Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
Paper • 2101.06983 • Published • 3
How to use Ehsanl/bidir1_7 with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Ehsanl/bidir1_7", trust_remote_code=True)
sentences = [
"That is a happy person",
"That is a happy dog",
"That is a very happy person",
"Today is a sunny day"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]This is a sentence-transformers model finetuned from BidirLM/BidirLM-1.7B-Embedding on the json dataset. It maps inputs to a 2048-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'unpad_inputs': False, 'architecture': 'BidirLMModel'})
(1): Pooling({'embedding_dimension': 2048, 'pooling_mode': 'mean', 'include_prompt': True})
)
per_device_train_batch_size: 512num_train_epochs: 1.0learning_rate: 5e-06lr_scheduler_type: cosinewarmup_steps: 102weight_decay: 0.01bf16: Truetf32: Trueper_device_eval_batch_size: 512push_to_hub: Truehub_private_repo: Truehub_model_id: Ehsanl/bidir1_7hub_strategy: checkpointload_best_model_at_end: Truedataloader_num_workers: 2dataloader_pin_memory: Falseddp_find_unused_parameters: Falseper_device_train_batch_size: 512num_train_epochs: 1.0max_steps: -1learning_rate: 5e-06lr_scheduler_type: cosinelr_scheduler_kwargs: Nonewarmup_steps: 102optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.01adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Truefp16: Falsebf16_full_eval: Falsefp16_full_eval: Falsetf32: Truegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 512prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Truehub_private_repo: Truehub_model_id: Ehsanl/bidir1_7hub_strategy: checkpointhub_always_push: Falsehub_revision: Noneload_best_model_at_end: Trueignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Truedataloader_num_workers: 2dataloader_pin_memory: Falsedataloader_persistent_workers: Falsedataloader_prefetch_factor: Nonedataloader_multiprocessing_context: Nonedataloader_in_order: Trueremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Falseddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonelocal_rank: -1prompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}warmup_ratio: None| Epoch | Step | Training Loss | Validation Loss | legal-val_cosine_ndcg@10 |
|---|---|---|---|---|
| 0.0010 | 1 | 5.9063 | - | - |
| 0.0098 | 10 | 5.8853 | - | - |
| 0.0196 | 20 | 5.8072 | - | - |
| 0.0294 | 30 | 5.5096 | - | - |
| 0.0392 | 40 | 5.2094 | - | - |
| 0.0490 | 50 | 5.0760 | - | - |
| 0.0588 | 60 | 4.8557 | - | - |
| 0.0686 | 70 | 4.7069 | - | - |
| 0.0784 | 80 | 4.5927 | - | - |
| 0.0881 | 90 | 4.4650 | - | - |
| 0.0979 | 100 | 4.3334 | - | - |
| 0.1077 | 110 | 4.2259 | - | - |
| 0.1175 | 120 | 4.1451 | - | - |
| 0.1273 | 130 | 4.0719 | - | - |
| 0.1371 | 140 | 4.0492 | - | - |
| 0.1469 | 150 | 4.0118 | - | - |
| 0.1567 | 160 | 3.9170 | - | - |
| 0.1665 | 170 | 3.9248 | - | - |
| 0.1763 | 180 | 3.9063 | - | - |
| 0.1861 | 190 | 3.8142 | - | - |
| 0.1959 | 200 | 3.7706 | 3.8215 | 0.4666 |
| 0.2057 | 210 | 3.7886 | - | - |
| 0.2155 | 220 | 3.7550 | - | - |
| 0.2253 | 230 | 3.6855 | - | - |
| 0.2351 | 240 | 3.6606 | - | - |
| 0.2449 | 250 | 3.6728 | - | - |
| 0.2547 | 260 | 3.6288 | - | - |
| 0.2644 | 270 | 3.6204 | - | - |
| 0.2742 | 280 | 3.6124 | - | - |
| 0.2840 | 290 | 3.5716 | - | - |
| 0.2938 | 300 | 3.5839 | - | - |
| 0.3036 | 310 | 3.5396 | - | - |
| 0.3134 | 320 | 3.5435 | - | - |
| 0.3232 | 330 | 3.4970 | - | - |
| 0.3330 | 340 | 3.5371 | - | - |
| 0.3428 | 350 | 3.5103 | - | - |
| 0.3526 | 360 | 3.4554 | - | - |
| 0.3624 | 370 | 3.4455 | - | - |
| 0.3722 | 380 | 3.4430 | - | - |
| 0.3820 | 390 | 3.4204 | - | - |
| 0.3918 | 400 | 3.4238 | 3.4868 | 0.5028 |
| 0.4016 | 410 | 3.3979 | - | - |
| 0.4114 | 420 | 3.4001 | - | - |
| 0.4212 | 430 | 3.4010 | - | - |
| 0.4310 | 440 | 3.3647 | - | - |
| 0.4407 | 450 | 3.3943 | - | - |
| 0.4505 | 460 | 3.3853 | - | - |
| 0.4603 | 470 | 3.3212 | - | - |
| 0.4701 | 480 | 3.2961 | - | - |
| 0.4799 | 490 | 3.3305 | - | - |
| 0.4897 | 500 | 3.3327 | - | - |
| 0.4995 | 510 | 3.3011 | - | - |
| 0.5093 | 520 | 3.2627 | - | - |
| 0.5191 | 530 | 3.3178 | - | - |
| 0.5289 | 540 | 3.2763 | - | - |
| 0.5387 | 550 | 3.3026 | - | - |
| 0.5485 | 560 | 3.2357 | - | - |
| 0.5583 | 570 | 3.2510 | - | - |
| 0.5681 | 580 | 3.2216 | - | - |
| 0.5779 | 590 | 3.2096 | - | - |
| 0.5877 | 600 | 3.2406 | 3.3413 | 0.5070 |
| 0.5975 | 610 | 3.1988 | - | - |
| 0.6072 | 620 | 3.2589 | - | - |
| 0.6170 | 630 | 3.2019 | - | - |
| 0.6268 | 640 | 3.2406 | - | - |
| 0.6366 | 650 | 3.2121 | - | - |
| 0.6464 | 660 | 3.2094 | - | - |
| 0.6562 | 670 | 3.1829 | - | - |
| 0.6660 | 680 | 3.2034 | - | - |
| 0.6758 | 690 | 3.1839 | - | - |
| 0.6856 | 700 | 3.1409 | - | - |
| 0.6954 | 710 | 3.1326 | - | - |
| 0.7052 | 720 | 3.2099 | - | - |
| 0.7150 | 730 | 3.1792 | - | - |
| 0.7248 | 740 | 3.1536 | - | - |
| 0.7346 | 750 | 3.1474 | - | - |
| 0.7444 | 760 | 3.1774 | - | - |
| 0.7542 | 770 | 3.1633 | - | - |
| 0.7640 | 780 | 3.1324 | - | - |
| 0.7738 | 790 | 3.1940 | - | - |
| 0.7835 | 800 | 3.1512 | 3.2747 | 0.5222 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{oord2019representationlearningcontrastivepredictive,
title={Representation Learning with Contrastive Predictive Coding},
author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
year={2019},
eprint={1807.03748},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/1807.03748},
}
Base model
Qwen/Qwen3-1.7B-Base