You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

SentenceTransformer based on BidirLM/BidirLM-1.7B-Embedding

This is a sentence-transformers model finetuned from BidirLM/BidirLM-1.7B-Embedding on the json dataset. It maps inputs to a 2048-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BidirLM/BidirLM-1.7B-Embedding
  • Maximum Sequence Length: 768 tokens
  • Output Dimensionality: 2048 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text
  • Training Dataset:
    • json

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'unpad_inputs': False, 'architecture': 'BidirLMModel'})
  (1): Pooling({'embedding_dimension': 2048, 'pooling_mode': 'mean', 'include_prompt': True})
)

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 512
  • num_train_epochs: 1.0
  • learning_rate: 5e-06
  • lr_scheduler_type: cosine
  • warmup_steps: 102
  • weight_decay: 0.01
  • bf16: True
  • tf32: True
  • per_device_eval_batch_size: 512
  • push_to_hub: True
  • hub_private_repo: True
  • hub_model_id: Ehsanl/bidir1_7
  • hub_strategy: checkpoint
  • load_best_model_at_end: True
  • dataloader_num_workers: 2
  • dataloader_pin_memory: False
  • ddp_find_unused_parameters: False

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 512
  • num_train_epochs: 1.0
  • max_steps: -1
  • learning_rate: 5e-06
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: None
  • warmup_steps: 102
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.01
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: True
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 512
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: True
  • hub_private_repo: True
  • hub_model_id: Ehsanl/bidir1_7
  • hub_strategy: checkpoint
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: True
  • dataloader_num_workers: 2
  • dataloader_pin_memory: False
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: False
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Epoch Step Training Loss Validation Loss legal-val_cosine_ndcg@10
0.0010 1 5.9063 - -
0.0098 10 5.8853 - -
0.0196 20 5.8072 - -
0.0294 30 5.5096 - -
0.0392 40 5.2094 - -
0.0490 50 5.0760 - -
0.0588 60 4.8557 - -
0.0686 70 4.7069 - -
0.0784 80 4.5927 - -
0.0881 90 4.4650 - -
0.0979 100 4.3334 - -
0.1077 110 4.2259 - -
0.1175 120 4.1451 - -
0.1273 130 4.0719 - -
0.1371 140 4.0492 - -
0.1469 150 4.0118 - -
0.1567 160 3.9170 - -
0.1665 170 3.9248 - -
0.1763 180 3.9063 - -
0.1861 190 3.8142 - -
0.1959 200 3.7706 3.8215 0.4666
0.2057 210 3.7886 - -
0.2155 220 3.7550 - -
0.2253 230 3.6855 - -
0.2351 240 3.6606 - -
0.2449 250 3.6728 - -
0.2547 260 3.6288 - -
0.2644 270 3.6204 - -
0.2742 280 3.6124 - -
0.2840 290 3.5716 - -
0.2938 300 3.5839 - -
0.3036 310 3.5396 - -
0.3134 320 3.5435 - -
0.3232 330 3.4970 - -
0.3330 340 3.5371 - -
0.3428 350 3.5103 - -
0.3526 360 3.4554 - -
0.3624 370 3.4455 - -
0.3722 380 3.4430 - -
0.3820 390 3.4204 - -
0.3918 400 3.4238 3.4868 0.5028
0.4016 410 3.3979 - -
0.4114 420 3.4001 - -
0.4212 430 3.4010 - -
0.4310 440 3.3647 - -
0.4407 450 3.3943 - -
0.4505 460 3.3853 - -
0.4603 470 3.3212 - -
0.4701 480 3.2961 - -
0.4799 490 3.3305 - -
0.4897 500 3.3327 - -
0.4995 510 3.3011 - -
0.5093 520 3.2627 - -
0.5191 530 3.3178 - -
0.5289 540 3.2763 - -
0.5387 550 3.3026 - -
0.5485 560 3.2357 - -
0.5583 570 3.2510 - -
0.5681 580 3.2216 - -
0.5779 590 3.2096 - -
0.5877 600 3.2406 3.3413 0.5070
0.5975 610 3.1988 - -
0.6072 620 3.2589 - -
0.6170 630 3.2019 - -
0.6268 640 3.2406 - -
0.6366 650 3.2121 - -
0.6464 660 3.2094 - -
0.6562 670 3.1829 - -
0.6660 680 3.2034 - -
0.6758 690 3.1839 - -
0.6856 700 3.1409 - -
0.6954 710 3.1326 - -
0.7052 720 3.2099 - -
0.7150 730 3.1792 - -
0.7248 740 3.1536 - -
0.7346 750 3.1474 - -
0.7444 760 3.1774 - -
0.7542 770 3.1633 - -
0.7640 780 3.1324 - -
0.7738 790 3.1940 - -
0.7835 800 3.1512 3.2747 0.5222

Training Time

  • Training: 1.5 days
  • Evaluation: 1.5 hours
  • Total: 1.6 days

Framework Versions

  • Python: 3.11.16
  • Sentence Transformers: 6.1.0
  • Transformers: 5.17.0
  • PyTorch: 2.8.0+cu128
  • Accelerate: 1.15.0
  • Datasets: 5.0.1
  • Tokenizers: 0.23.2

Additional Resources

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

WeightedMultiPositiveCachedLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
-
Safetensors
Model size
2B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Ehsanl/bidir1_7

Finetuned
(2)
this model

Papers for Ehsanl/bidir1_7

Evaluation results