redis/model-b-structured

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("redis/model-b-structured")
# Run inference
sentences = [
    'What is the difference between economic growth and economic development?',
    'What is the difference between economic growth and economic development?',
    'the difference between economic growth and economic development is What?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 0.9999,  0.9999, -0.0751],
#         [ 0.9999,  0.9999, -0.0751],
#         [-0.0751, -0.0751,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.835
cosine_accuracy@3	0.9092
cosine_accuracy@5	0.9345
cosine_precision@1	0.835
cosine_precision@3	0.3031
cosine_precision@5	0.1869
cosine_recall@1	0.835
cosine_recall@3	0.9092
cosine_recall@5	0.9345
cosine_ndcg@10	0.8986
cosine_mrr@1	0.835
cosine_mrr@5	0.874
cosine_mrr@10	0.8779
cosine_map@100	0.88

Training Details

Training Dataset

Unnamed Dataset

• Size: 713,743 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 16.07 tokens max: 53 tokens	min: 6 tokens mean: 16.03 tokens max: 53 tokens	min: 6 tokens mean: 16.81 tokens max: 58 tokens

• Samples:

  anchor	positive	negative
  Which one is better Linux OS? Ubuntu or Mint?	Why do you use Linux Mint?	Which one is not better Linux OS ? Ubuntu or Mint ?
  What is flow?	What is flow?	What are flow lines?
  How is Trump planning to get Mexico to pay for his supposed wall?	How is it possible for Donald Trump to force Mexico to pay for the wall?	Why do we connect the positive terminal before the negative terminal to ground in a vehicle battery?

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 7.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 40,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 15.52 tokens max: 74 tokens	min: 6 tokens mean: 15.51 tokens max: 74 tokens	min: 6 tokens mean: 16.79 tokens max: 69 tokens

• Samples:

  anchor	positive	negative
  Why are all my questions on Quora marked needing improvement?	Why are all my questions immediately being marked as needing improvement?	For a post-graduate student in IIT, is it allowed to take an external scholarship as a top-up to his/her MHRD assistantship?
  Can blue butter fly needle with vaccum tube be reused? Is it HIV risk? . Heard the needle is too small to be reused . Had blood draw at clinic?	Can blue butter fly needle with vaccum tube be reused? Is it HIV risk? . Heard the needle is too small to be reused . Had blood draw at clinic?	Can blue butter fly needle with vaccum tube be reused not ? Is it HIV risk ? . Heard the needle is too small to be reused . Had blood draw at clinic ?
  Why do people still believe the world is flat?	Why are there still people who believe the world is flat?	I'm not able to buy Udemy course .it is not accepting mine and my friends debit card.my card can be used for Flipkart .how to purchase now?

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 7.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• weight_decay: 0.0001
• max_steps: 5000
• warmup_ratio: 0.1
• fp16: True
• dataloader_drop_last: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 1
• load_best_model_at_end: True
• optim: adamw_torch
• ddp_find_unused_parameters: False
• push_to_hub: True
• hub_model_id: redis/model-b-structured
• eval_on_start: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0001
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: 5000
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 1
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: redis/model-b-structured
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	val_cosine_ndcg@10
0	0	-	0.7379	0.8652
0.0448	250	0.6516	0.4058	0.8939
0.0897	500	0.4799	0.3740	0.8953
0.1345	750	0.4486	0.3556	0.8962
0.1793	1000	0.4324	0.3486	0.8964
0.2242	1250	0.4225	0.3411	0.8971
0.2690	1500	0.4115	0.3385	0.8969
0.3138	1750	0.405	0.3327	0.8970
0.3587	2000	0.4017	0.3296	0.8969
0.4035	2250	0.3954	0.3278	0.8973
0.4484	2500	0.3928	0.3259	0.8974
0.4932	2750	0.3868	0.3245	0.8980
0.5380	3000	0.3844	0.3223	0.8978
0.5829	3250	0.3833	0.3226	0.8982
0.6277	3500	0.3845	0.3209	0.8984
0.6725	3750	0.3807	0.3192	0.8986
0.7174	4000	0.3797	0.3189	0.8983
0.7622	4250	0.3773	0.3177	0.8984
0.8070	4500	0.3763	0.3173	0.8985
0.8519	4750	0.3782	0.3171	0.8985
0.8967	5000	0.3769	0.317	0.8986

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}