---
license: mit
language:
- da
- sv
- 'no'
base_model:
- AI-Sweden-Models/Llama-3-8B-instruct
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- text-embedding
- embeddings
- information-retrieval
- beir
- text-classification
- text-clustering
- llm2vec
datasets:
- jealk/scandi-wiki-combined
- jealk/wiki40b-da-clean
- jealk/supervised-da
- DDSC/nordic-embedding-training-data
---
## TTC-L2V-2 (Danish, Swedish and Norwegian)

### Model Description

Supervised model for sentence embeddings.

- **Developed by:** Jesper Alkestrup, The Tech Collective
- **Model type:** Embedding model
- **Language(s) (NLP):** Danish, Swedish and Norwegian
- **Finetuned from model:** AI-Sweden-Models/Llama-3-8B-instruct
- **Finetuning procedure:** LLM2Vec

Trained by using the approach outlined in the paper **LLM2Vec: Large Language
Models Are Secretly Powerful Text Encoders**.

- MNTP finetuning on: https://huggingface.co/datasets/jealk/scandi-wiki-combined
- Initial Supervised SimCSE on: https://huggingface.co/datasets/jealk/supervised-da
- Followed by Supervised SimCSE with in-batch hard-negatives and instructions on:
  https://huggingface.co/datasets/DDSC/nordic-embedding-training-data

### Usage

This is a [sentence-transformers](https://www.sbert.net/) model — it loads
directly and needs no packages beyond `sentence-transformers`. The
bidirectional Llama encoder and the LLM2Vec instruction-aware mean pooling are
provided by small custom modules in this repo, loaded via `trust_remote_code`.

```bash
pip install sentence-transformers
```

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jealk/TTC-L2V-supervised-2", trust_remote_code=True)

# Queries are encoded with an instruction; documents without one.
instruction = "Givet et spørgsmål, find relevante tekstudsnit, der besvarer det:"
queries = [
    "Hvordan påvirker søvn vores koncentrationsevne",
    "Hvad skal man være opmærksom på, når man køber brugt cykel",
]
documents = [
    "Forskning viser, at for lidt søvn kan nedsætte både koncentration og hukommelse. "
    "Allerede efter én nat med dårlig søvn kan man opleve problemer med at fokusere.",
    "Når du køber en brugt cykel, bør du tjekke, om stellet har skader eller rust, og "
    "om gear og bremser fungerer korrekt.",
]

q_emb = model.encode(queries, prompt=instruction)   # or prompt_name="query"
d_emb = model.encode(documents)

# Cosine similarity
print(model.similarity(q_emb, d_emb))
```

The instruction is also registered as a named prompt, so
`model.encode(queries, prompt_name="query")` applies the default Danish
instruction shown above. Documents are encoded without an instruction.

Pooling is mean pooling over the content tokens; the instruction and the
beginning-of-text token are excluded — exactly as in LLM2Vec.

### Model size and quantization

The model has ~8B parameters and loads in roughly 14 GB (bfloat16). To run on
smaller GPUs it can be loaded quantized via `bitsandbytes`:

```python
from sentence_transformers import SentenceTransformer
from transformers import BitsAndBytesConfig

model = SentenceTransformer(
    "jealk/TTC-L2V-supervised-2", trust_remote_code=True,
    model_kwargs={"quantization_config": BitsAndBytesConfig(load_in_8bit=True)},
)
```

Approximate memory footprint: **bfloat16 ≈ 14 GB**, **8-bit ≈ 8 GB**,
**4-bit ≈ 5 GB** (`load_in_4bit=True`).

> The quantized variants have not been evaluated on benchmarks — embedding
> quality may differ from the full-precision model.

### Notes

- `trust_remote_code=True` is required: it loads the bidirectional Llama
  encoder and the LLM2Vec-style pooling module shipped in this repo.

### Credits

Approach from **LLM2Vec** (McGill-NLP). Related model:
https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised