EmbeddingGemma 2 (English + Nepali, 65k vocabulary) is a vocabulary-trimmed version of EmbeddingGemma 2, a text-only embedding model developed by Google DeepMind. It maps text, including code, into a shared 768-dimensional vector space for semantic search, retrieval-augmented generation (RAG), classification, clustering, and semantic similarity.

The vocabulary was reduced from 262,144 to 65,536 tokens and the tokenizer was rebuilt to match, keeping the tokens that English and Nepali text actually use. Everything else in the model is unchanged. This cuts the embedding table by 75% (about 100.7M parameters) for a smaller download and memory footprint.

Supported Languages

Language	Status
English (`en`)	Supported. Vocabulary chosen from English web text.
Nepali (`ne`, Devanagari script)	Supported. A dedicated share of the vocabulary is reserved for Nepali.
Code and numbers	Common tokens kept; unusual identifiers split into more tokens.
All other languages	Not targeted. Their tokens were mostly removed, so text is still encodable but split into many small pieces, and quality is expected to drop. Use the original model for them.

Vocabulary Reduction

	Original	This model
Vocabulary size	262,144	65,536
Embedding parameters (65,536 x 512 vs 262,144 x 512)	134.2M	33.6M
Parameters removed		100.7M (75% of the embedding table)
Transformer layers, projection layer, pooling	unchanged	unchanged

Only memory and file size shrink. Inference speed is unchanged, because the embedding is a table lookup.

How the vocabulary was chosen

Token counts: token usage was counted with the original tokenizer on FineWeb-2 web text, 70,000 English and 30,000 Nepali documents.
Always kept: all special tokens and all 256 byte-fallback tokens (<0x00> to <0xFF>), so any input remains encodable.
Language quota: the 12,288 most-used Nepali tokens first, then English tokens by English frequency. Nepali concentrates in a small set of tokens, so a small quota covers it, and the larger remainder goes to English.
BPE completeness: every kept token keeps the smaller tokens it is merged from, so no kept token becomes impossible to produce.
Rebuilt tokenizer: same pieces with new contiguous ids, only the merge rules whose parts survive, and remapped special-token ids (bos, eos, pad) in the configs.

A trimmed token is not lost. The tokenizer splits it into smaller pieces, so text decodes back exactly, but rare English words and Nepali words outside the kept set cost a few more tokens.

Fidelity to the original model

Embeddings from this model compared with the original EmbeddingGemma 2 on 256 random documents per language (unit-length 768-d embeddings):

Metric	English	Nepali
Cosine similarity to original, mean	0.9948	about 1.000
Cosine similarity to original, minimum	0.962	0.969
Correlation of pairwise similarities with original	0.9786	0.9985
Pairwise similarity difference, mean absolute	0.0063	0.0012

These measure how close the trimmed model's embeddings stay to the original's, not task accuracy. The documents come from the same corpus the vocabulary was counted on, so English figures are slightly optimistic. Nepali cosine marginally above 1.0 is bf16 rounding in the output norms. The benchmark scores below are for the original model; they have not been re-measured for this one, so evaluate on your own English and Nepali retrieval data before relying on it.

Model Overview

Component	Specification
Architecture	Gemma-based text embedding model
Backbone	130M parameters
Embedder	140M parameters
Text Model Parameters	270M original; about 100.7M fewer after trimming
Layers	24
Model Dimension	512
Hidden Dimension	2048
Sliding Window	1024 tokens
Vocabulary Size	65,536 (original: 262,144)
Number of Heads	4
Number of KV-Heads (Local/Global)	2/1
Local:Global	5:1
Attention	GQA/MQA
Activation	Gated FFN with GELU
Pooling	Mean Pooling
Projection Layer	512→768
Supported Input	Text and code
Context Window	8,192 tokens
Native Output Dimension	768
MRL Truncation Dimensions	128, 256, 512

EmbeddingGemma 2 builds upon the architectural and capability advancements of Gemma 4, offering several core features:

Text-only embeddings: Generates semantic representations of text and code for text-based retrieval and understanding tasks. It does not support image, video, or audio inputs.
Efficient footprint: Uses a text backbone and embedder designed for efficient inference, with a vocabulary trimmed to English and Nepali.
Matryoshka Representation Learning (MRL): Supports truncated embeddings across 128d, 256d, 512d, and 768d, enabling up to a 6x reduction in vector storage costs with minimal impact on quality.
Context length: Supports an 8K-token context window.
Task-steered representations: Uses lightweight text instruction prefixes to optimize embeddings for different tasks, including search, classification, clustering, and semantic similarity.

Benchmark Results (original model)

EmbeddingGemma 2 was evaluated across text and code embedding benchmarks. Results reported below use the original full-vocabulary full-precision checkpoint. They were not re-run on the trimmed model.

Overall Evaluation Results (768d)

Modality	Benchmark	Metric	EmbeddingGemma 2	EmbeddingGemma 1
Text	Massive Text Embedding Benchmark (MTEB, multilingual, v2)	Mean(Task), Multiple	61.36	61.15
Code	Massive Text Embedding Benchmark (MTEB, code, v1)	Mean(Task), NDCG@10	78.68	68.76

Evaluation Results with Vector Truncation

With MRL, EmbeddingGemma 2 representations can be truncated below the native 768d to 128d, 256d, and 512d representations and re-normalized.

This reduces storage requirements, with minimal quality impact down to 256 dimensions. The 128-dimensional representation is best suited to text-only workloads.

Output Dimension	Compression Ratio	MTEB (multilingual, v2) Mean(Task)	MTEB (English, v2) Mean(Task)	MTEB (code, v1) Mean(Task)
768d (Full)	1:1	61.36	68.46	78.68
512d	1:1.5	61.17	68.41	77.24
256d	1:3	60.41	67.78	76.18
128d	1:6	57.89	65.68	71.41

Quick Start

Install the sentence-transformers library:

pip install -U sentence-transformers transformers

Generate text embeddings:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("luluw/embeddinggemmatext2-65k-en-ne-trim")

query = "What causes the northern lights?"
document = "The northern lights are caused by charged particles from the sun."

query_emb = model.encode(query, prompt_name="SearchQuery")
doc_emb = model.encode(document, prompt_name="Document")
print(model.similarity(query_emb, doc_emb))

Best Practices

For optimal embedding quality and runtime efficiency, follow these configurations and best practices.

1. Task Instruction Prefixes

EmbeddingGemma 2 is trained with short task instruction prefixes prepended to text inputs. Using the right prefix improves quality; omitting it still works but may reduce precision.

Documents with a real title should be formatted as title: {title} | text: {content}. Use title: none when no title is available.

Prefix Notation & Usage

There are two types of tasks:

Asymmetric tasks (e.g., retrieval): Use a query prefix for queries and a document prefix for corpus items.
Symmetric tasks (e.g., classification, similarity): Apply the same task prefix to all inputs being compared.

Use Case	Task Type	Prompt Name	Query Task Instruction	Document Task Instruction (use `none` if no title)
Web / document search	Asymmetric	SearchQuery	`task: search result \| query: {query}`	`title: {title} \| text: {content}`
Question answering	Asymmetric	QuestionAnswering	`task: question answering \| query: {question}`	`title: {title} \| text: {passage}`
Fact checking	Asymmetric	FactChecking	`task: fact checking \| query: {claim}`	`title: {title} \| text: {evidence}`
Code search	Asymmetric	CodeRetrieval	`task: code retrieval \| query: {query}`	`title: {title or filename} \| text: {code}`
Text classification	Symmetric	Classification	`task: classification \| query: {content}`	N/A
Clustering	Symmetric	Clustering	`task: clustering \| query: {content}`	N/A
Measuring similarity	Symmetric	SentenceSimilarity	`task: sentence similarity \| query: {content}`	N/A

Note: prompt_name="Document" applies title: none. Titled documents should still be formatted manually:

model.encode(f"title: {title} | text: {document}")

2. Matryoshka Dimension Truncation

EmbeddingGemma 2 is trained with Matryoshka Representation Learning, so the 768-dimensional output vector can be shortened by keeping only its leading dimensions. The supported dimensions are 768, 512, 256, and 128. Shorter vectors reduce storage requirements and can speed up similarity search at some cost to quality.

At runtime, follow these guidelines:

Re-normalize after truncating: Slicing a unit-length vector does not preserve unit length. The shortened vector must be L2-normalized before it is used for cosine similarity. Skipping this step can silently degrade ranking quality.
Queries and documents must share a dimension: A 768-dimensional query cannot be scored against a 128-dimensional corpus.

To avoid truncating and normalizing manually, use the truncate_dim and normalize_embeddings parameters when calling model.encode():

query_emb = model.encode(
    query,
    truncate_dim=128,  # or 512, 256
    normalize_embeddings=True,
)

Model quality at each dimension is reported in the truncation table above (original model). Quality remains close to the full representation down to 256 dimensions. The 128-dimensional representation should be validated against your workload before adoption.

3. Numerical Precision

Run inference in bfloat16 or float32. Do not use float16.

EmbeddingGemma 2's activation range exceeds the dynamic range of float16. Using float16 may produce NaN or silently degraded embeddings.

bfloat16 is safe and carries the same 8-bit exponent as float32. It is the recommended default on hardware with native support and halves memory use relative to float32. Use float32 elsewhere, including on most CPUs.

In SentenceTransformers, configure the data type through model_kwargs:

import torch
from sentence_transformers import SentenceTransformer

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = SentenceTransformer(
    "luluw/embeddinggemmatext2-65k-en-ne-trim",
    model_kwargs={"torch_dtype": dtype},
)

Model Data

The base model's pre-training dataset is a large-scale, diverse collection of text and code, with a cutoff date of January 2025. The training data includes:

Web documents: Diverse web text covering a wide range of linguistic styles, topics, and vocabulary. The training dataset includes content in over 140 languages.
Code: Programming languages and code-related text to improve understanding of code semantics.

The vocabulary trimming step used only token counts from English and Nepali web text (FineWeb-2); the model weights were not retrained.

Data Processing

Several data cleaning and filtering methods were applied to the base model's training data:

CSAM filtering: Filtering was applied at multiple stages in data preparation to exclude harmful and illegal content.
Sensitive data filtering: Automated techniques were used to filter certain personal information and other sensitive data from training sets.
Additional methods: Filtering based on content quality and safety in line with Google's AI responsibility policies.

Ethics & Safety

EmbeddingGemma 2 is a pre-trained embedding model. Unlike generative models, it does not undergo post-training alignment, safety tuning, or output-level moderation. Safety mitigations during development focused on pre-training data filtering to reduce exposure to harmful content and severe biases in the learned embedding space, in alignment with Google's AI Principles.

Since embedding models produce representations rather than user-facing text, safety risks manifest downstream in how those representations are used.

Developers and deployers are responsible for evaluating and implementing application-level safeguards, such as retrieval filtering and fairness testing, appropriate to their production use cases.

Deployments must adhere to the Gemma Prohibited Use Policy.

Usage and Limitations

Intended Usage

This model generates text embeddings for English and Nepali downstream applications, including:

Retrieval: Semantic search across text and code, including document search, RAG over enterprise knowledge bases, and code search from natural-language queries.
Classification: Classifying text into predefined categories, such as sentiment or content categories.
Clustering: Grouping documents and text by semantic similarity, organizing document collections, and clustering customer feedback.
Semantic similarity: Measuring similarity between text inputs for recommendation systems, duplicate detection, paraphrase identification, and English-Nepali content alignment.
Fact verification: Retrieving evidence documents for a claim in automated fact-checking systems and source attribution pipelines.

Limitations

Languages: Trimmed for English and Nepali. Other languages the original model supports are not targeted. Their text still encodes through byte fallback but becomes much longer and less accurate, so use the original model for them.
Trimming effects: Rare English words and uncommon Nepali words are split into more tokens (English needs a few percent more tokens on average), and embeddings differ slightly from the original (see the fidelity table).
Not re-benchmarked: Published benchmark scores belong to the original model.
Training data: The quality and diversity of the base model's training data significantly influence its capabilities. Biases or gaps in that data carry over.
Context and task complexity: Models perform best on tasks that can be framed with clear instructions. Open-ended or highly complex tasks may be challenging.
Language ambiguity and nuance: Natural language is inherently complex. The model may struggle with subtle nuances, sarcasm, or figurative language.
Task instruction usage: Omitting recommended task prefixes may lead to suboptimal embedding quality.

Ethical Considerations and Risks

Bias and fairness
- Models trained on large-scale, real-world text data can reflect sociocultural biases in their training material.
- Training data underwent safety filtering to mitigate these risks.
Misinformation and misuse
- Embeddings can be misused to retrieve, classify, or organize content in false, misleading, or harmful ways. See the Responsible Generative AI Toolkit.
Transparency and accountability
- This model card summarizes the model's architecture, capabilities, limitations, and evaluation processes.
- Open embedding models make text representation technology accessible to developers and researchers.

Risks Identified and Mitigations

Misuse for malicious purposes: Technical limitations, developer education, and end-user education can help mitigate malicious applications of embedding models.
Privacy violations: Models were trained on data filtered to remove certain personal information and other sensitive data. Developers are encouraged to follow privacy regulations and use privacy-preserving techniques.
Perpetuation of biases: Continuous monitoring, evaluation metrics, human review, and debiasing techniques are encouraged during model development and deployment.

Benefits

An efficient, smaller text embedding model for English and Nepali retrieval and understanding, with flexible embedding dimensions and a lighter footprint for consumer hardware. Designed to support responsible AI development, like other models in the Gemma family.