Bedru/amharic-advanced-bpe-32k

Advanced Amharic Byte-Level BPE Tokenizer (32k)

An advanced, production-ready Byte-Level BPE tokenizer tailored for Amharic (አማርኛ) and generative LLM pre-training.

Key Features

• 32,000 Vocabulary: Highly optimized on the MasakhaNEWS Amharic corpus.
• Homophone Normalization: Redundant Ge'ez homophones (ሐ/ኀ → ሀ, ሠ → ሰ, ዐ → አ, ፀ → ጸ) are folded to prevent vocabulary waste.
• Math & Digit Isolation: Digits are strictly tokenized digit-by-digit for arithmetic and date reasoning (2024 → ['2', '0', '2', '4']).
• ChatML Native: Pre-equipped with <|im_start|>, <|im_end|>, and role tokens (<|system|>, <|user|>, <|assistant|>).
• Zero OOV: Byte-Level BPE guarantees 0% out-of-vocabulary fallback.

Fertility Benchmark

• Fertility Rate: ~1.37 - 1.50 tokens/word (Compared to 14.0 tokens/word on standard GPT-2).

Quick Usage

from transformers import AutoTokenizer

# 1. Load your published tokenizer from Hugging Face
tokenizer = AutoTokenizer.from_pretrained("Bedru/amharic-advanced-bpe-32k")

# =====================================================================
# 1. ENCODING
# =====================================================================
text = " ተማሪዎቹ ወደ ትምህርት ቤት ሄዱ።"

# Get raw token strings (byte representations)
raw_tokens = tokenizer.tokenize(text)
# Get numerical Token IDs
input_ids = tokenizer.encode(text)

print("--- ENCODING ---")
print(f"Original Text: {text}")
print(f"Raw Tokens:    {raw_tokens}")
print(f"Token IDs:     {input_ids}")

# =====================================================================
# 2. DECODING (Converting IDs back to readable text)
# =====================================================================
print("\n--- DECODING ---")

# A. Decode the full sequence back to original Amharic
decoded_full = tokenizer.decode(input_ids)
print(f"Full Decoded Text: '{decoded_full}'")

# B. Decode each token individually to inspect subword segmentation
decoded_subwords = [tokenizer.decode([token_id]) for token_id in input_ids]
print(f"Decoded Subwords:  {decoded_subwords}")

# =====================================================================
# 3. CHATML TEMPLATE (Instruction / Chat Formatting)
# =====================================================================
print("\n--- CHATML TEMPLATE ---")
messages = [
    {"role": "user", "content": "ሰላም፣ እንደምን ነዎት?"}
]

# Apply chat template
chat_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(chat_prompt)

# Tokenize and decode the ChatML prompt
chat_ids = tokenizer.encode(chat_prompt)
print("Chat Prompt Decoded Back:")
print(tokenizer.decode(chat_ids))