---
language:
- bal
license: apache-2.0
base_model: xlm-roberta-base
tags:
- balochi
- low-resource
- masked-language-modeling
- domain-adaptive-pretraining
- xlm-roberta
- transformers
- pytorch
- fill-mask
pipeline_tag: fill-mask
library_name: transformers
metrics:
- perplexity
- loss
widget:
- text: "اے دوئیں دبستانانی <mask> دیگءَ پرکے ھست۔"
  example_title: "Sentence Completion 1"
- text: "پاکستان ءِ ردءَ اے <mask> باز ارزښت دار اِنت۔"
  example_title: "Sentence Completion 2"
- text: "بلۆچی زبان ءِ <mask> زانتکار"
  example_title: "Sentence Completion 3"
model-index:
- name: BalBERT
  results:
  - task:
      type: fill-mask
      name: Masked Language Modeling
    dataset:
      name: Balochi Monolingual Corpus (sentences_clean.txt)
      type: text
    metrics:
    - type: perplexity
      value: 5.86
      name: Validation Perplexity
    - type: loss
      value: 1.7681
      name: Validation Loss
---

<div align="center">

# 🌟 BalBERT (`shahbakhsh/BalBERT`)

### *The Foundational Research-Grade Transformer Encoder for Balochi NLP*

[![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-shahbakhsh%2FBalBERT-ffc107.svg?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/shahbakhsh/BalBERT)
[![GitHub Repository](https://img.shields.io/badge/GitHub-shah--bakhsh%2FBalBERT-181717.svg?style=for-the-badge&logo=github&logoColor=white)](https://github.com/shah-bakhsh/BalBERT)
[![License](https://img.shields.io/badge/License-Apache_2.0-22c55e.svg?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)
[![Parameters](https://img.shields.io/badge/Parameters-278M-6366f1.svg?style=for-the-badge)]()
[![Perplexity Drop](https://img.shields.io/badge/Perplexity%20Drop-97.8%25-06b6d4.svg?style=for-the-badge)]()

</div>

---

## 📌 Model Overview

**BalBERT** is the first research-grade Transformer encoder domain-adapted specifically for the **Balochi language** (`bal`). Built via continuous **Domain-Adaptive Pretraining (DAPT)** of `xlm-roberta-base` on a normalized monolingual corpus of **97,624 Balochi sentences**, BalBERT provides a domain-adapted foundation model for all downstream Balochi natural language processing tasks.

* **Author / Developer:** Shah Bakhsh ([@shahbakhsh](https://huggingface.co/shahbakhsh))
* **Model Type:** Masked Language Model (Transformer Encoder)
* **Language:** Balochi (`bal`)
* **Base Architecture:** `xlm-roberta-base`
* **License:** Apache-2.0
* **Repository:** [github.com/shah-bakhsh/BalBERT](https://github.com/shah-bakhsh/BalBERT)

---

## 🚀 Benchmark & Performance Highlights

| Metric | `xlm-roberta-base` (Baseline) | **BalBERT** (Ours) | Improvement |
| :--- | :---: | :---: | :---: |
| **MLM Loss** | 5.8100 | **1.7681** | **-69.6% Reduction** |
| **Perplexity** | 333.63 | **7.40** (Val: **5.86**) | **+97.8% Drop** |
| **Masked Prediction Quality** | Generic Punctuation (`،`) | Contextual Balochi Words (`اے`, `آئی`) | **Linguistically Coherent** |
| **Embedding Affinity** | Collapsed uniform score | High dynamic margin | **Enhanced Representations** |

---

## 💻 Quick Start & Code Examples

### 1. Fill-Mask Pipeline (Inference)

You can run token completion directly using the Hugging Face `pipeline`:

```python
from transformers import pipeline

# Load pipeline directly from Hugging Face Hub
unmasker = pipeline("fill-mask", model="shahbakhsh/BalBERT")

# Predict masked word in Balochi
text = "اے دوئیں دبستانانی <mask> دیگءَ پرکے ھست۔"
results = unmasker(text)

print("Top Predictions:")
for res in results[:5]:
    print(f"Token: {res['token_str']:<15} | Score: {res['score']:.4f}")
```

---

### 2. Direct PyTorch Model & Tokenizer API

```python
import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

MODEL_ID = "shahbakhsh/BalBERT"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForMaskedLM.from_pretrained(MODEL_ID)

text = "پاکستان ءِ ردءَ اے <mask> باز ارزښت دار اِنت۔"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits

mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
mask_logits = logits[0, mask_token_index, :]
top_5_tokens = torch.topk(mask_logits, 5, dim=1).indices[0].tolist()

print("Top Mask Completions:")
for token_id in top_5_tokens:
    print(f" • {tokenizer.decode([token_id])}")
```

---

### 3. Extracting Contextual Sentence Embeddings

```python
import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("shahbakhsh/BalBERT")
model = AutoModel.from_pretrained("shahbakhsh/BalBERT")

sentence = "بلۆچی زبان ءِ شَرّێن زانتکار"
inputs = tokenizer(sentence, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    # Tensor Shape: [batch_size, sequence_length, 768]
    last_hidden_state = outputs.last_hidden_state

print("Embedding Shape:", last_hidden_state.shape)
```

---

## 📊 Evaluation & Qualitative Analysis

### Masked Word Prediction Comparison

Given the Balochi sentence:
> `اے جستءِ پسو گپءُ درجنی دگہ جنگے رودین ایت کہ ادا <mask> ءِ سرجمیں پہناتانی گیشواری المی نہ اِنت۔`

```
xlm-roberta-base Top Predictions:
1. ، (4.5%) 
2. ِ (3.0%) 
3. کی (2.9%) 
4.   (2.3%) 
5. ، (2.0%) 

BalBERT Top Predictions (Ours):
1. اے     (28.4%) [Demonstrative: "this"]
2. آئی    (28.3%) [Pronoun: "its/his/her"]
3. انسانی (13.1%) [Adjective: "human"]
4. پہ     (2.5%)  [Preposition: "for"]
5. کہ     (1.6%)  [Conjunction: "that"]
```

---

## ⚙️ Model Specifications & Training Configuration

### Architecture Specs

* **Base Model:** XLM-RoBERTa Encoder (`xlm-roberta-base`)
* **Total Parameters:** **278,295,186** (~278M)
* **Hidden Layers:** 12
* **Attention Heads:** 12
* **Hidden Dimension:** 768
* **Vocab Size:** 250,002
* **Max Sequence Length:** 128 tokens

### Pretraining Hyperparameters

* **Objective:** Masked Language Modeling ($p=0.15$ dynamic masking)
* **Learning Rate:** `5e-5` (Linear decay)
* **Warmup Ratio:** `0.06`
* **Weight Decay:** `0.05`
* **Per-Device Batch Size:** 16 (Gradient Accumulation: 2 -> Effective Batch Size: 32)
* **Precision:** `bf16` Mixed Precision
* **Hardware:** NVIDIA Tesla T4 GPU (16 GB VRAM) under CUDA 12.8

### Dataset Overview

* **Source File:** `sentences_clean.txt`
* **Cleaned Sentences:** **97,624 sentences**
* **Train / Val Split:** 95% Train (**92,742 sentences**) / 5% Validation (**4,882 sentences**)
* **Mean Length:** 91.3 chars / 19.8 words per sentence
* **Seed:** `42`

---

## 🗺 Balochi NLP Ecosystem Roadmap

BalBERT acts as the pre-trained feature extractor for the entire Balochi NLP stack:

$$\text{BalCorpus} \longrightarrow \text{BalTokenizer} \longrightarrow \mathbf{BalBERT} \longrightarrow \text{BalPOS v2} \longrightarrow \text{BalNER} \longrightarrow \text{BalMorph} \longrightarrow \text{BalParser}$$

- [x] **BalCorpus**: Curated large-scale monolingual Balochi text dataset.
- [x] **BalTokenizer**: Native Balochi BPE tokenizer.
- [x] **BalBERT**: Domain-adapted transformer backbone encoder.
- [ ] **BalPOS v2** *(Next Release)*: Universal Part-of-Speech tagger fine-tuned on `shahbakhsh/BalBERT`.
- [ ] **BalNER**: Named Entity Recognition model for Balochi.
- [ ] **BalMorph**: Morphological analyzer.
- [ ] **BalParser**: Dependency parser based on Universal Dependencies (UD).

---

## ⚠️ Intended Use & Limitations

### Intended Use
* Fine-tuning on downstream Balochi token classification (POS, NER) and text classification tasks.
* Extracting contextual token and sentence embeddings for Balochi NLP research.

### Limitations
* **Encoder-Only:** BalBERT is a 278M parameter encoder model and cannot generate long-form text.
* **Text Domain:** Trained primarily on written Balochi text; oral dialects or non-standard spelling variants may require additional fine-tuning.

---

## 📑 Citation

```bibtex
@misc{shahbakhsh2026balbert,
  author       = {Shah Bakhsh},
  title        = {BalBERT: Domain-Adaptive Pretraining of XLM-RoBERTa for Balochi Natural Language Processing},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/shahbakhsh/BalBERT}},
  note         = {GitHub Repository: shah-bakhsh/BalBERT}
}
```

---

## 📬 Author & Contact

* **Author:** Shah Bakhsh
* **Hugging Face:** [@shahbakhsh](https://huggingface.co/shahbakhsh)
* **GitHub:** [@shah-bakhsh](https://github.com/shah-bakhsh)
* **Repository:** [github.com/shah-bakhsh/BalBERT](https://github.com/shah-bakhsh/BalBERT)