🌐 EN · 🇷🇸 SR · 🇭🇷 HR · 🇧🇦 BS · 🇲🇰 MK · 🇸🇮 SL · 🇦🇱 SQ · 🇲 CNR · 🇧🇬 BG · 🇬🇷 EL · 🇹🇷 TR · 🇷🇴 RO · 🇭🇺 HU
Zora v1.12 — an open, honest LLM for the Balkans & Southeast Europe
зора = "dawn". One to unite them all. — by Sovasoft (ai.in.rs)
1 · What Zora is
Zora is an open 8B language model (built on Qwen3-8B) for 12 languages of the Balkans and Southeast Europe: Serbian, Croatian, Bosnian, Macedonian, Slovenian, Albanian, Montenegrin, Bulgarian, Greek, Turkish, Romanian, Hungarian.
Zora is not built to be the biggest model — it is built to be honest, in-language, and multi-perspective:
- thinks in the target language instead of pivoting through English,
- shows several perspectives on contested topics instead of one national view,
- and above all: admits when it doesn't know instead of inventing facts.
2 · The development story (v1.0 → v1.1 → v1.11 → v1.12)
| Version | Languages | BalkanBench | State |
|---|---|---|---|
| v1.0 | 6 | — | first public release |
| v1.1 | 12 | — | trained from scratch — but hallucinated facts (invented book titles, wrong authors). Never released. |
| v1.11 | 12 | 84/156 | the honest fix: says "I don't know", searches when unsure. #1 Balkan model. |
| v1.12 | 12 | 85/156 | the depth fix: better tool-calling, structured IDK, in-language thinking, RAG integration. |
v1.1 taught us the key lesson — a small model can't memorize every fact, so instead of faking it, v1.11 was retrained to be honest. v1.12 builds on that with deeper training and RAG.
3 · What's New in v1.12
Three Fixes from v1.11
Fix 1: IDK Mass Training (30-40% of SFT data)
- v1.11 had only 9% "I don't know" examples → model still guessed on unknowns
- v1.12: 216 curated IDK examples × 3 difficulty levels × 12 languages
- Result: Structured native-language refusals with reasoning ("Nemam pouzdanih podataka... neću da izmišljam")
Fix 2: Tool-Calling Cascade (769 examples)
- Priority chain: RAG (local docs) → web_search → IDK
- v1.11 had ZERO tool-calling examples (4get-cleanup removed everything)
- Result: SEARCH 4/12 → 7/12 (+3), TOOLBASE 11/12 → 12/12 (+1)
Fix 3: In-Language Thinking Traces (10K synthetic via Gemini)
- Reasoning in the target language (Serbian thinks Serbian, Croatian thinks Croatian)
- v1.11 had empty think blocks — no reasoning data at all
- Result: Better structured responses across all axes
Additional Improvements
- MAXLEN 8192 (8× longer than v1.11's 1024) — longer context, thinking traces have room
- RAG Integration — Zora can now use Retrieval-Augmented Generation
- CPT capped at 150 steps — faster, more stable training
4 · Benchmark (BalkanBench, 13 axes × 12 languages)
🔬 BalkanBench is open — test any model yourself: https://github.com/olivilo/balkanbench Deterministic scoring (script / language / keywords / numbers).
Axis-by-Axis Comparison (v1.11 → v1.12)
| Axis | v1.11 | v1.12 | Δ | What changed |
|---|---|---|---|---|
| FACT | 1/12 | 0/12 | ↓1 | 8B capacity limit; IDK now says "I don't know" instead of guessing |
| HALLU | 10/12 | 10/12 | = | Quality improved: structured native-language refusals (see Deep Dive below) |
| DETAIL | 8/12 | 10/12 | ↑2 | Better at recognizing fabricated content — IDK training at work |
| GRADED | 0/12 | 0/12 | = | Partial knowledge + honest uncertainty still hard for 8B |
| TEACH | 12/12 | 12/12 | = | Perfect — remains a core strength |
| REASON | 11/12 | 11/12 | = | Strong arithmetic reasoning |
| LOGIC | 0/12 | 0/12 | = | 8B capacity limit — needs v2 (27B) |
| LOGIC2 | 0/12 | 0/12 | = | Same as LOGIC |
| ANALYSIS | 0/12 | 0/12 | = | Same as LOGIC |
| INSTRUCT | 12/12 | 11/12 | ↓1 | Minor regression, within noise |
| LONGFORM | 12/12 | 12/12 | = | Perfect — remains a core strength |
| SEARCH | 4/12 | 7/12 | ↑3 | Tool-cascade works: RAG → web_search → IDK |
| TOOLBASE | 11/12 | 12/12 | ↑1 | Perfect: answers basics without calling tools |
| TOTAL | 81/156 | 85/156 | +4 |
Per-Language Scores
| Language | v1.11 | v1.12 | Δ |
|---|---|---|---|
| sq (Albanian) | 6/13 | 8/13 | +2 |
| cnr (Montenegrin) | 6/13 | 8/13 | +2 |
| hu (Hungarian) | 6/13 | 8/13 | +2 |
| bg (Bulgarian) | 7/13 | 8/13 | +1 |
| bs (Bosnian) | 8/13 | 8/13 | = |
| hr (Croatian) | 8/13 | 8/13 | = |
| ro (Romanian) | 7/13 | 7/13 | = |
| tr (Turkish) | 7/13 | 7/13 | = |
| el (Greek) | 8/13 | 7/13 | -1 |
| sr (Serbian) | 8/13 | 7/13 | -1 |
| mk (Macedonian) | 5/13 | 5/13 | = |
| sl (Slovenian) | 5/13 | 4/13 | -1 |
Biggest winners: Albanian, Montenegrin, Hungarian (+2 each) — the languages that benefited most from IDK + tool-training.
Charts
| Ranking | Evolution | Axis Matrix |
|---|---|---|
 |
 |
 |
| Delta (v1.11 → v1.12) | What Each Axis Tests |
|---|---|
 |
 |
5 · Deep Dive: Why HALLU Stayed at 10/12
The HALLU score (10/12) didn't change numerically — but the quality of how Zora says "I don't know" improved dramatically. Here's why the score stayed flat while the behavior improved, and what it would take to reach 12/12.
Why the Score Didn't Move
1. The 10/12 were already good. v1.11 already achieved 10/12 on HALLU. The test asks: "Does the model say one of the IDK marker words when asked about a fabricated person?" v1.11 already did that correctly for 10 of 12 languages. The last 2 languages (Macedonian, Slovenian) have the smallest training data — an 8B model simply doesn't have enough capacity for these underrepresented languages.
2. IDK training improved QUALITY, not SCORE. The BalkanBench HALLU test only checks: "Does the model say 'ne znam' / 'ne mogu da potvrdim' / etc.?" — a binary yes/no. What actually improved:
| Before (v1.11) | After (v1.12) |
|---|---|
| Short, sometimes truncated refusals | Full-sentence, structured refusals |
| Sometimes answered in English | Always answers in the question's language |
| No reasoning given | Explains why it can't answer |
| "Ne znam." | "Nemam pouzdanih podataka o 'X'. Ne mogu da potvrdim da postoji u pouzdanim izvorima, pa neću da izmišljam." |
This is a qualitative leap — the model sounds more natural, more trustworthy, and more helpful. But the binary score can't capture that.
3. The real hallucination improvement is in DETAIL (+2). DETAIL measures something harder: "A real author wrote a book that doesn't exist — does the model invent a plot?" v1.12 went from 8→10/12 here. This is where IDK training shows its real value — the model now recognizes it cannot describe a non-existent work, instead of making something up. The two new winners: Bulgarian and Hungarian.
4. LOGIC/ANALYSIS = 0/12 is a reasoning problem, not a hallucination problem. These axes test multi-step logic (cats-and-mice riddles, percentage calculations). The model doesn't hallucinate — it genuinely can't do the math. This is an 8B capacity limit, not a training issue.
What Would Move HALLU to 12/12
| Approach | Expected Impact | Effort |
|---|---|---|
| Larger model (v2 = 27B) | +1-2 languages (mk, sl) | High (new training run) |
| More IDK examples for mk/sl specifically | +0-1 languages | Medium (data generation) |
| RLHF with human feedback on refusal quality | Better quality (not score) | High (human annotation) |
| DPO (Direct Preference Optimization) | +1-2 languages | Medium (preference pairs) |
| More CPT data for mk/sl | +0-1 languages | High (data collection) |
Bottom line: The 8B model is near its ceiling for HALLU. The real gains in v2 (27B) will come from more parameters, not more training tricks.
6 · 🆕 RAG Feature
New in v1.12: Zora integrates with RAG (Retrieval-Augmented Generation) — a system that lets Zora search through a local knowledge base before answering.
What RAG gives Zora
- 87,284 chunks across 12 languages: Wikidata, Wikipedia, News Archive, EU Law, Statistics
- Live endpoint: https://rag.ai.in.rs
- Self-hostable: Clone the pipeline from https://github.com/olivilo/zora-v1.12
How it works
The tool-cascade: Zora first checks its RAG knowledge base (local documents, laws, statistics), then falls back to web search if needed, and finally says "I don't know" if neither helps.
User question → RAG (local docs) → web_search (live) → IDK (honest refusal)
Why this matters
- 8B models can't memorize everything — RAG gives Zora access to current, authoritative data without retraining
- Every answer carries source + date + license — full transparency
- Self-hostable: any organization can run their own Zora RAG with their own documents
7 · Training Details
| Parameter | Value |
|---|---|
| Base model | Qwen3-8B (Alibaba Cloud, Apache-2.0) |
| CPT steps | 150 (capped, not full epoch) |
| SFT examples | 9,379 (2 epochs) |
| MAXLEN | 8192 (8× longer than v1.11) |
| QLoRA | r=16, lora_alpha=16, 4bit |
| Data composition | 30-40% IDK, 15% Tool-calling, 10% Thinking, 35-45% Standard tasks |
| Infrastructure | Modal A100-80GB, ~4h total, ~$5-10 |
| Quantizations | Q5_K_M (5.4GB, recommended), Q6_K (6.7GB), Q8_0 (8.7GB) |
8 · Usage
Ollama (recommended):
ollama pull olivilo/zora:v1.12
ollama run olivilo/zora:v1.12
HuggingFace Transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sovasoft/zora-v1.12")
model = AutoModelForCausalLM.from_pretrained("sovasoft/zora-v1.12", device_map="auto")
GGUF (llama.cpp / Ollama manual): Download Q5_K_M, Q6_K, or Q8_0 from HuggingFace. Avoid Q4 and below — heavy quantization made the model hallucinate in our tests.
9 · Limitations
- 8B capacity: FACT, LOGIC, ANALYSIS are structurally weak — more parameters needed (v2 = 27B)
- Quantization: use Q5_K_M / Q6_K / Q8_0 only. Q4 and below degrade honesty.
- Smaller languages (mk, sl) have less training data — expect lower quality
- No real-time knowledge without RAG/web-search — the model's memory has a cutoff date
- Multi-step reasoning is unreliable — always verify critical calculations
10 · Benchmark Transparency & Limitations
BalkanBench is Sovasoft's own benchmark — designed, built, and scored by the same team that built Zora. This means:
- Design bias: The 13 axes (FACT, HALLU, DETAIL, etc.) were chosen to highlight Zora's strengths. A different benchmark design would produce different rankings.
- Scoring bias: The scoring functions in
matrix_ollama.pyare our own. How we define "correct" may favor Zora's training profile. - No frontier comparison: We compare only against open models (7-32B). Frontier models (GPT-4, Claude, Gemini) would outperform Zora — this benchmark is designed to evaluate within the open-source Balkan model ecosystem.
- Selection bias: We include models where Zora competes well. Inclusion criteria are not random.
- Training data overlap: Some benchmark questions may overlap with Zora's training data, which could inflate scores.
What the scores DO show: Zora v1.12 is the strongest open-source model we tested on our benchmark for 12 Balkan languages. It outperforms 3-4× larger models on BalkanBench v1.1 — a meaningful result for the open-source ecosystem, but not a claim of universal superiority.
What the scores do NOT show: That Zora is better than frontier models, that these rankings generalize beyond our test design, or that the scoring methodology is independent.
10 · What's Next: v2
| v1.12 (now) | v2 (planned) | |
|---|---|---|
| Base | Qwen3-8B | Qwen3.8-27B |
| BalkanBench | 85/156 | Target: 100+/156 |
| LOGIC/ANALYSIS | 0/12 | Target: 4-6/12 |
| HALLU | 10/12 | Target: 12/12 |
| Reasoning | Basic | Full chain-of-thought training |
11 · Acknowledgements
Zora exists because of open source. We give our formal, heartfelt thanks:
- Above all, to the Qwen team at Alibaba — for developing and open-sourcing Qwen3 (Apache-2.0), the foundation model Zora is built upon. Without their generosity, Zora would not exist.
- To the platforms and structures that made this possible — Kaggle, Modal, HuggingFace, Ollama, Unsloth — for the compute, the tools, and the open infrastructure.
- To the open-source community, for the models, code, and knowledge freely shared with everyone.
- To the people of the Balkans — whose languages, voices, stories and perspectives are Zora's very heart.
- To rag.ai.in.rs for the RAG infrastructure and 87,284 chunks of Balkan knowledge.
- And to all that is.
зора — the dawn belongs to everyone.
12 · Citation
@software{zora_v112,
author = {Vignjevic, Oliver},
title = {Zora v1.12: An Open, Honest LLM for the Balkans \& Southeast Europe},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/sovasoft/zora-v1.12},
license = {Apache-2.0},
base_model = {Qwen/Qwen3-8B},
languages = {sr, hr, bs, mk, sl, sq, cnr, bg, el, tr, ro, hu}
}
Sovasoft · ai.in.rs · one to unite them all
- Downloads last month
- 908