twinkle-ai/tw-privacy-guides
Viewer ā¢ Updated ā¢ 33 ā¢ 534
AI & ML interests
Traditional Chinese language models
Recent Activity
View all activity
cointeleportingĀ
updated a
dataset 5 days ago
Auto-update dataset - 2026-08-31
#59 opened 5 days ago
by 
whats2000
lianghsunĀ
updated a
model 9 days ago
lianghsunĀ
published a
model 9 days ago
lianghsunĀ
updated a
dataset 9 days ago
lianghsunĀ
published a
dataset 9 days ago
Post
153
š¹š¼ Releasing https://huggingface.co/lianghsun/tw-tokenizer-v1 ā a tokenizer trained from scratch for Traditional Chinese (Taiwan).
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex ā it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured Ļ = ā0.59), so we validated two levels deeper:
**Segmentation** ā boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
**Downstream** ā trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win ā with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info ā Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak TĆ¢i-lĆ“ support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
š https://huggingface.co/lianghsun/tw-tokenizer-v1
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex ā it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured Ļ = ā0.59), so we validated two levels deeper:
**Segmentation** ā boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
å°ę„ē“ é¤ćē¹č³Ŗęē¶å
¬ååÆ©ę„åŖå
ours: ['å°ę„ē“ é¤', 'ć', 'ē¹č³Ŗ', 'ęē¶', 'å
¬å', 'åÆ©ę„', 'åŖå']
Qwen: ['å°ę„', 'ē“ ', 'é¤', ...] ā ćē“ é¤ćsplit mid-word**Downstream** ā trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win ā with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info ā Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak TĆ¢i-lĆ“ support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
š https://huggingface.co/lianghsun/tw-tokenizer-v1
Auto-update dataset - 2026-08-24
#58 opened 13 days ago
by whats2000
Auto-update dataset - 2026-08-24
#58 opened 13 days ago
by whats2000
Auto-update dataset - 2026-08-17
#57 opened 20 days ago
by whats2000
Auto-update dataset - 2026-08-10
#56 opened 27 days ago
by whats2000
Auto-update dataset - 2026-08-03
#55 opened about 1 month ago
by whats2000
Auto-update dataset - 2026-07-27
#54 opened about 1 month ago
by whats2000
lianghsunĀ
updated a
collection about 2 months ago
Auto-update dataset - 2026-07-20
#53 opened about 2 months ago
by whats2000
Auto-update dataset - 2026-07-13
#52 opened about 2 months ago
by whats2000
Auto-update dataset - 2026-07-06
#51 opened 2 months ago
by whats2000
Auto-update dataset - 2026-06-29
#50 opened 2 months ago
by whats2000
Auto-update dataset - 2026-06-29
#50 opened 2 months ago
by whats2000
Auto-update dataset - 2026-06-15
#48 opened 3 months ago
by whats2000