Post
70
๐น๐ผ Releasing lianghsun/tw-tokenizer-v1 โ a tokenizer trained from scratch for Traditional Chinese (Taiwan).
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex โ it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ฯ = โ0.59), so we validated two levels deeper:
**Segmentation** โ boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
**Downstream** โ trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win โ with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info โ Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tรขi-lรด support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
๐ lianghsun/tw-tokenizer-v1
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex โ it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ฯ = โ0.59), so we validated two levels deeper:
**Segmentation** โ boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
ๅฐๆฅญ็ด ้คใ็น่ณชๆ็ถๅ
ฌๅๅฏฉๆฅๅชๅ
ours: ['ๅฐๆฅญ็ด ้ค', 'ใ', '็น่ณช', 'ๆ็ถ', 'ๅ
ฌๅ', 'ๅฏฉๆฅ', 'ๅชๅ']
Qwen: ['ๅฐๆฅญ', '็ด ', '้ค', ...] โ ใ็ด ้คใsplit mid-word**Downstream** โ trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win โ with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info โ Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tรขi-lรด support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
๐ lianghsun/tw-tokenizer-v1
