README
๐
AI & ML interests
None defined yet.
Recent Activity
lianghsunย
updated a
Space 7 days ago
lianghsunย
updated a
collection 7 days ago
lianghsunย
updated 2
datasets 7 days ago
lianghsunย
published a
dataset 10 days ago
lianghsunย
updated a
dataset 10 days ago
Post
386
๐น๐ผ Releasing https://huggingface.co/lianghsun/tw-tokenizer-v1 โ a tokenizer trained from scratch for Traditional Chinese (Taiwan).
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex โ it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ฯ = โ0.59), so we validated two levels deeper:
**Segmentation** โ boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
**Downstream** โ trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win โ with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info โ Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tรขi-lรด support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
๐ https://huggingface.co/lianghsun/tw-tokenizer-v1
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex โ it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ฯ = โ0.59), so we validated two levels deeper:
**Segmentation** โ boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
ๅฐๆฅญ็ด ้คใ็น่ณชๆ็ถๅ
ฌๅๅฏฉๆฅๅชๅ
ours: ['ๅฐๆฅญ็ด ้ค', 'ใ', '็น่ณช', 'ๆ็ถ', 'ๅ
ฌๅ', 'ๅฏฉๆฅ', 'ๅชๅ']
Qwen: ['ๅฐๆฅญ', '็ด ', '้ค', ...] โ ใ็ด ้คใsplit mid-word**Downstream** โ trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win โ with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info โ Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tรขi-lรด support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
๐ https://huggingface.co/lianghsun/tw-tokenizer-v1
Post
4,805
With the arrival of Twinkle April โ Twinkle AIโs annual open-source celebration held every April โ our community is excited to unveil its very first project:
๐ Twinkle Eval (https://github.com/ai-twinkle/Eval), a next-generation evaluation tool led by our contributor @tedslin .
Unlike traditional evaluation tools like iKalaโs ievals (https://github.com/ikala-ai/ievals), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient ๐ฒ โ for example, evaluating LRMs on the ikala/tmmluplus benchmark could take *
half a day without finishing.
One question we were especially curious about:
Does shuffling multiple-choice answer order impact model accuracy? ๐ค
โ See: "Change Answer Order Can Decrease MMLU Accuracy" โ arXiv:2406.19470v1
To address these challenges, Twinkle Eval brings three key innovations to the table:
1๏ธโฃ Parallelized evaluation of samples
2๏ธโฃ Multi-round testing for stability
3๏ธโฃ Randomized answer order to test robustness
After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15ร ๐๐. Interestingly, most models scored slightly lower under the 2๏ธโฃ3๏ธโฃ test settings compared to their claimed performance โ suggesting further benchmarking is needed.
This framework also comes with additional tunable parameters and detailed logging of LM behavior per question โ perfect for those who want to dive deeper. ๐
If you find Twinkle Eval useful, please โญ the project and help spread the word ๐ค
- 5 replies
Post
4,832
๐ Let me introduce the work I've done over the past three months: ๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ง๐ฎ๐ถ๐๐ฎ๐ป-๐ฏ๐ and ๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ง๐ฎ๐ถ๐๐ฎ๐ป-๐ฏ๐-๐๐ป๐๐๐ฟ๐๐ฐ๐, now open-sourced on ๐ค Hugging Face.
๐น๐ถ๐ฎ๐ป๐ด๐ต๐๐๐ป/๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ง๐ฎ๐ถ๐๐ฎ๐ป-๐ฏ๐: This model is built on top of ๐บ๐ฒ๐๐ฎ-๐น๐น๐ฎ๐บ๐ฎ/๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ฏ๐ with continual pretraining. The training dataset consists of a mixture of Traditional Chinese and multilingual texts in specific proportions, including 20B tokens of Traditional Chinese text.
๐น๐ถ๐ฎ๐ป๐ด๐ต๐๐๐ป/๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ง๐ฎ๐ถ๐๐ฎ๐ป-๐ฏ๐-๐๐ป๐๐๐ฟ๐๐ฐ๐: This is a fine-tuned conversational model based on the foundation model.
This Llama-3.2-Taiwan open-source project is currently a one-person effort (yes, I did everything from text preparation โ so exhausting!). If you're interested, feel free to join the Discord server for discussions.
๐ ฑ๐ ด๐ ฝ๐ ฒ๐ ท๐ ผ๐ ฐ๐๐ บ๐ ธ๐ ฝ๐ ถ
The evaluation was conducted using ikala/tmmluplus, though the README page does not yet reflect the latest results. The performance is close to the previous versions, indicating that further improvements might require adding more specialized knowledge in the datasets.
๐ ฐ ๐ ฒ๐ ฐ๐ ป๐ ป ๐ ต๐ พ๐ ๐๐๐ ฟ๐ ฟ๐ พ๐๐
If anyone is willing to provide compute resources, it would be greatly appreciated to help this project continue and grow. ๐ช
---
๐๏ธ Foundation model: lianghsun/Llama-3.2-Taiwan-3B
๐ค Instruction model: lianghsun/Llama-3.2-Taiwan-3B-Instruct
โก GGUF: lianghsun/Llama-3.2-Taiwan-3B-Instruct-GGUF
๐น๐ถ๐ฎ๐ป๐ด๐ต๐๐๐ป/๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ง๐ฎ๐ถ๐๐ฎ๐ป-๐ฏ๐: This model is built on top of ๐บ๐ฒ๐๐ฎ-๐น๐น๐ฎ๐บ๐ฎ/๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ฏ๐ with continual pretraining. The training dataset consists of a mixture of Traditional Chinese and multilingual texts in specific proportions, including 20B tokens of Traditional Chinese text.
๐น๐ถ๐ฎ๐ป๐ด๐ต๐๐๐ป/๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ฎ-๐ง๐ฎ๐ถ๐๐ฎ๐ป-๐ฏ๐-๐๐ป๐๐๐ฟ๐๐ฐ๐: This is a fine-tuned conversational model based on the foundation model.
This Llama-3.2-Taiwan open-source project is currently a one-person effort (yes, I did everything from text preparation โ so exhausting!). If you're interested, feel free to join the Discord server for discussions.
๐ ฑ๐ ด๐ ฝ๐ ฒ๐ ท๐ ผ๐ ฐ๐๐ บ๐ ธ๐ ฝ๐ ถ
The evaluation was conducted using ikala/tmmluplus, though the README page does not yet reflect the latest results. The performance is close to the previous versions, indicating that further improvements might require adding more specialized knowledge in the datasets.
๐ ฐ ๐ ฒ๐ ฐ๐ ป๐ ป ๐ ต๐ พ๐ ๐๐๐ ฟ๐ ฟ๐ พ๐๐
If anyone is willing to provide compute resources, it would be greatly appreciated to help this project continue and grow. ๐ช
---
๐๏ธ Foundation model: lianghsun/Llama-3.2-Taiwan-3B
๐ค Instruction model: lianghsun/Llama-3.2-Taiwan-3B-Instruct
โก GGUF: lianghsun/Llama-3.2-Taiwan-3B-Instruct-GGUF
- 4 replies