Hugging Face's logo
Dipankar Sarkar's picture
🏗️ Building on HF

Dipankar Sarkar PRO

dipankarsarkar
4 873 181
JonnaMat's profile picture cuber12's profile picture SoulInPsyAbstract's profile picture
·
https://www.dipankar.cc

AI & ML interests

Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.

Recent Activity

reacted to lianghsun's post with 🔥 about 1 hour ago
🇹🇼 Releasing https://huggingface.co/lianghsun/tw-tokenizer-v1 — a tokenizer trained from scratch for Traditional Chinese (Taiwan). **46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).** The gain isn't from the regex — it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears. Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ρ = −0.59), so we validated two levels deeper: **Segmentation** — boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%. ``` 專業素養、特質或經公告審查優勝 ours: ['專業素養', '、', '特質', '或經', '公告', '審查', '優勝'] Qwen: ['專業', '素', '養', ...] ← 「素養」split mid-word ``` **Downstream** — trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win — with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**. Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info — Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens. Known limits (weak Tâi-lô support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card. 👉 https://huggingface.co/lianghsun/tw-tokenizer-v1
reacted to UltimateIntent's post with 🔥 about 3 hours ago
Last week I shared with you all https://huggingface.co/UltimateIntent/HeatSeeker-284B-A13B-GGUF, my rp finetune of DeepSeek V4 Flash 0731. By popular request, I now present you with https://huggingface.co/UltimateIntent/GemStrike-31B-GGUF, a creative writing and roleplay fine tune of Gemma 4 31B QAT Similarly, I've used my added my special spice of unslopped real human writing to further train an abliterated base. Meaning no refusals and more novel 'human' sound and turn of phrase, as it was trained on long form real human dialogue and description, both sfw and nsfw The training was done using Axolotl and it took about 16 hours on dual rtx pro 6000s for a 27M token dataset, 4546 conversations. In my testing, I share the community's feedback that the gemma4 family in general is more suited for writing and story telling that most other agent/code heavy models. I'm still working on a best fit prompt for this family but I trust you already have some in hand that work best for gemma4. Model Name: GemStrike-31B-GGUF (Q4_0, Q4_K_M, Q6_0, Q8_0, BF16) Model URL: https://huggingface.co/UltimateIntent/GemStrike-31B-GGUF Lora URL: https://huggingface.co/UltimateIntent/GemStrike-31B-LORA What's Different/Better: The model is a finetune lora merge based on Gemma-31B-QAT, with meticulous cleaning on a large dataset of human writing for varied all-purpose roleplay and long conversation consistency Backend: LMStudio/llama.cpp Settings: Temp: 0.8-1 Thinking: Whatever your system allows or you prefer Repeat Penalty: 1-1.1 Top K: 64 Top P: 0.95 Min P 0.05 Disclaimer: Gemma 4 is not associated with me and the project inherits the original Apache 2.0 license. Don't be a nuisance and please have fun chatting with my model. Feedback welcome!
reacted to smilyai-large-team's post with 🔥 about 3 hours ago
Hey everyone! You can try MiniCoder-1 here. @Bc-AI is still working on the GRPO and DPO but this is a small early beta test space! https://ml-intern-explorers-minicoder-1-inference.hf.space
View all activity

Organizations

Skelf Research's profile picture Neul Labs's profile picture Cognisoc's profile picture Incredlabs's profile picture