alkinun

AtAndDev

29 158 867

alkinun

AI & ML interests

decentralize

Recent Activity

liked a dataset about 5 hours ago

IFM/Pretrain-Behaviors

reacted to juiceb0xc0de's post with 🔥 about 5 hours ago

Dude, Where's My Update? I'll tell you where! ~97.6% of my BF16 parameter coordinates didn't move at all, and the ones that did overshot by ~1.33x. It's nice to do research that doesn't end in disproving yourself once again and moving on to the next subject once in awhile. Back to the topic, if you've ever wondered why most of your weights are basically ghosting you nearly every step when you store your weights at bf16, Dude, I Measured It. https://huggingface.co/blog/juiceb0xc0de/intended-and-realized-updates-in-bf16-fine-tuning#dude-wheres-my-update

liked a dataset about 6 hours ago

marin-community/token-counts

View all activity

Organizations

reacted to juiceb0xc0de's post with 🔥 about 5 hours ago

Post

383

Dude, Where's My Update? I'll tell you where! ~97.6% of my BF16 parameter coordinates didn't move at all, and the ones that did overshot by ~1.33x.

It's nice to do research that doesn't end in disproving yourself once again and moving on to the next subject once in awhile.

Back to the topic, if you've ever wondered why most of your weights are basically ghosting you nearly every step when you store your weights at bf16, Dude, I Measured It.

https://huggingface.co/blog/juiceb0xc0de/intended-and-realized-updates-in-bf16-fine-tuning#dude-wheres-my-update

reacted to appvoid's post with 🔥 about 17 hours ago

Post

3002

We got gpt6 before gta6

14 replies

reacted to Bc-AI's post with 🔥 6 days ago

Post

2524

Building a 6.58B sparse MoE model from scratch on a single GPU.
Hey everyone! Today is day 1 of training Smilyai-Lab's new model I call G1-MINI. It's basically the smaller version of our planned model G1 which will be 20B and activate about 2B per token. MINI activates about 1.16B params per token and is currently training right now. If no errors spring up now, I'd say i can launch sometime around September 15th~ish. Thanks to our beta testers:
@guardamarcos
@ProCreations
@juiceb0xc0de
@Timmy6767
@Sbui503
@atom77777
@Fishtiks
@smartdigitalnetworks
@EmetTheGolum
@smilyai-large-team
@MUK-IS-GOAT
@Bc-AI
@Banaxi-Tech
@vovaRL
@Datdanboi25

9 replies

reacted to KlondikeDev's post with 🔥 6 days ago

Post

3439

A Preview of Boris-2!

Hello! Tomorrow, OpenCerebral will be releasing Boris-1.7-D60M-n30M — an experimental architecture. It will be testing a new data mixture, a new tokenizer, and testing Qwen4-like n-gram embeddings.

Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.

Then, Boris-2 will begin training!

Edit: the model is OUT NOW! opencerebral/Boris-1.7-D60M-n30M

70 replies

replied to KlondikeDev's post 6 days ago

wow... my eyes..

posted an update 6 days ago

Post

2784

SPECK 2 IS ALREADY OUT: specklabs/Speck2-140M

Pretrained on 4x more tokens than the previous releases (20b vs 5b).
Instruct tuned versions are coming soon.
Very interesting models are coming soon too (hint: super long context).

Thanks for everyone supporting!

2 replies

reacted to KlondikeDev's post with 🔥 6 days ago

Post

2384

Boris-1.7-D60M-n30M out NOW!

opencerebral/Boris-1.7-D60M-n30M

Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.

Then, Boris-2 will begin training!

reacted to Hoglet-33's post with 🔥 7 days ago

Post

3285

Pebble 10M and Pebble 10M Chat are now released!

Both models use our Mamba/Transformer 3:1 hybrid architecture and were pretrained on 25 billion tokens.

Pebble 10M Chat was additionally fine-tuned on 250 million tokens of Smol-SmolTalk to improve its conversational capabilities.

You can find them here:

- basically-ai/Pebble-10M
- basically-ai/Pebble-10M-Chat

We hope you enjoy using them. The rest of the Pebble family will be released soon.

Follow for more:
@Hoglet-33

basically-ai

32 replies

reacted to Banaxi-Tech's post with 👀 8 days ago

Post

3504

We're delaying BananaMind 2.1!
When BananaMind 2.1 Lite was almost done, we benchmarked it and the results we're worse than BananaMind 2 Mini.

We're going to spend alot more time in research on tiny models and then scaling up our techniques to the actual BananaMind 2.1 models!

We're also announcing these new models:
BananaMind 2.1 Coder: A 149M instruction tuned coder model trained on 75B tokens + 10B tokens of stack-v3-train.
BananaMind 2.1 Pico: A 1M parameter model trained on 22B tokens of data.
We also may release BananaMind 2.1 Large with around 100M parameters depending on how much compute we have.

Please give us a follow!

BananaMind
@Banaxi-Tech

---

@vovaRL
@DedeProGames

7 replies

reacted to sergiopaniego's post with 🔥 8 days ago

Post

395

catching up on some bookmarked reads from the summer, reading Antidoom from @liquidai

small reasoning models get stuck more easily when the task involves a long thinking trace and a hard problem. It starts repeating the same word over and over again ("Wait", "Alternatively"…), each repetition makes the next one likelier, and the generation is spent before it reaches an answer

they measured it, 10.2% of completions for an early LFM2.5-2.6B checkpoint and 22.9% for Qwen3.5-4B at greedy. After training those drop to 1.4% and 1.0%

the fix is FTPO (final token preference optimization). What I like is how narrow it is, it only touches the single token where the loop starts

three ways it differs from DPO:
> trains one token position, mid-generation, instead of whole sequences
> spreads probability across ~20 plausible alternatives instead of swapping one overtrained token for another
> keeps the regularizer in logit space, no softmax, so the rest of the vocabulary stays put

the third one is what makes it usable. If you want to edit one position without disturbing the model, you can't have a loss that reshuffles the other 150k logits on the way

and their explanation abt the result: the training teaches the model nothing new about math or code, it clears the failure mode that was blocking answers the model could already produce

full blog > https://www.liquid.ai/blog/antidoom

FTPO itself comes from Antislop, where it was built to strip overused phrasing. LiquidAI retargeted it to doom loops

and under the hood it's a subclass of TRL's DPOTrainer with compute_loss overridden, around 90 lines of loss and no new trainer

we documented that pattern in TRL's docs
https://huggingface.co/docs/trl/main/en/customization#change-the-training-objective

1 reply

reacted to appvoid's post with 🔥 8 days ago

Post

3290

Love how the small lm community is getting identity over time:

- Channel-Mixing
- XSA
- Three-tower
- Digit aware
- Loops

No one is doing the same! That's so cool.

36 replies

posted an update 9 days ago

Post

154

SPECK1.5 IS COMING SOON!
Same 5B token budget but much better corpus quality.

Also getting a ton of downloads, thanks for everyone downloading and liking <3

specklabs

reacted to OppaAI's post with 🔥 9 days ago

Post

2683

Congrat to HuggingFace Team

https://techcrunch.com/2026/08/26/nvidia-closes-in-on-hugging-face-acquisition/

3 replies

reacted to Banaxi-Tech's post with 🔥 10 days ago

Post

2145

We have updated the BananaMind Base Bench leaderboard!
We now have these benchmark cards, they make it way easier to see which models are actually good!
We've also added the model advisor. It asks you what you want to use the model for and the parameter range and gives you the best model for your task!

Try it out at BananaMind/BananaMindBench-Leaderboard

And please give us a follow to BananaMind!

BananaMind
@Banaxi-Tech

1 reply

reacted to Hoglet-33's post with ❤️👍🔥 10 days ago

Post

3047

We are announcing the first generation of the Pebble model family!

These are the models we are releasing:

- Pebble 10M
- Pebble 25M
- Pebble 50M

Each model will use a Mamba-Transformer 3:1 hybrid architecture and will be pretrained on 25 billion tokens before IFT and SFT.

Depending on development time and resources, we may also release:

- Pebble 5M
- Pebble 75M
- Pebble 1M (possibly)

We hope you're excited and enjoy the models!

Follow for more:
@Hoglet-33

basically-ai

7 replies

reacted to CodeSoft's post with 🔥❤️ 10 days ago

Post

2407

Wow, SLM Arena is getting a lot of traffic! Thank you guys for showing your interest!

To handle the growing demand, I’m moving SLM Arena from a CPU Space to a ZeroGPU Space. Hopefully, this will let me add more models to SLM Arena while keeping it running fast.

I've also added a separate arena + leaderboard for base models!

If there are any models or features you’d like to see, let me know in a reply to this post or in a Community post on the Space!

25 replies

reacted to appvoid's post with 🔥 10 days ago

Post

2635

Nobody knows what is doing, when you train a model, you are experimenting to advance the frontier, so keep failing 🫵

17 replies

alkinun

AI & ML interests

Recent Activity

Organizations

AtAndDev's activity