Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
appvoid 
posted an update about 19 hours ago
Post
721
You know, I can't believe we are actually doing it, literally advancing the frontier at the edge! We've come a long way since the first models on open slm leaderboard, and, as many things that change, performance is one of the main things that has exploded.

7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.

All I can say is that the best is yet to come. And will be agentic.

The 90m claim holds on the one board I have audited, and the gap is wider than the raw scores say.

On jiteshkprasad's 27-model MMLU run (14,007 items, letter scoring), Falcon-H1-Tiny-90M-Base gets 32.49. SmolLM2-360M gets 25.59, SmolLM2-135M 24.60.

But small models pick letters unevenly, so chance is not a flat 25.
Score each one against what its own letter mix would get by luck, and Falcon-90M is +7.9 points. SmolLM2-360M is +1.0, SmolLM2-135M +0.7.

So it is not a 90m model narrowly beating a 360m one. One is reading the question. The other mostly is not.

Is Falcon-H1-Tiny the 90m you meant, or is there another one I should run through the same check?

·

check klm-1.5

180M Agentic models on par with Qwen2.5 1.5B

·

Sooner than you might think