7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.
All I can say is that the best is yet to come. And will be agentic.
Join the community of Machine Learners and AI enthusiasts.
Sign UpThe 90m claim holds on the one board I have audited, and the gap is wider than the raw scores say.
On jiteshkprasad's 27-model MMLU run (14,007 items, letter scoring), Falcon-H1-Tiny-90M-Base gets 32.49. SmolLM2-360M gets 25.59, SmolLM2-135M 24.60.
But small models pick letters unevenly, so chance is not a flat 25.
Score each one against what its own letter mix would get by luck, and Falcon-90M is +7.9 points. SmolLM2-360M is +1.0, SmolLM2-135M +0.7.
So it is not a 90m model narrowly beating a 360m one. One is reading the question. The other mostly is not.
Is Falcon-H1-Tiny the 90m you meant, or is there another one I should run through the same check?
check klm-1.5
Sooner than you might think