June 1, 2025
🎉🎉🎉 Congratulations to the new top model in the world: mlx-community/gemma-3-12b-it-4bit 🎉🎉🎉
Gemma 3 12B currently holds the top spot on the Medium-Sized Model Plus Mildly Buggy Custom Logit Sampling Code Vibe Poetry Leaderboard™️
This model captures the very human experience of crafting and re-crafting computer code, in a poem that contains only two-syllable words.
Logic woven into glowing displays /
Commands crafted amidst hazy delays /
Functions firing throughout endless arrays /
Debug complete after countless relays /
This is a proprietary benchmark. Given the obvious opportunity in the space (cf the recent LMArena funding round) I will not be publishing inference code or data sets.
(In all seriousness, the Gemma 3 models are awesome.)
Related: "pixelated" is not in cmudict[1]
oops, yes, but no, that's not the bug. The bug is that pixelated is two tokens. A Byronic flourish by Gemma, so perhaps allowed. Possibly even to be admired.
