A 7.5B model beat a 24B on my coding benchmark.

16 local model configurations, 56 hidden-test coding tasks, 36 full runs, one 16 GB card

Every number below was recounted from the committed SCORES-*.tsv files, not transcribed from notes. The raw data - 36 rows, one per run, each with its full failure list - is RESULTS-q56.csv.

gemma-4-e4b is 7.5B parameters in a 4.97 GiB file. It scored 42/56.

devstral-small-2-24b is 24B in 11.90 GiB. It scored 40/56. gpt-oss-20b scored 38. Both

gemma-4-12b variants - one of them at Q8_0, more than twice the precision - landed at 42.5 and 41.

A 7.5B model beat a 24B on my coding benchmark.

Related reading

I Ran a 284B-Parameter LLM From 3.2GB of RAM — in Plain C

I Ran Every Gemma 4 Model on My Home Lab. E4B Crushes E2B. Here's the Data.

Gemma 4 31B Runs Fastest on SambaCloud

My Inferentia port matched its reference token-for-token — and still output…

I Tested Claude Opus 4, GPT-4.1, GPT-4o, Sonnet 4, and Gemini 2.5 Pro on 10…

Someone Fine-Tuned OpenBMB's MiniCPM5-1B on Claude Fable 5 Traces to Ship a…