What I learned trying to benchmark local LLMs honestly

I had eight models sitting on my laptop and no idea which one I should actually be using.

I could get tokens per second out of llama-bench. That told me llama3.2 was fast. It told me nothing about whether llama3.2 was good — whether the quantization I'd pulled had quietly broken something, whether the 8B model I'd been avoiding because it felt sluggish was worth the wait.

So I built the thing that measures both. It's called homebench, and most of the month I spent on it went not into the code but into deciding what the numbers were allowed to mean. Here's what turned out to be hard.

Tokens per second has a denominator problem

"Output tokens divided by time" sounds like it settles the question. It doesn't, because which time?

What I learned trying to benchmark local LLMs honestly

Related reading

I Benchmarked 3 Local LLMs on My Laptop — Here's What the Numbers Actually Show

Llamafile vs vLLM: Two Ways to Serve a Local Model, and When Each Makes Sense

vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB…

I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B

Running LLMs Locally on Consumer Hardware — Part 1: The Stack and First…

I A/B tested 4 LLMs on the same 500 queries. The results surprised me.