The most expensive bug I shipped last month was not in a model, and it was not in a prompt. It was in the thing doing the grading. I found it three times, in three different businesses, in the same month. Each time I had been blaming the wrong thing.
The best model I could buy scored 0.241
I run a small training platform. Five open-source models from the Qwen and DeepSeek families, one hard 54-task exam, and a gate that only promotes a model when the numbers say the win is real. For weeks none of them could get through the gate. The easy explanation was the comfortable one: small models, hard exam, keep training.
So I plugged a frontier model, gpt-5.5, into the exact same harness, just as a reference. Same tasks. Same fixtures. Same scoring.
It scored 0.241.
