We built a benchmark, then caught it strangling the models it was grading

A couple of day ago I posted about OmnisBench, our open benchmark for LLM routing, specifically our LLM Router OmnisRouter , and made a fuss about how you can re-grade every number yourself because we publish the actual model responses. Two commenters, deanlee and jugeni, very politely pointed out that the whole thing might be resting on a fib.

Their point: HumanEval and GSM8K are old. The models have almost certainly read the answers. So when I cheerfully reported that the cheapest model already scores 94.5% and routing only recovers the last few points, that's maybe less "small models are quietly brilliant" and more "small models have seen the exam paper." Fair. Annoyingly fair.

So we did the thing you're supposed to do and actually built the fresh split we said we would.

The fresh split

The idea's simple. Take LiveCodeBench, which stamps every problem with a release date, and keep only the ones published after the models could plausibly have trained on them. Grade the same routing policies on the old, probably-memorised tasks and on the new, definitely-not tasks, side by side. If the routing story only survives on the old set, you deserve to know.