There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything.
That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything:
No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers.
The placebo arm is the part almost nobody runs, and it turned out to be the most important one.
Why a placebo
