I published a LoRA fine-tuning run two weeks ago. The headline was that my first test set had lied to me: on data I generated myself the fine-tune scored 100% and prompting scored 94%, so fine-tuning looked pointless. On a test set rebuilt from real public corpora the fine-tune scored 95% and prompting collapsed to 66%.
A reader called Max Quimby left a comment that reframed the whole thing:
Five points and twenty-eight points. Your fine-tune lost the first moving to the rebuilt set, prompting lost the second, and the distance between those two drops tells you more than either one does. It says the original set was differentially easy for the prompted model rather than uniformly easy for both.
He then named a mechanism. The few-shot examples are part of the prompted method's input, so any kinship between the example pool and the test set inflates one arm and not the other. And he closed with a rule: when two methods move by very different amounts after a test-set swap, suspect the set before you suspect the methods.
I spent an afternoon checking it. He was right that my table was misleading, wrong about which arm was inflated, and the direct test of his mechanism produced a third answer that neither of us predicted.
