I fine-tuned Mistral 7B on my laptop to detect personal data in log lines and support messages. On my first test set it scored 100%. Perfect. Every single line classified correctly.
I did not publish that number, because the same test set gave few-shot prompting 94%, and a six-point gap over a prompt you can write in five minutes is not a reason to fine-tune anything. The honest conclusion looked like "this was a waste of an afternoon."
Then I threw my test set away and rebuilt it from real public data. The fine-tune dropped to 95%. Prompting collapsed to 66%.
Same model, same code, same training recipe. A 6 point gap became a 29 point gap, and the conclusion flipped completely. My benchmark had been choosing my answer for me, and it had chosen wrong.
This article is the whole run: what LoRA actually does, how to build a dataset that does not lie to you, the exact commands, and the measured results. Everything is reproducible from the repository on any Apple Silicon Mac.
