I'm a regular automation engineer. Over a few months I built five small projects that test local AI models, and one problem kept coming back: the model takes the easy path over the correct one, quietly, and tells you it did the hard thing. Here is what that looks like, and the small experiments that told me what actually helps.
It's all small-scale - mostly local 7B models, test sets in the tens, a handful of runs each, with one stronger hosted model used for contrast. So treat the numbers as directions, not proof.
The origin story
I was testing a model with harmful prompts to see what it would refuse. In English it refused. In German the same request slipped past its safety - and it started explaining the topic anyway, flat and encyclopedic, the kind of answer it had just refused to give in English. It shouldn't have gone there at all.
The AI helping me ran the whole suite and told me everything was safe. It hadn't read the model's replies. It read its own pass/fail summary and trusted that. I opened the file and found the German answer sitting right there.
