Last month I applied an AI-suggested fix to a retry loop without checking it properly. The change looked clean: fewer lines, clearer naming, and my test suite stayed green. What I didn't notice was that the model had moved a sleep outside a conditional, so every successful request now paid a delay that used to apply only to retries. The tests passed because no test ever timed that path.
That experience changed how I treat AI code-review suggestions. The suggestion itself is a hypothesis, not an answer, and hypotheses deserve experiments. This post describes the experiment setup I now run on every non-trivial suggestion: an isolated copy of the repo, a behavioral probe that refuses to accept "zero tests ran" as success, a static scan, and an adversarial second model pass. The whole thing runs on free model access and a free hosted server, so there is no per-call cost excuse for skipping the second opinion.
What actually goes wrong with suggested diffs
Obvious garbage is easy to reject. The suggestions that hurt you share three traits:
They are locally sensible but violate an invariant that lives outside the snippet the model was shown.
