A migration my schema tool generated passed every check I had. The final schema matched the target exactly — convergence, green. Then I looked at the plan it took to get there: DROP TABLE; CREATE TABLE. On a table with data. The destination was right; the path would have erased a production database.
The test was green because I was checking the wrong thing. I had an oracle for where it ended up and none for how it got there.
That gap is most of what this post is about. I build developer tools — SDKs, a compiler, a declarative schema-management system — the kind of software where one wrong edge case ships to everyone downstream. And I spent a long time trying to get coding agents to test this stuff for me. Claude Code, Codex — with enough prompting and the right skills you can get something that looks like a test suite. What I could never get was coverage that followed the real dev flow without quietly skipping the case that mattered. It's convincing right up until you check whether it checked anything.
What finally worked better was semantic fuzzing: constrained, deterministic random generation, run against properties instead of a reference output, with the agent writing the generators and domain rules rather than playing tester. It catches a lot. And the more I run the harness, the more it improves.
