Free Model Endpoints Make Good Fuzz Generators and Bad Oracles
The failure starts with a tempting shortcut: ask a low-cost model to "write tests for my CLI parser." The output looks confident. It returns a list of flags, expected return codes, and assertions. Some of those assertions are wrong. Worse, the model often encodes the same misunderstanding as the code it is supposed to test, so the test suite promises coverage while silently passing broken behavior.
The fix is not to buy a more expensive model. It is to take away the model's authority over correctness. Let the model generate candidate inputs. Let a deterministic reference implementation in your own code decide whether the behavior is right.
Build the generator/validator split
Use the model as a fuzzer, not as an oracle. The model emits argument arrays for a small C++ program. The program is your real system under test. A Python reference parser computes the expected outcome. Anything that disagrees is a finding.
