A generated patch can look correct and still fail in three ways that a prompt screenshot cannot show: it changes the contract, it imports a module the rest of the codebase does not allow, or it only works on the single example in the prompt. If you copy that patch into a real repository, you become the integration test, usually in a pull request late at night.
This article walks through a short validation loop that treats model output as untrusted input. The loop runs three checks before a candidate file can touch your working tree: contract, execution, and regression. The script is plain Python with no model-specific SDK, so the same gate works for output from any endpoint or manual file.
The three failure classes
Contract drift is the most common silent break. A model may rename a function, change its arguments, or return a different type while leaving the prompt's happy path intact. A static AST check can catch missing symbols before the code is ever imported.
Forbidden reach is the second issue. A coding model may decide to read the filesystem, shell out to git, or call a network when the task was supposed to be pure. If you allow generated code to run, its imports matter as much as its output.
