AI-generated code rarely fails loudly. The failure that scares me is the quiet one: a build report that says a security control is "Applied" when the code behind it was never written.
I hit this on a real build. "PII encryption at rest" was marked done in three places: a schema comment, an architecture invariant, and a compliance sheet. Three documents, all agreeing with each other.
Then I grepped for createCipheriv. Zero hits. The encryption key referenced in the docs was in no environment schema. The address columns were plaintext.
Every automated signal was green. The invariant that should have verified the control was marked "manual," so the linter skipped it. The tests passed because they only test what exists; nothing asserts the absence of a thing that was never written. The compliance sheet passed because it echoes the spec instead of auditing the code. Greenness was self-referential.
Once you look for this pattern you find weirder versions. Another build's invariant linter reported "0 violations" while inspecting zero files: the model had re-derived a glob-to-regex translation and applied the wildcard replacements in the wrong order, so the pattern matched nothing. A clean pass over an empty set. Across four builds I watched, the model's first attempt at that same translation was wrong in three, each differently.
