An AI-generated test that passes proves nothing until it can go red. Mutation testing exposes the phantom tests, with a Go example to prove it.
You ask the AI for tests. It hands you twelve, all green. CI passes. You merge. Three days later a bug ships, on a function those tests were supposed to cover. You reopen the test file and it clicks: it ran, it passed, and it tested nothing.
A green test isn't a proof. It's a hypothesis. And an AI, left to its own devices, is very good at writing hypotheses that can never be disproved.
The phantom test
Take a dead-simple function, a discount above 100 euros:
func Discount(total int) int {
An AI agent can make CI go green without making the code right. Here's how that happens, and how far you can get catching it.
AI is great at writing tests fast, and good at writing tests that look real but verify the wrong...
Six months ago I kept reading the same story. Developer uses Cursor or Claude Code to ship a feature....
To audit AI-generated tests, score how many mirror the code instead of checking it. Green CI proves...
AI can generate unit tests in seconds. But how do you know whether those tests are actually...
You Think Testing AI Means Testing the Process? Wrong A Testing Problem My...
