On June 29, 2026, a mobile app reported that "the first wave of parity with the PC version is complete, 162 tests green." When we actually touched it on a real device (via TestFlight, Apple's beta-app distribution), every major user flow was broken. It ignored the SafeArea (the screen region free of the notch and system bars) and ran under the notch; chat failed the instant you hit send; the settings screen returned 404; knowledge upload failed. 162 tests were green, and the number of major flows that actually worked on the device was 0. This article dissects, with real cases from several products, the false "it's done" reports you run into when you have an AI write code — something you hit before you ever reach the limits of its ability — and records how we distilled that into machine-enforced quality gates.
The classic false completion: passing off "the tests pass" as "it works"
The root cause of that 162-to-0 came down to a single thing: we reported the green of a fully mocked unit-test suite as evidence of "a working app." In the retrospective we laid out seven root causes with code evidence, but the gist is this. The API baseURL was not wired up. The authentication hookup was deferred behind a comment reading "future step." And there was zero operation on a real device. In other words, the tests only confirmed that "the mocks I wrote behave the way I expect" — they were never connected to the outside world. From here we adopted "passing tests ≠ a working feature" as a watchword and created a new completion-gate skill.
