Engrava's test suite had 3,845 tests and 94.22% line coverage. There was more test code in the repository than production code. By every number we had, the thing was well tested.
Then we ran a mutation audit over it and found that the suite did not protect a broken concurrency guard, several deletion paths, five configuration sections, two of its own safety tests, or a field that would silently discard a legitimate value, in the version people were running. That is nine findings in three groups, and the groups are not alike. The accounting is below.
This post is about how that happens, because the mechanism is more interesting than the individual bugs, and because we would have told you the suite was solid the day before.
Coverage answers a question nobody asked
Line coverage tells you a line executed while the tests ran. It does not tell you that anything would have noticed if the line were wrong.
