Originally published at ai.bedvibe.studio.
I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.
Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS.
So I wrote the QA framework I wished I had, packaged it as ttsproof, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.
The two failures WER cannot see
