When you are growing an audio dataset, you will inevitably run into the "Who's voice is this again?" problem. The anchors (target speakers for voice conversion) in this app were no exception. As I collected more material, I found anchors that had lost their labels—they were just numbered, like spk7 through spk18.
Files had missing names, some were renamed halfway through, and the mapping between the raw source directories and the extracted anchors became disconnected. For various reasons, the result is the same: you end up with "voice data that exists, but whose identity is unknown."
If left alone, this kind of material tends to become "dead data"—ignored because the source is unclear. But the clue actually lies within the sound itself. Since we know the vocal characteristics, we can just match them by voice quality. This article is a record of how I re-labeled unknown anchors by relying solely on speaker embedding similarity and rebuilt the anchor_sources.json file.
The Idea: Matching by Voice, Not by Label
Here is the breakdown of what I wanted to achieve:
