Twice now I have gone looking for a retrain and found the problem somewhere else.
The first time, a third of my output had collapsed into one template. Before spending anything I counted the pattern in my training data. 5 occurrences in 1,610 examples. Then I counted it in the output. 36%. The cause was a single example I had hardcoded into my own prompt months earlier, and the model was copying it.
The second time, quality had gone flat across the board. The system prompt had grown to 224,833 characters, roughly 56,000 tokens on every call, and ninety percent of it was accumulated rules about how the output should sound. The model was doing exactly what I had asked it to do, which was avoid things.
Neither needed new weights.
That is the pattern I would put in front of anyone about to fine-tune. It is the most expensive way to discover you had a prompt bug. About $30 and a five-hour job per run on a 70B model, and the compute is the small part. The real cost is that you now have two variables moving and no clean way to tell which one broke.
