I run a fine-tuned Llama 3.3 70B on Amazon Bedrock. It generates short first-person narrative posts: a setup, a few lines of story, and a closing line that lands the point.
Last week I noticed the closing lines had collapsed into a single template. Not similar in spirit, but literally the same grammatical shape, over and over:
and that's how [someone] [learns/teaches] [something].
Roughly a third of everything the model produced ended that way. The obvious diagnosis was overfitting: the training data must be saturated with that pattern, so the model learned it as the way to end a story. The obvious fix was equally clear. Scrub the training set, retrain, redeploy.
In my setup that runs to about $30 of training compute and a five-hour training job, plus the evaluation pass afterwards before I'd know whether it had worked. Not ruinous, but not free either, and this would have been the third retrain cycle.
