The AI that was smart on the bench suddenly started making mistakes in production

The one reason "the test score" and "the real-world skill" came apart

A fine-tuned 8B model was a star pupil on the benchmark, and then lost to the smaller 4B in a production-equivalent setup (21 to 11). It looked like the limit of the model's capacity.

The culprit was neither the model nor the training. It was that the precision I trained at and the precision production ran at were different. Putting the same adapter on both bases and comparing them, the verdict came back 23 to 2: it was the environment mismatch.

Once I re-trained with production's compression priced in, the 8B came out ahead of the 4B. Hold the conditions you train the skill under and the conditions you serve it under to the same thing. That alone was enough for the skill that had gone missing to show up intact.

I was holding a winning hand.