Why Your Machine Learning Model Performs Well but Fails in Production

A machine learning model can achieve 95% accuracy during development and still perform terribly after deployment. This is one of the frustrating realities of machine learning.

You train the model, evaluate it on your test set, see impressive results, and think the hard part is over. Then real users start interacting with it. The predictions become less accurate. Why? Because a model is not trained on “the real world.” It is trained on the data you gave it. And those two things are not always the same. In essence, a machine learning model is an abstraction of reality.

What follows are some reasons why well-performing models may still fail in production.

1. Your training data does not represent production

Suppose you build a model to predict whether a transaction is fraudulent. Your training data might look like this: