When Should You Retrain Your ML Model? A New Study Says Most Teams Are Solving the Wrong Problem

Every production ML model starts degrading the moment it ships. Customer behavior shifts, fraud patterns evolve, market conditions change — the technical term is concept drift, and it's the reason a model that scored great in testing quietly gets worse in the real world, often without anyone noticing until revenue or accuracy numbers show it.

The obvious question teams ask is: when should we retrain? A recent study, "When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints" by Sawan Dasari, ran nearly 4,000 experiments to actually answer that question — and the answer isn't the one most teams are optimizing for.

What the Study Actually Tested

The paper compares four approaches to keeping a deployed model current:

Doing nothing — the model is trained once and never touched again (the baseline).