I keep seeing teams pour compute into bigger base models and fancier RLHF pipelines, and the result is the same: the model gets smarter at gaming the reward, and the actual quality plateaus or drops. We've been scaling the wrong thing.
There's a paper from a couple years back — Scaling Laws for Reward Model Overoptimization — that names the problem precisely. When you optimize a policy against a learned reward model, the proxy reward keeps climbing while the true reward, the thing you actually care about, eventually turns around and falls. The better you get at maximizing the proxy, the worse you do at the real objective. It's not a bug in one training run. It's a structural property of optimizing against an imperfect signal.
The paper frames it in terms of KL divergence. As the policy drifts from the reference, the proxy and the true reward diverge. The relationship is concave — you get early gains, then diminishing returns, then a cliff. And here's the part that should scare anyone doing RLHF in production: the overoptimization isn't just a function of how many steps you run. It's a function of how far you let the policy wander. Every step of optimization against a flawed reward is borrowing against a debt that comes due.
