TL;DR — LoRA-based preference optimization is popular because it's cheap, but the low-rank constraint systematically biases the model toward the simplest direction that separates chosen from rejected outputs — which is usually style, not substance. Distilling from a LoRA-DPO model then bakes that shortcut into the student permanently. The fix isn't more data, it's rethinking where in the stack you spend your rank budget.
Everyone runs preference optimization through LoRA now. Full-rank RLHF or DPO on a large model is expensive, so the industry default is: freeze the base weights, bolt on a low-rank adapter, run DPO against it, ship. It's cheap, it's fast, and it mostly works — the eval scores go up, the chosen responses get preferred over rejected ones, everyone moves on.
Here's the problem nobody's pricing in: the rank constraint doesn't just make the update cheaper. It changes what kind of update is possible. And what's possible under a tight rank budget is almost never "understand why the rejected response was wrong." It's "find the cheapest linear direction that pushes chosen above rejected in this batch." Those are very different things, and low rank systematically picks the second one.
