Low-Rank Adapters Turn Preference Tuning Into Shortcut Tuning

Cheap LoRA-based preference tuning doesn't teach models your preferences. It teaches them the shortest path to look like they satisfy them.

Raccontata dadev.to