Manifold Drift Identified as Root Cause of Reward Hacking in Flow Optimization.

A new study has pinpointed "manifold drift" as a fundamental problem in applying preference optimization to continuous-time generative models, specifically in flow matching. This phenomenon occurs when reward-driven updates inadvertently push generated samples away from the original data distribution, leading to a form of reward hacking where models achieve high rewards without producing truly aligned outputs. The theoretical analysis shows that while optimal flow matching recovers the terminal data distribution, preference updates can cause drift if the induced terminal displacement has a non-zero normal component relative to the pretrained manifold. To counteract this issue, the researchers introduce ThermoDPO, a novel temperature-controlled objective. ThermoDPO works by anchoring pairwise preference optimization to preferred samples, effectively preventing the generated data from drifting too far from the original manifold. A weighted variant, ThermoDPO-weighted, further enhances performance by addressing diminished signals at low temperatures. Experimental results on a toy benchmark and SD3.5-M demonstrate that ThermoDPO-weighted significantly improves various metrics, including OCR and overall average performance, showcasing its effectiveness in maintaining data manifold integrity during preference optimization.