New Method Debias Reward Models, Improving AI Performance

A new research paper introduces "patterning," a novel technique designed to mitigate biases in reward models, which are crucial components in training large language models (LLMs) based on human preferences. Reward models often exhibit biases related to output length, formatting, or other stylistic elements, which can lead to suboptimal or incorrect model behavior. Patterning addresses this by reweighting each preference pair in the training data according to its "susceptibility," which measures its effect on the posterior expectation values of benchmark losses. Applying this method to a Gemma 2 9B Instruct reward model, trained on the Skywork-Reward-Preference v0.2 dataset, resulted in a significant improvement of +14.2 percentage points on the RM-Bench Hard split. This particular split is challenging because stylistic cues in the data often contradict the correct answer. The study also demonstrated that the computed weights are interpretable and can transfer effectively to other models, such as Gemma 2B and 27B, and partially to Llama 3.1 8B, without requiring recomputation. This marks the first application of patterning, a concept rooted in singular learning theory, to larger models and real-world tasks.