New Method Interprets MoE Reward Models by Response Contribution

Reward models are fundamental to training AI systems using human preferences, but understanding why they make certain predictions remains a significant challenge. While sparse Mixture-of-Experts (MoE) reward models aim to improve interpretability by routing prompts to specialized experts, current methods primarily focus on routing weights, which only indicate which prompts an expert receives, not how it judges responses. This provides an incomplete picture of expert behavior. To address this, a new interpretation method called Contribution-Contrast (CoCo) has been developed. CoCo offers a response-level interpretation by identifying chosen-rejected response pairs that exhibit the largest differences in expert contributions. This approach jointly captures both routing and preference behavior, leading to a more faithful and specialized understanding of each expert's role. Across various evaluations, CoCo consistently produced more coherent and accurate interpretations compared to existing router-based, score-based, and sparse autoencoder-based alternatives, all while maintaining strong reward modeling accuracy.