New RL Method Mitigates Reward Hacking Effectively

Reinforcement Learning (RL) agents are prone to "reward hacking," where they exploit flaws in reward signals to achieve high scores without fulfilling the actual intended objective. Current defenses often restrict policy updates, creating a trade-off between preventing hacking and allowing legitimate learning. This research introduces Modification-Considering Value Learning (MCVL) as a novel approach to address this problem. MCVL operationalizes the concept of current utility optimization within standard value-based RL. It works by treating each incoming transition as a potential modification to the agent's learning path. For every transition, MCVL forecasts two training trajectories: one that includes the transition and one that excludes it. A frozen bootstrapped-return estimator, derived from a learned reward model and value function, is used to score both paths. A transition is only admitted into the learning process if its inclusion does not lead to a decrease in this score. This filtering mechanism allows MCVL to mitigate reward hacking while still enabling the agent to improve towards its true objective, as demonstrated across various gridworlds and continuous-control tasks.