New Method Corrects RLHF Bias from Delayed Rewards

Reinforcement Learning from Human Feedback (RLHF) often faces a practical challenge: reward signals, such as those from human reviewers or complex verifiers, are not always immediately available. This delay breaks the synchronous reward assumption that many standard RL algorithms, like PPO, rely upon, leading to biased policy updates. Researchers have developed Retroactive Advantage Correction (RAC) to mitigate this issue. RAC works by queuing delayed reward completions and then reintroducing them into the advantage calculation of subsequent optimizer steps as a clipped residual. This process effectively corrects for the information lag. The method is mathematically proven to be unbiased when all delayed reward mass is eventually reinjected. In a tabular Markov Decision Process (MDP) experiment, RAC reduced policy bias by up to 47.9 times compared to simply waiting for slow rewards, while also being more computationally efficient. RAC can be integrated into existing PPO and GRPO frameworks with minimal code changes.