New RL Framework Mitigates Overestimation Bias in Off-Policy Learning

Deep off-policy reinforcement learning (RL) algorithms often suffer from overestimation bias, where the estimated value of actions is consistently higher than their true value. This issue stems from noisy temporal-difference (TD) targets and the recursive propagation of errors during policy updates, leading to unstable training and degraded performance in actor-critic methods. Existing solutions often overcorrect by focusing on high-uncertainty transitions, which can inadvertently amplify bias. The new Collaborative Weighting Actor-Critic (CWAC) framework tackles this by explicitly accounting for predictive uncertainty in value estimation. It employs a distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD-errors and uncertainty. Additionally, CWAC incorporates a stochastic pessimistic value estimation scheme, further mitigating error propagation. This framework is designed to be easily integrated into popular off-policy algorithms like SAC, TD3, and DDPG, demonstrating significant performance enhancements across various simulated tasks.