Separate Critics Boost Parallel Reinforcement Learning Performance

When training a single policy in parallel across multiple environments of the same task, such as procedurally generated levels or varied dynamics, it is common practice to use a single 'critic' network to estimate value functions. However, different environments can assign distinct expected returns to the same input, leading to a 'value mismatch' for a shared critic that lacks environment-specific information. This mismatch forces the critic to reconcile conflicting value targets, which can systematically distort the sampled advantages within individual environments. This distortion can reinforce unhelpful actions while weakening or even reversing the updates for beneficial ones. While theoretical analysis shows that, on average, shared and separate critics converge to the same optimal policy, their actual learning paths can diverge sharply, impacting practical performance. The research proposes a minimal yet effective intervention: simply providing the critic with a logged environment index. This allows the critic to condition its value estimates on the specific environment, effectively separating the value targets. Experiments on various tasks, including CartPole, MuJoCo, BipedalWalker, and 16 Procgen games, confirm that this intervention leads to more stable learning, higher returns, and a significant improvement in aggregate normalized return on unseen levels, demonstrating the critical importance of addressing value mismatch in parallel RL.