Robust Policy Learning Reduces Variance in RL Evaluation

In reinforcement learning (RL), evaluating the performance of a policy online often suffers from high variance, particularly when the behavior policy used for data collection is not optimized for evaluation. While existing methods attempt to learn data-collecting policies to reduce this variance, they typically overlook uncertainties in the environment's transition functions. This oversight can lead to policies trained in simulation performing poorly in real-world deployments. This paper proposes a novel double-loop gradient-based algorithm designed to learn behavior policies that are both efficient in data collection and robust to these transition uncertainties. The theoretical contributions include new gradient expressions for transition variance and global convergence guarantees. Numerical experiments confirm that the proposed method is less sensitive to environmental perturbations compared to existing approaches, highlighting its practical utility for real-world RL applications.