New Trust-Region Diffusion Policies Enhance Massively Parallel On-Policy RL

This new research presents Trust-region Diffusion Policies, or TruDi, a method designed to improve reinforcement learning in environments that use massively parallel simulations and on-policy training. Traditional approaches in this setting often rely on simpler policy models, while more expressive diffusion models have typically been limited to offline or off-policy training. TruDi addresses the inherent challenges of rapidly changing data distributions in on-policy training by incorporating a trust-region optimization rule. This rule enforces a KL-divergence constraint across the entire diffusion trajectory, ensuring stable learning even with complex policy structures. Empirical evaluations across 73 tasks in four massively parallel RL benchmarks demonstrate that TruDi consistently matches or surpasses existing strong baselines. It shows significant improvements on more complex humanoid control tasks, establishing a new benchmark for this domain.