DiPOD Stabilizes Diffusion Policy Optimization for Reinforcement Learning

This research tackles a significant challenge in reinforcement learning (RL) post-training, specifically the instability observed in existing diffusion policy-gradient methods. The authors identify a core issue they term the "double-drift phenomenon," where optimizing a variational surrogate causes the ELBO (Evidence Lower Bound) to diverge from the true log-likelihood, leading to misaligned policy gradients and unreliable policy improvement. To counteract this, the paper introduces DiPOD, a novel diffusion policy optimization framework. DiPOD is designed to maintain tight-bound behavior throughout the training process by strategically interleaving self-distillation with policy-improving gradient updates. This approach translates into a practical algorithm that augments each diffusion policy-gradient update with an on-policy ELBO regularizer. Empirical evaluations across both diffusion language model post-training and continuous-control diffusion policies demonstrate DiPOD's effectiveness. The framework substantially stabilizes training, allowing for the achievement of higher rewards compared to previous methods.