Continuous-Time RL Framework Fine-Tunes Discrete Diffusion Models.

This paper proposes a novel continuous-time reinforcement learning (RL) framework specifically designed for fine-tuning discrete diffusion models. It conceptualizes the state dynamics as a controlled continuous-time Markov chain (CTMC), allowing for policy optimization problems and the derivation of continuous-time variants of established methods like PPO and GRPO. A key advantage of this framework is its ability to perform reward-driven optimization even when reward signals are not differentiable, which is a common challenge in many real-world applications. Unlike existing GRPO-based approaches that typically rely only on terminal rewards, this new formulation permits the integration of intermediate reward or advantage signals throughout the entire denoising trajectory, offering more granular control and feedback. When applied to masked diffusion models (MDMs), the framework provides a unified perspective on exploration and policy optimization, supporting a rich class of policy parameterizations over the vocabulary simplex with analytically tractable probability ratios. For masked diffusion large language models (dLLMs), the researchers also introduce trajectory subsampling techniques to efficiently estimate computationally intensive trajectory likelihoods, significantly reducing the cost of computing per-position probability ratios. The effectiveness of these methods is demonstrated on both low-dimensional optimization problems and RL post-training of dLLMs for mathematical reasoning and coding tasks.