DigenRL Accelerates Disaggregated RL for Visual Generative LLMs

Reinforcement learning (RL) has become a key paradigm for post-training large language models (LLMs), with systems like veRL emerging for autoregressive LLMs. Concurrently, diffusion-oriented RL algorithms have extended RL's scope to visual and flow-based generation. However, efficient RL systems for diffusion generative LLMs have remained underexplored, with existing implementations often relying on colocated execution, which limits resource flexibility and independent scaling. To address these limitations, DigenRL, a disaggregated RL framework, has been introduced. DigenRL is specifically designed for diffusion-based generative LLMs, offering flexible resource allocation, accommodating heterogeneous GPUs, and facilitating efficient task scheduling. It incorporates several innovations to minimize execution bubbles in its disaggregated architecture: a generation-axis pipeline (GAP) and time-step parallelism (TSP) for finer-grained pipelining, an elastic trainer-assisted generation (TAG) approach where trainer GPUs dynamically aid rollout generations, and a tightly constrained asynchronous strategy. Extensive experiments on various hardware testbeds with multiple generative models demonstrated that DigenRL achieves 1.56-2.10x throughput improvements compared to state-of-the-art diffusion RL systems.