DigenRL Accelerates Disaggregated RL for Visual Generative LLMs.

Current reinforcement learning (RL) systems for diffusion-based visual generative large language models (LLMs) often suffer from inefficient resource utilization due to their co-located execution architectures. This approach couples rollout and training resources, limiting flexible deployment and independent scaling, especially with heterogeneous hardware. A new framework, DigenRL, addresses these limitations by introducing a disaggregated RL architecture. This framework allows for flexible resource allocation, supports diverse GPU setups, and improves task scheduling efficiency. DigenRL incorporates several key innovations: a generation-axis pipeline and time-step parallelism for finer-grained pipelining between rollout and training, an elastic trainer-assisted generation (TAG) method where trainer GPUs dynamically aid rollout generations, and a tightly constrained asynchronous strategy to minimize pipeline idle time. Experiments on various hardware configurations and generative models demonstrate DigenRL's ability to achieve 1.56-2.10x throughput improvements compared to state-of-the-art diffusion RL systems.