DynaResize Optimizes GPU Use for LLM Post-Training

Large Language Model (LLM) post-training, particularly reinforcement learning-based methods, often separates the "rollout" (generating data) and "training" (updating the model) phases onto distinct GPU resources. However, statically assigning GPUs to these phases leads to inefficiencies, known as pipeline bubbles, especially when rollout latencies are unpredictable. This results in GPUs sitting idle while waiting for the other stage to complete. DynaResize addresses this by introducing a dynamic GPU reallocation system. It intelligently switches GPUs between the rollout and training roles at runtime, ensuring better resource utilization and balancing the execution times of both stages without altering the core RL semantics. The system achieves this through fine-grained resizing operations, minimizing overhead by reusing communicators, staging states efficiently, and employing hysteresis-based resizing. Experiments show DynaResize can boost end-to-end throughput by 66.5% and cut total execution time by a third compared to optimal static configurations.