TideRL Boosts Agentic RL Training Efficiency by 5.6x

Reinforcement learning (RL) for large language models, particularly in multi-turn agentic workloads, faces significant efficiency challenges. These workloads involve tasks that frequently pause for external environments, resume with expanding contexts, and complete at highly variable times. In such scenarios, the true measure of efficiency, "training goodput" (effective training throughput), is often hampered by GPU waiting times and redundant prefill recomputation. Researchers introduce TideRL, a novel readiness-aware elastic RL system designed to optimize these complex workloads. TideRL incorporates three key components: Continuous Task Batching (CTB) to preserve useful rollout state, Resource-Aware Ref-Actor Pipelining (RA2P) to intelligently select between streaming and aggregation from ready tasks, and Elastic Resource Scaling (ERS) to dynamically reallocate resources between rollout and training based on readiness signals. Evaluations across both text-only and multi-modal agentic workloads demonstrate that TideRL significantly improves RL training goodput by up to 5.6 times compared to synchronous baselines, and over 33% compared to asynchronous baselines, all while maintaining similar task performance. The system also achieves a 1.58x improvement in KV cache hit rate, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.