Efficient RL for Long-Horizon Tool-Use Agents

A new modular training system, SINKFLEX-RL, has been developed to enhance the efficiency of reinforcement learning (RL) for agents performing long-horizon tasks that involve tool use. These agentic tasks often require complex reasoning over user goals, domain policies, and tool calls, leading to long contextual sequences and significant memory demands for traditional RL methods. SINKFLEX-RL addresses these challenges by integrating several key components. The system features a Gymnasium-compatible environment wrapper for broad applicability, a VERL-style dataflow for efficient rollouts, and group-relative policy optimization that eliminates the need for a separate value model. Crucially, it incorporates a sink-aware FlexAttention path, specifically designed to preserve model-specific sink scaling under various masking schemes while significantly reducing VRAM consumption. Preliminary results show improved validation reward in a retail environment and a nearly 20% reduction in peak VRAM usage, enabling the training of larger models and longer contexts that previously caused out-of-memory errors.