HARTS Accelerates Agentic RL for Hybrid-Attention Models

Agentic reinforcement learning (RL) often generates complex, irregular rollout trees where different trajectories share common historical prefixes. Traditional training methods recompute these shared prefixes repeatedly, leading to inefficiencies, especially with hybrid-attention models that combine different attention mechanisms. Existing systems typically focus on full-attention models and lack the necessary dense, differentiable execution for hybrid-attention with activation recomputation. HARTS (Hybrid-Attention RL over Tree Structures) addresses these challenges by jointly planning microbatches, data-parallel replica assignments, and microbatch-slot schedules. It employs non-replay compact-token work after compressing shared prefixes. For chunkwise linear attention, HARTS uses a linear-time algorithm to coordinate state recovery and replay, minimizing sequential linear-attention calls. The system preserves chunkwise state partitioning, avoiding redundant computations and performing only bounded state replay for numerical alignment. HARTS batches all branches into a single packed call per round, propagates gradients through differentiable state handoffs, supports activation recomputation, and restores per-token log-probabilities. On SWE-bench tasks, HARTS demonstrated 4.81-4.87x speedup in forward/backward/gradient passes with activation recomputation, making it a significant advancement for efficient agentic RL.