FACTOR Improves Multi-Turn Agent Learning with Credit Allocation

In multi-turn reinforcement learning for AI agents, effectively assigning credit for actions taken over a long sequence of interactions is a dual challenge. It involves determining how much credit an action deserves for the overall trajectory outcome, and then how to distribute that credit among the individual tokens that constitute the action itself. Existing methods often conflate these two decisions, leading to inefficiencies.Researchers have introduced FACTOR, a novel framework that explicitly separates these credit assignment processes. FACTOR first uses checkpoint-calibrated Temporal Difference (TD) residuals to assign a precise, per-action credit that accurately reflects its contribution to the overall trajectory advantage. This ensures that actions are appropriately rewarded or penalized based on their impact.Secondly, FACTOR employs feedback-conditioned teacher-student likelihood gaps to intelligently allocate each action's credit across its constituent tokens. This fine-grained distribution is further enhanced by per-action normalization, which prevents sign flips and preserves the action-average coefficient. The method also incorporates an action-mean reduction, decoupling an action's surrogate weight from its token length. Experiments across ALFWorld, WebShop, and ScienceWorld consistently show FACTOR outperforming competitive baselines, with the most significant gains observed in the longest-horizon environments. The approach also demonstrates strong transferability across different model backbones and families without hyperparameter retuning.