New Method Improves Agentic RL with Multi-Timescale Credit Assignment

Agentic reinforcement learning (RL) often struggles in real-world scenarios due to the common problem of delayed and sparse rewards, making it difficult for agents to understand which actions contributed to a successful outcome. A promising solution is credit assignment, which aims to break down overall rewards into more specific feedback for individual decisions. However, existing credit assignment methods frequently overlook the rich process information generated during an agent's interaction with its environment, such as the history of interactions. This paper introduces Environmental Feedback-based Credit Assignment (EFCA), a novel multi-timescale approach designed for long-horizon agentic RL. EFCA argues that environmental feedback provides valuable supervision for identifying the contribution of specific actions. It complements the traditional long-term outcome signal with two environment-grounded process signals: a short-term feedback signal that captures the immediate effect of the current action, and a medium-term state-history signal that helps identify ineffective patterns from recent interactions. These short- and medium-term signals are directly extracted from the environment's feedback and integrated into the learning process through a return reweighting mechanism. Experiments conducted on challenging environments like ALFWorld and WebShop demonstrated that EFCA consistently improved both the success rate and overall quality of tasks compared to strong baseline methods. This highlights the significant effectiveness of using environment-grounded, multi-timescale credit assignment for agentic RL in complex, long-horizon tasks.