Gated-BEPO Improves LLM Agent Credit Assignment in Long-Horizon Tasks

Training Large Language Model (LLM) agents to perform tasks over extended periods, especially in environments with sparse rewards, presents a significant challenge in accurately attributing success or failure to specific actions. Traditional methods often assign credit uniformly across an entire sequence of actions, making it difficult to discern effective actions from ineffective ones within a long trajectory. More recent approaches attempt to group similar states and compare actions, but still rely on direct trajectory outcomes and fixed-weight credit fusion. This new research introduces Gated-BEPO, a novel framework designed to enhance credit assignment for LLM agents. Gated-BEPO constructs empirical rollout graphs for groups of actions and estimates the value of states using a Bellman fixed-point approach, which reflects the agent's current policy. It then accumulates these temporal-difference residuals along trajectories to generate step-level Bellman advantages, capturing both immediate and future impacts of actions. A key innovation is the "confidence gate," which selectively incorporates step-level Bellman credit only when there's sufficient observational data (multiple observed successors for a state), otherwise defaulting to episode-level credit. This adaptive fusion mechanism allows for more nuanced and accurate credit assignment. Experiments across various environments, including WebShop and ALFWorld, demonstrate that Gated-BEPO consistently improves the performance of both language and vision-language models, highlighting the benefits of its Bellman fixed-point value estimation and selective credit incorporation.