GRPO Fails to Improve Small LLM Web Agents' Performance

A new study investigates the effectiveness of Group Relative Policy Optimization (GRPO) when applied to small language and vision-language model web agents, specifically those at the 4B to 8B parameter scale. The common practice is to run reinforcement learning, like GRPO, on a supervised checkpoint hoping to enhance agent capabilities. However, across a comprehensive control grid of 18 experimental runs, the researchers found no credible improvement in the success rate of a strong supervised baseline on tasks the agent had already largely mastered. In fact, for the text-based tasks, moderate to high learning rates actually led to a credible degradation in performance. This "null" finding remained consistent under various testing conditions, including paired testing, multiple evaluation and training seeds, recipe changes, and different observation types (text and Set-of-Marks screenshots), and even when scaling the backbone to 8B parameters. The detrimental effect was primarily observed on the text track. To confirm that the pipeline was functional, the identical setup was applied to tasks where the reward was easily reachable by sampling, and in those cases, the success rate significantly increased by 22 points. This indicates that GRPO is only beneficial when there's substantial room for the agent to improve, meaning the sampled policy already outperforms the greedy one. The paper further explains the mechanism of failure, showing that middle learning rates degrade the agent and high ones cause collapse, with distinct causal localizations for these effects within the model's architecture.