WebStep Benchmark Offers Process-Level Evaluation for Web Agents.

Evaluating web agents, which perform long sequences of interactions, has traditionally focused solely on terminal success, overlooking the detailed process. This approach provides limited insight into how agents can be improved. To address this, a new benchmark called WebStep has been developed, comprising 1,800 task instances with controlled difficulty and automatic semantic state tracking. WebStep operates by exposing a deterministic semantic Markov Decision Process (MDP) alongside the graphical user interface (GUI) of a website. This allows the environment to record high-level states and transitions in the background as the agent interacts with the interface, enabling fine-grained analysis without the need for manual annotation. Using WebStep, researchers demonstrated that process metrics reveal crucial differences in agent performance that are not apparent from outcome-based evaluations. For instance, agents with similar success rates showed divergent strengths in exploration versus execution accuracy. The benchmark also allows for skill decomposition, exposing opposite per-skill rankings within the same website, and bifurcation analysis to localize decisive errors, providing actionable insights for agent improvement.