Web Agents Monitored Without Internal Model Signals

Monitoring the reliability of web agents poses a challenge, particularly when internal model signals like token logits, which indicate uncertainty, are unavailable. This study investigates a method for predicting prefix-level risk in web agents by exclusively utilizing observable trajectory signals. The goal is to determine if an agent's current execution remains on track or is heading towards failure, based solely on its interactions with the environment. The researchers developed two types of observable trajectory representations: Macro features, which summarize the agent's cross-step behavior and feedback from the environment, and Micro features, which assess the consistency of the agent's intentions, actions, and anticipated state changes through repeated black-box queries. A novel labeling approach, called key-step boundary, is introduced. Instead of simply inheriting the final failure label, this method identifies the first critical error that remains uncorrected and leads to ultimate failure, allowing valid early prefixes of failed trajectories to still be considered "on track." Evaluations conducted on WebArena-Lite and Online Mind2Web benchmarks, using five different open- and closed-source LLM backbones, demonstrated that these observable trajectory signals are competitive with baselines that rely on internal model signals. The resulting predictors also support early intervention strategies within fixed false-cut budgets and exhibit transferability across different website categories, highlighting their potential for robust web agent monitoring.