SeerGuard Enhances Mobile GUI Agent Safety with World Model Prediction

Mobile Graphical User Interface (GUI) agents, while powerful for task automation, pose significant safety risks due to the potential for irreversible erroneous actions. Existing safety mechanisms are largely reactive, failing to assess risks proactively. To address this, researchers introduce SeerGuard, a novel consequence-aware safety framework designed to prevent such issues through pre-execution screening. SeerGuard operates on two levels: instruction-level screening and action-level risk assessment. The action-level assessment analyzes proposed agent actions within the current GUI state, predicting likely outcomes to identify potential risks before execution. This capability is powered by a unified safety-augmented world model (SAWM), which is built via multi-task learning to integrate semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate SeerGuard's effectiveness across various mobile GUI agents, showing a substantial increase in safety-utility scores and a reduction in risk-cost scores. Further analysis confirms the efficacy of both instruction-level screening and the SAWM's predictive capabilities.