Diagnosing GUI Agent Reliance on Pixels vs. Structure.

Multimodal GUI agents interact with interfaces by processing both visual information (screenshots) and structural data (like DOM trees). Before taking action, these agents form a belief about the current state of the interface. However, it's unclear whether this belief is primarily derived from what they "see" (pixels) or what they "read" (structure). This study introduces "visual state reliance," a new metric to diagnose how much an agent's state belief is attributed to pixels versus structure or prior knowledge. Using paired single-channel interventions on various web, mobile, and desktop interfaces, researchers found that when there's a conflict between visual and structural information, agents often prioritize the structural data. This "Perception-Fusion Gap" indicates that agents might correctly perceive visual information but still resolve conflicts by deferring to structural text, leading to incorrect actions and task failures in live environments. Coordinate-action agents, which rely more directly on visual positions, were found to be largely immune to this issue, suggesting a specific vulnerability in agents that process serialized text and indexed actions.