MAG Benchmark Unifies Web Agent Actions and Guide Generation

Digital Adoption Platforms (DAPs) are widely used to guide users through web systems, but current AI approaches often treat automated web actions and guide generation as separate problems, typically relying on textual page representations. This research introduces MAG, a novel benchmark that integrates these two tasks into a single "Multimodal Action and Guide" challenge. Unlike previous methods, MAG grounds its actions and guide generation directly on screenshots, using either Set-of-Mark element selection or raw pixel coordinates, mimicking how humans interact with web pages. The MAG harness provides a complete system for annotation (with LLM assistance and human verification), training, and evaluation in live web environments, using joint metrics for both actions and guides. Initial evaluations of frontier API models and open multimodal models reveal that even the strongest models complete fewer than 40% of tasks, indicating substantial room for future research. The paper also proposes a GRPO training method, augmented with expert trajectories, which nearly doubles the success rate of a supervised 9B agent, improving both task completion and guide quality.