New Framework Boosts GUI Agent Performance with Skill-Guided Distillation

A new research paper introduces Skill-Guided Continuation Distillation (SGCD), an iterative framework designed to enhance the performance of GUI (Graphical User Interface) agents. Current methods often rely on imitating expert demonstrations, but agents frequently encounter situations not covered by these pre-recorded paths, known as "off-trajectory states." SGCD tackles this supervision gap by first allowing the agent to explore and reach these realistic off-trajectory states. Then, a skill-guided policy takes over to successfully complete the task from these novel states, generating new, successful continuation trajectories. These newly generated continuations are then combined with the original expert data, providing the agent with crucial supervision for previously unseen scenarios. By extracting skills like "Continuation Plans" and "Failure Traps" from both successful and failed attempts, SGCD significantly improved the success rates of various base models on the OSWorld-Verified benchmark.