New AI Agent Improves Web Automation by Reusing Interaction Patterns

Large language model (LLM) web agents typically operate by observing a web page and then executing low-level tool actions. This method can lead to long action sequences and high computational costs, especially for complex tasks. To mitigate this, previous systems have introduced "web skills," which are higher-level callable tools that encapsulate repeated interaction fragments. However, these prior skill libraries often suffer from low reuse across different websites because they rely on instruction similarity or coarse site metadata for triggering. Researchers have developed SkillMigrator, an agent designed to overcome these limitations. SkillMigrator learns reusable web skills and effectively transfers them across various sites by focusing on matching layout structure rather than specific element references. Each learned skill is stored as a "transferable interaction pattern" (TIP), which combines the skill with a structural sketch of the web page at the time of its induction. During execution, SkillMigrator retrieves relevant TIPs based on layout similarity and then grounds their references onto the live web page. This innovative approach has been shown to reduce the average number of LLM actions on successful trajectories by 8-10% across benchmarks like WebArena and Mind2Web, all while maintaining comparable success rates. This advancement offers a more efficient and cost-effective way to automate complex web interactions.