BaRA Agent Improves Web Data Collection with BFS and Reflection

Large language model (LLM)-based web agents aim to automate web data collection, reducing the need for manual scripting. However, these agents often struggle on live websites, frequently missing relevant pages, returning incomplete multimodal outputs, or providing non-downloadable media URLs. To address these limitations, a new framework called BaRA (BFS-and-Reflection Agent) has been developed. BaRA is designed for site-level data collection within a fixed interaction budget, integrating a bounded breadth-first search (BFS) traversal strategy with a history-based self-reflection mechanism. Evaluations on 50 synthetic websites with ground-truth references, as well as three public websites featuring complex or dynamic layouts, demonstrated BaRA's superior performance. It significantly outperformed Pure LLM, SeeAct-Vision, and Browser-use agents in link discovery and the extraction of downloadable multimodal content, showing particular strength in recovering valid images and videos.