New Benchmark Evaluates Shopping Agents on Complex Tasks

As large language model (LLM)-based shopping agents become more prevalent, existing benchmarks often fall short in capturing the complexity of real-world shopper interactions. These benchmarks typically expose all requirements upfront and only grade the final product choice, failing to account for how a shopper's needs might be implicitly stated, stored in a profile, or revealed only through clarification questions. This limits their ability to assess an agent's performance on long-horizon tasks or diagnose specific missed requirements. To address this gap, researchers have introduced EComAgentBench, a novel benchmark comprising 662 tasks based on actual Amazon products and reviews. Each task in EComAgentBench distributes shopper requirements across a visible query, a tool-gated profile, and scripted clarification interactions. Agents must uncover this hidden intent, verify potential product candidates against attributes and review evidence, and commit to a single product within a limit of 100 tool calls. The benchmark employs typed, source-tagged rubrics to grade every task, attributing failures to specific requirements and their origins. This automated yet reliable construction ensures that every answer is fixed in code before text generation and every sample is validated. Evaluations of seven different models using EComAgentBench revealed that even the most capable models achieved only 57.1% overall accuracy, with rubric satisfaction degrading significantly from visible to hidden requirement sources. EComAgentBench is expected to serve as a foundational tool for advancing shopping agents beyond simple search to dependable, long-horizon assistance.