New Benchmark for Multimodal AI Shopping Agents Launched

The increasing reliance of online shoppers on AI assistants, often involving both images and multi-turn dialogue to express complex product needs, highlights a gap in current evaluation methods. Existing benchmarks for shopping agents typically rely on text-only or synthetic requests, failing to capture the intricacies of real-world multimodal interactions. To bridge this gap, researchers have developed MMShopBench, the first benchmark derived from real shopping logs for multimodal, multi-turn AI shopping agents. MMShopBench is built from meticulously cleaned and manually annotated shopping logs, providing ground-truth annotations for purchase intent and mandatory product requirements within each request. Agents evaluated on this benchmark must infer these requirements from a combination of user images and conversational dialogue. They then need to retrieve suitable candidate products using both image and text search, and finally, verify that these candidates meet all specified requirements by analyzing product images and structured attributes. The benchmark evaluates both open-source and proprietary models using an evidence-grounded multimodal protocol. A companion training set is also provided for fine-tuning open-source models. Experiments show that fine-tuning significantly narrows the performance gap between open-source models and leading proprietary solutions, demonstrating the effectiveness of the training data and the benchmark's utility in fostering reproducible experimentation within an offline shopping sandbox.