APeB Benchmarks LLM Agent Personalization for Raw Queries

Researchers have developed a new benchmark, Agent Personalized Benchmark (APeB), specifically designed to test the personalization abilities of LLM-powered agents. Existing benchmarks often fall short by relying on pre-refined user queries or simplified interaction histories, failing to capture real-world challenges. APeB focuses on "personalized product search" (PPS), where agents must infer user intent and extract preferences from raw, underspecified queries and complex interaction histories. The study constructed APeB using action logs, pairing vague intents with rich historical data and candidate items. Evaluations of state-of-the-art LLMs using multi-step agent workflows revealed that while models handle explicit queries well, they struggle significantly with early-stage queries requiring the discovery of intent and preferences. A detailed analysis attributed this gap primarily to ineffective use of historical data. A simple history-aware query-refinement pipeline, VQRA, demonstrated consistent improvements, underscoring the necessity for specialized modules dedicated to leveraging interaction history in personalized agents.