Setoka Benchmark Evaluates Deeper User Understanding for Personalized Agents

Personalized AI agents need more than just recalling explicit facts from past interactions; they require a deeper understanding of users, including abstract characteristics. Existing benchmarks primarily test simple information retrieval, failing to adequately assess this nuanced user comprehension. This paper introduces Setoka, a new benchmark specifically designed to evaluate memory-augmented personalized agents across multiple levels of user understanding. Setoka is grounded in cognitive and personality psychology, defining four hierarchical levels: semantic memory, episodic memory, behavior patterns, and personality traits. To enable realistic yet privacy-preserving evaluation, the benchmark employs a psychometrics-based pipeline to synthesize diverse and coherent heterogeneous user data and queries at scale. Initial evaluations using Setoka reveal that current language models combined with various memory systems perform well on semantic memory but struggle significantly with episodic memory, and even more so with tasks requiring integration of fragmented, long-term information to infer behavior patterns and personality traits. This highlights the need for more sophisticated memory mechanisms in personalized agents.