New Benchmark Evaluates Personalized LLMs with User Behavior.

Current benchmarks for personalized Large Language Models (LLMs) often rely on static textual personas or isolated behavioral signals, which fall short in evaluating how LLMs personalize responses based on a user's heterogeneous, real-world daily activities across multiple domains. To address this critical gap, researchers have introduced LUNAR, a pioneering benchmark specifically designed to assess LLMs' ability to personalize responses using longitudinal app interaction histories. This includes diverse domains such as clothing, food, housing, and mobility. LUNAR's construction employs a multi-stage, coarse-to-fine synthesis pipeline, grounded in real-world behavioral patterns, to ensure scalability while mitigating data sparsity and privacy concerns. Fidelity analyses confirm that LUNAR's synthetic data closely aligns with actual behavioral distributions, making it a robust tool for evaluation. Experiments conducted on 19 mainstream LLMs using LUNAR yielded significant insights. The findings indicate that while access to behavioral logs is necessary for deep personalization, it is not sufficient on its own. Simply providing more context or using larger models does not automatically guarantee improved performance. Instead, effective personalization critically depends on the intelligent selection and integration of relevant evidence across different domains. Furthermore, the research highlights a crucial trade-off: stronger personalization often comes at the cost of privacy protection. Direct retrieval of fine-grained behavioral records consistently outperformed compressed memory approaches, underscoring the challenges in balancing utility and privacy for personalized LLMs.