New Protocol Proposed for Evaluating Personal LLM Agents.

Current benchmarks for evaluating AI agents, particularly personal Large Language Model (LLM) agents, are often insufficient because they assess capabilities in isolation. These agents are unique in that they maintain persistent memories, learned skills, tool configurations, and policy states that continuously evolve with each user interaction. Existing evaluations typically test tools, memory, or safety as static, separate components, failing to capture the dynamic and interconnected nature of personal agents. The paper advocates for a fundamentally different evaluation protocol that focuses on user-conditioned adaptation under temporal interventions. This new approach requires replaying the same temporal intervention across various persistent, user-conditioned states to observe how failures might propagate across different agent components. The authors formalize this need with four key conditions: explicit temporal intervention, persistent state across the intervention, induced cross-dimensional effects, and variation in user-conditioned state. A focused audit of public benchmarks revealed that none of the reviewed protocols fully satisfy all four proposed conditions, highlighting a significant gap in current evaluation methodologies. This position paper not only identifies this gap but also proposes a minimal benchmark design and candidate reporting metrics specifically tailored for user-conditioned adaptation, providing a concrete framework for future personal-agent evaluation.