UserToolBench Benchmarks Personalized Decision-Making in Tool-Use LLMs.

As Large Language Models (LLMs) are increasingly tasked with acting as user delegates, the ability to make personalized decisions becomes crucial. Existing benchmarks often focus on aspects like profile recall or style imitation, but fail to adequately test an LLM's capacity for personalized decision-making in tool-use scenarios. A new benchmark, UserToolBench, aims to fill this gap by assessing whether models can infer user preferences from interaction history, identify when clarification is needed, and generate user-aligned tool-call trajectories even with incomplete information. UserToolBench is constructed from privacy-sanitized real interaction traces, incorporating structured persona profiles, public API-style tool ecosystems, and complex multi-turn trajectories. It features 10 user profiles, 36 tool sets, 1,065 turns, and 170 unique tools, with tasks covering scenarios like missing information, single-tool use, and multi-tool coordination. Experiments conducted with powerful tool-use LLMs on UserToolBench indicate that current models still face significant challenges in personalized delegation. Key bottlenecks include effective multi-tool coordination, inferring missing constraints, and maintaining long-horizon behavioral consistency. These findings suggest that future personalization evaluation should shift from merely assessing user-specific output phrasing to scrutinizing whether LLMs make genuinely correct decisions for the users they represent.