AgentWorld Evaluates AI Agent Reliability with Personality-Aware Users.

This research introduces AgentWorld, a novel simulation framework designed to rigorously evaluate the reliability of agentic information retrieval systems. Current evaluation methods often fall short by relying on scripted interactions with uniform users, failing to capture the natural diversity of human personalities or the adversarial vulnerabilities of AI agents. AgentWorld addresses this by integrating user populations driven by the Big Five (OCEAN) personality traits into stateful tool-use environments. The framework employs advanced metrics like pass$^k$ consistency, structured fault classification, and a dual-control handoff verification system. Crucially, it includes an adversarial Risk Analyser that identifies brittle trajectories by branching Monte-Carlo rollouts under various perturbation types and quantifies risk using sophisticated scoring and attribution methods. Experiments demonstrated that personality variations among simulated users exposed significant failure modes—such as cross-domain leakage and contextual drift—that uniform testing could not detect, leading to substantial quality gaps across personas. The Risk Analyser further revealed inherent trajectory brittleness and identified tool/infrastructure-layer attacks as dominant failure causes. This highlights the critical importance of personality-aware and adversarial testing for developing truly robust AI agents.