Incognita Evaluates Generative Agents in Socially Distributed Tasks.

This research addresses the challenge of evaluating generative agents in complex social environments where tasks require interaction and knowledge is distributed among various participants. Traditional benchmarks often lack either grounded actions or rich social interaction. The authors define "socially distributed task environments" as settings where task-relevant knowledge is isolated by roles, and consequential actions are only accessible through specific entities. To evaluate agents in such settings, they developed Incognita, a Concordia-based framework. Incognita separates social interaction from grounded execution, allowing agents to route messages to users or specialist entities. These specialists mediate operations, which are then executed in a deterministic sub-environment. An offline evaluator scores outcomes based on inherited rewards. Using Incognita-Retail, a transformation of a retail benchmark into a multi-entity environment, three generative agent models were evaluated across 18 tasks. While success rates remained low (0% to 17.2%), the evaluations showed significant behavioral progress, including increased knowledge elicitation and attempts at grounded actions. This indicates that socially distributed environments effectively expose nuanced agent behaviors before full task reliability is achieved.