GRID Learns Universal Behaviors from Diverse Agent Observations

Humans often learn new skills by observing others, inferring effective actions from demonstrated behaviors. However, when observations come from a diverse group of agents with varying goals, it becomes challenging to discern which behaviors are universally beneficial. This research addresses this problem with General Reward Inference and Disentanglement (GRID). GRID is a social learning methodology designed to extract universally useful behaviors from a heterogeneous population of demonstrators. It achieves this by decomposing each agent's reward function into two parts: a "general reward" that captures behaviors common across all agents, and "specific rewards" that reflect individual preferences and objectives. Training an agent exclusively on the general reward provides a novel paradigm for generalist pretraining. This approach yields a generalist agent that internalizes fundamental environmental competencies, such as safety and basic task proficiency, without the mode-averaging bias often seen in standard learning from demonstration techniques. This generalist agent then serves as a superior prior for fine-tuning to specific downstream tasks, even those with preferences not observed during initial training. Experiments across various environments, including multi-agent Craftax and an autonomous driving simulator, confirm GRID's ability to semantically disentangle reward structures, outperform baseline methods, and enable more efficient and stable specialization.