Multi-modal LLMs Guide Open-Ended Multi-Agent Learning Curricula

Training generally capable agents in Reinforcement Learning (RL) often relies on open-ended curricula, which involve identifying tasks that progressively build complex skills. A significant hurdle in designing these curricula is accurately gauging task difficulty relative to an agent's current learning stage. Traditional approaches have used scalar scores or textual summaries of agent behavior, but this research explores a novel method: directly observing policy behavior through recorded episode videos. The proposed method, termed Visual Inspection of Policies (VIP), utilizes a Video Language Model (VLM) to process these videos and generate curriculum recommendations. This approach allows for a richer understanding of agent performance and task complexity than simpler metrics. The researchers empirically evaluated VIP on the StarCraft Multi-Agent Challenge (SMAC), a complex multi-agent environment. Even with a lightweight and openly accessible VLM like VideoLLaMa2-7B, VIP demonstrated superior performance. It generated more effective curricula compared to both its text-only ablation and methods that rely solely on scalar task scores. This highlights the potential of multi-modal LLMs to significantly advance the development of sophisticated, self-improving AI agents by providing a more intuitive and comprehensive way to guide their learning process.