Multi-Agent RL Enables Unsupervised Reasoning in LLMs

This research presents Co-RL, a novel framework that facilitates unsupervised reasoning in large language models (LLMs) and vision-language models (VLMs) through cooperative multi-agent reinforcement learning. While traditional RL for reasoning often relies on costly ground-truth supervision, Co-RL addresses this by allowing multiple independent models to learn and optimize based on rewards derived from their peers. A key finding is that increasing the diversity within the cohort—by using different model families, sizes, and varied training samples—is crucial. This diversity helps to reduce correlated errors and biases that can lead to homogenized responses and training collapse, a common issue in self-rewarding systems. Co-RL consistently improves reasoning performance, maintains behavioral diversity, and prevents training collapse. It achieves significant gains across multiple text-only and multimodal benchmarks, matching or even surpassing supervised methods without requiring any ground-truth labels, marking a substantial step towards truly unsupervised reasoning.