Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce CO-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse.

Introduction. Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful approach for improving reasoning in large language models [Lightman et al., 2024, DeepSeek-AI, 2025], yet its strongest successes still depend heavily on ground-truth supervision. Such supervision is costly to obtain and becomes increasingly scarce as target reasoning capabilities approach or surpass what humans can reliably evaluate [Yue et al., 2025]. Self-rewarding RL reduces this dependence by deriving rewards from the model’s own completions, incorporating signals such as agreement with its majority-vote prediction [Zuo et al., 2025], self-certainty [Zhao et al., 2026], predictive entropy [Prabhudesai et al., 2025], or consistency across paraphrased inputs or moving-average policies [Zhang et al., 2026b]. However, these signals remain within a single model’s own predictions.

Discussion / Conclusion. In this work, we introduced CO-RL, a label-free multi-agent RL framework for reasoning tasks. In our framework, multiple agents learn from rewards constructed from their peers’ predictions rather than ground-truth labels or external judges. Across text-only and multimodal reasoning benchmarks, CO-RL consistently improves diverse LLMs and VLMs, outperforming prior self-rewarding and multi-agent RL approaches and, in many settings, matching or surpassing training with groundtruth rewards. Our theoretical analysis shows that cross-agent supervision expands the set of initial conditions that converge to the correct solution, allowing CO-RL to correct errors that self-rewarding RL would otherwise reinforce. An important direction for future work is to understand how the number, diversity, and interaction topology of agents shape cross-agent learning, and to develop adaptive supervision mechanisms that more effectively exploit complementary expertise.