Paper deep dive
Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/20/2026, 3:58:59 AM
Summary
The paper introduces Co-RL, a cooperative multi-agent reinforcement learning framework that enables unsupervised reasoning in language and vision-language models. By using multiple decoupled models with diverse architectures and sizes to supervise each other via peer-derived rewards, Co-RL mitigates the training collapse and bias amplification common in self-rewarding methods. The approach achieves significant performance gains across text-only and multimodal benchmarks without ground-truth labels.
Entities (12)
Relation Signals (11)
Co-RL → appliesto → LLMs
confidence 95% · Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs
Co-RL → appliesto → VLMs
confidence 95% · and 2.3-7.2% across four multimodal benchmarks for VLMs.
Co-RL → mitigates → Training Collapse
confidence 95% · This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse.
Co-RL → evaluatedon → We-Math
confidence 90% · The gains extend consistently to multimodal reasoning, where Co-RL improves five VLMs... across... We-Math.
Co-RL → evaluatedon → MathVista
confidence 90% · The gains extend consistently to multimodal reasoning, where Co-RL improves five VLMs... across... MathVista...
Co-RL → evaluatedon → MathVerse
confidence 90% · The gains extend consistently to multimodal reasoning, where Co-RL improves five VLMs... across... MathVerse...
Co-RL → evaluatedon → MathVision
confidence 90% · The gains extend consistently to multimodal reasoning, where Co-RL improves five VLMs... across MathVision...
Co-RL → uses → GRPO
confidence 90% · The resulting rewards drive policy optimization with GRPO (Shao et al. 2024) or REINFORCE++ (Hu et al. 2025).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.17253v2
- Canonical: https://arxiv.org/abs/2608.17253v2
Trouble viewing inline? Open PDF directly →
Full Text
108,081 characters extracted from source content.
Expand or collapse full text
Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL Yunhao Yang Yuexin Bian Yunjie Tian Affiliation: University of Exeter Independent Researcher[3pt] yijiangli, nuno@ucsd.edu Di Fu Affiliation: University of Exeter Independent Researcher[3pt] yijiangli, nuno@ucsd.edu Tianjin Huang Yuanyuan Shi Ziang Xiao Nuno Vasconcelos Yijiang Li [4pt] Johns Hopkins University UC San Diego Abstract Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0–8.6% across seven text-only benchmarks for LLMs and 2.3–7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL. †footnotetext: ∗Equal contribution. †Project lead.†footnotetext: Corresponding to Yijiang Li: yijiangli@ucsd.edu, Nuno Vasconcelos: nuno@ucsd.edu 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful approach for improving reasoning in large language models (Lightman et al. 2024; DeepSeek-AI 2025), yet its strongest successes still depend heavily on ground-truth supervision. Such supervision is costly to obtain and becomes increasingly scarce as target reasoning capabilities approach or surpass what humans can reliably evaluate (Yue et al. 2025). Self-rewarding RL reduces this dependence by deriving rewards from the model’s own completions, incorporating signals such as agreement with its majority-vote prediction (Zuo et al. 2025), self-certainty (Zhao et al. 2026), predictive entropy (Prabhudesai et al. 2025), or consistency across paraphrased inputs or moving-average policies (Zhang et al. 2026b). However, these signals remain within a single model’s own predictions. Without an external reference, such self-reinforcement can amplify existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to increasingly homogeneous outputs or even training collapse. This raises a fundamental question: how can a model obtain a sufficiently independent learning signal to improve without any ground-truth supervision? In this work, we show that such a signal can emerge from independently trained models. Since their errors are not perfectly correlated, each model can provide corrective feedback that the other cannot derive from its own generations. We therefore take the learning signal from a separate model, giving each agent decorrelated supervision: a target produced by independently updated weights that is less likely to echo its own biases (Blum and Mitchell 1998; Li et al. 2023). Building on this insight, we introduce Co-RL, a cooperative multi-agent label-free RL method in which multiple decoupled models, sharing no parameters, are optimized simultaneously using rewards derived from their cohorts. Given an unlabeled prompt, each agent samples multiple completions and aggregates them into a pseudo-answer through majority voting (Wang et al. 2023). The completions of one agent are then rewarded against another agent’s pseudo-answer, and the resulting rewards drive policy optimization with GRPO (Shao et al. 2024) or REINFORCE++ (Hu et al. 2025). Unlike self-rewarding methods, where the policy and the reward come from the same model, Co-RL draws supervision from an independently updated partner, which breaks the feedback loop that amplifies a model’s own bias. What a cohort teaches depends on how different its mistakes are. Highly similar models tend to make correlated errors and may reinforce the same incorrect answers. Therefore, cohort diversity is crucial to the effectiveness of Co-RL. Thus, we push it as far as the models allow: different families of architecture and pretrained weights, model size, and input formulation (e.g., rephrased prompt (Zhang et al. 2026b)). These differences expose agents to distinct inductive biases and decision boundaries, reducing correlated errors and strengthening the corrective signal available to their cohorts. By incorporating multiple agents in the training loop, Co-RL naturally constitutes a multi-agent RL framework. However, unlike prior multi-agent methods built around debate or iterative communication (Du et al. 2024; Liang et al. 2024), Co-RL requires no interaction between agents beyond the reward stage and eliminates the need for an external LLM judge or learned reward model (Xue et al. 2026; Park et al. 2025; Zhang et al. 2026a). The resulting framework is lightweight and symmetric: each agent serves simultaneously as a learner and a source of supervision, allowing all agents to improve within a single training run. Across text-only and multimodal domains, diverse model families, and training settings, Co-RL consistently improves upon the base models and prior label-free approaches, while matching or surpassing supervised training in several settings, without access to ground-truth labels. Across seven text-only benchmarks spanning mathematical reasoning, coding, and knowledge-intensive reasoning, Co-RL improves four LLMs by 3.0–8.6% on average across seven text-only benchmarks, outperforming the strongest self-rewarding baselines by 0.8–2.0%. The gains extend consistently to multimodal reasoning, where Co-RL improves five VLMs ranging from 2B to 12B parameters by 2.3–7.2% across MathVision, MathVerse, MathVista, and We-Math. Under the controlled evaluation setting of CoMAS (Xue et al. 2026), Co-RL further outperforms prior multi-agent RL methods by 4.0% on average while using only half as many agents. Together, these results demonstrate that cross-agent supervision provides an effective and general learning signal across modalities, model families, and reasoning domains without relying on labeled supervision or external judges. 2 Related Work Self-rewarding RL. While RLVR effectively improves LLM reasoning (Shao et al. 2024; DeepSeek-AI 2025; Yu et al. 2025b), its reliance on high-quality ground-truth labels remains a major bottleneck (Yue et al. 2025). Recent work therefore explores self-rewarding mechanisms that learn from unlabeled data. One prominent direction derives reward signals solely from a model’s own behavior, such as majority voting (Wang et al. 2023), self-consistency (Zuo et al. 2025), self-certainty (Zhao et al. 2026; Li et al. 2025), or predictive entropy (Prabhudesai et al. 2025; Zhang et al. 2025a). This line extends earlier work on self-rewarding models (Yuan et al. 2024) and self-play supervision (Chen et al. 2024c), and now includes unsupervised self-training (Xu et al. 2025a; Fang et al. 2025), self-correction based on a model’s own judgments (Xiong et al. 2025), and zero-data self-evolution, where one or more models generate their own curricula (Zhao et al. 2025a; Huang et al. 2026a; Liu et al. 2026). However, because these methods rely exclusively on a single model’s own view, they can reinforce and amplify existing biases and errors without providing an external corrective signal, often leading to substantial training collapse. Co-training, cross-view supervision, and its origins. Using a second view to supervise a learner is a longstanding idea. Co-training shows that two conditionally independent views can teach each other (Blum and Mitchell 1998), while subsequent analysis demonstrates that highly similar views tend to reinforce the same errors: the more alike the two views are, the more they simply agree on the same mistakes (Li et al. 2023). Related principles underlie self-supervised representation learning, where augmentations, momentum targets, or stop-gradient operations prevent collapse (Chen et al. 2020; Grill et al. 2020; Caron et al. 2021), and deep mutual learning, where peer networks provide reciprocal supervision (Zhang et al. 2018). Co-rewarding (Zhang et al. 2026b) brings this insight to label-free RL by deriving rewards from either paraphrased questions or a slowly updated model copy. However, because both views originate from the same model, their errors remain strongly correlated, only partially satisfying the independence required for effective two-view learning. Multi-agent RL. Several methods improve reasoning by letting multiple models interact. At inference time, multi-agent debate and round-table consensus have models critique and revise one another (Du et al. 2024; Liang et al. 2024; Chen et al. 2024a; Sun et al. 2024), and general orchestration frameworks compose such agents into pipelines (Wu et al. 2023; Zhang et al. 2024b; Kim et al. 2024). A more recent line trains the agents: CoMAS turns interactions scored by an LLM judge into rewards (Xue et al. 2026), MAPoRL and MARFT co-train agents against a learned or shared reward (Park et al. 2025; Liao et al. 2025), and other systems reinforce or bootstrap multi-agent cooperation directly (Motwani et al. 2024; Chen et al. 2025c; Zhao et al. 2025b; Zhang et al. 2026a; Chen et al. 2025d). RLVR for multimodal reasoning. RLVR has rapidly expanded to vision-language models, with R1-style rule-based rewards underpinning a broad family of methods (Huang et al. 2026b; Shen et al. 2025; Feng et al. 2025; Liu et al. 2025b; Meng et al. 2025; Peng et al. 2025; Yang et al. 2025b; Wang et al. 2025b). Existing work mainly improves reward design and training stability through curriculum or perception-aware rewards (Deng et al. 2025a; Yu et al. 2025a; Wang et al. 2025a), data augmentation and selection (Liu et al. 2025a; Wang et al. 2025c), and staged supervised-to-RL training (Deng et al. 2025b; Chen et al. 2025a), but still largely assumes verifiable labels. Label-free multimodal RL remains underexplored, and existing methods again rely on a single model’s own signals (Wei et al. 2025). Cross-view supervision is especially promising here, since VLMs rarely share a vision encoder, pairing InternViT (Chen et al. 2024b), SigLIP (Gemma Team 2025), or a native-resolution encoder (Bai et al. 2025; Zhang et al. 2025c; Zhang et al. 2025b) with different LLMs, yielding distinctive families of models. 3 Preliminary 3.1 Reinforcement Learning for Reasoning Given a prompt x∼x , an autoregressive language model πθ _θ generates a response y=(y1,…,yT)y=(y_1,…,y_T) according to πθ(y∣x)=∏t=1Tπθ(yt∣x,y<t) _θ(y x)= _t=1^T _θ(y_t x,y_<t). Reinforcement learning optimizes the policy using a scalar reward r(x,y)r(x,y) that evaluates the quality of the generated response. The corresponding objective is (θ)=x∼,y∼πθ(⋅∣x)[r(x,y)].J(θ)=E_x ,\,y _θ(· x) [r(x,y) ]. (1) For reasoning tasks, the response y typically contains both an intermediate reasoning trajectory and a final answer, while the reward is commonly assigned at the response level based on answer correctness or another outcome-based criterion. Among policy optimization algorithms, Group Relative Policy Optimization (GRPO) has been widely adopted as it improves reasoning performance without requiring a separately trained critic model (Shao et al. 2024; DeepSeek-AI 2025). Given a prompt x, the old policy πθold _ _old samples a group of K responses y1,…,yK\y^1,…,y^K\ and assigns each response a reward rk=r(x,yk)r^k=r(x,y^k). GRPO estimates the advantage of each response by normalizing rewards within the group: A^k=rk−mean(rjj=1K)std(rjj=1K)+ϵ A^k= r^k-mean(\r^j\_j=1^K)std(\r^j\_j=1^K)+ε, where ϵε is a small constant for numerical stability. Let ρk,t(θ)=πθ(ytk∣x,y<tk)πθold(ytk∣x,y<tk) _k,t(θ)= _θ(y_t^k x,y_<t^k) _ _old(y_t^k x,y_<t^k) denote the token-level importance ratio. GRPO optimizes the clipped surrogate objective GRPO(θ)=[1K∑k=1K1|yk|∑t=1|yk|[min(ρk,t(θ)A^k,clip(ρk,t(θ),1−δ,1+δ)A^k)−βKLk,t]],J_GRPO(θ)=E\! [ 1K _k=1^K 1|y^k| _t=1^|y^k| [ \! ( _k,t(θ) A^k,clip\! ( _k,t(θ),1-δ,1+δ ) A^k )- _KL^k,t ] ], (2) where δ is the clipping threshold, β controls the strength of KL regularization, and KLk,tD_KL^k,t penalizes deviation from a reference policy πref _ref. GRPO typically uses verifiable rewards from ground-truth answers or external verifiers. However, such supervision can be costly or difficult to obtain at scale. The following subsection considers model-generated rewards as an alternative, enabling reinforcement learning on unlabeled prompts. 3.2 Self-rewarding RL In the absence of external verifiers, reward functions can be constructed directly from a model’s own outputs, thereby enabling RL on unlabeled prompts. Given an unlabeled prompt x, the policy πθ _θ samples responses y1,…,yKy^1,…,y^K, with answers ak=g(yk)a^k=g(y^k) extracted from each response. TTRL (Zuo et al. 2025) defines the reward for response yky^k as rmajk=[ak=a^θ(x)]r_maj^k=1[a^k= a_θ(x)], where a^θ(x)=argmaxb∑j=1K[aj=b] a_θ(x)= _b _j=1^K1[a^j=b] is the policy’s majority-vote answer. Intuitor (Zhao et al. 2026) instead measures token-level confidence using rconfk=1|yk|∑t=1|yk|DKL(∥πθ(⋅∣x,y<tk))r_conf^k= 1|y^k| _t=1^|y^k|D_KL(U\| _θ(· x,y_<t^k)), where U denotes the uniform distribution over the vocabulary. RENT (Prabhudesai et al. 2025) operates on the predictive distribution, assigning rentk=−1|yk|∑t=1|yk|ℋ(πθ(⋅∣x,y<tk))r_ent^k=- 1|y^k| _t=1^|y^k|H( _θ(· x,y_<t^k)), where ℋ(⋅)H(·) denotes entropy. Thus, TTRL (Zuo et al. 2025) rewards responses that agree with the policy’s majority-vote prediction, whereas Intuitor (Zhao et al. 2026) and RENT (Prabhudesai et al. 2025) derive rewards from the model’s token-level predictive distributions. Despite their different reward constructions, these methods ultimately produce scalar rewards that can be used for policy optimization. Under GRPO, the self-generated rewards rkk=1Kr^k_k=1^K are normalized into group-relative advantages A^k A^k and used directly in Eq. (2). Despite their effectiveness, these signals originates from the same policy being optimized: without an external reference, training may reinforce existing biases and suboptimal behaviors, reduce response diversity, and eventually lead to homogenized responses or training collapse. We show this in (b) and (c) of Figure 2, where prolonged training causes TTRL to degenerate and lead to training collapse. Figure 1: Comparison of Co-RL with prior label-free RL methods. Both TTRL and Co-rewarding derive rewards from self-generated agreement, and CoMAS scores multi-turn interactions with one of its own agents acting as judge. Co-RL instead derives rewards directly from peer votes. Beyond two agents, the votes pass along a directed ring (N=3N=3 shown). 4 Method 4.1 Co-Reinforcement Learning (Co-RL) This motivates a fundamental question: how can a model obtain a sufficiently independent learning signal without any ground-truth supervision? Our key insight is that such a signal can emerge from independently trained models, since their errors are decorrelated; one model can provide corrective feedback that another cannot derive from its own generations (Blum and Mitchell 1998; Li et al. 2023). We provide an overview of our method in the right of Figure 1, in comparison with prior self-rewarding RL paradigms. Building on this principle, we introduce Co-Reinforcement Learning (Co-RL), a label-free multi-agent reinforcement learning method in which multiple decoupled agents independently generate completions to unlabeled problems and supervise one another through majority-voted pseudo reward from its cohort. The agents share neither parameters nor gradients; their optimization is coupled solely through the rewards they provide to one another. Formally, let x∼x denote an unlabeled reasoning problem. We consider a cohort of N agents with independently parameterized policies πθnn=1N\ _ _n\_n=1^N. The agents may be initialized from the same pretrained model or from models of different families and sizes. For each unlabeled reasoning problem x, each agent n∈1,…,Nn∈\1,…,N\ independently rollout a group of K completions ynk∼i.i.d.πθn(⋅∣x),k=1,…,Ky_n^k i.i.d. _ _n(· x),k=1,…,K and extracts the corresponding final answers as ank=g(ynk)a_n^k=g(y_n^k), with g(⋅)g(·) denoting an answer-extraction function. For each agent n, Co-RL constructs a supervision target exclusively from the answers generated by one designated peer. Specifically, the pseudo-label is constructed as a^−n(x)∈argmaxb∑j=1K[an−1j=b], a_-n(x)∈ _b _j=1^K1\! [a_n-1^j=b ], (3) where b ranges over the answers produced by agent n−1n-1, with the index taken cyclically so that agent 11 is supervised by agent N. Thus, a^−n(x) a_-n(x) represents the majority-vote answer of the peer that supervises agent n. The reward assigned to the k-th response of agent n is then rnk=[ank=a^−n(x)]r_n^k=1\! [a_n^k= a_-n(x) ]. A response therefore receives reward 11 when its extracted answer agrees with the constructed pseudo-label, and reward 00 otherwise. In the two-agent setting, the two agents supervise each other. Crucially, each agent does not contribute to its own supervision target a^−n(x) a_-n(x). Policy Optimization. Without loss of generality, we use GRPO (Shao et al. 2024) to train our agents. For each agent, the rewards rnkk=1K\r_n^k\_k=1^K are normalized within its own rollout group to obtain group-relative advantages, which are then used in the GRPO objective in Eq. 4 to optimize the agent. Let =(θ1,…,θN) θ=( _1,…, _N). The overall training objective can be written as maxCo-RL()=1N∑n=1NGRPO(θn,ynk,rnkk=1K), _ θ\;J_Co -RL( θ)= 1N _n=1^NJ_GRPO ( _n;\y_n^k,r_n^k\_k=1^K ), (4) where GRPO(θn)=[1K∑k=1K1|ynk|∑t=1|ynk|min(ρn,tkA^nk,clip(ρn,tk,1−ϵ,1+ϵ)A^nk)]−βDKL(πθn∥πref,n). _GRPO( _n)=E [ 1K _k=1^K 1|y_n^k| _t=1^|y_n^k| ( _n,t^k A_n^k,\,clip( _n,t^k,1-ε,1+ε) A_n^k ) ]-β D_KL ( _ _n\,\|\, _ref,n ). (5) A^nk=rnk−mean(rnjj=1K)std(rnjj=1K),ρn,tk=πθn(yn,tk∣xn,yn,<tk)πθnold(yn,tk∣xn,yn,<tk). A_n^k= r_n^k-mean\! (\r_n^j\_j=1^K )std\! (\r_n^j\_j=1^K ), 18.49988pt _n,t^k= _ _n (y_n,t^k x_n,y_n,<t^k ) _ _n^old (y_n,t^k x_n,y_n,<t^k ). (6) Figure 3 illustrates the two-agent setting. At each training step, all rollouts and majority-vote pseudo-labels are computed before any policy update. Given the resulting rewards, each policy is then updated independently. Algorithm 1 summarizes the training procedure, in which all agents are updated at each optimization step. Our framework is also compatible with other policy optimization methods that support sequence-level rewards. Algorithm 1 Co-Reinforcement Learning (Co-RL) 1: Agents πθnn=1N\ _ _n\_n=1^N initialized from different models 2: for each training step do 3: Sample a batch of unlabeled prompts ℬ⊂B 4: for x∈ℬx do 5: for n=1,…,Nn=1,…,N do ⊳ Generate responses 6: Sample ynk∼i.i.d.πθn(⋅∣x)y_n^k i.i.d. _ _n(· x) and extract answers ank←g(ynk)a_n^k← g(y_n^k) for k=1,…,Kk=1,…,K 7: end for 8: for n=1,…,Nn=1,…,N do ⊳ Construct cross-agent rewards 9: Compute a^−n(x) a_-n(x) using Eq. (3) 10: for k=1,…,Kk=1,…,K do 11: Assign rnk←[ank=a^−n(x)]r_n^k 1\! [a_n^k= a_-n(x) ] 12: end for 13: end for 14: end for 15: Update θnn=1N\ _n\_n=1^N with one GRPO step using the objective in (4) 16: end for 4.2 Diverse Cohorts Enable Unsupervised Reasoner What a cohort teaches depends on how different its mistakes are. Highly similar models tend to make correlated errors and may reinforce the same incorrect answers. Therefore, cohort diversity plays a crucial role in Co-RL. We push this diversity as far as the framework allows: policy optimization (Sec 4.1), model families and sizes, and input formation. Each source of diversity induces distinct inductive biases and reasoning behaviors, reducing correlated errors and strengthening the corrective signal available through peer supervision. Decoupled policy optimization. As discussed in Sec. 4.1, Co-RL realizes decoupled policy optimization by training two policies independently, with interaction occurring only through the reward in Eq. (4). Each policy maintains its own parameters and optimizer state with no gradient propagated between policies. Thus, this decoupled policy optimization design also serves as a source of diversity: independent updates prevent the two policies from being directly coupled, maintaining less-correlated predictions throughout the training process. We show in Figure 2(b) and (c) that decoupled policy optimization stabilizes training and yields consistent improvements over self-rewarding methods. Diversity across model families and sizes. Our primary source of diversity comes from independently pretrained model families. Distinct families differ throughout the model-development pipeline, including architecture, tokenization, pretraining data, and post-training, each of which brings inductive biases that can benefit co-learning. For example, Qwen2.5 and Llama 3 differ in their tokenizers, vocabulary sizes, architectural choices, and pretraining corpora (Yang et al. 2024; Grattafiori et al. 2024), while Gemma introduces a 256K-token vocabulary and interleaved local–global attention (Gemma Team 2025). Such differences are even more pronounced for VLMs, whose families employ distinct vision encoders: Qwen2.5-VL uses a natively trained dynamic-resolution ViT (Bai et al. 2025), InternVL adopts InternViT (Chen et al. 2024b), and Gemma 3 uses SigLIP (Gemma Team 2025). Appendix A further quantifies this diversity through the error-overlap ratios across model families. As shown in Figure 2(d), models from different families exhibit substantially less overlap in their errors. This complementary error structure is reflected by lower inter-model agreement during pseudo-labeling (Figure 2(a)) and, in turn, yields more accurate pseudo-labels (Figure 2(b)) and higher performance (Figure 2(c)) throughout training. Figure 2: (a) Agreement between the two models, (b) pseudo-label accuracy, and (c) evaluation performance; (d) Error overlap before RL for two pairs each from a different family, the same family, and the same model under a different seed. Model size provides an additional, orthogonal source of diversity. Models with different capacities exhibit distinct reasoning and prediction behaviors. Pairing models of different sizes therefore yields different error profiles, allowing each policy to provide informative reward signals on examples that the other does not solve correctly. The three-agent run in Section 6.2 trains models of different sizes together. Diversity across input formation. Despite model- and optimization-level decoupling, both agents are still trained on identical prompts. We further diversify their training data by rewriting each MATH problem with DeepSeek-V3 (DeepSeek-AI 2024), training one agent on the original prompt and the other on its rewrite. The rewrite preserves the answer and sample order while typically recasting the problem into a different concrete scenario rather than performing superficial lexical substitutions. The two agents therefore solve semantically equivalent problems expressed in different forms, reducing correlated errors induced by prompt-specific phrasing. Appendix C provides examples. We provide an illustration of diversified Co-RL in Figure 3. Figure 3: Overview of Co-RL with two agents. Each agent samples K responses to the same unlabeled question and generates a pseudo label with majority vote. Each rollout is then rewarded by agreement with the cohort’s pseudo label, and updates its own policy, with no sharing parameters and no gradient exchange except the cross-reward process. 5 Theoretical Analysis In this section, we theoretically characterize the learning dynamics of Co-RL and compare them with self-rewarding GRPO, where each agent uses its own majority vote as the pseudo-label. A key limitation of self-rewarding is that an agent learns from a pseudo-label derived from its own predictions. As a result, when the agent is systematically wrong, its supervision signal is likely to reinforce the same error rather than correct it. In contrast, Co-RL derives supervision from other agents, allowing one agent’s correct prediction to provide a corrective signal for another agent’s mistake. Specifically, we ignore clipping and the KL term for simplicity. Our analysis shows that Co-RL can exploit complementary strengths across agents to correct errors that self-rewarding would otherwise reinforce, thereby expanding the set of prompts that converge to the correct answer. To obtain a tractable characterization, we consider a fixed prompt x and reduce the induced answer distribution to two outcomes: the correct answer a⋆a and an aggregate incorrect answer. Let pn=Prπθn(a=a⋆∣x)p_n= _ _ _n(a=a x) denote the probability that agent n assigns to the correct answer; in the single-agent setting, we simply write p. We assume an odd number K of independently sampled rollouts, so that the majority vote has no ties, and let η denote the GRPO update rate. Our experiments use an even K, where a tie is resolved deterministically rather than discarded. 5.1 Comparison of training dynamics under self-rewarding and cross-agent supervision. We study how the probability of generating the correct answer p evolves during training. Proposition 1 (Training dynamics under self-rewarding and cross-agent supervision). Under the above definitions, the probability dynamics take the following forms. [1] Self-rewarding. Let C∼Bin(K,p)C (K,p) denote the number of correct responses among the K rollouts. The probability dynamics satisfy p˙=ηp(1−p)C∼Bin(K,p)[sign(C−K2)C(K−C)K]. p=η\,p(1-p)E_C (K,p) [sign\! (C- K2 ) C(K-C)K ]. (7) [2] Co-RL: Cross-agent supervision. For agent n, let Cn∼Bin(K,pn)C_n (K,p_n) denote the number of correct responses among its K rollouts, and let Z−n=[a^−n(x)=a⋆]Z_-n=1[ a_-n(x)=a ] indicate whether the pseudo-label constructed from the other agents is correct. The probability dynamics satisfy p˙n=ηnpn(1−pn)[sign(Z−n−12)]Cn∼Bin(K,pn)[Cn(K−Cn)K]. p_n= _np_n(1-p_n)E [sign\! (Z_-n- 12 ) ]E_C_n (K,p_n) [ C_n(K-C_n)K ]. (8) In the two-agent setting, define ϕK(p)=2VK(p)−1 _K(p)=2V_K(p)-1 as the signed majority-vote direction and qK(p)=ηp(1−p)C∼Bin(K,p)[C(K−C)K]>0q_K(p)=η p(1-p)E_C (K,p) [ C(K-C)K ]>0 as the positive update magnitude. Then Eq. (8) reduces to p˙A=qK(pA)ϕK(pB),p˙B=qK(pB)ϕK(pA). p_A=q_K(p_A) _K(p_B), p_B=q_K(p_B) _K(p_A). (9) Proof. A complete proof is provided in Appendix B.1. ∎ Proposition 1 highlights the structural difference between the two training mechanisms. Under self-rewarding, the pseudo-label is constructed from the same rollout group being optimized, making the update self-confirming: so the agent reinforces whichever answer is currently more likely, whether correct or not. In contrast, Co-RL decouples supervision from the optimized agent: since qK(pA)>0q_K(p_A)>0, the update direction of agent A is determined entirely by ϕK(pB) _K(p_B), the supervision signal provided by agent B. 5.2 Cross-agent supervision enlarges the basin of correct convergence. Following Proposition 1, Proposition 2 shows that self-rewarding is self-confirming: the expected GRPO update amplifies the currently favored answer, regardless of its correctness. Thus, when p<1/2p<1/2, self-rewarding further suppresses the correct answer instead of correcting the error. Proposition 2 (Self-confirming dynamics). For odd K, sign(GKself(p))=sign(p−12)sign(G_K^self(p))=sign(p- 12). Consequently, p(0)<12⇒p(t)→0p(0)< 12 p(t)→ 0 and p(0)>12⇒p(t)→1p(0)> 12 p(t)→ 1. Proof. A complete proof is provided in Appendix B.2. ∎ We next state our main result in Theorem 1, which characterizes the basin of attraction under cross-agent supervision. Theorem 1 (Co-RL enlarges the basin of correct convergence). Consider the symmetric two-agent dynamics in Eq. (9) with interior initialization (pA(0),pB(0))∈(0,1)2(p_A(0),p_B(0))∈(0,1)^2. The correct and incorrect consensus states (1,1)(1,1) and (0,0)(0,0) are asymptotically stable, while (1/2,1/2)(1/2,1/2) is a saddle point whose interior separatrix is pA+pB=1p_A+p_B=1. Consequently, pA(0)+pB(0)>1⟹(pA(t),pB(t))→(1,1),p_A(0)+p_B(0)>1 (p_A(t),p_B(t))→(1,1), whereas pA(0)+pB(0)<1⟹(pA(t),pB(t))→(0,0).p_A(0)+p_B(0)<1 (p_A(t),p_B(t))→(0,0). Proof. A complete proof is provided in Appendix B.3. ∎ Takeaway: Co-RL leverages complementary strengths to expand correct convergence. Consider an extreme example where (pA,pB)=(0.9,0.2)(p_A,p_B)=(0.9,0.2) on half of the prompts and (pA,pB)=(0.2,0.9)(p_A,p_B)=(0.2,0.9) on the other half. Each agent has average accuracy 0.550.55, but their strengths are perfectly complementary. Under self-rewarding, each agent succeeds only on the half of prompts where p>1/2p>1/2, yielding a final accuracy of 0.50.5. In contrast, under Co-RL, pA+pB=1.1>1p_A+p_B=1.1>1 on every prompt, so Theorem 1 predicts that both agents converge to the correct answer on all prompts, achieving accuracy 1.01.0. Thus, cross-agent supervision can exploit complementary expertise to correct errors that self-rewarding would otherwise reinforce. 6 Experiments 6.1 Experimental Setup Datasets. For the main experiments, language models are trained on the level 3 to 5 split of MATH (Hendrycks et al. 2021b), following MARTI (Zhang et al. 2026a). Vision-language models are trained on two multimodal math datasets. We use MMR1-Math (Leng et al. 2025) to match the training data of our baseline, M-UPT (Wei et al. 2025), and additionally use multimodal-open-r1 (LMMs-Lab 2025) to verify that the observed gains are not specific to a particular training dataset. Models. For language models, we consider Qwen3-1.7B (Yang et al. 2025a), Qwen2.5-3B paired with Llama-3.2-3B-Instruct, and Qwen2.5-7B paired with Llama-3.1-8B-Instruct (Yang et al. 2024; Grattafiori et al. 2024). For vision-language models, we use Qwen2.5-VL (3B, 7B), InternVL3.5 (2B, 8B), and Gemma-3 (4B, 12B) (Bai et al. 2025; Chen et al. 2024b; Gemma Team 2025), with models paired at comparable scales. Baselines. On language models, we compare our method against state-of-the-art label-free self-rewarding methods, including TTRL (Zuo et al. 2025), Intuitor (Zhao et al. 2026), RENT (Prabhudesai et al. 2025) and Co-rewarding-I (Zhang et al. 2026b). On vision-language models, we compare with the recent self-rewarding method M-UPT (Wei et al. 2025), which adopts the TTRL reward formulation. We also include GRPO with ground-truth rewards (GT-Reward) as a supervised reference (Shao et al. 2024). For multi-agent RL baselines, we compare against MAPoRL (Park et al. 2025) and CoMAS (Xue et al. 2026). Training details. All runs use the AdamW optimizer and sample rollouts at a temperature of 1.0. Following Co-rewarding (Zhang et al. 2026b), language models use a learning rate of 3×10−63× 10^-6, an effective batch of 128 prompts per agent and a 3072-token cap, and train with K=12K=12 responses per prompt for 2 epochs. Following R1-V (Chen et al. 2025b), vision-language models use a learning rate of 1×10−61× 10^-6, a 1024-token cap and K=8K=8 responses per prompt for 1 epoch. Because the rollout and policy distributions drift apart on Gemma-3, the vision-language runs additionally apply a token-level importance-sampling correction. All experiments use a single node of eight H100 GPUs, four per agent. Evaluation. Language models are scored on seven reasoning benchmarks. For mathematics, we use GSM8K (Cobbe et al. 2021), MATH-500 (Lightman et al. 2024) and AMC (Project-Numina 2024). For code generation, we use HumanEval (Chen et al. 2021), MBPP (Austin et al. 2021) and LiveCodeBench (Jain et al. 2025). For science, we use GPQA (Rein et al. 2024). Vision-language models are scored on four multimodal math benchmarks, MathVision (Wang et al. 2024a), MathVerse (Zhang et al. 2024a), MathVista (Lu et al. 2024) and We-Math (Qiao et al. 2025). Detailed experimental settings and the complete results are provided in Appendix D. 6.2 Main Results Co-RL outperforms all self-rewarding methods on language models. We first compare Co-RL against single-agent self-rewarding baselines. Table 1 reports results against TTRL, RENT, Intuitor, and Co-Rewarding-I, with GRPO using ground-truth rewards included as a supervised reference. Appendix D.1 extends the same comparison to 7B and 8B models. We evaluate three variants of our framework: Same family jointly trains two independent agents initialized from the same base model; Different family pairs one agent from each of the two model families; Different family+ further applies data decoupling. Notably, the readily accessible Same family setting already yields substantial gains over the base models, improving the average performance by 8.0% and 4.0% for Qwen2.5-3B and Llama-3.2-3B-Instruct, respectively. Introducing cross-family diversity provides further benefits overall, and, when combined with data decoupling (Different family+), achieves the strongest label-free average performance for both model families. Method GSM8K MATH500 AMC HEval GPQA MBPP LCB Avg Qwen2.5-3B Base 73.4 56.6 28.9 39.0 21.2 52.2 13.7 40.7 GT-Reward 76.2 64.6 36.1 65.2 20.7 54.4 14.5 47.4 TTRL 80.4 66.4 31.3 63.4 22.2 51.8 15.9 47.3 RENT 75.6 62.8 31.3 59.2 18.2 52.4 14.5 44.9 Intuitor 74.9 64.2 26.5 59.8 27.3 50.4 16.4 45.6 Co-rewarding-I 75.5 63.4 30.1 61.0 24.8 53.2 11.0 45.6 Co-RL (Same family) 78.5 66.0 37.4 65.8 22.2 56.0 15.2 48.7 Co-RL (Different family) 80.1 66.8 33.7 64.0 22.7 56.8 15.2 48.5 Co-RL (Different family+) 81.0 66.6 36.1 62.8 25.8 55.6 17.2 49.3 Llama-3.2-3B-Instruct Base 73.6 43.8 18.1 51.2 21.2 50.8 12.0 38.7 GT-Reward 78.8 53.8 25.3 60.4 20.7 50.2 12.1 43.0 TTRL 77.9 50.2 26.5 59.2 24.8 51.2 12.0 43.1 RENT 75.4 45.2 12.0 59.2 17.7 49.4 11.5 38.6 Intuitor 75.8 40.8 21.7 54.3 21.7 51.4 12.0 39.7 Co-rewarding-I 75.4 53.4 24.1 54.9 23.7 49.2 12.1 41.8 Co-RL (Same family) 78.4 52.4 26.5 57.9 21.7 49.6 12.4 42.7 Co-RL (Different family) 80.5 56.2 27.7 59.2 21.2 50.4 11.0 43.7 Co-RL (Different family+) 78.4 55.2 30.1 59.2 22.2 50.4 12.0 43.9 Table 1: Full performance across seven benchmarks for 3B models (%). For each benchmark, the best label-free result is shown in bold and the second best is underlined, with ties sharing the marking. Base and GT-Reward serve as references and are excluded from the ranking. Co-RL (Same family) trains two agents initialized from the same base model. Co-RL (Different family) pairs one agent from each of the two families. Co-RL (Different family+) further decouples the training data. Appendix D.1 extends the comparison to 7B and 8B models. Co-RL outperforms Multi-Agent RL. We next compare Co-RL with existing multi-agent RL methods. While these approaches also train multiple models jointly, existing state-of-the-art methods typically rely on an additional judging mechanism to construct rewards, such as an LLM judge in CoMAS or a learned reward model in MAPoRL. Following CoMAS, we adopt the same experimental setup, official implementation, and evaluation benchmarks to ensure a fair comparison. We report the CoMAS results as presented in the original paper. As shown in Table 2, Co-RL achieves the best average performance and leads on five of the seven benchmarks, outperforming CoMAS by 4.0% while using only half as many agents and requiring no additional judging mechanism. Scaling Co-RL to Three Agents. We further extend Co-RL beyond the two-agent setting by jointly training Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B in a single run. For each agent, we compare Co-RL against the same model trained independently using ground-truth rewards (GT-Reward) or self-generated majority-vote rewards (TTRL). Results are reported in Table 3. Co-RL consistently improves all three base models, with average gains of 7.8%, 6.0%, and 8.2%, respectively. Despite using no ground-truth supervision, three-agent Co-RL matches or outperforms GT-Reward in average performance for all three models, while also outperforming TTRL for Qwen2.5-3B and Llama-3.2-3B-Instruct. These results suggest that Co-RL naturally extends beyond pairwise training, allowing multiple heterogeneous agents to benefit from cross-agent supervision within a shared training run. Method GSM8K MATH-500 HumanEval MBPP MMLU GPQA SciBench Avg Base 85.40 55.00 73.78 55.80 63.20 28.79 36.47 56.92 MAPoRL 85.80 55.40 75.61 57.00 63.20 31.47 39.08 58.22 TTRL 88.20 56.80 73.78 59.00 63.80 27.23 38.48 58.18 CoMAS 87.20 55.80 77.44 59.20 65.60 29.69 37.68 58.94 Co-RL (Different family) 89.5 68.6 82.32 68.00 65.80 29.69 36.87 62.97 Table 2: Comparison under the CoMAS multi-agent RL setting (%). All methods train Qwen2.5-3B-Instruct on the same prompt mixture and are evaluated following the CoMAS protocol. Results for prior methods are reported from Xue et al. 2026. Co-RL Transfers to Vision-Language Models. Vision-language model families differ in both their visual encoders and language backbones, providing a stronger test of Co-RL under heterogeneous architectures. We therefore extend Co-RL to multimodal mathematical reasoning. Table 4 reports results for Qwen2.5-VL-3B and InternVL3.5-2B, while Appendix D.2 extends the evaluation to three model families from 7B to 12B. Co-RL achieves the best average performance in three of the four 2B-3B settings and remains competitive with ground-truth supervision. The gains persist at larger scales, where Co-RL consistently outperforms TTRL and even surpasses GT-Reward for Gemma-3-12B, demonstrating that Co-RL generalizes beyond text-only models. Model Method GSM8K MATH500 AMC HEval GPQA MBPP LCB Avg Qwen2.5-3B Base 73.4 56.6 28.9 39.0 21.2 52.2 13.7 40.7 GT-Reward 76.2 64.6 36.1 65.2 20.7 54.4 14.5 47.4 TTRL 80.4 66.4 31.3 63.4 22.2 51.8 15.9 47.3 Co-RL (Different family) 79.8 66.3 33.6 64.6 23.2 56.0 15.8 48.5 Llama-3.2-3B-Instruct Base 73.6 43.8 18.1 51.2 21.2 50.8 12.0 38.7 GT-Reward 78.8 53.8 25.3 60.4 20.7 50.2 12.1 43.0 TTRL 77.9 50.2 26.5 59.2 24.8 51.2 12.0 43.1 Co-RL (Different family) 77.8 54.2 28.8 64.4 25.1 50.9 11.7 44.7 Qwen3-1.7B Base 67.0 60.9 27.5 40.0 15.3 50.6 12.4 39.1 GT-Reward 67.1 67.0 34.3 70.1 25.2 51.2 15.2 47.2 TTRL 70.3 67.6 32.1 69.5 24.8 52.0 15.1 47.3 Co-RL (Different family) 69.3 67.6 32.7 64.2 27.1 54.6 15.3 47.3 Table 3: Three-agent Co-RL with heterogeneous model families (%). Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B are jointly trained in a single Co-RL run. For each model, we compare against the base model, training with ground-truth rewards (GT-Reward), and self-rewarding with majority-vote pseudo-labels (TTRL). Backbone Data Method MathVision MathVerse MathVista We-Math Avg InternVL-3.5-2B open-r1 GT-Reward 26.55 35.33 59.60 59.31 45.20 Base 24.77 34.21 55.60 57.87 43.11 TTRL 25.86 34.24 57.60 62.47 45.04 Co-RL (Different family) 26.25 34.92 58.90 61.55 45.40 MMR1 GT-Reward 25.99 34.37 59.00 59.25 44.65 Base 24.77 34.21 55.60 57.87 43.11 TTRL 26.38 35.36 57.70 61.78 45.30 Co-RL (Different family) 26.05 34.80 58.60 61.15 45.15 Qwen2.5-VL-3B open-r1 GT-Reward 21.71 31.29 60.90 57.99 42.97 Base 18.55 26.04 52.70 51.67 37.24 TTRL 21.15 30.05 57.40 61.55 42.54 Co-RL (Different family) 21.94 30.48 60.20 62.93 43.89 MMR1 GT-Reward 19.57 27.34 59.40 57.82 41.03 Base 18.55 26.04 52.70 51.67 37.24 TTRL 17.99 24.72 56.30 52.87 37.97 Co-RL (Different family) 21.05 28.91 57.20 57.30 41.12 Table 4: Vision-language results for the small pair, Qwen2.5-VL-3B with InternVL3.5-2B, trained separately on open-r1 and MMR1 (%). Base is graded once with the corrected multiple-choice grader and is therefore identical across the two training sets. Base and GT-Reward serve as references and are excluded from the ranking. Figure 4: Training dynamics at four scales, one column per backbone (Qwen2.5-3B, Llama-3.2-3B, Qwen2.5-7B, Llama-3.1-8B). (a) MATH-500 validation accuracy, (b) standard deviation of the reward within a rollout group, normalized to its value at the first step, and (c) mean completion length. Runs marked diverged leave the plotted range. 7 Ablation Study Training dynamics and stability. We further examine the training dynamics of Co-RL in Appendix D.3. As shown in Figure 4, across text models, Co-RL maintains stable reward variation and completion lengths, whereas self-rewarding baselines can exhibit reward collapse, length degeneration, or divergence. We observe a similar pattern for VLMs: the agents retain partial agreement while the accuracy of their exchanged pseudo-labels improves throughout training. These results are consistent with our theory: by decoupling an agent’s update from its own predictions, cross-agent supervision avoids self-reinforcing errors and preserves an informative learning signal throughout training. Controlling for training and inference budgets. To match the two-agent training budget of Co-RL, we construct a self-rewarding baseline with the same two base models. Each model is trained independently with TTRL. At inference, both Co-RL and TTRL ensemble the two models by pooling four rollouts from each for majority voting, thereby matching both training and test-time budgets. As shown in Appendix D.4, Co-RL consistently achieves the best average score across text and multimodal settings, showing that the gains come from cross-agent supervision rather than additional compute or ensembling alone. 8 Conclusion In this work, we introduced Co-RL, a label-free multi-agent RL framework for reasoning tasks. In our framework, multiple agents learn from rewards constructed from their peers’ predictions rather than ground-truth labels or external judges. Across text-only and multimodal reasoning benchmarks, Co-RL consistently improves diverse LLMs and VLMs, outperforming prior self-rewarding and multi-agent RL approaches and, in many settings, matching or surpassing training with ground-truth rewards. Our theoretical analysis shows that cross-agent supervision expands the set of initial conditions that converge to the correct solution, allowing Co-RL to correct errors that self-rewarding RL would otherwise reinforce. An important direction for future work is to understand how the number, diversity, and interaction topology of agents shape cross-agent learning, and to develop adaptive supervision mechanisms that more effectively exploit complementary expertise. References Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Bai et al. [2025] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. Blum and Mitchell [1998] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Conference on Computational Learning Theory (COLT), pages 92–100, 1998. Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630–9640, 2021. Chen et al. [2025a] Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. SFT or RL? an early investigation into training R1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468, 2025a. Chen et al. [2024a] Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. ReConcile: Round-table conference improves reasoning via consensus among diverse LLMs. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7066–7085, 2024a. Chen et al. [2025b] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-V: Reinforcing super generalization ability in vision-language models with less than $3. https://github.com/Deep-Agent/R1-V, 2025b. Accessed: 2026-08-17. Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597–1607, 2020. Chen et al. [2025c] Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Optima: Optimizing effectiveness and efficiency for LLM-based multi-agent system. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11534–11557, 2025c. Chen et al. [2025d] Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, and Jiaxuan You. Multi-agent evolve: LLM self-improve through co-evolution. arXiv preprint arXiv:2510.23595, 2025d. Chen et al. [2024b] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24185–24198, 2024b. Chen et al. [2024c] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In International Conference on Machine Learning (ICML), 2024c. Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Cohen [1960] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. DeepSeek-AI [2024] DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. DeepSeek-AI [2025] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Deng et al. [2025a] Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint arXiv:2503.07065, 2025a. Deng et al. [2025b] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. OpenVLThinker: Complex vision-language reasoning via iterative SFT-RL cycles. In Advances in Neural Information Processing Systems (NeurIPS), 2025b. Du et al. [2024] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning (ICML), 2024. Fang et al. [2025] Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. SeRL: Self-play reinforcement learning for large language models with limited data. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Feng et al. [2025] Yicheng Feng, Yijiang Li, Wanpeng Zhang, Sipeng Zheng, Hao Luo, Zihao Yue, and Zongqing Lu. VideoOrion: Tokenizing object dynamics in videos. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 20401–20412, 2025. Gemma Team [2025] Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Grattafiori et al. [2024] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 21271–21284, 2020. Hendrycks et al. [2021a] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021a. Hendrycks et al. [2021b] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021b. Hu et al. [2025] Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: Stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262, 2025. Huang et al. [2026a] Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-evolving reasoning LLM from zero data. In International Conference on Learning Representations (ICLR), 2026a. Huang et al. [2026b] Wenxuan Huang, Bohan Jia, Shaosheng Cao, Zheyu Ye, Zhe Xu, Yao Hu, Shaohui Lin, et al. Vision-R1: Incentivizing reasoning capability in multimodal large language models. In International Conference on Learning Representations (ICLR), 2026b. Jain et al. [2025] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations (ICLR), 2025. Kim et al. [2024] Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S. Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae W. Park. MDAgents: An adaptive collaboration of LLMs for medical decision-making. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Krogh and Vedelsby [1994] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 7, 1994. Kuncheva and Whitaker [2003] Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181–207, 2003. Leng et al. [2025] Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Yuming Jiang, Hang Zhang, Xin Li, Lidong Bing, et al. MMR1: Enhancing multimodal reasoning with variance-aware sampling and open resources. arXiv preprint arXiv:2509.21268, 2025. Li et al. [2025] Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot RL fine-tuning of language models. arXiv preprint arXiv:2506.06395, 2025. Li et al. [2023] Yijiang Li, Xinjiang Wang, Lihe Yang, Litong Feng, Wayne Zhang, and Ying Gao. Diverse cotraining makes strong semi-supervised segmentor. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. Liang et al. [2024] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. Liao et al. [2025] Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. MARFT: Multi-agent reinforcement fine-tuning. arXiv preprint arXiv:2504.16129, 2025. Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. Liu et al. [2026] Bo Liu, Simon Yu, Zichen Liu, Leon Guertler, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. SPIRAL: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. In International Conference on Learning Representations (ICLR), 2026. Liu et al. [2025a] Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Shieh. NoisyRollout: Reinforcing visual reasoning with data augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Liu et al. [2025b] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 2034–2044, 2025b. LMMs-Lab [2025] LMMs-Lab. Multimodal open R1. https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified, 2025. Accessed: 2026-08-17. Lu et al. [2024] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024. Ma et al. [2025] Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-Reasoner: Advancing LLM reasoning across all domains. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Meng et al. [2025] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. M-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025. Motwani et al. [2024] Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. MALT: Improving reasoning with multi-agent LLM training. arXiv preprint arXiv:2412.01928, 2024. Park et al. [2025] Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman E. Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. MAPoRL: Multi-agent post-co-training for collaborative large language models with reinforcement learning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025. Peng et al. [2025] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. LMM-R1: Empowering 3B LMMs with strong reasoning abilities through two-stage rule-based RL. arXiv preprint arXiv:2503.07536, 2025. Prabhudesai et al. [2025] Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660, 2025. Project-Numina [2024] Project-Numina. AIMO validation AMC. https://huggingface.co/datasets/AI-MO/aimo-validation-amc, 2024. Accessed: 2026-08-17. Qiao et al. [2025] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma Gongque, Shanglin Lei, Yifan Zhang, et al. We-Math: Does your large multimodal model achieve human-like mathematical reasoning? In Annual Meeting of the Association for Computational Linguistics (ACL), 2025. Rein et al. [2024] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. In Conference on Language Modeling (COLM), 2024. Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Yang Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Shen et al. [2025] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. VLM-R1: A stable and generalizable R1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025. Sun et al. [2024] Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. In Conference on Language Modeling (COLM), 2024. Wang et al. [2025a] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VL-Rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Wang et al. [2024a] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-Vision dataset. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024a. Wang et al. [2025b] Peiyu Wang, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, et al. Skywork R1V2: Multimodal hybrid reinforcement learning for reasoning. arXiv preprint arXiv:2504.16656, 2025b. Wang et al. [2024b] Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating college-level scientific problem-solving abilities of large language models. In International Conference on Machine Learning (ICML), 2024b. Wang et al. [2025c] Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. SoTA with less: MCTS-guided sample selection for data-efficient visual reasoning self-improvement. In Advances in Neural Information Processing Systems (NeurIPS), 2025c. Wang et al. [2023] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023. Wei et al. [2025] Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, and Lichao Sun. Unsupervised post-training for multi-modal LLM reasoning via GRPO. arXiv preprint arXiv:2505.22453, 2025. Wu et al. [2023] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. Xiong et al. [2025] Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.19613, 2025. Xu et al. [2025a] Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, and Zhiyong Wu. Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 13153–13167, 2025a. Xu et al. [2025b] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. KodCode: A diverse, challenging, and verifiable synthetic dataset for coding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6980–7008, 2025b. Xue et al. [2026] Xiangyuan Xue, Yifan Zhou, Guibin Zhang, Zaibin Zhang, Yijiang Li, Chen Zhang, Zhenfei Yin, Philip Torr, Wanli Ouyang, and Lei Bai. CoMAS: Co-evolving multi-agent systems via interaction rewards. In International Conference on Learning Representations (ICLR), 2026. Yang et al. [2024] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. Yang et al. [2025a] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Yang et al. [2025b] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-OneVision: Advancing generalized multimodal reasoning through cross-modal formalization. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 2376–2385, 2025b. Yu et al. [2025a] En Yu, Kangheng Lin, Liang Zhao, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, et al. Perception-R1: Pioneering perception policy with reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Yu et al. [2025b] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2025b. Yuan et al. [2024] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024. Yue et al. [2025] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In Advances in Neural Information Processing Systems (NeurIPS), 2025. Zhang et al. [2026a] Kaiyan Zhang, Kai Tian, Runze Liu, Sihang Zeng, Xuekai Zhu, Guoli Jia, Yuchen Fan, Xingtai Lv, Yuxin Zuo, Che Jiang, et al. MARTI: A framework for multi-agent LLM systems reinforced training and inference. In International Conference on Learning Representations (ICLR), 2026a. Zhang et al. [2025a] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised LLM reasoning incentivization. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Zhang et al. [2024a] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems? In European Conference on Computer Vision (ECCV), pages 169–186, 2024a. Zhang et al. [2025b] Wanpeng Zhang, Yicheng Feng, Hao Luo, Yijiang Li, Zihao Yue, Sipeng Zheng, and Zongqing Lu. Unified multimodal understanding via byte-pair visual encoding. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 12976–12986, 2025b. Zhang et al. [2025c] Wanpeng Zhang, Zilong Xie, Yicheng Feng, Yijiang Li, Xingrun Xing, Sipeng Zheng, and Zongqing Lu. From pixels to tokens: Byte-pair encoding on quantized visual modalities. In International Conference on Learning Representations (ICLR), 2025c. Zhang et al. [2018] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4320–4328, 2018. Zhang et al. [2024b] Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö. Arık. Chain of agents: Large language models collaborating on long-context tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2024b. Zhang et al. [2026b] Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-rewarding: Stable self-supervised RL for eliciting reasoning in large language models. In International Conference on Learning Representations (ICLR), 2026b. Zhao et al. [2025a] Andrew Zhao, Yiran Wu, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Zhao et al. [2025b] Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, and James Zou. SiriuS: Self-improving multi-agent systems via bootstrapped reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2025b. Zhao et al. [2026] Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. In International Conference on Learning Representations (ICLR), 2026. Zuo et al. [2025] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Xinwei Long, Ermo Hua, Biqing Qi, et al. TTRL: Test-time reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Appendix A Pre-training Error Decoupling Section 4.2 argues that a peer helps when its errors do not overlap with the agent’s own and Figure 2(d) previews this overlap for three kinds of model pairs. This appendix reports the full measurement. All numbers are computed on base checkpoints before any RL, so they describe the pretrained models themselves rather than anything our training produces. Setup and metrics. We score base checkpoints on 500 MATH problems at levels 3 to 5, zero-shot and single-sample at T=0.8T=0.8, with rule-based extraction and equivalence checking. For a pair of models, every problem lands in one of the four cells of Table 5, and the four diversity measures [Kuncheva and Whitaker 2003] are counts over these cells. Complementarity c is the share of problems where exactly one model is correct, so one model can correct the other. Oracle accuracy u is the share of problems outside the both-wrong cell, which is the accuracy a perfect selector would reach [Krogh and Vedelsby 1994]. Wrong-agreement w is the part of the both-wrong cell where the two models also return the same answer, which is the case a majority vote cannot detect. Cohen’s κ [Cohen 1960] measures how strongly the two models land in the same cells beyond chance, so lower values mean more decoupled errors. All four numbers come from the same four cells. κ and c are therefore two readings of one measurement rather than independent evidence, and across the twelve pairs below they correlate at r=−0.98r=-0.98. B correct B wrong A correct both correct only A correct A wrong only B correct both wrong Table 5: The four outcomes for a pair of models A and B. Every problem falls into exactly one cell, and all four diversity measures are counts over these cells. Three levels of decoupling. Table 6 groups pairs by what the two models differ in. Seed only pairs a checkpoint with itself under a different sampling seed, so the two views share every weight and differ in generation noise alone. Same family pairs models of one lineage across sizes or generations. Different family pairs models with separate architectures and pretraining data. The first level is what Co-RL has by construction, and the third is what a different-family cohort adds. The three levels separate without overlap. Every different-family pair reaches κ≤0.42κ≤ 0.42 and c≥29.4c≥ 29.4, and every same-family and seed-only pair sits at κ≥0.51κ≥ 0.51 and c≤24.6c≤ 24.6. No pair falls between the two groups, at either scale. Averaged within a group, crossing families lowers κ from 0.530.53 to 0.380.38 and raises c from 23.223.2 to 30.830.8. The same-family and seed-only groups are not distinguishable from one another, so changing the size, the generation or the sampling seed within a lineage leaves the error structure where it was. Holding the anchor fixed gives the same ordering. Group averages mix models of different strength, so we also fix one model and vary only its partner (Table 7). Both anchors give a monotone ladder. Relative to pairing a model with itself, a same-family partner lowers κ by 0.040.04 and 0.070.07, and a different-family partner lowers it by 0.180.18 and 0.160.16 while raising c by 9.2% and 10.4%. Because the seed-only row is the anchor paired with itself, capability is identical along each ladder and the source of the partner is the only variable. Wrong-agreement follows the same direction, and is lowest for the different-family partner under both anchors, though it does not order the middle rows. What this means for Co-RL. Two models trained by different groups on different data fail on different problems, and no amount of resampling or rescaling within one lineage reproduces that. A same-family cohort still gives each agent a target it did not produce itself, which is enough for stable training, but the target repeats what the agent would have answered anyway on three quarters of the problems. A different-family cohort is the cheapest way to buy the remaining headroom. Decoupling Pair κ↓κ\ c↑c\ (%) w↓w\ (%) u↑u\ (%) 3B tier different family Llama-3.2-3B × Phi-3.5-mini 0.31 32.8 3.0 53.0 different family Qwen2.5-3B × Llama-3.2-3B 0.38 31.2 2.4 63.0 different family Qwen2.5-3B × Phi-3.5-mini 0.38 31.2 4.0 55.4 different family Qwen2.5-3B × MiniCPM3-4B 0.41 29.4 4.4 60.4 same family Qwen2.5-3B × Qwen3-1.7B-Base 0.52 24.2 4.2 63.2 seed only Qwen3-1.7B-Base × itself 0.52 24.0 5.0 66.4 seed only Qwen2.5-3B × itself 0.56 22.0 4.4 62.6 7B tier different family Qwen2.5-7B × Llama-3.1-8B 0.42 29.4 1.8 71.4 same family Qwen2.5-7B × Qwen2.5-3B 0.51 24.2 3.8 69.8 same family Qwen2.5-7B × Qwen3-1.7B-Base 0.51 24.4 4.0 70.4 seed only Llama-3.1-8B × itself 0.51 24.6 3.0 62.0 seed only Qwen2.5-7B × itself 0.58 19.0 5.2 74.6 Table 6: Error decoupling before RL, by what the two models differ in, sorted by κ within each block. Partner Decoupling κ↓κ\ c↑c\ (%) w↓w\ (%) Anchor: Qwen2.5-3B itself, new seed seed only 0.56 22.0 4.4 Qwen3-1.7B-Base same family 0.52 24.2 4.2 MiniCPM3-4B different family 0.41 29.4 4.4 Phi-3.5-mini different family 0.38 31.2 4.0 Llama-3.2-3B different family 0.38 31.2 2.4 Anchor: Qwen2.5-7B itself, new seed seed only 0.58 19.0 5.2 Qwen2.5-3B same family 0.51 24.2 3.8 Qwen3-1.7B-Base same family 0.51 24.4 4.0 Llama-3.1-8B different family 0.42 29.4 1.8 Table 7: One model held fixed, partner varied. Capability is identical to the seed-only row along each ladder, so the source of the partner is the only variable. Appendix B Complete Proof B.1 Proof of Proposition 1 In this section, we derive the reward-induced GRPO dynamics in Proposition 1. We first derive a common update expression under a fixed pseudo-label and then specialize it to self-rewarding and cross-agent supervision. For a fixed prompt x, let Xk=[ak=a⋆]X^k=1[a^k=a ] indicate whether the k-th rollout is correct. Under the binary reduction, Xk∼i.i.d.Bernoulli(p)X^k i.i.d. Bernoulli(p), and C=∑k=1KXk∼Bin(K,p)C= _k=1^KX^k (K,p) denotes the number of correct responses among the K rollouts. To analyze the induced binary dynamics, we use the log-odds ℓ=logp1−p = p1-p as a one-dimensional coordinate. Equivalently, p=σ(ℓ)=1/(1+e−ℓ)p=σ( )=1/(1+e^- ). We analyze the reward-induced component of GRPO in the infinitesimal-update limit, where clipping is locally inactive. GRPO update under a fixed pseudo-label. Let Z∈0,1Z∈\0,1\ indicate whether a fixed pseudo-label is correct, where Z=1Z=1 corresponds to a⋆a and Z=0Z=0 to the aggregate incorrect answer. The binary reward for rollout k is rk=[Xk=Z]=(1−Z)+(2Z−1)Xk.r^k=1[X^k=Z]=(1-Z)+(2Z-1)X^k. Since C=∑k=1KXkC= _k=1^KX^k, the group-mean reward is r¯=(1−Z)+(2Z−1)CK, r=(1-Z)+(2Z-1) CK, and therefore rk−r¯=(2Z−1)(Xk−CK).r^k- r=(2Z-1) (X^k- CK ). (10) The within-group standard deviation of the binary rewards is sK(C)≜1K∑j=1K(rj−r¯)2=C(K−C)K.s_K(C) 1K _j=1^K(r^j- r)^2= C(K-C)K. For 0<C<K0<C<K, the normalized group-relative advantage is therefore A^k=rk−r¯sK(C)=2Z−1sK(C)(Xk−CK). A^k= r^k- rs_K(C)= 2Z-1s_K(C) (X^k- CK ). For C∈0,KC∈\0,K\, all rewards are identical and the centered update is zero. Treating the advantages as fixed during the policy update, the reward-induced GRPO gradient in the log-odds coordinate is gℓ(C,Z) g_ (C,Z) ≜∂ℓ[1K∑k=1KA^klogPrℓ(Xk)] ∂ [ 1K _k=1^K A^k _ (X^k) ] =1K∑k=1KA^k∂ℓ[Xklogp+(1−Xk)log(1−p)] = 1K _k=1^K A^k ∂ [X^k p+(1-X^k) (1-p) ] =1K∑k=1KA^k(Xk−p) = 1K _k=1^K A^k(X^k-p) =2Z−1KsK(C)∑k=1K(Xk−CK)(Xk−p). = 2Z-1Ks_K(C) _k=1^K (X^k- CK )(X^k-p). Since ∑k=1K(Xk−C/K)=0 _k=1^K(X^k-C/K)=0, the terms involving p cancel. Using (Xk)2=Xk(X^k)^2=X^k and ∑k=1KXk=C _k=1^KX^k=C gives ∑k=1K(Xk−CK)Xk=C−C2K=C(K−C)K. _k=1^K (X^k- CK )X^k=C- C^2K= C(K-C)K. Hence, gℓ(C,Z)=(2Z−1)C(K−C)K.g_ (C,Z)=(2Z-1) C(K-C)K. (11) [1] Self-rewarding. Under majority-vote self-rewarding, the pseudo-label is determined by the same rollout group: Zself=[C>K2].Z_self=1\! [C> K2 ]. Since K is odd, 2Zself−1=sign(C−K2).2Z_self-1=sign\! (C- K2 ). Substituting into Eq. (11) gives gℓ(C,Zself)=sign(C−K2)C(K−C)K.g_ (C,Z_self)=sign\! (C- K2 ) C(K-C)K. For a GRPO update with learning rate η, ℓ+−ℓ=ηgℓ(C,Zself). ^+- =η\,g_ (C,Z_self). Since C∼Bin(K,p)C (K,p) is random, the conditional expected one-step update is [ℓ+−ℓ∣p]=ηC∼Bin(K,p)[sign(C−K2)C(K−C)K].E[ ^+- p]=η\,E_C (K,p) [sign\! (C- K2 ) C(K-C)K ]. In the infinitesimal-update limit, the corresponding mean log-odds dynamics are ℓ˙=ηC∼Bin(K,p)[sign(C−K2)C(K−C)K]. =η\,E_C (K,p) [sign\! (C- K2 ) C(K-C)K ]. Since p=σ(ℓ)p=σ( ), p˙ p =∂p∂ℓℓ˙ = ∂ p∂ =ηp(1−p)C∼Bin(K,p)[sign(C−K2)C(K−C)K], =η\,p(1-p)E_C (K,p) [sign\! (C- K2 ) C(K-C)K ], which recovers Eq. (7). [2] Co-RL: Cross-agent supervision. For agent n, let Xnk=[ank=a⋆]X_n^k=1[a_n^k=a ] and Cn=∑k=1KXnk∼Bin(K,pn).C_n= _k=1^KX_n^k (K,p_n). Let Z−n=[a^−n(x)=a⋆]Z_-n=1[ a_-n(x)=a ] indicate whether the pseudo-label constructed from the other agents is correct. Conditional on the prompt and current policies, Z−nZ_-n is independent of agent n’s rollout group, and hence Z−n⟂Cn.Z_-n C_n. Applying Eq. (11) to agent n gives gℓn(Cn,Z−n)=sign(Z−n−12)Cn(K−Cn)K,g_ _n(C_n,Z_-n)=sign\! (Z_-n- 12 ) C_n(K-C_n)K, where ℓn=logpn1−pn _n= p_n1-p_n. Therefore, [ℓn+−ℓn∣pn,p−n] [ _n^+- _n p_n,p_-n] =ηn[sign(Z−n−12)Cn(K−Cn)K] = _n\,E [sign\! (Z_-n- 12 ) C_n(K-C_n)K ] =ηn[sign(Z−n−12)]Cn∼Bin(K,pn)[Cn(K−Cn)K], = _n\,E [sign\! (Z_-n- 12 ) ]E_C_n (K,p_n) [ C_n(K-C_n)K ], where the second equality follows from Z−n⟂CnZ_-n C_n. Taking the infinitesimal-update limit and using ∂pn/∂ℓn=pn(1−pn)∂ p_n/∂ _n=p_n(1-p_n) yields p˙n p_n =∂pn∂ℓnℓ˙n = ∂ p_n∂ _n _n =ηnpn(1−pn)[sign(Z−n−12)]Cn∼Bin(K,pn)[Cn(K−Cn)K], = _np_n(1-p_n)E [sign\! (Z_-n- 12 ) ]E_C_n (K,p_n) [ C_n(K-C_n)K ], which recovers Eq. (8). Symmetric two-agent dynamics. We finally specialize the result to two symmetric agents A and B. For agent A, its pseudo-label is the majority vote of agent B’s rollouts. Hence, if CB∼Bin(K,pB)C_B (K,p_B), Z−A=[CB>K2],Z_-A=1\! [C_B> K2 ], and therefore [sign(Z−A−12)] [sign\! (Z_-A- 12 ) ] =CB∼Bin(K,pB)[sign(CB−K2)] =E_C_B (K,p_B) [sign\! (C_B- K2 ) ] =2VK(pB)−1=ϕK(pB). =2V_K(p_B)-1= _K(p_B). Similarly, [sign(Z−B−12)]=ϕK(pA).E [sign\! (Z_-B- 12 ) ]= _K(p_A). Assuming the same learning rate η for both agents and defining qK(p)=ηp(1−p)C∼Bin(K,p)[C(K−C)K]>0,q_K(p)=η p(1-p)E_C (K,p) [ C(K-C)K ]>0, we obtain p˙A=qK(pA)ϕK(pB),p˙B=qK(pB)ϕK(pA), p_A=q_K(p_A) _K(p_B), p_B=q_K(p_B) _K(p_A), which recovers Eq. (9). B.2 Proof of Proposition 2 We prove that majority-vote self-rewarding reinforces the currently favored answer and therefore induces self-confirming dynamics. For convenience, denote GKself(p)=C∼Bin(K,p)[sign(C−K2)C(K−C)K].G_K^self(p)=E_C (K,p) [sign\! (C- K2 ) C(K-C)K ]. Expanding the expectation gives GKself(p)=∑c=0Ksign(c−K2)c(K−c)KPrp(C=c).G_K^self(p)= _c=0^Ksign\! (c- K2 ) c(K-c)K _p(C=c). Since the update magnitude satisfies c(K−c)K=(K−c)cK, c(K-c)K= (K-c)cK, each event C=c>K/2C=c>K/2 can be paired with its symmetric event C=K−c<K/2C=K-c<K/2. Moreover, the update magnitude vanishes at c=0c=0 and c=Kc=K. Hence, GKself(p)=∑c=(K+1)/2K−1c(K−c)K[Prp(C=c)−Prp(C=K−c)].G_K^self(p)= _c=(K+1)/2^K-1 c(K-c)K [ _p(C=c)- _p(C=K-c) ]. (12) For p∈(0,1)p∈(0,1) and c>K/2c>K/2, Prp(C=c)Prp(C=K−c) _p(C=c) _p(C=K-c) =(Kc)pc(1−p)K−c(K−c)pK−c(1−p)c = Kcp^c(1-p)^K-c KK-cp^K-c(1-p)^c =(p1−p)2c−K, = ( p1-p )^2c-K, where we use (Kc)=(K−c) Kc= KK-c. Since 2c−K>02c-K>0, Prp(C=c)−Prp(C=K−c)>0,p>12,=0,p=12,<0,p<12. _p(C=c)- _p(C=K-c) cases>0,&p> 12,\\ =0,&p= 12,\\ <0,&p< 12. cases The factor c(K−c)/K c(K-c)/K in Eq. (12) is strictly positive for 0<c<K0<c<K. Therefore, every term in the sum has the same sign, yielding sign(GKself(p))=sign(p−12).sign\! (G_K^self(p) )=sign\! (p- 12 ). (13) We next characterize the limiting behavior. From Eq. (7), p˙=ηp(1−p)GKself(p). p=η p(1-p)G_K^self(p). If 0<p(0)<1/20<p(0)<1/2, Eq. (13) implies p˙<0 p<0 whenever p∈(0,1/2)p∈(0,1/2). Thus, p(t)p(t) is monotonically decreasing and bounded below by zero, and therefore converges to some p∞∈[0,1/2)p_∞∈[0,1/2). Suppose p∞>0p_∞>0. By Eq. (13), p∞(1−p∞)GKself(p∞)<0.p_∞(1-p_∞)G_K^self(p_∞)<0. Since the right-hand side of Eq. (7) is continuous, p˙ p remains strictly negative in a neighborhood of p∞p_∞, which contradicts convergence to an interior limit. Hence, p(0)<12⟹p(t)→0.p(0)< 12 p(t)→ 0. Similarly, if 1/2<p(0)<11/2<p(0)<1, then p˙>0 p>0. Thus, p(t)p(t) is monotonically increasing and bounded above by one. The same argument rules out any interior limit, giving p(0)>12⟹p(t)→1.p(0)> 12 p(t)→ 1. Finally, at p=1/2p=1/2, symmetry gives GKself(1/2)=0G_K^self(1/2)=0, so p=1/2p=1/2 is an unstable equilibrium. This completes the proof. B.3 Proof of Theorem 1 We prove Theorem 1 for the symmetric two-agent dynamics from Eq. (9), p˙A=qK(pA)ϕK(pB),p˙B=qK(pB)ϕK(pA), p_A=q_K(p_A) _K(p_B), p_B=q_K(p_B) _K(p_A), (14) where qK(p)=ηp(1−p)C∼Bin(K,p)[C(K−C)K]>0q_K(p)=η p(1-p)E_C (K,p) [ C(K-C)K ]>0 for p∈(0,1)p∈(0,1), and ϕK(p)=2VK(p)−1 _K(p)=2V_K(p)-1. Proof. Symmetry of the dynamics. We first establish two useful symmetries. Since the distribution of K−CK-C under C∼Bin(K,p)C (K,p) is Bin(K,1−p)Bin(K,1-p) and C(K−C) C(K-C) is invariant under C↦K−C K-C, we have qK(1−p)=qK(p).q_K(1-p)=q_K(p). Moreover, for odd K, complementing every rollout reverses the majority outcome, so VK(1−p)=1−VK(p).V_K(1-p)=1-V_K(p). Therefore, qK(1−p)=qK(p),ϕK(1−p)=−ϕK(p).q_K(1-p)=q_K(p), _K(1-p)=- _K(p). (15) Since VK(p)V_K(p) is strictly increasing and VK(1/2)=1/2V_K(1/2)=1/2, sign(ϕK(p))=sign(p−12).sign( _K(p))=sign\! (p- 12 ). A conserved quantity. Define FK(p)≜∫1/2pϕK(u)qK(u)u.F_K(p) _1/2^p _K(u)q_K(u)\,du. (16) Along any interior trajectory of Eq. (14), dt[FK(pA)−FK(pB)] ddt [F_K(p_A)-F_K(p_B) ] =ϕK(pA)qK(pA)p˙A−ϕK(pB)qK(pB)p˙B = _K(p_A)q_K(p_A) p_A- _K(p_B)q_K(p_B) p_B =ϕK(pA)ϕK(pB)−ϕK(pB)ϕK(pA) = _K(p_A) _K(p_B)- _K(p_B) _K(p_A) =0. =0. Hence, FK(pA)−FK(pB)=constantF_K(p_A)-F_K(p_B)=constant (17) along every trajectory. From Eq. (15), FK(1−p)=FK(p).F_K(1-p)=F_K(p). Furthermore, FK′(p)=ϕK(p)/qK(p)F_K (p)= _K(p)/q_K(p), so FKF_K is strictly decreasing on (0,1/2)(0,1/2) and strictly increasing on (1/2,1)(1/2,1), with FK(1/2)=0F_K(1/2)=0. Separatrix and basins of attraction. Consider first the disagreement region pA<1/2<pBp_A<1/2<p_B. In this region, p˙A>0,p˙B<0, p_A>0, p_B<0, so the two agents move toward one another. Suppose pA+pB>1p_A+p_B>1. Then pB>1−pA>1/2p_B>1-p_A>1/2. Since FKF_K is strictly increasing on (1/2,1)(1/2,1) and FK(1−pA)=FK(pA)F_K(1-p_A)=F_K(p_A), FK(pB)>FK(1−pA)=FK(pA),F_K(p_B)>F_K(1-p_A)=F_K(p_A), and therefore FK(pA)−FK(pB)<0.F_K(p_A)-F_K(p_B)<0. (18) Because this quantity is conserved, agent B cannot reach 1/21/2 before agent A: if pB=1/2p_B=1/2, then FK(pA)−FK(pB)=FK(pA)≥0F_K(p_A)-F_K(p_B)=F_K(p_A)≥ 0, contradicting Eq. (18). Thus, agent A crosses the decision boundary first. Once pA,pB>1/2p_A,p_B>1/2, Eq. (14) gives p˙A>0,p˙B>0. p_A>0, p_B>0. Both probabilities therefore increase monotonically and are bounded above by one. No interior point of (1/2,1)2(1/2,1)^2 is an equilibrium because qK(p)>0q_K(p)>0 and ϕK(p)>0 _K(p)>0 there. Hence, (pA(t),pB(t))⟶(1,1).(p_A(t),p_B(t)) (1,1). Conversely, if pA+pB<1p_A+p_B<1, then pB<1−pAp_B<1-p_A, and the same symmetry gives FK(pA)−FK(pB)>0.F_K(p_A)-F_K(p_B)>0. The conserved quantity now prevents agent A from reaching 1/21/2 before agent B. Thus, B crosses below 1/21/2 first, after which both agents satisfy p˙A<0,p˙B<0, p_A<0, p_B<0, and consequently (pA(t),pB(t))⟶(0,0).(p_A(t),p_B(t)) (0,0). The case pB<1/2<pAp_B<1/2<p_A follows symmetrically. Now consider pA+pB=1p_A+p_B=1. Setting pB=1−pAp_B=1-p_A and using Eq. (15), p˙A+p˙B p_A+ p_B =qK(pA)ϕK(1−pA)+qK(1−pA)ϕK(pA) =q_K(p_A) _K(1-p_A)+q_K(1-p_A) _K(p_A) =0. =0. Hence, the line pA+pB=1p_A+p_B=1 (19) is invariant. Along this line, if pA<1/2<pBp_A<1/2<p_B, then p˙A>0 p_A>0 and p˙B<0 p_B<0; the reverse holds when pB<1/2<pAp_B<1/2<p_A. Thus, every interior trajectory on this line converges to (1/2,1/2)(1/2,1/2). The cases where both agents initially lie on the same side of 1/21/2 follow directly from Eq. (14). Combining all cases, pA(0)+pB(0)>1⟹(pA(t),pB(t))→(1,1),p_A(0)+p_B(0)>1 (p_A(t),p_B(t))→(1,1), whereas pA(0)+pB(0)<1⟹(pA(t),pB(t))→(0,0).p_A(0)+p_B(0)<1 (p_A(t),p_B(t))→(0,0). Therefore, pA+pB=1p_A+p_B=1 is the interior separatrix between the two consensus basins. Stability of the equilibria. In a neighborhood of (1,1)(1,1), both agents satisfy pA,pB>1/2p_A,p_B>1/2 and hence both coordinates increase monotonically toward one. Thus, (1,1)(1,1) is asymptotically stable. By symmetry, (0,0)(0,0) is also asymptotically stable. Finally, consider (1/2,1/2)(1/2,1/2). Since ϕK(1/2)=0 _K(1/2)=0, let pA=1/2+δAp_A=1/2+ _A and pB=1/2+δBp_B=1/2+ _B. Linearizing Eq. (14) gives [δ˙Aδ˙B]=qK(1/2)ϕK′(1/2)[0110][δAδB]. bmatrix δ_A\\ δ_B bmatrix=q_K(1/2) _K (1/2) bmatrix0&1\\ 1&0 bmatrix bmatrix _A\\ _B bmatrix. For odd K, ϕK′(1/2)=2K(K−1(K−1)/2)(12)K−1>0. _K (1/2)=2K K-1(K-1)/2 ( 12 )^K-1>0. The Jacobian therefore has eigenvalues λ±=±qK(1/2)ϕK′(1/2), _±=± q_K(1/2) _K (1/2), with corresponding eigenvectors (1,1)(1,1) and (1,−1)(1,-1). Thus, (1/2,1/2)(1/2,1/2) has one unstable and one stable direction and is therefore a saddle point. Its stable manifold is exactly the invariant line pA+pB=1p_A+p_B=1 established above. Finally, Proposition 2 gives the correct-convergence basin under independent self-rewarding as ℬself+=(pA,pB):pA>12,pB>12.B_self^+= \(p_A,p_B):p_A> 12,\;p_B> 12 \. For Co-RL, the result above gives ℬCo-RL+=(pA,pB):pA+pB>1.B_ Co-RL^+= \(p_A,p_B):p_A+p_B>1 \. Therefore, ℬself+⊊ℬCo-RL+,B_self^+ _ Co-RL^+, which establishes the strictly larger basin of correct convergence. ∎ Appendix C Rephrased Training Questions The data-decoupled runs train one agent on the original MATH questions and the other on a rephrased copy produced by DeepSeek-V3. The two copies are aligned by row and the answer is never changed. The rewrites go beyond word substitution: most of them place the problem in a concrete scenario, roughly doubling the question length. In a sample of 300 pairs, every rewrite preserved the answer and the row alignment. Two representative pairs follow. Example 1 Answer: 2 ORIGINAL How many vertical asymptotes does the graph of y=2x2+x−6y= 2x^2+x-6 have? REPHRASED The function f(t)=2t2+t−6f(t)= 2t^2+t-6 describes the temperature of a chemical reaction over time t. How many vertical asymptotes appear on the graph of this function? Example 2 Answer: 333 3 ORIGINAL In triangle ABCABC, AB=AC=14AB=AC=14 and BC=26BC=26. What is the length of the shortest angle bisector in ABCABC? Express your answer in simplest radical form. REPHRASED A triangular park has two equal sides of 14 meters and a third side of 26 meters. The city plans a path from each corner that bisects its angle, and will build only the shortest one. How long is that path? Express your answer in simplest radical form. Appendix D Complete Experiment Results D.1 Results on 7B and 8B language models We extend the comparison in Section 6.2 to larger models, pairing Qwen2.5-7B with Llama-3.1-8B-Instruct. Table 8 reports the full results across the same seven benchmarks. The trends observed for 3B models continue to hold at this scale. All Co-RL variants improve over their respective base models on average, and using agents from different model families generally provides stronger gains than using two agents initialized from the same family. Further decoupling the training data yields the strongest overall variant, with Co-RL (Different family+) improving Qwen2.5-7B from 49.0 to 53.6 and Llama-3.1-8B-Instruct from 44.7 to 47.7 on average. Compared with prior label-free methods, Co-RL (Different family+) achieves the best average performance for both models, outperforming the strongest self-rewarding baseline by 0.8% on Qwen2.5-7B and 1.1% on Llama-3.1-8B-Instruct. Notably, on Llama-3.1-8B-Instruct, it also surpasses GT-Reward (47.7 vs. 47.1) despite using no ground-truth labels. These results show that the benefits of cross-agent supervision persist as model scale increases, and further support the importance of diversity across agents for effective label-free learning. D.2 Results on 7B–12B Vision-Language Models We further evaluate Co-RL on larger vision-language models from three different families: Qwen2.5-VL-7B, InternVL3.5-8B, and Gemma-3-12B, with InternVL3.5-8B serving as the shared training partner. Table 9 reports results across the same four multimodal reasoning benchmarks. The gains observed at smaller scales persist consistently: Co-RL improves the corresponding base models by 7.2%, 6.3%, and 5.8% on average for Qwen2.5-VL-7B, InternVL3.5-8B, and Gemma-3-12B, respectively, and outperforms TTRL for all three model families. Despite using no ground-truth supervision, Co-RL approaches GT-Reward on Qwen2.5-VL-7B and InternVL3.5-8B, while surpassing it on Gemma-3-12B (47.56% vs. 45.17%). In particular, the improvement remains consistent across models with substantially different visual encoders and language backbones. Together with the 2B–3B results in Table 4, these results show that the benefits of cross-agent supervision extend across model scales and heterogeneous vision-language architectures. Method GSM8K MATH500 AMC HEval GPQA MBPP LCB Avg Qwen2.5-7B Base 82.9 70.0 39.8 47.6 18.7 62.8 21.1 49.0 GT-Reward 84.8 77.6 49.4 56.1 23.7 64.4 25.5 54.5 TTRL 80.6 74.8 39.8 51.8 25.8 65.4 23.9 51.7 RENT 78.8 75.4 47.0 50.6 29.8 61.6 26.2 52.8 Intuitor 82.9 75.4 41.0 51.8 28.3 64.0 24.8 52.6 Co-rewarding-I 81.9 72.6 43.4 52.4 26.8 64.0 25.9 52.4 Co-RL (Same family) 78.9 74.6 41.0 52.4 25.8 61.8 25.0 51.4 Co-RL (Different family) 81.3 75.2 44.6 52.4 26.3 65.6 26.5 53.1 Co-RL (Different family+) 80.2 74.4 38.6 54.3 37.9 63.2 26.6 53.6 Llama-3.1-8B-Instruct Base 82.9 49.6 18.1 65.2 22.2 58.4 16.8 44.7 GT-Reward 82.7 53.2 25.3 64.0 30.3 59.2 15.2 47.1 TTRL 83.9 51.0 27.7 64.6 21.2 58.2 16.3 46.1 RENT 79.5 48.2 21.7 67.7 19.7 60.0 16.0 44.7 Intuitor 79.7 45.8 21.7 65.8 26.8 58.0 16.1 44.8 Co-rewarding-I 84.7 52.0 24.1 67.1 22.2 59.8 16.5 46.6 Co-RL (Same family) 85.4 51.4 22.9 68.3 23.2 60.4 15.8 46.8 Co-RL (Different family) 83.6 54.8 27.7 67.7 18.2 57.6 17.7 46.8 Co-RL (Different family+) 85.4 55.6 26.5 64.6 27.3 57.2 17.1 47.7 Table 8: Results at 7B and 8B on the seven-benchmark suite (%). Base and GT-Reward serve as references and are excluded from the ranking. Backbone Method MathVision MathVerse MathVista We-Math Avg Qwen2.5-VL-7B GT-Reward 26.74 41.07 71.90 67.01 51.68 Base 23.36 33.32 56.60 62.47 43.94 TTRL 23.62 37.26 69.40 65.23 48.88 Co-RL (Different family) 26.87 38.43 71.00 68.22 51.13 InternVL-3.5-8B GT-Reward 37.24 43.35 69.30 73.51 55.85 Base 29.21 36.65 65.70 60.69 48.06 TTRL 35.07 41.24 68.60 71.72 54.16 Co-RL (Different family) 35.30 40.74 70.60 70.98 54.40 Gemma-3-12B GT-Reward 30.89 33.63 56.90 59.25 45.17 Base 27.20 32.70 46.70 60.50 41.78 TTRL 27.93 36.37 54.70 58.79 44.45 Co-RL (Different family) 32.01 35.91 55.60 66.72 47.56 Table 9: Vision-language results at 7B to 12B on open-r1, with InternVL3.5-8B as the shared partner. D.3 Training Dynamics and Stability Text models. Figure 4 compares validation accuracy, reward standard deviation, and mean completion length throughout training. Across all four model families, Co-RL maintains a non-degenerate reward standard deviation and relatively stable completion lengths while steadily improving validation accuracy. In contrast, several self-rewarding methods become unstable during training. RENT rapidly drives the reward standard deviation toward zero and often exhibits a sharp increase in completion length, accompanied by degraded accuracy or divergence. Intuitor similarly shows substantial reductions in reward variation and, for some models, degenerate completion lengths. TTRL is more stable, but its reward variation generally decreases and its validation performance remains below Co-RL. These observations align with the dynamics analyzed in Section 5. Under self-rewarding, the supervision signal is determined by the optimized agent itself. Consequently, incorrect predictions can be reinforced rather than corrected. As the model increasingly agrees with its own pseudo-labels, rollout rewards can also become homogeneous, reducing the group-relative learning signal. In Co-RL, supervision is instead provided by another agent: since the update direction of one agent is determined by its peer’s supervision signal, errors that would be self-reinforced can be corrected when the agents have complementary strengths. The stable reward variation observed in Figure 4 is consistent with this cross-agent signal remaining informative throughout training. Vision-language models. We observe the same qualitative behavior in the multimodal setting. As shown in Figure 5, Co-RL continues to improve evaluation accuracy while maintaining stable completion lengths, whereas TTRL eventually degrades in both accuracy and response length. More importantly, the agreement between the two agents remains well below full agreement throughout training, while the accuracy of the exchanged pseudo-labels steadily increases. Thus, the agents do not simply converge to identical behaviors; instead, they preserve meaningful differences while providing increasingly reliable supervision to one another. This provides further empirical support for the mechanism predicted by our theory: cross-agent learning benefits from complementary supervision rather than requiring the agents to collapse to the same predictions. Figure 5: Training dynamics for Qwen2.5-VL-7B trained with InternVL3.5-8B on open-r1. (a) Evaluation accuracy, (b) mean completion length, and (c) the accuracy of the exchanged pseudo-labels together with the agreement between the two agents. Co-RL keeps improving and holds its completion length, while TTRL peaks and then degrades in both. D.4 Controlling for the Two-Agent Training Budget Unlike single-agent self-rewarding baselines, Co-RL jointly trains two agents. To control for this additional training budget, we construct a matched self-rewarding baseline that trains the same two base models independently with TTRL on the same prompts. At inference time, we ensemble the two independently trained models: each model generates four responses, and the resulting eight responses are pooled for majority voting. We apply the same ensemble protocol to the two agents trained with Co-RL, such that the compared methods use both the same training-model budget and the same test-time sampling budget. We additionally report each trained agent individually to separate the effect of training from that of ensembling. Tables 10 and 11 report the results for text and multimodal reasoning, respectively. Simply training two self-rewarding agents and ensembling their predictions provides only limited gains over the stronger individual model. In contrast, ensembling the agents trained with Co-RL consistently gives the best macro-average on the text benchmarks and on the two multimodal settings. These results indicate that the advantage of Co-RL cannot be explained merely by training two models. Rather, cross-agent supervision produces agents whose predictions combine more effectively under the same training and inference budgets. Setting GSM8K MATH-500 AMC Avg TTRL (Qwen2.5-3B) 88.2 68.8 39.8 65.6 TTRL (Llama-3.2-3B) 65.7 56.0 27.7 49.8 TTRL (ensemble) 88.2 68.0 38.6 64.9 Co-RL (Qwen2.5-3B) 87.4 72.8 37.4 65.9 Co-RL (Llama-3.2-3B) 87.3 58.8 33.7 59.9 Co-RL (ensemble) 90.1 70.8 39.8 66.9 Table 10: Matched-budget comparison between TTRL and Co-RL on text reasoning benchmarks. Both settings train the same two base models, Qwen2.5-3B and Llama-3.2-3B-Instruct. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, T=0.6T=0.6). Avg is the macro-average over the three benchmarks. For each benchmark, the best result is in bold and the second best is underlined, with ties sharing the marking. Setting MathVision MathVerse MathVista We-Math Avg open-r1 TTRL (Qwen2.5-VL) 22.96 31.45 61.10 63.39 44.73 TTRL (InternVL3.5) 29.67 38.91 62.30 67.24 49.53 TTRL (ensemble) 27.24 35.13 65.40 67.41 48.80 Co-RL (Qwen2.5-VL) 25.43 35.66 64.80 65.80 47.92 Co-RL (InternVL3.5) 30.46 38.60 63.30 67.87 50.06 Co-RL (ensemble) 28.95 38.48 67.00 69.08 50.88 MMR1 TTRL (Qwen2.5-VL) 17.27 30.71 63.40 60.57 42.99 TTRL (InternVL3.5) 28.78 39.47 63.70 66.90 49.71 TTRL (ensemble) 25.53 37.77 67.00 66.44 49.19 Co-RL (Qwen2.5-VL) 25.86 34.59 66.00 64.94 47.85 Co-RL (InternVL3.5) 30.79 40.94 65.30 67.53 51.14 Co-RL (ensemble) 30.49 39.75 69.40 69.54 52.30 Table 11: Matched-budget comparison between TTRL and Co-RL on multimodal reasoning benchmarks. Both settings train the same two base models, Qwen2.5-VL-3B and InternVL3.5-2B. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, T=0.6T=0.6, top-p 0.95). Rows are grouped by training set. MMR1 runs use the corrected multiple-choice grader and open-r1 runs the legacy grader, so the two blocks are not compared against each other. D.5 Evaluation Details Language model evaluation. All language benchmarks except LiveCodeBench run through lm-evaluation-harness. Sampling uses a temperature of 0.6 with top-p 0.95 and a 3072-token generation budget. Trained checkpoints are evaluated with their tokenizer’s chat template, which matches the prompt format used during training. Base models never saw a template and are evaluated without one. For AMC we sample eight responses per problem and report avg@8, averaged over three evaluation seeds. All other benchmarks use a single sample. GPQA uses the Diamond subset with a boxed-answer prompt. Answers are scored by rule-based graders with math-aware normalization, and multiple-choice questions accept both the option letter and the option value. LiveCodeBench (release v6) runs through its official harness at its default temperature of 0.2 with one sample per problem. The harness hard-codes a gated Llama-3 tokenizer for prompt construction, and we substitute the openly available Llama-3.2-3B-Instruct tokenizer, which carries the same chat template. CoMAS comparison. The comparison with CoMAS uses its benchmark suite (GSM8K, MATH-500, HumanEval, MBPP, SciBench [Wang et al. 2024b], GPQA and MMLU [Hendrycks et al. 2021a]), its driver and its graders, with training prompts drawn from their blended 2,000 problems from MATH [Hendrycks et al. 2021b], KodCode [Xu et al. 2025b] and WebInstruct-verified [Ma et al. 2025]. Their protocol draws five samples per question at temperature 0.7 and then issues a sixth call that reasons over the five drafts, and only the sixth response is graded. On coding benchmarks this aggregation admits a loophole. A response that quotes several candidate solutions has every code block executed, and grading stops at the first block that passes, so a model that quotes many candidates is effectively scored at pass@5 while a decisive one is scored at pass@1. In our measurement this is worth 7.3% to the untrained baseline and 2.4% to our model. On coding benchmarks we therefore keep the five-sample budget but replace the aggregation with majority voting over candidates clustered by their execution behavior on the public example inputs. Three-agent runs. The three-agent runs use the same training configuration as the two-agent language runs and are evaluated with the protocol above. Vision-language evaluation. We follow the benchmark splits of M-UPT, the MathVision test set (3,040 problems), the MathVerse testmini split (3,940, all five versions), the MathVista testmini split (1,000) and the We-Math testmini split (1,740). Decoding is greedy with a 16384-token generation budget. Images are resized so that the long side does not exceed 1024 pixels, matching the training-time preprocessing. Trained checkpoints are prompted in the format they were trained on, with reasoning in think tags and the final answer in answer tags. Base models receive a standard boxed-answer prompt. Scoring is two-stage. A rule-based pass extracts the final answer and grades it with math-aware matching, and a response that never commits to an answer in a recognized format counts as incorrect. Responses that follow the format but fail the rule match are passed to an LLM judge, Qwen2.5-32B-Instruct at temperature 0, which accepts only semantically equivalent answers and rejects responses that are cut off. The judge can only recover rule-grading false negatives and never overturns a rule-credited answer. MMR1 runs use the corrected multiple-choice grader and open-r1 runs the legacy grader, so results across the two training sets are not compared. Engineering notes. All fixes below ship with the released code. None of them changes the training or evaluation semantics. They repair crashes or a wrong backend choice in the underlying libraries. Gemma-3, embedding initialization under ZeRO-3. At startup, the weight initializer zeroes the embedding row at padding_idx. Under DeepSpeed ZeRO-3, most ranks hold empty parameter shards, so this write fails before training begins. The branch is reached only when a model sets padding_idx, which Gemma-3 does and Qwen2.5-VL does not. We guard the initializer to skip embeddings whose local shard is empty. Gemma-3, batched prompt tokenization. The Gemma-3 processor builds token_type_ids by stacking the unpadded prompts of a batch into one array. Prompts of unequal length make this stacking fail at the first training step. We wrap the processor to tokenize with padding and strip the padding through the attention mask immediately after. The wrapper is a no-op for every other processor. Gemma-3, log-probability drift. Between the vLLM rollout engine and the training forward pass, Gemma-3 shows a systematic per-token log-probability drift of about 0.13. This is an architectural discrepancy rather than a removable bug. Following the reference recipes for this model, Gemma-3 runs, and only Gemma-3 runs, train with token-level truncation of the importance-sampling ratio. Qwen2.5-VL, vision-tower attention backend. In vLLM 0.11.2, the helper that selects the vision tower’s attention backend silently promotes xFormers to the bundled FlashAttention build. That build supports head dimensions that are multiples of 32 only, and the Qwen2.5-VL vision tower has head dimension 80, so the model crashes at load. We patch the helper to keep the original xFormers choice, which has no head-dimension restriction. Gemma-3 and InternVL vision towers have head dimensions that are multiples of 32 and are unaffected, and later vLLM releases fix the bug. InternVL3.5, processor and tiling. We use the transformers-native HF variants, whose checkpoints load through AutoProcessor without the legacy remote-code path. Dynamic patch tiling is disabled at both training and evaluation, so the image-token count per sample is identical in the two settings. Model families are detected from each checkpoint’s configuration file rather than from directory names, since Co-RL run directories contain both partners’ names.