Paper deep dive
Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving
Yiming Wang, Yikang Liu, Qingyuan Tian, Xingyu Chen, Zhuosheng Zhang, Zhaopeng Tu, Rui Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.
Tags
Links
- Source: https://arxiv.org/abs/2609.36750v1
- Canonical: https://arxiv.org/abs/2609.36750v1
Trouble viewing inline? Open PDF directly →
Full Text
98,999 characters extracted from source content.
Expand or collapse full text
_set:Ne tcboxmath _set:Ne Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving Yiming Wang Email: yiming.wang@sjtu.edu.cn Yikang Liu Qingyuan Tian Xingyu Chen Zhuosheng Zhang, Zhaopeng Tu, Rui Wang Affiliation: School of Computer Science, Shanghai Jiao Tong University Tencent Abstract Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response’s reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones. Figure 1: (a) GMAE Motivation: RLVR evaluates each response against external ground truth, whereas in self-rewarding RL its reward representation depends on the remaining responses in its rollout group, termed its group context. (b) GMAE Overview: Conventional methods use a single group-context realization to obtain each response’s reward representation, while GMAE aggregates rewards across possible contexts into a reward distribution and estimates the expected advantage. 1 Introduction Self-evolution enables large language models (LLMs) to learn autonomously from their own experience (Tao et al., 2024), yet becomes harder as models increasingly decide what, when, and how to evolve (Gao et al., 2025). A fundamental setting is zero-label learning, where models derive signals from environments and data without human labels. In RL post-training, this motivates self-rewarding reinforcement learning (Yuan et al., 2024; Zhao et al., 2026b), which uses self-generated rewards for policy optimization, unlike RL with verifiable rewards (RLVR) relying on ground-truth labels. Existing self-rewarding RL methods mainly derive supervision from two sources of self-generated information: probability-based and ensemble-based signals (He et al., 2026). Probability-based methods estimate response quality from token probabilities or entropy (Zhao et al., 2026b; Li et al., 2025), but often provide weak supervision and may lead to training instability or collapse (Zhang et al., 2025; Roy et al., 2025). Ensemble-based methods instead aggregate the rollout response group into a reward reference, typically a pseudo-label obtained through majority voting (Wang et al., 2023a; Zuo et al., 2026), and reward each response against this reference. Owing to its simplicity and effectiveness, this paradigm is widely adopted, with subsequent methods refining reward-reference construction and reward function (Zhang et al., 2026a; Wang et al., 2026a; Wu et al., 2026). Despite their promising performance, these ensemble-based methods share a hidden rewarding limitation, as illustrated in Figure 1(a). For a response used for policy optimization, its reward representation depends not only on the response itself but also on the other responses in its rollout group. These responses constitute its randomly sampled group context. This dependence on a random factor introduces uncertainty into the reward representation: a fixed response may receive the desired reward signal only under certain group contexts, as shown in Figure 2. However, existing methods represent this context-dependent reward using only a single group-context realization. If the sampled realization misses the desired signal, that signal cannot guide subsequent policy optimization. Thus, relying on a single group-context realization for reward representation makes policy updates sensitive to random rollout sampling, reducing policy optimization reliability. Following the above analysis, a reward representation restricted to a single group-context realization captures the desired signal only when that realization happens to be suitable for the response. Since this depends on both the self-rewarding rule and the response characteristics, it cannot be guaranteed in practice. Therefore, we consider preserving more reward realizations across different group contexts to reduce the risk of missing desired signals. This motivates our Group-Marginalized Advantage Estimation (GMAE) strategy, illustrated in Figure 1(b). GMAE aggregates these reward realizations into a distribution that serves as the response-level reward representation, then uses it to estimate the response’s expected advantage. Notably, GMAE is a general strategy agnostic to reward-reference construction and reward function: these components define the self-rewarding rules and serve as the “fuel”, while GMAE acts as a general “vehicle” for leveraging them. To apply GMAE in practice, we instantiate it with three representative self-rewarding rules. We conduct extensive experiments with four base models across eight benchmarks covering mathematics, science, knowledge, and coding. GMAE achieves strong performance and generalizes from mathematical training data to unseen domains, demonstrating more reliable reward estimation. It also exhibits stable learning dynamics and low variance across random seeds, demonstrating more stable optimization. Further experiments also confirm its applicability across training datasets and RL backbones. Overall, these results establish GMAE as a reliable general framework for zero-label self-evolution through self-rewarding RL. 2 Preliminaries: RLVR and Self-Rewarding RL In modern LLM post-training, RLVR is a common RL paradigm that improves reasoning through verifiable rewards. As a representative algorithm, GRPO (Shao et al., 2024) uses a PPO-style objective (Schulman et al., 2017; Ouyang et al., 2022), removing the value model and deriving advantages from group-relative rewards. At each training step, given a prompt x sampled from the dataset D, GRPO samples G responses ⊗G=1,…,GG G=\y_1,…,y_G\ from πθold(⋅∣) _ _old(· ). Let (i)A(y_i) denote the answer corresponding to response iy_i. Each response receives a scalar reward rir_i, forming =(r1,…,rG)r=(r_1,…,r_G), and its advantage is obtained by group-wise normalization: Ai=ri−μσ,μ=1G∑j=1Grj,σ=1G∑j=1G(rj−μ)2.A_i= r_i- _r _r, _r= 1G _j=1^Gr_j, _r= 1G _j=1^G (r_j- _r )^2. (1) Throughout, zero advantages are assigned whenever σ=0 _r=0. The response-level advantage is shared across all tokens, i.e., Ai,t=AiA_i,t=A_i. Defining ρi,t=πθ(yi,t∣,i,<t)/πθold(yi,t∣,i,<t), _i,t= _θ(y_i,t ,y_i,<t)/ _ _old(y_i,t ,y_i,<t), GRPO optimizes a clipped objective with KL regularization DKLD_KL: (θ)=∼P,⊗G∼πθold⊗G(⋅∣)[1G∑i=1G1|i|∑t=1|i|minρi,tAi,t,clip(ρi,t,1−ϵ,1+ϵ)Ai,t−βDKL(πθ∥πref)]. (θ)=E_x P_D,\,G G _ _old G(· ) [ 1G _i=1^G 1|y_i| _t=1^|y_i| \! \ _i,tA_i,t,clip\! ( _i,t,1-ε,1+ε )A_i,t \-β D_KL\! ( _θ\,\|\, _ref ) ]. (2) Standard RLVR. In Eq.1, the reward rir_i of each response iy_i is computed by a reward function ℛ(i∣ξ)R(y_i ξ), where ξ denotes the reward reference. In standard RLVR, this reference is given by the ground-truth answer atruea_true. The reward is therefore determined solely by the fixed external reference: ξ≡atrue,riVR=ℛ(i∣ξ)=[(i)=atrue].ξ≡ a_true, r_i^VR=R\! (y_i ξ )= 1\! [A(y_i)=a_true ]. (3) Mainstream Pipeline of Self-Rewarding RL. However, atruea_true is unavailable in self-rewarding RL, so ξ must be inferred from the sampled rollout group ⊗G=1,…,GG G=\y_1,…,y_G\. Specifically, a group-dependent function f constructs the reward reference ξ=f(⊗G)ξ=f(G G), against which the reward for iy_i is: riSR=ℛ(i∣ξ)=ℛ(i∣f(⊗G)).r_i^SR=R\! (y_i ξ )=R\! (y_i f(G G) ). (4) A common choice is majority voting (Wang et al., 2023a; Zuo et al., 2026), where ξ is the most frequent answer among (1),…,(G)\A(y_1),…,A(y_G)\, and ℛR is a binary indicator. Other methods refine f or ℛR while following the same pipeline. Their details are provided in Appendix B. 3 On the Limitation of Self-Reward Uncertainty Existing self-rewarding methods differ in their choice of (f,ℛ)(f,R), but share a common property: the reward assigned to each response depends on the rollout group it was sampled from. This dependence is implicit in Eq.4, where the pseudo-label f(⊗G)f(G G) is constructed from the entire rollout group. To make this dependence explicit, we fix a response iy_i and decompose its rollout group as ⊗G=i∪ℋi,ℋi≜jj≠i,G G=\y_i\ _i, _i \y_j\_j≠ i, (5) where ℋiH_i denotes the group context realized alongside iy_i. Since all responses are independently sampled from the same policy πθold(⋅∣) _ _old(· ), group contexts follow the same underlying distribution for all focal responses. We therefore use ℋH to denote a generic random group context. For a fixed response iy_i, its reward under ℋH is ri(ℋ)=ℛ(i∣f(i∪ℋ)),ℋ=1,…,G−1∼Q(⋅∣)≡πθold⊗(G−1)(⋅∣)r_i(H)=R\! (y_i f\! (\y_i\ ) ), =\z_1,…,z_G-1\ Q(· )≡ _ _old (G-1)(· ) (6) Although ri(ℋi)r_i(H_i) is deterministic once the realized context ℋiH_i is fixed, ℋH itself is randomly sampled. Thus, the reward intended to estimate the quality of iy_i depends not only on the response itself but also on its group context, introducing uncertainty into the resulting learning signal. Figure 2: Self-Reward Uncertainty Demonstration. (a) Reward distributions for a fixed correct response (top) and a fixed incorrect response (bottom) across group contexts under TTRL (Zuo et al., 2026), Self-Harmony (Wang et al., 2026a), SCOPE (Wang et al., 2026b), and SR-TTRL (Wu et al., 2026). Dots denote individual realizations and gray bars denote means. (b) Joint distributions of self-rewarding (x-axis) and RLVR (y-axis) advantages for the same responses. Results here are from Qwen3-8B-Base on a DAPO-14K prompt; more cases are provided in Appendix A. Empirical Validation. We examine how group-context variation affects the reward and advantage of the same response. Given a model π, we fix a prompt x and sample one correct focal response +y^+ and one incorrect focal response −y^-. For each response, we sample M=100M=100 independent group contexts, each containing G−1G-1 newly sampled responses with G=16G=16. These contexts produce M self-reward realizations and group-normalized advantages, forming their empirical distributions. For comparison, we compute the corresponding RLVR advantage under each context. Figure 2 presents results for four representative self-rewarding methods. As shown in (a), the reward of the same response varies substantially across group contexts. Although the desired signal is positive for a correct response and negative for an incorrect one, both may receive the opposite signal under some contexts. Thus, restricting reward assignment to a single group-context realization may miss the desired signal, limiting the reliability of self-reward as an estimate of response quality. In (b), RLVR advantages vary in magnitude but preserve the desired direction: correct responses remain non-negative, whereas incorrect ones remain non-positive. By contrast, self-rewarding advantages often change sign, causing the same response to be reinforced under one context but suppressed under another. Therefore, group-context uncertainty in reward assignment can propagate into advantage estimation, making the optimization direction dependent on the sampled context. 4 GMAE: Group-Marginalized Advantage Estimation Section 3 shows that a correct response may miss a positive reward signal under a single group context. However, the suitable context is response-dependent and unknown in advance, making it difficult to identify in practice. Therefore, instead of relying on one group context, we aggregate reward realizations across contexts into a distribution-level response representation, improving the chance of retaining desired signals. This idea motivates our Group-Marginalized Advantage Estimation (GMAE). In Section 4.1, we will introduce its core algorithm, and in Section 4.2, we will instantiate it with different self-rewarding rules. 4.1 Core Algorithm Main Step I: Group-Marginalized Reward Distribution. For a response iy_i, instead of using the scalar reward induced by the single realized group context ⊗G/iG G/\y_i\, we consider its reward over the group-context distribution in Eq.6. Let δv _v denote the Dirac measure concentrated at v, we define the group-marginalized reward distribution as Pri=∫δℛ(i∣f(i∪ℋ))Q(ℋ∣)=∫δℛ(i∣f(i,1,…,G−1))∏j=1G−1dπθold(j∣), [rgb]0.2031,0.5977,0.8594P_r_i= _R (y_i f (\y_i\ ) )\,dQ(H )= _R (y_i f (\y_i,z_1,…,z_G-1\ ) ) _j=1^G-1d _ _old(z_j ), (7) However, directly computing Eq.7 is intractable because it requires marginalizing over the (G−1)(G-1)-fold product distribution. Naive Monte Carlo estimation can approximate this marginalization, but requires repeatedly sampling fresh group contexts for each response, incurring substantial rollout cost. Therefore, we reuse sampled responses through a shared candidate pool (Wang et al., 2026c). Specifically, besides the G main responses ⊗G=1,…,GG G=\y_1,…,y_G\ used for policy optimization, we sample G′G auxiliary responses to form =1,…,G+G′C=\y_1,…,y_G+G \. For each main response iy_i (1≤i≤G1≤ i≤ G), choosing G−1G-1 companions from ∖iC \y_i\ yields (G+G′−1G−1) G+G -1G-1 possible group contexts. We uniformly sample K distinct contexts ikk=1K\S_i^k\_k=1^K up to τ, and construct the empirical distribution11 1 To avoid combinatorial explosion in enumeration, we set τ=104τ=10^4 by increasing the number of evaluated contexts tenfold from 1010 and observing negligible changes in the empirical reward distribution beyond 10410^4. P^ri=1K∑k=1Kδℛ(i∣f(i∪ik)),K=minτ,(G+G′−1G−1). [rgb]0.2969,0.7344,0.0898 P_r_i= 1K _k=1^K _R (y_i f (\y_i\ _i^k ) ), K= \τ, G+G -1G-1 \. (8) Now we measure the error from replacing the ideal PriP_r_i in Eq.7 with the empirical P^ri P_r_i in Eq.8 with the total variation distance dTV(⋅,⋅)d_TV(·,·). For a finite reward-value space i=supp(Pri)V_i=supp(P_r_i), with the confidence level and G fixed, the error scales as dTV(P^ri,Pri)=(|i|K+|i|G+G′). d_TV\! ( P_r_i,P_r_i )=O\! ( |V_i|K+ |V_i|G+G ). (9) It decreases with K and G′G but increases with |i||V_i|. The exact bound and proof are in Appendix C.1. Main Step I: Distribution-based Advantage Estimation. After marginalizing over group contexts, each P^ri P_r_i defines a response-level reward profile that is no longer tied to any particular group-context realization. Accordingly, GMAE treats these distributions as response-level reward representations and independently draws reward realizations from them for advantage estimation. Specifically, we construct the product distribution ⨂j=1GP^rj _j=1^G P_r_j over joint reward realizations =(v1,…,vG)v=(v_1,…,v_G), where each vj∼P^rjv_j P_r_j and ∈^=∏j=1G^jv∈ V= _j=1^G V_j. For each v, we compute the group-normalized advantage and then take its expectation under the product distribution, yielding Aigmae=∼⨂j=1GP^rj[vi−μσ]=∑∈^∏j=1G[1K∑k=1K[ℛ(j∣f(j∪jk))=vj]]vi−μσ. A_i^gmae=E_v _j=1^G P_r_j [ v_i- _v _v ]= _v∈ V \ _j=1^G [ 1K _k=1^K 1 [R (y_j f (\y_j\ _j^k ) )=v_j ] ] \ v_i- _v _v. (10) Attached Mechanism: Advantage Scale Calibration. Although group-marginalized advantages preserve richer reward information, marginalization can shrink their scale and weaken policy updates. Since ∑i=1GAigmae=0 _i=1^GA_i^gmae=0, their variance is Var(Aigmaei=1G)=1G∑i=1G(Aigmae)2Var(\A_i^gmae\_i=1^G)= 1G _i=1^G(A_i^gmae)^2, which directly reflects the advantage scale and effective update strength. By Jensen’s inequality (Jensen, 1906), Var(Aigmaei=1G)=1G∑i=1G([vi−μσ])2≤[1G∑i=1G(vi−μσ)2]≤1. \! (\A_i^gmae\_i=1^G )= 1G _i=1^G (E_v [ v_i- _v _v ] )^2 _v [ 1G _i=1^G ( v_i- _v _v )^2 ]≤ 1. (11) The right-hand side is the expected variance of conventional group-wise advantages without marginalization and is at most 1 under normalization. Thus, marginalization often reduces the advantage scale, motivating calibration to restore the non-marginalized update strength. Prompt-level calibration could counteract GMAE’s reduced advantage scale by matching each prompt to its scale from a single group context. However, scale differences across prompts can reflect prompt-specific reward uncertainty. A response favored in some contexts but disfavored in others has normalized advantages that partially cancel under marginalization. This reduces the prompt’s scale and makes its updates more conservative. Calibrating each prompt separately would erase these uncertainty-dependent differences. We formalize this relation in Appendix C.2. Therefore, we adopt batch-level calibration to restore the overall scale while preserving relative scale differences across prompts. For a batch of B prompts, let Ab,ibaseA_b,i^base denote the non-marginalized advantage. We denote the batch-level scales of Ab,ibaseA_b,i^base and Ab,igmaeA_b,i^gmae by σbase _base and σgmae _gmae: σbase=1BG∑b=1B∑i=1G(Ab,ibase)2,σgmae=1BG∑b=1B∑i=1G(Ab,igmae)2. _base= 1BG _b=1^B _i=1^G (A_b,i^base )^2, _gmae= 1BG _b=1^B _i=1^G (A_b,i^gmae )^2. (12) We use σbase _base as the reference because it retains the conventional update scale induced by group-wise normalization. We then calibrate all marginalized advantages using a shared factor λ, with a lower bound of 11 to ensure that calibration only compensates for scale contraction: A~b,igmae=λAb,igmae,λ=max(1,σbaseσgmae). [rgb]0.2969,0.7344,0.0898 A_b,i^gmae=λ A_b,i^gmae, λ= (1,\, _base _gmae ). (13) 4.2 Instantiating GMAE with Self-Rewarding Rules As shown in Section 4.1, GMAE is a general strategy agnostic to the reward-reference construction f and reward function ℛR. Different (f,ℛ)(f,R) configurations define different self-rewarding rules that provide the “fuel”, while GMAE serves as the “vehicle” that better leverages these signals for policy optimization. Naturally, concrete choices of f and ℛR are required to instantiate GMAE. We first instantiate ℛR with the binary reward ℛbin(i∣ξ)=[(i)=ξ]R_bin(y_i ξ)= 1[A(y_i)=ξ], where ξ denotes the reward reference. We adopt it for three reasons. First, it follows the standard reward design in RLVR. Second, it restricts i,^i⊆0,1V_i, V_i \0,1\, and hence |^|≤2G| V|≤ 2^G, minimizing the reward-value space enumerated in Eq.10. Third, |i|≤2|V_i|≤ 2 tightens the approximation bound in Eq.9. With ℛbinR_bin fixed, we combine it with three representative f, yielding three corresponding GMAE instantiations: GMAE0: Standard Voting. Following TTRL (Zuo et al., 2026), GMAE0 constructs the pseudo-label by majority voting (Wang et al., 2023a) over responses sampled from the current policy: ξ0=f0(⊗G)=argmaxa∑j=1G[(j)=a]. _0=f_0(G G)= _a _j=1^G 1\! [A(y_j)=a ]. (14) GMAECR: Co-Refinement. Following Co-Reward (Zhang et al., 2026b), GMAECR defines the group context ℋH as G responses sampled from a slowly updated co-teacher and constructs the reward reference by majority voting over this context, where αt _t denotes the teacher retention coefficient: θ~t=αtθ~t−1+(1−αt)θold,t,ℋ~=~jj=1G∼πθ~t⊗G(⋅∣),fCR(ℋ~)=argmaxa∑j=1G[(~j)=a]. θ_t= _t θ_t-1+(1- _t) _old,t, H=\ y_j\_j=1^G _ θ_t G(· ), f_CR\! ( H )= _a _j=1^G 1\! [A( y_j)=a ]. (15) GMAESR: Self-Refinement. Following SR-TTRL (Wu et al., 2026), GMAESR uses self-reflective verification instead of voting. For each distinct answer a, it summarizes a representative response and constructs the reward reference through self-reflective comparison: a=Summπθold(,Repj∣(j)=a),a∈,=(j)j=1G. _a=Summ_ _ _old (x,Rep \y_j (y_j)=a \ ), a , =\A(y_j)\_j=1^G. (16) ξSR=fSR(⊗G)=Reflectπθold(,(a,a)a∈). _SR=f_SR(G G)=Reflect_ _ _old (x,\(a,s_a)\_a ). Since our core contribution lies in the general GMAE strategy, we only briefly introduce the self-rewarding rules used in these three instantiations here. Their detailed pseudocode and computational complexity analyses are deferred to Appendix D. 5 Experiments Table 1: Main Results Across Models and Domains: Mean@16 scores on eight benchmarks across four base models. Average is computed over all eight benchmarks. Bold and underlined values denote the best and second-best results among all self-rewarding methods, respectively. Method Mathematics Science Knowledge Coding Average Mathematics Science Knowledge Coding Average MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode Qwen3-1.7B-Base Qwen3-4B-Base Raw Model 45.70.22 17.00.09 2.30.07 0.80.18 1.30.19 7.90.23 13.30.29 6.70.05 11.90.07 50.50.06 24.30.18 8.00.20 3.80.11 5.40.05 15.80.27 14.20.17 15.30.06 17.20.09 w/ Verifiable Reward 64.60.08 27.10.13 7.10.20 5.40.18 4.20.23 22.60.24 31.00.23 14.20.07 22.00.17 81.30.16 48.00.25 19.30.03 17.70.11 14.80.18 33.00.17 39.40.32 19.10.08 34.10.13 Intuitor∗ 52.10.14 19.80.10 1.10.24 1.20.13 0.80.18 13.30.24 15.70.30 10.70.22 14.30.21 67.10.18 32.90.04 9.20.21 2.70.20 0.70.20 30.00.22 41.10.23 15.80.08 24.90.21 EM-RL∗ 57.50.08 19.20.23 2.60.18 1.70.13 0.90.22 13.20.32 14.80.19 11.60.08 15.20.06 70.90.19 30.50.18 4.40.09 6.30.04 2.60.08 28.30.19 40.80.25 16.30.15 25.00.21 TTRL 61.10.13 24.00.22 3.40.24 2.70.07 1.10.04 20.20.22 23.70.22 11.30.24 18.40.15 74.60.23 37.80.04 8.80.11 5.00.19 5.30.11 32.90.16 35.40.30 16.00.14 27.00.05 CoVo 52.60.08 19.00.16 1.70.11 1.10.22 1.20.06 12.80.24 17.10.25 10.10.21 14.50.12 60.00.19 29.50.11 7.90.07 5.60.04 4.80.03 28.00.20 31.90.14 15.20.10 22.90.21 SCOPE 56.20.07 22.30.10 4.00.22 2.50.07 1.50.05 12.00.18 13.60.19 8.40.22 15.10.21 73.80.24 38.10.05 9.80.22 6.50.21 6.10.10 31.60.31 30.20.14 16.60.22 26.60.03 Self-Harmony 60.10.08 22.00.21 4.00.20 1.50.12 1.50.21 20.30.23 27.40.12 11.40.19 18.50.12 73.60.12 33.60.18 8.40.23 5.30.17 5.30.08 34.80.15 41.70.31 16.20.25 27.40.10 Co-Reward 61.20.23 23.60.21 4.00.05 2.60.21 1.10.15 18.90.27 23.20.27 12.30.25 18.40.06 74.40.06 38.00.13 9.40.19 4.80.12 5.70.15 33.70.25 33.80.13 17.20.09 27.10.10 RESTRAIN 58.30.17 22.30.16 3.60.10 2.10.15 1.90.19 16.20.25 19.80.30 11.70.16 17.00.11 71.00.24 37.20.04 8.20.23 5.50.17 4.90.12 31.00.28 35.20.15 16.90.23 26.20.09 SR-TTRL 60.90.08 24.30.23 4.10.12 2.50.11 0.70.21 20.20.25 24.70.19 11.90.24 18.70.18 74.30.16 38.70.13 9.80.09 5.90.23 5.50.21 32.10.19 31.00.18 16.60.09 26.70.19 GMAE0_0 (Ours) 63.00.11 25.80.06 5.50.09 4.20.04 2.50.12 22.20.18 25.40.22 14.30.07 20.40.10 75.70.05 39.70.12 10.90.08 8.40.06 6.50.10 34.10.15 38.80.21 17.10.09 28.90.07 GMAECR_CR (Ours) 63.80.06 26.60.10 6.40.05 5.00.08 3.00.11 22.00.14 25.00.19 14.50.12 20.80.06 76.20.09 40.20.05 11.50.11 8.10.07 7.80.04 35.30.17 43.40.20 18.30.10 30.10.08 GMAESR_SR (Ours) 63.60.08 26.40.05 6.90.12 4.60.09 3.30.06 22.20.20 26.30.16 14.80.04 21.00.11 75.70.12 41.20.08 12.00.06 9.20.10 7.00.05 34.30.22 40.40.18 17.50.07 29.70.09 Qwen3-8B-Base Llama3.1-8B-Instruct Raw Model 68.50.19 35.40.11 11.30.20 7.30.07 6.10.19 31.00.17 42.80.14 19.50.06 27.70.12 47.30.14 21.30.06 7.20.15 1.50.13 1.20.25 17.40.13 21.50.29 9.20.14 15.80.20 w/ Verifiable Reward 83.10.25 49.40.09 20.40.16 19.10.20 16.30.17 42.90.15 71.20.30 26.00.09 40.90.11 69.40.11 41.20.07 16.20.24 7.20.14 7.00.04 25.80.28 35.50.26 16.50.07 27.40.16 Intuitor∗ 71.10.03 37.80.04 11.90.15 9.70.23 6.80.24 35.20.14 60.60.14 19.80.05 31.60.17 52.10.06 27.60.11 7.70.19 2.70.06 1.60.06 19.90.17 27.60.32 10.20.05 18.70.21 EM-RL∗ 73.40.12 40.40.25 13.10.22 9.80.25 7.90.10 35.80.29 65.30.22 20.50.09 33.30.22 58.90.15 34.20.18 9.00.07 3.50.17 2.20.04 20.30.24 26.70.27 13.10.07 21.00.17 TTRL 77.80.25 43.60.03 13.10.14 10.50.09 6.10.16 39.70.24 64.10.14 20.90.17 34.50.04 63.80.24 36.60.10 11.60.16 5.00.19 4.10.21 21.10.30 31.80.13 14.30.12 23.50.17 CoVo 69.90.11 37.10.13 11.90.09 9.10.15 7.00.06 32.30.13 52.80.16 20.40.04 30.10.05 51.30.15 29.50.25 10.20.14 2.40.17 1.90.05 18.50.22 29.30.18 12.40.16 19.40.13 SCOPE 77.50.23 43.30.07 12.50.20 11.50.16 7.10.22 38.60.30 59.30.24 19.50.10 33.70.15 60.30.15 32.80.13 12.10.04 4.20.08 3.00.09 20.00.22 30.30.21 11.90.22 21.80.17 Self-Harmony 76.30.08 41.10.22 12.70.15 11.50.18 6.50.10 40.70.30 70.50.21 21.90.16 35.20.11 61.70.15 33.60.09 8.00.16 3.90.19 1.80.06 21.80.30 28.40.14 13.00.24 21.50.08 Co-Reward 78.00.16 43.50.15 13.40.06 10.40.05 6.90.25 40.60.23 65.30.14 21.10.15 34.90.17 65.10.12 36.80.09 12.10.19 5.30.20 4.70.09 21.60.27 32.40.24 15.30.19 24.20.23 RESTRAIN 77.30.19 44.80.10 13.80.17 9.80.10 8.10.12 39.00.12 65.00.21 20.70.23 34.80.20 64.20.11 30.60.21 10.90.23 4.30.14 4.00.18 20.70.20 30.80.31 13.90.11 22.40.16 SR-TTRL 77.30.16 44.50.18 13.80.18 11.10.11 7.20.17 39.20.31 63.70.31 20.10.14 34.60.06 65.00.06 36.30.13 12.10.15 5.30.25 5.50.15 22.00.30 31.60.29 13.50.06 23.90.12 GMAE0_0 (Ours) 79.80.05 46.20.11 16.20.07 12.50.13 9.20.06 41.70.19 67.40.23 23.50.08 37.10.10 66.50.12 38.40.06 13.20.09 6.50.04 5.80.11 22.90.16 32.80.21 15.40.05 25.20.09 GMAECR_CR (Ours) 81.60.09 47.00.04 17.40.10 13.50.06 9.50.12 41.30.15 69.20.18 24.30.11 38.00.07 67.70.06 39.80.10 14.00.05 6.60.08 6.40.13 23.90.17 34.20.20 16.70.09 26.20.06 GMAESR_SR (Ours) 80.40.07 46.40.12 16.50.04 13.10.09 8.60.05 40.70.21 66.60.17 22.60.06 36.90.11 68.00.10 39.20.05 13.60.12 6.10.07 6.00.09 23.30.19 33.70.16 16.20.11 25.80.08 5.1 Setup Models. We conduct our main experiments on Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen3-8B-Base (Yang et al., 2025) to evaluate GMAE across model scales. We also include Llama3.1-8B-Instruct (Grattafiori et al., 2024) to assess its adaptability across model families. Datasets. For training, we use the math dataset DAPO-14K (Yu et al., 2026). For evaluation, we select eight benchmarks across mathematics, science, knowledge, and coding domains. For mathematics, we use MATH500 (Hendrycks et al., 2021), AMC, AIME24, AIME25, and AIME26. The out-of-domain (OOD) benchmarks include GPQA (Rein et al., 2024) for science, MMLU-Pro (Wang et al., 2024b) for knowledge, and LiveCodeBench-v6 (Jain et al., 2025) for coding. Baselines. We primarily compare GMAE with ensemble-based methods, including TTRL (Zuo et al., 2026), CoVo (Zhang et al., 2026a), SCOPE (Wang et al., 2026b), Self-Harmony (Wang et al., 2026a), RESTRAIN (YU et al., 2026), Co-Reward (Zhang et al., 2026b), and SR-TTRL (Wu et al., 2026). Detailed descriptions are provided in Appendix B. We also include two probability-based methods: Intuitor (Zhao et al., 2026b), which uses self-certainty as its reward, and EM-RL (Agarwal et al., 2026), which uses trajectory entropy as its reward. We further include an RLVR reference to measure the remaining gap to verifiable supervision. We faithfully reproduce all baselines using official implementations when available; otherwise, we implement them based on the original papers. Implementation. We adopt GRPO (Shao et al., 2024) as the RL backbone and train each base model for one epoch with batch size 6464. We use a learning rate of 1×10−61× 10^-6 with 5%5\% warmup followed by cosine decay (Loshchilov and Hutter, 2017). For each prompt, we sample G=16G=16 main and G′=16G =16 auxiliary responses with temperature T=1.0T=1.0 and no top-p or top-k truncation, forming a candidate pool of 3232 responses. All 3232 responses are generated fully in parallel during rollout. We implement all experiments with verl on NVIDIA A100 80GB GPUs. Evaluation. For each evaluation dataset, we report Mean@16, defined as the average Pass@1 accuracy over 16 sampled responses per problem. The subscript reports the standard deviation of per-problem Pass@1 outcomes aggregated over the evaluation set. We use temperature T=0.6T=0.6, top-p=0.95p=0.95, and top-k=20k=20 for sampling. During training, we evaluate checkpoints on DAPO-500 (Yu et al., 2026) every 2020 steps and at training end, selecting the checkpoint with the highest validation Mean@16 for final benchmarking. Fair Cost Control. Training cost is reflected in two aspects: GPU memory and training duration. For GPU memory, all methods use G=16G=16 responses for policy updates. For a fair comparison, the baselines also generate 3232 responses per prompt, with the additional responses used only for reward-reference construction. This enlarged rollout scheme has been shown to improve performance (Zuo et al., 2026). For training duration, it varies with the computation required by each reward construction and is discussed separately in Section 5.3. 5.2 Main Results Overall Comparisons: Robust Improvement and Generalization. In Table 1, GMAE outperforms existing self-rewarding baselines across 30 of 32 model-benchmark combinations. It achieves improvements of 2.0-2.8 points over the strongest baselines, corresponding to 8.0%-12.3% relative gains across different models. Notably, GMAE approaches the performance of RLVR with verifiable rewards. For example, the performance gap on Qwen3-1.7B-Base is reduced to only 1.0 point. Moreover, although trained exclusively on mathematics, GMAE maintains strong performance on OOD benchmarks, demonstrating its broad generalization ability. Controlled Comparisons: GMAE as a Strong General Strategy. To isolate the contribution of GMAE from the underlying self-rewarding rules, we compare each GMAE instantiation with its base version: GMAE0 with TTRL, GMAECR with Co-Reward, and GMAESR with SR-TTRL. Across all comparisons, GMAE consistently improves the corresponding methods, yielding average relative gains of 8.2%, 10.3%, and 9.5%, respectively. Meanwhile, the relative ranking among different self-rewarding rules is largely preserved after applying GMAE, indicating that GMAE improves reward utilization while retaining their individual strengths. Notably, even the weakest GMAE instantiation consistently outperforms the strongest non-GMAE baseline across all evaluated models. These results demonstrate that the improvement comes from the GMAE strategy for leveraging self-reward signals, rather than from any specific self-rewarding rule alone. 5.3 Detailed Training Behavior Training Dynamic: Stability Analysis. Stable training dynamics are essential for sustained self-improvement. Figure 3 tracks performance on the DAPO-500 validation set throughout training. Several baselines improve initially but then degrade sharply, exhibiting a “rise-then-collapse” pattern. Notably, both probability-based methods show this behavior, suggesting that probability-based supervision can be unreliable (Zhang et al., 2025). In contrast, GMAE improves steadily and converges without collapse, demonstrating substantially more reliable training dynamics. Figure 3: Training Dynamics on the DAPO-500 validation set over training steps. Figure 4: Multi-Seed Training Variance. Results show the average Mean@16 standard deviation across eight benchmarks for Qwen3-8B-Base over five training seeds. Training Robustness: Multi-Seed Training Variance. As analyzed in Section 3, randomness in rollout sampling may flip the sign of the estimated advantage for the same response, substantially affecting policy updates. We therefore test whether GMAE produces more consistent outcomes across runs. We evaluate each method over five runs on Qwen3-8B-Base, keeping the hardware, initialization, training data, batch order, and hyperparameters fixed while varying only the rollout-sampling seed. Figure 4 reports the standard deviation of Mean@16 across runs, averaged over eight benchmarks. GMAE0 achieves the lowest variance at 0.48, with lower variance in every controlled comparison: TTRL → GMAE0 (0.69 → 0.48), Co-Reward → GMAECR_CR (0.76 → 0.55), and SR-TTRL → GMAESR_SR (0.72 → 0.57). These results show that beyond improving absolute performance, GMAE also makes training more robust. Figure 5: Training Duration vs. Performance. Each point represents one method. Duration is normalized to RLVR, and performance is the average Mean@16 across eight benchmarks on Qwen3-8B-Base. Training Duration: Cost Analysis. With equal rollout sampling budgets, we compare end-to-end training time under the same hardware and settings, normalized to RLVR. Figure 5 shows that all three GMAE instantiations remain within the low-to-moderate cost range of existing baselines and add little overhead over their corresponding base versions. This limited overhead arises because marginalization reuses cached responses and requires only lightweight scalar computation. For example, GMAE0 introduces only G(K−1)G(K-1) lightweight voting operations per prompt and at most 2G2^G scalar normalizations over TTRL, resulting in nearly identical training time. Meanwhile, the more elaborate GMAE instantiations require additional computation from their underlying self-rewarding rules but achieve correspondingly stronger performance, yielding a favorable cost-performance trade-off. 6 Extended Analysis 6.1 Why Does GMAE Work? A Correct-Response Frequency Argument The key motivation for GMAE is that a correct response may receive a positive reward only in certain group contexts, which a single realization may miss. By retaining reward realizations across contexts, GMAE preserves even sparse positive evidence. Extending this intuition to model performance, we initially expected GMAE to prevent rare correct responses from disappearing while keeping common ones frequent, as they are consistently rewarded across contexts. We test this explanation by tracking correct-response frequencies throughout training. Figure 6: Distribution of Correct Responses per Prompt. Illustrative proportions of training prompts by the number of correct responses among G=16G=16 sampled responses under GMAE0 and TTRL at steps 50 and 200. We compare illustrative count distributions for GMAE0 and its non-marginalized version TTRL on Qwen3-8B-Base at steps 50 and 200, representing early and near-final stages of the 211-step training. As shown in Figure 6, TTRL has more prompts with no correct responses, whereas GMAE0 has more prompts with a small or moderate number of correct responses. The two distributions have almost the same mass at 16 correct responses. This pattern suggests that GMAE helps keep rare correct responses from disappearing while maintaining success on prompts that are already solved consistently. 6.2 Ablation Study Figure 7: Auxiliary Response Size Ablation. Average Mean@16 across all eight benchmarks with different auxiliary response sizes. Auxiliary Response Size G′G . Eq.9 establishes that the upper bound on the estimation error of P^ri P_r_i decreases with larger G′G . We validate this result empirically by varying G′G under otherwise identical configurations and performing multiple runs. When G′=0G =0, the GMAE instantiation reduces to its base version. As shown in Figure 7, increasing G′G consistently improves performance, but the marginal benefit drops markedly beyond G′=16G =16. This suggests that our default choice of G′=16G =16 provides a strong balance between performance and rollout cost. Advantage Scale Calibration Mechanism. Section 4 theoretically motivates advantage scale calibration and explains why calibration should be applied at the batch rather than prompt level. Now we empirically validate this design. Table 2 shows that batch-level calibration performs best on both models, whereas prompt-level calibration performs worst, even underperforming no calibration. Table 2: Calibration Ablation on GMAE0. Mean@16 results for Qwen3-8B-Base. MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode Average w/o Calibration 78.4 45.2 16.0 10.3 9.7 39.2 63.6 22.4 35.6 w/ Prompt-level Calibration 78.0 44.7 15.5 10.8 8.6 39.4 63.8 21.3 35.3 w/ Batch-level Calibration (Ours) 79.8 46.2 16.2 12.5 9.2 41.7 67.4 23.5 37.1 Figure 8: Calibration Factor λ Dynamics. Figure 8 further shows how the calibration factor λ (Eq.13) evolves during training. Its value remains between 1.15 and 1.35, confirming that GMAE consistently requires moderate scale compensation. The factor gradually decreases as training proceeds, indicating that the gap between the non-marginalized and marginalized advantage scales narrows over time. 6.3 Scalability Study Table 3: Scalability Across Training Settings. Average Mean@16 across all eight benchmarks for Qwen3-8B-Base. Full benchmark-wise results are in Appendix E. Method RL Backbones Training Datasets GSPO REINFORCE++ Open-RS MATH-8K TTRL 35.8 32.8 34.0 33.2 Self-Harmony 36.1 33.6 34.9 33.0 Co-Reward 37.3 33.9 35.3 34.3 SR-TTRL 36.8 33.6 35.2 34.3 GMAE0 (Ours) 38.1 36.4 36.2 35.5 GMAECR_CR (Ours) 39.3 37.5 37.3 36.2 GMAESR_SR (Ours) 38.7 37.1 36.6 36.5 We evaluate GMAE’s scalability along two dimensions: RL backbones and training datasets. For the former, we replace the default GRPO backbone with GSPO (Zheng et al., 2025) and REINFORCE++ (Hu et al., 2025), keeping all other settings fixed. For the latter, we train all methods on Open-RS (Dang and Ngo, 2026) and MATH-8K (Hendrycks et al., 2021), in addition to DAPO-14K. As shown in Table 3, GMAE consistently retains its advantages over representative strong baselines from Table 1, demonstrating scalability across the two training settings. 7 Related Work Self-Evolving LLMs. Self-evolving LLMs improve through self-generated experience (Tao et al., 2024; Gao et al., 2025). They evolve data through self-generation or instruction evolution (Wang et al., 2023b; Xu et al., 2024) and failure-targeted augmentation (Lee et al., 2024b); reasoning through iterative refinement (Madaan et al., 2023; Shinn et al., 2023), latent-rationale bootstrapping (Zelikman et al., 2022; Zelikman et al., 2024), and structure discovery (Zhou et al., 2024); and policies or tasks through reinforced self-training, self-play (Gulcehre et al., 2023; Chen et al., 2024), and proposer–solver co-evolution (Zhao et al., 2026a; Huang et al., 2026). Yet many require demonstrations, labels, environmental feedback, or verifiers. We instead study zero-label evolution from prompts alone, without ground-truth answers or external reward models. Self-Rewarding RL. Self-rewarding RL uses self-generated supervision, extending RLAIF (Bai et al., 2022; Lee et al., 2024a) to closed self-evaluation loops (Yuan et al., 2024; Wang et al., 2024a; Wu et al., 2025). For reasoning, probability-based methods score individual rollouts using confidence or self-certainty (Zhao et al., 2026b; Li et al., 2025), while entropy-based objectives reward concentrated token distributions (Agarwal et al., 2026). Ensemble-based methods derive pseudo-supervision from multiple rollouts, beginning with answer-frequency consensus (Zuo et al., 2026) and extending to trajectory consistency (Zhang et al., 2026a), weighted or subgroup voting (Wang et al., 2026b), transformed-view agreement (Wang et al., 2026a), complementary or teacher-guided consensus (Zhang et al., 2026b), and self-reflection (Wu et al., 2026). In contrast, GMAE supports different self-reward rules and addresses their shared dependence on sampled rollout groups. 8 Conclusion In this paper, we introduced GMAE, a general advantage-estimation strategy for self-rewarding RL. By marginalizing rewards over possible group contexts, GMAE provides more reliable optimization signals while remaining compatible with different self-rewarding rules. Experiments across multiple models and benchmarks demonstrate consistent improvements and stable training, supporting GMAE as an effective approach to zero-label self-evolution. AI Use Statement In this work, we used generative AI tools for writing polishing and the retrieval of some related work. We have verified the correctness of the AI-generated content and take responsibility for the final content of this work, including any text, claims, or artifacts produced with the aid of generative AI. References Agarwal et al. (2026) S. Agarwal, Z. Zhang, L. Yuan, J. Han, and H. Peng The unreasonable effectiveness of entropy minimization in llm reasoning. Advances in Neural Information Processing Systems 38, p. 107150–107180. Cited by: §5.1, §7. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. Cited by: §7. Chen et al. (2024) Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335. Cited by: §7. Dang and Ngo (2026) Q. Dang and C. Ngo Reinforcement learning for reasoning in small LLMs: what works and what doesn’t. In Logical and Symbolic Reasoning in Language Models @ AAAI 2026, Cited by: §6.3. Gao et al. (2025) H. Gao, J. Geng, W. Hua, M. Hu, X. Juan, H. Liu, S. Liu, J. Qiu, X. Qi, Y. Wu, et al. A survey of self-evolving agents: what, when, how, and where to evolve on the path to artificial super intelligence. arXiv preprint arXiv:2507.21046. Cited by: §1, §7. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1. Gulcehre et al. (2023) C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998. Cited by: §7. He et al. (2026) B. He, Y. Zuo, Z. Liu, S. Zhao, Z. Fu, J. Yang, C. Qian, K. Zhang, Y. Fan, G. Cui, et al. How far can unsupervised rlvr scale llm training?. In International Conference on Learning Representations, Vol. 2026, p. 14823–14865. Cited by: §1. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Cited by: §5.1, §6.3. Hu et al. (2025) J. Hu, J. K. Liu, H. Xu, and W. Shen Reinforce++: stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262. Cited by: §6.3. Huang et al. (2026) C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu R-zero: self-evolving reasoning llm from zero data. In International Conference on Learning Representations, Vol. 2026, p. 130770–130790. Cited by: §7. Jain et al. (2025) N. Jain, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025, p. 58791–58831. Cited by: §5.1. Jensen (1906) J. L. W. V. Jensen Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica 30 (1), p. 175–193. Cited by: §4.1. Lee et al. (2024a) H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. R. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In Forty-first International Conference on Machine Learning, Cited by: §7. Lee et al. (2024b) N. Lee, T. Wattanawong, S. Kim, K. Mangalam, S. Shen, G. Anumanchipalli, M. Mahoney, K. Keutzer, and A. Gholami Llm2llm: boosting llms with novel iterative data enhancement. In Findings of the Association for Computational Linguistics: ACL 2024, p. 6498–6526. Cited by: §7. Li et al. (2025) P. Li, M. Skripkin, A. Zubrey, A. Kuznetsov, and I. Oseledets Confidence is all you need: few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.06395. Cited by: §1, §7. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, Cited by: §5.1. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: iterative refinement with self-feedback. Advances in neural information processing systems 36, p. 46534–46594. Cited by: §7. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §2. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, Cited by: §5.1. Roy et al. (2025) S. Roy, H. Hajimirsadeghi, M. Zhai, and G. Samei You need reasoning to learn reasoning: the limitations of label-free rl in weak base models. arXiv preprint arXiv:2511.04902. Cited by: §1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2, §5.1. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, p. 8634–8652. Cited by: §7. Tao et al. (2024) Z. Tao, T. Lin, X. Chen, H. Li, Y. Wu, Y. Li, Z. Jin, F. Huang, D. Tao, and J. Zhou A survey on self-evolution of large language models. arXiv preprint arXiv:2404.14387. Cited by: §1, §7. Wang et al. (2026a) R. Wang, W. Huang, Q. Cao, Y. Iwasawa, Y. Matsuo, and J. Guo Self-harmony: learning to harmonize self-supervision and self-play in test-time reinforcement learning. In International Conference on Learning Representations, Vol. 2026, p. 83112–83144. Cited by: §B.4, §1, Figure 2, §5.1, §7. Wang et al. (2024a) T. Wang, I. Kulikov, O. Golovneva, P. Yu, W. Yuan, J. Dwivedi-Yu, R. Y. Pang, M. Fazel-Zarandi, J. Weston, and X. Li Self-taught evaluators. arXiv preprint arXiv:2408.02666. Cited by: §7. Wang et al. (2026b) W. Wang, Y. Wang, K. Chen, and H. Huang Beyond majority voting: towards fine-grained and more reliable reward signal for test-time reinforcement learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 37251–37265. Cited by: §B.3, Figure 2, §5.1, §7. Wang et al. (2023a) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, Cited by: §1, §2, §4.2. Wang et al. (2026c) Y. Wang, Z. Zhang, and R. Wang On the overscaling curse of parallel thinking: system efficacy contradicts sample efficiency. arXiv preprint arXiv:2601.21619. Cited by: §4.1. Wang et al. (2023b) Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 13484–13508. Cited by: §7. Wang et al. (2024b) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: a more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems 37, p. 95266–95290. Cited by: §5.1. Wu et al. (2026) S. Wu, H. Tan, X. Zhang, B. Xia, S. Zhang, X. Qi, B. Yu, and J. Jia Beyond majority voting: self-reflective test-time reinforcement learning for LLM reasoning. In Forty-third International Conference on Machine Learning, Cited by: §B.7, §D.3, §1, Figure 2, §4.2, §5.1, §7. Wu et al. (2025) T. Wu, W. Yuan, O. Golovneva, J. Xu, Y. Tian, J. Jiao, J. E. Weston, and S. Sukhbaatar Meta-rewarding language models: self-improving alignment with llm-as-a-meta-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 11548–11565. Cited by: §7. Xu et al. (2024) C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang WizardLM: empowering large pre-trained language models to follow complex instructions. In International Conference on Learning Representations, Vol. 2024, p. 30745–30766. Cited by: §7. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1. Yu et al. (2026) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: §5.1, §5.1. YU et al. (2026) Z. YU, Z. Su, L. Tao, H. Wang, A. Singh, H. Yu, J. Wang, H. Gao, W. Yuan, J. E. Weston, P. Yu, and J. Xu RESTRAIN: from spurious votes to signals — self-training RL with self-penalization. In The Fourteenth International Conference on Learning Representations, Cited by: §B.5, §5.1. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 57905–57923. Cited by: §1, §7. Zelikman et al. (2024) E. Zelikman, G. R. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. Goodman Quiet-STar: language models can teach themselves to think before speaking. In First Conference on Language Modeling, Cited by: §7. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman Star: bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems 35, p. 15476–15488. Cited by: §7. Zhang et al. (2026a) K. Zhang, Q. Yao, S. Liu, Y. Wang, B. Lai, J. Ye, M. Song, and D. Tao Consistent paths lead to truth: self-rewarding reinforcement learning for llm reasoning. Advances in Neural Information Processing Systems 38, p. 59849–59887. Cited by: §B.2, §1, §5.1, §7. Zhang et al. (2025) Y. Zhang, Z. Zhang, H. Guan, Y. Cheng, Y. Duan, C. Wang, Y. Wang, S. Zheng, and J. He No free lunch: rethinking internal feedback for llm reasoning. arXiv preprint arXiv:2506.17219. Cited by: §1, §5.3. Zhang et al. (2026b) Z. Zhang, J. Zhu, X. Ge, Z. Zhao, X. Li, X. Feng, J. Yao, B. Han, et al. Co-rewarding: stable self-supervised rl for eliciting reasoning in large language models. In International Conference on Learning Representations, Vol. 2026, p. 82520–82558. Cited by: §B.6, §D.2, §4.2, §5.1, §7. Zhao et al. (2026a) A. Zhao, Y. Wu, T. Wu, Q. Xu, Y. Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang Absolute zero: reinforced self-play reasoning with zero data. Advances in Neural Information Processing Systems 38, p. 105816–105879. Cited by: §7. Zhao et al. (2026b) X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song Learning to reason without external rewards. In International Conference on Learning Representations, Vol. 2026, p. 2548–2581. Cited by: §1, §1, §5.1, §7. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §6.3. Zhou et al. (2024) P. Zhou, J. Pujara, X. Ren, X. Chen, H. Cheng, Q. V. Le, D. Zhou, S. Mishra, H. S. Zheng, et al. Self-discover: large language models self-compose reasoning structures. Advances in Neural Information Processing Systems 37, p. 126032–126058. Cited by: §7. Zuo et al. (2026) Y. Zuo, K. Zhang, L. Sheng, S. Qu, G. Cui, X. Zhu, H. Li, X. Long, E. Hua, B. Qi, et al. Ttrl: test-time reinforcement learning. Advances in Neural Information Processing Systems 38, p. 131459–131483. Cited by: §B.1, §D.1, §1, §2, Figure 2, §4.2, §5.1, §5.1, §7. Appendix A Additional Demonstrations of Self-Reward Uncertainty We provide four additional cases following the setup of Figure 2. For each DAPO-14K prompt, we sample one correct and one incorrect focal response from Qwen3-8B-Base. Each focal response is evaluated under M=100M=100 independent group contexts, each containing G−1G-1 newly sampled responses with G=16G=16. We record its reward and group-normalized advantage under the four self-rewarding methods, together with its RLVR advantage under the same context. Figure 9: Self-Reward Uncertainty (Additional Case 2). Figure 10: Self-Reward Uncertainty (Additional Case 3). Figure 11: Self-Reward Uncertainty (Additional Case 4). Figure 12: Self-Reward Uncertainty (Additional Case 5). Appendix B Self-Rewarding Rules of Baselines We describe the self-rewarding rules used by the baselines. Following Section 2, let ⊗G=1,…,GG G=\y_1,…,y_G\ be the sampled response group, where iy_i denotes a complete response and (i)A(y_i) extracts its final answer. Let =(i)i=1G,n(a)=∑i=1G[(i)=a]U= \A(y_i) \_i=1^G, n(a)= _i=1^G 1\! [A(y_i)=a ] (17) denote the set of distinct answers and the frequency of answer a, respectively. In general, each method constructs a reward reference z=f(⊗G)z=f(G G) and assigns ri=ℛ(i∣z)r_i=R(y_i z). Depending on the method, z may be a single pseudo-label, a collection of response-specific pseudo-labels, or structured group information. B.1 TTRL TTRL (Zuo et al., 2026) uses majority voting over the extracted answers. The most frequent answer is selected as the reward reference: zTTRL=fTTRL(⊗G)=argmaxa∈∑j=1G[(j)=a] z_TTRL=f_TTRL(G G)= _a _j=1^G 1\! [A(y_j)=a ] (18) Each response receives a binary reward according to whether its final answer matches this reference: ℛTTRL(i∣zTTRL)=[(i)=zTTRL] R_TTRL\! (y_i z_TTRL )= 1\! [A(y_i)=z_TTRL ] (19) B.2 CoVo CoVo (Zhang et al., 2026a) derives rewards from the internal reasoning trajectory of each response. Specifically, response iy_i is divided into TiT_i reasoning steps, and i,ℓs_i, denotes its intermediate state before step ℓ . For a candidate answer a, CoVo measures its distance from this state by d(i,ℓ,a)=−1|a|∑q=1|a|logπθold(aq∣i,ℓ,a<q).d(s_i, ,a)=- 1|a| _q=1^|a| _ _old\! (a_q _i, ,a_<q ). (20) Let ai=(i)a_i=A(y_i). The consistency and volatility of iy_i are defined as Con(i)=1Ti∑ℓ=0Ti−1[ai=argmina∈d(i,ℓ,a)],Con(y_i)= 1T_i _ =0^T_i-1 1\! [a_i= _a d(s_i, ,a) ], (21) Vol(i)=1Timax(ℓ:ai≠argmina∈d(i,ℓ,a)∪0).Vol(y_i)= 1T_i \! ( \ :a_i≠ _a d(s_i, ,a) \∪\0\ ). (22) Responses sharing the same extracted answer form ℐ(a)=j:(j)=a.I(a)= \j:A(y_j)=a \. (23) CoVo maps each response to i=Con(i)[cos(Vol(i))sin(Vol(i))]v_i=Con(y_i) bmatrix \! (Vol(y_i) )\\ \! (Vol(y_i) ) bmatrix (24) and aggregates these vectors within each answer group. Accordingly, its group-derived reference is zCoVo=fCoVo(⊗G)=ℐ(a),jj∈ℐ(a)a∈ z_CoVo=f_CoVo(G G)= \I(a), \v_j \_j (a) \_a (25) The intrinsic reward of iy_i is the normalized magnitude of the aggregated vector for its answer group: riint=1|ℐ(ai)|‖∑j∈ℐ(ai)j‖2.r_i^int= 1|I(a_i)| \| _j (a_i)v_j \|_2. (26) CoVo additionally assigns a curiosity reward ricurr_i^cur based on the likelihood of transitions between successive reasoning states, together with a KL penalty that prevents a few extremely unlikely tokens from dominating the reward. The final reward is ℛCoVo(i∣zCoVo)=riint+ricur R_CoVo\! (y_i z_CoVo )=r_i^int+r_i^cur (27) The token-level computation of ricurr_i^cur follows the original implementation of CoVo. B.3 SCOPE SCOPE (Wang et al., 2026b) combines step-wise confidence weighting with subgroup-specific pseudo-labels. Suppose response iy_i contains LiL_i reasoning steps, with step ℓ containing Ni,ℓN_i, tokens. Let Pi,ℓ,t(1),…,Pi,ℓ,t(Kc)P_i, ,t^(1),…,P_i, ,t^(K_c) denote the top-KcK_c next-token probabilities at token position t. The response-level confidence is ci=1Li∑ℓ=1Li1Ni,ℓ∑t=1Ni,ℓ(−1Kc∑q=1KclogPi,ℓ,t(q)).c_i= 1L_i _ =1^L_i 1N_i, _t=1^N_i, (- 1K_c _q=1^K_c P_i, ,t^(q) ). (28) SCOPE partitions the G responses into equal-sized subgroups. The subgroup size is dynamically selected through the Pareto criterion in the original method, which balances the agreement between responses and their subgroup labels against the diversity of labels across subgroups. Let ℐ1,…,ℐLI_1,…,I_L denote the resulting partition and let ℓ(i) (i) satisfy i∈ℐℓ(i)i _ (i). For each subgroup ℓ , SCOPE independently draws a bootstrap sample ℬℓB_ from the complete rollout group and applies confidence-weighted voting: zℓ=argmaxa∈∑j∈ℬℓcj[(j)=a].z_ = _a _j _ c_j 1\! [A(y_j)=a ]. (29) Therefore, SCOPE constructs one reward reference for each response according to its subgroup: zSCOPE=fSCOPE(⊗G)=(zℓ(1),…,zℓ(G)) z_SCOPE=f_SCOPE(G G)= (z_ (1),…,z_ (G) ) (30) Writing zSCOPE,i=zℓ(i)z_SCOPE,i=z_ (i), its reward is ℛSCOPE(i∣zSCOPE)=[(i)=zSCOPE,i] R_SCOPE\! (y_i z_SCOPE )= 1\! [A(y_i)=z_SCOPE,i ] (31) The candidate subgroup sizes, bootstrap configuration, and Pareto selection procedure follow the original SCOPE implementation. B.4 Self-Harmony Self-Harmony (Wang et al., 2026a) constructs two views of the same prompt. Besides the original prompt x, the model acts as a Reframer to generate a semantically equivalent prompt ′x . It then samples an additional response group ′⊗G′=1′,…,G′∼πθold⊗G′(⋅∣′).G G = \y _1,…,y _G \ _ _old G (· ). (32) For each candidate answer a, its empirical frequencies in the original and reframed views are p(a)=1G∑i=1G[(i)=a],p′(a)=1G′∑j=1G′[(j′)=a].p(a)= 1G _i=1^G 1\! [A(y_i)=a ], p (a)= 1G _j=1^G 1\! [A(y _j)=a ]. (33) Self-Harmony selects the answer with the largest harmonic mean of these two frequencies: zSH=fSH(⊗G,′⊗G′)=argmaxa2p(a)p′(a)p(a)+p′(a) z_SH=f_SH\! (G G,G G )= _a 2p(a)p (a)p(a)+p (a) (34) The score is defined as zero when p(a)+p′(a)=0p(a)+p (a)=0. The original responses are rewarded by agreement with this shared reference: ℛSH(i∣zSH)=[(i)=zSH] R_SH\! (y_i z_SH )= 1\! [A(y_i)=z_SH ] (35) Self-Harmony also trains the reframing branch using format and diversity rewards. Their prompt templates and implementation details follow the original paper. B.5 RESTRAIN RESTRAIN (YU et al., 2026) retains all distinct answers instead of selecting only the majority answer. Write =a1,…,aMU=\a_1,…,a_M\ and define cm=∑i=1G[(i)=am],pm=cmG.c_m= _i=1^G 1\! [A(y_i)=a_m ], p_m= c_mG. (36) Given the Gaussian shaping function g(p)=exp(−(p−k)22σ2),k∈[0,1],σ>0,g(p)= \! (- (p-k)^22σ^2 ), k∈[0,1], σ>0, (37) RESTRAIN assigns each candidate answer the normalized weight wm=g(pm)∑ℓ=1Mg(pℓ).w_m= g(p_m) _ =1^Mg(p_ ). (38) Its reward reference is therefore a weighted collection of candidate answers: zRES=fRES(⊗G)=(am,wm)m=1M z_RES=f_RES(G G)= \(a_m,w_m) \_m=1^M (39) Unlike methods using a single pseudo-label, RESTRAIN evaluates each response against every candidate answer: ℛRES(i∣zRES)=[(i)=am]m=1M R_RES\! (y_i z_RES )= \ 1\! [A(y_i)=a_m ] \_m=1^M (40) Let ri,m=[(i)=am]r_i,m= 1[A(y_i)=a_m] and let Ai,mA_i,m be the group-normalized advantage computed from rj,mj=1G\r_j,m\_j=1^G. When the majority count cmax=maxmcmc_ = _mc_m falls below a threshold κ, RESTRAIN sets the rewards to zero and subtracts a fixed offset δ from every advantage: r~i,m=ri,m,cmax≥κ,0,cmax<κ,A~i,m=Ai,m,cmax≥κ,Ai,m−δ,cmax<κ. r_i,m= casesr_i,m,&c_ ≥κ,\\ 0,&c_ <κ, cases A_i,m= casesA_i,m,&c_ ≥κ,\\ A_i,m-δ,&c_ <κ. cases (41) The resulting GRPO objectives are combined using the pseudo-label weights: RES(θ)=u∑m=1MwmGRPO(θ,A~i,mi=1G),J_RES(θ)=u_x _m=1^Mw_mJ_GRPO\! (θ;\ A_i,m\_i=1^G ), (42) where u_x is a fixed prompt weight obtained from the self-consistency of a frozen reference model, as defined in the original RESTRAIN implementation. B.6 Co-Rewarding Co-Rewarding (Zhang et al., 2026b) constructs its reward reference using a slowly evolving teacher policy. At training step t, the teacher parameters are updated by θ~t=αtθ~t−1+(1−αt)θold,t, θ_t= _t θ_t-1+(1- _t) _old,t, (43) where αt∈[0,1] _t∈[0,1] is the EMA coefficient. The teacher samples ~⊗G~=~1,…,~G~∼πθ~t⊗G~(⋅∣). G G= \ y_1,…, y_ G \ _ θ_t G(· ). (44) The majority answer in this teacher-generated group becomes the reward reference: zCR=fCR(~⊗G~)=argmaxa∑j=1G~[(~j)=a] z_CR=f_CR\! ( G G )= _a _j=1 G 1\! [A( y_j)=a ] (45) The student responses are evaluated against this reference: ℛCR(i∣zCR)=[(i)=zCR] R_CR\! (y_i z_CR )= 1\! [A(y_i)=z_CR ] (46) B.7 SR-TTRL SR-TTRL (Wu et al., 2026) constructs its pseudo-label through three stages: exploration, summarization, and reflection. Given the sampled group ⊗G=1,…,GG G=\y_1,…,y_G\, the exploration stage first collects the distinct extracted answers =(j)j=1G.U= \A(y_j) \_j=1^G. (47) For each a∈a , RepRep selects a representative response from those producing answer a. The summarization operator SummπθoldSumm_ _ _old then uses the current prompt and this representative response to produce a concise reasoning summary as_a. Finally, ReflectπθoldReflect_ _ _old jointly compares all answer–summary pairs and returns the selected answer as the pseudo-label: a=Summπθold(,Repj∣(j)=a),a∈,=(j)j=1G, _a=Summ_ _ _old (x,Rep \y_j (y_j)=a \ ), a , =\A(y_j)\_j=1^G, (48) zSR=fSR(⊗G)=Reflectπθold(,(a,a)a∈). z_SR=f_SR(G G)=Reflect_ _ _old (x,\(a,s_a)\_a ). Here, RepRep denotes the representative-response selection procedure, SummSumm summarizes the reasoning supporting each candidate answer, and ReflectReflect compares these summarized reasoning paths to select the final reward reference. The exact summarization and reflection prompts follow the original SR-TTRL implementation. Each sampled response is then evaluated against this reference using a binary reward: ℛSR(i∣zSR)=[(i)=zSR]. R_SR (y_i z_SR )= 1 [A(y_i)=z_SR ]. (49) Appendix C Theory of GMAE C.1 Approximation Error of Shared-Pool Estimation Here, we formalize the approximation error between the ideal group-marginalized reward distribution PriP_r_i in Eq.7 and its shared-pool estimate P^ri P_r_i in Eq.8. Let dTVd_TV denote total variation distance and assume that i=supp(Pri)V_i=supp(P_r_i) is finite. Theorem 1 (Approximation Error of Shared-Pool Estimation) For any δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ over the candidate-pool and context sampling, dTV(P^ri,Pri)≤|i|log2+log(4/δ)(12K+G−1G+G′−1).d_TV\! ( P_r_i,P_r_i )≤ |V_i| 2+ (4/δ) ( 1 2K+ G-1G+G -1 ). (50) Proof. Condition on the fixed response iy_i and define N=G+G′−1,m=G−1,q=⌊Nm⌋.N=G+G -1, m=G-1, q= Nm . (51) Let 1,…,Nz_1,…,z_N denote the remaining candidate responses and N,m=S⊆1,…,N:|S|=m S_N,m= \S \1,…,N\:|S|=m \ (52) denote the set of all valid companion-index subsets. For each S∈N,mS∈ S_N,m, define the resulting reward as ρi(S)=ℛ(i∣f(i∪j:j∈S)). _i(S)=R\! (y_i f\! (\y_i\∪\z_j:j∈ S\ ) ). (53) Averaging over all contexts in the candidate pool gives the complete-pool distribution P¯ri=(Nm)−1∑S∈N,mδρi(S). P_r_i^C= Nm^-1 _S∈ S_N,m _ _i(S). (54) If S1,…,SKS_1,…,S_K are sampled uniformly without replacement from N,m S_N,m, the empirical distribution used by GMAE is P^ri=1K∑k=1Kδρi(Sk). P_r_i= 1K _k=1^K _ _i(S_k). (55) By the triangle inequality, dTV(P^ri,Pri)≤ d_TV\! ( P_r_i,P_r_i )≤ dTV(P^ri,P¯ri) d_TV\! ( P_r_i, P_r_i^C ) (56) +dTV(P¯ri,Pri). +d_TV\! ( P_r_i^C,P_r_i ). The first term is the context-subsampling error caused by evaluating only K contexts, while the second is the finite-pool error caused by using N sampled candidate responses. For distributions supported on the finite space iV_i, dTV(P,Q)=sup⊆i|P()−Q()|.d_TV(P,Q)= _A _i |P(A)-Q(A) |. (57) We first bound the context-subsampling error. For any ⊆iA _i, define Xk=[ρi(Sk)∈].X_k^A= 1\! [ _i(S_k) ]. (58) Conditioned on the candidate pool, P^ri() P_r_i(A) is the mean of K values sampled without replacement from the finite population [ρi(S)∈]:S∈N,m\ 1[ _i(S) ]:S∈ S_N,m\, whose population mean is P¯ri() P_r_i^C(A). Hoeffding’s inequality for sampling without replacement therefore gives Pr[|P^ri()−P¯ri()|≥ϵ1|]≤2exp(−2Kϵ12). \! [ | P_r_i(A)- P_r_i^C(A) |≥ _1\, |\,C ]≤ 2 \! (-2K _1^2 ). (59) Applying a union bound over the at most 2|i|2^|V_i| subsets of iV_i yields Pr[dTV(P^ri,P¯ri)≥ϵ1]≤2|i|+1exp(−2Kϵ12). \! [d_TV\! ( P_r_i, P_r_i^C )≥ _1 ]≤ 2^|V_i|+1 \! (-2K _1^2 ). (60) The bound is unconditional because its right-hand side does not depend on the realization of C. We next bound the finite-pool error. For each ⊆iA _i, define the symmetric kernel h(1,…,m)=[ℛ(i∣f(i,1,…,m))∈].h_A(z_1,…,z_m)= 1\! [R\! (y_i f\! (\y_i,z_1,…,z_m\ ) ) ]. (61) Its population mean is p=[h(1,…,m)]=Pri().p_A=E\! [h_A(z_1,…,z_m) ]=P_r_i(A). (62) Moreover, P¯ri()=(Nm)−1∑S∈N,mh(j:j∈S), P_r_i^C(A)= Nm^-1 _S∈ S_N,mh_A(z_j:j∈ S), (63) which is a U-statistic of order m. To derive its concentration, let ΠN _N be the set of all permutations of 1,…,N\1,…,N\. For each π∈ΠNπ∈ _N, divide its first qmqm indices into q disjoint blocks of size m and define Vπ,=1q∑ℓ=1qh(π((ℓ−1)m+1),…,π(ℓm)).V_π,A= 1q _ =1^qh_A\! (z_π(( -1)m+1),…,z_π( m) ). (64) Every m-subset appears equally often among these blocks over all permutations, so P¯ri()=1N!∑π∈ΠNVπ,. P_r_i^C(A)= 1N! _π∈ _NV_π,A. (65) For any λ>0λ>0, Jensen’s inequality gives exp[λ(P¯ri()−p)] \! [λ ( P_r_i^C(A)-p_A ) ] (66) ≤1N!∑π∈ΠNexp[λ(Vπ,−p)]. ≤ 1N! _π∈ _NE \! [λ (V_π,A-p_A ) ]. For a fixed permutation, the q blocks are disjoint and hence their kernel evaluations are independent variables in [0,1][0,1], each with mean p_A. Hoeffding’s lemma therefore implies exp[λ(Vπ,−p)]≤exp(λ28q).E \! [λ (V_π,A-p_A ) ]≤ \! ( λ^28q ). (67) Combining this with Eq.66 and applying the Chernoff bound gives Pr[P¯ri()−p≥ϵ2]≤infλ>0exp(−λϵ2+λ28q)=exp(−2qϵ22), \! [ P_r_i^C(A)-p_A≥ _2 ]≤ _λ>0 \! (-λ _2+ λ^28q )= \! (-2q _2^2 ), (68) where the minimum is attained at λ=4qϵ2λ=4q _2. Applying the same argument to the lower tail yields Pr[|P¯ri()−Pri()|≥ϵ2]≤2exp(−2qϵ22). \! [ | P_r_i^C(A)-P_r_i(A) |≥ _2 ]≤ 2 \! (-2q _2^2 ). (69) A union bound over all subsets of iV_i then gives Pr[dTV(P¯ri,Pri)≥ϵ2]≤2|i|+1exp(−2qϵ22). \! [d_TV\! ( P_r_i^C,P_r_i )≥ _2 ]≤ 2^|V_i|+1 \! (-2q _2^2 ). (70) Let Lδ=|i|log2+log(4/δ)L_δ=|V_i| 2+ (4/δ) (71) and choose ϵ1=Lδ2K,ϵ2=Lδ2q. _1= L_δ2K, _2= L_δ2q. (72) By Eq.60 and Eq.70, each error exceeds its corresponding bound with probability at most δ/2δ/2. Thus, with probability at least 1−δ1-δ, dTV(P^ri,Pri)≤Lδ(12K+12q).d_TV\! ( P_r_i,P_r_i )≤ L_δ ( 1 2K+ 1 2q ). (73) Finally, since N/m≥1N/m≥ 1, q=⌊Nm⌋≥N2m,q= Nm ≥ N2m, (74) and therefore 12q≤mN=G−1G+G′−1. 1 2q≤ mN= G-1G+G -1. (75) Substituting this inequality into Eq.73 proves Eq.50. □ C.2 Uncertainty-Aware Advantage Scale For prompt b, let P^b=⨂i=1GP^rb,i P_b= _i=1^G P_r_b,i (76) denote the joint reward distribution. For a reward realization =(v1,…,vG)∼P^bv=(v_1,…,v_G) P_b, define its normalized advantage vector as b()=(v1−μσ,…,vG−μσ),σ>0,,σ=0.a_b(v)= cases ( v_1- _v _v,…, v_G- _v _v ),& _v>0,\\[8.0pt] 0,& _v=0. cases (77) The corresponding GMAE advantage vector is bgmae=[b()].A_b^gmae=E_v [a_b(v) ]. (78) Since every b()a_b(v) is zero-centered, ∑i=1GAb,igmae=0 _i=1^GA_b,i^gmae=0, and the prompt-wise GMAE scale satisfies (sbgmae)2=Var(Ab,igmaei=1G)=1G‖bgmae‖22. (s_b^gmae )^2=Var (\A_b,i^gmae\_i=1^G )= 1G \|A_b^gmae \|_2^2. (79) Let v and ′v be independent draws from P^b P_b. We define the context-induced reward uncertainty as b=Pr(σ=0)+12G,′[‖b()−b(′)‖22].U_b= _v\! ( _v=0 )+ 12GE_v,v [ \|a_b(v)-a_b(v ) \|_2^2 ]. (80) The first term captures reward realizations with no relative reward information, while the second measures disagreement between relative reward assignments across realizations. Theorem 2 (Reward Uncertainty and GMAE Scale) For every prompt b, (sbgmae)2=1−b. (s_b^gmae )^2=1-U_b. (81) Consequently, greater context-induced reward uncertainty yields a smaller prompt-wise GMAE scale. Proof. For any non-degenerate reward realization, group-wise normalization gives 1G‖b()‖22=1G∑i=1G(vi−μσ)2=1. 1G \|a_b(v) \|_2^2= 1G _i=1^G ( v_i- _v _v )^2=1. (82) When σ=0 _v=0, we assign b()=a_b(v)=0. Therefore, 1G[‖b()‖22]=1−Pr(σ=0). 1GE_v [ \|a_b(v) \|_2^2 ]=1- _v\! ( _v=0 ). (83) Because v and ′v are independent and identically distributed, 12G,′[‖b()−b(′)‖22] 12GE_v,v [ \|a_b(v)-a_b(v ) \|_2^2 ] (84) =1G[‖b()‖22]−1G‖[b()]‖22 = 1GE_v [ \|a_b(v) \|_2^2 ]- 1G \|E_v [a_b(v) ] \|_2^2 =1−Pr(σ=0)−(sbgmae)2. =1- _v\! ( _v=0 )- (s_b^gmae )^2. Rearranging Eq.84 gives (sbgmae)2 (s_b^gmae )^2 =1−[Pr(σ=0) =1- [ _v\! ( _v=0 ) (85) +12G,′[‖b()−b(′)‖22]] + 12GE_v,v [ \|a_b(v)-a_b(v ) \|_2^2 ] ] =1−b, =1-U_b, which proves the theorem. □ Theorem 2 explains why advantage signs matter. When a response is favored in some reward realizations but disfavored in others, its normalized advantage changes sign, increasing the disagreement term in Eq.80. Marginalization then causes stronger cancellation and produces a smaller scale, leading to more conservative policy updates. Prompt-level calibration would independently restore every prompt to its non-marginalized scale and remove these uncertainty-dependent differences. In contrast, batch-level calibration applies one shared factor λ, so s~bgmaes~b′gmae=λsbgmaeλsb′gmae=sbgmaesb′gmae, s_b^gmae s_b ^gmae= λ s_b^gmaeλ s_b ^gmae= s_b^gmaes_b ^gmae, (86) preserving the relative uncertainty-aware scales across prompts. Appendix D Detailed Implementation of GMAE Instantiation This section details the three GMAE instantiations introduced in Section 4.2. All three use the binary reward ℛbin(i∣z)=[(i)=z],R_bin(y_i z)= 1\! [A(y_i)=z ], (87) where z is the reward reference constructed by the corresponding self-rewarding rule. Given K reward realizations rikk=1K\r_i^k\_k=1^K for response iy_i, its empirical reward distribution is pi=1K∑k=1Krik,P^ri=piδ1+(1−pi)δ0.p_i= 1K _k=1^Kr_i^k, P_r_i=p_i _1+(1-p_i) _0. (88) For =(v1,…,vG)∈0,1Gv=(v_1,…,v_G)∈\0,1\^G, define w()=∏j=1Gpjvj(1−pj)1−vj,hi()=vi−μσ,σ>0,0,σ=0,w_p(v)= _j=1^Gp_j^v_j(1-p_j)^1-v_j, h_i(v)= cases v_i- _v _v,& _v>0,\\[6.0pt] 0,& _v=0, cases (89) where μ=1G∑j=1Gvj,σ=1G∑j=1G(vj−μ)2. _v= 1G _j=1^Gv_j, _v= 1G _j=1^G(v_j- _v)^2. (90) The marginalized advantage can therefore be computed exactly as Aigmae=∑∈0,1Gw()hi().A_i^gmae= _v∈\0,1\^Gw_p(v)h_i(v). (91) The first reward realization is produced by the original group used by the corresponding base method and provides AibaseA_i^base for calibration. After processing all prompts in a batch, we apply the batch-level calibration in Eq.13. Each instantiation requires G′G auxiliary rollouts beyond its base method. We therefore focus below on the additional operations performed after rollout generation. D.1 GMAE0: Standard Voting GMAE0 instantiates GMAE with the majority-voting rule used by TTRL (Zuo et al., 2026). For any response group ℬB, its reward reference is f0(ℬ)=argmaxa∑∈ℬ[()=a].f_0(B)= _a _y 1\! [A(y)=a ]. (92) The original rollout group is used as the first realization. For each focal response, the remaining K−1K-1 contexts are sampled uniformly without replacement from the shared candidate pool. Algorithm 1 GMAE0: Standard Voting Input: Prompt x, policy πθold _ _old, group sizes G,G′G,G , and number of contexts K Output: Advantages Aigmaei=1G\A_i^gmae\_i=1^G Sample =1,…,G+G′C=\y_1,…,y_G+G \ from πθold(⋅∣) _ _old(· ); Set ⊗G=1,…,GG G=\y_1,…,y_G\ and z1=f0(⊗G)z^1=f_0(G G); for i=1,…,Gi=1,…,G do Set ri1=ℛbin(i∣z1)r_i^1=R_bin(y_i z^1); Sample K−1K-1 distinct contexts ikk=2K\S_i^k\_k=2^K from ∖iC \y_i\, with |ik|=G−1|S_i^k|=G-1; for k=2,…,Kk=2,…,K do Set ℬik=i∪ikB_i^k=\y_i\ _i^k; Compute zik=f0(ℬik)z_i^k=f_0(B_i^k); Set rik=ℛbin(i∣zik)r_i^k=R_bin(y_i z_i^k); end for Compute pi=K−1∑k=1Krikp_i=K^-1 _k=1^Kr_i^k; end for Compute Aigmaei=1G\A_i^gmae\_i=1^G using Eq.91; return Aigmaei=1G\A_i^gmae\_i=1^G; Additional Computation. Compared with TTRL, GMAE0 evaluates K−1K-1 additional group contexts for each focal response. This adds G(K−1)G(K-1) majority-voting operations and the same number of binary reward evaluations. Since each vote scans G responses, these operations require (G2(K−1))O(G^2(K-1)) scalar computation. Exact distribution-based advantage estimation further enumerates at most 2G2^G joint binary reward realizations, requiring (G2G)O(G2^G) scalar computation. These operations require neither additional model generation nor backpropagation. D.2 GMAECR: Co-Refinement GMAECR follows Co-Rewarding (Zhang et al., 2026b). At training step t, its co-teacher parameters are updated as θ~t=αtθ~t−1+(1−αt)θold,t, θ_t= _t θ_t-1+(1- _t) _old,t, (93) where αt∈[0,1] _t∈[0,1] is the teacher update coefficient. For a co-teacher response group ℬ~ B, the reward reference is fCR(ℬ~)=argmaxa∑~∈ℬ~[(~)=a].f_CR( B)= _a _ y∈ B 1\! [A( y)=a ]. (94) The reward reference depends only on the co-teacher group. Therefore, the same K reward references can be reused for all G main responses. Algorithm 2 GMAECR: Co-Refinement Input: Prompt x, parameters θold,t _old,t and θ~t−1 θ_t-1, coefficient αt _t, group sizes G,G′G,G , and number of contexts K Output: Advantages Aigmaei=1G\A_i^gmae\_i=1^G Update θ~t θ_t using Eq.93; Sample ⊗G=ii=1GG G=\y_i\_i=1^G from πθold,t(⋅∣) _ _old,t(· ); Sample ~=~jj=1G+G′ C=\ y_j\_j=1^G+G from πθ~t(⋅∣) _ θ_t(· ); Set ℬ~1=~jj=1G B^1=\ y_j\_j=1^G; Sample K−1K-1 distinct groups ℬ~kk=2K\ B^k\_k=2^K from ~ C, with |ℬ~k|=G| B^k|=G; for k=1,…,Kk=1,…,K do Compute zCRk=fCR(ℬ~k)z_CR^k=f_CR( B^k); end for for i=1,…,Gi=1,…,G do for k=1,…,Kk=1,…,K do Set rik=ℛbin(i∣zCRk)r_i^k=R_bin(y_i z_CR^k); end for Compute pi=K−1∑k=1Krikp_i=K^-1 _k=1^Kr_i^k; end for Compute Aigmaei=1G\A_i^gmae\_i=1^G using Eq.91; return Aigmaei=1G\A_i^gmae\_i=1^G; Additional Computation. The co-teacher update is inherited from Co-Rewarding and therefore introduces no GMAE-specific computation. Compared with Co-Rewarding, GMAECR performs K−1K-1 additional majority votes over groups of G co-teacher responses. Because each reward reference is shared across all main responses, it also adds G(K−1)G(K-1) binary reward evaluations. Together with exact advantage estimation, the additional scalar complexity is (G(K−1)+G2G).O\! (G(K-1)+G2^G ). (95) D.3 GMAESR: Self-Refinement GMAESR follows the self-reflective reward construction of SR-TTRL (Wu et al., 2026). Instead of independently performing self-reflection for every group context, it first ranks all candidate answers in the shared pool and then reuses this ranking across contexts. Given the shared candidate pool =1,…,G+G′C=\y_1,…,y_G+G \, let =(j)j=1G+G′U_C= \A(y_j) \_j=1^G+G (96) denote its set of distinct answers. For each a∈a _C, GMAESR selects a representative response and summarizes its reasoning: a=Summπθold(,Repj∈∣(j)=a).s_a=Summ_ _ _old (x,Rep \y_j (y_j)=a \ ). (97) It then performs self-reflection once over all answer-summary pairs, producing a ranking ≻SR _SR over U_C: ≻SR=Reflectπθold(,(a,a)a∈). _SR=Reflect_ _ _old (x, \(a,s_a) \_a _C ). (98) Here, a≻SRa′a _SRa means that a is ranked above a′a by self-reflection. The representative-selection rule, summary prompt, and reflection prompt follow the original SR-TTRL implementation. For the k-th group context of response iy_i, define ℬik=i∪ik,ik=()∣∈ℬik.B_i^k=\y_i\ _i^k, _i^k= \A(y) _i^k \. (99) Its reward reference is the highest-ranked answer present in this context: zSR,ik=max≻SRik,rik=ℛbin(i∣zSR,ik).z_SR,i^k= _ _SRU_i^k, r_i^k=R_bin (y_i z_SR,i^k ). (100) Algorithm 3 GMAESR: Self-Refinement Input: Prompt x, policy πθold _ _old, group sizes G,G′G,G , and number of contexts K Output: Advantages Aigmaei=1G\A_i^gmae\_i=1^G Sample =1,…,G+G′C=\y_1,…,y_G+G \ from πθold(⋅∣) _ _old(· ); Set ⊗G=1,…,GG G=\y_1,…,y_G\; Construct the distinct-answer set U_C; foreach a∈a _C do Select arep=Repj∈:(j)=ay_a^rep=Rep\y_j :A(y_j)=a\; Compute a=Summπθold(,arep)s_a=Summ_ _ _old(x,y_a^rep); end foreach Compute the global answer ranking ≻SR _SR using Eq.98; for i=1,…,Gi=1,…,G do Set i1=⊗G∖iS_i^1=G G \y_i\; Sample K−1K-1 additional distinct contexts ikk=2K\S_i^k\_k=2^K from ∖iC \y_i\, with |ik|=G−1|S_i^k|=G-1; for k=1,…,Kk=1,…,K do Set ℬik=i∪ikB_i^k=\y_i\ _i^k; Extract the answers ik=():∈ℬikU_i^k=\A(y):y _i^k\; Select zSR,ik=max≻SRikz_SR,i^k= _ _SRU_i^k; Set rik=ℛbin(i∣zSR,ik)r_i^k=R_bin(y_i z_SR,i^k); end for Compute pi=K−1∑k=1Krikp_i=K^-1 _k=1^Kr_i^k; end for Compute Aigmaei=1G\A_i^gmae\_i=1^G using Eq.91; return Aigmaei=1G\A_i^gmae\_i=1^G; Additional Computation. Let UbaseU_base and UpoolU_pool denote the numbers of distinct answers in the original rollout group and the shared candidate pool, respectively. SR-TTRL summarizes UbaseU_base answer classes and performs one reflection, whereas GMAESR summarizes UpoolU_pool classes and also performs only one reflection. Therefore, GMAESR adds Upool−Ubase≤G′U_pool-U_base≤ G summary operations and does not increase the number of reflection operations, although the single reflection covers a larger candidate set. Once the global ranking is obtained, each additional context requires only extracting its answers and selecting the highest-ranked one. Compared with SR-TTRL, this introduces G(K−1)G(K-1) context-specific ranking lookups and binary reward evaluations. A direct implementation scans at most G responses per context, giving (G2(K−1))O(G^2(K-1)) scalar computation. Exact distribution-based advantage estimation further requires (G2G)O(G2^G) scalar operations. Thus, GMAESR reuses a single self-reflective ranking across all contexts rather than repeating the expensive reflection procedure K times. Appendix E Full Scalability Analysis Results Table 4: Scalability on RL backbone. Benchmark-wise Mean@16 results on Qwen3-8B-Base using GSPO as the RL backbone. Method Mathematics Science Knowledge Coding Average MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode TTRL 78.8 45.3 14.4 11.3 7.2 40.3 66.6 22.3 35.8 Self-Harmony 78.4 43.2 15.2 11.0 6.7 40.9 70.8 22.8 36.1 Co-Reward 80.3 47.7 15.5 12.7 7.9 42.0 68.7 23.2 37.3 SR-TTRL 79.7 47.1 14.7 12.6 8.0 41.4 67.6 22.9 36.8 GMAE0 (Ours) 81.9 48.7 17.3 14.0 9.0 42.1 67.9 24.2 38.1 GMAECR_CR (Ours) 83.3 50.0 17.6 15.0 9.7 43.8 69.7 25.0 39.3 GMAESR_SR (Ours) 82.4 49.8 17.5 14.5 9.1 43.2 68.5 24.8 38.7 Table 5: Scalability on RL backbone. Benchmark-wise Mean@16 results on Qwen3-8B-Base using REINFORCE++ as the RL backbone. Method Mathematics Science Knowledge Coding Average MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode TTRL 75.9 40.1 10.7 9.7 5.7 36.8 64.3 19.4 32.8 Self-Harmony 75.5 39.6 11.1 10.4 6.3 38.1 68.3 19.6 33.6 Co-Reward 76.7 41.5 11.9 11.5 6.6 37.7 64.5 20.4 33.9 SR-TTRL 76.9 40.8 11.7 10.8 6.1 37.4 65.0 20.1 33.6 GMAE0 (Ours) 79.8 45.5 13.8 12.9 9.3 40.2 66.7 22.8 36.4 GMAECR_CR (Ours) 80.5 47.8 14.9 14.2 10.2 41.0 66.8 24.3 37.5 GMAESR_SR (Ours) 80.0 47.1 14.4 13.7 9.2 41.0 67.4 23.7 37.1 Table 6: Scalability on training dataset. Mean@16 results on Qwen3-8B-Base trained with the Open-RS dataset. Method Mathematics Science Knowledge Coding Average MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode TTRL 76.4 42.7 13.5 10.4 7.6 38.8 62.1 20.3 34.0 Self-Harmony 75.6 41.6 13.7 10.8 7.2 39.3 69.3 21.4 34.9 Co-Reward 77.3 43.2 14.1 11.6 7.8 40.4 64.9 22.7 35.3 SR-TTRL 77.0 43.5 13.8 11.4 7.8 40.5 65.7 22.2 35.2 GMAE0 (Ours) 77.8 44.0 14.9 12.3 8.3 41.2 67.9 22.9 36.2 GMAECR_CR (Ours) 79.3 46.5 16.3 13.7 9.0 42.3 68.5 23.1 37.3 GMAESR_SR (Ours) 78.6 46.1 16.2 13.0 8.8 40.3 67.2 22.9 36.6 Table 7: Scalability on training dataset. Mean@16 results on Qwen3-8B-Base trained with the MATH-8K dataset. Method Mathematics Science Knowledge Coding Average MATH500 AMC AIME24 AIME25 AIME26 GPQA MMLU-Pro LiveCode TTRL 75.5 41.2 12.3 9.5 6.7 37.5 63.2 19.5 33.2 Self-Harmony 76.4 41.5 12.0 9.9 6.5 37.3 61.3 19.4 33.0 Co-Reward 77.0 42.3 13.6 8.9 7.1 38.2 66.0 21.2 34.3 SR-TTRL 77.1 42.3 13.6 10.4 7.8 38.6 63.9 21.0 34.3 GMAE0 (Ours) 79.1 43.7 15.0 11.1 8.3 40.1 65.1 21.2 35.5 GMAECR_CR (Ours) 80.3 44.0 15.7 12.0 9.0 41.0 64.4 22.9 36.2 GMAESR_SR (Ours) 80.8 43.5 15.4 12.3 9.5 41.2 66.1 23.2 36.5