Paper deep dive
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Hexuan Deng, Zihao Yan, Xuebo Liu, Shuo Nie, Yue Wang, Chen Wang, Zhaohua Zhang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Min Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/10/2026, 2:34:28 AM
Summary
The paper introduces GRPODropout, a method to mitigate policy entropy collapse in Reinforcement Learning (RL) for Large Language Models (LLMs). It proposes selectively removing high-probability positive-advantage rollouts before policy updates and recentering the remaining advantages. Theoretical analysis shows that such rollouts contribute negatively to policy entropy, and their removal improves exploration and accuracy while using fewer samples.
Entities (6)
Relation Signals (5)
GRPODropout → isbasedon → GRPO
confidence 95% · Reinforcement learning (RL) methods such as GRPO... To address this, we propose GRPODropout
GRPODropout → mitigates → Policy Entropy Collapse
confidence 95% · GRPODropout: Less is More for Online Reinforcement Learning Rollouts... entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates.
High-probability positive-advantage rollouts → causes → Policy Entropy Collapse
confidence 90% · high-probability positive rollouts... can both reduce policy entropy... further reinforcement may provide limited additional information while accelerating entropy collapse.
GRPODropout → improves → Accuracy
confidence 90% · Experiments show higher accuracy than original GRPO
GRPODropout → increases → Actor Entropy
confidence 90% · higher actor entropy while using fewer rollout samples for updates
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.11854v1
- Canonical: https://arxiv.org/abs/2610.11854v1
Trouble viewing inline? Open PDF directly →
Full Text
90,515 characters extracted from source content.
Expand or collapse full text
GRPODropout: Less is More for Online Reinforcement Learning Rollouts Hexuan Deng Zihao Yan Xuebo Liu Shuo Nie Yue Wang Chen Wang †thanks: ˜Xuebo Liu is the corresponding author. Affiliation: Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) Affiliation: Beijing Zhongguancun Academy Affiliation: XinzhuAI Zhaohua Zhang Tianwen Jiang Qiuyong Xiao Jihong Zhang Min Zhang Affiliation: Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) Abstract Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating “less is more.” This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at https://github.com/hexuandeng/GRPODropout/. 1 Introduction Online RL substantially improves the reasoning ability of large language models (Shao et al., 2024; Yu et al., 2025). However, these methods often suffer from policy entropy collapse during training, reducing sampling diversity and weakening exploration of alternative reasoning paths (Cui et al., 2025; Jin et al., 2026). Because online RL relies on newly sampled rollouts for subsequent updates, this reduced diversity can limit opportunities to discover new solutions and improve further (Yue et al., 2025). This highlights a fundamental challenge in RL: the exploration–exploitation trade-off (Auer et al., 2002). To address this problem, existing approaches follow two routes. The first modifies the RL algorithm through rewards or advantages (Hao et al., 2025; Jin et al., 2026), clipping bounds (Yu et al., 2025; Park et al., 2025), or temperature and sampling control (Wang et al., 2025a). The second focuses on data usage at the token level, targeting high-entropy tokens and critical branching points for masking or reweighting (Du et al., 2025; Cui et al., 2025; Wang et al., 2025b), branching (Li et al., 2025; Zhao et al., 2026), or calibration (Yang et al., 2026). These methods exploit differences in token importance to mitigate entropy collapse. Here, we argue that changing rollout usage alone can mitigate entropy collapse. However, existing work still offers limited theoretical and methodological guidance for selecting complete rollouts to balance exploration and exploitation. To bridge this gap, we propose GRPODropout, which selectively removes rollouts to improve performance, mitigate entropy collapse, and preserve exploration in the trained model. Specifically, under our rollout-level theoretical analysis, we find that high-probability positive rollouts and low-probability negative rollouts can both reduce policy entropy, with the former having a more pronounced effect. Intuitively, high-probability positive rollouts are already well represented under the current policy; further reinforcement may provide limited additional information while accelerating entropy collapse. We therefore selectively remove these rollouts and recenter the retained advantages using old-policy probability weights to correct the imbalance introduced by deletion. Across three models and ten benchmarks, the average score improves over original GRPO by 1.55 percentage points. Despite using 12.1% fewer retained rollouts in policy updates on average in groups with nonzero advantages, training and held-out actor entropy are 446.1% and 256.0% higher than original GRPO, respectively, at the end of training, demonstrating better results with fewer update rollouts. Our contributions are: • A rollout-level analysis of data importance. We extend the theoretical analysis to the rollout level to motivate our approach, and use experiments to further characterize which rollouts can reduce policy entropy, providing insight into how rollouts are used during RL training. • Rollout filtering and recentering. We design a data-selection algorithm to filter potentially harmful rollouts and recenter retained advantages to correct the imbalance introduced by deletion. • Empirical evidence across model sizes. We evaluate accuracy and entropy across three model sizes under reduced update-level rollout usage. 2 Related Work Token-level entropy interventions. One line of work stabilizes policy entropy by identifying less informative tokens and attenuating their updates. UloRL (Du et al., 2025) identifies high-probability tokens in successful rollouts as already well learned and masks them when policy entropy falls below a target. RSI-S (Lv et al., 2026) extends this idea by normalizing sampled-token surprisal by local entropy. Cui et al. (2025) analyze first-order entropy changes and use this analysis to motivate Clip-Cov and KL-Cov, which clip or penalize updates on high-covariance tokens. Wang et al. (2025b) show that updating only the 20% highest-entropy tokens can match or outperform full-token training. Later methods identify critical branching points to construct more diverse training data. CURE (Li et al., 2025) and ROSE (Zhao et al., 2026) branch at positions where likely next tokens are semantically distinct and train on the resulting trees using segment-level advantages. ERPO (Yu et al., 2026) detects Critical Decision Pivots using group-normalized entropy and combines entropy gating, reasoning progress, and final outcomes into token-level advantages. PAEC (Yang et al., 2026) combines normalized entropy with top-two competition to construct a detached soft position mask. STARE (Luo et al., 2026) derives an advantage–surprisal four-quadrant structure from first-order entropy variation and uses target-entropy feedback to regulate high-surprisal tokens. ICT (Feng et al., 2026) selects positions whose full next-token distributions exhibit large Jensen–Shannon divergences, capturing both high- and low-entropy forks. These token-level interventions leave open how entropy contributions can guide the selective removal of complete rollouts sampled for each prompt. We address this gap through rollout-level entropy analysis and principled rollout selection, as unmasked tokens may still reinforce existing paths and limit exploration (Du et al., 2025; Zhu et al., 2025). Entropy-aware reinforcement learning algorithms. Another line addresses entropy collapse by modifying the training signal or optimization rule. One group modifies rewards or advantages. STEER (Hao et al., 2025) estimates the first-order entropy change induced by each token update and downweights updates that sharply decrease or increase entropy. Jin et al. (2026) identify positive-advantage tokens as the main source of entropy reduction and reweight them according to the training stage or current entropy. HAPO (He et al., 2026) reallocates advantages toward high-entropy positions through capacity-guided credit assignment. Papo (Zhang et al., 2026) predicts the update direction from the advantage sign, sampled-token probability, and local distribution, and regulates opposing entropy polarities by training phase. A second group modifies the optimization rule directly. DAPO (Yu et al., 2025) introduces Clip-Higher to relax the upper importance-ratio bound. Clip-Low/High (Park et al., 2025) show the underlying optimizer bias: lower clipping tends to increase entropy, whereas upper clipping decreases it. SCOPE-RL (Wang et al., 2025a) uses temperature-adaptive positive samples and an auxiliary regularizer to control policy entropy. These methods mitigate entropy collapse through algorithmic changes to rewards, advantages, clipping, or regularization. We find that data-level interventions alone can also mitigate entropy collapse, through rollout selection and advantage recentering while preserving the RL objective form. 3 Method We find that not all rollouts benefit model training and demonstrate that removing a subset of the data can improve performance. We therefore propose GRPODropout, as illustrated in Figure 1. Each subsection presents a local theoretical analysis to motivate the corresponding design choice, while experiments evaluate the complete method. Section 3.2 develops rollout-level first-order entropy analysis and candidate deletion; Section 3.3 motivates positive-only deletion through a conditional comparison; Section 3.4 analyzes deletion-induced drift of zero-advantage sequences, derives recentering, and presents the full procedure. Figure 1: GRPODropout procedure. Filter removes only selected positive-advantage rollouts; Re-center applies probability-weighted advantage recentering. Other GRPO updates are unchanged. 3.1 Preliminaries Problem Formulation. A rollout is a complete output sequence generated by the policy for a given prompt. For a fixed prompt x, let πold _old denote the policy before the policy update, and let x=iiY_x=\y_i\_i be the set of all possible complete output sequences generated from x under πold _old. For this fixed prompt, its Shannon entropy is H(πold)=−∑i∈xπold(i∣x)logπold(i∣x)H( _old)=- [b]Σ _y_i _x _old(y_i x) _old(y_i x). We seek to optimize task return while mitigating policy entropy collapse and preserving exploration. For policy updates, we independently sample a training group of G complete rollouts from this space according to the old policy: (1),…,(G)∼i.i.d.πold(⋅∣x) [t]y^(1),…, [t]y^(G) [t] i.i.d. _old(· x). Here i indexes the space of complete output sequences, whereas g∈1,…,Gg∈\1,…,G\ indexes the sampled group. Rollout g is (g)=(y1(g),…,yTg(g)) [t]y^(g)=( [t]y_1^(g),…, y_T_g^(g)), with length TgT_g, giving sequence probability πold((g)∣x)=∏t=1Tgπold(yt(g)∣x,<t(g)) _old(y^(g) x)= _t=1^T_g _old(y_t^(g) x,y_<t^(g)). Policy optimization. Let πθ _θ be the policy to optimize, with parameters θ. Given a rollout reward R(x,)R(x,y), let A(x,)A(x,y) denote advantage relative to a baseline b(x)=R¯(x,∗)b(x)= R(x,*), e.g., R(x,)−b(x)R(x,y)-b(x). Using the old policy for sampling, we write the advantage objective with reference-policy regularization (Stiennon et al., 2020; Ouyang et al., 2022) and its sampled surrogate as maxθ∼πold(⋅∣x)[πθ(∣x)πold(∣x)A(x,)]−βKL(πθ(⋅∣x)∥πref(⋅∣x)) _θ \E_y _old(· x)\! [ _θ(y x) _old(y x)A(x,y) ]-β\,KL\! ( _θ(· x)\,\|\, _ref(· x) ) \ (1) ↝maxθ1G∑g=1GA(g)πθ((g)∣x)πold((g)∣x)−βKL(πθ(⋅∣x)∥πref(⋅∣x)). _θ \ 1G _g=1^GA^(g) _θ(y^(g) x) _old(y^(g) x)-β\,KL\! ( _θ(· x)\,\|\, _ref(· x) ) \. Here πref _ref is a fixed reference policy, β≥0β≥ 0 is its regularization coefficient, and KLKL denotes Kullback–Leibler divergence. The probability-ratio identity assumes that the candidate policy is supported within the old policy and the advantage expectation exists. The arrow denotes Monte Carlo approximation using sampled group of rollouts and advantages A(g)A^(g). 3.2 Rollout-Level Entropy Effect and Candidate Deletion In this subsection, we analyze which rollouts decrease policy entropy under a local update model. Motivated by this analysis and intuition, we design a filtering rule to mitigate entropy collapse. Theoretical motivation: rollout updates and entropy collapse. Reward optimization can improve return while increasing the risk of entropy collapse and loss of exploration. To identify which rollouts decrease entropy, we first characterize each rollout’s first-order entropy contribution and then give the criterion for the overall entropy change. To extend token-level entropy analysis to complete rollouts, we analyze a local KL-proximal policy update following Abdolmaleki et al. (2018). With the sampled group and its advantages fixed, let π(η)π^(η) be the updated policy at step size η, with π(0)=πoldπ^(0)= _old. Under this model, we obtain the following results: Proposition 1 (First-order entropy contribution of a single rollout). Fix a prompt x and adopt the local update model and assumptions in Appendix A.1. Let sg=−logπold((g)∣x)s_g=- _old(y^(g) x) be the surprisal of sampled rollout g under the old policy. Its additive first-order entropy contribution to the whole-group update is hg=A(g)G(sg−H(πold)).h_g= A^(g)G (s_g-H( _old) ). (2) Proposition 2 (Entropy derivative from sampled rollouts). Under the setting of Proposition 1, assume ∑g=1GA(g)=0 _g=1^GA^(g)=0. Using the surprisals sgs_g defined above, let s¯=G−1∑g=1Gsg s=G^-1 Σ _g=1^G s_g. For the fixed sampled group, the proximal update has entropy derivative dH(π(η))dη|η=0=Covg=1G(A(g),sg)=1G∑g=1GA(g)(sg−s¯). . dH(π^(η))dη |_η=0=Cov_g=1^G\! (A^(g),s_g )= 1G _g=1^GA^(g) (s_g- s ). (3) The proofs are in Appendix A.1. Proposition 1 shows that a positive-advantage rollout with sg<H(πold)s_g<H( _old) (relatively high probability) and a negative-advantage rollout with sg>H(πold)s_g>H( _old) (relatively low probability) both have negative first-order entropy contributions. Conversely, low-probability positives and high-probability negatives contribute to entropy increase. Intuitively, further reinforcing already likely positive-advantage rollouts or suppressing already unlikely negative-advantage rollouts may offer limited gains in reasoning ability. Repeatedly prioritizing such updates may instead accelerate entropy collapse and weaken exploration. Proposition 2 shows that the proximal update decreases entropy to first order when rollout advantages and log-probabilities are positively correlated, which is often indeed tend to be positively correlated in original models (Kadavath et al., 2022; Zenn and Geiping, 2026), providing an intuitive explanation for entropy collapse during reward optimization. Further, during RL training, by further increasing the likelihood of positive-advantage rollouts and decreasing that of negative-advantage ones, reward optimization can further reinforce this positive association and intensify entropy reduction over training. Method: candidate filtering. Based on the preceding analysis, we design a rollout-selection method that uses Proposition 1 to identify candidates with negative estimated entropy contributions and Proposition 2 to set the deletion threshold. Specifically, we reuse hgh_g from Eq. (2), estimating H(πold)H( _old) by the Monte Carlo mean s¯ s of the original sampled group. In the filtering rule below, hgh_g denotes the contribution computed with this substitution. Using Proposition 2, we employ a greedy algorithm to select the largest retained set that ensures the estimated entropy does not decrease: ℐkeep∈argmaxℐ⊆1,…,G|ℐ|s.t.Covg=1G(A(g)g∈ℐ,sg)≥0.I_keep∈ _I \1,…,G\|I| .t. _g=1^G\! (A^(g)1\g \,s_g )≥ 0. (4) Here ℐkeepI_keep is the set of rollout indices selected for retention. The covariance is evaluated over the original G rollouts, with the indicator setting advantages of the deleted rollouts to zero. To solve this argmax, we start with all rollouts and greedily remove those with the most negative estimated contributions, until the constraint in Eq. (4) is satisfied. In practice, we length-normalize log-probabilities for both filtering and recentering to avoid direct length bias. 3.3 Deleting Only Positive-Advantage Rollouts Section 3.2 shows that both high-probability positive-advantage and low-probability negative-advantage rollouts can decrease entropy. That result, however, uses a KL-proximal update whose logit coordinates contain old-policy-dependent scaling. Under ordinary gradient updates with a shared learning rate, their entropy contributions generally differ even when their hgh_g values are equal. To compare these two classes under the latter update, we additionally analyze one gradient-ascent step on independent sequence logits at the starting point. Theoretical motivation: conditional entropy effects of positive and negative rollouts. For comparison, we consider gradient updates of logits ziz_i and explicitly require the same learning rate for every logit coordinate. With L(z)=G−1∑gA(g)log[softmax(z)]i(g)L(z)=G^-1 _gA^(g) [softmax(z)]_i(g), where i(g)i(g) is the sequence-space index of sampled rollout g, the update is: zi(α)=zi+α∂L(z)∂zi,α>0.z_i(α)=z_i+α ∂ L(z)∂ z_i, α>0. (5) Here α is the learning rate shared by all logit coordinates: it is the multiplier of each coordinate’s gradient that is shared, not the logit or probability change of every sequence. Under this relatively more realistic assumption of a shared learning rate, we obtain the following result. Proposition 3 (Greater Entropy Impact of Positive-Advantage Rollouts). In the independent-logit model above, assume ∑gA(g)=0 _gA^(g)=0. Here hgh_g denotes rollout g’s additive first-order entropy contribution under the KL-proximal update in Proposition 1, and hgSGDh_g^SGD denotes its corresponding contribution under the logit gradient update above. Consider sampled rollouts +y_+ and −y_- with advantages A±A_± and surprisals s±=−logπold(±∣x)s_±=- _old(y_± x). Suppose A+>0>A−A_+>0>A_- and s+<H(πold)<s−s_+<H( _old)<s_-. If their entropy contributions in Proposition 1 are equal, h+=h−<0h_+=h_-<0, then the high-probability rollout contributes more to entropy decrease to first order in a single logit gradient update: |h+SGD||h−SGD|=πold(+∣x)πold(−∣x)=exp(s−s+)>1. |h_+^SGD||h_-^SGD|= _old(y_+ x) _old(y_- x)= (s_--s_+)>1. (6) Thus, even when the two rollouts have equal entropy contributions in Proposition 1, the high-probability positive-advantage rollout makes a larger first-order contribution to entropy reduction in a single logit update, strengthening the tendency toward entropy collapse (see Appendix A.2 for the proof). This conditional comparison provides additional motivation for preferentially filtering high-probability positive-advantage candidates. Method: final deletion rule. Based on the above analysis, considering that deleting data itself can negatively reduce data informativeness and potentially degrade accuracy to some extent, we aim to minimize perturbations to the original training process and preserve the original algorithm and data distribution as much as possible. Guided by the preceding theory, we therefore delete only positive-advantage rollouts while retaining negative-advantage ones. Starting from the retained set ℐkeepI_keep in Section 3.2, we restore any excluded rollouts with nonpositive advantages: ℐkeep←ℐkeep∪g∈1,…,G:A(g)≤0.I_keep _keep∪\g∈\1,…,G\:A^(g)≤ 0\. (7) 3.4 Recentering After Deletion and the Full Procedure Selective deletion changes the composition of retained samples, may make their distribution depart from the old policy, and disrupts the original centering of group advantages. This motivates recentering the retained advantages after deletion. Theoretical motivation: normalization drift after deletion. Generalization can legitimately increase the probability of unsampled solutions similar to rewarded rollouts. Beyond generalization, a normalization offset that does not depend on similarity can also change the probabilities of unsampled paths, which is what we aim to avoid. However, unrecentered advantages and the resulting perturbed distribution tend to exacerbate this issue. To isolate it, we assume i.i.d. sampling of the original rollout group for a fixed prompt and, conditional on the realized group and retained set, analyze an auxiliary distribution-space update without transfer through shared parameters. Using the retained set ℐkeepI_keep, define the accumulated post-deletion advantage as A~ikeep=∑g∈ℐkeepA(g)i=(g) A_i^keep= _g _keepA^(g)1\y_i=y^(g)\. We then obtain the following result: Proposition 4 (Invariance condition for zero-advantage sequences). Fix a prompt x and a realized group of rollouts sampled i.i.d. from πold(⋅∣x) _old(· x). Let U=i:πold(i∣x)>0,A~ikeep=0U=\i: _old(y_i x)>0,\ A_i^keep=0\ be the set of unsampled and other zero-advantage paths in the old-policy support. Evaluate πθ _θ along the auxiliary retained-sample proximal update specified in Appendix A.3, with θ varying with η and πθ=πold _θ= _old at η=0η=0. For every i∈Ui∈ U, dηlogπθ(i∣x)|η=0=−πold[A~keep]. ddη _θ(y_i x) |_η=0=-E_ _old[ A^keep]. (8) This proposition shows that all zero-advantage paths share the same first-order log-probability shift: regardless of their quality, their probabilities decrease when the old-policy-weighted mean advantage is positive and increase when it is negative. This shift expresses no preference for more promising paths, and its first-order strength is determined by |πold[A~keep]||E_ _old[ A^keep]|. Deletion changes the retained score and its old-policy-weighted mean, making such unwarranted probability increases and decreases more pronounced, thereby leading to training instability. The proof is in Appendix A.3. Method: probability-weighted recentering. Based on the above analysis, we recenter a nonempty retained set as follows. Recomputing A~keep A^keep with the recentered advantages yields a zero old-policy-weighted mean and cancels the first-order normalization shift in the proximal model of Proposition 4: bkeep=∑g∈ℐkeepe−sgA(g)∑g∈ℐkeepe−sg,A¯keep(g)=A(g)−bkeep,g∈ℐkeep.b_keep= _g _keepe^-s_gA^(g) _g _keepe^-s_g, A_keep^(g)=A^(g)-b_keep, g _keep. (9) This local analysis motivates the recentering rule; its effectiveness in training is evaluated empirically in Section 4.2. Note that this probability weighting is introduced precisely because our data deletion is biased, which violates the i.i.d. assumption. GRPODropout procedure. Based on the above theory and analysis, we present the full GRPODropout procedure. To minimize changes to the original training process, we use the selected rollouts and recentered advantages in the original loss, preserving its formulation. The procedure reuses existing rewards and log-probabilities and requires no value network, adding negligible computational overhead for filtering and recentering. GRPODropout Input: prompt x, old policy πold _old, sample count G, and the original RL loss. 1. Sample G rollouts and compute rewards, native advantages A(g)A^(g), and old-policy rollout surprisals sgs_g; set advantages to zero if the reward standard deviation is zero. 2. Estimate H(πold)H( _old) by s¯ s, compute and fix hgh_g using Eq. (2), then solve Eq. (4) by removing the most negative estimated contributions first to obtain ℐkeepI_keep. 3. Apply Eq. (7) to restore excluded rollouts with nonpositive native advantages to ℐkeepI_keep. 4. For the retained set, compute and subtract bkeepb_keep using Eq. (9) to recentered advantages. 5. Update the policy using the original RL loss on retained rollouts. 4 Experiments We compare GRPODropout, original GRPO, and other entropy-aware baselines across three model sizes (1.7B, 4B, and 8B), trained on mathematics with matched prompts. Evaluation spans ten benchmarks covering mathematical reasoning and general QA, with the latter assessing generalization beyond mathematics. Given their small test sets, AMC23/AIME24/AIME25/HMMT25 use Mean@32; the others use Mean@1. Full settings, baselines, and metric conventions appear in Appendix B.1. 4.1 Main Results Table 1: Main results on three backbones. Avg. averages ten benchmarks; Base denotes the untrained backbone. Bold marks the best result within each backbone block. Method Mathematical reasoning General QA Avg. GSM8k Math500 AMC23 Olympiad AIME24 AIME25 HMMT25 ARC-C MMLUP SGPQA Qwen3-1.7B Base 89.92 80.60 59.92 42.96 22.08 21.04 11.46 87.37 57.01 29.28 50.16 GRPO 90.60 84.00 72.27 51.56 28.54 23.44 14.06 87.80 57.91 31.07 54.13 DAPO 88.86 85.20 72.27 52.00 33.75 26.15 15.21 87.63 55.62 29.70 54.64 Clip-Cov 82.34 77.20 53.91 36.15 11.56 12.71 5.21 87.37 55.19 28.26 44.99 KL-Cov 90.07 86.40 70.39 50.07 31.67 25.73 14.79 88.31 56.60 29.68 54.37 Beyond8020 89.01 84.00 72.19 50.22 29.48 25.00 18.02 86.69 56.03 29.78 54.04 GRPODropout 90.83 85.60 75.55 52.15 31.46 25.62 15.83 86.35 56.64 30.02 55.01 Qwen3-4B Base 94.77 85.40 72.50 47.41 37.92 26.25 14.27 94.20 69.19 39.85 58.18 GRPO 94.77 91.80 89.45 58.37 42.60 33.23 18.96 94.11 68.35 38.64 63.03 DAPO 94.47 89.80 86.72 57.48 42.40 36.88 19.06 94.11 67.48 38.52 62.69 Clip-Cov 94.54 92.00 86.33 59.41 54.06 41.56 27.08 94.45 69.32 40.79 65.95 KL-Cov 94.92 92.00 87.89 59.11 54.79 40.83 27.60 94.03 69.27 40.83 66.13 Beyond8020 94.39 91.60 89.92 60.00 51.88 41.25 23.65 94.54 68.21 39.39 65.48 GRPODropout 94.77 91.80 88.28 58.96 51.98 43.02 25.94 94.11 69.35 40.25 65.85 Qwen3-8B Base 95.45 82.80 67.50 46.07 37.40 24.69 11.98 95.90 73.14 43.61 57.85 GRPO 95.30 91.20 85.86 59.11 53.54 38.44 24.38 95.73 74.22 45.54 66.33 DAPO 95.53 92.20 90.31 61.33 52.40 38.85 21.25 95.65 72.39 43.75 66.37 Clip-Cov 95.38 91.20 84.77 60.44 54.90 42.08 27.60 95.56 73.76 45.05 67.07 KL-Cov 95.75 93.20 86.72 61.33 54.17 40.31 27.50 95.48 73.81 44.88 67.32 Beyond8020 95.45 92.20 87.58 60.59 57.29 39.06 24.69 95.99 73.23 44.57 67.07 GRPODropout 95.53 92.20 87.66 60.74 55.31 42.81 25.21 94.97 73.50 44.99 67.29 Figure 2: Qwen3-4B actor entropy and rollout usage across checkpoints. (a–b) Token-mean entropy on training DAPO-17K and held-out MMLU-Pro (MMLUP); (c) mean update rollouts per question in training batches, excluding all-zero-advantage groups. The other five methods use 8 rollouts at every recorded checkpoint (gray dashed line). Curves connect raw observations without smoothing. GRPODropout targets rollouts that may reduce policy entropy. Under the same rollout sampling budget, we assess whether using fewer retained rollouts in policy updates can preserve or improve performance over original GRPO. Table 1 reports results on three model sizes: selective deletion with recentering improves average accuracy over GRPO by 1.55 percentage points on average. These results support our data-usage insight that using all sampled rollouts is not always the most effective training strategy, and that selectively removing a subset can instead yield better performance. GRPODropout also exceeds KL-Cov, the strongest entropy-aware baseline by average score, by 0.11 points on average. Finally, our method is orthogonal in design to these optimization methods, offering both a data-selection insight and a complementary strategy for mitigating entropy collapse. Removing few rollouts substantially mitigates entropy collapse. We seek to preserve performance and exploration with minimal changes to training. Because sequence-level entropy is strongly affected by response length, we follow prior work (Cui et al., 2025) and report token-level conditional entropy averaged over valid response tokens. We measure this entropy as −∑vπθ(v∣x,<t)logπθ(v∣x,<t)- _v _θ(v x,y_<t) _θ(v x,y_<t), where v ranges over the vocabulary. Figure 2 tracks this metric for Qwen3-4B on DAPO-17K and held-out MMLU-Pro. While GRPO exhibits entropy collapse, GRPODropout achieves 446.1% and 256.0% higher entropy, respectively, at the end of training. Across the ten recorded checkpoints, excluding all-zero-advantage groups, it retains 7.03 rollouts per question for the update—12.1% fewer than GRPO’s baseline of eight. Notably, compared to Beyond8020, which masks out 80% of tokens, our method removes a mere 12.1% of the data while achieving comparable or even superior policy entropy and performance, demonstrating the efficacy and value of rollout-level deletion. 4.2 Ablation Studies Our method has two main components: rollout deletion and advantage recentering. We ablate the effects of deleting positive-advantage (Pos.) and negative-advantage (Neg.) rollouts, enabling recentering (Center), and using probability weighting during recentering (P-wt.). Specifically, with Center/P-wt. set to ✓/✓ / , we subtract the probability-weighted mean advantage as in Eq. (9); with ✓/× /×, we subtract the equal-weight mean of retained advantages; with ×/×/×, no recentering is performed. The first row of each model block reports original GRPO without rollout deletion. Table 2 reports results on Qwen3-1.7B and Qwen3-4B. Table 2: Qwen3-1.7B and Qwen3-4B ablations of positive- and negative-advantage rollouts, advantage recentering, and old-policy probability weighting of the mean. ✓ /×: enabled/disabled; –: not applicable. Avg. Ent reports benchmark-average actor entropy. Deletion Advantages Mathematical reasoning General QA Avg. Avg. Ent Pos. Neg. Center P-wt. GSM8k Math500 AMC23 Olympiad AIME24 AIME25 HMMT25 ARC-C MMLUP SGPQA Qwen3-1.7B × × – – 90.60 84.00 72.27 51.56 28.54 23.44 14.06 87.80 57.91 31.07 54.13 0.0503 ✓ ✓ ✓ ✓ 90.30 86.80 70.47 49.04 30.94 24.90 13.75 87.71 56.53 29.77 54.02 0.2025 ✓ × × × 77.33 39.00 10.70 8.74 0.00 0.10 0.00 87.12 44.04 21.09 28.81 0.3282 ✓ × ✓ × 89.76 85.00 65.31 46.67 27.29 22.40 11.46 87.88 56.64 29.44 52.19 0.2239 ✓ × ✓ ✓ 90.83 85.60 75.55 52.15 31.46 25.62 15.83 86.35 56.64 30.02 55.01 0.1739 Qwen3-4B × × – – 94.77 91.80 89.45 58.37 42.60 33.23 18.96 94.11 68.35 38.64 63.03 0.0528 ✓ ✓ ✓ ✓ 94.77 91.00 87.03 58.22 51.46 41.25 25.00 93.34 69.09 40.10 65.13 0.2425 × ✓ ✓ ✓ 94.54 91.40 87.97 59.26 51.04 41.88 23.44 94.28 68.80 40.43 65.30 0.2043 ✓ × × × 0.38 0.20 2.19 0.00 0.00 0.00 0.00 88.57 42.35 26.34 16.00 0.0200 ✓ × ✓ × 94.62 92.20 89.30 58.22 52.81 42.08 25.10 93.94 69.48 40.44 65.82 0.2174 ✓ × ✓ ✓ 94.77 91.80 88.28 58.96 51.98 43.02 25.94 94.11 69.35 40.25 65.85 0.2140 Deleting only positive-advantage rollouts performs best. With probability-weighted recentering fixed, we compare deleting positive- and/or negative-advantage candidates. Deletion generally increases entropy and improves final accuracy in most settings, while the magnitude of the accuracy gain depends on the deletion pattern and recentering scheme. Deleting both positive and negative candidates yields the largest entropy gain but incurs a slight accuracy loss, showing that data removal can sacrifice performance and supporting our motivation to delete as little data as possible. Deleting only positive candidates achieves higher entropy and accuracy than deleting only negative candidates, consistent with our theoretical result that positive-advantage rollouts have a greater effect on entropy reduction. Recentering is necessary. With deletion restricted to positive-advantage candidates, removing recentering causes severe performance degradation and training collapse, demonstrating the importance of recentering in the tested deletion configurations. We further introduce probability weighting to calibrate the retained advantages under the old-policy measure. On Qwen3-4B, probability weighting has little effect, improving the average score by only 0.03 points over equal-weight recentering. Its benefit is more pronounced on Qwen3-1.7B, where removing the weighting lowers the average score by 2.82 points, suggesting that probability weighting can sometimes improve training stability. 4.3 More Precise Exploration and More Diverse Reasoning Paths The full GRPODropout method increases entropy while largely preserving or improving average accuracy across the three model sizes. We further examine the value of this retained entropy from two perspectives: exploration precision, to assess accuracy preservation; and reasoning-path diversity, to evaluate the preservation of exploration capacity. Metrics. We first assess how accurately model sampling produces correct solutions while limiting repeated incorrect final answers. We compare all methods on Qwen3-4B across AMC23, AIME24, AIME25, and HMMT25, sampling 32 rollouts per question. Pass@32 is computed per question within each benchmark and then averaged equally across AMC23, AIME24, AIME25, and HMMT25. For Table 3, Entropy is token-mean actor entropy over valid response tokens at the final checkpoint, averaged equally across the four benchmarks. Inspired by the answer-consistency perspective of Wang et al. (2022), we define answer-string dispersion W as the number of distinct nonempty final-answer strings among a question’s incorrect responses. W measures answer-string dispersion among incorrect final answers and can reveal recurring answer patterns. To distinguish relatively easier and harder questions, we use a fixed cross-method partition: a question enters the ≥8≥ 8 subset only if every one of the six systems has at least eight correct responses, and enters the <8<8 subset only if every system has fewer than eight. We average W within each subset. We next assess whether models approach the same problem through diverse reasoning paths. To this end, we use GLM-5.2 to summarize each response’s reasoning path in three to five sentences, excluding abandoned reasoning, and then encode the summaries with bge-m3 (Chen et al., 2024). Following Kirk et al. (2023); Chen et al. (2026), Summary distance measures the mean pairwise cosine distance between summary embeddings for the same question, reflecting semantic separation. Following Friedman and Dieng (2023); Encheng et al. (2026), Vendi measures effective diversity as the exponential of the entropy of the eigenvalues of the Gram matrix normalized by the number of summaries, accounting for similarity and repetition. Higher values indicate greater diversity. Here, W measures diversity among incorrect final answers, whereas Summary distance and Vendi measure diversity in the summarized reasoning paths, including paths that may lead to the same final answer. Full prompt and run settings are given in Appendix B.2. A favorable trade-off between correctness and diversity. The results show that our method achieves the best or second-best reported mean on all eight metrics and ranks first among the six systems, with an overall mean rank of 1.63 under equal weighting. In particular, our method attains the highest reasoning-path diversity on the ≥8≥ 8 subset and the highest actor entropy while achieving lower W than all baselines except KL-Cov. Compared with GRPO, higher solution coverage and actor entropy accompany fewer distinct incorrect answers and higher values on all four path-diversity measures, demonstrating that our improvement in Pass@1 accuracy does not come at the expense of Pass@32 or the model’s exploratory capability, but rather enhances them simultaneously. Notably, although DAPO exhibits relatively high entropy and reasoning-path diversity on the ≥8≥ 8 subset, this does not translate into the highest Pass@32, indicating that it sacrifices accuracy for exploratory diversity. Correspondingly, our method maximally preserves exploration accuracy—namely, higher Pass@1 and Pass@32, and fewer distinct incorrect answers—while fully maintaining the path diversity and entropy of correct answers, ensuring no loss of the model’s exploratory capability and achieving a simultaneous preservation of accuracy and entropy. Table 3: Qwen3-4B solution coverage, wrong-answer dispersion, and reasoning diversity. Definitions are in Section 4.3. Bold marks the best and underline marks the second-best in each column. Method Correctness Path diversity Entropy ↑ Overall mean rank ↓ pass@32 ↑ W≥8W_≥ 8 ↓ W<8W_<8 ↓ Dist.≥8 ↑ Dist.<8 ↑ Vendi≥8 ↑ Vendi<8 ↑ GRPO 63.54 1.1846 7.6216 0.1063 0.1641 1.8311 2.4194 0.0448 5.63 DAPO 69.38 1.6154 8.0811 0.1150 0.1798 1.9080 2.6013 0.1796 3.88 Clip-Cov 76.88 0.7692 5.2162 0.1139 0.2141 1.9041 2.9851 0.0900 2.63 KL-Cov 72.71 0.4615 3.0000 0.1134 0.2014 1.8930 2.8510 0.1509 3.25 Beyond8020 73.54 1.1231 7.8378 0.1142 0.1726 1.9037 2.5252 0.1671 4.00 GRPODropout 75.21 0.4923 4.2432 0.1151 0.2090 1.9114 2.8992 0.2291 1.63 5 Conclusion We find that some high-probability positive rollouts can hinder training in reinforcement learning algorithms such as GRPO. Removing these rollouts improves performance and mitigates entropy collapse. Based on this finding, we propose GRPODropout, which selectively removes high-probability positive rollouts and recenters the retained advantages to stabilize training, while leaving the original RL loss unchanged. We further extend the theoretical analysis of entropy dynamics from tokens to complete rollouts, providing theoretical motivation for our rollout-deletion strategy. Extensive experiments across three model scales and ten benchmarks demonstrate that GRPODropout outperforms the original GRPO while using fewer retained rollouts in policy updates. Limitations. First, GRPODropout does not outperform the strongest baseline at every model scale. It is worth pointing out that we do not merely provide an entropy collapse mitigation method; rather, our work includes the insight that reinforcing some rollouts—even correct ones—can be counterproductive, which also possesses clear value. Second, given the complexity of RL training, our theory itself cannot fully explain our method. In fact, the primary role of the theory is to serve as motivation for method design, acting as the initial starting point. The final method is validated through its advantages over the original GRPO and the ablation study results. Finally, GRPODropout relies on within-prompt differences in advantages and probabilities across multiple rollouts. It therefore does not directly apply to single-rollout training, while its effectiveness with very small groups remains to be evaluated. References Abdolmaleki et al. (2018) A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. A. Riedmiller Maximum a posteriori policy optimisation. CoRR abs/1806.06920. External Links: Link, 1806.06920 Cited by: §A.1, §3.2. Auer et al. (2002) P. Auer, N. Cesa-Bianchi, and P. Fischer Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47 (2-3), p. 235–256. External Links: Link, Document Cited by: §1. Chen et al. (2024) J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu M3-embedding: multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Findings of ACL, Vol. ACL 2024, p. 2318–2335. External Links: Link, Document Cited by: §4.3. Chen et al. (2026) X. Chen, W. Zhu, P. Qiu, X. Dong, H. Wang, H. Wu, H. Li, A. Sotiras, Y. Wang, and A. Razi DRA-GRPO: your GRPO needs to know diverse reasoning paths for mathematical reasoning. In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), p. 13995–14019. External Links: Link, Document Cited by: §4.3. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR abs/1803.05457. External Links: Link, 1803.05457 Cited by: §B.1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. CoRR abs/2110.14168. External Links: Link, 2110.14168 Cited by: §B.1. Cui et al. (2025) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, Z. Liu, H. Peng, L. Bai, W. Ouyang, Y. Cheng, B. Zhou, and N. Ding The entropy mechanism of reinforcement learning for reasoning language models. CoRR abs/2505.22617. External Links: Link, Document, 2505.22617 Cited by: §B.1, §1, §1, §2, §4.1. Du et al. (2025) D. Du, S. Liu, T. Yang, S. Chen, and Y. Li UloRL:an ultra-long output reinforcement learning approach for advancing large language models’ reasoning abilities. CoRR abs/2507.19766. External Links: Link, Document, 2507.19766 Cited by: §1, §2, §2. Encheng et al. (2026) C. Encheng, S. Peng, K. Ito, J. Xu, S. Hisada, S. Wakamiya, and E. Aramaki Single-agent generation surpasses multi-agent systems in semantic diversity. In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), p. 37993–38008. External Links: Link, Document Cited by: §4.3. Feng et al. (2026) X. Feng, Z. Li, Z. Liu, H. Li, Y. Jiang, B. Guo, J. Guo, J. Zhang, and S. Guo Beyond entropy: learning from token-level distributional deviations for LLM reasoning. CoRR abs/2606.19771. External Links: Link, Document, 2606.19771 Cited by: §2. Friedman and Dieng (2023) D. Friedman and A. B. Dieng The vendi score: A diversity evaluation metric for machine learning. Trans. Mach. Learn. Res. 2023. External Links: Link Cited by: §4.3. Hao et al. (2025) Z. Hao, H. Wang, H. Liu, J. Luo, J. Yu, H. Dong, Q. Lin, C. Wang, and J. Chen Rethinking entropy interventions in RLVR: an entropy change perspective. CoRR abs/2510.10150. External Links: Link, Document, 2510.10150 Cited by: §1, §2. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. CoRR abs/2402.14008. External Links: Link, Document, 2402.14008 Cited by: §B.1. He et al. (2026) Y. He, H. Wu, S. Liu, H. Ge, H. Zhou, K. Wu, Z. Zheng, Q. Lin, Z. Zhong, and Y. Zhang Rethinking token-level credit assignment in RLVR: A polarity-entropy analysis. CoRR abs/2604.11056. External Links: Link, Document, 2604.11056 Cited by: §2. Jin et al. (2026) R. Jin, P. Gao, Y. Ren, Z. Han, T. Zhang, W. Huang, W. Liu, J. Luan, and D. Xiong Revisiting entropy in reinforcement learning for large reasoning models. In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), p. 25300–25322. External Links: Link, Document Cited by: §1, §1, §2. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. E. Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan Language models (mostly) know what they know. CoRR abs/2207.05221. External Links: Link, Document, 2207.05221 Cited by: §3.2. Kirk et al. (2023) R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu Understanding the effects of RLHF on LLM generalisation and diversity. CoRR abs/2310.06452. External Links: Link, Document, 2310.06452 Cited by: §4.3. Li et al. (2025) Q. Li, R. Xue, J. Wang, M. Zhou, Z. Li, X. Ji, Y. Wang, M. Liu, Z. Yang, M. Qiu, and J. Yang CURE: critical-token-guided re-concatenation for entropy-collapse prevention. CoRR abs/2508.11016. External Links: Link, Document, 2508.11016 Cited by: §1, §2. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. CoRR abs/2305.20050. External Links: Link, Document, 2305.20050 Cited by: §B.1. Luo et al. (2026) H. Luo, Q. Sun, S. Wu, C. Xu, W. Deng, H. Hu, and Y. Tang STARE: surprisal-guided token-level advantage reweighting for policy entropy stability. CoRR abs/2606.19236. External Links: Link, Document, 2606.19236 Cited by: §2. Lv et al. (2026) O. Lv, Y. Zheng, Y. Zhang, Z. Huang, X. Wang, B. Dong, H. Zhu, and Y. Chen Which tokens matter? adaptive token selection for RLVR with the relative surprisal index. CoRR abs/2606.31575. External Links: Link, Document, 2606.31575 Cited by: §2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. CoRR abs/2203.02155. External Links: Link, Document, 2203.02155 Cited by: §3.1. Park et al. (2025) J. R. Park, J. Kim, G. Kim, J. Jo, S. Choi, J. Cho, and E. K. Ryu Clip-low increases entropy and clip-high decreases entropy in reinforcement learning of large language models. CoRR abs/2509.26114. External Links: Link, Document, 2509.26114 Cited by: §1, §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. CoRR abs/2402.03300. External Links: Link, Document, 2402.03300 Cited by: §A.1, §B.1, §1. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano Learning to summarize from human feedback. CoRR abs/2009.01325. External Links: Link, 2009.01325 Cited by: §3.1. Sutton et al. (1999) R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999], S. A. Solla, T. K. Leen, and K. Müller (Eds.), p. 1057–1063. External Links: Link Cited by: §A.1. Team et al. (2025) M. Team, X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, C. Zheng, K. Deng, S. Guo, S. Jia, S. Jiang, Y. Liao, R. Li, Q. Li, S. Li, Y. Li, Y. Li, D. Ma, Y. Ni, H. Que, Q. Wang, Z. Wen, S. Wu, T. Xing, M. Xu, Z. Yang, Z. M. Wang, J. Zhou, Y. Bai, X. Bu, C. Cai, L. Chen, Y. Chen, C. Cheng, T. Cheng, K. Ding, S. Huang, Y. Huang, Y. Li, Y. Li, Z. Li, T. Liang, C. Lin, H. Lin, Y. Ma, Z. Peng, Z. Peng, Q. Qi, S. Qiu, X. Qu, Y. Tan, Z. Wang, C. Wang, H. Wang, Y. Wang, Y. Wang, J. Xu, K. Yang, R. Yuan, Y. Yue, T. Zhan, C. Zhang, J. Zhang, X. Zhang, X. Zhang, Y. Zhang, Y. Zhao, X. Zheng, C. Zhong, Y. Gao, Z. Li, D. Liu, Q. Liu, T. Liu, S. Ni, J. Peng, Y. Qin, W. Su, G. Wang, S. Wang, J. Yang, M. Yang, M. Cao, X. Yue, Z. Zhang, W. Zhou, J. Liu, Q. Lin, W. Huang, and G. Zhang SuperGPQA: scaling llm evaluation across 285 graduate disciplines. External Links: 2502.14739, Link Cited by: §B.1. Team (2025) Q. Team Qwen3 technical report. CoRR abs/2505.09388. External Links: Link, Document, 2505.09388 Cited by: §B.1. Wang et al. (2025a) C. Wang, Z. Li, J. Bai, Y. Zhang, S. Cui, Z. Zhao, and Y. Wang Arbitrary entropy policy optimization: entropy is controllable in reinforcement fine-tuning. CoRR abs/2510.08141. External Links: Link, Document, 2510.08141 Cited by: §1, §2. Wang et al. (2025b) S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, Y. Liu, A. Yang, A. Zhao, Y. Yue, S. Song, B. Yu, G. Huang, and J. Lin Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR abs/2506.01939. External Links: Link, Document, 2506.01939 Cited by: §B.1, §1, §2. Wang et al. (2022) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, and D. Zhou Self-consistency improves chain of thought reasoning in language models. CoRR abs/2203.11171. External Links: Link, Document, 2203.11171 Cited by: §4.3. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-pro: A more robust and challenging multi-task language understanding benchmark. CoRR abs/2406.01574. External Links: Link, Document, 2406.01574 Cited by: §B.1. Williams (1992) R. J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, p. 229–256. External Links: Link, Document Cited by: §A.1. Yang et al. (2026) S. Yang, Y. Liu, J. Zheng, Z. Yang, and L. Li PAEC: position-aware entropy calibration for LLM reasoning in RLVR. CoRR abs/2606.08543. External Links: Link, Document, 2606.08543 Cited by: §1, §2. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source LLM reinforcement learning system at scale. CoRR abs/2503.14476. External Links: Link, Document, 2503.14476 Cited by: §B.1, §B.1, §1, §1, §2. Yu et al. (2026) S. Yu, L. Li, W. Zhao, and Z. Yang ERPO: token-level entropy-regulated policy optimization for large reasoning models. CoRR abs/2603.28204. External Links: Link, Document, 2603.28204 Cited by: §2. Yue et al. (2025) Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. CoRR abs/2504.13837. External Links: Link, Document, 2504.13837 Cited by: §1. Zenn and Geiping (2026) J. Zenn and J. Geiping When are likely answers right? on sequence probability and correctness in llms. CoRR abs/2606.27359. External Links: Link, Document, 2606.27359 Cited by: §3.2. Zhang et al. (2026) J. Zhang, Z. Fu, J. Shen, Y. Zhao, Y. Zhang, Z. Xi, L. Ma, C. An, Z. Zhang, S. Liu, D. Zhu, S. Dou, S. Liu, H. Li, W. Zhou, A. Adams, T. Gui, F. Huang, Q. Zhang, and X. Huang Entropy polarity in reinforcement fine-tuning: direction, asymmetry, and control. CoRR abs/2605.11775. External Links: Link, Document, 2605.11775 Cited by: §2. Zhao et al. (2026) Z. Zhao, Z. Ren, J. Zou, L. Yang, Z. Xu, X. Ge, Z. Chen, X. Ma, D. Shi, S. Wang, D. Yin, and X. Xin Reinforced efficient reasoning via semantically diverse exploration. CoRR abs/2601.05053. External Links: Link, Document, 2601.05053 Cited by: §1, §2. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin Group sequence policy optimization. CoRR abs/2507.18071. External Links: Link, Document, 2507.18071 Cited by: §A.1. Zhu et al. (2025) X. Zhu, M. Xia, Z. Wei, W. Chen, D. Chen, and Y. Meng The surprising effectiveness of negative reinforcement in LLM reasoning. CoRR abs/2506.01347. External Links: Link, Document, 2506.01347 Cited by: §2. Appendix A Supplementary Material A.1 Proofs of Proposition 1 and Proposition 2 Local update approximation. To extend token-level entropy analysis to complete rollouts, we adopt a distribution-space KL-proximal policy-improvement model following Abdolmaleki et al. (2018). For a candidate policy π(⋅∣x)π(· x), we define its proximal update as LA(π)=1G∑g=1GA(g)π((g)∣x)πold((g)∣x),π(η)=argmaxπ∈ΔoldLA(π)−1ηKL(π∥πold).L_A(π)= 1G _g=1^GA^(g) π(y^(g) x) _old(y^(g) x), π^(η)= _π∈ _old \L_A(π)- 1ηKL\! (π\,\|\, _old ) \. (A.1) Here η>0η>0 controls the size of the proximal update, and Δold _old contains normalized distributions over complete sequences supported on i:πold(i∣x)>0\y_i: _old(y_i x)>0\. Eq. (A.1) specifies an idealized update directly over sequence distributions. Its old-policy KL penalty specifies the update geometry, while the KL regularizer relative to the reference policy is omitted. We analyze the update around η=0η=0, taking π(0)=πoldπ^(0)= _old, with the sampled group and its finite advantages held fixed. We assume H(πold)<∞H( _old)<∞ and use rollout advantages without length normalization. We derive the sequence-score representation and closed-form update below. Notation for the proofs. For the fixed prompt x, we use the shorthand p()=πold(∣x)p(y)= _old(y x) and q()=π(∣x)q(y)=π(y x) for the old and candidate policies, with pi=p(i)p_i=p(y_i) and qi=q(i)q_i=q(y_i). The proximal policies are written qi(η)=π(η)(i∣x)q_i^(η)=π^(η)(y_i x) and qkeep,i(η)=πθ(i∣x)q_keep,i^(η)= _θ(y_i x), where in the latter case θ varies with η along the retained-sample proximal update. These abbreviations are used only in this appendix. Sampling-aligned sequence score. With the sampled group and its advantages fixed, express the sampled surrogate in sequence space using A~i=1Gpi∑g=1GA(g)i=(g),pi>0. A_i= 1Gp_i _g=1^GA^(g)1\y_i=y^(g)\, p_i>0. (A.2) Here ⋅1\·\ is the indicator function, and sums over sequence indices are restricted to the old-policy support. Repeated samples of the same sequence contribute additively, while a sequence not sampled in the current group has A~i=0 A_i=0 and receives no direct update signal. Indeed, ∑iqiA~i=∑iqi1Gpi∑g=1GA(g)i=(g)=1G∑g=1GA(g)q((g))p((g))=LA(q). _iq_i A_i= _iq_i 1Gp_i _g=1^GA^(g)1\y_i=y^(g)\= 1G _g=1^GA^(g) q(y^(g))p(y^(g))=L_A(q). Intuitively, A~i A_i measures the marginal value of assigning probability to sequence iy_i: transferring a small probability mass δ from jy_j to iy_i changes LAL_A by δ(A~i−A~j)δ( A_i- A_j). Consequently, at the starting point, ∇θLA=G−1∑gA(g)∇θlogπθ((g)∣x)|θ=θold _θL_A=G^-1 _gA^(g) _θ _θ(y^(g) x)|_θ= _old; the old policy and denominator are held fixed. Omitting 1/pi1/p_i from Eq. (A.2) would change the coefficient of sample g to e−sgA(g)/Ge^-s_gA^(g)/G, which cannot generally be corrected by one global learning rate. The inverse-probability factor is needed only for the mathematical correspondence and is not explicitly computed by the algorithm; its difference from a direct logit update is analyzed in Appendix A.2. Setup and regularity conditions. Fix a prompt x and consider only the positive-probability support of the old policy, i:pi>0\y_i:p_i>0\. Assume that this set is finite or countable, H(p)<∞H(p)<∞, the sample count G is finite, and every advantage A(g)A^(g) is a finite real number; samples and advantages are fixed throughout the update. Since the score in Eq. (A.2) can be nonzero only on finitely many sampled rollouts and each has pi>0p_i>0, the empirical score is bounded. Therefore Z(η)=∑ipieηA~iZ(η)= _ip_ie^η A_i exists and is positive for every finite η. In a neighborhood of zero, qi(η)/piq_i^(η)/p_i and its derivative with respect to η are uniformly bounded, and each summand of the entropy derivative is dominated by a constant multiple of pi(1+|logpi|)p_i(1+| p_i|). Interchanging differentiation and summation below is consequently valid. This argument is conditional on a fixed finite sampled group and does not mistake a finite first moment for exponential integrability of an arbitrary unbounded score. Taking a further expectation over prompts requires the corresponding additional integrability conditions. Closed-form KL-proximal update. The stationarity condition for Eq. (A.1) under normalization gives qi∝piexp(ηA~i)q_i p_i (η A_i). Normalizing over the old-policy support yields π(η)(i∣x)=πold(i∣x)exp(ηA~i)∑jπold(j∣x)exp(ηA~j).π^(η)(y_i x)= _old(y_i x) (η A_i) _j _old(y_j x) (η A_j). (A.3a) The updated policy converges to πold _old as η→0+η→ 0^+, so we extend the update continuously to π(0)=πoldπ^(0)= _old. Equivalently, using the shorthand defined above, qi(η)=piexp(ηA~i)Z(η),Z(η)=∑jpjexp(ηA~j).q_i^(η)= p_i (η A_i)Z(η), Z(η)= _jp_j (η A_j). (A.3b) To verify that this is the unique global optimizer rather than merely a stationary point, observe that for every q∈Δoldq∈ _old, ∑iqiA~i−1ηKL(q∥p)=1ηlogZ(η)−1ηKL(q∥q(η)). _iq_i A_i- 1ηKL(q\|p)= 1η Z(η)- 1ηKL(q\|q^(η)). By nonnegativity of KL, the right-hand side is maximized exactly at q=q(η)q=q^(η); a distribution with KL(q∥p)=∞KL(q\|p)=∞ cannot improve the objective. This argument applies to finite and countable supports. Taking logs of Eq. (A.3b) and differentiating yields 1qi(η)dqi(η)dη=A~i−Z′(η)Z(η)=A~i−q(η)[A~]. 1q_i^(η) dq_i^(η)dη= A_i- Z (η)Z(η)= A_i-E_q^(η)[ A]. Hence dqi(η)dη=qi(η)(A~i−q(η)[A~]). dq_i^(η)dη=q_i^(η) ( A_i-E_q^(η)[ A] ). (A.4) At η=0η=0, qi(0)=piq_i^(0)=p_i, and therefore dqi(η)dη|η=0=pi(A~i−p[A~]). . dq_i^(η)dη |_η=0=p_i ( A_i-E_p[ A] ). (A.5) Differentiating H(q(η))=−∑iqi(η)logqi(η)H(q^(η))=- _iq_i^(η) q_i^(η) gives dH(q(η))dη=−∑idqi(η)dη(1+logqi(η)). dH(q^(η))dη=- _i dq_i^(η)dη (1+ q_i^(η) ). Because ∑iqi(η)=1 _iq_i^(η)=1, we have ∑idqi(η)/dη=0 _idq_i^(η)/dη=0, so the constant term vanishes. Therefore, dH(q(η))dη=−∑idqi(η)dηlogqi(η). dH(q^(η))dη=- _i dq_i^(η)dη q_i^(η). Substituting η=0η=0 and Eq. (A.5) gives dH(q(η))dη|η=0=−∑ipi(A~i−p[A~])logpi=−Covp(A~,logp). . dH(q^(η))dη |_η=0=- _ip_i ( A_i-E_p[ A] ) p_i=-Cov_p( A, p). (A.6) This is the entropy derivative of the reward-driven KL-proximal update specified by Eq. (A.1); it does not apply to an arbitrary online-RL parameter update. Proposition 1: per-sample contribution. For each sampled index g, define the score component ai[g]=A(g)i=(g)/(Gpi)a_i^[g]=A^(g)1\y_i=y^(g)\/(Gp_i), so that A~i=∑gai[g] A_i= _ga_i^[g]. Since p[a[g]]=A(g)G,p[a[g]logp]=−A(g)Gsg,p[logp]=−H(p),E_p[a^[g]]= A^(g)G, _p[a^[g] p]=- A^(g)Gs_g, _p[ p]=-H(p), and Eq. (A.6) is linear in its score argument, hg=−Covp(a[g],logp)=A(g)G(sg−H(p)).h_g=-Cov_p(a^[g], p)= A^(g)G (s_g-H(p) ). (A.7) This proves Eq. (2). Throughout, g denotes a sampled index and i an index in the complete-sequence space. This additive decomposition includes the normalization-induced changes of other sequences, not only the entropy term of rollout g itself. Its sign is determined by A(g)(sg−H(p))A^(g)(s_g-H(p)): a relatively high-probability positive-advantage rollout and a relatively low-probability negative-advantage rollout decrease entropy; the reverse combinations increase it. The contribution is zero if A(g)=0A^(g)=0 or sg=H(p)s_g=H(p). Proposition 2: group entropy derivative. Summing Eq. (A.7) over sampled indices gives the entropy derivative of q(η)q^(η) at η=0η=0. Under the zero-sum condition of Proposition 2, p[A~]=G−1∑gA(g)=0E_p[ A]=G^-1 _gA^(g)=0, so dH(q(η))dη|η=0 . dH(q^(η))dη |_η=0 =∑ghg=1G∑gA(g)sg = _gh_g= 1G _gA^(g)s_g (A.8) =1G∑gA(g)(sg−s¯)=Covg=1G(A(g),sg). = 1G _gA^(g)(s_g- s)=Cov_g=1^G\! (A^(g),s_g ). This proves Eq. (3), using the same sgs_g and s¯ s as in the main text. Thus a negative empirical covariance means that the KL-proximal update driven by this fixed sampled group decreases entropy at the starting point, while a positive value means that it increases entropy. When the covariance is zero, only the first-order term vanishes; finite-step entropy change is undetermined. Eq. (A.8) is exact conditional on the current samples and weights, not an entropy guarantee for a neural-network update. Monte Carlo evaluation and greedy selection. For the sequence-level criterion, we estimate H(p)H(p) by the sample mean s¯ s of the existing G rollouts and substitute it into Eq. (2). Each sg=−∑t=1Tglogπold(yt(g)∣x,<t(g))s_g=- _t=1^T_g _old(y_t^(g) x,y_<t^(g)) sums negative old-policy log-probabilities over valid response tokens; the length-normalized implementation is described below. The covariance in Eq. (4) is computed over the original G rollouts. Since ∑g=1G(sg−s¯)=0 _g=1^G(s_g- s)=0, centering the masked advantages leaves Covg=1G(A(g)g∈ℐ,sg) _g=1^G\! (A^(g)1\g \,s_g ) =1G∑g∈ℐA(g)(sg−s¯) = 1G _g A^(g)(s_g- s) =∑g∈ℐhg|H(p)→s¯. = _g .h_g |_H(p)→ s. Thus the covariance constraint is exactly equivalent to a nonnegative sum of the remaining MC contributions. Recomputing the covariance on the retained subset would generally change the criterion because its mean surprisal need not equal s¯ s. For the greedy-selection argument, use hgh_g with the MC substitution specified in Section 3.2. Start with all G rollout indices. If their total contribution is negative, remove negative contributions in ascending order until the retained sum is nonnegative. Among sets with the same number of removals, removing the most negative contributions produces the largest retained sum, while removing nonnegative contributions cannot help. Thus the first feasible set uses the fewest removals and retains the most rollouts, solving Eq. (4). Removing all negative contributions leaves a nonnegative sum, so a feasible retained set exists. If ∑g=1Ghg≥0 _g=1^Gh_g≥ 0, retaining all G rollouts is optimal. This establishes maximum-cardinality optimality for the fixed MC contributions. Length normalization in implementation. To avoid direct length bias from summing token log-probabilities, the implementation uses the length-normalized surprisal sglen=sg/Tgs_g^len=s_g/T_g for both filtering and recentering, with TgT_g counting valid response tokens. Filtering uses the original-group mean of sglens_g^len, and recentering uses weights proportional to exp(−sglen)=πold((g)∣x)1/Tg (-s_g^len)= _old(y^(g) x)^1/T_g, computed by a softmax over retained rollouts. Thus responses with equal mean token log-probability receive equal recentering weights regardless of length. These normalized quantities are implementation proxies: their group mean is a filtering baseline rather than a complete-sequence entropy estimate, and the implemented zero-mean condition is defined by these weights. The theoretical results retain full sequence probabilities as their analysis model. General objective and probability-ratio surrogate. Eq. (A.1) specifies a distribution-space KL-proximal geometry for analyzing the sampled reward update. Its penalty η−1KL(q∥p)η^-1KL(q\|p) limits change relative to the old policy, whereas the reference penalty βKL(q∥πref)β\,KL(q\| _ref) in the original RL objective penalizes deviation from a fixed reference model. The generalized step size η and reference-KL coefficient β serve different purposes; the proximal model analyzes the reward contribution separately from clipping, length weights, and reference regularization. For the reward term, subtracting a fixed baseline that depends only on x does not change the expected policy gradient (Williams, 1992; Sutton et al., 1999). For any fixed advantage function A(x,)A(x,y), if the relevant expectation exists and the support of q is contained in that of p, the probability-ratio identity gives ∑iqiA(x,i)=∑ipiqipiA(x,i)=i∼p[qipiA(x,i)]. _iq_iA(x,y_i)= _ip_i q_ip_iA(x,y_i)=E_i p\! [ q_ip_iA(x,y_i) ]. Taking A(g)=A(x,(g))A^(g)=A(x,y^(g)) gives a sampled estimator of this expectation. More generally, an algorithm can construct advantage estimates from the full sampled group, in which case we directly define the empirical surrogate conditional on the current samples: LA(q) L_A(q) =1G∑g=1GA(g)q((g))p((g)), = 1G _g=1^GA^(g) q(y^(g))p(y^(g)), ∇θLA(πθ(⋅∣x))|θ=θold . _θL_A( _θ(· x)) |_θ= _old =1G∑g=1GA(g)∇θlogπθ((g)∣x)|θ=θold. = 1G _g=1^GA^(g) . _θ _θ(y^(g) x) |_θ= _old. The key condition for extending the local surrogate is therefore that the algorithm’s reward gradient at the starting point can be expressed in this rollout-level weighted form. Group-dependent advantage estimates are generally correlated with the current rollout, so the probability-ratio identity alone does not make this an unbiased estimator of the original expected-return gradient. Sequence-level policy gradients. When a complete generated sequence is treated as one action, the REINFORCE sampled gradient corresponds to A(g)=R(x,(g))−b(x)A^(g)=R(x,y^(g))-b(x) for a fixed prompt-level baseline b(x)b(x). An actor–critic update that treats the sequence as one action and holds its advantage fixed while differentiating has the same form. This correspondence concerns the reward gradient and excludes gradients from additional regularizers. Group-relative policy optimization. For outcome-supervised GRPO (Shao et al., 2024) and GSPO (Zheng et al., 2025), define the native group-relative advantage by A(g)=R(x,(g))−R¯σR,R¯=1G∑g=1GR(x,(g)),σR=[1G∑g=1G(R(x,(g))−R¯)2]1/2.A^(g)= R(x,y^(g))- R _R, R= 1G _g=1^GR(x,y^(g)), _R= [ 1G _g=1^G(R(x,y^(g))- R)^2 ]^1/2. If σR=0 _R=0, all advantages are defined as zero, so ∑gA(g)=0 _gA^(g)=0 always holds. Under outcome supervision, every token within a rollout shares A(g)A^(g). Original GRPO averages the token loss within each rollout, whereas GSPO uses the length-normalized sequence probability ratio [q((g))/p((g))]1/Tg[q(y^(g))/p(y^(g))]^1/T_g. Ignoring the reference KL and differentiating at θ=θoldθ= _old with clipping inactive, both correspond to replacing A(g)A^(g) in LAL_A by A(g)/TgA^(g)/T_g. If GRPO sums token losses rather than averaging within each rollout, it corresponds to LAL_A without length normalization. These correspondences match parameter gradients only at the starting point and do not make the clipped objectives or their off-starting-point gradients identical. A.2 Proof of Proposition 3 Independent-logit update and entropy derivative. Restrict the complete-sequence space to be finite, associate each sequence with an independent parameter ziz_i, and let p=softmax(z)p=softmax(z). Fix a sampled group with zero-sum advantages ∑gA(g)=0 _gA^(g)=0 and perform ordinary gradient ascent on L(z)=G−1∑gA(g)logp((g))L(z)=G^-1 _gA^(g) p(y^(g)) by z(α)=z+α∇zL(z)z(α)=z+α _zL(z), using the same learning rate α for every coordinate. The gradient with respect to each logit and the corresponding entropy derivative are ∂L∂zi ∂ L∂ z_i =1G∑gA(g)(i=(g)−pi)=1G∑gA(g)i=(g), = 1G _gA^(g) (1\y_i=y^(g)\-p_i )= 1G _gA^(g)1\y_i=y^(g)\, (A.9) ∂H(p)∂zi ∂ H(p)∂ z_i =pi(−logpi−H(p)). =p_i(- p_i-H(p)). Therefore dH(softmax(z(α)))dα|α=0=∑i∂H(p)∂zi∂L∂zi=∑ge−sgA(g)G(sg−H(p))⏟hgSGD. . dH(softmax(z(α)))dα |_α=0= _i ∂ H(p)∂ z_i ∂ L∂ z_i= _g e^-s_gA^(g)G(s_g-H(p))_h_g^SGD. (A.10) Finite-dimensional smoothness also gives H(softmax(z(α)))−H(p)=α∑ghgSGD+O(α2)H(softmax(z(α)))-H(p)=α _gh_g^SGD+O(α^2). The probability factor e−sge^-s_g arises from the entropy gradient with respect to logits, not from weighting a sampled rollout a second time. If ∑gA(g)≠0 _gA^(g)≠ 0, Eq. (A.10) acquires the additional term −(G−1∑gA(g))∑ipi2(−logpi−H(p))- (G^-1 _gA^(g) ) _ip_i^2(- p_i-H(p)). The contribution hgSGDh_g^SGD is therefore an additive attribution within a zero-sum whole-group update, not the complete entropy derivative obtained by training one rollout in isolation; it also cannot be reused automatically after deletion or length weighting. Conditional dominance of a high-probability positive rollout. Assume A+>0>A−A_+>0>A_-, s+<H(p)<s−s_+<H(p)<s_-, and equal entropy contributions h+=h−<0h_+=h_-<0 in Proposition 1. By Eq. (2), this equality is equivalent to A+(H(p)−s+)=|A−|(s−H(p))>0A_+(H(p)-s_+)=|A_-|(s_--H(p))>0. By Eq. (A.10), both rollouts decrease entropy, and their contribution magnitudes satisfy |h+SGD||h−SGD|=e−s+A+(H(p)−s+)e−s−|A−|(s−H(p))=es−s+>1. |h_+^SGD||h_-^SGD|= e^-s_+A_+(H(p)-s_+)e^-s_-|A_-|(s_--H(p))=e^s_--s_+>1. (A.11) This proves Eq. (6). The comparison concerns two rollouts whose entropy-decrease magnitudes before probability weighting are equal; it neither requires equal absolute advantages nor says that every positive-advantage rollout is more influential than every negative one. Scale difference from the KL-proximal update. For the same sampled group with ∑gA(g)=0 _gA^(g)=0, Eq. (A.2) satisfies A~i=pi−1∂L/∂zi A_i=p_i^-1∂ L/∂ z_i. Eq. (A.3a) can therefore be written q(η)=softmax(z+ηA~)q^(η)=softmax(z+η A). The two logit-coordinate changes are ΔziKL=ηpi∂L∂zi,ΔziSGD=α∂L∂zi. z_i^KL= ηp_i ∂ L∂ z_i, z_i^SGD=α ∂ L∂ z_i. (A.12) These are logit representatives of the corresponding distributions; adding the same constant to every coordinate does not change softmax. Section 3.2 also uses one global parameter η, but relative to the ordinary logit gradient its effective scale is η/piη/p_i, whereas Section 3.3 uses the shared gradient multiplier α. The inverse-probability factor in the former cancels the pip_i in the entropy gradient; the latter retains that probability factor. This is a difference between update rules, not a difference between having and not having a step size, and it does not make the independent-logit model more accurate than the KL-proximal model. Scope of the conclusion. Proposition 3 is a first-order comparison under independent logits, zero-sum whole-group advantages, and the stated equal-magnitude condition. It is not a direct consequence of the KL-proximal update in Section 3.2 and does not establish positive-sample dominance for arbitrary finite steps or actual neural-network updates. It supplies additional motivation for the final positive-advantage filter. The maximum-cardinality property of Eq. (4) applies only to the initial retained set with fixed original-group scores; the filtering in Eq. (7) and recentering in Eq. (9) do not inherit its threshold constraint or any global optimality claim. These results are rigorous within their stated local models and guide method design, while the experiments in Sections 4.1 and 4.2 evaluate the complete method. They do not guarantee entropy preservation under shared-parameter AdamW updates. A.3 Proof of Proposition 4 First-order change of zero-advantage sequences. With the native advantages fixed and satisfying ∑g=1GA(g)=0 _g=1^GA^(g)=0, deleting a nonempty set D of positive-advantage rollouts leaves ∑g∈ℐkeepA(g)=−∑g∈A(g)<0 _g _keepA^(g)=- _g A^(g)<0. Retention depends on rollout content and within-group comparisons, so the retained samples cannot generally be treated as fresh i.i.d. draws from p. These changes motivate reconsidering the advantage baseline after selection, but do not by themselves determine the probability weights. Fix a prompt x, a realized group of rollouts sampled i.i.d. from p, a nonempty retained set ℐkeepI_keep, and a bounded post-deletion advantage A~ikeep A_i^keep. To isolate normalization effects, assume the local update qkeep,i(η)=piexp(ηA~ikeep)/Zkeep(η)q_keep,i^(η)=p_i (η A_i^keep)/Z_keep(η), where Zkeep(η)=∑jpjexp(ηA~jkeep)Z_keep(η)= _jp_j (η A_j^keep). This has the same KL-proximal form as Appendix A.1, but uses the accumulated retained advantages as its score rather than the sampling-aligned score in Eq. (A.2). The distribution-space update couples paths only through its normalizing denominator and contains no shared-parameter effects. Let U=i:pi>0,A~ikeep=0U=\i:p_i>0,\ A_i^keep=0\. The set U contains sequences that receive no retained signal and may also include sampled sequences whose recentered advantage is zero or whose signals cancel; it must not be identified with the set of unsampled sequences. Under this update, qkeep,i(η)=pi/Zkeep(η)q_keep,i^(η)=p_i/Z_keep(η) for i∈Ui∈ U. For i∈Ui∈ U, the unnormalized weight remains pip_i, while normalization multiplies all such probabilities by the same factor 1/Zkeep(η)1/Z_keep(η). Equivalently, every zero-advantage path receives the same additive log-probability offset −logZkeep(η)- Z_keep(η), regardless of its similarity to the trained rollouts, and probability ratios between zero-advantage paths remain unchanged. In particular, any two paths i,j∈Ui,j∈ U satisfy qkeep,i(η)/qkeep,j(η)=pi/pjq_keep,i^(η)/q_keep,j^(η)=p_i/p_j: normalization cannot preferentially favor the one more similar to a rewarded rollout. The i.i.d. assumption concerns the original sampling group. The derivation is conditional on the realized group and retained set and does not require the retained samples to remain i.i.d.; excluding transfer through shared parameters is a separate assumption of the local update model. Using Zkeep(0)=1Z_keep(0)=1 and Zkeep′(0)=p[A~keep]Z_keep (0)=E_p[ A^keep] gives dqkeep,i(η)dη|0=−pip[A~keep],dlogqkeep,i(η)dη|0=−p[A~keep],i∈U. . dq_keep,i^(η)dη |_0=-p_iE_p[ A^keep], . d q_keep,i^(η)dη |_0=-E_p[ A^keep], i∈ U. (A.13) This proves Eq. (8). At η=0η=0, the identical log-probability derivatives imply equal relative changes across zero-advantage paths, while their absolute changes are scaled by pip_i. When U is nonempty, every sequence in it has zero first-order probability change if and only if p[A~keep]=0E_p[ A^keep]=0. If U is empty, there are no such sequences and one cannot infer a zero mean from a condition on an empty set. The result does not imply finite-step invariance: for a nontrivial post-deletion advantage with zero mean, typically Zkeep(η)>1Z_keep(η)>1 for η>0η>0, so zero-advantage sequences can still change at second and higher orders. Normalization here occurs only over the response space of a fixed prompt and does not prove cross-prompt or OOD invariance after a shared-network update. Subset entropy and global entropy attribution. Let HU(q)=−∑i∈UqilogqiH_U(q)=- _i∈ Uq_i q_i be the unnormalized entropy of the subset U. Eq. (A.13) yields dHU(qkeep(η))dη|0=p[A~keep](∑i∈Upi−HU(p)),−∑i∈Uqkeep,i′(0)logpi=−p[A~keep]HU(p). . dH_U(q_keep^(η))dη |_0=E_p[ A^keep] ( _i∈ Up_i-H_U(p) ), - _i∈ Uq_keep,i (0) p_i=-E_p[ A^keep]H_U(p). (A.14) The left expression is the derivative of the unnormalized subset entropy itself. The right expression is the contribution assigned to U after the constant term has been canceled globally using ∑iqkeep,i′(0)=0 _iq_keep,i (0)=0. They must not be conflated, and the sign of p[A~keep]E_p[ A^keep] alone does not determine the direction of global entropy. The general global expression remains −Covp(A~keep,logp)-Cov_p( A^keep, p) and reduces to the empirical covariance in Eq. (3) only for the original zero-sum score specified in Appendix A.1. Which zero-mean condition probability-weighted recentering satisfies. For the nonempty retained set in Eq. (9), replacing the native advantages in the accumulated post-deletion advantage by the recentered advantages gives A~ikeep=∑g∈ℐkeepA¯keep(g)i=(g),p[A~keep]=∑g∈ℐkeepe−sgA¯keep(g)=0. A_i^keep= _g _keep A_keep^(g)1\y_i=y^(g)\, _p[ A^keep]= _g _keepe^-s_g A_keep^(g)=0. (A.15) The last equality follows directly from A¯keep(g)=A(g)−bkeep A_keep^(g)=A^(g)-b_keep and bkeep=∑g∈ℐkeepe−sgA(g)/∑g∈ℐkeepe−sgb_keep= _g _keepe^-s_gA^(g)/ _g _keepe^-s_g. Eq. (9) therefore eliminates the first-order normalization drift of zero-advantage sequences in the KL-proximal model driven by this particular post-deletion advantage. The probability weights arise from the old-policy expectation in Eq. (A.13): Eq. (9) imposes this weighted zero-mean condition on the retained score without assuming that it held before deletion. Subtracting a common baseline preserves pairwise differences between retained advantages; the weights calibrate the retained score rather than reconstruct the original sampling distribution through importance weighting. Advantage contributions are accumulated by sampled index; when sequences repeat, the denominator ∑g∈ℐkeepe−sg _g _keepe^-s_g counts each sampled occurrence and need not equal the probability mass of the distinct retained sequences. Appendix B Experimental Details B.1 Experimental Setup Training settings. We conduct experiments on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B (Team, 2025), using DAPO-17K mathematics data (Yu et al., 2025) for reinforcement learning. All model sizes use the same training configuration and are trained for one epoch. Each training batch contains 128 prompts with eight rollouts per prompt, totaling 1,024 candidate rollouts before selection. The maximum prompt and response lengths are 2,048 and 8,192 tokens, respectively, and overlong prompts are filtered. Rollouts are sampled with vLLM at temperature 1.0 and top-p 1.0. Original GRPO uses all eight rollouts; GRPODropout removes selected rollouts and recenters the retained advantages before updating the policy. We use AdamW with learning rate 10−610^-6 and weight decay 0.01, a clipping width of 0.2, and a policy loss averaged over valid response tokens. The original GRPO baseline uses no additional entropy bonus or reference-policy KL regularization. Baselines. We use original GRPO (Shao et al., 2024) as the base control and compare representative methods from the two categories discussed in Related Work. The first category selects or masks token-level training signals: Beyond8020 (Wang et al., 2025b) optimizes only the approximately 20% highest-entropy tokens, while Clip-Cov (Cui et al., 2025) masks updates for a subset of high-covariance tokens. The second category modifies the RL objective or update rule: KL-Cov (Cui et al., 2025) applies KL constraints to selected high-covariance tokens, while DAPO (Yu et al., 2025) uses Clip-Higher to relax the upper clipping bound and facilitate exploration of low-probability actions. These baselines cover token-level data usage, regularization, and clipping, allowing us to compare complete-rollout selection with different strategies for mitigating entropy collapse. GRPODropout operates on data usage, applying the original RL loss to retained rollouts with recentered advantages while preserving its objective form. Evaluation settings. We evaluate on ten benchmarks: GSM8K (Cobbe et al., 2021), MATH-500 (Lightman et al., 2023), OlympiadBench (He et al., 2024), AMC23, AIME24, AIME25, and HMMT25 for mathematical reasoning, and ARC-C (Clark et al., 2018), MMLU-Pro (Wang et al., 2024), and SuperGPQA (Team et al., 2025) for general question answering. The latter evaluates generalization beyond the mathematics training domain. All evaluations use a maximum generation length of 8,192 tokens, temperature 0.6, top-p 0.95, top-k 20, and a fixed random seed. GSM8K, MATH-500, OlympiadBench, and the three QA benchmarks use one sampled response per question and report Mean@1. For the smaller AMC23, AIME24, AIME25, and HMMT25 test sets, we sample 32 responses per question through repeated single-response requests and report Mean@32, averaging correctness over the 32 responses and then over questions. Mathematical prompts request step-by-step reasoning and a final answer in . QA prompts request a final option letter, which is extracted, case-normalized, and compared with the reference label. Avg. is the unweighted mean accuracy across all ten benchmarks. B.2 Diversity-analysis protocol This subsection records the protocol used to compute the path-diversity columns in Table 3. The analysis covers 130 unique questions in AMC23 (40), AIME24 (30), AIME25 (30), and HMMT25 (30), with 32 sampled responses per question and system. Pass@32 is computed per question within each benchmark and then averaged equally across AMC23, AIME24, AIME25, and HMMT25. The Entropy column uses token-mean actor entropy over valid response tokens at the final checkpoint, averaged equally across the four benchmarks. All path-diversity aggregates are unweighted means over questions. Question partition and answer dispersion. Let cm,qc_m,q be the number of correct responses among the 32 rollouts for method m on question q. We define the common high-correctness set as ≥8=⋂mq:cm,q≥8,|≥8|=65,Q_≥ 8= _m\q:c_m,q≥ 8\, |Q_≥ 8|=65, and the common low-correctness set as <8=⋂mq:cm,q<8,|<8|=37.Q_<8= _m\q:c_m,q<8\, |Q_<8|=37. Thus the threshold is based on the original rollout correctness counts, and the same question sets are used for every method. The remaining 28 questions have mixed membership across systems and are not used in the two split columns. For each method and question, we compute Wm,q=|strip(ai):ri=0,strip(ai)≠∅|,W_m,q= | \strip(a_i):r_i=0,\ strip(a_i)≠ \ |, where aia_i is the extracted final answer and rir_i is its correctness indicator. W measures answer-string dispersion among incorrect final answers and can reveal recurring answer patterns. Summary generation and embedding. For each rollout, the summarizer receives the original question and the full generated response after removing explicit <think> tags. We use the model GLM-5.2, temperature 00, with thinking disabled. The summarizer is instructed to retain only the reasoning path that actually leads to the final boxed answer, place abandoned ideas in a separate list, and return strict JSON. Its exact prompt is: GLM-5.2 summarizer prompt You compress ONE chain-of-thought solution to a math problem. The trace is a stream-of-consciousness: the solver often floats several ideas, tries some, abandons most, and finally carries ONE line of reasoning through to the boxed answer. Your job is to separate the two: - FINAL PATH: the reasoning that was ACTUALLY carried through to produce the final (boxed) answer. This is the solution’s real solving path. Identify it by looking at how the last, answer-producing derivation actually went -- NOT by what was merely proposed earlier. - ATTEMPTED: ideas that were floated, tried, then dropped / not used for the final answer. Record each briefly with why it was abandoned. Be faithful. Do NOT invent steps. If the solver kept switching, the FINAL PATH is the one whose result equals the boxed answer at the very end. Output STRICT JSON only: "final_path": "summary": "<3-5 sentences: the actual path that yielded the answer>", "method": "<short tag: main machinery, e.g. cartesian-algebra / polar-form / calculus / geometric-series-identity>", "angle": "<short tag: the entry point / first decisive framing that led to the answer>", "key_trick": "<short tag: the crucial simplification/insight used, or ’none’>" , "attempted": [ "idea": "<short: an approach that was tried>", "why_abandoned": "<short>" ] The returned summary is encoded with the local bge-m3 checkpoint with L2 normalizations, and float32 arithmetic. For each question, Summary distance is the mean pairwise cosine distance over all 32 summary embeddings. Vendi is computed from the eigenvalues of the symmetrized Gram matrix divided by 32 as exp(−∑jλjlogλj) (- _j _j _j). Both measures therefore use all 32 summaries, including correct and incorrect responses.