Paper deep dive
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
Outongyi Lv, Yuanwei Zhang, Xiaoqun Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.
Tags
Links
- Source: https://arxiv.org/abs/2608.30632v1
- Canonical: https://arxiv.org/abs/2608.30632v1
Trouble viewing inline? Open PDF directly →
Full Text
59,629 characters extracted from source content.
Expand or collapse full text
darkbluergb0,0,0.5 GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning Outongyi Lv Email: harrylv@sjtu.edu.cn Yuanwei Zhang Email: sjtuzyw@sjtu.edu.cn Xiaoqun Zhang Affiliation: School of Mathematical Sciences, Shanghai Jiao Tong University Email: xqzhang@sjtu.edu.cn Affiliation: Institute of Natural Sciences, Shanghai Jiao Tong University Abstract Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models’ (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy–gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.11 1 Our code is available at https://github.com/outongyiLv/GMTS 11footnotetext: The first two authors contributed equally and are listed alphabetically.22footnotetext: Corresponding author. 1 Introduction Large language models (LLMs) have recently shown remarkable capabilities across diverse domains, such as mathematics, image quality assessment, speech, and multimodality 2023gpt,wu2025visualquality,shen2025vlm,bie2026llada2. Reinforcement learning (RL) has made substantial contributions to the advancement of LLMs, demonstrating outstanding effectiveness in enhancing performance 2022training,lee2024rlaif,shinn2023reflexion. DeepSeek introduced Group Relative Policy Optimization (GRPO) 2024deepseekmath, a method that dispenses with the value model in favor of simple rule-based rewards and adopts group-level advantage estimation, leading to significant improvements on reasoning benchmarks. Based on GRPO, several variants, such as Dynamic sAmpling Policy Optimization (DAPO) 2025dapo and Group Sequence Policy Optimization (GSPO) 2025group, have been proposed, showing that RLVR provides a scalable and efficient paradigm for advancing the reasoning abilities of LLMs. Nevertheless, these methods adopt a uniform objective function across all tokens, without fully accounting for the fact that different tokens contribute unevenly to RLVR. Figure 1: Overall architecture of the GMTS framework: High-entropy tokens selected by ETS may contribute less to the token gradient or be affected by gradient clipping. By computing δi,t _i,t, GMTS mitigates this problem. Moreover, GMTS can identify low-entropy tokens that are overlooked by ETS but still contribute more on token gradient. Recent studies have focused on distinguishing the token importance by designing token-level advantage functions 2025not,liu2026gdpo, ma2026fipo, introducing new objective functions 2025entropy,li2025cure, and filtering out low importance tokens 2025beyond,lv2026tokens. Concurrently, several studies have also emphasized the central role of high token entropy in enhancing RLVR 2025beyond,cui2025entropy,mengsparse, and controlling entropy provides a way to strengthen the RLVR reasoning ability. In these works, 2025beyond shows that training with only the top 20% high-entropy tokens yields substantially better performance across the reasoning tasks than training with all tokens. However, can entropy alone sufficiently indicate token importance? In this work, we observe that within a single answer, high-entropy tokens tend to be associated with large gradient magnitude. However, due to variations in reward signals and sample-specific characteristics, entropy alone fails to adequately capture token importance across different answers, whereas gradient magnitude provides a more reliable quantification. Based on this, we propose the Gradient Magnitude-based Token Selection (GMTS) method. In contrast to Entropy-based Token Selection (ETS) 2025beyond under the same settings, GMTS can more effectively distinguish token importance across different answers. Empirically, our experiments show that GMTS consistently improves performance across diverse reasoning tasks and model sizes. For example, GMTS achieves approximately 1-3 percentage point gains on math reasoning benchmarks. Furthermore, GMTS is also simple to implement and can be easily integrated into existing RLVR frameworks such as GRPO and DAPO. Since all the required components are already available during standard training, GMTS introduces minimal computational overhead. In summary, the contributions of this work are threefold: • Establish the relation between token entropy and gradient magnitude. We observe that within a single answer, tokens with high entropy correlate strongly with large gradient magnitude, which explains why entropy is an effective indicator of token importance. However, we further demonstrate that this correlation does not hold consistently across different answers when their reward signals vary significantly. This motivates the use of gradient magnitude as a more robust and reliable metric for quantifying token importance. • GMTS for quantifying token importance. Building upon the relationship between entropy and gradient magnitude, we propose GMTS for quantifying token importance. GMTS effectively retains the benefits of using entropy as a signal while overcoming its key limitations in quantifying token importance across different answers. • Empirical studies across reasoning domains and model scales. Our empirical studies show that GMTS consistently outperforms the baselines across three reasoning domains and multiple model scales. Detailed experimental results and settings are provided in Section 4. These results demonstrate the effectiveness of GMTS in improving RLVR performance, as well as its adaptability across different tasks and model scales. 1.1 Related Works Token-level advantage estimation. chen2025seed leverage the differences between positive and negative advantage-induced token-level signals, together with entropy, to redistribute token-level advantages. 2025gtpo assign entropy-weighted rewards to emphasize high-entropy tokens and down-weight low-entropy ones, mitigating model over-updating. At the same time, 2026reasoning show that high-entropy tokens correlate with exploration-facilitating behaviors and directly incorporate entropy into advantage estimation. 2025emergent focus on high-impact strategy tokens in reasoning and amplifies their learning signals by modifying the advantages, while 2026fipo use token-level future KL to reassess each token’s contribution. Objective function design. wang2025stabilizing introduce an entropy-aware clipping strategy that applies stricter constraints to low-entropy tokens while relaxing updates for high-entropy ones. 2025entropy observe that decreasing entropy can lead to entropy collapse and address this issue by dynamically adjusting the entropy regularization strength. Building on this direction, 2025cure identify critical positions using high-entropy tokens, resample them to construct branched trajectories, and jointly train RLVR on both the original and branched trajectories. In addition, 2025diversity systematically study diversity in RLVR reasoning and incorporate diversity-aware training into the RLVR objective. Low importance token filtering. wang2025beyond find that high-entropy tokens often serve as critical forking thinking tokens, and leverage this observation by selecting only the top 20% entropy tokens for RLVR training, achieving significant performance gains. GMTS also lies in this direction, filtering tokens with low gradient magnitude and using remaining tokens in RLVR training. 2 Preliminaries 2.1 LLM Formulation and Notations Let =x1,x2,…,xVV=\x^1,x^2,…,x^V\ denote the finite vocabulary of tokens, where V is the size of the vocabulary. Given an input query q, an LLM parameterized by θ defines an autoregressive policy πθ _θ that generates an output sequence =(o1,…,oT) o=(o_1,…,o_T) token by token. At step t, conditioned on the query and the previously generated tokens <t o_<t, the model predicts a distribution over the vocabulary πθ(⋅∣,<t)=[pt,1,pt,2,…,pt,V] _θ(· q, o_<t)=[p_t,1,p_t,2,…,p_t,V], from which the next token oto_t is sampled or selected. This process continues until an end-of-sequence token is generated or the maximum generation length is reached. We define the entropy of the predictive distribution at step t as Et:=−∑k=1Vpt,klogpt,k.E_t:=- _k=1^Vp_t,k p_t,k. This notation will be used throughout our method. 2.2 RLVR Methods This section briefly reviews two representative RLVR methods: GRPO and DAPO. GRPO 2024deepseekmath is a PPO-style method 2017proximal that estimates advantages from a group of sampled responses, thereby removing the need for a separate value model. Given a query ∼ q , GRPO samples G answers [1,2,…,G][ o_1, o_2,…, o_G] from πold _old, with rewards R=[R1,R2,…,RG]R=[R_1,R_2,…,R_G] to compute the advantage Ai=Ri−mean(R)std(R)A_i= R_i-mean(R)std(R). This response-level advantage is assigned to all tokens in i o_i, i.e., Ai,t=AiA_i,t=A_i. The optimization objective of GRPO is: maxθ[1G∑i=1G1|i|∑t=1|i|ℓi,t(θ)], _θE_ q [ 1G _i=1^G 1| o_i| _t=1^| o_i| _i,t(θ) ], (1) where ℓi,t(θ) _i,t(θ) denotes the token-level PPO-style objective: ℓi,t(θ):= _i,t(θ):= min(clip(ri,t(θ),1−ϵ1,1+ϵ2)Ai,t, (clip(r_i,t(θ),1- _1,1+ _2)A_i,t, OPENri,t(θ)Ai,t)−β⋅i,tKL(θ). r_i,t(θ)A_i,t )-β·D^KL_i,t(θ). In this expression, i,tKL(θ)D^KL_i,t(θ) denotes the KL divergence between the current and reference predictive distributions πθ(⋅∣q,<t) _θ(· q, o_<t) and πref(⋅∣q,<t) _ref(· q, o_<t), πref _ref is the reference policy and ri,t(θ)=πθ(oi,t|,i,<t)/πold(oi,t|,i,<t)r_i,t(θ)= _θ(o_i,t| q, o_i,<t)/ _old(o_i,t| q, o_i,<t). β is the parameter for the KL penalty term and ϵ1,ϵ2 _1, _2 are clipping hyperparameters. In practice, following 2020approximating, we estimate the per-token KL penalty using a single-sample unbiased estimator. DAPO builds on GRPO with several practical modifications 2025dapo. Compared with GRPO, DAPO removes the KL penalty from the objective and adopts dynamic sampling to discard uninformative groups in which all responses are either correct or incorrect. It also uses token-level loss averaging and a larger upper clipping threshold ϵ2 _2 to improve training stability. 3 Methodology Figure 2: We use DAPO to train Qwen2.5-math-1.5B on MATH-1212K dataset under the setting of G=16G=16. At training step 100100, we selected two answers with different advantages in one question group, and computed the gradients ∇θℓi,t(θ) _θ _i,t(θ), ∇θlogπθ(oi,t) _θ _θ(o_i,t), and entropy Ei,tE_i,t of all the tokens. With these collected data, we plotted entropy against the token-level log-probability gradients under the log scale (left), entropy against the true gradient magnitude under the log scale (middle), and the distributions of true gradient magnitude together with their 20th percentile boundaries (right). Figure 3: We select 10 questions from each of MATH-500 and KodCode, and evaluate Qwen2.5-Math and Qwen2.5-Coder at the 1.5B and 7B scales on the math and code benchmarks, respectively. For each question, we generate 16 responses. Within each group of responses to the same question, we compute the Pearson correlation coefficient between log(Ei,t) (E_i,t) and log(‖∇θlogπθ(oi,t)‖2) (\| _θ _θ(o_i,t)\|_2 ) (left), and the Spearman rank correlation coefficient between the GMTS score |Ei,t⋅ωi,t(θ)||E_i,t· _i,t(θ)| and the true gradient magnitude ‖∇θℓi,t(θ)‖2\| _θ _i,t(θ)\|_2 (middle). We further plot the distribution of ε over all 160 generated responses on MATH-500 (right). However, both GRPO and DAPO treat tokens in the same response equally in training objectives by assigning the same response-level advantage to all tokens, thereby ignoring their unequal contributions to policy updates. In this section, we first analyze token-level differences from the perspective of gradient magnitude and connect them to token entropy. We then introduce GMTS, a gradient-magnitude-based token selection method for RLVR training. Gradient of each token. Following 2025not, the gradient of the token-level objective ℓi,t(θ) _i,t(θ) can be expressed as a scalar coefficient times the score function: ∇θℓi,t(θ)=ωi,t(θ)⋅∇θlogπθ(oi,t), _θ _i,t(θ)= _i,t(θ)· _θ _θ(o_i,t), where we use πθ(oi,t) _θ(o_i,t) to represent πθ(oi,t|,i,<t) _θ(o_i,t| q, o_i,<t) for ease of exposition. The coefficient ωi,t(θ) _i,t(θ) is given by ωi,t(θ) _i,t(θ) =ri,t(θ)Ai,t⋅ϵ1,ϵ2(ri,t(θ),Ai,t) =r_i,t(θ)A_i,t·I_ _1, _2(r_i,t(θ),A_i,t) +βπref(oi,t)πθ(oi,t)−β. +β _ref(o_i,t) _θ(o_i,t)-β. Here, ϵ1,ϵ2(⋅,⋅)I_ _1, _2(·,·) denotes the clipping indicator function: ϵ1,ϵ2(a,b)=0if b>0,a>1+ϵ2,0if b<0,a<1−ϵ1,1otherwise.I_ _1, _2(a,b)= \ array[]l0&if b>0,\ a>1+ _2,\\ 0&if b<0,\ a<1- _1,\\ 1&otherwise. array . The decomposition of ∇θℓi,t(θ) _θ _i,t(θ) shows that ωi,t(θ) _i,t(θ) collects the effective token-level learning signals in RLVR, including the advantage value, clipping, and KL regularization terms. Therefore, it directly scales the policy gradient at each token and determines how strongly that token contributes to the overall parameter update. In DAPO, the KL term is omitted, so ωi,t(θ) _i,t(θ) reduces to ri,t(θ)Ai,tϵ1,ϵ2(ri,t(θ),Ai,t)r_i,t(θ)A_i,tI_ _1, _2(r_i,t(θ),A_i,t). When πθ _θ remains close to πold _ old and the clipping constraint is inactive, we have ri,t(θ)≈1r_i,t(θ)≈ 1 and hence ωi,t(θ) _i,t(θ) is approximately Ai,tA_i,t. Token gradient magnitude and entropy. As illustrated in the left panel of Figure 2, the magnitude of the token-level log-probability gradient ∇θlogπθ(oi,t|,<t) _θ _θ(o_i,t| q, o_<t) exhibits an approximately linear relationship with token entropy when both quantities are plotted on a logarithmic scale. To better understand this phenomenon, we analyze the gradient structure in detail. Consider a query input q and an output sequence o, and focus on the t-th generated token oto_t. Let t=[zt,1,zt,2,…,zt,V] z_t=[z_t,1,z_t,2,…,z_t,V] denote the logits produced by the final layer of the LLM at time step t, and let πθ(⋅∣,<t)=Softmax(t):=[pt,1,pt,2,…,pt,V] _θ(· q, o_<t)=Softmax( z_t):=[p_t,1,p_t,2,…,p_t,V] be the resulting predictive distribution over the vocabulary. Proposition 3.1. Let GtG_t be the ℓ2 _2-norm of the gradient of the log-probability logπθ(ot|,<t) _θ(o_t| q, o_<t) with respect to the logits ztz_t, EtE_t is the entropy of distribution πθ(⋅|,<t) _θ(·| q, o_<t). Let 1−πθ(ot|,<t)=ε1- _θ(o_t| q, o_<t)= , then we have: limε→0logGtlogEt=1. _ → 0 G_t E_t=1. The detailed derivation for Proposition 3.1 is provided in Section A.2. It establishes a direct connection between token entropy and the magnitude of the log-probability gradient with respect to the logits. As shown by the distribution of ε on the right panel of Figure 3, most tokens have ε values close to 00. This provides a theoretical explanation for the linearity observed in Figure 2, as well as its slope being close to 11. Furthermore, let ∂t∂θ ∂ z_t∂θ denote the Jacobian of the logits with respect to the model parameters. By the chain rule ∇θlogπθ(ot∣,<t)=∂t∂θ⋅∇tlogπθ(ot∣,<t) _θ _θ(o_t q, o_<t)= ∂ z_t∂θ· _ z_t _θ(o_t q, o_<t), which implies that tokens with higher entropy tend to induce larger gradients with respect to the model parameters. This analysis provides a principled explanation for the empirical observation that high-entropy tokens may contribute more significantly to parameter updates in RLVR training 2025beyond. Gradient magnitude token selection. However, the coefficients ωi,t(θ) _i,t(θ) in the token gradients can exhibit substantial variability and are particularly influenced by the clipping constraints and the advantage values derived from answer-level reward signals. As shown in the middle and right panels of Figure 2, tokens belonging to answers with different advantages display notable differences in gradient magnitudes, along with distinct percentile distributions. These observations suggest that entropy alone is insufficient to reflect a token’s contribution in RLVR training. To address this limitation, we propose the GMTS method, which assesses token importance directly from the gradient magnitude. Since directly computing the true gradient norm is computationally intractable during RL training, we approximate ∇θlogπθ(oi,t|,i,<t) _θ _θ(o_i,t| q, o_i,<t) using token entropy. And we define the GMTS score of token oi,to_i,t as: δi,t=|Ei,t⋅ωi,t(θ)|, _i,t=|E_i,t· _i,t(θ)|, which serves as a tractable measure of token importance. Based on that, the training objective of GMTS is as follows: maxθ[1Sρ∑i=1G∑t=1|i|[δi,t≥τρ]⋅ℓi,t(θ)]. _θE_ q [ 1S_ρ _i=1^G _t=1^| o_i|I[ _i,t≥ _ρ]· _i,t(θ) ]. (2) where (⋅)I(·) is the indicator function that evaluates to 11 if the condition inside holds and 00 otherwise. ρ is a predefined ratio specifying the top proportion to be selected, and τρ _ρ is the corresponding threshold such that only tokens with δi,t(θ)≥τρ _i,t(θ)≥ _ρ, comprising the top-ρ fraction of all tokens in the batch, are used to compute the gradient, SρS_ρ denotes the number of selected tokens in the group. Thus the loss is averaged only over the selected tokens. From a computational perspective, calculating ωi,t(θ) _i,t(θ) involves the advantage signal Ai,tA_i,t and the predicted probability from πθ,πold _θ, _old and πref _ref for each token. These quantities are already available during standard RLVR training. Therefore, the additional computational overhead introduced by GMTS is minimal. Moreover, GMTS can be readily integrated into other RLVR training frameworks, with the only difference being the specific formulation of the coefficient ωi,t(θ) _i,t(θ). By leveraging only a small subset of influential tokens for RLVR updates, GMTS offers the potential for more efficient RLVR training. Comparison with ETS. Both ETS and GMTS retain only the top-ρ fraction of tokens for policy updates, but they differ in how token importance is measured. ETS ranks tokens solely by entropy, whereas GMTS scores each token by δi,t _i,t, which incorporate the effective scalar coefficient in the token gradient. This difference matters in three aspects: (1) Tokens with similar entropy can have very different effects when they belong to responses with different advantages. As shown in the middle panels of Figure 2, the same level of entropy may correspond to distinct gradient magnitudes due to different response-level reward signals. (2) PPO-style clipping can suppress the policy-gradient term when the probability ratio falls outside the clipping range. In such cases, ETS may still select high-entropy tokens whose gradients are inactive or largely reduced, while GMTS downweights them through the clipping indicator in ωi,t(θ) _i,t(θ). (3) When KL regularization is used, the effective gradient is also affected by the reference-policy correction term, which is not reflected by entropy alone. Thus, GMTS prioritizes tokens that contribute more directly to the actual RLVR update. 4 Numerical Experiments 4.1 Training and Evaluation Settings We implement and evaluate GMTS and ETS within the verl framework22 2 https://github.com/volcengine/verl across GRPO and DAPO. Furthermore, we provide a standalone implementation33 3 https://github.com/policy-gradient/GRPO-Zero that not only gives a clear understanding of GMTS but also supports training under limited GPUs. In our experiments, unless otherwise specified, both ETS and GMTS are trained using the Top 20% of tokens, consistent with the selection ratio recommended by ETS 2025beyond. Training Details. We evaluate GMTS across three distinct domains: MATH, CODE, and COMMONSENSE (CS). Primary results are reported for the 1.5B and 7B model variants, with additional experiments conducted on the 8B model specifically for MATH domain. We use the following training sets: For MATH domain, we utilize the MATH-12K 2023let dataset (1.5B and 7B) and DAPO-MATH-17K 2025beyond (8B). For CODE domain, we utilize the KodCode 2025kodcode. For CS domain, we utilize the CS-QA 2019commonsenseqa. Other details for training on these tasks are all summarized in Section A.3. Evaluation Details. For MATH domain, we evaluate the 1.5B and 7B models on five benchmarks: AIME2024, AMC23, MATH-500, Minerva, and OlympiadBench. For the 8B model, we additionally report results on AIME2025, since it is particularly challenging for small models (1.5B and 7B). For CODE domain, we utilize standard benchmarks widely adopted in the Qwen3-Coder evaluation 44 4 https://github.com/QwenLM/Qwen3-Coder/: LiveCodeBench (202407 – 202411), MBPP, HumanEval, and Bigcode-Bench. For CS domain, we utilize the test of CS-QA and more general CS-QA2 2022commonsenseqa. For MATH and CS domains, we generate 16 candidate answers for each question and report the average accuracy as average@16. All generations are performed with the temperature of T=1.0T=1.0. For CODE domain, we employ the greedy strategy for evaluation. Figure 4: Overall performance comparison. GMTS consistently outperforms GRPO and DAPO baselines as well as the ETS method across multiple model scales (1.5B and 7B). 4.2 Main Results In Figure 4, we illustrate the global performance of ETS and GMTS across different domains. Next, we show more details of these results on each domain. 4.2.1 Results on MATH Domain We implement ETS/GMTS on GRPO/DAPO, and apply them to the Qwen2.5-math-1.5B and 7B model. All the detailed results are in Tables 1 and 2. Table 1: Evaluation on 5 math-reasoning benchmarks (ignore AIME2025) for ETS/GMTS with DAPO/GRPO on Qwen2.5-math-1.5B under average@16 accuracy (%). AIME2024 AMC23 MATH-500 Minerva OlympiadBench Avg. Qwen2.5-math-1.5B 2.702.70 18.7518.75 23.4023.40 6.206.20 14.1014.10 13.0313.03 DAPO 14.17 50.6250.62 73.85 29.4729.47 37.17 41.06 + ETS 14.17 48.4448.44 72.2572.25 29.0129.01 36.1636.16 40.0140.01 + GMTS 14.58 50.63 73.83 30.51 38.23 41.56 GRPO 10.6310.63 48.1248.12 71.5971.59 28.8128.81 34.9034.90 38.8138.81 + ETS 12.71 48.8948.89 72.39 28.4128.41 35.5435.54 39.5939.59 + GMTS 11.8811.88 53.33 73.06 30.11 36.09 40.89 Table 2: Evaluation on 5 math-reasoning benchmarks (ignore AIME2025) for ETS/GMTS with DAPO/GRPO on Qwen2.5-math-7B under average@16 accuracy (%). AIME2024 AMC23 MATH-500 Minerva OlympiadBench Avg. Qwen2.5-math-7B 7.297.29 21.0921.09 34.4534.45 6.096.09 10.9210.92 15.9715.97 DAPO 18.1218.12 62.5062.50 80.91 36.4936.49 44.35 48.4748.47 + ETS 20.00 63.75 80.0480.04 36.7236.72 43.5643.56 48.8148.81 + GMTS 25.00 65.16 80.4080.40 37.39 42.7542.75 50.14 GRPO 27.08 62.3462.34 79.7679.76 35.1835.18 42.87 49.45 + ETS 19.1719.17 60.6760.67 77.7277.72 33.6533.65 40.9640.96 46.4346.43 + GMTS 23.3323.33 64.83 81.13 36.32 43.60 49.84 Table 3: Evaluation on 6 math-reasoning benchmarks for ETS/GMTS with DAPO on Qwen3-8B under average@16 accuracy (%). AIME2024 AIME2025 AMC23 MATH-500 Minerva OlympiadBench Avg. Qwen3-8B 3.753.75 6.046.04 33.9133.91 62.9962.99 22.2922.29 23.6623.66 25.4425.44 DAPO 33.3333.33 25.4225.42 77.8177.81 89.2489.24 39.7739.77 56.6756.67 53.7153.71 + ETS 34.5834.58 26.2526.25 77.1977.19 89.7089.70 40.2640.26 57.43 54.2354.23 + GMTS 39.79 30.00 79.38 91.14 42.42 53.7653.76 56.08 Tables 1 and 2 show that GMTS Top (20%) consistently outperforms ETS across both DAPO and GRPO backbones (1.5B-DAPO: +1.55, 1.5B-GRPO: +1.30, 7B-DAPO: +1.33, 7B-GRPO: +3.41). To evaluate the performance of GMTS on larger reasoning models, we extended our experiments to Qwen3-8B, using results from 2025beyond as a baseline. As shown in Table 3, GMTS consistently delivers notable improvements over ETS across a wide range of math benchmarks, with substantial improvements in challenging tasks such as AIME2024 (+5.21) and AIME2025 (+3.75). With an overall average gain of 1.85%, these results demonstrate the generality of GMTS, suggesting it has the potential to scale effectively to larger and more complex reasoning models. 4.2.2 Results on CODE and CS Domains We implement ETS/GMTS on GRPO/DAPO, for code domain, we apply them to the Qwen2.5-coder-1.5B and 7B model. For commonsense domain, we apply them to the Qwen2.5-1.5B-base and 7B model. All the detailed results are in Tables 4 and 5. Table 4: Evaluation on 4 code-reasoning and 2 CS benchmarks for ETS/GMTS with DAPO/GRPO on Qwen2.5-coder-1.5B under greedy accuracy (%) (left) and Qwen2.5-base-1.5B under average@16 accuracy (%) (right). LiveCodeBench HumanEval MBPP Bigcode-Bench Avg. 1.5B Base Model 1.331.33 22.0422.04 12.2112.21 4.204.20 9.959.95 DAPO 11.0211.02 72.5072.50 73.1473.14 26.3126.31 45.7445.74 + ETS 9.009.00 73.5173.51 73.1273.12 23.2223.22 44.7144.71 + GMTS 11.0211.02 74.72 73.49 27.06 46.58 GRPO 10.3110.31 72.9172.91 74.1474.14 25.1025.10 45.6245.62 + ETS 9.009.00 72.9272.92 71.3471.34 25.7125.71 44.7444.74 + GMTS 11.02 75.03 74.21 26.33 46.64 CS-QA CS-QA2 Avg. 10.2610.26 4.144.14 7.207.20 77.2177.21 52.7352.73 64.9764.97 78.0978.09 52.8252.82 65.4665.46 79.15 53.51 66.33 77.4377.43 51.0751.07 64.2564.25 79.8279.82 50.9250.92 65.3765.37 79.95 52.16 66.04 Table 5: Evaluation on 4 code-reasoning and 2 CS benchmarks for ETS/GMTS with DAPO/GRPO on Qwen2.5-coder-7B under greedy accuracy (%) (left) and Qwen2.5-base-7B under average@16 accuracy (%) (right). LiveCodeBench HumanEval MBPP Bigcode-Bench Avg. 7B Base Model 3.203.20 36.3336.33 39.9239.92 13.5013.50 23.2423.24 DAPO 19.9419.94 85.1085.10 79.0279.02 36.8436.84 55.2055.20 + ETS 19.0619.06 84.3984.39 77.1877.18 38.42 54.7654.76 + GMTS 20.13 85.62 79.89 36.3736.37 55.50 GRPO 20.0020.00 84.2184.21 76.7176.71 36.3136.31 54.3154.31 + ETS 17.3117.31 85.4685.46 77.0277.02 37.98 54.4454.44 + GMTS 20.60 85.91 77.43 36.5836.58 55.13 CS-QA CS-QA2 Avg. 31.5631.56 14.8514.85 23.2123.21 84.7084.70 66.5766.57 75.6475.64 83.4483.44 66.0966.09 74.7774.77 85.65 66.95 76.30 84.0084.00 62.3962.39 73.2073.20 84.1784.17 63.8263.82 74.0074.00 84.60 65.92 75.26 Figure 5: The average@16 performance of DAPO, DAPO + ETS Bottom (80%, 90%) and DAPO + GMTS Bottom (80%, 90%) on Qwen2.5-math-1.5B (left) and the average@16 overall performance of varying selected ratios for DAPO, DAPO + ETS Top, and DAPO + GMTS Top on Qwen2.5-math-1.5B (right). For HumanEval and MBPP, which consist of Base and Plus subsets, the table reports the average accuracy across both. Similarly, for BigCode-Bench, the reported values represent the mean accuracy across the Full and Hard subsets. As shown in Tables 4 and 5, GMTS achieves consistent performance gains within the code domain (1.5B-DAPO: +1.87, 1.5B-GRPO: +1.90, 7B-DAPO: +0.74, 7B-GRPO: +0.69). Furthermore, in the commonsense domain (1.5B-DAPO: +0.87, 1.5B-GRPO: +0.67, 7B-DAPO: +1.53, 7B-GRPO: +1.26). These results validate the generalization of the GMTS method across diverse reasoning tasks. 4.3 Ablation Study Linearity and monotonicity consistency. We evaluated the linear and monotonic behavior of GMTS using Qwen2.5-math and Qwen2.5-coder at both the 1.5B and 7B scales. As shown in Figure 3 right and middle panels, the linear relationship consistently holds across domains and model scales, indicating that it is not an isolated empirical artifact. Moreover, GMTS exhibits strong monotonicity, suggesting that gradient-magnitude-based selection preserves the ordering of token contributions. Bottom selection. To examine the role of low gradient magnitude tokens, we follow 2025beyond to evaluate DAPO with bottom selection on Qwen2.5-math-1.5B. As shown in Figure 5 left and Table 6, GMTS consistently performs worse than ETS, with average drops of -1.08 and -1.28 at the 80% and 90% ratios, respectively. This suggests that low gradient magnitude tokens contribute less to RLVR training, whereas low-entropy tokens may still contain useful signals. Sensitivity to selection ratios. To evaluate the impact of selection intensity, we evaluate both DAPO and GRPO on Qwen2.5-math-1.5B with ratios 0.1, 0.2, 0.5, 0.7, and 0.9, with the results reported in Table 7 and Figure 5 right. It shows that GMTS consistently outperforms ETS in nine out of ten configurations. These findings suggest that the advantages of GMTS are not confined to a specific regime but extend across a broad spectrum of training conditions, providing a more reliable selection mechanism than ETS. 5 Conclusion In this work, we propose GMTS for improving RLVR in LLM reasoning. Although gradient magnitude is correlated with entropy within one answer, entropy alone cannot fully capture token importance across different answers, where the contribution of each token to policy update can vary substantially. GMTS addresses this limitation by leveraging a gradient-magnitude proxy for token selection. Empirically, GMTS achieves consistent improvements over ETS and remains robust across multiple selection ratios, model scales, and evaluation domains. These results provide empirical evidence supporting GMTS as an effective and efficient choice for RLVR. Limitations Despite the consistent performance gains of GMTS in RLVR, several limitations remain and point to directions for future research. First, although extensive empirical evaluations support the effectiveness of GMTS, its theoretical foundation remains only partially understood. In this work, we provide preliminary evidence on the relationship between token entropy and gradient magnitude, but a rigorous theoretical justification for why GMTS outperforms ETS, along with a deeper analytical characterisation of its behaviour, remains an important direction for future work. Second, due to computational constraints, we have not yet evaluated GMTS on larger models, such as 14B and 32B models. While our empirical results across different reasoning domains and model scales suggest that GMTS has the potential to extend to larger-scale models, a more comprehensive evaluation on such models is left for future work. Acknowledgments We thank the SJTU supercomputer center for providing the computing services and the anonymous reviewers very much for their careful reading and valuable comments. This work was supported by the National Natural Science Foundation of China (Grant No. 125B2026) and the Natural Science Foundation of Chongqing, China (No. CSTB2023NSCQ-LZX0054). References [Achiam et al.(2023)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, and et al.] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. [Bie et al.(2026)Bie, Cao, Cao, Chen, Chen, Chen, Du, Feng, Feng, Gong, and et al.] Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, and et al. 2026. Llada2. 1: Speeding up text diffusion via token editing. arXiv preprint arXiv:2602.08676. [Chen et al.(2025)Chen, Chen, Wang, and Yang] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. 2025. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505.12346. [Cheng et al.(2026)Cheng, Huang, Zhu, Dai, Zhao, Zhang, and Wei] Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. 2026. Reasoning with exploration: An entropy perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30377–30385. [Cui et al.(2025)Cui, Zhang, Chen, Yuan, Wang, Zuo, Li, Fan, Chen, Chen, and et al.] Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, and et al. 2025. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. [Lee et al.(2024)Lee, Phatale, Mansoor, Mesnard, Ferret, Lu, Bishop, Hall, Carbune, Rastogi, and et al.] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and et al. 2024. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In International Conference on Machine Learning, pages 26874–26901. PMLR. [Li et al.(2025)Li, Xue, Wang, Zhou, Li, Ji, Wang, Liu, Yang, Qiu, and et al.] Qingbin Li, Rongkun Xue, Jie Wang, Ming Zhou, Zhi Li, Xiaofeng Ji, Yongqi Wang, Miao Liu, Zheming Yang, Minghui Qiu, and et al. 2025. Cure: Critical-token-guided re-concatenation for entropy-collapse prevention. arXiv preprint arXiv:2508.11016. [Lightman et al.(2023)Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, and Cobbe] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations. [Liu et al.(2026)Liu, Dong, Lu, Diao, Belcak, Liu, Chen, Yin, Wang, Cheng, and et al.] Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and et al. 2026. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242. [Lv et al.(2026)Lv, Zheng, Zhang, Huang, Wang, Dong, Zhu, and Chen] Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang, Zhenghao Huang, Xingjun Wang, Baohua Dong, Hangcheng Zhu, and Yingda Chen. 2026. Which tokens matter? adaptive token selection for rlvr with the relative surprisal index. arXiv preprint arXiv:2606.31575. [Ma et al.(2026)Ma, Yang, Huang, Lu, Meng, Wang, Ding, Vosoughi, Wang, and Zhou] Chiyu Ma, Shuo Yang, Kexin Huang, Jinda Lu, Haoming Meng, Shangshang Wang, Bolin Ding, Soroush Vosoughi, Guoyin Wang, and Jingren Zhou. 2026. Fipo: Eliciting deep reasoning with future-kl influenced policy optimization. arXiv preprint arXiv:2603.19835. [Meng et al.(2026)Meng, Huang, Wei, Ma, Yang, Wang, Wang, Ding, and Zhou] Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou. 2026. Sparse but critical: A token-level analysis of distributional shifts in rlvr fine-tuning of llms. In The Fourteenth International Conference on Learning Representations. [Ouyang et al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, and et al.] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. [Schulman(2020)] John Schulman. 2020. Approximating kl divergence. John Schulman’s Homepage. [Schulman et al.(2017)Schulman, Wolski, Dhariwal, Radford, and Klimov] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [Shao et al.(2024)Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, and et al.] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. [Shen et al.(2025)Shen, Liu, Li, Fang, Ma, Liao, Shen, Zhang, Zhao, Zhang, and et al.] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, and et al. 2025. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615. [Shinn et al.(2023)Shinn, Cassano, Gopinath, Narasimhan, and Yao] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652. [Talmor et al.(2019)Talmor, Herzig, Lourie, and Berant] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158. [Talmor et al.(2022)Talmor, Yoran, Bras, Bhagavatula, Goldberg, Choi, and Berant] Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2022. Commonsenseqa 2.0: Exposing the limits of ai through gamification. arXiv preprint arXiv:2201.05320. [Tan and Pan(2025)] Hongze Tan and Jianfei Pan. 2025. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508.04349. [Wang et al.(2025a)Wang, Xu, Liu, Wu, Lin, and Chen] Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. 2025a. Emergent hierarchical reasoning in llms through reinforcement learning. arXiv preprint arXiv:2509.03646. [Wang et al.(2025b)Wang, Liu, Zhang, Li, and Zhou] Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou. 2025b. Stabilizing knowledge, promoting reasoning: Dual-token constraints for rlvr. arXiv preprint arXiv:2507.15778. [Wang et al.(2025c)Wang, Yu, Gao, Zheng, Liu, Lu, Dang, Chen, Yang, Zhang, and et al.] Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and et al. 2025c. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939. [Wu et al.(2025)Wu, Zou, Liang, Zhang, and Ma] Tianhe Wu, Jian Zou, Jie Liang, Lei Zhang, and Kede Ma. 2025. Visualquality-r1: Reasoning-induced image quality assessment via reinforcement learning to rank. arXiv preprint arXiv:2505.14460. [Xu et al.(2025)Xu, Liu, Yin, Zhou, and Poovendran] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. 2025. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6980–7008. [Yang et al.(2025)Yang, Luo, Wang, Han, He, Li, and Xu] Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. 2025. Do not let low-probability tokens over-dominate in rl for llms. arXiv preprint arXiv:2505.12929. [Yao et al.(2025)Yao, Cheng, Wu, Wu, and Tan] Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. 2025. Diversity-aware policy optimization for large language model reasoning. arXiv preprint arXiv:2505.23433. [Yu et al.(2025)Yu, Zhang, Zhu, Yuan, Zuo, Yue, Dai, Fan, Liu, Liu, and et al.] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and et al. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. [Zheng et al.(2025)Zheng, Liu, Li, Chen, Yu, Gao, Dang, Liu, Men, Yang, and et al.] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and et al. 2025. Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Appendix A Appendix A.1 More Experimental Results We provide additional complete numerical results for Section 4.3 in the main text, with detailed values reported in Tables 6 and 7. The Bottom selection experiment in Table 6 is designed to empirically examine whether the top ranked tokens under GMTS contain more training beneficial information than those selected by ETS. In addition, as shown in Table 7, GMTS clearly outperforms ETS in 9 out of 10 GRPO/DAPO settings, which empirically shows that GMTS generally remains superior to ETS across different top-selection ratios. Table 6: Evaluation on five math-reasoning benchmarks (ignore AIME2025) for ETS/GMTS with DAPO/GRPO on Qwen2.5-math-1.5B under average@16 accuracy (%). AIME2024 AMC23 MATH-500 Minerva OlympiadBench Avg. Qwen2.5-math-1.5B 2.702.70 18.7518.75 23.4023.40 6.206.20 14.1014.10 13.0313.03 DAPO 14.1714.17 50.6250.62 73.8573.85 29.4729.47 37.1737.17 41.0641.06 + ETS Bottom (80%) 12.2212.22 47.1747.17 65.5265.52 23.7723.77 30.4030.40 35.8135.81 + ETS Bottom (90%) 13.1113.11 49.5049.50 70.6070.60 28.7328.73 34.4934.49 39.2939.29 + GMTS Bottom (80%) 12.6712.67 46.6746.67 63.1163.11 22.0822.08 29.1429.14 34.7334.73 + GMTS Bottom (90%) 12.4412.44 50.1750.17 68.3568.35 26.7426.74 32.3332.33 38.0138.01 Table 7: Evaluation on five math-reasoning benchmarks for ETS/GMTS with DAPO/GRPO on Qwen2.5-math-1.5B under average@16 accuracy (%) in different selected ratios. AIME2024 AMC23 MATH-500 Minerva OlympiadBench Avg. Qwen2.5-math-1.5B 2.702.70 18.7518.75 23.4023.40 6.206.20 14.1014.10 13.0313.03 DAPO 14.1714.17 50.6250.62 73.8573.85 29.4729.47 37.1737.17 41.0641.06 + ETS Top (90%) 13.96 51.7251.72 73.9773.97 30.36 24.5824.58 38.9238.92 + GMTS Top (90%) 11.0411.04 52.34 74.20 30.36 37.61 41.11 + ETS Top (70%) 12.08 48.4448.44 74.07 29.26 36.3136.31 40.0340.03 + GMTS Top (70%) 11.4611.46 52.50 74.0074.00 28.8028.80 37.40 40.83 + ETS Top (50%) 12.5012.50 52.8152.81 73.2273.22 29.5529.55 36.97 41.0141.01 + GMTS Top (50%) 13.54 53.44 73.66 29.89 36.5036.50 41.41 + ETS Top (10%) 12.7112.71 46.8846.88 70.0370.03 27.0827.08 32.9332.93 37.9337.93 + GMTS Top (10%) 13.33 52.19 73.53 29.72 35.68 40.89 GRPO 10.6310.63 48.1248.12 71.5971.59 28.8128.81 34.9034.90 38.8138.81 + ETS Top (90%) 11.5611.56 50.0050.00 72.73 29.6129.61 36.07 40.0040.00 + GMTS Top (90%) 15.33 51.00 72.6872.68 30.51 35.9135.91 41.09 + ETS Top (70%) 11.8811.88 50.6250.62 73.5173.51 30.3130.31 36.6136.61 40.5940.59 + GMTS Top (70%) 12.92 52.50 74.22 30.61 37.11 41.47 + ETS Top (50%) 12.92 50.78 73.46 29.25 37.86 40.85 + GMTS Top (50%) 10.6310.63 49.3849.38 72.9572.95 28.7528.75 36.0936.09 39.5639.56 + ETS Top (10%) 10.6310.63 45.9445.94 70.0070.00 27.3027.30 32.9732.97 37.3737.37 + GMTS Top (10%) 15.00 49.84 72.24 29.60 35.65 40.47 A.2 Derivation of token gradient Proposition 3.1: Let GtG_t be the ℓ2 _2-norm of the gradient of the log-probability logπθ(ot|,<t) _θ(o_t| q, o_<t) with respect to the logits ztz_t, EtE_t is the entropy of distribution πθ(⋅|,<t) _θ(·| q, o_<t). Let 1−πθ(ot|,<t)=ε1- _θ(o_t| q, o_<t)= , then we have: limε→0logGtlogEt=1. _ → 0 G_t E_t=1. Proof. Let pt=πθ(⋅∣,<t)p_t= _θ(· q, o_<t) ∇zlog(pt,i) _z (p_t,i) =1pt,i∇zpt,i = 1p_t,i _zp_t,i =1pt,i[pt,i(ei−pt)] = 1p_t,i [p_t,i(e_i-p_t) ] =ei−pt, =e_i-p_t, where eie_i is the unit vector whose i-th entry is 11 and all other entries are 00. Let πθ(ot|,<t) _θ(o_t| q, o_<t) = pt,ip_t,i, we can get: Gt=(1−pt,i)2+∑j≠ipt,j2.G_t= (1-p_t,i)^2+ _j≠ ip_t,j^2. According to 1−πθ(ot|,<t)=ε1- _θ(o_t| q, o_<t)= , then we will have logGt G_t =log(ε2(1+∑j≠iqt,j2)) = ( ^2 (1+ _j≠ iq_t,j^2 )) =logε+log(1+∑j≠iqt,j2). = + ( 1+ _j≠ iq_t,j^2). where qt,k=pt,kεq_t,k= p_t,k . For entropy EtE_t, we have: Et E_t =−∑jpt,jlogpt,j =- _jp_t,j p_t,j =−pt,ilogpt,i−(1−pt,i)log(1−pt,i) =-p_t,i p_t,i-(1-p_t,i) (1-p_t,i) −(1−pt,i)∑j≠ipt,j1−pt,ilogpt,j1−pt,i -(1-p_t,i) _j≠ i p_t,j1-p_t,i p_t,j1-p_t,i Notice that −∑j≠ipt,j1−pt,ilogpt,j1−pt,i- _j≠ i p_t,j1-p_t,i p_t,j1-p_t,i is the entropy of the renormalized residual distribution after removing the i-th probability, we call it E(pt/pt,i)E(p_t/p_t,i), then we have: Et=−(1−ε)log(1−ε)−εlogε+εE(pt/pt,i)E_t=-(1- ) (1- )- + E(p_t/p_t,i) Using the Taylor expansion log(1−ε)=−ε−ε22+O(ε3) (1- )=- - ^22+O( ^3). We get logEt E_t =log(ε−εlogε+εE(pt/pt,i)−ε22+O(ε3)) = ( - + E(p_t/p_t,i)- ^22+O( ^3)) =logε+log(1−logε+E(pt/pt,i)−ε2+O(ε2)). = + (1- +E(p_t/p_t,i)- 2+O( ^2)). Thus, we have: limε→0logGtlogEt= _ → 0 G_t E_t= limε→0logε+log(1+∑j≠iqt,j2)logε+log(1−logε+E(pt/pt,i)−ε2+O(ε2)) _ → 0 + ( 1+ _j≠ iq_t,j^2) + (1- +E(p_t/p_t,i)- 2+O( ^2)) =1. =1. ∎ A.3 Training Details For fairness and reproducibility, we fix the training seed to 0 in all the experiments, which are performed on eight NVIDIA H100 GPUs, each equipped with 80 GB of memory. Hyperparameter settings for Qwen2.5-math-1.5B and Qwen2.5-math-7B. For the Qwen2.5-math-1.5B and Qwen2.5-math-7B models, the training configurations are as follows: the maximum response length is set to 2048 tokens, and the maximum prompt length to 1024 tokens. Each training batch contained 64 questions, with a mini-batch size of 64. For each question, 16 rollouts are sampled with a temperature of 1.0. The clipping parameters ϵ1 _1 and ϵ2 _2 are set to 0.2 and 0.28. The learning rate is fixed at 3×10−53× 10^-5 without applying any learning rate warm-up or scheduling. Hyperparameter settings for Qwen3-8B. Due to computational constraints, we could not fully adopt the settings of 2025beyond, and therefore adjusted the maximum output length. Specifically, we set the maximum response length to 4096 tokens and the maximum prompt length to 1024 tokens. Each training batch contained 512 questions and a mini-batch size of 32, resulting in 16 gradient steps per training batch. For each question, 16 rollouts are sampled with a temperature of 1.0. The clipping parameters ϵ1 _1 and ϵ2 _2 are set to 0.2 and 0.28. The learning rate is fixed at 1×10−61× 10^-6 without applying any learning rate warm-up or scheduling. Hyperparameter settings for Qwen2.5-coder-1.5B and Qwen2.5-coder-7B. For the Qwen2.5-coder-1.5B and Qwen2.5-coder-7B models, the training configurations are as follows: the maximum response length is set to 32768 tokens, and the maximum prompt length to 2048 tokens. Each training batch contained 512 questions, with a mini-batch size of 32. For each question, 16 rollouts are sampled with a temperature of 1.0. The clipping parameters ϵ1 _1 and ϵ2 _2 are set to 0.2 and 0.28. The learning rate is fixed at 1×10−61× 10^-6 without applying any learning rate warm-up or scheduling. Hyperparameter settings for Qwen2.5-base-1.5B and Qwen2.5-base-7B. For the Qwen2.5-base-1.5B and Qwen2.5-base-7B models, the training configurations are as follows: the maximum response length is set to 4096 tokens, and the maximum prompt length to 1024 tokens. Each training batch contained 512 questions, with a mini-batch size of 32. For each question, 16 rollouts are sampled with a temperature of 1.0. The clipping parameters ϵ1 _1 and ϵ2 _2 are set to 0.2 and 0.28. The learning rate is fixed at 1×10−61× 10^-6 without applying any learning rate warm-up or scheduling. Dialogue template For the math and CS domains, we adopt the following dialogue template: ⬇ <|im_start|>system reason step by step, and put your final answer within .<|im_end|> <|im_start|>user "<|im_end|> <|im_start|>assistant For the code domain, we adopt the following dialogue template: ⬇ <|im_start|>system are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user Please provide a self-contained Python script that solves the following problem: question will use the following starter code to write the solution to the problem: ‘python starter_code ‘ Please reason step by step, and put your code answer in a markdown code block. A.4 An Example for ETS Top (20%) and GMTS (20%) with DAPO on Qwen3-8B under AIME2025 In DAPO on Qwen3-8B, we selected an example under the ETS Top (20%) and GMTS Top (20%) settings to illustrate that the output of GMTS is not only more concise but also more accurate compared to ETS: Question and Ground-Truth Question: Find the number of ordered pairs (x,y)(x,y), where both x and y are integers between −100-100 and 100100, inclusive, such that 12x2−xy−6y2=012x^2-xy-6y^2=0. Ground-Truth: 117 117 ETS Top (20%) Okay, so I need to find the number of ordered pairs (x,y)(x,y) where both x and y are integers between −100-100 and 100100 inclusive, and they satisfy the equation 12x2−xy−6y2=0.12x^2-xy-6y^2=0. Let me think about how to approach this. First, the equation is quadratic in both x and y. Maybe I can factor it or find some relationship between x and y. Let me try rearranging the equation. Let me write it again: 12x2−xy−6y2=0.12x^2-xy-6y^2=0. Hmm. Maybe I can factor this quadratic equation. Let me see. If I treat this as a quadratic in x, then maybe I can factor it or use the quadratic formula. Alternatively, maybe factor it as a quadratic in y. Let me try both approaches. Let me try treating it as a quadratic in x. So, 12x2−yx−6y2=0.12x^2-yx-6y^2=0. So, in the form ax2+bx+c=0ax^2+bx+c=0, where a=12a=12, b=−yb=-y, and c=−6y2c=-6y^2. Then, using quadratic formula: x=y±y2−4⋅12⋅(−6y2)2⋅12.x= y± y^2-4· 12·(-6y^2)2· 12. Calculating discriminant: D=y2−4⋅12⋅(−6y2)=y2+288y2=289y2.D=y^2-4· 12·(-6y^2)=y^2+288y^2=289y^2. So D=289y2=17|y|. D= 289y^2=17|y|. Since y is an integer, |y||y| is non-negative, so D D is 17|y|17|y|. Therefore, x=y±17|y|24.x= y± 17|y|24. — Case 1: y≥0y≥ 0. Then |y|=y|y|=y. So x=y±17y24.x= y± 17y24. So two possibilities: x=y+17y24=18y24=3y4,x= y+17y24= 18y24= 3y4, or x=y−17y24=−16y24=−2y3.x= y-17y24= -16y24=- 2y3. Case 2: y<0y<0. Then |y|=−y|y|=-y. So x=y±17(−y)24.x= y± 17(-y)24. Let’s compute: First, with the plus sign: y+(−17y)24=−16y24=−2y3. y+(-17y)24= -16y24=- 2y3. With the minus sign: y−(−17y)24=y+17y24=18y24=3y4. y-(-17y)24= y+17y24= 18y24= 3y4. So regardless of the sign of y, the solutions for x are x=3y4orx=−2y3.x= 3y4 8.50012ptor 8.50012ptx=- 2y3. Therefore, for each y, x must be either 3y4 3y4 or −2y3- 2y3. But since x has to be an integer, this imposes conditions on y. Therefore, 3y/43y/4 must be integer or −2y/3-2y/3 must be integer. Therefore: Either 4|3y4 3y or 3|2y3 2y. But since 3 and 4 are coprime, 4|3y4 3y implies 4|y4 y. Similarly, 3|2y3 2y implies 3|y3 y. Therefore, for x to be integer: Either y is divisible by 44, or y is divisible by 33. Therefore, the solutions are: - For each y divisible by 4, x=3y/4x=3y/4 is integer. - For each y divisible by 3, x=−2y/3x=-2y/3 is integer. If y is divisible by both 3 and 4 (i.e., by 12), then both x’s would be integers. — Therefore, the total number of solutions would be the number of y’s divisible by 4 plus the number of y’s divisible by 3. However, we have to be careful not to double count the cases where y is divisible by both 3 and 4, i.e., by 12. Because those y’s would be counted in both sets. Therefore, using inclusion-exclusion, total solutions would be: N=(#y:4∣y)+(#y:3∣y)−(#y:12∣y).N=(\#\y:4 y\)+(\#\y:3 y\)-(\#\y:12 y\). — Now we compute. Range: −100≤y≤100-100≤ y≤ 100, i.e., 201 integers. - Number of y divisible by 4: ⌊100/4⌋×2+1=25×2+1=51 100/4 × 2+1=25× 2+1=51. - Number of y divisible by 3: ⌊100/3⌋×2+1=33×2+1=67 100/3 × 2+1=33× 2+1=67. - Number of y divisible by 12: ⌊100/12⌋×2+1=8×2+1=17 100/12 × 2+1=8× 2+1=17. Thus N=51+67−17=101.N=51+67-17=101. — Check special cases: - y=0y=0. Then the equation is 12x2=0⟹x=012x^2=0 x=0. So (0,0)(0,0) is one solution. This is correctly counted once by the formula. - y=4y=4. Then x=3x=3. Solution: (3,4)(3,4) works. - y=3y=3. Then x=−2x=-2. Solution: (−2,3)(-2,3) works. - y=12y=12. Then x=9x=9 or x=−8x=-8, both valid. All checks out. — Finally, also check x range: −100≤x≤100-100≤ x≤ 100. For |y|≤100|y|≤ 100, x=3y4x= 3y4 or x=−2y3x=- 2y3 also lies in [−100,100][-100,100]. Verified. — 101 101 GMTS Top (20%) Okay, so I need to find the number of ordered pairs (x,y)(x,y) where both x and y are integers between −100-100 and 100100 inclusive, and they satisfy the equation 12x2−xy−6y2=012x^2-xy-6y^2=0. Hmm, let me think about how to approach this. First, maybe I can try to factor the equation or simplify it somehow. Let me write down the equation again: 12x2−xy−6y2=0.12x^2-xy-6y^2=0. Hmm, quadratic in terms of x and y. Maybe I can treat this as a quadratic equation in x or y? Let me try treating it as a quadratic in x. Let me rearrange the terms: 12x2−xy−6y2=0.12x^2-xy-6y^2=0. If I consider this as a quadratic in x, then it’s of the form ax2+bx+c=0ax^2+bx+c=0, where: a=12,b=−y,c=−6y2.a=12, 8.50012ptb=-y, 8.50012ptc=-6y^2. Then, using the quadratic formula, x=−b±b2−4ac2a=y±y2−4⋅12⋅(−6y2)24.x= -b± b^2-4ac2a= y± y^2-4· 12·(-6y^2)24. Simplify inside the square root: y2−4⋅12⋅(−6y2)=y2+288y2=289y2.y^2-4· 12·(-6y^2)=y^2+288y^2=289y^2. So 289y2=17|y|. 289y^2=17|y|. Therefore, x=y±17|y|24.x= y± 17|y|24. — Case 1: y≥0y≥ 0. Then |y|=y|y|=y. Therefore: x=y±17y24.x= y± 17y24. So two possibilities: x=18y24=3y4,orx=−16y24=−2y3.x= 18y24= 3y4, 8.50012ptor 8.50012ptx= -16y24=- 2y3. Case 2: y<0y<0. Then |y|=−y|y|=-y. Therefore: x=y±17(−y)24.x= y± 17(-y)24. So: First possibility: y+(−17y)=−16y+(-17y)=-16y, hence x=−2y3x=- 2y3. Second possibility: y−(−17y)=18y-(-17y)=18y, hence x=3y4x= 3y4. — So regardless of the sign of y, the solutions for x are: x=3y4orx=−2y3.x= 3y4 8.50012ptor 8.50012ptx=- 2y3. Therefore, the solutions to the equation are all pairs (x,y)(x,y) with x,y∈[−100,100]x,y∈[-100,100] such that either x=3y/4x=3y/4 or x=−2y/3x=-2y/3, with integrality conditions. — Case 1: x=3y/4x=3y/4. For x to be integer, y must be divisible by 4. Let y=4ky=4k, then x=3kx=3k. Range: −100≤4k≤100⟹−25≤k≤25-100≤ 4k≤ 100 -25≤ k≤ 25. That gives 25−(−25)+1=5125-(-25)+1=51 solutions. Case 2: x=−2y/3x=-2y/3. For x to be integer, y must be divisible by 3. Let y=3my=3m, then x=−2mx=-2m. Range: −100≤3m≤100⟹−33≤m≤33-100≤ 3m≤ 100 -33≤ m≤ 33. That gives 33−(−33)+1=6733-(-33)+1=67 solutions. — Overlap check: Suppose (x,y)(x,y) satisfies both forms: (x,y)=(3k,4k)(x,y)=(3k,4k) and (x,y)=(−2m,3m)(x,y)=(-2m,3m). Then 3k=−2m3k=-2m and 4k=3m4k=3m. Solving gives k=0,m=0k=0,m=0, hence (0,0)(0,0). So there is exactly one overlap. — Final count: 51+67−1=117.51+67-1=117. 117 117