Paper deep dive
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization
Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, Xunliang Cai
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/20/2026, 8:24:31 PM
Summary
The paper introduces DynaMO, a dual-pronged optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR) in Large Language Models. It addresses two key challenges: suboptimal uniform rollout allocation and gradient attenuation in softmax policies. DynaMO employs dynamic rollout allocation based on Bernoulli variance to minimize gradient variance at the sequence level, and gradient-aware advantage modulation at the token level to compensate for attenuated gradients in high-confidence actions and stabilize excessive updates using entropy changes as indicators.
Entities (9)
Relation Signals (6)
DynaMO → addresses → RLVR
confidence 95% · DynaMO is proposed as an optimization framework for RLVR.
DynaMO → uses → Bernoulli Variance
confidence 92% · DynaMO establishes Bernoulli variance as a computable proxy for gradient informativeness.
DynaMO → uses → Policy Entropy
confidence 90% · DynaMO utilizes entropy changes as computable indicators to stabilize excessive update magnitudes.
DynaMO → evaluatedon → Qwen2.5-Math-1.5B
confidence 85% · Experiments conducted on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B.
DynaMO → outperforms → GRPO
confidence 85% · Extensive experiments demonstrate consistent improvements over strong RLVR baselines like GRPO.
DynaMO → evaluatedon → AIME24
confidence 80% · Experiments conducted on mathematical reasoning benchmarks including AIME24.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (ii) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradient-aware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.19208v2
- Canonical: https://arxiv.org/abs/2602.19208v2
Trouble viewing inline? Open PDF directly →
Full Text
72,756 characters extracted from source content.
Expand or collapse full text
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization Yangyi Fang1,2,*, Jiaye Lin1,*, Xiaoliang Fu1,3,*, Cong Qin1,4, Haolin Shi2, Chaowen Hu1, Lu Pan1, Ke Zeng1, Xunliang Cai1,† 1Meituan 2Tsinghua University 3Fudan University 4Peking University fangyangyi, linjiaye@meituan.com Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (i) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradient-aware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: GithubX-F/DynaMO-RL. How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization Yangyi Fang1,2,*, Jiaye Lin1,*, Xiaoliang Fu1,3,*, Cong Qin1,4, Haolin Shi2, Chaowen Hu1, Lu Pan1, Ke Zeng1, Xunliang Cai1,† 1Meituan 2Tsinghua University 3Fudan University 4Peking University fangyangyi, linjiaye@meituan.com †footnotetext: * Equal contribution. † Corresponding author. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful and promising paradigm for advancing Large Language Model (LLM) reasoning Ouyang et al. (2022); Bai et al. (2022). Recent breakthrough models, such as OpenAI o1 Jaech et al. (2024) and DeepSeek-R1 Guo et al. (2025), demonstrate emergent capabilities like long-form chain-of-thought and self-reflection. Building on these successes, numerous works have explored RLVR methods Shao et al. (2024); Fang et al. (2026), most commonly combining the GRPO algorithm Shao et al. (2024) or its variants Liu et al. (2025); Yu et al. (2025a) with outcome-based rewards for reinforcement learning. However, despite these advances, fundamental challenges persist in both computational resource allocation and policy optimization dynamics. Current RLVR methods uniformly distribute rollout budgets across training instances Shao et al. (2024), ignoring heterogeneous gradient informativeness among various problems. This overlooks a fundamental trade-off: while high-variance problems contribute more informative learning signals, they simultaneously introduce greater estimation noise that may destabilizes model performance. Existing adaptive strategies either prioritize sample selection over resource allocation Bengio et al. (2009); Schaul et al. (2015); Tong et al. (2024) or target different optimization frameworks Dong et al. (2023); Yao et al. (2025), leaving the variance-informativeness trade-off in policy gradient methods unaddressed. Moreover, this limitation is particularly pronounced on datasets with diverse difficulties, where effective allocation requires dynamically balancing informativeness and noise. Compounding this resource allocation challenge, the policy gradient dynamics in RLVR training expose fundamental token-level optimization issues. Theoretical analysis shows that the mathematical structure of the softmax policy induces an inherent and problematic tension Li (2025): high-confidence correct actions naturally yield smaller gradient magnitudes, leading to insufficient learning signals, whereas excessive gradient updates may severely destabilize training Luo et al. (2025). Our analysis further reveals that gradient magnitude is provably upper-bounded by policy entropy, allowing entropy changes to serve as computable indicators of such instability. Existing approaches attempt to mitigate this issue via ratio clipping Yu et al. (2025a); Yang et al. (2025), sample reweighting Zhu et al. (2025), or entropy-induced advantages Cheng et al. (2025); Tan and Pan (2025); Wang et al. (2025), but these coarse-grained interventions lack a unified theoretical grounding and even amplify rather than stabilize training fluctuations. To address these key challenges, we propose DynaMO, a dual-pronged optimization framework grounded in variance minimization theory and policy gradient analysis. At the sequence level, we derive dynamic rollout allocation that explicitly balances the informativeness-noise trade-off by gradient variance minimization specifically for policy gradient methods, establishing Bernoulli variance as a lightweight computable proxy. At the token level, we introduce gradient-aware advantage modulation through integrated compensation and stabilization mechanisms based on our gradient-entropy analysis: compensation for gradient attenuation in high-confidence correct actions and stabilization against excessive update magnitudes by monitoring entropy changes as an indicator. Our core contributions in this paper are summarized as follows: • We prove that uniform allocation is suboptimal and subsequently derive variance-minimizing rollout allocation with a lightweight proxy. • We establish the gradient-entropy relationship through theoretical analysis, enabling gradient-aware advantage modulation with compensation and stabilization mechanisms. • Extensive experiments across six benchmarks and three LLM scales demonstrate consistent improvements, with comprehensive ablations validating each component and visualizations revealing stable optimization dynamics. 2 Related Works 2.1 Reinforcement Learning for LLMs Reinforcement learning has emerged as a dominant paradigm for LLM post-training, with RLHF and RLVR demonstrating significant success Ouyang et al. (2022); Bai et al. (2022); Schulman et al. (2017). Recent breakthrough models, including DeepSeek-R1 Guo et al. (2025), DeepSeekMath Shao et al. (2024), OpenAI o1 Jaech et al. (2024), and Kimi k1.5 Team et al. (2025), further demonstrate the remarkable effectiveness of RLVR on complex reasoning tasks with verifiable rewards. While subsequent works have introduced various algorithmic refinements Liu et al. (2025); Yu et al. (2025a); Chu et al. (2025); Hu et al. (2025); Fang et al. (2025), fundamental challenges in computational efficiency and optimization stability still persist. 2.2 Entropy Dynamics in Policy Optimization Entropy regularization balances exploration and exploitation Haarnoja et al. (2018); Mnih et al. (2016), yet its precise role in LLM training remains contentious Ouyang et al. (2022); Shao et al. (2024); Yu et al. (2025a); Chu et al. (2025). Entropy collapse Luo et al. (2025) motivates various mitigation strategies such as ratio clipping Yu et al. (2025a); Yang et al. (2025), sample reweighting Zhu et al. (2025), or entropy-induced advantages Cheng et al. (2025); Tan and Pan (2025); Wang et al. (2025); Li et al. (2026a). Li (2025) further shows that high-confidence actions yield attenuated gradient magnitudes, thereby inducing asymmetric learning dynamics. Yet existing methods lack unified theoretical grounding, and the complex interplay between gradient attenuation and entropy dynamics remains largely underexplored. Our work addresses this through gradient-aware policy update control grounded in the gradient-entropy relationship, where entropy serves as a computable and interpretable indicator of update magnitude rather than a direct optimization target. 2.3 Sample Efficiency Sample efficiency is critically important for RLVR training, where generating multiple rollouts per problem incurs substantial computational cost. Standard methods employ uniform rollout budgets Shao et al. (2024), largely overlooking heterogeneous gradient informativeness across problems. Curriculum learning Bengio et al. (2009) and prioritized experience replay Schaul et al. (2015) choose which problems to train on, but primarily emphasize sample ordering rather than resource allocation. Offline methods Tong et al. (2024) repeatedly sample until obtaining a fixed number of correct responses, lacking dynamic scheduling for online training. EM-based methods Dong et al. (2023); Gulcehre et al. (2023); Yao et al. (2025); Liu et al. (2021a, b, 2022); Yu et al. (2025b) enhance efficiency via iterative rejection sampling, yet require gradient norm computations and target the EM framework rather than policy gradient methods. In contrast, we derive optimal rollout allocation by minimizing gradient variance for policy gradient methods, with a lightweight, gradient-free proxy based on historical success statistics. Figure 1: Overview of DynaMO, which operates at both the sequence and token levels, enabling fine-grained control over optimization. (i) Left: Dynamic allocation concentrates the rollout budget on high-variance problems. (i) Right: Gradient-aware advantage modulation compensates for attenuated gradients and stabilizes excessive updates. 3 Preliminaries Policy Optimization in RLVR. Formally, given a prompt q sampled from the training data D, let πθ _θ denotes the policy model parameterized by θ. In the context of RLVR, the model autoregressively generates a response o=o1,o2,…,oTo=\o_1,o_2,…,o_T\, where each token oto_t represents an action taken at step t and T represents the sequence length. The final objective is to maximize the expected reward: J(θ)=q∼,o∼πθ(⋅∣q)[R(o)],J(θ)=E_q ,o _θ(· q)[R(o)], (1) where R(⋅)R(·) is the reward function that evaluates the quality of the generated response o. To reduce variance, GRPO Shao et al. (2024) samples G responses oii=1G\o_i\_i=1^G per prompt, estimating advantages via group-wise normalization: Ai,t=R(oi)−mean(R(oj)j=1G)std(R(oj)j=1G),A_i,t= R(o_i)-mean(\R(o_j)\_j=1^G)std(\R(o_j)\_j=1^G), (2) where R(oi)R(o_i) denotes the reward for response oio_i, evaluated by the reward function R(⋅)R(·). mean(⋅)mean(·) and std(⋅)std(·) denote the mean and standard deviation of the rewards within the group. With the advantage Ai,tA_i,t shared across all tokens, GRPO maximizes the clipped surrogate objective: ℒ(θ) (θ) =q∼,oii=1G∼πold(⋅∣q)[1∑i=1G|oi|∑i=1G∑t=1|oi| =E_q ,\o_i\_i=1^G _old(· q) [ 1 _i=1^G|o_i| _i=1^G _t=1^|o_i| (3) min(ri,tAi,t,clip(ri,t,1−ϵ,1+ϵ)Ai,t)], (r_i,tA_i,t,clip(r_i,t,1-ε,1+ε)A_i,t ) ], where ri,t=πθ(oi,t∣q,oi,<t)πold(oi,t∣q,oi,<t)r_i,t= _θ(o_i,t q,o_i,<t) _old(o_i,t q,o_i,<t) is the importance sampling ratio and ϵε is the clipping parameter. Consistent with prior works Yu et al. (2025a); Liu et al. (2025), we omit the KL divergence penalty term and do not elaborate further on this component. Policy Entropy. Policy entropy quantifies the uncertainty in the model’s action selection process, serving as a key indicator of exploration in reinforcement learning. Given a policy model πθ _θ and training data D, the policy entropy is defined as: ℋ(πθ,) ( _θ,D) =q∼,oii=1G∼πθ(⋅∣q)[1∑i=1G|oi|∑i=1G∑t=1|oi| =E_q ,\o_i\_i=1^G _θ(· q) [ 1 _i=1^G|o_i| _i=1^G _t=1^|o_i| . (4) (−∑v∈πθ(v|q,oi,<t)logπθ(v|q,oi,<t))], . (- _v _θ(v|q,o_i,<t) _θ(v|q,o_i,<t) ) ], where V represents the vocabulary. This entropy metric reflects the model’s confidence distribution: higher values indicate greater uncertainty and exploration potential, whereas lower values suggest more deterministic behavior selection. 4 Methodology 4.1 Overview To effectively address the challenges of inefficient resource allocation and policy optimization dynamics highlighted in Section 1, we propose DynaMO (Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization). As illustrated in Figure 1, this dual-pronged framework operates at both the sequence and token levels, enabling fine-grained control over optimization. Notably, DynaMO comprises two key components: (i) dynamic rollout allocation that adaptively distributes computational budget based on gradient variance minimization, concentrating resources on problems with balanced success-failure distributions where learning signals are most informative, and (i) gradient-aware advantage modulation based on the gradient-entropy upper bound relationship, which compensates for gradient attenuation in high-confidence actions while using entropy changes to stabilize excessive updates. 4.2 Dynamic Rollout Allocation We establish a theoretical framework for optimal rollout allocation by formulating it as a gradient variance minimization problem, which considers a training dataset D with N prompts qii=1N\q_i\_i=1^N. 4.2.1 Optimization Theory For a prompt qiq_i with nin_i rollout budget, the corresponding gradient estimator g^i=1ni∑k=1nigi,k g_i= 1n_i _k=1^n_ig_i,k has variance Var[g^i]=σi2/niVar[ g_i]= _i^2/n_i. Minimizing the total estimation variance ∑iVar[g^i] _iVar[ g_i] under budget constraint ∑ini=B _in_i=B yields (proof in Appendix C): ni∗=B⋅σi∑k=1Nσk.n_i^*=B· _i _k=1^N _k. (5) This principle allocates more rollouts to problems with higher gradient variance. To implement this, we use Bernoulli variance PiP_i as a practical proxy for σi _i (derivation in Appendix C). 4.2.2 Bernoulli Variance as Practical Proxy For binary rewards, the variance follows p(1−p)p(1-p), where p represents the success probability. Then, we estimate this via historical statistics with kik_i correct responses out of GiG_i total rollouts generated: Pi=ki(Gi−ki)Gi(Gi−1),[Pi]=pi(1−pi).P_i= k_i(G_i-k_i)G_i(G_i-1), [P_i]=p_i(1-p_i). (6) This unbiased estimator can be efficiently computed from historical success counts, with problems that maximize PiP_i (characterized by balanced correct/incorrect responses) exhibiting large σi _i. Water-Level Implementation. Algorithm 1 implements the proportional budget allocation ni∝Pin_i P_i with the boundary constraints GminG_ (ensuring minimum coverage) and GmaxG_ (preventing over-concentration). After each training iteration, we incrementally update the statistics via Gi←Gi+GinewG_i← G_i+G_i^new and ki←ki+kinewk_i← k_i+k_i^new, thereby enabling an adaptive allocation as model proficiency evolves. Notably, the dynamic allocation strategy provably reduces variance against the conventional uniform allocation, with theoretical justification and operational details in Appendix C and D. Algorithm 1 Variance-Driven Dynamic Allocation 1:Historical statistics (Gi,ki)i=1N\(G_i,k_i)\_i=1^N, total rollout budget B, allocation bounds [Gmin,Gmax][G_ ,G_ ] 2:Compute priorities Pi=ki(Gi−ki)/[Gi(Gi−1)]P_i=k_i(G_i-k_i)/[G_i(G_i-1)] 3:Initialize Ginew=GminG_i^new=G_ , Brem=B−N⋅GminB_rem=B-N· G_ 4:while Brem>0B_rem>0 and ∃i:Ginew<Gmax∃ i:G_i^new<G_ do 5: ℰ=i:Ginew<GmaxE=\i:G_i^new<G_ \ 6: for i∈ℰi do 7: ΔGi=min(⌊BremPi∑j∈ℰPj⌋,Gmax−Ginew) G_i= ( B_remP_i _j P_j ,G_ -G_i^new ) 8: Ginew←Ginew+ΔGiG_i^new← G_i^new+ G_i 9: Brem←Brem−ΔGiB_rem← B_rem- G_i 10: end for 11:end while 12:return G1new,…,GNnew\G_1^new,…,G_N^new\ 4.3 Gradient-Aware Advantage Modulation Beyond resource allocation, the softmax policy structure causes high-confidence correct actions to produce attenuated gradient magnitudes, while excessive gradient updates may undermine training stability. Grounded in theoretical analysis of gradient dynamics, we design an integrated compensation and stabilization mechanism. 4.3.1 Gradient Compensation Our gradient analysis, detailed in Appendix B, reveals the fundamental relationship between the update magnitude and token-level entropy. Specifically, considering a softmax policy parameterized by logits z(s)z(s) at state s=(q,o<t)s=(q,o_<t), the advantage-weighted updates with learning rate η satisfy: ak∼πθ(⋅|s)[‖Δz(s)‖22]=η2[A2](1−∑k=1||πk2) _a_k _θ(·|s) [\| z(s)\|_2^2 ]=η^2\,E[A^2] (1- _k=1^|V| _k^2 ) (7) ≤η2[A2](1−exp(−ℋ(πθ|s))), ≤η^2\,E[A^2] (1- (-H( _θ|s) ) ), where ℋ(πθ|s)=−∑v∈πθ(v|s)logπθ(v|s)H( _θ|s)=- _v _θ(v|s) _θ(v|s) is the token-level entropy and ∑k=1||πk2 _k=1^|V| _k^2 measures policy concentration. Accordingly, high-confidence tokens (πk≈1 _k≈ 1) produce minimal expected update magnitudes (∑j=1||πj2≈1 _j=1^|V| _j^2≈ 1), resulting in gradient attenuation for confident correct actions—an observation that constitutes the theoretical basis for gradient compensation in our method. To effectively mitigate this gradient attenuation phenomenon for the confident correct actions, we introduce a gradient compensation factor: βi,tcomp=[Ai,t>0]⋅g(ℋi,t)+[Ai,t≤0], _i,t^comp=I[A_i,t>0]· g(H_i,t)+I[A_i,t≤ 0], (8) where the compensation function g(⋅)g(·) is designed to scale inversely with the entropy: g(ℋ)=1+α⋅ℋmax−ℋmax−ℋmin.g(H)=1+α· H_ -HH_ -H_ . (9) Here, α determines the maximum compensation factor, while ℋminH_ and ℋmaxH_ represent the minimum and maximum token-level entropy within the current training batch. This asymmetric design targets the confidence–update-magnitude asymmetry: for positive-advantage tokens, the compensation function scales inversely with entropy to counteract gradient attenuation, moderately amplifying learning signals for high-confidence correct actions; for negative-advantage tokens, compensation is bypassed to preserve the natural penalty signals. Furthermore, for uncertain actions with high entropy, the function returns to unity, maintaining natural update dynamics. 4.3.2 Update Magnitude Stabilization While gradient compensation addresses the attenuation for high-confidence actions, excessive gradient updates still may destabilize the training dynamics. By leveraging the gradient-entropy relationship established above, we utilize entropy changes as an indicator to detect such optimization instability. Specifically, our policy update decomposition reveals that changes in logits induce corresponding entropy changes through a factorized form: Δℋ(πθk|s)≈−η∑aπθk(a|s)2⋅Λθk(a|s)⋅ξi,t(a), ( _θ^k|s)≈-η _a _θ^k(a|s)^2· _θ^k(a|s)· _i,t(a), (10) where Λθk(a|s) _θ^k(a|s) denotes the centered log-probability (Definition 2) that captures the deviation of action a’s log-probability from the policy’s average entropy. ξi,t(a)=clip⋅ri,t⋅Ai,t _i,t(a)=I_clip· r_i,t· A_i,t represents the composite update coefficient combining clipping, importance sampling, and advantage estimation. The complete derivation is provided in Appendix A. This decomposition shows that when both the policy concentration πθk(a|s)2 _θ^k(a|s)^2 and the composite coefficient magnitude |ξi,t(a)|| _i,t(a)| are large, entropy changes become substantial. Building upon the established gradient-entropy relationship, such large entropy changes indicate excessive gradient magnitudes that may destabilize training. Accordingly, we define the token-level instability indicator: Ξi,t=|Δℋ(πθk|si,t)|, _i,t= | ( _θ^k|s_i,t) |, (11) which quantitatively measures the estimated contribution of each token to overall update instability. To stabilize tokens exhibiting excessive entropy changes while simultaneously maintaining learning efficiency, we formulate a stabilization factor: βi,tstab=f(Ξi,tmaxjΞj,t), _i,t^stab=f ( _i,t _j _j,t ), (12) where f(⋅)f(·) is a sigmoid-based decay function designed to reduce the modulation factor for tokens with large normalized entropy changes: f(x)=λmin+(1−λmin)⋅σ(−γ(x−τ)).f(x)= _ +(1- _ )·σ(-γ(x-τ)). (13) In this formulation, σ(z)=11+exp(−z)σ(z)= 11+ (-z) is the sigmoid function, γ>0γ>0 controls the transition sharpness between stable and unstable regions, τ∈[0,1]τ∈[0,1] determines the entropy change threshold triggering the decay activation, and λmin=1−α _ =1-α establishes a lower bound for the stabilization factor. 4.3.3 Integrated Advantage Modulation Our complete approach integrates both compensation and stabilization mechanisms through a unified advantage modulation formulation as: Ai,tfinal=Ai,t⋅βi,tcomp⋅βi,tstab.A_i,t^final=A_i,t· _i,t^comp· _i,t^stab. (14) Subsequently, the training objective incorporates these modulated advantages: ℒDynaMO(θ) _DynaMO(θ) =q∼,oii=1n∼πold(⋅∣q)[1∑i=1n|oi|∑i=1n∑t=1|oi| =E_q ,\o_i\_i=1^n _old(· q) [ 1 _i=1^n|o_i| _i=1^n _t=1^|o_i| (15) min(ri,tAi,tfinal,clip(ri,t,1−ϵ,1+ϵ)Ai,tfinal)], (r_i,tA_i,t^final,clip(r_i,t,1-ε,1+ε)A_i,t^final ) ], where n denotes the rollout budget allocated to prompt q via dynamic allocation, with the prompt subscript omitted for notational brevity. This integrated approach addresses both challenges, leveraging the gradient-entropy relationship: the compensation mechanism maintains sufficient learning signals for high-confidence positive actions, while the stabilization mechanism prevents training instability by dampening tokens with excessive entropy changes that indicate large gradient magnitudes, with both mechanisms unified through a single hyperparameter α for simplified tuning. Table 1: Comparison of benchmark results across Qwen2.5-Math-1.5B and Qwen2.5-Math-7B. Pass@K (%) is abbreviated as P@K. The best results are bold, and the second-best results are underlined, respectively. Method AIME24 AIME25 AMC23 MATH500 Minerva Olympiad Avg. P@1 P@32 P@1 P@32 P@1 P@32 P@1 P@32 P@1 P@32 P@1 P@32 P@1 P@32 Qwen2.5-Math-1.5B GRPO 13.2 32.3 7.6 31.5 56.0 90.0 54.4 79.2 17.2 42.8 25.6 47.0 29.0 53.8 Clip-Higher 12.4 34.7 6.4 30.6 50.6 89.9 56.8 80.2 16.8 41.3 26.4 46.8 28.2 53.9 Entropy Loss 12.6 33.7 5.8 28.4 55.6 86.9 56.3 78.5 17.6 43.6 25.4 46.4 28.9 52.9 Fork Tokens 9.4 32.0 5.9 31.4 52.5 85.6 54.3 74.2 16.6 36.8 25.5 45.2 27.4 50.9 Entropy Advantages 15.7 35.8 8.9 33.4 62.0 86.4 59.7 76.2 18.2 43.0 25.9 44.9 31.7 53.3 Clip-COV 13.5 36.4 6.6 34.4 59.5 89.7 57.6 75.6 15.8 44.3 25.8 47.6 29.8 54.7 KL-COV 12.6 33.9 9.0 33.4 55.8 91.3 54.2 78.1 14.8 40.3 25.4 48.1 28.6 54.2 W-REINFORCE 15.3 35.3 8.5 31.7 63.0 85.7 56.7 77.7 18.2 40.3 24.4 46.2 31.0 52.8 DynaMO (Ours) 17.2 37.2 9.8 32.5 63.6 91.9 58.8 81.0 19.4 44.0 27.2 47.1 32.7 55.6 Qwen2.5-Math-7B GRPO 28.8 52.5 11.7 34.8 68.3 90.8 63.3 75.0 22.6 45.4 28.6 44.7 37.2 57.2 Clip-Higher 27.0 51.9 12.1 39.5 67.8 89.9 64.2 83.6 24.0 46.1 28.1 46.3 37.2 59.6 Entropy Loss 30.6 54.6 13.2 40.6 66.0 87.0 60.6 79.6 23.3 45.9 30.2 41.1 37.3 58.1 Fork Tokens 27.1 52.5 13.4 43.5 71.0 87.3 65.8 79.3 26.1 42.4 30.9 47.3 39.1 58.7 Entropy Advantages 27.5 49.7 9.4 39.2 67.9 85.2 65.3 83.3 23.7 43.7 30.4 47.3 37.4 58.1 Clip-COV 32.2 52.7 13.2 40.4 72.7 89.3 64.3 76.8 25.4 45.9 29.5 44.6 39.5 58.3 KL-COV 32.8 53.3 11.7 36.1 70.6 88.5 64.6 75.3 24.5 39.9 30.2 44.2 39.1 56.2 W-REINFORCE 31.8 55.4 14.3 41.0 72.5 89.8 64.9 84.0 26.4 49.5 30.9 46.7 40.1 61.1 DynaMO (Ours) 34.4 59.0 15.4 46.8 74.4 92.9 66.4 84.0 27.3 47.2 31.6 50.1 41.6 63.3 5 Experiments 5.1 Experimental Setup Training Configuration. We conduct experiments on three different LLM scales: Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-14B. Our implementation builds upon the VeRL training framework Sheng et al. (2025), adapting it to incorporate our dual-pronged approach. Our training data is DAPO-Math-17k Yu et al. (2025a), which consists of mathematical reasoning problems with verifiable integer answers, ensuring reliable reward signals for RLVR optimization. Training is performed with top-p sampling at p=1.0 and temperature set to 1.0 to maintain exploration diversity. More details are provided in Appendix E. Benchmarks and Metrics. We evaluate DynaMO on six mathematical reasoning benchmarks: AIME24, AIME25 MAA (2025), AMC23 MAA (2023), MATH500 Hendrycks et al. (2021), Minerva Lewkowycz et al. (2022), and Olympiad He et al. (2024), covering various problem difficulties and types. Detailed descriptions of these benchmarks are presented in Appendix F. We report average Pass@1 and Pass@32 across three independent training runs with different random seeds, using Math-Verify HuggingFace (2025) for answer verification and top-p=1.0, temperature=1.0 for diverse generation. Following Yue et al. (2025), Pass@K equals 1 if at least one of K sampled outputs passes verification, with unbiased estimation adopted from Chen et al. (2021) to mitigate evaluation variance. Baseline Methods. We compare DynaMO with strong baselines, including standard GRPO Shao et al. (2024) and various entropy intervention techniques: Clip-Higher Yu et al. (2025a), Entropy Loss Cheng et al. (2025), Fork Tokens Wang et al. (2025), Entropy Advantages Cheng et al. (2025), coverage-based methods (Clip-COV, KL-COV) Zhu et al. (2025), and W-REINFORCE Tan and Pan (2025). All experiments use consistent hyperparameters for fair comparison. Table 2: Ablation study on Qwen2.5-Math-7B with Pass@1 (%). DRA: Dynamic Rollout Allocation, UMS: Update Magnitude Stabilization, GC: Gradient Compensation, w/o ALL: standard GRPO Method AIME24 AIME25 AMC23 MATH500 Minerva Olympiad Avg. DynaMO 34.4 15.4 74.4 66.4 27.3 31.6 41.6 - w/ow/o GC 33.8 15.0 71.9 65.0 26.7 29.3 40.3 - w/ow/o UMS 33.2 14.7 71.9 64.9 25.9 30.2 40.1 - w/ow/o DRA 31.9 15.2 73.4 65.7 23.0 30.4 39.9 - w/ow/o GC & DRA 31.9 14.5 69.0 64.4 23.7 29.7 38.9 - w/ow/o GC & UMS 30.5 14.3 70.2 63.5 22.2 29.4 38.4 - w/ow/o UMS & DRA 30.0 13.8 70.1 61.6 19.9 30.2 37.6 - w/ow/o ALL 28.8 11.7 68.3 63.3 22.6 28.6 37.2 5.2 Main Results Table LABEL:tab:main_results presents the comprehensive comparison results across six mathematical reasoning benchmarks on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B. DynaMO consistently outperforms all baseline methods by effectively addressing two complementary challenges: the variance-driven rollout allocation concentrates computational budget on problems with balanced success-failure distributions where Bernoulli variance signals high gradient informativeness, while the gradient-aware advantage modulation provides compensation for gradient attenuation in high-confidence correct actions and stabilization against excessive update magnitudes signaled by large entropy changes. Furthermore, a comparison with entropy intervention baselines reveals their critical limitations: coarse-grained clipping and sequence-level reweighting methods like Clip-Higher, Clip-COV, and KL-COV lack fine-grained control over token-level dynamics, while entropy-induced advantage methods such as Entropy Advantages and W-REINFORCE introduce training instability without principled stabilization mechanisms. Results on Qwen3-14B demonstrating effective scaling are presented in Section 5.5. Figure 2: Impact of DRA across different computational budgets on AIME24 with average Pass@1 (%). Solid lines denote the full DynaMO, and dashed lines are the variant that substitutes DRA with uniform allocation. 5.3 Ablation Study 5.3.1 Overall Component Analysis Table 2 conducts systematic ablation on Qwen2.5-Math-7B. DynaMO integrates three components: Dynamic Rollout Allocation (DRA), Update Magnitude Stabilization (UMS), and Gradient Compensation (GC). Removing individual components degrades average performance, with DRA showing the largest impact on Minerva, GC on Olympiad, and UMS providing stability across benchmarks. Removing multiple components reveals synergistic effects: w/o GC & UMS degrades more than the sum of individual removals, indicating UMS stabilizes the amplified gradients from GC. Figure 3: Hyperparameter sensitivity analysis on Qwen2.5-Math-7B across AIME24 and AMC23. 5.3.2 Impact of Dynamic Rollout Allocation Figure 2 evaluates DRA across varying computational budgets, specifically ranging from an average of 8 to 32 rollouts per problem. The results demonstrate that DRA provides stable performance gains across all configurations. During early training, limited historical statistics result in similar allocations across problems. However, as variance data accumulates, DRA progressively concentrates the budget on problems with balanced success-failure distributions where Bernoulli variance peaks. Crucially, these problems reside within the capability gap, being neither trivially solved nor currently inaccessible, where both positive and negative advantage signals are actively generated, thereby maximizing learning signal quality per computational unit. In contrast, uniform allocation continues wasting resources on problems outside this optimal learning zone throughout training. 5.4 Hyperparameter Analysis Figure 3 illustrates the hyperparameter sensitivity analysis on Qwen2.5-Math-7B across AIME24 and AMC23. The unified modulation parameter α exhibits an inverted-U pattern: performance degrades without modulation at α=0α=0, peaks at moderate values, and subsequently declines at excessive settings, validating that insufficient modulation fails to address gradient attenuation while over-intervention constrains the learning process. Allocation bounds GminG_ and GmaxG_ demonstrate stable performance across wide ranges, confirming variance-driven allocation avoids under-sampling issues with minimal tuning. Similarly, the stabilization sharpness γ exhibits a flat plateau around its optimal values, balancing selective intervention against learning flexibility. Entropy threshold τ displays smooth variation characterized by a broad optimal region, thereby demonstrating the reliability in identifying problematic tokens. 5.5 Scaling to Larger Models Figure 4: Performance comparison across different LLM scales (1.5B/7B/14B) with DynaMO and GRPO. To validate scalability, we extend experiments to Qwen3-14B across all benchmarks in Figure 4. DynaMO consistently outperforms GRPO with widening gaps as scale increases: moderate gains at 1.5B, amplified substantially at 7B, and expanded further at 14B. This trend confirms that our theoretically-grounded mechanisms become increasingly effective at larger scales, as variance-driven allocation and gradient-aware modulation better exploit the enhanced representational capacity while mitigating intensified optimization instabilities. 6 Case Study Figure 5: Training dynamics comparison. DynaMO maintains stable gradient norms and smooth entropy evolution, while GRPO exhibits severe spikes and fluctuations. Shaded regions show raw data range. To provide a deeper insight into the effectiveness of DynaMO, we compare it with GRPO through training dynamics analysis. As visually depicted in Figure 5, GRPO suffers from severe gradient spikes and erratic entropy fluctuations, whereas DynaMO maintains stable dynamics across both metrics. Two core components of DynaMO contribute to this stability: dynamic rollout allocation concentrates the budget on high-variance problems, while gradient-aware modulation prevents excessive update magnitudes. Notably, despite following similar overall trends, DynaMO achieves substantially smoother transitions in gradient norms and policy entropy, indicating more controlled optimization process throughout training. These phenomena confirm that DynaMO mitigates training instability at both sequence and token levels while simultaneously achieving superior performance. 7 Conclusion In this paper, we propose DynaMO, a theoretically-grounded dual-pronged framework designed to systematically address fundamental RLVR challenges. At the sequence level, we prove that uniform allocation is suboptimal and subsequently derive a variance-minimizing allocation strategy to concentrate computational resources on high-informativeness problems. At the token level, we establish the gradient-entropy relationship, enabling an integrated advantage modulation mechanism that compensates for gradient attenuation while stabilizing excessive updates. Extensive experiments conducted across multiple reasoning benchmarks and varying LLM scales demonstrate consistent improvements over strong baselines, with comprehensive ablation studies validating the independent contributions from both mechanisms. Limitations Our evaluation is conducted on the Qwen model family, spanning three distinct parameter scales (1.5B, 7B, and 14B). Although we have not yet systematically evaluated other model families, it is important to note that our method operates purely at the algorithmic level—modifying rollout allocation and advantage computation—without relying on any architecture-specific features, suggesting broad transferability potential. Future work could extend this evaluation to additional model architectures and scales, as well as multimodal reasoning scenarios where robustness against adversarial attacks Li et al. (2026b) becomes critical. References Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. (2022) Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §1, §2.1. Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In International Conference on Machine Learning (ICML), Cited by: §1, §2.3. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §5.1. D. Cheng, S. Huang, X. Zhu, B. Dai, W. X. Zhao, Z. Zhang, and F. Wei (2025) Reasoning with exploration: an entropy perspective. arXiv preprint arXiv:2506.14758. Cited by: §1, §2.2, §5.1. X. Chu, H. Huang, X. Zhang, F. Wei, and Y. Wang (2025) Gpg: a simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546. Cited by: §2.1, §2.2. H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang (2023) Raft: reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767. Cited by: §1, §2.3. T. Fang, Z. Zhang, X. Wang, R. Wang, C. Qin, Y. Wan, J. Ma, C. Zhang, J. Chen, X. Li, H. Zhang, H. Mi, and D. Yu (2025) Cognitive kernel-pro: A framework for deep research agents and agent foundation models training. CoRR abs/2508.00414. External Links: Link, Document, 2508.00414 Cited by: §2.1. Y. Fang, J. Lin, X. Fu, C. Qin, and H. Shi (2026) Placing puzzle pieces where they matter: a question augmentation framework for reinforcement learning. External Links: 2604.15830, Link Cited by: §1. C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. (2023) Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998. Cited by: §2.3. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2.1. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), Cited by: §2.2. C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. (2024) Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008. Cited by: §5.1. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §5.1. J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum (2025) Open-reasoner-zero: an open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290. Cited by: §2.1. HuggingFace (2025) Math-verify. External Links: Link Cited by: Appendix E, §5.1. A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. (2024) Openai o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §1, §2.1. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. (2022) Solving quantitative reasoning problems with language models. Conference on Neural Information Processing Systems (NeurIPS). Cited by: §5.1. M. Li, Q. Zeng, T. Fang, Z. Liang, L. Song, Q. Liu, H. Mi, and D. Yu (2026a) Verified critical step optimization for LLM agents. CoRR abs/2602.03412. External Links: Link, Document, 2602.03412 Cited by: §2.2. Y. Li (2025) Logit dynamics in softmax policy gradient methods. arXiv preprint arXiv:2506.12912. Cited by: §1, §2.2. Z. Li, Z. Ma, Y. Pan, Z. Zhang, X. Lv, B. Li, J. Gao, J. Zhang, C. Yuan, B. Li, and W. Hu (2026b) Making mllms blind: adversarial smuggling attacks in mllm content moderation. External Links: 2604.06950, Link Cited by: Limitations. P. Liu, S. Wang, X. Wang, W. Ye, and S. Zhang (2021a) QuadrupletBERT: an efficient model for embedding-based large-scale retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 3734–3739. Cited by: §2.3. P. Liu, X. Wang, S. Wang, W. Ye, X. Xi, and S. Zhang (2021b) Improving embedding-based large-scale retrieval via label enhancement. In Findings of the Association for Computational Linguistics: EMNLP 2021, p. 133–142. Cited by: §2.3. P. Liu, X. Xi, W. Ye, and S. Zhang (2022) Label smoothing for text mining. In Proceedings of the 29th international conference on computational linguistics, p. 2210–2219. Cited by: §2.3. Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: a critical perspective. arXiv preprint arXiv:2503.20783. Cited by: §1, §2.1, §3. M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, T. Zhang, L. E. Li, et al. (2025) Deepscaler: surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog. Cited by: §1, §2.2. MAA (2023) American mathematics competitions - amc. External Links: Link Cited by: §5.1. MAA (2025) American invitational mathematics examination - aime. External Links: Link Cited by: §5.1. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu (2016) Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), Cited by: §2.2. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Conference on Neural Information Processing Systems (NeurIPS). Cited by: §1, §2.1, §2.2. T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2015) Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §1, §2.3. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: Appendix C, §2.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix B, Appendix C, §1, §1, §2.1, §2.2, §2.3, §3, §5.1, Remark 1, Theorem 1. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In European Conference on Computer Systems (EuroSys), Cited by: §5.1. R. Sutton and A. Barto (2018) Reinforcement learning: an introduction. Cited by: Remark 1. H. Tan and J. Pan (2025) GTPO and grpo-s: token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508.04349. Cited by: §1, §2.2, §5.1. K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025) Kimi k1. 5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: §2.1. Y. Tong, X. Zhang, R. Wang, R. Wu, and J. He (2024) Dart-math: difficulty-aware rejection tuning for mathematical problem-solving. Conference on Neural Information Processing Systems (NeurIPS). Cited by: §1, §2.3. S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, et al. (2025) Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939. Cited by: §1, §2.2, §5.1. S. Yang, C. Dou, P. Guo, K. Lu, Q. Ju, F. Deng, and R. Xin (2025) DCPO: dynamic clipping policy optimization. arXiv preprint arXiv:2509.02333. Cited by: §1, §2.2. J. Yao, Y. Hao, H. Zhang, H. Dong, W. Xiong, N. Jiang, and T. Zhang (2025) Optimizing chain-of-thought reasoners via gradient variance minimization in rejection sampling and rl. arXiv preprint arXiv:2505.02391. Cited by: Appendix C, §1, §2.3. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. (2025a) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §1, §1, §2.1, §2.2, §3, §5.1, §5.1. W. Yu, Z. Liang, C. Huang, K. Panaganti, T. Fang, H. Mi, and D. Yu (2025b) Guided self-evolving llms with minimal human supervision. CoRR abs/2512.02472. External Links: Link, Document, 2512.02472 Cited by: §2.3. Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang (2025) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. arXiv preprint arXiv:2504.13837. Cited by: §5.1. X. Zhu, M. Xia, Z. Wei, W. Chen, D. Chen, and Y. Meng (2025) The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347. Cited by: §1, §2.2, §5.1. Appendix A Derivation of Entropy Change Estimation for Gradient-Aware Advantage Modulation We derive the first-order entropy change estimation for tabular softmax policy through a factorized decomposition approach. Let the actor policy πθ _θ be a tabular softmax policy where each state-action pair (s,a)(s,a) is associated with an individual logit parameter zs,a=θs,az_s,a= _s,a. A.1 Preliminary Definitions Definition 1 (Policy Concentration Measure). For a policy πθ(⋅|s) _θ(·|s) at state s, we define the concentration coefficient: Φ(πθ|s):=∑aπθ(a|s)2. ( _θ|s):= _a _θ(a|s)^2. (16) This measures the degree of policy concentration, with higher values indicating a more deterministic policy. Definition 2 (Entropy-Weighted Log-Probability). We define the centered log-probability for action a under policy πθ _θ at state s: Λθ(a|s):=logπθ(a|s)+ℋ(πθ|s). _θ(a|s)= _θ(a|s)+H( _θ|s). (17) This represents the deviation of action a’s log-probability from the policy’s average entropy, capturing the information-theoretic distance from uniform randomness. A.2 Entropy Gradient Derivation We use Taylor’s expansion under first-order approximation: ℋ(πθk+1∣s) ( _θ^k+1 s) (18) ≈ℋ(πθk∣s) ( _θ^k s) +⟨∇ℋ(πθk∣s),(zk+1−zk)⟩. + ( _θ^k s),(z^k+1-z^k) . To derive ∇ℋ(πθk∣s) ( _θ^k s), we start from the definition of entropy: ∇θℋ(πθ∣s) _θH( _θ s) (19) =∇θ(−a∼πθ(⋅∣s)[logπθ(a∣s)]) = _θ (-E_a _θ(· s)[ _θ(a s)] ) =−∇θa∼πθ(⋅∣s)[logπθ(a∣s)] =- _θE_a _θ(· s)[ _θ(a s)] =−a∼πθ(⋅∣s)[∇θlogπθ(a∣s)] =-E_a _θ(· s)[ _θ _θ(a s)] −a∼πθ(⋅∣s)[logπθ(a∣s) -E_a _θ(· s)[ _θ(a s) ×∇θlogπθ(a∣s)] × _θ _θ(a s)] =0−a∼πθ(⋅∣s)[logπθ(a∣s) =0-E_a _θ(· s)[ _θ(a s) ×∇θlogπθ(a∣s)] × _θ _θ(a s)] =−a∼πθ(⋅∣s)[logπθ(a∣s) =-E_a _θ(· s)[ _θ(a s) ×∇θlogπθ(a∣s)]. × _θ _θ(a s)]. The first term equals zero because a∼πθ(⋅∣s)[∇θlogπθ(a∣s)]=∇θ∑aπθ(a∣s)=∇θ1=0E_a _θ(· s)[ _θ _θ(a s)]= _θ _a _θ(a s)= _θ1=0. A.3 Softmax Derivative and Centered Logit Updates For a softmax policy πθ(a∣s)=exp(zs,a)∑a′exp(zs,a′) _θ(a s)= (z_s,a) _a (z_s,a ), the derivative of the log-probability with respect to any logit parameter zs,a′z_s,a is: ∂logπθ(a∣s)∂zs,a′=a=a′ ∂ _θ(a s)∂ z_s,a =1\a=a \ (20) −πθ(a′∣s). - _θ(a s). We now introduce a centered form of logit updates that exploits the translation invariance of softmax: Definition 3 (Centered Logit Change). Define the centered logit change as: δzs,a δ z_s,a :=zs,ak+1−zs,ak =z^k+1_s,a-z^k_s,a (21) −a′∼πθk(⋅|s)[zs,a′k+1−zs,a′k]. -E_a _θ^k(·|s)[z^k+1_s,a -z^k_s,a ]. This removes the global translation component, which does not affect softmax probabilities. Using Equation (20), we can express the first-order entropy change: ⟨∇θℋ(θk∣s),(zk+1−zk)⟩ _θH(θ^k s),(z^k+1-z^k) (22) =−a∼πθk(⋅∣s)[logπθk(a∣s) =-E_a _θ^k(· s) [ _θ^k(a s) ×∑a′(a=a′−πθk(a′∣s)) × _a (1\a=a \- _θ^k(a s) ) ×(zs,a′k+1−zs,a′k)] ×(z^k+1_s,a -z^k_s,a ) ] =−a∼πθk(⋅∣s)[logπθk(a∣s) =-E_a _θ^k(· s) [ _θ^k(a s) ×(zs,ak+1−zs,ak × (z^k+1_s,a-z^k_s,a −∑a′πθk(a′∣s)(zs,a′k+1−zs,a′k))] - _a _θ^k(a s)(z^k+1_s,a -z^k_s,a ) ) ] =−a∼πθk(⋅∣s)[logπθk(a∣s)⋅δzs,a]. =-E_a _θ^k(· s)[ _θ^k(a s)·δ z_s,a]. Expanding further and using the covariance decomposition [XY]=[X][Y]+Cov(X,Y)E[XY]=E[X]E[Y]+Cov(X,Y): ⟨∇θℋ(θk∣s),(zk+1−zk)⟩ _θH(θ^k s),(z^k+1-z^k) (23) =−a∼πθk(⋅∣s)[logπθk(a∣s) =-E_a _θ^k(· s)[ _θ^k(a s) ×(zs,ak+1−zs,ak)] ×(z^k+1_s,a-z^k_s,a)] +a∼πθk(⋅∣s)[logπθk(a∣s)] +E_a _θ^k(· s)[ _θ^k(a s)] ×a′∼πθk(⋅∣s)[zs,a′k+1−zs,a′k] ×E_a _θ^k(· s)[z^k+1_s,a -z^k_s,a ] =−a∼πθk(⋅∣s)[(logπθk(a∣s) =-E_a _θ^k(· s) [ ( _θ^k(a s) −a∼πθk(⋅∣s)[logπθk(a∣s)]) -E_a _θ^k(· s)[ _θ^k(a s)] ) ×δzs,a] ×δ z_s,a ] =−a∼πθk(⋅∣s)[Λθk(a|s)⋅δzs,a], =-E_a _θ^k(· s)[ _θ^k(a|s)·δ z_s,a], where we used Definition 2 in the last step, noting that −[logπθ]=ℋ(πθ|s)-E[ _θ]=H( _θ|s). A.4 GRPO Update Coefficient For GRPO, we define the composite update coefficient: ξi,t(a):=clip⋅ri,t⋅Ai,t, _i,t(a):=I_clip· r_i,t· A_i,t, (24) where clipI_clip is the clipping indicator function, ri,t=πθ(ai,t∣si,t)πref(ai,t∣si,t)r_i,t= _θ(a_i,t s_i,t) _ref(a_i,t s_i,t) is the importance sampling ratio, and Ai,tA_i,t is the advantage estimate. This coefficient combines three essential components: clipping for stability, importance weighting for off-policy correction, and advantage for policy improvement direction. Lemma 1 (Centered Logit Update for GRPO). Under the GRPO update rule, the per-sample stochastic logit update satisfies: zs,k+1−zs,k z_s,k^k+1-z_s,k^k =ηξi,t(a=k−πθ(k∣s)). =η\, _i,t\, (1\a=k\- _θ(k s) ). (25) Specifically, for the sampled action a and other actions a′≠a ≠ a: zs,ak+1−zs,ak z_s,a^k+1-z_s,a^k =ηξi,t(1−πθ(a∣s)) =η\, _i,t\,(1- _θ(a s)) (26) zs,a′k+1−zs,a′k z_s,a ^k+1-z_s,a ^k =−ηξi,tπθ(a′∣s). =-η\, _i,t\, _θ(a s). Remark 1. By the definition of advantage functions Sutton and Barto (2018) and GRPO’s group normalization Shao et al. (2024), advantages satisfy a∼πθ(⋅∣s)[Ai,t]=0E_a _θ(· s)[A_i,t]=0. A.5 Factorized Entropy Change Formula Substituting Lemma 1 into Equation (23): ⟨∇θℋ(θk∣s),(zk+1−zk)⟩ _θH(θ^k s),(z^k+1-z^k) (27) =−∑kπθk(k∣s)Λθk(k|s)(zs,k+1−zs,k) =- _k _θ^k(k s)\, _θ^k(k|s) (z^k+1_s,k-z^k_s,k ) =−ηξi,t[πθk(a∣s)Λθk(a|s) =-η\, _i,t\, [ _θ^k(a s)\, _θ^k(a|s) −∑kπθk(k∣s)2Λθk(k|s)]. - _k _θ^k(k s)^2\, _θ^k(k|s) ]. Taking expectation over a∼πθk(⋅∣s)a _θ^k(· s) under GRPO’s group normalization, the entropy change is dominated by: Δℋ(πθk|s) ( _θ^k|s) ≈−ηa∼πθk[πθk(a∣s)Λθk(a|s)ξi,t(a)]. ≈-η\,E_a _θ^k [ _θ^k(a s) _θ^k(a|s)\, _i,t(a) ]. (28) We now state our main result: Theorem 1 (Factorized Entropy Change). Following GRPO’s design Shao et al. (2024) with standard clipping, the first-order entropy change admits: Δℋ(πθk|s)≈−η∑aπθk(a|s)2⏟Concentration×Λθk(a|s)⏟Info Deviation×ξi,t(a)⏟Update Coeff. aligned ( _θ^k|s)&≈-η _a _θ^k(a|s)^2_Concentration\\ & × _θ^k(a|s)_Info Deviation\\ & × _i,t(a)_Update Coeff. aligned (29) where Λθk(a|s)=logπθk(a|s)+ℋ(πθk|s) _θ^k(a|s)= _θ^k(a|s)+H( _θ^k|s) (Definition 2), and: • πθk(a|s)2 _θ^k(a|s)^2: Policy concentration • Λθk(a|s) _θ^k(a|s): Info-theoretic deviation • ξi,t(a)=clip⋅ri,t⋅Ai,t _i,t(a)=I_clip· r_i,t· A_i,t: Update coefficient (Eq. 24) Proof. From Equation (28): Δℋ(πθk|s) ( _θ^k|s) (30) ≈−ηa∼πθk[πθk(a∣s)Λθk(a|s)ξi,t(a)] ≈-η\,E_a _θ^k [ _θ^k(a s) _θ^k(a|s)\, _i,t(a) ] =−η∑aπθk(a|s)⋅πθk(a∣s)×Λθk(a|s)ξi,t(a) =-η _a _θ^k(a|s)· _θ^k(a s)× _θ^k(a|s)\, _i,t(a) =−η∑aπθk(a|s)2Λθk(a|s)⋅ξi,t(a), =-η _a _θ^k(a|s)^2 _θ^k(a|s)· _i,t(a), where the second line expands the expectation as a∼π[f(a)]=∑aπ(a|s)⋅f(a)E_a π[f(a)]= _aπ(a|s)· f(a), and the third line simplifies the product. Substituting Definition 2 yields the boxed form. ∎ Remark 2. This factorization reveals three orthogonal mechanisms governing entropy dynamics: (i) concentration amplifies updates for high-probability actions, (i) information deviation directs change based on distance from maximum entropy, and (i) update coefficient modulates the magnitude via clipped importance-weighted advantages. The multiplicative structure implies that entropy change vanishes when any factor approaches zero, providing natural regularization. Appendix B Gradient-Entropy Relationship: Proof of Expected Gradient Norm Upper Bound This appendix establishes the relationship between the score function’s squared L2 norm and policy entropy in softmax policies, then extends it to advantage-weighted updates. Core Result. For softmax policy πθ(⋅|s)=(π1,…,π||) _θ(·|s)=( _1,…, _|V|) with logits z(s)=(z1(s),…,z||(s))z(s)=(z_1(s),…,z_|V|(s)) where πk=exp(zk)/∑jexp(zj) _k= (z_k)/ _j (z_j), the gradient of logπk _k with respect to logits is ∂logπk∂zi=δik−πi ∂ _k∂ z_i= _ik- _i (Kronecker delta δik _ik). The squared norm for action k is: ‖∇zlogπk‖2 \| _z _k\|^2 =∑i=1||(δik−πi)2 = _i=1^|V|( _ik- _i)^2 (31) =(1−πk)2+∑i≠kπi2 =(1- _k)^2+ _i≠ k _i^2 =1−2πk+∑j=1||πj2. =1-2 _k+ _j=1^|V| _j^2. Taking expectation over actions sampled from πθ _θ: ak∼πθ(⋅|s)[‖∇zlogπk‖2] _a_k _θ(·|s) [\| _z _k\|^2 ] (32) =∑k=1||πk(1−2πk+∑j=1||πj2) = _k=1^|V| _k (1-2 _k+ _j=1^|V| _j^2 ) =∑k=1||πk−2∑k=1||πk2 = _k=1^|V| _k-2 _k=1^|V| _k^2 +(∑j=1||πj2)∑k=1||πk⏟=1 + ( _j=1^|V| _j^2 ) _k=1^|V| _k_=1 =1−∑k=1||πk2. =1- _k=1^|V| _k^2. The term ∑k=1||πk2 _k=1^|V| _k^2 is the collision probability (probability that two independent samples from πθ _θ are identical), ranging from 1/||1/|V| (uniform) to 11 (deterministic). Entropy Upper Bound. To connect collision probability to Shannon entropy, apply Jensen’s inequality. Since x↦logx x is concave, ∑kπklogπk≤log(∑kπk2) _k _k _k≤ ( _k _k^2). Multiplying by −1-1 and exponentiating: ℋ(πθ|s) ( _θ|s) =−∑k=1||πklogπk =- _k=1^|V| _k _k (33) ≥−log(∑k=1||πk2) ≥- ( _k=1^|V| _k^2 ) ⇒∑k=1||πk2≥e−ℋ(πθ|s). _k=1^|V| _k^2≥ e^-H( _θ|s). Thus ak∼πθ(⋅|s)[‖∇zlogπk‖2]=1−∑k=1||πk2≤1−e−ℋ(πθ|s)E_a_k _θ(·|s)[\| _z _k\|^2]=1- _k=1^|V| _k^2≤ 1-e^-H( _θ|s). High entropy (ℋ(πθ|s)≫0H( _θ|s) 0) yields e−ℋ≈0e^-H≈ 0 and gradient norm ≈1≈ 1 (vigorous learning); low entropy (ℋ(πθ|s)≈0H( _θ|s)≈ 0) yields e−ℋ≈1e^-H≈ 1 and gradient norm ≈0≈ 0 (stable convergence). Advantage-Weighted Update Magnitude. For policy gradient update with learning rate η and advantage A, the logit update for sampled action aka_k is Δz(s)=ηA(k−πθ(⋅|s)) z(s)=η A(e_k- _θ(·|s)) where ke_k is the one-hot vector. The squared L2 norm is: ‖Δz(s)‖22 \| z(s)\|_2^2 =η2A2(1−2πk+∑j=1||πj2). =η^2A^2 (1-2 _k+ _j=1^|V| _j^2 ). (34) Taking expectation over ak∼πθa_k _θ gives [‖Δz(s)‖22]=η2[A2(1−2πk+∑jπj2)]E[\| z(s)\|_2^2]=η^2E[A^2(1-2 _k+ _j _j^2)]. Following GRPO’s design Shao et al. (2024) where advantages are sequence-level, the dependence between A and per-token probabilities πk _k is negligible, allowing the approximation: ak∼πθ(⋅|s)[‖Δz(s)‖22] _a_k _θ(·|s) [\| z(s)\|_2^2 ] (35) =η2[A2](1−∑k=1||πk2) =η^2E[A^2] (1- _k=1^|V| _k^2 ) ≤η2[A2](1−exp(−ℋ(πθ|s))). ≤η^2E[A^2] (1- (-H( _θ|s) ) ). This combines logit dynamics with advantage-weighted scaling and entropy bounds. High-confidence tokens (πk≈1 _k≈ 1) produce small expected update magnitudes (∑jπj2≈1 _j _j^2≈ 1), creating gradient attenuation for confident correct actions—the theoretical basis for gradient compensation in our method. Appendix C Gradient Variance Minimization for Dynamic Rollout Allocation This appendix provides the complete derivation of the optimal rollout allocation formula. Optimal Allocation via Lagrange Multipliers. For prompt qiq_i with nin_i rollouts, the gradient estimator g^i=1ni∑k=1nigi,k g_i= 1n_i _k=1^n_ig_i,k (where gi,k=1G∑j=1GA^j,k∇θlogπθ(oj,k|qi)g_i,k= 1G _j=1^G A_j,k _θ _θ(o_j,k|q_i) with A^j,k A_j,k the group-normalized advantage) has variance Var[g^i]=σi2/niVar[ g_i]= _i^2/n_i where σi2=[‖gi,k‖22] _i^2=E[\|g_i,k\|_2^2] is the single-sample gradient variance. Minimizing total variance ∑i=1Nσi2/ni _i=1^N _i^2/n_i subject to budget constraint ∑ini=B _in_i=B, we construct Lagrangian ℒ=∑iσi2/ni+λ(∑ini−B)L= _i _i^2/n_i+λ( _in_i-B). The first-order condition ∂ℒ/∂ni=−σi2/ni2+λ=0 /∂ n_i=- _i^2/n_i^2+λ=0 gives ni=σi/λn_i= _i/ λ. Substituting into the constraint ∑iσi/λ=B _i _i/ λ=B yields λ=(∑kσk)/B λ=( _k _k)/B, thus: ni∗=B⋅σi∑k=1Nσk.n_i^*=B· _i _k=1^N _k. (36) Variance Decomposition. To implement this principle, we decompose σi2=[‖gi,k‖22] _i^2=E[\|g_i,k\|_2^2] where gi,k=1G∑j=1GAj,k∇θlogπθ(oj,k|qi)g_i,k= 1G _j=1^GA_j,k _θ _θ(o_j,k|q_i). The squared norm is: ‖gi,k‖22 \|g_i,k\|_2^2 =1G2∥∑j=1GAj,k×∇θlogπθ(oj,k|qi)∥22. = 1G^2 \| _j=1^GA_j,k× _θ _θ(o_j,k|q_i) \|_2^2. (37) Following the treatment in high-dimensional gradient estimation Schulman et al. (2017); Yao et al. (2025), [⟨∇j,∇j′⟩]≈0E[ _j, _j ]≈ 0, yielding: [‖gi,k‖22] [\|g_i,k\|_2^2] ≈1G[A2∥∇θlogπθ(o|qi)∥22]. ≈ 1GE [A^2\| _θ _θ(o|q_i)\|_2^2 ]. (38) For GRPO with group-normalized advantages Shao et al. (2024), the gradient variance exhibits positive correlation with both reward variance and expected gradient magnitude: [‖gi,k‖22] [\|g_i,k\|_2^2] ∝Var(R)[∥∇θlogπθ(o|qi)∥22]. (R)E[\| _θ _θ(o|q_i)\|_2^2]. (39) For response o=o1,…,oTo=\o_1,…,o_T\, the policy gradient decomposes as ∇θlogπθ(o|qi)=∑t=1T∇θlogπθ(ot|qi,o<t) _θ _θ(o|q_i)= _t=1^T _θ _θ(o_t|q_i,o_<t). Applying the same approximation to token-level gradients: [∥∇θlogπθ(o|qi)∥22]≈[∑t=1T∥∇θlogπθ(ot|qi,o<t)∥22]. [\| _θ _θ(o|q_i)\|_2^2] [ _t=1^T\| _θ _θ(o_t|q_i,o_<t)\|_2^2 ]. (40) From Appendix B, the expected squared gradient norm for token t is [∥∇θlogπθ(ot|qi,o<t)∥22]=1−C(Pt)E[\| _θ _θ(o_t|q_i,o_<t)\|_2^2]=1-C(P_t) where C(Pt)=∑k=1||πk2(qi,o<t)C(P_t)= _k=1^|V| _k^2(q_i,o_<t) is the collision probability. Defining average collision C¯i=o∼πθ(⋅|qi)[1|o|∑t=1|o|C(Pt)] C_i=E_o _θ(·|q_i)[ 1|o| _t=1^|o|C(P_t)]: σi2∝Var(R)⋅(1−C¯i). _i^2 (R)·(1- C_i). (41) Computing C(Pt)=∑k=1||πk2C(P_t)= _k=1^|V| _k^2 for all tokens is expensive. Since high entropy corresponds to low collision probability (uniform distributions have low C(P)C(P), concentrated distributions have high C(P)C(P)), we use entropy as a proxy. Defining average per-token entropy ℋ¯i=o∼πθ(⋅|qi)[1|o|∑t=1|o|ℋ(πθ|qi,o<t)] H_i=E_o _θ(·|q_i) [ 1|o| _t=1^|o|H( _θ|q_i,o_<t) ], we approximate: σi∝std(R)⋅f(ℋ¯i) _i (R)· f( H_i) (42) where f(⋅)f(·) is an increasing function. Using f(ℋ)=1−e−ℋf(H)= 1-e^-H captures the relationship: higher entropy leads to larger gradient variance. Connection to Bernoulli Variance. For binary rewards, Var(R)=p(1−p)Var(R)=p(1-p) where p is success probability. The unbiased estimator using historical statistics (kik_i correct responses out of GiG_i total rollouts) is: Pi=ki(Gi−ki)Gi(Gi−1),[Pi]=pi(1−pi). P_i= k_i(G_i-k_i)G_i(G_i-1), [P_i]=p_i(1-p_i). (43) This PiP_i serves as a proxy for σi _i because: (i) it directly estimates Var(R)Var(R) through Bernoulli variance, (i) empirical accuracy p^i=ki/Gi p_i=k_i/G_i near the variance-maximizing value correlates with high policy entropy (models actively exploring multiple solution paths), and (i) together, PiP_i identifies high-σi _i problems per the variance decomposition. The water-level allocation algorithm implements ni∗∝Pin_i^* P_i with constraints Gmin,GmaxG_ ,G_ . Variance Reduction Guarantee. The optimal allocation achieves total variance Var∗=1B(∑iσi)2Var^*= 1B( _i _i)^2, while uniform allocation (ni=B/Nn_i=B/N) yields Varuniform=NB∑iσi2Var_uniform= NB _i _i^2. Their ratio is: Var∗Varuniform=(∑iσi)2N∑iσi2≤1, Var^*Var_uniform= ( _i _i)^2N _i _i^2≤ 1, (44) where the inequality follows from Cauchy-Schwarz, with equality only when all σi _i are equal. When gradient variances are heterogeneous across problems (typical in RLVR), this adaptive allocation provides substantial variance reduction. Appendix D Dynamic Rollout Allocation: Operational Details. Our allocation mechanism determines rollout quantities for each prompt in the training batch, operating independently of prompt selection strategies (e.g., random sampling, curriculum learning). Given a batch of N prompts and total rollout budget B, the water-level algorithm distributes nin_i rollouts per prompt proportionally to variance proxies PiP_i while respecting minimum and maximum constraints. The allocation process guarantees complete budget utilization through iterative refinement. Initially, each prompt receives a baseline allocation based on its PiP_i value. When constraints become active (some prompts hit minimum or maximum bounds), residual budget is redistributed among unconstrained prompts using the same proportional rule. This continues until all budget is assigned and ∑ini=B _in_i=B holds exactly. If the total minimum requirement exceeds available budget, we proportionally scale down the minimum allocation to ensure feasibility. For prompts with insufficient historical data (few prior observations), computing reliable variance proxies PiP_i is infeasible. In such cases, we apply uniform allocation as a fallback strategy, distributing rollouts equally among data-scarce prompts. As prompts accumulate responses through repeated selection in training batches, their historical statistics become sufficient to support adaptive allocation. This transition from uniform to variance-driven distribution occurs naturally as the PiP_i estimates stabilize with increased sample counts. The proportional allocation naturally adapts to estimation errors. If historical PiP_i overestimates current variance (due to policy improvement), subsequent rollouts reveal lower empirical variance, which updates PiP_i for future allocations. Conversely, underestimated prompts exhibit higher gradient variance, triggering increased allocation in later iterations. This feedback loop ensures the distribution tracks evolving problem difficulty without manual tuning. In boundary cases where all PiP_i values become negligibly small (indicating near-deterministic outcomes), the algorithm defaults to uniform distribution to maintain numerical stability. (a) Qwen2.5-Math-1.5B (b) Qwen2.5-Math-7B Figure 6: Per-step training time on Qwen2.5-Math models. Bold lines: smoothed; light lines: raw measurements. Appendix E Detailed Training Configuration All experiments are conducted on 8× NVIDIA A100 80GB GPUs per node (1-2 nodes depending on model scale) with mixed precision training (bfloat16) and FSDP for distributed training. The actor model uses Tensor Parallelism (TP=2) during rollout generation with vLLM for efficient inference. Training prompts are left-truncated to 2048 tokens if exceeding the maximum length, and we apply Math-Verify HuggingFace (2025) for answer verification during training and evaluation. The training dataset is shuffled at the beginning of each epoch, and rollout budget allocation is updated dynamically based on historical success statistics accumulated across iterations. Table 3 presents the complete hyperparameters and configuration details for our experiments. Table 3: Detailed training configuration for DynaMO experiments. Category Parameter Value Training Hyperparameters Batch Configuration Generation Batch Size 512 Update Batch Size 32 PPO Mini-batch Size 32 Rollout Allocation Average Rollouts per Prompt 16 Dynamic Range [8, 24] Optimization Learning Rate (Actor) 1×10−61× 10^-6 Learning Rate (Critic) 1×10−51× 10^-5 Weight Decay 0.1 / 0.01 Gradient Clipping 1.0 Warmup Steps 10 Warmup Style constant GRPO-Specific Settings Clipping Clip Ratio 0.2 Entropy Entropy Coefficient 0 KL Penalty KL Coefficient 0.0 Advantage Estimator Type GRPO Normalize by Std True Gamma (Discount Factor) 1.0 Lambda (GAE) 1.0 Inference & Rollout Sampling Strategy Temperature 1.0 Top-p 1.0 Top-k -1 (disabled) Sequence Lengths Max Prompt Length 2048 Max Response Length 8192 Rollout Engine Backend vLLM Tensor Parallel Size 2 GPU Memory Utilization 0.8 Max Num Sequences 1024 Max Batched Tokens 10240 Appendix F Detailed Description of Benchmarks To comprehensively evaluate mathematical reasoning capabilities, we employ six widely-adopted benchmarks spanning competition-level challenges (AIME, AMC), curriculum-aligned assessments (MATH), and specialized STEM domains (Minerva, OlympiadBench). These benchmarks encompass diverse mathematical subfields and problem formats, enabling rigorous evaluation across multiple dimensions of reasoning proficiency. Table 4 summarizes the key characteristics of each benchmark. Benchmark Core Description Key Characteristics AIME American Invitational Mathematics Examination - high school competition • 15 challenging problems per round • Integer answers (0-999) • Algebra, Geometry, Number Theory, Combinatorics • Multi-step reasoning required • Top AMC performers participate AMC American Mathematics Competitions - tiered assessment system • Three tiers (AMC 8/10/12) for different grades • 25 multiple-choice problems per tier • Curriculum-aligned design • Comprehensive secondary mathematics coverage • Standardized difficulty progression MATH-500 Curated subset from MATH dataset • 500 problems from comprehensive collection • Seven domains (Algebra, Geometry, Number Theory, etc.) • Five difficulty levels • Formal mathematical reasoning • Step-by-step solution verification Minerva Math Technical STEM benchmark • Undergraduate to graduate difficulty • Physics, Chemistry, Biology applications • Symbolic manipulation and formula derivation • Domain-specific knowledge integration • Quantitative problem-solving OlympiadBench Bilingual Olympiad-level benchmark • Large-scale competition problem collection • Bilingual (English and Chinese) • Mathematics, Physics, Chemistry, Biology • Theorem-proving and open-ended problems • Multimodal inputs (text, diagrams, equations) Table 4: Characteristics of mathematical reasoning benchmarks used in our evaluation. These benchmarks provide comprehensive coverage across difficulty levels (secondary to graduate), problem formats (multiple-choice, integer answers, open-ended), and mathematical domains (pure and applied mathematics), enabling thorough assessment of reasoning capabilities. Appendix G Efficiency Analysis Figure 6 presents per-step training time across model scales. DynaMO maintains comparable efficiency to baselines on both 1.5B and 7B models, as the additional computations—Bernoulli variance estimation, water-level allocation, and token-level modulation—involve only lightweight arithmetic operations on existing logits and statistics, incurring negligible overhead relative to model inference.