Paper deep dive
From $\boldsymbol{\logπ}$ to $\boldsymbolπ$: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Chaowen Hu, Cong Qin, Zekai Shao, Binbin Zheng, Lu Pan, Ke Zeng
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 5:08:47 AM
Summary
The paper introduces Decoupled Gradient Policy Optimization (DGPO), a novel reinforcement learning algorithm for Large Language Models (LLMs) that addresses the instability of 'soft clipping' methods. By shifting the optimization primitive from log-probability gradient to probability gradient, DGPO employs a decoupled decay mechanism to balance stability and exploration, effectively preventing weight divergence at clipping boundaries.
Entities (5)
Relation Signals (3)
DGPO → evaluatedon → DeepSeek-R1-Distill-Qwen
confidence 95% · Extensive experiments across DeepSeek-R1-Distill-Qwen series models (1.5B/7B/14B) demonstrate that DGPO consistently outperforms strong baselines
DGPO → improves → RLVR
confidence 95% · DGPO consistently outperforms strong baselines on various mathematical benchmarks, offering a robust and scalable solution for RLVR.
DGPO → replaces → Hard Clipping
confidence 90% · DGPO replaces 'hard clipping' with a decoupled decay mechanism applied to the probability gradient weight.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed a leap in Large Language Model (LLM) reasoning, yet its optimization dynamics remain fragile. Standard algorithms like GRPO enforce stability via ``hard clipping'', which inadvertently stifles exploration by discarding gradients of tokens outside the trust region. While recent ``soft clipping'' methods attempt to recover these gradients, they suffer from a critical challenge: relying on log-probability gradient ($\nabla_\theta\log \pi_\theta$) yields divergent weights as probabilities vanish, destabilizing LLM training. We rethink this convention by establishing probability gradient ($\nabla_\theta \pi_\theta$) as the superior optimization primitive. Accordingly, we propose Decoupled Gradient Policy Optimization (DGPO), which employs a decoupled decay mechanism based on importance sampling ratios. By applying asymmetric, continuous decay to boundary tokens, DGPO resolves the conflict between stability and sustained exploration. Extensive experiments across DeepSeek-R1-Distill-Qwen series models (1.5B/7B/14B) demonstrate that DGPO consistently outperforms strong baselines on various mathematical benchmarks, offering a robust and scalable solution for RLVR. Our code and implementation are available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.14389v1
- Canonical: https://arxiv.org/abs/2603.14389v1
Trouble viewing inline? Open PDF directly →
Full Text
96,976 characters extracted from source content.
Expand or collapse full text
From π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight Xiaoliang Fu1,2,*, Jiaye Lin1,*, Yangyi Fang1,3,*, Chaowen Hu1, Cong Qin1,4, Zekai Shao2, Binbin Zheng1,5, Lu Pan1, Ke Zeng1,† 1Meituan 2Fudan University 3Tsinghua University 4Peking University 5University of Science and Technology of China fuxiaoliang04, linjiaye, fangyangyi@meituan.com Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed a leap in Large Language Model (LLM) reasoning, yet its optimization dynamics remain fragile. Standard algorithms like GRPO enforce stability via “hard clipping”, which inadvertently stifles exploration by discarding gradients of tokens outside the trust region. While recent “soft clipping” methods attempt to recover these gradients, they suffer from a critical challenge: relying on log-probability gradient (∇θlogπθ _θ _θ) yields divergent weights as probabilities vanish, destabilizing LLM training. We rethink this convention by establishing probability gradient (∇θπθ _θ _θ) as the superior optimization primitive. Accordingly, we propose Decoupled Gradient Policy Optimization (DGPO), which employs a decoupled decay mechanism based on importance sampling ratios. By applying asymmetric, continuous decay to boundary tokens, DGPO resolves the conflict between stability and sustained exploration. Extensive experiments across DeepSeek-R1-Distill-Qwen series models (1.5B/7B/14B) demonstrate that DGPO consistently outperforms strong baselines on various mathematical benchmarks, offering a robust and scalable solution for RLVR. Our code and implementation are available at: github.com/VenomRose-Juri/DGPO-RL. From π π to π π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight Xiaoliang Fu1,2,*, Jiaye Lin1,*, Yangyi Fang1,3,*, Chaowen Hu1, Cong Qin1,4, Zekai Shao2, Binbin Zheng1,5, Lu Pan1, Ke Zeng1,† 1Meituan 2Fudan University 3Tsinghua University 4Peking University 5University of Science and Technology of China fuxiaoliang04, linjiaye, fangyangyi@meituan.com †footnotetext: * Equal contribution. † Corresponding author. 1 Introduction Reinforcement Learning (RL) has emerged as a transformative paradigm for aligning Large Language Models (LLMs) with human intent, shifting focus from imitation learning to goal-driven optimization Ouyang et al. (2022); Shao et al. (2024); Guo et al. (2025). In reasoning-intensive domains like mathematics, Reinforcement Learning with Verifiable Rewards (RLVR) achieves remarkable success by leveraging ground-truth feedback to enhance accuracy and logical coherence Lightman et al. (2023b); Shao et al. (2024). Beyond standard benchmarks, stronger reasoning has also shown promise in optimization modeling, tool-augmented mathematical problem solving, scientific explanation, and narrative generation Liu et al. (2026); Luo et al. (2025); Shao et al. (2025b, a). Figure 1: Schematic overview of our DGPO algorithm. While “hard clipping” methods (GRPO/ASPO) discard gradients at boundaries, and prior “soft clipping” approaches (CISPO/GPPO/CE-GPPO) risk divergence on the left boundary and limit exploration on the right boundary, DGPO optimizes the gradient decay mechanism accordingly. By enforcing a controlled “Slow Down” behavior for stability and a “Slow Down Gently” behavior to sustain exploration, DGPO effectively resolves the exploration-stability conflict while maintaining minimal bias against the true policy gradient. Despite this promise, RLVR optimization remains fragile from the conflict between exploration and stability. Algorithms like PPO Schulman et al. (2015) and GRPO Shao et al. (2024) enforce a trust region by “hard clipping” the Importance Sampling (IS) ratio (πθ/πθold _θ/ _ _old). While preventing destructive policy shifts, this inadvertently induces a vanishing gradient for exploration. Tokens drifting outside the trust region – representing valuable exploratory steps – receive zero updates, leading to entropy collapse and premature convergence Williams and Peng (1991); Eysenbach and Levine (2021). Recently, “soft clipping” approaches attempt to preserve gradients for out-of-bound tokens Chen et al. (2025); Su et al. (2025a, b). However, we identify a fundamental challenge: these methods predominantly operate on log-probability gradient (∇θlogπθ _θ _θ). As probability approaches zero, the gradient weight of log-probability-based methods grows divergently. This causes catastrophic instability at the left boundary (where the IS ratio is extremely low), disproportionately penalizing tokens with vanishing probabilities. Accordingly, we propose a paradigm shift: establishing probability gradient (∇θπθ _θ _θ) as the primary optimization primitive over log-probability gradient (∇θlogπθ _θ _θ). By analyzing the gradient landscape in probability space, we derive criteria for optimal weighting: (1) sustain exploration for clipped tokens, (2) ensure stability via convergent weights at boundaries, and (3) maximize alignment with the unbiased policy gradient. In this paper, we propose Decoupled Gradient Policy Optimization (DGPO) guided by these principles. DGPO replaces “hard clipping” with a decoupled decay mechanism applied to the probability gradient weight. It applies a polynomial decay to tokens with low IS ratios (left boundary) for stability, and a reciprocal radical decay to tokens with high IS ratios (right boundary) to foster exploration. This mathematically guarantees gradient continuity and prevents the weight divergence seen in prior methods. Our contributions are summarized as: • We introduce a novel perspective, establishing the gradient of probability, rather than log-probability, as the superior optimization primitive in LLMs, with two key insights: the inherent alignment of RL objectives and the geometric symmetry of probability space, which facilitates stable gradient design. • We propose DGPO, which leverages a decoupled adaptive decay mechanism to reconcile the exploration-stability conflict. Crucially, this design preserves gradients for clipped tokens while rigorously preventing weight divergence. • Comprehensive experiments against competitive baselines across mathematical reasoning benchmarks demonstrate the effectiveness of DGPO. Further results on diverse models scales confirm its scalability and robustness. 2 Related Works 2.1 RLVR in LLMs RLVR utilizes deterministic signals (e.g., correct answers) rather than learned reward models to enhance LLM reasoning Uesato et al. (2022). Recent advancements like DeepSeek-Math Shao et al. (2024) popularized GRPO, which efficiently normalizes rewards within a sampled group, eliminating the need for a critic. However, GRPO inherits the clipping-induced exploration limitations of PPO. 2.2 Importance Sampling and Clipping IS enables off-policy training by correcting distribution shifts via the ratio πθ/πθold _θ/ _ _old Precup et al. (2000); Schulman et al. (2015). To prevent variance explosion, PPO Schulman et al. (2017) clips this ratio within [1−ε,1+ε][1- ,1+ ]. This “hard clipping” mechanism zeros out gradients for outlier tokens, prioritizing exploitation while neglecting low-probability tokens essential for exploration, causing rapid entropy decay O’Donoghue et al. (2016); Yu et al. (2025). To mitigate exploration losses, several methods dynamically adjust clipping bounds Yu et al. (2025); Yang et al. (2025a), allowing more updates for specific tokens. However, they still rely on hard boundaries, inevitably discarding gradient information for tokens beyond the adjusted thresholds. 2.3 Soft Clipping Policy Optimization More recent approaches replace “hard clipping” with soft schemes. For instance, CISPO Chen et al. (2025) combines “soft clipping” with soft dual clip Ye et al. (2020), while GPPO Su et al. (2025a) retains a constant log-probability gradient weight for out-of-bound tokens. Crucially, both CISPO and GPPO suffer from left boundary instability: as πθ→0 _θ→ 0, the gradient grows indefinitely. Without proper decay mechanisms, this results in divergent updates that destabilize training. CE-GPPO Su et al. (2025b) attempts to scale boundary gradients via hyperparameters but fails to resolve the underlying divergence. Distinctly, ASPO Wang et al. (2025) proposes a reversed ratio to balance updates. Compared with these approaches, DGPO redefines the optimization target in probability space to ensure theoretical continuity and stability. 3 Preliminary Problem Definition. In this paper, we focus on RLVR settings. Given a query q from dataset D, a policy πθ _θ generates a response o. The rule-based reward function r(q,o)∈−1,1r(q,o)∈\-1,1\ evaluates the correctness of each response. Following GRPO Shao et al. (2024), for each query q, a group of G outputs oii=1G\o_i\_i=1^G is sampled from the old policy πθold _ _old. The advantage A^i A_i is computed by normalizing the group-level rewards: A^i=(r(q,oi)−μR)/σR A_i=(r(q,o_i)- _R)/ _R, where μR _R and σR _R denote the mean and standard deviation of the rewards within the group. Unified Gradient Formulation. To align off-policy training with reward maximization while constraining policy shifts for stability, most RLVR algorithms employ IS and clipping mechanisms. Thus, we formulate a unified gradient estimator that encompasses these methods. Let wi,t(θ)=πθ(oi,t|q,oi,<t)πθold(oi,t|q,oi,<t)w_i,t(θ)= _θ(o_i,t|q,o_i,<t) _ _old(o_i,t|q,o_i,<t) denotes the token-level IS ratio, and the gradient is then expressed as: ∇θ(θ)=q∼,oii=1G∼πθold(⋅|q)1∑i=1G|oi| _θJ(θ)=E_q ,\o_i\_i=1^G _ _old(·|q) 1 _i=1^G|o_i| ∑i=1G∑t=1|oi|ℱi,t(θ)A^i∇θlogπθ(oi,t|q,oi,<t). _i=1^G _t=1^|o_i|F_i,t(θ) A_i _θ _θ(o_i,t|q,o_i,<t). (1) The behavior of ℱi,t(θ)F_i,t(θ) typically depends on five distinct regions, which are simultaneously defined by the clipping boundaries (1−εlow1- _low and 1+εhigh1+ _high) and the sign of the advantage A^i A_i. We formally define these regions in Table 1. Table 1: Definition of clipping regions based on IS ratio wi,t(θ)w_i,t(θ) and advantage A^i A_i. Abbrev. denotes the abbreviation of five cases under different conditions. Case Abbrev. Condition Left Boundary LN wi,t(θ)<1−εloww_i,t(θ)<1- _low (Low ratio, Neg. adv.) ∧A^i<0 \ A_i<0 Right Boundary HP wi,t(θ)>1+εhighw_i,t(θ)>1+ _high (High ratio, Pos. adv.) ∧A^i>0 \ A_i>0 Reverse Left Boundary LP wi,t(θ)<1−εloww_i,t(θ)<1- _low (Low ratio, Pos. adv.) ∧A^i>0 \ A_i>0 Reverse Right Boundary HN wi,t(θ)>1+εhighw_i,t(θ)>1+ _high (High ratio, Neg. adv.) ∧A^i<0 \ A_i<0 In-Boundary M ¬LN∧¬HP (Medium ratio) ∧¬LP∧¬HN Instantiations of Clipping Strategies. Various existing methods can be interpreted as specific instantiations of ℱi,t(θ)F_i,t(θ). Standard PPO Schulman et al. (2017) and GRPO Shao et al. (2024) apply “hard clipping” to penalize excessive updates: ℱi,tGRPO(θ)=0,if LN∨HP,wi,t(θ),otherwise. ^GRPO_i,t(θ)= cases0,&if LN ,\\ w_i,t(θ),&otherwise. cases (2) CISPO Chen et al. (2025) preserves exploration for these tokens by the following gradient weight: ℱi,tCISPO(θ)=1−εlow,if LN∨LP,1+εhigh,if HP∨HN,wi,t(θ),otherwise. ^CISPO_i,t(θ)= cases1- _low,&if LN ,\\ 1+ _high,&if HP ,\\ w_i,t(θ),&otherwise. cases (3) This method inherently employs soft dual clip Ye et al. (2020), which clips the loss values for LP and HN (reverse cases) while preserving gradients. Subsequently, GPPO Su et al. (2025a) introduces a gradient weight more aligned with PPO, focusing solely on LN and HP cases: ℱi,tGPPO(θ)=1−εlow,if LN,1+εhigh,if HP,wi,t(θ),otherwise. ^GPPO_i,t(θ)= cases1- _low,&if LN,\\ 1+ _high,&if HP,\\ w_i,t(θ),&otherwise. cases (4) CE-GPPO Su et al. (2025b) further refines GPPO by introducing hyperparameters (β1 _1 and β2 _2) to control gradient scaling at both boundaries: ℱi,tCE(θ)=β1(1−εlow),if LN,β2(1+εhigh),if HP,wi,t(θ),otherwise. ^CE_i,t(θ)= cases _1(1- _low),&if LN,\\ _2(1+ _high),&if HP,\\ w_i,t(θ),&otherwise. cases (5) To address the imbalance in updating positive-advantage tokens, ASPO Wang et al. (2025) reverses gradient weight and adopts soft dual clip: ℱi,tASPO(θ)=0,if LN∨HP,1−εlow′,if LP′,1+εhigh′,if HN′,1wi,t(θ),otherwise. ^ASPO_i,t(θ)= cases0,&if LN ,\\ 1- _low ,&if LP ,\\ 1+ _high ,&if HN ,\\ 1w_i,t(θ),&otherwise. cases (6) The values of 1−εlow′1- _low and 1+εhigh′1+ _high for cases LP′ and HN′ in ASPO may be more extreme. 4 Methodology Figure 2: Comparative analysis of gradient dynamics. We systematically contrast DGPO with the standard GRPO, prior “soft clipping” enhancements (CISPO, GPPO, and CE-GPPO), and importance sampling improvements (ASPO). The visualization highlights the theoretical properties regarding the exploration capability of clipped tokens and the alignment with the true policy gradient, demonstrating DGPO’s superior stability and gradient consistency. 4.1 Shifting Focus: From π to π Classical policy gradient methods Williams (1992) typically employ the log-derivative trick to reformulate the gradient into an expectation, which establishes ∇θlogπθ _θ _θ as the canonical term. This has led to a pervasive focus on log-probability gradient in subsequent research. However, we rethink that probability gradient constitutes a superior analytical and design target for two primary reasons: (1) probability, rather than log-probability, acts as the superior optimization primitive in LLM training, and (2) probability exhibits superior geometric symmetry within its value range. Probability as the Optimization Primitive. To substantiate this claim, we contrast the token-level objectives of Supervised Fine-Tuning (SFT) and RL. SFT maximizes the mean of expert token log-probabilities, with the gradient estimator as: ∇θSFT(θ) _θJ_SFT(θ) =∑t=1|o|∇θlogπθ(ot|q,o<t). =E_D _t=1^|o| _θ _θ(o_t|q,o_<t). (7) Conversely, RL (under binary advantage assumptions) equivalently maximizes the mean of expert token probabilities (derivation provided in Appendix A.1), yielding the estimator: ∇θRL(θ)=∑t=1|o|∇θπθ(ot∣q,o<t). _θJ_RL(θ)=E_D _t=1^|o| _θ _θ(o_t q,o_<t). (8) Equations (7) and (8) reveal a fundamental distinction: SFT operates on log-probability, whereas RL inherently operates on probability. This distinction renders the SFT objective mathematically a lower bound of the RL objective Qin and Springenberg (2025), explaining the empirical performance ceiling often observed in SFT compared with RL Wu et al. (2025). Consequently, we identify probability as the superior optimization primitive. Since the probability aligns more closely with the core requirements of LLM training, its gradient should be prioritized in algorithm design. Geometric Symmetry and Boundedness. In LLM training, token probabilities reside in the symmetric and bounded interval (0,1)(0,1). This boundedness facilitates the analysis of gradient impacts on probability values and enables the design of symmetric gradient mechanisms. In contrast, log-probabilities span the asymmetric and unbounded interval (−∞,0)(-∞,0), complicating the gradient design. Table 2: Comparative analysis of gradient bias. We evaluate the bias magnitudes of DGPO and other baselines relative to the true policy gradient under different boundary conditions. Bias Condition Magnitude Comparison In-Boundary Bias 0=BiasDGPOM=BiasGRPOM=BiasCISPOM=BiasGPPOM=BiasCEM≤BiasASPOM0= [rgb]0.75390625,0,0Bias_DGPO^M=Bias_GRPO^M=Bias_CISPO^M=Bias_GPPO^M=Bias_CE^M _ASPO^M Left-Boundary Bias 0<BiasDGPOLN<BiasCISPOLN=BiasGPPOLN<BiasCELN<BiasGRPOLN=BiasASPOLN,n=10<BiasCISPOLN=BiasGPPOLN<BiasDGPOLN<BiasCELN<BiasGRPOLN=BiasASPOLN,n>1 cases0< [rgb]0.75390625,0,0Bias_DGPO^LN<Bias_CISPO^LN=Bias_GPPO^LN<Bias_CE^LN<Bias_GRPO^LN=Bias_ASPO^LN,&n=1\\ 0<Bias_CISPO^LN=Bias_GPPO^LN< [rgb]0.75390625,0,0Bias_DGPO^LN<Bias_CE^LN<Bias_GRPO^LN=Bias_ASPO^LN,&n>1 cases Right-Boundary Bias 0<BiasDGPOHP≤BiasCISPOHP=BiasGPPOHP=BiasCEHP<BiasGRPOHP=BiasASPOHP0< [rgb]0.75390625,0,0Bias_DGPO^HP _CISPO^HP=Bias_GPPO^HP=Bias_CE^HP<Bias_GRPO^HP=Bias_ASPO^HP Reverse Left-Boundary Bias 0=BiasDGPOLP=BiasGRPOLP=BiasGPPOLP=BiasCELP<BiasCISPOLP=BiasASPOLP0= [rgb]0.75390625,0,0Bias_DGPO^LP=Bias_GRPO^LP=Bias_GPPO^LP=Bias_CE^LP<Bias_CISPO^LP=Bias_ASPO^LP Reverse Right-Boundary Bias 0=BiasDGPOHN=BiasGRPOHN=BiasGPPOHN=BiasCEHN<BiasCISPOHN=BiasASPOHN0= [rgb]0.75390625,0,0Bias_DGPO^HN=Bias_GRPO^HN=Bias_GPPO^HN=Bias_CE^HN<Bias_CISPO^HN=Bias_ASPO^HN 4.2 Decoupled Gradient Policy Optimization Instability in Soft Clipping. While clipped tokens often contain critical information essential for model performance Liu et al. (2025); Su et al. (2025a), preserving their gradients requires a principled approach. Prior “soft clipping” works Chen et al. (2025); Su et al. (2025a) maintain constant log-probability gradient weights in LP and HN cases. However, the gradient weights are present asymmetrically at the boundaries, contradicting the symmetric nature of probability values. Thus, we define probability gradient weight as i,t(θ)=∇θ(θ)A^i∇θπθ(oi,t|q,oi,<t)W_i,t(θ)= _θJ(θ) A_i _θ _θ(o_i,t|q,o_i,<t). As illustrated in Figure 2, while the right-boundary gradient weight decreases convergently (promoting stability), the left-boundary weight grows divergently. This causes negative-advantage tokens to shrink disproportionately—the smaller the probability, the larger the update magnitude—leading to training instability Su et al. (2025b). Although CE-GPPO attempts to mitigate this via hyperparameters, it does not resolve the divergent growth, leaving the risk of collapse. DGPO Formulation. To address these limitations, we propose DGPO in this paper, designed to: (1) preserve gradient for clipped tokens, (2) stabilize exploration via adaptive gradient decay, and (3) minimize policy gradient bias for alignment. We define the weighting function i,tDGPO(θ)W^DGPO_i,t(θ) based on the regions defined in Table 1: i,tDGPO(θ)=Cleft⋅sgn[πθ(oi,t|q,oi,<t)],if LN,Cright⋅sg−1m[πθ(oi,t|q,oi,<t)],if HP,1πθold,otherwise.W^DGPO_i,t(θ)= casesC_left· sg^n[ _θ(o_i,t|q,o_i,<t)],&if LN,\\ C_right· sg^- 1m[ _θ(o_i,t|q,o_i,<t)],&if HP,\\ 1 _ _old,&otherwise. cases (9) Then, the objective function is formulated as: DGPO(θ)=q∼,oii=1G∼πθold(⋅|q)1∑i=1G|oi| _DGPO(θ)=E_q ,\o_i\_i=1^G _ _old(·|q) 1 _i=1^G|o_i| ∑i=1G∑t=1|oi|i,tDGPO(θ)A^iπθ(oi,t|q,oi,<t), _i=1^G _t=1^|o_i|W^DGPO_i,t(θ) A_i _θ(o_i,t|q,o_i,<t), (10) where n,m∈ℤ+n,m ^+ are hyperparameters controlling the decay rate, and sg[⋅]sg[·] denotes the stop-gradient operator. Cleft=(1−εlow)−nπθold−(n+1)(oi,t|q,oi,<t)C_left=(1- _low)^-nπ^-(n+1)_ _old(o_i,t|q,o_i,<t) and Cright=(1+εhigh)1mπθold1m−1(oi,t|q,oi,<t)C_right=(1+ _high) 1mπ 1m-1_ _old(o_i,t|q,o_i,<t) are constants ensuring continuity of gradient weights (derivation provided in Appendix A.2). 4.3 Theoretical Analysis and Advantages Gradient Preserving and Exploration. Following prior gradient-preserving methods, DGPO adopts “soft clipping” to maintain gradient of clipped tokens. By retaining adaptively reduced gradients in both LN (Low Ratio, Negative Advantage) and HP (High Ratio, Positive Advantage) scenarios, DGPO enables sustained exploration for clipped tokens, enhancing the model’s exploration potential and raising its convergence upper bound. Symmetric Stability Control. To ensure training stability while adhering to the principle of trust region optimization (limiting divergence between πθ _θ and πθold _ _old), gradient weights must decrease as the policy deviates. DGPO achieves this via a decoupled design: (1) Left Boundary: A positive integer power function of πθ _θ, ensuring weights decay as probability decreases. (2) Right Boundary: A reciprocal radical power function of πθ _θ, ensuring weights decay as probability increases. Furthermore, DGPO manages entropy dynamics by controlling the “openness” of gradient weights. Since LN tokens drive entropy reduction and HP tokens drive entropy increase Cui et al. (2025); Su et al. (2025b), the default symmetric setting (m=1m=1 and n=1n=1) may lead to rapid entropy decay. We balance this by adjusting n (reducing left-boundary openness) and m (increasing right-boundary openness). Additionally, multiplying gradient weights by CleftC_left and CrightC_right ensures continuity at the boundaries, ensuring a smooth transition between stable updates and decelerated exploration. Table 3: Comparison results of different methods on various benchmarks. Avg@32 (%) and Pass@32 (%) are abbreviated as A@32 and P@32. The best results are bold, and the second-best results are underlined, respectively. Method AIME24 AIME25 AMC23 MATH500 Minerva Olympiad Avg. A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 DeepSeek-R1-Distill-Qwen-1.5B GRPO 33.2 71.8 27.7 49.9 79.5 94.8 77.6 90.8 26.1 48.8 46.3 64.7 48.4 70.1 CISPO 34.8 69.1 25.8 53.3 76.9 94.9 76.8 91.8 26.5 54.2 45.8 65.8 47.8 71.5 GPPO 29.6 60.5 23.5 51.9 73.5 94.1 76.3 89.1 26.6 50.0 43.9 64.2 45.6 68.3 CE-GPPO 35.1 70.2 27.7 55.1 82.5 95.0 76.7 90.2 27.8 50.5 45.6 63.1 49.2 70.7 ASPO 36.4 73.2 28.3 51.5 83.1 94.7 74.6 90.5 26.0 49.8 44.9 63.7 48.9 70.6 Ours 43.3 79.3 32.8 56.1 86.0 95.0 77.9 91.0 28.2 50.4 48.0 66.4 52.7 73.0 DeepSeek-R1-Distill-Qwen-7B GRPO 48.2 82.5 37.4 60.5 88.1 96.6 84.8 92.4 37.4 57.2 57.2 73.9 58.9 77.2 CISPO 51.6 76.6 38.2 65.4 90.6 96.6 82.1 91.6 38.7 56.5 54.3 69.9 59.3 76.1 GPPO 43.1 72.5 31.7 62.5 85.6 94.9 83.2 95.4 33.1 59.3 53.2 74.3 55.0 76.5 CE-GPPO 48.7 76.9 36.4 60.4 90.5 95.0 84.3 93.3 39.0 55.4 54.9 72.5 59.0 75.6 ASPO 51.8 79.6 37.1 54.1 90.0 97.2 83.8 94.9 37.0 59.2 54.1 72.8 59.0 76.3 Ours 55.5 81.9 43.1 68.0 90.6 96.6 85.4 92.0 39.8 56.7 57.7 72.0 62.0 77.9 Minimal Bias Estimation. A critical challenge of existing RL algorithms is their deviation from the distribution correction principle of IS, resulting in significant bias relative to the true policy gradient. DGPO minimizes this bias to establish a stronger theoretical guarantee, which is the key factor for higher convergence bounds of model performance. Table 2 presents a comparative analysis of bias (proofs provided in Appendix A.3). Notably, while CISPO and GPPO theoretically achieve minimal left-boundary bias when n>1n>1, they suffer from divergent gradient weights, constituting the root cause of training collapse. DGPO uniquely balances these objectives: it ensures gradient continuity and adaptive convergence while achieving minimal bias at n=1n=1, and maintaining a constant bias ratio relative to CISPO/GPPO for n>1n>1, thus offering a robust trade-off between stability and theoretical correctness. 5 Experiments Figure 3: Comprehensive comparison of training dynamics and performance. Top row: DeepSeek-R1-Distill-Qwen-1.5B results. Bottom row: DeepSeek-R1-Distill-Qwen-7B results. Columns (L-R): Pass@K on AIME25, Avg@32 on AIME24/25, policy entropy, followed by hyperparameter analysis for Avg@32 and entropy. 5.1 Experimental Setup Configurations. For all experiments, we employ DeepSeek-R1-Distill-Qwen Guo et al. (2025) with various scales as backbone models to validate the effectiveness of DGPO. The open-source DAPO-Math-17K dataset Yu et al. (2025) is utilized for model training. During training, rule-based rewards are computed with math_verify Kydlíček (2025), while a conservative method prime_math is used for evaluation Lightman et al. (2023a, b). Moreover, to ensure consistent convergence dynamics across different model scales, we calibrate the learning rate to maintain a constant total gradient variance (derivation provided in Appendix A.4). Accordingly, the learning rates are set as follows: 1.0×10−61.0× 10^-6 for 1.5B, 4.63×10−74.63× 10^-7 for 7B, and 3.27×10−73.27× 10^-7 for 14B. We set the batch size to 512 and the mini-batch size to 32. This configuration enables 16 off-policy updates per IS step, sufficiently amplifying the influence of boundary tokens. εlow _low and εhigh _high are fixed to 0.2. Detailed configurations are available in Appendix B. Benchmarks and Metrics. We assess the generalization of reasoning capabilities on widely used mathematical benchmarks, i.e., AIME24 MAA (2024), AIME25 MAA (2025), AMC23 MAA (2023), MATH500 Hendrycks et al. (2021), Minerva Lewkowycz et al. (2022), and OlympiadBench He et al. (2024). More details about benchmarks are described in Appendix B. We report Avg@32 to measure the expected performance and Pass@32 to gauge the potential capability. Baselines. We benchmark DGPO against standard GRPO and variants employing distinct gradient weighting strategies, including “soft clipping” (CISPO and GPPO), scaled “soft clipping” (CE-GPPO, adopting the recommended β1=0.75 _1=0.75 and β2=1 _2=1), and reverse gradient weight (ASPO). 5.2 Main Results Overall Performance. Table 3 presents the performance comparison across 1.5B and 7B scales. DGPO demonstrates superior performance across the majority of benchmarks on both scales. On the 1.5B model, DGPO surpasses the vanilla GRPO by +4.3% and the best baseline (CE-GPPO) by +3.5% in average Avg@32. On the 7B model, the improvement is also significant, with DGPO outperforming GRPO by +3.1% and CISPO by +2.7%. As shown in Figure 3(a,g), DGPO consistently maintains higher Pass@K scores on AIME2025 compared with all baselines, demonstrating superior potential capability. Detailed results of Pass@K (K=1,…,32K=1,…,32) on AIME24 and AIME25 across all model scales are provided in Appendix C.3. Training Dynamics. Figure 3(b,c,h,i) illustrates the training dynamics of Avg@32 on AIME2024 and AIME2025. DGPO consistently outperforms all baselines in the mid-to-late training stages. Notably, algorithms with divergent gradient weights at the left boundary (CISPO, GPPO, and CE-GPPO) succumb to training collapse, and algorithms with larger policy gradient bias (GRPO and ASPO) exhibit suboptimal convergence. Figure 3(d,j) depicts the entropy dynamics. GRPO exhibits an entropy drop early in the training process, indicating premature exploitation that limits the achievable performance ceiling. Conversely, ASPO maintains excessively high entropy, reflecting over-exploration and insufficient exploitation, while CISPO, GPPO, and CE-GPPO show unstable entropy patterns leading to eventual collapse. DGPO demonstrates a moderate and controlled entropy reduction, signifying an optimal balance between exploration and exploitation. 5.3 Hyperparameter Analysis Figure 4: Comparison of gradient weight distributions (Prob: Probability, Grad: Gradient). (a) Overall distribution. (b) Detailed scatter plots of grad weight vs. IS ratio (top row) and prob (bottom row) across three methods: GRPO (left), GPPO (middle), and DGPO (right). Points are colored by advantage. (c) Boundary distribution analysis. We investigate the impact of hyperparameters n and m by expanding from the baseline (n=1n=1, m=1m=1) to four configurations: (1,1)(1,1), (1,2)(1,2), (2,1)(2,1), and (2,2)(2,2). Figure 3(e,k) shows their performance on AIME25 across 1.5B and 7B models. Robustness and Patterns. All DGPO configurations outperform GRPO (with the exception of n=2n=2 and m=2m=2 on the 7B model), demonstrating our algorithm’s robustness. However, the optimal configuration varies by scale: (2,2)(2,2) for 1.5B and (1,2)(1,2) for 7B. Analyzing Figure 3(e,f,k,l), which reveals a consistent pattern: increasing n or m generally yields: (1) improved performance, (2) elevated overall entropy levels, but (3) reduced entropy stability. Notably, significant entropy volatility is observed only in the 7B model under the (2,2)(2,2) setting that negates the performance benefits. Tuning Guideline. Based on these observations, we propose a heuristic for hyperparameter tuning: Enhance exploration by increasing n and m as long as entropy remains stable. Upon observing instability, revert to the preceding stable configuration. Empirically, we recommend n=1n=1 and m=2m=2 as a robust and conservative baseline configuration. Table 4: Scalability analysis of average Avg@32 (A) and Pass@32 (P) across 1.5B, 7B, and 14B models. Method 1.5B 7B 14B111The performance fluctuation is attributed to the base model difference: 1.5B/7B models are derived from Qwen-2.5-Math, whereas the 14B model is derived from Qwen-2.5-Base. (A / P) (A / P) (A / P) GRPO 48.4 / 70.1 58.9 / 77.2 53.6 / 67.4 DGPO 52.7 / 73.0 62.0 / 77.9 56.7 / 70.4 Improvement +4.3 / +2.9 +3.1 / +0.7 +3.1 / +3.0 Scaling to Larger Models. The instability of (2,2)(2,2) on 7B (absent in 1.5B) suggests that larger models exhibit higher intrinsic entropy volatility, necessitating more conservative hyperparameters. To validate this pattern, we apply the optimal 7B configuration (n=1,m=2n=1,m=2) to the 14B model. As summarized in Table 4, DGPO consistently outperforms GRPO across all model scales in both Avg@32 and Pass@32, which confirms that the benefits of our decoupled gradient mechanism effectively transfer to larger models. Furthermore, the training dynamics and entropy variations of the 14B model (shown in Appendix C.1) exhibit stable convergence patterns similar to 7B under DGPO. 5.4 Mechanistic Analysis via Visualization Mechanism of Stability (Left Boundary). Why does DGPO prevent collapse? Figure 4(a) visualizes the joint distribution of probability and IS ratios in the final mini-batch. Crucially, tokens at both boundaries are predominantly low-probability tokens. Figure 4(b) further details the relationship between probability, IS ratio, advantage, and relative gradient weight (normalized by πθold _ _old probability) for GRPO (zero weight), GPPO (divergent weight), and DGPO (convergent weight). As illustrated, GRPO’s zero gradients at the left boundary lead to the narrowest ratio distribution and insufficient exploration. Conversely, GPPO’s divergent weights induce an excessively broad distribution, eventually precipitating training collapse due to instability (Figure 3(b,c,h,i)) Yang et al. (2025b). In contrast, DGPO maintains convergent relative weights, resulting in a ratio distribution only slightly wider than GRPO that successfully balances stability with effective exploration. Mechanism of Improvement (Right Boundary). Why does DGPO perform better? Compared with GRPO (where right-boundary tokens have zero gradient and the narrowest ratio distribution), DGPO maintains gradients to foster exploration. Compared with GPPO (which uses a reciprocal standard weight equivalent to DGPO’s m=1m=1), DGPO with m=2m=2 employs a reciprocal radical weight. This design induces the widest ratio distribution on the right boundary (Figure 4(c)), significantly enhancing performance. This finding aligns with prior research suggesting that increasing the contribution of positive samples improves performance Yang et al. (2025b); Xi et al. (2025), and is consistent with our hyperparameter analysis in Section 5.3 where increasing m boosts results. 6 Conclusion In this paper, we revisit the fundamental optimization primitive in RLVR and establish probability—rather than log-probability—as the essential alignment target. We argue that the prevalent focus on log-probability gradients in prior “soft clipping” methods constitutes a misalignment with the true RL objective, leading to divergent gradient weights at boundaries and consequent training instability. To address this critical issue, we propose DGPO, which directly optimizes probability gradients via a decoupled decay mechanism based on IS ratios. This approach effectively resolves the inherent conflict between training stability and gradient preservation. Extensive experiments across the DeepSeek-R1-Distill-Qwen series models with multiple scales (1.5B, 7B, and 14B) demonstrate that DGPO consistently outperforms competitive baselines on various mathematical reasoning benchmarks, validating that aligning with the probability objective is crucial for unlocking the full potential of LLM reasoning capabilities through RL. 7 Limitations We discuss two practical limitations of our work. (1) Domain Specificity: In line with most RLVR studies, our experiments are primarily conducted on mathematical reasoning datasets (e.g., AIME, MATH500) where verifiable rewards are readily available. While we believe the principles of probability gradients are generalizable, the efficacy of DGPO in domains with sparse or subjective rewards (e.g., creative writing) remains to be verified. (2) Computational Constraints: Due to limited computational resources, our scaling laws analysis is restricted to models up to 14B parameters. Although we observed consistent performance gains and predictable hyperparameter patterns from 1.5B to 14B, validating these findings on significantly larger foundation models (e.g., 70B+) would provide further insights into the method’s scalability. 8 Ethical Considerations We have carefully considered the ethical implications of our research and provide the following statements: (1) Compliance and Transparency: Throughout this study, we have strictly followed established ethical guidelines. Our findings are reported honestly, and we provide comprehensive theoretical derivations to ensure transparency. (2) Data Usage: The datasets employed in our experiments (e.g., DAPO-Math-17k) originate from publicly available sources. No private, sensitive, or confidential user information was used at any stage of our research. (3) Reproducibility: We offer detailed descriptions of the hyperparameter configurations, including the specific tuning guidelines for n and m, to ensure the reproducibility of our results. (4) Open Source: In the interest of openness and to facilitate future research in the RLVR community, we have made our code available anonymously and will fully open-source it upon the acceptance of this paper. References A. Chen, A. Li, B. Gong, B. Jiang, B. Fei, B. Yang, B. Shan, C. Yu, C. Wang, C. Zhu, et al. (2025) MiniMax-m1: scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585. Cited by: §1, §2.3, §3, §4.2. G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. (2025) The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §4.3. B. Eysenbach and S. Levine (2021) Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257. Cited by: §1. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §5.1. C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. (2024) Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008. Cited by: §5.1. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §5.1. H. Kydlíček (2025) Math-Verify: Math Verification Library External Links: Link Cited by: §5.1. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. (2022) Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, p. 3843–3857. Cited by: §5.1. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023a) Let’s verify step by step. arXiv preprint arXiv:2305.20050. Cited by: §5.1. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023b) Let’s verify step by step. In The Twelfth International Conference on Learning Representations, Cited by: §1, §5.1. W. Liu, H. Wu, Y. Kuang, X. Han, T. Zhong, J. Feng, and W. Lu (2026) Automated optimization modeling via a localizable error-driven perspective. arXiv preprint arXiv:2602.11164. Cited by: §1. Z. Liu, M. Liu, S. Wen, M. Cai, B. Cui, C. He, and W. Zhang (2025) From uniform to heterogeneous: tailoring policy optimization to every token’s nature. arXiv preprint arXiv:2509.16591. Cited by: §4.2. H. Luo, H. Feng, Q. Sun, C. Xu, K. Zheng, Y. Wang, T. Yang, H. Hu, Y. Tang, and D. Wang (2025) AgentMath: empowering mathematical reasoning for large language models via tool-augmented agent. arXiv preprint arXiv:2512.20745. Cited by: §1. MAA (2023) American mathematics competitions - amc. Note: Accessed: 2023 External Links: Link Cited by: §5.1. MAA (2024) American invitational mathematics examination - aime. Note: Accessed: 2024 External Links: Link Cited by: §5.1. MAA (2025) American invitational mathematics examination - aime. Note: Accessed: 2025 External Links: Link Cited by: §5.1. B. O’Donoghue, R. Munos, K. Kavukcuoglu, and V. Mnih (2016) Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626. Cited by: §2.2. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1. D. Precup, R. S. Sutton, and S. Singh (2000) Eligibility traces for off-policy policy evaluation. Cited by: §2.2. C. Qin and J. T. Springenberg (2025) Supervised fine tuning on curated data is reinforcement learning (and can be improved). arXiv preprint arXiv:2507.12856. Cited by: §4.1. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International conference on machine learning, p. 1889–1897. Cited by: §1, §2.2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.2, §3. Z. Shao, L. Shen, H. Li, Y. Shan, H. Qu, Y. Wang, and S. Chen (2025a) Narrative player: reviving data narratives with visuals. IEEE Transactions on Visualization and Computer Graphics 31 (10), p. 6781–6795. Cited by: §1. Z. Shao, S. Yuan, L. Gao, Y. He, D. Yang, and S. Chen (2025b) Unlocking scientific concepts: how effective are llm-generated analogies for student understanding and classroom practice?. In Proceedings of the 2025 CHI conference on human factors in computing systems, p. 1–19. Cited by: §1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1, §2.1, §3, §3. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §B.1. Z. Su, L. Pan, X. Bai, D. Liu, G. Dong, J. Huang, W. Hu, F. Zhang, K. Gai, and G. Zhou (2025a) Klear-reasoner: advancing reasoning capability via gradient-preserving clipping policy optimization. arXiv preprint arXiv:2508.07629. Cited by: §1, §2.3, §3, §4.2. Z. Su, L. Pan, M. Lv, Y. Li, W. Hu, F. Zhang, K. Gai, and G. Zhou (2025b) CE-gppo: coordinating entropy via gradient-preserving clipping policy optimization in reinforcement learning. arXiv preprint arXiv:2509.20712. Cited by: §1, §2.3, §3, §4.2, §4.3. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins (2022) Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. Cited by: §2.1. J. Wang, R. Liu, L. Lin, W. Hu, X. Li, F. Zhang, G. Zhou, and K. Gai (2025) Aspo: asymmetric importance sampling policy optimization. arXiv preprint arXiv:2510.06062. Cited by: §2.3, §3. R. J. Williams and J. Peng (1991) Function optimization using connectionist reinforcement learning algorithms. Connection Science 3 (3), p. 241–268. Cited by: §1. R. J. Williams (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 (3), p. 229–256. Cited by: §4.1. Y. Wu, Y. Zhou, Z. Ziheng, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, and X. Yang (2025) On the generalization of sft: a reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629. Cited by: §4.1. Z. Xi, X. Guo, Y. Nan, E. Zhou, J. Shen, W. Chen, J. Liu, J. Huang, Z. Zhang, H. Guo, et al. (2025) BAPO: stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping. arXiv preprint arXiv:2510.18927. Cited by: §5.4. S. Yang, C. Dou, P. Guo, K. Lu, Q. Ju, F. Deng, and R. Xin (2025a) Dcpo: dynamic clipping policy optimization. arXiv preprint arXiv:2509.02333. Cited by: §2.2. Z. Yang, X. Luo, Z. Wang, D. Han, Z. He, D. Li, and Y. Xu (2025b) Do not let low-probability tokens over-dominate in rl for llms. arXiv preprint arXiv:2505.12929. Cited by: §5.4, §5.4. D. Ye, Z. Liu, M. Sun, B. Shi, P. Zhao, H. Wu, H. Yu, S. Yang, X. Wu, Q. Guo, et al. (2020) Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 6672–6679. Cited by: §2.3, §3. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §2.2, §5.1. Appendix A Derivation and Proof A.1 Derivation of RL Estimator in SFT Form The token-level objective of RL is to maximize the arithmetic mean expectation of the estimated advantage, formulated as follows: RL(θ)=q∼,o∼πθ(⋅|q)[∑t=1|o|At] _ RL(θ)=E_q ,\ o _θ(·|q) [ _t=1^|o|A_t ] (11) Assume that πe(⋅|q) _e(·|q) represents the ideal expert distribution, which satisfies two key properties: (1) its output follows a one-hot distribution; (2) every sampled token oto_t achieves the maximum advantage AtA_t. By applying Importance Sampling, we obtain: RL(θ)= _ RL(θ)= q∼,o∼πe(⋅|q) _q ,\ o _e(·|q) [∑t=1|o|πθ(ot∣q,o<t)πe(ot∣q,o<t)⋅At] [ _t=1^|o| _θ(o_t q,o_<t) _e(o_t q,o_<t)· A_t ] (12) Since πe(⋅|q) _e(·|q) is a one-hot distribution, the probability πe(ot∣q,o<t) _e(o_t q,o_<t) is always 100%, leading to: RL(θ)= _ RL(θ)= q∼,o∼πe(⋅|q) _q ,\ o _e(·|q) [∑t=1|o|πθ(ot∣q,o<t)⋅At] [ _t=1^|o| _θ(o_t q,o_<t)· A_t ] (13) Given that each action oto_t attains the highest possible AtA_t, under the binary advantage setting, AtA_t is consistently 1, resulting in: RL(θ)=(q,o)∼e[∑t=1|o|πθ(ot∣q,o<t)] _ RL(θ)=E_(q,o) _e [ _t=1^|o| _θ(o_t q,o_<t) ] (14) Consequently, the gradient is derived as: ∇θRL(θ)= _θJ_ RL(θ)= (q,o)∼e _(q,o) _e [∑t=1|o|∇θπθ(ot∣q,o<t)] [ _t=1^|o| _θ _θ(o_t q,o_<t) ] (15) A.2 Derivation of Continuity Constants In this section, we derive the constants CleftC_left and CrightC_right used in the DGPO weighting function. The primary objective is to ensure that the gradient estimator is continuous with respect to the policy probability πθ _θ at the clipping boundaries. From equation (4.2), the gradient equation of DGPO can be explicitly expressed as: ∇θDGPO(θ)=q∼,oii=1G∼πθold(⋅|q)1∑i=1G|oi| _θJ_DGPO(θ)=E_q ,\o_i\_i=1^G _ _old(·|q) 1 _i=1^G|o_i| ∑i=1G∑t=1|oi|i,tDGPO(θ)A^i∇θπθ(oi,t|q,oi,<t), _i=1^G _t=1^|o_i|W^DGPO_i,t(θ) A_i _θ _θ(o_i,t|q,o_i,<t), (16) Using the identity ∇θπθ=πθ∇θlogπθ _θ _θ= _θ _θ _θ, the effective coefficient applied to the standard score function ∇θlogπθ _θ _θ is ℱi,t=i,tDGPO⋅πθF_i,t=W^DGPO_i,t· _θ. Since πθ _θ is continuous, ensuring the continuity of i,tDGPOW^DGPO_i,t at the boundaries is sufficient to ensure the continuity of the entire gradient estimator. Let wi,t=πθπθoldw_i,t= _θ _ _old denote the importance sampling ratio. The In-Boundary (M) weight is given by: i,tM=1πθold.W^M_i,t= 1 _ _old. (17) A.2.1 Left Boundary Derivation The transition between the Left Boundary (LN) and the In-Boundary (M) region occurs when the importance sampling ratio is wi,t=1−εloww_i,t=1- _low. At this boundary, the current policy probability is: πθ=(1−εlow)πθold. _θ=(1- _low) _ _old. (18) We then equate the weighting functions for the LN and M regions at this point: i,tLN|wi,t=1−εlow ^LN_i,t |_w_i,t=1- _low =i,tM|wi,t=1−εlow =W^M_i,t |_w_i,t=1- _low Cleft⋅πθn C_left· _θ^n =1πθold = 1 _ _old Cleft⋅[(1−εlow)πθold]n C_left· [(1- _low) _ _old ]^n =1πθold. = 1 _ _old. (19) Solving for CleftC_left: Cleft C_left =1πθold⋅(1−εlow)nπθoldn = 1 _ _old·(1- _low)^n _ _old^n =1(1−εlow)nπθoldn+1. = 1(1- _low)^n _ _old^n+1. (20) Thus, Cleft=(1−εlow)−nπθold−(n+1)C_left=(1- _low)^-n _ _old^-(n+1). Table 5: Definition of bias relative to Policy Gradient Bias Type Mathematical Definition In-Boundary Bias BiasAlgoM=‖∇θAlgoM(θ)−∇θPGM(θ)‖Bias_Algo^M= \| _θJ_Algo^M(θ)- _θJ_PG^M(θ) \| Left-Boundary Bias BiasAlgoLN=‖∇θAlgoLN(θ)−∇θPGLN(θ)‖Bias_Algo^LN= \| _θJ_Algo^LN(θ)- _θJ_PG^LN(θ) \| Right-Boundary Bias BiasAlgoHP=‖∇θAlgoHP(θ)−∇θPGHP(θ)‖Bias_Algo^HP= \| _θJ_Algo^HP(θ)- _θJ_PG^HP(θ) \| Reverse Left-Boundary Bias BiasAlgoLP=‖∇θAlgoLP(θ)−∇θPGLP(θ)‖Bias_Algo^LP= \| _θJ_Algo^LP(θ)- _θJ_PG^LP(θ) \| Reverse Right-Boundary Bias BiasAlgoHN=‖∇θAlgoHN(θ)−∇θPGHN(θ)‖Bias_Algo^HN= \| _θJ_Algo^HN(θ)- _θJ_PG^HN(θ) \| A.2.2 Right Boundary Derivation Similarly, the transition between the Right Boundary (HP) and the In-Boundary (M) region occurs when wi,t=1+εhighw_i,t=1+ _high. At this boundary: πθ=(1+εhigh)πθold. _θ=(1+ _high) _ _old. (21) We similarly equate the weighting functions for the HP and M regions at this point: i,tHP|wi,t=1+εhigh ^HP_i,t |_w_i,t=1+ _high =i,tM|wi,t=1+εhigh =W^M_i,t |_w_i,t=1+ _high Cright⋅πθ−1m C_right· _θ^- 1m =1πθold = 1 _ _old Cright⋅[(1+εhigh)πθold]−1m C_right· [(1+ _high) _ _old ]^- 1m =1πθold. = 1 _ _old. (22) Solving for CrightC_right: Cright C_right =(1+εhigh)1mπθold1mπθold = (1+ _high) 1m _ _old 1m _ _old =(1+εhigh)1mπθold1m−1. =(1+ _high) 1m _ _old 1m-1. (23) This concludes the derivation of the constants. A.3 Proof of Policy Gradient Bias A.3.1 Standard Policy Gradient Estimator The gradient estimator for the standard policy gradient method is formally given by: ∇θPG(θ)= _θJ_PG(θ)= q∼,o∼πθ(⋅|q) _q ,\ o _θ(·|q) [At∇θlogπθ(ot|q,o<t)] [A_t _θ _θ(o_t|q,o_<t) ] (24) By introducing Importance Sampling with the old policy, we obtain the following form: ∇θPG(θ)=q∼,o∼πθold(⋅|q) _θJ_PG(θ)=E_q ,o _ _old(·|q) [πθ(ot|q,o<t)πθold(ot|q,o<t)At∇θlogπθ(ot|q,o<t)] [ _θ(o_t|q,o_<t) _ _old(o_t|q,o_<t)A_t _θ _θ(o_t|q,o_<t) ] (25) We adopt Equation (A.3.1) as the standard policy gradient estimator in the subsequent proofs. A.3.2 Decomposed Gradient Estimation To distinguish gradient estimators under different conditions, we define the following binary variables based on the importance sampling ratio rt(θ)=πθ(ot|q,o<t)πθold(ot|q,o<t)r_t(θ)= _θ(o_t|q,o_<t) _ _old(o_t|q,o_<t) and the advantage AtA_t. Note that while rtr_t is a function of θ, for the purpose of defining the estimator’s functional form, these regions are treated as piecewise conditions: vtLN=[rt(θ)<1−εlow∧At<0]vtHP=[rt(θ)>1+εhigh∧At>0]vtLP=[rt(θ)<1−εlow∧At>0]vtHN=[rt(θ)>1+εhigh∧At<0]vtM=1−(vtLN+vtHP+vtLP+vtHN). casesv_t^LN=I [r_t(θ)<1- _low A_t<0 ]\\ v_t^HP=I [r_t(θ)>1+ _high A_t>0 ]\\ v_t^LP=I [r_t(θ)<1- _low A_t>0 ]\\ v_t^HN=I [r_t(θ)>1+ _high A_t<0 ]\\ v_t^M=1- (v_t^LN+v_t^HP+v_t^LP+v_t^HN ) cases. (26) The policy gradient estimator is formally decomposed and expressed as a sum of five distinct terms: ∇θPG(θ)=∑X∈∇θPGX(θ), _θJ_PG(θ)= _X _θJ_PG^X(θ), (27) where ∈M, LN, LP, HP, HNX∈\M, LN, LP, HP, HN\, and the general mathematical form for each term is defined as: ∇θPGX(θ)=q∼,o∼πθold(⋅|q) _θJ_PG^X(θ)=E_q ,\,o _ _old(·|q) [vtX⋅rtAt∇θlogπθ(ot|q,o<t)]. [v_t^X· r_tA_t _θ _θ(o_t|q,o_<t) ]. (28) Similarly, gradient estimators for GRPO, CISPO, GPPO, CE-GPPO, ASPO, and DGPO can be consistently decomposed into five-term sums. The bias is defined as the magnitude (L2L_2 norm) of the difference vector between the algorithm’s gradient estimator and the standard policy gradient under identical conditions, as shown in Table 5. A.3.3 In-Boundary Bias The gradient estimates for each algorithm under in-boundary conditions are analyzed below. While most algorithms (GRPO, CISPO, GPPO, CE, DGPO) maintain an unbiased estimator locally (vtM=1v_t^M=1), ASPO introduces a non-linear regularization term specifically when the advantage is positive (At>0A_t>0). The magnitude of the bias is then computed as shown in Table 6. Table 6: In-Boundary bias among various policy optimization algorithms Algorithm In-Boundary Bias Magnitude GRPO, CISPO, BiasXM=‖∇θXM(θ)−∇θPGM(θ)‖=0 _X^M= \| _θJ_X^M(θ)- _θJ_PG^M(θ) \|=0 GPPO, CE-GPPO DGPO ASPO BiasASPOM=∥q∼,o∼πθold(⋅|q)[vtM⋅(1rt−rt)⋅At∇θlogπθ(ot|q,o<t)]∥≈∥q∼,o∼πθold(⋅|q)[vtM⋅−2(rt−1)⏟λt⋅At∇θlogπθ(ot|q,o<t)]∥≠0∗ aligned Bias_ASPO^M&= \|E_q ,\,o _ _old(·|q) [v_t^M· ( 1r_t-r_t )· A_t _θ _θ(o_t|q,o_<t) ] \|\\ &≈ \|E_q ,\,o _ _old(·|q) [v_t^M· -2(r_t-1)_ _t· A_t _θ _θ(o_t|q,o_<t) ] \|≠ 0^* aligned The relationship of in-boundary bias magnitudes is determined to be: 0=BiasDGPOM=BiasOthersM<BiasASPOM.0= [rgb]0.75390625,0,0Bias_DGPO^M=Bias_Others^M<Bias_ASPO^M. A.3.4 Left-Boundary Bias The gradient estimates for each algorithm under specific left-boundary conditions (rt<1−εlow,At<0r_t<1- _low,A_t<0) are presented in Eq. (37). Let r0=1−εlowr_0=1- _low. The magnitude of the bias is computed as shown in Table 7. Since 0<rt<r0<10<r_t<r_0<1, it follows universally that BiasDGPOLN<BiasGRPOLNBias_DGPO^LN<Bias_GRPO^LN and BiasDGPOLN<BiasASPOLNBias_DGPO^LN<Bias_ASPO^LN. However, ranking other algorithms requires a quantitative analysis of the integrals. Decoupling Assumption and Quantitative Analysis. To rigorously compare the bias magnitudes, we apply a decoupling assumption: we assume the gradient norm is locally independent of the importance sampling ratio rtr_t within the small boundary region. Let δ=q∼[∥∇θlogπθ(ot|q,o<t)∥]δ=E_q [ _θ _θ(o_t|q,o_<t) ] denote the average gradient magnitude. The explicit expressions analytically are derived as follows (integrals are over rtr_t): BiasGRPOLN _GRPO^LN =BiasASPOLN =Bias_ASPO^LN =∫0r0|0−rt|⏟Coeff. Diff⋅δ⋅krtγdrt = _0^r_0 |0-r_t|_Coeff. Diff·δ· kr_t^γ\,dr_t =kδr0γ+2γ+2 = kδ r_0^γ+2γ+2 (29) BiasCISPOLN _CISPO^LN =BiasGPPOLN =Bias_GPPO^LN =∫0r0|r0−rt|⏟Coeff. Diff⋅δ⋅krtγdrt = _0^r_0 |r_0-r_t|_Coeff. Diff·δ· kr_t^γ\,dr_t =kδr0γ+2(1γ+1−1γ+2) =kδ r_0^γ+2 ( 1γ+1- 1γ+2 ) (30) BiasCELN _CE^LN =∫0r0|β1r0−rt|⏟Coeff. Diff⋅δ⋅krtγdrt = _0^r_0 | _1r_0-r_t|_Coeff. Diff·δ· kr_t^γ\,dr_t =kδr0γ+2|β1γ+1−1γ+2| =kδ r_0^γ+2 | _1γ+1- 1γ+2 | (31) BiasDGPOLN _DGPO^LN =∫0r0|rtn+1r0n−rt|⏟Coeff. Diff⋅δ⋅krtγdrt = _0^r_0 | r_t^n+1r_0^n-r_t |_Coeff. Diff·δ· kr_t^γ\,dr_t =kδr0γ+2(1γ+2−1n+γ+2) =kδ r_0^γ+2 ( 1γ+2- 1n+γ+2 ) (32) where we utilize the property rt>rtn+1/r0nr_t>r_t^n+1/r_0^n for rt<r0r_t<r_0. In the extreme limiting case (γ→+∞γ→+∞), the resulting bias ratios are: limγ→∞BiasDGPOLNBiasCISPOLN _γ→∞ Bias_DGPO^LNBias_CISPO^LN =limγ→∞BiasDGPOLNBiasGPPOLN = _γ→∞ Bias_DGPO^LNBias_GPPO^LN =limγ→∞n(γ+1)γ+2+n=n = _γ→∞ n(γ+1)γ+2+n=n (33) limγ→∞BiasDGPOLNBiasCELN _γ→∞ Bias_DGPO^LNBias_CE^LN =limγ→∞n(γ+1)(γ+2+n)−1|(1−β1)γ+1−2β1| = _γ→∞ n(γ+1)(γ+2+n)^-1 |(1- _1)γ+1-2 _1 | =0 =0 (34) limγ→∞BiasCISPOLNBiasCELN _γ→∞ Bias_CISPO^LNBias_CE^LN =limγ→∞BiasGPPOLNBiasCELN = _γ→∞ Bias_GPPO^LNBias_CE^LN =limγ→∞1|(1−β1)γ+1−2β1| = _γ→∞ 1 |(1- _1)γ+1-2 _1 | =0 =0 (35) limγ→∞BiasGRPOLNBiasCELN _γ→∞ Bias_GRPO^LNBias_CE^LN =limγ→∞BiasASPOLNBiasCELN = _γ→∞ Bias_ASPO^LNBias_CE^LN =limγ→∞γ+1|(1−β1)γ+1−2β1| = _γ→∞ γ+1 |(1- _1)γ+1-2 _1 | =1|1−β1| = 1|1- _1| (36) Assuming n=1n=1 (linear decay), we obtain the strict ranking: 0<BiasDGPOLN<BiasCISPOLN=BiasGPPOLN<BiasCELN<BiasGRPOLN=BiasASPOLN0< [rgb]0.75390625,0,0Bias_DGPO^LN<Bias_CISPO^LN=Bias_GPPO^LN<Bias_CE^LN<Bias_GRPO^LN=Bias_ASPO^LN. A.3.5 Right-Boundary Bias The gradient estimates for the right-boundary case across algorithms are shown in Eq. (38). The bias magnitude is computed as shown in Table 8. Table 7: Left-Boundary bias among various policy optimization algorithms Algorithm Left-Boundary Bias Magnitude GRPO BiasGRPOLN=‖∇θGRPOLN(θ)−∇θPGLN(θ)‖=∥q∼,o∼πθold[vtLN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_GRPO^LN&= \| _θJ_GRPO^LN(θ)- _θJ_PG^LN(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^LN· r_t· A_t _θ _θ(o_t|q,o_<t) ] \| aligned CISPO BiasCISPOLN=‖∇θCISPOLN(θ)−∇θPGLN(θ)‖=∥q∼,o∼πθold[vtLN⋅(r0−rt)⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_CISPO^LN&= \| _θJ_CISPO^LN(θ)- _θJ_PG^LN(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^LN·(r_0-r_t)· A_t _θ _θ(o_t|q,o_<t) ] \| aligned GPPO BiasGPPOLN=‖∇θGPPOLN(θ)−∇θPGLN(θ)‖=∥q∼,o∼πθold[vtLN⋅(r0−rt)⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_GPPO^LN&= \| _θJ_GPPO^LN(θ)- _θJ_PG^LN(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^LN·(r_0-r_t)· A_t _θ _θ(o_t|q,o_<t) ] \| aligned CE-GPPO BiasCELN=‖∇θCELN(θ)−∇θPGLN(θ)‖=∥q∼,o∼πθoldvtLN⋅[β1r0−rt]⋅At∇θlogπθ(ot|q,o<t)∥ aligned Bias_CE^LN&= \| _θJ_CE^LN(θ)- _θJ_PG^LN(θ) \|\\ &= \|E_q ,\,o _ _old \v_t^LN·[ _1r_0-r_t]· A_t _θ _θ(o_t|q,o_<t) \ \| aligned ASPO BiasASPOLN=‖∇θASPOLN(θ)−∇θPGLN(θ)‖=∥q∼,o∼πθold[vtLN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_ASPO^LN&= \| _θJ_ASPO^LN(θ)- _θJ_PG^LN(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^LN· r_t· A_t _θ _θ(o_t|q,o_<t) ] \| aligned DGPO BiasDGPOLN=‖∇θDGPOLN(θ)−∇θPGLN(θ)‖=∥q∼,o∼πθold[vtLN⋅(rt−rtn+1r0n)⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_DGPO^LN&= \| _θJ_DGPO^LN(θ)- _θJ_PG^LN(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^LN·(r_t- r_t^n+1r_0^n)· A_t _θ _θ(o_t|q,o_<t) ] \| aligned ∇θPGLN(θ)=q∼,o∼πθold[vtLN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θGRPOLN(θ)=q∼,o∼πθold[vtLN⋅0⋅At∇θlogπθ(ot|q,o<t)]∇θCISPOLN(θ)=q∼,o∼πθold[vtLN⋅(1−εlow)⋅At∇θlogπθ(ot|q,o<t)]∇θGPPOLN(θ)=q∼,o∼πθold[vtLN⋅(1−εlow)⋅At∇θlogπθ(ot|q,o<t)]∇θCELN(θ)=q∼,o∼πθold[vtLN⋅β1(1−εlow)⋅At∇θlogπθ(ot|q,o<t)]∇θASPOLN(θ)=q∼,o∼πθold[vtLN⋅0⋅At∇θlogπθ(ot|q,o<t)]∇θDGPOLN(θ)=q∼,o∼πθold[vtLN⋅rtn+1(1−εlow)n⋅At∇θlogπθ(ot|q,o<t)] cases _θJ_PG^LN(θ)=E_q ,\,o _ _old [v_t^LN· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GRPO^LN(θ)=E_q ,\,o _ _old [v_t^LN· 0· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CISPO^LN(θ)=E_q ,\,o _ _old [v_t^LN·(1- _low)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GPPO^LN(θ)=E_q ,\,o _ _old [v_t^LN·(1- _low)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CE^LN(θ)=E_q ,\,o _ _old [v_t^LN· _1(1- _low)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_ASPO^LN(θ)=E_q ,\,o _ _old [v_t^LN· 0· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_DGPO^LN(θ)=E_q ,\,o _ _old [v_t^LN· r_t^n+1(1- _low)^n· A_t _θ _θ(o_t|q,o_<t) ] cases (37) Given the right-boundary condition 1<1+εhigh<rt1<1+ _high<r_t, and adopting the CE-GPPO configuration β2=1 _2=1, we derive the ranking: 0<BiasDGPOHP≤BiasCISPOHP=BiasGPPOHP=BiasCEHP<BiasGRPOHP=BiasASPOHP.0< [rgb]0.75390625,0,0Bias_DGPO^HP _CISPO^HP=Bias_GPPO^HP\\ =Bias_CE^HP<Bias_GRPO^HP=Bias_ASPO^HP. A.3.6 Reverse Left-Boundary Bias Based on the definition of ASPO, if the reverse left-boundary is treated as a clipping region 1−εlow1- _low, the gradient estimates are listed in Eq. (39). The bias magnitude is shown in Table 10. The reverse left-boundary bias magnitude relationship is directly derived as: 0=BiasDGPOLP=BiasGRPOLP=BiasGPPOLP=BiasCELP<BiasCISPOLP=BiasASPOLP.0= [rgb]0.75390625,0,0Bias_DGPO^LP=Bias_GRPO^LP=Bias_GPPO^LP\\ =Bias_CE^LP<Bias_CISPO^LP=Bias_ASPO^LP. Table 8: Right-Boundary bias among various policy optimization algorithms Algorithm Right-Boundary Bias Magnitude GRPO BiasGRPOHP=‖∇θGRPOHP(θ)−∇θPGHP(θ)‖=∥q∼,o∼πθold[vtHP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_GRPO^HP&= \| _θJ_GRPO^HP(θ)- _θJ_PG^HP(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^HP· r_t· A_t _θ _θ(o_t|q,o_<t) ] \| aligned CISPO BiasCISPOHP=‖∇θCISPOHP(θ)−∇θPGHP(θ)‖=∥q∼,o∼πθoldvtHP⋅[rt−(1+εhigh)]⋅At∇θlogπθ(ot|q,o<t)∥ aligned Bias_CISPO^HP&= \| _θJ_CISPO^HP(θ)- _θJ_PG^HP(θ) \|\\ &= \|E_q ,\,o _ _old \v_t^HP·[r_t-(1+ _high)]· A_t _θ _θ(o_t|q,o_<t) \ \| aligned GPPO BiasGPPOHP=‖∇θGPPOHP(θ)−∇θPGHP(θ)‖=∥q∼,o∼πθoldvtHP⋅[rt−(1+εhigh)]⋅At∇θlogπθ(ot|q,o<t)∥ aligned Bias_GPPO^HP&= \| _θJ_GPPO^HP(θ)- _θJ_PG^HP(θ) \|\\ &= \|E_q ,\,o _ _old \v_t^HP·[r_t-(1+ _high)]· A_t _θ _θ(o_t|q,o_<t) \ \| aligned CE-GPPO BiasCEHP=‖∇θCEHP(θ)−∇θPGHP(θ)‖=∥q∼,o∼πθoldvtHP⋅[rt−β2(1+εhigh)]⋅At∇θlogπθ(ot|q,o<t)∥ aligned Bias_CE^HP&= \| _θJ_CE^HP(θ)- _θJ_PG^HP(θ) \|\\ &= \|E_q ,\,o _ _old \v_t^HP·[r_t- _2(1+ _high)]· A_t _θ _θ(o_t|q,o_<t) \ \| aligned ASPO BiasASPOHP=‖∇θASPOHP(θ)−∇θPGHP(θ)‖=∥q∼,o∼πθold[vtHP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_ASPO^HP&= \| _θJ_ASPO^HP(θ)- _θJ_PG^HP(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^HP· r_t· A_t _θ _θ(o_t|q,o_<t) ] \| aligned DGPO BiasDGPOHP=‖∇θDGPOHP(θ)−∇θPGHP(θ)‖=∥q∼,o∼πθold[vtHP⋅[rt−(1+εhigh)1mrt1−1m]⋅At∇θlogπθ(ot|q,o<t)]∥ aligned Bias_DGPO^HP&= \| _θJ_DGPO^HP(θ)- _θJ_PG^HP(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^HP·[r_t-(1+ _high) 1mr_t^1- 1m]· A_t _θ _θ(o_t|q,o_<t) ] \| aligned ∇θPGHP(θ)=q∼,o∼πθold[vtHP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θGRPOHP(θ)=q∼,o∼πθold[vtHP⋅0⋅At∇θlogπθ(ot|q,o<t)]∇θCISPOHP(θ)=q∼,o∼πθold[vtHP⋅(1+εhigh)⋅At∇θlogπθ(ot|q,o<t)]∇θGPPOHP(θ)=q∼,o∼πθold[vtHP⋅(1+εhigh)⋅At∇θlogπθ(ot|q,o<t)]∇θCEHP(θ)=q∼,o∼πθold[vtHP⋅β2(1+εhigh)⋅At∇θlogπθ(ot|q,o<t)]∇θASPOHP(θ)=q∼,o∼πθold[vtHP⋅0⋅At∇θlogπθ(ot|q,o<t)]∇θDGPOHP(θ)=q∼,o∼πθold[vtHP⋅(1+εhigh)1mrt1−1m⋅At∇θlogπθ(ot|q,o<t)] cases _θJ_PG^HP(θ)=E_q ,\,o _ _old [v_t^HP· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GRPO^HP(θ)=E_q ,\,o _ _old [v_t^HP· 0· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CISPO^HP(θ)=E_q ,\,o _ _old [v_t^HP·(1+ _high)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GPPO^HP(θ)=E_q ,\,o _ _old [v_t^HP·(1+ _high)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CE^HP(θ)=E_q ,\,o _ _old [v_t^HP· _2(1+ _high)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_ASPO^HP(θ)=E_q ,\,o _ _old [v_t^HP· 0· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_DGPO^HP(θ)=E_q ,\,o _ _old [v_t^HP·(1+ _high) 1mr_t^1- 1m· A_t _θ _θ(o_t|q,o_<t) ] cases (38) A.3.7 Reverse Right-Boundary Bias Assuming the reverse right-boundary of ASPO is 1+εhigh1+ _high, the gradient estimates are listed in Eq. (40). The bias magnitude is shown in Table 10. The reverse right-boundary bias magnitude relationship is directly derived as: 0=BiasDGPOHN=BiasGRPOHN=BiasGPPOHN=BiasCEHN<BiasCISPOHN=BiasASPOHN.0= [rgb]0.75390625,0,0Bias_DGPO^HN=Bias_GRPO^HN=Bias_GPPO^HN\\ =Bias_CE^HN<Bias_CISPO^HN=Bias_ASPO^HN. A.4 Derivation of Learning Rate Scaling To ensure consistent training dynamics across models of varying sizes (1.5B, 7B, and 14B), we employ a learning rate scaling rule based on the principle of Constant Total Gradient Variance. Below, we provide the formal derivation of this scaling law. Assumptions. Consider two models with identical architecture but different parameter counts, denoted by N. We make the following standard assumptions for large-scale model training: • The training data, batch size, and optimizer configuration remain constant. • The gradient variance of a single parameter, denoted as σ2σ^2, is constant and independent of the total model size (assuming high redundancy in LLM parameters). • The gradients of individual parameters are approximately independent. Table 9: Reverse Left-Boundary bias among various policy optimization algorithms Algorithm Reverse Left-Boundary Bias Magnitude GRPO, GPPO BiasXLP=‖∇θXLP(θ)−∇θPGLP(θ)‖=0 aligned Bias_X^LP= \| _θJ_X^LP(θ)- _θJ_PG^LP(θ) \|=0 aligned CE-GPPO, DGPO CISPO, ASPO BiasCISPOLP=‖∇θCISPOLP(θ)−∇θPGLP(θ)‖=∥q∼,o∼πθold[vtLP⋅(1−εlow−rt)⋅At∇θlogπθ(ot|q,o<t)]∥≠0∗ aligned Bias_CISPO^LP&= \| _θJ_CISPO^LP(θ)- _θJ_PG^LP(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^LP·(1- _low-r_t)· A_t _θ _θ(o_t|q,o_<t) ] \|≠ 0^* aligned Table 10: Reverse Right-Boundary bias among various policy optimization algorithms Algorithm Reverse Left-Boundary Bias Magnitude GRPO, GPPO BiasXHN=‖∇θXHN(θ)−∇θPGHN(θ)‖=0 aligned Bias_X^HN= \| _θJ_X^HN(θ)- _θJ_PG^HN(θ) \|=0 aligned CE-GPPO, DGPO CISPO, ASPO BiasCISPOHN=‖∇θCISPOHN(θ)−∇θPGHN(θ)‖=∥q∼,o∼πθold[vtHN⋅[rt−(1+εhigh)]⋅At∇θlogπθ(ot|q,o<t)]∥≠0∗ aligned Bias_CISPO^HN&= \| _θJ_CISPO^HN(θ)- _θJ_PG^HN(θ) \|\\ &= \|E_q ,\,o _ _old [v_t^HN·[r_t-(1+ _high)]· A_t _θ _θ(o_t|q,o_<t) ] \|≠ 0^* aligned ∇θPGLP(θ)=q∼,o∼πθold[vtLP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θGRPOLP(θ)=q∼,o∼πθold[vtLP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θCISPOLP(θ)=q∼,o∼πθold[vtLP⋅(1−εlow)⋅At∇θlogπθ(ot|q,o<t)]∇θGPPOLP(θ)=q∼,o∼πθold[vtLP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θCELP(θ)=q∼,o∼πθold[vtLP⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θASPOLP(θ)=q∼,o∼πθold[vtLP⋅(1−εlow)⋅At∇θlogπθ(ot|q,o<t)]∇θDGPOLP(θ)=q∼,o∼πθold[vtLP⋅rt⋅At∇θlogπθ(ot|q,o<t)] cases _θJ_PG^LP(θ)=E_q ,\,o _ _old [v_t^LP· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GRPO^LP(θ)=E_q ,\,o _ _old [v_t^LP· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CISPO^LP(θ)=E_q ,\,o _ _old [v_t^LP·(1- _low)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GPPO^LP(θ)=E_q ,\,o _ _old [v_t^LP· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CE^LP(θ)=E_q ,\,o _ _old [v_t^LP· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_ASPO^LP(θ)=E_q ,\,o _ _old [v_t^LP·(1- _low)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_DGPO^LP(θ)=E_q ,\,o _ _old [v_t^LP· r_t· A_t _θ _θ(o_t|q,o_<t) ] cases (39) ∇θPGHN(θ)=q∼,o∼πθold[vtHN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θGRPOHN(θ)=q∼,o∼πθold[vtHN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θCISPOHN(θ)=q∼,o∼πθold[vtHN⋅(1+εhigh)⋅At∇θlogπθ(ot|q,o<t)]∇θGPPOHN(θ)=q∼,o∼πθold[vtHN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θCEHN(θ)=q∼,o∼πθold[vtHN⋅rt⋅At∇θlogπθ(ot|q,o<t)]∇θASPOHN(θ)=q∼,o∼πθold[vtHN⋅(1+εhigh)⋅At∇θlogπθ(ot|q,o<t)]∇θDGPOHN(θ)=q∼,o∼πθold[vtHN⋅rt⋅At∇θlogπθ(ot|q,o<t)] cases _θJ_PG^HN(θ)=E_q ,\,o _ _old [v_t^HN· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GRPO^HN(θ)=E_q ,\,o _ _old [v_t^HN· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CISPO^HN(θ)=E_q ,\,o _ _old [v_t^HN·(1+ _high)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_GPPO^HN(θ)=E_q ,\,o _ _old [v_t^HN· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_CE^HN(θ)=E_q ,\,o _ _old [v_t^HN· r_t· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_ASPO^HN(θ)=E_q ,\,o _ _old [v_t^HN·(1+ _high)· A_t _θ _θ(o_t|q,o_<t) ]\\ _θJ_DGPO^HN(θ)=E_q ,\,o _ _old [v_t^HN· r_t· A_t _θ _θ(o_t|q,o_<t) ] cases (40) Total Gradient Variance. Let ∇θℒ _θL represent the gradient vector of the loss function with respect to the model parameters θ∈ℝNθ ^N. Based on the independence assumption, the variance of the total gradient norm is the sum of the variances of individual parameter gradients: Var(∇θℒ)=∑i=1NVar(gi)=N⋅σ2Var( _θL)= _i=1^NVar(g_i)=N·σ^2 (41) Stability Condition. To maintain a consistent convergence speed across different scales, we require the variance of the parameter update step (the “step size” in the parameter space) to remain constant. Let η be the learning rate. The update step is Δθ=η⋅∇θℒ θ=η· _θL. The variance of this update step is: Var(Δθ) ( θ) =Var(η⋅∇θℒ) =Var(η· _θL) =η2⋅Var(∇θℒ)=η2Nσ2 =η^2·Var( _θL)=η^2Nσ^2 (42) We define a stability constant C such that: η2Nσ2=Cη^2Nσ^2=C (43) Scaling Law Derivation. Let (η1,N1)( _1,N_1) be the configuration for the base model (1.5B) and (η2,N2)( _2,N_2) be the configuration for the target model (e.g., 7B or 14B). From Eq. (43), we have: η12N1σ2=η22N2σ2=C _1^2N_1σ^2= _2^2N_2σ^2=C (44) Eliminating constant terms σ2σ^2 and C, we obtain the relationship: η12N1=η22N2⟹η2=η1N1N2 _1^2N_1= _2^2N_2 _2= _1 N_1N_2 (45) This establishes the Inverse Square Root Scaling Law relative to the number of parameters. Numerical Verification. Applying this rule to our experimental settings with the base learning rate η1.5B=1.0×10−6 _1.5B=1.0× 10^-6: • For 7B Model: η7B _7B =1.0×10−6×1.57 =1.0× 10^-6× 1.57 ≈1.0×10−6×0.4629≈4.63×10−7 ≈ 1.0× 10^-6× 0.4629≈ 4.63× 10^-7 • For 14B Model: η14B _14B =1.0×10−6×1.514 =1.0× 10^-6× 1.514 ≈1.0×10−6×0.3273≈3.27×10−7 ≈ 1.0× 10^-6× 0.3273≈ 3.27× 10^-7 These calculated values correspond exactly to the learning rates reported in Table 12. Appendix B Implementation Details B.1 Infrastructure and Environment Computational Resources. Our experiments are conducted on a high-performance computing cluster consisting of 30 nodes. Each node is equipped with 8 × NVIDIA A100 GPUs (80GB VRAM), interconnected via NVLink for high-bandwidth intra-node communication. The multi-node training relies on a robust Ethernet/InfiniBand fabric to ensure synchronization efficiency. The total effective training time for the largest model (14B) was approximately 500 hours. Software Stack. We build our reinforcement learning pipeline upon VeRL Sheng et al. (2025), a flexible framework designed for post-training. • Training Backend: We utilize Fully Sharded Data Parallel (FSDP) for distributed training. This setup manages memory efficiency through parameter sharding and offloading, allowing us to train 14B models with full-parameter updates without memory overflow. • Inference Engine: To maximize rollout throughput, we integrate vLLM as the inference backend. We leverage its PagedAttention mechanism to efficiently manage Key-Value (KV) cache memory, significantly reducing fragmentation during the generation of long reasoning chains. We align the environment configurations (CUDA version 12.4, PyTorch version 2.6.0) across all nodes to prevent numerical discrepancies. B.2 Data and Evaluation Benchmarks Training Dataset. We utilize the DAPO-Math-17k dataset for training. We employ the official tokenizer corresponding to the Qwen2.5-Math series to ensure consistent token mapping between the pre-trained backbone and the RL fine-tuning stage. The maximum response length is set to 8192 tokens. This length is empirically chosen to sufficiently accommodate the chain-of-thought reasoning steps required by benchmarks like AIME and MATH, while maintaining high training throughput. Benchmark Details To rigorously evaluate the mathematical reasoning capabilities of our models, we selected a suite of diverse benchmarks. These datasets cover a spectrum of difficulty levels, from foundational high school mathematics to expert-level olympiad problems. Table 11 provides a comprehensive overview of their characteristics. Dataset Core Description Key Characteristics AIME 2024 The 2024 edition of the American Invitational Mathematics Examination, serving as a bridge between the AMC and the USAMO. • Focuses on arithmetic precision and number theory. • Requires answers in a strict integer format (000–999). • Tests robustness against recent contamination. AIME 2025 The most recent iteration of the AIME competition, representing a strictly “held-out” set for evaluating generalization to unseen problems. • Zero data contamination risk due to recency. • High complexity requiring multi-step logical chains. • Validates the model’s potential for future reasoning. AMC 2023 Selected problems from the 2023 American Mathematics Competitions (AMC 10/12), representing the entry-level olympiad difficulty. • Covers broad topics: Algebra, Geometry, Counting. • Functions as a baseline for competitive math ability. • Requires mapping multiple-choice logic to open-ended generation. MATH-500 A curated subset of 500 representative problems from the widely used MATH dataset, designed by OpenAI for efficient evaluation. • Spans 7 categories including Calculus and Probability. • Reduces evaluation costs while maintaining distribution fidelity. • Heavily relies on LaTeX understanding. Minerva A collection of technical mathematics problems derived from scientific papers and undergraduate-level coursework. • Involves higher-order symbolic reasoning. • Contains domain-specific vocabulary and notation. • Tests capabilities beyond standard competition math. OlympiadBench A comprehensive aggregate of international mathematics competitions (e.g., IMO, CMO) spanning multiple languages and formats. • Represents the upper bound of mathematical reasoning. • Includes theorem proving and fill-in-the-blank types. • Challenges the model’s cross-lingual mathematical logic. Table 11: Detailed comparison of the mathematical reasoning benchmarks used in this study. The datasets used for evaluation can be accessed via the following repositories: • AIME 2024: https://huggingface.co/datasets/math-ai/aime24 • AIME 2025: https://huggingface.co/datasets/math-ai/aime25 • AMC 2023: https://huggingface.co/datasets/math-ai/amc23 • MATH-500: https://huggingface.co/datasets/HuggingFaceH4/MATH-500 • Minerva: https://huggingface.co/datasets/math-ai/minervamath • OlympiadBench: https://huggingface.co/datasets/math-ai/olympiadbench Reward Function. We employ a strict rule-based reward mechanism to rigorously verify the correctness of the generated solutions. • Format Verification: We first check if the output follows the required format (e.g., enclosing the answer in ). • Correctness Check: Using the math_verify toolkit, we compare the extracted answer against the ground truth. A reward of r=1r=1 is assigned for a correct match, and r=−1r=-1 otherwise. This binary reward setting poses a significant challenge for exploration, as no partial rewards are applied. B.3 Training Protocols Optimization Strategy. We employ the AdamW optimizer with β1=0.9 _1=0.9, β2=0.95 _2=0.95, and a small weight decay. Unlike standard pre-training schedules, we adopt a constant learning rate strategy without warm-up or cosine decay. This design choice eliminates the confounding factors of learning rate scheduling, allowing us to attribute performance gains solely to the algorithmic improvements. To ensure consistent convergence dynamics across different model scales (1.5B, 7B, 14B), we calibrate the learning rate based on the Constant Total Gradient Variance principle. Numerical Precision and Stability. All models are trained using bfloat16 precision. We apply gradient clipping with a norm threshold of 1.0 to mitigate gradient explosion. Random seeds for model initialization, data shuffling, and sampling are fixed to 42 to ensure reproducibility. B.4 Hyperparameter Specifications Table 12 summarizes the key hyperparameters used across all experiments. Table 12: Detailed hyperparameter configurations for all experiments. To ensure a fair comparison, we maintain identical training settings across all algorithms, varying only the learning rate according to the model scale. Model Scale Algorithm Learning Rate Specific Hyperparameters Common Settings: Mini-batch Size = 32, Rollout Batch Size = 512, Max Length = 8192 DeepSeek-R1-Distill-Qwen-1.5B GRPO 1.0×10−61.0× 10^-6 εlow=εhigh=0.2 _low= _high=0.2 CISPO εlow=εhigh=0.2 _low= _high=0.2 GPPO εlow=εhigh=0.2 _low= _high=0.2 CE-GPPO εlow=εhigh=0.2,β1=0.75,β2=1 _low= _high=0.2, _1=0.75, _2=1 ASPO εlow=εhigh=0.2,εlow′=0.33,εhigh′=3 _low= _high=0.2, _low=0.33, _high=3 DGPO εlow=εhigh=0.2,n=2,m=2 _low= _high=0.2,n=2,m=2 DeepSeek-R1-Distill-Qwen-7B GRPO 4.63×10−74.63× 10^-7 εlow=εhigh=0.2 _low= _high=0.2 CISPO εlow=εhigh=0.2 _low= _high=0.2 GPPO εlow=εhigh=0.2 _low= _high=0.2 CE-GPPO εlow=εhigh=0.2,β1=0.75,β2=1 _low= _high=0.2, _1=0.75, _2=1 ASPO εlow=εhigh=0.2,εlow′=0.33,εhigh′=3 _low= _high=0.2, _low=0.33, _high=3 DGPO εlow=εhigh=0.2,n=1,m=2 _low= _high=0.2,n=1,m=2 DeepSeek-R1-Distill-Qwen-14B GRPO 3.27×10−73.27× 10^-7 εlow=εhigh=0.2 _low= _high=0.2 CISPO εlow=εhigh=0.2 _low= _high=0.2 GPPO εlow=εhigh=0.2 _low= _high=0.2 CE-GPPO εlow=εhigh=0.2,β1=0.75,β2=1 _low= _high=0.2, _1=0.75, _2=1 ASPO εlow=εhigh=0.2,εlow′=0.33,εhigh′=3 _low= _high=0.2, _low=0.33, _high=3 DGPO εlow=εhigh=0.2,n=1,m=2 _low= _high=0.2,n=1,m=2 (a) AIME 2024 (Avg@32) (b) AIME 2025 (Avg@32) (c) Policy Entropy Figure 5: Training dynamics of DeepSeek-R1-Distill-Qwen-14B comparing GRPO and DGPO. Controlled Variables. To ensure a fair comparison and strictly isolate the impact of different gradient weighting strategies, we unify the clipping thresholds across all experiments. Specifically, for all methods involving trust region clipping or boundary definitions (including GRPO, CISPO, GPPO, CE-GPPO, ASPO and our DGPO), we fix: εlow=εhigh=0.2 _low= _high=0.2 This rigorous control ensures that any observed performance differences are primarily attributable to the gradient dynamics rather than minor variations in the trust region size. Common Configurations. We set the KL coefficient βKL=0 _KL=0. By removing the explicit KL penalty, we rely implicitly on the trust region constraints imposed by the clipping mechanisms to prevent policy collapse. The rollout generation uses a temperature of 1.0 and a top-p of 1.0. Method-Specific Settings. The method-specific configurations are as follows: • CE-GPPO: Adopts scaled soft clipping with β1=0.75 _1=0.75 and β2=1.0 _2=1.0, as recommended in the original paper. • ASPO: Use εlow′=0.33 _low=0.33 and εhigh′=3 _high=3 as soft dual clip threshold. • DGPO: We introduce the decoupled decay parameters n and m: – 1.5B: n=2n=2, m=2m=2. – 7B & 14B: n=1n=1, m=2m=2. The continuity constants CleftC_left and CrightC_right are automatically calculated based on πθold _ _old. Appendix C Additional Experimental Results C.1 14B Model Training Dynamics We visualize the training trajectories of the DeepSeek-R1-Distill-Qwen-14B model in Figure 5. Similar to the 1.5B and 7B models, DGPO on 14B scale demonstrates faster convergence and higher asymptotic performance on AIME benchmarks while maintaining a stable entropy reduction curve, avoiding the collapse issues seen in RLVR. C.2 Detailed 14B Performance Comparison Table 14 provides comprehensive evaluation results for 14B model, including both Avg@32 (expected performance) and Pass@32 (potential capability). C.3 Pass@K Statistics on AIME Benchmarks We report Pass@K (k∈1,2,4,8,16,32k∈\1,2,4,8,16,32\) metrics specifically for the AIME 2024 and AIME 2025 benchmarks across 1.5B, 7B, and 14B scales (Table 14). The results demonstrate that DGPO consistently achieves higher coverage of the solution space (higher Pass@K) compared to baselines, particularly as k increases. Table 13: Full performance comparison on DeepSeek-R1-Distill-Qwen-14B. Method AIME24 AIME25 AMC23 MATH500 Minerva Olympiad Avg. A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 A@32 P@32 DeepSeek-R1-Distill-Qwen-14B GRPO 56.6 82.9 40.5 67.8 92.2 97.4 66.5 70.4 22.6 35.3 43.2 50.7 53.6 67.4 DGPO 63.3 86.2 47.6 66.4 93.9 99.1 67.0 75.1 23.0 37.9 45.1 57.4 56.7 70.4 Table 14: Detailed Pass@K performance on AIME 2024 and AIME 2025 across all model scales. Method P@1 P@2 P@4 P@8 P@16 P@32 DeepSeek-R1-Distill-Qwen-1.5B Dataset: AIME 2024 GRPO 33.2 43.2 53.2 61.6 67.9 71.8 CISPO 34.8 44.1 52.0 58.7 64.4 69.1 GPPO 29.6 38.4 47.1 53.6 57.9 60.5 CE-GPPO 35.1 44.4 52.6 60.0 66.2 70.2 ASPO 36.4 44.4 53.1 61.7 68.5 73.2 DGPO 43.3 53.4 63.1 70.6 75.6 79.3 Dataset: AIME 2025 GRPO 27.7 33.0 38.1 42.6 46.5 49.9 CISPO 25.8 31.8 37.6 43.6 49.2 53.3 GPPO 23.5 29.3 34.3 40.0 46.4 51.9 CE-GPPO 27.7 33.1 38.3 44.1 50.1 55.1 ASPO 28.3 33.0 37.1 41.2 45.9 51.5 DGPO 32.8 38.3 43.6 48.6 52.7 56.1 DeepSeek-R1-Distill-Qwen-7B Dataset: AIME 2024 GRPO 48.2 58.0 66.6 73.4 78.6 82.5 CISPO 51.6 60.9 68.5 73.5 75.8 76.6 GPPO 43.1 52.1 59.3 65.0 69.7 72.5 CE-GPPO 48.7 55.7 62.3 68.1 73.0 76.9 ASPO 51.8 61.2 68.5 73.4 76.7 79.6 DGPO 55.5 64.1 71.6 77.0 80.2 81.9 Dataset: AIME 2025 GRPO 37.4 44.4 50.8 55.2 58.1 60.5 CISPO 38.2 45.2 51.7 56.5 60.8 65.4 GPPO 31.7 38.5 45.8 53.0 58.6 62.5 CE-GPPO 36.4 43.1 49.1 53.9 57.4 60.4 ASPO 37.1 43.3 48.4 51.2 52.6 54.1 DGPO 43.1 49.7 55.2 60.1 64.6 68.0 DeepSeek-R1-Distill-Qwen-14B Dataset: AIME 2024 GRPO 56.6 69.3 76.0 79.8 81.9 82.9 DGPO 63.3 70.4 76.4 81.1 84.6 86.2 Dataset: AIME 2025 GRPO 40.5 48.4 54.2 60.1 64.7 67.8 DGPO 47.6 53.7 58.7 61.7 63.9 66.4