Paper deep dive
Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning
Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 12:53:17 PM
Summary
The paper identifies that standard Reinforcement Learning with Verifiable Rewards (RLVR) algorithms suffer from diversity collapse due to uniform penalization of errors, which allows overconfident incorrect reasoning paths to persist. The authors propose Asymmetric Confidence-aware Error Penalty (ACE), a method that dynamically modulates negative advantages based on a per-rollout confidence shift metric. ACE amplifies penalties for overconfident errors while sparing exploratory and self-correcting errors, theoretically decomposing into a selective regularizer. Experiments on Qwen and Llama models demonstrate that ACE improves the full Pass@k spectrum on MATH-500 and AIME 2025 benchmarks.
Entities (13)
Relation Signals (9)
ACE → addresses → overconfident errors
confidence 95% · We identify a root cause that existing methods overlook: the uniform penalization of errors... allows overconfident errors... to persist... To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE).
ACE → improves → pass@k
confidence 95% · ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.
ACE → composeswith → DAPO
confidence 90% · ACE composes with both GRPO and DAPO
ACE → composeswith → GRPO
confidence 90% · ACE composes with both GRPO and DAPO
RLVR → suffersfrom → Diversity Collapse
confidence 90% · standard RLVR algorithms suffer from a well-documented pathology... narrow the model's reasoning boundary and reduce generation diversity.
Qwen2.5-Math-7B → evaluatedon → MATH-500
confidence 85% · Evaluated on MATH-500 and AIME 2025... across all three model families
Qwen2.5-Math-7B → evaluatedon → AIME 2025
confidence 85% · Evaluated on MATH-500 and AIME 2025... across all three model families
Qwen3-8B-Base → evaluatedon → MATH-500
confidence 85% · Evaluated on MATH-500 and AIME 2025... across all three model families
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while they improve Pass@1 accuracy through sharpened sampling, they simultaneously narrow the model's reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches -- whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes -- treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors (incorrect reasoning paths that the RL process has spuriously reinforced) to persist and monopolize probability mass, ultimately suppressing valid exploratory trajectories. To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE). ACE introduces a per-rollout confidence shift metric, c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x)), to dynamically modulate negative advantages. Theoretically, we demonstrate that ACE's gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer's strength. We conduct extensive experiments fine-tuning Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on the DAPO-Math-17K dataset using GRPO and DAPO within the VERL framework. Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2602.21420v1
- Canonical: https://arxiv.org/abs/2602.21420v1
Trouble viewing inline? Open PDF directly →
Full Text
80,926 characters extracted from source content.
Expand or collapse full text
Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning Yuanda Xu*† Hejian Sang* Zhengze Zhou* Ran He* Zhipeng Wang‡ LinkedIn Corporation, CA, USA *Co-first authors with equal contribution. †ericxu@linkedin.com ‡zhipwang@linkedin.com Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while improving Pass@1 through sharpened sampling, they simultaneously narrow the model’s reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches—whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes—treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors—incorrect reasoning paths that the RL process has spuriously reinforced—to persist and monopolize probability mass, suppressing valid exploratory trajectories. We propose the Asymmetric Confidence-aware Error Penalty (ACE), which introduces a per-rollout confidence shift metric ci=log(πθ(yi|x)/πref(yi|x))c_i= ( _θ(y_i|x)/ _ref(y_i|x)) to dynamically modulate negative advantages. Theoretically, we show that ACE’s gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer’s strength (Theorem 1). Experiments fine-tune Qwen2.5-Math-7B (Qwen Team et al., 2024), Qwen3-8B-Base (Qwen Team et al., 2025), and Llama-3.1-8B-Instruct (Grattafiori et al., 2024) on the DAPO-Math-17K dataset (Yu et al., 2025) using GRPO and DAPO with VERL (Volcano Engine, 2024), evaluating on MATH-500 (Hendrycks et al., 2021) and AIME 2025. ACE composes with both GRPO and DAPO, consistently improving the full Pass@k spectrum across all three model families and benchmarks. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) (DeepSeek-AI et al., 2025; OpenAI et al., 2024) has emerged as a primary method for post-training Large Language Models (LLMs) on reasoning tasks. By using binary correctness signals from deterministic verifiers, algorithms such as PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), and REINFORCE (Williams, 1992) iteratively refine the model’s Chain-of-Thought (CoT) generation (Wei et al., 2022). Despite its successes, a growing body of evidence reveals a fundamental tension in RLVR training. While RLVR models excel at Pass@1, they consistently underperform their own base models at Pass@k for large k (Chen et al., 2021; Yue et al., 2025; Brown et al., 2024), indicating a narrowing of the reasoning boundary rather than an expansion. This phenomenon has been attributed to diversity collapse: the training process concentrates probability mass on a small number of successful reasoning paths, suppressing the broader solution space. A prominent strategy addresses this: Difficulty-based curriculum learning filters prompts to maximize gradient signal. However, such methods operate at a macro level—selecting which problems to train on—ignoring a critical micro-level distinction: not all errors are equal. Within incorrect rollouts, we identify distinct regimes: exploratory errors (benign stochastic deviations), self-correcting errors (paths the model is already abandoning), and overconfident errors (spuriously reinforced paths acting as value traps). Standard RLVR penalizes these uniformly. While the global KL penalty βKL(πθ∥πref) _KL( _θ\| _ref) offers some correction, it is symmetric and indiscriminate, suppressing beneficial exploration alongside harmful overconfidence. Our contribution. We propose to break this dilemma by introducing asymmetric correction at the level of individual rollouts. Our method, ACE (Asymmetric Confidence-aware Error penalty), dynamically amplifies the penalty for overconfident errors using a per-rollout confidence shift metric, while leaving exploratory and self-correcting errors largely untouched. Concretely, our contributions are: 1. A new analytical dimension. We formalize error confidence shift ci=log(πθ(yi|x)/πref(yi|x))c_i= ( _θ(y_i|x)/ _ref(y_i|x)) as a per-rollout diagnostic that is orthogonal to prompt-level difficulty, and show empirically that overconfident errors accumulate during training (§3). 2. Theoretical foundations. We show that ACE’s gradient admits a decomposition into a selective regularizer targeting the overconfident portion of the policy, plus a residual term that partially moderates the regularizer’s correction strength (§4.4). 3. Empirical validation. ACE consistently improves the full Pass@k spectrum on MATH-500 and AIME 2025, across three model families (Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct) and two base algorithms (GRPO and DAPO), with particularly strong gains at large k, confirming that it preserves and expands the reasoning boundary (§5). Figure 1: ACE Method Overview. Top: Incorrect rollouts fall into three regimes based on the confidence shift ci=log(πθ(yi|x)/πref(yi|x))c_i= ( _θ(y_i|x)/ _ref(y_i|x)). Bottom-left: Standard GRPO assigns a uniform penalty |A^−|| A^-| to all errors regardless of regime. Bottom-right: ACE modulates the penalty via Softplus(ci)Softplus(c_i), strongly penalizing overconfident errors while leaving self-correcting errors nearly untouched. Figure 1 highlights the core idea: ACE reshapes the negative penalty as a smooth, confidence-dependent curve, amplifying penalties for overconfident errors while keeping exploratory and self-correcting errors close to the base level. 2 Related Work Curriculum and advantage shaping. Curriculum methods (Zeng et al., 2025; Parashar et al., 2025; Zhang et al., 2025) select prompts by difficulty, operating at the prompt level. Advantage shaping methods (Tang et al., 2025; Wen et al., 2025) balance correct vs. incorrect samples at the group level. ACE operates at the rollout level, modulating penalties within incorrect samples based on per-rollout confidence shift. KL regularization in RLHF/RLVR. KL divergence penalties are standard in RLHF pipelines to prevent reward hacking and mode collapse (Ouyang et al., 2022; Stiennon et al., 2020). The typical formulation adds a global term βKL(πθ∥πref) _KL( _θ\| _ref) that symmetrically penalizes all deviations from the reference. DPO (Rafailov et al., 2023a) implicitly constrains the KL divergence through its closed-form reward parameterization. However, all these methods apply KL penalties uniformly across correct and incorrect outputs alike, suppressing beneficial exploration alongside harmful overconfidence. ACE introduces an asymmetric and selective KL-like penalty that targets only overconfident errors while leaving correct outputs and self-correcting errors untouched. Entropy regularization and clipping strategies. Entropy bonuses have a long history in RL for encouraging exploration (Williams, 1992; Schulman et al., 2017). In the LLM context, DAPO (Yu et al., 2025) combats entropy collapse through its Clip-Higher strategy, which decouples the upper and lower clipping thresholds of the importance sampling ratio to give low-probability exploration tokens more room for probability increase. While such clipping-based strategies promote diversity globally, they operate at the token level and cannot distinguish between beneficial diversity on correct reasoning paths and harmful persistence of incorrect ones. ACE provides a complementary, more targeted mechanism: rather than modifying the clipping bounds, it modulates the penalty magnitude per rollout based on confidence shift, achieving diversity preservation as a consequence of selectively suppressing overconfident errors (see §5.4). Reward shaping. Potential-based reward shaping (Ng et al., 1999; Wiewiora et al., 2003; Devlin and Kudenko, 2012) transforms the reward function to accelerate learning while preserving the optimal policy. ACE can be viewed through the reward shaping lens: the confidence-dependent term α⋅Softplus(ci)α·Softplus(c_i) acts as an auxiliary reward signal derived from the policy–reference divergence. Unlike classical potential-based shaping, ACE’s shaping signal is asymmetric (applied only to negative advantages) and adaptive (it evolves with the policy). Process reward models (Lightman et al., 2023) offer another form of reward enrichment at the step level; ACE is complementary, operating at the trajectory level with zero additional annotation cost. Diversity loss in RLVR. Yue et al. (Yue et al., 2025) show RLVR narrows reasoning boundaries. Negative sample reinforcement (Zhu et al., 2025) demonstrates the importance of learning from incorrect rollouts but does not differentiate among error types. We identify overconfident errors as a key mechanism driving diversity collapse and propose confidence-based differential penalization to address this. 3 Preliminaries Setting. We consider a policy πθ _θ parameterized by θ, initialized from a reference model πref _ref. Given a prompt x∼x , the model generates G rollouts y1,…,yG\y_1,…,y_G\. Each rollout receives a reward ri∈ℝr_i from a reward function or verifier. We define: • Empirical mean reward: μ^x=1G∑i=1Gri μ_x= 1G _i=1^Gr_i • Empirical reward standard deviation: σ^x=1G∑i=1G(ri−μ^x)2 σ_x= 1G _i=1^G(r_i- μ_x)^2 • Above-average rollout set: +(x)=yi:ri>μ^xY^+(x)=\y_i:r_i> μ_x\ • Below-average rollout set: −(x)=yi:ri≤μ^xY^-(x)=\y_i:r_i≤ μ_x\ GRPO objective. In Group Relative Policy Optimization (Shao et al., 2024), the advantage for rollout yiy_i is computed via group normalization: A^i=ri−μ^xσ^x+ϵ A_i= r_i- μ_x σ_x+ε (1) where ϵε is a small constant for numerical stability. The clipped surrogate objective is: ℒGRPO(θ)=−x∼[1G∑i=1Gmin(ρiA^i,clip(ρi,1−ϵc,1+ϵc)A^i)]+βKL(πθ∥πref)L_GRPO(θ)=-E_x [ 1G _i=1^G \! ( _i A_i,\;clip( _i,1- _c,1+ _c) A_i ) ]+ _KL( _θ\| _ref) (2) where ρi=πθ(yi|x)/πold(yi|x) _i= _θ(y_i|x)/ _old(y_i|x) is the importance sampling ratio and ϵc _c is the clipping threshold. Observation: uniform penalty within groups. For rollouts with identical rewards ri=rjr_i=r_j, the advantages are also identical: A^i=A^j A_i= A_j. In the special case of binary rewards, all incorrect rollouts (ri=0r_i=0) share the same advantage: A^i−=−μ^xσ^x+ϵ A_i^-= - μ_x σ_x+ε (3) More generally, rollouts with the same reward receive identical advantage values regardless of their qualitative differences. The only per-rollout modulation comes from the importance ratio ρi _i, which is bounded by clipping and provides limited differentiation. Motivation: overconfident errors. Define the per-rollout confidence shift ci=log(πθ(yi|x)/πref(yi|x))c_i= ( _θ(y_i|x)/ _ref(y_i|x)): positive values indicate the policy has become more confident than the reference on rollout yiy_i, while negative values indicate the opposite. Training Qwen2.5-Math-7B with standard GRPO on DAPO-Math-17K (Yu et al., 2025), we observe that the distribution of cic_i among incorrect rollouts develops a heavy right tail as training progresses—a substantial fraction of errors become significantly more probable under the trained policy than under the reference, even though they remain incorrect. This is consistent with analyses of implicit reward distributions in preference optimization (Rafailov et al., 2023b; Meng et al., 2024). These overconfident errors consume probability mass that would otherwise support diverse reasoning paths, contributing to the diversity collapse documented by Yue et al. (2025). Crucially, the standard global KL penalty βKL(πθ∥πref) _KL( _θ\| _ref) cannot selectively address this: it penalizes all deviations from the reference proportionally, suppressing beneficial confidence growth on correct paths alongside harmful overconfidence on incorrect ones. This structural limitation motivates a targeted correction mechanism (see §5.3 for detailed quantitative tracking). 4 The ACE Method 4.1 Error Confidence Score Definition 1 (Error Confidence Score). For a prompt x and an incorrect rollout yi∈−(x)y_i ^-(x), the error confidence score is: ci≜logπθ(yi|x)πref(yi|x)=∑t=1Tilogπθ(yi(t)|x,yi(<t))πref(yi(t)|x,yi(<t))c_i _θ(y_i|x) _ref(y_i|x)= _t=1^T_i _θ(y_i^(t)|x,y_i^(<t)) _ref(y_i^(t)|x,y_i^(<t)) (4) where yi(t)y_i^(t) denotes the t-th token and TiT_i is the sequence length. The second equality decomposes the sequence-level confidence into a sum of token-level log-ratios. This is important for two reasons: (a) it shows that cic_i is already computed as a byproduct of standard RLVR training (which requires logπθ _θ and logπref _ref for the KL penalty and importance ratios), incurring zero additional compute; and (b) it reveals that cic_i aggregates confidence shifts across all reasoning steps, naturally weighting tokens where the policy has diverged most from the reference. Remark 1 (Three regimes). The sign of cic_i partitions incorrect rollouts into interpretable regimes: • ci>0c_i>0: Overconfident errors. The policy assigns higher probability than the reference. These are spurious patterns actively learned during RL. • ci≈0c_i≈ 0: Exploratory errors. Probability approximately unchanged from the reference. Natural stochastic deviations. • ci<0c_i<0: Self-correcting errors. The policy has already reduced probability mass relative to the reference. 4.2 The ACE Advantage We restructure the negative advantage to depend on the per-rollout confidence score cic_i. Definition 2 (ACE Advantage). For an incorrect rollout yi∈−(x)y_i ^-(x), the ACE advantage is: AACE,i−=A^i−⋅(1+α⋅Softplus(ci))A_ACE,i^-= A_i^-· (1+α·Softplus(c_i) ) (5) where A^i−=(ri−μ^x)/(σ^x+ϵ) A_i^-=(r_i- μ_x)/( σ_x+ε) is the standard GRPO advantage for incorrect rollouts and α≥0α≥ 0 is a hyperparameter controlling the correction strength. Since A^i−<0 A_i^-<0 and (1+α⋅Softplus(ci))≥1(1+α·Softplus(c_i))≥ 1, ACE strictly amplifies the magnitude of the penalty. For correct rollouts yi∈+(x)y_i ^+(x), we retain the standard GRPO advantage: AACE,i+=A^i=ri−μ^xσ^x+ϵA_ACE,i^+= A_i= r_i- μ_x σ_x+ε (6) Design rationale. The Softplus function Softplus(z)=log(1+ez)Softplus(z)= (1+e^z) is chosen for three properties: 1. Asymptotic behavior. When ci≫0c_i 0 (overconfident), Softplus(ci)≈ciSoftplus(c_i)≈ c_i: penalty scales linearly with the log-confidence ratio. When ci≪0c_i 0 (self-correcting), Softplus(ci)≈eci→0Softplus(c_i)≈ e^c_i→ 0: penalty converges to the base GRPO advantage A^i− A_i^-. 2. Smoothness. Unlike max(0,ci) (0,c_i) (which has a non-differentiable kink at 0), Softplus is infinitely differentiable everywhere, ensuring smooth gradient flow. 3. Monotonicity. SoftplusSoftplus is strictly increasing, so more confident errors always receive strictly larger penalties, consistent with our theoretical motivation. Comparison to uniform penalization. To illustrate the effect of ACE, consider the binary reward case where rollouts receive ri∈0,1r_i∈\0,1\. Under standard GRPO, all incorrect rollouts (ri=0r_i=0) share the same advantage A^−=−p^x/(σ^x+ϵ) A^-=- p_x/( σ_x+ε), where p^x p_x is the empirical pass rate and σ^x=p^x(1−p^x) σ_x= p_x(1- p_x). Difficulty-adaptive scaling. Since ACE multiplies the standard GRPO advantage A^i− A_i^- by (1+α⋅Softplus(ci))(1+α·Softplus(c_i)), it naturally inherits GRPO’s difficulty-dependent scaling: easy prompts (high pass rate) produce larger |A^i−|| A_i^-|, so errors on easy problems are penalized more heavily. The confidence modulation then provides additional per-rollout differentiation within each difficulty level. Penalty differentiation. Under ACE: AACE,i−=A^i−⋅(1+αlog(1+eci))⟹|AACE,i−| is strictly increasing in ciA_ACE,i^-= A_i^-· (1+α (1+e^c_i) ) |A_ACE,i^-| is strictly increasing in c_i (7) Therefore, within the same group, an overconfident error (ci=2c_i=2) receives a penalty |A^−|⋅(1+α⋅2.13)| A^-|·(1+α· 2.13) while an exploratory error (ci=0c_i=0) receives |A^−|⋅(1+α⋅0.69)| A^-|·(1+α· 0.69), and a self-correcting error (ci=−3c_i=-3) receives |A^−|⋅(1+α⋅0.05)| A^-|·(1+α· 0.05). This provides fine-grained differentiation that is impossible under uniform penalization. The same principle extends to continuous rewards, where ACE differentiates among below-average rollouts based on their confidence scores. ACE in One Sentence Standard RLVR punishes all wrong answers equally. ACE punishes wrong answers the model has learned to be confident in much harder, while leaving natural exploration mistakes alone. 4.3 ACE-GRPO: Integration and Algorithm ACE modifies only the advantage computation for negative samples. Substituting the ACE advantage (Definition 2) into the GRPO objective (Eq. 2), the full ACE-GRPO objective is: ℒACE(θ)=−x∼[1G∑i=1G([ri=1]⋅ℒi++[ri=0]⋅ℒi−)]+βKL(πθ∥πref)L_ACE(θ)=-E_x [ 1G _i=1^G (I[r_i=1]·L_i^++I[r_i=0]·L_i^- ) ]+ _KL( _θ\| _ref) (8) where: ℒi+ _i^+ =min(ρiA^i+,clip(ρi,1−ϵc,1+ϵc)A^i+) = \! ( _i A_i^+,\;clip( _i,1- _c,1+ _c) A_i^+ ) (9) ℒi− _i^- =min(ρiAACE,i−,clip(ρi,1−ϵc,1+ϵc)AACE,i−) = \! ( _iA_ACE,i^-,\;clip( _i,1- _c,1+ _c)A_ACE,i^- ) (10) The positive advantages A^i+ A_i^+ retain the standard GRPO formulation. Practical considerations. In practice, we normalize cic_i by sequence length (c¯i=ci/Ti c_i=c_i/T_i) to ensure comparable penalty magnitudes across rollouts of different lengths. Additional implementation details (sequence-level vs. token-level aggregation, clipping choices) and a PyTorch implementation are provided in Appendix C. The full algorithm is given below. Algorithm 1 ACE-GRPO: Asymmetric Confidence-aware Error Penalty 0: Policy πθ _θ, reference model πref _ref, prompt dataset D, group size G, ACE strength α, clipping ϵc _c, KL coefficient β 1: for each training step do 2: Sample prompt batch x1,…,xB∼\x_1,…,x_B\ 3: for each prompt x in batch do 4: Generate G rollouts y1,…,yG∼πθ(⋅|x)\y_1,…,y_G\ _θ(·|x) 5: Compute rewards ri∈0,1r_i∈\0,1\ via verifier 6: Compute standard group advantages A^i A_i via GRPO 7: for each incorrect rollout yiy_i with ri=0r_i=0 do 8: ci←(∑t=1Tilogπθ(yi(t)|⋅)−logπref(yi(t)|⋅))/Tic_i← ( _t=1^T_i _θ(y_i^(t)|·)- _ref(y_i^(t)|·) )/T_i // Already computed 9: AACE,i−←A^i−⋅(1+α⋅log(1+exp(ci)))A_ACE,i^-← A_i^-·(1+α· (1+ (c_i))) // Amplify uniform advantage 10: end for 11: Compute clipped surrogate loss (Eq. 8) 12: end for 13: Update θ via gradient descent 14: end for 4.4 Relationship to Selective Reverse KL Divergence We now characterize the theoretical relationship between ACE’s additional penalty and a selective regularizer that targets overconfident errors. Crucially, the equivalence is not exact: ACE implements a stop-gradient (reward-shaping) view of the confidence score, which omits a residual term compared to the full regularizer gradient. We state the exact decomposition below. Theorem 1 (Selective Regularization Decomposition). Let ℒstd(θ)L_std(θ) denote the standard policy gradient objective (Eq. 2) with uniform negative advantages A^− A^-, and let ℒACE(θ)L_ACE(θ) denote the objective with ACE advantages (Eq. 5). Define the selective regularizer: ℛsel(θ)=x∼[|A^−(x)|∑y∈−(x)πθ(y|x)⋅Softplus(logπθ(y|x)πref(y|x))]R_sel(θ)=E_x [| A^-(x)| _y ^-(x) _θ(y|x)·Softplus\! ( _θ(y|x) _ref(y|x) ) ] (11) where |A^−(x)|| A^-(x)| is the magnitude of the standard GRPO negative advantage for prompt x. Assume rollouts are sampled on-policy from πθ _θ. Then, in the infinite-sample limit (G→∞G→∞), the α-dependent additional gradient from ACE decomposes exactly as: Δ∇θ=−α∇θℛsel(θ)+αx∼[|A^−(x)|∑y∈−(x)πθ(y|x)σ(c(y))∇θlogπθ(y|x)] _θ\;=\;-α _θR_sel(θ)\;+\;α\,E_x \! [| A^-(x)| _y ^-(x) _θ(y|x)\,σ(c(y))\, _θ _θ(y|x) ] (12) where σ(c)=1/(1+e−c)σ(c)=1/(1+e^-c) is the sigmoid function (i.e., Softplus′(c)Softplus (c)). Equivalently, ACE implements the negative gradient of ℛselR_sel with the confidence modulation treated as a fixed reward signal (stop-gradient on cic_i), plus the residual term ℰ(θ)E(θ): ℰ(θ)=x∼[|A^−(x)|∑y∈−(x)πθ(y|x)σ(c(y))∇θlogπθ(y|x)]E(θ)=E_x \! [| A^-(x)| _y ^-(x) _θ(y|x)\,σ(c(y))\, _θ _θ(y|x) ] (13) Moreover, for overconfident errors where c(y)≫0c(y) 0, Softplus(c)≈cSoftplus(c)≈ c, and the dominant term in ℛselR_sel takes the form of a difficulty-weighted reverse KL divergence restricted to overconfident incorrect trajectories: ℛsel(θ)≈x∼[|A^−(x)|∑y∈−(x)c(y)>0πθ(y|x)⋅logπθ(y|x)πref(y|x)]R_sel(θ)\;≈\;E_x [| A^-(x)| _ subarraycy ^-(x)\\ c(y)>0 subarray _θ(y|x)· _θ(y|x) _ref(y|x) ] (14) The proof is provided in Appendix A. Intuitively, ACE’s stop-gradient treatment of Softplus(ci)Softplus(c_i) captures the dominant selective-regularization component (Term I: confidence-weighted probability suppression), while the residual ℰ(θ)E(θ) corresponds to the through-cic_i gradient (Term I) that the full regularizer would additionally apply. By omitting Term I, ACE implements a tempered version of ℛselR_sel—less aggressive than the full regularizer, but more targeted than standard GRPO. The per-prompt factor |A^−(x)|| A^-(x)| ensures that the selective regularizer inherits the difficulty-adaptive scaling of GRPO. In contrast to the global KL term βKL(πθ∥πref) _KL( _θ\| _ref) which indiscriminately pulls back all deviations, ℛselR_sel is (i) restricted to incorrect outputs (y∈−y ^-), (i) activated primarily by overconfidence (ci>0c_i>0) due to Softplus saturation, (i) independently tunable via α, and (iv) difficulty-adaptive via the |A^−(x)|| A^-(x)| factor. 4.5 Gradient Quality Analysis A natural question is whether ACE’s confidence-dependent reweighting improves or degrades gradient quality. We analyze this in detail in Appendix B and summarize the key results here. First, ACE necessarily increases both the total gradient second moment and the directional variance—unavoidable consequences of additive reweighting where (1+αϕi)>1(1+α _i)>1 for all ϕi>0 _i>0 (Proposition 1). However, this does not prevent quality improvement. We define the gradient quality ratio as Qd=μd2/σd2Q_d= _d^2/ _d^2, measuring the ratio of squared directional signal to directional variance. Under realistic conditions—specifically, when overconfident errors carry gradients aligned with the optimization direction (Cov(ϕi,ui)>0Cov( _i,u_i)>0) and the baseline gradient is noisy (Qdstd<1Q_d^std<1)—we prove that ACE strictly improves gradient quality: QdACE>QdstdQ_d^ACE>Q_d^std (Theorem 2). The key mechanism is that ACE’s selective amplification concentrates extra weight on the most informative gradients, causing the signal to grow faster than the noise along the optimization-relevant direction. Theoretical Insight: Gradient Efficiency ACE converts harmful variance into exploitable signal. By concentrating penalty weight on overconfident errors—which have gradients more aligned with the optimization direction—ACE achieves higher signal-to-noise ratio despite increasing total gradient variance. This is the statistical foundation for ACE’s improved learning efficiency. 5 Experiments: ACE Expands the Reasoning Boundary 5.1 Experimental Setup Models. We fine-tune Qwen2.5-Math-7B (Qwen Team et al., 2024), Qwen3-8B-Base (Qwen Team et al., 2025), and Llama-3.1-8B-Instruct (Grattafiori et al., 2024) using GRPO implemented with VERL (Volcano Engine, 2024). Note that Qwen3-8B-Base is evaluated without enabling the extended thinking mode (i.e., reasoning mode disabled). Llama-3.1-8B-Instruct is included in the main results (Tables 1 and 2) to test cross-family generalization beyond the Qwen model family. For the detailed diagnostic experiments (§5.3–§5.4), ablations (§5.5), and hyperparameter sensitivity (Appendix D), we focus on the two Qwen models because: (i) they serve as the primary experimental subjects and already span two distinct pretraining recipes (math-specialized vs. general-purpose base model), providing sufficient diversity to validate the generality of our findings; and (i) Llama-3.1-8B-Instruct operates in a substantially lower accuracy regime (e.g., near-floor on AIME 2025), which makes fine-grained diagnostics such as overconfident error distributions and entropy dynamics less statistically informative. Training data. We use the DAPO-Math-17K dataset (Yu et al., 2025) as the training prompts. For the GRPO and ACE-GRPO baselines we use standard GRPO (symmetric clipping, with KL penalty); for DAPO and ACE-DAPO we use the full DAPO algorithm (Yu et al., 2025) with Clip-Higher, dynamic sampling, and token-level loss. Evaluation. We evaluate on MATH-500 (Hendrycks et al., 2021) and AIME 2025 using a rule-based math verifier for correctness verification. Metrics. We report Pass@k for k∈1,2,4,8,16,32k∈\1,2,4,8,16,32\ using temperature 0.7 and top-p = 0.95. Pass@k measures the probability that at least one of k samples is correct. We use the unbiased estimator from Chen et al. (2021): Pass@k=x∼[1−(n−ck)(nk)]Pass@k=E_x [1- n-ck nk ] (15) where n is the total samples and c is the number correct. Pass@1 reflects exploitation; large-k reflects exploration and reasoning boundary. Baselines. • Base model: Unmodified pretrained model (upper bound for large-k diversity). • GRPO: Standard Group Relative Policy Optimization (Shao et al., 2024). • DAPO: The full DAPO algorithm (Yu et al., 2025), which uses asymmetric clipping (Clip-Higher), dynamic sampling, and token-level loss, trained on the same DAPO-Math-17K dataset. • ACE-GRPO: Our method (ACE applied to GRPO). • ACE-DAPO: Our method applied on top of DAPO, demonstrating composability with orthogonal diversity-preserving strategies. Hyperparameters. For ACE, we set α=1.0α=1.0 as the default. We use normalized confidence scores c¯i=ci/Ti c_i=c_i/T_i. Full training hyperparameters are provided in Appendix E. Fair comparison. Within each model, we keep the training recipe and budget matched across methods; see Appendix E. 5.2 Main Results: Full Pass@k Spectrum Table 1 reports Pass@k on MATH-500 and Table 2 reports results on AIME 2025. Table 1: Pass@k (%) on MATH-500. We report mean ± 95% confidence interval over 5 independent training runs. Bold = best within each model group; underline = second best. Model @1 @2 @4 @8 @16 @32 Qwen2.5-Math-7B 63.0 76.3 83.2 88.1 91.2 93.5 Qwen2.5-Math-7B + GRPO 73.4 ± 0.8 79.5 ± 0.7 83.2 ± 0.7 86.2 ± 0.5 89.7 ± 0.5 91.3 ± 0.5 Qwen2.5-Math-7B + DAPO 74.5 ± 1.0 80.8 ± 0.9 84.8 ± 0.8 89.5 ± 0.7 93.0 ± 0.7 94.6 ± 0.6 Qwen2.5-Math-7B + ACE-GRPO 74.2 ± 0.7 80.9 ± 0.7 84.5 ± 0.6 88.9 ± 0.5 92.6 ± 0.5 94.3 ± 0.4 Qwen2.5-Math-7B + ACE-DAPO 75.1 ± 0.8 82.4 ± 0.8 86.2 ± 0.6 91.2 ± 0.5 94.7 ± 0.5 96.1 ± 0.5 Qwen3-8B-Base 60.2 72.5 78.8 83.9 87.2 90.6 Qwen3-8B-Base + GRPO 69.4 ± 0.9 75.5 ± 0.9 79.3 ± 0.9 82.5 ± 0.9 86.2 ± 0.8 88.6 ± 0.7 Qwen3-8B-Base + DAPO 70.8 ± 1.0 76.8 ± 0.9 81.1 ± 0.9 84.3 ± 0.8 88.1 ± 0.8 90.4 ± 0.8 Qwen3-8B-Base + ACE-GRPO 70.1 ± 0.7 76.5 ± 0.7 81.1 ± 0.6 84.5 ± 0.6 88.5 ± 0.6 91.1 ± 0.5 Qwen3-8B-Base + ACE-DAPO 71.2 ± 0.9 77.5 ± 0.9 82.3 ± 0.8 85.4 ± 0.8 89.4 ± 0.7 91.6 ± 0.7 Llama-3.1-8B-Instruct 48.1 59.9 67.8 74.8 80.5 84.8 Llama-3.1-8B-Instruct + GRPO 52.9 ± 1.1 60.5 ± 1.0 67.3 ± 0.9 71.8 ± 0.9 75.5 ± 0.9 79.3 ± 0.8 Llama-3.1-8B-Instruct + DAPO 54.3 ± 1.0 61.8 ± 1.0 68.9 ± 1.0 72.9 ± 1.0 76.8 ± 0.9 80.4 ± 0.9 Llama-3.1-8B-Instruct + ACE-GRPO 54.1 ± 1.1 62.2 ± 1.1 69.1 ± 1.0 73.5 ± 1.0 76.8 ± 0.9 81.5 ± 0.9 Llama-3.1-8B-Instruct + ACE-DAPO 55.4 ± 1.1 62.8 ± 1.1 70.2 ± 1.0 74.1 ± 1.0 77.9 ± 0.9 82.1 ± 0.9 Table 2: Pass@k (%) on AIME 2025. AIME 2025 contains 30 problems; we report point estimates as confidence intervals are dominated by test-set size rather than training variance. Bold = best within each model group; underline = second best. Model 1 2 4 8 16 32 Qwen2.5-Math-7B 6.3 9.9 13.8 17.5 21.9 26.7 Qwen2.5-Math-7B + GRPO 10.5 14.9 19.7 23.9 28.6 33.7 Qwen2.5-Math-7B + DAPO 11.5 16.7 22.5 27.5 31.8 37.1 Qwen2.5-Math-7B + ACE-GRPO 11.2 16.0 21.2 26.1 30.6 36.4 Qwen2.5-Math-7B + ACE-DAPO 11.7 17.4 23.8 28.5 33.1 38.6 Qwen3-8B-Base 5.1 9.2 11.6 14.2 17.0 19.6 Qwen3-8B-Base + GRPO 9.7 13.9 17.4 22.5 25.7 29.8 Qwen3-8B-Base + DAPO 11.1 15.7 19.9 25.2 28.5 33.1 Qwen3-8B-Base + ACE-GRPO 10.5 15.5 19.6 24.7 27.9 32.4 Qwen3-8B-Base + ACE-DAPO 11.2 16.9 21.2 26.3 29.9 34.4 Llama-3.1-8B-Instruct 0.2 0.7 1.2 3.2 7.1 10.8 Llama-3.1-8B-Instruct + GRPO 0.2 0.3 0.5 2.1 3.0 7.0 Llama-3.1-8B-Instruct + DAPO 0.3 0.3 0.6 1.9 2.8 6.3 Llama-3.1-8B-Instruct + ACE-GRPO 0.3 0.3 0.5 2.2 3.9 8.2 Llama-3.1-8B-Instruct + ACE-DAPO 0.2 0.3 0.6 2.0 3.2 7.1 Key findings (Qwen2.5-Math-7B). As shown in Figure 2, ACE consistently improves larger-k metrics while maintaining comparable Pass@1. On MATH-500, ACE-GRPO improves Pass@32 from 91.3% to 94.3% (+3.0p) over GRPO; ACE-DAPO further pushes Pass@32 to 96.1% (+1.5p over DAPO’s 94.6%). On AIME 2025, ACE-GRPO improves Pass@32 from 33.7% to 36.4% (+2.7p); ACE-DAPO reaches 38.6% (+1.5p over DAPO’s 37.1%). Notably, ACE-DAPO achieves the strongest results across all k, demonstrating that ACE composes effectively with orthogonal diversity-preserving strategies. Key findings (Qwen3-8B-Base). The same pattern holds on a different model family. On MATH-500, ACE-GRPO improves Pass@32 from 88.6% to 91.1% (+2.5p); ACE-DAPO reaches 91.6% (+1.2p over DAPO’s 90.4%). On AIME 2025, ACE-GRPO improves Pass@32 from 29.8% to 32.4% (+2.6p); ACE-DAPO reaches 34.4% (+1.3p over DAPO’s 33.1%). Key findings (Llama-3.1-8B-Instruct). To test cross-family generalization, we evaluate on a non-Qwen model. On MATH-500, ACE-GRPO improves Pass@32 from 79.3% to 81.5% (+2.2p); ACE-DAPO reaches 82.1% (+1.7p over DAPO’s 80.4%). On AIME 2025—where Llama-3.1-8B-Instruct operates near the floor—ACE-GRPO improves Pass@32 from 7.0% to 8.2% (+1.2p), demonstrating that ACE’s mechanism transfers across model families even under low-accuracy regimes. These consistent gains across all three model families confirm the generality of ACE’s mechanism. Interaction with DAPO’s Clip-Higher. A natural observation is that ACE’s marginal gain over DAPO is smaller than over GRPO (e.g., on MATH-500 Qwen2.5-Math-7B Pass@32: +3.0p for ACE-GRPO vs. GRPO, but +1.5p for ACE-DAPO vs. DAPO). This reflects a genuine mechanism overlap: DAPO’s Clip-Higher preserves diversity by limiting how aggressively any incorrect path is suppressed at the token level, which indirectly reduces the overconfident-error pathology that ACE targets. However, DAPO’s protection is indiscriminate—it shields overconfident errors and exploratory errors alike, because token-level clipping cannot distinguish trajectory-level confidence regimes. ACE provides the missing selectivity: it amplifies suppression specifically for errors the model has learned to be confident in, while leaving exploratory errors untouched. The consistent gains of ACE-DAPO over DAPO across all model families and benchmarks indicate that this trajectory-level selectivity captures a dimension of the overconfidence problem that token-level clipping alone cannot resolve. The diminishing marginal returns are expected—both methods partially address the same pathology—but the residual improvement confirms that ACE’s rollout-level discrimination provides value beyond what DAPO’s uniform token-level mechanism achieves. Main Result ACE preserves Pass@1 performance while significantly expanding the reasoning boundary at large k. Across three model families (Qwen2.5-Math-7B, Qwen3-8B-Base, Llama-3.1-8B-Instruct) and two benchmarks (MATH-500 and AIME 2025), ACE-GRPO consistently improves Pass@32 by +2.2–3.0p over GRPO. Moreover, ACE composes with DAPO: ACE-DAPO achieves the best overall results (e.g., 96.1% on MATH-500 Pass@32), improving over DAPO by +1.2–1.7p, confirming that ACE provides complementary correction orthogonal to token-level diversity strategies and generalizes across model families. Figure 2: Performance Comparison across Benchmarks. Pass@k curves for all five methods on MATH-500 (left column) and AIME 2025 (right column) across three model families: Qwen2.5-Math-7B (top row), Qwen3-8B-Base (middle row), and Llama-3.1-8B-Instruct (bottom row). ACE-GRPO and ACE-DAPO consistently outperform their respective baselines (GRPO and DAPO) across all sampling budgets, model families, and benchmarks, with larger gains at higher k values. ACE-DAPO achieves the best overall performance, confirming that ACE’s rollout-level correction composes with DAPO’s token-level diversity preservation and generalizes across model families. 5.3 Experiment 1: Overconfident Error Dynamics Goal. Quantify the prevalence of overconfident errors during training and demonstrate that ACE effectively reduces them. Design. Track the distribution of cic_i among incorrect rollouts throughout training for both standard GRPO and ACE-GRPO on the two Qwen models.111We omit Llama-3.1-8B-Instruct from the diagnostic experiments as its lower baseline accuracy yields fewer correct rollouts per group, making the confidence shift statistics noisier and less informative. The main results in Tables 1–2 confirm that ACE’s gains transfer to Llama. At checkpoints every 25 training steps, generate 32 rollouts per prompt on a held-out set and record cic_i for all incorrect rollouts. Metrics. • Overconfident error fraction: OEF(t)=|yi∈−:ci>0|/|−|OEF(t)=|\y_i ^-:c_i>0\|/|Y^-| at step t. • Mean overconfidence magnitude: [ci∣ci>0,ri=0]E[c_i c_i>0,r_i=0] at step t. • Token-level entropy: Average per-token entropy of the policy. Results. The core claim of ACE is that standard GRPO allows incorrect rollouts to become increasingly overconfident during training, and that ACE’s asymmetric penalty should counteract this pathology. To test this, we track two complementary diagnostics at every checkpoint (Figure 3): (i) the overconfident error fraction (OEF), which measures the proportion of incorrect rollouts whose confidence has grown relative to the reference policy (ci>0c_i>0), and (i) the mean overconfidence magnitude among those overconfident errors, which captures the severity of the problem. Throughout training, ACE-GRPO maintains a lower OEF and a lower mean overconfidence magnitude than standard GRPO at every recorded checkpoint, indicating that ACE consistently suppresses both the prevalence and the severity of high-confidence incorrect rollouts. Figure 3: Overconfident Error Dynamics. Left: Overconfident error fraction (OEF) over training. Right: Mean overconfidence magnitude for ci>0c_i>0 errors. ACE-GRPO effectively suppresses both metrics compared to standard GRPO. 5.4 Experiment 2: Entropy Dynamics Goal. Verify that ACE preserves generation diversity by tracking entropy throughout training, and establish the connection between entropy and Pass@k performance. Design. Over the first 20 training steps, compute the average per-token entropy of the policy on a held-out subset of DAPO-Math-17K prompts: H(t)=−1|val|∑x∈val1T∑j=1T∑vπθ(v|x,y<j)logπθ(v|x,y<j)H(t)=- 1|D_val| _x _val 1T _j=1^T _v _θ(v|x,y_<j) _θ(v|x,y_<j) (16) where T is the average sequence length and v ranges over the vocabulary. Results. A key concern with aggressive error suppression is that it may cause premature mode collapse, concentrating probability mass on a narrow set of outputs and destroying the diversity needed for high Pass@k at large k. To diagnose this, we track average per-token entropy H(t)H(t) over the early phase of training, where entropy decay is most rapid, for both standard GRPO and ACE-GRPO (Figure 4). Standard GRPO exhibits a sharp entropy drop within the first 20 steps, retaining only a small fraction of its initial entropy. In contrast, ACE-GRPO decays substantially more slowly, preserving a much larger fraction of the initial entropy over the same period. This gap correlates with Pass@k performance at large k: the method that retains more entropy also achieves higher coverage, confirming that ACE’s selective penalty avoids premature mode collapse while still suppressing overconfident errors. Figure 4: Entropy Dynamics. Token-level entropy over the first 20 training steps. Left: On Qwen2.5-Math-7B, ACE-GRPO retains substantially more entropy than standard GRPO, which suffers rapid entropy collapse. Right: On Qwen3-8B-Base, ACE-GRPO maintains more stable entropy, demonstrating consistency across architectures. We report entropy dynamics for the two Qwen models only; Llama-3.1-8B-Instruct is excluded because its lower baseline accuracy makes the entropy signal less directly comparable (see §5 for discussion). 5.5 Ablation: Choice of Modulation Function A natural question is whether the choice of SoftplusSoftplus as the modulation function is important, or whether a simpler alternative such as ReLU(ci)=max(0,ci)ReLU(c_i)= (0,c_i) suffices. We compare the two variants on MATH-500 using Qwen2.5-Math-7B with α=1.0α=1.0 (the ablation uses a single representative model to isolate the effect of the modulation function; the main results in Table 1 confirm that ACE’s gains are consistent across all three model families): • ACE-Softplus (default): AACE,i−=A^i−⋅(1+α⋅Softplus(ci))A_ACE,i^-= A_i^-·(1+α·Softplus(c_i)) • ACE-ReLU: AACE,i−=A^i−⋅(1+α⋅ReLU(ci))A_ACE,i^-= A_i^-·(1+α·ReLU(c_i)) ReLU completely ignores self-correcting and exploratory errors (ci≤0c_i≤ 0), providing zero modulation in that regime, while Softplus provides a smooth, everywhere-positive modulation that transitions gradually. Table 3: Ablation: modulation function on MATH-500 (Qwen2.5-Math-7B, α=1.0α=1.0). Method @1 @2 @4 @8 @16 @32 GRPO (baseline) 73.4 79.5 83.2 86.2 89.7 91.3 ACE-ReLU 73.2 80.3 83.9 87.6 91.2 93.1 ACE-Softplus (ours) 74.2 80.9 84.5 88.9 92.6 94.3 Analysis. Both ACE-ReLU and ACE-Softplus outperform standard GRPO across all k>1k>1, confirming that confidence-aware modulation—regardless of the specific activation—is beneficial. However, ACE-Softplus consistently outperforms ACE-ReLU, with the gap widening at larger k (+1.2 p at Pass@32). This advantage stems from two properties of Softplus. First, smoothness: ReLU has a non-differentiable kink at ci=0c_i=0, creating a discontinuity in the gradient landscape that can destabilize training, whereas Softplus provides smooth gradient flow everywhere. Second, non-zero modulation near the boundary: ReLU assigns zero modulation to all errors with ci≤0c_i≤ 0, treating them identically to standard GRPO. In contrast, Softplus(0) =ln2≈0.69= 2≈ 0.69, providing a gentle baseline modulation that enables finer differentiation among borderline errors near ci≈0c_i≈ 0—precisely the regime where errors may be transitioning from exploratory to overconfident. These results empirically validate the design rationale in §4.2. 5.6 Analysis: Mechanism Behind Diversity Preservation The experimental results above (§5.3–§5.4) reveal a consistent mechanism: standard GRPO’s uniform penalties allow overconfident errors to form “probability sinks” that crowd out valid reasoning paths—the pathology identified by Yue et al. (2025) as the root cause of RLVR’s narrowing reasoning boundary. ACE’s asymmetric penalties break this cycle: the selective KL term (Theorem 1) acts as entropy regularization restricted to the overconfident region, while leaving exploratory errors (ci≤0c_i≤ 0) untouched. This targeted correction redistributes probability mass to alternative reasoning paths, explaining ACE’s improvements across the full Pass@k spectrum. 6 Limitations and Future Work Dependence on reference model quality. ACE uses πref _ref to define overconfidence. If the reference model is poorly calibrated, the confidence score cic_i may not reliably indicate spurious patterns. Exploring alternatives (e.g., using a moving average of recent checkpoints) is a direction for future work. Binary rewards only. Our current formulation assumes binary rewards (r∈0,1r∈\0,1\). Extending ACE to continuous or partial rewards (e.g., from process reward models) requires redefining what constitutes an “overconfident error” in the presence of graded feedback. Interaction with long CoT. Extended reasoning models (e.g., with >>10K token outputs) may exhibit different confidence shift dynamics. The sequence-length normalization (c¯i=ci/Ti c_i=c_i/T_i) may need refinement for very long chains. 7 Conclusion We identified a previously overlooked pathology in RLVR training: the accumulation of overconfident errors—incorrect reasoning paths that the RL process spuriously reinforces. We proposed ACE, a simple modification to the advantage function that dynamically amplifies penalties for overconfident errors while leaving exploratory errors untouched. Core Contributions (1) We formalize error confidence shift as a new per-rollout diagnostic orthogonal to prompt difficulty, revealing that overconfident errors accumulate during RLVR and drive diversity collapse. (2) ACE’s gradient decomposes into a selective regularizer on overconfident errors plus a tempering residual, providing principled theoretical grounding. (3) ACE improves the full Pass@k spectrum—especially at large k—without sacrificing Pass@1, adding only a single Softplus computation per incorrect rollout. References B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini (2024) Large language monkeys: scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. Cited by: §1. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §1, §5.1. DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, T. Piao, Y. Wu, and Z. Shao (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. S. Devlin and D. Kudenko (2012) Dynamic potential-based reward shaping. In AAMAS, p. 433–440. Cited by: §2. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mber, A. Kambadur, A. Zhernov, A. Rao, A. Lober, A. Ceballos, B. Chen, B. Eriksson, B. Perret, C. Lovejoy, and L. van der Maaten (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1, Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. In NeurIPS, Cited by: §5.1, Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. arXiv preprint arXiv:2305.20050. Cited by: §2. Y. Meng, M. Xia, and D. Chen (2024) SimPO: simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734. Cited by: §3. A. Y. Ng, D. Harada, and S. Russell (1999) Policy invariance under reward transformations: theory and application to reward shaping. In ICML, p. 278–287. Cited by: §2. OpenAI, A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carrel, A. Berber, and A. Glaese (2024) OpenAI o1 system card. Technical report. Cited by: §1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In NeurIPS, Cited by: §2. S. Parashar, S. Gui, X. Li, H. Ling, S. Vemuri, B. Olson, E. Li, Y. Zhang, and J. Caverlee (2025) Curriculum reinforcement learning from easy to hard tasks improves llm reasoning. arXiv preprint arXiv:2506.06632. Cited by: §2. Qwen Team, A. Yang, B. Yang, B. Hui, B. Zheng, C. Li, C. Bao, C. Song, D. Liu, F. Huang, H. Wei, H. Zhang, H. Zhang, J. Tu, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, W. Lin, X. Ren, X. Song, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Zhang, and Z. Liu (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1, Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning. Qwen Team, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2024) Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §5.1, Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023a) Direct preference optimization: your language model is secretly a reward model. In NeurIPS, Cited by: §2. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023b) Direct preference optimization: your language model is secretly a reward model. In NeurIPS, Cited by: §3. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §2. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §3, 2nd item. N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano (2020) Learning to summarize from human feedback. In NeurIPS, Cited by: §2. X. Tang, Y. Zhan, Z. Li, W. X. Zhao, Z. Zhang, Z. Wen, Z. Zhang, and J. Zhou (2025) Rethinking sample polarity in reinforcement learning with verifiable rewards. arXiv preprint arXiv:2512.21625. Cited by: §2. Volcano Engine (2024) VERL: Volcano Engine Reinforcement Learning for LLMs. Note: https://github.com/volcengine/verlOpen-source implementation of the HybridFlow paper Cited by: §5.1, Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, Cited by: §1. X. Wen, Z. Liu, S. Zheng, S. Ye, Z. Wu, Y. Wang, Z. Xu, X. Liang, and J. Li (2025) Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. Cited by: §2. E. Wiewiora, G. W. Cottrell, and C. Elkan (2003) Principled methods for advising reinforcement learning agents. In ICML, p. 792–799. Cited by: §2. R. J. Williams (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3–4), p. 229–256. Cited by: §1, §2. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, C. Hu, A. Ren, Z. Wang, X. Chen, P. Gao, W. Shao, Z. Yang, J. Chen, Y. Qiao, Y. Zheng, W. Luo, and L. Chen (2025) DAPO: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §2, §3, 3rd item, §5.1, Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning. Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Yang, and H. Hu (2025) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. In NeurIPS, Note: Oral Cited by: §1, §2, §3, §5.6. Y. Zeng, Z. Sun, B. Ji, E. Min, H. Cai, S. Wang, D. Yin, H. Zhang, and X. Chen (2025) CurES: from gradient analysis to efficient curriculum learning for rlvr. arXiv preprint arXiv:2510.01037. Cited by: §2. S. Zhang, G. Sun, K. Zhang, X. Guo, and R. Guo (2025) CLPO: curriculum learning meets policy optimization for llm reasoning. arXiv preprint arXiv:2509.25004. Cited by: §2. X. Zhu, M. Xia, Z. Wei, W.-L. Chen, D. Chen, and Y. Meng (2025) The surprising effectiveness of negative reinforcement in llm reasoning. In NeurIPS, Cited by: §2. Appendix A Proof of Theorem 1 (Selective Regularization Decomposition) Proof. The gradient of ℒACEL_ACE differs from ℒstdL_std only in the negative advantage terms. We analyze the α-dependent component. Since AACE,i−=A^i−⋅(1+α⋅Softplus(ci))A_ACE,i^-= A_i^-·(1+α·Softplus(c_i)), the additional gradient relative to standard GRPO is: Δ∇θ=−αx∼[|A^−(x)|G∑yi∈−(x)Softplus(ci)⋅∇θlogπθ(yi|x)] _θ=- _x [ | A^-(x)|G _y_i ^-(x)Softplus(c_i)· _θ _θ(y_i|x) ] (17) Here |A^−(x)|| A^-(x)| is a per-prompt scalar (constant across rollouts within a group) that does not depend on yiy_i. This is the standard REINFORCE form: |A^−(x)|⋅Softplus(ci)| A^-(x)|·Softplus(c_i) acts as a scalar reward multiplying the score function, with cic_i treated as not depending on θ (the “stop-gradient” convention standard in policy gradient methods). As G→∞G→∞, by the law of large numbers: Δ∇θ _θ →−αx∼[|A^−(x)|∑y∈−(x)πθ(y|x)⋅Softplus(c(y))⋅∇θlogπθ(y|x)] →- _x [| A^-(x)| _y ^-(x) _θ(y|x)·Softplus(c(y))· _θ _θ(y|x) ] =−αx∼[|A^−(x)|∑y∈−(x)Softplus(c(y))⋅∇θπθ(y|x)] =- _x [| A^-(x)| _y ^-(x)Softplus(c(y))· _θ _θ(y|x) ] (18) using πθ(y|x)∇θlogπθ(y|x)=∇θπθ(y|x) _θ(y|x) _θ _θ(y|x)= _θ _θ(y|x). Now, the true gradient of ℛsel(θ)R_sel(θ) requires differentiating |A^−(x)|⋅πθ(y|x)⋅Softplus(c(y))| A^-(x)|· _θ(y|x)·Softplus(c(y)) where πθ(y|x) _θ(y|x) and Softplus(c(y))Softplus(c(y)) both depend on θ (since c(y)=logπθ(y|x)−logπref(y|x)c(y)= _θ(y|x)- _ref(y|x)). Since |A^−(x)|| A^-(x)| is a per-prompt scalar, it factors out, and by the product rule: ∇θ[πθ(y|x)⋅Softplus(c)] _θ [ _θ(y|x)·Softplus(c) ] =Softplus(c)⋅∇θπθ(y|x)⏟Term I: captured by ACE+πθ(y|x)⋅σ(c)⋅∇θlogπθ(y|x)⏟Term I: residual = Softplus(c)· _θ _θ(y|x)_Term~I: captured by ACE+ _θ(y|x)·σ(c)· _θ _θ(y|x)_Term~I: residual (19) where σ(c)=Softplus′(c)=1/(1+e−c)σ(c)=Softplus (c)=1/(1+e^-c) and ∇θc=∇θlogπθ(y|x) _θc= _θ _θ(y|x). Multiplying by |A^−(x)|| A^-(x)|, summing over y∈−(x)y ^-(x), and taking expectations, Eq. (18) matches exactly −α⋅|A^−(x)|⋅Term I-α·| A^-(x)|·Term~I. Rearranging: Δ∇θ=−α∇θℛsel+αx∼[|A^−(x)|∑y∈−(x)πθ(y|x)σ(c)∇θlogπθ(y|x)]⏟ℰ(θ) _θ=-α _θR_sel+α E_x \! [| A^-(x)| _y ^-(x) _θ(y|x)\,σ(c)\, _θ _θ(y|x) ]_E(θ) (20) This is an exact identity with no approximation. The residual ℰ(θ)E(θ) arises because ACE treats Softplus(ci)Softplus(c_i) as a fixed reward signal, omitting the gradient through cic_i itself. ∎ Remark 2 (Residual term and contrast with global KL). The residual ℰ(θ)E(θ) is not negligible: for c∈[1,3]c∈[1,3], the ratio σ(c)/Softplus(c)σ(c)/Softplus(c) ranges from 31–56%. ℰE arises because ACE treats Softplus(ci)Softplus(c_i) as a fixed scalar (stop-gradient), omitting the gradient that the full regularizer ℛselR_sel would contribute by differentiating through cic_i (Term I in Eq. 19). This omitted gradient would suppress overconfident errors more aggressively: it drives the parameters to reduce not only πθ(y|x) _θ(y|x) but also the confidence gap c(y)c(y) itself. ACE therefore implements a tempered version of the full regularizer—correcting overconfident errors via the dominant Term I (confidence-weighted probability suppression) while forgoing Term I’s sharper through-c correction. Remark 3 (Why stop-gradient is preferable to the full regularizer). A natural question is whether one should retain Term I to implement the full ∇θℛsel _θR_sel instead of ACE’s tempered version. We argue against this for three reasons. (i) Precedent for detaching θ-dependent signals. Although the reward in vanilla REINFORCE does not depend on θ, modern policy gradient methods routinely stop-gradient through θ-dependent quantities used in the loss: PPO/GRPO detach the advantage A^i A_i (computed from the current policy’s rollouts) from the actor gradient; actor-critic methods detach the value baseline V(s;θ)V(s;θ) even under parameter sharing; and the “old policy” πold _old in importance ratios is always frozen. ACE’s treatment of Softplus(ci)Softplus(c_i) as a detached reward modifier follows the same principle: quantities that diagnose the policy state should inform gradient magnitude, not become optimization targets themselves. (i) Feedback loop. Retaining Term I means the penalty magnitude itself becomes an optimization target: the gradient would simultaneously try to reduce πθ(y|x) _θ(y|x) and reduce ci=log(πθ/πref)c_i= ( _θ/ _ref), creating a second-order feedback that can cause gradient oscillation and training instability. (i) Variance. The gradient quality analysis (Theorem 2) proves that ACE’s stop-gradient version improves the quality ratio QdQ_d under realistic conditions. Adding Term I introduces an additional score-function estimator σ(ci)∇θlogπθσ(c_i) _θ _θ, which increases gradient variance without a guaranteed commensurate signal gain—the sufficient condition for quality improvement (Eq. 49) would need to be re-derived and may no longer hold. Appendix B Gradient Quality Analysis This appendix provides the full formal analysis of ACE’s effect on gradient quality, summarized in §4.5. Assumption 1. For a fixed prompt x with pass rate p, let gi=Ai∇θlogπθ(yi|x)g_i=A_i _θ _θ(y_i|x) be the per-rollout gradient for incorrect rollouts (ri=0r_i=0), and let si=∇θlogπθ(yi|x)s_i= _θ _θ(y_i|x) denote the score function. Let ϕi=Softplus(ci) _i=Softplus(c_i). We assume: 1. Rollouts yiy_i are conditionally independent given x. 2. The signal direction is d^=[si∣ri=0]/∥[si∣ri=0]∥ d=E[s_i r_i=0]/\|E[s_i r_i=0]\|. 3. The directional covariance satisfies Cov(ϕi,(d^⊤si)2∣ri=0)>0Cov( _i,\,( d s_i)^2 r_i=0)>0, i.e., overconfident errors tend to have score functions more aligned with the expected gradient direction. Proposition 1 (Second Moment Increase). For any α>0α>0, ACE strictly increases the mean squared gradient norm of incorrect rollouts: [‖giACE‖2∣ri=0]>[‖gistd‖2∣ri=0]E[\|g_i^ACE\|^2 r_i=0]>E[\|g_i^std\|^2 r_i=0] (21) whenever [ϕi‖si‖2∣ri=0]>0E[ _i\|s_i\|^2 r_i=0]>0 (i.e., errors are not all zero-gradient). This is an unavoidable consequence of the purely additive penalty structure: (1+αϕi)>1(1+α _i)>1 for all ϕi>0 _i>0. Proof. Let a=|A^−(x)|>0a=| A^-(x)|>0 denote the per-prompt base penalty magnitude. Under standard GRPO: gistd=a⋅sig_i^std=a· s_i. Under ACE: giACE=a(1+αϕi)⋅sig_i^ACE=a(1+α _i)· s_i. Then: [‖giACE‖2]−[‖gistd‖2] [\|g_i^ACE\|^2]-E[\|g_i^std\|^2] =a2([(1+αϕi)2‖si‖2]−[‖si‖2]) =a^2 (E[(1+α _i)^2\|s_i\|^2]-E[\|s_i\|^2] ) =a2(2α[ϕi‖si‖2]⏟>0+α2[ϕi2‖si‖2]⏟≥0)>0 =a^2 ( 2α\,E[ _i\|s_i\|^2]_>0+ α^2\,E[ _i^2\|s_i\|^2]_≥ 0 )>0 (22) since a>0a>0, ϕi=Softplus(ci)>0 _i=Softplus(c_i)>0, α>0α>0, and ‖si‖2≥0\|s_i\|^2≥ 0 with [ϕi‖si‖2]>0E[ _i\|s_i\|^2]>0. ∎ Definition 3 (Directional Signal and Variance). For incorrect rollouts, let d^=[si∣ri=0]/∥[si∣ri=0]∥ d=E[s_i r_i=0]/\|E[s_i r_i=0]\| be the unit vector along the expected score function. The directional signal and directional variance of a gradient estimator gi=wi⋅sig_i=w_i· s_i are: μd _d =[d^⊤gi∣ri=0] =E[ d g_i r_i=0] (signal along d d) (23) σd2 _d^2 =Var[d^⊤gi∣ri=0] =Var[ d g_i r_i=0] (noise along d d) (24) The gradient quality ratio is Qd=μd2/σd2Q_d= _d^2/ _d^2. Theorem 2 (Improved Gradient Quality via ACE). Under Assumption 1, let d d be the signal direction. Define the directional projections ui=d^⊤siu_i= d s_i (scalar random variables). Assume: Cov(ϕi,ui2∣ri=0)>0Cov( _i,\,u_i^2 r_i=0)>0 (25) i.e., overconfident errors tend to have score functions more aligned with the expected gradient direction. Then: (a) Directional variance increase. For any α>0α>0, ACE increases the directional variance: Var[d^⊤giACE∣ri=0]>Var[d^⊤gistd∣ri=0]Var[ d g_i^ACE r_i=0]>Var[ d g_i^std r_i=0] (26) whenever Cov(ϕi,ui2)>0Cov( _i,u_i^2)>0 and [ϕi]>0E[ _i]>0. This is an unavoidable consequence of the additive reweighting structure, analogous to the total second-moment increase (Proposition 1). (b) Quality improvement under high-variance conditions. Assume additionally that the initial gradient is noisy relative to the signal, i.e., Var[ui]>([ui])2Var[u_i]>(E[u_i])^2 (equivalently, Qdstd<1Q_d^std<1). Then for sufficiently small α>0α>0, the gradient quality ratio of ACE strictly dominates that of standard GRPO: QdACE>QdstdQ_d^ACE>Q_d^std (27) Consequently, although ACE increases both the signal and the noise (directional variance), the signal grows faster, yielding a net improvement in gradient quality along the optimization-relevant direction. Proof. Consider a fixed prompt x with G rollouts sampled i.i.d. from πθ(⋅|x) _θ(·|x). Let si=∇θlogπθ(yi|x)s_i= _θ _θ(y_i|x) denote the score function for rollout yiy_i, and let ϕi=Softplus(ci) _i=Softplus(c_i). All expectations below are conditioned on ri=0r_i=0. Setup and notation. Let a=|A^−(x)|>0a=| A^-(x)|>0 denote the per-prompt base penalty magnitude. Under standard GRPO: gistd=a⋅sig_i^std=a· s_i. Under ACE: giACE=a(1+αϕi)sig_i^ACE=a(1+α _i)s_i. Since a is a positive scalar constant (per-prompt), it cancels in the gradient quality ratio Qd=μd2/σd2Q_d= _d^2/ _d^2. We therefore analyze the normalized weights wistd=1w_i^std=1 and wiACE=1+αϕiw_i^ACE=1+α _i without loss of generality. Let d^=[si]/‖[si]‖ d=E[s_i]/\|E[s_i]\| be the signal direction, and define the scalar projections ui=d^⊤siu_i= d s_i. Step 1: Directional variance analysis (Part (a)). The directional variance is: Var[wiui]=[wi2ui2]−([wiui])2Var[w_iu_i]=E[w_i^2u_i^2]-(E[w_iu_i])^2 (28) For standard GRPO (wi=1w_i=1): Varstd=[ui2]−([ui])2Var^std=E[u_i^2]-(E[u_i])^2. For ACE (wi=1+αϕiw_i=1+α _i): [wi2ui2] [w_i^2u_i^2] =[ui2]+2α[ϕiui2]+α2[ϕi2ui2] =E[u_i^2]+2α\,E[ _iu_i^2]+α^2\,E[ _i^2u_i^2] (29) ([wiui])2 (E[w_iu_i])^2 =([ui]+α[ϕiui])2 =(E[u_i]+α\,E[ _iu_i])^2 =([ui])2+2α[ui][ϕiui]+α2([ϕiui])2 =(E[u_i])^2+2α\,E[u_i]\,E[ _iu_i]+α^2(E[ _iu_i])^2 (30) Subtracting Eq. (30) from Eq. (29): VarACE ^ACE =Varstd+2α([ϕiui2]−[ui][ϕiui])⏟≡Δ1+O(α2) =Var^std+2α (E[ _iu_i^2]-E[u_i]\,E[ _iu_i] )_≡\, _1+O(α^2) (31) Step 2: Sign of Δ1 _1. Decompose using identities: [ϕiui2] [ _iu_i^2] =Cov(ϕi,ui2)+[ϕi][ui2] =Cov( _i,u_i^2)+E[ _i]\,E[u_i^2] (32) [ϕiui] [ _iu_i] =Cov(ϕi,ui)+[ϕi][ui] =Cov( _i,u_i)+E[ _i]\,E[u_i] (33) Substituting into Δ1 _1: Δ1 _1 =Cov(ϕi,ui2)+[ϕi][ui2]−[ui](Cov(ϕi,ui)+[ϕi][ui]) =Cov( _i,u_i^2)+E[ _i]\,E[u_i^2]-E[u_i] (Cov( _i,u_i)+E[ _i]\,E[u_i] ) =Cov(ϕi,ui2)−[ui]Cov(ϕi,ui)+[ϕi]Var[ui] =Cov( _i,u_i^2)-E[u_i]\,Cov( _i,u_i)+E[ _i]\,Var[u_i] (34) The third term [ϕi]Var[ui]>0E[ _i]\,Var[u_i]>0 always increases variance. In fact, Δ1>0 _1>0 under typical conditions: for the natural Gaussian linear model where ui∼(μ,σ2)u_i (μ,σ^2) and ϕi=a+bui _i=a+bu_i (a>0a>0, b>0b>0), we have Cov(ϕi,ui2)=2bμσ2Cov( _i,u_i^2)=2bμσ^2, Cov(ϕi,ui)=bσ2Cov( _i,u_i)=bσ^2, and [ϕi]=a+bμE[ _i]=a+bμ, giving: Δ1=2bμσ2−μ⋅bσ2+(a+bμ)σ2=2bμσ2+aσ2>0 _1=2bμσ^2-μ· bσ^2+(a+bμ)σ^2=2bμσ^2+aσ^2>0 (35) This confirms that directional variance increases under ACE—an unavoidable cost of additive reweighting. The condition Δ1<0 _1<0 would require: [ui]Cov(ϕi,ui)>Cov(ϕi,ui2)+[ϕi]Var[ui]E[u_i]\,Cov( _i,u_i)>Cov( _i,u_i^2)+E[ _i]\,Var[u_i] (36) which is violated in the Gaussian linear model and is difficult to satisfy in practice. However, as we show next, this does not prevent quality improvement: what matters is that the signal grows faster than the square root of variance. Step 3: Quality improvement (Part (b)). The gradient quality ratio is: Qd=([wiui])2Var[wiui]Q_d= (E[w_iu_i])^2Var[w_iu_i] (37) Since Δ1>0 _1>0 in general (Step 2), the directional variance increases. Nevertheless, the quality ratio can still improve because ACE also increases the signal [wiui]E[w_iu_i]. We now provide the complete derivation. Signal computation. For ACE with wi=1+αϕiw_i=1+α _i: μACE μ^ACE ≜[wiui]=[(1+αϕi)ui]=[ui]+α[ϕiui] [w_iu_i]=E[(1+α _i)u_i]=E[u_i]+α\,E[ _iu_i] =[ui]+α(Cov(ϕi,ui)+[ϕi][ui]) =E[u_i]+α (Cov( _i,u_i)+E[ _i]\,E[u_i] ) =[ui](1+α[ϕi])+αCov(ϕi,ui) =E[u_i](1+α\,E[ _i])+α\,Cov( _i,u_i) (38) Denote μ≜[ui]μ [u_i], ϕ¯≜[ϕi] φ [ _i], and C≜Cov(ϕi,ui)C ( _i,u_i). Then: μACE=μ(1+αϕ¯)+αCμ^ACE=μ(1+α φ)+α C (39) The squared signal is: (μACE)2 (μ^ACE)^2 =(μ(1+αϕ¯)+αC)2 = (μ(1+α φ)+α C )^2 =μ2(1+αϕ¯)2+2αCμ(1+αϕ¯)+α2C2 =μ^2(1+α φ)^2+2α Cμ(1+α φ)+α^2C^2 =μ2+2αμ2ϕ¯+α2μ2ϕ¯2+2αCμ+2α2Cμϕ¯+α2C2 =μ^2+2αμ^2 φ+α^2μ^2 φ^2+2α Cμ+2α^2Cμ φ+α^2C^2 =μ2+2α(μ2ϕ¯+Cμ)+O(α2) =μ^2+2α(μ^2 φ+Cμ)+O(α^2) (40) Variance computation. From Step 2, we have: VarACE=Varstd+2αΔ1+O(α2)Var^ACE=Var^std+2α\, _1+O(α^2) (41) where Varstd=[ui2]−μ2Var^std=E[u_i^2]-μ^2 and Δ1 _1 is given by Eq. (34). Quality ratio expansion. We compute the difference in quality ratios. For standard GRPO: Qdstd=μ2VarstdQ_d^std= μ^2Var^std (42) For ACE, using Eqs. (40) and (41): QdACE Q_d^ACE =(μACE)2VarACE=μ2+2α(μ2ϕ¯+Cμ)+O(α2)Varstd+2αΔ1+O(α2) = (μ^ACE)^2Var^ACE= μ^2+2α(μ^2 φ+Cμ)+O(α^2)Var^std+2α\, _1+O(α^2) (43) Using the first-order Taylor expansion (1+x)−1≈1−x(1+x)^-1≈ 1-x for small x: QdACE Q_d^ACE =μ2+2α(μ2ϕ¯+Cμ)Varstd(1−2αΔ1Varstd)+O(α2) = μ^2+2α(μ^2 φ+Cμ)Var^std (1- 2α\, _1Var^std )+O(α^2) =μ2Varstd+2α(μ2ϕ¯+Cμ)Varstd−2αμ2Δ1(Varstd)2+O(α2) = μ^2Var^std+ 2α(μ^2 φ+Cμ)Var^std- 2αμ^2\, _1(Var^std)^2+O(α^2) =Qdstd+2αVarstd(μ2ϕ¯+Cμ−μ2VarstdΔ1)+O(α2) =Q_d^std+ 2αVar^std (μ^2 φ+Cμ- μ^2Var^std\, _1 )+O(α^2) (44) Therefore: QdACE−Qdstd=2αVarstd(μ2ϕ¯+Cμ−QdstdΔ1)⏟≜Γ+O(α2)Q_d^ACE-Q_d^std= 2αVar^std (μ^2 φ+Cμ-Q_d^std\, _1 )_ \, +O(α^2) (45) Sufficient condition for improvement. Quality improves when Γ>0 >0. Substituting Δ1 _1 from Eq. (34): Γ =μ2ϕ¯+Cμ−Qdstd(Cov(ϕi,ui2)−μC+ϕ¯Varstd) =μ^2 φ+Cμ-Q_d^std (Cov( _i,u_i^2)-μ\,C+ φ\,Var^std ) =μ2ϕ¯+Cμ−QdstdCov(ϕi,ui2)+QdstdμC−Qdstdϕ¯Varstd =μ^2 φ+Cμ-Q_d^std\,Cov( _i,u_i^2)+Q_d^stdμ\,C-Q_d^std φ\,Var^std (46) Using Qdstd=μ2/VarstdQ_d^std=μ^2/Var^std, the last term becomes −μ2ϕ¯-μ^2 φ, which cancels with the first term: Γ =Cμ+QdstdμC−QdstdCov(ϕi,ui2) =Cμ+Q_d^stdμ\,C-Q_d^std\,Cov( _i,u_i^2) =Cμ(1+Qdstd)−QdstdCov(ϕi,ui2) =Cμ(1+Q_d^std)-Q_d^std\,Cov( _i,u_i^2) (47) Under Assumption 1, C=Cov(ϕi,ui)>0C=Cov( _i,u_i)>0 and Cov(ϕi,ui2)>0Cov( _i,u_i^2)>0 (overconfident errors have gradients more aligned with the signal direction). We analyze Γ>0 >0: Γ>0⟺Cμ(1+Qdstd)>QdstdCov(ϕi,ui2) >0 Cμ(1+Q_d^std)>Q_d^std\,Cov( _i,u_i^2) (48) Rearranging: CμCov(ϕi,ui2)>Qdstd1+Qdstd CμCov( _i,u_i^2)> Q_d^std1+Q_d^std (49) The right-hand side is a monotonically increasing function of QdstdQ_d^std that ranges from 0 (when Qdstd=0Q_d^std=0) to 11 (as Qdstd→∞Q_d^std→∞). Therefore, when Qdstd<1Q_d^std<1 (equivalently, Var[ui]>([ui])2Var[u_i]>(E[u_i])^2), we have: Qdstd1+Qdstd<12 Q_d^std1+Q_d^std< 12 (50) Under the Gaussian linear model (ui∼(μ,σ2)u_i (μ,σ^2), ϕi=a+bui _i=a+bu_i), we can verify: C C =Cov(ϕi,ui)=bσ2 =Cov( _i,u_i)=bσ^2 (51) Cov(ϕi,ui2) ( _i,u_i^2) =bCov(ui,ui2)=b⋅2μσ2=2bμσ2 =b\,Cov(u_i,u_i^2)=b· 2μσ^2=2bμσ^2 (52) Thus: CμCov(ϕi,ui2)=bσ2⋅μ2bμσ2=12 CμCov( _i,u_i^2)= bσ^2·μ2bμσ^2= 12 (53) Combined with Eq. (49), when Qdstd<1Q_d^std<1: 12>Qdstd1+Qdstd⟹Γ>0⟹QdACE>Qdstd 12> Q_d^std1+Q_d^std >0 Q_d^ACE>Q_d^std (54) This completes the proof that quality improves under the high-variance condition: Var[ui]>([ui])2⟺Qdstd<1Var[u_i]>(E[u_i])^2 Q_d^std<1 (55) This high-variance regime is the typical operating condition in stochastic policy gradient optimization, where individual rollout gradients are highly variable. Under this condition, the signal growth term dominates the variance growth term, ensuring QdACE>QdstdQ_d^ACE>Q_d^std. Summary. ACE’s confidence-dependent weighting increases both the total gradient second moment (Proposition 1) and the directional variance (Steps 1–2)—both unavoidable consequences of additive reweighting. However, ACE improves the gradient quality ratio (Step 3) under two conditions: (i) overconfident errors carry gradient signal aligned with the optimization direction (Cov(ϕi,ui)>0Cov( _i,u_i)>0), and (i) the initial quality ratio is low (Var[ui]>([ui])2Var[u_i]>(E[u_i])^2). The key mechanism is that ACE’s selective amplification of high-confidence errors concentrates extra weight on the most informative gradients, causing the signal to grow faster than the noise along the optimization-relevant direction. ∎ Appendix C Implementation Details Sequence-level vs. token-level aggregation. While Definition 1 defines cic_i at the sequence level, one can also define a token-level variant ci(t)c_i^(t) and apply ACE per-token. We use the sequence-level aggregation ci=∑tci(t)c_i= _tc_i^(t) in our main experiments to capture “trajectory confidence.” Compute overhead. ACE adds exactly one Softplus computation per incorrect rollout per training step. Given that the bottleneck of RLVR training is rollout generation (model inference), the overhead of ACE is negligible (<0.1%<0.1\% of wall-clock time). PyTorch implementation sketch. def ace_advantage(rewards, log_probs_policy, log_probs_ref, alpha=1.0): """ Args: rewards: (B, G) binary rewards log_probs_policy: (B, G) sequence-level log probs under pi_theta log_probs_ref: (B, G) sequence-level log probs under pi_ref alpha: ACE strength Returns: advantages: (B, G) modified advantages """ # Standard group statistics pass_rate = rewards.mean(dim=-1, keepdim=True) # (B, 1) std = rewards.std(dim=-1, keepdim=True) + 1e-8 std_advantage = (rewards - pass_rate) / std # (B, G) # Confidence score (already available, zero extra compute) c = log_probs_policy - log_probs_ref # (B, G) # Normalize by sequence length c = c / seq_lengths # ACE advantage for negative samples ace_neg = std_advantage * (1.0 + alpha * F.softplus(c)) # (B, G) # Combine: use standard advantage for correct, ACE for incorrect is_correct = (rewards == 1).float() advantages = is_correct * std_advantage + (1 - is_correct) * ace_neg return advantages Compatibility. The above can be dropped into any RLVR training loop that uses GRPO, PPO, or REINFORCE by replacing the advantage computation. No changes are needed to the model architecture, rollout generation, or reward computation. Appendix D Sensitivity to α We vary α∈0,0.1,0.5,1.0,2.0,5.0α∈\0,0.1,0.5,1.0,2.0,5.0\ on MATH-500 using Qwen2.5-Math-7B (α=0α=0 recovers standard GRPO). We conduct the sensitivity analysis on a single model to isolate the effect of α; since the main results (Tables 1–2) demonstrate consistent gains across all three model families at α=1.0α=1.0, we expect the optimal range to transfer. Table 4: Sensitivity to α on MATH-500 (Qwen2.5-Math-7B). α=1.0α=1.0 achieves optimal Pass@32 while preserving Pass@1. α Pass@1 (%) Pass@32 (%) 0.0 (GRPO) 73.4 91.3 0.1 73.5 91.9 0.5 73.8 93.2 1.0 (default) 74.2 94.3 2.0 73.5 93.5 5.0 72.4 92.0 Observations. As shown in Table 4, optimal performance is achieved at α=1.0α=1.0 with Pass@32 = 94.3% (+3.0p over GRPO). Performance remains stable across α∈[0.5,2.0]α∈[0.5,2.0], all outperforming standard GRPO. Pass@1 shows a slight decrease only at larger α values (≥2.0≥ 2.0), reflecting the exploration-exploitation trade-off. We adopt α=1.0α=1.0 as the default for all experiments. Appendix E Training Hyperparameters Table 5 summarizes the training hyperparameters for all three models. Fair comparison (matched recipe within each model). To ensure improvements are attributable to ACE rather than tuning differences, we use the same training recipe and training budget for all methods within a given model (GRPO vs. ACE-GRPO, and DAPO vs. ACE-DAPO). Concretely, for a fixed model we keep the data, verifier, rollout group size G, sampling settings, optimizer, learning rate schedule, batch sizes, clipping/KL coefficients, maximum sequence lengths, and the number of optimizer updates identical across methods; ACE changes only the computation of the negative advantages through Eq. (5) (controlled by α). Hyperparameters may differ across model families due to model-specific stability and context-length constraints, but cross-method comparisons are always performed under matched settings for the same model. Table 5: Training hyperparameters for ACE-GRPO experiments. Hyperparameter Qwen2.5-Math-7B Qwen3-8B-Base Llama-3.1-8B-Instruct Total epochs 10 10 10 Training batch size 2048 1024 1024 Mini-batch size 1024 1024 1024 Micro-batch size per GPU 16 16 16 Learning rate 1×10−51× 10^-5 5×10−75× 10^-7 1×10−61× 10^-6 Optimizer AdamW AdamW AdamW Temperature 1.0 1.0 1.0 Max prompt length 1024 1024 1024 Max response length 3000 8192 4096 Rollout samples per prompt 8 8 8 Validation samples 128 128 128 GPU memory utilization 0.75 0.75 0.75 KL coefficient β 0.001 0.001 0.001 Enable thinking (Qwen3) – False –