Paper deep dive
ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
Song Yu, Li Li
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/31/2026, 2:23:03 AM
Summary
ERPO (Entropy-Regulated Policy Optimization) is a reinforcement learning framework for large reasoning models that addresses the limitations of GRPO by introducing token-level credit assignment. It identifies 'Critical Decision Pivots' (CDPs) using entropy-based metrics and employs entropy-aware gating, bucket-based normalization, and outcome-anchored advantage synthesis to focus optimization on causal reasoning steps, resulting in improved accuracy and efficiency on mathematical benchmarks.
Entities (5)
Relation Signals (3)
ERPO → improvesupon → GRPO
confidence 95% · Extensive experiments on competitive mathematical benchmarks (e.g., MATH, AIME) demonstrate that ERPO significantly outperforms GRPO.
GRPO → isatypeof → RLVR
confidence 90% · the release of DeepSeek-R1 Guo et al. (2025) marks the significant effectiveness of reinforcement learning with verifiable rewards (RLVR) ... powered by Group Relative Policy Optimization
ERPO → utilizes → CDPs
confidence 90% · ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning from verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models. However, standard Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy's trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (ii) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (iii) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks (e.g., MATH, AIME) demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, establishing a new efficiency-accuracy frontier for large reasoning models.
Tags
Links
- Source: https://arxiv.org/abs/2603.28204v1
- Canonical: https://arxiv.org/abs/2603.28204v1
Trouble viewing inline? Open PDF directly →
Full Text
44,029 characters extracted from source content.
Expand or collapse full text
ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models Song Yu, Li Li School of Computer and Information Science Southwest University Chongqing, China 400715 yusong0929@email.swu.edu.cn lily@swu.edu.cn Corresponding Author. Abstract Reinforcement learning from verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models. However, standard Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy’s trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (i) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (i) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks (e.g., MATH, AIME) demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, establishing a new efficiency-accuracy frontier for large reasoning models. 1 Introduction (a) Token-level entropy distribution. (b) Causal impact on accuracy. Figure 1: Token-level entropy distribution and its impact on performance based on Qwen2.5-3B. (a) The LLM exhibits high entropy at high-entropy states (red), indicating that the model is making logical branch decisions, and low entropy at the inference process (yellow), indicating that the model is performing deterministic execution steps. (b) We sampled 50 questions that the model could definitely answer correctly and randomly perturbed the top 5% of high-entropy tokens and the bottom 5% of low-entropy tokens for each sequence. Perturbing these high-entropy hubs resulted in a significant decrease in final accuracy (p<0.001)(p<0.001), confirming their crucial role in inference. With the advancement of large language models (LLMs), reinforcement learning has emerged as a central paradigm for post-training base models in complex agent tasks Ouyang et al. (2022). OpenAI’s o1 OpenAI (2024) demonstrates the capability of solving complex logical problems through Chain-of-Thought (CoT), while the release of DeepSeek-R1 Guo et al. (2025) marks the significant effectiveness of reinforcement learning with verifiable rewards (RLVR) Lambert et al. (2024) in enhancing model reasoning abilities, powered by Group Relative Policy Optimization Shao et al. (2024) (GRPO). GRPO removes the critic network and adopts intra-group relative advantages for gradient updates, substantially reducing computational overhead and improving training efficiency Yue et al. (2025). Despite the remarkable success of GRPO, its underlying assumptions still exhibit limitations. Its assumes that each token shares the sequence-level advantage, implying an inherent premise that each token in the reasoning sequence contributes equally to the final advantage. However, in actual COT reasoning processes, the informational value of tokens demonstrates significant heterogeneity: when models encounter logical branches or critical decision points, token distributions exhibit high entropy characteristics Wang et al. (2025). At such junctures, models require substantial randomness and exploration to identify correct reasoning paths. During deterministic logical expansion phases, token distributions tend to converge. At this stage, redundant derivations often lead to inefficient generation and may even introduce extraneous computational overhead Liu and Xiao (2025); Dai et al. (2025). Existing verifiable reward designs, such as length penalties or KL divergence, are typically token-agnostic. This penalization suppresses exploration when models are hesitating, yet fails to effectively compress redundancy when models are confident. To systematically investigate this heterogeneity, we conducted a multi-stage empirical analysis revealing fundamental differences in lexical importance. We first observed significant heterogeneity in the predictive dynamics of LLMs during complex reasoning processes. By visualizing the lexical entropy of inference trajectories (e.g., on the AMC dataset), we identified specific high entropy token Dou et al. (2025) at which the model exhibits higher predictive entropy. This stands in stark contrast to deterministic computational steps that maintain near-zero entropy in Figure 1(a). We define these high-entropy regions as Critical Decision Pivots (CDPs). Unlike fixed logical keywords, CDPs represent the model’s reasoning frontier where the policy is most uncertain and sensitive to perturbations. We then determined the causal significance of these tokens through a large-scale perturbation study. By performing controlled truncation and random replacement on 2460 samples from the MATH dataset, we revealed a significant sensitivity gap: perturbing high-entropy token leads to a sharp 35.2% drop in inference accuracy Li et al. (2025), while the model remains highly robust to perturbations of low-entropy steps in Figure 1(b). This suggests that high-entropy states are not merely ’confused’ steps but represent irreplaceable structural transitions where any deviation leads to a cascading failure of the reasoning chain. However, standard reinforcement learning methods such as GRPO Shao et al. (2024) ignore this, imparting a uniform advantage throughout the sequence and diluting the crucial reward signal with gradient noise from redundant labels. Our empirical analysis confirms that disrupting these ’hesitation points’ leads to a disproportionate drop in reasoning accuracy, regardless of whether the token is a formal logical operator or a complex structural transition. Based on these insights, we propose Entropy-Regulated Relative Advantage (ERPO), a novel reinforcement learning paradigm designed to refine credit assignment in complex reasoning tasks. The core of ERPO lies in its ability to transform sparse, sequence-level outcomes into dense, token-level signals by integrating Implicit Process Rewards with a dynamic entropy-gating mechanism. ERPO prioritizes exploration at these CDPs to fortify the most fragile links in the reasoning chain. By selectively amplifying the advantage signal at high-entropy tokens while dampening it during deterministic computations, ERPO ensures that the gradient update is concentrated on the most causal steps of the inference chain. In summary, our main contributions are as follows: • We identify CDPs in LLM reasoning via token-level entropy and propose ERPO, which integrates implicit PRM signals and relative position bucketing for fine-grained credit assignment without additional reward training. • We introduce an adaptive gating function that dynamically adjusts reward density based on local predictive uncertainty, balancing exploration at critical junctions with stability in derivation steps. • Experiments on four datasets show ERPO outperforms GRPO, achieving a superior Pareto frontier between reasoning accuracy and sequence length redundancy, while rivaling or even surpassing models tens of times its size in parameter count. 2 Preliminaries We begin by formalizing the probabilistic framework of LLMs Brown et al. (2020), followed by an overview of RLVR. Then, we delineate the GRPO algorithm, providing the necessary background for our proposed method. LLMs. Specifically, given an input prompt x, an LLM πθ _θ sequentially generates a T-token response y=(y1,…,yT)y=(y_1,...,y_T): πθ(|)=∏t=1Tπθ(yt|,<t). _θ(y|x)= _t=1^T _θ(y_t|x,y_<t). (1) RLVR. RLVR Lambert et al. (2024) is a family of reinforcement learning methods that utilize verifiable reward signals rather than learned reward models. Unlike Reinforcement learning from human feedback (RLHF) Ouyang et al. (2022), RLVR employs rule-based objective rewards, such as the correctness of a programming output, the correctness of the final answer to a mathematical problem, or compliance with formatting. These rewards originate from tasks with explicit ground truth verification. Consider a dataset =(x,y)D=\(x,y)\ where x is the prompt and y is the ground truth. The optimization objective in RLVR is to maximize the expected reward: (θ)=(x,y)∼[y^∼πθ(⋅|x)[R(y^,y)]],J(θ)=E_(x,y) [E_ y _θ(·|x)[R( y,y)] ], (2) where R(y^,y)R( y,y) is a verifiable reward function that compares the generated output y y against the ground truth y. GRPO. GRPO Shao et al. (2024) is a more efficient policy optimization algorithm compared with Proximal Policy Optimization (PPO) Schulman et al. (2017), as it estimates advantages through group-based response sampling, eliminating the need for a separate value network. For each prompt x, GRPO samples G responses o1,o2,…,oGo_1,o_2,…,o_G from the policy model πθ _θ, with each response consisting of |oi||o_i| tokens. These responses are evaluated using a reward model or function R(x,oi)R(x,o_i), yielding a reward rir_i for each response. Token-level advantages A^i,t A_i,t are computed through within-group normalization. Specifically, for all tokens in response oio_i, the advantage is set to the normalized reward of that response: A^i,t=ri−mean()std()+δ,∀t∈1,…,|oi|, A_i,t= r_i-mean(r)std(r)+δ, ∀ t∈\1,...,|o_i|\, (3) where =[r1,r2,…,rG]r=[r_1,r_2,...,r_G] is the vector of rewards for all responses in the group, mean(⋅)mean(·) and std(⋅)std(·) denote the mean and standard deviation operations respectively, and δ is a small constant for numerical stability. This normalization provides a relative comparison of responses within the same group, effectively estimating advantages. This assignment assumes an equal contribution of all tokens to the final outcome reward, bypassing the need for per-token value estimation. To prevent the policy from diverging too far from the reference policy, GRPO incorporates KL divergence regularization. Specifically, for each token position, the KL divergence is estimated using a low-variance approximation Schulman (2020): KL[πθ∥πref]=πref(oi,t∣x,oi,<t)πθ(oi,t∣x,oi,<t)−logπref(oi,t∣x,oi,<t)πθ(oi,t∣x,oi,<t)−1,D_KL[ _θ\| _ref]= _ref(o_i,t x,o_i,<t) _θ(o_i,t x,o_i,<t)- _ref(o_i,t x,o_i,<t) _θ(o_i,t x,o_i,<t)-1, (4) This estimator provides a low-variance approximation of the KL divergence while maintaining computational efficiency. The GRPO optimization objective combines a clipped surrogate objective with a KL divergence penalty. After each generation, multiple updates can be performed using the following loss function: ℒGRPO(θ)=−1∑i=1G|oi|∑i=1G∑t=1|oi|[min(ρi,t(θ)A^i,t,clip(ρi,t(θ),1−ϵ,1+ϵ)A^i,t)−βKL[πθ∥πref]]L_GRPO(θ)=- 1 _i=1^G|o_i| _i=1^G _t=1^|o_i| [ ( _i,t(θ) A_i,t,clip( _i,t(θ),1-ε,1+ε) A_i,t )- _KL[ _θ\| _ref] ] (5) where ρi,t(θ)=πθ(oi,t∣x,oi,<t)πθold(oi,t∣x,oi,<t) _i,t(θ)= _θ(o_i,t x,o_i,<t) _ _old(o_i,t x,o_i,<t) is the probability ratio between the current policy and the old policy, clip(⋅,1−ϵ,1+ϵ)clip(·,1-ε,1+ε) constrains the probability ratio to the interval [1−ϵ,1+ϵ][1-ε,1+ε] to prevent excessively large policy updates, ϵε is the clipping hyperparameter, and β controls the strength of KL regularization. This loss function aims to maximize expected rewards while constraining the magnitude of policy updates, ensuring training stability. Through the combination of group-based advantage estimation and KL regularization, GRPO achieves stable policy optimization without requiring a value network, making it particularly suitable for reinforcement learning tasks with verifiable rewards. 3 Method To address the information heterogeneity in reasoning chains, we propose Entropy-Regulated Policy Optimization (ERPO). ERPO transforms the coarse-grained, sequence-level advantage into a dense, token-level signal that prioritizes causal bottlenecks while suppressing redundancy. The algorithm is shown in Algorithm 1. 3.1 Token-level Diagnostic Metrics We first define the diagnostic metrics used to characterize the internal states and progress of the policy πθ _θ during rollout. Uncertainty Estimation. We employ the policy entropy Hi,tH_i,t as a statistical proxy to identify CDPs. Formally, for each token oi,to_i,t, the entropy is defined as: Hi,t=−∑v∈πθ(v∣x,oi,<t)logπθ(v∣x,oi,<t).H_i,t=- _v _θ(v x,o_i,<t) _θ(v x,o_i,<t). (6) This metric quantifies the model’s instantaneous uncertainty, where peak entropy typically signifies a transition between distinct reasoning paths. Implicit Progress Signal. To provide dense feedback during the reasoning process, we derive a token-level signal si,ts_i,t that characterizes the confidence gain relative to the frozen reference policy πref _ref Setlur et al. (2025): si,t=βprogress(logπθ(oi,t∣x,oi,<t)−logπref(oi,t∣x,oi,<t)),s_i,t= _progress ( _θ(o_i,t x,o_i,<t)- _ref(o_i,t x,o_i,<t) ), (7) where βprogress>0 _progress>0 is a hyperparameter scaling the sensitivity to policy deviation. Intuitively, si,ts_i,t acts as an intrinsic motivator that rewards tokens reflecting a refined understanding over the base model. This term enables fine-grained credit assignment by distinguishing the relative contribution of individual tokens within a long CoT sequence, effectively mitigating the sparse reward challenge in RLVR. 3.2 Entropy aware Gating While Hi,tH_i,t provides a raw measure of uncertainty, to effectively leverage this signal for optimization, we introduce an adaptive gating function Wi,tW_i,t that translates local predictive uncertainty into a relative importance weight. We perform intra-group entropy calibration to account for varying task complexities. Specifically, for each token oi,to_i,t in a prompt group G, we compute the gated weight: Wi,t=σ(γ⋅Hi,t−μH,σH,+δ),W_i,t=σ (γ· H_i,t- _H,G _H,G+δ ), (8) where μH, _H,G and σH, _H,G denote the moving mean and standard deviation of entropy within the current group, respectively. σ(⋅)σ(·) is the sigmoid function and γ is a scaling factor. Essentially, Wi,tW_i,t acts as a diagnostic probe that surfaces the logic junctions where multiple reasoning paths diverge, ensuring that the gradient signal is concentrated on these high stakes decision points. 3.3 Bucketing Implicit Advantage As shown in Figure 1(a), the inference phase exhibits temporal heterogeneity. LLMs typically involve different stages, from step planning and logical branching to stepwise derivation and finally numerical verification. The implicit progress values of these steps cannot be directly compared horizontally. To ensure fairness, we propose relative position binning. We first define the relative progress of a token oi,to_i,t as τi,t=t/|oi| _i,t=t/|o_i|, where |oi||o_i| is the sequence length. The reasoning process is then partitioned into K discrete temporal buckets ℬ1,…,ℬKB_1,…,B_K, each representing a synchronized progress window. Within each prompt group G, we perform intra-group bucket normalization to isolate the relative quality of a token from its temporal position: s~i,t=si,t−μk,σk,+δ,∀(i,t)∈ℬk, s_i,t= s_i,t- _k,G _k,G+δ, ∀(i,t) _k, (9) where μk, _k,G and σk, _k,G are the mean and standard deviation of the signals belonging to bucket k across all G sequences in the group. By judging a token’s advantage only against its peers at the same reasoning stage, its filters out task variance and provides a pure measure of token-level contribution, effectively aligning the optimization signal with the model’s multi stage reasoning trajectory. 3.4 Final Advantages Synthesis We synthesize the refined process signals with the global sequence-level advantage. A primary concern in this fusion is ensuring that the process-level feedback does not lead to reward hacking, where the model might optimize for high-confidence tokens that ultimately lead to incorrect answers. We introduce an Outcome Anchoring mechanism, where the sign of the sequence-level advantage acts as a directional guardrail. Specifically, we compute the calibrated process reward Ψi,t _i,t and synthesize it into the final advantage as follows: A^i,tfinal=Norm(A^igroup+η⋅Ψi,t),withΨi,t=σtarget⋅Wi,t⋅sgn(A^i)⋅s~i,tstd(Ψ^active)+δ, A_i,t^final=Norm_G ( A_i^group+η· _i,t ), \;\; _i,t= _target· W_i,t·sgn( A_i)· s_i,tstd( _active)+δ, (10) where Norm(⋅)Norm_G(·) denotes the intra-group Z-score normalization to preserve the relative optimization property of GRPO. σtarget _target and η are parameters, respectively. This strategy ensures that A^i,tfinal A_i,t^final is unbiased and well calibrated, ensuring that even highly confident pivotal tokens are penalized if they ultimately lead to a fallacious conclusion. 1 Require: Initial policy πθ _θ, reference policy πref _ref; learning rate αlr _lr; group size G; number of buckets K; gating factor γ; scaling constants βprogress,η,σtarget _progress,η, _target. Ensure: Optimized policy parameters θ. 2 3for iteration n=1,…,Nn=1,…,N do 4 Sample prompt x∼x and generate G responses o1,…,oG∼πθ(⋅∣x)\o_1,…,o_G\ _θ(· x); 5 Compute verifiable rewards r1,…,rG\r_1,…,r_G\ and group advantages A^igroup A_i^group via standard GRPO; 6 /* Token-level Diagnostic Metrics */ 7 for each response oio_i and token t do 8 Hi,t←−∑v∈πθ(v∣x,oi,<t)logπθ(v∣x,oi,<t)H_i,t←- _v _θ(v x,o_i,<t) _θ(v x,o_i,<t); 9 si,t←βprogress(logπθ(oi,t∣⋅)−logπref(oi,t∣⋅))s_i,t← _progress ( _θ(o_i,t ·)- _ref(o_i,t ·) ); 10 τi,t←t/|oi|⟹Assign to bucket ℬk where k=⌊τi,t⋅K⌋ _i,t← t/|o_i| to bucket B_k where k= _i,t· K ; 11 12 /* Hierarchical Normalization & Gating */ 13 Compute Wi,t←Sigmoid(γ⋅Norm(Hi,t))W_i,t (γ·Norm_G(H_i,t)); 14 for each bucket k∈0,…,K−1k∈\0,…,K-1\ do s~i,t←Normℬk(si,t) s_i,t _B_k(s_i,t) /* Intra-bucket normalization per group */ 15 ; 16 17 /* Outcome-Anchored Synthesis */ 18 Ψi,t←Wi,t⋅sgn(A^igroup)⋅s~i,t _i,t← W_i,t·sgn( A_i^group)· s_i,t; 19 Rescale Ψi,t _i,t such that std(Ψactive)=σtargetstd( _active)= _target; 20 A^i,tfinal←Norm(A^igroup+η⋅Ψi,t) A_i,t^final _G ( A_i^group+η· _i,t ); 21 22 Update θ using ∇θJGRPO _θJ_GRPO with token-level advantages A^i,tfinal\ A_i,t^final\; 23 Algorithm 1 ERPO: Entropy-Regulated Policy Optimization 3.5 Theoretical Analysis We analyze whether ERPO preserves the theoretical guarantees of policy gradient methods and prevents pathological behaviors such as reward hacking. 3.5.1 Equivalence to Entropy-Weighted Regularization Statement. The ERPO gradient corresponds to optimizing the original GRPO objective augmented with a dynamic, entropy-weighted relative entropy regularization term. Formally, there exists a potential function F(πθ)F( _θ) such that: ∇θERPO(θ)=∇θGRPO(θ)+η⋅∇θF(πθ). _θJ_ERPO(θ)= _θJ_GRPO(θ)+η· _θF( _θ). (11) Proof. The policy gradient for the combined advantage A^i,tERPO A_i,t^ERPO is given by: ∇θERPO=∇θGRPO+η⋅[∑i,t∇θlogπθ(oi,t)⋅Ψi,t]. _θJ_ERPO= _θJ_GRPO+η·E [ _i,t _θ _θ(o_i,t)· _i,t ]. (12) Substituting the definition of Ψi,t _i,t and si,ts_i,t, and treating the scaling factors Λi,t=σtarget⋅Wi,t⋅sgn(A^i)std(Ψactive)+δ _i,t= _target· W_i,t·sgn( A_i)std( _active)+δ as constants during the gradient step, the additional term becomes: Δ∇θ=ηβprogress⋅[∑i,tΛi,t⋅∇θlogπθ(oi,t)(logπθ(oi,t)−logπref(oi,t))]. _θ=η _progress·E [ _i,t _i,t· _θ _θ(o_i,t) ( _θ(o_i,t)- _ref(o_i,t) ) ]. (13) Using the identity ∇θlogπθ⋅logπθ=12∇θ(logπθ)2 _θ _θ· _θ= 12 _θ( _θ)^2, we have: Δ∇θ=ηβprogress⋅[∑i,tΛi,t⋅∇θ(12(logπθ(oi,t))2−logπref(oi,t)logπθ(oi,t))]. _θ=η _progress·E [ _i,t _i,t· _θ ( 12( _θ(o_i,t))^2- _ref(o_i,t) _θ(o_i,t) ) ]. (14) This allows us to define the potential function F(πθ)F( _θ): F(πθ)=ηβprogress2∑i,tΛi,t(logπθ(oi,t)πref(oi,t))2+C,F( _θ)= η _progress2 _i,t _i,t ( _θ(o_i,t) _ref(o_i,t) )^2+C, (15) where C is a constant. This result demonstrates that ERPO is not merely repeating the KL penalty but is minimizing a weighted squared log ratio, which acts as a precision-guided regularizer. The gating weight Wi,tW_i,t ensures that this regularization is most active at critical decision pivots, transforming a passive constraint into an active, entropy-aware guidance signal. 3.5.2 Stability via Zero-Sum Normalization Statement. The intra-group normalization of A^i,tfinal A_i,t^final prevents reward hacking by ensuring that the total advantage across a prompt group remains conserved. Proof. ERPO applies Norm(⋅)Norm_G(·) to the combined advantages within each group G. The final advantages A^i,tfinal A_i,t^final satisfy: ∑(i,t)∈A^i,tfinal=0,Var(i,t)∈(A^i,tfinal)=1. _(i,t) A_i,t^final=0, _(i,t) ( A_i,t^final)=1. (16) Consider an attempt by the model to inflate the advantage by artificially increasing entropy. Due to the zero-sum constraint, any increase in advantage at one token must be exactly offset by a decrease elsewhere in the same group. This creates a zero-sum game for the gradient signal. The model cannot achieve an unbounded global advantage; it can only reallocate credit among tokens, forcing the optimization to prioritize the most causally significant steps relative to their group peers. 3.5.3 Causality and Consistency Statement. ERPO satisfies the requirements of the Policy Gradient Theorem as its advantage function does not rely on future information relative to the token being optimized. Proof. The components si,ts_i,t and Hi,tH_i,t depend strictly on the current token oi,to_i,t and context oi,<to_i,<t. The group-level statistics and bucket statistics are computed using the trajectories generated by the old policy πθold _ _old. In the gradient computation, these statistics are treated as constants. Therefore, the advantage function A^i,tfinal A_i,t^final does not involve future actions oi,t′>to_i,t >t of the current policy being optimized, preserving the causality of the trajectory. 4 Experiments 4.1 Experimental Setup Table 1: Performance evaluation of Qwen2.5 models across reasoning benchmarks. Acc (%) and Fmt (%) denote sample accuracy and boxed rate. Benchmarks are ordered by difficulty: AMC23, Minerva, AIME24, and AIME25. Bold values indicate the best performance within each parameter scale. AMC 23 Minerva AIME 24 AIME 25 Average Model Acc Fmt Acc Fmt Acc Fmt Acc Fmt Acc Fmt Commercial Baselines DeepSeek-R1-671B-0528 33.91 33.91 11.41 34.38 13.54 13.54 11.04 11.04 17.48 23.22 Qwen3-235B-A22B-Instr. 47.81 55.00 17.66 92.34 24.58 29.58 16.88 20.21 26.73 49.28 1.5B Scale Qwen2.5-1.5BBase_Base 0.78 27.97 0.31 27.97 0.21 28.33 0.00 26.67 0.33 27.74 Qwen2.5-1.5BSFT_SFT 8.13 98.28 1.41 84.84 0.83 95.42 0.42 93.13 2.70 92.92 Qwen2.5-1.5BGRPO_GRPO 25.31 95.31 4.06 93.44 3.54 87.08 2.08 94.17 8.75 92.50 Qwen2.5-1.5BERPO_ERPO 27.19 95.31 4.22 97.81 3.75 90.00 2.08 94.17 9.31 94.32 3B Scale Qwen2.5-3BBase_Base 14.84 72.66 3.13 62.66 2.08 75.83 1.46 78.96 5.38 72.53 Qwen2.5-3BSFT_SFT 10.63 99.38 2.03 74.69 0.83 95.42 0.83 95.63 3.58 91.28 Qwen2.5-3BGRPO_GRPO 32.81 97.03 7.34 97.03 5.21 87.71 3.33 93.96 12.17 93.93 Qwen2.5-3BERPO_ERPO 37.50 96.72 8.91 98.91 7.08 92.50 2.92 97.92 14.10 96.51 7B Scale Qwen2.5-7BBase_Base 23.75 86.09 4.38 77.03 3.54 82.50 1.25 83.54 8.23 82.29 Qwen2.5-7BSFT_SFT 17.03 96.88 4.84 71.88 1.46 88.75 1.46 94.79 6.20 88.08 Qwen2.5-7BGRPO_GRPO 47.50 97.19 12.50 98.91 11.25 93.75 6.46 94.17 19.43 96.01 Qwen2.5-7BERPO_ERPO 49.53 98.13 13.28 98.75 12.92 94.79 7.08 96.25 20.70 96.98 Datasets and Evaluation Benchmarks. For reinforcement learning, we utilized the MATH dataset Hendrycks et al. (2021), specifically filtering problems with difficulty levels 3 to 5. This curated subset ensures the model is exposed to high entropy reasoning tasks. To evaluate generalization and peak reasoning capabilities, we conducted testing on four prestigious competitive mathematics benchmarks: AMC23 Zhang and Math-AI (2023), AIME24 Zhang and Math-AI (2024), AIME25 Zhang and Math-AI (2025), and the Minerva Lewkowycz et al. (2022). Model Configurations. We implemented ERPO across three scales of the Qwen2.5 series Team (2025): 1.5B, 3B, and 7B. To ensure training efficiency, we employed Low-Rank Adaptation (LoRA) Hu et al. (2022) with a rank r=32r=32 and αlora=64 _lora=64, targeting all linear layers to provide sufficient capacity for complex reasoning updates. Training Specifications. The models were trained for a single epoch using the TRL framework von Werra et al. (2020), with the maximum sequence length set to 20482048 to accommodate CoT derivations. We utilized a global batch size of 16, with G=8G=8 rollouts per prompt. The learning rate was fixed at 5×10−65× 10^-6 with a 0.10.1 warmup ratio and a cosine decay schedule. Optimization was performed using AdamW with a weight decay of 0.0010.001, integrated with DeepSpeed for memory efficiency. Detailed settings are provided in Appendix. 4.2 Baselines To evaluate the effectiveness of ERPO, we benchmark our approach against the following baseline configurations: Base Model. The model without any fine-tuning, serving as the fundamental performance lower bound. SFT Model. A supervised fine-tuned version of the base model using the same MATH training set, representing the gain from standard cross entropy loss on reasoning chains. Instruct Model. The official models, which have undergone large-scale general instruction tuning and alignment. GRPO. The most critical baseline to isolate the specific contributions of our proposed entropy-gated credit assignment and temporal bucketing under identical rollout and reward configurations. 4.3 Overall Performance and Benchmarking Comparison with Large-Scale Models. According to the results in Table 1, ERPO achieves a dominant position across various parameter scales. Specifically, our 7B model reaches a level of performance that surpasses much larger commercial models, including DeepSeek-R1-0528 (671B) and Qwen3-235B-A22B-Instruct. This indicates that token-level credit assignment effectively compensates for smaller parameter counts by maximizing the reasoning potential of each layer. Furthermore, the format consistency rate (Fmt) also shows a steady improvement across all benchmarks. Table 2: Comprehensive pass@kpass@k (k∈2,4,8,16k∈\2,4,8,16\) performance evaluation. Bold values indicate the best performance within each parameter scale (1.5B, 3B, and 7B). AMC 23 Minerva AIME 24 AIME 25 Model @2 @4 @8 @16 @2 @4 @8 @16 @2 @4 @8 @16 @2 @4 @8 @16 Commercial Baselines DeepSeek-R1-671B-0528 47.62 58.72 66.19 72.50 16.35 20.94 23.78 25.00 21.03 28.07 32.14 33.33 17.33 23.68 28.82 33.33 Qwen3-235B-A22B-Instr. 54.35 59.66 63.00 65.00 19.50 21.44 23.40 25.00 27.03 28.32 30.00 33.33 20.03 22.21 24.22 26.67 1.5B Scale Qwen2.5-1.5BBase_Base 1.56 3.12 6.25 12.50 0.62 1.25 2.50 5.00 0.42 0.83 1.67 3.33 0.00 0.00 0.00 0.00 Qwen2.5-1.5BSFT_SFT 14.90 25.41 39.17 55.00 2.48 3.96 5.66 7.50 1.67 3.33 6.67 13.33 0.83 1.67 3.33 6.67 Qwen2.5-1.5BGRPO_GRPO 36.67 49.09 62.92 75.00 6.92 10.40 13.62 17.50 6.19 10.06 15.56 23.33 4.11 8.00 15.11 26.67 Qwen2.5-1.5BERPO_ERPO 38.25 49.62 61.46 72.50 6.21 8.53 11.92 17.50 6.81 11.40 17.08 23.33 3.72 6.12 9.21 13.33 3B Scale Qwen2.5-3BBase_Base 25.29 38.60 52.48 67.50 5.44 8.62 12.24 15.00 3.92 6.95 11.21 16.67 2.72 4.76 7.43 10.00 Qwen2.5-3BSFT_SFT 18.31 28.64 40.35 52.50 3.77 6.56 10.38 15.00 1.64 3.17 5.89 10.00 1.67 3.33 6.67 13.33 Qwen2.5-3BGRPO_GRPO 46.00 59.57 71.73 80.00 10.94 14.58 17.56 20.00 8.58 12.81 18.44 26.67 6.31 11.40 19.29 30.00 Qwen2.5-3BERPO_ERPO 49.79 61.91 71.11 77.50 13.25 17.73 22.57 27.50 11.22 16.43 23.40 33.33 5.56 10.07 16.56 23.33 7B Scale Qwen2.5-7BBase_Base 38.15 54.06 67.42 77.50 7.75 12.46 17.60 22.50 6.19 10.06 15.56 23.33 2.50 5.00 10.00 20.00 Qwen2.5-7BSFT_SFT 28.56 43.31 59.20 70.00 8.17 12.31 16.47 20.00 2.86 5.50 10.11 16.67 2.83 5.36 9.67 16.67 Qwen2.5-7BGRPO_GRPO 60.81 71.62 79.25 82.50 17.83 24.15 30.54 35.00 16.28 21.49 27.09 33.33 10.97 16.93 24.29 33.33 Qwen2.5-7BERPO_ERPO 63.40 75.22 84.21 90.00 17.92 23.10 28.29 32.50 17.89 22.45 27.38 33.33 12.08 18.60 26.44 36.67 The Generalization Gap in SFT. We observe that certain SFT models perform worse than their corresponding base models after fine-tuning. This phenomenon stems from the fact that our SFT phase is conducted on the relatively simple MATH dataset, while the evaluation benchmarks like AIME25 are significantly more difficult. SFT tends to force the model to mimic specific expert solution paths, which limits its ability to generalize to out-of-distribution problems. In contrast, ERPO encourages autonomous exploration, allowing the model to develop robust internal logic rather than simple pattern matching. 4.4 Multi-Sample Scaling Analysis Performance Gains in Pass@k. As demonstrated in Table 2, the number of solved problems increases for all models as the sampling count k rises. However, the improvement is significantly more pronounced for models trained with reinforcement learning (GRPO and ERPO). By optimizing the reasoning process, ERPO expands the effective search space of the policy. This allows our smaller models to partially exceed the performance of models with ten times more parameters when evaluated under high-throughput sampling conditions. 4.5 Scale-Dependent Training Dynamics Convergence and Performance on 3B and 7B Scales. The training curves in Figure 2 show that ERPO achieves higher final accuracy and faster convergence than the GRPO baseline on the 3B and 7B scales. The entropy-regulated mechanism immediately provides beneficial guidance to these larger models, which possess more stable logical representations. Initial Exploration Lag in 1.5B Models. On the 1.5B scale, we observe that ERPO initially underperforms GRPO before rising sharply in the later stages. Our analysis suggests that the entropy-regulation mechanism introduces significant perturbations to smaller models, which have less stable latent states. This causes the model to spend more time on exploration in the early phase. However, once correct reasoning paths are captured, the performance increases rapidly, eventually surpassing GRPO as seen in the late steps of Figure 2(a). This explains why ERPO might show lower metrics on certain difficult tasks like AIME25 within 1 epoch, and we anticipate that extended training would yield even greater gains. (a) Training dynamics of 1.5B models. (b) Training dynamics of 3B models. (c) Training dynamics of 7B models. Figure 2: Comparison of training efficiency and generalization performance between GRPO (Baseline) and ERPO (Ours) across three model scales (1.5B, 3B, 7B). Each row presents the sample accuracy (%) on AMC23, Minerva, AIME24, and AIME25 benchmarks, smoothed with EMA (α=0.2α=0.2). 4.6 Policy Stability and Information Metrics Entropy and Reward Evolution. Figure 3 visualizes the core training metrics. Although ERPO’s reward is initially lower than GRPO’s due to the emphasis on exploration, it eventually exceeds the baseline. Crucially, ERPO maintains a healthy entropy level between 0.2 and 0.4 in the late stages, whereas GRPO suffers from entropy collapse. This preservation of diversity is key to preventing mode collapse. Gradient and Divergence Stability. The gradient norms of ERPO remain stable throughout the process, showing no signs of gradient explosion. While our KL divergence is slightly higher than that of standard GRPO—reflecting a more aggressive departure from the base model—it remains well-controlled without the sharp spikes observed in the baseline. This suggests that ERPO maintains a more consistent optimization trajectory. 4.7 Inference Efficiency and Computational Overhead Reasoning Conciseness. A major advantage of ERPO is shown in Figure 4(a), where our models achieve higher accuracy while maintaining shorter sequence lengths. By encouraging exploration at high entropy points and suppressing redundancy at low-entropy steps, ERPO prevents the model from becoming overly verbose in straightforward reasoning steps. Evolution of Generation Length. Figure 4(c) shows that while our generation length is slightly higher during the initial exploration phase, it drops significantly below the GRPO length in the later stages while maintaining superior accuracy. This characteristic improves the overall quality of the reasoning chains. Figure 3: Training dynamics of ERPO vs. GRPO. We visualize the (a) Reward, (b) Entropy, (c) Grad Norm, and (d) KL Divergence. Note that the Entropy axis uses a symlog scale to highlight the significant difference in the late training stage (0.40.4 vs. 0.050.05), demonstrating that ERPO effectively prevents mode collapse. All curves are smoothed with EMA (α=0.12α=0.12) while raw data is shown in light colors. (a) Best checkpoint token efficiency. (b) Total training time. (c) Evolutionary trends of average generation tokens during training (EMA, α=0.12α=0.12). Figure 4: Comprehensive efficiency and training dynamics analysis for Qwen2.5-7B. Top row: (a) compares reasoning conciseness at the best checkpoints; (b) evaluates computational overhead. Bottom row: (c) displays the stability of token generation length across four benchmarks. ERPO achieves superior performance with significantly more concise reasoning paths and comparable training time to GRPO. Training Efficiency and Future Gains. As shown in Figure 4(b), the total training time for ERPO is comparable to GRPO with almost no increase. This is because the additional diagnostic signals used by ERPO are computationally inexpensive to derive. We expect that as training continues, ERPO will become even more efficient than GRPO because the decreasing rollout length directly translates into higher training throughput. 5 Conclusion In this work, we have presented ERPO, an entropy-regulated policy optimization framework designed to address the challenges of sparse and coarse-grained rewards in reinforcement learning for mathematical reasoning. By integrating token-level uncertainty diagnostics with a temporal bucketing mechanism, ERPO successfully transforms global sequence-level feedback into a dense, process-aware guidance signal. This approach allows the model to prioritize exploration at critical decision pivots while suppressing redundant computations in straightforward reasoning steps. Our empirical evaluation across multiple model scales and competitive benchmarks demonstrates that ERPO significantly outperforms standard GRPO and traditional supervised fine-tuning. Beyond pure accuracy, ERPO achieves a superior balance between performance and inference efficiency. By encouraging concise reasoning paths, ERPO models generate shorter sequences while maintaining higher precision, effectively reducing the computational footprint of long-form thought derivations. Given that the diagnostic signals used in ERPO are computationally inexpensive, the framework maintains a training efficiency comparable to vanilla GRPO. Future work will explore the application of ERPO to even more complex multi-modal reasoning tasks and investigate the long-term scaling laws of entropy-regulated exploration over extended training horizons. References [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §2. [2] M. Dai, S. Liu, and Q. Si (2025) Stable reinforcement learning for efficient reasoning. In NeurIPS 2025 Workshop on Efficient Reasoning, Cited by: §1. [3] Z. Dou, Q. Zhao, Z. Wan, D. Zhang, W. Wang, T. Raiyan, B. Chen, Q. Pan, Y. Ouyang, Z. Gao, et al. (2025) Plan then action: high-level planning guidance reinforcement learning for llm reasoning. arXiv preprint arXiv:2510.01833. Cited by: §1. [4] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. External Links: Link Cited by: §1. [5] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §4.1. [6] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §4.1. [7] N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, X. Lyu, et al. (2024) Tulu 3: pushing frontiers in open language model post-training. In Second Conference on Language Modeling, Cited by: §1, §2. [8] A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. (2022) Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, p. 3843–3857. Cited by: §4.1. [9] Y. Li, Z. Dong, Y. Sun, W. Wang, S. Xiong, Y. Luo, J. Liu, H. Lu, J. Wang, W. Su, et al. (2025) Attention illuminates llm reasoning: the preplan-and-anchor rhythm enables fine-grained policy optimization. arXiv preprint arXiv:2510.13554. Cited by: §1. [10] H. Liu and L. Xiao (2025) RE-grpo: leveraging hard negative cases through large language model guided self training. Neurocomputing, p. 132543. Cited by: §1. [11] OpenAI (2024) Learning to reason with llms. Note: https://openai.com/index/learning-to-reason-with-llms/Accessed: 19 March 2026 Cited by: §1. [12] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1, §2. [13] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2. [14] J. Schulman (2020-03) Approximating KL divergence. Note: http://joschu.net/blog/kl-approx.htmlAccessed: 2026-03-20 Cited by: §2. [15] A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar (2025) Rewarding progress: scaling automated process verifiers for llm reasoning. In ICLR, Cited by: §3.1. [16] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1, §2. [17] Q. Team (2025) Qwen2.5 technical report. External Links: 2412.15115 Cited by: §4.1. [18] TRL: Transformers Reinforcement Learning External Links: Link Cited by: §4.1. [19] S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, et al. (2025) Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1. [20] Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang (2025) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1. [21] Y. Zhang and T. Math-AI (2023) American mathematics competitions (amc) 2023. Note: https://huggingface.co/datasets/math-ai/amc23 Cited by: §4.1. [22] Y. Zhang and T. Math-AI (2024) American invitational mathematics examination (aime) 2024. Note: https://huggingface.co/datasets/math-ai/aime24 Cited by: §4.1. [23] Y. Zhang and T. Math-AI (2025) American invitational mathematics examination (aime) 2025. Note: https://huggingface.co/datasets/math-ai/aime25 Cited by: §4.1. Appendix A Appendix / supplemental material