Paper deep dive
Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
Yixuan Wang, Yifei Chen, Haichao Zhang, Haozheng Luo, Xander Wu, Jie Ni, Yun Fu, Nuno Vasconcelos, Yijiang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/23/2026, 1:22:19 AM
Summary
The paper introduces Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), a method that addresses limitations in standard multi-reward reinforcement learning where fixed weighted sums obscure reward profiles and ignore objective saturation. SA-MRPO standardizes each reward objective independently and adaptively discounts contributions based on a batch-level estimate of saturation, reallocating optimization effort toward under-optimized objectives. It generalizes GDPO and GRPO and demonstrates improved performance on mathematical reasoning, adaptive reasoning, and coding benchmarks compared to existing methods.
Entities (8)
Relation Signals (7)
SA-MRPO → generalizes → GDPO
confidence 95% · We further show that SA-MRPO strictly generalizes both Group reward-Decoupled Policy Optimization (GDPO) (19) and Group Relative Policy Optimization (GRPO) (26) as special cases.
SA-MRPO → generalizes → GRPO
confidence 95% · We further show that SA-MRPO strictly generalizes both Group reward-Decoupled Policy Optimization (GDPO) (19) and Group Relative Policy Optimization (GRPO) (26) as special cases.
Yixuan Wang → affiliatedwith → University of Florida
confidence 90% · Yixuan Wang Yifei Chen Haichao Zhang Affiliation: University of Florida
SA-MRPO → improves → AMC23
confidence 90% · On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8% on average and up to 9.2 % on AMC23
SA-MRPO → improves → AIME24
confidence 90% · SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5% on AIME24.
GRPO → suffersfrom → reward_resolution_loss
confidence 85% · First, scalarization loses reward resolution: different weights sum of objective rewards may produce the same scalar value and therefore receive the same advantage.
GRPO → suffersfrom → ignores_objective_saturation
confidence 85% · Second, fixed weights ignore objective saturation: Each objective retains the same relative weight throughout training, regardless of how close it is to saturation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce \textbf{Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization} (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to $5\%$ on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by $3.8\%$ on average and up to $9.2 \%$ on AMC23, and on coding benchmarks it improves pass rate by up to $2.3\%$, while in all settings maintaining the easier objectives near their already satisfied levels.
Tags
Links
- Source: https://arxiv.org/abs/2608.16072v1
- Canonical: https://arxiv.org/abs/2608.16072v1
Trouble viewing inline? Open PDF directly →
Full Text
50,150 characters extracted from source content.
Expand or collapse full text
Learn What’s Left, Not What’s Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization Yixuan Wang Yifei Chen Haichao Zhang Affiliation: University of Florida UC San Diego Northeastern University Haozheng Luo Xander Wu Jie Ni Affiliation: Northwestern University Stanford University, Zillion Network Universität Innsbruck Yun Fu Affiliation: University of Florida UC San Diego Northeastern University Nuno Vasconcelos Yijiang Li Abstract Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5%5\% on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8%3.8\% on average and up to 9.2%9.2\% on AMC23, and on coding benchmarks it improves pass rate by up to 2.3%2.3\%, while in all settings maintaining the easier objectives near their already satisfied levels. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has emerged as a standard approach for improving the reasoning capabilities of large language models. Group Relative Policy Optimization (GRPO) (26) and its variants (6) simplify this process by estimating advantages from groups of rollouts, avoiding the learned value function required by PPO (25). At its core, however, GRPO assumes a single scalar reward for each rollout. In practice, reasoning models are often optimized for multiple objectives (6; 3). A response should not only be correct, but may also need to satisfy constraints on length (21; 5; 4), format, safety (29; 1), or executability, etc. The standard approach is to combine these objectives through a fixed weighted sum and then standardize the resulting scalar reward within each rollout group. We identify two limitations of this design. First, scalarization loses reward resolution: different weights sum of objective rewards may produce the same scalar value and therefore receive the same advantage. Second, fixed weights ignore objective saturation: Each objective retains the same relative weight throughout training, regardless of how close it is to saturation. Consequently, optimization may continue to prioritize an well-optimized objective while the harder under-optimized objectives are overlooked. To this end, we introduce Saturatio-aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO). SA-MRPO preserves per-objective normalization while adaptively reweighting each objective according to its current degree of saturation. Specifically, we estimate saturation using the batch-mean reward relative to the objective’s attainable range, and progressively downweight objectives as they approach their maximum. The resulting advantage reallocates optimization emphasis toward under-optimized objectives while attenuating gradients from objectives that are already well optimized. A single exponent γ controls the strength of this saturation-aware reweighting. Substantially under-optimized objectives therefore receive greater emphasis, whereas objectives approaching saturation are progressively downweighted. SA-MRPO thus reallocates optimization effort throughout training toward objectives with greater remaining headroom while reducing emphasis on those that are already well learned, without modifying the underlying GRPO policy update. We show that this reweighting can change the direction, rather than merely the magnitude, of an update by reversing the sign of a rollout’s aggregate advantage. We further show that SA-MRPO strictly generalizes both Group reward-Decoupled Policy Optimization (GDPO) (19) and Group Relative Policy Optimization (GRPO) (26) as special cases. When saturation-aware reweighting is disabled, it reduces to GDPO, which independently normalizes each reward dimension before aggregation. In the single-objective setting, it further reduces to Group Relative Policy Optimization GRPO. Concurrent approaches such as DVAO (12) and GD2PO (17) adapt multi-reward optimization using reward variance or advantage consistency, but do not explicitly account for how close each objective is to saturation. Consequently, an already saturated objective can continue to exert comparable influence to one with substantially greater room for improvement. SA-MRPO addresses this limitation by making the allocation of optimization effort explicitly depend on each objective’s remaining headroom, while leaving the underlying GRPO policy update unchanged. We evaluate SA-MRPO on mathematical reasoning, controlled adaptive reasoning, and code generation across diverse base models, training data and configurations. Under standard two- and three-objective mathematical reasoning settings, SA-MRPO improves accuracy over GDPO in 12 of 15 benchmark comparisons while largely maintaining the performance of the saturated objective. On adaptive reasoning, SA-MRPO improves accuracy over GDPO on all five benchmarks, yielding a 3.8%3.8\% on average improvement while keeping response lengths within restriction limit. The same behavior extends to code generation: when jointly optimizing executability and test-case pass rate, SA-MRPO improves pass rate on three of four benchmarks, with gains of up to 2.3%2.3\%, while maintaining comparable executability. To summarize, our contributions are: • We identify two limitations of scalarized multi-reward policy optimization: reward-resolution loss and optimization ignores objective saturation, and show that the latter persists after reward decoupling. • We propose SA-MRPO, which dynamically reweights normalized reward objectives according to their degree of optimized saturation. We show that SA-MRPO can alter the direction of policy updates and that it strictly generalizes GDPO and GRPO as special cases. • We empirically validate the effectiveness of SA-MRPO across mathematical reasoning, adaptive reasoning, and code generation, showing that it successfully redistributes optimization effort from saturated objectives to less optimized ones, leading to consistently improvements on the under-optimized objectives while preserving the objectives that are already well optimized. 2 Related Work RLVR and multi objective alignment. Reinforcement learning with verifiable rewards has become a standard approach for improving language model reasoning, with GRPO (26), DeepSeek R1 (6), DAPO (28), and REINFORCE++ (10) developing increasingly effective critic free policy optimization schemes. Beyond single reward optimization, language model alignment frequently involves multiple objectives, including correctness, efficiency, helpfulness, harmlessness, and other preference dimensions. Prior work has studied this problem through conditional preference control, multi objective preference optimization, Pareto optimization, and adaptive reward weighting (31; 23; 13; 18; 20). These works establish that the relative importance of different objectives need not remain fixed throughout optimization. Multi reward group relative policy optimization. More recent work directly studies how multiple rewards should be combined within group relative policy optimization. GDPO (19) normalizes each reward dimension independently before aggregation, avoiding information loss caused by scalarizing heterogeneous rewards before group normalization. DVAO (12) adapts objective weights according to reward variance, while GD2PO (17) addresses conflicts among reward specific advantages. Related methods further consider reward correlation, task imbalance, and alternative multi reward optimization strategies (24; 16; 27). These approaches improve the construction of multi reward optimization signals, but objective importance is generally determined by reward statistics, agreement, or optimization structure rather than directly by the remaining attainable reward range. Dynamic objective allocation. Most closely related to our work, Dynamic Reward Weighting (20), SAW (8), and Focal Reward (11) recognize that objectives can progress at different rates and that optimization effort should evolve accordingly. SAW uses reward variability as a measure of objective informativeness, while Focal Reward estimates saturation for rubric based reward criteria. SA-MRPO instead focuses on bounded verifiable rewards and defines saturation directly from the fraction of the attainable reward range already achieved. This enables saturation aware reweighting of independently normalized reward advantages while retaining the standard GRPO policy update. 3 Preliminaries We study reinforcement learning for an auto-regressive language model under multiple reward objectives. Let πθ _θ denote the policy parameterized by θ. Given a query q, the policy defines an auto-regressive distribution over an output sequence o≜(o1,…,o|o|)o (o_1,…,o_|o|) according to πθ(o∣q)≜∏t=1|o|πθ(ot∣q,o1:t−1), _θ(o q) _t=1^|o| _θ(o_t q,o_1:t-1), where o1:t−1≜(o1,…,ot−1)o_1:t-1 (o_1,…,o_t-1) denotes the prefix preceding token oto_t. At each policy update, a batch of B queries qii=1B\q_i\_i=1^B is sampled from a data distribution D. For each query qiq_i, a frozen behavior policy πθold _ _old generates a group of G≥2G≥ 2 rollouts, oi,j∼πθold(⋅∣qi),j∈1,…,G. o_i,j _ _old(· q_i), j∈\1,…,G\. Let oi,j≜(oi,j,1,…,oi,j,|oi,j|)o_i,j (o_i,j,1,…,o_i,j,|o_i,j|) denote the j-th rollout associated with query qiq_i, and let oi,j,<t≜(oi,j,1,…,oi,j,t−1)o_i,j,<t (o_i,j,1,…,o_i,j,t-1) denote its prefix before token t. The policy is trained with n reward objectives. For objective k∈1,…,nk∈\1,…,n\, let R(k)R^(k) denote its reward function and let wk≥0w_k≥ 0 denote its prescribed weight. The reward assigned by objective k to rollout oi,jo_i,j is rk(i,j)≜R(k)(qi,oi,j).r_k^(i,j) R^(k)(q_i,o_i,j). For a finite collection S≜x1,…,xmS \x_1,…,x_m\, the mean and the standard deviation of S is represented as: mean(S)=1m∑l=1mxl,std(S)=1m∑l=1m(xl−mean(S))2, (S)= 1m _l=1^mx_l, (S)= 1m _l=1^m (x_l-mean(S) )^2, respectively. Group Relative Policy Optimization. GRPO estimates the advantage of a rollout by comparing its reward with the other rollouts generated for the same query. With multiple reward objectives, a standard approach first combines the individual rewards into a scalar score rsum(i,j)≜∑k=1nwkrk(i,j).r_sum^(i,j) _k=1^nw_kr_k^(i,j). The resulting score is then standardized within the group generated for query qiq_i. The GRPO advantage of rollout oi,jo_i,j is defined as AGRPO(i,j)≜rsum(i,j)−meanrsum(i,1),…,rsum(i,G)stdrsum(i,1),…,rsum(i,G). A_GRPO^(i,j) r_sum^(i,j)-mean \r_sum^(i,1),…,r_sum^(i,G) \std \r_sum^(i,1),…,r_sum^(i,G) \. For a clipping threshold ϵ>0ε>0, define clip(x,a,c)≜max(a,min(x,c)).clip(x,a,c) (a, (x,c) ). GRPO maximizes the clipped surrogate objective GRPO(θ)=[1G∑j=1G1|oi,j|∑t=1|oi,j|min(ρi,j,t(θ)AGRPO(i,j),clip(ρi,j,t(θ),1−ϵ,1+ϵ)AGRPO(i,j))], _GRPO(θ)=E [ 1G _j=1^G 1|o_i,j| _t=1^|o_i,j| ( _i,j,t(θ)A_GRPO^(i,j),\,clip ( _i,j,t(θ),1-ε,1+ε )A_GRPO^(i,j) ) ], where the expectation is taken over qi∼q_i and oi,jj=1G\o_i,j\_j=1^G sampled from πθold(⋅∣qi) _ _old(· q_i) and ρi,j,t(θ)≜πθ(oi,j,t∣qi,oi,j,<t)πθold(oi,j,t∣qi,oi,j,<t). _i,j,t(θ) _θ (o_i,j,t q_i,o_i,j,<t ) _ _old (o_i,j,t q_i,o_i,j,<t ). As in standard GRPO, a KL penalty against a fixed reference policy may additionally be included with coefficient β≥0β≥ 0. We omit this term because it is independent of the reward construction studied in this work. 4 Saturation Aware Advantage Reweighting The GRPO construction above combines all reward dimensions before computing the group relative advantage. This scalarization introduces two limitations. First, distinct reward profiles can collapse to the same scalar reward. For example, under equal weights, (1,0)(1,0) and (0,1)(0,1) become indistinguishable. Second, scalarization does not account for how much improvement remains for each objective. The policy can continue favoring an already saturated objective instead of directing more of the update toward the objective that still requires improvement, as illustrated in Figure 1. These two limitations motivate separating objective specific relative performance from the current optimization state of each objective. Figure 1: Comparison of GRPO, GDPO, and SA-MRPO on one group of G=4G=4 rollouts with a saturated format objective and an unsaturated correctness objective. Rollouts 22 and 33 have the same scalar reward but different reward profiles, causing GRPO to assign both zero advantage. GDPO distinguishes the two profiles but assigns a larger advantage to rollout 33, despite its zero correctness. SA-MRPO discounts the saturated format objective and instead assigns rollout 33 a more negative advantage than rollout 22, while favoring rollout 44, which achieves the highest correctness. 4.1 Saturation Aware Group Relative Advantage For each query qiq_i, let rk(i,j)r_k^(i,j) denote the reward assigned by objective k to rollout oi,jo_i,j, where i∈1,…,Bi∈\1,…,B\, j∈1,…,Gj∈\1,…,G\, and k∈1,…,nk∈\1,…,n\. A multi objective policy update must account for both the relative quality of a rollout under each objective and the current optimization state of that objective. We incorporate these two sources of information directly into a single group relative advantage. For each objective k, define the group statistics associated with query qiq_i as μk(i)≜meanrk(i,1),…,rk(i,G),σk(i)≜stdrk(i,1),…,rk(i,G). _k^(i) \r_k^(i,1),…,r_k^(i,G) \, _k^(i) \r_k^(i,1),…,r_k^(i,G) \. To characterize the current optimization state of objective k, we use its average reward over the current batch, r¯(k)≜meanrk(i,j):i∈1,…,B,j∈1,…,G. r^(k) \r_k^(i,j):i∈\1,…,B\,j∈\1,…,G\ \. Let rmin(k)r_ ^(k) and rmax(k)r_ ^(k) denote the attainable lower and upper reward bounds of objective k, respectively. We then define the saturation ratio of objective k as s(k)≜r¯(k)−rmin(k)rmax(k)−rmin(k)∈[0,1]. s^(k) r^(k)-r_ ^(k)r_ ^(k)-r_ ^(k)∈[0,1]. A smaller s(k)s^(k) indicates that a larger fraction of the attainable reward range remains unrealized, whereas a larger s(k)s^(k) indicates that the objective is closer to its reward ceiling. Given prescribed objective weights wkk=1n\w_k\_k=1^n and a saturation exponent γ≥0γ≥ 0, we define the saturation aware group relative advantage before final batch normalization as A~(i,j)≜∑k=1nw~kAk(i,j), A^(i,j) _k=1^n w_kA_k^(i,j), where Ak(i,j)≜rk(i,j)−μk(i)σk(i),w~k≜wk(1−s(k))γ. A_k^(i,j) r_k^(i,j)- _k^(i) _k^(i), w_k w_k (1-s^(k) )^γ. The term Ak(i,j)A_k^(i,j) measures the relative quality of rollout oi,jo_i,j under objective k within the corresponding rollout group, while w~k w_k modulates the contribution of objective k according to both its prescribed importance and its current saturation level. Specifically, the factor (1−s(k))γ (1-s^(k) )^γ decreases the influence of objective k as the current policy realizes a larger fraction of its attainable reward range. Thus, each objective contributes to the aggregate advantage according to its prescribed importance, its remaining room for improvement, and the relative quality of the current rollout. Since the saturation ratios evolve during training, the scale of A~(i,j) A^(i,j) may vary across policy updates. We therefore normalize the aggregate advantages over the current batch as A^SA(i,j)≜A~(i,j)−mean()std(),≜A~(i,j):i∈1,…,B,j∈1,…,G. A_SA^(i,j) A^(i,j)-mean(A)std(A), \ A^(i,j):i∈\1,…,B\,j∈\1,…,G\ \. (1) The resulting advantage is directly used in the standard clipped group relative surrogate objective, SA−MRPO(θ)=[1G∑j=1G1|oi,j|∑t=1|oi,j|min(ρi,j,t(θ)A^SA(i,j),clip(ρi,j,t(θ),1−ϵ,1+ϵ)A^SA(i,j))]. _SA-MRPO(θ)=E [ 1G _j=1^G 1|o_i,j| _t=1^|o_i,j| ( _i,j,t(θ) A_SA^(i,j),clip ( _i,j,t(θ),1-ε,1+ε ) A_SA^(i,j) ) ]. Thus, SA-MRPO changes only the construction of the rollout advantage while retaining the underlying group relative policy optimization objective. Algorithm 1 SA-MRPO policy update 1: queries qii=1B\q_i\_i=1^B, weights wkk=1n\w_k\_k=1^n, reward bounds (rmin(k),rmax(k))k=1n\(r_ ^(k),r_ ^(k))\_k=1^n, saturation exponent γ, clipping threshold ϵε 2: sample oi,jj=1G∼πθold(⋅∣qi)\o_i,j\_j=1^G _ _old(· q_i) for each i 3: compute rk(i,j)=R(k)(qi,oi,j)r_k^(i,j)=R^(k)(q_i,o_i,j) for all i,j,ki,j,k 4: for k=1k=1 to n do 5: r¯(k)←meanrk(i,j)i,j r^(k) \r_k^(i,j)\_i,j 6: s(k)←r¯(k)−rmin(k)rmax(k)−rmin(k)s^(k)← r^(k)-r_ ^(k)r_ ^(k)-r_ ^(k) 7: end for 8: for each query i and rollout j do 9: A~(i,j)←∑k=1nwk(1−s(k))γrk(i,j)−meanrk(i,1),…,rk(i,G)stdrk(i,1),…,rk(i,G) A^(i,j)← _k=1^nw_k (1-s^(k) )^γ r_k^(i,j)-mean\r_k^(i,1),…,r_k^(i,G)\std\r_k^(i,1),…,r_k^(i,G)\ 10: end for 11: A^SA(i,j)←(A~(i,j)−meanA~)/stdA~ A_SA^(i,j)← ( A^(i,j)-mean\ A\ )/std\ A\ for all i,ji,j 12: update θ by ascending ∇θSA−MRPO(θ) _θJ_SA-MRPO(θ) using A^SA(i,j)\ A_SA^(i,j)\ Algorithm 1 summarizes the resulting policy update. The saturation rule therefore provides an adaptive allocation mechanism, but changing the relative allocation across objectives also raises two questions: whether emphasizing an unsaturated objective can degrade an already optimized objective, and whether the saturation estimate itself can incorrectly represent the remaining optimization potential. The proposed construction contains static objective allocation as a special case. When γ=0γ=0, the saturation factors are identically one and w~k=wk,A~(i,j)=∑k=1nwkAk(i,j), w_k=w_k, A^(i,j)= _k=1^nw_kA_k^(i,j), which recovers the corresponding GDPO advantage under the same objective weights and normalization. The saturation aware reweighting also reduces the relative contribution of the more saturated objective compared with the prescribed allocation. Increasing γ further shifts the relative allocation toward objectives with larger remaining reward headroom. For any two objectives a and b satisfying wa>0w_a>0, wb>0w_b>0, and s(a),s(b)∈[0,1)s^(a),s^(b)∈[0,1), w~aw~b=wawb(1−s(a)1−s(b))γ. w_a w_b= w_aw_b ( 1-s^(a)1-s^(b) )^γ. If s(a)>s(b)s^(a)>s^(b), then (1−s(a))/(1−s(b))<1(1-s^(a))/(1-s^(b))<1. Therefore, the ratio w~a/w~b w_a/ w_b is strictly decreasing in γ. In particular, for every γ>0γ>0, we have w~a/w~b<wa/wb. w_a/ w_b<w_a/w_b. 4.2 Objective Conflict and Failure Modes Saturation aware reweighting reallocates optimization emphasis according to remaining reward headroom, but it does not impose a constraint that previously optimized objectives must be preserved. This distinction becomes important when two objectives induce conflicting policy updates. Let Jk(θ)J_k(θ) denote an objective specific policy surrogate and let gk(θ)≜∇θJk(θ)g_k(θ) _θJ_k(θ). For the saturation weighted ascent direction d(θ)≜∑k=1nw~kgk(θ), d(θ) _k=1^n w_kg_k(θ), the first order change of objective a along this direction is characterized by DJa(θ)[d]=w~a‖ga(θ)‖2+∑k≠aw~kga(θ)⊤gk(θ). DJ_a(θ)[d]= w_a \|g_a(θ) \|^2+ _k≠ a w_kg_a(θ) g_k(θ). The first term is the contribution of objective a to its own improvement, whereas the cross objective terms describe how updates induced by the remaining objectives affect objective a. If ga(θ)⊤gk(θ)≥0g_a(θ) g_k(θ)≥ 0 for every positively weighted objective k, then the aggregate direction cannot decrease JaJ_a to first order. In contrast, objective a decreases whenever ∑k≠aw~kga(θ)⊤gk(θ)<−w~a‖ga(θ)‖2. _k≠ a w_kg_a(θ) g_k(θ)<- w_a \|g_a(θ) \|^2. (2) Equation (2) gives the precise local condition under which conflict from the other objectives overwhelms the improvement induced by objective a itself. Because w~a=wa(1−s(a))γ w_a=w_a(1-s^(a))^γ decreases as objective a becomes more saturated, saturation aware allocation deliberately reduces the self improvement term protecting that objective. Therefore, when a less saturated objective has a sufficiently conflicting gradient, reallocating optimization effort toward that objective can reduce the performance of an objective that was previously well optimized. This behavior is particularly relevant when the objectives compete for the same finite model capacity, although limited capacity is only one possible mechanism that can produce negative gradient alignment. SA-MRPO should therefore be interpreted as an adaptive objective allocation rule rather than a constrained multi objective method that guarantees monotonic retention of every saturated objective. The empirical question is consequently whether the gain obtained on objectives with greater remaining headroom outweighs any degradation of objectives whose optimization pressure has been reduced. A separate consideration is that nominal reward headroom does not necessarily coincide with optimizable headroom. By construction, 1−s(k)1-s^(k) measures the fraction of the prescribed reward range that remains unrealized. When the reward bounds are specified directly by the reward function, this quantity is exactly observable and does not require estimation of the reward ceiling. However, a large remaining reward range does not imply that the current policy class has sufficient capacity to realize the corresponding improvement. In particular, an objective may remain far from its prescribed maximum even when the best policy representable by the current model can achieve only limited further improvement. Thus, s(k)s^(k) should be interpreted as a measure of remaining nominal reward headroom rather than a certificate of remaining achievable improvement. We do not regard this mismatch as a failure mode of the proposed saturation measure, since the saturation ratio correctly characterizes progress within the prescribed reward range and the unattainability of the remaining reward is instead imposed by the capacity of the underlying policy class. 5 Experiments We evaluate SA-MRPO with three questions in mind: whether saturation aware reweighting improves policy optimization under multiple reward objectives, whether the improvement is consistent with reallocating optimization effort away from objectives that are already saturated, and whether the same behavior extends beyond mathematical reasoning. We first evaluate SA-MRPO under standard multi-objective mathematical reasoning settings. We then construct an adaptive reasoning setting with an explicit saturation region to study the proposed mechanism more directly. Finally, we evaluate the method on code reasoning and study the effect of the saturation exponent γ. 5.1 Experimental Setup Training protocol. Unless otherwise specified, all experiments are implemented with verl and use vLLM for rollout generation. For each training prompt, we sample G=8G=8 responses and use the resulting group for reward computation and advantage estimation. We train all models for 3 epochs with a global batch size of 256 and a maximum response length of 4096 tokens. Mathematical and adaptive reasoning experiments are conducted on DeepScaleR-Preview 22, which contains approximately 40K competition-level mathematical reasoning problems. The model architecture, reward construction, and task-specific deviations from this protocol are described in the corresponding subsections. Evaluation protocol. For mathematical reasoning, we use vLLM with temperature 0.60.6, top-p=0.95p=0.95, and a maximum generation length of 4096 tokens. We sample 16 responses per problem and report the average pass@1 accuracy. For code reasoning, we use the same temperature and top-p with a maximum generation length of 2048 tokens. Task specific auxiliary metrics are introduced together with the corresponding experiments. In all evaluation tables, “Base” denotes the corresponding model before RL training. 5.2 Mathematical Reasoning We first investigate whether saturation-aware reweighting improves general multi-objective reasoning performance. We train Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct following the training protocol described in Section 5.1. We consider both two and three objective settings. The two objective setting optimizes correctness and response length compliance, while the three objective setting additionally includes a format objective. The latter introduces multiple auxiliary objectives that can reach high reward levels at different stages of training. The reward objectives are defined as follows. • Length reward. Let l=4000l=4000 denote the response length budget. We define ℛlength(o)=1,|o|≤l,0,|o|>l.R_length(o)= cases1,&|o|≤ l,\\ 0,&|o|>l. cases • Correctness reward. Let y denote the ground truth answer and let Extract(o)Extract(o) denote the final answer parsed from response o. We define ℛcorrect(o)=1,Extract(o)=y,0,Extract(o)≠y.R_correct(o)= cases1,&Extract(o)=y,\\ 0,&Extract(o)≠ y. cases • Format reward. The format reward evaluates whether the generated response follows the required XML style reasoning format. Specifically, the response must contain exactly one <answer> block and one </answer> block, and the complete response must match <think>...</think> <answer>...</answer>. We define ℛformat(o)=1,if o matches ˆ<think>.*?</think> <answer>.*?</answer>$,and N<answer>(o)=1,N</answer>(o)=1,0,otherwise.R_format(o)= cases1,& aligned &if o matches \^<think>.*?</think> n<answer>.*?</answer>\$,\\ &and N_ <answer>(o)=1, N_ </answer>(o)=1, aligned\\ 0,&otherwise. cases We evaluate on AIME24 (30), AMC23 11 1 https://huggingface.co/datasets/math-ai/amc23, MATH500 (9), Minerva Math 22 2 https://huggingface.co/datasets/math-ai/minervamath, and OlympiadBench (7). In addition to accuracy, we report Exceed, defined as the fraction of generated responses whose length exceeds the 4000 token budget. Table 1: Comparison of GDPO and SA-MRPO on mathematical reasoning under two and three reward objectives. Accuracy is higher is better, while Exceed denotes the fraction of responses exceeding the 4000 token length budget and is lower is better. The Qwen2.5-3B-Instruct base model is shared across the two reward settings. Benchmark Metric Qwen2.5-7B-Instruct Qwen2.5-3B-Instruct Three objectives Base ℛcorrect+ℛlengthR_correct+R_length ℛcorrect+ℛlength+ℛformatR_correct+R_length+R_format Base GDPO SA-MRPO GDPO2obj SA-MRPO2obj GDPO3obj SA-MRPO3obj AIME24 Acc ↑ 11.7% 11.5% 16.5% 0.6% 5.0% 8.5% 6.7% 8.1% Exceed ↓ 3.5% 1.2% 1.5% 6.2% 0.0% 0.6% 0.6% 1.7% Minerva Acc ↑ 16.1% 24.2% 24.8% 6.7% 16.2% 16.6% 16.9% 18.1% Exceed ↓ 0.4% 0.1% 0.0% 0.4% 0.1% 0.1% 0.1% 0.1% AMC23 Acc ↑ 41.1% 44.6% 43.5% 10.7% 33.2% 34.9% 31.5% 35.2% Exceed ↓ 1.0% 1.2% 1.1% 1.7% 0.0% 0.1% 0.4% 0.6% MATH500 Acc ↑ 50.0% 64.2% 67.7% 26.3% 57.1% 58.2% 58.9% 59.5% Exceed ↓ 0.7% 0.0% 0.1% 0.6% 0.0% 0.1% 0.1% 0.1% Olympiad Acc ↑ 23.8% 25.3% 26.1% 4.5% 20.6% 19.3% 20.6% 20.0% Exceed ↓ 2.5% 0.2% 0.7% 3.3% 0.1% 0.3% 0.2% 0.6% Table 1 compares GDPO and SA-MRPO across model scales and reward configurations. Across the three configurations, SA-MRPO achieves higher accuracy than GDPO in 12 of the 15 benchmark comparisons. For Qwen2.5-7B-Instruct with three reward objectives, SA-MRPO improves four of the five benchmarks, including gains of 5.05.0 percentage points on AIME24 and 3.53.5 percentage points on MATH500. The same pattern is observed for Qwen2.5-3B-Instruct, where SA-MRPO improves four of five benchmarks in both the two and three objective settings. These accuracy gains are generally accompanied by only small changes in Exceed, indicating that the improvement in correctness does not require abandoning the auxiliary length objective. 5.3 Adaptive Reasoning The previous experiment evaluates SA-MRPO under standard multi objective reward constructions. We next consider a setting in which saturation is explicitly built into one reward objective, allowing the proposed allocation mechanism to be examined more directly. We train DeepSeek-R1-Distill-Qwen-7B with two rule based objectives: correctness and length efficiency. No learned judge or external reward model is used. Training configuration follows Section 5.1. The correctness reward is identical to ℛcorrectR_correct defined in Section 5.2. Unlike the binary length constraint above, we define a graded length reward ℛlength(o)=1,|o|≤Bmin,Bmax−|o|Bmax−Bmin,Bmin<|o|<Bmax,0,|o|≥Bmax,R_length(o)= cases1,&|o|≤ B_ ,\\[4.0pt] B_ -|o|B_ -B_ ,&B_ <|o|<B_ ,\\[10.0pt] 0,&|o|≥ B_ , cases where Bmin=1024B_ =1024 and Bmax=2048B_ =2048. This construction has an explicit saturation region: once a response contains at most BminB_ tokens, the length reward reaches its maximum value and further shortening provides no additional reward. The setting therefore captures the regime motivating SA-MRPO, where an auxiliary objective can become saturated while correctness retains substantial room for improvement and evaluate on the same five mathematical reasoning benchmarks as in Section 5.2. We report accuracy together with Len, the average number of generated tokens. Benchmark SA-MRPO GDPO Δ Acc. Acc. Len. Acc. Len. AIME24 7.3 804 5.2 566 +2.1 Minerva 15.9 277 15.4 214 +0.5 AMC23 37.5 417 28.3 290 +9.2 MATH500 51.5 270 47.1 187 +4.4 Olympiad 20.9 529 18.1 406 +2.8 Average 26.6 459 22.8 333 +3.8 Table 2: Adaptive reasoning with an explicitly saturated length objective. Base GDPO SA-MRPO APPS Pass ↑ 43.8% 53.2% 53.8% Bug ↓ 19.6% 8.5% 9.9% CodeCont. Pass ↑ 12.4% 19.2% 20.6% Bug ↓ 32.9% 15.5% 15.5% Codeforces Pass ↑ 8.7% 10.6% 12.9% Bug ↓ 34.4% 8.6% 9.0% TACO Pass ↑ 29.0% 36.0% 35.6% Bug ↓ 23.1% 11.0% 12.4% Table 3: Code reasoning results for Qwen2.5-7B-Instruct. Table 2 shows that SA-MRPO improves accuracy over GDPO on all five benchmarks, with an average gain of 3.83.8 percentage points. The largest improvement occurs on AMC23, where accuracy increases from 28.3%28.3\% to 37.5%37.5\%. At the same time, SA-MRPO produces moderately longer responses, increasing the average response length from 333333 to 459459 tokens. Importantly, the average response length under both methods remains below the saturation threshold Bmin=1024B_ =1024. The additional tokens used by SA-MRPO are therefore consistent with the intended allocation mechanism: once the length objective is already in its high reward regime, aggressively shortening the response provides diminishing optimization value, while correctness still has substantial room for improvement. SA-MRPO reduces the relative influence of the saturated length objective and allows additional reasoning capacity to be used for correctness. The resulting 3.83.8 point average accuracy improvement provides direct empirical support for reallocating optimization effort according to remaining reward headroom. 5.4 Code Generation We next evaluate whether saturation aware reweighting extends beyond mathematical reasoning. We train Qwen2.5-7B-Instruct on the Eurus-2-RL dataset 2 using two rule based reward objectives: test case pass rate and executability. Training is performed with verl for three epochs. Unless otherwise specified, all remaining configurations follow Section 5.1. GDPO and SA-MRPO use identical training data, reward functions, and optimization hyperparameters and differ only in the construction of the aggregate advantage. The pass rate reward is defined as ℛpass=# passed test cases# total test cases.R_pass= \# passed test cases\# total test cases. The executability reward is ℛexec=1,if the generated program compiles and executes without errors,0,otherwise.R_exec= cases1,&if the generated program compiles and executes without errors,\\ 0,&otherwise. cases Executability captures a basic validity requirement that can become satisfied before full functional correctness. The pass rate objective is more demanding because it requires the generated program to produce correct behavior across the test cases. This setting therefore provides a qualitatively different instance in which one objective may become substantially easier to satisfy than another. We evaluate on APPS 9, CodeContests 15, Codeforces, and TACO 14. For each problem, we sample 16 responses with temperature 0.60.6, top-p=0.95p=0.95, and a maximum generation length of 2048 tokens. We report Pass, the average test case pass rate, and Bug, the fraction of generated programs that encounter compilation or runtime errors. Table 3 shows that both methods substantially improve functional correctness and executability over the base model. Compared with GDPO, SA-MRPO achieves higher pass rates on three of the four benchmarks, with improvements of 0.60.6, 1.41.4, and 2.32.3 percentage points on APPS, CodeContests, and Codeforces, respectively. On TACO, the pass rate is 0.40.4 percentage points lower than GDPO. The corresponding bug rates remain comparable, with both methods producing substantially fewer invalid programs than the base model. These results extend the saturation aware allocation effect beyond mathematical reasoning. Executability represents a relatively basic constraint, whereas test case pass rate captures the more difficult objective of functional correctness. SA-MRPO improves the latter on most benchmarks while largely retaining the executability gains achieved by GDPO, consistent with reallocating optimization effort toward the objective with greater remaining headroom. 5.5 Effect of the Saturation Strength We finally examine whether the saturation exponent γ controls the allocation of optimization effort in the manner predicted by the SA-MRPO weighting rule. Recall that w~k=wk(1−s(k))γ, w_k=w_k (1-s^(k) )^γ, so increasing γ more aggressively suppresses objectives with large saturation s(k)s^(k). For this experiment, we train Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on DeepScaleR-Preview for one epoch. All runs use 88 rollouts per problem, a batch size of 256256, and a maximum response length of 40964096 tokens. We use only the correctness and length rewards defined in Section 5.2. Figure 2: Training reward trajectories under different values of the saturation exponent γ. Larger values of γ place less relative weight on the highly saturated length objective and more relative emphasis on correctness. Figure 2 shows a systematic change in the two reward dimensions as γ increases. Compared with γ=0γ=0, positive values of γ generally produce higher correctness rewards during training. At the same time, the length reward decreases gradually as γ becomes larger, particularly for γ=0.75γ=0.75 and γ=1.0γ=1.0. This behavior is consistent with the proposed mechanism: increasing γ more strongly discounts the highly saturated length objective, thereby shifting relative optimization pressure toward correctness. The resulting change is therefore not merely a hyperparameter sensitivity effect, but reflects the allocation tradeoff controlled directly by the saturation weighting rule. Table 4: Effect of the saturation exponent γ on Qwen2.5-3B-Instruct under the two objective setting ℛcorrect+ℛlengthR_correct+R_length. Accuracy is reported in percentages. Exceed denotes the fraction of generated responses exceeding the predefined length budget. The γ=0.25γ=0.25 configuration corresponds to the SA-MRPO2obj result in Table 1. Benchmark Metric Base Qwen2.5-3B-Instruct, ℛcorrect+ℛlengthR_correct+R_length γ=0γ=0 γ=0.25γ=0.25 γ=0.5γ=0.5 γ=0.75γ=0.75 γ=1.0γ=1.0 AIME24 Acc ↑ 0.6% 5.0% 8.5% 9.0% 8.7% 7.4% Exceed ↓ 6.2% 0.0% 0.6% 0.7% 0.8% 1.1% Minerva Acc ↑ 6.7% 16.2% 16.6% 16.9% 17.0% 16.8% Exceed ↓ 0.4% 0.1% 0.1% 0.2% 0.1% 0.3% AMC23 Acc ↑ 10.7% 33.2% 34.9% 35.6% 35.3% 34.8% Exceed ↓ 1.7% 0.0% 0.1% 0.2% 0.2% 0.4% MATH500 Acc ↑ 26.3% 57.1% 58.2% 58.6% 58.8% 58.5% Exceed ↓ 0.6% 0.0% 0.1% 0.1% 0.2% 0.3% Olympiad Acc ↑ 4.5% 20.6% 19.3% 20.1% 19.4% 20.7% Exceed ↓ 3.3% 0.1% 0.3% 0.1% 0.7% 0.3% Table 4 confirms the same tradeoff at downstream evaluation. Every positive value of γ improves average mathematical reasoning accuracy relative to γ=0γ=0 across the five benchmarks. Among the evaluated settings, γ=0.5γ=0.5 achieves the highest average accuracy and provides the strongest performance on AIME24 and AMC23. Larger values of γ remain competitive in accuracy but generally increase Exceed, consistent with the decreasing length reward observed in Figure 2. Therefore, γ directly controls the balance between reallocating optimization effort toward correctness and preserving pressure on the already highly satisfied length objective. Moderate values provide the strongest overall balance in the present experiments. 6 Conclusion We studied multi reward policy optimization from the perspective of how optimization effort is allocated across objectives at different stages of training. While reward decoupling preserves information from individual reward dimensions, it does not distinguish between objectives that remain difficult and objectives that are already close to saturation. We introduced SA-MRPO, which addresses this limitation by adapting each objective’s contribution according to its current saturation while retaining the standard GRPO policy update. Across mathematical reasoning, adaptive reasoning, and coding tasks, SA-MRPO consistently shifts optimization toward objectives with greater remaining headroom, improving the harder objective in most settings while largely preserving objectives that are already satisfied. The ablation over the saturation exponent further shows that this reallocation is controllable, exposing a direct tradeoff between improving under optimized objectives and preserving saturated auxiliary objectives. These results suggest that effective multi reward policy optimization should account not only for the relative scale of reward signals, but also for how much useful improvement remains in each objective. References Chen et al. (2025) Z. Chen, M. Kang, and B. Li ShieldAgent: shielding agents via verifiable safety policy reasoning. In Forty-second International Conference on Machine Learning, Cited by: §1. Cui et al. (2025) G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cited by: §5.4. Fang et al. (2025) G. Fang, X. Ma, and X. Wang Thinkless: LLM learns when to think. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1. Feng et al. (2025) S. Feng, G. Fang, X. Ma, and X. Wang Efficient reasoning models: a survey. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856 Cited by: §1. Fu et al. (2025) T. Fu, Y. Ge, Y. You, E. Liu, Z. Yuan, G. Dai, S. Yan, H. Yang, and Y. Wang R2R: efficiently navigating divergent reasoning paths with small-large model token routing. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850. Cited by: §5.2. He et al. (2026) Y. He, B. Bi, S. Liu, H. Liao, Y. Ge, B. Wan, S. Tong, J. Chen, J. Guo, and X. Cheng SAW: stage-aware dynamic weighting for multi-objective reinforcement learning in large language models. arXiv preprint arXiv:2606.07705. Cited by: §2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §5.2, §5.4. Hu et al. (2025) J. Hu, J. K. Liu, H. Xu, and W. Shen Reinforce++: stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262. Cited by: §2. Huang et al. (2026) Y. Huang, Z. Zhao, Z. Huan, W. Gu, F. Hong, X. Ge, L. Yuan, W. Wu, Q. Hu, X. Zhang, et al. Focal reward: balanced reinforcement learning under rubric-based rewards. arXiv preprint arXiv:2605.26579. Cited by: §2. Jiang et al. (2026) G. Jiang, J. Song, G. Quan, C. Hao, G. Liu, and Y. Zhang DVAO: dynamic variance-adaptive advantage optimization for multi-reward reinforcement learning. arXiv preprint arXiv:2605.25604. Cited by: §1, §2. Li et al. (2025) C. Li, H. Zhang, Y. Xu, H. Xue, X. Ao, and Q. He Gradient-adaptive policy optimization: towards multi-objective alignment of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11214–11232. Cited by: §2. Li et al. (2023) R. Li, J. Fu, B. Zhang, T. Huang, Z. Sun, C. Lyu, G. Liu, Z. Jin, and G. Li Taco: topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852. Cited by: §5.4. Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science 378 (6624), p. 1092–1097. Cited by: §5.4. Liang et al. (2026) R. Liang, Y. Zhong, Y. Yuan, Y. Zheng, T. Tan, T. Wang, H. Guo, J. Wang, and X. Zhan Don’t mix rewards, mix policies: policy decomposition and optimization for multi-reward rl. arXiv preprint arXiv:2607.29246. Cited by: §2. Liu et al. (2026a) H. Liu, Y. Liu, J. Ni, S. Huang, X. Liu, P. Cheng, J. Song, R. Ding, J. Li, Z. Yu, et al. GD2 po: mitigating multi-reward conflicts via group-dynamic reward-decoupled policy optimization. arXiv preprint arXiv:2606.16771. Cited by: §1, §2. Liu et al. (2025) Q. Liu, J. Ruan, H. Li, H. Zhao, D. Wang, J. Chen, W. Guanglu, X. Cai, Z. Zheng, and T. Xu AMoPO: adaptive multi-objective preference optimization without reward models and reference models. In Findings of the Association for Computational Linguistics: ACL 2025, p. 8832–8866. Cited by: §2. Liu et al. (2026b) S. Liu, X. Dong, X. Lu, S. Diao, P. Belcak, M. Liu, M. Chen, H. Yin, Y. F. Wang, K. Cheng, et al. Gdpo: group reward-decoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242. Cited by: §1, §2. Lu et al. (2026) Y. Lu, Z. Wang, S. Li, X. Liu, C. Yu, Q. Yin, Z. Shi, Z. Zhang, and M. Jiang Learning to optimize multi-objective alignment through dynamic reward weighting. Transactions of the Association for Computational Linguistics 14, p. 1051–1073. Cited by: §2, §2. Luo et al. (2026) H. Luo, Z. Jiang, M. Z. Hasan, Y. Chen, and S. Sarkar FROST: filtering reasoning outliers with attention for efficient reasoning. In The Fourteenth International Conference on Learning Representations, Cited by: §1. Luo et al. (2025) M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, L. E. Li, R. A. Popa, et al. Deepscaler: surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog 3 (5). Cited by: §5.1. Mukherjee et al. (2024) S. Mukherjee, A. Lalitha, S. Sengupta, A. Deshmukh, and B. Kveton Multi-objective alignment of large language models through hypervolume maximization. arXiv preprint arXiv:2412.05469. Cited by: §2. Ramesh et al. (2026) S. S. Ramesh, X. Ji, M. Zimmer, S. Yoon, Z. Wang, H. B. Ammar, A. Lucchi, and I. Bogunovic Multi-task grpo: reliable llm reasoning across tasks. arXiv preprint arXiv:2602.05547. Cited by: §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1, §2. Wang et al. (2026) W. Wang, J. Bao, T. Yongsiqi, Y. Liu, H. Zhou, H. Ma, M. Zhou, W. Fan, J. He, X. Jiang, et al. SMOPD: multi-reward reinforcement learning via specialize-and-merge online policy distillation. arXiv preprint arXiv:2608.03092. Cited by: §2. Yu et al. (2026) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: §2. Zhang et al. (2026) Y. Zhang, Y. Ding, J. Yang, T. Luo, D. Li, R. Duan, Q. Liu, H. Su, Y. Dong, and J. Zhu Towards safe reasoning in large reasoning models via corrective intervention. In International Conference on Learning Representations, Vol. 2026, p. 53421–53444. Cited by: §1. Zhang and Math-AI (2024) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2024. Cited by: §5.2. Zhou et al. (2024) Z. Zhou, J. Liu, J. Shao, X. Yue, C. Yang, W. Ouyang, and Y. Qiao Beyond one-preference-fits-all alignment: multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, p. 10586–10613. Cited by: §2.