Paper deep dive
ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning
Yihang Chen, Yuanhao Ban, Cho-Jui Hsieh
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an $\alpha$-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.
Tags
Links
- Source: https://arxiv.org/abs/2609.35433v1
- Canonical: https://arxiv.org/abs/2609.35433v1
Trouble viewing inline? Open PDF directly →
Full Text
99,065 characters extracted from source content.
Expand or collapse full text
ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning Yihang Chen Affiliation: Department of Computer Science, University of California, Los Angeles Yuanhao Ban Affiliation: yhangchen, banyh2000, chohsieh@cs.ucla.edu Cho-Jui Hsieh Affiliation: Project Page: ReSPO Abstract Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an α-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning. 1 Introduction RL from verifiable rewards (RLVR) improves LLM reasoning using scalar rewards (Chen et al., 2025; DeepSeek-AI et al., 2025; Yang et al., 2025), and GRPO (Shao et al., 2024) computes group-relative advantages without a value function. Expensive and slow generation motivates rollout reuse, making the data off-policy in training: the current policy πθ _θ differs from the old policy πold _old that generated the responses. The sequence importance ratio W=πθ(∣)/πold(∣)W= _θ( o q)/ _old( o q) measures this mismatch on the prompt q and response o. A small ratio W means that the response is now less likely to be generated by the current policy than the old policy, while a large W means that it is more likely. GRPO clips the token importance ratio wt=πθ(ot|,<t)/πold(ot|,<t)w_t= _θ(o_t| q, o_<t)/ _old(o_t| q, o_<t) and GSPO (Zheng et al., 2025) clips the length-normalized ratio s=W1/||s=W^1/| o| to stabilize training. Both methods use clipping, so their in-region gradients scale linearly with the importance ratio. This linear weighting starves informative gradients in opposite ways depending on the advantage sign. We call responses with A^>0 A>0 positive responses and those with A^<0 A<0 negative responses, according to their group-relative reward. For positive responses, the gradient is proportional to wtw_t, so already familiar tokens near the upper clip threshold receive the largest updates, while under-generated tokens with wt≪1w_t 1 receive almost none. For negative responses, heavily over-generated tokens with wt≫1w_t 1 remain unclipped, so their unbounded weight can overshadow the gradient contributions from other tokens. The same failure carries over to the sequence-level ratio used by GSPO. We refer to this sign-dependent misallocation of learning signal as gradient starvation. We tackle the failures by reshaping the importance weight W at the sequence level. Instead of clipping, we replace the raw W with a smooth, sign-dependent kernel ϕ(±)(W)φ^(±)(W), and the two failures above determine the shape we need: ϕ(+)(0)>0φ^(+)(0)>0 must preserve under-generated positive responses, while ϕ(−)(0)=ϕ(−)(∞)=0φ^(-)(0)=φ^(-)(∞)=0 must suppress both already-suppressed and severely over-generated negative responses. GRPO and GSPO clipping miss these critical tails: their effective weight vanishes on the low-ratio tail but grows without bound on the high-ratio tail. VESPO (Shen et al., 2026) suppresses both tails, fitting negative advantages but starving positive ones at W→0W→ 0. We propose ReSPO (Reshaped Sequence Policy Optimization), which derives a smooth two-branch kernel from an α-divergence variational objective with exponential tilting for variance control. The choice α(+)>1α^(+)>1 gives the required positive tail, while the α(−)=1α^(-)=1 limit gives both required negative tails. On DAPO-MATH (Yu et al., 2025) with Qwen3-1.7B-Base and Qwen3-30B-A3B-Base, ReSPO attains a higher late-stage training score than every clipped baseline in all six model–N settings and a higher score than VESPO in five of six. 2 Related Work Policy optimization for LLM reasoning. PPO (Schulman et al., 2017) is the standard method for RLHF (Ouyang et al., 2022), using a clipped surrogate for stable updates. With automated verifiers (DeepSeek-AI et al., 2025), value-free alternatives have become popular. GRPO (Shao et al., 2024) computes group-relative advantages and clips per-token ratios; DAPO (Yu et al., 2025) adds decoupled clipping and dynamic sampling; and REINFORCE++ (Hu et al., 2025) uses global advantage normalization. These methods focus on token-level updates, whereas we study sequence-level gradient starvation. REAL (Zhai et al., 2026) reframes gradient starvation as a classification problem; we retain importance sampling to keep off-policy staleness explicit. This distinction matters increasingly as rollout reuse makes each batch more off-policy over successive optimizer steps. Sequence-level and off-policy optimization. Token-level importance correction is mismatched with sequence-level rewards. GSPO (Zheng et al., 2025) uses a length-normalized sequence ratio for stable MoE training, while VESPO (Shen et al., 2026) derives ϕKL(W)=Wβexp(λ(1−W)) _KL(W)=W^β (λ(1-W)) from a KL variational objective. Reusing rollouts across mini-batches causes off-policy drift (Noukhovitch et al., 2025), especially for MoE models (Zheng et al., 2025); Routing Replay (Ma et al., 2025) and truncated importance sampling (Liu et al., 2025) have been proposed to improve system stability. ReSPO complements these techniques by controlling how the importance weight allocates gradient across responses. We generalize VESPO’s framework to the α-divergence family with a sign-dependent two-branch kernel. Sec. 3.3.4 and Fig. 1 compare these kernel designs. Trust regions and divergence measures. TRPO (Schulman et al., 2015) limits updates with a KL bound, and PPO approximates this with hard clipping. Softer options include probability smoothing (Dwyer et al., 2025) and adaptive gating (Gao et al., 2025). Rényi-divergence bounds have also been used to analyze the variance of importance-sampling estimators in policy optimization (Metelli et al., 2018). ReSPO instead uses separable power-Bregman geometry to derive a sequence-level reshaping family. Its two branches allocate gradient according to the advantage sign and the importance-weight tail behavior that matters for off-policy optimization. 3 Method 3.1 Preliminaries: Token-Level and Sequence Policy Optimization LLM and GRPO. A large language model (LLM) parameterized by θ defines an autoregressive policy πθ(∣)=∏t=1Tπθ(ot∣,o<t) _θ( o q)= _t=1^T _θ(o_t q,o_<t) that generates a response =(o1,…,oT) o=(o_1,…,o_T) for a prompt q. In the RLVR setting, a deterministic scalar reward R(,)R( q, o) gives a sequence-level signal. GRPO samples G responses ii=1G\ o_i\_i=1^G from the old policy πold _old and forms the group-normalized advantage A^i=(R(,i)−meanjR(,j))/(stdjR(,j)+ϵA) A_i=(R( q, o_i)-mean_jR( q, o_j))/(std_jR( q, o_j)+ _A); implementation details are given in App. C. GRPO then optimizes a token-level clipped surrogate: GRPO(θ)=,i[1G∑i=1G1|i|∑t=1|i|min(wi,tA^i,clip(wi,t,1−ε,1+ε)A^i)],J_GRPO(θ)=E_ q,\ o_i\ [ 1G _i=1^G 1| o_i| _t=1^| o_i| \! (w_i,t\, A_i,\;clip(w_i,t,1- ,1+ )\, A_i ) ], (1) where wi,t=πθ(oi,t∣,i,<t)/πold(oi,t∣,i,<t)w_i,t= _θ(o_i,t q, o_i,<t)/ _old(o_i,t q, o_i,<t) is the per-token importance ratio. Because the reward is defined at the sequence level, applying a separate single-sample ratio to each token does not recover the exact sequence-level importance correction and can introduce noisy token-wise scaling as sequence length grows (Shen et al., 2026; Zheng et al., 2025). Routing fluctuations create an additional source of training–inference mismatch in MoE models, motivating stabilization techniques such as Routing Replay (Ma et al., 2025; Zheng et al., 2025). These instabilities compound as policy staleness grows. Sequence-level ratio: GSPO. A natural fix is to work on the sequence-level importance weight W()=∏twt=πθ(∣)/πold(∣)W( o)= _tw_t= _θ( o q)/ _old( o q), which aggregates token ratios at the same granularity as the reward. GSPO (Zheng et al., 2025) puts this view into practice with a length-normalized geometric-mean ratio si(θ)=(πθ(i∣)/πold(i∣))1/|i|s_i(θ)=( _θ( o_i q)/ _old( o_i q))^1/| o_i| and a clipped surrogate. GSPO(θ)=,i[1G∑i=1Gmin(si(θ)A^i,clip(si(θ),1−εlow,1+εhigh)A^i)].J_GSPO(θ)=E_ q,\ o_i\\! [ 1G _i=1^G \! (s_i(θ)\, A_i,\;clip(s_i(θ),1- _low,1+ _high)\, A_i ) ]. (2) GSPO trains more stably than GRPO, notably without Routing Replay on MoE models (Zheng et al., 2025), so the sequence-level weight is a natural starting point for off-policy reshaping. However, hard clipping of sis_i keeps GRPO’s gradient starvation inside the unclipped region. Moreover, the normalization W1/|i|W^1/| o_i| induces length-dependent bias because it tempers the sequence importance ratio by an exponent that depends on response length (see App. A.6). 3.2 Gradient Starvation Let ρ=wtρ=w_t in GRPO and ρ=siρ=s_i in GSPO. We call a response positive when the estimated advantage A^>0 A>0 and negative when A^<0 A<0. Inside the unclipped region, both objectives weight the gradient linearly by ρ. For GRPO and GSPO, the gradients are ∇θGRPO∝A^wt∇θlogπθ(ot∣,<t),and∇θGSPO∝A^isi|i|∑t=1|i|∇θlogπθ(oi,t∣,i,<t). _θJ_GRPO Aw_t _θ _θ(o_t q, o_<t),\ \ and\ \ _θJ_GSPO A_is_i| o_i| _t=1^| o_i| _θ _θ(o_i,t q, o_i,<t). Apart from the length factor in GSPO, the gradient magnitude of either objective grows linearly with |A^|ρ| A|ρ. Clipping discards large-ratio positive and small-ratio negative updates, but leaves two failures (Zhai et al., 2026): • Starved positive update. When A^>0 A>0 and ρ→0ρ→ 0, a positive response is becoming less likely, but its recovery signal vanishes: wt=0.01w_t=0.01 receives 100×100× less weight than wt=1w_t=1. • Dominating negative update. When A^<0 A<0 and ρ→∞ρ→∞, a negative response is becoming more likely. This side remains unclipped, so its weight grows without bound and can dominate repeated off-policy updates. ReSPO addresses these GRPO failures by reshaping W. The likelihood shift encoded by W, together with the advantage sign, gives the four cases below. How ReSPO should treat each tail. ↑ means “reinforce” and ↓ means “suppress” Response W→0W→ 0: less likely now W→∞W→∞: more likely now A^>0 A>0 ↑ : positive response being lost ↓ : positive response already reinforced A^<0 A<0 ↓ : negative response already suppressed ↓ : negative response being amplified Therefore, ReSPO requires ϕ(+)(0)>0φ^(+)(0)>0 and ϕ(+)(∞)=ϕ(−)(0)=ϕ(−)(∞)=0φ^(+)(∞)=φ^(-)(0)=φ^(-)(∞)=0. 3.3 ReSPO: Reshaped Sequence Policy Optimization To address gradient starvation, ReSPO applies a soft kernel to the sequence-level importance weight. The derivation has three stages: (i) interpret reshaping as a change of measure; (i) use an α-divergence objective to obtain an unconstrained kernel ϕ0 _0; and (i) project this measure under a moment constraint, producing a smooth exponential tilt ϕφ. We then choose α by advantage sign so that the two branches satisfy the requirements above; App. A provides the full derivations. 3.3.1 Weight Reshaping as Measure Change To find a ϕφ that satisfies the properties stated in Sec. 3.2, we first ask what replacing the raw importance weight W with a generic reshaped weight ϕ(W)φ(W) actually does to the gradient. Let μ=πoldμ= _old denote the behavior policy from which rollouts are sampled and π=πθπ= _θ the current target policy, so that the sequence-level importance weight defined in Sec. 3.1 is W()=π(∣)/μ(∣)W( o)=π( o q)/μ( o q). For the reward function R(⋅)R(·), the policy gradient of ∼πR()E_ o πR( o) is ∇θ∼πR()=∼πR()∇θlogπθ()=∼μW()⋅R()⋅∇θlogπθ(). _θE_ o πR( o)=E_ o πR( o) _θ _θ( o)=E_ o μW( o)· R( o)· _θ _θ( o)\,. (3) Applying any reshaping function ϕ:ℝ+→ℝ+φ:R_+ _+ to W implicitly defines an unnormalized measure τ. The measure-change identity is ∼μ[ϕ(W())⋅R()⋅∇θlogπθ()]=∑τ()R()⋅∇θlogπθ(),E_ o μ\! [φ(W( o))· R( o)· _θ _θ( o) ]= _ oτ( o)\,R( o)· _θ _θ( o), (4) where τ is given by τ()=μ()ϕ(W()).τ( o)=μ( o)\,φ(W( o)). (5) Eq. (5) fixes the notation used below: W is the raw sequence ratio, ϕφ is its reshaping kernel, and τ=μϕτ=μφ is an unnormalized measure. Let Z:=∑τ()=μ[ϕ(W)]Z:= _ oτ( o)=E_μ[φ(W)] and τ¯():=τ()/Z τ( o):=τ( o)/Z. Then only τ¯ τ is a probability distribution, and ∑τ()g()=Z∼τ¯[g] _ oτ( o)g( o)=Z\,E_ o τ[g]. This reframing suggests a design strategy. Rather than hand-picking ϕφ as clipping does, we first solve for an unnormalized τ, read off ϕφ through Eq. (5), and finally verify its tails against the requirements in Sec. 3.2. 3.3.2 Variational Derivation of the Reshaping Kernel Our goal is to design a measure τ that balances three goals: staying close to the behavior distribution μ for sample efficiency, moving toward the target policy π to reduce bias, and controlling variance for training stability. We begin with a brief review of the Bregman divergence. Bregman divergence and the dual-proximity objective. Let f:ℝ+→ℝf:R_+ be a strictly convex function, which we call the Bregman generator. The Bregman divergence induced by f for (unnormalized) measures p,q≥0p,q≥ 0 is Df(p∥q)=∑[f(p())−f(q())−f′(q())(p()−q())].D_f(p\,\|\,q)= _ o [f (p( o) )-f (q( o) )-f (q( o) )\, (p( o)-q( o) ) ]. (6) Borrowing the idea from annealed importance sampling (Neal, 2001), we formulate the dual-proximity objective for a general Bregman divergence: minτ≥0(1−β)Df(τ∥μ)+βDf(τ∥π), _τ≥ 0\;\;(1-β)\,D_f(τ\,\|\,μ)\;+\;β\,D_f(τ\,\|\,π), (7) where β∈(0,1)β∈(0,1) controls the trade-off. At β=0β=0 we aim to recover the behavior policy (τ=μτ=μ, zero importance-weight variance but maximum bias), and at β=1β=1 we target the current policy (τ=πτ=π, unbiased but with possibly high variance). Throughout this paper, τ is an unnormalized measure; only τ¯ τ denotes its normalized probability distribution. See App. A.1 for discussion. Choice of Bregman generator. The choice of f determines the geometry of the interpolation and, with it, the form and tail behavior of the reshaping kernel. KL divergence (f(x)=xlogx−xf(x)=x x-x). This is the choice in VESPO (Shen et al., 2026). The first-order condition of Eq. (7) gives the geometric mixture τKL∗()=μ()1−β⋅π()βτ^*_KL( o)=μ( o)^1-β·π( o)^β, with reshaping kernel ϕKL(W)=Wβ _KL(W)=W^β. This is also the continuous α→1α→ 1 limit of the power-mean family below. After exponential tilting it provides the two vanishing tails required by the negative branch, but its value at W→0W→ 0 cannot satisfy the positive branch. α-divergence (f(x)=1α(α−1)xαf(x)= 1α(α-1)x^α, α≠0,1α≠ 0,1). f is strictly convex on ℝ+R_+ for all α∉0,1α∉\0,1\. Solving the first-order condition gives the power-mean interpolation τ0∗()=[(1−β)μ()α−1+βπ()α−1]1α−1,τ^*_0( o)= [(1-β)\,μ( o)^α-1+β\,π( o)^α-1 ] 1α-1, (8) and substituting π()=μ()W()π( o)=μ( o)\,W( o) together with Eq. (5) yields the corresponding unconstrained reshaping kernel ϕ0(W)=[(1−β)+βWα−1]1/(α−1). _0(W)= [(1-β)+β\,W^α-1 ]^1/(α-1). (9) This kernel is valid for both 0<α<10<α<1 and α>1α>1 and satisfies ϕ0(1)=1 _0(1)=1 by construction. The full first-order derivation is given in App. A.1 for completeness. 3.3.3 Exponential Tilt for Variance Control The unconstrained kernel ϕ0 _0 from Eq. (9) redistributes gradient weight and requires a separate variance-control step. Since τ=μϕτ=μ\,φ, the natural second-moment target is μ[ϕ(W)2]=∑τ()ϕ(W())E_μ[φ(W)^2]= _ oτ( o)φ(W( o)). The final ϕφ is still unknown, so we replace it inside the constraint by the fixed proxy ϕ0 _0 and require ∑τ()ϕ0(W())≤C1 _ oτ( o) _0(W( o))≤ C_1. This proxy is sufficient because the resulting tilt leads to bounded μ[ϕ(W)2]E_μ[φ(W)^2], as shown in App. A.2. We additionally impose ∑τ()≤C2 _ oτ( o)≤ C_2 to prevent the measure from having unbounded mass. For this projection step, we use the generalized KL divergence for unnormalized measures p,q≥0p,q≥ 0: DKL(p∥q)=∑(p()logp()q()−p()+q())D_KL(p\|q)= _ o (p( o) p( o)q( o)-p( o)+q( o) ). We find the closest measure to τ0∗ _0^* that satisfies both constraints in this KL geometry (Csiszár, 1975): τ∗=argminτDKL(τ∥τ0∗)s.t.∑τ()ϕ0(W())≤C1,∑τ()≤C2.τ^*= _τ\;D_KL(τ\,\|\, _0^*) .t. _ oτ( o) _0(W( o))≤ C_1, _ oτ( o)≤ C_2. (10) The KKT conditions for this KL projection give an exponential tilt of τ0∗ _0^* with sufficient statistic ϕ0(W) _0(W): τ∗()∝τ0∗()⋅exp(−λϕ0(W())),τ^*( o) _0^*( o)· \! (-λ\, _0(W( o)) ), (11) where λ≥0λ≥ 0 is the Lagrange multiplier. Extracting the reshaping kernel via τ=μϕ(W)τ=μ\,φ(W) and using ϕ0(1)=1 _0(1)=1 to normalize ϕ(1)=1φ(1)=1 gives ϕ(W)=ϕ0(W)⏟α reshaping⋅exp(λ(1−ϕ0(W)))⏟exponential tilt.φ(W)= _0(W)_$α$ reshaping· \! (λ\,(1- _0(W)) )_exponential tilt. (12) The two divergences therefore play different roles: the α-divergence determines the power-mean interpolation and its tails, while the KL projection turns a linear moment constraint into a smooth multiplicative tilt rather than a hard boundary (App. A.2). As α→1+α→ 1^+, ϕ0(W)→Wβ _0(W)→ W^β (App. A.4) and the kernel becomes Wβexp(λ(1−Wβ))W^β (λ(1-W^β)). This differs from VESPO (Shen et al., 2026), whose exponential factor acts on the raw weight W rather than the reshaped weight WβW^β. 3.3.4 Tail Behavior and the Two-Branch Design The kernel of Eq. (12) depends on three hyperparameters (α,β,λ)(α,β,λ), and its tail behavior determines whether the gradient requirements of Sec. 3.2 are met. We extend the power-mean kernel continuously to α=1α=1 by defining ϕ0(W)=Wβ _0(W)=W^β (App. A.4). The key observation is that no single value of α satisfies both the positive- and negative-advantage requirements simultaneously. Tail analysis. For λ>0λ>0, the tilt sends ϕ(W)φ(W) to zero whenever ϕ0(W) _0(W) diverges. For α>1α>1, ϕ(0)>0φ(0)>0 and ϕ(∞)=0φ(∞)=0, matching the positive-branch requirements. For 0<α<10<α<1, ϕ(0)=0φ(0)=0 but ϕ(∞)>0φ(∞)>0, so extremely over-generated negative responses retain nonzero weight, and these outliers could impact the training. At the boundary α=1α=1, ϕ(0)=ϕ(∞)=0φ(0)=φ(∞)=0, exactly matching the negative-branch requirements. This is where the sign-specific design enters the variational construction. The ReSPO kernel. We use a sign-dependent parameterization: α(+)>1α^(+)>1 for positive advantages and the α(−)=1α^(-)=1 limiting form for negative advantages, both with 0<β(±)<10<β^(±)<1. The full ReSPO reshaping kernel is ϕ(W,A^)=ϕ0(±)(W)⋅exp(λ(±)(1−ϕ0(±)(W))),φ(W; A)= _0^(±)(W)· \! (λ^(±)\,(1- _0^(±)(W)) ), (13) where (+)(+) applies when A^≥0 A≥ 0 and (−)(-) applies when A^<0 A<0, and ϕ0(+)(W)=[(1−β(+))+β(+)Wα(+)−1]1/(α(+)−1),ϕ0(−)(W)=Wβ(−). _0^(+)(W)= [(1-β^(+))+β^(+)W^α^(+)-1 ]^1/(α^(+)-1), _0^(-)(W)=W^β^(-). (14) The negative branch is the α(−)→1α^(-)→ 1 limit of the same family. Both branches satisfy ϕ(1)=1φ(1)=1, and together they realize all four sign-specific tail requirements. Combining the kernel with group-normalized advantages gives the ReSPO policy gradient for a sampled batch from Eq. (4): ∇θReSPO=,i∼πold[1G∑i=1Gsg(ϕ(Wi,A^i))⋅A^i⋅∑t=1|i|∇θlogπθ(oi,t∣,i,<t)], _θJ_ReSPO=E_ q,\,\ o_i\ _old [ 1G _i=1^G sg (φ(W_i; A_i) )· A_i· _t=1^| o_i| _θ _θ(o_i,t q, o_i,<t) ], (15) Here sg(⋅) sg(·) is the stop-gradient operator and Wi=∏t=1|i|wi,tW_i= _t=1^| o_i|w_i,t is the sequence-level importance weight for response i. We apply the kernel to the unnormalized ratio W, because the variational derivation (Sec. 3.3.2) operates on distributions over sequences and W is the natural Radon–Nikodym derivative; length normalization in GSPO would distort the geometry. The detached weight ϕ(Wi,A^i)φ(W_i; A_i) acts as a per-sequence gradient scaling coefficient in a REINFORCE-style estimator. For an on-policy update, W=1W=1 and both branches reduce to unit weight (App. A.5). App. B gives the numerically stable log-space form, and Alg. 1 summarizes the complete update. 3.3.5 Hyperparameter Selection The three parameters have separate roles: α determines the tail regime, β sets the behavior–target interpolation, and λ controls the exponential tilt. The tail analysis fixes α by advantage sign. We set β(±)=0.5β^(±)=0.5 for a symmetric interpolation, then choose λ(±)λ^(±) according to the desired local shape. Positive branch: α(+)=2α^(+)=2, β(+)=0.5β^(+)=0.5, λ(+)=2λ^(+)=2. At α(+)=2α^(+)=2, the power mean reduces to the linear form ϕ0(+)(W)=(1−β(+))+β(+)W _0^(+)(W)=(1-β^(+))+β^(+)W. Setting β(+)=0.5β^(+)=0.5 gives the arithmetic mean ϕ0(+)(W)=12(1+W) _0^(+)(W)= 12(1+W), treating μ and π symmetrically. We require ϕ(+)(0)′=0φ^(+) (0)=0, which gives λ(+)=1/(1−β(+))=2λ^(+)=1/(1-β^(+))=2 and makes W=0W=0 the global maximum. Although an importance weight cannot be negative, the linear closed form extends to the real line and has its unique maximum at zero (App. A.3). The resulting kernel is ϕ(+)(W)=1+W2exp(1−W)φ^(+)(W)= 1+W2 (1-W), which decreases strictly for W>0W>0 and has maximum ϕ(+)(0)=12e≈1.36φ^(+)(0)= 12e≈ 1.36. Negative branch: α(−)=1α^(-)=1, β(−)=0.5β^(-)=0.5, λ(−)=2λ^(-)=2. The choice α(−)=1α^(-)=1 is made so that ϕ(−)(0)=ϕ(−)(∞)=0φ^(-)(0)=φ^(-)(∞)=0. With β(−)=0.5β^(-)=0.5, its unconstrained kernel is the geometric mean ϕ0(−)(W)=W _0^(-)(W)= W. We choose λ(−)λ^(-) by matching the two branches at the on-policy point: ϕ(+)(1)′=−12φ^(+) (1)=- 12, while ϕ(−)(1)′=β(−)(1−λ(−))φ^(-) (1)=β^(-)(1-λ^(-)). Equating the derivatives gives λ(−)=2λ^(-)=2 (App. A.3) and ϕ(−)(W)=Wexp(2(1−W))φ^(-)(W)= W \! (2(1- W) ). Thus the branches match in value and slope at W=1W=1. Fig. 1 shows that the negative kernel peaks at W∗=14W^*= 14 with value e/2e/2 and vanishes on both tails. Tail behavior Method ϕ(+)φ^(+) (0,∞)(0,∞) ϕ(−)φ^(-) (0,∞)(0,∞) Clipping (0,0)(0,0) (0,∞)(0,∞) VESPO (0,0)(0,0) (0,0)(0,0) ReSPO (e/2,0)(e/2,0) (0,0)(0,0) Figure 1: Kernel shapes and tail limits (ε=0.2 =0.2). The adjacent table gives the limits as W→(0,∞)W→(0,∞). ReSPO preserves positive-branch signal near zero while suppressing both negative tails. 4 Experiments We evaluate ReSPO on mathematical reasoning with one dense model, Qwen3-1.7B-Base, and one mixture-of-experts (MoE) model, Qwen3-30B-A3B-Base. For each model we vary the rollout-reuse ratio N to study how the methods degrade as off-policy drift grows. 4.1 Experimental Setup Training & evaluation protocol. All experiments use verl (Sheng et al., 2025), asynchronous vLLM rollouts (Kwon et al., 2023), mini-batch size M=32M=32, G=8G=8 rollouts per prompt, and the GRPO advantage estimator. The dense model uses FSDP with AdamW, whereas the MoE model uses Megatron with Adam; both use a learning rate of 10−610^-6 and no KL penalty. Tab. 2 lists the shared training settings. Both models are trained on DAPO-MATH-17k (Yu et al., 2025); we use a strict-box verifier as the training reward (App. D.1). The prompt limit is 1,0241,024 tokens, and the response limits are 15,36015,360 and 8,1928,192 for Qwen3-1.7B and Qwen3-30B-A3B. The 30B setting adds a soft length penalty: it is zero through 4,0964,096 tokens and decreases linearly to −1-1 at the 8,1928,192-token limit, encouraging concise responses without truncating them at 4,0964,096. We set the global batch size to NMNM and split each rollout batch into N consecutive mini-batches. Thus larger N reuses the same πold _old for more updates and induces more off-policy drift. We evaluate N∈8,16,32N∈\8,16,32\ and fix every run to 10241024 policy updates, corresponding to 32,76832,768 training prompts. Our evaluation uses the full test splits of AIME 2025, AIME 2024, and AMC 2023 (OpenCompass, 2025; Maxwell-Jia, 2024; math-ai, 2023), together with OlympiadBench (He et al., 2024), MinervaMath (Lewkowycz et al., 2022), and MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024). We sample with temperature 1.01.0, top-p=0.7p=0.7, and no top-k truncation, allowing a maximum response length of 16,38416,384. For each problem, we generate 1616 responses on AIME 2025, AIME 2024, and AMC 2023, and 44 on each remaining benchmark; each completion is scored using Math-Verify (Kydlíček, 2024) (App. D.2). Baselines, metrics, and reporting. We compare with GRPO (Shao et al., 2024), GSPO (Zheng et al., 2025), and VESPO (Shen et al., 2026) under matched data and optimizer settings. Tabs. 2 and 3 list the shared and method-specific settings, respectively. In particular, we use VESPO’s published best-performing sign-specific setting, (β(+),λ(+))=(2,3)(β^(+),λ^(+))=(2,3) and (β(−),λ(−))=(3,2)(β^(-),λ^(-))=(3,2). Tab. 1 reports per-benchmark results at N∈8,16,32N∈\8,16,32\. For training curves, we apply the affine transformation (score+1)/2( score+1)/2 to the recorded DAPO-MATH score and plot it against the number of policy updates, making experiments with different N directly comparable (App. D.1). We call this quantity training accuracy for 1.7B and penalized training accuracy for 30B. We report the maximum logged score within the first 256256 policy updates without an error bar and the policy-step-weighted trapezoidal mean over the last 128128 policy updates with its within-run temporal standard deviation. Tab. 1 uses a cluster-bootstrap standard error over evaluation problems (App. D.3). We discuss cross-seed standard deviations in Sec. 4.3. 4.2 Training and Evaluation Performance Fig. 3 and Tab. 1 compare optimization and generalization across both models and all three values of N. Figs. 5 and 6 additionally report evaluation accuracy, response length, approximate KL divergence, and entropy throughout training. Normalized training score and held-out pass@1 show the same broad pattern: the observed advantage of ReSPO is generally largest at N=32N=32. Training performance. ReSPO improves quickly and remains higher through most of the training trajectory. It has the highest observed early peak in five of six settings; for 30B at N=8N=8, its peak is 1.01.0 p below GRPO. For N∈16,32N∈\16,32\, its early-peak differences from the highest baseline are 1.01.0 and 3.03.0 p on 1.7B and 4.24.2 and 5.45.4 p on 30B. ReSPO also has the highest observed last-128128 mean in five of six settings; on 1.7B at N=16N=16, it is 0.40.4 p below VESPO, and each central value lies within the other’s temporal range. Its late mean exceeds the highest clipped baseline in all six settings by 4.24.2–8.48.4 p, with non-overlapping temporal ranges. At N=32N=32, these differences reach 7.97.9 p on 1.7B and 8.48.4 p on 30B. App. E.1 gives the complete model- and N-specific comparison. Early peak (t≤256t≤ 256) N GRPO GSPO VESPO ReSPO Qwen3-1.7B 88 0.1690.169 0.1580.158 0.1610.161 0.1920.192 1616 0.1560.156 0.1270.127 0.1510.151 0.1660.166 3232 0.1180.118 0.1060.106 0.1270.127 0.1570.157 Qwen3-30B-A3B 88 0.4780.478 0.4160.416 0.4140.414 0.4680.468 1616 0.4030.403 0.3690.369 0.3860.386 0.4450.445 3232 0.3620.362 0.3410.341 0.3640.364 0.4180.418 Late mean (last 128 steps) N GRPO GSPO VESPO ReSPO Qwen3-1.7B 88 0.213±0.0170.213_ ± 0.017 0.209±0.0150.209_ ± 0.015 0.275±0.0160.275_ ± 0.016 0.286±0.0160.286_ ± 0.016 1616 0.210±0.0070.210_ ± 0.007 0.199±0.0090.199_ ± 0.009 0.268±0.0090.268_ ± 0.009 0.264±0.0100.264_ ± 0.010 3232 0.197±0.0050.197_ ± 0.005 0.184±0.0030.184_ ± 0.003 0.230±0.0040.230_ ± 0.004 0.276±0.0040.276_ ± 0.004 Qwen3-30B-A3B 88 0.530±0.0150.530_ ± 0.015 0.538±0.0130.538_ ± 0.013 0.538±0.0140.538_ ± 0.014 0.595±0.0160.595_ ± 0.016 1616 0.498±0.0100.498_ ± 0.010 0.521±0.0080.521_ ± 0.008 0.531±0.0120.531_ ± 0.012 0.563±0.0210.563_ ± 0.021 3232 0.504±0.0050.504_ ± 0.005 0.493±0.0060.493_ ± 0.006 0.523±0.0080.523_ ± 0.008 0.588±0.0070.588_ ± 0.007 Figure 2: Normalized training-score curves and summaries. Tables report the peak through step 256256 and the last-128128 mean. Early peaks have no error bars; late-mean subscripts are temporal standard deviations. Bold marks each row’s maximum; for late means, it also marks methods whose central value and the maximum each fall within the other’s range. Figure 3: 1.7B accuracy across shared response-length bins. Q1–Q9 partition uncapped responses, while Q10 contains responses at the 16,38416,384-token cap. ReSPO has the highest accuracy in 88 of 1010 bins at each N. Model N Method AIME25 AIME24 AMC23 Olymp. Minerva MATH500 Avg. Qwen3 1.7B 8 GRPO 0.042 0.075 0.353 0.284 0.286 0.665 0.284±0.0130.284_ ± 0.013 GSPO 0.062 0.077 0.402 0.304 0.272 0.658 0.296±0.0130.296_ ± 0.013 VESPO 0.073 0.127 0.478 0.388 0.307 0.742 0.353±0.0150.353_ ± 0.015 ReSPO 0.090 0.133 0.484 0.368 0.289 0.726 0.348±0.0160.348_ ± 0.016 16 GRPO 0.071 0.083 0.431 0.303 0.278 0.663 0.305±0.0140.305_ ± 0.014 GSPO 0.046 0.075 0.419 0.319 0.283 0.672 0.302±0.0130.302_ ± 0.013 VESPO 0.065 0.106 0.464 0.364 0.301 0.712 0.335±0.0140.335_ ± 0.014 ReSPO 0.094 0.123 0.467 0.358 0.299 0.717 0.343±0.0150.343_ ± 0.015 32 GRPO 0.052 0.040 0.383 0.289 0.278 0.653 0.283±0.0120.283_ ± 0.012 GSPO 0.046 0.054 0.381 0.299 0.299 0.663 0.290±0.0130.290_ ± 0.013 VESPO 0.081 0.131 0.456 0.341 0.282 0.698 0.332±0.0150.332_ ± 0.015 ReSPO 0.088 0.117 0.488 0.367 0.280 0.729 0.345±0.0150.345_ ± 0.015 Qwen3 30B-A3B 8 GRPO 0.152 0.248 0.738 0.511 0.436 0.852 0.490±0.0180.490_ ± 0.018 GSPO 0.181 0.267 0.713 0.521 0.439 0.861 0.497±0.0180.497_ ± 0.018 VESPO 0.154 0.273 0.700 0.517 0.454 0.853 0.492±0.0180.492_ ± 0.018 ReSPO 0.252 0.306 0.791 0.427 0.411 0.783 0.495±0.0190.495_ ± 0.019 16 GRPO 0.138 0.246 0.733 0.490 0.424 0.831 0.477±0.0170.477_ ± 0.017 GSPO 0.156 0.265 0.700 0.493 0.452 0.848 0.486±0.0180.486_ ± 0.018 VESPO 0.152 0.300 0.700 0.515 0.436 0.854 0.493±0.0180.493_ ± 0.018 ReSPO 0.213 0.321 0.792 0.512 0.457 0.857 0.525±0.0180.525_ ± 0.018 32 GRPO 0.123 0.313 0.717 0.503 0.444 0.850 0.492±0.0170.492_ ± 0.017 GSPO 0.146 0.231 0.664 0.475 0.423 0.835 0.462±0.0180.462_ ± 0.018 VESPO 0.177 0.277 0.709 0.500 0.444 0.852 0.493±0.0180.493_ ± 0.018 ReSPO 0.202 0.269 0.795 0.510 0.451 0.844 0.512±0.0180.512_ ± 0.018 Table 1: Final-checkpoint pass@1. Bold marks the highest mean within each model–N block. Evaluation performance. At N=8N=8, the macro averages are similar relative to their evaluation sampling uncertainty. For N∈16,32N∈\16,32\, ReSPO has the highest observed mean in all four model–N settings, exceeding the highest baseline by 0.80.8 and 1.31.3 p on 1.7B and by 3.23.2 and 1.91.9 p on 30B. ReSPO has the highest value in 99 of the 1212 model–benchmark comparisons at N=16N=16 and 88 at N=32N=32. The pattern is most consistent on difficult competition benchmarks, where it has the highest AIME25 and AMC23 values in every model–N setting. Averaged over AIME25, AIME24, and AMC23 on 30B, its differences from the highest baseline are 6.36.3, 5.85.8, and 3.43.4 p for N∈8,16,32N∈\8,16,32\, respectively. App. E.1 provides a detailed summary of these comparisons. Accuracy across response lengths. Using 1010 quantile-based response-length bins shared across methods and benchmarks for each N (constructed in App. E.2), accuracy decreases toward the long-response tail for all methods (Fig. 3). ReSPO has the highest pooled accuracy in 2424 of the 3030 bins. At N=8N=8, it is highest in Q1–Q3 and Q5–Q9. At N=16N=16, ReSPO is highest in Q1–Q6 and Q9–Q10. The lead is largest at N=32N=32: ReSPO exceeds the highest baseline from Q1 through Q7 by 8.48.4, 16.016.0, 24.424.4, 21.621.6, 20.820.8, 15.115.1, and 8.58.5 p, respectively. The shared-bin comparison therefore shows that ReSPO’s observed advantage persists within response-length ranges and is generally largest across short and medium responses at N=32N=32. In App. E.2, Tab. 4 gives the token boundaries and method-specific bin counts and Fig. 4 gives the complete per-benchmark results. 4.3 Discussion Early learning from longer positive reasoning trajectories. The complete training trajectories in App. G complement the positive-branch diagnostics in App. E.3. Particularly at N=32N=32, ReSPO’s response length and training score begin to separate from the baselines during the same early stage of optimization (Fig. 5). To examine this initial rapid improvement, App. E.3 focuses on the first 512512 policy updates and the relationship between low-W positive responses and rapid response-length growth. Because logW W accumulates token-level policy drift across a response, longer responses tend to enter the low-W tail. The diagnostic shows that ReSPO retains more coefficient mass in this tail while mean positive-response length grows faster. Together, these observations indicate that ReSPO can effectively learn from longer positive reasoning trajectories during the earliest training steps, when retaining their signal is most important for rapid improvement. On 30B, this early advantage remains 4.24.2 and 5.45.4 p over the strongest baseline at N=16N=16 and N=32N=32, respectively, despite the soft length penalty. ReSPO also remains more accurate within shared response-length ranges (Fig. 3), indicating that its advantage extends beyond the change in length distribution. Robustness across seeds. To assess the robustness of the observed differences at N=32N=32, we augment each primary ReSPO run used in Fig. 3 and Tab. 1 with two additional seeds. All means and cross-seed population standard deviations below are computed over the three runs. On Qwen3-1.7B, the three runs attain a last-128128 mean of 27.07%27.07\%, with a standard deviation of 0.540.54 p. This mean exceeds VESPO (23.00%23.00\%), GRPO (19.70%19.70\%), and GSPO (18.41%18.41\%) by 4.074.07, 7.377.37, and 8.668.66 p. On Qwen3-30B-A3B, the three runs attain a last-128128 mean of 57.92%57.92\%, with a standard deviation of 0.950.95 p. This mean exceeds VESPO (52.30%52.30\%), GRPO (50.40%50.40\%), and GSPO (49.28%49.28\%) by 5.625.62, 7.527.52, and 8.648.64 p. On the final-checkpoint evaluations averaged over AIME25, AIME24, and AMC23, the three-run mean is 41.94%41.94\% with a standard deviation of 0.470.47 p, exceeding VESPO (38.77%38.77\%), GRPO (38.43%38.43\%), and GSPO (34.70%34.70\%) by 3.173.17, 3.513.51, and 7.247.24 p. Across both model scales, the standard deviations are substantially smaller than ReSPO’s performance gains over every baseline. Combination with Routing Replay. Routing Replay (Ma et al., 2025), which reuses rollout-time expert-routing decisions during optimization, is orthogonal to ReSPO’s control of sequence-level importance weights. In a matched 30B model ablation at N=16N=16, adding R3 increases the mean penalized training accuracy over the last 128128 policy steps from 56.32%56.32\% to 58.16%58.16\%. In this comparison, Routing Replay improves late-stage training score and smoothness; App. E.4 and Tabs. 8 report the complete comparison. 5 Conclusion We identify a sign-dependent gradient-allocation problem in off-policy learning: clipping suppresses under-generated positive responses on the low-importance-weight tail while allowing severely over-generated negative responses to dominate on the high-weight tail. ReSPO addresses both failures with separate positive- and negative-advantage reshaping branches derived from an α-divergence variational objective. The choice α(+)>1α^(+)>1 preserves a nonzero positive-branch limit as W→0W→ 0, while the continuous α(−)=1α^(-)=1 limit suppresses both negative-branch extremes. Across Qwen3-1.7B and Qwen3-30B-A3B at N∈8,16,32N∈\8,16,32\, ReSPO attains the highest observed early score in five of six settings and the highest observed late score in five of six settings, with the observed differences generally largest at N=32N=32. Positive-branch diagnostics show that ReSPO can effectively learn from long positive reasoning trajectories during the earliest training steps, even when accumulated policy drift might put them to the low-importance-weight tail; this early advantage remains influential on 30B despite its soft length penalty. ReSPO also obtains the highest observed final-checkpoint benchmark evaluation averages at N=16N=16 and N=32N=32 on both model scales, and has the highest accuracy in 2424 of 3030 response-length bins. The MoE ablation further shows that Routing Replay complements ReSPO by improving late-stage training score and smoothness. App. F discusses the limitations in compute budget and hyperparameter search. Reproducibility Statement Sec. 3.3 defines the ReSPO objective, its gradient, and the analytical conditions used to select the kernel. App. A provides the full derivations, while App. B gives the numerically stable log-space implementation and complete algorithm. The accompanying code artifact contains the ReSPO implementation and launch configurations for the dense and MoE experiments. Tabs. 2 and 3 record the shared training configuration and every method-specific hyperparameter, including model backends, parallelism, optimization settings, sampling parameters, response limits, batch construction, rollout-reuse ratios, and training horizon. Sec. 4.1 specifies the training data, model checkpoints, baseline configurations, and evaluation protocol. App. D.1 documents the DAPO-MATH reward and length penalty, and App. D.2 describes the evaluation-generation pipeline and mathematical-equivalence verifier. Tab. 1 reports every per-benchmark result used in the main comparison. App. D.3 defines the evaluation resampling procedure and its random seed; Apps. E.1, E.3, A.5, E.2, E.4, and G provide the detailed comparisons, sequence-weight diagnostics, on-policy limit, response-length analysis, Routing-Replay comparison, and auxiliary training statistics. Together, these materials specify how to reproduce the optimization method, training runs, evaluation scores, uncertainty estimates, and reported figures. References Chen et al. (2025) M. Chen, L. Sun, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, et al. ReSearch: learning to reason with search for LLMs via reinforcement learning. arXiv preprint arXiv:2503.19470. Cited by: §1. Csiszár (1975) I. Csiszár I-divergence geometry of probability distributions and minimization problems. The Annals of Probability, p. 146–158. Cited by: §A.2, §3.3.3. DeepSeek-AI et al. (2025) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2. Dwyer et al. (2025) M. Dwyer, A. Sobey, and A. Chapman It’s not you, it’s clipping: a soft trust-region via probability smoothing for LLM RL. External Links: 2509.21282, Link Cited by: §2. Gao et al. (2025) C. Gao, C. Zheng, X. Chen, K. Dang, S. Liu, B. Yu, A. Yang, S. Bai, J. Zhou, and J. Lin Soft adaptive policy optimization. External Links: 2511.20347, Link Cited by: §2. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850. Cited by: §4.1. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874. Cited by: §4.1. Hu et al. (2025) J. Hu, J. K. Liu, H. Xu, and W. Shen REINFORCE++: stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262. Cited by: §2. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611–626. External Links: Document, Link Cited by: §4.1. Kydlíček (2024) H. Kydlíček Math-Verify: math verification library. Note: https://github.com/huggingface/Math-VerifySoftware, accessed September 25, 2026 Cited by: §4.1. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems 35, p. 3843–3857. Cited by: §4.1. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Cited by: §4.1. Liu et al. (2025) J. Liu, Y. Li, Y. Fu, J. Wang, Q. Liu, and Z. Jiang When speed kills stability: demystifying RL collapse from the training-inference mismatch. Note: Web article, accessed September 25, 2026 External Links: Link Cited by: §2. Ma et al. (2025) W. Ma, H. Zhang, L. Zhao, Y. Song, Y. Wang, Z. Sui, and F. Luo Stabilizing MoE reinforcement learning by aligning training and inference routers. arXiv preprint arXiv:2510.11370. Cited by: §2, §3.1, §4.3. math-ai (2023) math-ai AMC 2023 dataset release. Note: https://huggingface.co/datasets/math-ai/amc23Competition problems originally published by the Mathematical Association of America; accessed September 25, 2026 Cited by: §4.1. Maxwell-Jia (2024) Maxwell-Jia AIME 2024 dataset release. Note: https://huggingface.co/datasets/Maxwell-Jia/AIME_2024Competition problems originally published by the Mathematical Association of America; accessed September 25, 2026 Cited by: §4.1. Metelli et al. (2018) A. M. Metelli, M. Papini, F. Faccio, and M. Restelli Policy optimization via importance sampling. Advances in Neural Information Processing Systems 31. Cited by: §2. Neal (2001) R. M. Neal Annealed importance sampling. Statistics and Computing 11 (2), p. 125–139. Cited by: §3.3.2. Noukhovitch et al. (2025) M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville Asynchronous RLHF: faster and more efficient off-policy RL for language models. In International Conference on Learning Representations, External Links: Link Cited by: §2. OpenCompass (2025) OpenCompass AIME 2025 dataset release. Note: https://huggingface.co/datasets/opencompass/AIME2025Competition problems originally published by the Mathematical Association of America; accessed September 25, 2026 Cited by: §4.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, p. 27730–27744. Cited by: §2. Schulman et al. (2015) J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz Trust region policy optimization. In International Conference on Machine Learning, p. 1889–1897. Cited by: §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2, §4.1. Shen et al. (2026) G. Shen, C. Zhao, X. Cheng, L. Huang, and X. Yu VESPO: variational sequence-level soft policy optimization for stable off-policy LLM training. External Links: 2602.10693, Link Cited by: §A.4, §A.6, Appendix C, §1, §2, §3.1, §3.3.2, §3.3.3, §4.1. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §4.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source LLM reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §1, §2, §4.1. Zhai et al. (2026) Z. Zhai, M. Chen, J. Zhao, J. Qian, L. Shen, and Y. Lu Rewards as labels: revisiting RLVR from a classification perspective. External Links: 2602.05630, Link Cited by: §2, §3.2. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin Group sequence policy optimization. External Links: 2507.18071, Link Cited by: §1, §2, §3.1, §3.1, §3.1, §4.1. Appendix A Derivations for the α Reshaping Kernel This appendix collects the derivations referenced in Sec. 3.3. A.1 α-Bregman divergence and first-order condition For f(x)=1α(α−1)xαf(x)= 1α(α-1)x^α with α≠0,1α≠ 0,1, f is strictly convex on ℝ+R_+. The induced Bregman divergence is Df(p∥q)=∑[1α(α−1)p()α−1α(α−1)q()α−1α−1q()α−1(p()−q())].D_f(p\,\|\,q)= _ o [ 1α(α-1)p( o)^α- 1α(α-1)q( o)^α- 1α-1q( o)^α-1 (p( o)-q( o) ) ]. (16) The objective in Eq. (7) has derivative ∂τ()[(1−β)Df(τ∥μ)+βDf(τ∥π)]=(1−β)(f′(τ)−f′(μ))+β(f′(τ)−f′(π)), ∂τ( o) [(1-β)\,D_f(τ\,\|\,μ)+β\,D_f(τ\,\|\,π) ]=(1-β) (f (τ)-f (μ) )+β (f (τ)-f (π) ), so setting it to zero yields f′(τ∗)=(1−β)f′(μ)+βf′(π)f (τ^*)=(1-β)\,f (μ)+β\,f (π). With f′(x)=1α−1xα−1f (x)= 1α-1x^α-1 this reads 1α−1(τ∗)α−1=1−βα−1μα−1+βα−1πα−1, 1α-1(τ^*)^α-1= 1-βα-1\,μ^α-1+ βα-1\,π^α-1, (17) and solving for τ∗τ^* gives the power-mean form of Eq. (8). Substituting π=μWπ=μ\,W and using τ=μϕ(W)τ=μ\,φ(W) (Eq. (5)) recovers the unconstrained kernel Eq. (9). The normalized distribution is obtained only afterward as τ¯=τ/Z τ=τ/Z. What if we optimize over a normalized distribution? To distinguish this alternative from our unnormalized measure τ, let q denote the optimization variable. For α>1α>1, imposing ∑q()=1 _ oq( o)=1 with multiplier η gives the KKT conditions for q≥0q≥ 0: f′(q∗)=(1−β)f′(μ)+βf′(π)−η,f (q^*)=(1-β)\,f (μ)+β\,f (π)-η, which yield q⋆()=[max(0,(1−β)μ()α−1+βπ()α−1−η(α−1))]1/(α−1),q ( o)=[ (0,(1-β)μ( o)^α-1+βπ( o)^α-1-η(α-1))]^1/(α-1), including a positivity projection. Substituting q=μϕ0(W)q=μ _0(W) and π=μWπ=μ W gives ϕ0(W,μ())=[(1−β)+βWα−1−η(α−1)⋅μ()1−α]+1/(α−1). _0 (W;μ( o) )= [(1-β)+β W^α-1-η(α-1)·μ( o)^1-α ]^1/(α-1)_+. This loses the ratio-only form needed for a reshaping kernel. The normalization multiplier enters as η(α−1)μ()1−αη(α-1)μ( o)^1-α rather than as a constant, so the optimum depends on the pair (W,μ())(W,μ( o)) instead of on W alone. Depending on the multiplier and local behavior mass, the positivity projection can also set the weight to zero, reintroducing a hard boundary. Optimizing over the unnormalized measure avoids both complications and yields the smooth kernel in Eq. (9). A.2 Positive branch: exponential tilt derivation Given the unconstrained solution τ0∗ _0^* from the dual-proximity problem (Sec. 3.3.2), we impose the second-moment constraint ∑τ()ϕ0(W())≤C1 _ oτ( o) _0(W( o))≤ C_1 by solving the KL projection in Eq. (10): τ∗=argminτDKL(τ∥τ0∗)s.t.∑τ()ϕ0(W())≤C1,∑τ()≤C2.τ^*= _τ\;D_KL(τ\,\|\, _0^*) .t. _ oτ( o)\, _0(W( o))≤ C_1, _ oτ( o)≤ C_2. (18) Taking the functional derivative of the Lagrangian ℒ(τ,λ,γ)=DKL(τ∥τ0∗)+λ(∑τ()ϕ0(W())−C1)+γ(∑τ()−C2)L(τ,λ,γ)=D_KL(τ\| _0^*)+λ( _ oτ( o) _0(W( o))-C_1)+γ( _ oτ( o)-C_2) gives logτ∗()=logτ0∗()−λϕ0(W())−γ, τ^*( o)= _0^*( o)-λ\, _0(W( o))-γ, (19) and hence τ∗()=τ0∗()⋅exp(−λϕ0(W())−γ)τ^*( o)= _0^*( o)· (-λ\, _0(W( o))-γ), as stated in Eq. (11). This is the standard KL information projection under a linear constraint (Csiszár, 1975): among all measures satisfying ∑τ()ϕ0(W())≤C1 _ oτ( o) _0(W( o))≤ C_1, the one closest to τ0∗ _0^* in KL divergence is obtained by tilting the density with exp(−λϕ0(W)) (-λ _0(W)). Substituting τ0∗=μϕ0(W) _0^*=μ\, _0(W) and extracting ϕφ via τ∗=μϕ(W)τ^*=μ\,φ(W) gives ϕ(W)=ϕ0(W)exp(−λϕ0(W)−γ).φ(W)= _0(W) (-λ\, _0(W)-γ). (20) The response-independent factor exp(−γ) (-γ) changes only the total mass and can be absorbed into the constraint constants C1C_1 and C2C_2. Rescaling the kernel so that ϕ(1)=1φ(1)=1, using ϕ0(1)=1 _0(1)=1, gives Eq. (12). Why use the proxy ϕ0 _0? The natural variance-control target is μ[ϕ(W)2]≤CE_μ[φ(W)^2]≤ C, which controls the second-moment term in the denominator of the standard importance-weight effective sample size. Since τ=μϕτ=μ\,φ, this is equivalent to ∑τ()ϕ(W())≤C _ oτ( o)φ(W( o))≤ C. However, the final kernel ϕφ is itself the unknown we are solving for: writing ϕ=ϕ0⋅hφ= _0· h where h is the tilt function, the constraint becomes ∑τ()ϕ0(W())h(W())≤C′ _ oτ( o) _0(W( o))h(W( o))≤ C , which is not linear in τ (since h depends on τ through the measure-change identity). The Gibbs variational principle requires a constraint of the form ∑τ()g(W())≤C _ oτ( o)g(W( o))≤ C for a fixed function g. We therefore replace ϕφ with its known proxy ϕ0 _0 (the unconstrained kernel from Step 1) and constrain ∑τ()ϕ0(W())≤C _ oτ( o) _0(W( o))≤ C. This is linear in τ, so the KL projection yields the clean exponential tilt exp(−λϕ0(W)) (-λ _0(W)). Under this proxy, we prove that the derived τ satisfies bounded moments. Since the tilt satisfies h(W)=exp(λ(1−ϕ0(W)))≤exp(λ)h(W)= (λ(1- _0(W)))≤ (λ), we have μ[ϕ(W)2]=∑τ()ϕ(W())≤exp(λ)∑τ()ϕ0(W()).E_μ[φ(W)^2]= _ oτ( o)φ(W( o))≤ (λ) _ oτ( o) _0(W( o)). (21) Thus, bounding ∑τ()ϕ0(W()) _ oτ( o) _0(W( o)) ensures μ[ϕ2]E_μ[φ^2] is also bounded. Why KL and not α-divergence for the projection? One might ask why we use the KL divergence in the projection step (Step 2) rather than the same α-divergence used in Step 1. Replacing the KL with Df(τ∥τ0∗)D_f(τ\| _0^*) for f(x)=1α(α−1)xαf(x)= 1α(α-1)x^α and using the linear constraint ∑τ()ϕ0(W())≤C _ oτ( o) _0(W( o))≤ C, the first-order condition becomes 1α−1τ∗α−1=1α−1τ0∗α−1−λϕ0(W). 1α-1τ^*α-1= 1α-1 _0^*α-1-λ\, _0(W). (22) Extracting ϕφ yields the constraint term inside the power-mean bracket: ϕ(W)∝[ϕ0(W)α−1−λ(α−1)μ1−αϕ0(W)]+1/(α−1).φ(W) [ _0(W)^α-1-λ(α-1)\,μ^1-α\, _0(W) ]_+^1/(α-1). (23) The factor μ1−αμ^1-α makes this expression depend on both W and the behavior mass. For sufficiently small μ, the bracket can become negative and activate the [⋅]+[·]_+ projection; when 1<α<21<α<2, the subtracted term also grows faster than the first term as ϕ0(W) _0(W) increases. The resulting kernel can therefore acquire a hard cutoff rather than the smooth, ratio-only decay we seek. The key distinction is geometric. The KL first-order condition is additive in log-space (logτ∗=logτ0∗−λϕ0−γ τ^*= _0^*-λ _0-γ), so the constraint enters as a multiplicative exponential factor on τ0∗ _0^*. In the power-Bregman alternative above, the constraint instead enters inside the power-mean bracket, introducing behavior-mass dependence and a possible hard boundary. The two-step procedure therefore uses each divergence where it is most natural: α-divergence for reshaping (power-mean interpolation between μ and π), and KL for variance control (multiplicative exponential tilt). A.3 Hyperparameter derivations Positive branch: deriving λ(+)λ^(+). For ϕ(W)=ϕ0(W)⋅exp(λ(1−ϕ0(W)))φ(W)= _0(W)· (λ(1- _0(W))), differentiating logϕ φ gives dlogϕdW=ϕ0′(W)(1ϕ0(W)−λ). d φdW= _0 (W) ( 1 _0(W)-λ ). (24) At α(+)=2α^(+)=2, the unconstrained kernel is ϕ0(+)(W)=(1−β(+))+β(+)W _0^(+)(W)=(1-β^(+))+β^(+)W, so ϕ0′(W)=β(+)>0 _0 (W)=β^(+)>0 is constant. At W=0W=0 we have ϕ0(0)=1−β(+) _0(0)=1-β^(+), so dlogϕ(+)dW|W=0=β(+)(11−β(+)−λ(+)). d φ^(+)dW |_W=0=β^(+)\! ( 11-β^(+)-λ^(+) ). We additionally require the full kernel to be stationary at the origin, ϕ(+)(0)′=0φ^(+) (0)=0. Because ϕ(+)(0)>0φ^(+)(0)>0, this is equivalent to setting the log derivative above to zero, which gives λ(+)=11−β(+).λ^(+)= 11-β^(+). (25) For any W>0W>0, ϕ0(W)>1−β(+)=1/λ(+) _0(W)>1-β^(+)=1/λ^(+), so 1/ϕ0(W)<λ(+)1/ _0(W)<λ^(+) and dlogϕ(+)dW<0 d φ^(+)dW<0: the kernel is strictly decreasing on its feasible domain (0,∞)(0,∞). Moreover, because α(+)=2α^(+)=2 makes ϕ0 _0 affine, the closed form has a differentiable extension to every W∈ℝW , even though importance weights themselves satisfy W≥0W≥ 0. On this extension, ϕ(+)(W)′=−(β(+))21−β(+)Wexp(λ(+)(1−ϕ0(+)(W))),φ^(+) (W)=- (β^(+))^21-β^(+)\,W \! (λ^(+)(1- _0^(+)(W)) ), which is positive for W<0W<0, zero at W=0W=0, and negative for W>0W>0. Thus W=0W=0 is the unique global maximizer both on the real-valued extension and on the feasible domain, with ϕ(+)(0)=(1−β(+))eβ(+)/(1−β(+)).φ^(+)(0)=(1-β^(+))\,e^\,β^(+)/(1-β^(+)). (26) With β(+)=0.5β^(+)=0.5, this gives λ(+)=2λ^(+)=2 and ϕ(+)(0)=12e1≈1.36φ^(+)(0)= 12e^1≈ 1.36. Why α=2α=2 is the natural choice. We seek a non-degenerate local condition at the origin: ϕ0′(0) _0 (0) should be finite and strictly positive, so requiring ϕ(+)(0)′=0φ^(+) (0)=0 uniquely determines λ(+)λ^(+). For 1<α<21<α<2, ϕ0′(W)∝Wα−2 _0 (W) W^α-2 is singular as W→0+W→ 0^+; for α>2α>2, ϕ0′(0)=0 _0 (0)=0, so stationarity of the full kernel holds for any λ and does not determine the tilt. At α=2α=2, ϕ0′(0)=β(+) _0 (0)=β^(+) is finite and strictly positive, and ϕ0 _0 is linear in W. Thus α(+)=2α^(+)=2 uniquely combines an affine unconstrained kernel, a non-degenerate zero-derivative condition that fixes λ(+)λ^(+), and the real-line maximum at W=0W=0 shown above. Negative branch: matching the derivative at W=1W=1. With α(−)=1α^(-)=1 and β(−)=0.5β^(-)=0.5, the continuous-limit kernel is ϕ0(−)(W)=Wβ(−)=W. _0^(-)(W)=W^β^(-)= W. We share β=0.5β=0.5 across the two branches so that both unconstrained kernels interpolate symmetrically between the behavior and target policies. The remaining parameter λ(−)λ^(-) is fixed by matching the derivatives of the full kernels at the on-policy point. For the positive branch, ϕ(+)(W)=1+W2e1−W,dϕ(+)dW=−W2e1−W,φ^(+)(W)= 1+W2e^1-W, dφ^(+)dW=- W2e^1-W, and hence ϕ(+)(1)′=−12φ^(+) (1)=- 12. For the negative branch, ϕ(−)(W)=Wβ(−)exp(λ(−)(1−Wβ(−))).φ^(-)(W)=W^β^(-) \! (λ^(-)(1-W^β^(-)) ). Since ϕ(−)(1)=1φ^(-)(1)=1, its derivative at W=1W=1 is ϕ(−)(1)′=β(−)(1−λ(−)).φ^(-) (1)=β^(-) (1-λ^(-) ). (27) Requiring ϕ(−)(1)′=ϕ(+)(1)′φ^(-) (1)=φ^(+) (1) and substituting β(−)=0.5β^(-)=0.5 gives 12(1−λ(−))=−12⟹λ(−)=2. 12 (1-λ^(-) )=- 12 λ^(-)=2. Thus the positive and negative kernels have identical value and first derivative at W=1W=1, making their local response to off-policy drift symmetric to first order. A.4 Continuous limit as α→1α→ 1 Taylor-expanding Wα−1W^α-1 around α=1α=1 gives Wα−1=e(α−1)logW≈1+(α−1)logW^α-1=e^(α-1) W≈ 1+(α-1) W to first order in α−1α-1. Substituting this expansion into the α factor of Eq. (12) gives ϕ0(W)=[1+β(Wα−1−1)]1/(α−1)≈[1+β(α−1)logW]1/(α−1), _0(W)= [1+β(W^α-1-1) ]^1/(α-1)≈ [1+β(α-1) W ]^1/(α-1), which tends to WβW^β as α→1α→ 1. The full kernel therefore reduces to Wβexp(λ(1−Wβ))W^β (λ(1-W^β)), which is the form used by ReSPO’s negative branch. For comparison, VESPO’s gamma-IS kernel (Shen et al., 2026) is Wβexp(λ(1−W))W^β (λ(1-W)): the exponential tilt penalizes the raw importance weight W rather than the reshaped weight ϕ0(W)=Wβ _0(W)=W^β. A.5 On-policy limit In the strict on-policy limit, πθ=πold _θ= _old and every sample has W=1W=1. Setting N=1N=1 removes staleness from rollout reuse and approximates this condition. At W=1W=1, the two kernels used in our experiments satisfy ϕ(+)(1)=1+12exp(1−1)=1,ϕ(−)(1)=1exp(2(1−1))=1.φ^(+)(1)= 1+12 (1-1)=1, φ^(-)(1)= 1 \! (2(1- 1) )=1. (28) Consequently, ReSPO assigns unit reshaping weight in the exact on-policy limit: its per-sequence multiplier is one. This recovers standard on-policy training, in which no importance-ratio weighting is applied. A.6 Length-dependent bias from sequence normalization Shen et al. (2026) identify a length-dependent bias in the geometric-mean normalization used by GSPO. Let T=||T=| o|, let ℓt=logwt _t= w_t be the per-token log-ratio, and write ℓ¯=T−1∑t=1Tℓt =T^-1 _t=1^T _t. The exact sequence importance weight and the GSPO weight are W=exp(Tℓ¯),s=W1/T=exp(ℓ¯).W= (T ), s=W^1/T= ( ). (29) The exponent 1/T1/T controls variance, but it also changes the implied measure correction with response length. For a fixed length T, weighting samples from μ by s induces the proposal QT()∝μ()s()=μ()1−1/Tπ()1/T.Q_T( o) μ( o)s( o)=μ( o)^1-1/Tπ( o)^1/T. (30) Thus the target-policy exponent decreases as T grows, and QTQ_T is increasingly tempered toward the behavior policy μ. In this sense, the strength of the off-policy correction depends on length rather than only on the likelihood shift between π and μ. The normalization also conflates sequences with different total likelihood shifts. If two responses have lengths T1≠T2T_1≠ T_2 but the same average log-ratio ℓ¯ , GSPO assigns both the same weight, s1=s2=eℓ¯s_1=s_2=e . Their true sequence weights are instead W1=eT1ℓ¯W_1=e^T_1 and W2=eT2ℓ¯W_2=e^T_2 , with W2W1=exp((T2−T1)ℓ¯). W_2W_1= \! ((T_2-T_1) ). (31) Consequently, a short response and a much longer response can receive the same GSPO weight even when their sequence-level distribution shifts differ exponentially. Clipping can further truncate these weights, but the length dependence is already introduced by the 1/T1/T normalization. ReSPO instead applies a fixed kernel ϕ(W)φ(W) to the unnormalized sequence weight. Like any non-identity reshaping, this intentionally trades bias for variance. The same map ϕφ is used at every response length, so length enters through the actual sequence likelihood ratio W without an additional length-dependent tempering exponent. Appendix B Log-Space Implementation of the Two-Branch Kernel The reshaping weight ϕ(Wi,A^i)φ(W_i; A_i) is evaluated per-sequence from the sequence-level log-ratio logWi=∑t=1|i|logwi,t,wi,t=πθ(oi,t∣,i,<t)/πold(oi,t∣,i,<t). W_i= _t=1^| o_i| w_i,t, w_i,t= _θ(o_i,t q, o_i,<t)\,/\, _old(o_i,t q, o_i,<t). We clamp logWi W_i to [−20,20][-20,20] before any non-linear operation and detach it from the gradient graph, since ϕφ plays the role of a gradient scaling coefficient. The branch computations use numerically stable forms. The tail limits analyzed above refer to the unclamped analytical kernels; the implementation evaluates a finite-range approximation to avoid numerical overflow. We describe the two branches separately; which one applies to a given sequence is selected by the sign of A^i A_i. Positive branch (A^i≥0 A_i≥ 0). With α(+)=2α^(+)=2, β(+)=0.5β^(+)=0.5, λ(+)=2λ^(+)=2, the kernel simplifies to ϕ(+)(Wi)=1+Wi2e1−Wi.φ^(+)(W_i)= 1+W_i2\,e^1-W_i. (32) No log-space tricks are needed: (1+Wi)/2≥1/2>0(1+W_i)/2≥ 1/2>0 for all Wi≥0W_i≥ 0. Negative branch (A^i<0 A_i<0). With α(−)=1α^(-)=1, the continuous-limit kernel has ϕ0(−)(Wi)=Wiβ(−) _0^(-)(W_i)=W_i^β^(-). We therefore compute it directly in log-space: logϕ0(−)(Wi)=β(−)logWi. _0^(-)(W_i)=β^(-) W_i. (33) The full kernel adds the exponential tilt: logϕ(−)(Wi)=logϕ0(−)(Wi)+λ(−)(1−exp(logϕ0(−)(Wi))). φ^(-)(W_i)= _0^(-)(W_i)+λ^(-)\! (1- \! ( _0^(-)(W_i) ) ). (34) Combining the branches. Given the two log-space scalars, we compute ϕi=exp(logϕi) _i= ( _i) on the branch selected by sign(A^i)sign( A_i), detach the result, and multiply it into the token-level loss as summarized in Alg. 1. In practice we compute both branches for every sequence and select the applicable one with a sign mask, which is simpler on a GPU than using a data-dependent branch and has negligible overhead since both kernels are O(1)O(1) per sequence. Algorithm 1 ReSPO: Reshaped Sequence Policy Optimization 0: Policy πθ _θ, prompts \ q\, group size G 0: α(+)=2α^(+)=2, α(−)=1α^(-)=1, β(+)=β(−)=0.5β^(+)=β^(-)=0.5, λ(+)=2λ^(+)=2, λ(−)=2λ^(-)=2 1: for each training iteration do 2: Sample a batch of prompts; for each q, sample G responses ii=1G∼πold\ o_i\_i=1^G _old 3: Compute rewards R(,i)R( q, o_i) and GRPO group-normalized advantages A^i A_i 4: for each response i o_i do 5: logWi←∑tlog[πθ(oi,t∣,i,<t)/πold(oi,t∣,i,<t)] W_i← _t \! [ _θ(o_i,t q, o_i,<t)/ _old(o_i,t q, o_i,<t) ]; clamp to [−20,20][-20,20] 6: if A^i≥0 A_i≥ 0 then 7: positive branch 8: ϕi←(1+elogWi)/2⋅exp(1−elogWi) _i←(1+e W_i)/2· (1-e W_i) 9: else 10: negative branch 11: ℓi←β(−)logWi _i←β^(-) W_i 12: ϕi←exp(ℓi+λ(−)(1−eℓi)) _i← ( _i+λ^(-)(1-e _i)) 13: end if 14: ϕi←ϕi _i← _i.detach() 15: end for 16: ∇θ←1G∑iϕi⋅A^i⋅∑t∇θlogπθ(oi,t∣,i,<t) _θJ← 1G _i _i· A_i· _t _θ _θ(o_i,t q, o_i,<t) 17: Update θ by gradient ascent using ∇θ _θJ 18: end for Appendix C Detailed Hyperparameters Tab. 2 lists the training hyperparameters for both experimental settings. Tab. 3 lists the method-specific hyperparameters for each baseline and ReSPO. Table 2: Training hyperparameters across experimental settings. Hyperparameter Qwen3-1.7B Qwen3-30B-A3B Model & infrastructure Training backend FSDP Megatron Tensor parallelism (train) – 2 Expert parallelism (train) – 8 Tensor parallelism (rollout) 2 4 Data Training set DAPO-MATH-17k DAPO-MATH-17k Max prompt length 1,0241,024 1,0241,024 Max response length 15,36015,360 8,1928,192 Optimization Optimizer AdamW Adam Learning rate 1×10−61× 10^-6 Warmup ratio 5%5\% Weight decay 0.10.1 Gradient clip norm 1.01.0 KL penalty None Rollout Rollout engine vLLM (async) Responses per prompt (G) 88 Temperature (train) 1.01.0 Top-p (train) 1.01.0 Batching & budget Mini-batch size (M) 3232 Rollout-reuse ratio (N) 8,16,32\8,16,32\ Global batch size N×32N× 32 Total policy updates 10241024 Advantage estimator GRPO (group-normalized) Aggregation token-mean Group-normalized advantages. The implementation computes GRPO advantages using the sample standard deviation with ϵA=10−6 _A=10^-6 added to the denominator. When all rewards in a group are identical, the centered numerators are zero, so all advantages are zero rather than undefined. Why this normalization? For a fixed prompt, Eq. (4) is an expectation over responses sampled from μ=πoldμ= _old. Given G such responses, its standard Monte Carlo estimator is therefore the sequence mean G−1∑i=1G^-1 _i=1^G used in Eq. (15) and Alg. 1; the outer expectation over prompts supplies the corresponding prompt average. The factor 1/G1/G averages sampled sequences and does not normalize the unnormalized measure τ or its weights, so replacing it by 1/∑iϕi1/ _i _i would instead produce a different, self-normalized estimator. In the reported experiments, we use the batch-wide token-mean normalization 1/∑i|i|1/ _i| o_i| shown in Tab. 2 for improved training stability. Relative to the sequence mean, this applies a common batch-level scale to all response contributions and therefore preserves their relative ReSPO weights within each batch. Table 3: Method-specific hyperparameters. All methods share the training configuration in Tab. 2. Method Hyperparameter Value GRPO Clip ratio ε 0.20.2 GSPO Clip ratio low εlow _low 3×10−43× 10^-4 Clip ratio high εhigh _high 4×10−44× 10^-4 VESPO β(+)β^(+) 22 λ(+)λ^(+) 33 β(−)β^(-) 33 λ(−)λ^(-) 22 ReSPO α(+)α^(+) 22 β(+)β^(+) 0.50.5 λ(+)λ^(+) 22 α(−)α^(-) 11 β(−)β^(-) 0.50.5 λ(−)λ^(-) 22 The VESPO values in Tab. 3 are the best-performing sign-specific configuration reported in the hyperparameter study of the original VESPO paper (Shen et al., 2026), and we use them directly. ReSPO uses the analytically motivated parameters derived in App. A. Appendix D Evaluation Implementation In this section, we describe the training reward implementation and the evaluation scoring procedure. D.1 DAPO-MATH training reward implementation In the reported runs, training examples carry the data-source key math_dapo, which selects the strict-box verifier rather than the Math-Verify evaluator used for Tab. 1. The relevant logic is summarized below; names are simplified for clarity. ⬇ # Training-reward pseudocode if use_strict_box_reward: strict_score = strict_box_verify(response, ground_truth) correct = (strict_score == +1) reward = +1.0 if correct else -1.0 if overlong_penalty_enabled: reward -= max(0.0, (response_length - 4096) / 4096) The strict-box verifier returns +1+1 or −1-1 after comparing the ground truth with the final boxed answer in the response suffix. Its wrapper explicitly checks for +1+1 before mapping the result to the floating-point reward +1.0+1.0 or −1.0-1.0. For the reported 30B experiments, the DAPO reward manager additionally applies the configured overlong-buffer term. With max_resp_len=8,192=8,192, buffer length 4,0964,096, and penalty factor 11, the additive penalty is −max0,(L−4,096)/4,096- \0,(L-4,096)/4,096\ for response length L. It is zero through 4,0964,096 tokens and reaches −1-1 at the response limit. The dense-model experiments disable this term and use only the signed strict-box score. Range of the penalized training accuracy. For the 1.7B experiments, the signed strict-box score is in −1,1\-1,1\, so the affine transformation (score+1)/2( score+1)/2 is in 0,1\0,1\. For the 30B experiments, the total score also includes the additive penalty in [−1,0][-1,0], so the transformed quantity can range from −0.5-0.5 to 11. We retain the name penalized training accuracy because the quantity reduces to ordinary training accuracy when the penalty is zero and uses the same affine scale; a negative value denotes an incorrect overlong response with an additional length penalty, not a negative probability. D.2 Math-Verify scoring implementation The evaluator tags all six benchmarks as MATH-lighteval, which the verl score dispatcher routes to Math-Verify rather than the DAPO strict-box verifier. The evaluation logic is summarized as follows. ⬇ # Evaluation-scoring pseudocode gold = parse_latex(box(ground_truth)) predictions = parse(completion, modes=[expression, latex]) correct = any( mathematically_equivalent(g, p) for g in gold for p in predictions ) score = 1.0 if correct else 0.0 Parsing the completion in both modes lets the verifier accept free-form final answers in addition to strict answers. If either side cannot be parsed, no pair is mathematically equivalent, or the scoring worker exceeds its 3030-second timeout, the completion receives zero credit. The pass@1 reported in Tab. 1 is the mean of this binary correctness indicator over samples and problems. The auxiliary majority-vote metric separately uses the DAPO boxed-answer utilities. D.3 Bootstrap evaluation uncertainty The error bars in Tab. 1 use a cluster bootstrap over evaluation problems, using B=10,000B=10,000 replicates with random seed 00. For problem i in benchmark j, let yjik∈0,1y_jik∈\0,1\ denote the correctness of sampled completion k. We first average the njin_ji completions for that problem, zji=1nji∑k=1njiyjik,a^j=1mj∑i=1mjzji,A^=16∑j=16a^j,z_ji= 1n_ji _k=1^n_jiy_jik, a_j= 1m_j _i=1^m_jz_ji, A= 16 _j=1^6 a_j, (35) where mjm_j is the number of problems in benchmark j: 3030 each for AIME25 and AIME24, 4040 for AMC23, 674674 for OlympiadBench, 272272 for MinervaMath, and 500500 for MATH-500. The reported macro average assigns equal weight to each benchmark. For bootstrap replicate b, we independently draw mjm_j problem indices with replacement within each benchmark, keeping all completions for a sampled problem together. If these indices are Ij1(b),…,Ijmj(b)I^(b)_j1,…,I^(b)_jm_j, we recompute a^j(b)=1mj∑ℓ=1mjzjIjℓ(b),A^(b)=16∑j=16a^j(b). a^(b)_j= 1m_j _ =1^m_jz_jI^(b)_j , A^(b)= 16 _j=1^6 a^(b)_j. (36) The subscripted error bar is the bootstrap standard error, SEboot=1B−1∑b=1B(A^(b)−A^¯)2,SE_boot= 1B-1 _b=1^B ( A^(b)- A )^2, (37) so an entry A^±SEboot A_ _boot reports one bootstrap standard error. For method contrasts, the same resampled problem indices are applied to both methods before taking the difference, giving a paired cluster bootstrap; percentile quantiles of that difference distribution form a confidence interval. These calculations quantify finite-evaluation sampling uncertainty conditional on the fixed final checkpoint and its stored generations. Appendix E Additional Robustness Checks E.1 Detailed training and evaluation comparisons Within the first 256256 policy updates, ReSPO has the highest peak normalized training score in five of six model–N settings. The exception is 30B at N=8N=8, where GRPO is higher by 1.01.0 p. ReSPO’s differences from the next-highest method are 2.32.3, 1.01.0, and 3.03.0 p on 1.7B for N∈8,16,32N∈\8,16,32\, respectively, and 4.24.2 and 5.45.4 p on 30B at N=16,32N=16,32. Its curves remain above the baselines through most of the subsequent trajectory. Over the last 128128 policy steps, ReSPO has the highest mean in five of six settings; on 1.7B at N=16N=16, it is 0.40.4 p below VESPO, and each central value lies within the other’s temporal standard-deviation range. It exceeds the highest clipped baseline in all six settings by 4.24.2–8.48.4 p, with non-overlapping temporal ranges. From N=8N=8 to N=32N=32, its mean decreases by 1.01.0 p on 1.7B and 0.70.7 p on 30B, compared with 2.52.5 and 4.54.5 p for GSPO and 4.54.5 and 1.51.5 p for VESPO. At N=32N=32, ReSPO exceeds the highest clipped baseline by 7.97.9 and 8.48.4 p, and VESPO by 4.64.6 and 6.56.5 p. Across the early and late summaries, ReSPO is highest in 1010 of 1212 settings. It is 1.01.0 p below the early maximum on 30B at N=8N=8 and 0.40.4 p below the late maximum on 1.7B at N=16N=16, where the temporal ranges overlap. At N=8N=8, the evaluation macro averages are similar relative to their bootstrap uncertainty: ReSPO is 0.50.5 p below VESPO on 1.7B and 0.20.2 p below GSPO on 30B. For N∈16,32N∈\16,32\, ReSPO has the highest observed mean in all four model–N settings, exceeding the highest baseline by 0.80.8 and 1.31.3 p on 1.7B and by 3.23.2 and 1.91.9 p on 30B. It has the highest value in 66, 99, and 88 of the 1212 model–benchmark pairs for N∈8,16,32N∈\8,16,32\, respectively. The largest observed differences occur on difficult competition benchmarks. On AIME25, ReSPO exceeds the highest baseline in all six model–N settings by 1.71.7, 2.32.3, and 0.70.7 p on 1.7B and by 7.17.1, 5.75.7, and 2.52.5 p on 30B as N increases. It also has the highest AMC23 value in all six settings and the highest AIME24 value in four of six. Averaged over these three benchmarks on 30B, its differences from the highest baseline are 6.36.3, 5.85.8, and 3.43.4 p for N∈8,16,32N∈\8,16,32\, respectively. At N=8N=8, the other three benchmarks offset this difference in the macro average; at larger N, ReSPO also attains the highest observed overall mean. E.2 Accuracy across response-length ranges We analyze the stored Qwen3-1.7B evaluation generations underlying Tab. 1 for all four methods, all six benchmarks, and N∈8,16,32N∈\8,16,32\. Each method contributes 7,3847,384 responses per value of N, and response length is the number of Qwen3-1.7B tokenizer tokens in the generated text. For each value of N, we pool the observed lengths to construct one set of boundaries shared by every method and benchmark. The generation cap produces a point mass at 16,38416,384 tokens, so Q10 contains exactly the capped responses; Q1–Q9 divide the uncapped pooled responses into nine equal-frequency intervals. This tie-preserving construction keeps identical capped lengths in the same bin. Accuracy within a bin is the fraction of its responses accepted by the evaluation verifier. Tab. 4 reports the resulting token upper bounds and the pooled number of responses from each method in every bin. These counts expose the method-dependent response-length distributions used to compute the pooled accuracies below. Table 4: Shared response-length-bin upper bounds (tokens) and pooled response counts for Qwen3-1.7B. Counts aggregate the six evaluation benchmarks; each method contributes 7,3847,384 responses for each N. Q10 contains responses at the 16,38416,384-token cap. N Entry Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 88 Upper bound 355 480 611 761 954 1,263 2,009 4,404 16,383 16,384 GRPO count 1,043 908 820 855 816 809 573 163 102 1,295 GSPO count 1,253 1,087 1,131 1,074 917 775 420 79 46 602 VESPO count 386 466 456 475 539 671 1,037 1,504 1,376 474 ReSPO count 207 429 482 482 605 630 856 1,140 1,360 1,193 1616 Upper bound 366 481 604 736 907 1,168 1,785 4,252 16,383 16,384 GRPO count 997 989 1,009 1,009 916 797 601 136 71 859 GSPO count 873 869 857 952 883 873 734 247 64 1,032 VESPO count 438 470 481 445 527 618 822 1,139 841 1,603 ReSPO count 506 483 450 405 472 506 640 1,281 1,827 814 3232 Upper bound 371 483 596 711 856 1,057 1,421 2,732 16,383 16,384 GRPO count 1,172 1,013 1,000 906 849 698 542 289 109 806 GSPO count 949 908 874 905 831 832 730 386 81 888 VESPO count 516 556 555 605 693 762 795 973 975 954 ReSPO count 255 394 453 469 499 589 805 1,230 1,714 976 Fig. 4(a) shows the pooled curves. Accuracy generally decreases toward the long tail, and the ReSPO curve decreases monotonically for all three values of N. ReSPO has the highest accuracy in eight of ten bins at each N. Its separation is widest at N=32N=32: the differences from the highest baseline are 8.48.4, 16.016.0, 24.424.4, 21.621.6, 20.820.8, 15.115.1, and 8.58.5 p from Q1 through Q7. VESPO is higher in Q8 and Q9 by 2.92.9 and 1.21.2 p, while ReSPO is higher in the cap bin by 1.51.5 p. At N=8N=8 and N=16N=16, the two reshaped methods are closer, but ReSPO still has the highest accuracy in most short, medium, and uncapped long-response bins. Fig. 4(b,c) separates the six benchmarks. The middle-length advantage at N=32N=32 appears broadly on AMC 2023, OlympiadBench, and MATH-500. The AIME panels have fewer observations and lower base accuracy, producing more irregular bin-level curves. Across benchmarks, capped responses are generally among the least accurate groups. (a) Pooled benchmarks (b) AIME 2025, AIME 2024, and AMC 2023 (c) OlympiadBench, MinervaMath, and MATH-500 Figure 4: Qwen3-1.7B accuracy by shared response-length bin. Q1–Q9 partition uncapped responses using boundaries shared across methods, while Q10 contains responses at the 16,38416,384-token cap. Columns correspond to N∈8,16,32N∈\8,16,32\; gaps in the benchmark panels indicate empty method–benchmark bins. E.3 Positive-response weight and length dynamics Sec. 4.3 suggests that ReSPO learns earlier from longer positive responses. To isolate this initial rapid-improvement phase, the diagnostic runs cover the first 1616 rollout batches, or 512512 of the 10241024 policy updates used in the complete experiments. Within this window, we ask two questions: when a positive response moves into the low-W tail, how much learning coefficient does it retain, and how does this allocation evolve with response length? Since W=πθ()/πold()W= _θ( o)/ _old( o), W<1W<1 means that the response has become less likely under the updated policy. For A^>0 A>0, this is precisely a response whose probability the update should increase. We measure W directly in matched Qwen3-1.7B-Base runs at N=32N=32 with a 16,38416,384-token response limit. ReSPO and VESPO use the same DAPO-MATH data, optimizer settings, GRPO advantages, and token-mean aggregation. Each run uses this first-half diagnostic window. For every sequence, we record its advantage sign and logW=∑t(logπθ−logπold) W= _t( _θ- _old) in fixed bins. Because this sum accumulates policy drift token by token, longer responses provide more opportunities for negative drift to build up and therefore tend to occur in the low-W tail. For a positive-branch bin B, we report pB(+)=∑i:A^i>0logWi∈B∑i:A^i>01,mB(+)=∑i:A^i>0|A^i|ϕ(+)(Wi)logWi∈B∑i:A^i>0|A^i|ϕ(+)(Wi).p_B^(+)= _i:\, A_i>01\ W_i∈ B\ _i:\, A_i>01, m_B^(+)= _i:\, A_i>0| A_i|φ^(+)(W_i)1\ W_i∈ B\ _i:\, A_i>0| A_i|φ^(+)(W_i). Here, pB(+)p_B^(+) is the fraction of positive responses in the bin, whereas mB(+)m_B^(+) is the fraction of the total positive-branch loss coefficient |A^|ϕ(+)(W)| A|φ^(+)(W) assigned to that bin. Their ratio mB(+)/pB(+)m_B^(+)/p_B^(+) is the amplification: a value above 11 means that each response in the bin receives more coefficient mass than the positive-branch average, while a value below 11 indicates suppression. The diagnostic logs each numerator and denominator under the same distributed reduction, so these ratios remain directly comparable. Table 5: Positive-branch deep-tail allocation on Qwen3-1.7B-Base at N=32N=32. Values are means over rollout batches 99–1616; amplification is the coefficient-mass share divided by the sequence share. Statistic ReSPO VESPO Positive responses with logW<−1 W<-1 22.8%22.8\% 16.7%16.7\% Coefficient mass on logW<−1 W<-1 38.5%38.5\% 15.5%15.5\% Amplification on logW<−1 W<-1 1.69×1.69× 0.93×0.93× Positive responses in −5<logW≤−2-5< W≤-2 7.4%7.4\% 3.7%3.7\% Coefficient mass in −5<logW≤−2-5< W≤-2 14.8%14.8\% 1.1%1.1\% Amplification in −5<logW≤−2-5< W≤-2 2.00×2.00× 0.30×0.30× The distinction between response share and coefficient-mass share isolates where the methods differ. In logW<−1 W<-1, ReSPO observes 22.8%22.8\% of its positive responses but assigns them 38.5%38.5\% of its positive-branch coefficient mass, giving 1.69×1.69× amplification. VESPO observes a similar 16.7%16.7\% response share but assigns the region only 15.5%15.5\% of its mass. The contrast sharpens in −5<logW≤−2-5< W≤-2: ReSPO amplifies the bin to 2.00×2.00× its response share, whereas VESPO suppresses it to 0.30×0.30×. At the final rollout batch, ReSPO places 58.5%58.5\% of its positive-branch mass in −5<logW≤−0.5-5< W≤-0.5; VESPO instead places its largest share, 35.6%35.6\%, in the region immediately below W=1W=1, −0.5<logW≤0-0.5< W≤ 0. Table 6: Mean logW W and response length on the positive branch during the N=32N=32 diagnostic runs. Mean logW W Mean response length Batch ReSPO VESPO ReSPO VESPO 11 −0.021-0.021 +0.134+0.134 752752 829829 88 −0.188-0.188 −0.064-0.064 829829 843843 1616 −0.385-0.385 −0.064-0.064 10761076 915915 The weight and length statistics evolve together. By batch 1616, mean logW W reaches −0.385-0.385 for ReSPO and −0.064-0.064 for VESPO, corresponding to geometric-mean weights of approximately 0.680.68 and 0.940.94. Over the same interval, mean positive-response length grows from 752752 to 10761076 under ReSPO (43%43\%) and from 829829 to 915915 under VESPO (10%10\%). Thus, the faster growth of ReSPO’s positive responses coincides with their movement farther into the low-W region where ReSPO retains more coefficient mass. In the complete 10241024-update trajectories, ReSPO’s aggregate response length and training score also separate from the baselines during the early stage of optimization and continue to diverge at N=32N=32 (Fig. 5). The two views together are consistent with ReSPO learning earlier from longer positive responses: this diagnostic identifies the retained learning coefficient, while App. G shows the corresponding full-training trajectory. To test the length–weight relationship directly, we also record positive-response length within each logW W bin over all 1616 rollout batches. Tab. 7 reports pooled, count-weighted mean lengths and normalizes each value by the corresponding method’s positive-branch mean over the same window. For logW≤−0.5 W≤-0.5, positive responses average 12761276 tokens under ReSPO and 936936 under VESPO, or 1.13×1.13× and 1.06×1.06× their branch-wide means, respectively. These pooled regions contain 71637163 and 49184918 positive responses. Responses immediately below W=1W=1 are shorter, so the relation turns upward in the deep low-W tail rather than changing monotonically. Individual extreme-bin means are descriptive because those bins are sparse. Combined with Tab. 5, the pooled result shows that the kernels differ in how much learning-coefficient mass they retain on these longer positive responses. Table 7: Count-weighted positive-response length by sequence-weight bin over rollout batches 11–1616 of the matched Qwen3-1.7B-Base follow-up runs at N=32N=32 and a 16,38416,384-token response limit. Parentheses give length relative to the corresponding method’s positive-branch mean over the same window; n is the positive-response count and “–” denotes an empty bin. ReSPO VESPO logW W bin W range Length n Length n (−∞,−20](-∞,-20] (0,2.1×10−9](0,2.1×10^-9] 2155(1.90×)2155\;(1.90×) 99 – 00 (−20,−10](-20,-10] 2.1×10−92.1×10^-9–4.5×10−54.5×10^-5 2208(1.95×)2208\;(1.95×) 5252 – 00 (−10,−5](-10,-5] 4.5×10−54.5×10^-5–0.0070.007 2227(1.97×)2227\;(1.97×) 240240 5593(6.35×)5593\;(6.35×) 88 (−5,−2](-5,-2] 0.0070.007–0.140.14 1501(1.32×)1501\;(1.32×) 16871687 1085(1.23×)1085\;(1.23×) 592592 (−2,−1](-2,-1] 0.140.14–0.370.37 1199(1.06×)1199\;(1.06×) 27532753 957(1.09×)957\;(1.09×) 20712071 (−1,−0.5](-1,-0.5] 0.370.37–0.610.61 1090(0.96×)1090\;(0.96×) 24222422 861(0.98×)861\;(0.98×) 22472247 (−0.5,0](-0.5,0] 0.610.61–11 885(0.78×)885\;(0.78×) 38673867 759(0.86×)759\;(0.86×) 38623862 (0,0.5](0,0.5] 11–1.651.65 994(0.88×)994\;(0.88×) 28772877 840(0.95×)840\;(0.95×) 30903090 (0.5,1](0.5,1] 1.651.65–2.72.7 1042(0.92×)1042\;(0.92×) 20362036 878(1.00×)878\;(1.00×) 20602060 (1,2](1,2] 2.72.7–7.47.4 1202(1.06×)1202\;(1.06×) 17941794 981(1.11×)981\;(1.11×) 17171717 (2,5](2,5] 7.47.4–148148 1835(1.62×)1835\;(1.62×) 535535 1259(1.43×)1259\;(1.43×) 364364 (5,10](5,10] 148148–2.2×1042.2×10^4 4985(4.40×)4985\;(4.40×) 66 13430(15.26×)13430\;(15.26×) 11 (10,20](10,20] 2.2×1042.2×10^4–4.9×1084.9×10^8 – 00 – 00 (20,∞)(20,∞) >4.9×108>4.9×10^8 – 00 – 00 Positive-branch mean – 11331133 18,27818,278 880880 16,01216,012 The allocation follows directly from the positive-branch kernels. ReSPO uses ϕ(+)(W)=1+W2exp(1−W),φ^(+)(W)= 1+W2 (1-W), which approaches e/2≈1.359e/2≈ 1.359 as W→0W→ 0 and remains 1.3481.348 at logW=−2 W=-2. VESPO uses ϕ(+)(W)=W2exp(3(1−W))φ^(+)(W)=W^2 (3(1-W)), which vanishes as W→0W→ 0 and falls to 0.2450.245 at logW=−2 W=-2 and 0.0060.006 at logW=−4 W=-4. At the same low value of W, ReSPO therefore retains a near-maximal coefficient on a positive response, whereas VESPO progressively removes its contribution. This kernel difference explains the coefficient-mass allocation in Tab. 5. E.4 MoE training and Routing Replay Both Qwen3-30B-A3B ReSPO runs at N=16N=16 reach the full budget of 10241024 policy updates. The Routing-Replay ablation adds R3 while matching the model, data, optimization, response cap, batch sizes, and update budget. Both use the current ReSPO parameters (α(+),α(−))=(2,1)(α^(+),α^(-))=(2,1), (β(+),β(−))=(0.5,0.5)(β^(+),β^(-))=(0.5,0.5), and (λ(+),λ(−))=(2,2)(λ^(+),λ^(-))=(2,2). Tab. 8 uses the same reporting protocol as Fig. 3. “Early peak” is the maximum through policy step 256256; late metrics are policy-step-weighted trapezoidal means over [896,1024][896,1024]; and “final” is the value at step 10241024. The score is (critic/score/mean+1)/2( critic/score/mean+1)/2. Because the 30B reward includes the soft overlong penalty described in App. D.1, this is a penalized training accuracy. The subscript on the late score denotes its within-run temporal standard deviation. Table 8: Matched Qwen3-30B-A3B ReSPO training diagnostics at N=16N=16, with and without R3 Routing Replay. All columns except the early peak and final score are time-weighted means over policy steps [896,1024][896,1024]. Setting Early peak Late score Final score ReSPO 0.4450.445 0.563±0.0210.563_ ± 0.021 0.5550.555 ReSPO ++ R3 0.4460.446 0.582±0.0080.582_ ± 0.008 0.5840.584 Setting Resp. len. Cap-hit rate Approx. KL Entropy Grad. norm ReSPO 19921992 3.505%3.505\% 3.37×10−43.37×10^-4 0.1280.128 1.131.13 ReSPO ++ R3 12131213 0.011%0.011\% 4.77×10−44.77×10^-4 0.1140.114 1.211.21 R3 leaves the early peak essentially unchanged but raises the late score by 1.841.84 p and reduces its temporal standard deviation from 2.112.11 to 0.750.75 p. It also substantially shortens responses and nearly eliminates response-cap hits in this window. Approximate KL rises slightly. In this matched N=16N=16 run, R3 combines with ReSPO, raising the late-stage score and reducing its temporal variation. Appendix F Limitations Compute resource limitation. Our compute budget does not support a longer Qwen3-30B-A3B study with a longer training response limit and additional policy updates. The reported 30B experiments instead use an 8,1928,192-token limit with the soft length penalty described in Sec. 4.1. Under our current training setup, extending both the context length and training horizon could require several weeks on 8×8×H200 GPUs. Owing to the same compute constraints, we run multiple ReSPO seeds only for N=32N=32, where the separation from the baselines is largest and robustness is most important to verify. Hyperparameter optimality. The ReSPO hyperparameters are selected primarily based on the analytical tail requirements and derivative matching at W=1W=1. This study does not exhaustively search over α, β, and λ; other choices may further improve particular model scales, reuse ratios, or response-length regimes. However, we want to emphasize that without extensive hyperparameter tuniing, matching the tail requirements only is enough to find a good algorithm. Appendix G Detailed Training Statistics and Reproducibility Fig. 3 uses the recorded critic/score/mean history. Each rollout batch supplies N policy updates, so the horizontal coordinate is training/global_step×N× N. The plotting script truncates every curve at 10241024 policy updates, applies (score+1)/2( score+1)/2 to the signed DAPO-MATH score, and validates the response cap and ReSPO negative-branch parameters. The early statistic is the maximum logged value through step 256256 and is reported without an error bar. For the late statistic, we linearly interpolate the curve over [896,1024][896,1024] and integrate its first and second moments to obtain the time-weighted mean and standard deviation. Figs. 5 and 6 use the same experiments and additionally report AIME25 evaluation accuracy, response length, approximate KL divergence, and entropy. For 1.7B at N=32N=32, App. E.3 resolves the aggregate response-length separation into positive-branch importance-weight and coefficient-mass statistics. The KL panel uses actor/ppo_kl for GRPO/GSPO and actor/approx_kl for VESPO/ReSPO. Figure 5: Detailed training statistics for Qwen3-1.7B-Base on DAPO-MATH. Rows correspond to N∈8,16,32N∈\8,16,32\. Training accuracy is (critic/score/mean+1)/2( critic/score/mean+1)/2, and the evaluation panel reports AIME25 best@4 accuracy. Figure 6: Detailed training statistics for Qwen3-30B-A3B-Base on DAPO-MATH. The training panel reports penalized training accuracy, and the evaluation panel reports AIME25 best@2 accuracy, the largest common best-of-k metric available for all four methods; the other panels follow Fig. 5.