Paper deep dive
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Yu Li, Xiuyu Li, Mingyang Yi, Jiaxing Wang, zhangliangxu, Zhaolong Xing, Zhen Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 7/7/2026, 3:28:21 PM
Summary
The paper introduces Selective Importance Sampling (SIS), a plug-in method for RL post-training of large language models that addresses the variance explosion problem inherent in off-policy training. By applying token-level rejection sampling, SIS converts off-policy tokens into on-policy tokens, assigning them a unit importance score while retaining standard correction for rejected tokens. This approach theoretically tightens the approximation error bound between token-level and sequence-level gradient estimators and empirically enhances training stability and performance across dense and Mixture-of-Experts architectures on math and agent benchmarks with negligible computational overhead.
Entities (12)
Relation Signals (10)
Selective Importance Sampling (SIS) → converts → Off-policy data to On-policy tokens
confidence 95% · transferring these off-policy token into on-policy token, so that the importance scores for correction are unnecessary.
Selective Importance Sampling (SIS) → theoreticallyproves → Reduces gap between token-level and sequence-level gradient estimators
confidence 95% · Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators.
Selective Importance Sampling (SIS) → modifies → Importance Sampling correction ratios
confidence 90% · only modifies the importance ratio in the policy loss
Selective Importance Sampling (SIS) → reduces → Variance explosion
confidence 90% · causing severe variance exploded... SIS implements by viewing off-policy model as proposal distribution... reduces the gap
Selective Importance Sampling (SIS) → uses → Rejection Sampling
confidence 90% · SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test
Selective Importance Sampling (SIS) → compatiblewith → GSPO
confidence 85% · can be combine with a vast vary of RL post-training algorithms e.g., GSPO
Selective Importance Sampling (SIS) → compatiblewith → GRPO
confidence 85% · can be combine with a vast vary of RL post-training algorithms e.g., GRPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.
Tags
Links
- Source: https://arxiv.org/abs/2607.04728v1
- Canonical: https://arxiv.org/abs/2607.04728v1
Trouble viewing inline? Open PDF directly →
Full Text
78,938 characters extracted from source content.
Expand or collapse full text
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment Yu Li1,∗, Xiuyu Li1,∗,‡, Mingyang Yi1,†, Jiaxing Wang2, zhangliangxu2, Zhaolong Xing2, Zhen Chen2 1Renmin University of China 2JD.com liyu0929,yimingyang@ruc.edu.cn ∗Equal contribution.†Corresponding author.‡Work done during an internship at JD.com. Abstract Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of “rollout then update”, which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is “transferring” these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data. 1 Introduction Reinforcement learning (RL) (Sutton et al., 1998) has become the dominant paradigm for post-training large language models, driving substantial progress in reasoning (Guo et al., 2025; Chen et al., 2025), code generation (Jiang et al., 2025), tool use (Li et al., 2026a), and broader capabilities (Li et al., 2026; 2026b; Zhang et al., 2026a; b; Tu et al., 2026). The RL process follows an order of “rollout then update”. However, owing to the considerations of efficiencies in sample reuse (Noukhovitch et al., 2025), asynchronous auto-regressive inference (Fu et al., 2026), researchers inevitably use off-policy data from stale policies to update model. The resulting distribution mismatch biases gradient estimates and degrades training stability (Ma et al., 2025), making effective off-policy gradient correction a fundamental challenge for RL-based post-training methods. Importance sampling (IS) (Tokdar and Kass, 2010) offers a principled correction to off-policy gradient by multiplying a correction ratio, but its variance can grow rapidly in long-horizon reasoning (Metelli et al., 2020), and is sensitive to some outliers (Schulman et al., 2017). To mitigate this, mainstream methods (Schulman et al., 2017; Shao et al., 2024; Yu et al., 2025) rely on hard clipping. Although effective for stabilization, hard clipping suppresses gradients from highly off-policy samples and loses useful learning signal. Some recent methods (Zheng et al., 2025b; Chen et al., 2025; Gao et al., 2025a) mitigate this issue by soft clipping, but remain heuristic. Instead of working on clipping method, we address this problem by “transferring” the off-policy tokens into on-policy tokens, which fundamentally resolve the off-policy problem. Our method, Selective Importance Sampling (SIS), a simple and effective method conduct this transfer via acceptance-rejection sampling (Robert and Casella, 2004). Concretely, SIS views the behavior policy create training data as a proposal distribution to the current policy. By implementing a rejection sampling, i.e., a token-level acceptance test, the accepted off-policy tokens from behavior policy can be viewed as on-policy tokens and receive unit weight; rejected tokens retain the standard IS correction. To further improve our SIS, we propose an efficient approximation to the acceptance test process, making our SIS brought negligible efforts in both computation and efficiency. More than the practical method, we further prove that our SIS strictly reduces the cumulative distributional mismatch across the sequence (Ma et al., 2025), tightening the off-policy approximation error bound, so that theoretically improves the current post-training methods. Empirically, SIS delivers consistent gains as a plug-in module across diverse backbones models (dense and Mixture-of-Experts (Yang et al., 2025a)), RL algorithms, and benchmarks (math (MAA, 2024) and agent tasks (Jin et al., 2025)). Moreover, it enhances training stability under challenging off-policy regimes, and ablation studies confirm that it is insensitive to hyperparameter choices. Our contributions are as follows: • We propose SIS, a plug-in method that can be combined with most mainstream RL post-training methods e.g., GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), GSPO (Zheng et al., 2025b). The method mitigates off-policy mismatch in LLM post-training in a rejection sampling regime. • We provide theoretical justification for SIS, proving that its token-level rejection mechanism reduces off-policy approximation error. • We conduct extensive experiments spanning diverse model architectures, RL algorithms, and task domains, demonstrating that SIS delivers consistent gains and stronger stability under challenging off-policy regimes. 2 Background Off-Policy Policy Gradient. In this section, we present the necessary background knowledge of this paper. Given a query x from the dataset D, the response denoted as y=(y1,y2,…,yT)y=(y_1,y_2,…,y_T) is sampled from LLM πθ(y∣x) _θ(y x). The response is evaluated by a reward model r(x,y)r(x,y). To optimize the policy LLM, we usually maximize the following regularized expected reward.111Here we ignore the KL regularization (Ouyang et al., 2022) to simplify the notation, our analysis can be similarly conducted with it involved. (θ)=x∼,y∼πθ(⋅∣x)[r(x,y)],J(θ)=E_x ,y _θ(· x)[r(x,y)], (1) The policy gradient of this objective with respect to the policy parameters can be computed via the log-trick (Sutton et al., 1998) by ∇θ(θ)=x∼,y∼πθ(⋅∣x)[(A(x,y))∇θlogπθ(y∣x)], _θJ(θ)=E_x ,y _θ(· x) [ (A(x,y) ) _θ _θ(y x) ], (2) where A(x,y)A(x,y) denotes the advantage associated with response y under prompt x, which results in unbiased gradient estimator e.g., A(x,y)=r(x,y)−cA(x,y)=r(x,y)-c for some constant c (Schulman et al., 2017). In typical large-scale settings, due to the limitations of sample reuse, rollout efficiency e.t.c. (Fu et al., 2026), online sampling from πθ _θ to estimate policy gradient is infeasible, and training is performed on a fixed set of responses collected from a stale behavior policy πθold _ _old. To reproduce the unbiased gradient estimator, we usually use importance sampling correction ratio w(θ)=πθ(y∣x)πθold(y∣x)w(θ)= _θ(y x) _ _old(y x) ∇θ(θ)=x∼,y∼πθold(⋅∣x)[w(θ)A(x,y)∇θlogπθ(y∣x)]. _θJ(θ)=E_x ,y _ _old(· x) [w(θ)A(x,y) _θ _θ(y x) ]. (3) The sequence-level correction ratio w(θ)w(θ) can be viewed as a factor to account for the discrepancy between the target and behavior policies. In practice, this ratio may become excessively large, leading to exploding gradients and unstable training (Schulman et al., 2017; Yu et al., 2025). To mitigate this, w(θ)w(θ) is typically modified through clipping or normalization to ensure more stable training as proposed in (Schulman et al., 2017; Zheng et al., 2025b). Token-level Importance Sampling. For autoregressive models, the sequence likelihood admits the following factorization πθ(y∣x)=∏t=1Tπθ(yt∣x,y<t). _θ(y x)= _t=1^T _θ(y_t x,y_<t). (4) Consequently, the sequence-level ratio w(θ)w(θ) decomposes into a product of token-level ratios w(θ)=∏t=1Twt(θ),wt(θ)=πθ(yt∣x,y<t)πθold(yt∣x,y<t).w(θ)= _t=1^Tw_t(θ), w_t(θ)= _θ(y_t x,y_<t) _ _old(y_t x,y_<t). (5) This product scales exponentially with sequence length, leading to severe variance in gradient estimation (Metelli et al., 2020). To mitigate this, practical implementations replace the sequence-level importance ratio in gradient estimation by single token-level ratios (Ma et al., 2025) and results in approximated gradient g~(θ)=y∼πθold(⋅∣x)[∑t=1Twt(θ)A(x,y)∇θlogπθ(yt∣x,y<t)]. g(θ)=E_y _ _old(· x) [ _t=1^Tw_t(θ)A(x,y) _θ _θ(y_t x,y_<t) ]. (6) Although g~(θ) g(θ) does not equal the true policy gradient ∇θ(θ) _θJ(θ), it provides a reliable approximation as long as πθ _θ stays sufficiently close to πθold _ _old(Kakade and Langford, 2002; Zheng et al., 2025a), since wt(θ)=πθ(yt∣x,y<t)/πθold(yt∣x,y<t)≈1w_t(θ)= _θ(y_t x,y_<t)/ _ _old(y_t x,y_<t)≈ 1. When the two policies diverge, this approximation gap grows, producing biased gradient estimates that destabilize training. Controlling this gap is therefore central to reliable off-policy LLM training; we formalize this in Section 3.3. 3 Method: Turning Off-Policy Tokens On-Policy In this section, we propose our Selective Importance Sampling (SIS), a simple yet effective method building upon rejection sampling that directly converts off-policy tokens into on-policy ones at the token level, so that obviate the aforementioned problems brought by off-policy data. Beyond the SIS, we propose an efficient implementation of it to make it a plug-in with negligible efforts of existing algorithms. Finally, we prove the theoretical improvements of SIS in terms of provably tight sequence-versus-token approximation error bound. Figure 1: Overview of SIS. At each token position, SIS applies a rejection sampling to determine whether the token can be treated as on-policy (w~t(θ)=1 w_t(θ)=1) or should retain its off-policy importance ratio. The modified ratios are then plugged into the gradient estimator of any existing algorithm. 3.1 Converting Off-Policy Tokens via Rejection Sampling The high-level idea of SIS is straightforward (see Figure 1 for an illustration): at each token position, the behavior policy πθold _ _old can be viewed as a proposal distribution to sample from current policy πθ _θ under acceptance-rejection sampling. Then, by a acceptance-rejection test, the accepted tokens from behavior policy can be recognized from current policy, and become on-policy tokens with identical importance ratio; otherwise, the token retains its original importance ratio as in existing methods. In this way, SIS reduces the variance of importance ratios while preserving gradient signal. Concretely, for each position t with prefix (x,y<t)(x,y_<t), we compute the constant Mt=maxv∈πθ(v∣x,y<t)πθold(v∣x,y<t),M_t= _v _θ(v x,y_<t) _ _old(v x,y_<t), (7) where V denotes the vocabulary, πθold(⋅∣x,y<t) _ _old(· x,y_<t), and πθ(⋅∣x,y<t) _θ(· x,y_<t) are respectively proposal and target distributions. Given the observed token yty_t with ratio wt(θ)=πθ(yt∣x,y<t)πθold(yt∣x,y<t)w_t(θ)= _θ(y_t x,y_<t) _ _old(y_t x,y_<t), we draw an acceptance indicator zt∼Bernoulli(wt(θ)Mt).z_t \! ( w_t(θ)M_t ). (8) By the standard theory of rejection sampling, acceptance (zt=1z_t=1) certifies that yty_t follows exactly the target distribution πθ(yt∣x,y<t) _θ(y_t x,y<t). We state this process formally as below. Proposition 1. Let yt∼πθold(⋅∣x,y<t)y_t _ _old(· x,y_<t) and zt∼Bernoulli(wt(θ)/Mt)z_t (w_t(θ)\,/\,M_t) as defined in (8). Then ℙ(yt=v∣zt=1)=πθ(v∣x,y<t),∀v∈.P(y_t=v z_t=1)= _θ(v x,y_<t), ∀\,v . (9) Proposition 1 states that, although yty_t is sampled from πθold _ _old, its conditional distribution over zt=1z_t=1 becomes the distribution of πθ _θ. In other words, accepted tokens are already on-policy samples and require no importance correction. Setting their weight to one therefore introduces zero bias. Accordingly, the modified token-level ratio w~t(θ)=1,zt=1(on-policy, weight removed),wt(θ),zt=0(off-policy, weight retained), w_t(θ)= cases1,&z_t=1 (on-policy, weight removed),\\ w_t(θ),&z_t=0 (off-policy, weight retained), cases (10) plugged in (6) results in identical gradient estimation. We visualize the accepted rate of the rejection process in Figure 2(A). As can be seen, the accept rate is substantial in both scenarios, demonstrating that a large fraction of tokens can be converted from off-policy to on-policy. This indicates that the modification introduced by SIS is no trivial. The lower accept rate in the agentic setting is expected, as interleaving reasoning with diverse tool calls introduces greater distributional shift between πθold _ _old and πθ _θ. Figure 2: Empirical validation of the three components of SIS on Qwen3-8B-Base. (A) Accept rate of rejection sampling on math and agentic search; higher means more off-policy tokens are converted to on-policy (Appendix D.2 for 30B-MoE and 14B). (B) Residual mass ξK _K outside the top-K set (upper) and cumulative coverage (lower) versus K; lower ξK _K and coverage near 1.01.0 indicate a tight top-K envelope. (C) Total log-importance deviation D for vanilla GRPO vs. GRPO+SIS; a lower curve and larger shaded gap correspond to a tighter optimization bound. Remark 1. In standard PPO algorithm, wt(θ)w_t(θ) is processed by hard clipping as in (15). This is originated from Trust-Region Policy Optimization (TRPO) (Schulman et al., 2015; Qi et al., 2026), which regularize the KL divergence between behavior policy and current policy. However, as pointed in (Qi et al., 2026), the clipping heavily relies on the sampled token, which introduces large noise in determining trust-region. However, our rejection process distributionally compare the behavior policy and current policy, so that also mitigate the noise introduced by sampling process. 3.2 Practical Top-K Envelope Approximation While the above rejection sampling procedure provides an unbiased mechanism for turning off-policy tokens on-policy, computing the maximal constant MtM_t in (7) requires maximizing over the entire vocabulary, which is computationally prohibitive. To mitigate this, we introduce a top-K approximation to this constant. Our method is built upon the fact that the probability mass of LLM is concentrated on a small subset of vocabulary, so that we can approximate MtM_t with the maximization restricted to the top-K. Concretely, we define M^t=maxv∈Kπθ(v∣x,y<t)πθold(v∣x,y<t),K=TopK(πθold(⋅∣x,y<t),K), M_t= _v _K _θ(v x,y_<t) _ _old(v x,y_<t), _K=TopK( _ _ old(· x,y_<t),\,K), (11) and replace MtM_t in (8) with it. Under this approximation, when computing acceptance indicator ztz_t (8), only tokens within KV_K are considered during rejection sampling, while tokens outside this set are still off-policy data. The approximated algorithm is summarized in Algorithm 1 in Appendix B. In this regime, we have the following proposition. Proposition 2. When approximating SIS with Algorithm 1, the accepted tokens follows conditional distribution π^θ(v∣x,y<t)=πθ(v∣x,y<t,v∈K) π_θ(v x,y_<t)= _θ(v x,y_<t,v _K). Beside that, let ξK=∑v∉Kπθ(v∣x,y<t) _K= _v _K _θ(v x,y_<t) denote the probability mass of the target policy outside the KV_K, then we have DTV(π^θ(⋅∣x,y<t)∥πθ(⋅∣x,y<t))=ξK,D_ TV\! ( π_θ(· x,y_<t) _θ(· x,y_<t) )= _K, (12) where DTVD_ TV is the total variation distance. The proposition indicates that the “on-policy” data obtained by our efficiency-aware implementation follows a conditional distribution, which is close to the target when ξK→0 _K→ 0. Since the top-K tokens of πθ _θ and πθold _ _ old will be overlapped with the increasing of K, and only a small number of high-probability tokens concentrate the vast majority of the probability mass, the ξK _K remains small in practice, even for a relative small K, e.g., K=5K=5 (see Figure 2(B)). Complexity analysis. Beyond approximation quality, the top-K formulation also keeps computational cost low. SIS reuses the old-policy logits from generation and the current-policy logits from training, requiring no extra model forward pass. The only additional work per token is selecting the top-K candidates under the old policy and computing their acceptance probabilities from cached logits, which adds only ∼1% 1\% wall-clock overhead per training step (see Appendix D.3). 3.3 Why SIS Works: A Theoretical Perspective The preceding subsections established what SIS does: selectively resetting off-policy tokens to on-policy via rejection sampling (§3.1), and how to implement it efficiently via a top-K approximation process (§3.2). We now turn to why this mechanism is effective by revisiting the fundamental question raised in (6) of Section 2: how does the gap between the token-level and sequence-level gradient estimates grow as the behavior policy drifts, and can SIS provably reduce it? To answer this, we first characterize the approximation error E=‖seq−tok‖E=\| g_ seq- g_ tok\| between the ground-truth gradient seq g_ seq (3) with sequence-level correction ratio (5) w(θ)w(θ) and its surrogate tok g_ tok (6) with token-level correction ratio wt(θ)w_t(θ) (5). Theorem 1. For a sequence of length T, we have E≤|A(x,y)|(eD−1)∑t=1T‖st‖,E≤|A(x,y)|\,(e^D-1) _t=1^T\|s_t\|, (13) where D=∑t=1T|logwt(θ)|D= _t=1^T|\! w_t(θ)|, A(x,y)A(x,y) is the advantage and st=∇θlogπθ(yt∣x,y<t)s_t= _θ _θ(y_t x,y_<t). The proposition is proved in Appendix A.2. As can be seen, the total log-importance deviation D=∑t=1T|logwt(θ)|D= _t=1^T|\! w_t(θ)| serves as the key quantity governing the approximation error E, which is consistent to the fact that for the gradient on each token, the real correction w(θ)=∏t=1Twt(θ)w(θ)= _t=1^Tw_t(θ) is replaced with wt(θ)w_t(θ). This exponential dependence explains why off-policy training destabilizes rapidly as the behavior policy drifts. The drift makes wt(θ)≉1w_t(θ) ≈ 1 (1≤t≤T1≤ t≤ T) so that wt(θ)≉w(θ)=∏t=1Twt(θ)w_t(θ) ≈ w(θ)= _t=1^Tw_t(θ). Next, we use a proposition to illustrate that estimating the policy gradient with our SIS strictly reduces D, thereby tightening the error bound (13). Proposition 3. Under the same conditions as Theorem 1, replacing the correction ratios wt(θ)w_t(θ) in (6) with the ratios w~t(θ) w_t(θ) (10) in SIS yields E≤|A(x,y)|(eDSIS−1)∑t=1T‖st‖≤|A(x,y)|(eD−1)∑t=1T‖st‖,E≤|A(x,y)|\,(e^D_SIS-1) _t=1^T\|s_t\|≤|A(x,y)|\,(e^D-1) _t=1^T\|s_t\|, (14) where DSIS=∑t=1T|logw~t(θ)|=∑t=1T(1−zt)|logwt(θ)|≤D_SIS= _t=1^T\,|\! w_t(θ)|= _t=1^T(1-z_t)\,|\! w_t(θ)|≤ D for ztz_t defined in (8). The proof to this proposition is in Appendix A.2. Intuitively, the correction ratio w(θ)=∏t=1Twt(θ)w(θ)= _t=1^Tw_t(θ) is contributed by each off-policy token wt(θ)w_t(θ). However, our SIS transfers most off-policy tokens (see Figure 2) into on-policy tokens, which strictly reduces the bias D of Theorem 1 into DSISD_SIS in Proposition 3. Figure 2(C) corroborates this empirically: SIS consistently maintains a lower deviation than vanilla GRPO on both mathematical reasoning and agentic search tasks, directly translating to a tighter approximation error bound and more stable policy optimization. 3.4 Plugging SIS into Existing Algorithms Notably, our SIS is algorithm-agnostic: it only modifies the token-level correction ratio wt(θ)→w~t(θ)w_t(θ)→ w_t(θ) (cf. (10)), while leaving the rest of the components unchanged. Thus it can be combined with any RL-post training algorithms implemented on off-policy data e.g., GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), GSPO (Zheng et al., 2025b). Below we illustrate the combination with with GRPO; instantiations with more algorithms are given in Appendix E.2. For a prompt x from query set D and a group of G responses yii=1G∼πθold(⋅∣x)\y_i\_i=1^G _ _old(· x), we write w~i,t(θ) w_i,t(θ) for the SIS-modified ratio of the t-th token in yiy_i. GRPO + SIS. Applying our proposed SIS to GRPO (Shao et al., 2024) yields GRPOSIS(θ) _ GRPO SIS(θ) =x∼,yii=1G∼πθold(⋅∣x) =E_x ,\y_i\_i=1^G _ _ old(· x) (15) [1G∑i=1G1|yi|∑t=1|yi|min(w~i,t(θ)A^i,clip(w~i,t(θ),1−ε,1+ε)A^i)−βDKL(πθ∥πref)], [ 1G _i=1^G 1|y_i| _t=1^|y_i| \! ( w_i,t(θ) A_i,\;clip( w_i,t(θ),1- ,1+ ) A_i )-β D_ KL( _θ _ ref) ], where A^i A_i is the standard group-based normalized advantage. The only modification is substituting wi,t(θ)→w~i,t(θ)w_i,t(θ)→ w_i,t(θ), confirming the plug-in nature of SIS. 4 Experiments We empirically verify SIS along three axes: (1) consistent plug-in gains across policy-gradient algorithms, backbones, and tasks; (2) improved training stability under challenging off-policy regimes; (3) robustness to its hyperparameter and interpretable token-level selectivity. 4.1 Experimental Setup Models and Datasets. We conduct experiments with three Qwen3 (Yang et al., 2025a) backbones: the dense Qwen3-8B-Base, Qwen3-14B-Base, and the Mixture-of-Experts (MoE) Qwen3-30B-A3B-Base. We mainly explore our results on math (Shao et al., 2024) and reasoning with search agent (Jin et al., 2025), and train our models on DAPO-Math-17K corpus (Yu et al., 2025) and the merged training sets of NQ (Kwiatkowski et al., 2019) and HotpotQA (Yang et al., 2018), respectively. More experiments on Llama3.2 (Grattafiori et al., 2024) backbones are in Appendix D.6. Evaluation. We evaluate on 10 benchmarks across three settings: Math: MATH500 (Hendrycks et al., 2021), AMC23 (Ouyang et al., 2022), AIME24 and AIME25 (MAA, 2024); General QA (Agent): Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2022); Multi-Hop QA (Agent): HotpotQA (Yang et al., 2018), Musique (Trivedi et al., 2022), and Bamboogle (Bamb) (Press et al., 2023). The QA datasets are designed for reasoning with search agent. We report Avg@1 accuracy for most tasks and Avg@32 for the smaller AMC23, AIME24, and AIME25. The experiments on math are conducted on Qwen3-8B-Base and Qwen3-30B-A3B-Base, and the experiments on agent are under Qwen3-8B-Base and Qwen3-14B-Base. Baselines. We integrate SIS into three representative RL objectives: token-level GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025), and sequence-level GSPO (Zheng et al., 2025b). We also compare against recent methods that address off-policy error through alternative mechanisms: clipping-based CISPO (Chen et al., 2025), divergence-based DPPO-TV (Qi et al., 2026), and entropy-based Clip-Cov (Cui et al., 2025). Each SIS run is paired with its corresponding baseline under identical hyperparameters and infrastructure; the only modification is replacing wi,t(θ)w_i,t(θ) with the SIS-modified weight w~i,t(θ) w_i,t(θ) in (10). All runs use FSDP (Zhao et al., 2023) for training and vLLM (Kwon et al., 2023) for rollouts. Detailed hyperparameters and descriptions of all comparison methods are provided in Appendix C. 4.2 Main Results We summarize our main results in Tables 1 and 2. As can be seen in Table 1, our SIS consistently improves all policy-gradient algorithms on dense (Qwen3-8B/14B-Base) and MoE (Qwen3-30B-A3B-Base) architectures across math and agentic reasoning tasks, yielding average gains of up to +6.37+6.37 on math and +2.71+2.71 on agent benchmarks. On other hands, Table 2 unifies recent policy-gradient-based methods, which are actually GRPO under varied wi,t(θ)=g(w)w_i,t(θ)=g(w). SIS instead decomposes in two-folds: accepted tokens are corresponded to wi,t(θ)=1w_i,t(θ)=1, while rejected tokens are handled by any base algorithm g(⋅)g(·). This composability gives SIS a guaranteed floor and a high ceiling: the trivial g(w)=wg(w)=w (no trick) already beats most methods, while pairing g(⋅)g(·) with a good algorithm e.g.., DAPO attains the best accuracy. Table 1: Performance of SIS as a plug-in across math and agent benchmarks. Shaded rows denote the application of SIS on top of each baseline. The best accuracy per model is bolded. Δ denotes the absolute gain over the corresponding baseline. Method Mathematical Reasoning Agentic Search MATH500 AMC23 AIME24 AIME25 Avg Δ NQ TriviaQA PopQA HotpotQA Musique Bamb Avg Δ Qwen3-8B-Base Qwen3-8B-Base GRPO 80.4 74.14 23.44 18.23 49.05 – 49.14 66.99 44.68 46.24 19.24 48.8 45.85 – !12 w/ SIS 85.6 79.53 25.83 19.38 52.59 +3.54 49.78 67.47 49.29 47.90 19.74 52.8 47.83 +1.98 DAPO 83.2 81.88 21.56 18.33 51.24 – 49.61 65.69 45.95 42.89 16.17 47.2 44.59 – !12 w/ SIS 84.2 86.79 26.98 22.60 55.14 +3.90 50.11 67.66 47.83 47.83 21.18 51.2 47.64 +3.05 GSPO 85.6 78.83 26.46 18.85 52.44 – 49.72 67.53 46.93 47.12 20.52 45.6 46.24 – !12 w/ SIS 85.2 77.19 28.33 23.44 53.54 +1.10 49.09 68.18 46.97 46.97 22.26 52.8 47.71 +1.47 Qwen3-30B-A3B-Base Qwen3-14B-Base GRPO 82.8 78.59 25.52 18.23 51.29 – 50.64 68.21 47.42 47.70 22.13 52.1 48.03 – !12 w/ SIS 85.8 83.91 35.94 25.00 57.66 +6.37 50.86 70.13 48.54 47.94 23.91 56.2 49.60 +1.57 DAPO 84.2 81.02 29.17 18.96 53.34 – 50.22 68.51 48.61 48.01 22.09 51.2 48.11 – !12 w/ SIS 85.2 84.77 35.10 27.40 58.12 +4.76 51.55 69.25 50.05 49.98 23.91 54.4 49.86 +1.75 GSPO 75.0 77.34 34.90 25.52 53.19 – 51.58 69.36 51.36 47.01 20.98 43.2 47.25 – !12 w/ SIS 85.4 79.38 35.41 24.38 56.14 +2.95 52.22 68.22 51.58 47.47 21.02 46.4 47.82 +0.57 Table 2: Trick-free training on Qwen3-8B-Base math benchmarks. All variants build on GRPO; the wi,t(θ)w_i,t(θ) column gives each method’s effective coefficient (definitions of TVi,tTV_i,t, Covi,tCov_i,t in Appendix E). SIS sets w~=1 w=1 for accepted tokens and applies any trick g(w)g(w) to the rejected ones. Best bolded, second-best underlined; Δ is the relative Avg gain over GRPO. Method wi,t(θ)w_i,t(θ) MATH500 AMC23 AIME24 AIME25 Avg Δ (%) No correction w/o IS 11 79.6 67.95 23.04 18.46 47.26 -3.65 w/o clip w 78.6 72.34 22.82 18.75 48.13 -0.92 Baseline GRPO clip(w, 1−ε, 1+ε)clip(w,\,1- ,\,1+ ) 80.4 74.14 23.44 18.23 49.05 – Stabilization tricks + Clip-Higher (DAPO) clip(w, 1−εlo, 1+εhi)clip(w,\,1- _ lo,\,1+ _ hi) 83.2 81.88 21.56 18.33 51.24 +4.46+4.46 + Seq-level IS (GSPO) clip(si, 1±ε),si=(∏twi,t)1/|yi|clip(s_i,\,1± ),\ s_i=( _tw_i,t)^1/|y_i| 85.6 78.83 26.46 18.85 52.44 +6.91+6.91 + Clipped-IS (CISPO) sg[min(w, 1+εhi)]sg[ (w,\,1+ _ hi)] 84.0 85.55 26.46 21.77 54.44 +11.0+11.0 + Divergence (DPPO-TV) w⋅[TVi,t≤δ]w· 1[TV_i,t≤δ] 81.4 73.83 23.85 18.85 49.48 +0.88+0.88 + Clip-Cov (Entropy) w⋅[Covi,t≤ω]w· 1[Cov_i,t≤ω] 80.8 76.56 24.17 18.23 49.94 +1.81+1.81 Ours accept reject g(w)g(w) + SIS (vanilla) 11 w 82.2 76.72 28.33 20.00 51.81 +5.63+5.63 + SIS (GRPO) 11 clip(w, 1±ε)clip(w,\,1± ) 85.6 79.53 25.83 19.38 52.59 +7.22+7.22 !12 + SIS (DAPO) 11 clip(w, 1−εlo, 1+εhi)clip(w,\,1- _ lo,\,1+ _ hi) 84.2 86.79 26.98 22.60 55.14 +12.4+12.4 4.3 SIS Improves Training Stability Next, we explore whether SIS improves training stability under challenging off-policy regimes, since it transfers most off-policy tokens into on-policy. We design three stress tests: (1) increasing policy staleness by reusing rollouts for up to 16 gradient updates, (2) amplifying train-inference mismatch via MoE routing divergence, and (3) removing clipping entirely. All comparisons keep the paired protocol from §4.1. Figure 3: Training dynamics under increasing policy staleness (N=4, 8, 16) on Qwen3-8B-Base. Columns: GRPO, DAPO, GSPO. Rows: AIME24 and AIME25 accuracy. Dashed lines are baselines; solid lines add SIS. Increasing N degrades the baselines, whereas the corresponding SIS variants stay consistently above their baselines at every staleness level. SIS resists degradation from stale rollouts. Reusing rollouts for multiple gradient updates is common practice to amortize generation cost, but increases the degree of off-policy-ness, and results in performance drop (Fu et al., 2026). We vary N, the number of mini-batch updates per rollout, from 4 to 16 on Qwen3-8B-Base to explore whether SIS remains effective as staleness grows. Figure 3 reveals two observations: (i) staleness is harmful in itself: increasing N degrades accuracy regardless of whether SIS is applied, confirming that reusing stale off-policy rollouts hurts RL, though increasing computational efficiency; (i) SIS yields a stable gain at every staleness level: each SIS variant stays consistently above its corresponding baseline across all N, because SIS maintains an appreciable accept rate even under large N (Appendix D.5). Together, these results show that SIS reliably recovers performance under stale rollouts (detailed scores in Tab. 8). SIS handles train-inference mismatch in MoE models. In MoE architectures, the rollout engine and the trainer may route tokens through different experts, producing mismatched log-probabilities even at the same checkpoint (Zheng et al., 2025a; Zhao et al., 2025). We test whether SIS can handle this system-level off-policy divergence on Qwen3-30B-A3B-Base, comparing against Rollout Routing Replay (R3) (Ma et al., 2025), a method specifically designed for MoE routing mismatch. As shown in Figure 4, vanilla DAPO collapses entirely: accuracy peaks and then degrades, entropy drops to near zero, and gradient norms spike by two orders of magnitude. In contrast, both R3 and our SIS mitigate part of this instability, while combining SIS with R3 yields the strongest accuracy and the most stable optimization, indicating that SIS and R3 address complementary sources of off-policy divergence. SIS enables clipping-free training. SIS is itself a stabilization mechanism: with clipping fully disabled, GRPO+SIS still beats clip-based GRPO on math and agent tasks (Appendix D.1). 4.4 Sensitivity and Token-Level Analysis We finally ablate the only hyperparameter of SIS and inspect its token-level acceptance behavior. Top-K sensitivity. SIS introduces a single hyperparameter K that defines the rejection envelope. Sweeping K∈10,50,100K∈\10,50,100\ on Qwen3-8B-Base, Table 5 shows that SIS is robust to K: average accuracy varies by at most ∼ 1 point, and we use K=10K=10 throughout the paper. Figure 4: Training dynamics on Qwen3-30B-A3B-Base (MoE) with DAPO. Columns: AIME24/25 accuracy, entropy, gradient norm. Vanilla DAPO (blue dashed) suffers entropy collapse and gradient spikes; R3 (orange dashed) and SIS (green solid) each mitigate part of the instability; combining SIS with R3 (red solid) gives the strongest accuracy and the most stable optimization. Token-level acceptance behavior. We further inspect what the rejection sampler accepts or rejects at the token level. Figure 5 visualizes the most distinctive tokens among accepted (green) and rejected (red) samples. Accepted tokens are dominated by mathematical reasoning terms (e.g., “frac”), while rejected ones contain more formatting and web artifacts (e.g., “$image”), suggesting that SIS preferentially grants on-policy status to tokens aligned with the target reasoning distribution. Table 3: Sensitivity to top-K truncation on Qwen3-8B-Base. Method K MATH500 AMC23 AIME24 AIME25 Avg GRPO + SIS 10 85.6 79.53 25.83 19.38 52.59 50 84.6 78.83 26.67 20.00 52.53 100 84.8 77.50 26.88 19.79 52.24 DAPO + SIS 10 84.2 85.00 25.52 22.19 54.23 50 84.2 84.45 26.98 22.60 54.56 100 84.0 85.08 26.77 22.19 54.51 GSPO + SIS 10 83.2 76.25 27.92 21.67 52.26 50 85.2 77.19 28.33 23.44 53.54 100 83.8 76.95 26.67 22.40 52.46 Figure 5: Token-level SIS decisions. Word clouds of distinctive tokens in accepted(left, green) vs. rejected(right, red) samples. SIS favors mathematical reasoning tokens and rejects formatting/web artifacts. 5 Related Work Importance Sampling Ratios for LLM RL. Importance-sampling correction for off-policy training dates back to Schulman et al. (2015); Sutton et al. (1998), but the per-token ratio introduces large variance and unstable training (Schulman et al., 2017; Qi et al., 2026). The dominant remedy is hard clipping (Shao et al., 2024; Yu et al., 2025; Cui et al., 2025), which bounds the ratio but discards gradient information from highly off-policy tokens; soft clipping (Chen et al., 2025; Gao et al., 2025a) retains gradients at the cost of sensitive hyperparameters, and other clipping variants (Çağatan et al., 2026; Le Roux et al., 2025; Lei et al., 2026) follow the same recipe—shrinking large ratios to suppress variance, which inevitably attenuates the gradient signal. SIS instead transfers off-policy tokens to on-policy, fixing the off-policy mismatch at its root. Rejection Sampling and Sample Selection. Rejection sampling has appeared in the LLM pipeline only as a data curation step rather than a sampling mechanism: RSO (Liu et al., 2024; Tajwar et al., 2026) filters preference pairs before reward-model training, best-of-N (Gui et al., 2024; Beirami et al., 2025; Sun et al., 2024; Li et al., 2026) re-ranks candidates at inference, and other work removes OOD or unsafe samples from the training corpus (Guo et al., 2025). Unlike SIS, these response-level pre/post-processing steps do not modify the policy gradient. Stabilizing RL Post-Training. A parallel line of work stabilizes training at the systems side, where staleness arises from asynchronous rollouts (Fu et al., 2026; Noukhovitch et al., 2025; Gao et al., 2025b; Team et al., 2025) or from train–inference engine mismatch in MoE models (Zheng et al., 2025a). Common remedies all limit the influence of off-policy data: masking high-deviation tokens (Li et al., 2025), reusing only sparse rollouts (Luo et al., 2026; Wan et al., 2026), replaying inference-time routing via R3 (Ma et al., 2025), or filtering discrepant tokens via IcePop (Zhao et al., 2025). SIS instead converts off-policy tokens into on-policy, and is orthogonal to these tricks; in particular, combining SIS with R3 on MoE yields strictly stronger stability than either alone. 6 Conclusion We present Selective Importance Sampling (SIS) to tackle the off-policy bottleneck in post-training of LLMs, where stale rollouts and train–inference mismatch bias the policy gradient. Rather than suppressing the correction ratio as in prior remedies, we reframe the problem as distribution conversion. Concretely, SIS performs a token-level acceptance–rejection test with a top-K approximation: accepted tokens are treated as fully on-policy, while rejected ones retain the standard IS correction. SIS only modifies the importance ratio, so it plugs into any policy-gradient objective with negligible overhead; it also provably tightens the off-policy approximation-error bound. Empirically, SIS delivers consistent gains across mainstream algorithms and dense/MoE backbones, improves stability under heavy rollout reuse and MoE routing mismatch. References A. Beirami, A. Agarwal, J. Berant, A. D’Amour, J. Eisenstein, C. Nagpal, and A. Suresh (2025) Theoretical guarantees on the best-of-n alignment policy. In Forty-second International Conference on Machine Learning, Cited by: §5. Ö. V. Çağatan, B. Akgün, G. G. Şahin, and X. Zhao (2026) Clipping-free policy optimization for large language models. Preprint Technical Report arXiv:2601.22801. Cited by: §5. A. Chen, A. Li, B. Gong, B. Jiang, B. Fei, B. Yang, B. Shan, C. Yu, C. Wang, C. Zhu, et al. (2025) Minimax-m1: scaling test-time compute efficiently with lightning attention. Preprint Technical Report arXiv:2506.13585. Cited by: §E.1, §1, §1, §4.1, §5. G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. (2025) The entropy mechanism of reinforcement learning for reasoning language models. Preprint Technical Report arXiv:2505.22617. Cited by: §E.1, §4.1, §5. W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, et al. (2026) Areal: a large-scale asynchronous reinforcement learning system for language reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1, §2, §4.3, §5. C. Gao, C. Zheng, X. Chen, K. Dang, S. Liu, B. Yu, A. Yang, S. Bai, J. Zhou, and J. Lin (2025a) Soft adaptive policy optimization. Preprint Technical Report arXiv:2511.20347. Cited by: §1, §5. J. Gao, W. Fu, M. Xie, S. Xu, C. He, Z. Mei, B. Zhu, and Y. Wu (2025b) Beyond ten turns: unlocking long-horizon agentic search with large-scale asynchronous rl. Preprint Technical Report arXiv:2508.07976. Cited by: §5. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. Preprint Technical Report preprint arXiv:2407.21783. Cited by: §D.6, §4.1. L. Gui, C. Garbacea, and V. Veitch (2024) BoNBoN alignment for large language models and the sweetness of best-of-n sampling. In The Thirty-seventh Annual Conference on Neural Information Processing Systems, Cited by: §5. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. Preprint Technical Report arXiv:2501.12948. Cited by: §1, §5. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Cited by: §4.1. X. Jiang, Y. Dong, M. Liu, H. Deng, T. Wang, Y. Tao, R. Cao, B. Li, Z. Jin, W. Jiao, et al. (2025) CodeRL+: improving code generation via reinforcement with execution semantics alignment. Preprint Technical Report arXiv:2510.18471. Cited by: §1. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. Preprint Technical Report arXiv:2503.09516. Cited by: §C.1, §C.2, §C.3, §1, §4.1. M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017) TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics, Cited by: §4.1. S. Kakade and J. Langford (2002) Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, Cited by: §2. V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Cited by: §C.1. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. (2019) Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics. Cited by: §4.1, §4.1. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Symposium on Operating Systems Principles, Cited by: §4.1. N. Le Roux, M. Bellemare, J. Lebensold, A. Bergeron, J. Greaves, A. Fréchette, C. Pelletier, E. Thibodeau-Laufer, S. Tóth, and S. Work (2025) Tapered off-policy REINFORCE: stable and efficient reinforcement learning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §5. S. Lei, Z. Cheng, and D. Tao (2026) A step back: prefix importance ratio stabilizes policy optimization. Preprint Technical Report arXiv:2601.22718. Cited by: §5. X. Li, J. Zhang, M. Yi, Y. Li, L. Wang, Y. Wang, and J. Fan (2026) ETS: energy-guided test-time scaling for training-free rl alignment. In Forty-third International Conference on Machine Learning, Cited by: §1, §5. Y. Li, J. Liu, J. Xu, Y. Tong, Z. Li, Q. Liu, and B. Wang (2025) Trust region masking for long-horizon llm reinforcement learning. arXiv preprint arXiv:2512.23075. Cited by: §5. Y. Li, M. Yi, X. Li, J. Fan, F. Jiang, B. Chen, P. Li, J. Song, and T. Zhang (2026a) Reasoning and tool-use compete in agentic rl: from quantifying interference to disentangled tuning. Preprint Technical Report arXiv:arXiv:2602.00994. Cited by: §1. Z. Li, M. Yi, Y. Wang, S. Cui, and Y. Liu (2026b) Towards a theoretical understanding to the generalization of rlhf. Preprint Technical Report arXiv:2601.16403. Cited by: §1. T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu (2024) Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, Cited by: §5. S. Luo, X. Zhang, Y. Hu, B. Zhang, K. Wang, J. Su, M. Sun, L. Liang, and J. Zhang (2026) Sparse-rl: breaking the memory wall in llm reinforcement learning via stable sparse rollouts. Preprint Technical Report arXiv:2601.10079. Cited by: §5. W. Ma, H. Zhang, L. Zhao, Y. Song, Y. Wang, Z. Sui, and F. Luo (2025) Stabilizing moe reinforcement learning by aligning training and inference routers. Preprint Technical Report arXiv:2510.11370. Cited by: §1, §1, §2, §4.3, §5. MAA (2024) American invitational mathematics examination (AIME). Note: https://maa.org/maa-invitational-competitions/ Cited by: §1, §4.1. A. Mallen, A. Asai, V. Zhong, R. Das, H. Hajishirzi, and D. Khashabi (2022) When not to trust language models: investigating effectiveness and limitations of parametric and non-parametric memories. Preprint Technical Report arXiv:2212.10511. Cited by: §4.1. A. M. Metelli, M. Papini, N. Montali, and M. Restelli (2020) Importance sampling techniques for policy optimization. Journal of Machine Learning Research. Cited by: §1, §2. M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville (2025) Faster, more efficient RLHF through off-policy asynchronous learning. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §5. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems. Cited by: §4.1, footnote 1. O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis (2023) Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Cited by: §4.1. P. Qi, X. Zhou, Z. Liu, T. Pang, C. Du, M. Lin, and W. S. Lee (2026) Rethinking the trust region in llm reinforcement learning. In Forty-third International Conference on Machine Learning, Cited by: §E.1, §4.1, §5, Remark 1. C. P. Robert and G. Casella (2004) Monte carlo statistical methods. Springer. Cited by: §1. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International conference on machine learning, Cited by: §5, Remark 1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. Preprint Technical Report arXiv:1707.06347. Cited by: §1, §2, §2, §5. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. Preprint Technical Report arXiv:2402.03300. Cited by: §E.1, 1st item, §1, §3.4, §3.4, §4.1, §4.1, §5. H. Sun, M. Haider, R. Zhang, H. Yang, J. Qiu, M. Yin, M. Wang, P. Bartlett, and A. Zanette (2024) Fast best-of-n decoding via speculative rejection. In The Thirty-seventh Annual Conference on Neural Information Processing Systems, Cited by: §5. R. S. Sutton, A. G. Barto, et al. (1998) Reinforcement learning: an introduction. MIT press Cambridge. Cited by: §1, §2, §5. F. Tajwar, G. Zeng, Y. Zhou, Y. Song, D. Arora, Y. Jiang, J. Schneider, R. Salakhutdinov, H. Feng, and A. Zanette (2026) Maximum likelihood reinforcement learning. Preprint Technical Report arXiv:2602.02710. Cited by: §5. L. Team, A. Shen, B. Li, B. Hu, B. Jing, C. Chen, C. Huang, C. Zhang, C. Yang, C. Lin, et al. (2025) Every step evolves: scaling reinforcement learning for trillion-scale thinking model. Preprint Technical Report arXiv:arXiv:2510.18855. Cited by: §5. S. T. Tokdar and R. E. Kass (2010) Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics. Cited by: §1. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2022) MuSiQue: multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics. Cited by: §4.1. S. Tu, C. Xu, Q. Zhang, Y. Ma, Y. Zhang, L. Li, D. Li, X. Lan, and D. Zhao (2026) UCOB: learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation. arXiv:2606.29502. Cited by: §1. X. Wan, Y. Wang, W. Huang, and M. Sun (2026) Buffer matters: unleashing the power of off-policy reinforcement learning in large language model reasoning. In The Fourteenth International Conference on Learning Representations, Cited by: §5. L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei (2022) Text embeddings by weakly-supervised contrastive pre-training. Preprint Technical Report arXiv:2212.03533. Cited by: §C.1. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025a) Qwen3 technical report. Preprint Technical Report arXiv:2505.09388. Cited by: §1, §4.1. A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. (2024b) Qwen2. 5-math technical report: toward mathematical expert model via self-improvement. (arXiv:2409.12122). Cited by: §C.3. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing, Cited by: §4.1, §4.1. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. Preprint Technical Report arXiv:2503.14476. Cited by: §E.1, 1st item, §1, §2, §3.4, §4.1, §4.1, §5. Y. Zhang, M. Yi, C. Wang, X. Xiang, T. Jia, Z. Dan, C. Zong, and Y. Wang (2026a) HTAM: hierarchical transition-attended memory for operator optimization. arXiv preprint arXiv:2605.29734. Cited by: §1. Y. Zhang, M. Fan, J. Fan, M. Yi, Y. Luo, G. Li, B. Wu, and W. Zhou (2026b) Reward-sql: boosting text-to-sql via stepwise execution-aware reasoning and process-supervised rewards. Proceedings of the ACM on Management of Data. Cited by: §1. X. Zhao, Y. Liu, K. Xu, J. Guo, Z. Wang, Y. Sun, X. Kong, Q. Cao, L. Jiang, Z. Wen, Z. Zhang, and J. Zhou (2025) Small leak can sink a great ship–boost rl training on moe with icepop!. External Links: Link Cited by: §4.3, §5. Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, et al. (2023) Pytorch fsdp: experiences on scaling fully sharded data parallel. Preprint Technical Report arXiv:2304.11277. Cited by: §4.1. C. Zheng, K. Dang, B. Yu, M. Li, H. Jiang, J. Lin, Y. Liu, A. Yang, J. Zhou, and J. Lin (2025a) Stabilizing reinforcement learning with llms: formulation and practices. Preprint Technical Report arXiv:2512.01374. Cited by: §2, §4.3, §5. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025b) Group sequence policy optimization. Preprint Technical Report arXiv:2507.18071. Cited by: §E.1, 1st item, §1, §2, §3.4, §4.1. Appendix A Proofs A.1 Proof of On-policy Certificate See 1 Proof. For any v∈v , applying Bayes’ rule gives ℙ(yt=v∣zt=1)=ℙ(zt=1∣yt=v)πθold(v∣x,y<t)ℙ(zt=1).P(y_t=v z_t=1)= P(z_t=1 y_t=v)\, _ _old(v x,y_<t)P(z_t=1). (16) The acceptance probability conditioned on yt=vy_t=v is ℙ(zt=1∣yt=v)=wt(v)Mt=πθ(v∣x,y<t)Mtπθold(v∣x,y<t).P(z_t=1 y_t=v)= w_t(v)M_t= _θ(v x,y_<t)M_t\, _ _old(v x,y_<t). (17) The marginal acceptance probability is ℙ(zt=1)=∑v′∈πθ(v′∣x,y<t)Mtπθold(v′∣x,y<t)⋅πθold(v′∣x,y<t)=1Mt∑v′∈πθ(v′∣x,y<t)=1Mt.P(z_t=1)= _v _θ(v x,y_<t)M_t\, _ _old(v x,y_<t)· _ _old(v x,y_<t)= 1M_t _v _θ(v x,y_<t)= 1M_t. (18) Substituting back, we have ℙ(yt=v∣zt=1)=πθ(v∣x,y<t)Mtπθold(v∣x,y<t)⋅πθold(v∣x,y<t)1/Mt=πθ(v∣x,y<t).P(y_t=v z_t=1)= _θ(v x,y_<t)M_t\, _ _old(v x,y_<t)· _ _old(v x,y_<t)1\,/\,M_t= _θ(v x,y_<t). (19) ∎ A.2 Proof of Approximation Error See 1 Proof. For a single response, the sequence-level gradient is seq=A(x,y)∏j=1Twj(θ)∑t=1Tst, g_seq=A(x,y) _j=1^Tw_j(θ) _t=1^Ts_t, (20) while the token-level surrogate uses tok=A(x,y)∑t=1Twt(θ)st, g_tok=A(x,y) _t=1^Tw_t(θ)\,s_t, (21) where st=∇θlogπθ(yt∣x,y<t)s_t= _θ _θ(y_t x,y_<t). Their discrepancy is E=‖A(x,y)∑t=1T(∏j=1Twj(θ)−wt(θ))st‖≤|A(x,y)|∑t=1T|∏j=1Twj(θ)−wt(θ)|‖st‖.E= \|A(x,y) _t=1^T ( _j=1^Tw_j(θ)-w_t(θ) )s_t \|≤|A(x,y)| _t=1^T | _j=1^Tw_j(θ)-w_t(θ) |\,\|s_t\|. (22) Define R=∏j=1Twj(θ)R= _j=1^Tw_j(θ) and Δj=logwj(θ) _j= w_j(θ), so that |R−wt(θ)|=wt(θ)|∏j≠twj(θ)−1||R-w_t(θ)|=w_t(θ)\, | _j≠ tw_j(θ)-1 |. Applying the elementary inequality |ex−1|≤e|x|−1|e^x-1|≤ e^|x|-1 to the product term gives |∏j≠twj(θ)−1|=|e∑j≠tΔj−1|≤e|∑j≠tΔj|−1≤e∑j≠t|Δj|−1. | _j≠ tw_j(θ)-1 |= |e _j≠ t _j-1 |≤ e | _j≠ t _j |-1≤ e _j≠ t| _j|-1. (23) Hence, |R−wt(θ)|≤e|Δt|(e∑j≠t|Δj|−1)=e∑j=1T|Δj|−e|Δt|≤eD−1,|R-w_t(θ)|≤ e^| _t| (e _j≠ t| _j|-1 )=e _j=1^T| _j|-e^| _t|≤ e^D-1, (24) where D=∑t=1T|logwt(θ)|D= _t=1^T|\! w_t(θ)|. Plugging this bound into (22) yields E≤|A(x,y)|(eD−1)∑t=1T‖st‖.E≤|A(x,y)|\,(e^D-1) _t=1^T\|s_t\|. (25) ∎ See 3 Proof. For accepted tokens (zt=1z_t=1), w~t(θ)=1 w_t(θ)=1 in the forward pass, so logw~t(θ)=0 w_t(θ)=0. Thus the total log-importance deviation becomes DSIS=∑t=1T|logw~t(θ)|=∑t=1T(1−zt)|logwt(θ)|.D_SIS= _t=1^T|\! w_t(θ)|= _t=1^T(1-z_t)\,|\! w_t(θ)|. (26) Defining C=∑t=1Tzt|logwt(θ)|≥0C= _t=1^Tz_t\,|\! w_t(θ)|≥ 0, we have DSIS=D−C≤D_SIS=D-C≤ D. Since f(u)=eu−1f(u)=e^u-1 is strictly increasing, eDSIS−1=eD−C−1≤eD−1,e^D_SIS-1=e^D-C-1≤ e^D-1, (27) with strict inequality if C>0C>0. Applying Theorem 1 to the SIS gradient estimate gives the desired bound on ESISE_SIS. ∎ A.3 Proof of top-K Approximation See 2 Proof. Let p(v)=πθ(v∣x,y<t)p(v)= _θ(v x,y_<t) denote the target distribution and q(v)=πθold(v∣x,y<t)q(v)= _ _old(v x,y_<t) the proposal. Define K=TopK(q,K)V_K=TopK(q,K) and ξK=∑v∉Kp(v) _K= _v _Kp(v). Within KV_K, the approximate envelope constant is M^=maxv∈Kp(v)q(v) M= _v _K p(v)q(v). The acceptance probability for v∈Kv _K is ℙ(A=1∣v)=p(v)M^q(v)P(A=1 v)= p(v) M\,q(v), and for v∉Kv _K, ℙ(A=1∣v)=0P(A=1 v)=0. The conditional distribution of accepted tokens is π^θ(v)=ℙ(v∣A=1)=ℙ(A=1∣v)q(v)∑v′∈Kℙ(A=1∣v′)q(v′)=p(v)/M^∑v′∈Kp(v′)/M^=p(v)∑v′∈Kp(v′), π_θ(v)=P(v A=1)= P(A=1 v)\,q(v) _v _KP(A=1 v )\,q(v )= p(v)\,/\, M _v _Kp(v )\,/\, M= p(v) _v _Kp(v ), (28) for v∈Kv _K, and π^θ(v)=0 π_θ(v)=0 for v∉Kv _K, so that proves our first conclusion. Next, we compute the total variation distance DTV(π^θ∥p) D_ TV( π_θ p) =12∑v∈|π^θ(v)−p(v)| = 12 _v | π_θ(v)-p(v)| =12∑v∈K(p(v)1−ξK−p(v))+12∑v∉Kp(v) = 12 _v _K ( p(v)1- _K-p(v) )+ 12 _v _Kp(v) =12∑v∈Kp(v)⋅ξK1−ξK+12ξK = 12 _v _Kp(v)· _K1- _K+ 12 _K =12(1−ξK)⋅ξK1−ξK+12ξK=ξK. = 12(1- _K)· _K1- _K+ 12 _K= _K. (29) ∎ Appendix B Algorithm We summarize the complete SIS procedure with the top-K envelope approximation in Algorithm 1. For each token position, the algorithm first identifies the top-K most probable tokens under the behavior policy and restricts the rejection envelope to this set. Tokens falling inside KV_K undergo an acceptance test: accepted tokens are certified as on-policy and receive unit importance weight, while rejected tokens and those outside KV_K retain their original importance ratio. The resulting modified ratios are then directly substituted into any standard policy-gradient objective. Algorithm 1 SIS with Top-K Envelope Approximation 0: Prompt x; responses yii=1G∼πθold(⋅∣x)\y_i\_i=1^G _ _old(· x); top-K parameter K 0: Modified importance ratios w~i,t(θ)\ w_i,t(θ)\ 1: for each response yiy_i, i=1,…,Gi=1,…,G do 2: for each token position t=1,…,|yi|t=1,…,|y_i| do 3: Compute token-level importance ratio wi,t(θ)w_i,t(θ) 4: Select top-K set KV_K under πθold(⋅∣x,yi,<t) _ _old(· x,y_i,<t) 5: Compute approximation constant M^t M_t restricted to KV_K ⊳ Eq. (7) 6: if yi,t∈Ky_i,t _K then 7: Accept token with probability wi,t(θ)/M^tw_i,t(θ)/ M_t; set zt←1z_t← 1 if accepted ⊳ Eq. (8) 8: else 9: zt←0z_t← 0 ⊳ Tokens outside KV_K remain off-policy 10: end if 11: Set w~i,t(θ)←1 w_i,t(θ)← 1 if zt=1z_t=1, else retain wi,t(θ)w_i,t(θ) ⊳ Eq. (10) 12: end for 13: end for 14: Plug w~i,t(θ)\ w_i,t(θ)\ into the policy-gradient objective ⊳ e.g., GRPO, DAPO, GSPO Appendix C Experimental Setup Details C.1 Math and Agent Experiments Table 5 (math) and Table 5 (agent) summarize the hyperparameters. For each baseline, the vanilla run and the SIS run share the same training and evaluation hyperparameters; SIS only adds the top-K approximation parameter used to compute the modified importance ratio. For agent training, we follow the settings of Search-R1 (Jin et al., 2025), using an E5 retriever (Wang et al., 2022) and the 2018 Wikipedia dump (Karpukhin et al., 2020) as the corpus. Table 4: Hyperparameters used in math experiments. Hyperparameter GRPO DAPO GSPO Learning rate 1×10−61× 10^-6 Mini-batch size (mbs) 256 Responses per prompt 8 Max prompt length 1,024 Max response length 3,072 Gradient steps 1,600 Train temperature 1.0 Eval temperature / top-p 1.0 / 0.7 Entropy loss coefficient 0 Training engine FSDP Inference engine vLLM KL loss coefficient 1×10−31× 10^-3 0 0 SIS top-K (dense / MoE) 10 / 10 30 / 30 50 / 30 εlow _ low 0.2 0.2 3×10−43× 10^-4 εhigh _ high 0.2 0.28 4×10−44× 10^-4 Global batch size (gbs) 256×N256× N GPUs 32 (colocated) Table 5: Hyperparameters used in agent experiments. Hyperparameter GRPO DAPO GSPO Learning rate 1×10−61× 10^-6 Mini-batch size (mbs) 64 Responses per prompt 8 Max prompt length 4,096 Max response length 4,096 Gradient steps 800 Train temperature 1.0 Eval temperature / top-p 1.0 / 1.0 Entropy loss coefficient 0 Training engine FSDP Inference engine vLLM KL loss coefficient 1×10−31× 10^-3 0 0 SIS top-K (dense) 50 50 50 εlow _ low 0.2 0.2 3×10−43× 10^-4 εhigh _ high 0.2 0.28 4×10−44× 10^-4 Global batch size (gbs) 256 GPUs 16 (colocated) For all training-dynamics figures we plot only the first 400400 gradient steps: beyond that the policy has largely converged and πθ≈πθold _θ\!≈\! _ _ old, so the curves flatten and the off-policy gap of interest is no longer visible. C.2 Reward Design All experiments use a purely outcome-based, rule-based reward; we apply no learned reward model, no format or length shaping, and no intermediate-step rewards. For a prompt x and a generated response y, let a^(y) a(y) be the answer extracted from y and a⋆a (or ⋆A ) denote the gold answer set. Math. We extract the final answer from the field of y and compare it against the reference with a symbolic equivalence checker Eq(⋅,⋅)Eq(·,·) that handles numeric and algebraic normalization. The reward is binary, rmath(x,y)=1,a^(y)is parseable from andEq(a^(y),a⋆)=True,0,otherwise.r_ math(x,y)\;=\; cases1,& a(y)\ is parseable from boxed\\\ and\ Eq\! ( a(y),\,a )=True,\\ 0,&otherwise. cases (30) Responses that omit a parseable answer receive reward 0. Agent. Following Search-R1 (Jin et al., 2025), the reward is the exact-match (EM) score of the final answer: ragent(x,y)=EM(apred,agold),r_ agent(x,y)\;=\;EM(a_ pred,\,a_ gold), (31) where apreda_ pred is the final answer extracted from the model response y (enclosed in the <answer> ⋅· </answer> tags), and agolda_ gold denotes the ground-truth answer. Search-engine invocations and retrieved passages are not rewarded directly; they affect the reward only through the final answer. C.3 Prompt Template For math reasoning, we adopt the same prompt format as Qwen-Math (Yang et al., 2024b). For agent reasoning, we follow Search-R1 (Jin et al., 2025) and use a template that enforces a minimal yet sufficient structure without introducing content-specific biases. The template organizes the model output into three iterative stages: (i) a reasoning phase, (i) a search engine invocation phase, and (i) a final answer. We illustrate the templates in Table 6. Table 6: Prompt templates for math and agent tasks. The placeholder question is replaced by the actual query during both training and inference. Task Prompt Template Math Please reason step by step, and put your final answer within . question. Agent Answer the given question. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <search> query </search>, and it will return the top searched results between <information> and </information>. You can search as many times as you want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer> without detailed illustrations. For example, <answer> x </answer>. Question: question. Appendix D Additional Experimental Results D.1 SIS Enables Stable Training Without Clipping To test whether SIS alone suffices for training stability, we disable clipping entirely in GRPO+SIS on Qwen3-8B-Base. Table 7 shows that SIS without clipping still outperforms the clip-based GRPO on both math and agent tasks, confirming that SIS is a stabilization mechanism in its own right, and one that does not sacrifice gradient information. Table 7: SIS does not require clipping on Qwen3-8B-Base. Best is bolded. Math Method MATH500 AMC23 AIME24 AIME25 Avg GRPO 80.4 74.14 23.44 18.23 49.05 + SIS 85.6 79.53 25.83 19.38 52.59 + SIS (w/o Clip) 82.2 76.72 28.33 20.00 51.81 Agent Method NQ TriviaQA PopQA HotpotQA Musique Bamb GRPO 49.14 66.99 44.68 46.24 19.24 48.8 + SIS 49.78 67.47 46.64 47.90 19.74 52.8 + SIS (w/o Clip) 49.58 67.90 46.88 46.56 20.94 52.0 D.2 Accept Rate Beyond Dense Math To show that the on-policy certificate of SIS remains effective beyond the dense-math regime, we further inspect its token-level accept rate on the larger MoE backbone (Qwen3-30B-A3B-Base) and on the agentic search setting (Qwen3-14B-Base). As shown in Figure 6, SIS maintains an appreciable accept rate across all three policy-gradient algorithms in both settings. The accept rate is naturally lower on the agentic task, since interleaving reasoning with diverse tool calls induces a larger distributional shift between πθold _ _old and πθ _θ; nevertheless it stays well above 0.60.6, indicating that a substantial fraction of tokens remain effectively on-policy under πθ _θ. Figure 6: Accept rate of SIS on Qwen3-30B-A3B-Base under math reasoning (left) and Qwen3-14B-Base under agentic search (right). SIS sustains an appreciable accept rate across all algorithms in both settings. The lower rate on the agentic task reflects the larger distributional shift introduced by interleaved tool calls between πθold _ _old and πθ _θ. D.3 Negligible Time Cost of SIS We profile the wall-clock cost introduced by SIS on top of vanilla GRPO. The setup follows our main math experiments: Qwen3-8B-Base, batch size 512512 with n=8n=8 rollouts per prompt, response length 4,0964,096, trained on 8×8×NVIDIA A800-80GB GPUs. We instrument the five SIS-specific operations: the top-K extraction during the old_log_prob stage, and the per-microbatch off-policy-to-on-policy correction during update_actor. We sweep K∈10,50,100K∈\10,50,100\ to assess how the overhead scales. Figure 7 decomposes the per-step wall-clock into rollout, πθold _ _old log-prob, actor update, and the additional SIS time. The total SIS overhead is ∼5.2 5.2 s per step, which corresponds to roughly 1%1\% of the 465465–499499 s step time across all three values of K. Furthermore, the overhead is essentially K-invariant: because the dominant cost is the off-policy-to-on-policy ratio reconstruction, which is fully parallelized on GPU and scales with O(B⋅L)O(B· L) rather than K, varying K from 1010 to 100100 leaves the wall-clock virtually unchanged. The only price paid by a larger K is GPU memory for the cached top-K logits, not compute time. Figure 7: Per-step wall-clock decomposition of GRPO+SIS on Qwen3-8B-Base across K∈10,50,100K∈\10,50,100\. The red segment is the SIS overhead: near-constant ∼5.2 5.2 s (∼1% 1\% of step time) and K-invariant. D.4 Training Dynamics Beyond final accuracy and accept rate, we further visualize three training-side dynamics for all SIS runs in Table 1: token-level training reward, policy entropy, and gradient norm, summarized in Figure 8. Training reward. The first column of Figure 8 shows that the three policy-gradient algorithms paired with SIS all exhibit smooth, monotonically increasing reward curves on Qwen3-8B (math), Qwen3-30B-A3B-Base (math), Qwen3-8B (agentic search) and Qwen3-14B (agentic search), and converge to comparable levels within each setting. This indicates that SIS optimizes the training objective stably regardless of the underlying policy-gradient algorithm or the model scale. Policy entropy. The middle column plots the policy entropy over the same horizon. Across all four settings, SIS keeps the entropy bounded away from zero, avoiding the entropy collapse commonly observed for aggressive off-policy training. On math tasks the entropy first drops as the policy sharpens on the training distribution, then stabilizes; GSPO+SIS in particular exhibits a mild early rebound before settling into a healthy range. On the agentic search tasks the entropy decreases more gradually due to the longer trajectories induced by tool calls, but again remains well above zero throughout training. Gradient norm. The right column shows the gradient norm on a log scale. The gradient norm stays bounded within roughly 10−210^-2–10110^1 throughout training and does not diverge in any of the four settings. Occasional spikes appear on the agentic search runs, mostly on GSPO, reflecting the higher variance induced by variable-length tool-augmented rollouts; however these spikes remain within one order of magnitude of the median and do not lead to sustained instability, consistent with the smooth reward curves in the first column. Figure 8: Training dynamics of SIS over the first 400400 gradient steps for all entries in Table 1. Rows: Qwen3-8B/Qwen3-30B-A3B-Base on math and Qwen3-8B/Qwen3-14B on agentic search. Columns: training reward, policy entropy, and gradient norm (log scale). SIS produces stable reward growth and non-collapsed entropy across all backbones and algorithms; occasional gradient-norm spikes on agentic search runs are transient and do not lead to sustained instability. D.5 Robustness Under Stale Rollouts We vary N, the number of mini-batch updates per rollout, to increase the drift from πθold _ _old to πθ _θ within each rollout cycle. Table 8 reports all three settings N=4N=4, 88, and 1616 grouped by algorithm, so the effect of staleness is directly comparable within each method. Two patterns emerge. First, staleness is harmful in itself: as N grows from 44 to 1616, average accuracy drops for every algorithm regardless of whether SIS is applied, confirming that reusing stale off-policy rollouts degrades RL. Second, SIS yields a stable gain at every staleness level: it improves over the corresponding baseline at all N for GRPO, DAPO, and GSPO. Accept rate under staleness. To further explain why SIS remains effective as N grows, we plot the token-level accept rate during training in Figure 9. Across all three algorithms and all staleness levels, the accept rate stays consistently in the 0.800.80–0.950.95 range and never collapses. This confirms that even with N=16N=16 a substantial fraction of tokens generated by πθold _ _old are still on-policy under πθ _θ, so the on-policy certificate of SIS continues to provide a meaningful correction signal under high staleness. Figure 9: Token-level accept rate of SIS during the first 400400 gradient steps on Qwen3-8B-Base under staleness N∈4,8,16N∈\4,8,16\. Across all three algorithms, the accept rate stays in 0.800.80–0.950.95 throughout training, indicating that most tokens remain effectively on-policy even at N=16N=16. Table 8: Performance of SIS under varying policy staleness on Qwen3-8B-Base, where staleness N is the number of mini-batch updates per rollout. Results are grouped by algorithm so that the effect of increasing N is directly comparable within each method. Shaded rows denote the application of SIS on top of each baseline. Within each algorithm–staleness pair, the higher accuracy per metric is bolded. N Method MATH500 AMC23 AIME24 AIME25 Avg GRPO Base 80.4 74.14 23.44 18.23 49.05 !12 4 w/ SIS 85.6 79.53 25.83 19.38 52.59 Base 81.6 71.33 22.91 20.73 49.14 !12 8 w/ SIS 83.8 78.59 24.08 20.10 51.64 Base 83.0 69.53 23.54 18.23 48.58 !12 16 w/ SIS 84.6 77.03 24.69 21.35 51.92 DAPO Base 83.2 81.88 21.56 18.33 51.24 !12 4 w/ SIS 84.2 86.79 26.98 22.60 55.14 Base 83.6 78.59 24.79 19.17 51.54 !12 8 w/ SIS 84.2 83.28 24.69 21.46 53.41 Base 83.4 73.83 20.42 17.60 48.81 !12 16 w/ SIS 85.0 81.17 24.48 21.04 52.92 GSPO Base 85.6 78.83 26.46 18.85 52.44 !12 4 w/ SIS 85.2 77.19 28.33 23.44 53.54 Base 83.4 73.75 25.31 18.02 50.12 !12 8 w/ SIS 83.4 78.20 25.52 21.77 52.22 Base 82.0 71.72 22.92 19.48 49.03 !12 16 w/ SIS 83.8 73.83 23.85 19.27 50.19 D.6 Generalization Beyond Qwen To verify that our findings generalize beyond a single model family, we further evaluate SIS on Llama-3.2-3B-Instruct (Grattafiori et al., 2024). We use the same DAPO-Math-17K training corpus and keep all hyperparameters identical to the Qwen3-8B-Base math setting (Table 5), ensuring a controlled comparison. As presented in Table 9, SIS again delivers consistent plug-in gains across all three algorithms, with the largest absolute lift on DAPO (+4.47+4.47 Avg). In relative terms the improvements are even more pronounced (+11.0%+11.0\% for GRPO and +14.4%+14.4\% for DAPO), notably larger than the corresponding relative gains on Qwen3-8B-Base. We attribute this to the smaller Llama model exhibiting a wider policy–rollout gap, making the off-policy correction provided by SIS even more impactful. These results confirm that SIS delivers robust gains independent of the underlying model architecture and family. Table 9: Performance of SIS as a plug-in on Llama-3.2-3B-Instruct across math benchmarks. Shaded rows denote the application of SIS on top of each baseline. The best accuracy per column is bolded. Δ denotes the absolute gain over the corresponding baseline. Method MATH500 AMC23 AIME24 AIME25 Avg Δ GRPO 50.0 44.77 12.40 0.10 26.82 – !12 w/ SIS 51.2 53.36 13.75 1.77 29.76 +2.94 DAPO 57.2 52.74 13.96 0.21 31.03 – !12 w/ SIS 58.0 65.47 17.50 1.04 35.50 +4.47 GSPO 50.2 43.36 11.15 0.52 26.31 – !12 w/ SIS 50.6 43.36 13.94 0.63 27.13 +0.82 Appendix E Stabilization Tricks Through the Importance-Sampling Lens We organize Table 2 by writing every stabilization trick as a token-level rescaling of the importance-sampling (IS) coefficient wi,t(θ)=πθ(yi,t∣x,yi,<t)πθold(yi,t∣x,yi,<t).w_i,t(θ)\;=\; _θ(y_i,t x,y_i,<t) _ _ old(y_i,t x,y_i,<t). (32) Under this lens, each method amounts to a specific surrogate g(wi,t(θ))g(w_i,t(θ)) used in place of wi,t(θ)w_i,t(θ) inside the policy-gradient objective. Clipping-based methods (GRPO, DAPO, GSPO, CISPO) reshape w directly; DPPO-TV and Clip-Cov instead constrain the update through an auxiliary signal, which we recast as multiplying w by an indicator that masks out-of-region tokens. We first restate the six representative methods in this unified form, and then use DAPO and GSPO as concrete cases to illustrate how SIS is instantiated on top of an existing trick. E.1 Representative stabilization methods GRPO (Shao et al., 2024). The vanilla token-level PPO-style ratio with a symmetric clip bound ε , gGRPO(w)=clip(w, 1−ε, 1+ε).g_ GRPO(w)\;=\;clip(w,\,1- ,\,1+ ). (33) DAPO (Yu et al., 2025). Decoupled clip bounds εlow,εhigh _ low, _ high with the KL regularization removed, DAPO(θ)= _ DAPO(θ)= (x,a)∼,yi∼πθold _(x,a) ,\y_i\ _ _ old (34) [1∑i|yi|∑i,tmin(wi,t(θ)A^i,t,clip(wi,t(θ),1−εlow,1+εhigh)A^i,t)], [ 1 _i|y_i|\! _i,t \! (w_i,t(θ) A_i,t,\;clip(w_i,t(θ),1- _ low,1+ _ high) A_i,t ) ], restricted to prompts with 0<|yi∣is_equivalent(a,yi)|<G0<|\y_i is\_equivalent(a,y_i)\|<G. GSPO (Zheng et al., 2025b). A sequence-level ratio obtained by averaging log-ratios, si(θ)=exp(1|yi|∑t=1|yi|logπθ(yi,t∣x,yi,<t)πθold(yi,t∣x,yi,<t)),s_i(θ)\;=\; \! ( 1|y_i| _t=1^|y_i| _θ(y_i,t x,y_i,<t) _ _old(y_i,t x,y_i,<t) ), (35) which then enters a sequence-level clipped objective with bounds εlow,εhigh _ low, _ high. CISPO (Chen et al., 2025). A stop-gradient one-sided clip that detaches the high tail, gCISPO(w)=sg[min(w, 1+εhi)].g_ CISPO(w)\;=\;sg\! [ (w,\,1+ _ hi) ]. (36) DPPO-TV (Qi et al., 2026). Instead of bounding w, it bounds the per-token total-variation distance between the current and behavior token distributions, TVi,t(θ)=12∥πθ(⋅∣x,yi,<t)−πθold(⋅∣x,yi,<t)∥1,TV_i,t(θ)\;=\; 12 \| _θ(· x,y_i,<t)- _ _ old(· x,y_i,<t) \|_1, (37) and discards the gradient of any token whose policy drift exceeds a threshold δ, which is equivalent to the effective coefficient w⋅[TVi,t≤δ]w· 1[TV_i,t≤δ]. Clip-Cov (Cui et al., 2025). It attributes entropy collapse to a small set of tokens with large log-probability–advantage covariance, Covi,t(θ)=(logπθ(yi,t∣x,yi,<t)−logπ¯)(A^i,t−A¯),Cov_i,t(θ)\;=\; ( _θ(y_i,t x,y_i,<t)- π ) ( A_i,t- A ), (38) where logπ¯ π and A¯ A are batch means. It detaches the gradient of tokens whose covariance exceeds a threshold ω, giving the effective coefficient w⋅[Covi,t≤ω]w· 1[Cov_i,t≤ω]. E.2 Instantiating SIS: DAPO and GSPO as cases Since SIS only replaces the token-level ratio wi,t(θ)w_i,t(θ) with its on-policy-equivalent counterpart w~i,t(θ) w_i,t(θ) (cf. (10)), instantiating SIS on top of any of the methods above amounts to a one-line substitution wi,t→w~i,tw_i,t\!→\! w_i,t in the corresponding objective. We illustrate this with DAPO and GSPO, the two cases used in our experiments; analogous substitutions apply to the remaining methods and are left to future empirical study. Case 1: DAPO + SIS. Substituting wi,t→w~i,tw_i,t\!→\! w_i,t into the DAPO objective yields DAPOSIS(θ)= _ DAPO SIS(θ)= (x,a)∼,yii=1G∼πθold(⋅∣x) _(x,a) ,\y_i\_i=1^G _ _ old(· x) (39) [1∑i=1G|yi|∑i=1G∑t=1|yi|min(w~i,t(θ)A^i,t,clip(w~i,t(θ),1−εlow,1+εhigh)A^i,t)], [ 1 _i=1^G|y_i| _i=1^G _t=1^|y_i| \! ( w_i,t(θ) A_i,t,\;clip( w_i,t(θ),1- _ low,1+ _ high) A_i,t ) ], s.t. 0<|yi∣is_equivalent(a,yi)|<G, 0< |\y_i is\_equivalent(a,y_i)\ |<G, where a is the ground-truth answer to the prompt x. Case 2: GSPO + SIS. For sequence-level GSPO, accepted tokens contribute a zero log-ratio, giving the SIS-modified sequence ratio si~(θ)=exp(1|yi|(∑(i,t)∈logπθ(yi,t∣x,yi,<t)sg[πθ(yi,t∣x,yi,<t)]+∑(i,t)∉logπθ(yi,t∣x,yi,<t)πθold(yi,t∣x,yi,<t))), s_i(θ)= \! ( 1|y_i| ( _(i,t) _θ(y_i,t x,y_i,<t)sg[ _θ(y_i,t x,y_i,<t)]+ _(i,t) _θ(y_i,t x,y_i,<t) _ _old(y_i,t x,y_i,<t) ) ), (40) which then enters the GSPO clipped objective GSPOSIS(θ)=x∼,yii=1G∼πθold(⋅∣x)[1G∑i=1Gmin(si~(θ)A^i,clip(si~(θ),1−εlow,1+εhigh)A^i)].J_ GSPO SIS(θ)=E_x ,\y_i\_i=1^G _ _ old(· x) [ 1G _i=1^G \! ( s_i(θ) A_i,\;clip( s_i(θ),1- _ low,1+ _ high) A_i ) ]. (41)