Paper deep dive
Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It
Tianrun Yu, Kaixiang Zhao, Shangzhe Li, Yuxiao Yang, Porter Jenkins, Weitong Zhang, Taylor W. Killian
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study training-inference mismatch in reinforcement learning with verifiable rewards (RLVR) for large language models, where rollouts are sampled by an inference engine while gradients are computed by a training engine, and the two engines assign different probabilities to the same tokens. To account for this discrepancy in policy updates, we introduce calibrated importance sampling (CIS). CIS is motivated by an empirically supported logit-displacement characterization that expresses the mismatch as an additive displacement $\varepsilon_t$ in log-odds, determined by the per-logit perturbation before the softmax, whose distribution is approximately invariant to token confidence. This characterization motivates a confidence-aware truncation: large positive displacements are truncated at a single constant threshold, which maps back to an importance-ratio cap that tightens as token confidence increases. Theoretically, we show that CIS replaces the unbounded second moment that governs the error of exact importance sampling with a term bounded by a constant, at the cost of a bias controlled by the truncated excess. In evaluation across three mixture-of-experts models and five mathematical reasoning benchmarks, CIS achieves the highest five-benchmark average on all three models among the evaluated baselines. Diagnostic analyses show that CIS places less truncation bias on low-confidence tokens than truncated importance sampling, while upward clipping of small importance weights reduces held-out accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2609.32444v1
- Canonical: https://arxiv.org/abs/2609.32444v1
Trouble viewing inline? Open PDF directly →
Full Text
120,207 characters extracted from source content.
Expand or collapse full text
Rethinking Training–Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It Tianrun Yu Kaixiang Zhao Shangzhe Li Affiliation: Brigham Young University, University of North Carolina at Chapel Hill*Corresponding author. tianruny, kzhao2@byu.edu, tkillian@cs.byu.edu Yuxiao Yang Affiliation: Brigham Young University, University of North Carolina at Chapel Hill*Corresponding author. tianruny, kzhao2@byu.edu, tkillian@cs.byu.edu Porter Jenkins Weitong Zhang Affiliation: Brigham Young University, University of North Carolina at Chapel Hill*Corresponding author. tianruny, kzhao2@byu.edu, tkillian@cs.byu.edu Taylor W. Killian Abstract We study training–inference mismatch in reinforcement learning with verifiable rewards (RLVR) for large language models, where rollouts are sampled by an inference engine while gradients are computed by a training engine, and the two engines assign different probabilities to the same tokens. To account for this discrepancy in policy updates, we introduce calibrated importance sampling (CIS). CIS is motivated by an empirically supported logit-displacement characterization that expresses the mismatch as an additive displacement εt _t in log-odds, determined by the per-logit perturbation before the softmax, whose distribution is approximately invariant to token confidence. This characterization motivates a confidence-aware truncation: large positive displacements are truncated at a single constant threshold, which maps back to an importance-ratio cap that tightens as token confidence increases. Theoretically, we show that CIS replaces the unbounded second moment that governs the error of exact importance sampling with a term bounded by a constant, at the cost of a bias controlled by the truncated excess. In evaluation across three mixture-of-experts models and five mathematical reasoning benchmarks, CIS achieves the highest five-benchmark average on all three models among the evaluated baselines. Diagnostic analyses show that CIS places less truncation bias on low-confidence tokens than truncated importance sampling, while upward clipping of small importance weights reduces held-out accuracy. †footnotetext: Code is available at https://github.com/kzhao5/CIS-RL. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving the reasoning abilities of large language models (LLMs) (Shao et al., 2024; DeepSeek-AI, 2025; Yu et al., 2025). To improve throughput, current RL post-training frameworks decouple rollout generation from gradient computation and assign them to an inference engine and a training engine, respectively (Sheng et al., 2025; Hu et al., 2025; Fu et al., 2025). Although the two engines load the same parameters, differences in their numerical implementations lead them to assign different probabilities to the same token. As a result, training that is nominally on-policy becomes off-policy (Yao et al., 2025; Qi et al., 2025; Zheng et al., 2025a). This training–inference mismatch is particularly severe for mixture-of-experts models, where it can cause training instability or even collapse (Ling Team and Inclusion AI, 2025; Zheng et al., 2025b; Ma et al., 2025; Zheng et al., 2025a). The standard remedy for this mismatch is importance sampling, which reweights the gradient of each token by the ratio between the probabilities that the training and inference engines assign to the sampled token (Yao et al., 2025; Liu et al., 2025; Zheng et al., 2025a). However, occasional sharp disagreements between the two engines on individual tokens can make this importance ratio extremely large (Zhao et al., 2025; Ma et al., 2025). Exact correction can therefore yield gradient estimates with uncontrolled variance (Ionides, 2008; Metelli et al., 2018; Liu et al., 2025). Truncating the importance sampling factor controls this variance by introducing the bias–variance trade-off. Understanding this bias–variance trade-off and designing better truncation methods has emerged as a new question to answer (Ionides, 2008; Liu et al., 2025). Figure 1: The inference and training engines assign different probabilities to the same tokens. CIS truncates the importance ratios with confidence-dependent caps, which are tighter for more confident tokens. The truncated ratios reweight the token-level GRPO loss (Shao et al., 2024). Existing methods largely rely on heuristic rules to truncate or mask this ratio. At the token level, TIS caps the ratio at a fixed upper bound (Yao et al., 2025; Qi et al., 2025), IcePop masks tokens whose ratio falls outside a fixed interval (Zhao et al., 2025; Ling Team and Inclusion AI, 2025), and KPop masks tokens by thresholding the KL divergence between the probabilities of the inference and training engines (Guo et al., 2026; Ling Team, 2026). At the sequence level, existing methods truncate or reject the ratio of the entire response (Liu et al., 2025), or replace the token-level objective with a sequence-level one (Zheng et al., 2025b). Other work addresses the problem through numerical precision and runs the entire pipeline in FP16 (Qi et al., 2025). Although these methods are effective to varying degrees, all of their rules act directly on the importance ratio or on quantities computed directly from the probabilities of the two engines. This ratio, however, mixes two factors: the actual discrepancy between the two engines and the confidence of the token itself. A discrepancy of the same size moves the ratio far from one on low-confidence tokens but barely changes it on high-confidence tokens. As a result, fixed thresholds act mainly on low-confidence tokens (Guo et al., 2026; Ling Team, 2026), and the truncation bias concentrates on these tokens. This motivates our question: Can we design a principled mismatch correction grounded in the statistical structure of the training–inference mismatch itself? We address this question by proposing CIS (Calibrated Importance Sampling), which is illustrated in Figure 1. CIS first separates the two factors in the importance ratio, which can be written exactly as a function of the token confidence and a component that reflects only the discrepancy between the two engines. Our measurements show that the distribution of this discrepancy component varies only slightly with token confidence, so anomalous discrepancies can be identified with the same threshold at every confidence level. CIS applies a single constant threshold to this component, which truncates anomalously large discrepancies at their source. When mapped back to the ratio, this threshold imposes a cap on the importance weight of each token, which depends on the token confidence and becomes tighter as confidence increases. This design determines where the truncation bias falls. Discrepancies of the same size are treated in the same way at every confidence level, so the truncation bias no longer concentrates on low-confidence tokens. Our contributions are as follows: • We decompose the importance ratio into token confidence and a discrepancy component, and find in large-scale analyses that the distribution of the discrepancy component barely varies with token confidence. With mixture-of-experts models, routing disagreement between engines substantially amplifies the heavy tail of this distribution. • Building on this characterization, we propose CIS, which truncates the discrepancy component rather than the ratio, yielding a ratio cap that tightens with token confidence. Theoretically, we prove that CIS bounds the potentially explosive second-moment term of exact correction, at the cost of a bias controlled by the truncated excess. • Experiments on three mixture-of-experts models and five mathematical reasoning benchmarks validate the effectiveness of CIS. Diagnostic analyses show that, unlike a fixed ratio cap, CIS does not concentrate its truncation bias on low-confidence tokens and attains a lower overall bias. 2 Preliminaries 2.1 Training–inference mismatch In current RL post-training frameworks, rollouts are generated by an inference engine (e.g., vLLM and SGLang), while gradients are computed by a separate training engine (e.g., FSDP and Megatron). Both engines load the same parameters θ, but differences in kernel implementations, reduction orders, and numerical precision lead them to assign different probabilities to the same token. We write ht:=(x,y<t)h_t:=(x,y_<t) for the prefix at position t. Then πtrain(yt∣ht;θ)≠πinfer(yt∣ht;θ) _train(y_t h_t;θ)≠ _infer(y_t h_t;θ), so rollouts that are nominally on-policy are in fact off-policy with respect to the training engine. The problem is more severe for mixture-of-experts models, where routing is a discrete top-k decision and a small perturbation can change which experts are activated. For the token-level GRPO surrogate evaluated on the observed rollout prefixes, correcting the conditional distribution of each sampled token introduces an importance ratio: (θ)=x∼,yii=1G∼πinfer(⋅∣x;θold)[1G∑i=1G1|yi|∑t=1|yi|ki,tmin(ρi,tA^i,t,clip(ρi,t)A^i,t)],J(θ)=E_x ,\;\y_i\_i=1^G _infer(· x; _old) [ 1G _i=1^G 1 y_i _t=1 y_i k_i,t\, ( _i,t A_i,t,\;clip ( _i,t ) A_i,t ) ], (1) where hi,t=(x,yi,<t)h_i,t=(x,y_i,<t), θold _old is the parameter snapshot at rollout time, A^i,t A_i,t is the group-normalized advantage, and clipclip restricts ρi,t _i,t to [1−clow, 1+chigh][1-c_low,\,1+c_high]. Let pi,t=πtrain(yi,t∣hi,t;θold)p_i,t= _train(y_i,t h_i,t; _old) and qi,t=πinfer(yi,t∣hi,t;θold)q_i,t= _infer(y_i,t h_i,t; _old) be the probabilities that the two engines assign to the sampled token. The update ratio is ρi,t=πtrain(yi,t∣hi,t;θ)/pi,t _i,t= _train(y_i,t h_i,t;θ)/p_i,t, and the mismatch ratio is ki,t=pi,t/qi,tk_i,t=p_i,t/q_i,t. The mismatch ratio ki,tk_i,t corrects only the next-token distribution given the observed prefix hi,th_i,t; it does not reweight the distribution of the prefix itself. The update ratio ρi,t _i,t is handled by the standard PPO clipping, while the mismatch ratio ki,tk_i,t carries the discrepancy between the two engines and is the focus of the rest of this paper. 2.2 Correcting the mismatch The most direct way to correct the training–inference mismatch is to use the exact ratio ki,tk_i,t in Equation 1. This correction is unbiased, but on the few tokens where the two engines disagree sharply, the ratio becomes very large and the gradient estimate has uncontrolled variance. Existing methods therefore truncate or mask ki,tk_i,t and accept some bias in exchange for lower variance. At the token level, TIS replaces ki,tk_i,t with min(ki,t,C) (k_i,t,C) for a fixed cap C (Yao et al., 2025). IcePop instead uses ki,t 1α≤ki,t≤βk_i,t\,1\α≤ k_i,t≤β\, which masks any token whose ratio falls outside a fixed interval (Zhao et al., 2025; Ling Team and Inclusion AI, 2025). KPop masks a token when the binary KL divergence between the probabilities of the two engines exceeds a fixed threshold (Guo et al., 2026; Ling Team, 2026). At the sequence level, Seq-TIS and Seq-MIS apply the same cap or mask to the product of the ratios over a response (Liu et al., 2025), and GSPO optimizes a sequence-level objective instead (Zheng et al., 2025b). A different line of work leaves the reweighting unchanged and reduces the discrepancy itself: FP16 training raises numerical precision (Qi et al., 2025), and routing replay makes the two engines select the same experts (Ma et al., 2025). All of these approaches are effective to some extent. However, the thresholds of the reweighting methods either act on ki,tk_i,t directly and treat every token in the same way, or are not derived from the statistics of the mismatch itself. In Section 3, we start from the logit perturbation between the two engines, characterize the structure of the mismatch, and build our correction on this characterization. 3 Calibrated Importance Sampling 3.1 From logit perturbation to log-odds displacement We measure, on large-scale rollouts, the probabilities that the two engines assign to the same sampled token (experimental details in Section A.1). On mixture-of-experts models, the top-k selection of the gating network is a discrete decision, so a small numerical difference can make the two engines activate different experts at some layer, and this difference propagates through all subsequent layers. These differences accumulate layer by layer and ultimately appear as different logits output by the two engines under the same parameters and the same prefix. Write =(z1,…,zV)∈ℝVz=(z_1,…,z_V) ^V for the logit vector of the training engine at the prefix hth_t, and +z+ δ for that of the inference engine, where =(δ1,…,δV)∈ℝV δ=( _1,…, _V) ^V is the per-logit perturbation between the two engines. Then pt=πtrain(yt∣ht;θold)=softmax()ytp_t= _train(y_t h_t; _old)=softmax(z)_y_t and qt=πinfer(yt∣ht;θold)=softmax(+)ytq_t= _infer(y_t h_t; _old)=softmax(z+ δ)_y_t. Substituting the two softmaxes into the mismatch ratio kt:=pt/qtk_t:=p_t/q_t yields the following identity. Figure 2: (a) Per-logit perturbation δj _j and (b) log-odds displacement εt _t, MoE vs. dense. Lemma 3.1 (Ratio–displacement identity). Let wj:=ezj/∑j′≠ytezj′w_j:=e^z_j / _j ≠ y_te^z_j for j≠ytj≠ y_t. Then the mismatch ratio satisfies kt=softmax()ytsoftmax(+)yt=pt+(1−pt)exp(εt),kt−1=(1−pt)(exp(εt)−1),k_t= softmax(z)_y_tsoftmax(z+ δ)_y_t=p_t+(1-p_t) ( _t), k_t-1=(1-p_t) ( ( _t)-1 ), (2) where εt:=−δyt+log∑j≠ytwjexp(δj) _t:=- _y_t+ _j≠ y_tw_j ( _j). The proof is given in Section B.1. We call εt _t the log-odds displacement of the inference engine relative to the training engine on the current token. From Equation 2, εt=logpt1−pt−logqt1−qt _t= p_t1-p_t- q_t1-q_t, that is, εt _t is exactly the difference between the log-odds that the two engines assign to the sampled token. In the log-odds coordinate, the discrepancy between the two engines is therefore an additive displacement determined entirely by the perturbation δ, and it can be computed directly from the log-probabilities recorded by the two engines. Figure 2 shows the measured δ and εt _t. On the dense control, δ is concentrated near zero and bounded, consistent with bf16 rounding error, and εt _t is correspondingly concentrated tightly around zero. On the MoE, the body of δ is likewise concentrated near zero, but is distinguished by a pronounced heavy tail. We analyze the source of this tail in Sections A.3 and A.4: rounding error accounts only for the body, while routing disagreement between the two engines is a major amplifier of the tail, and the tail of δ becomes heavier as the two engines select different experts at more layers. Since εt:=−δyt+log∑j≠ytwjexp(δj) _t:=- _y_t+ _j≠ y_tw_j ( _j), the heavy tail of δ carries over to εt _t, so the log-odds displacement on the MoE is heavy-tailed as well. Given ptp_t, by Equation 2, the ratio is a deterministic, increasing function of εt _t: the mismatch enters the importance weight through the displacement factor eεte _t, rescaled by 1−pt1-p_t. The same displacement therefore moves the ratio far from one on low-confidence tokens, whereas on high-confidence tokens the factor 1−pt1-p_t compresses it and the ratio barely moves. 3.2 Why exact correction is unstable A gradient update uses n tokens, indexed by m. Write the payload of the m-th token as ℓm′=∇θmin(ρmA^m,clip(ρm)A^m) _m~=~ _θ ( _m A_m,clip( _m) A_m ); the mismatch ratio kmk_m is the importance sampling factor used to scale this gradient. Since kmk_m does not depend on θ, the gradient of the token-level surrogate objective and its estimator are G0:=∇θ=[kℓ′],G^:=1n∑m=1nkmℓm′,G_0:= _θJ=E[k\, ], G:= 1n _m=1^nk_m\, _m, where the expectation is over the prompt, rollout, and numerical discrepancy between the two engines for each token. G0G_0 corrects the conditional distribution of each token given its prefix, not the distribution of the prefix itself. We measure the quality of the estimate by the mean-squared error ‖G^−G0‖22E \| G-G_0 \|_2^2. Figure 3: [eαεt]E[e^α _t] (left axis) and [((1−pt)eεt)α]E[((1-p_t)e _t)^α] (right axis) on the measured displacements of the trained MoE. Theorem 3.2 (Informal). Assume ‖ℓt′‖2≤b\| _t\|_2≤ b and that the contributions of different tokens are independent (for correlated contributions, replace n by neffn_eff; see Appendix B). Then ‖G^−G0‖22≤b2n[pt2+2pt(1−pt)exp(εt)+(1−pt)2e2εt].E \| G-G_0 \|_2^2\;≤\; b^2n\,E [p_t^2+2p_t(1-p_t) ( _t)+(1-p_t)^2\,e^2 _t ]. (3) Figure 3 shows both moments on the displacements of the MoE after RL training (Section A.7). The raw moment [eαεt]E[e^α _t] is only 7.077.07 at α=1α=1 but reaches 9.2×1069.2× 10^6 at α=2α=2. The weighted moment [((1−pt)eεt)α]E[((1-p_t)e _t)^α], which is the quantity that enters Equation 3, is compressed by the factor (1−pt)α(1-p_t)^α but follows the same pattern: it rises from 0.160.16 at α=1α=1 to 22.922.9 at α=2α=2. Since pt2≤1p_t^2≤ 1 and the cross term is at most 2×0.162× 0.16, the bound (Equation 3) is dominated by the second-moment term and is more than twenty times the corresponding bound without correction, for which the expectation equals one. Exact correction is therefore unbiased, but its variance is driven by the upper tail of εt _t. This changes the problem from whether the mismatch should be corrected to how the exact correction should be truncated. Some bias must be accepted in exchange for controlling its second moment. The next section constructs such a truncation directly in the coordinate from which the mismatch arises. 3.3 Correction in the displacement coordinate Figure 4: εt _t against token uncertainty 1−pt1-p_t on the base MoE; the black line is the median. Given ptp_t, the probability of generating a token, the mismatch enters the ratio only through eεte _t, by Equation 2: kt=pt+(1−pt)exp(εt).k_t=p_t+(1-p_t) ( _t). The variance problem is intrinsically one-sided. If eεt≤1e _t≤ 1, then pt<kt≤1,p_t<k_t≤ 1, so the importance weight on this side satisfies kt2≤1k_t^2≤ 1. In contrast, when exp(εt)>1 ( _t)>1, the ratio exceeds one and can grow without bound as the positive displacement increases. Thus the second-moment explosion identified in Section 3.2 can only arise from the upper tail. Modifying the lower side does not address this source of unbounded variance and instead introduces additional deviation from the exact correction. We therefore truncate only large positive displacements. It remains to choose where to truncate the upper tail. The displacement itself is only weakly related to confidence. Figure 4 plots εt _t against 1−pt1-p_t on the base MoE (Section A.6): across six orders of magnitude of uncertainty, the median stays at zero and the spread changes by less than a factor of two with a Spearman rank correlation of only −0.028-0.028. A displacement of a given amount is therefore approximately equally anomalous across confidence levels. This suggests placing a single upper threshold directly in the displacement coordinate with exp(εt)≤1+λ ( _t)≤ 1+λ where threshold λ≥0λ≥ 0. Truncating eεte _t at 1+λ1+λ and mapping back through Equation 2 gives fλ(kt,pt)=pt+(1−pt)mineεt, 1+λ=minkt, 1+λ(1−pt).f_λ(k_t,p_t)\;=\;p_t+(1-p_t) \e _t,\,1+λ \\;=\; \k_t,\;1+λ(1-p_t) \. (4) Equation 4 introduces calibrated importance sampling (CIS), our proposed correction for training–inference mismatch. The factor 1−pt1-p_t in the cap is not a design choice: it is the factor by which Equation 2 maps displacements to ratios. The two forms in Equation 4 are identical, token by token; we implement the right-hand form as it only requires the difference of the two log-probabilities, which remains finite when a probability rounds to one in bf16, whereas logitptlogitp_t and logitqtlogitq_t do not. For comparison, a fixed ratio cap kt≤Ck_t≤ C corresponds through Equation 2 to exp(εt)≤1+C−11−pt, ( _t)≤ 1+ C-11-p_t, (5) whose displacement threshold becomes increasingly permissive as confidence grows. CIS instead applies the same upper displacement threshold at every confidence level, and the confidence-dependent cap in Equation 4 follows automatically from the mapping back to ratio space. Theorem 3.3 (Informal). Assume ‖ℓt′‖2≤b\| _t\|_2≤ b and that the contributions of different tokens are independent. Then for any finite λ≥0λ≥ 0, ‖G^fλ−G0‖22≤ \| G_f_λ-G_0 \|_2^2\;≤ b2([(1−pt)(eεt−1−λ)+])2+b2n[(pt+(1−pt)mineεt,1+λ)2]. b^2 (E [(1-p_t)(e _t-1-λ)_+ ] )^2+ b^2n\,E [ (p_t+(1-p_t) \e _t,1+λ\ )^2 ]. (6) Compared with Equation 3, the unbounded second moment [e2εt]E[e^2 _t] is replaced by [min(eεt,1+λ)2]≤(1+λ)2E [ (e _t,1+λ)^2 ]≤(1+λ)^2, at the price of a bias from the truncated excess; λ controls this trade-off, and λ→∞λ→∞ recovers exact correction. Any cap with a finite maximum admits a similar bound, so the advantage of CIS over a fixed ratio cap lies not in this bound but in where the bias falls: CIS spreads the truncation bias more evenly across confidence levels and attains a lower overall bias (Section 4.3). Algorithm 1 Calibrated importance sampling (CIS) 1: ℓmtrain ^train_m, ℓminfer ^infer_m, ℓm′ _m for the n tokens of the batch; threshold λ; floor κ 2: for m=1,…,nm=1,…,n do 3: pm←exp(ℓmtrain)p_m← ( ^train_m), 4: km←exp(ℓmtrain−ℓminfer)k_m← ( ^train_m- ^infer_m) 5: φm←max(1−pm,κ) _m← (1-p_m,\,κ) 6: fm←minkm, 1+λφmf_m← \k_m,\;1+λ _m \ 7: return G^CIS=1n∑m=1nstopgrad(fm)ℓm′ G_CIS= 1n _m=1^nstopgrad(f_m)\, _m Algorithm. We treat λ as a hyperparameter; Section 4.4 shows that every positive threshold outperforms the uncorrected run. Applying Equation 4 in practice raises one issue: as 1−pt1-p_t becomes very small, the cap 1+λ(1−pt)1+λ(1-p_t) approaches one and becomes narrower than the storage resolution of the log-probabilities. The truncated tokens then have |logkt|| k_t| at the level of that resolution, so the truncation acts on rounding error rather than on the mismatch; without a remedy, the average accuracy drops from 34.7834.78 to 29.0229.02 (Section 4.3). We therefore floor 1−pt1-p_t at the storage resolution κ=5×10−3κ=5× 10^-3, replacing it by φt=max(1−pt,κ) _t= (1-p_t,κ), which gives Algorithm 1. For tokens with 1−pt≥κ1-p_t≥κ, Algorithm 1 applies exactly the displacement threshold exp(εt)≤1+λ ( _t)≤ 1+λ; below the floor, the threshold becomes eεt≤1+λκ/(1−pt)e _t≤ 1+λκ/(1-p_t), which is looser. The correction adds one elementwise pass and no forward or backward computation. The inference-side log-probability must be the raw value recorded at the sampling step, and the weight must be detached. 4 Experiments We organize the experiments around three questions. RQ1: Does CIS improve held-out performance over existing training–inference mismatch corrections? RQ2: Why does CIS work, and which design choices matter? RQ3: How sensitive is CIS to its hyperparameters? Section 4.1 describes the shared setup. Implementation details, training dynamics, additional diagnostics, and out-of-domain evaluations are provided in Appendices C and D. 4.1 Setup We evaluate RLVR on three MoE models: Qwen1.5-MoE-A2.7B-Chat (Qwen Team, 2024), DeepSeek-V2-Lite (DeepSeek-AI, 2024), and Qwen3-30B-A3B (Qwen Team, 2025). All models are trained on GSM8K (Cobbe et al., 2021) and evaluated with greedy pass@1 on five mathematical reasoning benchmarks: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021; Lightman et al., 2024), SVAMP (Patel et al., 2021), Minerva-Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). All methods share the same training recipe and code path and differ only in how they handle the training–inference mismatch. Unless stated otherwise, we report mean and standard deviation over three training seeds. We group the baselines by where they act: token-level (Exact Ratio (Yao et al., 2025; Liu et al., 2025), TIS (Yao et al., 2025), IcePop (Zhao et al., 2025; Ling Team and Inclusion AI, 2025), and KPop† (Guo et al., 2026; Ling Team, 2026)), sequence-level (Seq-TIS and Seq-MIS (Liu et al., 2025), and GSPO (Zheng et al., 2025b)), and numerical (FP16 (Qi et al., 2025)). The baseline “No Correction” provides the uncorrected reference, while CIS belongs to the token-level class. Full implementation details, hyperparameters, and evaluation protocols are given in Appendix C. 4.2 RQ1: Does CIS improve held-out performance? Table 1: Held-out accuracy (%) on five math benchmarks after RL on GSM8K. Mean ± standard deviation over three seeds; bold marks the best mean per column within each model. Model Method GSM8K MATH500 SVAMP Minerva Olympiad Avg Qwen1.5-MoE-A2.7B Base model (no RL) 55.5055.50 21.2021.20 59.0059.00 4.044.04 4.904.90 28.9328.93 No correction 61.21±1.1761.21_± 1.17 21.33±1.4021.33_± 1.40 64.33±1.7364.33_± 1.73 3.68±1.273.68_± 1.27 4.40±0.454.40_± 0.45 30.99±0.2230.99_± 0.22 Exact ratio 63.18±2.9763.18_± 2.97 22.00±1.2222.00_± 1.22 67.11±3.7567.11_± 3.75 5.51±0.645.51_± 0.64 4.60±0.154.60_± 0.15 32.48±1.2132.48_± 1.21 KPop† 64.80±1.2964.80_± 1.29 23.67±1.2923.67_± 1.29 68.67±3.7168.67_± 3.71 5.15±1.475.15_± 1.47 5.24±0.675.24_± 0.67 33.50±1.0433.50_± 1.04 IcePop 66.16±0.9666.16_± 0.96 24.33±1.0324.33_± 1.03 69.78±1.0269.78_± 1.02 5.51±0.645.51_± 0.64 5.09±1.115.09_± 1.11 34.18±0.5534.18_± 0.55 TIS 61.03±5.1161.03_± 5.11 20.73±3.9220.73_± 3.92 66.78±5.0966.78_± 5.09 4.54±0.764.54_± 0.76 3.91±0.313.91_± 0.31 31.40±2.4731.40_± 2.47 Seq-TIS 65.93±0.1965.93_± 0.19 22.93±0.6122.93_± 0.61 63.56±1.3963.56_± 1.39 3.68±1.603.68_± 1.60 5.29±0.895.29_± 0.89 32.28±0.8632.28_± 0.86 Seq-MIS 60.98±1.5060.98_± 1.50 22.07±0.5022.07_± 0.50 63.56±1.1763.56_± 1.17 4.41±0.644.41_± 0.64 4.75±0.264.75_± 0.26 31.15±0.6731.15_± 0.67 GSPO 60.10±2.7760.10_± 2.77 17.47±0.5017.47_± 0.50 58.56±7.7358.56_± 7.73 5.02±0.565.02_± 0.56 3.36±0.313.36_± 0.31 28.90±1.2528.90_± 1.25 FP16 60.02±1.1660.02_± 1.16 22.67±0.4622.67_± 0.46 62.78±1.5462.78_± 1.54 5.02±0.215.02_± 0.21 4.60±0.454.60_± 0.45 31.02±0.3931.02_± 0.39 CIS (ours) 67.88±1.6467.88_± 1.64 24.53±1.0324.53_± 1.03 70.22±1.3570.22_± 1.35 5.88±0.645.88_± 0.64 5.39±0.625.39_± 0.62 34.78±0.2534.78_± 0.25 DeepSeek-V2-Lite Base model (no RL) 70.6670.66 25.4025.40 67.6767.67 6.256.25 6.086.08 35.2135.21 No correction 71.06±0.8371.06_± 0.83 23.47±0.6423.47_± 0.64 70.89±2.9170.89_± 2.91 7.23±1.497.23_± 1.49 6.48±0.566.48_± 0.56 35.83±1.0835.83_± 1.08 Exact ratio 71.17±0.2771.17_± 0.27 24.67±0.6424.67_± 0.64 70.22±1.8470.22_± 1.84 6.37±1.496.37_± 1.49 4.95±0.564.95_± 0.56 35.47±0.3435.47_± 0.34 KPop† 71.14±0.5971.14_± 0.59 26.33±0.6426.33_± 0.64 70.89±2.5570.89_± 2.55 6.62±1.106.62_± 1.10 6.13±0.826.13_± 0.82 36.22±0.7436.22_± 0.74 IcePop 72.48±0.6972.48_± 0.69 25.60±0.9225.60_± 0.92 71.44±1.0271.44_± 1.02 7.72±1.607.72_± 1.60 6.43±0.766.43_± 0.76 36.73±0.7336.73_± 0.73 TIS 72.55±0.8472.55_± 0.84 24.80±1.0024.80_± 1.00 69.33±1.7669.33_± 1.76 7.72±0.977.72_± 0.97 5.69±1.125.69_± 1.12 36.02±0.5136.02_± 0.51 Seq-TIS 71.87±0.5071.87_± 0.50 24.53±0.7024.53_± 0.70 70.89±0.8470.89_± 0.84 6.74±1.186.74_± 1.18 5.74±0.455.74_± 0.45 35.95±0.2935.95_± 0.29 Seq-MIS 70.61±0.4670.61_± 0.46 25.93±1.0125.93_± 1.01 70.78±2.5970.78_± 2.59 6.86±0.566.86_± 0.56 6.33±0.456.33_± 0.45 36.10±0.7136.10_± 0.71 GSPO 72.50±0.8672.50_± 0.86 25.93±0.7025.93_± 0.70 71.33±1.0071.33_± 1.00 6.37±1.666.37_± 1.66 6.18±0.526.18_± 0.52 36.47±0.4336.47_± 0.43 FP16 71.80±0.6071.80_± 0.60 24.47±0.7024.47_± 0.70 69.89±2.1769.89_± 2.17 7.97±1.667.97_± 1.66 6.23±0.686.23_± 0.68 36.07±0.6136.07_± 0.61 CIS (ours) 72.58±0.7672.58_± 0.76 26.47±0.9926.47_± 0.99 71.89±2.0471.89_± 2.04 8.21±1.668.21_± 1.66 6.68±0.516.68_± 0.51 37.16±0.7337.16_± 0.73 Qwen3-30B-A3B Base model (no RL) 95.2295.22 74.0074.00 93.3393.33 31.2531.25 45.4045.40 67.8467.84 No correction 94.79±0.3294.79_± 0.32 72.40±0.7272.40_± 0.72 94.56±1.1794.56_± 1.17 33.33±0.9333.33_± 0.93 46.24±0.5246.24_± 0.52 68.26±0.2768.26_± 0.27 Exact ratio 95.35±0.3295.35_± 0.32 74.40±0.5374.40_± 0.53 93.22±1.0793.22_± 1.07 33.33±0.2133.33_± 0.21 47.28±1.0947.28_± 1.09 68.72±0.3068.72_± 0.30 KPop† 95.40±0.2395.40_± 0.23 74.80±0.5374.80_± 0.53 94.56±0.6994.56_± 0.69 33.21±0.9333.21_± 0.93 47.33±0.3947.33_± 0.39 69.06±0.2969.06_± 0.29 IcePop 95.43±0.4995.43_± 0.49 74.93±0.6474.93_± 0.64 94.89±1.1794.89_± 1.17 32.11±1.2932.11_± 1.29 47.58±0.7047.58_± 0.70 68.99±0.5168.99_± 0.51 TIS 95.15±0.2095.15_± 0.20 74.80±0.7274.80_± 0.72 94.56±0.8494.56_± 0.84 33.58±1.1833.58_± 1.18 47.63±0.4547.63_± 0.45 69.14±0.3069.14_± 0.30 Seq-TIS 95.12±0.1695.12_± 0.16 74.67±0.6174.67_± 0.61 93.78±1.0293.78_± 1.02 33.33±1.2933.33_± 1.29 48.07±0.4548.07_± 0.45 68.99±0.6168.99_± 0.61 Seq-MIS 95.15±0.3595.15_± 0.35 73.93±0.4273.93_± 0.42 93.89±1.0293.89_± 1.02 32.72±1.1032.72_± 1.10 47.73±0.4547.73_± 0.45 68.68±0.1568.68_± 0.15 GSPO 95.55±0.2495.55_± 0.24 74.47±0.5074.47_± 0.50 94.44±1.1794.44_± 1.17 33.21±0.5633.21_± 0.56 48.47±0.7348.47_± 0.73 69.23±0.1669.23_± 0.16 FP16 94.79±0.2794.79_± 0.27 74.13±0.4274.13_± 0.42 93.67±1.4593.67_± 1.45 33.21±1.6633.21_± 1.66 46.83±0.9146.83_± 0.91 68.53±0.3168.53_± 0.31 CIS (ours) 95.60±0.3395.60_± 0.33 75.00±0.9275.00_± 0.92 95.89±1.5095.89_± 1.50 34.31±0.5634.31_± 0.56 48.62±0.4548.62_± 0.45 69.88±0.5569.88_± 0.55 Table 1 compares all methods under the same training recipe. CIS achieves the highest five-benchmark average on all three MoE models. On Qwen1.5-MoE-A2.7B, the average improves from 30.9930.99 without correction to 34.7834.78 with CIS, compared with 32.4832.48, 33.5033.50, 34.1834.18, and 31.4031.40 for Exact Ratio, KPop†, IcePop, and TIS, respectively. CIS also achieves 67.8867.88 on the training-domain GSM8K benchmark. Its margins over IcePop on several transfer benchmarks are smaller than the across-seed variation, however, so we do not interpret these individual columns as statistically separated. The same trend holds across model families. CIS reaches an average of 37.1637.16 on DeepSeek-V2-Lite and 69.8869.88 on Qwen3-30B-A3B, the highest mean in both blocks. For Qwen3-30B-A3B, the base model already reaches 95.22%95.22\% on GSM8K, leaving little headroom on the training task; the five-benchmark average therefore provides the more informative comparison. Methods that act at other levels show different limitations. Sequence-level correction accumulates mismatch over the whole response, while changing numerical precision reduces but does not eliminate the residual engine discrepancy. We also evaluate the Qwen1.5-MoE checkpoints on three knowledge and two code benchmarks that the RL stage never uses (Table 13 in Section D.3); CIS has the highest pooled accuracy there, but its margin over IcePop is within the seed spread. Training dynamics and computational cost are reported in Sections D.1 and D.4. 4.3 RQ2: Why does CIS work, and which design choices matter? Figure 5: (a) Share of Var(kt)Var(k_t) on each side of logkt=0 k_t=0. (b) Truncation bias across confidence bins. (c) Overall bias relative to exact correction. (d) Added variance relative to no correction. Table 2: Ablations of CIS on Qwen1.5-MoE-A2.7B. Held-out accuracy (%) under the protocol of Table 1; mean ± standard deviation over three seeds. Variant GSM8K MATH500 SVAMP Minerva Olympiad Avg CIS (λ=2.3λ=2.3, κ=5×10−3κ=5× 10^-3) 67.88±1.6467.88_± 1.64 24.53±1.0324.53_± 1.03 70.22±1.3570.22_± 1.35 5.88±0.645.88_± 0.64 5.39±0.625.39_± 0.62 34.78±0.2534.78_± 0.25 Fixed ratio cap (TIS, C=2C=2) 61.03±5.1161.03_± 5.11 20.73±3.9220.73_± 3.92 66.78±5.0966.78_± 5.09 4.54±0.764.54_± 0.76 3.91±0.313.91_± 0.31 31.40±2.4731.40_± 2.47 Fixed ratio cap (TIS, C=1.13C=1.13) 14.99±17.0414.99_± 17.04 5.73±5.065.73_± 5.06 20.33±20.5320.33_± 20.53 1.47±1.911.47_± 1.91 1.88±1.861.88_± 1.86 8.88±9.248.88_± 9.24 Two-sided band (wt≥(1+pt)/2w_t≥(1+p_t)/2) 5.31±0.335.31_± 0.33 4.40±0.404.40_± 0.40 15.67±1.8615.67_± 1.86 1.47±0.641.47_± 0.64 3.21±0.453.21_± 0.45 6.01±0.506.01_± 0.50 Raised floor (κ=0.1κ=0.1) 64.42±1.6764.42_± 1.67 22.60±4.0622.60_± 4.06 64.44±1.9564.44_± 1.95 4.90±0.854.90_± 0.85 4.50±0.374.50_± 0.37 32.17±1.2832.17_± 1.28 No floor (κ=0κ=0) 55.62±2.6955.62_± 2.69 20.87±1.1020.87_± 1.10 59.78±2.8359.78_± 2.83 5.88±0.375.88_± 0.37 2.97±0.152.97_± 0.15 29.02±1.4329.02_± 1.43 Why truncate only the upper side? The variance is almost entirely carried by the upper tail: on the sampled tokens of the trained MoE at step 86, tokens with kt>1k_t>1 account for 99.96%99.96\% of Var(kt)Var(k_t), whereas those with kt<1k_t<1 contribute only 0.04%0.04\% (Figure 5a). The training ablation shows the same asymmetry: additionally lifting small displacements to eεt≥1/2e _t≥ 1/2, which gives the two-sided weight wt=maxminkt,1+λφt,(1+pt)/2w_t= \ \k_t,1+λ _t\,\,(1+p_t)/2\, reduces the average accuracy from 34.7834.78 to 6.016.01 (see Table 2); CIS applies no lower threshold. CIS therefore truncates only the upper side and leaves the bounded lower side unchanged. Figure 6: (a) Threshold λ and (b) floor κ on Qwen1.5-MoE-A2.7B. Why truncate at the source of the mismatch? TIS caps the final importance ratio ktk_t, whereas CIS caps the displacement factor eεte _t, through which the mismatch enters the ratio, at the common threshold 1+λ1+λ of Equation 4 and then maps this bound back to ratio space. Because ktk_t mixes the displacement with token confidence, a fixed cap on ktk_t does not represent the same amount of mismatch across tokens. In contrast, truncating eεte _t applies the same criterion to the mismatch itself, while the induced ratio bound automatically adapts to confidence. This changes where truncation bias is spent. Figure 5b bins the sampled tokens of the base MoE by ptp_t and reports [kt−ft]E[k_t-f_t] in each bin. TIS concentrates its bias on low-confidence tokens: below pt=0.3p_t=0.3, the bias induced by CIS is less than half that of TIS, whereas CIS takes on more bias at high confidence. CIS truncates more high-confidence tokens by count, but each such truncation removes little weight, because the factor 1−pt1-p_t compresses the ratio on these tokens; more than half of the weight that CIS removes comes from tokens with pt<0.9p_t<0.9. The two operators differ most in gradient bias at low confidence (Figure 5b): for pt<0.565p_t<0.565, the cap of CIS is looser than C=2C=2, so CIS removes less weight there than TIS. Section D.5 shows the tokens that each operator acts on during training. Figure 5c,d correspond to the two terms of the mean-squared error in Theorem 3.3: the bias ‖[(ft−kt)ℓt′]‖2\|E[(f_t-k_t) _t]\|_2 and the increase in the variance of ftℓt′f_t _t over ft≡1f_t≡ 1. We compute both with the gradient of logpt p_t with respect to the logits in place of ℓt′ _t, taking expectations over the next-token distributions at the 6,000 base-model positions of Section A.1; exact correction is unbiased for the conditional target G0G_0 and therefore has zero bias. Although CIS truncates more probability mass overall, its overall bias is lower (0.00640.0064 versus 0.00780.0078 for TIS), while its added variance remains well below that of Exact Ratio (0.00810.0081 versus 0.01320.0132). Thus, truncating at the mismatch source changes where the clipping occurs, not only how much is clipped, which yields a more favorable bias–variance trade-off. Ablations. Table 2 supports these design choices. Replacing the confidence-dependent CIS boundary with the fixed TIS cap reduces the average accuracy from 34.7834.78 to 31.4031.40, while the two-sided band drops it to 6.016.01, below the base model (28.9328.93). With C=1.13C=1.13, which truncates a similar share of tokens as CIS (4.3%4.3\% versus 3.8%3.8\%, Table 10), the average falls to 8.888.88: the same amount of truncation harms training when a fixed cap places it on low-confidence tokens. TIS with C=1C=1, which is CIS with λ=0λ=0, collapses as well (Figure 6a). The numerical floor is also necessary: removing it lowers the average accuracy to 29.0229.02, and raising it to κ=0.1κ=0.1 lowers it to 32.1732.17. Together, the results support the three components of CIS: upper-side truncation, truncation at the mismatch source, and a small numerical floor. 4.4 RQ3: How sensitive is CIS to its hyperparameters? Figure 6 varies one hyperparameter of CIS at a time on Qwen1.5-MoE-A2.7B, with the other fixed at its default; per-benchmark results are in Section D.2. For the threshold, λ=0λ=0 truncates every ratio above one, which is TIS with C=1C=1, and the average accuracy collapses to 8.758.75. Every positive threshold stays above the uncorrected run (30.9930.99), and the default λ=2.3λ=2.3 gives the highest accuracy (34.7834.78). The other positive thresholds lie between 32.5332.53 and 34.0834.08, so once λ>0λ>0 the average changes by at most 2.32.3 points. For the floor, accuracy peaks near the storage resolution of the log-probabilities, where κ=5×10−3κ=5× 10^-3 and κ=2×10−2κ=2× 10^-2 reach 34.7834.78 and 34.3934.39. A smaller floor of 10−310^-3 loses 1.41.4 points, and removing the floor lowers the average to 29.0229.02, below the uncorrected run, which is consistent with the cap then truncating rounding error on high-confidence tokens. Raising the floor to 0.10.1 lowers the average to 32.1732.17; the cap then stays constant on every token with 1−pt<0.11-p_t<0.1 and no longer tightens with confidence. 5 Related Work RL frameworks separate rollout generation from gradient computation to improve efficiency (Sheng et al., 2025; Hu et al., 2025; Fu et al., 2025), but numerical differences between the engines can yield different token probabilities even with identical parameters (Yao et al., 2025). In mixture-of-experts models, small perturbations can change discrete expert selections and amplify the mismatch (Ma et al., 2025; Ling Team and Inclusion AI, 2025). FP16 reduces numerical error (Qi et al., 2025), routing replay aligns expert selections (Ma et al., 2025), and adaptive learning-rate decay stabilizes optimization (Zhang et al., 2026b). These interventions address different mechanisms: improved precision does not enforce routing agreement, while replay requires routing traces. RSPO instead addresses routing drift across policy updates (Zhang et al., 2026a), which differs from engine mismatch at identical parameters. CIS complements these approaches by controlling the gradient impact of residual mismatch without changing numerical precision or recording routes. Importance sampling corrects distribution mismatch, while truncation trades bias for reduced variance (Ionides, 2008; Espeholt et al., 2018). For engine mismatch, TIS caps token ratios (Yao et al., 2025), and IcePop masks ratios outside a fixed interval (Zhao et al., 2025; Ling Team and Inclusion AI, 2025). These thresholds do not account for how token confidence affects the ratio, whereas KPop addresses this heterogeneity through binary-KL masking (Guo et al., 2026; Ling Team, 2026). Sequence-level alternatives truncate response weights (Liu et al., 2025), optimize sequence ratios (Zheng et al., 2025b), or reject responses that violate a trust-region criterion (Li et al., 2025). CISPO also clips weights to preserve token contributions, but uses fixed bounds on the current-to-old policy ratio (Chen et al., 2025). CIS instead derives its training–inference ratio cap from a common threshold on log-odds displacement. The resulting cap tightens with token confidence, and it keeps a nonzero weight for truncated tokens. 6 Conclusion We study training–inference mismatch in RLVR and show that it enters the importance ratio as an additive displacement in log-odds, whose distribution varies far less with token confidence than the ratio and has a heavy upper tail on mixture-of-experts models. Based on this characterization, calibrated importance sampling (CIS) truncates the upper tail of the displacement at a single threshold, which yields a ratio cap that tightens as token confidence increases. We prove that CIS bounds the second moment that makes exact correction unstable, at the cost of a bias controlled by the truncated excess. On three mixture-of-experts models, CIS achieves the highest five-benchmark average among the evaluated baselines. Diagnostic analyses show that CIS places less truncation bias on low-confidence tokens than TIS, and that truncating the lower side reduces held-out accuracy. References Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. arXiv. External Links: Document, Link Cited by: Table 9. Chen et al. (2025) A. Chen, A. Li, B. Gong, B. Jiang, B. Fei, B. Yang, B. Shan, C. Yu, C. Wang, C. Zhu, et al. Minimax-m1: scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585. Cited by: §5. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. arXiv. External Links: Document, Link Cited by: Table 9. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv. External Links: Document, Link Cited by: Table 9. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv. External Links: Document, Link Cited by: §C.2, Table 9, §4.1. DeepSeek-AI (2024) DeepSeek-AI DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model. CoRR abs/2405.04434. External Links: Document, Link Cited by: §C.1, §4.1. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. CoRR abs/2501.12948. External Links: Document, Link Cited by: §1. Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, Cited by: §5. Fu et al. (2025) W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, T. Yu, B. Yuan, and Y. Wu AReaL: a large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298. Cited by: §1, §5. Guo et al. (2026) J. Guo, Y. Sun, Z. Huang, Z. Wang, Z. Wen, Z. Zhang, J. Zhou, and S. Kok KPop: taming training–inference mismatch in reinforcement learning with adaptive masking regions. Note: Technical blog External Links: Link Cited by: §1, §2.2, §4.1, §5. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3828–3850. External Links: Link, Document Cited by: Table 9, §4.1. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: Table 9. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: Table 9, §4.1. Hu et al. (2025) J. Hu, X. Wu, W. Shen, J. K. Liu, W. Wang, S. Jiang, H. Wang, H. Chen, B. Chen, W. Fang, Xianyu, Y. Cao, H. Xu, and Y. Liu OpenRLHF: a ray-based easy-to-use, scalable and high-performance RLHF framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, I. Habernal, P. Schulam, and J. Tiedemann (Eds.), Suzhou, China, p. 656–666. External Links: Link, Document, ISBN 979-8-89176-334-0 Cited by: §1, §5. Ionides (2008) E. L. Ionides Truncated importance sampling. Journal of Computational and Graphical Statistics 17 (2), p. 295–311. Cited by: §1, §5. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra Solving quantitative reasoning problems with language models. arXiv. External Links: Document, Link Cited by: Table 9, §4.1. Li et al. (2025) Y. Li, J. Liu, J. Xu, Y. Tong, Z. Li, Q. Liu, and B. Wang Trust region masking for long-horizon llm reinforcement learning. arXiv preprint arXiv:2512.23075. Cited by: §5. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578–39601. Cited by: Table 9, §4.1. Ling Team and Inclusion AI (2025) Ling Team and Inclusion AI Every step evolves: scaling reinforcement learning for trillion-scale thinking model. CoRR abs/2510.18855. External Links: Document, Link Cited by: §1, §1, §2.2, §4.1, §5, §5. Ling Team (2026) Ling Team Ling and ring 2.6 technical report: efficient and instant agentic intelligence at trillion-parameter scale. CoRR abs/2606.15079. External Links: Document, Link Cited by: §1, §2.2, §4.1, §5. Liu et al. (2025) J. Liu, Y. Li, Y. Fu, J. Wang, Q. Liu, and Y. Shen When speed kills stability: demystifying RL collapse from the training-inference mismatch. External Links: Link Cited by: §1, §1, §2.2, §4.1, §5. Ma et al. (2025) W. Ma, H. Zhang, L. Zhao, Y. Song, Y. Wang, Z. Sui, and F. Luo Stabilizing MoE reinforcement learning by aligning training and inference routers. arXiv preprint arXiv:2510.11370. Note: Introduces Rollout Routing Replay (R3) Cited by: §D.6, §1, §1, §2.2, §5. Metelli et al. (2018) A. M. Metelli, M. Papini, F. Faccio, and M. Restelli Policy optimization via importance sampling. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, p. . External Links: Link Cited by: §1. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 2381–2391. External Links: Link, Document Cited by: Table 9. Patel et al. (2021) A. Patel, S. Bhattamishra, and N. Goyal Are NLP models really able to solve simple math word problems?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou (Eds.), Online, p. 2080–2094. External Links: Link, Document Cited by: Table 9, §4.1. Qi et al. (2025) P. Qi, Z. Liu, X. Zhou, T. Pang, C. Du, W. S. Lee, and M. Lin Defeating the training-inference mismatch via FP16. arXiv preprint arXiv:2510.26788. Cited by: §D.6, §1, §1, §2.2, §4.1, §5. Qwen Team (2024) Qwen Team Qwen1.5-MoE: matching 7B model performance with 1/3 activated parameters. Note: https://qwenlm.github.io/blog/qwen-moe/ Cited by: §4.1. Qwen Team (2025) Qwen Team Qwen3 technical report. CoRR abs/2505.09388. External Links: Document, Link Cited by: §C.1, §4.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Note: Introduces GRPO Cited by: Figure 1, §1. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. External Links: Link Cited by: §1, §5. Yao et al. (2025) F. Yao, L. Liu, D. Zhang, C. Dong, J. Shang, and J. Gao Your efficient rl framework secretly brings you off-policy rl training. External Links: Link Cited by: §1, §1, §1, §2.2, §4.1, §5, §5. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, j. liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference, p. 113222–113244. External Links: Document, Link Cited by: §C.2, §1. Zhang et al. (2026a) D. Zhang, X. Wu, S. Huang, L. Jiang, Y. Hao, L. Dong, Z. Chi, Z. Sui, and F. Wei Towards stable and effective reinforcement learning for mixture-of-experts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 25446–25457. Cited by: §5. Zhang et al. (2026b) Y. Zhang, Y. Li, J. Liu, J. Xu, Z. Li, Q. Liu, and H. Li Beyond precision: training-inference mismatch is an optimization problem and simple lr scheduling fixes it. arXiv preprint arXiv:2602.01826. Cited by: §5. Zhao et al. (2025) X. Zhao, Y. Liu, K. Xu, J. Guo, Z. Wang, Y. Sun, X. Kong, Q. Cao, L. Jiang, Z. Wen, Z. Zhang, and J. Zhou Small leak can sink a great ship—boost RL training on MoE with IcePop!. Note: Technical blog External Links: Link Cited by: §1, §1, §2.2, §4.1, §5. Zheng et al. (2025a) C. Zheng, K. Dang, B. Yu, M. Li, H. Jiang, J. Lin, Y. Liu, H. Lin, C. Wu, F. Hu, A. Yang, J. Zhou, and J. Lin Stabilizing reinforcement learning with LLMs: formulation and practices. CoRR abs/2512.01374. External Links: Document, Link Cited by: §1, §1. Zheng et al. (2025b) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §1, §1, §2.2, §4.1, §5. Appendix Appendix A Empirical Anatomy of the Training–Inference Mismatch Section 3.1 derives the ratio–displacement identity kt=pt+(1−pt)eεt,k_t=p_t+(1-p_t)e _t, starting from the per-logit perturbation between the training and inference engines. This appendix provides the measurements behind the empirical claims used in that derivation and in Sections 3.2 and 3.3. We first describe how the per-logit perturbation is measured (Section A.1). We then compare dense and mixture-of-experts architectures (Section A.2), examine how routing disagreement is associated with the MoE tail (Sections A.3 and A.4), and verify that the heavy tail propagates from the per-logit perturbation δj _j to the log-odds displacement εt _t (Section A.5). Finally, we examine the dependence of εt _t on token confidence (Section A.6) and show how the extreme upper tail dominates the second moment that enters the importance-sampling variance bound (Section A.7). A.1 Measurement protocol Models and engine configuration. We measure Qwen1.5-MoE-A2.7B-Chat and use Qwen1.5-14B-Chat as a dense control. All measurements in Sections A.2, A.3 and A.4 use the base checkpoints, with the training and inference engines loading exactly the same model parameters. The training engine is HuggingFace Transformers in bf16 with teacher forcing. The inference engine is vLLM 0.26 in bf16 with eager execution and prefix caching disabled. Per-logit measurement. For each model, we randomly select 250 trajectories from the first shard of the static mismatch measurement (Section C.4) and sample 24 generation positions from each trajectory, giving 6,000 positions. At each position, the training engine records its top-256 pre-softmax logits together with the corresponding log-normalizer. On the inference side, we truncate the trajectory immediately before that position and generate one token with logprobs=256, which returns the next-token distribution at the same prefix. The trajectory and position identifiers align on all sampled positions. The two top-256 sets have mean intersection rates of 0.9910.991 on the dense model and 0.9490.949 on the MoE, yielding 1,521,6721,521,672 and 1,457,4771,457,477 paired logit entries, respectively. The complete measurement takes approximately eight minutes on four A100 GPUs. Gauge convention. In Section 3.1, the training-engine logits are written as z and the inference-engine logits as +z+ δ. Raw logits are defined only up to an additive constant at each position. Because the displacement depends only on logit contrasts, adding the same constant to every coordinate of δ does not change εt _t. We therefore use the normalized log-probability gauge δj=logπinfer(j∣ht)−logπtrain(j∣ht). _j= _infer(j h_t)- _train(j h_t). (7) This differs from the raw logit perturbation only by a position-specific constant and is uniquely determined by the two observed distributions. Under this convention, εt=logitpt−logitqt, _t=logitp_t-logitq_t, consistent with the convention used throughout the main text. For the MoE model, the same measurement additionally records the top-4 expert identities selected by each engine at every layer, together with the routing margin between the fourth- and fifth-ranked experts. These records are used in Sections A.3 and A.4. A.2 Dense versus MoE: a heavy tail in the per-logit perturbation Figure 7: Complementary cumulative distributions of (a) |δj|| _j| and (b) |εt|| _t| on the dense and MoE models. Table 3: Per-logit perturbation δj _j on the base checkpoints. Dense MoE Paired entries 1,521,6721,521,672 1,457,4771,457,477 Median 0.00000.0000 0.00000.0000 Mean −0.0013-0.0013 −0.0036-0.0036 |mean|/sd|mean|/sd 1.6%1.6\% 1.1%1.1\% sd(δj)sd( _j) 0.0790.079 0.3360.336 q99(|δj|)q_99(| _j|) 0.2500.250 1.1441.144 q99.9(|δj|)q_99.9(| _j|) 0.3120.312 2.0622.062 Pr(|δj|>0.5) (| _j|>0.5) 0.003%0.003\% 9.87%9.87\% Pr(|δj|>1) (| _j|>1) 0.0001%0.0001\% 1.52%1.52\% maxj|δj| _j| _j| 1.001.00 10.4710.47 Range [−1.00, 0.91][-1.00,\,0.91] [−7.70, 10.47][-7.70,\,10.47] Figure 7(a) compares the full tails. The two distributions are both concentrated near zero, but they separate sharply beyond the central region. On the dense control, only 0.003%0.003\% of paired entries satisfy |δj|>0.5| _j|>0.5, and essentially none exceed one. On the MoE, 9.87%9.87\% exceed 0.50.5, 1.52%1.52\% exceed one, and the largest perturbation reaches 10.4710.47. The standard deviation is correspondingly 0.3360.336, more than four times the dense value of 0.0790.079. The difference is especially visible in the extreme quantiles: q99.9(|δj|)q_99.9(| _j|) grows from 0.3120.312 on the dense model to 2.0622.062 on the MoE. Thus the distinction between the two architectures is not merely a uniform rescaling of a narrow numerical perturbation; the MoE develops a pronounced tail that is absent from the dense control. A.3 Routing disagreement and the MoE tail To examine the source of this additional tail, we use the expert identities recorded on the MoE. For each sampled position, let NflipN_flip denote the number of layers, out of 24, for which the two engines select different top-4 expert sets. Routing disagreement is common: the mean is 3.693.69 disagreeing layers per position, and only 8.0%8.0\% of positions select the same expert set at all 24 layers. Figure 8: (a) Standard deviation of δj _j and (b) share of entries with |δj|>0.5| _j|>0.5 against the number of layers with routing disagreement; dashed: dense model. Table 4: Per-logit perturbation against the number of layers NflipN_flip with routing disagreement. NflipN_flip 00 11 22–33 44–66 ≥7≥ 7 Positions (%) 8.08.0 15.615.6 33.133.1 28.228.2 15.115.1 sd(δj)sd( _j) 0.2400.240 0.2690.269 0.3070.307 0.3520.352 0.4570.457 Pr(|δj|>0.5) (| _j|>0.5) (%) 4.514.51 5.425.42 7.857.85 11.8611.86 18.4818.48 Pr(|δj|>1) (| _j|>1) (%) 0.470.47 0.680.68 1.111.11 1.651.65 3.733.73 The two direct perturbation statistics show a clear monotonic trend. From positions with no routing disagreement to positions with disagreement in at least seven layers, sd(δj)sd( _j) increases from 0.2400.240 to 0.4570.457, a 90%90\% increase, while Pr(|δj|>0.5) (| _j|>0.5) rises from 4.51%4.51\% to 18.48%18.48\%, a factor of 4.14.1. The conclusion is unchanged under a stricter definition that counts only routing differences whose gate margin exceeds 10−310^-3. Under this “hard-flip” definition, the conditional standard deviations across the five bins are 0.2840.284, 0.3200.320, 0.4050.405, 0.4600.460, and 0.5970.597, while Pr(|δj|>0.5) (| _j|>0.5) increases from 6.40%6.40\% to 9.42%9.42\%, 14.72%14.72\%, 20.52%20.52\%, and 30.78%30.78\%. The association between routing disagreement and the width of the perturbation is therefore robust to the definition of a route change. A.4 Conditional routing control The previous analysis is observational: it stratifies positions by their realized routing disagreement rather than intervening on the routes. To separate the architecture-level difference from the additional effect associated with routing disagreement at the current position, we compare the dense control with MoE positions whose expert selections either agree at all 24 layers or disagree in at least one layer. Table 5: Per-logit perturbation on the dense model and on MoE positions with and without routing disagreement. sd(δj)sd( _j) Pr(|δj|>0.5) (| _j|>0.5) Pr(|δj|>1) (| _j|>1) Dense 0.0790.079 0.003%0.003\% 0.0001%0.0001\% MoE, Nflip=0N_flip=0 0.2400.240 4.51%4.51\% 0.47%0.47\% MoE, Nflip>0N_flip>0 0.3430.343 10.35%10.35\% 1.62%1.62\% Routing disagreement at the current position therefore does not explain the entire architecture gap. Even at Nflip=0N_flip=0, the MoE standard deviation is approximately three times that of the dense control. One possible source is routing disagreement earlier in the prefix, whose effect propagates through the hidden state to subsequent positions. The existing measurement cannot isolate this contribution: only 0.25%0.25\% of tokens have prefixes for which the two engines agree on every recorded route. Accordingly, we interpret the results of Figures 8 and 5 as evidence that routing disagreement is a major amplifier of the tail, rather than the sole source of the mismatch. A causal decomposition would require an intervention that explicitly replays identical routes across the two engines, which is outside the scope of the measurements reported here. A.5 The heavy tail propagates from δ to ε The identity in Section 3.1 maps the full perturbation vector δ into the sampled token’s log-odds displacement, εt=logitpt−logitqt. _t=logitp_t-logitq_t. We next verify that the architectural difference observed directly in δ remains visible after this mapping. Unlike Section A.2, which uses the 6,000-position logit dump, this analysis uses the full static sampled-token measurement: approximately 12.412.4M dense tokens and 12.312.3M MoE tokens on the corresponding base checkpoints. We compute εt _t whenever neither ptp_t nor qtq_t rounds exactly to one. Table 6: Log-odds displacement εt _t in the static measurement on the base checkpoints. Dense MoE Sampled tokens 12,389,11512,389,115 12,293,11512,293,115 Finite εt _t 11,934,87211,934,872 12,226,09212,226,092 Finite fraction 96.3%96.3\% 99.5%99.5\% sd(εt)sd( _t) 0.09970.0997 0.29470.2947 q95(|εt|)q_95(| _t|) 0.1650.165 0.5990.599 q99(|εt|)q_99(| _t|) 0.3360.336 1.0551.055 q99.9(|εt|)q_99.9(| _t|) 0.6930.693 2.0442.044 Pr(|εt|>1) (| _t|>1) 0.028%0.028\% 1.13%1.13\% Pr(|εt|>2) (| _t|>2) 00 0.106%0.106\% maxt|εt| _t| _t| 2.132.13 19.3819.38 As shown in Figure 7(b), the dense displacement decays rapidly, whereas the MoE retains a long tail. The 99.999.9th percentile of |εt|| _t| is 0.6930.693 on the dense model and 2.0442.044 on the MoE, and the maximum increases from 2.132.13 to 19.3819.38. Thus the heavy tail identified at the level of individual logit perturbations survives the nonlinear mapping to the log-odds displacement that enters the importance ratio. The dense CCDF contains several visible steps at high confidence. These arise from the discrete numerical structure of probabilities close to one and should not be interpreted as a continuous heavy tail. They do not affect the qualitative dense–MoE separation. A.6 The displacement is approximately invariant to token confidence The central design choice of CIS is to place a common upper threshold in the displacement coordinate rather than a common threshold on the importance ratio. This choice is motivated by the empirical observation that the scale of εt _t changes only mildly with token confidence, whereas the scale of the ratio deviation changes by orders of magnitude. We examine the static base-MoE measurement and retain tokens with 1−pt≥10−71-p_t≥ 10^-7. Figure 9(a) plots conditional quantiles of εt _t against token uncertainty 1−pt1-p_t. Figure 9(b) compares the median absolute deviation of εt _t with that of logkt k_t. Figure 9: (a) Median, 10–90% and 1–99% intervals of εt _t and (b) median absolute deviation of εt _t and logkt k_t against 1−pt1-p_t on the base MoE. Table 7: Conditional statistics of εt _t and logkt k_t on the base MoE. ptp_t range Tokens Med. ε MAD(ε)( ) q05(ε)q_05( ) q95(ε)q_95( ) MAD(logk)( k) pt<0.5p_t<0.5 2,824,0212,824,021 −0.0059-0.0059 0.1560.156 −0.375-0.375 0.2590.259 1.15×10−11.15×10^-1 0.50.5–0.90.9 2,337,5542,337,554 −0.0011-0.0011 0.1750.175 −0.376-0.376 0.3550.355 3.96×10−23.96×10^-2 0.90.9–0.990.99 1,951,8641,951,864 −0.0005-0.0005 0.1860.186 −0.453-0.453 0.4200.420 6.43×10−36.43×10^-3 0.990.99–0.9990.999 1,549,9151,549,915 −0.0019-0.0019 0.2030.203 −0.492-0.492 0.4720.472 6.78×10−46.78×10^-4 pt>0.999p_t>0.999 3,562,7383,562,738 0.00000.0000 0.2370.237 −0.504-0.504 0.5010.501 9.90×10−69.90×10^-6 Across 18 logarithmic bins spanning 1−pt≈1.5×10−61-p_t≈ 1.5× 10^-6 to 0.680.68, the conditional median of εt _t remains within 0.0060.006 of zero. Its median absolute deviation ranges from 0.1570.157 to 0.2880.288, a factor of only 1.81.8. The displacement distribution is therefore not exactly confidence-invariant, and in particular becomes moderately wider toward the highest-confidence region, but the variation is small relative to the ratio coordinate. Indeed, over the same bins, the median absolute deviation of logkt k_t changes by approximately 3.2×1053.2× 10^5. This contrast follows the mapping kt−1=(1−pt)(eεt−1):k_t-1=(1-p_t)(e _t-1): the same displacement is strongly compressed in ratio space when pt→1p_t→ 1. Consistently, on a random subsample of 300,000 tokens the Spearman correlation between εt _t and log(1−pt) (1-p_t) is only −0.028-0.028. Below the floor κ of Algorithm 1, εt _t is dominated by the storage resolution of the log-probabilities, so we repeat the comparison on the 7.67.6M tokens with 1−pt≥κ1-p_t≥κ, which is the region where CIS acts. There, the median absolute deviation of εt _t varies by a factor of 1.31.3 across 1818 logarithmic bins, whereas that of logkt k_t varies by a factor of 115115, and the Spearman correlation between |εt|| _t| and log(1−pt) (1-p_t) is −0.12-0.12. The same contrast holds between the SGLang and FSDP engines used for training. On the tokens with zero staleness in the training logs of the exact-ratio, TIS, and IcePop runs, the median absolute deviation of εt _t varies by a factor of 1.81.8 to 2.72.7 across confidence bins above the floor, against 150150 to 220220 for logkt k_t. These measurements support treating the displacement scale as approximately common across confidence levels, while making explicit that the approximation is not exact. A.7 The second moment on the trained model Figure 3 and Figure 5a use the MoE after RL training. We load the step-86 checkpoint of a CIS run on Qwen1.5-MoE-A2.7B into both engines of Section A.1 and score the 510,915510,915 sampled tokens of 4,0004,000 GSM8K rollouts, which are generated by the step-57 snapshot of the same run. The training engine scores them with teacher forcing and the inference engine by prefill, so the two engines score the same tokens with the same parameters. We compute εt _t whenever neither ptp_t nor qtq_t rounds to one, as in Section A.5. The moments of the displacement grow quickly with the order: [eεt]=7.07E[e _t]=7.07 and [e2εt]=9.2×106E[e^2 _t]=9.2× 10^6, and the weighted moments [((1−pt)eεt)α]E[((1-p_t)e _t)^α] that enter Theorem 3.2 are 0.160.16 at α=1α=1 and 22.922.9 at α=2α=2. The second moment is therefore determined by the upper tail of εt _t, which is the regime in which the truncated moment of CIS differs from exact correction. Appendix B Proofs and Technical Details for Section 3 B.1 Preliminaries and Key Lemmas In this subsection, we establish the minimal set of identities required for the proofs of Theorems 3.2 and 3.3. We adopt all notation and settings from Section 3. Throughout, pt,qt∈(0,1)p_t,q_t∈(0,1), kt:=pt/qtk_t:=p_t/q_t, and the n token contributions (kj,pj,ℓj′)j=1n(k_j,p_j, _j)_j=1^n are independent copies of (kt,pt,ℓt′)(k_t,p_t, _t). For a measurable correction rule f:(0,∞)×(0,1)→[0,∞)f:(0,∞)×(0,1)→[0,∞), define G^f:=1n∑j=1nf(kj,pj)ℓj′,G0:=[ktℓt′]. G_f:= 1n _j=1^nf(k_j,p_j)\, _j, G_0:=E[k_t _t]. (8) For λ≥0λ≥ 0, we write fλ(k,p):=mink, 1+λ(1−p).f_λ(k,p):= \k,\;1+λ(1-p) \. (9) Lemma B.1 (From logit perturbation to log-odds displacement). Fix the sampled token y=yty=y_t. Let pt=softmax(z)yp_t=softmax(z)_y and qt=softmax(z+δ)yq_t=softmax(z+δ)_y, and define wj:=ezj∑j′≠yezj′,j≠y,w_j:= e^z_j _j ≠ ye^z_j , j≠ y, (10) and εt:=−δy+log∑j≠ywjeδj. _t:=- _y+ _j≠ yw_je _j. (11) Then kt=ptqt=pt+(1−pt)eεt,kt−1=(1−pt)(eεt−1),k_t= p_tq_t=p_t+(1-p_t)e _t, k_t-1=(1-p_t) (e _t-1 ), (12) and εt=logpt1−pt−logqt1−qt. _t= p_t1-p_t- q_t1-q_t. (13) Proof. Substituting the two softmax probabilities directly into the ratio gives kt k_t =ezy/(ezy+∑j≠yezj)ezy+δy/(ezy+δy+∑j≠yezj+δj) = e^z_y/ (e^z_y+ _j≠ ye^z_j )e^z_y+ _y/ (e^z_y+ _y+ _j≠ ye^z_j+ _j ) =(a)ezyezy+∑j≠yezj(1+e−δy∑j≠yezj+δjezy) (a)= e^z_ye^z_y+ _j≠ ye^z_j (1+e^- _y _j≠ ye^z_j+ _je^z_y ) =(b)pt+e−δy∑j≠yezj+δjezy+∑j≠yezj (b)=p_t+ e^- _y _j≠ ye^z_j+ _je^z_y+ _j≠ ye^z_j =(c)pt+∑j≠yezjezy+∑j≠yezje−δy∑j≠yezj∑j′≠yezj′eδj (c)=p_t+ _j≠ ye^z_je^z_y+ _j≠ ye^z_je^- _y _j≠ y e^z_j _j ≠ ye^z_j e _j =(d)pt+(1−pt)e−δy∑j≠ywjeδj (d)=p_t+(1-p_t)e^- _y _j≠ yw_je _j =(e)pt+(1−pt)eεt, (e)=p_t+(1-p_t)e _t, where (a) expands the reciprocal of the inference probability, (b) uses the definition of ptp_t, (c) multiplies and divides by ∑j≠yezj _j≠ ye^z_j, (d) uses 1−pt=∑j≠yezjezy+∑j≠yezj1-p_t= _j≠ ye^z_je^z_y+ _j≠ ye^z_j together with (10), and (e) substitutes (11). Subtracting 1=pt+(1−pt)1=p_t+(1-p_t) immediately gives kt−1 k_t-1 =pt+(1−pt)eεt−pt−(1−pt) =p_t+(1-p_t)e _t-p_t-(1-p_t) =(1−pt)(eεt−1). =(1-p_t) (e _t-1 ). To obtain the log-odds form, note that qt1−qt q_t1-q_t =ezy+δy∑j≠yezj+δj = e^z_y+ _y _j≠ ye^z_j+ _j =(f)ezy∑j≠yezjeδy∑j≠ywjeδj (f)= e^z_y _j≠ ye^z_j e _y _j≠ yw_je _j =(g)pt1−pteδy∑j≠ywjeδj, (g)= p_t1-p_t e _y _j≠ yw_je _j, where (f) factors ∑j≠yezj+δj _j≠ ye^z_j+ _j using (10), and (g) uses pt/(1−pt)=ezy/∑j≠yezjp_t/(1-p_t)=e^z_y/ _j≠ ye^z_j. Taking logarithms and rearranging yields logpt1−pt−logqt1−qt p_t1-p_t- q_t1-q_t =−δy+log∑j≠ywjeδj =- _y+ _j≠ yw_je _j =εt, = _t, which proves (13). ∎ Lemma B.2 (Displacement-coordinate truncation identities). Let p∈(0,1)p∈(0,1) and suppose k=p+(1−p)eεk=p+(1-p)e . Then ε≤0⟹p<k≤1,ε>0⟹k>1. ≤ 0 p<k≤ 1, >0 k>1. (14) Moreover, for every λ≥0λ≥ 0, fλ(k,p)=p+(1−p)mineε, 1+λ,f_λ(k,p)=p+(1-p) \e ,\,1+λ \, (15) and k−fλ(k,p)=(1−p)(eε−1−λ)+.k-f_λ(k,p)=(1-p) (e -1-λ )_+. (16) Finally, for any fixed ratio cap C≥1C≥ 1, k≤C⟺eε≤1+C−11−p.k≤ C e ≤ 1+ C-11-p. (17) Proof. If ε≤0 ≤ 0, then eε≤1e ≤ 1, so k k =p+(1−p)eε =p+(1-p)e ≤p+(1−p)=1. ≤ p+(1-p)=1. At the same time, k−p=(1−p)eε>0k-p=(1-p)e >0, and hence p<k≤1p<k≤ 1. If ε>0 >0, then k−1=(1−p)(eε−1)>0,k-1=(1-p)(e -1)>0, which gives k>1k>1. For the clipped ratio, fλ(k,p) f_λ(k,p) =(a)minp+(1−p)eε, 1+λ(1−p) (a)= \p+(1-p)e ,\;1+λ(1-p) \ =(b)minp+(1−p)eε,p+(1−p)(1+λ) (b)= \p+(1-p)e ,\;p+(1-p)(1+λ) \ =(c)p+(1−p)mineε,1+λ, (c)=p+(1-p) \e ,1+λ \, where (a) substitutes the ratio–displacement identity, (b) rewrites 1+λ(1−p)=p+(1−p)(1+λ)1+λ(1-p)=p+(1-p)(1+λ), and (c) uses that x↦p+(1−p)x p+(1-p)x is strictly increasing. Therefore k−fλ(k,p) k-f_λ(k,p) =(1−p)(eε−mineε,1+λ) =(1-p) (e - \e ,1+λ\ ) =(1−p)(eε−1−λ)+, =(1-p)(e -1-λ)_+, using the pointwise identity x−minx,c=(x−c)+x- \x,c\=(x-c)_+. For a fixed cap C≥1C≥ 1, k≤C k≤ C ⟺p+(1−p)eε≤C p+(1-p)e ≤ C ⟺eε≤C−p1−p e ≤ C-p1-p ⟺eε≤1+C−11−p, e ≤ 1+ C-11-p, where the last equality follows from C−p=(1−p)+(C−1)C-p=(1-p)+(C-1). ∎ Lemma B.3 (Bias–variance decomposition). Let f:(0,∞)×(0,1)→[0,∞)f:(0,∞)×(0,1)→[0,∞) be measurable with [f(kt,pt)2‖ℓt′‖22]<∞E[f(k_t,p_t)^2\| _t\|_2^2]<∞, and abbreviate ft:=f(kt,pt)f_t:=f(k_t,p_t). Then ‖G^f−G0‖22 \| G_f-G_0 \|_2^2 =‖[(ft−kt)ℓt′]‖22+1n‖ftℓt′−[ftℓt′]‖22 = \|E [(f_t-k_t) _t ] \|_2^2+ 1nE \|f_t _t-E[f_t _t] \|_2^2 (18) ≤‖[(ft−kt)ℓt′]‖22+1n[ft2‖ℓt′‖22]. ≤ \|E [(f_t-k_t) _t ] \|_2^2+ 1nE [f_t^2\| _t\|_2^2 ]. (19) Proof. Write Xj:=f(kj,pj)ℓj′X_j:=f(k_j,p_j) _j and Gf:=[Xj]G_f:=E[X_j]. Then ‖G^f−G0‖22 \| G_f-G_0 \|_2^2 =(a)‖(G^f−Gf)+(Gf−G0)‖22 (a)=E \|( G_f-G_f)+(G_f-G_0) \|_2^2 =(b)‖Gf−G0‖22+‖G^f−Gf‖22 (b)=\|G_f-G_0\|_2^2+E \| G_f-G_f \|_2^2 =(c)‖Gf−G0‖22+1n2∑i,j=1n⟨Xi−Gf,Xj−Gf⟩ (c)=\|G_f-G_0\|_2^2+ 1n^2 _i,j=1^nE X_i-G_f,\;X_j-G_f =(d)‖Gf−G0‖22+1n‖X1−Gf‖22 (d)=\|G_f-G_0\|_2^2+ 1nE\|X_1-G_f\|_2^2 =(e)‖[(ft−kt)ℓt′]‖22+1n‖ftℓt′−[ftℓt′]‖22. (e)= \|E[(f_t-k_t) _t] \|_2^2+ 1nE \|f_t _t-E[f_t _t] \|_2^2. Here (a) inserts and subtracts GfG_f. The cross term vanishes in (b) because [G^f]=GfE[ G_f]=G_f. Step (c) expands G^f−Gf=n−1∑j(Xj−Gf) G_f-G_f=n^-1 _j(X_j-G_f), while (d) uses independence: for i≠ji≠ j, ⟨Xi−Gf,Xj−Gf⟩=0,E X_i-G_f,X_j-G_f =0, and the n diagonal terms are identical. Finally, Gf−G0=[ftℓt′]−[ktℓt′]=[(ft−kt)ℓt′],G_f-G_0=E[f_t _t]-E[k_t _t]=E[(f_t-k_t) _t], which gives (e) and proves (18). For the variance term, 1n‖X1−Gf‖22 1nE\|X_1-G_f\|_2^2 =1n(‖X1‖22−‖Gf‖22) = 1n (E\|X_1\|_2^2-\|G_f\|_2^2 ) ≤1n‖X1‖22 ≤ 1nE\|X_1\|_2^2 =1n[ft2‖ℓt′‖22], = 1nE [f_t^2\| _t\|_2^2 ], where the inequality simply drops the nonpositive term −∥Gf∥22/n-\|G_f\|_2^2/n. This gives (19). ∎ Remark (correlated token contributions). Independence enters only when the off-diagonal covariance terms are removed. If the contributions are identically distributed but correlated, define ρ¯f:=1n(n−1)∑i≠j⟨Xi−Gf,Xj−Gf⟩‖X1−Gf‖22. ρ_f:= 1n(n-1) _i≠ jE X_i-G_f,\;X_j-G_f E\|X_1-G_f\|_2^2. (20) If ρ¯f≤ρ¯ ρ_f≤ ρ, then 1n2∑i,j=1n⟨Xi−Gf,Xj−Gf⟩ 1n^2 _i,j=1^nE X_i-G_f,\;X_j-G_f =1n2(n‖X1−Gf‖22+∑i≠j⟨Xi−Gf,Xj−Gf⟩) = 1n^2 (nE\|X_1-G_f\|_2^2+ _i≠ jE X_i-G_f,X_j-G_f ) =1+(n−1)ρ¯fn‖X1−Gf‖22 = 1+(n-1) ρ_fnE\|X_1-G_f\|_2^2 ≤1neff‖X1−Gf‖22, ≤ 1n_effE\|X_1-G_f\|_2^2, where neff:=n1+(n−1)ρ¯.n_eff:= n1+(n-1) ρ. (21) Hence the variance bounds below remain valid with n replaced by neffn_eff. Appendix D reports the corresponding empirical dependence among the token contributions. B.2 Proof of Theorem 3.2 We first state the formal version of Theorem 3.2. Theorem B.4 (Formal version of Theorem 3.2). Under the conventions of Section B.1, assume ‖ℓt′‖2≤b\| _t\|_2≤ b. Then ‖G^−G0‖22≤b2n[pt2+2pt(1−pt)eεt+(1−pt)2e2εt],E \| G-G_0 \|_2^2≤ b^2nE [p_t^2+2p_t(1-p_t)e _t+(1-p_t)^2e^2 _t ], (22) where G^:=n−1∑j=1nkjℓj′ G:=n^-1 _j=1^nk_j _j. Proof of Theorem 3.2. By Lemma B.1, the right-hand side of (22) equals b2n[kt2] b^2nE[k_t^2]. If [kt2]=∞E[k_t^2]=∞, there is nothing to prove. Assume therefore that [kt2]<∞E[k_t^2]<∞. Since ‖ℓt′‖2≤b\| _t\|_2≤ b, [kt2‖ℓt′‖22] [k_t^2\| _t\|_2^2 ] ≤b2[kt2] ≤ b^2E [k_t^2 ] ∞, ∞, so Lemma B.3 applies with ft=ktf_t=k_t. Because exact correction has no truncation bias, ‖G^−G0‖22 \| G-G_0 \|_2^2 =1n‖ktℓt′−[ktℓt′]‖22 = 1nE \|k_t _t-E[k_t _t] \|_2^2 ≤(a)1n[kt2‖ℓt′‖22] (a)≤ 1nE [k_t^2\| _t\|_2^2 ] ≤(b)b2n[kt2] (b)≤ b^2nE[k_t^2] =(c)b2n[(pt+(1−pt)eεt)2] (c)= b^2nE [ (p_t+(1-p_t)e _t )^2 ] =b2n[pt2+2pt(1−pt)eεt+(1−pt)2e2εt]. = b^2nE [p_t^2+2p_t(1-p_t)e _t+(1-p_t)^2e^2 _t ]. Here (a) drops the nonpositive centered-mean term in the variance, (b) uses ‖ℓt′‖2≤b\| _t\|_2≤ b, and (c) applies Lemma B.1. ∎ Remark (where the instability enters). The proof imposes no structural assumption on the law of εt _t beyond the moments that appear explicitly in the bound. Exact correction has zero truncation bias, but its variance depends on the untruncated second-moment term [(1−pt)2e2εt]≤[e2εt]E[(1-p_t)^2e^2 _t] [e^2 _t]. Remark (correlated token contributions). Under correlated token contributions, the same argument holds with n replaced by neffn_eff from (21). The comparison below between the untruncated and truncated second moments is otherwise unchanged. B.3 Proof of Theorem 3.3 We first state the formal version of Theorem 3.3. Throughout this subsection, let mt:=mineεt,1+λm_t:= \e _t,1+λ\. Theorem B.5 (Formal version of Theorem 3.3). Under the conventions of Section B.1, assume ‖ℓt′‖2≤b\| _t\|_2≤ b. Then for every finite λ≥0λ≥ 0, ‖G^fλ−G0‖22 \| G_f_λ-G_0 \|_2^2 ≤b2([(1−pt)(eεt−1−λ)+])2 ≤ b^2 (E [(1-p_t)(e _t-1-λ)_+ ] )^2 +b2n[(pt+(1−pt)mineεt,1+λ)2]. + b^2nE [ (p_t+(1-p_t) \e _t,1+λ\ )^2 ]. (23) Proof of Theorem 3.3. Since fλ(kt,pt)≤1+λ(1−pt)≤1+λ,f_λ(k_t,p_t)≤ 1+λ(1-p_t)≤ 1+λ, we have [fλ(kt,pt)2‖ℓt′‖22]≤b2(1+λ)2<∞E[f_λ(k_t,p_t)^2\| _t\|_2^2]≤ b^2(1+λ)^2<∞, so Lemma B.3 applies: ‖G^fλ−G0‖22 \| G_f_λ-G_0 \|_2^2 ≤‖[(fλ(kt,pt)−kt)ℓt′]‖22 ≤ \|E[(f_λ(k_t,p_t)-k_t) _t] \|_2^2 +1n[fλ(kt,pt)2‖ℓt′‖22]. + 1nE [f_λ(k_t,p_t)^2\| _t\|_2^2 ]. For the bias term, fλ(kt,pt)≤ktf_λ(k_t,p_t)≤ k_t, and hence ‖[(fλ(kt,pt)−kt)ℓt′]‖2 \|E[(f_λ(k_t,p_t)-k_t) _t] \|_2 ≤[(kt−fλ(kt,pt))‖ℓt′‖2] [ (k_t-f_λ(k_t,p_t) )\| _t\|_2 ] ≤b[kt−fλ(kt,pt)] ≤ b\,E [k_t-f_λ(k_t,p_t) ] =(a)b[(1−pt)(eεt−1−λ)+], (a)=b\,E [(1-p_t)(e _t-1-λ)_+ ], where (a) uses (16). For the variance term, using (15), [fλ(kt,pt)2‖ℓt′‖22] [f_λ(k_t,p_t)^2\| _t\|_2^2 ] ≤b2[fλ(kt,pt)2] ≤ b^2E [f_λ(k_t,p_t)^2 ] =(b)b2[(pt+(1−pt)mt)2], (b)=b^2E [ (p_t+(1-p_t)m_t )^2 ], where (b) substitutes the displacement-coordinate form of the clipped ratio. Combining the two bounds gives (23). ∎ Remark (comparison with exact correction). The difference between Theorem B.4 and Theorem B.5 is the replacement of the untruncated displacement factor eεte _t by its truncated counterpart mtm_t, together with the corresponding truncation bias. Pointwise, min(eεt,1+λ)2 (e _t,1+λ )^2 ≤e2εt, ≤ e^2 _t, min(eεt,1+λ)2 (e _t,1+λ )^2 ≤(1+λ)2, ≤(1+λ)^2, and therefore [min(eεt,1+λ)2]≤min[e2εt],(1+λ)2.E [ (e _t,1+λ)^2 ]≤ \E[e^2 _t],\;(1+λ)^2 \. (24) Moreover, pt+(1−pt)mt≤minkt, 1+λp_t+(1-p_t)m_t≤ \k_t,\,1+λ\, so the variance term of (23) satisfies [(pt+(1−pt)mt)2]≤min[kt2],(1+λ)2E[(p_t+(1-p_t)m_t)^2]≤ \E[k_t^2],\,(1+λ)^2\. Thus, regardless of how heavy the positive tail of εt _t becomes, the second-moment term entering the CIS variance bound is at most (1+λ)2(1+λ)^2. The corresponding bias remains controlled by the first moment of the truncated excess, since (1−pt)(eεt−1−λ)+ (1-p_t)(e _t-1-λ)_+ ≤(eεt−1−λ)+ ≤(e _t-1-λ)_+ ≤eεteεt>1+λ ≤ e _t1\e _t>1+λ\ ≤eεt. ≤ e _t. CIS therefore trades a finite upper-tail bias for a uniformly bounded second-moment contribution, which is the bias–variance trade-off described in Section 3.3. Remark (correlated token contributions). The same result holds with n replaced by neffn_eff under the correlated-token setting above. The truncation term and the bound (24) are pointwise and therefore do not depend on the effective sample size. B.4 Numerical Floor in Algorithm 1 The theoretical operator in (9) uses the uncertainty 1−pt1-p_t. Algorithm 1 replaces it by ϕκ(p):=max1−p,κ _κ(p):= \1-p,κ\, where 0<κ≤10<κ≤ 1, and applies fλ,κ(k,p):=mink, 1+λϕκ(p).f_λ,κ(k,p):= \k,\;1+λ _κ(p) \. (25) This modification prevents the confidence-dependent cap from becoming narrower than the numerical resolution of the stored log-probabilities. Corollary B.6 (Effect of the numerical floor). For every λ≥0λ≥ 0 and κ∈(0,1]κ∈(0,1], fλ(k,p)≤fλ,κ(k,p)≤k,f_λ(k,p)≤ f_λ,κ(k,p)≤ k, (26) and consequently 0≤k−fλ,κ(k,p)≤k−fλ(k,p).0≤ k-f_λ,κ(k,p)≤ k-f_λ(k,p). (27) Moreover, fλ,κ(k,p)≤1+λ.f_λ,κ(k,p)≤ 1+λ. (28) Hence, if ‖ℓt′‖2≤b\| _t\|_2≤ b, [fλ,κ(kt,pt)2‖ℓt′‖22]≤b2(1+λ)2.E [f_λ,κ(k_t,p_t)^2\| _t\|_2^2 ]≤ b^2(1+λ)^2. (29) Proof. By definition, ϕκ(p)=max1−p,κ≥1−p _κ(p)= \1-p,κ\≥ 1-p. Since λ≥0λ≥ 0, 1+λϕκ(p)≥1+λ(1−p).1+λ _κ(p)≥ 1+λ(1-p). The map c↦mink,cc \k,c\ is nondecreasing, so fλ,κ(k,p) f_λ,κ(k,p) =mink,1+λϕκ(p) = \k,1+λ _κ(p)\ ≥mink,1+λ(1−p) ≥ \k,1+λ(1-p)\ =fλ(k,p). =f_λ(k,p). The definition of the minimum also gives fλ,κ(k,p)≤kf_λ,κ(k,p)≤ k, which proves (26). Subtracting both sides from k yields (27). Since both 1−p1-p and κ lie in (0,1](0,1], ϕκ(p)≤1 _κ(p)≤ 1. Therefore fλ,κ(k,p) f_λ,κ(k,p) ≤1+λϕκ(p) ≤ 1+λ _κ(p) ≤1+λ, ≤ 1+λ, which gives (28). If ‖ℓt′‖2≤b\| _t\|_2≤ b, it follows immediately that [fλ,κ(kt,pt)2‖ℓt′‖22] [f_λ,κ(k_t,p_t)^2\| _t\|_2^2 ] ≤b2[fλ,κ(kt,pt)2] ≤ b^2E [f_λ,κ(k_t,p_t)^2 ] ≤b2(1+λ)2, ≤ b^2(1+λ)^2, proving (29). ∎ Remark. The floor raises the cap only at the extreme high-confidence end, so fλ,κf_λ,κ removes no more of the exact importance weight than the idealized operator fλf_λ. At the same time, (28) keeps the applied weight uniformly bounded by 1+λ1+λ. The numerical safeguard in Algorithm 1 therefore preserves the central variance-control property of CIS while preventing the theoretical band from becoming narrower than the storage resolution. Appendix C Experimental Details C.1 Implementation and Hardware Recipe. Every arm uses the same recipe: learning rate 3×10−63× 10^-6, weight decay 0.010.01, PPO clip range (0.2,0.28)(0.2,0.28), no KL penalty, four samples per prompt at temperature one, at most 1,024 generated tokens, at most two versions of rollout staleness, three epochs over the 7,473 GSM8K training problems (87 optimizer steps), and one node with eight A100-80GB GPUs per arm, four serving SGLang and four training with FSDP and flash attention. Operators. Training uses AReaL v2.0.0 with its decoupled loss, in which the rollout-side and training-side log-probabilities are both available per token. All operators are implemented in the same token-level branch that AReaL uses for TIS and IcePop, so that every arm differs only in the weight on the token loss; CIS multiplies the loss by the weight minkt,1+λφt \k_t,1+λ _t\ of Algorithm 1, with φt=max(1−pt,κ) _t= (1-p_t,κ) and ptp_t taken from the training-side log-probability, and the weight carries no gradient. The two-sided band of Table 2 additionally lifts the displacement from below to eεt≥1/2e _t≥ 1/2, which maps through Equation 2 to the weight maxminkt,1+λφt,(1+pt)/2 \ \k_t,1+λ _t\,\,(1+p_t)/2\; CIS applies no lower threshold. Table 8 lists the operator and the hyperparameters of every arm. Beyond the operators, the changes to the framework are longer worker-readiness timeouts, a numerically corrected binary-KL metric for KPop†, product weights over a response for Seq-TIS and Seq-MIS, static loss scaling for the fp16 arm, and a token-level dump hook. The training engine must use flash attention; with packed sequences the scaled-dot-product fallback attends across sequence boundaries and corrupts the training-side log-probabilities, which we detected as a median logkt k_t far from zero on every method. Weights are sent from the trainer to the rollout engine by collective communication after every optimizer step. On Qwen1.5-MoE each arm takes about 1.0 to 1.4 hours for the 87 steps, and Appendix D.4 breaks this time down by phase. DeepSeek-V2-Lite (DeepSeek-AI, 2024) and Qwen3-30B-A3B (Qwen Team, 2025) use the same recipe with vLLM rollouts, weights reloaded from disk after every step, and optimizer states in bf16; the thinking mode of Qwen3-30B-A3B is disabled in both training and evaluation. Table 8: Operators and hyperparameters of the arms in Table 1. K=∏tktK= _tk_t is the product over a response and φt=max(1−pt,κ) _t= (1-p_t,κ). Method Weight on the token loss Hyperparameters No correction 11 (decoupled loss disabled) — Exact ratio ktk_t clamp to [10−6,106][10^-6,10^6] TIS min(kt,C) (k_t,C) C=2C=2 IcePop kt 1kt∈[α,β]k_t\,1\k_t∈[α,β]\ [α,β]=[0.5,5][α,β]=[0.5,5] KPop† kt 1max(KLB(p∥q),KLB(q∥p))≤ϕk_t\,1\ (KL_B(p\|q),KL_B(q\|p))≤φ\, float64 ϕ=2φ=2 Seq-TIS min(K,C) (K,C) for every token of the response C=2C=2 Seq-MIS K 1K∈[α,β]K\,1\K∈[α,β]\ for every token of the response [α,β]=[0.5,2][α,β]=[0.5,2] GSPO sequence-level surrogate, ktk_t uncorrected clip (3×10−4,4×10−4)(3× 10^-4,4× 10^-4) FP16 11, rollout and training in fp16 loss scale 10241024 CIS (ours) minkt,1+λφt \k_t,1+λ _t\ λ=2.3λ=2.3, κ=5×10−3κ=5× 10^-3 C.2 Datasets and Evaluation Protocol Training data. Every RL run trains on the 7,473 problems of the GSM8K training split (Cobbe et al., 2021). The prompt is the problem followed by an instruction to put the final answer in (Appendix C.3), and the reward is one when math-verify judges the final answer equal to the reference answer and zero otherwise. The framework evaluates the GSM8K test split once per epoch for logging; these numbers are not used to select a checkpoint, and every number that we report comes from the final checkpoint. Measurement data. The static measurement of the mismatch uses 2,500 problems drawn at random from DAPO-Math-17k (Yu et al., 2025). This set is not used for training. Protocols. Table 9 lists the evaluation benchmarks. Tables 1 and 2 use the chat template of each model with the instruction to put the final answer in , greedy decoding with vLLM, at most 1,536 new tokens, and the math-verify judge. The out-of-domain evaluation of Table 13 uses the same template and greedy decoding; a multiple-choice item is scored by the option letter that the response states, with at most 512 new tokens, and a code item is scored by running the completion against the benchmark tests with a limit of 15 seconds, with at most 768 new tokens. Changing the tensor-parallel degree of the evaluation engine changes a few greedy decisions (two HumanEval and six MBPP items on one checkpoint), so every checkpoint of a table is evaluated with the same degree. Table 9: Evaluation benchmarks. Benchmark Domain Split Scoring Items GSM8K (Cobbe et al., 2021) math test math-verify 1,319 MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024) math test subset math-verify 500 SVAMP (Patel et al., 2021) math test math-verify 300 Minerva-Math (Lewkowycz et al., 2022) math test math-verify 272 OlympiadBench (He et al., 2024) math English, text-only, open-ended math-verify 674 MMLU-STEM (Hendrycks et al., 2020) knowledge test, 19 STEM subjects option letter 3,153 ARC-Challenge (Clark et al., 2018) knowledge test option letter 1,172 OpenBookQA (Mihaylov et al., 2018) knowledge test option letter 500 HumanEval (Chen et al., 2021) code test unit tests 164 MBPP (Austin et al., 2021) code test unit tests 500 C.3 Prompts Every prompt below is the user message, formatted with the chat template of the model and no additional system message; the thinking mode of Qwen3-30B-A3B is disabled. Placeholders in braces are filled with the fields of each item. Prompt: RL training on GSM8K question Please put your final answer within . Prompt: evaluation on the five math benchmarks (Table 1) question Please reason step by step, and put your final answer within . Prompt: out-of-domain multiple choice (MMLU-STEM, ARC-Challenge, OpenBookQA) question A. option A B. option B C. option C D. option D Answer with the letter of the correct option. Put the single letter within . Prompt: out-of-domain code (HumanEval) Complete the following Python function. ```python function signature and docstring ``` Return the complete function in a single ```python code block. Do not include example usage or tests. Prompt: out-of-domain code (MBPP) problem description Your code must pass these tests: ```python three assert statements ``` Return the complete function in a single ```python code block. Do not include example usage or tests. Prompt: measurement of the mismatch (DAPO-Math-17k) problem C.4 Diagnostics Static measurement. Prompts are 2,500 problems drawn at random from DAPO-Math-17k. Generation uses vLLM 0.26 in bf16 with eager execution and without prefix caching; each prompt receives eight samples at temperature one and top-p one, at most 1,024 tokens per response, and the log-probability of every sampled token is taken from the sampling step itself rather than from a later prefill pass. Rescoring uses HuggingFace Transformers in bf16 with teacher forcing on the exact sampled sequence and the same parameter snapshot; the token at position t is scored from the logits at position t−1t-1 on both sides. For the MoE model we record the top-4 expert identities of every layer on both sides and the gate margin between the fourth and fifth expert; the routing read from forward hooks agrees with the identities returned by the fused kernel on 100%100\% of the (token, layer) pairs whose margin is positive. Two controls fix the measurement floor: rescoring a sequence twice on the training engine returns exactly zero difference, and the empirical mean of ktk_t is 1.00001.0000 on both architectures, which verifies the per-step calibration identity [kt∣ht]=1E[k_t h_t]=1. Two further controls accompany the main measurement: rescoring in fp32, and rescoring by a prefill pass on the inference engine, which reproduces the decode-time bulk. The displacement is computed as εt=logitpt−logitqt _t=logitp_t-logitq_t from the two log-probabilities. Training-time logging. The training runs dump, every ten optimizer calls, the rollout-side log-probability, the training-side log-probability at the start of the update, and the parameter version of the rollout for a sample of tokens. Version staleness is the difference between the current version and the rollout version. C.5 Preregistered Evaluation Contract Held-out accuracy at the final checkpoint is the arbiter of every comparison; training reward is reported but is not used to choose between methods. The following choices were fixed before the corresponding runs. The floor is κ=5×10−3κ=5× 10^-3, set from the storage resolution of the log-probabilities before any training run. The slope grid is λ∈1.6,2.3,3.1λ∈\1.6,2.3,3.1\. All arms use the recipe of Section 4.1 without per-arm tuning. Every ablation changes one element of the operator relative to the deployed configuration. The main comparison and the ablations of Table 2 use three seeds. Appendix D Additional Results and Limitations D.1 Training Dynamics (a) (b) (c) (d) (e) (f) Figure 10: Training dynamics on Qwen1.5-MoE-A2.7B (mean over seeds, band spanning the seeds, five-step smoothing). (a) Reward. (b) Gradient norm. (c) Entropy. (d) Response length. (e) Fraction of tokens acted on, counting whole responses for Seq-TIS and Seq-MIS. (f) Largest |logkt|| k_t| in the batch. Table 10: Training statistics on Qwen1.5-MoE-A2.7B, mean ± standard deviation over three seeds: third-epoch mean (median for the gradient norm and maxt|logkt| _t| k_t|), and peak gradient norm and fraction acted on over the whole run. Method Reward Grad. norm Peak grad. Entropy Length Acted on (%) maxt|logkt| _t| k_t| No correction 0.732±0.0040.732_± 0.004 0.75±0.020.75_± 0.02 2.5±0.42.5_± 0.4 0.220±0.0050.220_± 0.005 322±2322_± 2 0.00±0.000.00_± 0.00 9.5±0.59.5_± 0.5 Exact ratio 0.829±0.0120.829_± 0.012 3.58±0.753.58_± 0.75 69.5±41.469.5_± 41.4 0.102±0.0130.102_± 0.013 227±4227_± 4 0.06±0.030.06_± 0.03 25.9±3.425.9_± 3.4 KPop† 0.847±0.0160.847_± 0.016 3.09±0.763.09_± 0.76 29.2±23.529.2_± 23.5 0.093±0.0330.093_± 0.033 245±29245_± 29 1.33±0.051.33_± 0.05 27.3±1.727.3_± 1.7 IcePop 0.859±0.0050.859_± 0.005 1.12±0.041.12_± 0.04 2.8±1.52.8_± 1.5 0.113±0.0350.113_± 0.035 224±7224_± 7 3.64±0.403.64_± 0.40 24.0±2.324.0_± 2.3 TIS (C=2C=2) 0.861±0.0140.861_± 0.014 0.97±0.110.97_± 0.11 3.6±1.63.6_± 1.6 0.064±0.0040.064_± 0.004 226±9226_± 9 0.76±0.070.76_± 0.07 37.7±4.437.7_± 4.4 TIS (C=1.13C=1.13) 0.780±0.0650.780_± 0.065 0.91±0.080.91_± 0.08 8.5±11.78.5_± 11.7 0.200±0.1420.200_± 0.142 224±23224_± 23 4.32±0.644.32_± 0.64 29.7±4.429.7_± 4.4 Seq-TIS 0.803±0.0040.803_± 0.004 0.25±0.020.25_± 0.02 0.8±0.00.8_± 0.0 0.147±0.0160.147_± 0.016 276±7276_± 7 13.54±2.8313.54_± 2.83 15.6±2.915.6_± 2.9 Seq-MIS 0.759±0.0020.759_± 0.002 0.15±0.010.15_± 0.01 0.6±0.20.6_± 0.2 0.158±0.0250.158_± 0.025 292±9292_± 9 94.56±0.5094.56_± 0.50 18.2±1.718.2_± 1.7 FP16 0.740±0.0020.740_± 0.002 0.74±0.010.74_± 0.01 3.2±1.33.2_± 1.3 0.202±0.0020.202_± 0.002 313±3313_± 3 0.00±0.000.00_± 0.00 12.7±2.112.7_± 2.1 CIS (ours) 0.867±0.0090.867_± 0.009 1.47±0.311.47_± 0.31 3.0±1.13.0_± 1.1 0.071±0.0170.071_± 0.017 232±8232_± 8 3.81±0.423.81_± 0.42 26.9±5.126.9_± 5.1 Figure 10 and Table 10 show how the arms of Table 1 train on Qwen1.5-MoE. The token-level operators that bound the ratio (TIS, IcePop, and CIS) reach a training reward between 0.8590.859 and 0.8670.867 with a peak gradient norm below four, whereas the exact ratio and KPop† reach a lower reward with peak gradient norms of 7070 and 2929. The uncorrected run and the fp16 run stay near a reward of 0.740.74, and their entropy and response length stay close to their initial values. Seq-MIS removes 95%95\% of the responses, and its gradient norm is the smallest of all arms because most of the batch carries no gradient. The largest |logkt|| k_t| in the batch stays near 1010 for the uncorrected and fp16 runs and grows to between 1616 and 3838 for the other arms as the policy sharpens. The fraction of tokens that CIS acts on rises during the first third of training and then stays between 4%4\% and 5%5\% (panel e). D.2 Hyperparameter Sweeps Tables 11 and 12 list the per-benchmark accuracy behind Figure 6. Each sweep varies one hyperparameter of CIS on Qwen1.5-MoE-A2.7B with the other fixed at its default, and uses the recipe and evaluation protocol of Table 1; the default rows are the CIS entries of Table 1. Setting λ=0λ=0 truncates every ratio above one, which is TIS with C=1C=1, and the average accuracy collapses to 8.758.75 with a large spread across seeds. Every positive threshold stays above the uncorrected run (30.9930.99), and the default λ=2.3λ=2.3 gives the highest average (34.7834.78), while the other positive thresholds lie between 32.5332.53 and 34.0834.08. Table 11: Accuracy (%) of CIS on Qwen1.5-MoE-A2.7B for different thresholds λ with κ=5×10−3κ=5× 10^-3. Mean ± standard deviation over three seeds. λ GSM8K MATH500 SVAMP Minerva Olympiad Avg 00 14.71±13.4714.71_± 13.47 6.80±5.676.80_± 5.67 19.33±17.1719.33_± 17.17 0.98±1.390.98_± 1.39 1.93±1.121.93_± 1.12 8.75±7.448.75_± 7.44 0.450.45 64.82±1.8864.82_± 1.88 23.33±2.4423.33_± 2.44 70.33±2.0870.33_± 2.08 7.11±0.217.11_± 0.21 3.66±0.173.66_± 0.17 33.85±0.2733.85_± 0.27 0.890.89 66.59±1.9566.59_± 1.95 22.87±1.7522.87_± 1.75 70.22±1.9070.22_± 1.90 6.25±1.336.25_± 1.33 4.45±0.834.45_± 0.83 34.08±0.1934.08_± 0.19 1.61.6 65.68±0.4665.68_± 0.46 23.33±1.6323.33_± 1.63 68.78±2.1768.78_± 2.17 6.74±0.566.74_± 0.56 4.90±0.594.90_± 0.59 33.89±0.4633.89_± 0.46 2.32.3 (default) 67.88±1.6467.88_± 1.64 24.53±1.0324.53_± 1.03 70.22±1.3570.22_± 1.35 5.88±0.645.88_± 0.64 5.39±0.625.39_± 0.62 34.78±0.2534.78_± 0.25 3.13.1 65.07±0.7065.07_± 0.70 23.67±0.9223.67_± 0.92 67.89±2.5067.89_± 2.50 6.13±1.496.13_± 1.49 4.01±0.904.01_± 0.90 33.35±0.6633.35_± 0.66 4.64.6 64.54±1.1964.54_± 1.19 24.20±0.5324.20_± 0.53 69.22±1.2669.22_± 1.26 6.86±0.216.86_± 0.21 3.46±0.093.46_± 0.09 33.66±0.6433.66_± 0.64 9.29.2 62.27±1.6762.27_± 1.67 23.40±0.6023.40_± 0.60 66.89±1.8466.89_± 1.84 6.74±0.216.74_± 0.21 3.36±0.093.36_± 0.09 32.53±0.8732.53_± 0.87 For the floor, the accuracy peaks near the storage resolution of the log-probabilities, where κ=5×10−3κ=5× 10^-3 and κ=2×10−2κ=2× 10^-2 reach 34.7834.78 and 34.3934.39; a smaller floor of 10−310^-3 loses 1.41.4 points, and removing the floor or raising it to 0.10.1 lowers the average to 29.0229.02 and 32.1732.17. The gains of the default floor are largest on GSM8K and OlympiadBench. Table 12: Accuracy (%) of CIS on Qwen1.5-MoE-A2.7B for different floors κ with λ=2.3λ=2.3. Mean ± standard deviation over three seeds. κ GSM8K MATH500 SVAMP Minerva Olympiad Avg 00 55.62±2.6955.62_± 2.69 20.87±1.1020.87_± 1.10 59.78±2.8359.78_± 2.83 5.88±0.375.88_± 0.37 2.97±0.152.97_± 0.15 29.02±1.4329.02_± 1.43 10−310^-3 64.01±0.6564.01_± 0.65 24.00±0.2024.00_± 0.20 68.67±0.6768.67_± 0.67 6.86±0.216.86_± 0.21 3.51±0.093.51_± 0.09 33.41±0.3633.41_± 0.36 5×10−35× 10^-3 (default) 67.88±1.6467.88_± 1.64 24.53±1.0324.53_± 1.03 70.22±1.3570.22_± 1.35 5.88±0.645.88_± 0.64 5.39±0.625.39_± 0.62 34.78±0.2534.78_± 0.25 2×10−22× 10^-2 65.93±2.3165.93_± 2.31 24.73±0.9524.73_± 0.95 70.56±2.4670.56_± 2.46 7.11±0.217.11_± 0.21 3.61±0.093.61_± 0.09 34.39±1.2034.39_± 1.20 0.10.1 64.42±1.6764.42_± 1.67 22.60±4.0622.60_± 4.06 64.44±1.9564.44_± 1.95 4.90±0.854.90_± 0.85 4.50±0.374.50_± 0.37 32.17±1.2832.17_± 1.28 D.3 Out-of-Domain Evaluation Table 13: Out-of-domain accuracy (%) on Qwen1.5-MoE-A2.7B after RL on GSM8K. Mean ± standard deviation over three seeds; Science, Code, and All are item-weighted over the knowledge, code, and all benchmarks. Method MMLU ARC-C OBQA HumanEval MBPP Science Code All Base model (no RL) 47.9547.95 72.6172.61 66.6066.60 44.5144.51 33.4033.40 55.8855.88 36.1436.14 53.4953.49 No correction 49.96±0.3649.96_± 0.36 72.90±0.6572.90_± 0.65 69.67±0.4269.67_± 0.42 48.37±1.8648.37_± 1.86 33.67±1.4033.67_± 1.40 57.58±0.2357.58_± 0.23 37.30±1.5237.30_± 1.52 55.12±0.3455.12_± 0.34 Exact ratio 50.49±2.2150.49_± 2.21 67.49±5.6767.49_± 5.67 69.20±2.1169.20_± 2.11 50.00±1.2250.00_± 1.22 33.27±1.2133.27_± 1.21 56.56±3.0056.56_± 3.00 37.40±1.1137.40_± 1.11 54.24±2.5654.24_± 2.56 KPop† 51.33±0.2351.33_± 0.23 71.87±1.0871.87_± 1.08 71.07±1.2171.07_± 1.21 50.20±3.5750.20_± 3.57 33.33±1.8933.33_± 1.89 58.36±0.3358.36_± 0.33 37.50±2.1937.50_± 2.19 55.84±0.2555.84_± 0.25 IcePop 51.79±0.4051.79_± 0.40 72.81±1.4472.81_± 1.44 71.40±2.4271.40_± 2.42 52.64±0.3552.64_± 0.35 35.40±1.6435.40_± 1.64 58.93±0.1058.93_± 0.10 39.66±1.3039.66_± 1.30 56.60±0.0956.60_± 0.09 TIS 48.76±1.8148.76_± 1.81 69.77±6.2769.77_± 6.27 70.87±1.7970.87_± 1.79 48.17±1.2248.17_± 1.22 33.47±1.6033.47_± 1.60 56.15±2.8556.15_± 2.85 37.10±1.5137.10_± 1.51 53.85±2.3253.85_± 2.32 Seq-TIS 51.43±0.6551.43_± 0.65 73.41±0.5773.41_± 0.57 71.00±1.9171.00_± 1.91 50.20±2.7550.20_± 2.75 36.13±1.3336.13_± 1.33 58.80±0.5658.80_± 0.56 39.61±0.4539.61_± 0.45 56.48±0.4456.48_± 0.44 Seq-MIS 50.45±1.5750.45_± 1.57 72.35±0.9272.35_± 0.92 70.80±1.5970.80_± 1.59 50.81±1.8650.81_± 1.86 35.87±0.8335.87_± 0.83 57.88±1.2857.88_± 1.28 39.56±0.9239.56_± 0.92 55.66±1.0255.66_± 1.02 GSPO 50.32±1.8250.32_± 1.82 72.61±1.1972.61_± 1.19 70.67±3.2970.67_± 3.29 44.31±3.3644.31_± 3.36 32.53±1.3032.53_± 1.30 57.84±1.7257.84_± 1.72 35.44±1.7635.44_± 1.76 55.13±1.6955.13_± 1.69 FP16 49.19±0.7049.19_± 0.70 73.27±0.2673.27_± 0.26 68.67±0.4268.67_± 0.42 48.78±3.2348.78_± 3.23 34.47±1.1734.47_± 1.17 57.06±0.4957.06_± 0.49 38.00±0.8638.00_± 0.86 54.75±0.4454.75_± 0.44 CIS (ours) 51.87±0.2551.87_± 0.25 72.92±1.2872.92_± 1.28 72.13±1.7572.13_± 1.75 53.25±2.4653.25_± 2.46 35.80±2.4035.80_± 2.40 59.08±0.5259.08_± 0.52 40.11±2.2440.11_± 2.24 56.79±0.7156.79_± 0.71 Table 13 evaluates the checkpoints of Table 1 on Qwen1.5-MoE-A2.7B on three knowledge benchmarks (MMLU-STEM, ARC-Challenge, and OpenBookQA) and two code benchmarks (HumanEval and MBPP), none of which the RL stage uses. Every trained arm ends above the untrained model on the 5,4895,489 pooled items, so RL on GSM8K does not erase knowledge or coding ability. CIS has the highest pooled accuracy (56.7956.79), but its margin over IcePop (56.6056.60) is smaller than the standard deviations, so we do not claim a separation between the two. The exact ratio and TIS have the widest spreads on the pooled accuracy (±2.56± 2.56 and ±2.32± 2.32), as they do in Table 1. D.4 Computational Cost (a) (b) Figure 11: Compute time per step on Qwen1.5-MoE-A2.7B, excluding checkpointing and evaluation. (a) Time by phase; dashed: no correction. (b) Mean response length. Table 14: Compute time per optimizer step (seconds) on eight A100 GPUs, mean ± standard deviation over three seeds. Rescore is the training-engine forward pass that produces the training-side log-probabilities. Method Rollout Rescore Train step Advantage Total vs. no corr. Length No correction 0.44±0.020.44_± 0.02 0.00±0.000.00_± 0.00 29.41±0.0829.41_± 0.08 0.82±0.020.82_± 0.02 30.66±0.0930.66_± 0.09 — 321±1321_± 1 Exact ratio 0.48±0.030.48_± 0.03 6.50±0.086.50_± 0.08 23.26±0.4023.26_± 0.40 0.72±0.010.72_± 0.01 30.97±0.4830.97_± 0.48 +1.0%+1.0\% 252±3252_± 3 KPop† 0.45±0.010.45_± 0.01 6.64±0.586.64_± 0.58 23.90±2.0823.90_± 2.08 0.74±0.030.74_± 0.03 31.73±2.6831.73_± 2.68 +3.5%+3.5\% 260±24260_± 24 IcePop 0.49±0.030.49_± 0.03 6.51±0.236.51_± 0.23 23.12±0.6723.12_± 0.67 0.77±0.060.77_± 0.06 30.88±0.9330.88_± 0.93 +0.7%+0.7\% 250±8250_± 8 TIS 0.48±0.030.48_± 0.03 6.40±0.136.40_± 0.13 22.89±0.5622.89_± 0.56 0.74±0.010.74_± 0.01 30.51±0.6930.51_± 0.69 −0.5%-0.5\% 249±6249_± 6 Seq-TIS 0.49±0.020.49_± 0.02 7.39±0.047.39_± 0.04 26.44±0.2426.44_± 0.24 0.72±0.010.72_± 0.01 35.03±0.2835.03_± 0.28 +14.3%+14.3\% 289±2289_± 2 Seq-MIS 0.49±0.000.49_± 0.00 7.61±0.237.61_± 0.23 27.44±0.6627.44_± 0.66 0.74±0.010.74_± 0.01 36.28±0.8936.28_± 0.89 +18.3%+18.3\% 300±7300_± 7 FP16 0.46±0.030.46_± 0.03 0.00±0.000.00_± 0.00 28.81±0.3428.81_± 0.34 0.84±0.050.84_± 0.05 30.11±0.4030.11_± 0.40 −1.8%-1.8\% 316±2316_± 2 CIS (ours) 0.49±0.030.49_± 0.03 6.67±0.136.67_± 0.13 23.77±0.4823.77_± 0.48 0.72±0.010.72_± 0.01 31.65±0.6231.65_± 0.62 +3.2%+3.2\% 259±4259_± 4 Every operator is an elementwise map of the rollout-side and training-side log-probabilities, so no operator adds a forward or backward pass relative to the others. The arms that use the decoupled loss run one rescoring pass of the training engine per step (6.4 to 7.6 seconds), and the train step of these arms is shorter by a similar amount; the uncorrected and fp16 arms do not rescore. Figure 11 and Table 14 report the resulting compute time per step. The token-level operators lie between −0.5%-0.5\% and +3.5%+3.5\% of the uncorrected run, and CIS costs 3.2%3.2\% more. Seq-TIS and Seq-MIS cost 14%14\% and 18%18\% more, but they also generate longer responses (289 and 300 tokens against 249 to 260 for the token-level operators), and response length changes the step time independently of the operator. Normalizing by the number of tokens does not remove this confound either, because the uncorrected run processes the most tokens per step and is therefore the fastest per token. The wall-clock time between steps also includes checkpointing and evaluation, which differ between runs by up to 25 seconds per step because of how often each run was scheduled to evaluate, not because of the operator; we exclude them here. D.5 Where Each Operator Acts (a) (b) (c) (d) Figure 12: Tokens kept (blue) and truncated or masked (red) in the run of each operator (seed two), early, middle, and late in training; panel titles give the share acted on. (a) CIS. (b) IcePop. (c) TIS. (d) KPop†. Figure 12 shows which tokens each operator removes or down-weights during training. Early, middle, and late are the first, middle, and last twentieth of training; kept tokens are subsampled to 25,00025,000 per panel, and between 68%68\% and 85%85\% of all tokens have πtrn(yt)>0.99π^trn(y_t)>0.99 and overlap in the upper right corner. Early in training the two probabilities agree closely and every operator acts on less than 1%1\% of tokens. As the policy sharpens, the spread around the diagonal grows. The operators then act on different regions of the plane. CIS truncates only above the diagonal, and its truncated tokens concentrate at πtrn(yt)→1π^trn(y_t)→ 1, where the training engine is confident and the displacement exceeds the threshold. This concentration does not mean that the floor does most of the work. Applying Algorithm 1 to the logged tokens of the three CIS runs, about one third of the truncated tokens lie below the floor, where the cap equals 1+λκ≈1.011+λκ≈ 1.01, and these tokens account for only about 10%10\% of the total truncated excess ∑t(kt−ft) _t(k_t-f_t); more than half of it comes from tokens with 1−pt≥0.11-p_t≥ 0.1. IcePop masks mainly below the diagonal, on tokens whose ratio falls under 0.50.5, which the one-sided band of CIS leaves untouched. TIS caps the tokens whose training-side probability exceeds twice the inference-side probability, which lie mostly at low πinf(yt)π^inf(y_t). KPop† masks tokens near the edges of the plane, where one engine assigns a probability close to one and the other does not, at a rate that grows to 2.1%2.1\% late in training. CIS and IcePop therefore act on complementary sets of tokens. The KL mask and probability saturation. The binary-KL metric of the KPop baseline is KL(Bern(p)∥Bern(q))=plog(p/q)+(1−p)log((1−p)/(1−q))KL(Bern(p)\,\|\,Bern(q))=p (p/q)+(1-p) ((1-p)/(1-q)) evaluated in both directions, with p and q clamped to [10−8,1−10−8][10^-8,1-10^-8]. In single precision the upper clamp is the identity, so a token whose probability is exactly one on both engines yields 0⋅log(0/0)0· (0/0), and a token whose probability is exactly one on one engine yields +∞+∞; the bounds check treats both as out of bounds. On the tokens logged in the KPop run, 16.5%16.5\% have probability exactly one on both engines and 2.8%2.8\% on one engine, which together with the 0.45%0.45\% of tokens whose finite metric exceeds the threshold reproduces the 22%22\% intervention rate that the framework reports. Recomputing the metric in double precision with a representable clamp and mapping the undefined case to zero gives a mask rate of 0.45%0.45\%. The main text reports this corrected variant as KPop†, so that the comparison is not driven by probability saturation.Figure 12 counts tokens, so the truncations of CIS appear concentrated at high confidence: 83%83\% of the tokens that CIS truncates have pt≥0.9p_t≥ 0.9, compared with 25%25\% under TIS. By removed weight, however, more than half of the truncation of CIS comes from tokens with pt<0.9p_t<0.9. D.6 Limitations Models and engines. We train only mixture-of-experts models, on which the mismatch has a heavy tail (Appendix A). On the dense control the displacement is concentrated near zero, so we expect every correction to matter less there, but we do not train a dense model. The measurements and the training runs use bf16 and specific inference and training engines; other kernels or numerical formats can change the size of the mismatch, in which case λ may need to be retuned. In asynchronous training, ktk_t also contains the policy lag of stale rollouts, which our analysis does not model separately; stale tokens make up less than 1%1\% of the tokens in our runs. Hyperparameters and baselines. The threshold λ and the floor κ are chosen from a preregistered grid on Qwen1.5-MoE-A2.7B and kept fixed for the other two models, whereas every baseline uses the default hyperparameters of its reference implementation. A comparable sweep for each baseline, and combinations of CIS with methods that reduce the mismatch at its source, such as routing replay (Ma et al., 2025) or FP16 training (Qi et al., 2025), are left to future work. Scope of the theory. Theorems 3.2 and 3.3 give upper bounds under bounded per-token gradients and independent token contributions; for correlated contributions, the sample size n is replaced by neff=n/(1+(n−1)ρ¯)n_eff=n/(1+(n-1) ρ) (Appendix B). Any cap with a finite maximum admits a bound similar to Theorem 3.3, so the theory explains why truncation is needed but does not by itself rank CIS against other bounded caps; that comparison rests on the experiments of Section 4. What the token-level correction does not correct. The target gradient G0G_0 corrects the conditional distribution of each token given its prefix and not the distribution of the prefix itself. Neither the exact ratio nor CIS removes the shift in the prefix distribution. Seq-TIS and Seq-MIS act on the product of the ratios over a response, which addresses the prefix, but the product leaves any fixed interval for most responses of a few hundred tokens: Seq-MIS rejects 95%95\% of the responses in our runs (Table 10), and Seq-TIS caps 1212 to 17%17\% of them.