Paper deep dive
Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling
Shayan Mohajer Hamidi, Kayhan Behdin, Wenhui Zhu, Aida Rahmattalabi, Jincheng Cao, Jelena Markovic-Voronov, Alborz Geramifard
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/14/2026, 9:56:30 AM
Summary
The paper introduces Target-ESS Tempered Policy Optimization (TETPO), a method for stabilizing asynchronous reinforcement learning post-training of large language models. TETPO addresses the issue of heavy-tailed importance weights in off-policy settings by applying a data-adaptive power tempering exponent to importance weights. This exponent is selected to ensure the Effective Sample Size (ESS) meets a target fraction, providing smooth variance control without explicit zero-weight masking. Theoretical guarantees include deterministic bounds on weight dispersion and gradient norms, independent of policy drift. Empirical results on mathematical reasoning benchmarks show TETPO outperforms baselines like GRPO, GSPO, GEPO, and M2PO, particularly under high policy staleness.
Entities (13)
Relation Signals (10)
TETPO → appliesto → Large Language Models
confidence 95% · Asynchronous reinforcement-learning post-training of large language models... We introduce TETPO
TETPO → optimizes → Effective Sample Size
confidence 95% · TETPO selects the largest α whose tempered weights satisfy a target ESS fraction κ
TETPO → uses → Importance Sampling
confidence 95% · TETPO replaces each importance weight ρ by ρ^α... exact importance weights for a geometric bridge
TETPO → avoids → hard masking
confidence 90% · Power tempering also introduces no explicit zero-weight mask... distinguishing TETPO from M2PO’s hard masking.
TETPO → evaluatedon → AIME24
confidence 90% · Table 1 reports avg@16 accuracy... TETPO (Ours) 26.6±0.6
TETPO → evaluatedon → MATH-500
confidence 90% · Figure 1 shows the Math 500 trajectories... TETPO rises smoothly
TETPO → outperforms → GEPO
confidence 90% · GEPO suffers a catastrophic reward drop... TETPO rises smoothly to the highest reward.
TETPO → outperforms → GRPO
confidence 90% · On mathematical-reasoning benchmarks under high staleness, TETPO outperforms strong on- and off-policy baselines.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Asynchronous reinforcement-learning post-training of large language models is computationally efficient but inherently off-policy: the learner updates on rollouts generated by a stale policy, while correcting this mismatch requires importance weights that can be extremely heavy-tailed. A small number of trajectories may then dominate an update, collapsing the effective sample size (ESS) and increasing the variance and instability of the resulting gradient estimates. Existing stabilizers apply fixed interventions regardless of the realized severity of weight concentration, either discarding selected samples or reweighting them without adapting the correction strength. We introduce \emph{Target--ESS Tempered Policy Optimization} (TETPO), which replaces each importance weight $\rho$ by $\rho^\alpha$ for a data-adaptive exponent $\alpha\in[0,1]$: $\alpha=1$ preserves the original importance weights, whereas $\alpha=0$ assigns uniform coefficients. For each prompt group, TETPO selects the largest $\alpha$ whose tempered weights satisfy a target ESS fraction $\kappa$, applying little tempering to healthy groups and stronger tempering when weight degeneracy is severe. At the population level, the tempered coefficients are exact importance weights for a geometric bridge between the sampler and learner policies. The exponent is efficiently found by one-dimensional bisection, while the resulting weights satisfy deterministic bounds on normalized-weight dispersion and weighted-gradient norm that are independent of policy drift, together with a fixed-$\alpha$ bound on bridge-target discrepancy. Power tempering also introduces no explicit zero-weight mask. On mathematical-reasoning benchmarks under high staleness, TETPO outperforms strong on- and off-policy baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.14440v1
- Canonical: https://arxiv.org/abs/2610.14440v1
Trouble viewing inline? Open PDF directly →
Full Text
42,669 characters extracted from source content.
Expand or collapse full text
LinkedIn Target–ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling Shayan Mohajer Hamidi Email: smohajer@linkedin.com Kayhan Behdin Email: kbehdin@linkedin.com Wenhui Zhu Email: wenhzhu@linkedin.com Aida Rahmattalabi Email: arahmattalabi@linkedin.com Jincheng Cao Email: jincao@linkedin.com Jelena Markovic-Voronov Email: jmarkovicvoronov@linkedin.com Alborz Geramifard Email: agf@linkedin.com Affiliation: Abstract Asynchronous reinforcement-learning post-training of large language models is computationally efficient but inherently off-policy: the learner updates on rollouts generated by a stale policy, while correcting this mismatch requires importance weights that can be extremely heavy-tailed. A small number of trajectories may then dominate an update, collapsing the effective sample size (ESS) and increasing the variance and instability of the resulting gradient estimates. Existing stabilizers apply fixed interventions regardless of the realized severity of weight concentration, either discarding selected samples or reweighting them without adapting the correction strength. We introduce Target–ESS Tempered Policy Optimization (TETPO), which replaces each importance weight ρ by ραρ^α for a data-adaptive exponent α∈[0,1]α∈[0,1]: α=1α=1 preserves the original importance weights, whereas α=0α=0 assigns uniform coefficients. For each prompt group, TETPO selects the largest α whose tempered weights satisfy a target ESS fraction κ, applying little tempering to healthy groups and stronger tempering when weight degeneracy is severe. At the population level, the tempered coefficients are exact importance weights for a geometric bridge between the sampler and learner policies. The exponent is efficiently found by one-dimensional bisection, while the resulting weights satisfy deterministic bounds on normalized-weight dispersion and weighted-gradient norm that are independent of policy drift, together with a fixed-α bound on bridge-target discrepancy. Power tempering also introduces no explicit zero-weight mask. On mathematical-reasoning benchmarks under high staleness, TETPO outperforms strong on- and off-policy baselines. keywordsoff-policy reinforcement learning, large language models, importance sampling, effective sample size, variance reduction 1 Introduction and Motivation Modern reinforcement-learning (RL) post-training of large language models (LLMs) is dominated by the cost of rollout generation: producing several long completions y∼μ(⋅∣x)y μ(· x) per prompt x, scoring them, and taking policy-gradient steps on the resulting (x,y,R)(x,y,R) triples. In strictly synchronous schedules the learner sits idle during sampling. Asynchronous schedules (Noukhovitch et al., 2024; Fu et al., 2025; Wu et al., 2025; Piché et al., 2025) break this coupling: a sampler generates from a policy snapshot several gradient steps in the past while the learner trains on rollouts as they arrive, giving end-to-end speedups of 22–3×3× over synchronous training (Piché et al., 2025). The cost is that training becomes off-policy. Let πθ=πθt _θ= _ _t be the learner at step t and μ=πθt′μ= _ _t a sampler snapshot at step t′=t−kt =t-k. Assuming πθ(⋅∣x)≪μ(⋅∣x) _θ(· x) μ(· x) and θ-independent R, importance sampling (IS) yields the gradient identity ∇θJ(θ)=x∼,y∼μ(⋅∣x)[w(x,y)(R(x,y)−b(x))∇θlogπθ(y∣x)],w(x,y)≜πθ(y∣x)μ(y∣x), _θJ(θ)=E_x ,\,y μ(· x)\! [\,w(x,y)\, (R(x,y)-b(x) )\, _θ _θ(y x)\, ], w(x,y) _θ(y x)μ(y x), (1) for any y-independent baseline b(x)b(x). The group-relative advantages used in the practical surrogate are finite-sample variance-reduction terms; our results below concern the tempered weights conditional on the realised advantages, not unbiasedness of the full clipped update. But w(x,y)=∏t=1Tπθ(yt∣⋅)/μ(yt∣⋅)w(x,y)= _t=1^T _θ(y_t ·)/μ(y_t ·) is a product of T per-token ratios, with T in the thousands. Even small per-token shifts compound multiplicatively, so w is extremely heavy-tailed: the curse of the horizon (Liu et al., 2018; Liu et al., 2020). In practice a few trajectories dominate each update, and the effective sample size ESS()≜(∑i=1Bwi)2∑i=1Bwi2∈[1,B]ESS(w) ( _i=1^Bw_i )^2 _i=1^Bw_i^2∈[1,B] (2) collapses below the batch size B. When ESS collapses, weight concentration can sharply increase the variance and instability of the resulting gradient estimates, the learner-to-sampler KL spikes, and reward can collapse. The central question is therefore: how do we control the dispersion of heavy-tailed IS weights without losing so much information that learning stalls? Existing stabilizers. Two recent methods take complementary approaches. M2PO (Zheng et al., 2026) works per token: with ratio ri,t=πθ(yi,t∣⋅)/μ(yi,t∣⋅)r_i,t= _θ(y_i,t ·)/μ(y_i,t ·) and second-moment proxy Mi,t(2)=(logri,t)2M_i,t^(2)=( r_i,t)^2, it greedily zeroes the highest-M(2)M^(2) tokens in a trust region until the mean M(2)M^(2) falls below a threshold τM2 _M_2, yielding a masked surrogate ℒM2PO=−1N∑i,tmi,tri,tAiL_M2PO=- 1N _i,tm_i,t\,r_i,t\,A_i with mi,t∈0,1m_i,t∈\0,1\. GEPO (Zhang et al., 2026) works at the sequence level and never masks: rather than normalizing each likelihood ratio by the sum of the ratios in the batch, it divides the learner sequence-likelihood by a group-expectation denominator, i.e. an estimate ^grp[πθ] E_grp[ _θ] of the expected learner likelihood over the group, giving a group-expectation-normalized weight w^iGEPO=πθ(yi)/^grp[πθ] w^GEPO_i= _θ(y_i)/ E_grp[ _θ] (computed in log space for stability), so every sample contributes and nothing is discarded. Failure modes. Each buys stability at a cost that becomes acute as lag k grows. (i) M2PO collapses under heterogeneous staleness. The mask is discontinuous in ri,tr_i,t; when prompts in a batch are sampled at different lags, the M2M^2 statistic is dominated by high-lag tokens and the greedy mask zeroes a large fraction of the trust region, sometimes entirely, so that under mixed-staleness queues M2PO’s trusted-token fraction becomes erratic and the run can crash. In our k=32k=32 runs M2PO peaks early then declines on every benchmark. (i) GEPO has no variance-control knob. Its group-expectation normalization is fixed and has no tunable strength, so when a batch is dominated by a few heavy sequences it passes through highly concentrated weights without an explicit ESS constraint; at k=32k=32 GEPO suffers a catastrophic reward drop near step 200200. Our Approach. We want an estimator that (a) adapts its correction strength to the groupwise ESS: aggressive when ESS is collapsing, dormant when healthy; (b) is smooth in the IS ratios (no discrete masking); (c) provides a bound on the fixed-α bridge-target discrepancy; and (d) cannot collapse, so weights stay strictly positive even under an adversarial batch which motivate TETPO. 2 Methodology: Target–ESS Tempered Policy Optimization We follow the off-policy LLM-RL setting of M2PO (Zheng et al., 2026). A minibatch contains M prompts x1,…,xMx_1,…,x_M with G completions each; for prompt xmx_m the completions ym,1,…,ym,Gy_m,1,…,y_m,G are drawn from the recorded behaviour policy μm(⋅)≔μ(⋅∣xm) _m(·) μ(· x_m), and the sequence-level ratio is ρm,i≔πθ(ym,i∣xm)μm(ym,i)=exp(∑tlogπθ(ym,i,t∣xm,ym,i,<t)μm(ym,i,t∣xm,ym,i,<t)). _m,i _θ(y_m,i x_m) _m(y_m,i)= \! ( _t _θ(y_m,i,t x_m,y_m,i,<t) _m(y_m,i,t x_m,y_m,i,<t) ). (3) Because the bridge normaliser Zα(x)Z_α(x) (Theorem 1) is prompt-dependent, we normalise and select the tempering exponent within each prompt group, not across the pooled minibatch (which would implicitly reweight the prompt distribution). Definition 1 (Groupwise tempered weights and ESS). For α∈[0,1]α∈[0,1] and each group m, w¯m,i(α)≔ρm,iα,w~m,i(α)≔w¯m,i(α)1G∑j=1Gw¯m,j(α), w_m,i(α) _m,i^\,α, w_m,i(α) w_m,i(α) 1G _j=1^G w_m,j(α), (4) so 1G∑iw~m,i(α)=1 1G _i w_m,i(α)=1, and the groupwise effective sample size is ESSm(α)≔(∑iρm,iα)2∑iρm,i2α∈[1,G]. _m(α) ( _i _m,i^α)^2 _i _m,i^2α∈[1,G]. Definition 2 (Groupwise target-ESS exponent). Given a target ESS fraction κ∈(0,1]κ∈(0,1], αm⋆≔maxα∈[0,1]:ESSm(α)≥κG. _m \\,α∈[0,1]\;:\; _m(α)≥κ\,G\, \. (5) Since ρm,i0=1 _m,i^0=1 gives ESSm(0)=G≥κG _m(0)=G≥κ G, the feasible set is non-empty; as α↦ESSm(α)α _m(α) is continuous and non-increasing (Theorem 2) the maximum exists with ESSm(αm⋆)≥κG _m( _m )≥κ G for every group. We search all of [0,1][0,1] (no floor αmin _ ) by 11-D bisection. Stop-gradient surrogate. Let ℓm,i(θ)≔logπθ(ym,i∣xm) _m,i(θ) _θ(y_m,i x_m) and let sg[⋅]sg[·] be the stop-gradient operator. The exponent αm⋆ _m and weights w~m,i(αm⋆) w_m,i( _m ) are computed from detached ratios. Define the straight-through surrogate ratio rm,i(θ)≔sg[w~m,i(αm⋆)]exp(ℓm,i(θ)−sg[ℓm,i(θ)]),r_m,i(θ) \! [ w_m,i( _m ) ]\, \! ( _m,i(θ)-sg[ _m,i(θ)] ), (6) whose forward value is w~m,i(αm⋆) w_m,i( _m ) and whose gradient is w~m,i(αm⋆)∇θℓm,i(θ) w_m,i( _m )\, _θ _m,i(θ); at αm⋆=0 _m =0 the method reduces to a uniform-coefficient, GRPO-style score update on the stale samples (not an on-policy update, since the samples still come from μm _m). With group-relative advantage Am,iA_m,i, ℒTETPO(θ)=−1MG∑m=1M∑i=1Gmin(rm,i(θ)Am,i,clip(rm,i(θ), 1−εlo, 1+εhi)Am,i),L_TETPO(θ)=-\, 1MG _m=1^M _i=1^G \! (r_m,i(θ)\,A_m,i,\;clip (r_m,i(θ),\,1- _lo,\,1+ _hi )\,A_m,i ), (7) with Am,iA_m,i and all ESS-selection quantities treated as constants w.r.t. the learner gradient. This is a bridge-weighted PPO surrogate, not the exact gradient of the bridge objective: since π~α,θ π_α,θ depends on θ, ∇θπ~α[R]=απ~α[(R−π~α[R])∇θlogπθ] _θE_ π_α[R]=α\,E_ π_α[(R-E_ π_α[R]) _θ _θ], which (7) does not reproduce. Two regimes. Per group, (7) interpolates between two ends. Healthy (ESSm(1)≥κG _m(1)≥κ G): αm⋆=1 _m =1 and the group uses groupwise SNIS coefficients inside the clipped surrogate. Degenerate: the bisection returns αm⋆∈[0,1) _m ∈[0,1), tempering toward the uniform-weights limit (αm⋆=0 _m =0), which is always feasible. No group ever produces a zero tempering weight (Proposition 2), distinguishing TETPO from M2PO’s hard masking. 3 Theoretical Guarantees TETPO’s tempering enjoys four guarantees, stated and proved in Appendix A; we summarise them here. Exact bridge target (Thm. 1). Per prompt, the tempered coefficient ρα/Zα(x)ρ^α/Z_α(x), with Zα(x)=μ[ρα]Z_α(x)=E_μ[ρ^α], is the exact importance ratio of a geometric-bridge policy π~α∝πθαμ1−α π_α _θ^αμ^1-α against μ; α=1α=1 recovers πθ _θ and α=0α=0 recovers μ. Because Zα(x)Z_α(x) is prompt-dependent it cancels within a group but not across prompts, which is why normalisation is groupwise. This characterises the population target, not the finite-sample estimator (which is biased; see below). Well-posed selection (Thm. 2). α↦ESSm(α)α _m(α) is continuous and non-increasing with ESSm(0)=G _m(0)=G, so for every κ∈(0,1]κ∈(0,1] the target-ESS exponent αm⋆ _m exists (uniquely in the degenerate case) and is found by bisection on [0,1][0,1]; there is no need for a floor αmin _ . ESS-controlled dispersion (Thm. 3). The self-normalised weights obey 1G∑i(w~m,i(αm⋆)−1)2=G/ESSm(αm⋆)−1≤1/κ−1 1G _i( w_m,i( _m )-1)^2=G/ _m( _m )-1≤ 1/κ-1, independent of the πθ _θ-to-μ divergence, giving a weighted-gradient-norm bound ∥g^m∥2≤1κ⋅1G∑i∥gm,i∥2\| g_m\|^2≤ 1κ· 1G _i\|g_m,i\|^2. This is a deterministic realised-batch bound, not a bound on the estimator’s stochastic variance; in contrast, ordinary IS has Varμ[ρ]=eD2(πθ∥μ)−1 _μ[ρ]=e^D_2( _θ\|μ)-1, which grows with the drift (D2D_2 the order-two Rényi divergence). Bounded discrepancy (Thm. 4). For fixed α and |R|≤Rmax|R|≤ R_ , the bridge objective differs from the on-policy objective by at most Rmaxχ2(π~α∥πθ)R_ χ^2( π_α\| _θ). As χ2(π~α∥πθ)χ^2( π_α\| _θ) is non-increasing in α (Prop. 1) while dispersion is non-decreasing, αm⋆ _m is the least-tempered exponent meeting the ESS target, i.e. it minimises this χ2χ^2 bound among feasible exponents. The finite-sample and adaptation bias of the self-normalised estimator is not characterised. No masking (Prop. 2). All tempered weights are strictly positive, so tempering adds no explicit zero-weight mask, unlike M2PO, whose greedy mask can zero an entire trust region (the instability observed at high staleness). Positivity alone does not force a nonzero final PPO gradient when the advantage is zero or the clip is active. Complexity. Beyond a vanilla GRPO step, TETPO adds only O(niter⋅MG)O(n_iter· MG) work: the logρm,i _m,i are already available, and the 2424-iteration per-group bisections plus O(G)O(G) self-normalisation are negligible next to the per-token forward/backward passes. Table 2 (Appendix A) contrasts TETPO with prior stabilizers. 4 Experiments We evaluate TETPO against GRPO (Shao et al., 2024), GSPO (Zheng et al., 2025), GEPO (Zhang et al., 2026), and M2PO (Zheng et al., 2026) on math-reasoning benchmarks under high staleness (k=32k=32). Table 1 reports avg@16 accuracy (%): TETPO attains the best average on both models, improving over the strongest baseline by +2.1+2.1 points on Qwen3-Base-1.7B and +2.3+2.3 on Llama-3.2-3B-Instruct, and, consistent with the no-collapse guarantee, avoids the mid-training reward drop seen for GEPO and M2PO. Figure 1 shows the Math 500 trajectories at k=32k=32: GEPO collapses sharply (steps 200200, 450450) as low-ESS batches pass through highly concentrated weights, M2PO peaks then declines as its mask grows aggressive, and the on-policy baselines GRPO/GSPO plateau lower, whereas TETPO rises smoothly to the highest reward. Full details are in Appendix B. Figure 1: Math 500 avg@16 evaluation accuracy versus training step at staleness k=32k=32, for GRPO, GSPO, GEPO, M2PO, and TETPO (ours). GEPO collapses intermittently and M2PO peaks then declines, whereas TETPO rises smoothly and remains stable. Table 1: Accuracy (%) across mathematical-reasoning benchmarks, comparing GRPO, GSPO, GEPO, M2PO, and TETPO. Values are avg@16 accuracy under stochastic sampling, reported as mean ± std over three seeds. Within each model block, the best result per column is in bold and the second best is underlined. Method AIME24 AIME25 AMC23 AMC24 Math500 Avg. Qwen3-Base-1.7B GRPO (Shao et al., 2024) 15.6±1.1\,± 1.1 17.5±1.4\,± 1.4 42.1±0.8\,± 0.8 21.2±1.0\,± 1.0 74.6±0.5\,± 0.5 34.2±0.6\,± 0.6 GSPO (Zheng et al., 2025) 23.0±0.9\,± 0.9 18.7±1.2\,± 1.2 48.0±0.7\,± 0.7 25.6±0.9\,± 0.9 79.6±0.4\,± 0.4 39.0±0.5\,± 0.5 GEPO (Zhang et al., 2026) 19.5±1.6\,± 1.6 17.9±1.7\,± 1.7 43.4±1.2\,± 1.2 23.1±1.3\,± 1.3 77.1±0.8\,± 0.8 36.2±0.9\,± 0.9 M2PO (Zheng et al., 2026) 24.1±0.7\,± 0.7 24.2±1.0\,± 1.0 53.1±0.6\,± 0.6 27.5±0.8\,± 0.8 82.5±0.4\,± 0.4 42.3±0.5\,± 0.5 TETPO (Ours) 26.6±0.6\,± 0.6 24.6±0.8\,± 0.8 54.3±0.5\,± 0.5 32.8±0.7\,± 0.7 83.8±0.3\,± 0.3 44.4±0.4\,± 0.4 Llama-3.2-3B-Instruct GRPO (Shao et al., 2024) 1.5±0.7\,± 0.7 0.0±0.0\,± 0.0 15.5±1.1\,± 1.1 5.8±0.9\,± 0.9 40.2±0.6\,± 0.6 12.6±0.5\,± 0.5 GSPO (Zheng et al., 2025) 5.4±0.8\,± 0.8 0.6±0.5\,± 0.5 20.3±0.9\,± 0.9 9.8±1.0\,± 1.0 44.6±0.5\,± 0.5 16.1±0.4\,± 0.4 GEPO (Zhang et al., 2026) 2.8±0.9\,± 0.9 0.0±0.0\,± 0.0 22.1±1.3\,± 1.3 10.2±1.1\,± 1.1 45.9±0.7\,± 0.7 16.2±0.6\,± 0.6 M2PO (Zheng et al., 2026) 6.6±0.6\,± 0.6 0.3±0.4\,± 0.4 19.1±0.8\,± 0.8 8.7±0.7\,± 0.7 43.5±0.5\,± 0.5 15.6±0.4\,± 0.4 TETPO (Ours) 8.2±0.7\,± 0.7 1.6±0.6\,± 0.6 21.4±0.6\,± 0.6 14.9±0.9\,± 0.9 46.2±0.4\,± 0.4 18.5±0.3\,± 0.3 5 Limitations Tempering rescales all group weights uniformly; Theorem 4 bounds the fixed-α target discrepancy, but the finite-sample and adaptation bias of the estimator is not yet characterised. TETPO also uses sequence-level ratios; combining it with a per-token mask is future work. References Cai et al. (2026) Rui Cai, Wenhui Zhu, Xiwen Chen, Jincheng Cao, Han Yu, Shayan Mohajer Hamidi, Zelin He, Qiyao Ma, Daiwei Chen, Xuanzhao Dong, et al. Smaller models, better rejects: Preference distillation scaling. arXiv preprint arXiv:2609.38987, 2026. Fu et al. (2025) Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiacai Wang, Tingxun Yang, Binxing Yuan, and Yi Wu. AReaL: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025. Hamidi et al. (2026a) Shayan Mohajer Hamidi, Yize Cheng, Yuanda Xu, Zhengze Zhou, and Alborz Geramifard. Hierarchical reinforcement learning with stable temporal abstraction for language model agents. In SLM-Agents: 1st NeurIPS Workshop on SLMs for Agentic Systems, 2026a. URL https://openreview.net/forum?id=XYhgL4F1uZ. Hamidi et al. (2026b) Shayan Mohajer Hamidi, Linfeng Ye, and Konstantinos N. Plataniotis. Dreamphase: Offline imagination and uncertainty-guided planning for large-language-model agents. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?id=81PJ2KPnmK. Liu et al. (2018) Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in neural information processing systems, 31, 2018. Liu et al. (2020) Yao Liu, Pierre-Luc Bacon, and Emma Brunskill. Understanding the curse of horizon in off-policy evaluation via conditional importance sampling. In International Conference on Machine Learning, pages 6184–6193. PMLR, 2020. Noukhovitch et al. (2024) Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous RLHF: Faster and more efficient off-policy RL for language models. arXiv preprint arXiv:2410.18252, 2024. Piché et al. (2025) Alexandre Piché, Ehsan Kamaloo, Rafael Pardinas, and Dzmitry Bahdanau. PipelineRL: Faster on-policy reinforcement learning for long sequence generation. arXiv preprint arXiv:2509.19128, 2025. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Wu et al. (2025) Bo Wu, Sid Wang, Yu Tang, Jay Ding, Eric Helenowski, Liang Tan, Tao Xu, Thamme Gowda, Zhuokai Chen, Chen Zhu, Xin Tang, Yongqiang Qian, Bin Zhu, and Rui Hou. LlamaRL: A distributed asynchronous reinforcement learning framework for efficient large-scale LLM training. arXiv preprint arXiv:2505.24034, 2025. Zhang et al. (2026) Han Zhang, RuibinZheng, ZEXUAN YI, Zhuo Zhang, Hanyang Peng, Hui Wang, Jiayin Qi, Binxing Fang, Ruifeng Xu, and Yue Yu. GEPO: Group expectation policy optimization for stable heterogeneous reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=movNwgQtDt. Zheng et al. (2025) Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. Zheng et al. (2026) Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy RL reach with stale data on LLMs? In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=IIgl5MWelz. Appendix A Theoretical results and proofs Table 2: TETPO is the only method that (i) operates smoothly, (i) achieves a divergence-free (ESS-controlled) weight-dispersion bound, and (i) provably adds no explicit zero-weight mask. Second-moment entries use the order-two Rényi divergence D2D_2 (χ2=eD2−1χ^2=e^D_2-1). Method Granularity Smoothness Var./disp. bound Mask-collapse risk GSPO (Zheng et al., 2025) sequence PPO clip eD2e^D_2 low GEPO (Zhang et al., 2026) group group-exp. eD2−Ce^D_2-C low M2PO (Zheng et al., 2026) token hard mask R2M2R^2M_2 high SNIS+ESS sequence hard truncate 1/β−11/β-1 low TETPO (ours) sequence smooth 1/κ−1 1/κ-1 (disp.) none (Prop. 2) Theorem 1 (Geometric-bridge importance identity). Fix a prompt x with πθ(⋅∣x),μ(⋅∣x)>0 _θ(· x),μ(· x)>0 on a common support xY_x. For α∈[0,1]α∈[0,1] let Zα(x)≔y∼μ(⋅∣x)[ρ(y∣x)α]Z_α(x) _y μ(· x)[ρ(y x)^α] and define the geometric-bridge policy π~α(y∣x)≔πθ(y∣x)αμ(y∣x)1−α/Zα(x) π_α(y x) _θ(y x)^α\,μ(y x)^1-α/Z_α(x). Then π~α(y∣x)/μ(y∣x)=ρ(y∣x)α/Zα(x) π_α(y x)/μ(y x)=ρ(y x)^α/Z_α(x), so for every integrable h, y∼π~α(⋅∣x)[h(y)]=y∼μ(⋅∣x)[ρ(y∣x)αh(y)]y∼μ(⋅∣x)[ρ(y∣x)α],E_y π_α(· x)[h(y)]= E_y μ(· x)[ρ(y x)^α\,h(y)]E_y μ(· x)[ρ(y x)^α], (8) with endpoints π~0=μ π_0=μ and π~1=πθ π_1= _θ. Proof of Theorem 1 By the definition of π~α π_α, π~α(y∣x)μ(y∣x)=πθ(y∣x)αμ(y∣x)1−αZα(x)μ(y∣x)=1Zα(x)(πθ(y∣x)μ(y∣x))α=ρ(y∣x)αZα(x). π_α(y x)μ(y x)= _θ(y x)^α\,μ(y x)^1-αZ_α(x)\,μ(y x)= 1Z_α(x) ( _θ(y x)μ(y x) )^\!α= ρ(y x)^αZ_α(x). (9) Hence, for any integrable h, y∼π~α(⋅∣x)[h(y)]=∑y∈xπ~α(y∣x)h(y)=∑y∈xμ(y∣x)ρ(y∣x)αZα(x)h(y)=y∼μ(⋅∣x)[ρ(y∣x)αh(y)]Zα(x).E_y π_α(· x)[h(y)]= _y _x π_α(y x)\,h(y)= _y _xμ(y x)\, ρ(y x)^αZ_α(x)\,h(y)= E_y μ(· x)[ρ(y x)^α\,h(y)]Z_α(x). (10) Taking h≡1h≡ 1 gives Zα(x)=μ[ρα]Z_α(x)=E_μ[ρ^α], so π~α[h]=μ[ραh]μ[ρα],E_ π_α[h]= E_μ[ρ^α\,h]E_μ[ρ^α], (11) which is the stated ratio-of-expectations identity. The endpoints follow from π~0∝μ π_0 μ and π~1∝πθ π_1 _θ. Finally, Zα(x)Z_α(x) depends on x, so it cancels between numerator and denominator only within a fixed prompt and not under pooled cross-prompt normalisation. Theorem 2 (Existence, monotonicity, uniqueness, bisection). Fix a group with ρ1,…,ρG>0 _1,…, _G>0 and set f(α)≔ESSm(α)f(α) _m(α) on [0,1][0,1]. Then f is continuous, f(0)=Gf(0)=G, f(1)=ESS(ρ)f(1)= (ρ), and f is non-increasing (strictly decreasing on (0,1](0,1] when the ρi _i are not all equal). Hence for every κ∈(0,1]κ∈(0,1] the feasible set α:f(α)≥κG\α:f(α)≥κ G\ is a non-empty closed interval [0,αm⋆][0, _m ]: if f(1)≥κGf(1)≥κ G then αm⋆=1 _m =1; if κ<1κ<1 and f(1)<κGf(1)<κ G then αm⋆∈(0,1) _m ∈(0,1) is the unique root of f(α)=κGf(α)=κ G; and if κ=1κ=1 with non-constant ratios then αm⋆=0 _m =0. Bisection on [0,1][0,1] brackets αm⋆ _m to width 2−niter2^-n_iter. Proof of Theorem 2 Write zi≔logρiz_i _i and S(a)≔∑i=1Geazi,f(a)=S(a)2S(2a).S(a) _i=1^Ge^az_i, f(a)= S(a)^2S(2a). (12) Then f is continuous on [0,1][0,1], and f(0)=G2G=G,f(1)=(∑iρi)2∑iρi2=ESS(ρ).f(0)= G^2G=G, f(1)= ( _i _i)^2 _i _i^2= (ρ). (13) Introduce the exponentially tilted law pa(i)≔eazi/S(a)p_a(i) e^az_i/S(a). Since dalogS(a)=pa[z] dda S(a)=E_p_a[z], dalogf(a)=2dalogS(a)−dalogS(2a)=2(pa[z]−p2a[z]), dda f(a)=2\, dda S(a)- dda S(2a)=2 (E_p_a[z]-E_p_2a[z] ), (14) and, moreover, dapa[z]=Varpa(z)≥0. dda\,E_p_a[z]= _p_a(z)≥ 0. (15) Thus a↦pa[z]a _p_a[z] is non-decreasing, so dalogf(a)≤0 dda f(a)≤ 0 and f is non-increasing; it is strictly decreasing on (0,1](0,1] whenever the ziz_i are not all equal, since then Varpa(z)>0 _p_a(z)>0 for finite a. Because f(0)=G≥κGf(0)=G≥κ G, the feasible set a∈[0,1]:f(a)≥κG\a∈[0,1]:f(a)≥κ G\ is non-empty; by continuity it is closed, and by monotonicity it is an interval [0,αm⋆][0, _m ]. The stated cases follow: (i) if f(1)≥κGf(1)≥κ G then αm⋆=1 _m =1; (i) if κ<1κ<1 and f(1)<κGf(1)<κ G, the intermediate value theorem yields a unique root of f(α)=κGf(α)=κ G in (0,1)(0,1); (i) if κ=1κ=1 with non-constant ratios, f(α)<Gf(α)<G for all α>0α>0, forcing αm⋆=0 _m =0; (iv) if all ratios are equal, f≡Gf≡ G and αm⋆=1 _m =1. Bisection halves the bracket at each iteration. Theorem 3 (ESS-controlled dispersion and weighted-gradient norm). For each group m the self-normalised weights w~m,i(αm⋆) w_m,i( _m ) (empirical mean 11) satisfy the deterministic dispersion bound 1G∑i=1G(w~m,i(αm⋆)−1)2=GESSm(αm⋆)−1≤1κ−1, 1G _i=1^G ( w_m,i( _m )-1 )^2= G _m( _m )-1≤ 1κ-1, (16) independent of any πθ _θ-to-μ divergence. Consequently, for arbitrary vectors gm,ig_m,i, the weighted group mean g^m≔1G∑iw~m,i(αm⋆)gm,i g_m 1G _i w_m,i( _m )g_m,i obeys ∥g^m∥2≤1κ⋅1G∑i∥gm,i∥2\| g_m\|^2≤ 1κ· 1G _i\|g_m,i\|^2, and the minibatch mean g^≔1M∑mg^m g 1M _m g_m obeys ‖g^‖2≤1κMG∑m,i‖gm,i‖2\| g\|^2≤ 1κ MG _m,i\|g_m,i\|^2. Proof of Theorem 3 Fix a group m and abbreviate w~i≔w~m,i(αm⋆) w_i w_m,i( _m ). Self-normalisation gives 1G∑iw~i=1 1G _i w_i=1, i.e. ∑iw~i=G _i w_i=G. As ESS is invariant to a common positive rescaling of the weights, ESSm(αm⋆)=(∑iw~i)2∑iw~i2=G2∑iw~i2⟹1G∑i=1Gw~i2=GESSm(αm⋆). _m( _m )= ( _i w_i )^2 _i w_i^2= G^2 _i w_i^2 1G _i=1^G w_i^2= G _m( _m ). (17) Therefore, using 1G∑iw~i=1 1G _i w_i=1 and ESSm(αm⋆)≥κG _m( _m )≥κ G, 1G∑i=1G(w~i−1)2=1G∑iw~i2−1=GESSm(αm⋆)−1≤1κ−1. 1G _i=1^G ( w_i-1 )^2= 1G _i w_i^2-1= G _m( _m )-1≤ 1κ-1. (18) For the gradient bound, the Cauchy–Schwarz inequality gives ∥1G∑i=1Gw~igi∥2≤(1G∑iw~i2)(1G∑i∥gi∥2)≤1κ⋅1G∑i=1G∥gi∥2. \| 1G _i=1^G w_i\,g_i \|^2≤ ( 1G _i w_i^2 ) ( 1G _i\|g_i\|^2 )≤ 1κ· 1G _i=1^G\|g_i\|^2. (19) Averaging over groups and applying Jensen’s inequality, ‖g^‖2=‖1M∑m=1Mg^m‖2≤1M∑m=1M‖g^m‖2≤1κMG∑m=1M∑i=1G‖gm,i‖2.\| g\|^2= \| 1M _m=1^M g_m \|^2≤ 1M _m=1^M\| g_m\|^2≤ 1κ MG _m=1^M _i=1^G\|g_m,i\|^2. (20) Theorem 4 (Fixed-α bridge-target discrepancy). Fix a prompt x and α∈[0,1]α∈[0,1] with π~α(⋅∣x)≪πθ(⋅∣x) π_α(· x) _θ(· x), and suppose |R|≤Rmax|R|≤ R_ . Then |π~α(⋅∣x)[R]−πθ(⋅∣x)[R]|≤Rmaxχ2(π~α(⋅∣x)∥πθ(⋅∣x)), |E_ π_α(· x)[R]-E_ _θ(· x)[R] |≤ R_ \, χ^2\! ( π_α(· x)\,\|\, _θ(· x) ), (21) where χ2(p∥q)=q[(p/q−1)2]χ^2(p\|q)=E_q[(p/q-1)^2] is the Pearson divergence. Proof of Theorem 4 Let rα≔π~α(⋅∣x)/πθ(⋅∣x)r_α π_α(· x)/ _θ(· x), which is well defined because π~α≪πθ π_α _θ, and note πθ[rα]=1E_ _θ[r_α]=1. Then π~α[R]−πθ[R]=πθ[rαR]−πθ[R]=πθ[(rα−1)R].E_ π_α[R]-E_ _θ[R]=E_ _θ[r_αR]-E_ _θ[R]=E_ _θ\! [(r_α-1)R ]. (22) By the Cauchy–Schwarz inequality and |R|≤Rmax|R|≤ R_ , |πθ[(rα−1)R]|≤(πθ[(rα−1)2])1/2(πθ[R2])1/2≤Rmaxχ2(π~α∥πθ), |E_ _θ[(r_α-1)R] |≤ (E_ _θ[(r_α-1)^2] )^1/2 (E_ _θ[R^2] )^1/2≤ R_ χ^2\! ( π_α\,\|\, _θ ), (23) where the last step uses πθ[(rα−1)2]=χ2(π~α∥πθ)E_ _θ[(r_α-1)^2]=χ^2( π_α\| _θ). The bound concerns the population objective π~α[R]E_ π_α[R] only; it does not assert unbiasedness of the finite-sample self-normalised estimator, nor account for the data-adaptive selection of αm⋆ _m . Proposition 1 (Monotonicity of the bridge discrepancy). Fix a prompt x with μ(⋅∣x)μ(· x) and πθ(⋅∣x) _θ(· x) mutually absolutely continuous. Let X≔log(μ(Y∣x)/πθ(Y∣x))X (μ(Y x)/ _θ(Y x)) with Y∼πθ(⋅∣x)Y _θ(· x), and suppose M(t)≔πθ[etX]<∞M(t) _ _θ[e^tX]<∞ for t∈[0,2]t∈[0,2]. Then α↦χ2(π~α∥πθ)α χ^2( π_α\| _θ) is non-increasing on [0,1][0,1], and strictly decreasing on [0,1)[0,1) unless μ=πθμ= _θ almost surely. Proof Set t≔1−α∈[0,1]t 1-α∈[0,1] and K(t)≔logM(t)K(t) M(t). The bridge density relative to πθ _θ is π~α(Y∣x)πθ(Y∣x)=etX(Y)M(t), π_α(Y x) _θ(Y x)= e^tX(Y)M(t), (24) so 1+χ2(π~α∥πθ)=πθ[(etXM(t))2]=M(2t)M(t)2=exp(K(2t)−2K(t)).1+χ^2\! ( π_α\,\|\, _θ )=E_ _θ\! [ ( e^tXM(t) )^\!2 ]= M(2t)M(t)^2= \! (K(2t)-2K(t) ). (25) Differentiating, dtlog(1+χ2(π~α∥πθ))=2K′(2t)−2K′(t)≥0, ddt \! (1+χ^2( π_α\| _θ) )=2K (2t)-2K (t)≥ 0, (26) because K is a cumulant generating function and hence convex, so K′K is non-decreasing. Thus χ2χ^2 is non-decreasing in t=1−αt=1-α, i.e. non-increasing in α, and strictly so when X is non-degenerate. Proposition 2 (No hard masking by power tempering). For every α∈[0,1]α∈[0,1] and strictly positive ρ1,…,ρG _1,…, _G, the normalised tempered weights satisfy w~m,i(α)>0 w_m,i(α)>0. Thus power tempering never assigns an exactly zero coefficient to a sampled trajectory. (Strict positivity does not by itself imply a nonzero final PPO gradient: a sample still contributes zero when its advantage is zero or the clip is active.) Proof of Proposition 2 Fix α∈[0,1]α∈[0,1]. If α>0α>0, then ρm,i>0 _m,i>0 implies ρm,iα>0 _m,i^α>0; if α=0α=0, then ρm,i0=1>0 _m,i^0=1>0. In either case the group denominator 1G∑jρm,jα>0 1G _j _m,j^α>0, so w~m,i(α)=ρm,iα1G∑j=1Gρm,jα>0. w_m,i(α)= _m,i^α 1G _j=1^G _m,j^α>0. (27) Appendix B Experimental Details and Reproducibility Models and data. We post-train Qwen3-Base-1.7B (and, in the second block, Llama-3.2-3B-Instruct) with RL. Training prompts are the SimpleRL-Zoo competition-math split SimpleLR/simplelr_qwen_level3to5 (difficulty levels 3–5), loaded from the released train.parquet and filtered to prompts of at most 768768 tokens; we use the entire filtered split without subsampling. Evaluation uses five separate public benchmarks: MATH-500 (HuggingFaceH4/MATH-500), AMC23 (HanlardResearch/amc23), AMC24 (rawsh/2024_AMC12), AIME24 (HanlardResearch/aime_2024), and AIME25 (HanlardResearch/aime_2025), which come from sources disjoint from the training corpus, so evaluation items are excluded from training by construction; we did not run an additional n-gram decontamination pass. The reward is binary correctness computed by math-verify (v0.5.2): the gold answer is wrapped in and parsed, the completion is parsed with boxed-answer priority, and the reward is verify(answer, gold)∈0,11\ verify(answer, gold)\∈\0,1\, with no learned reward model. Rewards are not scaled (scale_rewards=False); advantages are the within-group mean-subtracted rewards (GRPO-style). Optimisation hyperparameters. All methods share: optimiser AdamW, learning rate 1×10−61×10^-6 with cosine schedule and warmup ratio 0.050.05, gradient clipping 1.01.0, KL coefficient β=0β=0, and 33 epochs with evaluation/checkpointing every 6464 steps. Rollouts use G=8G=8 completions per prompt with M=64M=64 prompts per optimisation step (effective batch B=MG=512B=MG=512 sequences; realised as per-device batch 88 × gradient accumulation 88 across 44 GPUs), max prompt length 768768 and max completion length 40964096. Decoding for rollouts uses temperature 0.60.6, top-p 0.950.95, top-k 2020, min -p 00. The PPO clip bounds are εlo=εhi=0.2 _lo= _hi=0.2. For TETPO we use target ESS fraction κ=0.85κ=0.85, floor αmin=0 _ =0, and a 2424-iteration bisection over [0,1][0,1] (∼10−7 10^-7 precision). Staleness protocol. We use a PipelineRL-style schedule (uniform_stale_k): the sampler serves from a pool of recent learner snapshots, and each rollout’s policy lag is drawn independently as ℓ∼Uniform1,…,k \1,…,k\ with k=32k=32. Thus k is the maximum lag and the realised lag is random (not fixed or averaged); the snapshot pool holds up to 256256 checkpoints. This produces the heterogeneous staleness that TETPO’s per-group adaptation targets. Importance ratios under truncated decoding. Our rollout sampler applies temperature scaling followed by hard top-k/top-p truncation. Let μ(h)S_μ(h) denote the token support retained by this decoding rule at history h. The resulting behaviour distribution is μ(a∣h)=μraw(a∣h)a∈μ(h)∑a′∈μ(h)μraw(a′∣h),μ^S(a h)= μ^raw(a h)1\a _μ(h)\ _a _μ(h)μ^raw(a h), where μrawμ^raw denotes the temperature-scaled distribution before top-k/top-p truncation. Because μ(a∣h)=0μ^S(a h)=0 outside μ(h)S_μ(h) while the learner generally assigns positive probability there, trajectories generated by μ^S cannot provide an exact importance-sampling estimator for the unrestricted learner policy. An exact change-of-measure identity is obtained instead for the learner restricted to the same behaviour support, πθ(a∣h)=πθraw(a∣h)a∈μ(h)∑a′∈μ(h)πθraw(a′∣h), _θ^S(a h)= _θ^raw(a h)1\a _μ(h)\ _a _μ(h) _θ^raw(a h), for which ρ(y∣x)=∏tπθ(yt∣x,y<t)μ(yt∣x,y<t)ρ^S(y x)= _t _θ^S(y_t x,y_<t)μ^S(y_t x,y_<t) satisfies πθ≪μ _θ^S μ^S and y∼μ[ρ(y∣x)]=1E_y μ^S[ρ^S(y x)]=1. In the experiments reported here, however, we do not retain μ(h)S_μ(h) or the corresponding truncation normalisers. We record the log-probability of the sampled token under the pre-truncation, temperature-scaled behaviour distribution and form the sequence-level proxy ρ^(y∣x)=∏tπθraw(yt∣x,y<t)μraw(yt∣x,y<t). ρ(y x)= _t _θ^raw(y_t x,y_<t)μ^raw(y_t x,y_<t). Consequently, whenever top-k/top-p truncation is active, ρ ρ is not the exact importance ratio for either the unrestricted learner policy or the support-restricted policy πθ _θ^S. TETPO’s empirical ESS and the selected exponent α⋆α are therefore computed from ρ ρ and should be viewed as using an approximate likelihood-ratio proxy. This distinction affects the interpretation of our theoretical results. The ESS monotonicity, target-ESS feasibility, deterministic weight-dispersion bound, and positivity results (Theorems 2–3 and Proposition 2) hold for any strictly positive input ratios and therefore continue to hold exactly when TETPO is applied to ρ ρ. In contrast, the exact geometric-bridge identity and the associated bridge-target discrepancy results (Theorems 1 and 4 and Proposition 1) apply to the exact support-restricted ratio ρ^S and do not constitute guarantees for the approximate ratios used in the present experiments. Logging the retained support (or otherwise evaluating the post-truncation normalisers) would permit exact computation of ρ^S; we leave this implementation and a direct comparison between exact and approximate ratios to future work. Evaluation and overhead. Reported numbers are avg@16 accuracy (%) on the five benchmarks above: for each prompt we draw 1616 independent samples with stochastic decoding (temperature 0.60.6, top-p 0.950.95, top-k 2020, matching the rollout settings) and average the binary correctness over those samples. We use avg@16 rather than single-sample greedy accuracy because several of these benchmarks are small (AIME24/AIME25 contain 3030 problems each), so a single greedy decode per prompt gives a very coarse, high-variance estimate. Each configuration is run with three training seeds, and Table 1 reports mean ± standard deviation over those seeds; per-seed results accompany the code. The only computation TETPO adds to a GRPO step is the per-group 2424-iteration bisection on already-computed log-ratios (O(niterMG)O(n_iter\,MG) elementwise operations) and the groupwise self-normalisation; these introduce no additional stored activations, and we observed no measurable change in per-step wall-clock relative to GRPO. Appendix C Additional Related Work RL post-training and LLM agents. TETPO targets the off-policy correction that arises in asynchronous RL post-training of LLMs (Noukhovitch et al., 2024; Fu et al., 2025; Wu et al., 2025; Piché et al., 2025), where group-relative objectives such as GRPO (Shao et al., 2024) and its sequence-level variants (Zheng et al., 2025) are trained on stale rollouts. Beyond single-turn reasoning, RL is increasingly used to train multi-step LLM agents, where unstable updates are a central difficulty: hierarchical RL with stable temporal abstraction (Hamidi et al., 2026a) stabilizes agent training by structuring decisions over temporally extended actions, and offline imagination with uncertainty-guided planning (Hamidi et al., 2026b) reduces reliance on costly online interaction by planning over imagined trajectories. Both settings reuse data generated under a policy other than the current learner, so the weight-degeneracy problem studied here also arises there, and TETPO’s group-wise ESS control is complementary to these designs. A related line of work studies preference-based post-training and distillation, e.g. how the choice of models used to generate rejected responses affects preference distillation at scale (Cai et al., 2026); such pipelines likewise train on samples drawn from a distribution that differs from the learner’s, which motivates distribution-mismatch corrections of the kind TETPO provides.