Paper deep dive
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Qijia He, Ruinan Jin, Jun Luo, Shaofeng Zou, Yingbin Liang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/4/2026, 2:06:04 AM
Summary
This paper addresses the challenge of data staleness in asynchronous reinforcement learning (RL) for large language model (LLM) post-training. The authors derive a convergence bound for GRPO-style algorithms that characterizes the tradeoff between the gradient estimator's second moment and bias introduced by stale rollouts. They propose Group-Mass Capping GRPO (GMC-GRPO), a novel method that minimizes a ratio-based bias bound while maintaining a common second-moment guarantee. Theoretical analysis shows GMC-GRPO improves the threshold dependence of the fourth-order delay term compared to TIC-GRPO. Experiments on Qwen3 models demonstrate improved robustness to stale rollouts and superior performance under large rollout delays.
Entities (6)
Relation Signals (4)
Asynchronous RL → causes → Stale Rollouts
confidence 95% · Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies.
GMC-GRPO → improves → TIC-GRPO
confidence 95% · compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from O(ϵ−4) to O(ϵ−2)
GMC-GRPO → isbasedon → GRPO
confidence 95% · we propose a novel group mass capping GRPO (GMC-GRPO) method
GMC-GRPO → demonstratesbetterperformanceon → Qwen3
confidence 90% · Experiments across Qwen3 models... demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator's second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from $O(\epsilon^{-4})$ to $O(\epsilon^{-2})$ as $\epsilon\to0$, where $1+\epsilon$ is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as $G^{-2/5}$ after tuning the step size, where $G$ is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as $G\to\infty$, whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among stable baselines under large rollout delays.
Tags
Links
- Source: https://arxiv.org/abs/2610.01896v1
- Canonical: https://arxiv.org/abs/2610.01896v1
Trouble viewing inline? Open PDF directly →
Full Text
171,478 characters extracted from source content.
Expand or collapse full text
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis Qijia He Affiliation: The Ohio State University Ruinan Jin Affiliation: The Ohio State University Jun Luo Affiliation: The Ohio State University Shaofeng Zou Affiliation: Arizona State University Yingbin Liang Affiliation: The Ohio State University Abstract Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator’s second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from O(ϵ−4)O(ε^-4) to O(ϵ−2)O(ε^-2) as ϵ→0ε→ 0, where 1+ϵ1+ε is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as G−2/5G^-2/5 after tuning the step size, where G is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as G→∞G→∞, whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among stable baselines under large rollout delays. 1 introduction Reinforcement learning (RL) has become a widely used paradigm for post-training large language models (LLMs), enabling them to improve from outcome-level feedback rather than relying solely on static demonstrations (Ouyang et al., 2022; Lambert et al., 2024). Given a prompt, RL allows an LLM to generate thinking tokens before producing a final answer, and reinforces the generated trajectory based on its final quality (Jaech et al., 2024; Guo et al., 2025). Models post-trained with algorithms such as GRPO (Shao et al., 2024) and its variants (Yu et al., 2026; Liu et al., 2025; Zheng et al., 2026) have shown strong capabilities on challenging reasoning problems, such as math (Cobbe et al., 2021; He et al., 2024), coding (Jain et al., 2025), and agentic tasks (Du et al., 2026). To reduce the inefficiency of repeatedly alternating between rollout generation and policy optimization, modern RL training pipelines increasingly adopt asynchronous execution (Zhong et al., 2024; Sheng et al., 2025). Such systems decouple rollout generation from policy optimization: rollout workers continuously generate responses using their local policy versions, while the learner updates the policy as soon as sufficient completed rollouts have accumulated and publishes the updated parameters without blocking generation (Fu et al., 2026; Hu et al., 2026; Noukhovitch et al., 2025). Together with optimized training engines and workload scheduling, this design allows inference and training to proceed concurrently, improving resource utilization and substantially increasing the training throughput of GRPO-style methods (Wang et al., 2025). The decoupling, however, introduces data staleness. Since rollout workers and the learner progress independently, the learner may update the policy several times before a generated rollout is consumed. Consequently, a training batch can mix trajectories collected under multiple earlier policy versions, creating a mismatch between the behavior policies that generated the data and the current policy being optimized (Fu et al., 2026). As the delay grows, these trajectories can become increasingly off-policy, making policy optimization more difficult and reducing training efficiency (Tian et al., 2026; Zheng et al., 2026; Xu et al., 2026). Despite the empirical success of delay-aware algorithms (Roux et al., 2025; Xi et al., 2026; Huang et al., 2026), how this staleness affects the convergence of GRPO-style algorithms, and how to mitigate its impact through principled algorithm design, remain poorly understood. This raises a natural but fundamental research question: Can we theoretically characterize how data staleness affects the convergence of GRPO-style algorithms, and use this to design RL algorithms better suited to asynchronous training systems? We answer this question by first identifying what any importance-weighted algorithm must control in order to converge, and then designing the correction that best meets this requirement under delay. Our key contributions are as follows: • Second moment and bias tradeoff. We derive a convergence bound for GRPO-style algorithms which explicitly captures the tradeoff between the second moment of gradient estimators and the bias error. With trajectory-level importance sampling, stale rollouts can produce large importance ratios and enlarge the gradient estimator’s second moment. Clipping these ratios controls the second moment but can introduce bias. When the resulting second-moment bound is independent of delay, delay enters the convergence bound through this clipping bias. This motivates algorithms that minimize a bias bound while maintaining a common second-moment guarantee. • GMC-GRPO. Guided by this principle, we introduce a class of weighted gradient estimators that satisfy a common second-moment bound. This class includes several GRPO-style methods as special cases. Within this class, we derive group mass capping (GMC)-GRPO by minimizing a bias bound and selecting a solution that preserves each group’s gradient direction, while retaining the same second-moment guarantee. This makes GMC-GRPO particularly well suited to asynchronous training, where its principled weighting rule effectively limits the bias. • Convergence of asynchronous GMC-GRPO. We establish a convergence guarantee for asynchronous GMC-GRPO with explicit dependence on stochastic rollout delays. Compared with TIC-GRPO (Pang et al., 2025), GMC-GRPO yields a strictly smaller delay-dependent penalty: in the tight-clipping regime, the penalty improves from O(ϵ−4)O(ε^-4) to O(ϵ−2)O(ε^-2), where 1+ϵ1+ε is the threshold for individual importance ratios in TIC-GRPO and their group average in GMC-GRPO. Moreover, because GMC-GRPO rescales a group only when its average importance ratio exceeds the stability cap, its correction bias vanishes as G grows and the group average concentrates. In contrast, trajectory-wise clipping acts separately on every ratio, leaving a nonzero clipping bias even in the large-group limit. • Experiments. Finally, we evaluate GMC-GRPO under both controlled fixed delays and in a simulated asynchronous pipeline where rollout staleness is correlated with response length. Across Qwen3 models and reasoning benchmarks, GMC-GRPO achieves the best performance among vanilla GRPO-style algorithms and other baselines designed for asynchronous training. These results demonstrate effectiveness of our theory-guided design under asynchronous rollout delays. Related Work. We defer the comprehensive discussion of related work to Appendix A. Our paper is most closely related to off-policy correction for stale LLM rollouts in asynchronous RL systems (Noukhovitch et al., 2025; Fu et al., 2026; Hu et al., 2026). Typical strategies include adaptive clipping thresholds (Roux et al., 2025; Xi et al., 2026; Zhao et al., 2026; Li et al., 2025), variance-aware step-size control (Zheng et al., 2026; Huang et al., 2026; Xu et al., 2026), and reuse of stale experience through staged training with selective filtering or replay buffers (Tian et al., 2026; Arnal et al., 2026; Ritter et al., 2026). Despite the empirical success of these methods, few works have studied GRPO-style algorithms from a theoretical perspective, especially under asynchronous rollout delays. Zhou et al. (2026) characterize the GRPO gradient as a U-statistic and analyze its finite-sample error and dependence on group size, while Pang et al. (2025) introduce TIC-GRPO and provide the first rigorous convergence analysis of GRPO-style methods. Nevertheless, these studies are not under asynchronous training. More recently, Xu et al. (2026) analyze asynchronous GRPO through a generic biased-gradient framework, but leave the staleness-induced bias and higher-order remainder terms as abstract quantities in the final bound. To the best of our knowledge, we are the first to provide an explicit delay-dependent convergence analysis of GRPO-style algorithms in asynchronous RL systems. 2 Preliminaries 2.1 RL post-training We denote πθ _θ as an autoregressive language model parameterized by θ∈ℝdθ ^d. Given a prompt x drawn from a training distribution D over the prompt space X, the likelihood of a response y=(y1,…,yT)y=(y_1,…,y_T) is πθ(y∣x)=∏t=1Tπθ(yt∣x,y<t) _θ(y x)= _t=1^T _θ(y_t x,y_<t), where A denotes the token vocabulary, y<t:=(y1,…,yt−1)y_<t:=(y_1,…,y_t-1) is the generated prefix, and T is the maximum response length. A verifier rϕ:×T→[0,R]r_φ:X×A^T→[0,R] scores each completed response. The goal of post-training is to maximize the expected verifier score. Following standard practice in reinforcement learning, we divide each global batch into mini-batches and update the parameters sequentially. The rollout samples used by an update may therefore be generated by a behavior policy πθbehav _ _behav that differs from the current policy. Using trajectory-level importance sampling to account for this mismatch, we write the objective as J(θ) J(θ) :=x∼y∼πθ(⋅|x)[rϕ(x,y)]=x∼y∼πθbehav(⋅∣x)[ρrϕ(x,y)], :=E_x E_y _θ(·|x) [r_φ(x,y) ]=E_x E_y _ _behav(· x) [ρ\,r_φ(x,y) ], (1) where ρ:=πθ(y∣x)/πθbehav(y∣x)ρ:= _θ(y x)/ _ _behav(y x). We further denote the optimal value by J⋆:=supθ∈ℝdJ(θ)J := _θ ^dJ(θ). To find an optimal solution, at learner iteration n∈1,…,Nn∈\1,…,N\, where N≥1N≥ 1 is the total number of updates, GRPO style algorithms draws B prompts x(1),…,x(B)x^(1),…,x^(B) from D and a group of G≥2G≥ 2 responses y(b,1),…,y(b,G)∼πθbehav(⋅∣x(b))y^(b,1),…,y^(b,G) _ _behav(· x^(b)) per prompt, and takes the ascent step θn+1=θn+ηgnIS _n+1= _n+η\,g_n^IS with step size η>0η>0, starting from θ0 _0 and ending at θN _N. Here gnISg_n^IS is the stochastic policy gradient of eq. 1 on the returned mini-batch, normalized by the response length T, gnIS:=1TBG∑b=1B∑i=1Gρn(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i),g_n^IS:= 1TBG _b=1^B _i=1^G _n^(b,i)∇ _ _n (y^(b,i) x^(b) )\, A^(b,i), (2) where ρn(b,i):=πθn(y(b,i)∣x(b))/πθbehav(y(b,i)∣x(b)) _n^(b,i):= _ _n(y^(b,i) x^(b))/ _ _behav(y^(b,i) x^(b)) is the importance ratio evaluated at θn _n, and A^(b,i):=r(b,i)−1G−1∑j≠ir(b,j) A^(b,i):=r^(b,i)- 1G-1 _j≠ ir^(b,j) with r(b,i):=rϕ(x(b),y(b,i))r^(b,i):=r_φ(x^(b),y^(b,i)) being the leave-one-out group advantage. Both the ratio and the score function ∇θlogπθ(y∣x) _θ _θ(y x) act on the whole trajectory, so a single scalar advantage is shared by all tokens of y(b,i)y^(b,i). The policy gradient in eq. 2 depends on the policy through its score function and the importance sampling ratio. Our later analysis will be based on the following token-level regularity and coverage assumptions. Assumption 1 (Token-level Lipschitz score). There is L>0L>0 such that, for every token context and all θ,θ′θ,θ , ‖∇θlogπθ(yt∣x,y<t)−∇θlogπθ′(yt∣x,y<t)‖2≤L‖θ−θ′‖2. \| _θ _θ(y_t x,y_<t)- _θ _θ (y_t x,y_<t) \|_2≤ L \|θ-θ \|_2. Assumption 2 (Token-level coverage). There exists ≥1C≥ 1 such that πθ(yt∣x,y<t)≥1/ _θ(y_t x,y_<t)≥ 1/C for every token, context, and learner or behavior policy encountered during training. Although the constant C can be large in LLM settings, it appears only through log in our analytical bounds, which justifies it as a controlled constant. 2.2 Asynchronous RL Systems and Data Staleness As illustrated in Section 2.2, we consider an asynchronous RL system in which multiple rollout workers generate responses continuously while a single learner updates the policy (Fu et al., 2026; Hu et al., 2026; Noukhovitch et al., 2025). Each worker draws a prompt x∼x and generates a group of G responses under the policy snapshot it holds at launch. Once all responses have been scored, the worker deposits the group and its behavior-policy version into a shared buffer. At iteration n, the learner takes B completed groups from the buffer, performs the update in eq. 2 at θn _n, and publishes θn+1 _n+1 while rollout generation continues. Since groups start under different snapshots and take different amounts of time to generate, score, and wait in the buffer, a single mini-batch can mix several degrees of staleness. Formally, let sn(b)∈0,…,ns_n^(b)∈\0,…,n\ denote the parameter-version index used to generate group b consumed at iteration n, and define its delay as τn(b):=n−sn(b) _n^(b):=n-s_n^(b), the lag between the behavior-policy version and the current learner version, measured in learner updates. All G responses in a group share one delay, whereas groups within a mini-batch need not. For example, a batch used at θ8 _8 may contain groups generated under θ5 _5 and θ7 _7, with delays 33 and 11. The behavior policy of Section 2.1 is thus group dependent, y(b,i)∼πθn−τn(b)(⋅∣x(b))y^(b,i) _ _n- _n^(b)(· x^(b)), and the ratio in eq. 2 becomes ρn(b,i)=πθn(y(b,i)∣x(b))πθn−τn(b)(y(b,i)∣x(b)). _n^(b,i)= _ _n(y^(b,i) x^(b)) _ _n- _n^(b)(y^(b,i) x^(b)). (3) To state the sampling conditions for eq. 3, let ℱn−F_n^- collect the randomness available before the mini-batch is returned, so that θmm=0n\ _m\_m=0^n is ℱn−F_n^--measurable while the prompts and delays carried by the current batch need not be, and let ℋn:=σ(ℱn−,x(b),τn(b)b=1B)H_n:=σ(F_n^-,\x^(b), _n^(b)\_b=1^B). Every behavior parameter θn−τn(b) _n- _n^(b) is ℋnH_n-measurable, so eq. 3 can be analyzed with the behavior policy held fixed. For our analysis, we impose the following regularity assumption on asynchronous sampling. Figure 1: Asynchronous rollout. Each worker generates a group using its launch-time policy, a batch may contain groups with different delays. Assumption 3 (Asynchronous sampling regularity). Conditional on ℋnH_n, the responses within each returned group are independent draws from πθn−τn(b)(⋅∣x(b)) _ _n- _n^(b)(· x^(b)). The returned mini-batch preserves conditional unbiasedness of the raw importance-sampling update: [gnIS∣ℱn−]=T−1∇J(θn)E[g_n^IS _n^-]=T^-1∇ J( _n). Queue admission and mini-batch selection may otherwise depend on launch and completion times. For the sharper delay-moment conversion only, we additionally require the delay-regularity condition [‖θn−θn−t‖22∣τn(b)=t]=‖θn−θn−t‖22E[ \| _n- _n-t \|_2^2 _n^(b)=t]=E \| _n- _n-t \|_2^2 whenever Pr(τn(b)=t)>0 ( _n^(b)=t)>0. Assumption 3 is a natural abstraction for stationary, large-scale RL training. With a fixed prompt stream, lossless rollout collection, and many concurrent workers, the delay of any individual rollout is approximately decoupled from both the distribution of returned samples and the aggregate learner trajectory. Thus, delay affects only the staleness of a rollout, without systematically biasing the returned data or the associated learner drift. We exclude RL training systems that explicitly filter rollouts by completion time, such as timeouts or maximum-delay gating (Fu et al., 2026; Zeng et al., 2026), as they can favor shorter responses and introduce length bias (Wang et al., 2025). 3 Second Moment-Bias tradeoff and Group-Mass-Capping GRPO How we design gradient estimators plays an important role in convergence, particularly in asynchronous training. In this section, we first establish a general convergence bound for a broad class of gradient estimators, which shows how their second moments and bias affect the convergence. This bound will then guide us toward a principled gradient estimator that minimizes a bias upper bound while maintaining a common second-moment guarantee, thereby reducing sensitivity to asynchronous rollout delays. Lemma 1. Suppose that θn+1=θn+ηgn _n+1= _n+η g_n and [gn∣ℱn−]=T−1∇J(θn)+bnE[g_n _n^-]=T^-1∇ J( _n)+b_n. Under Assumption 1-2, for every η>0η>0 and N≥1N≥ 1, ∑n=0N−1‖∇J(θn)‖22N≤2T(J⋆−[J(θ0)])ηN+T2N∑n=0N−1[ηRL(2log||+1)‖gn‖22+‖bn‖22]. _n=0^N-1E \|∇ J( _n) \|_2^2N\!≤\! 2T (J \!-\!E[J( _0)] )η N\!+\! T^2N _n=0^N-1 [η RL(2 |A|\!+\!1)E \|g_n \|_2^2\!+\!E \|b_n \|_2^2 ]. The proof of Lemma 1 is deferred to Appendix D. Lemma 1 characterizes the convergence bound into three terms. The first is the standard decay term O(1/(ηN))O(1/(η N)), which vanishes as the number N of iterations becomes large. The other two depend on the gradient estimator’s second moment ‖gn‖22E \|g_n \|_2^2 and its squared bias relative to the current policy gradient ‖bn‖22E \|b_n \|_2^2. A good gradient estimator should therefore keep both quantities small. A common way to prevent gradient’s second moment become large is to clip individual samples whose importance ratios exceed a prescribed threshold, as in GRPO and its variants. However, clipping can also enlarge bias and slow convergence (Zheng et al., 2026). This tradeoff becomes especially important in asynchronous post-training, where delayed rollouts increase the mismatch between the behavior and learner policies. As this mismatch grows, more importance ratios may exceed the clipping threshold, increasing the bias introduced by clipping. Although raising the clipping threshold allows more samples to contribute, it weakens control over the gradient estimator’s second moment and can lead to unstable training or even collapse (Zheng et al., 2026; Huang et al., 2026; Tian et al., 2026). This tradeoff motivates us to consider a class of weighted gradient estimators, in which each sample receives an additional weight to adjust its contribution. We require the average weighted importance ratio within each group to stay below a fixed threshold, while allowing individual weighted ratios to exceed it. As we show below, this constraint gives all estimators in the class a common second-moment guarantee. We can then compare different weighting rules and choose one that minimizes the bias bound. Definition 1 (Weighted gradient estimator class). Define the weighted gradient estimator as gnw:=1B∑b=1B1TG∑i=1Gn(b,i)ρn(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i),g_n^w:= 1B _b=1^B 1TG _i=1^G w_n^(b,i) _n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i), (4) where the weights wn(b,i)w_n^(b,i) should satisfy, for every realized group, 0≤wn(b,i)≤1for all i∈[G],1G∑i=1Gwn(b,i)ρn(b,i)≤1+ϵ. 0≤ w_n^(b,i)≤ 1 all i∈[G], 1G _i=1^Gw_n^(b,i) _n^(b,i)≤ 1+ε. (5) Remark 1. In eq. 5, the first condition prevents the correction from amplifying any trajectory, and the second precisely controls the group’s average gradient second moment. Such a class is broad, and includes many existing algorithms as special cases. For example wn(b,i)=ρn(b,i)≤1+ϵw_n^(b,i)=1\ _n^(b,i)≤ 1+ε\ corresponds to TIC-GRPO (Pang et al., 2025); wn(b,i)=min1,c/ρn(b,i)w_n^(b,i)= \1,c/ _n^(b,i)\ corresponds to VCPO (Huang et al., 2026) which applies sequence-level truncated importance sampling with a detached weight minρ,c \ρ,c\; and CISPO (Chen et al., 2025) and SAPO (Gao et al., 2025) can be viewed as token-level analogues. However, the vanilla GRPO (Shao et al., 2024) is not in this class, because for negative advantages, it retains the gradient of tokens with arbitrarily large ratios, so that its second moment scales as O(2)O(C^2) in the worst case (Pang et al., 2025). The following proposition shows that all gradient estimators in this class satisfy a common second-moment bound. This allows us to design a principled bias-minimal weighting rule. Proposition 1. Under Assumptions 1-3, the gradient estimators in the class given in Definition 1 share the following common bound [‖gnw‖22∣ℱn−]≤8T(1+ϵ)LR2log||. [ \|g_n^w \|_2^2 _n^-]≤ 8T(1+ε)LR^2 |A|. (6) Further, their group bias bn,bw:=−1TG∑i=1G(1−wn(b,i))ρn(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i)b_n,b^w:=- 1TG _i=1^G(1-w_n^(b,i)) _n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i) satisfies ‖[bn,bw∣ℋn]‖2≤2R2Llog[1G∑i=1G(1−wn(b,i))ρn(b,i)∣ℋn]. \|E[b_n,b^w _n] \|_2≤ 2R 2L \,E [ 1G _i=1^G(1-w_n^(b,i)) _n^(b,i) _n ]. (7) Proposition 1 shows that all weighting rules in this class share a common second-moment guarantee. With this guarantee in place, we focus on choosing weights that reduce the bias. Ideally, we would minimize the exact bias ‖[bn,bw∣ℋn]‖2\|E[b_n,b^w _n]\|_2. However, this quantity depends on the directions of individual gradient contributions and how they cancel, which cannot generally be determined from the importance ratios alone. We therefore minimize the upper bound in eq. 7, which depends only on the importance ratios and the chosen weights. This bound is proportional to the expected reduction in the group’s average importance ratio caused by weighting. Since the unweighted average 1G∑i=1Gρn(b,i) 1G _i=1^G _n^(b,i) is fixed for each realized group, minimizing this reduction is equivalent to maximizing the average weighted importance ratio subject to the class constraints: max0≤wn(b,i)≤11G∑i=1Gwn(b,i)ρn(b,i)subject to1G∑i=1Gwn(b,i)ρn(b,i)≤1+ϵ. _0≤ w_n^(b,i)≤ 1\ 1G _i=1^Gw_n^(b,i) _n^(b,i) to 1G _i=1^Gw_n^(b,i) _n^(b,i)≤ 1+ε. (8) The optimal weights are not unique in general. When the unweighted average importance ratio exceeds 1+ϵ1+ε, any feasible weighting rule whose average weighted importance ratio equals 1+ϵ1+ε attains the optimum. Different optimal rules can nevertheless produce different gradient directions within the same group. To select among these rules, we restrict attention to direction-preserving rules that assign a common positive weight to all responses in a group: wn(b,i)=αn(b)>0w_n^(b,i)= _n^(b)>0 for all i∈[G]i∈[G]. This scales the original importance-weighted group gradient estimator by a single factor, preserving the relative contributions of the responses and hence the direction of the group gradient estimate. The following proposition gives the optimal weighting rule under this restriction. Proposition 2. Fix a realized group and let ρ¯n(b):=G−1∑i=1Gρn(b,i) ρ_n^(b):=G^-1 _i=1^G _n^(b,i). Among weighted direction-preserving corrections, the unique solution to eq. 8 is αn(b),⋆=min1,(1+ϵ)/ρ¯n(b). _n^(b), = \1,(1+ε)/ ρ_n^(b) \. We call this weighting rule group mass capping (GMC), which caps the group’s average importance ratio by replacing each raw ratio with ρ^n(b,i):=αn(b),⋆ρn(b,i) ρ_n^(b,i):= _n^(b), _n^(b,i), yielding the following gradient estimator: gnGMC:=1TBG∑b=1B∑i=1Gρn(b,i)max1,ρ¯n(b)/(1+ϵ)∇logπθn(y(b,i)∣x(b))A^(b,i).g_n^GMC:= 1TBG _b=1^B _i=1^G _n^(b,i) \1, ρ_n^(b)/(1+ε)\∇ _ _n(y^(b,i) x^(b)) A^(b,i). (9) We provide a comprehensive pseudo code of asynchronous GMC-GRPO in Appendix C. 4 Convergence Analysis 4.1 Convergence Guarantees for GMC-GRPO We now apply Lemma 1 to establish convergence guarantees for asynchronous GMC-GRPO. Since Proposition 1 already controls the gradient second moment, it remains to bound the correction bias in terms of rollout delays. The resulting bound depends on the averaged fourth moment of the delay defined as τ¯4:=1NB∑n=0N−1∑b=1B[(τn(b))4] τ_4:= 1NB _n=0^N-1 _b=1^BE[( _n^(b))^4]. Theorem 1 (Convergence of asynchronous GMC-GRPO). Under Assumptions 1–3, for every η>0η>0 and N≥1N≥ 1, the convergence rate of GMC-GRPO satisfies 1N∑n=0N−1‖∇J(θn)‖22=O(1ηN+(1+ϵ)η+(1+ϵ)3log2(1+2ϵ)η4τ¯4). 1N _n=0^N-1E \|∇ J( _n) \|_2^2=O\! ( 1η N+(1+ε)η+ (1+ε)^3 ^2(1+2ε)η^4 τ_4 ). For τ¯4>0 τ_4>0, taking η⋆≍min1/(1+ϵ)N,(log2(1+2ϵ)/((1+ϵ)3τ¯4N))1/5η \1/ (1+ε)N,( ^2(1+2ε)/((1+ε)^3 τ_4N))^1/5\ gives 1N∑n=0N−1‖∇J(θn)‖22=O(1+ϵN+((1+ϵ)3τ¯4log2(1+2ϵ)N4)1/5). 1N _n=0^N-1E\|∇ J( _n)\|_2^2=O ( 1+εN+ ( (1+ε)^3 τ_4 ^2(1+2ε)N^4 )^1/5 ). When τ¯4=0 τ_4=0, the convergence recovers the standard O(N−1/2)O(N^-1/2) rate for fixed ϵε. Remark 2 (Effect of rollout delay on convergence rate). We use Theorem 1 to characterize how much rollout delay can be tolerated without degrading the convergence rate. Suppose τ¯4>0 τ_4>0. If τ¯4=O(N3/2) τ_4=O(N^3/2), choosing ϵ≍min1,τ¯41/7N−3/14ε \1, τ_4^1/7N^-3/14\ and η≍N−1/2η N^-1/2 preserves the no-delay O(N−1/2)O(N^-1/2) convergence guarantee; the delay contribution is lower order if τ¯4=o(N3/2) τ_4=o(N^3/2). For larger scale delays, where τ¯4=Θ(Np) τ_4= (N^p) with 3/2<p<43/2<p<4, the rate-optimal choices are ϵ=Θ(1)ε= (1) and η≍N−(p+1)/5η N^-(p+1)/5, which yield the slower convergence guarantee O(N−(4−p)/5)O(N^-(4-p)/5). Thus, the bound tolerates fourth delay moments up to N3/2N^3/2 without degrading the standard rate. Under polynomial scaling, moments below N4N^4 still yield a vanishing convergence guarantee, but at a slower convergence rate. Corollary 1 (Asymptotically optimal GMC threshold). Under the conditions of Theorem 1, assume that τ¯4>0 τ_4>0 does not scale with N. Then as N→∞N→∞, any minimizer of the leading ϵε-dependent terms over ϵ>0ε>0 in the derived GMC upper bound satisfies ϵGMC⋆=Θ(τ¯41/7N−3/14).ε_GMC = ( τ_4^1/7N^-3/14 ). Corollary 1 reflects a simple intuition: larger delays favor a higher threshold to avoid reducing samples’ contributions too aggressively, while more training iterations favor a lower threshold to keep the gradient estimator’s second moment small. Corollary 2. Under the conditions of Theorem 1, further suppose that for some finite δ≥0δ≥ 0, |ρn(b,i)−1|≤δ| _n^(b,i)-1|≤δ almost surely for all n,b,in,b,i. For τ¯4>0 τ_4>0, taking η⋆≍min1/(1+ϵ)N,(G2ϵ2/((1+δ)2(1+ϵ)3τ¯4N))1/5η \1/ (1+ε)N,(G^2ε^2/((1+δ)^2(1+ε)^3 τ_4N))^1/5\, then the group-concentration branch in the GMC bound gives 1N∑n=0N−1‖∇J(θn)‖22=O(1+ϵN+((1+δ)2(1+ϵ)3τ¯4G2ϵ2N4)1/5). 1N _n=0^N-1E\|∇ J( _n)\|_2^2=O ( 1+εN+ ( (1+δ)^2(1+ε)^3 τ_4G^2ε^2N^4 )^1/5 ). Corollary 2 shows that the delay-dependent term decreases as G−2/5G^-2/5, so larger groups reduce the effect of rollout delays on the convergence bound. Remark 3. Corollary 2 reflects a simple distinction between group-mass capping (GMC) and individual clipping. Importance ratios have conditional mean one under the behavior policy, and GMC intervenes only when their empirical group average is unusually large. As the group size grows, this average concentrates around one, so this excess as well as the bias introduced by correcting it vanishes. In contrast, individual clipping makes a separate decision for every trajectory (Pang et al., 2025; Huang et al., 2026); averaging more such decisions does not remove the clipping bias already introduced at the trajectory level. The condition |ρn(b,i)−1|≤δ| _n^(b,i)-1|≤δ in Corollary 2 identifies a near-policy regime in which this concentration benefit carries over to delayed training. When the behavior and current policies assign comparable likelihoods to the sampled trajectories, the importance ratios remain bounded, and the delay term decays with the group size. As this overlap deteriorates, the constant (1+δ)2(1+δ)^2 grows and the refinement loses its advantage, although Theorem 1 continues to apply. The group-size gain should be interpreted as a local benefit of aggregating trajectories with controlled likelihood ratios, rather than a universal claim that larger groups eliminate bias under arbitrary distribution shift. 4.2 Proof Outline and the Role of Rollout Delay We outline the proof of Theorem 1, which further explains how rollout delay affects convergence of asynchronous training. Theorem 1 follows from Lemma 1 by bounding the second moment and bias term separately. Proposition 1 controls the second-moment term uniformly, independently of delay, which yields the O(η)O(η) term. Delay therefore enters only through the correction bias, via a three-step chain. First, Proposition 1 bounds the bias of each group by the importance ratio it removes, which for GMC equals the group average’s excess over the cap: [(ρ¯n(b)−(1+ϵ))+∣ℋn]E[( ρ_n^(b)-(1+ε))_+ _n]. Second, a convexity argument bounds this excess by Dn(b)/log(1+2ϵ)D_n^(b)/ (1+2ε), where Dn(b):=DKL(πθn(⋅∣x(b))∥πθn−τn(b)(⋅∣x(b)))D_n^(b):=D_KL( _ _n(· x^(b))\,\|\, _ _n- _n^(b)(· x^(b))) measures the mismatch between the current and behavior policies. Third, Assumption 1 gives Dn(b)≤TL2‖θn−θn−τn(b)‖22D_n^(b)≤ TL2\| _n- _n- _n^(b)\|_2^2, and the uniform second-moment bound limits the parameter drift accumulated over τn(b) _n^(b) updates to O(η2(τn(b))2)O(η^2( _n^(b))^2). Squaring the bias thus produces the term of order η4τ¯4/log2(1+2ϵ)η^4 τ_4/ ^2(1+2ε). The second-moment control therefore plays a dual role: it stabilizes each update, and it also limits how far the policy can drift within a delay window. Remark 4 (Comparison with asynchronous SGD). Delay plays a different role in Theorem 1 than in classical asynchronous stochastic gradient descent (ASGD) (Tan et al., 2023; Maranjyan et al., 2025; Koloskova et al., 2022; Mishchenko et al., 2022). In ASGD, an update applied at iteration n is conditionally unbiased for the gradient at the stale iterate θn−τn _n- _n rather than at the current iterate θn _n, so staleness enters the descent argument directly through the discrepancy between the two gradients. In our setting, the trajectory-level ratio instead changes measure from the behavior policy πθn−τn(b) _ _n- _n^(b) to the current policy πθn _ _n, so the uncorrected estimator remains unbiased for ∇J(θn)∇ J( _n) regardless of delay. Staleness instead enlarges the spread of the importance ratios, which makes a second-moment correction necessary; delay then affects the bound only through the bias of that correction as explained above. This difference also changes the appropriate remedy. ASGD commonly bounds staleness by discarding updates whose delay exceeds a threshold (Tan et al., 2023; Maranjyan et al., 2025), but the analogous gating in LLM post-training preferentially discards long responses and thus induces a length bias (Wang et al., 2025). Improving the delay dependence therefore calls for RL algorithms that minimize the bias under the same second-moment control, which is precisely what group-mass-capping is designed to do. 4.3 Comparison with TIC-GRPO We now compare GMC-GRPO with TIC-GRPO (Pang et al., 2025), which replaces the token-level ratios of GRPO with a single trajectory-level ratio and clips it from above, corresponding to wn(b,i)=ρn(b,i)≤1+ϵw_n^(b,i)=1\ _n^(b,i)≤ 1+ε\ in eq. 4. TIC-GRPO is a natural baseline for two reasons. First, it provably converges faster than GRPO (Pang et al., 2025). Second, it belongs to the weighted gradient estimator class and therefore shares the second-moment bound of Proposition 1 with GMC-GRPO. The following theorem establishes the convergence of TIC-GRPO under stochastic delays. Theorem 2 (Convergence of asynchronous TIC-GRPO). Let cϵ:=log(1+ϵ)−ϵ/(1+ϵ)c_ε:= (1+ε)-ε/(1+ε). Under Assumptions 1–3, for every η>0η>0 and N≥1N≥ 1, TIC-GRPO satisfies the bound of Theorem 1 with log2(1+2ϵ) ^2(1+2ε) replaced by cϵ2c_ε^2. Consequently, for τ¯4>0 τ_4>0, taking η⋆≍min1/(1+ϵ)N,(cϵ2/((1+ϵ)3τ¯4N))1/5η \1/ (1+ε)N,(c_ε^2/((1+ε)^3 τ_4N))^1/5\ gives 1N∑n=0N−1‖∇J(θn)‖22=O(1+ϵN+((1+ϵ)3τ¯4cϵ2N4)1/5). 1N _n=0^N-1E\|∇ J( _n)\|_2^2=O ( 1+εN+ ( (1+ε)^3 τ_4c_ε^2N^4 )^1/5 ). Remark 5 (Convergence bound improvement). Theorems 1 and 2 differ only in the fourth-order delay coefficient: TIC-GRPO replaces log2(1+2ϵ) ^2(1+2ε) in the GMC-GRPO bound with cϵ2c_ε^2. Since cϵ<log(1+2ϵ)c_ε< (1+2ε) for every ϵ>0ε>0, the GMC coefficient is strictly smaller, and after optimizing the step size the delay-dependent rate improves by the factor (cϵ/log(1+2ϵ))2/5<1(c_ε/ (1+2ε))^2/5<1. This distinction is substantial for a tight threshold. As ϵ→0ε→ 0, cϵ=Θ(ϵ2)c_ε= (ε^2) while log(1+2ϵ)=Θ(ϵ) (1+2ε)= (ε); hence the TIC penalty scales as ϵ−4ε^-4, whereas the GMC penalty scales as ϵ−2ε^-2. GMC obtains this improvement without enlarging the conditional second-moment upper bound. This improvement arises from how much importance weight each algorithm removes. Following the discussion in Section 4.2, GMC-GRPO removes only the amount by which the group average exceeds 1+ϵ1+ε, whereas TIC-GRPO removes the full contribution of every trajectory above the threshold. The KL divergence between the current and behavior policies bounds these two quantities with denominators log(1+2ϵ) (1+2ε) and cϵc_ε, respectively, producing the coefficient difference above. Finally, the delay term in Theorem 2 does not improve with the group size, while GMC-GRPO benefits from larger groups under local overlap (Remark 3). Section 5 examines whether these theoretical gaps translate into faster training in practice. Figure 2: Sampled Pass@1 on MATH500 across training checkpoints under fixed rollout delays. GMC-GRPO remains robust as staleness increases, substantially outperforming vanilla GRPO and competitive with or exceeding TIC-GRPO. 5 Experiments In this section, we evaluate GMC-GRPO against several baselines: (i) vanilla GRPO (Shao et al., 2024); (i) GAC, which controls stale-aligned update directions through gradient projection (Xu et al., 2026); (i) VCPO, which combines effective-sample-size-guided step-size scaling with an off-policy minimum-variance baseline (Huang et al., 2026); and (iv) TIC-GRPO (Pang et al., 2025). Among these methods, GAC and VCPO are designed specifically for asynchronous training. We first evaluate robustness under controlled fixed delays and then construct a simulated asynchronous system in which rollout delay is correlated with response length. Detailed parameter settings and implementation details are deferred to Appendix B.1. We also discuss related approaches that selectively clip a fraction of importance ratios (Fu et al., 2026; Li et al., 2025) and report an additional comparison with A-3PO (Li et al., 2025) in Appendix B.2. Controlled Fixed Delays. To isolate the effect of staleness, we evaluate GRPO, TIC-GRPO, and GMC-GRPO on Qwen3-1.7B under fixed rollout delays. Figure 2 reports sampled Pass@1 on MATH500 for delays of D=0,64,128D=0,64,128 learner updates; see Appendix B.1. The full sweep over D∈0,32,64,128,256,512D∈\0,32,64,128,256,512\ is deferred to Appendix B.2. Without delay, all three methods attain similar final performance. At delay 64, GMC-GRPO learns faster than TIC-GRPO and reaches comparable final Pass@1, with both outperforming vanilla GRPO. At delay 128, GMC-GRPO achieves 59.03% Pass@1, compared with 55.41% for TIC-GRPO and 35.33% for vanilla GRPO. These results show that GMC supports faster learning and stronger final performance as staleness increases. Simulated Asynchronous Delays. The fixed-delay study isolates staleness, but real rollout workers finish at different times: longer responses typically take longer to generate and thus reach the learner under older policies. To capture this coupling, we simulate an asynchronous pipeline in which rollout delay depends on generation length, producing heterogeneous delays averaging approximately 160 learner updates for Qwen3-1.7B and 140 for Qwen3-4B and Qwen3-8B. This setting tests whether GMC-GRPO remains robust when staleness varies across rollouts. Among the methods reported in Table 1, GMC-GRPO achieves the highest mean avg@n scores across all three model sizes (41.47, 56.20, and 59.62) and the highest mean greedy scores on Qwen3-1.7B and Qwen3-8B (48.77 and 64.37). On Qwen3-4B, its mean greedy score of 60.63 is 0.83 points below TIC-GRPO. Notably, GMC-GRPO also outperforms the two of the most recent asynchronous-RL stabilization methods: GAC, which controls stale-aligned update directions via gradient projection, and VCPO, which combines ESS-guided step-size scaling with a minimum-variance baseline. Both improve over vanilla GRPO under delay but remain below TIC-GRPO-style trajectory correction at 4B and 8B. These results indicate that the robustness of group-mass capping is not restricted to fixed delays, but holds under heterogeneous, length-dependent staleness. Table 1: Performance under simulated asynchronous delays. Each benchmark cell is greedy/avg@n, with n=16n=16 for MATH500, Minerva, and AMC23, and n=8n=8 for GSM8K, Gaokao, and OlympiadBench. Avg. perf. is the average over the six benchmarks, reported as mean greedy/mean avg@n. Method Avg. Delay MATH500 GSM8K Gaokao Olympiad Minerva AMC23 Avg. perf. Qwen3-1.7B GRPO 158.70 48.20 / 31.04 72.18 / 47.90 39.74 / 26.27 14.69 / 6.77 16.18 / 8.32 22.50 / 9.53 35.58 / 21.64 GAC 160.02 47.00 / 31.03 72.18 / 47.09 40.26 / 25.16 15.73 / 6.70 15.81 / 8.18 20.00 / 11.09 35.16 / 21.54 VCPO 157.87 65.20 / 54.55 80.21 / 67.76 55.06 / 45.06 27.60 / 20.49 26.47 / 18.31 27.50 / 26.72 47.01 / 38.82 TIC-GRPO 159.60 62.20 / 53.61 82.34 / 67.14 53.25 / 43.02 27.60 / 19.51 26.84 / 17.53 32.50 / 26.25 47.45 / 37.84 GMC-GRPO 159.75 66.00 / 57.29 81.65 / 72.66 55.32 / 47.73 28.19 / 22.61 26.47 / 20.11 35.00 / 28.44 48.77 / 41.47 Qwen3-4B GRPO 137.92 57.40 / 43.26 74.30 / 51.56 48.57 / 34.35 22.11 / 9.66 18.01 / 11.99 35.00 / 17.19 42.57 / 28.00 GAC 138.85 58.00 / 42.39 77.79 / 55.32 50.39 / 33.64 21.81 / 10.55 16.18 / 12.22 35.00 / 18.91 43.19 / 28.84 VCPO 139.50 61.00 / 48.39 76.88 / 64.30 46.75 / 36.30 29.38 / 13.61 26.84 / 16.98 37.50 / 21.88 46.39 / 33.58 TIC-GRPO 138.87 76.40 / 68.31 90.75 / 81.70 65.19 / 55.68 42.88 / 33.33 36.03 / 27.55 57.50 / 43.91 61.46 / 51.75 GMC-GRPO 141.01 77.60 / 72.54 91.13 / 87.03 65.19 / 60.91 43.18 / 37.72 34.19 / 30.10 52.50 / 48.91 60.63 / 56.20 Qwen3-8B GRPO 140.90 61.80 / 49.41 88.70 / 72.74 55.06 / 38.83 27.30 / 16.49 21.69 / 16.96 25.00 / 22.03 46.59 / 36.08 GAC 141.7 61.20 / 48.34 88.10 / 72.09 54.55 / 39.29 26.41 / 16.26 22.79 / 15.99 27.50 / 21.41 46.76 / 35.56 VCPO 142.7 67.40 / 58.88 88.93 / 78.79 57.40 / 47.50 33.38 / 21.88 31.25 / 23.09 47.50 / 31.56 54.31 / 43.62 TIC-GRPO 141.6 79.20 / 75.14 92.87 / 87.92 68.31 / 61.20 45.10 / 38.41 40.07 / 34.81 60.00 / 50.47 64.26 / 57.99 GMC-GRPO 141.1 78.80 / 75.22 93.33 / 91.37 69.09 / 63.64 44.96 / 40.34 40.07 / 34.35 60.00 / 52.81 64.37 / 59.62 6 Conclusions and Limitations This work studies how rollout staleness affects the convergence of GRPO-style algorithms in asynchronous LLM training and introduces GMC-GRPO to improve learning from delayed rollouts. Our theory and experiments show that group-level capping improves robustness to stale rollouts while maintaining performance on fresh data. Within the proposed class of weighted gradient estimators, our analysis provides a foundation for understanding and improving delay tolerance. Future work can extend this framework to more general weighting rules, such as those that use information beyond importance ratios, to further improve the balance between second-moment control and bias. References Arnal et al. (2026) Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe, and Remi Munos. Efficient rl training for llms with experience replay. arXiv preprint arXiv:2604.08706, 2026. Art of Problem Solving (2024) Art of Problem Solving. AMC problems and solutions. https://artofproblemsolving.com/wiki/index.php/AMC_Problems_and_Solutions, 2024. Accessed: 2026-08-29. Bartoldson et al. (2026) Brian Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando Ceron, Yoshua Bengio, and Bhavya Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable llm post-training. Advances in Neural Information Processing Systems, 38:113901–113931, 2026. Chen et al. (2025) Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dettmers et al. (2021) Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021. Du et al. (2026) Zhenbang Du, Jun Luo, Zhiwei Zheng, Xiangchi Yuan, Kejing Xia, Dachuan Shi, Qirui Jin, Qijia He, Shaofeng Zou, Yingbin Liang, et al. Pact: Privileged trace co-training for multi-turn tool-use agents. arXiv preprint arXiv:2606.16215, 2026. Dwyer et al. (2025) Madeleine Dwyer, Adam Sobey, and Adriane Chapman. It’s not you, it’s clipping: A soft trust-region via probability smoothing for llm rl. arXiv preprint arXiv:2509.21282, 2025. Foster et al. (2025) Dylan J Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025. Fu et al. (2026) Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. Advances in Neural Information Processing Systems, 38:36256–36282, 2026. Gao et al. (2025) Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347, 2025. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850, 2024. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Hu et al. (2026) Tianhao Hu, Xiangcheng Liu, Yuchun Miao, Youshao Xiao, Hongyu Zang, Yang Zheng, Xuan Huang, Jinrui Ding, Yufei Zhang, Yu Yang, et al. Dora: A scalable asynchronous reinforcement learning system for language model training. arXiv preprint arXiv:2604.26256, 2026. Huang et al. (2025a) Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. In International Conference on Learning Representations, volume 2025, p. 76687–76739, 2025a. Huang et al. (2025b) Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason Lee, Wen Sun, Akshay Krishnamurthy, and Dylan Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In International Conference on Learning Representations, volume 2025, p. 92647–92701, 2025b. Huang et al. (2026) Luke J Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, and Song Han. Stable asynchrony: Variance-controlled off-policy rl for llms. arXiv preprint arXiv:2602.17616, 2026. Jaech et al. (2024) Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. Jain et al. (2025) Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, volume 2025, p. 58791–58831, 2025. Joulani et al. (2013) Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback. In International conference on machine learning, p. 1453–1461. PMLR, 2013. Koloskova et al. (2022) Anastasiia Koloskova, Sebastian U Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous sgd for distributed and federated learning. Advances in Neural Information Processing Systems, 35:17202–17215, 2022. Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626, 2023. Lambert et al. (2024) Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857, 2022. Li et al. (2025) Xiaocan Li, Shiliang Wu, and Zheng Shen. A-3po: Accelerating asynchronous llm training with staleness-aware proximal policy approximation. arXiv preprint arXiv:2512.06547, 2025. Liao et al. (2024) Minpeng Liao, Chengxi Li, Wei Luo, Wu Jing, and Kai Fan. Mario: Math reasoning with code interpreter output-a reproducible pipeline. In Findings of the Association for Computational Linguistics: ACL 2024, p. 905–924, 2024. Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, p. 39578–39601, 2024. Liu et al. (2025) Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. Maranjyan et al. (2025) Artavazd Maranjyan, Alexander Tyurin, and Peter Richtárik. Ringmaster asgd: The first asynchronous sgd with optimal time complexity. arXiv preprint arXiv:2501.16168, 2025. Mishchenko et al. (2022) Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake E Woodworth. Asynchronous sgd beats minibatch sgd under arbitrary delays. Advances in Neural Information Processing Systems, 35:420–433, 2022. Noukhovitch et al. (2025) Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. In International Conference on Learning Representations, volume 2025, p. 4003–4029, 2025. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Pang et al. (2025) Lei Pang, Jun Luo, and Ruinan Jin. Tic-grpo: Provable and efficient optimization for reinforcement learning from human feedback. arXiv preprint arXiv:2508.02833, 2025. Ritter et al. (2026) Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang, Kianté Brantley, and Wen Sun. Llms can learn to reason via off-policy rl. arXiv preprint arXiv:2602.19362, 2026. Roux et al. (2025) Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for llms. arXiv preprint arXiv:2503.14286, 2025. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Sheng et al. (2025) Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297, 2025. Tan et al. (2023) Qiao Tan, Feng Zhu, and Jingjing Zhang. Abs: Adaptive bounded staleness converges faster and communicates less. arXiv preprint arXiv:2301.08895, 2023. Tian et al. (2026) Minghao Tian, Yunfei Xie, and Chen Wei. How off-policy can grpo be? mu-grpo for efficient llm reinforcement learning. arXiv preprint arXiv:2605.17570, 2026. Tyurin et al. (2026) Alexander Tyurin, Andrei Spiridonov, and Varvara Rudenko. Asynchronous policy gradient aggregation for efficient distributed reinforcement learning. In International Conference on Learning Representations, volume 2026, p. 8826–8857, 2026. Wang et al. (2025) Liangyu Wang, Huanyi Xie, Xinhai Wang, Tianjin Huang, Mengdi Li, and Di Wang. Infinite sampling: Efficient and stable grouped rl training for large language models. arXiv preprint arXiv:2506.22950, 2025. Xi et al. (2026) Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Xun Deng, Zhihao Zhang, et al. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping. In International Conference on Learning Representations, volume 2026, p. 126204–126228, 2026. Xie et al. (2025) Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed H Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. In International Conference on Learning Representations, volume 2025, p. 43632–43669, 2025. Xu et al. (2026) Haofeng Xu, Junwei Su, Yukun Tian, Lansong Diao, Zhengping Qian, and Chuan Wu. Gac: Stabilizing asynchronous rl training for llms via gradient alignment control. arXiv preprint arXiv:2603.01501, 2026. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yu et al. (2026) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. Zeng et al. (2026) Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Zhao et al. (2026) Guanqun Zhao, Zijun Xie, Binbin Zheng, Enlei Gong, Jiafeng Lu, Yehan Yang, Aoqi Hu, and Zeyu Chen. Deconstructing off-policy ratios: Entropy-scaled trust regions for asynchronous reinforcement learning. arXiv preprint arXiv:2607.22186, 2026. Zheng et al. (2026) Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy rl reach with stale data on llms? In International Conference on Learning Representations, volume 2026, p. 82657–82679, 2026. Zhong et al. (2024) Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, et al. Rlhfuse: Efficient rlhf training for large language models with inter-and intra-stage fusion. arXiv preprint arXiv:2409.13221, 1, 2024. Zhou et al. (2026) Hongyi Zhou, Kai Ye, Erhan Xu, Jin Zhu, Ying Yang, Shijin Gong, and Chengchun Shi. Demystifying group relative policy optimization: Its policy gradient is a u-statistic. arXiv preprint arXiv:2603.01162, 2026. Appendix Contents A. Related Work.A B. Experiment Details and Supplemental Experiments.B B.1. Detailed Experiment Setup.B.1 B.2. Supplemental Experiment Results.B.2 C. Mixed-Delay Setting, GMC-GRPO and TIC-GRPO Algorithms.C D. Technical Preliminaries.D E. Optimization over a Second-Moment-Controlled Weighting Class.E F. Convergence Analysis of GMC-GRPO.F G. Convergence Analysis of TIC-GRPO.G H. From the Appendix Bounds to the Main-Text Rates.H Table 2: Summary of the main notation used in the paper. Notation Description ,,X,D,A Prompt space, training prompt distribution, and token vocabulary. x,y,yt,y<tx,y,y_t,y_<t Prompt, response, token at position t, and prefix (y1,…,yt−1)(y_1,…,y_t-1). T Maximum response length; also the normalization factor in each gradient update. θ,πθ(y∣x)θ, _θ(y x) Policy parameter in ℝdR^d and trajectory likelihood ∏t=1Tπθ(yt∣x,y<t) _t=1^T _θ(y_t x,y_<t). rϕ(x,y),Rr_φ(x,y),R Verifier score and its upper bound: rϕ(x,y)∈[0,R]r_φ(x,y)∈[0,R]. Jx(θ),J(θ),J⋆J_x(θ),J(θ),J Expected reward for prompt x, its average over x∼x , and supθJ(θ) _θJ(θ); see eq. 1. n,N,ηn,N,η Learner update index n∈0,…,N−1n∈\0,…,N-1\, N total updates, and step size η>0η>0. B,G,b,iB,G,b,i Groups per mini-batch, responses per group (G≥2G≥ 2), group index b∈[B]b∈[B], and response index i∈[G]i∈[G]. ℳnM_n Mini-batch of B completed prompt groups consumed at iteration n. θn,θn−τn(b) _n, _n- _n^(b) Current learner parameter and behavior parameter used to generate group b. τn(b) _n^(b) Number of learner updates between the launch and consumption of group b; τn(b)∈0,…,n _n^(b)∈\0,…,n\. τ¯k τ_k Averaged kkth delay moment: 1N∑n=0N−1[1B∑b=1B(τn(b))k] 1N _n=0^N-1E[ 1B _b=1^B( _n^(b))^k], k≥1k≥ 1. A^(b,i) A^(b,i) Leave-one-out advantage r(b,i)−1G−1∑j≠ir(b,j)r^(b,i)- 1G-1 _j≠ ir^(b,j), where r(b,i)=rϕ(x(b),y(b,i))r^(b,i)=r_φ(x^(b),y^(b,i)). ρn(b,i) _n^(b,i) Current-to-behavior trajectory likelihood ratio πθn(y(b,i)∣x(b))/πθn−τn(b)(y(b,i)∣x(b)) _ _n(y^(b,i) x^(b))/ _ _n- _n^(b)(y^(b,i) x^(b)). ρ¯n(b) ρ_n^(b) Average raw importance ratio in group b: G−1∑i=1Gρn(b,i)G^-1 _i=1^G _n^(b,i). ϵ,cϵε,c_ε Threshold parameter ϵ>0ε>0 (the cap is 1+ϵ1+ε), and cϵ=log(1+ϵ)−ϵ/(1+ϵ)c_ε= (1+ε)-ε/(1+ε). wn(b,i)w_n^(b,i) Correction weight, held fixed during differentiation, satisfying 0≤wn(b,i)≤10≤ w_n^(b,i)≤ 1 and G−1∑iwn(b,i)ρn(b,i)≤1+ϵG^-1 _iw_n^(b,i) _n^(b,i)≤ 1+ε. αn(b),⋆ _n^(b), Common GMC scaling factor min1,(1+ϵ)/ρ¯n(b) \1,(1+ε)/ ρ_n^(b)\. ρ^n(b,i) ρ_n^(b,i) GMC effective importance ratio αn(b),⋆ρn(b,i) _n^(b), _n^(b,i). gnGMC,gnTIC,gnwg_n^GMC,g_n^TIC,g_n^w Update estimators for GMC, TIC, and a general weighted correction; see eq. 9, eq. 35, and eq. 4. ℱn−,ℋnF_n^-,H_n Information before observing the returned mini-batch, and ℋn=σ(ℱn−,x(b),τn(b)b=1B)H_n=σ(F_n^-,\x^(b), _n^(b)\_b=1^B) after revealing its prompts and delays. bnb_n Conditional update bias [gn∣ℱn−]−T−1∇J(θn)E[g_n _n^-]-T^-1∇ J( _n) for the estimator gng_n under study. L,L,C Token-score Lipschitz constant and token-coverage constant (πθ(yt∣x,y<t)≥1/ _θ(y_t x,y_<t)≥ 1/C); see Assumptions 1 and 2. Dn(b),χn2,(b)D_n^(b), _n^2,(b) Trajectory KL and chi-square divergences from πθn(⋅∣x(b)) _ _n(· x^(b)) to πθn−τn(b)(⋅∣x(b)) _ _n- _n^(b)(· x^(b)); see Appendix C. δ Local-overlap bound: |ρn(b,i)−1|≤δ| _n^(b,i)-1|≤δ almost surely, for the refinement in Corollary 2. Appendix A Related Work Asynchronous RL Systems for LLM Post-Training. LLM RL pipelines typically alternate between autoregressive rollout generation and policy optimization, incurring idle time from variable response lengths and repeated switches between inference and training (Zhong et al., 2024; Sheng et al., 2025). Recent work improves this pipeline along two main directions. Asynchronous RLHF and LLM RL systems decouple rollout generation from policy optimization so that the two stages can proceed concurrently (Noukhovitch et al., 2025; Fu et al., 2026; Hu et al., 2026). Complementary approaches improve rollout-side utilization through continuous grouped sampling or scale exploration and learning through asynchronous replay (Wang et al., 2025; Bartoldson et al., 2026). This decoupling, however, causes the learner to consume trajectories generated by lagged behavior policies, making rollout staleness an intrinsic off-policy optimization issue. Rather than proposing another systems architecture, we aim at finding robust algorithms against data staleness in asynchronous RL systems: we analyze GRPO-style optimization under stochastic rollout delays and design an importance correction that controls the resulting bias without sacrificing stability. Off-Policy Correction for Stale LLM Rollouts. Stale rollouts can induce heavy-tailed importance ratios, allowing a small number of trajectories to dominate the gradient estimate and destabilize training. Learning from such data therefore requires a careful choice of how the update is corrected. Existing corrections differ mainly in where they intervene. A common strategy acts locally on individual ratios: TIC-GRPO clips a trajectory once its ratio exceeds the clipping threshold (Pang et al., 2025), while related methods replace fixed clipping with tapered weights, adaptive or asymmetric thresholds, and entropy-scaled trust regions (Roux et al., 2025; Xi et al., 2026; Zhao et al., 2026; Li et al., 2025). A second strategy controls batch- or update-level statistics through second-moment constraints, variance-aware step sizes, or gradient-alignment control (Zheng et al., 2026; Huang et al., 2026; Xu et al., 2026). A third changes how stale experience is reused through staged training with selective filtering, replay buffers, or objectives designed for lagged inference policies (Tian et al., 2026; Arnal et al., 2026; Ritter et al., 2026). Together, these methods show that stale data can remain informative when its effect on the update is controlled. Our work instead moves from individual correction to group-level control. We identify a cap on corrected importance mass that preserves gradient second-moment bound and derive a uniform rescaling rule that reduces delay-induced bias without selectively clipping trajectories. Theory of GRPO-Style Optimization under Delay. The convergence theory of asynchronous optimization is well developed, with rigorous and tight convergence analysis that matches the lower bound under various settings (Joulani et al., 2013; Koloskova et al., 2022; Mishchenko et al., 2022; Tyurin et al., 2026). In parallel, a growing line of work establishes theoretical guarantees for RL-based LLM post-training (Xie et al., 2025; Huang et al., 2025b; Foster et al., 2025; Huang et al., 2025a). Despite this progress, most existing RL-for-LLM theories are developed within the natural policy gradient and RLHF frameworks, and rigorous convergence guarantees for widely used GRPO-style algorithms remain limited, particularly under asynchronous rollout generation. Zhou et al. (2026) characterize the GRPO gradient as a U-statistic and analyze its finite-sample error and dependence on group size, while Pang et al. (2025) introduce TIC-GRPO and provide the first rigorous convergence analysis of GRPO-style methods. More recently, Xu et al. (2026) analyze asynchronous GRPO through a generic biased-gradient framework, but leave the staleness-induced bias and higher-order remainder terms as abstract quantities in the final bound. To the best of our knowledge, we are the first to provide an explicit delay-dependent convergence analysis of GRPO-style algorithms in asynchronous RL systems. Appendix B Experiment Details and supplemental Experiments B.1 Detailed Experiment Setup Models and training data. We run the experiments with the Qwen3-1.7B-Base, Qwen3-4B-Base and Qwen3-8B-Base models (Yang et al., 2025). All methods update the full actor; we use neither LoRA adapters nor an explicit KL-divergence penalty. The training set is constructed from the seven subject-specific training splits of MATH (Hendrycks et al., 2021): algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus. After removing examples for which the reference answer cannot be extracted, the resulting training set contains 7,498 problems. Training and evaluation use the same two-shot prompt, which instructs the model to reason step by step and place its final answer in . Shared training configuration. We generate rollouts with vLLM (Kwon et al., 2023) in BF16, using temperature 1.01.0 and top-p=1.0p=1.0. Prompts and completions are truncated at 1,024 and 4,096 tokens, respectively. Each rollout mini-batch contains eight prompts and eight responses per prompt, for a total of 64 trajectories. Except for the zero-delay runs in the fixed-delay suite described below, we set inner_steps=1, so these 64 trajectories produce one actor update. We optimize with 8-bit AdamW (Dettmers et al., 2021), a learning rate of 10−610^-6, β1=0.9 _1=0.9, β2=0.999 _2=0.999, and zero weight decay. The learning rate follows a cosine schedule with a 3% warmup, and the gradient norm is clipped at 1.01.0. We use a mix of NVIDIA RTX 6000 Ada Generation GPUs and NVIDIA A100 Tensor Core GPUs for training, depending on resource availability. Controlled fixed delays. The fixed-delay study uses Qwen3-1.7B-Base and sweeps the delay over D∈0,32,64,128,256,512D∈\0,32,64,128,256,512\ optimizer updates. For D=0D=0, we use inner_steps=4, so each rollout batch comprises four 64-trajectory optimizer mini-batches, for 256 responses in total. For every D>0D>0, we use inner_steps=1 and maintain a FIFO queue of D 64-trajectory rollout batches. The queue is initialized with batches generated by the initial policy. In these delayed runs, at each learner iteration, the oldest batch is consumed in one optimizer update and a new batch from the current policy is appended, yielding a steady-state behavior-policy lag of D optimizer updates after the initial fill transition. All remaining optimization, rollout, and sequence-length settings match the shared configuration above. Figure 3: A full version of Figure 2 with delay=0, 32, 64, 128, 256, 512. Figure 4: Current-policy token entropy across optimizer updates under fixed rollout delays. Faint curves show the per-update measurements, and solid curves show centered rolling means with a window of 25 updates. Dashed vertical lines mark queue drain, after which no new rollouts are generated. GMC-GRPO generally reaches a stable low-entropy regime earlier as the delay increases, while all methods behave similarly without delay. Simulated asynchronous delays. The second suite models the coupling between response length and asynchronous rollout latency. We maintain a hidden in-flight buffer and a virtual clock τ. Launching a 64-trajectory rollout batch advances the clock by Cgen=85C_gen=85. If the batch has mean completion length L, it becomes visible at τready=τlaunch+λL,λ=28. _ready= _launch+λ L, λ=28. Each optimizer update advances the clock by Cupdate=1C_update=1, and ready batches are exposed in completion-time order. Thus, longer batches tend to be consumed under older learner policies. A batch generated at optimizer step g and consumed at step t has realized delay d=t−gd=t-g; its approximate steady-state mean is [d]≃λ[L]Cgen+Cupdate=28[L]86.E[d] λ\,E[L]C_gen+C_update= 28\,E[L]86. Consequently, λ controls the delay distribution, while the measured mean delay—approximately 160 updates in the reported experiments—also depends on the model’s response-length dynamics. At initialization, we launch rollouts from policy 0 until the first batch is visible. If the visible queue later becomes empty, we launch the minimum number of additional batches needed to expose the next sample; both warmup and refill use a safety cap of 400 launches. For Qwen3-1.7B, rollout generation and actor updates execute sequentially on one GPU. For Qwen3-4B, one GPU serves vLLM rollouts and a second GPU performs actor optimization, with rollout generation overlapped with learner updates. This systems-level overlap does not alter the sample order or the simulated-delay schedule. Remark 6 (Scope of the asynchronous-sampling assumption). The fixed-delay suite satisfies Assumption 3 exactly, since every group’s delay is fixed in advance and independent of its responses. The simulated suite intentionally relaxes the conditional-independence part of the assumption: because delay is determined by response length, conditioning on a long delay tilts the returned responses toward longer completions. Our theorems therefore do not formally cover this setting. The key mechanism behind Assumption 3 is nevertheless preserved. The pipeline is lossless, so response length changes only when a rollout is consumed, not whether it is consumed, and each importance ratio uses the exact behavior policy that generated the rollout. As a result, the length–delay coupling largely averages out over the course of training, unlike completion-time filtering, which systematically removes long responses. Moreover, this coupling makes the problem harder rather than easier: long responses are both more stale and more prone to extreme importance ratios. We therefore view the simulated suite as a stress test beyond the idealized assumption, and GMC-GRPO’s robustness there suggests that its advantage does not hinge on this idealization. Method-specific settings. Vanilla GRPO (Shao et al., 2024) uses the two-sided clipping range ϵ=0.2ε=0.2 and the unnormalized group-centered advantage ri−r¯gr_i- r_g. VCPO (Huang et al., 2026) uses effective-sample-size-guided step-size scaling and its closed-form off-policy minimum-variance baseline. GAC (Xu et al., 2026) uses cosine-similarity thresholds 0.050.05 and 0.300.30. TIC-GRPO (Pang et al., 2025) and GMC-GRPO both use leave-one-out advantages, ϵhigh=0.28 _high=0.28. All other training, rollout, data-order, and delay parameters are held fixed across methods within each suite. Evaluation. For the simulated asynchronous study, we evaluate the step-800 checkpoint on six mathematical-reasoning benchmarks: MATH-500 (500 problems) (Hendrycks et al., 2021; Lightman et al., 2024), GSM8K (1,319) (Cobbe et al., 2021), Gaokao2023-Math-En (385) (Liao et al., 2024), the OE_TO_maths_en_COMP subset of OlympiadBench (674) (He et al., 2024), Minerva Math (272) (Lewkowycz et al., 2022), and AMC23 (40) (Art of Problem Solving, 2024). Greedy accuracy uses temperature zero and one output per problem. Sampled evaluation uses temperature 1.01.0, top-p=1.0p=1.0. We draw 16 outputs per problem for MATH-500, Minerva Math, and AMC23, and eight for GSM8K, Gaokao2023-Math-En, and OlympiadBench. Each main-table entry is reported as greedy/avg@n, where avg@n is the mean correctness across all n sampled outputs rather than pass@any. The final column is the unweighted macro-average over the six benchmarks, computed separately for greedy accuracy and avg@n. The fixed-delay study evaluates GRPO, TIC-GRPO, and GMC-GRPO on MATH-500 at checkpoint 800. We use greedy decoding and 16 sampled responses per problem to compute the unbiased pass@k estimators for k∈1,2,4,8,16k∈\1,2,4,8,16\. Both suites use the same 1,024-token prompt limit, 4,096-token generation limit, two-shot prompt, and last-complete- answer scorer as in training. B.2 Supplemental Experiment Results B.2.1 Additional experiment data and training dynamics Table 3: Performance on MATH500 under different fixed behavior-policy delays (Qwen3-1.7B-Base, checkpoint 800). Delay Method Greedy pass@1 pass@2 pass@4 pass@8 pass@16 – Base (no RL) 44.40 25.57 37.52 50.00 61.65 72.40 0 GMC-GRPO 66.40 62.94 72.39 79.25 84.64 89.00 TIC-GRPO 66.80 63.62 72.59 79.47 84.90 89.00 GRPO 66.20 63.32 72.41 79.25 84.84 88.80 32 GMC-GRPO 64.80 60.84 70.47 77.85 83.89 88.60 TIC-GRPO 66.40 61.70 71.09 78.38 84.24 88.80 GRPO 56.20 45.19 58.01 69.04 77.62 83.60 64 GMC-GRPO 65.20 60.38 69.97 77.30 83.30 88.00 TIC-GRPO 65.40 60.29 70.21 77.42 82.92 86.80 GRPO 52.00 40.91 53.93 65.43 74.47 81.20 128 GMC-GRPO 68.40 59.03 69.57 77.34 83.45 88.00 TIC-GRPO 63.20 55.41 66.86 75.64 82.59 87.80 GRPO 51.40 35.33 47.89 59.87 70.46 78.60 256 GMC-GRPO 59.60 48.33 61.64 72.02 79.85 85.40 TIC-GRPO 56.00 46.20 59.78 71.12 80.05 86.20 GRPO 47.60 32.11 44.78 57.10 68.17 77.60 512 GMC-GRPO 56.40 42.01 55.97 67.78 76.91 83.40 TIC-GRPO 53.80 38.17 51.07 62.59 72.47 80.60 GRPO 46.80 30.23 42.73 55.03 66.16 76.00 Greedy uses temperature 0. pass@k uses the unbiased estimator from n=16n=16 samples per question (temperature 1.0, top-p 1.0), averaged over 500 questions. Our theoretical results concern optimization convergence rather than only final task accuracy. To connect these guarantees to observable training dynamics, Figure 4 reports current-policy token entropy, which is the most direct diagnostic among our logged statistics of how quickly the policy distribution stabilizes. Without delay, all three methods exhibit similar entropy decay, consistent with their nearly identical Pass@1 trajectories in Figure 2. As the delay increases, GMC-GRPO typically enters a stable low-entropy regime earlier than TIC-GRPO and vanilla GRPO. Although entropy is not the formal stationarity measure used in our theory, its faster decay and earlier stabilization, together with the Pass@1 curves, provide empirical evidence consistent with the predicted faster convergence of GMC-GRPO under stale rollouts. Table 3 complements Figure 2 by reporting the complete checkpoint-800 results, including greedy accuracy and pass@k for k∈1,2,4,8,16k∈\1,2,4,8,16\. The three methods remain nearly indistinguishable at zero delay. Under stale rollouts, GMC-GRPO consistently improves over vanilla GRPO and is generally competitive with or stronger than TIC-GRPO, with the clearest gains at delays 128 and 512. Thus, the trend in the main-text Pass@1 curves extends to greedy decoding and a broader range of sampling budgets. B.2.2 GMC-GRPO benefits from larger group sizes Corollary 2 shows that larger groups reduce the delay-dependent term in the GMC-GRPO convergence bound. To examine this benefit empirically, we vary the group size over G∈4,8,32G∈\4,8,32\ while keeping all other experimental settings identical to the fixed-delay experiment with delay 128128. Figure 5 shows that GMC-GRPO establishes a reward advantage over TIC-GRPO progressively earlier as the group size increases: the curves begin to separate around step 300300 for G=4G=4, around step 200200 for G=8G=8, and around step 100100 for G=32G=32. This earlier separation indicates that GMC-GRPO benefits more promptly from larger groups, providing empirical evidence consistent with the faster convergence predicted by Corollary 2. The training-reward gap also becomes smaller at larger group sizes toward the end of training, with the two curves nearly meeting for G=32G=32. This pattern is consistent with both methods approaching a reward plateau: GMC-GRPO makes faster progress early on, while TIC-GRPO catches up as training continues. Thus, the benefit of larger groups is most apparent in the learning dynamics, even when the final training rewards are similar. Table 4 complements these training curves with MATH500 evaluations at checkpoints 200200, 300300, 400400, and 800800. At checkpoint 200200, GMC-GRPO’s sampled Pass@1 advantage over TIC-GRPO increases from −1.11-1.11 percentage points at G=4G=4 to 3.793.79 at G=8G=8 and 7.107.10 at G=32G=32, showing that the earlier training gains also translate into better evaluation performance. At the final checkpoint, GMC-GRPO retains higher sampled Pass@1 at all three group sizes and matches or exceeds TIC-GRPO in greedy accuracy. In particular, at G=32G=32, both methods achieve 66.00%66.00\% greedy accuracy, but GMC-GRPO still achieves higher sampled Pass@1 (61.18%61.18\% versus 59.80%59.80\%). Figure 5: Training-reward dynamics for GMC-GRPO and TIC-GRPO with group sizes G∈4,8,32G∈\4,8,32\ under the fixed-delay setting with delay 128128. All other experimental settings are unchanged. Faint curves show individual measurements, and solid curves show smoothed trends. GMC-GRPO establishes a reward advantage earlier as the group size increases, while the gap narrows toward the end of training, particularly for G=32G=32. Table 4: MATH500 evaluation of Qwen3-1.7B-Base at checkpoints 200200, 300300, 400400, and 800800 under the fixed-delay setting with delay 128128. All settings except group size match the fixed-delay experiment. Each method entry reports greedy accuracy / sampled Pass@1 (%), with sampled Pass@1 computed as mean correctness over 16 responses per problem (avg@16). Δ denotes the sampled Pass@1 of GMC-GRPO minus that of TIC-GRPO, measured in percentage points (p). Checkpoint G GMC-GRPO TIC-GRPO Δ Pass@1 (p) 200 4 51.40 / 33.70 51.60 / 34.81 −1.11-1.11 200 8 55.40 / 39.23 50.20 / 35.44 +3.79+3.79 200 32 62.80 / 46.51 53.60 / 39.41 +7.10+7.10 300 4 57.00 / 39.02 54.80 / 36.05 +2.97+2.97 300 8 61.40 / 46.34 55.20 / 41.01 +5.33+5.33 300 32 65.40 / 56.34 56.00 / 42.10 +14.24+14.24 400 4 62.60 / 48.52 58.60 / 42.29 +6.23+6.23 400 8 67.00 / 55.90 60.00 / 46.76 +9.14+9.14 400 32 64.00 / 58.55 62.80 / 51.41 +7.14+7.14 800 4 67.00 / 56.17 63.00 / 53.91 +2.26+2.26 800 8 68.40 / 59.03 63.20 / 55.41 +3.62+3.62 800 32 66.00 / 61.18 66.00 / 59.80 +1.38+1.38 B.2.3 Comparison against A-3PO. Table 5: Comparison of A-3PO and GMC-GRPO under simulated asynchronous delays. Each benchmark cell is greedy/avg@n, with n=16n=16 for MATH500, Minerva, and AMC23, and n=8n=8 for GSM8K, Gaokao, and OlympiadBench. Avg. perf. is the average over the six benchmarks, reported as mean greedy/mean avg@n. For Qwen3-1.7B and Qwen3-4B, the better result for each metric is bolded. A-3PO exhibited training collapse in our Qwen3-8B experiment. Method Avg. Delay MATH500 GSM8K Gaokao Olympiad Minerva AMC23 Avg. perf. Qwen3-1.7B A-3PO 160.80 67.40 / 61.26 81.88 / 76.42 53.77 / 51.69 30.71 / 25.56 28.68 / 24.20 32.50 / 32.50 49.16 / 45.27 GMC-GRPO 159.75 66.00 / 57.29 81.65 / 72.66 55.32 / 47.73 28.19 / 22.61 26.47 / 20.11 35.00 / 28.44 48.77 / 41.47 Qwen3-4B A-3PO 138.34 70.80 / 69.16 90.14 / 88.55 59.48 / 58.38 36.80 / 35.16 31.62 / 29.78 50.00 / 46.88 56.47 / 54.65 GMC-GRPO 141.01 77.60 / 72.54 91.13 / 87.03 65.19 / 60.91 43.18 / 37.72 34.19 / 30.10 52.50 / 48.91 60.63 / 56.20 Qwen3-8B A-3PO - Training collapsed GMC-GRPO 141.1 78.80 / 75.22 93.33 / 91.37 69.09 / 63.64 44.96 / 40.34 40.07 / 34.35 60.00 / 52.81 64.37 / 59.62 Figure 6: Training reward on Qwen3-8B under simulated asynchronous delays. Faint curves show individual measurements, and solid curves show smoothed trends. A-3PO initially improves rapidly, but its recorded reward falls to near zero around steps 430–440. GMC-GRPO continues to improve over the remaining training steps. In this section, we present an independent comparison between GMC-GRPO and A-3PO (Li et al., 2025), which achieves the best average performance on Qwen3-1.7B but suffers training collapse on Qwen3-8B in our experiments. We therefore report this comparison separately and discuss the observed instability in detail. To make use of stale rollouts without overly restricting policy updates, A-3PO follows decoupled policy optimization (Fu et al., 2026): it uses a proximal anchor for clipping while retaining importance correction relative to the behavior policy. Suppressing token arguments (yt∣x,y<t)(y_t x,y_<t), the decoupled surrogate is LdecoupledCLIP(θ)=[πproxπθbehavminπθπproxA^,clip(πθπprox,1−ϵ,1+ϵ)A^],L_decoupled^CLIP(θ)=E\! [ _prox _ _behav \! \ _θ _prox A,\,clip\! ( _θ _prox,1-ε,1+ε ) A \ ], (10) where the expectation is over sampled rollout tokens and A A is the group-based advantage. A-3PO approximates the proximal anchor by logπprox=αlogπθbehav+(1−α)logπθ,α=1max1,τ, _prox=α _ _behav+(1-α) _θ, α= 1 \1,τ\, This design is reasonable when delays are small: the policy shift is limited, and retaining more learning signals from stale rollouts can accelerate learning. Under large delays, however, α approaches zero, so the ratio used for clipping can remain close to one and the update can resemble unclipped GRPO. Since the outer importance factor πprox/πθbehav _prox/ _ _behav remains unclipped, large importance weights can destabilize updates and even lead to training collapse in extreme cases. This is also demonstrated in our experiments: Table 5 shows that A-3PO outperforms GMC-GRPO on Qwen3-1.7B in mean greedy/mean avg@n accuracy (49.16/45.27 versus 48.77/41.47). This advantage is consistent with prior evidence that hard clipping can discard useful gradients and restrict policy improvement (Dwyer et al., 2025; Zheng et al., 2026). Retaining more of these signals can therefore improve performance when extreme importance ratios are rare. The Qwen3-8B experiment illustrates the accompanying stability challenge. In Figure 6, A-3PO initially learns faster but its training reward collapses around step 430, whereas GMC-GRPO continues to improve and reaches mean greedy/mean avg@n accuracy of 64.37/59.62. This trajectory resembles the prosperity-before-collapse behavior reported by Zheng et al. (2026), where removing the trust region exposes useful stale-data signals but also permits destabilizing updates. The comparison highlights the motivation for GMC: preserving learning signals must be coupled with control of update magnitude. GMC caps the average importance mass within each group and preserves the relative contributions of its responses, controlling the update second moment under the assumptions of Proposition 1. Appendix C Mixed-Delay Setting, GMC-GRPO and TIC-GRPO Algorithms TIC-GRPO was originally proposed for synchronous training (Pang et al., 2025), so Algorithm 1 first restates this original synchronous form for reference. Algorithm 2 then gives the asynchronous versions of GMC-GRPO and TIC-GRPO analyzed in this paper. The two asynchronous methods share the same rollout and learner schedule and differ only in their importance correction. Each completed group stores its prompt, responses, rewards, behavior-policy version s, and behavior log-probabilities ℓ(i)=logπθs(y(i)∣x) ^(i)= _ _s(y^(i) x). For a group b consumed at iteration n, both algorithms compute ρn(b,i)=exp(logπθn(y(b,i)∣x(b))−ℓ(b,i)). _n^(b,i)= \! ( _ _n(y^(b,i) x^(b))- ^(b,i) ). For GMC-GRPO, the learner forms gnGMCg_n^GMC in eq. 9 using the group-rescaled ratios ρ^n(b,i) ρ_n^(b,i). For TIC-GRPO, it instead uses the individually truncated ratios ρn(b,i)ρn(b,i)≤1+ϵ _n^(b,i)1\ _n^(b,i)≤ 1+ε\ to form gnTICg_n^TIC, as defined in eqs. 34 and 35. In either case, the learner takes the ascent step θn+1=θn+ηgn _n+1= _n+η g_n. Both methods use the leave-one-out advantages defined in Section 2.1. Behavior log-probabilities and advantages are held fixed during each update. In the synchronous schedule, K mini-batches share one behavior-policy snapshot: rollout collection finishes before optimization begins, and the next global batch is generated only after these updates finish. Thus, synchronous training can still use stale data during successive mini-batch updates. In the asynchronous schedule, rollout workers and the learner run concurrently. The learner needs only B completed groups to update; it does not wait for all workers or require a common behavior-policy version. Algorithm 1 TIC-GRPO with synchronous rollouts 1: initial parameter θ0 _0; step size η; total updates N; group size G≥2G≥ 2; mini-batch size B; mini-batches per global batch K≥1K≥ 1; clipping threshold 1+ϵ1+ε; prompt distribution D; reward rϕr_φ. 2: n←0n← 0. 3: while n<Nn<N do 4: freeze the behavior-policy snapshot θs←θn _s← _n with s←ns← n; set Kn←minK,N−nK_n← \K,N-n\. 5: draw KnBK_nB prompts from D and generate G responses per prompt under the fixed policy πθs _ _s. 6: wait until all KnBK_nB groups are generated and scored by rϕr_φ; store their behavior log-probabilities and version s. 7: partition the completed global batch into KnK_n mini-batches of B whole groups each. 8: for each mini-batch in the partition do 9: denote its groups by b∈[B]b∈[B] and set sn(b)←s_n^(b)← s, τn(b)←n−s _n^(b)← n-s. 10: form A^(b,i) A^(b,i) from the rewards of all G responses in each group. 11: compute ρn(b,i) _n^(b,i) from the current policy and stored behavior log-probabilities for all b∈[B]b∈[B], i∈[G]i∈[G]. 12: form gnTICg_n^TIC using eq. 35 and update θn+1←θn+ηgnTIC _n+1← _n+η g_n^TIC. 13: n←n+1n← n+1. 14: end for 15: end while Algorithm 2 TIC-GRPO and GMC-GRPO with asynchronous rollouts 1: method ∈TIC,GMC method∈\TIC,GMC\; initial parameter θ0 _0; step size η; total updates N; group size G≥2G≥ 2; mini-batch size B; threshold 1+ϵ1+ε; rollout workers over D with reward rϕr_φ; shared buffer Q. 2: initialize ←∅Q← and publish the snapshot (0,θ0)(0, _0). 3: Run the following worker and learner loops concurrently. 4: Each rollout worker: 5: while training is active do 6: read a published snapshot (s,θs)(s, _s) and draw x∼x . 7: generate y(1),…,y(G)∼πθs(⋅∣x)y^(1),…,y^(G) _ _s(· x), keeping θs _s fixed throughout this group. 8: compute r(i)←rϕ(x,y(i))r^(i)← r_φ(x,y^(i)) and store ℓ(i)←logπθs(y(i)∣x) ^(i)← _ _s(y^(i) x) for all i∈[G]i∈[G]. 9: append (x,y(i),r(i),ℓ(i)i=1G,s)(x,\y^(i),r^(i), ^(i)\_i=1^G,s) to Q as one completed group. 10: end while 11: Learner: 12: for n=0,1,…,N−1n=0,1,…,N-1 do 13: once ||≥B|Q|≥ B, pop B completed groups ℳnM_n; their behavior-policy versions need not match. 14: denote each group’s stored version by sn(b)s_n^(b) and set τn(b)←n−sn(b) _n^(b)← n-s_n^(b). 15: form A^(b,i) A^(b,i) from the rewards of all G responses in each group. 16: compute ρn(b,i) _n^(b,i) from the current policy and stored behavior log-probabilities for all b∈[B]b∈[B], i∈[G]i∈[G]. 17: if =TIC method=TIC then 18: form gn←gnTICg_n← g_n^TIC using eq. 35. 19: else⊳ GMC-GRPO 20: for each group b∈[B]b∈[B] do 21: compute ρ¯n(b)←G−1∑i=1Gρn(b,i) ρ_n^(b)← G^-1 _i=1^G _n^(b,i) using log-sum-exp. 22: set ρ^n(b,i)←ρn(b,i)/max1,ρ¯n(b)/(1+ϵ) ρ_n^(b,i)← _n^(b,i)/ \1, ρ_n^(b)/(1+ε)\ for all i∈[G]i∈[G]. 23: end for 24: form gn←gnGMCg_n← g_n^GMC using eq. 9. 25: end if 26: update θn+1←θn+ηgn _n+1← _n+η g_n. 27: publish (n+1,θn+1)(n+1, _n+1) without waiting for ongoing rollouts; in-progress groups retain their launch snapshots. 28: end for 29: signal the rollout workers to stop. We use the asynchronous sampling model and notation introduced in Section 2.2. In particular, τn(b) _n^(b) is the delay of group b, ρn(b,i) _n^(b,i) is the delayed importance ratio in eq. 3, and A^(b,i) A^(b,i) is the leave-one-out advantage defined in Section 2.1. Let ℳnM_n denote the mini-batch of B completed groups returned at learner iteration n. To aggregate the mixed delays over the training horizon, for any integer k≥1k≥ 1, define the averaged kkth delay moment τ¯k:=1N∑n=0N−1[1B∑b=1B(τn(b))k]. τ_k:= 1N _n=0^N-1E\! [ 1B _b=1^B ( _n^(b) )^k ]. (11) Thus, τ¯2 τ_2 and τ¯4 τ_4 denote the averaged second and fourth delay moments, respectively. The second moment controls intermediate parameter-drift terms, while the fourth moment appears in the final convergence bounds after the delay-induced bias is squared. For the appendix proofs, we additionally write the average importance ratio in group b as ρ¯n(b):=G−1∑i=1Gρn(b,i) ρ_n^(b):=G^-1 _i=1^G _n^(b,i). Softmax policies have full support, so every ratio is finite and ρ¯n(b)>0 ρ_n^(b)>0 almost surely. Using the common scaling factor from Proposition 2, the GMC effective ratio admits the equivalent form ρ^n(b,i):=ρn(b,i)max1,ρ¯n(b)/(1+ϵ). ρ_n^(b,i):= _n^(b,i) \1, ρ_n^(b)/(1+ε)\. (12) This form is used in the groupwise decomposition and moment bounds below. The GMC branch of Algorithm 2 applies this groupwise rescaling within the shared asynchronous training loop. The behavior log-probabilities stored with each completed group are sufficient to compute every delayed importance ratio at consumption time. Remark 7 (Autodifferentiation implementation). Equation 9 defines the update directly. In a surrogate-loss implementation, max1,ρ¯n(b)/(1+ϵ) \1, ρ_n^(b)/(1+ε)\ and A^(b,i) A^(b,i) must be treated as constants using stop-gradient. The numerator ratio supplies the policy gradient. Differentiating through the common group denominator defines a different estimator and is not covered by the theorems below. Proof-specific conditioning and divergence notation. The filtration pair ℱn−⊆ℋnF_n^- _n is defined in Section 2.2, and Assumption 3 specifies the corresponding conditional sampling properties. For each returned group, define the current-to-behavior divergences Dn(b) D_n^(b) :=DKL(πθn(⋅∣x(b))∥πθn−τn(b)(⋅∣x(b))), :=D_KL\! ( _ _n(· x^(b))\,\|\, _ _n- _n^(b)(· x^(b)) ), χn2,(b) _n^2,(b) :=χ2(πθn(⋅∣x(b))∥πθn−τn(b)(⋅∣x(b))). :=χ^2\! ( _ _n(· x^(b))\,\|\, _ _n- _n^(b)(· x^(b)) ). Both divergences, as well as Jx(b)(θn)J_x^(b)( _n), are ℋnH_n-measurable and hence fixed inside conditional expectations given ℋnH_n. Because ℱn−⊆ℋnF_n^- _n, every integrable random variable Z satisfies [Z∣ℱn−]=[[Z∣ℋn]∣ℱn−].E[Z _n^-]=E\! [E[Z _n] _n^- ]. The proofs first condition on ℋnH_n to hold the prompt, delay, and behavior policy fixed, and then average back to ℱn−F_n^- to recover a statement about the learner iterate. With the algorithm and conditioning convention fixed, the next section collects the analytic tools shared by the GMC-GRPO and TIC-GRPO proofs. Appendix D Technical preliminaries The convergence proofs for both algorithms rely on a common set of analytic tools. Starting from Assumption 1, we establish trajectory-score bounds, control trajectory KL divergence by the squared parameter distance, and prove smoothness of the reward objective. We then use this smoothness bound to prove Lemma 1, which reduces convergence analysis to controlling the update second moment and the squared conditional bias. The following sections supply these controls for GMC-GRPO and TIC-GRPO. Lemma 2 (Trajectory score bounds). Under Assumption 1, for every prompt x, parameter θ, and length-T response y, y∼πθ(⋅∣x)‖∇logπθ(y∣x)‖22 _y _θ(· x) \|∇ _θ(y x) \|_2^2 ≤2LTlog||, ≤ 2LT |A|, (13) ‖∇logπθ(y∣x)‖22 \|∇ _θ(y x) \|_2^2 ≤−2LTlogπθ(y∣x). ≤-2LT _θ(y x). (14) If Assumption 2 also holds, then ‖∇logπθ(y∣x)‖2≤T2Llog. \|∇ _θ(y x) \|_2≤ T 2L . (15) Proof. Fix a token position t and its prefix (x,y<t)(x,y_<t), and write ft(θ):=−logπθ(yt∣x,y<t)f_t(θ):=- _θ(y_t x,y_<t). Assumption 1 states that ∇ft∇ f_t is L-Lipschitz. The descent lemma therefore gives, for every vector v, ft(θ+v)≤ft(θ)+⟨∇ft(θ),v⟩+L2‖v‖22.f_t(θ+v)≤ f_t(θ)+ ∇ f_t(θ),\,v + L2 \|v \|_2^2. Choose v=−L−1∇ft(θ)=L−1∇logπθ(yt∣x,y<t)v=-L^-1∇ f_t(θ)=L^-1∇ _θ(y_t x,y_<t). Since ftf_t is nonnegative, 0 0 ≤−logπθ+L−1∇logπθ(yt∣x,y<t)(yt∣x,y<t) ≤- _θ+L^-1∇ _θ(y_t x,y_<t)(y_t x,y_<t) ≤−logπθ(yt∣x,y<t)−12L‖∇logπθ(yt∣x,y<t)‖22, ≤- _θ(y_t x,y_<t)- 12L \|∇ _θ(y_t x,y_<t) \|_2^2, Rearranging the last inequality shows that each token score satisfies ‖∇logπθ(yt∣x,y<t)‖22≤−2Llogπθ(yt∣x,y<t). \|∇ _θ(y_t x,y_<t) \|_2^2≤-2L _θ(y_t x,y_<t). (16) Now let st:=∇logπθ(yt∣x,y<t)s_t:=∇ _θ(y_t x,y_<t). The autoregressive factorization gives ∇logπθ(y∣x)=∑t=1Tst∇ _θ(y x)= _t=1^Ts_t, so Cauchy–Schwarz and eq. 16 yield ‖∇logπθ(y∣x)‖22≤T∑t=1T‖st‖22≤−2LT∑t=1Tlogπθ(yt∣x,y<t)=−2LTlogπθ(y∣x), \|∇ _θ(y x) \|_2^2≤ T _t=1^T \|s_t \|_2^2≤-2LT _t=1^T _θ(y_t x,y_<t)=-2LT _θ(y x), which proves eq. 14. For the expected bound, first expand the conditional score identity: [st∣x,y<t]=∑a∈πθ(a∣x,y<t)∇logπθ(a∣x,y<t)=∑a∈∇πθ(a∣x,y<t)=∇1=0.E[s_t x,y_<t]= _a _θ(a x,y_<t)∇ _θ(a x,y_<t)= _a ∇ _θ(a x,y_<t)=∇ 1=0. If s<ts<t, then s_s is determined by (x,y<t)(x,y_<t). The tower property therefore gives ⟨s,st⟩=⟨s,[st∣x,y<t]⟩=0E s_s,\,s_t =E s_s,\,E[s_t x,y_<t] =0; hence distinct token scores are orthogonal in expectation. Expanding the square and applying eq. 16, y∼πθ(⋅∣x)‖∇logπθ(y∣x)‖22 _y _θ(· x) \|∇ _θ(y x) \|_2^2 =‖∑t=1Tst‖22=∑t=1T‖st‖22 =E \| _t=1^Ts_t \|_2^2= _t=1^TE \|s_t \|_2^2 ≤2L∑t=1T[−logπθ(yt∣x,y<t)]≤2LTlog||, ≤ 2L _t=1^TE\! [- _θ(y_t x,y_<t) ]≤ 2LT |A|, where the last inequality uses [−logπθ(yt∣x,y<t)]=y<t[H(πθ(⋅∣x,y<t))]≤log||.E[- _θ(y_t x,y_<t)]=E_y_<t\! [H ( _θ(· x,y_<t) ) ]≤ |A|. This proves eq. 13. Finally, Assumption 2 implies −logπθ(yt∣x,y<t)≤log- _θ(y_t x,y_<t)≤ . Combining this with eq. 16 and the triangle inequality, ‖∇logπθ(y∣x)‖2≤∑t=1T‖st‖2≤T2Llog, \|∇ _θ(y x) \|_2≤ _t=1^T \|s_t \|_2≤ T 2L , which is eq. 15. ∎ Lemma 3 (Trajectory KL is quadratically controlled). Under Assumption 1, for every prompt x and all θ,θ′θ,θ , DKL(πθ(⋅∣x)∥πθ′(⋅∣x))≤TL2‖θ−θ′‖22.D_KL ( _θ(· x)\,\|\, _θ (· x) )≤ TL2 \|θ-θ \|_2^2. (17) Proof. For a fixed prefix h=(x,y<t)h=(x,y_<t) and token a, the L-Lipschitz score implies the two-sided smoothness bound logπθ(a∣h)−logπθ′(a∣h)≤⟨∇logπθ(a∣h),θ−θ′⟩+L2‖θ−θ′‖22. _θ(a h)- _θ (a h)≤ ∇ _θ(a h),\,θ-θ + L2 \|θ-θ \|_2^2. The autoregressive KL chain rule sums this inequality over token positions: DKL(πθ(⋅∣x)∥πθ′(⋅∣x)) D_KL ( _θ(· x)\,\|\, _θ (· x) ) =y∼πθ(⋅∣x)∑t=1Tlogπθ(yt∣x,y<t)πθ′(yt∣x,y<t) =E_y _θ(· x) _t=1^T _θ(y_t x,y_<t) _θ (y_t x,y_<t) ≤∑t=1Ty∼πθ(⋅∣x)[⟨∇logπθ(yt∣x,y<t),θ−θ′⟩+L2‖θ−θ′‖22] ≤ _t=1^TE_y _θ(· x) [ ∇ _θ(y_t x,y_<t),\,θ-θ + L2 \|θ-θ \|_2^2 ] =TL2‖θ−θ′‖22. = TL2 \|θ-θ \|_2^2. For the last equality, condition on (x,y<t)(x,y_<t) in each linear term. The score identity proved above gives [⟨∇logπθ(yt∣x,y<t),θ−θ′⟩|x,y<t]=⟨∑a∈∇πθ(a∣x,y<t),θ−θ′⟩=0.E\! [ ∇ _θ(y_t x,y_<t),\,θ-θ \, |\,x,y_<t ]= _a ∇ _θ(a x,y_<t),\,θ-θ =0. Thus only the L‖θ−θ′‖22/2L \|θ-θ \|_2^2/2 remainder remains at each of the T positions, proving eq. 17. ∎ Lemma 4 (Smoothness of the reward objective). Under Assumption 1, the objective has TRL(2log||+1)TRL(2 |A|+1)-Lipschitz gradient. Consequently, any update θn+1=θn+ηgn _n+1= _n+η g_n satisfies J(θn+1)≥J(θn)+η⟨∇J(θn),gn⟩−TRL(2log||+1)η22‖gn‖22.J( _n+1)≥ J( _n)+η ∇ J( _n),\,g_n - TRL(2 |A|+1)η^22 \|g_n \|_2^2. (18) Proof. Fix a prompt–response pair (x,y)(x,y) and write Sθ(y∣x):=∇logπθ(y∣x)S_θ(y x):=∇ _θ(y x). The autoregressive decomposition and Assumption 1 imply ‖Sθ(y∣x)−Sθ′(y∣x)‖2≤TL‖θ−θ′‖2. \|S_θ(y x)-S_θ (y x) \|_2≤ TL \|θ-θ \|_2. Let Δ:=θ−θ′ :=θ-θ and γ(s):=θ′+sΔγ(s):=θ +s . The restriction s↦Sγ(s)(y∣x)s S_γ(s)(y x) is TL‖Δ‖2TL \| \|_2-Lipschitz, hence absolutely continuous and differentiable for almost every s, with ‖dsSγ(s)(y∣x)‖2≤TL‖Δ‖2. \| ddsS_γ(s)(y x) \|_2≤ TL \| \|_2. Moreover, ∇πθ(y∣x)=πθ(y∣x)Sθ(y∣x)∇ _θ(y x)= _θ(y x)S_θ(y x) and dsπγ(s)(y∣x)=πγ(s)(y∣x)⟨Sγ(s)(y∣x),Δ⟩ dds _γ(s)(y x)= _γ(s)(y x) S_γ(s)(y x),\, . Both factors are bounded on the compact segment, so their product is absolutely continuous. At almost every s, the one-dimensional product rule gives ds∇πγ(s)(y∣x)=πγ(s)(y∣x)⟨Sγ(s)(y∣x),Δ⟩Sγ(s)(y∣x)+dsSγ(s)(y∣x). dds∇ _γ(s)(y x)= _γ(s)(y x) \ S_γ(s)(y x),\, S_γ(s)(y x)+ ddsS_γ(s)(y x) \. Lemma 2 now allows us to sum the derivative bound over the finite trajectory space: ∑y‖ds∇πγ(s)(y∣x)‖2 _y \| dds∇ _γ(s)(y x) \|_2 ≤‖Δ‖2y∼πγ(s)(⋅∣x)[‖Sγ(s)(y∣x)‖22+TL] ≤ \| \|_2E_y _γ(s)(· x)\! [ \|S_γ(s)(y x) \|_2^2+TL ] ≤TL(2log||+1)‖Δ‖2. ≤ TL(2 |A|+1) \| \|_2. The fundamental theorem of calculus for absolutely continuous functions now yields the promptwise estimate ∑y‖∇πθ(y∣x)−∇πθ′(y∣x)‖2 _y \|∇ _θ(y x)-∇ _θ (y x) \|_2 ≤∫01∑y‖ds∇πγ(s)(y∣x)‖2ds ≤ _0^1 _y \| dds∇ _γ(s)(y x) \|_2\,ds ≤TL(2log||+1)‖θ−θ′‖2. ≤ TL(2 |A|+1) \|θ-θ \|_2. It remains to pass from a fixed prompt to the objective. First, ∑y‖∇πθ(y∣x)‖2=y∼πθ(⋅∣x)‖Sθ(y∣x)‖2≤2LTlog||, _y \|∇ _θ(y x) \|_2=E_y _θ(· x) \|S_θ(y x) \|_2≤ 2LT |A|, where Cauchy–Schwarz and eq. 13 give the last inequality. This uniform bound, together with |rϕ|≤R|r_φ|≤ R, dominates the first difference quotients of J. Therefore ∇J(θ)=x∑yrϕ(x,y)∇πθ(y∣x)∇ J(θ)=E_x _yr_φ(x,y)∇ _θ(y x), and the promptwise estimate above gives ‖∇J(θ)−∇J(θ′)‖2 \|∇ J(θ)-∇ J(θ ) \|_2 ≤Rx∑y‖∇πθ(y∣x)−∇πθ′(y∣x)‖2 ≤ RE_x _y \|∇ _θ(y x)-∇ _θ (y x) \|_2 ≤TRL(2log||+1)‖θ−θ′‖2. ≤ TRL(2 |A|+1) \|θ-θ \|_2. To obtain the claimed one-step inequality, integrate once more along θn+sηgn _n+sη g_n: J(θn+1)−J(θn)−η⟨∇J(θn),gn⟩ J( _n+1)-J( _n)-η ∇ J( _n),\,g_n =η∫01⟨∇J(θn+sηgn)−∇J(θn),gn⟩s =η _0^1 ∇ J( _n+sη g_n)-∇ J( _n),\,g_n \,ds ≥−TRL(2log||+1)η2‖gn‖22∫01sds ≥-TRL(2 |A|+1)η^2 \|g_n \|_2^2 _0^1s\,ds =−TRL(2log||+1)η22‖gn‖22. =- TRL(2 |A|+1)η^22 \|g_n \|_2^2. Rearranging proves eq. 18. ∎ Proof of Lemma 1. Conditioning eq. 18 on ℱn−F_n^-, substituting [gn∣ℱn−]=T−1∇J(θn)+bnE[g_n _n^-]=T^-1∇ J( _n)+b_n, and using that θn _n is ℱn−F_n^--measurable gives the first line below. For the second line, apply Young’s inequality in the explicit form ⟨u,v⟩≥−12T‖u‖22−T2‖v‖22(u,v∈ℝd) u,\,v ≥- 12T \|u \|_2^2- T2 \|v \|_2^2 (u,v ^d) with u=∇J(θn)u=∇ J( _n) and v=bnv=b_n: [J(θn+1)∣ℱn−]−J(θn) [J( _n+1) _n^-]-J( _n) ≥ηT‖∇J(θn)‖22+η⟨∇J(θn),bn⟩−TRL(2log||+1)η22[‖gn‖22∣ℱn−] ≥ ηT \|∇ J( _n) \|_2^2+η ∇ J( _n),\,b_n - TRL(2 |A|+1)η^22E[ \|g_n \|_2^2 _n^-] ≥η2T‖∇J(θn)‖22−ηT2‖bn‖22−TRL(2log||+1)η22[‖gn‖22∣ℱn−]. ≥ η2T \|∇ J( _n) \|_2^2- η T2 \|b_n \|_2^2- TRL(2 |A|+1)η^22E[ \|g_n \|_2^2 _n^-]. After taking expectations, move the objective increment to the right and sum from n=0n=0 to N−1N-1. The increments telescope according to ∑n=0N−1[J(θn+1)−J(θn)]=[J(θN)]−[J(θ0)]≤J⋆−[J(θ0)]. _n=0^N-1E[J( _n+1)-J( _n)]=E[J( _N)]-E[J( _0)]≤ J -E[J( _0)]. Consequently, η2T∑n=0N−1‖∇J(θn)‖22≤J⋆−[J(θ0)]+ηT2∑n=0N−1‖bn‖22+TRL(2log||+1)η22∑n=0N−1‖gn‖22. η2T _n=0^N-1E \|∇ J( _n) \|_2^2≤ J -E[J( _0)]+ η T2 _n=0^N-1E \|b_n \|_2^2+ TRL(2 |A|+1)η^22 _n=0^N-1E \|g_n \|_2^2. Dividing by N and multiplying by 2T/η2T/η gives exactly Lemma 1. ∎ Appendix E Optimization over a Second-Moment-Controlled Weighting Class This section justifies the weighting rule underlying GMC-GRPO. We first prove that the admissible mass constraint gives a uniform second-moment bound and, under coverage, bounds the correction bias by the removed importance mass. We then show that GMC uniquely maximizes retained mass among admissible direction-preserving corrections. Comparing the mass removed by GMC and TIC establishes the corresponding ordering of their certified bias envelopes, providing the basis for the convergence analysis that follows. Proof of Proposition 1. For brevity, denote the contribution of group b to eq. 4 by Un,bw:=1TG∑i=1Gwn(b,i)ρn(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i),gnw=1B∑b=1BUn,bw.U_n,b^w:= 1TG _i=1^Gw_n^(b,i) _n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i), g_n^w= 1B _b=1^BU_n,b^w. For one group, weighted Cauchy–Schwarz and eq. 5 give the pathwise bound ‖Un,bw‖22 \|U_n,b^w \|_2^2 ≤(1G∑i=1Gwn(b,i)ρn(b,i))×(1G∑i=1Gwn(b,i)ρn(b,i)‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22) ≤ ( 1G _i=1^Gw_n^(b,i) _n^(b,i) )× ( 1G _i=1^Gw_n^(b,i) _n^(b,i) \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2 ) ≤1+ϵG∑i=1Gρn(b,i)‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22. ≤ 1+εG _i=1^G _n^(b,i) \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2. The first factor is at most 1+ϵ1+ε, while wn(b,i)≤1w_n^(b,i)≤ 1 removes the weights from the second factor. Taking conditional expectation, using |A^(b,i)|≤2R| A^(b,i)|≤ 2R, and changing measure from the behavior policy to the current policy yield [‖Un,bw‖22∣ℋn] [ \|U_n,b^w \|_2^2 _n] ≤4R2(1+ϵ)T2G∑i=1G[ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22|ℋn] ≤ 4R^2(1+ε)T^2G _i=1^GE\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2\, |\,H_n ] =4R2(1+ϵ)T2y∼πθn(⋅∣x(b))‖∇logπθn(y∣x(b))‖22 = 4R^2(1+ε)T^2E_y _ _n(· x^(b)) \|∇ _ _n(y x^(b)) \|_2^2 ≤8(1+ϵ)LR2log||T, ≤ 8(1+ε)LR^2 |A|T, where the last line is eq. 13. Finally, convexity of the squared norm and the tower property give [‖gnw‖22∣ℱn−] [ \|g_n^w \|_2^2 _n^-] ≤[1B∑b=1B‖Un,bw‖22|ℱn−] \! [ 1B _b=1^B \|U_n,b^w \|_2^2\, |\,F_n^- ] =[1B∑b=1B[‖Un,bw‖22∣ℋn]|ℱn−]≤8(1+ϵ)LR2log||T. =E\! [ 1B _b=1^BE[ \|U_n,b^w \|_2^2 _n]\, |\,F_n^- ]≤ 8(1+ε)LR^2 |A|T. For the bias envelope, wn(b,i)≤1w_n^(b,i)≤ 1 implies (1−wn(b,i))ρn(b,i)≥0(1-w_n^(b,i)) _n^(b,i)≥ 0. Conditional Jensen’s inequality, the triangle inequality, |A^(b,i)|≤2R| A^(b,i)|≤ 2R, and eq. 15 then give ‖[bn,bw∣ℋn]‖2 \|E[b_n,b^w _n] \|_2 ≤2RTG∑i=1G[(1−wn(b,i))ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖2|ℋn] ≤ 2RTG _i=1^GE\! [(1-w_n^(b,i)) _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2\, |\,H_n ] ≤2R2Llog[1G∑i=1G(1−wn(b,i))ρn(b,i)|ℋn], ≤ 2R 2L \,E\! [ 1G _i=1^G(1-w_n^(b,i)) _n^(b,i)\, |\,H_n ], which is eq. 7. ∎ Proof of Proposition 2. For a direction-preserving correction with common weight α, the retained mass is αρ¯n(b)α ρ_n^(b). Since ρ¯n(b)>0 ρ_n^(b)>0, maximizing the retained mass—or, equivalently, minimizing the removed mass—amounts to choosing the largest α satisfying 0≤α≤10≤α≤ 1 and αρ¯n(b)≤1+ϵα ρ_n^(b)≤ 1+ε. The unique optimizer is therefore the stated αn(b),⋆ _n^(b), , which is exactly the common scaling in eq. 12. ∎ The same calculation also yields a comparison with TIC-GRPO. GMC removes importance mass (ρ¯n(b)−(1+ϵ))+( ρ_n^(b)-(1+ε))_+. This quantity is zero when ρ¯n(b)≤1+ϵ ρ_n^(b)≤ 1+ε. Otherwise, ρ¯n(b)−(1+ϵ) ρ_n^(b)-(1+ε) =1G∑i:ρn(b,i)>1+ϵρn(b,i)+1G∑i:ρn(b,i)≤1+ϵρn(b,i)−(1+ϵ)≤1G∑i:ρn(b,i)>1+ϵρn(b,i), = 1G _i:\, _n^(b,i)>1+ε _n^(b,i)+ 1G _i:\, _n^(b,i)≤ 1+ε _n^(b,i)-(1+ε)≤ 1G _i:\, _n^(b,i)>1+ε _n^(b,i), because the second normalized sum is at most 1+ϵ1+ε. Thus, in either case, GMC removes no more importance mass than TIC-GRPO. Applying eq. 7 and taking conditional expectations gives the corresponding comparison between their certified bias envelopes. The scope of Proposition 2 is deliberate. It does not claim that GMC minimizes the exact vector bias over arbitrary sample-specific rules: equal retained mass can still lead to different cancellations and hence different biases. Rather, Propositions 1 and 2 identify GMC as the unique direction-preserving rule that retains the shared second-moment guarantee while minimizing a certified bias envelope. TIC-GRPO enforces the same mass cap by deleting individual high-ratio responses; GMC instead applies the smallest uniform attenuation needed to satisfy the cap. The next section uses this groupwise structure to derive the GMC conditional-bias and convergence bounds. Appendix F Convergence analysis of GMC-GRPO This section establishes the convergence guarantee for GMC-GRPO under mixed rollout delays. By Lemma 1, it suffices to control the update second moment and the squared conditional bias. We first decompose the GMC update into the current-policy gradient and an explicit groupwise correction term, and show that the group-mass cap bounds the update second moment independently of the delay. We then bound the correction bias through trajectory divergences between the current and behavior policies, and relate these divergences to the parameter drift accumulated within each delay window. Combining these estimates with Lemma 1 yields the delay-explicit guarantee in Theorem 3. A final local-overlap argument converts the chi-square branch into the group-size refinement used in the main text. Throughout, conditional statements rely on the sampling and filtration conditions in Assumption 3. For each group, define the raw importance-sampling, self-normalized, and group-mass-capped estimators Un,bIS U_n,b^IS :=1TG∑i=1Gρn(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i), := 1TG _i=1^G _n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i), (19) Un,bSN U_n,b^SN :=1TG∑i=1Gρn(b,i)ρ¯n(b)∇logπθn(y(b,i)∣x(b))A^(b,i), := 1TG _i=1^G _n^(b,i) ρ_n^(b)∇ _ _n(y^(b,i) x^(b)) A^(b,i), Un,bGMC U_n,b^GMC :=1TG∑i=1Gρ^n(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i). := 1TG _i=1^G ρ_n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i). Proposition 3 (Conditional gradient decomposition). Under Assumption 1 and the sampling conditions in Assumption 3, for every n,bn,b, the following pathwise identities hold: 1G∑i=1Gρ^n(b,i) 1G _i=1^G ρ_n^(b,i) =minρ¯n(b),1+ϵ, = \ ρ_n^(b),1+ε\, Un,bGMC U_n,b^GMC =Un,bIS,ρ¯n(b)≤1+ϵ,(1+ϵ)Un,bSN,ρ¯n(b)>1+ϵ, = casesU_n,b^IS,& ρ_n^(b)≤ 1+ε,\\ (1+ε)U_n,b^SN,& ρ_n^(b)>1+ε, cases (20) Un,bGMC−Un,bIS U_n,b^GMC-U_n,b^IS =−(ρ¯n(b)−(1+ϵ))+Un,bSN. =-( ρ_n^(b)-(1+ε))_+U_n,b^SN. (21) Moreover, 0≤ρ^n(b,i)≤ρn(b,i)0≤ ρ_n^(b,i)≤ _n^(b,i) for every i. Define bn,b b_n,b :=−[(ρ¯n(b)−(1+ϵ))+Un,bSN|ℋn], :=-E [( ρ_n^(b)-(1+ε))_+U_n,b^SN\, |\,H_n ], (22) bn b_n :=[1B∑b=1Bbn,b|ℱn−]. :=E [ 1B _b=1^Bb_n,b\, |\,F_n^- ]. (23) Then [Un,bGMC∣ℋn] [U_n,b^GMC _n] =1T∇Jx(b)(θn)+bn,b, = 1T∇ J_x^(b)( _n)+b_n,b, (24) [gnGMC∣ℱn−] [g_n^GMC _n^-] =1T∇J(θn)+bn, = 1T∇ J( _n)+b_n, (25) Thus every subsequent bias lemma only needs to control the explicit correction term in eq. 22. Proof. The definition eq. 12 gives the pathwise chain 1G∑i=1Gρ^n(b,i) 1G _i=1^G ρ_n^(b,i) =ρ¯n(b)max1,ρ¯n(b)/(1+ϵ)=minρ¯n(b),1+ϵ, = ρ_n^(b) \1, ρ_n^(b)/(1+ε)\= \ ρ_n^(b),1+ε\, Un,bGMC U_n,b^GMC =minρ¯n(b),1+ϵUn,bSN,Un,bIS=ρ¯n(b)Un,bSN, = \ ρ_n^(b),1+ε\U_n,b^SN, U_n,b^IS= ρ_n^(b)U_n,b^SN, Un,bGMC−Un,bIS U_n,b^GMC-U_n,b^IS =−(ρ¯n(b)−(1+ϵ))+Un,bSN. =-( ρ_n^(b)-(1+ε))_+U_n,b^SN. This proves eq. 20–eq. 21; the denominator in eq. 12 is at least one, so also 0≤ρ^n(b,i)≤ρn(b,i)0≤ ρ_n^(b,i)≤ _n^(b,i). It remains to compute the mean of the raw estimator. Since ρn(b,i) _n^(b,i) is the current-to-behavior likelihood ratio, it cancels the behavior-policy density conditional on ℋnH_n. Hence, for the reward paired with the iith score, [ρn(b,i)∇logπθn(y(b,i)∣x(b))r(b,i)∣ℋn] \! [ _n^(b,i)∇ _ _n(y^(b,i) x^(b))r^(b,i) _n ] =y∼πθn(⋅∣x(b))[∇logπθn(y∣x(b))rϕ(x(b),y)] =E_y _ _n(· x^(b))\! [∇ _ _n(y x^(b))r_φ(x^(b),y) ] =∑yπθn(y∣x(b))∇logπθn(y∣x(b))rϕ(x(b),y)=∇Jx(b)(θn). = _y _ _n(y x^(b))∇ _ _n(y x^(b))r_φ(x^(b),y)=∇ J_x^(b)( _n). For a leave-one-out reward r(b,j)r^(b,j) with j≠ij≠ i, conditional independence separates the two sampled responses. The score factor has conditional mean zero: [ρn(b,i)∇logπθn(y(b,i)∣x(b))∣ℋn] \! [ _n^(b,i)∇ _ _n(y^(b,i) x^(b)) _n ] =∑yπθn(y∣x(b))∇logπθn(y∣x(b)) = _y _ _n(y x^(b))∇ _ _n(y x^(b)) =∑y∇πθn(y∣x(b))=0, = _y∇ _ _n(y x^(b))=0, [ρn(b,i)∇logπθn(y(b,i)∣x(b))r(b,j)∣ℋn] \! [ _n^(b,i)∇ _ _n(y^(b,i) x^(b))r^(b,j) _n ] =0⋅[r(b,j)∣ℋn]=0. =0·E[r^(b,j) _n]=0. Substitution of the leave-one-out advantage therefore yields [Un,bIS∣ℋn] [U_n,b^IS _n] =1TG∑i=1G(∇Jx(b)(θn)−1G−1∑j≠i0)=1T∇Jx(b)(θn), = 1TG _i=1^G (∇ J_x^(b)( _n)- 1G-1 _j≠ i0 )= 1T∇ J_x^(b)( _n), [Un,bGMC∣ℋn] [U_n,b^GMC _n] =[Un,bIS∣ℋn]−[(ρ¯n(b)−(1+ϵ))+Un,bSN∣ℋn]=1T∇Jx(b)(θn)+bn,b, =E[U_n,b^IS _n]-E\! [( ρ_n^(b)-(1+ε))_+U_n,b^SN _n ]= 1T∇ J_x^(b)( _n)+b_n,b, [gnGMC∣ℱn−] [g_n^GMC _n^-] =[1B∑b=1B[Un,bGMC∣ℋn]|ℱn−] =E\! [ 1B _b=1^BE[U_n,b^GMC _n]\, |\,F_n^- ] =1T[1B∑b=1B∇Jx(b)(θn)|ℱn−]+bn=1T∇J(θn)+bn. = 1TE\! [ 1B _b=1^B∇ J_x^(b)( _n)\, |\,F_n^- ]+b_n= 1T∇ J( _n)+b_n. The final equality is the conditional prompt-unbiasedness clause in Assumption 3. These are eq. 24 and eq. 25; bounded rewards and eq. 13 ensure integrability by Cauchy–Schwarz. ∎ Lemma 5 (Expected update second moment). Under Assumption 1 and the sampling conditions in Assumption 3, [‖gnGMC‖22∣ℱn−]≤8(1+ϵ)LR2log||T.E[ \|g_n^GMC \|_2^2 _n^-]≤ 8(1+ε)LR^2 |A|T. (26) Proof. We use the weighted Cauchy–Schwarz inequality ‖∑iaivi‖22≤(∑iai)(∑iai‖vi‖22),ai≥0. \| _ia_iv_i \|_2^2≤ ( _ia_i ) ( _ia_i \|v_i \|_2^2 ), a_i≥ 0. Apply it with ai=ρ^n(b,i)/Ga_i= ρ_n^(b,i)/G and vi=T−1∇logπθn(y(b,i)∣x(b))A^(b,i)v_i=T^-1∇ _ _n(y^(b,i) x^(b)) A^(b,i). Proposition 3 identifies the sum of the weights, while ρ^n(b,i)≤ρn(b,i) ρ_n^(b,i)≤ _n^(b,i) enlarges the second factor. Hence ‖Un,bGMC‖22 \|U_n,b^GMC \|_2^2 =‖1TG∑iρ^n(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i)‖22 = \| 1TG _i ρ_n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2 ≤(1G∑iρ^n(b,i))(1G∑iρ^n(b,i)‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22) ≤ ( 1G _i ρ_n^(b,i) ) ( 1G _i ρ_n^(b,i) \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2 ) ≤min1+ϵ,ρ¯n(b)G∑iρn(b,i)‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22 ≤ \1+ε, ρ_n^(b)\G _i _n^(b,i) \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2 ≤1+ϵG∑iρn(b,i)‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22. ≤ 1+εG _i _n^(b,i) \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2. The second inequality uses G−1∑iρ^n(b,i)=min1+ϵ,ρ¯n(b)G^-1 _i ρ_n^(b,i)= \1+ε, ρ_n^(b)\ and ρ^n(b,i)≤ρn(b,i) ρ_n^(b,i)≤ _n^(b,i); the third uses min1+ϵ,ρ¯n(b)≤1+ϵ \1+ε, ρ_n^(b)\≤ 1+ε. Taking conditional expectation and using |A^(b,i)|≤2R| A^(b,i)|≤ 2R gives [‖Un,bGMC‖22∣ℋn] [ \|U_n,b^GMC \|_2^2 _n] ≤4R2(1+ϵ)T2G∑i=1G[ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22|ℋn] ≤ 4R^2(1+ε)T^2G _i=1^GE\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2\, |\,H_n ] =4R2(1+ϵ)T2y∼πθn(⋅∣x(b))‖∇logπθn(y∣x(b))‖22 = 4R^2(1+ε)T^2E_y _ _n(· x^(b)) \|∇ _ _n(y x^(b)) \|_2^2 ≤8(1+ϵ)LR2log||T. ≤ 8(1+ε)LR^2 |A|T. The equality changes measure from the behavior policy to the current policy, and the final inequality is eq. 13. Finally, convexity gives the pathwise batch bound ‖gnGMC‖22=‖1B∑b=1BUn,bGMC‖22≤1B∑b=1B‖Un,bGMC‖22. \|g_n^GMC \|_2^2= \| 1B _b=1^BU_n,b^GMC \|_2^2≤ 1B _b=1^B \|U_n,b^GMC \|_2^2. Conditioning first on ℋnH_n and then on ℱn−F_n^- proves eq. 26. ∎ Lemma 6 (Bias controls). Under Assumptions 1 and 2 and the sampling conditions in Assumption 3, for every n,bn,b, ‖bn,b‖22≤min8R2Llog||T,8R2Llog||TcϵDn(b),8R2Lloglog2(1+2ϵ)(Dn(b))2,R2Llog2G2ϵ2(χn2,(b))2. \|b_n,b \|_2^2≤ \ 8R^2L |A|T, 8R^2L |A|Tc_εD_n^(b), 8R^2L ^2(1+2ε)(D_n^(b))^2, R^2L 2G^2ε^2( _n^2,(b))^2 \. (27) Proof. Let φ(u):=ulogu−u+1 (u):=u u-u+1 and abbreviate Eb:=ρ¯n(b)>1+ϵE_b:=\ ρ_n^(b)>1+ε\. Expanding the self-normalized update in eq. 22, bn,b=−1T[(ρ¯n(b)−(1+ϵ))+ρ¯n(b)1G∑i=1Gρn(b,i)∇logπθn(y(b,i)∣x(b))A^(b,i)|ℋn].b_n,b=- 1TE\! [ ( ρ_n^(b)-(1+ε))_+ ρ_n^(b) 1G _i=1^G _n^(b,i)∇ _ _n(y^(b,i) x^(b)) A^(b,i)\, |\,H_n ]. The scalar prefactor is at most Eb1_E_b. Conditional Jensen and |A^(b,i)|≤2R| A^(b,i)|≤ 2R therefore imply ‖bn,b‖2≤2RT1G∑i=1G[Ebρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖2|ℋn]. \|b_n,b \|_2≤ 2RT 1G _i=1^GE\! [1_E_b _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2\, |\,H_n ]. For each i, conditional Cauchy–Schwarz bounds the expectation inside the sum as [Ebρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖2|ℋn] \! [1_E_b _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2\, |\,H_n ] ≤[Ebρn(b,i)∣ℋn]1/2×[ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22|ℋn]1/2. [1_E_b _n^(b,i) _n]^1/2×E\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2\, |\,H_n ]^1/2. A second Cauchy–Schwarz inequality over i multiplies the averages of the two right-hand factors. The first average is [ρ¯n(b)Eb∣ℋn]E[ ρ_n^(b)1_E_b _n], so we obtain ‖bn,b‖22 \|b_n,b \|_2^2 ≤4R2T2[ρ¯n(b)ρ¯n(b)>1+ϵ∣ℋn]1G∑i=1G[ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22∣ℋn] ≤ 4R^2T^2E\! [ ρ_n^(b)1\ ρ_n^(b)>1+ε\ _n ] 1G _i=1^GE\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2 _n ] ≤8R2Llog||T[ρ¯n(b)ρ¯n(b)>1+ϵ∣ℋn]. ≤ 8R^2L |A|TE\! [ ρ_n^(b)1\ ρ_n^(b)>1+ε\ _n ]. In the last line, change of measure turns each second conditional expectation in the preceding display into the current-policy score moment, which is at most 2LTlog||2LT |A| by eq. 13. Moreover, [ρ¯n(b)∣ℋn]=1E[ ρ_n^(b) _n]=1 because each importance ratio has conditional mean one. Dropping Eb1_E_b proves the uniform branch. For the KL branch, φ(u)/u=logu−1+u−1 (u)/u= u-1+u^-1 has derivative (u−1)/u2>0(u-1)/u^2>0 for u>1u>1. Its value at u=1+ϵu=1+ε is cϵ=log(1+ϵ)−ϵ/(1+ϵ)c_ε= (1+ε)-ε/(1+ε), so the pointwise inequality cϵuu>1+ϵ≤φ(u),u≥0,c_εu1\u>1+ε\≤ (u), u≥ 0, holds. Apply it to u=ρ¯n(b)u= ρ_n^(b), then use convexity of φ : cϵ[ρ¯n(b)ρ¯n(b)>1+ϵ∣ℋn] c_ε\,E\! [ ρ_n^(b)1\ ρ_n^(b)>1+ε\ _n ] ≤[φ(ρ¯n(b))∣ℋn]≤1G∑i=1G[φ(ρn(b,i))∣ℋn]=Dn(b). [ ( ρ_n^(b)) _n]≤ 1G _i=1^GE[ ( _n^(b,i)) _n]=D_n^(b). For the last equality, change of measure gives [φ(ρn(b,i))∣ℋn]=DKL(πθn∥πθn−τn(b))=Dn(b)E[ ( _n^(b,i)) _n]=D_KL( _ _n\| _ _n- _n^(b))=D_n^(b). Substitution in the preceding bound proves the KL branch. The first two branches do not use Assumption 2. For the remaining branches, coverage and eq. 15 give ‖∇logπθn(y(b,i)∣x(b))‖2≤T2Llog \|∇ _ _n(y^(b,i) x^(b)) \|_2≤ T 2L for every sampled response. Since the normalized weights sum to one, the triangle inequality and |A^(b,i)|≤2R| A^(b,i)|≤ 2R yield the pathwise bound ‖Un,bSN‖2≤2RTG∑i=1Gρn(b,i)ρ¯n(b)‖∇logπθn(y(b,i)∣x(b))‖2≤2R2Llog. \|U_n,b^SN \|_2≤ 2RTG _i=1^G _n^(b,i) ρ_n^(b) \|∇ _ _n(y^(b,i) x^(b)) \|_2≤ 2R 2L . Using this inside eq. 22 and applying conditional Jensen gives ‖bn,b‖2≤[(ρ¯n(b)−(1+ϵ))+‖Un,bSN‖2|ℋn]≤2R2Llog[(ρ¯n(b)−(1+ϵ))+∣ℋn]. \|b_n,b \|_2 \! [( ρ_n^(b)-(1+ε))_+ \|U_n,b^SN \|_2\, |\,H_n ]≤ 2R 2L \,E[( ρ_n^(b)-(1+ε))_+ _n]. (28) For u≥1+ϵu≥ 1+ε, convexity and the tangent to φ at 1+2ϵ1+2ε give φ(u) (u) ≥φ(1+2ϵ)+φ′(1+2ϵ)(u−1−2ϵ) ≥ (1+2ε)+ (1+2ε)(u-1-2ε) =log(1+2ϵ)u−2ϵ = (1+2ε)u-2ε ≥log(1+2ϵ)(u−(1+ϵ)). ≥ (1+2ε)(u-(1+ε)). The last inequality is equivalent to (1+ϵ)log(1+2ϵ)≥2ϵ(1+ε) (1+2ε)≥ 2ε, which follows from logv≥2(v−1)/(v+1) v≥ 2(v-1)/(v+1) with v=1+2ϵv=1+2ε. Applying the resulting pointwise bound to u=ρ¯n(b)u= ρ_n^(b), followed by Jensen and change of measure, gives log(1+2ϵ)[(ρ¯n(b)−(1+ϵ))+|ℋn] (1+2ε)\,E [( ρ_n^(b)-(1+ε))_+\, |\,H_n ] ≤[φ(ρ¯n(b))∣ℋn]≤1G∑i=1G[φ(ρn(b,i))∣ℋn]=Dn(b). [ ( ρ_n^(b)) _n]≤ 1G _i=1^GE[ ( _n^(b,i)) _n]=D_n^(b). Substitution in eq. 28 proves the quadratic-KL branch. For the chi-square branch, the scalar inequality (u−1−ϵ)+≤(u−1)24ϵ,u≥0,(u-1-ε)_+≤ (u-1)^24ε, u≥ 0, is immediate when u≤1+ϵu≤ 1+ε; when u>1+ϵu>1+ε, it is equivalent to (u−1−2ϵ)2≥0(u-1-2ε)^2≥ 0. Next expand the conditional variance of the group mean. Because the responses are conditionally independent and each importance ratio has conditional mean one, for i≠ji≠ j, [(ρn(b,i)−1)(ρn(b,j)−1)∣ℋn]=0.E[( _n^(b,i)-1)( _n^(b,j)-1) _n]=0. Therefore [(ρ¯n(b)−1)2|ℋn] [( ρ_n^(b)-1)^2\, |\,H_n ] =1G2∑i=1G[(ρn(b,i)−1)2∣ℋn]=χn2,(b)G, = 1G^2 _i=1^GE[( _n^(b,i)-1)^2 _n]= _n^2,(b)G, [(ρ¯n(b)−(1+ϵ))+∣ℋn] [( ρ_n^(b)-(1+ε))_+ _n] ≤14ϵ[(ρ¯n(b)−1)2∣ℋn]=χn2,(b)4Gϵ. ≤ 14εE[( ρ_n^(b)-1)^2 _n]= _n^2,(b)4Gε. Substitution in eq. 28 proves the last branch. ∎ Lemma 7 (KL–delay conversions). Suppose the iterates are generated by the GMC update eq. 9, Assumption 1 holds, and both the sampling and delay-regularity conditions in Assumption 3 hold. [Dn(b)∣τn(b)]≤4(1+ϵ)L2R2log||η2(τn(b))2.E[D_n^(b) _n^(b)]≤ 4(1+ε)L^2R^2 |A|\,η^2( _n^(b))^2. (29) If Assumption 2 also holds, then Dn(b) D_n^(b) ≤4(1+ϵ)2TR2L2logη2(τn(b))2almost surely, ≤ 4(1+ε)^2TR^2L^2 \,η^2( _n^(b))^2 surely, [(Dn(b))2∣τn(b)] [(D_n^(b))^2 _n^(b)] ≤16(1+ϵ)3TL4R4log||logη4(τn(b))4. ≤ 16(1+ε)^3TL^4R^4 |A| \,η^4( _n^(b))^4. (30) Proof. Fix t≥0t≥ 0 with Pr(τn(b)=t)>0 ( _n^(b)=t)>0. The learner recursion gives θn−θn−t=η∑s=n−tn−1gsGMC,‖θn−θn−t‖22≤η2t∑s=n−tn−1‖gsGMC‖22, _n- _n-t=η _s=n-t^n-1g_s^GMC, \| _n- _n-t \|_2^2≤η^2t _s=n-t^n-1 \|g_s^GMC \|_2^2, where the inequality is Cauchy–Schwarz over the t learner updates. The delay-regularity condition in Assumption 3 removes the conditioning on the realized delay in the first line below. The tower property and Lemma 5 then bound every update moment: [‖θn−θn−t‖22∣τn(b)=t] [ \| _n- _n-t \|_2^2 _n^(b)=t] =‖θn−θn−t‖22≤η2t∑s=n−tn−1‖gsGMC‖22≤8(1+ϵ)LR2log||Tη2t2, =E \| _n- _n-t \|_2^2≤η^2t _s=n-t^n-1E \|g_s^GMC \|_2^2≤ 8(1+ε)LR^2 |A|Tη^2t^2, [Dn(b)∣τn(b)=t] [D_n^(b) _n^(b)=t] ≤TL2[‖θn−θn−t‖22∣τn(b)=t]≤4(1+ϵ)L2R2log||η2t2. ≤ TL2E[ \| _n- _n-t \|_2^2 _n^(b)=t]≤ 4(1+ε)L^2R^2 |A|\,η^2t^2. The second line applies the trajectory-KL bound eq. 17; this proves eq. 29. Under Assumption 2, the capped group mass and eq. 15 give, for every group, ‖Us,bGMC‖2 \|U_s,b^GMC \|_2 ≤1TG∑i=1Gρ^s(b,i)‖∇logπθs(y(b,i)∣x(b))‖2|A^(b,i)|≤2R(1+ϵ)2Llog. ≤ 1TG _i=1^G ρ_s^(b,i) \|∇ _ _s(y^(b,i) x^(b)) \|_2| A^(b,i)|≤ 2R(1+ε) 2L . The same bound holds for gsGMCg_s^GMC, because it is the average of the group updates. The triangle inequality along the learner path therefore yields ‖θn−θn−t‖2≤η∑s=n−tn−1‖gsGMC‖2≤2R(1+ϵ)2Llogηt. \| _n- _n-t \|_2≤η _s=n-t^n-1 \|g_s^GMC \|_2≤ 2R(1+ε) 2L \,η t. Combining this pathwise estimate with eq. 17 gives Dn(b) D_n^(b) ≤TL2‖θn−θn−t‖22≤4(1+ϵ)2TR2L2logη2t2=:Dmax, ≤ TL2 \| _n- _n-t \|_2^2≤ 4(1+ε)^2TR^2L^2 \,η^2t^2=:D_ , [(Dn(b))2∣τn(b)=t] [(D_n^(b))^2 _n^(b)=t] ≤4(1+ϵ)2TR2L2logη2t2[Dn(b)∣τn(b)=t] ≤ 4(1+ε)^2TR^2L^2 \,η^2t^2E[D_n^(b) _n^(b)=t] ≤16(1+ϵ)3TL4R4log||logη4t4, ≤ 16(1+ε)^3TL^4R^4 |A| \,η^4t^4, where the middle inequality uses 0≤Dn(b)≤4(1+ϵ)2TR2L2logη2t20≤ D_n^(b)≤ 4(1+ε)^2TR^2L^2 \,η^2t^2, hence (Dn(b))2≤DmaxDn(b)(D_n^(b))^2≤ D_ D_n^(b). This proves eq. 30. ∎ Define the local coverage penalty Γn,bGMC:=min8LTR2log||,8TR2Llog||cϵDn(b),8T2R2Lloglog2(1+2ϵ)(Dn(b))2,T2R2Llog2G2ϵ2(χn2,(b))2. _n,b^GMC:= \8LTR^2 |A|, 8TR^2L |A|c_εD_n^(b), 8T^2R^2L ^2(1+2ε)(D_n^(b))^2, T^2R^2L 2G^2ε^2( _n^2,(b))^2 \. (31) Since Dn(b)D_n^(b) and χn2,(b) _n^2,(b) are ℋnH_n-measurable, Γn,bGMC _n,b^GMC is a nonnegative ℋnH_n-measurable random variable bounded by 8LTR2log||8LTR^2 |A|. Theorem 3 (Delay-explicit convergence guarantee for GMC-GRPO). Suppose the iterates are generated by the GMC update eq. 9, Assumptions 1 and 2 hold, and the sampling conditions in Assumption 3 hold. Then, for every η>0η>0 and N≥1N≥ 1, 1N∑n=0N−1‖∇J(θn)‖22≤ 1N _n=0^N-1E \|∇ J( _n) \|_2^2≤ 2T(J⋆−[J(θ0)])ηN+8T(1+ϵ)L2R3log||(2log||+1)η 2T (J -E[J( _0)] )η N+8T(1+ε)L^2R^3 |A|\,(2 |A|+1)η +1N∑n=0N−1[1B∑b=1BΓn,bGMC]. + 1N _n=0^N-1E [ 1B _b=1^B _n,b^GMC ]. (32) If the delay-regularity condition in Assumption 3 also holds, the final term is at most min \ 8LTR2log||,32(1+ϵ)L3R4T(log||)2cϵη2τ¯2,128(1+ϵ)3L5R6T3log||(log)2log2(1+2ϵ)η4τ¯4, 8LTR^2 |A|, 32(1+ε)L^3R^4T( |A|)^2c_εη^2 τ_2, 128(1+ε)^3L^5R^6T^3 |A|( )^2 ^2(1+2ε)η^4 τ_4, (33) T2R2Llog2G2ϵ21N∑n=0N−1[1B∑b=1B(χn2,(b))2], T^2R^2L 2G^2ε^2 1N _n=0^N-1E [ 1B _b=1^B( _n^2,(b))^2 ] \, where a branch equal to +∞+∞ is simply inactive in the minimum. Each branch of eq. 33 is a valid bound on its own. The uniform branch shows that the correction bias always contributes a bounded amount. The second- and fourth-order branches make the dependence on rollout delay explicit through τ¯2 τ_2 and τ¯4 τ_4. The chi-square branch yields the group-size refinement in Corollary 2. Theorem 1 in the main text retains the fourth-order branch. Proof. The proof has three steps: we verify the hypothesis of Lemma 1, bound its update-moment and bias terms, and finally convert the bias penalty into delay moments. Step 1: conditional-mean structure. By eq. 25, [gnGMC∣ℱn−]=T−1∇J(θn)+bnE[g_n^GMC _n^-]=T^-1∇ J( _n)+b_n, where bnb_n is given by eq. 23 and is ℱn−F_n^--measurable. The second-moment bound in Lemma 5 and the uniform branch in eq. 27 show that gnGMCg_n^GMC and bnb_n are square integrable. Hence Lemma 1 applies and gives 1N∑n=0N−1‖∇J(θn)‖22≤ 1N _n=0^N-1E \|∇ J( _n) \|_2^2≤ 2T(J⋆−[J(θ0)])ηN+T2RL(2log||+1)ηN∑n=0N−1‖gnGMC‖22+T2N∑n=0N−1‖bn‖22. 2T(J -E[J( _0)])η N+ T^2RL(2 |A|+1)ηN _n=0^N-1E \|g_n^GMC \|_2^2+ T^2N _n=0^N-1E \|b_n \|_2^2. Step 2: update moment and bias. Taking expectations in Lemma 5 gives ‖gnGMC‖22≤8(1+ϵ)LR2log||/TE \|g_n^GMC \|_2^2≤ 8(1+ε)LR^2 |A|/T for every n. Therefore T2RL(2log||+1)ηN∑n=0N−1‖gnGMC‖22 T^2RL(2 |A|+1)ηN _n=0^N-1E \|g_n^GMC \|_2^2 ≤T2RL(2log||+1)η⋅8(1+ϵ)LR2log||T ≤ T^2RL(2 |A|+1)η· 8(1+ε)LR^2 |A|T =8T(1+ϵ)L2R3log||(2log||+1)η. =8T(1+ε)L^2R^3 |A|(2 |A|+1)η. For the bias term, substitute eq. 23. Conditional Jensen for the squared norm, followed by convexity of the squared norm over the uniform average of the B groups, gives ‖bn‖22=‖[1B∑b=1Bbn,b|ℱn−]‖22≤[‖1B∑b=1Bbn,b‖22|ℱn−]≤[1B∑b=1B‖bn,b‖22|ℱn−]. \|b_n \|_2^2= \|E\! [ 1B _b=1^Bb_n,b\, |\,F_n^- ] \|_2^2 \! [ \| 1B _b=1^Bb_n,b \|_2^2\, |\,F_n^- ] \! [ 1B _b=1^B \|b_n,b \|_2^2\, |\,F_n^- ]. The bound eq. 27 in Lemma 6 holds pathwise for every (n,b)(n,b), and Γn,bGMC _n,b^GMC is exactly T2T^2 times its four-branch minimum. Taking expectations therefore yields T2‖bn‖22≤T2[1B∑b=1B‖bn,b‖22]≤[1B∑b=1BΓn,bGMC].T^2E \|b_n \|_2^2≤ T^2E [ 1B _b=1^B \|b_n,b \|_2^2 ] [ 1B _b=1^B _n,b^GMC ]. Substituting the moment and bias bounds into the inequality of Step 1 proves eq. 32. Step 3: conversion to delay moments. Suppose now that the delay-regularity condition in Assumption 3 also holds. For nonnegative random variables X1,…,X4X_1,…,X_4, [minkXk]≤mink[Xk]E[ _kX_k]≤ _kE[X_k]. Hence the final term of eq. 32 is at most the minimum of the four averaged branches, and it suffices to bound each branch separately. The uniform branch is the constant 8LTR2log||8LTR^2 |A|. For the linear KL branch, the tower property and eq. 29 give [Dn(b)]=[[Dn(b)∣τn(b)]]≤4(1+ϵ)L2R2log||η2[(τn(b))2]E[D_n^(b)]=E [E[D_n^(b) _n^(b)] ]≤ 4(1+ε)L^2R^2 |A|\,η^2\,E[( _n^(b))^2]. Averaging over b and n and using the definition eq. 11 of τ¯2 τ_2, 1N∑n=0N−1[1B∑b=1B8TR2Llog||cϵDn(b)]≤32(1+ϵ)L3R4T(log||)2cϵη2τ¯2. 1N _n=0^N-1E [ 1B _b=1^B 8TR^2L |A|c_εD_n^(b) ]≤ 32(1+ε)L^3R^4T( |A|)^2c_εη^2 τ_2. For the quadratic-KL branch, the tower property and eq. 30 give [(Dn(b))2]≤16(1+ϵ)3TL4R4log||logη4[(τn(b))4]E[(D_n^(b))^2]≤ 16(1+ε)^3TL^4R^4 |A| \,η^4\,E[( _n^(b))^4]. Averaging in the same way, 1N∑n=0N−1[1B∑b=1B8T2R2Lloglog2(1+2ϵ)(Dn(b))2]≤128(1+ϵ)3L5R6T3log||(log)2log2(1+2ϵ)η4τ¯4. 1N _n=0^N-1E [ 1B _b=1^B 8T^2R^2L ^2(1+2ε)(D_n^(b))^2 ]≤ 128(1+ε)^3L^5R^6T^3 |A|( )^2 ^2(1+2ε)η^4 τ_4. The chi-square branch is kept in its averaged form T2R2Llog2G2ϵ21N∑n=0N−1[1B∑b=1B(χn2,(b))2]. T^2R^2L 2G^2ε^2 1N _n=0^N-1E [ 1B _b=1^B( _n^2,(b))^2 ]. Taking the minimum of these four upper bounds proves eq. 33. ∎ Corollary 3 (Local overlap converts chi-square to KL). Suppose additionally that, for some finite δ≥0δ≥ 0, the trajectory likelihood ratio satisfies |ρn(b,i)−1|≤δ| _n^(b,i)-1|≤δ almost surely for all n,b,in,b,i. Then χn2,(b)≤2(1+δ)Dn(b) _n^2,(b)≤ 2(1+δ)D_n^(b) almost surely for every n,bn,b. Proof. Let φ(u)=ulogu−u+1 (u)=u u-u+1. Softmax full support implies ρn(b,i)>0 _n^(b,i)>0 almost surely. For any u>0u>0 satisfying |u−1|≤δ|u-1|≤δ and every s∈[0,1]s∈[0,1], 0<1+s(u−1)≤max1,u≤1+δ0<1+s(u-1)≤ \1,u\≤ 1+δ. Hence φ′(1+s(u−1))≥(1+δ)−1 (1+s(u-1))≥(1+δ)^-1. Since φ(1)=φ′(1)=0 (1)= (1)=0, Taylor’s formula with integral remainder gives φ(u)=(u−1)2∫01(1−s)φ′(1+s(u−1))s≥(u−1)22(1+δ). (u)=(u-1)^2 _0^1(1-s) (1+s(u-1))\,ds≥ (u-1)^22(1+δ). Apply this pointwise to u=ρn(b,i)u= _n^(b,i) and take expectation under the behavior policy: Dn(b)=[φ(ρn(b,i))∣ℋn]≥12(1+δ)[(ρn(b,i)−1)2∣ℋn]=χn2,(b)2(1+δ),D_n^(b)=E[ ( _n^(b,i)) _n]≥ 12(1+δ)E[( _n^(b,i)-1)^2 _n]= _n^2,(b)2(1+δ), which proves the claim. ∎ Appendix G Convergence Analysis of TIC-GRPO The TIC-GRPO analysis follows the same proof structure, making the effect of individual ratio truncation explicit. We decompose its update into the raw importance-sampling estimator and the discarded contribution, establish the shared second-moment bound, and bound the clipping bias using trajectory divergences. Combining these estimates with the parameter-drift argument and Lemma 1 yields the TIC convergence bound. The resulting coefficients allow a direct comparison with GMC under the same assumptions. For every group, define the individually up-clipped estimator Un,bTIC:=1TG∑i=1Gρn(b,i)ρn(b,i)≤1+ϵ∇logπθn(y(b,i)∣x(b))A^(b,i),U_n,b^TIC:= 1TG _i=1^G _n^(b,i)1\ _n^(b,i)≤ 1+ε\∇ _ _n(y^(b,i) x^(b)) A^(b,i), (34) and the full mini-batch update gnTIC:=1B∑b=1BUn,bTIC,θn+1=θn+ηgnTIC.g_n^TIC:= 1B _b=1^BU_n,b^TIC, _n+1= _n+η g_n^TIC. (35) Proposition 4 (TIC-GRPO conditional gradient decomposition). Under Assumption 1 and the sampling conditions in Assumption 3, let Un,bISU_n,b^IS be the raw estimator in eq. 19. Pathwise, Un,bTIC−Un,bIS=−1TG∑i=1Gρn(b,i)ρn(b,i)>1+ϵ∇logπθn(y(b,i)∣x(b))A^(b,i).U_n,b^TIC-U_n,b^IS=- 1TG _i=1^G _n^(b,i)1\ _n^(b,i)>1+ε\∇ _ _n(y^(b,i) x^(b)) A^(b,i). (36) Set bn,bTIC:=[Un,bTIC−Un,bIS∣ℋn]b_n,b^TIC:=E[U_n,b^TIC-U_n,b^IS _n]. Then [Un,bTIC∣ℋn] [U_n,b^TIC _n] =1T∇Jx(b)(θn)+bn,bTIC, = 1T∇ J_x^(b)( _n)+b_n,b^TIC, (37) [gnTIC∣ℱn−] [g_n^TIC _n^-] =1T∇J(θn)+bnTIC, = 1T∇ J( _n)+b_n^TIC, (38) where bnTIC:=[B−1∑b=1Bbn,bTIC∣ℱn−]b_n^TIC:=E[B^-1 _b=1^Bb_n,b^TIC _n^-]. Proof. Subtracting eq. 19 from eq. 34 gives eq. 36. Proposition 3 already proves the mean identity for the raw estimator, including cancellation of the leave-one-out baseline. Hence [Un,bTIC∣ℋn] [U_n,b^TIC _n] =[Un,bIS∣ℋn]+bn,bTIC=1T∇Jx(b)(θn)+bn,bTIC, =E[U_n,b^IS _n]+b_n,b^TIC= 1T∇ J_x^(b)( _n)+b_n,b^TIC, [gnTIC∣ℱn−] [g_n^TIC _n^-] =[1B∑b=1B[Un,bTIC∣ℋn]|ℱn−]=1T∇J(θn)+bnTIC. =E\! [ 1B _b=1^BE[U_n,b^TIC _n]\, |\,F_n^- ]= 1T∇ J( _n)+b_n^TIC. This proves eq. 37–eq. 38. ∎ Remark 8 (The TIC bias does not depend on G). Fix ℋnH_n and write bE_b for expectation under the behavior policy πθn−τn(b)(⋅∣x(b)) _ _n- _n^(b)(· x^(b)). Since the responses in a group are conditionally i.i.d. and r(b,j)r^(b,j) is independent of y(b,i)y^(b,i) for j≠ij≠ i, substituting the leave-one-out advantage into eq. 36 gives bn,bTIC=−1Tb[ρ 1ρ>1+ϵ∇logπθn(y∣x(b))(rϕ(x(b),y)−b[rϕ(x(b),y)])],b_n,b^TIC=- 1T\,E_b [ρ\,1\ρ>1+ε\\,∇ _ _n(y x^(b)) (r_φ(x^(b),y)-E_b[r_φ(x^(b),y)] ) ], where ρ is the current-to-behavior ratio of y. The right-hand side does not depend on G and is nonzero in general. In contrast, the chi-square branch of eq. 27 shows that the GMC bias is O(χn2,(b)/(Gϵ))O ( _n^2,(b)/(Gε) ) and vanishes as G→∞G→∞ for fixed behavior and current policies. Lemma 8 (Expected TIC-GRPO update second moment). Under Assumption 1 and the sampling conditions in Assumption 3, [‖gnTIC‖22∣ℱn−]≤8(1+ϵ)LR2log||T.E[ \|g_n^TIC \|_2^2 _n^-]≤ 8(1+ε)LR^2 |A|T. (39) The GMC counterpart in Lemma 5 has the same constant. Proof. We use the weighted Cauchy–Schwarz inequality ‖∑iaivi‖22≤(∑iai)(∑iai‖vi‖22),ai≥0. \| _ia_iv_i \|_2^2≤ ( _ia_i ) ( _ia_i \|v_i \|_2^2 ), a_i≥ 0. Apply it with ai=G−1ρn(b,i)ρn(b,i)≤1+ϵa_i=G^-1 _n^(b,i)1\ _n^(b,i)≤ 1+ε\ and vi=T−1∇logπθn(y(b,i)∣x(b))A^(b,i)v_i=T^-1∇ _ _n(y^(b,i) x^(b)) A^(b,i). Since each retained ratio is at most 1+ϵ1+ε, ‖Un,bTIC‖22 \|U_n,b^TIC \|_2^2 ≤(1G∑i=1Gρn(b,i)ρn(b,i)≤1+ϵ)×(1G∑i=1Gρn(b,i)ρn(b,i)≤1+ϵ‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22) ≤ ( 1G _i=1^G _n^(b,i)1\ _n^(b,i)≤ 1+ε\ )× ( 1G _i=1^G _n^(b,i)1\ _n^(b,i)≤ 1+ε\ \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2 ) ≤1+ϵG∑i=1Gρn(b,i)‖1T∇logπθn(y(b,i)∣x(b))A^(b,i)‖22. ≤ 1+εG _i=1^G _n^(b,i) \| 1T∇ _ _n(y^(b,i) x^(b)) A^(b,i) \|_2^2. Taking conditional expectation and using |A^(b,i)|≤2R| A^(b,i)|≤ 2R gives [‖Un,bTIC‖22∣ℋn] [ \|U_n,b^TIC \|_2^2 _n] ≤4R2(1+ϵ)T2G∑i=1G[ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22|ℋn] ≤ 4R^2(1+ε)T^2G _i=1^GE\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2\, |\,H_n ] =4R2(1+ϵ)T2y∼πθn(⋅∣x(b))‖∇logπθn(y∣x(b))‖22 = 4R^2(1+ε)T^2E_y _ _n(· x^(b)) \|∇ _ _n(y x^(b)) \|_2^2 ≤8(1+ϵ)LR2log||T. ≤ 8(1+ε)LR^2 |A|T. The equality changes measure from the behavior policy to the current policy, and the last inequality is eq. 13. Finally, convexity gives ‖gnTIC‖22=‖1B∑b=1BUn,bTIC‖22≤1B∑b=1B‖Un,bTIC‖22. \|g_n^TIC \|_2^2= \| 1B _b=1^BU_n,b^TIC \|_2^2≤ 1B _b=1^B \|U_n,b^TIC \|_2^2. Conditioning first on ℋnH_n and then on ℱn−F_n^- proves eq. 39. ∎ Lemma 9 (TIC-GRPO clipping-error bounds). Under Assumptions 1 and 2 and the sampling conditions in Assumption 3, for every n,bn,b, ‖bn,bTIC‖22≤min8R2Llog||T,8R2Llog||TcϵDn(b),8R2Llogcϵ2(Dn(b))2,8R2Llog(1+ϵ)2ϵ4(χn2,(b))2. \|b_n,b^TIC \|_2^2≤ \ 8R^2L |A|T, 8R^2L |A|Tc_εD_n^(b), 8R^2L c_ε^2(D_n^(b))^2, 8R^2L (1+ε)^2ε^4( _n^2,(b))^2 \. (40) Proof. Let φ(u):=ulogu−u+1 (u):=u u-u+1. Expanding eq. 36, applying conditional Jensen, and using |A^(b,i)|≤2R| A^(b,i)|≤ 2R give ‖bn,bTIC‖2≤2RTG∑i=1G[ρn(b,i)ρn(b,i)>1+ϵ‖∇logπθn(y(b,i)∣x(b))‖2|ℋn]. \|b_n,b^TIC \|_2≤ 2RTG _i=1^GE\! [ _n^(b,i)1\ _n^(b,i)>1+ε\ \|∇ _ _n(y^(b,i) x^(b)) \|_2\, |\,H_n ]. For each i, conditional Cauchy–Schwarz gives [ρn(b,i)ρn(b,i)>1+ϵ‖∇logπθn(y(b,i)∣x(b))‖2|ℋn]2 \! [ _n^(b,i)1\ _n^(b,i)>1+ε\ \|∇ _ _n(y^(b,i) x^(b)) \|_2\, |\,H_n ]^2 ≤[ρn(b,i)ρn(b,i)>1+ϵ|ℋn][ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22|ℋn]. \! [ _n^(b,i)1\ _n^(b,i)>1+ε\\, |\,H_n ]E\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2\, |\,H_n ]. All G responses have the same conditional law. Squaring the preceding bound, changing measure in the score-moment factor, and applying eq. 13 therefore yield ‖bn,bTIC‖22 \|b_n,b^TIC \|_2^2 ≤4R2T2[ρn(b,i)ρn(b,i)>1+ϵ∣ℋn][ρn(b,i)‖∇logπθn(y(b,i)∣x(b))‖22∣ℋn] ≤ 4R^2T^2E[ _n^(b,i)1\ _n^(b,i)>1+ε\ _n]E\! [ _n^(b,i) \|∇ _ _n(y^(b,i) x^(b)) \|_2^2 _n ] ≤8R2Llog||T[ρn(b,i)ρn(b,i)>1+ϵ∣ℋn]. ≤ 8R^2L |A|TE[ _n^(b,i)1\ _n^(b,i)>1+ε\ _n]. The tail mass is at most [ρn(b,i)∣ℋn]=1E[ _n^(b,i) _n]=1, which proves the uniform branch. For the KL branch, φ(u)/u=logu−1+u−1 (u)/u= u-1+u^-1 is increasing for u>1u>1, and its value at u=1+ϵu=1+ε is cϵc_ε. Hence cϵuu>1+ϵ≤φ(u),u≥0.c_εu1\u>1+ε\≤ (u), u≥ 0. Applying this inequality to u=ρn(b,i)u= _n^(b,i) and changing measure gives cϵ[ρn(b,i)ρn(b,i)>1+ϵ∣ℋn]≤[φ(ρn(b,i))∣ℋn]=Dn(b).c_ε\,E[ _n^(b,i)1\ _n^(b,i)>1+ε\ _n] [ ( _n^(b,i)) _n]=D_n^(b). Substitution proves the KL branch. The first two branches do not use Assumption 2. For the remaining branches, eq. 15 bounds the score norm in the first display uniformly, so ‖bn,bTIC‖2 \|b_n,b^TIC \|_2 ≤2R2Llog[ρn(b,i)ρn(b,i)>1+ϵ∣ℋn] ≤ 2R 2L \,E[ _n^(b,i)1\ _n^(b,i)>1+ε\ _n] ≤2R2LlogminDn(b)cϵ,1+ϵ2χn2,(b), ≤ 2R 2L \, \ D_n^(b)c_ε, 1+ε^2 _n^2,(b) \, where the first alternative is the KL tail bound above. For the second, use the pointwise inequality uu>1+ϵ≤1+ϵ2(u−1)2.u1\u>1+ε\≤ 1+ε^2(u-1)^2. Indeed, u/(u−1)2u/(u-1)^2 is decreasing for u>1u>1, so on the event u>1+ϵu>1+ε it is at most (1+ϵ)/ϵ2(1+ε)/ε^2. Taking behavior-policy expectation turns (u−1)2(u-1)^2 into χn2,(b) _n^2,(b). Squaring the two alternatives proves the additional branches in eq. 40. ∎ For each realized group, define the local TIC penalty Γn,bTIC:=min8LTR2log||,8TR2Llog||cϵDn(b),8T2R2Llogcϵ2(Dn(b))2,8T2R2Llog(1+ϵ)2ϵ4(χn2,(b))2. _n,b^TIC:= \8LTR^2 |A|, 8TR^2L |A|c_εD_n^(b), 8T^2R^2L c_ε^2(D_n^(b))^2, 8T^2R^2L (1+ε)^2ε^4( _n^2,(b))^2 \. (41) Theorem 4 (Delay-explicit convergence guarantee for TIC-GRPO). Suppose the iterates are generated by the TIC update eq. 35, Assumptions 1 and 2 hold, and the sampling conditions in Assumption 3 hold. Then, for every η>0η>0 and N≥1N≥ 1, 1N∑n=0N−1‖∇J(θn)‖22≤ 1N _n=0^N-1E \|∇ J( _n) \|_2^2≤ 2T(J⋆−[J(θ0)])ηN+8T(1+ϵ)L2R3log||(2log||+1)η 2T (J -E[J( _0)] )η N+8T(1+ε)L^2R^3 |A|\,(2 |A|+1)η +1N∑n=0N−1[1B∑b=1BΓn,bTIC], + 1N _n=0^N-1E [ 1B _b=1^B _n,b^TIC ], (42) where Γn,bTIC _n,b^TIC is defined in eq. 41. If the delay-regularity condition in Assumption 3 also holds, the final term is at most min \ 8LTR2log||,32(1+ϵ)L3R4T(log||)2cϵη2τ¯2,128(1+ϵ)3L5R6T3log||(log)2cϵ2η4τ¯4, 8LTR^2 |A|, 32(1+ε)L^3R^4T( |A|)^2c_εη^2 τ_2, 128(1+ε)^3L^5R^6T^3 |A|( )^2c_ε^2η^4 τ_4, (43) 8T2R2Llog(1+ϵ)2ϵ41N∑n=0N−1[1B∑b=1B(χn2,(b))2]. 8T^2R^2L (1+ε)^2ε^4 1N _n=0^N-1E [ 1B _b=1^B( _n^2,(b))^2 ] \. Proof. Equation 38 verifies the conditional-mean hypothesis of Lemma 1. The update-moment bound eq. 39 contributes T2RL(2log||+1)η⋅8(1+ϵ)LR2log||T=8T(1+ϵ)L2R3log||(2log||+1)η.T^2RL(2 |A|+1)η\,·\, 8(1+ε)LR^2 |A|T=8T(1+ε)L^2R^3 |A|(2 |A|+1)η. For the bias term, conditional Jensen and eq. 40 give T2‖bnTIC‖22≤T2[1B∑b=1B‖bn,bTIC‖22]≤[1B∑b=1BΓn,bTIC].T^2E \|b_n^TIC \|_2^2≤ T^2E [ 1B _b=1^B \|b_n,b^TIC \|_2^2 ] [ 1B _b=1^B _n,b^TIC ]. Substituting both estimates into Lemma 1 gives 1N∑n=0N−1‖∇J(θn)‖22 1N _n=0^N-1E \|∇ J( _n) \|_2^2 ≤2T(J⋆−[J(θ0)])ηN+8T(1+ϵ)L2R3log||(2log||+1)η ≤ 2T(J -E[J( _0)])η N+8T(1+ε)L^2R^3 |A|(2 |A|+1)η +1N∑n=0N−1[1B∑b=1BΓn,bTIC]. + 1N _n=0^N-1E [ 1B _b=1^B _n,b^TIC ]. This proves eq. 42. Now additionally impose the delay-regularity condition in Assumption 3. Fix t≥0t≥ 0 with Pr(τn(b)=t)>0 ( _n^(b)=t)>0. The learner recursion and Cauchy–Schwarz over the t updates give θn−θn−t=η∑s=n−tn−1gsTIC,‖θn−θn−t‖22≤η2t∑s=n−tn−1‖gsTIC‖22. _n- _n-t=η _s=n-t^n-1g_s^TIC, \| _n- _n-t \|_2^2≤η^2t _s=n-t^n-1 \|g_s^TIC \|_2^2. This condition removes the conditioning on the realized delay, and eq. 39 bounds every update moment. Therefore [‖θn−θn−t‖22∣τn(b)=t] [ \| _n- _n-t \|_2^2 _n^(b)=t] =‖θn−θn−t‖22≤8(1+ϵ)LR2log||Tη2t2, =E \| _n- _n-t \|_2^2≤ 8(1+ε)LR^2 |A|Tη^2t^2, [Dn(b)∣τn(b)=t] [D_n^(b) _n^(b)=t] ≤TL2[‖θn−θn−t‖22∣τn(b)=t]≤4(1+ϵ)L2R2log||η2t2. ≤ TL2E[ \| _n- _n-t \|_2^2 _n^(b)=t]≤ 4(1+ε)L^2R^2 |A|\,η^2t^2. Under coverage, every retained ratio is at most 1+ϵ1+ε, and eq. 15 gives, for every group, ‖Us,bTIC‖2 \|U_s,b^TIC \|_2 ≤2RTG∑i=1Gρs(b,i)ρs(b,i)≤1+ϵ‖∇logπθs(y(b,i)∣x(b))‖2≤2R(1+ϵ)2Llog. ≤ 2RTG _i=1^G _s^(b,i)1\ _s^(b,i)≤ 1+ε\ \|∇ _ _s(y^(b,i) x^(b)) \|_2≤ 2R(1+ε) 2L . The same bound holds for gsTICg_s^TIC, because it is the average of the group updates. The triangle inequality along the learner path thus yields ‖θn−θn−t‖2≤η∑s=n−tn−1‖gsTIC‖2≤2R(1+ϵ)2Llogηt. \| _n- _n-t \|_2≤η _s=n-t^n-1 \|g_s^TIC \|_2≤ 2R(1+ε) 2L \,η t. Combining this pathwise estimate with eq. 17 gives Dn(b) D_n^(b) ≤TL2‖θn−θn−t‖22≤4(1+ϵ)2TR2L2logη2t2=:Dmax, ≤ TL2 \| _n- _n-t \|_2^2≤ 4(1+ε)^2TR^2L^2 \,η^2t^2=:D_ , [(Dn(b))2∣τn(b)=t] [(D_n^(b))^2 _n^(b)=t] ≤4(1+ϵ)2TR2L2logη2t2[Dn(b)∣τn(b)=t] ≤ 4(1+ε)^2TR^2L^2 \,η^2t^2E[D_n^(b) _n^(b)=t] ≤16(1+ϵ)3TL4R4log||logη4t4. ≤ 16(1+ε)^3TL^4R^4 |A| \,η^4t^4. The middle inequality uses the almost-sure upper bound on Dn(b)D_n^(b), namely (Dn(b))2≤DmaxDn(b)(D_n^(b))^2≤ D_ D_n^(b). For example, substituting the last line into the quadratic-KL branch gives 8T2R2Llogcϵ21N∑n=0N−1[1B∑b=1B(Dn(b))2]≤128(1+ϵ)3L5R6T3log||(log)2cϵ2η4τ¯4. 8T^2R^2L c_ε^2 1N _n=0^N-1E [ 1B _b=1^B(D_n^(b))^2 ]≤ 128(1+ε)^3L^5R^6T^3 |A|( )^2c_ε^2η^4 τ_4. The expected-drift line gives the second branch of eq. 43; the uniform branch is constant, and the empirical chi-square branch is left in its averaged form. The average pointwise minimum is no larger than any of these four branchwise averages, so their minimum proves eq. 43. ∎ Appendix H From the appendix bounds to the main-text rates This section derives the main-text rates from the detailed bounds above. We first prove a step-size balancing lemma for the optimization, stochastic, and fourth-order delay terms, then apply it to the GMC bound. We next optimize the GMC threshold asymptotically and use the local-overlap estimate to obtain the group-size refinement. Applying the same balancing lemma to TIC completes the main-text guarantees and makes their different delay-dependent coefficients explicit. Lemma 10 (Balancing optimization, stochastic, and delay terms). Let A,B>0A,B>0, C≥0C≥ 0, and M≥1M≥ 1. Suppose that, for every η>0η>0, FM(η)≤AMη+Bη+Cη4,F_M(η)≤ AMη+Bη+Cη^4, (44) then, with (A/(CM))1/5:=+∞(A/(CM))^1/5:=+∞ when C=0C=0, the choice η⋆:=minABM,(ACM)1/5,η := \ ABM, ( ACM )^1/5 \, (45) satisfies FM(η⋆)≤2ABM+2(A4CM4)1/5.F_M(η )≤ 2 ABM+2 ( A^4CM^4 )^1/5. (46) Proof. If C=0C=0, substituting η⋆=A/(BM)η = A/(BM) directly gives FM(η⋆)≤2AB/MF_M(η )≤ 2 AB/M. Suppose henceforth that C>0C>0. The definition eq. 45 implies AMη⋆ AMη =maxABM,(A4CM4)1/5≤ABM+(A4CM4)1/5, = \ ABM, ( A^4CM^4 )^1/5 \≤ ABM+ ( A^4CM^4 )^1/5, Bη⋆ Bη ≤ABM,C(η⋆)4≤(A4CM4)1/5, ≤ ABM, C(η )^4≤ ( A^4CM^4 )^1/5, where the equality uses 1/minu,v=max1/u,1/v1/ \u,v\= \1/u,1/v\ and the final inequality uses maxa,b≤a+b \a,b\≤ a+b for a,b≥0a,b≥ 0. Summing the three displayed bounds proves eq. 46. ∎ Proof of Theorem 1. Retain the quadratic-KL branch in eq. 33. With M=NM=N, the result has the form eq. 44 with A A =2T(J⋆−[J(θ0)]), =2T (J -E[J( _0)] ), B B =8T(1+ϵ)L2R3log||(2log||+1), =8T(1+ε)L^2R^3 |A|(2 |A|+1), C C =128(1+ϵ)3L5R6T3log||(log)2log2(1+2ϵ)τ¯4. = 128(1+ε)^3L^5R^6T^3 |A|( )^2 ^2(1+2ε) τ_4. This is the first bound in Theorem 1. Assume first that A>0A>0 and τ¯4>0 τ_4>0. Lemma 10 gives η⋆ η =minABN,(ACN)1/5, = \ ABN, ( ACN )^1/5 \, 1N∑n=0N−1‖∇J(θn)‖22 1N _n=0^N-1E \|∇ J( _n) \|_2^2 ≤2ABN+2(A4CN4)1/5. ≤ 2 ABN+2 ( A^4CN^4 )^1/5. Substituting C and suppressing the fixed problem constants gives η⋆≍min1(1+ϵ)N,(log2(1+2ϵ)(1+ϵ)3τ¯4N)1/5,η \ 1 (1+ε)N, ( ^2(1+2ε)(1+ε)^3 τ_4N )^1/5 \, and the preceding optimized bound gives the second claim of Theorem 1. When τ¯4=0 τ_4=0 and A>0A>0, the C=0C=0 branch of Lemma 10 instead gives η⋆=A/(BN)η = A/(BN) and the first term of the optimized bound. If A=0A=0, choose any positive step size of the main-text order and substitute it directly into eq. 44; the optimization term is then zero and the same stated upper order follows. ∎ Proof of Corollary 1. Suppressing fixed positive problem-dependent coefficients, the leading ϵε-dependent terms in the optimized GMC bound have the form ℬN(ϵ)=aN−1/21+ϵ+bτ¯41/5N−4/5(1+ϵ)3/5log−2/5(1+2ϵ),B_N(ε)=aN^-1/2 1+ε+b τ_4^1/5N^-4/5(1+ε)^3/5 ^-2/5(1+2ε), for constants a,b>0a,b>0 independent of N, τ¯4 τ_4, and ϵε. The first term increases with ϵε, while the second diverges as ϵ↓0ε 0 and captures the correction-bias penalty. For each N, ℬNB_N is continuous on (0,∞)(0,∞) and diverges at both endpoints, so it attains a global minimum in (0,∞)(0,∞). Let h(ϵ):=(1+ϵ)3/5log−2/5(1+2ϵ)h(ε):=(1+ε)^3/5 ^-2/5(1+2ε). Since log(1+2ϵ)∼2ϵ (1+2ε) 2ε as ϵ↓0ε 0, −h′(ϵ) -h (ε) =h(ϵ)5(4(1+2ϵ)log(1+2ϵ)−31+ϵ)≍ϵ−7/5(ϵ↓0). = h(ε)5 ( 4(1+2ε) (1+2ε)- 31+ε ) ε^-7/5 (ε 0). With τ¯4>0 τ_4>0 fixed, every minimizer tends to zero: at ϵ=N−1/5ε=N^-1/5, ℬN(ϵ)=aN−1/2+o(N−1/2)B_N(ε)=aN^-1/2+o(N^-1/2), whereas for any fixed ϵ0>0ε_0>0 and ϵ≥ϵ0ε≥ε_0, ℬN(ϵ)≥a1+ϵ0N−1/2B_N(ε)≥ a 1+ε_0\,N^-1/2. At any such minimizer, the first-order condition and dϵ1+ϵ≍1 ddε 1+ε 1 then give N−1/2≍τ¯41/5N−4/5(ϵGMC⋆)−7/5,N^-1/2 τ_4^1/5N^-4/5(ε_GMC )^-7/5, or, equivalently, (ϵGMC⋆)7/5≍τ¯41/5N−3/10.(ε_GMC )^7/5 τ_4^1/5N^-3/10. Taking both sides to the power 5/75/7 yields ϵGMC⋆=Θ(τ¯41/7N−3/14).ε_GMC = ( τ_4^1/7N^-3/14 ). ∎ Proof of Corollary 2. Corollary 3 gives (χn2,(b))2≤4(1+δ)2(Dn(b))2( _n^2,(b))^2≤ 4(1+δ)^2(D_n^(b))^2. Therefore the chi-square branch in eq. 31 and eq. 30 imply T2R2Llog2G2ϵ21N∑n=0N−1[1B∑b=1B(χn2,(b))2] T^2R^2L 2G^2ε^2 1N _n=0^N-1E [ 1B _b=1^B( _n^2,(b))^2 ] ≤2(1+δ)2T2R2LlogG2ϵ21N∑n=0N−1[1B∑b=1B(Dn(b))2] ≤ 2(1+δ)^2T^2R^2L G^2ε^2 1N _n=0^N-1E [ 1B _b=1^B(D_n^(b))^2 ] ≤32(1+δ)2(1+ϵ)3L5R6T3log||(log)2G2ϵ2η4τ¯4. ≤ 32(1+δ)^2(1+ε)^3L^5R^6T^3 |A|( )^2G^2ε^2η^4 τ_4. Substitution in eq. 32 proves the first displayed bound in the corollary. Lemma 10, with C equal to the complete coefficient of η4η^4 on the last line, gives the stated step size and optimized rate. ∎ Proof of Theorem 2. Retain the quadratic-KL branch in eq. 43. With M=NM=N, the result has the form eq. 44 with the same A and B as in the proof of Theorem 1, and C=128(1+ϵ)3L5R6T3log||(log)2cϵ2τ¯4.C= 128(1+ε)^3L^5R^6T^3 |A|( )^2c_ε^2 τ_4. This is the first bound in Theorem 2. Assume first that A>0A>0 and τ¯4>0 τ_4>0. Lemma 10 gives η⋆ η =minABN,(ACN)1/5, = \ ABN, ( ACN )^1/5 \, 1N∑n=0N−1‖∇J(θn)‖22 1N _n=0^N-1E \|∇ J( _n) \|_2^2 ≤2ABN+2(A4CN4)1/5. ≤ 2 ABN+2 ( A^4CN^4 )^1/5. Treating the objective gap and T,L,R,||,T,L,R,|A|,C as fixed constants reduces the first line to η⋆≍min1(1+ϵ)N,(cϵ2(1+ϵ)3τ¯4N)1/5,η \ 1 (1+ε)N, ( c_ε^2(1+ε)^3 τ_4N )^1/5 \, and the second line is exactly the optimized order stated in the theorem. When τ¯4=0 τ_4=0 and A>0A>0, the C=0C=0 branch of Lemma 10 instead gives η⋆=A/(BN)η = A/(BN) and the first term of the optimized bound. If A=0A=0, choose any positive step size of the main-text order and substitute it directly into eq. 44; the optimization term is then zero and the same stated upper order follows. ∎