Paper deep dive
Stabilizing Reinforcement Learning for Diffusion Language Models
Jianyuan Zhong, Kaibo Wang, Ding Ding, Zijin Feng, Haoli Bai, Yang Xiang, Jiacheng Sun, Qiang Xu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/13/2026, 12:15:46 AM
Summary
StableDRL is a reinforcement learning framework designed to stabilize the training of Diffusion Large Language Models (dLLMs) by addressing the reward collapse caused by noisy importance ratio estimation. It introduces unconditional clipping and self-normalization to break the self-reinforcing instability loop inherent in standard Group Relative Policy Optimization (GRPO) when applied to dLLMs.
Entities (5)
Relation Signals (4)
StableDRL → uses → unconditional clipping
confidence 100% · StableDRL... uses (i) unconditional clipping to suppress outlier-induced spikes
StableDRL → uses → self-normalization
confidence 100% · StableDRL... (ii) self-normalization to constrain updates within the convex hull
StableDRL → reformulates → GRPO
confidence 95% · we propose StableDRL, a reformulation of GRPO tailored for dLLMs
GRPO → causes → reward collapse
confidence 90% · its direct application to diffusion large language models (dLLMs) often triggers reward collapse
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Group Relative Policy Optimization (GRPO) is highly effective for post-training autoregressive (AR) language models, yet its direct application to diffusion large language models (dLLMs) often triggers reward collapse. We identify two sources of incompatibility. First, GRPO relies on importance ratios defined by sequence probabilities, which are intractable in dLLMs and must be estimated (e.g., via ELBO-based or mean-field likelihood proxies), yielding inherently noisy ratios. Second, standard GRPO's formulation is not designed for estimated ratios: its conditional clipping can be anomalously bypassed by model-agnostic estimation noise, producing gradient spikes, while its fixed group-size normalization amplifies gradient-magnitude fluctuations under high-variance ratio estimates. We show these effects form a self-reinforcing instability loop that drives policy drift and further increases ratio variance. To break this loop, we propose StableDRL, a reformulation of GRPO tailored for dLLMs that uses (i) unconditional clipping to suppress outlier-induced spikes and (ii) self-normalization to constrain updates within the convex hull of per-sample gradients. We further extend StableDRL to block-wise diffusion models via a staircase attention mechanism.
Tags
Links
- Source: https://arxiv.org/abs/2603.06743v1
- Canonical: https://arxiv.org/abs/2603.06743v1
Trouble viewing inline? Open PDF directly →
Full Text
93,680 characters extracted from source content.
Expand or collapse full text
1]Huawei Foundation Model Department 2]The Chinese University of Hong Kong 3]The Hong Kong University of Science and Technology ://github.com/JianyuanZhong/StableDRL Stabilizing Reinforcement Learning for Diffusion Language Models Jianyuan Zhong Kaibo Wang Ding Ding Zijin Feng Haoli Bai Yang Xiang Jiacheng Sun Qiang Xu [ [ [ Abstract Group Relative Policy Optimization (GRPO) is highly effective for post-training autoregressive (AR) language models, yet its direct application to diffusion large language models (dLLMs) often triggers reward collapse. We identify two sources of incompatibility. First, GRPO relies on importance ratios defined by sequence probabilities, which are intractable in dLLMs and must be estimated (e.g., via ELBO-based or mean-field likelihood proxies), yielding inherently noisy ratios. Second, standard GRPO’s formulation is not designed for estimated ratios: its conditional clipping can be anomalously bypassed by model-agnostic estimation noise, producing gradient spikes, while its fixed group-size normalization amplifies gradient-magnitude fluctuations under high-variance ratio estimates. We show these effects form a self-reinforcing instability loop that drives policy drift and further increases ratio variance. To break this loop, we propose StableDRL, a reformulation of GRPO tailored for dLLMs that uses (i) unconditional clipping to suppress outlier-induced spikes and (i) self-normalization to constrain updates within the convex hull of per-sample gradients. We further extend StableDRL to block-wise diffusion models via a staircase attention mechanism. 22footnotetext: Co-first authors.33footnotetext: Corresponding authors. Figure 1: StableDRL is the first method to enable stable full-parameter RL training on both full-attention and block dLLMs, better unlocking reasoning capability for dLLMs. The left panel reports performance on full-attention dLLMs (LLaDA-8B nie2025llada). Based on Table 1, Best Prior corresponds to WD1 tang2025wd1, and Best SOTA corresponds to the best performance between ESPO and SPG ou2025espo; wang2025spg for each task. The right panel demonstrates results for block diffusion models (SDAR-8B cheng2025sdar). 1 Introduction Discrete Diffusion Large Language Models (dLLMs) have emerged as a compelling alternative to autoregressive (AR) models, intrinsically supporting parallel decoding and bidirectional context modeling (sahoo2024simple; nie2025llada; wu2025fast; yang2025mmada). While Group Relative Policy Optimization (GRPO) has proven highly effective for reinforcement learning (RL) in the AR paradigm, its direct application to dLLMs leads to severe instability. As shown in Figure 2(a), full-parameter GRPO training on dLLMs exhibits an abrupt reward collapse at ∼300 300 steps. The incompatibility between GRPO and dLLMs stems from two factors: (i) the intractability of importance ratios in dLLMs ou2025absorbingdiscretediffusionsecretly; ou2025espo and (i) GRPO’s lack of adaptation to estimated importance ratios (Section 3.1). GRPO updates a target policy using data sampled from a behavior policy based on the importance ratios, defined as the ratio of their sequence probabilities. While this probability is tractable for AR models, it is intractable for dLLMs and often computed via estimations. Prior research has focused on the dLLM aspect, refining importance ratio estimation using mean-field approximations (zhao2025d1; tang2025wd1) or Evidence Lower Bound (ELBO) estimations (yang2025mmada; wang2025spg; ou2025espo). Although these approaches yield performance gains, they empirically remain prone to training instability. We attribute the instability in dLLMs to two design flaws in standard GRPO, which is inherently sensitive to the noisy importance ratios. First, the clipping mechanism in GRPO is conditional. In AR models, this mechanism accelerates the policy’s return to the trust region. In dLLMs, however, model-agnostic estimation noise allows the clipping condition to be anomalously bypassed, triggering gradient spikes. Second, GRPO normalizes updates by the fixed group size. Given the high variance of importance ratio estimation in dLLMs, this static normalization results in drastic fluctuations in gradient magnitude, destabilizing the optimization process. To address the instability, we first analyze the underlying mechanism and then propose a stable GRPO variant tailored for dLLMs. We theoretically and empirically demonstrate how these flaws precipitate a self-reinforcing instability loop, leading to the reward collapse. As shown in Figure 2(b), noisy importance ratios first induce gradient spikes under the GRPO update (Link 1). These spikes degrade the target policy, causing it to deviate significantly from the behavior policy (Link 2). This deviation, in turn, exacerbates the variance of importance ratios in subsequent steps (Link 3). We have proven that once the gradient norm exceeds a critical threshold, the probability of continued divergence increases, driving the policy toward irreversible reward collapse. To stabilize training, we propose StableDRL to break the instability loop at its source (Link 1). As illustrated in Figure 2(c), StableDRL incorporates two components. (i) We introduce unconditional clipping, which enforces strict bounds on importance ratios regardless of the advantage. This prevents outliers from generating gradient spikes. (i) We employ self-normalization. Instead of dividing by the group size, we normalize the update by the sum of clipped importance ratios. This constrains the update within the convex hull of per-sample gradients. Furthermore, we extend StableDRL to block diffusion models (cheng2025sdar) via a staircase attention mechanism, enabling leakage-free probability estimation. To the best of our knowledge, StableDRL is the first method to enable stable, full-parameter RL training on both full-attention and block dLLMs for over 1,000 steps. This sustained stability effectively increases the volume of valid training rollout samples, allowing the model to fully unlock its reasoning capabilities and empirically achieve state-of-the-art performance in dLLM reasoning tasks. Our contributions are summarized as follows: • We theoretically and empirically identify the self-reinforcing instability loop that causes reward collapse when GRPO is applied to dLLMs. • We propose StableDRL, a novel reinforcement learning framework for stabilizing the full-parameter training of dLLMs through unconditional clipping and self-normalization. • Comprehensive experiments validate the effectiveness of our StableDRL on both full-attention and block dLLMs, showing higher training stability and significant accuracy gain over prior best-in-class methods. 2 Background Figure 2: (a) Training instability. Naive integration of noisy importance ratios into GRPO leads to severe instability under full-parameter RL training with dLLMs. Notably, reward collapse occurs even with Policy Gradient, where the importance ratio is fixed at 1. (b) Instability loop. Estimation noise triggers gradient spikes and policy drift, creating a self-reinforcing cycle that amplifies the variance of future importance ratios. (c) StableDRL. To address this, we propose a reformulated GRPO for noisy importance ratios. By employing unconditional clipping and self-normalization, StableDRL effectively breaks the instability loop. 2.1 Masked Diffusion Language Models Masked diffusion language models (MDLMs) (nie2025llada; sahoo2024simpleeffectivemaskeddiffusion) formulate text generation as a discrete diffusion process modeled by a continuous-time Markov chain. Given a clean sequence x0∈nx_0 ^n, the forward process q(xt|x0)q(x_t|x_0) independently corrupts tokens by transitioning them to a special mask token M according to a schedule t∈[0,1]t∈[0,1]. The generative process reverses this corruption by learning a denoiser πθ(x0|xt) _θ(x_0|x_t) to reconstruct the original data from the latent state. Unlike AR models where the exact sequence log-likelihood logπθ(x0) _θ(x_0) is computationally tractable via the chain rule, that of MDLMs is intractable, as it requires marginalizing over all n!n! masking trajectories ou2025absorbingdiscretediffusionsecretly. Consequently, training optimizes the Evidence Lower Bound (ELBO) (wu2025fast; DBLP:conf/iclr/OuNXZSLL25), denoted as ℒθ(x0)L_θ(x_0) with ℒθ(x0)≤logπθ(x0)L_θ(x_0)≤ _θ(x_0): ℒθ(x0)=t,xt[1t∑i=1n(xti=M)logπθ(x0i|xt)].L_θ(x_0)=E_t,x_t [ 1t _i=1^n 1(x^i_t= M) _θ(x^i_0|x_t) ]. (1) In practice, this expectation is approximated via Monte Carlo (MC) sampling. We denote the single-sample MC estimator of the ELBO as ℒ^θ(x0) L_θ(x_0). 2.2 Reinforcement Learning with dLLMs We focus on fine-tuning dLLMs to maximize a reward function R(x)R(x) using policy gradient methods. The standard objective is to maximize the expected return (θ)=x∼πθ[R(x)]J(θ)=E_x _θ[R(x)]. Modern on-policy algorithms, such as PPO (schulman2017ppo) and GRPO (shao2024deepseekmath), improve sample efficiency by utilizing importance sampling to update the policy using trajectories collected from a behavior policy πθold _ _old. Group Relative Policy Optimization (GRPO). GRPO eliminates the value function critic by estimating the baseline using the group average of rewards. For a group of rollouts x1,…,xG\x_1,…,x_G\ sampled from πθold _ _old conditioned on prompt c, the gradient update formula is: ∇θGRPO=[1G∑j=1Gmin(ρjAj,clipϵ(ρj)Aj)gj], _θJ_GRPO=E [ 1G _j=1^G ( _jA_j,clip_ε( _j)A_j )g_j ], (2) where AjA_j is the advantage standardized within the group, ρj(x)=πθ(x)πθold(x) _j(x)= _θ(x) _ _old(x) is the importance ratio, and clip(⋅)clip(·) ensure a trused region of [1−ϵ,1+ϵ][1-ε,1+ε]. For simplicity, We omit dependencies on x,x∼πoldx,x _old and denote the gradient ∇θlogπθ(x) _θ _θ(x) by g. GRPO retains the min-clip operation from PPO, implicitly regularizing the divergence between the target policy and the behavior policy. When ρ tends to move away from the trust region (e.g., ρ>1+ϵ,A>0ρ>1+ε,A>0), the clip operation limits the update step size. When it tends to return to the trust region (ρ>1+ϵ,A<0ρ>1+ε,A<0), employing unclipped step sizes to accelerate training. Challenges in adapting GRPO to dLLMs. Prior works primarily improve importance ratio estimation methods and directly porting them to Eq. (2). While earlier methods like D1 and WD1 (zhao2025d1; tang2025wd1) attempted a one-step mean-field approximation, zhao2025diffpo showed this to be inaccurate. As a result, current state-of-the-art approaches (yang2025mmada; wang2025spg; ou2025espo) use multi-step Monte Carlo sampling to estimate likelihood via the Evidence Lower Bound (ELBO). However, in practical on-policy RL with limited MC steps (m≤5m≤ 5) ou2025espo; wang2025spg, the estimation suffers from noise and outliers. In Sec. 3.1, we show that the combination of this estimation noise and the standard GRPO formulation causes training instability. 3 Methodology In this section, we first diagnose the root cause of the observed reward collapse in GRPO, identifying an instability loop driven by the long-tail noise of importance ratios (Sec. 3.1). We then propose StableDRL, which mitigates this instability through unconditional clipping and self-normalization (Sec. 3.2). Finally, we provide a theoretical justification for our method (Sec. 3.3). 3.1 Understanding Instability in dLLM RL Training As current state-of-the-arts utilize Monte Carlo sampling to estimate the intractable importance ratios of dLLMs, we model the training instability as a three-stage process: (i) noise in estimated importance ratios forms a long-tail distribution; (i) high variance and outliers generate gradient spikes; and (i) these spikes induce policy drift, which amplifies future variances of the estimated importance ratios, closing the instability loop. (i) Variance in importance ratios. Let η(x)η(x) denote noise from the estimation error of the ELBO, such that ℒ^θ(x)=ℒθ(x)+η(x) L_θ(x)=L_θ(x)+η(x). The estimated ρ^(x) ρ(x) can be decomposed into a policy drift term and a noise term: ρ^(x)=expℒ^θ(x)expℒ^θold(x)=exp(Δℒ(x))⏟Policy Drift⋅exp(Δη(x))⏟Noise, ρ(x)= L_θ(x) L_ _old(x)= ( (x))_Policy Drift· ( η(x))_Noise, (3) where Δℒ(x)=ℒθ(x)−ℒθold(x) (x)=L_θ(x)-L_ _old(x) represents the true divergence between the target and behavior policies, and Δη(x)=ηθ(x)−ηθold(x) η(x)= _θ(x)- _ _old(x) stands for the net difference in estimation error between the two policy evaluations. The exp(⋅) (·) operator maps the symmetric noise Δη(x) η(x) to a long-tailed distribution with a non-negligible probability of yielding extreme values. For instance, the estimated ratio of a single rollout can explode to magnitudes of 10510^5 (Fig. 8). Consequently, for a group of rollouts x1,…,xG\x_1,…,x_G\, the resulting set of importance ratios ρ^(x1),…,ρ^(xG)\ ρ(x_1),…, ρ(x_G)\ exhibits extremely high variance. (i) Gradient spikes. We observe that the noise in ρ^(x) ρ(x) leads to gradient spikes through two mechanisms: individual anomalies and group anomalies. Individual Anomalies. In algorithms like GRPO, clipping is conditional. Specifically, when the advantage is negative (A<0A<0) and the ratio deviates significantly (ρ^>1+ϵ ρ>1+ε), the objective function simplifies to the unclipped term ρ^A ρA. This design allows the model to take large steps when returning to the trust region. However, in dLLMs, large ρ ρ values can be driven by the model-agnostic noise Δη(x) η(x) rather than true policy alignment. Consequently, whenever A<0A<0, there is a probability that a noise-induced outlier results in a massive, unclipped gradient. Group Anomalies. Due to the high variance of the estimator, importance ratios within a group ρ^jj=1G\ ρ_j\_j=1^G can be simultaneously large or small. Even if individual ratios are capped, the collective fluctuation of the sum ∑ρ^jΣ ρ_j causes the gradient magnitude to oscillate. In Sec. 3.3 and Sec. 4.1, we theoretically and empirically show that these frequent spikes, even bounded, can destabilize the training dynamics. (i) Policy drifts. When the target policy πθ _θ undergoes an update driven by a gradient spike, its behavior shifts abruptly, causing the policy divergence Δℒ(x) (x) to increase significantly. As shown in Eq. (3), a larger Δℒ(x) (x) acts as a multiplier, amplifying the variance of the importance ratios ρ^jj=1G\ ρ_j\_j=1^G in subsequent steps. This establishes an instability loop: estimation noise generates gradient spikes, which induce policy drift; this drift, in turn, exacerbates the variance of future importance ratios. This self-reinforcing loop destabilizes training and leads to reward collapse. 3.2 StableDRL To stabilize training, we propose StableDRL. Our method breaks the instability loop by preventing importance ratio noise from translating into gradient spikes. It consists of two components: unconditional clipping and self-normalization. Unconditional clipping. We replace the conditional clipping of GRPO with a strict, unconditional constraint. We enforce that the importance ratio ρ ρ is always bounded within [1−ϵ,1+ϵ][1-ε,1+ε], regardless of the sign of the advantage. Theoretically, this ensures the gradient is strictly bounded, avoiding the influence of extreme outliers. Self-normalization. While unconditional clipping mitigates individual outliers, the gradient can still oscillate violently between the lower and upper bounds due to group-level variance. As we show in Sec. 3.3, with unconditional clipping alone, the gradient frequently approaches the preset upper bound. This creates a trade-off where a loose bound leads to instability, while a tight bound conceals the true importance signal and slows learning. To address group-level anomalies, we replace the fixed group size normalizer G with the sum of clipped ratios, ∑i=1Gclipϵ(ρ^i) _i=1^Gclip_ε( ρ_i). By rescaling, we confine the update to the convex hull of the per-sample gradients, rather than allowing the magnitude to oscillate between preset bounds. The gradient update for StableDRL is formulated as: ∇θOurs=[1∑i=1Gclipϵ(ρ^i)∑j=1Gclipϵ(ρ^j)Ajgj]. _θJ_Ours=E [ 1 _i=1^Gclip_ε( ρ_i) _j=1^Gclip_ε( ρ_j)A_jg_j ]. (4) StableDRL is simple yet effective to suppresses gradient spikes, preventing training from entering the instability loop. 3.3 Theoretical Analysis We explain why GRPO becomes unstable when importance ratios are computed from noisy likelihood proxies in dLLMs. Our analysis models a self-reinforcing loop between estimation noise, gradient spikes, and policy drift. We first show that, under GRPO’s asymmetric unclipping on negative-advantage samples, the update norm has a nonzero probability of exceeding any threshold H. We then show a feedback mechanism: once a spike-induced step increases a drift state, the derived lower bound on the spike probability is nondecreasing for later inner steps on the same rollout group. Notations. Fix a rollout group ℬ=x1,…,xGB=\x_1,…,x_G\ sampled from πθold _ _old, and consider inner updates θ0=θold,θ1,θ2,… _0= _old, _1, _2,… on this fixed group. Following Sec. 3.1, let ρ^i,j=exp(Δℒi,j+Δηi,j) ρ_i,j= ( _i,j+ _i,j) be the estimated importance ratio at inner step i, where Δℒi,j _i,j is the noise-free drift and Δηi,j _i,j is the log-ratio estimation noise. Let gi,jg_i,j denote the advantage-weighted proxy gradient used in the update. We use a single constant B such that ‖gi,j‖≤B\|g_i,j\|≤ B for all inner steps i and samples j, which is standard in practice. Define the negative-advantage set =j:Aj≤−a0N=\j:A_j≤-a_0\ for some a0>0a_0>0, and the drift state Di:=maxj∈Δℒi,jD_i:= _j _i,j. Uniform tail envelope. To address the non-stationarity of noise across steps, we assume the right tails of Δηi,j _i,j admit a common lower bound (App. B.2). This assumption allows us to lower-bound the spike probability using a time-invariant function of the drift state DiD_i, ensuring that the instability risk is strictly defined by the magnitude of the drift. Why GRPO can spike. Standard GRPO allows unclipped multipliers on negative-advantage samples when Aj<0A_j<0 and ρ^i,j>1+ϵ ρ_i,j>1+ε. We first establish that gradient spikes are statistically inevitable and tightly coupled to the drift state. Lemma 3.1 (Informal, existence of drift-dependent spike probability). In inner step i of GRPO, for any threshold H>0H>0, there exists a lower bound Pi(H)∈(0,1)P_i(H)∈(0,1) such that Pr(‖∇θGRPO‖≥H|Di)≥Pi(H). \! ( \| _θJ_GRPO \|≥ H\, |\,D_i )\ ≥\ P_i(H). (5) Moreover, under the common tail-envelope condition on Δηi,j _i,j, the bound Pi(H)P_i(H) can be chosen as the same nondecreasing function of DiD_i for all inner steps (see App. B.2). As the policy drifts (Di↑D_i ), the noise margin required to push an importance ratio above any fixed level shrinks. Consequently, large-multiplier outliers become increasingly probable. Theorem 3.2 (Informal, self-reinforcing instability loop). Consider an inner step i where a gradient spike occurs (‖∇θGRPO‖≥H\| _θJ_GRPO\|≥ H) and the spike is driven by a single negative-advantage outlier that dominates the group update (sufficient conditions are in App. B.2). Then the resulting update increases the drift state: Di+1≥Di.D_i+1≥ D_i. (6) Consequently, since the bound Pi(H)P_i(H) in Lemma 3.1 is defined via a common tail envelope and is nondecreasing in DiD_i, the next-step spike lower bound cannot decrease on that realized step: Pi+1(H)≥Pi(H).P_i+1(H)≥ P_i(H). (7) Clipping alone can saturate. Unconditional two-sided clipping prevents unbounded spikes, but may enter a high-frequency “boundary-hitting” regime. Lemma 3.3 (Informal, existence of hitting probability). In unconditionally clipped GRPO, the update norm is deterministically bounded by Hmax=(1+ϵ)B.H_ =(1+ε)B. (8) For any threshold H close to HmaxH_ , there exists a saturation probability Qi(H)Q_i(H) that the update hits the upper boundary. Under the same common tail-envelope condition, Qi(H)Q_i(H) can be chosen to be nondecreasing in DiD_i (see App. B.3). Theorem 3.4 (Informal, self-reinforcing hitting loop). If at inner step i the update saturates near the upper boundary (‖∇θUC-GRPO‖≥H\| _θJ_UC -GRPO\|≥ H) and the saturated step induces non-decreasing drift (Di+1≥DiD_i+1≥ D_i under the appendix conditions), then, by monotonicity of Qi(H)Q_i(H) in Lemma 3.3, Qi+1(H)≥Qi(H).Q_i+1(H)≥ Q_i(H). (9) Thus, clipping alone can trade rare, unbounded spikes for frequent boundary-saturated updates. This creates a trade-off: a loose upper bound still destabilizes optimization, while an overly tight bound can obscure the importance-weight signal. Why StableDRL breaks the loop. Finally, we show how StableDRL structurally removes the remaining group-scale randomness. Theorem 3.5 (StableDRL). Let wi,j:=clipϵ(ρ^i,j)w_i,j:=clip_ε( ρ_i,j) be the clipped weights. The StableDRL update ∇θOurs _θJ_Ours is normalized by the sum of weights. Since wi,j>0w_i,j>0, the update always lies in the convex hull of the per-sample directions gi,j\g_i,j\: ‖∇θOurs‖=‖∑j=1Gwi,jgi,j∑j=1Gwi,j‖≤maxj‖gi,j‖≤B. \| _θJ_Ours \|= \| _j=1^Gw_i,j\,g_i,j _j=1^Gw_i,j \|≤ _j\|g_i,j\|≤ B. (10) Unlike clipping alone, self-normalization explicitly divides out the random group-scale factor 1G∑jwi,j 1G _jw_i,j, decoupling the update magnitude from group-level weight fluctuations. This breaks the instability mechanisms formalized in App. B.2 and App. B.3. In Sec. 4.1, we empirically validate these explanations. 3.4 Generalization to Block Diffusion Adapting block diffusion (wu2025fast; cheng2025sdar) to RL creates a dilemma between training efficiency and information leakage. Valid likelihood proxy estimation requires conditioning each block strictly on its clean history. Naive iterative implementations are prohibitively slow ((K)O(K)), while standard parallel attention invalidates gradient signals by allowing tokens to "cheat" and attend to their own ground truth. To resolve this, we introduce staircase attention, a structured masking primitive that enables leakage-free, single-pass evaluation. By utilizing a dual-stream input of frozen clean context and corrupted target, the mask enforces strict conditional independence through a unique geometry (Figure 3). A block-lower-triangular "staircase" grants target tokens in block k access to the clean history of preceding blocks (1…k−11… k-1) while mechanically occluding the current block’s ground truth. Simultaneously, a block-diagonal component permits parallel, independent denoising within the target stream. This structure satisfies ELBO requirements within a single computational graph ((1)O(1)), rendering full-parameter RL feasible for long-horizon tasks (formal derivation in Appendix A). Figure 3: Staircase Attention for Efficient Proxy Estimation. To evaluate the ELBO for block diffusion in a single pass (O(1)O(1)), we use a dual-stream construction. The Clean Context (top rows) provides immutable history. The Corrupted Target stream (bottom rows) uses a “staircase” mask (MstairM_ stair, bottom-left) to attend to valid history without peeking at the ground truth of the current block. The target self-attention (MintraM_ intra, bottom-right) is block-diagonal, ensuring independent parallel denoising. 3.5 Pratical Implementations Score-function surrogates. Since dLLMs lack a tractable ∇θlogπθ _θ _θ, StableDRL reweights gradient directions provided by stable score surrogates, current state-of-the-art dLLM RL methods SPG (yang2025mmada; ou2025espo; wang2025spg). For full-attention dLLMs, we implement block-wise masking (wang2025spg) by sampling structured mask blocks consistent with the inference denoising schedule. For block diffusion, we sample random positions to mask since the staircase attention runs in (1)O(1). Numerically stable log-space weights. Direct computation of Eq. (4) is numerically unstable due to the exponentiation of noisy log-ratios. We strictly compute weights in log-space. We define the clipped log-ratios ℓ~j=clip(ℒ^θ(xj)−ℒ^θold(xj),log(1−ϵ),log(1+ϵ)) _j=clip( L_θ(x_j)- L_ _old(x_j), (1-ε), (1+ε)) and compute the normalized coefficients via a stable softmax exp(ℓ~j−LSE(ℓ~kk=1G)) \! ( _j-LSE(\ _k\_k=1^G) ), where LSE(⋅)LSE(·) is the Log-Sum-Exp function. This clip-then-softmax approach preserves numerical precision even when raw probability ratios would underflow or overflow, ensuring stable optimization in mixed-precision training. 4 Experiments We evaluate StableDRL on two diffusion diffusion architectures: full-attention masked diffusion (LLaDA-8B-Instruct) and semi-autoregressive block diffusion (SDAR-8B). Specifically, we (i) empirically verify the theoretical instability mechanisms analyzed in Sec. 3.3, (i) show state-of-the-art reasoning performance on standard benchmarks, and (i) confirm the architectural generality of our framework. Ablation studies are also conduct to dissect the effects of unconditional clipping and self-normalization to training stability. Experimental setup. For both LLaDA-8B-Instruct and SDAR-8B, We perform RL fine-tuning using the AdamW optimizer with a learning rate of 1.0×10−61.0× 10^-6. To ensure optimization stability while maintaining sample efficiency, we set the unconditional importance weight clipping threshold to ϵ=5ε=5 by default. For a comprehensive description of the training infrastructure, model configurations, and hyperparameters, please refer to Appendix C.1. 4.1 Empirical Verification of Instability Mechanisms Figure 4: Verification of Instability Mechanisms across Methods. Left Column (GRPO): Unbounded drift (bottom) fuels an accelerating spike rate (middle), causing reward collapse (top). Middle Column (Unconditional Clipping): Clipping saturates the drift (bottom) but induces a high-frequency, stochastic spike regime (middle) that destabilizes learning (top). Right Column (StableDRL): Our method maintains a low, stable spike rate (middle) decoupled from drift (bottom), resulting in monotonic reward improvement (top). Experimental setup. To bridge the gap between our theoretical analysis of self-reinforcing instability in Sec 3.3 and observed training dynamics, we perform a fine-grained analysis of gradient norm evolution. We introduce the Relative Gradient Spike Rate to quantify instability. A step t is classified as a “spike” if the gradient norm exceeds the local moving average by a margin δ: spike(t)=[‖gt‖>(1+δ)⋅1W∑k=1W‖gt−k‖],I_spike^(t)= 1 [\|g_t\|>(1+δ)· 1W _k=1^W\|g_t-k\| ], where we set window W=50W=50 and δ=0.3δ=0.3. We compare the evolution of reward, spikes rate and the next spike threshold, (1+δ)⋅1W∑k=1W‖gt−k‖(1+δ)· 1W _k=1^W\|g_t-k\|, for GRPO, unconditional clipping, and StableDRL in Figure 4. We take ESPO ou2025espo as a representative for GRPO and follow its original training setting. We implement unconditional clipping within our training framework with the same clipping threshold ϵε and settings. For all three methods111For this Experiment, we train GRPO on GSM8K. For Unconditional Clipping and StableDRL, we train on CountDown., we perform full RL finetuning. As illustrated in Figure 4, GRPO exhibits divergent behavior in which unbounded importance ratios drive a steadily rise spike threshold, pushing the optimization into a high-variance regime that requires increasingly large gradients to destabilizing the reward signal. While unconditional clipping bounds the threshold, it results in a high saturation rate where gradient norms frequently impact the clipping limit. This intermittent saturation introduces oscillatory dynamics that corrupt AdamW’s momentum history, resulting in reward collapse. Conversely, StableDRL employs a structural convex-hull constraint that maintain a low and stable spike threshold, suppressing relative spikes and enables smooth, monotonic reward gain. 4.2 Mian Results 4.2.1 Full-Attention Diffusion Results Table 1: State-of-the-Art Reasoning Performance on LLaDA-8B-Instruct. We report pass@1 accuracy under three decoding budgets (N∈128,256,512N∈\128,256,512\) and the mean performance (Avg) for each dataset. Bold denote the best result and underline the second best. StableDRL achieves the highest average accuracy on all four benchmarks, demonstrating superior consistency across generation lengths. GSM8K MATH500 Countdown Sudoku Model / Seq Len 128 256 512 Avg 128 256 512 Avg 128 256 512 Avg 128 256 512 Avg LLaDA-8B-Inst. 69.5 77.2 79.8 75.5 28.2 32.4 34.6 31.7 18.8 16.8 16.8 17.5 5.7 27.7 26.2 19.9 LLaDA-1.5 70.4 80.5 81.9 77.6 26.8 32.2 35.8 31.6 31.9 21.1 21.5 24.8 7.4 26.9 29.0 21.1 D1 72.2 80.6 81.3 78.0 31.4 36.0 39.4 35.6 30.9 30.9 34.4 32.1 7.2 32.5 29.3 23.0 WD1 74.6 81.5 83.0 79.7 31.0 37.4 39.0 35.8 48.8 52.3 50.8 50.6 33.1 32.1 22.5 29.2 UniGRPO 74.9 82.5 82.7 80.0 32.4 37.4 39.4 36.4 44.5 43.0 57.0 48.2 59.0 67.0 62.9 63.0 ESPO 80.0 82.3 83.7 82.0 36.0 39.0 43.4 39.5 81.6 82.0 79.3 81.0 92.7 84.7 80.5 86.0 SPG w/ Mixture 78.5 86.1 84.5 83.0 33.4 40.0 41.8 38.4 68.8 70.7 70.3 69.9 82.9 94.0 93.1 90.0 StableDRL (Ours) 80.2 86.2 86.3 84.2 36.2 45.2 44.0 41.8 81.3 84.4 84.8 83.5 91.9 92.4 90.1 91.5 Experimental setup. We follow the experimental protocol of ESPO (ou2025espo) and SPG (wang2025spg), which build on the D1 and WD1 setup zhao2025d1; tang2025wd1: benchmarks include GSM8K cobbe2021training, MATH500 lightman2023lets, Countdown pan2025tinyzero, and Sudoku arel2025sudoku, with the same train and test splits, and evaluation procedure. Concretely, we evaluate at generation lengths 128,256,512\128,256,512\ and use confidence-based semi-autoregressive decoding with block size 3232 for both RL rollouts and evaluation. We set ϵ=5ε=5 for the best performance. Baselines. We compare StableDRL against a representative suite of reinforcement learning algorithms for dLLMs. Baselines include D1 (zhao2025d1) and UniGRPO (yang2025mmada), which adapt theGRPO framework by approximating the intractable log-likelihood via one-step unmasking or MC estimation of the ELBO. We also include WD1 (tang2025wd1), which formulates a weighted policy optimization objective to avoid direct likelihood estimation. Finally, we benchmark against SPG (wang2025spg), which mitigates gradient bias by sandwiching the policy objective between a tractable Evidence Upper Bound (EUBO) for negative rewards and the ELBO for positive rewards. All methods are initialized from the LLaDA-8B-Instruct (nie2025large). Enabling stable full fine-tuning. Unlike ESPO or SPG, which mitigate instability via LoRA or early stopping, we fully fine-tune LLaDA-8B-Instruct by explicitly suppressing the gradient spikes that typically destabilize training. This allows StableDRL to optimize the entire model backbone, better unlocking the latent reasoning capabilities of the dLLM. Notably, while our RL training is conducted at a sequence length of 256 tokens, the resulting model achieves consistent, high performance across all evaluated generation lengths (128128 to 512512 tokens). This suggests that stable, full-parameter reinforcement learning fosters superior length generalization compared to parameter-efficient or variance-constrained alternatives. State-of-the-Art performances. Table 1 demonstrates that StableDRL establishes a new state-of-the-art by achieving the highest average accuracy across all decoding budgets. Specifically, in complex reasoning (MATH500), it secures an average accuracy of 41.8%, outperforming ESPO and SPG, with a notable +5.2% margin over SPG at the 256-token budget. In long-horizon planning (Countdown), StableDRL overcomes the off-policy drift that plagues SPG, delivering a massive +13.7% gain at 256 tokens to reach 84.4%. Furthermore, unlike baselines such as ESPO that fluctuate significantly on consistency tasks, StableDRL maintains robustness across all lengths, achieving top average scores on both GSM8K (84.2%) and Sudoku (91.5%). These results confirm that resolving “Noise-Drift" instability is critical for scaling RL in dLLMs. 4.2.2 Generalization to Block Diffusion Table 2: Block Diffusion Reasoning Performance. Pass@1 accuracy on MATH500, GSM8K, and AIME, comparing StableDRL (SDAR-8B backbone) against AR baselines (Qwen3) and prior Block Diffusion methods. StableDRL notably outperforms the strong Qwen3-8B AR model on the rigorous AIME benchmark. Method MATH500 GSM8K AIME 24 Autoregressive (AR) Baselines Qwen3-4B 74.1 90.7 12.9 Qwen3-8B 78.4 92.8 10.0 Block Diffusion (Dynamic Sampling) SDAR-8B (Base) 70.6 90.4 08.3 Trado 75.0 91.2 11.0 StableDRL (Ours) 77.8 92.1 13.3 Block Diffusion (Static Sampling) SDAR-8B (Base) 75.4 91.1 11.8 Trado 78.5 92.3 13.3 StableDRL (Ours) 79.2 92.4 16.7 To demonstrate the architectural generality of our framework, we instantiate StableDRL on the SDAR-8B-Chat block diffusion model (cheng2025sdar). We utilize Staircase Attention (Sec. 3.4) to enable scalable, leakage-free proxy estimation during training. Experimental setup. We follow TraceRL’s training and evaluation conventions for the SDAR-8B-Chat cheng2025sdar model (B=4B=4). Each RL sampling iteration generates 1616 trajectories per prompt using dynamic sampling (T=0.9T=0.9, temp 1.01.0). We train on the selected MATH training data lightman2023lets. Evaluation uses both (i) static (greedy block-wise) and (i) dynamic (temperature 1.01.0) sampling. Baselines. We benchmark against the supervised base model SDAR-8B, Trado (wang2025revolutionizing) For fairness, we adopt the model with just TraceRL training to compare, and the autoregressive Qwen3 yang2025qwen3technicalreport (4B and 8B Base) to contextualize performance against standard LLMs. We exclude DiRL (zhu2026dirlefficientposttrainingframework) from our comparison, as it utilizes a fundamentally different data regime and a complex two-stage training pipeline. Performance analysis. Table 2 reports the comparative results. StableDRL consistently outperforms prior block diffusion methods. Notably, on the rigorous AIME 2024 benchmark, StableDRL achieves 16.7% (Static), significantly surpassing the base model (11.8%), Trado (13.3%), and even the autoregressive Qwen3-8B (10.0%). This indicates that stable on-policy RL can unlock reasoning capabilities often dormant in supervised baselines. Furthermore, while Trado degrades significantly under dynamic sampling (dropping to 11.0% on AIME), StableDRL maintains superior robustness (13.3%), indicating it effectively shapes the full probability landscape rather than merely optimizing the mode.222To enable computationally feasible training, we employ custom JetEngine cheng2025sdar inference kernels. 4.3 Stress testing exploding importance ratios. Figure 5: Robustness to Proxy Noise: The “Exploding Weight” Stress Test (GSM8K). We compare training stability under standard conditions (“Normal”, solid lines) versus an adversarial regime where importance weight variance is artificially amplified (“Exploding”, dashed lines; see App. C.2). (Left) Reward Trajectories: StableDRL (Green) demonstrates invariant stability, maintaining monotonic improvement in both regimes. In contrast, ESPO (Orange) suffers immediate, noise-accelerated collapse, confirming its sensitivity to ratio outliers. SPG (Blue) degrades in both settings, indicating that avoiding ratios (to reduce variance) fatally exposes the model to off-policy bias. (Right) Gradient Norm Density: Visualizing the failure mechanism. StableDRL maintains a condensed, low-variance gradient distribution. Conversely, ESPO exhibits a heavy right tail of explosive updates (log-norm >3>3), confirming that the “Asymmetric Clipping Failure” allows noise spikes to propagate unchecked. A central hypothesis of this work is that dLLM training instability is driven by heavy-tailed importance weights (ρ ρ) derived from stochastic ELBO proxies. To isolate this factor, we design an adversarial Exploding Weight Stress Test (protocol details in Appendix C.2). This protocol synthesizes “Exploding” weights for a subset of trajectories by pairing “easy” masking patterns (high ELBO) with “hard” masking patterns (low ELBO), amplifying proxy variance without altering the ground-truth data or rewards. Figure 5 compares StableDRL, SPG, and ESPO under both “Normal” (unbiased) and “Exploding” conditions. StableDRL (Ours): Invariant Stability. StableDRL is robust: in the Normal setting (green solid), it achieves the highest final reward; under Exploding weights (green dashed), training remains stable and monotonic, with only a minor performance degradation. ESPO: Noise-Accelerated Collapse. ESPO is highly sensitive to proxy noise: in the Normal setting (orange solid), it collapses later in training, while under Exploding weights (orange dashed) collapse is immediate and catastrophic. This supports our diagnosis that GRPO-style conditional clipping is a primary failure mode under heavy-tailed proxy noise (Sec. 3.1). SPG: Bias-Induced Failure. SPG (blue) collapses in both settings. Since SPG reuses rollouts without importance-sampling correction (implicitly assuming ρ=1ρ=1), it avoids weight explosions but accumulates off-policy bias as the policy drifts, leading to degradation regardless of proxy noise level. 4.4 Ablation Studies Dissecting the stability mechanisms. To verify the contributions of Unconditional Clipping and Group Self-Normalization, we analyze the training dynamics on Countdown (Figure 7). We observe that removing unconditional clipping leads to rapid training failure, as noise-induced outliers dominate the convex combination of gradients. Conversely, removing self-normalization while retaining clipping causes the aggregated update magnitude to oscillate significantly between bounds due to estimation noise, distorting the AdamW momentum history and eventually leading to reward collapse. Only StableDRL, which combines magnitude bounding with geometric constraints, yields a stable and monotonic learning curve. Figure 6: Deconstructing Stability Mechanisms. We isolate the effect of Clipping and Self-Normalization on GSM8K. w/o Self-Norm (Blue): Retaining clipping prevents immediate explosion, but random group scale induces high-variance oscillation that distorts momentum. w/o Clipping (Red): We early-stopped this experiment once observing an unrecoverable collapse in training rewards. Self-normalization alone fails because single noise outliers dominate the convex combination (α→1α→ 1), effectively collapsing the sample size and causing rapid failure. StableDRL (Green): Combining both controls yields monotonic stability. Figure 7: Sensitivity to Trust Region Size (ϵε). We evaluate performance across varying clipping thresholds. Small Thresholds (ϵ∈1,5ε∈\1,5\): Training remains stable, with ϵ=5ε=5 (Blue) offering a superior bias-variance trade-off compared to the stricter ϵ=1ε=1 (Green). Large Thresholds (ϵ∈100,1000ε∈\100,1000\): As constraints loosen, the “Trapdoor” failure re-emerges. Higher thresholds (Purple, Red) allow noise spikes sufficient leverage to destabilize the policy, resulting in sudden, catastrophic collapse. Sensitivity to trust region tightness (ϵε). We further analyze the trade-off between stability and learning speed across varying clipping thresholds (Figure 7). In the small threshold regime (ϵ∈1,5ε∈\1,5\), training remains stable, with ϵ=5ε=5 providing a superior exploration-stability trade-off than the stricter ϵ=1ε=1, achieving faster convergence and higher final rewards by preserving valid learning signals. However, as constraints loosen significantly (ϵ∈100,1000ε∈\100,1000\), higher thresholds allow noise spikes sufficient leverage to destabilize the policy before clipping takes effect, resulting in sudden and catastrophic collapse. 5 Related Work RL post-training for LMs. Policy-gradient RL underpins modern alignment and post-training pipelines (williams1992reinforce; schulman2015trpo; schulman2017ppo). RLHF popularized preference-based alignment for AR LMs (ouyang2022instructgpt), while RL with verifiable rewards has shown strong gains for mathematical reasoning and long-form solutions (shao2024deepseekmath; deepseek2025r1). In the AR context, IR correction lingteam2025stepevolvesscalingreinforcement; zheng2025gspo, targets the staleness in behavior and target policy. RL for diffusion LMs. Other methods such as LLaDa 1.5 zhu2025llada proposes a off-policy RL algorithm, which often yields less performance gain then the on-policy ones zhao2025d1; ou2025espo. Meanwhile, models like MDPO he2025mdpo models the diffusion process as a formal markov process, which is computationally expensive and face significant challenges in scaling to large-parameter models. Importance sampling robustness and off-policy stabilization. Variance and tail behavior of importance weights are classical concerns in Monte Carlo and off-policy estimation (hesterberg1988advances; owen2013monte; elvira2021importance). Truncation and clipping control extreme weights (ionides2008truncated), while diagnostics and smoothing characterize heavy-tail regimes (vehtari2015psis). In deep RL, clipped corrections such as V-trace and Retrace mitigate off-policy variance and improve stability (espeholt2018impala; munos2016retrace; liu2018breaking; greensmith2004variance). Our work adapts these robustness principles to the proxy-ratio setting of dLLM RL, where likelihood estimation noise is exponentiated inside the importance weights. 6 Conclusion This paper studies the instability of Group Relative Policy Optimization (GRPO) when applied to discrete diffusion large language models. We identify that GRPO instability in dLLMs stems from the noisy Monte Carlo importance ratio estimation, which triggers a self-reinforcing instability loop of gradient spikes and policy drift. To break this loop, we propose StableDRL, which employs unconditional clipping and self-normalization to eliminate spikes. Extensive experiments demonstrate that our proposed approach effectively stabilizes the training and significantly unlocks the reasoning potential of dLLMs. References Appendix A Details on Staircase Attention and Proxy Estimation In this section, we provide the theoretical details for adapting Reinforcement Learning to Block Diffusion models. We discuss the Monte Carlo estimation of the objective, the efficiency-leakage dilemma, and the formal construction of the Staircase Attention mask. A.1 Monte Carlo Estimation of ELBO For a fixed context c and sequence x, we estimate likelihood proxies by sampling m perturbations. Let ξ=(t,Mt)ξ=(t,M_t) collect the internal diffusion randomness (time and mask). A generic Monte Carlo (MC) estimator of the ELBO takes the form: ℒ^ELBO(x∣c;θ)=−1m∑τ=1m[w(tτ)∑i=1n(Mtτi=1)logπθ(xi∣xtτ(τ),c)], L_ELBO(x c;θ)=- 1m _τ=1^m [w(t_τ) _i=1^n1(M_t_τ^i=1) _θ(x^i x_t_τ^(τ),c) ], (11) where xtτ(τ)x_t_τ^(τ) is produced by the forward process using (tτ,Mtτ)(t_τ,M_t_τ). In standard full-attention models, the conditional logπθ _θ is computed under a full bidirectional mask. However, for block diffusion, we must enforce block-wise conditional independence to ensure the estimator remains a valid lower bound. A.2 The Efficiency-Leakage Dilemma For a sequence divided into K blocks, an exact ELBO estimate requires conditioning each block BkB_k strictly on its clean history x<Bkx_<B_k. • Naive Iterative Implementation (O(K)O(K)): This necessitates K separate forward passes, masking future tokens sequentially. For long sequences (e.g., 64 blocks), this increases the training cost linearly, rendering iterative RL prohibitively expensive. • Standard Single-Pass (Leakage): Conversely, standard bidirectional attention allows all tokens to attend to the full sequence. If applied naively in a single pass, denoising tokens in block BkB_k would attend to the ground-truth representations of their own block, mathematically invalidating the variational bound and the gradient signal. A.3 Dual-Stream Input and Mask Construction To achieve O(1)O(1) evaluation without leakage, we employ a dual-stream (“2L”) input construction. We concatenate a clean context stream xctxx_ctx (frozen history) and a corrupted target stream xtgtx_tgt (containing mask tokens). Let x~=[xctx;xtgt] x=[x_ctx;\,x_tgt] be the combined input of length 2n2n. We define a composite attention mask M∈0,12n×2nM∈\0,1\^2n× 2n that enforces strict block-causal dependency: M=[McausalMstairMintra].M\;=\; bmatrixM_ causal&0\\ M_ stair&M_ intra bmatrix. (12) The components are defined as follows: 1. Top-Left (McausalM_ causal): Standard causal mask for the clean context stream (Blue regions in Figure 3). 2. Top-Right (0): Zero matrix. The clean context cannot attend to the noisy target. 3. Bottom-Right (MintraM_ intra): A block-diagonal mask where (Mintra)ij=1(M_ intra)_ij=1 iff target positions i and j belong to the same block. This corresponds to the Pink regions in Figure 3 and enables intra-block denoising. 4. Bottom-Left (MstairM_ stair): The strictly block-lower-triangular component, corresponding to the Green regions in Figure 3. For a target token in block k and a context token in block l: (Mstair)k,l=1if l<k(Context: Attend to history)0if l≥k(Context: Occlude current/future)(M_ stair)_k,l= cases1&if l<k (Context: Attend to history)\\ 0&if l≥ k (Context: Occlude current/future) cases (13) This construction allows us to compute gradients for all K blocks simultaneously while mathematically preserving the autoregressive factorization required by the objective. Appendix B Proof of Main Results B.1 Formal theorem statements for Sec. 3.3 This subsection presents the formal statements of the instability mechanisms identified in Section 3.3. Theorems B.1–B.3 outline our theoretical framework in three logical steps. First, Theorem B.1 formally characterizes the drift–spike feedback loop inherent to standard GRPO. Second, Theorem B.2 states that two-sided unconditional clipping, while mitigating spikes, may lead to frequent boundary saturation. Finally, Theorem B.3 establishes that self-normalization structurally resolves the remaining random group-scale factor. Detailed proofs are provided in subsequent subsections. Mathematical setup. Fix a behavior policy θold _old and a rollout group ℬ=x1,…,xGB=\x_1,…,x_G\ sampled from πθold _ _old. GRPO performs updates θ0=θold,θ1,θ2,… _0= _old, _1, _2,… on this same fixed group. Write the estimated importance ratio on sample xjx_j at step i as ρ^i,j=exp(Δℒi,j+Δηi,j), ρ_i,j= ( _i,j+ _i,j), where Δℒi,j=ℒθi(xj)−ℒθold(xj) _i,j=L_ _i(x_j)-L_ _old(x_j) is the noise-free drift and Δηi,j _i,j is the corresponding log-ratio noise term. Let A^j A_j be the fixed, group-relative advantage, and define the negative set =j:A^j≤−a0for some a0>0.N=\j: A_j≤-a_0\ some a_0>0. Define the drift state, within-negative spread, and drift-maximizer index Di=maxj∈Δℒi,j,Si=Di−minj∈Δℒi,j,j†∈argmaxj∈Δℒi,j.D_i= _j _i,j, S_i=D_i- _j _i,j, j ∈ _j _i,j. (14) Finally, let g^GRPO,i g_GRPO,i denote the implemented GRPO update direction at step i. Theorem B.1 (GRPO drift–spike feedback loop). Assume the standing Conditions (C1)–(C5) in Appendix B.2. Fix any spike threshold H>0H>0 and define uH:=max1+ϵ,GH(1−λ)a0b0,u0,u_H:= \! \1+ε,\ GH(1-λ)a_0b_0,\ u_0 \, (15) where u0u_0 is the deterministic constant defined in Lemma B.6. Define the spike-probability lower bound Pi(H):=12F¯(loguH−Di).P_i(H)\ :=\ 12\, F\! ( u_H-D_i ). (16) Then, almost surely (conditioning on ℱi−1F_i-1, i.e., the current inner iterate and the fixed rollout group), ℙ(‖g^GRPO,i‖≥H|ℱi−1)≥Pi(H),a.s.P\! (\| g_GRPO,i\|≥ H\, |\,F_i-1 )\ ≥\ P_i(H), .s. (17) and Pi(H)P_i(H) is a nondecreasing function of the drift state DiD_i. Moreover, on any realized step where a single negative-advantage outlier dominates the group update and the local smoothness/geometry conditions in Appendix B.2 hold for that realized update, there exist step-dependent scalars csup,i>0c_sup,i>0 and camp,i∈ℝc_amp,i and indices j⋆∈j and j⋄∈∖j⋆j \j \ such that ℒθi(xj⋆)−ℒθi+1(xj⋆) _ _i(x_j )-L_ _i+1(x_j ) ≥csup,iρ^i,j⋆G, ≥ c_sup,i\, ρ_i,j G, (18) Di+1 D_i+1 ≥Di+(camp,iρ^i,j⋆−Si). ≥ D_i+ (c_amp,i ρ_i,j -S_i ). In particular, if camp,iρ^i,j⋆≥Sic_amp,i ρ_i,j ≥ S_i, then Di+1≥DiD_i+1≥ D_i, hence Pi+1(H)≥Pi(H)(on that realized step).P_i+1(H)\ ≥\ P_i(H) (on that realized step). (19) Theorem B.2 (Boundary saturation under two-sided clipping). Let gi,j:=A^j∇θℒθi(xj)g_i,j:= A_j _θL_ _i(x_j) and assume ‖gi,j‖≤B\|g_i,j\|≤ B (Condition (C1)). Define the two-sided clipped weight wi,j:=clip(ρ^i,j, 1−ϵ, 1+ϵ)w_i,j:=clip( ρ_i,j,\,1-ε,\,1+ε) and the clipping-only direction g^clip,i:=1G∑j=1Gwi,jgi,j g_clip,i:= 1G _j=1^Gw_i,jg_i,j. First, clipping prevents unbounded spikes: deterministically, ‖g^clip,i‖≤(1+ϵ)B\| g_clip,i\|≤(1+ε)B. Second, drift still increases the frequency of hitting the upper clipping boundary. Let j†∈argmaxj∈Δℒi,j ∈ _j _i,j be a drift-maximizer in the negative set. Then, almost surely, ℙ(ρ^i,j†≥1+ϵ|ℱi−1)≥F¯(log(1+ϵ)−Di),P\! ( ρ_i,j ≥ 1+ε\, |\,F_i-1 )\ ≥\ F\! ( (1+ε)-D_i ), (20) and the right-hand side is nondecreasing in DiD_i. Under the additional dominance event in Lemma B.14 and the same local smoothness/geometry conditions used in Appendix B.2, there exists a step-dependent scalar camp,i∈ℝc_amp,i such that Di+1≥Di+(camp,i(1+ϵ)−Si).D_i+1\ ≥\ D_i+ (c_amp,i(1+ε)-S_i ). Thus clipping alone can replace rare extreme spikes with frequent boundary-saturated updates once drift becomes large. Theorem B.3 (Self-normalization removes the random group-scale factor). With the same clipped weights wi,jw_i,j as in Theorem B.2, define the self-normalized direction g^sn,i g_sn,i :=∑j=1Gwi,jgi,j∑j=1Gwi,j, = _j=1^Gw_i,jg_i,j _j=1^Gw_i,j, (21) g^clip,i g_clip,i =(1G∑j=1Gwi,j)g^sn,i. = ( 1G _j=1^Gw_i,j ) g_sn,i. Since wi,j>0w_i,j>0 (because ρ^i,j=exp(⋅)>0 ρ_i,j= (·)>0) and ∑jwi,j>0 _jw_i,j>0, the coefficients wi,j/∑kwi,kw_i,j/ _kw_i,k form a convex combination, so g^sn,i g_sn,i always lies in the convex hull of gi,jj=1G\g_i,j\_j=1^G. In particular, if ‖gi,j‖≤B\|g_i,j\|≤ B then deterministically ‖g^sn,i‖≤B\| g_sn,i\|≤ B. Thus self-normalization explicitly divides out the random group-scale factor 1G∑jwi,j 1G _jw_i,j that remains under clipping-only. B.2 Proof of Theorem B.1 We present the proof of Theorem B.1. For clarity, we first state the necessary setup and assumptions. Deterministic proxy gradients and GRPO effective weights. Define deterministic proxy gradients hi,j:=∇θℒθi(xj),gi,j:=A^jhi,j.h_i,j:= _θL_ _i(x_j), g_i,j:= A_j\,h_i,j. We write the implemented GRPO direction in the equivalent “effective-weight” form g^GRPO,i:=1G∑j=1Gmi,jgi,j, g_GRPO,i:= 1G _j=1^Gm_i,j\,g_i,j, (22) where the (random) effective multiplier mi,jm_i,j is mi,j:=min(ρ^i,j, 1+ϵ),A^j≥0,max(ρ^i,j, 1−ϵ),A^j<0.m_i,j:= cases ( ρ_i,j,\,1+ε),& A_j≥ 0,\\ ( ρ_i,j,\,1-ε),& A_j<0. cases (23) This form is exactly equivalent to the usual GRPO “min–clip” surrogate: for A^j≥0 A_j≥ 0 the weight is clipped at 1+ϵ1+ε, while for A^j<0 A_j<0 the weight is not clipped from above. We do not assume a particular optimizer beyond the update form θi+1=θi+η0g^GRPO,i _i+1= _i+ _0\, g_GRPO,i for some learning rate η0>0 _0>0. Filtration. Let ℱiF_i denote the σ-field generated by all algorithmic randomness up to and including step i. Then θi _i is ℱi−1F_i-1-measurable; hence each drift value Δℒi,j _i,j is ℱi−1F_i-1-measurable. Standing conditions (C1–C5). We work under the following conditions. (C1) is standard and typically enforced by gradient clipping; (C3) holds when the Monte Carlo proxy evaluations use independent randomness across samples; and (C5) is empirically checkable by monitoring ∑j≠j†mi,j _j≠ j m_i,j. • (C1) Bounded per-sample directions. There exists B<∞B<∞ such that ‖gi,j‖≤B\|g_i,j\|≤ B for all inner steps i and all samples j. • (C2) Conditional common right-tail envelope for log-ratio noise. For each inner step i and sample j, define the conditional survival function F¯j,i(z):=ℙ(Δηi,j≥z|ℱi−1),z∈ℝ. F_j,i(z):=P ( _i,j≥ z\, |\,F_i-1 ), z . Assume that for every j∈j and every i, F¯j,i F_j,i has unbounded support in the sense that F¯j,i(z)>0 F_j,i(z)>0 for all z∈ℝz almost surely. Moreover, assume there exists a deterministic nonincreasing function F¯:ℝ→(0,1] F:R→(0,1] (a uniform tail lower envelope) such that almost surely, for all i and all j∈j , F¯j,i(z)≥F¯(z)∀z∈ℝ. F_j,i(z)≥ F(z) ∀ z . • (C3) Conditional independence across samples. For each inner step i, conditional on ℱi−1F_i-1, the noises Δηi,jj=1G\ _i,j\_j=1^G are independent. Equivalently, conditional on ℱi−1F_i-1, the ratios ρ^i,jj=1G\ ρ_i,j\_j=1^G (and thus the effective weights mi,j\m_i,j\) are independent across j. • (C4) Nontrivial proxy gradient at the drift-maximizer. There exists b0>0b_0>0 such that for all inner steps i, the drift-maximizer in the negative set satisfies ‖hi,j†‖≥b0\|h_i,j \|≥ b_0. • (C5) Residual effective-weight moment control. There exists a deterministic constant W<∞W<∞ such that for all inner steps i, [∑j≠j†mi,j|ℱi−1]≤W,a.s.E\! [ _j≠ j m_i,j\ |\ F_i-1 ]\ ≤\ W, .s. Remark B.4. Condition (C5) upper-bounds the expected total effective-weight mass of the samples other than the drift-maximizer. This quantity is directly measurable in experiments as the sum ∑j≠j†mi,j _j≠ j m_i,j. Controlling this expectation guarantees via Markov’s inequality that when ρ^i,j† ρ_i,j is large, the drift-maximizer dominates the group update with a constant probability. Lemma B.5 (Ratio exceedance identity and drift monotonicity). Fix an inner step i, an index j, and a threshold u>0u>0. Then ℙ(ρ^i,j≥u|ℱi−1)=ℙ(Δηi,j≥logu−Δℒi,j|ℱi−1)=F¯j,i(logu−Δℒi,j).P\! ( ρ_i,j≥ u\, |\,F_i-1 )=P\! ( _i,j≥ u- _i,j\, |\,F_i-1 )= F_j,i\! ( u- _i,j ). (24) Moreover, conditional on ℱi−1F_i-1, the map Δℒ↦F¯j,i(logu−Δℒ) F_j,i( u- ) is nondecreasing. Proof. By definition, ρ^i,j=exp(Δℒi,j+Δηi,j) ρ_i,j= ( _i,j+ _i,j). Since exp(⋅) (·) is strictly increasing, ρ^i,j≥u⇔Δℒi,j+Δηi,j≥logu⇔Δηi,j≥logu−Δℒi,j.\ ρ_i,j≥ u\ \ _i,j+ _i,j≥ u\ \ _i,j≥ u- _i,j\. Taking conditional probabilities given ℱi−1F_i-1 yields (24). For monotonicity, conditional on ℱi−1F_i-1 the survival function F¯j,i F_j,i is nonincreasing in its argument, while Δℒ↦logu−Δℒ u- is strictly decreasing; therefore their composition is nondecreasing. ∎ Lemma B.6 (Dominance from a large drift-maximizer ratio via a moment bound). Fix an inner step i and let j†∈argmaxj∈Δℒi,j ∈ _j _i,j. Fix any λ∈[0,1)λ∈[0,1) and define the (step-i) residual vector ri:=1G∑j≠j†mi,jgi,j.r_i:= 1G _j≠ j m_i,jg_i,j. Assume Conditions (C1)–(C5). Define u0:=2BWλa0b0.u_0:= 2BWλ a_0b_0. (25) Then for any u≥u0u≥ u_0, ℙ(‖ri‖≤λ1Gua0b0|ℱi−1,ρ^i,j†≥u)≥12,a.s.P\! (\|r_i\|≤λ\, 1G\,u\,a_0b_0\ |\ F_i-1,\ ρ_i,j ≥ u )\ ≥\ 12, .s. (26) Moreover, on the event ρ^i,j†≥u\ ρ_i,j ≥ u\ with u≥1+ϵu≥ 1+ε, since j†∈j we have mi,j†=ρ^i,j†m_i,j = ρ_i,j and thus g^GRPO,i=−1Gρ^i,j†|A^j†|hi,j†+ri. g_GRPO,i=- 1G\, ρ_i,j \,| A_j |\,h_i,j +r_i. (27) Proof. First, by (C1), ‖ri‖=‖1G∑j≠j†mi,jgi,j‖≤1G∑j≠j†mi,j‖gi,j‖≤BG∑j≠j†mi,j.\|r_i\|= \| 1G _j≠ j m_i,jg_i,j \|≤ 1G _j≠ j m_i,j\|g_i,j\|≤ BG _j≠ j m_i,j. Therefore, the event ∑j≠j†mi,j≤λBua0b0 \ _j≠ j m_i,j≤ λB\,u\,a_0b_0 \ implies ‖ri‖≤λ1Gua0b0\|r_i\|≤λ 1Gua_0b_0. Next, by (C3), conditional on ℱi−1F_i-1 the collection mi,jj≠j†\m_i,j\_j≠ j is independent of ρ^i,j† ρ_i,j , hence independent of the event ρ^i,j†≥u\ ρ_i,j ≥ u\. Thus, for any threshold t>0t>0, ℙ(∑j≠j†mi,j>t|ℱi−1,ρ^i,j†≥u)=ℙ(∑j≠j†mi,j>t|ℱi−1).P\! ( _j≠ j m_i,j>t\ |\ F_i-1,\ ρ_i,j ≥ u )=P\! ( _j≠ j m_i,j>t\ |\ F_i-1 ). Applying Markov’s inequality and (C5) yields ℙ(∑j≠j†mi,j>t|ℱi−1)≤[∑j≠j†mi,j∣ℱi−1]t≤Wt.P\! ( _j≠ j m_i,j>t\ |\ F_i-1 )≤ E\! [ _j≠ j m_i,j _i-1 ]t≤ Wt. Choose t=λBua0b0t= λBua_0b_0. If u≥u0=2BWλa0b0u≥ u_0= 2BWλ a_0b_0, then W/t≤1/2W/t≤ 1/2, hence ℙ(∑j≠j†mi,j≤λBua0b0|ℱi−1,ρ^i,j†≥u)≥12.P\! ( _j≠ j m_i,j≤ λBua_0b_0\ |\ F_i-1,\ ρ_i,j ≥ u )≥ 12. Combining with ‖ri‖≤BG∑j≠j†mi,j\|r_i\|≤ BG _j≠ j m_i,j gives (26). Finally, on ρ^i,j†≥u\ ρ_i,j ≥ u\ with u≥1+ϵu≥ 1+ε, since j†∈j we have mi,j†=max(ρ^i,j†,1−ϵ)=ρ^i,j†m_i,j = ( ρ_i,j ,1-ε)= ρ_i,j . Also gi,j†=A^j†hi,j†=−|A^j†|hi,j†g_i,j = A_j h_i,j =-| A_j |h_i,j . Substituting into (22) yields (27). ∎ Lemma B.7 (Dominance implies a gradient spike). Fix an inner step i and let j†j be as above. Assume (C1) and (C4). On the event ρ^i,j†≥u\ ρ_i,j ≥ u\ with u≥1+ϵu≥ 1+ε, and on any event where ‖ri‖≤λ1Gua0b0,\|r_i\|≤λ\, 1G\,u\,a_0b_0, (28) we have ‖g^GRPO,i‖≥(1−λ)1Gua0b0.\| g_GRPO,i\|≥(1-λ)\, 1G\,u\,a_0b_0. In particular, if u≥GH(1−λ)a0b0,u\ ≥\ GH(1-λ)a_0b_0, (29) then ‖g^GRPO,i‖≥H\| g_GRPO,i\|≥ H holds on the same event. Proof. On ρ^i,j†≥u\ ρ_i,j ≥ u\ with u≥1+ϵu≥ 1+ε, Lemma B.6 gives the decomposition g^GRPO,i=−1Gρ^i,j†|A^j†|hi,j†+ri. g_GRPO,i=- 1G\, ρ_i,j \,| A_j |\,h_i,j +r_i. Apply the reverse triangle inequality: ‖g^GRPO,i‖≥1Gρ^i,j†|A^j†|‖hi,j†‖−‖ri‖.\| g_GRPO,i\|≥ 1G ρ_i,j | A_j |\|h_i,j \|-\|r_i\|. Since j†∈j implies |A^j†|≥a0| A_j |≥ a_0, and (C4) gives ‖hi,j†‖≥b0\|h_i,j \|≥ b_0, and ρ^i,j†≥u ρ_i,j ≥ u, we obtain 1Gρ^i,j†|A^j†|‖hi,j†‖≥1Gua0b0. 1G ρ_i,j | A_j |\|h_i,j \|≥ 1Gua_0b_0. Together with (28) this yields ‖g^GRPO,i‖≥1Gua0b0−λ1Gua0b0=(1−λ)1Gua0b0.\| g_GRPO,i\|≥ 1Gua_0b_0-λ 1Gua_0b_0=(1-λ) 1Gua_0b_0. If (29) holds, then the right-hand side is at least H. ∎ Lemma B.8 (A drift-monotone lower bound on spike probability). Fix a step i and a spike threshold H>0H>0. Let j†∈argmaxj∈Δℒi,j ∈ _j _i,j so that Δℒi,j†=Di _i,j =D_i. Define uHu_H as in (15). Assume Conditions (C1)–(C5). Then, almost surely, ℙ(‖g^GRPO,i‖≥H|ℱi−1)≥12⋅ℙ(ρ^i,j†≥uH|ℱi−1).P\! (\| g_GRPO,i\|≥ H\, |\,F_i-1 )\ ≥\ 12·P\! ( ρ_i,j ≥ u_H\, |\,F_i-1 ). (30) Moreover, by Lemma B.5 and (C2), ℙ(ρ^i,j†≥uH|ℱi−1)=F¯j†,i(loguH−Di)≥F¯(loguH−Di),P\! ( ρ_i,j ≥ u_H\, |\,F_i-1 )= F_j ,i\! ( u_H-D_i )≥ F\! ( u_H-D_i ), (31) and the right-hand side is nondecreasing in DiD_i. Proof. Work conditionally on ℱi−1F_i-1. Since uH≥1+ϵu_H≥ 1+ε by definition, on the event ρ^i,j†≥uH\ ρ_i,j ≥ u_H\ we have mi,j†=ρ^i,j†m_i,j = ρ_i,j . By Lemma B.6 with u=uHu=u_H, we have ℙ(‖ri‖≤λ1GuHa0b0|ℱi−1,ρ^i,j†≥uH)≥12.P\! (\|r_i\|≤λ 1Gu_Ha_0b_0\ |\ F_i-1,\ ρ_i,j ≥ u_H )≥ 12. On the intersection of ρ^i,j†≥uH\ ρ_i,j ≥ u_H\ and ‖ri‖≤λ1GuHa0b0\\|r_i\|≤λ 1Gu_Ha_0b_0\, Lemma B.7 implies ‖g^GRPO,i‖≥H\| g_GRPO,i\|≥ H because uH≥GH/((1−λ)a0b0)u_H≥ GH/((1-λ)a_0b_0). Therefore, ℙ(‖g^GRPO,i‖≥H|ℱi−1) \! (\| g_GRPO,i\|≥ H\, |\,F_i-1 ) ≥ℙ(ρ^i,j†≥uH,‖ri‖≤λ1GuHa0b0|ℱi−1) \! ( ρ_i,j ≥ u_H,\ \|r_i\|≤λ 1Gu_Ha_0b_0\, |\,F_i-1 ) =ℙ(ρ^i,j†≥uH|ℱi−1)⋅ℙ(‖ri‖≤λ1GuHa0b0|ℱi−1,ρ^i,j†≥uH) =P\! ( ρ_i,j ≥ u_H\, |\,F_i-1 )·P\! (\|r_i\|≤λ 1Gu_Ha_0b_0\ |\ F_i-1,\ ρ_i,j ≥ u_H ) ≥12⋅ℙ(ρ^i,j†≥uH|ℱi−1), ≥ 12·P\! ( ρ_i,j ≥ u_H\, |\,F_i-1 ), which proves (30). The tail identity and lower bound (31) follow from Lemma B.5 with Δℒi,j†=Di _i,j =D_i, and (C2). Monotonicity in DiD_i follows from Lemma B.5. ∎ Lemma B.9 (Quadratic remainder for L-smooth functions). Let f:ℝd→ℝf:R^d be differentiable and L-smooth on the segment θ+t(θ′−θ):t∈[0,1]\θ+t(θ -θ):t∈[0,1]\. Then f(θ′)≤f(θ)+⟨∇f(θ),θ′−θ⟩+L2‖θ′−θ‖2,f(θ′)≥f(θ)+⟨∇f(θ),θ′−θ⟩−L2‖θ′−θ‖2.f(θ )≤ f(θ)+ ∇ f(θ),θ -θ + L2\|θ -θ\|^2, f(θ )≥ f(θ)+ ∇ f(θ),θ -θ - L2\|θ -θ\|^2. Proof. Let d:=θ′−θd:=θ -θ and define the univariate function φ(t):=f(θ+td),t∈[0,1]. (t):=f(θ+td), t∈[0,1]. Since f is differentiable on the segment θ+td:t∈[0,1]\θ+td:t∈[0,1]\, φ is differentiable and φ′(t)=⟨∇f(θ+td),d⟩. (t)= ∇ f(θ+td),\,d . By the fundamental theorem of calculus, f(θ′)−f(θ)=φ(1)−φ(0)=∫01φ′(t)t=∫01⟨∇f(θ+td),d⟩t.f(θ )-f(θ)= (1)- (0)= _0^1 (t)\,dt= _0^1 ∇ f(θ+td),\,d dt. Add and subtract ∇f(θ)∇ f(θ) inside the inner product: f(θ′)−f(θ)=⟨∇f(θ),d⟩+∫01⟨∇f(θ+td)−∇f(θ),d⟩t.f(θ )-f(θ)= ∇ f(θ),\,d + _0^1 ∇ f(θ+td)-∇ f(θ),\,d dt. Using Cauchy–Schwarz and L-smoothness (i.e., ‖∇f(u)−∇f(v)‖≤L‖u−v‖\|∇ f(u)-∇ f(v)\|≤ L\|u-v\| on the segment), for each t∈[0,1]t∈[0,1] we have |⟨∇f(θ+td)−∇f(θ),d⟩|≤‖∇f(θ+td)−∇f(θ)‖‖d‖≤Lt‖d‖2. | ∇ f(θ+td)-∇ f(θ),\,d |≤\|∇ f(θ+td)-∇ f(θ)\|\,\|d\|≤ L\,t\,\|d\|^2. Therefore, ∫01⟨∇f(θ+td)−∇f(θ),d⟩t≤∫01Lt‖d‖2t=L2‖d‖2, _0^1 ∇ f(θ+td)-∇ f(θ),\,d dt≤ _0^1Lt\|d\|^2\,dt= L2\|d\|^2, which gives f(θ′)≤f(θ)+⟨∇f(θ),θ′−θ⟩+L2‖θ′−θ‖2.f(θ )≤ f(θ)+ ∇ f(θ),θ -θ + L2\|θ -θ\|^2. Similarly, using ⟨∇f(θ+td)−∇f(θ),d⟩≥−Lt‖d‖2 ∇ f(θ+td)-∇ f(θ),\,d ≥-Lt\|d\|^2 yields f(θ′)≥f(θ)+⟨∇f(θ),θ′−θ⟩−L2‖θ′−θ‖2.f(θ )≥ f(θ)+ ∇ f(θ),θ -θ - L2\|θ -θ\|^2. ∎ Theorem B.10 (One-step decrease of ℒL on a dominating sample). Fix a step i and an index j⋆∈j . Assume that at this realized step the group update is dominated by j⋆j in the sense that g^GRPO,i=−1Gρ^i,j⋆|A^j⋆|hi,j⋆+ri⋆,‖ri⋆‖≤λ1Gρ^i,j⋆|A^j⋆|‖hi,j⋆‖. g_GRPO,i=- 1G\, ρ_i,j \,| A_j |\,h_i,j +r_i , \|r_i \|≤λ\, 1G\, ρ_i,j \,| A_j |\,\|h_i,j \|. (32) Define v:=hi,j⋆=∇θℒθi(xj⋆),η:=η0Gρ^i,j⋆|A^j⋆|,δ:=η0ri⋆.v:=h_i,j = _θL_ _i(x_j ), η:= _0G\, ρ_i,j \,| A_j |, δ:= _0r_i . Then θi+1=θi−ηv+δ _i+1= _i-η v+δ and ‖δ‖≤λη‖v‖\|δ\|≤λη\|v\|. Let f⋆(θ):=ℒθ(xj⋆)f_ (θ):=L_θ(x_j ). Assume f⋆f_ is L⋆L_ -smooth on the realized segment [θi,θi+1][ _i, _i+1]. Then ℒθi(xj⋆)−ℒθi+1(xj⋆)≥η‖v‖2((1−λ)−L⋆2(1+λ)2η).L_ _i(x_j )-L_ _i+1(x_j )≥η\|v\|^2 ((1-λ)- L_ 2(1+λ)^2η ). (33) In particular, if η≤1−λL⋆(1+λ)2η≤ 1-λL_ (1+λ)^2, then ℒθi(xj⋆)−ℒθi+1(xj⋆)≥1−λ2η‖v‖2.L_ _i(x_j )-L_ _i+1(x_j )≥ 1-λ2\,η\,\|v\|^2. (34) Proof. Apply Lemma B.9 to f⋆f_ at (θ,θ′)=(θi,θi+1)(θ,θ )=( _i, _i+1): f⋆(θi+1)≤f⋆(θi)+⟨∇f⋆(θi),θi+1−θi⟩+L⋆2‖θi+1−θi‖2.f_ ( _i+1)≤ f_ ( _i)+ ∇ f_ ( _i), _i+1- _i + L_ 2\| _i+1- _i\|^2. Rearrange: f⋆(θi)−f⋆(θi+1)≥−⟨v,−ηv+δ⟩−L⋆2‖−ηv+δ‖2=η‖v‖2−⟨v,δ⟩−L⋆2‖−ηv+δ‖2.f_ ( _i)-f_ ( _i+1)≥- v,-η v+δ - L_ 2\|-η v+δ\|^2=η\|v\|^2- v,δ - L_ 2\|-η v+δ\|^2. Bound ⟨v,δ⟩≤‖v‖‖δ‖≤λη‖v‖2 v,δ ≤\|v\|\|δ\|≤λη\|v\|^2. Also ‖−ηv+δ‖≤η‖v‖+‖δ‖≤(1+λ)η‖v‖\|-η v+δ\|≤η\|v\|+\|δ\|≤(1+λ)η\|v\|. Substitute to obtain (33). If η≤1−λL⋆(1+λ)2η≤ 1-λL_ (1+λ)^2, then the bracket is at least (1−λ)/2(1-λ)/2, yielding (34). ∎ Definition B.11 (Anti-alignment and directional curvature). Fix distinct indices j⋆≠j⋄j ≠ j and define x⋆:=xj⋆x :=x_j and x⋄:=xj⋄x :=x_j . Let v:=∇θℒθi(x⋆),u:=∇θℒθi(x⋄),γ:=−⟨u,v⟩.v:= _θL_ _i(x ), u:= _θL_ _i(x ), γ:=- u,v . Theorem B.12 (Cross-sample amplification with residual (proxy drift increase)). Fix a step i and two indices j⋆≠j⋄j ≠ j in N. Assume the outlier-dominance decomposition (32) holds on this realized step, and define v=∇θℒθi(xj⋆),u=∇θℒθi(xj⋄),γ=−⟨u,v⟩>0, v= _θL_ _i(x_j ), u= _θL_ _i(x_j ), γ=- u,v >0, η=η0Gρ^i,j⋆|A^j⋆|,θi+1=θi−ηv+δ,‖δ‖≤λη‖v‖. η= _0G ρ_i,j \,| A_j |, _i+1= _i-η v+δ, \|δ\|≤λη\|v\|. Let f⋄(θ):=ℒθ(xj⋄)f_ (θ):=L_θ(x_j ) and assume f⋄f_ is L⋄L_ -smooth on the realized segment [θi,θi+1][ _i, _i+1]. If η≤γ(1+λ)2L⋄‖v‖2,η≤ γ(1+λ)^2\,L_ \|v\|^2, (35) then on this realized step we have ℒθi+1(xj⋄)−ℒθi(xj⋄)≥η(γ2−λ‖u‖‖v‖),L_ _i+1(x_j )-L_ _i(x_j )≥η ( γ2-λ\|u\|\|v\| ), (36) and consequently, Δℒi+1,j⋄≥Δℒi,j⋄+η(γ2−λ‖u‖‖v‖). _i+1,j ≥ _i,j +η ( γ2-λ\|u\|\|v\| ). (37) Proof. Apply Lemma B.9 (lower bound form) to f⋄f_ at (θ,θ′)=(θi,θi+1)(θ,θ )=( _i, _i+1): f⋄(θi+1)≥f⋄(θi)+⟨∇f⋄(θi),θi+1−θi⟩−L⋄2‖θi+1−θi‖2.f_ ( _i+1)≥ f_ ( _i)+ ∇ f_ ( _i), _i+1- _i - L_ 2\| _i+1- _i\|^2. Substitute ∇f⋄(θi)=u∇ f_ ( _i)=u and θi+1−θi=−ηv+δ _i+1- _i=-η v+δ: f⋄(θi+1)−f⋄(θi)≥⟨u,−ηv+δ⟩−L⋄2‖−ηv+δ‖2=ηγ+⟨u,δ⟩−L⋄2‖−ηv+δ‖2.f_ ( _i+1)-f_ ( _i)≥ u,-η v+δ - L_ 2\|-η v+δ\|^2=ηγ+ u,δ - L_ 2\|-η v+δ\|^2. Use ⟨u,δ⟩≥−‖u‖‖δ‖≥−λη‖u‖‖v‖ u,δ ≥-\|u\|\|δ\|≥-λη\|u\|\|v\| and ‖−ηv+δ‖≤(1+λ)η‖v‖\|-η v+δ\|≤(1+λ)η\|v\| to get f⋄(θi+1)−f⋄(θi)≥ηγ−λη‖u‖‖v‖−L⋄2(1+λ)2η2‖v‖2.f_ ( _i+1)-f_ ( _i)≥ηγ-λη\|u\|\|v\|- L_ 2(1+λ)^2η^2\|v\|^2. Under (35), the quadratic term is at most 12ηγ 12ηγ, yielding (36). Equation (37) is just rewriting in terms of Δℒ . ∎ Lemma B.13 (From amplification of one sample to an increase in DiD_i). Fix a step i and suppose Theorem B.12 applies for some j⋆≠j⋄j ≠ j in N. Define η=η0Gρ^i,j⋆|A^j⋆|.η= _0G ρ_i,j \,| A_j |. Define the ℱi−1F_i-1-measurable coefficient camp,i:=η0|A^j⋆|G(γ2−λ‖u‖‖v‖)∈ℝ,c_amp,i:= _0| A_j |G ( γ2-λ\|u\|\|v\| ) , where u=∇θℒθi(xj⋄)u= _θL_ _i(x_j ), v=∇θℒθi(xj⋆)v= _θL_ _i(x_j ), and γ=−⟨u,v⟩>0γ=- u,v >0. Then on this realized step, Di+1≥Di+(camp,iρ^i,j⋆−Si).D_i+1≥ D_i+ (c_amp,i ρ_i,j -S_i ). (38) Proof. By Theorem B.12, Δℒi+1,j⋄≥Δℒi,j⋄+η(γ2−λ‖u‖‖v‖)=Δℒi,j⋄+camp,iρ^i,j⋆. _i+1,j ≥ _i,j +η ( γ2-λ\|u\|\|v\| )= _i,j +c_amp,i ρ_i,j . Since Di+1=maxj∈Δℒi+1,j≥Δℒi+1,j⋄D_i+1= _j _i+1,j≥ _i+1,j , we have Di+1≥Δℒi,j⋄+camp,iρ^i,j⋆.D_i+1≥ _i,j +c_amp,i ρ_i,j . By definition of SiS_i in (14), Δℒi,j⋄≥minj∈Δℒi,j=Di−Si. _i,j ≥ _j _i,j=D_i-S_i. Substituting yields (38). ∎ Proof of Theorem B.1. The spike-probability bound (17) follows from Lemma B.8 and (C2): ℙ(‖g^GRPO,i‖≥H|ℱi−1)≥12F¯(loguH−Di).P\! (\| g_GRPO,i\|≥ H\, |\,F_i-1 )≥ 12\, F( u_H-D_i). Monotonicity in DiD_i holds by Lemma B.5. For the one-step decrease bound on a dominating sample, consider a realized step i where a negative-advantage sample j⋆∈j with ρ^i,j⋆≥1+ϵ ρ_i,j ≥ 1+ε dominates the group update in the sense of (32) and where the local smoothness/step-size condition of Theorem B.10 holds, including (34). Then (34) gives ℒθi(xj⋆)−ℒθi+1(xj⋆)≥1−λ2η‖v‖2=1−λ2η0Gρ^i,j⋆|A^j⋆|‖v‖2,L_ _i(x_j )-L_ _i+1(x_j )≥ 1-λ2\,η\,\|v\|^2= 1-λ2\, _0G\, ρ_i,j | A_j |\,\|v\|^2, with v=∇θℒθi(xj⋆)v= _θL_ _i(x_j ). Thus the first inequality in (18) holds with csup,i:=1−λ2η0|A^j⋆|‖v‖2>0.c_sup,i:= 1-λ2\, _0\,| A_j |\,\|v\|^2>0. For the drift-state increment, Lemma B.13 gives Di+1≥Di+(camp,iρ^i,j⋆−Si),D_i+1≥ D_i+ (c_amp,i ρ_i,j -S_i ), establishing the second inequality in (18). Finally, if camp,iρ^i,j⋆≥Sic_amp,i ρ_i,j ≥ S_i then Di+1≥DiD_i+1≥ D_i. Since D↦F¯(loguH−D)D F( u_H-D) is nondecreasing (Lemma B.5), the lower bound Pi(H)=12F¯(loguH−Di)P_i(H)= 12 F( u_H-D_i) cannot decrease from step i to step i+1i+1 on that realized step, i.e., (19) holds. ∎ B.3 Proof of Theorem B.2 We prove Theorem B.2 for the two-sided unconditional clipping rule wi,j=clip(ρ^i,j, 1−ϵ, 1+ϵ)w_i,j=clip( ρ_i,j,\,1-ε,\,1+ε) and g^clip,i:=1G∑j=1Gwi,jgi,j g_clip,i:= 1G _j=1^Gw_i,jg_i,j. Lemma B.14 (A sufficient upper-bound dominance event under two-sided clipping). Fix an inner step i and let j†∈argmaxj∈Δℒi,j ∈ _j _i,j. Assume (C1) and (C4). Define the residual riclip:=1G∑j≠j†wi,jgi,j.r_i^clip:= 1G _j≠ j w_i,jg_i,j. On any realized step where ρ^i,j†≥1+ϵ ρ_i,j ≥ 1+ε and ∑j≠j†wi,j≤λa0b0B(1+ϵ), _j≠ j w_i,j\ ≤\ λ a_0b_0B\,(1+ε), (39) we have the deterministic decomposition g^clip,i=−1G(1+ϵ)|A^j†|hi,j†+riclip,‖riclip‖≤λ1G(1+ϵ)|A^j†|‖hi,j†‖. g_clip,i=- 1G(1+ε)\,| A_j |\,h_i,j +r_i^clip, \|r_i^clip\|≤λ\, 1G(1+ε)\,| A_j |\,\|h_i,j \|. Proof. On ρ^i,j†≥1+ϵ ρ_i,j ≥ 1+ε, we have wi,j†=1+ϵw_i,j =1+ε and gi,j†=A^j†hi,j†=−|A^j†|hi,j†g_i,j = A_j h_i,j =-| A_j |h_i,j . Thus g^clip,i=1Gwi,j†gi,j†+1G∑j≠j†wi,jgi,j=−1G(1+ϵ)|A^j†|hi,j†+riclip. g_clip,i= 1Gw_i,j g_i,j + 1G _j≠ j w_i,jg_i,j=- 1G(1+ε)| A_j |h_i,j +r_i^clip. Moreover, by (C1), ‖riclip‖≤1G∑j≠j†wi,j‖gi,j‖≤BG∑j≠j†wi,j.\|r_i^clip\|≤ 1G _j≠ j w_i,j\|g_i,j\|≤ BG _j≠ j w_i,j. Under (39), this yields ‖riclip‖≤BG⋅λa0b0B(1+ϵ)=λ1G(1+ϵ)a0b0≤λ1G(1+ϵ)|A^j†|‖hi,j†‖,\|r_i^clip\|≤ BG· λ a_0b_0B(1+ε)=λ\, 1G(1+ε)a_0b_0≤λ\, 1G(1+ε)| A_j |\|h_i,j \|, since |A^j†|≥a0| A_j |≥ a_0 and ‖hi,j†‖≥b0\|h_i,j \|≥ b_0 by (C4). ∎ Proof of Theorem B.2. Fix an inner step i. By (C1) we have ‖gi,j‖≤B\|g_i,j\|≤ B for all j. Moreover, since ρ^i,j>0 ρ_i,j>0 and wi,j=clip(ρ^i,j,1−ϵ,1+ϵ)w_i,j=clip( ρ_i,j,1-ε,1+ε), we have 0<wi,j≤1+ϵ0<w_i,j≤ 1+ε. Therefore, ‖g^clip,i‖=‖1G∑j=1Gwi,jgi,j‖≤1G∑j=1Gwi,j‖gi,j‖≤(1+ϵ)B,\| g_clip,i\|= \| 1G _j=1^Gw_i,j\,g_i,j \|≤ 1G _j=1^Gw_i,j\,\|g_i,j\|≤(1+ε)B, which proves the deterministic boundedness claim. Let j†∈argmaxj∈Δℒi,j ∈ _j _i,j so that Δℒi,j†=Di _i,j =D_i. By Lemma B.5 with u=1+ϵu=1+ε and (C2), ℙ(ρ^i,j†≥1+ϵ|ℱi−1)=F¯j†,i(log(1+ϵ)−Di)≥F¯(log(1+ϵ)−Di),P\! ( ρ_i,j ≥ 1+ε\, |\,F_i-1 )= F_j ,i\! ( (1+ε)-D_i )≥ F\! ( (1+ε)-D_i ), and the right-hand side is nondecreasing in DiD_i by Lemma B.5, which establishes (20). Finally, on any realized step where the sufficient dominance event in Lemma B.14 holds and where the local smoothness/geometry conditions required by Theorem B.12 (with the effective step size η=η0G(1+ϵ)|A^j†|η= _0G(1+ε)| A_j |) hold for some j⋄∈∖j†j \j \, the same argument as Lemma B.13 yields Di+1≥Di+(camp,i(1+ϵ)−Si).D_i+1\ ≥\ D_i+ (c_amp,i(1+ε)-S_i ). This completes the proof. ∎ B.4 Proof of Theorem B.3 Proof of Theorem B.3. Fix an inner step i and define the two-sided clipped weights wi,j:=clip(ρ^i,j, 1−ϵ, 1+ϵ),j=1,…,G.w_i,j:=clip( ρ_i,j,\,1-ε,\,1+ε), j=1,…,G. Since ρ^i,j>0 ρ_i,j>0, we have wi,j>0w_i,j>0 and thus ∑k=1Gwi,k>0 _k=1^Gw_i,k>0. Define g^sn,i:=∑j=1Gwi,jgi,j∑j=1Gwi,j. g_sn,i:= _j=1^Gw_i,j\,g_i,j _j=1^Gw_i,j. Let αi,j:=wi,j/∑k=1Gwi,k _i,j:=w_i,j/ _k=1^Gw_i,k. Then αi,j≥0 _i,j≥ 0 and ∑j=1Gαi,j=1 _j=1^G _i,j=1, hence g^sn,i=∑j=1Gαi,jgi,j∈convgi,1,…,gi,G. g_sn,i= _j=1^G _i,jg_i,j \g_i,1,…,g_i,G\. By (C1), ‖gi,j‖≤B\|g_i,j\|≤ B for all j, therefore ‖g^sn,i‖≤∑j=1Gαi,j‖gi,j‖≤∑j=1Gαi,jB=B.\| g_sn,i\|≤ _j=1^G _i,j\|g_i,j\|≤ _j=1^G _i,jB=B. This proves the deterministic bound and the convex-hull property. ∎ Appendix C Experimental Details C.1 Training and Hyperparameter Setup We provide detailed configurations for our experiments on both Full-Attention Diffusion and Block Diffusion architectures to ensure reproducibility. All experiments were conducted using the StableDRL framework, with hyperparameters chosen to isolate the contribution of our stability mechanisms. C.1.1 Full-Attention Diffusion (LLaDA-8B-Instruct) We fine-tune the LLaDA-8B-Instruct model using iterative decoding with a generation length of 256 tokens and a block size of 32. Optimization is performed using AdamW with a learning rate of 1.0×10−61.0× 10^-6 and a linear decay schedule over 2,000 steps. Crucially, we enable Self-Normalized Importance Sampling (SNIS) with an unconditional importance weight clipping threshold of 5.0. Table 3 summarizes the complete hyperparameter configuration. Table 3: Hyperparameter Configuration for Full-Attention Diffusion (LLaDA-8B-Instruct) Category Value Model & Initialization Base Model LLaDA-8B-Instruct Precision bfloat16 Activation Checkpointing Whole Layer Generation (Rollout) Decoding Strategy Iterative (128 steps) Generation Length 256 tokens Block Size 32 Temperature 0.9 Rollout Scale 8 generations × 2 repeats Training & Optimization Optimizer AdamW (β1=0.9,β2=0.99,λ=0.1 _1=0.9, _2=0.99,λ=0.1) Learning Rate 1.0×10−61.0× 10^-6 (Linear Decay) Batch Size 1 per GPU (Grad Accumulation = 4) Gradient Clipping 0.2 Inner Updates 2 per rollout cycle Total Steps 2000 StableDRL Specifics Loss Function Sandwiched (β=1.5,ω=0.5β=1.5,ω=0.5) ELBO Estimation 2 MC samples (perturbation p=0.15p=0.15) Stabilization SN enabled, Clip Threshold = 5.0 C.1.2 Block Diffusion (SDAR-8B-Chat) We instantiate StableDRL on the SDAR-8B-Chat architecture, following the conventions of TraceRL extended with our stability mechanisms. We utilize dynamic sampling with a threshold of τ=0.9τ=0.9 and a temperature of 1.0. The model is trained using AdamW with a learning rate of 1.0×10−61.0× 10^-6 and no weight decay. To stabilize the group-wise updates, we employ Group-wise SNIS with an asymmetric log-clipping threshold of 5.0 (log-space). We also enable mask resampling in the trainer to maintain valid drift coupling during optimization. Table 4 details the configuration for the block diffusion experiments. Table 4: Hyperparameter Configuration for Block Diffusion (SDAR-8B-Chat) Category Value Model & Initialization Base Model JetLM/SDAR-8B-Chat Architecture Block Diffusion (B=4B=4) Precision bf16 (TF32 enabled) Generation (Rollout) Sampling Strategy Dynamic (τ=0.9,T=1.0τ=0.9,T=1.0) Denoising Steps 4 per block Rollout Scale 16 responses per task Training & Optimization Optimizer AdamW (lr=1e-6,β2=0.999lr=1e-6, _2=0.999, no decay) Scheduler Linear Decay Micro Batch Size 1 (Gradient Accumulation = 2) Gradient Clipping 1.0 StableDRL Specifics Advantage Mode Raw Centered Importance Sampling Group-wise SNI Clip Threshold 5.0 (log-space) Mask Resampling Enabled C.2 Details of the Exploding Importance Weight Protocol To validate the robustness of StableDRL against the heavy-tailed noise characteristic of dLLMs, we use a controlled adversarial protocol that artificially inflates the variance of the importance ratio ρ ρ. C.3 Mechanism: Asymmetric Masking The importance ratio is estimated as ρ^=exp(ℒ^θ−ℒ^old) ρ= ( L_θ- L_old). We induce "exploding" weights by breaking the symmetry of the Monte Carlo estimation for a random 70% subset of the batch (the "stressed" samples). We employ two decoupled masking policies: 1. Numerator (ℒ^θ L_θ) → "Easy" Masking: We bias masking towards high-confidence regions (e.g., the sequence tail) and select the minimum number of masked tokens (tmint_ ). This yields a tighter, optimistic ELBO estimate. 2. Denominator (ℒ^old L_old) → "Hard" Masking: We bias masking towards low-confidence regions (e.g., the sequence head) and select the maximum number of masked tokens (tmaxt_ ). This yields a looser, pessimistic ELBO estimate. This systematic gap ensures that ℒ^θ≫ℒ^old L_θ L_old, driving ρ^→∞ ρ→∞ purely due to estimation variance, independent of the actual policy probability. C.4 Implementation We operationalize "Easy" vs. "Hard" based on the diffusion formulation (Block vs. Random Token). Algorithm 1 details the generation process. Algorithm 1 Adversarial Generation of Exploding Importance Weights 0: Batch X, Group size G, Coverage fraction γ=0.7γ=0.7 0: Bias Strength β=6.0β=6.0 (for Random), Masking Policy ∈Block,RandomP∈\Block,Random\ 1: for each group g in Batch do 2: Select indices Sg⊂gS_g⊂ g with size ⌈γ⋅G⌉ γ· G to stress. 3: for each sample xix_i in group g do 4: if i∈Sgi∈ S_g then 5: // 1. Numerator: "Easy" Masking (Tail Bias + Min Count) 6: if P is Block then 7: Mnum←M_num← Mask indices of the Last Block (Max Context) 8: else 9: W[k]∝exp(+β⋅k/L)W[k] (+β· k/L) Tail Position Bias 10: Mnum∼Multinomial(W)M_num (W) 11: Count(Mnum)←tminCount(M_num)← t_ Min Masked Tokens 12: end if 13: // 2. Denominator: "Hard" Masking (Head Bias + Max Count) 14: if P is Block then 15: Mden←M_den← Mask indices of the First Block (Min Context) 16: else 17: W[k]∝exp(−β⋅k/L)W[k] (-β· k/L) Head Position Bias 18: Mden∼Multinomial(W)M_den (W) 19: Count(Mden)←tmaxCount(M_den)← t_ Max Masked Tokens 20: end if 21: else 22: // Control: Standard Uniform Masking 23: Mnum,Mden∼UniformRandom(xi)M_num,M_den (x_i) 24: end if 25: ℒ^θ←ComputeELBO(xi,πθ,Mnum) L_θ (x_i, _θ,M_num) 26: ℒ^old←ComputeELBO(xi,πold,Mden) L_old (x_i, _old,M_den) 27: ρ^i←exp(ℒ^θ−ℒ^old) ρ_i← ( L_θ- L_old) 28: end for 29: end for 30: return Importance Weights ρ ρ C.5 Visual Diagnosis of Gradient Instability To empirically validate the “Instability Feedback Loop” and the structural failures diagnosed in Section 3.1, we visualize the joint distribution of importance weights (log10ρ _10ρ) and gradient norms (log10‖g^‖ _10\| g\|) recorded during training. Figure 8 presents a comparative diagnostic of ESPO, SPG-IS, and StableDRL, offering a direct geometric validation of our theoretical analysis. The “Chimney” Failure in ESPO. As observed in the left panel, ESPO exhibits a pathological “chimney” distribution. While the majority of samples cluster in a low-variance region, a sparse subset of noise-induced outliers (importance weights ρ>106ρ>10^6) drives gradient norms to catastrophic levels (‖g^‖>104\| g\|>10^4). This empirically confirms Failure 1 (Asymmetric Failure of the Clipped Surrogate) described in Section 3.1: when a sample with a large noise-induced importance weight has a negative advantage, it falls into the unclipped branch of the objective. Consequently, these “trapdoor” outliers bypass the trust region and act as unbounded multipliers on the step size, injecting massive shocks that destabilize the policy. Drift-Variance Correlation in SPG-IS. The center panel displays the dynamics of SPG-IS. Although SPG avoids explicit ratio computation to mitigate the “chimney” effect, the visualization reveals a strong positive correlation between the implicit weight magnitude and the update norm. This indicates that the method remains sensitive to policy drift: as the target policy diverges from the behavior policy, the accumulated “rollout-reuse bias” scales the variance of the updates proportionally. This prevents convergence, as the method lacks the structural constraints to mechanically decouple the update magnitude from distribution shifts. Geometric Stability in StableDRL. In contrast, the right panel demonstrates the efficacy of our proposed framework. StableDRL displays a compact, bounded distribution where gradient norms remain consistently low (<101.8<10^1.8) regardless of the importance weight magnitude. This confirms the effect of our dual stability mechanisms: Unconditional Clipping strictly censors extreme ratios before aggregation, while Self-Normalization ensures the update remains a convex combination of per-sample gradients. As predicted by Theorem 3.1, StableDRL effectively confines the update to the convex hull of the samples, maintaining deterministic stability even in the presence of heavy-tailed proxy noise. Figure 8: Diagnosing Gradient Instability in dLLM Training. We visualize the joint distribution of importance weights (log10ρ _10ρ) and gradient norms (log10‖g^‖ _10\| g\|) during training. (Left) ESPO: Exhibits a characteristic “chimney” failure where rare, noise-induced outliers bypass clipping on negative advantages, acting as unbounded step-size multipliers that drive gradients to explosion (>104>10^4). (Center) SPG-IS: Despite avoiding explicit ratios, the update variance is strongly correlated with policy drift, confirming that rollout-reuse bias accumulates to destabilize training. (Right) StableDRL (Ours): By enforcing strict clipping and self-normalization, our method decouples update magnitude from proxy noise, confining gradients to the convex hull of the samples (Theorem 3.1) and maintaining deterministic stability.