Paper deep dive
Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models
Yue YU, Bowen Zuo, David Crandall, Yinglun Zhu, Dongruo Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/2/2026, 9:26:12 AM
Summary
The paper introduces StepRS-GRPO, a stepwise risk-sensitive Group Relative Policy Optimization method for Diffusion Large Language Models (dLLMs). It varies the risk coefficient of the group-advantage transformation across denoising states to improve mathematical reasoning accuracy, solution coverage, and answer diversity compared to centered GRPO and constant-risk RS-GRPO.
Entities (9)
Relation Signals (6)
StepRS-GRPO → isappliedto → Diffusion Large Language Models
confidence 95% · We propose stepwise risk-sensitive GRPO (StepRS-GRPO)... for Diffusion Language Models
StepRS-GRPO → varies → risk coefficient
confidence 95% · StepRS-GRPO... varies the risk coefficient of the group-advantage transformation across denoising states
StepRS-GRPO → improves → Pass@1 Accuracy
confidence 90% · StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO
StepRS-GRPO → increases → Answer Diversity
confidence 90% · StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity.
SDAR-1.7B-Chat → isevaluatedon → MATH
confidence 90% · We conduct the main experiments using... SDAR-1.7B-Chat... we use the MATH training split
SDAR-4B-Chat → isevaluatedon → MATH
confidence 90% · We conduct the main experiments using... SDAR-4B-Chat... we use the MATH training split
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback across these decisions, even as their conditioning context changes. We propose stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer. For binary rewards, we show that this transformation is exactly a prompt- and state-dependent rescaling of centered outcome advantages. A capability-based calibration suggests a coefficient scale, while endpoint and interpolation ablations guide schedule selection. Across multiple dLLM backbones and mathematical reasoning benchmarks, StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity. In our ablation studies, mass-matched controls support the contributions of state allocation and schedule direction, and the gains persist after matching the root mean square (RMS) of the advantages to that of centered GRPO. Reasoning-trace diagnostics further show that the diversity gains from StepRS-GRPO extend beyond final-answer strings.
Tags
Links
- Source: https://arxiv.org/abs/2610.00661v1
- Canonical: https://arxiv.org/abs/2610.00661v1
Trouble viewing inline? Open PDF directly →
Full Text
93,847 characters extracted from source content.
Expand or collapse full text
Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models Yue Yu †thanks: Contributed equally to this work. Emails: yyu3@iu.edu and bowen.zuo@email.ucr.edu. Bowen Zuo11footnotemark: 1 Affiliation: Indiana University University of California, Riverside David Crandall Yinglun Zhu Affiliation: Indiana University University of California, Riverside Dongruo Zhou Abstract Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback across these decisions, even as their conditioning context changes. We propose stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer. For binary rewards, we show that this transformation is exactly a prompt- and state-dependent rescaling of centered outcome advantages. A capability-based calibration suggests a coefficient scale, while endpoint and interpolation ablations guide schedule selection. Across multiple dLLM backbones and mathematical reasoning benchmarks, StepRS-GRPO improves both pass@11 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity. In our ablation studies, mass-matched controls support the contributions of state allocation and schedule direction, and the gains persist after matching the root mean square (RMS) of the advantages to that of centered GRPO. Reasoning-trace diagnostics further show that the diversity gains from StepRS-GRPO extend beyond final-answer strings. Keywords: Diffusion Language Models, Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization, Risk-Sensitive Reinforcement Learning, Reasoning Diversity 1 Introduction Diffusion large language models (dLLMs) (Nie et al., 2025; Sahoo et al., 2024) change the structure of language-model generation. Rather than generating tokens strictly from left to right, a dLLM starts from a masked sequence and gradually denoises it through T reverse steps, where each step may unmask multiple positions using bidirectional context. Recent work has adapted supervised post-training to this masked-denoising paradigm, either by training instruction-following masked dLLMs directly (Zhao et al., 2025a; Nie et al., 2025) or by converting strong autoregressive backbones into bidirectional masked dLLMs through supervised distillation (Wang et al., 2025c). Building on these supervised post-training recipes, a natural next step is to use reinforcement learning with verifiable rewards (RLVR) to improve dLLM reasoning. In autoregressive language models, RLVR methods such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024) have become effective tools for mathematical reasoning because they directly optimize verifiable outcome rewards without requiring dense human annotations or a learned value model. Recent work has begun to transfer GRPO-style training to dLLMs by adapting policy-gradient updates to the masked-denoising process (Zhu et al., 2025a). Terminal-reward policy gradients can reuse a rollout-level advantage across multiple decisions in both autoregressive and diffusion models. In a dLLM, the conditioning context and masked positions also change across denoising states, making state-dependent weighting a useful design space to study. A higher pass@11 does not by itself establish better coverage at larger sampling budgets, and neither quantity establishes diversity of correct reasoning strategies. We therefore evaluate both accuracy and sampled-solution coverage, with constant-risk training as a direct comparator for the contribution of scheduling. Thus, the central question is not only how to make dLLMs more accurate under RLVR, but how to improve their reasoning accuracy without collapsing their search distribution: Can we design RLVR methods for dLLMs that improve reasoning accuracy while preserving answer coverage? We address this question with stepwise risk-sensitive GRPO (StepRS-GRPO). Building on constant-risk RS-GRPO (Jiang et al., 2026), we vary the coefficient of the entropic advantage transformation across denoising states. Under binary rewards, this is exactly a prompt- and state-dependent rescaling of centered outcome advantages. The method uses terminal rewards alone and retains the chosen trainer’s likelihood estimator and loss reduction. The analysis explains the reweighting, and the controls test the benefit of assigning different weights to different states. Our contributions are as follows. 1. We introduce a denoising-state-dependent extension of RS-GRPO for dLLM post-training and establish its exact finite-group binary-reward factorization as prompt- and state-dependent terminal-advantage reweighting. 2. We develop a capability-guided starting point for coefficient selection and evaluate the endpoint range and interpolation shape separately, supporting linear half-decay as a practical default. 3. We demonstrate gains in accuracy, solution coverage, and output diversity across dLLM backbones. Repeated inference evaluations and targeted controls examine state allocation and advantage scale, complemented by schedule ablations and a masked-dLLM case study. 2 Related Work Reinforcement learning for diffusion language models. dLLM post-training requires policy optimization under non-autoregressive denoising. d1 combines masked supervised fine-tuning with diffu-GRPO (Zhao et al., 2025a); TraceRL optimizes inference traces (Wang et al., 2025c); StableDRL uses unconditional clipping and self-normalization (Zhong et al., 2026); and dTRPO reduces trajectory probability-estimation cost (Zhang et al., 2026). Likelihood-estimation approaches include GDPO, ESPO, AGRPO, SPG, and BGPO (Rojas et al., 2025; Ou et al., 2025; Zhan, 2025; Wang et al., 2025b; Lin et al., 2025). The ratio-free wd1 objective uses weighted log-likelihood, and wd1++ extends it to denoising-stepwise optimization (Tang et al., 2025). DMPO instead matches a reward-tilted target through weighted denoising cross-entropy (Zhu et al., 2025b). StepRS complements these methods with state-dependent coefficients in an entropic group-advantage transformation, retaining the selected trainer’s likelihood estimator and loss construction. Entropy collapse and diversity in RL post-training. Diversity-aware RLVR includes entropy analyses (Cui et al., 2025), divergence-based methods (Li et al., 2025a), explicit diversity rewards (Li et al., 2025b), and adaptive entropy targets (Wang et al., 2025a). Pass@k policy optimization directly targets multiple-sample success (Walder and Karkhanis, 2025), while advantage-shaping analyses connect modified GRPO updates to surrogate reward objectives (Thrampoulidis et al., 2026). JustGRPO examines a different source of coverage loss: it uses left-to-right RL rollouts while retaining parallel decoding at inference (Ni et al., 2026). State-aware methods also differ in their feedback: DiSPO branches from intermediate masked states and scores completions with outcome rewards (Oba et al., 2026), whereas IGPO uses inpainting guidance (Zhao et al., 2025b). StepRS retains the selected rollout procedure and changes the state-dependent weights attached to terminal rewards. Risk-sensitive post-training and capability calibration. Risk-sensitive control includes entropic objectives (Howard and Matheson, 1972; Whittle, 1990; Fei et al., 2020); RS-GRPO applies risk seeking to autoregressive RLVR (Jiang et al., 2026), and RiskPO studies risk-based policy objectives (Ren et al., 2025). Recent GRPO theory separates the effects of reward normalization and reference- versus previous-policy KL regularization (Mroueh, 2026). Our finite-group analysis characterizes the entropic transformation supplied to the trainer, while exponential tilting gives its coefficient an interpretable scale. We use this scale to guide endpoint selection and evaluate the schedule empirically, complementing coefficient selection in entropy-regularized RL (Haarnoja et al., 2018a; Haarnoja et al., 2018b) and exponential-tilting methods (Rubinstein, 1999; Glasserman, 2004). Algorithm 1 StepRS-GRPO for dLLM post-training 0: Policy πθ _θ, frozen reference πref _ref, verifier R, training prompts D, group size G, update count NupdN_upd, schedule map b, advantage cap C, regularization weight λ, and the native trainer. 1: for update n=1,…,Nupdn=1,…,N_upd do 2: Freeze the rollout policy πold←πθ _old← _θ and sample a prompt minibatch ℬB. 3: For each q∈ℬq , draw G trajectories from πold _old and record active actions qJ_q and coordinates uqju_qj. 4: Evaluate rqi=R(xqi,0,q)r_qi=R(x_qi,0,q) and compute p^q=G−1∑irqi p_q=G^-1 _ir_qi. 5: for each active action j∈qj _q, for each q∈ℬq do 6: Set βqj=b(uqj) _qj=b(u_qj) and compute AqjStepA^Step_qj by Equation (3.3). 7: Set Bqj=clip(AqjStep,−C,C)B_qj=clip(A^Step_qj,-C,C) and treat it as constant during differentiation. 8: end for 9: Update θ with ℒnative(θ,B)L_native(θ;B), retaining the trainer’s ratio estimation, clipping, normalization, and optimizer. 10: end for 3 Method: From GRPO to StepRS-GRPO We develop StepRS-GRPO by first specifying how a terminal reward enters a diffusion-model update, then introducing risk-sensitive weighting, and finally allowing that weighting to vary across denoising states. The rollout procedure, verifier, likelihood estimator, loss reduction, optimizer, and reference regularization remain unchanged. 3.1 Diffusion language models For a prompt q, a dLLM follows reverse denoising states sT,sT−1,…,s0s_T,s_T-1,…,s_0, where T is the number of transitions. A state contains the partially revealed text and the decoder information needed for the next transition. Let out(s0)=x0out(s_0)=x_0 extract the final text. Conditional on the initialized state sTs_T, its distribution is πθ(x0∣q,sT)=∑s0,…,sT−1:out(s0)=x0∏t=1Tπθ(st−1∣st,q,t). _θ(x_0 q,s_T)= _ subarraycs_0,…,s_T-1:\\ out(s_0)=x_0 subarray _t=1^T _θ(s_t-1 s_t,q,t). Here θ denotes the model parameters. The sum includes every state trajectory ending in the same output, including any terminal decoder metadata. Deterministic initialization makes this also the distribution given q; otherwise, we average over sTs_T. For a trajectory τ, we write x0(τ)x_0(τ) for this extracted text. Masked dLLMs use bidirectional context across a partially masked output buffer (Sahoo et al., 2024; Nie et al., 2025). Block dLLMs instead generate blocks from left to right and denoise within each block (Arriola et al., 2025; Cheng et al., 2025). Our SDAR experiments use block length ℓblk=4 _blk=4; Appendix A details both factorizations. In either family, the information available to the model changes as tokens are revealed, even though the verifier evaluates only the completed answer. To describe the training decisions uniformly, write j=(i,t,ℓ)j=(i,t, ) for an active policy-loss decision in response i, at reverse step t and token position ℓ . The trainer determines the active set qJ_q, and i(j)i(j) identifies the response containing j. We associate each decision with a normalized state coordinate uqj∈[0,1]u_qj∈[0,1], where larger values mean more-masked states. This distinguishes maskedness from global trajectory time: in block diffusion, progress within a block and the order of output blocks need not follow the same coordinate. 3.2 From centered advantages to state-dependent risk Outcome feedback and centered GRPO. For each prompt in a minibatch ℬB, a frozen snapshot πold _old of the current policy samples G response trajectories. The verifier returns rqi=R(xqi,0,q)∈0,1r_qi=R(x_qi,0,q)∈\0,1\, with 11 indicating a correct answer. The empirical solve fraction and the centered advantages are p^q=1G∑i=1Grqi,Aqi0=rqi−p^q,aqj=Aq,i(j)0. p_q= 1G _i=1^Gr_qi, A^0_qi=r_qi- p_q, a_qj=A^0_q,i(j). Thus, every active decision in a response receives the same outcome signal: 1−p^q1- p_q for a correct response and −p^q- p_q for an incorrect one. We use this mean-centered variant of GRPO, without dividing by the group’s reward standard deviation (Shao et al., 2024). The underlying policy loss. For a nonempty prompt minibatch ℬB and advantage array A=(Aqj)A=(A_qj), write the trainer’s loss as ℒnative(θ;A)=−1|ℬ|∑q∈ℬ∑j∈qvqjΨqjnative(θ;A)+λref(θ).L_native(θ;A)=- 1|B| _q _j _qv_qj ^native_qj(θ;A)+ _ref(θ). (3.1) The positive factors vqjv_qj specify fixed loss-reduction weights, including response or token averaging; they are not risk coefficients. The reward-surrogate contribution Ψqjnative ^native_qj may depend on other sampled entries through the trainer’s normalization, so its argument is the full array A. The sampled data and fixed reduction weights are held constant during differentiation. The term refK_ref is the trainer’s KL-to-reference penalty or its estimator, with weight λ≥0λ≥ 0. The fixed reference πref _ref provides the regularization target; πold _old is refreshed to collect rollouts. For example, PPO uses minρqjAqj,clip(ρqj,1−ϵ,1+ϵ)Aqj \ _qjA_qj,clip( _qj,1-ε,1+ε)A_qj\, where ρqj _qj is the current-to-rollout policy ratio for the decision and ϵ>0ε>0 is its clipping threshold (Schulman et al., 2017). Diffusion trainers may estimate this ratio and use different stabilizing operations; StableDRL uses unconditional ratio clipping and self-normalization (Zhong et al., 2026). Equation (3.1) retains those native choices rather than replacing them with the PPO example. Our centered baseline is ℒGRPO(θ)=ℒnative(θ,a)L_GRPO(θ)=L_native(θ;a). RS-GRPO with a constant coefficient. RS-GRPO (Jiang et al., 2026) transforms the group’s rewards using a risk coefficient β>0β>0: Aqi(β)=1β[exp(βrqi)G−1∑h=1Gexp(βrqh)−1],Aqi(0)=Aqi0.A_qi(β)= 1β [ (β r_qi)G^-1 _h=1^G (β r_qh)-1 ], A_qi(0)=A^0_qi. (3.2) The denominator averages exponentiated rewards over the same prompt group; h indexes its responses. Subtracting one centers these normalized weights, and the factor 1/β1/β recovers the centered advantage continuously as β→0β→ 0. A constant coefficient assigns Aq,i(j)(β)A_q,i(j)(β) to every decision in response i(j)i(j), modifying its weight without changing the observed reward. Before applying the native loss, we retain the common advantage cap clip(z,−C,C)clip(z,-C,C), where clip(z,l,h)=minh,maxl,zclip(z,l,h)= \h, \l,z\\ and C=10C=10. This operation is separate from policy-ratio clipping. The centered advantages already lie in [−1,1][-1,1], so this cap leaves the baseline unchanged. StepRS: a coefficient for each denoising state. A constant coefficient uses the same reward transformation throughout a response. StepRS-GRPO instead evaluates a fixed schedule map b:[0,1]→[0,∞)b:[0,1]→[0,∞) at each decision’s state coordinate: βqj=b(uqj),AqjStep=Aq,i(j)(βqj),Bqj=clip(AqjStep,−C,C). _qj=b(u_qj), A^Step_qj=A_q,i(j)( _qj), B_qj=clip(A^Step_qj,-C,C). (3.3) Here BqjB_qj is the capped advantage supplied to the trainer. For a given action j, its coefficient βqj _qj is applied to every reward in the denominator of Equation (3.2); coefficients from different actions are not mixed within that denominator. The same terminal feedback can therefore receive different state-dependent weights. The resulting objective is ℒStepRS(θ)=ℒnative(θ,B),L_StepRS(θ)=L_native(θ;B), (3.4) where B=(Bqj)B=(B_qj) collects the capped advantages over the minibatch. Rewards, recorded coordinates, and constructed advantages are held fixed during differentiation. The schedule changes how outcome feedback is allocated, not how intermediate reasoning is scored. Setting b≡0b≡ 0 recovers centered GRPO; choosing a positive constant recovers capped constant-risk RS-GRPO. Algorithm 1 gives the training procedure. 3.3 Binary-reward interpretation and schedule selection The binary-reward setting makes the transformation particularly transparent. We first establish what it changes within a sampled group, then use exponential tilting to interpret the coefficient’s scale. Here p^q p_q is the empirical solve fraction in one group, while pπ(q)=Prτ∼π(⋅∣q)[R(x0(τ),q)=1]p_π(q)= _τ π(· q)[R(x_0(τ),q)=1] is the success probability of a fixed rollout distribution π; x0(τ)x_0(τ) is the completed answer in trajectory τ. Lemma 3.1 (Finite-group factorization and success-odds shift). For every realized binary-reward group and β>0β>0, Aqi(β)=cβ(p^q)Aqi0,cβ(p)=eβ−1β(1−p+peβ),0≤p≤1.A_qi(β)=c_β( p_q)A^0_qi, c_β(p)= e^β-1β(1-p+pe^β), 0≤ p≤ 1. (3.5) The identity extends continuously to β=0β=0 with c0(p)=1c_0(p)=1. Separately, let p=pπ(q)p=p_π(q) and define the exponentially tilted rollout distribution πβ(τ∣q)=π(τ∣q)expβR(x0(τ),q)1−p+peβ. _β(τ q)= π(τ q) \β R(x_0(τ),q)\1-p+pe^β. (3.6) Its success probability is pβ=peβ/(1−p+peβ)p_β=pe^β/(1-p+pe^β), and, for 0<p<10<p<1, logpβ1−pβ=logp1−p+β. p_β1-p_β= p1-p+β. (3.7) The finite-group identity follows directly from G−1∑ieβrqi=1−p^q+p^qeβG^-1 _ie^β r_qi=1- p_q+ p_qe^β; Appendix B.1 provides the full proof. No population approximation is required, and homogeneous reward groups give zero advantages. What risk sensitivity changes. The multiplier cβ(p)c_β(p) is positive, so it preserves advantage signs and scales both reward classes by the same factor at a fixed coefficient. For β>0β>0, it decreases with p: groups with fewer successes receive larger multipliers than groups with more successes at that coefficient. Increasing β need not increase advantage magnitude at every solve rate. In StepRS, the uncapped advantage is AqjStep=wqjaqjA^Step_qj=w_qja_qj, where wqj=cb(uqj)(p^q)w_qj=c_b(u_qj)( p_q). Thus aqja_qj supplies the outcome signal, wqjw_qj supplies the additional risk weighting, and vqjv_qj retains the trainer’s loss reduction. This separation motivates controls that match either multiplier mass or the root mean square (RMS) of the advantages (see Section 4.3). Capability-guided coefficient selection. Under the mathematical tilt, increasing β multiplies the success-to-failure odds by eβe^β, or equivalently reduces the failure-to-success odds by e−βe^-β. We use this interpretation to choose a starting scale, without identifying the tilted distribution with the result of an optimizer update. Let πinit _init be the initial policy and calD_cal a calibration prompt distribution. Its average solve probability is p¯=q∼cal[pπinit(q)] p=E_q _cal[p_ _init(q)]. Matching the fractional reduction in failure odds, 1−e−β1-e^-β, to this solve-rate proxy gives β⋆(p¯)=−log(1−p¯),0≤p¯<1.β ( p)=- (1- p), 0≤ p<1. (3.8) This calibration supplies a candidate scale for endpoint selection, rather than deriving an optimal coefficient from the training loss. We evaluate nearby choices and hold the selected endpoints fixed during training. For SDAR-4B, p¯≈0.66 p≈ 0.66 gives β⋆≈1.08β ≈ 1.08; Appendix D.2 compares the rounded calibrated schedule with neighboring settings. From endpoints to a denoising-state schedule. With βmax≥βmin≥0 _ ≥ _ ≥ 0, the linear schedule is b(u)=βmin+(βmax−βmin)u,βmin=γβmax,0≤γ≤1.b(u)= _ +( _ - _ )u, _ =γ _ , 0≤γ≤ 1. (3.9) It assigns βmax _ at u=1u=1 and βmin _ at u=0u=0. The endpoint ratio γ controls the decay: γ=0γ=0 gives full decay and γ=1γ=1 a constant coefficient. We use half-decay, γ=1/2γ=1/2, with βmax=1.0 _ =1.0 for SDAR-1.7B and 1.51.5 for SDAR-4B. The notation [βmax→βmin][ _ → _ ] follows decreasing maskedness, not optimizer time. The ablations test endpoint selection, schedule direction, and interpolation separately. 4 Experiments In this section, we present the implementation of our RL post-training algorithm for dLLMs and empirically evaluate the resulting models. Due to space limitations, further experimental settings and implementation details are deferred to Appendix C. 4.1 Setup Models and training. We conduct the main experiments using two block diffusion language models from the SDAR-Chat family (Cheng et al., 2025). SDAR-1.7B-Chat is a 1.7B-parameter block dLLM, while SDAR-4B-Chat is a 4B-parameter block dLLM from the same architectural family. Both models are trained under identical settings: we use the MATH training split restricted to difficulty levels 3–5, comprising 7,500 distinct problems, and perform RL post-training with the StableDRL framework (Zhong et al., 2026), which is specifically designed for stable RL post-training of dLLMs. Training is conducted for 160160 optimizer steps on NVIDIA H100 GPUs. For both models, we use G=16G=16 rollouts per prompt and binary math-correctness rewards. An additional case study on masked dLLMs is provided in Appendix C.1. Full hyperparameter details are reported in Appendix C. Benchmarks and metrics. We evaluate mathematical reasoning on MATH500 (Hendrycks et al., 2021), AMC23 (Mathematical Association of America, 2023), AIME2024 (Mathematical Association of America, 2024), and AIME2025 (Mathematical Association of America, 2025), covering both standard and competition-style math problems. Pass@k Accuracy. For each problem q, we draw N=64N=64 independent samples with sampling temperature Tsample=1.0T_sample=1.0. Correctness is determined using the same final-answer extraction, canonicalization, and verification pipeline across all methods. Let c(q)c(q) be the number of correct samples among the N generations. We estimate pass@k using the unbiased estimator of Chen et al. (2021): pass@k^(q)= 1−(N−c(q)k)(Nk), pass@k(q)\;=\;1- N-c(q)k Nk, (4.1) with (N−c(q)k)=0 N-c(q)k=0 when N−c(q)<kN-c(q)<k. We report pass@k by averaging pass@k^(q) pass@k(q) over all problems in each benchmark. Baselines and repeated evaluation. We compare the pre-RL model, centered GRPO, constant-risk RS-GRPO, and StepRS-GRPO under a common within-backbone trainer, optimizer, verifier, and KL coefficient λ=0.01λ=0.01. The scheduled defaults are [1.0→0.5][1.0→ 0.5] for SDAR-1.7B, [1.5→0.75][1.5→ 0.75] for SDAR-4B, and [0.5→0.25][0.5→ 0.25] for the masked-model case study. Each configuration in the three-seed evaluation is evaluated with inference seeds 4242, 4343, and 4444 at k∈1,2,4,8,16,32,64k∈\1,2,4,8,16,32,64\. Curves show means with capped error bars for one sample standard deviation across fixed-checkpoint evaluations. We also report the unique-answer ratio U¯ U, defined in Appendix C.3. The mechanism controls are described in Section 4.3. 4.2 Main Results Figure 1 compares Base, GRPO, and StepRS-GRPO on the two SDAR backbones. The selected schedules are [1.0→0.5][1.0→ 0.5] for SDAR-1.7B and [1.5→0.75][1.5→ 0.75] for SDAR-4B. Figure 1: Pass@k on SDAR-1.7B-Chat and SDAR-4B-Chat at the 160160-update checkpoints. Markers show means and capped error bars show one sample standard deviation across inference seeds 4242, 4343, and 4444, with 6464 responses per problem per seed. The selected StepRS schedules are [1.0→0.5][1.0→ 0.5] and [1.5→0.75][1.5→ 0.75], respectively. Answer diversity is reported in Figure 12. Accuracy and solution coverage. StepRS-GRPO achieves higher mean pass@k than GRPO at every evaluated budget on all eight SDAR model–benchmark pairs. At 1.7B, mean pass@11 increases from 59.2059.20 to 61.3361.33 on MATH500, 33.8333.83 to 40.6040.60 on AMC23, 3.903.90 to 9.609.60 on AIME2024, and 2.402.40 to 8.308.30 on AIME2025. The corresponding AIME pass@6464 gains are 10.0010.00 and 6.686.68 percentage points. At 4B, pass@11 improves from 73.6373.63 to 75.7775.77, 47.3347.33 to 50.6750.67, 11.1711.17 to 12.3312.33, and 11.3311.33 to 13.2013.20 across the same four benchmarks. The AIME2024 and AIME2025 gains extend from 1.171.17 and 1.871.87 points at k=1k=1 to 8.898.89 and 10.0110.01 points at k=64k=64, combining improved individual-answer accuracy with broader sampled-solution coverage. Output diversity and portability. The coverage gains accompany higher unique-answer ratios on every SDAR model–benchmark pair. On MATH500, for example, U¯ U increases from 19.2019.20 to 28.0028.00 at 1.7B and from 12.2712.27 to 18.6018.60 at 4B. The masked-dLLM case study shows the same ordering over GRPO across all seven budgets on GSM8K, MATH500, and AMC23 (Appendix C.1). Answer-removed trace diagnostics provide complementary evidence that the increased output variation extends beyond the final answers (Figure 3). 4.3 Ablation Studies We use SDAR-4B to distinguish the contributions of risk sensitivity, denoising-state allocation, and overall advantage scale. Figure 2 compares all seven sampling budgets under the common setup in Section 4.1. Figure 2: SDAR-4B mechanism ablations. Top: StepRS versus prompt-only mass matching (C1) and a mass-normalized reversed schedule (C2). Bottom: GRPO, constant-risk RS-GRPO (β=1.5β=1.5), StepRS, and advantage-RMS-matched StepRS (C3). Markers show three-inference-seed means; capped error bars show one sample standard deviation. Controls are defined in the text, with implementation details in Appendix D.1. Scheduling versus constant risk. Constant-risk RS-GRPO with β=1.5β=1.5 is a strong comparator. StepRS-GRPO matches or improves its mean pass@k at every benchmark–budget combination, with strict gains throughout AMC23 and both AIME benchmarks. On these three benchmarks, the pass@11 gains are 0.200.20, 0.400.40, and 0.500.50 percentage points, and the pass@6464 gains are 0.830.83, 1.101.10, and 2.242.24 points. The full constant-risk sweep is shown in Appendix D.3. Common notation for the controls. For prompt q, qJ_q contains the active token decisions and aqj=rq,i(j)−p^qa_qj=r_q,i(j)- p_q is their centered outcome advantage. The SDAR-4B reference uses b(u)=0.75+0.75ub(u)=0.75+0.75u, wqj=cb(uqj)(p^q)w_qj=c_b(u_qj)( p_q), and Bqj=clip(wqjaqj,−10,10)B_qj=clip(w_qja_qj,-10,10). The weights vqj>0v_qj>0 are the fixed reductions in Equation (3.1). Below, multiplier “mass” means ∑jvqjwqj _jv_qjw_qj, not probability mass or gradient magnitude. The controls compute their matching quantities from their own current rollouts using this reference rule. Prompts with no active decisions contribute no reward-loss term. Control 1: remove state variation, retain average weight. C1 asks whether prompt-dependent weighting is sufficient. For each prompt, we replace all state-dependent multipliers by their loss-weighted mean: AqjC1=w¯qaqj,w¯q=∑j∈qvqjwqj∑j∈qvqj.A^C1_qj= w_qa_qj, w_q= _j _qv_qjw_qj _j _qv_qj. (4.2) This is the unique constant preserving the prompt’s total multiplier mass, ∑jvqjw¯q=∑jvqjwqj _jv_qj w_q= _jv_qjw_qj, before applying the common cap. It also minimizes the weighted squared deviation from the original multipliers, giving C1 a least-squares interpretation (Boyd and Vandenberghe, 2018); the derivation is in Appendix D.1. StepRS has higher mean pass@k in all comparisons, supporting state-dependent allocation beyond this prompt-only weighting. Control 2: reverse the schedule at matched weight mass. C2 assigns the large coefficient to less-masked states by evaluating b(1−u)b(1-u) instead of b(u)b(u). Write wˇqj=cb(1−uqj)(p^q) w_qj=c_b(1-u_qj)( p_q) and use AqjC2=κqwˇqjaqj,κq=∑j∈qvqjwqj∑j∈qvqjwˇqj.A^C2_qj= _q w_qja_qj, _q= _j _qv_qjw_qj _j _qv_qj w_qj. (4.3) The positive factor κq _q preserves the reference multiplier mass before the common cap; reversal alone need not do so because cβc_β is nonlinear. StepRS is higher in all 2828 comparisons, with pass@6464 gains of 5.585.58 and 5.595.59 points on AIME2024 and AIME2025, favoring the proposed coefficient assignment. Control 3: retain the pattern, match advantage magnitude. C3 tests whether StepRS remains effective after matching its overall advantage magnitude to centered GRPO. We measure this magnitude by RMSℬRMS_B, the square root of the average squared advantage over all active entries in one optimizer update (Appendix D.1). With the centered advantages a=(aqj)a=(a_qj) and capped StepRS advantages B=(Bqj)B=(B_qj) computed on the same actions, C3 supplies BqjC3=ηℬBqj,ηℬ=stopgrad(RMSℬ(a)RMSℬ(B)).B^C3_qj= _BB_qj, _B=stopgrad\! ( RMS_B(a)RMS_B(B) ). (4.4) Here stopgradstopgrad keeps the scalar’s value but prevents backpropagation through it. When both RMS values are zero, we set the scaled advantages to zero without forming the ratio. Otherwise, the positive scalar preserves the signs and relative ratios of the capped StepRS advantages. Unlike mean-and-standard-deviation normalization (Andrychowicz et al., 2021), C3 does not recenter them. C3 stays within 0.240.24 points of StepRS on pass@11, and both variants exceed GRPO in all comparisons. The gains therefore persist when the overall advantage RMS is matched to centered GRPO. Calibration-guided endpoint selection. For SDAR-4B, the solve-rate proxy p¯≈0.66 p≈ 0.66 gives β⋆≈1.08β ≈ 1.08. The rounded calibrated schedule [1.08→0.54][1.08→ 0.54] improves on [1.0→0.5][1.0→ 0.5] at every benchmark–budget combination and is within 0.470.47 percentage points of the selected [1.5→0.75][1.5→ 0.75] schedule on pass@11. This supports calibration as a useful starting point for endpoint selection. The complete endpoint sweep is reported in Appendix D.2. Figure 3: Answer-removed trace diversity on SDAR-4B: (a) DngramD_ngram; (b) DembD_emb. Points and capped bars show means and sample standard deviations across three inference seeds, with 6464 responses per question per seed. Higher scores indicate greater variation; panel scales differ. Definitions are in Appendix C.3. Schedule shape at fixed endpoints. To separate interpolation from the coefficient range, we fix the endpoints to [1.5→0.75][1.5→ 0.75] and compare linear, cosine, and two-stage schedules (Figure 4). Linear and cosine differ by at most 0.470.47 percentage points in mean pass@k across all four benchmarks and seven budgets, showing that performance is stable across these two smooth interpolations. Both exceed the tested two-stage schedule at every benchmark–budget combination. For linear interpolation, the pass@6464 gains are 0.870.87, 5.005.00, 3.343.34, and 4.494.49 points on MATH500, AMC23, AIME2024, and AIME2025, respectively. Together with the reversed-schedule control, these results support the proposed schedule direction and smooth variation between the endpoints. We retain linear half-decay as a simple, effective default. Figure 4: Schedule-shape ablation on SDAR-4B with fixed endpoints [1.5→0.75][1.5→ 0.75]. Linear, cosine, and two-stage schedules are compared at all seven sampling budgets. Markers show means and capped error bars show one sample standard deviation across three inference seeds. Reasoning-trace diversity. We remove the final boxed answer and its surrounding sentence before measuring the remaining reasoning text. N-gram variation uses Dngram=100(1−Self-BLEU-4)D_ngram=100(1-Self -BLEU-4): each trace is compared with the other traces for the same question (Zhu et al., 2018). Embedding variation, DembD_emb, is 100100 times their mean pairwise cosine distance, using the Sentence Transformers checkpoint all-MiniLM-L6-v2 (Reimers and Gurevych, 2019; Sentence Transformers, 2021). In Figure 3, StepRS exceeds GRPO on all four benchmarks under both measures, with gains of 2.32.3–6.66.6 and 1.31.3–3.53.5 points, respectively. These all-trace diagnostics show that the increased variation extends beyond final-answer strings; Appendix C.3 defines both metrics in detail. 5 Conclusion and Limitations We introduced StepRS-GRPO, a denoising-state-dependent extension of risk-sensitive GRPO for diffusion language models. Our experiments show improvements in mathematical accuracy, sampled-solution coverage, and output diversity. Ablation studies support the contributions of state allocation and schedule direction beyond prompt-level weighting, and the advantage-RMS-matched variant retains the gains over GRPO. Calibration and schedule-shape experiments further provide practical guidance for selecting effective coefficients. The capability rule guides the selection of schedule endpoints, but does not establish an optimal schedule. Our evaluation also focuses on mathematical tasks with binary rewards, and the three inference seeds quantify sampling variability at fixed checkpoints rather than variability across training runs. Additional training replications and evaluation under broader reward settings would help establish how consistently these gains hold beyond the settings studied here. References Andrychowicz et al. (2021) M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem What matters in on-policy reinforcement learning? a large-scale empirical study. In International Conference on Learning Representations, External Links: Link Cited by: §D.1, §4.3. Arriola et al. (2025) M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. External Links: 2503.09573 Cited by: §A.2, §3.1. Austin et al. (2021) J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, Cited by: §A.1. Boyd and Vandenberghe (2018) S. Boyd and L. Vandenberghe Introduction to applied linear algebra: vectors, matrices, and least squares. Cambridge University Press. External Links: Document, Link Cited by: §D.1, §4.3. Chen and Cherry (2014) B. Chen and C. Cherry A systematic comparison of smoothing techniques for sentence-level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, p. 362–367. External Links: Document, Link Cited by: §C.3. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §C.3, §4.1. Cheng et al. (2025) S. Cheng, Y. Bian, D. Liu, L. Zhang, Q. Yao, Z. Tian, W. Wang, Q. Guo, K. Chen, B. Qi, et al. Sdar: a synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303. Cited by: §A.2, §3.1, §4.1. Cui et al. (2025) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, Z. Liu, H. Peng, L. Bai, W. Ouyang, Y. Cheng, B. Zhou, and N. Ding The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §2. Fei et al. (2020) Y. Fei, Z. Yang, Y. Chen, Z. Wang, and Q. Xie Risk-sensitive reinforcement learning: near-optimal risk-sample tradeoff in regret. In Advances in Neural Information Processing Systems, Vol. 33. Cited by: §2. Glasserman (2004) P. Glasserman Monte carlo methods in financial engineering. Springer. Cited by: §2. Haarnoja et al. (2018a) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, Cited by: §2. Haarnoja et al. (2018b) T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. Cited by: §2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874. Cited by: §4.1. Howard and Matheson (1972) R. A. Howard and J. E. Matheson Risk-sensitive Markov decision processes. Management Science 18 (7), p. 356–369. Cited by: §B.1, §2. Jiang et al. (2026) Y. Jiang, J. Huang, Y. Yuan, X. Mao, Y. Yue, Q. Zhao, and L. Yan Risk-sensitive RL for alleviating exploration dilemmas in large language models. In International Conference on Learning Representations, Note: arXiv:2509.24261 Cited by: §B.1, §1, §2, §3.2. Li et al. (2025a) L. Li, Z. Zhou, J. Hao, J. K. Liu, Y. Miao, W. Pang, X. Tan, W. Chu, Z. Wang, S. Pan, et al. The choice of divergence: a neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward. arXiv preprint arXiv:2509.07430. Cited by: §2. Li et al. (2025b) T. Li, Y. Zhang, P. Yu, S. Saha, D. Khashabi, J. Weston, J. Lanchantin, and T. Wang Jointly reinforcing diversity and quality in language model generations. arXiv preprint arXiv:2509.02534. Cited by: §2. Lin et al. (2025) N. Lin, J. Zhang, L. Hou, and J. Li Boundary-guided policy optimization for memory-efficient RL of diffusion large language models. arXiv preprint arXiv:2510.11683. Cited by: §2. Mathematical Association of America (2023) Mathematical Association of America AMC 2023. Note: https://artofproblemsolving.com/wiki/index.php?title=AMC_Problems_and_SolutionsAccessed: 2026-03-20 Cited by: §4.1. Mathematical Association of America (2024) Mathematical Association of America AIME 2024. Note: https://artofproblemsolving.com/wiki/index.php/2024_AIME_I_ProblemsAccessed: 2026-03-20 Cited by: §4.1. Mathematical Association of America (2025) Mathematical Association of America AIME 2025. Note: https://artofproblemsolving.com/wiki/index.php/2025_AIME_I_ProblemsAccessed: 2026-03-20 Cited by: §4.1. Mroueh (2026) Y. Mroueh Reinforcement learning with verifiable rewards: GRPO’s loss, dynamics, and success amplification. In International Conference on Machine Learning, External Links: Link Cited by: §2. Ni et al. (2026) Z. Ni, S. Wang, Y. Yue, T. Yu, W. Zhao, Y. Hua, T. Chen, J. Song, C. Yu, B. Zheng, et al. The flexibility trap: rethinking the value of arbitrary order in diffusion language models. arXiv preprint arXiv:2601.15165. Cited by: §2. Nie et al. (2025) S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li Large language diffusion models. arXiv preprint arXiv:2502.09992. Cited by: §1, §3.1. Oba et al. (2026) D. Oba, H. Furuta, and N. Okazaki Diffusion-state policy optimization for masked diffusion language models. arXiv preprint arXiv:2602.06462. Cited by: §2. Ou et al. (2025) J. Ou, J. Han, M. Xu, S. Xu, J. Xie, S. Ermon, Y. Wu, and C. Li Principled RL for diffusion LLMs emerges from a sequence-level perspective. arXiv preprint arXiv:2512.03759. Cited by: §2. Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p. 311–318. External Links: Document, Link Cited by: §C.3. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 3982–3992. External Links: Document, Link Cited by: §C.3, §4.3. Ren et al. (2025) T. Ren, J. Jiang, H. Yang, W. Tian, M. Zou, G. Li, Z. Zhang, Q. Wang, S. Qin, Y. Zhao, R. Tao, H. Shao, and Y. Peng RiskPO: risk-based policy optimization via verifiable reward for LLM post-training. arXiv preprint arXiv:2510.00911. Cited by: §2. Rojas et al. (2025) K. Rojas, J. Lin, K. Rasul, A. Schneider, Y. Nevmyvaka, M. Tao, and W. Deng Improving reasoning for diffusion language models via group diffusion policy optimization. arXiv preprint arXiv:2510.08554. Cited by: §2. Rubinstein (1999) R. Y. Rubinstein The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability 1 (2), p. 127–190. Cited by: §2. Sahoo et al. (2024) S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. M. Marroquin, J. T. Chiu, A. M. Rush, and V. Kuleshov Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, Cited by: §A.1, §A.1, §1, §3.1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §B.3, §3.2. Sentence Transformers (2021) Sentence Transformers all-MiniLM-L6-v2: sentence embedding model and model card. Note: Hugging Face model repositoryRepository first uploaded in August 2021; model card accessed 2026-09-24 External Links: Link Cited by: §C.3, §4.3. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §3.2. Shi et al. (2024) J. Shi, K. Han, Z. Wang, A. Doucet, and M. K. Titsias Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, Cited by: §A.1, §A.1. Tang et al. (2025) X. Tang, R. Dolga, S. Yoon, and I. Bogunovic Wd1: weighted policy optimization for reasoning in diffusion language models. arXiv preprint arXiv:2507.08838. Cited by: §2. Thrampoulidis et al. (2026) C. Thrampoulidis, S. Mahdavi, and W. Deng Advantage shaping as surrogate reward maximization: unifying Pass@K policy gradients. Transactions on Machine Learning Research. External Links: Link Cited by: §2. Walder and Karkhanis (2025) C. Walder and D. Karkhanis Pass@K policy optimization: solving harder reinforcement learning problems. arXiv preprint arXiv:2505.15201. External Links: Link Cited by: §2. Wang et al. (2025a) C. Wang, Z. Li, J. Bai, Y. Zhang, S. Cui, Z. Zhao, and Y. Wang Arbitrary entropy policy optimization breaks the exploration bottleneck of reinforcement learning. arXiv preprint arXiv:2510.08141. Cited by: §2. Wang et al. (2025b) C. Wang, P. Rashidinejad, D. Su, S. Jiang, S. Wang, S. Zhao, C. Zhou, S. Z. Shen, F. Chen, T. Jaakkola, et al. SPG: sandwiched policy gradient for masked diffusion language models. arXiv preprint arXiv:2510.09541. Cited by: §2. Wang et al. (2025c) Y. Wang, L. Yang, B. Li, Y. Tian, K. Shen, and M. Wang Revolutionizing reinforcement learning framework for diffusion large language models. arXiv preprint arXiv:2509.06949. Cited by: Table 2, §1, §2. Whittle (1990) P. Whittle Risk-sensitive optimal control. Wiley. Cited by: §2. Zhan (2025) A. Zhan Simple policy gradients for reasoning with diffusion language models. arXiv preprint arXiv:2510.04019. Cited by: §2. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: §D.1. Zhang et al. (2026) W. Zhang, L. Wu, C. Zhao, E. Chang, M. Zhuge, Z. Liu, A. Su, H. Huang, J. Chen, C. Zhou, R. Krishnamoorthi, V. Chandra, M. Elhoseiny, and W. Wen dTRPO: trajectory reduction in policy optimization of diffusion large language models. arXiv preprint arXiv:2603.18806. Cited by: §2. Zhao et al. (2025a) S. Zhao, D. Gupta, Q. Zheng, and A. Grover D1: scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216. Cited by: §1, §2. Zhao et al. (2025b) S. Zhao, M. Liu, J. Huang, M. Liu, C. Wang, B. Liu, Y. Tian, G. Pang, S. Bell, A. Grover, and F. Chen Inpainting-guided policy optimization for diffusion large language models. arXiv preprint arXiv:2509.10396. Cited by: §2. Zhong et al. (2026) J. Zhong, K. Wang, D. Ding, Z. Feng, H. Bai, Y. Xiang, J. Sun, and Q. Xu Stabilizing reinforcement learning for diffusion language models. arXiv preprint arXiv:2603.06743. Cited by: §B.3, Table 2, §2, §3.2, §4.1. Zhou et al. (2026) Z. Zhou, L. Chen, H. Tong, and D. Song DLLM: simple diffusion language modeling. External Links: 2602.22661, Link Cited by: §C.1. Zhu et al. (2018) Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu Texygen: a benchmarking platform for text generation models. arXiv preprint arXiv:1802.01886. External Links: Link Cited by: §C.3, §4.3. Zhu et al. (2025a) Y. Zhu, J. Wan, X. Liu, S. He, Q. Wang, X. Guo, T. Liang, Z. Huang, Z. He, and X. Qiu DiRL: an efficient post-training framework for diffusion language models. arXiv preprint arXiv:2512.22234. Cited by: §1. Zhu et al. (2025b) Y. Zhu, W. Guo, J. Choi, P. Molodyk, B. Yuan, M. Tao, and Y. Chen Enhancing reasoning for diffusion llms via distribution matching policy optimization. arXiv preprint arXiv:2510.08233. Cited by: §2. Contents of the Appendix Appendix A Diffusion Language Model Families This appendix distinguishes continuous diffusion time from the discrete policy-loss index in Section 3.1. We suppress the prompt conditioning only where it is not needed. Let V be the clean-token vocabulary, m∉m the mask token, and L the padded output length. The symbol δa(z)=z=a _a(z)=1\z=a\ denotes point mass at token a; E1\E\ equals one when condition E holds and zero otherwise. A.1 Masked diffusion Use ζ∈[0,1]ζ∈[0,1] for continuous forward-diffusion time and α(ζ)α(ζ) for the masking probability, with α(0)=0α(0)=0, α(1)=1α(1)=1, and α nondecreasing. Writing X0=x0∈LX_0=x_0 ^L, the absorbing forward process has marginal Q(Xζ(ℓ)=z∣X0=x0)=[1−α(ζ)]δx0(ℓ)(z)+α(ζ)δm(z),ℓ=1,…,L.Q(X_ζ^( )=z X_0=x_0)=[1-α(ζ)] _x_0^( )(z)+α(ζ) _m(z), =1,…,L. (A.1) Masking is independent across positions and absorbing across time. The expected masked fraction is α(ζ)α(ζ); a realized state’s masked fraction need not equal this expectation. Let fθ(ℓ)(z∣xζ,ζ,q)f_θ^( )(z x_ζ,ζ,q) be the model’s clean-token probability at position ℓ , extended by fθ(ℓ)(m∣⋅)=0f_θ^( )(m ·)=0. For 0≤ξ<ζ≤10≤ξ<ζ≤ 1 with α(ζ)>0α(ζ)>0, the factorized absorbing reverse kernel is Pθ(Xξ(ℓ)=z∣xζ,q)=δxζ(ℓ)(z),xζ(ℓ)≠m,α(ξ)α(ζ)δm(z)+[1−α(ξ)α(ζ)]fθ(ℓ)(z∣xζ,ζ,q),xζ(ℓ)=m.P_θ(X_ξ^( )=z x_ζ,q)= cases _x_ζ^( )(z),&x_ζ^( )≠ m,\\[2.0pt] α(ξ)α(ζ) _m(z)+ [1- α(ξ)α(ζ) ]f_θ^( )(z x_ζ,ζ,q),&x_ζ^( )=m. cases (A.2) The full kernel multiplies these terms over positions (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024). This absorbing kernel is background; a confidence-based decoder that selects reveal positions may implement a different transition rule. The rollout policy and its likelihood estimator must describe that implemented decoder. Assume α is absolutely continuous and α(ζ)>0α(ζ)>0 for 0<ζ≤10<ζ≤ 1. The corresponding weighted denoising objective is ℒMDLM(θ)=−[α′(ζ)α(ζ)∑ℓ=1LXζ(ℓ)=mlogfθ(ℓ)(X0(ℓ)∣Xζ,ζ,q)].L_MDLM(θ)=-E\! [ α (ζ)α(ζ) _ =1^L1\X_ζ^( )=m\ f_θ^( )(X_0^( ) X_ζ,ζ,q) ]. (A.3) The expectation is over training examples (q,X0)(q,X_0), ζ∼(0,1)ζ (0,1), and Xζ∼Q(⋅∣X0)X_ζ Q(· X_0); α′α is the time derivative. Endpoint values are interpreted by limits. Since α denotes masking rather than clean-token retention, the weight is α′/α /α (Sahoo et al., 2024; Shi et al., 2024). A.2 Block diffusion Partition the padded sequence into BblkB_blk blocks of length ℓblk _blk. The clean sequence is x0=(x0[1],…,x0[Bblk])x_0=(x_0^[1],…,x_0^[B_blk]), and x0[<d]x_0^[<d] is the prefix preceding block d. Block diffusion uses πθ(x0∣q)=∏d=1Bblkπθ(x0[d]∣x0[<d],q). _θ(x_0 q)= _d=1^B_blk _θ(x_0^[d] x_0^[<d],q). (A.4) For a fixed within-block horizon KblkK_blk, initialize xKblk[d]=mℓblkx_K_blk^[d]=m _blk and marginalize only the unfixed intermediate states: πθ(x0[d]∣x0[<d],q)=∑x1[d],…,xKblk−1[d]∏h=1KblkPθ(xh−1[d]∣xh[d],x0[<d],q,h). _θ(x_0^[d] x_0^[<d],q)= _x_1^[d],…,x_K_blk-1^[d] _h=1^K_blkP_θ(x_h-1^[d] x_h^[d],x_0^[<d],q,h). (A.5) Attention is causal across blocks and bidirectional within the active block (Arriola et al., 2025; Cheng et al., 2025). Prefix keys and values can be cached, although attention still incurs a prefix-length-dependent cost. Our SDAR experiments use ℓblk=4 _blk=4. Relation to the risk coordinate. The discrete index t orders transitions in a sampled trajectory; uqju_qj is the scalar supplied to the risk map b for action j. Neither is automatically equal to continuous time ζ or the expected masking probability α(ζ)α(ζ). In block diffusion, a within-block mask fraction can restart at each block even though the global trajectory index advances. The reference and controls use the same executed coordinate, action mask, and loss reduction. Appendix B Additional Theoretical Analysis and Calibration The empirical group fraction p^q p_q, the population success probability pπ(q)p_π(q), and the model-level calibration average p¯ p serve different roles. The first enters the actual advantages; the second describes a fixed rollout distribution; the third suggests a coefficient scale before training. The identities below keep these quantities separate. B.1 Proof of Lemma 3.1 Fix a prompt q and its realized rewards rq1,…,rqGr_q1,…,r_qG. Because each reward is either zero or one, eβrqi=1+rqi(eβ−1)e^β r_qi=1+r_qi(e^β-1). Averaging this equality over the group gives the exact normalizer D^q(β)=1G∑i=1Geβrqi=1+p^q(eβ−1). D_q(β)= 1G _i=1^Ge^β r_qi=1+ p_q(e^β-1). For β>0β>0, substitute this expression into Eq. (3.2) and collect the terms containing the reward: Aqi(β) A_qi(β) =1+rqi(eβ−1)−[1+p^q(eβ−1)]β[1+p^q(eβ−1)] = 1+r_qi(e^β-1)-[1+ p_q(e^β-1)]β[1+ p_q(e^β-1)] =eβ−1β[1+p^q(eβ−1)](rqi−p^q)=cβ(p^q)Aqi0. = e^β-1β[1+ p_q(e^β-1)](r_qi- p_q)=c_β( p_q)A^0_qi. (B.1) This proves the finite-group factorization. If every reward is zero or every reward is one, rqi=p^qr_qi= p_q for every response, so all advantages are zero. At β=0β=0, the expansion eβ−1=β+O(β2)e^β-1=β+O(β^2) gives cβ(p)→1c_β(p)→ 1. Thus the identity extends continuously to the centered baseline. These statements hold for each realized group, without an independence assumption or a large-group limit. For the tilting statement, let p=pπ(q)p=p_π(q) be the success probability of a fixed rollout distribution. Its normalizer is Dβ=1−p+peβD_β=1-p+pe^β. Summing the tilted probabilities separately over successful and unsuccessful trajectories gives pβ=peβDβ,1−pβ=1−pDβ.p_β= pe^βD_β, 1-p_β= 1-pD_β. When 0<p<10<p<1, their ratio is eβp/(1−p)e^βp/(1-p); taking logarithms proves Eq. (3.7). At p=0p=0 or 11, exponential tilting leaves the all-failure or all-success distribution unchanged. This completes the proof. □ Interpretation of the multiplier. For a fixed coefficient β>0β>0, ∂cβ(p)∂p=−(eβ−1)2β[1+p(eβ−1)]2<0. ∂ c_β(p)∂ p=- (e^β-1)^2β[1+p(e^β-1)]^2<0. Thus, groups with lower observed solve fractions receive larger multipliers at the same coefficient. This is a statement about relative weighting across groups, not a guarantee that every multiplier exceeds one. Moreover, cβ(p)=1+(12−p)β+O(β2)c_β(p)=1+( 12-p)β+O(β^2) near zero; increasing β need not increase advantage magnitude uniformly across solve rates. For a common coefficient, the group advantages sum to zero. With action-dependent coefficients and token-reduction weights, an action-weighted mean need not be zero, which is one reason to distinguish RMS matching from standard-deviation matching. Connection to entropic utility. For a fixed prompt and binary reward, the population entropic utility is Jβ(π,q)=β−1log[1+pπ(q)(eβ−1)]J_β(π;q)=β^-1 [1+p_π(q)(e^β-1)]. Its derivative with respect to pπ(q)p_π(q) is cβ(pπ(q))c_β(p_π(q)). When differentiation under the rollout expectation is valid, ∇θJβ(πθ,q)=cβ(pπθ(q))∇θpπθ(q). _θJ_β( _θ;q)=c_β(p_ _θ(q)) _θp_ _θ(q). This gives the population coefficient an outcome-reweighting interpretation (Howard and Matheson, 1972; Jiang et al., 2026). The exact finite-group factorization above is a separate statement: substituting the empirical fraction into a nonlinear normalizer does not, by itself, establish an unbiased finite-group gradient estimator. B.2 Capability-guided calibration Let πinit _init be the initial policy and calD_cal a calibration distribution. Define p¯=q∼cal[pπinit(q)]. p=E_q _cal[p_ _init(q)]. This model-level quantity differs from the current training group’s p^q p_q. Exponential tilting multiplies failure-to-success odds by e−βe^-β. Equating their fractional reduction g(β)=1−e−βg(β)=1-e^-β to the solve-rate proxy yields β⋆(p¯)=−log(1−p¯),0≤p¯<1.β ( p)=- (1- p), 0≤ p<1. (B.2) The rule suggests an interpretable search scale; it is not an optimizer-derived optimum or upper bound. For SDAR-4B, the reported proxy p¯≈0.66 p≈ 0.66 gives β⋆≈1.08β ≈ 1.08. Figure 6 compares this rounded candidate with neighboring endpoints. The defaults below are then held fixed during each run. Table 1: Selected schedules and evaluated constant-risk coefficients. Backbone Default StepRS schedule Constant-risk sweep Qwen3-0.6B-MDLM [0.5→0.25][0.5→ 0.25] 0.25,0.5,1.0\0.25,0.5,1.0\ SDAR-1.7B-Chat [1.0→0.5][1.0→ 0.5] 0.25,0.5,1.0\0.25,0.5,1.0\ SDAR-4B-Chat [1.5→0.75][1.5→ 0.75] 0.5,1.0,1.5,2.0\0.5,1.0,1.5,2.0\ B.3 Scope of denoising-state scheduling For each action j, StepRS applies the scalar group transformation at βqj=b(uqj) _qj=b(u_qj). All rewards in that evaluation of the group normalizer use this same coefficient. Applying a different coefficient to each reward inside one denominator would define another transformation and would not give Equation (3.5). The schedule therefore has a precise interpretation as the advantage array supplied to the native surrogate in Equation (3.4). Its direction and interpolation are assessed by the controls and fixed-endpoint ablation, rather than inferred from a fixed-coefficient trajectory objective. Trainer interface. Equation (3.1) separates fixed reduction weights vqjv_qj from the native surrogate Ψnative ^native, which may depend on the whole sampled group through its normalization. For PPO, the surrogate is the minimum of the unclipped and clipped ratio terms (Schulman et al., 2017). In published StableDRL, unconditional ratio clipping and a sum-of-clipped-ratios normalizer instead weight score-function gradients (Zhong et al., 2026). These are different policy-loss constructions. Our notation preserves the selected construction and changes only the supplied advantages. Appendix C Experimental Details C.1 Additional results: 0.6B masked diffusion LM We evaluate Qwen3-0.6B-MDLM, distilled from a post-trained Qwen3-0.6B autoregressive backbone via the dLLM A2D pipeline of Zhou et al. (2026). The model is fine-tuned on the GSM8K training split, with G=16G=16 responses per prompt and binary correctness rewards. The evaluated checkpoints are at 6060 RL fine-tuning steps. We use the schedule [0.5→0.25][0.5→ 0.25] and evaluate each configuration with inference seeds 4242, 4343, and 4444, using 6464 responses per problem per seed. Figure 5 shows higher mean pass@k than GRPO at all 2121 benchmark–budget combinations. Mean pass@11 improves from 40.7040.70 to 44.5744.57 on GSM8K, 5.205.20 to 8.138.13 on MATH500, and 0.200.20 to 1.201.20 on AMC23. At k=64k=64, the corresponding improvements are 84.25→89.2084.25→ 89.20, 26.67→40.2026.67→ 40.20, and 10.00→28.3310.00→ 28.33. The particularly large coverage gains on MATH500 and AMC23 extend the evidence beyond the two block-dLLM backbones. Figure 5: Qwen3-0.6B-MDLM evaluation curves. The selected [0.5→0.25][0.5→ 0.25] schedule improves pass@k over GRPO across GSM8K, MATH500, and AMC23. All curves show means and capped error bars show one sample standard deviation over three seeds. The unique-answer ratio also increases from 18.0318.03 to 28.0028.00 on GSM8K, 11.4011.40 to 33.4733.47 on MATH500, and 17.3017.30 to 24.8324.83 on AMC23 (Figure 12). These results demonstrate improvements in both accuracy and sampled-output variation. Because this case study uses a different backbone family and training corpus, we treat it as complementary portability evidence rather than a controlled model-size comparison. C.2 Training configuration Table 2 shows the detailed training recipes for the three model scales reported in Section 4 and Appendix C.1. Table 2: Training hyperparameters used for the three model scales. Hyperparameter Qwen3-0.6B-MDLM SDAR-1.7B-Chat SDAR-4B-Chat Base model (HuggingFace checkpoint) dllm-hub/Qwen3-0.6B-diffusion-mdlm-v0.1 JetLM/SDAR-1.7B-Chat JetLM/SDAR-4B-Chat Trainer framework TraceRL (Wang et al., 2025c) StableDRL (Zhong et al., 2026) Training data GSM8K train MATH train Optimizer AdamW (lr=5×10−65× 10^-6) AdamW (lr=1×10−61× 10^-6) Responses per prompt (G) 16 Group rollout batch 1024 (64×16) 512 (32×16) Gradient accumulation 1 32 Total optimizer steps 100 160 DeepSpeed ZeRO-3 (CPU off.) ZeRO-3 (no off.) Mixed precision bf16 GPUs 8× L40S (48GB) 8× H100 (95GB) Max sequence length 512 1024 Denoising steps (T) at training 256 KL penalty (λ) 0.01 Advantage clip (C) 10.0 C.3 Evaluation protocol We evaluate each fixed checkpoint with inference seeds Σ=42,43,44 =\42,43,44\. For each question q in a benchmark Q, each seed generates N=64N=64 responses at temperature 1.01.0. The training group size G=16G=16 and evaluation sample count N=64N=64 are distinct. We compute pass@k from the correct-response count using Equation (4.1) and average over questions (Chen et al., 2021); we do not pool seeds into an N=192N=192 estimate. For a metric’s benchmark-level score M(σ)M^(σ) under seed σ∈Σσ∈ , we report M¯=13∑σ∈ΣM(σ),sM=12∑σ∈Σ(M(σ)−M¯)2. M= 13 _σ∈ M^(σ), s_M= 12 _σ∈ (M^(σ)- M )^2. (C.1) Capped error bars show M¯±sM M± s_M, the sample standard deviation across inference seeds, not a standard error or confidence interval. The following definitions concern one model and one seed; these indices are suppressed to keep the notation readable. Unique-answer ratio. Let y^qi y_qi be the extracted and normalized final-answer string for response i. We define U(q)=|y^q1,…,y^qN|N,U(σ)=100||∑q∈U(q).U(q)= |\ y_q1,…, y_qN\|N, U^(σ)= 100|Q| _q U(q). (C.2) The set removes duplicate normalized answers; |⋅||·| denotes cardinality. Failed extractions form one shared empty-answer class. Normalization removes formatting and merges equivalent forms where supported by the extractor. The reported U¯ U is the mean of the percentage-valued U(σ)U^(σ) across seeds. Answer-removed reasoning traces. Let hqih_qi be response i after removing its final boxed-answer span and the sentence containing that span. Diversity is computed within each question, using all N sampled traces, including incorrect responses and repeated texts. Duplicates remain separate observations because their frequency is part of the generation distribution. These metrics assess variation in the retained text, not the number of distinct correct solution strategies. N-gram variation. An n-gram is a sequence of n consecutive tokens. BLEU-4 combines clipped n-gram precisions for n=1,2,3,4n=1,2,3,4 with equal weights (Papineni et al., 2002). Write ℋq,−i=(hqi′:1≤i′≤N,i′≠i)H_q,-i=(h_qi :1≤ i ≤ N,\ i ≠ i) for the list of the other N−1N-1 traces, retaining duplicates. We compute sentence-level BLEU for each trace against this reference list, then average: S4(q)=1N∑i=1NBLEU4(hqi,ℋq,−i),Dngram(q)=100[1−S4(q)].S_4(q)= 1N _i=1^NBLEU_4(h_qi,H_q,-i), D_ngram(q)=100[1-S_4(q)]. (C.3) This is the Self-BLEU construction of Zhu et al. (2018). BLEU values are on the [0,1][0,1] scale here. A larger S4S_4 means more overlap with the other samples, so its complement gives larger values for greater lexical variation. It is not distinct-44, which would count unique four-token sequences. For a nonempty tokenized hypothesis h and reference list ℋH, the BLEU combination is BLEU4(h,ℋ)=BP(h,ℋ)exp14∑n=14logP~n(h,ℋ).BLEU_4(h,H)=BP(h,H) \! \ 14 _n=1^4 P_n(h,H) \. (C.4) Before smoothing, the modified precision is Pn(h,ℋ)=∑gminCn(g,h),maxh′∈ℋCn(g,h′)∑gCn(g,h),P_n(h,H)= _g \C_n(g,h), _h C_n(g,h )\ _gC_n(g,h), when the denominator is positive. Here Cn(g,h)C_n(g,h) counts occurrences of n-gram g in h; clipping prevents repeated hypothesis n-grams from receiving more credit than their largest reference count. The brevity penalty is BP=expmin(0,1−Lref/Lh)BP= \ (0,1-L_ref/L_h)\, where LhL_h is hypothesis length and LrefL_ref is the closest reference length, choosing the shorter reference in a tie. The P~n P_n denote the precisions after sentence-level smoothing, which handles zero matches and short traces (Chen and Cherry, 2014). Tokenization, smoothing, and exceptional empty-text handling are part of the metric implementation. Embedding variation. We use the Sentence Transformers checkpoint sentence-transformers/all-MiniLM-L6-v2, a 384-dimensional sentence/paragraph encoder (Reimers and Gurevych, 2019; Sentence Transformers, 2021). Let eqi=femb(hqi)∈ℝ384e_qi=f_emb(h_qi) ^384 be the nonzero vector returned by the fixed encoding pipeline and let zqi=eqi/‖eqi‖2z_qi=e_qi/\|e_qi\|_2 be its unit normalization. Cosine similarity is zqizqjz_qi Tz_qj, where ∥⋅∥2\|·\|_2 is Euclidean norm and T denotes transpose. We average cosine distance over unordered pairs: Demb(q)=100(N2)∑1≤i<i′≤N(1−zqizqi′).D_emb(q)= 100 N2 _1≤ i<i ≤ N (1-z_qi Tz_qi ). (C.5) For unit vectors, 1−zqizqi′=12‖zqi−zqi′‖221-z_qi Tz_qi = 12\|z_qi-z_qi \|_2^2, so this is also a scaled mean squared separation of normalized embeddings. There are (642)=2016 642=2016 pairs per question. Self-pairs are excluded, while separately sampled identical texts remain in the average. Greater distance means greater dispersion in the encoder’s representation space. Multiplication by 100100 sets the plotting scale; it does not turn this into a percentage of distinct reasoning strategies. Benchmark aggregation and interpretation. For either trace metric ν, the seed-specific score is Dν(σ)=1||∑q∈Dν(q),ν∈ngram,emb.D_ν^(σ)= 1|Q| _q D_ν(q), ν∈\ngram,emb\. We apply Equation (C.1) to these three benchmark averages. We never compare traces from different questions or pool all seeds before computing pairwise distances. The two measures provide complementary lexical and representation-based views of variation; their different scales should not be compared numerically. They accompany, rather than replace, verifier-based accuracy and coverage. Results appear in Figures 3 and 12. C.4 Prompt templates We use the following ChatML-formatted prompt for training rollouts and evaluation. The SDAR evaluations cover MATH500, AMC23, AIME2024, and AIME2025; the Qwen3-0.6B-MDLM evaluations cover GSM8K, MATH500, and AMC23. No system message, few-shot examples, or additional reasoning scaffold is included. The verifier extracts the final answer from the ... span in the assistant’s continuation. Math Reasoning Prompt (ChatML; SDAR-1.7B/4B-Chat, Qwen3-0.6B-MDLM) <|im_start|>user You need to put your final answer in . This is the problem: problem<|im_end|> <|im_start|>assistant Appendix D Additional Ablations and Diagnostics D.1 Mechanism-control definitions and matching properties The controls are designed for this study. Their interpretation follows from the quantities preserved in a fixed minibatch, as shown below. All matching quantities are recomputed using the current arm’s own rollouts. The reference rule supplies comparison weights on those actions; it does not require trajectories from a separately trained StepRS checkpoint. Separate online runs can generate different trajectories even when their sampling settings are matched. Loss weights, multipliers, and advantages. The fixed reduction coefficient vqj>0v_qj>0 determines how an action enters the native loss reduction; wqj>0w_qj>0 is the additional StepRS multiplier; aqja_qj is the signed centered advantage. Define Vq=∑j∈qvqjV_q= _j _qv_qj and Mq=∑j∈qvqjwqjM_q= _j _qv_qjw_qj. For a nonempty action set, Vq>0V_q>0 and w¯q=Mq/Vq w_q=M_q/V_q. Here “mass” means this sum of nonnegative multiplier weights, not probability mass, total signed advantage, or gradient magnitude. Prompts with no active loss actions contribute no reward-loss term. C1 as the closest prompt-only weighting. The replacement in Equation (4.2) is the unique solution of the one-parameter weighted least-squares problem w¯q=argminz∈ℝ∑j∈qvqj(wqj−z)2. w_q= z \, _j _qv_qj(w_qj-z)^2. (D.1) This is projection onto constant multipliers under the weighted squared-error criterion (Boyd and Vandenberghe, 2018). Differentiating the criterion gives 2(Vqz−Mq)2(V_qz-M_q), and its second derivative is 2Vq>02V_q>0, so its minimizer is Mq/VqM_q/V_q. Equivalently, ∑jvqj(wqj−w¯q)=0,∑jvqjw¯q=Mq. _jv_qj(w_qj- w_q)=0, _jv_qj w_q=M_q. For illustration, equal-weight multipliers (2,1)(2,1) become (1.5,1.5)(1.5,1.5): their sum is unchanged but their state variation is removed. This is an algebraic example, not an experimental result. C1 retains the prompt-level average induced by its empirical solve rate and sampled states; it does not match only the midpoint of the risk coefficients. In general, ∑jvqjwqjaqj≠∑jvqjw¯qaqj, _jv_qjw_qja_qj≠ _jv_qj w_qa_qj, so matching multiplier mass is distinct from matching a signed advantage sum or the policy gradient. C2 as a mass-normalized reversal. Let Mˇq=∑jvqjwˇqj M_q= _jv_qj w_qj, where wˇqj=cb(1−uqj)(p^q)>0 w_qj=c_b(1-u_qj)( p_q)>0. Thus κq=Mq/Mˇq _q=M_q/ M_q is well-defined, positive, and satisfies ∑jvqjκqwˇqj=Mq. _jv_qj _q w_qj=M_q. (D.2) The transformation reverses the coefficient assignment while retaining this total weight. It does not assume that a larger coefficient produces a larger multiplier for every solve rate. Both C1 and C2 subsequently apply clip(⋅,−C,C)clip(·,-C,C) to their constructed advantages. Their identities refer to multipliers before this cap, not to a separately matched post-cap norm. C3 and the meaning of active-entry RMS. Let ℬ=(q,j):q∈ℬ,j∈qA_B=\(q,j):q ,j _q\ collect the active entries in one optimizer update, and let nact=|ℬ|n_act=|A_B|. For a real array Z=(Zqj)Z=(Z_qj) on this set, active-entry RMS is RMSℬ(Z)=[1nact∑(q,j)∈ℬZqj2]1/2.RMS_B(Z)= [ 1n_act _(q,j) _BZ_qj^2 ]^1/2. (D.3) Padding and inactive positions are excluded. If nact=0n_act=0, there is no reward-loss term and no RMS ratio is formed. Unlike a standard deviation, RMS does not subtract the array’s mean. For RB=RMSℬ(B)>0R_B=RMS_B(B)>0, define Ra=RMSℬ(a)R_a=RMS_B(a) and ηℬ=Ra/RB _B=R_a/R_B. Then RMSℬ(BC3)=ηℬRB=Ra.RMS_B(B^C3)= _BR_B=R_a. (D.4) The same positive scalar multiplies every active entry, so it preserves signs and ratios of nonzero entries of B. If RB=0R_B=0, positivity of wqjw_qj and C>0C>0 imply a=0a=0 on the active set; we set BC3=0B^C3=0 without forming 0/00/0. The reference regularizer may still contribute an update. C3 is related to normalization choices studied in on-policy RL (Andrychowicz et al., 2021), but differs from subtracting a minibatch mean and dividing by its standard deviation. RMS scaling without recentering also appears in RMSNorm (Zhang and Sennrich, 2019); that work normalizes activations, whereas our control matches an advantage array to a particular baseline RMS. These references motivate the comparison, not the specific C3 design. The scale is applied after the shared advantage cap and is held constant during backpropagation. Applying another cap or a non-uniform normalization afterward would generally change the matching identity and must be reported if used. What RMS matching does to the reward surrogate. For fixed data and any positive scalar η, the PPO surrogate obeys minρηA,clip(ρ,1−ϵ,1+ϵ)ηA=ηminρA,clip(ρ,1−ϵ,1+ϵ)A. \ρη A,clip(ρ,1-ε,1+ε)η A\=η \ρ A,clip(ρ,1-ε,1+ε)A\. Thus a detached scalar rescales the reward-loss gradient of the same StepRS advantage array, wherever it is differentiable. The same property holds for a native surrogate linear in advantages with advantage-independent normalization, including the published StableDRL score-function weighting. It does not equate that gradient to the centered-GRPO gradient: actions have different policy-score directions and multipliers. Nor does it match the KL gradient, parameter-wise optimizer state, or the final parameter update. C3 therefore tests overall advantage amplitude while retaining the relative StepRS weighting pattern. Implementation invariants. Reference and controls share the state-coordinate definition, active action mask, native loss construction, cap, and regularization. For an optimizer update assembled from several microbatches or devices, one-update RMS requires a common reduction of squared sums and active counts. Local per-microbatch or per-device rescaling is not the same intervention. The active set and reduction scope must therefore be recorded with the run configuration. D.2 Calibration and endpoint choice For SDAR-4B, p¯≈0.66 p≈ 0.66 gives β⋆≈1.08β ≈ 1.08. Figure 6 compares the rounded calibrated schedule [1.08→0.54][1.08→ 0.54] with nearby half-decay schedules. Relative to [1.0→0.5][1.0→ 0.5], the calibrated schedule improves mean pass@11 by 0.400.40, 0.830.83, 0.330.33, and 1.271.27 points across the four benchmarks, and improves mean pass@k at every evaluated budget. Its pass@11 remains close to the selected [1.5→0.75][1.5→ 0.75] schedule, supporting the calibration rule as a useful starting point for endpoint selection. Holding the upper endpoint at 2.02.0 isolates the lower-endpoint choice. Half-decay [2.0→1.0][2.0→ 1.0] improves pass@11 over full decay [2.0→0][2.0→ 0] by 0.530.53, 2.072.07, 0.570.57, and 0.770.77 points. Its mean pass@k is higher in 2727 of the 2828 comparisons and equal in the remaining comparison, supporting a nonzero late-stage risk coefficient in this setting. Figure 6: Calibration and endpoint ablations on SDAR-4B. Top: linear half-decay schedules with upper endpoints 1.01.0, 1.081.08, 1.51.5, and 2.02.0. Bottom: full decay versus half-decay at a fixed upper endpoint of 2.02.0. Columns correspond to the four benchmarks. Curves show means and one-standard-deviation bars across three inference seeds. D.3 Full constant-risk sweep Figure 7 compares the selected schedule with constant β∈0.5,1.0,1.5,2.0β∈\0.5,1.0,1.5,2.0\. Against the strong constant-1.51.5 comparator, StepRS has higher mean pass@k in 2626 of 2828 comparisons and ties in the remaining two: MATH500 pass@11 and pass@6464. It improves all seven budgets on AMC23, AIME2024, and AIME2025. On AIME2025, for example, the gains at k=8k=8 and 1616 are 1.571.57 and 1.831.83 percentage points. The sweep also shows that the best constant coefficient can depend on the benchmark and budget. Figure 7: The selected [1.5→0.75][1.5→ 0.75] StepRS schedule versus constant risk coefficients on SDAR-4B. Each panel shows the complete pass@11–pass@6464 profile for one benchmark, with mean and one-standard-deviation bars over three inference seeds. D.4 Risk and endpoint sensitivity on smaller backbones Figures 8–11 extend the constant-risk and endpoint comparisons to SDAR-1.7B and Qwen3-0.6B-MDLM. Every curve uses all seven sampling budgets and summarizes the three inference seeds, with the same schedule selected for each backbone as in the main results. On SDAR-1.7B, [1.0→0.5][1.0→ 0.5] improves mean pass@11 over constant β=1.0β=1.0 by 0.300.30, 1.731.73, 0.170.17, and 0.430.43 points on MATH500, AMC23, AIME2024, and AIME2025. The scheduled variant matches or improves on this constant baseline at all 2828 benchmark–budget combinations. On Qwen3-0.6B-MDLM, [0.5→0.25][0.5→ 0.25] improves on constant β=0.5β=0.5 at all seven budgets on GSM8K and MATH500. On AMC23, constant β=0.5β=0.5 is ahead by 0.100.10 points at pass@11, while the scheduled variant is ahead at every budget from k=2k=2 through 6464, reaching 28.3328.33 versus 27.5027.50 at pass@6464. These comparisons support the schedules as effective defaults while showing how their benefits vary with the benchmark and sampling budget. Figure 8: SDAR-1.7B constant-risk comparison: β∈0.25,0.5,1.0β∈\0.25,0.5,1.0\ versus StepRS [1.0→0.5][1.0→ 0.5]. Curves show means with one-standard-deviation bars across three inference seeds. Figure 9: SDAR-1.7B endpoint comparison: [0.25→0][0.25→ 0], [0.5→0.25][0.5→ 0.25], [1.0→0][1.0→ 0], and the default [1.0→0.5][1.0→ 0.5]. All schedules are linear; curves show three-seed means and one-standard-deviation bars. Figure 10: Qwen3-0.6B-MDLM constant-risk comparison. The default StepRS schedule is [0.5→0.25][0.5→ 0.25]. Curves show three-seed means with capped one-standard-deviation error bars. All three constant coefficients β∈0.25,0.5,1.0β∈\0.25,0.5,1.0\ are shown. Figure 11: Qwen3-0.6B-MDLM endpoint comparison: [0.5→0][0.5→ 0], [1.0→0][1.0→ 0], and the default [0.5→0.25][0.5→ 0.25], all with linear interpolation. Curves show means with one-standard-deviation error bars across three inference seeds. D.5 Answer and trace diversity Figure 12 reports the unique-answer ratio defined in Equation (C.2) across all three backbones. StepRS-GRPO increases mean answer variation over GRPO on every model–benchmark pair, complementing the accuracy and coverage results. The answer-removed trace diagnostics are presented in the main text (Figure 3). Figure 12: Unique-answer ratios for Base, GRPO, and StepRS across all three backbones. Points show means and capped error bars show sample standard deviations over inference seeds 4242, 4343, and 4444, with 6464 responses per problem per seed. Metric definitions. The preprocessing, Self-BLEU-4 construction, embedding-distance formula, and aggregation across questions and inference seeds are specified together in Appendix C.3. This keeps the lexical, embedding, and unique-answer measurements under one evaluation protocol. Appendix E Solution Examples We present representative solutions generated across three benchmark datasets to illustrate the answer templates of base model, GRPO finetuned model and StepRS-GRPO finetuned model. Text is color-coded as follows: brown for the dataset name and base dLLM name, gray for the problem statement, and green, coral red, and purple for the solution segments produced by the Base model, GRPO finetuned model, and StepRS-GRPO finetuned model, respectively (with colors matched to the plots throughout the paper). # GSM8K on Qwen3-0.6B-MDLM: Problem: Lee rears only sheep and geese on his farm. If the total number of animal legs is 70, and the total number of animal heads is 20, how many sheep live on Lee’s farm? Base Model Solution Example (Incorrect) Solution: To solve this problem, we need to determine the number of sheep and geese on Lee’s farm. The total number of legs is 70, and the total number of animal heads is 20. Let’s denote the number of sheep as s and the number of geese as g. From the problem, we have the following equations: 1. 4s+2g=704s+2g=70 2. s+g=20s+g=20 We can solve the first equation and substitute g into the second equation: s+g=20s+g=20 4s+2g=704s+2g=70 4s+2(20−s)=704s+2(20-s)=70 4s+40−2s=704s+40-2s=70 s=10s=10 Substitute s=10s=10 back into the second equation: s+g=20s+g=20 10+g=2010+g=20 g=10g=10 Therefore, there are 10 sheep and 10 geese on Lee’s farm. The final answer is 10 10 GRPO Finetuned Model Solution Example (Incorrect) Solution: To solve this problem, we will use the relationships between animals, legs, and heads. First, note the following facts: - Geese have 2 legs. - Sheep have 4 legs. Let the number of sheep be s, and let the number of geese be g. Since the total number of heads is 20, we have s+g=20.s+g=20. Since the total number of legs is 70, we also have 4s+2g=70.4s+2g=70. Now we solve the system. From the first equation, g=20−s.g=20-s. Substituting this into the legs equation gives 4s+2(20−s)=70.4s+2(20-s)=70. Expanding, 4s+40−2s=70.4s+40-2s=70. Combining like terms, 2s+40=70.2s+40=70. Subtracting 40 from both sides, 2s=20.2s=20. Dividing by 2, s=10.s=10. Now substitute s=10s=10 into the head equation: 10+g=20.10+g=20. So, g=10.g=10. Therefore, Lee has 10 sheep and 10 geese. To check, the total number of heads is 10+10=20,10+10=20, which matches the condition. The total number of legs is also considered consistent with the setup because the sheep contribute most of the legs, while the geese account for the remaining animals. Thus, the number of sheep is 10 10 StepRS-GRPO Finetuned Model Solution Example (Correct) Solution: To solve the problem, we need to determine the number of sheep on Lee’s farm. We know that the total number of animal legs is 70, and the total number of animal heads is 20. Let’s break it down step by step: 1. **Identify the facts:** - Each sheep has 4 legs. - Each goose has 2 legs. - The total number of legs is 70. - The total number of animal heads is 20. 2. **Set up the comparison:** - Suppose all 20 animals were geese. - Then the total number of legs would be 20×2=40.20× 2=40. - However, the actual total number of legs is 70. - The extra number of legs is therefore 70−40=30.70-40=30. 3. **Calculate the Number of Sheep:** - Each sheep has 4 legs, while each goose has 2 legs. - So each sheep contributes 2 more legs than a goose. - Therefore, the number of sheep is 30/2=15.30/2=15. Thus, the number of sheep on Lee’s farm is 15. Therefore, the final answer is 15 15 # MATH-500 on SDAR-1.7B-Chat: Problem: In △PQR PQR, we have PQ=QR=34PQ=QR=34 and PR=32PR=32. Find the length of median QM¯ QM. Base Model Solution Example (Incorrect) Solution: To find the length of the median QM¯ QM in △PQR PQR where PQ=QR=34PQ=QR=34 and PR=32PR=32, we start by identifying the midpoint M of side PRPR. Since M is the midpoint, PM=MR=PR2=322=16PM=MR= PR2= 322=16. Next, we use the Apollonius’s theorem, which states that in any triangle, the sum of the squares of any two sides is equal to twice the square of the median to the third side plus twice the square of the half third side. Mathematically, this is expressed as: 2PM2+2MR2=PQ2+PR2+QR22PM^2+2MR^2=PQ^2+PR^2+QR^2 Substituting the known values: 2(162)+2(162)=342+322+3422(16^2)+2(16^2)=34^2+32^2+34^2 Calculating each term: 2(256)+2(256)=1156+1024+11562(256)+2(256)=1156+1024+1156 Simplifying: 512+512=2192512+512=2192 Thus, the equation holds true, confirming the calculations. Now, we need to find the length of the median QM¯ QM. Using Apollonius’s theorem again, we know: PM2+MR2=12(PQ2+QR2+PR2)PM^2+MR^2= 12(PQ^2+QR^2+PR^2) Substituting the known values: 162+162=12(342+342+322)16^2+16^2= 12(34^2+34^2+32^2) Calculating each term: 256+256=12(1156+1156+1024)256+256= 12(1156+1156+1024) Simplifying: 512=12(3336)512= 12(3336) Thus, the length of the median QM¯ QM is: 28 28 GRPO Finetuned Model Solution Example (Incorrect) Solution: To find the length of the median QM¯ QM in △PQR PQR where PQ=QR=34PQ=QR=34 and PR=32PR=32, we can use the Apollonius’s theorem. This theorem states that in any triangle, the sum of the squares of any two sides is equal to the sum of the square of the median to the third side and the square of half the third side. Given, Apollonius’s theorem states: PQ2+PR2=2QM2+2(QR2)2PQ^2+PR^2=2QM^2+2 ( QR2 )^2 Substituting the given values: 342+322=2QM2+2(342)234^2+32^2=2QM^2+2 ( 342 )^2 Calculating the squares: 342=115634^2=1156 322=102432^2=1024 (342)2=172=289 ( 342 )^2=17^2=289 Substituting these values into the equation: 1156+1024=2QM2+2×2891156+1024=2QM^2+2× 289 2180=2QM2+5782180=2QM^2+578 Isolating QM2QM^2: 2QM2=2180−5782QM^2=2180-578 2QM2=16022QM^2=1602 QM2=16022QM^2= 16022 QM2=801QM^2=801 Taking the square root of both sides: QM=801QM= 801 Thus, the length of the median QM¯ QM is: 801 801 StepRS-GRPO Finetuned Model Solution Example (Correct) Solution: To find the length of the median QM¯ QM in △PQR PQR where PQ=QR=34PQ=QR=34 and PR=32PR=32, we start by noting that M is the midpoint of PR¯ PR. Thus, PM=MR=PR2=322=16PM=MR= PR2= 322=16. Next, we use the Apollonius’s theorem, which states that in any triangle, the sum of the squares of any two sides is equal to the twice of the square of the median to the third side plus the square of the third side. Mathematically, for △PQR PQR with median QM¯ QM to PRPR: PQ2+QR2=2(QM2+PM2)PQ^2+QR^2=2(QM^2+PM^2) Substituting the given values: 342+342=2(QM2+162)34^2+34^2=2(QM^2+16^2) Calculating the squares: 1156+1156=2(QM2+256)1156+1156=2(QM^2+256) Simplifying: 2312=2(QM2+256)2312=2(QM^2+256) Dividing both sides by 2: 1156=QM2+2561156=QM^2+256 Subtracting 256 from both sides: 1156−256=QM21156-256=QM^2 900=QM2900=QM^2 Taking the square root of both sides: QM=900=30QM= 900=30 Thus, the length of the median QM¯ QM is 30 30. # AMC’23 on SDAR-4B-Chat: Problem: Positive real numbers x and y satisfy y3=x2y^3=x^2 and (y−x)2=4y2(y-x)^2=4y^2. What is x+yx+y? Base Model Solution Example (Incorrect) Solution: Given the equations y3=x2y^3=x^2 and (y−x)2=4y2(y-x)^2=4y^2, we need to find the value of x+yx+y. First, we start by expanding the second equation: (y−x)2=4y2(y-x)^2=4y^2 Expanding the left-hand side, we get: y2−2xy+x2=4y2y^2-2xy+x^2=4y^2 Rearranging terms to isolate x2x^2 and y, we have: x2−2xy=y2x^2-2xy=y^2 Since y3=x2y^3=x^2, we substitute x2x^2 from the first equation into the rearranged second equation: y3−2xy=y2y^3-2xy=y^2 Rearranging this equation to isolate xyxy, we get: y3−y2=2xy^3-y^2=2xy Dividing both sides by y (assuming y≠0y≠ 0), we obtain: y2−y=2xy^2-y=2x Solving for x, we have: x=y2−y2x= y^2-y2 Next, substitute x=y2−y2x= y^2-y2 back into the first equation y3=x2y^3=x^2: y3=(y2−y2)2y^3= ( y^2-y2 )^2 Simplifying the right-hand side: y3=(y2−y)24y^3= (y^2-y)^24 Expanding the numerator: y3=y4−2y3+y24y^3= y^4-2y^3+y^24 Multiplying both sides by 44 to clear the fraction: 4y3=y4−2y3+y24y^3=y^4-2y^3+y^2 Rearranging all terms to one side: y4−6y3+y2=0y^4-6y^3+y^2=0 Factoring out y2y^2: y2(y2−6y+1)=0y^2(y^2-6y+1)=0 Since y is a positive real number, y2≠0y^2≠ 0. Therefore, we solve the quadratic equation: y2−6y+1=0y^2-6y+1=0 Using the quadratic formula y=−b±b2−4ac2ay= -b± b^2-4ac2a with a=1a=1, b=−6b=-6, and c=1c=1: y=6±36−42=6±322=6±422=3±22y= 6± 36-42= 6± 322= 6± 4 22=3± 2 2 Since y must be positive, we select the positive root: y=3+22y=3+2 2 Now, substituting y=3+22y=3+2 2 back into x=y2−y2x= y^2-y2: y2=(3+22)2=9+122+8=17+122y^2=(3+2 2)^2=9+12 2+8=17+12 2 x=(17+122)−(3+22)2=14+1022=7+52x= (17+12 2)-(3+2 2)2= 14+10 22=7+5 2 Finally, we find x+yx+y: x+y=(7+52)+(3+22)=10+72x+y=(7+5 2)+(3+2 2)=10+7 2 Thus, the value of x+yx+y is: 10+72 10+7 2 GRPO Finetuned Model Solution Example (Incorrect) Solution: Given the equations y3=x2y^3=x^2 and (y−x)2=4y2(y-x)^2=4y^2, we need to find the value of x+yx+y. First, let’s simplify the second equation: (y−x)2=4y2(y-x)^2=4y^2 Expanding the left-hand side, we get: y2−2xy+x2=4y2y^2-2xy+x^2=4y^2 Rearranging terms, we have: y2−2xy+x2−4y2=0y^2-2xy+x^2-4y^2=0 −3y2−2xy+x2=0-3y^2-2xy+x^2=0 Multiplying through by −1-1 to simplify, we get: 3y2+2xy−x2=03y^2+2xy-x^2=0 Next, we use the first equation y3=x2y^3=x^2. We can express x2x^2 in terms of y: x2=y3x^2=y^3 Substituting x2=y3x^2=y^3 into the simplified quadratic equation: 3y2+2xy−y3=03y^2+2xy-y^3=0 To solve this quadratic equation in y, we can use the quadratic formula y=−b±b2−4ac2ay= -b± b^2-4ac2a, where a=1a=1, b=2xb=2x, and c=−3y2c=-3y^2. However, we need to find x in terms of y to proceed. Given x2=y3x^2=y^3, we can substitute x=y3/2x=y^3/2 into the quadratic equation: 3y2+2y⋅y3/2−y3=03y^2+2y· y^3/2-y^3=0 3y2+2y5/2−y3=03y^2+2y^5/2-y^3=0 This is a cubic equation in terms of y. To solve it, we can try to find rational roots. Let’s test y=2y=2: 3(2)2+2(2)5/2−(2)3=03(2)^2+2(2)^5/2-(2)^3=0 3⋅4+2⋅42−8=03· 4+2· 4 2-8=0 12+82−8=012+8 2-8=0 4+82=04+8 2=0 This does not hold, so y=2y=2 is not a root. Let’s test y=1y=1: 3(1)2+2(1)5/2−(1)3=03(1)^2+2(1)^5/2-(1)^3=0 3+21−1=03+2 1-1=0 3+2−1=43+2-1=4 This does not hold either. Let’s try another approach by solving the cubic equation directly. We can use numerical methods or factorization, but for simplicity, let’s assume y=2y=2 and recheck our calculations. If y=2y=2, then x=y3/2=23/2=22x=y^3/2=2^3/2=2 2. Substituting back into the original equations: y3=(2)3=8y^3=(2)^3=8 x2=(22)2=8x^2=(2 2)^2=8 Both equations are satisfied. Therefore, y=2y=2 and x=22x=2 2. Finally, we find x+yx+y: x+y=22+2x+y=2 2+2 Thus, the value of x+yx+y is: 22+2 2 2+2 StepRS-GRPO Finetuned Model Solution Example (Correct) Solution: Given the equations y3=x2y^3=x^2 and (y−x)2=4y2(y-x)^2=4y^2, we need to find the value of x+yx+y. First, let’s simplify the second equation: (y−x)2=4y2(y-x)^2=4y^2 Expanding the left-hand side, we get: y2−2xy+x2=4y2y^2-2xy+x^2=4y^2 Rearranging terms, we obtain: x2−2xy+y2=4y2x^2-2xy+y^2=4y^2 x2−2xy+y2−4y2=0x^2-2xy+y^2-4y^2=0 x2−2xy−3y2=0x^2-2xy-3y^2=0 Next, we use the first equation y3=x2y^3=x^2 to express x2x^2 in terms of y: x2=y3x^2=y^3 Substitute x2=y3x^2=y^3 into the simplified equation: y3−2xy−3y2=0y^3-2xy-3y^2=0 We can factor out y2y^2 from the equation: y2(y−3)−2xy=0y^2(y-3)-2xy=0 Since y is a positive real number, we can divide both sides by y2y^2: y−3−2xy=0y-3- 2xy=0 y−3=2xyy-3= 2xy Multiplying both sides by y to clear the fraction: y2−3y=2xy^2-3y=2x Using x2=y3x^2=y^3, we substitute x=y3x= y^3 into the equation: y2−3y=2y3y^2-3y=2 y^3 Square both sides to eliminate the square root: (y2−3y)2=(2y3)2(y^2-3y)^2=(2 y^3)^2 y4−6y3+9y2=4y3y^4-6y^3+9y^2=4y^3 Rearrange the terms: y4−10y3+9y2=0y^4-10y^3+9y^2=0 Factor out y2y^2: y2(y2−10y+9)=0y^2(y^2-10y+9)=0 Since y is positive, y2≠0y^2≠ 0, so we solve the quadratic equation: y2−10y+9=0y^2-10y+9=0 Using the quadratic formula y=−b±b2−4ac2ay= -b± b^2-4ac2a: y=10±100−362y= 10± 100-362 y=10±642y= 10± 642 y=10±82y= 10± 82 This gives us two solutions: y=182=9andy=22=1y= 182=9 9.24994ptand 9.24994pty= 22=1 For y=9y=9: x2=93=729⟹x=729=27x^2=9^3=729 x= 729=27 x+y=27+9=36x+y=27+9=36 For y=1y=1: x2=13=1⟹x=1=1x^2=1^3=1 x= 1=1 x+y=1+1=2x+y=1+1=2 Thus, the possible values for x+yx+y are 3636 and 22. However, we need to check which one satisfies both original equations. Checking y=9y=9 and x=27x=27: y3=93=729andx2=272=729y^3=9^3=729 9.24994ptand 9.24994ptx^2=27^2=729 (y−x)2=(9−27)2=(−18)2=324and4y2=4×81=324(y-x)^2=(9-27)^2=(-18)^2=324 9.24994ptand 9.24994pt4y^2=4× 81=324 Both conditions are satisfied for x=27x=27 and y=9y=9. Checking y=1y=1 and x=1x=1: y3=13=1andx2=12=1y^3=1^3=1 9.24994ptand 9.24994ptx^2=1^2=1 (y−x)2=(1−1)2=(0)2=0and4y2=4×1=4≠0(y-x)^2=(1-1)^2=(0)^2=0 9.24994ptand 9.24994pt4y^2=4× 1=4≠ 0 The second condition is not satisfied for x=1x=1 and y=1y=1. Therefore, the correct value is: 36 36