Paper deep dive
CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing
Yuji Ren, Chenkai Xu, Zhuocheng Gong, Jianguo Li, Zhijie Deng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/17/2026, 4:36:40 AM
Summary
The paper introduces Consistency Forcing (CForce), a distillation method for Diffusion Large Language Models (dLLMs) that improves parallel decoding speed and quality. CForce aligns mask predictions from early denoising stages with those from later stages using self-rollout trajectories and a Confidence Adaptive KL Divergence (CAD) objective. This approach reduces error propagation in aggressive parallelism strategies and applies to both mask-to-token and edit-capable decoding models like LLaDA.
Entities (6)
Relation Signals (5)
CForce → improves → dLLMs
confidence 95% · CForce trains the model on pre-collected self-rollout trajectories, thereby improving training-inference alignment.
CForce → testedon → LLaDA
confidence 93% · Experiments on non-edit and edit-capable LLaDA models show improved speed-quality trade-offs
CForce → uses → Confidence Adaptive KL Divergence
confidence 92% · We introduce Confidence Adaptive KL Divergence as a distillation objective to conjoin the merits of forward and reverse KL.
CForce → appliesto → mask-to-token decoding
confidence 90% · Critically, the same formulation applies to both mask-to-token decoding and edit-capable decoding
CForce → appliesto → edit-capable decoding
confidence 90% · Critically, the same formulation applies to both mask-to-token decoding and edit-capable decoding
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion large language models (dLLMs) accelerate language generation by predicting multiple masks in a single forward pass. However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, leading to errors that can propagate to later stages. To tackle this issue, we present Consistency Forcing (CForce) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages. CForce trains the model on pre-collected self-rollout trajectories, thereby improving training-inference alignment. We introduce Confidence Adaptive KL Divergence as a distillation objective to conjoin the merits of forward and reverse KL. We further provide a theoretical analysis for the consistency objective to explain why CForce can approximately minimize the prediction error of early stages. Critically, the same formulation applies to both mask-to-token decoding and edit-capable decoding; in the edit-capable case, later token-to-token refinements provide additional supervision for earlier masked-state predictions. Experiments on non-edit and edit-capable LLaDA models show improved speed-quality trade-offs, especially under high-parallelism decoding budgets. Code is available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.13925v1
- Canonical: https://arxiv.org/abs/2608.13925v1
Trouble viewing inline? Open PDF directly →
Full Text
63,187 characters extracted from source content.
Expand or collapse full text
CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing Yuji Ren Affiliation: Shanghai Jiao Tong University Chenkai Xu Affiliation: Shanghai Jiao Tong University Zhuocheng Gong Affiliation: Ant Grouprenyj26, 132435xck, zhijied@sjtu.edu.cn, lijg.zero@antgroup.com Jianguo Li Zhijie Deng Abstract Diffusion large language models (dLLMs) accelerate language generation by predicting multiple masks in a single forward pass. However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, leading to errors that can propagate to later stages. To tackle this issue, we present Consistency Forcing (CForce) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages. CForce trains the model on pre-collected self-rollout trajectories, thereby improving training-inference alignment. We introduce Confidence Adaptive KL Divergence as a distillation objective to conjoin the merits of forward and reverse KL. We further provide a theoretical analysis for the consistency objective to explain why CForce can approximately minimize the prediction error of early stages. Critically, the same formulation applies to both mask-to-token decoding and edit-capable decoding; in the edit-capable case, later token-to-token refinements provide additional supervision for earlier masked-state predictions. Experiments on non-edit and edit-capable LLaDA models show improved speed-quality trade-offs, especially under high-parallelism decoding budgets. Code is available at: https://github.com/inclusionAI/dFactory. 11footnotetext: Equal contribution. 22footnotetext: Corresponding authors. 1 Introduction Figure 1: Speed–quality trade-off: accuracy vs. tokens per forward pass (TPF). Upper-right is better. Consistency Forcing moves both models toward a higher-parallelism trade-off. Figure 2: Per-task Accuracy Under Parallelism (AUP). AUP integrates the score–parallelism curve into a single metric. CForce yields consistent gains across all four benchmarks. Diffusion large language models (dLLMs) (Austin et al. 2021a; Zheng et al. 2023; Lou et al. 2024; Sahoo et al. 2024; Nie et al. 2026; Ye et al. 2025; Arriola et al. 2025; Gong et al. 2025) generate text by iteratively denoising masked sequences, committing multiple tokens per forward pass to achieve high parallelism (Wu et al. 2025; Bie et al. 2025; Bie et al. 2026; Wang et al. 2025; Cheng et al. 2025). However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, and the resulting errors can propagate to later stages. Thus, faster dLLM decoding depend not only on reducing denoising steps, but also on improving the reliability of early-stage predictions under high-parallelism decoding. This issue is especially important for edit-capable dLLMs. Conventional masked diffusion decoding is mainly mask-to-token (M2T), whereas edit-capable models such as LLaDA2.1 (Bie et al. 2026) also perform token-to-token (T2T) refinement, allowing later denoising states to revise earlier drafts. These later states contain richer context and possible T2T corrections, which could provide stronger supervision for early M2T predictions before token commitment. Existing acceleration and distillation methods do not train this early-to-late consistency along the model’s own threshold-decoding trajectory: some modify decoding or caching (Kim et al. 2026; Wu et al. 2025; Liu et al. 2025b), while others use constructed, teacher-generated, or privileged trajectories (Zhang et al. 2026; Kim et al. 2025; Liang et al. 2026) whose information conditions differ from the student’s inference states. We propose Consistency Forcing (CForce) for dLLMs, a distillation method inspired by consistency models (Salimans and Ho 2022; Song et al. 2023) that forces the mask predictions of early stages to align with those of later stages. Specifically, starting from a pretrained dLLM, CForce trains the model on self-rollout trajectories, thereby improving training-inference alignment. Then, CForce partitions each pre-collected trajectory into stages. Each stage boundary is set after a fixed number of newly revealed masked tokens, rather than at every native step, which would incur substantial trajectory-storage overhead. For each adjacent pair, the earlier-stage prediction is trained to match a stop-gradient prediction from the same model at the later stage, without relying on a frozen teacher. Still-masked positions are aligned with Confidence Adaptive KL Divergence (CAD), which dynamically interpolates between forward and reverse KL based on later-stage prediction confidence. CForce also applies a cross-entropy (CE) anchor to stabilize token commitment. A curriculum over reveal difficulty further stabilizes this adjacent-stage objective (Xu et al. 2025b; Liu et al. 2025a). We further provide a theoretical analysis supporting this design: early-stage prediction error is upper-bounded by adjacent-stage distributional drift plus reveal-boundary token error (Section 4.4), two terms that correspond directly to the CAD and CE components of our objective. Empirically, the clearest result appears on the edit-capable LLaDA2.1-mini (Bie et al. 2026): Consistency Forcing increases average Tokens Per Forward (TPF) from 6.94 to 9.08 while improving average accuracy from 85.57 to 86.41. On the non-edit LLaDA2.0-mini (Bie et al. 2025), the method shows an explicit speed-quality trade-off, increasing average TPF from 3.60 to 6.42 and improving few-step accuracy at fixed TPF budgets under aggressive decoding. Figure 1 and Figure 2 visualize this effect. Our contributions are fourfold: (1) we reframe dLLM acceleration around early-stage reliability under aggressive threshold decoding, rather than focusing only on fewer sampling steps; (2) we introduce Consistency Forcing, which aligns adjacent stages on the model’s own decoding trajectory using CAD and a CE anchor; (3) we instantiate this idea in edit-capable dLLMs, where later T2T-refined stages supervise earlier M2T predictions; and (4) we show that the same recipe improves few-step and high TPF behavior on a non-edit dLLM, with an explicit speed-quality trade-off. 2 Related Work Masked Diffusion Language Models. Masked diffusion language models (MDLMs) cast text generation as discrete denoising (Austin et al. 2021a; Zheng et al. 2023; Lou et al. 2024). Building on earlier masked generation methods (Ghazvininejad et al. 2019; Chang et al. 2022), recent dLLMs further scale this paradigm, enabling stronger text and code generation (Nie et al. 2026; Ye et al. 2025; Gong et al. 2025). To improve generation speed, several dLLMs introduce confidence-based parallel decoding (Wu et al. 2025; Wang et al. 2025; Cheng et al. 2025; Bie et al. 2025; Chen et al. 2026), while edit-capable models allow later denoising states to revise committed tokens (Bie et al. 2026). These later corrections are especially relevant under aggressive parallel decoding, where early M2T predictions are made with limited context and can benefit from supervision induced by later denoising states. Acceleration Methods for dLLMs. Existing acceleration methods follow several complementary directions. Inference-time approaches improve token selection or verification without changing model parameters (Kim et al. 2026; Xu et al. 2025a; Agrawal et al. 2025; Gao et al. 2025), while system-level approaches reduce computation through caching (Wu et al. 2025; Ma et al. 2026; Liu et al. 2025b). Training-based approaches instead improve few-step generation with constructed, teacher-generated, or privileged trajectories (Chen et al. 2025b; Zhang et al. 2026; Kim et al. 2025; Liang et al. 2026). These methods accelerate decoding but do not directly train the model on the low-context states produced by its own threshold-decoding path. Consistency Distillation in Diffusion Models. Diffusion distillation reduces sampling cost by compressing slow multi-step generation into faster samplers or models (Luhman and Luhman 2021; Salimans and Ho 2022; Meng et al. 2023). Consistency models extend this idea by enforcing agreement across trajectory states, enabling one-step or few-step generation (Song et al. 2023; Luo et al. 2023; Kou et al. 2024). Related objectives have also been studied for discrete generative models (Hayakawa et al. 2024; Sahoo et al. 2025; Xu et al. 2025b) and, more recently, for dLLMs (Zhang et al. 2026; Kim et al. 2025; Liang et al. 2026). These works demonstrate the usefulness of cross-state supervision. However, they typically depend on separate teachers, fixed representations, or teacher-generated targets. In contrast, Consistency Forcing constructs adjacent-stage constraints from the model’s own threshold-decoding trajectory without a frozen teacher, and adapts them to both masked-token reveal and edit transitions. 3 Preliminaries Figure 3: An illustration of edit-capable dLLM decoding. At each decoding step, high-confidence masked positions are denoised, while already generated tokens may also be revised to correct earlier mistakes. Masked Diffusion Language Models. Figure 4: Overview of Consistency Forcing. Left: self-rollout threshold-decoding trajectories are downsampled into adjacent stage pairs whenever the cumulative number of M2T reveals reaches the stage size S; right: the earlier low-context state is trained against same-model stop-gradient predictions from the later, more informative state. Masked diffusion language models formulate text generation as denoising in a discrete token space (Austin et al. 2021a; Lou et al. 2024; Sahoo et al. 2024). Given a prompt x∈x and a clean target sequence y=(y1,…,yL)y=(y^1,…,y^L) over vocabulary V, the forward process corrupts y by independently replacing tokens with [MASK][MASK]. Let αt∈[0,1] _t∈[0,1] denote the probability that a token remains unmasked at noise time t, and let sts_t be the corrupted sequence. Under the absorbing-mask process, q(stj∣yj)=αt,stj=yj,1−αt,stj=[MASK].q(s_t^j y^j)= cases _t,&s_t^j=y^j,\\ 1- _t,&s_t^j=[MASK]. cases The reverse model predicts a categorical distribution qθ(⋅∣x,st,j)q_θ(· x,s_t,j) for each masked position j. Let Mt=j:stj=[MASK]M_t=\j:s_t^j=[MASK]\ be the masked positions. Training can be written as the standard variational bound for masked diffusion, which reduces to a weighted cross-entropy over these positions: ℒMDLM _MDLM =(x,y),t,st[w(t)∑j∈MtCEθj], =E_(x,y),t,s_t [w(t) _j∈ M_tCE_θ^j ], (1) CEθj _θ^j =−logqθ(yj∣x,st,j). =- q_θ(y^j x,s_t,j). Here w(t)=−αt′/(1−αt)w(t)=-α _t/(1- _t) is the positive ELBO weight induced by the absorbing diffusion process. This objective trains the model to recover clean tokens from partially observed contexts, enabling multiple positions to be denoised in parallel at inference time. In current large-scale dLLMs, this masked diffusion objective is commonly used together with block diffusion (Arriola et al. 2025; Bie et al. 2025), which partitions the target sequence into blocks and denoises each block in parallel while generating blocks sequentially. Recent edit-capable dLLMs further augment this block-diffusion framework with token-to-token refinement, allowing committed tokens to be revised in later denoising states (Bie et al. 2026). Hereafter, we follow this block-diffusion setting and discuss decoding and training within a block unless otherwise specified. Confidence-based Parallel Decoding. At inference time, block diffusion generates one block by starting from an all-mask block state s0s_0 and iteratively denoising it into the all-clean state. This process produces a trajectory =(s0,s1,…,sT)T=(s_0,s_1,…,s_T), where T is the number of decoding steps allocated to the block. At step t, let Mt=j:stj=[MASK]M_t=\j:s_t^j=[MASK]\ denote the positions that remain masked, and let ptj(v)=qθ(v∣x,st,j)p_t^j(v)=q_θ(v x,s_t,j) be the model distribution at position j∈Mtj∈ M_t. Modern dLLMs reveal multiple positions in each forward pass according to model confidence (Bie et al. 2025). The confidence of a masked position is defined as ctj=maxv∈ptj(v)c_t^j= _v p_t^j(v). Given a threshold τ, all positions whose confidence exceeds the threshold are selected for commitment. To ensure progress, if no position exceeds the threshold, the most confident remaining position is selected: ΔMt=j∈Mt:ctj>τ,if j∈Mt:ctj>τ≠∅,argmaxj∈Mtctj,otherwise. M_t= cases\j∈ M_t:c_t^j>τ\,&if \j∈ M_t:c_t^j>τ\≠ ,\\ \ _j∈ M_tc_t^j\,&otherwise. cases (2) For each selected position j∈ΔMtj∈ M_t, the decoder commits the most probable token, st+1j=argmaxv∈ptj(v),s_t+1^j= _v p_t^j(v), (3) and the remaining positions stay masked. Equivalently, Mt+1=Mt∖ΔMt.M_t+1=M_t M_t. (4) Edit-capable Diffusion Decoding. Recent edit-capable dLLMs extend the confidence-based decoding process by allowing T2T refinement after M2T commitment (Bie et al. 2026). Given an edit threshold τedit _edit, positions selected for editing are Et=j∉Mt:maxv∈,v≠stjqθ(v∣x,st,j)>τedit.E_t= \j∉ M_t: _v ,\,v≠ s_t^jq_θ(v x,s_t,j)> _edit \. (5) For each j∈Etj∈ E_t, the decoder replaces the current token with the most probable alternative, st+1j=argmaxv∈,v≠stjqθ(v∣x,st,j).s_t+1^j= _v ,\,v≠ s_t^jq_θ(v x,s_t,j). (6) As illustrated in Figure 3, in edit-capable decoding, later states can correct tokens that were committed earlier. These T2T refinements are especially important for fast decoding when the M2T threshold τ is low. Consistency Forcing uses this later-stage refinement signal to supervise earlier masked-state predictions. 4 Method 4.1 Overview Consistency Forcing trains a dLLM to make more reliable predictions from early, low-context denoising states. The method collects threshold-decoding trajectories from the pretrained initialization qθ0q_ _0, converts them into staged adjacent pairs, and uses the later stage as a same-model stop-gradient target for the earlier stage. The training objective combines adjacent-stage distributional alignment, a reveal-boundary CE anchor, and a curriculum over transition difficulty; building on this, edit-capable models can further feed later T2T signals back into earlier states to improve M2T drafting. Figure 4 illustrates the overall pipeline. 4.2 Staged Trajectory Construction We need to construct trajectories used for consistency forcing. Specifically, we first collect the original confidence-based decoding trajectory =(s0,s1,…,sT)T=(s_0,s_1,…,s_T) using the cold-start initialization qθ0q_ _0 as described in Section 3. Rather than treating every consecutive native transition in T as a training pair, we use the native rollout only as the source trajectory: dense native-step supervision is inefficient and often provides only a weak training signal, because adjacent native states may differ by very few newly revealed tokens. Next, we maintain a reveal counter along the rollout. Whenever the cumulative number of newly revealed tokens reaches a fixed stage size S, we save the current state as a stage boundary and reset the counter. The initial all-mask state is always saved, and the final decoded state is appended if it has not already been saved. This produces a staged trajectory ¯=(s¯0,s¯1,…,s¯N) T=( s_0, s_1,…, s_N), where adjacent stages are separated by cumulative M2T progress rather than by the raw native update index. We then form adjacent pairs from ¯ T. Downsampling by cumulative M2T reveals forms coarser stages, so each adjacent pair contains a more substantial change in visible context. For each adjacent stage pair (s¯i,s¯i+1)( s_i, s_i+1), let M¯i=j:s¯ij=[MASK] M_i=\j: s_i^j=[MASK]\ denote the positions masked in the earlier stage. We split these positions into i=M¯i∩M¯i+1,Δi=M¯i∖M¯i+1.U_i= M_i∩ M_i+1, _i= M_i M_i+1. (7) Here, iU_i contains positions that remain masked in both stages, while Δi _i contains positions that are revealed between the two stages. These sets define the supports of the CAD and CE terms below. For edit-capable decoding, we also record native edit events during the trajectory rollout and aggregate them between adjacent stage boundaries; the resulting stage-level edit set E¯i E_i is introduced in Section 4.3. Since all stages are obtained from the model’s own confidence-based decoding process, the staged trajectories preserve the mask, reveal, and edit patterns that arise during fast inference. 4.3 Adjacent-stage Consistency Forcing For each sampled adjacent stage pair (s¯i,s¯i+1)( s_i, s_i+1), we use the earlier stage as the student input and the later, more informative stage as the target context. The target branch is evaluated under stop-gradient, denoted by sg[⋅]sg[·], so the model learns to approximate from s¯i s_i the prediction it would make after receiving the additional context in s¯i+1 s_i+1. We instantiate this adjacent-stage constraint with two complementary terms: CAD controls soft distributional drift on positions that remain latent across the transition, while a CE anchor stabilizes positions at the M2T reveal boundary. This decomposition follows the error-bound motivation in Section 4.4. Confidence Adaptive KL Divergence. On iU_i, both stages are still masked, so we align the student distribution pstuj=qθ(⋅∣x,s¯i,j)p_stu^j=q_θ(· x, s_i,j) with the stop-gradient target distribution ptarj=sg[qθ(⋅∣x,s¯i+1,j)]p_tar^j=sg[q_θ(· x, s_i+1,j)]. A standard forward KL minimizes drift but can be weak at sharpening predictions. Conversely, reverse KL is a strong mode-seeking objective but risks premature collapse when applied to uncertain targets. To combine their strengths, we design Confidence Adaptive KL Divergence (CAD), which dynamically mixes forward and reverse KL based on the later stage’s prediction confidence cj=maxvptarj(v)c_j= _vp_tar^j(v), defined as ℒCAD=1|i|∑j∈i[DKL(ptarj∥pstuj)+cjDKL(pstuj∥ptarj)],L_CAD= 1|U_i| _j _i [D_KL(p_tar^j\|p_stu^j)+c_jD_KL(p_stu^j\|p_tar^j) ], (8) where the forward KL term stably controls distributional drift connected to the upper bound in Section 4.4, while the confidence-weighted reverse KL adds mode-seeking pressure only when the later-stage prediction is confident. CE Anchor. On the newly revealed positions Δi _i, we apply a cross-entropy anchor to stabilize token commitments: ℒCE=−1|Δi|∑j∈Δilogqθ(yj∣x,s¯i,j),L_CE=- 1| _i| _j∈ _i q_θ(y^j x, s_i,j), (9) where yjy^j denotes the clean target token at position j. By anchoring the student prediction immediately before reveal, the CE term discourages errors at the boundary where masked positions become committed tokens. Edit-capable dLLMs. As described in Section 3, edit-capable dLLMs provide an additional source of later-stage supervision: T2T edits of already visible tokens. We define the stage-level edit set E¯i E_i as the positions whose visible tokens differ between adjacent stages s¯i s_i and s¯i+1 s_i+1: E¯i=j: E_i=\j: s¯ij≠[MASK],s¯i+1j≠[MASK], s_i^j≠[MASK],\; s_i+1^j≠[MASK], s¯ij≠s¯i+1j. s_i^j≠ s_i+1^j\. (10) Edit-capable trajectories contain both M2T reveals and later T2T corrections, so we broaden the CAD domain from iU_i to M¯i∪E¯i M_i∪ E_i. The M¯i M_i term aligns student-masked positions with later soft predictions under additional revealed context, while E¯i E_i lets visible tokens that change between stages contribute T2T correction signals. Since these later-stage soft distributions provide broader supervision, we also extend the CE anchor from Δi _i to all student-masked positions M¯i M_i. This uses edit-capable trajectories without changing the decoding algorithm. Curriculum Transition. At the beginning of optimization, the full stage gap can be difficult because the student has not yet learned to predict tokens that are revealed many positions ahead. We therefore expose the later-stage context gradually. At optimization step u, only a fraction ρu _u of the newly revealed positions in Δi _i are used to form the later-stage context: ρu=min(1,ρ0+(1−ρ0)uTtrain), _u= \! (1,\; _0+(1- _0) uT_train ), (11) where ρ0=0.1 _0=0.1 and TtrainT_train is the total training steps. Concretely, we select a subset Γi(u)⊆Δi _i(u) _i with |Γi(u)|=⌊ρu|Δi|⌋| _i(u)|= _u| _i| and form a partially revealed later state by revealing only Γi(u) _i(u) while keeping the remaining positions in Δi _i masked. Initially, the target state is close to the student state, stabilizing local consistency learning; as ρu _u increases, the model is gradually exposed to larger adjacent-stage transitions and eventually learns the consistency constraint used by fast decoding. Overall Objective. The training loss is ℒ=ℒCAD+λCEℒCE,L=L_CAD+ _CEL_CE, (12) where λCE _CE balances distributional forcing and token anchoring. 4.4 Theoretical Analysis Setup. We keep the main text focused on the intuition and key conclusion, and defer the full derivation to Appendix C. The goal is to justify why the Consistency Forcing objective targets early-stage reliability. For two categorical distributions p and q over vocabulary V, the total variation (TV) distance is defined as DTV(p,q)=12∑v∈|p(v)−q(v)|.D_TV(p,q)= 12 _v |p(v)-q(v)|. (13) Consider a position j revealed at stage rjr_j, with clean target token yj=s¯Njy^j= s_N^j, and let pij=qθ(⋅∣x,s¯i,j)p_i^j=q_θ(· x, s_i,j). For any earlier masked stage i<rji<r_j, the prediction error at s¯i s_i is measured as DTV(pij,δyj)D_TV(p_i^j, _y^j), where δyj _y^j denotes the one-hot distribution on yjy^j. Main Bound. The early-stage prediction error can be bounded by adjacent-stage distributional drift and the reveal-boundary token error: DTV(pij,δyj)≤∑r=irj−212DKL(pr+1j∥prj)−logprj−1j(yj). splitD_TV(p_i^j, _y^j)&≤ _r=i^r_j-2 12D_KL(p_r+1^j\|p_r^j)\\ & - p_r_j-1^j(y^j). split (14) The first term measures how much the model prediction drifts as more context is revealed before position j is committed, while the second term measures the token-level error immediately before reveal, which is controlled by the CE loss on the clean token under commitment. Connection to CForce. Eq. (14) directly matches our objective: the forward KL component in CAD reduces adjacent-stage drift, while the CE anchor controls the reveal-boundary error. The confidence-weighted reverse KL in CAD is non-negative and therefore preserves this forward KL alignment while sharpening predictions when the later-stage target is confident. Thus, Consistency Forcing optimizes a tractable surrogate for the upper bound on early-stage prediction error. Diffusion type Edit? Model GSM8K MATH500 MBPP HumanEval Score TPF AUP Score TPF AUP Score TPF AUP Score TPF AUP Pure diffusion × dUltra-coding-b32 81.52* 8.40 – 35.64 6.72* – 37.04 7.29* – 35.85 7.23* – d3LLM-LLaDA 73.09 9.11* 637.65* 30.36 5.74 107.64* 40.60 4.21 88.36 39.63 5.95 96.64 d3LLM-Dream 81.36 4.94 391.33 38.21* 3.92 97.50 55.60* 2.96 141.41* 57.10* 3.20 129.48* D2F-LLaDA 74.39 2.88 213.76 28.94 2.66 49.00 39.00 2.13 52.96 40.64 2.69 61.98 Block diffusion × LightningRL-8B-b32 90.30 5.58 492.40 63.00 6.28 407.50 58.30 11.10 641.60 72.60 6.30 450.10 LLaDA2.0-mini 93.25 2.85 266.82 81.80 3.46 286.71 78.69 3.80 305.84 83.54 4.28 367.52 LLaDA2.0-mini-CAP 91.74 4.09 374.03 81.00 4.85 394.10 72.13 5.43 400.00 73.18 5.89 387.79 LLaDA2.0-mini-CForce (Ours) 91.74 5.36 489.07 79.20 6.35 509.23 73.30 6.00 387.80 79.27 7.97 625.51 ✓ DMax 92.10 5.48 557.00 75.40 5.94 507.00 79.20 5.86 482.00 83.50 7.36 637.00 LLaDA2.1-mini 93.56 5.94 666.71 85.00 7.44 741.42 77.75 7.25 636.16 85.98 7.11 705.47 LLaDA2.1-mini-CForce (Ours) 92.27 7.63 845.03 84.80 10.15 966.35 81.97 10.07 1016.14 86.59 8.48 870.90 Table 1: Main Comparison by Diffusion Type and Edit Capability. Each benchmark group reports Score, TPF, and AUP. Bold and underline mark the best and second-best metrics within each comparable block; stars mark the best pure-diffusion results. 5 Experiments 5.1 Experimental Setup Models. We evaluate CForce on two LLaDA variants: LLaDA2.0-mini for the non-edit setting and LLaDA2.1-mini for the edit-capable setting described in Section 4.3. Detailed implementation settings are provided in Appendix B. Training Data. We use queries from OpenMath-Instruct-2 (Toshniwal et al. 2025) and OpenCodeInstruct (Ahmad et al. 2025) and decode responses using Algorithm 1. For LLaDA2.0, we use a confidence threshold τ=0.95τ=0.95; for LLaDA2.1, we use τ=0.85τ=0.85 and τedit=0.5 _edit=0.5. Responses exceeding 4096 tokens are discarded. Evaluation. We evaluate on four benchmarks that span mathematical reasoning and code generation: GSM8K (Cobbe et al. 2021), MATH500 (Lightman et al. 2024), MBPP (Austin et al. 2021b), and HumanEval (Chen et al. 2021). Inference is performed with SGLang (Zheng et al. 2024) using a block size of 32 and a max generation length of 4096. We report Score, TPF, and AUP (Qian et al. 2026). Score denotes exact-match accuracy on GSM8K and MATH500 and pass@1 on MBPP and HumanEval; higher TPF and AUP indicate greater parallelism and a better speed-quality operating point, respectively. Appendix D provides the exact operating points used to compute AUP for the LLaDA-family rows. Throughput results on GSM8K and HumanEval, together with the hardware setup, are reported in Appendix E. Baselines. We include LLaDA2.0-mini and LLaDA2.1-mini as the base models for the non-edit and edit-capable settings, respectively (Bie et al. 2025; Bie et al. 2026). For the LLaDA2.0-series comparison, we also include LLaDA2.0-mini-CAP, a confidence-aware parallel training variant of LLaDA2.0-mini, and refer to it as CAP (Chen et al. 2025b). Beyond these controlled baselines, Table 1 reports representative recent dLLM acceleration methods: dUltra-coding-b32, d3LLM-LLaDA, d3LLM-Dream, D2F-LLaDA, LightningRL-8B-b32, and DMax (Chen et al. 2025a; Qian et al. 2026; Wang et al. 2025; Hu et al. 2026; Chen et al. 2026). For LLaDA2.0-mini, CAP, and our LLaDA2.0-mini-CForce, evaluation uses τ=0.9τ=0.9; for LLaDA2.1-mini and LLaDA2.1-mini-CForce, evaluation uses τ=0.7τ=0.7 and τedit=0.5 _edit=0.5. 5.2 Main Results Edit-capable dLLMs. Table 1 reports controlled comparisons between each LLaDA baseline and its Consistency Forcing variant under the corresponding decoding setting, with the strongest gains appearing in the edit-capable case. On LLaDA2.1-mini, Consistency Forcing increases average TPF from 6.94 to 9.08 while also improving average score from 85.57 to 86.41. The improvement is strongest on the code benchmarks and comes with only minor changes on the math benchmarks. This result is consistent with the method design: edit-capable trajectories allow later corrections on already decoded tokens to feed back into the masked-position drafting ability of earlier states through the stop-gradient target. Non-edit dLLMs. Under the non-edit decoding setting, Consistency Forcing on LLaDA2.0-mini moves the model to a higher-parallelism operating point: it reaches 6.42 average TPF, compared with 3.60 for the base model and 5.07 for CAP. Its average score is lower than the base model but higher than CAP, making this setting an explicit speed-quality trade-off. Figure 5: Few-step Generation Results (Score %) on LLaDA2.0 Series with Fixed TPF Budgets. Vanilla, CAP, and Ours represent LLaDA2.0-mini, LLaDA2.0-mini-CAP, and LLaDA2.0-mini-CForce respectively. Stage size S AVG Score AVG TPF 4 80.62 5.64 8 80.88 6.42 16 75.05 7.97 Table 2: Ablation on Stage Size for LLaDA2.0-mini-CForce. Full benchmark-level results are in Appendix F. The shaded row denotes the default setting; bold marks the best value in each metric. Training AVG Curriculum Frozen Teacher Score TPF ✓ 80.88 6.42 ✓ ✓ 79.49 5.91 79.87 5.97 Table 3: Ablation on Curriculum Learning and Target Type for LLaDA2.0-mini-CForce. The frozen-teacher variant changes only the target predictor; the no-curriculum variant removes the gradual exposure schedule from the same-model stop-gradient target. Few-step Generation. Figure 5 evaluates fixed TPF decoding on the LLaDA2.0 series. We constrain the decoding budget so that the model operates at a fixed TPF of 8 or 4, which forces substantially more tokens to be committed per forward pass than in the standard dynamic-threshold setting. Consistency Forcing obtains the best average score at both budgets, with the largest margin under the more aggressive TPF =8=8 setting. This supports the main hypothesis that adjacent-stage forcing is most useful when early predictions must directly support large parallel commitments. 5.3 Ablation Studies We conduct ablations on LLaDA2.0-mini-CForce, where the confidence-threshold results expose a clear speed-quality trade-off. We vary the stage size, curriculum schedule, KL divergence form, CE anchor weight, and target type while keeping the rest of the training setup fixed. All rows are evaluated with threshold decoding at τ=0.9τ=0.9 and maximum generation length 4096. Stage Size. Table 3 studies the number of newly revealed tokens between saved trajectory stages. A small stage size (S=4S=4) yields nearby state pairs and lower parallelism, reaching 80.62 average score and 5.64 average TPF. Increasing the stage size to 16 raises average TPF to 7.97 but lowers average score to 75.05, consistent with larger transitions being harder consistency targets. We therefore use S=8S=8, which provides the best observed balance in this ablation, with 80.88 average score and 6.42 average TPF. Curriculum Learning. Table 3 isolates the curriculum schedule (Eq. 11), which gradually increases the fraction of newly revealed positions exposed by the later trajectory state from 10% to 100% over training. Removing the schedule lowers average score from 80.88 to 79.87 and average TPF from 6.42 to 5.97, confirming that gradual exposure to larger context gaps stabilizes learning. This suggests that the model benefits from first learning local adjacent-stage consistency before being exposed to the full transition gap. Target Type. Table 3 also compares the same-model stop-gradient target used by Consistency Forcing with a frozen-teacher variant. The same-model target outperforms the frozen-teacher baseline on both score (+1.39) and TPF (+0.51), suggesting that an evolving target better tracks the student’s own decoding distribution and provides more aligned supervision for its inference states. Component Variant AVG Score TPF KL divergence Forward KL 85.75 5.07 Reverse KL 65.57 8.72 CAD 80.88 6.42 CE anchor λCE=0 _CE=0 79.02 6.52 λCE=0.1 _CE=0.1 80.88 6.42 λCE=2.0 _CE=2.0 80.22 6.52 Table 4: Ablations of the KL Divergence Form and CE Anchor Weight on LLaDA2.0-mini-CForce. KL Divergence Form. Table 4 compares forward KL, reverse KL, and our CAD. Forward KL gives the highest average score, but it remains conservative in parallelism, with 5.07 average TPF. Reverse KL reaches much higher parallelism, 8.72 average TPF, but its average score drops to 65.57, consistent with an overly aggressive mode-seeking objective under threshold decoding. CAD selects an intermediate operating point: compared with forward KL, it increases average TPF from 5.07 to 6.42; compared with reverse KL, it avoids the large score collapse. CAD thus balances conservative drift control with controlled sharpening. CE Anchor. The lower block of Table 4 studies the CE anchor weight. Removing the anchor lowers average score from 80.88 to 79.02 while leaving TPF nearly unchanged, suggesting that the CE term improves the score side of the trade-off. A larger weight, λCE=2.0 _CE=2.0, recovers part of the score but remains below the moderate setting on average and drops noticeably on HumanEval. We therefore use λCE=0.1 _CE=0.1 as a modest token-level anchor: it improves average score while preserving the adjacent-stage KL-driven parallelism. 6 Conclusion We presented Consistency Forcing, a training method for improving the speed-quality frontier of diffusion language models under aggressive threshold decoding. The method constructs staged trajectories from the model’s own decoding path and aligns earlier low-context states with later, more informative states through same-model stop-gradient supervision. By combining Confidence Adaptive KL Divergence, a CE anchor, and a curriculum over reveal difficulty, Consistency Forcing improves the reliability of early-stage predictions and supports higher parallelism. We further extended the framework to edit-capable dLLMs, where later T2T refinements provide useful supervision for earlier M2T predictions. Experiments on both edit-capable and conventional dLLMs show that this trajectory-based forcing improves high-parallelism decoding while preserving the generation quality. Limitations For training stability, Consistency Forcing currently relies on trajectories collected in advance from a pretrained initialization. Although these trajectories are produced by the model’s own threshold-decoding process, they are still fixed before optimization begins. As training changes the model, the cached trajectories may gradually differ from the actual inference trajectories visited by the updated model. This remaining mismatch can limit how closely the training signal follows the final student distribution. In future work, we plan to explore online trajectory collection and policy-updating strategies so that the forcing objective can adapt to the model’s evolving inference behavior. Acknowledgments This work was supported by Ant Group Research Fund. References Agrawal et al. (2025) Sudhanshu Agrawal, Risheek Garrepalli, Raghavv Goel, Mingu Lee, Christopher Lott, and Fatih Porikli. 2025. Spiffy: Multiplying diffusion llm acceleration via lossless speculative decoding. arXiv preprint arXiv:2509.18085. Ahmad et al. (2025) Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, and Boris Ginsburg. 2025. Opencodeinstruct: A large-scale instruction tuning dataset for code llms. arXiv preprint arXiv:2504.04030. Arriola et al. (2025) Marianne Arriola, Aaron Gokaslan, Justin Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, volume 2025, pages 50726–50753. Austin et al. (2021a) Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. 2021a. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993. Austin et al. (2021b) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021b. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Bie et al. (2026) Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, and 1 others. 2026. Llada2. 1: Speeding up text diffusion via token editing. arXiv preprint arXiv:2602.08676. Bie et al. (2025) Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, and 1 others. 2025. Llada2. 0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745. Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. 2022. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Chen et al. (2025a) Shirui Chen, Jiantao Jiao, Lillian J Ratliff, and Banghua Zhu. 2025a. dultra: Ultra-fast diffusion language models via reinforcement learning. arXiv preprint arXiv:2512.21446. Chen et al. (2025b) Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. 2025b. dparallel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488. Chen et al. (2026) Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. 2026. Dmax: Aggressive parallel decoding for dllms. arXiv preprint arXiv:2604.08302. Cheng et al. (2025) Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and 1 others. 2025. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Gao et al. (2025) Yifeng Gao, Ziang Ji, Yuxuan Wang, Biqing Qi, Hanlin Xu, and Linfeng Zhang. 2025. Self speculative decoding for diffusion large language models. arXiv preprint arXiv:2510.04147. Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 6112–6121. Gong et al. (2025) Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. 2025. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639. Hayakawa et al. (2024) Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. 2024. Distillation of discrete diffusion through dimensional correlations. arXiv preprint arXiv:2410.08709. Hu et al. (2026) Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, and Zhijie Deng. 2026. Lightningrl: Breaking the accuracy-parallelism trade-off of block-wise dllms via reinforcement learning. arXiv preprint arXiv:2603.13319. Kim et al. (2025) Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman Singh, Ben Athiwaratkun, Ce Zhang, Kurt Keutzer, and Amir Gholami. 2025. Cdlm: Consistency diffusion language models for faster sampling. arXiv preprint arXiv:2511.19269. Kim et al. (2026) Seo Hyun Kim, Sunwoo Hong, Hojung Jung, Youngrok Park, and Se-Young Yun. 2026. Klass: Kl-guided fast inference in masked diffusion models. Advances in Neural Information Processing Systems, 38:92267–92301. Kou et al. (2024) Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. 2024. Cllms: Consistency large language models. In Forty-first International Conference on Machine Learning. Liang et al. (2026) Yihao Liang, Ze Wang, Hao Chen, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Emad Barsoum, Zicheng Liu, and Niraj K Jha. 2026. Cd4lm: Consistency distillation and adaptive decoding for diffusion language models. arXiv preprint arXiv:2601.02236. Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pages 39578–39601. Liu et al. (2025a) Yunpeng Liu, Boxiao Liu, Yi Zhang, Xingzhong Hou, Guanglu Song, Yu Liu, and Haihang You. 2025a. See further when clear: Curriculum consistency model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18103–18112. Liu et al. (2025b) Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. 2025b. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint arXiv:2506.06295. Lou et al. (2024) Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion modeling by estimating the ratios of the data distribution, 2024. URL https://arxiv. org/abs/2310.16834. Luhman and Luhman (2021) Eric Luhman and Troy Luhman. 2021. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388. Luo et al. (2023) Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. 2023. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. Ma et al. (2026) Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2026. dkv-cache: The cache for diffusion language models. Advances in Neural Information Processing Systems, 38:149009–149033. Meng et al. (2023) Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. 2023. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14297–14306. Nie et al. (2026) Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2026. Large language diffusion models. Advances in Neural Information Processing Systems, 38:50608–50646. Qian et al. (2026) Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, and Hao Zhang. 2026. d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation. arXiv preprint arXiv:2601.07568. Sahoo et al. (2024) Subham S Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. 2024. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184. Sahoo et al. (2025) Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. 2025. The diffusion duality. arXiv preprint arXiv:2506.10892. Salimans and Ho (2022) Tim Salimans and Jonathan Ho. 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512. Song et al. (2023) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models. Toshniwal et al. (2025) Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2025. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. In International Conference on Learning Representations, volume 2025, pages 19243–19275. Wang et al. (2025) Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. 2025. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192. Wu et al. (2025) Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618. Xu et al. (2025a) Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, and 1 others. 2025a. Lopa: Scaling dllm inference via lookahead parallel decoding. arXiv preprint arXiv:2512.16229. Xu et al. (2025b) Chenkai Xu, Xu Wang, Zhenyi Liao, Yishun Li, Tianqi Hou, and Zhijie Deng. 2025b. Unicms: A unified consistency model for efficient multimodal generation and understanding. arXiv preprint arXiv:2502.05415. Ye et al. (2025) Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. Zhang et al. (2026) Tunyu Zhang, Xinxi Zhang, Ligong Han, Haizhou Shi, Xiaoxiao He, Zhuowei Li, Hao Wang, Kai Xu, Akash Srivastava, Vladimir Pavlovic, and 1 others. 2026. T3d: Few-step diffusion language models via trajectory self-distillation with direct discriminative optimization. arXiv preprint arXiv:2602.12262. Zheng et al. (2024) Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, and 1 others. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583. Zheng et al. (2023) Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. 2023. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737. Appendix A Staged Trajectory Construction Algorithm Algorithm 1 gives the complete procedure for constructing staged self-rollout trajectories from native threshold-decoding traces. Algorithm 1 Staged Trajectory Construction Pretrained dLLM qθ0q_ _0, prompt set X, M2T threshold τ, optional edit threshold τedit _edit, stage size S Trajectory dataset D // Initialize dataset ←∅D← for each prompt x∈x do // Initialize native trajectory Set s0s_0 to the all-mask initial state and save s¯0←s0 s_0← s_0 Set stage counter n←0n← 0 and reveal counter R←0R← 0 for each native decoding step t do Compute predictions with qθ0(⋅∣x,st,j)q_ _0(· x,s_t,j) Select newly revealed positions ΔMt M_t by Eq. (2) For edit-capable decoding, record EtE_t by Eq. (5) Update the native state from sts_t to st+1s_t+1 Set R←R+|ΔMt|R← R+| M_t| if R reaches the stage size S then Save s¯n+1←st+1 s_n+1← s_t+1 Set n←n+1n← n+1 and R←0R← 0 end if end for // Finalize after full decoding if sTs_T is not already saved then Save s¯N←sT s_N← s_T end if Add the complete stage trajectory ¯=(s¯0,…,s¯N) T=( s_0,…, s_N) to D end for return D Appendix B Implementation Details All models use full-parameter fine-tuning with the AdamW optimizer, a learning rate of 1.0×10−51.0× 10^-5, and a cosine learning rate schedule. We set the stage size to 8, use a global batch size of 64, and train for 5 epochs. The CE anchor loss weight λCE _CE is set to 0.1, and the block size is 32. During KL distillation, we retain the top-20 tokens ranked by the target distribution ptarjp_tar^j and renormalize the retained probabilities before computing the KL terms. This top-k truncation both reduces the computation cost of the softmax and KL terms and focuses the student on the semantically meaningful region of the distribution. All training is conducted on 64 NVIDIA H800 GPUs. Appendix C Full Derivation of the Theoretical Analysis This appendix provides the full derivation of the bound used in Section 4.4. Using the TV distance defined in Section 4.4, consider a position j that is revealed at stage rjr_j, i.e., j∈M¯rj−1j∈ M_r_j-1 and j∉M¯rjj∉ M_r_j, with final token yj=s¯Njy^j= s_N^j. Let pij=qθ(⋅∣x,s¯i,j)p_i^j=q_θ(· x, s_i,j) denote the prediction at stage i. For any earlier masked stage i<rji<r_j, the triangle inequality gives DTV(pij,δyj)≤∑r=irj−2DTV(prj,pr+1j)+DTV(prj−1j,δyj),D_TV(p_i^j, _y^j)≤ _r=i^r_j-2D_TV(p_r^j,p_r+1^j)+D_TV(p_r_j-1^j, _y^j), (15) where δyj _y^j is the one-hot distribution on yjy^j. The first term accumulates distributional drift across adjacent stages, and the second term is the masked-state error immediately before token reveal. Pinsker’s inequality bounds each adjacent-stage TV term by forward KL: DTV(prj,pr+1j)≤12DKL(pr+1j∥prj).D_TV(p_r^j,p_r+1^j)≤ 12D_KL(p_r+1^j\|p_r^j). (16) For the reveal-boundary term, we have DTV(prj−1j,δyj)=1−prj−1j(yj)≤−logprj−1j(yj), splitD_TV(p_r_j-1^j, _y^j)&=1-p_r_j-1^j(y^j)\\ &≤- p_r_j-1^j(y^j), split (17) using 1−x≤−logx1-x≤- x for x∈(0,1]x∈(0,1]. Substituting Eqs. (16) and (17) into Eq. (15) yields DTV(pij,δyj)≤∑r=irj−212DKL(pr+1j∥prj)−logprj−1j(yj). splitD_TV(p_i^j, _y^j)&≤ _r=i^r_j-2 12D_KL(p_r+1^j\|p_r^j)\\ & - p_r_j-1^j(y^j). split (18) This bound matches the structure of Consistency Forcing. The forward KL component in Eq. (8) directly targets the adjacent-stage drift terms in Eq. (18). The CE anchor in Eq. (9) includes the reveal-boundary term in Eq. (17) and supplies token-level supervision at commitment boundaries. The confidence-weighted reverse KL further encourages mode-seeking predictions when the later-stage target is reliable. Since cj≥0c_j≥ 0, ℒCADj=DKL(ptarj∥pstuj)+cjDKL(pstuj∥ptarj)≥DKL(ptarj∥pstuj), splitL_CAD^j&=D_KL(p_tar^j\|p_stu^j)+c_jD_KL(p_stu^j\|p_tar^j)\\ &≥ D_KL(p_tar^j\|p_stu^j), split (19) so optimizing ℒCADL_CAD preserves the forward KL alignment term while sharpening the student distribution toward the cleaner state’s dominant mode. Therefore, Consistency Forcing reduces a tractable surrogate of the early-stage prediction error bound by aligning adjacent-stage distributions and anchoring masked-token predictions along the trajectory. Appendix D AUP Evaluation Details Model Decode config AVG GSM8K MATH500 MBPP HumanEval Score TPF Score TPF Score TPF Score TPF Score TPF LLaDA2.0-mini τ=0.95τ=0.95 86.52 3.09 93.78 2.42 84.20 2.99 81.50 3.21 86.59 3.74 τ=0.9τ=0.9 84.32 3.60 93.25 2.85 81.80 3.46 78.69 3.80 83.54 4.28 LLaDA2.0-mini-CAP τ=0.95τ=0.95 82.70 4.26 92.34 3.40 83.00 4.15 76.81 4.57 78.66 4.93 τ=0.9τ=0.9 79.51 5.07 91.74 4.09 81.00 4.85 72.13 5.43 73.18 5.89 LLaDA2.0-mini-CForce (Ours) τ=0.95τ=0.95 83.90 5.32 92.27 4.33 82.60 5.34 79.63 4.87 81.10 6.73 τ=0.9τ=0.9 80.88 6.42 91.74 5.36 79.20 6.35 73.30 6.00 79.27 7.97 LLaDA2.1-mini τ=0.7τ=0.7, τedit=0.5 _edit=0.5 85.57 6.94 93.56 5.94 85.00 7.44 77.75 7.25 85.98 7.11 τ=0.5τ=0.5, τedit=0.0 _edit=0.0 85.77 8.11 92.95 7.15 86.20 8.74 79.16 8.30 84.76 8.26 LLaDA2.1-mini-CForce (Ours) τ=0.7τ=0.7, τedit=0.5 _edit=0.5 86.41 9.08 92.27 7.63 84.80 10.15 81.97 10.07 86.59 8.48 τ=0.5τ=0.5, τedit=0.0 _edit=0.0 85.03 10.86 91.51 9.26 84.40 11.47 78.22 12.61 85.98 10.08 Table 5: Benchmark-level Operating Points Used to Compute AUP for the LLaDA-family Rows in Table 1. Each benchmark group reports the Score/TPF pair used by the two-point AUP frontier. Bold and underline mark the best and second-best metrics within each LLaDA-series block. All evaluations use maximum generation length 4096; LLaDA2.0 models are M2T-only and therefore do not use an edit threshold. AUP is computed from a small score–parallelism frontier rather than from only the single Score/TPF pair shown in Table 1. For each benchmark, an operating point is denoted by (ai,yi)(a_i,y_i), where aia_i is TPF and yiy_i is the task score in percentage points. Following the public AUP implementation, we sort operating points by TPF and compute AUP=a1y1+∑i=2mai−ai−12(yiW(yi)+yi−1W(yi−1)),AUP=a_1y_1+ _i=2^m a_i-a_i-12 (y_iW(y_i)+y_i-1W(y_i-1) ), (20) where W(y)=min(exp(−3(1−y/ymax)),1)W(y)= ( (-3(1-y/y_ )),1) and ymaxy_ is the maximum score for that benchmark among the compared methods. We keep operating points within five score points of the first, lowest-TPF point, matching the default AUP filtering threshold. This weighting penalizes operating points that gain parallelism by substantially degrading accuracy. Table 5 lists the exact decoding configurations and benchmark-level Score/TPF pairs used to construct the LLaDA-family AUP frontiers in Table 1. For the LLaDA2.0 series, including LLaDA2.0-mini, LLaDA2.0-mini-CAP, and LLaDA2.0-mini-CForce, we evaluate two threshold-decoding points, τ=0.95τ=0.95 and τ=0.9τ=0.9. For the edit-capable LLaDA2.1 series, including LLaDA2.1-mini and LLaDA2.1-mini-CForce, we use the official quality and speed modes: quality mode sets τ=0.7τ=0.7 and τedit=0.5 _edit=0.5, while speed mode sets τ=0.5τ=0.5 and τedit=0.0 _edit=0.0. The Score and TPF columns in Table 1 report the main comparison operating point, while the AUP column is computed from the corresponding two-point frontier. Appendix E Wall-clock Throughput We report wall-clock throughput on two benchmarks, GSM8K and HumanEval, in Table 6. All measurements use two NVIDIA H20 GPUs with tensor parallelism TP=2TP=2 and batch size 1. TPS denotes generated tokens per second. Model GSM8K TPS HumanEval TPS AVG TPS Non-edit dLLMs LLaDA2.0-mini 489.87 711.59 600.73 LLaDA2.0-mini-CAP 710.88 891.94 801.41 LLaDA2.0-mini-CForce (Ours) 783.09 932.82 857.96 Edit-capable dLLMs LLaDA2.1-mini 987.78 1116.10 1051.94 LLaDA2.1-mini-CForce (Ours) 1231.52 1352.86 1292.19 Table 6: Wall-clock Throughput on GSM8K and HumanEval. Bold marks the best throughput within each edit-capability group. In the non-edit LLaDA2.0 setting, CForce reaches 857.96 average TPS, improving over the base model by 42.82% and over CAP by 7.06%. In the edit-capable LLaDA2.1 setting, CForce reaches 1292.19 average TPS, improving over the base model by 22.84%. Appendix F Full Ablation Results Tables 7, 8, and 9 provide the benchmark-level results corresponding to the compact ablation tables in Section 5.3. Each benchmark group reports Score and TPF under threshold decoding at τ=0.9τ=0.9 and maximum generation length 4096. Stage size S AVG GSM8K MATH500 MBPP HumanEval Score TPF Score TPF Score TPF Score TPF Score TPF 4 80.62 5.64 91.51 4.56 79.80 5.64 74.94 5.22 76.22 7.14 8 80.88 6.42 91.74 5.36 79.20 6.35 73.30 6.00 79.27 7.97 16 75.05 7.97 90.30 7.61 75.20 9.86 62.76 6.71 71.95 7.70 Table 7: Full benchmark-level ablation on stage size for LLaDA2.0-mini-CForce. Each benchmark group reports Score and TPF. Curriculum Target type AVG GSM8K MATH500 MBPP HumanEval Score TPF Score TPF Score TPF Score TPF Score TPF ✓ Same-model target 80.88 6.42 91.74 5.36 79.20 6.35 73.30 6.00 79.27 7.97 ✓ Frozen teacher 79.49 5.91 91.21 4.67 79.20 5.57 70.73 5.63 76.83 7.75 × Same-model target 79.87 5.97 91.66 4.89 77.40 5.67 72.37 5.30 78.05 8.00 Table 8: Full benchmark-level ablation on curriculum learning and target type for LLaDA2.0-mini-CForce. The frozen-teacher variant changes only the later-state target predictor, while the no-curriculum row keeps the same-model stop-gradient target and removes the gradual exposure schedule. Each benchmark group reports Score and TPF. Component Variant AVG GSM8K MATH500 MBPP HumanEval Score TPF Score TPF Score TPF Score TPF Score TPF KL divergence Forward KL 85.75 5.07 92.95 4.00 82.80 4.72 81.26 4.74 85.98 6.81 Reverse KL 65.57 8.72 87.64 7.02 70.20 8.15 51.99 8.78 52.44 10.92 CAD 80.88 6.42 91.74 5.36 79.20 6.35 73.30 6.00 79.27 7.97 CE anchor λCE=0 _CE=0 79.02 6.52 91.58 5.34 78.80 6.36 68.85 6.06 76.83 8.32 λCE=0.1 _CE=0.1 80.88 6.42 91.74 5.36 79.20 6.35 73.30 6.00 79.27 7.97 λCE=2.0 _CE=2.0 80.22 6.52 92.49 5.34 78.60 6.06 75.41 6.34 74.39 8.35 Table 9: Full benchmark-level ablations of the KL Divergence Form and CE Anchor Weight on LLaDA2.0-mini-CForce. Each benchmark group reports Score and TPF. Appendix G Case Study We present qualitative examples comparing the generation quality of LLaDA2.0-mini-CForce (Ours), LLaDA2.0-mini-CAP, and the LLaDA2.0-mini baseline. All outputs are generated with a fixed TPF of 8, i.e., each forward pass produces exactly 8 tokens. These examples illustrate that CForce produces more coherent and accurate outputs under few-step generation compared to CAP and the baseline. G.1 Example 1 (Reference: 140) Prompt: ⬇ Question: Andy plants 90 geraniums and 40 fewer petunias that geraniums. How many flowers does he plant total? Please reason step by step, and put your final answer within . LLaDA2.0-mini-CForce (Ours): ⬇ Let’s solve the problem step by step: 1. **Number of geraniums**: Andy plants 90 geraniums. 2. **Number of petunias**: He plants 40 fewer petunias than geraniums. So, number of petunias = 90 - 40 = 50. 3. **Total number of flowers**: Add ger geranium and ger petunias: Total = geraniums + petunias = 90 + 50 = 140. Final answer: 140 LLaDA2.0-mini-CAP: ⬇ Let’s break this down step by step: 1. **Identify the number of geraniums planted:** Andy plants **90 geraniums**. 2. **Determine the number of petunias planted:** He plants **40 fewer petunias than geraniums**. So, number of petunias = 90 geraniumaniums)0 = **50**. 3. **Calculate the total number of flowers:** Total flowers = geraniums + petunias = 900 + 50 = **120 Rightarrow10 LLaDA2.0-mini (Baseline): ⬇ Let’s solve this problem step by step. 1. **Identify the number of geraniums plants**: Andy plants **90 geraniums**. 2. **Determine the number of petunias he plants**: He plants **40 fewer petununias than geraniums**. So, number of petunias = $$ 90 - 040 = 50 $$ 3. **Calculate the total number of flowers**: Add flowers number of geraniums and petunias $$ 90 + 50 = 140 $$ Wait -- the total written.: \ $$ - 50 = 130 \\ $$ No -- let’s re-check. **90 - 40 = 50**, correct.5 ias total. $$ \ 90 = \1100 $$ No -- restart. Correct: $$ 90 + 50 = 140 $$ Correcting 140 After Final step: $$ 140 $$ G.2 Example 2 (Reference: 160) Prompt: ⬇ Question: Tara bought 8 packs of 5 canvas bags for $4 each. She painted them and sold them at a craft fair for $8 each. How much profit did she earn on her bags? Please reason step by step, and put your final answer within . LLaDA2.0-mini-CForce (Ours): ⬇ Let’s solve this step by step. **Step 1: Determine the total number of canvas bags Tara bought.** Tara bought 8 packs, and each pack contains 5 canvas bags: $$ 8 packs × 5 bags per pack = 40 canvas bags $$ **Step 2: Calculate the total cost of the canvas bags.** Each canvas bag costs $4: $$ 40 bags × \$4 = \$160 $$ **Step 3: Calculate the total revenue from selling the bags.** She sold each bag for $8: $$ 40 bags × \$8 = \$320 $$ **Step 4: Calculate the profit.** Profit = Revenue - Total Cost $$ \$320 - \$160 = \$160 $$ **Final Answer:** $$ 160 $$ Tara earned a profit of **$160**. LLaDA2.0-mini-CAP: ⬇ Let’s solve this step step by step: **Step 1: Calculate the total cost Tara the bags.** bought bought 8 packs of 5 canvas bags, pack. Each pack pack costs $4. So, total number of bags bags = 8 packs x5 bags per pack = 4text bags $$ **Total cost** = 8 x 44 per pack = \32 $$ --- **Step 2: Calculate the much revenue selling the the bags bags $$ She sold each 40 bags for $8 each. Total, from all bags = 40 bags x $8 per bag \ \320 $$ --- **Step 3: Calculate the profit $$ProfitProfit = Total Revenue - Total cost \320 - \$32 = \288 --- Wait -- correction: $$ $$ boxedboxed2 LLaDA2.0-mini (Baseline): ⬇ Let’s break this down step by step: **Step 1: Calculate how total canvas bags Tara bought.** T bought bought 8 packs, each of 5 canvas bags. So, Total 5-canvas bags = 8 x packs 5 = =text40 bags. **Step 2: Calculate how Tara cost for bought on canvas bags.** Each pack $44, and she bought 8.. So, Cost = 8 x $4 = 3$2. **Step 3 Calculate how much Tara earned from selling the bags.** She sold each bag bag for for for88, and had 40 bags. . So, totalRevenue = 40 x $8 = 320. **Step 4: profit earned vs.** Profit = Total revenue - Total Cost. Profit = $320 - 32 = 288.. Wait -- let’s correct and recal values accurately. G.3 Example 3 Prompt: ⬇ You are an expert Python programmer, and here is your task: Write a function to find the median of three numbers. Your code should pass these tests: assert median_numbers(25,55,65)==55.0 assert median_numbers(20,10,30)==20.0 assert median_numbers(15,45,75)==45.0 Please enclose your code within delimiters as follows: ‘python # YOUR CODE HERE ‘ LLaDA2.0-mini-CForce (Ours): ⬇ ‘python def median_numbers(a, b, c): """ Find the median of three numbers. Args: a, b, c: three numbers Returns: The: of a float """ # Sort a three numbers list and return the median sorted_numbers = sorted([a, b, c]) return float(sorted_numbers[1]) ‘ LLaDA2.0-mini-CAP: ⬇ ‘python def median_numbers(a,,, c): """ Find the median of three numbers. Args: a, b, c: Three numbers Returns: The median value as a numbers or a float place numbers = [a, b, c] numbers.sort() return float(numbers[1]) ‘ LLaDA2.0-mini (Baseline): ⬇ ‘python # median_numbers(a, b, c): """ Find the median of three numbers. Args: a, b, c: Three numbers Returns: The median as a three-digit float float # Create a three numbers numbers and sort numbers = [] + [b] +c numbers.sort() # Return the median (middle value) return float(numbers[1]) ‘