Paper deep dive
Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
Pengxiang Li, Dilxat Muhtar, Tianlong Chen, Lu Yin, Shiwei Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/20/2026, 9:06:38 AM
Summary
This paper investigates why Diffusion Language Models (DLMs) often exhibit autoregressive (AR)-like decoding dynamics despite their potential for parallel token generation. The authors identify a mismatch between DLM objectives and the highly sequential structure of standard training data (e.g., pre-training corpora and long Chain-of-Thought data) as the primary cause. They propose NAP (Non-Autoregressive Parallel DLMs), a data-centric approach that curates examples with multiple independent reasoning trajectories and uses a parallel-forced decoding strategy. Experiments on math reasoning benchmarks show that NAP improves performance under parallel decoding compared to baselines trained on standard data, suggesting that aligning supervision with non-AR dynamics is key to unlocking true parallel generation.
Entities (11)
Relation Signals (7)
Dream-7B → isinstanceof → Diffusion Language Models
confidence 98% · widely used DLM families such as ... Dream
LLaDA-8B → isinstanceof → Diffusion Language Models
confidence 98% · widely used DLM families such as LLaDA
Diffusion Language Models → exhibitsbehavior → Autoregressive Decoding
confidence 95% · practical fast DLMs frequently converge to left-to-right, autoregressive (AR)-like decoding dynamics
NAP → uses → parallel decoding
confidence 95% · NAP curates examples as multiple independent reasoning trajectories and couples them with a parallel-forced decoding strategy
NAP → mitigates → Autoregressive Decoding
confidence 94% · NAP yields stronger performance under parallel decoding than DLMs trained on standard long CoT data... moving toward genuinely non-autoregressive parallel generation
training data → causes → Autoregressive Decoding
confidence 92% · a primary driver of AR-like decoding is a mismatch between DLM objectives and the highly sequential structure of widely used training data
OpenR1-Math → isinstanceof → training data
confidence 90% · long CoT reasoning datasets (i.e., OpenR1-Math)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion Language Models (DLMs) are often advertised as enabling parallel token generation, yet practical fast DLMs frequently converge to left-to-right, autoregressive (AR)-like decoding dynamics. In contrast, genuinely non-AR generation is promising because it removes AR's sequential bottleneck, better exploiting parallel hardware to reduce synchronization/communication overhead and improve latency scaling with output length. We argue that a primary driver of AR-like decoding is a mismatch between DLM objectives and the highly sequential structure of widely used training data, including standard pretraining corpora and long chain-of-thought (CoT) supervision. Motivated by this diagnosis, we propose NAP (Non-Autoregressive Parallel DLMs), a proof-of-concept, data-centric approach that better aligns supervision with non-AR parallel decoding. NAP curates examples as multiple independent reasoning trajectories and couples them with a parallel-forced decoding strategy that encourages multi-token parallel updates. Across math reasoning benchmarks, NAP yields stronger performance under parallel decoding than DLMs trained on standard long CoT data, with gains growing as parallelism increases. Our results suggest that revisiting data and supervision is a principled direction for mitigating AR-like behavior and moving toward genuinely non-autoregressive parallel generation in DLMs. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.23225v2
- Canonical: https://arxiv.org/abs/2602.23225v2
Trouble viewing inline? Open PDF directly →
Full Text
56,799 characters extracted from source content.
Expand or collapse full text
Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? Pengxiang Li 1 * Dilxat Muhtar 2 3 4 * Tianlong Chen 5 Lu Yin † 6 Shiwei Liu † 2 3 4 Abstract Diffusion Language Models (DLMs) are often ad- vertised as enabling parallel token generation, yet practical “fast” DLMs frequently converge to left- to-right, autoregressive (AR)-like decoding dy- namics. In contrast, genuinely non-AR generation is promising because it removes AR’s sequential bottleneck, better exploiting parallel hardware to reduce synchronization/communication overhead and improve latency scaling with output length. We argue that a primary driver of AR-like de- coding is a mismatch between DLM objectives and the highly sequential structure of widely used training data, including standard pre-training cor- pora and long chain-of-thought (CoT) supervision. Motivated by this diagnosis, we propose NAP (Non-Autoregressive Parallel DLMs), a proof-of- concept, data-centric approach that better aligns supervision with non-AR parallel decoding. NAP curates examples as multiple independent reason- ing trajectories and couples them with a parallel- forced decoding strategy that encourages multi- token parallel updates. Across math reasoning benchmarks, NAP yields stronger performance under parallel decoding than DLMs trained on standard long CoT data, with gains growing as parallelism increases. Our results suggest that revisiting data and supervision is a principled di- rection for mitigating AR-like behavior and mov- ing toward genuinely non-autoregressive parallel generation in DLMs. Our code is available at https://github.com/pixeli99/NAP. 1 The Hong Kong Polytechnic University, Hong Kong, China 2 ELLIS Institute T ̈ ubingen, T ̈ ubingen, Germany 3 Max Planck Institute for Intelligent Systems, T ̈ ubingen, Germany 4 T ̈ ubingen AI Center, T ̈ ubingen, Germany 5 The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA 6 University of Surrey, Guildford, United Kingdom. † Corresponding authors: Lu Yin <l.yin@surrey.ac.uk>, Shiwei Liu <sliu@tue.ellis.eu>. Preprint. March 2, 2026. 1. Introduction Large language models (LLMs) have become a cornerstone of modern AI, yet their rapidly growing computational and environmental footprints raise pressing sustainability con- cerns (Patterson et al., 2021; Luccioni et al., 2023). This mo- tivates renewed interest in alternative generation paradigms that can reduce inference latency and cost without sacrific- ing capability. Diffusion Language Models (DLMs) have recently emerged as a compelling candidate: by iteratively denoising a sequence, DLMs can in principle enable par- allel token generation, offering a path toward faster, more efficient generation (Austin et al., 2021b; Lou et al., 2023; Shi et al., 2024b; Sahoo et al., 2024a; Nie et al., 2025b; Gong et al., 2024; Ye et al., 2025). When paired with es- tablished inference accelerators, such as KV caching (Ma et al., 2025; Wu et al., 2025; Liu et al., 2025) and specu- lative decoding (Christopher et al., 2025; Gao et al., 2025; Chen et al., 2026), DLM-based systems are often claimed as substantially faster alternatives to standard autoregressive (AR) decoding. Yet, despite their promise, practical “fast” DLMs exhibit a striking and under-discussed behavior: many methods that aim for highly parallel decoding converge toward AR-like generation, where the effective reasoning trajectory pro- ceeds largely from left to right (Nie et al., 2025b; Israel et al., 2025; Wu et al., 2025; Gong et al., 2025). In other words, even when the model architecture permits bidirec- tional context and parallel refinement, the realized decoding dynamics can resemble a sequential construction of the out- put. This phenomenon makes real-world DLM usage more nuanced than the headline promise of “truly parallel decod- ing”: speedups are often coupled to subtle quality trade-offs, and the conditions under which DLMs depart meaningfully from AR behavior remain unclear (Kang et al., 2025). The payoff for achieving genuinely (non-AR) parallel de- coding is substantial: AR-style decoding is fundamentally sequential, every token depends on the previous one, so generation latency scales roughly with output length. Al- though we can switch to fast parallel decoding in subsequent blocks after earlier blocks have largely converged, the need to wait for upstream stabilization introduces a sequential critical path, leading to extra latency and communication 1 arXiv:2602.23225v2 [cs.CL] 27 Feb 2026 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? 050100150200250300350 Decoding Step 0 200 400 600 800 1000 Token Position Positions Unmasked per Step AR reference 050100150200250300350 Decoding Step 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Tokens Unmasked Tokens Unmasked per Step (Total: 1024 tokens in 340 steps) Mean: 3.01 tokens/step 050100150200250300350 Decoding Step 0 200 400 600 800 1000 Token Position Position Distribution per Step (black line = mean) AR reference 02004006008001000 Token Index (flattened order) 0 200 400 600 800 1000 Token Position Position vs Token Index (color = step index, red = fixed positions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Local AR-ness@k @1: 0.713 Local AR-ness vs k (consecutive next-token predictions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Global AR-ness@k @1: 0.735 Global AR-ness vs k (prioritize leftmost masks) Perfect AR 0 50 100 150 200 250 300 Step Index (a) LLaDA-8B (AO) 050100150200250300350 Decoding Step 0 200 400 600 800 1000 Token Position Positions Unmasked per Step AR reference 050100150200250300350 Decoding Step 0 1 2 3 4 5 6 Tokens Unmasked Tokens Unmasked per Step (Total: 1024 tokens in 340 steps) Mean: 3.01 tokens/step 050100150200250300350 Decoding Step 0 200 400 600 800 1000 Token Position Position Distribution per Step (black line = mean) AR reference 02004006008001000 Token Index (flattened order) 0 200 400 600 800 1000 Token Position Position vs Token Index (color = step index, red = fixed positions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Local AR-ness@k @1: 0.921 Local AR-ness vs k (consecutive next-token predictions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Global AR-ness@k @1: 0.006 Global AR-ness vs k (prioritize leftmost masks) Perfect AR 0 50 100 150 200 250 300 Step Index (b) Dream-7B (AO) 050100150200250300 Decoding Step 0 200 400 600 800 1000 Token Position Positions Unmasked per Step AR reference 050100150200250300 Decoding Step 0 2 4 6 8 Tokens Unmasked Tokens Unmasked per Step (Total: 1024 tokens in 326 steps) Mean: 3.14 tokens/step 050100150200250300 Decoding Step 0 200 400 600 800 1000 Token Position Position Distribution per Step (black line = mean) AR reference 02004006008001000 Token Index (flattened order) 0 200 400 600 800 1000 Token Position Position vs Token Index (color = step index, red = fixed positions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Local AR-ness@k @1: 0.003 Local AR-ness vs k (consecutive next-token predictions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Global AR-ness@k @1: 0.014 Global AR-ness vs k (prioritize leftmost masks) Perfect AR 0 50 100 150 200 250 300 Step Index (c) Random 050100150200250300350 Decoding Step 0 200 400 600 800 1000 Token Position Positions Unmasked per Step Fixed positions (8) AR reference 050100150200250300350 Decoding Step 0 1 2 3 4 5 6 Tokens Unmasked Tokens Unmasked per Step (Total: 1016 tokens in 340 steps) Mean: 2.99 tokens/step 050100150200250300350 Decoding Step 0 200 400 600 800 1000 Token Position Position Distribution per Step (black line = mean) AR reference 02004006008001000 Token Index (flattened order) 0 200 400 600 800 1000 Token Position Position vs Token Index (color = step index, red = fixed positions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Local AR-ness@k @1: 0.020 Local AR-ness vs k (consecutive next-token predictions) Perfect AR 2.55.07.510.012.515.017.520.0 k 0.0 0.2 0.4 0.6 0.8 1.0 Global AR-ness@k @1: 0.298 Global AR-ness vs k (prioritize leftmost masks) Perfect AR 0 50 100 150 200 250 300 Step Index (d) Ours Figure 1. Visualization of decoding dynamics. We plot the token position being unmasked (y-axis) against the decoding step (x-axis). (a, b) Despite using confidence-based Arbitrary Order (AO) decoding, standard DLMs (LLaDA and Dream) exhibit a strict linear diagonal pattern, revealing that their behavior collapses into autoregressive (left-to-right) generation. (c) Random decoding eliminates AR bias but lacks structure. (d) Our method (NAP) breaks the single-stream bottleneck, generating multiple reasoning trajectories simultaneously. cost (Wang et al., 2025; Fu et al., 2025). In contrast, truly non-AR parallel decoding is naturally compatible with the distributed hardware, i.e., when dependencies across spans are weak, decoding is naturally compatible with distributed hardware and can be distributed across devices, with only occasional synchronization to maintain global consistency. In this work, we argue that one primary caveat of this AR- bias is a mismatch between the learning objective and the training data. Existing DLM pipelines blindly reuse training data originally designed for AR models, where rea- soning trajectories are implicitly encoded as left-to-right progressions, e.g., next-token prediction–style ordering (Ye et al., 2025; Allal et al., 2025; Li et al., 2024), or sequen- tial Chain-of-Thought (CoT) rationales (Zhao et al., 2025; Lambert et al., 2024). As a result, even if the diffusion process is nominally position-agnostic, the model can learn denoising strategies that preferentially reconstruct outputs in an AR-shaped manner. This “AR-shaped data” effect not only limits the extent to which DLMs can exploit gen- uine parallelism, but also complicates evaluation: a method may appear effective while largely reproducing AR model’s dynamics under a different wrapper. To test this conjecture, we conduct a systematic analysis of the decoding behavior of commonly used DLMs. The main findings are summarized below. I. Widely used training corpora are strongly sequen- tial. We quantify the sequential dependency of datasets by measuring how strongly the token at one position is de- termined by its preceding context. We show a consistent trend: commonly used pre-training corpora (i.e., FineWeb (Penedo et al., 2024)) and long CoT reasoning datasets (i.e., Open-R1-Math (Team, 2025)) display strong sequence de- pendence. I. DLM decoding remains largely autoregressive. Across widely used DLM families such as LLaDA (Nie et al., 2025c) and Dream (Ye et al., 2025), ARness remains high: ARAORand Decoding Order 0 20 40 60 80 100 Pass@1 Accuracy (%) 78.278.2 33.9 71.9 51.9 51.6 GSM8K Dream-7B LLaDA-8B ARAORand Decoding Order 0 20 40 60 80 100 Pass@1 Accuracy (%) 38.2 39.0 17.2 32.4 21.8 17.6 MATH-500 Dream-7B LLaDA-8B Performance on GSM8K and MATH-500 by Decoding Order Figure 2. Performance on GSM8K (left) and MATH-500 (right). Forcing low-ARness behavior (Random decoding) gen- erally causes reasoning performance to collapse. Notably, for LLaDA, we employ a constrained block-wise decoding strategy to ensure generation validity. This preserves local structural in- tegrity, resulting in the Arbitrary Order (AO) decoding maintaining comparable performance, unlike the sharp drop observed in fully unstructured random decoding. the model still tends to “lock in” decisions in a quasi-left-to- right pattern, despite the nominally arbitrary decoding rules. Conversely, forcing genuinely low ARness behavior, for instance, by randomizing the update order aggressively, can reduce ARness but typically causes reasoning performance to collapse. Taken together, these results indicate a non- trivial tradeoff: in standard setups, either ARness stays high to maintain capacity, or lowering ARness breaks reasoning. I. Training on long CoT data escalates ARness. Con- tinued post-training on standard long CoT datasets further increases ARness over time. While DLMs trained from scratch (e.g., LLaDA) tend to exhibit lower ARness than those adapted from pre-trained AR models (e.g., Dream), this gap gradually narrows with sustained CoT supervision. Intuitively, long CoT supervision provides an explicit step- by-step trajectory with a privileged ordering. Matching such training targets rewards the model for producing and stabi- lizing earlier tokens before later ones, thereby progressively shifting the learned decoding dynamics toward increasingly 2 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? 02004006008001000 Token length 10 0 10 20 30 40 50 60 70 SeqDep SeqDep vs. Token Length Binned mean ±1 std (a) OpenR1-Math SeqDep 2004006008001000 Token length 10 0 10 20 30 40 50 60 70 SeqDep SeqDep vs. Token Length Binned mean ±1 std (b) Fineweb SeqDep Figure 3. Sequential Dependence (SeqDep) Analysis on (a) OpenR1-Math and (b) FineWeb Datasets. The consistently high and rising SeqDep scores indicate that standard training corpora possess strong intrinsic sequentiality, driving models to internalize AR-like dependencies. autoregressive behavior. IV. Recent parallel fast-DLM methods gain speed by amplifying, not removing, AR-like generation. Despite being motivated by parallel decoding, many recent fast- DLM approaches achieve practical speedups by reinforcing an underlying autoregressive computation pattern. In partic- ular, they rely on increasingly confident early predictions or staged block-wise updates that stabilize prefixes before allowing limited parallelism downstream. As a result, paral- lelism is effectively gated by an AR-like convergence order, and the achieved acceleration stems from exaggerating this sequential structure rather than eliminating it. The above findings suggest that even though the DLMs per- mit arbitrary decoding strategy, as DLMs are trained on highly sequentially structured data, the model tends to inter- nalize an AR-like computational strategy. In other words, the training distribution teaches the model that reasoning is a chain with a privileged order, and changing the decoding procedure alone is often insufficient to undo this learned reliance. Addressing the issue therefore requires revisiting the data and supervision that shape the model’s generation strategy in the first place. To this end, we propose NAP (Non-Autoregressive Parallel DLMs), a proof-of-concept approach that tackles the prob- lem from a data and decoding codesign perspective. First, we curate supervision in which each example consists of multiple independent reasoning trajectories generated in parallel, this format deemphasizes any privileged token or- der and is naturally compatible with denoising-style learning in DLMs. Second, we introduce a parallel-forced decod- ing strategy that explicitly encourages multi-token parallel updates at different reasoning traces, further steering gener- ation away from AR-like critical paths. Together, these two components provide a simple and effective way to better align DLM behavior with truly parallel decoding. Across a range of math reasoning benchmarks, our results show that NAP, fine-tuned with 103K samples, consistently yields stronger performance under parallel decoding than the base- line trained on standard long CoT datasets. Moreover, the improvement becomes more pronounced as we increase the degree of parallelism, indicating that NAP is better aligned with non-AR decoding dynamics rather than relying on an implicit sequential critical path. Note that our goal is not to claim that NAP fully resolves the challenges of non-AR parallel decoding. Rather, we aim to use this small-scale post-training only result to show that revisiting data and supervision design is a promising direction for mitigating AR-like behavior in DLMs and mov- ing toward genuinely non-autoregressive parallel generation. We hope our results motivate further work on data-centric approaches to unlock the full efficiency potential of DLMs. 2. Related Work 2.1. Diffusion Language Models Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021), best known for their success in image generation (Rombach et al., 2022; Nichol et al., 2022; Saharia et al., 2022), are increasingly studied as a non- autoregressive alternative for text generation. Bringing diffu- sion from continuous variables to discrete tokens can be for- malized by treating the forward corruption as a Markov pro- cess over a finite vocabulary: D3PM (Austin et al., 2021a) instantiates this idea with discrete-time transition matrices, while subsequent work extends it to continuous time through CTMC formulations (Campbell et al., 2022). A particularly practical family is masked diffusion, which can be viewed as an absorbing-state construction in the D3PM lineage and 3 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? operates directly in token space via random masking (Shi et al., 2024a). This paradigm has produced strong results across scales, from smaller models such as MDLM (Sa- hoo et al., 2024b) and RADD (Ou et al., 2025) to large systems like LLaDA (Nie et al., 2025a) and Dream (Ye et al., 2025). Beyond text-only settings, MMaDA (Yang et al., 2025) further generalizes large diffusion models to multimodal generation with a shared probabilistic view and modality-agnostic architecture, while the broader literature highlights potential benefits such as parallelizable decoding and flexible (non left-to-right) generation orders that may be useful for complex reasoning. 2.2. Decoding Order and Sampling Schedules A key degree of freedom in masked diffusion language models is the sampling path—which positions are updated (or committed) at each refinement step and in what or- der. Rather than being a mere implementation detail, sev- eral works treat order as an explicit control knob for qual- ity/efficiency trade-offs. P2 (Peng et al., 2025) cast order selection as a planning problem, where a separate planner chooses which tokens to denoise at each step, decoupling where/when to update from how to update. Prophet (Li et al., 2025) further leverages model confidence to early-commit, switching from iterative refinement to one-shot completion when the top-2 gap indicates convergence. Order-awareness has also been pushed into training, e.g., by encouraging sim- pler and more coherent sampling paths (Zhu et al.). Mean- while, Ni et al. (2026) caution that arbitrary-order flexibility can be a double-edged sword: models may preferentially resolve low-uncertainty tokens and bypass high-uncertainty branching points, collapsing the effective reasoning space, suggesting that constraining or regularizing generation order can sometimes improve reasoning. 3. Preliminaries 3.1. Diffusion Language Models We consider diffusion language models (DLMs), and in particular masked diffusion models (MDMs), which gen- erate discrete token sequences by iteratively denoising a partially masked state. Letxdenote the input prompt and lety 0 = (y 1 0 ,...,y L 0 )∈V L denote a clean output sequence of lengthLover vocabularyV. MDMs define a forward masking process indexed by a continuous time variable t ∈ [0, 1], wheretrepresents the masking ratio. Giveny 0 , the forward process independently masks each token with probability t: q y i t | y i 0 = ( [MASK], with prob. t, y i 0 ,with prob. 1− t, (1) and factorizes across positions asq(y t | y 0 ) = Q L i=1 q(y i t | y i 0 ) . Att = 1, the sequence is fully masked; att = 0, it remains unchanged. 3.2. Measuring Autoregressive Bias To quantify how autoregressive-like a DLM decoding tra- jectory is, we adopt the ARness metrics proposed by Gong et al. (2025), which distinguish between global left-to-right bias and local sequential continuity. Let the decoding pro- cess be represented by a sequence of unmasked positions p = (p 1 ,p 2 ,...,p L ), wherep c ∈ 1,...,Ldenotes the position index of the token committed at decoding stepc. Let M c−1 be the set of masked positions just before step c. Global ARness. This metric measures the tendency to prioritize unmasking the leftmost remaining tokens, captur- ing a front-to-back filling strategy. For a tolerance window k ≥ 1, we define an indicatorI global (c,k)that is1if the cho- sen position p c is among the k earliest positions in M c−1 : I global (c,k) = ( 1, if p c ∈ smallest-k(M c−1 ), 0, otherwise. (2) The Global ARness score is the average over the sequence: Global-ARness@k = 1 L L X c=1 I global (c,k)∈ [0, 1].(3) A score of 1.0 (atk = 1) indicates a strict autoregressive (left-to-right) generation order. Unless otherwise stated, we use Global-ARness@1 as the primary measure of ARness in our analysis, as it directly quantifies the adherence to a causal generation order. 3.3. Measuring Sequential Dependence (SeqDep) To quantify the intrinsic sequentiality of a dataset, we mea- sure how much the prediction of a current text segment relies on its preceding generation history compared to re- lying solely on the initial prompt. Letxdenote the input prompt. Suppose the corresponding output sequence is di- vided intoNsegmentss = (s 1 ,...,s N ). Using an external autoregressive scorerp AR (e.g., a pretrained LLM), we de- fine the Sequential Dependence (SeqDep) as the average log-probability gain provided by the prefix context: SeqDep(x,s) = 1 N − 1 N X n=2 logp AR (s n | x,s <n ) − logp AR (s n | x) (4) Intuitively, this metric captures the conditional dependence between reasoning steps. ASeqDepscore near0indicates 4 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? that the segments n is conditionally independent of previ- ous segmentss <n given the promptx, implying that the sequence components could theoretically be generated in parallel. Conversely, a high positiveSeqDepscore indi- cates a strong chain-like structure—meaning later tokens are heavily dictated by the immediate preceding context, a hallmark of standard left-to-right autoregressive reasoning. 4. Decoding Behaviors of DLMs In this section, we conduct a systematic analysis of the decoding behavior of commonly used DLMs. We fix the pretrained masked diffusion model, the token budget, the number of refinement steps, and the mask-ratio schedule, and vary only the decoding rule. This isolates the effect of the induced generation order from all other factors. Our main findings are summarized below. 4.1.Strong Sequential Dependence in Training Corpora A primary driver of sequential behavior is the data itself. We hypothesize that if the training distribution is highly sequen- tial, the model learns an implicit left-to-right dependency that persists even under parallel decoding objectives. We quantify this using theSeqDepmetric (Sec. 3.3) on two representative datasets: FineWeb (pre-training corpora) and OpenR1-Math (long-CoT reasoning). As shown in Fig- ure 3b and 3a, both datasets display strong sequence depen- dence. Notably, reasoning steps in OpenR1-Math exhibit increasing dependence as the chain progresses (p AR predicts later steps with much higher confidence given the prefix). This suggests that standard training data teaches the model that reasoning is a fundamentally ordered chain, creating a mismatch with position-agnostic diffusion objectives. 4.2. DLMs’ Decoding Remains Largely Autoregressive Given sequential training data, we examine how DLMs behave during inference. We evaluate two popular models, LLaDA-8B (Nie et al., 2025c) and Dream-7B (Ye et al., 2025), under three distinct decoding strategies: (i) Autore- gressive (AR) Order: committing the leftmost unresolved tokens at each step, mimicking standard left-to-right generation; (i) Arbitrary Order (AO): a confidence-based strategy that commits the most certain tokens first regard- less of their positions; and (i) Random: committing a uniformly random subset of tokens at each step. Unlike Dream-7B, which handles fully unstructured par- allel updates relatively well, LLaDA-8B exhibits severe degradation on structured mathematical tasks if unmasked entirely at random. This is largely an artifact of its specific supervised fine-tuning (SFT) phase. To ensure valid and comparable generation quality for LLaDA, we apply a con- strained block-wise modification (Arriola et al., 2025) to the AO and Random strategies. Table 1. Quantifying Autoregressive Bias (ARness) and Ac- curacy. Comparison of sequential bias and performance across different decoding strategies. While AR Order implies strict se- quentiality (1.00), AO (Conf) maintains high ARness and competi- tive accuracy. Model AR OrderAO (Conf) ARnessAccARnessAcc LLaDA-8B (Nie et al., 2025c)1.0071.90.7351.9 + Fast-dLLM (Wu et al., 2025)1.0071.90.8751.6 Dream-7B (Ye et al., 2025)1.0078.20.9278.2 + Fast-dLLM (Wu et al., 2025)1.0078.30.9478.1 High ARness in DLM Decoding. Table 1 reports the AR- ness scores. While AR order is 1.0 by definition, AO de- coding converges to extremely high ARness (∼ 0.92for Dream), indicating that the model’s most “confident” tokens are almost always the next tokens in the sequence. As a result, DLMs exhibit behavior closely resembling autore- gressive generation. The Accuracy–ARness Tradeoff. Is it possible to force genuinely parallel behavior? We test this using a Random decoding strategy, which successfully yields near-zero ARness. However, as shown in Figure 2, this comes at a severe cost: reasoning accuracy on GSM8K (Cobbe et al., 2021) and MATH 500 (Lightman et al., 2023) collapses when the model is prevented from following a sequential path. These results suggest that strong reasoning performance is often obtained at the cost of genuine parallelism, as improved accuracy tends to coincide with higher AR-ness under standard setups. 4.3. Long-CoT Supervision Escalates AR-ness We further investigate how supervised fine-tuning (SFT) on long Chain-of-Thought (CoT) data affects decoding dy- namics. We compare the ARness of base models against checkpoints post-trained on standard CoT datasets (Open- R1 Math (Team, 2025)). As shown in Table 2 and Figure 4, results indicate a clear trend: post-training further increases ARness. For in- stance, LLaDA’s base ARness under AO decoding rises from 0.73 to 0.81 after CoT tuning. Intuitively, CoT su- pervision provides explicit step-by-step trajectories with a privileged order. Minimizing the loss on such data rewards the model for stabilizing earlier tokens before later ones, effectively ”baking in” the AR order and making it harder for the model to utilize genuine parallel decoding during inference. 5 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? Table 2. Long-CoT Supervision Increases ARness. Comparison of Global ARness@1 scores (using AO decoding) before and after fine-tuning. ModelBase (Pretrained)Long-CoT (SFT)Change LLaDA-8B0.730.81↑ 0.08 Dream-7B0.920.93↑ 0.01 02k4k6k8k10k Training Steps (Long-CoT SFT vs. NAP Data) 0.68 0.70 0.72 0.74 0.76 0.78 0.80 0.82 Global ARness@1 Score +0.08 0.04 02k4k6k8k10k Training Steps (Long-CoT SFT vs. NAP Data) 0.90 0.91 0.92 0.93 0.94 Global ARness@1 Score +0.01 0.01 Figure 4. Long-CoT Supervision Increases ARness. The positive deltas show models converging toward strict left-to-right genera- tion (1.0), confirming that current supervision methods actively discourage non-autoregressive parallel decoding. 4.4. Current Fast DLMs Reinforce Sequentiality Finally, we analyze whether specialized ”fast” decoding algorithms can unlock genuine parallelism. We evaluate Fast-dLLM (Wu et al., 2025), a state-of-the-art acceleration method that employs block-wise parallel decoding. As shown in Table 1, these methods do not reduce sequen- tial dependence; in fact, they exacerbate it. For instance, while standard AO decoding for LLaDA has an ARness of 0.73, applying Fast-dLLM pushes this score up to 0.87. Similarly, for Dream-7B, the ARness rises to 0.94, nearly indistinguishable from strict autoregressive decoding (1.00). This empirical evidence suggests that current ”fast” DLMs achieve speedups not by enabling non-sequential generation, but by effectively identifying and accelerating the underly- ing autoregressive critical path. The parallelism in these sys- tems is gated by the convergence of the prefix, meaning they optimize the execution of the sequential chain rather than eliminating the bottleneck. This diagnosis reinforces our [Input Query] A shirt costs $50 and is on sale for 20% off. What is the final price? [Model Output] ⟨think 1⟩Method: Calculate discount amount first. Discount =50×0.20 = 10. Final Price = 50− 10 = 40.⟨/think 1⟩ ⟨ think 2⟩Direct multiplier. Since it’s 20% off, we pay100%− 20% = 80%. Final Price = 50× 0.8 = 40.⟨/think 2⟩ ⟨think 3⟩ 20%of50is10. Final Price =50− 10 = 30. [Calculation Error]⟨/think 3⟩ ⟨summary⟩By analyzing multiple reasoning processes above, I concluded that: The final answer is 40.⟨/summary⟩ Figure 5. A compact training instance. The model generates parallel paths (including distinct methods and a noisy path) and aggregates them into a correct summary. core premise: achieving true non-autoregressive parallelism requires revisiting the supervision signal itself, rather than relying solely on inference-time algorithmic optimizations. 5. NAP: Non-Autoregressive Parallel DLMs 5.1. Overview To bridge the gap between DLM objectives and the se- quential nature of reasoning data, we propose NAP (Non- Autoregressive Parallel DLMs). NAP is a data-decoding co-design framework that breaks the implicit autoregres- sive lock-in by restructuring both the supervision signal and the inference process. The framework operates on two levels: first, it curates training examples as multiple indepen- dent reasoning trajectories rather than a single linear chain, thereby removing the notion of a privileged order; second, it employs a parallel-forced decoding strategy that explicitly enforces multi-stream updates during inference, preventing the model from collapsing into a sequential critical path. 5.2. Data Curation Standard chain-of-thought (CoT) data typically encodes a single canonical left-to-right reasoning order, creating a natural mismatch with the objective of parallel DLM de- coding. To address this, we curate a datasetD parallel whose supervision is inherently parallel. Generating Parallel Reasoning Traces.Similar to Para- Thinker (Wen et al., 2025), given a queryx, we prompt a strong teacher model to generatePindependent reason- ing tracesr (1) ,...,r (P) . We employ a high sampling temperature (τ = 1.0) to induce diverse problem-solving approaches or distinct logical orderings. Unlike standard augmentation which treats these as separate samples, NAP groups them into a single training instance. This ensures that the parallel paths represent truly independent explorations rather than redundant copies. 6 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? Question: Solve the equation x³ − 7x + 6 = 0 ? Step 0▢ Step 1 <think1> Try▢root▢ </think1><think2> ▢factors▢ </think2> Step 2 <think1> Tryx=1rootworks </think1><think2> Tryfactorsx=2 </think2> Step N <think1> Tryx=1rootworks </think1><think2> Tryfactorsx=2 </think2> ▢ Masked token <think1><summary> Summary block <summary> x▢2= Thinking block Parallel Thinking Summary Commit <summary> x=−312 ▢ ▢ ▢ ▢ </think3><think3> <think3></think3> <think3> </think3> findremaining findroot root findremainingroot x= x= -3 -3 ▢ <summary> <summary> Summary: x=3 or x=1 <think2><think3> Figure 6. Overview of the parallel-forced decoding framework. The model concurrently generates multiple independent reasoning paths within structured thinking blocks. These parallel trajectories are then synthesized into a result within a designated summary block. Summary and Aggregation. To teach the model to re- solve conflicts, we construct the summary blockSby con- ditioning the ground-truth answeraon the concatenation of these diverse (and potentially noisy) paths. The final training instance follows the format in Eq.(5), as illustrated in Figure 5. In this setup, the model observes multiple par- allel paths—some of which may contain errors (e.g., Path 3)—followed invariably by the correct result inS. This supervision forces the model to implicitly learn how to iden- tify valid reasoning streams and filter out noise to match the ground truth, treating the parallel paths as supporting evidence rather than a linear chain. We fine-tune the DLM on this structured data using the standard masked diffusion objective. 5.3. Parallel-Forced Decoding To enable the model to reason in parallel, we design a de- coding canvas that spatially separates reasoning streams and enforce a structure-aware update schedule. Decoding Canvas. We define a structured output format containingmindependent reasoning blocks and one sum- mary block: Y = B 1 ,R (1) , B 2 ,R (2) , ..., B m ,R (m) , B S ,S , (5) whereB j are fixed textual headers (e.g., “<think #j>”), R (j) are free-form reasoning contents for thej-th path, and Sis a final summary containing the answer. Given a prompt x, we initialize a canvas of lengthL = P (|B j | + L j ) + (|B S | + L S ), where fixed headers are clamped and reason- ing slots are initialized to[MASK]. This layout effectively enforces conditional independence betweenR (i) andR (j) given the prompt, as there is no causal masking order be- tween them in a bidirectional model. Macro-Parallel, Micro-Confidence Updates. Standard arbitrary-order decoding often degenerates into global sequential generation because the model preferentially re- solves the immediate next tokens. NAP-D prevents this via a hierarchical schedule. At the macro level, we enforce strict parallelism: the unmasking budget is distributed across all mreasoning blocksR (1) ,...,R (m) at every step. This constraint prevents the model from stabilizing upstream paths before initiating downstream ones. At the micro level, within each individual blockR (j) , we apply a confidence- based strategy (i.e., masking low-confidence tokens). We do not enforce a left-to-right order locally; instead, tokens are committed based on their confidence scores. This com- bination ensures that the global process is parallel (evolving multiple trajectories simultaneously) while local generation retains the flexibility of non-autoregressive refinement. 6. Experiments This section evaluates whether our decoding strategy can (i) improve reasoning performance over standard diffusion decoding rules, (i) reshape the induced generation order as measured by ARness (Section 3.2), and (i) mitigate order sensitivity in regimes where long-form rationales exhibit strong sequential dependence (Eq.(4)). Unless otherwise stated, all results use the same pretrained masked diffusion model and differ only in the decoding rule. Evaluation protocol. We evaluate on a suite of reason- ing benchmarks including GSM8K (Cobbe et al., 2021), MATH-500 (Lightman et al., 2023), and GPQA (Rein et al., 2024). Each example is prompted to produce a think- ing path and a final answer in a fixed format; we extract answers with a deterministic parser and report accuracy. Models and Training.We conduct experiments on two state-of-the-art diffusion language models: LLaDA-8B-Instruct(Nieetal.,2025c)and 7 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? Table 3. Benchmark results on LLaDA-8B-Instruct and Dream-7B-Instruct under different step budgets. Tok/Step denotes the number of tokens decoded per decoding step; larger Tok/Step corresponds to higher decoding parallelism. BenchmarkStepsTok/StepLLaDA 8B LLaDA 8B (Long-CoT)NAP-LLaDA 8BDream-7B Dream-7B (Long-CoT)NAP-Dream-7B Mathematics & Scientific GSM8K 256446.454.156.1 (+2.0)35.046.560.9 (+14.4) 336354.460.963.3 (+2.4)49.456.970.9 (+14.0) 512262.082.082.6 (+0.4)58.566.879.2 (+12.4) 1024166.583.584.1 (+0.6)68.978.083.6 (+5.6) MATH-500 256417.821.426.6 (+5.2)8.816.223.8 (+7.6) 3363 20.626.635.4 (+8.8)11.425.631.4 (+5.8) 512228.041.243.0 (+1.8)20.840.043.0 (+3.0) 1024130.445.047.0 (+2.0)35.047.449.6 (+2.2) GPQA 336312.515.419.0 (+3.6)5.87.310.5 (+3.2) 512218.821.225.9 (+4.7)14.719.422.5 (+3.1) 10241 20.823.028.6 (+5.6)26.128.629.5 (+0.9) Dream-7B-Instruct(Ye et al., 2025).To vali- date our proposed method, we fine-tune these base models on the parallel reasoning datasetD parallel curated via the pipeline described in Section 5.2. For a fair comparison, we also train a Long-CoT baseline on the same set of reasoning trajectories but serialized in the standard autoregressive format. Crucially, this baseline is evaluated using standard decoding—its optimal inference setting—rather than our parallel strategy, ensuring a strong and fair comparison. Both variants are trained using the standard masked diffusion objective for 3 epochs. We use the AdamW optimizer with a learning rate of 2e-6 and a global batch size of 256. All experiments are conducted on 8 NVIDIA A800 GPUs. Decoding baselines. We compare several widely used un- masking rules under the common mask-and-predict frame- work. AR order commits the leftmost unresolved tokens at each step (a diffusion realization of left-to-right decod- ing). Arbitrary order (AO) commits the most confident positions. Random order (Rand) commits a uniformly ran- dom subset at each step, serving as a low-ARness control. Our method generatesmmultiple independent reasoning paths and a final summary commit on a structured canvas. To ensure a fair budget, PaS-Dec uses the same total token capLby allocating per-path budgets330and a summary budget32such that the overall canvas length matches the baseline. The summary block is the only region used for answer extraction and scoring. 6.1. Main Results Table 3 summarizes the performance across three bench- marks. Across all benchmarks and step budgets, our method achieves higher accuracy than both the Base model and the Long-CoT baseline. For instance, on GSM8K with Dream-7B (1024 steps), NAP-Dream-7B reaches 83.6%, surpassing the Long-CoT model (78.0%) despite using the same amount of compute and training data. This suggests that organizing reasoning into parallel streams is a more effective supervision signal for DLMs than forcing a single long chain. The most significant advantage of NAP appears in the low- step regime, e.g., 256 steps (4x parallel), where the model must generate more than one token per forward pass. Stan- dardLong-CoTmodels degrade sharply as parallelism increases. On Dream-7B/GSM8K, accuracy drops from 78.0% (1024 steps) to 46.5% (256 steps). This confirms that standard supervision creates a dependency on sequential stability; when forced to hurry, the reasoning collapses. In the same setting, NAP-Dream-7B maintains strong accuracy at 60.9%, compared to 46.5% of the Long-CoT baseline, thereby retaining substantially more capability. Notably, the gap between NAP and Long-CoT widens as parallel decod- ing is made more aggressive, increasing from +5.6% at 1024 steps to +14.4% at 256 steps. This result validates our core hypothesis: by training on data that lacks a privileged order, the model learns to be less reliant on the immediate left-side context, enabling effective Non-AR parallel decoding. To further understand how NAP achieves these results, we analyze the relationship between performance and the se- quential nature of generation (ARness). As shown in Fig- ure 1, standard models (LLaDA/Dream) using Arbitrary Order (AO) decoding exhibit a strict diagonal pattern. Even though they can decode anywhere, they effectively collapse into a left-to-right process (High ARness). In contrast, NAP (Figure 1(d)) displays distinct parallel bands, confirming that multiple reasoning trajectories are being generated si- multaneously. 6.2. Ablation Studies We investigate the individual contributions of the supervi- sion data and the decoding strategy using Dream-7B on the 8 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? GSM8K benchmark. The Necessity of Data-Decoding Co-design. We first iso- late the impact of our proposed decoding method versus the parallel-aligned data. As shown in Table 4, applying our Parallel-Forced Decoding strategy to a standard base model that has not been trained with our data leads to a larger performance drop than standard Arbitrary Order (AO) decoding. This suggests that without training support, the original Dream-7B struggles to handle the fragmented con- text of simultaneous generation. In addition, the decoding strategy becomes critical when parallelism is high. Specifi- cally at the aggressive 256-step budget, our Parallel-Forced decoding outperforms AO (60.9% vs. 57.4%). This con- firms that while the data provides the foundational reason- ing capability, aligning the decoding strategy is essential to maintain robustness when forcing the model to generate multiple tokens in parallel. Table 4. GSM8K accuracy using Dream-7B. Simply applying parallel decoding to a base model hurts performance; gains require aligned supervision. Training DataDecoding2565121024 Base (Pretrained)AO35.058.568.9 Base (Pretrained)Parallel-Forced31.052.660.2 NAP (Ours)AO57.478.985.1 NAP (Ours)Parallel-Forced60.979.283.6 Impact of Parallel Width (m). We further analyze how the number of parallel reasoning paths affects performance while keeping the total token budget constant. As detailed in Table 5, increasing the number of reasoning paths from a single chain (m = 1) to three (m = 3) provides consistent accuracy gains across both model families. Specifically, NAP-Dream sees a substantial improvement from 75.4% to 83.6%, while NAP-LLaDA rises from 79.4% to 84.1%. This monotonic trend supports the view that NAP benefits from an “internal ensemble” effect, where the final summary block effectively aggregates insights from multiple diverse trajectories generated in parallel to derive a more robust answer. Table 5. Accuracy on GSM8K with varyingm. Total token budget is fixed. Method1 Path2 Paths3 Paths NAP-Dream75.478.983.6 NAP-LLaDA79.482.684.1 Intrinsic Parallelism of Curated Data. To verify that our data curation pipeline effectively reduces the autoregressive bottleneck, we analyze the Sequential Dependence (SeqDep) of our constructed datasetD parallel . As illustrated in Figure 7, the SeqDep score remains remarkably stable (mean≈ 12) even as the sequence length grows from 500 to over 1000 5006007008009001000 Token length 10 0 10 20 30 40 50 60 70 SeqDep SeqDep vs. Token Length Binned mean ±1 std Figure 7. SeqDep Analysis onD parallel . We visualize the Sequen- tial Dependence (SeqDep) of our curated parallel reasoning data against token length. The green curve (binned mean) shows that SeqDep remains stable and relatively low across varying lengths. tokens. Unlike standard long-chain reasoning (as shown in Section 4), where dependence often escalates with depth, our parallel-structured data maintains a consistent level of information density. This ”flat” dependency profile confirms that the reasoning trajectories within our data possess high conditional independence, providing the necessary learning signal for the model to perform effective parallel updates during inference. 7. Conclusion In this work, we argue that the struggle of Diffusion Lan- guage Models (DLMs) to achieve genuine parallel decoding stems largely from the implicit sequentiality of standard training data. Our proposed method, NAP, demonstrates that aligning supervision with parallel decoding dynam- ics effectively mitigates this autoregressive collapse. By training on parallel reasoning trajectories and enforcing multi-stream updates, NAP decouples reasoning capability from sequential order, achieving superior performance in high-parallelism regimes while significantly reducing global ARness. These results suggest that unlocking the full po- tential of non-autoregressive generation requires moving beyond decoding heuristics to fundamentally rethink how we structure supervision for parallel reasoning. Limitations. While NAP demonstrates the feasibility of aligning supervision with genuinely parallel decoding, our current implementation serves primarily as a proof-of- concept. The method is evaluated in a post-training setting on a relatively small scale (∼100K samples). As scaling laws dictate much of DLMs’ behavior, a broader pre-training phase with inherently non-sequential, parallel-structured data may be required to completely eliminate the autoregres- sive bottleneck. 9 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? References Allal, L. B., Lozhkov, A., Bakouch, E., Bl ́ azquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydl ́ ı ˇ cek, H., Lajar ́ ın, A. P., Srivastav, V., et al. Smollm2: When smol goes big–data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025. Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V. Block diffusion: Inter- polating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025. Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, p. 17981–17993, 2021a. Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021b. Campbell, A., Benton, J., Bortoli, V. D., Rainforth, T., Deli- giannidis, G., and Doucet, A. A continuous time frame- work for discrete denoising models. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. Chen, J., Liang, Y., and Liu, Z.Dflash: Block dif- fusion for flash speculative decoding. arXiv preprint arXiv:2602.06036, 2026. Christopher, J. K., Bartoldson, B. R., Ben-Nun, T., Cardei, M., Kailkhura, B., and Fioretto, F. Speculative diffusion decoding: Accelerating language generation through dif- fusion. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), p. 12042–12059, 2025. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Fu, H., Huang, B., Adams, V., Wang, C., Srinivasan, V., and Jiao, J. From bits to rounds: Parallel decoding with exploration for diffusion language models. arXiv preprint arXiv:2511.21103, 2025. Gao, Y., Ji, Z., Wang, Y., Qi, B., Xu, H., and Zhang, L. Self speculative decoding for diffusion large language models. arXiv preprint arXiv:2510.04147, 2025. Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., et al. Scaling diffu- sion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024. Gong, S., Zhang, R., Zheng, H., Gu, J., Jaitly, N., Kong, L., and Zhang, Y. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639, 2025. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. Israel, D., den Broeck, G. V., and Grover, A. Accelerating diffusion llms via adaptive parallel decoding. CoRR, abs/2506.00413, 2025. Kang, W., Galim, K., Oh, S., Lee, M., Zeng, Y., Zhang, S., Hooper, C., Hu, Y., Koo, H. I., Cho, N. I., et al. Parallel- bench: Understanding the trade-offs of parallel decoding in diffusion llms. arXiv preprint arXiv:2510.04767, 2025. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E., Keh, S. S., Arora, K., et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Informa- tion Processing Systems, 37:14200–14282, 2024. Li, P., Zhou, Y., Muhtar, D., Yin, L., Yan, S., Shen, L., Liang, Y., Vosoughi, S., and Liu, S. Diffusion language models know the answer before decoding. arXiv preprint arXiv:2508.19982, 2025. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. Liu, Z., Yang, Y., Zhang, Y., Chen, J., Zou, C., Wei, Q., Wang, S., and Zhang, L. dllm-cache: Accelerating diffu- sion large language models with adaptive caching. arXiv preprint arXiv:2506.06295, 2025. 10 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? Lou, A., Meng, C., and Ermon, S. Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution. arXiv preprint arXiv:2310.16834, 2023. Luccioni, A. S., Viguier, S., and Ligozat, A.-L. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of machine learning research, 24(253): 1–15, 2023. Ma, X., Yu, R., Fang, G., and Wang, X. dkv-cache: The cache for diffusion language models, 2025. URLhttps: //arxiv.org/abs/2505.15781. Ni, Z., Wang, S., Yue, Y., Yu, T., Zhao, W., Hua, Y., Chen, T., Song, J., Yu, C., Zheng, B., et al. The flexibility trap: Rethinking the value of arbitrary order in diffusion language models. 2026. Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. GLIDE: towards photorealistic image generation and edit- ing with text-guided diffusion models. In Chaudhuri, K., Jegelka, S., Song, L., Szepesv ́ ari, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learn- ing, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, p. 16784–16804. PMLR, 2022. Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J., and Li, C. Large language diffusion models. CoRR, abs/2502.09992, 2025a. Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025b. Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025c. doi: 10.48550/arXiv.2502.09992. URLhttps://arxiv. org/abs/2502.09992. Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thir- teenth International Conference on Learning Representa- tions, ICLR 2025, Singapore, April 24-28, 2025. Open- Review.net, 2025. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, J. Car- bon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. Penedo, G., Kydl ́ ı ˇ cek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URLhttps://arxiv.org/abs/ 2406.17557. Peng, F. Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A. J., Tong, A., and Chatterjee, P. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03540, 2025. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, p. 10674–10685. IEEE, 2022. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Den- ton, E. L., Ghasemipour, S. K. S., Lopes, R. G., Ayan, B. K., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136– 130184, 2024a. Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Sim- ple and effective masked diffusion language models. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Pa- quet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024b. Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024a. Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024b. 11 Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. CoRR, abs/1503.03585, 2015. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. Team, O. Openr1-math-220k: A large-scale math reasoning dataset.https://huggingface.co/datasets/ open-r1/OpenR1-Math-220k, 2025.Accessed 2025. Wang, X., Xu, C., Jin, Y., Jin, J., Zhang, H., and Deng, Z. Diffusion llms can do faster-than-ar inference via dis- crete diffusion forcing. arXiv preprint arXiv:2508.09192, 2025. Wen, H., Su, Y., Zhang, F., Liu, Y., Liu, Y., Zhang, Y.-Q., and Li, Y. Parathinker: Native parallel thinking as a new paradigm to scale llm test-time compute. arXiv preprint arXiv:2509.04475, 2025. Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., and Xie, E. Fast-dllm: Training-free acceler- ation of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025. Yang, L., Tian, Y., Li, B., Zhang, X., Shen, K., Tong, Y., and Wang, M. Mmada: Multimodal large diffusion language models. CoRR, abs/2505.15809, 2025. Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. Zhao, S., Gupta, D., Zheng, Q., and Grover, A. d1: Scaling reasoning in diffusion large language models via rein- forcement learning. CoRR, abs/2504.12216, 2025. Zhu, Y., Chen, W., Kwok, J., and Zhao, Z. Spmdm: Enhanc- ing masked diffusion models through simplifing sampling path. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 12