Paper deep dive
$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
Zhenbang Du, Kejing Xia, Xinrui Zhong, Yonggan Fu, Nicolai Oswald, Binfei Ji, Brucek Khailany, Pavlo Molchanov, Yingyan Lin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/26/2026, 10:37:34 PM
Summary
The paper introduces $R^2$-dLLM, a unified framework designed to accelerate Diffusion Large Language Models (dLLMs) by reducing spatio-temporal redundancy. It identifies two main inefficiencies: spatial redundancy (confidence clusters and positional ambiguity) and temporal redundancy (remasking stable tokens). The framework proposes two components: a training-free decoding strategy using local confidence aggregation and multi-step consistency checks, and a redundancy-aware supervised fine-tuning (SFT) pipeline that aligns models with efficient decoding trajectories. Experimental results on LLaDA and Dream models across benchmarks like GSM8K and HumanEval show that $R^2$-dLLM significantly reduces the Number of Function Evaluations (NFE) by up to 75% while maintaining or improving generation quality.
Entities (8)
Relation Signals (4)
$R^2$-dLLM → addresses → Spatial Redundancy
confidence 100% · To address spatial redundancy, we apply a local confidence and token aggregation mechanism...
$R^2$-dLLM → addresses → Temporal Redundancy
confidence 100% · To handle temporal redundancy, we further introduce a multi-step consistency check...
LLaDA-Instruct-8B → isa → Diffusion Large Language Model
confidence 100% · We evaluate $R^2$-dLLM on two representative diffusion language models, LLaDA-Instruct-8B...
$R^2$-dLLM → improves → Diffusion Large Language Models
confidence 90% · Experiments demonstrate that $R^2$-dLLM consistently reduces the number of decoding steps by up to 75% compared to existing decoding strategies
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregressive generation by enabling parallel token prediction. However, practical dLLM decoding still suffers from high inference latency, which limits deployment. In this work, we observe that a substantial part of this inefficiency comes from recurring redundancy in the decoding process, including spatial redundancy caused by confidence clusters and positional ambiguity, and temporal redundancy caused by repeatedly remasking predictions that have already stabilized. Motivated by these patterns, we propose $R^2$-dLLM, a unified framework for reducing decoding redundancy from both inference and training perspectives. At inference time, we introduce training-free decoding rules that aggregate local confidence and token predictions, and finalize temporally stable tokens to avoid redundant decoding steps. We further propose a redundancy-aware supervised fine-tuning pipeline that aligns the model with efficient decoding trajectories and reduces reliance on manually tuned thresholds. Experiments demonstrate that $R^2$-dLLM consistently reduces the number of decoding steps by up to 75% compared to existing decoding strategies, while maintaining competitive generation quality across different models and tasks. These results validate that decoding redundancy is a central bottleneck in dLLMs, and that explicitly reducing it yields substantial practical efficiency gains.
Tags
Links
- Source: https://arxiv.org/abs/2604.18995v1
- Canonical: https://arxiv.org/abs/2604.18995v1
Trouble viewing inline? Open PDF directly →
Full Text
56,194 characters extracted from source content.
Expand or collapse full text
R 2 -dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Zhenbang Du 1 Kejing Xia 1 Xinrui Zhong 1 Yonggan Fu 2 Nicolai Oswald 2 Binfei Ji 1 Brucek Khailany 2 Pavlo Molchanov 2 Yingyan (Celine) Lin 1 Abstract Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregres- sive generation by enabling parallel token pre- diction. However, practical dLLM decoding still suffers from high inference latency, which lim- its deployment. In this work, we observe that a substantial part of this inefficiency comes from recurring redundancy in the decoding process, in- cluding spatial redundancy caused by confidence clusters and positional ambiguity, and temporal redundancy caused by repeatedly remasking pre- dictions that have already stabilized. Motivated by these patterns, we proposeR 2 -dLLM, a uni- fied framework for reducing decoding redundancy from both inference and training perspectives. At inference time, we introduce training-free decod- ing rules that aggregate local confidence and token predictions, and finalize temporally stable tokens to avoid redundant decoding steps. We further pro- pose a redundancy-aware supervised fine-tuning pipeline that aligns the model with efficient decod- ing trajectories and reduces reliance on manually tuned thresholds. Experiments demonstrate that R 2 -dLLM consistently reduces the number of de- coding steps by up to 75% compared to existing decoding strategies, while maintaining compet- itive generation quality across different models and tasks. These results validate that decoding redundancy is a central bottleneck in dLLMs, and that explicitly reducing it yields substantial prac- tical efficiency gains. 1. Introduction Diffusion Large Language Models (dLLMs) have recently emerged as a compelling alternative to the dominant Au- 1 Georgia Institute of Technology 2 NVIDIA. Correspondence to: Zhenbang Du <zdu89@gatech.edu>. Preprint. April 22, 2026. 304050607080 NFE 60.0 62.5 65.0 67.5 70.0 72.5 75.0 77.5 Accuracy LocalLeap Fast-dLLM dParallel R 2 -dLLM (Ours) 1.40× fewer NFE +2.5% ACC Better Figure 1. Benchmarking the accuracy versus the Number of Func- tion Evaluations (NFE) trade-offs between ourR 2 -dLLM and SOTA dLLM acceleration methods on the GSM8K dataset based on the LLaDA-Instruct-8B model. toRegressive (AR) paradigm (Brown et al., 2020; Ouyang et al., 2022). Unlike AR models, which are constrained by a sequential and token-by-token generation bottleneck, dLLMs leverage bidirectional attention and iterative decod- ing to predict multiple tokens in parallel (Nie et al., 2025; Ye et al., 2025). This non-autoregressive modeling capa- bility theoretically allows for significantly higher decoding throughput and higher hardware utilization for low-batch scenarios, offering a potential direction toward more effi- cient large-scale text generation. However, translating this theoretical benefit into practical speedups remains a significant challenge. Despite their generation flexibility, early-stage dLLMs often suffer from higher inference latency than AR models. Recently, many pioneering works are proposed to address this issue, such as Fast-dLLM (Wu et al., 2025), which introduces confidence- based parallel decoding to accelerate token unmasking; dLLM-Cache (Liu et al., 2025), which implements spe- cialized KV-Caching mechanisms to eliminate unnecessary attention computations across decoding steps; and D2F (Wang et al., 2025b), which adopts a semi-autoregressive attention mechanism instead of fully bidirectional attention to minimize unnecessary attention computations across dif- ferent decoding blocks. While these approaches reduce 1 arXiv:2604.18995v1 [cs.CL] 21 Apr 2026 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction computation from architectural or system perspectives, prac- tical dLLM decoding still exhibits inefficiencies within the decoding trajectory itself, namely redundancy in the decod- ing dynamics. In this paper, we study two forms of redundancy in dLLM decoding under a unified spatio-temporal view: 1) Spatial Redundancy: Unlike AR models that generate tokens at fixed positions, dLLMs must jointly determine token con- tent and positions. This leads to two common patterns. First, dLLMs often produce confidence clusters, where multiple neighboring tokens simultaneously reach high confidence (Kong et al., 2025). These tokens are strongly correlated and could be decoded together, but mainstream existing decoding strategies still process them independently, lead- ing to redundant decoding steps. Second, positional uncer- tainty causes token clusters, where the same token is repeat- edly predicted at multiple adjacent positions. The model then needs extra decoding steps to finalize the exact posi- tion. 2) Temporal Redundancy: During iterative decoding, many tokens converge early and their predictions remain unchanged for multiple steps (Shen et al., 2025). However, existing strategies continue to remask and re-decode these tokens. This leads to unnecessary computation on already confident predictions. To alleviate these issues, we proposeRedundancy-Reduced- dLLM (R 2 -dLLM), a unified framework that targets both spatial and temporal redundancies in dLLM decoding. We first introduce a training-free decoding strategy that directly reduces redundancy during inference. To address spatial redundancy, where models struggle with token positioning, we apply a local confidence and token aggregation mech- anism to stabilize consecutive confidence and token clus- ters. To handle temporal redundancy, we further introduce a multi-step consistency check that finalizes tokens with sta- ble predictions across consecutive steps, thereby avoiding unnecessary remasking. Beyond accelerating the decoding process, this training-free strategy also provides a practi- cal way to measure redundancy scores along generation trajectories. Building on this observation, we further pro- pose a redundancy-aware supervised fine-tuning pipeline that aligns the model with redundancy-less decoding be- haviors. Specifically, we construct the training dataset by filtering generation trajectories with minimal redundancy scores, enabling the model to directly learn redundancy re- duction patterns and reducing its dependence on manually tuned thresholds required by training-free decoding strate- gies. OurR 2 -dLLM, as shown in Figure 1, achieves a favor- able trade-off between decoding efficiency and generation quality by reducing redundancy in the decoding process. In summary, our contributions are as follows: •We present a unified view of spatio-temporal redun- dancy in dLLM decoding, covering locally adjacent high-confidence predictions, token-cluster positional ambiguity, and repeated decoding of already stable tokens. •We proposeR 2 -dLLM, a unified framework that re- duces redundancy from both inference and training perspectives, including training-free decoding strate- gies to alleviate inherent spatial and temporal redun- dancy, and a redundancy-aware supervised fine-tuning pipeline that aligns the model with efficient decoding trajectories. • Extensive experiments across representative bench- marks demonstrate thatR 2 -dLLM significantly re- duces the Number of Function Evaluations (NFE) and improves dLLM inference efficiency, while maintain- ing competitive generation quality. 2. Related Works 2.1. Diffusion Language Models Traditional AR language models generate text tokens from left to right (Brown et al., 2020; Ouyang et al., 2022). While they can produce high-quality text, their decoding through- put is fundamentally limited because tokens are generated sequentially. Recently, diffusion probabilistic models have been extended to discrete text generation (Austin et al., 2021a; Chen et al., 2022; Gulrajani & Hashimoto, 2023). In particular, dLLMs generate text by iteratively predict- ing and unmasking tokens in a sequence. This procedure views generation as denoising and enables multiple tokens to be updated in a single forward pass. LLaDA (Nie et al., 2025) scales diffusion language models to 8B parameters and trains them from scratch, demonstrating that dLLMs can approach the performance of strong AR models such as LLaMA3 8B (Grattafiori et al., 2024). Dream 7B (Ye et al., 2025) instead initializes from a pretrained AR model (Qwen- 2.5 7B) (Yang et al., 2025). While these efforts advance the scale and modeling choices of dLLMs, redundancy during decoding remains prevalent, and the potential parallelism of dLLM decoding is still not fully exploited. 2.2. dLLM Acceleration Recent work has explored accelerating dLLM decoding from multiple angles. Caching-based methods reduce com- putation by reusing KV caches across decoding steps (Ma et al., 2025; Liu et al., 2025; Hu et al., 2025). Fast- dLLM accelerates decoding by using a predefined confi- dence threshold to unmask multiple tokens in parallel (Wu et al., 2025). D2F further reduces inference cost by replac- ing fully bidirectional attention with a semi-autoregressive attention mechanism, avoiding unnecessary attention com- putation across decoding blocks (Wang et al., 2025b). 2 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Quantization-based approaches improve efficiency by com- pressing dLLMs to low-precision formats (Xu & Yang, 2025). Beyond reducing raw computation, several methods target more stable or decisive decoding dynamics. Credit- Decoding (Wang et al., 2025a) introduces Trace Credits to accumulate token-level consistency across steps. dParallel (Chen et al., 2025) applies certainty-forcing distillation to encourage more confident token predictions. LocalLeap (Kong et al., 2025) groups and finalizes locally consistent tokens to reduce repeated updates. Despite these advances, existing approaches often optimize only specific compo- nents of the decoding pipeline and frequently depend on manually tuned hyperparameters, limiting generalization and leading to suboptimal accuracy–efficiency trade-offs. 3. Preliminaries Diffusion Language Models. dLLMs formulate text gener- ation as a discrete denoising process over token sequences. Given a clean sequence x 0 = (x 0,1 ,...,x 0,L ), we define a corruption process parameterized by a noise levelt∈ (0, 1]: each token is independently replaced by a special mask to- kenMwith probabilityt, producing a partially observed sequencex t . Whent = 1, all tokens are masked; whentis close to0,x t is close tox 0 . A modelp θ is trained to predict the original tokens at masked positions conditioned on x t . Training Objective. For each training example, we sample tand constructx t by independent masking. We optimize the denoising objective L(θ) =−E t,x 0 ,x t " 1 t L X i=1 I[x t,i = M ] logp θ (x 0,i | x t ) # . (1) For conditional generation, we apply masking only to re- sponse tokens while keeping the prompt fixed. Inference Phase. Given a promptx p , dLLMs generate a responser 0 ∼ p θ (r 0 | x p )via an iterative unmasking– remasking procedure. Decoding starts from a fully masked response sequence. At each step, the model predicts token distributions for all currently masked positions in parallel; a subset of positions is then finalized (unmasked), while the remaining positions stay masked (or are remasked) for subsequent steps. Vanilla diffusion decoding typically finalizes the top-k(e.g., k = 1in LLaDA (Nie et al., 2025)) predictions based on confidence scores at each step. Fast-dLLM instead final- izes all predictions whose confidence exceeds a predefined thresholdτ, enabling more tokens to be unmasked per step and thereby reducing the number of decoding steps. While these strategies improve inference efficiency, they can still induce redundancy along the decoding trajectory, which we analyze in subsequent sections. 4. Redundancies in dLLMs In this section, we analyze how redundancy arises during dLLM decoding and introduce simple methods to alleviate spatial and temporal redundancy. Confidence Cluster Final answers: Token Cluster Final answers: howmanymore afteranhour Step how 0.92 many 0.90 more 0.91 how 0.82 many 0.83 more 0.86 t-3 t-2 t-1 t after 0.71 an 0.90 hour 0.73 an 0.82 an 0.87 hour 0.63 an 0.78 an 0.83 an 0.80 an 0.68 an 0.72 an 0.78 position uncertainty how 0.63 many 0.61 more 0.65 what 0.42 many 0.39 orange 0.44 decode together Step t-3 t-2 t-1 t Figure 2. An illustration of spatial redundancy. 4.1. Spatial Redundancy Unlike AR models that decode tokens in a fixed order, dLLMs generate tokens in a random order. While this en- ables parallel token prediction, it also introduces uncertainty over token positions. As a result, dLLMs must jointly de- termine both the token content and positions, which causes spatial redundancy during decoding. One common form of spatial redundancy is the confidence clusters. Here, multiple neighboring tokens reach relatively high confidence at the same decoding step and could be de- coded together, as illustrated in Figure 2. Existing decoding strategies mostly process such tokens independently, which delays their decoding and introduces unnecessary steps. Another form of spatial redundancy is the token cluster phenomenon, as shown in Figure 2. In particular, the same token is repeatedly predicted at multiple adjacent positions, where the model has correctly identified the token content but is still uncertain about its exact position. The model then requires multiple steps to resolve this positional ambiguity. To resolve spatial redundancy, we apply a local aggregation rule guided by confidence statistics. Confidence Cluster Aggregation.When multiple neigh- boring tokens reach relatively high confidence at the same step, we treat them as a confidence cluster. For a local windowW , we compute the average confidence γ ave = 1 |W| X i∈W γ i ,(2) whereγ i denotes the confidence score at positioni. If γ ave > τ s (a prespecified threshold), all tokens inWare decoded simutaneously. 3 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Step t text Step t-1 text Step t-2 now now. 0.64 0.73 0.81 0.72 . 0.79 0.720.85 Unmasked Token Original Masked Token Temporal Redundancy Token Cluster few 0.78 LLMfew LLM few few few 0.80 LLM Token Cluster Confidence Cluster Temporal Redundancy Confidence Cluster Decoding Steps Figure 3. Overview of the proposed training-free redundancy reduction strategy during diffusion decoding. Token Cluster Aggregation. When the same token is predicted at multiple adjacent positions (e.g.,≥ 2posi- tions), we treat them as a token cluster. Given a cluster C =i 1 ,...,i k , we select the position with the maximum confidence, i ∗ = arg max i∈C γ i .(3) Ifγ i ∗ > τ s , the token is directly decoded at positioni ∗ , resolving the positional ambiguity. 4.2. Temporal Redundancy 015304560 Token Position 0 8 16 24 32 40 Step Masked Token Redundant Step Decoding Step Unmasked Token Figure 4. An illustration of temporal redundancy, where a “redun- dant step” denotes a token that matches the final output but is not finalized. dLLMs generate text through an iterative unmasking– remasking process. At each decoding step, the model pre- dicts tokens at masked positions and only finalizes a subset of them based on the prediction confidence, while the re- maining tokens are remasked and unmasked in later steps. This can introduce temporal redundancy. In practice, many tokens converge early during decoding. Their predicted token remains unchanged across multiple consecutive steps, as shown in Figure 4. However, existing decoding strategies mostly remask these tokens until the final decoding step or until the corresponding confidence exceeds a sufficiently high threshold (e.g., 0.9). As a result, the model repeatedly decodes already stable tokens, leading to unnecessary computation and decoding steps. To reduce temporal redundancy, we introduce a multi-step consistency check to finalize early-converged tokens. Specif- ically, for a given positioni, if the predicted token remains unchanged formconsecutive decoding steps, and the confi- dence at the last step satisfies γ last i > τ t ,(4) then this token is finalized. In practice, we find that a small value ofm(e.g.,m = 3) works well across different models and tasks. This rule allows stable tokens to be decoded earlier and avoids redundant decoding. Together, these training-free rules explicitly reduce both spatial and temporal redundancy during the dLLM decoding process, as illustrated in Figure 3. 5.Redundancy-aware Supervised Fine-Tuning The aforementioned training-free decoding strategies ef- fectively reduce redundancy during inference. To fur- 4 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction ther enhance the achievable performance, we introduce a redundancy-aware supervised fine-tuning approach that en- courages the model to internalize efficient decoding behav- iors during training, which at the same time can reduce its dependence on heuristic rules at inference time. Candidate 1 Candidate 2 Candidate 3 Candidate 4 Prompt R 2 -dLLM SFT Figure 5. Redundancy-aware training dataset collection: For each prompt, candidate responses with correct answers and the lowest redundancy score R total are selected. 5.1. Training Dataset Collection We construct a redundancy-aware training dataset by ex- plicitly selecting decoding trajectories with minimal redun- dancy, as illustrated in Figure 5. Specifically, for each prompt, we first generate multiple candidate responses us- ing a pretrained dLLM. During generation, we track redun- dancy statistics along each decoding trajectory using the training-free rules introduced in Section 4. Each triggered spatial aggregation contributes one point of spatial redun- dancyR s , and each triggered temporal consistency check contributes one point of temporal redundancyR t . The over- all redundancy score of a candidate response is defined as R total = R s + R t . Given a set of candidate responses for the same prompt, we select the response with the lowestR total , which corresponds to a decoding trajectory that converges faster and exhibits fewer redundancies. The selected response is then paired with the original prompt to form a supervised training exam- ple. Repeating this process across prompts yields a training dataset that captures efficient diffusion decoding behavior. 5.2. Supervised Fine-Tuning To maintain consistency with the inference procedure, we adopt a semi-autoregressive training scheme for supervised fine-tuning (Chen et al., 2025). Given a prompt–response pair from the collected dataset, we concatenate the prompt and response to form a full sequencex 0 . During training, prompt tokens remain unmasked, while response tokens are divided into non-overlapping blocks of sizes(we use s = 32in all experiments, following existing dLLM work). At each training step, we randomly sample a block index i. Meanwhile, blocks with indices smaller thaniare kept unmasked, while blocks with indices larger thaniremain fully masked. Within blocki, we sample a binary mask m ∈ 0, 1 s with masking probabilityp m , wherem j = 1 indicates that tokenjis masked. The model is trained to predict the masked tokens in the selected block, with the loss defined as L SFT = − E 1 s× p m s X j=1 1[m j = 1] logp θ (x i,j | x <i ,x i,¬m ) (5) where x i,¬m denotes the unmasked tokens in block i. We also adopt a complementary sampling strategy (Wu et al., 2025). In particular, for each training example, we construct a complementary mask ̄m = 1− mand create two training views corresponding tomand ̄m. This ensures that each token within a block is supervised once as masked and once as unmasked, which helps reduce overfitting and stabilizes training. 6. Experiments 6.1. Experiment Settings Models and datasets. We evaluateR 2 -dLLM on two representative diffusion language models, LLaDA-Instruct- 8B (Nie et al., 2025) and Dream-v0-Instruct-7B (Ye et al., 2025). We conduct experiments on four benchmarks, in- cluding two math reasoning datasets, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), and two code generation datasets, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021b). We follow common settings and use 5-shot prompting for GSM8K, 4-shot for MATH, 0-shot for HumanEval, and 3-shot for MBPP. For evaluation metrics, we report averaged latency (second), the Number of Function Evaluations (NFE), and accuracy (ACC). Baselines. We compare our method with several base- lines to evaluate both efficiency and performance, includ- ing Vanilla dLLM, which uses Top-1 confidence decoding, Fast-dLLM (Wu et al., 2025), which decodes tokens whose confidence is above a fixed threshold (0.9 by default), Local- Leap (Kong et al., 2025), which decodes neighboring tokens with similar confidence jointly, and dParallel (Chen et al., 2025), which applies a distillation-based method to increase decoding confidence. All methods except Vanilla use the dual KV cache mechanism introduced in Fast-dLLM (Wu et al., 2025). For all experiments, the generation length is set to 256, and the decoding block size is set to 32. Similar to dParallel, we construct training data by sam- pling prompts from open-source datasets. For both models, prompts are sampled from the GSM8K training set (Cobbe 5 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Table 1. Efficiency and accuracy comparison under different decoding strategies (generation length is equal to 256). We use one NVIDIA H200 141 GB GPU to measure the latency (second). Rows of our methods are highlighted in light gray, latency speedup and NFE reduction over Vanilla is shown in green.R 2 -dLLM (F) represents only applying our training-free redundancy reduction, andR 2 -dLLM (T) utilizes SFTto reduce redundancy. TaskMethod LLaDA-Instruct-8BDream-v0-Instruct-7B Latency↓NFE↓ACC↑Latency↓NFE↓ACC↑ GSM8K (5-shot) Vanilla13.1256.077.0310.4256.078.77 Fast-dLLM2.7 ↓4.9× 86.4 ↓66% 78.011.7 ↓6.1× 63.6 ↓75% 76.57 LocalLeap2.3 ↓5.7× 64.4 ↓74% 78.161.4 ↓7.4× 61.0 ↓76% 76.19 dParallel1.4 ↓9.4× 39.7 ↓84% 75.361.3 ↓8.0× 38.7 ↓84% 78.47 R 2 -dLLM (F)2.3 ↓5.7× 63.1 ↓75% 77.561.4 ↓7.4× 53.3 ↓79% 75.51 R 2 -dLLM (T)1.7 ↓7.7× 51.9 ↓79% 77.861.3 ↓8.0× 44.9 ↓82% 80.82 MATH (4-shot) Vanilla9.7256.033.208.3256.038.08 Fast-dLLM3.3 ↓2.9× 107.8 ↓57% 32.602.4 ↓3.5× 90.3 ↓64% 37.58 LocalLeap2.8 ↓3.5× 85.3 ↓66% 32.181.9 ↓4.4× 81.5 ↓68% 37.18 dParallel1.7 ↓5.7× 53.8 ↓78% 30.221.6 ↓5.2× 57.3 ↓77% 35.94 R 2 -dLLM (F)2.9 ↓3.3× 79.7 ↓68% 32.261.7 ↓4.9× 76.3 ↓70% 37.56 R 2 -dLLM (T)2.1 ↓4.6× 68.5 ↓73% 32.601.8 ↓4.6× 66.1 ↓74% 36.88 HumanEval (0-shot) Vanilla5.8256.040.244.7256.057.93 Fast-dLLM2.5 ↓2.3× 90.1 ↓64% 36.591.7 ↓2.8× 80.5 ↓68% 57.93 LocalLeap2.1 ↓2.8× 69.8 ↓72% 35.981.4 ↓3.4× 75.0 ↓70% 53.66 dParallel1.1 ↓5.3× 37.9 ↓85% 33.531.1 ↓4.3× 50.4 ↓80% 53.66 R 2 -dLLM (F)2.3 ↓2.5× 81.7 ↓68% 35.981.3 ↓3.6× 67.5 ↓73% 53.05 R 2 -dLLM (T)1.7 ↓3.4× 64.3 ↓74% 36.590.9 ↓5.2× 44.6 ↓82% 54.27 MBPP (3-shot) Vanilla10.1256.029.407.9256.061.00 Fast-dLLM2.2 ↓4.6× 73.0 ↓71% 25.601.0 ↓7.9× 35.3 ↓86% 53.20 LocalLeap1.9 ↓5.3× 56.1 ↓78% 24.000.9 ↓8.8× 36.3 ↓85% 51.40 dParallel1.0 ↓10.1× 29.7 ↓88% 35.800.8 ↓9.9× 26.6 ↓89% 48.20 R 2 -dLLM (F)1.9 ↓5.3× 61.2 ↓76% 25.600.8 ↓9.9× 29.9 ↓88% 52.60 R 2 -dLLM (T)1.3 ↓7.7× 40.3 ↓84% 37.400.8 ↓9.9× 28.7 ↓88% 54.00 et al., 2021), PRM12K (Lightman et al., 2023), a subset of the Numina-Math dataset (Li et al., 2024), a subset of the AceCode dataset (Le et al., 2022), and a subset of the Big-Math CNK12 dataset (Albalak et al., 2025). During response generation, we use temperature 0.7 for LLaDA, temperature 0.7 and top-pis set to 0.9 for Dream. For each prompt, we generate four candidate responses and compute redundancy statistics for each. Note that no responses from external models or ground truth answers are used. All train- ing and inference are run on four NVIDIA H200 141 GB GPUs. Detailed training configurations are provided in the supplementary material. 6.2. Main Results Table 1 reports the efficiency and accuracy comparison across different decoding strategies, measured by latency, number of function evaluations (NFE), and task accuracy. Overall,R 2 -dLLM achieves consistent latency speedups and NFE reductions while maintaining competitive or im- proved accuracy on both LLaDA and Dream. For LLaDA, on GSM8K,R 2 -dLLM (T) reduces latency from 13.1 s to 1.7 s, corresponding to a7.7×speedup over Vanilla decoding. This is accompanied by a reduction in NFE from 256 to 51.9, while slightly improving accuracy from 77.03 to 77.86. On MATH,R 2 -dLLM (T) achieves a4.6×latency speedup (9.7 s→2.1 s) and reduces NFE to 68.5. On HumanEval, latency is reduced from 5.8 s to 1.7 s (3.4×), with NFE decreasing to 64.3 and accuracy comparable to Fast-dLLM. On MBPP,R 2 -dLLM (T) attains a7.7×latency speedup and reduces NFE from 256 to 40.3, while improving accuracy from 29.40 to 37.40. Although dParallel often achieves lower latency and NFE, it incurs noticeable accuracy degradation on some tasks. In contrast,R 2 -dLLM consistently delivers strong latency speedups with a more favorable accuracy–efficiency trade- off across all tasks. For Dream,R 2 -dLLM yields consistent improvements. On MATH, latency is reduced from 8.3 s to 1.8 s (4.6×), with NFE decreasing to 66.1 and accuracy remaining competitive. On HumanEval,R 2 -dLLM (T) achieves a5.2×latency speedup (4.7 s→0.9 s) and reduces NFE to 44.6, while maintaining accuracy. 6 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Table 2. Efficiency and accuracy comparison between Fast-dLLM andR 2 -dLLM on LLaDA-1.5. NFE reduction is computed relative to Fast-dLLM, shown in green. Method GSM8KMATHHumanEvalMBPP NFE↓ACC↑NFE↓ACC↑NFE↓ACC↑NFE↓ACC↑ Fast-dLLM84.579.81106.632.54104.736.5948.534.40 R 2 -dLLM50.5↓40% 79.2372.5↓32% 32.4081.8↓21% 37.2038.0↓21% 37.00 Overall, these results demonstrate thatR 2 -dLLM achieves substantial end-to-end latency acceleration by reducing re- dundancy while preserving generation quality. 6.3. Performance on dLLM Variant To evaluate the generality ofR 2 -dLLM, we apply it to LLaDA-1.5 (Zhu et al., 2025), a reinforcement learning tuned variant of LLaDA. Since LLaDA-1.5 is built on top of LLaDA-Instruct-8B, we directly reuse the redundancy- aware training dataset collected for LLaDA without addi- tional data collection. Table 2 reports the results on four benchmarks. Compared with Fast-dLLM,R 2 -dLLM consistently reduces the NFEs across all tasks. The relative NFE reduction ranges from about21%to40%. At the same time, accuracy remains comparable. These results show that redundant decoding behaviors still exist after reinforcement learning fine-tuning. Reducing redundancy at the decoding level remains effec- tive even when the base model has been optimized with reinforcement learning. 6.4. Design Choices of Training-Free Redundancy Reduction Components For token cluster aggregation, we decode the position with the highest confidence in each cluster. We compare this design with two alternatives: selecting the middle position (Mid) and a random position (Random). All methods are evaluated on LLaDA with GSM8K under a fixed threshold τ s = 0.75, as shown in Table 3. The confidence-based strategy achieves both lower NFE and higher accuracy than the alternatives, indicating that the most confident position is more likely to be the final correct one. Table 3. Comparison of different token cluster aggregation position selection strategies. StrategyNFE↓ACC↑ Fast-dLLM86.478.01 Mid83.175.82 Random 83.276.12 Conf 80.378.32 For temporal redundancy reduction, we finalize tokens based on their confidence at the last decoding step (Last). We com- pare this choice with using the average (Ave) or maximum (Max) confidence over the consecutive steps. Results in Table 4 show that Ave is overly conservative and yields limited NFE reduction, while Max is more aggressive. The Last strategy achieves the lowest NFE while maintaining high accuracy, suggesting that the most recent confidence provides a reliable signal of token stability. Table 4. Comparison of different temporal redundancy indicators. StrategyNFE↓ACC↑ Fast-dLLM86.478.01 Ave84.477.79 Max71.577.33 Last70.077.51 6.5. Ablation of Training-Free Redundancy Reduction Components Table 5 reports an ablation study of the three training-free redundancy reduction components on LLaDA. Each compo- nent is evaluated by enabling it individually while disabling the others. Enabling any single component consistently reduces NFE compared to the baseline, while accuracy remains compa- rable. Among the three components, confidence cluster aggregation and temporal redundancy provide larger effi- ciency gains, while token cluster aggregation still yields meaningful improvements with small impact on accuracy. On GSM8K, confidence cluster aggregation reduces NFE from 86.4 to 69.6, temporal redundancy reduces it to 70.0, and token cluster aggregation reduces it to 80.3, while main- taining comparable accuracy. A similar pattern is observed on MATH and HumanEval. On MATH, confidence and temporal redundancy reduce NFE from 107.8 to 89.4 and 90.7. On HumanEval, confidence and temporal redundancy reduce NFE from 90.1 to 79.1 and 88.8, while preserving accuracy. When all three components are applied, the model achieves the lowest NFE across all benchmarks. On GSM8K and MATH, NFE is reduced to 63.1 and 79.7, corresponding to reductions of 27.0% and 26.1% compared to the baseline. On HumanEval and MBPP, the full method further reduces NFE to 81.7 and 61.2, while accuracy remains comparable to the baseline. 7 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Table 5. Ablation of training-free redundancy reduction components on LLaDA. Redundancy TypeGSM8KMATHHumanEvalMBPP Spa. (Conf.)Spa. (Token.)Temp.NFE↓ACC↑NFE↓ACC↑NFE↓ACC↑NFE↓ACC↑ ✗86.478.01107.832.6090.136.5973.025.60 ✓69.677.3389.432.6279.136.5961.525.40 ✓80.378.32104.232.2086.535.3772.525.40 ✓70.077.5190.732.4088.835.9863.225.80 ✓63.177.5679.732.2681.735.9861.225.60 010002000300040005000 Iteration 1.5 2.0 2.5 3.0 3.5 Loss 010002000300040005000 Iteration 55 60 65 70 75 80 R total 010002000300040005000 Iteration 60 70 80 NFE 010002000300040005000 Iteration 65 70 75 80 85 90 Accuracy Figure 6. Training dynamics of LLaDA-Instruct-8B on GSM8K, showing loss, R total , NFE, and accuracy over training iterations. 6.6. Training Dynamics of Redundancy-aware Supervised Fine-Tuning Figure 6 shows the training dynamics of LLaDA-Instruct- 8B on GSM8K, including loss, sampledR total , NFE, and accuracy. As training proceeds, bothR total and NFE de- crease steadily, indicating that the model gradually learns to eliminate redundancies thus reduces the decoding steps. In particular,R total and NFE drops sharply during the early stage of training and continues to decrease steadily with further optimization. Meanwhile, accuracy remains stable throughout the training process, with only minor fluctua- tions. These results demonstrate that our method effectively reduces redundancy during training, leading to more effi- cient inference while preserving generation quality. 6.7. Training Sample Selection Strategy Our Redundancy-aware SFT selects the correct response with the lowestR total (Min). To assess the effectiveness of this selection strategy, we conduct an ablation study by selecting the correct response with the highestR total (Max). As shown in Table 6, the Min strategy consistently leads to lower NFE than the Max strategy. On GSM8K, Min reduces NFE from 58.6 to 51.9. On HumanEval, Min further reduces NFE from 80.0 to 64.3. Across both datasets, accuracy remains comparable between the two strategies. These results show that selecting responses with lower re- dundancy produces more efficient decoding trajectories, while maintaining generation quality. Table 6. Comparison of different response selection strategies based on redundancy on GSM8K and HumanEval. Strategy GSM8KHumanEval NFE↓ACC↑NFE↓ACC↑ Max59.977.9482.436.59 Min 51.977.8664.336.59 7. Conclusion In this paper, we proposeR 2 -dLLM, a unified framework for accelerating dLLM decoding by explicitly identifying and reducing spatio-temporal redundancy. Through an anal- ysis of decoding dynamics, we show that a large portion of inference inefficiency in dLLMs stems from redundant computation during iterative decoding, including spatial redundancy induced by confidence clusters and token clus- ters, as well as temporal redundancy caused by repeatedly remasking early-converged tokens. Building on these insights, we introduce a set of training- free redundancy reduction strategies that directly streamline the decoding process, and further develop a redundancy- aware supervised fine-tuning (SFT) pipeline that aligns model behavior with efficient decoding trajectories. Exten- sive experiments across GSM8K, MATH, HumanEval, and MBPP demonstrate thatR 2 -dLLM consistently achieves substantial reductions in decoding steps while preserving, and in some cases improving, generation accuracy, outper- forming existing diffusion decoding baselines in the accu- racy–efficiency trade-off. Overall, our results highlight decoding redundancy as a fun- damental bottleneck in dLLMs. By making redundancy explicit and optimizing against it at both inference and train- ing time,R 2 -dLLM provides a practical and effective way toward practical, scalable diffusion-based text generation. 8 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Impact Statement This paper presents work whose goal is to advance the field of Large Language Model. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References Albalak, A., Phung, D., Lile, N., Rafailov, R., Gandhi, K., Castricato, L., Singh, A., Blagden, C., Xiang, V., Mahan, D., et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502.17387, 2025. Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021a. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021b. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021. Chen, T., Zhang, R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. arXiv preprint arXiv:2208.04202, 2022. Chen, Z., Fang, G., Ma, X., Yu, R., and Wang, X. dparallel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488, 2025. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Gulrajani, I. and Hashimoto, T. B. Likelihood-based diffu- sion language models. Advances in Neural Information Processing Systems, 36:16693–16715, 2023. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. Hu, Z., Meng, J., Akhauri, Y., Abdelfattah, M. S., Seo, J.-s., Zhang, Z., and Gupta, U. Accelerating diffusion language model inference via efficient kv caching and guided diffusion. arXiv preprint arXiv:2505.21467, 2025. Kong, F., Zhang, J., Liu, Y., Wu, Z., Tian, Y., Zhou, G., et al. Accelerating diffusion llm inference via local deter- minism propagation. arXiv preprint arXiv:2510.07081, 2025. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), p. 1207–1216, Stan- ford, CA, 2000. Morgan Kaufmann. Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Ad- vances in Neural Information Processing Systems, 35: 21314–21328, 2022. Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S., Rasul, K., Yu, L., Jiang, A. Q., Shen, Z., et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. Liu, Z., Yang, Y., Zhang, Y., Chen, J., Zou, C., Wei, Q., Wang, S., and Zhang, L. dllm-cache: Accelerating diffu- sion large language models with adaptive caching. arXiv preprint arXiv:2506.06295, 2025. 9 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Ma, X., Yu, R., Fang, G., and Wang, X. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025. Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Shen, J., Sarkar, G., Ro, Y., Sridhar, S. N., Wang, Z., Akella, A., and Kundu, S.Improving the through- put of diffusion-based large language models via a training-free confidence-aware calibration. arXiv preprint arXiv:2512.07173, 2025. Wang, K., Jiang, Z., Feng, H., Zhao, W., Liu, L., Li, J., Lan, Z., and Lin, W. Creditdecoding: Accelerating parallel decoding in diffusion large language models with trace credits. arXiv preprint arXiv:2510.06133, 2025a. Wang, X., Xu, C., Jin, Y., Jin, J., Zhang, H., and Deng, Z. Diffusion llms can do faster-than-ar inference via dis- crete diffusion forcing. arXiv preprint arXiv:2508.09192, 2025b. Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., and Xie, E. Fast-dllm: Training-free acceler- ation of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025. Xu, C. and Yang, D. Dllmquant: Quantizing diffusion-based large language models. arXiv preprint arXiv:2508.14090, 2025. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Hu, J., Zhou, J., Chen, J., Lin, Y., Wen, J.-R., et al. Llada 1.5: Variance- reduced preference optimization for large language diffu- sion models. arXiv preprint arXiv:2505.19223, 2025. 10 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction A. Experiment Details We summarize the training configuration used for redundancy-aware supervised fine-tuning in Table 7. For all models, we standardize the generation length to a maximum of 768 tokens. Both prompt and response sequences are padded or truncated to this fixed length using the end-of-sequence token. We adopt a uniform LoRA (Hu et al., 2022) configuration across all models, with LoRA rank set to 32 and LoRA alpha set to 32. For LLaDA-based models, including LLaDA-Instruct-8B and LLaDA-1.5, we train the model for 6 epochs, while Dream-v0-Instruct-7B is trained for 3 epochs. All models are trained using four NVIDIA H200 141 GB GPU, with a per-GPU batch size of 8 and a gradient accumulation step of 2, resulting in an effective global batch size of 64. We provide detailed thresholds settings forR 2 -dLLM (F) in Table 8. We useτ t = τ s = 0.8for LLaDA andτ t = τ s = 0.85 for Dream to find redundancy in the dataset-collection process. The training dataset contains approximately 52k samples for LLaDA and 91k samples for Dream. Table 7. Training configuration for redundancy-aware supervised fine-tuning . Base ModelLoRA RankLoRA AlphaLearning RateBatch SizeEpoch LLaDA-Instruct-8B32322× 10 −5 646 LLaDA-1.532322× 10 −5 646 Dream-v0-Instruct-7B32322× 10 −5 643 Table 8. Selected τ t and τ s across datasets and models. Dataset LLaDA-Instruct-8BDream-v0-Instruct-7B τ t τ s τ t τ s GSM8K0.700.800.800.85 MATH0.700.800.800.85 HumanEval0.850.850.850.85 MBPP0.800.800.800.85 A.1. Threshold Sensitivity Analysis 0.850.800.750.70 s 60 65 70 75 80 NFE 0.850.800.750.70 s 70 72 74 76 78 80 Accuracy 0.850.800.750.70 t 68 70 72 74 76 78 80 NFE 0.850.800.750.70 t 75 76 77 78 79 Accuracy Figure 7. The sensitivity of τ s and τ t . We conduct experiments to study the sensitivity of the spatial thresholdτ s and the temporal thresholdτ t . Figure 7 reports the effect of varying each threshold on NFE and accuracy using LLaDA on GSM8K, while keeping other settings fixed. For the spatial thresholdτ s , decreasingτ s leads to a monotonic reduction in NFE, indicating that more aggressive spatial aggregation allows more tokens to be finalized earlier. However, accuracy also gradually decreases asτ s becomes smaller, showing a clear efficiency–accuracy trade-off. This suggests that overly permissive spatial aggregation may decode tokens prematurely. A similar trend is observed for the temporal thresholdτ t . Loweringτ t significantly reduces NFE by finalizing stable tokens earlier, but also results in a moderate accuracy drop. This indicates that temporal redundancy can be effectively reduced, but excessive early finalization may harm generation quality. Overall, the results show thatR 2 -dLLM is robust within a reasonable range of τ s and τ t . 11 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction A.2. Performance under Longer Generation Lengths To explore the performance ofR 2 -dLLM under longer generation lengths, we further evaluate LLaDA with generation lengths of 512, 768, and 1024 tokens. As shown in Figure 8,R 2 -dLLM scales more robustly as generation length increases. On GSM8K, dParallel achieves lower NFE thanR 2 -dLLM, but this comes with a clear accuracy drop, which becomes more significant for longer sequences. R 2 -dLLM maintains stable accuracy while keeping NFE much lower than Fast-dLLM. On MBPP, the trade-off of dParallel is more evident. Although dParallel attains low NFE, its accuracy degrades sharply as length increases. In contrast,R 2 -dLLM consistently preserves high accuracy with substantially lower NFE than Fast-dLLM across all lengths. These results indicate thatR 2 -dLLM generalizes well to longer decoding length, enabling efficient inference without sacrificing accuracy. 2565127681024 Length 40 60 80 100 120 140 NFE GSM8K NFE Fast-dLLM dParallel R 2 -dLLM 2565127681024 Length 73 74 75 76 77 78 Accuracy GSM8K ACC 2565127681024 Length 40 60 80 100 120 140 NFE MBPP NFE 2565127681024 Length 10 15 20 25 30 35 Accuracy MBPP ACC Figure 8. Effect of Generation Length on Decoding Efficiency and Accuracy. A.3. Quantitative Analysis of Redundancy We further quantify how often each redundancy pattern appears during decoding. We conduct this analysis on LLaDA GSM8K and report the average number of events per response. Table 9 reports the averaged event counts per response. For Fast-dLLM and R 2 -dLLM (T), we report potential trigger counts: on the generated trajectories, we evaluate how many times each redundancy rule would be triggered under the same criterion. For R 2 -dLLM (F), we report actual trigger counts, since the confidence-cluster, token-cluster, and temporal mechanisms are applied during real decoding. Therefore, the Fast-dLLM and R 2 -dLLM (T) rows measure the amount of redundancy that remains in the trajectories, while the R 2 -dLLM (F) row reflects what is actually triggered. Since earlier mechanisms in R 2 -dLLM (F) may already remove later redundancies, these counts are not fully decoupled. Table 9. Average redundancy-event counts per response on LLaDA GSM8K. MethodCount typeConfidence-clusterToken-clusterTemporal-finalization Fast-dLLMPotential47.453.1526.82 R 2 -dLLM (F)Actual40.041.116.17 R 2 -dLLM (T)Potential23.850.873.50 Overall, both R 2 -dLLM (F) and R 2 -dLLM (T) produce substantially fewer redundancy events than Fast-dLLM across all three categories. In particular, the training-free variant already sharply reduces token-cluster and temporal-finalization events, while the training-based variant further reduces the remaining redundancy in the generated trajectories. This per-type breakdown is consistent with Figure 6, where the overall redundancy score decreases during training, and supports our claim that the proposed framework improves decoding efficiency by reducing redundancy throughout the decoding process. A.4. Response-Level Analysis of Low-Redundancy Supervision A natural question is why standard supervised fine-tuning on the selected low-redundancy responses can improve decoding efficiency. To better understand this effect, we perform a response-level paired analysis between the lowest-R total and highest-R total candidate responses for the same prompt. Specifically, we compare the Min-R total and Max-R total candidates for the same prompt on GSM8K from LLaDA. 12 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction For clarity, All pairs reports percentages over all 7k matched pairs. Differ-only reports percentages over the subset of pairs where the two responses differ on the corresponding metric. Same-final-number reports the same comparison after restricting to the subset of pairs whose final numerical answers are identical. This control helps reduce the effect of answer mismatch and focuses the comparison on differences in response form. We use the following response-level metrics: • Same final number: whether the two responses end with the same final numerical answer. • Textually identical: whether the two responses are exactly identical at the text level. • Repeated-bigram rate: how often short two-word phrases are repeated within a response; lower values indicate less local repetition. •Lexical diversity: the ratio of unique tokens to total tokens; higher values indicate more varied wording and less formulaic phrasing. •Template markers: common fixed reasoning lead-ins such as “To solve this problem. . . ” or “Let’s break down. . . ”; fewer markers indicate less templated structure. As shown in Tables 10 and 11, low-redundancy responses are typically less repetitive, more lexically diverse, and less templated. This provides a concrete response-level explanation for why standard SFT on selected responses can still improve decoding efficiency. Table 10. Pairwise response-level comparison between the Min-R total and Max-R total candidates on matched GSM8K pairs. MetricAll pairs (%)Differ-only (%)Same-final-number (%) Same final number89.99– Textually identical1.70– Lower repeated-bigram rate60.465.265.0 Higher lexical diversity63.368.068.0 Fewer template markers–75.975.8 Table 11. Examples of overused lead-ins in the high-redundancy subset. Lead-inHigh-redundancy (%)Low-redundancy (%) “Let’s break down the problem step by . . . ”5.923.85 “To solve this problem, we need to determine . . . ”2.090.52 “To solve this problem, we need to calculate . . . ”1.920.76 B. Case Study We also show some cases generated with the vanilla model and ourR 2 -dLLM across different tasks. This demonstrates thatR 2 -dLLM maintains comparable generation quality while significantly reducing generation latency and NFE. All experiments in this section are conducted on two NVIDIA RTX 6000 (Blackwell) GPUs. Case: GSM8K (5-shot), Geneation Length 256, Block Length 32 Question: Stephanie is decorating 24 cupcakes for a birthday party, but she needs more candles. She currently has a total of 30 candles. She wants to decorate half of the cupcakes with 1 candle each and the other half of the cupcakes with 2 candles each. How many additional candles does Stephanie need to complete the cupcakes? Ground-truth answer: 6 LLaDA-8B-Instruct (Inference Step: 90, Latency: 3.16s) 13 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction For half of the cupcakes, Stephanie wants to use 1 candle each. Since half of the cupcakes is24/2and she plans to use 1 candle each for this half of the cupcakes, Stephanie needs (24/2)× 1 = 12 candles for this half of the cupcakes. For the other half of the cupcakes, Stephanie wants to use 2 candles. Therefore, she will need(24/2)× 2 = 24candles for this half of the cupcakes. Because Stephanie needs 12 candles for half of the cupcakes and 24 candles for the other half, she needs a total of12 + 24 = 36 candles. Since Stephanie needs 36 candles to decorate all the cupcakes and she currently has 30 candles, Stephanie needs36− 30 = 6 additional candles. Final Answer: The final answer is 6. LLaDA-8B-Instruct with R 2 -dLLM (Inference Step: 68, Latency: 2.57s) For half of the cupcakes, Stephanie wants to use 1 candle each. Since half of the cupcakes is24/2and she plans to use 1 candle each for this half of the cupcakes, Stephanie needs (24/2)× 1 = 12 candles for this half of the cupcakes. For the other half of the cupcakes, Stephanie wants to use 2 candles. Therefore, she will need(24/2)× 2 = 24candles for this half of the cupcakes. Because Stephanie needs 12 candles for half of the cupcakes and 24 candles for the other half, she needs a total of12 + 24 = 36 candles. Since Stephanie needs 36 candles to decorate all the cupcakes and she currently has 30 candles, Stephanie needs36− 30 = 6 additional candles. Final Answer: The final answer is 6. Case: MATH (4-shot), Geneation Length 256, Block Length 32 Question: If the system of equations 6x− 4y = a, 6y− 9x = b has a solution (x,y) where x and y are both nonzero, find a b (assuming b̸= 0). Ground-truth answer: − 2 3 LLaDA-8B-Instruct (Inference Step: 170, Latency: 5.38s) Multiply the first equation by− 3 2 : − 3 2 (6x− 4y) =− 3 2 a, 6y− 9x =− 3 2 a. Since the second equation gives 6y− 9x = b, we have b =− 3 2 a, so a b = − 2 3 . Final Answer: The final answer is− 2 3 . I hope it is correct. LLaDA-8B-Instruct with R 2 -dLLM(Inference Step: 109, Latency: 3.65s) If we multiply the first equation by− 3 2 , we obtain 6y− 9x =− 3 2 a. 14 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction Since we also know that 6y− 9x = b, we have − 3 2 a = b⇒ a b = − 2 3 . Final Answer: The final answer is− 2 3 . I hope it is correct. Case: humaneval (0-shot), Geneation Length 256, Block Length 32 Question: from typing import List, Tuple def rolling_max(numbers: List[int]) -> List[int]: """ From a given list of integers, generate a list of rolling maximum element found until given moment in the sequence. >>> rolling_max([1, 2, 3, 2, 3, 4, 2]) [1, 2, 3, 3, 3, 4, 4] """ LLaDA-8B-Instruct (Inference Step: 95, Latency: 2.84s) Here’s the implementation of the rolling_max function: from typing import List, Tuple def rolling_max(numbers: List[int]) -> List[int]: """ From a given list of integers, generate a list of rolling maximum element found until given moment in the sequence. >>> rolling_max([1, 2, 3, 2, 3, 4, 2]) [1, 2, 3, 3, 3, 4, 4] """ if not numbers: return [] result = [] for num in numbers: result.append(max(num, result[-1])) return result # Example usage: if __name__ == "__main__": import doctest doctest.testmod() This function iterates through the input list and keeps track of the rolling maximum by comparing each element with the last element in the result list. The result list is updated with the maximum of the current element and the last element in the result list. The doctest module is used to run the provided test cases. LLaDA-8B-Instruct with R 2 -dLLM (Inference Step: 73, Latency: 2.23s) Here’s the implementation of the rolling_max function: from typing import List, Tuple def rolling_max(numbers: List[int]) -> List[int]: """ From a given list of integers, generate a list of rolling maximum element found until given moment in the sequence. >>> rolling_max([1, 2, 3, 2, 3, 4, 2]) 15 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction [1, 2, 3, 3, 3, 4, 4] """ if not numbers: return [] rolling_max = [numbers[0]] for num in numbers[1:]: rolling_max.append(max(rolling_max[-1], num)) return rolling_max # Example usage: if __name__ == "__main__": import doctest doctest.testmod() This functionrolling_maxtakes a list of integers and returns a list of rolling maximum values. The rolling maximum is defined as the maximum value encountered so far in the sequence. The example usage provided in the docstring demonstrates how to use the function Case: mbpp (3-shot), Geneation Length 256, Block Length 32 Question: You are an expert Python programmer, and here is your task: Write a python function to find the last digit when factorial of a divides factorial of b. Your code should pass these tests: assert compute_Last_Digit(2,4) == 2 assert compute_Last_Digit(6,8) == 6 assert compute_Last_Digit(1,2) == 2 LLaDA-8B-Instruct (Inference Step: 143, Latency: 4.61s) def compute_Last_Digit(a, b): # Compute the factorial of a and b fact_a = 1 fact_b = 1 for i in range(1, a + 1): fact_a * = i fact_b * = i # Find the last digit of the result last_digit = fact_a % 10 return last_digit # Test cases assert compute_Last_Digit(2, 4) == 2 assert compute_Last_Digit(6, 8) == 6 assert compute_Last_Digit(1, 2) == 2 # Explanation: # 1. Compute the factorial of a and b. # 2. Find the last digit of the factorial of a. # 3. Return the last digit as the result. # Note: The problem asks for the last digit when the factorial of a divides the factorial of b. # This is equivalent to finding the last digit of the factorial of a. # Since a is a divisor of b, we only need to find the last digit of the factorial of a. LLaDA-8B-Instruct with R 2 -dLLM (Inference Step: 42, Latency: 1.62s) 16 Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction import math def factorial(n): result = 1 for i in range(1, n + 1): result * = i return result def compute_Last_Digit(a, b): fact_a = factorial(a) fact_b = factorial(b) result = fact_b // fact_a return result % 10 # Tests assert compute_Last_Digit(2, 4) == 2 assert compute_Last_Digit(6, 8) == 6 assert compute_Last_Digit(1, 2) == 2 17