Paper deep dive
Does Uniform Discrete Diffusion Need Time?
Chunsan Hong, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Jong Chul Ye, Yuki Mitsufuji
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/28/2026, 4:03:36 AM
Summary
This paper investigates the necessity of explicit time conditioning in Uniform Discrete Diffusion Models (UDMs). Theoretically, it proves that while the population-optimal UDM predictor depends on time to weigh hypotheses about which observed tokens are reliable, this dependence becomes negligible in finite-data settings typical of language modeling. Empirically, trained language UDMs show limited time sensitivity over most of the diffusion trajectory, and time-agnostic predictors often match or outperform time-conditioned models. The authors suggest that explicit time conditioning is often unnecessary and propose hybrid approaches that condition only in high-noise regimes.
Entities (6)
Relation Signals (6)
Masked Diffusion Models → istimeagnostic → True
confidence 95% · the population-optimal clean-token predictor does not need the diffusion time explicitly... time-agnostic MDMs achieve comparable empirical performance
Uniform Discrete Diffusion Models → uses → Time Conditioning
confidence 95% · Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning
Match Count → determines → Time Sensitivity
confidence 93% · time acts only through how many context positions match... time only reweights predictions across different match counts
Time Conditioning → isunnecessaryfor → Finite-Data Language UDMs
confidence 92% · this dependence can become negligible in finite-data settings relevant to language... time-agnostic predictors remain competitive with, and often outperform, time-conditioned models
Time Conditioning → isnecessaryfor → Population-Optimal Predictor
confidence 90% · we first show that the population-optimal UDM predictor generally depends on time
Time-Agnostic Predictors → outperforms → Time-Conditioned Models
confidence 88% · time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice.
Tags
Links
- Source: https://arxiv.org/abs/2609.30977v1
- Canonical: https://arxiv.org/abs/2609.30977v1
Trouble viewing inline? Open PDF directly →
Full Text
127,138 characters extracted from source content.
Expand or collapse full text
Does Uniform Discrete Diffusion Need Time? Chunsan Hong Chieh-Hsin Lai Satoshi Hayakawa Yuhta Takida Jong Chul Ye Yuki Mitsufuji KAIST Sony Group Corporation The University of Tokyo Abstract Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice. †footnotetext: † authors. 1 Introduction Diffusion models usually tell the neural network how noisy its input is (Ho et al., 2020; Song et al., 2021; Lai et al., 2025). This information, commonly provided through time conditioning, allows the same model to make different predictions at different stages of denoising. But does the model actually need to be told the time? Recent work on continuous diffusion suggests that explicit time conditioning may not always be essential. Sun et al. (2025) show that removing time conditioning causes only a small performance degradation in several settings. Separately, Wang and Du (2025) develop a generative formulation whose neural model is time-independent, further showing that generation need not always be built around an explicitly time-conditioned predictor. For discrete diffusion models, however, the answer depends strongly on the corruption mechanism. In masked diffusion models (MDMs), corrupted tokens are replaced by a special [MASK] symbol. The corrupted sequence therefore reveals which positions have been corrupted, and the population-optimal clean-token predictor does not need the diffusion time explicitly (Zheng et al., 2025). Consistent with this property, time-agnostic MDMs achieve comparable empirical performance (Sahoo et al., 2024) and have been adopted in large language diffusion models (Nie et al., 2025; Ye et al., 2025). Uniform diffusion models (UDMs) are fundamentally different. A corrupted token is replaced by an ordinary vocabulary token sampled from the uniform distribution, so the model cannot directly tell whether an observed token was preserved from the clean sequence or produced by noise (Liu et al., 2025; Amin et al., 2025). The diffusion time can therefore carry information that is absent from the corrupted sequence itself, by indicating how much the observed context should be trusted. This makes it unclear whether the time-agnostic property of MDMs can extend to UDMs. This question has become increasingly important as UDMs scale to large language modeling. Recent studies show that scaled UDMs can outperform MDMs in several settings (Sahoo et al., 2026b; von Rütte et al., 2026), followed by the development of large language UDMs (DiffusionGemma Team, 2026; Sahoo et al., 2026a). Some of these models already use time-agnostic predictors, yet it remains unclear why this works, when time actually changes the optimal prediction, and how much explicit time conditioning helps in practice. Throughout this work, we call a UDM time-agnostic when its neural predictor does not receive diffusion time as an explicit input. The forward corruption process, noise schedule, and other analytic time-dependent quantities remain unchanged. We ask: What role does time play in UDMs, and when do UDMs need explicit time conditioning? First, we characterize why and how the population-optimal UDM predictor depends on time. Uniform corruption creates uncertainty about which observed tokens are reliable. Consequently, every subset of observed tokens constitutes a possible hypothesis about the reliable context. We show that time only controls the relative weights assigned to these hypotheses. Thus, unlike MDMs where corrupted positions are explicitly identified, the population-optimal UDM predictor is generally time-dependent, but its dependence on time has a simple and structured form. Second, we show that finite training data can strongly suppress this dependence. Population-level time dependence arises from reweighting competing hypotheses about which observed tokens are reliable. In finite language data, however, the enormous discrete sequence space leaves training examples highly separated. A corrupted sequence therefore remains much closer to its originating clean sequence than to competing sequences, causing the posterior to collapse onto few hypotheses and weaken time-dependent reweighting. Only near the high-noise endpoint, where multiple hypotheses become plausible, can time affect the prediction. We make this intuition quantitative, deriving a bound that links the time sensitivity of the empirical oracle to the post-corruption separation margin. We empirically verify that language data exhibit large separation over most noise levels. Finally, we find that learned language UDMs make limited use of explicit time over most of the diffusion trajectory. The above finite-data result concerns the empirical oracle, not a learned neural network. A learned model may recover additional time dependence through generalization or architectural inductive bias; we therefore separately analyze trained UDMs. Their measured time sensitivity is weak over much of the trajectory and becomes substantially larger in the high-noise regime. The predictive benefit of time conditioning is similarly concentrated near high noise. A hybrid model that enables time conditioning only in this regime performs nearly the same as a fully time-conditioned model, while fully time-agnostic UDMs remain competitive with, and in several settings outperform, their time-conditioned counterparts across datasets and training objectives. In summary, our results shift the question from whether time matters in UDMs to where it matters enough to justify explicit conditioning. This helps explain why time-agnostic UDMs can work well despite a time-dependent population optimum, and suggests a broader design space for simpler architectures, training methods, and samplers that use time only where it is useful. 2 Background We briefly introduce the UDM formulation and the recent variants used in this work. See Appendix A for more details and related work. Notation. Let =1,…,K⊂ΔKX=\ e_1,…, e_K\⊂ _K be a finite token space, where each token is represented as a one-hot vector and ΔK _K is the K-category probability simplex. The vocabulary size is K=||K=|X|, and the uniform prior is K−1K^-1 1_K. A sequence is =(1,…,L)∈L x=( x^1,…, x^L)∈X^L, and t z_t denotes its corrupted version at time t∈[0,1]t∈[0,1], where t=0t=0 is data and t=1t=1 is the uniform prior. αt _t denotes a decreasing noise schedule with α0≈1 _0≈ 1 and α1≈0 _1≈ 0, and ρt≔Kαt/(1−αt) _t K _t/(1- _t). For simplicity, we use the standard αt=1−t _t=1-t hereafter; formal theoretical results for general schedules are provided in Appendix where the distinction matters. Unless stated otherwise, expectations are over ∼pdata x p_data and t∼qt(⋅∣) z_t q_t(· x). ⟨⋅,⋅⟩ ·,· , ⊙ , and ⋅ 1\·\ denote the dot product, Hadamard product, and indicator function, respectively. Uniform diffusion models. A UDM independently corrupts each token toward the uniform prior: qt(ti∣i)=Cat(ti,αti+(1−αt)K−1K). q_t( z_t^i x^i)=Cat ( z_t^i; _t x^i+(1- _t)K^-1 1_K ). (1) For s<ts<t, the single token posterior qs|0,tq_s|0,t follows from Bayes’ rule using the s→ts→ t transition and the s-marginal (Eq. (10)). Schiff et al. (2025) parameterize the reverse process by plugging θi(t,t) x_θ^i( z_t,t) into this posterior, ps|tθ(si∣t)≔qs|0,t(si∣i=θi(t,t),t),p_s|t^θ( z_s^i z_t) q_s|0,t ( z_s^i x^i= x_θ^i( z_t,t), z_t ), as detailed in Eq. (3). The standard UDM is trained with the continuous-time NELBO: ℒUDM(θ)=∫01,t∑i=1L−αt′Kαt[Kx¯θ,ℓi−Kx¯ℓi+∑j≠ℓix¯jix¯ℓilogx¯θ,ℓix¯jix¯θ,jix¯ℓi]t, _UDM(θ)= _0^1E_ x, z_t _i=1^L - _t K _t [ K x_θ, _i^i- K x_ _i^i+ _j≠ _i x_j^i x_ _i^i x_θ, _i^i x_j^i x_θ,j^i x_ _i^i ]dt, (2) where ¯i≔Kαti+(1−αt)K ν^i K _t ν^i+(1- _t) 1_K and ℓi _i denotes the non-zero index of ti z_t^i. Recent UDM formulations. Among several recent UDM formulations, we focus on two representative frameworks: Diffusion Duality (Sahoo et al., 2025) and the leave-one-out formulation of Gourevitch et al. (2026). Diffusion Duality retains the same uniform forward process and UDM NELBO, but represents the uniform-state marginal through the argmax of a Gaussian corruption and uses a curriculum from continuous relaxations to discrete inputs. In contrast, Gourevitch et al. (2026) show that the neural predictor in the UDM bridge has the leave-one-out (LOO) posterior as its population optimum: each clean token is predicted from the corrupted context excluding the observation at that same position, see Eq. (4). They further derive an analytic local-token correction that converts the LOO predictor into the full denoising posterior, allowing the underlying predictor to be trained with a simple cross-entropy loss. We refer to this CE-trained formulation as LOO+CE. 3 The Role of Time at the Population Optimum in UDMs The corruption mechanism determines whether time needs to enter the clean-token predictor. In MDMs, the special mask symbol reveals which positions have been corrupted, so the population-optimal predictor can rely directly on the visible tokens and is time-agnostic (Zheng et al., 2025). UDMs are different. Because uniform corruption replaces a token by another ordinary vocabulary token, the model cannot directly tell whether an observed token was copied from the clean sequence or produced by noise. Time can therefore be informative because it changes how likely the observed context is to have been preserved from the clean sequence. This motivates two questions: 1. Is the population-optimal UDM predictor inherently time-dependent? 2. If so, through what structure does time affect its prediction? The first determines whether removing explicit time conditioning changes the population optimum; the second reveals what information time actually contributes. 3.1 The Population Target of the Uniform Channel The UDM reverse kernel has a seemingly complicated plug-in form (Gourevitch et al., 2026; Sahoo et al., 2025; Schiff et al., 2025): ps|tθ(si∣t)=qs|0,t(si∣i=θi(t,t),t)=Cat(si,ρtti⊙θi(⋅,t)+ρtρsti+(1−ρtρs)ρsθi(⋅,t)+K+ρsρt⟨ti,θi(t,t)⟩+1). p_s|t^θ\! ( z_s^i z_t )=q_s|0,t( z_s^i x^i= x_θ^i( z_t,t), z_t)=Cat\! ( z_s^i; _t z_t^i x_θ^i(·,t)+ _t _s z_t^i+ (1- _t _s ) _s x_θ^i(·,t)+ 1_KK+ _s _t z_t^i, x_θ^i( z_t,t) +1 ). (3) Importantly, all neural-model dependence enters through the clean-token prediction θ(t,t) x_θ( z_t,t). Thus, rather than analyzing the reverse kernel directly, we ask what this predictor should output at the population optimum and how that prediction depends on time. Gourevitch et al. (2026) show that the neural prediction used by the UDM bridge has a leave-one-out (LOO) posterior as its population target. Fix a target position i∈[L]i∈[L]. Let X∼pdataX p_data denote a random clean sequence, T∼Unif([0,1])T ([0,1]) the corruption time, and Z∣(X=,T=t)∼qt(⋅∣)Z (X= x,T=t) q_t(· x) its corrupted version. We write Z−iZ^-i for the corrupted sequence with position i removed, and analogously for −i z^-i. Thus, Z−i=−iZ^-i= z^-i means that all corrupted tokens except the target position are observed. The LOO target predicts the clean token at position i from this remaining corrupted context: ⋆i(,t)≔[Xi|Z−i=−i,T=t]. x_ ^i( z,t) [X^i\, |\,Z^-i= z^-i,T=t ]. (4) Since XiX^i is one-hot, ⋆,ki(,t) x_ ,k^i( z,t) is the probability that the clean token is k e_k given the other corrupted tokens. The local observation i z^i can then be incorporated exactly through the known uniform corruption likelihood: Pr(Xi=k∣Z=,T=t)∝qt(i∣Xi=k)⋆,ki(,t) \! (X^i= e_k Z= z,T=t ) q_t\! ( z^i X^i= e_k ) x_ ,k^i( z,t). That is, ⋆i x_ ^i summarizes what the surrounding corrupted tokens imply about the clean token, while the bridge uses the known corruption process to account for the token observed at position i itself. Importantly, this LOO posterior is exactly the population minimizer of the UDM NELBO in Eq. (2): θ⋆(,t)=⋆(,t),for PT,Z-almost every (t,). x_θ ( z,t)= x_ ( z,t), P_T,Z-almost every (t, z). Substituting ⋆ x_ into Eq. (3) gives the corresponding population-optimal reverse kernel: ps|tθ⋆(si∣t)=qs|0,t(si∣i=⋆i(t,t),t). p_s|t^θ \! ( z_s^i z_t )=q_s|0,t( z_s^i x^i= x_ ^i( z_t,t), z_t). Eq. (4) tells us what the optimal predictor is, but it does not yet reveal how time changes its prediction: time appears only inside the conditioning distribution. To make this dependence explicit, recall that each token under uniform corruption can be viewed as arising either from the copy branch or from the uniform-noise branch. We therefore decompose the prediction over hypotheses about which context positions were copied from the clean sequence, which leads to the following property. Proposition 1 (Explicit time decomposition of the population-optimal UDM predictor). Fix i∈[L]i∈[L] and a corrupted observation z. For any index set S⊆[L]∖iS [L] \i\, write [S]≔(j)j∈S,X[S]≔(Xj)j∈S z[S] ( z^j )_j∈ S, X[S] (X^j )_j∈ S Then the population-optimal predictor satisfies ⋆i(,t)=∑S⊆[L]∖iPr(S∣Z−i=−i,T=t)[Xi|X[S]=[S]], x_ ^i( z,t)= _S [L] \i\ (S Z^-i= z^-i,T=t)E [X^i\, |\,X[S]= z[S] ], (5) Pr(S∣Z−i=−i,T=t)∝ρt|S|Pr(X[S]=[S]). (S Z^-i= z^-i,T=t) _t^|S| \! (X[S]= z[S] ). Here, Pr(X[S]=[S])=∑:[S]=[S]pdata() (X[S]= z[S])= _ x:\, x[S]= z[S]p_data( x) is the data probability mass of the partial observation [S] z[S] under the hypothesis index set S. Proposition 1 makes the role of time explicit: time controls how much of the hypothesis z[S] z[S] should be trusted. Given a corrupted context −i z^-i, the model does not know which observed tokens were from the clean sequence or from uniform noise. It therefore considers every subset S⊆[L]∖iS [L] \i\ as a possible set of copied positions. If the positions in S were indeed copied, then [Xi|X[S]=[S]]E\! [X^i\, |\,X[S]= z[S] ] is the corresponding prediction of the target token from that clean context. The final prediction averages over all such possibilities according to their posterior weights. We now interpret this time-weighted posterior in more detail. Using ρt=Kαt/(1−αt) _t=K _t/(1- _t), Pr(S∣Z−i=−i,T=t)∝ρt|S|Pr(X[S]=[S])=Pr(X[S]=[S])⏟DatadependentK|S|⏟Fixed vocabularyfactor(αt1−αt)|S|⏟Timedependent. (S Z^-i= z^-i,T=t) _t^|S| \! (X[S]= z[S] )= \! (X[S]= z[S] )_ subarraycData\\ dependent subarray K^|S|_ subarraycFixed vocabulary\\ factor subarray ( _t1- _t )^|S|_ subarraycTime\\ dependent subarray. This factorization cleanly separates the sources of the posterior weighting. The data-dependent term measures how compatible the observed values on S are with the data distribution. The time-dependent term changes the relative preference for subsets of different sizes as the corruption level varies, while K|S|K^|S| is a fixed channel factor that amplifies this size dependence. Near the clean endpoint, ρt _t is large, so the model places more weight on explanations in which many observed tokens were copied from the clean sequence. As noise increases, ρt _t decreases and explanations with fewer copied positions receive relatively more weight. Thus, unlike in MDMs, the population-optimal UDM predictor is generally time-dependent: ⋆i(,⋅)≢const x_ ^i( z,·) . 3.2 Time Acts Through the Amount of Matched Context Proposition 1 expresses the optimal prediction through all 2L−12^L-1 possible hypotheses of copied context positions. We can further simplify this optimum using the symmetry of the uniform channel (Eq. (1)): every position at which a candidate clean sequence matches the corrupted observation contributes the same factor 1+ρt1+ _t. Hence, the likelihood depends on a matching set M⊆[L]M [L] only through its size, via (1+ρt)|M|(1+ _t)^|M|, regardless of the token identities, the locations of the matches, or whether a matching token was copied from clean sequence or independently produced by uniform corruption. Thus, time acts only through how many context positions match. This motivates the following statistic. Definition 1 (Match count random variable). For random variables X and Z, and a fixed target position i, define Mi(X,Z)≔∑j∈[L]∖iXj=Zj.M^i(X,Z) _j∈[L] \i\ 1\X^j=Z^j\. Thus, MiM^i counts how many context tokens of a candidate clean sequence agree with the corrupted observation, excluding the target position i, without recording which particular positions match. Theorem 1 (Time dependence through the matched-context count). For every target position i, Xi⟂T|Z−i,Mi,X^i T Z^-i,M^i, i.e., XiX^i and T are conditionally independent given Z−iZ^-i and MiM^i. Moreover, Eq. (4) admits the exact decomposition ⋆i(,t)=∑m=0L−1wmi(,t)mi(), x_ ^i( z,t)= _m=0^L-1w_m^i( z,t)h_m^i( z), wmi(,t)≔Pr(Mi=m|Z−i=−i,T=t)∝(1+ρt)mPr(Mi(X,)=m), w_m^i( z,t) (M^i=m\, |\,Z^-i= z^-i,T=t ) (1+ _t)^m (M^i(X, z)=m ), mi()≔[Xi|Mi=m,Z−i=−i], _m^i( z) [X^i\, |\,M^i=m,Z^-i= z^-i ], where Pr(Mi(X,)=m)=∑Mi(,)=mpdata() (M^i(X, z)=m)= _M^i( x, z)=mp_data( x) is the data probability mass of clean sequences that match z in exactly m context positions. Theorem 1 gives a compact description of the entire time dependence. Once the corrupted context and its match count MiM^i are fixed, the distribution of the clean token XiX^i no longer depends on time. The component prediction mi()h_m^i( z) is therefore time-independent; time only changes how strongly different match counts m are weighted through (1+ρt)m(1+ _t)^m. In other words, the population-optimal UDM does not learn an unrelated clean-token prediction at every noise level. Time acts through a single scalar ρt _t that reweights predictions associated with different amounts of matched context. 3.3 Characterizing the Influence of Time in Uniform Diffusion Theorem 1 shows where time enters the population-optimal prediction: it changes the weights over the match count MiM^i. We next quantify how strongly this reweighting changes the prediction. Proposition 2 (Exact temporal sensitivity). Let ηt=log(1+ρt) _t= (1+ _t) and let k e_k denote the k-th one-hot vector. Then the temporal derivative of the context posterior is ∂ηt⋆i(,t)=Covt,(Mii(),Mi), ∂ _t x_ ^i( z,t)=Cov_t, z (h_M^i^i( z),M^i ), where the covariance is taken componentwise. Equivalently, for any candidate category k with ⋆,ki(,t)>0 x_ ,k^i( z,t)>0, ∂ηtlog⋆,ki(,t)=t,[Mi|Xi=k]−t,[Mi], ∂ _t x_ ,k^i( z,t)=E_t, z [M^i\, |\,X^i= e_k ]-E_t, z[M^i], (6) where the subscript (t,)(t, z) denotes the posterior distribution Pr(⋅∣Z−i=−i,T=t) (· Z^-i= z^-i,T=t). Proposition 2 makes the effect of time particularly transparent. Larger ηt _t corresponds to a cleaner channel. Eq. (6) therefore says that a candidate token k e_k gains probability toward the clean endpoint when clean sequences predicting k e_k have more average matches with the observed context than the posterior average. If they have fewer matches, its probability decreases. Thus, the strength of time dependence is governed by differences in matched-context support across candidate tokens. Population-level takeaway. The population-optimal UDM predictor is generally time-dependent, but time only reweights predictions across different match counts. Its effect is strongest when competing clean candidates are supported by different amounts of matched context. 4 Time Sensitivity of the Finite-Data Oracle At the population level, a rich data distribution can assign substantial probability to multiple clean sequences that plausibly explain the same corrupted context. In this regime, changing t can meaningfully reweight these competing explanations and thereby change the optimal prediction. The situation can be very different under finite training data. An empirical distribution places probability mass on only finitely many sequences in the enormous discrete space LX^L. If these sequences are sufficiently separated, a corrupted training sample z can remain much more compatible with its originating sequence x than with any competing sequence that predicts a different token at position i. The posterior may then already be strongly concentrated, leaving little room for the time-dependent reweighting to change the resulting prediction. This motivates the central question: When does finite training data suppress the time sensitivity of the oracle UDM predictor? We show that the answer is governed by the geometry of the finite training set: how well separated training sequences are in the enormous discrete sequence space. Our theoretical analysis proceeds in two steps: 1) when training sequences are sufficiently far apart, the originating sequence is likely to remain separated from competing sequences after corruption, and 2) whenever this separation is sufficiently large, the oracle Fisher sensitivity for corrupted sequence is strongly suppressed over a broad range of diffusion times, with the guarantee weakening toward the high-noise endpoint. 4.1 When Finite High-Dimensional Data Suppress Time Sensitivity We now make the finite-data mechanism precise. Given N training sequences 1,…,N∈L x_1,…, x_N∈X^L, define their empirical distribution as ≔1N∑n=1Nδn 1N _n=1^N _ x_n. This is the finite-data counterpart of pdatap_data considered in the previous section, with the oracle supported only on the observed training sequences. The first key question is how much of the separation between distant training sequences survives even after corruption. Consider a corrupted z generated from a training sequence x. For prediction at position i, only training sequences with ′i≠i x i≠ x^i can support a different target token. What matters is therefore whether the originating sequence x retains a match-count advantage over every such competitor after corruption. We quantify this advantage through the following separation margin. Definition 2 (Hamming separation and Δ -separability). For ∈supp() x ( ), define its minimum Hamming distance to the remaining training sequences as d()≔min′∈supp()∖∑j=1L′j≠j.d( x) _ x ( ) \ x\ _j=1^L 1\! \ x j≠ x^j \. For an integer Δ≥1 ≥ 1, we say that a corrupted sequence z is Δ -separable with respect to x at coordinate i if Mi(,)−Mi(′,)≥Δ∀′∈supp() with ′i≠i. M^i( x, z)-M^i( x , z)≥ ∀\, x ( ) with x i≠ x^i. (7) Thus, Δ is exactly the match-count advantage retained by the originating sequence over every training sequence that would predict a different token at position i. The following result shows that such an advantage survives when the training sequences are sufficiently separated. Proposition 3 (Δ -separation under UDM corruption (informal)). Given a training sequence x, let T∼Unif([0,1])T ([0,1]) and Z∼qT(⋅∣)Z q_T(· x). For any integer Δ≥1 ≥ 1 and i∈[L]i∈[L], when K≫LK L and d()d( x) is large, Pr(Z is Δ-separable w.r.t. at i)≳1−logN+Δ(1−e−1)d(). \! (Z is $ $-separable w.r.t.\ $ x$ at $i$ ) 1- N+ (1-e^-1)d( x). This simplified bound exposes the tradeoff directly. If training sequences are sufficiently separated, i.e. Hamming separation d()d( x) is large, the source x remains distinguishable after corruption. Conversely, more competing sequences or a larger required margin Δ make the event harder. Importantly, the dependence on dataset size is only logarithmic in N. If d()d( x) is too small relative to logN+Δ N+ , the bound can become uninformative. The formal finite-K statement is in Appendix B.4. The second key question is how separation suppresses the time sensitivity of the oracle prediction. For fixed z, coordinate i, and time t, define the Fisher time sensitivity ℐti()≔∑k:⋆,ki(,t)>0⋆,ki(,t)(∂tlog⋆,ki(,t))2=∑k:⋆,ki(,t)>0(∂t⋆,ki(,t))2⋆,ki(,t).I_t^i( z) _k:\, x_ ,k^i( z,t)>0 x_ ,k^i( z,t) ( ∂ t x_ ,k^i( z,t) )^2= _k:\, x_ ,k^i( z,t)>0 ( ∂ t x_ ,k^i( z,t) )^2 x_ ,k^i( z,t). (8) It measures how quickly the optimal categorical prediction changes when only the diffusion time is varied. Indeed, for infinitesimal δ, we have KL[⋆i(,t)∥⋆i(,t+δ)]=δ22ℐti()+o(δ2)D_KL [ x_ ^i( z,t)\, \|\, x_ ^i( z,t+δ) ]= δ^22I_t^i( z)+o(δ^2). Hence, small ℐti()I_t^i( z) means that nearby times produce nearly indistinguishable oracle predictions. The next result provides the key link: a large match-count advantage directly limits this time sensitivity. Theorem 2 (Fisher sensitivity with finite data (informal)). For Δ -separable z with respect to x at coordinate i, ℐti()≤NL2K2tΔ−2(K−(K−1)t)Δ+2.I_t^i( z)≤ NL^2 K^2t -2 (K-(K-1)t ) +2. Moreover, for Δ≥2 ≥ 2, the right-hand side is strictly increasing in t∈(0,1)t∈(0,1). The bound becomes weaker toward the high-noise endpoint, where corruption makes the originating sequence harder to distinguish from competing sequences. At lower noise levels, a larger separation margin Δ keeps these competitors much less plausible, leaving less room for changes in time to alter the oracle prediction. If Δ≥2 ≥ 2, a convenient consequence is: sup0<t≤1−K−1/4ℐti()≤NL2K−(3Δ−2)/4. _0<t≤ 1-K^-1/4I_t^i( z)≤ NL^2K^-(3 -2)/4. (9) For practical LLM vocabulary sizes, the interval (0,1−K−1/4](0,1-K^-1/4] already covers most of the diffusion trajectory. More importantly, Eq. (9) can be inverted to ask how much separation is sufficient for a desired sensitivity level. To guarantee sup0<t≤1−K−1/4ℐti()≤ζ _0<t≤ 1-K^-1/4I_t^i( z)≤ζ, it is sufficient that Δ≥max2,23+43logKNL2ζ ≥ \2,\, 23+ 43 _K NL^2ζ \. Thus, although the Fisher bound contains N and L2L^2, the required match-count advantage grows only logarithmically with dataset size and sequence length. Each additional unit of Δ further suppresses the bound by a factor of K−3/4K^-3/4. The bounds remain meaningful at LLM scale. Consider a K2-Horizon-375B-scale pretraining configuration (IFM Team, 2026) with N≈1.9×109N≈ 1.9× 10^9, L=8192L=8192, and K=250,624K=250,624. Here, by Eq. (9), only an integer margin Δ≥7 ≥ 7 guarantees sup0<t≤0.955ℐti()≤10−5 _0<t≤ 0.955I_t^i( z)≤ 10^-5. In other words, even with billions of training sequences, the originating sequence only needs to retain a seven-token match-count advantage over every competitor predicting a different target token. By the formal bound in Proposition 3 (Appendix B.4), d()≥900d( x)≥ 900, about 11%11\% of the sequence length, already gives a 0.950.95 lower bound on 77-separability, while d()≥451d( x)≥ 451 gives a 0.90.9. Finally, small Fisher sensitivity controls finite changes in the oracle prediction. If sup0<u≤1−ϵℐui()≤ζ _0<u≤ 1-εI_u^i( z)≤ζ, then for any t,s∈(0,1−ϵ)t,s∈(0,1-ε), ‖⋆i(,t)−⋆i(,s)‖2≤|t−s|2ζ. \| x_ ^i( z,t)- x_ ^i( z,s) \|_2≤ |t-s|2 ζ. Thus, once separation makes the Fisher bound small, the finite-data oracle is nearly invariant to time. 4.2 Oracle Has Near-Zero Time Sensitivity on Real Language Data Figure 1: Empirical separation of OWT training data. Left: distribution of the minimum Hamming distance d()d( x) from each training sequence to the remaining data, with median d()=988d( x)=988. Right: distribution of the largest post-corruption separation margin Δ across diffusion times. The dashed line marks Δ=6 =6, which implies an oracle Fisher-sensitivity bound below 10−510^-5 for t≤0.933t≤ 0.933. Small separation occurs primarily when z is sampled near the extreme high-noise endpoint. We now examine whether the finite-data regime predicted by Proposition 3 and Eq. (9) actually occurs in real language data. We consider OWT (Gokaslan et al., 2019), a standard benchmark in discrete diffusion language modeling, with K=50,258K=50,258, L=1024L=1024, and N=8.7×106N=8.7× 10^6. For these values, sup0<t≤0.933ℐti()≤ζ _0<t≤ 0.933I_t^i( z)≤ζ with ζ=10−5ζ=10^-5 only requires Δ≥6 ≥ 6. The formal bound in Appendix B.4 further shows that d()≥698d( x)≥ 698 is sufficient for at least a 0.950.95 lower bound on 66-separability. We now compute d()d( x) and the post-corruption separation margin Δ directly on OWT. Before diffusion corruption, clean OWT is highly separated in Hamming space. As shown in Figure 1 (Left), the distribution of d()d( x) is concentrated near the maximum sequence length L=1024L=1024, with median d()=988d( x)=988, well above the sufficient value 698698. We next test whether this separation survives UDM corruption. For each t∈0.05,0.1,…,0.95,0.99t∈\0.05,0.1,…,0.95,0.99\, we sample 20002000 pairs ∼ x and t∼qt(⋅∣) z_t q_t(· x) and measure the largest Δ for which t z_t is Δ -separable. Figure 1 (Right) shows that the observed margin remains above 66 over almost the entire diffusion trajectory, becoming small primarily when z is sampled near the extreme high-noise endpoint. These measurements confirm that OWT lies in the separation regime predicted by our theory: most of the training pairs (,)( x, z) satisfies 6-separability, and whenever the observed margin satisfies Δ≥6 ≥ 6, Eq. (9) gives sup0<t≤0.933ℐti()≤10−5 _0<t≤ 0.933I_t^i( z)≤ 10^-5, implying negligible oracle time sensitivity over most of the diffusion trajectory. Near the extreme high-noise endpoint, smaller separation margins weaken this guarantee and leave more room for time to influence the oracle prediction. Finite-data takeaway. When finite training sequences remain well separated after corruption, the originating sequence retains a clear match-count advantage and the oracle becomes nearly insensitive to time over most of the diffusion trajectory. This regime is empirically observed in practical language data, with the separation weakening mainly near the high-noise endpoint. 5 You Might Not Need Time in UDMs for Text Generation Sections 3 and 4 reveal a tension: the population-optimal UDM predictor generally depends on time, while the finite-data oracle can be nearly time-insensitive over most of the time. The remaining question is whether a learned UDM actually benefits from time conditioning. Two possibilities arise: • Time-variant UDMs may benefit from recovering useful time dependence through generalization or inductive bias beyond the finite-data oracle. • Time-agnostic UDMs may benefit from removing largely unnecessary time dependence. If time dependence remains weak over most of the trajectory, removing explicit time conditioning may simplify the model and improve learning. We test these directly by measuring the time sensitivity of learned UDMs and comparing time-conditioned and time-agnostic models. Throughout this section, Duality denotes the framework of Sahoo et al. (2025), while LOO+CE denotes the LOO denoiser trained with cross-entropy (Gourevitch et al., 2026). See Appendix A.2 for framework details and C.2 for experimental settings. 5.1 Learned UDMs Exhibit Limited Time Dependence in Language Figure 2: Left: Fisher sensitivity of Duality and LOO+CE, varying z sampling time tcorruption∈0.01,0.50,0.99t_corruption∈\0.01,0.50,0.99\ and evaluating Fisher sensitivity on every time coordinate. Right: Per-time NLL difference between time-agnostic and time-variant UDMs. Learned time sensitivity follows the finite-data prediction. We first examine whether the time sensitivity of learned UDMs exhibits the qualitative behavior predicted by our finite-data analysis. Using the official OWT checkpoints of Duality and LOO+CE, we sample corrupted sequences z at tcorruption∈0.01,0.50,0.99t_corruption∈\0.01,0.50,0.99\. For each fixed z, we then vary the time input used for prediction and measure its Fisher sensitivity across the entire diffusion trajectory. As shown in Figure 2 (left), the learned models qualitatively match our finite-data analysis: First, Fisher sensitivity increases with tcorruptiont_corruption. Stronger corruption tends to reduce the separation margin Δ , weakening the finite-data suppression guarantee in Eq. (9). Second, for a fixed corrupted input, Fisher sensitivity generally increases with the evaluation time and rises sharply near the high-noise endpoint, where the bound in Theorem 2 becomes weaker. Finally, we use held-out test sequences for this analysis, although our finite-data theory is stated for the empirical training distribution. The similar qualitative behavior on held-out data suggests that limited time sensitivity can extend beyond training examples, while our theory does not directly establish this behavior off the empirical support. The localized time sensitivity is robust and provides limited practical benefit. We next ask whether this concentration of time sensitivity can be resolved by how time is parameterized to the model. We train conventional UDMs (Schiff et al., 2025) on LM1B. We keep the forward process unchanged and reparameterize only the scalar passed to the time embedding as t^=f(t) t=f(t), where f(t)∈−log(1−t),logt,logit(t),tf(t)∈\- (1-t), t,logit(t),t\. These parameterizations allocate different resolution across diffusion times. Nevertheless, Figure 4 in Appendix shows a consistent pattern across all parameterizations: learned UDMs exhibit weak time sensitivity except near the high-noise endpoint. We next compare the time-conditioned model with its time-agnostic counterpart. Figure 2 (Right) shows that time conditioning improves per-time NLL mainly near t=1t=1, precisely where the learned Fisher sensitivity becomes substantial. Over most of the diffusion trajectory, however, the time-agnostic model achieves lower NLL and also performs better overall. Together, these results suggest that learned UDMs make substantial use of explicit time conditioning only in a limited high-noise region, and that modeling this localized dependence provides limited practical benefit. 5.2 Time-agnostic UDM versus Time-variant UDM We next compare time-agnostic and time-variant UDMs across broader settings, including different datasets, tokenizers, and training objectives. Beyond the conventional UDM studied in Section 5.1, we now evaluate Duality and LOO+CE. We follow widely adopted experimental settings in discrete diffusion language modeling (Sahoo et al., 2025; Arriola et al., 2025; Schiff et al., 2025), evaluating models on LM1B (Chelba et al., 2014), both with and without sentence packing, and OWT, all trained for 1M steps, with the same backbone. See Appendix C.2 for more detailed settings. Table 1: Validation perplexity (PPL) across several configurations. Lower is better. Hybrid uses the time-modulation layer only for t>0.8t>0.8. PPLs lower than the time-variant baseline are underlined. Dataset Objective Variant (139M) Hybrid (139M) Agnostic (131M) Agnostic (139M) LM1B Duality 29.99 29.92 30.32 29.76 LOO+CE 30.15 29.67 29.18 28.93 LM1B (Packed) Duality 33.48 33.17 33.69 33.35 LOO+CE 33.94 33.37 33.24 32.75 Results on LM1B. We compare four setttings for time conditioning: 1) Time-variant model, 2) Time-agnostic model where we remove the time-modulation layers, reducing the parameter count from 139M to 131M, 3) Time-agnostic model with 139M parameters obtained by adding one Transformer block to 2), and 4) Hybrid model that only activates time-modulation layer when t>0.8t>0.8. Table 1 shows that the hybrid model that only learns time-sensitivity on high-noise regime is almost equivalent to the time-variant model, and moreover, time-agnostic UDMs remain consistently competitive with their time-dependent counterparts across training frameworks. Table 2: Validation PPL on OWT. Agnostic Variant Duality 24.33 25.54 LOO+CE 25.29 25.29 Results on OWT. We compare time-variant and time-agnostic models with the same parameter size. As shown in Table 2, time-agnostic UDMs are comparable to time-variant UDMs. Generative PPL was comparable for Duality, while the time-agnostic model performed better for LOO+CE; see Appendix C.1. Learned-model takeaway. Time matters mainly near the high-noise endpoint. Consistently, time-agnostic UDMs remain competitive across datasets and objectives, suggesting that explicit time conditioning may offer limited benefit over most of the trajectory. 6 Conclusion We studied the role of time in population optimal UDMs, and why it weakens under finite training data. Consistently, learned UDMs exhibit limited time dependence over most of the trajectory, while time-agnostic models remain competitive across datasets and frameworks. We expect this view opens a door to future research that allows a simpler foundation for architectures, training, and samplers. References Amin et al. (2025) A. N. Amin, N. Gruver, and A. G. Wilson Why masking diffusion works: condition on the jump schedule for improved discrete diffusion. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §1. Arriola et al. (2025) M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations (ICLR), Cited by: §5.2. Austin et al. (2021) J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §A.1. Campbell et al. (2022) A. Campbell, J. Benton, V. De Bortoli, T. Rainforth, G. Deligiannidis, and A. Doucet A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §A.1, §A.1, §A.2, §A.2, §A.2. Chelba et al. (2014) C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson One billion word benchmark for measuring progress in statistical language modeling. In Interspeech 2014, p. 2635–2639. External Links: Document, ISSN 2958-1796 Cited by: §5.2. DiffusionGemma Team (2026) DiffusionGemma Team DiffusionGemma technical report. External Links: 2608.00146, Link Cited by: §A.1, §1. Gokaslan et al. (2019) A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex OpenWebText corpus. Note: http://Skylion007.github.io/OpenWebTextCorpus Cited by: §4.2. Gourevitch et al. (2026) S. Gourevitch, Y. Janati, D. Shariatian, U. Simsekli, E. Moulines, E. P. Xing, and A. Durmus Uniform diffusion models revisited: leave-one-out denoiser and absorbing state reformulation. External Links: 2605.22765, Link Cited by: §A.1, §A.2, §A.2, §B.1, §B.7, §C.2, §C.3, §2, §3.1, §3.1, §5. He et al. (2023) Z. He, T. Sun, Q. Tang, K. Wang, X. Huang, and X. Qiu DiffusionBERT: improving generative masked language models with diffusion models. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §C.2. Helbling et al. (2026) A. Helbling, S. G. Hernandez, B. Hoover, D. H. Chau, and P. Ram What time is it? how data geometry makes time conditioning optional for flow matching. External Links: 2605.08344, Link Cited by: §A.1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §1. Hong et al. (2026) C. Hong, S. An, M. Kim, and J. C. Ye Improving discrete diffusion unmasking policies beyond explicit reference policies. In International Conference on Learning Representations (ICLR), Cited by: §A.1. Hoogeboom et al. (2021) E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling Argmax flows and multinomial diffusion: learning categorical distributions. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §A.1. IFM Team (2026) IFM Team Introducing k2 horizon: frontier performance, radically open. External Links: Link Cited by: §4.1. Lai et al. (2025) C. Lai, Y. Song, D. Kim, Y. Mitsufuji, and S. Ermon The principles of diffusion models. External Links: 2510.21890, Link Cited by: §1. Lee et al. (2025) S. Lee, S. Kim, S. Kim, J. Park, and D. Park Effective test-time scaling of discrete diffusion through iterative refinement. External Links: 2511.05562, Link Cited by: §A.1. Liu et al. (2025) S. Liu, J. Nam, A. Campbell, H. Stark, Y. Xu, T. Jaakkola, and R. Gomez-Bombarelli Think while you generate: discrete diffusion with planned denoising. In International Conference on Learning Representations (ICLR), Cited by: §A.1, §1. Lou et al. (2024) A. Lou, C. Meng, and S. Ermon Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning (ICML), Cited by: §A.1, §A.1, §A.1, §A.2, §A.2, §A.2, §A.2, §A.2, §C.2. Nie et al. (2025) S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li Large language diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §1. Ou et al. (2025) J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations (ICLR), Cited by: §A.1. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In IEEE International Conference on Computer Vision (ICCV), Cited by: §C.2. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR) 21 (140), p. 1–67. Cited by: §C.2. Sahoo et al. (2024) S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §A.1, §C.2, §C.2, §1. Sahoo et al. (2026a) S. S. Sahoo, L. Chen, K. Pham, J. Geuter, C. Dwivedi, V. Pimpalkhute, Y. Akhauri, A. Moreno, M. Yurochkin, Z. Wang, M. Elhoushi, N. Dey, S. Bergsma, J. Hestness, J. Thickstun, E. Xing, and Z. Liu Unlocking lossless speedups in llms via discrete diffusion. External Links: 2609.04010, Link Cited by: §A.1, §1. Sahoo et al. (2025) S. S. Sahoo, J. Deschenaux, A. Gokaslan, G. Wang, J. Chiu, and V. Kuleshov The diffusion duality. In International Conference on Machine Learning (ICML), Cited by: §A.1, §A.1, §A.2, §A.2, §C.1, §C.2, §C.2, §C.3, §C.3, §2, §3.1, §5.2, §5. Sahoo et al. (2026b) S. S. Sahoo, J. Lemercier, Z. Yang, J. Deschenaux, J. Liu, J. Thickstun, and A. Jukić Scaling beyond masked diffusion language models. In International Conference on Machine Learning (ICML), Cited by: §A.1, §1. Sahraee-Ardakan et al. (2026) M. Sahraee-Ardakan, M. Delbracio, and P. Milanfar The geometry of noise: why diffusion models don’t need noise conditioning. External Links: 2602.18428, Link Cited by: §A.1. Schiff et al. (2025) Y. Schiff, S. S. Sahoo, H. Phung, G. Wang, S. Boshar, H. Dalla-torre, B. P. de Almeida, A. M. Rush, T. Pierrot, and V. Kuleshov Simple guidance mechanisms for discrete diffusion models. In International Conference on Learning Representations (ICLR), Cited by: §A.1, §A.2, §A.2, §A.2, §A.2, §B.7, §B.7, §C.2, §C.2, §C.3, §C.3, §2, §3.1, §5.1, §5.2. Shi et al. (2024) J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.1, §C.2. Sohl-Dickstein et al. (2015) J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), Cited by: §A.1. Song et al. (2021) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), Cited by: §A.1, §1. Sun et al. (2025) Q. Sun, Z. Jiang, H. Zhao, and K. He Is noise conditioning necessary for denoising generative models?. In International Conference on Machine Learning (ICML), Cited by: §A.1, §1. von Rütte et al. (2026) D. von Rütte, J. Fluri, O. Pooladzandi, B. Schölkopf, T. Hofmann, and A. Orvieto Scaling behavior of discrete diffusion language models. In International Conference on Learning Representations (ICLR), Cited by: §A.1, §1. Wang and Du (2025) R. Wang and Y. Du Equilibrium matching: generative modeling with implicit energy-based models. External Links: 2510.02300, Link Cited by: §1. Ye et al. (2025) J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong Dream 7b: diffusion large language models. External Links: 2508.15487, Link Cited by: §A.1, §1. Zheng et al. (2025) K. Zheng, Y. Chen, H. Mao, M. Liu, J. Zhu, and Q. Zhang Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In International Conference on Learning Representations (ICLR), Cited by: §A.1, §C.1, §1, §3. Contents 1 Introduction 2 Background 3 The Role of Time at the Population Optimum in UDMs 3.1 The Population Target of the Uniform Channel 3.2 Time Acts Through the Amount of Matched Context 3.3 Characterizing the Influence of Time in Uniform Diffusion 4 Time Sensitivity of the Finite-Data Oracle 4.1 When Finite High-Dimensional Data Suppress Time Sensitivity 4.2 Oracle Has Near-Zero Time Sensitivity on Real Language Data 5 You Might Not Need Time in UDMs for Text Generation 5.1 Learned UDMs Exhibit Limited Time Dependence in Language 5.2 Time-agnostic UDM versus Time-variant UDM 6 Conclusion References A Background A.1 Related Work A.2 Various Uniform Discrete Diffusion Frameworks B Omitted Proofs B.1 Proof of Proposition B.2 Proof of Theorem B.3 Proof of Proposition B.4 Proof of Proposition B.5 Proof of Theorem B.6 Proof of Eq. () B.7 Discussion on the Finite Data Setting C Experimental Details C.1 Additional Experimental Results C.2 Experimental Settings C.3 Details on Time-agnostic Models Appendix A Background A.1 Related Work Discrete diffusion models. Diffusion generative modeling was originally introduced by Sohl-Dickstein et al. (2015), and has become a dominant approach for continuous domains such as images (Ho et al., 2020; Song et al., 2021). By defining a time-variant stochastic differential equation (Song et al., 2021), diffusion models establish a continuous path between the data and noise distributions. This success has motivated extensions to discrete domains, leading to discrete diffusion models (Austin et al., 2021; Hoogeboom et al., 2021; Campbell et al., 2022). The forward corruption process is typically designed in one of two ways: (i) uniform corruption, which replaces tokens with random vocabulary elements (Lou et al., 2024; Sahoo et al., 2025), or (i) masking-based corruption, which maps tokens to an absorbing [MASK] state (Sahoo et al., 2024; Shi et al., 2024). Uniform discrete diffusion models and plug-in parametrization. Early uniform diffusion models (UDMs), together with masked diffusion models (MDMs), were formulated as discrete diffusion models by specifying particular forward corruption kernels (Austin et al., 2021; Campbell et al., 2022; Lou et al., 2024; Hoogeboom et al., 2021). Among those, the uniform bridge plug-in parameterization was first introduced in multinomial diffusion by Hoogeboom et al. (2021). Using uniform bridge plug-in parameterization, Schiff et al. (2025) developed a continuous-time formulation and derived its NELBO, reporting improved performance over earlier uniform-diffusion baselines. Building on this, Sahoo et al. (2025) reinterpreted the uniform kernel as the argmax pushforward of a Gaussian corruption kernel and proposed a curriculum that initially uses a biased continuous-input relaxation before returning to discrete-input UDM training. More recently, Gourevitch et al. (2026) characterized the unrestricted population optimum of the bridge plug-in predictor as a leave-one-out posterior, which enables training with a simple cross-entropy objective under a leave-one-out parametrization. Notably, recent studies have shown that UDMs can outperform MDMs when scaled up in the language domain (Sahoo et al., 2026b; von Rütte et al., 2026), and large UDMs such as DiffusionGemma (DiffusionGemma Team, 2026) and Uno (Sahoo et al., 2026a) have recently been released. Despite growing attention to the application of UDMs in the language domain, it remains unclear whether time conditioning is actually necessary for UDMs. In this regard, our work takes the population-optimum characterization as a starting point and analyzes its dependence on time in detail, showing that, in the language domain, the time sensitivity of UDMs can be largely confined to the endpoint region of t. Time-agnostic masked diffusion models. Earlier continuous-time formulations for mask corruption kernels commonly used time-conditioned networks to parameterize the reverse process (Campbell et al., 2022; Lou et al., 2024). However, Zheng et al. (2025); Ou et al. (2025) showed that MDMs do not inherently require time conditioning in the model, and Sahoo et al. (2024) demonstrated empirically that time-agnostic MDMs can slightly outperform their time-variant counterparts. Consequently, recent large-scale MDMs (Ye et al., 2025; Nie et al., 2025) have largely adopted time-agnostic architectures. Consequently, time-agnostic MDMs provide a simpler foundation, leading to subsequent developments in inference (Lee et al., 2025) and reinforcement learning (Hong et al., 2026). Accordingly, if time-agnostic modeling is also viable for UDMs, it would provide a simpler and more extensible framework. Unlike MDMs, however, where time-agnostic and time-variant models share the same population optimum, we show that the two optima generally differ for UDMs. Nevertheless, under a finite-data setting, we find that time-agnostic UDMs can still be practically viable. Time-agnostic continuous diffusion models. Recent studies have investigated when continuous diffusion models can dispense with explicit time conditioning (Sahraee-Ardakan et al., 2026; Sun et al., 2025; Helbling et al., 2026). In continuous diffusion, the state evolves in a differentiable continuous space, and the reverse process is described locally by a score ∇logpt() _ z p_t( z) or, equivalently in the ODE view, by a vector field over the continuous state space. The current state can therefore encode where the sample lies along the generative trajectory, allowing even a time-independent vector field to evolve according to its state. This continuous dynamics of particles enables geometric explanations for time-agnosticity. For example, Sahraee-Ardakan et al. (2026) study a time-independent vector field and relate the resulting dynamics to a marginal energy landscape and gradient flow. A complementary explanation is that the corrupted observation itself can reveal the noise level. Sun et al. (2025) relate concentration of T|Z=T Z= z to the gap between time-agnostic and time-conditioned predictors, and further extend their analysis to multiple data points under an i.i.d. Gaussian data model. Similarly, Helbling et al. (2026) construct a time estimator t^() t( z) under a spiked-covariance Gaussian model and show that the error from hiding time becomes small in high dimensions. In contrast, our analysis first reveals the distinct role of time in uniform discrete diffusion: time tells which subsets of observed tokens are plausibly inherited from the clean sequence. A key structural distinction from continuous diffusion lies in the corruption kernel. In continuous diffusion, Gaussian noise continuously perturbs every coordinate, so each corrupted coordinate generally retains a partial contribution from the clean state. In discrete diffusion, by contrast, the state evolves through tokenwise jumps, so only a subset of tokens may change. Consequently, a corrupted sequence contains tokens directly inherited from the clean sequence alongside tokens generated independently by noise, with the latter carrying no information about their original tokens. This distinction enables us to clearly understand the specific role of time in uniform discrete diffusion. Consequently, our analysis reveals a different mechanism through which explicit time conditioning can become unnecessary in uniform discrete diffusion. The key distinction is that the role of time itself can be limited for the prediction of z, rather than time being inferred from z: for a finite training set, sufficiently separated clean sequences concentrate the empirical posterior onto a small set of compatible hypotheses, making the empirical-optimal predictor nearly invariant to time. We quantify this through the post-corruption separation margin and directly bound the oracle time sensitivity. Other related works. Liu et al. (2025) decomposes uniform discrete diffusion into a planner that infers latent corruption identities and a denoiser that predicts clean tokens, highlighting the role of corruption-state uncertainty. Relatedly, Amin et al. (2025) conditions discrete diffusion on the jump schedule and connects uniform diffusion to masking diffusion when corruption-event information is revealed. While these works motivate reasoning about latent corruption states, we instead characterize how such ambiguity induces time dependence in the population-optimal UDM predictor and when this dependence becomes negligible. A.2 Various Uniform Discrete Diffusion Frameworks We briefly review several representative formulations of uniform discrete diffusion. We first describe the general CTMC view and its posterior- (Campbell et al., 2022) and score-based parameterizations (Lou et al., 2024). We then introduce the work of Schiff et al. (2025), which uses a plug-in UDM parameterization improving empirical performance over these earlier posterior- and score-based parameterizations, and forms the basis of the modern UDM family studied in this work. Finally, we review two recent developments built on this formulation, Diffusion Duality (Sahoo et al., 2025) and the leave-one-out parameterization of Gourevitch et al. (2026). Posterior and score parameterizations in continuous-time Markov chains. Discrete diffusion can be formulated as a continuous-time Markov chain (CTMC) (Campbell et al., 2022; Lou et al., 2024). We describe here only the single-token process on =1,…,KX=\ e_1,…, e_K\, where ,∈ x, z∈X are categorical states represented as one-hot vectors. For sequence-level extensions, we refer to the original works for these constructions. With a slight abuse of notation, write qt(∣)q_t( z x) for the forward transition kernel qt|0(∣)q_t|0( z x) and define its marginal by pt()=∑∈pdata()qt(∣). p_t( z)= _ x∈Xp_data( x)q_t( z x). Using the row-vector convention, let t=(pt())∈ℝ1×K, p_t= (p_t( z) )_ z∈X ^1× K, and let t∈ℝK×K Q_t ^K× K denote the forward transition-rate matrix. The forward dynamics satisfy dtdt=tt, d p_tdt= p_t Q_t, where the off-diagonal entries of t Q_t are nonnegative and each row sums to zero. We use this row-vector convention throughout. This is the transpose of the column-vector convention used by Lou et al. (2024), where dt/dt=ttd p_t/dt= Q_t p_t and the columns of t Q_t sum to zero. For uniform diffusion, the transition kernel and its corresponding rate matrix are t|0=αt+(1−αt)K−1KK⊤,t=−αt′αt(K−1KK⊤−). P_t|0= _t I+(1- _t)K^-1 1_K 1_K , Q_t=- _t _t (K^-1 1_K 1_K - I ). Thus, for two distinct states, the instantaneous transition rate is −αt′/(Kαt)- _t /(K _t). Equivalently, one may use the unnormalized uniform generator KK⊤−K 1_K 1_K -K I, as in Lou et al. (2024), with its scale absorbed into the scalar noise schedule. Since the uniform generator is symmetric, the row- and column-vector conventions yield the same numerical generator matrix in this particular case. The time reversal is again a CTMC. Under our row-vector convention, its exact reverse transition rate from z to ′≠ z ≠ z is ¯t(,′)=t(′,)pt(′)pt(). Q_t( z, z )= Q_t( z , z) p_t( z )p_t( z). The diagonal entries are determined by requiring each row of ¯t Q_t to sum to zero. Hence, constructing the reverse process amounts to estimating the marginal ratio pt(′)/pt()p_t( z )/p_t( z). The CTMC denoising formulation of Campbell et al. (2022) obtains this ratio indirectly through the clean posterior. By Bayes’ rule, pt(′)pt()=∑∈qt(′∣)qt(∣)q0|t(∣). p_t( z )p_t( z)= _ x∈X q_t( z x)q_t( z x)q_0|t( x z). Since the clean posterior q0|t(∣)q_0|t( x z) is generally intractable, it is replaced by a denoising model θPosterior(∣,t) μ_θ^Posterior( x z,t), yielding ¯tθ(,′)=t(′,)∑∈qt(′∣)qt(∣)θPosterior(∣,t),′≠. Q_t^θ( z, z )= Q_t( z , z) _ x∈X q_t( z x)q_t( z x) μ_θ^Posterior( x z,t), z ≠ z. Thus, this formulation parameterizes the reverse dynamics through a model of the clean posterior and trains the resulting reverse process using the continuous-time NELBO derived from the forward and reverse CTMCs. SEDD (Lou et al., 2024), in contrast, directly parameterizes the concrete score, i.e., the collection of marginal ratios st()′≔pt(′)pt(),′≠. s_t( z)_ z p_t( z )p_t( z), z ≠ z. Replacing the intractable score by a model θScore μ_θ^Score gives, in our row-vector convention, ¯tθ(,′)=t(′,)θScore(,t)′,′≠. Q_t^θ( z, z )= Q_t( z , z) μ_θ^Score( z,t)_ z , z ≠ z. SEDD trains these marginal ratios directly using diffusion-weighted denoising score entropy. Thus, both approaches target the same reverse CTMC but differ in how it is parameterized: the CTMC denoising formulation models the clean posterior and obtains the marginal ratio indirectly, whereas SEDD models the marginal ratio itself as a discrete score. Plug-in parametrized uniform diffusion models. Recent uniform diffusion models largely follow the plug-in parameterization adopted by Schiff et al. (2025), which yields empirical improvements over the score- and posterior-based parameterizations of the CTMC formulations described above. A UDM independently corrupts each token toward the uniform prior: qt(ti∣i)=Cat(ti,αti+(1−αt)K−1K). q_t( z_t^i x^i)=Cat ( z_t^i; _t x^i+(1- _t)K^-1 1_K ). For s<ts<t, the single token posterior qs|0,tq_s|0,t follows from Bayes’ rule using the s→ts→ t transition and the s-marginal: qs|0,t(si∣i,ti)=Cat(si,ρtti⊙i+ρtρsti+(1−ρtρs)(ρsi+K+ρs)ρt⟨ti,i⟩+1). q_s|0,t( z_s^i x^i, z_t^i)=Cat ( z_s^i; aligned & _t z_t^i x^i+ _t _s z_t^i+(1- _t _s)( _s x^i+ 1_KK+ _s) aligned _t z_t^i, x^i +1 ). (10) To mimic the true posterior, Schiff et al. (2025) design reverse process with plug-in parametrization: ps|tθ(si∣t) p_s|t^θ\! ( z_s^i z_t ) =qs|0,t(si∣i=θi(t,t),t) =q_s|0,t( z_s^i x^i= x_θ^i( z_t,t), z_t) =Cat(si,ρtti⊙θi(t,t)+ρtρsti+(1−ρtρs)ρsθi(t,t)+K+ρsρt⟨ti,θi(t,t)⟩+1). =Cat\! ( z_s^i; _t z_t^i x_θ^i( z_t,t)+ _t _s z_t^i+ (1- _t _s ) _s x_θ^i( z_t,t)+ 1_KK+ _s _t z_t^i, x_θ^i( z_t,t) +1 ). The training of UDM uses the variational upper bound (NELBO), and the discrete-time NELBO is: −logpθ() - p_θ( x) ≤[−logpθ(∣t(0))+∑iKL[qs(i)|0,t(i)(⋅)∥pθ(s(i)∣t(i))]+KL[qt(T)(⋅)∥pθ(t(T))]] [- p_θ( x z_t(0))+ _iD_KL[q_s(i)|0,t(i)(·)\|p_θ( z_s(i) z_t(i))]+D_KL[q_t(T)(·)\|p_θ( z_t(T))] ] The prior and reconstruction loss vanish for the continuous-time, and only the diffusion loss remains: ℒUDM(θ)=∫01,t∑i=1L−αt′Kαt[Kx¯θ,ℓi−Kx¯ℓi+∑j≠ℓix¯jix¯ℓilogx¯θ,ℓix¯jix¯θ,jix¯ℓi]t. _UDM(θ)= _0^1E_ x, z_t _i=1^L - _t K _t [ K x_θ, _i^i- K x_ _i^i+ _j≠ _i x_j^i x_ _i^i x_θ, _i^i x_j^i x_θ,j^i x_ _i^i ]dt. where ¯i≔Kαti+(1−αt)K ν^i K _t ν^i+(1- _t) 1_K and ℓi _i denotes non-zero dimension index of ti z_t^i. Diffusion duality. Diffusion Duality (Sahoo et al., 2025) retains the plug-in parameterization of Schiff et al. (2025), but provides an underlying Gaussian view of the uniform forward process. For each token, consider the Gaussian diffusion q~t(ti∣i;α~t)=(ti,α~ti,(1−α~t2)). q_t( w_t^i x^i; α_t)=N\! ( w_t^i; α_t x^i,(1- α_t^2) I ). Diffusion Duality shows that applying argmax to this Gaussian latent recovers the marginal of uniform discrete diffusion: qt(ti∣i;(α~t))=[argmax]⋆q~t(ti∣i;α~t),αt=(α~t), q_t\! ( z_t^i x^i;T( α_t) )=[ ]_ \, q_t\! ( w_t^i x^i; α_t ), _t=T( α_t), where [argmax]⋆[ ]_ denotes the pushforward under the argmax mapping and T is the diffusion transformation relating the Gaussian and uniform noise schedules. Thus, the uniform diffusion marginal can be viewed as the discrete projection of an underlying Gaussian diffusion. This duality is further used to construct a low-variance curriculum for training the same plug-in UDM. Diffusion Duality first rewrites the discrete NELBO exactly using Gaussian latents and a hard argmax projection. Its curriculum then replaces the hard input with a tempered softmax, introducing a lower-variance but biased relaxation that approaches the original discrete-input objective as the temperature decreases. More specifially, Diffusion Duality relaxes this input using the identity argmax(ti)=limτ→0+softmax(tiτ). ( w_t^i)= _τ→ 0^+softmax\! ( w_t^iτ ). During the curriculum, the denoising model therefore receives softmax(t/τ)softmax( w_t/τ) instead of the hard argmax(t) ( w_t) input. A positive temperature preserves more information from the Gaussian latent, making the denoising problem easier and reducing training variance; as τ is annealed toward zero, the model progressively approaches the original discrete UDM training problem. In particular, in their actual experiments, they use the hard argmax for last half of training, thereby retaining the original UDM’s discrete input and NELBO exactly. The final model retains the same plug-in parameterization and is evaluated as a standard uniform diffusion model. Leave-one-out parameterization. Gourevitch et al. (2026) characterize the target implicit in the UDM bridge plug-in parameterization. For each position i∈[L]i∈[L], the population-optimal network prediction is the leave-one-out (LOO) posterior ⋆i(,t)=[Xi|Z−i=−i,T=t], x_ ^i( z,t)=E [X^i\, |\,Z^-i= z^-i,T=t ], rather than the full denoising posterior conditioned on the local observation Zi=iZ^i= z^i. In particular, they show that the UDM NELBO is minimized when the plug-in predictor satisfies θ⋆i(,t)=⋆i(,t) x_θ ^i( z,t)= x_ ^i( z,t) for PT,ZP_T,Z-almost every (t,)(t, z). The local observation i z^i can then be incorporated analytically using the known uniform corruption likelihood. By Bayes’ rule, [Xi|Z=,T=t]=⋆i(,t)⊙(αti+1−αtKK)⟨⋆i(,t),αti+1−αtKK⟩. [X^i\, |\,Z= z,T=t ]= x_ ^i( z,t) ( _t z^i+ 1- _tK 1_K ) x_ ^i( z,t), _t z^i+ 1- _tK 1_K . Thus, the LOO posterior summarizes the information contained in the remaining corrupted context Z−iZ^-i, while the observation at the target position is inserted exactly through the known forward corruption process. Replacing the population-optimal LOO posterior ⋆i x_ ^i by the model prediction θi x_θ^i gives the corrected posterior denoiser θPosterior,i μ_θ^Posterior,i. Equivalently, using ρt=Kαt/(1−αt) _t=K _t/(1- _t), this correction can be written in logit form as θPosterior,i(,t) μ_θ^Posterior,i( z,t) =softmax(logθi(,t)+log(αti+1−αtKK)) =softmax ( x_θ^i( z,t)+ ( _t z^i+ 1- _tK 1_K ) ) =softmax(logθi(,t)+log(1+ρt)i). =softmax ( x_θ^i( z,t)+ (1+ _t) z^i ). (11) The second equality follows because the uniform-channel likelihood differs between the observed token and all other tokens only by the multiplicative factor 1+ρt1+ _t, while any common additive logit term vanishes under softmaxsoftmax. This corrected posterior can be trained using the usual denoising cross-entropy, ℒLOO+CE(θ)=−∫01,t∑i=1Llog⟨i,θPosterior,i(t,t)⟩dt. _LOO+CE(θ)=- _0^1E_ x, z_t _i=1^L x^i, μ_θ^Posterior,i( z_t,t) . At the unrestricted population optimum, the corrected posterior satisfies θ⋆Posterior,i(,t)=[Xi∣Z=,T=t] μ_θ ^Posterior,i( z,t)=E[X^i Z= z,T=t], and, crucially, the underlying predictor satisfies θ⋆i(,t)=⋆i(,t)=[Xi|Z−i=−i,T=t]. x_θ ^i( z,t)= x_ ^i( z,t)=E [X^i\, |\,Z^-i= z^-i,T=t ]. Thus, the corrected cross-entropy objective provides a convenient way to train the same LOO predictor required by the UDM plug-in parameterization. Once trained, θ x_θ is plugged into the original UDM bridge ps|tθp_s|t^θ to define the reverse transitions used for generation. Appendix B Omitted Proofs In this section, we provide omitted proofs in the main paper. Throughout the proofs, we consider 0<t<10<t<1 with 0<αt<10< _t<1. Conditional expectations on events of zero data probability are assigned arbitrary, time-independent probability vectors, since their corresponding mixture weights are zero. B.1 Proof of Proposition 1 See 1 Proof. Gourevitch et al. (2026) showed that the unrestricted population minimizer of the bridge plug-in UDM NELBO is the leave-one-out posterior. Therefore, θ⋆i(,t) x_θ ^i( z,t) =⋆i(,t)=[Xi|Z−i=−i,T=t]. = x_ ^i( z,t)=E [X^i\, |\,Z^-i= z^-i,T=t ]. By Bayes’ rule and marginalization, ⋆i(,t) x_ ^i( z,t) =pdata[Xiqt(−i∣X−i)]pdata[qt(−i∣X−i)]. = E_p_data [X^iq_t\! ( z^-i X^-i ) ]E_p_data [q_t\! ( z^-i X^-i ) ]. (12) Since the uniform forward process factorizes across positions, qt(−i∣X−i) q_t\! ( z^-i X^-i ) =∏j∈[L]∖i(αtXj=j+1−αtK) = _j∈[L] \i\ ( _t 1\! \X^j= z^j \+ 1- _tK ) =(1−αtK)L−1∏j∈[L]∖i(1+ρtXj=j) = ( 1- _tK )^L-1 _j∈[L] \i\ (1+ _t 1\! \X^j= z^j \ ) =(1−αtK)L−1∑S⊆[L]∖iρt|S|X[S]=[S], = ( 1- _tK )^L-1 _S [L] \i\ _t^|S| 1\! \X[S]= z[S] \, (13) where the last equality follows by expanding the product over all subsets of [L]∖i[L] \i\. Substituting Eq. (13) into Eq. (12) and canceling the common factor ((1−αt)/K)L−1 ((1- _t)/K )^L-1 gives ⋆i(,t) x_ ^i( z,t) =∑S⊆[L]∖iρt|S|pdata[XiX[S]=[S]]∑S⊆[L]∖iρt|S|Pr(X[S]=[S]). = _S [L] \i\ _t^|S|E_p_data [X^i 1\! \X[S]= z[S] \ ] _S [L] \i\ _t^|S| (X[S]= z[S] ). Using pdata[XiX[S]=[S]]=Pr(X[S]=[S])[Xi|X[S]=[S]], _p_data [X^i 1\! \X[S]= z[S] \ ]= (X[S]= z[S] )E [X^i\, |\,X[S]= z[S] ], we obtain ⋆i(,t) x_ ^i( z,t) =∑S⊆[L]∖iρt|S|Pr(X[S]=[S])∑S′⊆[L]∖iρt|S′|Pr(X[S′]=[S′])[Xi|X[S]=[S]]. = _S [L] \i\ _t^|S| (X[S]= z[S] ) _S [L] \i\ _t^|S | (X[S ]= z[S ] )E [X^i\, |\,X[S]= z[S] ]. (14) It remains to identify the normalized coefficient in Eq. (14). Interpret S as the set of context positions generated through the copy branch of the uniform forward channel. For a fixed S, the joint probability of this copy-set hypothesis and the observation Z−i=−iZ^-i= z^-i is Pr(S,Z−i=−i|T=t) (S,\,Z^-i= z^-i\, |\,T=t ) =αt|S|(1−αtK)L−1−|S|Pr(X[S]=[S]) = _t^|S| ( 1- _tK )^L-1-|S| (X[S]= z[S] ) =(1−αtK)L−1ρt|S|Pr(X[S]=[S]). = ( 1- _tK )^L-1 _t^|S| (X[S]= z[S] ). Therefore, normalizing over all S⊆[L]∖iS [L] \i\ gives Pr(S|Z−i=−i,T=t)=ρt|S|Pr(X[S]=[S])∑S′⊆[L]∖iρt|S′|Pr(X[S′]=[S′]). (S\, |\,Z^-i= z^-i,T=t )= _t^|S| (X[S]= z[S] ) _S [L] \i\ _t^|S | (X[S ]= z[S ] ). (15) Hence, Pr(S|Z−i=−i,T=t)∝ρt|S|Pr(X[S]=[S]), (S\, |\,Z^-i= z^-i,T=t ) _t^|S| (X[S]= z[S] ), and substituting Eq. (15) into Eq. (14) yields θ⋆i(,t) x_θ ^i( z,t) =∑S⊆[L]∖iPr(S|Z−i=−i,T=t)[Xi|X[S]=[S]], = _S [L] \i\ (S\, |\,Z^-i= z^-i,T=t )E [X^i\, |\,X[S]= z[S] ], which proves the claim. ∎ B.2 Proof of Theorem 1 See 1 Proof. By the law of total expectation, ⋆i(,t)=∑m=0L−1Pr(Mi=m|Z−i=−i,T=t)×[Xi|Z−i=−i,Mi=m,T=t]. x_ ^i( z,t)= _m=0^L-1 (M^i=m\, |\,Z^-i= z^-i,T=t )×E [X^i\, |\,Z^-i= z^-i,M^i=m,T=t ]. (16) Therefore, it remains to show that Pr(Mi=m|Z−i=−i,T=t) (M^i=m\, |\,Z^-i= z^-i,T=t ) ∝(1+ρt)mPr(Mi(X,)=m), (1+ _t)^m (M^i(X, z)=m), and [Xi|Z−i=−i,Mi=m,T=t]=[Xi|Z−i=−i,Mi=m]. [X^i\, |\,Z^-i= z^-i,M^i=m,T=t ]=E [X^i\, |\,Z^-i= z^-i,M^i=m ]. Posterior weighting. For any clean sequence ′ x , the uniform forward channel gives qt(−i|′−i)=∏j∈[L]∖i[αt′j=j+1−αtK]=(1−αtK)L−1(1+ρt)Mi(′,). q_t ( z^-i\, |\, x -i )= _j∈[L] \i\ [ _t 1 \ x j= z^j \+ 1- _tK ]= ( 1- _tK )^L-1(1+ _t)^M^i( x , z). (17) Hence, Pr(Mi=m,Z−i=−i|T=t) (M^i=m,Z^-i= z^-i\, |\,T=t ) =∑′∈LMi(′,)=mpdata(′)qt(−i|′−i) = _ subarrayc x ∈X^L\\ M^i( x , z)=m subarrayp_data( x )q_t ( z^-i\, |\, x -i ) =(1−αtK)L−1(1+ρt)m∑′∈LMi(′,)=mpdata(′) = ( 1- _tK )^L-1(1+ _t)^m _ subarrayc x ∈X^L\\ M^i( x , z)=m subarrayp_data( x ) =(1−αtK)L−1Pr(Mi(X,)=m)(1+ρt)m. = ( 1- _tK )^L-1 (M^i(X, z)=m)(1+ _t)^m. Therefore, Pr(Mi=m|Z−i=−i,T=t) (M^i=m\, |\,Z^-i= z^-i,T=t ) =Pr(Mi=m,Z−i=−i|T=t)∑r=0L−1Pr(Mi=r,Z−i=−i|T=t) = (M^i=m,Z^-i= z^-i\, |\,T=t ) _r=0^L-1 (M^i=r,Z^-i= z^-i\, |\,T=t ) =Pr(Mi(X,)=m)(1+ρt)m∑r=0L−1Pr(Mi(X,)=r)(1+ρt)r. = (M^i(X, z)=m)(1+ _t)^m _r=0^L-1 (M^i(X, z)=r)(1+ _t)^r. and hence Pr(Mi=m|Z−i=−i,T=t)∝Pr(Mi(X,)=m)(1+ρt)m. (M^i=m\, |\,Z^-i= z^-i,T=t ) (M^i(X, z)=m)(1+ _t)^m. Time-independence of the conditional prediction. For any ′ x satisfying Mi(′,)=mM^i( x , z)=m, Bayes’ rule and Eq. (17) give Pr(X=′|Z−i=−i,Mi=m,T=t) (X= x \, |\,Z^-i= z^-i,M^i=m,T=t ) =pdata(′)qt(−i|′−i)∑′∈LMi(′,)=mpdata(′)qt(−i|′−i) = p_data( x )q_t ( z^-i\, |\, x -i ) _ subarrayc x ∈X^L\\ M^i( x , z)=m subarrayp_data( x )q_t ( z^-i\, |\, x -i ) =pdata(′)∑′∈LMi(′,)=mpdata(′)=pdata(′)Pr(Mi(X,)=m), = p_data( x ) _ subarrayc x ∈X^L\\ M^i( x , z)=m subarrayp_data( x )= p_data( x ) (M^i(X, z)=m), where the second equality follows because all sequences satisfying Mi(′,)=mM^i( x , z)=m have the same uniform-channel likelihood, so the common factor cancels under normalization. The right-hand side is independent of t. Therefore, Xi⟂T|Z−i,Mi, X^i T Z^-i,M^i, and consequently [Xi|Z−i=−i,Mi=m,T=t]=[Xi|Z−i=−i,Mi=m]=mi(). [X^i\, |\,Z^-i= z^-i,M^i=m,T=t ]=E [X^i\, |\,Z^-i= z^-i,M^i=m ]=h_m^i( z). Substituting the two results into Eq. (16) gives ⋆i(,t)=∑m=0L−1wmi(,t)mi(), x_ ^i( z,t)= _m=0^L-1w_m^i( z,t)h_m^i( z), with wmi(,t)∝Pr(Mi(X,)=m)(1+ρt)m, w_m^i( z,t) (M^i(X, z)=m)(1+ _t)^m, which proves the claim. ∎ B.3 Proof of Proposition 2 See 2 Proof. By Theorem 1, ⋆i(,t)=∑m=0L−1wmi(,t)mi(), x_ ^i( z,t)= _m=0^L-1w_m^i( z,t)h_m^i( z), where, writing ηt=log(1+ρt) _t= (1+ _t), wmi(,t)=Pr(Mi(X,)=m)emηt∑r=0L−1Pr(Mi(X,)=r)erηt. w_m^i( z,t)= (M^i(X, z)=m)e^m _t _r=0^L-1 (M^i(X, z)=r)e^r _t. (18) Since mi()h_m^i( z) is independent of t, all dependence on ηt _t is contained in wmi(,t)w_m^i( z,t). Differentiating Eq. (18) gives ∂ηtwmi(,t) ∂ _tw_m^i( z,t) =wmi(,t)(m−∑r=0L−1rwri(,t)) =w_m^i( z,t) (m- _r=0^L-1r\,w_r^i( z,t) ) =wmi(,t)(m−t,[Mi]). =w_m^i( z,t) (m-E_t, z[M^i] ). Therefore, ∂ηt⋆i(,t) ∂ _t x_ ^i( z,t) =∑m=0L−1mi()wmi(,t)(m−t,[Mi]) = _m=0^L-1h_m^i( z)w_m^i( z,t) (m-E_t, z[M^i] ) =t,[Mii()Mi]−t,[Mii()]t,[Mi] =E_t, z [h_M^i^i( z)M^i ]-E_t, z [h_M^i^i( z) ]E_t, z[M^i] =Covt,(Mii(),Mi), =Cov_t, z (h_M^i^i( z),M^i ), which proves the claim. Now fix a candidate token k e_k such that ⋆,ki(,t)>0 x_ ,k^i( z,t)>0. Taking the k-th component of the previous expression, ∂ηtlog⋆,ki(,t) ∂ _t x_ ,k^i( z,t) =∑m=0L−1wmi(,t)hm,ki()m∑m=0L−1wmi(,t)hm,ki()−t,[Mi], = _m=0^L-1w_m^i( z,t)h_m,k^i( z)m _m=0^L-1w_m^i( z,t)h_m,k^i( z)-E_t, z[M^i], (19) where hm,ki()h_m,k^i( z) denotes the k-th component of mi()h_m^i( z). By Theorem 1, Xi⟂T|Z−i,MiX^i T Z^-i,M^i, and hence we can include t into the distribution: hm,ki() h_m,k^i( z) =Pr(Xi=k|Mi=m,Z−i=−i,T=t). = (X^i= e_k\, |\,M^i=m,Z^-i= z^-i,T=t ). Together with wmi(,t)=Pr(Mi=m|Z−i=−i,T=t), w_m^i( z,t)= (M^i=m\, |\,Z^-i= z^-i,T=t ), Bayes’ rule gives wmi(,t)hm,ki()⋆,ki(,t)=Pr(Mi=m|Xi=k,Z−i=−i,T=t). w_m^i( z,t)h_m,k^i( z) x_ ,k^i( z,t)= (M^i=m\, |\,X^i= e_k,Z^-i= z^-i,T=t ). Consequently, the first term in Eq. (19) is precisely t,[Mi|Xi=k], _t, z [M^i\, |\,X^i= e_k ], and therefore ∂ηtlog⋆,ki(,t)=t,[Mi|Xi=k]−t,[Mi]. ∂ _t x_ ,k^i( z,t)=E_t, z [M^i\, |\,X^i= e_k ]-E_t, z[M^i]. This proves Eq. (6). ∎ B.4 Proof of Proposition 3 We here provide a formal version of Proposition 3 in the main paper and the proof under the empirical data distribution setting. Assumption 1 (Finite-data setting). The data distribution is the uniform empirical distribution00 0 Note that every theoretical result in our paper holds within Assumption 1; see Appendix B.7 for why. =1N∑n=1Nδn = 1N _n=1^N _ x_n over N sequences 1,…,N∈L x_1,…, x_N∈X^L where ||=K|X|=K. Proposition 3 (Δ -separable event under UDM training). Under Assumption 1, fix ∈supp() x ( ) and i∈[L]i∈[L], and suppose d()≥2d( x)≥ 2. Let T∼Unif([0,1])T ([0,1]) and, conditioned on T=tT=t, let Z∼qt(⋅∣)Z q_t(· x) under the linear schedule αt=1−t _t=1-t. Then, for any integer Δ≥1 ≥ 1, Pr(Z is Δ-separable w.r.t. at i)≥max0, 1−log(N−1)+Δ+(e+e−1−2)L−1K(1−e−1)(d()−1). \! (Z is $ $-separable w.r.t.\ $ x$ at $i$ )≥ \0,\,1- (N-1)+ +(e+e^-1-2) L-1K(1-e^-1)(d( x)-1) \. (20) Thus, as given in the informal statement of the main paper, if K≫LK L and d()d( x) is sufficiently large, the approximate bound Pr(Z is Δ-separable w.r.t. at i)≳1−logN+Δ(1−e−1)d() \! (Z is $ $-separable w.r.t. $ x$ at $i$ ) 1- N+ (1-e^-1)d( x) holds. Note that we used Eq. (20) to measure quantitative bound in Section 4.2. We now provide the proof of formal statement: Proof. Fix any ′∈supp() x ( ) satisfying ′i≠i x i≠ x^i, and let ℋi(,′)≔j∈[L]∖i:′j≠j.H_i( x, x ) \j∈[L] \i\: x j≠ x^j \. Since ′i≠i x i≠ x^i and the Hamming distance between x and ′ x is at least d()d( x), |ℋi(,′)| |H_i( x, x ) | =∑j∈[L]∖i′j≠j≥d()−1. = _j∈[L] \i\ 1 \ x j≠ x^j \≥ d( x)-1. (21) Condition on T=tT=t. For each j∈ℋi(,′)j _i( x, x ), define Yj≔Zj=j−Zj=′j.Y_j 1 \Z^j= x^j \- 1 \Z^j= x j \. For j∉ℋi(,′)j _i( x, x ), x and ′ x contribute equally to the match count. Therefore, Mi(,Z)−Mi(′,Z)=∑j∈ℋi(,′)Yj. M^i( x,Z)-M^i( x ,Z)= _j _i( x, x )Y_j. (22) Under the linear schedule αt=1−t _t=1-t, the uniform forward kernel gives, for every j∈ℋi(,′)j _i( x, x ), Pr(Yj=1∣T=t) (Y_j=1 T=t) =1−t+tK, =1-t+ tK, Pr(Yj=0∣T=t) (Y_j=0 T=t) =t(K−2)K, = t(K-2)K, Pr(Yj=−1∣T=t) (Y_j=-1 T=t) =tK. = tK. Moreover, the variables Yjj∈ℋi(,′)\Y_j\_j _i( x, x ) are independent conditioned on T=tT=t. Hence, [e−Yj∣T=t] [e^-Y_j T=t ] =(1−t+tK)e−1+t(K−2)K+tKe = (1-t+ tK )e^-1+ t(K-2)K+ tKe =1−(1−e−1)(1−t)+(e+e−1−2)tK =1-(1-e^-1)(1-t)+(e+e^-1-2) tK ≤exp(−(1−e−1)(1−t)+(e+e−1−2)tK), ≤ (-(1-e^-1)(1-t)+(e+e^-1-2) tK ), (23) where the last inequality comes from 1+u≤eu1+u≤ e^u. The sequence Z fails to be Δ -separable from this particular ′ x whenever Mi(′,Z)>Mi(,Z)−Δ.M^i( x ,Z)>M^i( x,Z)- . Since the match counts are integer-valued, this is equivalent to Mi(,Z)−Mi(′,Z)≤Δ−1.M^i( x,Z)-M^i( x ,Z)≤ -1. Therefore, by Markov’s inequality and Eq. (22), Pr(Mi(,Z)−Mi(′,Z)≤Δ−1|T=t) (M^i( x,Z)-M^i( x ,Z)≤ -1\, |\,T=t ) =Pr(exp(−(Mi(,Z)−Mi(′,Z)))≥e−(Δ−1)|T=t) = ( \! (- (M^i( x,Z)-M^i( x ,Z) ) )≥ e^-( -1)\, |\,T=t ) ≤eΔ−1[exp(−(Mi(,Z)−Mi(′,Z)))|T=t] ≤ e -1E [ \! (- (M^i( x,Z)-M^i( x ,Z) ) )\, |\,T=t ] =eΔ−1[exp(−∑j∈ℋi(,′)Yj)|T=t] =e -1E [ (- _j _i( x, x )Y_j )\, |\,T=t ] =eΔ−1∏j∈ℋi(,′)[e−Yj∣T=t], =e -1 _j _i( x, x )E [e^-Y_j T=t ], Plugging Eq. (23) into the above inequality gives: Pr(Mi(,Z)−Mi(′,Z)≤Δ−1|T=t) (M^i( x,Z)-M^i( x ,Z)≤ -1\, |\,T=t ) ≤eΔ−1∏j∈ℋi(,′)exp(−(1−e−1)(1−t)+(e+e−1−2)tK) ≤ e -1 _j _i( x, x ) (-(1-e^-1)(1-t)+(e+e^-1-2) tK ) =exp(Δ−1−(1−e−1)(1−t)|ℋi(,′)|+(e+e−1−2)t|ℋi(,′)|K) = ( -1-(1-e^-1)(1-t) |H_i( x, x ) |+(e+e^-1-2) t |H_i( x, x ) |K ) ≤exp(Δ−1−(1−e−1)(1−t)(d()−1)+(e+e−1−2)L−1K). ≤ ( -1-(1-e^-1)(1-t)(d( x)-1)+(e+e^-1-2) L-1K ). where the last inequality uses Eq. (21), t≤1t≤ 1, and |ℋi(,′)|≤L−1 |H_i( x, x ) |≤ L-1. There are at most N−1N-1 empirical sequences distinct from x. Thus, by a union bound over all ′∈supp() x ( ) satisfying ′i≠i x i≠ x^i, Pr(Z is not Δ-separable w.r.t. at i|T=t) (Z is not $ $-separable w.r.t.\ $ x$ at $i$\, |\,T=t ) ≤min1,(N−1)exp(Δ−1−(1−e−1)(1−t)(d()−1)+(e+e−1−2)L−1K). ≤ \1,\,(N-1) ( -1-(1-e^-1)(1-t)(d( x)-1)+(e+e^-1-2) L-1K ) \. (24) It remains to average over the training time T∼Unif([0,1])T ([0,1]). For any A≥1A≥ 1 and B>0B>0, ∫01min1,Ae−B(1−t)t≤min1,logA+1B. _0^1 \1,\,Ae^-B(1-t) \dt≤ \1,\, A+1B \. (25) Indeed, if logA≥B A≥ B, the bound follows immediately since the integral is at most one. Otherwise, splitting the integral at t=1−logA/Bt=1- A/B gives ∫01min1,Ae−B(1−t)t=logAB+1−Ae−B≤logA+1B. _0^1 \1,\,Ae^-B(1-t) \dt= AB+ 1-Ae^-BB≤ A+1B. Applying Eq. (25) to Eq. (24) with A=(N−1)exp(Δ−1+(e+e−1−2)L−1K)A=(N-1) ( -1+(e+e^-1-2) L-1K ) and B=(1−e−1)(d()−1),B=(1-e^-1)(d( x)-1), we obtain Pr(Z is not Δ-separable w.r.t. at i)≤min1,log(N−1)+Δ+(e+e−1−2)L−1K(1−e−1)(d()−1). (Z is not $ $-separable w.r.t.\ $ x$ at $i$ )≤ \1,\, (N-1)+ +(e+e^-1-2) L-1K(1-e^-1)(d( x)-1) \. Taking the complement yields Pr(Z is Δ-separable w.r.t. at i)≥max0, 1−log(N−1)+Δ+(e+e−1−2)L−1K(1−e−1)(d()−1), (Z is $ $-separable w.r.t.\ $ x$ at $i$ )≥ \0,\,1- (N-1)+ +(e+e^-1-2) L-1K(1-e^-1)(d( x)-1) \, which proves the claim. ∎ B.5 Proof of Theorem 2 We here provide a formal version of Theorem 2 in the main paper under the empirical data distribution setting. We first define the Fisher sensitivity with respect to the noise-schedule parameter αt _t: ℐαti()≔∑k:⋆,ki(,t)>0⋆,ki(,t)(∂αtlog⋆,ki(,t))2.I_ _t^i( z) _k:\, x_ ,k^i( z,t)>0 x_ ,k^i( z,t) ( ∂ _t x_ ,k^i( z,t) )^2. Theorem 2 (Fisher sensitivity with finite data). Suppose that αt _t is strictly decreasing in t. Under the finite data assumption 1, for Δ -separable z with respect to x at coordinate i, ℐαti()≤(N−1)(L−1)2K2(1−αt)Δ−2(1+(K−1)αt)Δ+2.I_ _t^i( z)≤(N-1)(L-1)^2K^2 (1- _t) -2 (1+(K-1) _t ) +2. Moreover, for Δ≥2 ≥ 2, the right-hand side is strictly increasing in t∈(0,1)t∈(0,1). The bound admits the same interpretation as its time-based counterpart. What matters for denoising is how much the oracle prediction changes per unit change in the corruption level, which is measured directly by ℐαti()I_ _t^i( z). The theorem shows that as the corruption level increases (i.e., as αt _t decreases), the upper bound on this sensitivity becomes progressively weaker. Thus, time dependence is most weakly controlled in the high-noise regime. For ease of interpretation, the main text specializes this result to the linear schedule αt=1−t _t=1-t and expresses the Fisher sensitivity with respect to t. Under this schedule, the theorem reduces to ℐti()≤(N−1)(L−1)2K2tΔ−2(K−(K−1)t)Δ+2.I_t^i( z)≤(N-1)(L-1)^2 K^2t -2 (K-(K-1)t ) +2. Using (N−1)(L−1)2<NL2(N-1)(L-1)^2<NL^2 gives the simpler bound stated in the main paper: ℐti()≤NL2K2tΔ−2(K−(K−1)t)Δ+2.I_t^i( z)≤ NL^2 K^2t -2 (K-(K-1)t ) +2. We now prove the formal version of Theorem 2. Proof. By Proposition 2, with ηt=log(1+ρt) _t= (1+ _t), for every k such that ⋆,ki(,t)>0 x_ ,k^i( z,t)>0, ∂αtlog⋆,ki(,t) ∂ _t x_ ,k^i( z,t) =dηtdαt(t,[Mi|Xi=k]−t,[Mi]). = d _td _t (E_t, z [M^i\, |\,X^i= e_k ]-E_t, z[M^i] ). Therefore, by the definition of ℐαti()I_ _t^i( z), ℐαti() _ _t^i( z) =(dηtdαt)2∑k:⋆,ki(,t)>0⋆,ki(,t)(t,[Mi|Xi=k]−t,[Mi])2 = ( d _td _t )^2 _k:\, x_ ,k^i( z,t)>0 x_ ,k^i( z,t) (E_t, z [M^i\, |\,X^i= e_k ]-E_t, z[M^i] )^2 =(dηtdαt)2t,[(t,[Mi|Xi]−t,[Mi])2] = ( d _td _t )^2E_t, z [ (E_t, z [M^i\, |\,X^i ]-E_t, z[M^i] )^2 ] =(dηtdαt)2Vart,(t,[Mi|Xi]). = ( d _td _t )^2Var_t, z (E_t, z [M^i\, |\,X^i ] ). (26) For any choice of center, the variance is upper bounded by the corresponding mean squared deviation. We therefore choose the conditional mean associated with the source token i x^i, t,[Mi∣Xi=i]E_t, z[M^i X^i= x^i], where x is the clean sequence from which z was sampled. This choice is convenient because the deviation vanishes whenever Xi=iX^i= x^i. Moreover, since 0≤Mi≤L−10≤ M^i≤ L-1, all conditional expectations below lie in [0,L−1][0,L-1]. Hence, Vart,(t,[Mi|Xi]) _t, z (E_t, z [M^i\, |\,X^i ] ) ≤t,[(t,[Mi|Xi]−t,[Mi|Xi=i])2] _t, z [ (E_t, z [M^i\, |\,X^i ]-E_t, z [M^i\, |\,X^i= x^i ] )^2 ] ≤(L−1)2Pr(Xi≠i|Z−i=−i,T=t). ≤(L-1)^2 (X^i≠ x^i\, |\,Z^-i= z^-i,T=t ). (27) It remains to bound the posterior probability of a different target token. Under Assumption 1, =1N∑n=1Nδn. = 1N _n=1^N _ x_n. From Eq. (17), qt(−i|n−i)=(1−αtK)L−1(1+ρt)Mi(n,). q_t ( z^-i\, |\, x_n^-i )= ( 1- _tK )^L-1(1+ _t)^M^i( x_n, z). Hence, since the empirical prior is uniform, Pr(Xi≠i|Z−i=−i,T=t)=∑n:ni≠i(1+ρt)Mi(n,)∑r=1N(1+ρt)Mi(r,). (X^i≠ x^i\, |\,Z^-i= z^-i,T=t )= _ subarraycn:\, x_n^i≠ x^i subarray(1+ _t)^M^i( x_n, z) _r=1^N(1+ _t)^M^i( x_r, z). (28) By Δ -separability, every sequence satisfying ni≠i x_n^i≠ x^i obeys Mi(n,)≤Mi(,)−Δ. M^i( x_n, z)≤ M^i( x, z)- . Moreover, the denominator of Eq. (28) contains the contribution of the sampled sequence x. Thus, Pr(Xi≠i|Z−i=−i,T=t) (X^i≠ x^i\, |\,Z^-i= z^-i,T=t ) ≤(N−1)(1+ρt)Mi(,)−Δ(1+ρt)Mi(,) ≤ (N-1)(1+ _t)^M^i( x, z)- (1+ _t)^M^i( x, z) =(N−1)(1+ρt)−Δ. =(N-1)(1+ _t)^- . (29) Combining Eq. (26), Eq. (27), and Eq. (29) yields ℐαti() _ _t^i( z) ≤(N−1)(L−1)2(dαtlog(1+ρt))2(1+ρt)−Δ. ≤(N-1)(L-1)^2 ( dd _t (1+ _t) )^2(1+ _t)^- . (30) Using ρt=Kαt1−αt, _t= K _t1- _t, we have 1+ρt 1+ _t =1+(K−1)αt1−αt, = 1+(K-1) _t1- _t, (31) dαtlog(1+ρt) dd _t (1+ _t) =K(1−αt)(1+(K−1)αt). = K(1- _t) (1+(K-1) _t ). (32) Substituting Eq. (31) and Eq. (32) into Eq. (30) gives ℐαti() _ _t^i( z) ≤(N−1)(L−1)2K2(1−αt)Δ−2(1+(K−1)αt)Δ+2. ≤(N-1)(L-1)^2K^2 (1- _t) -2 (1+(K-1) _t ) +2. (33) It remains to establish the monotonicity of the bound. Define g(α)≔(1−α)Δ−2(1+(K−1)α)Δ+2.g(α) (1-α) -2 (1+(K-1)α ) +2. For Δ≥2 ≥ 2 and α∈(0,1)α∈(0,1), dαlogg(α) ddα g(α) =−Δ−21−α−(Δ+2)(K−1)1+(K−1)α<0. =- -21-α- ( +2)(K-1)1+(K-1)α<0. Hence, g(α)g(α) is strictly decreasing in α. Since αt _t is strictly decreasing in t, g(αt)g( _t) is strictly increasing in t. Therefore, the right-hand side of Eq. (33) is strictly increasing in t∈(0,1)t∈(0,1) for Δ≥2 ≥ 2, which proves the claim. ∎ B.6 Proof of Eq. (9) Proof. By Theorem 2, ℐαti() _ _t^i( z) ≤(N−1)(L−1)2K2(1−αt)Δ−2(1+(K−1)αt)Δ+2. ≤(N-1)(L-1)^2K^2 (1- _t) -2 (1+(K-1) _t ) +2. For Δ≥2 ≥ 2, we have (1−αt)Δ−2≤1(1- _t) -2≤ 1 for every αt∈(0,1) _t∈(0,1). Moreover, suppose αt≥K−β. _t≥ K^-β. Then 1+(K−1)αt 1+(K-1) _t ≥1+(K−1)K−β ≥ 1+(K-1)K^-β =K1−β+1−K−β =K^1-β+1-K^-β ≥K1−β. ≥ K^1-β. Consequently, ℐαti() _ _t^i( z) ≤(N−1)(L−1)2K2−(1−β)(Δ+2) ≤(N-1)(L-1)^2K^2-(1-β)( +2) =(N−1)(L−1)2K−Δ+β(Δ+2). =(N-1)(L-1)^2K^- +β( +2). Taking the supremum over all t satisfying αt≥K−β _t≥ K^-β gives supt:αt≥K−βℐαti() _t:\, _t≥ K^-βI_ _t^i( z) ≤(N−1)(L−1)2K−Δ+β(Δ+2). ≤(N-1)(L-1)^2K^- +β( +2). The exponent is negative whenever 0<β<Δ+2.0<β< +2. Thus, for any 0<β<Δ+20<β< +2 and Δ≥2 ≥ 2, the Fisher sensitivity is polynomially suppressed throughout the region αt≥K−β _t≥ K^-β. Choosing β=1/4β=1/4, which is valid for every Δ≥2 ≥ 2, gives supt:αt≥K−1/4ℐαti() _t:\, _t≥ K^-1/4I_ _t^i( z) ≤(N−1)(L−1)2K−(3Δ−2)/4. ≤(N-1)(L-1)^2K^-(3 -2)/4. This provides a useful intuition for a general schedule αt _t. If z is Δ -separable with Δ≥2 ≥ 2, its Fisher sensitivity is suppressed by a negative power of K whenever the expected clean-token fraction exceeds K−1/4K^-1/4. Thus, irrespective of the particular scheduler, a Δ -separable z has low empirical-optimal Fisher sensitivity with respect to the noise coordinate throughout this regime. Finally, under the linear schedule αt=1−t _t=1-t, the condition αt≥K−β _t≥ K^-β is equivalent to 0<t≤1−K−β,0<t≤ 1-K^-β, and ℐαti()=ℐti()I_ _t^i( z)=I_t^i( z). Therefore, sup0<t≤1−K−1/4ℐti() _0<t≤ 1-K^-1/4I_t^i( z) ≤NL2K−(3Δ−2)/4, ≤ NL^2K^-(3 -2)/4, where we intentionally upper-bound (N−1)(L−1)2<NL2(N-1)(L-1)^2<NL^2 to give a simpler and more intuitive bound in the main paper. ∎ B.7 Discussion on the Finite Data Setting In Section 4, we specialize the data distribution to the finite empirical distribution under Assumption 1. Here, we verify that this finite-data formulation is fully consistent with the theoretical results in previous works and in Section 3. For example, deriving the continuous-time UDM NELBO from its discrete-time counterpart could in principle rely on assumptions on the data distribution, such as full support; here, we explicitly verify that the finite empirical distribution does not invalidate these results. Specifically, we show that (i) the UDM NELBO remains a valid variational objective under , (i) its unrestricted optimum remains the uniform-channel leave-one-out posterior characterized in Proposition 1, and consequently, (i) Theorem 1 and Proposition 2 continue to hold under the finite empirical distribution. Validity of the UDM NELBO under finite data. We first verify that the UDM NELBO remains a valid variational objective under the finite empirical distribution in Assumption 1. The derivation of the UDM NELBO (Schiff et al., 2025) is pointwise in the clean sequence x and does not require any particular form of the outer data distribution. In particular, for a finite discretization with T diffusion steps, define t(i)=i+1T+1t(i)= i+1T+1, s(i)=iT+1s(i)= iT+1, for i=0,…,Ti=0,…,T. Then, for every ∈L x∈X^L, −logpθ()≤q[−logpθ(∣t(0)) - p_θ( x)≤\;E_q [- p_θ\! ( x z_t(0) ) +∑i=1TKL[q(s(i)∣t(i),)∥pθ(s(i)∣t(i))]] + _i=1^TD_KL\! [q\! ( z_s(i) z_t(i), x )\, \|\,p_θ\! ( z_s(i) z_t(i) ) ] ] +KL[q(t(T)∣)∥pθ(t(T))]. +D_KL\! [q\! ( z_t(T) x )\, \|\,p_θ\! ( z_t(T) ) ]. Crucially, this bound holds pointwise for every clean sequence ∈L x∈X^L. Its derivation does not depend on the support, cardinality, or any other property of the data distribution from which x is drawn. Therefore, the discrete-time NELBO remains valid when the data distribution is replaced by the finite empirical distribution in Assumption 1. We now analyze whether the continuous-time limit of the discrete-time NELBO derived by Schiff et al. (2025) holds in finite-data setting. Their derivation takes T→∞T→∞ for each fixed clean sequence x. In particular, the prior loss vanishes because αt(T)=0 _t(T)=0 implies q(t(T)i∣)=Cat(⋅,K−1K),q\! ( z_t(T)^i x )=Cat(·;K^-1 1_K), independently of x, so that setting pθ(t(T)i)=K−1Kp_θ( z_t(T)^i)=K^-1 1_K gives zero prior KL. Likewise, as t(0)=1/(T+1)→0t(0)=1/(T+1)→ 0 and αt(0)→1 _t(0)→ 1, q(t(0)i∣)⟶Cat(⋅,i),q\! ( z_t(0)^i x ) (·; x^i), and the endpoint denoising parameterization copies its input, making the reconstruction loss vanish. Neither argument depends on the support of the data distribution. Thus, the remaining diffusion term converges pointwise in x to ℒUDM(;θ)=∫01t∼qt(⋅∣)∑i=1Lαt′Kαt[ _UDM( x;θ)= _0^1E_ z_t q_t(· x) _i=1^L _t K _t [ Kx¯ℓi−Kx¯θ,ℓi−∑j≠ℓix¯jix¯ℓiilogx¯θ,ℓix¯jix¯θ,jix¯ℓi]dt, K x_ _i^i- K x_θ, _i^i- _j≠ _i x_j^i x_ _i^i x_θ, _i^i x_j^i x_θ,j^i x_ _i^i ]dt, where ¯i=Kαti+(1−αt)K ν^i=K _t ν^i+(1- _t) 1_K and ℓi _i denotes the non-zero dimension of ti z_t^i. Taking expectation of x then gives: ℒUDM(θ)=∫01,t∑i=1Lαt′Kαt[Kx¯ℓi−Kx¯θ,ℓi−∑j≠ℓix¯jix¯ℓilogx¯θ,ℓix¯jix¯θ,jix¯ℓi]t, _UDM(θ)= _0^1E_ x, z_t _i=1^L _t K _t [ K x_ _i^i- K x_θ, _i^i- _j≠ _i x_j^i x_ _i^i x_θ, _i^i x_j^i x_θ,j^i x_ _i^i ]dt, where x comes from pdatap_data or .Hence, the continuous-time UDM NELBO is likewise well defined for each individual clean sequence, irrespective of the support of the data distribution. Validity of the leave-one-out optimum under finite data. We next verify that the leave-one-out characterization of the UDM NELBO optimum remains valid under Assumption 1. We refer the readers to Appendix A.1 of Gourevitch et al. (2026) for a full understanding. The proof of their optimum results applies directly when pdatap_data is replaced by . Their argument relies only on Bayes’ rule, the Chapman--Kolmogorov identity, factorization of the bridge across coordinates, and the Bayes-compatible simplex extension of the plug-in bridge. None of these steps imposes any assumption on the cardinality or support structure of the clean-data distribution. Validity of the theoretical results in Section 3. Having established that the UDM NELBO and its leave-one-out optimum remain valid under Assumption 1, the remaining results in Section 3 follow without any additional assumption on the data distribution. Proposition 1 is obtained from the leave-one-out posterior using only Bayes’ rule, factorization of the uniform forward channel across coordinates, and a finite subset expansion. These arguments apply identically after replacing pdatap_data by . Likewise, Theorem 1 only groups the probability mass of the data distribution according to the matched-context count MiM^i and uses the fact that sequences with the same match count have the same uniform-channel likelihood. Proposition 2 then follows by differentiating the resulting finite normalized mixture weights. None of these derivations requires any assumption on the cardinality or support structure of pdatap_data. Therefore, Proposition 1, Theorem 1, and Proposition 2 all remain valid under the finite empirical distribution in Assumption 1. Appendix C Experimental Details C.1 Additional Experimental Results (a) Duality (b) LOO+CE Figure 3: Sampling performance of time-variant and time-agnostic UDMs across different numbers of function evaluations (NFEs). We compare generative perplexity as the sampling budget varies under (a) Duality and (b) LOO+CE; the annotated values report the entropy of generated samples. Figure 4: Fisher sensitivity under different time-input reparameterizations, where z sampling time and Fisher sensitivity evaluation time are equal. All sensitivities are measured with respect to the original diffusion time t, not reparametrized time t t for model input. We provide generative perplexity results for Duality and LOO+CE trained on OWT, which were omitted from the main paper. For each NFE, we generate 256 samples. Following Zheng et al. (2025) and Duality (Sahoo et al., 2025), we use float64 precision for sampling, as low-precision sampling can lead to reduced diversity and misleading generative perplexity in discrete diffusion models. As shown in Figure 3 (Left), time-agnostic and time-variant Duality achieve nearly identical generative perplexity across a wide range of NFEs. Their sample entropies are also closely matched, indicating similar sampling quality and diversity. On the other hand, as shown in Figure 3 (Right), LOO+CE exhibits similar sample entropy for the time-agnostic and time-variant models, while the time-agnostic model achieves lower generative perplexity. C.2 Experimental Settings Settings on training experiments. We use exactly the same experimental settings as the original UDM (Schiff et al., 2025) and Diffusion Duality (Sahoo et al., 2025). We detokenize the One Billion Words dataset following Lou et al. (2024); Sahoo et al. (2024), whose official code can be found. We tokenize LM1B using the bert-base-uncased tokenizer, consistent with He et al. (2023). We then concatenate and pack the sequences to a fixed length of 128 (Raffel et al., 2020). For OWT, we use the GPT-2 tokenizer and similarly concatenate and pack sequences to length 1,024; during packing, we insert an eos token between consecutive documents. As OWT does not provide an official validation split, we reserve the last 100k documents for validation. We parameterize the diffusion backbone of all models with the modified diffusion transformer architecture (Peebles and Xie, 2023) from Lou et al. (2024); Sahoo et al. (2024). For time-variant models, we use 12 layers, a hidden dimension of 768, 12 attention heads. For the diffusion backbone, we use the AdamW optimizer with a batch size of 512, constant learning rate warmup from 0 to a learning rate of 3e-4 for 2,500 steps, following prior works (Sahoo et al., 2024; Lou et al., 2024; Sahoo et al., 2025). We use a dropout rate of 0.1. See Appendix C.3 for time-agnostic and hybrid models. Experimental setting for the Hamming-distance and Δ analyses (Figure 1) The minimum hamming distance histogram is obtained by exhaustively comparing all 8,730,9668,730,966 OWT training rows of length L=1024L=1024. For each row, we compute its exact minimum Hamming distance as L minus the maximum position-wise match count over all other rows, excluding its own row index. Special tokens are treated like ordinary tokens throughout. For each t∈0.05,0.10,…,0.95,0.99t∈\0.05,0.10,…,0.95,0.99\, we independently sample 2,0002,000 source rows uniformly with replacement and corrupt each token by copying it with probability 1−t1-t or drawing uniformly from the full 50,25850,258-token vocabulary, including the added padding token, with probability t. For each pair, we independently sample i∼Unif([L])i ([L]) and compute Δi=Mi(,t)−max′:′i≠iMi(′,t) _i=M^i( x, z_t)- _ x :\, x i≠ x^iM^i( x , z_t) by scanning the entire training set, where MiM^i counts matches only outside i. The probability mass at each time is estimated from the empirical frequencies of these 2,0002,000 margins. Both measurements use exact exhaustive GPU scans, without approximate nearest-neighbor search. We store the entire token matrix losslessly in 16-bit integers (approximately 17.917.9 GB) on each H100 GPU and compute integer match counts across training rows. Experimental setting for the learned time-sensitivity analysis (Figure 2 (left)). We use the public checkpoint both for Duality and LOO+CE trained on OWT. We uniformly sample 10001000 held-out OWT sequences of length L=1024L=1024 without replacement. For each tcorruption∈0.01,0.50,0.99t_corruption∈\0.01,0.50,0.99\, we independently draw one corrupted sequence per clean sequence from qtcorruptionq_t_corruption and reuse identical inputs for both models. Each corrupted input z is then held fixed while the evaluation time varies over t∈0.01,0.02,…,0.99t∈\0.01,0.02,…,0.99\. We apply Eq. (8) to the raw network softmax prediction θi(,t) x_θ^i( z,t) in place of the oracle, before any analytic local-token correction. Derivatives are computed using automatic differentiation with respect to raw t. Each curve reports the arithmetic mean over all valid token positions, including unchanged tokens, followed by the mean over the 10001000 sequences. In particular, for LOO+CE, we measure Fisher sensitivity with respect to θ x_θ, the LOO denoiser actually used for generation and in the plug-in reverse kernel, rather than θPosterior μ_θ^Posterior, which is introduced through the LOO reparameterization in Eq. (11). Perplexity measure. In all experiments, we estimate validation/test perplexity by exponentiating the UDM NELBO. That is, we report perplexity that corresponds to an upper bound on the true perplexity, following the standard convention in most discrete diffusion language modeling works (Sahoo et al., 2024; Sahoo et al., 2025; Schiff et al., 2025; Gourevitch et al., 2026; Shi et al., 2024). Likewise, the time-resolved NLL plots report the expected token-normalized NELBO integrand at time t. C.3 Details on Time-agnostic Models Time-agnostic UDMs. Across all frameworks—including the conventional UDM (Schiff et al., 2025), Diffusion Duality (Sahoo et al., 2025), and the leave-one-out denoiser trained with cross-entropy (Gourevitch et al., 2026)—the only modification we make is to render the model time-agnostic. All other framework-specific components, including the loss function, forward process, noise schedule, and curriculum learning procedure, are kept exactly as originally proposed. More specifically, we directly build upon the original codebases and leave their framework-specific implementations unchanged, except for removing time conditioning from the model. We refer the reader to Appendix A.2 for a detailed description of these frameworks. For experiments on LM1B, we compared 1) time-variant model with 12 Transformer layers, 2) time-agnostic model with 12 Transformer layers, 3) time-agnostic model with 13 Transformer layers that matches parameter count of 1), and 4) hybrid model. For experiments on OWT, we compared the time-variant model with 12 Transformer layers and the time-agnostic model with 13 Transformer layers that matches the parameter count of the time-variant counterpart. All architectures are explained in detail below. Time-agnostic architecture. We use the same Transformer backbone for the time-conditioned and time-agnostic models, differing only in how the normalization and residual modulation parameters are specified. For the time-conditioned model, we follow the backbone of Sahoo et al. (2025); Schiff et al. (2025), where each Transformer block uses the adaLN-Zero parameterization: a timestep embedding produces time-dependent scale, shift, and residual-gating parameters. For time-agnostic models, we simply replaced time-dependent parameters with single learnable parameters. More precisely, let (t)∈ℝdcc(t) ^d_c denote the timestep embedding, where dcd_c is the conditioning dimension, and let d denote the Transformer hidden dimension. For the ℓ -th Transformer block, the modulation parameters are generated as ( ( OPENℓ,attn(t),ℓ,attn(t),ℓ,attn(t),ℓ,mlp(t),ℓ,mlp(t),ℓ,mlp(t))=ℓ(t)+ℓ, β_ ,attn(t),s_ ,attn(t),g_ ,attn(t), β_ ,mlp(t),s_ ,mlp(t),g_ ,mlp(t) )=W_ c(t)+b_ , where ℓ∈ℝ6d×dcW_ ^6d× d_c, ℓ∈ℝ6db_ ^6d, and each modulation vector lies in ℝdR^d. Here, β and s denote the channel-wise shift and scale modulation of LayerNorm, respectively, while g denotes the channel-wise residual gate. Let ℓ∈ℝL×dh_ ^L× d denote the sequence of hidden representations entering the ℓ -th block. Ignoring dropout for notational simplicity, the block is ~ℓ h_ =ℓ+ℓ,attn(t)⊙Attnℓ((d+ℓ,attn(t))⊙LNℓ,attn(ℓ)+ℓ,attn(t)), =h_ +g_ ,attn(t) _ ( ( 1_d+s_ ,attn(t) ) _w_ ,attn(h_ )+ β_ ,attn(t) ), ℓ+1 _ +1 =~ℓ+ℓ,mlp(t)⊙MLPℓ((d+ℓ,mlp(t))⊙LNℓ,mlp(~ℓ)+ℓ,mlp(t)). = h_ +g_ ,mlp(t) _ ( ( 1_d+s_ ,mlp(t) ) _w_ ,mlp( h_ )+ β_ ,mlp(t) ). Here, ℓ,attn,ℓ,mlp∈ℝdw_ ,attn,w_ ,mlp ^d are the learned channel-wise LayerNorm scales. All element-wise operations above act along the hidden dimension, with the d-dimensional modulation vectors broadcast across sequence positions. Thus, time enters the Transformer backbone only through the scale, shift, and residual-gating parameters of each block. For the time-agnostic model, we remove the timestep embedding (t)c(t) entirely and directly learn a time-independent set of modulation parameters for each block: (ℓ,attn,ℓ,attn,ℓ,attn,ℓ,mlp,ℓ,mlp,ℓ,mlp)∈ℝ6d, ( β_ ,attn,s_ ,attn,g_ ,attn, β_ ,mlp,s_ ,mlp,g_ ,mlp ) ^6d, where each of the six vectors belongs to ℝdR^d.The resulting static normalization modulation is equivalent to a standard affine LayerNorm. More precisely, for a token representation ∈ℝdh ^d, define LN0()=−μ()dVar()+ϵ,LN()=⊙LN0().LN_0(h)= h-μ(h) 1_d Var(h)+ε, _w(h)=w _0(h). Then, for time-independent ,∈ℝds, β ^d, (d+)⊙LN()+=(⊙(d+))⊙LN0()+=⊙LN0()+, ( 1_d+s) _w(h)+ β= (w ( 1_d+s) ) _0(h)+ β= γ _0(h)+ β, where =⊙(d+)∈ℝd γ=w ( 1_d+s) ^d. The last expression is precisely standard affine LayerNorm with learned channel-wise scale γ and shift β. Thus, the time-agnostic block is equivalent to ~ℓ h_ =ℓ+ℓ,attn⊙Attnℓ(LNℓattn(ℓ)), =h_ +g_ ,attn _ (LN^attn_ (h_ ) ), ℓ+1 _ +1 =~ℓ+ℓ,mlp⊙MLPℓ(LNℓmlp(~ℓ)), = h_ +g_ ,mlp _ (LN^mlp_ ( h_ ) ), where LNℓattnLN^attn_ and LNℓmlpLN^mlp_ are standard learned affine LayerNorms with channel-wise scale and shift in ℝdR^d, and ℓ,attn,ℓ,mlp∈ℝdg_ ,attn,g_ ,mlp ^d are learned, time-independent residual gates initialized to zero. We apply the same replacement to the time-conditioned modulation in the final output layer. Hybrid architecture. The hybrid model combines the time-conditioned and time-agnostic parameterizations above, using explicit time conditioning only in the high-noise regime. Specifically, for each Transformer block, we retain both the original time-modulation layer and a separate set of directly learned, time-independent modulation parameters. The six modulation vectors are given by (ℓ,r,ℓ,r,ℓ,r)r∈attn,mlp=ℓ(t)+ℓ,t>0.8,(ℓ,r0,ℓ,r0,ℓ,r0)r∈attn,mlp,t≤0.8. ( β_ ,r,s_ ,r,g_ ,r )_r∈\attn,mlp\= casesW_ c(t)+b_ ,&t>0.8,\\ ( β^0_ ,r,s^0_ ,r,g^0_ ,r )_r∈\attn,mlp\,&t≤ 0.8. cases where the six vectors in the second branch are learned directly and do not depend on time. Thus, the model is identical to the time-conditioned architecture for t>0.8t>0.8 and to the time-agnostic architecture for t≤0.8t≤ 0.8, with all other components left unchanged.