Paper deep dive
Generalized Discrete Diffusion from Snapshots
Oussama Zekri, Théo Uscidda, Nicolas Boullé, Anna Korba
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 2:26:49 AM
Summary
Generalized Discrete Diffusion from Snapshots (GDDS) is a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. By utilizing uniformization for forward noising and a snapshot-based ELBO for reverse training, GDDS improves training efficiency and generation quality, outperforming existing discrete diffusion methods and autoregressive models in large-vocabulary tasks.
Entities (5)
Relation Signals (3)
Discrete Diffusion → modelsdynamicsvia → CTMC
confidence 95% · In discrete diffusion, the dynamics of a single token are described by a continuous-time Markov chain (CTMC)
GDDS → optimizes → ELBO
confidence 95% · we derive a simple evidence lower bound (ELBO) based on snapshot latents
GDDS → utilizes → Uniformization
confidence 95% · The forward noising process relies on uniformization and enables fast arbitrary corruption.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce Generalized Discrete Diffusion from Snapshots (GDDS), a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : \href{this https URL}{this https URL}.
Tags
Links
- Source: https://arxiv.org/abs/2603.21342v1
- Canonical: https://arxiv.org/abs/2603.21342v1
Trouble viewing inline? Open PDF directly →
Full Text
162,968 characters extracted from source content.
Expand or collapse full text
Generalized Discrete Diffusion from Snapshots Oussama Zekri Théo Uscidda Nicolas Boullé Anna Korba Abstract We introduce Generalized Discrete Diffusion from Snapshots (GDDS), a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : https://oussamazekri.fr/gdds. Machine Learning, ICML boxthm]Proposition boxthm]Example boxthm]Corollary boxthm]Lemma boxthm]Definition boxthm]Design Principle 1 Introduction Diffusion models (ho2020denoising; song2020score) recently became a core component of generative modeling and achieved remarkable success in high-dimensional tasks defined on continuous domains, such as image (rombach2022high; saharia2022photorealistic), audio (kong2020diffwave; liu2023audioldm), and video generation (brooks2024video; wiedemer2025video). The extension of diffusion modeling to discrete data is of great interest since many data structures (including text, graphs, and molecules) are inherently discrete. This has led to the emergence of diffusion Large Language Models (dLLMs) (lou2023discrete; li2025survey). dLLMs offer a competitive alternative to the auto-regressive (AR) paradigm dominating language modeling (touvron2023llama; team2023gemini; liu2024deepseek) due to their ability to generate all tokens simultaneously. Figure 1: Zero-shot transfer of OWT-trained models. Zero-shot perplexity (↓ ) on three representative downstream validation sets from Table 3: PTB, LM1B, and Wikitext. Across this high-to-low perplexity range, GDDS Gauss consistently achieves the lowest transfer perplexity, highlighting the stronger generalization capability induced by semantically structured noising processes. Discrete diffusion models come in several variants, mainly differing in the choice of the noising process and how denoising is performed. Masked diffusion models (MDM) (sahoo2024simple; shi2024simplified; ou2024your; nie2025large) rely on a noising process where tokens are progressively replaced by a special [MASK] token. For uniform-state diffusion models (USDMs) (austin2021structured; schiff2024simple; sahoo2025diffusion), they are replaced with samples from the uniform distribution over the set of all possible tokens. These forward dynamics directly shape the reverse generation process: USDMs allow tokens to be updated continuously, whereas MDMs fix them once they are unmasked. Figure 2: Overview of GDDS. A clean sequence 0x_0 is first noised exactly by the forward CTMC at a sampled time t∈[0,1]t∈[0,1], yielding a snapshot sequence (t,t)(x_t,t). The mean parametrization is then used as a denoiser: given the snapshot, the model predicts the clean-token posterior directly from (t,t)(x_t,t), so training is performed on snapshots rather than through a full path-wise objective. The design space for discrete diffusion models remains surprisingly narrow. Most existing dLLMs pair a simplistic token-wise corruption rule (masking or uniform replacement) with the mean parametrization (austin2021structured). Here, a denoiser predicts a distribution over the clean token and reverse transition probabilities are derived from this prediction through an ELBO objective. This leads to two bottlenecks: (i) the forward process is blind to any notion of neighborhood in discrete spaces (e.g., semantic proximity in language), and (i) mean parametrization tightly constrains how denoising uncertainty can be translated into reverse dynamics, becoming increasingly restrictive beyond uniform/masked noise and at LLM scale. Advancing dLLMs calls for structure-aware noising and more flexible parametrizations that remain computationally scalable for large vocabularies and long contexts. In this work, we generalize discrete diffusion methods by considering arbitrary noising processes and propose a tractable associated training method. We introduce the Generalized Discrete Diffusion from Snapshots (GDDS) framework, which builds upon the most general formulation of interpolating discrete diffusion and extends it far beyond the restricted subclasses explored in prior work (sahoo2024simple; shi2024simplified; ou2024your; von2025generalized; zhou2025next; amin2025masking). GDDS introduces three key advances for discrete diffusion: 1) Generalized interpolating discrete diffusion: a mathematical framework covering arbitrary Markovian noising processes, encompassing all existing approaches. 2) Efficient noising process: a fast forward arbitrary corruption method for large vocabularies, requiring only column access to the rate matrix characterizing the noising process. 3) Parametrization and ELBO: a principled parametrization for reverse transition probabilities, yielding a simple ELBO training objective based on snapshot samples. These components form the first discrete diffusion framework that is fully general and computationally efficient. Our experiments on large-scale language modeling tasks demonstrate state-of-the-art modeling and generation quality. Figure 2 summarizes the two ingredients behind GDDS: exact forward noising to a snapshot and snapshot-level denoising. 2 Background and Preliminaries This section provides background material on discrete diffusion, including the definition of rate matrices that characterize the evolution of continuous-time Markov chains, as well as common choices used in the literature. 2.1 Discrete Diffusion In discrete diffusion, the dynamics of a single token are described by a continuous-time Markov chain (CTMC) (campbell2022continuous; lou2023discrete), which is a stochastic process (xt)t∈[0,T](x_t)_t∈[0,T] operating on a finite vocabulary =v1,…,vmV=\v_1,…,v_m\. We denote by δ[x] _[x] the one-hot encoding of x∈x . For a column-stochastic matrix111A column stochastic matrix is a matrix whose columns are probability distributions, hence ∑i=1mF(i,j)=1 _i=1^mF(i,j)=1 with F(i,j)∈[0,1]F(i,j)∈[0,1] for any column 1≤j≤m1≤ j≤ m. F∈ℝm×mF ^m× m, we use the shorthand F(⋅,x)≡Fδ[x]F(·,x)≡ F _[x] to denote the probability vector corresponding to the column indexed by x. Forward and reverse evolution. Let T>0T>0 be a fixed time horizon. At any time t∈[0,T]t∈[0,T], the distribution of xt∈x_t is denoted by qt∈Δmq_t∈ _m, where Δm _m is the probability simplex over V. We set q0:=qdataq_0:=q_data and qT:=qrefq_T:=q_ref a simple reference distribution. We represent the forward noising process through a family of Markov transition matrices (Kt)t∈[0,T](K_t)_t∈[0,T] acting on marginals as qt=Ktq0q_t=K_tq_0, with xt∼qtx_t q_t. Here, Kt∈ℝm×mK_t ^m× m describes how probability mass flows across tokens as corruption increases. Existing discrete diffusion schemes, such as uniform or masking corruption, correspond to particular choices of KtK_t. Given a clean token x0∼qdatax_0 q_data, the noised token xt∈x_t at time t is drawn from the categorical distribution: qt(xt∣x0)=Cat(xt;Kt(⋅,x0))=Kt(xt,x0).q_t(x_t x_0)=Cat(x_t;K_t(·,x_0))=K_t(x_t,x_0). (1) While KtK_t can be specified directly, we focus on the principled setting where these noising operators are induced by a continuous-time Markov process with (possibly time-inhomogeneous) rate matrix (also called infinitesimal generator) QtQ_t. This matrix is defined as Qt(i,j)=limh↓0ℙ(xt+h=i|xt=j)−δijhQ_t(i,j)= _h 0 P(x_t+h=i|x_t=j)- _ijh for i≠ji≠ j, and diagonal entries enforcing conservation of mass Qt(j,j)=−∑i≠jQt(i,j)Q_t(j,j)=- _i≠ jQ_t(i,j). In this case, KtK_t is defined as the solution to the Kolmogorov forward equation: dKtdt=QtKt,K0=Im,for t∈[0,T]. \,dK_t\,dt=Q_tK_t, K_0=I_m, t∈[0,T]. (2) Solving Eq. 2 yields Kt=exp(∫0tQsds),K_t=T ( _0^tQ_s\,ds), where T is the time-ordering operator (see Appendix A). Since each column Kt(⋅,x0)K_t(·,x_0) for x0∈x_0 lies in the probability simplex Δm _m and corresponds to the forward marginal qt(⋅∣x0)q_t(· x_0) as given in Eq. 1, a noisy token xtx_t can be obtained by directly sampling from this categorical distribution, without simulating the underlying continuous-time trajectory. Moreover, since qt=Ktq0q_t=K_tq_0, token distributions evolve according to the Kolmogorov forward equation for marginals: dqtdt=Qtqt,xt∼qt,for t∈[0,T]. \,dq_t\,dt=Q_tq_t, x_t q_t, t∈[0,T]. (3) The time reversal of this equation (kelly2011reversibility) defined through pt:=qT−tp_t:=q_T-t is dptdt=Q¯tpt,xT−t∼pt,for t∈[0,T], \,dp_t\,dt= Q_tp_t, x_T-t p_t, t∈[0,T], (4) where Q¯t(i,j)=(qt(i)/qt(j))Qt(j,i) Q_t(i,j)=(q_t(i)/q_t(j))Q_t(j,i) if i≠ji≠ j and Q¯t(i,i)=−∑j≠iQ¯t(j,i) Q_t(i,i)=- _j≠ i Q_t(j,i). Without loss of generality, we select T=1T=1 as any bounded interval [0,T][0,T] can be rescaled to [0,1][0,1] through the change of variable t↦t/Tt t/T. In discrete diffusion, a neural network learns to simulate the reverse dynamics given by Eq. 4 to reconstruct a clean data x0x_0 from a fully noised quantity x1x_1. Rate matrices. At any time t∈[0,1]t∈[0,1], the forward and reverse CTMCs evolve according to rate matrices QtQ_t and Q¯t Q_t with non-negative off-diagonal entries Qt(i,j)≥0Q_t(i,j)≥ 0 for i≠ji≠ j, and diagonal entries verifying Qt(j,j)=−∑i≠jQt(i,j)Q_t(j,j)=- _i≠ jQ_t(i,j), and analogously for Q¯t Q_t. We factorize them into exit rates and jump kernels as Qt Q_t =(Ft−Im)diag(f1(t),…,fm(t)), =(F_t-I_m)\,diag(f_1(t),…,f_m(t)), Q¯t Q_t =(Rt−Im)diag(r1(t),…,rm(t)), =(R_t-I_m)\,diag(r_1(t),…,r_m(t)), where fj(t)=∑i≠jQt(i,j)f_j(t)= _i≠ jQ_t(i,j) and rj(t)=∑i≠jQ¯t(i,j)r_j(t)= _i≠ j Q_t(i,j) are the (forward and reverse) exit rates, controlling how often the chain leaves state j. The matrices FtF_t and RtR_t specify where the chain jumps when it leaves a state and are defined as column-stochastic matrices: Ft(i,j)=Qt(i,j)fj(t),fj(t)>0,0,fj(t)=0, for i≠j,F_t(i,j)= cases Q_t(i,j)f_j(t),&f_j(t)>0,\\ 0,&f_j(t)=0, cases for i≠ j, and Ft(j,j)=1−∑i≠jFt(i,j)F_t(j,j)=1- _i≠ jF_t(i,j), analogously for RtR_t. Note that this factorization is not exploited in the literature. 2.2 Designs of the rate matrix A common choice to simplify the forward noising process is to select equal forward exit rates: f1(t)=…=fm(t)=f(t)f_1(t)=…=f_m(t)=f(t), and a time-independent forward jump kernel Ft=F_t=F. In this case, Qt=f(t)(F−Im),Q_t=f(t)(F-I_m), and a single matrix F∈ℝm×mF ^m× m must be stored. In this model, the time-ordered exponential T simplifies to a standard matrix exponential, and ensures that KtK_t admits the following closed form: Kt=exp(f¯(t)(F−Im)),where f¯(t)=∫0tf(s)ds.K_t= ( f(t)(F-I_m) ), f(t)= _0^tf(s)\,ds. Being able to sample from columns of the matrix exponential KtK_t is crucial to design a scalable noising process. Usual forms of F. For typical vocabulary sizes used in language models (m=50,257m=50,257 for GPT-2; radford2019language), storing a dense F∈ℝm×mF ^m× m requires more than 2.5×1092.5× 10^9 parameters (≈20≈ 20GB in double precision) and each matrix-vector products involving F costs (m2)O(m^2) time complexity, making it computationally impractical. Hence, forward kernels are usually highly structured (austin2021structured; campbell2022continuous; lou2023discrete), such as the ones related to the uniform and mask noising processes: Funiform≔1m⊤,Fabsorb≔δ[MASK]⊤,F uniform 1m11 , F absorb _ [MASK]1 , (5) for which KtK_t admits closed‐form expressions (austin2021structured; lou2023discrete), enabling efficient noising. Since these F matrices are idempotent, the exponential matrix writes Kt=exp(−f¯(t))Im+(1−exp(−f¯(t)))FK_t= (- f(t))I_m+(1- (- f(t)))F. However, these restrictive structures impose rigid corruption patterns on the tokens, motivating our flexible approach. 3 Forward noising with diffusion 3.1 Generalized interpolating discrete diffusion We consider a time-differentiable, decreasing, mixing rate t↦αt:[0,1]→[0,1]t _t:[0,1]→[0,1] such that α0=1 _0=1, α1=0 _1=0, and αt<1 _t<1 for t>0t>0. We introduce a time-differentiable column-stochastic mixing matrix Πt∈ℝm×m _t ^m× m, which specifies how probability mass is redistributed across tokens as noise increases, along with its interpolating matrix as222We assume that, for every t∈(0,1)t∈(0,1), KtK_t is invertible and the solution v(t,x) v^(t,x) to the linear system Kt⊤v(t,x)=K˙t⊤δx K_t v^(t,x)= K_t _x satisfies v[y](t,x)≥0 v^(t,x)_[y]≥ 0 for y≠xy≠ x to guarantee that KtK_t induces a valid CTMC. Kt:=αtIm+(1−αt)Πt,t∈[0,1].K_t:= _tI_m+(1- _t) _t, t∈[0,1]. (6) Here, Πt _t encodes the structure of the noising mechanism and αt _t its intensity. This formulation recovers common discrete diffusion schemes as special cases, such as masked or uniform. Yet, more general choices of Πt _t allow for structured and token-dependent corruption mechanisms. The rate matrix QtQ_t associated with KtK_t is given in Section 3.1. boxprop Let t≥0t≥ 0 and denote by K˙t K_t the time derivative of KtK_t. The rate matrix induced by Eq. 6 is Qt=K˙tKt−1Q_t= K_tK_t^-1. Choosing a column-constant mixing matrix Πt=πt⊤ _t= _t1 (a rank-one form with πt∈Δm _t∈ _m) in Eq. 6 yields the GIDD formulation of (von2025generalized, Lem. 3.6), namely Qt=α˙tαtIm+(1−αt)π˙t⊤−α˙tαtπt⊤Q_t= α_t _tI_m+(1- _t) π_t1 - α_t _t _t1 (see Section B.1). However, this formulation encompasses all existing frameworks, including (zhou2025next) and GenMD4 (shi2024simplified), unlike GIDD (von2025generalized). Expressiveness. Reversely, given any rate matrix QtQ_t, we aim to find a mixing matrix Πt _t such that the interpolating matrix KtK_t defined by Eq. 6 coincides with a solution to Eq. 2; which induces the marginal (qt)t≥0(q_t)_t≥ 0 as in Eq. 3. boxprop Let αt _t a mixing rate such that α˙0<0 α_0<0 and QtQ_t a rate matrix. There exists a unique mixing matrix Πt∈ℝm×m _t ^m× m such that for all t∈[0,1]t∈[0,1], Kt=αtIm+(1−αt)Πt=exp(∫0tQsds).K_t= _tI_m+(1- _t) _t=T ( _0^tQ_s\,ds ). Following Section 3.1, if Πt _t is known in closed-form, then simulating the noising process becomes possible. Indeed, qt(⋅∣x0)=Cat(⋅;Kt(⋅,x0))q_t(· x_0)=Cat(·;K_t(·,x_0)) requires only the evaluation of the column Πt(⋅,x0) _t(·,x_0) instead of costly matrix exponentiations. While columns are known in closed form for uniform or masked schemes, this is generally not the case for an arbitrary Πt _t. 3.2 Efficient forward noising through uniformization Since computing the marginals qt(⋅∣x0)q_t(· x_0) exactly is generally intractable, we employ an exact noising procedure based on uniformization. Classical uniformization provides an exact Poisson-based representation of the matrix exponential KtK_t (jensen1953markoff; stewart2009probability). Here, we use the same procedure to generate exact forward samples xt∼qt(⋅∣x0)x_t q_t(· x_0) without requiring exact knowledge of these marginals; hence avoiding computing the exponential. Following Section 3.1, the interpolating matrix in Eq. 6 is expressive enough to represent any rate matrix QtQ_t. Recall that any such QtQ_t can be written in factored form as Qt=(Ft−Im)diag(f1(t),…,fm(t))Q_t=(F_t-I_m)diag(f_1(t),…,f_m(t)). To simplify the exposition, we focus on the shared exit rates case: Qt=f(t)(Ft−Im),Q_t=f(t)(F_t-I_m), (7) which preserves the transition structure encoded in FtF_t, while making the uniformization-based noising process significantly easier to implement. This result can be extended to general non-shared exit rates through Poisson thinning. We denote by f¯(t)=∫0tf(s)ds f(t)= _0^tf(s)\,ds the integrated exit rate and set the mixing rate to αt=exp(−f¯(t)) _t= (- f(t)) for t∈[0,1]t∈[0,1]. boxprop [Uniformization] Consider a rate matrix QtQ_t of the form (7) and the mixing rate αt=exp(−f¯(t)) _t= (- f(t)), where f¯(t)=∫0tf(s)ds f(t)= _0^tf(s)\,ds. Let (Nt)t∈[0,1](N_t)_t∈[0,1] be a non-homogeneous Poisson process with intensity f¯(t) f(t), and denote by 0<T1<…<TNt≤t0<T_1<…<T_N_t≤ t its jump times on [0,t][0,t]. The unique matrix Πt _t provided by Section 3.1 is Πt=[FTNt…FT1∣Nt≥1] _t=E[F_T_N_t… F_T_1 N_t≥ 1]. Therefore, computing qt(⋅∣x0)q_t(· x_0) at any t∈[0,1]t∈[0,1] amounts to computing a column of [FTNt…FT1∣Nt≥1]E[F_T_N_t… F_T_1 N_t≥ 1], which can be done approximately even when the vocabulary size m is large (dingle2004uniformization). If we only need to draw samples xt∼qt(⋅∣x0)x_t q_t(· x_0) rather than evaluate the full distribution, we can instead sample exactly by performing NtN_t transitions with the matrix FtF_t, using only Poisson sampling and column access to QtQ_t (see Section B.2). This procedures implicitly builds a discrete-time Markov chain zk=xTkz_k=x_T_k for all 1≤k≤Nt1≤ k≤ N_t initialized at z0=x0z_0=x_0. Algorithm 1 details the resulting token-level noising procedure and returns the noised token zNt=xTNtz_N_t=x_T_N_t, which coincides exactly with xt∼qt(⋅∣x0)x_t q_t(· x_0) following Section 3.2. Algorithm 1 Exact general noising, token level 1: Input: clean token z0=x0z_0=x_0, time t∈[0,1]t∈[0,1], intensity f¯(t) f(t), rate matrix QtQ_t as in Eq. 7 2: Sample number of jumps Nt∼Poisson(f¯(t))N_t ( f(t)) 3: Sample and sort the jump times as T1<…<TNtT_1<…<T_N_t 4: for k=1k=1 to NtN_t do 5: Sample jump zk∼FTk(⋅,zk−1)z_k F_T_k(·,z_k-1) 6: end for 7: return noised token xt=zNtx_t=z_N_t and jumps (zk,Tk)k=1Nt(z_k,T_k)_k=1^N_t Algorithm 1 enables efficient noising for any continuous-time noising process (beyond masked and uniform), requiring only column access to the rate matrix QtQ_t (instead of its generally intractable matrix exponential). This procedure generalizes easily to a parallel sequence-level algorithm for a sequence 0=x01…x0nx_0=x^1_0… x^n_0 of length n≥1n≥ 1 (see Algorithm 3 in Section B.2). While the time input can be any value t∈[0,1]t∈[0,1], selecting t=1t=1 yields a full forward noising path of the form ω=N1,(zk,Tk)k=1N1ω=\N_1,(z_k,T_k)_k=1^N_1\. 4 Reverse learning: aligning the generative model, and the objective Readers mostly interested in the implementation and the loss function may refer to Sections 4.3 and 2. 4.1 The core mismatch in reverse parametrization A common choice in the discrete diffusion litterature to simulate the reverse dynamics is to use the mean parametrization (also known as x0x_0-parametrization). Concretely, μθ:×[0,1]→Δm _θ:V×[0,1]→ _m is a neural network outputting a probability vector on the token space V, which aims to approximate the posterior of the clean token from snapshots latents s=(xt,t)s=(x_t,t) generated by the forward noising process, i.e., μθ(xt,t)[x0]≈q(x0∣xt,t) _θ(x_t,t)_[x_0]≈ q(x_0 x_t,t). It is often plugged into the reverse-time model via Bayes’ rule as pu∣tθ,path(xu∣xt)=∑x0∈q(xu∣xt,x0)μθ(x0∣xt,t),p^θ,path_u t(x_u x_t)= _x_0 q(x_u x_t,x_0) _θ(x_0 x_t,t), (8) where q(xu∣xt,x0)=qt∣u(xt∣xu)qu(xu∣x0)qt(xt∣x0)q(x_u x_t,x_0)= q_t u(x_t x_u)q_u(x_u x_0)q_t(x_t x_0) and qt∣u(xt∣xu)q_t u(x_t x_u) denotes the forward conditional from time u∈[0,1]u∈[0,1] to t∈[0,1]t∈[0,1] with u≤tu≤ t. However, plugging μθ _θ into the reverse-time model through Eq. 8 does not generally enforce μθ(xt,t)[x0]≈q(x0∣xt,t) _θ(x_t,t)_[x_0]≈ q(x_0 x_t,t). This construction glues the mean denoiser to the entire reverse CTMC: the same μθ _θ controls when the chain jumps (reverse intensities) and where it jumps (reverse destinations), creating a training burden mismatch. Our insight is that the mean network naturally parametrizes a snapshot generative model that should be trained to actually achieve μθ(xt,t)[x0]≈q(x0∣xt,t) _θ(x_t,t)_[x_0]≈ q(x_0 x_t,t), whereas modeling the reverse CTMC calls for a jump network θ j_θ designed directly for the path-wise generative model with path-wise latents ω. To align the objective with the generative object, we first parametrize the reverse CTMC directly by disentangling jump times and jump destinations. Then, we focus on how one should design a snapshot generative model from the mean parametrization. 4.2 Path-wise model and loss function Jump-states parametrization. Inspired by the factorization Q¯t=(Rt−Im)diag(r1(t),…,rm(t)) Q_t=(R_t-I_m)\,diag(r_1(t),…,r_m(t)) of the true reverse generator, we directly learn RtR_t while keeping the exit-rate schedule ri(t)i=1m\r_i(t)\_i=1^m fixed to the true reverse rates. Note that even when the forward process uses shared exit rates as in Eq. 7, the reverse exit rates are not shared in general (see e.g. the masked diffusion example in Section B.3.4). We consider a neural network θ:×[0,1]→Δm j_θ:V×[0,1]→ _m that yields the following jump-states parametrization: Q¯tθ=(Rtθ−Im)diag(r1(t),…,rm(t)), Q_t^θ=(R_t^θ-I_m)\,diag(r_1(t),…,r_m(t)), (9) where Rtθ∈ℝm×mR_t^θ ^m× m is the column-stochastic infinitesimal reverse jump kernel (where the chain jumps) defined by Rtθδ[xt]:=θ(xt,t)R_t^θ _[x_t]:= j_θ(x_t,t), and the exit rates ri(t)r_i(t) (when the chain jumps) remain fixed. This parametrization of Eq. 4 is fundamentally different from the score parametrization of (lou2023discrete), the schedule-conditioned parametrization of (amin2025masking) and the parametrization in Eq. 8 (cf. Section B.3.1). Path-wise ELBO. We derive the Evidence Lower Bound (ELBO) associated with our jump-states parametrization given by Eq. 9 in Section 4.2. This parametrization is key to obtain a simple, CTMC-aligned, ELBO with a clean learning objective: a weighted cross-entropy that matches the model reverse jump kernel to the ideal reverse jump kernel, with θ-independent weights given by the reverse exit rates r[xt](t)r_[x_t](t). The result holds for any forward rate matrix QtQ_t, as the interpolating family Kt=αtIm+(1−αt)ΠtK_t= _tI_m+(1- _t) _t can represent an arbitrary rate matrix (Eqs. 6 and 3.1). Here, ptθ,pathp^θ,path_t is induced by Eq. 4 where we replace Q¯t Q_t by Q¯tθ Q_t^θ. boxprop [Path-wise ELBO] Let x0∈x_0 , the ELBO is logp0θ,path(x0)≥−ℒx0path(θ)+Cx0path p_0^θ,path(x_0)≥-L_x_0^path(θ)+C^path_x_0, where ℒx0path(θ)=∫01xt∼qt(⋅∣x0)[r[xt]x0(t)CE(Rtx0,Rtθ)|xt]dt, _x_0^path(θ)=\! _0^1\!E_x_t q_t(· x_0)\!\! [r_[x_t]^x_0(t) CE (R_t^x_0,R_t^θ) |_x_t\! ]\!\!\,dt, and Cx0pathC^path_x_0 is independent of θ. Here, CE(Rtx0,Rtθ)|xt CE (R_t^x_0,R_t^θ) |_x_t denotes the cross-entropy between the vectors Rtx0(⋅,xt)R_t^x_0(·,x_t) and Rtθ(⋅,xt)=θ(xt,t)R_t^θ(·,x_t)= j_θ(x_t,t). Rtx0R_t^x_0 and r[xt]x0(t)r_[x_t]^x_0(t) denote respectively the true conditional reverse jump kernel and its associated exit rate (see Section B.3.2). In the masked diffusion case, where Πt=δ[MASK]⊤ _t= _ [MASK]1 , our jump parametrization and ELBO coincide with the parametrization (8) and ELBO used in prior work (sahoo2024simple; shi2024simplified; ou2024your); see Section B.3.4. Beyond this setting, other objectives such as (von2025generalized; zhou2025next; lou2023discrete; amin2025masking) apply to broad classes of noising processes but do not isolate such a clean weighted cross-entropy signal and involve additional terms. Indeed, we show in Section B.3.2 that ℒx0path(θ)L_x_0^path(θ) can be written as ℒx0path(θ)=∫01xt∼qt(⋅∣x0)[∑y≠xtqt(y∣x0)qt(xt∣x0)Qt(xt,i)(−logθ(xt,t)y)]dt. _x_0^path(θ)\!=\!\! _0^1\!\!\!E_x_t q_t(· x_0)\!\! [\! _y≠ x_t\!\! q_t(y x_0)q_t(x_t x_0)Q_t(x_t,i)(- j_θ(x_t,t)_y)\! ]\!\!\!\,dt. This loss remains useful beyond masking: when the forward marginal is tractable (typically, when Πt _t is known so that qt(⋅∣x0)q_t(· x_0) can be evaluated; e.g., in the masked or uniform case), ℒx0path(θ)L_x_0^path(θ) is fully computable and directly trains the path-wise reverse CTMC. However, we seek to avoid knowledge of qt(⋅∣x0)q_t(· x_0) for a general CTMC QtQ_t, for which the associated Πt _t is unknown. Campbell estimator. To mitigate this issue, we introduce a path-wise Campbell estimator that is a clean rewriting of the path-wise loss. It consists of applying Campbell’s formula (campbell1909study; last2018lectures) to ℒx0path(θ) L_x_0^path(θ), which transforms the integral into a sum over the whole noising path in [0,1][0,1] given by the Poisson process of Algorithm 1. Importantly, this expression does not involve any other quantity than the network output and the uniformization path produced by Algorithm 1, making it computable even when qt(⋅∣x0)q_t(· x_0) (i.e., Πt _t) is unknown. boxprop [Campbell estimator] Let x0∈x_0 and ω∼q[0,1](⋅∣x0)ω q_[0,1](· x_0) denote the full forward noising path produced by Algorithm 1. Writing ω=N1,(zk,Tk)k=0N1ω=\N_1,(z_k,T_k)_k=0^N_1\ for its jump counts and marks, we have ℒx0path(θ)=ω∼q[0,1](⋅∣x0)[∑k=1N1−logθ(zk,Tk)[zk−1]]. _x_0^path(θ)=E_ω q_[0,1](· x_0)\! [ _k=1^N_1- j_θ(z_k,T_k)_[z_k-1] ]\!. This loss is reminiscent of any-order AR objectives (e.g., XLNet; yang2019xlnet), except that each term predicts the pre-jump token zk−1z_k-1 from the post-jump noised context (zk,Tk)(z_k,T_k), and the factorization order is induced by Poisson jump times (closer in spirit to denoising permutation objectives such as MPNet; song2020mpnet). As a result, training requires evaluating many snapshot-wise conditionals along a single path and, in practice, calls for a two-stream mechanism (separating a content stream encoding the clean tokens from a query stream used to predict the target token [zk−1][z_k-1], as in XLNet), which is not naturally aligned with standard transformer architectures and empirically underperforms them (see Appendix E). 4.3 Snapshot model and loss function This observation motivates our approach: if powerful models mostly train on snapshot noised contexts, why should the variational latent variable be the entire path ω? Therefore, we consider a snapshot-latent variational formulation, where s=(xt,t)s=(x_t,t) replaces ω as the latent variable. Crucially, this choice also aligns with the perspective of li2025back that denoising models should predict the clean quantity through the mean parametrization, rather than a noised quantity as the jump states parameterizations do. This aligns the mean parametrization μθ _θ to its coherent generative model, and yields an objective that is directly compatible with standard architectures for any general noising process. Snapshot ELBO. Consider the snapshot latent s=(xt,t)s=(x_t,t) and the variational distribution qsnap(s∣x0):=qt(xt∣x0)q^snap(s x_0):=q_t(x_t x_0). We define the snapshot predictor p0θ,snap(x0∣s):=μθ(xt,t)x0p_0^θ,snap(x_0 s):= _θ(x_t,t)_x_0 from the output of the mean network, and derive the associated snapshot ELBO in Section 4.3. boxprop [Snapshot ELBO] Let x0∈x_0 , the ELBO is logp0θ,snap(x0)≥−ℒx0snap(θ)+Cx0snap p_0^θ,snap(x_0)≥-L_x_0^snap(θ)+C^snap_x_0, where ℒx0snap(θ)=∫01xt∼qt(⋅∣x0)[−logμθ(xt,t)[x0]]dt,L_x_0^snap(θ)= _0^1E_x_t q_t(· x_0) [- _θ(x_t,t)_[x_0] ]\,dt, and Cx0snapC^snap_x_0 is independent of θ. This computable snapshot ELBO boils down to denoising training on (x0,xt,t)(x_0,x_t,t), without requiring the explicit knowledge of qt(⋅∣x0)q_t(· x_0).It is also well-suited to use with standard time-conditioned bidirectional transformer architectures (e.g., DDiT, peebles2023scalable). Note that (shi2025demystifying) derived the same ELBO expression as a reweighted form of the path-wise ELBO (Section 4.2), but this equivalence holds only in the masked diffusion setting and for a specific “simple” weight. Algorithm 2 GDDS training algorithm 1: Input: Training dataset 0(1),…,0(Ndata)∼datax_0^(1),…,x_0^(N_data) q_data 2: for k=1k=1 to NdataN_data (potentially several epochs) do 3: Sample t∼Unif[0,1]t [0,1] 4: Sample t∼t(⋅∣0(k))x_t q_t(· _0^(k)) with Algorithm 3 5: Minimize −log(θ(t,t)[0(k)])- ( μ_θ(x_t,t)_[x_0^(k)]) 6: end for 7: return θ μ_θ Algorithm 2 is reminiscent of the general procedure in (bengio2013generalized), but applied to the corruption process induced by the forward noising diffusion on discrete spaces. Information-calibration decomposition. To make precise the trade-off between using less information (a snapshot) and optimizing better (lower miscalibration), we compare the expected negative log-likelihood (NLL) of predicting a clean token x0∼qdatax_0 q_data either from the full forward path ω∼q[0,1](⋅∣x0)ω q_[0,1](· x_0) or from a randomly sampled snapshot s=(xt,t)s=(x_t,t) with t∼Unif[0,1]t [0,1] independently sampled, through the quantity ΔθNLL≔[−logp0θ,snap(x0∣s)]−[−logp0θ,path(x0∣ω)] ^NLL_θ [- p_0^θ,snap(x_0 s)]-E[- p_0^θ,path(x_0 ω)]. The resulting NLL gap admits a clean decomposition into an intrinsic information path gap (IPG) and a calibration gap (CG), where the calibration is Calθs≔[KL(q(⋅∣s)∥pθ(⋅∣s))]\,Cal_θ^s [ KL (q(· s)\,\|\,p_θ(· s))]. boxprop[Snapshot vs. path-wise NLL gap] For any conditional predictors p0θ,snap(⋅∣s)p_0^θ,snap(· s) and p0θ,path(⋅∣ω)p_0^θ,path(· ω), ΔθNLL=H(x0∣s)−H(x0∣ω)⏟IPG≥0+Calθs−Calθω⏟CG. ^NLL_θ= H(x_0 s)-H(x_0 ω)_IPG≥ 0+ \,Cal_θ^s-\,Cal_θ^ω_CG. Moreover, argminθ[ℒx0snap(θ)]=argminθCalθs _θE[L_x_0^snap(θ)]\!=\! _θ\!\!\,Cal_θ^s, but argminθ[ℒx0path(θ)]≠argminθCalθω _θE[L_x_0^path(θ)]≠ _θ\!\!\,Cal_θ^ω in general. This decomposition exposes the core trade-off in replacing path-wise latents ω by snapshots s=(xt,t)s=(x_t,t). Discarding the full path induces an intrinsic information loss as IPG≥0IPG≥ 0, which can be compensated by the additional information in ω when CG≤0CG≤ 0. Here, snapshot-latent ELBOs offer a principled trade: they sacrifice some information for an objective that is better aligned with the architecture and easier to optimize, often yielding stronger generative models. In particular, minimizing the snapshot objective enforces μθ(xt,t)[x0]≈q(x0∣xt,t) _θ(x_t,t)_[x_0]≈ q(x_0 x_t,t), which empirically produces better samples as it approximates the correct quantity in Eq. 8.This distinction between path-wise and snapshot-latent training is illustrated in Fig. 3. Figure 3: Snapshot vs. path-wise training. The forward process corrupts the clean sequence “My name is David”. The blue path shows the beginning of the noising trajectory ω=(xtk(4),tk)k≥1 [rgb]0.05,0,0.8 [named]pgfstrokecolorrgb0.05,0,0.8ω= \( [rgb]0.05,0,0.8 [named]pgfstrokecolorrgb0.05,0,0.8x_t_k^(4), [rgb]0.05,0,0.8 [named]pgfstrokecolorrgb0.05,0,0.8t_k)\_k≥ 1 of one tracked position (ℓ=4 =4). Path-wise objectives condition on the entire trajectory ω [rgb]0.05,0,0.8 [named]pgfstrokecolorrgb0.05,0,0.8ω, whereas our GDDS snapshot objective uses only one random-time observation s=(xt⋆,t⋆) [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0s=( [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0x_t , [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0t ). 5 Experiments We evaluate GDDS on two complementary settings: language modeling and language generation. Additional results and details are deferred to Appendix C. We train character-level models on Text8 (mahoney2024text8) for 11M steps, and BPE-tokenized models on the widely used OpenWebText (OWT) dataset (gokaslan2019openwebtext) for 500500k steps. We compare small-model families under matched compute. All retrained models share the same Transformer backbone. Diffusion models use a DDiT backbone with bidirectional attention and time conditioning (peebles2023scalable) (≈96≈ 96M non-embedding parameters); the autoregressive (AR) baseline uses the same backbone with causal self-attention and no time conditioning (≈89≈ 89M non-embedding parameters). We retrain AR, a decoder-only Transformer trained with next-token cross-entropy, prior discrete diffusion baselines (UDLM & MDM) using their respective objectives (schiff2024simple; sahoo2024simple; shi2024simplified; ou2024your), and our GDDS snapshot framework with different forward noising processes: GDDS Absorb (masked), GDDS Uniform (uniform), and GDDS Gauss (semantic-informed; Section C.2). All reported GDDS Gauss results use the KNN implementation with k=64k=64 neighbors per token. Table 1: Bits Per Character (↓ ) on Text8. Baseline results reported from (shi2024simplified). All models are trained for 11M steps. Best results per model family are in bold, while best results among retrained models are underlined. Method BPC (↓ ) Continuous Diffusion Plaid (gulrajani2023likelihood) ≤ 1.48 BFN (graves2023bayesian) ≤ 1.41 Any-order Autoregressive ARDM (hoogeboom2021autoregressive) ≤ 1.43 MAC (shih2022training) ≤ 1.40 Autoregressive IAF/SCF (ziegler2019latent) 1.88 AR Argmax Flow (hoogeboom2021argmax) 1.39 Discrete Flow (tran2019discrete) 1.23 AR (austin2021structured) 1.23 !20AR (retrain) !201.35 Uniform Discrete Diffusion Mult. Diffusion (hoogeboom2021argmax) ≤ 1.72 D3PM Uniform (austin2021structured) ≤ 1.61 SEDD Uniform (lou2023discrete) ≤ 1.47 UDLM (schiff2024simple) ≤ 1.44 !20UDLM (retrain) !20≤ 1.67 !30GDDS Uniform (Ours) !30≤ 1.50 Masked Discrete Diffusion D3PM Absorb (austin2021structured) ≤ 1.45 SEDD Absorb (lou2023discrete) ≤ 1.39 GenMD4 (shi2024simplified) ≤ 1.34 MD4 (shi2024simplified) ≤ 1.37 !20MDM (retrain) !20≤ 1.58 !30GDDS Absorb (Ours) !30≤ 1.16 Semantic-Informed Kernel (SIK). Following Section 3, one can implement a variety of semantic-informed kernels that depend on the distance between tokens in the embedding space to noise according to words semantic similarities. Let e[x]e_[x] denote the embedding vector associated with token x. Here, we use a Gaussian SIK and, for x≠yx≠ y, set FtGauss(x,y):=exp(−‖e[x]−e[y]‖22/τ(t))∑z≠yexp(−‖e[z]−e[y]‖22/τ(t)),F_t^Gauss(x,y)\;:=\; \! (-\|e_[x]-e_[y]\|_2^2/τ(t) ) _z≠ y (-\|e_[z]-e_[y]\|_2^2/τ(t) ), and FtGauss(y,y)=0F_t^Gauss(y,y)=0, where τ(t)τ(t) is chosen to increase with t so that the kernel progressively flattens and the forward process approaches the uniform distribution as its limiting distribution. The associated noising algorithm can then be implemented efficiently through Algorithm 3 together with either a KNN or a KeOps implementation. In our experiments, GDDS Gauss is trained with the KNN instantiation using k=64k=64 neighbors per token; further implementation details and benchmarks are given in Section C.2. More general noising processes can be defined analogously and implemented efficiently thanks to the GDDS framework. 5.1 Language modeling We compute three metrics: Text8 BPC (bits per character) for models trained on Text8 in Table 1, OWT in-domain validation perplexity on the OWT validation split for OWT-trained models in Table 2, and OWT-trained zero-shot perplexity on validation sets of downstream tasks for OWT-trained models in Table 3. For autoregressive models, these metrics are computed from exact likelihood evaluation. For diffusion models, exact likelihoods are generally not available in closed form; whenever an ELBO is available, we report the corresponding variational upper bound on BPC/perplexity (denoted by ≤), using the ELBO associated with the objective the model was trained with. Table 2: OWT validation perplexity. Validation perplexity (↓ ) on OWT. Best results are in bold. ⋆Trained on the WebText dataset. †Result taken from (zhou2025next). ‡Result taken from (sahoo2025diffusion). Method Training token PPL (↓ ) Autoregressive GPT-2†⋆ (radford2019language) unknown 23.40 AR† 262B 16.11 !20AR (retrain) !20262B !2020.49 Uniform Discrete Diffusion SEDD Uniform‡ (lou2023discrete) 524B ≤ 29.70 Duo (sahoo2025diffusion) 524B ≤ 25.20 !20UDLM (retrain) !20262B !20≤ 36.82 !30GDDS Uniform (Ours) !30262B !30≤ 10.97 Masked Discrete Diffusion SEDD Absorb‡ (lou2023discrete) 524B ≤ 24.10 GenMD4 (shi2024simplified) 524B ≤ 21.80 MDLM (sahoo2024simple) 327B ≤ 23.00 !20MDM (retrain) !20262B !20≤ 31.03 !30GDDS Absorb (Ours) !30262B !30≤ 08.98 General Discrete Diffusion HDLM (zhou2025next) 131B ≤ 23.25 !30GDDS Gauss (Ours) !30262B !30≤ 07.65 Figure 4: OWT training curves. Evolution of OWT validation perplexity during training for the retrained models reported in Table 2. This complements the final numbers in Table 2 by showing the full optimization trajectory; both axes are shown on logarithmic scales. Across both Text8 and OWT, GDDS substantially improves over prior discrete diffusion baselines under matched compute. On Text8, GDDS Absorb outperforms the AR baseline for the first time. On OWT, GDDS yields dramatically tighter variational bounds than retrained UDLM/MDM, and can outperform the matched AR baseline (see Tables 2 and 4). Empirically, we often observe a negative expected NLL gap ΔθNLL<0 ^NLL_θ<0, meaning that snapshot objectives improve calibration enough (i.e., CG<−IPGCG<-IPG in Section 4.3). Moreover, because PPL=exp(NLL)PPL= (NLL) (see Section C.3), this reduction in NLL can induce a much more pronounced multiplicative drop in perplexity. We find that the choice of forward process matters: on OWT, semantic-informed noise (e.g., GDDS Gauss) tends to outperform Uniform/Mask by proposing semantically proximal corruptions that are easier to denoise and better aligned with language structure, consistent with prior evidence that semantic similarity improves discrete diffusion modeling (zhou2025next). Table 3: Zero-shot transfer perplexity. Zero-shot perplexity (↓ ) of OWT-trained models. Models are evaluated on validation splits of 77 downstream datasets without additional fine-tuning. Best result per dataset is in bold; second-best is underlined. PTB Wikitext103 LM1B Lambada AG News Pubmed Arxiv Autoregressive !20AR (retrain) !20147.90 !2042.91 !2081.29 !2075.93 !20105.34 !2079.93 !2076.29 Masked Discrete Diffusion (upper bounds ≤) !20MDM (retrain) !20181.36 !2045.42 !2092.58 !2058.89 !20123.51 !2066.46 !2056.82 !30GDDS Absorb !30103.04 !3046.49 !3092.51 !3060.78 !30101.41 !3062.62 !3053.27 Uniform Discrete Diffusion (upper bounds ≤) !20UDLM (retrain) !20177.26 !2064.65 !20112.49 !2070.38 !20153.89 !2070.78 !2060.35 !30GDDS Uniform !30115.12 !3042.63 !30108.83 !3068.92 !30136.24 !3071.69 !3058.83 General Discrete Diffusion (upper bounds ≤) !30GDDS Gauss !3053.65 !3034.56 !3046.06 !3038.74 !3045.94 !3031.78 !3028.49 Overall, GDDS improves transfer compared to prior diffusion baselines under matched compute, as shown in Figs. 1 and 3. In the masked setting, GDDS Absorb lowers zero-shot perplexity relative to the retrained MDM on most datasets, indicating better out-of-distribution generalization of the learned denoiser. In the uniform setting, GDDS Uniform yields large gains over the retrained UDLM across all datasets but one. Most notably, GDDS Gauss consistently outperforms all baselines by an important margin across every OOD dataset, suggesting that diffusion processes built from semantically structured corruptions may provide a clear generalization advantage. 5.2 Language generation We now turn to evaluation generative performance of our models. For numerical stability, we follow (zheng2024masked) and cast logits to float64 during sampling. We evaluate Ngen=256N_gen=256 unconditional samples from OWT-trained models. In Fig. 5, we consider the Gen-PPL/entropy tradeoff, and report the generative perplexity of unconditional samples under a fixed evaluator (GPT2-large; radford2019language) against their sequence entropy, for multiple decoding budgets K. As a reference point for this entropy scale, zheng2024masked report that natural OWT text typically falls in the range 5.605.60–5.705.70. Figure 5: Generation quality-diversity tradeoff. Gen-PPL (↓ ) vs Entropy tradeoff. For K∈32,64,128,256,512,1024K∈\32,64,128,256,512,1024\ decoding steps, we plot the generative perplexity of Ngen=256N_gen=256 unconditional samples under a fixed evaluator (GPT2-large) against their sequence entropy (higher is better). Bubble radius increases with K. For reference, the AR baseline achieves Gen-PPL 56.8256.82 at entropy 5.605.60. Two consistent patterns emerge. First, UDLM attains low Gen-PPL but remains stuck at noticeably lower entropy, suggesting conservative generations with limited diversity. Second, MDM increases entropy as K grows, but this comes with a steep degradation in Gen-PPL at larger budgets, indicating that pushing diversity by running more steps can quickly harm sample quality. In contrast, GDDS improves the Pareto tradeoff under matched compute. GDDS Uniform shifts the uniform-diffusion frontier toward higher entropy while keeping Gen-PPL competitive, mitigating the low-diversity behavior of UDLM. More strikingly, GDDS Absorb yields a strictly better quality/diversity compromise than MDM in the highlighted regime: for comparable entropy, it achieves lower Gen-PPL, and reaches favorable points with substantially fewer decoding steps (smaller bubbles). In particular, at K=64K=64 decoding steps, GDDS Absorb attains a lower Gen-PPL at essentially the same entropy as MDM even when the latter is run with up to K=1024K=1024 decoding steps, highlighting a large gain in sampling efficiency. Table 4: Lexical diversity. Distinct-1/2/3 (↑ ) computed on Ngen=256N_gen=256 unconditional samples from OWT-trained models, measuring the fraction of unique n-grams among generated texts. Model Dist-1 (↑ ) Dist-2 (↑ ) Dist-3 (↑ ) !20AR (retrain) 0.10 0.57 0.88 !20MDM (retrain) 0.10 0.61 0.90 !30GDDS Absorb 0.10 0.60 0.89 !20UDLM (retrain) 0.08 0.53 0.85 !30GDDS Uniform 0.10 0.58 0.88 Next in Table 4, we consider judge-free quality metrics that do not rely on GPT2-large scoring, and report a diversity statistics (Distinct-n). This metric measures the fraction of unique n-grams in generated samples (higher is more diverse). Consistent with the low-entropy cluster in Fig. 5, UDLM exhibits the lowest lexical diversity (Distinct-1/2/3). GDDS Uniform increasing Distinct-1/2/3 over UDLM and matching the strong diversity of masked methods. Overall, Distinct-n corroborates the Pareto analysis: GDDS increases diversity without incurring the large Gen-PPL penalties observed for MDM at high decoding budgets. 6 Conclusion We introduced Generalized Discrete Diffusion from Snapshots (GDDS), a framework for discrete diffusion models that enables efficient noising processes with arbitrary rate matrix. Our training algorithm relies on a simple loss function based on snapshot samples instead of the entire noising path, and is compatible with standard architectures. GDDS beats previous discrete diffusion models as well as autoregressive models for the first time at this scale on language modeling tasks. Future work may build upon this approach and propose different Semantic-Informed Kernels (SIK) to enforce meaningful noising processes based on similarities between words. Impact Statement This paper presents work that aims to advance discrete diffusion-based generative modeling in machine learning. There are many potential societal consequences of our work, none of which must be specifically highlighted here. References Appendix Throughout the appendix, non-bold symbols refer to token-level objects on V, while bold symbols refer to their sequence-level counterparts on nV^n for n≥1n≥ 1. Appendix A Time-dependent matrix exponential A.1 Transition operator for time-inhomogeneous CTMCs Let t≥0t≥ 0, and consider two real d×d× d matrices Q, QtQ_t to be time-independent and time-dependent. The matrix exponential of Q is defined by the power series: exp(Q)=∑k=0∞Qk/k! (Q)= _k=0^∞Q^k/k!. When QtQ_t varies with time, one naturally extends this notion by considering the matrix Kt,sK_t,s with 0≤s≤t0≤ s≤ t that solves the following matrix-valued linear ODE: dtKt,s=QtKt,s,Ks,s=Im. \,d\,dtK_t,s=Q_tK_t,s, K_s,s=I_m. Its solution can be written using the time-ordered exponential (or Peano–Baker series) (brockett2015finite, Thm 1.3.1), Kt,s=exp(∫stQτdτ),K_t,s=T ( _s^tQ_τ\,dτ ), (10) where T is a time-ordering operator that orders products by decreasing time (later times to the left) as exp(∫stQτdτ)=Id+∫stQτ1dτ1+∫st∫sτ1Qτ1Qτ2dτ2dτ1+∫st∫sτ1∫sτ2Qτ1Qτ2Qτ3dτ3dτ2dτ1+⋯,T ( _s^tQ_τ\,dτ )=I_d+ _s^tQ_ _1\,d _1+ _s^t _s _1Q_ _1Q_ _2\,d _2\,d _1+ _s^t _s _1 _s _2Q_ _1Q_ _2Q_ _3\,d _3\,d _2\,d _1+·s, i.e., the k-fold term integrates over s≤τk≤⋯≤τ1≤ts≤ _k≤·s≤ _1≤ t the ordered product Qτ1⋯QτkQ_ _1·s Q_ _k. Note that if the family Qτ∈[s,t]\Q_τ\_τ∈[s,t] commutes pairwise (QτQτ′=Qτ′QτQ_τQ_τ =Q_τ Q_τ for all τ,τ′∈[s,t]τ,τ ∈[s,t]), time ordering is unnecessary and Kt,s=exp(∫stQτdτ)K_t,s= ( _s^tQ_τ\,dτ). The time-ordered exponential defined in Eq. 10 is useful to describe the fundamental solution to the Kolmogorov forward equation (see Eq. 3), which describes the time evolution of a probability vector qtq_t governed by a (possibly time-dependent) rate matrix QtQ_t. More precisely, the matrix Kt,sK_t,s acts as the linear operator that maps an initial distribution qsq_s to its future value qtq_t as qt=Kt,sqs=exp(∫stQτdτ)qs.q_t=K_t,sq_s=T ( _s^tQ_τ\,dτ )q_s. For any initial condition x∈x , the evolution of this point mass under the forward dynamics is obtained by considering q0=δx∈ℝmq_0= _x ^m at s=0s=0, i.e., qt(⋅∣x)=Kt,0δxq_t(· x)=K_t,0 _x. Hence, the x-th column of Kt,sK_t,s corresponds exactly to the conditional distribution of the CTMC transition matrix at time t given that it was at state x at time s, that is, Kt,s(y,x)=ℙ(xt=y∣xs=x).K_t,s(y,x)=P(x_t=y x_s=x). In this sense, the time-ordered exponential provides the explicit operator representation of the transition kernel solving the Kolmogorov forward equation. A.2 Coordinate-wise rate matrix and product-form marginals Recall that states are sequences =x1…xn∈nx=x^1… x^n ^n, and that jumps only modify one token at a time with token-level generator Qtℓ∈ℝm×mQ_t ^m× m at position ℓ∈1,…,n ∈\1,…,n\. Then, the joint rate matrix decomposes as a sum of coordinate-wise generators: t=∑ℓ=1n(Im⊗⋯⊗Qtℓ⊗⋯⊗Im)∈ℝmn×mn,Q_t= _ =1^n (I_m ·s Q_t ·s I_m ) ^m^n× m^n, where Qtℓ=QtQ_t =Q_t for all 1≤ℓ≤n1≤ ≤ n acts on coordinate ℓ and identities on the others (pauline2025foundations). Here, A⊗BA B denotes the Kronecker product of matrices A and B. This structure implies that each coordinate (xtℓ)1≤ℓ≤n(x_t )_1≤ ≤ n evolves as an independent time-inhomogeneous CTMC with rate matrix QtQ_t. In fact, let qt(⋅∣xℓ)q_t(· x ) solve the Kolmogorov forward equation: dtqt(⋅∣xℓ)=Qtqt(⋅∣xℓ),q0(⋅∣xℓ)=δxℓ. \,d\,dtq_t(· x )=Q_tq_t(· x ), q_0(· x )= _x . Define ~t(⋅∣):=∏ℓ=1nqt(⋅∣xℓ) q_t(· ):= _ =1^nq_t(· x ). By the product rule and the generator decomposition above, ~t q_t solves the joint forward equation dt~t(⋅∣)=t~t(⋅∣),~0(⋅∣)=δ. \,d\,dt q_t(· )=Q_t q_t(· ), q_0(· )= _x. Uniqueness of solutions to the forward equation Eq. 3 yields t(⋅∣)=~t(⋅∣)=∏ℓ=1nqt(⋅∣xℓ) q_t(· )= q_t(· )= _ =1^nq_t(· x ), i.e., the conditional marginal factorizes across coordinates. Appendix B Technical results B.1 CTMC rate matrices Proof of Section 3.1. Let t∈(0,1)t∈(0,1) such that KtK_t is invertible. For each x∈x , we denote by v(t,x)v^(t,x) the solution to linear system Kt⊤v(t,x)=K˙t⊤δxK_t v^(t,x)= K_t _x, which exists and is unique. We define a matrix Qt∈ℝm×mQ_t ^m× m by prescribing its x-th column as Qt⊤δx:=v(t,x),for all x∈.Q_t _x:=v^(t,x), all x . By construction, for every x∈x , we have K˙t⊤δx=Kt⊤(Qt⊤δx), K_t _x=K_t (Q_t _x), hence K˙t⊤=Kt⊤Qt⊤ K_t =K_t Q_t , which is equivalent to QtKt=K˙tQ_tK_t= K_t hence Qt=K˙tKt−1Q_t= K_tK_t^-1 since KtK_t is invertible. For any x≠y∈x≠ y , we have Qt(y,x)=(Qt⊤δx)[y]=v[y](t,x)≥0.Q_t(y,x)=(Q_t _x)_[y]=v^(t,x)_[y]≥ 0. Therefore, the off–diagonal entries of QtQ_t are nonnegative. Since Πt _t is column–stochastic and 0≤αt≤10≤ _t≤ 1, each KtK_t is column–stochastic: ⊤Kt=⊤(αtIm+(1−αt)Πt)=⊤1 K_t=1 ( _tI_m+(1- _t) _t)=1 . Differentiating with respect to t gives ⊤K˙t=01 K_t=0. Using the identity K˙t=KtQt K_t=K_tQ_t, we have 0=⊤K˙t=⊤KtQt=⊤Qt0=1 K_t=1 K_tQ_t=1 Q_t, because ⊤Kt=⊤1 K_t=1 . Hence every column of QtQ_t sums to 0. Consequently, for each x, Qt(x,x)=−∑y≠xQt(y,x)≤0.Q_t(x,x)=- _y≠ xQ_t(y,x)≤ 0. Let px(t):=Ktδxp_x(t):=K_t _x be the law at time t when starting from state x at time 0. Then, dtpx(t)=K˙tδx=QtKtδx=Qtpx(t), \,d\,dtp_x(t)= K_t _x=Q_tK_t _x=Q_tp_x(t), with px(0)=K0δx=δxp_x(0)=K_0 _x= _x because α0=1 _0=1 implies K0=ImK_0=I_m. Thus, QtQ_t drives the Kolmogorov forward equation, and since its off–diagonals are nonnegative and its columns sum to zero, QtQ_t is a valid CTMC rate matrix. Combining all previous steps yields the claimed expression Qt=K˙tKt−1Q_t= K_tK_t^-1. ∎ boxcor [GIDD case Πt=πt⊤ _t= _t1 , von2025generalized] Let t≥0,πt∈Δmt≥ 0, _t∈ _m and select Πt=πt⊤∈ℝd×d _t= _t1 ^d× d, then Qt=α˙tαtIm−α˙tαtπt⊤+(1−αt)π˙t⊤.Q_t= α_t _tI_m- α_t _t _t1 +(1- _t) π_t1 . Proof. We begin by verifying that our regularity assumption on KtK_t holds. Following the choice Πt=πt⊤ _t= _t1 , we have for all t∈[0,1]t∈[0,1], Kt=αtIm+(1−αt)πt⊤,andK˙t=α˙tIm−α˙tπt⊤+(1−αt)πt˙⊤.K_t= _tI_m+(1- _t) _t1 , K_t= α_tI_m- α_t _t1 +(1- _t) _t1 . Since KtK_t is a rank-11 update to an invertible matrix, the Sherman–Morrison formula (bartlett1951inverse) ensures that it is invertible and yields the following closed form for its inverse: Kt−1=1αt(Im−(1−αt)πt⊤αt(1+1−αtαt))=1αt(Im−(1−αt)πt⊤),t∈(0,1).K_t^-1= 1 _t (I_m- (1- _t) _t1 _t(1+ 1- _t _t) )= 1 _t(I_m-(1- _t) _t1 ), t∈(0,1). (11) Moreover for all x∈x and for all t∈(0,1)t∈(0,1), one has to verify that v(t,x)=(Kt⊤)−1K˙t⊤δx≥0v^(t,x)=(K_t )^-1 K_t _x≥ 0, i.e. that (1−αt)π˙t(x)−αt˙αtπt(x)≥0(1- _t) π_t(x)- _t _t _t(x)≥ 0, which is true since αt˙≤0 _t≤ 0 because αt _t is a decreasing function. Then, combining Sections 3.1 and 11 yields Qt Q_t =K˙tKt−1=(α˙tIm−α˙tπt⊤+(1−αt)π˙t⊤)1αt(Im−(1−αt)πt⊤) = K_tK_t^-1= ( α_tI_m- α_t _t1 +(1- _t) π_t1 ) 1 _t (I_m-(1- _t) _t1 ) =α˙tαt(Im−(1−αt)πt⊤)−α˙tαt(πt⊤−(1−αt)πt⊤πt⊤)+1−αtαt(π˙t⊤−(1−αt)π˙t⊤πt⊤). = α_t _t (I_m-(1- _t) _t1 )- α_t _t ( _t1 -(1- _t) _t1 _t1 )+ 1- _t _t ( π_t1 -(1- _t) π_t1 _t1 ). Using the identity ⊤πt=11 _t=1, the previous equality simplifies to Qt=α˙tαtIm−α˙tαtπt⊤+(1−αt)π˙t⊤.Q_t= α_t _tI_m- α_t _t _t1 +(1- _t) π_t1 . ∎ Proof of Section 3.1. Following the definition of KtK_t in Eq. 6, for any t∈[0,1]t∈[0,1], Kt=αtIm+(1−αt)ΠtK_t= _tI_m+(1- _t) _t. Differentiating the equation with respect to t yields: K˙t=α˙tIm+(1−αt)Π˙t−α˙tΠt=(1−αt)Π˙t+α˙t(Im−Πt). K_t= α_tI_m+(1- _t) _t- α_t _t=(1- _t) _t+ α_t(I_m- _t). (12) Let Πt _t be the matrix defined as the unique solution to the following linear matrix-valued ODE: Π0=Im−Q0α˙0,and(1−αt)Π˙t=(αtQt−α˙tIm)(Im−Πt)+QtΠt,t∈[0,1], _0=I_m- Q_0 α_0, (1- _t) _t=( _tQ_t- α_tI_m)(I_m- _t)+Q_t _t, t∈[0,1], which is valid since 0<1−αt≤10<1- _t≤ 1 for all t∈(0,1]t∈(0,1]. Inserting the ODE into Eq. 12 yields K˙t K_t =(1−αt)Π˙t+α˙t(Im−Πt)=(αtQt−α˙tIm)(Im−Πt)+QtΠt+α˙t(Im−Πt) =(1- _t) _t+ α_t(I_m- _t)=( _tQ_t- α_tI_m)(I_m- _t)+Q_t _t+ α_t(I_m- _t) =αtQt−αtQtΠt−α˙tIm+α˙tΠt+QtΠt+α˙tIm−α˙tΠt = _tQ_t- _tQ_t _t- α_tI_m+ α_t _t+Q_t _t+ α_tI_m- α_t _t =αtQt+(1−αt)QtΠt=Qt(αtIm+(1−αt)Πt)=QtKt. = _tQ_t+(1- _t)Q_t _t=Q_t( _tI_m+(1- _t) _t)=Q_tK_t. For the initial condition, using the Taylor expansion αt=1+α˙0t+o(t) _t=1+ α_0t+o(t) as t↓0t 0, we obtain limt↓0Kt=limt↓0(αtIm+(1−αt)Πt)=Im. _t 0K_t= _t 0( _tI_m+(1- _t) _t)=I_m. Hence, for t∈[0,1]t∈[0,1], KtK_t satisfies the linear matrix-valued ODE: K˙t=QtKt,K0=Im, K_t=Q_tK_t, K_0=I_m, whose unique solution is Kt=exp(∫0tQsds)K_t=T ( _0^tQ_s\,ds) for all t∈[0,1]t∈[0,1] (brockett2015finite, Theorem 1.3.1). Taking the column associated to the token x on both sides yields Kt(⋅,x)=exp(∫0tQsds)(⋅,x)K_t(·,x)=T ( _0^tQ_s\,ds)(·,x), so the induced marginals coincide with those of the CTMC. ∎ Remark B.1. Note that we never explicitly use the expression Π0=Im−Q0α˙0 _0=I_m- Q_0 α_0 in the proof. We only require Π0 _0 to be finite; the above choice merely ensures that Πt _t is uniquely defined as a continuous mixing matrix on [0,1][0,1]. In fact, we set Π0=Im−Q0α˙0 _0=I_m- Q_0 α_0 because 1−αt→01- _t→ 0 makes the coefficient of Π˙t _t in the ODE vanish at t=0t=0, and keeping Π˙t _t finite requires (Q0−α˙0Im)+α˙0Π0=0(Q_0- α_0I_m)+ α_0 _0=0. B.2 Uniformization and exact sampling Proof of Section 3.2. Let (Zt)t≥0(Z_t)_t≥ 0 be a CTMC with rate matrix Qt=f(t)(Ft−Im)Q_t=f(t)(F_t-I_m), started at Z0=xZ_0=x. Let NtN_t be its jump count process. Here, NtN_t is a non-homogeneous Poisson point process with parameter f¯(t)=∫0tf(s)ds f(t)= _0^tf(s)\,ds with marginal distribution ℙ(Nt=k)=exp(−f¯(t))f¯(t)k/k!P(N_t=k)= (- f(t)) f(t)^k/k! for all k∈ℕk . Conditionally on the event (Nt=k)(N_t=k), let 0<T1<…<Tk<t0<T_1<…<T_k<t denote the jump times. For each 1≤r≤k1≤ r≤ k, the transition probabilities at a jump are given by the columns of FTrF_T_r, namely ℙ(ZTr=i∣ZTr−1=j,Tr)=FTr(i,j)P(Z_T_r=i Z_T_r-1=j,T_r)=F_T_r(i,j). Therefore, for any i,j∈1,…,mi,j∈\1,…,m\, ℙ(Zt=i,Nt=k∣Z0=x)=[(FTk…FT1)(i,j)Nt=k∣Z0=j].P(Z_t=i,N_t=k Z_0=x)=E [(F_T_k… F_T_1)(i,j)1_N_t=k Z_0=j ]. Summing over k≥0k≥ 0 yields Kt(i,j)=ℙ(Zt=i∣Z0=x)=∑k≥0[(FTk…FT1)(i,j)Nt=k],K_t(i,j)=P(Z_t=i Z_0=x)= _k≥ 0E [(F_T_k… F_T_1)(i,j)1_N_t=k ], with the convention that FT0…FT1F_T_0… F_T_1 is the identity matrix when k=0k=0. Splitting the terms at k=0k=0 yields Kt(i,j)=ℙ(Nt=0)i=j+∑k≥1[(FTk…FT1)(i,j)Nt=k],K_t(i,j)=P(N_t=0)1_i=j+ _k≥ 1E [(F_T_k… F_T_1)(i,j)1_N_t=k ], Since ℙ(Nt=0)=αtP(N_t=0)= _t, we identify Kt(i,j)=αti=j+(1−αt)Πt(i,j)K_t(i,j)= _t1_i=j+(1- _t) _t(i,j), where Πt(i,j):=11−αt∑k≥1[(FTk…FT1)(i,j)Nt=k]=[(FTNt…FT1)(i,j)∣Nt≥1], _t(i,j):= 11- _t _k≥ 1E [(F_T_k… F_T_1)(i,j)1_N_t=k ]=E [(F_T_N_t… F_T_1)(i,j) N_t≥ 1 ], as ℙ(Nt≥1)=1−exp(−f¯(t))=1−αtP(N_t≥ 1)=1- (- f(t))=1- _t. Note that Πt _t is a correctly defined mixing matrix. Indeed, for a fixed k and any time sequence 0<t1<…<tn<t0<t_1<…<t_n<t, each FTrF_T_r is column stochastic and the product FTk…FT1F_T_k… F_T_1 as well. Hence, the random matrix FTNt…FT1F_T_N_t… F_T_1 conditioned on (Nt≥1)(N_t≥ 1) is column stochastic almost surely. Taking the conditional expectation (as a convex combination) preserves this property. Comparing this expression with the interpolating form Kt=αtIm+(1−αt)ΠtK_t= _tI_m+(1- _t) _t yields the unique Πt=[(FTNt…FT1)∣Nt≥1], _t=E [(F_T_N_t… F_T_1) N_t≥ 1 ], as claimed, where uniqueness follows from Section 3.1. ∎ Remark B.2 (Exact sampling of xt∼qt(⋅∣x0)x_t q_t(· x_0)). For a given continuous time t∈[0,1]t∈[0,1], the sampling of xt∼qt(⋅∣x0)x_t q_t(· x_0) can be performed exactly using only a Poisson random variable sampling, uniform random variable sampling, and columns of FtF_t as follows. 1. Sample Nt∼Poisson(f¯(t))N_t ( f(t)). 2. If Nt=0N_t=0, set xt←x0x_t← x_0. 3. If Nt≥1N_t≥ 1, set z0←xz_0← x then: (a) sample ordered jump time 0<T1<…<TNt≤t0<T_1<…<T_N_t≤ t according to the Poisson point process with parameter f¯(t) f(t). This can be done by sampling i.i.d. Ur∼([0,1])U_r ([0,1]), set Vr=f¯−1(Urf¯(t))V_r= f^-1(U_r f(t)) and take the ordered statistics of V1,…,VNV_1,…,V_N as T1,…,TNtT_1,…,T_N_t. (b) for each 1≤r≤Nt1≤ r≤ N_t, sample the next state zrz_r using the zr−1z_r-1-th column of FtF_t as ℙ(zr=i∣zr−1)=FTr(i,zr−1)for i=1,…,m.P(z_r=i z_r-1)=F_T_r(i,z_r-1) i=1,…,m. (c) Set xt←zNtx_t← z_N_t. This algorithm produces the conditional law ℙ(xt=x∣x0=y,Nt=k,T1=t1,…,Tk=tk)=Ftk…Ft1(x,y)P(x_t=x x_0=y,N_t=k,T_1=t_1,…,T_k=t_k)=F_t_k… F_t_1(x,y). Marginalizing over (Nt,T1,…,TNt)(N_t,T_1,…,T_N_t) yields ℙ(xt=x∣x0=y)=∑k≥0[(FTk…FT1)(x,y)Nt=k]=Kt(x,y).P(x_t=x x_0=y)= _k≥ 0E [(F_T_k… F_T_1)(x,y)1_N_t=k ]=K_t(x,y). Note that this procedure is exact. Algorithm 3 Exact general noising, sequence level (parallel token level) 1: Input: clean sequence 0=x01…x0nx_0=x_0^1… x_0^n of length n≥1n≥ 1, time t∈[0,1]t∈[0,1], intensity f¯(t) f(t), rate matrix QtQ_t as in Eq. 7 2: for ℓ=1 =1 to n in parallel do 3: Set z0ℓ←x0ℓz_0 ← x_0 4: Sample number of jumps Ntℓ∼Poisson(f¯(t))N_t ( f(t)) 5: Sample and sort the jump times as T1ℓ<…<TNtℓT_1 <…<T_N_t 6: for k=1k=1 to Ntℓ N_t do 7: Sample jump zkℓ∼FTkℓ(⋅,zk−1ℓ)z_k F_T_k (·,z_k-1 ) 8: end for 9: Set xtℓ←zNtℓx_t ← z_N_t 10: end for 11: return noised sequence t=xt1…xtnx_t=x_t^1… x_t^n and per-token jumps (Tkℓ,zkℓ)k=1Ntℓ=1n \(T_k ,z_k )_k=1^N_t \_ =1^n B.3 Path-wise parametrization and Evidence Lower Bound (ELBO) B.3.1 Reverse process Forward kernels and true reverse conditionals. Fix 0≤u<t≤10≤ u<t≤ 1 and consider a time-inhomogeneous forward Markov process: x0→xu→xt,whereq(x0,xu,xt)=qdata(x0)qu(xu∣x0)qt∣u(xt∣xu),x_0→ x_u→ x_t, q(x_0,x_u,x_t)=q_data(x_0)q_u(x_u x_0)q_t u(x_t x_u), and qt∣uq_t u denotes the forward transition kernel from time u to t (we use the shorthand qu:=qu∣0q_u:=q_u 0). The forward marginal from x0x_0 to time t is qt(xt∣x0)=∑xu∈qt∣u(xt∣xu)qu(xu∣x0).q_t(x_t x_0)= _x_u q_t u(x_t x_u)q_u(x_u x_0). (13) By Bayes’ rule, the true reverse conditional given x0x_0 reads q(xu∣xt,x0)=qt∣u(xt∣xu,x0)qu(xu∣x0)∑xu∈qt∣u(xt∣xu)qu(xu∣x0)=qt∣u(xt∣xu)qu(xu∣x0)qt(xt∣x0),q(x_u x_t,x_0)= q_t u(x_t x_u,x_0)q_u(x_u x_0) _x_u q_t u(x_t x_u)q_u(x_u x_0)= q_t u(x_t x_u)q_u(x_u x_0)q_t(x_t x_0), (14) where the second equality uses that the forward process is Markov hence qt∣u(xt∣xu,x0)=qt∣u(xt∣xu)q_t u(x_t x_u,x_0)=q_t u(x_t x_u) and the marginalization (13). Finally, conditioning on the observed endpoint xtx_t, the true reverse kernel is the mixture q(xu∣xt)=∑x0∈q(xu∣xt,x0)q0∣t(x0∣xt),q(x_u x_t)= _x_0 q(x_u x_t,x_0)q_0 t(x_0 x_t), (15) where q0∣t(x0∣xt)q_0 t(x_0 x_t) is the exact forward posterior of x0x_0 given the snapshot s=(xt,t)s=(x_t,t). Plug-in Bayes and realizability. At inference time x0x_0 is unknown while (xt,t)(x_t,t) is observed. Let μθ(⋅∣xt,t) _θ(· x_t,t) be a neural predictor (e.g. a transformer head; vaswani2017attention) that outputs a distribution over x0∈x_0 given (xt,t)(x_t,t). We define the plug-in reverse transition by mixing the exact conditional (14) with μθ _θ as pθ(xu∣xt,t):=∑x0∈q(xu∣xt,x0)μθ(x0∣xt,t).p_θ(x_u x_t,t):= _x_0 q(x_u x_t,x_0) _θ(x_0 x_t,t). (16) This parametrization is widely used in the literature (austin2021structured; campbell2022continuous; sahoo2024simple; shi2024simplified; von2025generalized; zhou2025next). It is interesting, as it verifies the following lemma. boxlem [Realizable limit] If μθ(⋅∣xt,t)=q0∣t(⋅∣xt) _θ(· x_t,t)=q_0 t(· x_t), then pθ(⋅∣xt,t)p_θ(· x_t,t) equals the true reverse kernel: pθ(xu∣xt,t)=q(xu∣xt).p_θ(x_u x_t,t)=q(x_u x_t). Proof. If μθ(⋅∣xt,t)=q0∣t(⋅∣xt) _θ(· x_t,t)=q_0 t(· x_t), then substituting into Eq. 16 gives pθ(xu∣xt,t)=∑x0∈q(xu∣xt,x0)q0∣t(x0∣xt)=q(xu∣xt),p_θ(x_u x_t,t)= _x_0 q(x_u x_t,x_0)q_0 t(x_0 x_t)=q(x_u x_t), where the last equality is exactly (15). ∎ A first crucial insight is that, μθ(⋅∣xt,t) _θ(· x_t,t) should be train to approach the posterior q0∣t(⋅∣xt)q_0 t(· x_t), so that pθ(xu∣xt,t)p_θ(x_u x_t,t) approaches q(xu∣xt)q(x_u x_t). CTMC view and factorization of the reverse dynamics. For an infinitesimal step u=t−ϵu=t-ε with ϵ↓0ε 0, the true reverse transition admits the standard “no-jump + jump” expansion: q(xt−ϵ∣xt)=(1−ϵr[xt](t))δxt+ϵr[xt](t)Rt(⋅,xt)+o(ϵ),q(x_t-ε x_t)=(1-ε r_[x_t](t)) _x_t+ε r_[x_t](t)R_t(·,x_t)+o(ε), where r[xt](t)r_[x_t](t) is the true reverse exit rate from state xtx_t and Rt(⋅,xt)∈ΔmR_t(·,x_t)∈ _m is the true jump destination distribution. Equivalently, the reverse generator factorizes as Q¯t=(Rt−Im)diag(r1(t),…,rm(t)), Q_t=(R_t-I_m)diag(r_1(t),…,r_m(t)), i.e., Q¯t(y,x)=r[x](t)Rt(y,x),y≠x,−r[x](t),y=x. Q_t(y,x)= casesr_[x](t)R_t(y,x),&y≠ x,\\ -r_[x](t),&y=x. cases This factorization separates when the chain jumps (through r[x](t)r_[x](t)) from where it jumps (through RtR_t). Why the plug-in Bayes kernel entangles “where” and “when”. Eq. 16 induces a time-inhomogeneous reverse kernel whose short-time behaviour is governed by a θ-dependent generator. Indeed, for an infinitesimal step u=t−ϵu=t-ε with ϵ↓0ε 0, the induced kernel pθ(xt−ϵ∣xt,t)p_θ(x_t-ε x_t,t) admits a first-order expansion of the form pθ(xt−ϵ∣xt,t)=(1−ϵr[xt]θ(t))δxt+ϵr[xt]θ(t)Atθ(⋅,xt)+o(ϵ),p_θ(x_t-ε x_t,t)=(1-ε r_[x_t]^θ(t)) _x_t+ε r_[x_t]^θ(t)A_t^θ(·,x_t)+o(ε), where both the effective exit rate r[xt]θ(t)r_[x_t]^θ(t) and the jump destination distribution Atθ(⋅,xt)∈ΔmA_t^θ(·,x_t)∈ _m depend on θ through μθ(⋅∣xt,t) _θ(· x_t,t). Equivalently, the induced reverse generator Q¯tθ Q_t^θ factorizes as Q¯tθ=(Atθ−Im)diag(r1θ(t),…,rmθ(t)), Q_t^θ=(A_t^θ-I_m)diag(r_1^θ(t),…,r_m^θ(t)), (17) so that learning μθ _θ implicitly learns both a jump kernel and a time-dependent clock. In other words, the Bayes plug-in construction couples where the chain jumps (through Atθ(⋅,xt)A_t^θ(·,x_t)) and when it jumps (through r[xt]θ(t)r_[x_t]^θ(t)), which complicates the path-wise ELBO and its optimization: the event-level objective contains gradients through both the destination cross-entropy term and the rate/normalization term, rather than isolating a single θ-dependent jump component. Jump-states parametrization: learn only the jump kernel, fix the exit rates. To mirror the true CTMC factorization, we instead parameterize only the jump destinations while prescribing the exit rates by a diffusion schedule. Concretely, we choose a nonnegative schedule r[x](t)≥0r_[x](t)≥ 0 independently of θ and define the neural reverse generator Q¯tθ=(Rtθ−Im)diag(r1(t),…,rm(t)),whereRtθ(y,x)=θ(x,t)[y], Q^θ_t=(R_t^θ-I_m)diag(r_1(t),…,r_m(t)), R_t^θ(y,x)= j_θ(x,t)_[y], (18) such that Q¯tθ(y,x)=r[x](t)θ(x,t)[y],y≠x,−r[x](t),y=x. Q^θ_t(y,x)= casesr_[x](t) j_θ(x,t)_[y],&y≠ x,\\ -r_[x](t),&y=x. cases Equivalently, for u=t−ϵu=t-ε the associated reverse kernel satisfies the first-order expansion: pθ(xt−ϵ∣xt)=(1−ϵr[xt](t))δxt+ϵr[xt](t)θ(xt,t)xt−ϵ+o(ϵ).p_θ(x_t-ε x_t)=(1-ε r_[x_t](t)) _x_t+ε r_[x_t](t) j_θ(x_t,t)_x_t-ε+o(ε). Compared to the plug-in Bayes transition in Eq. 16, the network now controls only where the chain jumps (via θ(xt,t) j_θ(x_t,t)), while when it jumps is entirely prescribed by the schedule r[xt](t)r_[x_t](t). B.3.2 Proof of Section 4.2 The following proposition is valid for any general forward CTMC and its parametrized time reversal. It has been established and rewritten in several previous works (e.g., campbell2022continuous; shi2024simplified; von2025generalized; zhou2025next), and we restate it with our notations. boxprop [General Path-wise ELBO, campbell2022continuous] Let x0∈x_0 , the ELBO is given by logp0θ,path(x0)≥−ℒ~x0path(θ)+C~x0path p_0^θ,path(x_0)≥- L_x_0^path(θ)+ C^path_x_0, where C~x0path C^path_x_0 is independent of θ and ℒ~x0path(θ)=∫01xt∼qt(⋅∣x0)[−Q¯tθ(xt,xt)+∑y≠xtQt(y,xt)(−logRtθ(xt,y))]dt. L_x_0^path(θ)= _0^1E_x_t q_t(· x_0) [- Q_t^θ(x_t,x_t)+ _y≠ x_tQ_t(y,x_t)(- R_t^θ(x_t,y)) ]\,dt. Let us apply Section B.3.2 to our jump-states parametrization detailed in Section 4.2. In our case, Q¯tθ(xt,xt)=−r[xt](t) Q_t^θ(x_t,x_t)=-r_[x_t](t) is θ-independent (see Eq. 9), we can discard it from the θ-dependent part, and define ℒx0path(θ):=∫01xt∼qt(⋅∣x0)[∑y≠xtQt(y,xt)(−logRtθ(xt,y))]dt,L_x_0^path(θ):= _0^1E_x_t q_t(· x_0) [ _y≠ x_tQ_t(y,x_t)(- R_t^θ(x_t,y)) ]\,dt, (19) and Cx0path:=C~x0path−∫01xt∼qt(⋅∣x0)[r[xt](t)]dtC^path_x_0:= C^path_x_0- _0^1E_x_t q_t(· x_0)[r_[x_t](t)]\,dt. We can then rewrite the ELBO as logp0θ,path(x0)≥−ℒx0path(θ)+Cx0path p_0^θ,path(x_0)≥-L_x_0^path(θ)+C^path_x_0. We now define the following quantities similarly to (shi2024simplified, Lem. 2). boxdef [True conditional reverse jump kernel Rtx0R_t^x_0 and conditional reverse rate rxx0(t)r^x_0_x(t)] Fix x0∈x_0 and let qt(⋅∣x0)q_t(· x_0) be the forward marginal at time t. For x∈x with qt(x∣x0)>0q_t(x x_0)>0, define the conditional reverse jump kernel Rtx0(⋅,x)R_t^x_0(·,x) by, for all y≠xy≠ x, Rtx0(y,x):=qt(y∣x0)qt(x∣x0)Qt(x,y)r[x]x0(t),where r[x]x0(t):=∑y≠xqt(y∣x0)qt(x∣x0)Qt(x,y).R_t^x_0(y,x):= q_t(y x_0)q_t(x x_0) Q_t(x,y)r_[x]^x_0(t), r_[x]^x_0(t):= _y≠ x q_t(y x_0)q_t(x x_0)Q_t(x,y). If qt(x∣x0)=0q_t(x x_0)=0, the value of Rtx0(⋅,x)R_t^x_0(·,x) is irrelevant under xt∼qt(⋅∣x0)x_t q_t(· x_0) and is set arbitrarily to 0. To conclude the proof, we provide the following lemma which gives two clean rewriting of the θ-dependent part ℒx0path(θ)L_x_0^path(θ). boxlem The quantity ℒx0path(θ)L_x_0^path(θ) defined in Eq. 19 satisfies the following identities: ℒx0path(θ) _x_0^path(θ) =∫01xt∼qt(⋅∣x0)[∑y≠xtqt(y∣x0)qt(xt∣x0)Qt(xt,y)(−logRtθ(y,xt))]dt = _0^1E_x_t q_t(· x_0) [ _y≠ x_t q_t(y x_0)q_t(x_t x_0)Q_t(x_t,y)(- R_t^θ(y,x_t)) ]\,dt =∫01xt∼qt(⋅∣x0)[r[xt]x0(t)CE(Rtx0(⋅,xt),Rtθ(⋅,xt))]dt. = _0^1E_x_t q_t(· x_0) [r_[x_t]^x_0(t) CE (R_t^x_0(·,x_t),R_t^θ(·,x_t)) ]\,dt. Here, CE(Rtx0,Rtθ)|xt CE (R_t^x_0,R_t^θ) |_x_t denotes the cross-entropy between Rtx0(⋅,xt)R_t^x_0(·,x_t) and Rtθ(⋅,xt)R_t^θ(·,x_t). Rtx0R_t^x_0 and rx0(t)r^x_0(t) denote respectively the conditional reverse jump kernel and its associated exit rate introduced in Section B.3.2. Proof. Let t∈[0,1]t∈[0,1], we start from the definition of ℒx0path(θ)L_x_0^path(θ) in Eq. 19 and expands the the expectation term inside the time integral to obtain xt∼qt(⋅∣x0)[∑y≠xtQt(y,xt)(−logRtθ(xt,y))] _x_t q_t(· x_0) [ _y≠ x_tQ_t(y,x_t) (- R_t^θ(x_t,y) ) ] =∑x∈qt(x∣x0)∑y≠xQt(y,x)(−logRtθ(x,y)) = _x q_t(x x_0) _y≠ xQ_t(y,x) (- R_t^θ(x,y) ) =∑x∈∑y≠xqt(x∣x0)Qt(y,x)(−logRtθ(x,y)). = _x _y≠ xq_t(x x_0)Q_t(y,x) (- R_t^θ(x,y) ). Swapping the variables names (x,y)↦(y,x)(x,y) (y,x) inside the double sum yields xt∼qt(⋅∣x0)[∑y≠xtQt(y,xt)(−logRtθ(xt,y))] _x_t q_t(· x_0) [ _y≠ x_tQ_t(y,x_t) (- R_t^θ(x_t,y) ) ] =∑x∈∑y≠xqt(y∣x0)Qt(x,y)(−logRtθ(y,x)) = _x _y≠ xq_t(y x_0)Q_t(x,y) (- R_t^θ(y,x) ) (20) =∑x∈qt(x∣x0)∑y≠xqt(y∣x0)qt(x∣x0)Qt(x,y)(−logRtθ(y,x)) = _x q_t(x x_0) _y≠ x q_t(y x_0)q_t(x x_0)Q_t(x,y) (- R_t^θ(y,x) ) =xt∼qt(⋅∣x0)[∑y≠xtqt(y∣x0)qt(xt∣x0)Qt(xt,y)(−logRtθ(y,xt))]. =E_x_t q_t(· x_0) [ _y≠ x_t q_t(y x_0)q_t(x_t x_0)Q_t(x_t,y) (- R_t^θ(y,x_t) ) ]. Integrating over t gives the first identity. For the second identity, we note that for y≠xy≠ x, Section B.3.2 can be restated as qt(y∣x0)qt(x∣x0)Qt(x,y)=r[x]x0(t)Rtx0(y,x). q_t(y x_0)q_t(x x_0)Q_t(x,y)=r_[x]^x_0(t)R_t^x_0(y,x). Plugging this into the inner sum in Eq. 20 yields, for each x∈x , ∑y≠xqt(y∣x0)qt(x∣x0)Qt(x,y)(−logRtθ(y,x))=r[x]x0(t)∑y≠xRtx0(y,x)(−logRtθ(y,x))=r[x]x0(t)CE(Rtx0(⋅,x),Rtθ(⋅,x)). _y≠ x q_t(y x_0)q_t(x x_0)Q_t(x,y) (- R_t^θ(y,x) )=r_[x]^x_0(t) _y≠ xR_t^x_0(y,x) (- R_t^θ(y,x) )=r_[x]^x_0(t) CE (R_t^x_0(·,x),R_t^θ(·,x) ). Taking the expectation over xt∼qt(⋅∣x0)x_t q_t(· x_0) and integrating over t∈[0,1]t∈[0,1] provides the desired quantity: ℒx0path(θ)=∫01xt∼qt(⋅∣x0)[r[xt]x0(t)CE(Rtx0(⋅,xt),Rtθ(⋅,xt))]dt.L_x_0^path(θ)= _0^1E_x_t q_t(· x_0) [r_[x_t]^x_0(t) CE (R_t^x_0(·,x_t),R_t^θ(·,x_t) ) ]\,dt. ∎ A notable simplification comes from the jump-state parametrization. As shown in Section B.3.2, the only θ-dependent contribution is a weighted cross-entropy over reverse jump matrices. By contrast, in the SEDD formulation of lou2023discrete, the ELBO contains two θ-dependent components: (i) a linear score term and (i) a cross-entropy term. In the mean parametrization of campbell1909study (recalled in Section B.3.2), the linear score term is replaced by a θ-dependent reverse-rate term. In both settings, training must therefore fit both jump destinations and a separate θ-dependent factor that controls the time-change, which adds burden and variance. Our jump-state parametrization removes this extra requirement: it keeps the reverse rate fixed and concentrates learning on the core problem (predicting where the chain jumps) via a single cross-entropy objective. The jump-state parametrization is closely related to the approach of amin2025masking. However, we do not condition the path-wise generative model on an explicit event schedule; instead, we keep the same joint distribution defining the generative model as in prior work. B.3.3 Proof of Section 4.2 Let x0∼q0x_0 q_0 and define the forward jump flow measure on the space Ω:=[0,1]× :=[0,1]×V×V as Λx0(dt,dy,dx):=qt(x∣x0)Qt(y,x)y≠xdt. ^x_0(\,dt,\,dy,\,dx):=q_t(x x_0)Q_t(y,x)1_\y≠ x\\,dt. We then rewrite the expression of ℒx0path(θ)L_x_0^path(θ) in Eq. 19 as ℒx0path(θ)=∫01x∼qt(⋅∣x0)[∑y≠xQt(y,x)(−logRtθ(x,y))]dt=∫Ω−logRtθ(x,y)Λx0(dt,dy,dx).L_x_0^path(θ)= _0^1E_x q_t(· x_0) [ _y≠ xQ_t(y,x)(- R_t^θ(x,y)) ]\,dt= _ - R_t^θ(x,y) ^x_0(\,dt,\,dy,\,dx). (21) We now introduce x0(dt,dy,dx):=∑k=1N1δ(Tk,zk,zk−1)(dt,dy,dx)J^x_0(\,dt,\,dy,\,dx):= _k=1^N_1 _(T_k,z_k,z_k-1)(\,dt,\,dy,\,dx), the jump random measure on Ω of the forward CTMC started at x0x_0. Here, N1N_1 and (Tk,zk)1≤k≤N1(T_k,z_k)_1≤ k≤ N_1 are the output of Algorithm 1 with inputs x0x_0, t=1t=1 and QtQ_t. Equivalently, this notation means that for any measurable set A⊂Ω,x0(A)=∑k=1N1(Tk,zk,zk−1)∈AA⊂ ,J^x_0(A)= _k=1^N_11_\(T_k,z_k,z_k-1)∈ A\. Its intensity measure is exactly Λx0 ^x_0 as for any measurable set A⊂ΩA⊂ , we have [x0(A)]=Λx0(A)E[J^x_0(A)]= ^x_0(A). To see this, it suffices to verify the identity on a rectangle A=(a,b]×y×xA=(a,b]×\y\×\x\ with 0≤a<b≤10≤ a<b≤ 1 and x≠y∈x≠ y , and extend to all measurable sets by a monotone class argument. For such a rectangle, we have [x0(A)]=[∑k=1N1Tk∈(a,b],zk=y,zk−1=x]≕[C(a,b]x→y],E[J^x_0(A)]=E [ _k=1^N_11_\T_k∈(a,b],z_k=y,z_k-1=x\ ] [C^x→ y_(a,b] ], where (xt)t∈[0,1](x_t)_t∈[0,1] is the forward CTMC started at x0x_0, and C(a,b]x→yC^x→ y_(a,b] is the number of jumps between states x→yx→ y in the time interval (a,b](a,b]. Let r≥1r≥ 1, we consider a uniform partition of (a,b](a,b] as t0=a<…<b=trt_0=a<…<b=t_r with step-size Δ=(b−a)/r =(b-a)/r. The Markov property and the generator give [C(ti,ti+1]x→y]=ℙ(Xti=x)(Qti(y,x)Δ+o(Δ))E[C^x→ y_(t_i,t_i+1]]=P(X_t_i=x) (Q_t_i(y,x) +o( ) ). Summing over 0≤i≤r−10≤ i≤ r-1 yields a Riemann sum along with a remainder term of the form ∑i=0r−1o(Δ) _i=0^r-1o( ), which vanishes as the mesh goes to 0. Therefore, [C(a,b]x→y]=∫abℙ(xt=x)Qt(y,x)dtE[C^x→ y_(a,b]]= _a^bP(x_t=x)Q_t(y,x)\,dt and, since ℙ(xt=x)=qt(x∣x0)P(x_t=x)=q_t(x x_0), we obtain [x0(A)]=Λx0(A)E[J^x_0(A)]= ^x_0(A). Hence, Campbell’s formula (last2018lectures, Prop. 2.72.7) states that for any nonnegative measurable function g:Ω→ℝ≥0g: _≥ 0, (Tk,zk,zk−1)k[∑k=1N1g(Tk,zk,zk−1)]=∫Ωg(t,y,x)Λx0(dt,dy,dx).E_(T_k,z_k,z_k-1)_k [ _k=1^N_1g(T_k,z_k,z_k-1) ]= _ g(t,y,x) ^x_0(\,dt,\,dy,\,dx). (22) To simplify the notations, we introduce ω=N1,(zk,Tk)k=1N1ω=\N_1,(z_k,T_k)_k=1^N_1\ and q[0,1](⋅∣x0)q_[0,1](· x_0) such that (Tk,zk,zk−1)k=ω∼q[0,1](⋅∣x0)E_(T_k,z_k,z_k-1)_k=E_ω q_[0,1](· x_0). Choosing g(t,y,x)=−logRtθ(x,y)≥0g(t,y,x)=- R_t^θ(x,y)≥ 0 and combining Eqs. 21 and 22 yield ℒx0path(θ)=ω∼q[0,1](⋅∣x0)[∑k=1N1−logRTkθ(zk−1,zk)]=ω∼q[0,1](⋅∣x0)[∑k=1N1−logjθ(zk,Tk)zk−1],L_x_0^path(θ)=E_ω q_[0,1](· x_0) [ _k=1^N_1- R_T_k^θ(z_k-1,z_k) ]=E_ω q_[0,1](· x_0) [ _k=1^N_1- j_θ(z_k,T_k)_z_k-1 ], by definition of jθj_θ. For an expression at the sequence level (of size n≥1)n≥ 1), let 0=x01…x0n∼datax_0=x_0^1… x_0^n q_data. Using t(⋅∣0)=∏ℓ=1nqt(⋅∣x0ℓ) q_t(· _0)= _ =1^nq_t(· x _0) implies that that the sequence-level measure is the superposition of the n independent token jump measure x0ℓJ^x_0 (i.e., 0=∑ℓ=1nx0ℓJ^x_0= _ =1^nJ^x_0 ). The linearity of the expectation implies that ℒ(θ)=∼[0,1](⋅∣0)[∑ℓ=1n∑k=1N1ℓ−logRTkℓθ(zk−1ℓ,zkℓ)],L(θ)=E_ ω q_[0,1](· _0) [ _ =1^n _k=1^N _1- R_T_k ^θ(z_k-1 ,z_k ) ], where =ωℓ=1n∼[0,1](⋅∣0) ω=\ω \_ =1^n q_[0,1](· _0) with ωℓ=N1ℓ,(Tkℓ,zkℓ)kω =\N _1,(T_k ,z_k )_k\, and [0,1](⋅∣0)=∏ℓ=1nq[0,1](⋅∣x0ℓ) q_[0,1](· _0)= _ =1^nq_[0,1](· x_0 ). B.3.4 Recovering the masked diffusion loss For masked diffusion, (ou2024your; sahoo2024simple; shi2024simplified) concurrently discovered a simplified expression of the (campbell2022continuous)-ELBO, that collapses with Section 4.2. This is due to the fact that masked diffusion makes the reverse exit rates independent of θ (amin2025masking). Hence, the seminal parametrization of (austin2021structured; campbell2022continuous) collapses with our jump-states parametrization in this special case. To our knowledge, this property is only true for masked diffusion, and other attempts of generalization from the parametrization of (austin2021structured; campbell2022continuous) led to more complicated losses than the weighted cross-entropy given by our jump-states parametrization (see e.g., von2025generalized; zhou2025next). Consider the forward generator Qtabsorb=f(t)(Fabsorb−Im)Q_t^absorb=f(t)(F^absorb-I_m) with a single mask state [MASK] (i.e., Ft=Fabsorb=δ[MASK]⊤F_t=F^absorb= _ [MASK]1 , or equivalently Πt=Π=δ[MASK]⊤ _t= = _ [MASK]1 ), so that Qtabsorb([MASK],j)=f(t)Q_t^absorb( [MASK],j)=f(t) for j≠[MASK]j≠ [MASK], and no other off-diagonal is nonzero. Remember that qt(xt∣x0)=αtδxt=x0+(1−αt)δxt=[MASK]q_t(x_t x_0)= _t _x_t=x_0+ (1- _t ) _x_t= [MASK]. This means xt∼qt(⋅∣x0)x_t q_t(· x_0) is either xt=x0x_t=x_0 or xt=[MASK]x_t= [MASK]. In this rank-1 absorbing case, the mean parametrization and the jump-states parametrization given by Eq. 9 coincide. Hence, θ only governs the transitions, while the scheduling is fixed by αt _t. In fact, one has the following lemma. boxlem In masked diffusion, the mean parametrization and the jump-states parametrization introduced in Eq. 9 coincide. In particular, the (conditional) reverse rate is independent of θ and given by r[x]x0(t)=r[x](t)=−α˙t1−αt,x=[MASK],0,x≠[MASK].r_[x]^x_0(t)=r_[x](t)= cases - α_t1- _t,&x= [MASK],\\ 0,&x≠ [MASK]. cases Proof. Recall that qt(y∣x0)=αtδy=x0,y≠[MASK],1−αt,y=[MASK],q_t(y x_0)= cases _t _y=x_0,&y≠ [MASK],\\ 1- _t,&y= [MASK], cases (23) and that Section B.3.2 defined the conditional reverse rate at state x as r[x]x0(t)=∑y≠xQtabsorb(x,y)qt(y∣x0)qt(x∣x0).r^x_0_[x](t)= _y≠ xQ_t^absorb(x,y) q_t(y x_0)q_t(x x_0). Computation for x=[MASK]x= [MASK]. Only the forward edges y→[MASK]y→ [MASK] (y≠[MASK]y≠ [MASK]) are nonzero, i.e., Qt([MASK],y)=f(t)Q_t( [MASK],y)=f(t). Therefore, r[MASK]x0(t)=∑y≠[MASK]Qtabsorb([MASK],y)qt(y∣x0)qt([MASK]∣x0)=∑y≠[MASK]f(t)αtδy=x01−αt=f(t)αt1−αt=−α˙t1−αt,r^x_0_ [MASK](t)= _y≠ [MASK]Q_t^absorb( [MASK],y) q_t(y x_0)q_t( [MASK] x_0)= _y≠ [MASK]f(t) _t _y=x_01- _t=f(t) _t1- _t= - α_t1- _t, as f(t)=−α˙t/αtf(t)=- α_t/ _t by definition of the mixing rate (cf. Section 3.2). Computation for x≠[MASK]x≠ [MASK]. There is no forward transition onto non-mask token x≠[MASK]x≠ [MASK] (Qt(x,y)=0Q_t(x,y)=0 for all y∈y ), hence r[x]x0(t)=0r^x_0_[x](t)=0 for all i≠[MASK]i≠ [MASK]. From r[x]x0(t)r^x_0_[x](t) to r[x](t)r_[x](t). Marginalizing over x0∼q0x_0 q_0 yields qt([MASK])=∑x0∈q0(x0)qt([MASK]∣x0)=∑x0∈q0(x0)(1−αt)=1−αt,q_t( [MASK])= _x_0 q_0(x_0)\,q_t( [MASK] x_0)= _x_0 q_0(x_0)\,(1- _t)=1- _t, where we used Eq. 23. Hence, qt([MASK]∣x0)=qt([MASK])=1−αtq_t( [MASK] x_0)=q_t( [MASK])=1- _t, which implies that ∑y≠[MASK]qt(y∣x0)=αt=∑y≠[MASK]qt(y). _y≠ [MASK]q_t(y x_0)= _t= _y≠ [MASK]q_t(y). Considering r[x](t)r_[x](t), we can repeat the proof of r[x]x0(t)r^x_0_[x](t) since r[x](t)=∑y≠xQtabsorb(x,y)qt(y)qt(x)r_[x](t)= _y≠ xQ_t^absorb(x,y) q_t(y)q_t(x), and obtain r[x](t)=−α˙t1−αt,x=[MASK],0,x≠[MASK],=r[x]x0(t).r_[x](t)= cases - α_t1- _t,&x= [MASK],\\ 0,&x≠ [MASK], cases=r^x_0_[x](t). ∎ We can now prove that our ELBO in Section 4.2 derived from the jump states parametrization coincides with the ELBO of masked diffusion models derived from the mean parametrization in (ou2024your; sahoo2024simple; shi2024simplified). boxprop [Recovering the MDM loss] For the case of masked diffusion, our ELBO in Section 4.2 coincides with the MDM loss: ℒx0path(θ)=∫01−α˙t1−αtxt∼qt(⋅∣x0)[xt=[MASK](−logμθ(xt,t))x0]dt.L_x_0^path(θ)= _0^1 - α_t1- _tE_x_t q_t(· x_0) [1_\x_t= [MASK]\(- _θ(x_t,t))_x_0 ]\,dt. Proof. Section B.3.4 directly implies that Atθ=RtθA_t^θ=R_t^θ in Eqs. 17 and 18, so the mean and jump parameterization collapses in the masked diffusion case (μθ=θ _θ= j_θ). Recovering the MDM loss is pretty straightforward from the expression ℒx0path(θ)=∫01xt∼qt(⋅∣x0)[∑y≠xtQtabsorb(y,xt)(−logμθ(y,t)xt)]dtL_x_0^path(θ)= _0^1E_x_t q_t(· x_0) [ _y≠ x_tQ_t^absorb(y,x_t)(- _θ(y,t)_x_t) ]\,dt. In fact, 1. if xt=x0x_t=x_0, then the token is still clean, and we have ∑y≠xtQtabsorb(y,xt)(−logμθ(y,t)xt)=Qtabsorb([MASK],x0)(−logμθ([MASK],t)x0)=f(t)(−logμθ([MASK],t)x0) _y≠ x_tQ_t^absorb(y,x_t)(- _θ(y,t)_x_t)=Q_t^absorb( [MASK],x_0)(- _θ( [MASK],t)_x_0)=f(t)(- _θ( [MASK],t)_x_0); 2. if xt=[MASK]x_t= [MASK], the token is masked. In this case, Qtabsorb(y,[MASK])=0Q_t^absorb(y, [MASK])=0 for all y≠[MASK]y≠ [MASK]. Hence, ∑y≠xtQtabsorb(y,xt)(−logμθ(y,t)xt)=0 _y≠ x_tQ_t^absorb(y,x_t)(- _θ(y,t)_x_t)=0. Combining these cases and using ℙ(xt=x0∣x0)=αtP(x_t=x_0 x_0)= _t and ℙ(xt=[MASK]∣x0)=1−αtP(x_t= [MASK] x_0)=1- _t, we obtain ℒx0path(θ)=∫01αtf(t)(−logμθ([MASK],t)xt)dt=∫01−α˙t(−logμθ([MASK],t)xt)dtL_x_0^path(θ)= _0^1 _tf(t)(- _θ( [MASK],t)_x_t)\,dt= _0^1- α_t(- _θ( [MASK],t)_x_t)\,dt, where we used α˙t=−αtf(t) α_t=- _tf(t). This is the simplest form of the loss, but is usually written differently by leveraging the identity xt∼qt(⋅∣x0)[xt=[MASK](−logμθ(xt,t)x0)]=(1−αt)(−logμθ([MASK],t)x0)E_x_t q_t(· x_0)[1_\x_t= [MASK]\(- _θ(x_t,t)_x_0)]=(1- _t)(- _θ( [MASK],t)_x_0). Finally, we recover the MDM loss as ℒx0path(θ)=∫01−α˙t1−αtxt∼qt(⋅∣x0)[xt=[MASK](−logμθ(xt,t))x0]dt.L_x_0^path(θ)= _0^1 - α_t1- _tE_x_t q_t(· x_0) [1_\x_t= [MASK]\(- _θ(x_t,t))_x_0 ]\,dt. ∎ B.4 Snapshot parametrization and Evidence Lower Bound (ELBO) Proof of Section 4.3. Consider the snapshot latent s=(xt,t)s=(x_t,t) and the variational distribution qsnap(s∣x0)=ρ(t)qt(xt∣x0)q^snap(s x_0)=ρ(t)q_t(x_t x_0), where ρ is the uniform distribution over [0,1][0,1] for simplicity (i.e., ρ(t)=1ρ(t)=1 for all t∈[0,1]t∈[0,1]). The snapshot marignal is then qsnap(s)=x0∼qdata[qsnap(s∣x0)]=x0∼qdata[qt(xt∣x0)]=qt(xt)q^snap(s)=E_x_0 q_data[q^snap(s x_0)]=E_x_0 q_data[q_t(x_t x_0)]=q_t(x_t). We define the snapshot predictor p0θ,snap(x0∣s)≔μθ(xt,t)x0p_0^θ,snap(x_0 s) _θ(x_t,t)_x_0 from the output of the mean network. This defines a latent-variable model with joint probability p0θ,snap(x0,s)=p0θ,snap(x0∣s)qsnap(s).p_0^θ,snap(x_0,s)=p_0^θ,snap(x_0 s)q^snap(s). A standard ELBO derivation (see e.g., lai2025principles) consists of applying Jensen’s inequality as follows, logp0θ,snap(x0)=logs∼qsnap(⋅∣x0)[p0θ,snap(x0,s)qsnap(s∣x0)]≥s∼qsnap(⋅∣x0)[logp0θ,snap(x0,s)qsnap(s∣x0)]. p_0^θ,snap(x_0)= _s q^snap(· x_0) [ p_0^θ,snap(x_0,s)q^snap(s x_0) ] _s q^snap(· x_0) [ p_0^θ,snap(x_0,s)q^snap(s x_0) ]. Expanding the right hand-side using p0θ,snap(x0,s)=p0θ,snap(x0∣s)qsnap(s)p_0^θ,snap(x_0,s)=p_0^θ,snap(x_0 s)q^snap(s) yields logp0θ,snap(x0)≥−ℒx0snap(θ)+Cx0snap, p_0^θ,snap(x_0)≥-L_x_0^snap(θ)+C^snap_x_0, where Cx0snap≔−s∼qsnap(⋅∣x0)[logqsnap(s∣x0)qsnap(s)]C^snap_x_0 -E_s q^snap(· x_0) [ q^snap(s x_0)q^snap(s) ] is independent of θ and ℒx0snap(θ)≔s∼qsnap(⋅∣x0)[−logp0θ,snap(x0∣s)]=∫01xt∼qt(⋅∣x0)[−logμθ(xt,t)x0]dt.L_x_0^snap(θ) _s q^snap(· x_0) [- p_0^θ,snap(x_0 s) ]= _0^1E_x_t q_t(· x_0) [- _θ(x_t,t)_x_0 ]\,dt. ∎ We now move on to the proof of Section 4.3. Consider a clean token x0∼qdatax_0 q_data. The path-wise latent ω=N1,(zk,Tk)k=1N1∼q[0,1](⋅∣x0)ω=\N_1,(z_k,T_k)_k=1^N_1\ q_[0,1](· x_0) is the full forward CTMC path on [0,1][0,1] and s=(xt,t)s=(x_t,t) is the snapshot latent from this path. This means that we have sampled t∼ρ(t)t ρ(t) (with ρ being the uniform distribution over [0,1][0,1]) independently of x0x_0, and extracted the associated token xtx_t from ω. This token is defined as follows : there exists some kt∈1,…,N1k_t∈\1,…,N_1\ such that t∈(Tkt−1,Tkt]t∈(T_k_t-1,T_k_t], and xt=zkt−1x_t=z_k_t-1 (where the edge case kt=1k_t=1 is covered by setting T0=0T_0=0 and z0=x0z_0=x_0). More formally, if Ω is the path space and S the snapshot space, there exists a measurable map Ψ:[0,1]×Ω→S :[0,1]× → S such that Ψ(t,ω)=s (t,ω)=s. For v∈s,ωv∈\s,ω\, the joint law of both generative models can be written as q(x0,v)=qdata(x0)q(v∣x0)q(x_0,v)=q_data(x_0)q(v x_0) and the corresponding posterior as q(x0∣v)=q(x0,v)/q(v)q(x_0 v)=q(x_0,v)/q(v). Let pθsnap(⋅∣s)p_θ^snap(· s) and pθpath(⋅∣ω)p_θ^path(· ω) be any conditional predictors. We define the expected NLL gap as ΔθNLL=[−logp0θ,snap(x0∣s)]−[−logp0θ,path(x0∣ω)] ^NLL_θ=E[- p_0^θ,snap(x_0 s)]-E[- p_0^θ,path(x_0 ω)] and the calibration error as Calθv=[KL(q(x0∣v)∥pθ(⋅∣v))]\,Cal_θ^v=E[ KL (q(x_0 v)\,\|\,p_θ(· v))]. Let us start by introducing a simple lemma. boxlem [Expected NLL decomposition] Let (X,V)∼q(⋅,⋅)(X,V) q(·,·) be any pair, and let pθ(⋅∣v)p_θ(· v) be any conditional predictor for v∈s,ωv∈\s,ω\. Then, (X,V)∼q(⋅,⋅)[−logpθ(X∣V)]=H(X∣V)+V[KL(q(⋅∣V)∥pθ(⋅∣V))].E_(X,V) q(·,·) [- p_θ(X V) ]=H(X V)+E_V [ KL (q(· V)\,\|\,p_θ(· V) ) ]. Proof. After conditioning on V=vV=v, we obtain [−logpθ(X∣V)]=V[X∼q(⋅∣V)[−logpθ(X∣V)]].E [- p_θ(X V) ]=E_V [E_X q(· V)[- p_θ(X V)] ]. For each fixed v∈v , the inner expectation is the cross-entropy between q(⋅∣v)q(· v) and pθ(⋅∣v)p_θ(· v): X∼q(⋅∣v)[−logpθ(X∣v)]=H(q(⋅∣v),pθ(⋅∣v))=H(q(⋅∣v))+KL(q(⋅∣v)∥pθ(⋅∣v)).E_X q(· v)[- p_θ(X v)]=H (q(· v),p_θ(· v) )=H (q(· v) )+ KL (q(· v)\,\|\,p_θ(· v) ). Taking expectation over V gives [−logpθ(X∣V)]=V[H(q(⋅∣V))]+V[KL(q(⋅∣V)∥pθ(⋅∣V))].E[- p_θ(X V)]=E_V[H(q(· V))]+E_V[ KL (q(· V)\,\|\,p_θ(· V))]. However, V[H(q(⋅∣V))]=H(X∣V)E_V[H(q(· V))]=H(X V) by definition of conditional entropy. ∎ Proof of Section 4.3. Applying Section B.4 twice with V=sV=s and V=ωV=ω, and subtracting the two yields the following decomposition: ΔθNLL=H(x0∣s)−H(x0∣ω)⏟IPG+Calθs−Calθω⏟CG, ^NLL_θ= H(x_0 s)-H(x_0 ω)_IPG+ \,Cal_θ^s-\,Cal_θ^ω_CG, where Calθs≔[KL(q(⋅∣s)∥pθ(⋅∣s))]\,Cal_θ^s [ KL (q(· s)\,\|\,p_θ(· s))]. Note that, by measurability of Ψ:[0,1]×Ω→S :[0,1]× → S, the data processing inequality states that the mutual information can only decrease, i.e., I(x0;ω)≥I(x0;s)I(x_0;ω)≥ I(x_0;s), or equivalently H(x0∣s)≥H(x0∣ω)H(x_0 s)≥ H(x_0 ω), which implies that IPG≥0IPG≥ 0. Snapshot minimizer. The snapshot loss is given by ℒx0snap(θ)=∫01xt∼qt(⋅∣x0)[−logμθ(xt,t)x0]dt=[−logpθsnap(X0∣s)|X0=x0].L^snap_x_0(θ)= _0^1E_x_t q_t(· x_0) [- _θ(x_t,t)_x_0 ]\,dt=E [- p_θ^snap(X_0 s)\, |\,X_0=x_0 ]. (24) Note that it would be weighted by ρ(t)ρ(t) inside the time integral if ρ was chosen differently from the uniform density. Averaging Eq. 24 over x0∼qdatax_0 q_data yields exactly the unconditional expected snapshot NLL: x0∼qdata[ℒx0snap(θ)]=x0∼qdatas∼qsnap(⋅∣x0)[−logpθsnap(x0∣s)].E_x_0 q_data[L^snap_x_0(θ)]=E_x_0 q_dataE_s q^snap(· x_0)[- p_θ^snap(x_0 s)]. Following Section B.4, x0∼qdata[ℒx0snap(θ)]=H(x0∣s)+CalθsE_x_0 q_data[L^snap_x_0(θ)]=H(x_0 s)+\,Cal_θ^s, where the term H(x0∣s)H(x_0 s) depends only on the data distribution and the forward process, and not on θ. Therefore argminθ[ℒx0snap(θ)]=argminθCalθs. _θE[L^snap_x_0(θ)]= _θ\,Cal_θ^s. Path-wise minimizer. Contrary to a snapshot ELBO, a path-wise ELBO (i.e., a diffusion ELBO; see lai2025principles) contains an additional diffusion term that depends on θ. This makes the model calibrates the local conditionals zk−1∣(zk,Tk)z_k-1 (z_k,T_k) instead of x0∣ωx_0 ω. This can be clearly seen here with the Campbell form of the path-wise ELBO. Indeed, recall that Calθω:=ω[KL(q(x0∣ω)∥pθ(⋅∣ω))]\,Cal_θ^ω:=E_ω [ KL (q(x_0 ω)\,\|\,p_θ(· ω) ) ] measures calibration of a predictor of the initial token x0x_0 given the full path ω. In contrast, the Campbell path-wise objective ℒx0path(θ)=ω∼q[0,1](⋅∣x0)[∑k=1N1−logθ(zk,Tk)zk−1]L_x_0^path(θ)=E_ω q_[0,1](· x_0) [ _k=1^N_1- j_θ(z_k,T_k)_z_k-1 ] is the negative log-likelihood of predicting the previous state zk−1z_k-1 at each jump time from the event (zk,Tk)(z_k,T_k). More precisely, using the other form given by Section 4.2, ℒx0path(θ)=∫01xt∼qt(⋅∣x0)[rxtx0(t)CE(Rtx0(⋅,xt),Rtθ(⋅,xt))]dt,L_x_0^path(θ)= _0^1E_x_t q_t(· x_0) [r_x_t^x_0(t)\, CE (R_t^x_0(·,x_t),R_t^θ(·,x_t) ) ]\,dt, hence, up to a θ-independent term, ℒx0path(θ)=const+∫01xt∼qt(⋅∣x0)[rxtx0(t)KL(Rtx0(⋅,xt)∥Rtθ(⋅,xt))]dt.L_x_0^path(θ)=const+ _0^1E_x_t q_t(· x_0) [r_x_t^x_0(t)\, KL (R_t^x_0(·,x_t)\,\|\,R_t^θ(·,x_t) ) ]\,dt. Therefore, minimizing [ℒX0path(θ)]E[L_X_0^path(θ)] calibrates the local reverse jump kernel Rtθ(⋅,x)R_t^θ(·,x), not the posterior predictor pθ(x0∣ω)p_θ(x_0 ω). Since Calθω\,Cal_θ^ω concerns a different conditional distribution (namely x0∣ωx_0 ω), the minimizers do not coincide in general. ∎ Appendix C Experimental details C.1 Experimental setting We train GDDS models on Text8 and OWT with bf16 precision (including the loss), global batch size 512, AdamW with learning rate 3.5×10−43.5× 10^-4, weight decay 10−210^-2, (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95), gradient clipping at 1.0, and EMA with decay 0.9999, for 1M optimizer steps, validating every 10k steps. All runs are executed on a single node with 4× NVIDIA H100 GPUs using DDP. More details are given in Table 5. Table 5: GDDS Training configuration Category Setting Sequence length 256 for Text8, 1024 for OWT Hidden size / heads 768 / 12 MLP ratio / dropout 4 / 0.1 Time conditioning AdaLN Layers 12 Optimizer AdamW (β1=0.9,β2=0.95,ϵ=10−8 _1=0.9, _2=0.95,ε=10^-8), weight decay=0.01=0.01 Learning rate 3.5×10−43.5× 10^-4 LR schedule Constant warmup for the 2500 first steps Precision bf16 (training and loss precision) EMA 0.9999 Batch size Global batch size 512 (eval global batch size 512) Gradient clipping 1.0 Max steps 1,000,000 Eval validation every 10,000 steps Noise log-linear Hardware 1 node, 4× NVIDIA H100 GPUs; 4 tasks per node (DDP) Text8 dataset. We train models on Text8 for 1 million optimizer steps. While results in Table 1 are reported from (shi2024simplified), we additionally retrained three baselines: a standard autoregressive Transformer (AR), MDM (sahoo2024simple; shi2024simplified; ou2024your) and UDLM (schiff2024simple) using the same training recipe as for GDDS (optimizer, learning rate, schedule, batch size, precision, number of steps, and evaluation protocol). The only architectural difference is that the AR baseline uses no time conditioning and causal attention. Under this unified setup, we obtain 1.351.35, 1.581.58 and 1.671.67 BPC for AR, MDM and UDLM respectively (see Table 1). Notably, these values are worse than the corresponding numbers reported in Table 1 in (shi2024simplified), suggesting that cross-paper comparisons are sensitive to training details (e.g., optimization and implementation choices) and experimental conditions (e.g., hardware and system-level settings). Note that our AR result exactly matches the one found by (hoogeboom2021autoregressive). Since GDDS Absorb still outperforms these baselines by a large margin under our unified setup, we do not emphasize this mismatch further but mention it for completeness. OpenWebText dataset. We follow the same unified protocol on OWT, training all models for 500500k optimizer steps at sequence length 10241024 (see Table 5), resulting in a total of 262262B training tokens. We tokenize OpenWebText using the GPT-2 tokenizer, then concatenate documents and chunk the resulting stream into sequences of length 10241024, inserting an <|endoftext|> token between consecutive documents. Since OpenWebText has no official validation split, we reserve the last 100100k tokens for validation. Similarly than Text8, we retrain the matched-compute baselines (AR, MDM, and UDLM) with the same optimizer, learning-rate schedule, batch size, precision, EMA, and evaluation pipeline as our GDDS variants. Under this controlled setup, we observe again that our retrained baselines do not reproduce the perplexities reported in prior papers (see Table 2), highlighting the sensitivity of OWT results to implementation details and system-level factors (e.g., data processing, optimization hyperparameters, and hardware/software stacks). For this reason, we primarily rely on our retrained baselines for fair comparisons, while still reporting prior-work numbers when available. Importantly, GDDS yields substantially tighter likelihood bounds and stronger generation quality under the same protocol, and the main conclusions remain unchanged despite the mismatch with previously reported figures. In Table 3, we evaluate zeroshot perplexity by taking the models trained on OpenWebText and evaluating likelihoods on the validation splits of 7 datasets: Penn Tree Bank (PTB; marcus1993building), Wikitext103 (merity2016pointer), One Billion Word Language Model Benchmark (LM1B; chelba2013one), Lambada (paperno2016lambada), AG News (zhang2015character), and scientific papers (Pubmed and Arxiv subsets; cohan2018discourse). Since the zeroshot datasets have different conventions for sequence segmentation, we wrap sequences to 1024 but do not add <|endoftext|> tokens in between sequences. C.2 Semantic-Informed Kernel (SIK) Recall that m is the vocabluary size. Let E∈ℝm×dE ^m× d be a fixed embedding table, where ei∈ℝde_i ^d denote the embedding vector of the token number i∈1,…,mi∈\1,…,m\. Let ρii=1m\ _i\_i=1^m be positive local bandwidths (e.g., ρi _i is the squared distance from eie_i to its k-th nearest neighbor). We fix (i) a decaying profile η:[0,∞)→ℝ+η:[0,∞) _+, (i) a distance on the embedding space demb:ℝd×ℝd→[0,∞)d_emb:R^d×R^d→[0,∞), and (i) a temperature schedule τ:[0,1]→(0,∞)τ:[0,1]→(0,∞) that controls the amount of mixing (small τ(t)τ(t) yields a near-identity kernel; large τ(t)τ(t) yields a flatter kernel). Following self-tuning Diffusion Maps (zelnik2004self), we define the (symmetric) affinity as Wt(x,y):=η(demb(e[x],e[y])2τ(t)ρ[x]ρ[y]),for x≠y∈,Wt(x,x)=0.W_t(x,y):=η\! ( d_emb(e_[x],e_[y])^2τ(t) _[x] _[y] ), x≠ y , W_t(x,x)=0. (25) Hence, it is possible to define the Semantic-Informed Kernel by normalizing the affinity: FtSIK(x,y):=Wt(x,y)∑z≠yWt(z,y),x≠y,FtSIK(y,y)=0.F_t^SIK(x,y):= W_t(x,y) _z≠ yW_t(z,y), x≠ y, F_t^SIK(y,y)=0. For the Gaussian metric, demb(e[x],e[y])=‖e[x]−e[y]‖22d_emb(e_[x],e_[y])=\|e_[x]-e_[y]\|_2^2, while for the cosine metric, demb(e[x],e[y])=1−⟨e¯[x],e¯[y]⟩d_emb(e_[x],e_[y])=1- e_[x], e_[y] after normalizing embeddings to unit norm. The corresponding CTMC rate matrix is QtSIK:=f(t)(FtSIK−Im)Q_t^SIK:=f(t) (F_t^SIK-I_m ), with f such that αt=exp(−f¯(t)) _t= (- f(t)) as in Section 3.2. The embeddings are extracted from GPT-2 (radford2019language), where d=768d=768 and m=50257m=50257. Gaussian and cosine SIKs. In the experiments, we consider two concrete choices for the affinity defining FtSIKF_t^SIK. The first is the Gaussian SIK, based on squared Euclidean distance in embedding space. The second is a cosine SIK, obtained by first normalizing embeddings to unit norm and then using the cosine distance demb(e[x],e[y])=1−⟨e¯[x],e¯[y]⟩d_emb(e_[x],e_[y])=1- e_[x], e_[y] . The Gaussian version favors tokens that are close in Euclidean geometry, while the cosine version instead depends only on the angle between embeddings. The main text notation FtGaussF_t^Gauss corresponds precisely to this Gaussian choice of the SIK affinity. Two implementations: KNN and KeOps. There are then two ways to implement FtSIKF_t^SIK in practice. The first is a KNN implementation, where for each source token y we precompute a sparse neighbor set k(y)N_k(y) containing only its top-k nearest candidate neighbors in embedding space (in our benchmark, k=64k=64). In that case, we use the sparse approximation FtKNN(x,y) F_t^KNN(x,y) =(1−λ(t))exp(−demb(e[x],e[y])ερ[x]ρ[y])x∈k(y)∑z∈k(y)exp(−demb(e[z],e[y])ερ[z]ρ[y])+λ(t)x≠ym−1 = (1-λ(t) ) \! (- d_emb(e_[x],e_[y]) \, _[x] _[y] )1_x _k(y) _z _k(y) \! (- d_emb(e_[z],e_[y]) \, _[z] _[y] )\;+\;λ(t) 1_x≠ ym-1 so sampling takes place only on this sparse candidate set, which makes the method very fast in practice. The second is a KeOps implementation, denoted FtKeOpsF_t^KeOps, where we do not build a sparse graph and do not materialize the dense m×m× m kernel either; instead, we evaluate the entries of FtSIKF_t^SIK on-the-fly with blockwise GPU reductions. Thus, KNN should be viewed as a sparse approximation of FtSIKF_t^SIK, whereas KeOps is the dense lazy implementation of the same normalized kernel. Why this is KeOps-friendly. For the dense implementation FtKeOpsF_t^KeOps, we only specify the symbolic pairwise map (ℓ,j)⟼η(demb(eℓ,ej)2τ(t)ρℓρj).( ,j) \ η\! ( d_emb(e_ ,e_j)^2τ(t) _ _j ). Thus, we let KeOps generate fused GPU kernels that compute the needed sums on-the-fly. This avoids storing QtSIKQ_t^SIK explicitly and keeps forward noising practical. Benchmark interpretation. Table 6 reports both similarity choices (Gaussian and cosine) and both implementations (KNN and KeOps), in addition to the absorbing and uniform baselines. In our setup, the KNN versions are substantially faster because they sample only from a sparse neighbor list, while the KeOps versions operate over the full vocabulary through lazy blockwise reductions. This induces an overhead because the dense KeOps path still evaluates normalized kernel scores over large vocabulary blocks and then performs exact categorical sampling over those blocks. Our implementation uses a custom CUDA kernel for the blockwise sampler on top of the KeOps score construction, which makes the dense path practical. We acknowledge that this CUDA kernel can still be optimized further to reduce the remaining overhead. However, even in its current form, around 150150–160160 ms to noise an entire batch of 512×1024=524,288512× 1024=524,288 tokens is small compared with the cost of a full forward-and-backward training step on batches of that size. The point of the KeOps implementation is therefore not to beat the KNN approximation in raw latency, but to make the dense normalized kernel FtSIKF_t^SIK feasible without ever materializing the full transition matrix. Table 6: Noising-time benchmark. Mean wall-clock latency in milliseconds for sampling xt∼qt(⋅∣x0)x_t q_t(· x_0) on batches of size 512512 and sequence length 10241024 (512×1024=524,288512× 1024=524,288 positions). Reported values are mean ± standard deviation over 55 random seeds; for each seed, latency is averaged over 1010 timed runs after 33 warmup runs. Results use absorbing, uniform, and SIK-based forward processes with KNN and KeOps implementations. Method Mean ± Std (ms) Absorbing 0.09±0.010.09± 0.01 Uniform 0.07±0.000.07± 0.00 SIK Gauss (KNN) 8.97±0.268.97± 0.26 SIK Gauss (KeOps) 152.49±1.86152.49± 1.86 SIK Cosine (KNN) 8.94±0.278.94± 0.27 SIK Cosine (KeOps) 162.45±2.19162.45± 2.19 All reported GDDS Gauss training results in this paper use the KNN implementation with k=64k=64 neighbors per token. C.3 Metrics We report both likelihood-based metrics (BPC / perplexity) and generation metrics computed from unconditional samples (Generative Perplexity, Sequence Entropy, Distinct-n). Throughout, let N denote the number of evaluated sequences (validation examples or generated samples, depending on the metric) and let (j)=(x(j),1,…,x(j),n)x^(j)=(x^(j),1,…,x^(j),n) denote the sequence j∈1,…,nj∈\1,…,n\ of length n. C.3.1 Negative log-likelihood (NLL), BPC, and PPL For likelihood-based evaluation, consider a model that defines a probability pθ()p_θ(x) over sequences =(x1,…,xn)x=(x^1,…,x^n). The (per-token) negative log-likelihood on N sequences (i)i=1N\x^(i)\_i=1^N is NLL≔−1Nn∑i=1Nlogpθ((i)).NLL\; \;- 1N\,n _i=1^N p_θ\! (x^(i) ). When the model admits a tractable factorization (e.g. autoregressive), logpθ()=∑j=1nlogpθ(xj∣x<j), p_θ(x)= _j=1^n p_θ(x^j x^<j), which recovers the standard “next-token prediction loss” expression. In general, logpθ() p_θ(x) may be intractable; in that case we report a variational upper bound on NLL (equivalently a lower bound on logpθ() p_θ(x)), such as the ELBO induced by the training objective. Bits per character (BPC). On character-level datasets (Text8), we report BPC, i.e. the NLL expressed in base 22: BPC≔−1Nn∑i=1Nlog2pθ((i))=NLLlog2.BPC\; \;- 1N\,n _i=1^N _2p_θ\! (x^(i) )\;=\; NLL 2. Perplexity (PPL). On tokenized datasets (OWT), we report perplexity, defined as the exponential of the NLL: PPL≔exp(NLL)=exp(−1Nn∑i=1Nlogpθ((i))).PPL\; \; (NLL)\;=\; \! (- 1N\,n _i=1^N p_θ\! (x^(i) ) ). Lower NLL (equivalently lower BPC/PPL) indicates better likelihood-based performance. C.3.2 Generative Perplexity Likelihood metrics do not directly assess sample quality when generation uses an approximate sampler. Following prior work (lou2023discrete; sahoo2024simple), we therefore report Generative Perplexity (Gen-PPL) under a fixed external evaluator (GPT2-large in our experiments). Given unconditional samples (i)i=1N\x^(i)\_i=1^N produced by a model, we compute Gen-PPL=exp(−1Nn∑i=1N∑j=1nlogpeval(x(i),j∣x(i),<j)),Gen -PPL\;=\; \! (- 1N\,n _i=1^N _j=1^n p_eval\! (x^(i),j x^(i),<j ) ), where pevalp_eval denotes the evaluator next-token distribution (GPT2-large). Lower Gen-PPL indicates that generated samples are more predictable under the reference model, which empirically correlates with higher perceived fluency. C.3.3 Sequence Entropy A low Gen-PPL can be achieved by overly repetitive or mode-collapsed generations, as noticed by (zheng2024masked). To quantify diversity, we compute the average unigram entropy of each generated sequence and average across samples. Let c(v,(i))c(v,x^(i)) be the count of token v∈v in sample (i)x^(i). Define the empirical unigram distribution within a sample as p^(i)(v)=c(v,(i))/n p^(i)(v)=c(v,x^(i))/n. We report Huni=−1N∑i=1N∑v∈p^(i)(v)logp^(i)(v).H_uni=- 1N _i=1^N _v p^(i)(v)\, p^(i)(v). Higher values indicate that a sample uses a broader set of tokens more evenly (i.e., less repetition). In the paper, we plot Gen-PPL against this entropy statistic to visualize the quality–diversity tradeoff, following (zheng2024masked). C.3.4 Distinct-n We additionally report Distinct-n statistics, which measure lexical diversity via the proportion of unique n-grams. To avoid overloading notation, we denote by Distinct-kDistinct-k the metric computed with k-grams. Let kgramsk((i))kgrams_k(x^(i)) denote the multiset of length-k contiguous k-grams in sample (i)x^(i). We compute corpus-level Distinct-k over the full set of samples: Distinct-k=|⋃i=1Nuniq(kgramsk((i)))|∑i=1N|kgramsk((i))|,k∈1,2,3,Distinct-k= |\; _i=1^Nuniq\! (kgrams_k(x^(i)) )\; | _i=1^N |kgrams_k(x^(i)) |, k∈\1,2,3\, where uniq(⋅)uniq(·) removes duplicates within a sequence. Higher Distinct-k indicates fewer repeated k-grams and typically correlates with more diverse generations. C.4 Qualitative samples on Text8 We report in Tables 7 and 8 two unconditional samples for the GDDS Absorb and Uniform models trained on the Text8 dataset. Table 7: Two unconditional samples for the GDDS Absorb model trained on Text8. [BOS]sessions studies professional present topics international literature university of global history of international shock artist as well as its global no actibilative clocks to ruin the strip individual martials in the image of information both side east[EOS] [BOS]g strewn nearly confrentative and the bible tag strews which were very officialy dilgress in the usa japanepic first considered only parathislatic influences arabic christianity have played in their language slightly best known is the japanene term in ma[EOS] Table 8: Two unconditional samples for the GDDS Uniform model trained on Text8. [BOS]heir ownest possession is the case despite cyclic subgraphs resulting in its first higher community area or less expensive [EOS]major national companies again below the company is a supporter of city trains or a record historically system is cut the large bu[EOS] [BOS]ree the oxford university vol two six seven two two p two seven three two x are not human definitions of oxford since others are seaset fara la conventions university of chicago based on bissaudi and known as the karlowe large content of the world elbia[EOS] C.5 Qualitative samples on OpenWebText For readability, we apply a lightweight post-processing to unconditional samples before rendering them in LaTeX. Concretely, we replace paragraph markers by a line breaks, decode common Unicode escape sequences into standard typography (e.g., 2019↦~ ~’, 201c↦~ ~“, 201d↦~ ~”, 2264↦≤~ ~≤), and render special tokens such as <|endoftext|> in . We report in Tables 9, 10 and 11 unconditional samples for the GDDS Absorb, Uniform, and Gauss models trained on the OpenWebText dataset. Table 9: Unconditional sample for the GDDS Absorb model trained on OpenWebText. <|endoftext|> to conform to routine training procedures for the handling of a crime or any mass deaths committed by police when it investigates crimes. Opponents of the entertainer’s departure say the department will offer guarantees in coverage for a crisis that in 2002 and 2009 frees law enforcement. The protesters, say police are taught to commit crime continuously with tear gas or bullets. The chaos comes amid sweeping changes in policing. Illinois law enforcement took last month’s a surprise given the growing appetite in some places for police from function as order in the state. “But more generally, it’s a safe haven,” said Lake-based Howard Hexano. Theneighbor Police Association of the United States said it replaced the training road used by law enforcement for initially international training exercises, rather opting to resemble standard “economic training” methods. Sarley opened questions about the leadership of Kenney Village, the same police department, that has the traditional candidate for the license for two men accused of shooting someone at the Colorado New Year’s Eve pageant annual competition, but it pays paid for the monthly pay of the victims. The agency won’t face any disciplinary action against officers in the scandal, and it has addressed the community about the past. Earley on Feb. 28 will officially investigate the incident, concluding that the men accused were not teaching acting well, or causing harm to others. The Tribune’s Matt Steward declined to comment. The department’s chief officer, J. Durkin, who was a local pastor following his Feb. 24 firing, did not respond to requests for earring for comment on the incidents. All grievances in the investigation came and there’s need for further technical inquiry. Even local law enforcement companies, including Illinois Public Affairs and other state law enforcement officials continue to have witness cooperation, such as W-Ferguson and Louisiana C-M., in the works. “We actively consult — municipal agencies and other social health and architectural agencies to deliver information services and information regarding current events,” Charlie Paley said in an announcement. “C-M., Under-Chief Shawn Leslie and Nick Stout, Executive Director are working closely with the person conducting the investigation for evaluation and possible update on future events. For more about what happened in Kidney Village, read on here.” The real highlights the six basic changes and guidelines recommended throughout the town’s history. For additional units, if completed, the Westlake Union Marriott Hotel will be located in the 650 block. Comprehensive plans include a spa spa. It is 45-foot by 40- feet wide and have a barbed roof. Regional — urban — will be 10,000 square feet and include RGB lighting, natural gas development, natural fire and swimming pool and golf courses. As the area grows diverse enough, police will become concerned about the potential threat to its security. Some cops want hotel gives new applicants for a recreational site to meet location hotel is home to the training headquarters of the U.S. Marines, home of about 700 horse and elderly confrontations with law enforcement officers. It’s normal for the U.S. government to increase its activist role — asserting that American police do slow to inflict damage on Americans during First World War. The U.S. Congress loosened constitutional restraints when it comes to police violations trial except that makes it a fine, similar to what is guilty under the federal Criminal Code. Also floating around is potential limited impacts — and public safety and safety have increased as a deterrent since when a school student was shot while running a road trip. Why is that? But tourists won’t allow it in. That common sense will prove little change for the town, either. And the impact will no longer homes. But any feeling of diversity will be gaining almost all in the surrounding institution, many of its core buildings and colleges. The Cenna-The Performing Arts exhibition is “Hop Art Thursday” by Sen. David D Ducey (D-D.). “To make everyday Americans realize that I think the museum stands out throughout much of American history,” he says when a 65-year-old drops an invitation from Harvard’s Dr. Richard B. Hoffman to enter the White House in 2006. And always will have been the center of the museum’s popularity. The visitors that the design be more limited in future exhibits, said Fucakis, are themes from past films, such as the arenas and cop offices, “brain longening” and “Hey, Captain. I have the power of a cat.” Fucakis, also a pollster, announced Thursday that the megamstitute is opening a doghouse and would be introducing video of historical footage<|endoftext|> Table 10: Unconditional sample for the GDDS Uniform model trained on OpenWebText. <|endoftext|> the United Kingdom), in hopes that the U. president and his global leader will lodge a demand or promise to use its stated goal to sustain support to combat ISIL. (The United Arab Emirates responded for the first time.) On Saudi Arabia and its United States side on the matter, less than ever will affect the U.S. to longer series,” co-Nubal Hender explained. This kind of bullsrina zone is in the shape that Syria’s non-American interests may be in the West 2007. U.S recent political and economic quellings of the Egyptian mass-democratic regime, destabilizing Shiite and anti-Muslim militias will only escalate tensions between the core allies of Syria, Saudi Arabia, and Syria, independent of the central U.S., while maintaining a fair-minded, external foe whose leaders seek to assist them, as head of White House argued. Even if American and Pres. Kerry efforts succeed slack push, they are unlikely to continue to broker lobbying strategies. The best way to look at that is the case across the world that rather than accept the negative effects of Nazi method are much to completely out of reach for those who have demons. In fact, patients who are in progressanic – for their malign position limiting behavior may turn into a - or recommended - - for becausefama. Elimuting unsenate drugs as long as Western medical prohibition on marketing and other ministries have harm’s effect, with a perils annual mortality rate, despite often everywhere no-one makes patients plan for a continued suicide away. Indeed, suicide rates among the American population are abnormally low, and coronary myocardic degeneration remains at a preexisting rate the single highest.41. A third study conducted by the IBM School of Medicine also archives patient interpretation data for physicians and health researchers and encouraged the study. The study in July 2015 was published in Dec. 4 of the Proceedings of the National Academy of Sciences, which was produced this year. The authors responded with an order of published papers published in each major journal for the study of hypertension. The journal’s published papers were published online from Thursday, February 16 to Jan. 6, 2015 by Mawall and Sarah Leonard. “The 1VHP trials were the most selective Tract-based randomized trial to be administered under the ausp against the fact where they reported consistent that they were moderately popular in the internal health industry. But the motive of taking support for high-risk cardiovascular care tends to lead at even higher risk,” write Begojin Krishnan, who is the senior author of the New England Journal of Medicine. 41. B̈iopathological benefits of shoulder plans was again useful in triggering several potentially dangerous activity that could trigger strokes.Ḧe found that these outcomes cannot be altered because coronary artery consumption is harmful to the researchers, and his team among its patients to criticize the customary practice of nasation from shorter milliseconds of knee propelled toparse pain and flow from a higher-value knee plateau. In a study that took 36 weeks, Pro Medel’s 37 weeks of placebo caused s̈udden activities-̈-no orgasm for years of experience–and survive, and decreases were detected at ≤ 191 levels of cancer-seriousctal core actal. Backstroke-induced metabolic syndrome is a major shift in vascular function to dependence, which may cause time distances among Traitor intensive coronary artery consumption more associated with chronic stroke-staffing,n̈otes Schuning, the row center in the study. T̈his positively affected the vascular system almost all the time, and it represents his lower, older, rested region.4̈2 Despite by scientists its efforts to reduce coronary artery injury, concussion therapy impacted fatal deaths among documented patients with Naondigran, who suffered increased cardiovascular toxicity and significantly increased strokes among the patients investigated the November 1 to 12 June 2015. The efficacy of placebo The scope of the study for shoulder disease was completely triletified when the study showed moderate improvements in coronary artery activity. The main test of effectiveness was the use of alcohol-free root oils for impact and the intervention. free-toverorption properties also contained wide variety of complimentary placebo techniques. Only a run on the annual National Study of Clinical Study, significant clinical correlation was achieved by the study 1 trial that did not show the effects of treatment from their respective programmes. When fails all studies measuring menstrual side effects, another systematic review showed that one trial showed a linear scheme of the retrospective performance across knowledge units 1VHP. Modeling was predicted to be followed for each stroke due to a health insurance plan featuring RCT and Morse sequences over the day of visits. Thus the commensatory between risk of death and risk of strokes was reached, soon agreement with in the 321 1VHP trials involving practitioner using mathematical reasoning and the distribution provided by himself.44 Thus, septoristic treatments show a significant sum of the energy received in one dose of intervention; only a fraction of the energy is collected from the release of<|endoftext|> Table 11: Unconditional sample for the GDDS Gauss model trained on OpenWebText. <|endoftext|> stories went in to TV in 2014 on Abboun on Zahora and became a busy town, rapidly knit and destabilized to men of almost 800 soldiers. Most frequently, Abcindi. the Aba Biba, boss of his lawyers, in Nahida, became a woman who enjoys his wife Helen‘s penchant for writing and writing to the LA Times. Abbindi, the wrinkled young woman, is sent fishing for the beach for one part in the incredibly horrific story find on eyewitness accounts and videos involving an immigrant family and local high school friends in AinsZi, a coastal enclave of Felton, N.C., Liberia and Dominican Republic. The Cairo story of The Los Angeles Times magazine tells the Egyptian State government to Abi ’Sadei was a huge hoax. The Prison cells of the LahaAidan of Cairo National University, a local university for the United States, built military antennas for the Egyptian army to cover the Ghouta bombing in Idlib. The Egyptian University builds anti-Islamic propaganda, challenged the group of soldier to get their flack on a private plane, leading to a photographers – Abno Ueh – having portrayed in photographs of Muslims. One explanation of what had caused the mayhem is for what’s worse happened on the covers and does not get to interest. That, they says, plenty of telefilmed spy community, of fear to beding in to the many stories and photos in the Abnot Ueh story. A little apart, however, the idea of the investigation was so near to this sites it became obvious that some that those individuals were presented for speculative speculation on that search result - and it is the way that has remained hidden for almost a year. It was not just a disturbing problem in trying to relate to personal data. Artificial intelligence is no confident in ability to pull a plot for it was as difficult to it was on a list the opponent has been to it. Research is to prone as it say to when we have requests in Google searches, they inevitably do asked to American espionage who care. Now they were looking for evidence pointing to action - to try to avoid that, and simply to try to get the first, or simply to get to the point for conjecture. Weaknesses could turn to brilliance on the very most dramatic weeks - power and intelligence are not mere real technologies, they are a incurable danger they have face maybe to a decade or we didn’t want to fully understand. The debate there seems interesting.Is it to several: Perhaps to be certain, the answer may sound much more than the midnight hour of 9/11. And in November 2015, Donald Trump’s first response on an outreach to try to infiltrate Twitter on Muslims was to open a hotline for U.S. for control of the World Trade Centre. And it is not a reference to what was happening in Saudi Arabia, as it is ISIS providing explicit terrorism was meant for attack that as detonate the target. It took much more room to laid the groundwork for it for many people looking to that - for anticipatory details that is not easier to find. there is an increased degree of concern when the group specifically targets terror-related activists and the National security adviser, Donald Trump tells audience gather in Saudi Arabia. to provide his account to Donald Trump as the story for put ubaida on Europe’s emerging refugee crises. – Kimberler Michael A- Doh6 Follow us on Twitter Google Whatsapp Tumblr Tumblr Pinterest Pinterest LinkedIn Google sharePARIS, New Jersey A building is deserted. It didn’t get cashing in for several years or it bombed to several.The biggest problems for that necessity of the huge building were dizors, patio cushions and wooden stitched.The journalist could hardly make up the case.A columnist in Ashraf Darrow, a reporter who is helped Donald Trump keep his watch as he is keen to become the non-American.This month, on Sunday a Faredan revealed the plan to embark in Montreal- doomed flights. “My plane had come up and I was actually only say to hell I’m have.” he told Time.“So I couldn’t do that It wasn’t anything for the metal gros and a communication tools I did,…It was sort of like an crazy …, ’cause, it got me on how I had to, [I Rediscovered that].I just wanted to prove I can have what to have on,” … French Transport Minister UEP S, Mariese Louse (C) and how Canada Led The War on fiscal Policy to a Future’S international Airport Centre. available Online at: 893/2313,349, online 915, (13,A), Part 2 The Economic Analysis Of Inflorating Railway Product Taxes (2013). French Transport Minister UEP S.Louse (C), Center to Transport Policy Group <|endoftext|> Appendix D Sampling procedures Setup and notation. We consider a discrete-time grid 1=tK>tK−1>⋯>t0=01=t_K>t_K-1>…>t_0=0 with decoding budget K∈32,64,128,256,512,1024K∈\32,64,128,256,512,1024\. The forward process is conditionally independent across positions (given x0x_0), so all formulas below apply token-wise and are executed in parallel for a length-n sequence. For any 0≤u≤10≤ u≤ 1, we denote by qu(⋅∣x0)q_u(· x_0) the forward marginal at time u starting from token x0∈x_0 , and by qt∣s(⋅∣xs)q_t s(· x_s) the forward transition from time s to t. At sampling time, the model provides a distribution over clean tokens, μθ(⋅∣xt,t)∈Δm _θ(· x_t,t)∈ _m (our “mean network”). D.1 Ancestral sampling Model-implied forward marginal. Given μθ(⋅∣xt,t) _θ(· x_t,t), define the model-implied forward marginal at any t∈[0,1]t∈[0,1] by qt(xt∣μθ)=∑x′∈Vμθ(x′∣xt,t)qt(xt∣x′).q_t(x_t _θ)= _x ∈ V _θ(x x_t,t)\;q_t(x_t x ). (26) Ancestral reverse kernel. The exact Bayes reverse conditional is q(xs∣xt,x0)=qt∣s(xt∣xs)qs(xs∣x0)qt(xt∣x0).q(x_s x_t,x_0)= q_t s(x_t x_s)\,q_s(x_s x_0)q_t(x_t x_0). All our samplers use the standard plug-in approximation x0≈μθ(⋅∣xt,t)x_0≈ _θ(· x_t,t), yielding the ancestral reverse kernel pθanc(xs∣xt)=q(xs∣xt,x0=μθ)=qt∣s(xt∣xs)qs(xs∣μθ)qt(xt∣μθ),p_θ^anc(x_s x_t)=q(x_s x_t,x_0= _θ)=q_t s(x_t x_s)\; q_s(x_s _θ)q_t(x_t _θ), where qs(⋅∣μθ)q_s(· _θ) and qt(xt∣μθ)q_t(x_t _θ) are defined in Eq. 26. Time-discretization. Given a discretization 0=t0<t1<⋯<tK=10=t_0<t_1<…<t_K=1 (decoding budget K), we use the plug-in Bayes/ancestral kernel pθanc(xtk−1∣xtk)=q(xtk−1∣xtk,x0=μθ)=qtk∣tk−1(xtk∣xtk−1)qtk−1(xtk−1∣μθ)qtk(xtk∣μθ),p^anc_θ(x_t_k-1 x_t_k)=q(x_t_k-1 x_t_k,x_0= _θ)=q_t_k t_k-1(x_t_k x_t_k-1)\; q_t_k-1(x_t_k-1 _θ)q_t_k(x_t_k _θ), (27) where the “mixture forward marginal” induced by the predictor μθ _θ is qu(xu∣μθ):=∑x′∈[m]μθ(xtk,tk)x′qu(xu∣x′),for u∈tk,tk−1.q_u(x_u _θ):= _x ∈[m] _θ(x_t_k,t_k)_x \,q_u(x_u x ), u∈\t_k,t_k-1\. Algorithm 4 summarizes the generic ancestral sampler. Below we give closed forms (Uniform / Absorb, already known in the literature (sahoo2024simple; shi2024simplified; ou2024your; schiff2024simple)) and the operator form (SIK). Algorithm 4 Generic ancestral sampler (token-wise, parallel over positions) 1: Input: Time grid 1=tK>⋯>t0=01=t_K>…>t_0=0; model μθ(⋅∣x,t) _θ(· x,t). 2: Sample xtK∼qtK(⋅)x_t_K q_t_K(·). 3: for k=K,K−1,…,1k=K,K-1,…,1 do 4: t←tk,s←tk−1t← t_k,\;\;s← t_k-1 5: μ←μθ(⋅∣xt,t)μ← _θ(· x_t,t) 6: Compute qs(⋅∣μ)q_s(· μ) and qt(xt∣μ)q_t(x_t μ) via Eq. 26 7: Compute pθanc(⋅∣xt)p_θ^anc(· x_t) via Eq. 27 8: Sample xs∼pθanc(⋅∣xt)x_s p_θ^anc(· x_t) 9: end for 10: return xt0x_t_0 D.2 Instantiations of ancestral sampling Case 1: Uniform diffusion. The forward marginal is qt(y∣x0)=αtδy=x0+1−αtm.q_t(y x_0)= _t\, _y=x_0+ 1- _tm. Moreover, for 0≤s<t≤10≤ s<t≤ 1, letting αt∣s≔αt/αs _t s _t/ _s, we have qt∣s(y∣x)=αt∣sδy=x+1−αt∣sm.q_t s(y x)= _t s\, _y=x+ 1- _t sm. Plugging these into Eq. 27 yields pθanc(xs∣xt)=[αt∣sδxs=xt+1−αt∣sm]∑x′μθ(xt,t)x′qs(xs∣x′)∑x′μθ(xt,t)x′qt(xt∣x′).p^anc_θ(x_s x_t)= [ _t s _x_s=x_t+ 1- _t sm ] _x _θ(x_t,t)_x \,q_s(x_s x ) _x _θ(x_t,t)_x \,q_t(x_t x ). (28) Case 2: Absorbing / masked diffusion. The forward marginal is qt(y∣x0)=αtδy=x0+(1−αt)δy=[MASK].q_t(y x_0)= _t\, _y=x_0+(1- _t)\, _y= [MASK]. For 0≤s<t≤10≤ s<t≤ 1 with αt∣s≔αt/αs _t s _t/ _s, the conditional kernel is qt∣s(y∣x)=αt∣sδy=x+(1−αt∣s)δy=[MASK].q_t s(y x)= _t s\, _y=x+(1- _t s)\, _y= [MASK]. Thus, pθanc(xs∣xt)=[αt∣sδxs=xt+(1−αt∣s)δxs=[MASK]]∑x′μθ(xt,t)x′qs(xs∣x′)∑x′μθ(xt,t)x′qt(xt∣x′).p^anc_θ(x_s x_t)= [ _t s _x_s=x_t+(1- _t s) _x_s= [MASK] ] _x _θ(x_t,t)_x \,q_s(x_s x ) _x _θ(x_t,t)_x \,q_t(x_t x ). (29) Case 3: Semantic-Informed Kernel (SIK). Let Ft(⋅∣x)F_t(· x) be a (column-stochastic) semantic jump kernel with Ft(x∣x)=0F_t(x x)=0, and let the (time-inhomogeneous) generator be Qt=f(t)(Ft−I).Q_t=f(t)\,(F_t-I). The exact transition operator is the time-ordered exponential Kt,s:=exp(∫stQτdτ),qt∣s(⋅∣xs)=(Kt,sδxs)(⋅),qt(⋅∣x0)=(Kt,0δx0)(⋅).K_t,s:=T \! ( _s^tQ_τ\,dτ ), q_t s(· x_s)\;=\;(K_t,s _x_s)(·), q_t(· x_0)\;=\;(K_t,0 _x_0)(·). Therefore the plug-in ancestral kernel Eq. 27 can be written as pθanc(xs∣xt)=(Kt,sδxs)(xt)∑x′μθ(xt,t)x′(Ks,0δx′)(xs)∑x′μθ(xt,t)x′(Kt,0δx′)(xt).p^anc_θ(x_s x_t)=(K_t,s _x_s)(x_t)\; _x _θ(x_t,t)_x \,(K_s,0 _x )(x_s) _x _θ(x_t,t)_x \,(K_t,0 _x )(x_t). (30) Practical computation and difficulty. For SIK, each factor in Eq. 30 is significantly harder to access than in the uniform and absorbing cases. The bridge term (Kt,sδxs)(xt)(K_t,s _x_s)(x_t) requires a short-time forward transition between two arbitrary tokens, while the numerator and denominator require evaluating the mixture marginals qs(⋅∣μθ)q_s(· _θ) and qt(xt∣μθ)q_t(x_t _θ), i.e. applying the forward operators Ks,0K_s,0 and Kt,0K_t,0 to many latent candidates weighted by μθ _θ. In our implementation, these quantities are approximated through uniformization-based matrix-vector products with caching across timesteps and blocks. This makes ancestral decoding feasible, but also substantially more delicate than in the closed-form uniform and absorbing settings. Empirically, this is reflected by the fact that GDDS-SIK models can achieve very strong validation losses, while the corresponding ancestral samplers remain difficult to calibrate and, in our current experiments, do not yet outperform the GDDS-uniform and GDDS-absorb samplers reported in the main text. We therefore interpret the present SIK results as evidence that the model class is strong, but that sampling for semantic continuous-time kernels still requires additional work; the appendix ablations document this point. Table 12: GDDS-SIK sampling ablation on OpenWebText. We report decoding budget K, average sequence entropy, and Gen-PPL for unconditional samples. Natural OWT text typically lies around entropy 5.605.60–5.705.70 (zheng2024masked). K Entropy (↑ ) Gen-PPL (↓ ) 8 5.34 402.25 16 5.45 230.64 32 5.59 207.48 64 5.59 176.68 128 5.66 189.06 256 5.67 254.15 The ablation in Table 12 clarifies the current GDDS-SIK behavior. On the positive side, the sampler does reach the desired entropy range: from K=32K=32 onward, the generated entropy is already close to that of natural OWT text, and by K∈128,256K∈\128,256\ it lands squarely in the target regime. Qualitatively, the resulting generations also look reasonable; see the generated samples in Table 11. The difficulty is instead on the quality side. Unlike uniform and absorbing diffusion, where the ancestral kernel admits a closed form and each reverse step can be sampled directly, SIK requires approximating the time-ordered exponential through uniformization-based cached matrix-vector products. This has two drawbacks. First, it is slower, because each reverse step requires truncating a Poisson series and performing several matvecs, whereas ancestral sampling for uniform and absorbing diffusion is direct. Second, the approximation error appears to accumulate along the trajectory: Gen-PPL improves up to K=64K=64, but then worsens as K increases further. This suggests that once the discretization error is sufficiently small, the remaining operator-approximation error dominates and compounds across steps. In other words, we can already sample from the trained GDDS-SIK denoiser, but faithfully turning that denoiser into a strong ancestral sampler remains challenging. Future work: avoiding ancestral sampling for SIK and more. While ancestral sampling via Eq. 27 is conceptually simple, its SIK instantiation remains computationally expensive. Indeed, even with sparsity and caching, evaluating the plug-in ratios requires repeatedly approximating forward operators (e.g., Kt,0K_t,0 and Ks,0K_s,0) and/or bridge terms, for which uniformization-based matvecs dominate the runtime. A natural direction is therefore to develop adaptive sampling procedures that better exploit the strength of the trained denoiser without committing to full ancestral updates at every step. This is consistent with recent evidence that adaptive or confidence-based schedules can outperform standard ancestral decoding in diffusion language models (von2025scaling; nie2025large; kim2025train). For GDDS-SIK in particular, such methods are especially attractive because they need not rely on repeated explicit approximations of Kt,sK_t,s or of closed-form forward marginals. Ultimately, we aim at samplers (and corresponding training objectives, as snapshot-ELBOs already encourage) in which GDDS with semantic kernels is truly blind to the exact forward transition operators, thereby removing the need to approximate KtK_t in closed form in both training and decoding. Appendix E Architectural details for Campbell E.1 Campbell objective at the sequence-level : an any-order autoregressive objective Recall that 0=x01…x0nx_0=x_0^1… x_0^n is clean and t=xt1…xtnx_t=x_t^1… x_t^n is the noised sequence at time t. At a jump time τ=Tkℓτ=T_k of coordinate ℓ , we denote by τ−x_τ^- (resp. τx_τ) the sequence immediately before (resp. after) the jump. By construction, only coordinate ℓ changes at time τ, so τ−x_τ^- and τx_τ coincide at all positions j≠ℓj≠ , and the observed pair is zk−1ℓ=xτ−ℓz_k-1 =x_τ^- and zkℓ=xτℓz_k =x_τ . Let τθ(′,)R_τ^θ(x ,x) denote the reverse kernel on sequences of size n at time τ, interpreted as the conditional probability of the predecessor sequence ′x given the current sequence x. Since consecutive states along ω differ in exactly one coordinate, τθ(τ−,τ)R_τ^θ(x_τ^-,x_τ) only concerns the predecessor token at the updated position ℓ given the post-jump state τx_τ, and for τ=Tkℓτ=T_k we have τθ(τ−,τ)=Rτθ(zk−1ℓ,zkℓ)R_τ^θ(x_τ^-,x_τ)=R_τ^θ(z_k-1 ,z_k ). Consider the conditional333We write τ for a generic realized jump time; in the products/sums below, τ always refers to τ=Tkℓτ=T_k for some (ℓ,k)( ,k). 0θ(0∣):=(0;)∏ℓ=1n∏k=1Nℓτθ(τ−,τ), p_0^θ(x_0 ω):=C(x_0; ω)\; _ =1^n _k=1^N R_τ^θ (x_τ^-,x_τ ), where (0;)C(x_0; ω) collects all terms independent of θ. Then −log0θ(0∣)- p_0^θ(x_0 ω) equals the event-wise cross-entropy sum up to an additive constant. Indeed, Jensen’s inequality yields the sequence-level ELBO log0θ(0)≥∼[0,1](⋅∣0)[log0θ(0∣)]⏟−(θ)+(0), p_0^θ(x_0)≥ E_ ω q_[0,1](· _0) [ p_0^θ(x_0 ω) ]_- L(θ)+C(x_0), where (0)=[log(0;)]=∑ℓ=1nC(x0ℓ)C(x_0)=E_ ω[ (x_0; ω)]= _ =1^nC(x_0 ) is independent of θ, expanding the token-level ELBO of Section 4.2 beyond the case n=1n=1. Here, the θ-dependent term is exactly the Campbell objective of Section 4.2, (θ)=∼[0,1](⋅∣0)[∑ℓ=1n∑k=1Nℓ−logτθ(τ−,τ)]. L(θ)\!=\!E_ ω q_[0,1](· _0)\!\! [ _ =1^n _k=1^N \!- _τ^θ(x_τ^-,x_τ) ]\!.\! (31) This mirrors an any-order autoregressive training objective, such as XLNet (yang2019xlnet). There, the factorization is induced by a random permutation of clean tokens, whereas in Eq. 31 it is induced by the time-ordered Poisson jump events along the diffusion path. Hence, the conditioning contexts τx_τ are noised, making it closer to the MPNet objective (song2020mpnet). However, our objective still remains fundamentally different; first, this path-wise formulation also applies beyond masked diffusion to general forward corruption processes. Second, it trains over all jumps encountered along the forward path, yielding ∑ℓ=1nNℓ _ =1^nN token-level supervision terms per clean sequence (rather than one). E.2 Two-stream architecture for the Campbell estimator A naive neural implementation (either XLNet/MPNet or a bidirectional-attention decoder only transformer) would require ∑ℓ=1nNℓ _ =1^nN NFEs per clean sequence to evaluate Eq. 31. This may be enormous for any reasonable sequence length n and hinders scalable training. This motivates a two-stream attention architecture that treats the whole path in [Nℓ]E[N ] NFEs in average (now independent of the sequence length n), i.e. in roughly one pass. Note that for masked diffusion, Nℓ=1N =1 for any ℓ , so it is exactly one pass in that case. Neural parametrization (from 0θ p_0^θ to θj_θ). Recall from Eq. 31 that the Campbell objective maximizes the path-conditioned product likelihood 0θ(0∣) p_0^θ(x_0 ω), whose θ-dependent part factorizes over the observed jumps of the forward path ω. Each factor is a sequence-level reverse probability τθ(τ−,τ)R_τ^θ(x_τ^-,x_τ) at some realized jump time τ. Since only one coordinate ℓ changes at time τ, τθ(τ−,τ)R_τ^θ(x_τ^-,x_τ) is the probability assigned to the pre-jump token at position ℓ given the post-jump state τx_τ. We parameterize these factors with a single neural network θ:n×ℝ≥0n→Δmnj_θ:V^n×R_≥ 0^n→ _m^n that outputs, for each context sequences =(τ)1≤ℓ≤n∈n×nx_ τ=(x_τ)_1≤ ≤ n ^n× n and jump times =(τ)1≤ℓ≤n∈ℝ≥0n τ=(τ)_1≤ ≤ n _≥ 0^n, a categorical distribution over predecessor tokens at positions 1≤ℓ≤n1≤ ≤ n: θℓ(τ,τ)=softmax(lθℓ(τ,τ))∈Δm,lθℓ(τ,τ)∈ℝm. j_θ (x_τ,τ)=softmax(l_θ (x_τ,τ))∈ _m, l_θ (x_τ,τ) ^m. For an observed jump of coordinate ℓ at time τ=Tkℓτ=T_k along ω, the predecessor token is zk−1ℓ=xτ−ℓz_k-1 =x_τ^- , so we set θℓ(τ,τ)zk−1ℓ=τθ(τ−,τ) j_θ (x_τ,τ)_z_k-1 =R_τ^θ(x_τ^-,x_τ). Hence, each Campbell term is exactly the cross-entropy loss contribution −logθℓ(τ,τ)zk−1ℓ- j_θ (x_τ,τ)_z_k-1 . Minimizing (θ) L(θ) effectively corresponds to the maximum likelihood on the path-wise model 0θ(0∣) p_0^θ(x_0 ω), i.e., to maximize 0θ(0) p_0^θ(x_0) (up to θ-independent factors). Two-stream architecture. We introduce a two-stream architecture based on the XL-Net/MPNet idea: an encoder-decoder transformer (vaswani2017attention) that combines ideas from XLNet (yang2019xlnet) and DiT (peebles2023scalable). The main challenge is to predict the pre-jump token xτℓ−ℓx _ _ ^- at an event time τℓ _ using only the clean set j:τj>τℓ\j: _j> _ \, which induces a sample-dependent (non-causal) factorization that cannot be enforced by a fixed left-to-right mask. We therefore introduce the rank rℓ:=rank(τℓ)r_ :=rank( _ ), defined as the position of τℓ _ in the sorted (decreasing) list of masking times (with ties broken so that r is a permutation), so that τj>τℓ⇔rj<rℓ _j> _ r_j<r_ . Inspired by XLNet, our two-stream architecture enforces this permutation-style factorization with two streams and rank-based attention masks: an encoder (content stream) builds contextual representations, while a decoder (query stream) predicts with masked queries and is restricted to attend only to keys j such that rj<rℓr_j<r_ , preventing leakage from x0ℓx_0 or yet-unrevealed tokens. The decoder is conditioned on continuous time via Adaptive LayerNorm (AdaLN), using a local embedding of τℓ _ to modulate its blocks and output, while the encoder remains strictly time-agnostic. This yields the one-pass training loop of Algorithm 5. Algorithm 5 Training with Campbell estimator 1: Input: distribution data q_data, network θj_θ, batch size B. 2: Sample a batch of sequences 0(1),…,0(B)∼datax^(1)_0,…,x^(B)_0 q_data. 3: For each sequence 0(b)x^(b)_0 and token position ℓ , run Algorithm 3 with t=1t=1 to sample Nℓ,(b)N ,(b) jumps and pairs (Tkℓ,(b),zkℓ,(b))k=1Nℓ,(b)\(T_k ,(b),z_k ,(b))\_k=1^N ,(b). 4: Compute the Campbell loss estimate Eq. 31 (θ)=1B∑b=1B∑ℓ=1n∑k=1Nℓ,(b)−logθℓ(Tkℓ,(b)(b),Tkℓ,(b))zk−1ℓ,(b), L(θ)= 1B _b=1^B _ =1^n _k=1^N ,(b)- j_θ (x^(b)_T_k ,(b),\,T_k ,(b))_z_k-1 ,(b), where t(b)x^(b)_t denotes the noised sequence at time t. E.3 Empirical results We train the two-stream architecture on both Text8 and OWT, with the same experimental setup as previously. We used an absorbing forward noising. Table 13: Validation perplexity of the two-stream architecture. We train the two-stream architecture on Text8 and OWT under the same experimental setup as in Appendix C, and report the BPC and validation perplexity. Text8 BPC OWT PPL ≤ 1.75 ≤ 76.07 Figure 6: Training loss stability on Text8. Train loss curves for AR, MDM, GDDS Absorb, and Campbell two-stream training, using the same setup as in Appendix C. While snapshot-based training exhibit noticeably higher short-term fluctuations, Campbell training yields a markedly smoother optimization trajectory, comparable to AR. We plot the training loss of the two-stream model “Campbell” in Fig. 6, as well as the training losses of MDM, GDDS Absorb and AR. Quantitatively, the standard deviation (std) of the training loss over the last 300300k steps is std≈3.08×10−2std≈ 3.08\!×\!10^-2 (AR), 1.63×10−11.63\!×\!10^-1 (MDM), 1.65×10−11.65\!×\!10^-1 (GDDS Absorb), and 2.92×10−22.92\!×\!10^-2 (Campbell). We remark that Campbell training yields a more stable optimization curve than snapshot-based objectives (MDM, GDDS, etc.), even though it conditions on full path information ω (as do AR models) rather than a single snapshot. Indeed, the Campbell estimator sums per-jump cross-entropies along the uniformization path(yielding at least n supervision terms per clean sequence, versus a single term for snapshot-based training) which reduces the variance of the learning objective across iterations. However, this stability does not translate into better likelihood performance under our architectural constraints (see Table 13). Indeed, compared to the results of Tables 1 and 2, we found that the two-stream architecture clearly underperforms.