Paper deep dive
Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy
Jingyuan Li, Xiaoyi Jiang, Yixuan Jiang, Wei Liu, Yi Zhu, Zuoqiang Shi, Pipi Hu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/24/2026, 3:07:18 AM
Summary
The paper introduces Mean-to-Score (M2S), a parameterization for discrete diffusion models that enforces Bayes realizability by predicting the clean-token posterior mean and mapping it to scores via an exact linear transformation. This approach addresses limitations in Score Entropy Discrete Diffusion (SEDD), where unconstrained score ratios may violate the bridge polytope, leading to negative pre-normalization weights and suboptimal sampling. M2S improves performance on CIFAR-10 and OpenWebText compared to SEDD, GIDD, and Neural CTMC baselines.
Entities (8)
Relation Signals (6)
M2S → enforces → Bayes Realizability
confidence 98% · To enforce Bayes realizability by construction rather than through post-hoc projection, we introduce mean-to-score (M2S)
M2S → improves → CIFAR-10
confidence 95% · M2S lowers test BPD from 3.173 to 3.129 and FID-50k from 42.83 to 28.09 on CIFAR-10.
M2S → outperforms → SEDD
confidence 95% · A 170M-parameter M2S model ... outperforms the evaluated pure-uniform SEDD ... at every tested sampling budget
M2S → outperforms → GIDD
confidence 95% · outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints
SEDD → failstoenforce → Bayes Realizability
confidence 90% · While positivity guarantees nonnegative reverse jump rates, it does not ensure Bayes realizability
M2S → recovers → MD4
confidence 85% · for absorbing-mask corruption, the resulting objective recovers MD4 exactly.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios. While positivity guarantees nonnegative reverse jump rates, it does not ensure Bayes realizability: ratios at a noisy state need not be jointly induced by any clean-token posterior under the forward kernel. The score-entropy loss has the correct population optimum but does not enforce this constraint away from it. In a trained pure-uniform SEDD checkpoint, roughly one quarter of complete score vectors violate the coordinate box, while more than half lie inside it yet remain materially incompatible with any valid posterior. Such violations can produce negative pre-normalization weights in finite-step sampling. Projecting raw scores onto the bridge polytope removes all observed negative weights and improves external generative PPL from $203.6$ to $175.1$ without changing the sampler. We introduce \emph{mean-to-score} (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map. The construction applies to any known coordinate-wise continuous-time Markov chain (CTMC) satisfying a mild support condition. For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly. In a controlled 28.4M-parameter CIFAR-10 comparison, M2S lowers test BPD from $3.173$ to $3.129$ and FID-50k from $\CifarSEDDFID$ to $\CifarMtwoSFID$. A 170M-parameter M2S model trained on about 262B OpenWebText token slots outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget, reaching generative PPL $143.3$ at 128 steps versus $183.6$ for the strongest pure-uniform baseline.
Tags
Links
- Source: https://arxiv.org/abs/2607.21372v1
- Canonical: https://arxiv.org/abs/2607.21372v1
Trouble viewing inline? Open PDF directly →
Full Text
91,586 characters extracted from source content.
Expand or collapse full text
Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy Jingyuan Li2,3,4 Xiaoyi Jiang1,411footnotemark: 1 Yixuan Jiang1,4 Wei Liu3 Yi Zhu1,2,4 Zuoqiang Shi1,2,4 Pipi Hu2,4 1Tsinghua University 2Beijing Institute of Mathematical Sciences and Applications 3Wuhan University 4MathonAI Equal contribution.Corresponding author. Abstract Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios. Although positivity ensures nonnegative reverse jump rates, it does not ensure Bayes realizability: at a fixed noisy state, all candidate score ratios must arise jointly from a single clean-token posterior under the forward kernel. We show that the score-entropy loss has the correct population optimum but does not enforce Bayes realizability away from it. In a trained pure-uniform SEDD checkpoint, roughly one quarter of complete score vectors violate the coordinate box, while more than half satisfy every coordinate bound but remain materially incompatible with any valid clean-token posterior. Although the corresponding continuous-time reverse jump rates remain nonnegative, these violations can induce negative pre-normalization weights in the finite-step sampler update. Projecting the checkpoint’s raw scores onto the bridge polytope removes all observed negative weights and lowers external generative PPL from 203.6203.6 to 175.1175.1 without changing the sampler. To enforce Bayes realizability by construction rather than through post-hoc projection, we introduce mean-to-score (M2S): the network predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map. The map applies to any known coordinate-wise continuous-time Markov chain (CTMC) satisfying a mild support condition. For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly. In a controlled 28.4M-parameter CIFAR-10 comparison, M2S lowers test BPD from 3.1733.173 to 3.1293.129 and FID-50k from 42.8342.83 to 28.0928.09. A 170M-parameter M2S model trained on approximately 262B OpenWebText token slots outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget, reaching generative PPL 143.3143.3 at 128 steps compared with 183.6183.6 for the strongest pure-uniform baseline. Figure 1: M2S occupies the Bayes-realizable subset of the scalar envelope, whereas SEDD can output any positive score. In a K=3K=3 uniform toy, panels (a)–(b) vary one score along s^(c)=(csy1⋆,sy2⋆) s(c)=(cs_y_1 ,s_y_2 ) around the shared optimum c=1c=1. The labels c−c_- and c+c_+ mark the limits imposed by the ρt _t-based coordinate envelope, while the inner dark-blue band is the smaller Bayes-realizable subset enforced by M2S and enlarged in the inset. Panel (a) shows the score-entropy excess and log-score gradient; panel (b) shows the terminal reverse-process TV error; and panel (c) shows the joint score region. Point A passes every scalar bound but admits no valid joint posterior, while point B lies on the M2S boundary. 1 Introduction Discrete diffusion models generate finite-valued data by reversing a continuous-time Markov chain (CTMC) that progressively corrupts a sample [1, 4]. We study pure-uniform corruption, which treats vocabulary states symmetrically without an absorbing mask token. For site i, let X0iX_0^i denote the clean token and μi⋆(x,t):=Pr(X0i=⋅∣Xt=x)∈ΔK−1 _i (x,t):=Pr(X_0^i=· X_t=x)∈ ^K-1 its posterior given the noisy sequence Xt=xX_t=x, where ΔK−1 ^K-1 is the probability simplex over a vocabulary of size K. Unlike absorbing-mask training, which reduces to a weighted clean-token cross-entropy on marked positions, uniform corruption does not reveal which positions changed and requires the reverse model to coordinate transitions across the full vocabulary [21, 23]. MD4 [23] already highlights a closely related parameterization issue for absorbing-mask diffusion: a freely parameterized score need not be induced by the conditional clean-token mean under the forward process. MD4 therefore predicts this mean and constructs the masked reverse model from it. We ask a vector-level question: whether one valid clean-token posterior induces the complete concrete-score vector predicted by SEDD. We call this property Bayes realizability. It reduces to MD4’s score–mean constraint for an absorbing mask, but under pure-uniform corruption it couples all K−1K-1 scores. In score-based CTMC models, each reverse jump rate uses a concrete score ratio pt(xi→y)/pt(x)p_t(x^i→ y)/p_t(x). SEDD [15] predicts each ratio with an exponentiated network output, guaranteeing positive scores and nonnegative continuous-time reverse jump rates. Positivity alone, however, does not ensure that all candidate score ratios arise from one clean-token posterior. Formally, at noisy state x and site i with k=xik=x^i, a candidate-score vector s=(sy)y≠ks=(s_y)_y≠ k satisfies Bayes realizability if s=Bt,kμs=B_t,kμ for some posterior μ∈ΔK−1μ∈ ^K-1. Under uniform corruption, any score induced by a clean-token posterior must lie in the coordinate box t,k:=[ρt,ρt−1]K−1C_t,k:=[ _t, _t^-1]^K-1. This restriction concerns Bayes realizability, not requirement for nonnegative continuous-time reverse rates, which only require sy>0s_y>0. The coordinate box is necessary but not sufficient for Bayes realizability: the complete score vector must lie in the strictly smaller bridge polytope t,k=Bt,k(ΔK−1)⊊t,kP_t,k=B_t,k( ^K-1) _t,k. Thus, constraining a SEDD head to positive outputs is insufficient to ensure Bayes realizability: its output may lie outside both the bridge polytope and even the coordinate box. Score entropy therefore has the correct population optimum but does not enforce Bayes realizability away from that optimum. Figure 1 visualizes the distinction for K=3K=3. The coordinate-box interval strictly contains its intersection with the bridge polytope, and panel (c) shows t,k⊊t,k⊊ℝ>02P_t,k _t,k _>0^2. Point A passes every coordinate bound but has a negative signed inverse component. Failure of Bayes realizability has measurable consequences at language scale. In a pure-uniform SEDD checkpoint, 6.69%6.69\% of 3.293.29 billion audited candidate scores leave the coordinate box. To test Bayes realizability during generation, we run this checkpoint with SEDD’s released sampler on 128 sequences and audit the unmodified score vector at every position and sampling step. Of these vectors, 25.02%25.02\% contain a coordinate violation, while another 56.23%56.23\% pass every scalar check but admit no valid joint posterior. During these runs, the sampler update encounters negative pre-normalization weights and operationally samples from their positive part. To isolate the effect, we project the scores onto t,kP_t,k for 1,024 paired sequences while holding the time grid, initial states, random-number stream, final denoising, and sampler code fixed. Projection removes all observed negative weights and lowers external generative PPL from 203.60203.60 to 175.07175.07. Rather than project scores at inference time, we enforce Bayes realizability in the model. For any known coordinate-wise forward kernel satisfying a mild support condition, the one-site clean-token posterior recovers every concrete score: si⋆(x,t;y)=[Pt(i)(y∣X0i)Pt(i)(xi∣X0i)|Xt=x],s_i (x,t;y)=E\! [ P_t^(i)(y X_0^i)P_t^(i)(x^i X_0^i)\, |\,X_t=x ], (1) where the integrand depends only on the clean token at site i. Mean-to-score (M2S) predicts μθi(xt,t)=softmax(xθi(xt,t)) _θ^i(x_t,t)=softmax(x_θ^i(x_t,t)) and applies the known linear map BtB_t to obtain the score. Because μθi _θ^i is a distribution, M2S enforces Bayes realizability. It changes the admissible off-optimum geometry, not the population target. The bridge uses only a one-site posterior, not a posterior over the full sequence. For uniform corruption it is injective on the simplex, has a closed form, and evaluates all scores in O(K)O(K) time per site. For absorbing-mask corruption, the same construction recovers the weighted clean-token cross-entropy of MD4 exactly. We also show that the conditional score-entropy risk is uniquely minimized at the true score (Proposition 4.3) and derive the rank condition rank([B+⊤,]⊤)=Krank([B_+ ,1] )=K under which this optimum uniquely recovers the clean-token posterior (Theorem 4.4); pure-uniform corruption satisfies this condition. Our experiments ask whether enforcing Bayes realizability matters for image and language generation. On 256-state MNIST, we compare M2S with SEDD: the two models share the architecture, forward process, objective, training budget, and sampler, and M2S improves FID from 126.1±0.4126.1± 0.4 to 71.1±4.371.1± 4.3. On CIFAR-10, an augmentation-free 28.4M-parameter M2S checkpoint reaches a test BPD upper bound of 3.1293.129, compared with 3.1733.173 for SEDD under the same evaluation protocol; under the same 256-step Euler sampler, M2S also lowers FID-50k from 42.8342.83 to 28.0928.09. On OpenWebText [9], the best configuration of our 169.9M-parameter M2S model reaches generative PPL 143.3143.3 at 128 steps and outperforms the evaluated pure-uniform SEDD [15], GIDD [28], and Neural CTMC [14] baselines at all tested budgets. Figure 2 further compares M2S with pure-uniform SEDD through Bayes realizability and score-entropy audits, and adds a CIFAR-10 optimization trace under identical compute. Keeping the SEDD checkpoint fixed, we also project its scores onto t,kP_t,k during sampling using the simplex-constrained Euclidean projection in Eq. (11), lowering external GenPPL from 203.60203.60 to 175.07175.07 (Section 5.1). Figure 2: M2S enforces Bayes realizability and achieves lower final loss on both text and image generation. Panels (a)–(b) audit score realizability and sampler-weight validity. Panels (c)–(d) compare the score-entropy loss of M2S and SEDD on OpenWebText and CIFAR-10, respectively. Contributions. 1. We introduce joint Bayes realizability for discrete score vectors. A complete score vector satisfies Bayes realizability exactly when it is induced by one valid clean-token posterior. Positivity and even coordinate-wise feasibility do not suffice. Under uniform corruption, such vectors form a strict bridge polytope t,k⊊t,kP_t,k _t,k, and score entropy does not enforce this constraint away from its population optimum. 2. We derive M2S to enforce Bayes realizability. M2S maps a one-site clean-token posterior to all concrete scores through an exact kernel-dependent linear bridge. We establish score consistency and posterior recovery, derive the uniform-kernel form, and recover the MD4 objective under absorbing-mask corruption. 3. We show empirically that Bayes realizability matters. For a fixed pure-uniform SEDD checkpoint, simplex-constrained Euclidean projection removes negative sampler weights and lowers generative PPL from 203.6203.6 to 175.1175.1. On MNIST and CIFAR-10, M2S improves FID over SEDD under identical settings, while a 170M-parameter OpenWebText model outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC baselines at every tested sampling budget. 2 Related Work Discrete diffusion. Diffusion probabilistic models originate from progressively corrupting Markov chains [24]. Discrete variants use multinomial, structured, or absorbing transition kernels [12, 1, 3, 11, 30]. Continuous-time formulations learn reverse CTMC rates or more general denoising Markov dynamics [4, 25, 2]. Discrete score learning includes concrete and target-concrete score matching [16, 29], while SEDD learns marginal probability ratios with score entropy [15]. Recent scalable systems use score, clean-token denoiser, interpolating-kernel, or factorized-rate parameterizations [21, 23, 17, 28, 14]. M2S retains SEDD’s score-entropy CTMC and changes how the complete score vector is parameterized. Posterior and score parameterizations. Posterior parameterizations construct reverse dynamics from clean-token predictions, as in D3PM and subsequent reparameterized or absorbing models [1, 11, 30]. For absorbing corruption, RADD factors the concrete score through conditional clean-data probabilities [17], while MDLM and MD4 reduce their objectives to weighted clean-token cross-entropy and MD4 highlights score–mean consistency [21, 23]. GIDD interpolates between masked and uniform corruption [28], and concurrent work derives exact coordinate-level score–denoiser conversions for uniform diffusion [10]. These works characterize individual score coordinates or population objectives. M2S instead asks whether the complete SEDD score vector is induced by one clean-token posterior, audits this joint condition in trained checkpoints, and enforces it by construction. 3 Preliminaries Let =1,…,KV=\1,…,K\ be a finite state space. A time-inhomogeneous CTMC on V over [0,T][0,T] is specified by a rate matrix QtQ_t satisfying Qt(a,b)≥0Q_t(a,b)≥ 0 for a≠ba≠ b and ∑bQt(a,b)=0 _bQ_t(a,b)=0. Definition 3.1. For some start time s and end time t=s+Δt=s+ (t>st>s) as Δ→0 → 0, we have qt∣s(b∣a)=δa,b+Qt(a,b)Δ+o(Δ),q_t s(b a)= _a,b+Q_t(a,b) +o( ), (2) where QtQ_t is called the forward transition rate. The Kolmogorov forward equation dtqt=qtQt ddtq_t=q_tQ_t determines the finite-time transition kernel Pt∣sP_t s [4]. For the corruption processes considered here, we use the kernel in Definition 3.2, which covers both uniform and absorbing corruption. Definition 3.2. The cumulative transition probabilities of the CTMC are given by qt∣0(a∣z)=Cat(a;Pt(z,⋅)),Pt=αtI+βtπ⊤,q_t 0(a z)=Cat\! (a;P_t(z,·) ), P_t= _tI+ _t1π , (3) where αt+βt=1 _t+ _t=1, with α0=1 _0=1 and αT=0 _T=0, and π is a fixed distribution on V. For uniform corruption, π=/Kπ=1/K; for absorbing corruption, π=emπ=e_m, where eme_m is the one-hot vector for the mask state m. Here 1 is the all-ones vector. As t→Tt→ T, qt∣0(⋅∣z)→πq_t 0(· z)→π for every z∈z , so the reference distribution is pref=πp_ref=π. For sequences, the forward process acts independently across sites. With site-wise kernels Pt(i)P_t^(i) and rates Qt(i)Q_t^(i), qt(xt∣x0)=∏i=1LPt(i)(xti∣x0i)q_t(x_t x_0)= _i=1^LP_t^(i)(x_t^i x_0^i) and pt(x)=∑x0pdata(x0)∏j=1LPt(j)(xj∣x0j)p_t(x)= _x_0p_data(x_0) _j=1^LP_t^(j)(x^j x_0^j). Definition 3.3. For a sequence state x∈Lx ^L and a candidate token y≠xiy≠ x^i, let xi→yx^i→ y denote the sequence obtained by replacing site i of x with y. Whenever pt(x)>0p_t(x)>0, the concrete score and its corresponding reverse transition rate are si⋆(x,t;y)=pt(xi→y)pt(x),Q¯t(i)(xi,y∣x)=Qt(i)(y,xi)si⋆(x,t;y),s_i (x,t;y)= p_t(x^i→ y)p_t(x), Q_t^(i)(x^i,y x)=Q_t^(i)(y,x^i)\,s_i (x,t;y), (4) respectively [4, 15]. MDLM and GIDD use an x0x_0 parameterization, whereas SEDD directly predicts the concrete score in Definition 3.3. M2S instead predicts the site-wise clean posterior μi(xt,t)∈ΔK−1μ^i(x_t,t)∈ ^K-1, with [μi(xt,t)]z=Pr(X0i=z∣Xt=xt),[μ^i(x_t,t)]_z=Pr(X_0^i=z X_t=x_t), and maps it to the concrete score in Section 4. 4 Methodology This section presents the M2S bridge, its training loss, and its connection to MD4. All proofs are provided in Appendix A. Theorem 4.1. Under Assumption A.1, for any site i, candidate token y, and noisy state x with pt(x)>0p_t(x)>0, let π⋆(z)=Pr(X0i=z∣Xt=x)π (z)=Pr(X_0^i=z X_t=x). For every z∈supp(π⋆)z (π ), we have Pt(i)(xi∣z)>0P_t^(i)(x^i z)>0. Then si⋆(x,t;y)=[Pt(i)(y∣X0i)Pt(i)(xi∣X0i)|Xt=x]=∑z∈supp(π⋆)π⋆(z)Pt(i)(y∣z)Pt(i)(xi∣z).s_i (x,t;y)=E\! [ P_t^(i)(y X_0^i)P_t^(i)(x^i X_0^i)\, |\,X_t=x ]= _z (π )π (z)\, P_t^(i)(y z)P_t^(i)(x^i z). (5) Replacing the exact posterior in Theorem 4.1 with the neural network prediction μθi(x,t) _θ^i(x,t) defines the M2S score as sθ,i(x,t;y)=∑z=1K[μθi(x,t)]zPt(i)(y∣z)Pt(i)(xi∣z)=:(Bμθi)y.s_θ,i(x,t;y)= _z=1^K[ _θ^i(x,t)]_z\, P_t^(i)(y z)P_t^(i)(x^i z)=:(B _θ^i)_y. (6) For the uniform process, define the off-diagonal-to-diagonal kernel ratio as ρt=(βt/K)/(αt+βt/K)∈(0,1) _t=( _t/K)/( _t+ _t/K)∈(0,1). At a noisy input xtx_t, Equation (6) then simplifies, for each candidate y≠xtiy≠ x_t^i, to sθ,i(xt,t;y)=1+(ρt−1)[μθi(xt,t)]xti+(ρt−1−1)[μθi(xt,t)]y.s_θ,i(x_t,t;y)=1+( _t-1) [ _θ^i(x_t,t) ]_x_t^i+( _t^-1-1) [ _θ^i(x_t,t) ]_y. (7) All candidate scores in Equation (7) are computed in O(K)O(K) time. We next introduce the training objective. Theorem 4.2. For the uniform CTMC with αt=e−σ(t) _t=e^-σ(t) and σ˙(t)≥0 σ(t)≥ 0, let x0∼pdatax_0 p_data, t∼Unif[0,1]t [0,1], and xt∼qt(⋅∣x0)x_t q_t(· x_0). For y≠xtiy≠ x_t^i, define ri(x0,xt,t;y)=Pt(i)(y∣x0i)/Pt(i)(xti∣x0i)r_i(x_0,x_t,t;y)=P_t^(i)(y x_0^i)/P_t^(i)(x_t^i x_0^i). For s,r>0s,r>0, define h(s,r)=s−rlogs+rlogr−r=r(s/r−1−log(s/r))≥0h(s,r)=s-r s+r r-r=r (s/r-1- (s/r) )≥ 0, with equality if and only if s=rs=r, and extend this definition continuously to r=0r=0 by h(s,0)=sh(s,0)=s. Define the M2S objective ℒM2S(θ) _M2S(θ) =t,x0,xt[∑i=1L∑y≠xtiwt,i(xti,y)h((Bμθi)y,ri(x0,xt,t;y))], =E_t,x_0,x_t\! [ _i=1^L _y≠ x_t^iw_t,i(x_t^i,y)\,h ((B _θ^i)_y,r_i(x_0,x_t,t;y) ) ], (8) wt,i(xti,y) w_t,i(x_t^i,y) =Qt(i)(y,xti)=σ˙(t)K(y≠xti). =Q_t^(i)(y,x_t^i)= σ(t)K\; (y≠ x_t^i ). Let pθ,0p_θ,0 be the time-zero marginal obtained by initializing the reverse process from prefp_ref and replacing si⋆s_i with sθ,is_θ,i in its rates. Then x0∼pdata[−logpθ,0(x0)]≤ℒM2S(θ)+x0∼pdataDKL(qT(⋅∣x0)∥pref).E_x_0 p_data[- p_θ,0(x_0)] _M2S(θ)+E_x_0 p_dataD_KL\! (q_T(· x_0)\, \|\,p_ref ). (9) The following results show that minimizing this objective recovers the true score and the clean posterior. Proposition 4.3. For fixed t,x,i,yt,x,i,y with si⋆(x,t;y)>0s_i (x,t;y)>0, the conditional risk Rx(s):=[h(s,ri)∣Xt=x]R_x(s):=E[h(s,r_i) X_t=x] satisfies Rx(s)−Rx(si⋆)=h(s,si⋆)≥0R_x(s)-R_x(s_i )=h(s,s_i )≥ 0, with equality if and only if s=si⋆s=s_i . Theorem 4.4. For fixed t,x,it,x,i, let π⋆π be the clean posterior from Theorem 4.1, let +=y≠xi:wt,i(xi,y)>0Y_+=\y≠ x^i:w_t,i(x^i,y)>0\, and let B+B_+ contain the rows of B indexed by +Y_+. Assume si⋆(x,t;y)>0s_i (x,t;y)>0 for every y∈+y _+. Then μ minimizes the conditional M2S risk if and only if B+μ=B+π⋆B_+μ=B_+π . Moreover, B+B_+ is injective on ΔK−1 ^K-1 if and only if ker(B+)∩v∈ℝK:⊤v=0=0,equivalentlyrank[B+⊤]=K. (B_+)∩\v ^K:1 v=0\=\0\, \! bmatrixB_+\\ 1 bmatrix=K. (10) When this rank condition holds, the unique minimizer is μ=π⋆μ=π . For the uniform kernel, σ˙(t)>0 σ(t)>0 makes every candidate positively weighted, and αt∈(0,1) _t∈(0,1) makes the augmented matrix in Eq. (10) full rank. Therefore, μ⋆=π⋆=[eX0i∣Xt=x]μ =π =E[e_X_0^i X_t=x]. Beyond the realizable case, this exact-recovery result suggests a canonical repair for an arbitrary predicted score: project it onto the bridge image and then invert the bridge. The uniform bridge is injective on the probability simplex, so each realizable score vector corresponds to a unique posterior. Fix t,x,it,x,i, let k=xik=x^i, and let Bt,k∈ℝ(K−1)×KB_t,k ^(K-1)× K map a posterior to its scores for candidates y≠ky≠ k. Given any score vector s∈ℝK−1s ^K-1, define the posterior whose induced score is closest to s by μproj(s):=argminμ∈ΔK−1∥Bt,kμ−s∥22.μ^proj(s):= *arg\,min_μ∈ ^K-1 B_t,kμ-s _2^2. (11) This projection distinguishes coordinate-wise feasibility from joint Bayes realizability. Define the coordinate box and the bridge polytope by t,k:=[ρt,ρt−1]K−1,t,k:=Bt,k(ΔK−1).C_t,k:=[ _t, _t^-1]^K-1, _t,k:=B_t,k( ^K-1). (12) Equation (7) gives t,k⊆t,kP_t,k _t,k, with strict inclusion for K>2K>2. Hence the score space has the following exact disjoint decomposition: ℝK−1=ℝK−1∖t,k⏟coordinate-outside∪˙t,k∖t,k⏟joint-onlynon-realizable∪˙t,k⏟Bayes-realizable.R^K-1= R^K-1 _t,k_coordinate-outside\; ∪\; C_t,k _t,k_ subarraycjoint-only\\ non-realizable subarray\; ∪\; P_t,k_Bayes-realizable. (13) The coordinate envelope is therefore necessary but not sufficient: a vector can satisfy every scalar bound while its unique affine inverse has a negative posterior component. Appendix D evaluates this same decomposition on raw SEDD scores. To separate material violations from sign-level floating-point effects, it divides the middle class at εμ=10−6 _μ=10^-6 into material and numerical-boundary subclasses; their union corresponds to t,k∖t,kC_t,k _t,k under the computed-sign convention. Proposition 4.5. Let π⋆=Pr(X0i=⋅∣Xt=x)π =Pr(X_0^i=· X_t=x) and let s⋆=(si⋆(x,t;y))y≠k=Bt,kπ⋆s =(s_i (x,t;y))_y≠ k=B_t,kπ . For the uniform kernel with αt∈(0,1) _t∈(0,1), the projection in Eq. (11) uniquely recovers the clean posterior: μproj(s⋆)=π⋆.μ^proj(s )=π . (14) Proposition 4.5 guarantees exact recovery only at the population optimum. For an arbitrary learned score s, μproj(s)μ^proj(s) is a valid posterior but need not be the true one. We next show that, for the absorbing-mask process, the M2S objective in Eq. (8) reduces exactly to the MD4 loss [23]. Theorem 4.6. Let m be an absorbing mask. For every clean token z≠mz≠ m, let Pt(z∣z)=αtP_t(z z)= _t and Pt(m∣z)=1−αtP_t(m z)=1- _t, and let Pt(m∣m)=1P_t(m m)=1, with no transitions between distinct clean tokens. Suppose that each μθi _θ^i is a distribution over the clean vocabulary, so [μθi]m=0[ _θ^i]_m=0. Then ℒM2Sabs(θ)=ℒMD4(θ):=t,x0,xt[∑i:xti=m−α˙t1−αt(−log[μθi(xt,t)]x0i)].L_M2S^abs(θ)=L_MD4(θ):=E_t,x_0,x_t [ _i:\,x_t^i=m - α_t1- _t\, (- [ _θ^i(x_t,t)]_x_0^i ) ]. (15) 5 Experiments 5.1 Same-Checkpoint Score Repair To isolate Bayes realizability at inference time from training differences, we generate 1,024 paired sequences from the epoch-32 pure-uniform SEDD checkpoint with the same released sampler and coupled random-number stream. Replacing only sSEDDs^SEDD by its Euclidean projection onto the bridge polytope lowers external GenPPL from 203.60203.60 to 175.07175.07; the paired average-NLL difference is −0.1510-0.1510 with 95% bootstrap interval [−0.1774,−0.1251][-0.1774,-0.1251]. The complete protocol, the distinction between direct, reconstructed, and projected scores, and the partition relative to the bridge polytope are given in Appendix D. 5.2 Image and Language Generation We evaluate M2S on discrete image and language generation tasks. All models use the uniform forward process (Appendix B) with αt=1−t _t=1-t and βt=t _t=t. For a fair comparison, M2S adopts the same DiT [19] backbone and parameter count as the mainstream baselines; full hyperparameters are given in Appendix C. Image Generation: We train M2S on MNIST, representing each 28×2828× 28 image as a sequence over =0,1,…,255S=\0,1,…,255\. Using identical architectures, parameter counts, optimization hyperparameters, training budgets, and samplers, M2S reduces FID by more than 52 points relative to SEDD at every tested sampling budget, with full details provided in Appendix C.1. We also evaluate M2S on CIFAR-10 using a 28.4M-parameter U-Net with self-attention. Following the MD4 setup, we match its total training budget of 512M image presentations while using pure-uniform corruption and the M2S parameterization. Without data augmentation, the model reaches a test BPD upper bound of 3.1293.129, compared with 3.1733.173 for SEDD under the same evaluation protocol. Thus, M2S lowers test BPD by approximately 0.0450.045. With the same 256-step Euler sampler and coupled random-number stream, M2S also lowers FID-50k from 42.8342.83 to 28.0928.09 (a 34.4%34.4\% reduction); qualitative samples and the complete protocol are reported in Appendix C.2. It also improves over the D3PM and Campbell et al. results in Table 1. This result provides evidence that the Bayes-realizable parameterization remains effective beyond MNIST. Full training and evaluation details are given in Appendix C.2. Table 1: CIFAR-10 test BPD (↓ ). M2S and SEDD are evaluated under the same test protocol; the remaining baseline values are published results. ≤ denotes a variational upper bound. The M2S and SEDD estimates are rounded to three decimal places. M2S is trained without data augmentation. A dash denotes an unreported parameter count. Method # Params BPD (↓ ) Autoregressive PixelRNN [26] – 3.003.00 Gated PixelCNN [27] – 3.033.03 PixelCNN++ [22] 53M 2.922.92 PixelSNAIL [5] 46M 2.852.85 Image Transformer [18] – 2.902.90 Sparse Transformer [6] 59M 2.802.80 Method # Params BPD (↓ ) Pure-uniform discrete diffusion M2S (ours) 28.4M ≤3.129 ≤ 3.129 SEDD [15] 28.4M ≤3.173≤ 3.173 Absorbing-mask discrete diffusion D3PM Absorb [1] 37M ≤4.40≤ 4.40 Campbell et al. Absorb [4] 28M ≤3.52≤ 3.52 MD4 [23] 28M ≤2.75 2.75 Discrete-Gaussian diffusion D3PM Gauss + logistic [1] 36M ≤3.44≤ 3.44 Campbell et al. (τ ) [4] 36M ≤3.59≤ 3.59 Language Modeling: We compare M2S against SEDD [15], MDLM [21], GIDD [28], and Neural CTMC [14] on OpenWebText [9]. To ensure a fair comparison, M2S and all baselines (SEDD, MDLM, GIDD, Neural CTMC) use an identical backbone architecture (12-layer DiT-style Transformer, 768 hidden dim, 12 heads, ∼ 169M parameters); the only difference across methods is the parameterization of the reverse process and its associated loss. For GIDD we report results with punif∈0.0,0.1,0.2p_unif∈\0.0,0.1,0.2\, where punif=0.0p_unif=0.0 corresponds to a pure mask process. For evaluation, we draw 1024 unconditional samples from each model and score them with a pretrained Gemma2-9B model to obtain generative perplexity (PPL); each method is run with multiple sampling seeds and we report the best PPL across seeds. (a) (b) Figure 3: (a) Generative-PPL comparison between M2S and the evaluated baselines. (b) Generative-PPL comparison of M2S Euler and Bayes samplers under linear and cosine time grids. Table 2: Generative perplexity (↓ ) on OpenWebText for varying numbers of sampling steps. All checkpoints use DiT-style backbones at a comparable parameter scale. Bold: best overall per column; underline: best among rows with a reported 262B-token training budget. Type Method Train Toks Max Len 16 32 64 128 mask SEDD [15] 682B 1024 825.5825.5 337.9337.9 186.5186.5 127.2127.2 MDLM [21] 262B 1024 1432.81432.8 553.7553.7 301.6301.6 210.5210.5 GIDD (punif=0.0p_unif=0.0) [28] 262B 512 2773.12773.1 993.7993.7 529.6529.6 414.3414.3 mixture GIDD (punif=0.1p_unif=0.1) [28] 262B 512 702.0702.0 398.9398.9 270.8270.8 249.8249.8 GIDD (punif=0.2p_unif=0.2) [28] 262B 512 770.4770.4 430.1430.1 344.3344.3 293.0293.0 uniform Neural CTMC [14] 262B 512 578.3578.3 258.8258.8 189.7189.7 183.6183.6 SEDD [15] 262B 512 963.9963.9 353.2353.2 247.7247.7 204.1204.1 GIDD [28] 262B 512 2134.02134.0 455.6455.6 271.7271.7 226.0226.0 M2S (Bayes, cosine) 262B 512 254.6¯ 254.6 175.3¯ 175.3 152.6¯ 152.6 148.8¯ 148.8 Note: (1) The losses of SEDD (mask) and GIDD (punif=0.0p_unif=0.0) are equivalent to the MDLM loss. (2) Checkpoint and sampler details are given in Appendix C.3. Table 2 shows that M2S with Bayes sampling on the cosine grid achieves the best overall PPL at 16–64 steps (254.6254.6, 175.3175.3, and 152.6152.6) and the best PPL among 262B-token checkpoints at 128 steps (148.8148.8). Figure 3(b) compares the M2S sampling configurations and shows that Bayes sampling on the linear grid achieves the best 128-step PPL of 143.3143.3. Overall, M2S outperforms every evaluated uniform and mixture model at all four budgets. Against the strongest uniform baseline, M2S reduces PPL by 56%56\% at 16 steps (578.3578.3 to 254.6254.6) and by 22%22\% at 128 steps (183.6183.6 to 143.3143.3), demonstrating a consistent advantage across the full sampling range. At 128 steps, M2S ranks second overall only to SEDD (mask), which uses 682B training tokens compared with 262B for M2S. 6 Conclusion This work identifies Bayes realizability as a structural requirement for discrete score vectors. Positive SEDD scores define valid reverse CTMC rates, but the complete vector need not be induced by any clean-token posterior. We characterize this gap and introduce M2S, which predicts the one-site clean-token posterior and maps it to all candidate scores through the known forward kernel. The construction applies to known coordinate-wise kernels under a mild support condition, restricts uniform-corruption scores to the bridge polytope, and recovers MD4 under absorbing-mask corruption. Empirically, M2S improves FID over SEDD under identical MNIST and CIFAR-10 settings and outperforms the evaluated pure-uniform baselines on OpenWebText at every tested sampling budget. A fixed-checkpoint intervention further shows that projecting SEDD scores onto the bridge polytope removes the observed negative sampler weights and improves generative PPL. These results establish Bayes realizability as a practical design principle for score-based discrete diffusion. References [1] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg (2021) Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, Vol. 34, p. 17981–17993. External Links: Link Cited by: §1, §2, §2, Table 1, Table 1. [2] J. Benton, Y. Shi, V. De Bortoli, G. Deligiannidis, and A. Doucet (2024) From denoising diffusions to denoising Markov models. Journal of the Royal Statistical Society Series B: Statistical Methodology 86 (2), p. 286–301. Cited by: §2. [3] S. Bond-Taylor, P. Hessey, H. Sasaki, T. P. Breckon, and C. G. Willcocks (2022) Unleashing transformers: parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes. In Computer Vision – ECCV 2022, Lecture Notes in Computer Science, Vol. 13683, p. 170–188. External Links: Document, Link Cited by: §2. [4] A. Campbell, J. Benton, V. De Bortoli, T. Rainforth, G. Deligiannidis, and A. Doucet (2022) A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, Vol. 35, p. 28266–28279. External Links: Document, Link Cited by: Appendix A, §1, §2, Definition 3.3, §3, Table 1, Table 1. [5] X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel (2018) PixelSNAIL: an improved autoregressive generative model. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 864–872. External Links: Link Cited by: Table 1. [6] R. Child, S. Gray, A. Radford, and I. Sutskever (2019) Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. Cited by: Table 1. [7] L. Condat (2016) Fast projection onto the simplex and the ℓ1 _1 ball. Mathematical Programming 158 (1–2), p. 575–585. Cited by: §D.1. [8] J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra (2008) Efficient projections onto the ℓ1 _1-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, New York, NY, USA, p. 272–279. External Links: Document, Link Cited by: §D.1. [9] A. Gokaslan and V. Cohen (2019) OpenWebText corpus. Note: http://Skylion007.github.io/OpenWebTextCorpus Cited by: §1, §5.2. [10] S. Gourevitch, Y. Janati, D. Shariatian, U. Simsekli, E. Moulines, E. P. Xing, and A. Durmus (2026) Uniform diffusion models revisited: leave-one-out denoiser and absorbing state reformulation. arXiv preprint arXiv:2605.22765. Cited by: §2. [11] Z. He, T. Sun, Q. Tang, K. Wang, X. Huang, and X. Qiu (2023) DiffusionBERT: improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, p. 4521–4534. External Links: Document, Link Cited by: §2, §2. [12] E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling (2021) Argmax flows and multinomial diffusion: learning categorical distributions. In Advances in Neural Information Processing Systems, Vol. 34, p. 12454–12465. External Links: Link Cited by: §2. [13] A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §C.2. [14] J. Li, X. Jiang, F. Wen, W. Liu, R. Luo, Y. Zhu, Z. Shi, and P. Hu (2026) Neural continuous-time Markov chain: discrete diffusion via decoupled jump timing and direction. arXiv preprint arXiv:2604.15694. Cited by: §C.3, §1, §2, §5.2, Table 2. [15] A. Lou, C. Meng, and S. Ermon (2024) Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 32819–32848. External Links: Link Cited by: Appendix A, §C.3, §1, §1, §2, Definition 3.3, §5.2, Table 1, Table 2, Table 2. [16] C. Meng, K. Choi, J. Song, and S. Ermon (2022) Concrete score matching: generalized score matching for discrete data. In Advances in Neural Information Processing Systems, Vol. 35, p. 34532–34545. External Links: Document, Link Cited by: §2. [17] J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li (2025) Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations, External Links: Link Cited by: §2, §2. [18] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran (2018) Image transformer. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 4055–4064. External Links: Link Cited by: Table 1. [19] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4195–4205. External Links: Link Cited by: §5.2. [20] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. Technical report OpenAI. Cited by: §C.3. [21] S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov (2024) Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, Vol. 37, p. 130136–130184. External Links: Document, Link Cited by: §C.3, §1, §2, §2, §5.2, Table 2. [22] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma (2017) PixelCNN++: improving the PixelCNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, External Links: Link Cited by: Table 1. [23] J. Shi, K. Han, Z. Wang, A. Doucet, and M. K. Titsias (2024) Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, Vol. 37, p. 103131–103167. External Links: Document, Link Cited by: §C.2, §1, §1, §2, §2, §4, Table 1. [24] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 37, p. 2256–2265. External Links: Link Cited by: §2. [25] H. Sun, L. Yu, B. Dai, D. Schuurmans, and H. Dai (2023) Score-based continuous-time discrete diffusion models. In International Conference on Learning Representations, External Links: Link Cited by: §2. [26] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu (2016) Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 48, p. 1747–1756. External Links: Link Cited by: Table 1. [27] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu (2016) Conditional image generation with PixelCNN decoders. In Advances in Neural Information Processing Systems, Vol. 29, p. 4790–4798. External Links: Link Cited by: Table 1. [28] D. von Rütte, J. Fluri, Y. Ding, A. Orvieto, B. Schölkopf, and T. Hofmann (2025) Generalized interpolating discrete diffusion. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 61810–61843. External Links: Link Cited by: §C.3, §1, §2, §2, §5.2, Table 2, Table 2, Table 2, Table 2. [29] R. Zhang, S. Zhai, Y. Zhang, J. Thornton, Z. Ou, J. M. Susskind, and N. Jaitly (2025) Target concrete score matching: a holistic framework for discrete diffusion. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 76716–76753. External Links: Link Cited by: §2. [30] L. Zheng, J. Yuan, L. Yu, and L. Kong (2024) A reparameterized discrete diffusion model for text generation. In Conference on Language Modeling, External Links: Link Cited by: §2, §2. Appendix Contents Appendix A Proofs Assumption A.1 (Support condition). Scores and posteriors are evaluated only at states x with pt(x)>0p_t(x)>0. For every site i, candidate y, and clean token z in the site-i data support, Pt(i)(xi∣z)=0⟹Pt(i)(y∣z)=0.P_t^(i)(x^i z)=0\; \;P_t^(i)(y z)=0. Theorem 4.1. Under Assumption A.1, for any site i, candidate token y, and noisy state x with pt(x)>0p_t(x)>0, let π⋆(z)=Pr(X0i=z∣Xt=x)π (z)=Pr(X_0^i=z X_t=x). For every z∈supp(π⋆)z (π ), we have Pt(i)(xi∣z)>0P_t^(i)(x^i z)>0. Then si⋆(x,t;y)=[Pt(i)(y∣X0i)Pt(i)(xi∣X0i)|Xt=x]=∑z∈supp(π⋆)π⋆(z)Pt(i)(y∣z)Pt(i)(xi∣z).s_i (x,t;y)=E\! [ P_t^(i)(y X_0^i)P_t^(i)(x^i X_0^i)\, |\,X_t=x ]= _z (π )π (z)\, P_t^(i)(y z)P_t^(i)(x^i z). Proof of Theorem 4.1. For each clean token z, define Az:=∑x0:x0i=zpdata(x0)∏j≠iPt(j)(xj∣x0j),Pz:=Pt(i)(xi∣z).A_z:= _x_0:\,x_0^i=zp_data(x_0) _j≠ iP_t^(j)(x^j x_0^j), P_z:=P_t^(i)(x^i z). These quantities collect all contributions outside site i. They give pt(x)=∑zAzPz,pt(xi→y)=∑zAzPt(i)(y∣z).p_t(x)= _zA_zP_z, p_t(x^i→ y)= _zA_zP_t^(i)(y z). The one-site posterior satisfies Pr(X0i=z∣Xt=x)=AzPzpt(x).Pr(X_0^i=z X_t=x)= A_zP_zp_t(x). Let ⋆=supp(π⋆)=z:AzPz>0Z_ =supp(π )=\z:A_zP_z>0\. For every z∈⋆z _ , we have Pz>0P_z>0, so define g(z):=Pt(i)(y∣z)Pt(i)(xi∣z)g(z):= P_t^(i)(y z)P_t^(i)(x^i z) on ⋆Z_ . Expanding the conditional expectation over its support gives [g(X0i)∣Xt=x] [g(X_0^i) X_t=x] =∑z∈⋆π⋆(z)g(z) = _z _ π (z)g(z) =∑z∈⋆AzPzpt(x)Pt(i)(y∣z)Pz = _z _ A_zP_zp_t(x) P_t^(i)(y z)P_z =1pt(x)∑z∈⋆AzPt(i)(y∣z). = 1p_t(x) _z _ A_zP_t^(i)(y z). If z∉⋆z _ , then either Az=0A_z=0 or Pz=0P_z=0. In the latter case, whenever Az>0A_z>0, the token z occurs in the clean support at site i, so Assumption A.1 implies Pt(i)(y∣z)=0P_t^(i)(y z)=0. Thus AzPt(i)(y∣z)=0A_zP_t^(i)(y z)=0 outside ⋆Z_ , and the restricted sum can be extended to all z. Hence [g(X0i)∣Xt=x]=∑zAzPt(i)(y∣z)pt(x)=pt(xi→y)pt(x).E[g(X_0^i) X_t=x]= _zA_zP_t^(i)(y z)p_t(x)= p_t(x^i→ y)p_t(x). The first equality above is the posterior expansion in the theorem, and the last equality is the concrete score si⋆(x,t;y)s_i (x,t;y). ∎ Theorem 4.2. For the uniform CTMC with αt=e−σ(t) _t=e^-σ(t) and σ˙(t)≥0 σ(t)≥ 0, let x0∼pdatax_0 p_data, t∼Unif[0,1]t [0,1], and xt∼qt(⋅∣x0)x_t q_t(· x_0). For y≠xtiy≠ x_t^i, define ri(x0,xt,t;y)=Pt(i)(y∣x0i)/Pt(i)(xti∣x0i)r_i(x_0,x_t,t;y)=P_t^(i)(y x_0^i)/P_t^(i)(x_t^i x_0^i). For s,r>0s,r>0, define h(s,r)=s−rlogs+rlogr−r=r(s/r−1−log(s/r))≥0h(s,r)=s-r s+r r-r=r (s/r-1- (s/r) )≥ 0, with equality if and only if s=rs=r, and extend this definition continuously to r=0r=0 by h(s,0)=sh(s,0)=s. Define the M2S objective ℒM2S(θ) _M2S(θ) =t,x0,xt[∑i=1L∑y≠xtiwt,i(xti,y)h((Bμθi)y,ri(x0,xt,t;y))], =E_t,x_0,x_t\! [ _i=1^L _y≠ x_t^iw_t,i(x_t^i,y)\,h ((B _θ^i)_y,r_i(x_0,x_t,t;y) ) ], wt,i(xti,y) w_t,i(x_t^i,y) =Qt(i)(y,xti)=σ˙(t)K(y≠xti). =Q_t^(i)(y,x_t^i)= σ(t)K\; (y≠ x_t^i ). Let pθ,0p_θ,0 be the time-zero marginal obtained by initializing the reverse process from prefp_ref and replacing si⋆s_i with sθ,is_θ,i in its rates. Then x0∼pdata[−logpθ,0(x0)]≤ℒM2S(θ)+x0∼pdataDKL(qT(⋅∣x0)∥pref).E_x_0 p_data[- p_θ,0(x_0)] _M2S(θ)+E_x_0 p_dataD_KL\! (q_T(· x_0)\, \|\,p_ref ). Proof of Theorem 4.2. For u=s/r>0u=s/r>0, the inequality u−1−logu≥0u-1- u≥ 0 gives h(s,r)=r(u−1−logu)≥0h(s,r)=r(u-1- u)≥ 0, with equality only at u=1u=1. Theorem 3.6 of Lou et al. [15], based on the continuous-time ELBO of Campbell et al. [4], includes the target-only normalization r(logr−1)=rlogr−r( r-1)=r r-r. Its integrand is therefore exactly h(s,r)h(s,r). Since T=1T=1, the theorem gives, for every x0x_0, −logpθ,0(x0)≤ - p_θ,0(x_0)≤ ∫01xt∼qt(⋅∣x0)[∑x~≠xtQt(x~,xt)h(sθ(xt,t;x~),qt(x~∣x0)qt(xt∣x0))]dt _0^1E_x_t q_t(· x_0) [ _ x≠ x_tQ_t( x,x_t)\,h\! (s_θ(x_t,t; x), q_t( x x_0)q_t(x_t x_0) ) ]dt +DKL(qT(⋅∣x0)∥pref), +D_KL\! (q_T(· x_0)\, \|\,p_ref ), written here in our row-generator convention. The coordinate-wise generator has a nonzero off-diagonal entry only when x~=xti→y x=x_t^i→ y for some i and y≠xtiy≠ x_t^i. For this pair, Qt(x~,xt)=Qt(i)(y,xti),qt(x~∣x0)qt(xt∣x0)=Pt(i)(y∣x0i)Pt(i)(xti∣x0i)=ri(x0,xt,t;y).Q_t( x,x_t)=Q_t^(i)(y,x_t^i), q_t( x x_0)q_t(x_t x_0)= P_t^(i)(y x_0^i)P_t^(i)(x_t^i x_0^i)=r_i(x_0,x_t,t;y). For the uniform process, Qt(i)(y,xti)=σ˙(t)/KQ_t^(i)(y,x_t^i)= σ(t)/K for y≠xtiy≠ x_t^i. Hence the integral, after averaging over x0∼pdatax_0 p_data, is exactly ℒM2S(θ)L_M2S(θ) because t∼Unif[0,1]t [0,1]. Averaging the pointwise bound proves Eq. (9). ∎ Proposition 4.3. For fixed t,x,i,yt,x,i,y with si⋆(x,t;y)>0s_i (x,t;y)>0, the conditional risk Rx(s):=[h(s,ri)∣Xt=x]R_x(s):=E[h(s,r_i) X_t=x] satisfies Rx(s)−Rx(si⋆)=h(s,si⋆)≥0R_x(s)-R_x(s_i )=h(s,s_i )≥ 0, with equality if and only if s=si⋆s=s_i . Proof of Proposition 4.3. Condition on Xt=xX_t=x and write s⋆=si⋆(x,t;y)s =s_i (x,t;y). Since [ri∣Xt=x]=s⋆E[r_i X_t=x]=s by Theorem 4.1, Rx(s) R_x(s) :=[h(s,ri)∣Xt=x] :=E[h(s,r_i) X_t=x] =s−s⋆logs+Cx, =s-s s+C_x, where CxC_x is independent of s. Moreover, Rx′(s)=1−s⋆s,Rx′(s)=s⋆s2>0.R_x (s)=1- s s, R_x (s)= s s^2>0. Thus RxR_x is strictly convex and uniquely minimized at s⋆s , with Rx(s)−Rx(s⋆)=s−s⋆−s⋆logs⋆=h(s,s⋆).R_x(s)-R_x(s )=s-s -s ss =h(s,s ). ∎ Theorem 4.4. For fixed t,x,it,x,i, let π⋆π be the clean posterior from Theorem 4.1, let +=y≠xi:wt,i(xi,y)>0Y_+=\y≠ x^i:w_t,i(x^i,y)>0\, and let B+B_+ contain the rows of B indexed by +Y_+. Assume si⋆(x,t;y)>0s_i (x,t;y)>0 for every y∈+y _+. Then μ minimizes the conditional M2S risk if and only if B+μ=B+π⋆B_+μ=B_+π . Moreover, B+B_+ is injective on ΔK−1 ^K-1 if and only if ker(B+)∩v∈ℝK:⊤v=0=0,equivalentlyrank[B+⊤]=K. (B_+)∩\v ^K:1 v=0\=\0\, \! bmatrixB_+\\ 1 bmatrix=K. When this rank condition holds, the unique minimizer is μ=π⋆μ=π . For the uniform kernel, σ˙(t)>0 σ(t)>0 makes every candidate positively weighted, and αt∈(0,1) _t∈(0,1) makes the augmented matrix in Eq. (10) full rank, and hence μ⋆=π⋆=[eX0i∣Xt=x]μ =π =E[e_X_0^i X_t=x]. Proof of Theorem 4.4. By Theorem 4.1, Bπ⋆=si⋆(x,t;⋅).Bπ =s_i (x,t;·). Write wy=wt,i(xi,y)w_y=w_t,i(x^i,y) and let ℛx(μ)R_x(μ) denote the conditional M2S risk summed over y∈+y _+. Applying Proposition 4.3 to each candidate gives, for every μ with finite conditional risk, ℛx(μ)−ℛx(π⋆) _x(μ)-R_x(π ) =∑y∈+wyh((Bμ)y,(Bπ⋆)y) = _y _+w_y\,h ((Bμ)_y,(Bπ )_y ) ≥0. ≥ 0. Here every wyw_y and (Bπ⋆)y=si⋆(x,t;y)(Bπ )_y=s_i (x,t;y) is positive. Since h(s,r)=0h(s,r)=0 if and only if s=rs=r, equality holds exactly when B+μ=B+π⋆.B_+μ=B_+π . This proves the first claim. We next prove the rank characterization. If μ,ν∈ΔK−1μ,ν∈ ^K-1 satisfy B+μ=B+νB_+μ=B_+ν, then v=μ−νv=μ-ν satisfies B+v=0,⊤v=⊤μ−⊤ν=0.B_+v=0, 1 v=1 μ-1 ν=0. Thus the null-space condition in Eq. (10) implies v=0v=0 and hence μ=νμ=ν, so B+B_+ is injective on ΔK−1 ^K-1. For the converse, suppose there exists a nonzero v∈ker(B+)v∈ (B_+) with ⊤v=01 v=0. Let μ¯=1K,0<ε<1K∥v∥∞,μ±=μ¯±εv. μ= 1K1, 0< < 1K v _∞, μ^±= μ± v. The choice of ε ensures that every coordinate of μ±μ^± is positive, while ⊤v=01 v=0 gives ⊤μ±=11 μ^±=1. Hence μ+,μ−∈ΔK−1μ^+,μ^-∈ ^K-1 and μ+≠μ−μ^+≠μ^-. However, B+μ+−B+μ−=2εB+v=0,B_+μ^+-B_+μ^-=2 B_+v=0, so B+B_+ is not injective on ΔK−1 ^K-1. This proves B+ is injective on ΔK−1⟺ker(B+)∩ker(⊤)=0.B_+ is injective on ^K-1 (B_+)∩ (1 )=\0\. Finally, ker[B+⊤]=ker(B+)∩ker(⊤). \! bmatrixB_+\\ 1 bmatrix= (B_+)∩ (1 ). Because the augmented matrix has K columns, its kernel is trivial if and only if its column rank is K. This establishes both equivalences in Eq. (10). When they hold, B+μ=B+π⋆B_+μ=B_+π forces μ=π⋆μ=π . For the uniform kernel, σ˙(t)>0 σ(t)>0 makes every candidate y≠xiy≠ x^i positively weighted, so B+=B_+=B. Moreover, αt∈(0,1) _t∈(0,1) implies ρt∈(0,1) _t∈(0,1). Let k=xik=x^i and suppose μ,ν∈ΔK−1μ,ν∈ ^K-1 satisfy Bμ=BνBμ=Bν. For any η∈ΔK−1η∈ ^K-1, summing Eq. (7) over y≠ky≠ k gives F(η) F(η) :=∑y≠k((Bη)y−1) := _y≠ k ((Bη)_y-1 ) =(K−1)(ρt−1)ηk+(ρt−1−1)(1−ηk). =(K-1)( _t-1) _k+( _t^-1-1)(1- _k). The coefficient of ηk _k is (K−1)(ρt−1)−(ρt−1−1)=(ρt−1)((K−1)+ρt−1),(K-1)( _t-1)-( _t^-1-1)=( _t-1) ((K-1)+ _t^-1 ), which is nonzero for ρt∈(0,1) _t∈(0,1). Since Bμ=BνBμ=Bν implies F(μ)=F(ν)F(μ)=F(ν), it follows that μk=νk _k= _k. For each y≠ky≠ k, Eq. (7) then gives 0 0 =(Bμ)y−(Bν)y =(Bμ)_y-(Bν)_y =(ρt−1)(μk−νk)+(ρt−1−1)(μy−νy) =( _t-1)( _k- _k)+( _t^-1-1)( _y- _y) =(ρt−1−1)(μy−νy). =( _t^-1-1)( _y- _y). Because ρt−1−1≠0 _t^-1-1≠ 0, we obtain μy=νy _y= _y for every y≠ky≠ k. Thus μ=νμ=ν, proving that the uniform bridge is injective on ΔK−1 ^K-1; by the equivalence above, its augmented matrix has rank K. The unique minimizer is therefore μ=π⋆=[eX0i∣Xt=x]μ=π =E[e_X_0^i X_t=x]. ∎ Proposition 4.5. Let π⋆=Pr(X0i=⋅∣Xt=x)π =Pr(X_0^i=· X_t=x) and let s⋆=(si⋆(x,t;y))y≠k=Bt,kπ⋆s =(s_i (x,t;y))_y≠ k=B_t,kπ . For the uniform kernel with αt∈(0,1) _t∈(0,1), the projection in Eq. (11) uniquely recovers the clean posterior: μproj(s⋆)=π⋆.μ^proj(s )=π . Proof of Proposition 4.5. Because π⋆∈ΔK−1π ∈ ^K-1 and s⋆=Bt,kπ⋆s =B_t,kπ , choosing μ=π⋆μ=π in Eq. (11) gives objective value zero. Hence every minimizer μ μ satisfies Bt,kμ^=s⋆=Bt,kπ⋆B_t,k μ=s =B_t,kπ . The uniform bridge is injective on ΔK−1 ^K-1 when αt∈(0,1) _t∈(0,1) by Theorem 4.4, so μ^=π⋆ μ=π . Thus the minimizer is unique and equals the clean posterior. ∎ Theorem 4.6. Let m be an absorbing mask. For every clean token z≠mz≠ m, let Pt(z∣z)=αtP_t(z z)= _t and Pt(m∣z)=1−αtP_t(m z)=1- _t, and let Pt(m∣m)=1P_t(m m)=1, with no transitions between distinct clean tokens. Suppose that each μθi _θ^i is a distribution over the clean vocabulary, so [μθi]m=0[ _θ^i]_m=0. Then ℒM2Sabs(θ)=ℒMD4(θ):=t,x0,xt[∑i:xti=m−α˙t1−αt(−log[μθi(xt,t)]x0i)].L_M2S^abs(θ)=L_MD4(θ):=E_t,x_0,x_t [ _i:\,x_t^i=m - α_t1- _t\, (- [ _θ^i(x_t,t)]_x_0^i ) ]. Proof of Theorem 4.6. Masked sites. Fix a site with xti=mx_t^i=m and let ct=αt/(1−αt)c_t= _t/(1- _t). The bridge and conditional target reduce to sθ,i(xt,t;y)=ct[μθi(xt,t)]y,ri(x0,xt,t;y)=cty=x0i.s_θ,i(x_t,t;y)=c_t[ _θ^i(x_t,t)]_y, r_i(x_0,x_t,t;y)=c_t1\y=x_0^i\. Using h(s,0)=sh(s,0)=s, substitution into the inner score-entropy sum in Eq. (8) gives ∑y≠m[ct[μθi]y−cty=x0ilog(ct[μθi]y)+cty=x0ilogct−cty=x0i]. _y≠ m [c_t[ _θ^i]_y-c_t1\y=x_0^i\ (c_t[ _θ^i]_y)+c_t1\y=x_0^i\ c_t-c_t1\y=x_0^i\ ]. Because μθi _θ^i is supported on the clean vocabulary, ∑y≠m[μθi]y=1 _y≠ m[ _θ^i]_y=1. The first and last terms therefore cancel after summation, and the two occurrences of logct c_t also cancel. The remaining loss is −ctlog[μθi]x0i.-c_t [ _θ^i]_x_0^i. The Kolmogorov equation gives the clean-to-mask rate Qt(i)(y,m)=−α˙t/αtQ_t^(i)(y,m)=- α_t/ _t, independent of the clean candidate y. Therefore, Qt(i)(y,m)ct=−α˙tαtαt1−αt=−α˙t1−αt.Q_t^(i)(y,m)\,c_t= - α_t _t _t1- _t= - α_t1- _t. Thus, summing over masked sites and taking the expectation gives exactly Eq. (15). Unmasked sites. Now let xti=x0i=k≠mx_t^i=x_0^i=k≠ m. For every candidate y≠ky≠ k, Qt(i)(y,k)=0Q_t^(i)(y,k)=0: clean-to-clean rates vanish, and the absorbing state has no outgoing rate. Thus every candidate weight is zero, so unmasked sites make no contribution. This completes the reduction. ∎ Appendix B Algorithms B.1 Training Algorithm During training, a clean token z is corrupted according to the uniform kernel qt(y∣z)=αty=z+1−αtK,αt=1−t,q_t(y z)= _t1\y=z\+ 1- _tK, _t=1-t, (16) where y is the corrupted token, K is the vocabulary size, and t∈[0,1]t∈[0,1]. Algorithm 1 summarizes the training procedure. Given a clean sequence x0x_0 and a diffusion time t, we sample xtx_t from the forward kernel in Eq. (16). The model predicts the clean-token posterior, converts it to concrete scores through the M2S bridge, and is optimized with the loss in Eq. (8). Algorithm 1 M2S Training 1:model fθf_θ, data distribution pdatap_data, vocabulary size K, learning rate η 2:while not converged do 3: Sample x0∼pdatax_0 p_data and t∼Unif[0,1]t [0,1] 4: Corrupt x0x_0 with Eq. (16) to obtain xtx_t 5: Predict μθi=softmax(fθi(xt,t)) _θ^i=softmax(f_θ^i(x_t,t)) for all sites i 6: Compute sθ,i(xt,t;y)s_θ,i(x_t,t;y) for all i and y≠xtiy≠ x_t^i using Eq. (6) 7: Compute rir_i and wt,iw_t,i from the known forward process as in Eq. (8) 8: θ←θ−η∇θℒM2S(θ)θ←θ-η _θL_M2S(θ) 9:end while 10:return θ B.2 Sampling Algorithms M2S pairs Euler and Bayes updates with either a linear or cosine time grid. This gives four samplers: Euler-linear, Euler-cosine, Bayes-linear, and Bayes-cosine. For M sampling steps, let t0>t1>⋯>tMt_0>t_1>·s>t_M denote the reverse-time grid: tmlin=1−mM,tmcos=cos(πm2M),m=0,…,M,t_m^lin=1- mM, t_m^cos= \! ( π m2M ), m=0,…,M, (17) Let Δm=tm−tm+1 _m=t_m-t_m+1. Both grids start at t0=1t_0=1 and end at tM=0t_M=0. All samplers initialize Xt0X_t_0 from the uniform distribution and update all sites in parallel. At step m, the model predicts the clean-token posterior μθ,i _θ,i from the current sequence x=Xtmx=X_t_m and maps it to scores using Eq. (7). Euler sampling uses the reverse rates Rm,i(y)=Qtm(i)(y,xi)sθ,i(x,tm;y),y≠xi.R_m,i(y)=Q_t_m^(i)(y,x^i)\,s_θ,i(x,t_m;y), y≠ x^i. (18) Algorithm 2 M2S Euler Sampling (Linear or Cosine Grid) 1:model fθf_θ, steps M, grid type g∈lin,cosg∈\lin,cos\, vocabulary size K, sequence length L 2:Construct tmm=0M\t_m\_m=0^M from Eq. (17) 3:Sample Xt0i∼Unif(1,…,K)X_t_0^i (\1,…,K\) for all sites i 4:for m=0,…,M−1m=0,…,M-1 do 5: x←Xtmx← X_t_m and Δm←tm−tm+1 _m← t_m-t_m+1 6: Predict μθ,i _θ,i and compute scores using Eq. (7) 7: Compute Rm,iR_m,i using Eq. (18) 8: for all sites i in parallel do 9: pi(y)←ΔmRm,i(y)p_i(y)← _mR_m,i(y) for y≠xiy≠ x^i 10: pi(xi)←1−∑y≠xipi(y)p_i(x^i)← 1- _y≠ x^ip_i(y) 11: Sample Xtm+1i∼Categorical(pi)X_t_m+1^i (p_i) 12: end for 13:end for 14:return XtMX_t_M Bayes sampling draws directly from the finite-time posterior instead of discretizing the reverse rates. For the current token k=Xtmik=X_t_m^i, marginalizing over μθ,i _θ,i gives p~m,i(y)=∑z=1K[μθ,i]zqtm+1(y∣z)qtm∣tm+1(k∣y)qtm(k∣z). p_m,i(y)= _z=1^K[ _θ,i]_z q_t_m+1(y z)\,q_t_m t_m+1(k y)q_t_m(k z). (19) Algorithm 3 M2S Bayes Sampling (Linear or Cosine Grid) 1:model fθf_θ, steps M, grid type g∈lin,cosg∈\lin,cos\, vocabulary size K, sequence length L 2:Construct tmm=0M\t_m\_m=0^M from Eq. (17) 3:Sample Xt0i∼Unif(1,…,K)X_t_0^i (\1,…,K\) for all sites i 4:for m=0,…,M−1m=0,…,M-1 do 5: x←Xtmx← X_t_m 6: Predict μθ,i _θ,i from (x,tm)(x,t_m) 7: for all sites i in parallel do 8: Compute p~m,i(y) p_m,i(y) for all y using Eq. (19) 9: Normalize pm,i←p~m,i/∑yp~m,i(y)p_m,i← p_m,i/ _y p_m,i(y) 10: Sample Xtm+1i∼Categorical(pm,i)X_t_m+1^i (p_m,i) 11: end for 12:end for 13:return X0X_0 Appendix C Experimental Details C.1 MNIST Experiments We compare M2S and SEDD on 28×2828× 28 MNIST images quantized to K=256K=256 gray levels. For a fair comparison, both methods use the same width-64 U-Net architecture and parameter count, optimization hyperparameters (AdamW with a learning rate of 2×10−42× 10^-4), uniform transition kernel and forward process αt=e−t _t=e^-t, 50-epoch training budget and generate samples with the same sampler, while differing only in the training parameterization: M2S predicts the posterior mean, whereas SEDD predicts the log-score directly. We generate 5000 samples for each model and compute FID against the first 5000 MNIST test images. Table 3 compares the two methods with 50, 100, and 200 sampling steps while holding all other settings fixed. M2S lowers FID by more than 52 points at every budget. Table 3: MNIST FID (↓ ) across sampling budgets. Objective 50 steps 100 steps 200 steps M2S (ours) 72.8072.80 72.8772.87 73.4473.44 SEDD 127.18127.18 126.81126.81 126.19126.19 Figure 4 shows the first 64 uncurated samples from the 200-step runs. M2S produces clearer strokes and fewer isolated bright pixels than SEDD, consistent with the FID comparison. (a) MNIST test data (b) M2S, FID 73.473.4 (c) SEDD, FID 126.2126.2 Figure 4: Qualitative comparison on MNIST. C.2 CIFAR-10 Experiments Training setup. We follow the CIFAR-10 [13] setup of MD4 [23]. Each 32×3232× 32 RGB image is represented as a length-3072 sequence with 256 states per color channel. We use the same approximately 28M-parameter U-Net with self-attention, AdamW with learning rate 4×10−44× 10^-4 and weight decay 0.010.01. The warmup covers 25,600 image presentations, equal to 100 MD4 batch-256 updates and 6.25 of our large-batch updates, and is followed by cosine learning-rate decay. The model is trained without data augmentation. M2S retains the clean-token posterior head but uses pure-uniform corruption and maps its output to concrete scores as described in Section 4. MD4 trains for 2M updates with batch size 256. We use a global batch size of 4096 for 125,000 updates, matching its total budget of 512M image presentations (10,240 CIFAR-10-equivalent epochs). The resulting 28,427,520-parameter checkpoint achieves a CIFAR-10 test BPD upper bound of 3.1293.129. Re-evaluating SEDD under exactly the same test protocol gives 3.1733.173, so M2S lowers test BPD by approximately 0.0450.045. Optimization trace. The two parameterizations are trained with the same complete score-entropy objective: M2S maps posterior logits to a log-score through the uniform bridge, whereas SEDD predicts the log-score directly, after which both use the same time distribution, corruption kernel, token sum, and batch mean. We log the all-reduced raw objective over all 16 ranks at every optimizer step and convert step u to 4096u/50,0004096u/50,000 CIFAR-10-equivalent epochs. M2S starts higher and optimizes more slowly early in training, but its trailing 625-step mean makes its final crossover below SEDD at equivalent epoch 3963.63963.6 and remains lower thereafter. Over the final 5,000 updates, the mean raw loss is 6770.16770.1 for M2S and 6901.96901.9 for SEDD. This curve is an optimization diagnostic, distinct from the held-out BPD estimates above. Figure 5: M2S converges to a lower CIFAR-10 training objective. Panel (a) includes every optimizer step in the 512M-image-presentation run; panel (b) enlarges the final 2,048 equivalent epochs. Faint curves show raw per-step losses (subsampled only for rendering), and bold curves show trailing 625-step means, a 51.2-equivalent-epoch window. Both axes use equivalent epoch rather than wall-clock time or dataloader passes. Generation protocol. We compare the EMA weights at update 125,000 for M2S and SEDD. Both checkpoints have 28,427,520 parameters and were trained without augmentation using the same optimizer, schedule, global batch size, and 512M image-presentation budget. For each method, we generate 50,000 images with the same 256-step categorical Euler discretization, cosine time grid, eight-GPU rank assignment, per-GPU batch size 256, and seed 20260723. Every rank-local batch is independently seeded from its first global sample index, so corresponding M2S and SEDD images use the same initial noise and categorical random-number stream even after a restart. We do not apply a final clean-token projection, rejection, or sample filtering. The generation paths differ only in the learned model output and the corresponding reverse-rate construction: M2S maps a posterior through the bridge, whereas SEDD exponentiates a directly predicted log-score. Table 4: M2S improves both likelihood and sample quality in the controlled CIFAR-10 comparison. BPD is the four-pass variational upper bound on all 10,000 test images. FID-50k uses pytorch-fid v0.3.0 with 2,048-dimensional Inception pool-3 features against all 50,000 CIFAR-10 training images. Lower is better. Method Admissible score set Test BPD (↓ ) FID-50k (↓ ) M2S (ours) t,k=Bt,k(ΔK−1)P_t,k=B_t,k( ^K-1) ≤3.129≤ 3.129 28.0928.09 SEDD ℝ>0K−1R_>0^K-1 ≤3.173≤ 3.173 42.8342.83 (a) M2S, FID-50k 28.0928.09 (b) SEDD, FID-50k 42.8342.83 Figure 6: M2S suppresses the isolated chromatic artifacts visible in SEDD under a fully paired sampler. Each panel contains generated indices 0–63 from its corresponding 50,000-image FID run, in index order, without ranking, filtering, or manual selection. Corresponding positions use the same seeded random-number stream, 256-step Euler discretization, and cosine grid. Connection to Bayes realizability. Theorem 4.1 shows that the true score vector is induced by one clean-token posterior. Because M2S predicts that posterior in the simplex, its entire off-diagonal score vector lies in the bridge polytope t,kP_t,k from Eq. (12) at every network iterate. SEDD enforces positivity coordinate by coordinate but does not enforce this joint constraint. Under finite data, capacity, and optimization, the M2S parameterization therefore removes non-Bayesian score degrees of freedom and promotes a coherent reverse-rate field. In the controlled experiment, this structural restriction coincides with a 14.7414.74-point FID reduction (34.4%34.4\%), the lower test BPD in Table 4, and fewer isolated high-saturation pixels and local texture breaks in Figure 6. Both parameterizations contain the population-optimal score, so the theorem alone does not imply a universal FID ordering; the paired result is empirical evidence consistent with the proposed finite-training mechanism. C.3 OpenWebText Experiments M2S training. We train a 169.7M-parameter diffusion Transformer on OpenWebText with GPT-2 [20] tokenization and a maximum sequence length of 512. We use the uniform forward process and the M2S loss in Eq. (8), setting the time-sampling cutoff to 10−310^-3. The global batch size is 1584 across 24 H100 GPUs. We use AdamW with a peak learning rate of 5×10−45× 10^-4, 3200 linear warmup steps, cosine decay, gradient clipping at 1.01.0, and bf16 precision. The model is trained on 262B tokens. Figure 7: M2S training loss over 161.8k optimization steps. The gray curve shows raw loss recorded every 100 steps, the blue curve shows its trailing 5,000-step average, and the dashed line marks the end of warmup. GenPPL evaluation. For each method and sampling-step budget, we draw 1024 unconditional samples and score 512 tokens per sample with the same Gemma2-9B evaluator. For checkpoints with max_len=1024, we evaluate only the first 512 generated tokens; all other checkpoints generate 512-token sequences. We report GenPPL as the exponentiated average next-token NLL assigned by the evaluator. To evaluate the baselines with their intended sampling procedures, MDLM, GIDD, SEDD, and Neural CTMC each use the sampler specified in the corresponding paper [21, 28, 15, 14]. The sampling algorithms used for M2S are described in Appendix B. Generated samples are shown in Appendix E. Appendix D SEDD Bayes Realizability: Projection Repair, Violation Rates, and Sample Quality This appendix develops and evaluates projection repair for SEDD scores. We first distinguish affine reconstruction, which reveals whether a score has a valid clean-token posterior, from simplex-constrained projection, which maps an non-realizable score to the nearest Bayes-realizable one. We then measure Bayes-realizability violations during SEDD sampling. Finally, we compare sample quality before and after projection repair at a fixed checkpoint and describe the projected SEDD sampler. D.1 Projection Repair For current token k, time t, and an SEDD candidate-score vector s, we seek the valid clean-token posterior whose induced score is closest to s: μproj(s)=argminμ∈ΔK−1∥Bt,kμ−s∥22,sproj(s)=Bt,kμproj(s).μ^proj(s)= *arg\,min_μ∈ ^K-1 B_t,kμ-s _2^2, s^proj(s)=B_t,kμ^proj(s). Thus sproj(s)s^proj(s) is the Euclidean projection of s onto the set of scores produced by valid posteriors, t,k=Bt,k(ΔK−1)P_t,k=B_t,k( ^K-1). Euclidean projection onto the probability simplex is a standard optimization primitive [8, 7]. Our objective measures distance after the linear score map Bt,kB_t,k, so Section D.4 derives a specialized solver for the uniform bridge. The simplex constraint is what makes this projection nontrivial. The matrix Bt,k∈ℝ(K−1)×KB_t,k ^(K-1)× K has full row rank, so if μ were allowed to be an arbitrary real vector, every s∈ℝK−1s ^K-1 could be written exactly as Bt,kμB_t,kμ and the projection would leave s unchanged. Even after imposing ⊤μ~=11 μ=1, every s has a unique affine inverse, but this inverse may contain negative entries. Requiring μ~∈ΔK−1 μ∈ ^K-1, including nonnegativity, is what restricts the score to t,kP_t,k. For the uniform bridge, this affine inverse has a closed form. Let n=K−1n=K-1, dt=1−ρtd_t=1- _t, and bt=dt/ρtb_t=d_t/ _t. Then m(s)=∑y≠k(sy−ρt)dt(n+ρt−1),μ~k=1−m(s),μ~y=sy−ρt−dtm(s)bt.m(s)= _y≠ k(s_y- _t)d_t(n+ _t^-1), μ_k=1-m(s), μ_y= s_y- _t-d_tm(s)b_t. It is important to distinguish reconstruction from projection. The SEDD head produces sySEDD=exp(aθ,y),y≠k.s_y^SEDD= (a_θ,y), y≠ k. Applying the affine inverse above gives a unique signed vector μ~(sSEDD) μ(s^SEDD) with ⊤μ~=11 μ=1. Reapplying the bridge defines the reconstruction srec:=Bt,kμ~(sSEDD)=sSEDD.s^rec:=B_t,k μ(s^SEDD)=s^SEDD. The last equality is an algebraic identity on the full affine hyperplane, so srecs^rec is a numerical consistency check, not a repair. In particular, a negative entry of μ~ μ can coexist with srec=sSEDDs^rec=s^SEDD. Bayes realizability requires μ~∈ΔK−1 μ∈ ^K-1. Consequently, only the simplex-constrained projection changes a score outside the bridge polytope: μproj:=μproj(sSEDD),sproj:=Bt,kμproj.μ^proj:=μ^proj(s^SEDD), s^proj:=B_t,kμ^proj. Thus sproj=sSEDDs^proj=s^SEDD if and only if μ~∈ΔK−1 μ∈ ^K-1. Section D.4 derives the scalar-threshold solver used by the projected SEDD sampler. D.2 Bayes-Realizability Violation Rates We use three sets of samples for different purposes: four trajectories for the diagnostic in Figure 2(b), 128 trajectories to measure Bayes-realizability violations, and 1,024 paired sequences to evaluate projection repair. Because the three analyses use different units, their reported percentages should be interpreted separately. Table 5: Data used in the three SEDD score analyses. The first two rows inspect SEDD scores while leaving the generated trajectories unchanged. The third applies projection repair during sampling. Data Purpose Reported unit 4 SEDD trajectories Negative-weight diagnostic in Figure 2(b) Step-averaged candidate-score and position rates 128 SEDD trajectories Bayes-realizability violation rates 8,388,6088,388,608 position-step score vectors 1,024 paired sequences Sample quality before and after projection repair 67,633,15267,633,152 position updates and sequence quality For the first two analyses, we used the original SEDD scores throughout sampling. At each state and position, we recorded the complete score vector and checked whether every coordinate lay in the required interval and whether the complete vector was induced by a valid clean-token posterior. These checks did not alter the generated trajectories. Only the projection experiment in Section D.3 replaced sSEDDs^SEDD with sprojs^proj during sampling. Sampler-weight violations. We ran four length-512 sequences with the 128-step SEDD sampler. At each step we evaluated the SEDD score sSEDDs^SEDD and its projection on the same current states. Averaged over steps, 9.781%9.781\% of off-diagonal candidate scores violated the coordinate box, and 37.632%37.632\% of positions had at least one negative pre-normalization weight. Projection removed all observed negative weights, giving the diagnostic in Figure 2(b). For the 128-trajectory analysis, we used the partition in Eq. (13). To distinguish material violations from numerical effects near the boundary, we split t,k∖t,kC_t,k _t,k at εμ=10−6 _μ=10^-6. Table 6 reports the resulting four classes over 512 positions and 128 sampler steps per trajectory. Table 6: SEDD score vectors over 128 trajectories, using t,k=Bt,k(ΔK−1)⊊t,k⊊ℝ>0K−1P_t,k=B_t,k( ^K-1) _t,k _>0^K-1. The boundary row separates numerical from material violations. Position-step class Criterion Fraction Count Outside t,kC_t,k sSEDD∉t,ks^SEDD _t,k 25.0177%25.0177\% 2,098,6352,098,635 In t,k∖t,kC_t,k _t,k, material sSEDD∈t,k∖t,ks^SEDD _t,k _t,k and minμ~<−10−6 μ<-10^-6 56.2310%56.2310\% 4,716,9944,716,994 In t,k∖t,kC_t,k _t,k, boundary sSEDD∈t,k∖t,ks^SEDD _t,k _t,k and −10−6≤minμ~<0-10^-6≤ μ<0 18.7419%18.7419\% 1,572,1881,572,188 In t,kP_t,k sSEDD∈t,ks^SEDD _t,k 0.0094%0.0094\% 791791 Across position-steps, 25.0177%25.0177\% lay outside t,kC_t,k, 56.2310%56.2310\% were material violations in t,k∖t,kC_t,k _t,k, 18.7419%18.7419\% formed its numerical boundary band, and only 0.0094%0.0094\% lay in t,kP_t,k. The 56.2310%56.2310\% rate was counted directly because the marginal coordinate and posterior-sign violations overlap (1.2584%1.2584\%). D.3 Sample Quality after Projection Repair To isolate the effect of projection repair, we kept the pure-uniform SEDD checkpoint and sampler fixed and changed only the score passed to the sampler. It received either sSEDDs^SEDD or sproj=Bt,kμprojs^proj=B_t,kμ^proj. For each setting, we generated 1,024 sequences of length 512 with 128 sampling steps. Each pair started from the same initial noise and used the same random numbers and final denoising update. With projected scores, none of the observed pre-normalization sampler weights was negative. We decoded the samples with the GPT-2 tokenizer and measured external GenPPL using the same Gemma2-9B evaluator as in the main language experiment. Table 7 summarizes the comparison; confidence intervals are estimated by sequence bootstrap, with samples paired across the two settings. Table 7: Effect of projection repair at a fixed SEDD checkpoint. The checkpoint and sampler are identical in both rows; only the score input changes. Intervals are 95% sequence-bootstrap confidence intervals. Score input Average NLL External GenPPL 95% GenPPL interval sSEDDs^SEDD 5.316155.31615 203.60203.60 [196.64,210.45][196.64,210.45] sproj=Bt,kμprojs^proj=B_t,kμ^proj 5.165185.16518 175.07175.07 [169.10,181.19][169.10,181.19] Using projected scores lowered external GenPPL from 203.60203.60 to 175.07175.07, a 14.0%14.0\% reduction. The paired bootstrap consistently favored projection, and all generated sequences were distinct in both settings, so the gain was not explained by sequence collapse. Because the checkpoint and sampler were unchanged, this comparison isolates the effect of projection repair at inference time for this checkpoint. It does not imply that post-hoc projection is equivalent to training M2S; external GenPPL is also an external measure of sample quality, not the model’s likelihood. D.4 Projected SEDD Sampler The uniform bridge permits an exact reduction of the simplex-constrained projection to a scalar threshold. Let u=μ−ku= _-k, z=s−kSEDD−ρtz=s^SEDD_-k- _t1, n=K−1n=K-1, d=1−ρtd=1- _t, and b=d/ρtb=d/ _t. Since μk=1−⊤u _k=1-1 u, Eq. (7) gives minu≥0, 1⊤u≤1‖(bI+d⊤)u−z‖22. _u≥ 0,\ 1 u≤ 1 \|(bI+d11 )u-z \|_2^2. (20) For the objective in Eq. (20), define c=b2,h=d2(n+2ρt),gy=bzy+d∑v≠kzv.c=b^2, h=d^2 (n+ 2 _t ), g_y=bz_y+d _v≠ kz_v. The KKT conditions give uy=[gy−θ]+c.u_y= [g_y-θ]_+c. If the mass constraint is inactive, the scalar threshold satisfies ∑y≠kuy=θ/h _y≠ ku_y=θ/h with θ∈[0,h]θ∈[0,h]. If it is active, it satisfies ∑y≠kuy=1 _y≠ ku_y=1 with θ∈[h,maxy≠kgy]θ∈[h, _y≠ kg_y]. At the branch point θ=hθ=h, let mh=c−1∑y≠k[gy−h]+m_h=c^-1 _y≠ k[g_y-h]_+. The mass constraint is active exactly when mh>1m_h>1. In either branch, the mass on the left decreases monotonically while the target on the right is nondecreasing, so batched bisection solves all positions without sorting the vocabulary. Here [a]+=maxa,0[a]_+= \a,0\. The resulting projection is applied at every position before the standard SEDD update, as summarized in Algorithm 4. Algorithm 4 Projected SEDD Sampler 1:trained SEDD model, reverse-time grid t0>⋯>tMt_0>·s>t_M, terminal distribution 2:Sample Xt0X_t_0 from the terminal distribution 3:for m=0,…,M−1m=0,…,M-1 do 4: x←Xtmx← X_t_m 5: Evaluate the SEDD score siSEDDs_i^SEDD for every position i 6: for all positions i in parallel do 7: k←xik← x^i and μiproj←μproj(siSEDD) _i^proj←μ^proj(s_i^SEDD) using Eq. (11) 8: siproj←Btm,kμiprojs_i^proj← B_t_m,k _i^proj 9: end for 10: Apply the standard SEDD update with sprojs^proj to sample Xtm+1X_t_m+1 11:end for 12:Apply the standard final denoising update 13:return X0X_0 Appendix E Sample Generation The following passages are randomly selected excerpts from samples generated by the M2S model using 128 sampling steps. Sample 1 The European Union (EU) said the European Union would “fight” Russia’s aggression against ISIS with “love for neighbour” and achieve a “defaulted” nuclear agreement and prevent the launch of a nuclear weapon. Among the top pro-Moscow-Kremlin claims about the Minsk peace agreement were that in 2014, however the war a far east pro-Russian separatists had stopped their long and violent clashes with Russia. That was still false now, the EU leaders said in a joint statement delivered by German Chancellor Angela Merkel late on Monday. They claimed that ceasefire negotiations would continue after 2014, but added that their “government has to compromise,” before rushing to Minsk. This came during a brief discussion with President Merkel at a joint news conference of the Joint Forces Committee of the Russian Aerospace Forces. Sample 2 A Republican proposal to change Senate rules would force the Senate budget in line for 10 percent, but under current law the regs spending is $77 million. The budget proposal submitted on Tuesday by Sen. Marco Rubio, R-Fla., the Senate’s third stiffer member and didn’t include the first shortfall, a Republican said, one that he now faces would cost a Republican incumbent $3,000 in the U.S. Senate, the unnamed Republican said. While Senate Democrats were unavailable for comment about the Rubio proposal, the Senate’s budget from 2009 to fiscal 2013 was $120 million, with Senate Majority Leader Harry Reid, D-Nev., calling for savings to narrow the gap. Whitehouse communications director Matt Lunt responded, saying that while any new bill would be pork, full funding would have been required, if the usual funds made financial sense. Leavint Reid, D-Nev., had not responded to questions about the proposal. According to the Senate, Rubio would have requested $14 million—a sum that became somewhat problematic in 2011. In the final 15 days that he had given into campaign finance, he closed a public college and policy research board, which had no students. The board of education was appointed by Obama. Sample 3 Apple has revealed a new 5K tablet with a redesigned, bezel-less Touch Cover design. It’s a bit cumbersome and a little more of a departure than the first redesign will be expected. It looks harmless, especially considering launching the invitation ring on the new 5800 later this year, with another refreshed version soon. What we should see in the huge next iPhone 5S is a tablet stand, costing $420,000–$850,000, along with a full top shelf smartphone. Such mini-boxes will serve them the power down and thus the memories, and possibly drive ports are all possible. You’d most likely wish they’d wrap this thing up, rather than with their own logo but at least that’s what we can expect. Apple will also offer the keyboard that pairs with it like you’re forced to the top of its sides. Obviously we find that as a nice gesture and you’d think that wouldn’t have to be available during launch too, but if you take the $300–$400, the setup will fit very nicely. Apple has also prepared a new iPad, just so you can have it with it too, but this idea is a myth. Not bad, since the iPhone offers a 4K resolution to 4k video and up to 12GB should be very respectable, but its not why the phone comes with Android 4.1/2.0 OS. Stay tuned for more hands-on details about the latest, and we’l report back to iPad TV Line in the future for more info. Sample 4 Junior Robinson is a genius. He’s always been blessed with natural size and speed. Lemon speed is about any player. But the speed he cultivated in the offseason, and he has averaged just six minutes in practice ice in a season starting the past three is at all impressive. And it’s not something you can’t. People can carry a a couple of minutes in five preseason games, able to attend a freeze from assembling the NHL roster. They are versatile, guys. The 24-year-old former 18th overall selection chose Detroit as the first-round pick of the 2012 NHL Pending Draft, two years after averaging 25 points in 24 games in various stints at 82 points, and starting since this past October, for Detroit, in the same spot some of his most promising stints in short-re 30 games, and 24 points in 27, for a rookie season, he performed very well. He returned this offseason, maximizing his own potential by one season at the NHL—subject to arbitration until 2012 at his due date. In order for the Red Wings to make use of roster room, they need just a couple of 25-man reinforcements in their rotation, without shipping John O’Neil or others for a roster spot. Sample 5 The year is once again in 2015, as the number of layoffs has been a record year high. Consequently, the banking giant Chase & Chase Bank recently reported that the sector had lost more than 19,000 jobs over the past three years. Read more Markers, who are criticised for feeling a stagnation, usually cite strong employment growth trends to speak to its current trends. “Today, the unemployment rate is down by 330 per cent since 2010,” said in its annual report. “And that depends on the shifts in opportunities in business.” However, in 2014–2015, one Fargo Ontario office employing 17,000 from was downsized—by 5,200—it backtracked this. It eliminated more than two-thirds of production, filling out the need among its banking sector employees. Earlier, the Bank had hinted that it may tighten its finances more. It released evidence that a low in wage workers caused the unemployment cut by almost $14,000 to workers before a tax of 0.25 percent. This would push the GDP worse in 2015. The recovery could not have been struck without these defects. As Fargo is expected to come out soon about the disinitation of its economy, the report, however, states the need to change infrastructure in a more sustainable way. The sale of Fargo operating Business Hummer division was taken in as the expense by example cited above. Sample 6 The main web apps on the desktop are 8GB allowed KB. They have a 40% download speed, and get a chunk of that from the Google Play Store, you can bet that Google also intends to bring the fun back into the store by having the same much functionality as in Firefox browsers like Chrome and Safari. On the mobile side, we have Hoothing, Magazines, Cycle Enhancement Browser, and Goggles apps, though this technically means you can upgrade later this year if you tried and missed before. Full apps can go on now too. Apps are also available on self-link for third-party developers to help with the Web apps as well as the Mobile settings also. However, such details were not available at this stage of the above launch. Chrome users, however, need to enable options even for Android. That’s been going on for many years now, with the Google Chrome desktop live app (which has been live for nearly a year now) and the mobile browsers on the U.S. and Chinese Web versions. Still enough stuff, you know? Apps first launched in early September will now be supported including on Google’s website, therefore making the Google Chrome app easier to sign up. There is still an open beta. All things are ready for everyone, always and soon. Google is testing out the redboards in China and other countries and is testing the locations available on the site. Sample 7 When you sign up for an AWS Elastic Load Network (ELT), get connected with your email or even the online language in your applications, it gets you go immediately. This may have only gotten so rare in the past, but the full power of computing is up for grabs in cloud computing and with its acquisition by PxCom, Inc. its cloud-delivered services has always been unusually designed with one important one: customer delivery in mind. “As the dominant broadband-only provider in the world, Seattle-based AWS has particular plans to modernize its computing infrastructure and give it real-time on-demand service and production-ready broadband customer delivery, with direct access to its delivery systems,” its agency says in a release. It also understands that it can be a layer used by its other large providers and coordinate their services to their customers’s data centers and connect them in their case at home, an indication that not all of the customer experiences in the cloud are necessarily different from each of the largest ISPs’ portfolio companies. P&Com, founded in partnership with Yormia and Escape Networks in January of 2014, is a provider of enterprise cloud service for “cloud” services AWS, Microsoft Azure, Amazon Dropbox, and Google Docs. Sample 8 Microsoft has announced it will announce and released of AI assistant in around the holidays. Last Thursday, Microsoft revealed that its AI assistant device, which was shown as a prototype over at CES, “is the successor to Google tablet- AI operating system with a text and call interface.” It will likely be paired with the upcoming Microsoft-powered Xbox One. Now there is another hugely interesting and interesting rumor floating around, and it got execs caught at CES anyway. An official Windows blog post is claiming that the new software will feature in users’ mobile systems, and it will provide a “personal voice assistant for device hubs with limited sharing space,” Microsoft said. Unlike Siri, it works with older storage devices, but there stored multiple SD cards or at least a volume on them. That’s obviously a waste of time, as your device can be gone out-of-state after being hooked up to the SD card. The AI device is all tied to Microsoft’s tech future, the AI+ operating system. Siri that will be capable of communicating with Internet-connected PCs, and the goal will be to work with iOS, Mac, and other groups of devices in common, require AI+ using operating system. We said, but wait. The full video is available here. μ Sample 9 With a new law pending before Congress, California faces an uphill labor battle, chief executive officer Xavier Becerra said recently. There is an uproar over California’s pay rate for the corporate behemoth that sells health care, which makes home appliances and other products. Western leaders on Capitol Hill still struggling with lower wages in countries that lean to union workers and rely on pliers’ compensation. But here were signs of a change of heart that amounts to a higher rate for California, partly because taxpayers in the state have invested heavily in company labor. The federal labor department said rates in its factories in Tennessee were also getting higher. “As there has been such a measure of growth in the states, labor costs have risen,” Mr. Cosino said in ODFO’s fact sheet March 20. Sample 10 The International Maoist-China forum called in return for “a world leader in international stability and international goods and services”. “China is proposing that its vision of state-denialing is a new machine, based on international cooperation, and its own market, rather than a country who would use its culture and opportunity to lead,” Hu Bingkao told participants, in an announcement of the convention on Saturday. Saturday, as the party celebrated the first anniversary of the start of the Maoist Commission, which was held in October when it warned of economic and social unrest in the country. The government, at the time felt there was no warning about the new trend, which included the rise of crack economy, ultra-high unemployment rates, levels of corruption, and the growth of militant groups. The convention took place just north of the Helfen, where also collected in some over 60 addresses take place annually. “Our society will be part of the discussion, but our ideological reason is simply the objective,” the Bingkao said. Meanwhile, Hu said People needed to grow the environment, dealing with drug dealing, and said that the methods to boost social stability was needed to respond to problems for the population.