Paper deep dive
One-Step Text Generation by Seq-Drifting
Jiayu Liu, Ruizhi Zhao, Tianyun Ji, Yee Whye Teh, Zhenya Huang, Ning Miao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/12/2026, 6:29:19 AM
Summary
The paper introduces Seq-Drifting, a framework for one-step text generation that adapts the drifting paradigm to discrete token sequences. Unlike traditional diffusion language models requiring multiple sampling steps, Seq-Drifting learns a direct map from Gaussian noise to text via a single forward pass. It employs context-dependent support sets for attraction and per-position repulsion to ensure coherent, diverse outputs. The method achieves competitive perplexity on LM1B and demonstrates applicability in translation, summarization, and reasoning tasks.
Entities (9)
Relation Signals (10)
Seq-Drifting → achieves → One-Step Generation
confidence 98% · The generator learns from this drift field during training and produces all tokens jointly in one pass at inference.
Seq-Drifting → evaluatedon → LM1B
confidence 95% · Empirically, Seq-Drifting achieves a generative perplexity of 36.01 on LM1B in one step
Seq-Drifting → uses → Repulsion
confidence 95% · while repulsion discourages collapse across samples and repetition within sequences.
Seq-Drifting → uses → Attraction
confidence 95% · At each position, attraction guides each output toward a set of contextually plausible token representations
Seq-Drifting → appliesto → summarization
confidence 90% · Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework
Seq-Drifting → appliesto → Translation
confidence 90% · Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework
Seq-Drifting → appliesto → Reasoning
confidence 90% · Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework
Repulsion → prevents → Repetition
confidence 90% · repulsion discourages collapse across samples and repetition within sequences.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion language models generate tokens in parallel, but still require multiple sampling steps to produce high-quality text. In this paper, we investigate whether a model can learn to generate an entire sequence in a single forward pass. We propose Seq-Drifting, a framework that applies drifting to sequences of continuous token representations and directly learns a map from Gaussian noise to text. Unlike existing DLMs that derive the supervision at each position directly from the observed token in a training sequence, Seq-Drifting builds a drift field whose position-wise targets adapt to the current generated context. At each position, attraction guides each output toward a set of contextually plausible token representations, while repulsion discourages collapse across samples and repetition within sequences. The generator learns from this drift field during training and produces all tokens jointly in one pass at inference. Empirically, Seq-Drifting achieves a generative perplexity of 36.01 on LM1B in one step and performs competitively in both unconditional and conditional generation. Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework beyond open-ended generation. These results suggest that drifting over token sequences offers a promising path toward effective one-step text generation. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.12879v1
- Canonical: https://arxiv.org/abs/2610.12879v1
Trouble viewing inline? Open PDF directly →
Full Text
71,700 characters extracted from source content.
Expand or collapse full text
One-Step Text Generation by Seq-Drifting Jiayu Liu Ruizhi Zhao Tianyun Ji Yee Whye Teh Zhenya Huang Ning Miao Abstract Diffusion language models generate tokens in parallel, but still require multiple sampling steps to produce high-quality text. In this paper, we investigate whether a model can learn to generate an entire sequence in a single forward pass. We propose Seq-Drifting, a framework that applies drifting to sequences of continuous token representations and directly learns a map from Gaussian noise to text. Unlike existing DLMs that derive the supervision at each position directly from the observed token in a training sequence, Seq-Drifting builds a drift field whose position-wise targets adapt to the current generated context. At each position, attraction guides each output toward a set of contextually plausible token representations, while repulsion discourages collapse across samples and repetition within sequences. The generator learns from this drift field during training and produces all tokens jointly in one pass at inference. Empirically, Seq-Drifting achieves a generative perplexity of 36.0136.01 on LM1B in one step and performs competitively in both unconditional and conditional generation. Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework beyond open-ended generation. These results suggest that drifting over token sequences offers a promising path toward effective one-step text generation. Our code is available at https://github.com/jyliu-ai/Seq-Drifting. 00footnotetext: 1Hong Kong Institute of AI for Science, City University of Hong Kong, 2Department of Data Science, City University of Hong Kong, 3Shenzhen University, 4State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, 5Department of Statistics, University of Oxford.00footnotetext: *Correspondence to: Ning Miao (ningmiao@cityu.edu.hk) Figure 1: Seq-Drifting achieves lower generative perplexity than prior few-step distilled DLMs, and is on par with or better than multi-step DLMs, with a 44–11×11× speedup over the few-step baselines and a 900900–2500×2500× speedup over the multi-step ones in wall-clock time. (Gen-PPL under GPT-2 Large; dataset: LM1B, T=128T=128. Detailed comparison in Figure 3 and Table 6.) 1 Introduction Autoregressive (AR) language models are the dominant paradigm for text generation and have shown strong capabilities across a wide range of domains (Radford et al., 2019; Brown et al., 2020). At inference, however, they generate one token at a time, making sequential decoding a fundamental latency bottleneck that grows with sequence length. Non-autoregressive alternatives such as diffusion language models (DLM) generate multiple tokens in parallel (Sahoo et al., 2024; Li et al., 2022; Hu et al., 2026; Lee et al., 2026), but they still rely on multiple refinement steps to produce high-quality samples. Distillation methods such as SDTT (Deschenaux and Gulcehre, 2025), Di4C (Hayakawa et al., 2025), and FMLM (Lee et al., 2026) reduce the number of steps, but generation quality remains limited at small sampling budgets. A key question, then, is whether we can directly learn high-quality one-step generation through a single forward pass from noise to text. Most generative models can be seen as a map from a noise vector to data. Because this map is highly multimodal, previous methods often need to decompose it into many small steps. Drifting (Deng et al., 2026) offers a different route, learning the map directly as a single forward pass in continuous domains. Rather than learning a trajectory that is unrolled at inference, a drifting model evolves its pushforward map during training. As the parameters are updated, each noise sample is carried a little closer toward the data distribution, a movement governed by a drifting field that vanishes once the generated and data distributions match. At convergence the map can reach a fixed point, so a single forward pass can turn noise into a sample without an iterative sampler at inference. Although drifting has shown promising results on images, transferring it to text is far from straightforward. A text sample is a sequence in which every position contains a token drawn from a finite vocabulary. Since valid text sequences are composed of discrete token choices, a continuous drift step will generally land between valid token embeddings rather than on one. The drift may therefore operate in regions containing no valid token. The space is also semantically non-smooth, as a single substituted token can overturn the meaning of a sentence or make it incoherent, which means nearness in embedding space does not necessarily imply nearness in meaning. One might hope to sidestep these issues by drifting a single sentence-level embedding, where the space is continuous. However, this only relocates the difficulty, because mapping such a vector back to a faithful token sequence is itself a hard sequence-decoding problem, and a single vector exposes none of the per-position targets a drift step needs. In this work we introduce Seq-Drifting, which adapts drifting to sequences of continuous token representations and learns the noise-to-text map in a single forward pass. Our core idea is to provide each position with supervision from a context-dependent set of plausible tokens. This support set anchors the drift to actual vocabulary tokens and identifies which choices are compatible with the current generated context. Seq-Drifting then attracts each output toward these tokens, allowing different noise inputs to select different valid alternatives without pulling them all toward an average representation. Because the support tokens depend on the generated context, the supervision couples token choices across positions and guides the generator toward coherent sequences. Seq-Drifting establishes a new framework for one-step text generation and brings three advantages. First, it generates a sample in a single forward pass, so its inference cost does not grow with a step budget as in multi-step diffusion and few-step distillation, while matching their quality at a fraction of the inference cost. Second, it addresses the ambiguity of one-step generation without relying on iterative refinement. Rather than assigning each noise input a unique target sequence, it allows different noise to naturally settle on their own coherent continuations while preserving multiple generation modes. Finally, it unifies two ways of learning the one-step map by constructing support sets either from corpus retrieval or scoring by a frozen language model, which opens the possibility of turning an existing autoregressive language model into a one-step generator. Empirically, we show that Seq-Drifting combines competitive generation quality with substantially lower inference cost. For example, in Figure 1, it reaches a generative perplexity of 36.0136.01 with an 11.2×11.2× wall-clock speedup over the strongest four-step distilled baseline, even outperforming several DLMs that use 10241024 sampling steps. This advantage extends to prefix-conditioned generation, such as machine translation, summarization, and reasoning, which broadens its applicability beyond language continuation. Overall, these results demonstrate the feasibility of directly learning a one-step text generator and point to a new direction for diffusion language models. 2 Related Work Diffusion language models (DLMs) provide an iterative non-autoregressive alternative to the token-by-token decoding of autoregressive models (Radford et al., 2019; Brown et al., 2020), and can be broadly categorized into discrete and continuous formulations (Li et al., 2025). Discrete DLMs operate directly over categorical tokens. D3PM (Austin et al., 2021) provides a general framework for discrete corruption, while MDLM (Sahoo et al., 2024) uses masking to learn token recovery from partially observed sequences. LLaDA (Nie et al., 2025) further demonstrates masked diffusion at the scale of large language models. Unlike standard irreversible unmasking, the uniform-state formulation of Duo (Sahoo et al., 2025) allows previously generated tokens to be revised. Block diffusion methods (Arriola et al., 2025; Arriola et al., 2026) instead combine parallel generation within blocks with autoregressive conditioning across blocks. However, parallel per-position predictions do not by themselves capture the joint uncertainty of tokens updated simultaneously, making inter-token dependencies particularly important when sampling is compressed into large jumps. Continuous DLMs instead model language generation in a continuous representation space. Diffusion-LM (Li et al., 2022) and DiffuSeq (Gong et al., 2023) operate on token embeddings, whereas SSD-LM (Han et al., 2023) and TESS (Mahabadi et al., 2024) use simplex-based representations. Beyond the state space, an important distinction is how discrete supervision constrains the trajectory. FLM (Lee et al., 2026) learns continuous denoising over one-hot encodings with token-level cross-entropy, and LangFlow (Chen et al., 2026) connects embedding-space diffusion to Flow Matching through Bregman divergence. ELF (Hu et al., 2026) instead reserves token-level supervision for the final decoding step, allowing the preceding dynamics to remain in continuous embedding space. These advances improve continuous generation, but an effective trajectory still requires tens or even hundreds of sampling steps at inference. Few-step methods address this problem by learning to traverse or replace an existing sampling trajectory. SDTT (Deschenaux and Gulcehre, 2025) compresses denoising through self-distillation, Di4C (Hayakawa et al., 2025) explicitly addresses the dimensional correlations lost in factorized reverse transitions, and FMLM (Lee et al., 2026) distills a continuous flow into a map between distant timepoints. Drifting (Deng et al., 2026) provides a different training formulation, evolving the generator’s outputs through attraction and repulsion while retaining a single forward pass at inference. Seq-Drifting extends this formulation to token sequences. Context-dependent, per-position supervision supplies plausible token targets for its current outputs. The support sets can draw on corpus retrieval or a frozen language model, while the objective directly trains the one-step map without reproducing a diffusion or flow trajectory. 3 Preliminaries We consider a one-step generator GθG_θ that maps a noise input ϵ∼pϵε p_ε to an output =Gθ(ϵ) x=G_θ(ε) with distribution qθq_θ, and training seeks qθ≈pdataq_θ≈ p_data. Since the parameters are updated step by step, the same noise ϵε produces a slightly different output after each step. The displacement Δi=Gθi+1(ϵ)−Gθi(ϵ) x_i\;=\;G_ _i+1(ε)-G_ _i(ε) (1) comes entirely from the parameter update, and is called the drift of x at iteration i. A drifting field Vp,qV_p,q (Deng et al., 2026) prescribes this drift sample by sample, with p fixed to the target pdatap_data and q to the current model qθiq_ _i, i+1=i+Vpdata,qθi(i). x_i+1= x_i+V_p_data,\,q_ _i( x_i). (2) The field vanishes when the two distributions agree, Vp,p=V_p,p= 0, so qθ=pdataq_θ=p_data is a fixed point of training. Regressing each output onto its drifted target with a stop gradient turns this into the loss, ℒ(θ)=ϵ‖Gθ(ϵ)−sg(Gθ(ϵ)+V(Gθ(ϵ)))‖2,L(θ)=E_ε \|\,G_θ(ε)-sg\! (G_θ(ε)+V(G_θ(ε)) ) \|^2, (3) whose value equals ϵ‖V(Gθ(ϵ))‖2E_ε\|V(G_θ(ε))\|^2, so minimizing it drives the drift toward zero. A concrete field comes from a kernel mean-shift. For a sample x, a positive set P drawn from pdatap_data attracts and a negative set N drawn from qθq_θ repels, V()=∑∈k~(,)(−)⏟attraction−∑∈k~(,)(−)⏟repulsion,V( x)= _ y k( x, y)\,( y- x)_attraction\;-\; _ y k( x, y)\,( y- x)_repulsion, (4) where k~ k is a similarity kernel normalized over each set. The attraction moves x toward the data it already resembles and the repulsion moves it off the other samples, and the two cancel once the generated and data distributions agree. Defining the drift target. The field in Eq. (4) specifies how a generated sample should move during training. Positive samples provide the destinations for attraction, while other generated samples provide the reference points for repulsion. The similarity kernel determines which samples contribute most strongly to each force. Combining the forces gives a displacement, which defines the stop-gradient target +V() x+V( x) in Eq. (3). The generator learns to approach this target through parameter updates. Adapting drifting to a new domain therefore requires suitable positive and negative samples, a geometry for comparing them, and a rule for converting the field into a training target. Difficulties in applying drifting to text. A text sequence is composed of discrete token choices, which creates three difficulties for drifting. First, valid outputs lie in a discrete and semantically non-smooth space. A continuous displacement can change the decoded token, but proximity between embeddings does not guarantee that the new token fits the surrounding sequence. Second, the tokens that form a coherent continuation depend on the preceding sequence. However, a training example supplies only the continuation observed after its own context, which may differ substantially from the generator’s current context. Using that observed continuation directly as a drift target can therefore yield a weak or misaligned update. Third, under cosine decoding, radial separation can increase Euclidean distances without changing token identity. Unconstrained repulsion can therefore spend its force on growing norms while producing no additional token-level diversity. Figure 2: Seq-Drifting. In one forward pass the generator GθG_θ maps a noise vector ϵ∼(0,IC)ε\! \!N(0,I_C) (optionally with a prefix p1,…,pMp_1,…,p_M) to token embeddings 1,…,T h_1,…, h_T, each projected onto the unit sphere as t=t/‖t‖∈SD−1 e_t= h_t/\| h_t\|∈ S^D-1. Supervision is applied per position. A positive set tS_t of plausible tokens attracts t e_t, while the negative sets NtoutN_t^out (other samples) and NtinN_t^in (other positions of the same sample) repel it, weighted by a similarity kernel φ and strengths λin/out _in/out. The two combine into the drift force t F_t; a single geodesic step along t F_t gives the stop-gradient target goaltgoal_t, onto which the generator is regressed, ℒ=‖t−sg(goalt)‖2L=E\| e_t-sg(goal_t)\|^2. 4 Seq-Drifting To address the difficulties above, Seq-Drifting represents a sequence with per-position token representations and provides each position with a context-dependent set of plausible tokens. These representations and targets are placed in a shared spherical space, where the resulting drift remains anchored to valid vocabulary while the noise-to-text map is learned directly. Figure 2 summarizes the framework. We first describe the one-step generator and its representation space, then introduce support-set construction, the attractive and repulsive forces, and the training objective. 4.1 Spherical one-step generator We generate a sequence of continuous representations, each one corresponding to a token. This allows the generator to produce all positions jointly while exposing each token choice to local drift supervision. Specifically, our generator GθG_θ maps a shared noise vector ϵ∈ℝCε ^C to T token representations tt=1T\ h_t\_t=1^T, with t∈ℝD h_t ^D, in one forward pass. Architectural details are given in Appendix A.1. Then, we recover tokens by cosine similarity to a frozen vocabulary embedding table. Since this decoding rule depends only on direction, we normalize both the output vectors t h_t and the vocabulary embeddings v w_v, i.e., t=t/‖t‖∈SD−1 e_t= h_t/\| h_t\|∈ S^D-1 and ‖v‖=1\| w_v\|=1. Attraction, repulsion, and token recovery then depend on the same directional geometry. The decoded token is tokt=argmaxvt⊤vtok_t= _v e_t w_v, with non-text tokens excluded. This operation produces the final text at inference, requiring no separate vocabulary softmax. 4.2 Context-dependent token support The main challenge in selecting positives is to provide useful guidance for the sequence the generator currently produces. A reference sentence may differ from that sequence at many positions, so its tokens need not be suitable targets for the corresponding positions. We instead construct positive targets at each position, asking which tokens are plausible given the generated context. For position t, we collect the positive targets into a support set tS_t (i.e., P in § 3), with each candidate represented by its unit embedding. The set supplies concrete token targets without prescribing a single continuation. It separates contextual plausibility from the choice made by an individual sample. The support set determines which tokens are eligible for attraction, while the local kernel introduced next determines how strongly each candidate attracts the current output. There are two natural ways to form tS_t. The off-policy route reads it from the corpus, matching each generation against real sequences and taking the tokens they place at position t, so the candidates come from observed continuations. The on-policy route queries a frozen language model (hereafter, the support model) on the generator’s current prefixes to propose contextually plausible next-token candidates, where one forward pass over the generated sequence returns a next-token distribution ptp_t at every position at once. tS_t collects its most probable tokens (the top k), so the target adapts to wherever the current generator sits. The same construction also applies to a closed-source support model, which can be queried multiple times to sample continuations from the generated context and form tS_t. We empirically find the on-policy support the stronger of the two (§5.4) and adopt it by default. The support model is queried only during training to construct the support set tS_t, and is not required at inference. The resulting supervision is set-valued and context-dependent rather than a fixed per-position target. 4.3 Angular mode-seeking attraction The attraction pulls each position toward the supported token nearest its own embedding. We measure closeness by angular distance d(t,)=1−t⊤d( e_t, w)=1- e_t w and form a low-temperature similarity kernel φ , normalized over the support set, φ(,t)∝exp(−d(t,)stτ),t=∑∈tφ(,t)(−t),st=std∈td(t,), ( w, e_t)\; \; \! (\!- d( e_t, w)s_t\,τ ), u_t=\!\! _ w _t\! ( w, e_t)\,( w- e_t), s_t=std_ w _td( e_t, w), (5) where sts_t is the standard deviation of d(t,)d( e_t, w) for candidates at position t and τ is a temperature. Normalizing by sts_t accounts for each position’s candidate spread, while τ<1τ<1 sharpens the kernel around nearby candidates. It is therefore mode-seeking and concentrates the weight on the near supported token, so different noise inputs commit to different tokens rather than to the blurred centroid of tS_t. The mean-shift then gives the attractive force t=∑∈tφ(,t)(−t) u_t= _ w _t ( w, e_t)( w- e_t). 4.4 Per-position angular repulsion The repulsion stops distinct noise inputs from collapsing to the same output. It acts per position against two negative sets, i.e., ≜t=Ntout,NtinN _t=\N_t^out,N_t^in\. The cross-sample set NtoutN_t^out contains the position-t representations from the other samples in the batch. The intra-sequence set NtinN_t^in contains the representations at the other positions of the same sequence, and pushing away from it suppresses within-sentence repetition. Each uses the same angular kernel φ as the attraction, tout=∑∈Ntoutφ(,t)(t−),tin=∑∈Ntinφ(,t)(t−). r^out_t=\!\! _ n∈ N_t^out\!\! ( n, e_t)\,( e_t- n), r^in_t=\!\! _ n∈ N_t^in\!\! ( n, e_t)\,( e_t- n). (6) Both are then rescaled to the magnitude of the attraction before the update weights them by λout _out and λin _in. Restricting the cross-sample set NtoutN_t^out to the samples that share a context is studied in §5.4. On the sphere, repulsion separates outputs by changing their directions rather than increasing their norms. This prevents radial growth and aligns repulsion with cosine decoding. Together with the bounded geodesic update in §4.5, it defines the displacement of each training target. 4.5 Geodesic update and training The update sums the attraction and the two repulsions into one drift force, t=t+λouttout+λintin F_t= u_t+ _out\, r^out_t+ _in\, r^in_t, projects it onto the tangent plane at t e_t to drop the radial part, t′=t−(t⊤t)t F _t= F_t-( F_t e_t)\, e_t, and applies it as one step along the great circle by the exponential map, goalt=cos(α)t+sin(α)t′‖t′‖,α=min(‖t′‖,αmax).goal_t= (α)\, e_t+ (α)\, F _t\| F _t\|, α= \! (\| F _t\|,\ _ ). (7) Here we use the norm of t′ F _t as the arc length of the geodesic step, capped by αmax _ to prevent overshooting. Treating goaltgoal_t as a fixed target during differentiation, the loss of Eq. (3) becomes the squared distance ‖t−sg(goalt)‖2E\,\| e_t-sg(goal_t)\|^2, which is bounded because both points lie on the sphere. Appendix A.2 analyzes the conditions under which the field vanishes as the temperature approaches zero. Appendix A.3 provides the training pseudocode for the on-policy support-set implementation. Conditional generation. For conditional generation, we additionally provide the generator with the conditioning token embeddings, which the output positions access through cross-attention. The support model also receives the conditioning sequence followed by the generated continuation to provide token support sets. To stabilize learning this conditional map from scratch, we first train with an unconditional warm-up. During this phase, the context is filled with a constant padding token (e.g., <|endoftext|> for GPT-2) under normal attention, which keeps the conditioning path active while the generator learns fluent text. We then replace the constant context with the real prefix, allowing the generator to learn context dependence from this fluent initialization. 5 Experiments 5.1 Setup Baselines. Our main comparison is against the few-step distilled models that compress a multi-step sampler into a handful of steps, including FMLM (Lee et al., 2026), Duo+DCD (Sahoo et al., 2025), MDLM+SDTT (Deschenaux and Gulcehre, 2025), and Duo/MDLM+Di4C (Hayakawa et al., 2025). We report them at 11, 22, and 44 steps. We also compare against multi-step models that generate through multiple (e.g., 1024) denoising steps, namely FLM (Lee et al., 2026), Duo (Sahoo et al., 2025), MDLM (Sahoo et al., 2024), CANDI (Pynadath et al., 2026), RDLM (Jo and Hwang, 2026), LangFlow (Chen et al., 2026), and ELF (Hu et al., 2026). Data and protocol. We follow the evaluation protocol of Lee et al. (2026) and report at two output lengths. In the length-128128 setup the models are trained on One Billion Word (LM1B) (Chelba et al., 2013), and in the length-10241024 setup on OpenWebText (Gokaslan et al., 2019) using the corresponding benchmark preprocessing. We draw 10241024 samples per configuration. For each method we report the number of forward passes per sample (Step), the generative perplexity of the samples under GPT-2 Large (Gen-PPL↓ ), and their unigram entropy (Entropy). Implementation details. We consider two implementations of on-policy support-set construction. For Seq-DriftingS, we first train a GPT-2 from scratch on the same corpus as the baselines for 10510^5 steps, then freeze it and use it as the support model. This setting uses no additional text beyond the baselines. For Seq-DriftingP, we directly use a public pretrained GPT-2 checkpoint to construct the support sets in the same way. The support set uses k=64k=64, the attraction temperature is τ=0.3τ=0.3, and the cross-sample and intra-sequence repulsions use λout=5 _out=5 and λin=100 _in=100. The geodesic step is capped at αmax=1.5708 _ =1.5708 radians. We train with AdamW (learning rate 2×10−42× 10^-4, weight decay 00, cosine schedule) on 44 H200 GPUs. Unconditional models are trained for 2×1052× 10^5 steps. Conditional models use a 2×1052× 10^5-step unconditional warm-up, followed by 1×1051× 10^5 conditional steps, keeping an exponential moving average of the weights with decay 0.9990.999. Therefore, the total training step of Seq-Drifting is shorter than the 1.0–1.5M-step recipes usually used by the baselines. The effective batch is 20482048 at length 128128 and 128128 at length 10241024, the longer sequence being reduced to fit memory. Method Step LM1B OWT Gen-PPL↓ Ent. Gen-PPL↓ Ent. FMLM 1 119.34 4.16 169.32 5.17 2 110.19 4.21 135.68 5.30 4 98.76 4.21 111.49 5.36 Duo+DCD 1 1006.75 4.33 6330.89 5.98 2 466.46 4.24 983.80 5.52 4 195.52 4.23 252.04 5.49 Duo+Di4C 1 292.94 3.79 370.51 3.92 2 247.69 3.87 210.22 4.63 4 150.67 4.00 154.67 4.85 MDLM+SDTT 1 1099.64 4.23 1264.02 5.25 2 448.78 4.22 860.40 5.34 4 181.08 4.23 319.73 5.36 MDLM+Di4C 1 1217.10 4.38 1272.27 5.28 2 621.59 4.37 742.99 5.34 4 247.32 4.00 226.95 5.37 Seq-DriftingS 1 93.81 4.21 183.98 4.95 Seq-DriftingP 1 36.01 4.42 128.33 5.08 Figure 3: Unconditional generation. Left: Gen-PPL (under GPT-2 Large) and unigram entropy for the few-step distilled models at 11/22/44 steps and Seq-Drifting at a single pass, on LM1B (T=128T=128) and OpenWebText (T=1024T=1024); multi-step baselines are in Table 6. Right: Gen-PPL vs. sampling steps and wall-clock running time per sample, showing Seq-Drifting achieves lower Gen-PPL in a single forward pass at a fraction of the cost. 5.2 Unconditional generation Figure 3 compares the few-step distilled models with Seq-Drifting at a single pass. The full multi-step results are reported in Table 6 in Appendix B.3. Seq-DriftingS produces competitive unconditional samples. On LM1B, it outperforms the strongest four-step baseline and even beat several 10241024-step baselines, which directly reflect the effectiveness of our drifting framework. Seq-DriftingP further reduces Gen-PPL relative to FMLM by 69.8%69.8\% on LM1B and 24.2%24.2\% on OpenWebText. The entropy and Self-BLEU results in Appendix B.1 show that they preserve generation diversity. To verify that the comparison is not tied to the GPT-2 Large evaluator, Appendix B.4 recomputes Gen-PPL with Pythia-1B (Biderman et al., 2023) and OPT-1.3B (Zhang et al., 2022). The resulting comparisons remain consistent with the main table, showing that the observed gains are not specific to the evaluator. Table 1: Conditional generation. A 6464-token prefix followed by a continuation. Method Step 64→6464→64 64→51264→512 Gen-PPL↓ Ent. Gen-PPL↓ Ent. FMLM 1 162.20 3.78 418.37 3.32 2 143.62 3.82 441.09 3.55 4 124.74 3.82 320.22 3.52 Duo+DCD 1 972.09 3.77 7639.94 3.85 2 384.35 3.72 846.20 3.72 4 209.32 3.74 205.79 3.75 Duo+Di4C 1 409.44 3.02 2784.97 3.74 2 507.34 3.30 1007.87 3.75 4 476.22 3.48 254.55 3.75 MDLM+SDTT 1 912.05 3.75 2770.06 3.74 2 333.40 3.74 1077.27 3.75 4 139.73 3.74 298.50 3.75 MDLM+Di4C 1 1018.23 3.83 2791.44 3.74 2 509.84 3.83 944.53 3.75 4 231.36 3.82 226.34 3.75 Seq-DriftingS 1 79.13 3.74 119.13 4.10 Seq-DriftingP 1 76.16 4.40 168.34 4.78 The running-time in Figure 3(right) shows that Seq-Drifting has the lowest single-sample latency compared with the baselines. Relative to four-step FMLM, which reaches a comparable quality level, Seq-Drifting is 11.2×11.2× faster on LM1B and 10.1×10.1× faster on OpenWebText. A broader comparison of sampling time across multi-step baselines is provided in Appendix B.2. Qualitative samples are shown in Appendix B.9. 5.3 Conditional generation Table 1 summarizes two conditional-generation settings, each with a 64-token context. On LM1B the model generates 64 output tokens. On OpenWebText it generates 512 output tokens so that the context and output remain within GPT-2’s 1024-token context window. Both Seq-DriftingS and Seq-DriftingP outperform all few-step distilled baselines in both settings. Compared with unconditional generation, the gap between Seq-DriftingS and Seq-DriftingP narrows, and Seq-DriftingS is even stronger on the 512512-token output setting. One possible explanation is that the observed prefix narrows the plausible continuations, reducing ambiguity in the support sets and making the conditional map easier to learn. Overall, the results suggest that Seq-Drifting can preserve both fluency and conditional relevance in conditional generation. 5.4 Ablations We report ablations in both the unconditional (128128) and conditional (64→6464→64) settings in Table 2. The geometry ablation shows that the spherical update is essential. Replacing it with a Euclidean update allows repulsion to increase the embedding norm and causes training to diverge. The positive-selection ablation highlights that support construction should match the task. For open-ended generation, fixed corpus-based targets do not follow the generator’s evolving contexts. When the attraction becomes sharp, this mismatch can produce oscillatory training and rising generative perplexity and entropy (Appendix B.8). For tasks with explicit target answers, dataset-derived supervision can remain appropriate, as in our reasoning experiments in §5.6. For repulsion, the cross-sample term is not enough on its own. An intra-sequence term that pushes each position off the other positions of the same generation is also necessary, and dropping it (w/o intra) collapses the output into within-sentence token repetition. In the conditional setting, whether the cross-sample repulsion is restricted to samples that share a prefix makes little difference. Finally, unconditional warm-up stabilizes from-scratch conditional training by allowing the generator to learn fluency before it must also model prefix dependence. Table 2: Ablation study on unconditional (128128) and conditional (64→6464→64) generation. Unconditional (128128) Conditional (64→6464→64) Decision Variant Gen-PPL↓ Entropy Gen-PPL↓ Entropy Seq-DriftingP 36.01 4.42 76.16 4.40 Geometry Euclidean diverges (loss blow-up) Positives off-policy >>1000 >>6 >>10000 >>4 Anti-repetition w/o intra 10.01 1.82 12.18 1.31 Repulsion per-context (block) – – 81.27 4.27 Warm-up w/o warm-up – – 53.57 3.87 Figure 4: Hyperparameter sweeps of the repulsion strength λout _out, the anti-repetition weight λin _in, the attraction temperature τ, and the support size k. Setting: unconditional 128128-token generation (solid), conditional 64→6464→64 generation (dashed). 5.5 Hyperparameters Figure 4 further sweeps parameters λout,λin,τ _out, _in,τ, and k. We observe that the first three parameters show the same broad pattern. Gen-PPL first improves and then worsens as the parameter moves away from a favorable middle range, revealing a balance between a target that is too weak and one that is too sharp or too disruptive. λout _out has a larger effect than λin _in. When repulsion between different noise samples is too weak, the generator is encouraged to learn an average token direction rather than preserve distinct modes. λin _in mainly controls repetition within a sequence and therefore provides a narrower correction. For τ, making the temperature too small makes the target at each position nearly unique, discarding the useful ambiguity in the support tokens. For the support size, k=4k=4 collapses the output (entropy below 3.53.5). Increasing the support size generally restores diversity and improves Gen-PPL. 5.6 Downstream task evaluation Translation and summarization. Following Hu et al. (2026), we evaluate Seq-Drifting on WMT14 (German–English translation) (Bojar et al., 2014) and XSum (abstractive summarization) (Narayan et al., 2018). Experimental details are provided in Appendix B.5. Table 3 shows that Seq-Drifting outperforms the few-step baselines on translation and on most summarization metrics, although a gap remains relative to the multi-step results. We hypothesize that this gap partly reflects a mismatch between the support-set candidates used during training and the single reference used for evaluation. When evaluated against results sampled from the support model, Seq-Drifting obtains substantially higher scores. Scaling the support model further improves one-step generation quality (Figure 5). This observation motivates future work on using increasingly capable language models to construct support sets, with the broader goal of turning existing LLMs into efficient one-step generators. Model Step WMT14 XSum BLEU↑ ROUGE-1↑ ROUGE-2↑ ROUGE-L↑ AR – 25.2† 30.5±0.13± 0.13† 10.2±0.11± 0.11† 24.4±0.12± 0.12† MDLM 4 << 10 19.0±0.06± 0.06 2.4±0.01± 0.01 13.2±0.04± 0.04 64 18.4† 33.4±0.11± 0.11† 11.6±0.10± 0.10† 25.8±0.10± 0.10† Duo 4 << 10 24.4±0.06± 0.06 5.6±0.01± 0.01 18.1±0.04± 0.04 1000 21.3† 31.4±0.12± 0.12† 10.1±0.10± 0.10† 25.0±0.12± 0.12† ELF 1 11.5 23.7±0.11± 0.11 5.7±0.08± 0.08 19.4±0.10± 0.10 64 26.4† 36.0±0.13± 0.13† 12.2±0.11± 0.11† 27.8±0.12± 0.12† Seq-Drifting (vs. dataset) 1 18.6 24.7±0.36± 0.36 5.3±0.24± 0.24 20.1±0.32± 0.32 Seq-Drifting (vs. AR) 1 30.8 33.2±0.44± 0.44 11.6±0.41± 0.41 28.7±0.44± 0.44 Table 3: Performance on WMT14 De–En and XSum. The two Seq-Drifting rows evaluate results on the dataset ground truth and samples from our support model, respectively. † denotes results taken from Hu et al. (2026). Figure 5: Support model scaling. S, M, and L denote small, medium, and large GPT-2. Reasoning. We further evaluate Seq-Drifting on math reasoning (SVAMP (Patel et al., 2021) and ASDiv (Miao et al., 2020)) and logical reasoning (ProofWriter (Tafjord et al., 2021)). We scale Seq-Drifting and all the baselines to 0.50.5B. The implementation and training details are provided in Appendix B.6. Despite using a single forward pass, Seq-Drifting approaches the accuracy of existing latent-reasoning method Coconut (Hao et al., 2025) on the mathematical benchmarks and remains competitive with AR on ProofWriter, while achieving substantially lower per-sample latency. These results show that a drifting field can also be learned from supervision with explicit target answers, extending the one-step framework beyond open-ended generation. Table 4: Performance on SVAMP, ASDiv, and ProofWriter. Model SVAMP ASDiv ProofWriter ACC↑ Time/sample (ms)↓ ACC↑ Time/sample (ms)↓ ACC↑ Time/sample (ms)↓ AR 27.92±0.36± 0.36 49.42±4.36± 4.36 44.97±0.50± 0.50 31.07±1.18± 1.18 85.90±0.38± 0.38 41.90±0.10± 0.10 NAR 4.72±0.76± 0.76 2.70±0.04± 0.04 1.73±0.57± 0.57 1.30±0.07± 0.07 0.00±0.00± 0.00 1.42±0.06± 0.06 Coconut 17.76±0.01± 0.01 25.80±0.52± 0.52 22.59±0.01± 0.01 26.63±0.13± 0.13 – – Seq-Drifting 13.60±0.50± 0.50 5.88±0.17± 0.17 19.11±0.79± 0.79 1.05±0.03± 0.03 79.31±0.47± 0.47 3.68±0.02± 0.02 6 Conclusion In this paper, we presented Seq-Drifting, a framework for directly learning one-step text generation by drifting sequences of continuous token representations. Context-dependent support sets provide multiple plausible targets at each position, while mode-seeking attraction, repulsion, and spherical geometry help preserve coherent and diverse outputs. Across unconditional generation, conditional continuation, translation, summarization, and reasoning, Seq-Drifting provides a competitive one-step operating point and outperforms the evaluated few-step baselines at a fraction of their inference cost. The remaining gap to the strongest multi-step systems motivates future work on stronger support models, richer semantic supervision, and improved token recovery. AI Use Statement In this work, we used generative AI tools to edit the manuscript for readability and clarity. All AI-assisted text was reviewed and revised by the authors. The authors take full responsibility for the final content of this work, including all claims, analyses, and conclusions. References Arriola et al. (2025) M. Arriola, A. Gokaslan, J. Chiu, Z. Yang, Z. Qi, J. Han, S. Sahoo, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, Vol. 2025, p. 50726–50753. Cited by: §2. Arriola et al. (2026) M. Arriola, Y. Schiff, H. Phung, A. Gokaslan, and V. Kuleshov Encoder-decoder diffusion language models for efficient training and inference. Advances in Neural Information Processing Systems 38, p. 15612–15644. Cited by: §2. Austin et al. (2021) J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems 34, p. 17981–17993. Cited by: §2. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: a suite for analyzing large language models across training and scaling. In International conference on machine learning, p. 2397–2430. Cited by: §5.2. Bojar et al. (2014) O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, p. 12–58. Cited by: §5.6. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1, §2. Chelba et al. (2013) C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. Cited by: §5.1. Chen et al. (2026) Y. Chen, C. Liang, H. Sui, R. Guo, C. Cheng, J. You, and G. Liu Langflow: continuous diffusion rivals discrete in language modeling. arXiv preprint arXiv:2604.11748. Cited by: §B.3, §2, §5.1. Deng et al. (2026) M. Deng, H. Li, T. Li, Y. Du, and K. He Generative modeling via drifting. External Links: 2602.04770, Link Cited by: §A.2, §1, §2, §3. Deng et al. (2023) Y. Deng, K. Prasad, R. Fernandez, P. Smolensky, V. Chaudhary, and S. Shieber Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460. Cited by: §B.6. Deschenaux and Gulcehre (2025) J. Deschenaux and C. Gulcehre Beyond autoregression: fast llms via self-distillation through time. In International Conference on Learning Representations, Vol. 2025, p. 5007–5045. Cited by: §1, §2, §5.1. Gokaslan et al. (2019) A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex OpenWebText corpus. Note: http://Skylion007.github.io/OpenWebTextCorpus Cited by: §5.1. Gong et al. (2023) S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong Diffuseq: sequence to sequence text generation with diffusion models. In International Conference on Learning Representations (ICLR), Cited by: §2. Han et al. (2023) X. Han, S. Kumar, and Y. Tsvetkov Ssd-lm: semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11575–11596. Cited by: §2. Hao et al. (2025) S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. E. Weston, and Y. Tian Training large language models to reason in a continuous latent space. In Second Conference on Language Modeling, Cited by: §B.6, §5.6. Hayakawa et al. (2025) S. Hayakawa, Y. Takida, M. Imaizumi, H. Wakaki, and Y. Mitsufuji Distillation of discrete diffusion through dimensional correlations. In Forty-second International Conference on Machine Learning, Cited by: §1, §2, §5.1. Hu et al. (2026) K. Hu, L. Qiu, Y. Lu, H. Zhao, T. Li, Y. Kim, J. Andreas, and K. He Elf: embedded language flows. arXiv preprint arXiv:2605.10938. Cited by: §B.3, §1, §2, §5.1, §5.6, Table 3. Jo and Hwang (2026) J. Jo and S. J. Hwang Continuous diffusion model for language modeling. Advances in Neural Information Processing Systems 38, p. 97394–97430. Cited by: §B.3, §5.1. Lee et al. (2026) C. Lee, J. Yoo, M. Agarwal, S. Shah, J. Huang, A. Raghunathan, S. Hong, N. M. Boffi, and J. Kim One-step language modeling via continuous denoising. arXiv e-prints, p. arXiv–2602. Cited by: §B.3, §1, §2, §2, §5.1, §5.1. Li et al. (2025) T. Li, M. Chen, B. Guo, and Z. Shen A survey on diffusion language models. arXiv preprint arXiv:2508.10875. Cited by: §2. Li et al. (2022) X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto Diffusion-lm improves controllable text generation. Advances in neural information processing systems 35, p. 4328–4343. Cited by: §1, §2. Lin (2004) C. Lin Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, p. 74–81. Cited by: §B.5. Mahabadi et al. (2024) R. K. Mahabadi, H. Ivison, J. Tae, J. Henderson, I. Beltagy, M. E. Peters, and A. Cohan Tess: text-to-text self-conditioned simplex diffusion. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2347–2361. Cited by: §2. Miao et al. (2020) S. Miao, C. Liang, and K. Su A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, p. 975–984. Cited by: §5.6. Narayan et al. (2018) S. Narayan, S. B. Cohen, and M. Lapata Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 1797–1807. Cited by: §5.6. Nie et al. (2025) S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li Large language diffusion models. arXiv preprint arXiv:2502.09992. Cited by: §2. Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, p. 311–318. Cited by: §B.5. Patel et al. (2021) A. Patel, S. Bhattamishra, and N. Goyal Are nlp models really able to solve simple math word problems?. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, p. 2080–2094. Cited by: §5.6. Pynadath et al. (2026) P. Pynadath, J. Shi, and R. Zhang CANDI: hybrid discrete-continuous diffusion models. In Forty-third International Conference on Machine Learning, Cited by: §5.1. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: §1, §2. Sahoo et al. (2024) S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems 37, p. 130136–130184. Cited by: §B.3, §1, §2, §5.1. Sahoo et al. (2025) S. S. Sahoo, J. Deschenaux, A. Gokaslan, G. Wang, J. Chiu, and V. Kuleshov The diffusion duality. Proceedings of machine learning research 267, p. 52584. Cited by: §B.3, §2, §5.1. Tafjord et al. (2021) O. Tafjord, B. Dalvi, and P. Clark ProofWriter: generating implications, proofs, and abductive statements over natural language. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, p. 3621–3634. Cited by: §5.6. Zhang et al. (2022) S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. Cited by: §5.2. Zhu et al. (2018) Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu Texygen: a benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, p. 1097–1100. Cited by: §B.1. Appendix A Method details and analysis A.1 Generator architecture A shared noise vector ϵ∈ℝCε ^C is projected to T⋅DT\!·\!D dimensions, reshaped to (T,D)(T,D), and combined with learned positional embeddings. Our generator GθG_θ is a 1212-layer pre-LN Transformer with width 768768, 1212 attention heads, and an FFN width of 30723072. The noise dimension is C=128C=128. We use QK-norm on the per-head queries and keys and a LayerScale gate on each residual branch as stabilization mechanisms. For conditional generation, conditioning token embeddings are supplied as a prefix and the output positions access them through ungated cross-attention. A.2 A conditional fixed-point analysis We analyze conditions under which matching the generated and data distributions makes the idealized drift field zero. For qθ=pdataq_θ=p_data to be a fixed point of training, the drift V, and hence the loss of Eq. (3), must be zero there. Anti-symmetry Vp,q=−Vq,pV_p,q=-V_q,p is one way to guarantee this and is how the plain mean-shift field of Deng et al. (2026) does it, but it is a sufficient condition, not a necessary one. Seq-Drifting’s per-position field is not anti-symmetric, so instead of appealing to that argument we check V=V= 0 at qθ=pdataq_θ=p_data directly. The field. At position t, with unit embedding t e_t, the drift of §4 is t=∑∈tφ(,t)(−t)⏟attraction t−λout∑∈Ntoutφ(,t)(−t)−λin∑∈Ntinφ(,t)(−t), F_t= _ w _t ( w, e_t)( w- e_t)_attraction u_t- _out\!\! _ n∈ N_t^out\!\! ( n, e_t)( n- e_t)- _in\!\! _ n∈ N_t^in\!\! ( n, e_t)( n- e_t), (8) with φ(,)∝exp(−d(,)/(sτ)) ( a, b) \! (\!-d( a, b)/(s\,τ) ) and d(,)=1−⊤d( a, b)=1- a b the angular kernel, normalized over each set. The attraction pulls t e_t toward the support set tS_t of plausible tokens, and the two repulsions push it off the other generations NtoutN_t^out and the other positions of its own sequence NtinN_t^in. Attraction and rescaled repulsion at matching distributions. At qθ=pdataq_θ=p_data the generator reproduces the data. Suppose each output embedding coincides with a real token embedding, t=⋆ e_t= w with ⋆∈t w _t. Importantly, this does not depend on any particular support model. In the mode-seeking limit τ→0τ\!→\!0, a generation already sitting on a supported token has nothing to be pulled toward, so the attraction is zero, t=∑∈tφ(,t)(−t)⟶φ(⋆,t)(⋆−t)=, u_t= _ w _t ( w, e_t)( w- e_t)\; \; ( w , e_t)( w - e_t)= 0, (9) The repulsions then vanish with it, because each is rescaled to the attraction magnitude before entering t F_t, with coefficient arms/rrmsa_rms/r_rms and arms=RMS(t)a_rms=RMS( u_t), so the repulsion magnitude is ∝arms→0 a_rms→ 0. Under these conditions, t F_t is zero at every position and qθ=pdataq_θ=p_data is a fixed point of the idealized field. The attraction-relative rescaling is what lets a strong anti-collapse repulsion coexist with this fixed point, because the repulsion is rescaled by the attraction magnitude and therefore switches off with it. In practice, the temperature is positive and the decoded token distribution can match the data distribution even when generated representations slightly differ from their corresponding vocabulary embeddings. We now quantify how such discrepancies affect the drift. Let t⋆=tokt w_t = w_tok_t and consider generated sequences for which t⋆∈t w_t _t is the unique nearest candidate. Since the kernel weights sum to one, the attraction decomposes exactly as t=t⋆−t+∑∈t∖t⋆φ(,t)(−t⋆). u_t= w_t - e_t+ _ w _t \ w_t \ ( w, e_t)( w- w_t ). (10) The first term is the representation discrepancy, and the second is the finite-temperature correction. For a support set with at least two candidates, define Δt _t =min∈t∖t⋆[d(t,)−d(t,t⋆)]>0, = _ w _t \ w_t \ [d( e_t, w)-d( e_t, w_t ) ]>0, (11) bt b_t =(|t|−1)exp(−Δtstτ). =(|S_t|-1) \! (- _ts_tτ ). Each other candidate has at most exp(−Δt/(stτ)) (- _t/(s_tτ)) times the nearest candidate’s kernel weight, giving 1−φ(t⋆,t)≤bt/(1+bt)1- ( w_t , e_t)≤ b_t/(1+b_t). All vocabulary embeddings have unit norm, so their pairwise distances are at most two. It follows that ‖t−(t⋆−t)‖2≤2[1−φ(t⋆,t)]≤2bt1+bt. \| u_t-( w_t - e_t) \|_2≤ 2 [1- ( w_t , e_t) ]≤ 2b_t1+b_t. (12) For singleton supports, the sole candidate has weight one and we set bt=0b_t=0. Define ‖RMS=(‖t‖22)1/2\| a\|_RMS=(E\| a_t\|_2^2)^1/2, averaging over generated samples and positions, and write δ=(‖t−t⋆‖22)1/2,ρτ=[(2bt1+bt)2]1/2.δ= (E\| e_t- w_t \|_2^2 )^1/2, _τ= [E ( 2b_t1+b_t )^2 ]^1/2. (13) The L2L^2 triangle inequality gives ‖RMS≤δ+ρτ\| u\|_RMS≤δ+ _τ. Under the attraction-relative rescaling above, each rescaled repulsion has RMS no greater than that of the attraction, using the same normalization groups and leaving zero repulsion at zero. Tangent projection cannot increase the force norm, and the geodesic displacement Vt=goalt−tV_t=goal_t- e_t satisfies ‖Vt‖2=2|sin(αt/2)|≤αt≤‖t′‖2\|V_t\|_2=2| ( _t/2)|≤ _t≤\| F _t\|_2, where αt _t is the step angle at position t and zero tangent force gives zero displacement. Applying the triangle inequality to the combined force therefore yields ‖V‖RMS≤‖RMS≤(1+λout+λin)(δ+ρτ).\|V\|_RMS≤\| F\|_RMS≤(1+ _out+ _in)(δ+ _τ). (14) The loss averaged over samples and positions is bounded by the square of this expression. For a fixed distribution of representations satisfying the support and uniqueness conditions above, ρτ→0 _τ→ 0 as τ→0τ→ 0 by dominated convergence, since bt→0b_t→ 0 and 2bt/(1+bt)≤22b_t/(1+b_t)≤ 2. At positive temperature, the bound remains small when the representation discrepancy and the temperature correction are small enough for the chosen repulsion weights. Thus, when continuous representations do not coincide exactly, matching the decoded distribution can ensure that the drift is bounded. A.3 Seq-Drifting algorithm The pseudocode in Figure 6 summarizes Seq-Drifting training with on-policy support sets constructed by a frozen support model. Each iteration generates token representations, builds context-dependent support sets, and combines attraction and repulsion to form supervision targets on the sphere. The generator is then updated by regressing toward these fixed targets, with optional prefixes enabling conditional generation. Algorithm 1: Seq-Drifting training. Input: generator GθG_θ, frozen support model pψp_ψ, normalized token table vv∈\ w_v\_v , batch size B, support size k, τ,λout,λin,αmaxτ, _out, _in, _ . repeat Sample noise ϵ1,…,ϵB _1,…, _B and optional prefixes 1,…,B p_1,…, p_B. Compute one-step outputs b,1:T=Gθ(ϵb,b) h_b,1:T=G_θ( _b, p_b) and normalize b,t=b,t/‖b,t‖ e_b,t= h_b,t/\| h_b,t\|. Decode provisional tokens x^b,t=argmaxvb,t⊤v x_b,t= _v e_b,t w_v. Run one causal forward pass of pψp_ψ on each generated context and form b,tS_b,t from the top-k plausible next tokens. for each sample b and position t do Compute angular attraction b,t u_b,t from b,tS_b,t. Compute cross-sample and intra-sequence repulsions b,tout r^out_b,t and b,tin r^in_b,t. Rescale each repulsion to the attraction magnitude: ~b,tj=(arms/rrmsj)b,tj r^j_b,t=(a_rms/r_rms^j) r^j_b,t for j∈out,inj∈\out,in\, where the RMS values are computed over the batch and positions; set ~j= r^j= 0 when rrmsj=0r_rms^j=0. Set b,t=b,t+λout~b,tout+λin~b,tin F_b,t= u_b,t+ _out r^out_b,t+ _in r^in_b,t. Project b,t F_b,t to the tangent plane and map it to goalb,tgoal_b,t by Eq. (7). end for Update θ with ∇θ1BT∑b,t‖b,t−sg(goalb,t)‖22 _θ 1BT _b,t\| e_b,t-sg(goal_b,t)\|_2^2. until convergence Figure 6: Seq-Drifting training pseudocode. Appendix B Additional results B.1 Self-BLEU Table 5 reports Self-BLEU (Zhu et al., 2018) for the few-step distilled baselines, at each of their step counts (11, 22, 44), and for Seq-Drifting at a single step. Self-BLEU measures the n-gram overlap among a model’s own generations. A fully mode-collapsed generator that emits the same sample every time attains the maximum of 1.01.0. Table 5: Self-BLEU across the step budget. Per-dataset Self-BLEU for the few-step distilled models and Seq-Drifting; a fully mode-collapsed generator scores 1.01.0 (bottom row). Method Step Self-BLEU LM1B OpenWebText FMLM 1 0.201 0.249 2 0.198 0.251 4 0.204 0.260 Duo+DCD 1 0.042 0.016 2 0.065 0.064 4 0.093 0.140 Duo+Di4C 1 0.054 0.082 2 – 0.105 4 – 0.190 MDLM+SDTT 1 0.049 0.082 2 0.061 0.098 4 0.104 0.177 MDLM+Di4C 1 0.023 0.080 2 0.039 0.111 4 0.061 0.204 Seq-DriftingS 1 0.066 0.146 Seq-DriftingP 1 0.042 0.206 Mode collapse – 1.00 1.00 B.2 Sampling time across step budgets Figure 7 reports seconds per sample as a function of the sampling budget for the multi-step baselines. The horizontal dashed line marks the one-step latency of Seq-Drifting. Sampling time grows with the number of denoising steps, while Seq-Drifting remains constant at one step. Figure 7: Sampling time across step budgets. Seconds per sample on LM1B (T=128T=128) and OpenWebText (T=1024T=1024). The x-axis is logarithmic in the number of sampling steps and the y-axis is logarithmic in seconds per sample. The dashed line marks Seq-Drifting. B.3 Multi-step baselines Table 6 gives the full multi-step baselines (FLM, Duo, MDLM, CANDI, RDLM, LangFlow, ELF) at their 11/88/10241024 step budgets on both datasets, with Seq-Drifting included for reference. Table 7 gives the multi-step baselines on the conditional task at the same step budgets. Multi-step baseline implementations. For FLM, we use the authors’ released checkpoints for LM1B and OpenWebText (Lee et al., 2026). For Duo, MDLM, and CANDI on LM1B, we use the reproduction checkpoints distributed by Lee et al. (2026). On OpenWebText, we use the released Duo and MDLM weights (Sahoo et al., 2024; Sahoo et al., 2025). For CANDI on OpenWebText, we report the published result from Lee et al. (2026), while no public conditional OpenWebText checkpoint is available. For RDLM, we use the released LM1B checkpoint and its accompanying SDE configuration (Jo and Hwang, 2026). We do not report OpenWebText results because no corresponding public checkpoint is available. For LangFlow, we use the authors’ released checkpoints (Chen et al., 2026). For ELF, we use the released OpenWebText results and checkpoints (Hu et al., 2026); no corresponding LM1B checkpoint is available. Table 6: Unconditional generation: multi-step models across the step budget, with Seq-Drifting at a single pass. LM1B (T=128T=128) and OpenWebText (T=1024T=1024). A dash indicates that no corresponding released checkpoint or published reference result was available for that dataset–model setting. Method Step LM1B (T=128T=128) OpenWebText (T=1024T=1024) Gen-PPL↓ Entropy Gen-PPL↓ Entropy FLM 1 6.27 1.15 27.54 1.53 8 215.84 4.24 453.49 5.20 1024 85.24 4.29 63.97 5.35 Duo 1 1027.77 4.34 4007.98 5.80 8 121.37 4.27 196.45 5.57 1024 85.97 4.30 78.11 5.54 MDLM 1 1348.21 4.37 4180.42 5.82 8 177.21 4.33 785.33 5.87 1024 69.39 4.30 42.36 5.30 CANDI 1024 108.07 4.35 143.13 5.71 RDLM 1 22.70 3.20 – – 8 229.32 4.01 – – 1024 274.94 4.35 – – LangFlow 1 1.69 0.30 14.53 1.00 8 536.60 4.38 514.24 5.80 1024 121.73 4.31 36.53 5.25 ELF 1 – – 2.97 1.99 8 – – 70.95 5.25 1024 – – 7.73 4.74 Seq-DriftingS 1 93.81 4.21 183.98 4.95 Seq-DriftingP 1 36.01 4.42 128.33 5.08 Table 7: Conditional generation: multi-step models across the step budget, with Seq-Drifting at a single pass. A 6464-token prefix, then a continuation (64→6464→64 on LM1B, 64→51264→512 on OWT). A dash indicates that no corresponding released checkpoint or published reference result was available for that dataset–model setting. Method Step 64→6464→64 64→51264→512 Gen-PPL↓ Entropy Gen-PPL↓ Entropy FLM 1 1.95 0.83 201.21 2.18 8 6.00 1.81 522.85 3.63 1024 9.47 2.19 304.01 3.44 Duo 1 1103.17 3.83 7616.55 3.86 8 99.98 3.78 181.91 3.81 1024 72.04 3.81 84.65 3.82 MDLM 1 1064.26 3.83 6026.52 3.85 8 147.58 3.82 490.74 3.86 1024 58.64 3.80 61.71 3.79 CANDI 1024 93.09 3.84 – – RDLM 1024 72.79 3.21 – – LangFlow 1 2.94 0.32 72.60 0.74 8 242.77 3.87 743.79 3.78 1024 64.05 3.81 83.30 3.31 ELF 1 – – 4.50 1.82 8 – – 91.49 4.99 1024 – – 10.59 4.49 Seq-DriftingS 1 79.13 3.74 119.13 4.10 Seq-DriftingP 1 76.16 4.40 168.34 4.78 B.4 Alternative Gen-PPL evaluators The main comparison reports Gen-PPL under GPT-2 Large. To test whether the results depend on that evaluator, we recompute unconditional Gen-PPL with Pythia-1B and OPT-1.3B in Table 8. All other sampling settings are unchanged. Table 8: Unconditional Gen-PPL under alternative evaluators. Gen-PPL computed with Pythia-1B and OPT-1.3B for LM1B (T=128T=128) and OpenWebText (T=1024T=1024). Method Step LM1B OpenWebText Pythia-1B OPT-1.3B Pythia-1B OPT-1.3B FMLM 1 94.75 97.29 170.10 152.67 2 87.55 88.58 134.89 122.45 4 79.86 80.37 109.44 100.41 Duo+DCD 1 829.59 849.31 4654.30 4401.95 2 435.40 453.47 905.38 792.78 4 254.97 260.45 246.35 219.22 Duo+Di4C 1 – – 1029.73 1061.16 2 – – 736.75 728.92 4 – – 252.60 251.94 MDLM+SDTT 1 919.99 935.31 1024.45 1086.82 2 397.09 414.49 802.18 809.25 4 161.86 162.99 306.35 307.63 MDLM+Di4C 1 1088.63 1120.40 1025.45 1067.07 2 601.01 626.70 702.27 695.89 4 266.63 273.17 216.24 215.50 Seq-DriftingS 1 97.04 98.70 130.48 151.38 Seq-DriftingP 1 36.07 43.07 110.36 119.68 B.5 Translation and summarization setup For WMT14, the input contains a 64-token source sequence and the model generates a 64-token target sequence. For XSum, the input contains up to 1024 source tokens and the model generates a 64-token summary. We report corpus BLEU (Papineni et al., 2002) for WMT14 and ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004) for XSum. For Seq-Drifting, we first train an autoregressive model from scratch on each task’s training split. In the small support-model setting, this model is a 12-layer pre-LN Transformer with hidden width 768768, 12 attention heads, FFN width 30723072, and the same tokenizer and context lengths used by the corresponding experiment. It uses a causal attention mask and is trained with the standard next-token cross-entropy objective. We then train Seq-Drifting using the frozen autoregressive model’s token distributions as its on-policy support signal, without initializing from a pretrained language model. At evaluation time, we compare generated outputs against both the dataset ground-truth references and samples generated by the autoregressive model, using the same task metrics in both cases. B.6 Reasoning implementation details For mathematical reasoning, we train on GSM8K-Aug (Deng et al., 2023) and evaluate out-of-domain generalization on SVAMP and ASDiv. Ground-truth solution sequences define the support set for the attraction objective. For logical reasoning, we use the binary (true/false) subset of ProofWriter, training on the combined D0–D5 training split and evaluating on the corresponding test split; unknown-labeled examples are excluded. Seq-Drifting uses a 0.50.5B-parameter backbone with the Qwen2.5 architecture. We compare it with autoregressive (AR), non-autoregressive (NAR), and latent-reasoning baseline Coconut (Hao et al., 2025). We report answer accuracy and wall-clock time per sample for each benchmark in Table 4. • Coconut performs latent reasoning by feeding continuous thought representations back into the model’s hidden states instead of emitting explicit chain-of-thought tokens. B.7 Training dynamics Figure 8 tracks Gen-PPL and unigram entropy over training for the conditional runs. The first 200200k steps are the unconditional warm-up phase, during which the context is filled with a constant token under normal attention. The drift reaches coherent English and drives Gen-PPL down by orders of magnitude; real prefix conditioning then turns on and perplexity keeps falling. Figure 8: Training dynamics of the conditional runs. Gen-PPL (log scale, left axis) and unigram entropy (right axis) versus training step, for Seq-DriftingP 64→6464→64 (top left), Seq-DriftingS 64→6464→64 (top right), Seq-DriftingP 64→51264→512 (bottom left), and Seq-DriftingS 64→51264→512 (bottom right). All panels show training through 300300k steps for both output lengths. The dashed line marks the warm-up→ switch at 200200k steps. B.8 Off-policy positives The off-policy ablation of §5.4 constructs support sets from corpus retrieval instead of language-model predictions. We precompute a memory bank of the token n-gram embeddings of a set of real sequences, and at each generated position we take as the support set tS_t the tokens that real sequences place at the position where they best match the current generation, ranked by the position-aligned min-n-gram distance of §4.2. Everything downstream, the angular attraction, the two repulsions, and the geodesic update, is unchanged. The one difference from the on-policy signal is that these targets are fixed by the data and do not follow the generator. Observed training behavior. Figure 9 tracks this run at length 128128. During the temperature warm-up, over which the attraction temperature τ anneals from 1.01.0 to 0.10.1 across the first 1010k steps, Gen-PPL falls as the soft, mode-covering attraction pulls the batch toward real n-grams. Once τ reaches its sharp steady value at 1010k, each generation commits hard to its single nearest real n-gram. Because those targets are fixed and do not adapt, the sharp commitment lets the anti-collapse repulsion over-scatter the batch faster than the fixed attraction can pull it back, which sets up a limit cycle. The repulsion over-scatters the batch, so entropy and Gen-PPL spike. The samples are then far apart, the repulsion weakens, and the attraction pulls them back toward real tokens, so entropy and Gen-PPL fall. Once re-consolidated the repulsion strengthens again and the cycle repeats, which is the sawtooth in Figure 9. The cycle does not close, each swing ends higher than the last, so Gen-PPL and entropy trend upward overall and the run moves toward higher-entropy text with reduced global coherence. The on-policy support model avoids this this because it scores the generation’s own prefix, so its targets follow the generator and the attraction stays balanced against the repulsion. Figure 9: Off-policy training dynamics (unconditional, length 128128). Gen-PPL (log scale, left axis) and unigram entropy (right axis) versus training step for the off-policy positives ablation. The dashed line marks the end of the temperature warm-up at 1010k steps, where τ reaches 0.10.1 and the run begins to oscillate and degrade. B.9 Qualitative samples The following examples are produced in a single forward pass and truncated for display. Seq-DriftingP entropy: 4.30 Gen-PPL: 36.01 It will provide an excellent overview of a wide variety of topics: "Innovations & Challenges of the Future," "A History of Technology" and, the latest developments and advancements that the technology has in general and the development and future capabilities in specific applications. This is one of our largest conferences ever organized for industry-leading attendees in North American technology industry, which will give you more insight as you go to work with technology. About the Author: Klaus K. Krieg was awarded his doctor, … Seq-DriftingS entropy: 4.21 Gen-PPL: 93.81 A spokeswoman at the Brown’s Downing Street offices said: "There it would not happen here at home, will be able to make up his mind whether that’s right another very me. i guess he has been very, but if he did have been opportunity, if he has been very very me. the so far as well ... and this time around there, if he did. i think terms people who knows about what happened on afternoon, … Seq-Drifting WMT Source: Wir sind stolz auf unsere Leistung, aber wir wollen jedes Spiel gewinnen. Reference: We are proud of our performance, but we want to win every game. Prediction: We are proud of our performance, but we want to win every game. Seq-Drifting XSum Source: Police said officers from the North West counter terrorism unit searched an address on Peakdale Avenue, Crumpsall, Manchester on Friday. The suspect, 26, was arrested the following day on suspicion of offences under the Terrorism Act. Police would not comment on the nature of the alleged offence but said it believed it caused "no threat" to the community. Reference: A man is being held in Manchester on suspicion of terrorism offences. Prediction: A man has been arrested on suspicion of terrorism offences Seq-Drifting Math Reasoning Question: A group of students are preparing for a water balloon fight. Each student needs 5 balloons. If there are 12 students, how many balloons are needed in total? Prediction: <<5*12=60>> #### 60 Seq-Drifting ProofWriter Question: Theory: Bob is kind. Bob is nice. Bob is white. Charlie is kind. Charlie is white. Erin is red. Erin is rough. Harry is blue. Harry is kind. Harry is red. Blue people are nice. All white people are red. If someone is white and blue then they are nice. All rough people are red. If someone is smart then they are blue. If someone is kind then they are red. If someone is nice then they are rough. If someone is red then they are smart. If someone is red and rough then they are nice. Query: Bob is not nice. Give the proof in order and finish with #### true or #### false. Prediction: Fact: Bob is nice. #### false