Paper deep dive

In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion

Lingxiao Yang, Liu Liu, Moran Li, Han Feng, Wenjian Cao, Jiangning Zhang, Ye Shi

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 72

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 8/8/2026, 2:40:12 AM

Summary

The paper introduces 'In-Context Forcing', a progressive autoregressive paradigm for few-step video diffusion models. It addresses the issue of excessive local detail leakage from clean context frames by utilizing contexts with decreasing noise levels (higher noise for adjacent frames, lower for distant ones). This approach ensures robust temporal consistency and high inter-frame dynamics. The method also introduces a Step-wise Rolling KV Cache for training consistency and enables cross-frame parallel denoising via causal attention, significantly accelerating inference speed without sacrificing visual fidelity.

Entities (8)

In-Context Forcing · method · 98%Step-wise Rolling KV Cache · component · 92%Cross-frame Causal Attention · component · 90%VBench · benchmark · 90%Distribution Matching Distillation (DMD) · training-objective · 85%Rolling Forcing · baseline-method · 85%Self Forcing · baseline-method · 85%Diffusion Transformers (DiTs) · model-architecture · 80%

Relation Signals (6)

In-Context Forcing → uses → contexts with decreasing noise levels

confidence 95% · In-Context Forcing... utilizes contexts with decreasing noise levels. By applying less masking to distant frames and more masking to adjacent ones...

In-Context Forcing → enables → cross-frame parallel denoising

confidence 92% · by decoupling the strict dependence on previous clean frames, our paradigm enables cross-frame parallel denoising

In-Context Forcing → outperforms → state-of-the-art approaches

confidence 90% · Extensive experiments on VBench demonstrate that our method significantly outperforms state-of-the-art approaches

Cross-frame Causal Attention → reduces → memory requirement

confidence 88% · reduces the memory requirement to a single KV cache

Self Forcing → suffersfrom → excessive local detail leakage

confidence 88% · These clean frames leak excessive local details, which causes the model to take shortcuts

Step-wise Rolling KV Cache → supports → train-test consistency

confidence 85% · accurately simulates inference behavior, successfully preserving train-test consistency

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Current few-step autoregressive video diffusion models depend on previous fully denoised clean frames as context for all denoising steps of the current frame. However, these clean frames leak excessive local details, which causes the model to take shortcuts, resulting in compromised temporal semantics and dynamics. Inspired by the perspective of diffusion as masking, we explore the impact of noisy contexts on few-step autoregressive generation. Yet, simply applying contexts with the same noise levels provides insufficient guidance, leading to poor temporal consistency. To resolve this dilemma, we introduce In-Context Forcing, a progressive autoregressive paradigm that utilizes contexts with decreasing noise levels. By applying less masking to distant frames and more masking to adjacent ones, this approach provides adaptive guidance, effectively ensuring both robust temporal consistency and high inter-frame dynamics. Furthermore, by decoupling the strict dependence on previous clean frames, our paradigm enables cross-frame parallel denoising, achieving substantial inference acceleration without sacrificing performance. Extensive experiments on VBench demonstrate that our method significantly outperforms state-of-the-art approaches in both visual fidelity and inference speed.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

70,492 characters extracted from source content.

Expand or collapse full text

In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion †Lingxiao Yang, †Liu Liu, †Moran Li, Han Feng, Wenjian Cao, Jiangning Zhang, and Ye Shi † L. Yang, L. Liu and M. Li contributed equally to this work. (Corresponding author: Ye Shi.)L. Yang, L. Liu, and Y. Shi are with the School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China (e-mail: yanglx23@shanghaitech.edu.cn; liuliu2025@shanghaitech.edu.cn; shiye@shanghaitech.edu.cn).M. Li , W. Cao and H. Feng are with Tencent Youtu Lab (e-mail: moranli.aca@gmail.com; weijiancao@tencent.com; whuerfff@whu.edu.cn).J. Zhang is with the College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China (e-mail: 186368@zju.edu.cn). Abstract Current few-step autoregressive video diffusion models depend on previous fully denoised clean frames as context for all denoising steps of the current frame. However, these clean frames leak excessive local details, which causes the model to take shortcuts, resulting in compromised temporal semantics and dynamics. Inspired by the perspective of diffusion as masking, we explore the impact of noisy contexts on few-step autoregressive generation. Yet, simply applying contexts with the same noise levels provides insufficient guidance, leading to poor temporal consistency. To resolve this dilemma, we introduce In-Context Forcing, a progressive autoregressive paradigm that utilizes contexts with decreasing noise levels. By applying less masking to distant frames and more masking to adjacent ones, this approach provides adaptive guidance, effectively ensuring both robust temporal consistency and high inter-frame dynamics. Furthermore, by decoupling the strict dependence on previous clean frames, our paradigm enables cross-frame parallel denoising, achieving substantial inference acceleration without sacrificing performance. Extensive experiments on VBench demonstrate that our method significantly outperforms state-of-the-art approaches in both visual fidelity and inference speed. I Introduction Recent advances in video generation [2, 3, 4, 11, 12, 13, 15, 20, 26, 34, 35, 43] have enabled the synthesis of high-fidelity clips with remarkable temporal coherence and visual detail. Yet, most state-of-the-art approaches, typically based on diffusion models [14, 29, 32], rely on bidirectional attention mechanisms in Diffusion Transformers (DiTs) [25, 35] to generate entire sequences simultaneously. These methods often require multiple denoising steps, resulting in slow inference and limiting their use to offline scenarios. In contrast, many real-world interactive applications, such as game simulation [9, 33, 46], live content creation [7, 22], and robotics [21, 42], require videos to be generated sequentially under strict real-time constraints. While the autoregressive generation paradigm naturally fits this streaming setup, AR-only models [5, 19, 37, 38, 40] often struggle to match the visual quality of their diffusion-based counterparts. Bridging this gap to enable sequential generation with both high fidelity and low latency is therefore a critical challenge for real-time applications. To enable real-time video generation, recent studies have explored combining autoregressive paradigms with diffusion distillation. CausVid [45] introduces an asymmetric distillation strategy within the DMD [44] framework and follows the training paradigm of Diffusion Forcing [6, 8, 28, 30]. However, this creates a critical train-test gap: training on ground-truth contexts while inferring with imperfect self-generated contexts leads to accumulated errors. Recently, Self Forcing [16] alleviates this train-test gap by reusing the previously generated clean frames as context to generate the next frame during training, as shown in Fig. 1(a). However, these models depend on previous clean frames as context for all denoising steps of the current frame. These clean contexts leak local details, leading the model to directly replicate patterns from previous frames rather than treating them as meaningful guidance. Particularly during early denoising stages, this flaw causes the model to take shortcuts, as evidenced by the excessively high attention allocated to adjacent regions in previous frames, resulting in compromised temporal semantics and dynamics. Moreover, Rolling Forcing [23] adopts bidirectional attention within denoising windows to suppress error accumulation. However, this design inherently forces a split training paradigm: during training, a random exit flag determines which window writes the generator output, causing different frames within the same video to be predicted at heterogeneous denoising steps. Consequently, the resulting x^0 x_0 estimates exhibit drastically different quality—some frames are coarse one-step predictions from pure noise while others are refined predictions near t0t_0—creating a mixed-quality output tensor that the DMD critic evaluates on. This trains the fake score to track a heterogeneous mixture distribution rather than the inference-time distribution, where every frame originates from t0t_0, fundamentally misaligning the optimization objective. Inspired by the perspective of diffusion as masking, we explore the impact of noisy contexts to mitigate this issue. We first discover that simply applying contexts with the same noise levels matching the current frame masks excessive information, providing insufficient guidance and leading to poor temporal consistency, as illustrated in Fig. 1(b). To resolve this dilemma, we introduce In-Context Forcing, a progressive autoregressive paradigm that utilizes contexts with decreasing noise levels. By applying less masking to distant frames and more masking to adjacent ones, this approach provides adaptive guidance, effectively ensuring both robust temporal consistency and high inter-frame dynamics. Figure 1: Comparison of contextual paradigms in few-step autoregressive video generation. (a) Self Forcing: depends on previous fully denoised clean frames as context, which causes excessive local detail leakage and compromises temporal semantics and dynamics. (b) Identical Noise Context: applies contexts with the same noise level as the current frame, which masks excessive information and provides insufficient guidance. (c) In-Context Forcing: introduces a progressive paradigm utilizing contexts with decreasing noise levels. By applying less masking to distant frames and more masking to adjacent ones, it provides adaptive guidance, ensuring robust temporal consistency and high inter-frame dynamics.Video frames are generated by a text-to-video (T2V) model. While this progressive autoregressive paradigm offers significant advantages, maintaining distinct progressive contexts for each denoising step introduces additional complexity during training and inference. To address this during training, we propose a Step-wise Rolling Key-Value (KV) Cache mechanism. At each denoising step, this approach stores the KV cache corresponding to the current noise level. Then, it performs a bottom-up rolling update after each frame’s denoising is completed, dynamically maintaining contexts at decreasing noise levels. Simultaneously, this mechanism accurately simulates inference behavior, successfully preserving train-test consistency while supporting our progressive autoregressive paradigm. During inference, applying this context update strategy would incur substantial additional VRAM overhead due to the introduction of multi-level KV caches. Fortunately, by decoupling the strict dependence on fully denoised previous frames inherent in current few-step autoregressive models, our paradigm naturally supports inter-frame parallelization. Therefore, we introduce cross-frame causal attention, which not only reduces the memory requirement to a single KV cache but also achieves substantial inference acceleration on a single GPU through cross-frame parallel denoising. In summary, our contributions are threefold. • We propose In-Context Forcing, a novel progressive autoregressive paradigm that mitigates excessive local detail leakage from clean contexts in current models. Our approach utilizes contexts with decreasing noise levels, specifically applying higher noise to adjacent frames and lower noise to distant ones. This provides adaptive guidance, effectively ensuring robust temporal consistency and high inter-frame dynamics. • We introduce a Step-wise Rolling KV Cache to address the complexity of maintaining distinct progressive contexts during training. By storing the KV cache corresponding to the current noise level at each denoising step and performing a bottom-up rolling update after each frame’s denoising is completed, this approach dynamically maintains contexts at decreasing noise levels. • Extensive experiments on both standard short-video and extended long-video generation tasks demonstrate the superiority of our method. By decoupling the strict dependence on fully denoised previous frames, our paradigm naturally supports inter-frame parallelization. This enables the use of cross-frame causal attention during inference, which reduces the memory requirement to a single KV cache and achieves substantial acceleration without compromising performance. I Related Work I-A Video Generation Models Video generation models [2, 20, 26, 35, 43] typically employ bidirectional attention to generate all frames simultaneously, yielding high visual fidelity and temporal consistency. To support streaming video applications, recent autoregressive models [5, 19, 27, 40] equipped with causal attention adopt a next-token prediction paradigm. While this sequential framework reduces inference latency, it often suffers from degraded visual quality due to the accumulation of prediction errors. AR-diffusion hybrid models [1, 6, 18, 24, 39] combine the advantages of both paradigms and show promising potential. Among these methods, Diffusion Forcing [6] adds independent noise to each frame during training, enabling the model to adapt to different noise contexts during inference and support a pyramid-style inference schedule. However, its reliance on ground-truth training contexts creates an exposure bias that is significantly amplified during few-step inference. To address this, our In-Context Forcing proposes a train-test consistent formulation. By constructing training contexts via self-simulation rather than ground truth, we force the model to explicitly learn error correction, effectively bridging the train-test gap for robust progressive generation. I-B Diffusion Distillation Video diffusion models produce high-quality results but suffer from slow iterative inference. To address this, distilling multi-step teachers into fewer-step students has emerged as a key direction to maintain generation quality while significantly reducing latency. Trajectory distillation and score distillation are the two main paradigms for this task. Trajectory distillation, such as consistency models [31], uses fewer-step student models to simulate the multi-step teacher’s ODE trajectory, whereas score distillation matches output distributions across noise levels using the teacher’s score function. Methods such as DMD [44] and SiD [47] minimize distribution divergence, e.g., KL or Fisher divergence, for high-fidelity one-step generation, while DMD2 incorporates self-simulation and gradient truncation to enable efficient few-step generation. Score distillation particularly enables asymmetric setups, such as distilling a bidirectional teacher into a causal student. CausVid [45] first formalized this approach but suffers from a train-test gap due to its dependence on ground-truth noisy inputs during training. Self Forcing [16] mitigates this issue through self-simulation, reusing its previously generated clean frames as context to generate the next frame. Furthermore, Rolling Forcing [23] incorporates extrapolation techniques such as attention sinks and window-wise bidirectional attention, effectively alleviating quality drift over extended sequences to enable minute-level video generation. To avoid the detail leakage caused by clean contexts in existing methods, our progressive autoregressive paradigm utilizes contexts with decreasing noise levels. This provides two key benefits. First, it offers adaptive guidance for robust temporal consistency and dynamics. Second, decoupling the reliance on fully denoised frames enables seamless integration with cross-frame causal attention, achieving memory-efficient parallel denoising and substantial inference acceleration. I Methodology: In-Context Forcing We first formalize the autoregressive video diffusion paradigm in Section I-A. We then introduce In-Context Forcing, a progressive paradigm that utilizes contexts with decreasing noise levels in Section I-B. In Section I-C, we describe the Step-wise Rolling KV Cache, a mechanism designed to maintain these distinct progressive contexts while ensuring train-test consistency. Finally, Section I-D details the cross-frame parallel denoising scheme enabled by our paradigm, which achieves substantial inference acceleration. I-A Preliminaries: Few-step Autoregressive Video Diffusion The autoregressive video diffusion model combines the advantages of both diffusion and autoregressive models, preserving the autoregressive properties between frames while maintaining the iterative characteristics within each frame. Specifically, given a video consisting of N frames x1:N=(x1,x2,…,xN)x^1:N=(x^1,x^2,…,x^N), the frame-wise joint distribution can be expressed as p(x1:N)=∏i=1Np(xi∣x<i),p(x^1:N)= _i=1^Np(x^i x^<i), (1) where x<i=(x1,x2,…,xi−1)x^<i=(x^1,x^2,…,x^i-1) denotes the sequence of preceding i−1i-1 frames. In each frame, given the (T+1)(T+1)-step denoising schedule t0,t1,…,tT\t_0,t_1,…,t_T\ where t0=0t_0=0 and tT=1000t_T=1000, the step-wise conditional distribution of frame i can be expressed as p(x0i∣x0<i)=∏j=1T−1p(xtj−1i∣xtji,x0<i),p(x^i_0 x^<i_0)= _j=1^T-1p(x^i_t_j-1 x^i_t_j,x^<i_0), (2) where a diffusion process can be applied to achieve better visual quality. We can also generate a chunk of frames simultaneously to improve parallelism and frame-wise consistency. For simplicity of notation, each such chunk is treated as a single frame throughout this section. Most existing few-step autoregressive video diffusion models are distilled from bidirectional teacher diffusion models using Distribution Matching Distillation (DMD) loss: ∇ℒDMD _DMD =t(∇θKL(pfake,t∥preal,t)) =E_t ( _θKL(p_fake,t\|p_real,t) ) =−t,z,xt(sreal(xt,t)−sfake(xt,t))dGθ(z)dθ, =-E_t,z,x_t (s_real(x_t,t)-s_fake(x_t,t) ) dG_θ(z)dθ, (3) where z∼(0,I)z (0,I) is the input noise, GθG_θ denotes the student model, and xt=Ψ(x^0,ϵ,t)=αtx^0+σtϵx_t= ( x_0,ε,t)= _t x_0+ _tε (4) represents the noisy version of the student’s predicted clean video x^0=Gθ(z) x_0=G_θ(z). Here, sfakes_fake and sreals_real correspond to the score functions belonging to the student’s and teacher’s distributions, respectively. Since the DMD loss depends only on the student’s distribution and imposes no constraints on the student architecture, it naturally supports asymmetric distillation, enabling knowledge transfer from a multi-step bidirectional teacher to a few-step autoregressive student. Figure 2: Overview of the proposed In-Context Forcing framework. (a) Progressive autoregressive form: applies decreasing noise levels to preceding frames to provide adaptive guidance. (b) Step-wise Rolling KV Cache: maintains these progressive contexts during training via self-simulation to ensure train-test consistency. (c) Parallel inference via cross-frame causal attention: enables inter-frame parallel denoising to significantly accelerate inference. I-B In-Context Forcing While previous few-step autoregressive video diffusion models demonstrate strong performance, they typically treat inter-frame autoregression and intra-frame denoising as independent processes, which leads to two primary challenges. First, these models depend on previous clean frames as context for all denoising steps of the current frame. These clean contexts leak local details, causing the model to directly replicate patterns rather than treating them as meaningful guidance. Particularly during early denoising stages, this flaw leads the model to take shortcuts, which is evidenced by the high attention allocated to adjacent regions in previous frames. This results in compromised temporal semantics and dynamics. Second, this paradigm imposes a strict sequential dependency where subsequent frames can only begin denoising after the preceding ones are fully completed, thereby preventing cross-frame parallelization. Building on this foundation and inspired by the perspective of diffusion as masking, we introduce the progressive autoregressive paradigm. Under this view, the noise level acts as a mask that hides local details. This allows us to provide adaptive guidance by explicitly controlling the noise levels of preceding frames. The conditional distribution of this paradigm can be expressed as p(xtji∣x0:T<i)=p(xtji∣xtj−1i−1,…,xtj−ki−k,…),p(x^i_t_j x^<i_0:T)=p (x^i_t_j x^i-1_t_j-1,…,x^i-k_t_j-k,… ), (5) where 0<k≤i0<k≤ i, and the noise level tj−kt_j-k is clipped to t0t_0 when j−k<0j-k<0. As shown in Fig. 2(a), this design incorporates contextual information from preceding frames across decreasing noise levels to facilitate the denoising of the current frame. By explicitly applying higher noise to adjacent frames and lower noise to distant ones, this approach provides adaptive guidance. This effectively prevents the model from taking shortcuts, ensuring robust temporal consistency and high inter-frame dynamics. The underlying intuition is that early denoising stages primarily require coarse structural guidance to establish global layout and motion trajectories, whereas later stages demand fine-grained spatial details for precise texture synthesis. By masking adjacent frames heavily at early steps, our paradigm suppresses the leakage of low-level details that would otherwise be directly copied, forcing the model to rely on higher-level semantic signals from noisier contexts. Conversely, as denoising progresses and the current frame’s content becomes more determined, progressively cleaner contexts from distant frames provide the precise spatial correspondence needed for coherent detail refinement. This coarse-to-fine contextual scheduling thus naturally aligns the information content of each context frame with the denoising objective at each step. I-C In-Context Forcing Training with Step-wise Rolling KV Cache Step-wise Rolling KV Cache. To maintain distinct progressive contexts for each denoising step, we propose the Step-wise Rolling KV Cache, a set of dedicated caches, each corresponding to a specific denoising timestep t. This design enables a progressive autoregressive schedule through recursive updates between caches, as shown in Fig. 2(b). Specifically, during the denoising of a single frame, we store the KV cache corresponding to the noise level of each timestep t. After completing the denoising of a frame, a bottom-up rolling update is performed across adjacent caches to maintain the progressive causal context. Bridging the Train-Test Gap via Self-Simulation. In DMD-based few-step autoregressive diffusion, train-test consistency is particularly important because the critic directly optimizes the distribution of the student’s predicted clean outputs. Rolling Forcing’s split training violates this consistency: a random exit flag selects one window for the DMD loss, but different blocks within the selected window may correspond to different denoising stages. As a result, the resulting x^0 x_0 tensor contains predictions with heterogeneous quality, ranging from near-clean estimates at t0t_0 to coarse one-step estimates from high-noise inputs. The fake score network is therefore trained to fit a mixed-quality output distribution, rather than the inference-time distribution in which each generated frame is obtained after the full denoising trajectory and output at t0t_0. This mismatch causes the DMD objective to optimize a biased target and can be repeatedly propagated across rolling windows, especially in long-video generation. To achieve full train-test consistency, we integrate inference-stage scheduling into training via a self-simulation mechanism. Our KV cache design integrates seamlessly into this framework, requiring minimal modifications to Self Forcing while improving overall performance. Specifically, for a model with T+1T+1 steps, we randomly sample a step s≤Ts≤ T during training and simulate step-wise rolling KV caches for s+1s+1 steps. The resulting iterative output XθX_θ is then used as the student’s prediction for computing the DMD loss. However, directly computing parameter gradients along the entire diffusion trajectory often leads to prohibitive memory overhead. Therefore, we retain gradients only in the final denoising stage and truncate gradients flowing through the KV caches of preceding frames. Since we do not compute gradients for the KV caches, this truncation strategy also allows all KV caches except those needed for the current denoising step to be offloaded to the CPU, enabling memory consumption comparable to that of Self Forcing during training. Furthermore, even without CPU offloading, maintaining multiple KV caches increases the overall memory footprint by less than 30% compared to the standard training baseline. Algorithm 1 In-Context Forcing Training via Step-wise Rolling KV Cache 1: Require: Denoising timesteps t0,…,tT\t_0,…,t_T\ 2: Require: Number of video frames N 3: Require: AR diffusion model GθG_θ which returns predicted clean image x^0i x^i_0 and KV embeddings kvjikv^i_j of frame i and timestep index j 4: loop 5: Initialize model output Xθ←[]X_θ←[\ ] and KVKV cache pool KVPool←[[]]KVPool←[[\ ]] 6: Sample s∼Uniform(1,2,…,T)s (1,2,…,T) 7: for i=1,…,Ni=1,…,N do 8: Initialize xtTi∼(0,I)x^i_t_T (0,I) 9: for j=T,…,sj=T,…,s do 10: KV←KVPool[j] [j] 11: if j=sj=s then 12: Enable gradient computation 13: Set x^0i,kvji←Gθ(xtji;tj,KV) x^i_0,kv^i_j← G_θ(x^i_t_j;t_j,KV) 14: Xθ.append(x^0i)X_θ.append( x^i_0) 15: KV.append(kvji).append(kv^i_j) 16: Disable gradient computation 17: KV←KVPool[s−1] [s-1] 18: Cache kv0i←GθKV(x^0i;0,KV)kv^i_0← G^KV_θ( x^i_0;0,KV) 19: KV.append(kv0i)KV.append(kv^i_0) 20: else 21: Disable gradient computation 22: Cache kvji←GθKV(xtji;tj,KV)kv^i_j← G^KV_θ(x^i_t_j;t_j,KV) 23: KV.append(kvji).append(kv^i_j) 24: Sample ϵ∼(0,I)ε (0,I) 25: Set xtj−1i←Ψ(x^0i,ϵ,tj−1)x^i_t_j-1← ( x^i_0,ε,t_j-1) 26: end if 27: end for 28: for m=T,…,sm=T,…,s do 29: Set KVPool[m]←KVPool[m−1]KVPool[m] [m-1] 30: end for 31: end for 32: Update θ via distribution matching loss 33: end loop I-D Parallel Denoising via Cross-frame Causal Attention Reducing Memory Footprint by a Unified KV Cache. While Section I-C introduces a progressive autoregressive framework that ensures train-test consistency, its inference-time memory overhead remains challenging. Maintaining multiple KV caches consumes substantial GPU memory, and offloading them to the CPU introduces communication latency that degrades inference speed. However, we observe that this progressive autoregressive paradigm, by decoupling the strict inter-frame dependencies, enables cross-frame parallelism, as shown in Fig. 2(c). This allows us to replace multiple per-noise-level KV caches with a single unified cache, matching Self Forcing’s memory footprint while avoiding CPU offloading overhead. Figure 3: Visual Evidence of Train-Test Inconsistency: x^0 x_0 Quality Disparity at Output. Top: Rolling Forcing’s split training with random exit flag produces frames at heterogeneous denoising timesteps. At the output layer (t0t_0 row, red dashed box), the resulting x^0 x_0 estimates exhibit drastically inconsistent quality—Frame n+1n+1 is a clean prediction (denoised from t0t_0), Frame n shows weak residual noise (from t1t_1), Frame n+2n+2 retains strong noise artifacts (from t2t_2), and Frame n+3n+3 remains near-pure noise (from t3t_3). This mixed-quality tensor corrupts DMD training by forcing the fake score to fit a heterogeneous mixture distribution. Bottom: In-Context Forcing ensures every frame follows an identical denoising trajectory from t3→t0t_3→ t_0, producing uniformly high-quality x^0 x_0 outputs across all frames (green dashed box). This strict train-test consistency avoids the mixed-output distribution that degrades motion diversity in Rolling Forcing. Formally, we define a schedule matrix ∈ℝN×MT ^N× M as: =[tTtTtT⋯tTtT−1tTtT⋯tTtT−2tT−1tT⋯tT⋮⋱⋮t0t1t2⋯tT0t0t1⋯tT−1⋮⋯t1t0t0t0⋯t0],T= bmatrixt_T&t_T&t_T&·s&t_T\\ t_T-1&t_T&t_T&·s&t_T\\ t_T-2&t_T-1&t_T&·s&t_T\\ & & & & \\ t_0&t_1&t_2&·s&t_T\\ 0&t_0&t_1&·s&t_T-1\\ & & &·s&t_1\\ t_0&t_0&t_0&·s&t_0 bmatrix, (6) where each row represents the noise levels for an N-frame sequence, and each column tracks the evolution across M denoising steps. This matrix exhibits a key structural property: each column follows a monotonically non-decreasing trajectory from t0t_0 to tTt_T, encoding the progressive nature of our context schedule. The lower-triangular dominance of the matrix ensures that at any denoising step, a frame can only attend to preceding frames at equal or higher noise levels, which is precisely the constraint imposed by causal attention combined with our progressive noise scheduling. Importantly, the staircase pattern does not correspond to a strictly sequential autoregressive process where frame n+1n+1 must wait until frame n is fully denoised. Instead, it encodes a relaxed causal schedule: a new frame can begin denoising once its predecessors have advanced to sufficiently informative context states, enabling multiple frames to be processed in parallel while preserving the causal conditioning order. At each step, we identify active denoising indices and corresponding noise levels through inter-row comparison, process frames in parallel via cross-frame causal attention, and then re-noise outputs for the next iteration. Cross-frame Parallelism for Fast Inference. This parallelization addresses a key limitation in Self Forcing’s inference efficiency: while KV caching reduces the number of tokens in a single attention computation, it leads to suboptimal GPU utilization. This inefficiency explains why the standard frame-wise configuration, despite performing fewer total attention operations, results in slower inference than the chunk-wise setup. Our approach effectively mitigates this sequential bottleneck through cross-frame parallelism, which dramatically improves GPU utilization across all denoising stages without sacrificing generation quality. Consequently, our method achieves a maximum relative speedup of 82%, equivalent to a 45.1% reduction in total inference time, under the frame-wise setting. Furthermore, compared to Self Forcing, it still yields a 9.3% time reduction even in the already optimized chunk-wise configuration. Plug-and-Play Capability. Notably, by directly integrating this scheduling into the weights of the Self Forcing baseline in a training-free manner, we observe a seamless improvement in generation quality. We attribute this to the model’s initialization: Self Forcing inherits its weights from CausVid, which is initialized similarly to Diffusion Forcing and inherently learns to process contexts across diverse noise levels. Consequently, this robust foundation in generalized contextual awareness is well preserved even after subsequent distillation. Algorithm 2 In-Context Forcing Inference via Cross-frame Causal Attention 1: Require: Denoising timesteps t0,…,tT\t_0,…,t_T\ 2: Require: Number of generated video frames M 3: Require: Number of parallel sampling steps N 4: Require: AR diffusion model GθG_θ which returns predicted clean image x^0i x_0^i and KVKV embeddings kvkv of frame i. 5: Require: Schedule matrix ∈ℝN×MT ^N× M , with [i,j]T[i,j] as the noise level ti,jt_i,j at denoising step i for frame j. 6: Initialize model output Xθ←[]X_θ←[\ ] 7: Initialize KV cache KV←[]KV←[\ ] 8: Initialize Xθ←xtT1:M∼(0,I)X_θ← x^1:M_t_T (0,I) 9: for s=2,…,Ns=2,…,N do 10: ActiveMask←[i,:]−[i−1,:]ActiveMask [i,:]-T[i-1,:] 11: ActiveIndices←find(ActiveMask>0)ActiveIndices (ActiveMask>0) 12: tcurrent←[i,ActiveIndices]t_current [i,ActiveIndices] 13: xactive←Xθ[ActiveIndices]x_active← X_θ[ActiveIndices] 14: x^0,kv←Gθ(xactive;tcurrent,KV) x_0,kv← G_θ(x_active;t_current,KV) 15: KV.append(kv)KV.append(kv) 16: tnext←[i+1,ActiveIndices]t_next [i+1,ActiveIndices] 17: Sample ϵ∼(0,I)ε (0,I) 18: xnext←Ψ(x^0,ϵ,tnext)x_next← ( x_0,ε,t_next) 19: Xθ[ActiveIndices]←xnextX_θ[ActiveIndices]← x_next 20: end for 21: return XθX_θ IV Experiments IV-A Implementation Details Training. We implement In-Context Forcing using the causal variant of the Wan2.1-T2V-1.3B [35] architecture as the base model. The corresponding variant based on bidirectional attention can generate 5-second 480p video clips in 20 to 50 steps. We follow CausVid’s initialization procedure [45], in which the model is finetuned using 16K ODE pairs sampled from the original bidirectional model together with the causal attention mask. We conduct experiments using a 4-step chunk-wise autoregressive diffusion model, where each chunk jointly generates 3 frames unless otherwise specified. The prompts used for ODE initialization and DMD distillation are drawn exclusively from text prompts provided by a filtered and LLM-extended version of VidProM [36]. Consequently, no video data is required for the experiments. For DMD asymmetric distillation, we employ the Wan2.1-T2V-14B model based on bidirectional attention as the teacher, serving as both the real and fake score networks. Additional hyperparameters are provided in Appendix B. Evaluation. We holistically evaluate our method on both standard short-video and extended long-video generation tasks. For quantitative assessment, we adopt VBench [17] to measure comprehensive dimensions such as visual quality, temporal consistency, and semantic alignment. Specifically, we evaluate standard short clips alongside 30-second long sequences, for which we utilize inference-time extrapolation techniques following Rolling Forcing [23]. Furthermore, we conduct a rigorous blind, randomized user study to gauge overall human preference through A/B testing, recorded as Better, Same, or Bad. Finally, inference speed is measured by throughput, i.e., FPS, following Self Forcing [16]. IV-B Comparison with State-of-the-Art We compare our method with the most relevant open-source models. The baselines include two diffusion models, Wan2.1 [35] and LTX-Video [12], where Wan2.1 also serves as the base model used in our distillation. We further include several autoregressive models such as MAGI-1 [28], SkyReels-V2 [8], NOVA [10], Pyramid Flow [18], CausVid [45], Self Forcing [16], and Rolling Forcing [23]. TABLE I: Comparison with state-of-the-art methods on short-video generation. We compare In-Context Forcing with current models in terms of inference speed and VBench scores on standard short clips. Model #Params Throughput (FPS) ↑ Evaluation Scores ↑ Total Score Quality Score Semantic Score Dynamic Degree Wan2.1 1.3B [35] 1.3B 0.78 84.26 85.30 80.09 61 LTX-Video [12] 1.9B 8.98 80.00 82.30 70.79 46 MAGI-1 [28] 4.5B 0.19 79.18 82.04 67.74 42 SkyReels-V2 [8] 1.3B 0.49 82.67 84.70 74.53 37 NOVA [10] 0.6B 0.88 80.12 80.39 79.05 46 Pyramid Flow [18] 2B 6.7 81.72 84.74 69.62 16 CausVid [45] 1.3B 17.0 81.20 84.05 69.80 62 Self Forcing* [16] 1.3B 17.0 83.95 84.67 81.05 63 Rolling Forcing* [23] 1.3B 13.6 83.03 83.69 80.38 43 In-Context Forcing* 1.3B 18.4 84.34 84.99 81.77 72 *The evaluation was conducted with a consistent random seed across experiments. Figure 4: Qualitative comparison with state-of-the-art methods of the same architecture, i.e., 1.3B parameters, on short-video generation. Our method produces frames with richer motion dynamics and stronger semantic coherence with the text prompt, while Self Forcing and CausVid tend to replicate patterns from preceding frames, resulting in static or repetitive motion. Video frames featuring a person are generated by a text-to-video (T2V) model. Superior Video Quality and Semantic Alignment. The qualitative and quantitative results for both standard short clips and long sequences are shown in Table I, Table I, Fig. 4, Fig. 5, and Fig. 6. Our method surpasses all baseline models in VBench scores, demonstrating superior visual quality, dynamics, and semantic alignment across both short and long video generation tasks, as well as in the user preference study. This improvement stems from the diverse contextual signals used during both training and inference, which ensure that the full information from preceding frames influences the current frame only in the later denoising stages. This prevents the model from directly copying frame-specific patterns and thereby enhances temporal dynamics. Moreover, since the contextual features come from higher noise levels during the early denoising stages, they primarily convey coarse semantic structure rather than fine details. This provides stronger global semantic guidance, which in turn improves semantic coherence in the generated videos. Figure 5: User preference study on short-video generation. Participants performed blind A/B testing between our method and each baseline, rating videos as “Better,” “Same,” or “Worse.” Our approach is consistently preferred across all comparisons. User Study Setup. We conduct a blind, randomized A/B user study to assess human preference. From VBench [17], we randomly sample a diverse set of extended prompts covering varied scenes, subjects, and motion types, and use them to generate videos with our method as well as three baselines: Wan2.1 [35], CausVid [45], and Self Forcing [16]. For each prompt and each baseline, the two videos are presented side by side in a randomized left-right order with method identities hidden. Participants are then asked to compare overall generation quality by selecting one of three labels: Better, Same, or Worse. Each video pair is rated by multiple independent participants to mitigate individual bias, and the aggregated results across all comparisons are reported in Fig. 5. Further details are given in Appendix D. Accelerated Inference via Parallel Denoising. In addition, our method yields substantial improvements in inference speed, as detailed in Table I. Under the frame-wise setting, where standard autoregressive models typically suffer from severe sequential bottlenecks and low GPU utilization, our cross-frame parallelism dramatically mitigates this issue. It boosts the throughput from 8.9 FPS to 16.2 FPS, achieving an 82% relative speedup, or a 45.1% reduction in total inference time, over Self Forcing. Furthermore, even under the already optimized chunk-wise setting, our model still achieves a 9.3% reduction in total inference time. These gains stem from our cross-frame causal attention mechanism, which processes multiple frames simultaneously to maximize hardware utilization, resulting in significantly accelerated inference without loss of accuracy. TABLE I: Inference speed comparison between Self-forcing and In-Context Forcing. The relative time reductions are highlighted in green. Model Total Time ↓ Diffusion Time ↓ VAE Time ↓ FPS ↑ Self-forcing (frame-wise) 9.10s 7.25s 1.85s 8.9 In-Context Forcing (frame-wise) 5.00s (-45.1%) 3.15s (-56.6%) 1.85s 16.2 Self-forcing (chunk-wise) 4.85s 3.00s 1.85s 17.0 In-Context Forcing (chunk-wise) 4.40s (-9.3%) 2.54s (-15.3%) 1.85s 18.4 TABLE I: VBench evaluation for 30-second long-video generation. Method Total Score Quality Score Semantic Score Dynamic Degree Self Forcing 80.44 80.95 78.40 52.51 Rolling Forcing 82.40 82.81 80.82 40.63 In-Context Forcing 82.97 83.49 80.90 86.40 Figure 6: Qualitative comparison with state-of-the-art methods of the same architecture, i.e., 1.3B parameters, on 30-second long-video generation. Our method sustains diverse motion throughout extended sequences.Video frames are generated by a text-to-video (T2V) model. IV-C Ablation Study Attention Map Analysis. Fig. 7 visualizes the cross-frame causal attention during the initial denoising step. As highlighted in the zoomed-in regions, baselines such as Self Forcing exhibit a sharply concentrated diagonal, placing excessive attention on the exact spatial locations of the preceding frame. Consequently, this strict spatial localization causes the model to over-emphasize local details, leading to excessive pattern replication. In contrast, our In-Context Forcing yields a smoother, more dispersed attention distribution. By utilizing contexts with decreasing noise levels, it prevents local over-reliance and maintains a broader semantic receptive field. As generation progresses, our attention maps adaptively converge to those of Self Forcing to facilitate fine-grained detail refinement. A complete visualization together with its extraction protocol is provided in Appendix C-A. Figure 7: Cross-frame causal attention maps in the initial denoising step. Visualizing attention from the current frame’s Query (Q) to the preceding frame’s Key (K), Self Forcing exhibits a sharply concentrated diagonal that over-emphasizes local details. In contrast, our method yields a smoother, more dispersed distribution to maintain a broader semantic receptive field. Plug-and-Play Improvement. We compare the clean context used in Self Forcing with the progressive context introduced in our approach. Experiments show that even when the progressive context is applied in a plug-and-play manner to the same model weights trained with Self Forcing, it still achieves plug-and-play performance improvement, as shown in Table IV. Notably, without specific training on the progressive context, the model demonstrates strong generalization capability by effectively supporting progressive context inference, which may be attributed to the model initialization strategy described in Section I-D. TABLE IV: Quantitative comparison with state-of-the-art methods across six dimensions of VBench. Model Dynamic Degree Subject Consistency Aesthetic Quality Object Class Spatial Relationship Scene Self Forcing 0.633 0.956 0.660 0.938 0.814 0.553 In-Context Forcing* 0.653 0.962 0.663 0.942 0.819 0.563 In-Context Forcing 0.719 0.959 0.665 0.964 0.842 0.572 *Our method using the model weights from Self Forcing with the modified inference procedure in Section I-D. Training for Semantic and Motion Enhancement. Table IV presents detailed VBench evaluations across six dimensions, including three visual quality metrics, i.e., dynamic degree, subject consistency, and aesthetic quality, and three semantic alignment metrics, i.e., object class, spatial relationship, and scene. Our method significantly outperforms Self Forcing in terms of dynamic degree. We attribute this improvement to the high-noise contexts applied during early denoising stages, which effectively prevent the direct replication of preceding frames and thereby enhance overall temporal dynamics. Furthermore, our approach demonstrates clear improvements in semantic alignment, particularly in the trained model. This suggests that the contextual paradigm of In-Context Forcing better captures global semantic information, serving as an additional benefit of utilizing high-noise contexts during the initial denoising phase. The gain is especially pronounced in long-video generation, where the dynamic degree gap over Rolling Forcing widens dramatically (86.40 vs. 40.63, Table I). This confirms that Rolling Forcing’s train-test inconsistency disproportionately degrades motion diversity over extended sequences, as the mismatch is repeatedly propagated across successive rolling windows. In contrast, our cross-frame causal attention preserves strict train-test consistency throughout generation, sustaining motion diversity regardless of video length. V Conclusion This work identified a key limitation in few-step autoregressive video diffusion models: the isolation between inter-frame autoregression and intra-frame denoising. Strictly relying on fully denoised frames as context leaks excessive local details, causing shortcut learning that degrades temporal dynamics and prevents parallel generation. To solve these issues, we introduced In-Context Forcing, a progressive paradigm that utilizes contexts with decreasing noise levels. By applying higher noise to adjacent frames and lower noise to distant ones, this approach provides adaptive guidance to prevent pattern replication. To ensure strict train-test consistency, we further proposed the Step-wise Rolling KV Cache through self-simulation. Finally, by decoupling the reliance on clean contexts, our method enables cross-frame causal attention for inter-frame parallel denoising. Extensive evaluations have demonstrated that our approach achieves superior semantic alignment and dynamic degree, along with substantial inference acceleration. Limitations and future directions are discussed in Appendix E. VI Acknowledgments This work was supported by the National Natural Science Foundation of China under Grants 62406195, the HPC Platform of ShanghaiTech University, and Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), Ministry of Education. References [1] M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov (2025) Block diffusion: interpolating between autoregressive and diffusion language models. In Proc. ICLR, Cited by: §I-A. [2] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. (2023) Stable video diffusion: scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127. Cited by: §I, §I-A. [3] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis (2023) Align your latents: high-resolution video synthesis with latent diffusion models. In Proc. CVPR, Cited by: §I. [4] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, N. Ng, R. Wang, and A. Ramesh (2024) Video generation models as world simulators. Note: https://openai.com/research/video-generation-models-as-world-simulatorsAccessed: Jul. 13, 2026 Cited by: §I. [5] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, N. Apps, et al. (2024) Genie: generative interactive environments. In Proc. ICML, Cited by: §I, §I-A. [6] B. Chen, D. M. Monsó, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann (2024) Diffusion forcing: next-token prediction meets full-sequence diffusion. In Proc. NeurIPS, Cited by: §I, §I-A, §I-A. [7] F. Chen, Z. Yang, B. Zhuang, and Q. Wu (2024) Streaming video diffusion: online video editing with diffusion models. arXiv preprint arXiv:2405.19726. Cited by: §I. [8] G. Chen, D. Lin, J. Yang, C. Lin, J. Zhu, M. Fan, H. Zhang, S. Chen, Z. Chen, C. Ma, et al. (2025) SkyReels-V2: infinite-length film generative model. arXiv preprint arXiv:2504.13074. Cited by: §I, §IV-B, TABLE I. [9] Decart, J. Quevedo, Q. McIntyre, S. Campbell, X. Chen, and R. Wachen (2024) Oasis: a universe in a transformer. Note: https://oasis-model.github.io/Accessed: Jul. 13, 2026 Cited by: §I. [10] H. Deng, T. Pan, H. Diao, Z. Luo, Y. Cui, H. Lu, S. Shan, Y. Qi, and X. Wang (2025) Autoregressive video generation without vector quantization. In Proc. ICLR, Cited by: §IV-B, TABLE I. [11] A. Gupta, L. Yu, K. Sohn, X. Gu, M. Hahn, F.-F. Li, I. Essa, L. Jiang, and J. Lezama (2024) Photorealistic video generation with diffusion models. In Proc. ECCV, Cited by: §I. [12] Y. HaCohen, N. Chiprut, B. Brazowski, D. Shalem, D. Moshe, E. Richardson, E. Levin, G. Shiran, N. Zabari, O. Gordon, et al. (2024) LTX-Video: realtime video latent diffusion. arXiv preprint arXiv:2501.00103. Cited by: §I, §IV-B, TABLE I. [13] J. Ho, W. Chan, C. Saharia, C. Whang, R. Gao, A. A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans (2022) Imagen video: high definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. Cited by: §I. [14] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Proc. NeurIPS, Cited by: §I. [15] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet (2022) Video diffusion models. In Proc. NeurIPS, Cited by: §I. [16] X. Huang, Z. Li, G. He, M. Zhou, and E. Shechtman (2025) Self forcing: bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009. Cited by: Appendix B, Appendix D, §I, §I-B, §IV-A, §IV-B, §IV-B, TABLE I. [17] Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu (2024) VBench: comprehensive benchmark suite for video generative models. In Proc. CVPR, Cited by: Appendix B, Appendix D, §IV-A, §IV-B. [18] Y. Jin, Z. Sun, N. Li, K. Xu, H. Jiang, N. Zhuang, Q. Huang, Y. Song, Y. Mu, and Z. Lin (2025) Pyramidal flow matching for efficient video generative modeling. In Proc. ICLR, Cited by: §I-A, §IV-B, TABLE I. [19] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M. C. Chiu, et al. (2024) VideoPoet: a large language model for zero-shot video generation. In Proc. ICML, Cited by: §I, §I-A. [20] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) HunyuanVideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Cited by: §I, §I-A. [21] S. Li, Y. Gao, D. Sadigh, and S. Song (2025) Unified video action model. arXiv preprint arXiv:2503.00200. Cited by: §I. [22] F. Liang, A. Kodaira, C. Xu, M. Tomizuka, K. Keutzer, and D. Marculescu (2025) Looking backward: streaming video-to-video translation with feature banks. In Proc. ICLR, Cited by: §I. [23] K. Liu, W. Hu, J. Xu, Y. Shan, and S. Lu (2025) Rolling forcing: autoregressive long video diffusion in real time. arXiv preprint arXiv:2509.25161. Cited by: §I, §I-B, §IV-A, §IV-B, TABLE I. [24] Y. Liu, Y. Ren, X. Cun, A. Artola, Y. Liu, T. Zeng, R. H. Chan, and J.-M. Morel (2024) Redefining temporal modeling in video diffusion: the vectorized timestep approach. arXiv preprint arXiv:2410.03160. Cited by: §I-A. [25] W. S. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proc. ICCV, Cited by: §I. [26] A. Polyak, A. Zohar, A. Brown, A. Tjandra, A. Sinha, A. Lee, A. Vyas, A. Shi, B. Ma, C. Y. Chuang, et al. (2024) Movie Gen: a cast of media foundation models. arXiv preprint arXiv:2410.13720. Cited by: §I, §I-A. [27] S. Ren, S. Ma, X. Sun, and F. Wei (2025) Next block prediction: video generation via semi-auto-regressive modeling. arXiv preprint arXiv:2502.07737. Cited by: §I-A. [28] Sand-AI (2025) MAGI-1: autoregressive video generation at scale. Note: https://static.magi.world/static/files/MAGI_1.pdfAccessed: Jul. 13, 2026 Cited by: §I, §IV-B, TABLE I. [29] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. ICML, Cited by: §I. [30] K. Song, B. Chen, M. Simchowitz, Y. Du, R. Tedrake, and V. Sitzmann (2025) History-guided video diffusion. arXiv preprint arXiv:2502.06764. Cited by: §I. [31] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. In Proc. ICML, Cited by: §I-B. [32] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In Proc. ICLR, Cited by: §I. [33] D. Valevski, Y. Leviathan, M. Arar, and S. Fruchter (2025) Diffusion models are real-time game engines. In Proc. ICLR, Cited by: §I. [34] R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan (2022) Phenaki: variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399. Cited by: §I. [35] A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Cited by: Appendix B, Appendix D, §I, §I-A, §IV-A, §IV-B, §IV-B, TABLE I. [36] W. Wang and Y. Yang (2024) VidProM: a million-scale real prompt-gallery dataset for text-to-video diffusion models. In Proc. NeurIPS, Cited by: §IV-A. [37] Y. Wang, T. Xiong, D. Zhou, Z. Lin, Y. Zhao, B. Kang, J. Feng, and X. Liu (2024) Loong: generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757. Cited by: §I. [38] D. Weissenborn, O. Täckström, and J. Uszkoreit (2020) Scaling autoregressive video models. In Proc. ICLR, Cited by: §I. [39] D. Xie, Z. Xu, Y. Hong, H. Tan, D. Liu, F. Liu, A. Kaufman, and Y. Zhou (2025) Progressive autoregressive video diffusion models. In Proc. CVPR, p. 6322–6332. Cited by: §I-A. [40] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas (2021) VideoGPT: video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104.10157. Cited by: §I, §I-A. [41] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. (2024) Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: Appendix B. [42] M. Yang, Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel (2024) Learning interactive real-world simulators. In Proc. ICLR, Cited by: §I. [43] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2025) CogVideoX: text-to-video diffusion models with an expert transformer. In Proc. ICLR, Cited by: §I, §I-A. [44] T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park (2024) One-step diffusion with distribution matching distillation. In Proc. CVPR, Cited by: §I, §I-B. [45] T. Yin, Q. Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang (2025) From slow bidirectional to fast autoregressive video diffusion models. In Proc. CVPR, Cited by: Appendix D, §I, §I-B, §IV-A, §IV-B, §IV-B, TABLE I. [46] J. Yu, Y. Qin, X. Wang, P. Wan, D. Zhang, and X. Liu (2025) GameFactory: creating new games with generative interactive videos. arXiv preprint arXiv:2501.08325. Cited by: §I. [47] M. Zhou, H. Zheng, Z. Wang, M. Yin, and H. Huang (2024) Score identity distillation: exponentially fast distillation of pretrained diffusion models for one-step generation. In Proc. ICML, Cited by: §I-B. Supplementary Material for In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion Overview of the Appendices The appendices provide additional details and experimental results that complement the main paper. Appendix A describes the accompanying video demonstrations. Appendix B reports additional implementation details. Appendix C presents extended experimental analyses, including attention-map and denoising-procedure visualizations. Appendix D details the user study, covering both the A/B preference protocol and the comprehensive score evaluation. Appendix E discusses limitations and future directions. Finally, Appendix F provides additional qualitative results for short, 30-second, and 60-second generation. Figures and tables in the appendices continue the numbering of the main paper. Appendix A Video Demonstrations Supplementary Video Comparisons. While the static frames provided in the main paper and these appendices illustrate the visual quality of our generated samples, dynamic video demonstrations better convey temporal dynamics and motion smoothness. We therefore release side-by-side video comparisons between our In-Context Forcing and the Self Forcing baseline as part of the supplementary material accompanying this paper. Appendix B Additional Implementation Details We employ Wan2.1-T2V-14B as the teacher model and its causal variant Wan2.1-T2V-1.3B as the student model, with a classifier-free guidance (CFG) scale of 33. For optimization, we use AdamW with β1=0 _1=0, β2=0.999 _2=0.999, and a weight decay of 0.010.01. The learning rates for the student model GθG_θ and the fake score network sfakes_fake are set to 2×10−62× 10^-6 and 4×10−74× 10^-7, respectively, with an update ratio of 55 (i.e., the student model updates once every 55 updates of the fake score network). Training is conducted with a total batch size of 6464 for 1,9001,900 iterations, using the exponential moving average (EMA) with a decay rate of 0.990.99. Consistent with the evaluation protocol of the base Wan2.1 model [35] and Self Forcing [16], we evaluate on VBench [17] by rewriting the test prompts using Qwen2.5-7B-Instruct [41]. Appendix C Additional Experimental Results C-A Attention Map Visualization Implementation Details. We compare the contextual differences between Self Forcing and our In-Context Forcing through attention-map visualization in Fig. 8. The visualization focuses on the second chunk generation process, where key tokens 0 to 7878 correspond to the previous chunk and 7979 to 156156 correspond to the current chunk. Each chunk contains 46804680 tokens, and with a downsample rate of 6060, each chunk is represented by 7878 tokens in the visualization. Notably, the attention visualization presented in Fig. 7 is explicitly extracted from this configuration. Specifically, it crops the region spanning Key token indices 5252 to 7878 and Query token indices 105105 to 131131, which corresponds to the attention map between the 5th frame and the 2nd frame. All maps are generated under identical prompts, denoising steps, transformer blocks, and random seeds, with average pooling applied to downsample high-dimensional attention maps and averaging across all attention heads for clarity. Frame boundaries are marked with grid lines, with the horizontal axis representing key length and the vertical axis indicating query length. Figure 8: Attention Map Comparison between Self Forcing and In-Context Forcing. The visualization depicts attention patterns during the second chunk generation (with token indices ranging from 0 to 7878 for the previous chunk and 7979 to 156156 for the current chunk) under consistent prompts, transformer blocks, and random seeds. During early denoising at Step 0 (top row), Self Forcing rigidly concentrates attention on the exact spatial locations of previous frames. In contrast, In-Context Forcing effectively mitigates this over-reliance and prevents direct pattern replication by maintaining a more dispersed attention distribution with a stronger focus on the current frame. In late denoising at Step 3 (bottom row), both methods converge to a sharply concentrated diagonal, reflecting our approach’s adaptive transition from global semantic alignment to fine-grained detail refinement. Attention Maps in Early Denoising Stage. Distinct differences emerge in the early denoising stages: Self Forcing, which strictly relies on fully denoised contexts, demonstrates excessive pattern replication from previous frames. This over-reliance on local details is manifested as disproportionately high attention weights at exact corresponding spatial locations of preceding frames, which even surpass the attention allocated to the current frame. In contrast, our In-Context Forcing utilizes contexts with higher noise levels to provide adaptive guidance. This maintains a broader semantic receptive field during initial denoising, evidenced by a significantly more dispersed attention distribution. Rather than collapsing into sharp point-to-point mappings, the attention smoothly spans broader spatial neighborhoods while maintaining a stronger relative emphasis on the current frame. Attention Maps in Late Denoising Stage. As generation progresses to later denoising stages (e.g., Step 3), In-Context Forcing naturally incorporates contexts with lower noise levels. Consequently, its attention distribution transitions from the previously dispersed pattern to a sharply concentrated diagonal, adaptively converging to the localized attention patterns of Self Forcing. This structural convergence explicitly demonstrates our method’s capability to shift its focus toward fine-grained detail refinement in the final stages, perfectly complementing the robust global semantic alignment established during the early denoising phases. C-B Denoising Procedure Visualization In Figs. 9 and 10, we visualize the progressive denoising results of Self Forcing and our In-Context Forcing across different sampling steps (displaying the first 20 frames, sampled every 5 frames). The comparison reveals that Self Forcing, which strictly conditions on fully denoised contexts, forces an early restoration of fine-grained details. As shown in the top row of Fig. 9, the model achieves near-complete detail formation as early as the initial denoising step. While this appears to accelerate generation, it forces the model to directly replicate local details from preceding frames, severely compromising motion dynamics and leaving almost no capacity for meaningful refinement in subsequent steps. In contrast, In-Context Forcing leverages contexts with higher noise levels during the early stages. This provides adaptive guidance that prioritizes global semantic coherence and natural motion patterns over immediate detail synthesis. As generation progresses to later steps (subsequent rows in Fig. 10), the model smoothly transitions to fine-grained detail refinement. This behavior perfectly aligns with the fundamental coarse-to-fine hierarchy inherent in diffusion models, enabling continuous improvement in both visual fidelity and temporal dynamics throughout the entire generation process. Figure 9: Self Forcing Denoising Visualization. Results across sequential denoising steps (rows) for the first 20 frames sampled every 5 frames (columns). By strictly relying on fully denoised contexts, Self Forcing leads to an overly rapid detail restoration as early as the first denoising step. This over-reliance causes excessive local pattern replication, heavily compromising temporal dynamics and severely limiting the capacity for refinement in later stages. Figure 10: In-Context Forcing (Ours) Denoising Visualization. The multi-step generation (first 20 frames, sampled every 5 frames) across sequential denoising steps (rows) and frames (columns) explicitly demonstrates our method’s adaptive coarse-to-fine hierarchy. By utilizing contexts with higher noise levels in early stages, the model effectively prevents excessive replication of previous frames, thereby achieving superior semantic alignment and natural motion coherence. As generation progresses, it naturally transitions to fine-grained detail refinement, ensuring continuous visual improvement while maintaining robust inter-frame dynamics. Appendix D User Study User Preference Study. In the user preference study, we randomly selected extended prompts from VBench [17] and compared our method against three baselines: Wan2.1 [35], CausVid [45], and Self Forcing [16]. In each A/B test trial, participants evaluated video pairs randomly sampled from our method and each baseline. Results presented in Fig. 5 demonstrate a consistent user preference for our approach across all comparisons. Comprehensive Score Evaluation. For comprehensive score evaluation, each participant assessed randomly selected videos generated by our method and Self Forcing, providing ratings across three key dimensions: range of motion, visual quality, and semantic consistency. Fig. 11 illustrates the user study interface, where the left panel displays the test video and the right panel presents the corresponding evaluation dimensions and rating buttons. After completing the ratings for all dimensions, the participants can proceed to the next video using the navigation button below. Figure 11: User Study Interface. The interface presents the test video in the left panel and the evaluation controls, i.e., the three rating dimensions and the navigation button, in the right panel. Results Analysis. As detailed in Table V, the comprehensive score evaluation specifically compares our method against the primary baseline, Self Forcing. Our In-Context Forcing consistently achieves higher ratings across all dimensions. Most notably, the substantial improvement in the motion metric (2.732.73 versus 2.312.31) provides strong quantitative evidence that our progressive contextual modeling effectively prevents shortcut learning, resulting in vastly superior temporal dynamics. Furthermore, our approach yields clear enhancements in both global semantic alignment (2.552.55 versus 2.372.37) and overall visual quality (2.532.53 versus 2.492.49). TABLE V: Comprehensive Score Evaluation. Method Semantic Motion Quality Self Forcing 2.37 2.31 2.49 In-Context Forcing (Ours) 2.55 2.73 2.53 Appendix E Limitations and Future Directions Limitations. A primary limitation of our approach lies in an additional training convergence time. Because our method introduces progressive contexts with decreasing noise levels, the model requires more training iterations to fully adapt to these diverse contextual signals compared to the baseline. Specifically, under identical experimental settings, the Self Forcing baseline reaches optimal performance at 15001500 iterations, whereas our method requires 19001900 iterations. However, this computational overhead is strictly confined to the training phase. We consider this additional training duration a justified investment for the significantly enhanced semantic alignment, robust temporal dynamics, and highly efficient inference achieved during deployment. Future Directions. While this work primarily focuses on exploring the impact of progressive contexts in few-step autoregressive generation, our attention-map visualizations (Fig. 8) reveal an interesting phenomenon: the attention patterns remain inherently sparse even after autoregressive distillation. This persistent sparsity suggests significant potential for further structural optimization. Therefore, a highly promising direction for future research is to investigate the integration of attention sparsification techniques with few-step autoregressive models. Leveraging this inherent sparsity could substantially reduce the computational overhead and memory requirements for per-frame attention calculations, thereby amplifying the benefits of parallel denoising and enabling even greater scalability. Appendix F Additional Qualitative Results We present additional qualitative results to demonstrate the advantages of In-Context Forcing in visual quality and temporal coherence across varying video lengths. These results complement the comparisons already reported in the main paper, i.e., Fig. 4 for short clips and Fig. 6 for 30-second sequences. Specifically, Figs. 12 to 15 show short-video comparisons on VBench against the baselines, while Figs. 16 and 17 extend these comparisons to 30-second scenarios. Finally, Fig. 18 highlights our method’s scalability with a 60-second video. Across all cases, In-Context Forcing sustains richer motion and stronger semantic alignment with the prompt, whereas the baselines tend to replicate patterns from preceding frames and gradually lose temporal dynamics. Figure 12: Additional qualitative comparisons of short-video generation on VBench. Rows from top to bottom: CausVid, Wan2.1, Self Forcing, and In-Context Forcing (Ours). All visual results are generated through text-to-video (T2V) inference using the prompt: “A close-up of a baby crawling on a soft, carpeted floor. The baby is on all fours, with chubby arms and legs pushing and pulling themselves forward. They have a curious expression, with their head tilted upwards as they explore their surroundings. The baby is dressed in a bright, colorful onesie. The camera remains static, focusing solely on the baby’s movements and facial expressions, capturing each small step and moment of discovery.” Figure 13: Additional qualitative comparisons of short-video generation on VBench. Rows from top to bottom: CausVid, Wan2.1, Self Forcing, and In-Context Forcing (Ours). All visual results are generated through text-to-video (T2V) inference using the prompt: “A person is driving a modern sedan on a winding road surrounded by lush greenery. The driver is focused, with a neutral expression, and their hands are positioned at ten and two o’clock on the steering wheel. The car is moving smoothly along the curving path, and the scenery outside the window changes as the vehicle progresses. The background includes tall trees and rolling hills, creating a serene driving environment. The scene is captured from inside the car, providing a first-person perspective of the driver and the road ahead. Medium shot focusing on the driver and the immediate surroundings of the car.” Figure 14: Additional qualitative comparisons of short-video generation on VBench. Rows from top to bottom: CausVid, Wan2.1, Self Forcing, and In-Context Forcing (Ours). All visual results are generated through text-to-video (T2V) inference using the prompt: “A romantic close-up of two people kissing passionately. They are standing outdoors under a softly lit streetlamp at night. Both individuals have their eyes closed, leaning into each other with gentle expressions of affection. The man has short brown hair and is wearing a casual shirt, while the woman has shoulder-length blonde hair and is dressed in a flowy evening gown. Their arms are wrapped around each other, pulling them closer together. The background shows blurred city lights and buildings, adding to the intimate atmosphere. The scene captures the moment of deep connection between them, with a soft focus and warm lighting.” Figure 15: Additional qualitative comparisons of short-video generation on VBench. Rows from top to bottom: CausVid, Wan2.1, Self Forcing, and In-Context Forcing (Ours). All visual results are generated through text-to-video (T2V) inference using the prompt: “A middle-aged man with a casual outfit, including a t-shirt and jeans, is tasting a frothy beer from a pint glass. He has a mustache and is sitting at a wooden bar table with several empty glasses nearby. His face shows a thoughtful expression as he savors the taste, tilting his head slightly and closing his eyes. The bar has warm, ambient lighting and rustic decor, with wooden panels and dimly lit bottles of liquor on shelves behind him. Medium close-up shot focusing on his face and the beer glass.” Figure 16: Additional qualitative comparisons of 30-second long-video generation on VBench. All visual results are generated through text-to-video (T2V) inference using the prompt: “A majestic great white shark is swimming gracefully through the vast ocean in a watercolor painting style. The shark’s sleek body is painted with shades of grey and white, blending smoothly with the surrounding water. The water is depicted with soft blues and greens, showing gentle waves and sunlight filtering through, creating a serene underwater atmosphere. Schools of smaller fish swim alongside the shark, adding to the vibrant marine life. The background showcases a distant coral reef and floating seaweed, enhancing the sense of depth and life in the ocean. The painting captures the shark in a mid-swim pose, with its powerful tail propelling it forward. Medium shot, focusing on the shark and immediate surroundings.” Figure 17: Additional qualitative comparisons of 30-second long-video generation on VBench. All visual results are generated through text-to-video (T2V) inference using the prompt: “A close-up view of a ripe yellow banana and a black keyboard lying side by side on a wooden table. The banana is curved with a few spots, indicating it is just right for eating. The keyboard has a sleek design with white keys and black letters. The banana is positioned closer to the front of the frame, while the keyboard is slightly behind it, creating depth. The background is blurred, focusing attention on these two items. The banana appears fresh and inviting, contrasting with the utilitarian nature of the keyboard. Static scene, no camera movement.” Figure 18: Qualitative results of In-Context Forcing for 60-second long-video generation on VBench. The visual results are generated through text-to-video (T2V) inference using the prompt: “A cute happy Corgi playing in park, sunset.”

Instant research discovery