Paper deep dive

Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video

Chang-Bae Bang, Hyungjin Chung, Byung-Hoon Kim

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 84

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 10/4/2026, 3:12:04 AM

Summary

This study investigates the alignment between internal representations of video diffusion models and the human visual cortex. It hypothesizes that representations used for generating future video frames align better with the predictive nature of human visual processing than representations of observed video. Using fMRI data from the BOLD Moments Dataset, the authors compare an autoregressive (AR) video diffusion model (Helios) and its non-AR base model (Wan-2.1). Results show that future video generation representations align more strongly with the visual cortex, particularly in higher-order regions, compared to observed video or reconstruction representations. Behavioral experiments confirm that humans prefer videos generated by amplifying layers that align better with the visual cortex.

Entities (8)

BOLD Moments Dataset · dataset · 99%Visual Cortex · biological-structure · 99%Helios · model · 98%Wan 2.1 · model · 98%fMRI · method · 97%Autoregressive Model · model-type · 95%CausVid · model · 92%Self-Forcing · model · 92%

Relation Signals (8)

BOLD Moments Dataset → usedin → fMRI Study

confidence 99% · We use the BOLD Moments Dataset (Lahner et al., 2024), in which ten subjects watched 1,102 3 s naturalistic videos while whole-brain fMRI was recorded

Wan 2.1 → isbasemodelof → Helios

confidence 98% · helios-base-v2v ... fine-tuned from the non-AR base model Wan-2.1 T2V-14B

Helios → istypeof → Autoregressive Model

confidence 98% · The main analysis uses helios-base-v2v (Yuan et al., 2026a) as the AR model

Future Video Generation → alignsbetterwith → Visual Cortex

confidence 95% · we show that the representations for future video generation align better with the visual cortex than the representations of the observed video

Future Video Generation → alignswith → Higher-order Visual Cortex

confidence 93% · that of future video generation is concentrated in higher-order visual cortex

Observed Video Reconstruction → alignswith → Lower-order Visual Cortex

confidence 93% · the alignment of observed video reconstruction is concentrated in lower-order visual cortex

CausVid → istypeof → Autoregressive Model

confidence 92% · CausVid (Yin et al., 2024), both 30-layer autoregressive video models

Self-Forcing → istypeof →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Studying the alignment between the internal representations of vision models and the responses of the visual cortex to the same observed visual stimuli has enabled us to better understand human visual processing. However, studies so far have largely overlooked the fact that the human brain not only processes observed visual stimuli, but also predicts upcoming stimuli based on what has been observed. Accordingly, we hypothesize that internal representations for generating future video frames are better aligned with the predictive nature of human visual processing than representations of the observed video itself. To this end, we compare the alignment between human video-watching fMRI responses in the visual cortex and the internal representations from two types of video diffusion models, an autoregressive (AR) model and its non-AR base model. We first conduct a within-model analysis of the AR video diffusion model and show that the representations for future video generation align better with the visual cortex than the representations of the observed video. We then compare the internal representations of the AR model with those of its non-AR base model and again show that the representations for future video generation align better with the visual cortex than the representations for observed video reconstruction by the base model. Specifically, the alignment of observed video reconstruction is concentrated in lower-order visual cortex, whereas that of future video generation is concentrated in higher-order visual cortex. Finally, we show in a human behavioral experiment that humans prefer videos generated by amplifying the contributions of individual layers that align better with the visual cortex.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

80,336 characters extracted from source content.

Expand or collapse full text

Preprint. Under review. FUTURE VIDEO GENERATION BETTER ALIGNS WITH THE HUMAN VISUAL CORTEX THAN OBSERVED VIDEO Chang-Bae Bang 1,2,3 Hyungjin Chung 4† Byung-Hoon Kim 1,2,3,5† 1 Department of Psychiatry, Yonsei University College of Medicine 2 Institute of Behavioral Sciences in Medicine, Yonsei University College of Medicine 3 Department of Biomedical Systems Informatics, Yonsei University College of Medicine 4 Department of Computer Science & Engineering, Korea University 5 Yonsei Institute for Digital Healthcare, Yonsei University changbae.bang@yonsei.ac.kr, hj chung@korea.ac.kr, egyptdj@yonsei.ac.kr ABSTRACT Studying the alignment between the internal representations of vision models and the responses of the visual cortex to the same observed visual stimuli has enabled us to better understand human visual processing. However, studies so far have largely overlooked the fact that the human brain not only processes observed vi- sual stimuli, but also predicts upcoming stimuli based on what has been observed. Accordingly, we hypothesize that internal representations for generating future video frames are better aligned with the predictive nature of human visual pro- cessing than representations of the observed video itself. To this end, we compare the alignment between human video-watching fMRI responses in the visual cor- tex and the internal representations from two types of video diffusion models, an autoregressive (AR) model and its non-AR base model. We first conduct a within- model analysis of the AR video diffusion model and show that the representations for future video generation align better with the visual cortex than the representa- tions of the observed video. We then compare the internal representations of the AR model with those of its non-AR base model and again show that the repre- sentations for future video generation align better with the visual cortex than the representations for observed video reconstruction by the base model. Specifically, the alignment of observed video reconstruction is concentrated in lower-order vi- sual cortex, whereas that of future video generation is concentrated in higher-order visual cortex. Finally, we show in a human behavioral experiment that humans prefer videos generated by amplifying the contributions of individual layers that align better with the visual cortex. 1INTRODUCTION Studying the representational alignment between vision models and the human brain has enabled us to better understand human visual processing (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014; Kriegeskorte & Douglas, 2018). For example, by examining which features or layers of the model best explain neural responses, we can infer how visual information may be processed in the human brain (Kriegeskorte et al., 2008; Naselaris et al., 2011; Schrimpf et al., 2018). While these brain–vision model representation alignment studies have provided insights into human visual processing, studies to date have largely focused on representations of the observed visual stimu- lus (Conwell et al., 2024; Sartzetaki et al., 2025; H ̈ ofling et al., 2026). The brain, however, does not only passively respond to the observed stimulus, but also predicts the stimulus that is about to † Co-corresponding authors. 1 arXiv:2609.38819v1 [cs.CV] 30 Sep 2026 Preprint. Under review. 1s2s3s Video Watching Future Generation Observed + Future Observed Reconstruction Base Model Observed Video Noised Video AR Model InOut OutIn 4s 1s2s3s 1s2s3s 1s2s3s Which Representation Aligns Better with the Visual Cortex? Future Generation > Observed Video 0.0 0.1 0.2 0.3 R² / NC Unique O bserved Video Shared Unique F uture G eneration 00.07 Unique Observed Reconstruction 00.09 Unique Future Generation R² / NC Observed Reconstruction → Lower Cortex Future Generation → Higher Cortex Neural Alignment to Behavioral Validation Neural Alignment Human Preference AB LayersLayers Study Design Main FindingsMain Findings (a) (b) Figure 1: Study Design and Main Findings. (a) We map the representations of the observed video, future video generation, and observed video reconstruction onto the visual cortex. (b) Left, future video generation aligns better with the visual cortex than the observed video. Middle, observed video reconstruction aligns better with lower-order visual cortex, whereas future video generation aligns better with higher-order visual cortex. Right, humans prefer videos generated by amplifying the layers that align better with the visual cortex. arrive (Rao & Ballard, 1999; Keller & Mrsic-Flogel, 2018; Hogendoorn & Burkitt, 2018). Accord- ingly, we compare the internal representations of a model that generates future video frames with those of the human visual cortex. Recently, video generative models based on the diffusion transformer (Peebles & Xie, 2022) have demonstrated the capability to produce photorealistic videos (Brooks et al., 2024; Team Wan et al., 2025). Conventionally, video diffusion models generate video sequences by progressively denois- ing latent representations across all frames simultaneously. However, such designs disregard the inherent temporal order of causality. To align with the causal order of time in which visual stimuli naturally unfold, recent approaches have integrated the autoregressive (AR) framework with diffu- sion models to generate the future video conditioned on the observed video (Yin et al., 2024; Huang et al., 2025; Yuan et al., 2026a). The AR model represents how the visual stimulus unfolds over time to generate a future video that is semantically and physically plausible based on the observed video. We hypothesize that the internal representations for generating future video frames are better aligned with the predictive nature of human visual processing than the representations of the observed video itself. To this end, we compare the internal representations from two types of video diffusion models, an AR model and its non-AR base model. Using human fMRI responses recorded while subjects watch 3 s naturalistic videos (Lahner et al., 2024), we map the representations of the observed video stimulus, future video generation, and observed video reconstruction (Figure 1a). Moreover, we test whether the alignment of the representation for future video generation is validated at the behavioral level with a human video preference experiment. 2 Preprint. Under review. Contributions (Figure 1b). • We show that future video generation aligns better with the visual cortex than the observed video. The representation for future video generation aligns better than the rep- resentation of the observed video that both the AR model and the human brain receive. • We show that the two representations, for future video generation and observed video reconstruction, align with different levels of the visual hierarchy. The alignment of observed video reconstruction is concentrated in lower-order visual cortex, whereas that of future video generation is concentrated in higher-order visual cortex. • We validate the layer profile of neural alignment in human behavior. Amplifying the contribution of an individual layer during future video generation, we show that humans prefer videos generated by amplifying the layers that align better with the visual cortex. 2RELATED WORK 2.1BRAIN–VISION MODEL REPRESENTATION ALIGNMENT The internal representations of vision models turn out to partly resemble the representations of the human visual cortex (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014; Schrimpf et al., 2018). Presenting the same visual stimulus to a human and to a vision model, and compar- ing their internal representations, enables us to understand human visual processing. The internal representations of vision models optimized for object recognition align with the representation in the visual cortex (Yamins et al., 2014), and deeper layers align best with higher-order visual cor- tex (G ̈ uc ̧l ̈ u & van Gerven, 2015). Large-scale recordings of human cortical responses to natural images and videos (Allen et al., 2022; Lahner et al., 2024) have enabled systematic comparisons across a large pool of vision models. These comparisons show that, for images, the visual training diet exerts a larger and more consistent effect on representation alignment than architecture or task objective (Conwell et al., 2024). For videos, temporal modeling drives representation alignment to lower-order visual cortex while the classification task drives alignment to higher-order visual cor- tex (Sartzetaki et al., 2025). The comparison has also been extended to the internal representations of an image generative model. The U-Net bottleneck of a latent image diffusion model aligns with the visual cortex best at the noisiest denoising step, and as denoising proceeds the first layer aligns better with lower-order visual cortex while the bottleneck aligns better with higher-order visual cor- tex (Takagi & Nishimoto, 2023). 2.2PREDICTIVE NATURE OF HUMAN VISUAL PROCESSING The human visual cortex represents not only the response to the observed stimulus but also the future stimulus. According to predictive coding theory, the brain uses its prior experience to build an internal generative model of the external world (Rao & Ballard, 1999; Lee, 2015; Keller & Mrsic- Flogel, 2018). With this internal model, the brain can infer the cause of the observed visual stimuli and predict future stimuli. Indeed, the representation of future visual stimuli has been confirmed in human neuroimaging studies with visual tasks. The human brain represents the next position of a moving object before its input arrives (Hogendoorn & Burkitt, 2018; Blom et al., 2020), and when presented with a partial cue, even lower-order visual cortex represents the full sequence of previously experienced visual stimuli (Ekman et al., 2017). Higher-order visual cortex is thought to represent visual stimuli further into the future than lower-order visual cortex while humans observe naturalistic video (de Vries & Wurm, 2023; Lee et al., 2021). 2.3INTERNAL REPRESENTATION OF DIFFUSION VIDEO MODELS A non-AR diffusion video model, which denoises all frames of the video together, learns an internal representation that carries rich visual information (Xiang et al., 2023; Li et al., 2023; Yu et al., 2025; Geyer et al., 2024; Nam et al., 2025). This representation is rich enough to be adapted to vision tasks on the observed video, such as action recognition (Guimar ̃ aes et al., 2025), depth estimation (King et al., 2026), point tracking (Son et al., 2025; Nam et al., 2025; Yuan et al., 2026b), and physical plausibility estimation (Esmati et al., 2026). In contrast to the non-AR model, an AR diffusion 3 Preprint. Under review. video model sequentially generates the future video based on the observed video, allowing video generation to follow the natural temporal order. The AR model represents how the future visual stimulus unfolds over time to generate a future video that is semantically and physically plausible. Indeed, aligning the representation of an AR diffusion video model to the representation of the future video improves its future video generation quality (Yu et al., 2026). 3PRELIMINARIES Human Video fMRI Dataset. We use the BOLD Moments Dataset (Lahner et al., 2024), in which ten subjects watched 1,102 3 s naturalistic videos while whole-brain fMRI was recorded, 1,000 of them for training and 102 for testing. Preprocessing follows the MOSAIC pipeline (Lahner et al., 2025) and leaves one response per video at each cortical vertex (Appendix A.4), Y ∈R N×P , m∈1,..., 10, p∈1,...,P, r ⊆1,...,P,(1) where Y is the response of one subject over N videos and P vertices, m indexes subjects, p indexes vertices, r is a region composed of a group of vertices, and|r| is the number of vertices it contains. Diffusion Video Generative Model. A diffusion video generative model generates a video by iteratively denoising noisy video tokens. A variational autoencoderE maps a video v to a latent z 0 = E (v)∈R C×T×H×W , and generation follows a flow-matching path between noise and data (Lipman et al., 2023). A latent on this path at noise level σ interpolates the clean latent z 0 with Gaussian noise, z σ = (1− σ)z 0 + σ ε, ε∼N (0,I).(2) We order the schedule σ 0 > σ 1 >· > σ S−1 , so that step s = 0 is the noisiest and step s = S− 1 the cleanest. The transformer has B layers, numbered b = 0,...,B− 1. Let H s,b denote the hidden states that layer b produces at noise level σ s , and let H in s denote the sequence that enters the first layer. 4METHODS AND EXPERIMENT SETUP Human Brain Data. Every vertex is labeled with the HCP-MMP atlas (Glasser et al., 2016). Our primary focus is the Visual System, 7,894 vertices grouped into five divisions. The five divisions are Primary Visual, Early Visual, Dorsal Stream, Ventral Stream, and MT+ Complex, and a region r is either one division or the Visual System as a whole. Only vertices with a noise ceiling NC > 10% enter the analysis. Models. The main analysis uses helios-base-v2v (Yuan et al., 2026a) as the AR model, a 40-layer autoregressive video model fine-tuned from the non-AR base model Wan-2.1 T2V-14B (Team Wan et al., 2025). We replicate the analysis on two further AR models, Self-Forcing (Huang et al., 2025) and CausVid (Yin et al., 2024), both 30-layer autoregressive video models fine-tuned from the non- AR base model Wan-2.1 T2V-1.3B. Video Preparation. Every BOLD Moments Dataset video is 3 s long and square. For helios-base- v2v and Wan-2.1 T2V-14B we resample each clip to 272× 272 at 33 fps. For Self-Forcing and CausVid we fit it inside an 832× 480 frame, keeping the aspect ratio and padding the sides with black, and sample it at 15 fps. Autoregressive Model. We use an AR model built on a diffusion video generative model that generates future video frames based on observed video stimuli. The AR model generates the future video by denoising future video tokens. Let z obs denote the clean latent of the observed video and z fut σ s the latent of the future video at noise level σ s . The input sequence for every step carries both observed and future tokens at once, H in s = patch(z obs ) | z observed video tokens, clean ;patch(z fut σ s ) | z future video tokens at noise level σ s ,(3) with the observed tokens fed in clean at every step and the future tokens carried over from the previous step as they are denoised. Let c obs = patch(z obs ) denote the observed video embedding, 4 Preprint. Under review. which is the same at every step. The layer activation of the AR model at step s and layer b splits into an observed part and a future part, H ar s,b = H obs s,b ; H fut s,b .(4) We extract future tokens H fut s,b , so that each video yields a grid of representations indexed by the denoising step s = 0,...,S− 1 and the layer b = 0,...,B− 1. Future Video Generation with AR Model. For each AR model we feed the 3 s observed video and let the model generate the 1 s future video that follows. Because subjects only observed the 3 s video with no instruction, we leave the text prompt empty. The AR model helios-base-v2v runs 50 denoising steps, and Self-Forcing and CausVid run 4 steps, the default for each model. For each model we extract c obs once, and H fut s,b at every step and layer. Base Model. We use a non-AR base model built on a diffusion generative model that reconstructs noised video stimuli. The base model reconstructs the observed video by denoising observed video tokens. We noise z obs by Equation 2 and run one forward pass at each of the noise levels σ s , so that at every noise level the base model reconstructs the observed video from its noised form. Let H rec s,b denote the layer activation of the base model at step s and layer b. We extract H rec s,b , which gives a grid over noise levels and layers of the same shape as H fut s,b . Observed Video Reconstruction with Base Model. For each base model we feed the 3 s observed video noised at the same schedule as its AR model. At each noise level we run one forward pass to denoise the observed video with no text prompt. For each model we extract H rec s,b at every step and layer. Vertex-Wise Encoding Model. We map three representations onto the cortical response recorded while subjects observed the videos (Figure 1a). The first is the observed video embedding c obs that the AR model receives (Equation 3). The second is the representation for future video generation H fut s,b of the AR model (Equation 4). The third is the representation for observed video reconstruction H rec s,b of the base model. Each of them is flattened and reduced to d = 6004 dimensions with a sparse random projection (Appendix A.5), which gives the feature matrices X obs , X fut s,b , and X rec s,b , each in R N×d with one row per video. For each representation we fit a vertex-wise encoding model that predicts the response of one cortical vertex (Naselaris et al., 2011). Let X denote any one of these feature matrices. For vertex p we fit ridge regression weights, w p = arg min w∈R d Y :,p − Xw 2 2 + λ p ∥w∥ 2 2 ,(5) with a ridge penalty λ p chosen for that vertex, and take ˆ Y :,p = Xw p as the predicted response of that vertex. The penalty is chosen by nested cross-validation on the training videos, following the stacked-regression scheme of Lin et al. (2024), and for the two grid representations, H fut s,b and H rec s,b , the best step and layer are selected on the training folds (Appendix A.5). The test videos take no part in fitting or in this selection, and every result we report is measured on test videos. Encoding Accuracy. Encoding accuracy at a vertex is the coefficient of determination R 2 p of the test-set predictions. Every vertex is assigned a noise ceiling NC p , the upper bound on how well any model can predict its response given the noise in the fMRI response (Appendix A.4). We report the noise-normalized encoding accuracy ̃ R 2 p = R 2 p /NC p .(6) The encoding accuracy of a region r is its mean over the vertices of the region, ̃ R 2 r = 1 |r| P p∈r ̃ R 2 p . Variance Partitioning. Two representations can explain the same part of the cortical response. To separate what each explains on its own from what the two explain in common, we fit a joint encoding model that combines representations a and b and measure the unique contribution of each 5 Preprint. Under review. 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / NC (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Layers S0 L11 Figure 2: Within-model analysis. (a) The AR model receives the 3 s observed video and generates the 1 s future video. (b) Encoding accuracy of the representation for future video generation of the AR model over denoising steps and layers in the Visual System. (c) Unique and shared contributions of the representation for future video generation against the observed video in the Visual System and its five divisions. (d) The same contributions plotted on the cortical surface. representation. Let ̃ R 2 stack be the encoding accuracy of the joint model, and ̃ R 2 a and ̃ R 2 b the encoding accuracies of a and b alone (Lin et al., 2024), U a = ̃ R 2 stack − ̃ R 2 b , U b = ̃ R 2 stack − ̃ R 2 a , U ab = ̃ R 2 a + ̃ R 2 b − ̃ R 2 stack .(7) U a and U b are the unique contributions of the two representations and U ab is their shared contribu- tion. The joint model is explained in Appendix A.6. We partition two pairs. The within-model pair sets the observed video against future video generation, (X obs ,X fut ). The cross-model pair sets observed video reconstruction against future video generation, (X rec ,X fut ). Here X fut and X rec are the grid cells X fut s,b and X rec s,b at the best-aligned step and layer, selected on the training folds for each region and subject (Appendix A.5, Equation 12). 5RESULTS 5.1WITHIN-MODEL ANALYSIS The AR model receives the 3 s observed video and generates the 1 s future video. We map both the representation of the observed video X obs and the representation for future video generation X fut s,b onto the visual cortex, and then use variance partitioning to measure the unique contribution of each representation. We report Helios here and repeat the same analysis on CausVid and Self-Forcing in Appendix A.1. Future video generation aligns better with the visual cortex than the observed video. The representation for future video generation aligns best with the Visual System at step 0 and layer 11, the cell selected on the training folds (Figure 2b). In the Visual System the unique contribution of future video generation is 0.222, the shared contribution is 0.111, and the unique contribution of the observed video is −0.006 (Figure 2c). The unique contribution of future video generation is above the unique contribution of the observed video in every division (p = 0.002, Appendix A.9). Only in Primary Visual does the shared contribution, 0.151, exceed the unique contribution of future video generation, 0.089. In every other division the unique contribution of future video generation exceeds both the unique contribution of the observed video and the shared contribution. The shared contribution and the unique contribution of future video generation are 0.142 and 0.175 in Early 6 Preprint. Under review. Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / NC (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best step 0.0 0.1 0.2 0.3 R² / NC AR Model (S0) Base Model (S16) 02449 Steps @ best layer 0.0 0.1 0.2 0.3 R² / NC AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Early Dorsal Ventral MT+ AR Model Base Model (e) Figure 3: Cross-model analysis. (a) The AR model generates the 1 s future video and its base model reconstructs the 3 s noised observed video. (b) Encoding accuracy of the representation for observed video reconstruction of the base model over denoising steps and layers in the Visual System. (c) Unique and shared contributions of future video generation against observed video reconstruction in the Visual System and its five divisions. (d) The same contributions plotted on the cortical sur- face. (e) Encoding accuracy of the representations for future video generation and observed video reconstruction in the five divisions. (f, g) Encoding accuracy of the two representations along the step axis at each representation’s own selected layer and along the layer axis at each representation’s own selected step. Visual, 0.072 and 0.227 in Dorsal Stream, 0.083 and 0.314 in Ventral Stream, and 0.087 and 0.350 in MT+ Complex. On the cortical surface the unique contribution of future video generation grows continuously toward higher-order visual cortex, whereas the shared contribution and the unique contribution of the observed video are concentrated in lower-order visual cortex (Figure 2d). 5.2CROSS-MODEL ANALYSIS The base model reconstructs the 3 s observed video from its noised form. We map the representation for observed video reconstruction X rec s,b onto the visual cortex, and then partition the variance with the representation for future video generation X fut s,b (Figure 3a). We report Helios here and repeat the same analysis on CausVid and Self-Forcing in Appendix A.1. Future video generation aligns better with the visual cortex than observed video reconstruc- tion. The representation for observed video reconstruction peaks at step 16 and layer 8, the cell selected on the training folds (Figure 3b). Its best encoding accuracy, 0.301, is below the best en- coding accuracy of future video generation, 0.334. In Primary Visual and Early Visual the encoding accuracies of future video generation and observed video reconstruction are similar, 0.240 and 0.240 7 Preprint. Under review. (a) Encoding Accuracy AB Layers Preference Testing No STG STG @ L0 STG @ L32 STG @ L38 ... (b) 01939 Layers 0.0 0.1 0.2 0.3 R² / NC AR Model (S0) Human preference −1 0 1 BT log-worth (c) 3D Self-Attention Q K V A Skipped Self-Attention (STG-A) Q K V I x t ε cond ε CFG ε perturb + w − w ε x t−1 0.10.20.3 R² / NC −1 0 1 BT log-worth L0 L5 L11 L16 L22 L28 L32 L38 r = 0 . 7 5 (d) STG Schematic Flow Figure 4: Layer-wise guidance and human preference. (a) STG skips the self-attention of one layer and pushes the sample away from the perturbed prediction (Equation 15). The schematic follows the layout of Ahn et al. (2024). (b) Pairwise preference test among the nine arms, the unguided arm (No STG) and eight STG-guided arms. (c) Encoding accuracy of the AR model at step 0 in the Visual System and human preference as Bradley–Terry log-worth along the layer axis. (d) Encoding accuracy against human preference, with the Pearson r. in Primary Visual and 0.317 and 0.310 in Early Visual (Figure 3e). However, future video generation aligns better in higher-order visual cortex. In Dorsal Stream future video generation is at 0.299 and observed video reconstruction at 0.272, in Ventral Stream 0.396 and 0.329, and in MT+ Complex 0.436 and 0.363 (p = 0.002 in all three). Observed video reconstruction aligns with lower-order visual cortex, and future video gener- ation aligns with higher-order visual cortex. In Primary Visual the two unique contributions are 0.023 for observed video reconstruction and 0.022 for future video generation, in Early Visual 0.027 and 0.034, in Dorsal Stream 0.015 and 0.042, in Ventral Stream 0.004 and 0.071, and in MT+ Com- plex 0.006 and 0.080 (Figure 3c). On the cortical surface the unique contribution of future video generation grows continuously toward higher-order visual cortex, whereas the unique contribution of observed video reconstruction is concentrated in lower-order visual cortex (Figure 3d). Alignment of observed video reconstruction peaks at a less noisy step than future video gener- ation. The alignment profiles of observed video reconstruction and future video generation differ along the step axis (Figure 3f). The alignment of future video generation is highest at the noisiest step and decreases monotonically as denoising proceeds, whereas the alignment of observed video reconstruction peaks early in the noise schedule and declines after it. Along the layer axis the align- ment of both representations follows an inverted U with the maximum at an early-to-middle layer (Figure 3g). 5.3NEURAL ALIGNMENT TO BEHAVIORAL VALIDATION We finally ask whether the neural alignment of future video generation extends to human behavior. Spatiotemporal skip guidance (STG; Hyung et al., 2025) amplifies the contribution of a layer during video generation (Figure 4a, Appendix A.7). We test whether the neural alignment of future video generation is validated by human preference for videos generated under layer-wise amplification with STG. We apply STG to one layer b ∈ 0, 5, 11, 16, 22, 28, 32, 38 at a time while the AR model generates a 5 s future video conditioned on the observed 3 s video. We then run A/B tests on the combined 8 s videos with five human observers, 720 preference trials in total (Figure 4b, Appendix A.8). Humans prefer videos generated by amplifying well-aligned layers. The neural alignment of future video generation and human preference along the layer axis both follow an inverted U (Fig- 8 Preprint. Under review. ure 4c). Among the eight guided layers, encoding accuracy at step 0 is highest at layer 11 and preference at layer 16. Over the eight layers the two measures correlate at r = 0.75 (p = 0.029, Figure 4d). 6DISCUSSION Future video generation aligns better than the observed video. The representation for future video generation aligns better with the visual cortex than the observed video itself. Moreover, the observed tokens themselves add no unique contribution beyond future video generation in any divi- sion (Appendix A.2), which further confirms that future video generation is the key to brain align- ment. This may be indirect evidence that the visual cortex performs future-oriented computation, especially in higher-order visual cortex, where human experiments have also shown predictive sig- nals (Lee et al., 2021; de Vries & Wurm, 2023; Tarder-Stoll et al., 2024). Why the noisiest step and an early-to-middle layer. The representation for future video gener- ation aligns best at the noisiest step and an early-to-middle layer. As denoising goes on, the future video settles into one of the many futures that could follow. Interestingly, this uncommitted state aligned best with the visual cortex. This finding is in line with Bayesian theories of neural coding, where the activity of a neural population encodes the whole range of possible stimuli rather than a single-valued prediction (Ma et al., 2006; van Bergen & Jehee, 2019). On the other hand, the rep- resentation for observed video reconstruction aligns best at a less noisy step. Moreover, the unique contribution of observed video reconstruction is concentrated in lower-order visual cortex, which processes simple visual features such as edges, contours, and textures (Rolls, 2024). As such fine details are refined in the later stages of the denoising schedule (Wang & Vastola, 2023), these results suggest that, compared with future video generation, observed video reconstruction focuses more on refining such fine details than on grasping the global semantics of the video. Along the layer axis, prior work reports that the visual representation is rich in the intermediate layers of diffusion transformers (Nam et al., 2025; Yuan et al., 2026b; Yu et al., 2025) and poor in the first and last layers (Yuan et al., 2026b), which agrees with our finding. Comparison with a wide range of vision models. Previous studies compared the internal repre- sentations of vision models of various architectures, training datasets, and objectives to understand human visual processing (Conwell et al., 2024; Sartzetaki et al., 2025). We also compared the internal representations of a wide range of vision models with the representation for future video generation (Appendix A.3). The internal representations of self-supervised video models aligned best with the visual cortex, as reported before (H ̈ ofling et al., 2026; Tang et al., 2025). However, the gap in brain alignment between the representation for future video generation and the repre- sentations of self-supervised video models narrowed toward higher-order visual cortex. This may be further indirect evidence that higher-order visual cortex implements a mechanism that predicts future visual stimuli. From neural alignment to human preference. STG guides the video generation path away from a perturbed weak model in which the self-attention of one layer is skipped (Hyung et al., 2025). Applying STG to a layer amplifies the representation of that layer. In our experiment, humans preferred the future videos generated by amplifying the layers that aligned better with the visual cortex. A layer that aligns better with the visual cortex may hold a richer visual representation, and amplifying it may make the generated video more natural to a human. This is in line with previous studies showing that, for diffusion-based generative models, aligning the internal representation to a richer visual representation improves the quality of the generated visual contents (Yu et al., 2025; Zhang et al., 2025). Limitations. First, our analysis framework, which feeds a 3 s observed video and generates the 1 s future video that follows, limits understanding of how the brain represents the past and the future over a longer context and a longer horizon (Hasson et al., 2008; Lee et al., 2021; Brunec & Momen- nejad, 2022; Tarder-Stoll et al., 2024). Second, the cortical response evolves over time while the subject watches the video (de Vries & Wurm, 2023), but we map one response per video onto the visual cortex and do not compare the time course of the response with the internal representation of the AR model. Finally, our behavioral experiment shows that the layer profile of neural alignment 9 Preprint. Under review. is reflected in human preference, but it does not provide a causal account of how neural alignment gives rise to preference. 7CONCLUSION In this study, we examined the representational alignment of an AR and a non-AR video generative model with the visual cortex. We tested which of the three representations, the observed video, future video generation, and observed video reconstruction, aligns best with the visual cortex. With vertex-wise encoding and variance partitioning, we showed that future video generation aligns better than both the observed video and its reconstruction. More specifically, the alignment advantage of the representation for future video generation grows along the visual hierarchy. In contrast, the unique contributions of the observed video and its reconstruction are concentrated in lower-order visual cortex. Finally, by amplifying one layer during future video generation, we showed that humans preferred the videos generated from the layers that align well with the visual cortex. AI USE STATEMENT In this work, we used generative AI tools to provide feedback on the research methodology and experiments, to implement methods, to assist with translation, and to interpret results. We have not used generative AI tools to generate synthetic datasets, to develop theoretical models or conceptual frameworks, to formulate mathematical claims, to provide ingredients for or write mathematical proofs, to propose or refine hypotheses, to clean or reformat datasets, or to support qualitative or thematic data analysis. Additionally, we used generative AI tools to create and edit software code, to draft parts of the paper, to summarize existing literature, to identify relevant literature and search for information, for brainstorming, to edit the paper to improve readability, and to format references. We have reviewed all AI-assisted work. LLM-generated code was verified and tested by the authors, and all AI-assisted text, translations, and references were checked and revised by the authors. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. ETHICS STATEMENT The fMRI data are from the publicly released BOLD Moments Dataset and are used in their anonymized form. The human preference experiment was a video-watching task with no risk to the observers, who were adult volunteers and gave informed consent, and no identifying informa- tion about them was collected or reported. REFERENCES Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jung, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In European Conference on Computer Vision (ECCV), 2024. doi: 10.48550/arXiv.2403.17377. Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Nase- laris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25:116–126, 2022. doi: 10.1038/s41593-021-00962-x. Tessel Blom, Daniel Feuerriegel, Philippa Johnson, Stefan Bode, and Hinze Hogendoorn. Predic- tions drive neural representations of visual events ahead of incoming sensory information. Pro- ceedings of the National Academy of Sciences, 117(13):7510–7515, 2020. doi: 10.1073/pnas. 1917777117. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029. Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video 10 Preprint. Under review. generation models as world simulators.OpenAI Technical Report, 2024.URL https: //openai.com/index/video-generation-models-as-world-simulators/. Iva K. Brunec and Ida Momennejad. Predictive representations in hippocampal and prefrontal hierar- chies. Journal of Neuroscience, 42(2):299–312, 2022. doi: 10.1523/JNEUROSCI.1327-21.2021. Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. A large- scale examination of inductive biases shaping high-level visual representation in brains and ma- chines. Nature Communications, 15(1):9383, 2024. doi: 10.1038/s41467-024-53147-y. Ingmar E. J. de Vries and Moritz F. Wurm. Predictive neural representations of naturalistic dynamic input. Nature Communications, 14:3858, 2023. doi: 10.1038/s41467-023-39355-y. Matthias Ekman, Peter Kok, and Floris P. de Lange. Time-compressed preplay of anticipated events in human primary visual cortex. Nature Communications, 8:15276, 2017. doi: 10.1038/ ncomms15276. Parsa Esmati, Somjit Nath, Katja Hofmann, Derek Nowrouzezahrai, Samira Ebrahimi Kahou, and Majid Mirmehdi. The invisible hand of physics: When video diffusion models know more than they show. arXiv preprint arXiv:2606.05328, 2026. Oscar Esteban, Christopher J. Markiewicz, Ross W. Blair, Craig A. Moodie, A. Ilkay Isik, Asier Erramuzpe, James D. Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, Hiroyuki Oya, Satrajit S. Ghosh, Jessey Wright, Joke Durnez, Russell A. Poldrack, and Krzysztof J. Gor- golewski. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nature Methods, 16 (1):111–116, 2019. doi: 10.1038/s41592-018-0235-4. Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In International Conference on Learning Representations (ICLR), 2024. doi: 10.48550/arXiv.2307.10373. Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smith, and David C. Van Essen. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171–178, 2016. doi: 10.1038/nature18933. Umut G ̈ uc ̧l ̈ u and Marcel A. J. van Gerven. Deep neural networks reveal a gradient in the com- plexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27): 10005–10014, 2015. doi: 10.1523/JNEUROSCI.5023-14.2015. Rog ́ erio Guimar ̃ aes, Frank Xiao, Pietro Perona, and Markus Marks. Diffusion-based action recog- nition generalizes to untrained domains. arXiv preprint arXiv:2509.08908, 2025. Uri Hasson, Eunice Yang, Ignacio Vallines, David J. Heeger, and Nava Rubin. A hierarchy of temporal receptive windows in human cortex. Journal of Neuroscience, 28(10):2539–2550, 2008. doi: 10.1523/JNEUROSCI.5487-07.2008. Larissa H ̈ ofling, Matthias Tangemann, Lotta Piefke, Susanne Keller, Katrin Franke, and Matthias Bethge. Only brains align with brains: Cross-region alignment patterns expose limits of normative models. In International Conference on Learning Representations (ICLR), 2026. doi: 10.48550/ arXiv.2604.21780. Hinze Hogendoorn and Anthony N. Burkitt. Predictive coding of visual object position ahead of moving objects revealed by time-resolved EEG decoding. NeuroImage, 171:55–61, 2018. Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009, 2025. Junha Hyung, Kinam Kim, Susung Hong, Min-Jung Kim, and Jaegul Choo. Spatiotemporal skip guidance for enhanced video diffusion sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. doi: 10.48550/arXiv.2411.18664. Georg B. Keller and Thomas D. Mrsic-Flogel. Predictive processing: A canonical cortical compu- tation. Neuron, 100(2):424–435, 2018. doi: 10.1016/j.neuron.2018.10.003. 11 Preprint. Under review. Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Computational Biology, 10(11):e1003915, 2014. doi: 10.1371/journal.pcbi.1003915. Michael King, Aravindh Mahendran, Matthew Koichi Grimes, Fedor Kitashov, Adham Elarabawy, Pedro Velez, Maks Ovsjanikov, and Viorica P ̆ atr ̆ aucean. Gen4u: Unifying video generation and understanding via diffusion. arXiv preprint arXiv:2607.06856, 2026. Nikolaus Kriegeskorte and Pamela K. Douglas. Cognitive computational neuroscience. Nature Neuroscience, 21:1148–1160, 2018. doi: 10.1038/s41593-018-0210-5. Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis: connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, 2008. doi: 10.3389/neuro.06.004.2008. Benjamin Lahner, Kshitij Dwivedi, Polina Iamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, SouYoung Jin, N. Apurva Ratan Murty, Kendrick Kay, Aude Oliva, and Radoslaw Cichy. Modeling short visual events through the BOLD moments video fMRI dataset and metadata. Nature Communications, 15(1):6241, 2024. doi: 10.1038/s41467-024-50310-3. Benjamin Lahner, Mayukh Deb, N. Apurva Ratan Murty, and Aude Oliva. MOSAIC: A scalable framework for fMRI dataset aggregation and modeling of human vision. bioRxiv preprint, 2025. Caroline S. Lee, Mariam Aly, and Christopher Baldassano. Anticipation of temporally structured events in the brain. eLife, 10:e64972, 2021. doi: 10.7554/eLife.64972. Tai Sing Lee. The visual system’s internal model of the world. Proceedings of the IEEE, 103(8): 1359–1378, 2015. doi: 10.1109/JPROC.2015.2434601. Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffu- sion model is secretly a zero-shot classifier. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. Ping Li, Trevor J. Hastie, and Kenneth W. Church. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 287–296, 2006. doi: 10.1145/1150402.1150436. Ruogu Lin, Thomas Naselaris, Kendrick Kay, and Leila Wehbe. Stacked regressions and structured variance partitioning for interpretable brain maps. NeuroImage, 298:120772, 2024. doi: 10.1016/ j.neuroimage.2024.120772. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023. doi: 10.48550/arXiv.2210.02747. Wei Ji Ma, Jeffrey M. Beck, Peter E. Latham, and Alexandre Pouget. Bayesian inference with probabilistic population codes. Nature Neuroscience, 9(11):1432–1438, 2006. Jisu Nam, Soowon Son, Dahyun Chung, Jiyoung Kim, Siyoon Jin, Junhwa Hur, and Seungryong Kim. Emergent temporal correspondences from video diffusion transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2025. doi: 10.48550/arXiv.2506.17220. Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fMRI. NeuroImage, 56(2):400–410, 2011. doi: 10.1016/j.neuroimage.2010.07.073. William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022. Jacob S. Prince, Ian Charest, Jan W. Kurzawski, John A. Pyles, Michael J. Tarr, and Kendrick N. Kay. Improving the accuracy of single-trial fMRI response estimates using GLMsingle. eLife, 11:e77599, 2022. doi: 10.7554/eLife.77599. 12 Preprint. Under review. Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional inter- pretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79–87, 1999. doi: 10.1038/4580. Edmund T. Rolls. Two what, two where, visual cortical streams in humans. Neuroscience & Biobe- havioral Reviews, 160:105650, 2024. doi: 10.1016/j.neubiorev.2024.105650. Christina Sartzetaki, Gemma Roig, Cees G. M. Snoek, and Iris I. A. Groen. One hundred neural networks and brains watching videos: Lessons from alignment. In International Conference on Learning Representations (ICLR), 2025. doi: 10.1101/2024.12.05.626975. Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Score: Which artificial neural network for object recognition is most brain-like? bioRxiv preprint, 2018. Soowon Son, Honggyu An, Chaehyun Kim, Hyunah Ko, Jisu Nam, Dahyun Chung, Siyoon Jin, Jung Yi, Jaewon Min, Junhwa Hur, and Seungryong Kim. Repurposing video diffusion transformers for robust point tracking. arXiv preprint arXiv:2512.20606, 2025. Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14453–14463, 2023. Yingtian Tang, Abdulkadir Gokce, Khaled Jedoui Al-Karkari, Daniel Yamins, and Martin Schrimpf. Many-two-one: Diverse representations across visual pathways emerge from a single objective. bioRxiv, 2025. doi: 10.1101/2025.07.22.664908. Hannah Tarder-Stoll, Christopher Baldassano, and Mariam Aly. The brain hierarchically represents the past and future during multistep anticipation. Nature Communications, 15, 2024. doi: 10. 1038/s41467-024-53293-3. Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Shiwei Zhang, Siyang Sun, Wei Lin, Wenyuan Yu, Yingya Zhang, Yun Zheng, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. Ruben S. van Bergen and Janneke F. M. Jehee. Probabilistic representation in human visual cortex reflects uncertainty in serial decisions. Journal of Neuroscience, 39(41):8164–8176, 2019. Matteo Visconti, Fatma Deniz, Tom Dupr ́ e la Tour, and Jack L. Gallant. Encoding models in functional magnetic resonance imaging: The voxelwise encoding model framework. PsyArXiv preprint (v4), 2026. URL https://osf.io/preprints/psyarxiv/nt2jq_v4. Binxu Wang and John J. Vastola. Diffusion models generate images like painters: An analytical theory of outline first, details later. arXiv preprint arXiv:2303.02490, 2023. Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619–8624, 2014. doi: 10. 1073/pnas.1403112111. Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. arXiv preprint arXiv:2412.07772, 2024. Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In International Conference on Learning Representations, 2025. 13 Preprint. Under review. Yonghao Yu, Lang Huang, Runyi Li, Zerun Wang, and Toshihiko Yamasaki. Video-mirai: Autore- gressive video diffusion models need foresight. arXiv preprint arXiv:2606.03971, 2026. Shenghai Yuan, Yuanyang Yin, Zongjian Li, Xinwei Huang, Xiao Yang, and Li Yuan. Helios: Real real-time long video generation model. arXiv preprint arXiv:2603.04379, 2026a. Tianyu Yuan, Yuanbo Yang, Lin-Zhuo Chen, Yao Yao, and Zhuzhong Qian. Denoise to track: Harnessing video diffusion priors for robust correspondence. arXiv preprint arXiv:2512.04619, 2026b. doi: 10.48550/arXiv.2512.04619. Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang, Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng. Videorepa: Learning physics for video generation through relational alignment with foundation models. arXiv preprint arXiv:2505.23656, 2025. doi: 10.48550/arXiv.2505.23656. AAPPENDIX A.1REPLICATION ON OTHER AUTOREGRESSIVE VIDEO MODELS We repeat the within-model and cross-model analyses on two other AR video models, CausVid (Yin et al., 2024) and Self-Forcing (Huang et al., 2025). Both are fine-tuned from the non-AR base model Wan-2.1 T2V-1.3B and generate the future video in four denoising steps. We extract the representations for future video generation and for observed video reconstruction from all four steps and all 30 layers, and run the same encoding and variance partitioning as in the main analysis (Figures 5–8). Vi sual P rimary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Video Generation 00.0100.3400.36 Unique Observed VideoShared Unique Future Video Generation R² / NC 0123 Steps 0 14 29 Layers S0 L10 0 0.33 R² / NC 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 CausVid Figure 5: Within-model analysis of CausVid. A.2ABLATION STUDY Token Readout. The activation of the AR model while it generates the future video has an ob- served part and a future part, H ar s,b = [H obs s,b ; H fut s,b ] (Equation 4). In the main analysis we read out the future part H fut s,b . For the ablation we also read out the observed part H obs s,b and the concatenation [H obs s,b ; H fut s,b ], and apply the same sparse random projection and vertex-wise ridge regression as in the main analysis. 14 Preprint. Under review. Vi sual P rimary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Video Generation 00.0100.3400.35 Unique Observed VideoShared Unique Future Video Generation R² / NC 0123 Steps 0 14 29 Layers S0 L11 0 0.32 R² / NC 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 00.0200.3300.39 Unique Observed VideoShared Unique Future Generation R² / NC Vi sual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Shared Unique Future Generation (c)(d) 0 0.34 R² / N C (b) AR Model Observed Video Future Generation Within Model Analysis (a) AR Model 02549 Steps 0 20 39 Laye rs S0 L11 SelfForcing Figure 6: Within-model analysis of Self-Forcing. 00.0400.5000.07 Unique Observed Video ReconstructionShared Unique Future Video Generation R² / NC 0123 Steps 0 14 29 Layers S2 L10 0 0.33 R² / NC 0.20.4 R² / NC Primary Early Dorsal Ventral MT+ 0123 Best step 01429 Best layer AR Model Base Model 123 Best step 01429 Best layer AR Model Base Model 0123 Steps @ best layer 0.0 0.1 0.2 0.3 R² / NC Future Video Generation (L10) Observed Video Reconstruction (L10) 01429 Layers @ best step 0.0 0.1 0.2 0.3 R² / NC Future Video Generation (S0) Observed Video Reconstruction (S2) Vi sual P rimary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Reconstruction Shared Unique Future Video Generation Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) CausVid Figure 7: Cross-model analysis of CausVid. Future Tokens Align Best with the Visual Cortex. Reading out only the future tokens aligns best with the visual cortex (Figure 9). In the Visual System the encoding accuracy of the future tokens reaches ̃ R 2 = 0.334 against 0.211 for the observed tokens, with the concatenated tokens between them at 0.309. Variance Partitioning against the Observed Tokens. The observed tokens H obs s,b also form a grid over denoising steps and layers, so we partition the variance between H obs s,b and H fut s,b . The cell of each representation is selected on the training folds for each region and subject. 15 Preprint. Under review. 00.0500.4900.07 Unique Observed Video ReconstructionShared Unique Future Video Generation R² / NC 0123 Steps 0 14 29 Layers S2 L12 0 0.32 R² / NC 0.20.4 R² / NC Primary Early Dorsal Ventral MT+ 0123 Best step 01429 Best layer AR Model Base Model 123 Best step 01429 Best layer AR Model Base Model 0123 Steps @ best layer 0.0 0.1 0.2 0.3 R² / NC Future Video Generation (L11) Observed Video Reconstruction (L12) 01429 Layers @ best step 0.0 0.1 0.2 0.3 R² / NC Future Video Generation (S0) Observed Video Reconstruction (S2) Vi sual P rimary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Video Reconstruction Shared Unique Future Video Generation Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) SelfForcing Figure 8: Cross-model analysis of Self-Forcing. Visual P rimary Early Dorsal V entral MT+ 0.0 0.2 0.4 R² / NC Future Tokens All Tokens Observed Tokens Figure 9: Encoding accuracy of future video generation under different token readouts. Future Tokens Keep Their Unique Contribution against the Observed Tokens. The observed tokens align best with the Visual System at step 2 and layer 11 (Figure 10b). Again, future video generation aligns better than the observed tokens. In the Visual System the shared contribution is 0.213, the unique contribution of future video generation is 0.121, and the unique contribution of the observed tokens is−0.001 (Figure 10c). The unique contribution of future video generation is above the unique contribution of the observed tokens in every division (p = 0.002), and it grows along the hierarchy, from 0.055 in Primary Visual and 0.093 in Early Visual to 0.137 in Dorsal Stream, 0.171 in Ventral Stream, and 0.189 in MT+ Complex. On the cortical surface the unique contribution of future video generation is again concentrated in higher-order visual cortex (Figure 10d). A.3COMPARISON WITH A WIDE RANGE OF VISION MODELS We compared the internal representations of a wide range of vision models with the representation for future video generation of the AR model. We included both video and image vision models, 80 models in total. Vision Model Activation Extraction. A video vision model that takes n frames spaced δ frames apart sees nδ of the T frames of a video in one pass, so we run K =⌈T/(nδ)⌉ passes v (1) ,...,v (K) to cover the whole video. An image vision model has no temporal window, so a pass is a single frame and K = T . Let h ℓ denote the activation at layer ℓ of a vision model, where ℓ = 1,...,L indexes 16 Preprint. Under review. 0.20.4 R² / NC Primary Early Dorsal Ventral MT+ 02449 Best step 01939 Best layer AR Model Observed Tokens AR Model Observed Tokens 02449 Steps @ best layer 0.0 0.1 0.2 0.3 R² / NC Future Video Generation (L11) Observed Video Tokens (L11) 01939 Layers @ best step 0.0 0.1 0.2 0.3 R² / NC Future Video Generation (S0) Observed Video Tokens (S2) Vi sual P rimary Early Dorsal Ventral MT+ 0.0 0.1 0.2 R² / NC Unique Observed Video Tokens Shared Unique Future Video Generation 00.0300.4200.21 Unique Observed Video TokensShared Unique Future Video Generation R² / NC 02449 Steps 0 19 39 Layers S2 L11 0 0.34 R² / NC Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Cross Model Analysis AR Model Base Model 02549 Steps 0 20 39 Layers S16 L8 0 0.34 R² / N C (a)(b) Base Model 00.0700.4700.09 Unique Observed ReconstructionShared Unique Future Generation R² / NC Visual Primary Early Dorsal Ventral MT+ 0.0 0.1 0.2 0.3 R² / NC Unique Observed Reconstruction Shared Unique Future Generation (c) (d) 01939 Layers @ best s tep 0.0 0.1 0.2 0.3 R² / N C AR Model (S0) Base Model (S16) 02449 Steps @ best l ayer 0.0 0.1 0.2 0.3 R² / N C AR Model (L11) Base Model (L8) (f)(g) 0.20.4 R² / NC Primary Earl y Dorsal Ventral MT+ AR Model Base Model (e) Observed Tokens Future Tokens Within Model Analysis (a) AR Model Observed Tokens Figure 10: Within-model analysis against the observed tokens. the composite stages of the backbone. The representation of the vision model X vis ℓ holds one row per video, the layer activation flattened over tokens and channels and averaged over passes, X vis ℓ v = 1 K K X k=1 vech ℓ v (k) .(8) Vision models have no denoising axis, so their grid reduces to the layer axis alone. X vis ℓ goes through the same sparse random projection and the same nested cross-validation (Appendix A.5). Figure 11 shows the encoding accuracy of every model in the Visual System, and Figure 12 the best model of each group in each division. A.4FMRI DATASET AND PREPROCESSING Stimuli and Acquisition. The BOLD Moments Dataset records whole-brain fMRI responses of ten subjects to 1,102 naturalistic videos of 3 s (Lahner et al., 2024). Each video was shown at a visual angle of 5 ◦ while the subject held central fixation, and each trial was the 3 s video followed by a 1 s inter-trial interval. The 1,000 training videos were presented three times each and the 102 test videos ten times each. Data were acquired on a 3 T Siemens Trio scanner with a 32-channel head coil. Functional images were T2 ∗ -weighted gradient-echo echo-planar images at 2.5 m isotropic resolution with a repetition time of 1.75 s, an echo time of 30 ms, a flip angle of 71 ◦ , 54 slices, and a multi-band acceleration factor of 2. A T1-weighted anatomical image at 1 m isotropic resolution was acquired as the anatomical reference, and dual-echo field maps were acquired at the beginning of every session to correct the spatial distortion of the functional images. Preprocessing. We use the release of the BOLD Moments Dataset under the MOSAIC framework, which aggregates vision fMRI datasets under a shared preprocessing pipeline (Lahner et al., 2025). 17 Preprint. Under review. 0.20 0.25 0.30 0.35 0.40 R² / NC AR Model 0.334 Base Model 0.301 video-ssl n = 7 video-supervised n = 24 video-language n = 5 0.20 0.25 0.30 0.35 0.40 R² / NC image-ssl n = 5 image-supervised n = 35 image-language n = 4 Figure 11: Encoding accuracy of vision models in the Visual System. Primary Early Dorsal Ventral MT+ −40 −30 −20 −10 0 10 20 30 % change vs AR Model AR Model Base Model V-JEPA2.1-g V-JEPA2.1-g V-JEPA2-g V-JEPA2.1-L V-JEPA2-g MViTv2-S UniFormer-B UniFormer-B ir-CSN-IG65M ir-CSN-IG65M IntVideo2-1B IntVideo2-1B IntVideo2-1B IntVideo2-1B IntVideo2-1B I-JEPA-H I-JEPA-H DINOv2-L DINOv2-L DINOv2-L MViT-B-IN1k MViT-B-IN1k C2D-R101 TSN-R50 TSN-R50 CLIP-RN50 CLIP-RN50 CLIP-RN50 CLIP-RN50 CLIP-RN50 video-ssl video-supervised video-language image-ssl image-supervised image-language Figure 12: Best vision model of each group in each division. fMRIPrep preprocesses each subject and registers the data to the fsLR32k cortical surface (Este- ban et al., 2019), and GLMsingle estimates one beta per presentation from the preprocessed time series (Prince et al., 2022). The betas of each session are normalized by the mean and standard deviation of the training videos presented in that session. Averaging the normalized betas over the repetitions of a video leaves one response per video at each vertex. Noise Ceiling. The noise ceiling is the upper bound on how well any model can predict the re- sponse of a vertex. In the MOSAIC pipeline the noise ceiling is calculated from the ten presentations of each test video. The variance across the ten presentations of the same video is the noise variance σ 2 noise , and the remaining variance of the response is the signal variance σ 2 signal . The response we 18 Preprint. Under review. model is the average over n = 10 presentations, which reduces the noise variance to σ 2 noise /n. The noise ceiling is the fraction of the variance of the averaged response that is signal, NC = σ 2 signal σ 2 signal + σ 2 noise /n .(9) Atlas. The HCP-MMP atlas parcellates each hemisphere into 180 areas grouped into 22 divisions by function and anatomy, and those 22 divisions into six major systems (Glasser et al., 2016). One of the six is the Visual System, which comprises the five divisions. Primary Visual is V1, 1,618 vertices. Early Visual is V2, V3, and V4, 2,565 vertices. The Dorsal Stream is V6, V3A, V7, IPS1, V3B, and V6A, 1,116 vertices. The Ventral Stream is V8, FFC, PIT, VMV1, VMV2, VMV3, and VVC, 1,250 vertices. The MT+ Complex is MST, LO1, LO2, MT, PH, V4t, FST, V3CD, and LO3, 1,345 vertices. A.5FEATURE CONSTRUCTION AND ENCODING MODEL Sparse Random Projection. Every representation is flattened over tokens and channels and re- duced to d = 6004 dimensions with a sparse random projection Φ (Li et al., 2006), X obs = Φ c obs , X fut s,b = Φ H fut s,b , X rec s,b = Φ H rec s,b ,(10) each lying inR N×d . The number of projections follows the Johnson–Lindenstrauss lemma, and ε = 0.1 for the N = 1,102 videos gives d = 6004. Ridge Regression. We follow the ridge regression and parameter selection of Lin et al. (2024). The encoding model of Equation 5 is solved in closed form, ˆ B = (X ⊤ X + λI) −1 X ⊤ Y, ˆ Y = X ˆ B.(11) The penalty λ is chosen for each vertex from 16 values spaced by factors of ten between 10 −6 and 10 9 . The outer loop divides the 1,000 training videos into five folds and predicts each fold from a model fit on the remaining four, which yields an out-of-fold prediction for every training video. The inner loop divides those four folds again into five and chooses λ at each vertex by the inner cross-validation error. The model is then refit on all training videos with the chosen λ. Cell Selection. Let g = (s,b) denote a grid cell and let X g be its feature matrix. Let R 2,oof m,p (X) denote the out-of-fold R 2 at vertex p in subject m. For a region r in subject m we select the cell g ⋆ r,m that maximizes that subject’s own out-of-fold score, averaged over the vertices of the region, g ⋆ r,m = arg max g 1 |r| X p∈r R 2,oof m,p X g ,(12) and use it for every vertex of r in that subject. Both the ridge penalty and g ⋆ r,m come from the training folds, so the test videos take no part in model selection. The encoding accuracy reported at a cell is the mean over subjects of each subject’s test score at the cell selected on their own training set. A.6VARIANCE PARTITIONING We use variance partitioning to separate what each feature space explains uniquely from what they explain in common (Visconti et al., 2026; Lin et al., 2024). For a pair a and b their predictions are combined at each vertex into a joint model, with convex weights fit to the training responses, ˆ Y stack = w a ˆ Y a +w b ˆ Y b ,(w a ,w b ) = argmin w a +w b =1, w≥0 Y train −(w a ˆ Y a oof +w b ˆ Y b oof ) 2 , (13) which has a closed-form solution at each vertex. Let ̃ R 2 a , ̃ R 2 b , and ̃ R 2 stack denote the normalized test scores of the two feature spaces and of their combination. The unique and shared contributions are U a = ̃ R 2 stack − ̃ R 2 b , U b = ̃ R 2 stack − ̃ R 2 a , U ab = ̃ R 2 a + ̃ R 2 b − ̃ R 2 stack .(14) 19 Preprint. Under review. A.7SPATIOTEMPORAL SKIP GUIDANCE Spatiotemporal skip guidance (STG) is a sampling-time guidance for video diffusion models (Hyung et al., 2025). In its attention variant (STG-Attention) it forms a perturbed weak model by skipping the self-attention of one layer b and pushes the generation path away from that weak model, ε = ε CFG + w ε cond − ε perturb , ε CFG = ε uncond + s ε cond − ε uncond .(15) We set w = 1.5 and the classifier-free guidance scale s = 5.0. Here ε cond and ε uncond are the predictions of the intact model under the conditional and the unconditional prompt, and ε perturb is the prediction under the conditional prompt with the self-attention of layer b skipped. In the skipped layer the attention scores are replaced by the identity. We apply STG while the AR model generates the 5 s video that follows the 3 s observed video. The unguided arm uses classifier-free guidance alone, and each of the other eight arms adds the STG term for one layer b∈0, 5, 11, 16, 22, 28, 32, 38. A.8HUMAN PREFERENCE EVALUATION Stimuli. For each of the 102 test videos the AR model generates the 5 s continuation under nine arms, eight with STG applied at one layer each and one baseline that uses classifier-free guidance alone. Each video is 8 s long, the 3 s original video followed by the 5 s continuation, rendered at 272× 272 and 33 fps without sound. Trial Design. In each trial an observer watches two videos side by side and chooses the one they prefer. The nine arms give 9 2 = 36 pairs of arms, and an observer sees every pair four times, each time with a different source video, so an observer completes 144 trials. The left and right positions are balanced within each pair. Five observers completed the test, 720 trials in total. Bradley–Terry Model. To rank the arms from the pooled choices we fit a Bradley–Terry model (Bradley & Terry, 1952), in which arm i is chosen over arm j with probability P (i≻ j) = π i π i + π j ,(16) where π i is the worth of arm i. We report the log-worth θ i = logπ i with the unguided arm anchored at θ = 0. A pseudo-count of 0.1 wins and 0.1 losses between every pair of arms keeps the estimate finite. A.9STATISTICAL TESTING Paired comparisons across the ten subjects use an exact two-sided sign-flip test, whose smallest attainable p is 0.002. The correlation between the layer profiles of encoding accuracy and of pref- erence is the Pearson r over the eight guided layers, with a two-sided p from an exact permutation test. Intervals on the log-worth are 95% intervals from 2,000 bootstrap resamples of observers. 20

Instant research discovery