Paper deep dive

Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex

Iishaan Inabathini, Margaret M. Henderson

Year: 2026Venue: arXiv preprintArea: q-bio.NCType: PreprintEmbeddings: 108

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the majority of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding models address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fitting this model to fMRI responses to short video clips. We compare joint spatiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides interpretable, stimulus-specific attention maps that dynamically follow moving objects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category-selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

104,451 characters extracted from source content.

Expand or collapse full text

CROSS-ATTENTION ENCODING MODELS REVEAL DYNAMIC SPATIOTEMPORAL ROUTING ACROSS HUMAN HIGHER VISUAL CORTEX Iishaan InabathiniMargaret M. Henderson Carnegie Mellon UniversityCarnegie Mellon University iishaan@cmu.edu mmhender@cmu.edu ABSTRACT Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the major- ity of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding mod- els address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fit- ting this model to fMRI responses to short video clips. We compare joint spa- tiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides in- terpretable, stimulus-specific attention maps that dynamically follow moving ob- jects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category- selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception. 1INTRODUCTION Human observers effortlessly parse objects, events, and actions from a dynamic visual input, sup- ported by computations in higher dorsal, ventral, and lateral visual cortex (Ungerleider & Mishkin, 1982; Pitcher & Ungerleider, 2021; Grill-Spector & Weiner, 2014). Recent work using methods such as functional magnetic resonance imaging (fMRI) along with deep neural network (DNN)- based computational models, has substantially advanced understanding of how the higher visual system processes static content such as natural scenes (Yamins & DiCarlo, 2016; Doerig et al., 2023). However, while a number of recent studies have applied similar methods to video-evoked data (e.g., Sartzetaki et al., 2025; Garcia et al., 2025; Tang et al., 2025), we still know comparatively much less about the neural basis of dynamic visual perception. Modeling neural responses to dynamic videos is challenging due to the spatiotemporal complexity of both videos and video-evoked neural responses. Visual cortex populations, as measured with fMRI, integrate information across time nonlinearly (Hasson et al., 2008; Zhou et al., 2018; Groen et al., 2022), making it challenging to identify which temporal moment is most predictive of the measured neural response. In addition to temporal structure, natural videos are spatially complex, often including multiple objects or agents that compete for neural representation (Zoccolan et al., 2005; Reddy & Kanwisher, 2007), which similarly makes it challenging to identify which spatial region of an image is responsible for driving a neural population. Evidence suggests that neural pop- ulations in the higher visual system respond to a preferred stimulus at multiple possible locations 1 arXiv:2609.36366v1 [q-bio.NC] 28 Sep 2026 A Extract Route (joint 3D cross-attention) Read out Video Video model W H T Spatiotemporal tokens x n 3D PE K V Learned parcel queries Per-voxel linear readout Predicted voxel responses B Mean pool a= 1/N Temporal a(t)/HW Spatial a(h,w)/T Factorized a(t)a(h,w) Joint 3D a(t,h,w) Softmax over all (t, h, w) tokens More flexible routing t = 5 Right MT S1 P556 Overlay = LN W k W v t = 8 t = 6 t = 7 t = 9 t = 10 t = 11 t = 12 h r a r C W o FFN Figure 1: Cross-attention encoding model. (A) A video backbone (V-JEPA 2) encodes each clip into a grid of spatiotemporal tokens. A learned query for each cortical parcel attends over the tokens, and the attended feature predicts the responses of the parcel’s voxels through a voxel-specific linear readout. (B) Constrained routing models, shown as example attention patterns over the token grid with the form of their attention weights. (C) Attention of the joint model for a parcel in right MT, overlaid on selected frames of a test clip of a weightlifter. within a large spatial receptive field (Rust & DiCarlo, 2010), and their response to the stimulus may depend on other objects (Zoccolan et al., 2005; Reddy & Kanwisher, 2007; Bao & Tsao, 2018) or surrounding context (Bar, 2004). The resulting neural response properties may then be described as having spatial selectivity that is stimulus-dependent, as opposed to being driven by features within a fixed spatial receptive field. In videos, where objects can also move in time, the relative weight- ing of information across time may similarly depend on the input, which could result in adaptive spatiotemporal weighting that depends on characteristics of the input video. To model such adaptive stimulus-dependent weighting of visual features by neural populations, recent work has incorporated attention operations, commonly used in Transformer neural networks (Vaswani et al., 2017; Dosovitskiy et al., 2021), into fMRI encoding models (Adeli et al., 2026; Hwang et al., 2025). While standard fMRI voxelwise encoding model approaches map from DNN features to voxel responses using a simple linear mapping (Naselaris et al., 2011), these newer approaches use a cross-attention module with a region-specific query vector to perform a more flexible routing operation that can improve model performance in higher visual regions (Adeli et al., 2026; Hwang et al., 2025). However, it is not clear whether and how such approaches can be extended to modeling brain responses to video stimuli, where spatial and temporal information may be independently and/or jointly reflected in neural response properties. Here, we present a new cross-attention encoding model framework for predicting voxelwise fMRI activation in response to natural video clips. Using fMRI data from the BOLD Moments Dataset (BMD; Lahner et al. (2024)) and features extracted from a self-supervised video model (V-JEPA-2 (Assran et al., 2025)), we compare the predictive accuracy of models using spatial, temporal, and joint spatiotemporal cross-attention. Our contributions are as follows: • We extend previous work by showing that a cross-attention readout leveraging spatial at- tention over video tokens can improve predictive accuracy of voxelwise encoding models for video-evoked fMRI data, across multiple DNN backbones. • We show that the novel addition of joint spatiotemporal attention boosts encoding model accuracy across several higher visual regions, notably in lateral and parietal visual cortex. 2 • We demonstrate how joint spatiotemporal attention leads to more interpretable and accurate attention maps compared to factorized attention, tracking the movement of relevant objects across a video clip. • We show dissociations between the attention maps derived from different functionally- localized higher visual cortex regions, with attention weights dynamically tracking the lo- cation of region-relevant video content. 2RELATED WORK Neural coding during dynamic visual perception. Classic models of vision suggest that static object recognition and dynamic spatial vision are anatomically segregated into the ventral and dor- sal stream, respectively (Ungerleider & Mishkin, 1982; Goodale & Milner, 1992; Grill-Spector & Weiner, 2014), with dorsal areas (MT, MST) key to visual motion processing (Albright, 1984; An- dersen, 1997). Recent work also provides evidence for a lateral visual pathway specializing in dy- namic social perception (Pitcher et al., 2011; Pitcher & Ungerleider, 2021; McMahon et al., 2023), including regions along the superior temporal sulcus (STS). Other research on temporal dynamics has suggested a hierarchical posterior-to-anterior organization based on temporal receptive window size (Hasson et al., 2008; Groen et al., 2022; Zhou et al., 2018). Here, we develop a novel strategy to computationally model higher visual cortex during dynamic natural event perception. Category-selective organization in visual cortex. Anatomically localized regions of ventral and lateral visual cortex exhibit functional selectivity for human faces (FFA, OFA, STS), human bodies (EBA), and scenes (PPA, TOS, RSC), among other categories (Epstein & Kanwisher, 1998; Kan- wisher et al., 1997; Downing et al., 2006; Epstein & Baker, 2019). These regions were classically identified using isolated objects, with recent work extending this using naturalistic fMRI and com- putational modeling (Khosla et al. (2022); Ratan Murty et al. (2021); Luo et al. (2023), and others). However, because natural videos often include multiple categories that interact over time, modeling category-selective responses during natural video perception remains challenging. Here, we show how joint spatiotemporal attention can be used to dissociate the features driving responses in distinct category-selective networks during complex video perception. Voxelwise fMRI encoding models. To model cortical fMRI responses during complex natural vi- sion, forward encoding models construct a stimulus-computable function mapping from the input (i.e., a stimulus viewed by an observer) to the response of an fMRI voxel (Naselaris et al., 2011; Serences & Saproo, 2012; Wehbe et al., 2014). This is typically done by extracting stimulus fea- tures from a task-optimized DNN model, then learning a voxel-specific mapping from the DNN activations to the individual voxel’s response (Naselaris et al., 2011; Yamins & DiCarlo, 2016). Benchmarking the cross-validated accuracy of DNN encoding models can be used to evaluate the alignment of DNN models and neural representations (Schrimpf et al., 2020; Wang et al., 2023a; Conwell et al., 2024; Sartzetaki et al., 2025; Doerig et al., 2023). The voxel-specific mapping is most commonly linear (Naselaris et al., 2011), and is often preceded by spatial and/or temporal averaging to handle high-dimensional DNN features. Other approaches use a “factorized” strategy which applies a shared spatial weighting function across feature maps (Klindt et al., 2017; St-Yves & Naselaris, 2018; Lurz et al., 2021; Henderson et al., 2023), assuming separable and input-invariant spatial selectivity. Recent work replaces this with a cross-attention module that dynamically weights space based on the input (Adeli et al., 2026; Hwang et al., 2025). Here, we extend this flexible routing strategy to the spatiotemporal domain. Video encoding and decoding models for fMRI. Large-scale video fMRI datasets (Lahner et al., 2024; Gifford et al., 2025) have spurred efforts to build predictive video encoding models based on DNN features (Sartzetaki et al., 2025; Tang et al., 2025; H ̈ ofling et al., 2026; Gamal et al., 2026; Han et al., 2024; Al-Karkari et al., 2025; Garcia et al., 2025; Pushpita & Wehbe, 2025). Other work uses video encoding models to perform video reconstruction (Chen et al., 2023; Lu et al., 2024; Yeung et al., 2025), or to synthesize an optimal video stimulus (Tang et al., 2026). Recent papers find that V-JEPA-2 features (Assran et al., 2025) achieve top predictive accuracy among video models (Tang et al., 2025; 2026; H ̈ ofling et al., 2026). Our approach is distinct in introducing a spatiotemporal cross-attention framework that yields interpretable maps of how video features are weighted over space and time by cortical populations. 3 3METHODS 3.1VIDEO CROSS-ATTENTION ENCODING MODEL FRAMEWORK We propose a cross-attention encoding model framework for video-evoked fMRI responses (Fig. 1) with three goals. First, each cortical population should read out from a video in a stimulus-dependent way, selecting where and when in the clip to draw features from, rather than relying on a fixed receptive field or on features pooled over the video. Second, the contribution of each form of routing should be testable: spatial, temporal, factorized and joint spatiotemporal routing are constrained versions of the same attention operation, so they can be compared with the backbone, readout and training held fixed. Third, the routing should be interpretable: the attention weights form a stimulus- specific map over the video that can be compared directly with its content. Following cross-attention encoding models for static images (Adeli et al., 2026; Hwang et al., 2025), a learned query for each brain parcel attends over the token grid of a frozen video backbone, and we extend the attention to range jointly over time and space. Below, we describe the model, the constrained comparison models and the parcel-based queries. Section 3.2 describes the data, Section 3.3 the fitting, evaluation and ridge regression baseline, and Appendix 6.13 the analyses of category selectivity in the attention maps. Spatiotemporal token grid. A frozen video backbone maps each clip to a grid of tokens X ∈ R T×H×W×D , with token x n ∈R D at grid position (t n ,h n ,w n ) and N = THW tokens per clip (T = 16, H = W = 24, D = 1408 and N = 9216 for our backbone; Appendix 6.3). Each channel is z-scored over training clips. The tokens are computed once; all learning takes place in the readout. Cross-attention readout. Each parcel r has a learned query q r ∈R D that does not depend on the stimulus. Keys and values are computed from the tokens as k n = W k LN x n + α PE(t n ,h n ,w n ) , v n = W v x n , where W k ,W v ∈R D×D , LN is layer normalization, and PE is a fixed sinusoidal 3D positional encoding whose D channels are split into three blocks encoding t, h and w (α = 1). Because the positional encoding enters the keys but not the values, a query can prefer locations and moments as well as content, while the values carry content only. The attention of parcel r is a single softmax over all N tokens, a rn = exp q ⊤ r k n / √ D P N m=1 exp q ⊤ r k m / √ D , X n a rn = 1, which, reshaped to the token grid, gives the attention map a r (t,h,w) (Fig. 1). The fixed query encodes a stable preference, and the map shows where and when a clip meets it. The attended feature z r = P n a rn v n is passed through an output projection and a residual feed-forward network, h r = W o z r , h r ← h r + FFN LN(h r ) , where the FFN is a two-layer MLP with a GELU nonlinearity and hidden width 4D. We use a single attention head, and W k , W v , W o and the FFN are shared across parcels, so parcels differ only in their query and voxel readout. Each voxel is predicted from the output h r of its parcel by a voxel- specific linear readout, trained jointly with the attention module by gradient descent. All voxels in a parcel thus share one attention map but weight the attended features differently. Constrained routing models. To isolate the contributions of spatial, temporal and coupled spa- tiotemporal routing, we compare the joint model with models that differ from it only in their at- tention weights. Their attention factorizes into a temporal and a spatial distribution, a r (t,h,w) = a T r (t)a S r (h,w), and each factor is uniform, fixed or routed. A uniform factor weights all frames (or locations) equally. A fixed factor is a softmax over learned logits for each parcel, a stimulus- independent receptive field that is the same for every clip. A routed factor is a softmax of the parcel’s query against keys computed, with their own key projection and positional encoding, from tokens averaged over the other axis (the spatially averaged token of each frame, or the temporally averaged token at each location). We focus on four of these models: • Mean pool (both factors uniform, a rn = 1/N ): every parcel reads the same clip-average feature, with no routing. 4 • Temporal (a T r routed, a S r uniform): the parcel selects moments but pools over space. • Spatial (a T r uniform, a S r routed): the parcel selects locations but pools over time, as in spatial routing for static images applied to the time-averaged video. • Factorized (both factors routed): the parcel selects moments and locations, but attends to the same locations at every moment, so it cannot follow moving content. Because the joint model’s attention is a single softmax over all tokens, a parcel can attend to different locations at different moments. Comparing the joint and factorized models therefore isolates the benefit of coupled spatiotemporal routing, and comparing the spatial and temporal models with mean pooling isolates the benefit of routing along each axis. Parcel-based queries. We assign one query to each parcel of the Schaefer 1000-parcel, 7-network atlas (Schaefer et al., 2018), resampled to the voxel grid of the dataset (Appendix 6.2), which gives 423–433 queries per subject, including one for voxels without a parcel label. We judged this the ap- propriate granularity: fine enough for differences within and between functional regions to emerge in the attention maps, and coarse enough that each map is estimated from many voxels (Appendix 6.1). 3.2FUNCTIONAL MRI VIDEO DATASET BOLD Moments Dataset. We use the BOLD Moments Dataset (BMD; Lahner et al. (2024)), in which 10 subjects watched 1,102 naturalistic 3 s video clips of everyday events during 3T fMRI (2.5 m isotropic voxels, TR 1.75 s): 1,000 training clips shown 3 times each and 102 test clips shown 10 times each. We predict the single-trial response amplitudes released with the dataset (GLMsingle betas; Prince et al. (2022)), averaged over the repeats of each clip, in the 17.2–17.5k voxels per subject that lie in cortex with reliable responses to the videos or in the functional ROIs (preprocessing details in Appendix 6.2). We also use the per-voxel noise ceiling released with the dataset, computed for responses averaged over the 10 test repeats. Functional ROIs. We use the 23 ROIs released with the dataset, pooled across hemispheres: retino- topic ROIs from the atlas of Wang et al. (2015), MT and parietal ROIs from the HCP multimodal parcellation (Glasser et al., 2016), and category-selective ROIs defined in each subject with a video- based functional localizer (Appendix 6.2). To summarize prediction accuracy (Fig. 2), we pool the ROIs into four groups: early visual (V1–V3, dorsal and ventral, V3ab and hV4), lateral/face/body (MT, EBA, FFA, OFA, LOC, STS), scene (PPA, RSC, TOS) and parietal (IPS0, IPS1-2-3, 7AL, BA2, PFop, PFt). We also report all modeled voxels and the voxels outside all ROIs. The groups follow a clustering of the ROIs by their responses to the videos (Appendix 6.4). 3.3MODEL FITTING AND EVALUATION PROCEDURES We use frozen features from layer 32 of 40 of V-JEPA 2 ViT-g/16 (Assran et al., 2025), a self- supervised video model whose features give the most accurate fMRI predictions among current large video models (Tang et al., 2025; 2026; H ̈ ofling et al., 2026); layer 32 gave the most accurate predictions in a ridge-regression sweep over the layers of several video and image backbones (Ap- pendix 6.5). To test whether our results depend on the backbone and its depth, we also fit the joint and all constrained models to features from an early and a late layer of another video model and two image models (Appendix 6.11). Each subject is fit separately, with the mean squared error over all voxels as the loss. Every model is an ensemble of 10 members, each trained on a different random split of the 1,000 training clips into 900 clips for training and 100 for early stopping, and, following Adeli et al. (2026), the test predictions of the members are averaged voxel by voxel before eval- uation. We train with AdamW and early stopping, with the same hyperparameters for all models, backbones and subjects. As a standard linear encoding model, we also fit voxelwise ridge regression to the same backbone layer, averaged over time and space into one 1408-dimensional vector per clip, with the same splits and ensembling (token extraction, optimization and ridge details in Ap- pendix 6.3). For each voxel, we compute the Pearson correlation r between the ensemble predictions and the measured responses across the 102 test clips and report the signed r 2 = sign(r)r 2 . 5 Joint 3D − factorizedJoint 3D − spatial Joint 3D − uniform Joint 3D − ridge -0.04 0 0.04 Δ signed r² 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.14 0.16 0.18 0.20 0.22 Signed r² uniform spatial factorized joint 3D 0.150 0.175 0.200 0.225 uniform spatial factorized joint 3D 0.225 0.250 0.275 0.300 0.325 uniform spatial factorized joint 3D 0.075 0.100 0.125 0.150 uniform spatial factorized joint 3D 0.10 0.15 0.20 uniform spatial factorized joint 3D 0.125 0.150 0.175 0.200 0.225 A B C ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 2: Comparing accuracy of joint spatiotemporal attention versus other mappings (V-JEPA 2 ViT-g, layer 32; 102 test clips). (A) Joint model minus the factorized, spatial and mean-pool (“uni- form”) models and ridge regression on mean-pooled features in voxelwise signed r 2 , averaged over subjects (fsaverage flatmap, unthresholded; each subject in Appendix 6.7). (B) Region-mean signed r 2 of every routing configuration (T = temporal factor, S = spatial factor; each uniform, fixed or routed), the joint model and ridge regression. Large dots: mean± SEM over subjects; small dots: individual subjects; grey band: noise ceiling (mean± SEM). Region means include only voxels with a noise ceiling≥ 5%. (C) Zoomed inset from (B); shows region means of the mean-pool, spatial, factorized and joint models in each subject (colored lines) and their mean± SEM (black). Table 1: Region-mean signed r 2 on the 102 test clips (V-JEPA 2 ViT-g, layer 32; mean over 10 subjects; voxels with noise ceiling ≥ 5%). The best model in each row is in bold. All routing configurations and the other backbones are given in Table 7. RegionRidgeMean poolTemporalSpatialFactorizedJointNoise ceiling All voxels.174.180.180.187.186.191.34 Early visual.178.177.178.189.189.189.36 Lateral/face/body.242.249.248.255.255.263.40 Scene.118.115.114.116.115.119.24 Parietal.109.122.121.125.125.129.29 Outside ROIs.165.172.171.178.177.181.33 6 4RESULTS Comparing model performance across attention types. We first examine performance averaged across all voxels, then examine region-specific differences. Across all modeled voxels, we observed a substantial advantage for joint spatiotemporal routing, with the joint model having higher r 2 than every constrained model and than ridge regression in all 10 subjects (Fig. 2B, Table 1), including the closest, the spatial model (0.191 vs. 0.187; t(9) = 5.85, p FDR < 0.001). Unless stated otherwise, we compare models with two-sided paired t-tests of region-mean signed r 2 across the 10 subjects (FDR corrected; p FDR < 0.05; Benjamini & Hochberg (1995)); see Table 3. The constrained models show which kind of routing produces this gain. Of the 0.010 by which the joint model exceeds mean pooling, 0.006 is recovered by routing over space alone (spatial minus mean pool: t(9) = 5.07, p FDR < 0.001; see Table 4 for additional comparisons). This extends to video previous work showing an advantage of stimulus-dependent spatial routing over linear readouts for static images (Adeli et al., 2026). This pattern holds for every backbone and layer we tested, with the exception of scene-selective cortex and for the last stage of CLIP (Appendix 6.8). Routing over time alone does not improve over uniform pooling (temporal model, 0.180), and adding temporal routing to spatial routing does not improve beyond spatial routing alone (factorized model, 0.186). We also find that the benefit of spatial routing depends on the selection being driven by the stimulus: a fixed spatial distribution learned for each parcel, which acts as a static receptive field, does not exceed the performance of mean pooling (0.181; Table 7, ablations in Appendix 6.9). These results provide insight into what drives the benefit of the joint spatiotemporal routing mechanism: it allows parcels to attend to different locations at different moments in a stimulus-dependent manner. This benefit of coupled routing differs between regions. On the flatmaps, the advantage of the joint model over the factorized and spatial models is concentrated in lateral occipitotemporal cortex (Fig. 2A). At the level of functionally-defined ROIs, the advantage is largest in lateral/face/body regions, where the joint model reaches 0.263 (66% of the noise ceiling) and exceeds the factorized, spatial, mean-pool and ridge regression models in every subject (joint minus factorized: t(9) = 6.60, p FDR < 0.001; Fig. 2C; individual ROIs in Appendix 6.10). Parietal regions similarly show an advantage for joint routing over all other models (Table 3). In early visual cortex, by contrast, the spatial, factorized and joint models are tied (0.189; joint minus factorized: t(9) = −0.03, p FDR = 0.98), and all three improve on mean pooling by 0.012. Stimulus-dependent selection of spatial locations thus accounts for the whole gain in early visual cortex, whereas lateral and parietal regions additionally benefit from following content over time. In scene-selective regions, we find more mixed evidence for a benefit of spatial or temporal routing. Neither the joint model nor the spatial model significantly exceed the the ridge baseline (Table 3, Table 4). This may reflect the role of scene-selective cortex in encoding global scene layout, which features averaged over the whole clip may sufficiently capture (Appendix 6.4). The advantage of coupled routing in lateral regions is not specific to V-JEPA 2 (Appendix 6.11). With features from the late layers of a second video model (VideoMAE v2) and two image mod- els (DINOv2 and CLIP), the joint model is again more accurate than the factorized model in lat- eral/face/body regions (joint minus factorized: t(9) = 2.88, 2.49 and 3.47; p FDR = 0.027, 0.048 and 0.011, respectively; FDR-corrected over the 192 tests of the sweep). The pattern depends on the features, however. With features from early layers, the joint model is less accurate than models that route over space alone, and for the last stage of CLIP, ridge regression on mean-pooled features is the most accurate model in every region except early visual (Table 7). Interpreting cross-attention maps. Fig. 3 shows the attention maps of the joint and factorized models for one parcel in each of two test clips, both in lateral/face/body regions, where coupled routing improves accuracy most. For a parcel in left EBA, the attention of the joint model stays on the body of a fencer and, after a cut to a close-up, follows the arm and sword through the swing. For a parcel in left STS, it stays on the head and body of a panda as it climbs. The factorized model cannot: its spatial map is fixed across frames and only its temporal weights change, so it concentrates its attention on a few frames and, within them, spreads it over a large region that includes the background. In the fencer clip, a single spatial map must serve both the wide shot and the close-up, although the body is in different places in the two shots (more examples in Appendix 6.12). Cross-attention maps differentiate category-selective networks. If the attention of a parcel re- flects what its voxels respond to, the attention of category-selective regions should land on their 7 t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Fencer Left EBA S6 P059 1× 3× ≥5× × uniform t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Panda Left STS S9 P377 1× 3× ≥5× × uniform Figure 3: Joint attention follows moving content, whereas factorized attention cannot. Attention maps of one parcel in each of two test clips, for the joint model (upper two rows of each block) and the factorized model (lower two rows), over all 16 frames of the clip. Left of each block: the subject, the parcel and its location on the flatmap within its functional ROI. Attention is shown in units of the uniform level 1/N on the same scale for both models: tokens at or below the uniform level are transparent, and color and opacity increase up to 5 times the uniform level. preferred category, and it should be possible to find these regions from the attention alone, without fMRI localizer data. We test both with the attention of the joint model on the 102 test clips, using the intersection over union (IoU) of each parcel’s 92 most-attended tokens (the top 1%) with category masks obtained by segmenting the clips with SAM 3 (Carion et al., 2025) (Appendix 6.13). Because the raw IoU mostly reflects the size of each category (Fig. 4b), we z-score it across parcels to better isolate parcel differences (Fig. 4c). This metric indicates that FFA, OFA and EBA attend more than the average parcel to faces, bodies and animals, which usually appear together in the clips, and less to scences, whereas the scene-selective PPA and RSC show the reverse pattern. For the majority of face-, body-, and scene-selective ROIs, the highest category in this metric is the expected category (with exceptions OFA and STS, where animal and scene are slightly above face respectively). Based on these z-scored IoU values, we compute “localizer contrasts” that quantify face, body, and scene-selectivity. As expected, the parcels with the highest contrast overlap with the corresponding ROIs above chance for all three localizers: face 0.19 (chance 0.09), body 0.14 (chance 0.02) and scene 0.28 (chance 0.10; two-sided one-sample t-tests of overlap minus chance across subjects, FDR-corrected over the three localizers, all p FDR < 0.05; Table 8). Additional supporting analysis can be found in Appendix 6.14. These results indicate that attention maps reliably differentiate the selectivity of distinct functionally-defined cortical networks. 5DISCUSSION We developed a novel framework for modeling human fMRI responses to natural video clips, lever- aging joint spatiotemporal cross-attention over embeddings from a self-supervised video model. Our results showed that joint attention outperformed spatial-only or factorized attention in multiple 8 -7.507.5 FFAOFASTSEBAPPARSCTOSLOCAll parcels 0.000 0.005 0.010 IoU (raw) FaceBodyAnimalObjectScenefROI's own category FFAOFASTSEBAPPARSCTOSLOCAll parcels −1 0 1 IoU (z) Attention contrast (t) Derived: top-k parcelsLocalizer fROIs Face FFA+OFA+STS face − object k = 34, 41% in fROIs Body EBA body − object k = 12, 8% in fROIs Scene PPA+RSC+TOS scene − rest k = 46, 43% in fROIs b c a Figure 4: The attention of the joint model differentiates category-selective regions. (A) Localizing category-selective regions from attention alone in one subject (S8). Left: t of each localizer contrast of z-scored IoU across test clips, per parcel. Middle: the K highest-t parcels (K = number of ROI parcels) and their overlap with the ROIs. Right: the localizer-defined ROIs (parcels with at least 20% of their voxels inside). (B) Raw IoU of the 92 most-attended tokens (top 1%) with the masks of five categories, for eight category-selective ROIs and for all parcels (mean± SEM over 10 subjects). (C) The same IoU z-scored across all parcels, so that all parcels are 0 by definition. Outlined bars mark the category that defines each ROI (for LOC, all four foreground categories). regions of higher visual cortex, with the most consistent advantage in lateral and parietal regions. This aligns with past evidence that dorsolateral visual cortex regions play a key role in dynamic processing of motion, events and social actions (McMahon et al., 2023; Andersen, 1997; Pitcher et al., 2011), and expands upon these past findings by showing how cross-attention maps can yield interpretable tracking of the spatiotemporal features that drive cortical response on a per-frame basis. Our results could suggest that higher visual cortex implements a mechanism analogous to spatiotem- poral cross-attention over feature maps during video perception, but this remains to be empirically tested. Previous work suggests that normalization across multiple objects in images (Bao & Tsao, 2018; Zoccolan et al., 2005) and/or normalization across time (Zhou et al., 2018; Groen et al., 2022) can explain neural response properties in the visual system, and it is possible that scaled dot-product attention (Vaswani et al., 2017) approximates a similar normalization mechanism (Kozachkov et al., 2023). To evaluate this, future work should compare performance of cross-attention versus more biologically-inspired normalization models (Reynolds & Heeger, 2009; Carandini & Heeger, 2011). Other limitations of this work include the short length of the video stimuli (3s) used in the BMD dataset (Lahner et al., 2024). Expanding our framework to modeling longer movie stimuli (Boyle et al., 2025; Gifford et al., 2025), would allow investigation of temporal integration over longer time windows. The slow temporal resolution of fMRI recordings presents another limitation, so future work should also explore how this model can predict data in more temporally-resolved modalities including MEG and EEG. These extensions will lead to more comprehensive understanding of how neural populations adaptively weight time-varying information during dynamic natural vision. 9 REFERENCES Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses, 2026. URL https://arxiv.org/abs/2505.17329. Khaled Jedoui Al-Karkari, Yingtian Tang, Martin Schrimpf, and Daniel L. K. Yamins. Do dy- namics matter for neural alignment? A comparative study of video and static vision mod- els.In Proceedings of the Conference on Cognitive Computational Neuroscience (CCN), 2025.URL https://2025.ccneuro.org/abstract_pdf/Al-Karkari_2025_ Do_Dynamics_Matter_Neural_Alignment_A.pdf. Thomas D. Albright. Direction and orientation selectivity of neurons in visual area MT of the macaque. Journal of Neurophysiology, 52(6):1106–1130, 1984. doi: 10.1152/jn.1984.52.6.1106. Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Nase- laris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116–126, 2022. doi: 10.1038/s41593-021-00962-x. Richard A. Andersen. Neural mechanisms of visual motion perception in primates. Neuron, 18: 865–872, 1997. Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khali- dov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning, 2025. URL https://arxiv.org/abs/2506.09985. Pinglei Bao and Doris Y. Tsao. Representation of multiple objects in macaque category-selective areas. Nature Communications, 9:1774, 2018. doi: 10.1038/s41467-018-04126-7. Moshe Bar. Visual objects in context. Nature Reviews Neuroscience, 5(8):617–629, August 2004. doi: 10.1038/nrn1476. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. doi: 10.1111/j.2517-6161.1995.tb02031.x. Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Doll ́ ar, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181, 2025. J.A. Boyle, B. Pinsard, et al. CNeuroMod data collection complete: 200h of individual fMRI across diverse naturalistic and controlled tasks to build NeuroAI models. In Conference on Cognitive Computational Neuroscience (CCN 2025), 2025. Abstract. Matteo Carandini and David J. Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1):51–62, November 2011. doi: 10.1038/nrn3136. Erratum in: Nat Rev Neurosci. 2013 Feb;14(2):152. Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. SAM 3: Seg- ment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video re- construction from brain activity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=i913TUOvTK. 10 Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. A large- scale examination of inductive biases shaping high-level visual representation in brains and ma- chines. Nature Communications, 15(9383), 2024. doi: 10.1038/s41467-024-53838-2. Adrien Doerig, Rowan P. Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W. Lindsay, Konrad P. Kording, Talia Konkle, Marcel A. J. van Gerven, Nikolaus Kriegesko- rte, and Tim C. Kietzmann. The neuroconnectionist research programme. Nature Reviews Neuroscience, 24:431–450, May 2023. doi: 10.1038/s41583-023-00705-w. URL https: //w.nature.com/articles/s41583-023-00705-w. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929. P.E. Downing, A.W. Chan, M.V. Peelen, C.M. Dodds, and N. Kanwisher. Domain specificity in visual cortex. Cerebral Cortex, 16(10):1453–1461, 10 2006. doi: 10.1093/CERCOR/BHJ086. R. Epstein and N. Kanwisher. A cortical representation of the local visual environment. Nature, 392:598–601, 4 1998. doi: 10.1038/33402. Russell A. Epstein and Chris I. Baker. Scene perception in the human brain. Annual Review of Vision Science, 5:373–397, 2019. doi: 10.1146/annurev-vision-091718-014809. Oscar Esteban, Christopher J. Markiewicz, Ross W. Blair, Craig A. Moodie, A. Ilkay Isik, Asier Erramuzpe, James D. Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, Hiroyuki Oya, Satrajit S. Ghosh, Jessey Wright, Joke Durnez, Russell A. Poldrack, and Krzysztof J. Gor- golewski. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nature Methods, 16 (1):111–116, 2019. doi: 10.1038/s41592-018-0235-4. Mai Gamal, Mohamed Rashad, Eman Ehab, Seif Eldawlatly, and Mennatullah Siam. Pixel-level understanding of a world in motion within a neural encoding framework. Scientific Reports, 16:12188, 2026. doi: 10.1038/s41598-025-34141-w. URL https://doi.org/10.1038/ s41598-025-34141-w. Kathy Garcia, Emalie McMahon, Colin Conwell, Michael Bonner, and Leyla Isik. Modeling dy- namic social vision highlights gaps between deep learning and humans. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=wAXsx2MYgV. Alessandro T. Gifford, Domenic Bersch, Marie St-Laurent, Basile Pinsard, Julie Boyle, Lune Bellec, Aude Oliva, Gemma Roig, and Radoslaw M. Cichy. The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies, 2025. URL https://arxiv.org/abs/ 2501.00504. Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smith, and David C. Van Essen. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171–178, 2016. doi: 10.1038/nature18933. Melvyn A. Goodale and A. David Milner. Separate visual pathways for perception and action. Trends in Neurosciences, 15(1):20–25, January 1992. doi: 10.1016/0166-2236(92)90344-8. Kalanit Grill-Spector and Kevin S. Weiner. The functional architecture of the ventral temporal cortex and its role in categorization. Nature Reviews Neuroscience, 15(8):536–548, 6 2014. doi: 10.1038/nrn3747. URL https://w.nature.com/articles/nrn3747. Iris I. A. Groen, Giovanni Piantoni, Stephanie Montenegro, Adeen Flinker, Sasha Devore, Orrin Devinsky, Werner Doyle, Patricia Dugan, Daniel Friedman, Nick F. Ramsey, Natalia Petridou, and Jonathan Winawer. Temporal dynamics of neural responses in human visual cortex. Journal of Neuroscience, 42(40):7562–7580, 2022. doi: 10.1523/JNEUROSCI.1812-21.2022. 11 Hyewon Willow Han, Ruchira Dhar, Qingqing Yang, Maryam Hoseini Behbahani, Mar ́ ıa Alejan- dra Mart ́ ınez Ortiz, Tolulope Samuel Oladele, Diana C Dima, Hsin-Hung Li, Anders Søgaard, and Yalda Mohsenzadeh. Investigating the role of modality and training objective on representa- tional alignment between transformers and the brain. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024. URL https://openreview.net/ forum?id=t4CnKu6yXn. Uri Hasson, Eunice Yang, Ignacio Vallines, David J. Heeger, and Nava Rubin. A hierarchy of tem- poral receptive windows in human cortex. Journal of Neuroscience, 28(10):2539–2550, March 2008. doi: 10.1523/JNEUROSCI.5487-07.2008. Margaret M. Henderson, Michael J. Tarr, and Leila Wehbe. A texture statistics encoding model reveals hierarchical feature selectivity across human visual cortex. Journal of Neuroscience, 43 (22):4144–4161, 2023. doi: 10.1523/JNEUROSCI.1822-22.2023. Larissa H ̈ ofling, Matthias Tangemann, Lotta Piefke, Susanne Keller, Matthias Bethge, and Katrin Franke. Only brains align with brains: Cross-region alignment patterns expose limits of normative models. In ICLR, 2026. URL https://openreview.net/forum?id=cMGJcHHI7d. Ethan Hwang, Hossein Adeli, Wenxuan Guo, Andrew Luo, and Nikolaus Kriegeskorte. In silico mapping of visual categorical selectivity across the whole brain. In Advances in Neural Informa- tion Processing Systems, 2025. Nancy Kanwisher, J McDermott, and M M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11):4302–4311, 6 1997. doi: 10.1523/JNEUROSCI.17-11-04302.1997. Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32:4159–4171.e9, 10 2022. doi: 10.1016/j.cub.2022.08.009. David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge.Neural system identification for large populations separating “what”and “where”.In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.URL https://proceedings.neurips.c/paper_files/paper/2017/ file/8c249675aea6c3cbd91661bbae767f1-Paper.pdf. Leo Kozachkov, Ksenia V. Kastanenka, and Dmitry Krotov. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120(34):e2219150120, August 2023. doi: 10.1073/pnas.2219150120. Benjamin Lahner, Kshitij Dwivedi, Polina Iamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, SouYoung Jin, N. Apurva Ratan Murty, Kendrick Kay, Aude Oliva, and Radoslaw Cichy.Modeling short visual events through the BOLD moments video fMRI dataset and metadata. Nature Communications, 15: 6241, 2024.doi: 10.1038/s41467-024-50310-3.URL https://doi.org/10.1038/ s41467-024-50310-3. Yizhuo Lu, Changde Du, Chong Wang, Xuanliu Zhu, Liuyun Jiang, and Huiguang He. Animate your thoughts: Decoupled reconstruction of dynamic natural vision from slow brain activity. arXiv preprint arXiv:2405.03280, 2024. Andrew F. Luo, Margaret M. Henderson, Leila Wehbe, and Michael J. Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Konstantin-Klemens Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay Jagadish, Eric Wang, Edgar Y. Walker, Santiago A Cadena, Taliah Muhammad, Erick Cobos, Andreas S. Tolias, Alexander S Ecker, and Fabian H. Sinz. Generalization in data-driven models of primary vi- sual cortex. In International Conference on Learning Representations, 2021. URL https: //openreview.net/forum?id=Tp7kI90Htd. 12 Emalie McMahon, Michael F. Bonner, and Leyla Isik. Hierarchical organization of social action fea- tures along the lateral visual pathway. Current Biology, 33(23):5035–5047.e8, December 2023. doi: 10.1016/j.cub.2023.10.015. Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fMRI. NeuroImage, 56(2):400–410, May 2011. doi: 10.1016/j.neuroimage.2010.07.073. Maxime Oquab, Timoth ́ e Darcet, Th ́ eo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. David Pitcher and Leslie G. Ungerleider. Evidence for a third visual pathway specialized for social perception. Trends in Cognitive Sciences, 25:100–110, 2021. David Pitcher, Daniel D. Dilks, Rebecca R. Saxe, Christina Triantafyllou, and Nancy Kanwisher. Differential selectivity for dynamic versus static information in face-selective cortical regions. NeuroImage, 56(4):2356–2363, June 2011. doi: 10.1016/j.neuroimage.2011.03.067. Jacob S. Prince, Ian Charest, Jan W. Kurzawski, John A. Pyles, Michael J. Tarr, and Kendrick N. Kay. Improving the accuracy of single-trial fMRI response estimates using GLMsingle. eLife, 11, 2022. doi: 10.7554/ELIFE.77599. Subha Nawer Pushpita and Leila Wehbe. Two cortical mechanisms of audiovisual processing in the human brain. bioRxiv, 2025. doi: 10.1101/2025.11.05.686819. URL https://w. biorxiv.org/content/early/2025/11/06/2025.11.05.686819. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning (ICML), p. 8748–8763, 2021. N.A. Ratan Murty, Pouya Bashivan, Alex Abate, James J. DiCarlo, and Nancy Kanwisher. Com- putational models of category-selective brain regions enable high-throughput tests of selectivity. Nature Communications, 12(5540), 2021. doi: 10.1038/s41467-021-25409-6. Leila Reddy and Nancy Kanwisher. Category selectivity in the ventral visual pathway confers robustness to clutter and diverted attention. Current Biology, 17(23):2067–2072, 2007. doi: 10.1016/j.cub.2007.10.043. John H. Reynolds and David J. Heeger. The normalization model of attention. Neuron, 61(2): 168–85, January 2009. doi: 10.1016/j.neuron.2009.01.002. Nicole C. Rust and James J. DiCarlo. Selectivity and tolerance (“Invariance”) both increase as visual information propagates from cortical area V4 to IT. Journal of Neuroscience, 30(39): 12978–12995, September 2010. doi: 10.1523/JNEUROSCI.0179-10.2010. Gabriel H. Sarch, Michael J. Tarr, Katerina Fragkiadaki, and Leila Wehbe. Brain dissection: fMRI- trained networks reveal spatial selectivity in the processing of natural images. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Christina Sartzetaki, Gemma Roig, Cees G. M. Snoek, and Iris Groen. One hundred neural networks and brains watching videos: Lessons from alignment. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https://openreview.net/forum?id= LM4PYXBId5. Alexander Schaefer, Ru Kong, Evan M. Gordon, Timothy O. Laumann, Xi-Nian Zuo, Avram J. Holmes, Simon B. Eickhoff, and B. T. Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cerebral Cortex, 28(9):3095–3114, 2018. doi: 10.1093/cercor/bhx179. Martin Schrimpf, Jonas Kubilius, Michael J Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413–423, 2020. doi: 10.1016/j.neuron.2020.07.040. 13 John T. Serences and Sameer Saproo. Computational advances towards linking BOLD and behavior. Neuropsychologia, 50:435–446, 3 2012. doi: 10.1016/j.neuropsychologia.2011.07.013. Oriane Sim ́ eoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ̈ el Ramamonjisoa, et al. DINOv3. arXiv preprint arXiv:2508.10104, 2025. Ghislain St-Yves and Thomas Naselaris. The feature-weighted receptive field: an interpretable encoding model for complex feature spaces. NeuroImage, 180:188–202, 10 2018. doi: 10.1016/ J.NEUROIMAGE.2017.06.035. Yingtian Tang, Abdulkadir Gokce, Khaled Jedoui Al-Karkari, Daniel Yamins, and Martin Schrimpf. Diverse perceptual representations across visual pathways emerge from a single objective. bioRxiv, 2025.doi: 10.1101/2025.07.22.664908.URL https://w.biorxiv.org/ content/early/2025/10/08/2025.07.22.664908. Yingtian Tang, Sogand Salehi, Ming Zhou, Amir Zamir, Leyla Isik, and Martin Schrimpf. NEvo: Neural-guided evolutionary video synthesis for dynamic visual selectivity, 2026. URL https: //arxiv.org/abs/2607.02317. Leslie G. Ungerleider and Mortimer Mishkin. Two cortical visual systems. In David J. Ingle, Melvyn A. Goodale, and Richard J. W. Mansfield (eds.), Analysis of Visual Behavior. MIT Press, 1982. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017. Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe.Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.Nature Machine Intelligence, 5(12):1415–1426, 11 2023a.doi: 10.1038/s42256-023-00753-y. Liang Wang, Ryan E. B. Mruczek, Michael J. Arcaro, and Sabine Kastner. Probabilistic maps of visual topography in human cortex. Cerebral Cortex, 25(10):3911–3931, 2015. doi: 10.1093/ cercor/bhu277. Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE V2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14549– 14560, 2023b. Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading sub- processes. PLOS ONE, 9:e112575, 11 2014. doi: 10.1371/JOURNAL.PONE.0112575. Daniel L. Yamins and James J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3):356–365, March 2016. doi: 10.1038/n.4244. Jacob Yeung, Andrew F. Luo, Gabriel Sarch, Margaret M. Henderson, Deva Ramanan, and Michael J. Tarr. Reanimating images using neural representations of dynamic stimuli. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 5331–5343, 6 2025. Jingyang Zhou, Noah C. Benson, Kendrick N. Kay, and Jonathan Winawer. Compressive temporal summation in human visual cortex. Journal of Neuroscience, 38(3):691–709, January 2018. doi: 10.1523/JNEUROSCI.1724-17.2017. Davide Zoccolan, David D. Cox, and James J. DiCarlo. Multiple object response normalization in monkey inferotemporal cortex. Journal of Neuroscience, 25(36):8150–8164, September 2005. doi: 10.1523/JNEUROSCI.2058-05.2005. 14 6APPENDIX 6.1CHOICE OF PARCEL-BASED QUERIES We assign one query to each parcel of the Schaefer 1000-parcel atlas (Section 3.2; Appendix 6.2). The number of queries sets a trade-off. With one query per voxel, each attention map would be estimated from a single noisy voxel; with one query per large region, a single map would be imposed on a heterogeneous population, and the regions would have to be defined in advance. Parcels sit between these extremes. Every modeled voxel belongs to exactly one parcel query, so the whole measured volume is predicted, with a single additional query for voxels without a parcel label. The parcels are fine enough for differences within and between functional regions to emerge in the attention maps. Because the atlas is defined independently of our fMRI data and functional localizers, any category selectivity that emerges in the attention maps (Appendix 6.13) is not built into the model. Finally, with about 430 queries per subject, each attention map is shared by many voxels and is therefore estimated from many voxels’ responses. This follows the observation that finer query granularity improves cross-attention encoding models (Adeli et al., 2026), and the use of one query per Schaefer parcel in a whole-brain cross-attention encoder for static images (Hwang et al., 2025). Here we use the parcels in volumetric space for video, with attention over both space and time. 6.2DATASET AND PREPROCESSING DETAILS Preprocessing and voxel selection. We use the BMD release recommended by the dataset authors, preprocessed with fMRIPrep (Esteban et al., 2019) in MNI152NLin2009cAsym volume space at 2.5 m. We model all voxels in the union of the dataset’s group-level mask of cortex with reliable responses to the videos (“BMDGeneral” mask) and the 23 functional ROIs. The single-trial betas were estimated by the dataset authors with GLMsingle and z-scored per voxel within each scanning session. After averaging over the repeats of each clip, we z-score each voxel again separately within the training and the test clips. The second normalization puts training and test targets on the same scale, since the test responses are averaged over more repeats (10 vs. 3) and are therefore less noisy. The prediction target is thus a single repeat-averaged response per voxel and clip; we do not model the response time course within a clip. The noise ceiling released with the dataset follows the procedure of the Natural Scenes Dataset (Allen et al., 2022). Parcellation. We use the volumetric MNI152 version of the Schaefer atlas, resample it to the BMD voxel grid with nearest-neighbor interpolation and intersect it with each subject’s voxel mask. Each parcel with at least one voxel in the mask becomes a query, with the two hemispheres parcellated separately, and the voxels without a parcel label (mostly subcortical and edge voxels) are assigned to one additional query. Every modeled voxel therefore belongs to exactly one query. Functional ROIs. The 23 ROIs are V1v, V1d, V2v, V2d, V3v, V3d, V3ab, hV4, IPS0 and IPS1- 2-3 from the retinotopic atlas of Wang et al. (2015); MT, 7AL, BA2, PFop and PFt from the HCP multimodal parcellation (Glasser et al., 2016); and EBA, LOC, FFA, OFA, STS, PPA, RSC and TOS from the subject-specific functional localizer (bodies > objects for EBA; objects > scrambled objects for LOC; faces > objects for FFA, OFA and STS; scenes > objects for PPA, RSC and TOS). 6.3IMPLEMENTATION DETAILS Backbone tokens. We use the V-JEPA 2 ViT-g/16 model with 384 px, 64-frame input. From each 3 s clip we sample 16 frames uniformly and repeat each frame four times to fill the 64-frame input. The 2× 16× 16 tubelets of the model give a 32× 24× 24 token grid, and we average each pair of adjacent temporal tokens (which cover the same repeated frame), giving 16× 24× 24 tokens of dimension D = 1408. Each token therefore covers one sampled frame (≈0.19 s) and a 16× 16 px patch. Model size. For a typical subject the joint model has about 47M parameters, of which about 0.6M are queries, 21.8M are the shared attention and FFN block, and 24.5M are the voxel readouts. Data splits and ensembling. The 102 test clips are held out and used only for the final evaluation. For each of the 10 ensemble members, the 1,000 training clips are split at random into 900 clips 15 for training and 100 for validation, with a different split and initialization for each member, and the validation clips are used only for early stopping. Because each training clip is held out for validation in about one of the 10 members, the ensemble as a whole uses all training clips. All models, including the ridge baseline, use the same 10 splits. Optimization. We use AdamW with a learning rate of 10 −3 , halved every 3 epochs, weight decay 0.05, batches of 8 clips, gradient-norm clipping at 1 and mixed precision. The queries are initialized fromN (0, 0.02 2 ), and attention dropout of 0.1 is applied during training. Each member is trained for up to 12 epochs with early stopping on validation R 2 (patience 4), and the weights from the best validation epoch are restored (typically the third epoch). All hyperparameters are identical for the joint and constrained models, for all backbones and for all subjects. Ridge regression. We chose mean pooling for the ridge baseline because it predicted more accu- rately than ridge regression on the flattened token grid after dimensionality reduction with PCA. Features are standardized using each member’s training clips, and the penalty of each voxel is cho- sen from six values between 10 2 and 10 7 (log-spaced) by efficient leave-one-out cross-validation on the member’s 900 training clips. 6.4ROI CLUSTERING FOR THE REGION GROUPS To group the ROIs by function rather than by a priori anatomical assignment, we clustered the 23 ROIs by their responses to the videos. For each subject, we averaged the z-scored responses over each ROI’s reliable voxels (noise ceiling ≥ 5%), giving a matrix of 23 ROIs × 1,102 clips, and correlated the ROIs across clips. The correlation matrices were Fisher-z averaged over the 10 subjects and hierarchically clustered (distance 1− r, average linkage). Cutting the dendrogram into four clusters gives an early visual cluster (V1–V3, V3ab and hV4), a lateral/face/body cluster (OFA, FFA, EBA, MT, LOC and STS), a scene cluster (RSC, PPA, TOS, IPS0 and IPS1-2-3) and a parietal cluster (PFt, BA2, PFop and 7AL). This departs from a conventional ventral/lateral/dorsal division: FFA, OFA and LOC cluster with EBA, MT and STS rather than with the scene-selective ROIs, and IPS0 and IPS1-2-3 merge with each other first (distance≈0.25) and then join the scene ROIs. The region groups used in the main text follow these clusters except that IPS0 and IPS1-2-3 are placed in the parietal group because they lie in posterior parietal cortex; this exception is not data- driven. With the clustering assignment instead (both IPS ROIs in the scene group), only the scene and parietal groups change, and the joint model still outperforms the factorized, spatial, mean-pool and ridge models in both (two-sided paired t-tests across the 10 subjects, Benjamini–Hochberg FDR-corrected over these 8 tests). The differences in signed r 2 are +0.0036, +0.0029, +0.0054 and +0.0073 in the scene group (t(9) = 4.08, 2.50, 3.69 and 4.70; p FDR = 0.007, 0.034, 0.010 and 0.005) and +0.0040, +0.0035, +0.0059 and +0.0206 in the parietal group (t(9) = 3.47, 2.95, 2.64 and 6.87; p FDR = 0.011, 0.022, 0.031 and < 0.001). In the scene-selective ROIs alone (PPA, RSC and TOS), the joint model outperforms the factorized model in every subject (+0.0041 signed r 2 , t(9) = 6.16, p FDR < 0.001, 10/10 subjects) but matches ridge regression on mean-pooled features (+0.0007, t(9) = 0.42, p FDR = 0.75, 8/10; two-sided paired t-tests, FDR-corrected over the 24 tests of Table 3). The advantage over ridge in the scene group of the clustering assignment therefore comes from the IPS ROIs, where the joint model gains substantially over ridge (+0.018 signed r 2 in both IPS0 and IPS1-2-3; t(9) = 5.34 and 5.55, both p < 0.001, two-sided paired t-tests, uncorrected; 10/10 subjects). This is consistent with scene-selective cortex responding to the global layout of a scene, which is already captured by features averaged over the whole clip, so that routing to specific locations and moments adds little for these ROIs. 6.5BACKBONE AND LAYER SELECTION We chose the backbone and layer with voxelwise ridge regression on mean-pooled features from each layer of several video and image models, fit and evaluated in the same way as the ridge baseline (Section 3.3). Table 2 gives the best layer of each backbone and its whole-brain R 2 on the test clips, averaged over all voxels in each subject, then over the 10 subjects. The V-JEPA 2 models are the most accurate, and the three V-JEPA 2 sizes are nearly tied. 16 Table 2: Best layer of each backbone in the ridge-regression layer sweep (mean-pooled features; test R 2 , mean over all voxels in 10 subjects). BackboneBest layer R 2 V-JEPA 2 ViT-g (Assran et al., 2025)32.134 V-JEPA 2 ViT-H24.133 V-JEPA 2 ViT-L18.132 V-JEPA 2 ViT-L, fine-tuned on SSv218.125 VideoMAE v2 ViT-g (Wang et al., 2023b)20.109 V-JEPA 2.1 ViT-g [cite]44.108 Perception Encoder Spatial-L (Bolya et al., 2025)20.100 Perception Encoder Spatial-G38.098 DINOv3 ViT-7B, per frame (Sim ́ eoni et al., 2025)32.092 6.6SUBJECT-LEVEL TESTS FOR FIGURE 2 Table 3: Joint model minus each comparison model in region-mean signed r 2 on the 102 test clips (mean ± SEM over 10 subjects), with two-sided paired t-tests across subjects (df = 9). p is un- corrected; p FDR is Benjamini–Hochberg corrected over all 24 tests; values below 0.05 are in bold. Positive ∆ indicates that the joint model performed better. “Joint better” column indicates how many individual subjects have a numerical advantage for the joint model in each comparison. d z : mean divided by the standard deviation of the per-subject differences. RegionComparison∆ signed r 2 t(9)p FDR Joint betterd z All voxelsfactorized+0.0043± 0.00085.394.4e-048.7e-0410/101.71 spatial+0.0040± 0.00075.852.4e-045.8e-0410/101.85 mean pool+0.0103± 0.00176.161.7e-044.5e-0410/101.95 ridge+0.0162± 0.00198.521.3e-051.6e-0410/102.69 Early visualfactorized−0.0000± 0.0010-0.030.9760.9765/10-0.01 spatial+0.0000± 0.00090.040.9660.9764/100.01 mean pool+0.0122± 0.00254.849.2e-040.0019/101.53 ridge+0.0108± 0.00283.830.0040.0069/101.21 Lateral / face / bodyfactorized+0.0084± 0.00136.601.0e-043.4e-0410/102.09 spatial+0.0086± 0.00117.792.7e-051.6e-0410/102.46 mean pool+0.0145± 0.00216.807.9e-053.2e-0410/102.15 ridge+0.0216± 0.002210.043.5e-068.3e-0510/103.17 Scenefactorized+0.0041± 0.00076.161.7e-044.5e-0410/101.95 spatial+0.0029± 0.00112.720.0240.0278/100.86 mean pool+0.0033± 0.00113.070.0130.0178/100.97 ridge+0.0007± 0.00170.420.6870.7498/100.13 Parietalfactorized+0.0037± 0.00093.990.0030.0059/101.26 spatial+0.0034± 0.00122.930.0170.0209/100.93 mean pool+0.0067± 0.00223.110.0130.01710/100.98 ridge+0.0197± 0.00286.976.5e-053.1e-0410/102.21 Outside ROIsfactorized+0.0040± 0.00084.987.6e-040.00110/101.58 spatial+0.0035± 0.00074.987.6e-040.00110/101.58 mean pool+0.0094± 0.00175.593.4e-047.4e-0410/101.77 ridge+0.0163± 0.00208.122.0e-051.6e-0410/102.57 6.7VOXELWISE ACCURACY AND DIFFERENCES IN EACH SUBJECT Figure 5 shows the voxelwise accuracy of the joint model, averaged over subjects and in each subject, and Figure 6 shows the difference between the joint model and each comparison model in each subject; Fig. 2A shows the average of these differences over subjects. All modeled voxels are shown, without a noise-ceiling threshold, on the fsaverage flatmap used in Fig. 2A. The single-subject maps 17 are noisier than the average, but the region-level advantage of the joint model in lateral/face/body regions holds in every subject (Table 3). 18 Mean over subjects S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 0.00.30.6 Signed r² (joint 3D) Figure 5: Voxelwise accuracy of the joint model (V-JEPA 2 ViT-g, layer 32; signed r 2 on the 102 test clips; all modeled voxels), averaged over the 10 subjects (top left) and in each subject, on the fsaverage flatmap. Values outside 0–0.6 are shown at the ends of the color scale. 19 Joint 3D − factorizedJoint 3D − spatialJoint 3D − uniformJoint 3D − ridge −0.080.000.08 Δ signed r² S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 Figure 6: Joint model minus the factorized, spatial and mean-pool (“uniform”) models and ridge regression on mean-pooled features in voxelwise signed r 2 , in each subject (rows), on the flatmap of Fig. 2A. All panels share one color scale; differences beyond±0.08 are shown at its ends. 20 6.8SPATIAL ROUTING COMPARED WITH MEAN POOLING AND RIDGE REGRESSION Cross-attention encoders of static images improve on linear encoding models by routing each re- gion’s readout to stimulus-dependent spatial locations (Adeli et al., 2026). Table 4 tests the video analogue of this result, the spatial model (temporal factor uniform, spatial factor routed), against the mean-pool model and against ridge regression on mean-pooled features, for every backbone and layer of the routing sweep (Appendix 6.11). The spatial model is more accurate than both in all regions for every configuration except in scene-selective cortex, where it is not significantly differ- ent from either for V-JEPA 2 layer 32 and VideoMAE v2 layer 20, and for the last stage of CLIP, where the spatial model is less accurate than ridge regression in four of the six regions and less accurate than mean pooling in lateral/face/body regions. Of the 96 tests, 83 favor the spatial model at p FDR < 0.05. 6.9CONTRIBUTIONS OF THE READOUT COMPONENTS Besides routing, the joint model differs from ridge regression on mean-pooled features in three ways: it is trained by gradient descent, it passes the tokens through value and output projections shared by all voxels, and it applies a feedforward network (FFN) to each parcel’s representation before the voxel readouts. To separate these components, we trained three further models on V-JEPA 2 ViT-g layer 32, with the same data splits, ensembling, optimization and early stopping as the joint model (Appendix 6.3): • Linear readout: the voxel readouts applied directly to the mean-pooled tokens and trained by gradient descent, the counterpart of ridge regression. • Uniform, no FFN: the mean-pool model without the FFN, so that every parcel’s represen- tation is the output projection of the mean of the value projections. Because both projec- tions are linear, this model is still linear in the mean-pooled tokens; it differs from the linear readout only in its parametrization. • Joint, no FFN: the joint model without the FFN. With ridge regression, the mean-pool model and the joint model, these form a sequence that adds one component at a time (Fig. 7, Table 5; two-sided paired t-tests across the 10 subjects, df = 9, FDR- corrected over the 30 tests of the table). Routing contributes the most. Without the FFN, joint routing improves on uniform attention in all six regions and in every subject (+0.0115 across all voxels; t(9) = 10.51, p FDR < 0.001), as much as it does with the FFN (joint minus mean pool, +0.0103; Table 3). The FFN adds less, with uniform attention (+0.0039; t(9) = 5.29, p FDR = 0.001) and with joint routing (+0.0026; t(9) = 4.74, p FDR = 0.002), and in early visual cortex it adds nothing significant in either case. Gradient descent by itself does not explain the advantage of the attention models: the linear readout is less accurate than ridge regression in every region (−0.0045 across all voxels; t(9) = −7.11, p FDR < 0.001). The shared projections make up this deficit and more in lateral/face/body and parietal regions, but not in early visual or scene regions. 21 0.150 0.175 0.200 0.225 Signed r² All voxels 0.15 0.20 Early visual 0.20 0.25 0.30 Lateral/face/body ridge linear uniform, no FFN uniform joint, no FFN joint 3D 0.075 0.100 0.125 0.150 Signed r² Scene ridge linear uniform, no FFN uniform joint, no FFN joint 3D 0.10 0.15 0.20 Parietal ridge linear uniform, no FFN uniform joint, no FFN joint 3D 0.125 0.150 0.175 0.200 Outside ROIs ridge (closed form) linear readout (SGD) uniform, no FFN uniform joint 3D, no FFN joint 3D S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 Figure 7: Contributions of the readout components (V-JEPA 2 ViT-g, layer 32; region-mean signed r 2 on the 102 test clips; voxels with noise ceiling ≥ 5%). From left to right: ridge regression on mean-pooled features, a linear readout of the mean-pooled features trained by gradient descent, the mean-pool model without and with the FFN (“uniform”), and the joint model without and with the FFN. Open markers: models without an FFN. Colored lines: individual subjects; markers: mean± SEM over subjects. 22 Table 4: Spatial model minus the mean-pool model and minus ridge regression on mean-pooled features, in region-mean signed r 2 on the 102 test clips (∆ in units of 10 −3 , mean over 10 subjects; voxels with noise ceiling≥ 5%). t(9): two-sided paired t-test across subjects. p FDR : Benjamini– Hochberg corrected over all 96 tests in the table; values below 0.05 are in bold. n + : subjects in which the spatial model is better. Spatial− mean poolSpatial− ridge Region∆t(9)p FDR n + ∆t(9)p FDR n + V-JEPA 2 ViT-g, layer 32 of 40 All voxels+6.35.078.9e-49/10 +12.28.263.3e-510/10 Early visual+12.25.584.7e-410/10 +10.84.240.0039/10 Lateral/face/body +5.93.970.0049/10 +13.08.732.5e-510/10 Scene+0.50.370.7346/10 −2.2-1.110.3135/10 Parietal+3.32.380.0457/10 +16.38.093.7e-510/10 Outside ROIs+5.94.600.0029/10 +12.88.253.3e-510/10 V-JEPA 2 ViT-g, layer 4 of 40 All voxels+14.311.295.7e-610/10 +18.418.771.3e-610/10 Early visual+18.79.781.2e-510/10 +18.211.804.9e-610/10 Lateral/face/body +14.79.801.2e-510/10 +21.214.052.3e-610/10 Scene+10.45.883.4e-410/10 +14.18.343.2e-510/10 Parietal+7.98.213.4e-510/10 +12.211.814.9e-610/10 Outside ROIs+14.49.941.2e-510/10 +18.916.601.4e-610/10 VideoMAE v2 ViT-g, layer 20 of 40 All voxels+14.88.912.3e-510/10 +24.610.906.9e-610/10 Early visual+20.66.361.9e-410/10 +23.45.227.3e-410/10 Lateral/face/body +19.612.414.0e-610/10 +34.516.201.4e-610/10 Scene+2.01.610.1528/10 +1.40.760.4835/10 Parietal+8.54.810.00110/10 +21.67.377.1e-510/10 Outside ROIs+13.67.277.8e-510/10 +23.710.001.1e-510/10 VideoMAE v2 ViT-g, layer 4 of 40 All voxels+19.38.253.3e-510/10 +25.69.451.5e-510/10 Early visual+32.46.401.9e-410/10 +30.65.763.8e-410/10 Lateral/face/body +22.48.702.5e-510/10 +33.212.434.0e-610/10 Scene+10.26.142.5e-410/10 +17.69.012.1e-510/10 Parietal+8.68.542.8e-510/10 +16.114.192.3e-610/10 Outside ROIs+17.37.218.2e-510/10 +23.98.682.5e-510/10 DINOv2 ViT-B/14, block 12 of 12 All voxels+22.910.717.2e-610/10 +27.19.851.2e-510/10 Early visual+32.08.522.8e-510/10 +31.06.791.3e-410/10 Lateral/face/body +27.213.792.3e-610/10 +34.214.332.3e-610/10 Scene+9.411.545.1e-610/10 +8.64.500.00210/10 Parietal+11.55.465.4e-410/10 +19.07.029.9e-510/10 Outside ROIs+22.09.591.4e-510/10 +26.48.812.4e-510/10 DINOv2 ViT-B/14, block 2 of 12 All voxels+23.010.777.1e-610/10 +25.513.752.3e-610/10 Early visual+51.611.754.9e-610/10 +40.110.587.7e-610/10 Lateral/face/body +20.410.807.1e-610/10 +29.117.661.3e-610/10 Scene+6.74.260.0039/10 +14.56.481.7e-410/10 Parietal+5.56.651.5e-410/10 +11.512.454.0e-610/10 Outside ROIs+20.88.832.4e-510/10 +23.612.014.9e-610/10 CLIP RN50, stage 4 of 4 All voxels+0.00.010.9915/10 −6.7-2.730.0263/10 Early visual+15.74.150.0039/10 +10.02.650.0298/10 Lateral/face/body −15.2-3.460.0081/10 −22.2-5.783.8e-40/10 Scene+6.54.200.00310/10 −5.3-4.140.0032/10 Parietal+2.60.960.3787/10 −2.9-1.300.2414/10 Outside ROIs+0.30.130.9105/10 −6.4-2.970.0183/10 CLIP RN50, stage 2 of 4 All voxels+28.79.291.7e-510/10 +41.911.625.1e-610/10 Early visual+50.77.486.5e-510/10 +54.87.566.0e-510/10 Lateral/face/body +35.012.714.0e-610/10 +53.815.142.0e-610/10 Scene+5.23.350.01010/10 +17.811.275.7e-610/10 Parietal+12.37.655.6e-510/10 +24.610.388.7e-610/10 Outside ROIs+25.48.043.9e-510/10 +39.110.956.9e-610/10 23 Table 5: Each model of Fig. 7 minus the model it extends, in region-mean signed r 2 (mean± SEM over 10 subjects), with two-sided paired t-tests across subjects (df = 9). p is uncorrected; p FDR is Benjamini–Hochberg corrected over all 30 tests; values below 0.05 are in bold. n + : subjects in which the first model is better. ContrastRegion∆ signedr 2 t(9)p FDR n + Linear readout (SGD)− ridgeAll voxels −0.0045± 0.0006-7.115.6e-054.2e-040/10 Early visual −0.0053± 0.0010-5.374.5e-040.0010/10 Lateral/face/body −0.0069± 0.0012-5.882.3e-040.0010/10 Scene−0.0055± 0.0010-5.265.2e-040.0010/10 Parietal−0.0021± 0.0008-2.530.0320.0391/10 Outside ROIs −0.0037± 0.0006-6.718.8e-054.4e-040/10 Uniform, no FFN− linear readoutAll voxels+0.0065± 0.00144.530.0010.00210/10 Early visual+0.0007± 0.00260.270.7900.7905/10 Lateral/face/body +0.0112± 0.00205.722.9e-040.00110/10 Scene−0.0035± 0.0028-1.240.2480.2654/10 Parietal+0.0115± 0.00225.334.8e-040.0019/10 Outside ROIs+0.0063± 0.00134.760.0010.00210/10 Uniform− uniform, no FFNAll voxels+0.0039± 0.00075.295.0e-040.0019/10 Early visual+0.0033± 0.00171.880.0930.1047/10 Lateral/face/body +0.0028± 0.00073.930.0030.00510/10 Scene+0.0063± 0.00134.700.0010.00210/10 Parietal+0.0036± 0.00103.510.0070.0098/10 Outside ROIs+0.0043± 0.00085.503.8e-040.0019/10 Joint, no FFN− uniform, no FFNAll voxels+0.0115± 0.001110.512.4e-063.3e-0510/10 Early visual+0.0161± 0.00246.778.2e-054.4e-0410/10 Lateral/face/body +0.0147± 0.001410.662.1e-063.3e-0510/10 Scene+0.0055± 0.00105.384.4e-040.00110/10 Parietal+0.0074± 0.00164.590.0010.00210/10 Outside ROIs+0.0103± 0.001010.093.3e-063.3e-0510/10 Joint− joint, no FFNAll voxels+0.0026± 0.00064.740.0010.0029/10 Early visual −0.0006± 0.0017-0.370.7200.7444/10 Lateral/face/body +0.0026± 0.00112.290.0480.0558/10 Scene+0.0041± 0.00162.620.0280.0359/10 Parietal+0.0029± 0.00093.340.0090.0119/10 Outside ROIs+0.0033± 0.00065.553.6e-040.00110/10 24 6.10ROUTING IN EACH FUNCTIONAL ROI Figure 8 repeats Fig. 2B for each of the 23 functional ROIs, and Table 6 compares the joint model with the factorized, spatial and mean-pool models and ridge regression in each ROI (two-sided paired t-tests across the 10 subjects, df = 9, Benjamini–Hochberg FDR-corrected over the 92 tests of the table). The ROIs follow the pattern of their region groups. In each of the eight early visual ROIs, the joint model does not differ significantly from the factorized or the spatial model, and it is better than mean pooling in seven of them. In the lateral/face/body group, it is better than the spatial model in all six ROIs and than the factorized model in all but OFA (p FDR = 0.054), with the largest gains over the factorized model in MT (+0.016; t(9) = 6.54, p FDR < 0.001) and EBA (+0.013; t(9) = 8.50, p FDR < 0.001). Among the scene-selective ROIs, the joint model is better than every comparison model in PPA, but not significantly better than any of them in TOS, and in RSC it is better only than the factorized model. Table 6: Joint model minus each comparison model in ROI-mean signed r 2 on the 102 test clips (∆ in units of 10 −3 , mean over 10 subjects; voxels with noise ceiling ≥ 5%). t(9): two-sided paired t-test across subjects. p FDR : Benjamini–Hochberg corrected over all 92 tests in the table; values below 0.05 are in bold. n + : subjects (of 10) in which the joint model is better. Joint− factorizedJoint− spatialJoint− mean poolJoint− ridge ROI∆t(9)p FDR n + ∆t(9)p FDR n + ∆t(9)p FDR n + ∆t(9)p FDR n + Early visual V1v −1.7 -1.260.2815 −1.7 -1.530.2033 +8.33.380.0178 +3.81.400.2406 V1d −0.8 -0.590.5904 −1.9 -2.260.0723 +13.94.300.0069 +12.03.630.0138 V2v −0.5 -0.400.7174 −1.5 -1.840.1304 +5.52.180.0806 +2.50.950.4027 V2d −1.8 -1.230.2904 −0.7 -0.680.5464 +18.35.920.0019 +18.05.380.0029 V3v+1.20.920.4126 +0.30.310.7725 +8.34.590.0049 +5.63.010.0268 V3d −1.6 -1.100.3394 +0.20.260.8036 +18.74.080.0078 +18.53.510.0159 V3ab+2.51.440.2297 +2.71.620.1778 +9.53.220.0218 +11.73.340.0188 hV4+3.01.950.1128 +3.92.250.0728 +16.05.140.00210 +14.03.470.01610 Lateral / face / body MT+15.66.549.8e-410 +12.36.220.00110 +24.76.270.00110 +30.78.312.8e-410 EBA+12.78.502.8e-410 +13.38.192.8e-410 +22.35.730.00110 +27.97.713.4e-410 FFA+5.13.290.0198 +5.33.700.0129 +9.14.320.0069 +16.68.162.8e-410 OFA+3.72.490.0548 +4.74.060.00710 +4.83.280.0198 +12.74.150.0079 LOC+9.44.860.0039 +9.96.300.00110 +16.96.170.00110 +24.08.872.8e-410 STS+4.75.000.00310 +4.45.680.00110 +6.25.810.00110 +14.0 11.679.0e-510 Scene PPA+3.54.540.00510 +3.53.020.0267 +6.74.410.0059 +7.33.440.0169 RSC+4.53.930.0099 +2.72.540.0518 +1.30.960.4025 −3.4 -1.720.1564 TOS+2.01.040.3656 +1.10.600.5877 +4.71.190.3038 +7.42.410.0608 Parietal IPS0+4.11.930.1147 +4.62.980.0279 +11.44.230.00610 +17.95.340.00210 IPS1-2-3 +1.71.320.2645 +1.20.740.5106 +5.31.980.1087 +17.55.550.00210 7AL+5.33.720.0129 +1.91.330.2615 +4.62.530.0518 +21.88.042.8e-410 BA2+3.32.350.0668 +3.63.140.0229 +5.72.250.0727 +18.66.270.00110 PFop+3.52.670.0429 +2.92.970.0279 +5.63.170.0228 +17.87.493.8e-410 PFt+4.53.110.0238 +5.52.830.03310 +7.52.280.0728 +25.24.650.00410 25 V1vV1dV2vV2dV3vV3dV3abhV4 0.0 0.1 0.2 0.3 0.4 0.5 Signed r² Early visual MTEBAFFAOFALOCSTSPPARSCTOS 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Signed r² Lateral / face / body Scene IPS0IPS1-2-37ALBA2PFopPFt 0.0 0.1 0.2 0.3 0.4 0.5 Signed r² Parietal ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling Figure 8: Region-mean signedr 2 of every routing configuration, the joint model and ridge regression in each functional ROI (V-JEPA 2 ViT-g, layer 32; 102 test clips; voxels with noise ceiling≥ 5%), grouped by region as in Fig. 2. Large dots: mean ± SEM over subjects; small dots: individual subjects; grey band: noise ceiling (mean± SEM). 26 6.11ROUTING SWEEP Table 7 gives the region-mean signed r 2 of every routing configuration and of ridge regression for two layers of each of four backbones, all fit and evaluated as for V-JEPA 2 layer 32 (Section 3.3). Figure 9 summarizes the table as the difference between the joint model and the factorized, spatial and mean-pool models and ridge regression. Figures 10–16 show every configuration other than V-JEPA 2 layer 32 in the format of Fig. 2B,C. 27 All voxels Early visual Lateral /face /body Scene Parietal Outside ROIs −0.03 −0.02 −0.01 0.00 0.01 Δ signed r² Joint 3D − factorized All voxels Early visual Lateral /face /body Scene Parietal Outside ROIs −0.03 −0.02 −0.01 0.00 0.01 Joint 3D − spatial All voxels Early visual Lateral /face /body Scene Parietal Outside ROIs 0.00 0.02 0.04 Joint 3D − uniform All voxels Early visual Lateral /face /body Scene Parietal Outside ROIs −0.02 0.00 0.02 0.04 Joint 3D − ridge V-JEPA 2-g, layer 32 V-JEPA 2-g, layer 4 VideoMAE v2-g, layer 20 VideoMAE v2-g, layer 4 DINOv2-B, block 12 DINOv2-B, block 2 CLIP RN50, stage 4 CLIP RN50, stage 2 Figure 9: Joint model minus the factorized, spatial and mean-pool (“uniform”) models and ridge regression on mean-pooled features of the same layer, in region-mean signed r 2 on the 102 test clips, for every backbone and layer of the routing sweep (mean ± SEM over 10 subjects; voxels with noise ceiling≥ 5%). 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.03 0.04 0.05 0.06 Signed r² uniform spatial factorized joint 3D 0.04 0.06 0.08 uniform spatial factorized joint 3D 0.04 0.06 0.08 uniform spatial factorized joint 3D 0.02 0.04 0.06 0.08 uniform spatial factorized joint 3D 0.02 0.03 uniform spatial factorized joint 3D 0.03 0.04 0.05 0.06 V-JEPA 2-g, layer 4 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 10: Fig. 2B,C for V-JEPA 2 ViT-g, layer 4 of 40: region-mean signed r 2 of every routing configuration, the joint model and ridge regression (top), and of the mean-pool, spatial, factorized and joint models in each subject (bottom). 28 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.150 0.175 0.200 Signed r² uniform spatial factorized joint 3D 0.15 0.20 uniform spatial factorized joint 3D 0.20 0.25 0.30 uniform spatial factorized joint 3D 0.075 0.100 0.125 0.150 uniform spatial factorized joint 3D 0.10 0.15 uniform spatial factorized joint 3D 0.125 0.150 0.175 0.200 VideoMAE v2-g, layer 20 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 11: As Fig. 10, for VideoMAE v2 ViT-g, layer 20 of 40. 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.06 0.08 0.10 0.12 Signed r² uniform spatial factorized joint 3D 0.10 0.15 uniform spatial factorized joint 3D 0.075 0.100 0.125 0.150 uniform spatial factorized joint 3D 0.04 0.06 0.08 0.10 uniform spatial factorized joint 3D 0.03 0.04 0.05 0.06 uniform spatial factorized joint 3D 0.06 0.08 0.10 0.12 VideoMAE v2-g, layer 4 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 12: As Fig. 10, for VideoMAE v2 ViT-g, layer 4 of 40. 29 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.100 0.125 0.150 0.175 Signed r² uniform spatial factorized joint 3D 0.10 0.15 uniform spatial factorized joint 3D 0.20 0.25 uniform spatial factorized joint 3D 0.075 0.100 0.125 0.150 uniform spatial factorized joint 3D 0.05 0.10 0.15 uniform spatial factorized joint 3D 0.100 0.125 0.150 0.175 DINOv2-B, block 12 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 13: As Fig. 10, for DINOv2 ViT-B/14, block 12 of 12. 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.04 0.06 0.08 Signed r² uniform spatial factorized joint 3D 0.05 0.10 uniform spatial factorized joint 3D 0.06 0.08 0.10 uniform spatial factorized joint 3D 0.02 0.04 0.06 0.08 uniform spatial factorized joint 3D 0.02 0.03 0.04 uniform spatial factorized joint 3D 0.04 0.06 0.08 DINOv2-B, block 2 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 14: As Fig. 10, for DINOv2 ViT-B/14, block 2 of 12. 30 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.10 0.12 0.14 Signed r² uniform spatial factorized joint 3D 0.075 0.100 0.125 0.150 uniform spatial factorized joint 3D 0.15 0.20 0.25 uniform spatial factorized joint 3D 0.050 0.075 0.100 0.125 uniform spatial factorized joint 3D 0.050 0.075 0.100 0.125 uniform spatial factorized joint 3D 0.10 0.12 0.14 CLIP RN50, stage 4 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 15: As Fig. 10, for CLIP RN50, stage 4 of 4. 0.0 0.1 0.2 0.3 0.4 Signed r² All voxels Early visual Lateral / face / body SceneParietalOutside ROIs uniform spatial factorized joint 3D 0.075 0.100 0.125 0.150 Signed r² uniform spatial factorized joint 3D 0.10 0.15 uniform spatial factorized joint 3D 0.10 0.15 0.20 uniform spatial factorized joint 3D 0.050 0.075 0.100 uniform spatial factorized joint 3D 0.04 0.06 0.08 uniform spatial factorized joint 3D 0.075 0.100 0.125 CLIP RN50, stage 2 ridge (mean-pooled features) fix-T × unif-S unif-T × fix-S fix-T × fix-S route-T × unif-S route-T × fix-S fix-T × route-S uniform (unif-T × unif-S) spatial (unif-T × route-S) factorized (route-T × route-S) joint 3D noise ceiling S1S2S3S4S5S6S7S8S9S10mean ± SEM Figure 16: As Fig. 10, for CLIP RN50, stage 2 of 4. 31 Table 7: Region-mean signed r 2 on the 102 test clips (mean over 10 subjects; voxels with noise ceiling≥ 5%) for every routing configuration and backbone. The factorized models are labelled by their temporal factor (column groups) and spatial factor (U = uniform, F = fixed, R = routed), where a fixed factor is a learned, stimulus-independent distribution for each parcel. Temporal uniform with spatial uniform is the mean-pool model, temporal uniform with spatial routed is the spatial model, temporal routed with spatial uniform is the temporal model, and temporal routed with spatial routed is the factorized model. Ridge is ridge regression on the mean-pooled features of the same layer. The best model in each row is in bold. Temporal uniformTemporal fixedTemporal routed RegionRidgeUFRUFRUFR Joint V-JEPA 2 ViT-g (Assran et al., 2025), layer 32 of 40 All voxels.174 .180 .181.187 .181 .181 .187 .180 .180 .186.191 Early visual.178 .177 .178.189 .177 .177 .190 .178 .179 .189.189 Lateral/face/body.242 .249 .250.255 .250 .250 .255 .248 .249 .255.263 Scene.118 .115 .117.116 .117 .117 .116 .114 .114 .115.119 Parietal.109 .122 .122.125 .122 .122 .126 .121 .122 .125.129 Outside ROIs.165 .172 .172.178 .172 .172 .178 .171 .172 .177.181 V-JEPA 2 ViT-g, layer 4 of 40 All voxels.035 .039 .039.053 .039 .039 .052 .039 .040 .050.042 Early visual.046 .046 .047.064 .046 .046 .063 .048 .049 .060.048 Lateral/face/body.046 .053 .053.067 .053 .052 .066 .052 .053 .063.056 Scene.043 .046 .047.057 .047 .044 .056 .045 .046 .054.053 Parietal.013 .017 .017.025 .017 .017 .025 .018 .018 .023.020 Outside ROIs.030 .035 .035.049 .035 .035 .048 .035 .036 .046.038 VideoMAE v2 ViT-g (Wang et al., 2023b), layer 20 of 40 All voxels.148 .158 .158.173 .158 .159 .174 .156 .157 .172.173 Early visual.148 .151 .153.171 .150 .153 .173 .151 .152 .171.169 Lateral/face/body.206 .221 .221.240 .220 .221 .242 .218 .219 .239.243 Scene.119 .118 .119.120 .119 .120 .121 .119 .119 .118.120 Parietal.093 .106 .106.114 .107 .107 .115 .104 .104 .115.115 Outside ROIs.139 .149 .149.163 .149 .150 .164 .147 .148 .162.162 VideoMAE v2 ViT-g, layer 4 of 40 All voxels.067 .073 .074.092 .073 .074 .092 .074 .074 .089.089 Early visual.099 .097 .100.129 .097 .101 .129 .098 .099 .124.122 Lateral/face/body.078 .089 .090.111 .089 .089 .111 .089 .088 .107.108 Scene.057 .064 .065.075 .065 .064 .075 .064 .063 .075.073 Parietal.033 .040 .041.049 .041 .040 .049 .041 .041 .047.048 Outside ROIs.061 .067 .068.084 .067 .068 .084 .068 .068 .082.081 DINOv2 ViT-B/14 (Oquab et al., 2024), block 12 of 12 All voxels.125 .129 .130.152 .129 .130 .152 .130 .131 .151.151 Early visual.108 .107 .110.139 .107 .110 .139 .108 .112 .139.132 Lateral/face/body.185 .192 .193.219 .192 .194 .218 .193 .195 .218.222 Scene.110 .109 .109.118 .110 .111 .119 .110 .111 .119.115 Parietal.078 .085 .085.097 .086 .086 .096 .084 .084 .096.099 Outside ROIs.117 .121 .122.143 .121 .122 .143 .122 .123 .143.142 DINOv2 ViT-B/14, block 2 of 12 All voxels.042 .045 .045.068 .044 .045 .067 .045 .045 .067.060 Early visual.065 .053 .055.105 .053 .055 .104 .056 .058 .104.095 Lateral/face/body.050 .059 .059.079 .059 .059 .079 .059 .059 .079.073 Scene.048 .055 .055.062 .055 .055 .062 .054 .055 .061.055 Parietal.016 .022 .023.028 .022 .022 .028 .022 .022 .028.024 Outside ROIs.037 .040 .040.060 .039 .040 .060 .040 .040 .060.053 CLIP RN50 (Radford et al., 2021), stage 4 of 4 All voxels.132 .125 .124.125 .124 .125 .126 .123 .124 .126.126 Early visual.102 .097 .095.112 .095 .097 .113 .094 .095 .115.110 Lateral/face/body.205 .198 .196.183 .196 .197 .185 .197 .198 .184.191 Scene.107 .095 .097.102 .096 .097 .102 .094 .095 .102.099 Parietal.082 .077 .077.079 .076 .076 .080 .075 .075 .078.077 Outside ROIs.123 .117 .115.117 .115 .116 .118 .115 .115 .117.118 CLIP RN50, stage 2 of 4 All voxels.071 .084 .085.113 .085 .086 .113 .084 .086 .113.099 Early visual.076 .080 .083.130 .080 .083 .130 .082 .085 .130.099 Lateral/face/body.102 .121 .123.156 .122 .123 .156 .120 .122 .157.146 Scene.065 .078 .078.083 .078 .078 .083 .079 .079 .083.080 Parietal.035 .047 .047.059 .048 .047 .059 .047 .047 .059.053 Outside ROIs.065 .078 .079.104 .079 .080 .104 .079 .080 .104.091 32 6.12ADDITIONAL EXAMPLES OF ATTENTION MAPS Figures 17–19 show nine further test clips in the format of Fig. 3, for parcels in right MT (golfer, cat and weightlifter), left EBA (surfer, skier and child on stairs), left STS (toddlers and child in snow) and right FFA (eagle). We chose the clips by eye as examples of moving people and animals; they illustrate the attention maps and are not a quantitative test. For the golfer, the surfer, the skier, the toddlers and the child on stairs, the attention of the joint model stays on the moving person across frames, and for the cat it moves with the head and paw as the cat turns in its basket. The factorized model instead concentrates its attention on a few frames and, within them, on fixed parts of the frame that often include the background, such as the basket. For the weightlifter, the joint model attends mostly to the barbell as it is dropped and rolls, rather than to the lifter, whereas the factorized model spreads its attention over a large region on the left of the frame. In the eagle and child-in-snow clips, where the attended head moves little, the two models attend to similar locations. 33 t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Golfer Right MT S1 P556 1× 3× ≥5× × uniform t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Surfer Left EBA S7 P182 1× 3× ≥5× × uniform t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Toddlers Left STS S9 P377 Figure 17: Further examples of attention maps in the format of Fig. 3: joint model (upper two rows of each block) and factorized model (lower two rows) over all 16 frames of each clip, in units of the uniform level 1/N , from transparent at or below the uniform level to saturated at 5 times the uniform level. 34 t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Skier Left EBA S2 P042 1× 3× ≥5× × uniform t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Child in snow Left STS S8 P412 1× 3× ≥5× × uniform t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Eagle Right FFA S2 P502 1× 3× ≥5× × uniform Figure 18: Further examples of attention maps, as in Fig. 17. 35 t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Cat Right MT S1 P556 1× 3× ≥5× × uniform t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Weightlifter Right MT S1 P556 t = 0–7 t = 8–15 Joint 3D t = 0–7 t = 8–15 Factorized Child on stairs Left EBA S2 P042 1× 3× ≥5× × uniform Figure 19: Further examples of attention maps, as in Fig. 17. 36 6.13CATEGORY SELECTIVITY AND LOCALIZATION FROM ATTENTION MAPS All analyses of category selectivity use the attention maps of the joint model (V-JEPA 2 ViT-g, layer 32) on the 102 test clips, with one model per subject trained as described in Section 3.3. We assign a parcel to a functional ROI if at least 20% of its voxels fall in that ROI. Because each parcel has a single attention map, the whole parcel is used, so a parcel can belong to more than one ROI and may also contain voxels outside the ROI. Category masks. For each test clip we take 16 frames evenly spaced through the clip, one for each temporal position of the token grid, and segment them with SAM 3 (Carion et al., 2025), which returns all instances matching a text prompt; we keep detections with a confidence of at least 0.35. We use 28 prompts and merge them into mutually exclusive categories: face (face), body (person, hand), animal (animal, animal face), food (food), object (car, vehicle, ball, toy, musical instrument, tool, utensil, phone, book, furniture, bag, text, sign) and scene (building, tree, sky, grass, road, water, mountain, wall, floor). A pixel claimed by several categories is assigned to the first in this order. The order runs from the most to the least specific category, so that a face is not also counted as part of the person, and a hand holding a phone is not counted as an object. Pixels matched by no prompt are unlabeled. The masks are computed at 64× 64 pixels per frame and area-averaged onto the 24× 24 token grid, which gives the fraction M c (t,h,w)∈ [0, 1] of every token covered by category c. Because the categories are exclusive, these fractions sum to 1 over the categories and unlabeled. The unlabeled pixels are mostly scene surfaces that the scene prompts miss (for example snow, dirt paths, plain walls and courts), so when we score the known functional ROIs we merge scene and unlabeled into a single place category. Category selectivity of known functional ROIs. We measure where a parcel reads in a clip by the 92 tokens (the top 1% of the 16× 24× 24 grid) that it attends to most, the same fraction as the 2,000 most-attended pixels of each image used by Adeli et al. (2026). Restricting to the most-attended tokens measures where the parcel reads, rather than the diffuse tail of its attention that is spread over the whole clip. The IoU of category c is the intersection over union of these tokens with the binary mask of c (tokens at least half covered), as in analyses of spatial selectivity in fMRI-trained networks (Sarch et al., 2023). We average it over the clips in which c covers between 1% and 90% of the clip: when a category covers very few tokens, its IoU is determined by those few tokens and dominated by noise, and when it fills nearly the whole clip, any 92 tokens overlap it, so the IoU carries no information about where the parcel attends. The raw IoU mostly reflects the size of each category (Fig. 4b). A place mask covers about 6,500 tokens, so 92 tokens can reach an IoU of only about 0.014 even for a parcel that attends only to places, whereas a face mask covers about 550 tokens and is hard to hit. We therefore z-score the IoU across parcels: the score of a functional ROI for category c is the mean over its parcels minus the mean over all parcels, divided by the standard deviation across parcels. Because every parcel sees the same clips and masks, the size of each category is the same for all parcels and cancels, and the score asks whether the ROI puts more of its attention on the category than a typical parcel does. We report the raw and z-scored IoU for the eight category-selective ROIs (FFA, OFA, STS, EBA, PPA, RSC, TOS and LOC), and the raw IoU of all parcels for reference, for five categories (face, body, animal, object and place) as the mean ± SEM over subjects (Fig. 4b,c). Localizing category-selective regions from attention. Functional localizers contrast the responses of each voxel to different stimulus categories, for example faces and objects. We apply the same logic to the attention maps: for each parcel we measure how much attention it puts on each category in each clip, form the localizer contrasts from these measures, and ask which parcels show the contrast consistently across clips. Contrasts. In each clip, we compute the IoU of each parcel’s 92 most-attended tokens with the mask of each category, as above, and leave it undefined for a clip in which the category covers less than 1% or more than 90% of the clip. Here the scene category is the scene mask alone, without the unlabeled pixels. Within each subject, we z-score the IoU of each category across all clips and parcels where it is defined. The z-scoring puts the categories on a common scale, so that a contrast is not dominated by the category whose IoU is largest or varies most, which depends on its size. It also sets 0 to the average parcel on the average clip, so that a contrast measures a parcel’s preference relative to other parcels rather than a preference that all parcels share (for example, if faces draw attention from every parcel). From the z-scored IoU we form, in each clip, contrasts that mirror the localizers used to define the functional ROIs: face minus object for FFA, OFA and STS, and 37 body minus object for EBA. For PPA, RSC and TOS we contrast scene with the mean of face, body, animal and object. The scene localizer compares scenes with isolated objects, but in natural video every foreground item appears within a scene, so the analogous contrast is between scene and every kind of labeled foreground, not objects alone. A contrast is undefined in a clip if any of its terms is undefined; for the scene contrast, the mean is taken over the foreground categories that are defined. For each parcel and contrast, a one-sample t-test across clips then gives a t value, treating clips as independent replicates, as trials are in a localizer. Parcels with fewer than 5 clips in which the contrast is defined are left unscored (Fig. 4a). Scoring. To compare these attention-derived maps with the functional ROIs, we pool the parcels of the ROIs that each localizer defines (face: FFA, OFA and STS; body: EBA; scene: PPA, RSC and TOS), with parcels assigned to ROIs as above. Let K be the number of these ROI parcels. We take the K scored parcels with the highest t and measure the overlap as the fraction of them that belong to the ROIs (Fig. 4a shows the t map, the top-K parcels and the ROIs of each contrast for one subject, S8). Setting the size of the derived region equal to the size of the ROI avoids choosing a t threshold and makes the overlap equal to both the precision and the recall of the derived region. If the ranking were unrelated to ROI membership, the number of ROI parcels among the top K would follow a hypergeometric distribution, with an expected overlap equal to the fraction of scored parcels that belong to the ROIs (0.02–0.10). We therefore test the overlap in each subject with a one-sided hypergeometric test (uncorrected), and across subjects with a two-sided one-sample t-test of the overlap minus chance (df = 9), with p-values corrected over the three localizers with the Benjamini–Hochberg false discovery rate (FDR) procedure (Benjamini & Hochberg, 1995). As a threshold-free complement, we report the area under the ROC curve (AUC) of t for ROI versus non-ROI scored parcels, which measures the separation of the two groups over the whole ranking. Detailed results. The raw IoU mostly reflects the size of each category: faces, which are small, have the lowest IoU in most ROIs and in the average parcel, although the face IoU of FFA (0.0107) is well above that of all parcels (0.0063; Fig. 4b). After z-scoring (Fig. 4c), FFA, OFA and EBA put more attention than the average parcel on faces, bodies and animals (z ≈ 0.5–1.3) and less on places; because these categories usually appear together in the clips, an area that attends to one of them also tends to attend to the others. PPA and RSC show the reverse pattern (place z = 0.84 and 0.75, all foreground categories negative). LOC is high for faces, bodies and animals (z = 1.15–1.25), lower for objects (z = 0.34) and strongly negative for places (z = −1.24), consistent with a preference for foreground content in general. In STS and TOS, all categories are close to the average parcel (|z|≤ 0.28). In the localization, the overlap is significant in 5, 3 and 8 of the 10 individual subjects for the face, body and scene localizers (one-sided hypergeometric test, p < 0.05, uncorrected), and ROI parcels rank above other parcels across the whole ranking, not only at the top (AUC 0.61–0.77; Table 8). Fig. 4a shows the subject with the highest mean overlap across the three localizers (S8; face 0.41, body 0.08, scene 0.43). 6.14SPATIAL-PRIOR CONTROL FOR THE LOCALIZATION The IoU of a parcel with a category can differ between parcels even if no parcel’s attention follows the content of the clips, because each parcel’s attention has a spatial prior and categories occupy typical positions in the frame. The IoU is computed from the most-attended tokens, and which tokens are most attended depends on the prior. Z-scoring does not remove this confound: unlike the size of a category, which is the same for all parcels and cancels, the prior differs between parcels. To measure this contribution, we repeat the localization with each clip’s attention scored against the category masks of a different test clip. The pairing is a single random permutation of the 102 test clips in which no clip is paired with itself, drawn once and used for every parcel and subject. The mismatched masks keep each parcel’s spatial prior and the typical layout of the categories, but remove any correspondence between the attention and the content of the clip. Table 8 compares the localization with the correct and the mismatched masks; the two are compared with a two-sided paired t-test across subjects (df = 9), FDR-corrected over the three localizers. For faces, the mismatched masks give an overlap at chance (0.10, chance 0.09; t(9) = 0.25, p FDR = 0.81), and the correct masks give a significantly larger overlap (t(9) = 3.13, p FDR = 0.036), so the localization of the face-selective ROIs depends on the content of each clip. For bodies, the mismatched-mask overlap (0.08) is not significantly above chance (t(9) = 2.00, p FDR = 0.11), but the correct masks do not exceed it significantly either (t(9) = 1.70, p FDR = 0.18). For scenes, 38 the mismatched masks give nearly the same overlap as the correct masks (0.26 vs. 0.28; t(9) = 1.11, p FDR = 0.30) and are themselves well above chance (t(9) = 3.74, p FDR = 0.014), so the localization of the scene-selective ROIs mostly reflects the parcels’ spatial priors rather than attention to the scenery in each clip. Table 8: Localization of category-selective ROIs from z-scored IoU contrasts with the correct masks and with the masks of a different clip (joint model, 10 subjects, 102 test clips; face: FFA, OFA and STS; body: EBA; scene: PPA, RSC and TOS). Overlap: fraction of the top-K parcels that belong to the ROIs, mean (SD) over subjects; chance is the fraction of scored parcels in the ROIs. t(9) and p FDR : two-sided one-sample t-test of overlap minus chance across subjects, Benjamini– Hochberg corrected over the three localizers within each condition. Subjects: number with a one- sided hypergeometric p < 0.05 (uncorrected). Correct− mismatched: two-sided paired t-test of the overlaps across subjects, FDR-corrected over the three localizers. Correct masksMismatched masksCorrect− mismatched Localizer ChanceOverlapt(9)p FDR AUC SubjectsOverlapt(9)p FDR Subjectst(9)p FDR Face0.09 0.19 (0.11)2.810.0210.615/10 0.10 (0.05)0.250.810/103.13 0.0120.036 Body0.02 0.14 (0.09)4.020.0050.763/10 0.08 (0.09)2.000.113/101.700.120.18 Scene0.10 0.28 (0.12)4.850.0030.778/10 0.26 (0.14)3.740.0148/101.110.300.30 39

Instant research discovery