Paper deep dive
Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE
Angeliki Papathanasiou, Jascha Achterberg, Thomas E. Nichols, Rui Ponte Costa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 6/20/2026, 7:38:16 AM
Summary
The paper introduces MED-VAE (Multi-Encoder-Decoder Variational Autoencoder), a novel framework for cross-subject neural alignment that does not require shared stimuli. Unlike traditional methods like SRM or Hyperalignment which rely on overlapping data, MED-VAE uses a pretrained Artificial Neural Network (ANN), specifically ResNet-50, as a computational scaffold to anchor subject-specific fMRI representations into a common latent space. The method demonstrates superior semantic organization, better generalization to held-out stimuli, and improved cross-subject neural prediction and image decoding compared to classical alignment baselines.
Entities (7)
Relation Signals (4)
MED-VAE → evaluatedon → Natural Scenes Dataset
confidence 100% · Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces
MED-VAE → usesasscaffold → ResNet-50
confidence 100% · anchoring representations to a common scaffold provided by a pretrained ANN... ResNet-50 features
ResNet-50 → extractsfeaturesfrom → Natural Scenes
confidence 95% · ResNet-50 activations (pretrained on ImageNet) were extracted for all the images viewed by subjects.
MED-VAE → outperforms → Shared Response Model
confidence 90% · achieving higher cross-subject alignment than common methods... MED-VAE, by contrast, is trained on each subject's full non-shared responses.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Aligning neural activity across subjects offers the promise of discovering shared computational principles and generalizable decoders. However, traditional alignment methods require shared stimuli across subjects, a constraint that limits applicability to naturalistic paradigms with limited or non-overlapping data. We introduce a Multi-Encoder-Decoder Variational Autoencoder (MED-VAE) that achieves cross-subject alignment without shared stimuli by anchoring representations to a common scaffold provided by a pretrained ANN. Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces with superior semantic organisation, achieving higher cross-subject alignment than common methods while maintaining robust generalisation to held-out stimuli where traditional methods degrade. Reconstructing from these common spaces back to each subject's original neural space, MED-VAE preserves equal stimulus-driven signal in its cross-subject latent space. Finally, we show that this superior alignment directly enables cross-subject neural prediction, as demonstrated via cross-subject image decoding. In summary, we introduce a framework to identify generalisable common subspaces for cross-subject predictions and downstream tasks, demonstrated here for visual cortex responses to static images.
Tags
Links
- Source: https://arxiv.org/abs/2606.15989v1
- Canonical: https://arxiv.org/abs/2606.15989v1
Trouble viewing inline? Open PDF directly →
Full Text
95,312 characters extracted from source content.
Expand or collapse full text
10.32470/7dlktq8 Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE Angeliki Papathanasiou1 Jascha Achterberg1 Thomas E. Nichols2 Rui Ponte Costa1 1 Centre for Neural Circuits and Behaviour Department of Physiology Anatomy and Genetics University of Oxford 2 Big Data Institute Nuffield Department of Medicine University of Oxford Abstract Aligning neural activity across subjects offers the promise of discovering shared computational principles and generalizable decoders. However, traditional alignment methods require shared stimuli across subjects, a constraint that limits applicability to naturalistic paradigms with limited or non-overlapping data. We introduce a Multi-Encoder-Decoder Variational Autoencoder (MED-VAE) that achieves cross-subject alignment without shared stimuli by anchoring representations to a common scaffold provided by a pretrained ANN. Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces with superior semantic organisation, achieving higher cross-subject alignment than common methods while maintaining robust generalisation to held-out stimuli where traditional methods degrade. Reconstructing from these common spaces back to each subject’s original neural space, MED-VAE preserves equal stimulus-driven signal in its cross-subject latent space. Finally, we show that this superior alignment directly enables cross-subject neural prediction, as demonstrated via cross-subject image decoding. In summary, we introduce a framework to identify generalisable common subspaces for cross-subject predictions and downstream tasks, demonstrated here for visual cortex responses to static images. 1 Introduction Neural responses to identical stimuli vary substantially across individuals, yet humans demonstrate remarkably consistent perceptual and behavioral capabilities. This coexistence of individual neural variability with functional equivalence raises a fundamental question: to what extent do different brains employ shared computational principles when processing identical information? Understanding this relationship constitutes a central challenge in systems neuroscience, where identifying common neural motifs across individuals can reveal both universal coding mechanisms and the constraints that shape cortical organisation [16, 13]. Addressing cross-subject neural alignment requires methods that identify common representational structure while accommodating individual anatomical and functional differences to transform neural data from different individuals into a shared representational space enabling direct comparison [12, 6]. Emerging evidence suggests that such shared structure is a fundamental property of neural systems, not merely a convenient modelling assumption. [19] demonstrated that neural population latent dynamics are preserved across animals performing similar motor behaviours, arguing these reflect species-level constraints on circuit organisation. In visual cortex, hyperalignment recovers common high-dimensional representational spaces across subjects [12, 13], indicating that shared computational organisation persists despite idiosyncratic cortical topographies. Cross-subject alignment enables population-level inference, identification of common neural motifs that reveal universal coding mechanisms, and characterisation of individual differences across subjects [22, 19, 2, 18]. Beyond population analysis, aligned representations enable cross-subject neural prediction and universal decoding: neural responses from one subject can reconstruct another’s brain activity, enabling decoders that generalise across individuals with minimal calibration [8, 21, 9, 25], a capability particularly valuable for brain-computer interfaces and clinical applications where individual subjects cannot provide extensive training data. Traditional alignment methods, e.g., Shared Response Models [6, SRM;], Procrustes analysis [11], and Hyperalignment [12], derive transformations to perform alignment from neural responses to shared stimuli viewed by all subjects. While effective when substantial stimulus overlap exists, they face critical limitations: they require shared data for deriving transformations, and generalisation to non-shared stimuli depends entirely on representational coverage of the shared subset. For naturalistic paradigms where subjects view largely non-overlapping stimuli, these constraints become prohibitive and they preclude the large-scale, cross-study data aggregation that the increasing availability of open neuroimaging data makes both desirable and technically feasible. We introduce MED-VAE, a Multi-Encoder Multi-Decoder Variational Autoencoder achieving cross-subject alignment of visual cortex responses to natural scenes in a common space without requiring shared stimuli. Our key insight is that a pre-trained artificial neural network (ANN) processing identical images can serve as an alignment scaffold. By training subject-specific encoders to project fMRI responses into a shared latent space, and requiring this space to reconstruct ANN features via a shared decoder, we create implicit alignment pressure: all subjects’ representations in the shared space must be compatible with the same computational structure encoding hierarchical visual features and semantic relationships. The contributions of our method are: 1. MED-VAE architecture: it eliminates the need for shared stimuli between subjects through ANN-guided latent alignment. 2. Semantic organization: our method is able to preserve behaviorally relevant categorical information in its aligned space, evidenced by higher silhouette scores and cross-subject category decoding accuracy. 3. Cross-subject alignment: it achieves higher alignment and more robust generalization to held-out stimuli where traditional methods degrade. 4. Cross-trial reconstruction: reconstructing from the aligned space back to each subject’s original neural space, it obtains equal cross-trial reconstruction revealing equal preservation of stimulus-related signal. 5. Cross-subject neural prediction and image decoding: we show that we obtain more accurate cross-subject neura prediction and image decoding. 6. Framework for finding common task-relevant latent spaces : the shared (vision) latent space supports multiple downstream tasks, with potential for aggregating neural data across subjects and studies without requiring shared stimuli. 2 Related Work Traditional alignment methods. Traditional alignment approaches transform individual subjects’ representational spaces into unified frameworks where neural activity can be directly compared and analyzed to determine what information is systematically shared across individuals. Shared Response Model [6, SRM;] decomposes neural responses into shared and subject-specific components through probabilistic matrix factorisation, learning transformation matrices from each subject’s individual neural space to the common space, based on the responses to common stimuli. Procrustes analysis [11] finds optimal orthogonal transformations minimising distances between corresponding points across subjects, enforcing rigid geometric transformations that preserve within-subject distances and angles. Hyperalignment [12] iteratively applies Procrustes across subjects to derive a common space. While effective for datasets with extensive stimulus overlap, these methods require substantial shared data for deriving alignment transformations, a strict constraint for naturalistic paradigms where individual subjects view largely non-overlapping stimuli. Moreover, generalisation to non-shared stimuli depends entirely on representational coverage of the shared subset; if alignment transformations learned from shared images fail to capture relevant neural dimensions, the remaining subject-specific responses remain suboptimally aligned. Pairwise cross-subject transfer methods. Recent work has used pretrained ANNs to bypass the shared-stimulus requirement for pairwise cross-subject transfer. [24] use ANN features as a content-loss training signal for pairwise neural code converters, while [26] use per-subject ANN-based encoding models to synthesise paired fMRI data for training pairwise linear transformations. Crucially, both approaches create a separate transformation for each source–target subject pair, i.e., scaling as O(n2)O(n^2) for n subjects, and neither constructs a shared representational space: the output is converted voxel patterns in the target subject’s native brain space, not a common manifold in which all subjects can be simultaneously compared. These methods therefore address a complementary goal, i.e., pairwise brain-to-brain transfer, rather than the population-level aligned space that is the focus of this work. MED-VAE trains a single model across all subjects, producing a shared latent space supporting population-level analyses. Our empirical comparisons accordingly focus on methods that, like MED-VAE, construct common spaces, enabling evaluation on alignment metrics defined only within a shared representational framework. 3 Methods Our approach rests on the finding that ANNs trained on visual tasks produce internal representations that predict neural responses in visual cortex [17, 27, 7]. This model-brain correspondence means the ANN carries task-relevant structure making it a principled alignment scaffold. MED-VAE achieves cross-subject alignment implicitly through architectural constraints rather than explicit stimulus matching or prescribed correspondences. The framework constructs a common space wherein all subjects’ latent representations must reconstruct, along with each subject’s own neural activity via subject-specific decoders, a single shared ANN feature space via a common decoder. Simultaneously, the subject-specific fMRI decoders reconstruct their respective brain activity from this same unified ANN encoding, projected into the latent space via a shared ANN encoder (Fig. 1). This bidirectional constraint establishes pressure toward representational universality across all subjects without explicitly defined correspondence functions. The ANN scaffold overpasses the need for shared stimuli, as each fMRI representation aligns to the ANN representation, despite not having identical correspondence to another subject. It also provides principled task-relevant structure, i.e., encoding categories, hierarchical features, and semantic relationships, that guides alignment toward computationally meaningful dimensions rather than arbitrary geometric correspondences. MED-VAE does not prescribe any relationship between ANN semantic structure and fMRI latent organisation; alignment emerges from reconstruction objectives. Crucially, the ANN scaffold is required only during training to induce this alignment; at inference each subject is projected into the shared space by their fMRI encoder alone, without the ANN model. 3.1 MED-VAE Architecture Our framework employs a Multi-Encoder Multi-Decoder Variational Autoencoder projecting subject-specific neural responses into a shared latent space organized by task-relevant representational structure. The architecture, as shown in Fig. 1, includes four pathway types: Subject-specific fMRI encoders. For each subject i∈1,…,Ni∈\1,…,N\, an encoder EncifMRI:ℝVi→ℝdEnc_i^fMRI:R^V_i ^d maps fMRI responses to a shared d-dimensional latent space: i∼(i,diag(i2)),i,logi=EncifMRI(ifMRI)z_i ( μ_i,diag( σ_i^2)), μ_i, σ_i=Enc_i^fMRI(x_i^fMRI) (1) where ViV_i denotes voxel count for subject i. Shared ANN encoder. A shared encoder EncANN:ℝK→ℝdEnc^ANN:R^K ^d maps ResNet-50 features to the same latent space, establishing the computational structure toward which fMRI representations align. Subject-specific fMRI decoders. Individual decoders DecifMRI:ℝd→ℝViDec_i^fMRI:R^d ^V_i reconstruct each subject’s voxel responses from latent representations originating from both their respective fMRI encoder and the ANN encoder. Shared ANN decoder. A universal decoder DecANN:ℝd→ℝKDec^ANN:R^d ^K reconstructs ANN features from any latent representation. The shared ANN decoder constitutes the primary alignment mechanism: requiring all subjects’ latent representations to reconstruct identical ANN features enforces cross-subject alignment within the latent space. For subjects i and j viewing not identical but similar stimuli, iz_i and jz_j must reconstruct similar ANN features through the shared decoder, enforcing iz_i and jz_j to be close in the latent space, implicitly resulting in cross-subject alignment. A secondary alignment mechanism arises from the subject-specific fMRI decoders needing to reconstruct neural activity from both same-subject fMRI and ANN-encoder latent representations. The shared ANN encoder output serves as a universal input that regularizes subject-specific fMRI decoders toward functionally equivalent transformations, making them more compatible with other subjects’ fMRI encoders, which is beneficial for cross-subject neural prediction. Notably, at inference the ANN scaffold is discarded: a subject’s responses are mapped into the shared space by their fMRI encoder alone EncifMRI(ifMRI)Enc_i^fMRI(x_i^fMRI), and all the inference use cases we demonstrate below operate without requiring the ANN model. The ANN-derived scaffold is only used to encourage latent alignment during training. Another thing to note is that because encoders and decoders are subject-specific with independently parameterised input/output dimensions, the architecture naturally accommodates subjects with different voxel counts or ROI definitions, including, in principle, subjects from different studies or scanners. Figure 1: Multi-Encoder Multi-Decoder Variational Autoencoder (MED-VAE). Subject-specific fMRI encoders map neural responses to a shared latent space; a shared ANN encoder maps ResNet-50 features to the same space. Subject-specific fMRI decoders reconstruct voxel patterns; a shared ANN decoder reconstructs ResNet-50 features from all latent representations, enforcing cross-subject alignment. 3.2 Training Objective The MED-VAE loss integrates the four reconstruction pathways with KL regularization: ℒ =ℒfMRI→fMRI⏟Within-modality fMRI+ℒANN→fMRI⏟Cross-modal: ANN→fMRI = L_fMRI _Within-modality fMRI -4.0mu+ -4.0mu L_ANN _Cross-modal: ANN fMRI (2) +w[ℒANN→ANN⏟Within-modality ANN+ℒfMRI→ANN⏟Cross-modal: fMRI→ANN]+ℒKL -4.0mu+ -4.0muw [ L_ANN _Within-modality ANN -4.0mu+ -4.0mu L_fMRI _Cross-modal: fMRI ANN ] -4.0mu+ -4.0muL_KL The weighting parameter w controls alignment pressure through the shared ANN decoder pathway. Higher w values strengthen the requirement that fMRI encoders produce latent codes compatible with the shared ANN decoder. All results reported here use w=5w=5, selected on a validation set as the value maximising downstream alignment metrics while maintaining NC-normalised reconstruction above 100% (Appendix E). Detailed training objective equations can be found in Appendix B.1. 3.3 Evaluation Metrics Alignment quality. Component-wise correlation measures dimension-specific alignment across subjects and Representational Similarity Analysis (RSA) evaluates geometric structure similarity between subjects’ latent representations. Both metrics are computed for subject pairs in the common aligned space. More information on the metrics can be found in Appendix A. Within-subject reconstruction. Within-subject reconstruction is evaluated by calculating the correlation between the original subject data (input) and the reconstructed subject data (output), i.e., when projecting from the common space back to each subject’s original neural space, thus quantifying the level of subject-specific neural information preserved in the common space. This correlation can be evaluated within-trial or cross-trial, i.e., correlate the reconstructed output with the original data of the same trial (or trial averages of the same stimuli) or a different trial for the same stimulus. Cross-trial evaluation. Within-trial metrics allow noise artifacts, e.g., thermal noise, moment-to-moment physiological fluctuations (respiration phase, cardiac phase), and cognitive state variations, that are in both the input and the reconstructed output, to inflate performance by reconstructing this type of noise in addition to stimulus-related signal. Cross-trial evaluation addresses this confound by correlating reconstructions from Trial i with independent Trial j≠ij≠ i of identical stimuli, breaking the correlation between input noise and evaluation target. NC-normalised reconstruction. To quantify whether a method’s reconstruction reflects stimulus-driven signal or trial-specific noise, we normalise each voxel’s reconstruction correlation by its per-voxel correlation noise ceiling derived from the NSD-provided ncsnr values. Values above 100% indicate reconstruction of non-stimulus-driven variance. Full derivation in Appendix D.3. Category encoding. We evaluate semantic organisation in the common space using two complementary metrics. Silhouette scores, adapted for multi-label images (Appendix A), are computed on the pooled latent representations of all subjects, quantifying how coherently stimuli cluster by semantic category in the shared space. Additionally, leave-one-subject-out multi-label logistic regression decoding, reporting exact match and balanced accuracy, assesses whether category-discriminative structure generalises across subjects. Cross-subject neural prediction. For subject pairs (i,j)(i,j), we compute corr(DecjfMRI(i),jfMRI)corr(Dec_j^fMRI(z_i),x_j^fMRI): how accurately subject i’s latent representation, decoded through subject j’s decoder, reconstructs subject j’s actual neural responses. 3.4 Statistical Analysis For all metrics we report p-values and effect sizes (Cohen’s dzd_z). Methods were compared with two-tailed paired t-tests over the four subjects (n=4n=4). Reconstruction and decoding metrics are naturally per-subject; component correlation and RSA are pairwise, but as these values are not independent, e.g., in pairwise RSA each subject appears in three pairs, the tests use one value per subject, i.e, its mean correlation with the other three subjects. 4 Experimental Setup fMRI Dataset. We use the Natural Scenes Dataset [1, NSD;], comprising 8 subjects viewing mostly unshared ∼ 10,000 natural images each across 30–40 scanning sessions, with ∼ 1,000 images shared among all or a subset of subjects. The NSD data were downloaded from MIT Algonauts Project 2023 Challenge [10]. Cross-subject evaluation focuses on two image sets: 872 images viewed by all participants (shared) and 128 images viewed by a subset of four participants, subjects 1, 2, 5 and 7 (semi-shared). MED-VAE is trained exclusively on subject-specific (non-overlapping) responses. MED-VAE therefore never observes cross-subject correspondences during training, i.e., it cannot exploit stimulus overlap to learn alignment, relying entirely on the ANN scaffold to discover shared structure. SRM and Procrustes, by contrast, derive their alignment transformations from the 872 shared images. The held-out 128 images can be used as a generalization test for all methods and a proxy for the cross-subject alignment quality of the non-shared images. For ROI selection, we followed [7], where neural responses derive from occipitotemporal cortex (∼ 20,730 voxels per subject) in fsaverage surface space, encompassing object- and category-selective regions (LOC, FFA, PPA, EBA). For the cross-subject image decoding evaluation, we use a frozen MindEye2 decoder [21]; for this evaluation we use the ROIs specified in the original MindEye2 work rather than our standard ROIs. ANN features. ResNet-50 activations (pretrained on ImageNet) were extracted for all the images viewed by subjects. We chose ResNet-50 as it has been shown to produce representations particularly well-aligned with human high-level visual cortex [7]. We sampled one layer every 8 to reduce dimensionality of data while obtaining activations from all depths. Each selected layer’s activations were reduced via Sparse Random Projection (SRP) to 3,500 dimensions each. Core alignment metrics vary <2% across SRP seeds, confirming SRP seed choice is not a meaningful variance source. We assesed the robustness of MED-VAE framework on the use of different ANNs as scaffold, by training with CLIP ViT-L/14 (contrastive Vision Transformer). This analysis can be found on Appendix G. Model Training. Encoders and decoders are two-layer MLPs with ReLU activations and dropout (p=0.3p=0.3). At dlatent=32d_latent=32, the hidden dimension is dhidden=256d_hidden=256, giving layer widths of inputdimension→512→256→32input\,dimension→ 512→ 256→ 32 (encoder) and 32→256→512→outputdimension32→ 256→ 512→ output\,dimension (decoder). In experiments where dlatent=512d_latent=512 (Section 5.5), dhiddend_hidden scales to 10241024. Models were trained for 30 epochs with Adam (lr=10−4lr=10^-4, batch size 128128). Full architectural details are given in Appendix B.2. Classical alignment baselines. We compared MED-VAE against Shared Response Model (SRM; [6]) and generalized Procrustes analysis [11, 13], each learning a per-subject linear map into a common dlatent=32d_latent=32 space. Both were fit on the 872 images viewed by all subjects (shared) and evaluated on the 128-image set shared only by subjects 1,2,5,7 (semi-shared); MED-VAE, by contrast, is trained on each subject’s full non-shared responses, the data volume the baselines cannot exploit by construction. Because Procrustes is dimensionality-preserving, i.e., its output dimension equals its input dimension, we first reduced each subject to the dlatent=32d_latent=32 latent with PCA, then aligned subjects iteratively, namely rotating each to the current group-mean configuration via the orthogonal-Procrustes solution and re-estimating the mean until convergence. SRM is itself dimensionality-reducing, and we used its deterministic variant (DetSRM). 5 Results Figure 2: MED-VAE yields a more semantically structured latent manifold, facilitating superior cross-subject alignment and enhanced generalization to unseen stimuli. A. Common subspaces found by MED-VAE, SRM and Procrustes. MED-VAE latent space better encodes stimulus category information. Multi-label images plotted as multiple jittered dots coloured by label; the eight most frequent label combinations shown as single grey-edged dots with blended colours. B. Category encoding quality in the shared space. Silhouette scores measuring category cluster separability in the pooled multi-subject latent space; bars show the bootstrap mean with 95% CI, and the silhouette significance bracket is a paired image-bootstrap over the 128 images (B = 5000; two-sided p). Exact match and balanced accuracy for leave-one-subject-out multi-label category decoding; violins show the per-subject values and their brackets denote two-tailed paired t-tests across subjects (n=4)(n=4). MED-VAE achieves superior category encoding across all metrics. C. Performance comparison across alignment and reconstruction metrics. Alignment: component-wise correlation and RSA. Reconstruction: within-trial and cross-trial voxel correlation—traditional methods’ within-trial advantage disappears under cross-trial evaluation, indicating the reconstruction of within-trial noise rather than stimulus-related signal. Significance brackets denote two-tailed paired t-tests across subjects (n=4n=4). For the alignment metrics the violins show each metric’s native distribution—the 32 component-wise correlations (component correlation) and the 6 subject-pair values (RSA)—but, because these component- and pair-level values are not independent observations, e.g., in RSA each subject appears in three pairs, the tests are run on one value per subject, e.g., each subject’s mean correlation with the other three (n=4n=4); reconstruction metrics are intrinsically per-subject (n=4n=4). Stars reflect uncorrected p (p∗∗<0.001^***p<0.001, p∗<0.01^**p<0.01, p∗<0.05^*p<0.05; n.s., not significant). D. Generalisation gap: difference in component-wise correlation between images used for alignment training for SRM/Procrustes and held-out images, across latent dimensionalities (32, 64, 128). Shaded regions show ±1± 1 s.d. across components. SRM and Procrustes exhibit pronounced generalisation degradation on unseen stimuli. MED-VAE has not used any of these subsets for alignment training. 5.1 Enhanced Semantic Organisation in Aligned Spaces We first examine whether aligned common spaces preserve behaviourally relevant categorical structure. Fig. 2A shows fMRI projections onto latent spaces produced by MED-VAE, SRM, and Procrustes, coloured by semantic category. MED-VAE latent spaces exhibit better categorical organisation: semantically related stimuli cluster coherently, with clear separation between semantic clusters. In contrast, SRM and Procrustes produce diffuse representations with substantial category overlap and poorly defined cluster boundaries. To quantify the degree of semantic structure, we used two complementary metrics. Silhouette scores are computed on the pooled latent representations of all subjects, measuring how coherently stimuli cluster by semantic category in the shared space, (Fig. 2B): MED-VAE achieves a markedly higher silhouette than both SRM and Procrustes (0.34 [0.29, 0.42] vs 0.22 [0.18, 0.28] and 0.21 [0.18, 0.27]; non-overlapping bootstrap 95% CIs, paired image bootstrap B=5000, p<0.001 for both). Additionally, leave-one-subject-out decoding assesses whether category-discriminative structure generalises across individuals: a multi-label classifier is trained on three subjects’ aligned representations and tested on the held-out subject’s. MED-VAE yields higher exact-match accuracy than SRM and Procrustes (0.184 vs. 0.104 and 0.074, respectively; paired t-tests: dz=4.6d_z=4.6 and 3.3, p=0.003p=0.003 and 0.007) and higher balanced accuracy (0.833 vs. 0.758 and 0.754, respectively; dz=6.3d_z=6.3 and 10.5, p=0.001p=0.001 and 0.0002). To demonstrate that these results require task-relevant representations, we trained a MED-VAE framework with the activations of an untrained ResNet; we observe no cross-subject category clustering in the latent space; (Fig.5, Appendix C). As an additional control, we trained MED-VAE after shuffling the 1:1 correspondence between ResNet and fMRI data. We observe that representations cluster by subject identity rather than semantic content (Fig.5, Appendix C). In both control cases, silhouette score when combining all subjects’ representations is near zero. This enhanced semantic organization emerges from the ANN scaffold: the shared decoder requires neural latent representations to reconstruct ResNet-50 features encoding hierarchical visual structure and categorical boundaries. VAEs inherently produce latent spaces where semantically similar inputs cluster together, a property that holds even for ANN-to-ANN or image-to-image autoencoders. MED-VAE leverages this property while allowing the bidirectional pathways, i.e, fMRI→ANN, ANN→fMRI, fMRI→fMRI, to jointly shape the latent geometry. These results demonstrate that a common representational space exists across subjects preserving the computational organisation of visual representation, i.e., categorical structure and semantic relationships, not merely arbitrary statistical correspondence. The ANN-informed scaffold is essential: as our untrained ResNet control confirms, this organisation depends on the scaffold carrying meaningful computational structure, not on the VAE or ResNet architecture alone. 5.2 Cross-Subject Alignment with Robust Generalisation MED-VAE achieves higher cross-subject alignment than SRM and Procrustes, respectively, as measured by both component-wise correlation (0.64 vs. 0.52 and 0.45; paired t-tests: dz=7.7d_z=7.7 and 11.4, p=0.0006p=0.0006 and 0.0002) and RSA (0.63 vs. 0.41 and 0.41; dz=9.0d_z=9.0 and 8.1, p=0.0004p=0.0004 and 0.0005); Fig. 2C, Alignment panel. t-tests were performed on metrics computed per subject (n=4n=4), as described in Section 3.4. Critically, MED-VAE maintains superior performance despite being trained purely with non-overlapping stimuli across subjects. To quantify this effect and contrast it with standard benchmarks, we calculated a generalization gap, defined as the difference in component correlation between the shared (Nshared=872N_shared=872) and semi-shared (Nsemi−shared=128N_semi-shared=128) image sets: Δcorr=ρshared−ρsemi-shared _corr= _shared- _semi-shared. While both SRM and Procrustes are fitted directly on the 872 shared images, they exhibit weak generalization that scales poorly with latent dimensionality d (Fig. 2D). In contrast, MED-VAE demonstrates robust generalization, indicating that it captures subject-invariant representations rather than over-fitting to stimulus-specific features. This generalization gap exposes an important flaw in classical alignment: by deriving transformations solely from shared stimuli, methods like SRM and Procrustes fail to capture the latent dimensions required to align the broader dataset. Without ground-truth correspondences for non-shared stimuli, the 128-image test set serves as a vital proxy for global alignment quality. In contrast, MED-VAE leverages an ANN scaffold to extract alignment pressure from the entire stimulus set, eliminating the risk of overfitting to a "privileged" shared subset. By imposing computational structures—such as categorical hierarchies and representational geometries—the scaffold forces subject convergence into a unified manifold. Per-layer analysis (Appendix F) confirms that brain-ANN alignment increases monotonically with scaffold depth, with late layers, which encode categorical and object-selective representations, reaching 80%. 5.2.1 Cross-Subject Latent Space Retrieval Cross-subject retrieval in the shared latent space provides a direct test of whether aligned representations preserve stimulus identity across individuals. For each image in the 128 semi-shared image dataset, we used subject A’s latent representation as a query and retrieved the nearest neighbour among subject B’s latent representations, measuring whether the retrieved embedding corresponded to the same image. MED-VAE achieved substantially higher retrieval accuracy than both SRM and Procrustes across all top-k thresholds, i.e., more than double the top-1 accuracy of either baseline (46.2% vs 21.8% and 18.4%), as shown in Table 1. This reflects both coherent categorical structure, as also shown in Section 5.1, and stimulus-level alignment enforced by the ANN scaffold. Table 1: Average cross-subject retrieval top-k accuracy on the 128 semi-shared held-out dataset. MED-VAE substantially outperforms traditional alignment methods at preserving stimulus identity across subjects. Method Top-1 Top-2 Top-5 Top-10 Chance 0.8% 1.6% 3.9% 7.8% Procrustes 18.4% 25.6% 40.9% 54.1% SRM 21.8% 30.0% 44.5% 57.8% MED-VAE 46.2% 62.0% 80.3% 91.1% 5.3 Cross-Trial Evaluation Reveals Preservation of Signal A key question is whether MED-VAE’s more aligned, semantically better-organized common space preserves the same level of stimulus-driven neural information. Within-trial reconstruction, where models reconstruct from trial-averaged input and are evaluated against the same trial average, suggests a trade-off: SRM and Procrustes achieve significantly higher voxel correlations than MED-VAE (0.71 and 0.75 vs. 0.61, respectively; paired t-tests: |dz|=17.7|d_z|=17.7 and 21.0, both p<0.001p<0.001, n=4n=4 subjects; Fig. 2C, Reconstruction panel). However, cross-trial evaluation, which isolates stimulus-driven signal by correlating reconstructions from one trial with ground truth from independent trials of the same stimulus (thereby removing noise correlations between input and evaluation target), reveals no significant differences between methods (0.19/0.18/0.17; all p>0.11p>0.11). The expected level difference between within- and cross-trial values given these ROIs’ noise ceiling SNR is calculated in Appendix D.2 and agrees with the observed gap. The difference reflects the noise component that within-trial metrics erroneously credit as signal, while also the fact that independent realizations of noise can be negatively correlated. This cross-trial result demonstrates that traditional methods’ within-trial advantage derives from averaged trial-specific measurement noise rather than stimulus-related signal. NC-normalised self-reconstruction provides a complementary perspective on the same finding, i.e., normalising each voxel’s reconstruction correlation by its per-voxel correlation noise ceiling (derived from the NSD-provided noise ceiling SNR; see Appendix D.3 for the full formulation), with values above 100% indicating reconstruction of non-stimulus-driven variance. Procrustes and SRM achieve 154.8% and 138.3% respectively, substantially exceeding the ceiling, while MED-VAE achieves 102.5%, confirming that the within-trial gap is attributable to noise reconstruction rather than superior signal preservation. MED-VAE, thus, does not sacrifice signal for alignment; it suppresses noise while preserving stimulus-driven neural structure. MED-VAE achieves aligned spaces with superior semantic organization, superior alignment, and comparable preservation of stimulus-driven signal. 5.4 Higher Alignment Enables Superior Cross-Subject Neural Prediction Cross-subject neural prediction, i.e., using subject i’s latent representation with subject j’s decoder to reconstruct subject j’s voxel-level responses, provides functional validation of alignment quality. This metric directly tests whether the common space captures genuinely shared neural structure generalisable across individuals. MED-VAE outperforms the other methods as shown in Fig. 3 for all pairs of subjects. This performance advantage follows directly from the properties established above: a semantically organised, well-aligned common space that preserves stimulus-driven signal naturally supports accurate translation of neural representations between individuals. Figure 3: Cross-subject functional alignment comparison via fMRI Voxel Correlation. Voxel-level prediction accuracy across twelve subject transfer pairs (i→ji→ j). MED-VAE (purple) outperforms SRM and Procrustes. Results are on the 128 held-out images. Table 2: Average cross-subject image decoding performance across all source–target pairs (subjects 1, 2, 5, 7). Higher is better (↑ ) for all metrics except Eff and SwAV (↓ ). Method PixCorr ↑ SSIM ↑ Alex2 ↑ Alex5 ↑ Incep ↑ CLIP ↑ Eff ↓ SwAV ↓ Fwd% ↑ Bwd% ↑ VAE 32D 0.160 0.338 0.846 0.925 0.870 0.827 0.776 0.463 76.7 64.4 SRM 32D 0.113 0.342 0.774 0.862 0.769 0.731 0.868 0.529 64.3 55.9 Proc 32D 0.107 0.345 0.758 0.836 0.736 0.699 0.885 0.541 49.5 45.3 VAE 512D 0.166 0.344 0.867 0.946 0.904 0.856 0.742 0.440 86.5 76.3 SRM 512D 0.156 0.355 0.859 0.928 0.835 0.800 0.806 0.485 83.1 81.1 Proc 512D 0.152 0.355 0.845 0.916 0.825 0.786 0.816 0.489 81.3 80.3 5.5 Cross-subject Image Decoding as a Semantic Probe Cross-subject neural prediction evaluates alignment quality at the voxel level but does not assess what kind of information the predicted responses carry. We therefore used cross-subject image decoding to probe whether cross-subject predictions preserve sufficient semantic content to drive a complex downstream task. The output of cross-subject neural prediction, i.e., subject j’s voxel-level responses reconstructed from subject i’s latent representation via subject j’s decoder (Section 5.4), is fed into a frozen MindEye2 decoder [21] pre-trained exclusively on subject j’s data. No MindEye2 parameters are updated at any stage. Neither MED-VAE nor SRM or Procrustes include decoding-related objectives during training. Image decoding is used here purely as a probe of what semantic information survives the cross-subject prediction pipeline for each method, not as a benchmark against purpose-built decoding systems. We evaluated all pairwise transfer configurations among subjects 1, 2, 5, and 7 on held-out images, using the NSD ROIs specified in the original MindEye2 work rather than our standard occipitotemporal ROIs, to ensure compatibility with the pre-trained decoder weights. Table 2 presents results at 32D and 512D for the latent dimension, i.e., the dimension of the subjects’ aligned space. At 32D, MED-VAE substantially outperforms both baselines across nearly all metrics, with the advantage most pronounced on high-level semantic measures and retrieval accuracy Fwd%. At 512D, the gap between methods narrows considerably though MED-VAE retains a clear advantage on high-level measures (CLIP: 85.6% vs 80.0% vs 78.6%; Inception: 90.4% vs 83.5% vs 82.5%). This dimensionality-dependent pattern reflects MED-VAE’s semantic scaffold: at 32D, it already carries semantically relevant information, while SRM/Procrustes require higher capacity to separate signal from noise. That MED-VAE at 32D surpasses both baselines at 512D on high-level metrics underscores the efficiency of scaffold-guided alignment. The baselines’ relative strengths are confined to SSIM, a metric with known limitations, as [21] showed even degenerate reconstructions can score well on it, and backward retrieval at 512D, which rewards pattern distinctiveness rather than semantic fidelity. The detailed results for each pair of subjects and information on the metrics can be found in Appendix J. Qualitative examples of cross-subject image reconstructions are shown in Fig.4 for selected transfer pairs and with the full set of all 12 transfer directions provided in Fig. 6 in Appendix 6. MED-VAE reconstructions more consistently preserve the semantic content of the original stimuli and exhibit smaller variance across different transfer pairs. Figure 4: Cross-subject image reconstruction examples comparing MED-VAE and SRM alignment. For six selected stimuli, reconstructions are shown for three transfer pairs where subject 1 serves as the source (1→2, 1→5, 1→7). Ground truth images are displayed in the leftmost column. For each stimulus, the top row shows MED-VAE-based reconstructions and the bottom row shows SRM-based reconstructions. Results are for the 512D setting. 6 Discussion and conclusions Population-level neuroscience implications: Aligning subjects without shared stimuli into a semantically organised common space enables population-level analyses that were previously impractical. Data from subjects scanned under different protocols, viewing different stimuli, could in principle be projected into a common representational framework, enabling group-level analyses of representational geometry and individual differences analysis at the neural pattern level rather than the summary-statistic level. MED-VAE provides response-level alignment without shared stimuli, and the resulting common space exhibits qualitatively superior semantic organisation. While the current experiments use subjects from the same dataset, the architecture naturally accommodates heterogeneous inputs, e.g., different voxel counts, ROI definitions, and in principle, different scanners, making cross-study alignment a natural next step. Cross-subject neural prediction and downstream applications: Superior alignment translates to more accurate prediction of individual subjects’ neural responses from other subjects’ representations. This enables practical applications including cross-subject image decoding with minimal per-subject calibration, data augmentation for subjects with limited scanning time by borrowing aligned data from other subjects, and clinical applications where scanning time is constrained. A new participant can be integrated into an existing shared space by training only a subject-specific encoder–decoder pair on their own non-overlapping stimuli recovering ∼ 98% of the jointly-trained alignment quality, with strong alignment retained from as little as 10% of the NSD experiment scanning time without retraining the shared space. More information on this analysis can be found in Appendix I. Potential extensions beyond vision: The ANN scaffold approach could, in principle, be extended to other domains where good model-brain alignment exists, though this remains to be demonstrated. Encoding models for language processing [4, 5], auditory processing [14], and multimodal cognition [23] demonstrate that suitable scaffolds exist across cognitive domains. Extension to temporal domains such as language and auditory processing would require adapting the framework from trial-level static responses to continuous temporal streams, a non-trivial but tractable modification. Limitations and future work: Several limitations warrant discussion. All subjects in the current study are drawn from NSD, sharing scanner hardware and preprocessing pipelines; cross-study alignment with genuinely heterogeneous acquisition parameters remains to be demonstrated, though the architecture is designed to support it. The ResNet-50 scaffold imposes a particular computational lens on alignment: the common space can only capture shared neural structure that is reflected in the scaffold’s representational geometry. Consequently, the quality of cross-subject alignment is fundamentally bounded by the scaffold’s ability to explain neural responses. Using scaffolds with stronger model-brain correspondence, e.g., recurrent architectures that better capture the dynamics of ventral-stream processing [15], represents a natural direction for improving alignment quality. Systematic comparison across scaffold architectures would clarify which computational features are most relevant for cross-subject alignment. Finally, behavioral analysis in the latent space would strengthen the claim that the common space captures not just representational content but processing dynamics with behavioral relevance. Conclusions: MED-VAE bypasses the shared-stimulus bottleneck of traditional neural alignment by replacing explicit matching with implicit constraints from a pretrained ANN scaffold. This architecture yields a cascade of advantages: superior semantic organization, robust generalization to held-out stimuli where classical methods fail, and preserved stimulus-driven signal fidelity. Overall, our framework offers an approach to identify task-informed manifolds shared across human brains that can be used for downstream tasks and neuroscientific investigation. Reproducibility: The code is available at https://github.com/pangelu9/MEDVAE_NSD. Acknowledgments A.P. is supported by the EPSRC Centre for Doctoral Training in Health Data Science [EP/S02428X/1]. R.P.C is funded by a ERC-UKRI Frontier Research Guarantee Starting Grant (EP/Y027841/1). References [1] E. J. Allen, G. St-Yves, Y. Wu, J. L. Breedlove, J. S. Prince, L. T. Dowdle, M. Nau, B. Caron, F. Pestilli, I. Charest, et al. (2022) A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience 25 (1), p. 116–126. Cited by: §4. [2] Z. Anderson et al. (2024) Application of hyperalignment to resting state data in individuals with psychosis reveals systematic changes in functional networks and identifies distinct clinical subgroups. Aperture Neuro 4. Cited by: §1. [3] A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan (2019) Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems 32. Cited by: Appendix F, Appendix H. [4] R. Antonello, A. Vaidya, and A. Huth (2023) Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems 36, p. 21895–21907. Cited by: §6. [5] C. Caucheteux and J. King (2022) Brains and algorithms partially converge in natural language processing. Communications biology 5 (1), p. 134. Cited by: §6. [6] P. C. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. Haxby, and P. J. Ramadge (2015) A reduced-dimension fMRI shared response model. Advances in Neural Information Processing Systems 28. Cited by: §1, §1, §2, §4. [7] C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle (2024) What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?. bioRxiv. Cited by: §3, §4, §4. [8] Y. Dai et al. (2025) MindAligner: explicit brain functional alignment for cross-subject visual decoding from limited fMRI data. International Conference on Machine Learning. Cited by: §1. [9] M. Ferrante, T. Boccato, F. Ozcelik, R. VanRullen, and N. Toschi (2024) Through their eyes: multi-subject brain decoding with simple alignment techniques. Imaging Neuroscience 2, p. 1–21. Cited by: §1. [10] A. T. Gifford, B. Lahner, S. Saba-Sadiya, M. G. Vilas, A. Lascelles, A. Oliva, K. Kay, G. Roig, and R. M. Cichy (2023) The Algonauts Project 2023 challenge: how the human brain makes sense of natural scenes. arXiv preprint arXiv:2301.03198. Cited by: §4. [11] J. C. Gower (1975) Generalized Procrustes analysis. Psychometrika 40 (1), p. 33–51. Cited by: §1, §2, §4. [12] J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, M. Hanke, and P. J. Ramadge (2011) A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 72 (2), p. 404–416. Cited by: §1, §1, §1, §2. [13] J. V. Haxby, J. S. Guntupalli, S. A. Nastase, and M. Feilong (2020) Hyperalignment: modeling shared information encoded in idiosyncratic cortical topographies. elife 9, p. e56601. Cited by: §1, §1, §4. [14] A. J. Kell, D. L. Yamins, E. N. Shook, S. V. Norman-Haignere, and J. H. McDermott (2018) A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98 (3), p. 630–644. Cited by: §6. [15] T. C. Kietzmann, C. J. Spoerer, L. K. Sörensen, R. M. Cichy, O. Hauk, and N. Kriegeskorte (2019) Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences 116 (43), p. 21854–21863. Cited by: §6. [16] N. Kriegeskorte, M. Mur, and P. A. Bandettini (2008) Representational similarity analysis–connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience 2, p. 4. Cited by: Appendix A, §1. [17] N. Kriegeskorte (2015) Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science 1 (1), p. 417–446. Cited by: §3. [18] M. Meshulam, L. Hasenfratz, H. Hillman, Y. Liu, M. Nguyen, K. A. Norman, and U. Hasson (2021) Neural alignment predicts learning outcomes in students taking an introduction to computer science course. Nature Communications 12 (1), p. 1922. Cited by: §1. [19] M. Safaie, J. C. Chang, J. Park, L. E. Miller, J. T. Dudman, M. G. Perich, and A. Bhattacharya (2023) Preserved neural dynamics across animals performing similar behaviour. Nature 623 (7988), p. 765–771. Cited by: §1, §1. [20] P. Scotti, A. Banerjee, J. Goode, S. Shabalin, A. Nguyen, A. Dempster, N. Verlinde, E. Yundler, D. Weisberg, K. Norman, et al. (2023) Reconstructing the mind’s eye: fMRI-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems 36, p. 24705–24728. Cited by: Appendix J. [21] P. S. Scotti, M. Tripathy, C. K. T. Villanueva, R. Kneeland, T. Chen, A. Narang, C. Santhirasegaran, J. Xu, T. Naselaris, K. A. Norman, et al. (2024) MindEye2: shared-subject models enable fMRI-to-image with 1 hour of data. International Conference on Learning Representations. Cited by: §1, §4, §5.5, §5.5. [22] I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, E. Grant, I. Groen, J. Achterberg, et al. (2023) Getting aligned on representational alignment. arXiv preprint arXiv:2310.13018. Cited by: §1. [23] J. Tang, M. Du, V. Vo, V. Lal, and A. Huth (2023) Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in Neural Information Processing Systems 36, p. 29654–29666. Cited by: §6. [24] H. Wang, J. K. Ho, F. L. Cheng, S. C. Aoki, Y. Muraki, M. Tanaka, J. Park, and Y. Kamitani (2025) Inter-individual and inter-site neural code conversion without shared stimuli. Nature Computational Science 5 (7), p. 534–546. Cited by: §2. [25] Z. Wang, Z. Zhao, L. Zhou, and P. Nachev (2024) Unibrain: a unified model for cross-subject brain decoding. arXiv preprint arXiv:2412.19487. Cited by: §1. [26] N. Wasserman, R. Beliy, R. Urbach, and M. Irani (2026) Functional brain-to-brain transformation without shared stimuli. NeuroImage, p. 121741. Cited by: §2. [27] D. L. Yamins and J. J. DiCarlo (2016) Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience 19 (3), p. 356–365. Cited by: §3. Appendix A Metrics Component-wise Correlation. To quantify the consistency of individual latent dimensions across subjects, we computed component-wise Pearson correlation. For each latent dimension d∈1,…,Dd∈\1,…,D\, we extracted the activation vector d(s)∈ℝNz^(s)_d ^N across all N test images for each subject s. We then computed the pairwise Pearson correlation coefficient r(d(si),d(sj))r(z^(s_i)_d,z^(s_j)_d) for all subject pairs (si,sj)(s_i,s_j) and averaged across pairs to obtain a single correlation value per component. This metric assesses whether each latent dimension encodes consistent information across subjects, independent of the remaining dimensions. Representational Similarity Analysis (RSA). We used RSA [16] to compare the representational geometry across subjects. For each subject s, we constructed a representational dissimilarity matrix (RDM) (s)∈ℝN×ND^(s) ^N× N, across all N test images, where each entry Dij(s)=1−r(i(s),j(s))D^(s)_ij=1-r(z^(s)_i,z^(s)_j) corresponds to the Pearson correlation distance between the latent representations of fMRI responses to images i and j. Pairwise RDM similarity between subjects was then quantified using the Pearson correlation between the upper-triangular elements of their respective RDMs. Clustering Quality—Adapted Multi-Label Silhouette Score. Standard silhouette analysis assumes single-label clustering, but natural scene images inherently contain multiple semantic categories (e.g., “person” + “bicycle” + “outdoor”). We implemented a modified silhouette score adapted for multi-label data: 1. For multi-label samples, we computed pairwise label similarity using Jaccard index: J(i,j)=|labelsi∩labelsj||labelsi∪labelsj|J(i,j)= |labels_i _j||labels_i _j| (3) where labelsilabels_i represents the set of ground-truth categories for sample i. 2. For each sample i, we partitioned other samples into “similar” (Jaccard >> median_similarity) and “dissimilar” (Jaccard ≤ median_similarity) groups. 3. Distances in latent space were computed using Euclidean metric. 4. The silhouette coefficient for sample i was calculated as: si=bi−aimax(ai,bi)s_i= b_i-a_i (a_i,b_i) (4) where aia_i = mean distance to similar samples and bib_i = mean distance to dissimilar samples. This modified silhouette score quantifies how effectively the latent space organises stimuli according to semantic category structure while accommodating the multi-label nature of natural images. For all methods, silhouette scores are computed not on the raw d-dimensional latents but on a 10-dimensional UMAP embedding of the pooled multi-subject representations, fit identically across methods so that scores remain directly comparable. Appendix B Methods B.1 Training Objective The complete MED-VAE loss function integrates the four complementary reconstruction pathways with KL regularization: The MED-VAE loss integrates the four reconstruction pathways with KL regularization: ℒ =ℒfMRI→fMRI⏟Within-modality fMRI+ℒANN→fMRI⏟Cross-modal: ANN→fMRI = L_fMRI _Within-modality fMRI -4.0mu+ -4.0mu L_ANN _Cross-modal: ANN fMRI (5) +w[ℒANN→ANN⏟Within-modality ANN+ℒfMRI→ANN⏟Cross-modal: fMRI→ANN]+ℒKL -4.0mu+ -4.0muw [ L_ANN _Within-modality ANN -4.0mu+ -4.0mu L_fMRI _Cross-modal: fMRI ANN ] -4.0mu+ -4.0muL_KL where the single weighting parameter w governs alignment pressure through the shared ANN decoder pathway. and controls the alignment-reconstruction trade-off. Component 1: Within-Modality fMRI Reconstruction ℒrecfMRI→fMRI=∑i=1Nq(i|ifMRI)[‖ifMRI−DecifMRI(i)‖2]L_rec^fMRI fMRI= _i=1^NE_q(z_i|x_i^fMRI) [\|x_i^fMRI-Dec_i^fMRI(z_i)\|^2 ] (6) where q(zi∣xifMRI)=(μi,diag(σi2))q(z_i x_i^fMRI)=N( _i,diag( _i^2)) is the approximate posterior parameterised by subject i’s encoder (Eq. 1). Pathway: ifMRI→EncifMRIi→DecifMRI^ifMRIx_i^fMRI Enc_i^fMRIz_i Dec_i^fMRI x_i^fMRI. Standard autoencoding preserves subject-specific neural information, ensuring latent representations iz_i retain sufficient detail for within-subject voxel-level reconstruction. Component 2: Cross-Modal ANN→fMRI Reconstruction ℒrecANN→fMRI=∑i=1Nq(ANN|ANN)[‖ifMRI−DecifMRI(ANN)‖2]L_rec^ANN fMRI= _i=1^NE_q(z_ANN|x^ANN) [\|x_i^fMRI-Dec_i^fMRI(z_ANN)\|^2 ] (7) where q(zANN∣xANN)q(z^ANN x^ANN) by the shared ANN encoder. Pathway: ANN→EncANNANN→DecifMRI^ifMRIx^ANN Enc^ANNz_ANN Dec_i^fMRI x_i^fMRI. Enforces decoder universality by requiring subject-specific fMRI decoders to reconstruct neural activity from ANN-encoder outputs. Component 3: Within-Modality ANN Reconstruction ℒrecANN→ANN=q(ANN|ANN)[‖ANN−DecANN(ANN)‖2]L_rec^ANN ANN=E_q(z_ANN|x^ANN) [\|x^ANN-Dec^ANN(z_ANN)\|^2 ] (8) Pathway: ANN→EncANNANN→DecANN^ANNx^ANN Enc^ANNz_ANN Dec^ANN x^ANN. Establishes task-relevant latent space organization – hierarchical visual features, categorical boundaries, semantic relationships – constituting the target representational geometry for cross-subject alignment. Component 4: Cross-Modal fMRI→ANN Reconstruction ℒrecfMRI→ANN=∑i=1Nq(i|ifMRI)[‖ANN−DecANN(i)‖2]L_rec^fMRI ANN= _i=1^NE_q(z_i|x_i^fMRI) [\|x^ANN-Dec^ANN(z_i)\|^2 ] (9) Pathway: ifMRI→EncifMRIi→DecANN^ANNx_i^fMRI Enc_i^fMRIz_i Dec^ANN x^ANN. Enforces encoder alignment: subject-specific fMRI encoders must produce latent codes iz_i compatible with the shared ANN decoder. For subjects i and j viewing identical/similar stimuli, both iz_i and jz_j must reconstruct the same/similar ANN features. Component 5: KL Divergence Regularization ℒKL=∑i=1NDKL(q(i|ifMRI)∥p())+DKL(q(ANN|ANN)∥p())L_KL= _i=1^ND_KL(q(z_i|x_i^fMRI)\|p(z))+D_KL(q(z_ANN|x^ANN)\|p(z)) (10) where p()=(,)p(z)=N(0,I) denotes the standard Gaussian prior. B.2 Model Architecture Each encoder is a two-layer MLP with LayerNorm, ReLU activations, and dropout (p). The encoder produces μ and logσ2 σ^2 via two parallel projection heads from the final hidden layer. Each decoder mirrors this structure, with output s⋅tanh(⋅)s· (·), where s is a learnable scale parameter. Table 3 specifies the full layer sequence. Table 3: Encoder and decoder architecture. LN = LayerNorm, D(p) = Dropout. Encoder 1 Linear(dind_in, 2dh2d_h) → LN → ReLU → D(p) 2 Linear(2dh2d_h, dhd_h) → LN → ReLU → D(p) μ Linear(dhd_h, dzd_z) logσ2 σ^2 Linear(dhd_h, dzd_z) Decoder 1 Linear(dzd_z, dhd_h) → LN → ReLU → D(p) 2 Linear(dhd_h, 2dh2d_h) → LN → ReLU → D(p) out Linear(2dh2d_h, doutd_out) → s⋅tanh(⋅)s· (·) All encoders and decoders share this architecture but have independent weights. During training, latent samples are drawn via z=μ+exp(0.5logσ2)εz=μ+ (0.5 σ^2)\, , ε∼(0,I) (0,I); at evaluation, z=μz=μ. Hyperparameters. We report results at dz=32d_z=32, dh=256d_h=256, and p=0.3p=0.3. For the experiments where dz=512d_z=512 (Section 5.5), dh=1024d_h=1024. Models were trained for 30 epochs with Adam (lr =10−4=10^-4, batch size 128128). Loss weights: w=wfMRI→ANN=wANN→ANN=5w=w_fMRI =w_ANN =5, wfMRI→fMRI=wANN→fMRI=1w_fMRI =w_ANN =1. Appendix C Control Analysis: Untrained ANN To verify that cross-subject alignment in MED-VAE depends on meaningful ANN representations, we trained two control cases; one with untrained ResNet-50 and one with destroying the 1:1 correspondence between the ResNet representations and the fMRI data. In the untrained control, the ANN encoder received activations from a randomly initialised ResNet-50, which carries no learned semantic information. In the shuffled control, the ANN encoder received activations from a fully trained ResNet-50, but the mapping between stimuli and activations was randomly permuted, breaking the one-to-one correspondence between what a subject viewed and the ANN features fed to the model. In the shuffled condition, the UMAP projection of the 32-dimensional latent space reveals eight clearly separated clusters, each corresponding to a single subject as seen in Fig. 5 (right). Because the fMRI-to-fMRI reconstruction pathway operates independently within each subject, each encoder still learns a compact representation of its own subject’s data. However, without a valid shared ANN signal to anchor these representations to one another, the per-subject latent spaces remain unaligned. Consequently, no category structure emerges across subjects: points from different subjects that depict the same semantic category are scattered across separate clusters rather than co-localised as in the trained model (see. Figure 2A). For the untrained RN50 condition, in Fig. 5 (left), we observe one diffuse blob with thoroughly mixed colours, i.e., the latent has no semantic-category organization at all This is expected as untrained ANN features carry no category structure to align to. Quantitatively, silhouette score in MED-VAE latent is 0.056 for the untrained RN50 control and 0.018 for the shuffled control (trained RN50 MED-VAE is 0.34). Retrieval top-1 score is 6.6% for the untrained control and 1.0% for the shuffled control, compared to 46.2% for trained RN50 MED-VAE in Table 1. This confirms that cross-subject alignment requires the combination of the VAE architecture with a scaffold carrying meaningful computational structure; neither the VAE’s architectural constraints alone nor the ResNet architecture without task-relevant learned representations is sufficient to achieve alignment. Figure 5: UMAP projection of latent representations from control MED-VAE models. (Left) Untrained MED-VAE: the ANN encoder receives activations from a randomly initialised (untrained) ResNet-50, providing no meaningful semantic signal. (Right) Shuffled MED-VAE: the ANN encoder receives activations from a trained ResNet-50, but the stimulus-to-activation mapping is randomly permuted, destroying the correspondence between ANN and fMRI responses. Points are coloured by COCO super-category label; the absence of category structure within clusters confirms that cross-subject alignment depends on a valid ANN–fMRI correspondence. Appendix D Noise Ceiling Analysis D.1 Theoretical Framework We model each fMRI trial as a sum of signal and noise: Ytrial=S+NY_trial=S+N (11) where S represents the true stimulus-driven neural signal (consistent across trials of the same stimulus) and N represents trial-specific noise. We assume S and N are uncorrelated: cov(S,N)=0cov(S,N)=0. NCSNR metric. The Natural Scenes Dataset provides per-voxel noise ceiling signal-to-noise ratio (ncsnr), defined as: ncsnr=σSσN=var(S)var(N)ncsnr= _S _N= var(S) var(N) (12) This is a single scalar per voxel, estimated across all images in the dataset and reflecting the voxel’s overall signal quality. Different subjects and ROIs yield different average ncsnr values. Noise ceiling as signal variance fraction. The noise ceiling (NC) represents the proportion of total variance attributable to signal: NC=var(S)var(S)+var(N)=ncsnr2ncsnr2+1NC= var(S)var(S)+var(N)= ncsnr^2ncsnr^2+1 (13) Trial-to-trial correlation equals NC. When correlating two independent trials of the same stimulus, both containing independent noise realisations: corr(trial1,trial2) (trial_1,trial_2) =corr(S+N1,S+N2) =corr(S+N_1,S+N_2) =cov(S+N1,S+N2)std(S+N1)⋅std(S+N2) = cov(S+N_1,S+N_2)std(S+N_1)·std(S+N_2) =var(S)var(S)+var(N)=NC = var(S)var(S)+var(N)=NC (14) since cov(S,Ni)=0cov(S,N_i)=0 and cov(N1,N2)=0cov(N_1,N_2)=0 for independent noise. Correlation ceiling. When correlating a perfect signal prediction (noise-free) with noisy data: corr(S,S+N) (S,S+N) =var(S)var(S)⋅var(S)+var(N) = var(S) var(S)· var(S)+var(N) =var(S)var(S)+var(N)=NC=Rceil = var(S)var(S)+var(N)= NC=R_ceil (15) The square root arises because only one side contains noise. This RceilR_ceil represents the theoretical maximum correlation achievable by any model producing perfect signal estimates. Effect of trial averaging. When averaging n trials, signal remains constant while noise variance reduces: NCn=ncsnr2ncsnr2+1/nNC_n= ncsnr^2ncsnr^2+1/n (16) D.2 Expected Gap Between Within-Trial and Cross-Trial Within-Subject Reconstruction The within-trial and cross-trial reconstruction metrics reported in Fig. 2C (Reconstruction panel) differ in their evaluation protocol, and consequently face different noise ceilings. Here, we derive the expected gap between them for the ncsnr value of the ROIs we used in our analysis, which is provided by NSD and shown in Table 4, to verify that the observed difference in values is consistent with the noise ceiling framework rather than reflecting a loss of neural information. Within-trial evaluation. In our within-trial protocol, we reconstruct from trial-averaged input (n=3n=3 repetitions) and correlate the reconstruction S S with the same trial-averaged ground truth S+NavgS+N_avg, where var(Navg)=var(N)/3var(N_avg)=var(N)/3. For a model that perfectly recovers stimulus-driven signal, the ceiling is: Rceil,n=3=NCn=3=ncsnr2ncsnr2+1/3R_ceil,n=3= NC_n=3= ncsnr^2ncsnr^2+1/3 (17) The values for each subject’s Rceil,n=3R_ceil,n=3 are given in Table 4, with mean Rceil,n=3=0.618R_ceil,n=3=0.618. Cross-trial evaluation. In our cross-trial protocol, we reconstruct from a single trial and correlate the reconstruction with an independent single trial of the same stimulus. Even if the model perfectly extracts signal from its input trial, the evaluation target S+NjS+N_j contains full single-trial noise (var(Nj)=var(N)var(N_j)=var(N)), so the ceiling is: Rceil,n=1=NCn=1=ncsnr2ncsnr2+1R_ceil,n=1= NC_n=1= ncsnr^2ncsnr^2+1 (18) As shown in Table 4, mean Rceil,n=1=0.449R_ceil,n=1=0.449. Table 4: Per-subject mean NCSNR, noise ceilings, and correlation ceilings, averaged across all ∼ 20,730 occipitotemporal cortex (Streams ROI) voxels. K=3K=3: within-trial (trial-averaged) evaluation. K=1K=1: cross-trial (single-trial) evaluation. Subject NCSNR NC (K=3K=3) RceilR_ceil (K=3K=3) NC (K=1K=1) RceilR_ceil (K=1K=1) S1 0.501 0.407 0.622 0.201 0.448 S2 0.538 0.437 0.640 0.224 0.474 S5 0.576 0.464 0.662 0.249 0.499 S7 0.406 0.323 0.549 0.142 0.376 Overall 0.505 0.408 0.618 0.204 0.449 Summary. We denote by S S the model’s reconstruction from a given input. Evaluation Correlation computed Ceiling Within-trial corr(S^avg,S+Navg)corr( S_avg,\;S+N_avg) Rceil,n=3=0.618R_ceil,n=3=0.618 Cross-trial corr(S^triali,S+Nj),j≠icorr( S_trial_i,\;S+N_j),\;\;j≠ i Rceil,n=1=0.449R_ceil,n=1=0.449 Baseline corr(S+Ni,S+Nj),j≠icorr(S+N_i,\;S+N_j),\;\;j≠ i NCn=1=0.204NC_n=1=0.204 The within-trial ceiling (Rceil,n=3=0.618R_ceil,n=3=0.618) is substantially higher than the cross-trial ceiling (Rceil,n=1=0.449R_ceil,n=1=0.449) for two reasons: (i) the within-trial evaluation target contains noise reduced by n=3n=3 averaging, and (i) the model reconstructs from this less noisy averaged input to itself. The cross-trial ceiling of 0.4470.447 would be reached only if the model produced a perfectly denoised signal estimate from a single trial. In practice, all methods achieve cross-trial correlations of approximately 0.200.20 (Fig. 2C, Reconstruction panel), coinciding with the no-model baseline NCn=1≈0.20NC_n=1≈ 0.20. The key observation is that all methods—MED-VAE, SRM, and Procrustes—converge to the same cross-trial performance despite within-trial values ranging from ≈0.6≈ 0.6 (MED-VAE) to ≈0.7≈ 0.7 (SRM). The entire within-trial gap between methods vanishes under cross-trial evaluation, confirming that it reflects non stimulus-driven signal. D.3 NC-Normalised Within-Subject Reconstruction The cross-trial analysis above demonstrates that all methods preserve equal stimulus-driven signal. To further quantify the extent to which each method’s within-trial reconstruction reflects signal versus noise, we normalise self-reconstruction performance by the per-voxel noise ceiling, that the dataset provides. For each voxel v, we compute: NC-normv=rvRceil,vNC-norm_v= r_vR_ceil,v (19) where rvr_v is the method’s Pearson correlation between reconstructed and original responses at voxel v (computed across images), and Rceil,v=NCvR_ceil,v= NC_v is the per-voxel correlation ceiling computed from that voxel’s ncsnr with K=3K=3 trial averaging. We then average across all voxels: NC-norm=1V∑v=1VrvRceil,vNC-norm= 1V _v=1^V r_vR_ceil,v (20) expressed as a percentage. This per-voxel normalisation accounts for heterogeneous signal quality across voxels. Values above 100% indicate that the method reconstructs more variance than can be attributed to stimulus-driven signal at individual voxels, i.e., it captures trial-specific noise shared between input and evaluation target. Table 5 reports NC-normalised performance for all three alignment methods on the 128 held-out images. Table 5: Within-subject reconstruction and cross-subject prediction with NC-normalisation. NC-norm values above 100% on the diagonal (within-subject reconstruction) indicate reconstruction of non-stimulus-driven variance. Metric Procrustes SRM MED-VAE within-subj recon r (within-trial) 0.746 0.707 0.606 NC-norm (diagonal) 154.8% 138.3% 102.5% Cross-subj prediction r 0.332 0.344 0.383 NC-norm (off-diagonal) 33.4% 35.4% 40.9% Interpretation. The NC-normalised within-subject reconstruction values confirm the noise-capture interpretation from the cross-trial analysis: Procrustes (154.8%) and SRM (138.3%) substantially exceed the noise ceiling. This confirms that their within-trial within-subject reconstruction advantage derives from capturing trial-specific measurement noise, i.e., spatially correlated noise patterns that inflate within-trial correlations but vanish under cross-trial evaluation. Overall, cross-trial analysis and NC-normalised values demonstrate that MED-VAE does not sacrifice stimulus-driven signal for alignment. Its lower within-trial reconstruction relative to classical methods reflects principled noise suppression, not inferior signal capture. Appendix E Hyperparameter w Sensitivity We swept w∈0.5,1,2,5,10,20w∈\0.5,1,2,5,10,20\ on the 872 shared images (held out from training). Table 6 reports key metrics. Table 6: Sensitivity to scaffold weight w. Selected value in bold. w CompCorr Ret@1 Silh. rvoxr_vox (cross) rvoxr_vox (self) NC-norm 0.5 0.558 14.5% 0.331 0.374 0.657 131.6% 1 0.584 17.0% 0.336 0.376 0.647 127.8% 2 0.615 18.9% 0.352 0.379 0.633 122.0% 5 0.655 19.6% 0.358 0.383 0.610 112.9% 10 0.671 18.8% 0.348 0.384 0.590 105.6% 20 0.682 17.9% 0.340 0.382 0.568 97.3% Increasing w monotonically improves component-wise correlation (CompCorr) but degrades within-subject reconstruction correlation (rvoxr_vox (self)). Noise-ceiling-normalised within-reconstruction (NC-norm; Appendix D.3) remains above 100% for w≤10w≤ 10, dropping below only at w=20w=20. Downstream alignment metrics cross-subject retrieval top-1 accuracy (Ret@1) and silhouette score peak at w=5w=5 and decline at higher values, while cross-subject voxel prediction correlation (rvoxr_vox) is stable across the sweep (0.374–0.384). We selected w=5w=5 as the highest value maintaining NC-norm above 100% while maximising downstream alignment. Appendix F Per-Layer Reconstruction Analysis To understand what the shared latent space captures across the scaffold hierarchy, we performed a per-layer analysis evaluating two reconstruction pathways: ANN→ → (how much of each layer’s information survives the 32D bottleneck) and fMRI→ → (how much of each layer’s information is mapped to brain data, and thus reconstructed by the fMRI→ → pathway). All values are per-feature Pearson correlations. Raw fMRI→ reconstruction is confounded by each ANN layer’s intrinsic dimensionality: layers with high intrinsic dimensionality are disfavoured by the low-dimensional bottleneck regardless of their brain relevance, simply because their variance cannot be compressed into 32 dimensions. We therefore report the ratio of fMRI→ to ANN→ correlation as the primary metric, quantifying how each layer maps to the fMRI data. This ratio controls for compressibility and indexes brain–ResNet-50 alignment at each hierarchical level of ResNet-50: it measures what fraction of each layer’s preserved information (i.e., the information that survives the bottleneck) is neurally accessible, i.e., maps to the fMRI data. For the ResNet-50 data, we sampled one layer every 8 (both pre- and post-activation), yielding 15 sampled layers spanning the full depth of the network. Each layer’s activations were reduced to 3,500 dimensions via Sparse Random Projection before concatenation into a single 51K-dimensional feature vector. The results for all layer activations, are the following: Table 7: Per-layer reconstruction for ResNet-50 (MED-VAE, w=5w=5). We have included activations from Conv2d layers (pre-ReLU) and post-ReLU. Per-feature Pearson correlation on 872 shared images. Ratio = fMRI→ / ANN→ . Layer Module ANN→ fMRI→ Ratio Conv2d-1 conv1 0.514 0.263 0.51 ReLU-1 conv1 0.420 0.218 0.52 Conv2d-9 layer1 0.209 0.145 0.69 ReLU-9 layer1 0.141 0.100 0.71 Conv2d-17 layer2 0.455 0.259 0.57 ReLU-17 layer2 0.322 0.231 0.72 Conv2d-25 layer3 0.225 0.161 0.71 ReLU-25 layer3 0.339 0.240 0.71 Conv2d-33 layer3 0.335 0.253 0.76 ReLU-33 layer3 0.344 0.256 0.74 Conv2d-41 layer3 0.409 0.315 0.77 ReLU-41 layer3 0.292 0.204 0.70 Conv2d-49 layer4 0.534 0.428 0.80 ReLU-49 layer4 0.464 0.363 0.78 AvgPool final avgpool 0.586 0.469 0.80 Table 7 reveals three distinct patterns across the ResNet-50 hierarchy: ANN→ reconstruction is ∪-shaped. Early layers (conv1: 0.514) and late layers (avgpool: 0.586) reconstruct best, while intermediate layers show a pronounced trough. This profile mirrors the inverse of the known intrinsic dimensionality (ID) profile across CNN layers: [3] showed that ID peaks at intermediate ResNet layers before dropping again for the final layers. The brain–RN50 ratio increases monotonically with depth. The ratio rises from 0.51 at conv1 to 0.80 at layer4/avgpool. At every successive level of the hierarchy, a larger fraction of what the bottleneck preserved is brain-accessible, i.e., maps to the fMRI data. This is consistent with the functional properties of our occipitotemporal ROIs (LOC, FFA, PPA, EBA), which encode high-level categorical and object-selective representations corresponding to late network layers. For a compact summary, Table 8 aggregates across pre- and post-ReLu activation layers at three depth levels. Table 8: Reconstruction summary at different ResNet-50 depths summary (averaging pre- and post-activation layers). Depth ANN→ fMRI→ Ratio Early (conv1, layer1) 0.321 0.182 0.57 Mid (layer2, layer3) 0.340 0.240 0.71 Late (layer4, avgpool) 0.528 0.420 0.80 Appendix G Scaffold Comparison: ResNet-50 vs CLIP ViT-L/14 To assess the sensitivity of MED-VAE’s alignment on the use of different scaffold architectures, we trained the full framework with two distinct scaffolds: ResNet-50 (ImageNet-pretrained, ∼ 51K features after per-layer SRP) and CLIP ViT-L/14 (contrastive vision transformer, ∼ 26K features after per-layer SRP). In both cases, MED-VAE uses a 32-dimensional latent bottleneck and, here, evaluations are on the 872 shared images across all 8 NSD subjects (held-out from training). Table 9 reports the full comparison across all alignment, reconstruction, and downstream metrics. Table 9: Full scaffold comparison across all evaluation metrics. Bold indicates the better-performing scaffold. Evaluated on 872 shared images, all 8 NSD subjects. Metric ResNet-50 CLIP ViT-L/14 Latent space alignment Compositional Corr. 0.647 0.549 RSA (Pearson) 0.652 0.582 Category encoding Silhouette 0.336 0.353 Balanced Accuracy 0.832 0.809 Cross-subject retrieval Retrieval Top-1 48.9% 31.4% Retrieval Top-2 63.6% 48.1% Reconstruction fMRI Recon (voxel corr.) 0.606 0.606 fMRI Recon (NC norm%) 102.5 102.7 Cross-Trial r 0.269 0.257 Cross-subject neural prediction Cross-Subj Prediction 0.383 0.370 Cross-Subj NC% 40.8 38.6 ResNet-50 outperforms CLIP ViT-L/14 on the majority of cross-subject alignment metrics, with particularly large advantages in retrieval accuracy (48.9% vs 31.4% top-1) and latent alignment metrics (CompCorr: 0.647 vs 0.549). Importantly though, MED-VAE with CLIP ViT-L/14 still substantially outperforms all classical baselines benchmarked in the paper (SRM, Procrustes), showcasing a robustness to the choice of the ANN scaffold. Appendix H ANN Reconstruction Performance Here we explore the ANN reconstruction performance of the ANN→ pathway measured by Pearson correlation between the original activations and the reconstructed ones through the pathway. To isolate the cost of sharing the latent space with fMRI, we compare MED-VAE, where the latent is trained on ANN and fMRI concurrently, against an ANN-only VAE, i.e., same 32D architecture, trained with only the ANN encoder/decoder — no fMRI encoders or decoders. Table 10 reports per-feature Pearson correlation on the 128 held-out images. Table 10: ANN reconstruction: MED-VAE vs ANN-only control. Model Per-feature r ANN-only VAE 0.368 MED-VAE (w=5w=5) 0.354 The ∼ 4% difference confirms that sharing the latent space with fMRI does not reduce the ANN reconstruction performance; the 32D bottleneck compressing 51K ResNet-50 features is the binding constraint, not the trade-off with fMRI pathways. A PCA analysis of the ResNet-50 features explains this constraint: 32 PCs capture only 14.6% of the total variance, and ∼ 1,810 PCs are needed to reach 50% — reflecting the high intrinsic dimensionality of intermediate ResNet layers [3]. Per layer ANN→ reconstruction, along with fMRI→ reconstruction, i.e., how much of each layer’s reconstructed information is mapped to the fMRI data, can be found in Appendix F. Appendix I Generalisation to Novel Subjects A practical requirement for cross-subject alignment is that a new participant can be added to an existing shared space efficiently—ideally without retraining the space and without requiring stimuli shared with the existing cohort. We assess this in two ways: (i) a leave-one-subject-out integration experiment testing whether a held-out subject can be aligned to a frozen shared space, and (i) a data-efficiency analysis quantifying how much of the new participant’s data is needed to achieve good alignment. I.1 Leave-one-subject-out integration We trained MED-VAE on 7 of the 8 NSD subjects to shape the shared latent space, then integrated the held-out subject by training only a new subject-specific encoder and decoder to align with the ANN scaffold, keeping the shared ANN encoder and decoder—and hence the shared space itself—frozen. This was repeated with each of the 8 subjects held out in turn. Crucially, the new subject’s encoder and decoder are trained on that subject’s own (non-overlapping) stimuli; no stimulus overlap with the other 7 subjects is required, and only an encoder–decoder pair is fitted, making integration computationally cheap. We compared this finetuned integration against the full jointly-trained 8-subject model (MED-VAE trained with all 8 subjects jointly), evaluating both on the 872 images shared across all subjects (held out from MED-VAE training). Table 11 reports performance averaged over all held-out subjects. Table 11: Leave-one-subject-out integration into a frozen shared space vs. the full jointly-trained 8-subject model. Values are averaged across all held-out subjects, evaluated on the 872 shared images. Metric Joint Finetuned Compositional Corr. 0.664 0.655 RSA (Pearson) 0.656 0.638 Retrieval Top-1 (%) 18.1 17.8 Self-recon. (voxel r) 0.624 0.587 Self-recon. (NC%) 118.9% 104.1% Finetuning into the frozen shared space recovers approximately 98% of the jointly-trained component correlation, RSA, and retrieval accuracy. The raw voxel reconstruction correlation drops by 0.037 on average, but noise-ceiling normalisation (Appendix D.3) shows this does not reflect a loss of stimulus-driven signal: NC-normalised reconstruction remains above 100%, i.e. the finetuned model continues to recover essentially all of the recoverable stimulus-driven variance. MED-VAE therefore provides a simple route to integrate novel subjects, achieving performance comparable to including the subject in the initial joint training. (Metrics here are measured on the 872 shared images and so differ from the 128-image values reported in the main text.) I.2 Data efficiency To quantify how much data a new participant needs in order to align to the group that defines the shared space, we used subject 5 as a representative held-out subject: we trained MED-VAE on the remaining 7 subjects, froze the shared ANN encoder and decoder, and then trained a new encoder and decoder for subject 5 on 10%, 20%, 30%, and 100% of its available data. Table 12 reports the results. Table 12: Data efficiency of integrating a held-out subject (S5) into a frozen shared space, as a function of the fraction of S5 data used to train its encoder–decoder pair. Self-recon. columns report within-subject reconstruction. Comp. RSA Self-recon. Data Corr. (Pears.) voxel r NC% 10% 0.655 0.664 0.526 63.9 20% 0.661 0.662 0.621 89.6 30% 0.668 0.672 0.632 92.9 100% 0.679 0.672 0.655 99.5 The latent alignment metrics are remarkably robust to data reduction: at just 10% of the training data, component correlation reaches 96.5% of the full-data value and RSA reaches 98.8%. Self-reconstruction shows the largest improvement between 10% and 20% of the data (0.526→ 0.621, i.e. 64% → 90% of the noise ceiling), with diminishing returns thereafter. Together these results indicate that a new participant can be aligned to an existing, well-scanned cohort with substantially less data, without shared stimuli and without retraining the shared representational space—a regime directly relevant to clinical settings where scanning time is constrained. Appendix J Cross-Subject Image Decoding Results: Per-Pair Results Tables 13 and 14 present cross-subject image decoding performance without refinement, while Tables 15 and 16 present results with refinement, for 32-dimensional and 512-dimensional common spaces, respectively. Refinement refers to the second stage of the MindEye2 pipeline [20], in which initial reconstructions generated via SDXL unCLIP are passed through a base Stable Diffusion XL image-to-image process conditioned on predicted text captions, improving perceptual quality without substantially altering high-level semantic content. Notation i→ji→ j denotes alignment of subject i’s neural responses to subject j’s space, with subsequent decoding via a frozen MindEye2 decoder. For each subject pair and metric, the superior method is indicated in bold. Table 13: MED-VAE 32D vs SRM 32D vs Procrustes 32D decoding performance (No Refinement). Bold indicates the best-performing method per pair. Higher is better (↑ ) for all metrics except Eff and SwAV (↓ ). Pair Method PixCorr↑ SSIM↑ Alex2↑ Alex5↑ Incep↑ CLIP↑ Eff↓ SwAV↓ Fwd%↑ Bwd%↑ 2→12→ 1 MED-VAE 0.182 0.349 0.868 0.941 0.880 0.826 0.790 0.465 85.0 65.6 SRM 0.134 0.350 0.828 0.906 0.783 0.746 0.859 0.517 78.5 71.0 Proc 0.123 0.352 0.831 0.909 0.795 0.728 0.866 0.523 69.9 62.4 5→15→ 1 MED-VAE 0.154 0.347 0.837 0.924 0.857 0.814 0.769 0.461 78.6 53.8 SRM 0.134 0.345 0.770 0.874 0.817 0.737 0.845 0.511 64.5 52.7 Proc 0.101 0.359 0.745 0.851 0.794 0.704 0.868 0.529 41.9 40.9 7→17→ 1 MED-VAE 0.132 0.315 0.796 0.888 0.827 0.795 0.819 0.492 59.1 53.8 SRM 0.077 0.321 0.717 0.805 0.717 0.693 0.895 0.554 45.2 36.2 Proc 0.077 0.332 0.678 0.753 0.665 0.641 0.919 0.564 35.5 25.8 1→21→ 2 MED-VAE 0.226 0.354 0.885 0.960 0.923 0.878 0.717 0.421 90.3 77.4 SRM 0.145 0.347 0.804 0.893 0.804 0.774 0.848 0.509 78.5 69.9 Proc 0.141 0.345 0.799 0.883 0.793 0.737 0.873 0.527 67.7 65.6 5→25→ 2 MED-VAE 0.145 0.334 0.842 0.932 0.897 0.821 0.790 0.476 71.0 62.4 SRM 0.131 0.344 0.752 0.859 0.815 0.746 0.847 0.527 61.3 56.3 Proc 0.130 0.348 0.737 0.842 0.772 0.699 0.861 0.539 44.1 43.0 7→27→ 2 MED-VAE 0.126 0.315 0.793 0.893 0.826 0.772 0.805 0.488 61.3 57.6 SRM 0.062 0.312 0.706 0.805 0.716 0.699 0.905 0.564 41.9 33.3 Proc 0.081 0.314 0.686 0.752 0.655 0.661 0.908 0.562 31.2 26.9 1→51→ 5 MED-VAE 0.170 0.353 0.882 0.944 0.911 0.881 0.712 0.418 90.3 74.2 SRM 0.101 0.342 0.799 0.864 0.775 0.716 0.877 0.528 76.9 71.0 Proc 0.085 0.337 0.783 0.871 0.774 0.701 0.892 0.539 51.6 48.4 2→52→ 5 MED-VAE 0.173 0.344 0.860 0.933 0.874 0.839 0.787 0.470 75.3 69.9 SRM 0.117 0.355 0.796 0.894 0.776 0.732 0.873 0.527 69.9 55.9 Proc 0.131 0.353 0.785 0.866 0.715 0.722 0.895 0.542 57.0 50.5 7→57→ 5 MED-VAE 0.121 0.306 0.799 0.886 0.842 0.784 0.814 0.492 64.5 55.9 SRM 0.097 0.311 0.718 0.809 0.726 0.682 0.913 0.558 36.6 35.5 Proc 0.085 0.319 0.708 0.765 0.628 0.643 0.925 0.568 31.2 21.5 1→71→ 7 MED-VAE 0.189 0.361 0.914 0.955 0.895 0.887 0.736 0.423 89.3 74.2 SRM 0.114 0.366 0.813 0.870 0.801 0.768 0.868 0.521 79.0 69.9 Proc 0.096 0.365 0.777 0.866 0.740 0.718 0.882 0.533 57.0 54.8 2→72→ 7 MED-VAE 0.163 0.342 0.851 0.923 0.864 0.830 0.788 0.453 81.7 65.6 SRM 0.136 0.367 0.815 0.890 0.736 0.742 0.844 0.503 68.8 62.4 Proc 0.131 0.368 0.795 0.838 0.760 0.727 0.864 0.528 54.7 57.0 5→75→ 7 MED-VAE 0.139 0.341 0.827 0.911 0.839 0.797 0.791 0.475 75.3 62.4 SRM 0.107 0.345 0.770 0.873 0.762 0.738 0.846 0.523 69.9 55.9 Proc 0.104 0.348 0.766 0.837 0.744 0.710 0.869 0.539 52.7 47.3 Mean MED-VAE 0.160 0.338 0.846 0.925 0.870 0.827 0.776 0.463 76.7 64.4 SRM 0.113 0.342 0.774 0.862 0.769 0.731 0.868 0.529 64.3 55.9 Proc 0.107 0.345 0.758 0.836 0.736 0.699 0.885 0.541 49.5 45.3 Table 14: MED-VAE 512D vs SRM 512D vs Procrustes 512D decoding performance (No Refinement). Bold indicates the best-performing method per pair. Higher is better (↑ ) for all metrics except Eff and SwAV (↓ ). Pair Method PixCorr↑ SSIM↑ Alex2↑ Alex5↑ Incep↑ CLIP↑ Eff↓ SwAV↓ Fwd%↑ Bwd%↑ 2→12→ 1 MED-VAE 0.192 0.366 0.906 0.966 0.923 0.854 0.746 0.431 94.6 85.0 SRM 0.175 0.361 0.896 0.948 0.854 0.837 0.791 0.479 96.8 98.9 Proc 0.187 0.366 0.896 0.947 0.853 0.808 0.794 0.477 94.6 95.7 5→15→ 1 MED-VAE 0.156 0.345 0.851 0.939 0.911 0.837 0.736 0.439 85.0 68.8 SRM 0.174 0.355 0.829 0.925 0.877 0.808 0.793 0.476 77.4 79.6 Proc 0.156 0.349 0.822 0.920 0.877 0.819 0.795 0.474 80.7 77.9 7→17→ 1 MED-VAE 0.138 0.323 0.838 0.932 0.868 0.849 0.776 0.460 73.1 67.4 SRM 0.126 0.334 0.795 0.866 0.755 0.763 0.858 0.518 61.8 59.1 Proc 0.118 0.336 0.798 0.886 0.757 0.735 0.867 0.528 61.3 62.4 1→21→ 2 MED-VAE 0.206 0.357 0.899 0.960 0.928 0.882 0.714 0.422 92.8 87.1 SRM 0.221 0.371 0.886 0.959 0.873 0.833 0.760 0.448 96.8 94.6 Proc 0.224 0.373 0.911 0.959 0.848 0.805 0.789 0.455 97.8 96.8 5→25→ 2 MED-VAE 0.146 0.345 0.839 0.950 0.915 0.863 0.734 0.445 88.2 73.1 SRM 0.143 0.350 0.842 0.927 0.873 0.831 0.796 0.478 80.6 78.5 Proc 0.156 0.354 0.850 0.952 0.872 0.852 0.789 0.469 79.6 76.3 7→27→ 2 MED-VAE 0.130 0.328 0.814 0.919 0.871 0.822 0.752 0.454 76.3 65.6 SRM 0.113 0.350 0.760 0.873 0.755 0.739 0.849 0.521 67.7 63.4 Proc 0.103 0.347 0.753 0.850 0.754 0.711 0.858 0.532 57.0 59.1 1→51→ 5 MED-VAE 0.170 0.344 0.883 0.961 0.929 0.877 0.711 0.423 87.1 80.6 SRM 0.145 0.357 0.890 0.929 0.848 0.809 0.808 0.481 93.5 88.2 Proc 0.137 0.357 0.864 0.929 0.840 0.771 0.820 0.486 91.4 87.1 2→52→ 5 MED-VAE 0.194 0.358 0.895 0.959 0.922 0.896 0.728 0.430 94.6 83.9 SRM 0.169 0.373 0.881 0.941 0.850 0.810 0.795 0.481 90.3 87.1 Proc 0.170 0.370 0.877 0.941 0.794 0.787 0.816 0.496 87.1 89.2 7→57→ 5 MED-VAE 0.134 0.320 0.825 0.921 0.906 0.828 0.759 0.452 71.0 65.0 SRM 0.121 0.345 0.794 0.848 0.746 0.723 0.848 0.517 57.0 63.4 Proc 0.101 0.339 0.749 0.809 0.703 0.709 0.874 0.531 59.1 57.0 1→71→ 7 MED-VAE 0.197 0.350 0.911 0.952 0.888 0.856 0.752 0.436 95.7 82.8 SRM 0.159 0.365 0.872 0.945 0.856 0.819 0.814 0.487 96.8 93.5 Proc 0.174 0.366 0.886 0.943 0.866 0.811 0.798 0.473 94.6 95.7 2→72→ 7 MED-VAE 0.188 0.352 0.892 0.949 0.905 0.846 0.748 0.432 92.5 86.0 SRM 0.183 0.361 0.896 0.945 0.847 0.814 0.795 0.460 93.5 90.3 Proc 0.173 0.358 0.897 0.934 0.858 0.800 0.801 0.467 92.5 93.5 5→75→ 7 MED-VAE 0.142 0.342 0.849 0.942 0.883 0.862 0.747 0.453 87.1 69.9 SRM 0.143 0.354 0.862 0.932 0.842 0.812 0.781 0.475 83.9 76.3 Proc 0.127 0.344 0.837 0.924 0.875 0.820 0.795 0.482 79.6 73.1 Mean MED-VAE 0.166 0.344 0.867 0.946 0.904 0.856 0.742 0.440 86.5 76.3 SRM 0.156 0.355 0.859 0.928 0.835 0.800 0.806 0.485 83.1 81.1 Proc 0.152 0.355 0.845 0.916 0.825 0.786 0.816 0.489 81.3 80.3 Table 15: MED-VAE 32D vs SRM 32D decoding performance (with default Refinement; tp=13, CFG=5). Bold indicates the better-performing method per pair. Higher is better (↑ ) for all metrics except Eff and SwAV (↓ ). Pair Method PixCorr↑ SSIM↑ Alex2↑ Alex5↑ Incep↑ CLIP↑ Eff↓ SwAV↓ Fwd%↑ Bwd%↑ 2→12→ 1 MED-VAE 0.193 0.370 0.860 0.929 0.856 0.851 0.787 0.456 85.0 65.6 SRM 0.144 0.371 0.805 0.906 0.764 0.789 0.873 0.520 78.5 71.0 5→15→ 1 MED-VAE 0.168 0.359 0.840 0.935 0.847 0.826 0.777 0.452 79.2 53.8 SRM 0.124 0.359 0.776 0.873 0.792 0.810 0.841 0.511 64.5 51.6 7→17→ 1 MED-VAE 0.122 0.340 0.809 0.885 0.808 0.811 0.816 0.482 59.1 53.8 SRM 0.059 0.358 0.752 0.817 0.706 0.701 0.894 0.549 45.2 35.5 1→21→ 2 MED-VAE 0.210 0.379 0.880 0.958 0.902 0.904 0.727 0.412 90.3 77.8 SRM 0.159 0.370 0.823 0.889 0.755 0.762 0.871 0.511 78.5 69.9 5→25→ 2 MED-VAE 0.120 0.364 0.834 0.928 0.840 0.840 0.803 0.468 71.0 62.4 SRM 0.087 0.370 0.746 0.886 0.811 0.785 0.841 0.497 61.3 56.3 7→27→ 2 MED-VAE 0.130 0.348 0.779 0.872 0.830 0.810 0.812 0.477 61.3 58.1 SRM 0.071 0.351 0.719 0.828 0.685 0.718 0.902 0.547 41.2 33.3 1→51→ 5 MED-VAE 0.195 0.373 0.903 0.960 0.924 0.907 0.680 0.399 90.3 74.2 SRM 0.154 0.375 0.796 0.886 0.792 0.782 0.861 0.515 76.3 71.0 2→52→ 5 MED-VAE 0.163 0.371 0.836 0.926 0.865 0.844 0.769 0.442 75.3 68.8 SRM 0.118 0.381 0.832 0.902 0.748 0.741 0.865 0.503 69.9 55.9 7→57→ 5 MED-VAE 0.122 0.344 0.838 0.891 0.835 0.826 0.792 0.477 64.5 55.9 SRM 0.055 0.350 0.753 0.830 0.690 0.696 0.894 0.552 36.6 34.8 1→71→ 7 MED-VAE 0.211 0.382 0.894 0.948 0.882 0.896 0.712 0.406 89.3 74.2 SRM 0.154 0.378 0.785 0.872 0.774 0.770 0.866 0.523 79.0 69.9 2→72→ 7 MED-VAE 0.186 0.369 0.856 0.914 0.865 0.893 0.754 0.440 81.7 65.6 SRM 0.144 0.387 0.807 0.889 0.778 0.786 0.842 0.504 68.8 62.4 5→75→ 7 MED-VAE 0.134 0.353 0.822 0.923 0.862 0.820 0.787 0.454 76.3 62.4 SRM 0.120 0.372 0.760 0.864 0.798 0.810 0.839 0.502 69.9 55.9 Mean MED-VAE 0.163 0.363 0.846 0.922 0.860 0.852 0.768 0.447 77.0 64.4 SRM 0.116 0.369 0.779 0.870 0.754 0.764 0.867 0.517 64.1 55.6 Table 16: MED-VAE 512D vs SRM 512D decoding performance (with default Refinement; tp=13, CFG=5). Bold indicates the better-performing method per pair. Higher is better for all metrics except Eff and SwAV (lower is better). Pair Method PixCorr↑ SSIM↑ Alex2↑ Alex5↑ Incep↑ CLIP↑ Eff↓ SwAV↓ Fwd%↑ Bwd%↑ 2→12→ 1 MED-VAE 0.215 0.382 0.888 0.948 0.915 0.896 0.752 0.435 93.9 85.0 SRM 0.214 0.377 0.882 0.948 0.827 0.847 0.794 0.463 96.8 98.9 5→15→ 1 MED-VAE 0.163 0.357 0.862 0.940 0.881 0.887 0.742 0.433 85.0 68.8 SRM 0.186 0.371 0.828 0.915 0.840 0.863 0.782 0.463 77.4 79.6 7→17→ 1 MED-VAE 0.128 0.349 0.842 0.909 0.848 0.864 0.770 0.452 73.1 68.8 SRM 0.104 0.364 0.787 0.858 0.773 0.735 0.848 0.516 61.8 59.1 1→21→ 2 MED-VAE 0.204 0.383 0.903 0.967 0.901 0.898 0.709 0.404 92.5 87.1 SRM 0.212 0.392 0.905 0.951 0.835 0.805 0.802 0.460 96.8 94.6 5→25→ 2 MED-VAE 0.130 0.367 0.848 0.937 0.887 0.862 0.760 0.443 88.2 73.1 SRM 0.120 0.382 0.820 0.916 0.840 0.843 0.804 0.478 80.6 78.5 7→27→ 2 MED-VAE 0.140 0.361 0.825 0.904 0.873 0.848 0.766 0.447 76.3 65.6 SRM 0.116 0.377 0.775 0.875 0.754 0.765 0.849 0.518 67.7 63.4 1→51→ 5 MED-VAE 0.202 0.370 0.908 0.963 0.895 0.903 0.713 0.417 87.1 80.6 SRM 0.174 0.376 0.887 0.946 0.838 0.795 0.783 0.464 93.5 88.2 2→52→ 5 MED-VAE 0.191 0.383 0.885 0.959 0.900 0.897 0.729 0.421 94.6 83.9 SRM 0.180 0.392 0.876 0.934 0.849 0.802 0.788 0.465 90.3 87.1 7→57→ 5 MED-VAE 0.133 0.358 0.858 0.926 0.872 0.867 0.731 0.432 71.0 64.5 SRM 0.088 0.368 0.785 0.841 0.692 0.762 0.827 0.507 57.0 63.4 1→71→ 7 MED-VAE 0.217 0.374 0.896 0.959 0.883 0.874 0.730 0.421 95.7 82.8 SRM 0.200 0.388 0.890 0.941 0.841 0.814 0.795 0.470 96.8 93.5 2→72→ 7 MED-VAE 0.205 0.377 0.891 0.944 0.883 0.869 0.731 0.425 92.5 86.0 SRM 0.174 0.382 0.870 0.922 0.835 0.829 0.794 0.462 93.5 90.3 5→75→ 7 MED-VAE 0.149 0.356 0.833 0.941 0.903 0.860 0.739 0.430 87.1 69.9 SRM 0.128 0.372 0.816 0.914 0.847 0.846 0.783 0.462 83.9 76.3 Mean MED-VAE 0.173 0.368 0.870 0.941 0.887 0.877 0.739 0.430 86.4 76.3 SRM 0.158 0.379 0.844 0.919 0.813 0.809 0.805 0.477 83.1 81.1 Figure 6: Cross-subject image reconstruction comparison between VAE and SRM alignment methods. Each block shows reconstructions for a single stimulus GT (ground truth shown in the leftmost column) across all 12 subject transfer pairs (i→j). For each stimulus, the top row shows VAE reconstructions and the bottom row shows SRM reconstructions. Both methods use 512-dimensional latent representations with nearest-neighbor refinement (k=5)