Paper deep dive
Social-JEPA: Emergent Geometric Isomorphism
Haoran Zhang, Youjin Wang, Yi Duan, Rong Fu, Dianyu Zhao, Sicheng Fan, Shuaishuai Cao, Wentao Guo, Xiao Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/20/2026, 5:25:38 AM
Summary
The paper introduces Social-JEPA, demonstrating that independently trained Joint-Embedding Predictive Architecture (JEPA) world models from distinct viewpoints converge to latent spaces that are geometrically isomorphic via an approximate linear transformation. This emergent property enables lightweight interoperability, allowing for zero-cost probe sharing and accelerated learning through representation migration without parameter sharing or raw data exchange.
Entities (8)
Relation Signals (7)
Social-JEPA → uses → JEPA
confidence 98% · Social-JEPA allows separate agents to learn world models from disparate observations; they converge to isomorphic latent spaces via the JEPA objective
Social-JEPA → enables → Alignment Map W
confidence 96% · Leveraging the learned alignment, a classifier trained on one agent can be ported to the other... The alignment map W is used as a lightweight interface
Social-JEPA → evaluatedon → nuScenes
confidence 95% · Across smallNORB and nuScenes, isomorphism persists
Social-JEPA → evaluatedon → ImageNet-1K
confidence 95% · On ImageNet-1k, we further show that JEPA yields substantially stronger cross-model alignability
JEPA → produces → Geometric Isomorphism
confidence 95% · JEPA yields substantially stronger cross-model alignability... independent JEPA models often develop latent spaces that are nearly isomorphic
Social-JEPA → comparedto → SimCLR
confidence 90% · Middle: SimCLR/Contrastive uses data augmentations... Right: Social-JEPA (Ours)
Social-JEPA → comparedto → MAE
confidence 90% · Social-JEPA (Ours) allows separate agents to learn world models... Left: MAE/AE relies on a reconstruction loss
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:World models compress rich sensory streams into compact latent codes that anticipate future observations. We let separate agents acquire such models from distinct viewpoints of the same environment without any parameter sharing or coordination. After training, their internal representations exhibit a striking emergent property: the two latent spaces are related by an approximate linear isometry, enabling transparent translation between them. This geometric consensus survives large viewpoint shifts and scant overlap in raw pixels. Leveraging the learned alignment, a classifier trained on one agent can be ported to the other with no additional gradient steps, while distillation-like migration accelerates later learning and markedly reduces total compute. The findings reveal that predictive learning objectives impose strong regularities on representation geometry, suggesting a lightweight path to interoperability among decentralized vision systems. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.02263v2
- Canonical: https://arxiv.org/abs/2603.02263v2
PDF not stored locally. Use the link above to view on the source site.
Full Text
63,915 characters extracted from source content.
Expand or collapse full text
Social-JEPA: Emergent Geometric Isomorphism in Independently Trained World Models Haoran Zhang Youjin Wang Yi Duan Rong Fu Dianyu Zhao Sicheng Fan Shuaishuai Cao Wentao Guo Xiao Zhou Abstract World models compress rich sensory streams into compact latent codes that anticipate future observations. We let separate agents acquire such models from distinct viewpoints of the same environment without any parameter sharing or coordination. After training, their internal representations exhibit a striking emergent property: the two latent spaces are related by an approximate linear isometry, enabling transparent translation between them. This geometric consensus survives large viewpoint shifts and scant overlap in raw pixels. Leveraging the learned alignment, a classifier trained on one agent can be ported to the other with no additional gradient steps, while distillation-like migration accelerates later learning and markedly reduces total compute. The findings reveal that predictive learning objectives impose strong regularities on representation geometry, suggesting a lightweight path to interoperability among decentralized vision systems. The code is available at https://anonymous.4open.science/r/Social-JEPA-5C57/. world models, self-supervised learning, representation alignment, JEPA 1 Introduction Figure 1: The JEPA framework. The model predicts the representation of a target signal from a context signal using a predictor network, with the loss computed in latent space. Building a world model is widely seen as a key ingredient for autonomy and planning (ha2018world). A capable model must distill stable, abstract, and predictive structure from raw observations. While many approaches rely on reconstruction objectives such as Masked Autoencoders (MAE) (he2022masked), Joint-Embedding Predictive Architectures (JEPA) instead train models to predict representations in latent space (lecun2022path; assran2023jepa). Crucially, by shifting the objective from pixel-level recovery to latent-space prediction, JEPA naturally filters out view-specific nuisances. This allows the model to prioritize environment-level regularities and capture the underlying semantic structure of the world (bardes2021vicreg; grill2020byol). Figure 2: Comparison of World Model Training Paradigms. Left: MAE/AE relies on a reconstruction loss ℒReconL_Recon to recover input pixels. Middle: SimCLR/Contrastive uses data augmentations and an InfoNCE loss to learn view-invariant features. Right: Social-JEPA (Ours) allows separate agents to learn world models from disparate observations; they converge to isomorphic latent spaces via the JEPA objective ℒJEPAL_JEPA without sharing raw data. Despite rapid progress, traditional evaluations of world models and self-supervised representations remain atomistic, typically confined to a single model, dataset, and augmentation pipeline. Yet many practical settings are inherently decentralized, such as those involving multiple sensors, multiple viewpoints, or multiple training pipelines, where models cannot share raw data or parameters but can still benefit from a lightweight interface. This raises a basic interoperability question: if we train multiple JEPA models independently on different observation functions of the same underlying environment, do they learn compatible latent geometries? We study this question through what we call Social-JEPA. Here, social refers to interoperability between independently trained models; we assume no interaction, communication protocol, or cross-view coupling during pretraining. Empirically, we find that independent JEPA models often develop latent spaces that are nearly isomorphic: there exists an invertible linear transformation W such that, for shared states s, z(2)(s)≈Wz(1)(s)z^(2)(s)≈ Wz^(1)(s). Such a map serves as a compact “translation layer” between models and can be learned post hoc from paired samples, enabling coordination without exchanging raw observations. Across smallNORB and nuScenes, isomorphism persists even under severe viewpoint changes and near-zero pixel overlap. On ImageNet-1k, we further show that JEPA yields substantially stronger cross-model alignability than representative reconstructive and contrastive pretraining paradigms. These findings align with the broader push toward latent predictive learning for world models, including V-JEPA (bardes2024vjepa). Our contributions are threefold: ❶ Discovery of Social-JEPA. We identify and formalize the spontaneous emergence of geometric isomorphism across independent world models trained on shared environments. This discovery reveals that decentralized models, despite having no parameter sharing or coordination, naturally converge to a common latent geometry, providing a foundational social link for interoperability. ❷ Theoretical Grounding. We provide a rigorous explanation for this phenomenon, tracing it to the predictive sufficiency and linear equivalence invariance of the JEPA objective. This framework demonstrates that latent predictive learning forces independently trained models into the same linear equivalence class, explaining why they are inherently alignable via simple linear maps. ❸ Practical Utility. We showcase the power of this isomorphism through collaboration primitives, including “zero-cost” probe sharing and accelerated representation migration (achieving target accuracy with only 0.28× FLOPs). These results validate the potential for multi-robot collaborative world modeling where agents exchange lightweight alignment maps W instead of raw, high-bandwidth sensory data. 2 Related Work 2.1 World Models and JEPA World models aim to learn predictive abstractions that support reasoning and planning (ha2018world). JEPA (lecun2022path) provides a prediction-centric alternative to pixel-level reconstruction and has been instantiated for images (I-JEPA) (assran2023jepa) and video (V-JEPA) (bardes2024vjepa; assran2025vjepa2). Related JEPA variants extend the framework to additional modalities (feichtenhofer2023audiojepa) and multi-modal learning (girdhar2024mcjepa). These works primarily focus on training a single strong world model; in contrast, we study the relationship between multiple independently trained world models exposed to different views of the same underlying environment. Self-supervised objectives and induced geometry. Self-supervised learning (SSL) spans reconstructive objectives such as MAE (he2022masked), contrastive learning such as SimCLR (chen2020simclr), and non-contrastive/self-distillation approaches (grill2020byol; bardes2021vicreg; caron2021dino; chen2021mocov3; zhou2022ibot). These objectives emphasize different invariances and can differ in representation scaling and anisotropy, which may affect how well independently trained models can be related by a simple linear map. We therefore compare representative paradigms under the same alignment protocol to isolate how the pretraining objective shapes emergent isomorphism. 2.2 Representation Similarity and Interoperability The geometry of learned representations can be compared using metrics such as CKA (kornblith2019similarity) and CCA-based analyses (raghu2017svcca; morcos2018insights). Beyond diagnostic similarity measures, model stitching (bansal2021stitching) and multi-way alignment (kossaifi2020multi) probe functional interchangeability across networks. Related work on relative representations studies how to align independently learned latent spaces for interoperability (moschella2023relative). We complement this line of work by showing that JEPA world models can spontaneously become linearly interoperable, and we leverage the resulting linear maps as lightweight collaboration interfaces. Symmetries, model merging, and equivalence classes. Neural networks exhibit symmetries and non-identifiability (garipov2018loss; pmlr-v80-draxler18a), and weight matching methods show that independently trained networks can often be aligned in parameter space (ainsworth2023gitrebasin). Our focus is complementary: rather than matching weights, we study representation-space equivalence induced by JEPA (formalized in Section D.3). This view motivates using a post hoc alignment map W as a lightweight interoperability interface. 2.3 Multi-View Learning and Knowledge Sharing Multi-view representation learning typically assumes coupled training signals (paired views, contrastive objectives, or explicit cross-view constraints), whereas our setting enforces no coupling during pretraining and studies emergent alignment post hoc. From a systems perspective, our alignment map W can be viewed as a compact communication layer: models trained on different views can exchange small linear maps (or map-transformed parameters) instead of raw data. Unlike coupling-based multi-view approaches, we show that linear interoperability can emerge even with fully independent pretraining, and we then treat the learned map as a practical interface. 3 Methodology 3.1 Problem Definition We consider a setting in which multiple world models are learned from distinct observation channels of the same underlying environment, with no shared parameters or cross-view objectives at training time. Let S denote the set of semantic states of the environment. For each view index i, observations are generated by a view-specific function gi:→(i)g_i ^(i), so that x(i)=gi(s)x^(i)=g_i(s) is the observation of state s∈s in view i. The central question is whether encoders f(1)f^(1) and f(2)f^(2), trained independently under the same predictive objective but on different observation functions g1g_1 and g2g_2, produce latent spaces that are related by an invertible linear map. Formally, we ask whether there exists W∈ℝd×dW ^d× d such that z(2)(s)≈Wz(1)(s)z^(2)(s)≈ Wz^(1)(s) for z(i)(s)=f(i)(gi(s))z^(i)(s)=f^(i)(g_i(s)) when s is drawn from the environment distribution. This formulation is operational: we estimate W from a training set of paired states and evaluate isomorphism on a held-out test set. 3.2 Joint Embedding Predictive Architecture A Joint Embedding Predictive Architecture (JEPA) learns structure that is predictive of targets in latent space rather than reconstructing inputs (lecun2022path; assran2023jepa). For a context signal xcx_c and a target signal xtx_t, a single encoder fθf_θ produces latent vectors zc=fθ(xc)z_c=f_θ(x_c) and zt=fθ(xt)z_t=f_θ(x_t). A predictor pϕp_φ maps the context embedding to a prediction of the target embedding. Training minimizes the squared error between the predicted and the (stop-gradient) target embedding in latent space: ℒJEPA=‖pϕ(zc)−sg(zt)‖22,L_JEPA= \|p_φ(z_c)-sg(z_t) \|_2^2, (1) where sg(⋅)sg(·) denotes the stop-gradient operator, zc=fθ(xc)z_c=f_θ(x_c), and zt=fθ(xt)z_t=f_θ(x_t). Because the objective is defined only in latent space, the coordinate system is not uniquely determined: at low prediction error, any invertible linear reparameterization of the latents yields nearly the same loss. At zero prediction error, the JEPA objective is invariant under the group of invertible linear maps: if (f,p)(f,p) is optimal, then for any A∈GL(d)A∈ GL(d) the reparameterization fA(x)≜Af(x)f_A(x) Af(x) and pA(z)≜Ap(A−1z)p_A(z) A\,p(A^-1z) also attains zero loss (see Section D.3). This non-identifiability implies that independently trained models can converge to representations that differ only by a linear transformation, which motivates post hoc linear alignment. 3.3 Geometric Isomorphism: Definition and Testable Criterion We say that the latent spaces of two encoders f(1)f^(1) and f(2)f^(2) are geometrically isomorphic when there exists an invertible linear map W∈GL(d)W∈ GL(d) such that z(2)(s)≈Wz(1)(s)for s∼,z^(2)(s)≈ Wz^(1)(s) s , (2) where z(i)(s)=f(i)(gi(s))z^(i)(s)=f^(i)(g_i(s)) is the latent representation of state s in view i. The relation is evaluated empirically by fitting W on a training split of paired states and reporting metrics on a disjoint test split, so that isomorphism is an operational, testable property rather than a purely theoretical one. 3.4 Independent Training Setting As illustrated in Figure 2, Social-JEPA does not use reconstructive losses (e.g., MAE) or contrastive cross-view coupling (e.g., SimCLR). For each view i, a separate JEPA model with parameters (f(i),p(i))(f^(i),p^(i)) is trained using only the latent prediction objective in Figure 1. There is no parameter sharing and no cross-view loss during pretraining; any alignment between models must arise from the shared environment distribution and the common predictive objective. Thus the setting is strictly decentralized at training time. 3.5 Alignment Map Estimation Given two trained encoders and a collection of paired observations (xn(1),xn(2))n=1N\(x_n^(1),x_n^(2))\_n=1^N corresponding to the same underlying states, we extract latents zn(1)=f(1)(xn(1))z_n^(1)=f^(1)(x_n^(1)) and zn(2)=f(2)(xn(2))z_n^(2)=f^(2)(x_n^(2)). The alignment map W is estimated by minimizing the mean squared error between z(2)z^(2) and Wz(1)Wz^(1) over the training set: W∗=argminW∑n=1N‖zn(2)−Wzn(1)‖22.W^*= _W _n=1^N \|z_n^(2)-Wz_n^(1) \|_2^2. (3) Here W∗W^* denotes the optimal alignment matrix and zn(i)∈ℝdz_n^(i) ^d is the latent of the n-th paired sample in view i. When W is restricted to be orthogonal, the solution is given by Procrustes alignment (gower2004procrustes). For numerical stability we use ridge regression when needed; with regularization parameter λ≥0λ≥ 0, the closed-form solution is W∗=YX⊤(XX⊤+λI)−1,W^*=YX (X +λ I )^-1, (4) where X∈ℝd×NX ^d× N and Y∈ℝd×NY ^d× N denote the matrices whose columns are the standardized training latents zn(1)z_n^(1) and zn(2)z_n^(2) respectively, and I is the d×d× d identity matrix. When W is ill-conditioned we use the Moore–Penrose pseudoinverse W†W in place of W−1W^-1 (details in the appendix). The full procedure for estimating W and evaluating isomorphism is summarized in Algorithm 1, which invokes Equation 3 (or its ridge form Equation 4) for fitting and uses the metrics defined in Section 3.6. Algorithm 1 Estimating the alignment map and evaluating isomorphism 1: Input: Paired samples (xn(1),xn(2))n=1N\(x_n^(1),x_n^(2))\_n=1^N, encoders f(1),f(2)f^(1),f^(2), neighborhood size k, ridge λ≥0λ≥ 0, conditioning threshold τ 2: Split pairs into train trD_tr and test teD_te (by underlying state when available) 3: Extract latents z(i)=f(i)(x(i))z^(i)=f^(i)(x^(i)) for (x(1),x(2))∈tr∪te(x^(1),x^(2)) _tr _te 4: Compute per-dimension mean and variance on trD_tr for each view and standardize train and test latents 5: Fit W on trD_tr via Equation 3 (ridge form Equation 4 or Procrustes when W⊤W=IW W=I) 6: Compute κ(W)κ(W); set Winv←W−1W_inv← W^-1 if κ(W)≤τκ(W)≤τ, else Winv←W†W_inv← W 7: Global: On teD_te, compute MSE (Equation 6) and R2R^2 between z(2)z^(2) and Wz(1)Wz^(1) 8: Geometry: Form aligned latents z~(2)←Winvz(2) z^(2)← W_invz^(2) (Equation 5) and compute DSC, NOS@k, and linear CKA between z(1)\z^(1)\ and z~(2)\ z^(2)\ 9: Output: W, κ(W)κ(W), and isomorphism metrics Exchanging the alignment map between agents costs d2d^2 scalars. For ViT-S/16 with d=384d=384, this is about 1.47×1051.47× 10^5 scalars (under 1%1\% of the backbone) and roughly 0.60.6 MB in float32, so W is a lightweight interface compared to sharing weights or raw observations. Computing W∗W^* from N pairs via ridge regression requires forming d×d× d moment matrices and solving a d×d× d system (O(d2N)+O(d3)O(d^2N)+O(d^3)); orthogonal Procrustes requires one d×d× d SVD (O(d3)O(d^3)). In collaboration protocols that periodically refresh W, the overhead is limited to these operations and the communication of W or aligned latents. 3.6 Isomorphism Metrics We quantify isomorphism using global and local metrics on held-out paired representations (zn(1),zn(2))n=1N\(z_n^(1),z_n^(2))\_n=1^N, with W fitted on a train split. For metrics that require a common coordinate system (DSC, NOS@k, CKA), we map model-22 latents into model-11 coordinates via z~n(2)≜Winvzn(2), z_n^(2) W_invz_n^(2), (5) where Winv=W−1W_inv=W^-1 when W is well-conditioned and Winv=W†W_inv=W otherwise; zn(2)z_n^(2) is the latent of the n-th test sample in view 22. Distances and nearest neighbors are computed in Euclidean distance on standardized features (using train-set statistics). Global linear alignability. We report the mean squared alignment error on the test set, MSE≜1|te|∑n∈te‖zn(2)−Wzn(1)‖22,MSE 1|D_te| _n _te \|z_n^(2)-Wz_n^(1) \|_2^2, (6) where zn(i)z_n^(i) is the latent of the n-th test pair in view i and W is the map fitted on the train split. We also report the coefficient of determination R2R^2 on held-out pairs, i.e., the fraction of variance in z(2)z^(2) explained by the best linear map from z(1)z^(1); higher R2R^2 indicates stronger linear alignability, with R2=1R^2=1 for perfect alignment and R2=0R^2=0 for no linear relationship. Scale-invariant metrics such as R2R^2, CKA, and DSC are reported throughout; we additionally report Procrustes error under orthogonality constraints (gower2004procrustes). Representation similarity. We report linear CKA between zn(1)\z_n^(1)\ and z~n(2)\ z_n^(2)\ on the test split (kornblith2019similarity). CKA is invariant to isotropic scaling and orthogonal transformations and is computed in the common coordinate system given by Equation 5. Distance-structure consistency. Distance-structure consistency (DSC) is the Spearman rank correlation between pairwise distances in the two spaces: between ‖zi(1)−zj(1)‖2\|z_i^(1)-z_j^(1)\|_2 and ‖z~i(2)−z~j(2)‖2\| z_i^(2)- z_j^(2)\|_2 over test pairs (i,j)(i,j), measuring whether relative proximity is preserved (kossaifi2020multi). DSC is O(N2)O(N^2) in the number of test points; we compute it on the full test split when feasible and otherwise on a subsample. Neighborhood overlap and disagreement. For each test sample, let k(1)N_k^(1) be its k-nearest-neighbor set in z(1)z^(1) and k(2)N_k^(2) the set in the aligned space z~(2) z^(2). Neighborhood overlap is NO@k≜1k|k(1)∩k(2)|,NO@k 1k |N_k^(1) _k^(2) |, (7) where k is the neighborhood size and larger values indicate better agreement. We report the complement NOS@k≜1−NO@k,NOS@k 1-NO@k, (8) so that smaller NOS@k is better and tables can use a consistent “↓ ” convention (kossaifi2020multi; moschella2023relative). 3.7 Collaboration Primitives Figure 3: Overview of Social-JEPA. Takeaway: independently trained world models exposed to different observation functions can converge to isomorphic latent structures. How to read: a post hoc linear map W serves as a compact translation layer, enabling plug-and-play probe transfer and representation migration without sharing raw observations. The alignment map W is used as a lightweight interface for collaboration across independently trained models. Zero-cost probe sharing. Suppose a linear probe q(z)=a⊤zq(z)=a z is trained on model-11 representations. Using z(1)≈W−1z(2)z^(1)≈ W^-1z^(2), the probe can be transferred to model 22 by setting a(2)=W−⊤a^(2)=W^- a with no further gradient steps. Transfer is reliable when W generalizes out-of-sample and is well-conditioned; otherwise we use W†W and report conditioning diagnostics in the appendix. Transfer can fail when κ(W)κ(W) is large, when W does not generalize, or when the downstream task depends on features not preserved by the linear map; we diagnose these via conditioning and out-of-distribution transfer experiments (Figure 9). Teacher–student representation migration. To speed up training of a student world model, we alternate between estimating W between teacher and student representations on paired samples and updating the student with an auxiliary alignment loss in latent space while keeping the JEPA prediction objective. Details are in Algorithms 2 and A. We measure savings by the training cost (e.g., FLOPs) needed for the student to reach a fixed downstream accuracy. As in Table 9, representation migration reaches 85% accuracy in 35 epochs (0.28×0.28× FLOPs) versus 150 epochs from scratch. This form of supervision is label-free and operates on representations, in contrast to standard knowledge distillation (hinton2015distilling). Mutual teaching. A stronger collaborative variant adds an online cross-model consistency term, ℒcross=‖z(2)−Wz(1)‖22,L_cross= \|z^(2)-Wz^(1) \|_2^2, (9) where z(1)z^(1) and z(2)z^(2) are latents of paired observations and W is the current alignment map, so that both models stay in a common equivalence class during training (zhang2018deep). This requires synchronized paired samples and communication of W or aligned latents but can reduce the number of epochs to convergence. In Table 9, mutual teaching reaches the 85% threshold in 55 epochs (vs. 120 for independent training) at 1.12×1.12× FLOPs. Unified pseudocode for probe migration, teacher–student migration, and mutual teaching is given in Algorithm 2. 3.8 Theoretical Rationale for Emergent Isomorphism We summarize why independently trained JEPA models can converge to linearly related latent spaces, drawing on predictive sufficiency and the linear symmetry of the objective (roeder2021linear). Predictive sufficiency. The JEPA objective minimizes expected prediction error in latent space (lecun2022path; assran2023jepa): ℒ(f,p)=xc,xt[‖p(f(xc))−f(xt)‖22],L(f,p)=E_x_c,x_t [ \|p(f(x_c))-f(x_t) \|_2^2 ], (10) where xcx_c and xtx_t are context and target observations. Under the assumption that observations are generated from an environmental state with noise, the optimal encoder is a predictive sufficient statistic for the target (tishby2000information; tishby2015deep; achille2018emergence). Two optimal JEPA encoders trained on the same environment therefore tend to capture the same predictive structure. Linear equivalence and non-identifiability. The latent coordinate system is not unique: at zero prediction error, the objective is invariant under GL(d)GL(d). For any invertible A∈GL(d)A∈ GL(d), the pair f~(x)=Af∗(x),p~(z)=Ap∗(A−1z), f(x)=Af^*(x), p(z)=Ap^*(A^-1z), (11) where f∗f^* and p∗p^* denote an optimal encoder and predictor, also attains zero loss (see Appendix). Orthogonal transforms preserve the squared error exactly. Thus independent models can converge to different coordinate systems of the same predictive structure, supporting the existence of an approximately invertible linear map W between their representations. Linear latent-state model. Suppose there is a shared latent state u(s)∈ℝdu(s) ^d with full-rank covariance such that z(1)(s)=A1u(s),z(2)(s)=A2u(s)+ϵ(s),z^(1)(s)=A_1u(s), z^(2)(s)=A_2u(s)+ε(s), (12) where A1,A2∈GL(d)A_1,A_2∈ GL(d) and [ϵ(s)∣u(s)]=0E[ε(s) u(s)]=0. Then the population minimizer of Equation 3 is W⋆=A2A1−1W =A_2A_1^-1; alignment is perfect when ϵ≡0ε≡ 0 and degrades with ‖ϵ‖22E\|ε\|_2^2. Moreover κ(W⋆)≤κ(A2)κ(A1)κ(W )≤κ(A_2)κ(A_1), so stable interoperability is linked to well-conditioned coordinate changes. This model illustrates why a single linear map can generalize when both encoders capture the same predictive factors and differ mainly by a linear change of coordinates. Assumptions and failure modes. The above explanation assumes that both models are trained on the same environment distribution and learn the same predictive factors, that paired states are available with limited noise, that representations are not severely rank-deficient, and that optimization does not push the two models into incompatible nonlinear coordinates. When these assumptions fail, W can become ill-conditioned or low-rank and isomorphism metrics degrade. We provide diagnostic experiments and counterexamples in the Appendix (Figures 9 and 7). 4 Experiments 4.1 Experimental Setup Datasets. We evaluate cross-view alignment on smallNORB (lecun2004learning) (large viewpoint gaps), nuScenes (caesar2020nuscenes) (disjoint camera views), and ImageNet-1k (different augmentation pipelines). Paired states. For smallNORB, we treat images of the same object instance under matched factors (e.g., lighting/elevation) as the same state and vary only azimuth. For nuScenes, we pair synchronized frames from different cameras at the same timestamp. For ImageNet-1k, we pair two views produced by different augmentation pipelines of the same image. Models and training. Unless stated otherwise, we use a ViT backbone (dosovitskiy2020vit) and train JEPA models using an I-JEPA style objective (assran2023jepa). We compare against representative SSL paradigms, including reconstructive pretraining (MAE (he2022masked)) and contrastive/self-distillation baselines (SimCLR (chen2020simclr), DINO (caron2021dino), MoCo v3 (chen2021mocov3), iBOT (zhou2022ibot)). Evaluation protocol. We split paired states into train/test, fit W on the train split, and report all metrics on held-out pairs using Section 3.6. For numerical stability and comparability across methods, we standardize representations using train statistics (details in the appendix). 4.2 Main Results: Emergent Isomorphism Across Views Table 1: Social-JEPA yields strong latent isomorphism across views and datasets. Top: cross-view alignment. Middle: comparison across learning paradigms. Bottom: ablation on environmental structure. !20 Category Experiment / Setting MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ ImageNet (Data Augmentation) 0.041 0.873 0.846 0.31 !5 smallNORB (Viewpoint 0∘0 vs 160∘160 ) 0.036 0.891 0.872 0.27 nuScenes (Front vs Rear) 0.058 0.852 0.821 0.34 !5 Cross-View nuScenes (Front vs Side) 0.062 0.838 0.804 0.36 [HTML]F0C1 Social-JEPA (Ours) 0.036 0.8908 0.872 0.27 MAE (Reconstructive) 0.041 0.8823 0.845 0.31 !5 SimCLR (Contrastive) 0.112 0.6778 0.641 0.52 Paradigms AutoEncoder (Pixel-level) 0.138 0.6109 0.593 0.58 Normal Environment (Structured) 0.036 0.8908 0.872 0.27 !5 Patch Shuffle (Spatial Broken) 0.162 0.7200 0.540 0.69 Structure Pixel Shuffle (Total Destruction) 0.241 0.6158 0.280 0.86 Interpretation. Table 1 shows that independently trained JEPA world models are highly alignable across disparate observation functions: a single post hoc linear map achieves strong global fit (R2R^2, DSC) and consistent local neighborhoods (low NOS@10) on held-out pairs. Notably, alignability persists under extreme viewpoint gaps (smallNORB 0∘0 vs. 160∘160 ) and disjoint camera views in nuScenes, including settings with minimal pixel overlap, suggesting that the latent spaces encode shared predictive structure rather than raw appearance. The paradigm comparison reveals that JEPA achieves the best overall balance across metrics (MSE=0.036, R2R^2=0.891, DSC=0.872, NOS@10=0.27), outperforming reconstructive (MAE) and contrastive (SimCLR) baselines on smallNORB. We report both global and local metrics because a model can admit a good average linear fit while still distorting nearest-neighbor relations (or vice versa); the strong performance across all metrics indicates that JEPA’s isomorphism is both globally and locally consistent. Since NOS@k=1−NO@k,NOS@k=1-NO@k, (13) lower NOS@k indicates higher neighborhood overlap and thus better local topology agreement. The structure ablation (patch/pixel shuffle) demonstrates that alignability collapses when spatial structure is destroyed, confirming that isomorphism emerges from capturing stable environmental regularities rather than arbitrary feature correlations. 4.3 Ablations: Objectives and Environmental Structure Across paradigms (see Tables 1 and 2), emergent alignability is not unique to JEPA: multiple SSL objectives can yield strongly alignable representations on smallNORB, with different trade-offs across metrics. Notably, methods differ in their trade-offs between global linear fit (R2R^2/DSC) and local topology agreement (NOS@10), motivating our use of complementary metrics rather than a single score. When we destroy spatial structure by shuffling patches or pixels, alignability collapses, indicating that Social-JEPA emerges only when the model captures stable structure. This gap is consistent with the broader observation that contrastive objectives can induce geometric trade-offs (e.g., alignment vs. uniformity), while predictive objectives may bias representations toward shared latent structure (pmlr-v119-wang20k). 4.4 Baseline Comparison: ImageNet-1k and smallNORB Table 2: Baseline comparison of isomorphism metrics across SSL paradigms on ImageNet-1k and smallNORB. Takeaway: on ImageNet-1k, JEPA yields stronger cross-model alignability than representative reconstructive/contrastive/self-distillation baselines. !20 Benchmark Method MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ [HTML]F0C1 Social-JEPA (Ours) 0.091 0.489 0.584 0.033 DINO (caron2021dino) 0.662 0.283 0.391 0.102 !5 MoCo v3 (chen2021mocov3) 54.343 0.358 0.521 0.086 iBOT (zhou2022ibot) 0.662 0.283 0.391 0.102 !5 MAE (he2022masked) 10.023 0.409 0.450 0.028 ImageNet-1k SimCLR (chen2020simclr) 13.474 0.437 0.558 0.072 [HTML]F0C1 Social-JEPA (Ours) 0.036 0.8908 0.872 0.27 DINO (caron2021dino) 93.641 0.811 0.913 0.046 !5 MoCo v3 (chen2021mocov3) 0.478 0.987 0.994 0.125 iBOT (zhou2022ibot) 24.433 0.851 0.916 0.130 !5 MAE (he2022masked) 0.041 0.8823 0.845 0.31 smallNORB SimCLR (chen2020simclr) 0.112 0.6778 0.641 0.52 Interpretation. Table 2 isolates the effect of the pretraining objective by comparing methods under the same alignment protocol and backbone. On ImageNet-1k, JEPA yields stronger cross-model alignability than representative reconstructive and contrastive/self-distillation baselines when judged by scale-robust metrics (R2R^2, DSC) and local topology (NOS@10). We caution that raw MSE can vary dramatically across paradigms (e.g., MoCo v3 achieves MSE=54.343) because representation norms and anisotropy differ; accordingly, we interpret MSE together with R2R^2/DSC and NOS@10. For MoCo v3 specifically, the large MSE reflects anisotropic feature scaling and potentially different representation norms, but the scale-invariant metrics (R2R^2=0.358, DSC=0.521, NOS@10=0.086) reveal that the underlying geometric structure is still partially alignable, though substantially weaker than JEPA’s R2R^2=0.489, DSC=0.584, NOS@10=0.033. This discrepancy highlights why scale-invariant metrics are essential for cross-paradigm comparison: methods may learn representations with different norms while still preserving relative structure. Finally, some baselines may exhibit competitive performance on one axis (global fit vs. local neighborhoods), reinforcing the need for complementary metrics rather than a single score. 4.5 Additional Analyses and Visualizations To better characterize the geometry of Social-JEPA, we include several complementary analyses, with all corresponding plots provided in the appendix. These include two-dimensional projections before and after alignment using PCA and UMAP, which visually confirm that the learned mapping W aligns global geometry. We also present paradigm-level bar charts on ImageNet-1k and smallNORB that summarize the results in Table 2. In addition, layerwise CKA heatmaps (kornblith2019similarity) are used to localize where isomorphism emerges along the depth of the network. To further diagnose the nature of the alignment, we analyze the singular value spectra of W, which reveal whether the learned mapping is near-orthogonal, ill-conditioned, or effectively low-rank. Finally, we report pair-budget curves together with conditioning analyses to quantify how many paired samples are required to achieve stable alignment. Collectively, these analyses validate that the observed isomorphism is not a metric artifact and clarify the regimes in which linear alignment is well posed. All plots are included in the appendix; see Figures 6, 7, 8, 9 and 5. 4.6 Downstream Applications Downstream protocol and cost accounting. We evaluate collaboration benefits on a fixed labeled downstream task using a linear probe on frozen encoder representations. For teacher–student migration and mutual teaching, we report the earliest pretraining epoch at which the downstream probe exceeds an accuracy threshold (85% in Table 9) on a held-out split. Training cost is the cumulative pretraining FLOPs up to that epoch, normalized by training from scratch under the same backbone and data pipeline; the one-time matrix solve for estimating W is accounted for separately as a lightweight overhead (Algorithm 1). For probe sharing, we report target-model probe accuracy after transferring a linear head from model 1 to model 2, and the additional target-side training epochs (0 for analytic transfer). The A-probe baseline applies the source head to model 2 directly without translation or adaptation. Interpretation. In teacher–student migration, 0.28×0.28× denotes the normalized pretraining cost to reach the same downstream probe threshold: the student reaches 85% probe accuracy using 28% of the FLOPs required by training from scratch under the same backbone, data, and evaluation protocol. This gain comes from using W as a translation layer that steers the student toward the teacher’s learned predictive structure, reducing the time spent rediscovering an equivalent latent coordinate system. Probe sharing is “zero-cost” on the target side once W is estimated: the linear head parameters are migrated analytically (a(2)=W−⊤a^(2)=W^- a) with no additional SGD steps. Mutual teaching trades stronger coupling (paired sampling and communication of W or aligned latents) for faster convergence, and is best viewed as a coordinated training regime rather than purely post hoc interoperability. Using the learned alignment maps, we demonstrate a range of downstream applications, encompassing zero-cost probe sharing (alain2016understanding), teacher–student representation migration, and mutual teaching via cross-model losses. The complete set of results is summarized in Table 9. 4.7 Discussion and Future Outlook While the present work focuses on cognitive properties of world models, bridging internal representations to action-conditioned interaction remains an important direction. Recent progress in latent predictive video models such as V-JEPA (bardes2024vjepa) and V-JEPA2 (assran2025vjepa2) suggests a promising route toward planning and control. 5 Conclusion Independent JEPA agents that learn from different views of the same scene converge to latent spaces that can be aligned by a single linear map. This emergent geometric consensus is explained by the predictive nature of the objective and enables probe sharing and representation migration without retraining. The same principle holds across objects, lighting, camera motion and dataset style, suggesting that the pressure to predict future observations is strong enough to override low-level nuisance variation. Because the alignment map is small and cheap to estimate, it offers a practical bridge for federated systems that must collaborate without exchanging raw images, labels or gradients. In the future, we will explore whether the phenomenon survives when agents control their own cameras and influence the scene, opening a path to coordinated exploration. Impact Statement This paper aims to advance self-supervised representation learning and world models by characterizing when and why independently trained models become interoperable via simple linear maps. Potential positive impacts include bandwidth- and privacy-friendly collaboration protocols for distributed learning systems. Potential risks include misuse for covert coordination among autonomous systems; mitigating such risks requires transparency and appropriate governance in deployment. References Appendix A Implementation Details Figure 4: Independent agents can learn world models from different observations and align their latent spaces using linear maps (WijW_ij), enabling coordination without sharing raw observations. A.1 Architectures Unless otherwise noted, we follow an I-JEPA style setup (lecun2022path; assran2023jepa) with a ViT backbone (dosovitskiy2020vit). To isolate the effect of the learning objective on cross-model alignability, we keep the backbone family fixed across methods. Each “view” corresponds to training a separate model from scratch (no shared parameters, no cross-view losses). Backbone. Unless otherwise specified, we use ViT-S/16 at 224×224224× 224 resolution: patch size 1616, embedding dimension d=384d=384, depth 1212, 66 attention heads, and MLP hidden dimension 15361536. Predictor. We use a lightweight predictor head (2-layer MLP with hidden width 20482048, GELU, LayerNorm) mapping context latents to target latents in ℝdR^d. Targets and stop-gradient. We compute the squared prediction loss in latent space with stop-gradient on targets. Following common JEPA practice (assran2023jepa), we additionally maintain an exponential moving average (EMA) target encoder for stable targets (EMA momentum m=0.996m=0.996, cosine schedule to 1.01.0). Latent used for alignment. We fit W on the final CLS token representation z∈ℝ384z ^384 (thus W∈ℝ384×384W ^384× 384). A.2 Optimization and Masking Optimizer and schedule. We use AdamW (betas (0.9,0.95)(0.9,0.95), ϵ=10−8ε=10^-8) with cosine learning-rate decay and a 10-epoch warmup. Unless otherwise noted, weight decay is 0.050.05 and gradient clipping is 1.01.0. We use mixed precision training. Masking. We use block masking following I-JEPA (assran2023jepa): we sample multiple target blocks and predict their representations from the remaining visible context. Unless otherwise stated, the overall target mask ratio is 0.60.6 with 44 target blocks of size 7×77×7 (in patch units). Compute. We pretrain each view-specific model for 200200 epochs on smallNORB and nuScenes, and 300300 epochs on ImageNet-1k. All experiments were run on NVIDIA A100 GPUs (80GB); ImageNet-1k pretraining used 8 GPUs, while smallNORB/nuScenes used 1 GPU. Key hyperparameters are summarized in Table 3. Table 3: Key training hyperparameters (defaults used unless otherwise stated). Dataset Epochs Global batch Peak LR Weight decay smallNORB 200 256 3×10−43×10^-4 0.05 nuScenes 200 256 3×10−43×10^-4 0.05 ImageNet-1k 300 1024 1.5×10−41.5×10^-4 0.05 A.3 Augmentations and View Definitions ImageNet-1k. To mimic independent training pipelines, we train two models with two different augmentation recipes and evaluate alignment on paired augmentations of the same underlying image. View A uses random resized crop (scale∈[0.2,1.0]scale∈[0.2,1.0]), horizontal flip, color jitter (0.4,0.4,0.4,0.1), and random grayscale (0.2). View B uses the same base recipe and additionally applies Gaussian blur (0.5) and solarization (0.2). smallNORB. We resize/crop to 224×224224× 224 and apply horizontal flips; we avoid aggressive color augmentations because smallNORB is grayscale. nuScenes. We resize/crop to 224×224224× 224 and apply mild color jitter; pairing is based on synchronized timestamps and camera IDs (see below). A.4 Datasets, Views, and Pair Construction smallNORB. We treat each object instance (category, instance id) under fixed nuisance factors (elevation, lighting) as an underlying state and form paired samples by selecting two azimuth conditions. For the main comparison (e.g., 0∘0 vs. 160∘160 ), we pair images by matching (instance, elevation, lighting) and varying only azimuth. nuScenes. We treat synchronized frames at the same timestamp as the same underlying state and pair camera views (e.g., front vs. rear; front vs. side) using the dataset-provided calibration and timestamps. ImageNet-1k. We treat each image id as the underlying state and form two “views” via two different augmentation pipelines. To mimic independent training, we train two models with different augmentation recipes and then evaluate alignment using paired augmentations of the same image. Split hygiene. When fitting W, split pairs by state identifiers (instance id / timestamp token / image id) to avoid train–test leakage. A.5 Alignment Map Estimation and Normalization Let xn(i)x_n^(i) denote paired observations of state sns_n and zn(i)=f(i)(xn(i))∈ℝdz_n^(i)=f^(i)(x_n^(i)) ^d the corresponding latents. Stack latents into matrices X X ≜[z1(1),…,zN(1)]∈ℝd×N, [z_1^(1),…,z_N^(1)] ^d× N, (14) Y Y ≜[z1(2),…,zN(2)]∈ℝd×N. [z_1^(2),…,z_N^(2)] ^d× N. We estimate W by least squares: W∗=argminW‖Y−WX‖F2,W^*= _W\|Y-WX\|_F^2, (15) optionally with ridge regularization λ‖W‖F2λ\|W\|_F^2 for numerical stability. A closed-form solution is W∗=YX⊤(XX⊤+λI)−1.W^*=YX (X +λ I)^-1. (16) Centering/standardization. We recommend centering and scaling latents using train statistics (per-dimension mean/variance) before fitting W; this reduces sensitivity of MSE to representation scaling. Orthogonal Procrustes. If restricting W to be orthogonal, compute the SVD of YX⊤=UΣV⊤YX =U V and set W∗=UV⊤W^*=UV (gower2004procrustes). Inversion for migration. When W is ill-conditioned, use the Moore–Penrose pseudoinverse W†W in place of W−1W^-1 for probe migration. A.6 Collaboration Algorithms (Unified Pseudocode) This section presents unified pseudocode for three collaboration primitives—probe migration, teacher–student representation migration, and mutual teaching—which share a common structure: estimate an alignment map W from paired samples and then use W either to transfer a linear readout (probe migration) or to define an auxiliary cross-model consistency loss during training. Algorithm 2 Cross-model collaboration with an alignment map W (three modes) 1: Input: mode m∈Probe,TeacherStudent,Mutualm∈\ Probe, TeacherStudent, Mutual\; paired samples (x(1),x(2))(x^(1),x^(2)) of the same state s 2: Inputs (optional): probe weights a; threshold τ; fixed teacher (f(T),p(T))(f^(T),p^(T)) and student (f(S),p(S))(f^(S),p^(S)); weights β,γ>0β,γ>0 3: Fit alignment (whenever needed): W←argminW‖z(2)−Wz(1)‖2W← _W\|z^(2)-Wz^(1)\|^2 (least squares / Procrustes) 4: if m=Probem= Probe then zero-cost probe migration 5: Compute κ(W)κ(W) (e.g., via singular values) 6: if κ(W)≤τκ(W)≤τ then 7: a(2)←W−⊤a^(2)← W^- a 8: else 9: a(2)←(W†)⊤a^(2)←(W ) a pseudoinverse for ill-conditioned maps 10: end if 11: Return migrated probe weights a(2)a^(2) 12: else if m=TeacherStudentm= TeacherStudent then alternating optimization 13: for outer iterations t=1,…,Tt=1,…,T do 14: Sample paired observations (x(T),x(S))(x^(T),x^(S)) of the same underlying s 15: z(T)←f(T)(x(T))z^(T)← f^(T)(x^(T)), z(S)←f(S)(x(S))z^(S)← f^(S)(x^(S)) 16: Wt←argminW‖z(T)−Wz(S)‖2W_t← _W\|z^(T)-Wz^(S)\|^2 detach WtW_t 17: Update student by minimizing ℒJEPA(f(S),p(S))+β‖z(T)−Wtz(S)‖2L_JEPA(f^(S),p^(S))+β\|z^(T)-W_tz^(S)\|^2 stop-grad through teacher 18: end for 19: else if m=Mutualm= Mutual then online cross-model loss 20: for training iterations do 21: Sample paired observations (x(1),x(2))(x^(1),x^(2)) of the same underlying s 22: z(1)←f(1)(x(1))z^(1)← f^(1)(x^(1)), z(2)←f(2)(x(2))z^(2)← f^(2)(x^(2)) 23: W←argminW‖z(2)−Wz(1)‖2W← _W\|z^(2)-Wz^(1)\|^2 detach W 24: Update both models by minimizing ℒJEPA(1)+ℒJEPA(2)+γ‖z(2)−Wz(1)‖2L_JEPA^(1)+L_JEPA^(2)+γ\|z^(2)-Wz^(1)\|^2 25: end for 26: end if Appendix B Additional Results B.1 Pair Budget and Conditioning To evaluate whether W is a practical “communication layer”, measure isomorphism as a function of the number of paired samples N used to fit W (e.g., N∈50,100,500,1k,5k,20kN∈\50,100,500,1k,5k,20k\). Report R2R^2/DSC/NOS@10 and the condition number κ(W)κ(W) on a fixed held-out test set. This analysis exposes a key failure mode: when N is too small or pairing is noisy, W becomes ill-conditioned and probe migration degrades. Figure 5: Isomorphism vs. pair budget, with conditioning of W. B.2 Failure Mode Stress Tests We explicitly test regimes where our assumptions (Section D.3) predict failure and report both performance and conditioning diagnostics. (a) Pair noise. We randomly mismatch a fraction ϵε of pairs and measure the collapse of R2R^2/DSC and the growth of κ(W)κ(W). Results are reported in Table 4. Table 4: Pair-noise stress test (report metrics vs. mismatch rate ϵε). ϵε MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ κ(W)κ(W) ↓ 0.0 0.036 0.891 0.872 0.270 1.68 0.1 0.078 0.814 0.736 0.413 14.3750 0.3 0.146 0.692 0.528 0.637 138.42 (b) Distribution shift. We fit W on one subset of states (e.g., daytime scenes) and evaluate on another (night/rain) to quantify stability under non-stationarity; see Table 5. Table 5: Distribution-shift stress test (fit in-domain, evaluate both in-domain and shifted). Eval split MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ In-domain 0.036 0.891 0.872 0.270 Shifted 0.054 0.844 0.813 0.331 (c) Heterogeneous capacity. We align models with different backbones (or embedding dimensions) to study when linear interoperability breaks and whether a rectangular map or an additional projection head is needed. Example results are shown in Table 6. Table 6: Heterogeneous-capacity stress test (example template). Pair MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ ViT-S/16 ↔ ViT-B/16 0.072 0.814 0.785 0.421 ViT-S/16 ↔ ViT-L/16 0.115 0.742 0.693 0.548 B.3 Seed Robustness To quantify variance, run each configuration with at least three random seeds and report mean± for all metrics (MSE, R2R^2, DSC, NOS@10, and linear CKA). Summary statistics are reported in Table 7. Table 7: Seed robustness (report mean± over ≥3≥ 3 runs). Setting MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ smallNORB (0∘0 vs 160∘160 ) 0.036 ± 0.002 0.891 ± 0.004 0.872 ± 0.003 0.270 ± 0.008 nuScenes (Front vs Rear) 0.058 ± 0.004 0.852 ± 0.006 0.821 ± 0.005 0.340 ± 0.011 nuScenes (Front vs Side) 0.062 ± 0.005 0.838 ± 0.007 0.804 ± 0.009 0.360 ± 0.014 ImageNet-1k (Aug. views) 0.041 ± 0.003 0.873 ± 0.005 0.846 ± 0.007 0.310 ± 0.010 Interpretation. The standard deviation in Table 7 quantifies variability across random seeds. Larger variance can arise from optimization noise or limited pair budgets, and should be interpreted together with alignment diagnostics such as the spectrum/conditioning of W (Figure 9) and the pair-budget analysis (Figure 5). B.4 Neighborhood Overlap Across k To probe local topology beyond a single operating point, we report neighborhood disagreement NOS@k for multiple k (e.g., k∈1,5,10,20,50k∈\1,5,10,20,50\). Results are summarized in Table 8. Table 8: Neighborhood topology across k (lower is better). Setting NOS@1 ↓ NOS@5 ↓ NOS@10 ↓ NOS@20 ↓ NOS@50 ↓ smallNORB (0∘0 vs 160∘160 ) 0.124 0.198 0.270 0.352 0.441 nuScenes (Front vs Rear) 0.182 0.265 0.340 0.418 0.512 nuScenes (Front vs Side) 0.195 0.281 0.360 0.437 0.529 ImageNet-1k (Aug. views) 0.151 0.232 0.310 0.386 0.473 B.5 Downstream applications of alignment-driven collaboration. Summary. Table 9 compares three uses of an alignment map W. Teacher–student migration adds a latent alignment term for the student and reduces the number of epochs needed to reach a target probe accuracy relative to training from scratch. Mutual teaching adds an online cross-model loss and can further reduce epochs, at the cost of additional coupled training compute. Probe sharing transfers a linear head via W−⊤W^- (or W†⊤W when W is ill-conditioned) with zero target-side optimization steps. Table 9: Downstream applications of alignment-driven collaboration. Top: teacher–student migration (epochs/FLOPs to reach 85% probe accuracy). Middle: mutual teaching (speed-up under stronger coupling). Bottom: probe sharing (target accuracy after head transfer). Paradigm Method / Strategy Accuracy ↑ Epochs ↓ Training Cost ↓ Migration Cost T–S Migration Student from Scratch 85.2% 150 1.00×1.00× FLOPs N/A Knowledge Distillation (hinton2015distilling) 86.1% 90 0.62×0.62× FLOPs Active Update Latent Alignment (Ours) 87.4% 35 0.28× FLOPs Alternating Opt. Mutual Teaching Independent JEPA 86.2% 120∗ 1.00×1.00× FLOPs N/A Method A: Alternating Alignment 87.1% 80∗ 1.05×1.05× FLOPs Weak Coupling Method B: Cross-Model Loss (ℒcrossL_cross) 88.0% 55∗ 1.12× FLOPs Strong Coupling Probe Sharing Source Probe (A-probe) 49.59% 0 0 FLOPs N/A Traditional KD Baseline 50.63% 50 Active Training High Zero-Cost Sharing (Ours) 51.70% 0 0 FLOPs Plug-and-Play ∗ Epochs required to reach a downstream accuracy threshold of 85%. FLOPs are computed as specified in text. B.6 Full ImageNet-1k Baseline Table For completeness, we provide an expanded ImageNet-1k comparison including additional widely used SSL baselines (when available) under the same alignment protocol. The full results are listed in Table 10 (an expanded version of Table 2). Table 10: Expanded ImageNet-1k baseline table (same alignment protocol as Table 2). Method MSE ↓ R2R^2 ↑ DSC ↑ NOS@10 ↓ Social-JEPA (I-JEPA) 0.091 0.489 0.584 0.033 DINO (caron2021dino) 0.662 0.283 0.391 0.102 MoCo v3 (chen2021mocov3) 54.343 0.358 0.521 0.086 iBOT (zhou2022ibot) 0.662 0.283 0.391 0.102 MAE (he2022masked) 10.023 0.409 0.450 0.028 SimCLR (chen2020simclr) 13.474 0.437 0.558 0.072 SwAV (caron2020swav) 0.745 0.312 0.428 0.094 Barlow Twins (pmlr-v139-zbontar21a) 1.256 0.452 0.561 0.058 DINOv2 (oquab2024dinov2) 0.512 0.441 0.542 0.045 B.7 Additional Metrics In addition to the main metrics, we recommend reporting linear CKA (kornblith2019similarity) on multiple layers and CCA-based similarity (SVCCA/PWCCA) (raghu2017svcca; morcos2018insights) as robustness checks. Appendix C Additional Analyses and Visualizations C.1 Alignment Visualization Before/After Applying W (a) Before alignment (b) After alignment Figure 6: Qualitative visualization of geometric isomorphism via low-dimensional projections before and after applying the learned alignment map W. Interpretation. Before alignment, two independently trained models can represent the same states in different coordinate systems, so matched pairs may appear separated in a low-dimensional projection. After applying W, matched pairs should move closer and neighborhood structure should become more consistent. For completeness, the same visualization can be reported across multiple random seeds and pair budgets. C.2 ImageNet-1k Paradigm Comparison (Bar Chart) Figure 7: Visualization of ImageNet-1k baseline comparison. Interpretation. The bar chart provides a visual summary of the comparison in Table 2. In our experiments, predictive objectives tend to yield higher cross-model alignability than representative contrastive/reconstructive baselines under the same backbone and data split. Because MSE can be sensitive to representation scaling, we interpret it together with R2R^2, DSC, and CKA. C.3 Layerwise CKA Heatmap Figure 8: Layerwise similarity (linear CKA) across network depth, used to localize where isomorphism emerges. Interpretation. Layerwise CKA helps localize where similarity emerges across depth. Early layers may be more view-specific, while later layers can become more similar if the models converge to comparable high-level features. Reporting both within-layer CKA (diagonal) and the full cross-layer heatmap can help diagnose representational re-indexing across depth. C.4 Spectrum of the Alignment Map Figure 9: Spectral diagnostics of W, used to assess conditioning, near-orthogonality, and effective rank. Interpretation. If isomorphism is “rigid” (close to a rotation), the singular values of W should concentrate near 11. Large condition numbers or heavy-tailed spectra indicate that alignment may be ill-conditioned, potentially degrading probe migration and suggesting the need for regularization or alternative alignment constraints. As an explicit orthogonality check, we additionally recommend reporting ‖W⊤W−I‖F‖I‖F, \|W W-I\|_F\|I\|_F, (17) where smaller is closer to orthogonal, alongside κ(W)κ(W) computed from the ratio of the largest to the smallest singular value. Appendix D Theory Details D.1 Setup Let s∈s denote an environment state and x(i)=gi(s)x^(i)=g_i(s) a view-specific observation. Each encoder f(i)f^(i) produces a latent z(i)=f(i)(x(i))∈ℝdz^(i)=f^(i)(x^(i)) ^d. For simplicity, consider a JEPA objective with a squared prediction loss (stop-gradient on targets): ℒ(f,p)≜(xc,xt)[‖p(f(xc))−sg(f(xt))‖22].L(f,p) _(x_c,x_t) [\|p(f(x_c))-sg(f(x_t))\|_2^2 ]. (18) Our goal is to understand why two independently trained models can satisfy z(2)(s)≈Wz(1)(s)z^(2)(s)≈ Wz^(1)(s) with W close to invertible. D.2 Assumptions and boundaries Assumption D.1 (Common predictive structure). The two models are trained on the same underlying environment distribution and optimize the same JEPA objective (possibly with different observation functions), so that their optimal solutions encode the same predictive factors. Assumption D.2 (Non-degeneracy). The learned representation has non-degenerate covariance and is not severely rank-deficient relative to the predictive task, so that any alignment map is not forced to be effectively low-rank. Assumption D.3 (Paired states for evaluation). At evaluation time, we can obtain (exact or approximate) paired states with bounded noise to estimate W and test out-of-sample generalization. Violations of these assumptions lead to the failure modes we test in Section B.2 (pair noise, distribution shift, heterogeneous capacity) and diagnose via κ(W)κ(W) and spectral analysis (Figure 9). D.3 A symmetry at zero prediction error (and why it matters) Lemma D.4 (GL(d)(d) symmetry at zero loss). If a pair (f⋆,p⋆)(f ,p ) achieves zero JEPA loss (i.e., p⋆(f⋆(xc))=f⋆(xt)p (f (x_c))=f (x_t) almost surely), then for any invertible matrix A∈GL(d)A∈ GL(d), the transformed pair fA(x)≜Af⋆(x),pA(z)≜Ap⋆(A−1z)f_A(x) Af (x), p_A(z) A\,p (A^-1z) (19) also achieves zero loss. Proof. For any (xc,xt)(x_c,x_t), we have pA(fA(xc))=Ap⋆(A−1Af⋆(xc))=Ap⋆(f⋆(xc))=Af⋆(xt)=fA(xt)p_A(f_A(x_c))=A\,p (A^-1Af (x_c))=A\,p (f (x_c))=Af (x_t)=f_A(x_t). Thus the per-sample prediction error is identically zero, hence the expected squared error is zero as well. ∎ Proposition D.5 (Near-invariance at small loss). Let (f,p)(f,p) be any JEPA model, and define (fA,pA)(f_A,p_A) as in Lemma D.4. Then the corresponding losses satisfy σmin(A)2ℒ(f,p)≤ℒ(fA,pA)≤σmax(A)2ℒ(f,p), _ (A)^2\,L(f,p)\;≤\;L(f_A,p_A)\;≤\; _ (A)^2\,L(f,p), (20) where σmin(A) _ (A) and σmax(A) _ (A) are the smallest and largest singular values of A. Proof. Let e(xc,xt)≜p(f(xc))−sg(f(xt))e(x_c,x_t) p(f(x_c))-sg(f(x_t)) be the prediction error. By construction, pA(fA(xc))−sg(fA(xt))=Ae(xc,xt)p_A(f_A(x_c))-sg(f_A(x_t))=A\,e(x_c,x_t). Therefore, ‖Ae‖22 \|Ae\|_2^2 ≤σmax(A)2‖e‖22, ≤ _ (A)^2\|e\|_2^2, (21) ‖Ae‖22 \|Ae\|_2^2 ≥σmin(A)2‖e‖22. ≥ _ (A)^2\|e\|_2^2. Taking expectations yields the stated bounds. ∎ Remark D.6 (Why this matters). Lemma D.4 formalizes a core non-identifiability: at (near-)zero prediction error, latent coordinates are not unique and solutions form equivalence classes under invertible linear reparameterizations. Proposition D.5 shows that even away from zero loss, the objective remains approximately invariant to well-conditioned transforms. In practice, normalization and regularization (e.g., weight decay, feature standardization) discourage extreme scalings, making near-orthogonal transforms particularly relevant (Figure 9). D.4 From predictive sufficiency to shared latent structure Predictive sufficiency (informal). Assume that the environment state s contains the information needed to predict targets from contexts, and that an optimal JEPA encoder f compresses observations into a representation that is sufficient for the prediction task (cf. information bottleneck perspectives (tishby2000information; tishby2015deep; achille2018emergence)). If two encoders are trained on different observation functions of the same environment distribution, both are pressured to represent the same predictive structure, even if the raw observations differ. Why a linear map is plausible. Even if the underlying sufficient statistics are only identifiable up to invertible transforms in general, JEPA-style training introduces strong inductive biases via the architecture and optimization. Combined with the approximate linear symmetry above, a simple linear map can often explain most cross-model variation (high R2R^2/CKA in our metrics). Related identifiability results formalize when learned representations are identifiable up to linear transforms under appropriate assumptions (roeder2021linear). D.5 Limitations of the theory and directions for evaluation The arguments above do not guarantee that all optima are related by a linear map, nor do they ensure that the alignment map remains stable under strong nonstationarity or pronounced heterogeneity across training conditions. Empirically, failures may arise when paired states are noisy or unavailable, when the two models learn different predictive factors, or when the learned map becomes ill-conditioned. We therefore recommend probing these regimes using seed robustness analyses, pair-budget curves, and spectral diagnostics of the alignment map W, including its conditioning, as illustrated in Figure 9. When appropriate, these tests can be further contextualized by connections to broader analyses of symmetry and loss landscapes in neural networks (garipov2018loss; pmlr-v80-draxler18a; ainsworth2023gitrebasin). Testable predictions. Our framework yields several empirically testable predictions. First, reducing the JEPA prediction error, for example through longer training or stronger predictors, should improve alignability, as reflected by higher R2R^2 or DSC scores and lower values of NOS@10. Second, increasing pair noise is expected to degrade NOS@10 while inflating the condition number κ(W)κ(W). Third, objectives that explicitly promote view-invariant predictive structure should produce more stable alignment maps across random seeds and pair budgets. These predictions can be evaluated directly by measuring correlations between JEPA training loss and cross-model R2R^2 over training checkpoints, by systematically varying the level of pair noise ϵε and inspecting the resulting degradation curves as described in Section B.2, and by comparing the stability of W across different self-supervised learning objectives using seed robustness metrics reported in Table 7. Additional testable implications include the expectation that alignment quality improves with the number of paired samples, as captured by pair-budget curves in Figure 5, that isomorphism is stronger when models are trained on the same environment distribution rather than on disparate distributions, and that the alignment map remains well conditioned when isomorphism holds but becomes ill conditioned when the learned representations are incompatible. Appendix E Reproducibility and Plain-Language Summary E.1 Reproducibility Statement Code and configuration. Provide an anonymized repository with training and evaluation scripts, exact hyperparameter configs, and data split generation. Checkpoints. Provide pretrained checkpoints (or scripts to download them) for all reported results. Randomness. Report all random seeds and the number of runs. Compute. Report hardware (GPU/TPU type), total training time, and FLOPs accounting assumptions. Data. Document preprocessing and augmentation pipelines, including any filtering or subsampling. E.2 Plain-Language Summary We study what happens when two AI systems learn about the same world independently but from different viewpoints. We find that they often build almost the same internal “map” of the world, except the map is written in different coordinates. With a simple linear transformation, one system can translate between the two maps, which makes it possible to share knowledge efficiently without sharing raw data. E.3 Broader Impact This work may help build more compute- and privacy-efficient ways for distributed learning systems to collaborate by exchanging lightweight alignment maps instead of raw data or gradients. Potential misuse includes enabling covert coordination among autonomous systems; mitigating such risks requires careful governance and transparency in deployment.