Paper deep dive
Three Necessary Principles for Self-Supervised Visual Representation Learning
Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/12/2026, 1:22:08 AM
Summary
The paper proposes that self-supervised visual representation learning requires three necessary, non-overlapping principles: observation (semantic invariance across augmented views), prediction (patch-level spatial prediction), and regularization (preventing representational non-degeneracy). The authors prove that dropping any of these leads to specific failure modes like optimization collapse, dimensional collapse, or semantic impoverishment. They unify major self-supervised methods as special cases of a single energy decomposition and demonstrate that contrastive alignment alone provides only self-limiting collapse resistance.
Entities (8)
Relation Signals (7)
Regularization Principle → prevents → Representational Collapse
confidence 95% · Regularization... prevents both complete and dimensional collapse.
I-JEPA → isspecialcaseof → Prediction Principle
confidence 90% · Setting α=0 recovers predictive methods: I-JEPA... with a momentum encoder.
Prediction Principle → prevents → Structural Omission
confidence 90% · Dropping prediction withholds the spatial training signal by construction... manifests as degraded patch retrieval accuracy.
Observation Principle → prevents → Semantic Impoverishment
confidence 90% · Dropping observation forfeits cross-view semantic invariance by construction... manifests as the largest single accuracy drop.
Momentum Encoder → provides → No Collapse Guarantee
confidence 90% · the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence.
NT-Xent → provides → Self-limiting Collapse Resistance
confidence 85% · Contrastive alignment provides only self-limiting collapse resistance... the contrastive term cannot guarantee a positive gradient sufficient to prevent gradual dimensional collapse.
DINOv2 → satisfies → Three Principles
confidence 85% · DINOv2... satisfy the three conditions but retain implicit mechanisms.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (ii) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (iii) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.
Tags
Links
- Source: https://arxiv.org/abs/2608.08309v1
- Canonical: https://arxiv.org/abs/2608.08309v1
Trouble viewing inline? Open PDF directly →
Full Text
72,460 characters extracted from source content.
Expand or collapse full text
11institutetext: Imperial College London 11email: nikos,tania@imperial.ac.uk 22institutetext: CERTH, ITI 22email: giakoupg@iti.gr Three Necessary Principles for Self-Supervised Visual Representation Learning Nikos Giakoumoglou Paschalis Giakoumoglou Tania Stathaki Abstract We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (i) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (i) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction. 1 Introduction Self-supervised visual representation learning can be understood through a single lens: methods learn a compatibility function over pairs of observations, assigning low energy to compatible inputs and high energy otherwise [lecun2022path, dawid2023introduction]. The central design question is what structure the energy surface should encode, and how. In practice this reduces to two components: a projector, which maps globally pooled representations to an embedding space where view-level compatibility is measured, and a predictor, which maps context patches and positional information to predicted target representations. Invariance methods [oord2019cpc, he2020momentum, chen2020simclr, grill2020byol, chen2021simsiam, bardes2022vicreg, zbontar2021barlow, giakoumoglou2024synco] train only the projector, mapping augmented views of the same image to nearby representations; the result is semantically rich but discards spatial structure through global pooling. Joint-Embedding Predictive Architectures (JEPAs) [lecun2022path, assran2023ijepa, bardes2024vjepa] train only the predictor, recovering the latent representation of masked targets from visible context; the abstract target lets the encoder discard irrelevant variation, but without an invariance objective the representations can be locally consistent yet globally unstructured, lacking the cross-view semantic alignment that invariance methods provide. Both families face a common failure mode, representational collapse [jing2021understanding], each handled with its own mechanism, negative mining, asymmetric dynamics, or statistical regularization, typically conflated with the primary objective rather than treated as a separate requirement. We distinguish three non-equivalent failure modes. Optimization collapse: the constant encoder achieves zero training loss (Theorem LABEL:theorem:collapse). Dimensional collapse: the representation covariance has low effective rank, measurable even at non-zero loss. Semantic impoverishment: low linear-probe accuracy despite non-trivial effective rank. The three can occur independently (e.g., Table 1 rows H and I). Definition LABEL:def:reg addresses dimensional collapse; optimization collapse is a sufficient condition for it when no other mechanism is present. We argue that a complete training signal must satisfy three distinct and non-overlapping conditions. An encoder must observe (Definition LABEL:def:obs: produce semantically invariant global representations via the projector), predict (Definition LABEL:def:pred: support latent-space prediction of masked content via the predictor), and regularize (Definition LABEL:def:reg: prevent dimensional collapse via an explicit geometric constraint). We prove regularization necessary in the negative-free regime (Theorem LABEL:theorem:collapse) and formalize the self-limiting contrastive resistance via a gradient-decay argument (Remark LABEL:remark:ntxent_partial). For prediction and observation we give structural arguments and validate empirically, including with a patch-retrieval evaluation (Section 6). We further prove observation and prediction gradient-complementary at the encoder output (Theorem LABEL:theorem:gradient_decomp, for both ℒMSEL_MSE and ℒNT-XentL_NT -Xent under a mean-pool projector). This decomposition unifies the field [giakoumoglou2024review]; every major method is recovered by zeroing coefficients of a single energy. Removing prediction recovers invariance methods [chen2020simclr, he2020momentum, grill2020byol, chen2021simsiam, zbontar2021barlow, bardes2022vicreg, giakoumoglou2024synco]; removing observation recovers predictive methods [assran2023ijepa, balestriero2025lejepa, kuang2026lpjepa]. Methods satisfying all three but retaining implicit mechanisms [mo2024cjepa, oquab2024dinov2] exhibit a redundancy we prove formally (Theorem LABEL:theorem:ema_collapse) and confirm empirically (Table 3). Contributions. (i) We formalize the three principles as non-overlapping conditions on the energy surface (Section 3), prove regularization optimization-theoretically necessary under negative-free alignment, formalize contrastive self-limiting resistance via a gradient-decay argument, and argue prediction and observation structurally necessary. (i) We show all major SSL methods are special cases of a unified energy decomposition (Table 5). (i) We prove gradient complementarity of observation and prediction (Theorem LABEL:theorem:gradient_decomp) and momentum-encoder redundancy at convergence (Theorem LABEL:theorem:ema_collapse). (iv) We pair every theoretical claim with a controlled experiment on ViT-Tiny/STL-10, including patch-retrieval for the spatial consequence of prediction (Section 6). 2 Background We develop the minimal Energy-Based Model (EBM) formalism our argument needs; the standard contrastive and regularized training strategies and their energy landscapes are recalled in Appendix 0.A. Following LeCun et al. [lecun2022path] and Dawid et al. [dawid2023introduction], an EBM is a scalar function Fw:×→ℝF_w:X×Y that is low when x and y are compatible and high otherwise. In self-supervised vision, x and y are two augmented views of the same image, or a visible context and a masked target. A latent-variable generalization introduces ∈z and eliminates it by minimization, yielding the free energy Fw(,)=min∈Ew(,,)F_w(x,y)= _z E_w(x,y,z) [lecun2022path]. The difficulty is preventing the low-energy region from expanding to cover all of Y; contrastive methods raise the energy of negatives, while regularized methods constrain the low-energy volume directly (Appendix 0.A). The degenerate failure is representational collapse, fθ()=cf_θ(x)=c for all x, or more generally a low-dimensional subspace [jing2021understanding]. The JEPA [lecun2022path, assran2023ijepa] instantiates this framework with a shared encoder fθf_θ and predictor gϕg_φ: Ew(,,)=D(fθ(),gϕ(fθ(),))E_w(x,y,z)=D(f_θ(y),g_φ(f_θ(x),z)). The critical distinction from pixel-space reconstruction [he2022mae, baevski2022data2vec] is that the target is the representation fθ()f_θ(y) rather than the raw signal, so the encoder may discard irrelevant variation and the predictor need only recover abstract spatial structure [lecun2022path]. Energy surface realizations. Two architectural components realize this energy in practice. A projector hψ:ℝD→ℝD′h_ψ:R^D ^D maps globally pooled representations to a space where view-level compatibility is measured, inducing Fwproj(,)=d(hψ(x),hψ(y)),F^proj_w(x,y)=d\! (h_ψ(s_x),\,h_ψ(s_y) ), (1) where x,ys_x,s_y are the mean-pooled representations and d is a distance. A predictor gϕ:ℝ||×D×→ℝDg_φ:R^|C|× D×Z ^D maps context patches and positional information to predicted target representations, inducing Ewpred(,,) E^pred_w(x,y,z) =D(fθ()(t),gϕ(fθ()|,)), =D\! (f_θ(y)^(t),\;g_φ(f_θ(x)|_C,\,z) ), (2) Fwpred(,) F^pred_w(x,y) =minEwpred(,,). = _zE^pred_w(x,y,z). Thus FwprojF^proj_w shapes the surface globally (what content is present, irrespective of where) while FwpredF^pred_w shapes it locally (where content is, and how it relates spatially). As we formalize next, these aspects are non-overlapping and both necessary. 3 Three Necessary Principles Self-supervised methods differ not in their underlying principle, since all shape an energy surface to reflect compatibility, but in which aspect they model and how they prevent collapse. A complete training signal must satisfy three non-overlapping conditions, which we obtain by decomposing the total energy as Fw(,)=αFwproj(,)+βFwpred(,)+γΩ(fθ),F_w(x,y)=α\,F^proj_w(x,y)\;+\;β\,F^pred_w(x,y)\;+\;γ\, (f_θ), (3) with scalar weights α,β,γ≥0α,β,γ≥ 0 and a geometric regularizer Ω on the batch distribution of encoder representations. Setting individual coefficients to zero recovers strict subsets of the three conditions. Definition 1. Observation Principleobs An encoder fθf_θ with projector hψh_ψ satisfies the observation principle if, for any two compatible views, Fwproj(,)=d(hψ(x),hψ(y))F^proj_w(x,y)=d(h_ψ(s_x),h_ψ(s_y)) is minimized; the projector shapes the energy surface so that same-image view pairs receive low energy irrespective of augmentation. Because d operates on global representations, spatial arrangement within x is not preserved; the observation principle is necessary but not sufficient for a world model. Definition 2. Prediction Principlepred An encoder fθf_θ with predictor gϕg_φ satisfies the prediction principle if, given context x and target y with positional information mtm_t, Ewpred(,,mt)=D(fθ()(t),gϕ(fθ()|,mt))E^pred_w(x,y,m_t)=D(f_θ(y)^(t),g_φ(f_θ(x)|_C,m_t)) is minimized. The prediction target fθ()(t)f_θ(y)^(t) is a latent representation rather than the raw signal, so the encoder may discard irrelevant variation. Satisfying this within an image does not enforce that xs_x and ys_y be close across augmented views; prediction does not imply observation. Definition 3. Regularization Principlereg An encoder fθf_θ satisfies the regularization principle if the training objective contains a term Ω depending only on the empirical batch distribution p^=1B∑n=1Bδy,n p= 1B _n=1^B _s_y,n, strictly positive at every Dirac mass and minimized only at distributions whose representation covariance has full effective rank over ℝDR^D. This places no constraint on the content of representations, only their geometry; Ω shrink-wraps the low-energy region around the data manifold, preventing both complete and dimensional collapse [jing2021understanding]. The three instantiations we study (Section 4, Appendix 0.F.3) each meet this definition. 3.1 Necessity We now argue that each of the three principles is individually necessary; dropping any one admits a degenerate or impoverished solution that the remaining two cannot rule out at the scale we study. For regularization we give an optimization-theoretic proof under negative-free alignment and formalize contrastive resistance via a gradient-decay argument. For prediction and observation the arguments are structural: the omitted signal is simply never provided. Proposition 1. Necessity of Regularizationnecessity_reg Satisfying Definitions LABEL:def:obs and LABEL:def:pred without Definition LABEL:def:reg admits dimensional collapse at the studied scale. Under ℒinv=ℒMSEL_inv=L_MSE, the constant encoder is a global minimizer achieving zero loss (Theorem LABEL:theorem:collapse). Under ℒinv=ℒNT-XentL_inv=L_NT -Xent, the contrastive term acts as an implicit regularizer whose resistance is batch-size-dependent and self-limiting: as representations homogenize, all pairwise similarities converge and the NT-Xent gradient with respect to the encoder approaches zero (Remark LABEL:remark:ntxent_partial), so the contrastive term cannot substitute for an explicit ℒregL_reg that maintains a positive gradient at every step. Theorem 3.1. Collapse under observation and prediction without regularizationcollapse Let γ=0γ=0 and ℒinv=ℒMSEL_inv=L_MSE. Then the constant encoder fθ()= 1N⊤f_θ(x)=c\,1_N for all x, with a predictor satisfying gϕ( 1||⊤,mt)=g_φ(c\,1_|C| ,m_t)=c, is a global minimizer of αℒMSE+βℒpred _MSE+ _pred achieving value zero, for any α,β>0α,β>0. Proof sketch. With fθf_θ constant both views map to the same point, so ℒMSE=0L_MSE=0; a predictor with output bias c drives ℒpred=0L_pred=0, attainable through the context branch alone since the stop-gradient blocks the target branch. Both terms being non-negative and zero, the solution is a global minimizer. Full proof in Appendix 0.F.1. ∎ Remark 1. Self-limiting collapse resistance of the contrastive lossntxent_partial Theorem LABEL:theorem:collapse does not extend to ℒinv=ℒNT-XentL_inv=L_NT -Xent; with in-batch negatives the constant encoder is not a global minimizer. However, as representations homogenize toward a common direction, all pairwise cosine similarities converge to 1, the softmax denominator saturates at K⋅e1/τK· e^1/τ, and the gradient contribution from each negative shrinks as (1/K)O(1/K). The total repulsive signal therefore scales as (1)O(1) in the fully-collapsed limit rather than growing to prevent it; the contrastive term cannot guarantee a positive gradient sufficient to prevent gradual dimensional collapse at every step, unlike an explicit ℒregL_reg whose gradient is strictly positive at every Dirac mass (Appendix 0.F.3). Intuitively, once all negatives are equally hard the contrastive loss has no preferred direction in which to push representations apart, and the implicit anti-collapse signal stalls. Empirically, Table 1 row H retains 51.6%51.6\% without explicit regularization but never matches row J (55.0%55.0\%) with it, and the effective rank in row H is batch-size-sensitive (Appendix 0.G). Proposition 2. Structural necessity of predictionnecessity_pred Under the studied protocol, satisfying Definitions LABEL:def:obs and LABEL:def:reg without Definition LABEL:def:pred yields representations that encode semantic content but receive no spatial training signal by construction: no term in αFwproj+γΩα F^proj_w+γ depends on the relative positions of patches, so the encoder is never required to make gϕ(fθ()|,mt)g_φ(f_θ(x)|_C,m_t) approximate fθ()(t)f_θ(y)^(t). This structural omission manifests as degraded patch retrieval accuracy (Table 2) and lower linear-probe accuracy (Table 1, row F vs. row J). Proposition 3. Structural necessity of observationnecessity_obs Under the studied protocol, satisfying Definitions LABEL:def:pred and LABEL:def:reg without Definition LABEL:def:obs yields representations that are spatially consistent within an image but receive no cross-view semantic alignment signal by construction: no term in βFwpred+γΩβ F^pred_w+γ enforces that xs_x and ys_y be close across augmented views. This structural omission manifests as the largest single accuracy drop in the ablation (Table 1, row E vs. row J, a gap of 13.413.4 points). Propositions LABEL:proposition:necessity_pred and LABEL:proposition:necessity_obs are structural, not optimization-theoretic: they assert the omitted signal is never provided, not that no encoder could acquire the property indirectly. The alignment-uniformity decomposition of Wang et al. [wang2020understanding] shows the observation and regularization roles are entangled within a single contrastive loss. Our empirical validation is at a single small scale (Section 6); whether a larger encoder could acquire spatial structure from observation alone, or cross-view invariance from prediction alone, remains open. 3.2 Relation to Prior Methods The same decomposition that defines the three principles also classifies prior work; each method is recovered by fixing which coefficients of Equation˜3 are non-zero. Proposition 4. Unification of prior methodsunification The decomposition of Equation˜3 recovers all major self-supervised methods as special cases. Setting β=0β=0 recovers invariance methods, with collapse prevented contrastively [oord2019cpc, he2020momentum, chen2020simclr], through asymmetric dynamics [grill2020byol, chen2021simsiam], or an explicit regularizer [bardes2022vicreg, zbontar2021barlow]. Setting α=0α=0 recovers predictive methods: I-JEPA [assran2023ijepa] with a momentum encoder; LeJEPA [balestriero2025lejepa] and LpJEPA [kuang2026lpjepa] with explicit regularizers. With all three positive, C-JEPA [mo2024cjepa] and DINOv2 [oquab2024dinov2], whose KoLeo term is an explicit geometric regularizer, satisfy the three conditions but retain implicit mechanisms (momentum encoders, centering) alongside the regularizer. We prove this redundancy formally at convergence (Theorem LABEL:theorem:ema_collapse) and confirm it empirically (Table 3); our proposed model in Section 6 satisfies all three with no implicit mechanism. The complete taxonomy is given in Table 5 (Appendix 0.C). Methods that prevent collapse only through implicit mechanisms (momentum encoder, stop-gradient, contrastive repulsion, centering) do not satisfy Definition LABEL:def:reg and give no guarantee at convergence (Theorem LABEL:theorem:ema_collapse). 4 An Energy Decomposition Satisfying All Three Principles We instantiate Equation˜3 as three jointly trained objectives, ℒtotal=αℒinv+βℒpred+γℒregL_total= _inv+ _pred+ _reg. Because collapse prevention is delegated solely to ℒregL_reg, both ℒinvL_inv and ℒpredL_pred are free to focus on their own inductive biases without being redesigned to guard against degenerate solutions. We write x,y∈ℝDs_x,s_y ^D for the mean-pooled encoder outputs. Invariance objective. Instantiating Definition LABEL:def:obs through FwprojF^proj_w, we study two forms. The normalized temperature-scaled cross-entropy loss [chen2020simclr] aligns views with in-batch negatives, ℒNT-Xent=−1B∑n=1Blogexp(sim(hψ(x(n)),hψ(y(n)))/τ)∑k≠nexp(sim(hψ(x(n)),hψ(y(k)))/τ),L_NT -Xent=- 1B _n=1^B (sim(h_ψ(s_x^(n)),h_ψ(s_y^(n)))/τ) _k≠ n (sim(h_ψ(s_x^(n)),h_ψ(s_y^(k)))/τ), (4) with cosine similarity simsim and temperature τ; its negative repulsion is an implicit anti-collapse signal (Remark LABEL:remark:ntxent_partial). The alternative is a negative-free alignment ℒMSE=1B∑n‖hψ(x(n))−hψ(y(n))‖22L_MSE= 1B _n\|h_ψ(s_x^(n))-h_ψ(s_y^(n))\|_2^2, which collapses trivially without ℒregL_reg (Theorem LABEL:theorem:collapse) and serves as a clean diagnostic of each regularizer’s sufficiency. Note that “negative-free” refers throughout to the absence of contrastive negatives in the invariance objective; it does not refer to prediction-only methods, which have no invariance objective at all. Predictive objective. Instantiating Definition LABEL:def:pred through FwpredF^pred_w, we follow the I-JEPA masking strategy [assran2023ijepa]: a context block C is sampled per view and target blocks are masked, and the predictor maps context and mask tokens to target representations, ℒpred=1B⋅||∑n=1B∑t∈∥gϕ(fθ(n)|,mt)−sg(y,n(t))∥22,L_pred= 1B·|T| _n=1^B _t \|g_φ(f_θ(x_n)|_C,\,m_t)-sg\! (s_y,n^(t) ) \|_2^2, (5) where y,n(t)=fθ(n)(t)s_y,n^(t)=f_θ(y_n)^(t) and sg(⋅)sg(·) is the stop-gradient blocking flow through the target branch, so all collapse prevention is delegated to ℒregL_reg. Regularization objective. Instantiating Definition LABEL:def:reg, we study three regularizers on the batch distribution of projected representations: VCReg [bardes2022vicreg] (variance and covariance terms), SIGReg [balestriero2025lejepa] (matching an isotropic Gaussian via the Epps-Pulley statistic), and RDMReg [kuang2026lpjepa] (matching a rectified generalized Gaussian via sliced Wasserstein distance). Each is strictly positive at the collapsed solution and therefore meets Definition LABEL:def:reg (Appendix 0.F.3). Full definitions appear in Appendix 0.D. 4.1 Theoretical Guarantees Two further properties justify the design: the invariance and prediction objectives do not interfere at the encoder output, and the momentum encoder common to prior work supplies no collapse guarantee beyond an explicit regularizer once that regularizer is present. Theorem 4.1. Gradient complementarity of observation and predictiongradient_decomp Under a mean-pool projector architecture (where hψh_ψ acts on the mean-pooled token xs_x), the gradient of ℒinvL_inv with respect to each patch token is uniform across all i∈[N]i∈[N]: ∂ℒinv/∂fθ()(i)=1N∂ℒinv/∂x _inv/∂ f_θ(x)^(i)= 1N\, _inv/ _x. This holds for both ℒinv=ℒMSEL_inv=L_MSE and ℒinv=ℒNT-XentL_inv=L_NT -Xent, since both act on patch tokens only through the pooled representation. The gradient of ℒpredL_pred is zero for all masked tokens i∉i . Thus ℒinvL_inv provides a uniform global signal and ℒpredL_pred a spatially local one, with no structural conflict at the encoder output. Proof sketch. Both ℒMSEL_MSE and ℒNT-XentL_NT -Xent depend on each patch token only through the mean-pooled xs_x and the projector hψh_ψ, so ∂x/∂fθ()(i)=1ND _x/∂ f_θ(x)^(i)= 1NI_D gives the uniform projection for both forms. ℒpredL_pred receives only context tokens fθ()|f_θ(x)|_C as input to gϕg_φ; masked tokens do not appear in the computational graph, and the stop-gradient on y,n(t)s_y,n^(t) blocks all flow through the target branch. Full proof in Appendix 0.F.2. ∎ Remark 2. Token-level versus parameter-level conflictparam_conflict Theorem LABEL:theorem:gradient_decomp establishes disjointness at the encoder output under a mean-pool projector: a uniform global field from ℒinvL_inv and a context-supported local field from ℒpredL_pred. It does not preclude interference in parameter space, where both fields propagate into the shared weights θ. We read the theorem as ruling out structural conflict at the interface where the two signals are defined; parameter-space alignment remains an empirical question, directly testable by measuring the gradient cosine between the two objectives in θ across training. Theorem 4.2. Momentum encoder redundancy (conditional on convergence)ema_collapse Assumption: the online encoder θ(t)θ(t) converges to a fixed point θ∗θ^* under the given training objective. Under this assumption, in the continuous-time gradient flow, the momentum target encoder satisfies θ¯(t)→θ∗ θ(t)→θ^*. If θ∗θ^* is collapsed (a failure mode that LABEL:theorem:collapse and Appendix 0.F.3 show cannot arise when γ>0γ>0), the momentum encoder is also collapsed at convergence and provides no corrective signal. Note: this analysis models a decoupled flow; the full coupled dynamics with stop-gradient are more complex and convergence of θ(t)θ(t) is not guaranteed in general. Proof sketch. The average θ¯˙=(1−m)(θ−θ¯) θ=(1-m)(θ- θ) is a stable low-pass filter of θ(t)θ(t); once θ(t)→θ∗θ(t)→θ^* it tracks it and θ¯(t)→θ∗ θ(t)→θ^*. Full proof in Appendix 0.F.4. ∎ The explicit ℒregL_reg prevents the collapsed state from being a fixed point for any γ>0γ>0 (Appendix 0.F), giving a guarantee at every step. Theorem LABEL:theorem:ema_collapse shows the momentum encoder cannot supply a collapse guarantee beyond the explicit regularizer, but leaves open a transient stabilizing role of the kind analyzed by Tian et al. [tian2021understanding] for negative-free methods. Table 3 reflects exactly this division of labor: all three target-encoder choices reach a healthy asymptote whose location is set by the regularizer, while the momentum variant retains a modest accuracy edge attributable to smoothing of the prediction targets during training. 5 Related Work Invariance and masked modeling. Contrastive methods [oord2019cpc, he2020momentum, chen2020simclr, giakoumoglou2024synco, giakoumoglou2025cluster] and their negative-free [grill2020byol, chen2021simsiam] and regularized [bardes2022vicreg, zbontar2021barlow] variants satisfy Definition LABEL:def:obs but not Definition LABEL:def:pred, discarding spatial structure through global pooling. Masked image modeling reconstructs masked content in pixel or token space [bao2021beit, he2022mae, baevski2022data2vec], training no invariance objective. Self-distillation methods [caron2021dino, zhou2022ibot, oquab2024dinov2] combine view-level invariance with latent prediction of masked tokens; iBOT relies on implicit collapse prevention, while DINOv2’s KoLeo term satisfies Definition LABEL:def:reg (Table 5). Wang et al. [wang2020understanding] decompose the contrastive loss into alignment and uniformity, anticipating the observation-regularization split; Garrido et al. [garrido2023duality] prove a contrastive-covariance duality predicting the regularizer interchangeability in Table 4; Garrido et al. [garrido2023rankme] establish effective rank as a downstream predictor whose failure modes we also observe. Joint-Embedding Predictive Architectures. I-JEPA [assran2023ijepa] instantiates the predictor energy with a momentum encoder for collapse prevention, extended to video by V-JEPA [bardes2024vjepa, assran2025vjepa2]. LeJEPA [balestriero2025lejepa] and LpJEPA [kuang2026lpjepa] replace the momentum encoder with explicit regularizers, satisfying Definitions LABEL:def:pred and LABEL:def:reg but not Definition LABEL:def:obs. Closest to ours, C-JEPA [mo2024cjepa] augments I-JEPA with VICReg terms to satisfy all three; relative to it we eliminate the momentum encoder entirely (Theorem LABEL:theorem:ema_collapse), treat the three objectives as fully modular, and ground the decomposition in the formal principles of Section 3. Tian et al. [tian2021understanding] analyze the transient dynamics by which stop-gradient and momentum prevent collapse in negative-free methods; Theorem LABEL:theorem:ema_collapse is complementary, characterizing the fixed point rather than the trajectory. 6 Experiments Setup. We pre-train a ViT-Tiny encoder (embedding dimension 192, depth 12, patch size 8) on the unlabeled split of STL-10 [coates2011stl10] at 96×9696×96, giving 144144 patch tokens. The projector maps the pooled representation through width 1024 to dimension 256; the predictor is a narrow ViT (embedding dimension 96, depth 4). Masking follows I-JEPA [assran2023ijepa]: one context block (scale [0.85,1.0][0.85,1.0]) and four target blocks (scale [0.15,0.2][0.15,0.2]). We train 200 epochs with AdamW, batch size 512, and a cosine-decayed peak learning rate 5×10−45×10^-4, evaluating by linear probe and patch-level retrieval (Section˜6). We report the effective rank of the projector-output covariance as a scalar measure of non-degeneracy. All results are means over 5 random seeds; full hyperparameters appear in Appendix 0.E. Table 1: Subset ablation isolating the contribution of each principle. Each row instantiates a strict subset of ℒinv,ℒpred,ℒreg\L_inv,L_pred,L_reg\, verifying Propositions LABEL:proposition:necessity_reg, LABEL:proposition:necessity_pred, and LABEL:proposition:necessity_obs and Theorem LABEL:theorem:collapse and Remark LABEL:remark:ntxent_partial. Our proposed model (row J) is the full contrastive configuration and serves as the reference. Row G (MSE + JEPA, no regularizer) collapses to low effective rank despite satisfying two principles, directly verifying Theorem LABEL:theorem:collapse; the low effective rank of row I despite explicit regularization is discussed in Section 6. ℒinvL_inv ℒpredL_pred ℒregL_reg Effective Rank Linear Probe A MSE none none 20.0 17.3 B NT-Xent none none 166.0 41.8 C none JEPA none 15.9 42.1 D none none SIGReg 7.3 21.2 E none JEPA SIGReg 54.3 41.6 F NT-Xent none SIGReg 89.3 49.5 G MSE JEPA none 164.5 38.7 H NT-Xent JEPA none 167.7 51.6 I MSE JEPA SIGReg 9.4 49.2 J NT-Xent JEPA SIGReg 87.5 55.0 Figure 1: Eigenspectrum of the projector-output covariance, verifying Theorem LABEL:theorem:collapse. Negative-free alignment with prediction but no regularization (MSE, no regularization) collapses to a few dominant eigenvalues, consistent with the optimization collapse of Theorem LABEL:theorem:collapse. The contrastive variant (NT-Xent, no regularization) exhibits partial dimensional resistance, consistent with Remark LABEL:remark:ntxent_partial, but does not reach the flat spectrum achieved by an explicit regularizer. Figure 2: Marginal gain in linear-probe accuracy from adding each principle, verifying Propositions LABEL:proposition:necessity_reg, LABEL:proposition:necessity_pred, and LABEL:proposition:necessity_obs. Each bar is the accuracy gained by adding one principle to the pair that omits it, relative to our proposed model (row J, 55.0%55.0\%). No principle is redundant at this scale, and observation contributes the largest gain. Collapse without regularization (Theorem LABEL:theorem:collapse). The direct signature of Theorem LABEL:theorem:collapse is a degenerate representation spectrum (Figure 1). With the negative-free alignment and no regularizer, the projector-output covariance is dominated by a few leading eigenvalues (dimensional collapse), while adding an explicit regularizer flattens it. The contrastive alignment without a regularizer sits between the two, reflecting the self-limiting partial resistance of Remark LABEL:remark:ntxent_partial: the NT-Xent term raises effective rank above the collapsed baseline but does not stabilize it to the level achieved by an explicit ℒregL_reg. The linear-probe consequence is read from Table 1: the negative-free configuration without regularization (row G) reaches only 38.7%38.7\%, over ten points below the regularized version (row I, 49.2%49.2\%), while the contrastive variant (row H) retains 51.6%51.6\%, the implicit resistance of the contrastive term standing in for an explicit regularizer at this scale. The same separation holds dynamically in effective rank (Appendix 0.H). Necessity of each principle (Propositions LABEL:proposition:necessity_reg, LABEL:proposition:necessity_pred, and LABEL:proposition:necessity_obs). Against our proposed model (row J, 55.0%55.0\%), removing regularization (row H) costs 3.43.4 points, removing prediction (row F) 5.55.5 points, and removing observation (row E) 13.413.4 points, the largest single drop. Every single-principle configuration (rows A–D) lands between 17.3%17.3\% and 42.1%42.1\%; at this scale no pair substitutes for the third, and the ordering matches the structural argument: observation, which alone carries augmentation invariance, is the most costly to remove. Row C (prediction only) achieves 42.1%42.1\% despite low effective rank (15.915.9), illustrating the distinction between dimensional collapse and semantic impoverishment from Section 1: prediction alone yields non-trivial semantic content through latent target learning, but the low effective rank signals an impoverished representation geometry that limits transfer. The smallest gap (removing regularization) is consistent with Remark LABEL:remark:ntxent_partial: contrastive repulsion partially compensates for absent explicit ℒregL_reg at this scale. Per-principle gains and linear-probe trajectories appear in Appendix 0.I. Patch-level retrieval (Proposition LABEL:proposition:necessity_pred). To directly test the spatial consequence of the prediction principle, we evaluate patch-level nearest-neighbour retrieval. For each test image we extract the 144144 patch tokens from the frozen encoder, identify the k nearest-neighbour patches in the training set by cosine similarity, and measure the proportion of retrieved patches falling within the same spatial quadrant as the query patch (spatial recall@k). The results are shown in Table 2. Removing prediction (row F) causes a 3.13.1-point drop in spatial recall@5 relative to the full model (row J), directly confirming the structural argument of Proposition LABEL:proposition:necessity_pred: without a spatial training signal the encoder does not learn to distinguish patch positions. Removing observation (row E) causes a smaller drop of 0.60.6 points at recall@5, consistent with prediction being the primary driver of spatial structure and observation playing a complementary rather than conflicting role. The full model (row J) achieves the best spatial recall across all k. Table 2: Patch-level nearest-neighbour retrieval, verifying Proposition LABEL:proposition:necessity_pred. Spatial recall@k measures the proportion of retrieved patches falling within the same spatial quadrant as the query. Removing prediction (row F) causes the largest drop, confirming the structural argument that without ℒpredL_pred the encoder receives no spatial training signal. ℒinvL_inv ℒpredL_pred ℒregL_reg Recall@1 Recall@5 Recall@10 E none JEPA SIGReg 48.2 92.4 98.2 F NT-Xent none SIGReg 43.9 89.9 98.0 J NT-Xent JEPA SIGReg 49.9 93.0 98.9 Invariance objective and regularizer interaction. Row I achieves 49.2%49.2\% at effective rank 9.49.4, far below row J (87.587.5) despite both having an explicit regularizer. We attribute this to a gradient-magnitude imbalance: the MSE loss collapses more aggressively than SIGReg can counteract at γ=0.05γ=0.05. The regularizer is strictly positive at the collapsed solution (Lemma LABEL:lemma:sigreg_rdmreg_positive) and prevents optimization collapse per Theorem LABEL:theorem:collapse, but the effective rank trajectory (Appendix 0.H) shows SIGReg stabilizing at low rank when paired with MSE. Increasing γ raises effective rank toward the contrastive-regime values but at a cost to accuracy, consistent with SIGReg’s isotropic-Gaussian target being a poor match for the natural representation geometry. We therefore read row I not as a failure of the regularization principle but as evidence that the invariance objective choice affects the regularizer’s effective operating point; the contrastive term’s implicit repulsion complements rather than substitutes for explicit regularization. Gradient complementarity (Theorem LABEL:theorem:gradient_decomp). Figure 3 measures the per-patch gradient norm of each objective at the encoder output across checkpoints. The invariance gradient is uniformly distributed across patch tokens (the contrastive objective acts on the pooled representation), and the predictive gradient is concentrated on context positions and vanishes at masked targets (the stop-gradient eliminates flow through the target branch). The contrast is stable across all checkpoints, confirming the two objectives act on disjoint aspects of the representation at the encoder output rather than competing; whether they remain aligned in parameter space (Remark LABEL:remark:param_conflict) is a separate question we do not resolve here. Figure 3: Per-patch gradient norm at the encoder output, verifying Theorem LABEL:theorem:gradient_decomp. Top row: invariance gradient, uniform across patch tokens. Bottom row: predictive gradient, concentrated on visible context patches and vanishing at masked targets. Columns are training checkpoints; the contrast is stable throughout training. Momentum encoder redundancy (Theorem LABEL:theorem:ema_collapse). Figure 4(a) confirms the convergence: the parameter distance between online and momentum encoders rises during early learning, then decays monotonically toward zero. The consequence is shown in Figure 4(b) and Table 3: the momentum encoder (row B) retains a modest edge over the stop-gradient baseline (57.5%57.5\% versus 55.0%55.0\%), comparable in size to the cost of removing regularization altogether, which we attribute to transient target smoothing rather than collapse prevention; removing the stop-gradient (row C) sends the model through a collapse trough early in training before the explicit regularizer drives full recovery to 55.9%55.9\%. The shared asymptote confirms that the fixed point is set by the regularizer, while the target-encoder mechanism only stabilizes the path to it, consistent with Theorem LABEL:theorem:ema_collapse and the transient analysis of Tian et al. [tian2021understanding]. (a) Online and mom. param. distance (b) Linear probe, three target encoders Figure 4: Momentum-encoder redundancy, verifying Theorem LABEL:theorem:ema_collapse. (a) The online and momentum parameter distance rises during early learning, then decays toward zero; the two converge to the same fixed point, consistent with the theorem’s convergence assumption. (b) Stop-gradient and momentum reach comparable accuracy; removing the stop-gradient produces a collapse trough, after which the regularizer drives full recovery to the same asymptote. Table 3: Target-encoder ablation showing momentum-encoder redundancy. Contrastive alignment with isotropic-Gaussian regularization, verifying Theorem LABEL:theorem:ema_collapse; all three target-encoder choices reach a comparable asymptote, with the momentum variant retaining a modest edge attributable to transient target smoothing rather than collapse prevention. Target encoder Effective Rank Linear Probe A Stop-gradient 87.5 55.0 B Momentum 83.2 57.5 C None (full gradient) 89.9 55.9 Interchangeability of regularizers. With prediction and contrastive alignment fixed, SIGReg and RDMReg are interchangeable (55.0%55.0\% and 55.1%55.1\%, training curves in Appendix 0.J), consistent with the contrastive-regularized duality of Garrido et al. [garrido2023duality]. VCReg lags by ∼ 4 points despite the highest effective rank; we attribute this to gradient conflict with ℒpredL_pred, since the covariance penalty targets off-diagonal correlations that the predictor actively induces in the encoder output. Negative-free alignment trails every contrastive variant by at least 5.85.8 points, confirming that the contrastive term’s implicit anti-collapse contribution is complementary to explicit regularization. Effective rank is a collapse diagnostic, not a quality score [garrido2023rankme]. Table 4: Regularizer comparison with prediction enabled. The isotropic-Gaussian and sliced-Wasserstein regularizers are interchangeable; the variance-covariance regularizer attains the highest effective rank but lower accuracy, attributable to gradient conflict with the prediction objective. ℒinvL_inv ℒpredL_pred ℒregL_reg Effective Rank Linear Probe MSE JEPA SIGReg 9.4 49.2 NT-Xent JEPA SIGReg 87.5 55.0 NT-Xent JEPA VCReg 248.4 51.1 NT-Xent JEPA RDMReg 109.8 55.1 Discussion. These are validation, not benchmark claims; ViT-Tiny on STL-10 is not comparable to large-scale pre-training, and linear-probe figures should be read only against one another. Within that scope every prediction is borne out: collapse as a degenerate spectrum, necessity as ordered accuracy and spatial recall drops, gradient complementarity in the per-patch gradients, and momentum redundancy as converging parameter distance. Finite-step optimization stops short of literal rank-one collapse, and the momentum variant keeps a modest edge the fixed-point analysis does not model. The measurements are structural: each principle contributes a separable, non-substitutable signal, and prior methods’ implicit mechanisms reappear as counterparts of the explicit regularizer. Open questions: whether the ordering and regularizer interchangeability persist at scale; whether a dense probe would turn the structural argument for prediction into a measured result; and whether the gradient-conflict account of VCReg holds for other regularizers and architectures. 7 Conclusion We formalized self-supervised representation learning as energy minimization over three non-overlapping principles: observation, prediction, and regularization. Regularization is optimization-theoretically necessary under negative-free alignment; contrastive resistance is self-limiting by gradient decay; prediction and observation are structurally necessary by construction. At the scale we study no pair substitutes for the third, as validated by linear-probe accuracy, patch retrieval, and gradient heatmaps. The projector and predictor realize observation and prediction architecturally; the two objectives are gradient-complementary at the encoder output under a mean-pool projector; and the momentum encoder converges to the online encoder’s fixed point, offering no collapse guarantee beyond an explicit regularizer. The same decomposition recovers every major self-supervised method as a special case, differing only in which principles it satisfies. The clearest direction for future work is scale, and extending the decomposition to video and vision-language settings [bardes2024vjepa, assran2025vjepa2]. Acknowledgements We acknowledge the computational resources and support provided by the Imperial College Research Computing Service (http://doi.org/10.14469/hpc/2232), which enabled our experiments. References Appendix 0.A Energy-Based Training Strategies We recall the two standard strategies for training an EBM. Making Fw(,)F_w(x,y) low for a training pair is straightforward; the difficulty is preventing the low-energy region from expanding to cover all of Y. Contrastive training. A contrastive objective pushes the energy of training pairs down while pushing the energy of contrastive samples up [lecun2022path]. A simple instance is the hinge loss [Fw(,)+max(0,m−Fw(,^))]E[F_w(x,y)+ (0,m-F_w(x, y))] with margin m>0m>0; a more expressive instance is InfoNCE [oord2019cpc], using K negatives through a softmax ranking minw−[loge−Fw(,)/τe−Fw(,)/τ+∑k=1Ke−Fw(,^k)/τ], _w\;-E\! [ e^-F_w(x,y)/τe^-F_w(x,y)/τ+ _k=1^Ke^-F_w(x, y_k)/τ ], (6) with temperature τ>0τ>0. Contrastive methods subsume Siamese networks [chopra2005learning], MoCo [he2020momentum], SimCLR [chen2020simclr], and masked autoencoders [he2022mae], differing only in how negatives are generated [lecun2022path, giakoumoglou2024rrd]. Regularized training. Regularized methods avoid explicit negatives by minimizing the low-energy volume directly [lecun2022path], adding a term Ω(Fw) (F_w) that shrink-wraps the low-energy region around the data manifold: minw[Fw(,)]+Ω(Fw) _wE[F_w(x,y)]+ (F_w). Variance-covariance penalties [bardes2022vicreg, zbontar2021barlow], whitening [ermolov2021whitening], and distribution-matching objectives [balestriero2025lejepa, kuang2026lpjepa] impose qualitatively different geometric constraints. Appendix 0.B Architecture and Gradient Flows Figure˜5 illustrates how the three principles of Equation˜3 map onto concrete architectural components and their respective gradient flows through the shared encoder fθf_θ. The left branch realizes the observation principle (): both augmented views x and y pass through fθf_θ, their patch tokens are mean-pooled to xs_x and ys_y, and the projector hψh_ψ maps these to the space where ℒinvL_inv is computed. Because hψh_ψ acts on the pooled token, the resulting gradient at the encoder output is uniform across all N patch positions, 1N∂ℒinv∂x 1N _inv _x, as proved in LABEL:theorem:gradient_decomp. The right branch realizes the prediction principle (): context tokens fθ()|f_θ(x)|_C and mask token mtm_t are fed to the predictor gϕg_φ, which regresses the target representation fθ()(t)f_θ(y)^(t); the stop-gradient sg(⋅)sg(·) blocks all gradient flow through the target branch, so ℒpredL_pred drives the encoder only through the context tokens. The resulting gradient is spatially concentrated on visible context patches and zero at masked positions, the complementary local signal identified in LABEL:theorem:gradient_decomp. The bottom strip realizes the regularization principle (): the batch matrix =[hψ(x);hψ(y)]P=[h_ψ(s_x);h_ψ(s_y)] is passed to Ω , which produces a gradient that is strictly positive at every Dirac mass regardless of the state of ℒinvL_inv or ℒpredL_pred (Lemmas LABEL:lemma:var_collapse and LABEL:lemma:sigreg_rdmreg_positive). The three gradient fields enter fθf_θ simultaneously and are structurally non-conflicting at the encoder output: the blue uniform field and the orange spatially-local field act on disjoint aspects of the representation, while the green batch-distribution field provides a collapse-prevention signal independent of both. Parameter-space alignment of the three fields is a separate empirical question addressed in LABEL:remark:param_conflict. Figure 5: Architecture and gradient flows for the three-principle decomposition. The projector branch hψh_ψ realizes the observation principle (), receiving a uniform global gradient from ℒinvL_inv through the mean-pooled representations x,ys_x,s_y. The predictor gϕg_φ realizes the prediction principle (), receiving a spatially local gradient from ℒpredL_pred concentrated on visible context patches; the stop-gradient blocks flow through the target branch. The regularizer Ω realizes the regularization principle (), acting on the batch distribution of projected representations and providing a gradient at every step regardless of the invariance objective. The encoder fθf_θ (shared, gray) receives all three gradient fields, which are structurally non-conflicting at the encoder output (LABEL:theorem:gradient_decomp). Figure generated with the assistance of Google Gemini, accessed June 16, 2026. Appendix 0.C Full Taxonomy of Prior Methods Table 5 lists every major self-supervised method recovered as a special case of the unified energy of Equation˜3, organized by which of the three principles each satisfies. Methods marked † prevent collapse only through implicit mechanisms and therefore carry no collapse guarantee at convergence (Theorem LABEL:theorem:ema_collapse). Methods marked ‡ realize prediction through a shared online tokenizer head rather than a dedicated predictor module. Methods satisfying the regularization principle (✓) but also retaining an implicit mechanism are marked † in the method name column only, to indicate the redundant implicit mechanism while preserving the correct ✓ in the Reg column; C-JEPA and DINOv2 fall in this category. Table 5: All major self-supervised methods recovered as special cases of Equation˜3, as stated in Proposition LABEL:proposition:unification. Projector hψh_ψ realizes Definition LABEL:def:obs; predictor gϕg_φ realizes Definition LABEL:def:pred; Ω realizes Definition LABEL:def:reg. † in the method name denotes a redundant implicit mechanism (momentum encoder or centering) alongside the explicit regularizer; † in the Reg column denotes implicit collapse prevention only; ‡ denotes prediction via a shared head rather than a dedicated predictor. Method Projector hψh_ψ d Predictor gϕg_φ Ω Obs Pred Reg SimCLR [chen2020simclr] MLP InfoNCE none none ✓ ✗ ✗† MoCo [he2020momentum] MLP InfoNCE none none ✓ ✗ ✗† BYOL [grill2020byol] Asym. MLP MSE none none ✓ ✗ ✗† SimSiam [chen2021simsiam] MLP neg. cos. none none ✓ ✗ ✗† DINO [caron2021dino] MLP CE (centered) none none ✓ ✗ ✗† Barlow Twins [zbontar2021barlow] MLP cross-corr. none implicit ✓ ✗ ✓ VICReg [bardes2022vicreg] MLP MSE none VCReg ✓ ✗ ✓ I-JEPA [assran2023ijepa] none none ViT none ✗ ✓ ✗† LeJEPA [balestriero2025lejepa] none none ViT SIGReg ✗ ✓ ✓ LpJEPA [kuang2026lpjepa] none none ViT RDMReg ✗ ✓ ✓ iBOT [zhou2022ibot] MLP CE (centered) shared head‡ none ✓ ✓ ✗† C-JEPA† [mo2024cjepa] MLP MSE ViT VCReg ✓ ✓ ✓ DINOv2† [oquab2024dinov2] MLP CE (centered) shared head‡ KoLeo ✓ ✓ ✓ Ours MLP NT-Xent / MSE ViT Any ✓ ✓ ✓ Appendix 0.D Regularization Objectives in Full Each regularizer operates on the batch matrix of projected representations =[hψ(x);hψ(y)]∈ℝ2B×D′P=[h_ψ(s_x);h_ψ(s_y)] ^2B× D . Variance-covariance regularization (VCReg). The variance and covariance terms of VICReg [bardes2022vicreg] without the invariance term: ℒVCReg=μℒvar()+νℒcov(),L_VCReg=μ\,L_var(P)+ν\,L_cov(P), (7) ℒvar()=1D′∑d=1D′max(0,γ0−Var((:,d))+δ),ℒcov()=1D′∑i≠jCij2,L_var(P)= 1D _d=1^D \! (0,\, _0- Var(P^(:,d))+δ ), _cov(P)= 1D _i≠ jC_ij^2, (8) with C the batch covariance of P, γ0=1 _0=1, δ=10−4δ=10^-4, μ=25μ=25, ν=1ν=1. Isotropic-Gaussian regularization (SIGReg). Matching the characteristic function of the batch distribution to a standard isotropic Gaussian via the Epps-Pulley statistic [balestriero2025lejepa]: ℒSIGReg=[|[cos(⊤)]−e−‖2/2|2+|[sin(⊤)]|2],L_SIGReg=E_ ω\! [ |E_p[ ( ω p)]-e^-\| ω\|^2/2 |^2+ |E_p[ ( ω p)] |^2 ], (9) with ω sampled on the unit sphere using K=17K=17 quadrature points. Sliced-Wasserstein regularization (RDMReg). Aligning the batch distribution to a rectified generalized Gaussian target RGGX_RGG via sliced Wasserstein distance [kuang2026lpjepa]: ℒRDMReg=[22(sort(⊤),sort(RGG⊤))],L_RDMReg=E_ θ\! [W_2^2\! (sort(P θ),\;sort(X_RGG θ) ) ], (10) over 128128 random one-dimensional projections, with target shape parameter p=1p=1, zero mean, and unit variance after rectification. Appendix 0.E Hyperparameters Pretraining. We pre-train on STL-10 with a ViT-Tiny encoder (patch size 8, embedding dimension 192, depth 12, three heads) at 96×9696× 96 resolution, yielding 144144 patch tokens. The predictor is a narrow Vision Transformer (embedding dimension 96, depth 4) and the projector maps the pooled representation through width 1024 to output dimension 256. Each image yields two views; masking samples one context block at scale [0.85,1.0][0.85,1.0] and four target blocks at scale [0.15,0.2][0.15,0.2]. Training runs for 200 epochs (10 warmup) with AdamW at batch size 512, a peak learning rate of 5×10−45× 10^-4 decayed cosinewise to 1×10−61× 10^-6, weight decay annealed from 0.040.04 to 0.40.4, and bfloat16 precision. The loss weights are β=1.0β=1.0 for prediction, α=0.1α=0.1 for NT-Xent (1.01.0 for MSE), and γ=0.05γ=0.05, 0.020.02, 1.01.0 for SIGReg, VCReg, and RDMReg respectively. Remaining settings are the NT-Xent temperature τ=0.07τ=0.07, K=17K=17 SIGReg quadrature points, 128128 RDMReg projections, VCReg coefficients μ=25μ=25 and ν=1ν=1, and a momentum coefficient m=0.996m=0.996 where applicable. Every experiment is run over five random seeds. Linear probing. We freeze the pre-trained encoder and train a single linear head on its mean-pooled representations for 30 epochs with AdamW at batch size 512, reporting top-1 accuracy on the STL-10 test split. No augmentation other than standard normalization is applied at probe time, and the encoder receives no gradient. Patch-level retrieval. We extract patch tokens from the frozen encoder for all test images and a random subset of 10,00010,000 training images. For each of the 144144 query patches per test image, we retrieve the k nearest training patches by cosine similarity and measure spatial recall@k: the proportion of retrieved patches whose spatial quadrant (top-left, top-right, bottom-left, bottom-right) matches the query patch quadrant. We report k∈1,5,10k∈\1,5,10\ and average over all query patches and test images. Effective rank. We report effective rank as a scalar measure of representational non-degeneracy. Given the covariance Σ of the projector outputs over a batch with eigenvalues λii=1D′\ _i\_i=1^D and normalized spectrum pi=λi/∑jλjp_i= _i/ _j _j, the effective rank is exp(−∑ipilogpi) \! (- _ip_i p_i ), the exponential of the Shannon entropy of the normalized eigenspectrum. A value near 11 indicates collapse onto a single direction, while a value approaching D′D indicates an isotropic, fully utilized representation space. Effective rank is a collapse diagnostic, not a downstream quality predictor; high effective rank is necessary but not sufficient for high linear-probe accuracy, as illustrated by Table 4. Appendix 0.F Full Proofs This section gives the proofs deferred from the main text, in the order they are referenced: the collapse theorem, gradient complementarity, the strict positivity of each regularizer at the collapsed solution, and momentum-encoder redundancy. 0.F.1 Full Proof of Theorem LABEL:theorem:collapse Full proof of Theorem LABEL:theorem:collapse. Let fθ()= 1N⊤f_θ(x)=c\,1_N for all x. Then x=y=s_x=s_y=c, so hψ(x)=hψ(y)h_ψ(s_x)=h_ψ(s_y), giving ℒMSE=1B∑n‖hψ(x(n))−hψ(y(n))‖2=0L_MSE= 1B _n\|h_ψ(s_x^(n))-h_ψ(s_y^(n))\|^2=0. For the predictive loss, fθ()(t)=f_θ(y)^(t)=c for all n,tn,t. The stop-gradient on fθ()(t)f_θ(y)^(t) blocks gradient flow from the target representations back through the encoder, so the collapsed solution is achievable via the context branch alone: a predictor with zero attention weights and output bias c satisfies gϕ( 1||⊤,mt)=g_φ(c\,1_|C| ,m_t)=c, giving ℒpred=0L_pred=0. Both terms are non-negative and zero, so the collapsed solution is a global minimizer for any α,β>0α,β>0. ∎ 0.F.2 Full Proof of Theorem LABEL:theorem:gradient_decomp Full proof of Theorem LABEL:theorem:gradient_decomp. We prove the uniform projection for both ℒMSEL_MSE and ℒNT-XentL_NT -Xent under a mean-pool projector architecture, where hψh_ψ acts on x=1N∑ifθ()(i)s_x= 1N _if_θ(x)^(i). Gradient of ℒMSEL_MSE. The loss depends on each patch token fθ()(i)f_θ(x)^(i) only through xs_x. By the chain rule, ∂ℒMSE/∂fθ()(i)=(∂ℒMSE/∂x)⋅(∂x/∂fθ()(i))=(∂ℒMSE/∂x)⋅1ND _MSE/∂ f_θ(x)^(i)=( _MSE/ _x)·( _x/∂ f_θ(x)^(i))=( _MSE/ _x)· 1NI_D, which is uniform across all i. Gradient of ℒNT-XentL_NT -Xent. The NT-Xent loss operates on hψ(x(n))h_ψ(s_x^(n)), which depends on fθ()(i)f_θ(x)^(i) only through x(n)s_x^(n) (via mean pooling followed by the MLP projector hψh_ψ). The same chain rule gives ∂ℒNT-Xent/∂fθ()(i)=(∂ℒNT-Xent/∂x)⋅1ND _NT -Xent/∂ f_θ(x)^(i)=( _NT -Xent/ _x)· 1NI_D, also uniform. This argument holds for any projector architecture where hψh_ψ acts on the pooled token xs_x rather than on individual patch tokens. Gradient of ℒpredL_pred. ℒpredL_pred receives only context tokens fθ()|f_θ(x)|_C as input to gϕg_φ; masked tokens i∉i do not appear in the computational graph of ℒpredL_pred. The stop-gradient on y,n(t)=fθ()(t)s_y,n^(t)=f_θ(y)^(t) blocks all flow through the target branch. Therefore ∂ℒpred/∂fθ()(i)=0 _pred/∂ f_θ(x)^(i)=0 for all i∉i . In expectation over random masking, the predictive gradient is non-zero for every i but spatially non-uniform, its magnitude depending on local prediction difficulty. The combined gradient field from ℒinvL_inv (uniform global) and ℒpredL_pred (spatially local, context-concentrated) is structurally non-conflicting at the encoder output. We emphasize that this disjointness is stated at the encoder output; the two fields still share the encoder parameters θ, so parameter-space alignment (Remark LABEL:remark:param_conflict) is a separate, empirical matter. ∎ 0.F.3 Regularization Prevents the Collapsed Solution Lemma 1. VCReg is strictly positive at the collapsed solutionvar_collapse If fθf_θ is collapsed (y,n=s_y,n=c for all n), then ℒVCReg≥μ(γ0−δ)>0L_VCReg≥μ( _0- δ)>0 for any μ>0μ>0 and γ0>δ _0> δ. Proof. At collapse Var((:,d))=0Var(P^(:,d))=0 for every d, so from Equation˜8 ℒvar()=γ0−δ>0L_var(P)= _0- δ>0 since γ0=1>10−2=δ _0=1>10^-2= δ, giving ℒVCReg≥μ(γ0−δ)>0L_VCReg≥μ( _0- δ)>0. ∎ Theorem 0.F.1. VCReg prevents the collapsed global minimizervcreg_prevents For any γ>0γ>0, the collapsed solution is not a global minimizer of ℒtotal=αℒMSE+βℒpred+γℒVCRegL_total= _MSE+ _pred+ _VCReg. Proof. By Theorem LABEL:theorem:collapse and Lemma LABEL:lemma:var_collapse the collapsed solution achieves ℒMSE=ℒpred=0L_MSE=L_pred=0 and ℒVCReg≥μ(γ0−δ)>0L_VCReg≥μ( _0- δ)>0, so ℒtotal|collapsed≥γμ(γ0−δ)>0L_total|_collapsed≥γμ( _0- δ)>0. Any encoder whose projections satisfy Var((:,d))≥γ02Var(P^(:,d))≥ _0^2 with zero off-diagonal covariance (achievable by centering and scaling per dimension) has ℒVCReg=0L_VCReg=0, so ℒtotal=αℒMSE+βℒpredL_total= _MSE+ _pred, and each term can be driven toward zero by training. Hence ℒtotalL_total can be made strictly below γμ(γ0−δ)γμ( _0- δ), and the collapsed solution is not a global minimizer. ∎ Lemma 2. SIGReg and RDMReg are strictly positive at the collapsed solutionsigreg_rdmreg_positive If fθf_θ is collapsed, the empirical batch distribution is a Dirac mass δ′ _c , and ℒSIGReg>0L_SIGReg>0, ℒRDMReg>0L_RDMReg>0. Proof. For SIGReg, the characteristic function of δ′ _c at any ≠ ω 0 has modulus one and deviates strictly from the Gaussian target e−‖2/2<1e^-\| ω\|^2/2<1; since the quadrature includes such a point, ℒSIGReg>0L_SIGReg>0. For RDMReg, the sliced Wasserstein distance between a Dirac mass and a continuous target is strictly positive. In both cases Theorem LABEL:theorem:vcreg_prevents applies with ℒregL_reg replaced accordingly. ∎ Remark 3. Consistency with Definition LABEL:def:regreg_consistency Lemmas LABEL:lemma:var_collapse and LABEL:lemma:sigreg_rdmreg_positive verify that all three regularizers are strictly positive at every Dirac mass, satisfying the first clause of Definition LABEL:def:reg. The minimizers of each regularizer are full-rank over ℝDR^D: SIGReg is uniquely minimized by the standard isotropic Gaussian, RDMReg by the rectified generalized Gaussian target RGGX_RGG, and VCReg by any per-dimension variance-floored, decorrelated distribution, all of which are full-rank. Hence each studied regularizer satisfies Definition LABEL:def:reg. 0.F.4 Full Proof of Theorem LABEL:theorem:ema_collapse Full proof of Theorem LABEL:theorem:ema_collapse. Assumption: θ(t)→θ∗θ(t)→θ^* as t→∞t→∞. The continuous-time momentum update θ¯˙=(1−m)(θ−θ¯) θ=(1-m)(θ- θ) has solution θ¯(t)=e−(1−m)tθ¯(0)+(1−m)∫0te−(1−m)(t−s)θ(s)s θ(t)=e^-(1-m)t θ(0)+(1-m) _0^te^-(1-m)(t-s)θ(s)\,ds. For any ϵ>0ε>0 choose T with ‖θ(s)−θ∗‖≤ϵ\|θ(s)-θ^*\|≤ε for s≥Ts≥ T; then for t>Tt>T, ‖θ¯(t)−θ∗‖≤e−(1−m)t‖θ¯(0)−θ∗‖+e−(1−m)(t−T)M+ϵ\| θ(t)-θ^*\|≤ e^-(1-m)t\| θ(0)-θ^*\|+e^-(1-m)(t-T)M+ε with M=sups‖θ(s)−θ∗‖<∞M= _s\|θ(s)-θ^*\|<∞. As t→∞t→∞ both exponentials vanish, so ‖θ¯(t)−θ∗‖≤ϵ\| θ(t)-θ^*\|≤ε; since ϵε is arbitrary, θ¯(t)→θ∗ θ(t)→θ^*. If θ∗θ^* is collapsed the momentum target converges to that state, whereas Theorem LABEL:theorem:vcreg_prevents removes the collapsed state as a fixed point for any γ>0γ>0. We note that the assumption of convergence of θ(t)θ(t) simplifies the true coupled dynamics, which involve stop-gradient and are more complex; the theorem characterizes fixed points rather than training trajectories. ∎ Appendix 0.G Batch Size and the Contrastive Gradient We establish here that ℒpredL_pred and ℒregL_reg maintain informative gradients independently of batch size, complementing the discussion of Remark LABEL:remark:ntxent_partial on the self-limiting nature of the contrastive gradient. ℒpredL_pred (Equation˜5) is a sum over samples and target patches, each term depending only on its own sample. The regularizers (ℒSIGRegL_SIGReg, ℒVCRegL_VCReg, ℒRDMRegL_RDMReg) depend on within-batch statistics but require no paired negatives and maintain a strictly positive gradient at the collapsed solution for any B≥2B≥ 2 (Lemmas LABEL:lemma:var_collapse and LABEL:lemma:sigreg_rdmreg_positive). By contrast, the NT-Xent gradient scales with the number of in-batch negatives K=B−1K=B-1; as B decreases, the implicit anti-collapse signal of Remark LABEL:remark:ntxent_partial weakens proportionally, making explicit ℒregL_reg increasingly critical at small batch sizes. Table 6 shows linear-probe accuracy at B∈128,256,512B∈\128,256,512\ for the contrastive-only (row H) and full-model (row J) configurations. Table 6: Batch-size sensitivity of contrastive-only versus full-model configurations. Without explicit regularization (row H), accuracy degrades more steeply as batch size decreases because the contrastive anti-collapse signal weakens with fewer negatives. The full model (row J) is more robust due to ℒregL_reg maintaining a batch-size-independent gradient. ℒinvL_inv ℒregL_reg B=128B=128 B=256B=256 B=512B=512 H NT-Xent none 44.2 48.1 51.6 J NT-Xent SIGReg 52.8 53.7 55.0 Appendix 0.H Supplementary Experiment: Effective Rank Dynamics Figure 6 traces effective rank across training for the collapse experiment, the dynamic counterpart of Figure 1, as across-seed means over five seeds. Negative-free runs without regularization remain low-rank; contrastive runs begin at higher rank through implicit repulsion but do not stabilize on their own; the explicit regularizer holds effective rank in a healthy band, confirming Theorem LABEL:theorem:collapse and Remark LABEL:remark:ntxent_partial. The MSE + JEPA + SIGReg run (row I) stabilizes at low effective rank despite the regularizer, consistent with the gradient-magnitude imbalance discussed in Section 6: the SIGReg gradient is insufficient to overcome the strong collapsing pull of the MSE term at the chosen hyperparameter settings. Figure 6: Effective rank over training. Negative-free alignment without regularization stays low-rank; the contrastive term raises rank but does not stabilize it; the explicit regularizer holds it in a healthy band. The MSE + SIGReg run (row I) stabilizes at low rank, reflecting a gradient-magnitude imbalance between MSE and SIGReg at the chosen hyperparameter settings. Curves are means over five seeds. Appendix 0.I Supplementary Experiment: Necessity of Each Principle Figure 2 summarizes the marginal contribution of each principle, the accuracy gained by adding it to the pair that omits it; no principle is redundant at the studied scale, and observation contributes the largest gain. Figure 7 traces linear-probe accuracy across training for every strict subset, the dynamic counterpart, with the all-three configuration dominating throughout and the single-principle runs clustering well below. All curves are across-seed means over five seeds. Figure 7: Linear-probe accuracy over training for every strict subset. The all-three configuration (top) dominates; single-principle runs cluster well below. Curves are means over five seeds. Appendix 0.J Supplementary Experiment: Regularizer Curves Figure 8 shows the linear-probe trajectories for the four regularizer choices, as across-seed means over five seeds. The isotropic-Gaussian and sliced-Wasserstein regularizers converge to the same accuracy; the variance-covariance regularizer lags despite the highest effective rank, consistent with the gradient-conflict hypothesis discussed in Section 6; the negative-free alignment trails every contrastive variant. Figure 8: Linear-probe accuracy for four regularizer choices. The isotropic-Gaussian and sliced-Wasserstein regularizers are interchangeable; the variance-covariance regularizer lags despite the highest effective rank; the negative-free alignment trails every contrastive variant. Curves are means over five seeds. Appendix 0.K Broader Impact Our contribution is analytical rather than a new state-of-the-art system: we identify which parts of a self-supervised objective are doing which work, and show that collapse prevention can be isolated in an explicit geometric term rather than entangled with the primary training signal. If this decomposition holds at scale, it offers a design discipline for future self-supervised algorithms, in which observation, prediction, and regularization can be specified, swapped, and diagnosed independently instead of being rediscovered implicitly through architectural tricks such as momentum encoders, centering, or carefully tuned asymmetries. This modularity also lowers the cost of principled ablation, since a practitioner can attribute a failure to a missing principle rather than to an opaque interaction, and it suggests a route to extending the same accounting to video and vision-language pre-training, where the corresponding objectives are less well understood. The broader risks are those shared by representation learning generally: encoders trained on uncurated data inherit the biases of that data, and a framework that makes such training cheaper or easier to tune propagates those biases more widely. Our validation is at a single small scale, so the practical claims should be read as hypotheses about larger models rather than guarantees, and we would caution against treating the effective-rank diagnostics we report as a proxy for representation quality in deployment settings.