Paper deep dive
Which Tasks Survive Self-Supervised Learning?
Achleshwar Luthra, Lucas Bryant, Tracy Zhu, Tomer Galanti
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/1/2026, 8:09:58 AM
Summary
This paper investigates which downstream tasks remain recoverable from representations learned via same-instance self-supervised learning (SSL). It introduces 'semantic recoverability' as the amount of a task's posterior score captured by the representation's function space. The authors derive exact relationships between recoverability and directional class-distance-normalized variance (CDNV), few-shot nearest-centroid classification error, and multitask centroid geometry under centered and whitened representations. Furthermore, they show that for canonical two-view SSL objectives, the learned subspace spans the leading cross-view-stable modes of the two-view operator, providing a spectral characterization of which tasks survive SSL based on the overlap between the task posterior and the selected spectral subspace.
Entities (8)
Relation Signals (6)
Captured Posterior Energy → measures → Semantic Recoverability
confidence 96% · We call this quantity captured posterior energy, B(F), and use it to measure semantic recoverability
Semantic Recoverability → determines → Directional Class-Distance-Normalized Variance
confidence 95% · We show that, for centered and whitened representations, recoverability exactly determines directional class-distance-normalized variance (CDNV)
Semantic Recoverability → controls → Few-Shot Nearest-Centroid Classification
confidence 94% · recoverability ... controls few-shot nearest-centroid classification
Two-View Operator → hasmodes → Cross-View-Stable Modes
confidence 93% · its population optimum spans the leading cross-view-stable modes of the associated two-view operator.
SSL Objective → selects → Spectral Subspace
confidence 92% · a downstream task is preserved to the extent that its posterior lies in the selected spectral subspace.
Population Linear Probe → coincideswith → Centroid Axis
confidence 91% · The population linear probe and centroid axis coincide
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Same-instance self-supervised learning (SSL) learns representations by enforcing consistency across two views of the same underlying instance. This principle alone, however, does not determine which downstream tasks remain recoverable from the learned representation. We study this question through \emph{semantic recoverability}, defined as the amount of a task's posterior score captured by the represented function space. We show that, for centered and whitened representations, recoverability exactly determines directional class-distance-normalized variance (CDNV), controls few-shot nearest-centroid classification, and governs the strength of task-relevant semantic directions. The population linear probe and centroid axis coincide, and multiple well-recovered tasks approach a factorial centroid geometry. We then analyze a canonical two-view SSL objective and show that its population optimum spans the leading cross-view-stable modes of the associated two-view operator. This yields a closed-form spectral characterization of semantic recoverability: a downstream task is preserved to the extent that its posterior lies in the selected spectral subspace. We validate these predictions on synthetic and real datasets across several SSL methods, testing the predicted relationships among recoverability, directional geometry, spectral structure, and few-shot transfer. Together, these results give a task-level account of what information survives same-instance SSL and how the retained information appears in downstream geometry and transfer.
Tags
Links
- Source: https://arxiv.org/abs/2609.38393v1
- Canonical: https://arxiv.org/abs/2609.38393v1
Trouble viewing inline? Open PDF directly →
Full Text
104,881 characters extracted from source content.
Expand or collapse full text
Which Tasks Survive Self-Supervised Learning? Achleshwar Luthra Lucas Bryant Tracy Zhu Tomer Galanti †thanks: Correspondence to galanti@tamu.edu Affiliation: Department of Computer Science and Engineering Affiliation: Texas A&M University Abstract Same-instance self-supervised learning (SSL) learns representations by enforcing consistency across two views of the same underlying instance. This principle alone, however, does not determine which downstream tasks remain recoverable from the learned representation. We study this question through semantic recoverability, defined as the amount of a task’s posterior score captured by the represented function space. We show that, for centered and whitened representations, recoverability exactly determines directional class-distance-normalized variance (CDNV), controls few-shot nearest-centroid classification, and governs the strength of task-relevant semantic directions. The population linear probe and centroid axis coincide, and multiple well-recovered tasks approach a factorial centroid geometry. We then analyze a canonical two-view SSL objective and show that its population optimum spans the leading cross-view-stable modes of the associated two-view operator. This yields a closed-form spectral characterization of semantic recoverability: a downstream task is preserved to the extent that its posterior lies in the selected spectral subspace. We validate these predictions on synthetic and real datasets across several SSL methods, testing the predicted relationships among recoverability, directional geometry, spectral structure, and few-shot transfer. Together, these results give a task-level account of what information survives same-instance SSL and how the retained information appears in downstream geometry and transfer. 1 Introduction Self-supervised learning (SSL) (Balestriero et al., 2023) learns representations without labels and performs well across vision, language, speech, and multimodal learning (Chen et al., 2020; He et al., 2020; Zbontar et al., 2021; He et al., 2022; Oquab et al., 2024; Gao et al., 2021; Reimers and Gurevych, 2019; Schneider et al., 2019; Baevski et al., 2020; Hsu et al., 2021; Baevski et al., 2022; Radford et al., 2021; Jia et al., 2021; Zhai et al., 2023; Tschannen et al., 2025). A common principle is simple: two views of the same instance should receive compatible representations. Yet two views can share many kinds of information: semantic content, style, background, nuisance structure, or accidental regularities. Agreement alone therefore does not reveal which downstream distinctions survive. This leads to a basic task-level question: given a representation learned only from paired views, which semantic tasks can still be recovered from it? For example, a representation learned from color-augmented face images might preserve whether a person is smiling while losing information about hair color. Existing theory provides two pieces of the answer. Operator and spectral analyses characterize which cross-view-stable directions are preferred by idealized SSL objectives (Zhai et al., 2025; Zhai, 2025; Ermolov et al., 2021; Weng et al., 2022), while geometric analyses show that SSL representations can develop strong label-aligned directional structure without global class collapse (Luthra et al., 2025; Wang and Isola, 2020; Garrido et al., 2023; Balestriero and LeCun, 2022; Qiu et al., 2023). We therefore ask: Which downstream tasks survive same-instance SSL, and what geometry and few-shot behavior follow from their recoverability? Our starting point is that this question is task-dependent. A representation can preserve one semantic partition while discarding another. For a fixed downstream task, the relevant object is the part of its posterior score that remains visible in the represented function space, as illustrated in Fig. 2. We call this quantity captured posterior energy, B(F)B(F), and use it to measure semantic recoverability independently of how the representation was learned. Our theory addresses two questions: why semantic recoverability matters, and when SSL makes a task recoverable. Our contributions are threefold. (i) We formulate task survival through captured posterior energy, separating recoverability from any particular SSL objective. (i) For centered and whitened representations, we derive an exact relation between recoverability and directional class-distance-normalized variance (CDNV) (Luthra et al., 2025), establish probe–centroid alignment, obtain a direct finite-sample guarantee for few-shot nearest-centroid classification, and characterize the joint centroid geometry of multiple well-recovered tasks. (i) For a canonical two-view population SSL objective, we show that the learned subspace is spanned by the leading cross-view-stable modes of the associated two-view operator. Consequently, semantic recoverability is exactly the spectral overlap between the task posterior and the selected subspace, yielding a criterion for which downstream tasks survive SSL. We further extend this subspace perspective to covariance-regularized and prediction-based SSL surrogates in the appendix. 1.1 Related Work Our work connects two lines of SSL theory that are usually studied separately: task-relevant representation geometry and the population subspaces selected by two-view objectives. Directional geometry and downstream transfer. SSL representations can support strong downstream classification and retrieval, while remaining globally anisotropic and retaining substantial variance in nuisance directions (Chen et al., 2020; Caron et al., 2021; Ben-Shaul et al., 2023; Weng et al., 2025; Oquab et al., 2024; Wang and Isola, 2020; Wang and Liu, 2021; Chen et al., 2021; HaoChen et al., 2021; Saunshi et al., 2019). This motivates task-aware geometric measures rather than global collapse. In particular, directional CDNV isolates within-class variation along the semantic decision direction and has been shown to track linear-probe and few-shot transfer across SSL methods (Luthra et al., 2025). Related work on neural collapse and few-shot transfer connects class geometry to generalization (Papyan et al., 2020; Han et al., 2022; Galanti et al., 2022c; Galanti et al., 2022b; Galanti et al., 2022a; Wang et al., 2023; Goldblum et al., 2020). Our goal is complementary: we ask what property of a label-free learned subspace predicts whether this favorable task geometry will appear. Spectral and operator views of SSL. Many SSL methods combine cross-view agreement with mechanisms that prevent representational collapse (Wang and Isola, 2020; Zbontar et al., 2021; Bardes et al., 2022; Garrido et al., 2023; Balestriero and LeCun, 2022; Ermolov et al., 2021). At the population level, whitening-style formulations connect naturally to CCA, maximal correlation, and conditional expectation operators (Hotelling, 1936; Hannan, 1961; Breiman and Friedman, 1985; Lai and Fyfe, 2000; Hardoon et al., 2004; Fukumizu et al., 2007; Andrew et al., 2013; Michaeli et al., 2016; Asoodeh et al., 2015). Recent analyses characterize the subspace selected by such objectives through the leading spectrum of an induced two-view operator (Zhai et al., 2025; Zhai, 2025). We use this characterization for computing which task posteriors are recoverable from the selected subspace. Relation to broader SSL theory. A broader literature studies why SSL captures useful structure through mutual-information views (Bachman et al., 2019; McAllester and Stratos, 2020; Tschannen et al., 2020), alignment and uniformity (Wang and Isola, 2020; Wang and Liu, 2021; Chen et al., 2021), latent-variable recovery and augmentation structure (Saunshi et al., 2019; Tosh et al., 2021; Zimmermann et al., 2021; Ash et al., 2022; Nozawa and Sato, 2021; HaoChen et al., 2021; Shen et al., 2022; Awasthi et al., 2022; Saunshi et al., 2022), and optimization, projection heads, and sample complexity (HaoChen and Ma, 2023; Parulekar et al., 2023; Wen and Li, 2021; Tian, 2023; Tian et al., 2020; Gupta et al., 2022; Gui et al., 2023; Alon et al., 2024). Our question is narrower and task-conditional: given the information retained by a learned representation, which downstream posterior scores survive, and what geometric and few-shot consequences follow? 2 Problem Setup We consider a latent instance that generates an observed sample and two conditionally independent views, together with a semantic task defined at the instance level. A representation is learned from paired views without access to the task labels and is then evaluated by how well the task can be recovered from a small labeled support set. 2.1 Latent Instance Model and Two-View Generation Let C denote the latent instance (for example, the underlying object, scene, or document) taking values in a measurable space C, with distribution PCP_C. The semantic label is determined at the instance level: Y=y(C)∈±1Y~=~y(C)∈\± 1\, ℙ(Y=+1)=ℙ(Y=−1)=12P(Y=+1)~=~P(Y=-1)~=~ 12. Thus C may contain much more information than Y: many latent instances can share the same label. We let X(0)X^(0) denote the sample associated with C, drawn as X(0)∼P0(⋅∣C)X^(0) P_0(· C). A positive pair is formed by drawing two independent views, X(1),X(2)∼i.i.d.P(⋅∣X(0))X^(1),X^(2) .i.d. P(· X^(0)). In vision, for example, C may encode image content, while each view is produced by a random augmentation such as cropping, color jitter, or blur. Let PXP_X denote the marginal law of a single view. 2.2 Downstream Few-Shot Learning Let F:→ℝrF:X ^r be a representation. We evaluate its downstream quality through the expected m-shot error of a classifier built on top of F. Given a support set with m labeled examples per class, with all support examples sampled independently, Sm=(Xs,i,s):s∈±1S_m=\(X_s,i,s):s∈\± 1\, Xs,i∼P(⋅∣Y=s)X_s,i P(· Y=s), (i∈[m]i∈[m])\, let g^SmA g^A_S_m denote the classifier returned by a learning rule AA from the embedded support set. We define errmA(F):=Sm[ℙ(X,Y)(g^SmA(F(X))≠Y)] err^A_m(F):=E_S_m\! [P_(X,Y)\! ( g^A_S_m(F(X))≠ Y ) ], where (X,Y)(X,Y) is drawn from the single-view distribution induced by the latent-instance model. We focus on nearest-class-centroid (NCC) classification: gSmNCC(z):=argmins∈±1‖z−μ^s‖2g^NCC_S_m(z):= _s∈\± 1\\|z- μ_s\|_2, where μ^s:=1m∑i=1mF(Xs,i) μ_s:= 1m _i=1^mF(X_s,i). This gives an operational notion of task survival: a useful representation should make the task learnable from only a small labeled support set. The next section identifies a population quantity that controls when this happens. 3 Semantic Recoverability and Its Consequences Task stableacross viewsRecoverabilityBr=∑j≤rβj2B_r= _j≤ r _j^2 (i) low directional CDNV V~F=1−Br2Br V_F= 1-B_r2B_r (i) few-shot NCC errmNCC≲1−Br+rmerr^NCC_m 1-B_r+ rm (i) multitask geometry mY≈(Br(t)Yt)t≤km_Y≈ ( B_r^(t)\,Y_t )_t≤ k top-r modesof T Figure 1: Conceptual flow of the theory. Same-instance SSL retains the most cross-view-stable modes of the two-view operator T (Prop. 4.1), so a task survives to the extent that its posterior overlaps that subspace (Cor. 4.2). Under centered whitening, BrB_r fixes the directional collapse law (Prop. 3.1) and few-shot nearest-centroid guarantee (Thm. 3.4); the task-specific Br(t)B_r^(t) determine the multitask centroid geometry (Thm. 3.5). 3.1 Captured Posterior Energy For the balanced task, define the posterior score η(x):=[Y∣X=x]η(x):=E[Y X=x]. The posterior contains the information about Y available from a single view. For a representation F=(F1,…,Fr)F=(F_1,…,F_r), let F:=spanF1,…,Fr⊂L2(PX)S_F:=span\F_1,…,F_r\⊂ L^2(P_X) denote its represented function space. We define the captured posterior energy B(F):=‖ΠFη‖L2(PX)2B(F):=\| _S_Fη\|_L^2(P_X)^2, where ΠF _S_F is the orthogonal projection onto FS_F. This definition does not require SSL or whitening. It asks directly how much of the Bayes-relevant posterior score remains in the function space represented by F. The dependence on the downstream task is intentional: the same representation may preserve one semantic partition and discard another. If one wants a literal fraction of the single-view task signal that is retained, it is B(F)/‖η‖L2(PX)2B(F)/\|η\|_L^2(P_X)^2. In particular, large B(F)B(F) means that most of the posterior is expressible through the representation, while small B(F)B(F) means that the task is largely absent from the represented subspace. The quantity is invariant to invertible changes of coordinates in feature space because such changes leave FS_F unchanged. Whitening enters below for another reason: it converts this intrinsic subspace quantity into exact Euclidean statements about feature geometry and few-shot classification. 3.2 Why Recoverability Matters Under Whitening Assume for the remainder of this section that F is centered and whitened, so that [F(X)]=0E[F(X)]=0 and [F(X)F(X)⊤]=IrE[F(X)F(X) ]=I_r. The coordinate functions of F are then orthonormal in L2(PX)L^2(P_X), and the intrinsic recoverability quantity has the simple feature-space form B(F)=‖[YF(X)]‖22B(F)=\|E[YF(X)]\|_2^2. Let μs:=[F(X)∣Y=s] _s:=E[F(X) Y=s], Σs:=Cov(F(X)∣Y=s) _s:=Cov(F(X) Y=s), and Δ:=μ+−μ− := _+- _-. When Δ≠0 ≠ 0, let u:=Δ/‖Δ‖2u:= /\| \|_2. The CDNV (VFV_F) and directional CDNV (V~F V_F) are VF:=(Tr(Σ+)+Tr(Σ−))/‖Δ‖22,V~F:=(u⊤Σ+u+u⊤Σ−u)/‖Δ‖22.V_F:=(Tr( _+)+Tr( _-))/\| \|_2^2, V_F:=(u _+u+u _-u)/\| \|_2^2. (1) When Δ=0 =0, we set VF=V~F=+∞V_F= V_F=+∞. Under whitening, total within-class variation can remain large even when variation along the semantic direction is small. 3.2.1 Exact directional-collapse law Proposition 3.1. Let F:→ℝrF:X ^r be centered and whitened, and let Y∈±1Y∈\± 1\ be balanced. Then VF=(r−B(F))/(2B(F))V_F=(r-B(F))/(2B(F)) and V~F=(1−B(F))/(2B(F)) V_F=(1-B(F))/(2B(F)), with both quantities equal to +∞+∞ when B(F)=0B(F)=0. Prop. 3.1 gives an exact population law: directional CDNV vanishes as recoverability approaches one and diverges as recoverability vanishes. Thus the representation need not collapse globally for the task to become geometrically simple; it only needs to collapse along the semantic direction that survives in the represented subspace. Remark 3.2 (Directional vs. global collapse). Even at B(F)=1B(F)=1, the global CDNV is VF=(r−1)/2V_F=(r-1)/2, large whenever r≥3r≥ 3. Whitening fixes the total representation variance, so within-class spread remains in directions orthogonal to the task axis. Global collapse is therefore neither expected nor necessary. When the label is fully determined by a single view, Y=g(X)Y=g(X), we have B(F)=1B(F)=1 if and only if η∈Fη _F, giving perfect directional collapse despite potentially large global CDNV. 3.2.2 Probe–centroid alignment A second consequence of whitening is that the population squared-loss probe and the centroid decision axis coincide. Corollary 3.3 (MSE probe and decision-axis alignment). Let F:→ℝrF:X ^r be centered and whitened, and let Y∈±1Y∈\± 1\ be balanced. Define μs:=[F(X)∣Y=s] _s:=E[F(X) Y=s] for s∈±1s∈\± 1\, and let Δ:=μ+−μ− := _+- _-. Then the unique population MSE probe, (w⋆,b⋆)∈argminw∈ℝr,b∈ℝ[(Y−(w⊤F(X)+b))2](w ,b )∈ _w ^r,b E[(Y-(w F(X)+b))^2], satisfies b⋆=0b =0 and w⋆=[YF(X)]=12Δw =E[YF(X)]= 12 . In particular, μ−=−μ+ _-=- _+, and Δ=2μ+=2w⋆ =2 _+=2w . Cor. 3.3 shows that linear probing and NCC use the same separating direction. Moreover, ‖w⋆‖22=B(F)\|w \|_2^2=B(F), so the strength of the population probe is itself a measurement of semantic recoverability. 3.2.3 Few-shot transfer guarantees The geometric identities above translate recoverability into an operational downstream guarantee. In the whitened setting, the same scalar that controls directional geometry also controls the expected error of an m-shot nearest-centroid classifier. Theorem 3.4 (Few-shot NCC bound via captured posterior energy). Let F:→ℝrF:X ^r be centered and whitened, and let Y∈±1Y∈\± 1\ be balanced. Then, for every m≥1m≥ 1, errmNCC(F)≤1−B(F)+r−B(F)m+1−B(F)1−B(F)+2mB(F)=2V~F1+2V~F+(r−1)+2rV~Fm(1+2V~F)+V~FV~F+m,err^NCC_m(F)≤ 1-B(F)+ r-B(F)m+ 1-B(F)1-B(F)+2mB(F)= 2 V_F1+2 V_F+ (r-1)+2r V_Fm(1+2 V_F)+ V_F V_F+m, where the second expression is understood by continuity when B(F)=0B(F)=0. The bound separates two effects. The leading term reflects task information not captured by the representation, while the remaining terms arise from estimating class centroids from finitely many examples. Whitening is what makes this a one-scalar Euclidean statement: without it, coordinate rescaling could change ‖[YF]‖22\|E[YF]\|_2^2 and Euclidean NCC would no longer coincide with the natural population geometry. The coordinate-invariant analogue is obtained by rewhitening, or equivalently by using a Mahalanobis metric in the original feature space. 3.2.4 Multitask semantic geometry Recoverability also determines how several semantic tasks are organized jointly. We use a superscript to index tasks, reserving the subscript r for representation rank. Consider k balanced binary tasks Y1,…,Yk∈±1Y_1,…,Y_k∈\± 1\, and write Y:=(Y1,…,Yk)Y:=(Y_1,…,Y_k). For each task t, let ηt(x)=[Yt∣X=x] _t(x)=E[Y_t X=x], B(t)(F)=‖ΠFηt‖L2(PX)2B^(t)(F)=\| _S_F _t\|_L^2(P_X)^2, and εt=[Var(Yt∣X)]=1−‖ηt‖L2(PX)2 _t=E[Var(Y_t X)]=1-\| _t\|_L^2(P_X)^2. Let wt=[YtF(X)]w_t=E[Y_tF(X)]. Assuming B(t)(F)>0B^(t)(F)>0, define ut=wt/‖wt‖2u_t=w_t/\|w_t\|_2 and Zt(X)=⟨ut,F(X)⟩Z_t(X)= u_t,F(X) . Theorem 3.5 (Near-orthogonality and centroid geometry). Let F:→ℝrF:X ^r be centered and whitened. Assume each Yt∈±1Y_t∈\± 1\ is balanced and B(t)(F)>0B^(t)(F)>0 for all t; arbitrary dependence among Y1,…,YkY_1,…,Y_k is allowed. For i≠ji≠ j, define ρij:=[YiYj] _ij:=E[Y_iY_j]. Then, for any i≠ji≠ j, |ui⊤uj|≤(|ρij|+εiεj+(‖ηi‖L22−B(i)(F))(‖ηj‖L22−B(j)(F)))/B(i)(F)B(j)(F).|u_i u_j|~≤~ (| _ij|+ _i _j+ (\| _i\|_L^2^2-B^(i)(F) ) (\| _j\|_L^2^2-B^(j)(F) ) )/ B^(i)(F)B^(j)(F). Moreover, letting mY:=[Z(X)∣Y]∈ℝkm_Y:=E[Z(X) Y] ^k, [‖mY−(B(1)(F)Y1,…,B(k)(F)Yk)‖22]≤∑t=1k(1−B(t)(F)).E [ \|m_Y- ( B^(1)(F)Y_1,…, B^(k)(F)Y_k ) \|_2^2 ]~≤~ _t=1^k (1-B^(t)(F) ). Thm. 3.5 shows that well-recovered tasks induce strong semantic axes that become nearly orthogonal when task correlations and residual posterior uncertainty are small. Their joint centroids approach (B(t)(F)Yt)t≤k ( B^(t)(F)Y_t )_t≤ k, yielding an approximately axis-aligned hyperrectangle whose side lengths are set by the task-specific capture values. Thus recoverability controls the strength of each semantic direction, while cross-task dependence determines how closely the overall geometry approaches a factorial product. The results so far deliberately make no claim about how F was learned. They say that if a representation captures a task, then B(F)B(F) determines its directional geometry, its few-shot behavior, and its role in multitask centroid structure. We now turn to the complementary question: why should a same-instance SSL objective produce a large value of B(F)B(F) for some tasks and not others? 4 Recoverability Under Same-Instance SSL The preceding results show that the recoverability quantity B(F)B(F) characterizes the task-relevant geometry of a representation and controls its few-shot behavior. The next question is whether, for a representation learned by same-instance SSL, this quantity can itself be identified explicitly from the SSL objective. We show that this is indeed the case: under the whitening-based population objective, the learned representation spans the leading eigenspace of the two-view operator, and B(F)B(F) becomes the spectral overlap between the downstream posterior and these view-stable modes. 4.1 A Canonical Two-View Population Objective Many SSL methods (Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2022; Chen et al., 2020; Caron et al., 2021; Ermolov et al., 2021; Tao et al., 2022) combine instance-level agreement with an anti-collapse mechanism that promotes global spread, decorrelation, or isotropy (Wang and Isola, 2020; Garrido et al., 2023; Balestriero and LeCun, 2022). To obtain a clean population characterization of the represented subspace, we study the canonical whitening-based surrogate (W-MSE) (Ermolov et al., 2021; Weng et al., 2022) maxF:→ℝr[⟨F(X(1)),F(X(2))⟩]subject to[F(X)]=0,[F(X)F(X)⊤]=Ir. _F:X ^rE\! [ F(X^(1)),F(X^(2)) ] to [F(X)]=0, [F(X)F(X) ]=I_r. (2) The objective rewards agreement across two views of the same instance, while whitening rules out collapse and fixes the feature metric. Its role in our theory is specific: it gives an analyzable rule for which r-dimensional function space is selected from the two-view distribution. Strict whitening is the cleanest setting for this identification and for the exact geometry above. App. C studies a covariance-regularized surrogate related to Barlow Twins (Zbontar et al., 2021) and VICReg (Bardes et al., 2022), while App. D gives a corresponding reduced-rank regression view of prediction-based objectives such as BYOL (Grill et al., 2020), SimSiam (Tao et al., 2022), and JEPA (Assran et al., 2023). Figure 2: Geometric interpretation of semantic recoverability. The shaded plane is the SSL-selected subspace r=spanψ1,ψ2U_r=span\ _1, _2\ of leading two-view eigenfunctions. The task posterior η decomposes into its projection Πrη _rη onto this subspace and an orthogonal residual. From left to right, the posterior is fully captured, partially captured, or discarded. The captured posterior energy Br=‖Πrη‖L2(PX)2B_r=\| _rη\|_L^2(P_X)^2 measures the retained task-relevant information. 4.2 The Two-View Operator and the Selected Subspace Define the two-view conditional expectation operator T:L2(PX)→L2(PX)T:L^2(P_X)→ L^2(P_X) by (Tf)(x):=[f(X(2))∣X(1)=x](Tf)(x):=E[f(X^(2)) X^(1)=x]. Thus T maps a function of one view to its conditional expectation from another view of the same instance. In the conditionally i.i.d. two-view model, exchangeability gives ⟨f,Tg⟩=⟨Tf,g⟩ f,Tg = Tf,g , while conditional independence given X(0)X^(0) gives ⟨f,Tf⟩=[([f(X)∣X(0)])2]≥0 f,Tf =E[(E[f(X) X^(0)])^2]≥ 0. Hence T is self-adjoint and positive semidefinite, and an eigenvalue measures the cross-view stability of the corresponding mode. Proposition 4.1. Let L02(PX):=f∈L2(PX):[f(X)]=0L_0^2(P_X):=\f∈ L^2(P_X):E[f(X)]=0\, and assume that T restricted to L02(PX)L_0^2(P_X) admits an orthonormal eigenbasis (ψj)j≥1( _j)_j≥ 1, with eigenvalues λ1≥λ2≥⋯≥0 _1≥ _2≥·s≥ 0. Assume λr>0 _r>0 and λr>λr+1 _r> _r+1. Then ψ1,…,ψr _1,…, _r are feasible for (2) and achieve the optimal value ∑j=1rλj _j=1^r _j. Moreover, if F:→ℝrF:X ^r is any optimal solution of (2), then its coordinates span the same r-dimensional subspace as ψ1,…,ψr _1,…, _r. Prop. 4.1 says that the population objective keeps the r most cross-view-stable modes. The representation is determined only up to an orthogonal change of basis within this selected subspace. This result identifies what SSL preserves without referring to any downstream labels; the task enters only when we ask how much of its posterior lies in that subspace. 4.3 Closed-Form Recoverability at the SSL Optimum Let F⋆F be any population-optimal solution of (2), and define r:=spanψ1,…,ψr⊂L2(PX)U_r:=span\ _1,…, _r\⊂ L^2(P_X), with Πr _r denoting the orthogonal projection onto rU_r. For a balanced binary label with posterior score η(x)=[Y∣X=x]η(x)=E[Y X=x], write βj:=⟨η,ψj⟩L2(PX) _j:= η, _j _L^2(P_X). Corollary 4.2. Under the assumptions of Prop. 4.1, Πrη=∑j=1rβjψj _rη= _j=1^r _j _j, and B(F⋆)=‖Πrη‖L2(PX)2=∑j=1rβj2=:BrB(F )=\| _rη\|_L^2(P_X)^2= _j=1^r _j^2=:B_r. For multiple downstream tasks, we write Br(t)B_r^(t) for the rank-r capture of task t; equivalently, Br(t)=B(t)(F⋆)B_r^(t)=B^(t)(F ) for an optimal rank-r representation. This is the closed-form answer to when recoverability is large under the SSL model. The task is preserved when its posterior score is concentrated on the modes that are most stable across the two views, and it is discarded when its posterior lies primarily in view-variant directions outside the selected subspace. The two-view objective itself remains label-free; labels enter only through the overlap used to evaluate a particular downstream task. Combining this spectral identity with the previous section immediately turns BrB_r into task-level predictions. A large spectral overlap implies low directional CDNV, strong few-shot NCC performance, and pronounced semantic axes; across several tasks, the values Br(t)B_r^(t) determine the side lengths of the approximate hyperrectangle. In the recoverable regime where the augmentations preserve the task and the increasing view-stable subspaces capture its posterior, BrB_r approaches one as the representation dimension grows. Thus the overall mechanism is: SSL selects view-stable structure, a task survives according to its posterior overlap with that structure, and the amount that survives determines the downstream geometry and transfer behavior. 5 Experiments (a) W-MSE-CelebA (b) VICReg-CelebA (c) Barlow-IM1K (d) I-JEPA-IM1K Figure 3: Captured posterior energy on CelebA. Mean Br(t)B_r^(t) over the 1, 5, and 10 most recoverable CelebA attributes and for all 40 attributes, shown across spectral ranks r. Shaded regions denote the corresponding interquartile ranges. The dashed curve is the mean for a matched randomly initialized encoder, with dotted curves showing its interquartile range. (a) W-MSE-CelebA (b) VICReg-CelebA (c) Barlow-IM1K (d) I-JEPA-IM1K Figure 4: Predicted versus observed directional CDNV on CelebA. Each point is one CelebA attribute at a given rank r. The prediction V~^pred=(1−B^A(t))/(2B^A(t)) V_pred=(1- B_A^(t))/(2 B_A^(t)) is computed on split A. Class centroids, the semantic direction, and observed directional CDNV are estimated independently on held-out split B. The dashed line is y=xy=x, and each panel contains all 40 attributes. (a) W-MSE-CelebA (b) I-JEPA-IM1K Figure 5: Captured posterior energy controls few-shot NCC error. Empirical m-shot NCC error versus task-specific Br(t)B_r^(t) across CelebA attributes, for several ranks r (panels) and support sizes m (colors). Each point corresponds to a downstream attribute, and dashed curves are the corresponding upper bounds from Thm. 3.4. (a) W-MSE-CelebA (b) I-JEPA-IM1K (c) Centroid RMSE (d) Task-axis overlap Figure 6: Multitask centroid geometry on CelebA. (a–b) Illustrative held-out joint centroids for three attributes, compared with the hyperrectangle predicted by Thm. 3.5; triplets are selected on the training split. (c–d) Results over all 1646 eligible triplets per encoder, without recoverability or orthogonality screening. Larger mean task recoverability is associated with lower normalized centroid RMSE and smaller maximum pairwise task-axis overlap. Curves show bin means and shaded regions show interquartile ranges; selection and normalization details are in App. B.1. (a) W-MSE-CelebA (b) VICReg-CelebA (c) Barlow-IM1K (d) I-JEPA-IM1K Figure 7: Recoverability from the paired-view spectral subspace. Directly measured task recoverability B^(t)(F) B^(t)(F) versus its reconstruction B^r⋆(t) B_r ^(t) from the label-free paired-view spectral modes. Each point is one CelebA attribute; the dashed line is perfect agreement. For ImageNet-pretrained encoders, the retained subspace is reduced-rank and the median absolute discrepancy is below 0.050.05 across models; additional encoders are shown in App. B.1. 5.1 Settings We test the main empirical consequences of the theory on natural and synthetic data. Full dataset, preprocessing, and training details are given in App. A. Datasets. The main experiments use CelebA (Liu et al., 2015), whose 40 binary attributes provide a collection of downstream semantic tasks. For each attribute, we balance the two classes by subsampling so that the empirical problem matches the binary setting of Sec. 2. Controlled experiments on dSprites (Matthey et al., 2017) and 3DShapes (Burgess and Kim, 2018) are reported in App. B.2. SSL methods and rank-r representations. We train W-MSE (Ermolov et al., 2021) and VICReg (Bardes et al., 2022) from scratch on CelebA with ResNet-50 (He et al., 2016) backbones. We also evaluate publicly available ImageNet-1K (Deng et al., 2009)-pretrained VICReg and Barlow Twins (Zbontar et al., 2021) ResNet-50 encoders and an I-JEPA (Assran et al., 2023) ViT-H/14 encoder. To vary representation rank, we estimate a paired-view spectral basis from unlabeled data and retain its leading r directions. The retained features are then centered and whitened before the label-dependent measurements of BrB_r below. Estimating semantic recoverability. For a balanced downstream task, let β^=^[YF(X)] β= E[YF(X)] and G^=^[F(X)F(X)⊤] G= E[F(X)F(X) ]. We estimate captured posterior energy by B^(F)=β^⊤G^−1β B(F)= β G^-1 β. This form accounts for finite-sample deviations from exact whitening. When G^=Ir G=I_r, it reduces to B^(F)=‖^[YF(X)]‖22 B(F)=\| E[YF(X)]\|_2^2. Unless noted otherwise, all reported recoverability values use this estimator. 5.2 Results Recoverability is task-dependent. Because the rank-r subspaces are nested, Br(t)B_r^(t) is nondecreasing in r. Fig. 3 nevertheless shows substantial variation across CelebA attributes: the most recoverable tasks approach Br(t)≈1B_r^(t)≈ 1 at moderate ranks, while the average over all 40 attributes remains lower. SSL representations also exceed random-initialization baselines across a broad range of ranks. Tasks differ markedly in both saturation rate and attainable recoverability. The same pattern appears for additional pretrained encoders in App. B.1 and on dSprites and 3DShapes in App. B.2. Recoverability predicts directional geometry. Prop. 3.1 gives the exact relation V~F=(1−B(F))/(2B(F)) V_F=(1-B(F))/(2B(F)) for centered, whitened representations. We test it with a split-sample protocol: recoverability is estimated on split A and directional CDNV independently on split B. Fig. 4 shows close agreement over more than two orders of magnitude, across ranks and SSL representations. Additional encoder results are given in App. B.1. Recoverability controls few-shot transfer. Thm. 3.4 predicts that, at fixed rank and shot count, larger Br(t)B_r^(t) yields lower NCC error, while increasing the support size m reduces centroid-estimation error. Fig. 5 shows both trends across CelebA attributes: empirical error decreases with recoverability, and the bound tightens as m grows. The bound remains above the observed error throughout the evaluated configurations and becomes more informative for well-recovered tasks. App. B.1 reports the remaining encoders and Tab. 1 shows comparisons with earlier neural-collapse-based transfer bounds (Galanti et al., 2023; Luthra et al., 2025; Luthra et al., 2026). Multiple semantic factors. Thm. 3.5 predicts two related effects: well-recovered tasks should have nearly orthogonal semantic axes when cross-task correlations and residual uncertainty are small, and their joint centroids should approach the corresponding factorial hyperrectangle. Fig. 6(a–b) shows two illustrative triplets selected on the training split using fixed recoverability and near-orthogonality thresholds (see App. B.1); the displayed centroids are then measured on held-out test images. Panels (c–d) use every eligible triplet, with no additional recoverability or orthogonality screening. Across encoders, higher mean task recoverability is associated with both lower centroid RMSE and smaller task-axis overlap, consistent with the geometry predicted by the theorem. The spectral subspace recovers semantic energy. Cor. 4.2 identifies recoverability at the population optimum with posterior energy in the leading two-view spectral subspace. We test this empirically by comparing B^(t)(F) B^(t)(F), measured directly from the learned representation, with B^r⋆(t) B_r ^(t), reconstructed from the estimated paired-view spectral modes at the label-free rank r⋆r defined in App. B.1. Fig. 7 shows close agreement. For CelebA-trained W-MSE and VICReg, the retained basis spans nearly the full projector space, so near-exact agreement is expected. For ImageNet-pretrained encoders, the lower-dimensional spectral subspace still recovers most of the full-representation semantic energy, with median absolute discrepancies below 0.050.05 across models. The subspace is estimated without downstream labels. For I-JEPA, the paired-view operator is a surrogate analysis kernel rather than the masking distribution used in pretraining; more details can be found in App. A. 6 Discussion and Limitations Our results identify semantic recoverability as the key quantity linking same-instance SSL to downstream geometry and transfer. The task-specific captured posterior energy Br(t)B_r^(t) provides a unified account of directional neural collapse, probe–centroid alignment, multitask centroid structure, and few-shot performance. These results are derived primarily for population-optimal, centered and whitened representations. We also analyze a covariance-regularized surrogate, but extending the theory to finite-sample training and imperfect optimization remains an important direction for future work. Our main analysis also focuses on balanced binary tasks and conditionally i.i.d. two-view generation. Extending the framework to multiclass tasks and more general view dependencies is another natural direction. AI use statement In this work, we used generative AI tools (ChatGPT and Claude Code) to assist with developing and refining theoretical frameworks, formulating mathematical claims, discussing proof strategies, and providing feedback on experimental methodology. These tools also assisted with experimental design and implementation, including code generation and refinement, as well as research brainstorming, literature exploration, manuscript organization, drafting and editing, and LaTeX presentation. All AI-assisted theoretical arguments, mathematical derivations, experimental designs, and generated or modified code were reviewed by the authors for correctness and consistency. Relevant literature was independently examined, and all experimental results and their interpretation were evaluated by the authors. The authors take full responsibility for the final content of the paper, including its text, claims, code, results, and other artifacts produced with the assistance of generative AI. Ethics statement This work studies theoretical and empirical properties of representations learned through self-supervised learning. It does not involve new human-subject studies or the collection of new personal data. The experiments use existing datasets and pretrained models for research purposes. The work does not develop or evaluate a high-risk application. Reproducibility statement We state the assumptions and formal results in the main text and provide complete proofs in the appendix. The experimental sections and supplementary material document the datasets, pretrained representations, evaluation protocols, preprocessing, training procedures, and implementation details needed to reproduce our empirical results. We also specify the procedures used to estimate semantic recoverability, directional CDNV, spectral structure, and few-shot transfer performance. References Alon et al. (2024) N. Alon, D. Avdiukhin, D. Elboim, O. Fischer, and G. Yaroslavtsev Optimal sample complexity of contrastive learning. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1.1. Andrew et al. (2013) G. Andrew, R. Arora, J. Bilmes, and K. Livescu Deep canonical correlation analysis. In Proceedings of the 30th International Conference on Machine Learning, S. Dasgupta and D. McAllester (Eds.), Proceedings of Machine Learning Research, Vol. 28, Atlanta, Georgia, USA, p. 1247–1255. External Links: Link Cited by: §1.1. Ash et al. (2022) J. Ash, S. Goel, A. Krishnamurthy, and D. Misra Investigating the role of negatives in contrastive representation learning. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, G. Camps-Valls, F. J. R. Ruiz, and I. Valera (Eds.), Proceedings of Machine Learning Research, Vol. 151, p. 7187–7209. External Links: Link Cited by: §1.1. Asoodeh et al. (2015) S. Asoodeh, F. Alajaji, and T. Linder On maximal correlation, mutual information and data privacy. In 2015 IEEE 14th Canadian Workshop on Information Theory (CWIT), p. 27–31. External Links: Document Cited by: §1.1. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. G. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, p. 15619–15629. External Links: Link Cited by: Appendix A, §4.1, §5.1. Awasthi et al. (2022) P. Awasthi, N. Dikkala, and P. Kamath Do more negative samples necessarily hurt in contrastive learning?. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 1101–1116. External Links: Link Cited by: §1.1. Bachman et al. (2019) P. Bachman, R. D. Hjelm, and W. Buchwalter Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, Vol. 32, p. 15509–15519. External Links: Link Cited by: §1.1. Baevski et al. (2022) A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli Data2vec: a general framework for self-supervised learning in speech, vision and language. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 1298–1312. External Links: Link Cited by: §1. Baevski et al. (2020) A. Baevski, H. Zhou, A. Mohamed, and M. Auli Wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. External Links: ISBN 9781713829546 Cited by: §1. Balestriero et al. (2023) R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum A cookbook of self-supervised learning. External Links: 2304.12210, Link Cited by: §1. Balestriero and LeCun (2022) R. Balestriero and Y. LeCun Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §1.1, §1, §4.1. Bardes et al. (2022) A. Bardes, J. Ponce, and Y. LeCun VICReg: variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A, Appendix A, §1.1, §4.1, §4.1, §5.1. Ben-Shaul et al. (2023) I. Ben-Shaul, R. Shwartz-Ziv, T. Galanti, S. Dekel, and Y. LeCun Reverse engineering self-supervised learning. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1.1. Breiman and Friedman (1985) L. Breiman and J. H. Friedman Estimating optimal transformations for multiple regression and correlation. Journal of the American Statistical Association 80 (391), p. 580–598. External Links: Document, Link Cited by: §1.1. Burgess and Kim (2018) C. Burgess and H. Kim 3D shapes dataset. Note: https://github.com/deepmind/3d-shapes/ Cited by: §5.1. Caron et al. (2021) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9650–9660. Cited by: §1.1, §4.1. Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, H. D. I and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, p. 1597–1607. External Links: Link Cited by: §1.1, §1, §4.1. Chen et al. (2021) T. Chen, C. Luo, and L. Li Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems 34, p. 11834–11845. Cited by: §1.1, §1.1. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, p. 248–255. External Links: Document Cited by: §5.1. Ermolov et al. (2021) A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe Whitening for self-supervised representation learning. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 3015–3024. External Links: Link Cited by: Appendix A, Appendix A, §1.1, §1, §4.1, §5.1. Fukumizu et al. (2007) K. Fukumizu, F. R. Bach, and A. Gretton Statistical consistency of kernel canonical correlation analysis. Journal of Machine Learning Research 8 (14), p. 361–383. External Links: Link Cited by: §1.1. Galanti et al. (2023) T. Galanti, L. Galanti, and I. Ben-Shaul Comparative generalization bounds for deep neural networks. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §B.1, Table 1, §5.2. Galanti et al. (2022a) T. Galanti, A. György, and M. Hutter Generalization bounds for few-shot transfer learning with pretrained classifiers. External Links: 2212.12532, Link Cited by: §1.1. Galanti et al. (2022b) T. Galanti, A. György, and M. Hutter Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, External Links: Link Cited by: §1.1. Galanti et al. (2022c) T. Galanti, A. György, and M. Hutter On the role of neural collapse in transfer learning. In International Conference on Learning Representations, External Links: Link Cited by: §1.1. Gao et al. (2021) T. Gao, X. Yao, and D. Chen SimCSE: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 6894–6910. External Links: Link, Document Cited by: §1. Garrido et al. (2023) Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. LeCun On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1.1, §1, §4.1. Goldblum et al. (2020) M. Goldblum, S. Reich, L. Fowl, R. Ni, V. Cherepanova, and T. Goldstein Unraveling meta-learning: understanding feature representations for few-shot tasks. In Proceedings of the 37th International Conference on Machine Learning, H. D. I and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, p. 3607–3616. External Links: Link Cited by: §1.1. Goyal et al. (2017) P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He Accurate, large minibatch SGD: training ImageNet in 1 hour. External Links: 1706.02677, Link Cited by: Appendix A. Grill et al. (2020) J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko Bootstrap your own latent - a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 21271–21284. External Links: Link Cited by: §4.1, §4.1. Gui et al. (2023) Y. Gui, C. Ma, and Y. Zhong Unraveling projection heads in contrastive learning: insights from expansion and shrinkage. External Links: 2306.03335, Link Cited by: §1.1. Gupta et al. (2022) K. Gupta, T. Ajanthan, A. van den Hengel, and S. Gould Understanding and improving the role of projection head in self-supervised learning. In NeurIPS 2022 Workshop: Self-Supervised Learning – Theory and Practice, External Links: 2212.11491, Link Cited by: §1.1. Han et al. (2022) X.Y. Han, V. Papyan, and D. L. Donoho Neural collapse under MSE loss: proximity to and dynamics on the central path. In International Conference on Learning Representations, External Links: Link Cited by: §1.1. Hannan (1961) E. J. Hannan The general theory of canonical correlation and its relation to functional analysis. Journal of the Australian Mathematical Society 2 (2), p. 229–242. External Links: Document Cited by: §1.1. HaoChen and Ma (2023) J. Z. HaoChen and T. Ma A theoretical study of inductive biases in contrastive learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1.1. HaoChen et al. (2021) J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 5000–5011. External Links: Link Cited by: §1.1, §1.1. Hardoon et al. (2004) D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor Canonical correlation analysis: an overview with application to learning methods. Neural Computation 16 (12), p. 2639–2664. External Links: Document Cited by: §1.1. He et al. (2022) K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16000–16009. Cited by: §1. He et al. (2020) K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9729–9738. External Links: Link Cited by: §1. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. External Links: Link Cited by: Appendix A, §5.1. Hotelling (1936) H. Hotelling Relations between two sets of variates. Biometrika 28 (3/4), p. 321–377. External Links: Document, Link Cited by: §1.1. Hsu et al. (2021) W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 29, p. 3451–3460. External Links: ISSN 2329-9290, Link, Document Cited by: §1. Jia et al. (2021) C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 4904–4916. External Links: Link Cited by: §1. Lai and Fyfe (2000) P. L. Lai and C. Fyfe Kernel and nonlinear canonical correlation analysis. International Journal of Neural Systems 10 (5), p. 365–377. External Links: Document, Link Cited by: §1.1. Liu et al. (2015) Z. Liu, P. Luo, X. Wang, and X. Tang Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 3730–3738. External Links: Document, Link Cited by: Appendix A, §5.1. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Luthra et al. (2026) A. Luthra, Y. Salunkhe, and T. Galanti Directional neural collapse explains few-shot transfer in self-supervised learning. In International Conference on Machine Learning, External Links: 2603.03530, Link Cited by: §B.1, Table 1, §5.2. Luthra et al. (2025) A. Luthra, T. Yang, and T. Galanti Self-supervised contrastive learning is approximately supervised contrastive learning. In Advances in Neural Information Processing Systems, Vol. 38, p. 46671–46708. External Links: Document, Link Cited by: §B.1, Table 1, §1.1, §1, §1, §5.2. Matthey et al. (2017) L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner DSprites: disentanglement testing sprites dataset. Note: https://github.com/deepmind/dsprites-dataset/ Cited by: §5.1. McAllester and Stratos (2020) D. McAllester and K. Stratos Formal limitations on the measurement of mutual information. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, S. Chiappa and R. Calandra (Eds.), Proceedings of Machine Learning Research, Vol. 108, p. 875–884. External Links: Link Cited by: §1.1. Michaeli et al. (2016) T. Michaeli, W. Wang, and K. Livescu Nonparametric canonical correlation analysis. In Proceedings of The 33rd International Conference on Machine Learning, M. F. Balcan and K. Q. Weinberger (Eds.), Proceedings of Machine Learning Research, Vol. 48, New York, New York, USA, p. 1967–1976. External Links: Link Cited by: §1.1. Nozawa and Sato (2021) K. Nozawa and I. Sato Understanding negative samples in instance discriminative self-supervised representation learning. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 5784–5797. External Links: Link Cited by: §1.1. Oquab et al. (2024) M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research. External Links: Link Cited by: §1.1, §1. Papyan et al. (2020) V. Papyan, X. Y. Han, and D. L. Donoho Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40), p. 24652–24663. External Links: Document, Link, https://w.pnas.org/doi/pdf/10.1073/pnas.2015509117 Cited by: §1.1. Parulekar et al. (2023) A. Parulekar, L. Collins, K. Shanmugam, A. Mokhtari, and S. Shakkottai InfoNCE loss provably learns cluster-preserving representations. In Proceedings of Thirty Sixth Conference on Learning Theory, G. Neu and L. Rosasco (Eds.), Proceedings of Machine Learning Research, Vol. 195, p. 1914–1961. External Links: Link Cited by: §1.1. Qiu et al. (2023) Z. Qiu, Q. Hu, Z. Yuan, D. Zhou, L. Zhang, and T. Yang Not all semantics are created equal: contrastive self-supervised learning with automatic temperature individualization. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 28389–28421. External Links: Link Cited by: §1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. External Links: Link Cited by: §1. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 3982–3992. External Links: Link, Document Cited by: §1. Saunshi et al. (2022) N. Saunshi, J. Ash, S. Goel, D. Misra, C. Zhang, S. Arora, S. Kakade, and A. Krishnamurthy Understanding contrastive learning requires incorporating inductive biases. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 19250–19286. External Links: Link Cited by: §1.1. Saunshi et al. (2019) N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 5628–5637. External Links: Link Cited by: §1.1, §1.1. Schneider et al. (2019) S. Schneider, A. Baevski, R. Collobert, and M. Auli Wav2vec: unsupervised pre-training for speech recognition. In Interspeech 2019, p. 3465–3469. External Links: Document, Link Cited by: §1. Shen et al. (2022) K. Shen, R. M. Jones, A. Kumar, S. M. Xie, J. Z. Haochen, T. Ma, and P. Liang Connect, not collapse: explaining contrastive learning for unsupervised domain adaptation. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 19847–19878. External Links: Link Cited by: §1.1. Tao et al. (2022) C. Tao, H. Wang, X. Zhu, J. Dong, S. Song, G. Huang, and J. Dai Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14431–14440. Cited by: §4.1, §4.1. Tian et al. (2020) Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola What makes for good views for contrastive learning?. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 6827–6839. External Links: Link Cited by: §1.1. Tian (2023) Y. Tian Understanding the role of nonlinearity in training dynamics of contrastive learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1.1. Tosh et al. (2021) C. Tosh, A. Krishnamurthy, and D. Hsu Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, p. 1179–1206. Cited by: §1.1. Tschannen et al. (2020) M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic On mutual information maximization for representation learning. In International Conference on Learning Representations, External Links: Link Cited by: §1.1. Tschannen et al. (2025) M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. External Links: 2502.14786, Link Cited by: §1. Wang and Liu (2021) F. Wang and H. Liu Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2495–2504. Cited by: §1.1, §1.1. Wang and Isola (2020) T. Wang and P. Isola Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, p. 9929–9939. Cited by: §1.1, §1.1, §1.1, §1, §4.1. Wang et al. (2023) Z. Wang, Y. Luo, L. Zheng, Z. Huang, and M. Baktashmotlagh How far pre-trained models are from neural collapse on the target dataset informs their transferability. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 5526–5535. External Links: Document Cited by: §1.1. Wen and Li (2021) Z. Wen and Y. Li Toward understanding the feature learning process of self-supervised contrastive learning. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 11112–11122. External Links: Link Cited by: §1.1. Weng et al. (2025) X. Weng, J. An, X. Ma, B. Qi, J. Luo, X. Yang, J. S. Dong, and L. Huang Clustering properties of self-supervised learning. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 66597–66616. External Links: Link Cited by: §1.1. Weng et al. (2022) X. Weng, L. Huang, L. Zhao, R. M. Anwer, S. Khan, and F. Khan An investigation into whitening loss for self-supervised learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §1, §4.1. Zbontar et al. (2021) J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny Barlow twins: self-supervised learning via redundancy reduction. In International conference on machine learning, p. 12310–12320. Cited by: Appendix A, §1.1, §1, §4.1, §4.1, §5.1. Zhai et al. (2025) R. Zhai, K. Yang, B. Varıcı, C. Tsai, J. Z. Kolter, and P. K. Ravikumar Contextures: representations from contexts. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 74318–74347. External Links: Link Cited by: §1.1, §1. Zhai (2025) R. Zhai Contextures: the mechanism of representation learning. Ph.D. Thesis, Carnegie Mellon University. Note: PhD dissertation, CMU-CS-25-104 External Links: 2504.19792, Link Cited by: §1.1, §1. Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 11975–11986. Cited by: §1. Zimmermann et al. (2021) R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel Contrastive learning inverts the data generating process. In International Conference on Machine Learning, p. 12979–12990. Cited by: §1.1. Appendix A Implementation Details Datasets. CelebA (Liu et al., 2015) contains 202,599 RGB face images at resolution 178×218178× 218, covering 10,177 identities and 40 binary attributes. We use the standard identity-disjoint split of 162,770 training, 19,867 validation, and 19,962 test images. The dSprites dataset contains 737,280 binary 64×6464× 64 images generated from six latent factors. We retain two of the three shapes, the four most extreme scale levels, and the eight most extreme values of each position coordinate, leaving orientation unrestricted. This gives 20,480 images, split randomly into 18,432 training and 2,048 held-out test images. Binary downstream tasks are formed by thresholding selected generative factors. 3DShapes contains 480,000 RGB 64×6464× 64 images generated from six latent factors. We keep six of the ten object-hue levels and the four most extreme scale levels, giving 144,000 images. We use 129,600 images for training and 14,400 for held-out testing. As with dSprites, binary downstream tasks are obtained by thresholding selected latent factors; the factors used in each analysis are indicated in the corresponding figure. Evaluation protocol. The held-out test split is used for the saturation curves and the centroid-geometry evaluations. For the recoverability and directional-CDNV analyses, we construct disjoint subsets A and B from the training data. The subsets contain 81,385 images each for CelebA, 2,048 images each for dSprites, and 14,400 images each for 3DShapes. The paired-view spectral basis, whitening transform, and recoverability estimates are computed on A; directional CDNV is then measured independently on B. Evaluation preprocessing. All label-dependent quantities (recoverability, directional CDNV, NCC error, and linear-probe quantities) are measured on clean, single-view images. CelebA images are center-cropped to 160×160160× 160 and resized to 128×128128× 128 for the models trained from scratch. dSprites and 3DShapes are kept at their native 64×6464× 64 resolution and are bilinearly upsampled to the input resolution of externally pretrained backbones when needed. Inputs are normalized with ImageNet-1K channel statistics. On CelebA, augmented pairs are used only to estimate the paired-view spectral basis. On dSprites and 3DShapes, the same deterministic resize-and-normalize transform is used both for paired-view spectral estimation and for clean feature extraction. VICReg and W-MSE augmentations. We use the same view distribution for the CelebA-trained VICReg and W-MSE models. Each view is generated by a random resized crop with scale (0.2,1.0)(0.2,1.0) and aspect ratio (3/4,4/3)(3/4,4/3), resized to 128×128128× 128, followed by horizontal flipping with probability 0.50.5. We apply color jitter with brightness 0.40.4, contrast 0.40.4, saturation 0.20.2, and hue 0.050.05 with probability 0.80.8, grayscale conversion with probability 0.10.1, and Gaussian blur with probability 0.50.5 using a 7×77× 7 kernel and σ∼[0.1,2.0]σ [0.1,2.0]. We use a minimum crop scale of 0.20.2, rather than the ImageNet value 0.080.08, to preserve more facial structure. Unlike the original VICReg pipeline (Bardes et al., 2022), we do not use solarization and use the same blur probability for both views. Pretrained encoders. For ImageNet-1K-pretrained VICReg, Barlow Twins, and I-JEPA, CelebA images are resized to a shorter side of 256 pixels and center-cropped to 224×224224× 224. To estimate the paired-view spectral basis, we use an ImageNet-scale version of the augmentation pipeline above: random resized crops with scale (0.08,1.0)(0.08,1.0), color-jitter hue 0.10.1, grayscale probability 0.20.2, and a 23×2323× 23 Gaussian blur kernel. For I-JEPA, these augmented pairs define a surrogate two-view kernel for analysis; they are not the masking-based views used during I-JEPA pretraining. For dSprites and 3DShapes, externally pretrained encoders receive bilinearly upsampled 224×224224× 224 inputs. Training details. We train VICReg (Bardes et al., 2022) and W-MSE (Ermolov et al., 2021) from scratch on the CelebA training split for 1000 epochs. Both use AdamW (Loshchilov and Hutter, 2019) with β1=0.9 _1=0.9, β2=0.999 _2=0.999, ϵ=10−8ε=10^-8, and weight decay 10−610^-6. The learning rate is scaled linearly with global batch size, η=ηbaseB/256η= _baseB/256, with a 10-epoch linear warm-up (Goyal et al., 2017) followed by cosine decay (Loshchilov and Hutter, 2017). For VICReg, the projection head is Linear(2048 → 2048, bias=False) → BN → ReLU → Linear(2048 → 2048). The invariance, variance, and covariance coefficients are λ=25λ=25, μ=25μ=25, and ν=1ν=1. We use a global batch size of 1024 across two NVIDIA H100 GPUs and ηbase=1.5×10−4 _base=1.5× 10^-4. For W-MSE, the projection head is Linear(2048 → 2048, bias=False) → BN → ReLU → Linear(2048 → 128). Following (Ermolov et al., 2021), embeddings from each view are split into sub-batches of 256 and whitened independently using the inverse Cholesky factor of the empirical covariance, with no covariance regularization (ϵ=0ε=0). The loss is normalized mean-squared error between corresponding whitened embeddings. The two views are processed in one concatenated forward pass. We use a global batch size of 512 on one NVIDIA H100 and ηbase=10−3 _base=10^-3. Architectures. Both CelebA-trained models use a standard ResNet-50 (He et al., 2016). At 128×128128× 128 resolution, the final residual stage produces a 4×44× 4 feature map, which is globally averaged to a 2048-dimensional representation. Unless stated otherwise, representation analyses use these pooled backbone features rather than projection-head outputs. We also evaluate public ImageNet-1K-pretrained VICReg and Barlow Twins (Zbontar et al., 2021) ResNet-50 encoders and an I-JEPA (Assran et al., 2023) ViT-H/14 encoder, without fine-tuning. The ResNet-50 encoders produce 2048-dimensional pooled features. I-JEPA has 32 transformer layers, 16 attention heads, hidden dimension 1280, and MLP dimension 5120. At 224×224224× 224, it produces 256 patch tokens and no classification token; we average the final-layer-normalized patch tokens to obtain a 1280-dimensional representation. Appendix B Additional Results B.1 Additional CelebA Results This section reports the remaining CelebA results using the same protocol as Sec. 5. Semantic recoverability. Fig. 8 (a) shows the rank-dependent recoverability curves for ImageNet-1K-pretrained VICReg. It reproduces the main-text pattern: recoverability varies substantially across attributes, the strongest tasks saturate at lower rank, and the learned representations remain above their random-initialization baselines. (a) Captured posterior energy (b) Predicted vs. observed V~ V (c) Spectral identification Figure 8: Additional recoverability results for VICReg-IM1K on CelebA. (a) Captured posterior energy across representation ranks, using the same attribute groups and random-initialization reference as Fig. 3. (b) Directional CDNV predicted from recoverability estimated on split A versus directional CDNV measured independently on held-out split B; the dashed line is y=xy=x. (c) Recoverability measured directly from the learned representation versus its reconstruction from the paired-view spectral subspace; the dashed line denotes perfect agreement. Directional geometry. Fig. 8 (b) gives the held-out test of Prop. 3.1. Predictions are computed from split A and observed directional CDNV from split B. The agreement with y=xy=x remains tight across attributes and ranks. Few-shot transfer. Fig. 9 reports the remaining few-shot results. Across models, ranks, and shot counts, empirical NCC error decreases with recoverability and the bound tracks the same rank–shot tradeoff seen in the main text. (a) VICReg-CelebA (b) VICReg-IM1K (c) Barlow-IM1K Figure 9: Captured posterior energy and few-shot NCC error. Results for VICReg-CelebA, VICReg-IM1K, and Barlow Twins-IM1K across ranks r and support sizes m. Dashed curves are the upper bound from Thm. 3.4. Multitask geometry and spectral identification. Each encoder is evaluated at a label-free rank r⋆r , defined as the number of paired-view spectral directions whose eigenvalue is at least 10−310^-3 times the largest eigenvalue. This gives r⋆=946r =946 for W-MSE, 2046 for VICReg-CelebA, 763 for VICReg-IM1K, 806 for Barlow-IM1K, and 263 for I-JEPA-IM1K. For each attribute triplet, the task-specific capture values Br⋆(t)B_r ^(t) and the task axes are estimated on the training split, weighting the eight joint-label cells equally; the centroids are then measured on the held-out test split. We write B¯r⋆:=13∑t=13Br⋆(t) B_r := 13 _t=1^3B_r ^(t) for the triplet mean recoverability used in Fig. 6(c–d). Normalized centroid RMSE is the root-mean-square distance between observed and predicted centroids divided by the root-mean-square radius of the predicted box. Fig. 6(c–d) use all triplets drawn from the 26 attributes for which each binary class contains at least 10% of the data and every joint-label cell contains at least 1000 training and 100 test examples. This yields 1646 triplets per encoder. No recoverability or orthogonality screening is applied to these quantitative panels. The curves are means in bins of width 0.05 containing at least 12 triplets; shaded regions show interquartile ranges. The illustrative cubes in Figs. 6(a–b) and 10(a–b) are selected on the training split only from triplets satisfying mintBr⋆(t)≥0.10 _tB_r ^(t)≥ 0.10 and pairwise |cos(ui,uj)|≤0.12| (u_i,u_j)|≤ 0.12. Fig. 10 adds held-out centroid visualizations for VICReg-IM1K and Barlow Twins-IM1K, and Fig. 8 (c) and Fig. 7 (c) reports the corresponding spectral-identification plots for VICReg-IM1K and Barlow Twins-IM1K respectively. The latter compare recoverability measured in the full feature space with recoverability reconstructed from the label-free paired-view spectral subspace. (a) VICReg-IM1K (b) Barlow-IM1K Figure 10: Multitask centroid geometry on CelebA. Held-out joint centroids and the hyperrectangles predicted by Thm. 3.5 for ImageNet-1K-pretrained VICReg and Barlow Twins. Experiment setup is same as Fig. 6 Comparison with prior transfer bounds. Table 1 compares Thm. 3.4 with earlier neural-collapse-based transfer bounds (Galanti et al., 2023; Luthra et al., 2025; Luthra et al., 2026). On W-MSE trained on CelebA, the earlier bounds remain above 0.50.5 for all 40 attributes and all five shot counts shown. Our bound is non-vacuous in several rank–shot configurations and falls below 0.50.5 already at m=1m=1. The minimizing rank increases as more support examples become available, matching the rank–shot tradeoff in Thm. 3.4. Table 1: Comparison of few-shot transfer bounds on CelebA. Bounds on NCC classification error for W-MSE trained on CelebA (“Male/Female” attribute) . Our bound is minimized over the evaluated ranks separately for each shot count m. Prior bounds use ℓ2 _2-normalized full-dimensional features, whereas ours uses whitened top-r features. Values below 0.50.5 are non-vacuous relative to chance classification. Bound = m=1 = m=5 = m=10 = m=100 = m=500 (Galanti et al., 2023) 2923.1 709.9 433.2 184.3 162.1 (Luthra et al., 2025) 138.1 32.1 20.6 6.70 3.53 (Luthra et al., 2026) 1544.4 282.5 126.7 10.27 2.05 Our bound 0.490 0.253 0.223 0.190 0.161 B.2 Results on Synthetic Datasets We use dSprites and 3DShapes to check that the same recoverability–geometry relationships are visible in controlled factorized data. Figs. 11–13 report rank-dependent recoverability, the directional-CDNV identity, and held-out multitask centroid geometry. The displayed encoders are ImageNet-1K-pretrained VICReg, I-JEPA, and Barlow Twins. Across both datasets, the qualitative behavior matches the CelebA results. (a) VICReg (dSprites) (b) I-JEPA (dSprites) (c) Barlow Twins (dSprites) (d) VICReg (3DShapes) (e) I-JEPA (3DShapes) (f) Barlow Twins (3DShapes) Figure 11: Semantic recoverability on dSprites and 3DShapes. Task-specific captured posterior energy Br(t)B_r^(t) across representation ranks for ImageNet-1K-pretrained VICReg, I-JEPA, and Barlow Twins. (a) VICReg (dSprites) (b) I-JEPA (dSprites) (c) Barlow Twins (dSprites) (d) VICReg (3DShapes) (e) I-JEPA (3DShapes) (f) Barlow Twins (3DShapes) Figure 12: Predicted versus observed directional CDNV on synthetic datasets. Results for VICReg, I-JEPA, and Barlow Twins on dSprites and 3DShapes. Each point is one attribute at one rank r; the prediction is computed on split A and the observed directional CDNV on held-out split B. Dashed lines show the theoretical prediction y=xy=x. (a) VICReg (dSprites) (b) I-JEPA (dSprites) (c) Barlow Twins (dSprites) (d) VICReg (3DShapes) (e) I-JEPA (3DShapes) (f) Barlow Twins (3DShapes) Figure 13: Multitask centroid geometry on synthetic datasets. Held-out joint centroids compared with the hyperrectangles predicted by Thm. 3.5. Appendix C Covariance-Regularized SSL Objectives In the main text we studied the population surrogate (2), which enforces strict whitening [F(X)F(X)⊤]=IrE[F(X)F(X) ]=I_r. This yields an exact strict-whitening theory in which the main directional geometry is governed by the single semantic capture parameter Br=‖Πrη‖L2(PX)2B_r=\| _rη\|_L^2(P_X)^2. Modern SSL objectives typically do not enforce exact whitening, but instead penalize covariance deviations through variance/covariance regularizers, as in Barlow Twins and VICReg. In this appendix we show that a natural covariance-regularized surrogate preserves the same spectral subspace identified by the two-view operator T in Prop. 4.1, but the resulting geometry is no longer governed by a single scalar. Instead, three explicit quantities (Bγ,Cγ,Sγ)(B_γ,C_γ,S_γ) appear, with the strict-whitening formulas recovered as γ→∞γ→∞. C.1 Population covariance-regularized objective We consider the following population surrogate: maxF:→ℝr[⟨F(X(1)),F(X(2))⟩]−γ2‖[F(X)F(X)⊤]−Ir‖F2s.t.[F(X)]=0, _F:X ^r\ E\! [ F(X^(1)),F(X^(2)) ]- γ2 \|E[F(X)F(X) ]-I_r \|_F^2 .t. [F(X)]=0, (3) where γ>0γ>0 controls the strength of the covariance penalty. As γ→∞γ→∞, the penalty enforces [F(X)F(X)⊤]→IrE[F(X)F(X) ]→ I_r, recovering the strict whitening setting. The centering constraint [F(X)]=0E[F(X)]=0 models the fact that practical implementations explicitly center features and, in a population analysis, can be imposed without loss of generality by working with centered features F¯(X):=F(X)−[F(X)] F(X):=F(X)-E[F(X)] and rewriting the objective in terms of F¯ F. C.2 Operator form and canonical modes Recall the two-view conditional expectation operator (Tf)(x):=[f(X(2))∣X(1)=x](Tf)(x):=E[f(X^(2)) X^(1)=x], T:L2(PX)→L2(PX)T:L^2(P_X)→ L^2(P_X). For any f,g∈L2(PX)f,g∈ L^2(P_X), we have [f(X(1))g(X(2))]=⟨f,Tg⟩L2(PX).E [f(X^(1))g(X^(2)) ]= f,Tg _L^2(P_X). (4) In particular, [f(X(1))f(X(2))]=⟨f,Tf⟩L2(PX).E [f(X^(1))f(X^(2)) ]= f,Tf _L^2(P_X). Let (λj,ψj)j≥1( _j, _j)_j≥ 1 be the orthonormal eigenbasis of T on L02(PX)L_0^2(P_X) from Prop. 4.1, ordered as λ1≥λ2≥⋯≥0. _1≥ _2≥·s≥ 0. Thus Tψj=λjψj,⟨ψi,ψj⟩L2(PX)=δij,[ψj(X)]=0.T _j= _j _j, _i, _j _L^2(P_X)= _ij, [ _j(X)]=0. These are exactly the canonical observable modes identified in Prop. 4.1. This appendix keeps the same operator and asks how covariance regularization rescales its top-r T-eigenspace. C.3 Structure of the population optimum We show that covariance regularization preserves the T-selected spectral subspace and only introduces mode-dependent scalings within that subspace. Proposition C.1 (Structure of the regularized optimum). Assume λr>0 _r>0 and λr>λr+1 _r> _r+1. Then any population maximizer Fγ⋆F _γ of (3) can be written as Fγ⋆(x)=R(s1ψ1(x),…,srψr(x)),F _γ(x)=R\, (s_1 _1(x),…,s_r _r(x) ), (5) where R∈ℝr×rR ^r× r is orthogonal and s1,…,sr≥0s_1,…,s_r≥ 0 are scalars. Moreover, for the quadratic penalty in (3), the optimal scalings satisfy sj2=1+λjγ,j=1,…,r.s_j^2=1+ _jγ, j=1,…,r. Prop. C.1 implies that the regularized optimum can be reduced to the whitened setting by an explicit population rewhitening step, and therefore inherits the same multitask geometric structure established in the main text. Let Fγ⋆F _γ be a population maximizer of (3), and define Σγ:=Cov(Fγ⋆(X)),F~γ(x):=Σγ−1/2Fγ⋆(x). _γ:=Cov(F _γ(X)), F_γ(x):= _γ^-1/2F _γ(x). For the maximizer in Prop. C.1, we have sj2=1+λj/γ>0s_j^2=1+ _j/γ>0, hence Σγ≻0 _γ 0 and Σγ−1/2 _γ^-1/2 is well-defined. Then F~γ F_γ is feasible for the strictly whitened objective (2) and is population-optimal. Indeed, by Prop. C.1 we may write Fγ⋆(x)=Rdiag(s1,…,sr)(ψ1(x),…,ψr(x)),F _γ(x)=R\,diag(s_1,…,s_r)\, ( _1(x),…, _r(x) ), so Σγ=Rdiag(s12,…,sr2)R⊤,Σγ−1/2=Rdiag(s1−1,…,sr−1)R⊤, _γ=R\,diag(s_1^2,…,s_r^2)\,R , _γ^-1/2=R\,diag(s_1^-1,…,s_r^-1)\,R , and therefore F~γ(x)=R(ψ1(x),…,ψr(x)). F_γ(x)=R\, ( _1(x),…, _r(x) ). By Prop. 4.1, any representation of the form R(ψ1,…,ψr)R( _1,…, _r) is feasible for (2) and achieves the optimal value ∑j=1rλj _j=1^r _j, hence F~γ F_γ is a whitened population optimum. Consequently, all main-text results that rely only on the whitened-optimum structure, including the multitask near-orthogonality and factorial-centroid geometry in Thm. 3.5, apply verbatim to F~γ F_γ. In the original feature coordinates of Fγ⋆F _γ, this same structure is obtained by applying the linear map Σγ1/2 _γ^1/2, so the hyperrectangle in rewhitened probe coordinates becomes a parallelepiped in the raw Fγ⋆F _γ coordinates. This highlights that covariance regularization preserves the operator-selected semantic organization, but can distort Euclidean geometry unless one explicitly rewhitens features, or equivalently uses a Mahalanobis nearest-centroid rule with metric Σγ−1 _γ^-1. C.4 How the CDNV formulas change Let Y=y(C)∈±1Y=y(C)∈\± 1\ be balanced and define the posterior score η(x)=[Y∣X=x].η(x)=E[Y X=x]. Write the posterior directly in the T-eigenbasis: η=∑j≥1βjψj,βj:=⟨η,ψj⟩L2(PX).η= _j≥ 1 _j _j, _j:= η, _j _L^2(P_X). Then, by Cor. 4.2, Πrη=∑j=1rβjψj,Br=‖Πrη‖L2(PX)2=∑j=1rβj2. _rη= _j=1^r _j _j, B_r=\| _rη\|_L^2(P_X)^2= _j=1^r _j^2. Define the scalars Bγ B_γ :=∑j=1rβj2sj2, := _j=1^r _j^2s_j^2, Cγ C_γ :=∑j=1rβj2sj4, := _j=1^r _j^2s_j^4, Sγ S_γ :=∑j=1rsj2. := _j=1^rs_j^2. (6) Proposition C.2. Let Fγ⋆F _γ be any population maximizer of (3), written as in (5). Let μ± _± and Σ± _± be the class means and class covariances defined as in the main text. Then μ−=−μ+ _-=- _+ and Δ=μ+−μ−=2μ+ = _+- _-=2 _+. Moreover, when Bγ>0B_γ>0, the CDNV and directional CDNV defined in (1) satisfy V~Fγ⋆=12(CγBγ2−1),VFγ⋆=12(SγBγ−1). V_F _γ= 12 ( C_γB_γ^2-1 ), V_F _γ= 12 ( S_γB_γ-1 ). (7) When Bγ=0B_γ=0, we set both quantities to +∞+∞ by the same convention as in the main text. For Bγ>0B_γ>0, Prop. C.2 can be plugged directly into a general CDNV-based few-shot guarantee. Namely, with F=Fγ⋆F=F _γ, substituting (7) yields the explicit γ-dependent bound errmNCC(Fγ⋆)≤4(CγBγ2−1)+8m12(SγBγ−1)+(8m+4m)12(SγBγ−1).err^NCC_m(F _γ)≤ 4 ( C_γB_γ^2-1 )+ 8 m 12 ( S_γB_γ-1 )+ ( 8 m+ 4m ) 12 ( S_γB_γ-1 ). (8) In particular, this bound depends on γ only through the three scalars Bγ,Cγ,SγB_γ,C_γ,S_γ defined in (6). Thus γ controls a concrete tradeoff in (8). The leading directional term depends on Cγ/Bγ2C_γ/B_γ^2, while the finite-m terms depend on Sγ/BγS_γ/B_γ. Under the quadratic penalty sj2=1+λj/γs_j^2=1+ _j/γ, decreasing γ inflates all selected modes, with larger inflation on more stable modes. This increases total variance through SγS_γ, but it also increases signal strength along the task-relevant direction through BγB_γ and CγC_γ. The resulting few-shot behavior is therefore governed by how the posterior coefficients βj _j overlap with the modes that are most amplified by the regularizer. In the strict-whitening limit sj→1s_j→ 1, we recover Bγ→Br,Cγ→Br,Sγ→r,B_γ→ B_r, C_γ→ B_r, S_γ→ r, and therefore V~Fγ⋆→1−Br2Br,VFγ⋆→r−Br2Br, V_F _γ→ 1-B_r2B_r, V_F _γ→ r-B_r2B_r, exactly as in Prop. 3.1. Appendix D Prediction-Based SSL Objectives as Reduced-Rank Regression This appendix gives a simple population-level connection between prediction-based SSL objectives, such as BYOL, SimSiam, and JEPA, and classical reduced-rank regression / partial least squares. We do not aim to model algorithmic details such as stop-gradient or EMA teachers. Instead, we study a natural whitened population surrogate and show that its optimizer selects the same T-spectral subspace as (2). As a result, the main-text conclusions that depend only on the selected subspace, and therefore on the induced capture value BrB_r, carry over directly to this prediction-based setting. D.1 A stylized whitened prediction surrogate Let F:→ℝrF:X ^r be a shared representation used on both views, and let W∈ℝr×rW ^r× r be a linear predictor. Consider the population objective minF:→ℝrW∈ℝr×r[∥F(X(2))−WF(X(1))∥22]s.t.[F(X)]=0,[F(X)F(X)⊤]=Ir, _ subarraycF:X ^r\\ W ^r× r subarray\ E [ \|F(X^(2))-WF(X^(1)) \|_2^2 ] .t. [F(X)]=0,\ \ E[F(X)F(X) ]=I_r, (9) where (X(1),X(2))(X^(1),X^(2)) is a same-instance pair with conditionally i.i.d. views as in Sec. 2.1. Define C12(F):=[F(X(1))F(X(2))⊤],C21(F):=C12(F)⊤.C_12(F):=E [F(X^(1))F(X^(2)) ], C_21(F):=C_12(F) . (10) Proposition D.1 (R reduction to a spectral objective). Fix any feasible F in (9). The unique minimizer over W is W⋆(F)=C21(F)=C12(F)⊤,W (F)=C_21(F)=C_12(F) , and the resulting minimum equals minW[‖F(X(2))−WF(X(1))‖22]=r−‖C21(F)‖F2. _W\ E [ \|F(X^(2))-WF(X^(1)) \|_2^2 ]=r-\|C_21(F)\|_F^2. (11) Hence minimizing (9) is equivalent to maximizing ‖C21(F)‖F2\|C_21(F)\|_F^2 over feasible F. Proof. Using [F(X)F(X)⊤]=IrE[F(X)F(X) ]=I_r, ‖F(X(2))−WF(X(1))‖22=r−2Tr(WC12(F))+‖W‖F2=r+‖W−C21(F)‖F2−‖C21(F)‖F2.E\|F(X^(2))-WF(X^(1))\|_2^2=r-2Tr(WC_12(F))+\|W\|_F^2=r+\|W-C_21(F)\|_F^2-\|C_21(F)\|_F^2. Thus the unique minimizer is W⋆(F)=C21(F)=C12(F)⊤W (F)=C_21(F)=C_12(F) , and the minimum is r−‖C21(F)‖F2r-\|C_21(F)\|_F^2. ∎ D.2 Selected subspace and relationship to the alignment objective For scalar f,g∈L2(PX)f,g∈ L^2(P_X) with [f(X)]=[g(X)]=0E[f(X)]=E[g(X)]=0, the same-instance model gives [f(X(2))g(X(1))]=⟨f,Tg⟩L2(PX).E[f(X^(2))g(X^(1))]= f,Tg _L^2(P_X). Thus, if F=(f1,…,fr)F=(f_1,…,f_r) is feasible and the coordinates fk\f_k\ are orthonormal in L2(PX)L^2(P_X), then (C21(F))ij=[fi(X(2))fj(X(1))]=⟨fi,Tfj⟩L2(PX). (C_21(F) )_ij=E[f_i(X^(2))f_j(X^(1))]= f_i,Tf_j _L^2(P_X). (12) Consequently, ‖C21(F)‖F2\|C_21(F)\|_F^2 is the squared Hilbert–Schmidt norm of the compression PUTPUP_UTP_U, where U=spanf1,…,fr⊂L02(PX)U=span\f_1,…,f_r\⊂ L_0^2(P_X). Let (λj,ψj)j≥1( _j, _j)_j≥ 1 be the orthonormal eigenbasis of T from Prop. 4.1, with λ1≥λ2≥⋯≥0 _1≥ _2≥·s≥ 0. Proposition D.2 (Prediction objective selects the top-r operator subspace). Assume λr>0 _r>0 and λr>λr+1 _r> _r+1. Then the maximum of ‖C21(F)‖F2\|C_21(F)\|_F^2 over feasible F equals ∑j=1rλj2 _j=1^r _j^2, and any maximizer F⋆F has coordinates spanning spanψ1,…,ψrspan\ _1,…, _r\. In particular, there exists an orthogonal R∈ℝr×rR ^r× r such that F⋆(x)=R(ψ1(x),…,ψr(x)).F (x)=R\,( _1(x),…, _r(x)). (13) Proof. Let U=spanf1,…,frU=span\f_1,…,f_r\ and let PUP_U be the orthogonal projector onto U. In the orthonormal basis fj\f_j\, C21(F)C_21(F) is the matrix of PUTPUP_UTP_U, and hence ‖C21(F)‖F2=‖PUTPU‖HS2≤‖TPU‖HS2=Tr(PUT2)≤∑j=1rλj2.\|C_21(F)\|_F^2=\|P_UTP_U\|_HS^2≤\|TP_U\|_HS^2=Tr(P_UT^2)≤ _j=1^r _j^2. The last inequality is Ky Fan’s variational principle applied to the positive semidefinite operator T2T^2. Equality is attained for U=spanψ1,…,ψrU=span\ _1,…, _r\, and the eigengap λr>λr+1 _r> _r+1 makes this maximizing subspace unique. ∎ D.3 Consequences for recoverability, CDNV, and downstream guarantees Prop. D.2 shows that the whitened prediction surrogate (9) selects the same r-dimensional T-eigenspace as (2). In particular, any maximizer F⋆F of (9) has coordinates spanning spanψ1,…,ψrspan\ _1,…, _r\, so F⋆F is feasible for (2). Since the strictly whitened alignment objective (2) depends only on the chosen r-dimensional subspace and is maximized by any orthonormal basis of the top-r eigenspace, as shown in Prop. 4.1, F⋆F is also population-optimal for (2) and achieves the optimal value ∑j=1rλj _j=1^r _j. Therefore, all main-text results that apply to whitened population optima of (2) apply directly to F⋆F . In particular, for any downstream binary label Y=y(C)Y=y(C) with posterior score η(x)=[Y∣X=x]η(x)=E[Y X=x], writing η=∑j≥1βjψj,η= _j≥ 1 _j _j, the capture value remains Br=∑j=1rβj2,B_r= _j=1^r _j^2, and the closed-form directional CDNV and CDNV laws for whitened optima from Prop. 3.1 hold unchanged for F⋆F . Consequently, the same few-shot NCC guarantees from Thm. 3.4 and the same multitask geometry conclusions from Thm. 3.5 follow immediately for this stylized prediction-based surrogate. Appendix E Proofs E.1 Proofs of the Results in the Main Text See 3.1 Proof. Let μs:=[F(X)∣Y=s] _s:=E[F(X) Y=s] and Σs:=Cov(F(X)∣Y=s) _s:=Cov(F(X) Y=s) for s∈±1s∈\± 1\, and set Δ:=μ+−μ− := _+- _- with u:=Δ/‖Δ‖2u:= /\| \|_2 when Δ≠0 ≠ 0. Balancedness of Y and [F(X)]=0E[F(X)]=0 give 12μ++12μ−=0 12 _++ 12 _-=0, so μ−=−μ+ _-=- _+ and Δ=2μ+ =2 _+. Moreover, [YF(X)]=12μ+−12μ−=μ+E[YF(X)]= 12 _+- 12 _-= _+, and by iterated expectation this also equals [η(X)F(X)]E[η(X)F(X)]. Write F=(f1,…,fr)F=(f_1,…,f_r). Centering and whitening imply that f1,…,fr\f_1,…,f_r\ is orthonormal in L2(PX)L^2(P_X), so F=spanf1,…,frS_F=span\f_1,…,f_r\ is an r-dimensional subspace and (μ+)i=[η(X)fi(X)]=⟨η,fi⟩( _+)_i=E[η(X)f_i(X)]= η,f_i . Hence ΠFη=∑i(μ+)ifi _S_Fη= _i( _+)_if_i, and by orthonormality ‖μ+‖22=‖ΠFη‖L22=B(F)\| _+\|_2^2=\| _S_Fη\|_L^2^2=B(F), giving ‖Δ‖22=4B(F)\| \|_2^2=4B(F). If B(F)=0B(F)=0 then Δ=0 =0 and both ratios are +∞+∞; assume B(F)>0B(F)>0 below. By the law of total covariance and whitening, Ir=12(Σ++Σ−)+Cov(μY)I_r= 12( _++ _-)+Cov( _Y), where μY∈μ+,μ− _Y∈\ _+, _-\ uniformly. Since μ−=−μ+ _-=- _+, Cov(μY)=μ+μ+⊤Cov( _Y)= _+ _+ , yielding Σ++Σ−=2(Ir−μ+μ+⊤). _++ _-=2(I_r- _+ _+ ). (14) Taking traces in (14) gives Tr(Σ+)+Tr(Σ−)=2(r−B(F))Tr( _+)+Tr( _-)=2(r-B(F)), so VF=(r−B(F))/(2B(F))V_F=(r-B(F))/(2B(F)) after dividing by ‖Δ‖22=4B(F)\| \|_2^2=4B(F). For the directional ratio, u=μ+/‖μ+‖2u= _+/\| _+\|_2 gives u⊤(Σ++Σ−)u=2(1−‖μ+‖22)=2(1−B(F))u ( _++ _-)u=2(1-\| _+\|_2^2)=2(1-B(F)) by (14), so V~F=(1−B(F))/(2B(F)) V_F=(1-B(F))/(2B(F)). ∎ See 3.3 Proof. Let Z:=F(X)Z:=F(X), so that [Z]=0E[Z]=0, [ZZ⊤]=IrE[Z ]=I_r, and [Y]=0E[Y]=0. Expanding the squared loss and using these identities gives ℛ(w,b):=[(Y−(w⊤Z+b))2]=1−2w⊤[YZ]+‖w‖22+b2,R(w,b):=E [(Y-(w Z+b))^2 ]=1-2w E[YZ]+\|w\|_2^2+b^2, a strictly convex quadratic in (w,b)(w,b) whose unique minimizer is w⋆=[YZ]w =E[YZ] and b⋆=0b =0. Let μs:=[Z∣Y=s] _s:=E[Z Y=s]. Balancedness gives [YZ]=12(μ+−μ−)=12ΔE[YZ]= 12( _+- _-)= 12 and [Z]=12μ++12μ−=0E[Z]= 12 _++ 12 _-=0, so μ−=−μ+ _-=- _+ and hence Δ=2μ+=2w⋆ =2 _+=2w . ∎ See 3.4 Proof. If B(F)=0B(F)=0, the stated bound is trivial since its right-hand side is at least 11. Assume henceforth that B(F)>0B(F)>0. Let Z:=F(X)Z:=F(X) and w:=[YZ]w:=E[YZ], so [Z]=0E[Z]=0, [ZZ⊤]=IrE[Z ]=I_r, and ‖w‖22=B(F)\|w\|_2^2=B(F). Balancedness and [Z]=0E[Z]=0 give μ±:=[Z∣Y=±1]=±w _±:=E[Z Y=± 1]=± w, hence Δ=2w =2w and ‖Δ‖22=4B(F)\| \|_2^2=4B(F). As in the proof of Prop. 3.1, the law of total covariance yields Σ++Σ−=2(Ir−ww⊤), _++ _-=2(I_r-w ), (15) where Σs:=Cov(Z∣Y=s) _s:=Cov(Z Y=s); taking traces gives Tr(Σ+)+Tr(Σ−)=2(r−B(F))Tr( _+)+Tr( _-)=2(r-B(F)), and contracting with u:=w/‖w‖2u:=w/\|w\|_2 gives u⊤(Σ++Σ−)u=2(1−B(F))u ( _++ _-)u=2(1-B(F)). NCC rule as a margin. Draw an m-shot support set with empirical class means μ^±:=1m∑i=1mZ±,i μ_±:= 1m _i=1^mZ_±,i, and set w^:=(μ^+−μ^−)/2 w:=( μ_+- μ_-)/2 and c:=(μ^++μ^−)/2c:=( μ_++ μ_-)/2. The NCC rule is gSmNCC(z)=sign(⟨w^,z−c⟩)g_S_m^NCC(z)= sign( w,z-c ), since its decision boundary is the perpendicular bisector of μ^± μ_±. Define the signed margin MSm:=2Y⟨w^,Z−c⟩M_S_m:=2Y w,Z-c . Regardless of how ties are broken, gSmNCC(Z)≠Y⊆MSm≤0\g_S_m^NCC(Z)≠ Y\ \M_S_m≤ 0\. Cantelli bound conditional on the support set. Since (Z,Y)(Z,Y) is independent of SmS_m with [YZ]=wE[YZ]=w, [Z]=0E[Z]=0, [ZZ⊤]=IrE[Z ]=I_r, [MSm∣Sm]=2⟨w^,w⟩,Var(MSm∣Sm)=4(‖w^‖22+⟨w^,c⟩2−⟨w^,w⟩2).E[M_S_m S_m]=2 w,w , (M_S_m S_m)=4 (\| w\|_2^2+ w,c ^2- w,w ^2 ). On the event G:=⟨w^,w⟩>0G:=\ w,w >0\, Cantelli’s inequality gives ℙ(MSm≤0∣Sm)≤1−f(Sm)P(M_S_m≤ 0 S_m)≤ 1-f(S_m), where f(Sm):=⟨w^,w⟩2‖w^‖22+⟨w^,c⟩2,w^≠0,0,w^=0.f(S_m):= cases w,w ^2\| w\|_2^2+ w,c ^2,& w≠ 0,\\[2.84526pt] 0,& w=0. cases Since 0≤f(Sm)≤10≤ f(S_m)≤ 1, bounding f(Sm)G≥f(Sm)−Gcf(S_m)1_G≥ f(S_m)-1_G^c and taking expectations gives errmNCC(F)≤1−[f(Sm)]+ℙ(Gc).err_m^NCC(F)≤ 1-E[f(S_m)]+P(G^c). (16) Lower bound on [f(Sm)]E[f(S_m)]. When w^≠0 w≠ 0, set p:=⟨w^,w⟩2/‖w^‖22=‖Pspan(w^)w‖22p:= w,w ^2/\| w\|_2^2=\|P_span( w)w\|_2^2; then f(Sm)=p/(1+⟨w^/‖w^‖2,c⟩2)f(S_m)=p/(1+ w/\| w\|_2,c ^2). The inequality x/(1+y)≥x−yx/(1+y)≥ x-y for 0≤x≤10≤ x≤ 1, y≥0y≥ 0, combined with p≤‖w‖22=B(F)≤1p≤\|w\|_2^2=B(F)≤ 1, yields f(Sm)≥p−‖c‖22f(S_m)≥ p-\|c\|_2^2. Bounding the distance from w to span(w^)span( w) by ‖w−w^‖2\|w- w\|_2 gives p≥B(F)−‖w^−w‖22p≥ B(F)-\| w-w\|_2^2; when w^=0 w=0 the inequality f(Sm)≥B(F)−‖w^−w‖22−‖c‖22f(S_m)≥ B(F)-\| w-w\|_2^2-\|c\|_2^2 still holds since then ‖w−w^‖22=B(F)\|w- w\|_2^2=B(F). Writing w^−w=(ε+−ε−)/2 w-w=( _+- _-)/2 and c=(ε++ε−)/2c=( _++ _-)/2 with ε±:=μ^±−μ± _±:= μ_±- _±, class independence gives ‖w^−w‖22=‖c‖22=Tr(Σ+)+Tr(Σ−)4m=r−B(F)2m,E\| w-w\|_2^2=E\|c\|_2^2= Tr( _+)+Tr( _-)4m= r-B(F)2m, so [f(Sm)]≥B(F)−(r−B(F))/mE[f(S_m)]≥ B(F)-(r-B(F))/m. Bound on ℙ(Gc)P(G^c). Since ⟨w^,w⟩=B(F)+⟨w^−w,w⟩ w,w =B(F)+ w-w,w , we have [⟨w^,w⟩]=B(F)E[ w,w ]=B(F), and Var(⟨w^,w⟩)=w⊤Cov(w^−w)w=B(F)4mu⊤(Σ++Σ−)u=B(F)(1−B(F))2mVar( w,w )=w Cov( w-w)w= B(F)4m\,u ( _++ _-)u= B(F)(1-B(F))2m using (15). Cantelli gives ℙ(Gc)≤(1−B(F))/(1−B(F)+2mB(F))P(G^c)≤(1-B(F))/(1-B(F)+2mB(F)). Combining this with (16) and the lower bound on [f(Sm)]E[f(S_m)] yields errmNCC(F)≤1−B(F)+(r−B(F))/m+(1−B(F))/(1−B(F)+2mB(F))err_m^NCC(F)≤ 1-B(F)+(r-B(F))/m+(1-B(F))/(1-B(F)+2mB(F)). ∎ See 3.5 Proof. Write F=(f1,…,fr)F=(f_1,…,f_r). Centering and whitening make f1,…,fr\f_1,…,f_r\ orthonormal in L2(PX)L^2(P_X), so F:=spanf1,…,frS_F:=span\f_1,…,f_r\ is r-dimensional with orthogonal projector ΠF _S_F. For each task t, let wt:=[YtF(X)]∈ℝrw_t:=E[Y_tF(X)] ^r with coordinates (wt)ℓ=[ηt(X)fℓ(X)]=⟨ηt,fℓ⟩(w_t)_ =E[ _t(X)f_ (X)]= _t,f_ , so ΠFηt=∑ℓ(wt)ℓfℓ _S_F _t= _ (w_t)_ f_ and hence ‖wt‖22=‖ΠFηt‖L22=B(t)(F).\|w_t\|_2^2=\| _S_F _t\|_L^2^2=B^(t)(F). (17) Write ηt⟂:=(I−ΠF)ηt _t :=(I- _S_F) _t, so ‖ηt⟂‖L22=‖ηt‖L22−B(t)(F)\| _t \|_L^2^2=\| _t\|_L^2^2-B^(t)(F). Near-orthogonality. By (17), ui⊤uj=⟨ΠFηi,ΠFηj⟩/B(i)(F)B(j)(F)u_i u_j= _S_F _i, _S_F _j / B^(i)(F)B^(j)(F). Since ⟨ΠFηi,ΠFηj⟩=⟨ηi,ηj⟩−⟨ηi⟂,ηj⟂⟩ _S_F _i, _S_F _j = _i, _j - _i , _j , Cauchy–Schwarz on the residual term gives |⟨ΠFηi,ΠFηj⟩|≤|⟨ηi,ηj⟩|+(‖ηi‖L22−B(i)(F))(‖ηj‖L22−B(j)(F)). | _S_F _i, _S_F _j |≤| _i, _j |+ (\| _i\|_L^2^2-B^(i)(F))(\| _j\|_L^2^2-B^(j)(F)). (18) To bound the first term, set ξt:=Yt−ηt(X) _t:=Y_t- _t(X); then [ξt∣X]=0E[ _t X]=0 and [ξt2]=εtE[ _t^2]= _t. Expanding YiYj=(ηi+ξi)(ηj+ξj)Y_iY_j=( _i+ _i)( _j+ _j) and using conditional mean zero on the cross terms, ρij=[ηi(X)ηj(X)]+[ξiξj] _ij=E[ _i(X) _j(X)]+E[ _i _j], so |⟨ηi,ηj⟩|≤|ρij|+εiεj| _i, _j |≤| _ij|+ _i _j by Cauchy–Schwarz. Substituting into (18) yields the stated bound on |ui⊤uj||u_i u_j|; no independence among the YtY_t was used. Centroid geometry. Let Z(X)=(Z1(X),…,Zk(X))Z(X)=(Z_1(X),…,Z_k(X)) and mY(t):=[Zt(X)∣Y]m_Y^(t):=E[Z_t(X) Y]. By (17), [YtZt(X)]=ut⊤wt=‖wt‖2=B(t)(F)E[Y_tZ_t(X)]=u_t w_t=\|w_t\|_2= B^(t)(F), and iterated expectation gives [YtmY(t)]=B(t)(F)E[Y_tm_Y^(t)]= B^(t)(F). Expanding the square and using Yt2=1Y_t^2=1, [(mY(t)−B(t)(F)Yt)2]=[(mY(t))2]−B(t)(F)≤[Zt2]−B(t)(F)=1−B(t)(F),E [(m_Y^(t)- B^(t)(F)\,Y_t)^2 ]=E[(m_Y^(t))^2]-B^(t)(F) [Z_t^2]-B^(t)(F)=1-B^(t)(F), where Jensen gives the inequality and whitening gives [Zt2]=ut⊤Irut=1E[Z_t^2]=u_t I_ru_t=1; this step also used no joint structure of the YtY_t. Summing over t yields the claimed centroid bound. ∎ See 4.1 Proof. Write F=(f1,…,fr)F=(f_1,…,f_r). The constraints [F(X)]=0E[F(X)]=0 and [F(X)F(X)⊤]=IrE[F(X)F(X) ]=I_r are equivalent to requiring f1,…,fr\f_1,…,f_r\ to be an orthonormal family in L02(PX)L_0^2(P_X). Using (4), the objective is therefore [⟨F(X(1)),F(X(2))⟩]=∑ℓ=1r⟨fℓ,Tfℓ⟩L2(PX).E\! [ F(X^(1)),F(X^(2)) ]= _ =1^r f_ ,Tf_ _L^2(P_X). Expand fℓ=∑j≥1aℓjψjf_ = _j≥ 1a_ j _j in the orthonormal eigenbasis of T, and define sj:=∑ℓ=1raℓj2s_j:= _ =1^ra_ j^2. Since Tψj=λjψjT _j= _j _j, ∑ℓ=1r⟨fℓ,Tfℓ⟩=∑j≥1λjsj,0≤sj≤1,∑j≥1sj=r. _ =1^r f_ ,Tf_ = _j≥ 1 _js_j, 0≤ s_j≤ 1, _j≥ 1s_j=r. The inequalities follow because the coefficient matrix A=(aℓj)A=(a_ j) has orthonormal rows. Since λ1≥λ2≥⋯ _1≥ _2≥·s, the maximum is attained by s1=⋯=sr=1s_1=·s=s_r=1 and sj=0s_j=0 for j>rj>r, with optimal value ∑j=1rλj _j=1^r _j. Taking F0=(ψ1,…,ψr)F_0=( _1,…, _r) attains this value, so ψ1,…,ψr _1,…, _r are feasible and optimal. Finally, the eigengap λr>λr+1 _r> _r+1 makes the maximizing allocation unique. Hence any optimal F has coordinate span spanψ1,…,ψrspan\ _1,…, _r\. ∎ See 4.2 Proof. By Prop. 4.1, F⋆=rS_F =U_r. Since (ψj)j≥1( _j)_j≥ 1 is an orthonormal eigenbasis of T on L02(PX)L_0^2(P_X) and η∈L02(PX)η∈ L_0^2(P_X), write η=∑j≥1βjψjη= _j≥ 1 _j _j, where βj=⟨η,ψj⟩L2(PX) _j= η, _j _L^2(P_X). Hence Πrη=∑j=1rβjψj _rη= _j=1^r _j _j, and orthonormality gives B(F⋆)=‖Πrη‖L2(PX)2=∑j=1rβj2B(F )=\| _rη\|_L^2(P_X)^2= _j=1^r _j^2. ∎ E.2 Proofs for Appendix C See C.1 Proof. Let G=[F(X)F(X)⊤]G=E[F(X)F(X) ]. Both terms in (3) are invariant under orthogonal changes of feature coordinates, so diagonalize G and write the resulting coordinates as gj=tjhjg_j= t_j\,h_j, where tj≥0t_j≥ 0 and the nonzero hjh_j’s are orthonormal in L02(PX)L_0^2(P_X); complete them arbitrarily to an orthonormal family when some tj=0t_j=0. Reorder so that t1≥⋯≥trt_1≥·s≥ t_r. The objective becomes ∑j=1rtj⟨hj,Thj⟩−γ2∑j=1r(tj−1)2. _j=1^rt_j h_j,Th_j - γ2 _j=1^r(t_j-1)^2. By the weighted Ky Fan principle, ∑jtj⟨hj,Thj⟩≤∑jtjλj _jt_j h_j,Th_j ≤ _jt_j _j. Hence the objective is at most ∑j=1r[tjλj−γ2(tj−1)2], _j=1^r [t_j _j- γ2(t_j-1)^2 ], whose j-th term is uniquely maximized at tj=1+λj/γt_j=1+ _j/γ. Equality is attained by taking hj=ψjh_j= _j. The eigengap fixes the maximizing r-dimensional subspace, while rotations within eigenspaces of repeated eigenvalues are absorbed into an orthogonal output matrix. Thus every maximizer has the stated form with sj2=tj=1+λj/γs_j^2=t_j=1+ _j/γ. ∎ See C.2 Proof. Let D=diag(s12,…,sr2)D=diag(s_1^2,…,s_r^2) and b=(s1β1,…,srβr)⊤b=(s_1 _1,…,s_r _r) . Then Cov(Fγ⋆(X))=RDR⊤Cov(F _γ(X))=RDR , while balancedness and centering give μ+=Rb _+=Rb, μ−=−μ+ _-=- _+, and ‖μ+‖22=Bγ\| _+\|_2^2=B_γ. By total covariance, Σ++Σ−=2(RDR⊤−μ+μ+⊤). _++ _-=2 (RDR - _+ _+ ). Taking traces and using ‖Δ‖22=4Bγ\| \|_2^2=4B_γ yields VFγ⋆=12(Sγ/Bγ−1)V_F _γ= 12(S_γ/B_γ-1). For u=μ+/Bγu= _+/ B_γ, we have u⊤RDR⊤u=Cγ/Bγu RDR u=C_γ/B_γ, so contraction along u gives V~Fγ⋆=12(Cγ/Bγ2−1) V_F _γ= 12(C_γ/B_γ^2-1). The case Bγ=0B_γ=0 follows from the stated convention. ∎