Paper deep dive
Sparse Prototype Code Underlies Classification and Prediction Across Modalities
Yehonatan Avidan, Daniel D. Lee, Haim Sompolinsky
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/22/2026, 2:40:55 AM
Summary
This paper proposes a sparse prototype code theory to explain classification accuracy in deep neural networks across vision, audio, and language modalities. The authors demonstrate that within-class variability is not random but strongly correlated with the true class centroid and rival class centroids. By deriving an analytical mean-field theory based on these centroid-aligned correlations, they accurately predict classification performance. The study highlights that accurate prediction relies on a sparse set of centroid coordinates, connecting representational geometry to sparse feature extraction methods like sparse autoencoders.
Entities (8)
Relation Signals (6)
Sparse Prototype Code → predicts → Classification Accuracy
confidence 95% · The theory accurately predicts classification accuracy across architectures and modalities.
Within-class variability → correlateswith → true class centroid
confidence 94% · its classifier-relevant component has strong and structured correlations with the class's own centroid
Within-class variability → correlateswith → rival class centroids
confidence 94% · and with the centroids of its competing classes.
Prototype Classifier → uses → centroid
confidence 92% · the class-specific readout vector is not a learned weight but the centroid - the empirical mean of the class in representation space
Mean-Field Theory → governedby → variability along true-class and rival-class centroid coordinates
confidence 90% · an analytical mean-field theory governed mainly by the variability along true-class and rival-class centroid coordinates
Sparse Prototype Code → connectsto → Sparse Autoencoders
confidence 85% · connecting our framework to sparse-feature extraction approaches such as sparse autoencoders.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural representations have become a central tool for studying the internal mechanisms of modern AI models, yet their complex high-dimensional structure makes them difficult to interpret. We show that classification tasks give rise to a universal representational geometry, shared across state-of-the-art models in vision, audio, and language processing. The key structure is that within-class variability is not random in representation space. Instead, its classifier-relevant component has strong and structured correlations with the class's own centroid and with the centroids of its competing classes. Building on this observation, we derive an analytical mean-field theory governed mainly by the variability along true-class and rival-class centroid coordinates, together with a global renormalization of the class radius that compensates for the non-Gaussian statistics of real representations. The theory accurately predicts classification accuracy across architectures and modalities. The relevant geometric quantities improve systematically with model scale, mirroring the observed gains in accuracy. A striking feature of the theory is its sparsity: accurate prediction requires only a small set of centroid coordinates associated with the true class and its strongest rivals - connecting our framework to sparse-feature extraction approaches such as sparse autoencoders. Together, these results provide a parsimonious predictive theory of neural representations and suggest that classification in deep networks is governed by a sparse, centroid-aligned structure embedded within the full high-dimensional representation space.
Tags
Links
- Source: https://arxiv.org/abs/2608.15632v1
- Canonical: https://arxiv.org/abs/2608.15632v1
Trouble viewing inline? Open PDF directly →
Full Text
125,072 characters extracted from source content.
Expand or collapse full text
Sparse Prototype Code Underlies Classification and Prediction Across Modalities Yehonatan Avidan Affiliation: The Racah Institute of Physics, Hebrew University Affiliation: Edmond and Lily Safra Center for Brain Sciences, Hebrew University Daniel D. Lee Affiliation: Department of Electrical and Computer Engineering, Cornell Tech, Cornell University Haim Sompolinsky Affiliation: The Racah Institute of Physics, Hebrew University Affiliation: Edmond and Lily Safra Center for Brain Sciences, Hebrew University Affiliation: Center for Brain Science, Harvard UniversityYehonatan.Avidan@mail.huji.ac.il Abstract Neural representations have become a central tool for studying the internal mechanisms of modern AI models, yet their complex high-dimensional structure makes them difficult to interpret. We show that classification tasks give rise to a universal representational geometry, shared across state-of-the-art models in vision, audio, and language processing. The key structure is that within-class variability is not random in representation space. Instead, its classifier-relevant component has strong and structured correlations with the class’s own centroid and with the centroids of its competing classes. Building on this observation, we derive an analytical mean-field theory governed mainly by the variability along true-class and rival-class centroid coordinates, together with a global renormalization of the class radius that compensates for the non-Gaussian statistics of real representations. The theory accurately predicts classification accuracy across architectures and modalities. The relevant geometric quantities improve systematically with model scale, mirroring the observed gains in accuracy. A striking feature of the theory is its sparsity: accurate prediction requires only a small set of centroid coordinates associated with the true class and its strongest rivals - connecting our framework to sparse-feature extraction approaches such as sparse autoencoders. Together, these results provide a parsimonious predictive theory of neural representations and suggest that classification in deep networks is governed by a sparse, centroid-aligned structure embedded within the full high-dimensional representation space. 1 Introduction One of the central challenges in modern Deep Neural Network (DNN) research is to open the "black box" of computation and to understand the internal mechanisms by which networks transform their inputs into useful predictions. Such understanding is essential for building safer, more reliable AI systems (3; 23). A primary tool toward this goal is the analysis of representations - the hidden-layer activations induced by specific inputs. These representations form structured point clouds whose geometry reflects both the data and the model, but extracting interpretable principles from this structure remains a significant challenge. A growing body of work in mechanistic interpretability addresses this challenge by extracting sparse, human-interpretable features from network activations (15; 4; 26; 49). Sparse autoencoders, feature dictionaries, and circuit analyses have shown that semantically meaningful concepts are often encoded along specific directions in activation space and can be isolated within otherwise opaque representations. These approaches are powerful tools for identifying the essential features of the network; they are typically not aimed, however, at providing a quantitative theory linking the geometric structure of a representation to how well it supports a downstream task. Such a theory would help address fundamental questions, such as why some networks generalize better than others. Predictive theories of representational geometry —manifold capacity (6; 7; 48; 51) and neural collapse (39; 21; 17) — have made important progress in connecting geometry to performance within their respective regimes. Capacity theory studies the linear separability of object manifolds under random binary dichotomies, relating global capacity to geometric quantities such as manifold radius, effective dimension, and inter-manifold correlations. Neural collapse describes the final phase of supervised training, in which within-class variability collapses and class means converge to a simplex equiangular tight frame. Modern pretrained foundation models, however, typically operate in a different regime: they are evaluated using multiclass accuracy and are often trained with early stopping or self-supervised objectives, leaving substantial within-class variability. Recent work connects pairwise class geometry to performance and generalization bounds in few-shot learning (47; 33). A predictive geometric theory of multiclass accuracy in the finite-radius regime that accounts for both competition among multiple classes and the rich within-class statistics of many examples is currently missing. Our analytical lens is prototype-based classification. A long line of work - from nearest-class-mean classifiers (35) to prototypical networks for few-shot learning (46) - has established that classifying inputs by proximity to class centroids is a strong strategy whenever representations are well-structured. Adopting the centroid-based classifier in this work has a clear advantage: the centroids are geometric objects, allowing classification accuracy to be related directly to geometric measures of the representation - while remaining relevant for realistic classification settings. Thus, we leverage this framework to close the theoretical gap, deriving and empirically validating a quantitative geometric theory of multi-class classification accuracy using prototype classification, and apply it across architectures and modalities. Our main contributions are: Universal centroid-aligned structure: We identify a universal representational geometry shared across vision, language, and audio backbones. Its defining property is that within-class variability is strongly correlated with both the true-class centroid and the centroids of competing classes (see Fig. 1a). We explicitly demonstrate this by surgically removing only these centroid-aligned correlations (using rotations, see Fig. 1b) - leaving centroid-centroid correlations, total variance, and example-example structure intact - and find that doing so drives per-class accuracy to essentially 100%. The centroid-aligned correlations are therefore precisely the geometric source of classification errors. Sparsity of competition: These correlations do not spread evenly across the other classes; for each class, the geometry concentrates on a small set of active rivals, which is sufficient to recover the full geometric picture and reproduce the predicted per-class accuracy. These rivals also appear to carry semantically meaningful relations to the true class (see Fig. 1a). Each example’s classification is therefore faithfully captured by a sparse, semantically meaningful centroid-aligned basis embedded in the full high-dimensional representation - a grounded analog of the sparse-feature picture pursued in mechanistic interpretability. Compact predictive mean-field theory: To connect representational geometry to classification, we adopt a prototype classifier - scoring examples by dot product with unit-normalized class centroids (46) - which serves as an empirically reliable proxy for the trained linear head (see Appendix A.1). Building on this, we derive an analytical mean-field theory in the sparse centroid basis (Sec. 5). Its inputs are the variability along the true- and rival-class centroid directions and a single global radius renormalization that absorbs the heavy-tailed statistics of real representations (see Fig. 3c). Crucially, this factor is tightly correlated with a direct measure of tail heaviness (Fig. 4e). From these inputs alone, the theory predicts per-class accuracy across classes, architectures, and modalities (see Fig. 4(a-d), Appendix H). Geometry across model scales: Tracking the theory’s geometric ingredients across model families, we find that they systematically improve as the models scale up, mirroring the gains in accuracy (Table 4). Importantly, we show that larger models achieve their accuracy advantage without having the smallest overall within-class variance. Our theory identifies the specific geometric measures that drive this improvement - taking a step toward understanding the key geometrical ingredients required for good generalization in real state-of-the-art models. We hypothesize that these measures are practically actionable: targeting them during training could improve generalization; matching them between teacher and student could guide more geometry-aware distillation (24) and could help identify the best-suited pretrained backbone for transfer (27). Figure 1: Centroid-aligned variability, Qwen3-32B, TinyStories representations. The variability of each class is highly correlated with other class centroids. (a) To visualize it, we use Qwen3-32B representations (54) on the dataset TinyStories (14). We project each example onto the three centroids with the largest correlations among the class variability. For 217/320 classes (68%68\%), the true class centroid is among these top 3 (the nine shown here are of this sort). The rivals carry semantic meaning (e.g., the rivals of "cookie" are "food" and "delicious"; the rivals of "head" are "face" and "hand"). The mean prototype classification accuracy of these classes (see Def. 2.1) is 76.4%76.4\%. (b) To causally test the role of the variability-centroid correlations, we rotate the class manifold around its mean (denoted as δ-rotation), breaking these correlations while leaving the class radius and the distance between class centroids unchanged. The manipulation removes all classification errors, and the projections onto the top-3 directions shrink to 11.3%11.3\% of their original magnitude on average, and those directions no longer carry semantic meaning. 2 Setup and Notation We study how the geometry of neural representations determines classification accuracy. For each input we consider a representation vector taken from some layer of a neural network, and the task is to assign it to one of P classes. Prototype classification: Rather than training a linear readout on top of the representation, we use a prototype classifier (46), in which the class-specific readout vector is not a learned weight but the centroid - the empirical mean of the class in representation space (Def. 2.1), normalized to a unit vector. Centroids are estimated from a labeled set, and at inference, the learner classifies one example against all P centroids (computed without it, "Leave one out" (29)). This has a direct geometric definition while remaining relevant for realistic classification, can be applied at every layer of every model without additional fine-tuning, and empirically tracks trained linear readouts closely (see Appendix A.1), making it both a flexible evaluator in its own right and a useful proxy for the standard linear head. Experimental setup: We evaluate frozen representations from pretrained backbones spanning vision (ImageNet-1K (10)), language (masked and next-token prediction on TinyStories (14)), and audio (VGGSound (5)), across transformer and convolutional architectures, supervised, self-supervised, and contrastive objectives. Full lists of datasets, models, and extraction details are in Appendix B. For language, the task is to predict a content word from its representation. In autoregressive models this is the embedding of the token preceding the word; in bidirectional models it is the embedding at the word’s masked position. We group each word’s inflections under a single lemma label (e.g. say/says/said, friend/friends), and keep only lemmas with at least 1000 examples, so their statistics are well estimated. Full technical details are in Appendix B. Definition 2.1 (prototype classification): Let ∈ℝNx ^N be the representation of an input at a given layer of a neural network, and consider a P-class task with labels y∈1,…,Py∈\1,…,P\ and dataset =(i,yi)i=1nD=\(x_i,y_i)\_i=1^n. For each class μ, define the centroid μ=[∣y=μ]c_μ=E[x y=μ] and its unit direction ^μ=μ/‖μ‖ c_μ=c_μ/\|c_μ\|, which serves as the prototype class readout. Throughout the paper, we use the terms centroid and prototype interchangeably. The prototype logits and accuracy are defined as tμ()=^μ⋅,Acc=(,y)∈[argmaxμtμ=y].t_μ(x)= c_μ·x, =E_(x,y) [1\ _μt_μ=y\]. (1) Definition 2.2 (manifold geometry): The class manifold is defined as the point cloud of a particular class k: ℳk=i:yi=kM_k=\x_i:y_i=k\. Each ∈ℳkx _k decomposes as =k+δx=c_k+ with ∈ℳk[δ]=E_x∈ _k[ ]=0. We name δ the residual vector. The class radius R measures the overall within-class variance, and σμ2∈[0,1] _μ^2∈[0,1] is the fraction of the residual variance along a particular centroid direction ^μ c_μ: R2=k[‖δ‖2]‖k‖2,σμ2=k[(δ⋅^μ)2]k[‖δ‖2].R^2\;=\; E_k[\| \|^2]\|c_k\|^2, _μ^2\;=\; E_k[( · c_μ)^2]E_k[\| \|^2]. (2) Throughout, geometric quantities depend on a fixed true class k. We treat k as fixed and drop the subscript to lighten notation: R≡RkR≡ R_k, σkμ≡σμ _kμ≡ _μ, and σk2 _k^2 is the fraction of residual variance along the true centroid ^k c_k. In addition, kE_k, denotes the average over all the points in the manifold k, k≡∈ℳkE_k _x _k. 3 Isotropic Uncorrelated Model We first examine a baseline of an uncorrelated model, in which the centroids μc_μ and the residuals δ are drawn independently from an isotropic Gaussian distribution. The model yields testable predictions for how geometry shapes accuracy, which we evaluate in Sec.4 on real data: deviations from these predictions quantify the role of correlations in the classification task. Assumption 3.1 (uncorrelated statistics). For each class μ=1,…,Pμ=1,…,P, the centroid is drawn as μ∼iid(,IN)c_μ\! _iid\!N(0,I_N). Conditional on the centroids, examples of class k are generated by sampling =k+δx=c_k+ , with residuals drawn as δ∼iid(,R2IN) \! _iid\!N(0,R^2I_N). Assumption 3.2 (high-dimensional geometry). We work in the high-dimensional limit N→∞N→∞, with P=N(1)P=N^O(1) classes and a radius R=R(N)R=R(N) that may scale with N. We refer to Assumptions 3.1-3.2 jointly as the isotropic model. Theorem 3.3 (isotropic model accuracy). Under the isotropic model, Acciso.(R,N,P)=z∼(0,1)[ΦP−1(Rz+NR2+1)],Acc^iso.(R,N,P)=E_z (0,1)\! [ ^P-1\! ( Rz+ N R^2+1 ) ], (3) where Φ is the standard Gaussian CDF. The supporting lemmas (centroid-overlap CLT, self-averaging of AccAcc, and the joint Gaussian distribution of the logits), together with the proof of Theorem 3.3, are given in Appendix C.1. Predictions. Several consequences follow directly from Theorem 3.3: (i) For any R=(1)R=O(1), the classes are well separated, and classification is trivial: Acciso.→1Acc^iso.→ 1. (i) In the infinite radii limit, R/N→∞R/ N→∞, the isotropic residual dominates the input, which thus retains no information about the true class: all P classes are equally likely to be predicted, and accuracy collapses to chance, Acciso.→1/PAcc^iso.→ 1/P. (i) Non-perfect accuracy requires large radii, Acciso.<1⇒R=Ω(N/lnP)Acc^iso.<1 R= ( N/ P). (iv) In the high-dimensional regime, the accuracy is self-averaging: its variance across centroid realizations is ((PR2)−1)O((PR^2)^-1). We note that according to (i), R also scales with N for nontrivial accuracy. These predictions are rigorously derived in Appendix C.1 (Corollaries C.1.6-C.1.9), and verified numerically with synthetic data in Appendix Fig.6. Figure 2: Geometric signatures of multiclass classification (a) Per-class accuracy vs. empirical radius R: sharp decrease across the narrow empirical R range and a wide spread at fixed R. After random rotation of within-class variability δx (δ-rotation), the accuracy is perfect. (b) We artificially scale the variability δ→(R⋆/R)δ \!→\!(R /R) . The accuracy decreases fast but stabilizes at 39%~39\% accuracy for R⋆→∞R →∞ due to strong true centroid correlations (blue line). After random rotation of the variability δ-rotation, the accuracy is well described by the isotropic model (MSE=1.26×10−3MSE=1.26× 10^-3, orange line). Additional breaking of centroid-centroid correlations (full rotation) has a very small effect; lines overlap. 4 Empirical Results In this section we characterize the geometric structure of trained representations that governs prototype classification. We begin from the predictions of the isotropic model (Sec.3) and show that they are not satisfied by real representations. This pattern of failure is informative: it identifies which correlations dominate the geometry responsible for accuracy, and isolates the components needed to build the analytical framework of Sec.5. Results are presented for DINOv3 7B (44) representations on ImageNet-1K (10), and later replicated in many models and modalities (Sec. 5.3). Accuracy changes on an (1)O(1) radius scale: The empirical class radii lie in a narrow R∈[0.4,4.3]R∈[0.4,4.3] range. In Sec.3, we saw that an uncorrelated model predicts that accuracy should remain unchanged throughout (1)O(1) radii, and decline only in R=Ω(N/lnP)∼20R= ( N/ P) 20. However, we observe a dramatic change in accuracy, ranging from 100%100\% to 61%61\% averaged over a small R window (see Fig.2a). The sharp accuracy decline therefore indicates structure that is not described by the isotropic model. Large class-to-class variability at fixed radius: The isotropic model predicts that class-to-class fluctuations are of order 1/RP1/R P (∼1%−3% 1\%-3\%). Empirically, however, the spread of accuracies at fixed R is large. For example, the 89 classes in the range R∈[1.2,1.3]R∈ [1.2,1.3 ] have a mean accuracy of 90.4%90.4\%, with a minimum accuracy of 61.8%61.8\%, and a maximum of 99.6%99.6\% (see Fig.2a). The radius alone therefore cannot explain classification performance: additional class-dependent geometric structure must determine which classes are easy to classify and which are difficult. 4.1 Geometric manipulations To causally test which components cause the deviation from the isotropic model, we use geometric manipulations to probe specific structures of the representations. We apply two geometric manipulations: (i) Variability inflation, where we set each class radius to a chosen R⋆R artificially by scaling its deviations, (R⋆)=k+(R⋆/R)δx(R )=c_k+(R /R) , where R is the empirical radius as defined in Sec.2. (i) Geometric rotations, where we surgically destroy specific correlations in the data while preserving all other aspects of the representation geometry. Strong residual–true-centroid correlations: We first examine the representations under variability inflation (R⋆)=k+(R⋆/R)δx(R )=c_k+(R /R) . Naively, as R⋆→∞R →∞ one expects chance accuracy, as we saw in the isotropic model. Empirically, the large-radius accuracy instead stays far above chance - accuracy approaches 39.2%39.2\%, over two orders of magnitude above 1/P=0.1%1/P\!=\!0.1\% (Fig.2b). The within-class variability is thus far more correlated with the true centroid than with the rival centroids. To quantify this directly, we measure the projection variance along each centroid direction σμ _\!μ (see Sec. 2). The true-class σk _k exceeds all rivals in 93%93\% of classes. Moreover, this dominance is itself predictive of a well-structured class geometry: in the 930930 classes where σk _k exceeds all rivals, the mean accuracy is 87.4%87.4\%, whereas in the remaining 7070 classes it drops to 60.1%60.1\%. Residual-centroid correlations are the main drivers of errors: We manipulate the representations by rotating the residuals δ around their own class centroid (δ-rotations): for each class k we replace =k+δx=c_k+ by δrot=k+Okδx^δ rot=c_k+O_k , with OkO_k a class-specific random orthogonal matrix (see Fig. 1b for illustration). This preserves the centroid-centroid correlations, the class radii, and the example-example relations, while breaking the correlations between δ and the centroid directions (residual-centroid correlations). At the empirical radius, the class-averaged accuracy rises from the baseline of 85.5%85.5\% to 100%100\%, completely eliminating classification errors (see Fig.2a, orange line). This is achieved without breaking centroid-centroid correlations. Breaking residual–centroid correlations recovers the isotropic model: To look for any further role of residual–centroid correlations, we go outside the empirical radius range, where δ-rotation has already saturated the accuracy. We set each class radius to a chosen R⋆R by variability inflation. Strikingly, after removing only the residual-centroid correlations, the data is already well described by the isotropic model accuracy (Eq.3), with MSE=1.26×10−3MSE=1.26× 10^-3. The accuracy declines only at R=Ω(N/lnP)R= ( N/ P) , far beyond the empirical R range (see Fig.2b). The accuracy at R⋆→∞R →∞ is chance (0.1%0.1\%), and the per-class standard deviation around the mean accuracy shrinks by an order of magnitude to 2.3×10−32.3× 10^-3 compared to 1.7×10−21.7× 10^-2 in the unmanipulated representations. Centroid-centroid correlations play a minor role: To test the role of centroid-centroid correlations, we apply a stronger manipulation: rotation of the whole class manifold around the origin by another random orthogonal matrix after δ-rotations, full-rot=Uk(k+Okδ)x^full -rot=U_k (c_k+O_k ), which breaks centroid–centroid correlations on top of the δ-rotation. Even at these larger radii the extra effect is small: the full-rotation curve differs from the δ-rotation curve by only MSE=×10−5MSE=2.0\!×\!10^-5 (see Fig. 2b). We infer that centroid–centroid correlations play a minor role in accuracy. This echoes the picture from neural collapse (39), where the centroids are as spread as possible. Together, these empirical results identify the key geometric structure of trained representations: the class-relevant geometry is governed by strong correlations between within-class variability and a sparse set of centroid directions—dominated by the true-class centroid. 5 Centroid-Aligned Variability Model The empirical results of Sec. 4 identify the geometry of prototype classification in pretrained representations: Within-class variability δ is centroid-aligned. Surgically removing only the δ -centroid correlations drives accuracy to ∼100% \!100\%, while breaking centroid-centroid correlations has only a secondary effect. We introduce the centroid-aligned model that incorporates this geometry. The resulting theory takes as input geometric quantities measurable from the representation, together with a single global renormalization of the class radius (Sec. 5.2), and predicts per-class accuracy across architectures and modalities. All proofs, lemmas, and intermediate computations are deferred to Appendix C.2; the main text retains only the core assumptions and central results. We stress that the following assumptions are introduced to define an analytically solvable model and are not intended as exact claims about the statistics of the data. Nevertheless, the model is motivated by empirical observations (Sec. 4) and, as we show below, yields accurate predictions across models spanning different architectures and modalities. We therefore believe that it captures important aspects of real model representations. Definition 5.1 (rival support). Fix a class k and an integer K. Define its rival support ℛKR_K as the set of indices μ≠kμ≠ k corresponding to the K largest values of σμ _μ (Def. 2.2), and set ℛK+≡k∪ℛKR_K^+≡\k\ _K. For each μ∈ℛKμ _K, define the true-rival centroid correlation by gμ=^k⋅^μg_μ= c_k· c_μ. Definition 5.2 (standardized centroid projections). Conditioned on class k, each example is represented by the following projection coefficients: sμ=δ⋅^μstdk[δ⋅^μ],μ∈ℛK+s_μ= · c_μstd_k [ · c_μ ], μ _K^+ (4) Assumption 5.3 (centroid-aligned variability model). Let Gμν=^μ⋅^νG_μν= c_μ· c_ν be the centroids Gram matrix. Assume that the restricted centroid Gram matrix of the rival support (Gμν)μ,ν∈ℛK+(G_μν)_μ,ν _K^+ is invertible. We assume the following generative model of examples of class k ()=‖k‖[^k+R(∑μ,ν∈ℛK+σμsμGμν−1^ν+σ⟂s⟂^⟂)]x(s)= \|c_k \| [ c_k+R ( _μ,ν ^+_K\!\!\! _μs_μ\,G^-1_μν c_ν+ __ s__ e__ ) ] (5) where ^⟂~ e__ is perpendicular to the span(^μ∈ℛk+)~span( c_μ _k^+), σμ2 _μ^2 is the variance fraction along the centroid μ direction, and R is the empirical class radius (see Def. 2.2). Eq. 5 encodes the geometry identified in Sec. 4 by placing the residuals δ along rival centroid directions in the rival support. Assumption 5.4 (Gaussian statistics). We assume the projection coefficients sμs_μ are jointly Gaussian with zero mean and unit variance, and that their covariance has the following form Cov(sμ,sν)=δμν(1−ρμ2)+ρμρν,Cov (s_μ,s_ν )= _μν (1-ρ^2_μ )+ _μ _ν, (6) where ρμ=k[sksμ] _μ=E_k [s_ks_μ ], and by definition ρk=1 _k=1. These are the Pearson correlations between the residual projections δ⋅^k · c_k and δ⋅^μ · c_μ. Equivalently, all dependence among the rival projections is captured by sks_k: conditioned on sks_k, they are mutually independent. Theorem 5.5 (centroid-aligned model accuracy). Under assumptions 5.3-5.4 Acckth(R)=z∼(0,1)[∏μ∈ℛKΦ(σk−ρμσμσμ1−ρμ2z+1−gμRσμ1−ρμ2)].Acc^th_k(R)=E_z (0,1)\! [ _μ _K ( _k- _μ _μ _μ 1-ρ^2_μz+ 1-g_μR _μ 1-ρ^2_μ ) ]. (7) The inputs to Eq. 7 are entirely geometric and class-specific: the empirical radius R, the rival projected variances σμ\ _μ\, the centroid-centroid overlaps gμ\g_μ\, and the correlation terms ρμ _μ are all measured directly from the representations. For convenience, we summarize our notations and how we measure them in Table. 2. Figure 3: Sparsity and heavy tail statistics: (a) K-way classification of rivals chosen by centroid projection variance σμ _μ converges quickly to the full P-way classification. (b) We define the strongest rival of each example μ∗()=argmaxμ≠ktμ()μ^*(x)= _μ≠ kt_μ(x), and count the number of examples captured by each rival. The strongest rival of each class captures more than half of the examples on average, with 90%90\% of the examples captured by only 14.4 rivals. (c) The standardized centroid projections sμs_μ (see Eq.4) have heavy tail power law statistics, which inflate the extreme values that control classification accuracy. The magnified panel shows the log density of the far tail. 5.1 Sparsity The rival count K is a hyperparameter of the theoretical model controlling how many rival directions enter the prediction in Eq. 7. Because Φ approaches one quickly as its argument grows, small projections σμ _μ have very little effect on the overall accuracy. Thus, when ranking rivals by their projection magnitudes σμ _μ, predictive power plateaus rapidly once K exceeds the head of the spectrum. We assess the effective number of rivals both theoretically and empirically. We first replace the full P-way task with a K-way task restricted to the top rivals ranked by σμ _μ. We find that only 6 rivals are needed to come within 3%3\% of the full P-way classification accuracy, 12 within 2%2\%, and 37 within 1%1\% (see Fig.3a). The competition is thus not about the full set of P (1000) classes, but rather a sparse set of strongly correlated rivals. In Appendix E, we compare several rival-selection methods and show that ranking rivals by σμ _μ yields results comparable to more complex selection rules, while remaining simple, interpretable, and easy to compute. In particular, it outperforms selection based on centroid proximity, providing additional evidence that centroid correlations play only a secondary role in the task. We further examine whether the correlations are homogeneously distributed across the rivals or dominated by a few. For each example x of class k, we record its strongest rival, μ∗()=argmaxμ≠ktμ()μ^*(x)= _μ≠ kt_μ(x), and count the number of distinct rivals that appear across the ∼1300 1300 examples of that class. Competition is highly concentrated: on average only 6767 distinct rivals appear, 14.414.4 of them cover 90%90\% of the examples, and one rival alone accounts for more than half (Fig. 3b). The theory’s predictions also saturate quickly as the number of rivals K increases. We find that K∈[10,20]K∈[10,20] is already enough in most models to achieve accurate predictions, with very small gain from increasing K further (see Appendix E). The results in the main text are shown for K=20K=20 for all models. We also note that in many cases, the rivals chosen by σμ _μ carry semantic meaning. In the Qwen3-32B (54) model on TinyStories (14), many rival words are similar in both grammatical role ("smile"↦ "reply", "nod", "say") and semantic meaning ("forest"↦ "park", "garden", "grass"). In vision (DINOv3 ViT-7B), we observe either similar objects ("leopard"↦ "jaguar", "cheetah") or frequently co-occurring objects ("king crab"↦ "plate", "flute"↦ "stage"). For more examples, see Table 1 and Fig. 1a. model class top-5 rivals (by σμ _μ) Qwen3-32B happy excite, proud, glad, surprised, curious bad sad, sorry, scare, angry, tired voice loud, noise, sound, big, funny smile reply, nod, say, think, tell cake delicious, cookie, food, ice, big forest park, garden, grass, world, sky ground floor, grass, swing, slide, garden DINOv3 ViT-7B leopard snow leopard, jaguar, cheetah, impala, tusker king crab Dungeness crab, rock crab, hermit crab, plate, American lobster flute panpipe, oboe, ocarina, bassoon, stage orange lemon, banana, grocery store, Granny Smith, strawberry Table 1: Rival classes selected by the projection’s variance σμ _μ are semantically related words (Qwen3-32B, TinyStories) or visually similar and co-occurring objects (DINOv3 ViT-7B, ImageNet). For more examples see also Fig. 1a. 5.2 Non-Gaussian tails and a global radius renormalization Notation Name Definition kc_k Centroid k[]E_k[x] ^μ c_μ Unit centroid μ/∥μ∥c_μ/ _μ δ Residual −kx-c_k R2R^2 Radius2 k[∥δ∥2]/∥k∥2E_k\! [ ^2 ] / _k ^2 σμ2 _μ^2 Variance fraction k[(δ⋅^μ)2]/k[∥δ∥2]E_k\! [( · c_μ)^2 ] /E_k\! [ ^2 ] sμs_μ Std. projection (δ⋅^μ)/stdk[δ⋅^μ] ( · c_μ )/std_k [ · c_μ ] ρμ _μ Proj. correlation Corrk(δ⋅^k,δ⋅^μ)Corr_k ( · c_k, · c_μ ) gμg_μ Centroid overlap ^k⋅^μ c_k· c_μ ℛKR_K Rival set μ≠k:Klargestσμ\\,μ≠ k:\ K\ largest\ _μ\,\ Table 2: Definitions. kE_k is defined as average over all samples x belong to the manifold k The theoretical accuracy predicted by Eq.7 alone consistently overestimates the accuracy on all the models we checked (see Appendix F). Our main simplifying assumption (Assumption 5.4) is that the statistics of the standardized centroid projections sμs_μ are Gaussian. Empirically, the projections have markedly heavy-tailed statistics (see Fig.3c). Although the variance of sμs_μ is normalized, classification depends on extreme statistics (maxμtμ _μt_μ), and heavier tails inflate these extremes relative to a Gaussian of equal variance. We find that the right tail of the variables sμs_μ that controls the max projection on a certain centroid is well fitted by a Student’s-t power-law distribution p(s)∝(a2+s2)−ν/2p(s) (a^2+s^2)^-ν/2 (see Fig.3c), where the exponent ν shapes the decay of the tail. We adopt the simplest solution: a uniform global variance inflation of all manifolds R↦λR λ R in Theorem 5.5, which adjusts the scale of the Gaussian theory such that its extreme matches the empirical heavy-tailed distribution. The global parameter λ is fitted once per model by λ⋆=argminλ(∑k=1P[Acckemp(R)−Acckth(λR)]2),λ = _λ ( _k=1^P\! [Acc_k^emp(R)-Acc_k^th(λ R) ]^2 ), (8) Thus λ is a single global calibration of non-Gaussian extreme-value statistics of a specific model, not a per-class fitting parameter. Empirically λ⋆λ is well correlated with the inverse tail exponent 1/ν1/ν of the model, r=0.90r=0.90 (see Fig. 4e). This relation is global across 36 models spanning vision, audio, and language, supporting our interpretation of λ as a compensation for non-Gaussian statistics. Figure 4: Centroid-aligned theory matches per-class accuracy across modalities. (a, c) Per-class accuracy vs. empirical radius R for DINOv3 ViT-7B on ImageNet (P=1000P=1000, K=20K=20) and Qwen3-32B on TinyStories next-token prediction (P=320P=320, K=20K=20). Per-class scatter is presented with empirical binned mean. The λ-rescaled theory (λ⋆=1.316λ =1.316, 1.1191.119 resp.) accounts for the empirical accuracy accurately. (b, d) Accurate per-class prediction of the accuracy. MSE=1.83×10−3=1.83× 10^-3 (vision) and 4.43×10−34.43× 10^-3 (language), Pearson r=0.951r=0.951 and 0.9550.955. (e) Optimal λ⋆λ vs. heavy tail measurement (1/ν) on representations for 15 vision backbones, 14 autoregressive language models, 4 masked language models, and 3 audio models. Overall Pearson correlations between λ and 1/ν1/ν are r=0.90r=0.90, confirming that λ is a byproduct of the heavy-tail shape of the projection variables. 5.3 Shared geometry across architectures and modalities The empirical structure of Sections 4-5 was characterized on DINOv3 ViT-7B. Table 3 reports the same measurements across a diverse set of vision, audio, and language backbones: Every observation persists: R is in an (1)O(1) range for any N, with large accuracy fluctuations for fixed R (independent of N). δ-rotation drives the class-averaged accuracy to near-perfect on all backbones; the saturation accuracy (R⋆→∞R →∞) is consistently orders of magnitude above chance, indicating true-class dominance among the residual correlations. K90%K_90\% is the number of rivals that are the strongest competitor for 90% of a class’s examples. This rival set is sparse, K90%≪PK_90\%\! \!P in all cases. The projection sμs_μ right tail is heavy on every backbone. The geometric characterizations required by the theory of Section 5 are therefore present and consistent across different architectures, modalities, and training procedures. Appendix H and Figs. 10-13 report the theory-empirical comparison for all 3636 backbones. Table 3: Universality of the centroid-aligned geometry. For each backbone, we report: the number of classes P; feature dimension N; per-class radius range; per-class mean accuracy ⟨Acc⟩ and its fluctuation around the mean accuracy at a fixed R; mean accuracy after δ-rotation; saturation accuracy at R⋆→∞R →∞ (controlled only by the residual δδ x); K90%K_90\% (mean rivals covering 90%90\% of examples); right-tail Student-t tail index ν (K=20K=20). Backbone P N R range ⟨Acc⟩ Fixed R fluctuations δ-rot R⋆→∞R →∞ ⟨K90%⟩ K_90\% ν Vision – ImageNet-1K DINOv3 ViT-7B/16 1000 4096 [0.4, 4.3] 0.855 0.099 1.000 0.392 14.4 4.5 DINOv3 ConvNeXt-L 1000 1536 [0.2, 3.1] 0.814 0.100 1.000 0.312 14.2 4.4 CLIP ViT-L/14 1000 768 [0.6, 2.7] 0.792 0.124 0.999 0.077 13.9 16.4 ViT-B/16 (IN21K) 1000 768 [0.5, 2.7] 0.853 0.088 1.000 0.265 14.1 4.1 Audio – VGGSound SSLAM 310 768 [0.1, 3.7] 0.628 0.155 0.999 0.067 11.0 7.9 CED 310 768 [0.1, 4.0] 0.628 0.160 1.000 0.062 11.3 7.9 Language - TinyStories masked-token RoBERTa-large 320 1024 [0.6, 2.6] 0.767 0.108 1.000 0.120 13.3 9.6 ModernBERT-large 320 1024 [0.6, 2.7] 0.734 0.112 1.000 0.118 12.4 11.3 BERT-large 320 1024 [0.6, 2.7] 0.710 0.112 1.000 0.120 15.8 10.1 Language - TinyStories next-token Qwen3-8B (L34, −2-2) 320 4096 [0.9, 4.2] 0.540 0.118 1.000 0.107 33.7 11.3 OLMo2-13B (L36, −4-4) 320 5120 [1.0, 4.8] 0.563 0.116 1.000 0.112 35.8 10.2 Llama-3.1-8B (L29, −3-3) 320 4096 [0.8, 4.1] 0.550 0.119 1.000 0.101 31.7 12.4 6 Geometry and Accuracy Across Model Scales The theory of Sec. 5 reduces accuracy to a handful of ingredients of the centroid-aligned geometry. We use these quantities as a diagnostic lens across the DINOv3 and Qwen3 model families at different scales (S(21M)/S+(29M)/B(86M)/L(300M)/H+(840M)/7BS(21M)/S_+(29M)/B(86M)/L(300M)/H_+(840M)/7B, 0.6B/1.7B/4B/8B/14B/32B0.6B/1.7B/4B/8B/14B/32B resp.). The theory reveals which geometric reorganizations drive the improvements in classification performance of larger models. A natural concern is that the accuracy gains of larger models are a trivial variability-reduction effect — that bigger models simply produce tighter manifolds. The Qwen3 family rules this out directly: the mean radius actually grows with scale (⟨R⟩=1.98→2.20 R =1.98→ 2.20 from 0.6B to 32B), yet accuracy rises monotonically from 41%41\% to 57%57\%, the opposite of what a naive approach predicts. In the DINOv3 family, the radius does shrink with scale, but the trend breaks at the largest model: ViT-7B carries a mean radius 0.210.21 larger than ViT-H+ yet matches its accuracy. Reduced variability, therefore, cannot be the mechanism behind improved generalization. Our other geometric measures are independent of R, so they isolate genuine reorganizations of the geometry across scale, and correctly identify the components that make the larger models’ representations better. We specifically look at the worst rival μ∗μ^*, measured by the maximum σμ _μ (most correlated centroid among the variability) for each class, and average its geometric statistics over all classes. The theory exposes a coherent set of reorganizations, shared among the two families. Within-class variability becomes more aligned with the true centroid and less aligned with the rival (the ratio σk/σμ∗ _k/ _μ^* increases, σμ∗ _μ decreases in both families). The centroids decorrelate - the overlap between the true-centroid and its hardest rival gμ∗g^*_μ drops - and the projections of the residual on the centroids also become less correlated (ρ∗ρ^* decreases). In language, we see similar trends to vision but with less favorable values, which explains the overall lower accuracy. Finally, representations become more heavy-tailed with scale (ν decreases, 19.3↦4.519.3 4.5 in vision and 21.9↦11.421.9 11.4 in language) . While we cannot directly link this trend to improved accuracy, it is consistent. A large body of literature (40; 30; 50; 45) reports that in many cases, the best representations for tasks are found in the middle hidden layers, and not in the final layer. We add our geometric angle to it in finding that in autoregressive models, the best prototype classification is consistently 1-3 layers before the final layer. We report here the geometry for the best layer found for prototype classification, and analyze how the geometry changes across depth in Appendix G. Table 4: Geometric measures across model scales: DINOv3 ViT family on ImageNet-1K (top block) and the Qwen3 language-model family on TinyStories (bottom block). The geometric quantities are defined in Sec. 5 and Table 2; the superscript ∗ flags the hardest competitor of a class, identified by σμ _μ. Best measure within each family marked in bold. Model ⟨R⟩ R σk/σμ∗ _k/σ^*_μ σμ∗σ^*_μ gμ∗g^*_μ ρμ∗ _μ^* ν Acc DINOv3 ViT-S 1.68 1.06 0.172 0.574 0.558 19.3 0.706 DINOv3 ViT-S++ 1.60 1.09 0.172 0.555 0.535 14.4 0.738 DINOv3 ViT-B 1.52 1.19 0.163 0.542 0.529 9.0 0.794 DINOv3 ViT-L 1.39 1.36 0.147 0.447 0.322 7.0 0.845 DINOv3 ViT-H++ 1.29 1.59 0.164 0.236 -0.011 5.3 0.855 DINOv3 ViT-7B 1.50 2.01 0.120 0.289 0.038 4.5 0.855 Qwen3-0.6B (L27, −1-1) 1.98 0.85 0.241 0.576 0.515 21.9 0.406 Qwen3-1.7B (L26, −2-2) 2.09 0.95 0.212 0.542 0.494 13.9 0.474 Qwen3-4B (L34, −2-2) 2.05 0.98 0.202 0.526 0.463 12.7 0.519 Qwen3-8B (L34, −2-2) 2.12 0.99 0.188 0.514 0.453 11.3 0.540 Qwen3-14B (L37, −3-3) 2.10 1.00 0.186 0.504 0.447 11.3 0.556 Qwen3-32B (L61, −3-3) 2.20 1.01 0.175 0.503 0.451 11.4 0.566 7 Discussion This work identifies geometric organization principles for multiclass representations: prototype accuracy is controlled by a sparse, centroid-aligned code embedded in the full activation space. Across vision, audio, and language models, and across architectures and training objectives, the same geometry recurs. Within-class variability is not isotropic around each class center; its classifier-relevant components are arranged along the true centroid and a small set of rival centroids. The evidence is causal as well as descriptive: breaking this alignment, while preserving the remaining parts of the geometry, nearly eliminates prototype errors. Thus the full N-dimensional representation can be replaced, for the purpose of predicting class accuracy, by a much smaller set of task-aligned coordinates. The theory makes this reduction quantitative. Per-class accuracy is predicted from the geometric measures of centroid-projection statistics, and a single global scale λ that captures the effect of heavy tails in the marginal projection distributions. Radius alone is insufficient to explain the results. We show that as models become larger, the improved accuracy is not due to reduced class variance but because of reorganization of the geometry: variability shifts toward the true-class coordinate, correlations with the strongest rivals decrease, and centroid directions become more decorrelated. Importantly, the theory tracks not only prototype classification but also standard linear classification (Appendix A.1). For linear readouts, a train–test split is essential, since a readout can fit arbitrary labels whenever the data remain separable. Prototype classification, by contrast, achieves high accuracy only for genuinely well-organized geometry — shuffled labels or random representations perform near chance — so the train–test split (calculating the centroids on different data than the one classified) contributes only finite-sample effects, negligible for large sample sizes when train and test are drawn from the same distribution. This perspective complements existing theories of representational geometry. Manifold-capacity theory (6; 7) models class manifolds as ellipsoids and characterizes their linear separability under random binary dichotomies. In that setting, the primary component of the geometry is the random structure of the whole P manifolds, and correlations enter as corrections on top of the random theory (51). We showed that in our task the centroid-aligned component of within-class variability is the dominant part, while the random fluctuations of inter-centroid overlaps have little to no effect. The two frameworks are complementary regimes of the same underlying geometry, and it would be interesting to further investigate where the two descriptions meet. The sparse code uncovered here also suggests links to interpretability and neuroscience. Sparse autoencoders and dictionary-learning methods seek low-dimensional, semantically meaningful directions in activations (34; 26; 49; 18); our results identify a complementary sparse class-supervised set of directions whose functional role is directly tied to classification. In parallel, deep networks have been shown to predict responses in primate ventral-stream areas, including IT cortex (53; 43), and object-manifold theory has been applied to biological populations (16; 47; 28). It would be interesting to examine whether the universal geometry we identify across many artificial models also applies to representations in the brain. Finally, the theory points to practical applications. Because its ingredients are compact and inexpensive to estimate, they can serve as diagnostics for representation quality, to select which pretrained model is most suitable for fine-tuning on a new task (27). They could also suggest geometry-aware objectives: training could explicitly encourage strong own-centroid alignment, weak rival alignment, and sparse competition, while distillation could match the teacher and student geometry rather than the entire N-dimensional representation or the logit distribution (24). Following these directions would turn the centroid-aligned description from a predictive theory of existing representations into a constructive principle for designing better ones. Acknowledgments: We thank David G. Clark, Yoni Ankri, and Nadav Lederman for fruitful discussions. This research is supported by the Gatsby Charitable Foundation, the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University, the Office of Naval Research grant No. N0014-23-1-2051, and the Institute of Information and Communications Technology Planning and Evaluation (IITP), grant funded by the Korean government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project). References Alex et al. (2025) T. Alex, S. Atito, A. Mustafa, M. Awais, and P. Jackson Sslam: enhancing self-supervised models with audio mixtures for polyphonic soundscapes. In International Conference on Learning Representations, Vol. 2025, p. 22608–22626. Cited by: §B.2. Arps et al. (2024) D. Arps, L. Kallmeyer, Y. Samih, and H. Sajjad Multilingual nonce dependency treebanks: understanding how language models represent and process syntactic structure. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 7822–7844. Cited by: §B.1. Bommasani et al. (2021) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: §1. Bricken et al. (2023) T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread. Note: https://transformer-circuits.pub/2023/monosemantic-features/index.html Cited by: §1. Chen et al. (2020) H. Chen, W. Xie, A. Vedaldi, and A. Zisserman Vggsound: a large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 721–725. Cited by: §B.1, §2. Chung et al. (2018) S. Chung, D. D. Lee, and H. Sompolinsky Classification and geometry of general perceptual manifolds. Physical Review X 8 (3), p. 031003. Cited by: §1, §7. Cohen et al. (2020) U. Cohen, S. Chung, D. D. Lee, and H. Sompolinsky Separability and geometry of object manifolds in deep neural networks. Nature communications 11 (1), p. 746. Cited by: §1, §7. Crammer and Singer (2001) K. Crammer and Y. Singer On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research 2 (Dec), p. 265–292. Cited by: §A.1. De Marneffe et al. (2021) M. De Marneffe, C. D. Manning, J. Nivre, and D. Zeman Universal dependencies. Computational linguistics 47 (2), p. 255–308. Cited by: §B.1. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: §B.1, §2, §4. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §B.2. Dinkel et al. (2024) H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang CED: consistent ensemble distillation for audio tagging. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 291–295. Cited by: §B.2. Dosovitskiy et al. (2020) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §B.2. Eldan and Li (2023) R. Eldan and Y. Li Tinystories: how small can language models be and still speak coherent english?. arXiv preprint arXiv:2305.07759. Cited by: §B.1, §B.2, Figure 1, §2, §5.1. Elhage et al. (2022) N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652. Cited by: §1. Froudarakis et al. (2020) E. Froudarakis, U. Cohen, M. Diamantaki, S. Patel, Z. Tan, T. Muhammad, E. Y. Walker, J. Reimer, P. Berens, H. Sompolinsky, et al. Object manifold geometry across the mouse cortical visual hierarchy. BioRxiv, p. 2020–08. Cited by: §7. Galanti et al. (2021) T. Galanti, A. György, and M. Hutter On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121. Cited by: §1. Gao et al. (2025) L. Gao, T. Dupre la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations, Vol. 2025, p. 26721–26754. Cited by: §7. Gong et al. (2021) Y. Gong, Y. Chung, and J. Glass Ast: audio spectrogram transformer. arXiv preprint arXiv:2104.01778. Cited by: §B.2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §B.2. Han et al. (2021) X. Han, V. Papyan, and D. L. Donoho Neural collapse under mse loss: proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073. Cited by: §1. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778. Cited by: §B.2. Hendrycks et al. (2021) D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916. Cited by: §1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §1, §7. Honnibal et al. (2020) M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd spaCy: Industrial-strength Natural Language Processing in Python. External Links: Document Cited by: §B.1. Huben et al. (2024) R. Huben, H. Cunningham, L. Smith, A. Ewart, and L. Sharkey Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, Vol. 2024, p. 7827–7845. Cited by: §1, §7. Kornblith et al. (2018) S. Kornblith, J. Shlens, and Q. V. Le Do better imagenet models transfer better?. arXiv preprint arXiv:1805.08974. Cited by: §1, §7. Kuoch et al. (2024) M. Kuoch, C. Chou, N. Parthasarathy, J. Dapello, J. J. DiCarlo, H. Sompolinsky, and S. Chung Probing biological and artificial neural networks with task-dependent neural manifolds. In Conference on Parsimony and Learning, p. 395–418. Cited by: §7. Lachenbruch and Mickey (1968) P. A. Lachenbruch and M. R. Mickey Estimation of error rates in discriminant analysis. Technometrics 10 (1), p. 1–11. Cited by: §2. Liu et al. (2019a) N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 1073–1094. Cited by: §6. Liu et al. (2019b) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §B.2. Liu et al. (2022) Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie A convnet for the 2020s. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 11966–11976. Cited by: §B.2. Luthra et al. (2026) A. Luthra, Y. Salunkhe, and T. Galanti Directional neural collapse explains few-shot transfer in self-supervised learning. arXiv preprint arXiv:2603.03530. Cited by: §1. Makhzani and Frey (2013) A. Makhzani and B. Frey K-sparse autoencoders. arXiv preprint arXiv:1312.5663. Cited by: §7. Mensink et al. (2013) T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka Distance-based image classification: generalizing to new classes at near-zero cost. IEEE transactions on pattern analysis and machine intelligence 35 (11), p. 2624–2637. Cited by: §1. Nussbaum et al. (2024) Z. Nussbaum, J. X. Morris, B. Duderstadt, and A. Mulyar Nomic embed: training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613. Cited by: §B.2. OLMo et al. (2024) T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. Cited by: §B.2. Oquab et al. (2023) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §B.2. Papyan et al. (2020) V. Papyan, X. Y. Han, and D. L. Donoho Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40), p. 24652–24663. Cited by: §1, §4.1. Peters et al. (2018) M. E. Peters, M. Neumann, L. Zettlemoyer, and W. Yih Dissecting contextual word embeddings: architecture and representation. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 1499–1509. Cited by: §6. Qwen et al. (2025) Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §B.2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §B.2. Schrimpf et al. (2020) M. Schrimpf, J. Kubilius, M. J. Lee, N. A. R. Murty, R. Ajemian, and J. J. DiCarlo Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron 108 (3), p. 413–423. Cited by: §7. Siméoni et al. (2025) O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104. Cited by: §B.2, §4. Skean et al. (2025) O. Skean, M. R. Arefin, D. Zhao, N. Patel, J. Naghiyev, Y. LeCun, and R. Shwartz-Ziv Layer by layer: uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013. Cited by: §6. Snell et al. (2017) J. Snell, K. Swersky, and R. Zemel Prototypical networks for few-shot learning. Advances in neural information processing systems 30. Cited by: §1, §1, §2. Sorscher et al. (2022) B. Sorscher, S. Ganguli, and H. Sompolinsky Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences 119 (43), p. e2200800119. Cited by: §1, §7. Stephenson et al. (2021) C. Stephenson, S. Padhy, A. Ganesh, Y. Hui, H. Tang, and S. Chung On the geometry of generalization and memorization in deep neural networks. arXiv preprint arXiv:2105.14602. Cited by: §1. Templeton et al. (2026) A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, et al. Scaling monosemanticity: extracting interpretable features from claude 3 sonnet. arXiv preprint arXiv:2605.29358. Cited by: §1, §7. Tenney et al. (2019) I. Tenney, D. Das, and E. Pavlick BERT rediscovers the classical nlp pipeline. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 4593–4601. Cited by: §6. Wakhloo et al. (2023) A. J. Wakhloo, T. J. Sussman, and S. Chung Linear classification of neural manifolds with correlated variability. Physical Review Letters 131 (2), p. 027301. Cited by: §1, §7. Warner et al. (2025) B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, et al. Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: Long papers), p. 2526–2547. Cited by: §B.2. Yamins et al. (2014) D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences 111 (23), p. 8619–8624. Cited by: §7. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §B.2, Figure 1, §5.1. Appendix A Prototype Classification A.1 Prototype classification and linear readout comparison One natural concern regarding our work is that it is applied only to prototype classification, with no relation to the more common linear readout. We test this by splitting the data into 90%90\% train and 10%10\% validation, training a max-margin SVM (using the Crammer–Singer method [8]) on the train data, and evaluating on the validation set. We compare it with computing the centroids over the 90%90\% train representations, and computing the prototype scores on the 10%10\%. The gap between the two methods is small (see Fig.5 and Table 5), especially in vision (0.047) and autoregressive models (0.063). We note that in the bidirectional language models, the gap is substantially larger than in the other modalities (0.121), which is also reflected in a lower per-class correlation between the two classifiers (r=0.58r=0.58, against 0.870.87 in vision and 0.760.76 in the autoregressive models). The additional structure that allows the linear readout to separate the data better than the centroids in these models remains an open question. Figure 5: Centroid vs trained linear-readout accuracy, per backbone. The gap between methods is small (see also Table 5). Backbone r Δ Vision (ImageNet-1K) ViT-B/16 (IN21K) 0.9370.937 −0.005-0.005 ConvNeXt-B (IN1K) 0.8930.893 −0.001-0.001 DINOv3 ViT-L/16 0.8850.885 +0.028+0.028 DINOv3 ViT-H+/16 0.8730.873 +0.028+0.028 DINOv3 ConvNeXt-S 0.8720.872 +0.058+0.058 CLIP ViT-L/14 0.8690.869 +0.057+0.057 DINOv3 ConvNeXt-T 0.8650.865 +0.076+0.076 DINOv3 ViT-S+/16 0.8640.864 +0.084+0.084 ResNet-50 0.8610.861 +0.016+0.016 DINOv3 ViT-S/16 0.8590.859 +0.100+0.100 DINOv3 ViT-B/16 0.8570.857 +0.058+0.058 DINOv3 ConvNeXt-B 0.8560.856 +0.054+0.054 DINOv3 ConvNeXt-L 0.8540.854 +0.052+0.052 DINOv3 ViT-7B/16 0.8390.839 +0.030+0.030 DINOv2 ViT-g/14 0.7910.791 +0.074+0.074 mean 0.8650.865 +0.047+0.047 Audio (VGGSound) SSLAM 0.8210.821 +0.084+0.084 AST 0.8150.815 +0.091+0.091 CED 0.8080.808 +0.088+0.088 mean 0.8150.815 +0.088+0.088 Backbone r Δ Masked LM (TinyStories, masked-word) ModernBERT-large 0.5840.584 +0.126+0.126 NomicBERT 0.5810.581 +0.159+0.159 BERT-large 0.5680.568 +0.102+0.102 RoBERTa-large 0.5650.565 +0.096+0.096 mean 0.5750.575 +0.121+0.121 Autoregressive LM (TinyStories, next-token) Qwen3-4B 0.7850.785 +0.070+0.070 Qwen3-1.7B 0.7770.777 +0.089+0.089 Qwen3-8B 0.7720.772 +0.050+0.050 Qwen2.5-0.5B 0.7690.769 +0.102+0.102 Qwen3-0.6B 0.7650.765 +0.122+0.122 Llama-3.1-8B 0.7640.764 +0.041+0.041 Qwen2.5-3B 0.7620.762 +0.076+0.076 Qwen3-32B 0.7610.761 +0.019+0.019 Qwen3-14B 0.7540.754 +0.038+0.038 OLMo-2-13B 0.7500.750 +0.029+0.029 Qwen2.5-1.5B 0.7490.749 +0.102+0.102 Qwen2.5-32B 0.7470.747 +0.036+0.036 Qwen2.5-7B 0.7400.740 +0.052+0.052 Qwen2.5-14B 0.7360.736 +0.050+0.050 mean 0.7590.759 +0.063+0.063 Overall 0.7870.787 +0.065+0.065 Table 5: Per-class Pearson correlation r between trained linear-readout and prototype-classifier accuracy, together with the mean accuracy gap ΔAcc (readout minus prototype) on the held-out split. The two classifiers align closely, with the exception of masked language models (see Sec. A.1). Appendix B Experimental Setup The main text evaluates frozen representations from pretrained backbones spanning vision, language, and audio, across transformer and convolutional architectures, supervised, self-supervised, and contrastive objectives. This appendix gives the full list of datasets, models, and extraction details. No backbone is fine-tuned or changed for our analysis. B.1 Datasets Vision-ImageNet-1K: We use ImageNet-1K (ILSVRC 2012) [10], the standard 10001000-class image benchmark with ≈1.28≈\!1.28M images. Language-TinyStories: For language, the task is token prediction on the validation dataset of TinyStories [14], a corpus of 20K short synthetic children’s stories. We treat token prediction as a multi-class classification problem. Content words and lemmatizing: In order to get semantically meaningful representations, we keep only data points of content words. Following [2], content words are those whose part-of-speech is an open lexical class of the Universal Dependencies scheme [9]: noun, propn, verb, adj, or adv. We assign part-of-speech tags and lemmas with the spaCy pipeline [25], and unify all tokens sharing a lemma into a single class (for example, "say, said, says" or "friend, friends"). We keep only classes with at least 10001000 occurrences, so that the manifold geometry is well estimated. Finally, to obtain an identical class set across models with different tokenizers, we discard the 1111 lemmas that are not realized consistently in every tokenizer - fragmented or unified into/from sub-words differently in at least one model ("benny, bunny, fire, frog, grandma, grow, mia, mommy, sweet, window, yummy", for example one tokenizer treats "fireman"↦\, \,"fire+man"). This leaves 320320 classes shared by all backbones. Masked-word task (bidirectional encoders). We mask a single content-word position at a time, run a full bidirectional forward pass, and read out the encoder’s representation at the masked position. The representation is the final hidden state feeding the model’s masked-LM head. The class label is the lemma of the masked word, restricted to the same open-class content-word construction as above. Next-token task (autoregressive models): From a standard causal forward pass, the representation is the hidden state (at a specified layer) at the last token preceding the predicted word, and the class label is the lemma of the actual next word, restricted to the same open-class content-word construction as above. Audio - VGGSound. For audio we use VGGSound [5], a large-scale dataset of ∼10 \!10-second clips sourced from web video. We evaluate on the train split: 310310 single-label classes and 183,630183,630 clips (∼592 592 for each class), decoded to 1616 kHz mono. B.2 Models The canonical roster comprises 3636 pretrained backbones across the four modality groups (Table 6). Vision contains the self-supervised DINOv3 family [44]: DINOv3 ViT-S/16 through ViT-7B/16 and DINOv3 ConvNeXt-T through ConvNeXt-L, together with the self-supervised DINOv2 ViT-g/14 [38], contrastive CLIP ViT-L/14 [42], the supervised ImageNet-21K ViT-B/16 [13], the supervised ImageNet-1K ConvNeXt-B [32], and the supervised ResNet-50 [22]. The language backbones are evaluated on the TinyStories dataset [14]. The masked-word encoders are RoBERTa-large [31], BERT-large [11], ModernBERT-large [52], and NomicBERT [36]. The next-token autoregressive models are the Qwen2.5-Instruct family (0.50.5B–3232B) [41], the Qwen3 family (0.60.6B–3232B) [54], Llama-3.1-8B [20], and OLMo-2-13B [37]. The audio backbones are SSLAM [1], an EAT-style self-supervised model fine-tuned on AudioSet-2M; CED-base [12], an AudioSet-supervised distilled transformer; and AST [19], an AudioSet-supervised spectrogram transformer. Table 6: Model roster (3636 backbones). N is the feature dimension of the extracted representation; P is the number of classes in the task. Family Model N Data / task P Objective Vision — ImageNet-1K Vision DINOv3 ViT-S/16 384384 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ViT-S+/16 384384 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ViT-B/16 768768 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ViT-L/16 10241024 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ViT-H+/16 12801280 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ViT-7B/16 40964096 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ConvNeXt-T 768768 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ConvNeXt-S 768768 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ConvNeXt-B 10241024 ImageNet-1K 10001000 Self-supervised Vision DINOv3 ConvNeXt-L 15361536 ImageNet-1K 10001000 Self-supervised Vision DINOv2 ViT-g/14 15361536 ImageNet-1K 10001000 Self-supervised Vision CLIP ViT-L/14 768768 ImageNet-1K 10001000 Contrastive (image-text) Vision ViT-B/16 (IN21K) 768768 ImageNet-1K 10001000 Supervised (IN21K) Vision ConvNeXt-B (IN1K) 10241024 ImageNet-1K 10001000 Supervised (IN1K) Vision ResNet-50 20482048 ImageNet-1K 10001000 Supervised (IN1K) Audio — VGGSound Audio SSLAM 768768 VGGSound 310310 Self-supervised + AudioSet-2M FT Audio CED-base 768768 VGGSound 310310 Supervised distillation (AudioSet) Audio AST 768768 VGGSound 310310 Supervised (AudioSet) Masked LM — TinyStories masked-word Masked LM RoBERTa-large 10241024 TinyStories (masked) 320320 Masked language modeling Masked LM BERT-large 10241024 TinyStories (masked) 320320 Masked language modeling Masked LM ModernBERT-large 10241024 TinyStories (masked) 320320 Masked language modeling Masked LM NomicBERT 768768 TinyStories (masked) 320320 Masked language modeling Autoregressive LM — TinyStories next-token Autoregressive LM Qwen2.5-0.5B 896896 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen2.5-1.5B 15361536 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen2.5-3B 20482048 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen2.5-7B 35843584 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen2.5-14B 51205120 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen2.5-32B 51205120 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen3-0.6B 10241024 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen3-1.7B 20482048 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen3-4B 25602560 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen3-8B 40964096 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen3-14B 51205120 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Qwen3-32B 51205120 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM Llama-3.1-8B 40964096 TinyStories (next-token) 320320 Causal language modeling Autoregressive LM OLMo-2-13B 51205120 TinyStories (next-token) 320320 Causal language modeling B.3 Representation extraction For vision transformers, we take the class token of the final layer as the representation. For convolutional architectures (ConvNeXt, ResNet), we take the global-average-pooled feature vector of the final stage. For audio backbones, we take the model’s pre-classifier pooled embedding: SSLAM its fc-norm of the class token, CED-base the mean over its final-layer hidden states, AST its standard pre-classifier pooled outputs. For masked language models the representation is the final hidden state at the masked position — the input to the masked-LM head. For autoregressive language models the representation is the hidden state (in a specific layer, see Sec. G) of the last token preceding the predicted word. In every case, the feature dimensions N reported in Table 6 are those of the stored arrays. Before any analysis, we z-score each representation dimension-wise, using the empirical mean and standard deviation of the data. B.4 Resources Feature extraction ran on GPU (A100 and H100); the numerical integration of the theory and the fitting of the rescaling factor λ ran on CPU. The largest extractions were Qwen-14B/32B and OLMo on language, and DINOv3 ViT-7B on vision. Appendix C Theoretical Proofs This appendix contains all material deferred from Sec. 3 and Sec. 5. Appendix C.1 works out the isotropic uncorrelated model and derives the corollaries that show how the model fails qualitatively on real data. Appendix C.2 gives the full proofs of the centroid-aligned theory used in the main text. C.1 Isotropic model We analyze the maximally unstructured baseline, in which class centroids and within-class fluctuations are both isotropic Gaussian. It retains only the ambient dimension N, the number of classes P, and the noise-to-signal scale R, discarding every correlation between centroids and fluctuations. We derive its exact large-N accuracy, show that accuracy is self-averaging, and extract corollaries delineating its regimes. In Section 4 these predictions are shown to match real representations only after the residual-centroid correlations are removed. Assumption C.1.1 (Isotropic uncorrelated model). For each class μ=1,…,Pμ=1,…,P the centroid is drawn μ∼iid(,IN)c_μ _iidN(0,I_N) and held fixed (quenched). Conditioned on kc_k, an example of class k is =k+δx=c_k+ with δ∼(,R2IN) (0,R^2I_N), sampled independently across examples. Assumption C.1.2 (High-dimensional regime). N→∞N→∞, the number of classes grows at most polynomially, P=N(1)P=N^O(1), and the radius R=R(N)R=R(N) may scale with N. We refer to Assumptions C.1.1–C.1.2 jointly as the isotropic model. The classifier is the unit-centroid (prototype) rule y^()=argmaxμ^μ⋅ y(x)= _μ\, c_μ\!·\!x, with ^μ:=μ/∥μ∥ c_μ:=c_μ/ _μ , and its logits are tμ:=^μ⋅t_μ:= c_μ\!·\!x. Lemma C.1.3 (Overlaps). Let ,∼iid(,IN)u,v _iidN(0,I_N). Then, as N→∞N→∞, ∥/N→1 / N p1 and N(^⋅^)→(0,1) N\,( u\!·\! v) dN(0,1). Consequently, the dot product of the target centroid with the true and rival directions obeys ^μ⋅k=εμ+δμk(N−εμ),εμ∼iid(0,1). c_μ\!·\!c_k= _μ+ _μ k ( N- _μ ), _μ _iidN(0,1). (9) Proof. ⋅=∑iuiviu\!·\!v= _iu_iv_i is a sum of N i.i.d. centred unit-variance variables, so ⋅/N→(0,1)u\!·\!v/ N dN(0,1) by the CLT, while ∥/N→1 / N p1 by the law of large numbers. □ Theorem C.1.4 (Mean accuracy). Under the isotropic model, δ[Acc]¯=z∼(0,1)[ΦP−1(N+Rz1+R2)]. E_ [Acc]=E_z (0,1)\! [ ^P-1\! ( N+Rz 1+R^2 ) ]. (10) where δ[Acc]¯ E_δ x[Acc] denotes the average over the quenched disorder μ\c_μ\ after averaging over the thermal δ . Proof. Conditional on the example and centroids, the accuracy is the probability that the correct logit is the largest, Acc|δ,ν=∫−∞dtkP(tk)∏μ≠k∫−∞tkdtμP(tμ),P(tμ)=δ(tμ−^μ⋅).Acc\, |\, ,\c_ν\= _-∞^∞\!dt_k\,P(t_k) _μ≠ k _-∞^t_k\!dt_μ\,P(t_μ), P(t_μ)=δ\! (t_μ- c_μ\!·\!x ). (11) Writing each δ through its Fourier representation introduces conjugate variables t^μ t_μ: Acc|δ,ν=∫−∞dtkdt^k2π∏μ≠k∫−∞tkdtμdt^μ2πexp(i∑μ=1Pt^μ(tμ−^μ⋅(k+δ))).Acc\, |\, ,\c_ν\= _-∞^∞\! dt_k\,d t_k2π _μ≠ k _-∞^t_k\! dt_μ\,d t_μ2π \! (i _μ=1^P t_μ (t_μ- c_μ\!·\!(c_k+ ) ) ). (12) Carrying out the Gaussian δ and t^μ t_μ integrals, δx[Acc|μ]=∫−∞dtk2πR2∏μ≠k∫−∞tkdtμ2πR2exp(−12R2∑μ=1P(tμ−^μ⋅k)2).E_δ x\! [Acc\, |\,\c_μ\ ]= _-∞^∞\! dt_k 2π R^2 _μ≠ k _-∞^t_k\! dt_μ 2π R^2 \! (- 12R^2 _μ=1^P (t_μ- c_μ\!·\!c_k )^2 ). (13) Lemma C.1.3 sets ^μ⋅k=εμ+δμk(N−εμ) c_μ\!·\!c_k= _μ+ _μ k( N- _μ). The target mean concentrates at N N, while integrating each rival overlap εμ∼(0,1) _μ (0,1) inflates the rival variance R2→1+R2R^2→ 1+R^2: Acc¯=∫−∞dtk2πR2∏μ≠k∫−∞tkdtμ2π(1+R2)exp(−(tk−N)22R2−12∑μ≠ktμ21+R2). Acc= _-∞^∞\! dt_k 2π R^2 _μ≠ k _-∞^t_k\! dt_μ 2π(1+R^2) \! (- (t_k- N)^22R^2- 12 _μ≠ k t_μ^21+R^2 ). (14) The rival integrals factorize into Φ(tk/1+R2) \! (t_k/ 1+R^2 ); substituting tk=N+Rzt_k= N+Rz yields Eq. 10. □ The average Eq. 10 is taken over both variability δδ x and centroids; the next result shows that the accuracy is self-averaging, such that taking the annealed average over the centroids disorder is valid. Proposition C.1.5 (Self-averaging). Under the isotropic model, the accuracy is self-averaging. Var(Acc)=(1/(PR2))→0Var(Acc)=O\! (1/(PR^2) )→ 0. Proof. We bound the variance of the disorder-averaged accuracy. Introduce two independent replicas ,′x,x of class k sharing the centroids. As in Eq.13, we get, δx[Acc2|μ] _δ x\! [Acc^2\, |\,\c_μ\ ] =∫−∞dtkdtk′2πR2∏μ≠k∫−∞tkdtμ2πR2∫−∞tk′dtμ′2πR2 = _-∞^∞\! dt_k\,dt_k 2π R^2 _μ≠ k _-∞^t_k\! dt_μ 2π R^2 _-∞^t_k \! dt_μ 2π R^2 =exp(−12R2∑μ[(tμ−^μ⋅k)2+(tμ′−^μ⋅k)2]). = \! (- 12R^2 _μ\! [(t_μ- c_μ·c_k)^2+(t_μ - c_μ·c_k)^2 ] ). (15) Averaging over the centroids couples the replicas through the shared mean. The integral becomes bivariate normal with variance 1+R21+R^2 and covariance 11, Acc2¯=∫dtkdtk′2πR2∏μ≠k1∫−∞tk+Ndtμ∫−∞tk′+Ndtμ′exp(−tk2+tk′22R2−(1+R2)(tμ2+tμ′2)−2tμtμ′2R2(R2+2)), Acc^2= \! dt_k\,dt_k 2π R^2 _μ≠ k 1Z _-∞^t_k+ Ndt_μ _-∞^t_k + Ndt_μ \! (- t_k^2+t_k 22R^2- (1+R^2)(t_μ^2+t_μ 2)-2t_μt_μ 2R^2(R^2+2) ), (16) with the normalization =2πRR2+2Z=2π R R^2+2. The regime R=(1)R=O(1): The target shift N→∞ N→∞ pushes the rival limits to +∞+∞, the rival integrals yield 11 up to an exponentially small correction whenever lnP=o(N) P=o(N), and Acc2¯=Acc¯2=1 Acc^2= Acc^2=1, so Var→0Var→ 0. The regime R∼NR N: Set r=R/Nr=R/ N and rescale. The inter-replica coupling is weak, of order 1/(Nr2)1/(Nr^2), and we can expand in the coupling. Solving the uncoupled Gaussian integrals yields: Acc2¯ Acc^2 =z,z′[(Φ(z+1r)Φ(z′+1r)+1Nr2ϕ(z+1r)ϕ(z′+1r)+(N−2))P−1] =E_z,z \! [ ( (z+ 1r) (z + 1r)+ 1Nr^2φ(z+ 1r)φ(z + 1r)+O(N^-2) )^P-1 ] (17) =Acc¯2+P−1Nr2(z[ΦP−2(z+1r)ϕ(z+1r)])2+(lnPN2P2). = Acc^2+ P-1Nr^2 (E_z [ ^P-2(z+ 1r)φ(z+ 1r) ] )^2+O\! ( PN^2P^2 ). where ϕ(x)φ(x) is a standard normal PDF, and Φ(x) (x) its CDF. The expectation can be performed using the Laplace method, yielding the classic extreme-value extremum z∗+1/r=2lnPz +1/r= 2 P, ϕ(z∗+1/r)=2lnP/Pφ(z +1/r)= 2 P/P, and the width of the Laplace region Δz=1/(2lnP) z=1/ (2 P ). We note that the higher binomial terms are suppressed not by the 1/N1/N prefactors but by the Gaussian PDF densities ϕφ, each one of the order 2lnP/P 2 P/P , which cancel the binomial growth. Var(Acc)=P−1Nr2(z[ΦP−2(z+1r)ϕ(z+1r)])2=(1PNr2)=(1PR2)→0Var(Acc)= P-1Nr^2 (E_z [ ^P-2(z+ 1r)φ(z+ 1r) ] )^2=O( 1PNr^2)=O( 1PR^2)→ 0 (18) In both regimes Var(Acc)→0Var(Acc)→ 0. □ The mean accuracy Eq.10 has four immediate consequences, delineating the regimes of the isotropic model. Corollary C.1.6 (Separable regime). For R=(1)R=O(1) the classes are well separated and classification is trivial: Acc→1Acc^→ 1. Proof. The scale of the maximum of P i.i.d. Gaussians with variance σ2=1+R2σ^2=1+R^2 converges to σ2lnPσ 2 P for large P. The signal is [tk]=NE[t_k]= N. From Assumption C.1.2, lnP=(lnN) P=O( N) thus for any R=(1)R=O(1), σ2lnP≪Nσ 2 P N which yields Acc→1Acc→ 1. □ Corollary C.1.7 (Saturation regime). In the infinite-radius limit R/N→∞R/ N→∞ the isotropic residual dominates the input, which retains no information about the true class: all P classes are equally likely and accuracy collapses to chance, Acc→1/PAcc^→ 1/P. Proof. Dividing through by R, the argument of Φ in Eq.10 tends to z pointwise, so bounded convergence gives Acc→z[ΦP−1(z)]=ℙ(ξk>maxμ≠kξμ)=1/PAcc _z[ ^P-1(z)]=P( _k> _μ≠ k _μ)=1/P for ξ1,…,ξP∼iid(0,1) _1,…, _P _iidN(0,1), by permutation symmetry. □ Corollary C.1.8 (Critical radius). Non-perfect accuracy requires large radii: Acc<1⇒R=Ω(N/lnP)Acc^<1 R= ( N/ P). Proof. Contrapositive. If R=o(N/lnP)R=o( N/ P) then 1+R2=o(N/lnP)1+R^2=o(N/ P), so the argument of Φ in Eq.10 is (N+Rz)/1+R2=ω(lnP)≫2lnP( N+Rz)/ 1+R^2=ω( P) 2 P, and Corollary C.1.6 forces Acc→1Acc^→ 1. □ Corollary C.1.9 (Per-class fluctuations). Per-class accuracy fluctuations are of order 1/RP1/R P. Proof. Calculation of the variance in Proposition C.1.5 yields Var(Acc)=(1/PR2)Var(Acc)=O(1/PR^2). Different classes are equivalent to a resampling of the centroids, so per-class fluctuations are of the same order. □ C.2 Centroid-aligned model Definition C.2.1 (Centroid geometry). Fix the true class k. For μ,ν∈1,…,Pμ,ν∈\1,…,P\ let Gμν:=^μ⋅^νG_μν:= c_μ\!·\! c_ν be the Gram matrix of the unit centroids, and let gμ:=^k⋅^μg_μ:= c_k\!·\! c_μ be the centroid overlap of class μ with the true class, so that gk=1g_k=1. The rival support is ℛKR_K and ℛK+:=k∪ℛKR^+_K:=\k\ _K, as defined in Sec. 5. Assumption C.2.2 (Deterministic centroids and Gaussian projections). We assume that the centroids μ=1P\c_μ\_μ=1^P are deterministic and held fixed, and that the restricted Gram matrix of ℛK+R^+_K is invertible. The source of stochasticity in the system is the sampling of Gaussian projection coefficients sμ=1P\s_μ\_μ=1^P and s⟂s__ . Definition C.2.3 (Centroid-aligned generative model). An example of class k is generated by ()=‖k‖[^k+R(∑μ,ν∈ℛK+σμsμGμν−1^ν+σ⟂s⟂^⟂)],x(s)= \|c_k \| [ c_k+R ( _μ,ν ^+_K\!\! _μs_μ\,G^-1_μν c_ν+ __ s__ e__ ) ], (19) where ^⟂ e__ is orthogonal to span(^μ)μ∈ℛK+span( c_μ)_μ ^+_K, R is the empirical class radius, and σμ _μ is the projected standard deviation along ^μ c_μ. Definition C.2.4 (Normalized logits). The prototype logits of (Def 2.1) are defined as tμ=^μ⋅t_μ= c_μ\!·\!x. Throughout this section it is convenient to divide all of them by the common factor ∥k∥ _k , and we set tμ:=⋅^μ∥k∥,μ∈ℛK+.t_μ:= x\!·\! c_μ _k , μ ^+_K. (20) Since ∥k∥ _k is a common factor independent of μ, the argmaxargmax properties of the logits do not change. The classifier and its accuracy are therefore those of Definition 2.1, and no generality is lost. We restrict the logits only to the sparse set of rivals ℛKR_K, and neglect weak competitors. Lemma C.2.5 (Logit representation). Under Assumption C.2.2 and Definitions C.2.3–C.2.4, tμ=gμ+Rσμsμt_μ=g_μ+R\, _μs_μ for every μ∈ℛK+μ ^+_K. Proof. Taking the inner product of Eq. 19 with ^μ c_μ and dividing by ∥k∥ _k gives tμ=gμ+R(∑α,ν∈ℛK+σαsαGαν−1Gνμ+σ⟂s⟂^⟂⋅^μ)t_μ=g_μ+R ( _α,ν ^+_K _αs_αG^-1_ανG_νμ+ __ s__ \, e__ \!·\! c_μ ). The last term vanishes since ^⟂^μ e__ c_μ for μ∈ℛK+μ ^+_K, and ∑νGαν−1Gνμ=δαμ _νG^-1_ανG_νμ= _αμ by Assumption C.2.2. □ Definition C.2.6 (Projection covariance). The coefficients are centered and standardized by definition, k[sμ]=0E_k[s_μ]=0 and k[sμ2]=1E_k[s^2_μ]=1. We write k[sμsν]=ΣμνR2σμσν,μ,ν∈ℛK+.E_k[s_μs_ν]= _μνR^2 _μ _ν, μ,ν ^+_K. (21) By Lemma C.2.5 this makes Σμν=Covk[tμ,tν] _μν=Cov_k[t_μ,t_ν] the covariance matrix of the logit vector itself. No structure is imposed on Σ beyond that of a well-conditioned covariance matrix. Proposition C.2.7 (Accuracy as an orthant probability). Under Lemma C.2.5, Definition C.2.6 and Assumption C.2.2, Accth(R)=1detΣ∫−∞dtk2π∏μ∈ℛK∫−∞tkdtμ2πexp(−12∑μ,ν∈ℛK+(tμ−gμ)Σμν−1(tν−gν)).Acc^th(R)= 1 _-∞^∞\! dt_k 2π _μ _K _-∞^t_k\! dt_μ 2π\, \! (- 12 _μ,ν ^+_K (t_μ-g_μ ) ^-1_μν (t_ν-g_ν ) ). (22) Proof. By Lemma C.2.5, Definition C.2.6 and Assumption C.2.2 the vector (tμ)μ∈ℛK+(t_μ)_μ ^+_K is Gaussian with mean (gμ)(g_μ) and covariance Σ . An example is classified correctly iff tμ<tkt_μ<t_k for all μ∈ℛKμ _K. □ Eq. 22 is a multivariate orthant probability, which in general can be evaluated only numerically (for example by Monte-Carlo simulation). A specific structure admits an analytical solution. Lemma C.2.8 (Gaussian conditioning). Under Assumption C.2.2, set ρμ:=k[sμsk] _μ:=E_k[s_μs_k] for μ∈ℛKμ _K. Then for all μ,ν∈ℛKμ,ν _K, k[tμ∣tk]=gμ+ρμσμtk−1σk,Cov[tμ,tν∣tk]=R2σμσν(k[sμsν]−ρμρν),E_k\! [t_μ t_k ]=g_μ+ _μ _μ t_k-1 _k, \! [t_μ,t_ν t_k ]=R^2 _μ _ν (E_k[s_μs_ν]- _μ _ν ), (23) and in particular the conditional covariance does not depend on tkt_k. Proof. By Definition C.2.6, Cov(tμ,tν)=ΣμνCov(t_μ,t_ν)= _μν, Σμk=R2σμσkρμ _μ k=R^2 _μ _k _μ and Σkk=R2σk2 _k=R^2σ^2_k, while k[tμ]=gμE_k[t_μ]=g_μ and k[tk]=1E_k[t_k]=1. For a jointly Gaussian vector, [tμ∣tk]=[tμ]+ΣμkΣkk−1(tk−[tk])E[t_μ t_k]=E[t_μ]+ _μ k _k^-1(t_k-E[t_k]) and Cov[tμ,tν∣tk]=Σμν−ΣμkΣνkΣkk−1Cov[t_μ,t_ν t_k]= _μν- _μ k _ν k _k^-1. Substituting gives Eq. 23. □ Assumption C.2.9 (Conditional independence). Conditionally on tkt_k the rival logits are uncorrelated: Cov[tμ,tν∣tk]=0Cov[t_μ,t_ν t_k]=0 for all μ≠νμ≠ν in ℛKR_K. Equivalently, by Lemma C.2.8, k[sμsν]=δμν(1−ρμ2)+ρμρν,so thatCov[tμ,tν∣tk]=δμνR2σμ2(1−ρμ2).E_k[s_μs_ν]= _μν (1-ρ^2_μ )+ _μ _ν, that \! [t_μ,t_ν t_k ]= _μνR^2σ^2_μ (1-ρ^2_μ ). (24) Theorem C.2.10 (Conditional factorization). Under Assumption C.2.9, Accth(R)=∫−∞dtkRσk2πexp(−(tk−1)22R2σk2)∏μ∈ℛKΦ(tk(1−ρμσμ/σk)+ρμσμ/σk−gμRσμ1−ρμ2).Acc^th(R)= _-∞^∞ dt_kR _k 2π (- (t_k-1)^22R^2 _k^2 ) _μ _K \! ( t_k (1- _μ _μ/ _k )+ _μ _μ/ _k-g_μR _μ 1-ρ^2_μ ). (25) Proof. Condition on tkt_k. By Lemma C.2.8 and Assumption C.2.9 the variables (tμ)μ∈ℛK(t_μ)_μ _K are then independent Gaussians with mean gμ+ρμσμ(tk−1)/σkg_μ+ _μ _μ(t_k-1)/ _k and variance R2σμ2(1−ρμ2)R^2σ^2_μ(1-ρ^2_μ), so the orthant probability factorizes into ∏μPr[tμ<tk∣tk] _μ [t_μ<t_k t_k], which yields Eq. 25. □ Corollary C.2.11 (Standardized form). Substituting tk=Rσkz+1t_k=R _kz+1 in Eq. 25 yields Acckth(R)=z∼(0,1)[∏μ∈ℛKΦ(σk−ρμσμσμ1−ρμ2z+1−gμRσμ1−ρμ2)],Acc^th_k(R)=E_z (0,1) [ _μ _K \! ( _k- _μ _μ _μ 1-ρ^2_μ\,z+ 1-g_μR _μ 1-ρ^2_μ ) ], (26) which is Theorem 5.5 of the main text. Proposition C.2.12 (Unconditional moments). Under Assumption C.2.9, for μ,ν∈ℛKμ,ν _K, k[tμ]=gμ,Σμν=Covk[tμ,tν]=R2(δμνσμ2(1−ρμ2)+σμσνρμρν),E_k\! [t_μ ]=g_μ, _μν=Cov_k\! [t_μ,t_ν ]=R^2 ( _μνσ^2_μ (1-ρ^2_μ )+ _μ _ν _μ _ν ), (27) which is a rank-one correction to a diagonal covariance. This is the covariance structure of s stated in Assumption 5.4 of the main text. Proof. From Lemma C.2.5, k[tμ]=gμE_k[t_μ]=g_μ and Σμν=R2σμσνk[sμsν] _μν=R^2 _μ _νE_k[s_μs_ν]. Inserting the covariance structure from Eq. 24 yields the result. □ Remark C.2.13 (Quenched versus deterministic centroids). Assumption C.2.2 treats the centroids as deterministic. Drawing them instead from a Gaussian ensemble whose second moments reproduce the overlap structure G does not change the result for large N: the induced fluctuations of the coefficients gμg_μ, σμ _μ and ρμ _μ entering Eq. 25 are subleading corrections to their (1)O(1) values, and the accuracy can be shown to be self-averaging by the same argument as in Proposition C.1.5. Appendix D Synthetic Data We validate the two generative models of the main text on synthetic data, where the geometry is controlled exactly and the analytic predictions can be checked without any of the confounds of real representations. Figure 6 tests the isotropic uncorrelated model of Section 3, and Figure 7 tests the centroid-aligned model of Section 5. For each model we verify three claims: (i) the analytic accuracy matches the simulation; (i) the accuracy is self-averaging, i.e. a single quenched centroid bank gives the same per-class accuracy as a fresh, independently drawn set of rivals for every example; and (i) the fluctuation of the accuracy across realizations shrinks with the number of classes P and for the isotropic model also with the class radius R as predicted by the previous section. Figure 6: Validation of the isotropic model’s theory on synthetic data. (a) Class-averaged accuracy versus R/NR/ N: simulation (markers, mean ± between-class std) against Theorem 3.3 (line), N=512N=512, P=500P=500. (b) Self-averaging: quenched per-class accuracy (one shared centroid bank) versus the fresh-rival accuracy (rivals redrawn per example), the line is y=xy=x. r=0.985r=0.985, confirming self-averaging - the thermal average is equivalent to the quenched average. (c) Across-realization variance of the accuracy versus P at fixed radius R=9.05R=9.05, fitted power law ∝P−1.00 P^-1.00, predicted by Proposition C.1.5. (d) Variance versus radius R (swept via the dimension at fixed R/NR/ N), fitted power law ∝R−1.96 R^-1.96, predicted by Proposition C.1.5. Figure 7: Synthetic validation of the centroid-aligned model. Examples are generated according to the centroid-aligned variability model (Sec. 5 and Sec. C.2) with K=20K=20 rivals, N=200N=200, P=500P=500. The geometry is calibrated to DINOv3 ViT-7B: own-class projected std σk=0.24 _k=0.24, rival stds σμ=0.12μ−0.4 _μ=0.12\,μ^-0.4. (a) Class-averaged accuracy versus class radius R: simulation (markers) against Theorem 5.5 (line). The dotted line is the R→∞R→∞ plateau 0.3160.316, which is set by σk/σμ _k/ _μ alone. (b) Quenched centroid versus fresh per-example sampling of the centroids, y=xy=x line, r=0.993r=0.993, validating self-averaging. (c) Across-realization variance versus P at fixed radius, fitted ∝P−0.97 P^-0.97, similar to the isotropic model (Proposition C.1.5). Appendix E Rival Selection For a given example, only a small number of the P classes are genuine competitors. This is the reason that the theory of Section 5 evaluates the accuracy over a small rival set ℛKR_K. This appendix (i) compares four possible rival-selection rules on convergence to the full P-way classification (i) shows that the theory’s error saturates by K≈10K≈ 10-2020, so K=20K=20 is a principled choice. E.1 Rival-selection rules Fix a class k. We define a ranking method for every candidate rival μ≠kμ≠ k: Centroid correlations: gμ=^k⋅^μg_μ= c_k· c_μ , (descending). Projection variance: σμ=k[(δ⋅^μ)2]/k[∥δ∥2] _μ= E_k\! [( · c_μ)^2 ] /E_k\! [ ^2 ], (descending). Theory margin: (1−gμ)/(Rσμ1−ρμ2)(1-g_μ)/ (R _μ 1-ρ^2_μ ), (ascending). The z-independent argument in Eq.7. Gaussian margin: (1−gμ)/σk2+σμ2−2σkσμρμ(1-g_μ)/ _k^2+ _μ^2-2\, _k _μ _μ, (ascending). The exact z-expectation over Eq.7 for one rival. E.2 Convergence of the K-way classification accuracy Restricting the classification to a smaller rival set can only improve accuracy, so the K-way accuracy exceeds the full P-way accuracy and decreases monotonically to it as K grows. Table 7 reports, for each selector, the median number of rivals a backbone needs before its K-way accuracy is within a fixed band of the full accuracy, aggregated over all 3636 backbones. The margin and variance selectors predict the P-way classification at the smallest K; the centroid-correlation and Gaussian-margin selectors need substantially more rivals at the tight 1%1\% band, confirming that ranking by projected variance (or by the conditional margin) is the most efficient way to choose the true competitors. The conclusion is consistent for all modalities. Since the projection variance and the conditional margin perform comparably (within five rivals at every band), we adopt the projection variance in the main text, as it is a single measured quantity and is therefore cheaper to compute and easier to interpret. E.3 Optimal K for theory predictions Increasing K enlarges the rival set the theory integrates over. Table 9 tracks the root-mean-square error between the predicted and empirical per-class accuracies as a function of K. The error drops steeply up to K≈10K≈ 10 and is essentially flat thereafter: at K=20K=20 the mean error is within 0.0030.003 of its K→300K\!→\!300 floor for 3131 of the 3636 backbones, and the median number of rivals needed to come within 0.0010.001 of the floor is 2020, the value we chose for K in the theory throughout. Table 7: Rivals needed to reach the full accuracy. Median number of rivals K (over 3636 backbones) at which the restricted K-way accuracy comes within 5/3/2/1%5/3/2/1\% of the full P-way accuracy, per selector. Smaller is better. Selector within 5%5\% within 3%3\% within 2%2\% within 1%1\% Centroid correlations (gμg_μ) 14 24 36 87 Projection variance (σμ _μ) 14 20 30 54 Theory margin 11 18 25 49 Gaussian margin 10 19 28 73 Table 8: Rivals needed within 1%1\%, by modality. Median K to reach within 1%1\% of the full accuracy, per selector and modality (n backbones). Modality n Centroids corr. Proj. variance Theory margin Gaussian margin Vision 15 76 47 45 56 Autoregressive LM 14 102 89 82 108 Masked LM 4 48 42 35 43 Audio 3 60 39 37 49 Table 9: The theory’s error saturates by K≈10K≈ 10–2020. Mean per-class accuracy RMSE of the theory at K=10K=10 and K=20K=20, its large-K floor, and the number of backbones already within 0.0030.003 of that floor (large K) at K=20K=20, by modality. Modality n RMSE K=10K=10 RMSE K=20K=20 floor within 0.0030.003 by K=20K=20 Vision 15 0.049 0.047 0.045 13/15 Audio 3 0.081 0.076 0.064 0/3 Masked LM 4 0.064 0.062 0.061 4/4 Autoregressive LM 14 0.080 0.075 0.075 14/14 All 36 0.065 0.062 0.060 31/36 Appendix F Heavy tail statistics and renormalization The theory of Sect. 5 treats the standardized centroid projections sμs_μ as jointly Gaussian. Empirically, the variables sμs_μ (see Table. 2 for the definition) have a pronounced right tail. Since a prediction is decided by the largest rival score maxμtμ _μt_μ, this positive tail controls the error rate, and it is exactly where a Gaussian underestimates the extreme value. Fitting the pooled positive projections gives a power-law p(s)∝(a2+s2)−ν/2p(s)\; \; (a^2+s^2 )^-ν/2 (28) obtained by a maximum-likelihood fit of a Student-t distribution of parameters a,νa,ν to the positive s values. Here ν is the tail (power-law) exponent, a smaller ν means a heavier tail, and ν→∞ν\!→\!∞ recovers a Gaussian. A heavier tail inflates the extreme rival projections that produce errors, so the Gaussian theory is over-optimistic; the single global rescaling R→λR→λ R used in Section 5 (fit per model) compensates for this by enlarging the effective within-class radius, and the fitted λ tracks the tail heaviness 1/ν1/ν (see Fig.4 and Table 11). Figure 8 contrasts the density of s across six backbones spanning three modalities, ordered from the lightest to the heaviest tail, each with its fitted curve (28). In Sec. 5.2 we claim that without the additional rescaling of the variability by λ, the theory is overoptimistic. This is demonstrated explicitly in Table 10. The theory with λ=1λ=1 consistently predicts accuracy high by 6%6\% on average across all backbones, compared to only 0.5%0.5\% with λ. Figure 8: Standardized centroid projections have a heavy right tail that varies across models. Density of the standardized projection s (filled) with its fitted half-Student-t right tail p(s)∝(a2+s2)−ν/2p(s) (a^2+s^2)^-ν/2 (solid), K=20K=20. Panels run light → heavy tail (left to right, top to bottom), with the fitted exponent ν and model in each title. Color encodes modality (vision blue, autoregressive LM orange, masked LM green). As ν decreases, the peak sharpens, and the right tail fattens. Table 11 lists the tail exponent ν for all 3636 backbones, grouped by modality and sorted from heaviest to lightest tail, alongside the rescaling λ. The exponent varies substantially, from ν≈3.7ν≈ 3.7 (DINOv2 ViT-g, the heaviest tail on the roster) to ν≈22ν≈ 22 (Qwen3-0.6B, nearly Gaussian); the vision backbones span the widest range, while the audio and language backbones have larger ν (lighter tail) and less variability between models. Across the roster the heaviness 1/ν1/ν correlates strongly with the fitted rescaling λ (Pearson r=0.90r=0.90 over the 3636 backbones): the heaviest-tail models require the largest correction (λ up to 1.341.34) and the near-Gaussian ones need almost none. The main text shows this λ-versus-1/ν1/ν relationship as a scatter (Fig. 4). Table 10: Without global radius rescaling, the theory is over-optimistic. Per-class accuracy predicted by the centroid-aligned theory of Sec. 5.2, evaluated at λ=1λ=1 (no rescaling of the variability) and at the fitted λ, compared with the empirical accuracy. The theoretical accuracy with λ=1λ=1 is higher than the empirical one, while the theory with λ is within 1%1\% of the empirical one, with significantly better RMSE. λ=1λ=1 fitted λ Modality n ⟨Acc⟩ ⟨Accth⟩ ^th RMSE ⟨Accth⟩ ^th RMSE ⟨λ⟩ λ Vision 15 0.812 0.876 0.080 0.819 0.047 1.228 Audio 3 0.613 0.657 0.087 0.617 0.076 1.084 Masked LM 4 0.715 0.785 0.090 0.724 0.062 1.131 Autoregressive LM 14 0.522 0.583 0.096 0.524 0.075 1.111 All 36 0.672 0.734 0.088 0.677 0.062 1.160 Backbone ν λ Vision (ImageNet-1K) DINOv2 ViT-g 3.73.7 1.2721.272 ConvNeXt-B 4.04.0 1.2541.254 ViT-B 4.14.1 1.3441.344 DINOv3 CNX-L 4.44.4 1.3051.305 DINOv3 ViT-7B 4.54.5 1.3161.316 DINOv3 CNX-B 4.54.5 1.2961.296 ResNet50 4.64.6 1.2281.228 DINOv3 CNX-S 5.25.2 1.2291.229 DINOv3 ViT-H+ 5.35.3 1.2971.297 DINOv3 CNX-T 6.86.8 1.1481.148 DINOv3 ViT-L 7.07.0 1.2211.221 DINOv3 ViT-B 9.09.0 1.1581.158 DINOv3 ViT-S+ 14.414.4 1.1341.134 CLIP-L 16.416.4 1.1081.108 DINOv3 ViT-S 19.319.3 1.1141.114 r(λ, 1/ν)r(λ,\,1/ν) 0.8880.888 Audio (VGGSound) SSLAM 7.97.9 1.1061.106 CED 7.97.9 1.0891.089 AST 9.59.5 1.0571.057 r(λ, 1/ν)r(λ,\,1/ν) 0.9480.948 Backbone ν λ Masked LM (TinyStories, masked-word) RoBERTa-large 9.69.6 1.1571.157 BERT-large 10.110.1 1.1271.127 ModernBERT-large 11.311.3 1.1241.124 NomicBERT 14.214.2 1.1151.115 r(λ, 1/ν)r(λ,\,1/ν) 0.8050.805 Autoregressive LM (TinyStories, next-token) OLMo-2-13B 10.210.2 1.1521.152 Qwen2.5-32B 10.910.9 1.1171.117 Qwen2.5-7B 11.211.2 1.1091.109 Qwen3-8B 11.311.3 1.1281.128 Qwen2.5-1.5B 11.311.3 1.1101.110 Qwen3-14B 11.311.3 1.1241.124 Qwen3-32B 11.411.4 1.1191.119 Qwen2.5-14B 11.511.5 1.1121.112 Llama-3.1-8B 12.412.4 1.1071.107 Qwen2.5-3B 12.612.6 1.1121.112 Qwen3-4B 12.712.7 1.1181.118 Qwen3-1.7B 13.913.9 1.1031.103 Qwen2.5-0.5B 19.219.2 1.0881.088 Qwen3-0.6B 21.921.9 1.0591.059 r(λ, 1/ν)r(λ,\,1/ν) 0.8870.887 Overall 0.9010.901 Table 11: Right-tail exponent ν across all backbones. ν is the power-law exponent of the fitted right tail p(s)∝(a2+s2)−ν/2p(s) (a^2+s^2)^-ν/2; a smaller ν means a heavier tail. λ is the global radius rescaling of the theory. Grouped by modality, sorted from heaviest to lightest tail. Pearson correlations between λ and 1/ν1/ν are high in every family (r=0.89,0.95,0.81,0.89r=0.89,0.95,0.81,0.89 in vision, audio, masked language models and autoregressive models, respectively) and globally over all models (r=0.90r=0.90). Backbone P N Acc Accth r λ RMSE Vision (ImageNet-1K) ConvNeXt-B 1000 1024 0.9430.943 0.9520.952 0.9540.954 1.2541.254 0.02560.0256 ResNet50 1000 2048 0.8610.861 0.8710.871 0.9360.936 1.2281.228 0.04540.0454 DINOv3 ViT-H+ 1000 1280 0.8560.856 0.8620.862 0.9600.960 1.2971.297 0.03900.0390 DINOv3 ViT-7B 1000 4096 0.8550.855 0.8610.861 0.9510.951 1.3161.316 0.04270.0427 ViT-B 1000 768 0.8530.853 0.8610.861 0.9590.959 1.3441.344 0.03780.0378 DINOv3 ViT-L 1000 1024 0.8450.845 0.8520.852 0.9640.964 1.2211.221 0.03840.0384 DINOv3 CNX-L 1000 1536 0.8140.814 0.8190.819 0.9610.961 1.3051.305 0.04400.0440 DINOv3 CNX-B 1000 1024 0.8040.804 0.8080.808 0.9580.958 1.2961.296 0.04550.0455 DINOv2 ViT-g 1000 1536 0.7950.795 0.8040.804 0.9410.941 1.2721.272 0.06310.0631 DINOv3 ViT-B 1000 768 0.7940.794 0.8000.800 0.9540.954 1.1581.158 0.04600.0460 CLIP-L 1000 768 0.7920.792 0.8040.804 0.9360.936 1.1081.108 0.05960.0596 DINOv3 CNX-S 1000 768 0.7860.786 0.7930.793 0.9560.956 1.2291.229 0.04810.0481 DINOv3 CNX-T 1000 768 0.7400.740 0.7490.749 0.9370.937 1.1481.148 0.06020.0602 DINOv3 ViT-S+ 1000 384 0.7380.738 0.7420.742 0.9510.951 1.1341.134 0.05040.0504 DINOv3 ViT-S 1000 384 0.7060.706 0.7100.710 0.9440.944 1.1141.114 0.05550.0555 mean 0.8120.812 0.8190.819 0.9510.951 1.2281.228 0.04680.0468 Audio (VGGSound) SSLAM 310 768 0.6280.628 0.6310.631 0.9520.952 1.1061.106 0.07200.0720 CED 310 768 0.6280.628 0.6320.632 0.9470.947 1.0891.089 0.07390.0739 AST 310 768 0.5830.583 0.5870.587 0.9440.944 1.0571.057 0.08190.0819 mean 0.6130.613 0.6170.617 0.9480.948 1.0841.084 0.07590.0759 Masked LM (TinyStories, masked-word) RoBERTa-large 320 1024 0.7670.767 0.7760.776 0.9440.944 1.1571.157 0.05420.0542 ModernBERT-large 320 1024 0.7340.734 0.7430.743 0.9420.942 1.1241.124 0.06140.0614 BERT-large 320 1024 0.7100.710 0.7200.720 0.9420.942 1.1271.127 0.06300.0630 NomicBERT 320 768 0.6490.649 0.6580.658 0.9450.945 1.1151.115 0.06830.0683 mean 0.7150.715 0.7240.724 0.9430.943 1.1311.131 0.06170.0617 Autoregressive LM (TinyStories, next-token) Qwen2.5-32B 320 5120 0.5670.567 0.5720.572 0.9450.945 1.1171.117 0.07220.0722 Qwen3-32B 320 5120 0.5660.566 0.5690.569 0.9550.955 1.1191.119 0.06650.0665 OLMo-2-13B 320 5120 0.5630.563 0.5670.567 0.9480.948 1.1521.152 0.07010.0701 Qwen3-14B 320 5120 0.5560.556 0.5580.558 0.9540.954 1.1241.124 0.06940.0694 Qwen2.5-14B 320 5120 0.5520.552 0.5560.556 0.9420.942 1.1121.112 0.07580.0758 Llama-3.1-8B 320 4096 0.5500.550 0.5530.553 0.9430.943 1.1071.107 0.07720.0772 Qwen2.5-7B 320 3584 0.5440.544 0.5500.550 0.9400.940 1.1091.109 0.07560.0756 Qwen3-8B 320 4096 0.5400.540 0.5420.542 0.9550.955 1.1281.128 0.06930.0693 Qwen2.5-3B 320 2048 0.5280.528 0.5310.531 0.9460.946 1.1121.112 0.07850.0785 Qwen3-4B 320 2560 0.5190.519 0.5200.520 0.9530.953 1.1181.118 0.06970.0697 Qwen2.5-1.5B 320 1536 0.4910.491 0.4940.494 0.9310.931 1.1101.110 0.08220.0822 Qwen3-1.7B 320 2048 0.4740.474 0.4740.474 0.9490.949 1.1031.103 0.07470.0747 Qwen2.5-0.5B 320 896 0.4520.452 0.4500.450 0.9440.944 1.0881.088 0.08530.0853 Qwen3-0.6B 320 1024 0.4060.406 0.4030.403 0.9390.939 1.0591.059 0.08390.0839 mean 0.5220.522 0.5240.524 0.9460.946 1.1111.111 0.07500.0750 Overall 0.6720.672 0.6770.677 0.9480.948 1.1601.160 0.06180.0618 Table 12: Centroid-aligned theory: The theory was tested on all 36 models. For each backbone, we report: number of classes P, feature dimension N, empirical prototype accuracy, per-class Pearson r between predicted and empirical accuracy, the fitted global rescaling λ, and the per-class prediction error RMSE. The theory consistently predicts the accuracy well across all models (r>0.93r>0.93). Figure 9: The best prototype layer is a few blocks before the final one for language, but is the final block for vision. Per-layer prototype classification accuracy versus fractional depth. Color runs from light (small) to dark (large) within each modality, and the peak performance is marked. (a) The six Qwen3 autoregressive language models; the optimal accuracy is a few blocks before the final layer, with a big drop at the final layer in most models. (b) the six DINOv3 vision transformers. The final layer is the best, with a plateau on DINOv3 7B. Appendix G Depth analysis The prototype classifier of Sec. 5 reads out a single hidden layer. Here we analyze how prototype classification accuracy and geometry change across depth, and identify the best layer for this task. For the autoregressive language models the best layer for content-word prediction is consistently a few blocks before the final one, and accuracy dips at the last block; for vision the final block is best (in DINOv3 7B, there is a plateau). We subsequently analyze the geometry of the largest language model, Qwen3-32B across depth, similar to the analysis in Sec. 6 for the Qwen3 family across scale. Similar to the scale effect, we see that the best layer has large true centroid correlation compared to rivals σk/σμ∗ _k/ _μ^*, decorrelates centroids (small gμ∗g_μ^*), and has small correlations between projections ρμ∗ _μ^*. We note that, as observed before with scale, the class radius R does not predict the overall best accuracy. Table 13: Qwen3-32B across depth. Class-averaged geometric measures for the seven last blocks of Qwen3-32B (6464 blocks). ∗ marks each class’s hardest competitor μ⋆μ , measured by σμ _μ, the projection standard deviation. ⟨R⟩ R is the mean class radius; σk/σμ∗ _k/σ^*_μ is the ratio between the standard deviation of the projection onto the true-class and hardest-rival directions. σμ∗σ^*_μ is the hardest rival projected std, gμ∗=^k⋅^μ⋆g^*_μ= c_k\!·\! c_μ the centroid overlap and ρμ∗ρ^*_μ the correlation between projections. ν is the right-tail exponent. Best measure highlighted in bold. Layer ⟨R⟩ R σk/σμ∗ _k/σ^*_μ σμ∗σ^*_μ gμ∗g^*_μ ρμ∗ρ^*_μ ν Acc L58 (0.91) 2.40 0.993 0.172 0.514 0.496 10.9 0.521 L59 (0.92) 2.32 0.996 0.175 0.511 0.502 11.0 0.542 L60 (0.94) 2.24 0.999 0.175 0.510 0.474 10.9 0.557 L61 (0.95) 2.20 1.007 0.175 0.503 0.451 11.4 0.566 L62 (0.97) 2.17 0.957 0.181 0.517 0.463 13.7 0.553 L63 (0.98) 2.16 0.894 0.191 0.521 0.448 18.1 0.528 L64 (final) 2.10 0.754 0.224 0.533 0.471 98.5 0.470 Appendix H Theoretical Predictions Across Models and Modalities The main text validates the centroid-aligned variability theory (Sec. 5) on a handful of representative backbones. Here we apply it, unchanged, to a comprehensive list of 36 state-of-the-art models, spanning four modalities: 15 vision encoders (ImageNet), 14 autoregressive language models and 4 masked language models (TinyStories), and 3 audio encoders (VGGSound). The models have different architectures (transformers and convolutional neural networks), training methods (supervised, self-supervised, and contrastive), and training datasets. For every model, we compute the per-class prediction with a single fitted global rescaling λ and the measured quantities the theory predicts, and compare it to the empirical prototype accuracy class by class. No per-model tuning beyond the scalar λ is used. Table 12 reports, for each backbone, the number of classes P, the feature dimension N, the empirical prototype accuracy, the per-class Pearson correlation r between predicted and empirical accuracy, the fitted λ, and RMSE. Across all four modalities the per-class correlation between the theory and the empirical accuracy is high: r=0.951r=0.951 (vision), 0.9460.946 (autoregressive), 0.9430.943 (masked) and 0.9480.948 (audio). Per-class errors have nearly zero bias (mean per modality bias 0.002−0.0090.002-0.009) and only a few percent mean deviation (mean per modality RMSE 0.0470.047–0.0760.076). Thus our theory predicts prototype accuracy class by class across vision, language and audio representations. For each model we also reproduce the two per-class panels of the theory vs the empirical accuracy as in Fig. 4. The four figures show all backbones grouped by modality. The theory not only tracks the general R trend, but often closely follows the individual pattern of the model. Figure 10: 15 vision models. For each backbone: (left) per-class accuracy versus empirical radius R, empirical binned mean and theory binned mean nearly overlap in most models; (right) per-class predicted versus empirical accuracy with r, RMSE and λ. The dashed line is y=xy=x. Mean Pearson r=0.951r=0.951, mean RMSE =0.047=0.047. Figure 11: Masked language models (4 models, TinyStories content-word prediction, best layer). Panels as in Fig. 10. Mean r=0.943r=0.943, mean RMSE =0.062=0.062. Figure 12: Autoregressive language models (14 models, TinyStories content-word prediction, best layer). Panels as in Fig. 10; equally spaced R bins fixed for this modality. Mean r=0.946r=0.946, mean RMSE =0.075=0.075. Figure 13: Audio (3 encoders, VGGSound training split, final layer). Panels as in Fig. 10. Mean r=0.948r=0.948, mean RMSE =0.076=0.076. Appendix I Variants of the theory The prototype-accuracy theory of the main text (Theorem 5.5) is one point in a small family of predictors, obtained by making different assumptions about the covariance of the standardized centroid projections sμs_μ. Here we compare the main theory against a simpler baseline and one richer refinement. All three predictors are evaluated at a fixed rival K=20K=20, on the same 36 backbones and the same per-class accuracy targets. The global rescaling factor λ was fitted separately for each variant. Uncorrelated s: The model is the same as the one presented in Sec. 5, with ρμ≠k=0 _μ≠ k=0, so that different sμs_μ are uncorrelated. The uncorrelated theory fits the data significantly worse than the theory of Sec. 5, where we consider a simple off-diagonal structure, and thus it is not used in the main text. Cluster expansion: We start from Eq. 22. We assume that Σμν _μν has the structure identified in Sec. 5, plus a small correction on the terms outside the diagonal, Σ=Σ0+δΣ,δΣμμ=0 = _0+δ , 5.0ptδ _μ=0. We shift the logits to have zero mean for convenience, and write the orthant probability conditioned on tkt_k, which has a diagonal element from the original structure plus the small correction Accth(R)∝∏μ∈ℛK∫−∞tk−fμ(tk)−gμdtμ2πexp(−12∑μ,ν∈ℛKtμ(δμνΛμ2+δΣμν)−1tν)Acc^th(R) _μ _K ^t_k-f_μ (t_k )-g_μ_-∞\! dt_μ 2π\, \! (- 12 _μ,ν _Kt_μ ( _μν ^2_μ+δ _μν )^-1t_ν ) (29) , where fμ(tk)=ρμσμ(tk−1)/σkf_μ (t_k )= _μ _μ (t_k-1 )/ _k is the conditional mean similar to Sec. C.2, and Λμ _μ is the conditional standard deviation Λμ=Rσμ1−ρμ2 _μ=R _μ 1- _μ^2. Expanding to linear order in δΣμνδ _μν: Accth(R)=∏μ∈ℛK∫−∞tk−fμ(tk)−gμdtμ2πΛμ2exp(−12∑μ∈ℛKtμ2Λμ2)(1+12∑μ≠νδΣμνtμtνΛμ2Λν2)Acc^th(R)= _μ _K ^t_k-f_μ (t_k )-g_μ_-∞\! dt_μ 2π ^2_μ\, \! (- 12 _μ _K t^2_μ ^2_μ ) (1+ 12 _μ≠ν δ _μνt_μt_ν ^2_μ ^2_ν ) (30) Define the conditional margin Fμ(tk)=tk−fμ(tk)−gμΛμ,F_μ(t_k)= t_k-f_μ (t_k )-g_μ _μ, (31) and perform the truncated Gaussian integrals Accth(R)=∏μ∈ℛKΦ(Fμ(tk))(1+12∑μ≠νδΣμνΛμΛνm(Fμ(tk))m(Fν(tk)))Acc^th(R)= _μ _K (F_μ (t_k ) ) (1+ 12 _μ≠ν δ _μν _μ _νm (F_μ (t_k ) )m (F_ν (t_k ) ) ) (32) We used the common inverse Mills ratio m(x)=ϕ(x)/Φ(x)m(x)=φ(x)/ (x), where ϕ(x)φ(x) is a standard normal PDF and Φ(x) (x) is a standard normal CDF. Substitute tk=Rzσk+1t_k=Rz _k+1 and averaging over z we get the final result: Accth(R)=z∼(0,1)[∏μ∈ℛKΦ(Fμ(z))(1+12∑μ≠νδΣμνΛμΛνm(Fμ(z))m(Fν(z)))]Acc^th(R)=E_z (0,1 ) [ _μ _K (F_μ (z ) ) (1+ 12 _μ≠ν δ _μν _μ _νm (F_μ (z ) )m (F_ν (z ) ) ) ] (33) With Fμ(z)=σk−ρμσμΛμ/Rz+1−gμΛμF_μ (z )= _k- _μ _μ _μ/Rz+ 1-g_μ _μ (34) Although the cluster expansion theory remains analytic, and fits the data slightly better than the original theory derived in Sec. 5, it adds another K(K−1)/2K(K-1)/2 parameters δΣμνδ _μν that need to be measured, and it is less transparent than the original simpler theory. Thus, we bring it here as an additional theoretical result with its performance on the 36 backbones we tested (Table. 14), but do not use it in the main text. Uncorrelated Main theory Cluster expansion Backbone λ RMSE λ RMSE λ RMSE Vision (ImageNet-1K) ConvNeXt-B 0.9790.979 0.04930.0493 1.2541.254 0.02560.0256 1.2811.281 0.02340.0234 ResNet50 1.1141.114 0.05890.0589 1.2281.228 0.04540.0454 1.3201.320 0.04340.0434 DINOv3 ViT-H+ 1.2121.212 0.05780.0578 1.2971.297 0.03900.0390 1.3171.317 0.03770.0377 DINOv3 ViT-7B 1.2481.248 0.05740.0574 1.3161.316 0.04270.0427 1.3371.337 0.04020.0402 ViT-B 1.1961.196 0.05930.0593 1.3441.344 0.03780.0378 1.3931.393 0.03390.0339 DINOv3 ViT-L 0.9960.996 0.07330.0733 1.2211.221 0.03840.0384 1.2481.248 0.03660.0366 DINOv3 CNX-L 1.1601.160 0.07060.0706 1.3051.305 0.04400.0440 1.3601.360 0.04170.0417 DINOv3 CNX-B 1.0871.087 0.07990.0799 1.2961.296 0.04550.0455 1.3371.337 0.04200.0420 DINOv2 ViT-g 1.1141.114 0.11660.1166 1.2721.272 0.06310.0631 1.3111.311 0.05610.0561 DINOv3 ViT-B 0.8640.864 0.09290.0929 1.1581.158 0.04600.0460 1.1961.196 0.04160.0416 CLIP-L 0.6710.671 0.11880.1188 1.1081.108 0.05960.0596 1.1991.199 0.04710.0471 DINOv3 CNX-S 0.9660.966 0.09350.0935 1.2291.229 0.04810.0481 1.2751.275 0.04500.0450 DINOv3 CNX-T 0.8550.855 0.12240.1224 1.1481.148 0.06020.0602 1.2091.209 0.05350.0535 DINOv3 ViT-S+ 0.8610.861 0.09810.0981 1.1341.134 0.05040.0504 1.1791.179 0.04380.0438 DINOv3 ViT-S 0.8380.838 0.10610.1061 1.1141.114 0.05550.0555 1.1731.173 0.04640.0464 mean 1.0111.011 0.08370.0837 1.2281.228 0.04680.0468 1.2761.276 0.04220.0422 Audio (VGGSound) SSLAM 0.9200.920 0.11940.1194 1.1061.106 0.07200.0720 1.1791.179 0.05550.0555 CED 0.8940.894 0.11640.1164 1.0891.089 0.07390.0739 1.1831.183 0.05990.0599 AST 0.8610.861 0.12560.1256 1.0571.057 0.08190.0819 1.1731.173 0.06170.0617 mean 0.8920.892 0.12050.1205 1.0841.084 0.07590.0759 1.1781.178 0.05900.0590 Masked LM (TinyStories, masked-word) RoBERTa-large 0.9430.943 0.08560.0856 1.1571.157 0.05420.0542 1.2351.235 0.04770.0477 ModernBERT-large 0.8840.884 0.09850.0985 1.1241.124 0.06140.0614 1.2251.225 0.05110.0511 BERT-large 0.9430.943 0.09300.0930 1.1271.127 0.06300.0630 1.2351.235 0.04920.0492 NomicBERT 0.9200.920 0.10480.1048 1.1151.115 0.06830.0683 1.2381.238 0.05650.0565 mean 0.9230.923 0.09550.0955 1.1311.131 0.06170.0617 1.2341.234 0.05110.0511 Autoregressive LM (TinyStories, next-token) Qwen2.5-32B 0.9370.937 0.10690.1069 1.1171.117 0.07220.0722 1.3631.363 0.07550.0755 Qwen3-32B 0.9400.940 0.10040.1004 1.1191.119 0.06650.0665 1.3761.376 0.07070.0707 OLMo-2-13B 0.9860.986 0.09220.0922 1.1521.152 0.07010.0701 1.3931.393 0.07260.0726 Qwen3-14B 0.9530.953 0.09840.0984 1.1241.124 0.06940.0694 1.3831.383 0.07160.0716 Qwen2.5-14B 0.9330.933 0.10930.1093 1.1121.112 0.07580.0758 1.3701.370 0.07730.0773 Llama-3.1-8B 0.9230.923 0.10610.1061 1.1071.107 0.07720.0772 1.3731.373 0.07500.0750 Qwen2.5-7B 0.9460.946 0.11140.1114 1.1091.109 0.07560.0756 1.3501.350 0.07800.0780 Qwen3-8B 0.9630.963 0.09470.0947 1.1281.128 0.06930.0693 1.3631.363 0.07030.0703 Qwen2.5-3B 0.9560.956 0.10580.1058 1.1121.112 0.07850.0785 1.3571.357 0.07800.0780 Qwen3-4B 0.9500.950 0.09930.0993 1.1181.118 0.06970.0697 1.3471.347 0.06850.0685 Qwen2.5-1.5B 0.9530.953 0.11660.1166 1.1101.110 0.08220.0822 1.3301.330 0.08650.0865 Qwen3-1.7B 0.9400.940 0.10030.1003 1.1031.103 0.07470.0747 1.3371.337 0.07180.0718 Qwen2.5-0.5B 0.9370.937 0.10720.1072 1.0881.088 0.08530.0853 1.3501.350 0.08080.0808 Qwen3-0.6B 0.9000.900 0.10960.1096 1.0591.059 0.08390.0839 1.3111.311 0.07720.0772 mean 0.9440.944 0.10420.1042 1.1111.111 0.07500.0750 1.3571.357 0.07530.0753 Overall 0.9650.965 0.09600.0960 1.1601.160 0.06180.0618 1.2951.295 0.05740.0574 Table 14: Fitted global rescaling λ and per-class prediction error RMSE for three predictors at K=20K=20, on all 36 backbones: the uncorrelated-projections baseline (ρμ=0 _μ=0), the main theory (rank-one true–rival correlation), and the first-order cluster expansion. λ is refitted separately for each predictor.