Paper deep dive
A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics
Louie Hong Yao, Yuhao Li, Shengchao Liu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/14/2026, 1:58:53 AM
Summary
The paper presents a minimal theoretical model to explain representation collapse in self-supervised learning. It identifies 'frustration'—samples that cannot be consistently classified—as the primary driver of collapse, which manifests as a slow-time-scale degradation of embedding geometry. The authors demonstrate that techniques like shared projection heads and stop-gradient mechanisms mitigate this by enabling non-collapsed fixed points and stabilizing class separation, a finding validated through both dynamical mean-field theory and linear teacher-student models.
Entities (5)
Relation Signals (3)
Frustration → induces → Representation Collapse
confidence 95% · a small fraction of frustrated samples that cannot be classified consistently induces collapse
Stop-Gradient → prevents → Representation Collapse
confidence 95% · stop-gradient enables non-collapsed solutions and stabilizes finite class separation under frustration
Dynamical Mean-Field Theory → analyzes → Representation Collapse
confidence 90% · develop a dynamical mean-field style self-consistency description, showing that stop-gradient enables non-collapsed solutions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised representation learning is central to modern machine learning because it extracts structured latent features from unlabeled data and enables robust transfer across tasks and domains. However, it can suffer from representation collapse, a widely observed failure mode in which embeddings lose discriminative structure and distinct inputs become indistinguishable. To understand the mechanisms that drive collapse and the ingredients that prevent it, we introduce a minimal embedding-only model whose gradient-flow dynamics and fixed points can be analyzed in closed form, using a classification-representation setting as a concrete playground where collapse is directly quantified through the contraction of label-embedding geometry. We illustrate that the model does not collapse when the data are perfectly classifiable, while a small fraction of frustrated samples that cannot be classified consistently induces collapse through an additional slow time scale that follows the early performance gain. Within the same framework, we examine collapse prevention by adding a shared projection head and applying stop-gradient at the level of the training dynamics. We analyze the resulting fixed points and develop a dynamical mean-field style self-consistency description, showing that stop-gradient enables non-collapsed solutions and stabilizes finite class separation under frustration. We further verify empirically that the same qualitative dynamics and collapse-prevention effects appear in a linear teacher-student model, indicating that the minimal theory captures features that persist beyond the pure embedding setting.
Tags
Links
- Source: https://arxiv.org/abs/2604.09979v1
- Canonical: https://arxiv.org/abs/2604.09979v1
Trouble viewing inline? Open PDF directly →
Full Text
86,447 characters extracted from source content.
Expand or collapse full text
A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics Louie Hong Yao, 1,∗ Yuhao Li, 2,† and Shengchao Liu 2,‡ 1 Center for Soft Matter and Biological Physics, Virginia Tech, Blacksburg, VA 24061, USA 2 Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China (Dated: April 14, 2026) Self-supervised representation learning is central to modern machine learning because it extracts structured latent features from unlabeled data and enables robust transfer across tasks and domains. However, it can suffer from representation collapse, a widely observed failure mode in which em- beddings lose discriminative structure and distinct inputs become indistinguishable. To understand the mechanisms that drive collapse and the ingredients that prevent it, we introduce a minimal embedding-only model whose gradient-flow dynamics and fixed points can be analyzed in closed form, using a classification-representation setting as a concrete playground where collapse is directly quantified through the contraction of label-embedding geometry. We illustrate that the model does not collapse when the data are perfectly classifiable, while a small fraction of frustrated samples that cannot be classified consistently induces collapse through an additional slow time scale that follows the early performance gain. Within the same framework, we examine collapse prevention by adding a shared projection head and applying stop-gradient at the level of the training dynam- ics. We analyze the resulting fixed points and develop a dynamical mean-field style self-consistency description, showing that stop-gradient enables non-collapsed solutions and stabilizes finite class separation under frustration. We further verify empirically that the same qualitative dynamics and collapse-prevention effects appear in a linear teacher-student model, indicating that the minimal theory captures features that persist beyond the pure embedding setting. I. INTRODUCTION Recent advances in artificial intelligence (AI) have de- livered striking performance across a wide range of tasks, from perception [1–3] and language understanding [4–8] to scientific and engineering applications [9–13]. Much of this progress is driven by large-scale data, large mod- els, and effective training recipes, which together yield strong generalization and rapid adaptation to new do- mains. However, the same ingredients introduce an enor- mous number of degrees of freedom and a training pro- cess that is highly coupled, nonlinear, and often messy in its details, so a clear theoretical picture remains limited. At the same time, physics has long developed tools to analyze complex many-body systems, where qualitative behavior emerges from collective effects rather than mi- croscopic details [14]. This perspective naturally suggests that modern learning systems may also admit robust ef- fective theories. In line with this idea, physics has interacted with mod- ern AI in two complementary directions. On one side, physicists have brought analytical tools together with physical intuition to understand learning systems. Ex- amples include nonequilibrium-dynamics viewpoints on optimization [15, 16], mean-field and dynamical mean- field approaches for high-dimensional networks and their representation dynamics [17–19], renormalization-group ∗ lhyao731@gmail.com; Equal contribution † Equal contribution ‡ scliu@cuhk.edu.hk and symmetry viewpoints on generative model behav- ior [20, 21], and spin-glass perspectives on emergent com- putation in large models [22–24]. On the other side, ideas from physics have shaped both model classes and training principles in modern AI. For instance, thermody- namic and dynamical formulations motivate diffusion and flow-based generative models [25–27]. Related perspec- tives draw on phase transitions and spontaneous sym- metry breaking to organize generative dynamics across scales [28, 29], and on synchronization phenomena to in- spire coupled-oscillator style architectures [30, 31]. Despite these advances, many concrete phenomena in learning systems still lack a clean dynamical account that isolates which ingredients control qualitative outcomes. A prominent example is representation collapse in self- supervised representation learning [32–34], which aims to assign each data sample a vector representation whose embedding geometry captures meaningful relations and supports transfer to downstream tasks. In this setting, collapse refers to a degeneration of the embedding ge- ometry, where representations lose discriminative struc- ture and many distinct inputs are mapped to nearly the same point. A range of strategies have been proposed to prevent this outcome. Explicit approaches enforce non- collapse by construction, for example through contrastive objectives with negative pairs [35–37] or through explicit covariance regularization [38]. More intriguingly, implicit approaches such as BYOL [39] and SimSiam [40] do not impose an explicit repulsive force, yet still prevent col- lapse through architectural or dynamical asymmetries. This raises the basic questions of when and why collapse arises in the first place and how these methods mitigate arXiv:2604.09979v1 [cond-mat.dis-n] 11 Apr 2026 2 data embedding Loss Function Eq. 2 datalabel label embedding encoder f θ encoder g θ classifier prediction Loss Function Eq. 1 datalabel one-hot embedding Loss Function Eq. 26 data embedding label embedding data embedding label embedding projection head projection head data embedding label embedding Loss Function Eq. 51 data encoder Loss Function Eq. 54 data embedding label embedding data embedding label embedding projection head projection head data data data embedding label embedding Loss Function Eq. 5 Problem FormulationFrustration Leads to CollapseProjection and SG Prevent Collapse Teacher-Student Scenario data embedding label embedding Loss Function Eq. 10 embedding space no frustration data embedding label embedding frustration embedding space Loss Function Eq. 26 data embedding label embedding data embedding label embedding projection head projection head encoder stop-gradient = = stop-gradient = = (a)(c)(e)(g) (b)(d)(f) (h) FIG. 1.Overview of model collapse in representation learning. (a) Standard contrastive-based classification pipeline with fixed one-hot labels, where only data representations are learned. (b) Generative-based representation-learning pipeline used in this work, with learnable data and label embeddings. (c) Embedding model under Eq. (5) without frustration, where class-wise dynamics remain decoupled. (d) Same model with frustration induced by shared data embeddings, leading to competing alignment constraints across classes. (e) Architecture with projection heads and stop-gradient [Eq. (26)], which modifies the effective gradient flow between branches. (f) Same architecture without stop-gradient, restoring symmetric coupling between embeddings. (g) Teacher–student pipeline where data embeddings are generated by a parametrized encoder. (h) Teacher–student model with a learned encoder, projection head, and stop-gradient, extending the mechanism beyond the embedding-only setting. it through their effects on the dynamics. Previous studies have sought to explain why im- plicit methods avoid collapse mainly in two ways, ei- ther through extensive empirical simulations [41, 42] or through simplified toy networks with specific architec- tures [43, 44]. When theory is pursued, it typically pro- ceeds by simplifying the network, for instance to two- layer models, to obtain a tractable description of the training dynamics. While these analyses are valuable, they often remain closely tied to microscopic architec- tural details, making it difficult to distill the dynamics into a simple effective theory. One reason is that even in such simplified settings the analysis typically starts from a detailed parameter-level, microscopic or ultravi- olet (UV) description and then attempts to derive the emergent infrared (IR) effective theory from it. In this work, we take a complementary route and for- mulate a minimal theory directly at the infrared level. Rather than starting from ultraviolet variables such as specific weights and architectural constraints, we treat the embeddings themselves as the effective degrees of freedom and focus on the collective dynamics they obey. We study these dynamics within a classification play- ground, where sample embeddings are trained to match labels. Unlike mainstream approaches, where represen- tations are trained to match labels in a categorical space, we instead consider alignment in a learnable label embed- ding space, where collapse is quantified directly through the contraction of the label-embedding geometry. An overview is illustrated in Fig. 1. This representation-level formulation admits closed- form dynamics and fixed-point analysis, and it shows that when all samples are perfectly classifiable, the learned structure can persist without collapse. Collapse is in- stead induced by frustration, defined as the fraction of samples that cannot be classified consistently. Such frustration is not a modeling artifact, but a generic feature of realistic learning problems, where it may arise from imperfect data, label noise, or limited model expressivity. At the representation level, how- ever, these different sources share the same effective con- sequence, with some embeddings failing to align con- 3 sistently with their class labels. When the fraction of such frustrated samples is small, it introduces a separate slow time scale, resulting in rapid early improvement fol- lowed by late-time degradation. This slow time scale is governed by the frustration fraction: as the fraction increases, it approaches the fitting time scale, and the separation between the two stages gradually vanishes. Within the same framework, we also study how tech- niques such as adding a shared projection head and ap- plying stop-gradient mitigate collapse. We show that stop-gradient enables non-collapsed fixed points by open- ing non-collapsed directions in the representation space. From the Dynamical Mean Field Theory (DMFT) self- consistency equations, we further find that this non- collapsed eigensubspace emerges from the interaction be- tween the projection head and frustration through the structure of the propagator. Moreover, to test whether these mechanisms persist once one restores a learned input-to-representation map, we validate the results in a linear teacher-student model, which reintroduces a parametrized mapping from inputs to embeddings. The same qualitative time-scale separa- tion and stop-gradient stabilization persist beyond the pure embedding setting. Taken together, our minimal model identifies frustration as the driving ingredient be- hind collapse and clarifies how stop-gradient avoids it by enabling non-collapsed fixed points through additional non-collapsing directions in representation space. The paper is organized as follows.We begin in Sec. I with the motivating empirical observations from classification-representation training and define the geo- metric quantities we use to diagnose collapse through the behavior of label embeddings. Sec. I and IV then de- velop the minimal model and use it to study both sides of the story, how frustration produces collapse through the fixed points and time scales and how a shared pro- jection head with stop-gradient opens up non-collapsed fixed points and alters the dynamics. To connect back to a more standard learning setup, Sec. V revisits the same phenomena in a linear teacher–student framework. We finish in Sec. VI with a brief discussion of implications, limitations, and future directions. I. MODE COLLAPSE IN REPRESENTATION LEARNING In this paper, we focus on the classification problem and later analyze it in a simplified setting, where the col- lapse phenomenon becomes easier to detect. Fig. 1(a) shows a standard classification problem, which is char- acterized as a supervised problem. We are given data points x α,i , where α = 1, 2,...,N labels the class to which each data point belongs. A parametrized function f θ (·) is then learned by optimizing the parameters θ over a loss function L, such as the typical contrastive loss, cross-entropy: L class = X α,i CrossEntropy f θ (x αi ),y α ,(1) where y α is the corresponding one-hot label of class α. In this setting, the labels are fixed as one-hot vectors in the spaceR N , with N denoting the number of classes. The optimization over θ is to find a specific function f θ (·) that makes f θ (x αi ) as close to y α as possible. Since the y α are fixed, there is no issue of mode collapse. Here we depart from the traditional contrastive- based classification setup and consider a generative-based representation-learning variant, as shown in Fig. 1(b). In this case, we further employ an encoding function for the class labels g θ (·), and the loss function becomes a func- tional of both f θ and g θ . We write f θ and g θ for the data and label encoders respectively, and let θ = (θ f ,θ g ) de- note the collection of all trainable parameters. In partic- ular, f and g do not share model architecture or weights; θ is only a shorthand for the joint parameter space. To train the model, we consider two types of loss. The first is the mean-square-error (MSE) loss, correspond- ing to the matching in the Euclidean distance. Another common choice is the negative cosine loss, which instead measures distance on the unit sphere. Given the MSE loss, the functions f θ and g θ are optimized over L MSE = X α,i f θ (x αi )− g θ (y α ) 2 .(2) This setting naturally appears in multiscale learning in the physical sciences [12, 45–47], where, due to scale sep- aration and universality, different microscopic settings correspond to similar mesoscopic and/or macroscopic be- haviors. Moreover, when the class labels are taken to be the data points themselves and the x are augmented ver- sions of these data points, it reduces to the multi-view learning setting [48] in computer vision and natural lan- guage processing. Given the loss in Eq. (2), we optimize the parameters θ by (stochastic) gradient descent with learning rate η. For a mini-batch B, one update step takes the form θ ← θ− η∇ θ L MSE (B).(3) In many settings, one additionally includes weight de- cay, which is equivalent to augmenting the loss by an L 2 regularization term λ 2 ∥θ∥ 2 2 . We will not discuss that case further here, as our goal is to isolate the intrinsic collapse dynamics without introducing an additional decay time scale. A. Mode collapse Since the label representations g θ (y) are not fixed in the representation space, there is a natural global min- imum of Eq. (2), achieved when f θ (x) = C = g θ (y) for 4 all x and y, where C is a constant vector. In this case, L MSE = 0, achieving the global minimum. The optimizer can reach this state by driving every representation to- ward the same vector C, so that all pairwise differences vanish and the dataset collapses to a single point. At this collapsed solution, class structure is erased: any classifier built on the representations cannot separate classes, and the accuracy reduces to chance level (e.g., 1/N for bal- anced N -way classification). Besides the complete mode collapse that gives the global minimum, partial mode col- lapse can also occur in this classification setting, for ex- ample, when g θ (α 1 ) = g θ (α 2 ) for α 1 ̸= α 2 . In this case, the distinction between class α 1 and class α 2 is lost, and the classification accuracy will suddenly drop by a step of order 1/N . In Fig. 2, we show the results on MNIST dataset with LeNet model [49] and on CIFAR-10 dataset [50] with ResNet model [51]. Since the collapse behavior is most apparent on the training set, we focus on the training dynamics only. The training-set accuracy is shown in the left panel: for both MNIST and CIFAR-10, the accuracy increases rapidly in the initial period and then starts to decay. This is generally in contrast to the usual intuition that, with a proper learning rate, continued training on the training set should keep improving performance on the training set and eventually overfit it, especially when the model is overparameterized. To better understand this behavior, we also plot the loss and the minimal L 2 distance between embeddings of any pair of labels, MinL2 ≡ min α̸=β ∥g θ (y α )− g θ (y β )∥ 2 , in the right panel. Since MinL2 characterizes the sepa- ration between label embeddings, we use it as a metric to capture mode collapse in the training dynamics. For easier comparison, we normalize both quantities by their values at epoch zero and show them on a log- arithmic y-axis. As seen in the plot, the loss exhibits a fast initial decay compared to MinL2, corresponding to the rapid growth in accuracy. However, after a crossover period, the decay of the loss gradually becomes paral- lel to the decay of MinL2, indicating that the dynamics are essentially governed by the collapsing behavior. This change of regime suggests the existence of two distinct time scales: one that controls the intended performance at early times and another defined by the collapsing dy- namics of the system. Note that in both cases we optimize the system using stochastic gradient descent without weight decay or other regularization terms. Regularizations generally introduce an additional timescale, which could interfere with the two time scales intrinsic to the system. B. Mitigation of Collapse To mitigate representation collapse, both explicit and implicit approaches have been developed in the com- munity. Explicit approaches prevent collapse by con- struction, for example, by introducing contrasted exam- 0.4 0.6 0.8 MNIST / LeNet Train accuracy 10 5 10 3 10 1 MNIST / LeNet (norm.) Loss MinL2 050100150200 Epoch 0.4 0.5 0.6 0.7 CIFAR10 / ResNet18 Train accuracy 050100150200 Epoch 10 2 10 1 10 0 CIFAR10 / ResNet18 (norm.) Loss MinL2 FIG. 2. Training dynamics of the classification–representation model on MNIST and CIFAR-10. In both experiments, we use an embedding dimension of 10, batch size 128, and stochastic gradient descent without weight decay or other regularization; the learning rate is 0.05 for MNIST and 0.01 for CIFAR-10. Left: training accuracy as a function of epochs for MNIST with LeNet and CIFAR-10 with ResNet-18. Right: corre- sponding training loss and minimal L 2 distance between la- bel embeddings (MinL2 ), both normalized by their values at epoch zero and shown on a logarithmic scale, highlight- ing a fast accuracy-improvement regime followed by a slower collapse-dominated regime. ples through contrastive objectives [35–37] or by explic- itly regularizing the embedding covariance, as in JEPA [38]. In contrast, implicit approaches such as BYOL [39] and SimSiam [40] stabilize non-collapsed representations through architectural or dynamical asymmetries, without relying on negative samples. In this work, we study the stop-gradient as the key dynamical ingredient underlying this class of implicit approaches. To adapt the stop-gradient mechanism to the classification–representation setting considered here, we introduce a shared projection network h θ (·) that is ap- plied symmetrically to both data and label embeddings. Specifically, we replace the original mean-squared-error objective in Eq. (2) with the projected variant L MSE = 1 2 X α,i h θ f θ (x αi ) − sg[g θ (y α )] 2 + 1 2 X α,i h θ g θ (y α ) − sg[f θ (x αi )] 2 , (4) which symmetrically matches projected data embeddings to label embeddings and vice versa. The stop-gradient operator sg[·] blocks gradient flow through the target branch during backpropagation, so that the correspond- ing partial derivatives do not appear in the parameter updates. Equivalently, in the gradient descent updates, all terms proportional to∇ θ sg[·] vanish identically, while the forward value of the target is retained in the loss. Im- 5 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Train Accuracy w/o stop-gradient w/ stop-gradient 0.00 0.05 0.10 0.15 0.20 0.25 Train Loss w/o stop-gradient w/ stop-gradient 050100150200 Epoch 0.00 0.25 0.50 0.75 1.00 1.25 1.50 MinL2 Before Projection w/o stop-gradient w/ stop-gradient 050100150200 Epoch 0.00 0.25 0.50 0.75 1.00 1.25 MinL2 After Projection w/o stop-gradient w/ stop-gradient FIG. 3.Effect of stop-gradient on representation collapse. Training dynamics on MNIST with LeNet under standard op- timization and stop-gradient. Top: training accuracy (left) and loss (right).Bottom: minimal L 2 distance between label embeddings before (left) and after (right) projection. Stop-gradient preserves early accuracy improvement while suppressing the decay of inter-label distances, stabilizing non-collapsed representations, and preventing the collapse- dominated regime. portantly, this operation does not modify the loss land- scape defined by Eq. (4), which may be viewed as an en- ergy function. Instead, it changes the gradient-induced vector field that governs the training dynamics. As a re- sult, the parameter updates no longer correspond to a steepest-descent flow with respect to this loss. In Fig. 3, we show the training dynamics of Eq. (4) on MNIST. The experiment implements the symmetric pro- jected loss with stop-gradient defined in Eq. (4), where the projection network h θ is chosen to be a single lin- ear layer without bias. All other aspects of the net- work architecture and optimization settings are kept un- changed. By tracking both task performance and inter- label separation, the figure shows that even with a mini- mal linear projection, stop-gradient leaves the early task- learning dynamics intact while suppressing the late-time collapse of label representations. Without stop-gradient, the training accuracy initially increases but subsequently decreases at late times, even as the training loss continues to decay. Additionally, the minimal inter-label distance MinL2 decreases monotonically both before and after projection, indicating a progressive collapse of label rep- resentations. When stop-gradient is applied, the training accuracy stabilizes at a high level, while the training loss no longer decreases to the extremely low values observed without stop-gradient. At the same time, the early-time decay of MinL2 remains similar with and without stop- gradient, but its late-time behavior changes qualitatively. Without stop-gradient, MinL2 continues to decay toward zero; in contrast, with stop-gradient, it saturates at a fi- nite value, indicating the absence of representation col- lapse. Discussion.The dynamics shown here correspond to a single training run over a finite time window of 200 epochs. Because SGD introduces stochasticity, rare col- lapse events can still occur in single training runs, espe- cially when the dynamics are evolved for long times, even if the overall probability of collapse is low. Nevertheless, models in the machine learning community are typically trained based on a single run, [52] and training is usually terminated well before extremely long-time dynamics are reached. Within this regime, the results here show that the stop-gradient mechanism can prevent collapse in a meaningful dynamical sense. In the following section, we introduce a minimal model to provide insight into these dynamics and the mechanisms underlying representation collapse and its mitigation. I. FRUSTRATION LEADS TO REPRESENTATION COLLAPSE We adopt a minimal modeling perspective in which the neural network is integrated out entirely, and the dynam- ics are formulated directly in representation space. This lets us bypass the complexity of network-level training and focus on the effective gradient-flow dynamics of the embeddings. Within this reduced description, collapse is not generic, but is instead driven by frustration: a frac- tion of samples cannot be classified perfectly by the net- work and thus induce competing alignment constraints. Such frustration may arise from imperfect data, label noise, limited model expressivity, or other microscopic causes. In the reduced representation-level description, however, all of these mechanisms enter through the same effect: a subset of embeddings cannot be brought into consistent alignment with their class labels. We there- fore start from the unfrustrated case and then examine how frustration qualitatively changes the dynamics and fixed-point structure. A. Dynamics without Frustration We consider a simplified model of a classification prob- lem with n classes. Each class is assumed to contain N samples. We reduce the problem to directly optimiz- ing embeddings for both the data samples and the class labels. Specifically, we denote by u iα ∈R d the embed- ding of the α-th sample from class i, and by v i ∈R d the embedding of the corresponding class label, where i = 1, 2,...,n and α = 1, 2,...,N . Here d denotes the embedding dimension. In this reduced formulation, these embeddings are optimized directly. As a starting point, we analyze the simplest setting in which there is no coupling between different classes, as shown in Fig. 1(c). The loss is a mean-squared error between each sample embedding and its corresponding 6 020406080100 Training step 10 2 10 1 10 0 | v i | n = 10, d = 2 n = 2, d = 5 n = 10, d = 5 n = 2, d = 10 FIG. 4.Mean inter-class deviation ⟨|δv i |⟩ as a function of training step for unfrustrated runs at learning rate 0.5. The vertical axis is shown on a logarithmic scale. All curves relax rapidly to small but nonzero plateaus, indicating stable class separation rather than complete collapse. class-label embedding, ℓ(u,v) = 1 2 n X i=1 N X α=1 u iα − v i 2 .(5) Because the loss decomposes as a sum over classes and embedding coordinates, there are no interactions between different classes. Each class is therefore optimized in- dependently, and embeddings associated with different labels remain completely decoupled. Moreover, the per- mutation symmetry among classes implies that all classes relax in the same manner. The gradient-flow dynamics are derived explicitly in Appendix A. Here, we focus on the resulting fixed-point structure. Within each class, all sample embeddings converge to the corresponding label embedding, u iα = v i for all α,(6) while the label embedding itself relaxes to a constant de- termined by the initialization. Due to the permutation symmetry among samples within a class, the long-time behavior depends only on the within-class sample mean. In particular, the asymptotic value of the label embed- ding is given by v i (∞) = N ̄u i (0) + v i (0) N + 1 ,(7) where ̄u i (0) = 1 N P α u iα (0) denotes the initial within- class sample mean. The asymptotic separation can be derived directly as follows. Let δv i = v i (∞)− 1 n n X j=1 v j (∞).(8) Using Eq. (7), this deviation can be expressed in terms of the initial conditions as: δv i = N N + 1 h ̄u i (0)− 1 n X j ̄u j (0) i + 1 N + 1 h v i (0)− ̄v(0) i , (9) where ̄v(0) = 1 n P j v j (0) denotes the center of the label embedding at initialization. For finite N the deviation δv i is generically nonzero unless the initial class means and label embeddings are identical across all classes. In particular, the second term in δv i does not vanish under generic Gaussian initialization and therefore prevents ex- act collapse at finite N . To confirm this behavior, we directly simulate the dy- namics and measure the resulting class separation. As shown in Fig. 4 we plot the mean inter class deviation ⟨|δv i |⟩ as a function of training time for different numbers of classes n and embedding dimensions d. Across all con- figurations the inter class deviations exhibit a rapid ex- ponential decay at early times and subsequently relax to small but nonzero plateau values, indicating stable class separation throughout training rather than collapse. Taken together, these observations indicate that in this minimal unfrustrated model, there is no intrinsic mecha- nism that drives different classes to collapse onto a com- mon representation. Collapse is therefore not a generic outcome of the dynamics. B. Dynamics with Frustration and the Emergence of Representation Collapse So far, we have assumed that embeddings are free to move so as to satisfy all alignment constraints imposed by the loss. By integrating out the neural network, we effec- tively allowed the embedding of each sample to evolve in- dependently, without interacting with other samples. In practice, however, perfect alignment is typically impos- sible, leaving some samples that cannot be consistently assigned to a single class. To model this effect at a minimal level, we introduce frustration directly in representation space by allowing a subset of samples to interact with multiple class labels. For a fraction 1− r of the samples, embeddings u iα are paired exclusively with their corresponding class label v i . For the remaining fraction r, embeddings s α are simulta- neously paired with all class labels, as shown in Fig. 1(d). These shared samples represent data points that cannot be uniquely assigned to a single class and therefore im- pose competing alignment constraints across classes. The resulting loss function takes the form ℓ = n X i=1 (1−r)N X α=1 u iα − v i 2 + n X i=1 rN X α=1 s α − v i 2 .(10) Unlike the unfrustrated case, this loss no longer decom- poses into a sum over independent classes. Each embed- 7 ding s α in the second term is simultaneously constrained to align with multiple label embeddings v i , generating competing forces that cannot, in general, be minimized simultaneously. We consider full-batch gradient descent with “raw” learning rate ̃γ. In practice, training proceeds via dis- crete updates and the loss is averaged over all samples. At fixed N , the total number of samples scales linearly with the number of classes n, introducing an overall nor- malization that rescales the gradient flow by a factor 1/n. When analyzing the dynamics, we work in a continuous- time description, in which both discretization effects and the normalization induced by loss averaging are absorbed into the definition of an effective learning rate γ. Differ- entiating the loss in Eq. (10) then yields the equations of motion ̇u iα =−γ(u iα − v i ), ̇s α =−γ n X i=1 (s α − v i ) =−γ ns α − n X i=1 v i , ̇v i =−γ Nv i − (1−r)N X α=1 u iα − rN X α=1 s α , (11) where overdots denote derivatives with respect to time. Fixed-point structure.The fixed-points of the gradient-flow dynamics are obtained by setting all time derivatives in Eq. (11) to zero. At any fixed point, all class-specific samples satisfy u iα = v i , while all shared samples take a common value s α = ̄v ≡ 1 n P i v i . Sub- stituting these relations back into the stationarity condi- tions yields v i = (1− r)v i + r ̄v,(12) which implies v 1 = v 2 =· = v n = ̄v.(13) Thus, all label embeddings coincide at the fixed point, and the class representations collapse onto a single shared value. This collapse is a direct consequence of frustra- tion and holds independently of the optimization path taken to reach the fixed point. This behavior is not sur- prising, as frustration effectively introduces attractive in- teractions between label embeddings through the shared samples. In the following, we analyze the dynamics by which the system approaches this collapsed fixed-point configuration. Dynamics and time scales. The full gradient-flow dynamics Eq. (11) admits an exact decomposition into three invariant sectors. Each sector is spanned by a set of symmetry-adapted effective degrees of freedom (DOF), evolves autonomously under a linear system, and is char- acterized by a distinct set of eigenvalues. a. Sector I: sample-level fluctuations. For each class i, define the within-class mean and residuals ̄u i ≡ 1 (1− r)N (1−r)N X α=1 u iα , δu iα ≡ u iα − ̄u i ,(14) with P α δu iα = 0. For the shared samples, define ̄s≡ 1 rN rN X α=1 s α , δs α ≡ s α − ̄s,(15) with P α δs α = 0. Sector I is spanned by the residual DOFδu iα andδs α . Subtracting the mean equations from the full dynamics yields ̇ δu iα =−γδu iα , ̇ δs α =−γnδs α .(16) Thus, sector I has eigenvalues λ =−γ, λ =−γn,(17) corresponding to dispersion within class-specific and shared samples, respectively. b. Sector I: class-level deviations. Define global means across classes ̄u≡ 1 n n X i=1 ̄u i , ̄v ≡ 1 n n X i=1 v i ,(18) and class-deviation variables δu i ≡ ̄u i − ̄u, δv i ≡ v i − ̄v,(19) which satisfy P i δu i = P i δv i = 0. Sector I is spanned byδu i ,δv i and captures inter-class contrasts. For each independent deviation direction, the dynamics are d dt δu i δv i = γ −11 N (1− r) −N δu i δv i .(20) The corresponding eigenvalues are λ (I) ± =− γ 2 h N + 1± p (N + 1) 2 − 4Nr i .(21) c. Sector I: global mean sector. Sector I is spanned by the fully symmetric DOF ( ̄u, ̄v, ̄s).(22) Their dynamics are governed by d dt ̄u ̄v ̄s = γ −110 N (1− r) −N Nr 0n −n ̄u ̄v ̄s .(23) This subsystem has one neutral eigenvalue λ (I) 0 = 0,(24) reflecting global translation symmetry, and two decaying eigenvalues λ (I) ± =− γ 2 h N +n + 1± p (N − n + 1) 2 + 4Nr(n− 1) i . (25) The decomposition shows that, at the level of the lin- earized dynamics, the system admits several relaxation 8 10 8 10 6 10 4 10 2 10 0 Magnitude n = 2, d = 5, r = 0.05 02004006008001000 Training steps (×100) 10 4 10 3 10 2 10 1 10 0 Magnitude n = 10, d = 2, r = 0.10 |u i | | v i | FIG. 5. Training dynamics of the frustrated representation model, showing a clear separation of time scales. Curves show the square root of the loss √ ℓ, the mean magnitude of sample- level deviations ⟨|δu i |⟩, and class-level deviations ⟨|δv i |⟩ on a logarithmic scale. Top: n = 2, d = 5, r = 0.05. Bottom: n = 10, d = 2, r = 0.10. All experiments use N = 1000 and a learning rate of 0.5. Fast sample-level relaxation is followed by slow, frustration-controlled inter-class dynamics. rates associated with different symmetry sectors. Not all of these modes, however, are relevant for classification or for the observed training dynamics. The global mean sector (Sector I) governs only the re- laxation of fully symmetric degrees of freedom and leaves relative class geometry unchanged. As a result, its modes do not manifest in the classification performance. The relevant dynamics are therefore contained in Sectors I and I. Sector I captures sample-level fluctuations, with a rate of order γ governing sample-to-class alignment and a rate of order γn associated with internal relaxation among shared, frustrated samples. The latter affects only relative positions within the frustrated subset and does not directly contribute to class discrimination. Sector I captures class-level deviations and controls the evolution of inter-class structure. In the classification regime of in- terest, the number of samples per class N is large. In this limit, the two eigenvalues of Sector I separate into a fast mode scaling as γN and a slower mode scaling as γr. The characteristic time scale associated with the fast mode, 1/(γN ), vanishes as N → ∞ and is therefore not observable in practice. Consequently, the effective learning dynamics reduce to two distinct time scales. The first is the sample- to-representation alignment rate γ, which governs the 0204060 1 10 4 10 3 10 2 10 1 10 0 (a) Fast stage 050001000015000 2 10 4 10 3 10 2 10 1 10 0 (b) Slow stage n = 2, d = 2 r = 0.05,= 0.5 n = 2, d = 5 r = 0.05,= 0.5 n = 2, d = 5 r = 0.05,= 1.0 n = 2, d = 5 r = 0.10,= 0.5 n = 2, d = 5 r = 0.30,= 0.5 n = 2, d = 10 r = 0.05,= 0.5 n = 10, d = 5 r = 0.05,= 0.5 FIG. 6.Two-stage relaxation of the training loss in the frustrated representation model. We plot the square root of the mean-squared loss, √ ℓ, using the fast rescaled time τ 1 in (a) and the slow rescaled time τ 2 in (b). The early-time decay in (a) reflects rapid sample-to-class alignment, while the long-time relaxation in (b) is controlled by frustration- induced inter-class coupling. Curves correspond to different choices of the number of classes n, embedding dimension d, frustration level r, and effective learning rate ̃γ, as indicated in the legend. rapid fitting of individual samples. The second is the frustration-controlled rate γr, which sets the slow, col- lective time scale associated with inter-class coupling and representation collapse. Empirical results. Fig. 5 shows two representative ex- amples of the training dynamics in the frustrated repre- sentation model. In both cases, the number of samples per class is fixed at N = 1000, while the number of classes n, the embedding dimension d, and the frustration level r are varied, as indicated in the figure. Despite these differences, both examples exhibit the same qualitative behavior. We plot the square root of the loss, √ ℓ, since the loss is a mean-squared error and taking the square root restores the natural distance scale in representation space. The loss dynamics separate into two stages, with an initial fast decay corresponding to rapid sample-to- class alignment, followed by a much slower relaxation driven by frustration-induced inter-class coupling, dur- ing which class representations gradually approach one another. The two stages of the loss dynamics are naturally cap- tured by two rescaled time variables. We define the fast time scale as τ 1 = γt, which governs the dynamics of sample-to-class alignment, and the slow time scale as τ 2 = γrt, which incorporates the effect of frustration through the fraction of shared samples. Fig. 6 shows the 9 loss plotted against these rescaled times. In panel (a), the early-time decay collapses when expressed in terms of τ 1 , indicating that this stage is dominated by a universal alignment process. In panel (b), the long-time decay col- lapses under τ 2 , revealing a distinct frustration-controlled regime. This second stage corresponds to the gradual collapse of class representations, during which inter-class distinctions are reduced under competing alignment con- straints. These results show that the observed two-stage be- havior is a direct consequence of a separation of relax- ation time scales in the frustrated representation dy- namics. The fast mode governs rapid sample-to-class alignment and corresponds to the regime in which model performance improves. At longer times, a much slower, frustration-controlled mode drives the gradual collapse of class representations. This separation provides a unified explanation for the empirical observation that training accuracy can initially increase before eventually degrad- ing as collapse develops, as shown in Fig. 2. IV. PROJECTION AND STOP-GRADIENT PREVENT REPRESENTATION COLLAPSE UNDER FRUSTRATION In this section, we study how the projection head and the stop-gradient mechanism help prevent collapse in the minimal frustrated model introduced above. We assume that the data and target branches share a common pro- jection head. The difference between the cases with and without stop-gradient is shown in Fig. 1(e) and 1(f). For simplicity, we consider a linear projection without a bias term, which can therefore be represented by a weight matrix W . The loss function can then be written as ℓ = 1 2 n X i=1 (1−r)N X α=1 (Wu iα − v i ) 2 + (Wv i − u iα ) 2 + 1 2 n X i=1 rN X α=1 (Ws α − v i ) 2 + (Wv i − s α ) 2 . (26) Here, we do not explicitly annotate the stop-gradient op- eration in the loss function. Instead, its effect is made explicit at the level of the gradient-flow equations, which are given by ̇ W =−γ n X i=1 (1−r)N X α=1 (Wu iα − v i )u T iα + (Wv i − u iα )v T i − γ n X i=1 rN X α=1 (Ws α − v i )s T α + (Wv i − s α )v T i , ̇u iα =−γW T (Wu iα − v i )−γ(u iα − Wv i ), ̇s α =−γ X i W T (Ws α − v i )−γ X i (s α − Wv i ), ̇v i =−γW T NWv i − rN X α=1 s α − (1−r)N X α=1 u iα −γ Nv i − rN X α=1 Ws α − (1−r)N X α=1 Wu iα . (27) The terms highlighted in color blue are those that van- ish when stop-gradient is applied. They arise solely from the absence of stop-gradient and correspond to additional reciprocal feedback between the data and label embed- dings. Throughout this section, we further assume that W has no zero eigenvalues. If W possesses zero modes, the dynamics project out the corresponding eigenspaces, so that the effective degrees of freedom evolve only within the nonzero eigensubspace of W . This induces an effec- tive dimensional reduction that we do not consider here. A. Fixed-Point Structures Induced by Projected Heads, With vs. Without Stop-Gradient We characterize the fixed-point manifold by impos- ing stationarity, i.e., setting all time derivatives to zero. These stationary configurations govern the long-time be- havior and therefore determine the system’s final state. We begin with the stop-gradient scenario, in which the stop-gradient operation eliminates certain terms and sim- plifies the dynamics substantially. With stop-gradient. We start with the embedding sec- tor of the fixed-point equations. Since W has no zero modes, they can be simplified as Wu iα = v i , Ws α = ̄v = 1 n n X j=1 v j , NWv i = rN X α=1 s α + (1−r)N X α=1 u iα . (28) 10 Substituting Eq. (28) into ̇ W = 0 shows that the con- tributions involving u iα and s α vanish separately, with the latter dropping out because P i ( ̄v− v i ) = 0. The re- maining terms then cancel exactly by the global balance condition in the third line of Eq. (28). The stationarity condition for the projection matrix W therefore imposes no additional constraint, and it is sufficient to analyze the embedding-sector equations alone. Acting with W on the last equation in (28) and using the first two equa- tions then yields a closed operator identity relating the label embeddings to W 2 : W 2 − (1− r)I v i = r ̄v.(29) This identity restricts the spectrum of W 2 to at most two distinct eigenvalues within the subspace spanned by the label embeddings: λ 0 = 1 and λ r = 1−r. Let Π 0 and Π r denote the projection operators onto the corresponding eigenspaces. Projecting Eq. (29) onto these two sectors decouples the constraint into the following relations: Π 0 v i = Π 0 ̄v,Π r ̄v = 0.(30) The first relation enforces a geometric collapse within the λ 0 sector, requiring individual embeddings to coin- cide with their global mean. The second relation demon- strates that any non-collapsed components are strictly confined to the λ r sector and are admissible only under the condition that the global mean has no projection onto this subspace, Π r ̄v = 0. The existence of the λ r sector suggests that non-collapsed fixed points can emerge as vi- able physical solutions. In this regime, the system avoids a total loss of dimensionality. While the λ 0 component of the embeddings is forced into a singular configuration, the λ r component provides a manifold where class-wise separation can be sustained. Such solutions are permissi- ble only when the embeddings are ”centered” within this subspace, such that their collective centroid vanishes. Without stop-gradient. The scenario without stop- gradient is more involved due to the additional terms. Without stop-gradient, the terms that would otherwise be removed remain coupled to the projection dynamics. As a result, the embedding-sector stationarity conditions are modified, and, as we show below, they exclude any non-collapsed fixed point for r > 0. As before, we begin with the embedding-sector stationarity conditions, which can be simplified to (W T W + I)u iα = (W T + W )v i (W T W + I)s α = (W T + W ) ̄v (W T W + I)v i = 1 N (W T + W ) rN X α=1 s α + (1−r)N X α=1 u iα . (31) Unlike the stop-gradient case, solutions of these embedding-sector conditions do not automatically satisfy the stationarity condition for the projection W , and thus impose additional constraints. For now, we focus on the embedding sector: substituting the first two equations into the last one gives (W T W + I)v i = B [r ̄v + (1− r)v i ],(32) where B = (W T + W )(W T W + I) −1 (W T + W ) and W T W + I is positive definite and invertible. Averaging Eq. (32) over i and subtracting the mean relation yields the reduced condition (W T W + I) 1/2 I − (1− r)C 2 (W T W + I) 1/2 δv i = 0, (33) where C = (W T W + I) −1/2 (W T + W )(W T W + I) −1/2 and δv i = v i − ̄v. Therefore, to avoid collapse (i.e., to allow δv i ̸= 0), the operator C 2 must have an eigenvalue equal to 1/(1− r). Since r > 0, this required eigenvalue is strictly greater than 1. However, we can bound the spectrum of C directly. Let λ be an eigenvalue of C with eigenvector x, such that Cx = λx. By defining v = (W T W + I) −1/2 x and substituting the definition of C, we obtain the generalized eigenvalue equation (W T + W )v = λ(W T W + I)v.(34) Left-multiplying this equation by v T yields 2v T Wv = λ(∥Wv∥ 2 +∥v∥ 2 ). We can then bound the magnitude of λ as follows: |λ| = |2v T Wv| ∥Wv∥ 2 +∥v∥ 2 ≤ 2∥v∥Wv∥ ∥Wv∥ 2 +∥v∥ 2 ≤ 1.(35) This demonstrates that all eigenvalues of C fall within [−1, 1], meaning C 2 cannot possibly admit an eigenvalue of 1/(1−r) > 1. Consequently, the operator I−(1−r)C 2 is strictly positive definite. Eq. (33) therefore uniquely enforces δv i = 0. As a result, the geometric constraints of the embedding sector alone intrinsically prohibit any non-collapsed solutions, rendering a non-trivial difference sector impossible. Discussion. The fixed-point structure shows that intro- ducing the projection W in the SimSiam spirit enlarges the dynamical system, but does not by itself resolve the collapse mechanism. In the fully coupled dynamics, the additional terms that remain without stop-gradient tighten the embedding-sector constraints. Concretely, Eq. (33) shows that sustaining fluctuations δv i ̸= 0 would require a nontrivial spectral sector incompatible with the bound on the interaction operator, so the embedding- sector equations force δv i = 0 for any r > 0. The conse- quence is that the fully collapsed configuration remains the unique stationary state. Stop-gradient, in contrast, alters the feedback struc- ture between the two branches and removes the geomet- ric obstruction that otherwise enforces collapse. The sta- tionary constraints then separate into two spectral sec- tors of W 2 : the λ 0 = 1 sector enforces collapse onto the mean, while the λ r = 1− r sector can sustain non- trivial class-wise structure provided the embeddings are centered, Π r ̄v = 0. Thus, stop-gradient is the ingredi- ent that stabilizes a non-collapsed fixed-point manifold, whereas the projection alone does not prevent collapse once the full coupling is retained. 11 B. Dynamics Under Projection and Stop-Gradient We briefly discuss the dynamics of the system. Since the introduction of the projection W , the system is non- linear and thus a closed-form analytical solution is not explicit. Here, we derive self-consistent equations in the spirit of DMFT [17, 53]. As in Sec. I B, we can decom- pose the dynamics into a mean sector, a class-deviation sector, and a within-class fluctuation sector. The dif- ference is that each sector evolves in a time-dependent medium set by W (t). This yields a Dyson-type integral formulation with a memory kernel. W -sector. We start with the projection sector. The dynamics of W can be written as the linear matrix ODE ̇ W =−γ WS(t)− R(t) ,(36) where S(t) = n X i=1 h (1− r)N ̄u i ̄u T i + C i,u (t) + Nv i v T i i + n rN ̄s ̄s T + C s (t) , R(t) = n X i=1 h (1− r)N (v i ̄u T i + ̄u i v T i ) + rN (v i ̄s T + ̄sv T i ) i . (37) Here C i,u (t) and C s (t) are the within-group covariances, defined as C i,u (t) = (1−r)N X α=1 u iα (t)− ̄u i (t) u iα (t)− ̄u i (t) T , C s (t) = rN X α=1 s α (t)− ̄s(t) s α (t)− ̄s(t) T . (38) The solution of (36) admits the integral representation W (t) = W (0)Φ(t, 0) + γ Z t 0 R(τ )Φ(t,τ )dτ,(39) where the propagator Φ(t,τ ) is defined by ∂ t Φ(t,τ ) =−γΦ(t,τ )S(t),Φ(τ,τ ) = I.(40) Here R(t) collects the mixed second moments between the projection outputs and the label embeddings. It is not an external input. Instead, R(t), and similarly S(t), is determined self-consistently by the embedding trajec- tories through (37) together with the embedding-sector dynamics derived below. Embedding sector. With the projection W (t), the em- beddings evolve in a time-dependent medium set by the Gram operator M (t)≡ W (t) T W (t).(41) 1 10 MinL2 02004006008001000 Training steps (×100) 1 10 | v i | n=2, d=2, r=0.1 n=2, d=5, r=0.1 n=2, d=10, r=0.05 n=2, d=10, r=0.1 n=5, d=5, r=0.05 n=5, d=10, r=0.1 FIG. 7. Training dynamics of the minimal model with pro- jection head and stop-gradient, shown for different numbers of classes n, embedding dimensions d, and frustration ratios r (legend). Top: the minimum pairwise distance between label embeddings, MinL2. Bottom: the mean magnitude of label- embedding deviations, ⟨|δv i |⟩. Both panels use a logarithmic y-axis and share the same y-range for direct comparison; the horizontal axis denotes training steps (in units of 10 2 ). This operator acts as an effective medium for the embed- ding dynamics, replacing the constant relaxation rates of the unprojected model with matrix-valued couplings. We follow the same sector decomposition as in Sec. I B, separating the dynamics into the mean sector, the class- deviation sector, and sample-level fluctuations, all evolv- ing in the medium set by M (t), with M (t) ultimately determined by the coupled projection dynamics. Under this decomposition, the mean sector evolves as ̇ ̄u =−γM ̄u + γW T ̄v, ̇ ̄v =−γNM ̄v + γNW T (1− r) ̄u + r ̄s , ̇ ̄s =−γnM ̄s + γnW T ̄v, (42) the class-deviation modes satisfy ̇ δu i =−γMδu i + γW T δv i , ̇ δv i =−γNMδv i + γN (1− r)W T δu i , (43) and, away from the permutation-symmetric manifold, the sample-level fluctuations are summarized by the within- group covariances (C i,u ,C s ) obeying the anticommutator flow ̇ C i,u (t) =−γM (t),C i,u (t), ̇ C s (t) =−γnM (t),C s (t), (44) with A,B = AB + BA. 12 We introduce the propagators generated by M (t), ∂ t G 1 (t,τ ) =−γM (t)G 1 (t,τ ), G 1 (τ,τ ) = I, ∂ t G N (t,τ ) =−γNM (t)G N (t,τ ), G N (τ,τ ) = I, ∂ t G n (t,τ ) =−γnM (t)G n (t,τ ), G n (τ,τ ) = I. (45) These propagators describe transport in a time- dependent effective medium M (t),replacing the constant-rate exponential decay of the unprojected model. In terms of these propagators, the mean sector admits the integral representation ̄u(t) = G 1 (t, 0) ̄u(0) + γ Z t 0 G 1 (t,τ )W (τ ) T ̄v(τ )dτ, ̄s(t) = G n (t, 0) ̄s(0) + γn Z t 0 G n (t,τ )W (τ ) T ̄v(τ )dτ, ̄v(t) = G N (t, 0) ̄v(0) + γN Z t 0 G N (t,τ )W (τ ) T × h (1− r) ̄u(τ ) + r ̄s(τ ) i dτ, (46) and the class-deviation sector is equivalently δu i (t) = G 1 (t, 0)δu i (0) + γ Z t 0 G 1 (t,τ )W (τ ) T δv i (τ )dτ, δv i (t) = G N (t, 0)δv i (0) + γN (1− r) Z t 0 G N (t,τ ) × W (τ ) T δu i (τ )dτ. (47) At the sample level, the microscopic modes propagate as δu iα (t) = G 1 (t, 0)δu iα (0), δs α (t) = G n (t, 0)δs α (0), (48) which implies the covariance propagation laws C i,u (t) = G 1 (t, 0)C i,u (0)G 1 (t, 0) T , C s (t) = G n (t, 0)C s (0)G n (t, 0) T . (49) The integral representations (46)-(49) make explicit how the embedding degrees of freedom are transported in the effective medium M (t), while the projection sector is en- coded by the propagator Φ(t,τ ) defined in (40) and the variation-of-constants formula (39). Although all embed- ding modes evolve in the same medium, the three chan- nels retain distinct characteristic rates set by γ, γN , and γn through G 1 , G N , and G n , respectively. The frus- tration ratio r does not enter the propagators directly. It appears in the source terms, for instance through the combination (1− r) ̄u + r ̄s in (46). It also enters indi- rectly through the learned projection, which determines M (t) = W (t) T W (t). As a result, the frustration-induced attraction acts only through the projection sector. Its ef- fect on the embedding dynamics is mediated by W (t) and can be reduced when M (t) relaxes toward fixed points that suppress this coupling, as discussed in Sec. IV A. 0 1 2 3 4 MinL2 0100200300400500 Training steps 0 1 2 3 4 | v i | n=2, d=2, r=0.1 n=2, d=5, r=0.1 n=2, d=10, r=0.05 n=2, d=10, r=0.1 n=5, d=5, r=0.05 n=5, d=10, r=0.1 FIG. 8. Training dynamics of the minimal model with pro- jection head but without stop-gradient, shown for different numbers of classes n, embedding dimensions d, and frustra- tion ratios r (legend). Top: the minimum pairwise distance between label embeddings, MinL2. Bottom: the mean mag- nitude of label-embedding deviations,⟨|δv i |⟩. Both quantities rapidly decay toward zero, indicating convergence to a fully collapsed configuration in which label embeddings become in- distinguishable. The horizontal axis denotes training steps. Together, these relations provide a convenient starting point for further analytical reductions. For example, if the projection and embedding sectors separate in time scales, one may derive effective dynamics by integrating out the fast sector. In this way, the projection can be ap- proximated as quasi-static while the embeddings follow a reduced evolution. Conversely, one can obtain an effec- tive equation for the slow drift of W near the collapsed configuration. We leave a detailed development of these reduced descriptions for future work and do not pursue it further here. C. Empirical Results We next validate our theoretical results with empiri- cal training runs of the minimal model with projection head. Figures 7 and 8 compare the training dynamics with and without stop-gradient. In both figures we track two quantities. The first is the mean deviation magni- tude ⟨|δv i |⟩, which is the main object in our theoretical analysis and measures the overall spread of the label em- beddings. The second is the minimum pairwise distance between label embeddings, MinL2 = min i̸=j ∥v i − v j ∥ 2 , which directly tests whether any two classes have be- come indistinguishable. While the theory is formulated in a class-symmetric setting, a single finite run need not realize the permutation symmetry between classes. The dynamics can therefore exhibit partial collapse in which only a subset of labels collapse onto one another. In that 13 Class 0 Class 1 Class 2 Class 3 Class 4 Frustration End FIG. 9. Schematic training trajectories in the minimal model with a single frustrated sample. Shown are the 2D paths of the projected class label embeddings Wv i for n = 5 classes (colored curves) together with the frustration embedding s (black dashed curve), with embedding dimension d = 2 and N = 50 samples per class. The single frustrated sample corre- sponds to a frustration ratio r = 1/50. Positions are recorded every 1000 optimization steps (open circles), and the final state is marked by a cross. case ⟨|δv i |⟩ can remain finite even though at least one pair has already collapsed, and MinL2 provides a sensi- tive indicator of the onset of collapse. With stop-gradient in Figure 7, ⟨|δv i |⟩ decays rapidly after a short transient, while MinL2 typically settles to a nonzero plateau whose value depends on the initial- ization as well as (n,d,r). The label embeddings there- fore remain separated rather than collapsing to a single point. Without stop-gradient in Figure 8, both MinL2 and ⟨|δv i |⟩ rapidly decay toward zero across all tested settings. The dynamics then converge to a fully collapsed configuration in which the label embeddings become in- distinguishable, consistent with an attractive collapsed fixed point under full-gradient updates. To complement these aggregate curves, Figure 9 plots an illustrative two-dimensional trajectory of the pro- jected label embeddings Wv i , with one curve per class, together with the frustration embedding s as training evolves. We observe a strongly anisotropic evolution in which the trajectories contract and align along one di- rection while remaining spread along a transverse direc- tion, so that separation is lost only partially rather than uniformly. This pattern supports our theoretical picture in which collapse occurs within the eigenspace of W 2 with eigenvalue 1, whereas non-collapse can persist in the eigenspace with eigenvalue 1− r. Figure 10 provides a spectral summary by plotting the eigenvalues of W 2 at the end of training for each stop- gradient run. The spectrum concentrates near a small number of well-separated values. For convenience, our 0.00.20.40.60.81.0 Eigenvalue of W 2 n=2, d=10, r=0.05 n=5, d=5, r=0.05 n=2, d=2, r=0.1 n=2, d=5, r=0.1 n=2, d=10, r=0.1 n=5, d=10, r=0.1 FIG. 10. Eigenvalue spectra of W 2 at the end of training (100,000 steps) for the stop-gradient (SG) runs. Each row corresponds to one setting (n,d,r), and markers denote the eigenvalues of W 2 . Across settings, the spectrum concentrates near a small number of well-separated values, with an eigen- value near 0 and the remaining eigenvalues grouped near 1 and 1− r. theoretical analysis assumes that W has no zero modes. In practice, some eigenvalues can appear near 0, indi- cating that the dynamics effectively evolve in a lower- dimensional subspace, with collapse occurring along the corresponding directions. Apart from these modes, the spectra align with the theoretical prediction that the re- maining eigenvalues cluster near two values, 1 and 1−r. The eigenvalue-1 sector identifies directions that undergo collapse, whereas the eigenvalue-(1− r) sector identifies directions that remain non-collapsed. V. REPRESENTATION COLLAPSE FROM EMBEDDING-WISE FRUSTRATION TO A LINEAR TEACHER–STUDENT MODEL We now move beyond the embedding-only minimal model to a parametrized setting, where representations are generated by a learned map from inputs rather than treated as free variables. We adopt a linear teacher- student construction because it provides a simple con- trolled setting: in a teacher-student model, a fixed teacher generates the target labels while a student model is trained to learn them. Here, the task is separable in the absence of corruption, while tunable label corruption gives direct control over the frustration ratio. This allows us to test whether the collapse mechanism identified in the embedding-only model persists once representations can be learned through a function class under controlled levels of frustration. A. Teacher Model and Data Generation We use a simple teacher–student construction in which the teacher is a linear binary classifier in d dimensions. The teacher assigns labels according to the sign of a fixed 14 42024 4 2 0 2 4 PCA projection 42024 4 2 0 2 4 Random affine map FIG. 11.Two-dimensional visualizations of the synthetic teacher-student dataset in d = 64, where different colors de- note the two classes. Left: PCA projection. Right: random affine map toR 2 . direction, y(x) = sign(w ⊤ x),(50) where we take d = 64 and choose w to be the normalized alternating vector (1,−1, 1,−1,... ). Given this teacher, we generate two Gaussian classes inR 64 with isotropic noise and opposite mean shifts along w. Concretely, we sample x∼N (+2μ,I) for class +1 and x ∼ N (−2μ,I) for class −1. To avoid sam- ples near the decision boundary, we discard points whose projection onto μ has magnitude smaller than 0.2. Fig- ure 11 shows two subfigures that visualize the same 64- dimensional dataset. The left subfigure shows a PCA projection that clearly separates the two classes and re- flects the underlying linear separability. The right subfig- ure shows a random affine map toR 2 that can make the classes appear overlapping and indicates that a randomly initialized linear layer does not separate them. To introduce frustration, given N samples in each class, we choose rN points from each class uniformly at random and overwrite their feature vectors by assigning them to the opposite class’s peak, i.e., to ±2μ, while keeping their original labels unchanged. This yields a fraction r of frustrated samples per class and keeps the dataset perfectly balanced. Because the same operation is applied to both classes, it preserves the permutation symmetry between classes and isolates the effect of frus- tration from trivial class-imbalance effects. B. Representation Collapse and Dynamics Under Frustration We first analyze the collapse dynamics in a minimal linear setting. We model the sample embeddings with a linear student x7→ Wx +b and represent each class by a learned label embedding v i , as shown in Fig. 1(g). Using the mean-squared error (MSE) objective, the loss is ℓ = X i,α Wx iα + b− v i 2 ,(51) 02004006008001000 Training Steps 10 1 10 0 (a) 0200400600800 10 2 10 1 10 0 / r = 0 (b) r = 0.05 r = 0.10 r = 0.20 r = 0.30 FIG. 12. Training dynamics of the linear student-label em- bedding model with embedding dimension 5 at learning rate 0.005 for different frustration rates r. (a) √ ℓ as a function of training steps on a log scale. (b) Rescaled loss ratio p ℓ/ℓ r=0 versus the effective time τ . where α indexes samples within a class, i∈1, 2 indexes the two classes, v i is the label embedding for class i, and W and b are the weights and bias of the student model. Before turning to empirical results, we briefly discuss two key differences between this setup and the pure embedding models. Scaling of the loss ℓ. This MSE objective has a simple scale symmetry. If we multiply the entire representation by a constant factor k, meaning we rescale the student output and the label embeddings together, W → kW, b→ kb, v i → kv i ,(52) then every residual Wx iα + b − v i is multiplied by k. Since the loss is a sum of squared residuals, it rescales as ℓ → k 2 ℓ. This scaling direction becomes relevant when the student model cannot fit the targets perfectly, so the residuals Wx iα + b− v i cannot be driven to zero. That is the generic situation here, because a single linear layer cannot, in general, align all sample embeddings Wx iα +b with the corresponding label embeddings v i . As a result, gradient descent can keep reducing the loss by shrinking the overall scale of (W,b,v i ), which preserves the rela- tive embedding geometry and therefore leaves the deci- sion structure essentially unchanged. At the same time, this contraction decreases the absolute distances between label embeddings, driving them closer together. In con- trast, in the pure embedding case the model can drive ℓ to zero, so this scaling freedom plays no role. Notably, the 15 same rescaling mechanism is present even when r = 0, so it does not rely on frustration. Effective frustration rate. A second difference is the way frustration is implemented. Rather than introducing universally frustrated samples that couple to all classes, we select a balanced set of samples from each class and overwrite their feature vectors by placing them at the op- posite class peak, while keeping their labels unchanged. In this construction, each frustrated sample acts as a mis- matched constraint of strength comparable to two label- flips in the pure embedding setting, effectively amplifying the frustration. It is therefore convenient to define an ef- fective frustration rate r eff = 2r 1 + r ,(53) and to express the relevant time scale in terms of r eff rather than r. Empirical results. We now turn to the empirical train- ing dynamics in Fig. 12. In these experiments, the linear student maps 64-dimensional inputs to a 5-dimensional embedding space. Panel (a) plots √ ℓ versus training steps for several frustration rates r and reveals a clear two-stage dynamics. The loss first drops rapidly in an early transient. This early-stage decay is not well de- scribed by a single exponential, reflecting the coupled evolution induced by the model structure even in this linear setting. At late times, the dynamics cross over to a regime with an approximately exponential decay. The separation of time scales is pronounced, and the dura- tion of the initial transient decreases as r increases. To factor out the trivial decay shared by all runs, we nor- malize the loss by the no-frustration baseline and plot p ℓ/ℓ r=0 . Panel (b) shows this ratio as a function of the effective time τ = r eff t. After normalization, the curves nearly collapse in the early stage, indicating that the initial transient is largely set by the r = 0 time scale. At late times, the curves approach straight lines on the log scale, indicating an approximately exponential decay. Moreover, these lines are nearly parallel across different r, which implies that the decay rates are the same once time is measured in τ = r eff t. This is consistent with our theory that frustration drives the late-time collapse and sets the relevant time scale through the (effective) frustration ratio. C. Dynamics Under Projection and Stop-Gradient Lastly, we study training in a linear teacher–student setting. As shown in Fig. 1(h), we focus on the variant with projection head and stop-gradient, which prevents complete collapse and supports stable non-collapsed fixed points. For simplicity, we use a linear projection head 0200040006000800010000 Training Steps 2.6 2.7 2.8 2.9 3.0 v 0 v 1 2 0200040006000800010000 Training Steps 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Accuracy r = 0.00 r = 0.05 r = 0.10 r = 0.20 r = 0.30 FIG. 13. Training dynamics of the model with projection head and stop-gradient in the linear teacher–student model for different frustration rates r. Left: separation between the learned label embeddings, ∥v 0 − v 1 ∥ 2 , versus training steps. Right: classification accuracy versus training steps. without bias. The loss function is then written as ℓ = X iα W ′ (Wx iα + b)− sg[v i ] ! 2 + X iα W ′ v i − sg[Wx iα + b] ! 2 , (54) where W ′ is the projection-head weight and sg[·] denotes the stop-gradient operator during backpropagation. In Fig. 13, we report the training dynamics in the lin- ear teacher–student model under stop-gradient for dif- ferent frustration rates r, using an embedding space of dimension 5 as in the collapse-dynamics setting. The left panel shows the distance between the two label embed- dings, ∥v 0 −v 1 ∥ 2 , which quickly reaches a plateau for all r, indicating that the dynamics with stop-gradient avoid complete collapse even as frustration increases. Since we consider two classes, this separation is simply twice the distance of each label embedding from their mean. The right panel reports the classification accuracy, which ap- proaches the ideal value 1− r, consistent with only the frustrated fraction of samples being intrinsically ambigu- ous while the remaining (1− r) fraction is, in principle, perfectly classifiable. In particular, the non-collapsing behavior in the left panel does not come at the expense of performance: near-ideal accuracy is preserved while the label separation remains nonzero. Notably, the stop-gradient also breaks the scale equiv- ariance of ℓ by removing the reciprocal coupling in the updates: because only one branch receives gradients, the scale-transformation direction that would otherwise shrink the absolute distances between embeddings, even when the geometric structure is preserved, is no longer available. As a result, the label embeddings remain at a finite separation instead of slowly approaching each other through pure rescaling, as clearly seen for r = 0. VI. SUMMARY AND OUTLOOK In this paper, we studied the widely observed phe- nomenon of representation collapse, where learned repre- 16 sentations lose discriminative structure and different in- puts become indistinguishable. We analyzed this mech- anism in a classification-representation setting, where sample embeddings were trained to match learned la- bel embeddings under an alignment objective. This set- ting let us quantify collapse more directly by tracking the shrinkage of pairwise separations between label em- beddings. To understand the underlying mechanism, we introduced a minimal model that optimized only the em- beddings, so the dynamics and fixed points could be an- alyzed in closed form. We showed both analytically and empirically that, in the perfectly classifiable case, this minimal model did not exhibit collapse. Collapse was in- stead induced by frustration, i.e. samples that could not be distinguished perfectly by the network. Frustration introduces an additional time scale on top of the original fitting time scale. When the fraction of indistinguishable samples is small, this new time scale is slow and well sep- arated from the original one. This separation captures the characteristic behavior observed in practice: perfor- mance improves rapidly at early times, then degrades at late times as the system drifts toward a collapsed state. Notably, the resulting two-time-scale structure in our toy model echoes the two time scales reported for diffusion models [54]. Here, the fast time scale corresponds to a generalization regime, where the model finds a “middle ground” that fits the data well, while the slow time scale corresponds to a memorization regime, where the model attempts to overfit the unclassifiable samples. Under the same minimal model framework, we stud- ied how commonly used intrinsic mechanisms prevent collapse. By adding a shared projection head and ap- plying stop-gradient, we showed that the dynamics ad- mit non-collapsed fixed points that are absent without stop-gradient, thereby stabilizing finite class separation. While the resulting dynamics were no longer exactly solv- able due to the nonlinearity introduced by the projection head, we derived a set of self-consistency equations in the spirit of dynamical mean-field theory (DMFT) to char- acterize the evolution. This formalism consistently ex- plained how frustration-induced collapse was mitigated, as reflected by the fixed-point structure and supported by empirical simulations. Finally, we demonstrated empiri- cally that the same qualitative phenomena and collapse- prevention effects appeared in a linear teacher-student framework, indicating that the minimal model captured robust features of collapse dynamics beyond the pure em- bedding setting. While our model effectively described the time-scale separation in representation collapse and how projection head and stop-gradient prevent collapse, it still missed behaviors and mechanisms that are worth exploring in future work. In our minimal model, sample embeddings moved freely and could converge exactly to their corre- sponding label embeddings. This differed from real data, where even in the best case samples from the same class form a finite-width cluster around the label embedding rather than collapsing to a single point. As a result, our setting was not well suited for studying accuracy di- rectly. Once the classifiable samples collapsed to single points, accuracy remained essentially unchanged unless the label embeddings themselves nearly coincided. Even if their separation decayed exponentially, it did not reach zero in finite time, so any accuracy change tied to exact collapse would be effectively unobservable on practical training horizons. A natural extension is to introduce an effective repulsive interaction among samples within the same class, which would stabilize finite clusters around each label embedding and enable a more faithful link be- tween collapse dynamics and classification performance. Second, we ignored several sources of stochasticity and additional couplings that are standard in modern train- ing. Practical optimization uses SGD rather than full- batch gradient descent, so a single run can be driven by fluctuations across basins toward a collapsed fixed point even when the average dynamics would avoid collapse, which becomes especially relevant at large scale where re- training is costly. Given an explicit dynamical equation together with a noise model, these stochastic effects could in principle be analyzed using the Martin–Siggia–Rose– Janssen–De Dominicis (MSRJD) formalism [55–58]. We also did not study weight decay, momentum, or explicit manifold constraints, all of which introduce additional in- teractions and time scales that can qualitatively change the long-time behavior and potentially mask the effective mechanisms isolated by our minimal model. AI USAGE We used large language models as writing and coding assistants to improve clarity and productivity. For writ- ing, AI tools were used to polish grammar, refine phras- ing, and suggest edits to the presentation. For coding, AI tools were used to help draft and refactor utility functions (e.g., plotting, data loading, and experiment scaffolding) and to suggest debugging checks. All technical content, including the modeling assumptions, theoretical deriva- tions, experimental design, code verification, figures, and final scientific claims, was developed and validated by the authors. The authors reviewed, edited, and take full responsibility for the final manuscript and the released code. ACKNOWLEDGMENTS LHY is indebted to Tianyu Jiang for fruitful discus- sions and providing computational resources. 17 [1] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, Transactions on Machine Learning Research (2024), featured Certifi- cation. [2] O. Sim ́eoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa,et al., arXiv preprint arXiv:2508.10104 (2025). [3] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) p. 10684–10695. [4] A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, et al., arXiv preprint arXiv:2512.02556 (2025). [5] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., arXiv preprint arXiv:2501.12948 (2025). [6] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al., arXiv preprint arXiv:2505.09388 (2025). [7] H. Wei,Y. Sun, and Y. Li, arXiv preprint arXiv:2510.18234 (2025). [8] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al., arXiv preprint arXiv:2507.06261 (2025). [9] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Fig- urnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇ Z ́ıdek, A. Potapenko, et al., nature 596, 583 (2021). [10] R. Acharya, D. A. Abanin, L. Aghababaie-Beni, et al., Nature 638, 920 (2025). [11] A. Novikov, N. V ̃u, M. Eisenberger, E. Dupont, P.- S. Huang, A. Z. Wagner, S. Shirobokov, B. Ko- zlovskii, F. J. Ruiz, A. Mehrabian, et al., arXiv preprint arXiv:2506.13131 (2025). [12] S. Liu, W. Du, H. Xu, Y. Li, Z. Li, V. Bhethanabotla, D. Yan, C. Borgs, A. Anandkumar, H. Guo, et al., Nature Communications (2025). [13] ˇ Z. Avsec, N. Latysheva, J. Cheng, G. Novati, K. R. Tay- lor, T. Ward, C. Bycroft, L. Nicolaisen, E. Arvaniti, J. Pan, et al., bioRxiv , 2025 (2025). [14] P. W. Anderson, Science 177, 393 (1972). [15] Y. Feng and Y. Tu, Proceedings of the National Academy of Sciences 118, e2015617118 (2021). [16] N. Yang, Y. Zhang, Q. Ouyang, C. Tang, and Y. Tu, arXiv preprint arXiv:2601.10962 (2026). [17] H. Sompolinsky, A. Crisanti, and H.-J. Sommers, Physi- cal review letters 61, 259 (1988). [18] A. Cowsik, T. Nebabu, X. Qi, and S. Ganguli, Physical Review E 112, 055301 (2025). [19] B. Epping, A. Ren ́e, M. Helias, and M. T. Schaub, Ad- vances in Neural Information Processing Systems 37, 48164 (2024). [20] M.KambandS.Ganguli,arXivpreprint arXiv:2412.20292 (2024). [21] A. Sheshmani, Y.-Z. You, B. Buyukates, A. Ziasha- habi, and S. Avestimehr, Physical Review E 111, 015304 (2025). [22] W. Hou, L. Zhou, H.-Y. Hu, Y. Chen, Y.-Z. You, and X.-L. Qi, arXiv preprint arXiv:2511.00763 (2025). [23] Y. Li, R. Bai, and H. Huang, Physical Review E 112, L013301 (2025). [24] R. Barney, M. Winer, and V. Galitski, arXiv preprint arXiv:2408.06421 (2024). [25] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, in International conference on machine learn- ing (pmlr, 2015) p. 2256–2265. [26] R. Wang and Y. Du, arXiv preprint arXiv:2510.02300 (2025). [27] W. Du, S. Chang, J. Tang, Y. Rong, F. Wang, and S. Liu, arXiv preprint arXiv:2504.19353 (2025). [28] M. Yaguchi, K. Sakamoto, R. Sakamoto, M. Tanabe, M. Akagawa, Y. Hayashi, M. Suzuki, and Y. Matsuo, Transactions on Machine Learning Research (2025), fea- tured Certification. [29] G. Raya and L. Ambrogioni, in Thirty-seventh Confer- ence on Neural Information Processing Systems (2023). [30] Y. Song, T. A. Keller, S. Brodjian, T. Miyato, Y. Yue, P. Perona, and M. Welling, arXiv preprint arXiv:2509.15328 (2025). [31] T. Miyato, S. L ̈owe, A. Geiger, and M. Welling, in The Thirteenth International Conference on Learning Repre- sentations (2025). [32] R. Balestriero and Y. LeCun, in NeurIPS 2024 Work- shop: Self-Supervised Learning - Theory and Practice (2024). [33] R. Shwartz Ziv and Y. LeCun, Entropy 26, 252 (2024). [34] Y. LeCun and I. Misra, Self-supervised learning: The dark matter of intelligence, Meta AI Blog (2021). [35] S. Chopra, R. Hadsell, and Y. LeCun, in 2005 IEEE com- puter society conference on computer vision and pattern recognition (CVPR’05), Vol. 1 (IEEE, 2005) p. 539–546. [36] A. v. d. Oord, Y. Li, and O. Vinyals, arXiv preprint arXiv:1807.03748 (2018). [37] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, in Proceedings of the 37th International Conference on Ma- chine Learning (2020) p. 1597–1607. [38] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vin- cent, M. Rabbat, Y. LeCun, and N. Ballas, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023) p. 15619–15629. [39] J.-B. Grill, F. Strub, F. Altch ́e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko, in Advances in Neural Information Pro- cessing Systems, Vol. 33 (2020) p. 21271–21284. [40] X. Chen and K. He, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021) p. 15750–15758. [41] X. Wang, H. Fan, Y. Tian, D. Kihara, and X. Chen, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022) p. 16570–16579. [42] X. Wang, X. Chen, S. S. Du, and Y. Tian, arXiv preprint arXiv:2110.04947 (2021). [43] Y. Tian, X. Chen, and S. Ganguli, in Proceedings of the 38th International Conference on Machine Learning , Vol. 139 (2021) p. 10268–10278. 18 [44] C. Zhang, K. Zhang, C. Zhang, T. X. Pham, C. D. Yoo, and I. S. Kweon, in International Conference on Learning Representations (2022). [45] M. Maggioni, S. Minsker, and N. Strawn, The Journal of Machine Learning Research 17, 43 (2016). [46] M. Kalimuthu, D. Holzm ̈uller, and M. Niepert, in ICLR 2025 Workshop on Machine Learning Multiscale Pro- cesses (2025). [47] N. Parsan, D. J. Yang, and J. J. Yang, in ICLR 2025 Workshop on Machine Learning Multiscale Processes (2025). [48] Y. LeCun, Open Review 62, 1 (2022). [49] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Pro- ceedings of the IEEE 86, 2278 (2002). [50] A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images. (2009). [51] K. He, X. Zhang, S. Ren, and J. Sun, in Proceedings of the IEEE conference on computer vision and pattern recognition (2016). [52] While ensemble methods such as bagging are common in traditional machine learning, they are rarely applied in neural network training due to computational cost. [53] A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Reviews of modern physics 68, 13 (1996). [54] T. Bonnaire, R. Urfin, G. Biroli, and M. Mezard, in The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025). [55] P. C. Martin, E. D. Siggia, and H. A. Rose, Physical Review A 8, 423 (1973). [56] C. de Dominicis, in J. Phys.(Paris), Colloq.;(France), Vol. 1 (1976). [57] H.-K. Janssen, Zeitschrift f ̈ur Physik B Condensed Mat- ter 23, 377 (1976). [58] U. C. T ̈auber, Critical dynamics: a field theory ap- proach to equilibrium and non-equilibrium scaling behav- ior (Cambridge University Press, 2014). Appendix A: Dynamics of the Unfrustrated MSE Model In this appendix, we provide the detailed derivation of the gradient-flow dynamics and the fixed-point structure for the unfrustrated mean-squared-error (MSE) model in- troduced in Sec. I. We consider the loss ℓ(u,v) = 1 2 n X i=1 N X α=1 u iα − v i 2 ,(A1) where u iα ∈R d denotes the embedding of sample α from class i and v i ∈R d denotes the embedding of the corresponding class label. Since different embed- ding coordinates do not couple, the dynamics can be analyzed component-wise. We therefore restrict to the one-dimensional case without loss of generality. Under full-batch gradient descent with learning rate ̃γ, the discrete-time updates are ∆u iα =− ̃γ u iα − v i , ∆v i =− ̃γ N X α=1 v i − u iα . (A2) Taking the continuous-time limit with normalized learn- ing rate γ, we obtain the gradient-flow equations ̇u iα =−γ u iα − v i , ̇v i =−γ N X α=1 v i − u iα . (A3) A key feature of Eq. (A3) is that the dynamics are fully decoupled across classes. All forces acting on (u iα ,v i ) de- pend only on the class index i, and there is no coupling between different classes. We therefore analyze the dy- namics for a fixed class i and suppress the class index where no confusion arises. Define the within-class sum U ≡ N X α=1 u α ,(A4) so that summing Eq. (A3) over α yields ̇ U =−γ U − Nv , ̇v =−γ Nv− U . (A5) The fixed-point manifold of this system is given by U = Nv,(A6) corresponding to perfect alignment between sample em- beddings and the label embedding. To analyze relaxation toward this manifold, define the deviation δ(t)≡ U (t)− Nv(t).(A7) 19 Using Eq. (A5), its time derivative is ̇ δ = ̇ U − N ̇v =−γ U − Nv + γN Nv− U =−γ(N + 1)δ. (A8) Thus, δ(t) = δ(0)e −γ(N +1)t ,(A9) where δ(0) = U (0)− Nv(0). Since ̇v = γδ, integration gives v(t) = v(0) + γ Z t 0 δ(s)ds = v(0) + δ(0) N + 1 1− e −γ(N +1)t . (A10) Taking the long-time limit t→∞, we obtain v(∞) = v(0) + U (0)− Nv(0) N + 1 = U (0) + v(0) N + 1 .(A11) Writing U (0) = N ̄u(0) in terms of the initial within-class mean ̄u(0) = 1 N N X α=1 u α (0),(A12) yields the fixed-point value quoted in the main text, v(∞) = N ̄u(0) + v(0) N + 1 .(A13) Finally, substituting v(∞) back into the solution for u α shows that u α (t) −→ v(∞)for all α,(A14) demonstrating that all sample embeddings within a class align with the label embedding, while different classes remain independent. This completes the derivation of the unfrustrated dynamics. Appendix B: Exact Spectrum of the Frustrated Dynamics In this section we derive the full mode decomposition and eigenvalues of Eq. (11). Although the system has O(nN ) degrees of freedom, its permutation symmetries imply an exact block-diagonalization into invariant sub- spaces. Throughout we assume r > 0 so that the shared- sample variables are well-defined. 1. Decomposition of u iα and s α into Means and Fluctuations Define the within-class mean and fluctuations of the non-shared samples: ̄u i ≡ 1 (1− r)N (1−r)N X α=1 u iα , δu iα ≡ u iα − ̄u i , (B1) so that P α δu iα = 0 for each i. Similarly, define the shared-sample mean and fluctuations: ̄s≡ 1 rN rN X α=1 s α , δs α ≡ s α − ̄s,(B2) so that P α δs α = 0. Averaging Eq. (11) over α yields the closed dynamics for the means: ̇ ̄u i =−γ( ̄u i − v i ),(B3) ̇ ̄s =−γ n ̄s− n X j=1 v j ,(B4) ̇v i =−γ Nv i − (1− r)N ̄u i − rN ̄s .(B5) Subtracting (B3) from the equation for u iα gives the fluctuation dynamics ̇ δu iα =−γδu iα .(B6) Likewise, subtracting (B4) from the equation for s α gives ̇ δs α =−γnδs α .(B7) Equations (B6)–(B7) show that the internal sample fluc- tuations decouple exactly from ( ̄u i , ̄s,v i ). 2. Eigenvalues of the Fluctuation Subspaces Equation (B6) implies that each independent compo- nent of δu iα decays with eigenvalue λ u =−γ.(B8) The multiplicity is n (1 − r)N − 1 (one constraint P α δu iα = 0 per class). Equation (B7) implies that each independent compo- nent of δs α decays with eigenvalue λ s =−γn,(B9) with multiplicity (rN − 1) (one constraint P α δs α = 0). These modes describe within-group dispersion and do not affect class contrast. 20 3. Reduction of the Mean Dynamics into Deviation and Mean Sectors The remaining dynamics (B3)–(B5) live in the (2n+1)- dimensional space of ( ̄u i ,v i , ̄s). Define global means and deviations across classes: ̄u≡ 1 n n X i=1 ̄u i , ̄v ≡ 1 n n X i=1 v i ,(B10) δu i ≡ ̄u i − ̄u, δv i ≡ v i − ̄v, n X i=1 δu i = n X i=1 δv i = 0. (B11) Substituting into (B3)–(B5) yields two independent blocks: a. (i) Deviation sector. For each of the (n− 1) in- dependent deviation directions, d dt δu δv = γ −11 N (1− r) −N δu δv ,(B12) with eigenvalues λ dev ± =− γ 2 h N + 1± p (N + 1) 2 − 4Nr i .(B13) Each of λ dev ± has multiplicity (n−1). These modes control the decay of inter-class contrasts, e.g. v i − ̄v. b. (i) Mean sector. The triplet ( ̄u, ̄v, ̄s) evolves as d dt ̄u ̄v ̄s = γ −110 N (1− r) −N Nr 0n −n ̄u ̄v ̄s .(B14) This block has one exact zero eigenvalue, λ mean 0 = 0,(B15) reflecting the global translation symmetry of the squared loss. A convenient conserved quantity is r ̄s(t) + n N ̄v(t) + n(1− r) ̄u(t) = const.(B16) The remaining two eigenvalues are λ mean ± =− γ 2 h N +n + 1± p (N − n + 1) 2 + 4Nr(n− 1) i . (B17) 4. Full Spectrum and Multiplicities Collecting the invariant subspaces, the spectrum of the full linear system (11) consists of: • λ u =−γ with multiplicity n (1−r)N− 1 (within- class u fluctuations); • λ s =−γn with multiplicity (rN−1) (within-shared s fluctuations); • λ dev ± from (B13), each with multiplicity (n − 1) (inter-class deviation modes); • λ mean ± from (B17), each with multiplicity 1, and a neutral mode λ mean 0 = 0(global mean sector). In particular, the inter-class contrasts v i − ̄v lie en- tirely in the deviation sector and are governed only by λ dev ± . The alignment errors ̄u i − v i receive contributions from both the deviation and mean sectors, while internal sample dispersions decay independently according to λ u (and λ s for shared-sample dispersion).