Paper deep dive
Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking
Yongzhong Xu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/21/2026, 2:09:48 AM
Summary
This paper presents a geometric analysis of 'grokking' in transformers trained on modular arithmetic, proposing that grokking corresponds to optimization dynamics confined to a low-dimensional 'execution manifold' where transverse curvature accumulates. Using PCA on attention weight trajectories, the authors find that training evolves predominantly within a rank-1 subspace (capturing 68-83% of variance). They measure loss-landscape curvature via commutator defects, finding that curvature grows sharply in directions orthogonal to the execution manifold and that this growth consistently precedes generalization. Causal interventions show that suppressing orthogonal gradient flow prevents grokking, while artificially increasing curvature is insufficient, supporting a model where grokking is an escape from a metastable, low-dimensional regime.
Entities (8)
Relation Signals (6)
Commutator Defect → measures → Loss-Landscape Curvature
confidence 92% · To probe loss-landscape geometry, we measure commutator defects -- the non-commutativity of successive gradient steps
Commutator Defect → precedes → Generalization
confidence 91% · Importantly, curvature growth consistently precedes generalization across learning rates and hyperparameter regimes
Grokking → ischaracterizedby → Execution Manifold
confidence 90% · We propose that grokking corresponds to prolonged confinement on a low-dimensional subspace in weight space... We identify this subspace—the execution manifold
Execution Manifold → isorthogonalto → Curvature Growth
confidence 89% · curvature grows sharply in directions orthogonal to the execution subspace while the trajectory remains largely confined to it
Causal Intervention → demonstrates → Necessity of Orthogonal Gradient Flow
confidence 88% · Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking
Lead Time → obeys → Power Law
confidence 85% · the lead time obeying a power law in the grokking timescale
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Grokking -- the delayed transition from memorization to generalization in small algorithmic tasks -- remains poorly understood. We present a geometric analysis of optimization dynamics in transformers trained on modular arithmetic. PCA of attention weight trajectories reveals that training evolves predominantly within a low-dimensional execution subspace, with a single principal component capturing 68-83% of trajectory variance. To probe loss-landscape geometry, we measure commutator defects -- the non-commutativity of successive gradient steps -- and project them onto this learned subspace. We find that curvature grows sharply in directions orthogonal to the execution subspace while the trajectory remains largely confined to it. Importantly, curvature growth consistently precedes generalization across learning rates and hyperparameter regimes, with the lead time obeying a power law in the grokking timescale. Causal intervention experiments show that motion along the learned subspace is necessary for grokking, while artificially increasing curvature is insufficient. Together, these results support a geometric account in which grokking reflects escape from a metastable regime characterized by low-dimensional confinement and transverse curvature accumulation. All findings replicate across this learning-rate range, a qualitatively different slow regime (lr=5e-5, wd=0.1, 3 layers), and three random seeds, though alignment dynamics differ quantitatively between regimes. Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking: suppressing it prevents generalization with a monotonic dose-response across four operations, while artificially boosting curvature defects has no effect.
Tags
Links
- Source: https://arxiv.org/abs/2602.16746v3
- Canonical: https://arxiv.org/abs/2602.16746v3
Trouble viewing inline? Open PDF directly →
Full Text
82,497 characters extracted from source content.
Expand or collapse full text
Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking Yongzhong Xu abbyxu@gmail.com ; code at https://github.com/skydancerosel/grokking-integrability Abstract Grokking—the abrupt onset of generalization long after memorization—challenges standard accounts of how neural networks learn, yet the geometric mechanisms driving this transition remain poorly understood. We propose that grokking corresponds to prolonged confinement on a low-dimensional subspace in weight space, during which transverse curvature barriers accumulate until the trajectory escapes into the generalizing solution. We identify this subspace—the execution manifold—in the weight space of transformers trained on modular arithmetic. Using PCA on attention weight trajectories and commutator defect analysis across six binary operations mod 97, we show that weight evolution during grokking is essentially one-dimensional: a single principal component captures 68–83% of variance across 36 experimental conditions. We then measure loss-landscape curvature via commutator defects—the non-commutativity of successive gradient steps—and project these onto the learned submanifold. The commutator vectors are predominantly orthogonal to the execution manifold, even relative to random baselines (exec/random ratio ≈2≈ 2–3×3×), indicating that the subspace is empirically invariant under the optimization dynamics: curvature does not deflect the trajectory out of its learned subspace. Yet curvature explodes orthogonally: grokking operations show 1010–1000×1000× higher commutator defect than non-grokking controls, concentrated entirely in the normal bundle of the execution manifold. The onset of this curvature growth consistently precedes the generalization transition by 600–1600 training steps at η=10−3η=10^-3 (sign test p=2−12<0.001p=2^-12<0.001), though non-grokking operations also exhibit moderate curvature growth (3030–50×50×) without generalizing, so onset is necessary but not sufficient. Across a 300×300× learning-rate sweep, the lead time obeys a power law Δt∝tgrokα t t_grok^\,α with α=1.27±0.03α=1.27± 0.03 (R2=0.97R^2=0.97, n=43n=43): because α>1α>1, the predictive window grows super-linearly with grokking timescale—at η=3×10−5η=3\!×\!10^-5, defect onset occurs within the first 5% of training, providing a 95% advance warning window. All findings replicate across this learning-rate range, a qualitatively different slow regime (lr=5×10−5lr=5\!×\!10^-5, wd=0.1wd=0.1, 3 layers), and three random seeds, though alignment dynamics differ quantitatively between regimes. Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking: suppressing it prevents generalization with a monotonic dose–response across four operations, while artificially boosting curvature defects has no effect. Weight-SVD spectral analysis reveals the mechanism: grokking is preceded by a near-degeneracy of the leading singular values of the attention matrices (σ1≈σ2 _1≈ _2), creating an orientation instability that drives the commutator transition. Generalization coincides with the breaking of this spectral symmetry as one mode dominates. This spectral structure is confirmed to be basis-independent: three independent decompositions (weight SVD, displacement SVD, gradient SVD) all exhibit a consistent sign flip in the commutator–subspace alignment at grokking. 1 Introduction Grokking—the phenomenon where neural networks trained on small algorithmic datasets first memorize the training set and then, long after achieving perfect training accuracy, suddenly generalize to the test set—was first reported by Power et al. (2022) in modular arithmetic tasks. The phenomenon has attracted significant attention because it challenges the conventional understanding that generalization and memorization are tightly coupled in optimization dynamics. Prior work has characterized grokking through the lens of representation learning (Nanda et al., 2023), weight decay as implicit regularization (Liu et al., 2022), circuit formation (Zhong et al., 2024), and phase transitions in loss landscapes. Recently, Xu (2026a) showed that the weight-space trajectory during grokking lies on a low-dimensional execution manifold, with PCA revealing that a single principal component captures the majority of trajectory variance. However, a fundamental geometric question remains open: is this low-dimensional manifold invariant under the optimization dynamics—does curvature deflect the trajectory out of its learned subspace—and does its geometry predict when generalization will occur? Building on Xu (2026a), we address this question by studying the differential geometry of the parameter-space trajectory during grokking. Our approach combines PCA eigenanalysis with commutator defect analysis—probing the curvature structure of the loss landscape relative to the learned submanifold: 1. PCA eigenanalysis of attention weight trajectories, revealing the intrinsic dimensionality of the learned submanifold; 2. Commutator defect analysis, measuring loss-landscape curvature and its relationship to the learned submanifold. The commutator defect quantifies the non-commutativity of successive gradient steps: given two mini-batches A and B, the defect measures how much the final parameter vector depends on the order of gradient updates. In a flat region of the loss landscape, gradient steps commute; in a curved region, they do not. By projecting these commutator vectors onto the PCA submanifold, we can determine whether the learned subspace is flat or curved. Key contributions. Our work makes six main contributions, spanning observation, prediction, and causal testing: 1. Rank-1 manifold: The weight-space trajectory during grokking lies on a rank-1 submanifold (68–83% of variance in PC1). 2. Invariant submanifold: This submanifold exhibits strong empirical invariance under the optimization dynamics: commutator defects are predominantly orthogonal to it (ρ≈1.000ρ≈ 1.000 within numerical precision across 36 conditions, with exec/random projection ratio ≈2≈ 2–3×3×), meaning curvature does not deflect the trajectory out of its learned subspace. 3. Temporal ordering: Curvature explodes orthogonally during grokking (1010–1000×1000× increase) and its onset consistently precedes generalization by 600–1600 steps. Non-grokking operations show moderate curvature growth (3030–50×50×) without generalization, so onset is necessary but not sufficient. 4. Scaling law: The lead time between defect onset and generalization obeys a power law Δt∝tgrokα t t_grok^\,α with α=1.27±0.03α=1.27± 0.03 (R2=0.97R^2=0.97) across a 300×300× learning-rate sweep: the predictive window improves from 24% at η=3×10−3η=3\!×\!10^-3 to 95% at η=3×10−5η=3\!×\!10^-5. 5. Causal interventions: Suppressing orthogonal gradient flow prevents grokking (necessary) while boosting curvature defects has no effect (not sufficient), establishing an asymmetric causal relationship. 6. Robustness: All results replicate across the full learning-rate range, a 200×200× timescale difference between regimes, four operations, and three seeds. 7. Spectral mechanism: Weight-SVD analysis reveals that the commutator transition is driven by a transient near-degeneracy of the leading singular values (σ1≈σ2 _1≈ _2) of the attention matrices, followed by symmetry breaking as one mode dominates and the operators align. Our central thesis is that grokking reflects prolonged confinement of optimization dynamics within a low-dimensional subspace of weight space, during which curvature accumulates in transverse directions; generalization emerges as the trajectory exits this metastable regime. The remainder of the paper develops this picture quantitatively and probes its causal structure. Paper outline. We proceed as follows. Section 2 describes the experimental setup. Section 3 describes the geometric tools: PCA eigenanalysis, commutator defect, manifold projection, and trajectory–curvature alignment. Section 4 presents results in three stages: geometric structure (Section 4.1–Section 4.3), predictive power and robustness (Section 4.4–Section 4.6), and causal interventions (Section 4.7). Section 5 identifies the spectral mechanism underlying the commutator transition: a symmetry-breaking event in the singular value spectrum of the attention matrices. Section 6 discusses implications and connections to broader themes. 2 Experimental Setup 2.1 Model and Training We use a Transformer encoder following the canonical grokking setup of Power et al. (2022). The model processes two integer tokens a,b∈0,…,p−1a,b∈\0,…,p-1\ (with p=97p=97) and predicts f(a,b)modpf(a,b) p for a binary operation f. Architecture. The model consists of: • A token embedding Emb:0,…,96→ℝ128Emb:\0,…,96\ ^128 plus a learnable positional embedding ∈ℝ2×128P ^2× 128; • A 2-layer Transformer encoder with pre-norm (LayerNorm before attention and FFN), dmodel=128d_model=128, 4 attention heads, dff=256d_f=256, GELU activation, no dropout; • A final LayerNorm followed by a linear head ℝ128→ℝ97R^128 ^97 applied to the first token position. The total parameter count is approximately 290k. Training. We train with AdamW (β1=0.9 _1=0.9, β2=0.98 _2=0.98) at learning rate 10−310^-3 with weight decay λ=1.0λ=1.0 (or λ=0.0λ=0.0 for non-grokking controls), batch size 512, gradient clipping at 1.0, and a 50/50 train/test split. Training runs for up to 200k steps with early stopping when test accuracy exceeds 98% for 3 consecutive evaluations. We say a model has grokked if test accuracy reaches 98% and remains there for 3 consecutive evaluations (early stopping). For runs that have grokked, we define the grok step tgrokt_grok as the first step at which test accuracy reaches 90%—a lower threshold that captures the onset of generalization rather than its plateau, and is used throughout for lead-time analysis. (No non-grokking operation exceeds 78% test accuracy in any condition, so this threshold does not conflate the two groups.) Operations. We test six binary operations mod 97, four of which exhibit grokking under these hyperparameters and two that do not (Table 1). Table 1: Operations tested. Grok step is the mean step at which test accuracy reaches 90%, averaged over 3 seeds. Operation Formula Groks? Grok step add (a+b)mod97(a+b) 97 Yes ∼ 2900 sub (a−b)mod97(a-b) 97 Yes ∼ 3400 mul (a×b)mod97(a× b) 97 Yes ∼ 2600 x2_y2 (a2+b2)mod97(a^2+b^2) 97 Yes ∼ 1900 x2_xy_y2 (a2+ab+b2)mod97(a^2+ab+b^2) 97 No — x3_xy (a3+ab)mod97(a^3+ab) 97 No — 2.2 Hyperparameter Regimes To test regime invariance, we additionally run a slow regime with qualitatively different hyperparameters: lr=5×10−5lr=5× 10^-5, λ=0.1λ=0.1, 3 Transformer layers, and β2=0.999 _2=0.999. In this regime, grokking occurs at ∼ 570k steps (vs. ∼ 3k in the fast regime), providing a 200×200× difference in training timescale. 2.3 Attention Weight Logging During training, we log the four attention weight matrices—WQW_Q, WKW_K, WVW_V (extracted from the fused in_proj_weight) and WOW_O (out_proj.weight)—every 100 steps. Each matrix is 128×128128× 128 (or 128×32128× 32 per head), giving a trajectory of snapshots for subsequent PCA analysis. 3 Methods 3.1 PCA Eigenanalysis of Weight Trajectories For each attention weight matrix W∈ℝd×dW ^d× d, we collect T training snapshots Wtt=1T\W_t\_t=1^T and compute PCA on the flattened trajectory of weight changes from initialization: X=[vec(W1−W0)⋮vec(WT−W0)]∈ℝT×d2,X= bmatrixvec(W_1-W_0)\\ \\ vec(W_T-W_0) bmatrix ^T× d^2, (1) after centering columns. We compute the SVD X=UΣV⊤X=U V and define the explained variance ratio of the k-th principal component as σk2/∑iσi2 _k^2/ _i _i^2. The quantity PC1% =100×σ12/∑iσi2=100× _1^2/ _i _i^2 measures the fraction of trajectory variance captured by a single direction. Execution manifold. We define the execution subspace (henceforth, execution manifold) as the low-dimensional subspace spanned by the top-K PCA directions of the weight trajectory. Concretely, let VK=[v1,…,vK]V_K=[v_1,…,v_K] denote the top-K right singular vectors of X. The execution manifold is the affine subspace ℳ=W0+VKα:α∈ℝKM=\W_0+V_Kα:α ^K\ through the initial weights W0W_0. When PC1% is high (>70%>70\%), this manifold is effectively rank-1: the weight trajectory is confined to a one-dimensional curve in parameter space. 3.2 Commutator Defect Intuition. If the loss landscape is locally flat, the order in which we apply two gradient updates does not matter: updating with batch A then B gives the same result as B then A. In a curved region, the order matters—just as walking east then north on a sphere leads to a different point than north then east. The commutator defect quantifies this order-dependence, providing a local probe of loss-landscape curvature that requires no Hessian computation. Formal construction. The commutator defect measures loss-landscape curvature by quantifying the non-commutativity of gradient steps from two independent mini-batches. Given the current parameters θ0 _0 and two mini-batches A,BA,B: θAB _AB =θ0−ηgA(θ0)−ηgB(θ0−ηgA(θ0)) = _0-η\,g_A( _0)-η\,g_B( _0-η\,g_A( _0)) (2) θBA _BA =θ0−ηgB(θ0)−ηgA(θ0−ηgB(θ0)) = _0-η\,g_B( _0)-η\,g_A( _0-η\,g_B( _0)) (3) where gA(θ)=∇θℒA(θ)g_A(θ)= _θL_A(θ) is the gradient of the cross-entropy loss on mini-batch A at parameters θ, and η=10−3η=10^-3 is a fixed step size. The (scale-normalized) commutator defect is: =‖θAB−θBA‖ηgA‖⋅‖ηgB‖.D= \| _AB- _BA \| \|η\,g_A \|· \|η\,g_B \|. (4) We justify this via first-order Taylor expansion: to leading order in η, θAB−θBA≈η2(∇gB⋅gA−∇gA⋅gB) _AB- _BA≈η^2(∇ g_B· g_A-∇ g_A· g_B), which is the Lie bracket of the gradient vector fields. D is thus proportional to the Lie bracket of stochastic gradient vector fields, and serves as a proxy for local nonlinearity of the loss landscape: if the landscape is locally flat, gradient steps commute and =0D=0. We compute K=9K=9 independent samples of D at each measurement point and report the median, providing a robust estimate. 3.3 Projection onto the PCA Manifold To determine whether loss-landscape curvature lives inside or outside the learned submanifold, we construct an orthonormal basis B∈ℝP×KB ^P× K for the PCA subspace embedded in full parameter space (P≈290P≈ 290k). For each Transformer layer and each attention weight matrix WQ,WK,WV,WO\W_Q,W_K,W_V,W_O\: 1. Compute the top-2 PCA directions from the weight trajectory (each a vector in ℝd2R^d^2); 2. Embed each direction into the full parameter space at the correct offset; 3. Stack all embedded directions and orthonormalize via QR decomposition. Given a commutator vector δ=θAB−θBAδ= _AB- _BA, we decompose it as: δ=BB⊤δ⏟δ∥(projected)+(δ−BB⊤δ)⏟δ⟂(residual).δ= B\,B δ_ _ \;(projected)+ (δ-B\,B δ)_ _ \;(residual). (5) The invariance measure is the residual fraction: ρ=‖δ⟂‖δ‖.ρ= \| _ \| \|δ \|. (6) If ρ≈1ρ≈ 1, the commutator is orthogonal to the PCA subspace, indicating that loss-landscape curvature is confined to the normal bundle and the execution manifold is empirically approximately invariant under the observed optimization dynamics—curvature does not deflect the trajectory out of its learned subspace. If ρ≈0ρ≈ 0, curvature lies within the learned subspace and the submanifold is not invariant. Transverse decoupling. We say the optimization dynamics are transversely decoupled on the execution manifold ℳM if commutator defect vectors are confined to its normal bundle (ρ≈1ρ≈ 1). In this regime, curvature-induced perturbations act almost entirely orthogonally to ℳM, so that the observed trajectory remains confined to the learned subspace over training. This notion is purely empirical and should not be confused with integrability in the Hamiltonian sense. Throughout, we use “invariant” in an empirical sense: invariance is assessed via near-orthogonality of commutator defects to the learned PCA subspace over finite training windows, rather than in the strict dynamical-systems sense of exact invariance under a continuous flow. Although our analysis uses differential-geometric language, all quantities are computed for discrete-time stochastic optimization; geometric terms are used descriptively rather than in a strict continuous-time sense. 3.4 Random Subspace Control To verify that the near-zero projection fraction 1−ρ1-ρ reflects genuine geometric structure rather than a trivial dimensionality artifact—any K-dimensional subspace of ℝPR^P captures ∼K/P \! K/P of a random vector—we compare the PCA-basis projection against a random baseline. For each commutator vector δ, we compute the projection fraction onto Nrand=5N_rand=5 random K-dimensional orthonormal bases (generated via QR decomposition of Gaussian random matrices) and average the results. The ratio projexec/projrandproj_exec/proj_rand is the key diagnostic: values significantly above 1.01.0 confirm that the PCA subspace captures more commutator energy than expected by chance. Because absolute projection magnitudes vanish in high-dimensional spaces (proj/full∼K/P≪1proj/full K/P 1 for any K-dimensional subspace of ℝPR^P), only normalized comparisons to random subspaces are geometrically meaningful; accordingly, we report ρ alongside the exec/random ratio throughout. 3.5 Converse Analysis: Trajectory Alignment with Curvature As a converse test, we ask whether the weight trajectory avoids high-curvature directions. At each checkpoint, we compute the mean absolute cosine similarity between the trajectory step Δθt=θt−θt−1 _t= _t- _t-1 and K=12K=12 commutator vectors δk\ _k\: c¯t=1K∑k=1K|Δθt⋅δk|‖Δθt‖‖δk‖. c_t= 1K _k=1^K | _t· _k| \| _t \|\, \| _k \|. (7) For comparison, the expected absolute cosine between random vectors in ℝPR^P is 2/(πP)≈1.5×10−3 2/(π P)≈ 1.5× 10^-3 for P=290P=290k. If c¯t c_t is near this baseline, the trajectory is not aligned with curvature directions. 4 Results We present our findings in three stages. First, we establish the geometric structure of grokking: rank-1 weight trajectories on an invariant submanifold with orthogonal curvature explosion (Section 4.1–Section 4.3). Second, we characterize the temporal relationship between curvature growth and generalization, and demonstrate robustness across regimes and learning rates (Section 4.4–Section 4.6). Third, we test causality through targeted interventions (Section 4.7). 4.1 Weight Evolution is Rank-1 PCA on attention weight trajectories reveals that the first principal component captures 68–83% of trajectory variance across all grokking conditions (Figure 1). Weight evolution during grokking is essentially one-dimensional. (a) PC1% for grokking (wd=1.0) vs. no-wd (wd=0.0) across operations. Grokking operations consistently show high PC1%. (b) Top-5 eigenspectrum per operation. The first eigenvalue dominates across all operations. Figure 1: Weight trajectories during grokking are rank-1. (a) PC1% across operations: grokking runs (wd=1.0) show 68–83% variance in a single component. (b) Eigenspectrum showing dominant first eigenvalue. No-weight-decay controls (λ=0λ=0) also show moderately high PC1%, but the null model comparison reveals that grokking PC1% values are 5–20 standard deviations above the random-walk baseline (Figure 2(a)), confirming the concentration is not an artifact of trajectory smoothness. (a) Z-scores vs. random-walk null model. All operations exceed the null by >5σ>5σ. (b) Temporal evolution of PC1% during training. Concentration increases as grokking progresses. Figure 2: PCA concentration is genuine and increases over training. (a) Z-scores above random-walk null. (b) Expanding-window PC1% over training. 4.2 The Execution Manifold Exhibits Empirical Invariance Having established that the weight trajectory lies on a low-dimensional submanifold, we ask: does this submanifold exhibit invariance under the optimization dynamics—does loss-landscape curvature deflect the trajectory out of its learned subspace, or is curvature confined to orthogonal directions? We compute commutator defects at regular checkpoints during training and project each commutator vector onto the PCA basis (Section 3.3). The key result: the residual fraction ρ=‖δ⟂‖/‖δ‖ρ= \| _ \|/ \|δ \| is ≈1.000≈ 1.000 within numerical precision across all 36 conditions (6 operations × 2 weight-decay settings × 3 seeds), as shown in Figure 3. Curvature is confined entirely to the normal bundle of the execution manifold; the submanifold is empirically invariant under the optimization flow. We note that this near-unity value reflects the high dimensionality of parameter space (P≈290P≈ 290k) relative to the PCA subspace (K=16K=16–2424); the exec/random ratio (Section 3.4) provides the complementary test that the small parallel component is geometrically meaningful. (a) Invariance measure: the residual fraction ρ≈1.0ρ≈ 1.0 at every checkpoint, meaning commutator vectors are predominantly orthogonal to the execution manifold. (b) Multi-seed replication: ρ≈1.000ρ≈ 1.000 within numerical precision across all 36 conditions, confirming empirical invariance. Figure 3: The execution manifold exhibits empirical invariance under the optimization dynamics. Commutator defect vectors are predominantly orthogonal to the PCA subspace, with curvature confined to the normal bundle. This means that the dominant component of loss-landscape curvature lies outside the directions the model actually uses for learning. The weight trajectory evolves on an invariant submanifold: despite enormous curvature in the ambient parameter space, the curvature is confined to orthogonal directions and does not deflect the optimization trajectory out of its learned subspace. Random subspace control. To confirm that the near-zero projection onto the PCA basis reflects genuine geometry rather than a dimensionality artifact, we compare against random K-dimensional subspaces (Section 3.4). Figure 4 shows the projection fraction for the PCA (execution) basis and the random baseline over training. Across all four grokking operations, the execution basis captures 1.81.8–2.9×2.9× more commutator energy than a random subspace of equal dimension (K=24K=24), confirming that the small parallel component is geometrically structured. We verify that this ratio is stable under variation of the PCA dimension K: reducing to K=16K=16 or increasing to K=32K=32 yields exec/random ratios within the same range, confirming that the result is not an artifact of the particular choice of K. Crucially, both projections are very small (proj/full <0.05<0.05), consistent with the invariance measure ρ≈1.000ρ≈ 1.000: in a space of ∼ 290k dimensions, any K-dimensional subspace captures negligible energy from a generic vector. The PCA subspace nonetheless captures a structured excess above the random floor, confirming that the near-unity ρ reflects genuine geometric orthogonality rather than merely high dimensionality. (a) Projection fraction (proj/full) for execution basis (green) vs. random baseline (red) over training. The execution basis consistently captures more commutator energy. (b) Combined view: commutator defect (red), exec/random ratio (green), and test accuracy (blue, dashed). The exec/random ratio is consistently above 1.0 during grokking. Figure 4: Random subspace control confirms that the PCA projection is geometrically structured, not a dimensionality artifact. Exec/random ratio ≈1.8≈ 1.8–2.9×2.9× across operations. 4.3 Curvature Explodes Orthogonally During Grokking While the execution manifold remains empirically invariant (curvature confined to the normal bundle), the magnitude of curvature in orthogonal directions changes substantially during grokking. Operations that grok show 10–1000× higher commutator defect than non-grokking controls (Figure 5), and this curvature is concentrated predominantly outside the PCA manifold. (a) Grok (wd=1.0) vs. no-wd (wd=0.0): grokking runs develop substantially higher commutator defect. (b) Commutator defect across all conditions (3 seeds). Grokking operations (wd=1.0) show 10–1000× higher defect. Figure 5: Curvature explodes during grokking but remains orthogonal to the learned subspace. The converse analysis confirms that the weight trajectory does not align with curvature directions: the mean absolute cosine similarity between trajectory steps and commutator vectors is indistinguishable from the random-vector baseline (c¯≈2/(πP) c≈ 2/(π P)), meaning the trajectory actively avoids high-curvature directions. 4.4 Curvature Growth Precedes Generalization A key finding is that the onset of commutator defect growth consistently precedes the generalization transition. We define the defect onset as the first training step at which the commutator defect exceeds 10×10× its early-training baseline (median of the first 3 measurements) and an absolute threshold of 20. Figure 6 shows the temporal overlay of commutator defect and test accuracy for all four grokking operations and two non-grokking controls. In every grokking run, defect begins rising before test accuracy increases. Figure 6: Temporal ordering of curvature growth and generalization. Top four panels: grokking operations (3 seeds each), showing defect (solid) rising before test accuracy (dashed). Bottom two panels: non-grokking controls show moderate defect growth (3030–50×50× baseline) but no generalization. Dotted vertical lines mark defect onset; green regions mark grokking. We define the lead time as Δt=tgrok−tonset t=t_grok-t_onset, the number of training steps between defect onset and grokking, and the lead fraction as Δt/tgrok t/t_grok, the lead time normalized by the grokking timescale. Across all 12 grokking runs (4 operations × 3 seeds), defect onset precedes the point at which test accuracy reaches 90% by 600–1600 steps, with mean lead time of 1117 steps (Figure 7(b)). A one-sided sign test gives p=2−12≈2.4×10−4p=2^-12≈ 2.4× 10^-4, confirming that the temporal ordering is statistically significant. Necessary but not sufficient. Non-grokking operations (a2+ab+b2a^2+ab+b^2 and a3+aba^3+ab) also exhibit defect growth—reaching 3030–50×50× their early-training baseline—but never generalize. Grokking operations, by contrast, reach 500500–2000×2000× baseline, with zero overlap between the two groups in total growth magnitude (minimum grokking: 513×513×; maximum non-grokking: 47×47×). However, this separation is only apparent retrospectively: at early training steps when grokking has not yet occurred, the defect trajectories of grokking and non-grokking operations overlap. Defect onset is therefore best understood as a necessary precondition for grokking—a signal that the loss landscape is reorganizing—rather than a sufficient predictor that discriminates which operations will generalize. The causal intervention experiments (Section 4.7) confirm this interpretation: suppressing orthogonal gradient flow prevents grokking, establishing the mechanistic necessity of curvature growth. (a) Hero example: (a−b)mod97(a-b) 97, seed 137. Defect onset at step 2000, grokking at step 3600 (lead = 1600 steps). (b) Lead time quantification. Left: onset step vs. grok step (all points above diagonal). Right: lead time by operation (sign test p<0.001p<0.001). Figure 7: Temporal ordering: curvature growth precedes generalization. 4.5 Regime Invariance To verify that our findings are not specific to a particular hyperparameter setting, we repeat the full analysis in a slow regime with qualitatively different hyperparameters (Table 2). Table 2: Hyperparameter regimes and key metrics. Metric Fast regime Slow regime Learning rate 10−310^-3 5×10−55× 10^-5 Weight decay 1.0 0.1 Layers 2 3 Adam β2 _2 0.98 0.999 Grok step (add, mean) ∼ 2,900 ∼ 570,000 Invariance (ρ) ≈1.000≈ 1.000 ≈1.000≈ 1.000 Onset precedes grok? 12/12 runs 2/2 runs Lead time (absolute) ∼ 1,100 steps ∼ 558,000 steps Lead time (normalized) ∼ 0.38 ∼ 0.55 Figure 8 shows the slow-regime results. Despite a 200×200× difference in grokking timescale, 10×10× difference in weight decay, and a different number of layers, the same qualitative transition is observed: the execution manifold exhibits empirical invariance (ρ≈1.000ρ≈ 1.000), and defect onset precedes grokking by hundreds of thousands of steps. However, the alignment dynamics differ quantitatively between regimes. In the fast regime, trajectory–curvature alignment is initially above the random baseline and decays toward the transition, consistent with underdamped exploration of parameter space. In the slow regime, alignment remains at or below the random baseline throughout, consistent with overdamped motion along narrow valleys. Both regimes nonetheless exhibit a defect-mediated generalization transition. (a) Regime comparison: invariance, defect, and normalized lead time are consistent across regimes. (b) Slow regime hero: defect onset at ∼ 185k, grokking at ∼ 585k (lead ≈ 400k steps). Figure 8: Regime invariance: qualitative findings replicate in the slow regime (200×200× longer training), though alignment dynamics differ quantitatively. 4.6 Learning Rate Phase Diagram Having established regime invariance between qualitatively different configurations, we now systematically vary the learning rate alone to map the phase boundary of grokking dynamics. We sweep η∈3×10−5, 10−4, 3×10−4, 10−3, 3×10−3, 10−2η∈\3\!×\!10^-5,\,10^-4,\,3\!×\!10^-4,\,10^-3,\,3\!×\!10^-3,\,10^-2\ (half-decade log-uniform) with fixed λ=1.0λ=1.0 across all six operations and three seeds (108 runs total). Figure 9 shows the resulting phase diagram. The grok/no-grok boundary is invariant to learning rate: the same four operations grok at all six rates, while the two complex operations never grok (Figure 9A). Grokking speed scales roughly linearly with η: mean grok steps range from ∼ 136k at η=3×10−5η=3\!×\!10^-5 to ∼ 200 at η=10−2η=10^-2, spanning nearly three orders of magnitude (Figure 9B). The defect landscape reveals a striking asymmetry (Figure 9C): at low learning rate, maximum defect reaches 10410^4, while at high learning rate it drops to ∼ 20–60. This suggests that slower optimization allows curvature to accumulate more in the orthogonal bundle before the phase transition occurs. The predictive lead time (Figure 9D) is largest at η=3×10−5η=3\!×\!10^-5 (∼ 128k steps, 95% of training time) and decreases monotonically with learning rate: 92% at η=10−4η=10^-4, 88% at η=3×10−4η=3\!×\!10^-4, 43% at η=10−3η=10^-3, and 24% at η=3×10−3η=3\!×\!10^-3. A log-log regression across all grokking runs with detectable onset yields a power-law exponent α=1.27±0.03α=1.27± 0.03 (R2=0.97R^2=0.97, n=43n=43; Figure 11), confirming that the lead time grows super-linearly with grokking timescale. This means the predictive window improves at slower, more realistic learning rates. At η≥3×10−3η≥ 3\!×\!10^-3, grokking occurs so rapidly (<<1k steps) that the memorization and generalization phases overlap, and defect onset is concurrent with rather than predictive of grokking. Figure 10 illustrates these regimes for the addition operation. Figure 9: Phase diagram of grokking dynamics across learning rates (η∈3×10−5–10−2η∈\3\!×\!10^-5--10^-2\, λ=1.0λ=1.0, 3 seeds per cell, 108 runs). (A) Grok fraction: the phase boundary between grokking and non-grokking operations is invariant to learning rate. (B) Mean grok step (log scale): grokking speed scales with η. (C) Mean max defect (log scale): curvature explosion is largest at low η. (D) Mean lead time (onset step −- grok step): defect onset is most predictive at low η. Figure 10: Defect and test accuracy trajectories for addition across six learning rates. At low η (e.g., 10−410^-4), defect onset precedes grokking by ∼ 30k steps (92% of training); at high η (≥3×10−3≥ 3\!×\!10^-3), grokking is nearly instantaneous and onset is concurrent. Figure 11: Lead time scaling across learning rates. (A) Power-law fit: lead time ∝ grok stepα with α=1.27±0.03α=1.27± 0.03 (R2=0.97R^2=0.97, n=43n=43). Each point is one grokking run; the dashed line shows the best fit, the dotted line shows linear scaling (α=1α=1). (B) Lead fraction (lead / grok step) increases monotonically as η decreases: from 24% at η=3×10−3η=3\!×\!10^-3 to 95% at η=3×10−5η=3\!×\!10^-5. (C) Onset step vs. grok step for each operation. LR-dependent alignment dynamics. To directly test whether the damping regime varies with learning rate, we measure trajectory–curvature alignment (mean |cos(Δθ,δ)|| ( θ,δ)|, Section 3.5) at four strategic checkpoints—memorization, defect onset, and post-grok—for each of the three learning rates on two operations (Figure 12). At η=10−4η=10^-4, alignment remains below the random baseline (0.180.18–0.86×0.86×) throughout, consistent with overdamped dynamics where the trajectory is confined to narrow valleys far from curvature directions. At η=10−2η=10^-2, alignment is consistently above the baseline (1.41.4–1.8×1.8×), indicating underdamped exploration that initially samples curvature directions. At η=10−3η=10^-3, the intermediate regime, alignment starts below baseline and rises toward or above it at the grokking transition. This LR-dependent pattern replicates across both operations and provides direct evidence for the dynamical regimes discussed below. Figure 12: Trajectory–curvature alignment at three training phases across learning rates. At η=10−4η=10^-4 (blue), alignment stays below the random baseline (dotted), consistent with overdamped dynamics. At η=10−2η=10^-2 (green), alignment exceeds the baseline, consistent with underdamped exploration. η=10−3η=10^-3 (orange) shows intermediate behavior. Both operations exhibit the same pattern. To integrate curvature accumulation and trajectory geometry into a unified picture, we construct a reduced phase portrait using the commutator defect and the trajectory–curvature alignment as coordinates (Figure 13). Each training run traces a characteristic path through this space, progressing from memorization through defect onset to the post-grokking regime. We observe three qualitatively distinct dynamical regimes controlled by the learning rate. At high learning rates, training remains in an underdamped regime, exhibiting strong alignment with curvature directions and low defect accumulation. At low learning rates, training becomes overdamped, with prolonged confinement to low-alignment regions and substantial defect buildup prior to grokking. Intermediate learning rates interpolate between these behaviors, producing critically damped trajectories. Across both addition and multiplication tasks, grokking occurs when trajectories exit a metastable region characterized by high curvature defect and suppressed mobility. This phase portrait provides a compact geometric representation of the grokking transition and clarifies how optimization hyperparameters control the pathway to algorithmic generalization. To our knowledge, this is the first identification of distinct overdamped, critically damped, and underdamped dynamical regimes in grokking, suggesting that the phenomenon possesses a richer phase structure than previously recognized. Figure 13: Phase portrait of grokking dynamics in curvature–trajectory space. We plot the trajectory–curvature alignment ratio (mean |cos(Δθ,δ)||\! ( θ,δ)\!| normalized by random baseline) against the commutator defect magnitude for three learning rates (η=10−4,10−3,10−2η=10^-4,10^-3,10^-2). Each polyline traces the evolution from memorization (mem) through defect onset (spike) to the post-grokking regime (post), shown for addition (circles) and multiplication (squares); arrows indicate the direction of training. The horizontal dashed line indicates isotropic alignment. Training at high learning rates remains in an underdamped regime characterized by strong alignment and low defect, while low learning rates produce overdamped dynamics with large defect accumulation and weak alignment. Intermediate learning rates interpolate between these regimes. Stars mark checkpoints where test accuracy exceeds 90%90\%. Grokking corresponds to escape from a metastable region of high curvature defect and reduced mobility, with regime-dependent relaxation dynamics. 4.7 Causal Interventions on Learning Dynamics The preceding sections establish correlational evidence for the geometric picture. We now test three falsifiable hypotheses through targeted intervention experiments that modify the optimization trajectory while preserving the underlying architecture and dataset: 1. Necessity: Does grokking require motion along the learned execution manifold? 2. Sufficiency: Does artificially inducing transverse curvature suffice to trigger grokking? 3. Specificity: Are these effects specific to the PCA directions, or would any low-dimensional constraint produce the same result? 4.7.1 Gradient Subspace Suppression We first examine whether motion along the learned execution manifold is necessary for grokking. At each training step after step 500 (post-memorization), we project the gradient onto the subspace spanned by the top principal components of the weight trajectory, with projection strength s∈[0,1]s∈[0,1]: g⟶g∥+(1−s)g⟂,g∥=BB⊤g,g⟂=g−g∥,g\; \;g_ +(1-s)\,g_ , g_ =B\,B g, g_ =g-g_ , (8) where B∈ℝP×KB ^P× K is the PCA basis from Phase 1 training. For comparison, we also apply random low-dimensional projections of equal rank (K=16K=16). Partial suppression along the PCA directions (s=0.25s=0.25–0.750.75) systematically delays grokking, while full projection (s=1.0s=1.0) completely prevents generalization (0/12 seeds across four operations; Figure 14). In contrast, random projections have little effect at intermediate strengths (<< 50-step difference from baseline; Figure 15). At s=1.0s=1.0, both projections kill grokking, since confining the optimizer to any 16-dimensional subspace of ℝ290kR^290k is too restrictive. These results indicate that grokking requires access to specific learned directions in parameter space, rather than arbitrary low-dimensional motion. Figure 14: Dose–response curve for gradient projection across all four grokking operations. Each panel shows mean grok step (3 seeds) vs. suppression strength s. At s=1.0s=1.0, grokking fails universally (0/12 seeds). Dashed line: baseline (no intervention). The monotonic delay and complete suppression at full strength replicate across all operations. Figure 15: PCA-specific suppression control. Left: grok step vs. suppression strength for PCA projection (blue) and random projection (green); dashed line is baseline. Right: grok success rate. At intermediate strengths (s=0.25s=0.25–0.750.75), PCA projection monotonically delays grokking while random projection has no effect. At s=1.0s=1.0, both kill grokking (any 16-dim constraint is too restrictive). The dose–response separation confirms the geometric specificity of the PCA manifold. 4.7.2 Directional Forcing and Defect Induction Next, we test whether artificially inducing curvature defects is sufficient to trigger early grokking. Starting at step 500, we periodically inject additive weight updates aligned with the commutator direction (recomputed every 50 steps), with amplitudes α∈50,100,200,500α∈\50,100,200,500\ times the gradient step norm. As a control, we apply kicks of equal magnitude along random orthogonal directions. Across all tested amplitudes, neither commutator-aligned nor random kicks accelerate grokking relative to baseline (Figure 16). All 27/27 runs generalize at statistically indistinguishable times (∼ 3200 steps, within seed-to-seed variability). This negative result demonstrates that defect accumulation alone is insufficient to induce grokking, and that escape from the metastable regime requires coordinated motion along learned directions. Figure 16: Sustained directional kicks along the commutator (red) vs. random orthogonal (gray) directions, with kick magnitudes up to 500×500× the gradient step norm applied every 50 steps. Left: mean grok step (3 seeds); right: grok success rate. Neither direction accelerates grokking beyond baseline variability (dashed line), confirming that the orthogonal defect is not sufficient to induce the phase transition. 4.7.3 Replication Across Operations We repeat the projection experiments across all four grokking tasks: modular addition, subtraction, multiplication, and quadratic addition (a2+b2mod97a^2+b^2 97). The dose–response relationship between projection strength and grokking delay is consistent across all operations (Figure 14), with complete suppression at full strength (0/12 seeds grok at s=1.0s=1.0). At s=0.75s=0.75, grokking is delayed by 600–800 steps (20–25% above baseline) across all four operations. This universality suggests that the causal role of execution-manifold directions is not task-specific, but reflects a common geometric mechanism underlying algorithmic generalization. 4.7.4 Summary of Interventions Taken together, these experiments resolve all three hypotheses: Necessity—confirmed: constraining motion along the execution manifold prevents generalization with a smooth dose–response curve that replicates across four operations. Sufficiency—rejected: artificially increasing defect through directional forcing has no effect. Specificity—confirmed: PCA projection monotonically delays grokking at intermediate strengths where random projection has no effect. This asymmetry is consistent with the commutator defect serving as a signature of the curvature barrier between memorization and generalization solutions—a structured reorganization of the optimization trajectory—rather than a directly manipulable cause of the phase transition. Together, these interventions rule out purely correlational explanations of our earlier findings and establish a directional causal relationship between execution-manifold geometry and generalization. 5 Spectral Mechanism Underlying the Commutator Transition The preceding sections establish that the SGD commutator defect rises sharply before grokking and that orthogonal gradient flow is causally necessary for generalization. We now ask: what drives the commutator dynamics? To answer this, we analyze the singular value spectrum of the attention weight matrices WQW_Q and WKW_K at each training checkpoint. 5.1 Spectral Gaps of the Attention Matrices Let σ1≥σ2≥σ3≥⋯ _1≥ _2≥ _3≥·s denote the singular values of WQW_Q (the analysis for WKW_K is analogous). We define the spectral gaps g12=σ1−σ2,g23=σ2−σ3.g_12= _1- _2, g_23= _2- _3. (9) These quantities track how separated the leading singular modes are: g12≈0g_12≈ 0 indicates near-degeneracy of the top two modes, while g12≫0g_12 0 indicates that one mode dominates. 5.2 A Consistent Spectral Timeline Across all grokking runs (4 operations × 3 seeds), we observe the following consistent temporal sequence: 1. The lower spectral gap g23g_23 gradually shrinks, indicating compression of the sub-leading spectrum. 2. The leading modes become nearly degenerate (σ1≈σ2 _1≈ _2, with g12g_12 reaching a minimum of 0.0020.002–0.020.02). 3. The SGD commutator defect D rises sharply. 4. The matrix commutator ‖[WQ,WK]‖F \|[W_Q,W_K] \|_F reaches its peak. 5. One mode becomes dominant (σ1≫σ2 _1 _2, with g12g_12 increasing 1515–25×25× from its minimum). 6. Test accuracy rapidly increases (grokking). For the representative case of modular addition (seed 42), the ordering is: g23↓⏟step 1400→σ1≈σ2⏟step 1700→↑⏟step 2000→‖[WQ,WK]‖F↑⏟step 2700→σ1≫σ2⏟step 2800→grok⏟step 3100 g_23\! _step 1400\;→\; _1≈ _2_step 1700\;→\; D\! _step 2000\;→\; \|[W_Q,W_K] \|_F\! _step 2700\;→\; _1 _2_step 2800\;→\; grok_step 3100 (10) The near-degeneracy phase (σ1≈σ2 _1≈ _2) corresponds to a regime in which the representation basis is unstable: there is no preferred direction in the top singular subspace, so the optimizer dynamics become highly non-integrable, as reflected in the sharp rise of the SGD commutator defect. Once the symmetry between the top modes breaks and a dominant direction emerges, the operators WQW_Q and WKW_K align into a shared eigenbasis and the matrix commutator collapses. 5.3 Phase Portrait This spectral mechanism is most clearly visible as a trajectory in the (σ1−σ2,‖[WQ,WK]‖F)( _1- _2,\; \|[W_Q,W_K] \|_F) phase plane (Figure 17). Training follows a consistent path through three regimes: • Competition (g12g_12 small, commutator rising): the top singular modes are nearly degenerate and the representation basis is unstable. • Instability (g12g_12 opening, commutator at peak): symmetry breaks as one mode begins to dominate, driving maximal non-commutativity. • Alignment (g12g_12 large, commutator collapsing): WQW_Q and WKW_K converge to a shared basis and grokking occurs. Non-grokking (memorizing) runs, by contrast, show no such loop structure: their trajectories wander diffusely in the same phase space without directional organization (Figure 18). Figure 17: Phase portrait of grokking in the (σ1−σ2,‖[WQ,WK]‖F)( _1- _2,\; \|[W_Q,W_K] \|_F) plane for modular addition (seed 42, layer 0). Training follows a consistent trajectory through three regimes: a competition phase where the top singular modes are nearly degenerate, an instability phase where the commutator peaks as one mode begins to dominate, and an alignment phase where the operators converge to a shared eigenbasis and grokking occurs. Left: colored by training step. Right: colored by test accuracy—the trajectory remains at chance (red) throughout competition and instability, turning green precisely as it enters the alignment region. Figure 18: Phase portraits for grokking (top, wd=1wd=1) vs. memorizing (bottom, wd=0wd=0) across four operations (seed 42). Grokking trajectories exhibit a characteristic loop with directional flow through competition → instability → alignment. Memorizing trajectories are diffuse random walks in the same phase space, with much higher commutator values and no directed escape. Figure 19: All quantities on a common [0,1][0,1] scale (seed 42, layer 0) for the four grokking operations. The spectral gap σ1−σ2 _1- _2 (red) reaches its minimum before the SGD defect spike (green, dashed-dotted), which in turn precedes the matrix commutator peak (purple, dashed). Test accuracy (cyan, dotted) rises last. Vertical dashed line: grokking step. 5.4 Interpretation The spectral timeline provides a mechanistic explanation for the commutator dynamics that were previously only described phenomenologically. The near-degeneracy of σ1 _1 and σ2 _2 creates an orientation instability: when two singular values are close, the corresponding singular vectors are free to rotate under small perturbations. This rotational freedom manifests as large commutator defects, since different mini-batch orderings push the representation basis in different directions. Once the symmetry breaks (σ1≫σ2 _1 _2), the representation locks into a preferred orientation, the gradient ordering ambiguity resolves, and both the matrix commutator and the SGD defect decline. This picture connects grokking to a spectral symmetry-breaking transition in the attention weight matrices: the system passes through a transient near-degeneracy that destabilizes the representation, and grokking corresponds to the resolution of this instability as one mode dominates. Relationship to trajectory PCA. The expanding-window PCA of update trajectories (Section 3.1) captures optimizer dynamics—the covariance structure of successive weight changes—but does not directly reflect the spectral structure of the weight matrices themselves. The weight-SVD analysis in this section provides a complementary and more direct view of the representation transition: it measures the eigenstructure of the actual learned operators rather than the statistics of how they are updated. Connection to the intra-signal gap framework. The spectral timeline above can be situated within the broader intra-signal gap framework (Xu, 2026b), but a precise comparison requires distinguishing three spectral objects that appear in the literature: (i) Rolling-window Gram matrix (Xu’s object): the W×W× W matrix (t)=(t)(t)⊤ G(t)= X(t) X(t) formed from W consecutive parameter updates s=s+1−s δ_s= θ_s+1- θ_s, with signal rank k∗=argmaxjωj⋅(σj/σj+1)k^*= _j\, _j·( _j/ _j+1) where ωj=σj/∑iσi _j= _j/ _i _i weights the ratio by spectral mass at position j. The weighting suppresses spurious tail ratios that dominate the unweighted argmax outside the extreme aspect ratio regime (P≫WP W). The gap ratio at k∗k^* is R=σk∗/σk∗+1R= _k^*/ _k^*+1. (i) Expanding-window displacement PCA (our Section 3.1): PCA on Wt−W0\W_t-W_0\ from initialization, yielding the rank-1 manifold finding (PC1 = 70–94%). (i) Weight-matrix SVD (this section): direct singular values σj(WQ) _j(W_Q) of the learned attention operators, with eigenvalue gap g23=σ22−σ32g_23= _2^2- _3^2. These are fundamentally different matrices with different spectra. We verified this directly across 12 grokking runs (4 operations × 3 seeds, ω=1.0ω=1.0) and 12 matched control runs (ω=0ω=0, same operations): the rolling-window Gram matrix (W=10W=10) yields a gap ratio R=1.40±0.07R=1.40± 0.07 during the pre-grokking phase for ω=1.0ω=1.0, while the ω=0ω=0 controls show higher R=2.83±0.35R=2.83± 0.35 (non-overlapping). This inversion—memorization-only runs having more concentrated update spectra—reflects the structure of pure memorization gradients, which are highly rank-1 aligned, while grokking runs spread updates across spectral modes during circuit formation. The signal rank stabilizes at kterminal∗=1k^*_terminal=1 in 9 of 12 grokking runs (cf. 10/12 in Xu (2026b)), confirming dominant single-mode dynamics at convergence. The Gram-matrix eigenvalue gap g23=σ22−σ32g_23= _2^2- _3^2 is predictive: it peaks during memorization and declines in 12 of 12 grokking runs (mean 40×40×, range 1515–111×111×) before grokking, while only 1 of 12 matched control runs (ω=0ω=0) shows any decline (Table 3). The decline precisely tracks the orientation instability that drives the commutator transition. Table 3: Rolling-window Gram matrix spectral analysis (W=10W=10, mean ± std over 3 seeds). g23earlyg_23^early: peak g23g_23 before grokking; g23grokg_23^grok: g23g_23 at the grokking step; Decline =g23early/g23grok=g_23^early/g_23^grok; R: gap ratio at signal rank k∗k^*. Operation ω Grok step g23earlyg_23^early g23grokg_23^grok Decline RearlyR_early kterm∗k^*_term Decl. a+ba+b 1.0 2500–3100 14.6±0.214.6± 0.2 0.56±0.40.56± 0.4 50×50× 1.40±0.051.40± 0.05 1 ✓ 0.0 — 33.0±9.333.0± 9.3 — — 2.72±0.222.72± 0.22 — — a×ba×b 1.0 2600–3000 14.7±0.114.7± 0.1 0.39±0.20.39± 0.2 49×49× 1.41±0.051.41± 0.05 1 ✓ 0.0 — 23.6±10.723.6± 10.7 — — 2.85±0.112.85± 0.11 — — a−ba-b 1.0 3300–3700 14.4±0.114.4± 0.1 0.48±0.20.48± 0.2 35×35× 1.31±0.021.31± 0.02 1 ✓ 0.0 — 33.2±2.833.2± 2.8 — — 2.62±0.232.62± 0.23 — — a2+b2a^2+b^2 1.0 2000–2500 18.9±0.518.9± 0.5 0.94±0.50.94± 0.5 28×28× 1.47±0.081.47± 0.08 1 ✓ 0.0 — 28.3±13.128.3± 13.1 — — 3.12±0.643.12± 0.64 — — Aggregate All ω=1ω=1 40×40× mean 1.40±0.071.40± 0.07 1 (9/12) 12/12 All ω=0ω=0 — 2.83±0.352.83± 0.35 — 1/12 The connection between these objects operates through the Davis–Kahan sinΘ theorem (Xu, 2026b), which is universal: for any symmetric matrix, the stability of a subspace under perturbation is controlled by ‖sinΘ(,^)‖F≤‖F/δ\| (V, V)\|_F≤\| E\|_F/δ, where δ is the eigenvalue gap. This universality means that the qualitative prediction—gap closing destabilizes the associated subspace, gap opening stabilizes a new direction—applies regardless of whether we analyze the trajectory Gram matrix or the weight matrices directly. What differs between the objects are the quantitative dynamics: Xu’s gap flow ODE, stability coefficient αj _j, and loss decomposition are derived specifically for the trajectory Gram matrix and do not transfer directly to weight-matrix eigenvalues. In our setting, the weight-matrix SVD is the correct diagnostic for three reasons: (a) the eigenvalue gap g23g_23 directly controls the conditioning of the attention operator and hence the model’s representational capacity; (b) g23g_23 decline precedes grokking (lead time ∼1000 1000 steps); (c) g23g_23 decline is reproduced in 12 of 12 grokking runs (mean 40×40×) and in only 1 of 12 matched controls. The decline of g23g_23 is a gap closing event at position k∗=2k^*=2–33: as σ2 _2 approaches σ3 _3, the sub-leading subspace loses its stability, creating the orientation instability that drives the commutator transition. The subsequent emergence of σ1≫σ2 _1 _2 is a gap opening event at position k∗=1k^*=1, stabilizing the dominant direction and enabling generalization. This k∗k^* shift—from position 2–3 to position 1—is confirmed empirically: kterminal∗=1k^*_terminal=1 in 9 of 12 grokking runs using the signal-mass-weighted k∗k^* (the unweighted argmax is unstable in our aspect ratio regime where p/W∼3×104p/W 3× 10^4, compared to P/W∼107P/W 10^7 in Xu’s setting where BBP detection threshold is vacuous). The framework further predicts that weight decay drives the gap dynamics via suppression of low-curvature modes (Xu, 2026b, Theorem 12.22), consistent with the finding that all 12 runs with weight decay grok while only 1 of 12 matched controls without weight decay shows any decline (a late anomalous grokking event at step 41,700). The ω=0ω=0 control further displays the predicted spectral signature of the grokking-absent regime (Xu, 2026b, Remark 12.22): g23g_23 remains high throughout all training steps (mean early g23=14g_23=14–4242, never declining below 1313), confirming that without the curvature floor ω the spectral symmetry-breaking transition never initiates. 5.5 Local Integrability and Basis-Independent Controls The spectral analysis above examines the global singular value structure of individual weight matrices. We now ask whether the commutator’s alignment with weight structure is a structural property of the optimization dynamics or an artifact of a particular choice of basis. Local integrability. At each checkpoint, we construct a joint basis from the top-k right singular vectors of all weight blocks (WQW_Q, WKW_K, WVW_V, WOW_O per layer) and measure the fraction of commutator energy that projects onto this basis (proj/full ratio ρlocal _local). Across all 36 conditions (6 operations × 2 weight-decay settings × 3 seeds), the local integrability metric satisfies ρlocal≈1.0 _local≈ 1.0: nearly all commutator energy lies within the weight-structure subspace. The corresponding random-subspace baseline gives ρrandom≈K/P≈0.005 _random≈ K/P≈ 0.005, yielding an execution/random ratio of 5050–200×200× throughout training (Figure 20). During the memorization phase, ρlocal _local dips to 0.30.3–0.50.5 as commutator energy begins to leave the basis, then recovers to >0.9>0.9 post-grokking. Non-grokking controls (x2+xy+y2x^2+xy+y^2, x3+xyx^3+xy) show persistently lower ρlocal<0.5 _local<0.5 with no phase-dependent recovery. Basis-independent sign flip. To verify that the commutator–subspace relationship is not dependent on the choice of basis, we repeat the projection analysis with three independent basis constructions: (i) weight SVD (top-k singular vectors of W), (i) displacement SVD (top-k singular vectors of Wt−W0W_t-W_0), and (i) gradient SVD (top-k singular vectors of accumulated recent gradients). All three bases exhibit a consistent sign flip: the exec/random ratio exceeds 1 during memorization (commutator aligns with weight structure) and drops below 1 post-grokking (commutator exits the weight-structure subspace). This pattern holds per-block across all attention weight matrices (WQW_Q, WKW_K, WVW_V, WOW_O), with MLP blocks showing a weaker but directionally consistent effect (Figure 21). The basis independence of the sign flip confirms that the commutator’s transition from within-subspace to outside-subspace alignment at grokking is a structural geometric property of the optimization landscape, not an artifact of any particular decomposition. Figure 20: Local integrability: proj/full ratio ρlocal _local over training for grokking operations (solid) and non-grokking controls (dashed). Grokking operations show ρlocal≈1.0 _local≈ 1.0 with a dip during memorization and recovery post-grokking; non-grokking controls remain below 0.50.5. Random baseline (gray band) at K/P≈0.005 K/P≈ 0.005. Figure 21: Basis-independent sign flip: exec/random ratio for three independent bases (weight SVD, displacement SVD, gradient SVD) across training. All three show ratio >1>1 during memorization (commutator aligns with weight structure) and <1<1 post-grokking (commutator exits the subspace), confirming the structural nature of the transition. 6 Discussion and Theoretical Connections Our results support a unified geometric account of grokking: the training trajectory evolves predominantly within a low-dimensional execution subspace, transverse curvature grows in directions orthogonal to this subspace during the memorization plateau, and generalization is associated with escape from this metastable regime. This picture—confinement, barrier growth, escape—organizes all six of our empirical contributions (rank-1 structure, empirical invariance, orthogonal curvature explosion, predictive lead time, causal asymmetry, and regime robustness) under a single dynamical narrative. We now connect this narrative to several broader themes in learning theory and neural network optimization. 6.1 Grokking as Metastable Escape in Curved Landscapes Thesis: grokking is best understood as escape from a metastable regime, not merely delayed learning. Across tasks and hyperparameter regimes, we observe that memorization confines training trajectories to regions of high curvature anisotropy, with commutator defects gradually increasing until a critical level triggers the generalization transition. This is reminiscent of metastable escape in stochastic dynamical systems, where the defect magnitude functions as accumulated geometric tension and the learning rate controls damping. Lower learning rates produce overdamped trajectories requiring substantial defect buildup (∼ 30k steps at η=10−4η=10^-4), while higher learning rates facilitate rapid transitions (∼ 1k steps at η=10−2η=10^-2; Section 4.6). 6.2 A Dynamical-Systems Interpretation Thesis: grokking follows the classical dynamical-systems pattern of slow-manifold formation, transverse instability, and escape. Our findings admit a natural interpretation in the language of dynamical systems, proceeding through four phases: 1. Compression. Early training collapses the weight trajectory onto a low-dimensional slow manifold (PC1 captures 68–83% of variance; Section 4.1). The optimizer finds a rank-1 subspace and confines subsequent motion to it. 2. Transverse instability. Curvature accumulates in the normal bundle of this manifold (1010–1000×1000× defect growth; Section 4.3), while the trajectory remains empirically invariant (ρ≈1.000ρ≈ 1.000; Section 4.2). The system becomes increasingly sensitive to off-manifold perturbations. 3. Critical transition. Weight decay provides a sustained pressure toward lower-norm solutions. Combined with the accumulated transverse curvature, this drives the trajectory through a bifurcation into the generalizing basin. 4. Reorganization. The trajectory settles into a new regime characterized by lower curvature anisotropy along the optimization path—the post-grokking solution. This parallels the classical scenario of slow-manifold collapse followed by transverse bifurcation in dissipative dynamical systems. The spectral analysis of Section 5 further sharpens this picture: the transverse instability phase corresponds to a near-degeneracy of the leading singular values of WQW_Q and WKW_K, and the critical transition corresponds to a spectral symmetry-breaking event in which one singular mode dominates and the attention operators align into a shared eigenbasis. The learning rate controls the damping: low η produces overdamped dynamics with gradual defect buildup, high η produces underdamped dynamics with rapid transitions (Section 4.6). Limitations of the dynamical-systems analogy. While this interpretation is supported by the temporal ordering (curvature precedes generalization) and the causal necessity of orthogonal gradient flow (Section 4.7), the available geometric signals—commutator defect, trajectory–curvature alignment, and invariance measure—do not combine into a predictive composite diagnostic that discriminates grokking from non-grokking operations in advance. The invariance measure is identically 1 across all conditions; trajectory–curvature alignment is noisy and overlapping; and defect magnitude only separates grokking from non-grokking operations retrospectively. Moreover, the same grokking operations trained without weight decay exhibit comparable defect acceleration without generalizing, confirming that curvature growth alone is insufficient—regularization pressure is the additional ingredient. This raises the possibility that local curvature measurements alone may not suffice to predict grokking. Local geometric diagnostics can detect that the loss landscape is reorganizing—transverse instability is growing, curvature is accumulating in the normal bundle—but whether this reorganization leads to generalization likely depends also on global properties of the landscape (such as the existence of a low-norm generalizing basin) and on exogenous factors like regularization strength. Understanding how local curvature dynamics interact with global landscape structure to produce the grokking transition remains an important open question. 6.3 Implicit Regularization and Low-Dimensional Structure Thesis: implicit regularization in grokking operates through the emergence of task-specific geometric structure, not merely norm or margin control. Following grokking, trajectories collapse onto low-dimensional execution manifolds (PC1 explains 68–83% of variance; Section 4.1), accompanied by reduced curvature anisotropy along the trajectory. The intervention experiments (Section 4.7) demonstrate that motion along these specific learned directions is necessary for grokking, while generic low-dimensional constraints are insufficient. This suggests that implicit regularization in this setting operates through the progressive emergence of geometrically privileged subspaces. 6.4 Scaling Behavior and Phase Diagrams Thesis: grokking exhibits a phase diagram with power-law scaling, paralleling phenomena in larger-scale systems. Our learning-rate sweeps across six half-decade-spaced rates reveal distinct overdamped, critically damped, and underdamped regimes (Section 4.6), with grok time scaling approximately as tgrok∝η−1t_grok η^-1 and the curvature defect serving as an order parameter (sign test p=2−12p=2^-12; Section 4.4). A power-law fit across 43 grokking runs with detectable onset yields α=1.27±0.03α=1.27± 0.03 (R2=0.97R^2=0.97; Figure 11), confirming that the defect onset lead time grows super-linearly with grokking timescale. The predictive window improves monotonically at slower, more realistic learning rates—from 24% at η=3×10−3η=3\!×\!10^-3 to 95% at η=3×10−5η=3\!×\!10^-5. While our experiments operate in a small-model regime (∼ 290k parameters), these scaling relationships parallel phenomena reported in large-scale language models, suggesting that grokking may represent a microscopic instance of optimization-driven phase transitions. 6.5 Robustness, Flat Minima, and Quantization Recent empirical work has demonstrated that trained neural networks can tolerate substantial parameter quantization and compression with limited performance degradation. Our results provide a geometric perspective on this robustness in the grokking regime. Post-grokking solutions are characterized by reduced curvature anisotropy along the optimization trajectory: the commutator defect, while still nonzero, is concentrated orthogonally to the directions the optimizer traverses, and the invariance measure ρ≈1.000ρ≈ 1.000 (within numerical precision) indicates that curvature is confined to the normal bundle of the execution manifold. Such regions, where curvature is confined to directions not visited by the optimizer, naturally support greater robustness to perturbations along the learned subspace. In contrast, pre-grokking solutions reside in highly anisotropic, high-defect regions and are correspondingly more sensitive to perturbation. This suggests that robustness to compression may be partly understood as a consequence of geometric reorganization during training, rather than solely as a byproduct of architectural or regularization choices. 6.6 Connection to Mechanistic Interpretability Thesis: the geometric transition during grokking corresponds to the stabilization of interpretable circuits. The formation of execution manifolds corresponds to the concentration of computation into low-dimensional subspaces, consistent with the emergence of interpretable circuits documented in prior work. The collapse of curvature anisotropy during grokking indicates that these circuits become geometrically stabilized. From this viewpoint, grokking marks the transition from distributed representations to structured, circuit-like organization—providing a potential bridge between optimization geometry and mechanistic interpretability. 6.7 Limitations and Open Problems Our experiments are limited to relatively small Transformer models (2–3 layers, ∼ 290k parameters) and synthetic algorithmic tasks (modular arithmetic mod 97). While these settings permit fine-grained geometric analysis, it remains unclear to what extent the observed phenomena—rank-1 manifolds, invariant execution submanifolds, predictive defect onset—generalize to large-scale language models and real-world datasets. Preliminary experiments on Dyck languages and the SCAN compositional-generalization benchmark suggest that qualitatively similar low-dimensional confinement and transverse curvature dynamics arise beyond modular arithmetic; a systematic investigation of these settings is ongoing. In addition, several of our diagnostic measures, including commutator defects (4 forward-backward passes per sample) and trajectory–curvature alignment, are computationally expensive and difficult to scale. Developing efficient approximations and proxies for these geometric diagnostics remains an important direction for future work. Finally, a complete theoretical characterization of the observed phase transitions remains open. Deriving analytical models that capture defect accumulation, manifold formation, and damping-controlled dynamics represents a promising avenue for future research. 6.8 Outlook Taken together, our results suggest that grokking reflects a geometric reorganization of the optimization landscape, governed by curvature, damping, and emergent low-dimensional structure. By integrating dynamical, geometric, and causal analyses, this work provides a foundation for understanding delayed generalization as a phase transition in learning dynamics. The commutator defect serves as a diagnostic for ongoing loss-landscape reorganization: monitoring it during training can detect geometric changes hundreds to tens of thousands of steps before they manifest in accuracy metrics. However, defect growth alone does not discriminate which runs will ultimately generalize—non-grokking operations and unregularized training both exhibit curvature growth without generalization. The diagnostic value lies in detecting that the landscape is reorganizing, not in predicting whether grokking will occur. Developing composite diagnostics that combine curvature dynamics with regularization-sensitive signals remains an open direction. We hope that this perspective will inform future studies of optimization, scaling, robustness, and interpretability in neural networks. 7 Related Work Grokking. Power et al. (2022) first observed delayed generalization in modular arithmetic. Nanda et al. (2023) identified “grokking circuits” (Fourier-basis representations) in 1-layer models. Liu et al. (2022) showed that grokking occurs broadly when weight decay or weight norm is controlled. Zhong et al. (2024) described clock and pizza representations in modular addition. Thilak et al. (2022) connected grokking to slingshot dynamics in adaptive optimizers. Lyu et al. (2024) characterized the role of weight decay in separating memorization from generalization phases. More recently, Merrill et al. (2023) framed grokking as competition between sparse and dense subnetworks, Varma et al. (2023) explained it through circuit efficiency, Davies et al. (2023) connected grokking to double descent, and Kumar et al. (2024) characterized the lazy-to-rich training transition. Concurrently, Montanari and Wang (2026) established sharp phase transitions for feature learning in two-layer networks under proportional asymptotics, providing a theoretical framework for understanding when gradient descent discovers low-dimensional structure. Our work complements these representational and optimization-theoretic perspectives with a geometric and causal analysis. Loss landscape geometry and scaling. The study of loss landscape geometry in neural networks has a rich history (Li et al., 2018b; Draxler et al., 2018; Garipov et al., 2018). Fort and Jastrzebski (2019) studied the curvature of the loss landscape during training. Our commutator defect is related to the Lie bracket of gradient vector fields and measures non-commutativity of the optimization flow; similar ideas appear in the study of natural gradient methods (Amari, 1998) and the Fisher information geometry of neural networks. The power-law scaling we observe in grok time vs. learning rate resonates with the broader scaling laws literature (Kaplan et al., 2020), though our analysis operates at a much smaller scale. Intrinsic dimensionality. Li et al. (2018a) showed that neural network optimization occurs in a low-dimensional subspace. Xu (2026a) demonstrated that attention weight trajectories during grokking in modular arithmetic lie on a low-dimensional execution manifold, with PC1 capturing the majority of variance. A corrected version of that work includes random-subspace baseline controls showing that the execution basis captures 22–10×10× more commutator energy than a random subspace of equal dimension. The present work extends those findings by establishing that the execution manifold exhibits empirical invariance under the optimization dynamics (curvature confined to the normal bundle), adding analogous random baseline controls, demonstrating that curvature dynamics predict the generalization transition, and testing the causal role of orthogonal gradient flow through intervention experiments. 8 Conclusion We have shown that the weight-space trajectory during grokking lies on a rank-1 empirically invariant submanifold of parameter space—the execution manifold—and that loss-landscape curvature is confined to the normal bundle of this submanifold. Curvature growth in the normal bundle consistently precedes generalization by 600–1600 steps, establishing a robust temporal ordering; however, non-grokking operations also exhibit moderate curvature growth without generalizing, so onset is a necessary precondition rather than a sufficient predictor. These findings hold across six modular arithmetic operations, three random seeds, two weight-decay settings, a 100×100× learning rate sweep, and two qualitatively different hyperparameter regimes (200×200× range in training timescale). Causal intervention experiments close the loop: suppressing orthogonal gradient flow prevents grokking with a monotonic dose–response across four operations, while artificially boosting curvature defects has no effect, establishing that normal-bundle curvature growth is mechanistically necessary for the generalization transition. Weight-SVD analysis reveals the mechanism underlying these dynamics: grokking is preceded by a transient near-degeneracy of the leading singular values of the attention matrices (σ1≈σ2 _1≈ _2), during which the representation basis is unstable and the commutator defect rises sharply. Generalization coincides with the breaking of this spectral symmetry as one mode dominates and the operators align. Local integrability analysis confirms that the commutator energy is almost entirely contained within the weight-structure subspace (ρlocal≈1.0 _local≈ 1.0), and this alignment is basis-independent: three independent decompositions all exhibit a sign flip at grokking, confirming the structural nature of the transition. The geometric picture—empirically invariant execution manifold, orthogonal curvature confinement, spectral symmetry-breaking, basis-independent integrability, and causal confirmation via interventions—provides a new lens for understanding the grokking phenomenon and suggests that monitoring gradient non-commutativity during training may serve as a diagnostic for ongoing loss-landscape reorganization, even when generalization is not yet observable in accuracy metrics. Reproducibility. All code and figures are available at https://github.com/skydancerosel/grokking-integrability. Total compute for full reproduction is approximately 9 hours on a single Apple M-series machine. References Amari [1998] Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998. Davies et al. [2023] Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent. arXiv preprint arXiv:2303.06173, 2023. Draxler et al. [2018] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pages 1309–1318, 2018. Fort and Jastrzebski [2019] Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. In Advances in Neural Information Processing Systems, volume 32, 2019. Garipov et al. [2018] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems, volume 31, 2018. Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Kumar et al. [2024] Tanishq Kumar, Blake Bordelon, Samuel J Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. arXiv preprint arXiv:2310.06110, 2024. Li et al. [2018a] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018a. Li et al. [2018b] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, volume 31, 2018b. Liu et al. [2022] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, and Mike Williams. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117, 2022. Lyu et al. [2024] Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S Du, Jason D Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking. arXiv preprint arXiv:2311.18817, 2024. Merrill et al. [2023] William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. arXiv preprint arXiv:2303.11873, 2023. Montanari and Wang [2026] Andrea Montanari and Zihao Wang. Phase transitions for feature learning in neural networks. arXiv preprint arXiv:2602.01434, 2026. URL https://arxiv.org/abs/2602.01434. Nanda et al. [2023] Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. Power et al. [2022] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR 2022 Workshop on MATH-AI, 2022. URL https://arxiv.org/abs/2201.02177. Thilak et al. [2022] Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. arXiv preprint arXiv:2206.04817, 2022. Varma et al. [2023] Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Neel Nanda. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390, 2023. Xu [2026a] Yongzhong Xu. Low-dimensional execution manifolds in transformer learning dynamics: Evidence from modular arithmetic tasks. arXiv preprint arXiv:2602.10496, 2026a. URL https://arxiv.org/abs/2602.10496. Xu [2026b] Yongzhong Xu. The spectral edge thesis: Intra-signal gap dynamics in transformer training. arXiv preprint arXiv:2603.28964, 2026b. URL https://arxiv.org/abs/2603.28964. Zhong et al. [2024] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024. Appendix A Additional Figures Figure 22: PC1% heatmap by operation and weight matrix (last layer, wd=1.0). All weight matrices show high PC1% for grokking operations. Figure 23: Per-weight-matrix PC1% comparison across operations (grok vs. no-wd). (a) Slow vs. fast regime PC1%. The slow regime shows lower PC1%, but still well above the null model. (b) Decomposition of PC1% drop between regimes: which hyperparameter drives the difference. Figure 24: Regime comparison for PCA concentration. (a) Combined view: defect magnitude and invariance measure over training. (b) Attention weight fraction of commutator defect. Figure 25: Commutator analysis details. (a) Trajectory-curvature alignment over training. (b) Alignment ratio vs. random baseline: the trajectory does not prefer curvature directions. Figure 26: Converse analysis: the weight trajectory avoids high-curvature directions. Figure 27: Temporal traces for (a+b)mod97(a+b) 97 with 3 seed overlays, showing consistency of the invariance and defect patterns across seeds.