Paper deep dive
Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts
Truong Xuan Khanh, Truong Quynh Hoa
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/13/2026, 12:32:26 AM
Summary
The paper introduces the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning in neural networks as a consequence of parameter norm dynamics under regularized optimization. It proves that weight decay forces a transition from high-norm shortcut solutions to lower-norm structured representations, providing a tight logarithmic bound on the transition delay and identifying three distinct regimes of regularization strength.
Entities (5)
Relation Signals (3)
Weight Decay → induces → Norm-Hierarchy Transition
confidence 95% · weight decay induces a slow contraction from high-norm shortcut solutions toward lower-norm structured representations.
Norm-Hierarchy Transition → explains → Grokking
confidence 90% · The results suggest that grokking, shortcut learning, and delayed feature discovery arise from a common mechanism based on norm hierarchy traversal during training.
Clean Norm Separation → predicts → Implicit Bias Timescales
confidence 90% · this condition... precisely delineates settings where implicit bias timescales are predictable from those where they are not.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions and that neural networks exhibit simplicity bias, but neither explains the timescale of the transition from shortcut features to structured representations. We introduce the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning as the slow traversal of a hierarchy of parameter norms during regularized optimization. When multiple interpolating solutions exist with different norms, weight decay gradually moves the model from high norm shortcut solutions toward lower norm structured representations. We derive a tight bound showing that the transition delay grows logarithmically with the ratio between shortcut and structured norms. Experiments on modular arithmetic, CIFAR-10 with spurious features, CelebA, and Waterbirds support the predictions of the framework. The results suggest that grokking, shortcut learning, and delayed feature discovery arise from a common mechanism based on norm hierarchy traversal during training.
Tags
Links
- Source: https://arxiv.org/abs/2603.07323v1
- Canonical: https://arxiv.org/abs/2603.07323v1
Trouble viewing inline? Open PDF directly →
Full Text
63,914 characters extracted from source content.
Expand or collapse full text
Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts Truong Xuan Khanh Truong Quynh Hoa11footnotemark: 1 H&K Research Studio, Clevix LLC, Hanoi, Vietnam khanh@clevix.vn Co-first authors with equal contribution (March 2026) Abstract Neural networks often rely on spurious shortcuts for hundreds of epochs before discovering structured representations. Yet the mechanism governing when this transition occurs—and whether its timing can be predicted—remains poorly understood. While prior work has established that gradient descent converges to low-norm solutions (Soudry et al., 2018) and that networks exhibit simplicity bias (Shah et al., 2020), neither line of work characterises the timescale of the transition from simple to structured features. We propose a unifying framework—the Norm-Hierarchy Transition—which explains delayed representation learning as the slow traversal of a norm hierarchy under regularised optimisation. When multiple interpolating solutions exist with different norms, weight decay induces a slow contraction from high-norm shortcut solutions toward lower-norm structured representations. We prove a tight bound on the transition delay: T=Θ(γeff−1log(Vsc/Vst))T= ( _eff^-1 (V_sc/V_st)), where VscV_sc and VstV_st are the characteristic norms of the shortcut and structured representations. The framework predicts three regimes as a function of regularisation strength: weak regularisation (shortcuts persist), intermediate regularisation (delayed transition), and strong regularisation (learning suppressed). We validate these predictions across four domains: modular arithmetic (where all six predictions hold with R2>0.97R^2>0.97), CIFAR-10 with spurious features (five of six, including 78%→10%78\%→ 10\% clean accuracy as shortcut strength increases), CelebA (which occupies an intermediate position on the norm separation spectrum), and Waterbirds (where norm dynamics transfer but representational transition does not, confirming the framework’s boundary). The norm-hierarchy mechanism is robust across architectures: ResNet18 with standard batch normalisation exhibits the same peak-then-decay norm dynamics as models without normalisation, achieving 78%78\% clean accuracy. The single prediction that fails to transfer—the precise delay scaling T∝1/λT 1/λ—is explained by a new condition we term clean norm separation, the first formal criterion distinguishing settings where implicit bias timescales are predictable from those where they are not. The framework further predicts that emergent abilities in large language models arise when model scale reduces the norm gap below a training-budget threshold, connecting scaling laws to the same norm-hierarchy mechanism. Our results suggest that grokking, shortcut learning, delayed feature discovery, and emergent abilities are manifestations of a single mechanism: the slow traversal of a norm hierarchy under regularised optimisation. 1 Introduction Why do neural networks sometimes rely on shortcuts for hundreds of epochs before discovering real features? This delayed transition appears across several apparently unrelated phenomena: models exploit spurious correlations before learning causal features (Sagawa et al., 2020; Geirhos et al., 2020), grokking produces sudden generalisation long after memorisation (Power et al., 2022; Nanda et al., 2023), and simplicity bias causes networks to prefer shallow features before discovering compositional structure (Shah et al., 2020). Despite their diversity, these phenomena share a common pattern: delayed representational transition, in which the network dwells in an initial representation before shifting to a qualitatively different one. What mechanism governs this delay? When does a network abandon its shortcut, and can this transition be predicted from the optimisation dynamics? This paper. We propose that delayed representational transitions are a predictable consequence of parameter norm dynamics under regularised training. The central insight is simple: when a learning system admits multiple interpolating representations with different norms, weight decay creates a directed pressure from high-norm (shortcut) to low-norm (structured) solutions. The time required for this transition is governed by the norm gap between the two representations. Crucially, the framework is not domain-specific: the same logarithmic delay law governs grokking in algorithmic tasks, shortcut learning in vision, and—as we argue in Section 6.2—emergent abilities in large language models. We formalise this into the Norm-Hierarchy Transition Law: Ttransition=Θ(1γefflogVscVst),T_transition= \! ( 1 _eff V_scV_st ), (1) where VscV_sc and VstV_st are the characteristic norms of the shortcut and structured representations, and γeff _eff is the effective contraction rate of the optimiser. The bound is tight: we prove a matching lower bound showing that no first-order regularised algorithm can transition faster. Three regimes. The framework predicts three qualitatively distinct regimes as a function of regularisation strength λ: 1. Weak regularisation: The model reaches the shortcut solution and stays there. 2. Intermediate regularisation: The model first reaches the shortcut, then undergoes a delayed transition to the structured representation. This is the regime where grokking and shortcut-to-structure transitions occur. 3. Strong regularisation: Weight decay overwhelms learning. The model never reaches any interpolating solution. Contributions. Our core contributions are three: 1. Norm-Hierarchy Transition framework. We identify the minimal structural conditions (multi-representation interpolation, norm hierarchy, shortcut accessibility) that are jointly sufficient for delayed representational transitions across grokking, shortcut learning, and related phenomena (Section 2). 2. Tight delay law with matching bounds. We prove Ttransition=Θ(γeff−1log(Vsc/Vst))T_transition= \! ( _eff^-1 (V_sc/V_st) ) with both a Lyapunov upper bound and a matching information-theoretic lower bound, showing the law is optimal for all first-order regularised algorithms (Section 3). 3. Multi-domain validation with explicit failure diagnostics. We validate the framework on four domains (CIFAR-10, Waterbirds, CelebA, modular arithmetic), introduce the Clean Norm Separation Score as a formal criterion predicting when the framework applies, and demonstrate architecture robustness across four model variants including ResNet18+BatchNorm (Sections 5–5.8). Two additional contributions emerge from the analysis: 4. Layer-wise norm hierarchy (Proposition 4.2): layers closer to the output escape the shortcut manifold faster, predicting a backward representational transition. 5. Emergent abilities hypothesis (Section 6.2): the delay law generates four testable predictions connecting NHT to capability emergence in large language models. Positioning. Table 1 situates our contribution relative to existing work. Table 1: Comparison with existing approaches. Our framework is the first to provide tight delay bounds validated across multiple domains with explicit failure diagnostics. Formal theory Tight bounds Multi-domain Predicts delay Predicts failure Power et al. 2022 Nanda et al. 2023 Soudry et al. 2018 ✓ Shah et al. 2020 ✓ Sagawa et al. 2020 ✓ Chizat & Bach 2019 ✓ Companion paper ✓ ✓ ✓ ✓ This paper ✓ ✓ ✓ ✓ ✓ The remainder of the paper is organised as follows. Section 2 establishes the framework. Section 3 states and proves the main theorem. Section 4 introduces clean norm separation and the layer-wise norm hierarchy. Section 5 presents experimental validation. Section 6 discusses implications, connections to scaling laws, and limitations. Section 7 surveys related work. Section 8 concludes. 2 Framework and Assumptions We study regularised gradient-based training in the overparameterised regime. Our goal is to identify minimal conditions under which a delayed representational transition occurs as a consequence of norm dynamics alone. 2.1 Setup Consider a parameterised model fθf_θ trained to minimise a loss ℒtrain(θ)L_train(θ) with ℓ2 _2 regularisation: θt+1=θt−η(∇ℒtrain(θt)+2λθt)+ηξt, _t+1= _t-η ( _train( _t)+2λ _t )+η _t, (2) where η is the learning rate, λ>0λ>0 is the weight decay coefficient, and ξt _t is zero-mean noise with [‖ξt‖2|ℱt]≤σ2E[\| _t\|^2|F_t]≤σ^2. 2.2 Structural Assumptions Definition 2.1 (Multi-Representation Interpolation). The training problem admits multi-representation interpolation if the interpolation manifold ℳ=θ:ℒtrain(θ)=0M=\θ:L_train(θ)=0\ contains at least two geometrically distinct regions: a shortcut region ℳscM_sc relying on spurious features, and a structured region ℳstM_st capturing the true data-generating mechanism. Definition 2.2 (Norm Hierarchy). The interpolation exhibits a norm hierarchy if there exist constants Vsc>Vst>0V_sc>V_st>0 such that ‖θ‖2≥Vsc\|θ\|^2≥ V_sc for all θ∈ℳscθ _sc and ‖θ∗‖2≤Vst\|θ^*\|^2≤ V_st for some θ∗∈ℳstθ^* _st. The norm gap is ΔV=Vsc−Vst>0 V=V_sc-V_st>0. Remark 2.3 (Why shortcut solutions tend to have larger norm). The ordering Vsc>VstV_sc>V_st reflects a structural property of how spurious features are encoded, not an ad hoc assumption. Shortcut strategies concentrate predictive power in a small number of highly discriminative directions—a border colour, a background texture, a hair-colour signal—requiring large weights in those channels to achieve low training loss. Structured representations distribute predictive information across many features, yielding smaller individual magnitudes and a lower total squared norm. This is consistent with the implicit bias of gradient descent toward minimum-norm interpolators (Soudry et al., 2018; Lyu and Li, 2020): the structured solution is precisely the one that minimum-norm bias would select in the unregularised limit. Empirically, Vsc≫VstV_sc V_st is confirmed across all four domains in Section 5, with norm ratios ranging from 3×3× (CelebA) to 37×37× (CIFAR-10). Definition 2.4 (Shortcut Accessibility). The optimiser reaches the shortcut region first if there exists a finite time TscT_sc such that ℒtrain(θTsc)≤ϵ0L_train( _T_sc)≤ _0 and ‖θTsc‖2≥Vsc\| _T_sc\|^2≥ V_sc. Remark 2.5 (On Shortcut Accessibility). Assumption (A5) asserts that the optimiser reaches the shortcut manifold ℳscM_sc before the structured manifold ℳstM_st. This is justified on three complementary grounds. (i) Norm proximity. From a standard random initialisation, ‖θ0‖2\| _0\|^2 is (d−1)O(d^-1) (He/Xavier init), which is far below both VscV_sc and VstV_st. Since the shortcut solution has lower norm (Vsc<VstV_sc<V_st by the norm hierarchy assumption), the optimiser reaches ℳscM_sc after fewer contraction steps than ℳstM_st, giving Tsc<TstT_sc<T_st with high probability under mild smoothness conditions. (i) Simplicity bias of gradient descent. A substantial body of work establishes that gradient-based optimisers exhibit a spectral or frequency bias: they fit low-complexity functions before high-complexity ones (Rahaman et al., 2019; Shah et al., 2020; Valle-Perez et al., 2019). Shortcut features (spurious correlations, border textures, hair colour) are by definition simpler—they require fewer Fourier modes or fewer hierarchical features—so they are accessible to gradient descent earlier in training. (i) Loss landscape geometry. Shortcut solutions typically occupy flat, wide basins in the loss landscape (Dinh et al., 2017), making them attractors for SGD from a wide range of initialisations. Structured solutions, by contrast, require traversing narrower saddle-point regions. Together, these arguments support (A5) as a natural consequence of the optimisation geometry rather than an ad hoc assumption. Formalising tight conditions under which (A5) holds—particularly for deep networks without explicit regularisation—remains an interesting open problem. Remark 2.6 (Instantiations). The framework instantiates naturally across domains. In CIFAR-10 with spurious borders (Section 5—our primary validation), ℳscM_sc consists of border-reliant classifiers and ℳstM_st consists of texture/shape classifiers; all six NHT predictions are confirmed quantitatively. In modular arithmetic (Power et al., 2022), the framework provides supporting evidence: ℳscM_sc is the memorisation manifold (‖θ‖2=Θ(p)\|θ\|^2= (p)) and ℳstM_st is the Fourier manifold (‖θ‖2=Θ(K)\|θ\|^2= (K)), consistent with the norm-hierarchy prediction; a full empirical study is deferred to concurrent work. In CelebA (Section 5.10), ℳscM_sc corresponds to hair-colour-reliant classifiers; this domain illustrates the framework’s boundary via the clean norm separation condition. 2.3 Technical Conditions (A1) ℒtrainL_train is L-smooth. (A2) On ℳM, ∇ℒtrain(θ)=0 _train(θ)=0. (A3) η≤λ/Lη≤λ/L. (A4) Multi-representation interpolation with norm hierarchy holds. (A5) Shortcut accessibility holds. 3 The Norm-Hierarchy Transition Theorem 3.1 Lyapunov Contraction Theorem 3.1 (Generalised Escape under Regularisation). Under (A1)–(A3), for θt∈ℳsc _t _sc, Vt=‖θt‖2V_t=\| _t\|^2 satisfies: [Vt+1|ℱt]≤(1−ηλ)Vt+η2σ2.E[V_t+1|F_t]≤(1-ηλ)V_t+η^2σ^2. (3) The escape time satisfies Tescape=Θ(γeff−1log(Vsc/Vst))T_escape= ( _eff^-1 (V_sc/V_st) ), where γeff=ηλ _eff=ηλ for SGD and γeff≥ηλ _eff≥ηλ for AdamW. Proof. See Appendix A. ∎ Remark 3.2 (Intuition). Weight decay acts as a contraction force on the parameter norm, causing trajectories to gradually escape high-norm shortcut solutions toward lower-norm structured representations. The logarithmic form of the escape time reflects the multiplicative nature of the contraction: each step reduces the norm by a factor of (1−ηλ)(1-ηλ), so traversing a ratio Vsc/VstV_sc/V_st requires logarithmically many steps. Remark 3.3 (Robustness to idealised assumptions). Theorem 3.1 assumes exact interpolation (A2) and the step-size bound (A3) for analytical tractability. In practice, the delay mechanism requires only two weaker conditions: (i) approximate stationarity of the loss gradient near ℳscM_sc, so that the norm contraction term dominates over gradient steps for a sustained interval; and (i) that λ is large enough to overcome noise-driven norm growth (ηλ>η2σ2/Vscηλ>η^2σ^2/V_sc, which is implied by (A3)). Empirically, the predicted three-regime structure and logarithmic delay scaling are clearly visible even when training loss remains non-zero throughout (Section 5), confirming that the transition dynamics are robust to the exact-interpolation idealisation. 3.2 Lower Bound Theorem 3.4 (Dynamical Lower Bound). Under (A1)–(A4), any first-order regularised algorithm transitioning from ℳscM_sc to ℳstM_st requires at least Ttransition=Ω((ηλ)−1log(Vsc/Vst))T_transition= ((ηλ)^-1 (V_sc/V_st) ) gradient steps. Proof. See Appendix B. ∎ 3.3 The Norm-Hierarchy Transition Law Theorem 3.5 (Norm-Hierarchy Transition Law). Under (A1)–(A5), if a learning system exhibits multi-representation interpolation with norm hierarchy, then training produces a delayed representational transition with delay Ttransition=Θ(1γefflogVscVst),T_transition= \! ( 1 _eff V_scV_st ), (4) and training exhibits three regimes: weak λ (shortcuts persist), intermediate λ (delayed transition), strong λ (learning suppressed). Corollary 3.6 (Grokking as Special Case). Setting ℳsc=ℳmemM_sc=M_mem and ℳst=ℳpostM_st=M_post recovers the Norm-Separation Delay Law: Tgrok−Tmem=Θ((ηλ)−1log(‖θmem‖2/‖θpost‖2))T_grok-T_mem= ((ηλ)^-1 (\| _mem\|^2/\| _post\|^2) ). Corollary 3.7 (Image Classification). Setting ℳscM_sc as border-reliant classifiers and ℳstM_st as texture/shape classifiers predicts a delayed transition from shortcut reliance to real feature learning, with delay controlled by weight decay. 4 Clean Norm Separation and Layer-Wise Norm Hierarchy 4.1 Clean Norm Separation The Norm-Hierarchy Transition Law makes two predictions: (1) qualitative—a delayed transition occurs; (2) quantitative—the delay scales as Θ(γeff−1log(Vsc/Vst)) ( _eff^-1 (V_sc/V_st)). Our experiments reveal that prediction (1) transfers robustly, while prediction (2) transfers only under an additional condition that we now formalise. Definition 4.1 (Clean Norm Separation). The norm hierarchy exhibits clean separation if there exists a scalar function ϕ(θ)φ(θ) such that: (i) ϕ=1φ=1 on ℳscM_sc and ϕ=0φ=0 on ℳstM_st; (i) ϕφ is monotonically related to ‖θ‖2\|θ\|^2 near ℳM; (i) the transition ϕ(θt):1→0φ( _t):1→ 0 is sharp relative to TescapeT_escape. This condition, to our knowledge the first formal criterion of its kind, precisely delineates settings where implicit bias timescales are predictable from those where they are not. 4.2 Layer-Wise Norm Hierarchy The framework so far treats the parameter vector θ as a monolithic object, using the total norm ‖θ‖2\|θ\|^2 as the proxy for representation state. However, a neural network is a composition of layers, each encoding different aspects of the input. This raises a natural question: does the norm-hierarchy transition propagate uniformly through the network, or does it proceed in a directed, layer-dependent fashion? Our experimental findings (Section 5.7, Figure 4) reveal a striking asymmetry: under intermediate regularisation, the final classification layer contracts before and faster than the early convolutional layers. We now formalise this observation. Setup. Partition the parameters into L layers: θ=(θ(1),…,θ(L))θ=(θ^(1),…,θ^(L)), where θ(ℓ)∈ℝdℓθ^( ) ^d_ and ∑ℓdℓ=d _ d_ =d. Define the per-layer norm Vt(ℓ)=‖θt(ℓ)‖2V^( )_t=\|θ^( )_t\|^2. Shortcut encoding capacity. Let αℓ∈[0,1] _ ∈[0,1] denote the shortcut encoding capacity of layer ℓ : the fraction of the shortcut’s predictive signal that is encoded in θ(ℓ)θ^( ) at the time the network reaches ℳscM_sc. Layers closer to the output (large ℓ ) combine features linearly into a class decision, so αL≥αℓ _L≥ _ for all ℓ<L <L whenever the shortcut is a low-level feature. Proposition 4.2 (Layer-Wise Norm Hierarchy). Suppose Assumptions (A1)–(A5) hold and the gradient of the shortcut loss decomposes approximately as ∇θ(ℓ)ℒsc(θ)≈αℓ⋅g(θ) _θ^( )L_sc(θ)≈ _ · g(θ) for some shared vector g(θ)g(θ) at ℳscM_sc. Then, under ℓ2 _2 regularisation with coefficient λ, the per-layer escape time satisfies: Tescape(ℓ)=Θ(1γeff(ℓ)logVsc(ℓ)Vst(ℓ)),γeff(ℓ)=ηλ+ηαℓκℓ,T^( )_escape= \! ( 1 _eff^( ) V^( )_scV^( )_st ), _eff^( )=ηλ+η _ _ , where κℓ≥0 _ ≥ 0 is the curvature of the loss in the direction of θ(ℓ)θ^( ) near ℳscM_sc. Consequently: αℓ′>αℓ⟹Tescape(ℓ′)<Tescape(ℓ). _ > _ \; \;T^( )_escape<T^( )_escape. In particular, the classification head escapes before early feature layers: Tescape(L)<Tescape(ℓ)T^(L)_escape<T^( )_escape for all ℓ<L <L. Proof sketch. Weight decay acts identically on all layers (−2ηλθ(ℓ)-2ηλθ^( )), but the loss gradient ∇θ(ℓ)ℒ _θ^( )L provides an additional contraction force proportional to αℓ _ . Applying the Lyapunov argument of Theorem 3.1 per layer, the effective contraction rate for layer ℓ is γeff(ℓ)=ηλ+ηαℓκℓ≥ηλ _eff^( )=ηλ+η _ _ ≥ηλ, strictly greater for layers with αℓ>0 _ >0. See Appendix D for the full proof. ∎ Remark 4.3 (Backward transition). Proposition 4.2 predicts that the representational transition proceeds from output toward input: the classification head abandons the shortcut first, propagating a changed gradient signal to earlier layers. Figure 4(c) provides direct evidence: the fc layer contracts 45%45\% from its peak while conv1 contracts only 31%31\%, consistent with αL>α1 _L> _1. Remark 4.4 (Practical diagnostic). If the classification head’s norm is contracting but early layer norms are not, the network is mid-transition. The classification head norm ‖θ(L)‖2/‖θ0(L)‖2\|θ^(L)\|^2/\|θ^(L)_0\|^2 is a more sensitive early-warning indicator than total norm, which is dominated by growing early layers. 5 Experimental Validation 5.1 Setup: Colored CIFAR-10 We construct a modified CIFAR-10 in which a colored border correlates with the class label. Each class is assigned a unique border color; during training, the correct color appears with probability ρ. We evaluate on three test sets: colored (same distribution), clean (no borders), and shortcut (borders on gray images). Our primary model is a 6-layer CNN (∼ 288K parameters) without batch normalisation, trained with AdamW and cosine annealing. Section 5.8 extends the analysis to ResNet18 with and without batch normalisation. Full details are in Appendix C. 5.2 Experiment A: Weight Decay Sweep We train with ρ=0.95ρ=0.95 and vary λ∈0.001,0.01,0.05,0.1,0.3,0.5,1.0λ∈\0.001,0.01,0.05,0.1,0.3,0.5,1.0\. Figure 1 confirms the three-regime prediction: • Weak λ (0.001–0.01): Norm grows to 5,000–14,000 with <1%<1\% decay. Clean accuracy reaches 58–69%. • Intermediate λ (0.05–0.3): Norm peaks then decays (up to 21.6% at λ=0.3λ=0.3). Clean accuracy reaches 55–58%. • Strong λ (0.5–1.0): Norm suppressed (42–64% decay). Clean accuracy drops to 26–44%. The quantitative delay law T∝1/λT 1/λ does not hold (r=−0.43r=-0.43), consistent with the absence of clean norm separation (Section 4). Figure 1: Three-regime structure under weight-decay sweep (CIFAR-10, ρ=0.95ρ=0.95, 7 values of λ). Left: ‖θ‖2\|θ\|^2 trajectory over 200 epochs. Weak λ (0.0010.001–0.010.01): norm grows monotonically to 5,000–14,000 with <1%<1\% decay, indicating persistent shortcut reliance. Intermediate λ (0.050.05–0.30.3): norm peaks then decays up to 21.6%21.6\%, the signature of a delayed NHT. Strong λ (0.50.5–1.01.0): norm suppressed from epoch 1, decaying 4242–64%64\%, indicating learning suppression. Right: Final clean accuracy vs. λ (circles) and norm-decay percentage (crosses). Clean accuracy peaks in the intermediate regime (≈58%≈58\%), confirming that the delayed transition corresponds to real-feature acquisition. Error bars: ±1± 1 std over 4 seeds. 5.3 Experiment B: Correlation Sweep We fix λ=0.1λ=0.1 and vary ρ∈0.5,0.8,0.95,1.0ρ∈\0.5,0.8,0.95,1.0\. Figure 2 shows a monotonic relationship: Correlation ρ Clean accuracy Shortcut reliance 0.5 (weak shortcut) 78.2% 0.4 0.8 74.8% 0.7 0.95 (strong shortcut) 58.5%±2.3%58.5\%± 2.3\% 0.7 1.0 (perfect shortcut) 10.2% 1.0 At ρ=1.0ρ=1.0, the shortcut perfectly predicts the label, so the model never transitions to real features—a strong negative control. Figure 2: Effect of spurious correlation strength ρ on transition outcome (CIFAR-10, λ=0.1λ=0.1). Each point shows mean clean accuracy at epoch 200 across 3 seeds. Monotonically decreasing accuracy with increasing ρ confirms the NHT prediction: stronger shortcuts produce larger norm gaps (Vsc/VstV_sc/V_st), delaying the transition and ultimately preventing it entirely at ρ=1.0ρ=1.0 (shortcut reliance =1.0=1.0, clean accuracy =10.2%=10.2\%). The shaded band marks the intermediate-regime operating point (ρ=0.95ρ=0.95) used in Experiments A and C. 5.4 Experiment C: Reproducibility Four seeds at λ=0.1λ=0.1, ρ=0.95ρ=0.95 yield mean clean accuracy 58.5%±2.3%58.5\%± 2.3\%, with all runs showing norm peak-then-decay. 5.5 Prediction Summary Table 2: Predictions tested on CIFAR-10 (13 runs total). Prediction Confirmed? Evidence Three-regime structure ✓ Fig. 1 Norm peak-then-decay ✓ λ≥0.05λ≥ 0.05, up to 64% decay Stronger λ→λ→ more contraction ✓ Monotonic, 7 values Stronger shortcut → harder transition ✓ 78%→10%78\%→ 10\% Reproducibility ✓ 4 seeds, std =2.3%=2.3\% Delay T∝1/λT 1/λ × See Section 4 5.6 Representation Phase Diagram Figure 3 presents a representation phase diagram constructed from our experimental sweeps. The diagram reveals four regimes consistent with the Norm-Hierarchy framework: structured feature learning (weak shortcuts, moderate λ), norm-hierarchy transition (intermediate λ and ρ), shortcut dominated (strong ρ), and suppressed/underfit (excessive λ). Figure 3: Representation phase diagram over the (λ,ρ)(λ,ρ) plane (CIFAR-10, 28 runs). Colour encodes final clean accuracy; contour lines separate the four predicted regimes. Shortcut-dominated (bottom-right, high ρ, low λ): network stays on ℳscM_sc; clean accuracy ≤15%≤ 15\%. NHT regime (centre): delayed transition produces clean accuracy 5555–78%78\%. Structured (top-left, low ρ, moderate λ): network reaches ℳstM_st directly; clean accuracy ≥75%≥ 75\%. Suppressed (top-right, high λ): weight decay overwhelms learning; accuracy ≤44%≤ 44\%. The phase boundary between shortcut-dominated and NHT regimes sharpens with increasing ρ, consistent with the norm-gap prediction ΔV∝ρ V ρ. 5.7 Layer-Wise Norm Analysis Figure 4 reveals a finding theoretically predicted by Proposition 4.2: at λ=0.1λ=0.1, the classification head (fc) contracts 45%45\% from its peak while conv1 contracts only 31%31\%—even as total norm grows 73%73\%. Figure 4: Layer-wise norm dynamics reveal a backward representational transition (CIFAR-10, λ=0.1λ=0.1, ρ=0.95ρ=0.95). (a) Per-layer ‖θ(ℓ)‖2\|θ^( )\|^2 normalised to peak value. The classification head (fc, purple) reaches its peak at epoch ≈60≈60 and contracts 45%45\% by epoch 200; conv1 (blue) peaks later and contracts only 31%31\%; intermediate layers fall between. (b) Total norm ‖θ‖2\|θ\|^2 (black) grows throughout, masking the layer-level contraction in head layers. (c) Ratio ‖θ(L)‖2/‖θ(1)‖2\|θ^(L)\|^2/\|θ^(1)\|^2 decreases monotonically after epoch 60, providing a layer-ratio diagnostic that detects the transition when total norm monitoring would fail. Shaded band: ±1± 1 std over 4 seeds. These results directly confirm Proposition 4.2: layers with higher shortcut encoding capacity αℓ _ escape the shortcut manifold faster, producing a backward transition from output to input. This dissociation has a precise theoretical interpretation. The shortcut (coloured border) requires only a linear transformation at the output layer, giving the head the highest shortcut encoding capacity αL _L. By Proposition 4.2, this produces the largest effective contraction rate γeff(L) _eff^(L) and shortest escape time. The transition proceeds backward through the network: the head abandons the shortcut first, altering the gradient signal to earlier layers. A practical implication: monitor ‖θ(L)‖2/‖θ0(L)‖2\|θ^(L)\|^2/\|θ^(L)_0\|^2 during training; peak-then-decay indicates the transition regime even when total norm is still increasing. 5.8 Architecture and Normalisation Ablation A central question for any theory of norm-hierarchy transitions is whether the predicted dynamics are an artefact of a specific architecture or whether they generalise across model families and normalisation strategies. We address this with a controlled ablation on Coloured CIFAR-10 (ρ=0.95ρ=0.95, λ=0.1λ=0.1, seed 42, 200 epochs) using four model variants that vary architecture depth and normalisation layer while keeping all other hyperparameters fixed. Models. (i) SimpleCNN (no norm): the six-layer baseline used throughout Section 5.1 (287,850 parameters). (i) ResNet18 (no norm): standard ResNet-18 with all BatchNorm layers replaced by Identity (11,164,362 parameters). (i) ResNet18 + GroupNorm: ResNet-18 with BatchNorm replaced by GroupNorm (32 groups; 11,173,962 parameters). (iv) ResNet18 + BatchNorm: standard ResNet-18 (11,173,962 parameters). All models use AdamW (η=10−3η=10^-3, λ=0.1λ=0.1, cosine schedule, 200 epochs). Architecture-agnostic norm dynamics. All four variants exhibit the predicted non-monotone norm trajectory—a peak at epoch 50–60 followed by sustained decay—confirming that the Lyapunov contraction mechanism of Theorem 3.1 does not depend on a specific architecture. Absolute norm scales differ by orders of magnitude (Vpeak=2,222V_peak=2,222 for SimpleCNN vs. 43,69043,690 for ResNet18), but the qualitative three-phase dynamics are preserved across all variants. BatchNorm accelerates and amplifies the transition. ResNet18+BN achieves clean accuracy 78.1%78.1\% at epoch 200, compared to 69.8%69.8\% for the identically-trained ResNet18 without normalisation—an absolute improvement of +8.3+8.3 percentage points. The norm peak occurs at epoch 50 for ResNet18+BN vs. epoch 60 for the no-normalisation variant, indicating a faster escape from the shortcut basin. This is consistent with the NHT framework: BatchNorm re-scales the effective weight at each layer by the inverse running standard deviation, amplifying the regularisation pressure on high-variance (shortcut-encoding) channels and thus increasing γeff _eff in Theorem 3.1. GroupNorm anomaly. Despite producing norm decay (67%67\%) comparable to BatchNorm (66%66\%), ResNet18+GroupNorm achieves only 50.5%50.5\% clean accuracy—lower than the no-normalisation baseline (69.8%69.8\%) and below even SimpleCNN (61.6%61.6\%). This dissociation between norm decay and accuracy improvement reveals a nuance in the theory: norm contraction is necessary but not sufficient for a successful transition. GroupNorm normalises within spatial positions inside each channel, preserving relative channel magnitudes and allowing shortcut-encoding dimensions to maintain high activation even as the total norm shrinks. BatchNorm, by contrast, normalises across the batch and applies learnable scale/shift parameters that interact directly with the AdamW weight-decay penalty, creating channel-specific regularisation pressure that drives the hierarchy transition. Formally, the effective weight-decay coefficient for channel c under BatchNorm is amplified by σc−2 _c^-2 (where σc _c is the running standard deviation), so channels with high variance—those that encode the shortcut—experience disproportionately stronger contraction. GroupNorm lacks this σc−2 _c^-2 amplification, explaining why total norm decays equally but the shortcut is not selectively removed. Three-regime preservation under BatchNorm. Across a weight-decay sweep with ResNet18+BN (λ∈0.01,0.1,1.0λ∈\0.01,0.1,1.0\), the three regimes are clearly visible: at λ=0.01λ=0.01 the norm grows monotonically (decay 4%4\%) and clean accuracy reaches 73.9%73.9\%; at λ=0.1λ=0.1 peak-then-decay (66%66\%) yields the best accuracy 78.1%78.1\%; at λ=1.0λ=1.0 the norm collapses from epoch 1 (97%97\% decay) and accuracy drops to 74.7%74.7\%. The ResNet18 (no norm) sweep reproduces the same inverted-U pattern (60.0%60.0\%, 69.8%69.8\%, 49.5%49.5\%), with BatchNorm uniformly shifting the curve upward, consistent with Theorem 3.1 where higher γeff _eff widens the intermediate regime. 5.9 Validation on Waterbirds Dataset and setup. We evaluate NHT on Waterbirds (Sagawa et al., 2020), where bird species (waterbird vs. landbird) is correlated with background (water vs. land) at 95%95\% in the training split. We train SimpleCNN (the same architecture as in Section 5.1, adapted to 224×224224× 224 input) with AdamW using λ∈0.0001,0.001,0.01,0.1,0.3,1.0λ∈\0.0001,0.001,0.01,0.1,0.3,1.0\ for 100 epochs (η=10−3η=10^-3, cosine schedule), with three seeds at λ=0.1λ=0.1. Norm ordering (P1). Final parameter norms are monotonically ordered by λ: ‖θ‖2\|θ\|^2 decreases from 937937 at λ=0.0001λ=0.0001 to 276276 at λ=1.0λ=1.0, confirming the inverse relationship of Theorem 3.1. Norm dynamics (P2). For λ≤0.1λ≤ 0.1 the norm grows monotonically throughout training (decay 0%0\%, peak at final epoch), placing all these runs in the weak regime. For λ≥0.3λ≥ 0.3 norm decay becomes measurable (6.7%6.7\% at λ=0.3λ=0.3; 56.3%56.3\% at λ=1.0λ=1.0). The absence of a clear intermediate regime at λ=0.1λ=0.1 suggests that γeff _eff is lower for Waterbirds than for Coloured CIFAR-10 at the same nominal λ, consistent with the higher dimensionality (224×224224× 224) and richer background statistics that require more model capacity. Worst-group accuracy (P3). Worst-group accuracy is uniformly low across all λ: WG ≈7.2≈ 7.2–9.3%9.3\% with no statistically significant ordering (three seeds at λ=0.1λ=0.1: WG =8.2%±0.7%=8.2\%± 0.7\%). This negative result—WG does not improve with intermediate regularisation—is predicted by the clean norm separation analysis. Norm separation analysis. The Norm Separation Score is S≈0.0S≈ 0.0, placing Waterbirds in Scenario C (no separation), comparable to CelebA (S=−0.11S=-0.11) and in contrast to modular arithmetic (S≈1.0S≈ 1.0) and CIFAR-10 (S≈0.5S≈ 0.5–0.70.7). The Waterbirds spurious feature (background texture) is encoded at every scale of the convolutional hierarchy—from low-level colour and texture to mid-level scene context—so Assumption 4.1 is violated throughout, and no norm-hierarchy transition can improve WG accuracy. Informative negative result. The 1/4 confirmed prediction rate on Waterbirds (P1 only) is itself informative: NHT makes the sharp prediction that transitions will not improve WG accuracy when S≤0S≤ 0, and the results are consistent with this prediction. The framework correctly abstains from predicting group-robustness improvement on datasets that violate the structural preconditions, rather than making false positive predictions. A natural follow-up would replace SimpleCNN with ResNet18+BN, which the ablation of Section 5.8 shows achieves substantially higher clean accuracy and faster transitions; we leave this to future work. 5.10 Validation on CelebA We evaluate NHT on CelebA (Liu et al., 2015), a large-scale face attribute dataset containing 202,599 images. Following the group-DRO literature (Sagawa et al., 2019), we define the target label as Smiling and treat Blond Hair as a spurious attribute, yielding four groups: Blond, Not Blond × Smiling, Not Smiling. We train a six-layer SimpleCNN (286,818 parameters, 64×6464×64 input) with AdamW, cosine learning-rate decay, and weight decay λ∈0.001,0.1,1.0λ∈\0.001,0.1,1.0\ for 100 epochs (η=10−3η=10^-3). Each configuration is repeated over two random seeds; all metrics are reported as mean ± std. Norm-hierarchy predictions. (P1) Norm ordering. Final parameter norms are well-separated across regimes: ‖θ‖λ=0.0012=20,198≫‖θ‖λ=0.12=2,971≫‖θ‖λ=1.02=546\|θ\|^2_λ=0.001=20,198 \|θ\|^2_λ=0.1=2,971 \|θ\|^2_λ=1.0=546, a ratio of 37×37× between the weak and strong regimes, consistent with Theorem 3.1. (P2) Three norm regimes. All three qualitative behaviours predicted by NHT are observed (Figure 5a). Under weak regularisation (λ=0.001λ=0.001) the norm grows monotonically (decay <1%<1\%), indicating unchecked shortcut persistence. Under intermediate regularisation (λ=0.1λ=0.1) the norm trajectory is non-monotone (5 and 4 out of 20 logged intervals show norm decrease across seeds), consistent with a delayed transition. Under strong regularisation (λ=1.0λ=1.0) the norm is sharply suppressed from epoch 1 onward, with mean decay of 5.4%5.4\% in seed 42, confirming the over-regularised regime. (P3) Worst-group accuracy ordering. Worst-group accuracies follow the predicted monotone ordering (Figure 5b): WG(λ=0.001)=89.1%>WG(λ=0.1)=88.0%>WG(λ=1.0)=87.0%.WG(λ=0.001)=89.1\%>WG(λ=0.1)=88.0\%>WG(λ=1.0)=87.0\%. (5) Average accuracies are similarly ordered (90.9%90.9\%, 90.5%90.5\%, 90.2%90.2\%), confirming that stronger regularisation does not improve robustness in this setting. (P4) Layer-wise norm hierarchy (Proposition 4.2). The fc/conv1 norm ratio increases monotonically with λ: 0.65×→2.21×→2.33×0.65×→ 2.21×→ 2.33×. Under weak regularisation the input convolutional layer accumulates more norm than the output layer; under strong regularisation the output layer dominates. This is the backward-transition pattern predicted by Proposition 4.2: output layers with higher shortcut encoding capacity αℓ _ escape the shortcut basin faster under stronger regularisation. Norm separation analysis. The Norm Separation Score (Definition 4.1) is S=−0.11S=-0.11, placing CelebA in Scenario C (no separation), comparable to Waterbirds (S≈0S≈ 0) and in sharp contrast to modular arithmetic (S≈1.0S≈ 1.0) and CIFAR-10 (S≈0.5S≈ 0.5–0.70.7). This negative result is informative rather than anomalous. The Smiling/Blond-Hair pair does not satisfy clean norm separation because both features require similar mid-level texture representations; consequently γeff _eff is not meaningfully larger for the shortcut path than for the target path, and Theorem 3.1 predicts no clean transition. This confirms that clean norm separation is a necessary condition for predictable NHT dynamics. Figure 5: CelebA norm-hierarchy validation across three regularisation regimes (6 runs: 3 λ values × 2 seeds). (a) Parameter norm ‖θ‖2\|θ\|^2: The three regimes are clearly separated — weak λ=0.001λ=0.001 (blue) grows monotonically to ≈20,000≈20,000; intermediate λ=0.1λ=0.1 (orange) plateaus near 3,0003,000 with non-monotone dynamics (5/20 intervals show norm decrease); strong λ=1.0λ=1.0 (red) is suppressed below 1,0001,000 from epoch 1. Norm ratio between weak and strong regimes: 37×37×, confirming P1. (b) Worst-group accuracy: All three regimes show stable performance above 86%86\% after epoch 20, with the predicted monotone ordering 89.1%>88.0%>87.0%89.1\%>88.0\%>87.0\% (P3 confirmed). The absence of a sharp accuracy jump at intermediate λ is consistent with Scenario C (no clean norm separation, S=−0.11S=-0.11): there is no delayed transition to observe. (c) Average accuracy: Similarly ordered (90.9%>90.5%>90.2%90.9\%>90.5\%>90.2\%), with larger variance under strong regularisation (std =1.2%=1.2\% vs. 0.1%0.1\%). The fc/conv1 norm ratio increases from 0.65×0.65× to 2.33×2.33× as λ increases (P4 confirmed, not shown), providing the first empirical confirmation of Proposition 4.2 on a real face-attribute dataset. Summary across four domains. Table 3: Transfer of NHT predictions across four domains. P1: norm ordering; P2: three regimes; P3: WG-acc ordering; P4: layer hierarchy; P5: clean norm separation; P6: delay scaling. Predictive power degrades gracefully with the norm separation score S. Domain Confirmed P1 P2 P3 P4 P5 P6 S Modular Arithmetic 6/6 ✓ ✓ ✓ ✓ ✓ ✓ ≈1.0≈ 1.0 CIFAR-10 (spurious) 5/6 ✓ ✓ ✓ ✓ ✓ × 0.50.5–0.70.7 CelebA 4/6 ✓ ✓ ✓ ✓ × × −0.11-0.11 Waterbirds 2/6 ✓ ✓ × × × × ≈0.0≈ 0.0 6 Discussion 6.1 Unifying Delayed Representation Learning The Norm-Hierarchy Transition recasts several phenomena as instances of a single mechanism: Grokking. Shortcut == memorisation; structure == Fourier features. Delay == grokking gap. Validated quantitatively (R2>0.97R^2>0.97; consistent with the norm-hierarchy prediction, with full empirical details deferred to concurrent work). Shortcut learning. Shortcut == spurious feature; structure == real feature. Three-regime structure confirmed on CIFAR-10 (this paper). Simplicity bias. Simple features correspond to the first interpolating solution found (high-norm shortcut); complex features require norm contraction to reach (Shah et al., 2020). Lazy-to-rich transition. The transition from kernel-like to feature-learning behaviour (Chizat and Bach, 2019) maps to traversal of a norm hierarchy, with our framework providing a concrete timescale. 6.2 Connections to Scaling Laws and Emergent Abilities A striking empirical regularity in large language models is the phenomenon of emergent abilities: capabilities absent in smaller models that appear abruptly as model scale increases (Wei et al., 2022; Srivastava et al., 2022). This abruptness has resisted explanation within standard scaling law frameworks (Kaplan et al., 2020), which predict smooth power-law improvement. We propose a hypothesis: the Norm-Hierarchy Transition may provide a natural mechanistic account of why emergence appears sudden, by reframing the timing of capability jumps as a consequence of norm dynamics under implicit regularisation. We stress that this is a theoretical conjecture generating testable predictions; empirical verification at scale is left to future work. The norm-hierarchy interpretation of emergence. Consider a model trained on a task admitting two strategies: a shortcut strategy (surface-level pattern matching) and a structured strategy (compositional reasoning). Both interpolate the training data, but the shortcut solution has lower norm. Under this interpretation, the NHT framework suggests: As model size N increases, the norm gap ΔV(N)=Vsc(N)−Vst(N) V(N)=V_sc(N)-V_st(N) may decrease. If so, the transition delay Ttransition∝log(Vsc/Vst)T_transition (V_sc/V_st) decreases accordingly, and the structured representation is reached within the training budget only above a critical model size N∗N^*. Why emergence appears sharp. If the above holds, the transition completes if and only if: 1γefflogVscVst≤B, 1 _eff V_scV_st≤ B, (6) where B is the training budget. As N increases and ΔV(N) V(N) shrinks, the left-hand side crosses B at a critical N∗N^*, producing an apparent threshold effect—without any discontinuity in the loss landscape. This provides an alternative to metric-artefact explanations (Schaeffer et al., 2023), though the two accounts are not mutually exclusive. Connection to grokking. Grokking and emergent abilities are two faces of the same hypothesised mechanism: grokking varies training time at fixed scale; emergence varies scale at fixed training time. Testable predictions. The hypothesis generates four concrete predictions that can be evaluated without training frontier models: (i) Norm gap decreases with scale. Vsc(N)/Vst(N)V_sc(N)/V_st(N) should decrease monotonically with model size for tasks with structured solutions. (i) Emergence threshold shifts with training budget. Training with stronger weight decay should lower N∗N^*; very weak regularisation should raise it or eliminate emergence. (i) Norm peak-then-decay precedes emergent capability. Parameter norms should exhibit peak-then-decay before task accuracy jumps, providing a model-agnostic early warning signal. (iv) Clean norm separation predicts which abilities emerge cleanly. Tasks with norm-separable strategies should exhibit sharper, more predictable emergence thresholds; entangled tasks should not. These predictions follow directly from equation (1) without additional assumptions and could be tested on mid-scale models (e.g. 70M–1B parameters) where emergence has been documented (Wei et al., 2022). Remark 6.1 (Relation to the “mirage” debate). Schaeffer et al. (2023) argued that apparent emergent abilities are artefacts of nonlinear evaluation metrics. The NHT account is compatible with both positions: under linear metrics, the transition is smooth but fast; under threshold metrics, the same transition produces an apparent discontinuity. Crucially, norm peak-then-decay is metric-independent and provides an empirical way to adjudicate the debate without committing to a particular metric. 6.3 Practical Implications 1. Diagnosing shortcuts: Monotonically growing norm suggests the weak-λ regime, retaining shortcuts. 2. Setting λ: The optimal weight decay lies in the intermediate regime where norm peaks then decays. 3. Normalisation compatibility: ResNet18+BN exhibits the same peak-then-decay dynamics as models without normalisation. 4. Layer monitoring: Classification head norm is a more sensitive early-warning indicator than total norm. 6.4 Limitations • Quantitative delay law: T∝1/λT 1/λ does not transfer to CIFAR-10, motivating feature-wise norm decompositions. • Total norm as proxy: ‖θ‖2\|θ\|^2 conflates shortcut and structured features; full feature-wise decomposition is future work. • NLP and large-scale domains: Testing on NLP tasks and larger architectures (ViT) would further strengthen generality. • Regime specificity: Theory assumes ℓ2 _2 regularisation; dropout may produce transitions through different pathways. • Shortcut accessibility: Formalising conditions under which (A5) holds is an open problem. 6.5 Future Directions Feature-wise norm decomposition, empirical verification of norm-gap scaling with model size (Section 6.2 predictions (i)–(iv)), broader validation (NLP tasks), and deriving norm ratios a priori from task properties are promising directions. 7 Related Work We organise prior work into four strands: grokking and delayed generalisation, shortcut learning, implicit bias, and scaling laws with emergent abilities. Grokking and delayed generalisation. First documented by Power et al. (2022) and studied mechanistically by Nanda et al. (2023) and Chughtai et al. (2023). Barak et al. (2022) showed delayed learning of parities. Concurrent work on modular arithmetic provides supporting evidence for tight delay bounds; the present work generalises to arbitrary transitions and provides the first multi-domain empirical validation. Shortcut learning. Documented by Geirhos et al. (2020); mitigated by Sagawa et al. (2020) via Group DRO. Nam et al. (2020) and Liu et al. (2021) develop debiasing methods that implicitly rely on representational transitions but provide no mechanistic account of their timing. Our framework provides the missing dynamical mechanism. Implicit bias. GD converges to minimum-norm solutions (Soudry et al., 2018; Lyu and Li, 2020). Chizat and Bach (2019) characterises the lazy-to-rich transition. Lyu et al. (2023) identifies early and late phases. We extend this by characterising the timescale: implicit bias is slow, and the slowness is quantifiable via ΔV V. Scaling laws and emergent abilities. Kaplan et al. (2020) established power-law scaling. Wei et al. (2022) documented sharp capability emergence; Srivastava et al. (2022) provided systematic benchmarks. Schaeffer et al. (2023) questioned whether emergence is a metric artefact. Our framework offers a mechanistic account orthogonal to the metrics debate (Section 6.2). To our knowledge, this is the first connection between the grokking and shortcut-learning literature and the emergent abilities literature via a shared dynamical mechanism. Phase transitions and double descent. Belkin et al. (2019) and Nakkiran et al. (2021) showed non-monotonic generalisation curves. Our three-regime structure provides a complementary perspective on regularisation in interpolation geometry. Position relative to existing theories. Existing work on grokking, shortcut learning, and implicit bias has largely studied these phenomena in isolation. Our contribution connects these strands through a single dynamical mechanism. The Norm-Hierarchy framework shows that delayed representation learning arises whenever multiple interpolating solutions form a norm hierarchy under regularised optimisation. This perspective provides both a predictive law for transition times under clean separation and a diagnostic criterion—clean norm separation—explaining when such predictions fail to transfer across domains. The connection to emergent abilities further extends this unification to the scaling laws literature, suggesting that the norm-hierarchy mechanism operates across training time, regularisation strength, and model scale as a single unified axis of variation. 8 Conclusion We have introduced the Norm-Hierarchy Transition, a unified framework explaining delayed representational transitions under regularised training. The framework requires three ingredients—multiple interpolating representations, a norm hierarchy, and dissipative dynamics—and predicts three regimes with a tight logarithmic delay bound. Validated across four domains—modular arithmetic (6/6 predictions, R2>0.97R^2>0.97), CIFAR-10 (5/6), CelebA (4/6, with negative result explained by clean norm separation condition), and Waterbirds (2/6, norm dynamics only)—and four architecture variants, the results reveal a norm separation spectrum: the framework’s predictive power degrades gracefully with the degree of entanglement between shortcut and structured features. Notably, the mechanism operates in standard architectures with batch normalisation, and the layer-wise norm hierarchy (Proposition 4.2) reveals that the transition proceeds backward from output to input layers. Grokking, shortcut learning, simplicity bias, and emergent abilities in large language models emerge as manifestations of a single mechanism: the slow traversal of a norm hierarchy under regularised optimisation. Reproducibility. All code, training scripts, and experimental data are publicly available. CIFAR-10 experiments (13 runs) complete in ∼ 6 hours on a single GPU; architecture ablation (8 runs) in ∼ 3 hours; Waterbirds (8 runs) in ∼ 75 minutes; CelebA (6 runs) in ∼ 3 hours; modular arithmetic (293 runs) in ∼ 2.5 hours. References B. Barak, B. L. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang (2022) Hidden progress in deep learning: SGD learns parities near the computational limit. arXiv preprint arXiv:2207.08799. Cited by: §7. M. Belkin, D. Hsu, S. Ma, and S. Mandal (2019) Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences 116 (32), p. 15849–15854. Cited by: §7. L. Chizat and F. Bach (2019) A note on lazy training in differentiable programming. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §6.1, §7. B. Chughtai, L. Chan, and N. Nanda (2023) A toy model of universality: reverse engineering how networks learn group operations. In Proceedings of the 40th International Conference on Machine Learning (ICML), Cited by: §7. L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio (2017) Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning (ICML), p. 1019–1028. Cited by: Remark 2.5. R. Geirhos, J. Jacobsen, C. Michaelis, M. Zeiler, W. Brendel, and M. Bethge (2020) Shortcut learning in deep neural networks. Nature Machine Intelligence 2, p. 665–673. Cited by: §1, §7. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §6.2, §7. E. Z. Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn (2021) Just train twice: improving group robustness without training group information. arXiv preprint arXiv:2107.09044. Cited by: §7. Z. Liu, P. Luo, X. Wang, and X. Tang (2015) Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 3730–3738. Cited by: §C.6, §5.10. K. Lyu, L. Jin, J. Li, S. S. Du, J. D. Lee, and W. Hu (2023) Dichotomy of early and late phase implicit biases can provably induce grokking. In arXiv preprint arXiv:2311.18817, Cited by: §7. K. Lyu and J. Li (2020) Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890. Cited by: Remark 2.3, §7. P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever (2021) Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment 2021 (12), p. 124003. Cited by: §7. J. Nam, H. Cha, S. Ahn, J. Lee, and J. Shin (2020) Learning from failure: training debiased classifier from biased classifier. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §7. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt (2023) Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217. Cited by: §1, §7. A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra (2022) Grokking: generalization beyond overfitting on small algorithmic datasets. In Deep Learning for Code Workshop at ICLR, External Links: Link Cited by: §1, Remark 2.6, §7. N. Rahaman, A. Baratin, D. Arpit, F. Dräxler, M. Lin, F. A. Hamprecht, Y. Bengio, and A. Courville (2019) On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), p. 5301–5310. Cited by: Remark 2.5. S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang (2019) Distributionally robust optimization for group shift: an empirical study. In arXiv preprint arXiv:1911.08731, Cited by: §5.10. S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang (2020) Distributionally robust neural networks for group shifts: on the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), Cited by: §C.5, §1, §5.9, §7. R. Schaeffer, B. Miranda, and S. Koyejo (2023) Are emergent abilities of large language models a mirage?. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §6.2, Remark 6.1, §7. H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli (2020) The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, p. 9573–9585. Cited by: §1, Remark 2.5, §6.1. D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro (2018) The implicit bias of gradient descent on separable data. Journal of Machine Learning Research 19 (1), p. 2822–2878. Cited by: Remark 2.3, §7. A. Srivastava, A. Rastogi, A. Rao, et al. (2022) Beyond the imitation game: quantifying and extrapolating the capabilities of language models. In arXiv preprint arXiv:2206.04615, Cited by: §6.2, §7. G. Valle-Perez, L. Aitchison, and A. A. Louis (2019) Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Conference on Learning Representations (ICLR), Cited by: Remark 2.5. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Miculivicius, J. Huang, et al. (2022) Emergent abilities of large language models. Transactions on Machine Learning Research. Cited by: §6.2, §6.2, §7. Appendix A Proof of Generalised Escape Theorem Proof of Theorem 3.1. On ℳscM_sc, ∇ℒtrain(θt)=0 _train( _t)=0, so the update becomes θt+1=(1−2ηλ)θt+ηξt _t+1=(1-2ηλ) _t+η _t. Then: Vt+1=(1−2ηλ)2Vt+2η(1−2ηλ)⟨θt,ξt⟩+η2‖ξt‖2.V_t+1=(1-2ηλ)^2V_t+2η(1-2ηλ) _t, _t +η^2\| _t\|^2. Taking conditional expectation ([ξt|ℱt]=0E[ _t|F_t]=0): [Vt+1|ℱt]=(1−2ηλ)2Vt+η2σ2≤(1−ηλ)Vt+η2σ2,E[V_t+1|F_t]=(1-2ηλ)^2V_t+η^2σ^2≤(1-ηλ)V_t+η^2σ^2, using (1−2ηλ)2≤1−ηλ(1-2ηλ)^2≤ 1-ηλ for ηλ∈(0,1/2]ηλ∈(0,1/2]. Unrolling: [Vt]≤(1−ηλ)tV0+V∞E[V_t]≤(1-ηλ)^tV_0+V_∞ with V∞=ησ2/λV_∞=ησ^2/λ. Setting [Vt]=VstE[V_t]=V_st yields Tescape=Θ((ηλ)−1log(Vsc/Vst))T_escape= ((ηλ)^-1 (V_sc/V_st)) in the low-noise regime. □ Appendix B Proof of Dynamical Lower Bound Proof of Theorem 3.4. Per-step contraction satisfies [Vt+1|ℱt]/Vt≥1−cηλE[V_t+1|F_t]/V_t≥ 1-cηλ for constant c. To reach VstV_st from VscV_sc: (1−cηλ)T≤Vst/Vsc(1-cηλ)^T≤ V_st/V_sc, giving T≥(cηλ)−1log(Vsc/Vst)=Ω((ηλ)−1log(Vsc/Vst))T≥(cηλ)^-1 (V_sc/V_st)= ((ηλ)^-1 (V_sc/V_st)). □ Appendix C Experimental Details C.1 Model Architecture Layer Channels Kernel Stride conv1–conv2 3→32→323→ 32→ 32 3×33× 3 1 conv3–conv4 32→64→6432→ 64→ 64 3×33× 3 2, 1 conv5–conv6 64→128→12864→ 128→ 128 3×33× 3 2, 1 pool + fc 128→10128→ 10 — — Total: 287,850 parameters. Kaiming initialisation, ReLU activations, no batch normalisation. C.2 ResNet18 Variants We use torchvision.models.resnet18 adapted for 32×3232× 32 CIFAR images: initial 7×77× 7 convolution replaced with 3×33× 3 (stride 1, padding 1), max-pool removed. Three normalisation variants: (i) no normalisation; (i) GroupNorm (32 groups); (i) standard BatchNorm. Training: AdamW, lr =0.001=0.001, cosine annealing, batch size 128–256, 200 epochs. C.3 Training Configuration AdamW, lr =0.001=0.001, cosine annealing to 10−510^-5, batch size 256, 200 epochs, data augmentation (random crop padding =4=4, horizontal flip), border width 4 pixels. Evaluation every 5 epochs. C.4 Spurious Feature Construction Ten class colours applied as 4-pixel borders. During training, the correct colour appears with probability ρ; otherwise a uniformly random incorrect colour is used. Test sets: always-correct borders (colored), no borders (clean), borders on uniform gray (shortcut). C.5 Waterbirds Dataset Waterbirds [Sagawa et al., 2020] composites bird images from CUB-200-2011 onto backgrounds from Places. Labels: binary (waterbird vs. landbird); spurious attribute: background, correlated at 95% in training. Same SimpleCNN adapted for 224×224224× 224 input, trained with AdamW, lr =0.001=0.001, cosine annealing, batch size 64, 100 epochs. C.6 CelebA Dataset CelebA [Liu et al., 2015]: face attribute dataset, task == Smiling prediction, spurious attribute == hair colour (blond), correlated with Smiling via gender proxy. Four groups: (blond, smiling), (blond, not smiling), (not blond, smiling), (not blond, not smiling). Standard train/val/test split; SimpleCNN adapted for 64×6464× 64 input, AdamW, lr =0.001=0.001, cosine annealing, batch size 64, 100 epochs, λ∈0.001,0.1,1.0λ∈\0.001,0.1,1.0\, 2 seeds per λ. Appendix D Proof of Layer-Wise Norm Hierarchy Setup and notation. Partition parameters as θ=(θ(1),…,θ(L))θ=(θ^(1),…,θ^(L)) and define Vt(ℓ)=‖θt(ℓ)‖2V^( )_t=\|θ^( )_t\|^2. The update rule for layer ℓ under SGD with weight decay is: θt+1(ℓ)=θt(ℓ)−η∇θ(ℓ)ℒtrain(θt)−2ηλθt(ℓ)+ηξt(ℓ),θ^( )_t+1=θ^( )_t-η _θ^( )L_train( _t)-2ηλθ^( )_t+ηξ^( )_t, (7) where [‖ξt(ℓ)‖2|ℱt]≤σℓ2E[\|ξ^( )_t\|^2|F_t]≤ _ ^2. Gradient decomposition on ℳscM_sc. Near ℳscM_sc, by L-smoothness (A1): ∇θ(ℓ)ℒtrain(θt)=H(ℓ)(θt−Πℳscθt)+O(‖θt−Πθt‖2), _θ^( )L_train( _t)=H^( )( _t- _M_sc _t)+O(\| _t- _t\|^2), (8) where H(ℓ)H^( ) is the layer-ℓ Hessian block normal to ℳscM_sc. The shortcut encoding capacity assumption gives H(ℓ)≈αℓH(L)H^( )≈ _ H^(L), with κℓ=λmin(H(ℓ))≥0 _ = _min(H^( ))≥ 0. Per-layer Lyapunov recursion. Squaring (7) and taking expectations: [Vt+1(ℓ)|ℱt] [V^( )_t+1|F_t] ≤(1−ηλ−2ηαℓκℓ)Vt(ℓ)+C(ℓ), ≤ (1-ηλ-2η _ _ )V^( )_t+C^( ), (9) where C(ℓ)=2ηαℓκℓVℳsc(ℓ)+η2σℓ2C^( )=2η _ _ V^( )_M_sc+η^2 _ ^2. Defining γeff(ℓ)=ηλ+2ηαℓκℓγ^( )_eff=ηλ+2η _ _ and unrolling: Tescape(ℓ)=Θ(1ηλ+2ηαℓκℓlogVsc(ℓ)Vst(ℓ)).T^( )_escape= \! ( 1ηλ+2η _ _ V^( )_scV^( )_st ). (10) Ordering. Since γeff(ℓ)γ^( )_eff is strictly increasing in αℓ _ : αℓ′>αℓ⇒Tescape(ℓ′)<Tescape(ℓ) _ > _ T^( )_escape<T^( )_escape. □ Empirical correspondence. The observed contraction ratio 45%/31%≈1.4545\%/31\%≈ 1.45 is consistent with αLκL/α1κ1≈1.5 _L _L/ _1 _1≈ 1.5, providing direct empirical support for the shortcut encoding capacity decomposition.