Paper deep dive
Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive
Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen, Shishir Muralidhara, Federico Raue, René Schuster, Stanislav Frolov, Andreas Dengel
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/22/2026, 3:15:12 AM
Summary
The paper argues that continual learning regularizers like EWC fail to account for layer-specific curvature, leading to suboptimal stability-plasticity trade-offs. By analyzing the Hessian spectrum, the authors demonstrate that forgetting is driven by the top eigenvalue of each layer's Hessian block. They propose a layer-adaptive regularization strategy where penalty strength is proportional to these eigenvalues, effectively protecting early layers more than deeper ones, which significantly improves performance over uniform regularization.
Entities (8)
Relation Signals (6)
EWC → uses → diagonal Fisher
confidence 95% · Setting W to the diagonal Fisher gives EWC
early_layers → require → strong_protection
confidence 94% · protect early layers strongly, let deeper layers move
diagonal Fisher → failstocapture → top Hessian eigenvalue
confidence 93% · Diagonal-Fisher weights cannot recover this eigenvalue
Layer Condition Number → correlateswith → performance_loss
confidence 92% · uniform regularization loses new-task performance by an amount scaling with the layer condition number
SLCA → benefitsfrom → layer_adaptive_regularization
confidence 91% · We apply this recipe to EWC and SLCA and show clear improvements
ResNet-50 → has → high_layer_condition_number
confidence 90% · On ResNet-50 we measure κ=134.6
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer's diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC's existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer's top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics.
Tags
Links
- Source: https://arxiv.org/abs/2608.15901v1
- Canonical: https://arxiv.org/abs/2608.15901v1
Trouble viewing inline? Open PDF directly →
Full Text
129,382 characters extracted from source content.
Expand or collapse full text
Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive Brian B. Moser Ahmed Anwar Affiliation: German Research Center for Artificial Intelligence (DFKI) Tobias Christian Nauen Affiliation: German Research Center for Artificial Intelligence (DFKI) Affiliation: RPTU University Kaiserslautern-Landaufirst.last@dfki.de \ first_second.last@dfki.de Shishir Muralidhara Affiliation: German Research Center for Artificial Intelligence (DFKI) Affiliation: RPTU University Kaiserslautern-Landaufirst.last@dfki.de \ first_second.last@dfki.de Federico Raue Affiliation: German Research Center for Artificial Intelligence (DFKI) René Schuster Affiliation: German Research Center for Artificial Intelligence (DFKI) Affiliation: RPTU University Kaiserslautern-Landaufirst.last@dfki.de \ first_second.last@dfki.de Stanislav Frolov Affiliation: German Research Center for Artificial Intelligence (DFKI) Andreas Dengel Affiliation: German Research Center for Artificial Intelligence (DFKI) Affiliation: RPTU University Kaiserslautern-Landaufirst.last@dfki.de \ first_second.last@dfki.de Abstract Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer’s diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC’s existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer’s top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics. 1 Introduction Continual learning (CL) trains a single neural network on a sequence of tasks 1,2,…T_1,T_2,… presented one after another. The defining difficulty of this setting is catastrophic forgetting: the network keeps performing well on the most recent task but loses competence on the earlier ones. To prevent forgetting, the literature has converged on three broad families. Replay-based methods store or generate examples from past tasks [29, 5, 40], parameter-isolation methods route different tasks through different parts of the network [44, 26, 49], and regularization-based methods anchor parameters near the previous-task optimum [23, 59, 1]. This paper advances the third family. Regularization-based methods prevent forgetting by adding a penalty against parameter drift, Ω(θ)=12‖θ−θ⋆‖W2, (θ)= 12\|θ-θ \|_W^2, (1) where θ⋆θ is the previous-task optimum, ‖x‖W2:=x⊤Wx\|x\|_W^2:=x Wx, and W⪰0W 0 encodes the importance assigned to each direction in parameter space. Setting W to the diagonal Fisher gives EWC [23], to path-integral importance gives SI [59], and to synaptic salience gives MAS [1]. Later refinements such as Online EWC [43] and RWalk [4] change how W is estimated but keep the form of Eq. 1 unchanged. What the form does not do is reflect any quantity describing the layer as a whole. Two layers with similar parameter-wise Fisher-averages get the same per-layer penalty in EWC, even when one is ten times more sensitive to weight change than the other [37, 42, 35]. We call this the static, layer-uniform view of the network, where every parameter wears its own scalar tag, which is determined without considering the layer. A growing body of evidence from outside CL says the static view is wrong. Adversarial bit-flip attacks [37, 55, 38, 39, 6] drop ImageNet accuracy by more than 60%60\% by flipping fewer than 3030 bits, and those bits cluster in a small number of layers because those layers dominate the loss response to weight perturbation. Empirical Hessian-spectrum analyses [41, 42, 35, 14] as an independent research direction find per-layer eigenvalues that span two to three orders of magnitude on standard backbones. Both lines of evidence point to the top eigenvalue sℓs_ of the per-layer Hessian block. If sℓs_ varies across layers by a factor of 100100, then any regularizer blind to it spends its forgetting budget in the wrong places, which leads to a worse stability-plasticity trade-off. This paper closes the gap between the static view used by EWC and the layer-heterogeneous structure that real networks exhibit. Under a block-diagonal Hessian assumption (the layer-level analogue of the diagonal-Fisher assumption EWC already uses), forgetting decomposes cleanly into per-layer contributions weighted by sℓs_ (Theorem 2). Diagonal-Fisher weights cannot recover this structure, because averaging the diagonal of a positive semidefinite matrix loses the rank information that controls the top eigenvalue (Proposition 5). At a fixed forgetting budget, uniform regularization incurs regret scaling with the layer condition number κ=maxℓsℓ/minℓsℓκ= _ s_ / _ s_ (Theorem 6). Within the scalar-per-layer family, the minimum-regret choice is λℓ∝sℓ _ s_ (Theorem 8). We measure κ on six ImageNet-pretrained backbones (ResNet-18/34/50/101, ViT-B/16, ViT-L/16) and find values in the range 2222 to 195195, so the static view is leaving real performance unused. Two recent CL methods already make a layer-dependent choice. TUNA [51] restricts each layer’s update to a per-layer low-rank adapter. SLCA [60] uses a two-group learning-rate schedule for backbone and head. Each implicitly imposes a different prior on sℓs_ , and our analysis explains where each helps and where each does not. To probe the prescription on a method the framework directly covers, we also extend EWC with a one-parameter geometric schedule λℓ=c/αℓ−1 _ =c/α -1, sweep α from below one to above one, and compare against λℓ∝sℓ _ s_ from measured Hessians. The contribution of this paper is therefore both diagnostic and prescriptive. It identifies the static view in regularization-based CL as the part of the design space leaving performance unused, and gives the closed-form recipe λℓ∝sℓ _ s_ for what to do instead. Protect early layers strongly, let deeper layers move. 2 Related work Regularization-based CL. EWC [23], as introduced in Eq. 1, sets W to the diagonal Fisher. SI [59] and MAS [1] use path-integral and output-sensitivity importance, RWalk [4] unifies the two through a KL-based Riemannian formulation, and Progress & Compress [43] introduces online EWC. All produce per-parameter diagonal weights. Replay [29, 5, 40] and parameter-isolation [44] families are orthogonal to this analysis. Layer-adaptive CL. SLCA [60] applies a smaller learning rate to the pretrained backbone than to the head, a two-group layer-adaptive update rule on ViT. The prompt family [53, 52, 50, 46, 48] can be read as the limit where most of the backbone is frozen and only an input-prompt subspace updates. Each of these methods imposes its own prior on which layers are more sensitive. Layer-wise scaling. Layer-wise update scaling has independent roots in optimization (LARS [58], LAMB [57]) and fine-tuning (ULMFiT [19]). Our scalar-per-layer family is the regularization-side analog. Second-order weight importance descends from Optimal Brain Damage [27]. More recent instances include WoodFisher [45], EigenDamage [47], and SparseGPT [13], alongside first-order signals such as magnitude pruning [17] and the Lottery Ticket Hypothesis [12]. Proposition 5 is in the same spirit, specialised to the per-layer aggregation CL penalties implicitly perform. Per-layer Hessian sensitivity. Adversarial bit-flip attacks [37, 38, 39, 55, 6] concentrate the destructive bits in a small number of layers, an empirical lower bound on per-layer sensitivity spread. The empirical Hessian of trained networks has a bulk-plus-outlier structure [41, 42, 35, 34, 33, 31], with scalable estimators in PyHessian [56] and Lanczos [14], both validating the per-layer power-iteration we use for sℓs_ . Block-diagonal preconditioners K-FAC [30, 15] and Shampoo [16] exploit the same structure; Kunstner et al. [24] caution against empirical-Fisher approximations that we sidestep by computing the true Hessian. The flat-minima line [22, 18, 9, 11, 25] operationalises top-eigenvalue sensitivity at training time. Theoretical CL through the NTK lens [2, 10, 32] provides landscape-level evidence that the CL optimum lives in the low-curvature region we exploit per layer. 3 Methodology Let θ be the parameters of a neural network, partitioned by architecture into L layer blocks θ=(θ(1),…,θ(L))θ=(θ^(1),…,θ^(L)) with θ(ℓ)∈ℝdℓθ^( ) ^d_ , where each residual or transformer block counts as one layer. For a vector v∈ℝdv ^d, v(ℓ)v^( ) denotes its layer-ℓ part. For a matrix A∈ℝd×dA ^d× d acting on the full parameter vector (such as the Hessian), A(ℓ,ℓ′)A^( , ) denotes the block of rows in layer ℓ and columns in layer ℓ′ . Tasks 1,2,…T_1,T_2,… arrive sequentially with losses LkL_k. Let θ⋆θ be the parameters after training on tasks 1,…,k−11,…,k-1, and let θ⋆+Δθ + θ be the parameters after additionally training on task k. Forgetting on task k−1k-1 is ℱ(Δθ):=Lk−1(θ⋆+Δθ)−Lk−1(θ⋆)≥ 0.F( θ)\;:=\;L_k-1(θ + θ)-L_k-1(θ )\;≥\;0. (2) Let H=∇2Lk−1(θ⋆)H=∇^2L_k-1(θ ). The gradient vanishes at θ⋆θ , so Taylor expansion gives ℱ(Δθ)=12Δθ⊤HΔθ+(‖Δθ‖3).F( θ)\;=\; 12\, θ H\, θ\;+\;O ( θ ^3 ). (3) This second-order regime is the same one EWC [23] uses to derive its Fisher penalty, and every result in the paper concerns the quadratic term. Assumption 1 (Block-diagonal Hessian). H(ℓ,ℓ′)=0H^( , )=0 for ℓ≠ℓ′ ≠ . The Hessian couples parameters within a layer but not across layers. Assumption 1 is the layer-level analogue of the diagonal-Fisher approximation EWC already relies on. It is exact in fully factorized architectures and approximate in real ones. Skip connections, LayerNorm, and attention all create off-diagonal blocks. Appendix A gives a continuous relaxation. All main-text results use the block-diagonal form. Theorem 2 (Per-layer decomposition). Under Assumption 1, to the second order in ‖Δθ‖ θ , we have ℱ(Δθ)=12∑ℓ=1L(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ)+(‖Δθ‖3).F( θ)\;=\; 12 _ =1^L ( θ^( ) ) H^( , )\, θ^( )+O( θ ^3). (4) Proof. Apply the Taylor expansion of Eq. 3. Under Assumption 1, Δθ⊤HΔθ=∑ℓ(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ) θ H θ= _ ( θ^( )) H^( , ) θ^( ). ∎ Every unit of forgetting now belongs to a specific layer, which turns the question of how to spread a forgetting budget across layers into a well-posed optimization rather than a heuristic. Definition 3 (Layer sensitivity). The sensitivity of layer ℓ is the top eigenvalue of the layer’s Hessian block, namely sℓ:=λmax(H(ℓ,ℓ)).s_ := _ (H^( , )). (5) The upper bound on forgetting follows immediately: ℱ(Δθ)≤12∑ℓsℓ‖Δθ(ℓ)‖22.F( θ)\;≤\; 12 _ s_ \, θ^( ) _2^2. (6) The sensitivity sℓs_ acts as a per-layer price tag on displacement, and the rest of the paper revolves around this bound. 3.1 The layer condition number Definition 4 (Layer condition number). κ:=maxℓsℓmax(minℓsℓ,ε)κ:= _ s_ ( _ s_ ,\, ), with a floor ε>0 >0 to handle layers with a flat direction. We use ε=10−8maxℓsℓ =10^-8 _ s_ . Our empirical claim, which we call the Layer-Importance Hypothesis (LIH), is that on standard deep networks trained to convergence, κ is non-trivial and falls in the range 1010–10310^3. We verify this directly in Figure 1, which reports power-iteration measurements of sℓs_ on ImageNet-pretrained ResNet-50 and ViT-B/16 (full procedure and per-layer numbers in Appendix E). On ResNet-50 we measure κ=134.6κ=134.6, with the early conv stage layer1 dominating (slayer1=3,167s_layer1=3,167) and the fc head 134× less sensitive (sfc=23.5s_fc=23.5). On ViT-B/16 we measure κ=27.4κ=27.4, with the patch-embedding stem dominating (spatch_embed=78.2s_patch\_embed=78.2) and the transformer blocks roughly within an order of magnitude of each other. Both profiles are heavy-tailed and consistent with what the bit-flip, Optimal Brain Damage, and Hessian-spectrum literature already report [37, 55, 27, 35, 42]. Figure 1: Measured per-layer Hessian top eigenvalue sℓs_ on ResNet-50 (left) and ViT-B/16 (right), pretrained on ImageNet, evaluated at a 100-class cross-entropy minibatch. y-axis is log10sℓ _10s_ . ResNet-50: κ=134.6κ=134.6, with layer1 dominating. ViT-B/16: κ=27.4κ=27.4, with patch_embed dominating and the twelve transformer blocks within an order of magnitude of each other. Both profiles are heavy-tailed. To measure sℓs_ on a real network, run power iteration on H(ℓ,ℓ)H^( , ) using Hessian-Vector Products (HVPs). The cost is k HVPs per layer. 10 to 20 iterations suffice whenever the second eigenvalue is well separated from the first, which is the typical regime for trained networks because the per-layer Hessian is near low-rank. On very wide layers where power iteration is expensive, tr(H(ℓ,ℓ))/dℓ (H^( , ))/d_ recovered from a single Hutchinson HVP is a cheaper proxy. The two quantities can disagree by a factor in [1,dℓ][1,d_ ] (Proposition 5). 3.2 Diagonal Fisher cannot see layer sensitivity EWC’s per-parameter Fisher weights do vary across layers in the sense that the sum of Fisher values in one layer differs from another. But that sum is a weak summary of the layer’s actual curvature. The next proposition quantifies how weak. Proposition 5 (Diagonal-spectral gap). For any positive semidefinite matrix A∈ℝd×dA ^d× d, 1≤λmax(A)1d∑pApp≤d.1\;≤\; _ (A) 1d _pA_p\;≤\;d. (7) The left inequality is an equality if and only if A=σIA=σ I. The right is an equality if and only if A has rank at most 11. Every value in [1,d][1,d] is realized by some positive semidefinite A. Proof in Appendix J. For the per-layer Hessian block H(ℓ,ℓ)H^( , ), Proposition 5 says the following. Two layers with identical mean diagonal (and hence identical mean Fisher under the standard F≈HF≈ H identity at exponential-family optima [30]) can have sℓs_ values that differ by a factor up to the smaller of their widths. The low-rank Hessian regime observed in trained networks [33, 31] pushes this bound close to tight. EWC treats the two cases as identical, but they produce very different amounts of forgetting (Appendix E). 3.3 Uniform regularization pays a price The performance cost of the static, layer-uniform view comes from comparing two regularizers held to the same forgetting budget B>0B>0. The oracle regularizer uses the true per-layer Hessian: minΔθLk(θ⋆+Δθ)subject to12∑ℓ(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ)≤B. _ θ\;L_k(θ + θ) to 12 _ ( θ^( )) H^( , ) θ^( )\;≤\;B. (8) The constraint equals forgetting to second order by Theorem 2. The uniform regularizer treats all displacement as equally costly: minΔθLk(θ⋆+Δθ)subject to12μ‖Δθ‖22≤B, _ θ\;L_k(θ + θ) to 12\,μ\, θ _2^2\;≤\;B, (9) with μ chosen so that the resulting Δθ θ produces the same actual forgetting B under the true Hessian. This μ-calibration step blocks the trivial reply that uniform could just tighten its penalty. Both regularizers now respect the same forgetting budget, so the comparison isolates the stability-plasticity trade-off, namely the new-task accuracy each regularizer sacrifices to hit that fixed forgetting level. Only the allocation of the budget across layers differs. The bound below uses two further quantities. Write g(ℓ):=∇θ(ℓ)Lk(θ⋆)g^( ):= _θ^( )L_k(θ ) for the new-task gradient at the old optimum, restricted to layer ℓ , and Q(ℓ,ℓ):=∇θ(ℓ)2Lk(θ⋆)Q^( , ):=∇^2_θ^( )L_k(θ ) for the new-task Hessian block. We assume the new-task Hessian is well-conditioned in the standard sense, with smallest and largest eigenvalues satisfying μQ≤λmin(Q(ℓ,ℓ)) _Q≤ _ (Q^( , )) and λmax(Q(ℓ,ℓ))≤MQ _ (Q^( , ))≤ M_Q uniformly across layers. Equivalently: μQI⪯Q(ℓ,ℓ)⪯MQI _QI Q^( , ) M_QI, where μQ _Q is a uniform lower curvature bound and MQM_Q a uniform upper curvature bound on the new-task quadratic. Let v(ℓ)v^( ) be a top eigenvector of H(ℓ,ℓ)H^( , ), and define the alignment χℓ:=‖g(ℓ)⊤v(ℓ)‖g(ℓ)‖2∈[0,1]. _ \;:=\; g^( ) v^( ) g^( ) _2\;∈\;[0,1]. (10) The alignment χℓ _ measures how much of the new-task gradient on layer ℓ points along the old-task fragile direction. When χℓ _ is small, the new task is not pushing on anything the old task cares about. Theorem 6 (Regret of uniform regularization). Assume Q(ℓ,ℓ′)=0Q^( , )=0 for ℓ≠ℓ′ ≠ (Appendix B treats the coupled case). The regret of the uniform regularizer, Regret:=L~k(Δθunif⋆)−L~k(Δθoracle⋆)Regret:= L_k( θ _unif)- L_k( θ _oracle), satisfies Regret≥c0⋅κ−1κ⋅maxℓ∈ℒ+χℓ2‖g(ℓ)‖22sℓ,c0:=μQMQ+μQ∈(0,12],Regret\;≥\;c_0· κ-1κ· _ ^+\, _ ^2\, g^( ) _2^2s_ , c_0:= _QM_Q+ _Q∈ (0, 12 ], (11) where ℒ+:=ℓ:sℓ>minℓ′sℓ′L^+:=\ :s_ > _ s_ \ is everything except the most flexible layer. The gap vanishes when κ=1κ=1, vanishes when every χℓ _ for ℓ∈ℒ+ ^+ is zero, and is strictly positive and monotone non-decreasing in κ otherwise. Three factors multiply. The heterogeneity amplifier (κ−1)/κ(κ-1)/κ grows from 00 to 11 as the layer spectrum spreads. The alignment χℓ2 _ ^2 captures whether the new-task gradient actually pushes on fragile directions. When it does not, uniform and oracle agree. The per-layer unit ‖g(ℓ)‖22/sℓ g^( ) _2^2/s_ is the ratio of new-task pressure on layer ℓ to the layer’s rigidity. Proof in Appendix B. Corollary 7 (Large-κ asymptotic). Fix smax:=maxℓsℓs_ := _ s_ and let κ→∞κ→∞ by minℓsℓ↓0 _ s_ 0. Writing ℓ^:=argmaxℓsℓ := _ s_ , limκ→∞Regret≥c0⋅χℓ^2‖g(ℓ^)‖22smax. _κ→∞\,Regret\;≥\;c_0· _ ^2\, g^( ) _2^2s_ . (12) 3.4 The optimal per-layer strength Staying inside the scalar-per-layer family Ω(Δθ)=12∑ℓλℓ‖Δθ(ℓ)‖22 ( θ)\;=\; 12 _ _ \, θ^( ) _2^2 (13) is the simplest strict improvement on uniform, and it is what our proposed adaptive methods in Section 4 actually implement. Theorem 8 (Optimal λℓ _ ). Fix the worst-case displacement direction in each layer Δθ(ℓ) θ^( ) aligned with the top eigenvector v(ℓ)v^( ) of H(ℓ,ℓ)H^( , ). Among regularizers of the form in Eq. 13, the choice that matches the oracle’s penalty on this direction, up to one global constant shared across layers, is λℓ∝sℓ. _ \; \;s_ . (14) No scalar choice reproduces the oracle on every direction unless every H(ℓ,ℓ)H^( , ) is a scalar multiple of the identity. Proof in Appendix C. Theorem 8 fixes λℓ _ to a noisy per-layer top eigenvalue, and on standard backbones the dynamic range of sℓs_ spans more than two orders of magnitude (Figure 1), so a literal schedule over-protects the most curvature-dominant layer in absolute terms and starves the rest of plasticity. A one-parameter approximation that fits only the depth direction provides implicit smoothing across layers, motivating the following corollary below. Corollary 9 (Geometric schedules). If sℓs_ decays or grows geometrically with depth, sℓ∝γℓ−1s_ γ -1, then the optimal λℓ∝γℓ−1 _ γ -1. This is the one-parameter family our experiments in Section 4 fit. 3.5 Practical implications The math reduces the design to one empirical question: which depth direction grows sℓs_ ? On six pretrained backbones (Figure 1, Appendix E), early conv stages on ResNets and patch embeddings on ViTs dominate sℓs_ . Given the corollary 9, so combined with λℓ∝sℓ _ s_ , this gives the rule of thumb: protect early layers strongly, let deeper layers move. Concretely, methods that enforce weight penalty should set λ1≥λ2≥…≥λL, _1\;≥\; _2\;≥\;…\;≥\; _L, (15) larger penalty on the early layers and smaller on the head. Methods that enforce per-layer learning rate (e.g., SLCA) should set η1≤η2≤…≤ηL, _1\;≤\; _2\;≤\;…\;≤\; _L, (16) smaller LR on the early layers and larger on the head. The two schedules are opposite in sign because constraining LR is the inverse of constraining penalty strength. We test both forms next. 4 Experiments Our LIH predicts that a layer-adaptive regularizer should outperform a uniform one whenever the layer condition number κ is non-trivial, and that the optimal scalar-per-layer strength tracks the per-layer sensitivities (sℓ)(s_ ). We test these predictions in two settings. The first uses a depth-weighted EWC variant we implement directly on small from-scratch CNNs. The second applies the same depth-weighted rule inside SLCA on a pretrained ViT-B/16 backbone. We also tested the rule on TUNA, but the gains there are smaller than for EWC and SLCA. TUNA’s adapter family already restricts each layer’s update to a 1616-dimensional subspace, which leaves limited headroom for layer-wise reweighting, so we report those results in Appendix H. Experiment 1: Depth-weighted EWC on Split-C (small CNNs). We extend EWC by replacing its single scalar λ with the geometric decreasing schedule λℓ=c/αℓ−1 _ =c/α -1, fulfilling Eq. 15, and grid-search over α∈2−2,2−1,20,21,22,23α∈\2^-2,2^-1,2^0,2^1,2^2,2^3\ and c∈200,1000,5000,20000c∈\200,1000,5000,20000\. The point α=20=1α=2^0=1 is exactly uniform EWC. Everything else is layer-adaptive. The backbones are deliberately small so we can sweep the full grid: SmallCNN (3 conv + 2 fc, L=5L=5 layers) on Split-CIFAR-10 (5 tasks of 2 classes), and MediumCNN (4 conv + 2 fc, L=6L=6 layers) on Split-CIFAR-100 (10 tasks of 10 classes). Each task trains for two epochs. Remaining hyperparameters are in the supplementary materials. We report four metrics across the grid: average accuracy across all tasks (the standard CL metric), final-task accuracy, forgetting (max accuracy reached minus final accuracy, averaged over tasks), and backward transfer (BWT [29], end-of-sequence accuracy on each earlier task minus its end-of-task accuracy, averaged over earlier tasks). Experiment 2: SLCA depth-weighted schedule on ViT-B/16, Split-CIFAR-100. SLCA enforces the per-layer constraint differently from Experiment 1. Instead of weighting the EWC penalty per layer, it scales the per-layer learning rate (default backbone 0.1×0.1×, head 1×1×). Protection here happens through how far each layer is allowed to move per step rather than through how strongly it is anchored to θ⋆θ , so the depth-weighted schedule has a different sign convention. Larger α means deeper layers receive a larger LR multiplier, not a smaller penalty. We extend SLCA’s two-group LR schedule to a per-block depth-weighted chain λl=c⋅αl−L _l=c·α^l-L, fulfilling Eq. 16, where L=15L=15 is the number of ViT-B blocks plus stem, norm, and head, and at (c,α)=(1,1)(c,α)=(1,1) we reproduce default SLCA exactly. We sweep c∈0.5,1.0,2.0c∈\0.5,1.0,2.0\ and α∈0.85,0.92,1.0,1.08,1.18α∈\0.85,0.92,1.0,1.08,1.18\ on two pretrainings of the same backbone, namely MoCoV3 (self-supervised contrastive, used in the SLCA paper’s ablations) and ImageNet-21k (supervised classification). The framework predicts the same equation in both cases but allows the optimum to land at different α if the two pretrainings produce different per-layer sensitivities. 4.1 Depth-weighted EWC on small from-scratch CNNs Figure 2 reports the MediumCNN sweep on Split-CIFAR-100 across all four metrics, with 3-seed mean ± std bands. The avg-acc peak at c=200c=200 sits at α=2α=2 (59.8%±1.559.8\%± 1.5) and beats α=1α=1 (57.2%±1.357.2\%± 1.3) with non-overlapping bands. The forgetting and BWT panels prefer α≤1α≤ 1 instead. The framework reads that as deeper layers being the more sensitive ones on this from-scratch backbone, opposite of the shallow-dominated pattern measured on pretrained backbones (Figure 1). The avg-acc preference for α>1α>1 on the longer task sequence is then plasticity dominating the metric rather than a sensitivity prediction. The remaining EWC sweeps (SmallCNN/Split-CIFAR-10 and ResNet-18/Split-CIFAR-100) are reported in Appendix F. The SmallCNN effect is statistically marginal and the ResNet-18 sweep gains nothing from a one-parameter geometric prior, consistent with the non-monotone ResNet-50 sensitivity in Figure 1 that a geometric schedule cannot fit. Figure 2: Depth-weighted EWC on Split-CIFAR-100 (MediumCNN, 10 tasks, 3 seeds, mean ± std bands). Top, average accuracy across tasks (left) and forgetting (right). Bottom, final-task accuracy (left) and backward transfer (right). The dotted vertical line marks α=1α=1 (uniform EWC). The avg-acc peak at c=200c=200 sits at α=2α=2 with non-overlapping seed bands over uniform. 4.2 SLCA depth-weighted schedule on a pretrained ViT-B/16 Figure 3 shows the SLCA depth-weighted sweep on Split-CIFAR-100 (10 incremental tasks, ViT-B/16 + ImageNet-21k pretraining). Default SLCA corresponds exactly to (c=1,α=1)(c=1,α=1) at 91.29%91.29\% final-task accuracy. The optimum moves away from α=1α=1 to α∈1.08,1.18α∈\1.08,1.18\ for every c. The best configuration (c=1,α=1.18)(c=1,α=1.18) achieves 92.01%92.01\%, a +0.7%+0.7\% absolute gain over the default. On the average-incremental metric the gain narrows to +0.13%+0.13\% (95.03%95.03\% vs 94.90%94.90\%) but is direction-consistent across all three c values. The ImageNet-21k backbone benefits from a schedule that protects shallow layers more than default SLCA’s two-group LR choice does. The complementary MoCoV3-pretrained sweep (Figure 7, Appendix G) is much flatter, consistent with MoCoV3 having per-layer sensitivities that default SLCA already fits well. The contrast between the two pretrainings is the sharper test of the framework: the same method on the same backbone with different pretraining has different optima. Figure 3: SLCA depth-weighted sweep on Split-CIFAR-100 (ViT-B/16 + ImageNet-21k pretraining, 10 tasks). Per-block LR multiplier λl=c⋅αl−L _l=c·α^l-L, L=15L=15. Top: final-task accuracy (left) and forgetting (right). Bottom: average incremental accuracy (left) and BWT (right). The dotted vertical line is default SLCA (α=1α=1). The optimum is consistently away from α=1α=1 at α∈1.08,1.18α∈\1.08,1.18\, with best configuration (c=1,α=1.18)(c=1,α=1.18) giving +0.7%+0.7\% final-task accuracy over default SLCA at (c=1,α=1)(c=1,α=1). Table 1 places this result in context under SLCA’s protocol [60] (10 tasks of 10 classes, ViT-B/16); each row’s method is cited inline. Our depth-weighted SLCA uses the dominant configuration found in the sweep above (c=1,α=1.18c=1,α=1.18 on ImageNet-21k; c=0.5,α=0.92c=0.5,α=0.92 on MoCoV3). Table 1: Class-incremental learning on Split-CIFAR-100 (10 tasks, ViT-B/16), two pretrainings: ImageNet-21k supervised and ImageNet-1k MoCoV3 [7]. Last-acc and Inc-acc as in [60]. Prior-method numbers reproduced under SLCA’s protocol; the SLCA rows report multi-seed mean± over task-order seeds (Appendix G). Bold = best in column among non-upper-bound rows. ImageNet-21k pretrain MoCoV3 pretrain Method Memory-free Last-acc Inc-acc Last-acc Inc-acc Joint training (upper bound) – 93.22 — 89.11 — GDumb [36] 81.92 89.46 69.72 80.95 DER++ [3] 84.50 91.49 63.64 79.55 BiC [54] 88.45 93.37 80.57 89.39 L2P [53] ✓ 82.76 88.48 — — DualPrompt [52] ✓ 85.56 90.33 — — EWC [23] ✓ 89.30 92.31 81.62 87.56 LwF [28] ✓ 87.99 92.13 77.94 86.90 Seq FT ✓ 88.86 92.01 81.47 87.55 SLCA [60] ✓ 91.35± 0.23 94.40± 0.69 84.44± 0.29 90.12± 0.32 SLCA + ours (depth-weighted) ✓ 91.80± 0.23 94.51± 0.66 84.26± 0.48 90.08± 0.32 On ImageNet-21k pretraining, depth-weighted SLCA improves on every memory-free baseline reproduced in [60], including default SLCA itself (91.35→91.8091.35→ 91.80 last-acc, 94.40→94.5194.40→ 94.51 inc-acc). On MoCoV3 the depth-weighted variant matches default SLCA in the noise (84.2684.26 vs 84.4484.44), consistent with the framework’s reading that MoCoV3 already lands near the per-layer schedule the default two-group LR ratio implements. 4.3 Empirical takeaways Theorem 8 prescribes λℓ∝sℓ _ s_ as the optimal scalar-per-layer schedule, and Corollary 9 specializes this to λℓ∝γℓ−1 _ γ -1 when sℓs_ decays geometrically with depth. Our experiments fit this one-parameter family. The measurements say sℓs_ is shallow-dominated on every standard pretrained backbone. The rule of thumb that follows says protect shallow layers strongly, let deeper layers move. The experiments confirm this where the rule applies. The cleanest single bridge between theory and experiments is the cross-pretraining contrast on SLCA (Figures 3 and 7). The same method on the same backbone with different pretraining shifts the optimal α, in the direction the framework predicts. Depth-weighted EWC on MediumCNN/CIFAR-100 peaks at α=2α=2 on avg-acc with non-overlapping 3-seed bands (Figure 2), and SLCA on ViT-B/ImageNet-21k peaks at α∈1.08,1.18α∈\1.08,1.18\ across all c with +0.7%+0.7\% over default (Figure 3). The rule is silent (or has the opposite forgetting signal) where its domain assumption fails, namely on small from-scratch CNNs whose sℓs_ profile is not shallow-dominated, and we report those cases honestly in Appendix F. The TUNA adapter-orthogonality sweep is consistent in direction (Appendix H) but the gain is small because the adapter family already constrains each layer’s update to a 16-dimensional subspace. As a robustness check, we also swap the geometric schedule for a linear and a two-group “step” schedule on MediumCNN/CIFAR-100; all three monotone-decreasing shapes consistently beat uniform within ∼ 1–4% of each other, indicating the rule of thumb is the load-bearing piece, not the specific shape (Appendix I). Across backbones, the magnitude of the forgetting penalty for under-protecting deep layers tracks the measured layer condition number κ, the empirical analog of Theorem 6’s regret bound (Appendix L). A direct test of the literal λℓ∝sℓ _ s_ prescription on EWC is reported in Appendix K, where the geometric proxy outperforms the literal schedule. This is the empirical counterpart of the dynamic-range and measurement-noise concerns introduced in Section 3.4, and confirms that the framework’s prediction carries through the direction of the schedule rather than through the exact per-layer ratios. The same per-layer measurement procedure on a language backbone (BERT-base) recovers a qualitatively similar shallow-dominated profile with κ=9.15κ=9.15, indicating the LIH and the resulting prescription extend beyond computer vision (Appendix N). 5 Discussion and limitations The framework is deliberately minimal. It inherits EWC’s second-order approximation and adds the block-diagonal analogue of EWC’s diagonal assumption, isolating per-layer operator norms sℓs_ as the missing piece. This minimality has costs. The quadratic surrogate is only valid inside a trust region around θ⋆θ (Appendix B). The block-diagonal assumption is violated at the boundaries created by skip connections, LayerNorm, and attention. Appendix A gives a continuous relaxation and a granularity diagnostic for that case. The empirical Fisher information matrix can diverge from H [24], but Proposition 5 applies to any positive semidefinite block, so the gap result survives. The SLCA reduction is functional rather than algebraic. The sensitivity sℓs_ is measured at θ⋆θ and then used to regularize motion away from it, and we have no stability argument beyond the trust region. 6 Conclusion The diagonal Fisher-penalty at the heart of EWC treats every layer as equally sensitive, but per-layer Hessian top eigenvalues span two orders of magnitude on every standard backbone we measured. The framework formalises this gap (forgetting decomposes by sℓs_ ; diagonal Fisher-penalty cannot recover sℓs_ ; uniform regularization pays regret (κ−1)/κ(κ-1)/κ) and gives the closed-form remedy λℓ∝sℓ _ s_ , computable from one cheap power iteration per layer. SLCA approximates this implicitly through its layer-wise learning rate, our depth-weighted EWC is the direct test, and the resulting rule of thumb (Section 4.3) holds on pretrained backbones, namely protect shallow layers strongly and let deeper layers move. Acknowledgements This work was supported by the BMFTR project Albatross (Grant 16IW24002). References [1] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars (2018) Memory aware synapses: learning what (not) to forget. In Proc. ECCV, p. 144–161. Cited by: §1, §1, §2. [2] M. A. Bennani, T. Doan, and M. Sugiyama (2006) Generalisation guarantees for continual learning with orthogonal gradient descent. In arXiv:2006.11942, Cited by: §2. [3] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara (2020) Dark experience for general continual learning: a strong, simple baseline. In Proc. NeurIPS, Cited by: Appendix P, Table 1. [4] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. S. Torr (2018) Riemannian walk for incremental learning: understanding forgetting and intransigence. In Proc. ECCV, Cited by: §1, §2. [5] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny (2019) Efficient lifelong learning with a-gem. In Proc. ICLR, Cited by: §1, §2. [6] H. Chen, C. Fu, J. Zhao, and F. Koushanfar (2021) ProFlip: targeted trojan attack with progressive bit flips. In Proc. ICCV, Cited by: Appendix Q, §1, §2. [7] X. Chen, S. Xie, and K. He (2021) An empirical study of training self-supervised vision transformers (moco v3). In Proc. ICCV, Cited by: Table 1. [8] J. Devlin, M. Chang, K. Lee, and K. N. Toutanova (2018) BERT: pre-training of deep bidirectional transformers for language understanding. External Links: Link Cited by: Appendix N. [9] L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio (2017) Sharp minima can generalize for deep nets. In Proc. ICML, Cited by: §2. [10] T. Doan, M. A. Bennani, B. Mazoure, G. Rabusseau, and P. Alquier (2021) A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In Proc. AISTATS, Cited by: §2. [11] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur (2021) Sharpness-aware minimization for efficiently improving generalization. In Proc. ICLR, Cited by: §2. [12] J. Frankle and M. Carbin (2019) The lottery ticket hypothesis: finding sparse, trainable neural networks. In Proc. ICLR, Cited by: §2. [13] E. Frantar and D. Alistarh (2023) SparseGPT: massive language models can be accurately pruned in one-shot. In Proc. ICML, Cited by: §2. [14] B. Ghorbani, S. Krishnan, and Y. Xiao (2019) An investigation into neural net optimization via hessian eigenvalue density. In Proc. ICML, Cited by: §1, §2. [15] R. Grosse and J. Martens (2016) A kronecker-factored approximate fisher matrix for convolution layers. In Proc. ICML, Cited by: §2. [16] V. Gupta, T. Koren, and Y. Singer (2018) Shampoo: preconditioned stochastic tensor optimization. In Proc. ICML, Cited by: §2. [17] S. Han, J. Pool, J. Tran, and W. Dally (2015) Learning both weights and connections for efficient neural networks. In Proc. NeurIPS, Cited by: §2. [18] S. Hochreiter and J. Schmidhuber (1997) Flat minima. In Neural Computation, Cited by: §2. [19] J. Howard and S. Ruder (2018) Universal language model fine-tuning for text classification. In Proc. ACL, Cited by: §2. [20] Y. Huang, Y. Zhang, J. Chen, X. Wang, and D. Yang (2021) Continual learning for text classification with information disentanglement based regularization. In Proc. NAACL, Cited by: Appendix N. [21] S. Jastrzebski, Z. Kenton, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey (2019) On the relation between the sharpest directions of dnn loss and the sgd step length. In Proc. ICLR, Cited by: Appendix M. [22] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang (2017) On large-batch training for deep learning: generalization gap and sharp minima. In Proc. ICLR, Cited by: §2. [23] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13). Cited by: §1, §1, §2, §3, Table 1. [24] F. Kunstner, P. Hennig, and L. Balles (2019) Limitations of the empirical fisher approximation for natural gradient descent. In Proc. NeurIPS, p. 4156–4167. Cited by: §2, §5. [25] J. Kwon, J. Kim, H. Park, and I. K. Choi (2021) ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proc. ICML, Cited by: §2. [26] M. De. Lange et al (2022) A continual learning survey: defying forgetting in classification tasks. IEEE TPAMI 44 (7). Cited by: §1. [27] Y. LeCun, J. S. Denker, and S. A. Solla (1990) Optimal brain damage. In Advances in Neural Information Processing Systems 2, p. 598–605. Cited by: §2, §3.1. [28] Z. Li and D. Hoiem (2016) Learning without forgetting. In Proc. ECCV, Cited by: Table 1. [29] D. Lopez-Paz and M. Ranzato (2017) Gradient episodic memory for continual learning. In Proc. NeurIPS, Cited by: §1, §2, §4. [30] J. Martens (2020) New insights and perspectives on the natural gradient method. Journal of Machine Learning Research 21 (146). Cited by: §2, §3.2. [31] C. H. Martin and M. W. Mahoney (2021) Implicit self-regularization in deep neural networks: evidence from random matrix theory and implications for learning. Journal of Machine Learning Research 22 (165). Cited by: §2, §3.2. [32] S. I. Mirzadeh, M. Farajtabar, D. Gorur, R. Pascanu, and H. Ghasemzadeh (2021) Linear mode connectivity in multitask and continual learning. In Proc. ICLR, Cited by: §2. [33] V. Papyan, X. Y. Han, and D. L. Donoho (2020) Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40). Cited by: §2, §3.2. [34] V. Papyan (2019) Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. In Proc. ICML, Cited by: §2. [35] V. Papyan (2020) Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research 21 (252). Cited by: §1, §1, §2, §3.1. [36] A. Prabhu, P. H. S. Torr, and P. K. Dokania (2020) GDumb: a simple approach that questions our progress in continual learning. In Proc. ECCV, Cited by: Appendix P, Table 1. [37] A. S. Rakin, Z. He, and D. Fan (2019) Bit-flip attack: crushing neural network with progressive bit search. In Proc. ICCV, p. 1211–1220. Cited by: Appendix Q, §1, §1, §2, §3.1. [38] A. S. Rakin, Z. He, and D. Fan (2020) TBT: targeted neural network attack with bit trojan. In Proc. CVPR, Cited by: §1, §2. [39] A. S. Rakin, Z. He, J. Li, F. Yao, C. Chakrabarti, and D. Fan (2022) T-bfa: targeted bit-flip adversarial weight attack. IEEE TPAMI. Cited by: §1, §2. [40] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert (2017) ICaRL: incremental classifier and representation learning. In Proc. CVPR, Cited by: §1, §2. [41] L. Sagun, L. Bottou, and Y. LeCun (2016) Eigenvalues of the hessian in deep learning: singularity and beyond. In arXiv:1611.07476, Cited by: §1, §2. [42] L. Sagun, U. Evci, V. U. Guney, Y. Dauphin, and L. Bottou (2018) Empirical analysis of the hessian of over-parametrized neural networks. In ICLR Workshop Track, Cited by: §1, §1, §2, §3.1. [43] J. Schwarz, W. M. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell (2018) Progress & compress: a scalable framework for continual learning. In Proc. ICML, Cited by: §1, §2. [44] J. Serrà, D. Surís, M. Miron, and A. Karatzoglou (2018) Overcoming catastrophic forgetting with hard attention to the task. In Proc. ICML, Cited by: §1, §2. [45] S. P. Singh and D. Alistarh (2020) WoodFisher: efficient second-order approximation for neural network compression. In Proc. NeurIPS, Cited by: §2. [46] J. S. Smith, L. Karlinsky, V. Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira (2023) CODA-prompt: continual decomposed attention-based prompting for rehearsal-free continual learning. In Proc. CVPR, Cited by: §2. [47] C. Wang, R. Grosse, S. Fidler, and G. Zhang (2019) EigenDamage: structured pruning in the kronecker-factored eigenbasis. In Proc. ICML, Cited by: §2. [48] L. Wang, J. Xie, X. Zhang, M. Huang, H. Su, and J. Zhu (2023) Hierarchical decomposition of prompt-based continual learning: rethinking obscured sub-optimality. In Proc. NeurIPS, Cited by: §2. [49] L. Wang, X. Zhang, H. Su, and J. Zhu (2024) A comprehensive survey of continual learning: theory, method and application. IEEE TPAMI. Cited by: §1. [50] Y. Wang, Z. Huang, and X. Hong (2022) S-prompts learning with pre-trained transformers: an occam’s razor for domain incremental learning. In Proc. NeurIPS, Cited by: §2. [51] Y. Wang, D.-W. Zhou, and H.-J. Ye (2025) Integrating task-specific and universal adapters for pre-trained model-based class-incremental learning. In Proc. ICCV, Cited by: §1. [52] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister (2022) DualPrompt: complementary prompting for rehearsal-free continual learning. In Proc. ECCV, Cited by: §2, Table 1. [53] Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister (2022) Learning to prompt for continual learning. In Proc. CVPR, Cited by: §2, Table 1. [54] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu (2019) Large scale incremental learning. In Proc. CVPR, Cited by: Table 1. [55] F. Yao, A. S. Rakin, and D. Fan (2020) DeepHammer: depleting the intelligence of deep neural networks through targeted chain of bit flips. In Proc. USENIX Security Symposium, p. 1463–1480. Cited by: Appendix Q, §1, §2, §3.1. [56] Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney (2020) PyHessian: neural networks through the lens of the hessian. In Proc. IEEE BigData, Cited by: §2. [57] Y. You et al (2020) Large batch optimization for deep learning: training bert in 76 minutes. In Proc. ICLR, Cited by: §2. [58] Y. You, I. Gitman, and B. Ginsburg (2017) Large batch training of convolutional networks. In arXiv:1708.03888, Cited by: §2. [59] F. Zenke, B. Poole, and S. Ganguli (2017) Continual learning through synaptic intelligence. In Proc. ICML, p. 3987–3995. Cited by: §1, §1, §2. [60] G. Zhang, L. Wang, G. Kang, L. Chen, and Y. Wei (2023) SLCA: slow learner with classifier alignment for continual learning on a pre-trained model. In Proc. ICCV, Cited by: §1, §2, §4.2, §4.2, Table 1, Table 1. [61] Z. Zhu, J. Wu, B. Yu, L. Wu, and J. Ma (2019) The anisotropic noise in stochastic gradient descent: its behavior of escaping from sharp minima and regularization effects. In Proc. ICML, p. 7654–7663. Cited by: Appendix M. Appendix A Relaxed block-diagonal Hessian This appendix promotes Assumption 10 (bounded cross-layer coupling) into a quantitative replacement for Theorems 2 and 6. Assumption 10 (Bounded cross-layer coupling). There exists ρ∈[0,1)ρ∈[0,1) such that for all ℓ≠ℓ′ ≠ , ‖H(ℓ,ℓ′)‖op≤ρλmax(H(ℓ,ℓ))λmax(H(ℓ′,ℓ′)). H^( , ) _op\;≤\;ρ _ (H^( , )) _ (H^( , )). (17) Proposition 11 (Decomposition with cross-layer coupling). Under Assumption 10, the second-order term in ℱ(Δθ)F( θ) satisfies |Δθ⊤HΔθ−∑ℓ(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ)|≤ 2ρ∑ℓ<ℓ′sℓsℓ′‖Δθ(ℓ)‖2‖Δθ(ℓ′)‖2, |\, θ H\, θ- _ ( θ^( )) H^( , ) θ^( )\, |\;≤\;2ρ _ < s_ \,s_ \, θ^( ) _2 θ^( ) _2, (18) which after Cauchy-Schwarz is bounded by ρ(∑ℓsℓ‖Δθ(ℓ)‖2)2−ρ∑ℓsℓ‖Δθ(ℓ)‖22≤ρ(L−1)∑ℓsℓ‖Δθ(ℓ)‖22.ρ\,( _ s_ θ^( ) _2)^2-ρ _ s_ θ^( ) _2^2≤ρ\,(L-1) _ s_ θ^( ) _2^2. Proof. Δθ⊤HΔθ=∑ℓ,ℓ′(Δθ(ℓ))⊤H(ℓ,ℓ′)Δθ(ℓ′) θ H θ= _ , ( θ^( )) H^( , ) θ^( ). Subtracting the diagonal terms gives ∑ℓ≠ℓ′(Δθ(ℓ))⊤H(ℓ,ℓ′)Δθ(ℓ′) _ ≠ ( θ^( )) H^( , ) θ^( ), bounded in absolute value by ∑ℓ≠ℓ′‖H(ℓ,ℓ′)‖op‖Δθ(ℓ)‖‖Δθ(ℓ′)‖ _ ≠ H^( , ) _op θ^( ) θ^( ) . Substituting Assumption 10, factoring the symmetric pair, and applying Cauchy-Schwarz (∑ℓsℓ‖Δθ(ℓ)‖)2≤L∑ℓsℓ‖Δθ(ℓ)‖2( _ s_ θ^( ) )^2≤ L _ s_ θ^( ) ^2 gives the claim. ∎ Corollary 12. With Assumption 10, Theorem 6’s regret bound holds with κ replaced by an effective κeff(ρ)=κ/(1+ρ(L−1))κ^eff(ρ)=κ/(1+ρ(L-1)). As ρ→0ρ→ 0, κeff→κ^eff→κ. As ρ→1ρ→ 1, κeff→κ/Lκ^eff→κ/L, attenuating the framework’s gain proportionally to depth-coupling. Granularity diagnostic. For a candidate partition into L blocks, estimate ρ empirically by the largest off-diagonal block’s spectral norm divided by the geometric mean of the corresponding diagonal blocks’ top eigenvalues. A working threshold such as ρ≤0.3ρ≤ 0.3 is not attained at any granularity we measured. On ResNet-50 the mean adjacent ρ is 1.341.34 for a coarse partition into 3 super-blocks, 1.271.27 at residual-stage level (L=6L=6), and 0.920.92 per block (L=18L=18), so coarser partitions do not lower ρ. Table 2 below gives the quantitative guidance instead, reporting how much bound content κeffκ^eff retains at the measured coupling. Measured effective condition numbers. Table 2 evaluates the corollary’s κeff(ρ)=κ/(1+ρ(L−1))κ^eff(ρ)=κ/(1+ρ(L-1)) at the measured mean adjacent ρ of each backbone (measurement protocol in Appendix E). The bound retains substantial content on all four ResNets, with κeffκ^eff between 9.49.4 and 27.727.7. On ViT-B/16 it drops to 2.62.6, and only ViT-L/16 approaches the vacuous value 11 (κeff=1.11κ^eff=1.11). ResNet-50 and ResNet-101 have mean adjacent ρ≥1ρ≥ 1, outside the domain of Assumption 10; the closed form is monotone in ρ, and we report its value there as an extrapolation, marked †. Table 2: Effective condition number κeff=κ/(1+ρ(L−1))κ^eff=κ/(1+ρ(L-1)) evaluated at the measured mean adjacent ρ per backbone. †: measured ρ≥1ρ≥ 1, outside the domain of Assumption 10; the entry extrapolates the monotone closed form. Backbone L κ mean adjacent ρ κeffκ^eff ResNet-18 6 55.1 0.975 9.4 ResNet-34 6 75.4 0.967 12.9 ResNet-50 6 195.3 1.211† 27.7† ResNet-101 6 181.4 1.337† 23.6† ViT-B/16 14 31.5 0.867 2.6 ViT-L/16 26 22.3 0.764 1.11 All-pairs coupling. Adjacent pairs do not tell the whole story. We measured ρℓ,ℓ′ _ , for every block pair on ResNet-50 (15 pairs) and ViT-B/16 (91 pairs). The mean over non-adjacent pairs, 1.801.80 on ResNet-50 and 0.980.98 on ViT-B/16, exceeds the adjacent mean in the same runs (1.251.25 and 0.810.81), so the coupling is not banded around the diagonal. Part of the magnitude is a normalization artifact: on ResNet-50 the fc block’s small sℓs_ inflates every entry in its column, up to 2.92.9. Because coupling is real and non-local, we quantify its effect on the bound through κeffκ^eff and through the displacement-level probes below, not through a banded-structure argument. Direct tightness probes. We also probe Proposition 11 on actual continual-learning displacements rather than on operator norms alone. For MediumCNN, ResNet-18, and ResNet-50 on Split-CIFAR-100 (3 seeds each), we take the displacement Δθ θ accumulated while training task 2 and compare the full quadratic form Δθ⊤HΔθ θ H θ against its block-diagonal part. In 9 of 9 runs the gap |Δθ⊤HΔθ−∑ℓ(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ)|| θ H θ- _ ( θ^( )) H^( , ) θ^( )| lies within the coupling bound of Proposition 11 evaluated with the measured off-diagonal norms. Table 3 reports the ratios. The block-diagonal term captures 0.71±0.160.71± 0.16 of the full quadratic on ResNet-18 and 0.50±0.080.50± 0.08 on ResNet-50; the MediumCNN ratio is noisy because the raw magnitudes are tiny. Two caveats apply. First, 2 of the 9 runs show negative actual forgetting (backward transfer), which no positive-semidefinite quadratic model can reproduce. Second, at a 5-epoch drift distance the quadratic surrogate is rough; the trust-region caveat of Section 5 applies. Finally, the upper bound 12∑ℓsℓ‖Δθ(ℓ)‖22 12 _ s_ θ^( ) _2^2 sits two to three orders of magnitude above the realized quadratic. It is a worst-case ranking device, and Theorem 6 discounts it by the alignment factor χℓ2 _ ^2; we do not claim it is tight. Table 3: Tightness probes on the task-2 displacement (Split-CIFAR-100, 3 seeds per backbone, mean± ). “blockdiag/full” is the block-diagonal quadratic divided by the full quadratic; “sℓs_ -bound/full” is the upper bound 12∑ℓsℓ‖Δθ(ℓ)‖22 12 _ s_ θ^( ) _2^2 divided by the full quadratic; “bound holds” counts runs where |full−blockdiag||full-blockdiag| is within the measured coupling bound of Proposition 11. Backbone blockdiag/full sℓs_ -bound/full bound holds MediumCNN 1.56±2.341.56± 2.34 2463±36612463± 3661 3/3 ResNet-18 0.71±0.160.71± 0.16 780±399780± 399 3/3 ResNet-50 0.50±0.080.50± 0.08 954±514954± 514 3/3 Appendix B Full proof of Theorem 6 We work on the quadratic surrogate of LkL_k near θ⋆θ , with g=∇Lk(θ⋆)g=∇ L_k(θ ) and Q=∇2Lk(θ⋆)Q=∇^2L_k(θ ) satisfying μQI⪯Q⪯MQI _QI Q M_QI. Both Q and H are block-diagonal across layers. Per-layer KKT. At a fixed forgetting budget B>0B>0 and Lagrange multiplier β≥0β≥ 0, the per-layer subproblem for the oracle is minΔθ(ℓ)12(Δθ(ℓ))⊤Q(ℓ,ℓ)Δθ(ℓ)+g(ℓ)⊤Δθ(ℓ)+β2(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ). _ θ^( )\; 12( θ^( )) Q^( , ) θ^( )+g^( ) θ^( )+ β2( θ^( )) H^( , ) θ^( ). (19) First-order optimality gives Δθoracle(ℓ)=−(Q(ℓ,ℓ)+βH(ℓ,ℓ))−1g(ℓ) θ^( )_oracle=-(Q^( , )+β H^( , ))^-1g^( ). The uniform regularizer replaces the bracket by Q(ℓ,ℓ)+βμIQ^( , )+βμ I. Loss difference. Substituting both into the new-task quadratic surrogate L~k L_k and subtracting, L~k(Δθunif⋆)−L~k(Δθoracle⋆)=12∑ℓg(ℓ)⊤[(Q(ℓ,ℓ)+βμI)−1−(Q(ℓ,ℓ)+βH(ℓ,ℓ))−1]g(ℓ)+ℛ, L_k( θ _unif)- L_k( θ _oracle)\;=\; 12 _ g^( ) \! [(Q^( , )+βμ I)^-1-(Q^( , )+β H^( , ))^-1 ]g^( )+R, (20) where ℛR collects the cross terms and is non-negative by convexity. Spectral resolution per layer. Diagonalize H(ℓ,ℓ)=Uℓdiag(h1(ℓ),…,hdℓ(ℓ))Uℓ⊤H^( , )=U_ \,diag(h_1^( ),…,h_d_ ^( ))U_ with h1(ℓ)≤⋯≤hdℓ(ℓ)h_1^( )≤…≤ h_d_ ^( ). Let g~(ℓ)=Uℓ⊤g(ℓ) g^( )=U_ g^( ) with components g~i(ℓ) g_i^( ). Using Q⪰μQIQ _QI to upper-bound the inverse, g(ℓ)⊤[(Q(ℓ,ℓ)+βμI)−1−(Q(ℓ,ℓ)+βH(ℓ,ℓ))−1]g(ℓ)≥∑i(g~i(ℓ))2β(hi(ℓ)−μ)(μQ+βhi(ℓ))(μQ+βμ).g^( ) \! [(Q^( , )+βμ I)^-1-(Q^( , )+β H^( , ))^-1 ]g^( )\;≥\; _i( g_i^( ))^2 β(h_i^( )-μ)( _Q+β h_i^( ))( _Q+βμ). (21) Budget matching forces μ between min and max. The budget constraint ℱ~(Δθunif⋆)=B F( θ _unif)=B implies, after substitution, B=12∑ℓ∑ihi(ℓ)(g~i(ℓ))2(μQ+βμ)2.B= 12 _ _i h_i^( )( g_i^( ))^2( _Q+βμ)^2. (22) The same equation with hi(ℓ)h_i^( ) replaced by μ on the LHS would give the constraint of a uniform regularizer with sensitivity μ. Since the LHS averages hi(ℓ)h_i^( ) weighted by (g~i(ℓ))2( g_i^( ))^2, when κ>1κ>1 this average lies strictly between minℓsℓ _ s_ and maxℓsℓ _ s_ , so μ∈(minℓsℓ,maxℓsℓ)μ∈( _ s_ , _ s_ ). Lower bound by the worst layer. Restrict the sum in Eq. 21 to the maximum-h mode of layer ℓ∈ℒ+ ^+ (those with sℓ>minℓ′sℓ′s_ > _ s_ ). For any such layer the top mode satisfies hdℓ(ℓ)=sℓ>μh_d_ ^( )=s_ >μ when sℓs_ is on the larger side of μ (which happens for at least one ℓ when κ>1κ>1). Substituting the alignment (g~dℓ(ℓ))2≥χℓ2‖g(ℓ)‖22( g_d_ ^( ))^2≥ _ ^2 g^( ) _2^2 and bounding the denominator by (μQ+MQ)2( _Q+M_Q)^2 from βhi(ℓ)≤MQβ h_i^( )≤ M_Q in the relevant regime, (21)≥χℓ2‖g(ℓ)‖22⋅β(sℓ−μ)(μQ+βsℓ)(μQ+βμ)≥μQMQ+μQ⋅sℓ−μsℓ⋅χℓ2‖g(ℓ)‖22sℓ. eq:per-mode-gap\;≥\; _ ^2 g^( ) _2^2· β(s_ -μ)( _Q+β s_ )( _Q+βμ)\;≥\; _QM_Q+ _Q· s_ -μs_ · _ ^2 g^( ) _2^2s_ . (23) Using μ≤maxℓsℓ/κ=maxℓsℓ⋅minℓsℓ/maxℓsℓ=minℓsℓμ≤ _ s_ /κ= _ s_ · _ s_ / _ s_ = _ s_ in the worst case, we get (sℓ−μ)/sℓ≥(κ−1)/κ(s_ -μ)/s_ ≥(κ-1)/κ. Plugging in the constant c0=μQ/(MQ+μQ)c_0= _Q/(M_Q+ _Q) yields Theorem 6’s bound. Coupled new-task Hessian. If Q couples layers, the per-layer KKT step does not decouple cleanly. A weaker bound follows by replacing μQ _Q with the smallest eigenvalue of the off-block-diagonal-augmented Q matrix. In the worst case c0c_0 shrinks by a factor of L, but the (κ−1)/κ(κ-1)/κ structure survives. Appendix C Full proof of Theorem 8 Worst-case-direction matching. On the displacement Δθ(ℓ)=cℓv(ℓ) θ^( )=c_ v^( ), the oracle penalty contributes 12sℓcℓ2 12s_ c_ ^2 and the scalar-family penalty contributes 12λℓcℓ2 12 _ c_ ^2. The two penalties agree on this family iff λℓ/sℓ _ /s_ is independent of ℓ , hence λℓ∝sℓ _ s_ . Average-case extension. Take Δθ(ℓ) θ^( ) uniform on dℓ−1S^d_ -1 at fixed norm ‖Δθ(ℓ)‖=rℓ θ^( ) =r_ . Then [(Δθ(ℓ))⊤H(ℓ,ℓ)Δθ(ℓ)]=rℓ2tr(H(ℓ,ℓ))/dℓ=rℓ2sℓtrE [( θ^( )) H^( , ) θ^( ) ]=r_ ^2\, (H^( , ))/d_ =r_ ^2\,s_ ^tr, while the scalar-family penalty contributes 12λℓrℓ2 12 _ r_ ^2. Matching gives λℓ∝sℓtr _ s_ ^tr. Non-reproducibility on every direction. A scalar λℓ∈ℝ _ has one degree of freedom per layer. The matrix H(ℓ,ℓ)∈SymdℓH^( , ) _d_ has dℓ(dℓ+1)/2d_ (d_ +1)/2 of them. The map λℓ↦λℓI _ _ I is into the one-dimensional subspace of SymdℓSym_d_ spanned by the identity, and matching λℓI=H(ℓ,ℓ) _ I=H^( , ) requires H(ℓ,ℓ)H^( , ) itself to lie in that subspace. Appendix D SLCA LR-to-λ equivalence Consider gradient descent with step size η on the new-task quadratic surrogate L~k(Δθ)=12Δθ⊤QΔθ+g⊤Δθ L_k( θ)= 12 θ Q θ+g θ, starting from Δθ0=0 _0=0. The trajectory under LR scaling γℓ _ on layer ℓ is, per layer, Δθt+1(ℓ)=Δθt(ℓ)−ηγℓ(Q(ℓ)Δθt(ℓ)+g(ℓ)), θ^( )_t+1\;=\; θ^( )_t-η _ (Q^( ) θ^( )_t+g^( ) ), (24) which iterated for T steps with Q(ℓ)Q^( ) block-diagonal yields ΔθT(ℓ)=−[I−(I−ηγℓQ(ℓ))T](Q(ℓ))−1g(ℓ), θ^( )_T\;=\;- [I-(I-η _ Q^( ))^T ](Q^( ))^-1g^( ), (25) valid for ηγℓ<2/λmax(Q(ℓ))η _ <2/ _ (Q^( )) (numerical stability). The norm ‖ΔθT(ℓ)‖ θ^( )_T is monotone increasing in γℓ _ and saturates at ‖Q(ℓ)−1g(ℓ)‖ Q^( )\,-1g^( ) as T→∞T→∞ or γℓ _ grows. The trajectory under λ-penalized gradient descent (with no LR scaling) on the loss L~k(Δθ)+12λℓ‖Δθ(ℓ)‖2 L_k( θ)+ 12 _ θ^( ) ^2 is Δθt+1(ℓ)=Δθt(ℓ)−η((Q(ℓ)+λℓI)Δθt(ℓ)+g(ℓ)), θ^( )_t+1\;=\; θ^( )_t-η ((Q^( )+ _ I) θ^( )_t+g^( ) ), (26) with the same iterated form but Q(ℓ)→Q(ℓ)+λℓIQ^( )→ Q^( )+ _ I. The fixed point is −(Q(ℓ)+λℓI)−1g(ℓ)-(Q^( )+ _ I)^-1g^( ), which has the same direction as −(Q(ℓ))−1g(ℓ)-(Q^( ))^-1g^( ) but a smaller magnitude. Functional equivalence. The two trajectories produce different fixed points, but for any choice of the LR-scaling parameter γℓ∈(0,1] _ ∈(0,1] and total step budget T, one can find a λℓ≥0 _ ≥ 0 such that the resulting ΔθT(ℓ) θ^( )_T-norms agree. The exact match is implicit. SLCA’s design choice γbackbone=0.1,γhead=1 _backbone=0.1, _head=1 is therefore equivalent in displacement-norm to a two-group λ-schedule with λbackbone/λhead _backbone/ _head implicitly determined by T. The framework’s prediction concerns the sign (smaller LR for the more-sensitive group), not an exact algebraic identity. Appendix E Direct sℓs_ measurements Procedure. For a backbone fθf_θ and a calibration loss L(θ)=(x,y)∼p[ℓ(fθ(x),y)]L(θ)=E_(x,y) p[ (f_θ(x),y)] on a batch sampled from CL benchmark p, the per-layer Hessian block is H(ℓ,ℓ)=∇θ(ℓ)2L(θ)H^( , )=∇^2_θ^( )L(θ). We compute its top eigenvalue by power iteration on Hessian-vector products: starting from a random v∈ℝdℓv ^d_ , iterate v←HVP(L,θ(ℓ),v)v (L,θ^( ),v), v←v/‖v‖v← v/ v , returning ‖HVP(L,θ(ℓ),v)‖ (L,θ^( ),v) after k=15k=15 iterations. HVP is implemented in PyTorch via ‘torch.autograd.grad(grad, params, vec)‘. The cost is O(k)O(k) HVPs per layer (two backward passes each). For ViT-B/16 we force the math SDPA kernel since the flash and memory-efficient kernels do not support double-backward. Measured profiles. Table 4 reports the full per-layer values for both backbones, with dℓd_ included for context. Both architectures use ImageNet-pretrained weights from torchvision, evaluated on a batch of 32 random samples with a 100-class head (matching CIFAR-100 / Split-CIFAR-100 cardinality). Table 4: Measured per-layer top Hessian eigenvalue sℓs_ on a 100-class CIFAR-100-style cross-entropy minibatch. ResNet-50 ViT-B/16 block dℓd_ sℓs_ block dℓd_ sℓs_ stem 9,536 39.05 patch_embed 590,592 78.21 layer1 215,808 3,167.48 block0 7,087,872 10.70 layer2 1,219,584 519.36 block1 7,087,872 2.85 layer3 7,098,368 139.32 block2-11 ∼ 7M each 3–7 layer4 14,964,736 57.73 head 76,900 6.61 fc 204,900 23.53 κ=134.6κ=134.6 κ=27.4κ=27.4 Stability across batches. We measure sℓs_ over 5 independent calibration batches per backbone. The relative ordering of layers is stable across batches, and absolute sℓs_ values fluctuate by ∼ 5–15%. The reported κ in Table 5 is the mean-over-seeds. Backbone scan, κ vs model size. Table 5 measures κ across six standard backbones, all ImageNet-pretrained. ResNets show κ growing with depth and capacity (R18: 55, R34: 75, R50: 195, with mild saturation at R101: 181). The dominant block is consistently in the early conv stage (layer1 or layer3). Vision transformers show smaller κ (∼ 25–30) and the dominant block varies by depth (patch embedding for ViT-B, a mid-stack block for ViT-L). Both families satisfy the LIH (κ≫1κ 1). Table 5: Per-layer Hessian condition number κ=maxℓsℓ/minℓsℓκ= _ s_ / _ s_ measured across six ImageNet-pretrained backbones, with means taken over 5 calibration batches. Full per-layer numbers are in Appendix E. L is the number of architectural blocks used in the partition. Backbone L Params (M) κ Most-sensitive Least-sensitive ResNet-18 6 11.2 55.1 layer1 stem ResNet-34 6 21.3 75.4 layer3 stem ResNet-50 6 23.7 195.3 layer1 fc ResNet-101 6 42.7 181.4 layer1 layer4 ViT-B/16 14 85.7 31.5 patch_embed block1 ViT-L/16 26 303.2 22.3 block9 block0 Off-diagonal coupling ρ. We also estimate the off-diagonal block norm ‖H(ℓ,ℓ′)‖op H^( , ) _op between adjacent layer pairs and report ρℓ,ℓ′=‖H(ℓ,ℓ′)‖op/sℓsℓ′ _ , = H^( , ) _op/ s_ s_ (the quantity bounded by Assumption 10). Across the six backbones the mean adjacent ρ ranges from 0.760.76 (ViT-L/16) to 1.341.34 (ResNet-101); Table 2 in Appendix A lists all six values. Values ρ>1ρ>1 indicate that Assumption 10’s quantitative bound (which requires ρ<1ρ<1) is not strictly satisfied at adjacent residual stages. The qualitative block-diagonal structure still holds, but the regret-bound degradation in Appendix A is a genuine caveat at large coupling. Harmonized profiles across pretrainings. The ViT-B/16 profile in Table 4 uses the supervised ImageNet-1k checkpoint (IN1k-sup). Figure 4 repeats the measurement on the same architecture under two further pretrainings, ImageNet-21k supervised (IN21k) and MoCoV3 self-supervised, with an identical protocol: the same 14-block partition, the same 100-class calibration head, and a fixed calibration seed shared across the three checkpoints. Small differences between the IN1k-sup column and Table 4 come from this fixed seed. The MoCoV3 checkpoint was rebuilt from the official release; the conversion was validated with zero missing or unexpected keys on loading. Under this protocol κ is 31.531.5 for IN1k-sup (matching the 5-batch mean in Table 5), 14.614.6 for IN21k, and 27.427.4 for MoCoV3. The three profiles disagree in depth ordering: IN1k-sup and MoCoV3 peak early, at the patch embedding, while IN21k peaks late, at block10. No single fixed α therefore transfers across pretrainings; the framework conditions the schedule on the measured profile of the actual checkpoint. The MoCoV3 mid-stack is nearly flat (blocks 1–11 span 2.72.7–5.25.2), which is consistent with the near-flat sweep in Appendix G: apart from the early peak there is little heterogeneity for a depth-monotone schedule to exploit. Figure 4: Per-layer sℓs_ (log scale) on ViT-B/16 under three pretrainings, measured with the same 14-block partition, 100-class calibration head, and a fixed calibration seed. κ: 31.5 (IN1k-sup), 14.6 (IN21k), 27.4 (MoCoV3). IN1k-sup and MoCoV3 peak at the patch embedding; IN21k peaks at block10. Random-initialization profiles. All profiles above are measured on trained checkpoints. At random initialization the ordering inverts and the deepest blocks are hottest (3 seeds each). On ResNet-50, sℓs_ rises monotonically from the stem (sstem≈301s_stem≈ 301) to the fc head (sfc≈127,000s_fc≈ 127,000). ResNet-18 peaks at layer4 (≈32,000≈ 32,000), and SmallCNN has κ≈200κ≈ 200 at initialization. Heterogeneity is therefore present at init; it is the depth ordering that flips, not the size of κ. The early-layer rule of thumb is a property of trained checkpoints, not of the architecture alone. This does not affect the from-scratch EWC results in Appendix F: EWC anchors its penalty at the trained parameters after each task, and the trajectory measurements in Appendix O show the profile migrating during task 1 and becoming rank-stable from task 2 on, so the schedule is consumed at anchor points where the profile is already that of a trained network. Appendix F Depth-weighted EWC on from-scratch backbones Split-CIFAR-100 with MediumCNN. This is the discriminating from-scratch setting in our EWC experiments and is reported in the main text as Figure 2. One quantitative addition that the main-text discussion does not include is the magnitude of the forgetting trend, with forgetting climbing from ∼0.005 0.005 at α=2−2α=2^-2 to ∼0.16 0.16 at α=23α=2^3. The opposing avg-acc and forgetting signals do not contradict the framework. MediumCNN is trained from scratch and its per-layer sensitivities are not the shallow-dominated pattern Figure 1 measures on pretrained backbones, so the rule-of-thumb’s domain assumption fails here. The framework still applies because it predicts the optimum tracks sℓs_ , whatever the depth ordering of sℓs_ happens to be. The avg-acc preference for α>1α>1 on this longer task sequence is then dominated by plasticity on the new task rather than by stability on the old one. Split-CIFAR-10 with SmallCNN. On the smaller backbone and shorter task sequence, the average-accuracy effect of layer-adaptivity is statistically marginal. The avg-acc maximum at c=200c=200 is at α∈1,2α∈\1,2\ with 3-seed std bands overlapping α=0.5α=0.5, so a strict “non-uniform beats uniform” claim is not supported on this backbone. The forgetting and BWT panels are clearer: forgetting at α=23α=2^3 is ∼ 8× higher than at α=2−2α=2^-2, so deep-layer sensitivity is real even when the avg-acc metric does not separate the schedules. Figure 5: Depth-weighted EWC on Split-CIFAR-10 (SmallCNN, 5 tasks, 3 seeds, mean± bands). Same panel layout as Figure 2. Split-CIFAR-100 with ResNet-18. On a deeper backbone the geometric schedule does not consistently beat uniform. The avg-acc maximum for c=200c=200 is at α=1α=1 (mean 0.550.55) with overlapping std bands at α∈0.5,4α∈\0.5,4\. This is consistent with the measured ResNet-50 per-layer sensitivities (Figure 1), which is non-monotone in depth (a layer1 spike, then decreasing). A one-parameter geometric prior λℓ∝γℓ−1 _ γ -1 cannot fit a non-monotone sℓs_ . A non-monotone schedule (e.g., a per-block estimate of sℓs_ ) would be expected to recover the gain. Figure 6: Depth-weighted EWC on Split-CIFAR-100 with a deeper backbone (ResNet-18, 6 architectural blocks, 10 tasks, 3 seeds). Same panel layout as Figure 2. Appendix G SLCA depth-weighted sweep on MoCoV3 pretraining For comparison with the ImageNet-21k pretraining in the main text (Figure 3), we report the same depth-weighted SLCA sweep on a ViT-B/16 backbone pretrained with MoCoV3 (the pretraining used in the SLCA paper’s ablations). The full 3×53× 5 bounded grid spans 80.080.0–84.8%84.8\% final-task accuracy (87.987.9–90.4%90.4\% incremental average). Default SLCA corresponds to (c=1,α=1)(c=1,α=1) at 84.77%84.77\%. The grid optimum is (c=0.5,α=0.92)(c=0.5,α=0.92) at 84.79%84.79\%, within noise of the default. The interesting structure is row-internal: at c=2c=2 the row is monotone increasing in α (final-acc rises from 80.0%80.0\% at α=0.85α=0.85 to 84.5%84.5\% at α=1.18α=1.18), so when one over-regularizes the entire backbone, transferring some protection from shallow to deep blocks is the right correction. The near-flatness near (c=1,α=1)(c=1,α=1) is consistent with MoCoV3 having per-layer sensitivities that the default two-group LR ratio (backbone 0.1×0.1×, head 1×1×) already approximates well. Figure 7: SLCA depth-weighted sweep on Split-CIFAR-100 (ViT-B/16 + MoCoV3, 10 tasks). Same parametrization and panel layout as Figure 3. Multi-seed comparison. Table 6 repeats the key configurations of both sweeps over multiple seeds, together with the literal measured-sℓs_ schedule (the SLCA analog of Appendix K). The SLCA trainer pins the torch seed, so the seed varies only the task/class ordering; the reported spread measures robustness to task order. On ImageNet-21k the geometric schedule beats uniform on every seed. Paired by seed, the Last-acc gain is +0.450+0.450 (per-seed +0.52/+0.56/+0.51/+0.44/+0.22+0.52/+0.56/+0.51/+0.44/+0.22), paired t-test p=0.0018p=0.0018; the Wilcoxon signed-rank test gives p=0.0625p=0.0625, its smallest attainable value at n=5n=5. The Inc-acc gain is +0.112+0.112 (paired t, p=0.022p=0.022). On MoCoV3 the paired difference is −0.18-0.18 and not significant (p=0.29p=0.29). This is the parity the measured sensitivity profiles (Appendix E) predict: default SLCA’s two-group ratio already fits the MoCoV3 profile, so a depth-monotone reweighting has no room to win. The literal measured-sℓs_ schedule trails uniform on both pretrainings, mildly on ImageNet-21k and severely on MoCoV3 (−7.84-7.84 Last-acc, paired t, p=0.0021p=0.0021), consistent with the dynamic-range failure of the literal prescription documented in Appendix K. Table 6: Multi-seed SLCA on Split-CIFAR-100 (10 tasks, ViT-B/16), mean± over n task-order seeds, all values in %. Last-acc and Inc-acc as in Table 1. Geometric and measured-sℓs_ rows use each pretraining’s best (c,α)(c,α) and best c, respectively. Pretraining Schedule n Last-acc Inc-acc Forgetting ImageNet-21k uniform (c=1c=1, α=1α=1) 5 91.35±0.2391.35± 0.23 94.40±0.6994.40± 0.69 5.83±0.425.83± 0.42 ImageNet-21k geometric, ours (c=1c=1, α=1.18α=1.18) 5 91.80±0.2391.80± 0.23 94.51±0.6694.51± 0.66 4.37±0.494.37± 0.49 ImageNet-21k measured sℓs_ (c=1c=1) 3 90.96±0.1890.96± 0.18 94.45±0.2194.45± 0.21 6.61±0.286.61± 0.28 MoCoV3 uniform (c=1c=1, α=1α=1) 3 84.44±0.2984.44± 0.29 90.12±0.3290.12± 0.32 9.33±0.449.33± 0.44 MoCoV3 geometric, ours (c=0.5c=0.5, α=0.92α=0.92) 3 84.26±0.4884.26± 0.48 90.08±0.3290.08± 0.32 10.09±0.6210.09± 0.62 MoCoV3 measured sℓs_ (c=0.5c=0.5) 3 76.60±0.7176.60± 0.71 86.68±0.2286.68± 0.22 20.41±0.8520.41± 0.85 Appendix H TUNA layer-wise orthogonality sweep TUNA inserts per-layer adapters with an orthogonality penalty across tasks. The penalty has a per-layer weight λj=c/αj _j=c/α^j, fulfilling Eq. 15, with the same parametrization as Experiment 1 in the main text (here j indexes transformer blocks, L=12L=12). Sweeping α at fixed c probes whether the optimal orthogonality protection grows or shrinks with depth in a pretrained ViT-B/16 backbone. The adapter family already restricts Δθ(ℓ) θ^( ) to a 1616-dimensional subspace per transformer block, so the residual room for layer-wise reweighting is small. The result is consistent with the rule of thumb in direction but smaller in magnitude than what we see on EWC and SLCA, which is why we report it here rather than in the main text. We test on two benchmarks, namely CIFAR-100 B0-Inc10 (1010 tasks of 1010 classes) and ImageNet-R B0-Inc20 (2020 tasks, harder distribution shift). Same four metrics as Experiment 1. ImageNet-R B0-Inc20. Figure 8 shows the result on the more demanding ImageNet-R B0-Inc20 setting. The optimum is visibly above α=1α=1, around α∈21,22α∈\2^1,2^2\, suggesting that on this dataset the orthogonality penalty should be relaxed on the late blocks. Figure 8: TUNA orthogonality sweep on ImageNet-R B0-Inc20 (ViT-B/16). Per-layer orthogonality weight λj=c/αj _j=c/α^j. Top, final average accuracy (left) and forgetting (right). Bottom, average incremental accuracy (left) and BWT (right). The optimum is visibly above α=1α=1, suggesting that on this benchmark late-block orthogonality should be relaxed relative to early blocks. CIFAR-100 B0-Inc10. The accuracy span across the (α,c)(α,c) grid on this easier benchmark is only about 0.25%0.25\%, since the adapter constraint already does the bulk of the work and the residual layer-wise tuning has limited room to act. Within that small span the optimum lies away from α=1α=1 for every c, with the typical preference around α∈2−1,21α∈\2^-1,2^1\. Figure 9: TUNA orthogonality sweep on CIFAR-100 B0-Inc10 (ViT-B/16, 10 tasks). Per-layer orthogonality weight λj=c/αj _j=c/α^j. Same panel layout as Figure 8. Appendix I Schedule-shape comparison The framework predicts that the per-layer optimum tracks sℓs_ , but in practice we replace the prescription with a one-parameter family. If the rule of thumb (protect shallow layers strongly, let deeper layers move) is the load-bearing piece, the specific monotone-decreasing shape should matter less than the principle. We test this directly by running two additional schedule families on MediumCNN/Split-CIFAR-100 (3 seeds each): • Linear: λℓ=c⋅(1−(α−1)(ℓ−1)/(L−1)) _ =c· (1-(α-1)( -1)/(L-1) ), clipped to [c/100,c][c/100,c], with α∈1.5,2,3α∈\1.5,2,3\. • Step (two-group): λℓ=c _ =c for ℓ≤L/2 ≤ L/2 and λℓ=c/α _ =c/α otherwise, with α∈2,4,8α∈\2,4,8\. Figure 10 reports the best avg-acc across each family’s α grid, alongside the geometric family already in the main text. All three monotone-decreasing schedules consistently beat uniform; the absolute gains over uniform are ∼ 2–7% depending on c, and the three families’ best points lie within ∼ 1–4% of each other. The step family is in fact the strongest at higher c, edging out geometric. The shape of the schedule is therefore a second-order choice once the rule of thumb is followed. Figure 10: Best avg-acc per schedule family on MediumCNN/Split-CIFAR-100 (3-seed mean ± std). Bars: best over α within each family. Dashed line and shaded band: uniform EWC (α=1α=1). All three monotone-decreasing schedules consistently beat uniform; the differences between schedule shapes are within or near the seed-to-seed std bands. Linear and step are not run at c=20,000c=20,000 since the rank ordering is already clear at smaller c. Appendix J Full proof of Proposition 5 Let A∈ℝd×dA ^d× d be positive semidefinite with eigenvalues λ1≥⋯≥λd≥0 _1≥…≥ _d≥ 0. Left inequality. ∑pApp=tr(A)=∑iλi _pA_p= (A)= _i _i. The arithmetic mean of non-negative reals is at most the maximum, so 1d∑pApp=1d∑iλi≤λmax(A) 1d _pA_p= 1d _i _i≤ _ (A), with equality iff all λi _i are equal, i.e., A=σIA=σ I. Right inequality. All eigenvalues being non-negative implies ∑iλi≥λmax(A) _i _i≥ _ (A), hence 1d∑pApp≥λmax(A)/d 1d _pA_p≥ _ (A)/d, which rearranges to λmax(A)/(1d∑pApp)≤d _ (A)/( 1d _pA_p)≤ d. Equality holds iff exactly one eigenvalue is nonzero, i.e., rank(A)≤1rank(A)≤ 1. Intermediate values. Take At=tI+(1−t)⊤A_t=tI+(1-t)11 with t∈[0,1]t∈[0,1]. Every diagonal entry equals 11, so 1d∑p(At)pp=1 1d _p(A_t)_p=1. The eigenvalues of AtA_t are t+(1−t)dt+(1-t)d (once, with eigenvector /d1/ d) and t (with multiplicity d−1d-1, on the orthogonal complement). Thus λmax(At)/(1d∑p(At)pp)=t+(1−t)d _ (A_t)/( 1d _p(A_t)_p)=t+(1-t)d, which is continuous in t and sweeps [1,d][1,d] as t runs from 11 down to 00. ∎ Appendix K Direct test of the prescription on EWC Theorem 8 prescribes λℓ∝sℓ _ s_ . The main-text experiments use a one-parameter geometric proxy. For completeness we also tested the literal prescription on EWC, using the per-layer sℓs_ measured on the ImageNet-pretrained checkpoint (Appendix E) directly as the schedule. Three seeds, four c values, ResNet-18 and ResNet-50 on Split-CIFAR-100. Table 7: Direct measured-s schedule vs. uniform EWC on Split-CIFAR-100 (3 seeds, mean± ). Best c per backbone in bold. c uniform (α=1α=1) measured-s schedule ResNet-50 / Split-CIFAR-100 50 — 0.330±0.1170.330± 0.117 200 0.453±0.0420.453± 0.042 0.312±0.0580.312± 0.058 1000 0.407±0.0360.407± 0.036 0.278±0.0540.278± 0.054 5000 0.366±0.0130.366± 0.013 0.195±0.0270.195± 0.027 ResNet-18 / Split-CIFAR-100 50 — 0.510±0.0510.510± 0.051 200 0.555±0.0090.555± 0.009 0.498±0.0090.498± 0.009 1000 0.515±0.0220.515± 0.022 0.409±0.0510.409± 0.051 5000 0.415±0.0760.415± 0.076 0.249±0.0480.249± 0.048 The measured-s schedule underperforms uniform on both backbones, by 4–14% absolute. This is a real negative result that the framework does not, by itself, predict away. We see two factors that plausibly contribute. The first is dynamic range, in that on ResNet-50 the measured ratio slayer1/sfcs_layer1/s_fc is 134134, so the literal λℓ∝sℓ _ s_ schedule sets λlayer1/λfc≈134 _layer1/ _fc≈ 134, which over-protects layer1 in absolute terms and starves the head of plasticity. The second is per-layer measurement noise, since each sℓs_ is estimated from a single calibration batch and the literal schedule applies the raw ratios with no smoothing across layers. Trajectory drift away from the pretraining checkpoint, where the ratios at θ⋆θ need not hold along the optimizer path, is a third candidate; the re-measurement experiment below isolates it. The one-parameter geometric proxy used in the main text trades exact-ratio matching for a regularising prior across α, and that single-slope smoothing is what we believe carries the gain. The framework’s prediction is therefore narrower than “apply λℓ∝sℓ _ s_ literally and you win.” A correct reading is that the optimum within the scalar-per-layer family lies in the direction of sℓs_ , and the geometric one-parameter approximation is a robust way to sweep that direction without committing to exact per-layer ratios that depend on noisy per-batch curvature estimates. The schedule-shape comparison in Appendix I reinforces this, showing that the broad family of monotone-decreasing schedules consistently beats uniform. Interpolated schedules. The dynamic-range and measurement-noise factors suggest smoothing the literal schedule rather than abandoning it. We test the one-parameter family λℓ∝(sℓ/smed)β _ (s_ /s_med)^β, where smeds_med is the median of the measured sℓs_ : β=0β=0 recovers uniform EWC, β=1β=1 recovers the literal measured schedule above, and intermediate β compresses the per-layer ratios toward one. We sweep β∈0,0.25,0.5,0.75,1β∈\0,0.25,0.5,0.75,1\ and c∈50,200,1000,5000c∈\50,200,1000,5000\ with 3 seeds on Split-CIFAR-100. Figure 11 reports avg-acc at the best c per cell. ResNet-18 has an interior optimum at β∗=0.5β^*=0.5 (0.590±0.0140.590± 0.014 vs 0.557±0.0210.557± 0.021 for uniform), so a measured-s schedule beats uniform on the very backbone where the one-parameter geometric prior gains nothing (Appendix F). ResNet-50 has a boundary optimum at β=0β=0 and decreases monotonically in β; its extreme dynamic range (κ=195.3κ=195.3, with the non-monotone layer1 spike) defeats power scaling. Forgetting decreases monotonically in β on both backbones, the stability-plasticity direction the framework predicts, and avg-acc varies smoothly in β with no cliffs. The reading is consistent with the theory. Theorem 8 fixes the direction of the schedule from the worst-case displacement. The average-case optimum λℓ∝tr(H(ℓ,ℓ))/dℓ _ (H^( , ))/d_ (Appendix C) keeps the same ordering while compressing the dynamic range (Appendix M). The exponent β<1β<1 plays the same compressing role and acts as shrinkage against per-batch measurement noise. Figure 11: The interpolated family λℓ∝(sℓ/smed)β _ (s_ /s_med)^β on Split-CIFAR-100 (left: ResNet-18, right: ResNet-50; 3 seeds, mean± , one curve per c). At the best c per cell, ResNet-18 peaks at β∗=0.5β^*=0.5 (0.590±0.0140.590± 0.014 vs 0.557±0.0210.557± 0.021 uniform); ResNet-50 is best at β=0β=0 (0.441±0.0150.441± 0.015) and decreases monotonically to 0.330±0.1200.330± 0.120 at β=1β=1. Per-task re-measurement. The drift factor can be isolated directly. We re-measure sℓs_ at every task boundary and rebuild the schedule with the interpolated family above, in two arms: online β=1β=1 (literal) and online β=0.5β=0.5 (smoothed). Same benchmark, seeds, and c grid; every measurement averages three calibration batches. Table 8 reports avg-acc at the best c per arm. The online-smoothed arm is the strongest ResNet-18 arm in this comparison and halves forgetting relative to uniform (0.0300.030 vs 0.0620.062). On ResNet-50, re-measurement repairs the static schedule’s failure back to uniform parity (0.462±0.0560.462± 0.056 vs 0.453±0.0340.453± 0.034). The static baseline on ResNet-18 also lands above the single-calibration-batch schedule of Table 7 (0.5750.575 vs 0.5100.510), consistent with the measurement-noise factor above. The overhead is measured rather than estimated: 216216 HVPs per boundary (1212 power iterations × 66 blocks × 33 calibration batches), ≈ 1.5 s per boundary on ResNet-18 (∼ 15% of training wall-clock) and ≈ 3.6 s on ResNet-50. Together with the β sweep, this locates the causes of the negative result above: drift is real but secondary, and magnitude smoothing is primary. Table 8: Per-task re-measured schedules on Split-CIFAR-100: avg-acc at the best c per arm (3 seeds, mean± ). Best arm per backbone in bold. Arm ResNet-18 ResNet-50 uniform (α=1α=1) 0.555±0.0080.555± 0.008 0.453±0.0340.453± 0.034 static measured 0.575±0.0090.575± 0.009 0.330±0.0950.330± 0.095 online β=1β=1 0.570±0.0110.570± 0.011 0.462±0.0560.462± 0.056 online β=0.5β=0.5 0.589±0.0080.589± 0.008 0.460±0.0330.460± 0.033 Appendix L Forgetting swing vs measured κ Theorem 6 predicts that the regret of layer-uniform regularization grows with the layer condition number κ. We give a coarse quantitative test by plotting, for each EWC backbone, the forgetting swing: the gap between forgetting at the most-deep-protective α=2−2α=2^-2 and at the least-deep-protective α=23α=2^3, taken as the maximum across c values. This is the empirical analog of the regret bound: the more sensitive the deep layers are (high κ), the more forgetting one pays for under-protecting them. Figure 12: Per-backbone forgetting swing on Split-CIFAR-10/CIFAR-100 (forg at α=23α=2^3 minus forg at α=2−2α=2^-2, max across c) vs measured κ. Backbones with measured κ (ResNet-18, ResNet-50) are plotted at their true κ; SmallCNN/MediumCNN are trained from scratch and are placed on the left margin since their sℓs_ profile was not measured. The trend is monotone in κ on the two pretrained backbones and consistent in magnitude with the bound’s (κ−1)/κ(κ-1)/κ scaling. Appendix M Trace proxy vs operator-norm sensitivity Theorem 8 sets λℓ∝sℓ _ s_ , the top Hessian eigenvalue, because forgetting is dominated by the worst displacement direction. If one instead averages forgetting over isotropic random displacement directions, the optimum becomes λℓ∝tr(H(ℓ,ℓ))/dℓ _ (H^( , ))/d_ , the average eigenvalue per layer. The two summaries can be very different (Proposition 5), and SGD’s actual trajectories preferentially follow top eigendirections [21, 61] rather than isotropic ones, so sℓs_ is the right summary for the regret bound. The trace summary is nevertheless useful as a cheap proxy, costing one Hutchinson HVP per layer instead of 1010–2020 power iterations. Figure 13 plots both quantities side by side on ResNet-50 and ViT-B/16, each normalised by its own median for visual comparability. Figure 13: Per-layer operator-norm sℓs_ (blue) and trace-divided-by-width sℓtr=tr(H(ℓ,ℓ))/dℓs_ ^tr= (H^( , ))/d_ (orange), each normalised by its median. Both signals identify the same dominant block (layer1 on ResNet-50, patch_embed on ViT-B/16), but the operator norm has a sharper peak. The trace proxy is a reasonable cheap surrogate for the qualitative ordering, but understates the dynamic range of κ. The takeaway: the trace proxy preserves the qualitative ordering and identifies the same most-sensitive block, but compresses the spread by a factor of several. For a practitioner choosing a depth-weighted schedule, the cheap trace estimate is a good first sweep direction; for the regret bound in Theorem 6, which depends on the operator-norm κ, the power-iteration estimate remains the right quantity. Appendix N LIH on the BERT-base language backbone The LIH measurements in the main text and Appendix E are on six standard vision backbones (ResNet-18/34/50/101, ViT-B/16, ViT-L/16). The framework is stated for any layer-partitioned neural network, but a reasonable concern is whether the shallow-dominated sℓs_ pattern transfers beyond computer vision. We test this on BERT-base [8]: same per-layer power-iteration procedure as Appendix E, calibration batch of 3232 random token sequences (max length 6464, CLS/SEP boundary tokens preserved), 1010-class linear head on the CLS embedding, 1515 power-iteration steps, 33 calibration seeds. Figure 14 reports the per-layer profile. The token-embedding block dominates with sℓ=70.4±0.0s_ =70.4± 0.0, the head sits at 20.820.8, and the twelve transformer blocks span sℓ∈[7.7,26.3]s_ ∈[7.7,26.3]. The measured layer condition number is κ=9.15κ=9.15, modest compared to the 2020–200200 range observed on vision backbones but still strictly above 11. The qualitative pattern matches the LIH: a small number of layers (here, the embedding) dominate the loss curvature, with the rest of the architecture spread across roughly an order of magnitude. The framework’s prescription therefore extends to NLP backbones in principle, with the specific schedule depending on the per-layer profile of the chosen model. Figure 14: Per-layer log10sℓ _10s_ on BERT-base, evaluated on a random-token calibration batch with a 1010-class linear head on CLS. Embeddings dominate (sembed=70s_embed=70), with the twelve transformer blocks in [8,26][8,26] and the head at shead=21s_head=21. Measured κ=9.15κ=9.15. Two caveats. First, κ on BERT is smaller than on vision backbones; the rule-of-thumb gain from a depth-weighted regularizer should be correspondingly smaller per Theorem 6. Second, our calibration batch is random tokens rather than a real text-classification benchmark, so the absolute magnitudes will shift on a downstream task. The per-layer ordering is what the framework consumes, and it is consistent with the vision-backbone pattern: a small number of high-curvature layers, here the token embedding, dominate the loss. A text continual-learning benchmark. The measurement above is a curvature profile, not a task-performance result. To close the loop, we ran an end-to-end benchmark on the five-dataset text-classification sequence of Huang et al. [20]: AG News, Yelp Review Full, Amazon Review Full, Yahoo Answers, and DBpedia, in that fixed order. The backbone is BERT-base with a separate classification head per task; EWC regularizes the shared encoder parameters. We use the reduced setting of IDBR [20], with 20002000 training and up to 10001000 test examples per class, 33 seeds, and c∈100,1000,10000c∈\100,1000,10000\. Table 9 reports sequential fine-tuning and both EWC arms at the strongest regularization strength. Table 9: Five-dataset text continual learning with BERT-base (3 seeds, mean± ). SeqFT is sequential fine-tuning without regularization; both EWC arms are shown at c=10000c=10000. Method Avg-acc Forgetting SeqFT 0.624±0.0100.624± 0.010 0.158±0.0110.158± 0.011 Uniform EWC (c=10000c=10000) 0.728±0.0020.728± 0.002 0.027±0.0060.027± 0.006 Measured-s EWC (c=10000c=10000) 0.722±0.0050.722± 0.005 0.035±0.0040.035± 0.004 EWC itself transfers to language: uniform EWC gains 1010 percentage points of avg-acc over sequential fine-tuning and cuts forgetting from 0.1580.158 to 0.0270.027. The measured-s schedule matches uniform within noise. The difference is between −0.6-0.6 and +0.3+0.3 percentage points at every c and is never significant (p≥0.09p≥ 0.09). This null is the outcome Theorem 6 predicts. The measured κ=9.15κ=9.15 on BERT-base is the smallest backbone-level value we observe, and the regret bound in Theorem 6 is monotone non-decreasing in κ, so the predicted layer-adaptive gain is smallest exactly on this backbone. The benchmark therefore confirms the scope condition: layer-adaptive schedules pay off where the measured κ is large, and the same measurement predicts in advance where they do not. Appendix O Per-task sℓs_ variation and ordering stability The Layer-Importance Hypothesis (Figure 1) is implicitly a claim about the architecture: sℓs_ varies dramatically across layers, and this variation should be a property of the network rather than of any particular calibration batch. We test this directly on ResNet-50 by drawing the calibration batch from each Split-CIFAR-100 task in turn (10-class subsets, 3232 samples per batch) and measuring per-layer sℓs_ at each. Figure 15 shows the resulting trajectories. Figure 15: Per-layer sℓs_ on ResNet-50 measured with calibration batches drawn from each of the 1010 Split-CIFAR-100 tasks separately. Each line is one architectural block; the x-axis is task index. The per-block ordering (layer1 on top, then layer3 and fc, then layer2 and layer4, with stem at the bottom) is preserved across all 1010 tasks. Absolute magnitudes vary by a factor of ∼2 2 within each block, but the rank is invariant. Two observations. First, the per-layer rank is stable: layer1 dominates on every task (sℓ=12.3±1.5s_ =12.3± 1.5 across tasks), with layer3 and fc consistently second-tier and stem consistently lowest. The shallow-conv-dominated pattern from Figure 1 therefore does not depend on a specific calibration batch and reads through to every task’s data slice. Second, absolute sℓs_ values vary within a factor of ∼2 2 across tasks, and the per-task κ values cluster in 4.54.5–6.46.4. This is much smaller than the κ=195κ=195 reported on the same backbone with a 100-class head in Appendix E, because here each task has only 1010 classes and the cross-entropy loss has correspondingly less spread. The framework’s prescription is invariant to the absolute scale of κ (it only uses the per-layer ratios), so this scale shift does not affect the rule of thumb; it does say that quoting a single κ for a backbone hides real data dependence in the absolute magnitudes. Table 10: Per-experiment compute summary. EWC and SLCA depth-weighted sweeps use 3 seeds where indicated. “Wallclock per job” is the typical end-to-end time on a single allocation; “Total” aggregates across the full grid. Experiment # jobs Wallclock/job GPU-h (approx.) EWC sweep, SmallCNN/CIFAR-10 (3 seeds, 6×4 grid) 72 5–10 min 10 EWC sweep, MediumCNN/CIFAR-100 (3 seeds, 6×4 grid) 72 10–20 min 20 EWC sweep, ResNet-18/CIFAR-100 (3 seeds, 6×4 grid) 72 20–40 min 40 EWC sweep, ResNet-50/CIFAR-100 (3 seeds, 6×4 grid) 72 30–60 min 60 EWC measured-s, ResNet-18/CIFAR-100 (3 seeds, 4 c, Appendix K) 12 20–40 min 7 EWC measured-s, ResNet-50/CIFAR-100 (3 seeds, 4 c, Appendix K) 12 30–60 min 10 EWC schedule-shape, MediumCNN/CIFAR-100 (linear+step, Appendix I) 54 10–20 min 15 EWC schedule-shape, SmallCNN/CIFAR-10 (linear+step, Appendix I) 54 5–10 min 7 EWC schedule-shape, ResNet-18/CIFAR-100 (linear+step, Appendix I) 54 20–40 min 30 EWC 20-task Split-CIFAR-100 (3 seeds, 6×4 grid) 72 20–40 min 35 TUNA, ImageNet-R B0-Inc20 ∼ 24 60–120 min 36 TUNA, CIFAR-100 B0-Inc10 (Appendix H) ∼ 24 60–120 min 36 SLCA depth-weighted, MoCoV3 (15 configs, 90 ep/task) 15 240–360 min 75 SLCA depth-weighted, ImageNet-21k (15 configs, 20 ep/task) 15 25–40 min 8 Hessian power iteration, 6 backbones (Appendix E) 6 30–60 min 4 Hessian extras: per-task sℓs_ , empirical Fisher, granularity, BERT-base NLP 4 20–60 min 3 Total reported runs ∼ 397 Within-task and across-task trajectories. The measurements above vary the calibration task at a fixed trained checkpoint. A complementary question is how sℓs_ evolves along the training trajectory itself. We measure sℓs_ at 1111 checkpoints during task 1 of Split-CIFAR-100 (33 seeds) on pretrained ResNet-50 and on from-scratch ResNet-18 and SmallCNN, and summarize rank stability as the Spearman correlation between the profile at step t and the final within-task profile. Figure 16 shows the trajectories. On pretrained ResNet-50 the correlation stays at or above 0.700.70 at every checkpoint: the ranking measured at θ⋆θ does not go stale within a task, only the magnitudes move. The from-scratch runs are rank-unstable before roughly step 200200 and stabilize thereafter. Mid-training, the top-curvature estimate on ResNet-18 and ResNet-50 can be negative; the trajectory measurements use a signed Rayleigh-quotient estimator, and we report magnitudes with a sign flag. Figure 16: Per-layer log10|sℓ| _10|s_ | at 11 checkpoints during task 1 of Split-CIFAR-100 (one panel per backbone; mean over 3 seeds with seed band; open markers flag negative signed-Rayleigh curvature; dashed line: κ(t)κ(t)). Across task boundaries, we track a from-scratch ResNet-50 (CIFAR variant; blocks stage1–stage3 and fc) trained with EWC (c=200c=200) on Split-CIFAR-100, snapshotting the profile after tasks 1, 2, 5, and 10. At initialization the profile is fc-heavy (sℓ=32,600s_ =32,600 on fc vs 11,50011,500 on stage3). After task 1 it has reordered to stage3-heavy (21.221.2 vs 1.91.9). From task 2 onward the ranking is stable while the magnitudes grow: stage3 passes through 478478, 1,7681,768, and 3,2013,201, and fc through 2020, 8181, and 126126, after tasks 2, 5, and 10 respectively. The two views agree. Profiles anchored at a trained θ⋆θ are rank-stable, both within a task and across tasks from task 2 on, whereas profiles taken at initialization or early in from-scratch training do not predict the trained ordering. This supports re-measuring the schedule at task boundaries rather than fixing it at initialization. Appendix P Comparison with experience replay The main text treats replay as an orthogonal family (Section 2). Here we quantify the comparison directly. We run experience replay (ER) on Split-CIFAR-100 with the same task split, backbones, and training protocol as the EWC experiments (Section 4): class-balanced index replay over past-task data, memory budgets M∈500,2000,6200M∈\500,2000,6200\ stored examples, replay batch size equal to the current-task batch, 3 seeds. The largest budget is chosen so that the buffer roughly matches the extra storage EWC itself carries on MediumCNN, anchor parameters plus Fisher values, i.e., 2×2× the parameter count at 4 bytes each. Table 11: Average accuracy on Split-CIFAR-100 (3 seeds, mean± ): ER at three memory budgets, the EWC arms at their best configuration, and the composition of the two families. Best c is 200200 throughout; the best geometric α is 22 on MediumCNN and ResNet-50 and 44 on ResNet-18. Method MediumCNN ResNet-18 ResNet-50 ER (M=500M=500) 0.441±0.0120.441± 0.012 0.658±0.0090.658± 0.009 0.620±0.0270.620± 0.027 ER (M=2000M=2000) 0.443±0.0140.443± 0.014 0.725±0.0120.725± 0.012 0.684±0.0310.684± 0.031 ER (M=6200M=6200) 0.493±0.0020.493± 0.002 0.775±0.0120.775± 0.012 0.720±0.0110.720± 0.011 EWC uniform (best c) 0.572±0.0040.572± 0.004 0.555±0.0090.555± 0.009 0.453±0.0420.453± 0.042 EWC geometric (best α, c) 0.598±0.0230.598± 0.023 0.542±0.0380.542± 0.038 0.431±0.0230.431± 0.023 ER+EWC (M=500M=500, c=200c=200, α=2α=2) 0.574±0.0050.574± 0.005 0.593±0.0440.593± 0.044 0.505±0.0170.505± 0.017 Three observations. First, ER dominates both EWC arms on the ResNets at every budget, consistent with prior reports that replay is a strong baseline on deep networks [3, 36]. Second, the ordering reverses on the small from-scratch CNN: every EWC arm beats ER at every budget, and the geometric schedule is the best arm overall. Third, the two families compose. The ER+EWC row combines a small buffer (M=500M=500) with the geometric schedule at α=2α=2 and c=200c=200; on MediumCNN this beats ER alone at every budget. Replay also carries a cost the table does not show: the buffer stores raw past-task data, which is unavailable under privacy or data-retention constraints. Layer-adaptive regularization applies unchanged in that regime and, per the last row, remains useful when storage is allowed. Appendix Q Broader impacts The paper’s contribution is theoretical and diagnostic. The most direct positive impact is efficiency: a layer-adaptive regularizer that recovers the same continual-learning quality with a smaller forgetting budget reduces the compute needed to keep deployed models current with new data, an effect that compounds with the total number of model updates a system performs over its lifetime. The associated diagnostic (κ measurement) is a single-number test for whether layer-adaptive regularization is worth using on a given backbone, which spares practitioners from running a full sweep when uniform regularization is already adequate. The paper also has a dual-use facet that we want to acknowledge directly. The per-layer top Hessian eigenvalue sℓs_ that our framework prescribes for protective regularization is the same per-layer sensitivity quantity that the adversarial weight-attack literature uses to identify which layers are most worth attacking with bit-flip, weight-perturbation, or trojan-implant procedures [37, 55, 6]. Our work does not introduce a new attack capability (the bit-flip literature already operates on the same quantity), but reframing sℓs_ as a CL hyperparameter could in principle make the per-layer sensitivity profile a more visible target for adversaries planning weight-space attacks on deployed models. We mitigate this by working only with publicly available, non-safety-critical benchmarks (Split-CIFAR-10/100, ImageNet-R) and by releasing no new model checkpoints. We recommend that practitioners who measure sℓs_ on production models treat the resulting per-layer profile with the same care they would apply to any other model-internal artefact that informs an attack surface. A second consideration is more diffuse. Improved continual learning makes long-running personalized systems (recommenders, assistants, on-device adapters) easier to keep current. This is a property of better CL in general rather than a specific contribution of this paper, but it interacts with privacy in the usual way: a system that retains and integrates information well across tasks also retains and integrates information well across users and time, which sharpens the existing tension between personalization quality and data minimization. Any deployment of CL methods on user data should pair the stability gains we discuss with explicit data-retention and consent policies. Appendix R Compute resources All experiments ran on an internal HPC cluster with NVIDIA GPUs (B200, H200, H100, A100-80GB, A100-40GB, RTXA6000) inside an NVIDIA PyTorch 25.01 enroot container, scheduled by SLURM. Per-job resources requested: 1 GPU, 4–8 CPUs, 32–64 GB RAM, 3–10 h walltime depending on workload. Approximate wallclock and total compute per experiment block, including the appendix-only sweeps: The full research effort (preliminary sweeps, debugging runs, abandoned configurations not reported here) consumed roughly 22–3×3× the total in Table 10, i.e., on the order of 800800–12001200 GPU-hours. A later round of experiments, including the replay comparison in Appendix P, added roughly 300300 further GPU-jobs of 0.1–2 h each. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and Section 1 state the four main results (Theorems 2, 6, 8 and Proposition 5). They also state how two existing layer-adaptive CL methods (TUNA and SLCA) and our own depth-weighted EWC variant fit the scalar-per-layer family. Each claim is formalized in Section 3 and tested in Section 4. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 5 explicitly lists five caveats: local validity of the second-order model (trust region), block-diagonal Hessian approximation, F=HF=H identity at exponential-family optima, functional (not algebraic) SLCA reduction, and locality of sℓs_ measurement. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: All theorems, propositions, corollaries, and definitions are numbered and cross-referenced. Assumption 1 is stated before every result that uses it. Theorem 2 and Proposition 5 are proved in full in the main text; Theorems 6 and 8 have main-text proof sketches with full proofs deferred to Appendices B and C. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 4 reports the full grid for each experiment: depth-weighted EWC sweeps over α∈0.25,0.5,1,2,4,8α∈\0.25,0.5,1,2,4,8\ and c∈200,1000,5000,20000c∈\200,1000,5000,20000\ on SmallCNN (5 layers) on Split-C10, MediumCNN (6 layers) on Split-C100, and ResNet-18 on Split-C100, all over 3 seeds with 2 epochs per task; TUNA orthogonality sweeps over the same (c,α)(c,α) parametrization on ViT-B/16 with C100 B0-Inc10 and ImNet-R B0-Inc20; SLCA depth-weighted sweeps over c∈0.5,1,2c∈\0.5,1,2\ and α∈0.85,0.92,1,1.08,1.18α∈\0.85,0.92,1,1.08,1.18\ on ViT-B/16 + MoCoV3 and ViT-B/16 + ImNet-21k, both on Split-C100. The remaining hyperparameters (optimizer, batch size, Fisher-sample count, milestones) are listed in the companion repository. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The main contribution is theoretical, but every empirical figure in the paper (the per-layer Hessian measurement of Figure 1, the multi-seed depth-weighted EWC sweeps, the TUNA orthogonality sweeps, and the two SLCA depth-weighted sweeps) is produced by an anonymous companion codebase. We will release this codebase together with the camera-ready, including the launch scripts and per-task JSON outputs that generate every plot. The submission text gives the full (c,α)(c,α) grids, backbones, datasets, and seeds needed to reproduce the runs from public sources alone. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Section 4 reports architecture (SmallCNN, 5 layers; MediumCNN, 6 layers), dataset (Split-C10, 5 tasks; Split-C100, 10 tasks), epochs per task, and the full (α,c)(α,c) grid. Full hyperparameter specifications (optimizer, batch size, Fisher-sample size) are in the companion repository. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: The depth-weighted EWC sweeps report mean curves with shaded ±1± 1 std bands across 3 seeds (216 runs total). The TUNA and SLCA sweeps are single-seed grid sweeps following the protocol of the original methods. Each of the two SLCA (c,α)(c,α) grids spans 15 configs, and the trend is read across the grid rather than from per-cell error bars. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Appendix R (Table 10) reports per-experiment GPU type, memory, walltime per job, and total GPU-hours, including a full-research-effort estimate that covers preliminary and abandoned runs. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: The paper is theoretical and the supporting experiments use only public continual-learning benchmarks (Split-C10/100, ImNet-R/A, ObjectNet, CUB-200, Cars-196). No human subjects, private data, or deployment risks are involved. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No] , they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Appendix Q discusses positive impacts (compute efficiency for keeping deployed models current; a single-number diagnostic that spares unnecessary sweeps), the dual-use overlap between the per-layer sensitivity sℓs_ used here for regularization and the same quantity used by the adversarial weight-attack literature, and the indirect privacy interaction between better continual learning and long-running personalized systems. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No] , they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: The paper releases no new models or datasets; it analyzes existing continual-learning methods and uses public benchmarks only. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All prior methods discussed (EWC, SI, MAS, TUNA, SLCA, K-FAC) and datasets used by the companion experiments (Split-C10/100, ImNet-R/A, ObjectNet, CUB-200, Cars-196) are cited with their original publications in Section 2 and the References. The companion repository documents dataset licenses for the empirical runs. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [N/A] Justification: The paper does not release new datasets, models, or trained checkpoints with this submission. Companion experimental code will be released separately with the camera-ready version. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing or research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve human subjects or crowdsourcing, so IRB approval is not applicable. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: LLMs are not part of the methodology or any core component of this research; any use of LLM tools was limited to writing and formatting, which per NeurIPS policy does not require declaration. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.