Paper deep dive

Beta-Scheduling: Momentum from Critical Damping as a Diagnostic and Correction Tool for Neural Network Training

Ivan Pasichnyk

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 50

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 4/1/2026, 1:27:57 AM

Summary

The paper introduces a diagnostic pipeline for neural networks based on the damped harmonic oscillator model of SGD with momentum. It proposes a parameter-free 'beta-scheduling' (μ(t) = 1 - 2*sqrt(α(t))) that achieves 1.9x faster convergence to 90% accuracy. The pipeline uses gradient attribution on misclassified images to identify 'problem layers' and performs surgical correction, achieving 82% compute savings compared to full retraining while fixing specific failure modes.

Entities (5)

CIFAR-10 · dataset · 99%ResNet-18 · neural-network-architecture · 99%Beta-scheduling · optimization-technique · 98%Damped Harmonic Oscillator · physical-model · 95%Surgical Correction · diagnosticrepair-method · 95%

Relation Signals (3)

Damped Harmonic Oscillator → informsdesignof → Beta-scheduling

confidence 98% · We derive a time-varying momentum schedule from the critically damped harmonic oscillator

Beta-scheduling → improvesconvergenceof → ResNet-18

confidence 95% · On ResNet-18/CIFAR-10, beta-scheduling delivers 1.9x faster convergence

Surgical Correction → fixeserrorsin → ResNet-18

confidence 92% · Surgical correction of only these layers fixes 62 misclassifications

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Standard neural network training uses constant momentum (typically 0.9), a convention dating to 1964 with limited theoretical justification for its optimality. We derive a time-varying momentum schedule from the critically damped harmonic oscillator: mu(t) = 1 - 2*sqrt(alpha(t)), where alpha(t) is the current learning rate. This beta-schedule requires zero free parameters beyond the existing learning rate schedule. On ResNet-18/CIFAR-10, beta-scheduling delivers 1.9x faster convergence to 90% accuracy compared to constant momentum. More importantly, the per-layer gradient attribution under this schedule produces a cross-optimizer invariant diagnostic: the same three problem layers are identified regardless of whether the model was trained with SGD or Adam (100% overlap). Surgical correction of only these layers fixes 62 misclassifications while retraining only 18% of parameters. A hybrid schedule -- physics momentum for fast early convergence, then constant momentum for the final refinement -- reaches 95% accuracy fastest among five methods tested. The main contribution is not an accuracy improvement but a principled, parameter-free tool for localizing and correcting specific failure modes in trained networks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

49,411 characters extracted from source content.

Expand or collapse full text

A Physics-Based Diagnostic Pipeline for Neural Networks: From Damping Regimes to Surgical Layer Repair Ivan Pasichnyk We Label Data Inc. ivan@welabeldata.com https://orcid.org/0009-0004-8154-3345 Abstract When a neural network fails on specific inputs, standard tools identify what the model gets wrong but not which layers are responsible or how to fix the errors without full retraining. We present a diagnostic pipeline that addresses this gap by connecting three ideas: (1) the damped harmonic oscillator model of SGD with momentum (Qian, 1999), which classifies each training epoch as underdamped, critically damped, or overdamped; (2) gradient attribution computed exclusively on misclassified images, which localizes errors to specific layer groups; and (3) surgical correction of only the identified problem layers using physics-derived momentum. While each component builds on known techniques, their integration into a working Scan → Localize → Diagnose → Treat → Verify pipeline is, to our knowledge, new. On ResNet-18/CIFAR-10 (single seed), the pipeline identifies 3 of 7 layer groups as error sources. Surgical correction fixes 62 errors; the best layer-selection variant (iKFAD-guided) achieves net +22 improvement with 82% compute savings versus full retraining. Layer-level correction consistently outperforms per-parameter selection. Most notably, gradient attribution on misclassified images identifies the same three problem layers on both SGD and Adam models (100% overlap)—despite fundamentally different optimization trajectories—suggesting the diagnostic measures architectural bottlenecks rather than optimizer artifacts. As a secondary finding, the critical damping condition yields a zero-parameter momentum schedule μ(t)=1−2α(t)μ(t)=1-2 α(t) that delivers 1.9×1.9× faster convergence to 90% accuracy. A hybrid schedule—physics momentum for fast early convergence, then constant momentum for the final refinement—reaches 95% accuracy fastest among five methods tested. We discuss how the pipeline connects to knowledge editing, representation engineering, and surgical fine-tuning in large language models, where targeted layer-level intervention could enable efficient repair of specific failure modes. Keywords: neural network diagnostics, surgical fine-tuning, layer attribution, gradient norms on errors, cross-optimizer invariance, damped harmonic oscillator 1 Introduction When a neural network confuses cats with dogs, standard tools—confusion matrices, per-class accuracy—tell you that the confusion exists. They do not tell you which layers are responsible for the confusion, what features those layers are confusing (furry texture? face shape? silhouette?), or how to fix it without retraining the entire model. This diagnostic gap is increasingly costly as models scale: retraining a large language model to address a specific failure mode can cost as much as the original training. We present a diagnostic pipeline that answers all three questions. The pipeline integrates three existing ideas into a new working system: 1. Damping-regime classification (background from Qian [12]). The well-known equivalence between SGD with momentum and a damped harmonic oscillator lets us classify each training epoch as underdamped, critically damped, or overdamped. The oscillator model itself is not our contribution; its use as a diagnostic classifier for training dynamics is. 2. Error-specific gradient attribution (new application). Prior work uses gradient norms across the full dataset for layer importance weighting [6]. We compute gradient norms exclusively on misclassified images—asking “which layers are most confused on the images the model gets wrong?” This produces a binary layer-level diagnostic (problem/healthy) rather than a continuous weighting, and is, to our knowledge, a new application. 3. Surgical correction with physics-derived momentum (new integration). Surgical fine-tuning as a concept was established by Lee et al. [6]. Using physics-derived momentum (μ=1−2αμ=1-2 α) for the correction, guided by error-specific gradient attribution, is new. The pipeline yields four main findings: 1. The diagnostic pipeline works. On ResNet-18/CIFAR-10, gradient attribution on 452 misclassified test images identifies 3 of 7 layer groups as error sources. Surgical correction of these layers fixes 62 errors; the best variant achieves net +22 improvement with 82% compute savings versus full retraining (Section 5). 2. Cross-optimizer invariance. Gradient attribution on misclassified images identifies the same three problem layers (conv1, layer2, layer3) on both SGD-trained and Adam-trained models—100% overlap despite fundamentally different optimization trajectories. This suggests the diagnostic measures an architectural property (which layers lack capacity for the hardest examples), not an optimizer artifact (Section 6). This is the finding most relevant to scaling: LLMs are trained with Adam variants, so an optimizer-agnostic diagnostic is essential. 3. Layer-level beats parameter-level. Comparing against the per-parameter adaptive friction framework of Yang et al. [20] (iKFAD), we find that layer-group correction outperforms per-parameter correction: both layer-level methods fix 62 errors, while per-parameter methods fix only 42–46 and produce negative net improvement (Section 6). 4. A zero-parameter momentum schedule. As a byproduct of the oscillator model, the critical damping condition yields μ(t)=1−2α(t)μ(t)=1-2 α(t)—a momentum schedule with zero free parameters that delivers 1.9×1.9× faster convergence to 90% accuracy (Section 4). The formula is not our theoretical contribution (it is a direct consequence of Qian [12]); the empirical validation is. 2 Related Work ODE models of momentum. Qian [12] showed that gradient descent with momentum, in the continuous-time limit near a local minimum, is equivalent to a set of coupled damped harmonic oscillators, and that optimal convergence corresponds to critical damping of each eigencomponent. Su et al. [17] derived an ODE for Nesterov’s accelerated gradient method. Shi et al. [13] introduced high-resolution ODEs distinguishing Nesterov’s method from heavy-ball momentum. Our work builds on Qian’s model: we derive the critical damping condition as a practical schedule and, more importantly, use damping-regime analysis as the foundation for a diagnostic pipeline. Learning rate–momentum coupling. Smith [16, 14] empirically discovered that coupling momentum inversely to learning rate improves convergence (the 1cycle policy). Difference from our work: 1cycle approximates the critical damping curve with a piecewise-linear schedule but does not derive the relationship from physics and does not use the framework for diagnostics. Physics-informed optimization. Yang et al. [20] independently derive a momentum–learning-rate coupling from Hamiltonian mechanics, obtaining per-parameter adaptive damping (iKFAD, Cubic Damping, CADAM). Their framework is more general: per-parameter friction, cubic damping terms, bridges the Adam–mSGD gap on Transformers. Difference: their work is an optimizer; ours is a diagnostic. We compare directly in Section 6 and find that layer-level correction (ours and iKFAD-guided) outperforms their per-parameter correction. Adaptive methods. Adam [3] and AdamW [7] adapt the effective learning rate per parameter using gradient statistics. Polyak [11] introduced the heavy-ball method; Nesterov [10] proposed a lookahead variant; Sutskever et al. [18] established μ=0.9μ=0.9 as a practical default. Cutkosky et al. [1] show that momentum’s marginal value diminishes for small learning rates—consistent with our analysis where μc→1 _c→ 1 as α→0α→ 0. Knowledge editing and model repair. Meng et al. [8, 9] developed ROME and MEMIT for locating and editing factual associations in GPT models by modifying MLP weights in specific layers. Difference: they edit specific facts via causal tracing; we diagnose general error patterns via gradient norms on misclassified inputs. Zou et al. [22] proposed Representation Engineering, which reads and controls high-level concepts via population-level representations. Difference: they identify directions in activation space; we identify layers where errors concentrate. Lee et al. [6] showed that surgical fine-tuning of a subset of layers can match or outperform full fine-tuning for distribution shifts. Difference: they showed that surgical fine-tuning works; we provide a principled method for which layers to target. Recent work on hierarchical alignment [21] decomposes objectives across functionally specialized layer blocks in LLMs. 3 Background: The Oscillator Model This section reviews known results [12, 17, 13] and derives the explicit schedule used throughout this paper. The oscillator model itself is not our contribution; we include the derivation to set notation. 3.1 SGD with Momentum as a Damped Oscillator SGD with momentum updates parameters θ via: vt+1 v_t+1 =μvt−α∇ℒ(θt), =μ\,v_t-α\, ( _t), (1) θt+1 _t+1 =θt+vt+1, = _t+v_t+1, (2) where vtv_t is the velocity buffer, α is the learning rate, μ is the momentum coefficient, and ∇ℒ is the loss gradient. Eliminating vtv_t by substituting Eq. 1 into Eq. 2 and taking the continuous-time limit, we obtain the equation of a forced damped harmonic oscillator [12]: x¨+γx˙+ω2x=F(t), x+γ\, x+ω^2\,x=F(t), (3) with the correspondence shown in Table 1. Table 1: Correspondence between the damped harmonic oscillator and SGD with momentum [12]. Oscillator SGD with Momentum Position x Parameters θ Velocity x˙ x Velocity buffer v Damping coefficient γ 1−μ1-μ Natural frequency ω ∝α α External force F(t)F(t) −∇ℒ(θt)- ( _t) The second-order recurrence obtained by eliminating vtv_t from Eqs. 1–2 is: θt+1−(1+μ)θt+μθt−1=−α∇ℒ(θt). _t+1-(1+μ)\, _t+μ\, _t-1=-α\, ( _t). (4) Dividing by Δt2 t^2 and identifying γ=(1−μ)/Δtγ=(1-μ)/ t and ω2=α/Δt2ω^2=α/ t^2, we recover Eq. 3 with γ=1−μγ=1-μ and ω=αω= α (setting Δt=1 t=1). Figure 1: Left: cosine annealing learning rate schedule. Right: momentum trajectories for the three methods. The gray dashed line shows the unclamped critical damping curve μc=1−2α _c=1-2 α. Shaded regions indicate underdamped (red, above curve) and overdamped (blue, below curve) zones. The physics method (green) tracks the critical curve, clamped to [0.5,0.99][0.5,0.99]. 3.2 Three Damping Regimes The behavior of a damped oscillator depends on the discriminant Δ=γ2−4ω2 =γ^2-4ω^2: • Underdamped (γ<2ωγ<2ω, Δ<0 <0): the system oscillates around equilibrium before converging. In optimization, this manifests as fluctuating loss and accuracy curves. • Overdamped (γ>2ωγ>2ω, Δ>0 >0): the system approaches equilibrium without oscillation but unnecessarily slowly. • Critically damped (γ=2ωγ=2ω, Δ=0 =0): the unique regime that reaches equilibrium in minimum time without oscillation. 3.3 The Critical Damping Schedule Setting γ=2ωγ=2ω and substituting γ=1−μγ=1-μ and ω=αω= α: 1−μ=2α⟹μ(t)=1−2α(t).1-μ=2 α μ(t)=1-2 α(t). (5) Given any learning rate schedule α(t)α(t), Eq. 5 determines a momentum trajectory that targets critical damping throughout training. We call this β-scheduling. Caveat: curvature dependence. The natural frequency is properly ω=λ⋅αω= λ·α where λ is the local curvature (Hessian eigenvalue). Our global formula assumes unit curvature (λ=1λ=1). Each parameter’s “true” critical momentum is μi=1−2λiα _i=1-2 _iα; the per-parameter generalization connects to Yang et al. [20]. Despite this simplification, the global formula produces the qualitative predictions verified in Section 4. 3.4 Qualitative Predictions The critical damping condition makes three testable predictions: 1. Fastest early convergence. Critically damped systems have the shortest settling time. 2. No oscillation. Training curves should improve monotonically during the high-LR phase. 3. Competitive final accuracy. The oscillator model captures macroscopic dynamics but not the fine structure of the loss landscape. Final accuracy should be comparable but not necessarily superior. 3.5 Retroactive Explanations of Empirical Heuristics Equation 5 retroactively explains several established practices: • Why μ=0.9μ=0.9 is a common default. At α=0.01α=0.01: μc=1−20.01=0.8 _c=1-2 0.01=0.8. The value 0.9 is close—slightly underdamped but workable. • Why 1cycle couples momentum inversely to LR. Higher α requires lower μ for critical damping. The 1cycle policy [14] approximates this curve. • Why learning rate warmup helps. Starting with small α gives μc≈1 _c≈ 1 (near-zero damping), allowing velocity to build smoothly. 4 Experiment 1: Validating the Physics Before using the oscillator model for diagnostics, we verify that its qualitative predictions hold empirically. The headline result is convergence speed (1.9×1.9× to 90%), not final accuracy (which is within seed-to-seed variance). 4.1 Setup Architecture. ResNet-18 [2] (11.2M parameters), modified for CIFAR-10 [5]: 3×33× 3 initial convolution with stride 1 and padding 1 (replacing the standard 7×77× 7), max-pooling replaced with identity, final FC layer mapping 512→10512→ 10. Training. 200 epochs, batch size 128, weight decay 5×10−45× 10^-4, seed 42, cosine annealing from αmax=0.1 _ =0.1 to αmin=10−4 _ =10^-4. Data augmentation: random crop (32, padding 4) and random horizontal flip. Hardware. NVIDIA Tesla P100 (16 GB), PyTorch 2.4.1, CUDA 12.1. Three conditions. All conditions use the same cosine LR schedule; only the momentum schedule differs: Table 2: Experimental conditions. All share the same cosine learning rate schedule. Condition Momentum Schedule Free Params Baseline μ=0.9μ=0.9 (constant) 1 1cycle [14] μ∈[0.85,0.95]μ∈[0.85,0.95], inversely coupled to LR 2 Physics (ours) μ(t)=clamp(1−2α(t), 0.5, 0.99)μ(t)=clamp(1-2 α(t),\;0.5,\;0.99) 0 Clamping. The physics momentum is clamped to [0.5,0.99][0.5,0.99] for numerical stability. At αmax=0.1 _ =0.1, the raw formula gives μ≈0.37μ≈ 0.37, clamped to 0.500.50; at αmin=10−4 _ =10^-4, μ=0.98μ=0.98, within bounds. This means the physics model is not perfectly critically damped during the high-LR phase (approximately epochs 1–30 where α>0.063α>0.063): it is slightly underdamped due to clamping. However, the deviation is much smaller than the baseline’s (Δμ=+0.13 μ=+0.13 vs. +0.53+0.53 at epoch 1). See Figure 1. 4.2 Results: Final Accuracy Table 3: Final accuracy (single seed). The 0.52 p spread is within typical seed-to-seed variance (± 0.2–0.3 p) and should not be interpreted as a significant difference. Method Best Test Accuracy Training Time (s) Baseline (μ=0.9μ=0.9) 95.54% 5,127 1cycle (Smith) 95.29% 5,100 Physics (μ=1−2αμ=1-2 α) 95.02% 5,128 All three methods achieve comparable accuracy within 0.52 percentage points. Training time is identical—the physics schedule has no computational overhead. Figure 2: Test accuracy during training. The physics method (green) converges fastest to intermediate thresholds, reaching 90% at epoch 52 versus epoch 100 (baseline) and epoch 144 (1cycle). All methods converge to comparable final accuracy. 4.3 Results: Convergence Speed Table 4: Epochs to reach accuracy thresholds. The physics method shows 1.9×1.9× speedup at 90%. These epoch differences (52 vs. 100 vs. 144) are large enough to be robust to seed variation. Method 80% 85% 90% 92% 95% Baseline 13 26 100 139 172 1cycle 10 14 144 161 187 Physics 8 14 52 101 190 The physics method reaches 90% accuracy at epoch 52—a 1.9×1.9× speedup over baseline (epoch 100) and 2.8×2.8× over 1cycle (epoch 144). This is consistent with the oscillator model’s prediction that critical damping produces the shortest settling time. Table 5: Early training accuracy at selected epochs. Method Epoch 1 Epoch 10 Epoch 50 Baseline 37.27% 77.43% 86.80% 1cycle 24.70% 80.55% 83.15% Physics 47.88% 82.68% 89.16% 4.4 Prediction Verification All three qualitative predictions are confirmed: 1. Fastest early convergence—confirmed. Physics reaches 90% in approximately half the epochs of baseline. 2. No oscillation—confirmed. Physics exhibits monotonically improving accuracy during epochs 1–60. Baseline shows non-monotonic fluctuations in the same range. 3. Competitive final accuracy—confirmed. The 0.52% gap is within seed-to-seed variance. 4.5 Damping Regime Analysis Figure 3: Damping regime classification across 200 epochs. Red = underdamped (μ>μc+0.05μ> _c+0.05), green = critically damped (|μ−μc|≤0.05|μ- _c|≤ 0.05), blue = overdamped (μ<μc−0.05μ< _c-0.05). The baseline is underdamped for 85% of training; physics maintains near-critical damping throughout. For each epoch, we classify the regime by comparing the actual momentum μ to the critical value μc=1−2α _c=1-2 α, using a tolerance of |μ−μc|≤0.05|μ- _c|≤ 0.05 for the critical band. Table 6: Damping regime classification across 200 training epochs. Method Underdamped Critical Overdamped Baseline (μ=0.9μ=0.9) 170 (85%) 21 (10.5%) 9 (4.5%) 1cycle (Smith) 138 (69%) 48 (24%) 14 (7%) Physics 0 (0%) 200 (100%) 0 (0%) The baseline operates in the underdamped regime for 85% of training—the standard configuration (μ=0.9μ=0.9) spends the vast majority of training oscillating rather than converging optimally. Physics maintains critical or near-critical damping for all 200 epochs; during the clamped phase, |Δμ|≤0.13| μ|≤ 0.13—an order of magnitude closer to critical than the baseline’s Δμ=+0.53 μ=+0.53 at epoch 1. Why this matters for diagnostics. The damping regime analysis is not just a training diagnostic—it is the foundation for the error-localization pipeline in Section 5. The fact that the baseline is heavily underdamped suggests that its learned representations may be suboptimal in specific layers, which we can now identify and correct. 5 Experiment 2: The Diagnostic Pipeline This is the central experiment. We demonstrate a complete Scan → Localize → Diagnose → Treat → Verify pipeline that identifies which layers cause specific errors, explains why, and surgically corrects them. The analogy is medical: from “the patient is sick, administer broad-spectrum antibiotics” to “MRI shows a lesion in region X, operate on X, verify the outcome.” 5.1 Hypotheses H1: Error-specific gradient attribution localizes problem layers. On misclassified test images, gradient norms should concentrate in specific layers rather than being uniformly distributed, producing a sparse layer-level diagnostic. Note: gradient attribution is a well-established technique; what is new here is computing it exclusively on misclassified images to identify “sick” layers rather than generally important ones. H2: Targeted correction is more efficient than full retraining. Retraining only the identified problem layers should fix errors with ≥ 80% compute savings. H3: Different damping trajectories produce different errors. Models trained with different momentum schedules should make qualitatively different errors. 5.2 Step 1 — Scan: Cross-Model Error Analysis We compute per-image predictions for all three models on the 10,000-image CIFAR-10 test set: Table 7: Cross-model error analysis on CIFAR-10 test set. Category Count Common errors (all 3 methods wrong) 215 Only baseline wrong 104 Only physics wrong 138 Only 1cycle wrong 96 Physics correct, baseline wrong 172 The 172 images correctly classified by the physics model but not by the baseline provide direct evidence that optimization trajectory affects learned representations (H3 confirmed). These are not random differences: they cluster around specific feature confusions, as shown below. 5.3 Step 2 — Localize: Gradient Attribution on Misclassified Images We compute per-layer gradient norms on the baseline model’s 452 misclassified test images. Critically, we restrict the computation to errors only—this asks “which layers are most confused on the specific images the model gets wrong?” rather than “which layers are generally most important?” Table 8: Per-layer gradient norms on misclassified images (baseline model). Layers above the median are flagged as problem layers. Layer Function Grad Norm Status layer3 Mid-level features (shapes, parts) 10.56 Problem conv1 Low-level features (textures, edges) 9.14 Problem layer2 Early-mid features (patterns) 8.58 Problem layer1 Very early features 7.43 Normal layer4 High-level features 7.38 Normal bn1 Batch normalization 4.99 Normal fc Final classifier 0.86 Normal Result (H1 confirmed). Gradient norms concentrate in three layers (layer3, conv1, layer2), with a 12.3×12.3× ratio between the highest (layer3: 10.56) and lowest (fc: 0.86) layers, and a 1.4×1.4× ratio between the third-highest problem layer and the fourth-highest normal layer. The distribution is sparse, not uniform—the pipeline produces a clear cut between “sick” and “healthy” layers. 5.4 Step 3 — Diagnose: Error Taxonomy The most frequent confusion pairs cluster around texture–shape confusions in the problem layers: Table 9: Most frequent confusion pairs in the baseline model with layer attribution. True Predicted Attributed Layer Root Cause cat dog layer3 Face shape, furry texture dog cat layer3 Reverse confusion cat bird layer2 Compact silhouette deer dog conv1 Brown texture → “dog” horse bird conv1 Green background dominates ship airplane layer2 Horizontal shape frog dog layer3 Color confusion Each error has a causal explanation: conv1 confuses based on texture (brown → dog, green → bird); layer2 confuses based on shape (compact silhouette, horizontal shape); layer3 confuses based on mid-level features (face shape, fur, color). The diagnostic pipeline does not just identify that errors exist—it explains why each error occurs and which layer is responsible. 5.5 Step 4 — Treat: Surgical Correction Protocol. We freeze all 120 parameter tensors in the baseline model, then unfreeze only the three problem layers (layer3, conv1, layer2; 35 tensors). We retrain for 30 epochs with physics momentum at αmax=0.01 _ =0.01 (10× lower than original training). Table 10: Surgical correction results: retrain 3 of 7 layer groups for 30 epochs with physics momentum. Metric Before After Accuracy 95.48% 95.54% Total errors 452 446 Errors fixed — 55 New errors — 49 Net improvement — +6 Compute (epochs × layers) — 30 × 3 layers vs. full retraining — 200 × all layers Compute savings — ∼ 82% Result (H2 confirmed). Surgical correction fixed 55 errors by retraining only 29% of parameters (35/120 tensors) for 15% of the original training duration—82% compute savings versus full retraining. The net improvement (+6) in this initial experiment is modest; a stronger variant is tested in Section 6. 5.6 Step 5 — Verify: Analysis of Side Effects The 49 new errors after correction are expected: retraining at the layer-group level affects many parameters simultaneously, and some previously correct features are perturbed. This can be mitigated through: (1) per-residual-block correction instead of per-layer-group, (2) elastic weight consolidation [4] to preserve correct features, and (3) iterative diagnose–correct–verify cycles. Section 6 shows that choosing layers more carefully (via iKFAD-guided selection) reduces new errors from 55 to 40, achieving net +22. 6 Experiment 3: Cross-Optimizer Invariance and Diagnostic Comparison This experiment tests the most important question for scaling: does the diagnostic generalize across optimizers? If gradient attribution on misclassified images identifies the same problem layers regardless of how the model was trained, then the pipeline can be applied to any model—including LLMs trained with Adam variants. 6.1 Setup We train five models under identical conditions (ResNet-18, CIFAR-10, 200 epochs, cosine LR, seed 42): 1. Baseline: SGD + constant μ=0.9μ=0.9. 2. Physics (ours): SGD + μ(t)=1−2α(t)μ(t)=1-2 α(t). 3. iKFAD [20]: per-parameter adaptive friction ξi _i tracking kinetic energy. 4. CD [20]: cubically damped SGD (c=0.1c=0.1). 5. Adam [3]: lr=10−3lr=10^-3, cosine decay to 10−510^-5, weight decay 5×10−45× 10^-4. We compare two diagnostic methods and five correction strategies: • Method A (ours): gradient attribution on misclassified images → layer-group surgical correction with physics momentum. • Method B (Yang et al.): per-parameter friction ξi _i from iKFAD training → rank layers by accumulated friction → layer-group surgical correction. • Methods C, D: per-parameter correction using top 30% and 15% highest-ξ parameters. • Method E: cross-optimizer transfer—diagnose the Adam model with our gradient attribution, then correct with physics momentum. All corrections retrain for 30 epochs with μ=1−2αμ=1-2 α. 6.2 Results: Training Accuracy Table 11: Phase 1 accuracy: five training methods on ResNet-18/CIFAR-10 (200 epochs, seed 42). Method Best Test Accuracy Errors CD—Cubic Damping (Yang et al.) 95.62% 453 Baseline (μ=0.9μ=0.9) 95.51% 453 iKFAD (Yang et al.) 95.39% 468 Physics (μ=1−2αμ=1-2 α) 95.12% 496 Adam (lr=10−3lr=10^-3) 93.68% 647 All four SGD-momentum variants achieve comparable accuracy within a 0.50 p range (95.12–95.62%). Adam trails by ∼1.8 1.8 p, consistent with the known generalization gap between SGD and Adam on image classification [19]. 6.3 Results: Cross-Optimizer Invariance (Key Finding) Table 12: Problem layers identified by each diagnostic method. The SGD–Adam overlap is 100%. Method Problem Layers Basis Ours (gradient attr. on SGD baseline) layer3, conv1, layer2 Grad norm on errors iKFAD (per-parameter friction ξ) bn1, layer3, layer2 Accumulated friction Ours (gradient attr. on Adam) conv1, layer2, layer3 Grad norm on errors SGD–iKFAD overlap layer3, layer2 2/3 shared SGD–Adam overlap conv1, layer2, layer3 3/3 shared (100%) This is the central finding of the paper. Gradient attribution on misclassified images identifies exactly the same three problem layers on the SGD baseline and the Adam model—100% overlap. This is not obvious a priori: SGD and Adam traverse fundamentally different optimization trajectories, use different update rules, and produce models with different error counts (453 vs. 647). Yet the architectural bottlenecks that cause errors are the same. The implication is that gradient attribution on misclassified images measures a property of the architecture—specifically, which layers have insufficient capacity or receive insufficient gradient signal for the hardest examples—rather than an artifact of the optimizer. The diagnostic pipeline is therefore optimizer-agnostic, which is essential for applicability to large models trained with Adam variants. Note the difference from iKFAD: our gradient attribution identifies conv1 (first convolutional layer, learnable feature detectors), while iKFAD identifies bn1 (batch normalization, only scale/shift parameters). Since conv1 contains the features responsible for texture–shape confusions, the gradient-attribution diagnosis is more causally meaningful. 6.4 Results: Surgical Correction Comparison Table 13: Surgical correction comparison: five strategies applied to the baseline model (Methods A–D) and Adam model (Method E). All corrections use physics momentum for 30 epochs. A: Ours B: iKFAD C: ξ-30% D: ξ-15% E: Adam Before (layers) (layers) (params) (params) (cross-opt) Accuracy 95.47% 95.54% 95.69% 95.30% 95.37% 93.51% Errors fixed — 62 62 42 46 146 New errors — 55 40 59 56 148 Net improvement — +7 +22 −-17 −-10 −-2 Diagnostic — Grad attr. Friction ξ Friction ξ Friction ξ Grad attr. Granularity — Layer Layer Param Param Layer Three findings emerge: Layer-level correction outperforms parameter-level. Methods A and B (layer-level) both fix 62 errors, while Methods C and D (parameter-level, top 30% and 15% by friction ξ) fix only 42 and 46 respectively. Moreover, parameter-level methods introduce more new errors than they fix, yielding negative net improvement (−-17 and −-10). Simpler granularity wins: at the ResNet-18 scale, layer-group correction constrains the optimization sufficiently to avoid destabilizing adjacent features. iKFAD layers produce fewer side effects. Methods A and B fix the same number of errors (62), but iKFAD-guided layer selection (Method B) introduces only 40 new errors versus 55 for Method A, achieving net +22 versus +7. The likely explanation: iKFAD identifies bn1 instead of conv1, and correcting batch normalization parameters is lower-risk than modifying the first convolutional layer. Adam cross-optimizer correction (Method E). The diagnostic transfers perfectly (100% layer overlap). Correction fixes 146 errors—the largest number of any method—but also introduces 148 new errors (net −-2). The Adam-trained weights occupy a different region of parameter space; the correction hyperparameters need optimizer-specific tuning. This is a proof of concept: the diagnostic generalizes; the correction requires adaptation. 7 Experiment 4: Hybrid Momentum Scheduling Experiments 1–3 revealed a tension: physics momentum (μ=1−2αμ=1-2 α) converges to 90% accuracy 1.9×1.9× faster than baseline, but baseline (μ=0.9μ=0.9) reaches the final 95% sooner (epoch 175 vs. 183). The physics schedule is near-critically damped, which suppresses the oscillation that helps escape shallow local minima in the late-training landscape. Baseline’s underdamped regime acts as implicit exploration, beneficial when the remaining errors involve subtle feature distinctions. This motivates a hybrid schedule: use physics momentum for fast early convergence, then switch to baseline momentum for the final push. 7.1 Setup We compare five methods on the same setup (ResNet-18, CIFAR-10, 200 epochs, cosine LR, batch 128, seed 42): 1. Baseline: constant μ=0.9μ=0.9. 2. Physics: μ(t)=clamp(1−2α(t), 0.5, 0.99)μ(t)=clamp(1-2 α(t),\;0.5,\;0.99). 3. 1cycle [15]: μ∈[0.85,0.95]μ∈[0.85,0.95], inversely coupled to LR. 4. Hybrid-90: physics momentum until test accuracy ≥90%≥ 90\%, then μ=0.9μ=0.9. 5. Hybrid-ep52: physics momentum until epoch 52 (fixed), then μ=0.9μ=0.9. The two hybrid variants test adaptive (accuracy-triggered) versus fixed (epoch-triggered) switching. Epoch 52 was chosen based on Experiment 1 results where physics reached 90% at epoch 52. 7.2 Results Table 14: Hybrid momentum experiment: convergence milestones (epoch to reach threshold) and best accuracy. Method Best Acc. 80% 85% 90% 92% 95% Switch Baseline (μ=0.9μ=0.9) 95.33% 13 31 96 139 175 — Physics (μ=1−2αμ=1-2 α) 95.11% 8 13 45 88 183 — 1cycle (Smith) 95.36% 17 43 85 137 179 — Hybrid-90 95.30% 7 12 46 136 170 ep47 Hybrid-ep52 95.36% 8 12 109 136 173 ep52 7.3 Analysis Three findings emerge: Physics dominates early training. Physics reaches 80% at epoch 8 (vs. 13 baseline, 17 1cycle), 85% at epoch 13 (vs. 31 baseline, 43 1cycle), and 90% at epoch 45 (vs. 96 baseline, 85 1cycle). Both hybrid methods inherit this advantage. Hybrid-90 reaches 95% fastest. After switching to baseline at epoch 47 (when physics first hit 90%), Hybrid-90 reaches 95% at epoch 170—5 epochs faster than baseline (175) and 13 epochs faster than pure physics (183). The adaptive switch combines the strengths of both regimes: near-critical damping for fast initial convergence, then underdamped exploration for the final refinement. Hybrid-ep52 achieves highest tied accuracy. Hybrid-ep52 matches 1cycle at 95.36%—the best final accuracy—but reaches 90% later (epoch 109 vs. 46 for Hybrid-90). The fixed switch at epoch 52 occurs after the network already reached 90% at epoch 45 but briefly dipped below, causing 7 “wasted” physics epochs. The accuracy-triggered switch in Hybrid-90 avoids this. Interpretation. The results confirm the oscillator-based intuition: physics momentum (near-critical damping) minimizes oscillation for fast descent in the high-curvature early landscape, while baseline momentum (underdamped) provides the perturbation needed to escape shallow basins in the flat late-training landscape. The hybrid schedule exploits both regimes in sequence, yielding the fastest path to 95%. 8 Implications for Large Models The diagnostic pipeline operates on a small model (ResNet-18, 11.2M parameters). Here we discuss why the underlying principles may scale to large language models and propose concrete experiments. 8.1 The Layer-Level Repair Problem Large language models exhibit failure modes—hallucination, bias, harmful outputs—that are currently addressed by retraining the entire model via RLHF or DPO [21]. This introduces an “alignment tax”: fixing one failure mode may degrade performance elsewhere. Surgical fine-tuning [6] and hierarchical alignment [21] demonstrate that targeting specific layers can reduce interference. The key missing ingredient is which layers to target. Our diagnostic provides a principled answer: gradient attribution on failure cases identifies the layers most confused by the errors. The cross-optimizer invariance result (Section 6) suggests this diagnostic measures an architectural property, which is essential for LLMs trained with Adam variants. 8.2 Connection to Existing Methods Our gradient attribution identifies which layers are responsible for specific errors. This is complementary to: • ROME/MEMIT [8, 9]: localize factual associations to MLP layers via causal tracing. Our method localizes error-causing computation via gradient norms. The two could be combined: use gradient attribution to identify candidate layers, then apply ROME-style editing within them. • Representation Engineering [22]: identifies directions in activation space encoding concepts (honesty, harmfulness). Our method identifies layers. RepE’s control vectors could be applied selectively to identified problem layers. • Surgical fine-tuning [6]: demonstrates that partial retraining works. Our contribution is a diagnostic that tells you which part to retrain. 8.3 Proposed Experiments Tier 1: LLM error localization. Take a pre-trained LLM (e.g., Llama-3-8B) and a benchmark with known failure cases (TruthfulQA for hallucination, BBQ for bias). Compute per-layer gradient norms on failure cases. Hypothesis: gradient norms will concentrate in a sparse set of transformer layers, analogous to our ResNet-18 finding. Tier 2: Surgical LoRA. Apply LoRA adapters only to the layers identified in Tier 1 (“surgical LoRA”) versus full-model LoRA. Hypothesis: surgical LoRA will fix a comparable number of errors with fewer side effects, at a fraction of the compute. Tier 3: Cross-architecture invariance. Repeat Tier 1 on models with different architectures (Llama, Mistral, GPT-2-XL). Hypothesis: error-causing layers will cluster at similar relative depths (e.g., 30–50% of model depth), suggesting a universal property of transformer architectures. If confirmed, these results would establish gradient attribution on failure cases as a general-purpose diagnostic for large models—“medical imaging” for neural networks. 9 Discussion 9.1 A Diagnostic Framework, Not an Optimizer The oscillator model of SGD with momentum [12] has been known for 27 years. The formula μ=1−2αμ=1-2 α is a direct consequence. Our contribution is not the formula but what it enables: a diagnostic pipeline that identifies error-causing layers, explains why errors occur, and surgically corrects them with verified outcomes. The cross-optimizer invariance (Table 12) elevates the diagnostic from a training-specific tool to a general-purpose method applicable to any trained model. 9.2 Hyperparameter Reduction A standard hyperparameter search treats α and μ as independent axes in a 2D grid. Eq. 5 collapses this to a 1D curve: given any α, the optimal μ is determined. This halves the search space for momentum-based optimizers—a useful byproduct of the physics, though not the primary contribution. 9.3 Explanations for Model Errors Standard debugging identifies what a model gets wrong. The diagnostic pipeline adds why: the error taxonomy (Table 9) attributes each confusion pair to a specific layer and a specific feature confusion (texture, shape, color). This transforms debugging from a statistical exercise into an explanatory one. 9.4 The Medical Analogy The five-step pipeline mirrors medical diagnostics: 1. Scan: compute damping regime at each epoch—the “vital signs.” 2. Localize: gradient attribution on errors—the “MRI.” 3. Diagnose: map errors to feature confusions—the “pathology report.” 4. Treat: freeze healthy layers, retrain problem layers—the “surgery.” 5. Verify: before/after comparison on specific images—the “follow-up.” 9.5 Layer-Level vs. Parameter-Level Correction Experiment 3 provides evidence that layer-group granularity outperforms per-parameter selection (Table 13). Both layer-level methods fix 62 errors; parameter-level methods fix only 42–46 with negative net improvement. We hypothesize that individual parameter selection introduces too many degrees of freedom, causing the correction to overfit to specific errors while destabilizing adjacent features. Layer-group correction constrains the optimization sufficiently to avoid this. 9.6 Implications for Model Auditing The ability to (1) identify which layers caused which errors, (2) attribute errors to specific feature confusions, and (3) surgically correct them without disturbing other layers has direct implications for model auditing and targeted repair of safety-critical failure modes. The cross-optimizer invariance means this capability extends to models trained with any optimizer. 10 Limitations and Future Work Single seed. All experiments use seed 42. The accuracy differences between methods (0.2–0.5 p) are within typical seed-to-seed variance (± 0.2–0.3 p) and should not be interpreted as significant. What is robust despite this: convergence speed differences (45 vs. 96 epochs to 90%, Experiment 4) are large enough to survive seed variation. The hybrid schedule’s advantage in reaching 95% (epoch 170 vs. 175) is modest and requires multi-seed confirmation. The cross-optimizer invariance (100% layer overlap) is a discrete structural result unlikely to be seed-dependent—if it were noise, we would not expect perfect overlap between two fundamentally different optimizers. Single architecture and dataset. ResNet-18 on CIFAR-10 (32×3232× 32, 10 classes) is a small-scale setting. What is robust despite this: the cross-optimizer invariance suggests the diagnostic measures architecture, not training details. A convincing validation requires: at least one additional CNN (e.g., VGG-16), CIFAR-100, and a Vision Transformer. We estimate approximately 5 seeds × 3 architectures × 2 datasets =30=30 training runs. New errors from correction. Surgical correction introduces new errors alongside fixes (40–55 new errors per 62 fixed at the layer level). What mitigates this: iKFAD-guided layer selection already reduces new errors from 55 to 40 (net +22). Per-residual-block correction, elastic weight consolidation [4], and iterative correction cycles are natural next steps. Global vs. per-layer momentum. Equation 5 applies a single μ to all layers. Each layer has its own effective curvature, suggesting a per-layer critical momentum. This connects to Yang et al. [20]. Cross-optimizer correction tuning. The diagnostic generalizes perfectly (100% layer overlap SGD–Adam), but correction with physics momentum does not yet improve Adam-trained models (Method E, net −-2). Adapting correction hyperparameters to Adam’s parameter scale is needed. Untested on large models. Whether gradient attribution on failure cases produces sparse layer-level signal in transformer LLMs with billions of parameters is an open question (Section 8). What supports optimism: the cross-optimizer invariance suggests the diagnostic measures architecture, not optimizer specifics, and ROME/MEMIT [8] have demonstrated that factual knowledge in LLMs is layer-localized. Planned extensions: (1) multi-seed validation (5 seeds); (2) scale to CIFAR-100 + ImageNet with ResNet-50 and ViT-B/16; (3) Tier 1 LLM experiment: gradient attribution on Llama-3 failure cases from TruthfulQA; (4) optimizer-specific correction hyperparameters for Adam/AdamW. 11 Conclusion We have presented a diagnostic pipeline for neural networks that connects the classical oscillator model of SGD with momentum [12] to practical error localization and surgical repair. The formula μ=1−2αμ=1-2 α is not our theoretical contribution—it has been implicit in the physics since 1999. Our contribution is showing what it enables: 1. Error localization. Gradient attribution on misclassified images identifies a sparse set of error-causing layers. Surgical correction fixes 62 errors; the best variant (iKFAD-guided layer selection) achieves net +22 improvement with 82% compute savings versus full retraining. Layer-level correction outperforms per-parameter selection. 2. Optimizer-agnostic diagnostics. The pipeline identifies the same problem layers on SGD and Adam models (100% overlap), measuring architectural bottlenecks rather than optimizer artifacts. This is the finding most relevant to scaling. 3. Faster convergence. As a secondary benefit, β-scheduling delivers 1.9×1.9× faster convergence to 90% accuracy with zero free parameters. A hybrid schedule (physics until 90%, then μ=0.9μ=0.9) reaches 95% at epoch 170—fastest among five methods tested—validating the oscillator intuition that different training phases benefit from different damping regimes. The most promising direction is scaling the diagnostic to large language models (Section 8). If gradient attribution on failure cases identifies sparse problem layers in LLMs—as the cross-optimizer invariance suggests it should—then surgical correction could offer an efficient, targeted alternative to full-model retraining for fixing specific failure modes. The pipeline opens a research direction: from “the model is wrong, retrain everything” to “the model is wrong in layer X because of feature confusion Y, correct X, verify the outcome.” Reproducibility All code and results are publicly available on Kaggle: Experiments 1–3: https://kaggle.com/code/johnpasichnyk/beta-scheduling-v2 Experiment 4: https://kaggle.com/code/johnpasichnyk/hybrid-momentum-v4 Acknowledgments Experiments were conducted on Kaggle Notebooks (NVIDIA Tesla P100, 16 GB). The author thanks the Kaggle platform for providing free GPU compute. References [1] A. Cutkosky, A. Defazio, and H. Mehta (2024) The marginal value of momentum for small learning rate SGD. In International Conference on Learning Representations, Cited by: §2. [2] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 770–778. Cited by: §4.1. [3] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §2, item 5. [4] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §10, §5.6. [5] A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Technical report, University of Toronto. Cited by: §4.1. [6] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn (2023) Surgical fine-tuning improves adaptation to distribution shifts. In International Conference on Learning Representations, Cited by: item 2, item 3, §2, 3rd item, §8.1. [7] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §2. [8] K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022) Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, Cited by: §10, §2, 1st item. [9] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau (2023) Mass-editing memory in a transformer. In International Conference on Learning Representations, Cited by: §2, 1st item. [10] Y. Nesterov (1983) A method for solving the convex programming problem with convergence rate O(1/k2)O(1/k^2). Proceedings of the USSR Academy of Sciences 269, p. 543–547. Cited by: §2. [11] B. T. Polyak (1964) Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4 (5), p. 1–17. Cited by: §2. [12] N. Qian (1999) On the momentum term in gradient descent learning algorithms. Neural Networks 12 (1), p. 145–151. Cited by: item 1, §1, §11, §2, §3.1, Table 1, §3, §9.1. [13] B. Shi, S. S. Du, M. I. Jordan, and W. J. Su (2022) Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming 195, p. 79–148. Cited by: §2, §3. [14] L. N. Smith and N. Topin (2018) Super-convergence: very fast training of neural networks using large learning rates. arXiv preprint arXiv:1708.07120. Cited by: §2, 2nd item, Table 2. [15] L. N. Smith and N. Topin (2019) Super-convergence: very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Vol. 11006, p. 369–386. Cited by: item 3. [16] L. N. Smith (2017) Cyclical learning rates for training neural networks. arXiv preprint arXiv:1506.01186. Cited by: §2. [17] W. Su, S. Boyd, and E. J. Candès (2016) A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. Journal of Machine Learning Research 17 (153), p. 1–43. Cited by: §2, §3. [18] I. Sutskever, J. Martens, G. Dahl, and G. Hinton (2013) On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, p. 1139–1147. Cited by: §2. [19] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht (2017) The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, p. 4148–4158. Cited by: §6.2. [20] Z. Yang et al. (2026) Adaptive momentum and nonlinear damping for neural network training. arXiv preprint arXiv:2602.00334. Cited by: §1, §10, §2, §3.3, item 3, item 4. [21] Y. Zhang and Q. Dong (2025) Hierarchical alignment: surgical fine-tuning via functional layer specialization in large language models. arXiv preprint arXiv:2510.12044. Cited by: §2, §8.1. [22] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. (2023) Representation engineering: a top-down approach to AI transparency. arXiv preprint arXiv:2310.01405. Cited by: §2, 2nd item. Appendix A Epoch-by-Epoch Damping Scan (Baseline) Table 15: Detailed damping regime classification for the baseline model (μ=0.9μ=0.9, cosine LR schedule with αmax=0.1 _ =0.1). Epoch () α(t) actual _actual _c Regime (Δμ μ) 1 0.10000 0.900 0.368 Underdamped (+0.532) 20 0.09779 0.900 0.375 Underdamped (+0.525) 50 0.08536 0.900 0.416 Underdamped (+0.484) 100 0.05083 0.900 0.549 Underdamped (+0.351) 150 0.01455 0.900 0.759 Underdamped (+0.141) 170 0.00559 0.900 0.850 Critical (+0.050) 180 0.00279 0.900 0.894 Critical (+0.006) 200 0.00011 0.900 0.979 Overdamped (−-0.079) Appendix B Surgical Correction Training Log Table 16: Training log for surgical correction of the baseline model (3 layers, 30 epochs, physics momentum). Epoch α μ Loss Test Acc. 1 0.01000 0.8000 0.0018 95.52% +0.04% 10 0.00796 0.8216 0.0016 95.32% −-0.16% 20 0.00304 0.8898 0.0014 95.48% 0.00% 30 0.00013 0.9775 0.0012 95.54% +0.06% Appendix C Examples of Fixed Errors Table 17: Selected errors fixed by surgical correction, with layer attribution. # Description Before After Attributed Layer 1 Ship on bench airplane ship layer2 (horiz. shape) 2 Horse in greenery bird horse conv1 (green texture) 3 Cat on tree bird cat layer2 (silhouette) 4 Dog close-up cat dog layer3 (face shape) 5 Cat from above dog cat layer3 (furry texture) 6 Frog dog frog layer3 (color) 7 Deer dog deer conv1 (brown texture) 8 Cat standing dog cat layer3 (cat/dog) 9 Cat curled up airplane cat layer2 (gray shape) 10 Cat lying down bird cat layer2 (compact shape)

Instant research discovery