Paper deep dive
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization
Theophilus Amaefuna, Hitesh Vaidya, Anshuman Chhabra, Ankur Mali
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 3:50:09 AM
Summary
The paper introduces a curvature-aware, Minimum Description Length (MDL)-inspired framework for optimizing large language models by allocating capacity and pruning layers. It defines a curvature-adjusted layer gain based on the inverse Hessian and gradient, which serves as a surrogate for reducible risk. This gain is used to formulate two convex optimization programs: one for allocating resources (e.g., expert slots) under a hardware budget, and another for pruning layers while protecting high-gain layers. The framework provides closed-form solutions via bisection and proves transfer stability bounds.
Entities (8)
Relation Signals (7)
Curvature-Weighted Capacity Allocation → uses → Layer Gain
confidence 95% · We introduce a curvature-aware, MDL-inspired framework built around the layer gain...
Curvature-Weighted Capacity Allocation → evaluatedon → Mistral-7B
confidence 92% · Experiments on Mistral-7B and Gemma-7B show clear allocation gains
Curvature-Weighted Capacity Allocation → evaluatedon → Gemma 7B
confidence 92% · Experiments on Mistral-7B and Gemma-7B show clear allocation gains
Theophilus Amaefuna → affiliatedwith → University of South Florida
confidence 90% · Theophilus Amaefuna ... University of South Florida
Layer Gain → derivedfrom → Hessian
confidence 90% · ζ²_k = g_k^T H̃_kk^-1 g_k ... incorporates inverse local curvature
Curvature-Weighted Capacity Allocation → inspiredby → Minimum Description Length
confidence 90% · Our objective functions are motivated by Minimum Description Length (MDL)
LayerIF → comparedto → Layer Gain
confidence 85% · The closest prior measure to ζ²_k is the Layer Influence score (LayerIF)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Layer-wise capacity in large language models is highly non-uniform: some layers contribute disproportionately to loss reduction, whereas others are nearly redundant. Existing layer-scoring methods provide sensitivity estimates but do not give a principled rule for converting those estimates into allocation or pruning decisions under a global hardware budget. We introduce a curvature-aware, MDL-inspired framework built around the layer gain $\zeta_k^2=g_k^\top\widetilde H_{kk}^{-1}g_k$. This quantity equals twice the maximal decrease predicted by the regularized layer-restricted quadratic model and incorporates inverse local curvature; it is therefore a local surrogate for reducible risk, not a universal dominance claim over gradient-norm scores. After normalizing the gains into scores $q_k$, we formulate two convex programs: one allocates expert slots under diminishing returns, and the other assigns layer-wise pruning ratios while protecting high-score layers. Both continuous programs have unique globally optimal solutions characterized by one dual variable and computable in $O(K\log(1/\varepsilon))$ time by bisection. We also prove a quadratic transfer-regret bound: when source and target score vectors differ by at most $\delta$, the target surrogate cost of the transferred decision is within $O(\delta^2)$ of the target optimum. Experiments on Mistral-7B and Gemma-7B show clear allocation gains in some settings and competitive, though mixed, pruning performance. The framework therefore replaces an empirical score-to-decision heuristic with a budget-feasible optimization procedure whose guarantees apply to the stated continuous surrogates. Code is available on github repo - [TKAI-LAB-Mali/Curvature-Weighted-Capacity-Allocation](this https URL)
Tags
Links
- Source: https://arxiv.org/abs/2603.00910v2
- Canonical: https://arxiv.org/abs/2603.00910v2
Trouble viewing inline? Open PDF directly →
Full Text
85,094 characters extracted from source content.
Expand or collapse full text
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization Theophilus Amaefuna * 1 Hitesh Vaidya *1 Anshuman Chhabra 1 Ankur Mali 1 1 Bellini College of Artificial Intelligence, Cybersecurity and Computing University of South Florida, Tampa, Florida, USA Abstract Layer-wise capacity in large language models is highly non-uniform: some layers contribute dis- proportionately to loss reduction, whereas others are nearly redundant. Existing layer-scoring meth- ods provide sensitivity estimates but do not give a principled rule for converting those estimates into allocation or pruning decisions under a global hardware budget. We introduce a curvature-aware, MDL-inspired framework built around the layer gainζ 2 k = g ⊤ k e H −1 k g k . This quantity equals twice the maximal decrease predicted by the regularized layer-restricted quadratic model and incorporates inverse local curvature; it is therefore a local surro- gate for reducible risk, not a universal dominance claim over gradient-norm scores. After normal- izing the gains into scoresq k , we formulate two convex programs: one allocates expert slots under diminishing returns, and the other assigns layer- wise pruning ratios while protecting high-score lay- ers. Both continuous programs have unique glob- ally optimal solutions characterized by one dual variable and computable inO(K log(1/ε))time by bisection. We also prove a quadratic transfer- regret bound: when source and target score vec- tors differ by at mostδ, the target surrogate cost of the transferred decision is withinO(δ 2 )of the target optimum. Experiments on Mistral-7B and Gemma-7B show clear allocation gains in some settings and competitive, though mixed, pruning performance. The framework therefore replaces an empirical score-to-decision heuristic with a budget- feasible optimization procedure whose guarantees apply to the stated continuous surrogates. Code is available on github repo - TKAI-LAB- Mali/Curvature-Weighted-Capacity-Allocation * Equal contribution. 1 INTRODUCTION The representational capacity of a neural network is not uniformly distributed across its layers.Empirical studies of large language models (LLMs) consistently reveal that lay- ers differ substantially in their contribution to the training objective: some layers hold the bulk of the model’s expres- sive power, while others are near-redundant, contributing little to no loss reduction [Zhang et al., 2022, Vitel and Chhabra, 2026]. This non-uniformity has two practical con- sequences that current practice addresses only in isolation. On one hand, capacity bottlenecks: layers where representa- tional power is insufficient, limit model performance even when global parameter counts are large. On the other hand, capacity redundancy: layers where parameters contribute negligibly to the objective, inflates model complexity with- out commensurate benefit. As models scale into the hundreds of billions of parameters, both problems are exacerbated by the underlying physical reality: hardware constraints impose strict limits on mem- ory, compute, and communication bandwidth. The central challenge, therefore, is not simply to make models larger or smaller, but to allocate capacity where it matters and remove it where it does not, within a global resource budget. The missing ingredient: curvature.Existing approaches to layer importance estimation rely primarily on gradient magnitudes, activation statistics, or held-out accuracy drops [Askari et al., 2025, Song et al., 2024, Liu et al., 2021]. These signals share a common limitation: they do not ac- count for the local curvature of the loss landscape. A layer may exhibit a large gradient norm yet reside in a region of high curvature, where the actual achievable loss reduc- tion per unit of capacity is small. Conversely, a layer with a moderate gradient in a flat curvature region may offer substantial reducible risk. Without curvature information, capacity decisions are systematically misallocated. This work.We develop a unified, curvature-aware frame- work for simultaneously allocating and pruning model ca- pacity across layers under a global resource constraint. Our arXiv:2603.00910v2 [cs.IT] 9 Jul 2026 central quantity is the curvature-adjusted layer gain: ζ 2 k = g ⊤ k e H −1 k g k , whereg k is the layer-kgradient and e H k is a positive- definite surrogate for the layer-restricted Hessian block. We show thatζ 2 k /2is the maximal decrease of the regularized layer-restricted quadratic model. Appendix B.3 bounds its approximation error for the nonlinear empirical objective; for layerskandℓ, the induced ordering is certified when- ever 1 2 |ζ 2 k − ζ 2 ℓ | > ε k + ε ℓ , whereε k andε ℓ bound their respective approximation errors. With the normalized scores q k = ζ 2 k / P j ζ 2 j , we then derive two complementary convex programs: •Capacity allocation (Section 2.3): given a global hardware budgetB, distribute additional capacity (e.g. mixture-of-experts slots) preferentially to high- q k layers, with diminishing log-returns penalizing over-allocation. The program admits a closed-form curvature-weighted water-filling solution, computed in O(K log 1/ε) via bisection. •Capacity pruning (Section 3.1): given a global spar- sity targetS, remove parameters aggressively from the low-q k layers while protecting the high-gain layers from degradation. The program is strongly convex with a unique closed-form minimizer, again computed by bisection. We further analyze transfer stability (Appendix C): when curvature scores drift between a source domain and a target domain by∥q (A) − q (B) ∥ 2 ≤ δ, the excess cost of using source-derived allocations on the target task is bounded by O(δ 2 ), with explicit constants given by the condition num- ber of the target program. This justifies warm-starting allo- cation and pruning decisions from source-domain curvature estimates, a practically important property for fine-tuning and domain adaptation. Connections to information theory.Our objective func- tions are motivated by Minimum Description Length (MDL) [Rissanen, 1978, 1989]: model complexity is penalized by description length, while data fit is rewarded by a concave utility reflecting diminishing returns in code-length reduc- tion [Lotfi et al., 2023]. This perspective connects our frame- work to compression-based generalization bounds, where shorter valid descriptions can yield tighter generalization bounds [Wilson, 2025, Schmidhuber, 1997]. The concrete objectives below are MDL-inspired surrogates rather than literal prefix-code lengths. Unless the empirical loss is a negative log-likelihood and the logarithm base and sample- size conversion are specified,ζ 2 k /2is measured in units of average loss, not bits. Contributions. We summarize our contributions as fol- lows. 1. Curvature-adjusted layer gain. We deriveζ 2 k from first principles as twice the maximal second-order ob- jective decrease attributable to layerk, and character- ize the approximation error introduced by Tikhonov regularization of the Hessian block (Lemma 1, Ap- pendix B.3). 2.Curvature-weighted water-filling. We formulate and solve in closed form a convex allocation program that distributes capacity according toq k under diminishing returns and a global hardware budget (Theorem 2). 3.Curvature-protected pruning. We formulate and solve in closed form a strongly convex pruning pro- gram that concentrates sparsity on low-gain layers while meeting a global sparsity target (Theorem 3). 4. Transfer stability. We prove anO(δ 2 )transfer regret bound under score drift, with explicit constants tied to the condition number and score-gradient Lipschitz constant of the target program (Theorem 4). 5.Efficient algorithms. We provideO(K log 1/ε)bisec- tion algorithms for both programs, with explicit bisec- tion brackets and compatibility with standard Hessian approximations (Algorithms 1 and 2). 2 BACKGROUND Minimum Description Length. The Minimum Descrip- tion Length (MDL) principle [Rissanen, 1978] formalizes the tradeoff between model complexity and data fit: the best model is the one that minimizes the total codelength required to describe both the model and the data under the model, Cl(D) =Cl(θ) |z model complexity + Cl(D | θ) |z data fit ,(1) whereCl(θ)is the codelength (in bits) required to describe the parametersθ, andCl(D | θ)is the codelength required to describe the data given the model. Under this principle, a model with lowerCl(D)simultaneously achieves better compression and generalization, as well as lower unneces- sary complexity [Lotfi et al., 2023, Prada et al., 2025]. MDL for capacity allocation.Adding capacity at layerk (e.g. adding mixture-of-experts slots bye k ≥ 0) increases model codelength byCl(θ e k )while reducing data-fit code- length by∆ e k Cl(D | θ)[Blum and Langford, 2003]. Since the base termsCl(θ)andCl(D | θ)are constant with respect to the allocation decision, minimizing the total codelength Eq. (1) over e = (e 1 ,...,e K ) reduces to, min e k ≥0 K X k=1 h Cl(θ e k )− ∆ e k Cl(D | θ) i .(2) The first term penalizes complexity growth; the second rewards data-fit improvement. We instantiate Eq.(2)con- cretely in Section 2.3 by modelingCl(θ e k )∝ αc k e k (linear in resource usage) and∆ e k Cl(D | θ) ∝ γq β k log(1 + e k ) (concave, reflecting diminishing returns in code-length reduction).These proportional models specify a tractable MDL-inspired objective; they are not derived as exact code- length identities. Consequently, the optimization results be- low establish optimality for the stated surrogate, not univer- sal MDL optimality over all possible codes. MDL for pruning. Pruning layerkby sparsity ratio ρ k ∈ [0, 1]reduces the model codelength by−∆ ρ k Cl(θ) but increases data-fit codelength by∆ ρ k Cl(D | θ)[Fran- kle and Carbin, 2019]. Minimizing total codelength over ρ k = (ρ 1 ,...,ρ K ) subject to a global sparsity target S min 0≤ρ k ≤1 K X k=1 h ∆ ρ k Cl(θ) + ∆ ρ k Cl(D | θ) i .(3) We instantiate Eq.(3)in Section 3.1 by modeling ∆ ρ k Cl(θ) = −bn k ρ k (bits saved by removingn k ρ k pa- rameters) and∆ ρ k Cl(D | θ) = η q κ k ρ 2 k (convex degradation penalty, weighted by layer qualityq k ). The quadratic degra- dation term is likewise a modeling surrogate. Its empirical adequacy is separate from the convexity and closed-form analysis of the resulting program. Layer quality via second-order information. Both pro- grams Eq.(2)and Eq.(3)depend on layer quality scores q k that measure how much each layer contributes to re- ducible empirical risk. A natural candidate is the New- ton decrement restricted to layerk: given gradientg k = E k ∇L(θ)and positive-definite Hessian surrogate e H k = E k ∇ 2 L(θ)E ⊤ k + τI, the quantityζ 2 k is twice the maximal second-order decrease inLachievable by updating layerk alone (derived in Section 2.2). We use normalized scores: q k = ζ 2 k P K j=1 ζ 2 j ,throughout, ensuring scale invari- ance with respect to the global curvature magnitude. Layers with largeq k carry more reducible risk and should receive more capacity; layers with smallq k are candidates for prun- ing. Layer Influence scores and their limitations. The clos- est prior measure toζ 2 k is the Layer Influence score (LayerIF) of Askari et al. [2025], which localizes the classical influ- ence function [Koh and Liang, 2017] to individual layers. Given training dataD train = x i m i=1 and validation data D valid = x ν j n j=1 , the global influence score of training sample x i is I(x i ) = − n X j=1 ∇ θ ℓ(x ν j ,θ) ⊤ H(θ) −1 ∇ θ ℓ(x i ,θ),(4) measuring the sensitivity of validation loss to upweighting x i . The LayerIF score restricts Eq.(4)to layerkby replac- ing full-model quantities with their layer-k counterparts: I (k) (x i ) = − n X j=1 ∇ θ (k) ℓ(x ν j ,θ) ⊤ H (k) (θ) −1 ·∇ θ (k) ℓ(x i ,θ). (5) A large|I (k) (x i )|indicates that layerkis sensitive to the training samplex i , and Askari et al. [2025] used the aggre- gated magnitude as a proxy for layer quality to guide expert allocation and pruning. LayerIF combines training and val- idation gradients through an inverse-Hessian operator. Its magnitude therefore reflects gradient alignment and inverse curvature and cannot, by itself, be identified with high cur- vature. The gainζ 2 k /2instead has an exact interpretation as the decrease of the regularized quadratic surrogate. WhenL is an average negative log-likelihood measured in nats, the corresponding local total-codelength scale isnζ 2 k /(2 log 2) bits, up to damping and Taylor-remainder errors. While LayerIF captures data-dependent sensitivity, it has two structural limitations that motivate our approach. First, I (k) depends on individual training samples and must be aggregated heuristically into a layer-level score; in contrast, ζ 2 k is defined directly on the empirical objective and has a closed-form interpretation as reducible risk. Second, and more importantly, LayerIF provides a signal but not an objective: translatingI (k) into concrete expert counts or pruning ratios requires a separate heuristic (a knapsack assignment with first-come-first-served residual allocation [Askari et al., 2025]), with no budget constraint and no optimality guarantee. Our MDL-inspired programs Eq.(2) and Eq.(3)replace this two-stage heuristic with a single convex program that jointly determines the allocation of all layers under a global resource constraint, with closed-form solutions and provable optimality. We study the problem of allocating limited model capacity across layers to maximize locally reducible empirical risk under a global resource constraint. The method consists of two components: (i) a curvature-aware measure of layer- wise reducible risk derived from a second-order expansion of the training objective, and (i) a convex allocation pro- gram that distributes capacity according to these gains under diminishing returns. See Appendix A for related works. 2.1 OBJECTIVE AND NOTATION Let:L(θ) = 1 n P n i=1 φ(f (x i ;θ), y i ),denote the empirical objective, whereφis a per-sample loss andf (· ;θ)is the model. We write the gradient and Hessian atθ ∈R p as: g := ∇ θ L(θ), H := ∇ 2 θ L(θ).PartitionθintoK disjoint layer blocksθ = (θ 1 ,...,θ K ), whereθ k ∈R p k and P k p k = p. For each layerk, letE k ∈ 0, 1 p k ×p be the coordinate-selection matrix and define:g k := E k g ∈ R p k , H k := E k HE ⊤ k ∈R p k ×p k . 2.2 SECOND-ORDER EXPANSION AND LAYER-RESTRICTED DECREASE Second-order Taylor expansion. Suppose∇ 2 Lis locally Lipschitz with constantM > 0in a neighborhood of θ. Taylor’s theorem with integral remainder gives, L(θ + ∆)− L(θ) = g ⊤ ∆ + 1 2 ∆ ⊤ H∆ + R(∆), |R(∆)| ≤ M 6 ∥∆∥ 3 . (6) Layer-restricted quadratic model.Restrict perturbations to layerkby setting∆ = E ⊤ k d k ford k ∈R p k . Since E k E ⊤ k = I p k and the blocks are disjoint, substitution into Eq. (6) yields the layer-restricted quadratic: Q k (d k ) = g ⊤ k d k + 1 2 d ⊤ k H k d k . Because neural network Hessians are generally indefinite, minimizingQ k directly is ill-posed. We introduce the Tikhonov-regularized surrogate: e H k := H k + τI, τ > 0,(7) and analyze e Q k (d k ) = g ⊤ k d k + 1 2 d ⊤ k e H k d k . This regular- ization is standard in second-order deep learning [Martens and Grosse, 2015],Botev et al. [2017]. Quadratic damping is the Lagrangian form associated with a trust-region sub- problem for a suitable multiplier; an arbitrary fixed τ Lemma 1 (Layer-restricted optimum). If e H k ≻ 0, the unique minimizer of e Q k (d k ) over d k ∈R p k is: d ⋆ k = − e H −1 k g k , and the corresponding decrease equals, min d k e Q k (d k ) = − 1 2 g ⊤ k e H −1 k g k . The proof for lemma 1 is shown in Appendix B.1. Curvature-adjusted layer gain. Define: ζ 2 k := g ⊤ k e H −1 k g k ≥ 0.(8) By Lemma 1,ζ 2 k /2is exactly the maximal decrease pre- dicted by the regularized quadratic model e Q k . Ford ⋆ k = − e H −1 k g k and ∆ ⋆ k = E ⊤ k d ⋆ k , Taylor’s theorem gives, L(θ + ∆ ⋆ )− L(θ) =− 1 2 ζ 2 k − τ 2 ∥d ⋆ k ∥ 2 2 + R(∆ ⋆ ), |R(∆ ⋆ )|≤ M 6 ∥d ⋆ k ∥ 3 2 . Thusζ 2 k is a local surrogate for reducible empirical risk rather than a global characterization of the nonlinear loss landscape. Moreover, ∥g k ∥ 2 2 λ max ( e H k ) ≤ ζ 2 k ≤ ∥g k ∥ 2 2 λ min ( e H k ) , so the score incorporates curvature but need not preserve the ranking induced by∥g k ∥ 2 2 . Appendix B.3 gives the asso- ciated approximation and ranking conditions. 2.3 CAPACITY ALLOCATION UNDER DIMINISHING RETURNS Setup. Lete k ≥ 0denote continuous capacity allocated to layerk(e.g., number of active units or effective precision bits), with per-unit resource costc k > 0. We impose the global budget constraint K X k=1 c k e k ≤ B.(9) Define the normalized curvature weights: q k := ζ 2 k P K j=1 ζ 2 j , q k ≥ 0, X k q k = 1(10) This definition assumes P j ζ 2 j > 0 . When zero or near-zero scores are possible, we use the smoothed simplex weights q (ε) k = ζ 2 k + ε q P K j=1 ζ 2 j + Kε q , ε q > 0, which satisfyq (ε) k > 0and provide the positive lower bound needed for uniform strong-convexity and score-Lipschitz constants. Optimization program. We seek an allocation that concen- trates capacity where reducible risk is largest, while penal- izing over-allocation via diminishing log-returns. Setting α k = αc k (so that linear cost scales with resource usage and no layer-dependent free parameter is introduced), we solve: min e k ≥0 K X k=1 h αc k e k − γ q β k φ(e k ) i s.t. K X k=1 c k e k ≤ B, (11) whereα,γ > 0are scalar hyperparameters,β ≥ 0controls gain emphasis (see below) andφ(e k ) = log(1 + e k ). The objective is convex:αc k e k is linear, and−γq β k log(1 + e k ) is convex one k ≥ 0since− log(1 + e)is convex. The feasible set is a convex polytope. Slater’s condition Boyd and Vandenberghe [2004] holds (any strictly feasiblee k > 0 with P k c k e k < Bis a Slater point), so strong duality applies. Closed-form solution. Forming the Lagrangian: L(e,λ) = X k h αc k e k −γq β k log(1+e k ) i +λ X k c k e k −B , and imposing stationarity: ∂L/∂e k = 0 for e k > 0 gives, αc k + λc k − γq β k 1 + e k = 0. Solving and incorporating the non-negativity constraint yields the curvature-weighted water-filling allocation, e k (λ) = max ( γ q β k (α + λ)c k − 1, 0 ) . (12) If the unconstrained optimum (λ = 0) violates the bud- get, the constraint is active and the dual variableλ ⋆ > 0 satisfies P k c k e k (λ ⋆ ) = B. The mapλ 7→ P k c k e k (λ) is continuous and nonincreasing and is strictly decreasing on every interval containing at least one active coordinate. Strict convexity gives a unique primal minimizer; when the active budget equation is nondegenerate, its dual multiplier is unique and can be computed inO(K log(1/ε))time by bisection. A zero-allocation plateau can admit several dual values representing the same primal solution. Role ofβ. The exponentβinterpolates between three regimes: (i)β = 0:q β k = 1uniformly, so curvature in- formation is discarded and the allocation depends only on costsc k ; (i)β = 1: capacity is allocated proportionally to normalized gainsq k , recovering a natural baseline; (i) As βincreases, relative weight is increasingly placed on the largest scores. With fixedγand normalizedq k < 1, how- ever, allq β k vanish asβ →∞, so the allocation may collapse to zero. Literal concentration on the maximizers requires rescalingγwithβor normalizing the powered weights as q β k / P j q β j . In practice,β = 1orβ = 2work well across architectures (Section 3.3). 3 METHOD In this section, we present our MDL framework for expert allocation and pruning as displayed in Figure 1. 3.1 CAPACITY ALLOCATION AND PRUNING UNDER CURVATURE-WEIGHTED TRADEOFFS The curvature-adjusted layer gainsζ 2 k defined in Eq.(8)and derived in Section 2.2, quantify how much empirical risk can be locally reduced by updating layerkalone. We now use these gains to drive two complementary capacity decisions: allocating additional capacity to under-resourced layers, and pruning parameters from layers that contribute little to risk reduction (see Figure 1). In both programs we work with the normalized quality scores introduced in Eq.(10). Nor- malization ensures that the allocation parameters(γ,β)are invariant to the global scale of the curvature signal, and thatq k can be interpreted directly as the relative share of reducible risk attributable to layerk. A layer with largeq k has high potential to reduce empirical risk; accordingly, it should receive more capacity and be protected from pruning. Capacity allocation. Given the setup in Section 2.3 and Eq.(9)lete k denote the effective capacity added at layer k(e.g., LoRA rank or number of mixture-of-experts slots) [Baldi and Vershynin, 2019], settingα k = αc k so that linear penalty scales with hardware usage (and no layer-dependent free parameter is introduced), we model the tradeoff be- tween model complexity and risk reduction via the separa- ble convex program in Eq.(11), whereγ > 0scales benefit strength,β ≥ 0controls curvature emphasis, andφis in- creasing and strictly concave withφ(0) = 0. The concavity ofφmodels diminishing returns: each additional unit of ca- pacity yields progressively smaller reductions in empirical risk. Theorem 2 (Convexity and closed form for allocation). The objective in Eq.(11)is convex. Ifφis strictly concave and q k > 0for allk, the objective is strictly convex and the minimizer is unique. Forφ(e) = log(1 + e), there exists a uniqueλ ⋆ ≥ 0such that the optimal allocation is as shown in Eq. (13). e ⋆ k = max ( γ q β k (α + λ ⋆ )c k − 1, 0 ) .(13) Whenλ ⋆ > 0the budget constraint is active, P k c k e ⋆ k = B; whenλ ⋆ = 0the unconstrained optimum is feasible. On the active setk : e ⋆ k > 0,e ⋆ k is strictly increasing inq k . A detailed proof for this theorem is discussed in Appendix B.2.1 Interpretation.At optimality, the marginal benefit equals the marginal cost at every active layer: γq β k φ ′ (e ⋆ k ) | z marginal benefit = (α + λ ⋆ )c k | z marginal cost . Layers with larger curvature signalq k receive more capacity, while the global multiplierλ ⋆ enforces the budget. The exponentβinterpolates between three regimes:β = 0 produces a uniform allocation that ignores curvature; β = 1allocates proportionally to normalized gains; and asβ → ∞, allocation concentrates on the layer(s) with the largestζ 2 k . In practice,β ∈ 1, 2works well across architectures (Section 3.3). Layer-wise pruning. We now consider the complementary problem of removing parameters from layers that carry lit- tle curvature signal. Letρ k ∈ [0, 1]denote the fraction of parameters pruned at layerk, and letn k denote the total number of parameters in layerk. The number of retained parameters isn k (1− ρ k ), contributingbn k (1− ρ k )bits to model size, where b > 0 is bits per parameter. Pruning a fractionρ k of layerkdegrades data fit. Sinceζ 2 k measures the maximal second-order decrease achievable by updating layerk(Section 2.2), layers with largerq k ∝ ζ 2 k are more sensitive to parameter removal. We model the resulting degradation asηq κ k ψ(ρ k ), whereψis convex with ψ(0) = 0,κ ≥ 0controls curvature emphasis, andη > 0 scales the penalty. This is a design choice grounded in the shared upstream Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 . . . Layer K ζ 2 k low high (a) Non-uniform layer capalcity high gain: allocate more low gain: prune more Curvature Estimation e H k = H k + τI d ⋆ k =− e H −1 k g k Layer Gain (Lem. 1) ζ 2 k = g ⊤ k e H −1 k g k Quality Scores q k = ζ 2 k P j ζ 2 j Allocation (Thm. 1) e ⋆ k = max ( γq β k (α+λ ⋆ )c k −1, 0 ) P c k e k ≤ B, bisection Pruning (Thm. 2) ρ ⋆ k = clip (b+λ ⋆ )n k 2ηq κ k , 0,ρ max P n k ρ k ≥ S, bisection Transfer Stability (Thm. 3) Regret A→B ≤ L x L 2 xq 2σ 2 q (A) − q (B) 2 2 expand compress (b) MDL Capacity Framework g k , e H k Figure 1: (a) Layer-wise curvature scoresζ 2 k vary substantially across the transformer stack. High-gain layers (dark bars) hold dispropor- tionate reducible risk and should receive additional capacity; low-gain layers (light bars) are candidates for aggressive pruning. (b) Our framework computesζ 2 k from per-layer gradientsg k and regularized Hessian blocks e H k , normalizes them to quality scoresq k , and solves two convex programs: a capacity allocation program (Theorem 2) that enriches high-gain layers, and a pruning program (Theorem 3) that concentrates sparsity on low-gain layers. Both programs admit closed-form solutions viaO(K log 1/ε)bisection (Algorithms 1–2). Theorem 4 bounds the cost of transferring source-domain allocations to a target domain. curvature interpretation ofq k ; layers with larger curvature signal attract a higher degradation penalty, so the optimizer naturally protects them. We enforce a minimum global sparsity targetS ≤ P k n k ρ k and solve: min ρ 1 ,...,ρ k K X k=1 h bn k (1− ρ k ) + η q κ k ψ(ρ k ) i s.t. K X k=1 n k ρ k ≥ S, 0≤ ρ k ≤ ρ max . (14) The objective minimizes total model size subject to bounded degradation, with the constraint enforcing that at leastS parameters are pruned globally. Theorem 3 (Strong convexity and closed form for pruning). The objective in Eq.(14)is convex. Ifψis strongly convex andq κ k > 0for allk, the objective is strongly convex and the primal minimizer is unique. A uniform modulus requires q k ≥ q min > 0, or the smoothed scores above, becauseq κ k vanishes at zero whenκ > 0. Feasibility requires0≤ S ≤ ρ max P k n k . Forψ(ρ) = ρ 2 andq κ k > 0for allk, there existsλ ⋆ ≥ 0 such that ρ ⋆ k = clip (b + λ ⋆ )n k 2η q κ k , 0, ρ max .(15) Whenλ ⋆ > 0the sparsity constraint is active, P k n k ρ ⋆ k = S ; whenλ ⋆ = 0, the unconstrained solution already meets or exceeds the target. On interior coordinates (0 < ρ ⋆ k < ρ max ),ρ ⋆ k is strictly decreasing inq k . A detailed proof for this theorem is discussed in Appendix B.2.2. Interpretation. The pruning solution mirrors the alloca- tion solution but in the opposite direction: layers with small q k (low curvature signal, little reducible risk) are pruned ag- gressively, while high-gain layers are protected. The global multiplierλ ⋆ calibrates the pruning depth to meet the spar- sity targetS, playing the same role as the budget multiplier in the allocation program. The quadratic choiceψ(ρ) = ρ 2 is a surrogate selected for analytic transparency. Convexity requires convexψ; monotone degradation is modeled by nondecreasingψ; uniqueness requires strict or strong con- vexity; and a closed-form inverse-gradient representation requiresψ ′ to be invertible on the relevant interval. Together, the two programs provide a unified curvature-aware frame- work: allocation enriches layers where gradient information is underexploited, and pruning removes redundancy where unnecessary. Transfer Stability Under Score DriftThe allocation and pruning programs depend on the normalized score vectorq. A domain change can move this vector fromq (A) toq (B) . Appendix C bounds the target-objective excess cost incurred when the source score vector is used in place of the target score vector. 3.2 ALGORITHMS: CLOSED-FORM SOLUTIONS VIA ONE-DIMENSIONAL DUAL SEARCH The convex programs of Theorems 2 and 3 share a common structure (see Figure 1): strict (or strong) convexity guar- antees a unique primal minimizer, and the KKT conditions reduce each constrained program to a monotone scalar equa- tion in the Lagrange multiplierλ. Both programs therefore admitO(K log(1/ε))algorithms via bisection, with primal variables recovered in closed form at each function evalua- tion. Throughout,q k = ζ 2 k / P j ζ 2 j denotes the normalized curvature score from Eq.(10),α k = αc k as established in Section 2.3, and we use the canonical utility choices φ(e) = log(1 +e)andψ(ρ) = ρ 2 . Since capacity is equiva- lent to an expert in mixture-of-expert, from here on out, we use expert allocation in place of capacity allocation. Expert Allocation. From Theorem 2, the unique mini- mizer of Eq.(11)takes the closed form in Eq.(12)where λ ≥ 0is the dual variable for the budget constraint. The sumλ 7→ P k c k e k (λ)is continuous and strictly decreas- ing: each active terme k (λ) > 0decreases strictly inλ, and oncee k hits zero it stays there. Thereforeλ ⋆ is unique when the constraint is active. To bracket the bisection, note that e k (0) = maxγq β k /(αc k )− 1, 0gives the unconstrained solution. A valid upper bracket is λ max = max ( 0, max k γq β k c k − α !) , because e k (λ) = 0 whenever λ≥ γq β k /c k − α. Layer-wise Pruning. From Theorem 3, the unique min- imizer of Eq.(14)is obtained as follows. The Lagrangian subtracts the dual term for the lower-bound constraint in Eq. (17), The full KKT stationarity condition is −bn k +ηq κ k ψ ′ (ρ k )−λn k −μ k +ν k = 0, λ,μ k ,ν k ≥ 0, whereμ k andν k correspond to−ρ k ≤ 0andρ k −ρ max ≤ 0. On an interior coordinate, 0 < ρ k < ρ max , complementary slackness gives μ k = ν k = 0. For ψ(ρ) = ρ 2 , this yields, ρ int k (λ) = (b + λ)n k 2η q κ k . Strong convexity guarantees a unique primal minimizer, although a plateau of the projected scalar map can make the dual multiplier non-unique. The positive numeratorb +λ > 0(sinceb,λ ≥ 0) ensuresρ int k ≥ 0, consistent with the lower-bound structure of the sparsity constraint. Projection onto[0,ρ max ]enforces the box constraints Eq.(15).The sumλ 7→ P k n k ρ k (λ)is continuous and non-decreasing: the interior solution increases linearly inλ, and clipping to[0,ρ max ]preserves monotonicity since eachρ k is non- decreasing before and after projection. Strict increase holds whenever at least one coordinate remains in the interior (0,ρ max ), guaranteeing a unique λ ⋆ when binding. The unconstrained solution atλ = 0givesρ k (0) = clip(bn k /(2ηq κ k ), 0, ρ max ) . Feasibility under the common layer cap requiresS ≤ ρ max P k n k . A valid upper bisec- tion bracket is λ max = max 0, max k 2ηq κ k ρ max n k − b , which places every coordinate at its upper cap. Remark (Budget flexibility). The budgetBin Algorithm 1 need not equal the base-model FLOPs used in our exper- iments. Any computationally feasible value ofByields a valid allocation that is globally optimal for the continuous surrogate program; the bisection in Algorithm 1 adapts au- tomatically to the chosen constraint. Complexity and practical notes. Each bisection iteration evaluates the primal sum inO(K)time. Both algorithms therefore run inO(K log(1/ε))time to achieve dual gapε. This is substantially cheaper than general-purpose interior- point methods, which requireO(K 3 )per iteration. The curvature scoresq k enter only through the closed-form expressions and need not be recomputed during the dual search, so the algorithms are fully compatible with any ap- proximation of e H −1 k g k (e.g., diagonal Fisher, K-FAC, or ran- domized Nyström sketches). Together, Algorithms 1 and 2 provide practical implementations of the curvature-weighted water-filling framework developed throughout this section. 3.3 EXPERIMENTAL SETUP Models. All experiments are conducted on two publicly available 7B-parameter large language models: Mistral-7B- v0.1 Jiang et al. [2023] and Gemma-7B Team et al. [2024]. Parameter-efficient fine-tuning is performed using LoRA- MoE Gao et al. [2024], which augments each layer with a mixture of low-rank adapters and serves as the capacity expansion mechanism targeted by Algorithm 1. Score estimation.The theoretical score is the normalized Newton-decrement gainq k . For scalability, the experiments instead use the proxy bq k = bs k P j bs j , wherebs k is the chosen nonnegative aggregation of LayerIF scores. Algorithms 1 and 2 are instantiated withbq, not with the theoretical Newton-decrement weightsq. The convex optimization results apply to any fixed positive weight vec- tor; the experiments therefore evaluate the allocation and pruning decision rules under this proxy. Expert allocation.Datasets. We evaluate on five classifi- cation and question-answering benchmarks: CoLA Warstadt et al. [2019], MRPC Dolan and Brockett [2005], Common- senseQA Talmor et al. [2019], ScienceQA Lu et al. [2022], and OpenBookQA Mihaylov et al. [2018]. Curvature scores. Layer-wise influence scores are computed for each (model, dataset) pair following the procedures of Askari et al. [2025] and Kwon et al. [2023]. We consider two variants of the influence score pool: All, in which every computed influence score is used, and +ve, in which only positively influential samples (those with negative influence score values, indicating a beneficial effect on validation loss) are retained. Both variants are evaluated on Mistral-7B; only the +ve variant is used for Gemma-7B. Algorithm 1 MDL-Inspired Expert Allocation Require:Scoresq k , costsc k , budgetB, hyperparam- eters α,γ,β > 0 1: Define closed-form primal: Eq. (12) 2: Setλ min ← 0andλ max ← max0, max k (γq β k /c k − α). 3: if P k c k e k (0)≤ B then 4: λ ⋆ ← 0 budget constraint inactive 5: else 6: Findλ ⋆ ∈(λ min ,λ max )via bisection on P k c k e k (λ) = B 7: end if 8: return e ⋆ k ← e k (λ ⋆ ) for all k Algorithm 2 MDL-Inspired Layer-wise Pruning Require:Scoresq k , sizesn k , sparsity targetS, hyper- parameters b,η,κ > 0 1: Define projected primal: Eq. (15) 2: Set λ min ← 0 and λ max ← max0, max k (2ηq κ k ρ max /n k − b). 3: if P k n k ρ k (0)≥ S then 4: λ ⋆ ← 0 sparsity constraint inactive 5: else 6: Findλ ⋆ ∈(λ min ,λ max )via bisection on P k n k ρ k (λ) = S 7: end if 8: return ρ ⋆ k ← ρ k (λ ⋆ ) for all k Budget and allocation. Following He et al. [2023], the FLOPs of a multi-expert LoRA model are empirically upper- bounded by the FLOPs of the base network. We therefore set the budgetBin Algorithm 1 equal to the FLOPs of a stan- dard single-expert layer in the respective base model, with scaling factorsσ = 0.0276for Mistral-7B andσ = 0.02for Gemma-7B (i.e.,B = σ×base model FLOPs). Per-layer ex- pert counts are computed using Algorithm 1, and fine-tuning follows the protocols of Qing et al. [2024] and Gao et al. [2024] for 5 epochs per (model, dataset) pair. The convex program produces continuous allocationse ∗ k . We convert them to integer expert counts by settingm k = ⌊e ∗ k ⌋ . This rule preserves budget feasibility, P k c k m k ≤ P k c k e ∗ k ≤ B , but it can leave unused budget and is not, in general, the exact solution of the corresponding integer program. Layer-wise Pruning. Datasets. Calibration uses the C4 dataset Raffel et al. [2023]. Zero-shot post-pruning evalu- ation is performed on seven benchmarks: RTE Wang et al. [2019], OpenBookQA Mihaylov et al. [2018], ARC-Easy and ARC-Challenge Clark et al. [2018], HellaSwag Zellers et al. [2019], BoolQ Clark et al. [2019], and WinoGrande Sakaguchi et al. [2021]. Pruning configurations. Layer-wise pruning ratiosρ ⋆ k are computed using Algorithm 2 with influence scores carried over from the expert allocation experiments. The computed ratios are applied under three structural pruning configura- tions — Magnitude Han et al. [2015], SparseGPT Frantar and Alistarh [2023], and Wanda Sun et al. [2023] — using the framework of Lu et al. [2024]. A global sparsity target ofS = 0.5× (total parameters)is enforced for both models, corresponding to 50% sparsity. Evaluation. We report the mean zero-shot accuracy for ex- pert allocation after finetuning and across all seven evalua- tion benchmarks for each pruning configuration and com- pare against the baseline Askari et al. [2025]. Refer to Appendix D for hardware and parameter settings used in the experiments. 3.4 RESULTS AND DISCUSSION CoLAMRPCCommonQOpenBookScienceQAverage Datasets 75 77 79 81 83 85 87 89 91 Accuracies 85.6 83.6 81.2 85.2 66.4 80.4 87.4 82.7 80.4 85.0 79.8 83.1 86.1 82.8 81.4 85.8 80.8 83.4 85.9 84.2 80.2 86.4 83.6 84.1 layerIF (all)mdl (all, our)layerIF (+ve)mdl (+ve, our) Figure 2: Expert allocation accuracy (%) on Mistral-7B-v0.1 (5 epochs) Expert allocation. Figures 2 and 3 report zero-shot ac- curacy after five epochs of fine-tuning on Mistral-7B and Gemma-7B respectively, with per-layer expert counts de- termined by Algorithm 1 (MDL) and the LayerIF heuristic baseline. We note that LayerIF has previously been shown to outperform AlphaLoRA Qing et al. [2024] and MoLA Gao et al. [2024] on these benchmarks; we omit those comparisons here for brevity and focus on the MDL-vs- LayerIF contrast. On Mistral-7B, the MDL allocation out- performs LayerIF on average under both influence score variants: 83.07% vs. 80.41% (All) and 84.06% vs. 83.39% (+ve), corresponding to absolute improvements of 2.66 and 0.67 percentage points respectively. The gains are most pro- nounced on ScienceQA, where MDL improves over LayerIF by 13.4 points (All) and 2.8 points (+ve), suggesting that curvature-weighted allocation is particularly beneficial for knowledge-intensive reasoning tasks where representational capacity is unevenly demanded across layers. On Gemma- 7B, Algorithm 1 produces identical expert counts under the All and +ve variant (hence only +ve results are reported). The MDL allocation yields a marginal improvement to the LayerIF allocation (87.52% vs. 87.46%), confirming that the two methods agree in structure when curvature scores CoLAMRPCCommonQOpenBookScienceQAverage Datasets 75 77 79 81 83 85 87 89 91 93 95 97 99 Accuracies 87.2 84.4 82.6 88.4 94.7 87.5 87.3 85.0 81.6 88.8 94.9 87.5 layerIF (+ve) mdl (+ve, our) Figure 3: Expert allocation accuracy (%) on Gemma-7B, +ve variant only (5 epochs) are relatively uniform, MDL provides a cleaner theoretical justification for the same decision. These results demon- strate that replacing the knapsack heuristic of LayerIF with the convex MDL program of Theorem 2 yields consistent improvements in some cases without additional compute: Both methods share the same influence-score inputs, and Algorithm 1 adds only anO(K log 1/ε)bisection step. See more in Tables 5, 7 and 6 for results on Mistral-7B and Gemma-7B respectively. Layer-wise Pruning. Tables 1 and 2 report mean zero- shot accuracy across seven evaluation benchmarks at 50% global sparsity, under Magnitude, Wanda, and SparseGPT pruning configurations. We impose the box constraint be- fore optimization, usingρ max = 0.51for Mistral-7B and ρ max = 0.55for Gemma-7B. The global 50% target re- mains feasible without fully pruning any targeted parameter block. For Mistral-7B, the combination of a 50% global target andρ max = 0.51leaves little room for heterogeneous layer ratios, which partly explains the near parity between the two decision rules. Fully pruning a targeted transforma- tion can cause severe degradation, although residual connec- tions may still preserve a network-level information path. On Mistral-7B (Table 1), MDL pruning ratios match Lay- Table 1: Mean zero-shot accuracy (%) across 7 benchmarks for pruning on Mistral-7B-v0.1 at 50% sparsity (layer cap 0.51). Rows show the calibration dataset used to compute influence scores. Calibration datasetMagnitudeWandaSparseGPT CoLA56.4758.5959.66 MRPC56.2358.4860.12 CommonsenseQA56.3258.6560.77 OpenBookQA56.2258.4660.05 ScienceQA55.8958.9960.30 Average (MDL)56.2358.6360.18 LayerIF56.4158.6760.18 erIF closely across all three configurations: average accu- racies are within 0.05 points for Wanda and identical for SparseGPT (60.18%), while Magnitude shows a marginal difference of 0.18 points (56.23% MDL vs. 56.41% Lay- erIF). On Gemma-7B (Table 2), MDL outperforms LayerIF under Magnitude (33.34% vs. 32.91%) while LayerIF leads under Wanda (52.3% vs. 49.47%) and SparseGPT (50.79% vs. 49.22%). The pruning parity between MDL and LayerIF is itself a meaningful result: it shows that the principled con- vex program of Theorem 3 recovers the empirically tuned LayerIF ratios without manual calibration, while providing the theoretical guarantees — strong convexity, unique mini- mizer, and budget feasibility, which are lacking in LayerIF. The Wanda and SparseGPT gaps on Gemma-7B suggest that the quadratic degradation modelψ(ρ) = ρ 2 may under- estimate pruning sensitivity in certain architectural regimes; exploring richerψ(e.g.,ψ(ρ) = − log(1− ρ)) is a natu- ral direction for future work. Overall, the allocation results are strongest on Mistral-7B, while Gemma-7B allocation is essentially tied with LayerIF and the pruning results are mixed. The smaller pruning differences are partly explained by the 50% global sparsity target and the tight per-layer caps, which leave limited freedom for heterogeneous prun- ing ratios. Our optimality guarantees apply to the continuous surrogate programs; downstream accuracy can additionally be affected by integer rounding and the specific structural pruning method. Table 2: Mean zero-shot accuracy (%) across 7 benchmarks for pruning on Gemma-7B at 50% sparsity (layer cap 0.55). Rows show the calibration dataset used to compute influence scores. Calibration datasetMagnitudeWandaSparseGPT CoLA33.0348.3649.28 MRPC33.4952.0650.77 CommonsenseQA33.5150.2649.28 OpenBookQA33.5248.0948.11 ScienceQA33.1548.5848.66 Average (MDL)33.3449.4749.22 LayerIF32.9152.3050.79 4 CONCLUSION We presented a curvature-aware framework for layer-wise capacity allocation and pruning in large language models, grounded in the Minimum Description Length principle. The central quantity,ζ 2 k = g ⊤ k e H −1 k g k , measures reducible empirical risk at each layer and drives two convex pro- grams with unique closed-form solutions, each computable inO(K log 1/ε)via bisection. Experiments on Mistral-7B and Gemma-7B show clear allocation gains in one model, an allocation tie in the other, and competitive but mixed pruning performance. Because both methods use the same influence-derived proxy scores, the comparison isolates the decision rule. The formal guarantees are global optimality for the continuous surrogate programs and quadratic stabil- ity with respect to score perturbations under the stated posi- tivity and strong-convexity assumptions. See Appendix G and Appendix H for limitations and directions for future work, respectively. References Hadi Askari, Shivanshu Gupta, Fei Wang, Anshuman Chhabra, and Muhao Chen. LayerIF: Estimating Layer Quality for Large Language Models using Influence Func- tions. In Advances in Neural Information Processing Systems, 2025. Pierre Baldi and Roman Vershynin. The capacity of feed- forward neural networks. Neural Netw., 116(C):288–311, August 2019. ISSN 0893-6080. doi: 10.1016/j.neunet. 2019.04.009. URLhttps://doi.org/10.1016/ j.neunet.2019.04.009. Avrim Blum and John Langford. Pac-mdl bounds. In Bern- hard Schölkopf and Manfred K. Warmuth, editors, Learn- ing Theory and Kernel Machines, pages 344–357, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. ISBN 978-3-540-45167-9. Aleksandar Botev, Hippolyt Ritter, and David Barber. Prac- tical gauss-newton optimisation for deep learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learn- ing, volume 70 of Proceedings of Machine Learning Re- search, pages 557–565, Sydney, Australia, 2017. PMLR. URLhttps://proceedings.mlr.press/v70/ botev17a.html. Stephen Boyd and Lieven Vandenberghe. Convex Opti- mization. Cambridge University Press, Cambridge, UK, 2004. Available athttps://web.stanford.edu/ ~boyd/cvxbook/. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota, June 2019. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ N19-1300. URLhttps://aclanthology.org/ N19-1300/. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URLhttps: //arxiv.org/abs/1803.05457. Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005). Asia Federation of Natural Language Processing, January 2005. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learn- ing Research, 23(120):1–39, 2022. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URLhttps://openreview.net/forum? id=rJl-b3RcF7. Elias Frantar and Dan Alistarh. Sparsegpt: Massive lan- guage models can be accurately pruned in one-shot, 2023. URL https://arxiv.org/abs/2301.00774. Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts, 2024. URLhttps://arxiv.org/ abs/2402.08562. Song Han, Jeff Pool, John Tran, and William J. Dally. Learn- ing both weights and connections for efficient neural net- works. In Proceedings of the 29th International Confer- ence on Neural Information Processing Systems - Volume 1, NIPS’15, page 1135–1143, Cambridge, MA, USA, 2015. MIT Press. B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain sur- geon and general network pruning. In IEEE International Conference on Neural Networks, pages 293–299 vol.1, 1993. doi: 10.1109/ICNN.1993.298572. Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improv- ing computational efficiency of mixture of experts. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14685–14691, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.emnlp-main. 907. URLhttps://aclanthology.org/2023. emnlp-main.907/. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie- Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URLhttps://arxiv. org/abs/2310.06825. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International con- ference on machine learning, pages 1885–1894. PMLR, 2017. Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora- tuned llms and diffusion models.arXiv preprint arXiv:2310.00902, 2023. Yann LeCun, John Denker, and Sara Solla.Op- timal brain damage.In D. Touretzky, editor, AdvancesinNeuralInformationProcessing Systems,volume2.Morgan-Kaufmann,1989. URLhttps://proceedings.neurips. c/paper_files/paper/1989/file/ 6c9882bbac1c7093bd25041881277658-Paper. pdf. Hongyang Liu, Sara Elkerdawy, Nilanjan Ray, and Mostafa Elhoushi. Layer importance estimation with imprinting for neural network quantization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2408–2417, 2021. doi: 10. 1109/CVPRW53098.2021.00273. Sanae Lotfi, Marc Finzi, Yilun Kuang, Tim G. J. Rudner, Micah Goldblum, and Andrew Gordon Wilson. Non- vacuous generalization bounds for large language mod- els. In International Conference on Machine Learning, 2023.URLhttps://api.semanticscholar. org/CorpusID:266573256. Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphaprun- ing: Using heavy-tailed self regularization theory for im- proved layer-wise pruning of large language models. In Thirty-eighth Conference on Neural Information Process- ing Systems, 2024. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. James Martens and Roger Grosse. Optimizing neural net- works with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417. PMLR, 2015. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Pro- cessing, pages 2381–2391, Brussels, Belgium, October- November 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/D18-1260. URLhttps: //aclanthology.org/D18-1260/. Benjamin Prada, Shion Matsumoto, Abdul Malik Zekri, and Ankur Mali.Bridging predictive coding and mdl: A two-part code framework for deep learn- ing.ArXiv, abs/2505.14635, 2025.URLhttps: //api.semanticscholar.org/CorpusID: 278768469. Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. Alphalora: As- signing lora experts based on layer training quality, 2024. URL https://arxiv.org/abs/2410.10054. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URLhttps: //arxiv.org/abs/1910.10683. Jorma Rissanen.Modeling by shortest data descrip- tion*. Autom., 14:465–471, 1978. URLhttps://api. semanticscholar.org/CorpusID:30140639. Jorma Rissanen. Stochastic complexity in statistical in- quiry. In World Scientific Series in Computer Science, 1989.URLhttps://api.semanticscholar. org/CorpusID:9365056. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, August 2021. ISSN 0001-0782. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. Jürgen Schmidhuber. Discovering neural nets with low kol- mogorov complexity and high generalization capability. Neural Networks, 10(5):857–873, 1997. ISSN 0893-6080. doi: https://doi.org/10.1016/S0893-6080(96)00127-X. URLhttps://w.sciencedirect.com/ science/article/pii/S089360809600127X. Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning : from theory to algorithms. Cambridge University Press, Cambridge, 2014. ISBN 1-107-29801- 6. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URLhttps:// openreview.net/forum?id=B1ckMDqlg. Zichen Song, Sitan Huang, Yuxin Wu, and Zhongfeng Kang. Layer importance and hallucination analysis in large language models via enhanced activation variance- sparsity, 2024. URLhttps://arxiv.org/abs/ 2411.10069. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answer- ing challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, edi- tors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapo- lis, Minnesota, June 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421/. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, An- drea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Mu- raru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Milli- can, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Os- car Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shak- eri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahra- mani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024. URL https://arxiv.org/abs/2403.08295. Leslie G. Valiant.A theory of the learnable.Com- mun. ACM, 27:1134–1142, 1984. URLhttps://api. semanticscholar.org/CorpusID:59712. Dmytro Vitel and Anshuman Chhabra. First is Not Re- ally Better Than Last: Evaluating Layer Choice and Ag- gregation Strategies in Language Model Data Influence Estimation. In International Conference on Learning Representations, 2026. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URLhttps://arxiv.org/ abs/1804.07461. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625– 641, 09 2019. ISSN 2307-387X. doi: 10.1162/tacl_a_ 00290. URLhttps://doi.org/10.1162/tacl_ a_00290. Andrew Gordon Wilson. Position: Deep learning is not so mysterious or different. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URLhttps://openreview.net/forum? id=42Au7FoD8F. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really fin- ish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? J. Mach. Learn. Res., 23(1), January 2022. ISSN 1532-4435. Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization Theophilus Amaefuna * 1 Hitesh Vaidya *1 Anshuman Chhabra 1 Ankur Mali 1 1 Bellini College of Artificial Intelligence, Cybersecurity and Computing University of South Florida, Tampa, Florida, USA SUPPLEMENTARY OVERVIEW This supplement follows the same conceptual progression as the main paper. Section A places the proposed curvature- weighted framework in the context of layer-quality estimation, second-order compression, MDL, and adaptive capacity. Section B then supplies the detailed proofs and the surrogate-regularization analysis underlying the main theoretical claims. Section C develops the transfer-stability result and its cross-task diagnostic. The remaining sections provide implementation details, complete experimental results, ablations, proxy validation, statistical analysis, limitations, future work and the LLM-use disclosure. * Equal contribution. * Equal contribution. CONTENTS 1 Introduction1 2 Background2 2.1Objective and Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 2.2Second-Order Expansion and Layer-Restricted Decrease . . . . . . . . . . . . . . . . . . . . . . . . . .3 2.3Capacity Allocation Under Diminishing Returns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 3 Method5 3.1Capacity Allocation and Pruning Under Curvature-Weighted Tradeoffs . . . . . . . . . . . . . . . . . . .5 3.2Algorithms: Closed-Form Solutions via One-Dimensional Dual Search . . . . . . . . . . . . . . . . . . .6 3.3Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 3.4Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 4 Conclusion9 Supplementary Overview13 A Related Work16 A.1 Layer Quality Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.2 Second-Order Methods for Model Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.3 Generalization Bounds and Minimum Description Length . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.4 Mixture-of-Experts and Adaptive Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B Mathematical Foundations and Proofs16 B.1 Layer-Restricted Quadratic Optimum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B.2 Closed-Form Allocation and Pruning Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 B.3 Validity Under Surrogate Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 C Transfer Stability Under Score Drift18 C.1 Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 C.2 Regularity Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 C.3 Transfer-Regret Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 C.4 Interpretation and Boundary Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 C.5 Cross-Task Transfer Diagnostic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 D Experimental Details and Full Results20 D.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 D.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 D.3 Complete Expert-Allocation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 E Ablations and Proxy Validation21 E.1 Mistral-7B Expert-Allocation Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 E.2 Sensitivity to β . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 E.3 Sensitivity to κ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 E.4 Direct and Proxy Estimates of ζ 2 on an 8-Layer MLP . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 F Statistical Significance and Interpretation24 G Limitations25 H Future work25 ILLM Use Disclosure25 A RELATED WORK The main paper combines a layer-wise curvature score with globally constrained allocation and pruning programs. We first position these two ingredients relative to prior work, beginning with layer-quality estimation and then moving to second-order compression, MDL, and adaptive-capacity models. A.1 LAYER QUALITY ESTIMATION Quantifying the per-layer contribution to model performance is central to pruning, expert allocation in mixture-of-experts (MoE) models, and network compression. Askari et al. [2025] proposed LayerIF, which uses influence functions to estimate layer quality and applies it to predict expert counts in Mistral-7B and to determine layer-wise pruning ratios. Song et al. [2024] introduced the Activation Variance-Sparsity Score (AVSS), a combined measure of normalized activation variance and sparsity that quantifies each layer’s contribution to model performance. Liu et al. [2021] developed an accuracy-aware criterion for ranking layer importance, used to guide quantization decisions. A common limitation shared by these approaches is the absence of curvature information: gradient magnitudes and activation statistics do not account for the local geometry of the loss landscape. Our gainζ 2 k addresses this gap directly by incorporating the inverse Hessian block, yielding a measure of reducible risk rather than raw gradient magnitude. A.2 SECOND-ORDER METHODS FOR MODEL COMPRESSION Second-order information has been used for pruning since the seminal Optimal Brain Damage LeCun et al. [1989] and Optimal Brain Surgeon frameworks Hassibi et al. [1993], which identify parameters to remove by computing the Hessian of the training loss. More recent work scales these ideas to modern architectures using diagonal Fisher approximations Martens and Grosse [2015], Kronecker-factored curvature Botev et al. [2017], and randomized sketches. Our work extends this tradition from individual weight pruning to layer-level capacity allocation, and introduces a convex program that jointly optimizes the allocation across all layers under a global budget. A.3 GENERALIZATION BOUNDS AND MINIMUM DESCRIPTION LENGTH Generalization theory asks how well a model trained on finite data performs on unseen examples. Valiant [1984] formalized PAC learning, establishing that generalization depends on hypothesis space complexity and sample size. Shalev-Shwartz and Ben-David [2014] extend this to infinite hypothesis spaces via VC dimension. For LLMs, classical bounds are vacuous due to the enormous number of parameters; Lotfi et al. [2023] address this by introducing compression-based bounds via SubLoRA, a low-dimensional nonlinear parametrization that yields non-vacuous guarantees. Our MDL objective is directly motivated by this line of work: minimizing description length simultaneously controls generalization and penalizes unnecessary model complexity, grounding our convex programs in information-theoretic principles. A.4 MIXTURE-OF-EXPERTS AND ADAPTIVE CAPACITY Sparse MoE models Shazeer et al. [2017], Fedus et al. [2022] increase model capacity without proportional compute cost by routing tokens to a subset of expert layers. Existing routing mechanisms are learned end-to-end and do not explicitly account for the curvature-adjusted utility of adding capacity at a given layer. Our allocation program provides a principled, optimization-based alternative that is complementary to learned routing: given a fixed routing mechanism, it determines how much capacity to assign to each layer based on reducible risk. Together, these lines of work motivate the two-part construction analyzed next: a layer-wise second-order score that estimates locally reducible risk, and a convex resource-allocation rule that converts those scores into globally feasible decisions. B MATHEMATICAL FOUNDATIONS AND PROOFS With the surrounding literature established, we now give the detailed mathematical arguments omitted from the main paper. The presentation follows the logical dependency of the framework: first the layer-restricted quadratic optimum, then the allocation and pruning programs, and finally the effect of surrogate regularization. B.1 LAYER-RESTRICTED QUADRATIC OPTIMUM We begin with Lemma 1, because its regularized Newton step defines the layer gain used by every subsequent optimization program. Recall that e Q k (d k )is the layer-restricted quadratic surrogate for the change in empirical loss produced by a perturbation d k to layer k. Proof.The gradient of e Q k with respect tod k is∇ d k e Q k = g k + e H k d k . Setting this to zero and invoking e H k ≻ 0yields the unique stationary point d ⋆ k =− e H −1 k g k . Substituting back, e Q k (d ⋆ k ) = g ⊤ k (− e H −1 k g k ) + 1 2 (− e H −1 k g k ) ⊤ e H k (− e H −1 k g k ) =−g ⊤ k e H −1 k g k + 1 2 g ⊤ k e H −1 k g k = − 1 2 g ⊤ k e H −1 k g k . B.2 CLOSED-FORM ALLOCATION AND PRUNING SOLUTIONS Having established the layer-wise gain, we next show how the two global resource-allocation programs inherit convexity and reduce to one-dimensional dual searches. B.2.1 Allocation: Convexity and Closed Form We first prove Theorem 2, which characterizes the optimal continuous capacity allocation e ⋆ k under the global budget. Proof. Each termαc k e k is linear. Sinceφis concave,−φis convex, and sinceγq β k ≥ 0 , the term−γq β k φ(e k ) is convex on e k ≥ 0. The sum of convex functions over a convex feasible set is convex. Whenφis strictly concave andq k > 0, each term −γq β k φ(e k ) is strictly convex, giving strict convexity of the full objective. Slater’s condition Boyd and Vandenberghe [2004] holds (any strictly feasible point satisfies the constraint with strict inequality), so strong duality applies. The Lagrangian is L(e,λ) = X k h αc k e k − γq β k φ(e k ) i + λ X k c k e k − B ! ,λ≥ 0.(16) Stationarity at an interior point e k > 0 gives (α + λ)c k − γq β k φ ′ (e ⋆ k ) = 0. For φ(e) = log(1 + e), φ ′ (e) = 1/(1 + e), so (α + λ ⋆ )c k = γq β k 1 + e ⋆ k , which yields Eq.(13)after incorporating non-negativity. Sincee k (λ)is strictly decreasing and continuous inλ, the sum P k c k e k (λ)is strictly decreasing, soλ ⋆ is unique and can be computed inO(K log(1/ε))time by bisection. Strict monotonicity of e ⋆ k in q k follows directly from Eq. (13). B.2.2 Pruning: Strong Convexity and Closed Form The pruning program is complementary to allocation: instead of distributing additional capacity, it assigns layer-wise sparsity while enforcing a global target. We now prove the closed-form characterization in Eq. (15). Proof.The term−bn k ρ k is linear. Ifψis strongly convex,ηq κ k ψ(ρ k )is strongly convex forq k > 0, making the full objective strongly convex. The feasible set[0, 1] K ∩ P k n k ρ k ≥ Sis convex and compact, so a unique minimizer exists. The constraint P k n k ρ k ≥ S is a lower-bound constraint, so the Lagrangian is formed by subtracting the dual term: L(ρ,λ) = P k h bn k (1− ρ k ) + ηq κ k ψ(ρ k ) i − λ P k n k ρ k − S − P K k=1 μ k ρ k + P K k=1 ν k (ρ k − ρ max ), λ≥ 0, μ k ≥ 0, ν k ≥ 0 (17) Stationarity at an interior point ρ k ∈ (0,ρ max ) gives −bn k + η q κ k ψ ′ (ρ ⋆ k )− λ ⋆ n k − μ k + ν k = 0. For ψ(ρ) = ρ 2 , ψ ′ (ρ) = 2ρ, so ρ ⋆ k = (b + λ ⋆ )n k 2η q κ k , and box constraints[0,ρ max ]induce clipping, giving Eq.(15). Sinceλ ⋆ ≥ 0, the numeratorb + λ ⋆ > 0, soρ ⋆ k > 0on the interior. The mappingλ7→ P k n k ρ k (λ)is strictly increasing (largerλincreases everyρ k through the numeratorb + λ), soλ ⋆ is unique when binding and can be found by bisection. Strict decrease of ρ ⋆ k in q k follows directly from Eq. (15). B.3 VALIDITY UNDER SURROGATE REGULARIZATION The preceding proofs establish optimality for the regularized quadratic surrogate. We now connect that surrogate back to the actual empirical objective and state the additional separation condition needed for layer rankings to be certified. The step∆ ⋆ = E ⊤ k d ⋆ k is derived from the surrogate e H k , not fromH k directly. We now show that the resulting ranking of layers by ζ 2 k is consistent with the true objective decrease, and characterize the approximation gap explicitly. Applying Eq. (6) to ∆ ⋆ and using ∆ ⋆⊤ H k ∆ ⋆ = ∆ ⋆⊤ e H k ∆ ⋆ − τ∥∆ ⋆ ∥ 2 gives L(θ + ∆ ⋆ )− L(θ) = − 1 2 ζ 2 k − τ 2 ∥∆ ⋆ ∥ 2 + R(∆ ⋆ ).(18) We bound the bias term as follows. Since d ⋆ k =− e H −1 k g k and∥E ⊤ k d∥ =∥d∥ for any d, ∥∆ ⋆ ∥ 2 = ∥d ⋆ k ∥ 2 = g ⊤ k e H −2 k g k . The spectral inequality e H −2 k ⪯ λ min ( e H k ) −1 e H −1 k (which follows because all eigenvalues of e H −1 k lie in(0,λ min ( e H k ) −1 ]) then yields τ 2 ∥∆ ⋆ ∥ 2 ≤ τ 2λ min ( e H k ) ζ 2 k .(19) Combining Eq. (18) and Eq. (19), L(θ + ∆ ⋆ )− L(θ) ≥ − 1 2 1 + τ λ min ( e H k ) ! ζ 2 k + R(∆ ⋆ ). The factorτ/λ min ( e H k )measures the relative damping term in this bound. WhenH k ⪰ 0, one hasλ min ( e H k )≥ τand the factor is at most one. For an indefinite block,λ min ( e H k ) = λ min (H k ) + τmay be smaller thanτ; positive definiteness requires τ >−λ min (H k ), and the ratio need not be at most one. Small damping error alone does not preserve rankings when scores are nearly tied. Define ε k := τ 2 ∥d ⋆ k ∥ 2 2 + M 6 ∥d ⋆ k ∥ 3 2 . The ordering of layers k and ℓ is certified whenever 1 2 |ζ 2 k − ζ 2 ℓ | > ε k + ε ℓ . Neither batch normalization nor weight decay alone guarantees that an empirical Hessian block is positive semidefinite. This completes the within-task justification of the curvature score. We next study a different source of error: replacing the target-task score vector by a score vector estimated on a related source task. C TRANSFER STABILITY UNDER SCORE DRIFT The allocation and pruning programs depend on a normalized score vector. We quantify the target-objective excess cost incurred when a source score vectorq (A) is substituted for the target score vectorq (B) , while keeping the optimization objective, hyperparameters, and feasible set fixed. C.1 PROBLEM SETUP LetJ B (x;q)denote the MDL program objective on taskB, as a function of the decision variablexand quality inputsq. Herex = e∈R K ≥0 for the allocation program (Section 2.3) andx = ρ∈ [0,ρ max ] K for the pruning program (Section 3.1), with ∆ K :=q ∈R K + : 1 ⊤ q = 1, q ∈ ∆ K . The score vector has K coordinates and affine dimension K− 1; it is distinct from the decision variable x. LetXdenote the corresponding feasible set (budget constraint or sparsity constraint). Since Theorems 2 and 3 guarantee a unique minimizer for each admissible q, define bx B (q) := arg min x∈X J B (x; q). The transfer regret of deploying the source-derived decision bx B (q (A) ) on the target task is Regret A→B := J B bx B (q (A) ); q (B) −J B bx B (q (B) ); q (B) .(20) Both terms are evaluated under the target objective and target scoresq (B) , so Eq.(20)is the excess cost of a misspecified score input. The notationbx B (q (A) )keeps the target objective and feasible set fixed. Usingbx A (q (A) )would additionally introduce objective, hyperparameter, or feasible-set drift and would require separate error terms. C.2 REGULARITY CONDITIONS Let ∆ K (q min ) =q ∈ ∆ K : q k ≥ q min for all k for some q min > 0. We assume: (i) J B (·;q) is σ-strongly convex on the common closed convex feasible setX for every q ∈ ∆ K (q min ); (i) the decision gradient is Lipschitz in the score vector: ∥∇ x J B (x;q)−∇ x J B (x;q ′ )∥ 2 ≤ L xq ∥q− q ′ ∥ 2 for all x∈X and q,q ′ ∈ ∆ K (q min ). For allocation with φ(e) = log(1 + e), one may take L alloc xq ≤ γβ max t∈[q min ,1] t β−1 . For pruning with ψ(ρ) = ρ 2 , L prune xq ≤ 2ηκρ max max t∈[q min ,1] t κ−1 . The positive lower bound onq k is necessary for uniform constants when0 < β < 1or0 < κ < 1, and it also prevents the strong-convexity modulus from vanishing in the pruning program. C.3 TRANSFER-REGRET BOUND With the score domain and regularity conditions fixed, we can state the target-objective excess-cost guarantee. Theorem 4 (Transfer regret under score drift). Under assumptions (i)–(i), 0≤ Regret A→B ≤ L 2 xq 2σ ∥q (A) − q (B) ∥ 2 2 .(21) If|q (A) k − q (B) k |≤ δ k for every k, then Regret A→B ≤ L 2 xq 2σ K X k=1 δ 2 k . Proof. Write x A = bx B (q (A) ) and x B = bx B (q (B) ). Strong convexity ofJ B (·;q (B) ) gives J B (x A ;q (B) )−J B (x B ;q (B) )≤ D ∇ x J B (x A ;q (B) ),x A − x B E − σ 2 ∥x A − x B ∥ 2 2 . Because x A minimizesJ B (·;q (A) ) over the closed convex setX , its variational inequality is D ∇ x J B (x A ;q (A) ),x B − x A E ≥ 0. Equivalently,⟨∇ x J B (x A ;q (A) ),x A − x B ⟩≤ 0. Subtracting this nonpositive term and applying assumption (i) yields Regret A→B ≤ L xq ∥q (A) − q (B) ∥ 2 ∥x A − x B ∥ 2 − σ 2 ∥x A − x B ∥ 2 2 . The right-hand side is maximized overr = ∥x A − x B ∥ 2 ≥ 0atr = L xq ∥q (A) − q (B) ∥ 2 /σ, which gives Eq.(21). Nonnegativity follows from the optimality ofx B under the target score vector. The coordinate-wise statement follows from ∥q (A) − q (B) ∥ 2 2 ≤ P k δ 2 k . C.4 INTERPRETATION AND BOUNDARY VALIDITY The bound is valid for interior and boundary solutions, including active budget, sparsity, and box constraints; no multiplier- sensitivity argument or additional smoothness constant is required. The factorL 2 xq /(2σ)separates sensitivity to score perturbations from the strong-convexity conditioning of the target surrogate program. C.5 CROSS-TASK TRANSFER DIAGNOSTIC We conduct a cross-task diagnostic on Gemma-7B by deriving the expert allocation from CommonsenseQA and deploying that allocation on the remaining tasks. Because the experimental weights are LayerIF-derived proxies, we report the squared proxy-score drift ∥bq (source) − bq (target) ∥ 2 2 , with each proxy vector normalized to sum to one. The accompanying performance quantity is a downstream accuracy difference, not the nonnegative optimization regret in Eq. (20). Table 3: Cross-task proxy-score drift and downstream accuracy difference for a CommonsenseQA-derived allocation. DatasetProxy- score drift Accuracy difference CommonQA (Source)0.00000.00 MRPC0.2447-0.34 CoLA0.62731.82 OpenBookQA0.31271.20 ScienceQA0.66981.44 The negative MRPC entry confirms that the third column is not the theorem’s optimization regret, which is nonnegative by definition. The table therefore provides a qualitative cross-task diagnostic. The direct numerical quantity associated with Theorem 4 is R J =J B (bx B (q (A) );q (B) )−J B (bx B (q (B) );q (B) )≥ 0, with upper bound L 2 xq ∥q (A) − q (B) ∥ 2 2 /(2σ). The theorem controls the surrogate optimization objective, whereas the cross-task experiment additionally reflects training noise, integer expert counts, and downstream evaluation. We therefore keep the two quantities separate and now turn to the implementation details and complete empirical results. D EXPERIMENTAL DETAILS AND FULL RESULTS D.1 HARDWARE Mistral-7B experiments are run on 4×NVIDIA L40S GPUs; Gemma-7B experiments are run on 4×NVIDIA A6000 GPUs. D.2 HYPERPARAMETERS Table 4 summarizes all hyperparameter settings. For the pruning program (Algorithm 2):b = 16bits per retained parameter, η = 2, κ = 1. For the allocation program (Algorithm 1): α = 0.5, γ = 0.9, β = 1. Layer sizes n k are set to the number of parameters in layer k, and per-unit costs c k are set to the FLOPs of a single LoRA expert at layer k. Table 4: Hyperparameter configurations for allocation and pruning. ProgramParameterMistral-7BGemma-7B Allocation α0.50.5 γ0.90.9 budget scaling, σ0.02760.02 Pruning b (bits)1616 η22 κ11 Sparsity target, S50%50% D.3 COMPLETE EXPERT-ALLOCATION RESULTS Tables 5 and 6 provide the dataset-level results summarized in Section 3.4 of the main paper. These complete tables separate the effect of the convex decision rule from the influence-style score estimator shared with LayerIF. Table 5: Expert allocation accuracy (%) on Mistral-7B-v0.1 (5 epochs). Best average per category in bold. All+ve DatasetLayerIFMDLLayerIFMDL CoLA85.6287.4486.1085.90 MRPC83.5982.7282.8484.23 CommonsenseQA81.2480.4381.4080.18 OpenBookQA85.2085.0085.8086.40 ScienceQA66.4179.77 80.8083.59 Average80.4183.0783.3984.06 Table 6: Expert allocation accuracy (%) on Gemma-7B, +ve variant only (5 epochs). Best average in bold. DatasetLayerIFMDL (+ve) CoLA87.1587.34 MRPC84.4184.99 CommonsenseQA82.6481.57 OpenBookQA88.4088.80 ScienceQA94.6994.92 Average87.4687.52 The full results establish the primary comparison. We next examine which parts of the framework drive those outcomes by varying the allocation and pruning hyperparameters and by comparing alternative score-to-decision pipelines. E ABLATIONS AND PROXY VALIDATION This section complements the main experiments in three steps. We first compare the proposed allocation rule with uniform, hand-designed, and heuristic alternatives. We then study sensitivity to the exponentsβandκ. Finally, we compare direct evaluations ofζ 2 k with scalable proxy scores on a smaller network where the regularized curvature quantity can be computed explicitly. E.1 MISTRAL-7B EXPERT-ALLOCATION COMPARISON We compare uniform and non-uniform MoLA allocations [Gao et al., 2024], AlphaLoRA [Qing et al., 2024], a gradient- norm/Fisher score combined with the LayerIF heuristic, the original LayerIF decision rule [Askari et al., 2025], and our convex allocation program using the same scalable influence-style score inputs. All methods are evaluated on Mistral-7B-v0.1 under the common five-epoch protocol. Table 7: Accuracy (%) across expert-allocation strategies (Mistral-7B-v0.1, 5 epochs, All). DatasetUniformNon-uniformAlphaLoraFisherLayerIFMDL (MoLA 5555)MoLA 2468MoLA 8642(grad. norm) Cola85.5286.2985.4387.4485.3385.6287.44 MRPC83.9482.6184.2383.5480.8783.5982.72 CommonsenseQA81.2481.0863.4781.9079.5281.2480.43 OpenbookQA85.4086.6082.2082.0086.2085.2085.00 ScienceQA81.2176.3982.5572.2678.3366.4179.77 Table 7 shows that the proposed method remains competitive with the strongest alternatives while providing an explicit continuous objective and a globally enforced resource budget. The comparison also clarifies that the theoretical contribution concerns the score-to-decision rule rather than a uniformly superior score estimator. E.2 SENSITIVITY TO β We next isolate the gain-emphasis exponentβin Algorithm 1. Using Gemma-7B, we varyβwhile holding the remaining allocation hyperparameters and training protocol fixed. This experiment measures how strongly concentrating capacity on the largest scores affects dataset-level and average accuracy. colamrpccommonqopenbookscienceQ 75 80 85 90 95 100 Accuracy (%) 87.15 84.41 82.64 88.40 94.69 87.34 82.37 82.80 86.60 94.56 86.77 83.30 79.27 87.60 94.38 87.34 84.99 81.57 88.80 94.92 LayerIF (Baseline) MDL (+ve) Beta=1 MDL (+ve) Beta=2 MDL (+ve) Beta=3 Performance Breakdown by Dataset: LayerIF vs MDL (+ve) (a) 123 Beta 86.4 86.6 86.8 87.0 87.2 87.4 Average Accuracy (%) Average Performance vs Beta MDL (+ve) LayerIF (Constant) (b) Figure 4: (a) reveals the detailed accuracy per dataset as Beta goes from1− 3. (b) As shown for Gemma-7B,β = 3gives, on average, a better accuracy across the datasets. E.3 SENSITIVITY TO κ The pruning exponentκcontrols how sharply the degradation penalty distinguishes high- and low-score layers. We varyκ on Gemma-7B while holding the remaining pruning hyperparameters fixed, and report zero-shot accuracy for Magnitude, Wanda, and SparseGPT pruning over the tasks listed in Section 3.3 of the main paper. E.4 DIRECT AND PROXY ESTIMATES OF ζ 2 ON AN 8-LAYER MLP The preceding ablations vary the optimization programs while keeping the score pipeline fixed. We now examine that pipeline directly on an 8-layer multilayer perceptron, whereζ 2 k (τ ) = g ⊤ k e H −1 k g k can be computed and compared with scalable alternatives. The damping parameter τ is central to this comparison: as τI increasingly dominates H k in Eq. (7), e H −1 k ≈ τ −1 I, ζ 2 k (τ )≈ τ −1 ∥g k ∥ 2 2 , colamrpccommonqopenbookscienceQ 0 10 20 30 40 50 60 70 Zero-Shot Accuracy (%) 33.03 33.49 33.51 33.52 33.15 48.36 52.06 50.26 48.09 48.58 49.28 50.77 50.18 48.11 48.66 Kappa = 1 Magnitude Wanda Sparsegpt colamrpccommonqopenbookscienceQ Zero-Shot Accuracy (%) 33.21 32.96 33.44 33.03 33.36 47.37 52.17 50.51 48.93 48.48 46.97 50.99 49.75 48.15 49.18 Kappa = 2 colamrpccommonqopenbookscienceQ 0 10 20 30 40 50 60 70 Zero-Shot Accuracy (%) 32.97 33.34 33.61 33.00 33.25 45.08 54.20 50.11 48.48 48.95 46.38 52.12 50.70 47.89 49.43 Kappa = 3 colamrpccommonqopenbookscienceQ Zero-Shot Accuracy (%) 32.72 33.27 33.63 32.69 33.37 45.20 52.52 49.11 47.84 48.46 45.22 51.51 50.49 48.44 48.29 Kappa = 4 Performance Breakdown by Dataset and Method (Kappa 1-4) (a) 1234 Kappa Value 32.5 35.0 37.5 40.0 42.5 45.0 47.5 50.0 52.5 Average Zero-Shot Accuracy (%) Average Performance vs Kappa Magnitude Wanda Sparsegpt LayerIF Computed Mag (33.91) LayerIF Computed Wanda (52.3) LayerIF Computed Sparsegpt (50.79) (b) Figure 5: (a) Performance breakdown for the three pruning types across varying values ofκ(b) Across the pruning types, an increase inκ doesn’t seem to be helpful when compared to the LayerIF baselines. so the regularized second-order score approaches a rescaled gradient-norm ranking. Tables 8–12 show how the proxy correlations evolve across this regime. The results reveal a damping-dependent transition. At small-to-moderate damping, the LayerIF-style proxy is most strongly Table 8: Tau / damping = 0.001 Proxy/BaselineSpearmanKendall LayerIF_style0.659± 0.2240.571± 0.210 diag_fisher0.484± 0.2040.381± 0.187 grad_norm0.611± 0.1570.452± 0.135 Table 9: Tau / damping = 0.01 Proxy/BaselineSpearmanKendall LayerIF_style0.754± 0.1850.643± 0.210 diag_fisher0.349± 0.2380.310± 0.187 grad_norm0.706± 0.1370.571± 0.117 Table 10: Tau / damping = 0.1 Proxy/BaselineSpearmanKendall LayerIF_style0.881± 0.1680.833± 0.236 diag_fisher0.214± 0.4180.167± 0.379 grad_norm0.690± 0.0850.571± 0.058 Table 11: Tau / damping = 1 Proxy/BaselineSpearmanKendall LayerIF_style0.738± 0.1780.619± 0.221 diag_fisher0.381± 0.1520.333± 0.121 grad_norm0.825± 0.1190.738± 0.121 Table 12: Tau / damping = 2 Proxy/BaselineSpearmanKendall LayerIF_style0.595± 0.1850.476± 0.168 diag_fisher0.444± 0.0110.381± 0.034 grad_norm0.929± 0.0340.833± 0.067 aligned with the direct regularized score, whereas at larger damping the gradient norm becomes the closest proxy, as predicted by e H −1 k ≈ τ −1 I. This diagnostic does not establish proxy equivalence for LLMs, but it clarifies when an influence-style proxy can retain information beyond raw gradient magnitude. Having examined both program hyperparameters and score proxies, we finally assess whether the observed allocation differences are statistically distinguishable under the current evaluation budget. F STATISTICAL SIGNIFICANCE AND INTERPRETATION To assess the reliability of the allocation comparison, we performed pairedt-tests between the proposed MDL-inspired decision rule and LayerIF on the matched model–dataset evaluations reported below. Because the two methods use the same influence-style layer-score inputs, this analysis isolates the difference between their allocation rules rather than the quality of the underlying sensitivity estimator. Table 13: Paired comparisons between LayerIF and the proposed allocation rule. None of the reported differences is significant at the 0.05 level. Algorithm (Mistral)Average (%)p-value LayerIF (+ve)83.390.1906 MDL (+ve)84.06 − Mean diff0.67 − Algorithm (Mistral)Average (%)p-value LayerIF (All)80.410.1917 MDL (All)83.07 − Mean diff2.66 − Algorithm (Gemma)Average (%)p-value LayerIF (All)87.460.4162 MDL (All)87.52 − Mean diff0.07 − The differences are not statistically significant at the 0.05 level, so the experiments support comparability rather than a universal accuracy improvement. This outcome should be interpreted in the context of the design: our method and LayerIF intentionally share the same influence-style layer scores, and the comparison therefore isolates the decision rule. Under the current evaluation budget, replacing the LayerIF knapsack heuristic with the convex MDL-inspired program preserves competitive empirical performance while adding an explicit risk–complexity objective, global budget feasibility, and the transfer-stability guarantee proved above. G LIMITATIONS The quadratic degradation modelψ(ρ) = ρ 2 may underestimate pruning sensitivity in architectures with highly heteroge- neous layer widths, as seen in the Gemma-7B Wanda results. The Hessian surrogate e H k requires a curvature approximation (e.g., diagonal Fisher or K-FAC) whose quality affects the accuracy ofζ 2 k . The experiments instantiate the programs with LayerIF-derived proxy weights rather than directly computed Newton-decrement gains. They therefore evaluate the convex decision rule, but do not directly establish the empirical quality of the proposed curvature estimator. H FUTURE WORK Natural extensions include: (i) richer degradation penaltiesψcalibrated to specific pruning methods; (i) joint optimization of allocation and pruning in a single program; and (i) online updating ofζ 2 k during fine-tuning to track curvature drift adaptively. This concludes the theoretical and empirical supplement. We finish with the required disclosure concerning the use of language-model tools during preparation. I LLM USE DISCLOSURE We employed Large Language Models (LLMs) to refine the text for grammar and clarity. Additionally, LLMs were used for debugging the published codebases of Askari et al. [2025], Kwon et al. [2023] and in writing visualization code. We confirm that LLMs were not used to implement any of the core algorithms or methodologies proposed in this work.