Paper deep dive
Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
Hossein Mobahi, Peter L. Bartlett
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/24/2026, 3:06:21 AM
Summary
The paper introduces HOPE (Hilbert Operator for Progressive Encoding), a mathematical framework for deconstructing deep neural network representations by shifting compression from the discrete parameter space to a continuous Hilbert space. HOPE models neurons as rank-1 Hilbert-Schmidt operators, unifying pruning and merging as low-rank subspace projections. It utilizes Batch Normalization statistics to construct a data-free Gaussian surrogate distribution, enabling unbiased architectural decisions and capacity measurement without requiring training data or hyperparameters.
Entities (6)
Relation Signals (5)
HOPE โ uses โ Hilbert-Schmidt Operator
confidence 95% ยท By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection.
HOPE โ leverages โ Batch Normalization
confidence 92% ยท HOPE applies the Maximum Entropy principle directly to the empirical BN statistics embedded within the trained model checkpoint.
HOPE โ enables โ Data-free Compression
confidence 90% ยท A key advantage of HOPE is that it operates entirely data-free for networks utilizing Batch Normalization (BN).
HOPE โ mitigates โ Magnitude-based Pruning
confidence 85% ยท Simple heuristics such as magnitude-based pruning... often fail... To address the shortcomings... we propose to transition from the physical parameter space to the function space
HOPE โ appliesto โ ResNet-50
confidence 80% ยท Throughout this work, we contextualize our framework within standard modern vision architectures, specifically the ResNet-50 (V1) architecture.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.
Tags
Links
- Source: https://arxiv.org/abs/2607.21366v1
- Canonical: https://arxiv.org/abs/2607.21366v1
Trouble viewing inline? Open PDF directly โ
Full Text
198,222 characters extracted from source content.
Expand or collapse full text
2026-07-24 Hilbert Operator for Progressive Encoding (HOPE) A Mathematical Framework for Deconstructing Learned Representations in Deep Networks Hossein Mobahi 1 and Peter L. Bartlett 1,2 1 Google DeepMind, 2 University of California, Berkeley Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory. 1. Introduction While deep neural networks learn complex representations, deconstructing this knowledge from numerical weights remains challenging. In this work, we use model compression as a measurable proxy task to study these internal representations objectively. Given the fundamental link between compression and learning [Rissanen, 1978,Hinton and Van Camp, 1993], which has recently been reinforced by demonstrating that LLMs are general-purpose compressors [Delรฉtang et al., 2023] and amortized algorithmic predictors [Genewein et al., 2026], we believe compression provides a promis- ing lens to study this issue objectively. Deconstructing opaque networks through capacity reduction has foundational roots [Mozer and Smolensky, 1988]. Viewed through a modern information-theoretic lens, learning is essentially the systematic discarding of task-irrelevant noise to isolate generalizable core patterns [Tishby et al., 1999, Shwartz-Ziv and Tishby, 2017]. Consequently, we posit that progressive compression is an effective tool for achieving this separation. Because core invariants resist pruning significantly longer than malleable slack, iteratively reducing capacity with minimal distortion naturally peels away the periphery to expose the networkโs universal feature space [He et al., 2026,Nguyen et al., 2026,Wang et al., 2025]. This post-training decon- struction mirrors the networkโs learning dynamics: the โcoarse-grainingโ dimensionality reduction that gradient descent originally used to build them [Dandi et al., 2025]. Indeed, theoretical analyses confirm that networks learn by incrementally adding effective units to model increasingly complex functions [Zhang et al., 2025]. While biologically inspired paradigms like [Behrouz et al., 2026] achieve this core-slack segregation by expanding a network to consolidate memories, we demonstrate that this separation can be accomplished efficiently through data-free compression. Despite compressionโs promise, the opacity of deep networks poses hurdles for objectively identi- fying and reducing capacity. Simple heuristics such as magnitude-based pruning (e.g. norm of raw Corresponding author(s): hmobahi@google.com, peterbartlett@google.com ยฉ 2026 Google. All rights reserved arXiv:2607.21366v1 [cs.LG] 23 Jul 2026 Hilbert Operator for Progressive Encoding (HOPE) weights) often fail because these magnitudes are typically optimization artifacts rather than impor- tance indicators [Scholl et al., 2021,Tanaka et al., 2020,Hooker et al., 2019], as also long observed in network sparsification [LeCun et al., 1989,Hassibi and Storkey, 1992,Frankle and Carbin, 2019]. To address the shortcomings of magnitude based notions of capacity, we propose to transition from the physical parameter space to the function space, specifically the underlying transformation a neuron applies to its input. This perspective treats the entire neuron, rather than individual weight matrices, as the atomic unit of the network. Note that this transformation must be analyzed over the relevant data manifold rather than the entire input space. For instance, characterizing a neuronโs behavior within a dog-versus-cat image classification task requires isolating the data manifold specific to those two classes, while discarding irrelevant classes and non-natural images. While one might attempt to approximate the input manifold to each neuron empirically by passing a finite dataset through the network, such finite-dimensional approximations tether the evaluation to specific samples. This reliance on explicit data sets can render the resulting architecture brittle to distribution shifts and disproportionately degrade performance on long-tail features [Hooker et al., 2019]. Furthermore, empirical approaches that rely on continuous activation matching or curvature approximations [Molchanov et al., 2017,Luo et al., 2017] incur a severe computational penalty, as they require numerically evaluating neural activations across the dataset repeatedly during every iteration of the progressive model encoding. To avoid both the drawbacks of weight-based parameterization and the pitfalls of empirical data dependence, we introduce the Hilbert Operator for Progressive Encoding (HOPE) framework. The core philosophy of HOPE is that evaluation and analysis of neurons must occur in the function space, the resulting compression must be executed discretely on the networkโs parameters. By lifting the parameter vectors and matrices of individual neurons into continuous functions, HOPE models each neuron as a rank-1 Hilbert-Schmidt operator. This abstraction unifies pruning and merging under a single theoretical paradigm: an optimal low-rank projection within a functional tensor space, where distances are measured by โฅ ยท โฅ H . Once the optimal reductions (pruning or merging) are computed in this pure functional space, the framework projects the resulting continuous operators back onto the network parameters to execute the compression. A key advantage of HOPE is that it operates entirely data-free for networks utilizing Batch Normalization (BN). 1 While prior research has successfully exploited BN moving averages to generate synthetic images for data-free knowledge distillation [Yin et al., 2020,Micaelli and Storkey, 2019], HOPE leverages this information in a different manner. Instead of generating synthetic spatial data or relying on massive real datasets to drive input signals, HOPE applies the Maximum Entropy principle directly to the empirical BN statistics embedded within the trained model checkpoint. This yields a continuous surrogate for the local input distribution at each neuron, allowing the framework to analytically evaluate the integrals required for Hilbert space norms โฅ ยท โฅ H . This fully continuous, analytical perspective largely mitigates scaling symmetries inherent in raw parameters, and thus enables unbiased capacity measurements across heterogeneous layers without requiring a single real or synthetic data sample. The primary value of HOPE lies in establishing a rigorous, hyperparameter free (and also data free in presence of BN) mathematical framework for the progressive encoding of trained deep networks. We present proof-of-concept applications for model compression and fine-tuning experiments to empirically validate these theoretical capabilities, rather than to establish exhaustive, large-scale benchmarks. 1 This data-free approach relies on global BN statistics. For architectures lacking BN, HOPE easily adapts: it requires only a simple, one-time calibration pass over a small data batch to capture the necessary pre-activation statistics. 2 Hilbert Operator for Progressive Encoding (HOPE) Paper Organization. We begin with a literature review in Section 2 and formally define a neuron within our framework in Section 3. To enable data-free evaluation, Section 4 constructs a surrogate distribution constrained by Batch Normalization (BN) statistics. In Section 5, we lift discrete neurons into continuous Hilbert-Schmidt operators. Section 6 then derives a cost functionalJthat captures the distortion induced by pruning or merging via subspace projection. Building on this, Section 7 derives the optimal parent neuron in function space for a given pair of merged neurons, detailing how this continuous representation is mapped back to the networkโs discrete parameters. Section 8 extends this projection metric to the macro level to evict larger residual blocks. Section 9 introduces a rate-distortion-inspired objective that balances the distortion costJagainst the resulting reduction in parameter count; this objective is used to greedily select the next optimal compression action. Section 10 presents the encoding process as a loop over greedy selection of compression actions. Finally, Section 11 provides a proof-of-concept evaluation of our framework for model compression and fine-tuning. 2. Related Works Pruning and Parameter-Space Methods. Network compression has historically relied on parameter- space pruning, evolving from early Taylor-expansion techniques [LeCun et al., 1989,Hassibi and Storkey, 1992] to modern unstructured [Han et al., 2015,Frankle and Carbin, 2019] and structured [He et al., 2017, Wen et al., 2016] approaches. HOPE directly compares against structured baselines utilizingํฟ 1 -norms [Li et al., 2017] and BN scaling [Liu et al., 2017]. To capture functional importance, methods rang- ing from early skeletonization via error derivatives [Mozer and Smolensky, 1988] to modern empir- ical activation tracking [Molchanov et al., 2017,Molchanov et al., 2019,Singh and Alistarh, 2020, Luo et al., 2017] rely on dataset passes; however, this introduces computational bottlenecks and brittleness to distribution shifts [Hooker et al., 2019]. Conversely, parameter-centric heuristics suffer from โscale symmetryโ [Blalock et al., 2020,Renda et al., 2020,Badrinarayanan et al., 2015, Dinh et al., 2017], where overparameterized many-to-one mappings [Neyshabur et al., 2015] mean raw magnitudes often reflect optimization artifacts rather than true importance [Scholl et al., 2021, Tanaka et al., 2020,Hooker et al., 2019]. Although methods like [Lee et al., 2021] can partially mit- igate global scale symmetries, they remain bound to parameter-space magnitude heuristics, which are vulnerable to within-neuron scaling artifacts. By migrating evaluation to a continuous Hilbert space, HOPE circumvents both empirical data bottlenecks and parameter-space artifacts. This shift from struc- tural to functional mappings aligns with the Platonic Representation Hypothesis [Huh et al., 2024], which posits that diverse networks converge to a shared statistical model of reality, treating weight spaces as mere shadows. HOPE operationalizes this intra-model, defining identity via continuous operators rather than superficial parameters. Neuron Alignment and Model Merging. Some recent methods on merging networks include permuta- tion invariances [Entezari et al., 2022,Ainsworth et al., 2023], Optimal Transport [Singh and Jaggi, 2020], alignment strategies [Tatro et al., 2020], or feature zipping [Stoica et al., 2024]. Other approaches perform data-free neuron merging by evaluating pairwise parameter similarities [Srinivas and Babu, 2015] or clustering weights via adaptive scalar hashing [Yvinec et al., 2021]. However, these methods typ- ically rely on combinatorial matching or parameter averaging. HOPE advances this paradigm by unifying pruning and merging under a single continuous operation: optimal low-rank projection in a functional tensor space, penalizing distortion via the Hilbert-Schmidt norm. This geometric approach mirrors using subspace embeddings to capture hierarchical and compositional representa- tions [Moreira et al., 2026]. Merging acts as the inverse to โfeature splittingโ [Bricken et al., 2023] or computation in superposition [Hรคnni et al., 2024], where overparameterized models fragment concepts across correlated sub-features, a phenomenon rooted in foundational vector space arithmetic 3 Hilbert Operator for Progressive Encoding (HOPE) [Mikolov et al., 2013,Pennington et al., 2014] and the Linear Representation Hypothesis [Park et al., 2023, Engels et al., 2024]. By projecting a rank-2 neuron pair into an optimal rank-1 Hilbert-Schmidt parent, HOPE reconsolidates this distributed knowledge, reclaiming capacity while preserving the networkโs underlying linear geometry. Macro Architecture and Layer Pruning. To reduce networkโs depth, standard methods employ stochastic depth [Huang et al., 2016], heuristic layer dropping [Fan et al., 2020], Neural Architecture Search [Zoph and Le, 2017,Liu et al., 2019], or dynamic routing gates [Veit and Belongie, 2018, Wang et al., 2018]. However, these approaches decouple macro-architectural decisions from granular feature selection, relying on separate optimization phases or custom hyperparameters. HOPE unifies these scales by formalizing block eviction as a macroscopic function subspace projection. Mapping this architectural deletion to the identical capacity costJallows macro-reductions to compete against granular pruning and merging within a single, hyperparameter-free decision engine. Data-Free Compression and Maximum Entropy Surrogates. Bypassing the original training data often involves inverting BN statistics to generate synthetic inputs [Lopes et al., 2017,Cai et al., 2020, Nagel et al., 2019,Micaelli and Storkey, 2019,Yin et al., 2020] or conserving discrete synaptic flow at initialization [Tanaka et al., 2020]. Rather than generating explicit samples, HOPE elevates these empirical statistics via the Maximum Entropy principle [Jaynes, 1957] to construct a continuous analytical surrogate, similar to some task-agnostic pruning of LLMs [Ma et al., 2023]. Paralleling theoretical analyses of infinite-width networks and Gaussian Processes [Neal, 1996,Lee et al., 2018, Jacot et al., 2018,Yang, 2019], this framework allows infinite-dimensional integrals to be resolved without a single forward pass. Parameter Plasticity, Continual Learning, and Representation Deconstruction. HOPE concep- tually bridges progressive compression with transfer learning [Yosinski et al., 2014,Hu et al., 2022, Houlsby et al., 2019] and the Stability-Plasticity dilemma of continual learning [Grossberg, 1987, Kirkpatrick et al., 2017,Zenke et al., 2017]. Drawing parallels to Complementary Learning Systems in cognitive neuroscience [McClelland et al., 1995,Kumaran et al., 2016], and aligning with the growing recognition across deep learning [Kong et al., 2026], we hypothesize that the learned rep- resentation must be explicitly segregated into a universal core of invariants and a peripheral slack of malleable volume in order to allow learning without forgetting. While previous frameworks attempt to protect foundational knowledge against representational drift using computationally massiveO(ํ 3 )orthogonal matrix projections [Saha et al., 2021,Zeng et al., 2019,Yang et al., 2025, HuggingFace Research Team et al., 2026] or dataset-dependent quadratic penalties [Kirkpatrick et al., 2017], HOPE provides a purely data-free alternative. By computing neuron capacity inO(ํ)time, its progres- sive pruning and merging peel away slack and expose core abstractions, computationally mirroring how awake biological circuits rapidly decorrelate co-activated neurons to maintain network sta- bility [Andrei et al., 2023]. This perspective is corroborated by recent literature utilizing capacity reduction to deconstruct hierarchies. For example, targeted parameter removal has been used to explain robustness [He et al., 2026], uncover heavy-tailed synaptic backbones [Nguyen et al., 2026], and demonstrate hierarchical learning through progressive feature compression [Wang et al., 2025]. Collectively, these works support the premise that HOPEโs neuron capacity may efficiently identify foun- dations versus plastic slack, laying the groundwork for transfer learning and downstream algorithmic interpretability [Bau et al., 2017, Morcos et al., 2018, Olah et al., 2020, Neyshabur et al., 2017]. 3. The Neuron Scale invariances are inherently relational and that is why isolated parameters fail to capture them. The minimal architectural unit where these symmetries fully manifest is the neuron in its entirety 4 Hilbert Operator for Progressive Encoding (HOPE) (encapsulating its incoming weights, BN parameters, non-linear activation, and outgoing weights). In this section we discuss how these within-neuron scale symmetries can be factored out, while deferring global scale symmetries to Section 5. For clarity, we derive our theoretical framework under the assumption of a fully connected architecture. However, this formalism seamlessly extends to convolutional networks (see Appendix B.1). Indeed, the architectures evaluated in Section 11 rely on this adaptation. Consider a neuronํwith input weightsํ raw,ํ โ โ ํ and output weightsํ out,ํ โ โ ํ . Let the neuron be subject to learnable affine BN parameters(ํพ ํ , ํฝ ํ )and empirical moving dataset statistics(ํ ํ , ํ 2 ํ ), whereํ ํ โ ํผ X [ํ ํ raw,ํ ํ] andํ 2 ํ โ Var X (ํ ํ raw,ํ ํ) . To capture the signal reaching the non-linearity, we absorb the normalization operations into a set of effective parameters: ํ eff in,ํ โ ํพ ํ / โ๏ธ ํ 2 ํ + ํํ raw,ํ , ํ ํ โ ํฝ ํ โ ํพ ํ ํ ํ / โ๏ธ ํ 2 ํ + ํ,(1) whereํ >0 is a small numerical stability constant. The neuronโs end-to-end signal mapping, capturing its functional contribution to the subsequent layer for an inputํ, is defined by the continuous function: ํ ํ (ํ)=ฮจ(ํฆ ํ )ํ out,ํ , ํฆ ํ โ (ํ eff in,ํ ) ํ ํ+ ํ ํ ,(2) whereฮจ(ยท) denotes a PH-1 activation function defined as below: Positively Homogeneous of degree 1 (PH-1) Functions An activation functionฮจ:โโ โis Positively Homogeneous of degree 1 (PH-1) if it satisfies the scaling propertyฮจ(ํํง)= ํฮจ(ํง) for all ํง โ โ and all scalars ํโฅ 0. Examples include ReLU, Leaky ReLU, PReLU, and the linear functions. Throughout the HOPE framework, we formally designate this continuous functionํ ํ as the neuron. Operating at this atomic level mitigates two primary sources of scaling symmetry: normalization invariance (from BN) and re-parameterization invariance (from cross-layer weight resharding). We detail the former below, and defer the latter to Section 5. 3.1. Mitigating Normalization Invariance Normalization invariance arises from BNโs standardizing mechanics. Scaling raw input weightsํ raw,ํ by a constant factorํ >0 increases the pre-activation variance byํ 2 . The subsequent BN layer divides by the standard deviation, cancelingํbefore the non-linearity. Because downstream output remains unchanged, raw weight magnitudes can be deceptive. HOPE mitigates this failure mode by evaluating capacity through effective parameters (ํ eff in,ํ and ํ ํ ). While HOPE uses raw weightsํ raw,ํ alongside BN statistics to construct the data-constrained surrogate datasetํ X (Section 4), their utility ends there. The framework then evaluates the neuronโs continuous-functional impact on this surrogate. Physically, the non-linear activation processes the normalized, shifted signal, not the raw parameter projection. Thus, when computing the continuous integral to compute Hilbert space norms or inner product, HOPE defines the pre-activation signal asํฆ ํ = (ํ eff in,ํ ) ํ ํ + ํ ํ . Becauseํ eff in,ํ divides by the empirical standard deviationํ ํ , any magnitude inflation is canceled. By evaluating the signal impacting the activation function, HOPE guarantees the capacity criterion reflects functional utility rather than scale artifacts. 5 Hilbert Operator for Progressive Encoding (HOPE) 4. The Neural Signal Distribution Since HOPE operates in a data-free regime, the true input distributionํ โ X (encompassing both initial data and subsequent layer activations) is inaccessible. However, because we model neurons as continuous Hilbert-Schmidt operators, evaluating their inner products requires integrating over the data distribution. To resolve this, we invoke the Maximum Entropy principle [Jaynes, 1957] to construct a Gaussian surrogate constrained by BN statistics. While a Gaussian approximation may seem overly idealistic, we explain below in two steps why it aligns closely with modern neural architectures. Step 1: Gaussianity of Pre-Activation. While a neuronโs true input distributionํ โ X often lies on a complex, highly non-Gaussian manifold, each neurons observes its inputํonly through 1-dimensional linear projectionsํฆ= ร ํ ํ=1 ํค ํ ํฅ ํ . As the fan-in dimensionํgrows, by the Central Limit Theorem and the Diaconis-Freedman effect [Diaconis and Freedman, 1984], these aggregated signals converge to a Gaussian distribution. Consequently, neurons remain oblivious to the complex data manifold; from their perspective the pre-activationํฆ ํ is Gaussian. Although nonlinear activations (e.g., ReLU) disrupt this Gaussianity, subsequent high-dimensional linear transformations recursively smooth the signals back into Gaussian pre-activations across layers. Step 2: Gaussian Surrogate for the Inputํ. While neurons are oblivious to the true shape of the data manifold and perceive their input as a Gaussian signalํฆ. Based on this observation, for theoretical convenience, we substitute the complex true data with a tractable surrogate. For architectural consistency, the surrogate distribution must satisfy the same observational bottleneck: its 1D linear projections must remain Gaussian. By definition, if every linear combination of a random vector is Gaussian, the vector itself must be multivariate Gaussian. Thus, to construct a surrogate distribution aligned with a world where every linear observer (neuron) sees a Gaussian, that surrogate is necessarily a multivariate Gaussian, ํ X =N(หํ ํฅ , ห ฮฃ ํฅ ). Architectural Context Throughout this work, we contextualize our framework within standard modern vision architec- tures, specifically the ResNet-50 (V1) architecture. The canonical computational block follows the sequence: convolution, followed by BN, followed by a ReLU activationConvโ BNโ ReLU. Consequently, the input vectorํpresented to any internal convolutional layer is the output of a preceding ReLU activation. Post-ReLU Support Paradox Using a multivariate Gaussian surrogateํ X whose support is the entireโ ํ might seem problematic, given that post-ReLU inputs are non-negativeํ โฅ0. However, the purpose ofํ X is to model the inner product ํ ํ , ํ ํ H , not the true input distribution. Because ํ ํ , ํ ํ H = ํผ ํโผํ X [ฮจ(ํฆ ํ )ฮจ(ํฆ ํ )] ยท ํ out,ํ , ํ out,ํ โ ํ , this integral reduces to a 2D subspace de- fined by the pre-activationsํฆ ํ andํฆ ํ . In high dimensions, the Central Limit Theorem and the Diaconis-Freedman effect ensure that projecting high-dimensional vectors into a low- dimensional subspace rapidly converges to a bivariate Gaussian distribution. Since the integra- tion depends solely on this 2D slice, relaxing the ambient non-negativity constraint yields a tractable surrogate while maintaining an accurate asymptotic approximation of the bivariate distribution. Optimal Surrogate. To define the optimal parameters(หํ ํฅ , ห ฮฃ ํฅ )of the unknown data distribution, we can incorporate the two empirical constraintsํ โ โ ํ :ํผ[ํ ํ raw,ํ ํ]= ํ ํ andVar(ํ ํ raw,ํ ํ)= ํ 2 ํ 6 Hilbert Operator for Progressive Encoding (HOPE) imposed by BN for anyํ. We define a shared surrogate for each layer. Since the empirical means ํ BN โ โ ํ represent the 1D shadow of the datasetโs center cast through the raw weights, to find the best mean for a layer we compute the optimal least-squares approximation of the datasetโs center using the Moore-Penrose pseudo-inverseํพ + raw , which leads toหํ ํฅ = ํพ + raw ํ BN . In underdetermined scenarios (e.g., when compressing layers whereํ > ํ), this pseudo-inverse yields the minimum-norm solution, which sets the unobserved orthogonal components of the data manifold to a zero mean. To find the optimal covariance matrix ห ฮฃ ํฅ for the layer, we maximize the differential entropy of the multivariate Gaussian,ํป(ํ) โ log det(ฮฃ ํฅ )subject to the BN variance constraintsํ ํ raw,ํ ฮฃ ํฅ ํ raw,ํ = ํ 2 ํ forํ โ 1, . . . , ํ. Conceptually, applying Lagrange multipliers yields ห ฮฃ ํฅ = ร ํ ํ=1 ํ ํ ํ raw,ํ ํ ํ raw,ํ โ1 , whereํ ํ are optimized to satisfy variance equality constraints. However, inverting this covariance matrix is computationally expensive. Fortunately, the framework bypasses the computationally expensive need to compute and invert the full high-dimensional joint covariance matrix entirely: micro-operations evaluate pairwise merges restricted to a rank-2 subspace, allowing a closed-form solution via a pairwise neural kernel (Appendix E.3), while macro block eviction evaluates destruction using the ํฟ 1 cumulative distance of surviving capacities (Section 8). 5. A Hilbert Functional Perspective on Neurons To mitigate parameter shape bias and facilitate the identification of dead neurons, we transition from discrete parameter analysis to a continuous function formulation. By embedding each neuron into a Hilbert space 2 Hand treating it as a rank-1 Hilbert-Schmidt operator, our framework evaluates the actual function the neuron computes, effectively abstracting away its physical matrix shape. Furthermore, by integrating this continuous function over the analytically derived surrogateํ X , HOPE identifies dead neurons via a closed-form expectation, bypassing the need for computationally expensive empirical forward passes over a dataset. Together, these features shift the evaluation criterion from raw parameter counts to functional capacity, quantified as the norm of the neuronโs function inH. We model each neuron as a rank-one Hilbert operatorํ ํ โH. This way, a neuronโs identity is not defined by its evaluation on a single input pointํ, but its continuous behavior over the entire surrogate distributionX. We quantify the capacity of the neuron by โฅ ํ ํ โฅ H . We now present this formally. We define our space of neural functions asHโ ํฟ 2 (X, ํ X ;โ ํ ), the space of square-integrable functions mappingXto theํ-dimensional output space. Defineํ ํ :X โ โ asํ ํ (ํ)โฮจ (ํ eff in,ํ ) ํ ํ+ ํ ํ . We embedํ ํ as an element of the scalar Hilbert spaceH in โ ํฟ 2 (X, ํ X ;โ). Furthermore, the scalar activation of a neuron is sent to the next layer by scaling with the finite- dimensional output weight vectorํ out,ํ โ H out โ โ ํ . Since the output across allํdimensions is confined to the one-dimensional subspace spanned byํ out,ํ , this entire continuous landscape is embedded exclusively along a single vector direction. By taking the tensor product of the input function and the output vector, we construct a linear mapping across these spaces:H H in โH out . Thus the vector-valued functionํ ํ :X โ H out , i.e. the neuron, is an element within this tensor product spaceํ ํ โ ํ ํ โ ํ out,ํ . Since this element is constructed from the outer product of one input function and one output vector, each individual neuronํ ํ is a rank-1 Hilbert-Schmidt operator. See Figure 1 for a visualization. This tensor formulation is fundamental for defining the merging operation in HOPE as we will discuss in Sections 6 and 7.1. Hilbert-Schmidt Inner Product and Capacity. Because our neurons are defined as rank-1 operators residing in the tensorH H in โ H out , we must evaluate their geometric relationship using the 2 See Appendix A for a brief introduction to Hilbert spaces. 7 Hilbert Operator for Progressive Encoding (HOPE) Input Function ํ ํ โH in โ Output Vector ํ out,ํ โH out ํฅ 1 ํฅ 2 ํฅ 3 ํ out,ํ = Tensor Rank-1 Operator ํ ํ โH ํฅ 1 ํฅ 2 ํฅ 3 Image of landscape restricted to line ํ out,ํ Figure 1|Visualizing the rank-1 tensor of a neuron. Left: The input phase computes a continuous (infinite- dimensional) scalar landscapeํ ํ (ํ). Middle: The output phase defines a single, finite-dimensional weight vectorํ out,ํ . Right: The tensor product binds them. The entire functionโs landscape (represented by the colored points) is mapped along the 1D subspace spanned by ํ out,ํ , which enforces the definition of a rank-1 operator. inner product defined on this composite space: ํ ํ , ํ ํ H = ํ ํ โ ํ out,ํ , ํ ํ โ ํ out,ํ H = ํ ํ , ํ ํ H in ยท ํ out,ํ , ํ out,ํ H out = ํผ ํโผํ X [ฮจ(ํฆ ํ )ฮจ(ํฆ ํ )]ยท ํ out,ํ , ํ out,ํ โ ํ . We define the capacity of a neuron as its Hilbert-Schmidt normโฅ ํ ํ โฅ H = โ๏ธ โจ ํ ํ , ํ ํ โฉ H , which we will later use to decide what neuron to prune and which macro block to evict. Kernel Formulation. We define the kernel of two neuronsํ, ํasํพ(ํ, ํ)โ ํผ ํโผํ X [ฮจ(ํฆ ํ )ฮจ(ํฆ ํ )] . Under this, the capacity of a neuron simplifies toโฅ ํ ํ โฅ H =โฅํ out,ํ โฅ 2 ยท โ๏ธ ํพ(ํ, ํ)and the inner product of two neurons becomes ํ ํ , ํ ํ H = ํ out,ํ , ํ out,ํ โ ํ ํพ(ํ, ํ) . Closed form expression for these kernels are provided at the end of this section, and their derivation is presented in Appendix E whenฮจis the ReLU activation function. Neuron Scale Invariance. In networks with PH-1 activations, scalingํ eff in,ํ andํ ํ of a neuron by ํ >0 andํ out,ํ by 1/ํalters weight magnitudes without changing the downstream function. This symmetry confounds raw magnitude-based criteria. HOPE mitigates this issue by defining capacity as the Hilbert norm:โฅ ํ ํ โฅ H =โฅํ out,ํ โฅ 2 โ๏ธ ํพ(ํ, ํ). Because positive homogeneity scales the kernelํพbyํ, the opposing factors cancel. Consequently, HOPE guarantees an invariant capacity score regardless of weight resharding. Neuron Shape Invariance. By definition, the neuronโs functional capacityโฅ ํ ํ โฅ H =โฅํ out,ํ โฅ 2 โ๏ธ ํพ(ํ, ํ) depends on the input spaceXsolely through the kernel termํพ(ํ, ํ)โ ํผ ํโผํ X [ฮจ 2 (ํฆ ํ )]. Rather than counting discrete incoming parametersํor computing weight magnitudes such asโฅํ eff in,ํ โฅorโฅํ raw,ํ โฅ, this formulation abstracts away the physical dimensionality of the input tensorํ โ โ ํ . 3 Furthermore, when paired with the layer-wise magnitude neutrality axiom, this function-based approach ensures the neuronโs evaluation is entirely invariant to both its input and output dimensions. 4 3 While increasing the fan-inํnaturally inflates the pre-activation varianceVar X (ํ ํ raw,ํ ํ), this scaling artifact is then mitigated by the absorbed BN parameters. Becauseํ eff in,ํ โ (ํพ ํ / โ๏ธ ํ 2 ํ + ํ)ํ raw,ํ , the variance of the pre-activation signalํฆ ํ is bounded entirely by the learned scaleํพ 2 ํ . Consequently, the expected activation energyํผ ํโผํ X [ฮจ 2 (ํฆ ํ )]remains decoupled from the physical width of the input tensor. 4 Although capacityโฅ ํ ํ โฅ H scales with the output dimensionํviaโฅํ out,ํ โฅ 2 , HOPE mitigates this downstream. As shown in Section 6, compression is governed by the distortion costJ. Derived axiomatically,Jnormalizes a neuronโs capacity 8 Hilbert Operator for Progressive Encoding (HOPE) ํธ = 5 ํธ = 4 ํธ = 3 ํธ = 2 Rank-1 Subspace ( ํ 1 = ํ 2 ) ํ 1 โH ํ 2 โH โซ 1 0 ยค J proj ํ ํ โ ํซ ( ํฝ ํ , ฬ ํฝ ํ ) โซ 1 0 ยค J capacity ํ ํ โ ฮ ํธ = โ 1 Initial Stateฮฆ(0) Terminal State ฬ ฮฆ(1) Origin (ํธ= 0) Figure 2|Visualization of Transition Costs of Merging inH 2 . The axes schematically repre- sent the space of continuous functionsH, illus- trated as a smooth transition from linear to sinu- soidal. During a merge operation, the layer tran- sitions from an initial stateฮฆ(0)=ฮฆ ํ to a pre- deletion targetฮฆ(1)= ฬ ฮฆ ํ whereํ 1 = ํ 2 . For the sake of illustration, supposeํธ(ํก) โ ํธ ํ and ํ(ฮฆ(ํก))= ํ 0 for 0โค ํก โค1. Then โซ 1 0 ยค J capacity ํํก โ โ ํ 0 ํธ ํ โซ 1 0 ยค ํธ(ํก) ํํก= โ ํ 0 ํธ ํ (ํธ ํ โ ํธ ํ )= โ ํ 0 ํธ ํ ฮํธ โ โฮํธ , yielding a penalty proportional toโฮํธ. How- ever, โซ 1 0 ยค J proj ํํก โ ํ 0 ํธ ํ โซ 1 0 ยค ํ (ํก) ํํก= ํ 0 ํธ ํ (ํ ( 1)โ ํ (0)) โ ํท(ฮฆ ํ , ฬ ฮฆ ํ ), the Euclidean projection distance shown as the straight purple line. Neuron Merging via Hilbert-Schmidt Projection. Similar to pruning, merging also reduces the networkโs neuron count by one, but it can provide a higher-fidelity reduction when the selected pair have a strong cosine similarity inH. We define the merger through an optimal Hilbert-Schmidt projection. Since neuronsํ, ํare each rank-1 operators, their joint contribution[ ํ ํ , ํ ํ ]spans a 2- dimensional subspace inHconsisting of operators of rank at most 2. Merging them into a single parent neuron is defined as finding the optimal rank-1 approximation of this tensor subspace. Self-Kernel ofReLU Neurons Let ํ andฮฆ be the standard Normal PDF and CDF respectively. Then: ํพ(ํ, ํ)=(ํพ 2 ํ + ํฝ 2 ํ )ฮฆ ํฝ ํ |ํพ ํ | + ํฝ ํ |ํพ ํ |ํ ํฝ ํ |ํพ ํ | (3) Cross-Kernel ofReLU Neurons For brevity, let: ํ eff โ D ํ eff in,ํ , ํ eff in,ํ E ํ eff in,ํ 2 ํ eff in,ํ 2 , ํ โ ํ eff 1โ ํ 2 eff ! ยฉ ยญ ยญ ยซ |ํพ ํ | ํ eff in,ํ 2 ยช ยฎ ยฎ ยฌ ยฉ ยญ ยญ ยซ |ํพ ํ | ํ eff in,ํ 2 ยช ยฎ ยฎ ยฌ , หํ ํํ โ 2ํ 1+ โ 1+ 4ํ 2 . (4) Then the cross-kernel has the form a : ํพ(ํ, ํ) โ 1 ํ โ๏ธ 1โ หํ 2 ํํ +(ํโ arccos หํ ํํ )หํ ํํ โ๏ธ ํพ(ํ, ํ)ํพ(ํ, ํ)(5) a While the exact cross-kernel can be derived analytically, it requires evaluating a bivariate normal CDF for every neuron pair, which is computationally prohibitive for large networks. Instead, we approximate the kernel by assuming zero biasํฝ ํ , ํฝ ํ โ0. This isolates the angular alignment,หํ ํํ , as the primary driver of redundancy and avoids costly CDF evaluations. Full derivations of both the exact and approximate kernels are in Appendix E. against the layerโs total capacity. Since all neurons in a layer share the same output spaceโ ํ , this emergent normalization factors out ํ. 9 Hilbert Operator for Progressive Encoding (HOPE) 6. Layer Transition Costs Scale symmetries also manifest globally: shallow layers processing high-variance data often yield neurons with larger capacities than deeper layers operating on compressed latent representations [Hanin and Rolnick, 2018,Tanaka et al., 2020]. For a compression method to account for this bias, it must evaluate individual neurons within the context of their entire layer [Lee et al., 2021]. We formalize this global context via a layer stateฮฆโ ( ํ 1 , ํ 2 , . . . , ํ ํ ), whereํis the number of active neurons in the layer. A single compression step maps an initial stateฮฆ ํ to a reduced stateฮฆ ํ , where |ฮฆ ํ |= ํโ1. The entire compression process is thus a chain of discrete state transitions across various layers, guided by a costJ(ฮฆ ํ ,ฮฆ ํ ) >0 that quantifies the resulting model distortion. This section focuses on deriving this J. 6.1. Continuous-Time Relaxation Despite the conceptual clarity of these transitions, their discrete nature (where the architecture hops from stateฮฆ ํ toฮฆ ํ via pruning or merging) presents a significant barrier to mathematical analysis. To bridge the gap between abstract analysis and algorithmic execution, we proceed in two steps. First, we perform a continuous relaxation: instead of an instantaneous jump, we define a continuous deformationฮฆ(ํก)overํก โ [0,1], interpolating betweenฮฆ(0)=ฮฆ ํ and a pre-deletion targetฮฆ(1)= ฬ ฮฆ ํ (Figure 2). This allows us to use differential equations to express the infinitesimal cost of shrinking a layerโs capacity. Second, to compute the total transition cost, we integrate this differential cost with respect toํก. Our objective is to resolve this integral into an expression that depends only on the physically realizable endpoints (ฮฆ ํ and ฬ ฮฆ ํ ), bypassing the need to evaluate fictitious intermediate states along the continuous path. However, since this integral generally lacks a closed-form solution, and numerical evaluation incurs computationally prohibitive runtime overhead, we instead derive a closed-form upper bound on the analytically intractable integral. Layer Capacity. To develop a layer costJ, we first extend the single-neuron capacity,โฅ ํ ํ โฅ, to define a layer capacityํธ(ฮฆ)for stateฮฆ, whereํธ( ํ 1 )= โฅ ํ 1 โฅ. A natural requirement is thatํธ(ฮฆ)remains invariant to arbitrary neuron partitioning. Assumingํธ(ฮฆ)is a symmetric, separable, and homoge- neous functional of individual capacities, this condition uniquely determinesํธ(ฮฆ)= ร ํ ํ=1 โฅ ํ ํ โฅ H (by Lemma C.1). For some intuition, suppose thatํธ(ฮฆ)= ร โฅ ํ ํ โฅ ํ H 1/ํ . Partitioning a neuronํ 0 intoํ fractionsํ 0 /ํyieldsํ (1โํ)/ํ โฅ ํ 0 โฅ H . Capacity invariance for anyํrequires(1โ ํ)/ํ=0, yielding ํ= 1. Axiomatic Cost Formulation. To ensure a well-posed definition ofJ, we introduce the following natural axioms: 1. Magnitude Neutrality:Jmust be scale invariant:โํ >0;J(ํฮฆ ํ , ํฮฆ ํ )= J(ฮฆ ํ ,ฮฆ ํ ) . 2. Connectivity Preservation:Jmust establish an asymptotic barrier preventing layer extinction:lim ํธ(ฮฆ ํ )โ0 + J= โ. 3. Infinitesimal Capacity Dependence:Jmust be additive along continuous paths and be driven by the reduction in layer capacity:J(ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 โํ(ฮฆ(ํก)) ยค ํธ(ํก)ํํก, where ยค ํธ(ํก)โ ํํธ(ฮฆ(ํก))/ํํกandํ(ฮฆ(ํก)) >0 is a state-dependent density function. While Axioms 1 and 2 define boundaries of the theory, Axiom 3 acts as an idealized analytical tool modeling a continuous capacity drain ยค ํธ(ํก) <0. This allows us to deduce the fundamental shape of the cost function. Under these premises, we can prove (By Theorem C.2) thatJmust obeyJ capacity (ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 โํ(ฮฆ(ํก)) ยค ํธ(ํก) ํธ(ฮฆ(ํก)) ํํก, where ยค ํธ(ํก) < 0 (due to capacity reduction) and ํ(ฮฆ(ํก)) > 0 is a scale-invariant factor (i.e., ํ(ํฮฆ)= ํ(ฮฆ) for anyฮฆ โH ํ and ํ > 0). Piecewise Constantํ(ฮฆ(.)). To bridge continuous theory with discrete execution, we restrictํ(ฮฆ) to remain constant along any discrete state transitionฮฆ ํ โฮฆ ํ , e.g.ํ(ฮฆ(ํก))= ํ(ฮฆ ํ )forํก โ [0,1]. This allows us to factorํ(ฮฆ)out of the integral for bothJ capacity and all subsequently derived cost 10 Hilbert Operator for Progressive Encoding (HOPE) functionals; forJ capacity , this directly yields the analytical solutionJ capacity = ํ(ฮฆ ํ ) ln( ํธ ํ ํธ ํ ) . Upon reaching the terminal state, physically removing extinguished neurons causesํ(ฮฆ(ํก))to snap to a new valueํ(ฮฆ ํ ). Consequently,ํ(ฮฆ(ํก))acts as a globally piecewise constant function that remains locally constant during any integration step. WhileJ capacity is not yet the final objective used in our optimizer, confirming that it satisfies Axioms 1 and 2 ensures we are on track, while its derivation via integration inherently satisfies the idealized capacity dependence assumption. 6.2. Bounding the Projection Cost While ยค J(ํก) is driven by the relative capacity reductionโ ยค ํธ/ํธ, our framework needs to minimize projection error 5 (Section 5). We bridge the two by calibrating along an orthogonal trajectory where ํํ =โํํธtranslates the abstract capacity lossโ ยค ํธinto a geometric speed ยค ํ . Hereํ (ํก)= โซ ํก 0 โฅ ยค ฮฆ(ํ)โฅ H ํ ํํ is the arc-length swept byฮฆ(ํก)through the spaceH ํ . BecauseH ํ is isotropic, this substitution generalizes to any arbitrary deformation path, yielding ยค J proj (ํก)= ํ(ฮฆ(ํก)) ยค ํ (ํก) ํธ(ฮฆ(ํก)) (Definition 44). This substitution shiftsJfrom pure capacity loss to any distance traversed, meaning the strict ยค ํธ(ํก) <0 assumption from the idealized model is no longer required along the physical path. The compression algorithm executes discrete leaps (e.g., snapping neuronsํ ํ andํ ํ to a shared parentํ ํ ). Evaluating the cost of this transition conceptually requires integrating ยค J proj = ํยท ยค ํ /ํธ over the jump path. However, because runtime integration is computationally prohibitive, we seek a fast, closed-form proxy. Since underestimating this integral risks destructive jumps (e.g., removing orthogonal features) and breaching layer depletion barriers before the continuous cost can diverge, we derive a closed-form upper bound to enforce cautious greedy optimization. We construct this bound by exploiting the inverse relationship between ยค J proj and ํธ in ยค J proj = ํยท ยค ํ /ํธ. For any arbitrary deformation path connectingฮฆ ํ toฮฆ ํ , we can establish an upper bound on the integral cost by replacing the dynamic capacityํธ(ํก)with a constant minimum,ํธ min , allowing us to pull the denominator outside the integral. This yields a bounded fractional cost where the numerator is the pathโs total arc length, โซ 1 0 ยค ํ (ํก)ํํก. Because infinitely many curves inH ํ connect the two states, this establishes a family of valid upper bounds. To tighten this proxy cost, we minimize the numerator by selecting the path with the shortest arc length: the straight-line trajectory inH ํ . This evaluates to the traversed Euclidean distance ํท(ฮฆ ํ , ฬ ฮฆ ํ )โ โฅฮฆ ํ โ ฬ ฮฆ ํ โฅ H ํ =( ร ํ ํ=1 โฅ ํ (ํ) ํ โ ฬ ํ (ํ) ํ โฅ 2 H ) 1 2 . Next, to complete this bound, we must safely approximate the denominatorโs minimumํธ min along this chosen straight-line path. Because the straight-line geometric path acts as a secant across the space of functions (abandoning the strict ยค ํธ(ํก) < 0 assumption), the capacityํธ(ํก)can temporarily dip below the pre-deletion targetํธ( ฬ ฮฆ ํ ). To safely absorb this without breaking the integral bound, we introduce a safety buffer by evaluating the denominator at the true terminal stateํธ(ฮฆ ํ ). For highly correlated neuron pairs, ํธ(ํก) โฅ ํธ(ฮฆ ํ ) throughout the straight-line transition (Lemma C.3). Substituting the minimized numeratorํทand evaluating the constant denominator asํธ(ฮฆ ํ )yields the final boundJ proj (ฮฆ ํ ,ฮฆ ํ ) โค ํ(ฮฆ ํ ) ํท(ฮฆ ํ , ฬ ฮฆ ํ ) ํธ(ฮฆ ํ ) โก J bound (ฮฆ ํ ,ฮฆ ํ ) (see Theorem C.4). Here, ฬ ฮฆ ํ โH ํ is the pre-deletion target atํก=1 (e.g., a duplicated parent[ ํ ํ , ํ ํ ]) but theํ-dimensional structure remains intact. Conversely,ฮฆ ํ โ H ํโ1 is the true terminal state: the layer after the extinguished neuron is dropped. This separation ensures no dimensional mismatch in the arguments ofํท, while the denominator only relies on the (ํโ 1)-dimensional post-deletion capacity ํธ(ฮฆ ํ ). 5 Transitioning fromJ capacity toJ proj ensures sensitivity to feature alignment. For instance, merging two orthogonal neurons introduces a severe subspace projection error whileJ capacity evaluates this catastrophic alignment loss identically to a merge between two collinear (hence redundant) neurons due to their equivalent linear capacity reductions. Shifting toJ proj reorients the optimization objective from macroscopic reduction to minimizing distortion within the networkโs internal mapping. 11 Hilbert Operator for Progressive Encoding (HOPE) Axiomatic Consistency of the Bounded Proxy. While the continuous functionalJ capacity was derived from our foundational axioms, the subsequent derivation ofJ bound alters the underlying differential equation. Specifically, to bypass the expensive runtime integration, we introduced a surrogate curve and bounded the capacity denominator. Because these approximations manipulate the original differential equation, it is no longer guaranteed a priori that the resulting closed-form proxy inherits the axiomatic properties of its continuous predecessor. However, we can prove that J bound (and consequentlyJ final ) still preserves the foundational axioms of Magnitude Neutrality and Connectivity Preservation (by Proposition C.5). However, the Infinitesimal Capacity Dependence assumption acts primarily as an analytical tool rather than a fundamental necessity, and is intentionally relaxed. Specifically,J capacity relies on integration over a path characterized by a monotonic capacity drain ยค ํธ(ํก) < 0. However, deriving the closed-formJ bound abandons this path integration in favor of a straight-line approximation evaluated at endpoints. Because this straight-line projection cuts directly acrossH ํ , the intermediate capacity along the path may temporarily fluctuate, violating the assumption of monotonic decrease required by the original differential equation. Consequently, J bound knowingly sacrifices the path-additivity required by the modeling assumption. This relaxation is necessary to translate abstract continuous theory into an efficientO(1)evaluation of discrete state transitions. Practical Notes The Correlation Constraint. The assumptionํธ(ํก) โฅ ํธ(ฮฆ ํ )holds only for highly correlated neurons, but this poses no practical limitation. Because the projection errorํท(ฮฆ ํ ,ฮฆ ํ )vanishes for collinear candidates, the greedy optimizer naturally minimizesJ bound by actively selecting highly correlated pairs, inherently satisfying this requirement. Locality of the Projection Error. EvaluatingJacross a wide layer might seem com- putationally intractable. However, for reductions modifying only a small subset of neu- ronsS(e.g., pruning or merging), the cost restricts entirely to the perturbed subspace: J bound (ฮฆ ํ ,ฮฆ ํ )= ํ(ฮฆ ํ ) โ๏ธ ร ํโS โฅ ํ (ํ) ํ โ ํ (ํ) ํ โฅ 2 H /ํธ(ฮฆ ํ ) (Corollary C.6). This isolates the com- putation from the total architectural width, guaranteeingO(1) execution time. Choice ofํ(ฮฆ)= ํ. We previously specifiedํ(ฮฆ)to be piecewise constant; we now propose a more specific definition: settingํ(ฮฆ)= ํfor each continuous piece. This is to avoid unfair removal of critical diversity from wide layers by the global optimizer before addressing obvious redundancies in narrow bottlenecks, which may occur as capacityํธ(ฮฆ)intrinsically scales with layer width. To mitigate this width bias, we normalize J using the average feature capacity 6 ํธ(ฮฆ)/ํ, which can be implemented by settingํ(ฮฆ)= ํ. Substituting thisํ(ฮฆ)intoJ bound yields the final costJ final โ ํยทํท(ฮฆ ํ , ฬ ฮฆ ํ ) ํธ(ฮฆ ํ ) . We can instantiate the pruning and merging costs as special cases of J final . 6.3. Final Pruning and Merging Costs Pruning a neuronํ ํ corresponds to projecting its rank-1 operator down to the null operator 0. Because the perturbed subspace only contains this single neuronS= ํand its terminal state is 6 Consider a mean-field assumption where each active neuron contributes an average capacity ฬํ. The incremental cost of pruning a single neuron evaluates toJ prune โ ํยท ฬํ ํยท ฬํ =1. This normalization renders the penalty invariant to the instantaneous layer width. Without this dynamic coupling (e.g., ifํwere anchored toํ initial ), the incremental cost would artificially explode as the live capacity shrinks, forcing an artificial uniformity that prevents the optimizer from fully extinguishing noisy, redundant blocks. 12 Hilbert Operator for Progressive Encoding (HOPE) 0, the projection error simplifies toํท= โ๏ธ โฅ ํ ํ โ 0โฅ 2 H =โฅ ํ ํ โฅ H . By evaluating the terminal capacity as ํธ(ฮฆ ํ )= ํธ ํ โโฅ ํ ํ โฅ H we get J prune = ํยทโฅ ํ ํ โฅ H ํธ ํ โโฅ ํ ํ โฅ H . Merging a neuron pair is slightly more involved. For neuronsํandํ, their joint operator[ ํ ํ , ํ ํ ] spans a rank-2 subspace inH. Becauseํ ํ andํ ํ are vector-valued functions, their joint operator is matrix-valued, denoted asํพ joint โ [ ํ ํ , ํ ํ ]. Merging compresses this into a rank-1 approximation ํพ โฒ joint . Classic unconstrained rank truncation (Eckart-Young-Mirsky) prescribes a rank-one basisํ ํ and independent scaling factorsํผ, ํฝ โ โby solvingmin ํ ํ โH,ํผ,ํฝโโ โฅํพ joint โ ํพ โฒ joint โฅ 2 H , whereํพ โฒ joint = [ํผํ ํ , ํฝ ํ ํ ]. However, because a physical neuron must produce a single unified output, we must restrict the valid replacement pair toํพ โฒ joint = [ ํ ํ , ํ ํ ] . This enforces the constraintํผ= ํฝ=1, yielding the constrained objectivemin ํ ํ โH โฅํพ joint โ ํพ โฒ joint โฅ 2 H and rendering standard unconstrained projections inapplicable. Deferring the derivation of the optimal parentํ ํ to Section 7.1, we first establish the objective functional itself. Since the distanceํทis the expected Frobenius projection error under the Hilbert- Schmidt norm, we expand it as follows: ํท 2 (ฮฆ ํ ,ฮฆ ํ )=โฅํพ joint โ ํพ โฒ joint โฅ 2 H = ํผ ํโผํ X h โฅํพ joint (ํ)โ ํพ โฒ joint (ํ)โฅ 2 ํน i =โฅ ํ ํ โ ํ ํ โฅ 2 H +โฅ ํ ํ โ ํ ํ โฅ 2 H . The terminal capacityํธ ํ updates by swapping the eliminated children for the new parent:ํธ ํ = ํธ ํ โโฅ ํ ํ โฅ H โโฅ ํ ํ โฅ H +โฅ ํ ํ โฅ H . Substituting ํท and ํธ ํ yields the final merging cost. Pruning and Merging Costs J prune = ํโฅ ํ ํ โฅ H ํธ ํ โโฅ ํ ํ โฅ H , J merge = ํ โ๏ธ โฅ ํ ํ โ ํ ํ โฅ 2 H +โฅ ํ ํ โ ํ ํ โฅ 2 H ํธ ํ โโฅ ํ ํ โฅ H โโฅ ํ ํ โฅ H +โฅ ํ ํ โฅ H .(6) 7. Generating the Parent Neuron 7.1. The Parent Neuron in Hilbert Space We determine the optimal parent neuronํ โ ํ by minimizingJ merge ( ํ ํ )subject toํ ํ โ N, whereN denotes the space of realizable neurons: Nโ ํ | ํ(ํ)= ํ out ฮจ( ฬํ in ยท ฬํ) โ H .(7) Here ฬํ=[ํ,1] ํ and ฬํ in =[ํ eff in , ํ] ํ denote the augmented inputs and weights. Any non-zero function ํ โHcan be decomposed into a scalar magnitudeํ >0 and a directionํโH, such thatํ= ํ ํand โฅํโฅ H =1. Applying this to the parent neuronํ ํ allows us to reformulate the search forํ โ ํ as the following nested optimization problem: min ํ โโ + min ํโN โ๏ธ โฅ ํ ํ โ ํ ํ โฅ 2 H +โฅ ํ ํ โ ํ ํ โฅ 2 H ํธ ํ โโฅ ํ ํ โฅ H โโฅ ํ ํ โฅ H +โฅ ํ ํ โฅ H s.t ํ ํ = ํ ํ , โฅํโฅ H = 1 , ํ > 0(8) 7.1.1. Optimal Direction We first focus on the inner optimization problem of (8). Expanding the squared numerator of the cost functional reveals that for a fixed magnitudeํ >0, minimizing the cost inํis equivalent to 13 Hilbert Operator for Progressive Encoding (HOPE) maximizing the alignment ํ, ํ ํ + ํ ํ H inํ. To solve the latter, we enforce the realizabilityํโN and unit-normโฅํโฅ H =1 constraints by decoupling the input and output parameters, yielding the parametric form: ํ= ฮจ(ํยท ฬํ) โ๏ธ ํพ(ํ, ํ) ํ .(9) Substituting this parametric form into the unconstrained alignment objective and distributing the Hilbert inner product via the kernel identity isolates the output directionํ. By the Cauchy-Schwarz inequality, the optimalํ โ must align with ร ํโํ,ํ ํพ(ํ โ , ฬํ ํ in )ํ ํ out . Substituting this optimalํ โ back into the objective simplifies the alignment inner product to the Euclidean norm of that sum, yielding the final objective for the optimal ํ โ (Theorem C.7): ํ โ = ร ํโํ,ํ ํพ(ํ โ , ฬํ ํ in )ํ ํ out โฅ ร ํโํ,ํ ํพ(ํ โ , ฬํ ํ in )ํ ํ out โฅ , ํ โ = argmax โฅํโฅ=1 โฅ ร ํโํ,ํ ํพ(ํ, ฬํ ํ in )ํ ํ out โฅ โ๏ธ ํพ(ํ, ํ) s.t. ํพ(ํ, ํ) > 0 (10) The above optimization 7 inํgenerally lacks a closed-form solution due to the non-linear nature of the kernelํพ. To maintain computational tractability, we introduce an approximation scheme that reduces the objective to an eigenvalue problem. Our approximation assumes that for any unit vectorํand non-zeroํ, the kernel factors asํพ(ํ, ํ)= โฅํโฅํ D ํ, ํ โฅํโฅ E for some angular function ํ:[โ1,1] โ โbounded byํ(ํ) โค1. Additionally, we requireํ(1)= ํ โฒ (1)andํ(1) >0. These conditions naturally hold for all PH-1 functions (piecewise linear with a single knot at the origin), e.g., ReLU, Leaky-ReLU; see Propositions C.8 to C.10. For highly correlated neuron pairings, the optimal parent directionํaligns closely with its children, pushing their cosine similarityํโ D ํ, ฬํ in โฅ ฬํ in โฅ E toward 1. Expandingํ(ํ)to first order aroundํ=1 and applying theํ(1)= ํ โฒ (1)identity cancels the constant terms, yielding the linear approximationํ(ํ) โ ํํ(1). While this degrades for unaligned vectors, the phase-check provided later in the section corrects anti-alignment by flipping the sign ofํ, ensuring the optimization trajectory remains safely within this linear domain. Applying the linear approximation to the numerator of the objective, and defining the constant matrixํจโ ํ ํ out ( ฬํ ํ in ) ํ + ํ ํ out ( ฬํ ํ in ) ํ , the summation factors neatly: ร ํโํ,ํ ํพ(ํ, ฬํ ํ in ) ํ ํ out โ ํ( 1)ํจํ. Conversely, the denominator requires no approximation; becauseโฅํโฅ=1, self-alignmentํ=1 evaluates toํพ(ํ, ํ)= ํ(1). Substituting these into the original optimization problem gives: b ํ= arg max ํ โฅํ(1)ํจํโฅ โ ํ(1) s.t. โฅํโฅ=1. Becauseํ(1) >0, the scalars pull out. Dropping these constants and squaring the strictly non-negative objective simplifies the unconstrained problem to a standard quadratic form: b ํ= arg max ํ ํ ํ ํจ ํ ํจํs.t. โฅํโฅ=1. The optimal direction b ํis simply the principal eigenvector ofํจ ํ ํจ. While explicitly constructing this ambient matrix is computationally prohibitive,ํจis fundamentally rank-2. Restricting the eigendecomposition to this rank-2 subspace bypasses the ambient dimension entirely, yielding the principal eigenvector via a fast closed-form solution. Determining the Sign ofํ. For PH-1 activations, the kernel is sign-sensitiveํพ(ํ, ฬํ in )โ ํพ(โํ, ฬํ in ). However, because our linearization approximation relies on the leading eigenvector b ํ ofํจ ํ ํจ, we only recover the solution up to a sign ambiguity. We resolve this by evaluating both candidate polarities ยฑ b ํ in the exact, non-linearized objective (10): ํ correct = argmax ํโ b ํ,โ b ํ ร ํโํ,ํ ํพ(ํ, ฬํ ํ in )ํ ํ out โ๏ธ ํพ(ํ, ํ) .(11) 7 By the PH-1 property ofฮจ, the mappingํโฆโ ํพ(ํ, ํ)is homogeneous, implying that the objective is invariant to the transformation ํโ ํํ for any ํ > 0. We arbitrarily enforce โฅํโฅ= 1 to keep the problem well-posed. 14 Hilbert Operator for Progressive Encoding (HOPE) 7.1.2. Optimal Scale Recall from (9) that the unit-norm directionํ โ Nis parameterized by unit vectorsํandํas ํ= ฮจ(ํยท ฬํ) โ ํพ(ํ,ํ) ํ . Substitutingํ= ํ ํinto the merging cost(8)and defining constantsํโ โฅ ํ ํ โฅ 2 H +โฅ ํ ํ โฅ 2 H , ํโ ํ, ํ ํ + ํ ํ H , andํธ rem โ ํธ ํ โโฅ ํ ํ โฅ H โโฅ ํ ํ โฅ H , minimizing the squared objective reduces to the 1D problemํ โ = argmin ํ >0 2ํ 2 โ2ํํ +ํ (ํ +ํธ rem ) 2 . Setting the derivative with respect toํ to zero yields the unique minimizerํ โ = ํ+ํํธ rem 2ํธ rem +ํ . This solution is also stable. By definition, the residual capacityํธ rem โฅ0, and the prior phase-check ensures the alignment in function spaceํ >0. Thus, the denominator is strictly positive, guaranteeing a unique global minimum in the positive domain (simplifying cleanly toํ โ = ํ/ํin the event of a total layer collapse whereํธ rem =0). Once the optimal scaleํ โ >0 is determined, the parent neuron is fully characterized as shown below. Optimal Parent Neuron ํ โ ํ ( ฬํ)= ํ โ ํ โ ( ฬํ) , ํ โ ( ฬํ)= ฮจ(ํ c ยท ฬํ) โ๏ธ ํพ(ํ c , ํ c ) ํ โ , ํ โ = โฅ ํ ํ โฅ 2 H +โฅ ํ ํ โฅ 2 H + ํธ rem ํ โ , ํ ํ + ํ ํ H 2ํธ rem + ํ โ , ํ ํ + ํ ํ H (12) ํ c = argmax ํโ b ํ,โ b ํ โฅ ร ํโํ,ํ ํพ(ํ, ฬํ ํ in )ํ ํ out โฅ โ๏ธ ํพ(ํ, ํ) , ํ โ = ร ํโํ,ํ ํพ(ํ c , ฬํ ํ in )ํ ํ out โฅ ร ํโํ,ํ ํพ(ํ c , ฬํ ํ in )ํ ํ out โฅ (13) b ํ= arg max โฅํโฅ=1 ํ ํ out ( ฬํ ํ in ) ํ + ํ ํ out ( ฬํ ํ in ) ํ ํ .(14) 7.2. From Hilbert Space to Physical Parameters This section bridges the abstract function space and the physical parameter space by mapping the mathematical operator derived inHback into physical parameters. This parameter recovery is only necessary for merging. For pruning, the projection target is simply the null operator 0, which leads toํ(ํ)=0; this is trivially realized by zeroing out the neuronโs incoming weights, outgoing weights, and BN parameters. However, deploying the parent neuronํ โ ํ โHderived in (12) requires determining the physical parameters (weightsํ raw ํ , ํ ํ , ํ ํ,out and BN statisticsํฝ ํ , ํพ ํ , ํ ํ , ํ ํ ) that will configure the forward pass to reproduce its targeted non-zero activation profile. 7.2.1. Input/Output Scaling To form a standard realizable neuron as described in (7), we equateํ โ ํ ( ฬํ)= ํ โ out ฮจ( ฬํ โ in ยท ฬํ)and then specify the parameters ฬํ โ in andํ โ out . Because the PH-1 activationฮจexhibits scale symmetry, the amplitudeํ โ / โ๏ธ ํพ(ํ โ , ํ โ )can be factored into arbitrary input and output scales ฬํ โ in = ํ in ํ โ and ํ โ out = ํ out ํ โ , for anyํ in , ํ out โฅ0 satisfyingํ in ํ out = ํ โ / โ๏ธ ํพ(ํ โ , ํ โ ). While any factorization yields the same mappingX โ โ ํ , amplitude distribution impacts fine-tuning dynamics. To preserve the original layerโs balance, we define the subspace Frobenius ratioํ ํน โ โฅํพ in โฅ ํน /โฅํพ out โฅ ํน , whereํพ in =[ ฬํ ํ in | ฬํ ํ in ] andํพ out =[ํ ํ out | ํ ํ out ]. Constraining the parent neuron to this ratio requiresโฅ ฬํ โ in โฅ 2 /โฅํ โ out โฅ 2 = ํ ํน . Sinceํ โ andํ โ are unit vectors,ํ in /ํ out = ํ ํน . This uniquely determines the scale factors, yielding the final parameters: ฬํ โ in = โ ํ โ ํ ํน ยท ํพ โ1/4 self ํ โ , ํ โ out = โ๏ธ ํ โ ํ ํน ยท ํพ โ1/4 self ํ โ , ํพ self โ ํพ(ํ โ , ํ โ ) .(15) 15 Hilbert Operator for Progressive Encoding (HOPE) 7.2.2. Raw Input and BN Parameters While the Hilbert space formulation operates entirely on the effective input parameters ฬํ in โ (ํ eff in , ํ), realizing the physical network requires recovering the underlying physical parameters:ํ raw , ํฝ, ํพ, ํ, andํ. Since the parent direction b ํlies within the 2D subspace spanned by the augmented children, there exist projection coefficients ํ 1 and ํ 2 that produce the effective parameters: ํ eff ํ,in = ํ 1 ํ eff in,ํ + ํ 2 ํ eff in,ํ and ํ ํ = ํ 1 ํ ํ + ํ 2 ํ ํ .(16) By mapping these coefficients through the pre-activation distributions of the children, we can deduce the required BN statistics for the parent neuron. Because the physical BN equations form an under- constrained system, we resolve the ambiguity by anchoring the variance such thatํ 2 ํ = max(0, ํพ 2 ํ โํ). As rigorously derived in Appendix D, this anchoring yields a closed-form recovery of all physical parameters. For the active regime ํพ 2 ํ โฅ ํ, these evaluate to: ํ raw in,ํ = ํ eff in,ํ , ํ ํ = ํ 1 ํฝ ํ + ํ 2 ํฝ ํ โ ํ ํ (17) ํฝ ํ = ํ 1 ํฝ ํ + ํ 2 ํฝ ํ , ํ ํ = ํพ ํ = โ๏ธ ํ 2 1 ํพ 2 ํ + ํ 2 2 ํพ 2 ํ + 2ํ 1 ํ 2 |ํพ ํ ||ํพ ํ |หํ ํํ .(18) whereหํ ํํ is from (4). Note thatํ ํ โ ํพ ํ is an approximation that assumes the numerical stability constantํis negligible. The exact boundary-safe formulationํ 2 ํ = max(0, ํพ 2 ํ โ ํ)and the edge case for inactive featuresํพ 2 ํ < ํare deferred to Appendix B.5. Furthermore, Appendix D provides the full step-by-step derivation, along with a proof demonstrating that the physical forward pass acts as a self-correcting mechanism that ensures the networkโs mapping remains invariant to the sign of the recovered scale ํพ ํ . 8. Block Eviction This section expands the granular compression costJ bound established in Section 6 to a new macro- level operation: block eviction. Focusing on residual blocks in architectures like ResNet-50, we extend the previously developed continuous integral to evaluate block eviction alongside granular operations within a single, unified mathematical framework. Consider the canonical residual block, which processes an input representationํ(capitalized to distinguish it from the flattened vectorํ) through a three-stage mapping pathwayํน(ํ). This pathway sequentially applies weight parametersํ 1 ,ํ 2 , andํ 3 , and the result is added to a skip connection to yield the final pre-activationํ= ํ+ ํน(ํ). We define Block Eviction as forcingํน(ํ) โ0, and thus collapsing the block into a pure identity mapping ํ= ํ (see Figure 3). Targeted for Block Eviction: ํน(ํ) โ 0 ํ ํ 1 1x1 Conv (Reduction) BN ReLU ํ 2 3x3 Conv (Spatial) BN ReLU ํ 3 1x1 Conv (Expansion) BN + ReLU ํ ํ Skip Connection (Preserves ํ= ํ) Residual Function ํน(ํ) Figure 3|Illustration of the canonical ResNet V1 residual block and its eviction process. Eviction forces the internal pathwayํน(ํ) โ0, collapsing the block into a pure identity pre-activationํ= ํ. 16 Hilbert Operator for Progressive Encoding (HOPE) 8.1. Motivation A dedicated macro-level operation is required because standard granular pruning cannot remove the blockโs final layerํ 3 . The output dimensionality ofํน(ํ)must match the skip connectionํfor element-wise addition. Consequently, granular compression can only deplete the internal layers ํ 1 ,ํ 2 , which leaves the output channels ofํ 3 locked at their ambient size. Leaving a residual pathway active under these conditions creates two issues: โข Model Generalization: Whenํ 1 andํ 2 are heavily depleted, the pathway functionally reduces to injecting an uncalibrated BN effective biasํต eff into the skip connectionํ= ํ + ํต eff . This shifts downstream feature maps out of their calibrated domain, often causing catastrophic ReLU clipping and irreversible information loss when going from ํ to ํ. โข Execution Efficiency: Retaining the massiveํ 3 parameter tensor simply to process a negligible subspace violates the core objective of compression. Block eviction resolves both issues by projecting the pathwayํน(ํ)to the null operator. By yielding a pure identity mappingํ= ํ, we avoid uncalibrated bias injection and leverage the fact that residual architectures are inherently designed to be robust to identity mappings (e.g., standard ํพ=0 initialization practices) [Goyal et al., 2017,He et al., 2016]. Full mathematical details of this degradation are provided in Appendix F. 8.2. The Unified Macro Cost J evict To evaluate this macro-operation within our framework, we must expand our definition of layer state. To see why, observe that Axiom 2 imposes an infinite cost penalty on projecting an entire layer to zero to prevent disconnecting the network graph. However, this penalty creates an artificial barrier here, as the parallel identity mapping preserves overall connectivity of the block and keeps it alive. To account for this skip pathway, we formulate a macroscopic stateฮฉ (ํ) โ (ฮฆ (ํ) ,I)that couples the targeted internal layerฮฆ (ํ) with the ambient skip connectionI. The skip connection provides a parallel survival capacityํธ identity that keeps the mathematical projection stable. As rigorously derived in Appendix F, integrating the continuous capacity cost over this macro-state and applying a linear upper bound to safely govern massive discrete architectural leaps yields a closed-form distortion criterion. For a standard residual bottleneck comprising two internal convolution layersํ โ 1,2, the total macroscopic distortion is the linear sum of their independent projection bounds: J evict = 2 โ๏ธ ํ=1 J layer (ฮฉ (ํ) ํ ,ฮฉ (ํ) ํ )= 2 โ๏ธ ํ=1 ํ (ํ) active ํธ (ํ) active ํธ identity ! .(19) Here,ํ (ํ) active andํธ (ํ) active represent the active operator count and surviving capacity of internal layer ํ, respectively. The parallel survival capacity evaluates to the expected RMS energy of the identity operators conditioned by the preceding BN layer: ํธ identity = ร ํ amb ํ=1 โ๏ธ ํพ 2 ํ + ํฝ 2 ํ . ResNet Block Eviction Cost J evict = ร 2 ํ=1 ํ (ํ) active ํธ (ํ) active ร ํ amb ํ=1 โ๏ธ ํพ 2 ํ + ํฝ 2 ํ .(20) 17 Hilbert Operator for Progressive Encoding (HOPE) 9. Balancing Compression and Distortion All cost functionals discussed thus far (J prune andJ merge for granular reductions, andJ evict for block evictions) measure the projection error incurred when transitioning from a given state to a reduced state. However, rate-distortion theory establishes that distortion alone cannot fully characterize a lossy compression scheme: lower signal distortion requires a higher bit count, while stronger compression inevitably increases distortion. To balance these competing objectives, we aim to minimize total distortion under a fixed bit count budget. Progressive compression is therefore formulated as a trajectory planning problem within the action space. The goal is to craft a sequence of compression operations that yields a final model satisfying the allowable bit budget while minimizing the total accumulated distortion along the trajectory. Solving this represents a highly complex planning problem due to two primary challenges: 1. Dynamic State Dependency: The cost of an action changes continuously as the model transitions between states. For example, pruning a single neuron shrinks the layerโs residual capacity, which instantaneously alters the costJof subsequent operations, such as pruning another neuron or evicting an entire block. Consequently, the mathematical cost landscape is constantly shifting. 2. Mutually Exclusive Actions: The action space contains complex combinatorial dependencies. If the optimizer merges Neuron A with Neuron B, independent actions like โPrune Aโ or โMerge A with Cโ become permanently invalid. For computational tractability, we must relax these constraints. At each iteration, we temporarily assume all currently admissible operations will remain valid for future iterations, ignoring their mutually exclusive nature. While this generates a complete theoretical action sequence, executing the full trajectory would introduce compounding errors in both state transitions and capacity counts. Instead, we adopt a receding-horizon strategy [Camacho and Bordons, 2013,Bertsekas, 2012]: we compute the optimal sequence, but execute only the immediate next action. Then we physically update the network and then re-evaluate all admissible functions from scratch. This single-step execution acts as an inherent auto-correction mechanism that ensures adherence to constraints over each short-term step. Formally, letA= 1,2, . . . , ํพdenote the set of all admissible compression operations at the current encoding iteration, encompassing all feasible granular and macro operations. Each actionํ incurs a distortion penaltyJ ํ and releasesฮํ ํ parameters (see Appendix B.2 for details on computing ฮํ). Assuming a standard fixed-precision representation (e.g., 32-bit floating-point), bit reduction is proportional to parameter reduction. This direct scaling allows us to express the allowable budget directly in terms of the parameter footprint. We frame this optimization as: (ํ โ 1 ,ยท , ํ โ ํพ )= arg min ํ 1 ,ยท ,ํ ํพ ํพ โ๏ธ ํ=1 ํ ํ J ํ s.t. ํพ โ๏ธ ํ=1 ํ ํ ฮํ ํ โฅ ํ 0 โ ํ budget , โํ ; ํ ํ โ 0, 1 . (21) whereํ 0 is the initial parameter count andํ budget is the maximum allowable parameter footprint for the final model. This formulation is a discrete knapsack problem, which is well-known to be NP- Hard [Karp, 1972,Garey and Johnson, 1979]. We resolve this using a continuous relaxation heuristic, replacing the binary constraintํ ํ โ 0,1with a continuous bound 0โค ํ ํ โค1. This transforms the objective into a continuous knapsack problem (a specific class of linear programming) that admits a highly efficient analytical solution. As established by Dantzig [Dantzig, 1957], the exact optimal solution is found greedily: candidates are sorted by their distortion rate (DR), defined as the cost-to-capacity ratioJ ํ /ฮํ ํ , and assignedํ ํ =1 in ascending order until the budget constraint is 18 Hilbert Operator for Progressive Encoding (HOPE) saturated. Because our receding-horizon framework executes only the single next action, the problem reduces to selecting the operation with the minimal DR: ํ โ = argmin ํโA J ํ ฮํ ํ .(22) While the receding-horizon strategy mitigates the dynamic dependency ofJ, the continuous knapsack solver still requiresฮํto satisfy Dantzigโs Axiom of Item Independence: the weight of one item cannot depend on the selection state of another. Particularly in our problem, evaluating operations using the dynamically shrinking live parameter footprintฮํviolates this axiom because adjacent layers share weight matrices; pruning a neuron physically shrinks theฮํ of its neighbors. A naive optimization using this liveฮํtriggers a failure mode: as a layer is compressed, the expected DR of neighboring structures artificially inflates. This repels the optimizer and may trap the architecture in a fragmented state that prevents the removal of contiguous blocks. Decoupling parameter yield from dynamic state via the static surrogateฮํ init ํ restores item independence and avoids this failure mode: Action Selection Criterion ํ โ = argmin ํโA J ํ ฮํ init ํ .(23) 10. The Encoding Loop With the optimal action selection now formally defined, we execute progressive encoding as a greedy dynamical system. At a high level, the algorithm continuously identifies the optimal actionํ โ offering the lowest DRJ ํ /ฮํ init ํ , performs a localized recalculation exclusively for the modified structures (e.g., a newly generated parent neuronํ โ ํ ) and their immediate neighbors, decrements the relevant dimension count, and repeats. Specifically, the process operates in the following three phases and terminates once the target physical parameter budget is reached or no admissible compression operations remain: 1. Initialization: Before compression begins, the algorithm precomputes and caches the individual capacities of all neurons, the pairwise geometric cross-capacities of all valid merging pairs, and the total initial capacity of every layer (establishing the starting value for ํธ rem ). 2. The Greedy Scan: At each iteration, the algorithm scans allํฟlayers (each containing roughly ํactive neurons) to find the single optimal compression actionํ โ yielding the lowest DRJ ํ /ฮํ init ํ . For pruning, evaluating every individual candidate across the network requiresO(ํฟยท ํ)operations. For merging, evaluating every valid pair requires checking ํ(ํโ1) 2 combinations per layer, leading to O(ํฟยท ํ 2 )operations. Because querying the cachedJfor each candidate takesO(1)time 8 , the total computational complexity to find the optimal action at any step is bounded by the pairwise merge evaluations atO(ํฟยท ํ 2 ). 3. Localized Update: Once the globally optimal action is identified and executed, the network state must be synchronized. The algorithm decrements the layerโsํธ rem by the capacity flux removed by the operation, and decrements the neuron countํ. If the action was a merge, the algorithm also 8 As established in the practical notes of Section 6, the distortion costJrelies on local variables: the capacity of the targeted neurons and the remaining capacity of their specific layer. Because evaluatingJdoes not require querying the global network state, calculating the DR of any individual prune, merge, or block eviction operation isO(1). 19 Hilbert Operator for Progressive Encoding (HOPE) computes the capacity of the newly generated parent neuronํ โ ํ and calculates the cross-capacities as well as optimal projection vectors) between this new parent and theํโ1 surviving neighbors in its layer. These updated constants are injected into the cache, guaranteeing that the evaluation ofJ during subsequent greedy scans remainsO(1). This limits the network state recalculation to anO(ํ) local update. 11. Proof-of-Concept Applications 11.1. Model Compression Because its encoding is progressive, any intermediate iteration serves as a valid compressed model, providing users with flexible trade-offs between compression rate and fidelity. Taxonomically, HOPE is a structured method: it eliminates entire neurons rather than zeroing out individual weights. This provides greater practical utility than unstructured pruning, which generates randomly sparse matrices requiring specialized hardware to realize actual computational speedups. We compare HOPE against three structured baselines that eliminate neurons below specific magnitude thresholds:ํฟ 1 - Norm Input Pruning [Liu et al., 2017] (scored by incoming weightํฟ 1 norms);ํฟ 1 -Norm Joint Pruning (scored by concatenated incoming and outgoingํฟ 1 norms); and BN Scale Pruning [Liu et al., 2017] (using the BN scaling factor ํพ as a proxy for importance). Our experiments investigated the rela- tionship between test set accuracy and model density (defined as the ratio of ac- tive to initial neurons across the entire network). For our compression assess- ment, we utilized Kerasโ publicly avail- able ResNet-50 model checkpoint that is pre-trained on ImageNet. As demon- strated in the plot, HOPE yields models with superior accuracy compared to the baselines. 11.2. Cross-Domain Transfer Learning HOPEโs capacity evaluation can be used for resolving the stability-plasticity dilemma in transfer learning. By merging redundancies, we can partition the network into a protected core and a plastic periphery and leverage it for parameter-efficient transfer. 11.2.1. The Stability-Plasticity Dilemma and Current Bottlenecks Intelligent systems face a fundamental challenge: adapting to new domains without erasing founda- tional knowledge, a trade-off known as the Stability-Plasticity dilemma [Grossberg, 1987]. Cog- nitive neuroscience models this via Complementary Learning Systems [McClelland et al., 1995, Kumaran et al., 2016], proposing that the brain insulates a stable, domain-specific core from a plastic periphery, as empirically supported by recent neuroimaging [Billot et al., 2024,Blank et al., 2016, Casto and Fedorenko, 2026]. Specifically, the brain extracts generalizable schemata from noisy state transitions by applying low-dimensional regularization to its representational geometry [Kimmel et al., 2026]. During standard training, deep neural networks self-organize into a similar, albeit noisy and imper- fect, dichotomy [Martin and Mahoney, 2021]. They develop a sparse, load-bearing core surrounded by low-capacity representational slack [Frankle and Carbin, 2019,El Cheairi et al., 2026]. Naively 20 Hilbert Operator for Progressive Encoding (HOPE) exploiting this emergent segregation by freezing the core fails; because layers remain entangled, peripheral updates shift activation flows, causing representational drift. Early solutions like PackNet [Mallya and Lazebnik, 2018] circumvented this using binary masks during inference, but these scale poorly and require a priori task identities. Robust continual learning requires explicit interventions, rather than relying on noisy emer- gent segregation. During source training, regularization [Wen et al., 2016,Scardapane et al., 2017] can amplify the segregation. During downstream adaptation, penalty-based methods like EWC [Kirkpatrick et al., 2017], Synaptic Intelligence [Zenke et al., 2017], and second-order pruning [LeCun et al., 1989, Hassibi and Storkey, 1992,Singh and Alistarh, 2020] prevent drift using locally convex Fisher Infor- mation Matrices (FIMs) or Hessians. However, relying on local approximations makes these algorithms brittle to large domain shifts. Conversely, orthogonal projection methods stop drift by restrict- ing updates to the null-space of previous tasks [Jaeger, 2014,Saha et al., 2021,Zeng et al., 2019, Wang et al., 2021,Yang et al., 2025,HuggingFace Research Team et al., 2026]. However, they re- main computationally expensive due to theO(ํ 3 )operations and source forward passes needed to compute covariance matrices. 11.2.2. DEFT To address the above challenges, we introduce Dispersed Elastic Fine-Tuning (DEFT). Aligning with the Information Bottleneck principle [Tishby et al., 1999], DEFT treats learning as the compression of irrelevant slack space via global Hilbert-Schmidt operators, bypassing local loss curvature and empirical data passes. Leveraging the HOPE framework, DEFT analytically computes each neuronโs capacity inO(ํ)time to partition the network into a Universal Core and a Peripheral Slack. To prevent representational drift, DEFT severs weight projections from the slack to the core prior to transfer. This ensures the core remains frozen while the slack adapts, eliminating the need for inference-time masking or task identities. DEFT governs parameter plasticity through a binary elasticity mapํธ โ 0,1. While prior methods also regulate plasticity [Zhou et al., 2026], their reliance on weight sensitivity leaves them vulnerable to the scaling symmetries that HOPE mitigates. We formalize this by evaluating the pruning cost assigned to each neuron ํ upon its removal during HOPEโs progressive encoding process: J (ํ) prune = ํ (ํ) ยทโฅ ํ ํ โฅ H ํธ (ํ) ํ (24) whereํ (ํ) andํธ (ํ) ํ โ ํธ (ํ) ํ โโฅ ํ ํ โฅ H denote the active neuron count and the remaining layer capacity, at the step neuronํis pruned. To establish a global freezing threshold, we collect the set of all such costs across the entire encoding process, and filter out extinction artifacts resulting from near-zero capacities: C= n J (ํ) prune ํธ (ํ) ํ > ํ o (25) Given a target percentile hyperparameterํ โ [0,100], we compute the thresholdํฝ ํ = Percentile(C, ํ) and the supremumํฝ sup = max(C). For numerical stability against edge capacity regimes, the final locking threshold ํฝ lock is defined: ํฝ lock = ๏ฃฑ ๏ฃด ๏ฃด ๏ฃด๏ฃฒ ๏ฃด ๏ฃด ๏ฃด ๏ฃณ ํฝ ํ if ํฝ ํ โฅ ํ ํฝ sup if ํฝ ํ < ํ and ํฝ sup โฅ ํ 1otherwise (26) 21 Hilbert Operator for Progressive Encoding (HOPE) The elasticity of neuron ํ is formulated as: ํธ ํ = ( 1 if J (ํ) prune < ํฝ lock 0 if J (ํ) prune โฅ ํฝ lock (27) Under this formulation, high capacity neurons essential to the source architectureJ (ํ) prune โฅ ํฝ lock are frozen by ํธ ํ = 0, whereas low-capacity slack neurons are granted high plasticity via ํธ ํ = 1. Dynamic Resolution of Redundancy: Deep networks frequently fragment a single feature across multiple correlated neurons. If we freeze the network based on a static capacity threshold, we incorrectly lock up this redundant volume and deprive the target task of parameter space. As illustrated in Figure 4(a), DEFT resolves this by compressing these redundant features into a single rank-1 parent neuron. This consolidates the foundational source knowledge while releasing the freed child neurons into the plastic slackํธ ํ =1. By transforming redundant copies into uncommitted parameter space, DEFT actively generates capacity for the target task. Frozen CorePlastic Slack ํ 1 ํ 2 ํ 3 ํ Copies Parent Empty Empty MERGE Error โค ร ํถ ํ (a) Dynamic Resolution of Redundancy Layer ํ Layer ํ+ 1 Core Slack Core Slack ํค coreโcore ํค slackโslack Severed at ํก = 0 (b) The Structural Mask Figure 4|The algorithmic mechanisms of DEFT. (a) Redundant features within the frozen core are compressed to generate new elastic target capacity. (b) A structural mask permanently severs cross-connections to protect the core from target-driven drift. Consistency at Initialization (The Structural Mask): To prevent target-driven updates of the plastic slack from corrupting the frozen core, DEFT applies a structural mask at initialization (Figure 4b). It severs all connections pointing from upstream plastic neurons to downstream frozen core neurons. For a weight tensor connecting an upstream layer (with elasticitiesํฌ in ) to a downstream layer (with elasticities ํฌ out ), the mask ํด enforces: ํ ํ,ํ = ( 0 if ํธ in,ํ > 0 and ํธ out,ํ = 0 1 otherwise (28) The initial weights for the target task are thus constrained to ํพ 0 = ํดโ ํพ source . Theoretical Guarantees: We prove in Appendix H that these mechanisms protect the source representation through a layer-to-layer bounding framework. First, the Static Initialization Shock Bound establishes that severing the slack-to-core connections introduces a static error strictly bounded byO(ํ). Second, Dynamic Decoupling guarantees the core experiences zero dynamic interference during target fine-tuning. Because the cross-connections are severed at initialization and their weights are frozen from updating, any drifting signal from the learning slack subset is multiplied by zero, nullifying it before it can penetrate the core. Combined with the bounded projection errors of the merging operation, this framework ensures the cumulative degradation of the source task cannot compound exponentially, remaining anchored to an algorithmically verifiable constant. 22 Hilbert Operator for Progressive Encoding (HOPE) Gradient Scaling: During optimization, the target gradients are element-wise scaled by the downstream neuronโs elasticity ํฌ out (uniformly broadcast across the input channels): ํ ํก = ํฌ out โโ ํพ L target (ํพ ํก )(29) whereํ ํก represents the effective gradient passed to the optimizer state 9 , e.g., SGD with momentum. 11.2.3. Experimental Setup To evaluate the stability-plasticity tradeoff, we rank methods using the H-Score commonly used in continual learning [Qiu et al., 2024,Xie et al., 2025,Islam et al., 2025]. H-Score, defined as the harmonic mean of source retention and target accuracy, heavily penalizes poor performance in either domain. This ensures a high score is achieved only when a model excels on both tasks. We evaluate models pre-trained on multi-class classification tasks derived from the CIFAR-100 dataset [Krizhevsky, 2009]. Each source task is constructed by randomly sampling 4 superclasses, which yields 20 fine-grained categories (5 per superclass). Building the source task around dense semantic clusters rather than sampling 20 arbitrary classes forces the network to learn hierarchical features to distinguish closely related concepts. We then transfer these specialized models to the full 10-class digit classification task in the SVHN dataset (street-level house numbers) [Netzer et al., 2011]. As summarized in Table 1, we benchmark DEFT against the following baseline methods: โขStandard Full FT: The entire pre-trained backbone is unfrozen, allowing the optimizer to alter representations across all layers. While maximizing target plasticity, it is highly susceptible to catastrophic forgetting. โข Head-Only FT (Standard FT): Representing the opposite extreme (linear probing), the pre- trained backbone is completely frozen, acting as a static feature extractor. Only the final linear classification head is optimized. โขPEFT (BN-Tuning): Operating on the premise that spatial feature extraction logic should remain invariant [Frankle et al., 2021], this method applies a binary gradient mask: core convolutional and dense kernels are frozen, while plasticity is isolated entirely to the affine BN parameters (scale ํพ and shift ํฝ) and the newly initialized classification head. โขEWC (Elastic Weight Consolidation): Allows all parameters to update but applies a quadratic penalty constraining parameters deemed critical to the source task [Kirkpatrick et al., 2017]. To accurately lock foundational features, the empirical diagonal Fisher Information Matrix (FIM) is computed strictly over the source training dataset. See Appendix G for the per-example FIM derivation and integration protocol. Table 1 provides a conceptual summary of these methodologies. Full details regarding network architecture, hyperparameter optimization, and reproducibility protocols are provided in Appendix G. 11.2.4. Results and Discussion The final test set evaluations across 4 independent cross-domain trials (20 discrete CIFAR-100โSVHN scenarios) are presented in Table 2. While Standard Full FT achieves the highest target performance 94.09% by freely overwriting network weights, it completely destroys the source representation, crashing source retention to a baseline low of 7.52%. Conversely, Head-Only FT best preserves source knowledge, but its 36.11% target accuracy highlights the severe domain gap; pre-trained features are insufficient to separate SVHN digits linearly. EWC behaves remarkably similarly to 9 Scaling the gradient before the optimizer step prevents velocity drift for frozen parameters. 23 Hilbert Operator for Progressive Encoding (HOPE) Table 1 | Comparative Summary of Transfer Learning Methodologies PropertyFull FT Head-Only PEFT EWC DEFT (Ours) Updates Backbone Featuresโ--โ Mitigates Catastrophic Forgetting-โ Parameter-Specific Modulation---โ Source Data Independenceโ-โ Structure & Redundancy Aware----โ Full FT: it learns the target domain (93.94%) but fails to significantly arrest catastrophic forgetting (6.74%). DEFT successfully bridges this stability-plasticity gap. By routing target gradients into elastic neurons characterized by their low capacity, DEFT captures nearly all the plasticity of Standard Full FT (94.09% vs. 89.79%). Simultaneously, by masking the core foundational features, it halts catastrophic forgetting, retaining 52.14% of the source accuracy. Computing the harmonic mean of the two accuracy values leads to the H-Score of method, in which DEFT significantly outperform all the baselines. Table 2|Cross-Domain Transfer Learning Results (4 Trials, 5 tasks each). Metrics represent Test Set accuracy evaluated at the optimal target validation epoch. All metrics are averaged across all tasks and trials. MethodTarget Acc (SVHN) Source Retention (CIFAR)H-Score DEFT (Ours)89.79ยฑ 0.8452.14ยฑ 5.2965.82ยฑ 3.96 Head-Only36.11ยฑ 2.7963.13ยฑ 4.6245.79ยฑ 2.05 Full FT94.09ยฑ 0.287.52ยฑ 1.6313.88ยฑ 2.84 EWC93.94ยฑ 0.226.74ยฑ 1.7412.54ยฑ 2.99 PEFT81.91ยฑ 0.495.44ยฑ 0.9810.18ยฑ 1.63 12. Acknowledgment We thank Juno Kim, Vaishnavh Nagarajan, Atish Agarwala, Spencer Frei, Lisa Schut, Alan Malek, Gil shamir, and Bruno Mlodozeniec of Google DeepMind for their helpful comments and discussions. 24 Hilbert Operator for Progressive Encoding (HOPE) References [Ainsworth et al., 2023]Ainsworth, S. K., Hayase, J., and Srinivasa, S. (2023). Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations. [Andrei et al., 2023]Andrei, A., Akil, A., Kharas, N., Rosenbaum, R., Josiฤ, K., and Dragoi, V. (2023). Rapid compensatory plasticity revealed by dynamic correlated activity in monkeys in vivo. Nature Neuroscience, 26(11):1960โ1969. [Badrinarayanan et al., 2015]Badrinarayanan, V., Mishra, B., and Cipolla, R. (2015). Understanding symmetries in deep networks. arXiv preprint arXiv:1511.01029. [Bau et al., 2017]Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017). Network dis- section: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541โ6549. [Behrouz et al., 2026]Behrouz, A., Hashemi, F., Javanmard, A., and Mirrokni, V. (2026). Language models need sleep: Learning to self-modify and consolidate memories. [Bertsekas, 2012]Bertsekas, D. P. (2012). Dynamic programming and optimal control: Volume I. Athena scientific. [Billot et al., 2024] Billot, A., Jhingan, N., Varkanitsa, M., Blank, I., Ryskin, R., Kiran, S., and Fedorenko, E. (2024). The language network ages well: Preserved selectivity, lateralization, and within-network functional synchronization in older brains. bioRxiv. [Blalock et al., 2020]Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., and Guttag, J. (2020). What is the state of neural network pruning? In Proceedings of Machine Learning and Systems, volume 2, pages 129โ146. [Blank et al., 2016]Blank, I., Kanwisher, N., and Fedorenko, E. (2016). A functional mri investigation of the language network. Journal of Neurophysiology, 116(4):1968โ1984. [Bricken et al., 2023] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., et al. (2023). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread. [Cai et al., 2020]Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2020). Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169โ13178. [Camacho and Bordons, 2013]Camacho, E. F. and Bordons, C. A. (2013). Model predictive control. Springer Science & Business Media. [Casto and Fedorenko, 2026]Casto, E. and Fedorenko, E. (2026). Cerebellar language hubs: Func- tional segregation of linguistic processing from motor and cognitive domains. Nature Neuroscience, 29(2):145โ158. [Cho and Saul, 2009]Cho, Y. and Saul, L. (2009). Kernel methods for deep learning. In Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C., and Culotta, A., editors, Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc. [Dandi et al., 2025] Dandi, Y., Pesce, L., Zdeborova, L., and Krzakala, F. (2025). The computational advantage of depth in learning high-dimensional hierarchical targets. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 25 Hilbert Operator for Progressive Encoding (HOPE) [Dantzig, 1957]Dantzig, G. B. (1957). Discrete-variable extremum problems. Operations research, 5(2):266โ288. [Delรฉtang et al., 2023] Delรฉtang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. (2023). Language modeling is compression. arXiv. [Diaconis and Freedman, 1984] Diaconis, P. and Freedman, D. (1984). Asymptotics of graphical projections. The Annals of Statistics, pages 793โ815. [Dinh et al., 2017]Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017). Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019โ1028. PMLR. [El Cheairi et al., 2026]El Cheairi, H., Gamarnik, D., and Mazumder, R. (2026). Theoretical com- pression bounds for wide multilayer perceptrons. In 39th Annual Conference on Learning Theory, pages 1โ59. Proceedings of Machine Learning Research. [Engels et al., 2024]Engels, J., Liao, I., Michaud, E. J., Gurnee, W., and Tegmark, M. (2024). Not all language model features are linear. arXiv preprint arXiv:2405.14860. [Entezari et al., 2022]Entezari, R., Sedghi, H., Saund, O., and Neyshabur, B. (2022). The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations. [Fan et al., 2020]Fan, A., Grave, E., and Joulin, A. (2020). Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations (ICLR). [Frankle and Carbin, 2019]Frankle, J. and Carbin, M. (2019). The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [Frankle et al., 2021]Frankle, J., Schwab, D. J., and Morcos, A. S. (2021). Training batchnorm and only batchnorm: On the expressive power of random features in cnns. In International Conference on Learning Representations. [Garey and Johnson, 1979]Garey, M. R. and Johnson, D. S. (1979). Computers and intractability: A guide to the theory of NP-completeness. W. H. Freeman and Company. [Genewein et al., 2026]Genewein, T., Grau-Moya, J., Wenliang, L. K., Orseau, L., and Hutter, M. (2026). Algorithmic compression via pretrained neural networks. Entropy, 28:596. [Goyal et al., 2017] Goyal, P., Dollรกr, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677. [Grossberg, 1987] Grossberg, S. (1987). Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23โ63. [Han et al., 2015] Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, volume 28. [Hanin and Rolnick, 2018] Hanin, B. and Rolnick, D. (2018). How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems, pages 571โ581. 26 Hilbert Operator for Progressive Encoding (HOPE) [Hassibi and Storkey, 1992]Hassibi, B. and Storkey, D. G. (1992). Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 5. [He et al., 2016]He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity mappings in deep residual networks. In European conference on computer vision (ECCV), pages 630โ645. Springer. [He et al., 2026] He, S., Sun, G., Zhang, H., Fu, Y., and Li, A. (2026). Demystifying when pruning works via representation hierarchies. [He et al., 2017]He, Y., Zhang, X., and Sun, J. (2017). Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389โ1397. [Hinton and Van Camp, 1993]Hinton, G. E. and Van Camp, D. (1993). Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, pages 5โ13. [Hooker et al., 2019]Hooker, S., Courville, A., Clark, G., Yannakakis, Y., and Murphy, K. (2019). What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05248. [Houlsby et al., 2019]Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790โ2799. PMLR. [Hu et al., 2022]Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). [Huang et al., 2016] Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. (2016). Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pages 646โ661. Springer. [HuggingFace Research Team et al., 2026] HuggingFace Research Team, von Werra, L., and Man- grulkar, S. (2026). Orthogonal subspace fine-tuning (osf): A unified framework for interference-free peft. arXiv preprint arXiv:2602.04519. [Huh et al., 2024]Huh, M., Cheung, B., Wang, T., and Isola, P. (2024). The platonic representation hypothesis. arXiv preprint arXiv:2405.07987. [Hรคnni et al., 2024] Hรคnni, K., Mendel, J., Vaintrob, D., and Chan, L. (2024). Mathematical models of computation in superposition. arXiv. [Islam et al., 2025] Islam, M., Maโsum, M. A., Pratama, M., and Skrjanc, I. (2025). Latest advance- ments towards catastrophic forgetting under data scarcity: A comprehensive survey on few-shot class incremental learning. arXiv preprint arXiv:2502.08209. [Jacot et al., 2018] Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, volume 31. [Jaeger, 2014] Jaeger, H. (2014). Controlling recurrent neural networks by conceptors. arXiv. [Jaynes, 1957]Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical review, 106(4):620. 27 Hilbert Operator for Progressive Encoding (HOPE) [Karp, 1972]Karp, R. M. (1972). Reducibility among combinatorial problems. In Complexity of computer computations, pages 85โ103. Springer. [Kimmel et al., 2026]Kimmel, D. L., Stachenfeld, K. L., Salzman, C., and Shohamy, D. (2026). Neural representations supporting generalization under continual learning. bioRxiv. [Kirkpatrick et al., 2017] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017). Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521โ3526. [Kong et al., 2026]Kong, L., Liu, X., Chen, G., Ma, M. Q., Song, X., Sun, Y., Yurochkin, M., Killian, T. W., Salakhutdinov, R., Zhang, K., Xing, E. P., and Liu, Z. (2026). From reasoning traces to reusable modules: Understanding compositional generalization in language model reasoning. [Krizhevsky, 2009]Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. pages 32โ33. [Kumaran et al., 2016] Kumaran, D., Hassabis, D., and McClelland, J. L. (2016). What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512โ534. [LeCun et al., 1989]LeCun, Y., Denker, J. S., and Solla, S. (1989). Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS). [Lee et al., 2018]Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J. (2018). Deep neural networks as gaussian processes. In International Conference on Learning Representations. [Lee et al., 2021] Lee, N., Ajanthan, T., van de Weijer, J., Chang, P. H. S., and Torr, P. (2021). Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations. [Li et al., 2017]Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. (2017). Pruning filters for efficient convnets. In International Conference on Learning Representations (ICLR) Workshop. [Liu et al., 2019]Liu, H., Simonyan, K., and Yang, Y. (2019). Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR). [Liu et al., 2017]Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736โ2744. [Lopes et al., 2017] Lopes, R. G., Fenu, S., and Starner, T. (2017). Data-free knowledge distillation for deep neural networks. In NIPS Workshop on Machine Learning on the Phone and other Consumer Devices. [Luo et al., 2017]Luo, J.-H., Wu, J., and Lin, W. (2017). Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 5058โ5066. [Ma et al., 2023]Ma, X., Fang, G., and Wang, X. (2023). Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems (NeurIPS). 28 Hilbert Operator for Progressive Encoding (HOPE) [Mallya and Lazebnik, 2018]Mallya, A. and Lazebnik, S. (2018). Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7765โ7773. [Martin and Mahoney, 2021]Martin, C. H. and Mahoney, M. W. (2021). Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1โ73. First presented/arXiv in 2018. [McClelland et al., 1995]McClelland, J. L., McNaughton, B. L., and OโReilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419โ457. [Micaelli and Storkey, 2019]Micaelli, P. and Storkey, A. J. (2019). Zero-shot knowledge transfer via adversarial belief matching. In Advances in Neural Information Processing Systems, volume 32. [Mikolov et al., 2013]Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013. [Molchanov et al., 2019]Molchanov, P., Mallya, A., Tyagi, S., Bourezak, I., and Kautz, J. (2019). Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264โ11272. [Molchanov et al., 2017]Molchanov, P., Tyagi, S., Natsev, A., and Krause, J. (2017). Pruning convo- lutional neural networks for resource efficient inference. In International Conference on Learning Representations. [Morcos et al., 2018]Morcos, A. S., Barrett, D. G., Rabinowitz, N. C., and Botvinick, M. (2018). On the importance of single directions for generalization. In International Conference on Learning Representations. [Moreira et al., 2026]Moreira, G., Marinho, Z., Marques, M., Costeira, J. a. P., and Xiong, C. (2026). Native hierarchical and compositional representations with subspace embeddings. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD โ26). [Mozer and Smolensky, 1988]Mozer, M. C. and Smolensky, P. (1988). Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Touretzky, D., editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann. [Nagel et al., 2019]Nagel, M., Baalen, M. v., Blankevoort, T., and Welling, M. (2019). Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325โ1334. [Neal, 1996] Neal, R. M. (1996). Bayesian learning for neural networks, volume 118. Springer Science & Business Media. [Netzer et al., 2011]Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. [Neyshabur et al., 2017]Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. In Advances in neural information processing systems, volume 30. 29 Hilbert Operator for Progressive Encoding (HOPE) [Neyshabur et al., 2015]Neyshabur, B., Tomioka, R., and Srebro, N. (2015). In search of the real inductive bias: On the role of implicit regularization in deep learning. In International Conference on Learning Representations. [Nguyen et al., 2026]Nguyen, Q., Pham, H. H., Cassi, D., and Bellingeri, M. (2026). Depth fragility and skeletal universality: Decoupling topology and function in deep neural networks. Mathematics, 14(9):1438. [Olah et al., 2020]Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020). Zoom in: An introduction to circuits. Distill. [Park et al., 2023]Park, K., Choe, Y. J., and Veitch, V. (2023). The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. [Pennington et al., 2014]Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532โ1543. [Qiu et al., 2024] Qiu, W., Yin, M., Wang, M., Bartlett, P., and Zanette, A. (2024). Continual learning in the frequency domain. In Advances in Neural Information Processing Systems (NeurIPS). [Renda et al., 2020] Renda, A., Frankle, J., and Carbin, M. (2020). Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations. [Rissanen, 1978]Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5):465โ 471. [Saha et al., 2021]Saha, G., Garg, I., and Roy, K. (2021). Gradient projection memory for continual learning. In International Conference on Learning Representations (ICLR). [Scardapane et al., 2017]Scardapane, S., Comminiello, D., Hussain, A., and Uncini, A. (2017). Group sparse regularization for deep neural networks. Neurocomputing, 241:81โ89. [Scholl et al., 2021]Scholl, C., Rule, M. E., and Hennig, M. H. (2021). The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules. PLOS Computational Biology, 17(10):1โ23. [Shwartz-Ziv and Tishby, 2017] Shwartz-Ziv, R. and Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810. [Singh and Alistarh, 2020]Singh, S. P. and Alistarh, D. (2020). Woodfisher: Efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems, volume 33, pages 18098โ18109. [Singh and Jaggi, 2020] Singh, S. P. and Jaggi, M. (2020). Model fusion via optimal transport. In Advances in Neural Information Processing Systems, volume 33, pages 22045โ22055. [Srinivas and Babu, 2015] Srinivas, S. and Babu, R. V. (2015). Data-free parameter pruning for deep neural networks. arXiv. [Stoica et al., 2024]Stoica, G., Bolya, D., Bales, J., and Hoffman, J. (2024). Zipit! merging models from different tasks without training. In International Conference on Learning Representations. [Tanaka et al., 2020] Tanaka, H., Kunin, D., Yamins, D. L. K., and Ganguli, S. (2020). Pruning neural networks without any data by iteratively conserving synaptic flow. ArXiv, abs/2006.05467. 30 Hilbert Operator for Progressive Encoding (HOPE) [Tatro et al., 2020]Tatro, N., Chen, P.-Y., Das, P., Sattigeri, P., Lai, R., and Huan, Z. (2020). Optimiz- ing mode connectivity via neuron alignment. In Advances in Neural Information Processing Systems, volume 33, pages 15300โ15311. [Tishby et al., 1999]Tishby, N., Pereira, F. C., and Bialek, W. (1999). The information bottleneck method. In The 37th Annual Allerton Conference on Communication, Control, and Computing, pages 368โ377. [Veit and Belongie, 2018] Veit, A. and Belongie, S. (2018). Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3โ18. [Wang et al., 2025]Wang, P., Li, X. L., Yaras, C., Zhu, Z., Balzano, L., Hu, W., and Qu, Q. (2025). Understanding deep representation learning via layerwise feature compression and discrimination. Journal of Machine Learning Research, 26(47):1โ71. [Wang et al., 2021]Wang, S., Li, X., Sun, J., and Xu, Z. (2021). Training networks in null space of feature covariance for continual learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 184โ193. [Wang et al., 2018]Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., and Gonzalez, J. E. (2018). Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409โ424. [Wen et al., 2016]Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016). Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (NIPS). [Xie et al., 2025]Xie, J., Yang, J., Luo, Z., Cao, Y., Gao, Q., Zhang, M., and Hu, W. (2025). AdaDARE-ํพ: Balancing stability and plasticity in multi-modal LLMs through efficient adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [Yang, 2019]Yang, G. (2019). Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes. In Advances in Neural Information Processing Systems, volume 32. [Yang et al., 2025]Yang, M., Zhang, W., and Liu, H. (2025). Lora-null: Directing parameter-efficient adaptation into orthogonal null-spaces. In Proceedings of the International Conference on Machine Learning (ICML). [Yin et al., 2020]Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha, N. K., and Kautz, J. (2020). Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715โ 8724. [Yosinski et al., 2014]Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks? In Advances in neural information processing systems, volume 27. [Yvinec et al., 2021]Yvinec, E., Dapogny, A., Cord, M., and Bailly, K. (2021). Red : Looking for redundancies for data-free structured compression of deep neural networks. arXiv. [Zeng et al., 2019]Zeng, G., Chen, Y., Cui, B., and Yu, S. (2019). Continual learning of context- dependent processing in neural networks. Nature Machine Intelligence, 1(8):364โ372. [Zenke et al., 2017]Zenke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987โ3995. PMLR. 31 Hilbert Operator for Progressive Encoding (HOPE) [Zhang et al., 2025]Zhang, Y., Saxe, A., and Latham, P. E. (2025). Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures. arXiv preprint arXiv:2512.20607. [Zhou et al., 2026]Zhou, X., Zhao, H., and Mehr, N. (2026). Taco: Temporal consensus optimization for continual neural mapping. arXiv preprint arXiv:2602.04516. [Zoph and Le, 2017] Zoph, B. and Le, Q. V. (2017). Neural architecture search with reinforcement learning. In International Conference on Learning Representations. 32 Hilbert Operator for Progressive Encoding (HOPE) Appendix Table of Contents A Hilbert Spaces35 A.1 Introduction: Why a Hilbert Space? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 A.2 The Inner Product: The Ruler of Geometry . . . . . . . . . . . . . . . . . . . . . . . . 35 A.3 The Ambient Space: ํฟ 2 (X, ํ X ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 A.4 The Tensor Product: Splitting Continuous and Discrete Spaces . . . . . . . . . . . . . 36 A.5 Uniqueness and the Total Set Property . . . . . . . . . . . . . . . . . . . . . . . . . . 37 A.6 Neuron Synthesis by Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 A.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 B Implementation Notes38 B.1 Adaptation for Convolutional Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 B.2 Deriving the Parameter Footprintฮํ . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 B.3 Cross-Action Overlap and Uniform Scaling . . . . . . . . . . . . . . . . . . . . . . . . 39 B.4 Computational Complexity and the Decoupled Cache . . . . . . . . . . . . . . . . . . 40 B.5 Numerical Stability and BN Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 41 C Main Paper Proofs42 C.1 Layer Transition Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 C.2 Generating Parent Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 C.3 Block Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 D Derivation of Physical BN Parameters50 E Kernel Formulation52 E.1 Pre-Activation Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 E.2 Self-Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 E.3 Cross-Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 F Derivations for Block Eviction56 F.1 Generalization and Execution Degradation in Depleted Blocks . . . . . . . . . . . . . 56 F.2 Derivation of the Unified Macro Cost J evict . . . . . . . . . . . . . . . . . . . . . . . . 56 F.3 Generalization to Non-Residual Architectures . . . . . . . . . . . . . . . . . . . . . . 59 G Reproducibility Protocols for Cross-Domain Transfer59 33 Hilbert Operator for Progressive Encoding (HOPE) G.1 Task Construction and Data Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . 59 G.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 G.3 Base Training Regimen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 G.4 EWC Exact Empirical Fisher Calculation . . . . . . . . . . . . . . . . . . . . . . . . . 61 G.5 Hyperparameter Tuning and Final Evaluation . . . . . . . . . . . . . . . . . . . . . . 61 H Theoretical Guarantees of DEFT62 H.1 Algorithmic Axioms and Partitioning of Neurons . . . . . . . . . . . . . . . . . . . . . 63 H.2 Layer-to-Layer Bounding Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 H.3 Dynamic Resolution of Redundancy via Bounded Trade-off . . . . . . . . . . . . . . . 66 I Algorithms67 34 Hilbert Operator for Progressive Encoding (HOPE) A. Hilbert Spaces This section provides a quick introduction to Hilbert spaces, focusing on the concepts of inner products, completeness, and total sets. We bridge abstract functional analysis with the specific architectural choices of the HOPE framework, demonstrating how theํฟ 2 embedding of neural functions creates a unique and complete geometric environment for optimization over network structures. A.1. Introduction: Why a Hilbert Space? We treat compression operations like pruning and merging as projections. When we replace two neurons with one, we are attempting to find a single element that "represents" a multi-dimensional subspace. To perform this operation rigorously, we need three things: 1. A Space that contains all possible neural identities. 2. A Metric (Inner Product) to measure "closeness" and "alignment." 3. Completeness to ensure that our optimizations actually have solutions. A Hilbert spaceH provides these three pillars. A.2. The Inner Product: The Ruler of Geometry The defining feature of a Hilbert space is the inner product. While a vector space only lets us add and scale elements, an inner product space lets us talk about angles and lengths. Definition A.1 (Inner Product Axioms). An inner product on a vector spaceํoverโis a mapping โจ ยท,ยท โฉ : ํร ํ โ โ satisfying for all ํ, ํ, โโ ํ and ํโ โ: โข Symmetry: โจ ํ, ํ โฉ = โจ ํ, ํ โฉ . โข Linearity: โจ ํํ + ํ, โ โฉ = ํ โจ ํ, โ โฉ + โจ ํ, โ โฉ . โข Positive Definiteness: โจ ํ, ํ โฉ โฅ 0, and โจ ํ, ํ โฉ = 0 โโ ํ= 0. A.2.1. Connection to HOPE: The Expectation Metric In the HOPE framework, we operate on functionsํ:X โ โ ํ . We define our inner product relative to a surrogate distribution ํ X : ํ ํ , ํ ํ H โ ํผ ํโผํ X ํ ํ (ํ) ํ ํ ํ (ํ) (30) Proposition A.1 (Validity of the HOPE Metric). The functional defined in (30) satisfies the inner product axioms. Proof.Linearity and symmetry follow directly from the linearity of the expectation operatorํผand the symmetry of the Euclidean dot product. Positive definiteness is guaranteed because โจ ํ, ํ โฉ = ํผ[ โฅ ํ(ํ) โฅ 2 ] โฅ0. The definiteness โจ ํ, ํ โฉ = 0=โ ํ=0 is satisfied in theํฟ 2 sense (i.e.,ํis zero "almost everywhere").โก 35 Hilbert Operator for Progressive Encoding (HOPE) A.3. The Ambient Space: ํฟ 2 (X, ํ X ) A Hilbert space is more than just an inner product space; it must be complete. In finite dimensions (like โ ํ ), every inner product space is complete. In function spaces, this is not true. Definition A.2 (Completeness). A space is complete if every Cauchy sequence ํ ํ (a sequence where elements get arbitrarily close to each other) converges to an element ํ that is also inside the space. If we worked only with continuous functions, the space would not be complete. For example, a sequence of continuous functions can converge to a step function (which is discontinuous). This would be a disaster for compression, as our "best parent" might not even exist in our space. A.3.1. The ํฟ 2 Embedding To avoid this, HOPE embeds neurons into ํฟ 2 (X, ํ X ), the space of square-integrable functions. โข Energy Bound: Every function inH has finite energy: ํผ[ โฅ ํ โฅ 2 ] <โ. โขClosure: By definition,ํฟ 2 is complete. Every optimization we perform (minimizing the cost functional of compression operations) is guaranteed to have a valid result within the ambient space. A.4. The Tensor Product: Splitting Continuous and Discrete Spaces While we defined our ambient space as vector-valued functionsH= ํฟ 2 (X, ํ X ;โ ํ ), computing the inner product directly in this monolithic space obscures the internal structure of a neural network. HOPE simplifies this by utilizing a tensor product space. A neuronโs operation naturally splits into two phases: 1.The Continuous Input Landscape: The effective input weights (which absorb BN statistics) and the activation function create a continuous scalar landscapeํ ํ (ํ)=ฮจ((ํ eff in,ํ ) ํ ํ+ ํ ํ ). We embed this function into a scalar Hilbert spaceH in โ ํฟ 2 (X, ํ X ; โ). 2.The Discrete Output: This scalar activation is broadcast to the next layer along a finite- dimensional output weight vector ํ out,ํ . We define this output space asH out โ โ ํ . By taking the tensor product of these two spaces, we construct the full ambient space mapping: H H in โH out . Under this formulation, each individual neuron is modeled as a rank-1 Hilbert- Schmidt operator, represented by the outer product of its input function and output vector: ํ ํ โ ํ ํ โ ํ out,ํ (31) A.4.1. Factoring the Metric This tensor structure is what makes HOPE computationally tractable. The inner product of two rank-1 tensors elegantly factors into the product of their individual space inner products: ํ ํ , ํ ํ H = ํ ํ โ ํ out,ํ , ํ ํ โ ํ out,ํ H = ํ ํ , ํ ํ H in ยท ํ out,ํ , ํ out,ํ H out (32) Because ํ ํ , ํ ํ H in is the expected alignment of their non-linear activations over the distribution ํ X , we define this as the kernelํพ(ํ, ํ). This allows us to separate the continuous-functional evaluation from the discrete parameters, reducing the full Hilbert space inner product to: ํ ํ , ํ ํ H = ํพ(ํ, ํ)ยท ํ out,ํ , ํ out,ํ โ ํ (33) 36 Hilbert Operator for Progressive Encoding (HOPE) A.5. Uniqueness and the Total Set Property One might ask: "We defined the inner product for any functionsํ, ํ. But in HOPE, we only ever calculate it for single ReLU neurons. Is that enough to define the whole space?" This is the most critical part of the theory. Definition A.3 (Total Set). A setํ โ His a total set if the set of all finite linear combinations of elements in ํ is dense inH. Ifํis total, then knowing the inner product for every pair inํuniquely determines the inner product for the entire Hilbert space. A.5.1. The Universal Approximation Theorem In the context of HOPE, our "dictionary" of functions is the set of single neuronsN: N= ํ(ํ)= ํ ํํขํก ฮจ( ฬํ in ยท ฬํ)(34) Theorem A.2 (HOPE Uniqueness). The setN is a total set in ํฟ 2 (X, ํ X ). Discussion.By the Universal Approximation Theorem, linear combinations of ReLU neurons can approximate any square-integrable function to arbitrary precision. In the language of Hilbert spaces, this means ํ ํํํ(N) is dense inH. BecauseNis total, the definition of the function correlation kernelํพ(ํ, ํ)= ํผ[ฮจ(ํฆ ํ )ฮจ(ํฆ ํ )]is sufficient to uniquely characterize the metric of the entire ambient space. We do not need a "separate" definition for the inner product of sums of neurons; it is uniquely forced upon the space by the behavior of the single neurons.โก A.6. Neuron Synthesis by Projection Finally, we see why this matters. The generation process relies on finding an Optimal Subspace Projection. When we want to merge neuronsํandํ, their joint contribution[ ํ ํ , ํ ํ ]spans a 2- dimensional tensor subspace. Because a physical parent neuron must produce a single unified output, it is modeled as a constrained rank-1 approximation [ ํ ํ , ํ ํ ]. Rather than a simple orthogonal projection of a sum, HOPE finds the optimal parentํ โ ํ by minimizing the expected Frobenius projection error under the Hilbert-Schmidt norm, scaled by the remaining capacity of the layer: ํ โ ํ = argmin ํ ํ โN โ๏ธ ํ ํ โ ํ ํ 2 H + ํ ํ โ ํ ํ 2 H ํธ ํ โ โฅ ํ ํ โฅ H โ ํ ํ H + ํ ํ H (35) In a Hilbert space, the numerator translates to a rigorous geometric projection distance. Without the Hilbert space structure (inner products and completeness), the concepts of "closest operator" and "functional alignment" would have no meaning. A.7. Summary A summary of the key concepts and their utility in HOPE are provided in Table 3. 37 Hilbert Operator for Progressive Encoding (HOPE) Hilbert ConceptHOPE Implementation Inner Product โจ ํ, ํ โฉ Expected dot product over ํ X Norm โฅ ํ โฅ Functional Capacity (Square root of Signal Energy) Ambient SpaceHTensor product spaceH in โH out OperatorsNeurons modeled as rank-1 Hilbert-Schmidt operators Basis / Total SetN The manifold of realizable single neurons CompletenessGuarantees that the "best parent" is well-defined Table 3 | Mapping of Abstract Hilbert Concepts to the HOPE Framework. B. Implementation Notes B.1. Adaptation for Convolutional Layers Our formalism extends to convolutional networks by defining a โneuronโ as a filter in layerํดproducing feature mapํ. The joint parameter vectorํ ํ = [ํ โค ํ , ํ โค ํ ] โค is constructed by vectorizing the filterโs respective input and output kernels. The input space vectorํ ํ encapsulates the filterโs local receptive field. Assuming layerํดpossesses a weight tensorํฒ ํด โ โ โ ํด รํค ํด รํถ in รํถ out , the vectorํ ํ corresponding to filterํis the flattened spatial sliceํฒ ํด [:,:,:, ํ] โ โ ํ , whereํ= โ ํด ร ํค ํด รํถ in . Conversely, the output space vectorํ ํ captures the filterโs downstream influence on the subsequent layerํต. Because the activation map of filterํacts as theํ-th input channel to layerํต,ํ ํ is formed by extracting and flattening the corresponding slice of the downstream tensorํฒ ํต [:,:, ํ,:] โ โ ํ , whereํ= โ ํต รํค ํต รํถ out,ํต . To allow the surrogate distribution to serve as a location-invariant prior without requiring in- tractable coordinate-specific covariance modeling, we assume spatial stationarity (ergodicity) across the feature map and construct it using the globally averaged BN varianceํ 2 ํ . While boundary zero- padding breaks local stationarity, the high-dimensional spatial aggregation of modern BN buffers absorbs these edge-effects into a into a single global average. B.2. Deriving the Parameter Footprintฮํ As established in Section 9, the parameter footprintฮํquantifies the number of physical parameters removed by a compression action. To preserve Dantzigโs Axiom of Item Independence, this crite- rion uses a static surrogate,ฮํ init , evaluated on the initial network state and decoupled from the dynamically changing network. For any operation, the footprint tracks the total removed parameters: ฮํ init =โฅํ in โฅ 0 +โฅํ out โฅ 0 +โฅํ aux โฅ 0 (36) whereํ in ,ํ out , andํฝ aux denote the input weights, output weights, and auxiliary parameters (e.g., BN parameters) respectively, and โฅยทโฅ 0 counts the number of non-zero elements. B.2.1. Granular Operations For pruning or merging,ฮํ init comprises the incoming weights, outgoing weights, and BN parameters of a single target neuron or filter. For example: โขTransformer MLP: Removing a neuron in an MLP with aํ model โ4ํ model expansion yields ฮํ init = 2ํ model + 1. โขConvolutional Networks: For a filter with spatial dimensionsํปร ํ, the footprint scales with the receptive field and projective kernel:ฮํ init = ํปยท ํยท ํถ in + ํถ out +4, where 4 accounts for the BN parameters ํพ, ํฝ, ํ, ํ 2 . 38 Hilbert Operator for Progressive Encoding (HOPE) Example: Architectural Symmetry in ResNet-50 Evaluatingฮํ init locally reveals symmetries across different layer types. Consider a ResNet-50 bottleneck block with a base channel width ํ and a 4ร expansion ratio: โข 1ร1 Squeeze Layer: Removing one filter deletes 4ํinput connections and 9ํoutput connections to the subsequent 3ร 3 layer. Yield:ฮํ init = 13ํ. โข3ร3 Spatial Layer: Removing one filter deletes 9ํinput connections and 4ํoutput connections to the subsequent 1ร 1 expansion layer. Yield:ฮํ init = 13ํ. Despite differing spatial tensor shapes, the parameter yield per filter in both positions evaluates to 13ํ. This symmetry allows the costJto arbitrate compression across heterogeneous layers without being biased by raw parameter array shapes. B.2.2. Macro Operations: Block Eviction Unlike granular operations that yield incremental savings, block eviction removes entire layers simultaneously. For a residual block with internal convolutional layersํ 1 andํ 2 , and a terminal expansion layer ํ 3 , the static parameter yield expands to: ฮํ init evict =โฅํ 1 โฅ 0 +โฅํ 2 โฅ 0 +โฅํ aux โฅ 0 (37) whereโฅํ 1 โฅ 0 andโฅํ 2 โฅ 0 denote the parameters of the internal layers, andโฅํ aux โฅ 0 counts the BN parametersํ, ํ 2 , ํพ, ํฝacross the entire block, including those ofํ 3 . Note that the weight matrix ํ 3 โ โ ํ amb รํ bottleneck is not explicitly included in this sum. Because removing a filter inํ 2 removes its corresponding outgoing connections inํ 3 , the memory footprint ofํ 3 is naturally accounted for when evaluatingํ 2 . Explicitly adding the ambient dimensionsโฅํ 3 โฅ 0 = ํ amb ร ํ bottleneck would count the same parameters twice within a single macro action. This intra-action double-counting would inflate the parameter footprint of block eviction and artificially lower its Distortion Rate (DR), giving it an unfair advantage over granular operations. B.3. Cross-Action Overlap and Uniform Scaling While we avoid counting parameters twice within a single action (as seen withํ 3 ), evaluating the entire decision space using a static footprintฮํ init introduces an overlap between different competing actions. Letํandํbe targeted neurons in adjacent layersํandํ+1. To preserve Dantzigโs Axiom of Item Independence, an action targeting layerํmust not alter the state variables used to evaluate layerํ+1. Consequently, their shared weightํ (ํ) ํ,ํ is counted independently in both evaluations: ํ (ํ) ํ,ํ โฮํ init ํ and ํ (ํ) ํ,ํ โฮํ init ํ . For any sequence of actionsS, this cross-action overlap overestimates the true number of parameters recovered ร ํโS ฮํ init ํ >ฮํ live S . This overestimation artificially lowers the computed DR compared to the live network state: DR ํ = J ํ ฮํ init ํ < J ํ ฮํ live ํ (38) To correct this approximation without violating item independence, HOPE relies on uniform scaling. Because this cross-action overlap applies systematically across the entire action spaceA(all granular and macro candidates interact with their neighbors), it acts as a uniform scaling factorํผโฅ1 such thatฮํ init ํ โ ํผฮํ live ํ . 39 Hilbert Operator for Progressive Encoding (HOPE) Because the distortion costJis evaluated independently of the parameter counts, andํผapplies uniformly, the relative ordering of the distortion rates is preserved: J ํ ฮํ init ํ < J ํ ฮํ init ํ โโ J ํ ฮํ live ํ < J ํ ฮํ live ํ (39) Since the greedy continuous knapsack solver selectsarg min ํโA DR ํ , this uniform scaling ensures that the theoretical fairness of the optimal action selection remains intact. B.4. Computational Complexity and the Decoupled Cache Evaluating the transition costJefficiently poses a computational challenge because it is inversely coupled to the monotonically decreasing layer capacityํธ rem . In a layer withํneurons, recomputing the non-linear weight-space geometry (e.g., Rank-2 Singular Value Decompositions) for allO(ํ 2 ) candidate pairs every time ํธ rem decreases requiresO(ํ 3 ) execution time. Conversely, cachingJvalues in a standard priority queue via submodular approximations (e.g., Minouxโs Lazy Update) leads to stale estimations. Because the transition cost goes toโas capacity approaches 0, small capacity reductions cause the true costs to spike. A delayed queue would underestimate these costs and cause the optimizer to select sub-optimal actions and potentially collapse the layer. We resolve this bottleneck using anO(1) Decoupled Cache. Decoupling. The computationally expensive optimal projectionsํ โ , ํ โ and the cost components (ํ= โฅ ํ ํ โฅ 2 H + โฅ ํ ํ โฅ 2 H andํ= ํ โ , ํ ํ + ํ ํ H , derived in Section 7.1) depend only on weights and static BN parameters associated with that layer. Since our method does not rely on any cross-layer criterion (such as Fisher Information), we guarantee thatํ โ , ํ โ , ํ, ํremain independent of both the downstream architecture and the dynamic capacity ํธ rem . The only variable dependent onํธ rem is the optimal scalar magnitudeํ โ . At initialization, the framework computesํ โ andํ โ to evaluate and cache only the scalar constantsํandํ. To prevent O(ํ 2 )memory exhaustion, the high-dimensional vectorsํ โ , ํ โ are then discarded. During the greedy search, evaluating the cost of any action requires querying the cached constants and the live remaining capacity ํธ rem = max(ํธ ํ โโฅ ํ ํ โฅ H โโฅ ํ ํ โฅ H , ํ) to compute ํ โ and J analytically inO(1) time: ํ โ = ํ+ ํํธ rem 2ํธ rem + ํ (40) Index Determinism and JIT Generation. We sort the active neurons to evaluate undirected pairs only whereํ < ํ. By enforcing this index determinism, the algorithm guarantees a 100% cache hit rate and makes it viable to abandon priority queues entirely. At every step, the algorithm executes an O(1)global scan over all remaining pairs using the live layer capacity, maintaining mathematically perfect freshness. Once the global minimum is selected, the framework executes a Just-In-Time (JIT) generation, re-evaluating the fast Rank-2 SVD only for the single winning pair (โผ1 ms) to retrieve its ํ โ , ํ โ for physical deployment. Computing the BN Variance. Deploying the JIT-generated parent requires setting its BN varianceํพ 2 ํ . Under the surrogate distributionํ X ,ํพ 2 ํ depends only on the warped correlationหํ ํํ . A naive approach might substitute the simple correlation of the raw weightsํ raw into the variance equation. However, this incorrectly mixes parameter-space metrics with signal-space statistics. Doing so produces incorrect BN moving averages, which miscalibrates the network during the forward pass. The framework avoids 40 Hilbert Operator for Progressive Encoding (HOPE) this by computing the physical variance usingหํ ํํ directly retrieved from the cache. This ensures the deployed parameters maintain the correct statistical behavior without requiring empirical forward passes. B.5. Numerical Stability and BN Parameters Section 7.2.2 derives the mapping from the parent neuronโs effective parameters (ํ eff in,ํ andํ ํ ) to its physical network variables: the raw input weightsํ raw in,ํ and the BN parametersํพ ํ , ํฝ ํ , ํ ํ , ํ 2 ํ . A naive assignment of these physical variables alters the pre-activation signal and breaks the critical equivalenceํฆ ํ =(ํ eff in,ํ ) ํ ํ+ ํ ํ . This mismatch miscalibrates the network and immediately degrades accuracy on the first forward pass. To prevent this, HOPE enforces the exact analytical mapping. However, deploying these formulas in practice requires safely handling numerical bounds, such as preventing negative varianceํ 2 ํ <0 and avoidingNaNin the BN denominator โ๏ธ ํ 2 ํ + ํ . This section details how to resolve these numerical edge cases and ensure the deployed model remains mathematically well-defined and numerically stable. B.5.1. The ํ Boundary Regime and Variance Clamping To resolve the under-constrained BN system, the framework fixes the physical variance asํ 2 ํ = max(0, ํพ 2 ํ โ ํ), whereํis a small stability constant (e.g., 10 โ5 ). For active featuresํพ 2 ํ โฅ ํ, this evaluates smoothly. This allows the denominator of the BN transformation to simplify to |ํพ ํ |. However, as the progressive encoder shrinks the network, the derived parent varianceํพ 2 ํ may occasionally fall below the numerical floorํ. Without themax(0,ยท)operator, enforcingํ 2 ํ = ํพ 2 ํ โ ํ would result in a negative variance, which causesNaNduring inference. The clamping operator safely bounds the physical variance at ํ 2 ํ = 0. Whenํ 2 ํ is clamped to 0, the BN denominator โ๏ธ ํ 2 ํ + ํevaluates to โ ํ. Consequently, the scale factor becomes ํพ ํ / โ ํ. Substituting this into the effective bias equation yields: ํ ํ = ํฝ ํ โ ํพ ํ โ ํ ํ ํ =โ ํ ํ = โ ํ ํพ ํ (ํฝ ํ โ ํ ํ )(41) As the parent neuronโs scaleํพ ํ approaches 0, dividing byํพ ํ causesํ ํ โ โ. To prevent this, the framework bypasses this calculation for inactive features and sets ํ raw in,ํ = 0. B.5.2. Running Variance Offset During Fine-Tuning Setting the initial BN running variance toํ 2 ํ = ํพ 2 ํ โ ํensures the networkโs output is perfectly preserved during inference. However, when the model resumes training for fine-tuning, the actual batch variance computed during the forward pass evaluates toํ 2 batch = ํพ 2 ํ . This introduces a minor mismatch between the empirical batch variance and the stored running variance: ํ 2 batch โ ํ 2 ํ = ํพ 2 ํ โ(ํพ 2 ํ โ ํ)= ํ(42) Because this discrepancy evaluates to the numerical constantํ(typically 10 โ5 ), its impact is negligible. Standard optimizers (e.g., Adam, AdamW) seamlessly absorb thisO(ํ)offset during the initial training steps without destabilizing the network or degrading performance. 41 Hilbert Operator for Progressive Encoding (HOPE) C. Main Paper Proofs C.1. Layer Transition Costs For convenience, we first recall the axioms of the cost J from the main paper. Axioms of the Cost Functional J To ensure a well-posed definition of the costJ, the framework introduces the following three axioms: โข Magnitude Neutrality: J must be scale invariant:โํ > 0 ; J(ํฮฆ ํ , ํฮฆ ํ )=J(ฮฆ ํ ,ฮฆ ํ ). โข Connectivity Preservation:Jmust establish an asymptotic barrier preventing layer extinction: lim ํธ(ฮฆ ํ )โ0 + J=โ. โขInfinitesimal Capacity Dependence:Jmust be additive along continuous paths and driven by the reduction in layer capacity:J(ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 โํ(ฮฆ(ํก)) ยค ํธ(ํก)ํํก, where ยค ํธ(ํก)โ ํํธ(ฮฆ(ํก))/ํํก and ํ(ฮฆ(ํก)) > 0 is a state-dependent density function. Lemma C.1 (Uniqueness of theํฟ 1 Capacity). Letฮฆ=( ํ 1 , . . . , ํ ํ ) โH ํ denote a layer state. Assume the capacity functional ํธ : ร โ ํ=1 H ํ โ โ โฅ0 satisfies: 1. Identity:โํ โH, ํธ(( ํ))= โฅ ํ โฅ H . 2.Symmetry & Separability:โ ํ:โ โฅ0 โ โ(continuous and strictly monotonic) and a function โ such thatโฮฆ โH ํ , ํธ(ฮฆ)= โ ร ํ ํ=1 ํ( โฅ ํ ํ โฅ H ) . 3. Partition Invariance:โํ โH, โํ โ โค โฅ1 , ํธ(( ํ))= ํธ ( ํ/ํ, . . . , ํ/ํ | z ํ times ) . Then ํธ(ฮฆ)= ร ํ ํ=1 โฅ ํ ํ โฅ H . Proof. For a single-neuron stateฮฆ=( ํ), Conditions 1 and 2 imply: ํธ(( ํ))= โ ํ( โฅ ํ โฅ H ) = โฅ ํ โฅ H =โ โโก ํ โ1 on Im(ํ) Thus, the functional simplifies to ํธ(ฮฆ)= ํ โ1 ร ํ ํ=1 ํ( โฅ ํ ํ โฅ H ) . By Condition 3 and the positive homogeneity of the norm ( โฅ ํ/ํ โฅ H = โฅ ํ โฅ H /ํ for ํ โฅ 1): โฅ ํ โฅ H = ํธ ( ํ/ํ, . . . , ํ/ํ | z ํ times ) = ํ โ1 ํ โ๏ธ ํ=1 ํ โฅ ํ โฅ H ํ ! = ํ โ1 ํํ โฅ ํ โฅ H ํ Applying ํ to both sides and substituting ํฅโ โฅ ํ โฅ H โฅ 0 yields: ํ(ํฅ)= ํํ ํฅ ํ โํฅ โฅ 0, โํ โ โค โฅ1 Forํฅ=0,ํ(0)= ํํ(0)=โ ํ(0)=0. For any rationalํ= ํ/ํ >0, substitutingํฆ= ํฅ/ํyields ํ(ํํฆ)= ํํ(ํฆ)= ํํํ(ํฆ)= ํํ(ํํฆ), meaningํ(ํํฅ)= ํํ(ํฅ). Sinceํis continuous, this linearity extends to all ํฅ โ โ โฅ0 , yielding ํ(ํฅ)= ํํฅ for some constant ํ. Since ํ is strictly monotonic, ํโ 0. 42 Hilbert Operator for Progressive Encoding (HOPE) Substituting ํ(ํฅ)= ํํฅ and ํ โ1 (ํฆ)= ํฆ/ํ into the expression for ํธ(ฮฆ) gives: ํธ(ฮฆ)= 1 ํ ํ โ๏ธ ํ=1 ํ โฅ ํ ํ โฅ H = ํ โ๏ธ ํ=1 โฅ ํ ํ โฅ H โก Theorem C.2 (Integral Formulation of Scale-Invariant Cost). Under the Axioms of Magnitude Neutrality, Connectivity Preservation, and Infinitesimal Capacity Dependence, the transition cost along a continuous deformation pathฮฆ:[0,1] โH ํ with boundary conditionsฮฆ(0)=ฮฆ ํ andฮฆ(1)=ฮฆ ํ , is determined as the integral: J(ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 โํ(ฮฆ(ํก)) ยค ํธ(ํก) ํธ(ฮฆ(ํก)) ํํก a (43) whereํธ(ฮฆ(ํก))is the instantaneous capacity, ยค ํธ(ํก) <0 is the rate of capacity reduction, and ํ(ฮฆ(ํก)) > 0 is a scale-invariant factor. a Since the Hilbert norm โฅ ํ โฅ H is non-differentiable atํ=0, this integral is defined over paths where the active capacity remains positive ํธ(ฮฆ(ํก)) > 0. Proof.1. Differential Form: By the Infinitesimal Capacity Dependence axiom, the differential cost along a path is driven by capacity reduction: ยค J(ํก)=โํ(ฮฆ(ํก)) ยค ํธ(ํก) Since active capacity decreases during compression ยค ํธ(ํก) < 0 and the cost rate must be positive ยค J(ํก) > 0, we require ํ(ฮฆ(ํก)) > 0. 2. Magnitude Neutrality: For any scalarํ >0, Magnitude Neutrality requiresJ(ํฮฆ ํ , ํฮฆ ํ )= J(ฮฆ ํ ,ฮฆ ํ ). By Lemma C.1, capacity scales linearlyํธ(ํฮฆ)= ํํธ(ฮฆ), so ยค ํธ(ํฮฆ(ํก))= ํ ยค ํธ(ํก). Integrating over the scaled path yields: โซ 1 0 โํ(ํฮฆ(ํก)) ํ ยค ํธ(ํก)ํํก= โซ 1 0 โํ(ฮฆ(ํก)) ยค ํธ(ํก)ํํก Assuming continuous integrands, since this equality holds for any valid continuous path, the integrands must be identical pointwise: ํ ํ(ํฮฆ) ยค ํธ(ํก)= ํ(ฮฆ) ยค ํธ(ํก)=โ ํ(ํฮฆ)= ํ โ1 ํ(ฮฆ) Thus, ํ is a homogeneous function of degreeโ1. 3. Scale-Invariant Factor: Define ํ(ฮฆ)โ ํ(ฮฆ)ํธ(ฮฆ). Scaling the state by ํ yields: ํ(ํฮฆ)= ํ(ํฮฆ)ํธ(ํฮฆ)=(ํ โ1 ํ(ฮฆ))(ํํธ(ฮฆ))= ํ(ฮฆ) This showsํ(ฮฆ)is scale-invariant. Substitutingํ(ฮฆ)= ํ(ฮฆ)/ํธ(ฮฆ)back into the differential form gives the integral: J(ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 โํ(ฮฆ(ํก)) ยค ํธ(ํก) ํธ(ฮฆ(ํก)) ํํก 43 Hilbert Operator for Progressive Encoding (HOPE) 4. Connectivity Preservation: This axiom mandates an infinite cost barrier against layer extinction: lim ํธ ํ โ0 + J(ฮฆ ํ ,ฮฆ ํ )= โ, whereํธ ํ = ํธ(ฮฆ ํ )andํธ ํ = ํธ(ฮฆ ํ ). Applying the change of variables ํํธ= ยค ํธ(ํก)ํํกand reversing the limits (which absorbs the negative sign sinceํธ ํ < ํธ ํ due to ยค ํธ(ํก) <0) gives: J(ฮฆ ํ ,ฮฆ ํ )= โซ ํธ ํ ํธ ํ โํ(ฮฆ) ํํธ ํธ = โซ ํธ ํ ํธ ํ ํ(ฮฆ) ํํธ ํธ If ํ(ฮฆ) is bounded below by a constant ํ min > 0 along the path to extinction: J(ฮฆ ํ ,ฮฆ ํ ) โฅ โซ ํธ ํ ํธ ํ ํ min ํํธ ํธ = ํ min ln(ํธ ํ )โ ln(ํธ ํ ) Taking the limit as ํธ ํ โ 0 + yieldsโ, satisfying the axiom.โก Relative Differential Cost For any differentiable deformation pathฮฆ:[0,1] โH ํ , the rate of geometric projection cost accumulation ยค J proj (ํก) is defined as: ยค J proj (ํก)โ ํ(ฮฆ(ํก)) ยค ํ (ํก) ํธ(ฮฆ(ํก)) (44) where ยค ํ (ํก)= ยค ฮฆ(ํก) H ํ โฅ 0 is the instantaneous geometric speed of the state vector. Derivation.The scalar capacity rate ยค ํธ(ํก)from Theorem C.2 assigns zero penalty to geometric deformations (e.g., orthogonal rotations) where scalar capacity is conserved ยค ํธ(ํก)=0. To capture structural distortion, we calibrate the criterion along a localized orthogonal path. Consider a compressive path where a single active neuronํ ํ is scaled toward 0 by a decreasing scalarํผ(ํก) โ [0,1] ยค ํผ(ํก) <0, while all other neurons remain static. The capacity isํธ(ฮฆ(ํก))= ํผ(ํก) โฅ ํ ํ โฅ H + ร ํโ ํ โฅ ํ ํ โฅ H , giving ยค ํธ(ํก)= ยค ํผ(ํก) โฅ ํ ํ โฅ H < 0. The geometric speed ยค ํ (ํก) is the norm of the state derivative vector. Since only theํ-th coordinate changes, this vector is 1-sparse, ensuring the norms coincide: ยค ํ (ํก)= ยค ฮฆ(ํก) H ํ = | ยค ํผ(ํก) |โฅ ํ ํ โฅ H =โ ยค ํผ(ํก) โฅ ํ ํ โฅ H =โ ยค ํธ(ํก) Substituting ยค ํ (ํก)=โ ยค ํธ(ํก)into the cost baseline ยค J capacity (ํก)=โํ(ฮฆ) ยค ํธ ํธ yields ยค J(ํก)= ํ(ฮฆ) ยค ํ ํธ for this orthogonal axis. Because the ambient spaceH ํ is isotropic, we define the geometric projection cost ยค J proj (ํก) as this ratio to generalize to arbitrary trajectories.โก Lemma C.3 (Capacity Bound for Correlated Projections). Letํ ํ , ํ ํ โHbe active candidate neurons โฅ ํ ํ โฅ H >0, ํ ํ H >0, and letํ ํ โ Hbe their optimal parent neuron. Letํธ rem โ ํธ ํ โโฅ ํ ํ โฅ H โโฅ ํ ํ โฅ H โฅ 0. Define the functional correlation as ํ ํํ โ โจ ํ ํ ,ํ ํ โฉ H โฅ ํ ํ โฅ H โฅ ํ ํ โฅ H . For the continuous straight-line deformation pathฮฆ(ํก)inH ํ connecting the initial stateฮฆ ํ to the pre-deletion target state ฬ ฮฆ ํ , there exists a correlation thresholdํ โ ( ํ ํ , ํ ํ ) โ (0,1)such that if ํ ํํ โฅ ํ โ , then: ํธ(ฮฆ(ํก)) โฅ ํธ(ฮฆ ํ ) โํก โ [0, 1](45) where ํธ(ฮฆ ํ )= ํธ rem + ํ ํ H is the capacity of the post-deletion terminal stateฮฆ ํ โH ํโ1 . 44 Hilbert Operator for Progressive Encoding (HOPE) Proof.Parameterizing the path asฮฆ(ํก)=(1โ ํก)ฮฆ ํ + ํก ฬ ฮฆ ํ forํก โ [0,1], the targeted neurons transition viaํ ํ (ํก)= (1โ ํก) ํ ํ + ํก ํ ํ andํ ํ (ํก)= (1โ ํก) ํ ํ + ํก ํ ํ . By Lemma C.1 and the triangle inequality โฅ ํข โฅ H + โฅ ํฃ โฅ H โฅ โฅ ํข+ ํฃ โฅ H : ํธ(ฮฆ(ํก)) โฅ ํธ rem + (1โ ํก)( ํ ํ + ํ ํ )+ 2ํก ํ ํ H To prove ํธ(ฮฆ(ํก)) โฅ ํธ rem + ํ ํ H , we require: (1โ ํก)( ํ ํ + ํ ํ )+ 2ํก ํ ํ H > ํ ํ H โํก โ [0, 1](46) We evaluate this in the collinear limitํ ํํ โ1. Letํ ํ = ํฅหํข,ํ ํ = ํฆหํข, andํ ํ = ํงหํขfor a shared unit vector หํข and scalars ํฅ, ํฆ, ํง > 0. The condition simplifies to: (1โ ํก)(ํฅ+ ํฆ)+ 2ํกํง > ํง Since this expression is linear in ํก, its minimum occurs at the boundaries: โข At ํก= 1: 2ํง > ํง, which inherently holds since ํง > 0. โขAtํก=0:ํฅ+ ํฆ > ํง. From the optimal scale derivation (Section 7.1),ํง= ํ+ํํธ rem 2ํธ rem +ํ . In the collinear limit, ํ= ํฅ 2 + ํฆ 2 and ํ= ํฅ+ ํฆ. Thus: ํฅ+ ํฆ > ํฅ 2 + ํฆ 2 +(ํฅ+ ํฆ)ํธ rem 2ํธ rem + ํฅ+ ํฆ (ํฅ+ ํฆ)(2ํธ rem + ํฅ+ ํฆ) > ํฅ 2 + ํฆ 2 +(ํฅ+ ํฆ)ํธ rem 2ํธ rem (ํฅ+ ํฆ)+ ํฅ 2 + 2ํฅํฆ+ ํฆ 2 > ํฅ 2 + ํฆ 2 + ํธ rem (ํฅ+ ํฆ) ํธ rem (ํฅ+ ํฆ)+ 2ํฅํฆ > 0 Since ํฅ > 0, ํฆ > 0, and ํธ rem โฅ 0, this strict inequality unconditionally holds. Since both endpoints satisfy the strict inequality, it holds for allํก โ [0,1]in the collinear limit. Because the Hilbert norm andํ ํ are continuous with respect toํ ํํ , this inequality is preserved in a neighborhood aroundํ ํํ =1. Thus, there exists a thresholdํ โ ( ํ ํ , ํ ํ ) โ (0,1)ensuring the condition for ํ ํํ โฅ ํ โ .โก Theorem C.4 (Discrete Transition Cost Bound). For any structural reduction from an initial stateฮฆ ํ โ H ํ to a terminal stateฮฆ ํ โ H ํโ1 (specifically, pruning a neuron, or merging a correlated pair withํ ํํ โฅ ํ โ ), the continuous projection costJ proj evaluated along the straight-line path to the pre-deletion target ฬ ฮฆ ํ โH ํ is upper-bounded by the discrete proxy J bound : J proj (ฮฆ ํ ,ฮฆ ํ ) โค ํ(ฮฆ ํ ) ํท(ฮฆ ํ , ฬ ฮฆ ํ ) ํธ(ฮฆ ํ ) โก J bound (ฮฆ ํ ,ฮฆ ํ )(47) whereํธ(ฮฆ ํ )is the post-deletion capacity, andํท(ฮฆ ํ , ฬ ฮฆ ํ )= ฮฆ ํ โ ฬ ฮฆ ํ H ํ is the Euclidean distance in the configuration space. Proof. By Definition 44, the cost alongฮฆ(ํก)=(1โ ํก)ฮฆ ํ + ํก ฬ ฮฆ ํ is: J proj (ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 ํ(ฮฆ(ํก)) ยค ํ (ํก) ํธ(ฮฆ(ํก)) ํํก 45 Hilbert Operator for Progressive Encoding (HOPE) Since the active neuron count remains invariant prior toํก=1,ํ(ฮฆ(ํก))= ํ(ฮฆ ํ )almost everywhere on [0, 1] and can be factored out. Next, we bound the dynamic capacity ํธ(ฮฆ(ํก)): โข Pruning: ํธ(ฮฆ(ํก))= ํธ ํ โ ํก โฅ ํ ํ โฅ H . Its minimum is ํธ(1)= ํธ ํ โ โฅ ํ ํ โฅ H = ํธ(ฮฆ ํ ). โข Merging: Lemma C.3 guarantees ํธ(ฮฆ(ํก)) โฅ ํธ(ฮฆ ํ ) for all ํก โ [0, 1]. In both cases,ํธ(ฮฆ(ํก)) โฅ ํธ(ฮฆ ํ ) >0. Substituting 1/ํธ(ฮฆ(ํก)) โค1/ํธ(ฮฆ ํ )into the integral establishes an upper bound: J proj (ฮฆ ํ ,ฮฆ ํ ) โค ํ(ฮฆ ํ ) ํธ(ฮฆ ํ ) โซ 1 0 ยค ํ (ํก)ํํก(48) The geometric speed ยค ํ (ํก)= ยค ฮฆ(ํก) H ํ = ฬ ฮฆ ํ โฮฆ ํ H ํ โก ํท(ฮฆ ํ , ฬ ฮฆ ํ ) is constant. Evaluating the integral yields ํท(ฮฆ ํ , ฬ ฮฆ ํ ), leading to: J proj (ฮฆ ํ ,ฮฆ ํ ) โค ํ(ฮฆ ํ ) ํท(ฮฆ ํ , ฬ ฮฆ ํ ) ํธ(ฮฆ ํ ) โก J bound (ฮฆ ํ ,ฮฆ ํ ) โก Proposition C.5 (Axiomatic Consistency of the Bounded Proxy). The bounded projection costJ bound (ฮฆ ํ ,ฮฆ ํ )= ํ(ฮฆ ํ ) ํท(ฮฆ ํ , ฬ ฮฆ ํ ) ํธ(ฮฆ ํ ) satisfies Axiom 1 (Magnitude Neutrality) and Axiom 2 (Connectivity Preservation). Proof.Axiom 1 (Magnitude Neutrality): For anyํ >0, we haveํ(ํฮฆ ํ )= ํ(ฮฆ ํ )(scale-invariant by Theorem C.2),ํท(ํฮฆ ํ , ํ ฬ ฮฆ ํ )= ํํท(ฮฆ ํ , ฬ ฮฆ ํ ), andํธ(ํฮฆ ํ )= ํํธ(ฮฆ ํ )(linear scaling by Lemma C.1). Thus, ํ cancels out: J bound (ํฮฆ ํ , ํฮฆ ํ )= ํ(ฮฆ ํ ) ํํท(ฮฆ ํ , ฬ ฮฆ ํ ) ํํธ(ฮฆ ํ ) =J bound (ฮฆ ํ ,ฮฆ ํ ) Axiom 2 (Connectivity Preservation): For an initial state withํธ(ฮฆ ํ ) >0, we evaluatelim ํธ(ฮฆ ํ )โ0 + J bound (ฮฆ ํ ,ฮฆ ํ ). By the definitions of pruning and merging,ํธ( ฬ ฮฆ ํ ) โค2ํธ(ฮฆ ํ ). 10 Thus,ํธ(ฮฆ ํ ) โ0 + =โ ํธ( ฬ ฮฆ ํ ) โ0. Because norms are equivalent on a finite-dimensional space, โฅ ฮฆ โฅ H ํ โค ํธ(ฮฆ) โค โ ํ โฅ ฮฆ โฅ H ํ , which implies ฬ ฮฆ ํ H ํ โ 0. Applying the reverse triangle inequality gives: ํท(ฮฆ ํ , ฬ ฮฆ ํ )= ฮฆ ํ โ ฬ ฮฆ ํ H ํ โฅ โฅ ฮฆ ํ โฅ H ํ โ ฬ ฮฆ ํ H ํ Taking the limit as ฬ ฮฆ ํ H ํ โ 0 yields a positive lower bound: lim ํธ(ฮฆ ํ )โ0 + ํท(ฮฆ ํ , ฬ ฮฆ ํ )= โฅ ฮฆ ํ โฅ H ํ โฅ 1 โ ํ ํธ(ฮฆ ํ ) > 0 Since the numerator is bounded below by a positive constant, dividing byํธ(ฮฆ ํ ) โ0 + strictly diverges toโ.โก 10 For pruning, ํธ( ฬ ฮฆ ํ )= ํธ(ฮฆ ํ ). For merging, ํธ( ฬ ฮฆ ํ )= ํธ rem + 2 ํ ํ H โค 2(ํธ rem + ํ ํ H )= 2ํธ(ฮฆ ํ ). 46 Hilbert Operator for Progressive Encoding (HOPE) Corollary C.6 (Locality of the Projection Error). For a structural reduction modifying a localized subset of neuronsS โ 1, . . . , ํ(e.g.,|S|=1 for pruning,|S|=2 for merging), the discrete projection bound evaluates over the perturbed subspace: J bound (ฮฆ ํ ,ฮฆ ํ )= ํ(ฮฆ ํ ) โ๏ธ ร ํโS ํ (ํ) ํ โ ฬ ํ (ํ) ํ 2 H ํธ(ฮฆ ํ ) (49) where ฬ ํ (ํ) ํ are the components of the pre-deletion target state ฬ ฮฆ ํ โH ํ . Proof. By Theorem C.4, the Euclidean distance expands as: ํท(ฮฆ ํ , ฬ ฮฆ ํ )= v t โ๏ธ ํโS ํ (ํ) ํ โ ฬ ํ (ํ) ํ 2 H + โ๏ธ ํโS ํ (ํ) ํ โ ฬ ํ (ํ) ํ 2 H For any unperturbed coordinate ํโS, ํ (ํ) ํ = ฬ ํ (ํ) ํ , meaning the second summation vanishes. The terminal capacityํธ(ฮฆ ํ )is also computed locally viaํธ(ฮฆ ํ )= ํธ(ฮฆ ํ )โ ร ํโS ํ (ํ) ํ H + ร ํโS new ํ (ํ) ํ H , whereS new represents newly generated components (e.g., ํ ํ ). Because|S|and|S new |depend solely on the localized operation, evaluating J bound requiresO(1) operations, given ํธ(ฮฆ ํ ) is cached.โก C.2. Generating Parent Neuron Theorem C.7 (Exact Optimal Parent Direction). For a fixed magnitudeํ >0, the inner optimization for merging neurons ํ ํ , ํ ํ โN: ํ โ = argmin ํโN โฅ ํ โฅ H =1 โ๏ธ โฅ ํ ํโ ํ ํ โฅ 2 H + ํ ํโ ํ ํ 2 H ํธ ํ โ โฅ ํ ํ โฅ H โ ํ ํ H + ํ (50) is minimized by the parameterized function: ํ โ = ฮจ(ํ โ ยท ฬํ) โ๏ธ ํพ(ํ โ , ํ โ ) ํ โ (51) where the unit vectors ํ โ โ โ ํ+1 and ํ โ โ โ ํ are given by: ํ โ = argmax โฅ ํ โฅ 2 =1 ํพ(ํ,ํ)>0 ร ํโํ,ํ ํพ(ํ, ฬํ ํ in )ํ ํ out 2 โ๏ธ ํพ(ํ, ํ) (52) ํ โ = ร ํโํ,ํ ํพ(ํ โ , ฬํ ํ in )ํ ํ out ร ํโํ,ํ ํพ(ํ โ , ฬํ ํ in )ํ ํ out 2 (53) 47 Hilbert Operator for Progressive Encoding (HOPE) Proof. Since ํ > 0 and the denominator of (50) is positive and independent of ํ (due to โฅ ํ โฅ H = 1), minimizing the objective is equivalent to minimizing the squared numerator: โฅ ํ ํโ ํ ํ โฅ 2 H + ํ ํโ ํ ํ 2 H = 2 โฅ ํ ํ โฅ 2 H + โฅ ํ ํ โฅ 2 H + ํ ํ 2 H โ 2 ํ ํ, ํ ํ + ํ ํ H = 2ํ 2 + โฅ ํ ํ โฅ 2 H + ํ ํ 2 H โ 2ํ ํ, ํ ํ + ํ ํ H (54) Becauseํ >0 and โฅ ํ ํ โฅ H , ํ ํ H are constant with respect toํ, this reduces to maximizing the inner product: ํ โ = argmax ํโN โฅ ํ โฅ H =1 ํ, ํ ํ + ํ ํ H (55) To enforceํโN, we decompose it into unit directionsํโ โ ํ+1 , ํโ โ ํ and magnitudesํผ, ํฝ >0. By the PH-1 property ofฮจ, ํ= ํฝํฮจ(ํผํยท ฬํ)= ํผํฝํฮจ(ํยท ฬํ). The unit-norm constraint โฅ ํ โฅ H = 1 requires: 1=(ํผํฝ) 2 โฅ ํ โฅ 2 2 ํผ ํโผํ X ฮจ 2 (ํยท ฬํ) =(ํผํฝ) 2 ํพ(ํ, ํ)=โ ํผํฝ= 1 โ๏ธ ํพ(ํ, ํ) Assuming ํพ(ํ, ํ) > 0 (otherwise ํ= 0), substituting ํผํฝ gives: ํ= ฮจ(ํยท ฬํ) โ๏ธ ํพ(ํ, ํ) ํ Substituting this into (55) and expanding the tensor inner product โจ ํโ ํ, โโ ํ โฉ H = โจ ํ, โ โฉ H in โจ ํ, ํ โฉ 2 yields: * ฮจ(ํยท ฬํ) โ๏ธ ํพ(ํ, ํ) ํ, ํ ํ + ํ ํ + H = โ๏ธ ํโํ,ํ * ฮจ(ํยท ฬํ) โ๏ธ ํพ(ํ, ํ) ํ, ํ ํ out ฮจ( ฬํ ํ in ยท ฬํ) + H = 1 โ๏ธ ํพ(ํ, ํ) โ๏ธ ํโํ,ํ ํผ ํโผํ X ฮจ(ํยท ฬํ)ฮจ( ฬํ ํ in ยท ฬํ) ํ, ํ ํ out 2 = * ํ, 1 โ๏ธ ํพ(ํ, ํ) โ๏ธ ํโํ,ํ ํพ(ํ, ฬํ ํ in )ํ ํ out + 2 (56) Letํ(ํ)= ร ํโํ,ํ ํพ(ํ, ฬํ ํ in )ํ ํ out . For a fixedํ, maximizing โจ ํ, ํ(ํ) โฉ 2 subject to โฅ ํ โฅ 2 =1 requires ํ to align with ํ(ํ) via the Cauchy-Schwarz inequality: ํ โ (ํ)= ํ(ํ) โฅ ํ(ํ) โฅ 2 Substitutingํ โ (ํ)back into (56) simplifies the inner product to โฅ ํ(ํ) โฅ 2 / โ๏ธ ํพ(ํ, ํ), leavingํ โ as the sole maximizer in Equation (52).โก Proposition C.8 (Separability and Homogeneity of PH-1 Kernels). Letฮจ:โโ โbe a PH-1 activation functionโํ >0,ฮจ(ํํง)= ํฮจ(ํง). Let the functional kernel over an isotropic probability distributionํ V beํพ(ํ, ํ)= ํผ ํโผํ V [ฮจ(ํ ํ ํ)ฮจ(ํ ํ ํ)]. For any unit vectorํ โฅ ํ โฅ 2 =1 and any ํโ 0, the kernel factorizes as: ํพ(ํ, ํ)= โฅ ํ โฅ 2 ํ(ํ) where ํ= ํ, ํ โฅ ํ โฅ 2 2 (57) 48 Hilbert Operator for Progressive Encoding (HOPE) Proof.Letหํ= ํ/ โฅ ํ โฅ 2 . By the PH-1 property and linearity of expectation, the positive magnitude โฅ ํ โฅ 2 factors out: ํพ(ํ, ํ)= ํผ ํโผํ V [ฮจ(ํ ํ ํ)ฮจ( โฅ ํ โฅ 2 หํ ํ ํ)]= โฅ ํ โฅ 2 ํผ ํโผํ V [ฮจ(ํ ํ ํ)ฮจ(หํ ํ ํ)] Sinceํ V is isotropic, it is invariant under orthogonal transformations. Letํนbe an orthogonal matrix such that ํนํ= ํ 1 and ํนหํ= ํํ 1 + โ๏ธ 1โ ํ 2 ํ 2 , where ํ= โจ ํ, หํ โฉ 2 . Applying the change of variablesํ= ํนํ, we haveํโผ ํ V . Substitutingํ ํ ํ= ํ ํ ํน ํ ํ=(ํนํ) ํ ํ= ํ ํ 1 ํ= ํข 1 and similarly หํ ํ ํ= หํ ํ ํน ํ ํ=(ํนหํ) ํ ํ= ํํข 1 + โ๏ธ 1โ ํ 2 ํข 2 yields: ํผ ํโผํ V [ฮจ(ํ ํ ํ)ฮจ(หํ ํ ํ)]= ํผ ํโผํ V [ฮจ(ํข 1 )ฮจ(ํํข 1 + โ๏ธ 1โ ํ 2 ํข 2 )] Because this expectation depends only onํ, we can define it asํ(ํ), establishing the separable form ํพ(ํ, ํ)= โฅ ํ โฅ 2 ํ(ํ).โก Proposition C.9 (Boundary Identity). Letฮจbe a PH-1 function. Under a standard bivari- ate Gaussian reference distribution with correlationํ, the induced angular kernelํ(ํ)= ํผ[ฮจ(ํ)ฮจ(ํ)] satisfies ํ(1)= ํ โฒ (1). Proof.By the PH-1 property,ฮจ(ํฅ)= ํถ + ํฅํ ํฅ>0 + ํถ โ ํฅํ ํฅ<0 for constantsํถ + =ฮจ(1)andํถ โ =โฮจ(โ1). Its derivative (defined almost everywhere) isฮจ โฒ (ํฅ)= ํถ + ํ ํฅ>0 + ํถ โ ํ ํฅ<0 . Let(ํ,ํ)be standard bivariate Gaussian with correlationํ. Atํ=1,ํ= ํalmost surely, givingํ(1)= ํผ[ฮจ(ํ) 2 ]. By Priceโs Theorem ํ ํํ ํผ[ ํ(ํ)ํ(ํ)]= ํผ[ ํ โฒ (ํ)ํ โฒ (ํ)] , we haveํ โฒ (ํ)= ํผ[ฮจ โฒ (ํ)ฮจ โฒ (ํ)], which at ํ= 1 gives ํ โฒ (1)= ํผ[ฮจ โฒ (ํ) 2 ]. Evaluating these expectations: ํ(1)= ํผ (ํถ + ํํ ํ>0 + ํถ โ ํํ ํ<0 ) 2 = ํถ 2 + ํผ[ํ 2 ํ ํ>0 ]+ ํถ 2 โ ํผ[ํ 2 ํ ํ<0 ] ํ โฒ (1)= ํผ (ํถ + ํ ํ>0 + ํถ โ ํ ํ<0 ) 2 = ํถ 2 + ํผ[ํ ํ>0 ]+ ํถ 2 โ ํผ[ํ ํ<0 ] By the symmetry of the standard normal distribution,ํผ[ํ 2 ํ ํ>0 ]= ํผ[ํ 2 ํ ํ<0 ]= 1 2 ํผ[ํ 2 ]= 1 2 and ํผ[ํ ํ>0 ]= ํผ[ํ ํ<0 ]= 1 2 . Substituting these yieldsํ(1)= 1 2 (ํถ 2 + + ํถ 2 โ )andํ โฒ (1)= 1 2 (ํถ 2 + + ํถ 2 โ ). Thus, ํ(1)= ํ โฒ (1).โก Proposition C.10 (Kernel Positivity at Boundary). For any non-trivial PH-1 functionฮจ(ํฅ) .0, the induced angular kernel satisfies ํ(1) > 0. Proof.As derived in Proposition C.9,ํ(1)= 1 2 (ํถ 2 + + ํถ 2 โ ). Sinceฮจis non-trivial, at least one ofํถ + or ํถ โ is non-zero. Consequently, ํถ 2 + + ํถ 2 โ > 0, implying ํ(1) > 0.โก 49 Hilbert Operator for Progressive Encoding (HOPE) C.3. Block Eviction Proposition C.11. Letฮฉ= (ฮฆ 1 , . . . ,ฮฆ ํ )be a macroscopic state, where eachฮฆ ํ โ H ํ ํ ํ is a layer state consisting of rank-1 operatorsํ ํ,ํ โ H ํ . If the macroscopic capacityํธ(ฮฉ)is a symmetric, separable, and homogeneous function of the capacities of its constituent operators, and is invariant under arbitrary operator partitioning, it is uniquely defined as the sum of the constituent Hilbert-Schmidt norms: ํธ(ฮฉ)= ํ โ๏ธ ํ=1 ํธ(ฮฆ ํ )= ํ โ๏ธ ํ=1 ํ ํ โ๏ธ ํ=1 ํ ํ,ํ H ํ .(58) Proof.Becauseํธ(ฮฉ)is a symmetric, separable, and homogeneous functional of the individual capaci- ties, it must take the form of an ํฟ ํ norm: ํธ(ฮฉ)= ยฉ ยญ ยซ ํ โ๏ธ ํ=1 ํ ํ โ๏ธ ํ=1 ํ ํ,ํ ํ H ํ ยช ยฎ ยฌ 1 ํ for some ํ > 0 .(59) The capacity invariance axiom requires thatํธ(ฮฉ)remains invariant if any operator is partitioned. Consider an arbitrary operatorํ 0 โฮฉ. Partitioningํ 0 intoํidentical fractional operators yieldsํ operators, each defined asํ 0 /ํ. By the positive homogeneity of the Hilbert-Schmidt norm, each fractional operator has capacity โฅ ํ 0 /ํ โฅ H = 1 ํ โฅ ํ 0 โฅ H . Evaluating the capacity of this partitioned subset under the ํฟ ํ functional yields: ํธ subset = ํ โ๏ธ ํ=1 ํ 0 ํ ํ H ! 1 ํ = ํ 1 ํ โฅ ํ 0 โฅ H ํ 1 ํ = ํ 1โํ ํ โฅ ํ 0 โฅ H .(60) For the macroscopic capacity to remain invariant for any partition scaleํ โฅ1, we must have ํธ subset = โฅ ํ 0 โฅ H . This equality holds if and only ifํ 1โํ ํ = 1. Sinceํis arbitrary, the exponent must be zero: 1โ ํ ํ = 0=โ ํ= 1 .(61) Substitutingํ=1 reduces theํฟ ํ norm to theํฟ 1 sum of scalar norms, yieldingํธ(ฮฉ)= ร ํ ํ=1 ร ํ ํ ํ=1 ํ ํ,ํ H ํ . โก D. Derivation of Physical BN Parameters This appendix provides the complete derivation for recovering the physical BN parameters and raw weights from the effective parameters, expanding upon Section 7.2.2. Since the parent direction b ํ lies within the 2D subspace spanned by the augmented children, there exist projection coefficients ํ 1 , ํ 2 such that: ํ eff ํ,in = ํ 1 ํ eff in,ํ + ํ 2 ํ eff in,ํ and ํ ํ = ํ 1 ํ ํ + ํ 2 ํ ํ .(62) 50 Hilbert Operator for Progressive Encoding (HOPE) By the definition of the pre-activation signalํฆ= (ํ eff in ) ํ ํ+ ํ, it follows thatํฆ ํ = ํ 1 ํฆ ํ + ํ 2 ํฆ ํ . The physical BN parameters ํฝ ํ and ํพ ํ correspond to the mean and standard deviation of ํฆ ํ . Taking the expectation and variance yields: ํฝ ํ = ํผ[ํฆ ํ ]= ํ 1 ํผ[ํฆ ํ ]+ ํ 2 ํผ[ํฆ ํ ]= ํ 1 ํฝ ํ + ํ 2 ํฝ ํ (63) ํพ 2 ํ = Var[ํฆ ํ ]= ํ 2 1 ํพ 2 ํ + ํ 2 2 ํพ 2 ํ + 2ํ 1 ํ 2 Cov(ํฆ ํ , ํฆ ํ ) .(64) Substituting the closed-form covarianceCov(ํฆ ํ , ํฆ ํ )=|ํพ ํ ||ํพ ํ |หํ ํํ (whereหํ ํํ is defined in Equation 4) gives: ํพ ํ =ยฑ โ๏ธ ํ 2 1 ํพ 2 ํ + ํ 2 2 ํพ 2 ํ + 2ํ 1 ํ 2 |ํพ ํ ||ํพ ํ |หํ ํํ .(65) Next, we determine the raw weights ํ raw in,ํ and dataset statistics ํ ํ , ํ ํ . By definition (1): ํ eff in,ํ = ํพ ํ โ๏ธ ํ 2 ํ + ํ ํ raw in,ํ and ํ ํ = ํฝ ํ โ ํพ ํ โ๏ธ ํ 2 ํ + ํ ํ ํ .(66) This system is under-constrained. To resolve the ambiguity, we anchor the variance such that ํ 2 ํ = max(0, ํพ 2 ํ โ ํ). As noted in the main text, we focus on the active regimeํพ 2 ํ โฅ ํand defer the edge caseํพ 2 ํ < ํ to Appendix B.5. In the active regime, the physical scaling factor simplifies directly to its sign: ํพ ํ โ๏ธ ํ 2 ํ + ํ = ํพ ํ |ํพ ํ | = sign(ํพ ํ ) .(67) Substituting this back into (66) yields: ํ eff in,ํ = sign(ํพ ํ )ํ raw in,ํ and ํ ํ = ํฝ ํ โ sign(ํพ ํ )ํ ํ .(68) Multiplying both equations bysign(ํพ ํ )(and noting thatsign(ํพ ํ ) 2 =1 forํพ ํ โ 0) isolates the raw physical parameters: ํ raw in,ํ = sign(ํพ ํ )ํ eff in,ํ and ํ ํ = sign(ํพ ํ )(ํฝ ํ โ ํ ํ ) .(69) Becauseํพ ํ in (65) can take either sign, we arbitrarily choose the positive rootํพ ํ >0 without loss of generality. Under this choice, sign(ํพ ํ )= 1, and the recovered parameters simplify to: ํ raw in,ํ = ํ eff in,ํ , ํ ํ = ํ 1 ํฝ ํ + ํ 2 ํฝ ํ โ ํ ํ , ํ ํ = ํพ ํ = โ๏ธ ํ 2 1 ํพ 2 ํ + ํ 2 2 ํพ 2 ํ + 2ํ 1 ํ 2 |ํพ ํ ||ํพ ํ |หํ ํํ . (70) To prove that the physical forward pass is invariant to the chosen sign ofํพ ํ , we substitute the recovered parameters back into the BN inference equation. The true pre-activation signalํฆ ํ is computed as: ํฆ ํ = ํพ ํ โ๏ธ ํ 2 ํ + ํ (ํ raw in,ํ ) ํ ํโ ํ ํ + ํฝ ํ = sign(ํพ ํ ) h sign(ํพ ํ )ํ eff in,ํ ํ ํโ sign(ํพ ํ )(ํฝ ํ โ ํ ํ ) i + ํฝ ํ = sign(ํพ ํ ) 2 h (ํ eff in,ํ ) ํ ํโ(ํฝ ํ โ ํ ํ ) i + ํฝ ํ .(71) Since sign(ํพ ํ ) 2 = 1, the shift ํฝ ํ cancels out: ํฆ ํ =(ํ eff in,ํ ) ํ ํโ ํฝ ํ + ํ ํ + ํฝ ํ =(ํ eff in,ํ ) ํ ํ+ ํ ํ .(72) Thus, regardless of the polarity ofํพ ํ , the deployed pre-activation robustly realizes the target effective geometry. 51 Hilbert Operator for Progressive Encoding (HOPE) E. Kernel Formulation Expressing the Hilbert space inner products and norms via a kernel decouples finite-dimensional vector operations from infinite-dimensional function integrals. We define the correlation kernelํพ(ํ, ํ) of two neurons as: ํพ(ํ, ํ)โ ํผ ํโผํ X [ฮจ(ํฆ ํ )ฮจ(ํฆ ํ )](73) Under this definition, the inner product inH evaluates to: ํ ํ , ํ ํ H = ํ ํ โ ํ out,ํ , ํ ํ โ ํ out,ํ H = ํ ํ , ํ ํ H in ํ out,ํ , ํ out,ํ โ ํ = ํพ(ํ, ํ) ํ out,ํ , ํ out,ํ โ ํ (74) Consequently, the capacity of a single neuron is formulated as: โฅ ํ ํ โฅ H = ํ out,ํ 2 โ๏ธ ํพ(ํ, ํ)(75) where ํพ(ํ, ํ)= ํผ ํโผํ X [ฮจ(ํฆ ํ ) 2 ] represents the expected squared energy of the activation signal. Evaluatingํพ(ํ, ํ)requires integrating over a high-dimensional space. We derive closed-form analytical expressions for these integrals under the ReLU activation. 11 E.1. Pre-Activation Distribution To evaluate the integrals analytically, we first determine the distribution of the pre-activation signal ํฆ ํ =(ํ eff in,ํ ) ํ ํ+ ํ ํ under the surrogate distribution ํ X =N(หํ ํฅ , ห ฮฃ ํฅ ). Becauseํฆ ํ is an affine transformation of a Gaussian vector, it is univariate Gaussian. Recall from Section 4 that the surrogate mean is defined asหํ ํฅ = ํพ + raw ํ BN . Becauseํ ํ raw,ํ หํ ํฅ is theํ-th element ofํพ raw หํ ํฅ , we haveํพ raw หํ ํฅ = ํพ raw ํพ + raw ํ BN . Since the empirical BN mean vectorํ BN resides in the column space ofํพ raw , this projection simplifies toํ BN . Thus, the expected value of the raw projection recovers the empirical mean:ํผ[ํ ํ raw,ํ ํ]= ํ ํ . Similarly, the surrogate covariance ห ฮฃ ํฅ is constrained to satisfy Var(ํ ํ raw,ํ ํ)= ํ 2 ํ . Using the effective parametersํ eff in,ํ = ํพ ํ โ๏ธ ํ 2 ํ +ํ ํ raw,ํ andํ ํ = ํฝ ํ โ ํพ ํ ํ ํ โ๏ธ ํ 2 ํ +ํ , the mean ofํฆ ํ evaluates to: ํผ[ํฆ ํ ]= ํพ ํ โ๏ธ ํ 2 ํ + ํ ํผ[ํ ํ raw,ํ ํ]+ ํ ํ = ํพ ํ ํ ํ โ๏ธ ํ 2 ํ + ํ + ํฝ ํ โ ํพ ํ ํ ํ โ๏ธ ํ 2 ํ + ํ = ํฝ ํ (76) Similarly, its variance evaluates to: Var(ํฆ ํ )= ํพ 2 ํ ํ 2 ํ + ํ Var(ํ ํ raw,ํ ํ)= ํพ 2 ํ ํ 2 ํ ํ 2 ํ + ํ ! (77) Assuming the numerical stability constantํis negligible compared to the data varianceํโ0, the variance simplifies to ํพ 2 ํ . Thus, under ํ X , the pre-activation density is: ํฆ ํ โผN(ํฝ ํ , ํพ 2 ํ )(78) 11 This evaluation connects to the Arc-Cosine Kernel family of orderํ=1 [Cho and Saul, 2009]. Because HOPE relies on the translation vectorํฝto capture the full statistical state of the signal, we provide the formal proof for the biased formulation here to ensure the paper remains self-contained. 52 Hilbert Operator for Progressive Encoding (HOPE) Realizing Empirical Constraints in Practice Adapting HOPE to unnormalized networks requires a lightweight empirical calibration pass over a small, unlabeled data batch to measure the marginal pre-activation statistics(ํ ํ , ํ 2 ํ ) . Defining the effective scaling asํพ ํ = ํ ํ and the shift asํฝ ํ = ํ ํ + ํ raw,ํ anchors the evaluation to the data manifold and recovers the decoupling required to evaluate the kernel analytically. a a Networks using LayerNorm or GroupNorm still require this empirical calibration pass to determine the marginal channel statistics. Because these layers condition the signal variance, their calibration converges with remarkably few samples, and the resulting pre-activations adhere to the Gaussian surrogate assumptions. E.2. Self-Kernel Proposition E.1 (Dimensionality Reduction). For a Gaussian pre-activationํฆ ํ โผN(ํฝ ํ , ํพ 2 ํ ) and any activation functionฮจ, the expected energyํพ(ํ, ํ)= ํผ[ฮจ(ํฆ ํ ) 2 ]reduces to a 1D integral, independent of the input space dimensionality. Proof.By definition,ํพ(ํ, ํ)= ํผ ํโผํ X [ฮจ(ํฆ ํ ) 2 ]. Becauseํฆ ํ is a scalar random variable with density ํ(ํฆ ํ )=N(ํฆ ํ ;ํฝ ํ , ํพ 2 ํ ), the expectation evaluates as โซ โ โโ ฮจ(ํฆ) 2 ํ(ํฆ)ํํฆ , decoupling the computation from the ambient spaceX.โก Theorem E.2 (Closed-Form Self-Kernel for ReLU). Forํฆ ํ โผN(ํฝ ํ , ํพ 2 ํ )with|ํพ ํ | >0, the expected squared energy forฮจ(ํฆ ํ )= max(0, ํฆ ํ ) evaluates to: ํพ(ํ, ํ)=(ํพ 2 ํ + ํฝ 2 ํ )ฮฆ ํฝ ํ |ํพ ํ | + ํฝ ํ |ํพ ํ |ํ ํฝ ํ |ํพ ํ | (79) where ํ andฮฆ are the standard Normal PDF and CDF. Proof.By Proposition E.1,ํพ(ํ, ํ)= โซ โ 0 ํฆ 2 ํ(ํฆ)ํํฆ. Applying the substitutionํง= ํฆโํฝ ํ |ํพ ํ | , we have ํฆ=|ํพ ํ |ํง+ ํฝ ํ and ํํฆ=|ํพ ํ |ํํง. Setting the integration limit ํ= ํฝ ํ |ํพ ํ | , we obtain: ํพ(ํ, ํ)= โซ โ โํ (|ํพ ํ |ํง+ ํฝ ํ ) 2 ํ(ํง)ํํง= ํฝ 2 ํ โซ โ โํ ํ(ํง)ํํง+ 2ํฝ ํ |ํพ ํ | โซ โ โํ ํงํ(ํง)ํํง+ ํพ 2 ํ โซ โ โํ ํง 2 ํ(ํง)ํํง We evaluate each term using integration by parts and the identity ํ โฒ (ํง)=โํงํ(ํง): โข โซ โ โํ ํ(ํง)ํํง= 1โฮฆ(โํ)=ฮฆ(ํ) โข โซ โ โํ ํงํ(ํง)ํํง=[โํ(ํง)] โ โํ = ํ(ํ) โข โซ โ โํ ํง 2 ํ(ํง)ํํง=[โํงํ(ํง)] โ โํ + โซ โ โํ ํ(ํง)ํํง=ฮฆ(ํ)โ ํํ(ํ) Substituting these evaluations yields: ํพ(ํ, ํ)= ํฝ 2 ํ ฮฆ(ํ)+ 2ํฝ ํ |ํพ ํ |ํ(ํ)+ ํพ 2 ํ ฮฆ(ํ)โ ํํ(ํ) =(ํพ 2 ํ + ํฝ 2 ํ )ฮฆ(ํ)+ 2ํฝ ํ |ํพ ํ |โ ํพ 2 ํ ํ ํ(ํ) Sinceํพ 2 ํ ํ= ํพ 2 ํ ( ํฝ ํ |ํพ ํ | )=|ํพ ํ |ํฝ ํ , the coefficient forํ(ํ)simplifies toํฝ ํ |ํพ ํ |. Substitutingํ= ํฝ ํ |ํพ ํ | completes the proof.โก 53 Hilbert Operator for Progressive Encoding (HOPE) E.3. Cross-Kernel E.3.1. The Local Pairwise Surrogate Distribution Letํ eff = D ํ eff in,ํ ,ํ eff in,ํ E ํ eff in,ํ 2 ํ eff in,ํ 2 be the cosine similarity between the effective input weights. The cross-kernel relies on the joint pre-activation distribution ofํฆ ํ andํฆ ํ . Because optimizing the global covariance ห ฮฃ ํฅ is computationally prohibitive, we restrict the maximum entropy formulation to the local 2ร2 subspace spanned by the neuron pair. Proposition E.3 (Pairwise Warped Correlation). Under a local pairwise maximum entropy surrogate, the correlation หํ ํํ between ํฆ ํ and ํฆ ํ is given analytically by: หํ ํํ = 2ํ 1+ โ 1+ 4ํ 2 (80) whereํ is the blending constant uniquely defined by the input weight geometry and empirical standard deviations a : ํ = ํ eff 1โ ํ 2 eff ! ยฉ ยญ ยญ ยซ |ํพ ํ | ํ eff in,ํ 2 ยช ยฎ ยฎ ยฌ ยฉ ยญ ยญ ยซ |ํพ ํ | ํ eff in,ํ 2 ยช ยฎ ยฎ ยฌ (81) a To prevent undefined 0/0 states, this evaluation is restricted to active features where the effective weight norm is non-zero ํ eff in 2 > 0. Proof.Letํพ= [ํ raw,ํ , ํ raw,ํ ] โ โ ํร2 be the raw weights andํฎ= ํพ ํ ํพbe their Gram matrix. The maximum entropy distribution constrained by the variances of these two projections has a precision matrix ห ฮฃ โ1 ํฅ = ํฐ + ํพํฒํพ ํ , whereํฒ= diag(ํ ํ , ํ ํ )contains the Lagrange multipliers. The output covariance is ํช= ํพ ํ ห ฮฃ ํฅ ํพ= ํพ ํ (ํฐ+ ํพํฒํพ ํ ) โ1 ํพ. Applying the Woodbury matrix identity to the inner inverse yields: (ํฐ+ ํพํฒํพ ํ ) โ1 = ํฐโ ํพ(ํฒ โ1 + ํพ ํ ํพ) โ1 ํพ ํ = ํฐโ ํพ(ํฒ โ1 + ํฎ) โ1 ํพ ํ Substituting this back into the expression for ํช: ํช= ํพ ํ ํฐโ ํพ(ํฒ โ1 + ํฎ) โ1 ํพ ํ ํพ= ํฎโ ํฎ(ํฒ โ1 + ํฎ) โ1 ํฎ We factor out ํฎ to simplify the expression: ํช= ํฎ ํฐโ(ํฒ โ1 + ํฎ) โ1 ํฎ = ํฎ (ํฒ โ1 + ํฎ) โ1 (ํฒ โ1 + ํฎ)โ(ํฒ โ1 + ํฎ) โ1 ํฎ = ํฎ(ํฒ โ1 + ํฎ) โ1 ํฒ โ1 Inverting both sides yields the localized precision matrix ํช โ1 : ํช โ1 =ํฒ(ํฒ โ1 + ํฎ)ํฎ โ1 =ํฒ โ1 ํฎ โ1 +ํฒํฎ โ1 = ํฎ โ1 +ํฒ Becauseํฒis diagonal, its addition only perturbs the diagonal entries ofํฎ โ1 . Therefore, the off- diagonal entries are identical: [ํช โ1 ] 12 =[ํฎ โ1 ] 12 . 54 Hilbert Operator for Progressive Encoding (HOPE) Letํ raw be the correlation inํฎandํ raw be the target correlation inํช. Evaluating the inverse off-diagonal elements of these 2ร 2 matrices yields: [ํฎ โ1 ] 12 = โํ raw ํ raw,ํ 2 ํ raw,ํ 2 (1โ ํ 2 raw ) , [ํช โ1 ] 12 = โํ raw ํ ํ ํ ํ (1โ ํ 2 raw ) Equating them gives: ํ raw 1โ ํ 2 raw = ํ raw 1โ ํ 2 raw ํ ํ ํ raw,ํ 2 ! ํ ํ ํ raw,ํ 2 ! By the definition of the effective weights,ํ eff in,ํ = ํพ ํ โ๏ธ ํ 2 ํ +ํ ํ raw,ํ . The effective correlations relate to the raw correlations via the signs of these scaling parameters:ํ raw = sign(ํพ ํ ํพ ํ )ํ eff andํ raw = sign(ํพ ํ ํพ ํ )หํ ํํ . Substituting these into the odd functionํ(ํฅ)= ํฅ 1โํฅ 2 causes the sign terms to cancel. Taking the limit as ํโ 0, we substitute ํ ํ โฅ ํ raw,ํ โฅ 2 = |ํพ ํ | ํ eff in,ํ 2 , which yields the blended constant ํ : หํ ํํ 1โ หํ 2 ํํ = ํ =โ ํ หํ 2 ํํ + หํ ํํ โ ํ = 0 Solving forหํ ํํ via the quadratic formula and selecting the root that satisfies the boundary condition lim ํ eff โ0 หํ ํํ =0, we obtain โ1+ โ 1+4ํ 2 2ํ . Multiplying the numerator and denominator by the conjugate ( โ 1+ 4ํ 2 + 1) ensures numerical stability as ํ โ 0, producing the final formulation 2ํ 1+ โ 1+4ํ 2 .โก Consequently, the joint pre-activation distribution under the surrogate space is: ํฆ ํ ํฆ ํ โผN ํฝ ํ ํฝ ํ , ํพ 2 ํ |ํพ ํ ||ํพ ํ |หํ ํํ |ํพ ํ ||ํพ ํ |หํ ํํ ํพ 2 ํ (82) Regardless of the activationฮจ, any valid Cross-Kernelํพ(ํ, ํ)= ํผ[ฮจ(ํฆ ํ )ฮจ(ํฆ ํ )]must satisfy three properties: 1.Diagonal Consistency: Ifหํ ํํ =1 and the marginals are identical (ํฝ ํ = ํฝ ํ ,|ํพ ํ |= |ํพ ํ |), the interaction recovers the self-kernel: ํพ(ํ, ํ)= ํพ(ํ, ํ). 2. Cauchy-Schwarz Compliance: The magnitude is bounded: | ํพ(ํ, ํ) | โค โ๏ธ ํพ(ํ, ํ)ํพ(ํ, ํ). 3. Weight-Space Correlation Dependency: The interaction is monotonic with respect to หํ ํํ . E.3.2. Exact Bivariate Cross-Kernel for Biased ReLUs Evaluatingํพ(ํ, ํ)= ํผ[max(0, ํฆ ํ ) max(0, ํฆ ํ )]requires integrating over the joint positive orthant. Definingํ ํ = ํฝ ํ |ํพ ํ | andํ ํ = ํฝ ํ |ํพ ํ | , the exact cross-kernel evaluates to the closed-form moments of a truncated bivariate normal: ํพ(ํ, ํ)=|ํพ ํ ํพ ํ | " (ํ ํ ํ ํ + หํ ํํ )ฮฆ 2 (ํ ํ , ํ ํ ; หํ ํํ )+ ํ ํ ํ(ํ ํ )ฮฆ(ํ ํ| ํ ) + ํ ํ ํ(ํ ํ )ฮฆ(ํ ํ|ํ )+(1โ หํ 2 ํํ )ํ 2 (ํ ํ , ํ ํ ; หํ ํํ ) # (83) whereฮฆ 2 andํ 2 are the standard bivariate normal CDF and PDF evaluated at(ํ ํ , ํ ํ )with correlation หํ ํํ ,ํandฮฆare the standard univariate normal PDF and CDF, and the conditional integration boundaries areํ ํ| ํ = ํ ํ โหํ ํํ ํ ํ โ๏ธ 1โหํ 2 ํํ andํ ํ|ํ = ํ ํ โหํ ํํ ํ ํ โ๏ธ 1โหํ 2 ํํ . By centering the evaluation onํฝ ํ rather than the weight bias ํ ํ , we correctly evaluate active feature detectors in the function space. 55 Hilbert Operator for Progressive Encoding (HOPE) E.3.3. Zero-Bias Approximation for Large-Scale Networks Because calculatingฮฆ 2 for all neuron pairs is computationally intensive, we approximate the interac- tion by assuming bias shifts are negligibleํฝ ํ , ํฝ ํ โ0. Under this assumption,ํพ(ํ, ํ)factors into the geometric mean of the capacities scaled by a normalized interaction functionI(หํ ํํ ): ํพ(ํ, ํ) โI(หํ ํํ ) โ๏ธ ํพ(ํ, ํ)ํพ(ํ, ํ)(84) For the ReLU activation, applying the Arc-Cosine kernel of orderํ=1 yields the approximate cross-kernel: ํพ(ํ, ํ) โ 1 ํ โ๏ธ 1โ หํ 2 ํํ +(ํโ arccos หํ ํํ )หํ ํํ โ๏ธ ํพ(ํ, ํ)ํพ(ํ, ํ)(85) This formulation complies with the Cauchy-Schwarz inequality|ํพ(ํ, ํ)| โค โ๏ธ ํพ(ํ, ํ)ํพ(ํ, ํ)and ensures diagonal consistency (ํพ(ํ, ํ)= ํพ(ํ, ํ) when หํ ํํ = 1). F. Derivations for Block Eviction This appendix provides the mathematical framework and extended derivations for the block eviction operation introduced in Section 8. F.1. Generalization and Execution Degradation in Depleted Blocks A dedicated macro-level operation is necessary because standard granular pruning cannot fully deplete a residual block due to architectural constraints. Because the residual connection computesํ + ํน(ํ), the output dimension ofํน(ํ)(and thus the final weight tensorํ 3 ) must match that ofํ. While granular pruning can safely compress internal neurons withinํ 1 andํ 2 , pruning the output filters ofํ 3 would cause a dimensional mismatch in the element-wise additionํ + ํน(ํ). Consequently, granular compression leaves the output channels ofํ 3 intact. Leaving a residual pathway active with heavily depleted ํ 1 and ํ 2 degrades both model generalization and execution efficiency: โขModel Generalization. The output of the final BN layer evaluates toํน(ํ)= ํธโ ํ 3 ํป 2 โํ โ ํ 2 +ํ + ํท. When ํป 2 โ 0, this reduces to ํน(ํ)= ํทโ ํธโ ํ โ ํ 2 +ํ . As defined in (1), this is the effective bias vectorํ. Once the block is depleted,ํloses its normalization purpose and instead acts as an uncalibrated bias injected into the skip connection, yieldingํ= ํ + ํ. This forcefully shifts downstream feature maps out of their calibrated domain. Since the block terminates with a ReLU activationํ= ReLU(ํ + ํ), a negativeํcauses catastrophic clipping and irreversible information loss. โข Execution Efficiency. Even ifํป 2 partially survives, the shape ofํ 3 remains locked at its ambient output size. Retaining this massive parameter tensor simply to process a negligible, low-rank subspace is computationally wasteful. Block eviction resolves these issues by projectingํน(ํ) โ0, yielding a pure identity mappingํ= ํ. Unlikeํ= ํ + ํ, residual architectures are natively robust to pure identity mappings. For instance, initializingํพ=0 in the final layer of each residual branch [Goyal et al., 2017,He et al., 2016] ensures blocks begin training as ํน(ํ)= 0. F.2. Derivation of the Unified Macro Cost J evict Recall that a state tuple in layerํhas the formฮฆ (ํ) =( ํ (ํ) 1 , . . . , ํ (ํ) ํ ํ ) โH ํ ํ ํ . LetI=(ํผ 1 , . . . , ํผ ํ amb ) โ H ํ amb ํด be the tuple of identity mappings comprising the skip connection, whereH ํด is the Hilbert 56 Hilbert Operator for Progressive Encoding (HOPE) space of operators over the ambient dimensionโ ํ amb โ โ ํ amb . The aggregate mapping of any tuple is the sum of its constituent operators: F(ฮฆ (ํ) )โ ํ ํ โ๏ธ ํ=1 ํ (ํ) ํ , F(I)= ํ amb โ๏ธ ํ=1 ํผ ํ .(86) The composite block operatorB โH ํด is formalized as the addition of the residual pathwayF res โH ํด and the skip connection: B=F res ฮฆ (1) ,ฮฆ (2) +F(I) .(87) We define eviction iteratively: we project internal layers to null operators until the block is fully depleted. Evaluating the projection of an internal layerฮฆ (ํ) overํก โ [0,1]induces a continuous trajectoryฮฆ (ํ) (ํก) โ (0, . . . ,0). LetF res (ํก)denote the sequential pathway whereฮฆ (ํ) (ํก) โ0 while other layers are held constant. The composite block trajectory is: B(ํก)=F res (ํก)+F(I) .(88) From Section 6, the capacity cost for a microscopic layer transition is: J capacity (ฮฆ ํ ,ฮฆ ํ )= โซ 1 0 โํ(ฮฆ(ํก)) ํ ํํก ํธ(ฮฆ(ํก)) ํธ(ฮฆ(ํก)) ํํก .(89) To generalize this to macroscopic blocks, we expand the state to a composite tupleฮฉโ (ํ 1 , . . . ,ํ ํ ), whereํ ํ โH ํ ํ ํ . By Proposition C.11, extendingํธ(ยท)under constraints of symmetry, separability, and homogeneity yields: ํธ(ฮฉ)= ํ โ๏ธ ํ=1 ํธ(ํ ํ )= ํ โ๏ธ ํ=1 ํ ํ โ๏ธ ํ=1 โฅ ํ ํ,ํ โฅ H ํ .(90) For consistency, the constituent tuples ofฮฉmust interact strictly additively withinB. Grouping se- quentially composed layers (e.g.,ฮฆ (1) andฮฆ (2) ) into the same state creates a contradiction: projecting ฮฆ (1) โ0 collapses the entire sequential pathwayF res โ0, leavingB=F(I)with a true capacity of ํธ(I). However,ํธ(ฮฉ)would erroneously evaluate toํธ(ฮฆ (2) )+ ํธ(I). To avoid this, we must restrict the macroscopic state strictly to additive components: a single internal layerํand the skip connection I. Furthermore, because the mappingฮฆ (ํ) and the skip connectionIoperate in different dimensional spaces (ํ bottleneck versusํ amb ), direct operator additionฮฆ (ํ) +Iis mathematically undefined. We bypass this by defining the state as the composite pairฮฉ (ํ) โ (ฮฆ (ํ) ,I). The capacity cost for this macroscopic state is: J capacity (ฮฉ (ํ) ํ ,ฮฉ (ํ) ํ )= โซ 1 0 โํ(ฮฉ (ํ) (ํก)) ํ ํํก ํธ(ฮฆ (ํ) (ํก))+ ํ ํํก ํธ(I) ํธ(ฮฆ (ํ) (ํก))+ ํธ(I) ํํก .(91) To mitigate layer-width bias,ํ(ยท)acts as a counting measure of compressible operators. Being additive over disjoint sets,ํ(ฮฉ (ํ) )= ํ(ฮฆ (ํ) )+ ํ(I). Since the skip connectionIis fixed, it has no compressible operatorsํ(I)=0. Thus,ํ(ฮฉ (ํ) (ํก))= ํ(ฮฆ (ํ) (ํก)). Combined with the stationarity of the skip connection ํ ํํก ํธ(I)= 0, the integral simplifies to: J capacity (ฮฉ (ํ) ํ ,ฮฉ (ํ) ํ )= โซ 1 0 โํ(ฮฆ (ํ) (ํก)) ํ ํํก ํธ(ฮฆ (ํ) (ํก)) ํธ(ฮฆ (ํ) (ํก))+ ํธ(I) ํํก .(92) 57 Hilbert Operator for Progressive Encoding (HOPE) During the projectionฮฆ (ํ) ํ โ0,ํ(ฮฆ (ํ) (ํก))drops fromํ (ํ) active to 0. Bounding this density by its maximum pre-action valueํ (ํ) active allows a closed-form integration. Definingํธ(ํก)โ ํธ(ฮฆ (ํ) (ํก)), ยค ํธ(ํก)โ ํ ํํก ํธ(ํก), and ํธ identity โ ํธ(I): J capacity (ฮฉ (ํ) ํ ,ฮฉ (ํ) ํ ) โค ํ (ํ) active โซ 1 0 โ ยค ํธ(ํก) ํธ(ํก)+ ํธ identity ํํก= ํ (ํ) active ln 1+ ํธ (ํ) active ํธ identity ! .(93) This logarithmic formulation penalizes capacity reduction continuously. However, compression ex- ecutes finite capacity removalsฮํธ= ํธ (ํ) active in discrete leaps. Evaluating macroscopic evictions logarithmically while governing granular micro-actions (Section 6) via linear ratios creates an incon- sistent optimization hierarchy. Driven by this logarithmic discount, a greedy optimizer would view massive architectural deletions as artificially cheap. To align the macro-eviction cost with the linear micro-transition cost and ensure optimization stability, we apply the standard inequalityln(1+ ํฅ) โค ํฅ for ํฅ โฅ 0. Substituting ํฅ= ํธ (ํ) active /ํธ identity establishes a strict linear upper bound: J layer (ฮฉ (ํ) ํ ,ฮฉ (ํ) ํ )โ ํ (ํ) active ํธ (ํ) active ํธ identity .(94) F.2.1. Parallel Survival Capacity ํธ identity Flattening the tensorํinto a vectorํ โ โ ํ amb , we decompose it intoํ amb independent identity operatorsI ํ (ํ)= โจํ ํ , ํโฉํ ํ = ํฅ ํ ํ ํ . Over the input distributionํ X , the Hilbert-Schmidt capacity evaluates to the Root Mean Square (RMS) energy. Given โฅํ ํ โฅ 2 = 1: โฅI ํ โฅ H = ํผ ํโผํ X [โฅํฅ ํ ํ ํ โฅ 2 2 ] 1/2 = โ๏ธ ํผ ํโผํ X [ํฅ 2 ํ ] .(95) Assumingํis conditioned by a preceding BN layer with scaleํพ ํ and shiftํฝ ํ , the expected energy evaluates toํผ[ํฅ 2 ํ ]= Var(ํฅ ํ )+(ํผ[ํฅ ํ ]) 2 = ํพ 2 ํ + ํฝ 2 ํ . The aggregate capacity of the skip connection is therefore: ํธ identity = ํ amb โ๏ธ ํ=1 โ๏ธ ํพ 2 ํ + ํฝ 2 ํ .(96) For a stably normalized networkํพ ํ โ1, ํฝ ํ โ0, this naturally evaluates to the ambient physical dimension: ํธ identity โ ํ amb . F.2.2. Resolving the Extinction Divergence Block eviction projects a layerํto 0. Since PH-1 activations satisfyฮจ(0)=0, this ensures the entire sequential residual pathway vanishesF res โ0, reducing the composite block mapping toB=F(I). Axiom 2 (Connectivity Preservation) requires the transition cost to divergeJ โ โas the state capacity approaches zero, penalizing graph disconnections. If evaluated solely on the isolated layerฮฆ (ํ) , the terminal capacity would evaluate to zero, yielding an infinite penalty that would permanently prevent the optimizer from selecting an eviction operation. However, because the post-eviction mapping of the block isF(I), the network retains a minimum capacityํธ identity >0. Evaluating the integral over the macroscopic stateฮฉ (ํ) = (ฮฆ (ํ) ,I)naturally tracks this non-vanishing capacity. Since the differential change is driven entirely by the active layer ํํธ(ฮฉ)= ํํธ(ฮฆ (ํ) ), we have: J (ํ) layer = โซ 0 ํธ (ํ) active โํ (ํ) active ํํธ ํธ+ ํธ identity .(97) Asํธ โ0, the denominator is bounded bylim ํธโ0 ํธ(ฮฉ)= ํธ identity >0. The extinction divergence is thus mitigated. 58 Hilbert Operator for Progressive Encoding (HOPE) F.3. Generalization to Non-Residual Architectures Whileํธ identity structurally resolves the division-by-zero limit for residual networks, architectures without additive skip connections lack this architectural anchor. Examples include sequential VGG- style blocks, Inception modules, DenseNets, or intermediate expansion layers inside Transformer FFNs (prior to the global residual addition). In these architectures, evicting a block leaves no parallel identity mapping. This means the surviving capacity reaches zero. To evaluate block eviction in these domains without encountering an infinite penaltyJ โ โ, we replace the dynamic composite denominator with a static historical constant: the blockโs initial, pre-pruning capacity ํธ (ํ) init . The individual layer-wise cost becomes: J layer (ฮฉ (ํ) ํ ,ฮฉ (ํ) ํ )= ํ (ํ) active ํธ (ํ) active ํธ (ํ) init (98) For a block containing multiple internal layers, the total macroscopic eviction cost is the linear sum of these bounds: J evict = ร ํ J (ํ) layer . From the perspective of pure functional analysis,ํธ init acts as a heuristic. It violates the Markov property of continuous state transitions (because the system must maintain memory of a pristine architectural state that no longer exists) and departs from Axiom 2 by freezing the denominator rather than tracking an architectural asymptote. However, it serves as an optimal proxy for three reasons: โขScale Invariance: Using an arbitrary absolute constantํถwould yieldJ โ ํธ (ํ) active /ํถ , violating scale neutrality. The ratioํธ (ํ) active /ํธ (ํ) init โ [0,1]normalizes the capacity reduction against the layerโs own baseline. This ensures blocks of varying widths are penalized fairly based on the relative fraction of capacity destroyed. โขAvoiding Normalization Collapse: Normalizing by the current active capacityํธ (ํ) active would collapse the cost toJ layer = ํ (ํ) active . This artificially discounts wide, pristine layers, driving the greedy optimizer to evict them prematurely instead of targeting heavily pruned, redundant layers. โขDynamic Cost Decay: The progressive encoder selects actions that minimize the Distortion Rate DR=J/ฮํ. Under a mean-field assumption, active capacity scales linearly with surviving width ํธ active โ ํ active . By locking the denominator toํธ init , the structural cost decays quadratically as the layer is prunedJ โ ํ active ํธ active โ ํ 2 active . Because the physical parameter yield also scales linearlyฮํ โ ํ active , the overall distortion rate decreases linearly:DRโ ํ active . This guarantees that the relative cost of evicting a block naturally decreases as granular micro-actions deplete it. Thus, whileํธ identity is the rigorously derived anchor for residual networks,ํธ init provides the necessary constraints to enable automated, progressive eviction in non-residual architectures. G. Reproducibility Protocols for Cross-Domain Transfer This section details the experimental setup, algorithmic formulations, and hyperparameter grids necessary to reproduce the cross-domain transfer results presented in Section 11.2. G.1. Task Construction and Data Partitioning Tasks are dynamically constructed from the datasets. We perform 4 independent trials, with each trial comprised of 5 tasks to yield 20 distinct cross-domain transfer scenarios. 59 Hilbert Operator for Progressive Encoding (HOPE) โขSource Tasks (CIFAR-100): To ensure the source task requires learning cohesive structural features, the task comprises 20 classes sampled by selecting 4 random superclasses and utilizing all 5 of their constituent fine classes. โขTarget Tasks (SVHN): The target tasks consist of 10-class classification utilizing all SVHN digits. Both CIFAR-100 and SVHN datasets are partitioned into an 80% training split and a 20% validation split. The original test sets are preserved entirely for final evaluation. Prior to training, images are cast to floating-point tensors and scaled to[0,1]by dividing by 255. No data augmentation is applied during training. G.2. Network Architecture All experiments utilize a VGG-style 8-layer sequential baseline, modernized with BN. To prevent the final classification head from dominating the networkโs parameter footprint, we replace the classic flattened Dense layers with Global Average Pooling (GAP). This ensures that the modelโs overall capacity, and consequentlyฮํevaluated by our compression algorithm, remains localized to the convolutional filters rather than spatial dense transitions. This prevents artificially skewed Distortion Rates (DR). The network consists of the following blocks: 1. Block 1: Conv2D (128 filters, 3x3)โ BNโ ReLUโ MaxPool (2x2) 2. Block 2: Conv2D (256 filters, 3x3)โ BNโ ReLUโ MaxPool (2x2) 3.Block 3: Conv2D (512 filters, 3x3)โBNโReLUโConv2D (512 filters, 3x3)โBNโReLU โ MaxPool (2x2) 4.Block 4: Conv2D (512 filters, 3x3)โBNโReLUโConv2D (512 filters, 3x3)โBNโReLU โ MaxPool (2x2) 5. Transition: GlobalAveragePooling2D 6. Bottleneck: Dense (512 neurons)โ BNโ ReLU 7. Classification Head: Dense (10 or 20 neurons) for multi-class logit output. Note that BN layers are positioned between the affine transformations (Conv2D/Dense) and the ReLU non-linearities. To prevent redundancy and ensure capacity accounting, biases in the Conv2D and Dense layers preceding a BN layer are disabled. G.3. Base Training Regimen Both pre-training and fine-tuning utilize SGD with standard heavy-ball momentum of 0.9, a fixed batch size of 16, and a Sparse Categorical Cross-Entropy (CE) loss function. โขSource Pre-training: The model is trained from a random initialization for 100 epochs. To suppress mini-batch noise and ensure the parameters in the slack space converge to zero, the learning rate follows a Cosine Decay schedule starting atํ=0.05 and decaying to a near- zeroํผ=0.001. Early stopping is deliberately disabled. To actively clear out the networkโs peripheral slack during this phase, anํฟ 2 regularization penalty of 5ร10 โ4 is applied to all multi-dimensional kernels as well as all 1D BN scale parameters ํพ. โขTarget Fine-Tuning: All algorithms fine-tune for 30 epochs and use a fixed learning rate rather than a decay schedule. During target adaptation, theํฟ 2 penalty is restricted to tensors with a rank greater than 1 to leave the calibrated 1D BN scale parameters immune to weight decay. 60 Hilbert Operator for Progressive Encoding (HOPE) G.4. EWC Exact Empirical Fisher Calculation To properly lock foundational source features for the EWC baseline, the empirical diagonal Fisher Information Matrix (FIM) is evaluated over the source training dataset (bypassing validation sets). Note that the mathematical formulation calculates the expected squared gradient of the source CE loss at the per-example level before batch averaging: FIM ํ = ํผ ํโผX source_train " ํL CE ํํฝ ํ 2 # (99) Many naive EWC implementations incorrectly square the averaged mini-batch gradient, which misrepresents the true Fisher Information Matrix. By computing the per-example squared gradients directly, our evaluation avoids this theoretical pitfall. During target domain optimization, the target CE loss is augmented by the EWC penalty: L total =L target_CE + ํ 2 โ๏ธ ํโBackbone FIM ํ (ํฝ ํ โ ํฝ โ ํ ) 2 (100) whereํis a global regularization hyperparameter, andํฝ โ ํ represents the model parameters of the optimal pre-trained source. Note that the summation is restricted to the backbone parameters; the newly initialized target classification head is excluded from the penalty. G.5. Hyperparameter Tuning and Final Evaluation To ensure a fair comparison, all algorithms undergo a hyperparameter pre-sweep prior to final evaluation. Hyperparameters are selected by maximizing the H-Score, defined as the Harmonic Mean of the validation accuracies on the target and source tasks: H-Score= 2ยท Acc target val ยท Acc source val Acc target val + Acc source val (101) This evaluation uses the Validation Sets of both SVHN (target) and CIFAR-100 (source) to prevent data leakage. To accelerate the pre-sweep, the target training set is artificially capped at 1,000 samples during the tuning phase. Rather than re-evaluating the grid for all 20 distinct transfer scenarios, the algorithm isolates the very first task (Task 0) as a representative sample, sweeps the grids to find the globally optimal parameters, and locks them in for the entire benchmark run. The sweep evaluates grids specific to each methodology: โขDEFT: A grid search across the target percentileํ โ 60,40,30,20, and fixed fine-tuning learning rate ํ โ 0.04, 0.02, 0.01, 0.005, 0.001. โขEWC: A grid search across the regularization strengthํ โ 0.1,0.5,1.0,5.0,10.0,20.0,50.0 and learning rate ํ โ 0.01, 0.005, 0.001, 0.0005, 0.0001. โข PEFT: Unfreezes all 1D tensors (including layer biases and BN affine parameters) along- side the target head, while freezing all 2D/4D kernels. Optimized via a line search for ํ โ 0.01, 0.005, 0.0025, 0.001, 0.0005, 0.0001. โข Standard Full FT: Unfreezes the entire network architecture. Optimized via a line search for ํ โ 0.02, 0.01, 0.005, 0.001, 0.0005, 0.0001. โขStandard Head-Only FT: Freezes the entire backbone, optimizing only the final classification head. Optimized via a line search for ํ โ 0.04, 0.02, 0.01, 0.005, 0.001, 0.0005. 61 Hilbert Operator for Progressive Encoding (HOPE) Once optimal hyperparameters are secured, models are fine-tuned on the full SVHN training set for 30 epochs. Performance is tracked epoch-by-epoch using the SVHN Validation Set, and the algorithm identifies the best epoch yielding peak target validation accuracy. The final reported metrics in the main text are extracted at this best epoch by running the model against the unseen SVHN Test Set and CIFAR-100 Test Set. Source Retention Protocol: To evaluate source retention, the original pre-trained source clas- sification head is temporarily grafted back onto the fine-tuned backbone. Furthermore, a universal mask is applied to the grafted source head. This zeros out connections originating from upstream โslackโ featuresํธ >0 to ensure target-adapted weights do not corrupt the source logits. Critically, the BN moving statisticsํ, ํ 2 are reverted; they are unlocked from the target-adapted backbone and restored to their original source states to guarantee a fair evaluation. H. Theoretical Guarantees of DEFT Continual learning seeks a parameter updateฮํฝthat minimizes target risk while bounding the degradation of the source representation. Rather than analyzing non-convex loss landscapes, we abstract the network into continuous-functional operators and evaluate layer-wise distortion. Under HOPE, the capacity of neuron ํ is its Hilbert-Schmidt norm over ํ X : โฅ ํ ํ โฅ H =โฅํ out,ํ โฅ 2 โ๏ธ ํพ(ํ, ํ) where ํพ(ํ, ํ)= ํผ ํโผํ X ฮจ(ํฆ ํ (ํ)) 2 .(102) The Setup: Core vs. Slack. DEFT partitions the network into two disjoint sets based on a capacity threshold ํ: โขUniversal Coreโฅ ํ ํ โฅ H > ํ: Highly active neurons carrying source knowledge. We freeze these ํธ ํ = 0 to prevent forgetting. โขSlack Subsetโฅ ํ ํ โฅ H โค ํ: Weak, inactive neurons. We make these fully plasticํธ ํ =1 to learn the target task. To prevent the target-driven updates of the slack subset from corrupting the core, DEFT applies a structural mask at initializationํก=0, severing all connections from upstream slack neurons to downstream core neurons. We establish the stability and plasticity of this setup through four main guarantees: 1. Bounded Initialization Shock (Theorem H.1). Severing the slack-to-core connections atํก=0 introduces a static error. As visualized in Figure 5(a), the severed core-directed weightsํ core,ํ of a slack neuronํform a sub-vector of its total outgoing weightsํ out,ํ . Thus,โฅํ core,ํ โฅ 2 โค โฅํ out,ํ โฅ 2 . Because we only sever connections from slack neurons (whereโฅ ํ ํ โฅ H โค ํ), the initial distortion injected into the core is strictly bounded by |N (ํ) slack |ํ (ํ) . 2. Dynamic Decoupling (Theorem H.2). During trainingํก >0, slack neurons drift to learn the target task. Because the structural mask severs cross-connections at initializationํ (0) core,ํ = 0 and the zero elasticity prevents gradient updatesํํ core,ํ /ํํก=0, these weights remain zero. As illustrated in Figure 5(b), a changing signal multiplied by zero is zero; thus, target parameter drift cannot penetrate the core. 3. Freeing Space Safely (Proposition H.4). Freezing the core might leave insufficient parameter space for the target task. Deep networks often fragment a feature acrossํcorrelated neurons. Statically freezing them incorrectly locks redundant volume. Instead, DEFT compresses theseํ 62 Hilbert Operator for Progressive Encoding (HOPE) neurons into a single rank-1 parent operator. This preserves the feature while releasing the remaining ํโ1 children to the slack subset, freeing the parameter spaceโ (ํโ1)รํ for optimization. Because the optimal parent minimizes the Hilbert-Schmidt projection error, this structural distortion is upper- bounded by the sum of the childrenโs initial capacities:ํฟ ํ โค ร ํ ํ=1 โฅ ํ ํ โฅ H (ํ) . (While we structurally guarantee this capacity release, we do not formally prove gradient descent finds a global optimum in this non-convex space). 4. Unified Cumulative Bound (Corollary H.5). Cutting connections and merging neurons across layers introduces multiple distortions. However, because Theorem H.2 guarantees zero interference during training, the networkโs total error does not compound exponentially. By the triangle inequality inH (ํ) , the global degradation in the function space is static and bounded by the linear sum of the severed slack connections and the merge projection errors. To Slack To Core Cut Component Kept Component Total w out (a) Vector Geometry (Theorem H.1) Preceding Layerโs Slack Drift Signal (ฮํ) (Highly chaotic) ร Cut Connection ํ core = 0 Current Layerโs Core Update (ฮs=0) ฮโ 0 0 (b) Dynamic Signal Blocking (Theorem H.2) Figure 5|Visual intuition for the theoretical bounds of DEFT. (a) Vector decomposition demonstrates the initialization shock is bounded by the slack capacities. (b) A changing signal multiplied by the zeroed structural mask ensures dynamic decoupling. H.1. Algorithmic Axioms and Partitioning of Neurons Here we formally define how DEFT partitions neurons into the peripheral slack and universal core. We also present a few axioms that merely mirror the mechanisms designated in the DEFT algorithm itself. These axioms do not impose any extra assumptions or restrictions on DEFT beyond the entirety of the algorithm itself. Definition H.1 (Neurons Partition). For any intermediate layerํcontaining a set of active neurons I (ํ) , a continuous capacity threshold ํ (ํ) > 0 divides the layer into two disjoint functional subsets: N (ํ) core =ํโI (ํ) | โฅ ํ ํ โฅ H (ํ) > ํ (ํ) (103) N (ํ) slack =ํโI (ํ) | โฅ ํ ํ โฅ H (ํ) โค ํ (ํ) (104) Axiom 1 (The Algorithmic Structural Mask). At initializationํก=0, DEFT prevents cross-subset interference by severing the projections from the upstream slack subset to the downstream core. For any weightํค ํข,ํ from an upstream slack neuronํ โ N (ํ) slack to a downstream core neuron ํขโN (ํ+1) core , the mask ํด enforces ํค ํข,ํ = 0. 63 Hilbert Operator for Progressive Encoding (HOPE) Axiom 2 (The Gradient Elasticity). During target optimizationํก >0, the parameter update for any parameter vector ํฝ ํ belonging to a destination neuron ํ is governed by: ํํฝ ํ ํํก = ํธ ํ ยทโ ํฝ ํ L target (105) whereํธ ํ โ 0,1is the elasticity multiplier. Slack neurons are fully plasticํธ ํ =1โํโN slack and core neurons are strictly frozen ํธ ํข = 0โํขโN core . Axiom 3 (The Merged-Vessel Release). When the continuous encoder collapses redundant neurons into a rank-1 parent operator, DEFT assigns full plasticityํธ ํ =1 to the structurally released child neurons. H.2. Layer-to-Layer Bounding Framework We now establish a rigorous bound on the structural distortion, decoupled entirely from target parameter drift. Assumption 1 (Bipartite Separability). Let the pre-activationํฆ ํข of a downstream core neuron ํข โ N (ํ+1) core be a linear combination of upstream activationsํ ํ (ํ)=ฮจ(ํฆ ํ (ํ))forํ โ I (ํ) , governed by parameterized weightsํค ํข,ํ and unparameterized identity connections denoted by an adjacency indicatorํ ํข,ํ โ 0,1. We assume that for all upstream slack neuronsํโN (ํ) slack , the network identity routing is strictly zero: ํ ํข,ํ = 0 โํโN (ํ) slack ,โํขโN (ํ+1) core (106) This ensures that the mask strictly dictates inter-subset signal flow. Remark on Feed-Forward Architectures: For purely sequential architectures (e.g., standard MLPs or VGG),ํ ํข,ํ โก0 globally. This assumption only becomes non-trivial in architectures with unparame- terized residual connections (e.g., ResNets), where an identity mapping could bypass the mask if the architecture cross-wired the subsets. Theorem H.1 (The Static Initialization Shock Bound). Letํ ํ (ํ)โฮจ(ํฆ ํ (ํ)), and letํ core (ํ)โ ร ํโI (ํ) ํ core,ํ ํ ํ (ํ) be the input to the downstream core. The initialization maskํดintroduces a distortionฮํ init core (ํ)โ ร ํโN (ํ) slack ํ core,ํ ํ ํ (ํ) bounded over ํ X by: ฮํ init core H (ํ) โค |N (ํ) slack | ํ (ํ) (107) Proof. By Assumption 1, ํ core (ํ) decomposes strictly over the parameterized upstream subsets: ํ core (ํ)= โ๏ธ ํโN (ํ) core ํ ํ (ํ) ํ core,ํ + โ๏ธ ํโN (ํ) slack ํ ํ (ํ) ํ core,ํ (108) whereํ core,ํ is the sub-vector of structural weights connecting upstream neuronํto the downstream core. By Axiom 1, DEFT enforcesํ core,ํ =0โํโN (ํ) slack . The resulting functional distortion evaluates 64 Hilbert Operator for Progressive Encoding (HOPE) to the severed projections: ฮํ init core (ํ)= โ๏ธ ํโN (ํ) slack ํ ํ (ํ) ํ core,ํ (109) Applying the triangle inequality inH (ํ) yields: ฮํ init core H (ํ) โค โ๏ธ ํโN (ํ) slack ํ core,ํ 2 ํ ํ H (ํ) in (110) By Definition 102, ํ ํ H (ํ) in = โ๏ธ ํพ(ํ, ํ). Furthermore, becauseํ out,ํ = ํ core,ํ โ ํ slack,ํ , its Euclidean norm bounds its sub-vectors: ํ core,ํ 2 โค ํ out,ํ 2 . Substituting these properties gives: ฮํ init core H (ํ) โค โ๏ธ ํโN (ํ) slack ํ out,ํ 2 โ๏ธ ํพ(ํ, ํ)= โ๏ธ ํโN (ํ) slack โฅ ํ ํ โฅ H (ํ) (111) By Definition H.1, the capacity of every slack neuron satisfiesโฅ ํ ํ โฅ H (ํ) โค ํ (ํ) . Therefore, the functional distortion is strictly bounded by: ฮํ init core H (ํ) โค โ๏ธ ํโN (ํ) slack ํ (ํ) = ํ (ํ) |N (ํ) slack |(112) โก H.2.1. Dynamic Decoupling The dynamic decoupling of the core emerges as a direct consequence of Axiom 2. Theorem H.2 (Dynamic Decoupling). During target fine-tuningํก >0, the operatorsํ (ํ) ํ of all neurons within the core experience zero dynamic deviation: โํ , ํโN (ํ) core ; ํ (ํ),(ํก) ํ โ ํ (ํ),(0) ํ H (ํ) = 0(113) Proof. We proceed by structural induction over the network layers. Base Case: The input ํ โผ ํ X is a static anchor unmodified by optimizationฮํ= 0. Inductive Step: Assume that for layerํ, the core functional mappings exhibit zero drift:ํ (ํก) ํ (ํ)= ํ (0) ํ (ํ) for all ํ โN (ํ) core . The dynamic deviation of the signal injected into the downstream core of layerํ+1 at timeํกis defined byฮํ core (ํ)= ํ (ํก) core (ํ)โ ํ (0) core (ํ). Expanding this difference over the core and slack subsets yields: ฮํ core (ํ)= โ๏ธ ํโN (ํ) core ํ (ํก) core,ํ ํ (ํก) ํ (ํ)โ ํ (0) core,ํ ํ (0) ํ (ํ) + โ๏ธ ํโN (ํ) slack ํ (ํก) core,ํ ํ (ํก) ํ (ํ)โ ํ (0) core,ํ ํ (0) ํ (ํ) (114) For the first summation, Axiom 2 freezes downstream core weightsํธ ํข =0=โ ํ (ํก) core,ํ = ํ (0) core,ํ , and the inductive hypothesis guarantees ํ (ํก) ํ (ํ)= ํ (0) ํ (ํ). The first summation is therefore zero. 65 Hilbert Operator for Progressive Encoding (HOPE) For the second summation, Axiom 1 severs cross-subset weights at initializationํ (0) core,ํ =0. Becauseํธ ํข =0, these structural weights receive zero gradient updates and remain zeroํ (ํก) core,ํ =0. Thus, the second summation evaluates to zero irrespective of the slack drift ํ (ํก) ํ . Consequently,ฮํ core (ํ) โก0. Since the downstream core parametersํฝ ํข โ ํ in,ํข , ํ ํข , ํพ ํข , ํฝ ํข are also frozenํธ ํข =0, the downstream core mappings experience zero driftํ (ํก) ํข (ํ)= ํ (0) ํข (ํ). By induction, the target learning is decoupled from the core representation across all layers.โก Remark on Covariate Shift: Theorem H.2 guarantees the immutability of the core operators relative to the source distribution anchorํ X . In practice, the network processes novel target data D ํ during fine-tuning. Consequently, empirical activations flowing through the core will inevitably shift due to standard covariate shift, not parameter degradation. DEFT ensures that any shift in downstream representations is driven entirely by the target data itself, free from the compounding noise of parameter degradation. This preserves foundational knowledge as an uncorrupted lens for processing new domains. Corollary H.3 (Cumulative Masking-Induced Bound). Assuming a masking-based reduction without feature merging, the global structural distortionํท mask global evaluates as the cumulative sum of normalized layer-wise distortions. Normalizing the initialization shock at each layer by the downstream core capacity ํธ (ํ+1) core โ ร ํขโN (ํ+1) core โฅ ํ ํข โฅ H (ํ+1) , the total distortion is bounded by: ํท mask global โ ํฟโ1 โ๏ธ ํ=1 ฮํ init,(ํ+1) core H (ํ) ํธ (ํ+1) core โค ํฟโ1 โ๏ธ ํ=1 |N (ํ) slack |ํ (ํ) ํธ (ํ+1) core ! (115) This confirms that source task degradation is strictly bounded by the chosen capacity thresholds, circumventing the exponential scaling issues of global Lipschitz constants. H.3. Dynamic Resolution of Redundancy via Bounded Trade-off A static capacity threshold evaluates functional volume in isolation, which ignores the redundancy trap where networks fragment a feature across multiple correlated neurons. To compress this redundant volume and avoid incorrectly locking it, DEFT employs HOPEโs MERGE operation. Proposition H.4 (Bounded Distortion of Merging). Consolidatingํcorrelated neurons into a rank-1 parent neuron guarantees the release of target optimization volume while bounding the distortion of the source representation. Proof.Supposeํcorrelated neurons ํ ํ ํ ํ=1 are assigned to the coreํธ ํ =0. By generating an optimal parentํ ํ โH (ํ) , DEFT collapsesํcopies into 1 and freesํโ1 child vessels. By Axiom 3, DEFT assigns these children full plasticityํธ ํ =1, successfully releasing the parameter spaceโ (ํโ1)รํ for optimization. The distortionํฟ ํ introduced by this operation is the composite projection error between the parent ํ ํ and its children ํ ํ ํ ํ=1 . Because the optimal parent minimizes this distance inH (ํ) , the error is 66 Hilbert Operator for Progressive Encoding (HOPE) upper-bounded by the sub-optimal projection to the null operator ํ ํ โก 0: ํฟ ํ โ min ํ ํ ํ โ๏ธ ํ=1 โฅ ํ ํ โ ํ ํ โฅ 2 H (ํ) ! 1 2 โค ํ โ๏ธ ํ=1 โฅ ํ ํ โ 0โฅ 2 H (ํ) ! 1 2 (116) Applying the standardโ ํ -norm inequalityโฅํโฅ 2 โค โฅํโฅ 1 , we bound this sub-optimal projection by the linear sum of the child capacities: ํฟ ํ โค ํ โ๏ธ ํ=1 โฅ ํ ํ โฅ H (ํ) (117) Thus, the algorithm safely releases massive target parameter volume while the distortion remains strictly bounded by the initial capacities.โก Corollary H.5 (The Unified Cumulative Bound). By the triangle inequality inH (ํ) , the total perturbation injected into the core of layerํ+1 is bounded by the sum of the missing slack projections (Theorem H.1) and the structural projection errors fromํพ (ํ) merge operations. Because dynamic interference evaluates to zero via structural induction (Theorem H.2), the global normalized functional distortion is bounded as: ํท total global โ ํฟโ1 โ๏ธ ํ=1 ฮํ (ํ+1) core H (ํ) ํธ (ํ+1) core โค ํฟโ1 โ๏ธ ํ=1 ยฉ ยญ ยซ |N (ํ) slack |ํ (ํ) + ร ํพ (ํ) ํ=1 ํฟ ํ ํธ (ํ+1) core ยช ยฎ ยฌ (118) whereํฟ ํ is the bounded projection error of theํ-th merge operation. Therefore, DEFT releases target parameter volume while bounding cumulative source degradation to an algorithmically verifiable constant. I. Algorithms 67 Hilbert Operator for Progressive Encoding (HOPE) Algorithm 1 HOPE Progressive Encoding Loop Require: Pre-trained modelM, Target density ํ target 1: Initialize CostManager() โฒ PHASE 1: O(ํ 2 ) Initialization 2: for each layer ํฟโM do 3:InitializeCaches(ํฟ)โฒ Computes initial capacities andO(ํ 2 ) geometry 4:Anchor initial uncompressed capacity ํธ (ํฟ) 0 5: end for โฒ PHASE 2: True O(1) Greedy Scan 6: while Density(M) > ํ target do 7: ํ โ โ null, DR min โโ โฒ Evaluate all cached actions using live residual capacities 8: for each active layer ํฟ do 9:ํธ rem โ GetResidualCapacity(ํฟ)โฒ Dynamic tracking of layer shrinkage 10:ํ live โ ํฟ.ActiveCount()โฒ Live layer width for unbiased normalization 11:for each candidate ํ โ ํฟ.ActiveActions() doโฒ Prunes, Merges, Evicts 12:ฮํ static โ GetStaticPayoff(ํ)โฒ Action-specific static payoff 13:J cost โ ComputeDistortion(ํ, ํธ rem , ํ live )โฒ O(1) scalar arithmetic 14:DRโ J cost /ฮํ static 15:if DR < DR min then 16:DR min โ DR 17:ํ โ โ ํ 18:end if 19:end for 20: end for โฒ Execution and Local Synthesis 21:ExecutePhysicalReduction(M, ํ โ ) 22:UpdateResidualCapacity(ํ โ .ํฟ, ํ โ .J cost ) โฒ O(1) Local Recalculation (Strict intra-layer isolation) 23: if ํ โ .type== MERGE then 24:ํ ํ โ ํ โ .parent 25:Cache.UpdatePruneCapacity( ํ ํ )โฒ Update solitary capacity for new parent 26:for each ํ neighbor โ ํ โ .ํฟ.ActiveNeurons() do 27:geo new โ PrecomputePairGeometry( ํ ํ , ํ neighbor ) 28:Cache.Insert(( ํ ํ , ํ neighbor ), geo new ) 29:end for 30: end if โฒ ZERO downstream recalculation required due to the staticฮํ approximation 31: end while 32: returnM 68 Hilbert Operator for Progressive Encoding (HOPE) Algorithm 2 HOPE Subroutine: UpdateResidualCapacity Require:Executed compression actionํ โ (contains action type, target layer(s)ํฟ, and targeted neurons) 1: ํโ 10 โ12 โฒ Numerical stability floor for capacity tracking 2: if ํ โ .type== PRUNE then 3: ํฟโ ํ โ .ํฟ 4: ํ vic โ ํ โ .victim 5:ฮํธโ Cache.GetPruneCapacity( ํ vic ) 6: ํฟ.ํธ rem โ max(ํฟ.ํธ rem โฮํธ, ํ) 7: ํฟ.RemoveNeuron( ํ vic ) 8: else if ํ โ .type== MERGE then 9: ํฟโ ํ โ .ํฟ 10: ํ ํ , ํ ํ โ ํ โ .children 11: ํ ํ โ ํ โ .parent โฒ Net flux: capacity of the extinguished children minus the new parent 12:ฮํธโ Cache.GetPruneCapacity( ํ ํ )+ Cache.GetPruneCapacity( ํ ํ )โ ComputeCapacity( ํ ํ ) 13: ํฟ.ํธ rem โ max(ํฟ.ํธ rem โฮํธ, ํ) 14: ํฟ.RemoveNeuron( ํ ํ )โฒ Child ํ is purged; Child ํ serves as the vessel for ํ ํ 15: else if ํ โ .type== EVICT then โฒ Eviction targets all internal reduction layers within the residual pathway 16: for each internal layer ํฟโ ํ โ .block do 17:ํฟ.ํธ rem โ ํโฒ Pathway capacity forcibly collapsed 18:ํฟ.ClearAllActiveNeurons()โฒ All internal operators are projected to 0 19: end for 20: end if 69 Hilbert Operator for Progressive Encoding (HOPE) Algorithm 3 Dispersed Elastic Fine-Tuning (DEFT) Require:Pre-trained source parametersํ src , Source validation dataD val ํ , Target dataD ํก , Percentile threshold ํ โฒ PHASE 1: Capacity Evaluation & Core/Slack Partitioning 1:Merge Redundancies: Use HOPE to compress highly correlated neurons into parent operators, freeing child vessels. 2: Compute Capacities: Calculate the functional capacity โฅ ํ ํ โฅ H for all active neurons. 3: Thresholding: Determine global capacity threshold ํ corresponding to the ํ-th percentile. 4: for each neuron ํ in the network do 5: if โฅ ํ ํ โฅ H > ํ and neuron ํ is active then 6:ํธ ํ โ 0โฒ Universal Core: Freeze to protect source knowledge 7: else 8:ํธ ํ โ 1โฒ Plastic Slack: Freed vessels and weak neurons learn target 9: end if 10: end for โฒ PHASE 2: Consistency at Initialization (The Structural Mask) 11: Initialize target weights ํ 0 โ ํ src . 12: for each weight ํค ํข,ํ connecting an upstream neuron ํ to a downstream neuron ํข do 13: if ํธ ํ = 1 and ํธ ํข = 0 then 14:ํ ํข,ํ โ 0โฒ Sever cross-connections to prevent Slack drift from entering Core 15: else 16:ํ ํข,ํ โ 1 17: end if 18: end for 19: Apply Mask: ํ 0 โ ํดโ ํ 0 โฒ PHASE 3: Elastic Target Fine-Tuning (Dynamic Decoupling) 20: ํ ํก โ ํ 0 , best_h_scoreโโ1 21: for epochโ 1, . . . , FINETUNE_EPOCHS do 22: for mini-batch (ํ, ํ) โ D train ํก do 23:ํ โโ ํ ํก L target (ํ ํก ; ํ, ํ)โฒ Compute raw target gradients 24:ํ mod โ ํฌโ ํโฒ Nullify gradients for frozen Core (ํธ ํ = 0) 25:ํ ํก โ OptimizerStep(ํ ํก , ํ mod )โฒ Only plastic Slack updates 26: end for // Dual-Domain Evaluation (Harmonic Score Optimization) 27:Acc tgt โ Evaluate(ํ ํก ,D val ํก ) 28: ํ src_eval โ MaskSlackDrift(ํ ํก , ํ src , ํฌ)โฒ Restore pristine Core / Mask out Slack drift 29:Acc src โ Evaluate(ํ src_eval ,D val ํ ) 30:H-Scoreโ 2ยทAcc tgt ยทAcc src Acc tgt +Acc src 31: if H-Score > best_h_score then 32:best_h_scoreโ H-Score 33:ํ best โ ํ ํก 34: end if 35: end for 36: return ํ best 70