Paper deep dive
Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures
Yanglin Zhang, Anneke von Seeger, Gilad Lerman, Ron Levie
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We develop a principled framework for constructing neural networks whose specific parameter realizations admit an explicit algorithmic interpretation. Existing algorithm-inspired architectures can explain the computational structure of a network, yet after standard training the learned parameters need not retain a clear relation to the motivating algorithm. We address this gap as follows. First, we model each data point as a sample of a class-dependent stochastic process and assume that statistics of this process can be estimated from a single sample and these statistics are sufficient to distinguish the classes. We then construct a neural network whose initial parameters exactly implement an algorithm for estimating these statistics, making the network fully interpretable. To account for mismatch between the idealized model and real data, we fine-tune this network while controlling its deviation from the algorithmic initialization. The trained network hence roughly retains the interpretation of the initial network. A PAC-Bayesian analysis yields a uniform generalization bound whose complexity term scales with the fine-tuning radius, providing a statistical motivation for our approach. We instantiate the framework for texture classification using the scattering transform to estimate the discriminative statistics.
Tags
Links
- Source: https://arxiv.org/abs/2610.04413v1
- Canonical: https://arxiv.org/abs/2610.04413v1
Trouble viewing inline? Open PDF directly →
Full Text
132,878 characters extracted from source content.
Expand or collapse full text
Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures Yanglin Zhang* Affiliation: Technion – Israel Institute of Technology, Faculty of Mathematics Anneke von Seeger* Affiliation: University of Minnesota, School of Mathematics Gilad Lerman Affiliation: University of Minnesota, School of Mathematics Ron Levie Affiliation: Technion – Israel Institute of Technology, Faculty of Mathematics Abstract We develop a principled framework for constructing neural networks whose specific parameter realizations admit an explicit algorithmic interpretation. Existing algorithm-inspired architectures can explain the computational structure of a network, yet after standard training the learned parameters need not retain a clear relation to the motivating algorithm. We address this gap as follows. First, we model each data point as a sample of a class-dependent stochastic process and assume that statistics of this process can be estimated from a single sample and these statistics are sufficient to distinguish the classes. We then construct a neural network whose initial parameters exactly implement an algorithm for estimating these statistics, making the network fully interpretable. To account for mismatch between the idealized model and real data, we fine-tune this network while controlling its deviation from the algorithmic initialization. The trained network hence roughly retains the interpretation of the initial network. A PAC-Bayesian analysis yields a uniform generalization bound whose complexity term scales with the fine-tuning radius, providing a statistical motivation for our approach. We instantiate the framework for texture classification using the scattering transform to estimate the discriminative statistics. 11footnotetext: Equal contribution. Keywords: Explainable AI, PAC-Bayesian Analysis, Scattering Transform, Texture Classification 1 Introduction Classical algorithms are typically constructed as explicit sequences of computational operations. Each step is specified by design and serves a known role in transforming the input toward the desired output, making the underlying computational procedure transparent. Neural networks, in contrast, are produced through data-driven learning processes. Their behavior is determined jointly by the computational structure of the network and a collection of parameters optimized from data. Although the forward computation of a trained network is fully specified mathematically, understanding how these learned parameters and computations collectively give rise to the underlying decision process is often difficult. The boundary between classical algorithms and neural networks, however, is not absolute. Many algorithms can be expressed as a sequence of operations that naturally correspond to the layers or modules of a neural network. This connection is exploited, for example, in algorithm unfolding, where the steps of an iterative algorithm are mapped to network layers, and is closely related to algorithmic alignment, where the computational structure of a network is chosen to reflect the decomposition of a target algorithm (Monga et al., 2021; Xu et al., 2020). An algorithmic interpretation of the architecture, however, does not necessarily provide an interpretation of a particular trained network. Consider a family of networks parameterized by weights. It may be possible to assign the same algorithmic interpretation to every member of this family, even though different parameter values instantiate different computations. Such an interpretation explains the algorithmic role of the architecture, but does not necessarily explain what is encoded by the learned parameters. We refer to this notion as generic algorithmic interpretability. Examples of such architecture-level interpretations include viewing neural networks as unfolded sparse-coding or pursuit algorithms (Papyan et al., 2017; Sulam et al., 2020), transformer blocks as iterations of sparse rate reduction (Yu et al., 2023), deep networks through the information bottleneck principle (Tishby and Zaslavsky, 2015), and deep architectures through the variational renormalization group (Mehta and Schwab, 2014). In this work, we seek a stronger notion, which we call specific algorithmic interpretability. Here, the interpretation depends on the particular values of the parameter: different values of the parameters can be related to different explicit algorithmic objects or operations. Thus, rather than asking only what algorithmic template a network architecture follows, we ask what particular algorithm is implemented by a trained network in view of the values of its parameters. Our goal is to develop a principled framework for constructing neural networks whose learned parameters admit such an explicit algorithmic interpretation, allowing the underlying decision process of the trained network to be understood. Concretely, we begin with a predefined network whose parameters implement an explicit algorithm. To decide what the target of the algorithm should be, we first propose a model of the data. We model observations as realizations of class-dependent stochastic processes and interpret the algorithm as estimating, from a single observation, statistics of its generating process that are sufficient to distinguish the class. This gives the predefined parameter realization a specific algorithmic interpretation: its computation can be understood in terms of the statistics it estimates. More broadly, this leads to a stochastic view of pattern recognition in which recognition amounts to inferring discriminative properties of an observation’s underlying generating process. A difficulty arises when such an algorithmically motivated network is trained in the usual way: unconstrained optimization may move its parameters to a realization with no clear relation to the original algorithm. We call this the algorithmic alignment gap. We address it by initializing the network as the interpretable algorithm and fine-tuning on real data by controlling the distance traveled in parameter space. A PAC-Bayesian analysis shows that, under a local stability condition, the resulting generalization bound scales with this fine-tuning radius, providing a statistical justification for training large neural networks in the low-data regime. We then instantiate the framework in texture classification, where ergodicity allows statistics of an underlying stationary process to be estimated from a single image. The scattering transform provides an explicit interpretable algorithm that we realize as predefined neural networks and use as initializations for subsequent fine-tuning. 1.1 Related Work Algorithmic interpretations of neural networks. A broad literature seeks to make learned models interpretable, including intrinsic approaches, which design models to be interpretable by construction , and post-hoc approaches, which explain already trained models (Rudin, 2018; Arrieta et al., 2019). Our work belongs to the former setting and focuses specifically on algorithmic interpretations of neural networks. Closely related are algorithm-unfolding methods, which map iterations of an optimization algorithm to network layers (Monga et al., 2021); a canonical example is LISTA, which unfolds the Iterative Shrinkage-Thresholding Algorithm and learns its operators from data (Gregor and LeCun, 2010). Related algorithmic interpretations have also been developed for CNNs and transformer-like architectures (Papyan et al., 2017; Yu et al., 2024). These works typically provide algorithmic interpretations of the computational structure shared across a family of parameterizations, which we refer to as generic algorithmic interpretability. For example, a sparse-coding network may still be interpreted as performing sparse inference after its dictionary has been learned, even though this interpretation does not by itself explain the particular learned dictionary. Our work instead targets specific algorithmic interpretability, where the particular learned parameters themselves can be related to explicit algorithmic objects and operations, thereby revealing the particular algorithm implemented by the trained network. Further connections to interpretability literature are discussed in Appendix A.1. Neural Collapse Neural collapse (NC) is an empirical phenomenon that emerges in the penultimate-layer representations of deep classification networks during the terminal phase of training (Papyan et al., 2020). In the input domain, samples from different classes may be highly entangled and difficult to separate directly. During training, the network progressively disentangles these inputs by learning representations in which features from the same class become concentrated while those from different classes become increasingly separated. Under neural collapse, this disentangling process culminates in representations for which the classes are readily separable by the final linear classifier. Subsequent theoretical works have shown that, when the feature extractor is treated as infinitely expressive, neural collapse arises as an optimal solution of the classification objective (E and Wojtowytsch, 2022; Lu and Steinerberger, 2022; Tirer and Bruna, 2022; Nguyen et al., 2022; Mixon et al., 2022). These results suggest that neural collapse can be viewed as a limiting behavior toward which training drives the learned representation, progressively transforming an initially entangled classification problem into one that is simple to solve with a linear classifier. In our approach, the target of the initial algorithm is to promote neural collapse. PAC-Bayesian bounds for neural networks PAC-Bayesian theory provides generalization bounds in terms of a distribution over predictors, typically controlled by the KL divergence between a learned posterior and a chosen prior (McAllester, 1999; McAllester, 2003; Langford and Shawe-Taylor, 2002). For neural networks, Neyshabur et al. (2018) derived PAC-Bayesian margin bounds by relating parameter perturbations to changes in the network output through layer-wise spectral norms. Subsequent work extended and refined such analyses for convolutional architectures and other structured neural networks (Behboodi et al., 2022). These results provide an alternative to classical uniform complexity bounds, which can become vacuous for overparameterized networks (Shalev-Shwartz and Ben-David, 2014). The wavelet scattering transform Wavelet-based models provide explicit multiscale statistics for stationary images. Wavelet transform. The wavelet transform represents images as superpositions of local details at different scales. Let ψλ∈Λ\ _λ\_λ∈ be a so called family of complex wavelets. Here, λ indexes scale and orientation, and each wavelet ψλ _λ is typically chosen as a local bump with oscillations at the direction and rate parameterized by λ. The wavelet transform represents an image x through the coefficients x⋆ψλ¯x _λ, where ⋆ denotes correlation, which describe its local content at different scales and orientations. We also let ϕJ _J denote a low-pass filter performing spatial averaging at scale 2J2^J. More details on the wavelet transform are given in Appendix A.2.1. Wavelet scattering transform. The wavelet scattering transform constructs multiscale statistics by cascading wavelet convolutions with complex modulus nonlinearities, followed by spatial averaging (Mallat, 2012). For a scattering path p=(λ1,…,λq)p=( _1,…, _q), define U[∅]x=xU[ ]x=x and recursively U[p]x=|U[(λ1,…,λq−1)]x∗ψλq|.U[p]x=|U[( _1,…, _q-1)]x* _ _q|. The windowed scattering coefficient associated with p is SJ[p]x=U[p]x∗ϕJ.S_J[p]x=U[p]x* _J. More details on the scattering transform are given in Section A.2.2. Mallat (2012) also formulates scattering for stationary stochastic processes through the expected scattering coefficients [U[p]X].E[U[p]X]. The windowed scattering coefficients SJ[p]xS_J[p]x can therefore be viewed as estimates of these statistics from one single realization. In particular, when ϕJ _J is normalized to integrate to one, they are unbiased estimators of S¯X(p) S_X(p); see the explanation of Equation 9 in Section A.2.2 for more details. It provides a predefined family of statistics for stationary processes, which we use as the selected statistics in our stochastic model to construct an explicit feature extractor for texture classification. 2 Specific algorithmic interpretability via a model of perception In this section we present our general approach for specific algorithmic interpretability. Notation. For an integer m≥1m≥ 1, let [m]:=1,…,m[m]:=\1,…,m\. We consider a C-class classification problem with input space :=x∈ℝn:‖x‖2≤BX:=\x ^n:\|x\|_2≤ B\ and representation space (,d)(Z,d). Let FΘF_ denote a neural network parameterized by Θ . Depending on the context, FΘF_ denotes either the feature extractor FΘ:→,F_ :X , or the corresponding predictor with a single-layer linear head FΘ:→ℝC.F_ :X ^C. For j∈[C]j∈[C], we write FΘ(x)[j]F_ (x)[j] for the j-th coordinate of FΘ(x)F_ (x). When the parameters consist of d matrix-valued blocks, Θ=vec(Wii=1d), =vec (\W_i\_i=1^d ), so that ‖Θ‖22=∑i=1d‖Wi‖F2.\| \|_2^2= _i=1^d\|W_i\|_F^2. Throughout, ∥⋅∥F\|·\|_F denotes the Frobenius norm, while ∥⋅∥2\|·\|_2 denotes the Euclidean norm for vectors and the spectral norm for matrices. We use ∥⋅∥∞\|·\|_∞ for the maximum norm. 2.1 From neural collapse to data disentangling Neural collapse as a geometric objective. We use neural collapse as a guiding geometric objective for understanding the computation performed by the neural network feature extractor; see Section 1.1 for background. Rather than explaining all properties associated with neural collapse, we focus on its basic geometric principle: representations should concentrate observations from the same class while separating observations from different classes. If FΘF_ denotes the neural network feature extractor, our question is how the computation x⟼FΘ(x)x F_ (x) transforms an initially entangled input signal space into a class-organized representation that is simple to separate. We therefore seek explicit computations for which within-class concentration and between-class separation can be established mathematically. Data disentanglers. We begin with the simplest setting in which the computation of a feature extractor can be understood explicitly: a neural network whose parameters are chosen to implement a prescribed algorithm rather than learned from data. Let FΘ:→F_ :X denote a generic neural network feature extractor. We are interested in a particular predefined parameter realization Θ0 _0 for which the computation x⟼FΘ0(x)x F_ _0(x) has an explicit algorithmic interpretation. We call FΘ0F_ _0 a data disentangler if it maps observations to a class-separating representation under an appropriate model of the data. Unlike clustering, which organizes observations using relations among the samples in a dataset, a data disentangler applies the same transformation to each observation independently. Any class information encoded in the representation must therefore be recoverable from properties of the individual observation itself. This motivates a stochastic view of the data. We regard each observation as a realization of an underlying stochastic process, and interpret the role of the data disentangler as estimating from that realization statistics of the generating process that are sufficient to distinguish its class. 2.2 A transparent stochastic data model Let S denote a family of stochastic processes, where each X∈X∈ S induces a probability law PXP_X on the observation space X. We place a probability measure μ over S, noting that there is a natural measurable structure over S. The data distribution P is then modeled as the mixture (1) P(A)=∫PX(A)μ(X),A⊆.P(A)= _ SP_X(A)\,dμ(X), A . Equivalently, an observation is generated in two stages: first a stochastic process X∼μX μ is selected, and then a realization x∼PXx P_X is drawn from that process. For a classification problem with C classes, we assume that the process space is partitioned into disjoint subsets =1∪⋯∪C, S= S_1∪·s∪ S_C, where processes in c S_c generate observations belonging to class c. Let μc=μ(⋅∣c) _c=μ(\,·\, S_c) denote the distribution of processes conditioned on class c. We denote by YcY_c the mixture process associated with class c, whose induced law is (2) Pc(A)=∫cPX(A)dμc(X).P_c(A)= _ S_cP_X(A)\,d _c(X). Thus, variation within a class may arise both from randomness within an individual process and from variation among the processes belonging to that class. Assumption: estimable class-separating statistics. We assume that there exists a metric space (,d)(Z,d) and a map s:→s: S that associates each stochastic process X with a collection of statistics s(X)s(X). For each class, define c=s(X):X∈c.A_c=\s(X):X∈ S_c\. We assume that these sets are separated across classes, so that the statistics s(X)s(X) are sufficient to determine the class of the generating process. We further assume that the processes in S are ergodic, in the sense that these statistics can be estimated from a single realization x∼PXx P_X. In particular, there exists an estimator s s such that (3) s^(x)≈s(X) s(x)≈ s(X) whenever x∼PXx P_X, in an appropriate asymptotic regime. Under these assumptions, the predefined network becomes a data disentangler when its parameters Θ0 _0 are chosen so that (4) FΘ0(x)=s^(x).F_ _0(x)= s(x). Indeed, if x is generated by a process X∈cX∈ S_c, then FΘ0(x)F_ _0(x) approximates s(X)∈cs(X) _c. The network therefore maps individual observations to a class-separating representation determined by statistics of their underlying generating processes. Moreover, the parameters Θ0 _0 have an explicit algorithmic meaning: they implement a prescribed procedure for estimating these statistics. The stochastic pattern recognition hypothesis. The transparent stochastic model suggests a broader view of pattern recognition. Suppose an observation x is generated by a stochastic process X∈X∈ S. A conventional view of pattern recognition treats x itself as the object to be classified. Here we adopt a different perspective: x is viewed as a realization that carries information about its underlying generating process. In this view, pattern recognition is therefore formulated as inferring from x the statistics s(X)s(X) that distinguish this process from the relevant alternatives. This motivates what we call the stochastic pattern recognition hypothesis: pattern recognition consists of inferring, from an individual observation, statistics that distinguish its underlying stochastic generating process from the other processes relevant to the task. Under this view, the fundamental objects being distinguished are generating mechanisms rather than the particular samples appearing in a finite dataset. We conjecture that neural feature extractors can be interpreted as progressively estimating increasingly discriminative statistics of these mechanisms, thereby transforming the input into a representation in which recognition becomes simple. This statistical view has strong precedents in texture analysis. Julesz’s classical work proposed that human texture perception can be characterized through statistics of an underlying stochastic image process (Julesz, 1962). In a later formulation of Julesz’s hypothesis, one assumes that there exists a collection of functions ϕkk=1K\ _k\_k=1^K such that two stochastic texture processes P and Q are perceptually indistinguishable if and only if X∼P[ϕk(X)]=X∼Q[ϕk(X)],k=1,…,K.E_X P[ _k(X)]=E_X Q[ _k(X)], k=1,…,K. This statistical viewpoint has subsequently appeared in increasingly rich texture models and scattering-based representations of stationary processes (Portilla and Simoncelli, 2000; Mallat, 2012), and related formulations arise in the classification of stationary ergodic processes (Ryabko and Ryabko, 2010). In Section 3 we instantiate our setting for texture classification. 2.3 The algorithmic alignment gap The transparent stochastic model provides a particular parameter realization Θ0 _0 for which the network FΘ0F_ _0 implements an explicit data-disentangling algorithm. This gives the predefined network a specific algorithmic interpretation. However, this property is generally lost under standard end-to-end training. Algorithm unfolding and related approaches construct a network architecture from the computational structure of an algorithm, but the resulting network is then typically fully trained without requiring its learned parameters to remain related to the parameters that implement the original algorithm. Consequently, the architecture may retain a generic algorithmic interpretation while the particular trained parameter realization doesn’t admit an explicit algorithmic interpretation. We refer to this discrepancy as the algorithmic alignment gap: an architecture may be aligned with an interpretable algorithm, while the parameters obtained after training need not retain a specific algorithmic interpretation. Bridging this gap therefore requires controlling not only the architecture of the network, but also how its parameters change during learning. 2.4 Bridging the algorithmic alignment gap: fine tuning To address the algorithmic alignment gap, we propose using the interpretable realization Θ0 _0 as the initialization for learning and fine-tuning the network, i.e., restricting how far its parameters move from this initialization. Let Θ denote the parameters after fine-tuning. The quantity ‖Θ−Θ0‖2\| - _0\|_2 measures how far learning has moved the network from the algorithmic realization. One motivation of fine-tuning is that Θ0 _0 is constructed under an idealized stochastic model, whereas real data need not satisfy this model exactly. Some adaptation is therefore desirable to compensate for this mismatch. Accordingly, the predefined network FΘ0F_ _0 is not required to perform competitively with modern trained models; rather, we suppose that a well-performing predictor can be reached without moving too far from Θ0 _0. Our theoretical result gives a statistical reason to restrict this movement. For R>0R>0, define the neighborhood BR(Θ0)=Θ:‖Θ−Θ0‖2≤R.B_R( _0)=\ :\| - _0\|_2≤ R\. We show that for networks in this neighborhood, the generalization bound, which bounds the gap between the statistical and empirical risks, scales linearly with R and inversely with M M, where M is the number of training samples. Thus, restricting the distance from the algorithmic initialization controls an aspect of the learned model beyond its algorithmic interpretation: its generalization bound. This is the sense in which fine-tuning bridges the algorithmic alignment gap. The transparent stochastic model gives us an explicit, data-independent parameter realization Θ0 _0 with a known algorithmic meaning. By restricting learning to BR(Θ0)B_R( _0), we can retain most of the interpretation of this reference point while analyzing the generalization gap. To formulate our result, define the classification margin mΘ(x,y)=FΘ(x)[y]−maxj≠yFΘ(x)[j].m_ (x,y)=F_ (x)[y]- _j≠ yF_ (x)[j]. Let R0(FΘ)R_0(F_ ) denote the statistical classification risk and R^γ(FΘ) R_γ(F_ ) the empirical γ-margin risk. Our analysis uses what we call an (R,τ,ϵ)(R,τ,ε)-stability condition. Informally, it requires that all networks within distance R of Θ0 _0 change their outputs by at most ϵε under parameter perturbations of size at most τ. Formal definitions and the PAC-Bayesian derivation are given in Appendix B.1. Corollary 1 (Uniform Generalization Bound Around a Data-Independent Initialization). Let ℱF be a parametrized hypothesis family, and let Θ0=vec(Wii=1d) _0=vec(\W_i\_i=1^d) be an initialization independent of the training data, where Wi∈ℝmi×niW_i ^m_i× n_i, and define h:=maxi∈[d]mi,ni.h:= _i∈[d]\m_i,n_i\. Fix R,γ,τ>0R,γ,τ>0, and suppose that ℱF is (R,τ,γ4R,τ, γ4)-stable under bounded parameter perturbations around Θ0 _0. Then, for any δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ over the draw of a training set of size M, the following holds for all Θ∈BR(Θ0) ∈ B_R( _0): (5) R0(FΘ)≤R^γ(FΘ)+4(R2hln(4dh)/τ2+ln(6M/δ))/(M−1)R_0(F_ )≤ R_γ(F_ )+4 (R^2h (4dh)/τ^2+ (6M/δ) )/ (M-1 ) Implications for training. We use the bound primarily as an a priori design principle rather than as a post-training numerical certificate. We note that after training, predictive performance can be assessed directly on held-out data. Hence, evaluating the numerical value of a worst-case PAC-Bayesian bound is not the purpose of our analysis. Corollary 1 shows that for a fixed architecture and perturbation-stability scale τ, the generalization gap scales with O(RM).O( R M). Hence, a smaller adaptation radius yields a tighter bound; equivalently, for a fixed target generalization gap, a smaller adaptation radius allows the same guarantee to be obtained with fewer training samples. This suggests fine-tuning an algorithmically meaningful initialization can be particularly useful in the low-data regime. It therefore motivates to control ‖Θ−Θ0‖2. - _0 _2. Such control can be imposed explicitly through a regularizer λ‖Θ−Θ0‖22,λ - _0 _2^2, or encouraged by a conservative optimization procedure, such as a sufficiently small learning rate. Importantly, the bound does not require the predefined disentangler itself to achieve competitive performance on the real data. Its role is to provide a data-independent and algorithmically interpretable reference point from which a good predictor can be reached with limited adaptation. This leaves substantial freedom in the construction of the initial algorithm. 3 Specific Algorithmic Interpretability In Texture classification In this section we instantiate the above theory on the concrete example of texture classification. 3.1 Textures as stationary processes with subset statistics A texture is an image exhibiting repeated local structure without requiring strict periodicity. Textures are commonly modeled as realizations of stationary random fields (Julesz, 1962; Portilla and Simoncelli, 2000; Mallat, 2012). Let Y=(Y(u))u∈ℤ2Y=(Y(u))_u ^2 be a random field. For g∈ℤ2g ^2, define the translation action on a random field by (g.Y)(u):=Y(u−g)(g.Y)(u):=Y(u-g). A random field Y is stationary if its law is invariant under this action: g.Y=Yg.Y d=Y for every g∈ℤ2g ^2. A texture image is viewed as a finite observation of this field in a spatial rectangular window. We next discuss the ergodic decomposition of the stationary process Y. For consistent notation, we denote a generic ergodic process arising in this decomposition by X. Ergodicity has a precise definition in probability theory. We give here an equivalent definition which is more in line with our discussion, i.e., the empirical average over any integrable function of the array converges to the expected value of this function. The equivalent formulation follows the so called Birkhoff’s ergodic theorem (Krengel and Brunel, 2011, Sec. 1.2). Definition 1. The process X=(X(u))u∈ℤ2X=(X(u))_u ^2 is called ergodic if for every integrable function f with respect to the law of X, i.e. f∈L1(PX)f∈ L_1(P_X), we have (6) limN→∞14N2∑g∈[−N,N]×[−N,N]f(g.X)=a.s.[f(X)], _N→∞ 14N^2 _g∈[-N,N]×[-N,N]f(g.X) .s.=E[f(X)], where expected value is with respect to the law of X. A stationary process need not be ergodic. However, the ergodic decomposition theorem (Viana and Oliveira, 2016) states that any stationary process is a mixture of ergodic processes in the sense of (1), where each PXP_X is ergodic. Hence, stationary processes are well suited to our analysis. One can think about the set of all integrable functions f in (6) as the set of sufficient statistics.11 1 In fact, this claim can be made precise. By taking all indicator functions as f, one can extract the ergodic components. More accurately, the set of statistics is X↦[f(X)]\X [f(X)]\. However, this set is too large to be of any practical use. Instead, practical definitions of statistics consider subsets of the integrable functions as the statistics. By taking a subset S of L1(PX)L_1(P_X), we reduce the resolution of the statistics. Namely, there are separate ergodic components with the same set of S statistics. One hence has to carefully choose a meaningful subset statistics for the task, in the sense that images with the same statistics tend to be in the same class. Finite observations. The stochastic processes above are defined on the full lattice ℤ2Z^2, whereas an observed texture image has finite spatial support. We therefore model an image as a finite restriction as follows. The generative procedure first produces an infinite random field X as described above, after which only its restriction X|ΛX|_ to a predefined finite rectangular domain Λ⊂ℤ2 ^2 is observed. Consequently, the limiting statistics in (6) cannot be computed exactly from an image. In practice, they are estimated by averaging over the observed domain Λ . We assume that Λ is sufficiently large for these finite spatial averages to provide good approximations to the corresponding ergodic limits. The preceding discussion gives the texture-specific realization of the stochastic model introduced in Section 2.2. In principle, a sufficiently rich collection of statistics can distinguish the underlying ergodic components. In practice, however, only a finite subset can be computed. The choice of this subset determines which differences between generating processes are retained. For classification, we therefore seek statistics that can be estimated reliably from a finite image while retaining the distinctions relevant for separating texture classes. Classical statistical approaches to texture recognition characterize processes through moments and correlations. Because higher-order moments quickly become computationally expensive, practical methods have traditionally focused largely on first- and second-order statistics. More expressive representations can be obtained by choosing structured nonlinear statistics. In what follows, we consider wavelet scattering transform as a concrete example (Mallat, 2012). 3.2 Wavelet scattering as a data disentangler We now instantiate the statistical construction above using the wavelet scattering transform, reviewed in Section 1.1 and described in detail in Appendix A.2.2. Let P denote the collection of scattering paths used in the representation. For each path p∈p , define fp(X):=U[p]X(0).f_p(X):=U[p]X(0). Since wavelet filtering and the pointwise modulus commute with translations, evaluating fpf_p on translated realizations corresponds to evaluating U[p]XU[p]X at different spatial locations. Hence each fpf_p is one of the functions to which the ergodic averaging principle in (6) applies. The windowed scattering coefficient SJ[p]x(u)=U[p]x∗ϕJ(u)S_J[p]x(u)=U[p]x* _J(u) is precisely a weighted spatial average of these translated evaluations. Unlike the limiting average in (6), this average is weighted against the scaling function ϕJ _J. When ϕJ _J is sufficiently high scale, integrating against it provides a finite-image approximation to the corresponding process statistic [fp(X)].E[f_p(X)]. Thus, scattering amounts to selecting the finite family scat=fp:p∈S_scat=\f_p:p \, from the much larger collection of functions appearing in the ergodic decomposition, and acts as a data disentangler. In Appendix B.2, we show that this disentangler satisfies the local parameter-stability condition required by Corollary 1. Qualitative assessment through texture synthesis. Before evaluating the resulting representation for classification, we first inspect whether the selected scattering statistics retain meaningful texture information. Given a reference texture x, we synthesize an image x~ x whose scattering statistics match those of x by solving x~≈argminz‖SJ(z)−SJ(x)‖22. x≈ _z \|S_J(z)-S_J(x) \|_2^2. Here, we use the micro-canonical approaches proposed by Bruna and Mallat (2018); implementation details are given in Appendix C.1. We then visually inspect whether the synthesized image depicts a texture similar to the reference. If matching the statistics consistently reproduces the characteristic appearance of the reference texture, this suggests that the selected statistics preserve perceptually relevant properties of the underlying texture process. Figure 1 shows representative results of the synthesis. Figure 1: Texture synthesis by matching wavelet scattering statistics. Synthesized images show that the selected scattering statistics retain perceptually meaningful texture information. 4 Experiments We realize the predefined scattering transform as a trainable convolutional network. At initial parameters Θ0 _0, the feature extractor exactly implements the scattering transform, with wavelet convolutions applied independently to each input channel. During fine-tuning, the filters can adapt and information across channels can be combined. Gauthier et al. (2022) propose a related approach in which only a small number of parameters controlling the filters are learned; see Appendix C.2.2. We compare against three baselines: the same architecture with random initialization, the original fixed scattering transform, and the learnable scattering-network block architecture proposed by Zarka et al. (2021) (ScatNet B.A.). ScatNet B.A. keeps the wavelet filters and scattering modulus fixed, while adding learnable 1×11× 1 convolutions to combine and reduce channels, followed by learnable tight-frame analysis layers. All methods use a single linear classification head. Together, these baselines isolate the benefit of fine-tuning from a predefined scattering realization, relative to random initialization, no adaptation, and an alternative trainable scattering architecture. Model details are given in Appendix C.2.2. We evaluate on DTD, EuroSAT, MinC-2500, and SuroTex (Cimpoi et al., 2014; Helber et al., 2019; Bell et al., 2015; Putra et al., 2025). Classical texture benchmarks such as KTH-TIPS and CUReT are not used since they are relatively small, and fixed scattering representations already perform strongly on them, leaving limited room to study the benefit of fine-tuning(Fritz et al., 2004; Dana et al., 1999). SuroTex instead provides a large-scale texture dataset, while DTD, EuroSAT, and MinC-2500 introduce increasingly richer variability beyond classical texture images. Together, these datasets allow us to study the regime between textures well described by the predefined model and more general images for which adaptation becomes important. We also evaluate performance across different training-set sizes to study sample efficiency; dataset splits and sampling details are given in Appendix C.2.1. For our architecture, we set the fine-tuning regularization parameter λ with values of 0,0.1,1.00,0.1,1.0. For ScatNet B.A. and the randomly initialized architecture, we set λ=0λ=0 to avoid anchoring to a prior that is not meaningful. We report test accuracy and generalization gap in this section. We further conduct an ablation study to separate the effect of regularizing the fine-tuning distance from that of the scattering prior, and analyze the effect of fine-tuning on neural collapse and interpretability; these additional results are provided in Section C.3. The ablation study shows that anchoring the model to the scattering prior improves accuracy compared with either no regularization or standard weight decay, while the interpretability analysis shows that fine-tuned filters remain wavelet-like. Figure 2: Percentage of data used during training versus test accuracy. Color denotes method and line style denotes regularization strength. The proposed method gives the best performance when the amount of training data is small. Figure 2 displays the percentage of available data used during training versus test accuracy, calculated as the number of correct predictions out of total predictions on the test dataset. Corollary 1 focuses on controlling the generalization gap and does not directly address test accuracy. However, having low generalization gap is typically related to high performance, so long as the training accuracy is high, as we demonstrate in Figure 5. Figure 2 demonstrates that the fine-tuned scattering network outperforms all baseline methods when training data is extremely limited. On the least texture-like dataset, MinC-2500, ScatNet B.A. outperforms the fine-tuned scattering network as training data increases. On the remaining datasets, the fine-tuned scattering network either performs the best or performs competitively with baseline methods as data increases. Figure 3: Fine-tuning distance versus generalization gap. Color denotes method, line style denotes regularization strength, and size denotes amount of training data used. Gray lines between fine-tuned scattering networks trained on the same data size allow comparison across regularization strengths. Smaller fine-tuning distance correlates with lower generalization gap. Figure 3 displays the fine-tuning distance against generalization gap, where fine-tuning distance is the L2L^2 distance between initial and final parameters, normalized by the initial parameter norm. As predicted by Corollary 1, the generalization gap tends to decreases as the fine-tuning distance decreases. The effect is strongest on MinC-2500, the least texture-like dataset, suggesting that control of the fine-tuning distance helps most when the data departs from the theoretical model and the parameters would otherwise travel further. The randomly initialized network does not follow the trend established by the fine-tuned scattering network, and has a higher generalization gap than the trend suggests, given its fine-tuning distance. The deviation from the trend suggests that anchoring the networks parameters to an interpretable algorithm improves performance, a finding supported by Figure 7, which analyzes the effect of anchoring a randomly initialized model to the scattering network’s parameters. 5 Conclusion We introduced a framework for specific algorithmic interpretability that ties a particular neural-network parameter realization to an explicit algorithm rather than merely an algorithm-inspired architecture. The key idea is to preserve this algorithmic reference during learning through controlled fine-tuning, whose distance measures the deviation from the disentangler and allows controlling the generalization bound. Significance. Our framework provides a way to train highly over-parameterized networks in low-data regimes without overfitting and without requiring a network pretrained on a large external dataset. In contrast to standard transfer learning, where the useful initialization is typically obtained by pretraining on vast amounts of data, here it is supplied directly by an explicit interpretable algorithm. Additionally, the framework suggests a new role for classical algorithms in the deep-learning era: rather than being discarded in favor of flexible learned models, they can serve as algorithmically interpretable initializations that combine explicit domain knowledge with data-driven adaptation. Limitations and Future Work. The present framework is mainly suited to classification problems. Future work may extend this perspective to other settings, including generative models, and instantiate the framework for other data modalities. Such extensions may clarify when explicit algorithmic structure provides a useful foundation for modern learned models. Acknowledgement. AvS and GL were partially supported by NSF award DMS-2427955. RL was supported by a grant from the United States-Israel Binational Science Foundation (BSF), Jerusalem, Israel, and the United States National Science Foundation (NSF), (NSF-BSF, grant No. 2024660), and by the Israel Science Foundation (ISF grant No. 1937/23). References Andreux et al. (2020) M. Andreux, T. Angles, G. Exarchakis, R. Leonarduzzi, G. Rochette, L. Thiry, J. Zarka, S. Mallat, J. Andén, E. Belilovsky, et al. Kymatio: scattering transforms in python. Journal of Machine Learning Research 21 (60), p. 1–6. Cited by: §C.1, §C.2.2. Arrieta et al. (2019) A. B. Arrieta, N. D. Rodríguez, J. D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. Inf. Fusion 58, p. 82–115. External Links: Link Cited by: §A.1, §1.1. Beaglehole et al. (2024) D. Beaglehole, P. Súkeník, M. Mondelli, and M. Belkin Average gradient outer product as a mechanism for deep neural collapse. Advances in Neural Information Processing Systems 37, p. 130764–130796. Cited by: §C.3.3. Behboodi et al. (2022) A. Behboodi, G. Cesa, and T. S. Cohen A PAC-bayesian generalization bound for equivariant networks. Advances in Neural Information Processing Systems 35, p. 5654–5668. External Links: Link Cited by: §1.1. Bell et al. (2015) S. Bell, P. Upchurch, N. Snavely, and K. Bala Material recognition in the wild with the materials in context database. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3479–3487. Cited by: §C.2.3, §4. Brochard et al. (2021) A. Brochard, S. Zhang, and S. Mallat Generalized rectifier wavelet covariance models for texture synthesis. In International Conference on Learning Representations, External Links: Link Cited by: §C.2.3. Bruna and Mallat (2013) J. Bruna and S. Mallat Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence 35 (8), p. 1872–1886. Cited by: §A.2.2, §C.2.2, §C.2.2. Bruna and Mallat (2018) J. Bruna and S. Mallat Multiscale sparse microcanonical models. Mathematical Statistics and Learning 1 (3/4), p. 257–315. Cited by: §C.1, §3.2. Bruna (2022) J. Bruna The scattering transform. In Mathematical Aspects of Deep Learning, P. Grohs and G. Kutyniok (Eds.), p. 338–399. External Links: ISBN 978-1-009-02509-6 978-1-316-51678-2, Link, Document Cited by: §A.2.2. Chen et al. (2019) C. Chen, O. Li, C. Tao, A. J. Barnett, J. Su, and C. Rudin This looks like that: deep learning for interpretable image recognition. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Cited by: §A.1. Cimpoi et al. (2014) M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Cited by: §C.2.3, §4. Dana et al. (1999) K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink Reflectance and texture of real world surfaces. ACM Transactions on Graphics 18 (1), p. 1–34. Cited by: §4. E and Wojtowytsch (2022) W. E and S. Wojtowytsch On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, p. 270–290. External Links: ISSN 2640-3498, Link Cited by: §1.1. Fritz et al. (2004) M. Fritz, E. Hayman, B. Caputo, and J. O. Eklundh THE kth-tips database. In Computational Vision and Active Perception Laboratory, KTH, External Links: Link Cited by: §4. Gauthier et al. (2022) S. Gauthier, B. Thérien, L. Alsène-Racicot, M. Chaudhary, I. Rish, E. Belilovsky, M. Eickenberg, and G. Wolf Parametric scattering networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5749–5758. External Links: Link Cited by: §C.2.2, §C.2.2, §4. Gregor and LeCun (2010) K. Gregor and Y. LeCun Learning fast approximations of sparse coding. In International Conference on Machine Learning, External Links: Link Cited by: §A.1, §1.1. Guth et al. (2022) F. Guth, J. Zarka, and S. Mallat Phase collapse in neural networks. In International Conference on Learning Representations, Cited by: §C.2.2. Helber et al. (2019) P. Helber, B. Bischke, A. Dengel, and D. Borth Eurosat: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7), p. 2217–2226. Cited by: §C.2.3, §4. Jacot et al. (2025) A. Jacot, P. Súkeník, Z. Wang, and M. Mondelli Wide neural networks trained with weight decay provably exhibit neural collapse. In International Conference on Learning Representations, Vol. 2025, p. 84780–84806. Cited by: §C.3.3. Julesz (1962) B. Julesz Visual pattern discrimination. IRE Transactions on Information Theory 8 (2), p. 84–92. External Links: Document Cited by: §2.2, §3.1. Koh et al. (2020) P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, H. D. I and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, p. 5338–5348. External Links: Link Cited by: §A.1. Krengel and Brunel (2011) U. Krengel and A. Brunel Ergodic theorems. De Gruyter Studies in Mathematics, De Gruyter. External Links: ISBN 9783110844641, LCCN 85004457, Link Cited by: §3.1. Langford and Shawe-Taylor (2002) J. Langford and J. Shawe-Taylor PAC-bayes & margins. Advances in neural information processing systems 15. Cited by: §1.1. Lu and Steinerberger (2022) J. Lu and S. Steinerberger Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis 59, p. 224–241. External Links: ISSN 1063-5203, Link, Document Cited by: §1.1. Mallat (2009) S. Mallat A wavelet tour of signal processing. Academic Press. External Links: Document Cited by: §C.2.2. Mallat (2012) S. Mallat Group invariant scattering. Communications on Pure and Applied Mathematics 65 (10), p. 1331–1398. Cited by: §A.2.1, §A.2.2, §A.2.2, §A.2.3, §1.1, §2.2, §3.1, §3.1, Proposition 1. McAllester (1999) D. A. McAllester PAC-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, p. 164–170. Cited by: §1.1. McAllester (2003) D. McAllester Simplified pac-bayesian margin bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, p. 203–215. Cited by: §1.1. Mehta and Schwab (2014) P. Mehta and D. J. Schwab An exact mapping between the variational renormalization group and deep learning. External Links: 1410.3831, Link Cited by: §1. Mixon et al. (2022) D. G. Mixon, H. Parshall, and J. Pi Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis 20 (2), p. 11. External Links: ISSN 2730-5724, Link, Document Cited by: §1.1. Monga et al. (2021) V. Monga, Y. Li, and Y. C. Eldar Algorithm unrolling: interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine 38 (2), p. 18–44. External Links: Document Cited by: §A.1, §1.1, §1. Neyshabur et al. (2018) B. Neyshabur, S. Bhojanapalli, and N. Srebro A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations, External Links: Link Cited by: §1.1, Lemma 1. Nguyen et al. (2022) D. A. Nguyen, R. Levie, J. Lienen, E. Hüllermeier, and G. Kutyniok Memorization-dilation: modeling neural collapse under noise. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1.1. Olah et al. (2020) C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter Zoom in: an introduction to circuits. External Links: Link Cited by: §A.1. Oyallon and Mallat (2014) E. Oyallon and S. Mallat Deep roto-translation scattering for object classification. CoRR abs/1412.8659. External Links: Link, 1412.8659 Cited by: §C.2.2. Papyan et al. (2020) V. Papyan, X. Y. Han, and D. L. Donoho Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40), p. 24652–24663. External Links: Link, Document Cited by: §1.1. Papyan et al. (2017) V. Papyan, Y. Romano, and M. Elad Convolutional neural networks analyzed via convolutional sparse coding. Journal of Machine Learning Research 18 (83), p. 1–52. External Links: Link Cited by: §A.1, §1.1, §1. Portilla and Simoncelli (2000) J. Portilla and E. Simoncelli A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, p. . External Links: Document Cited by: §2.2, §3.1. Putra et al. (2025) M. A. Putra A. Harjoko et al. SuroTex: surrounding texture dataset. Data in Brief 59, p. 111292. Cited by: §C.2.3, §4. Rudin (2018) C. Rudin Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 1, p. 206 – 215. External Links: Link Cited by: §A.1, §1.1. Ryabko and Ryabko (2010) D. Ryabko and B. Ryabko Nonparametric statistical inference for ergodic processes. IEEE Transactions on Information Theory 56 (3), p. 1430–1435. External Links: Document Cited by: §2.2. Shalev-Shwartz and Ben-David (2014) S. Shalev-Shwartz and S. Ben-David Understanding machine learning: from theory to algorithms. Cambridge University Press. Cited by: §1.1. Simoncelli and Freeman (1995) E.P. Simoncelli and W.T. Freeman The steerable pyramid: a flexible architecture for multi-scale derivative computation. In Proceedings., International Conference on Image Processing, Vol. 3, p. 444–447 vol.3. External Links: Document Cited by: §C.2.2. Sulam et al. (2020) J. Sulam, A. Aberdam, A. Beck, and M. Elad On multi-layer basis pursuit, efficient algorithms and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, p. 1968–1980. External Links: Document Cited by: §1. Tirer and Bruna (2022) T. Tirer and J. Bruna Extended unconstrained features model for exploring deep neural collapse. In Proceedings of the 39th International Conference on Machine Learning, p. 21478–21505. External Links: ISSN 2640-3498, Link Cited by: §1.1. Tirer et al. (2023) T. Tirer, H. Huang, and J. Niles-Weed Perturbation analysis of neural collapse. In International Conference on Machine Learning, p. 34301–34329. Cited by: §C.3.3. Tishby and Zaslavsky (2015) N. Tishby and N. Zaslavsky Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26 - May 1, 2015, p. 1–5. External Links: Link, Document Cited by: §1. Tropp (2011) J. A. Tropp User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics 12 (4), p. 389–434. External Links: ISSN 1615-3383, Link, Document Cited by: §B.1.3. Viana and Oliveira (2016) M. Viana and K. Oliveira Ergodic decomposition. In Foundations of Ergodic Theory, Cambridge Studies in Advanced Mathematics, p. 142–156. Cited by: §3.1. Xu et al. (2020) K. Xu, J. Li, M. Zhang, S. S. Du, K. Kawarabayashi, and S. Jegelka What can neural networks reason about?. In International Conference on Learning Representations, External Links: Link Cited by: §A.1, §1. Yu et al. (2024) Y. Yu, S. Buchanan, D. Pai, T. Chu, Z. Wu, S. Tong, H. Bai, Y. Zhai, B. D. Haeffele, and Y. Ma White-box transformers via sparse rate reduction: compression is all there is?. Journal of Machine Learning Research 25 (300), p. 1–128. External Links: Link Cited by: §A.1, §1.1. Yu et al. (2023) Y. Yu, S. Buchanan, D. Pai, T. Chu, Z. Wu, S. Tong, B. Haeffele, and Y. Ma White-box transformers via sparse rate reduction. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 9422–9457. External Links: Document, Link Cited by: §1. Zarka et al. (2021) J. Zarka, F. Guth, and S. Mallat Separation and concentration in deep networks. In ICLR 2021-9th International Conference on Learning Representations, Cited by: §C.2.2, §C.2.2, §C.2.2, §C.2.3, §C.2.3, §C.2.3, §4. Appendix Appendix A Background and Preliminaries A.1 Additional related work on interpretability Interpretability methods can be divided into intrinsic approaches, which design models whose structure is intended to be understandable by construction, and post-hoc approaches, which analyze an already trained model [Rudin, 2018, Arrieta et al., 2019]. Our work belongs to the intrinsic setting and, more specifically, to the literature connecting neural networks with explicit computational algorithms [Monga et al., 2021, Gregor and LeCun, 2010, Papyan et al., 2017, Yu et al., 2024]. As discussed in the main text, our focus is on the algorithmic meaning of a particular learned parameter realization, rather than only the algorithmic structure shared by a parameterized family. Concept and prototype-based models. Some intrinsic approaches design neural networks whose decisions pass through human-interpretable intermediate representations. Concept bottleneck models predict predefined semantic concepts before producing the final prediction [Koh et al., 2020], while prototype-based networks classify examples through similarity to learned representative parts [Chen et al., 2019]. These approaches make the decision process interpretable in terms of concepts or examples; our work instead seeks to associate the network computation itself with an explicit algorithm. Algorithmic alignment. Our work is also related to algorithmic alignment, which studies the correspondence between the computational structure of a neural network and the decomposition of a target algorithm [Xu et al., 2020]. When such a correspondence exists, network modules can be associated with computational subroutines of the target algorithm, which may facilitate learning and generalization. Algorithmic alignment primarily concerns whether the organization of the network is compatible with the decomposition of an algorithmic computation. Such structural correspondence does not by itself guarantee an algorithmic interpretation of the fully trained network, since the computation it implements is determined jointly by its architecture and learned parameters. Our focus is complementary: we ask whether the particular parameter realization of the trained network retains an explicit algorithmic meaning. Mechanistic interpretability. Mechanistic interpretability seeks to reverse-engineer trained neural networks by identifying internal features, circuits, and computational mechanisms that give rise to their behavior [Olah et al., 2020]. It is closely related to our objective in that both aim to understand the computation performed by a particular trained model rather than only characterize the architecture at a generic level. The main difference is methodological. Mechanistic interpretability typically begins with an already trained model and attempts to recover its internal computation post hoc. Our approach instead begins from a parameter realization that implements a known algorithm and constrains subsequent learning so that the trained model remains related to this interpretable reference point. A.2 Wavelet and Scattering Preliminaries We review here the wavelet-based statistical construction used in our texture experiments, namely, the wavelet scattering transform. The scattering transform constructs a hierarchy of representations by cascading predefined wavelet convolutions with pointwise modulus nonlinearities along a collection of scattering paths, followed by spatial averaging. The resulting scattering coefficients form a finite family of statistics. Under the stationary ergodic model introduced in Section 3.1, these spatial averages provide estimators of statistics of the underlying generating process. A.2.1 Oriented wavelet transform We define the oriented wavelet transform following Mallat [2012], specialized to two-dimensional images and dyadic scales. Definition 2 (Wavelet Transform). Let ψ∈L2(ℝ2)ψ∈ L^2(R^2) be a complex function that we call a mother wavelet, and let ϕ∈L2(ℝ2)φ∈ L^2(R^2) be a complex function with ∫ϕ=1 φ=1, which we call a low-pass function. For an integer L≥1L≥ 1, let G=rkk=02L−1⊂SO(2)G=\r_k\_k=0^2L-1⊂ SO(2) be the rotations by the angles θk=kπ/L _k=kπ/L. A dilation by 2j2^j, j∈ℤj , and rotation by r∈Gr∈ G of ψ produce ψλ(u)=2−2jψ(2−jr−1u) _λ(u)=2^-2jψ(2^-jr^-1u), where we write λ=2jrλ=2^jr. For J∈ℤJ , the low-pass filter at scale 2J2^J is ϕJ(u)=2−2Jϕ(2−Ju) _J(u)=2^-2Jφ(2^-Ju); since ∫ϕJ=1 _J=1, convolution with ϕJ _J performs a weighted spatial average over a region of size proportional to 2J2^J. The wavelet transform at scale 2J2^J of a signal x∈L2(ℝ2)x∈ L^2(R^2) is the collection Jx=x∗ϕJ,(x∗ψλ)λ∈ΛJ,ΛJ=2jr:r∈G,j<J,W_Jx=\x* _J,(x* _λ)_λ∈ _J\, _J=\2^jr:r∈ G,\ j<J\, with norm ‖Jx‖2=‖x∗ϕJ‖2+∑λ∈ΛJ‖x∗ψλ‖2.\|W_Jx\|^2=\|x* _J\|^2+ _λ∈ _J\|x* _λ\|^2. The coefficient x∗ψλ(u)x* _λ(u) is complex. Its modulus measures the local response of x to structures at the scale 2j2^j and orientation indexed by λ, while its phase encodes the precise position of the oscillation. The scattering transform below retains the modulus, which varies slowly in space and can therefore be averaged stably, and recovers part of the information discarded with the phase through a cascade of further wavelet transforms. Symmetry for real-valued inputs. For Morlet wavelets, ψ(−u)=ψ(u)¯ψ(-u)= ψ(u). Since rk+L=−rkr_k+L=-r_k, this gives ψ2jrk+L=ψ2jrk¯ _2^jr_k+L= _2^jr_k, so for real-valued x we have |x∗ψ2jrk+L|=|x∗ψ2jrk||x* _2^jr_k+L|=|x* _2^jr_k|. All inputs to the scattering cascade below are real-valued (the image itself and the moduli computed from it), so in practice only the L orientations θk _k, k=0,…,L−1k=0,…,L-1, in [0,π)[0,π) are computed. Frame bounds. Recall that a family fkk∈Γ\f_k\_k∈ in a Hilbert space ℋH is a frame with bounds 0<A≤B0<A≤ B if A‖f‖2≤∑k∈Γ|⟨f,fk⟩|2≤B‖f‖2A\|f\|^2≤ _k∈ | f,f_k |^2≤ B\|f\|^2 for all f∈ℋf . For the wavelet transform, frame bounds are guaranteed by the Littlewood–Paley condition. Proposition 1 (Littlewood–Paley condition; Mallat 2012). If there exists ϵ∈[0,1)ε∈[0,1) such that, for almost all ω∈ℝ2ω ^2, 1−ϵ≤|ϕ^(2Jω)|2+12∑j<J∑r∈G|ψ^(2jr−1ω)|2≤1,1-ε≤| φ(2^Jω)|^2+ 12 _j<J _r∈ G| ψ(2^jr^-1ω)|^2≤ 1, then JW_J is a frame with bounds 1−ϵ1-ε and 11 on real-valued signals: (1−ϵ)‖x‖2≤‖Jx‖2≤‖x‖2,x∈L2(ℝ2)real-valued.(1-ε)\|x\|^2≤\|W_Jx\|^2≤\|x\|^2, x∈ L^2(R^2)\ real-valued. The lower bound ensures that JW_J retains all the information in x, and the upper bound makes JW_J non-expansive. Since the complex modulus is also non-expansive, the scattering transform defined below inherits this property (Appendix B.2). Discrete setting. In practice, images are defined on a finite grid, all convolutions are discrete, and the Littlewood–Paley condition is imposed on the discrete frequency grid. The wavelet transform is computed at finitely many scales 0≤j<J0≤ j<J, where j=0j=0 is the finest scale resolved by the grid. With a slight abuse of notation, we write ΛJ=2jrk: 0≤j<J, 0≤k<L _J=\2^jr_k:\ 0≤ j<J,\ 0≤ k<L\ for this finite index set of computed wavelets. A.2.2 Wavelet scattering transform The wavelet scattering transform constructs nonlinear multiscale statistics by cascading wavelet convolutions and complex modulus nonlinearities [Mallat, 2012]. We define it formally as follows. Definition 3 (Windowed Scattering Transform). For λ∈ΛJλ∈ _J, define the scattering operator U[λ]x:=|x∗ψλ|U[λ]x:=|x* _λ|. A sequence p=(λ1,…,λq)∈ΛJqp=( _1,…, _q)∈ _J^q is called a path of length q≥0q≥ 0; the empty path (q=0q=0) is denoted ∅ . The scattering propagator along p is defined by U[∅]x:=xU[ ]x:=x and U[p]x:=U[λq]⋯U[λ1]x=|U[(λ1,…,λq−1)]x∗ψλq|.U[p]x:=U[ _q]·s U[ _1]x= |U[( _1,…, _q-1)]x* _ _q |. The windowed scattering coefficient associated with p is (7) SJ[p]x:=U[p]x∗ϕJ,S_J[p]x:=U[p]x* _J, and, for a set of paths P, the windowed scattering transform is the collection SJ(x):=(SJ[p]x)p∈S_J(x):= (S_J[p]x )_p . We use the standard choice of frequency-decreasing paths of length at most qmaxq_ , J:=(2j1rk1,…,2jqrkq)∈ΛJq: 0≤q≤qmax,j1<j2<⋯<jq.P_J:= \(2^j_1r_k_1,…,2^j_qr_k_q)∈ _J^q:\ 0≤ q≤ q_ ,\ j_1<j_2<·s<j_q \. Since U[λ]xU[λ]x has most of its energy at frequencies below the central frequency of ψλ _λ, paths along which the scale does not increase carry little energy and are discarded [Bruna and Mallat, 2013]. In the discrete setting, JP_J is finite. Unless stated otherwise, =JP=P_J and SJ(x)=(SJ[p]x)p∈JS_J(x)=(S_J[p]x)_p _J. Under the Littlewood–Paley condition, the windowed scattering transform over all paths is non-expansive [Mallat, 2012, Proposition 2.5], [Bruna, 2022, Proposition 8.3]. Hence, so is its restriction to any subset of paths, such as JP_J: ‖SJ(x)−SJ(x′)‖≤‖x−x′‖.\|S_J(x)-S_J(x )\|≤\|x-x \|. A.2.3 Scattering coefficients as statistics of an ergodic process. The connection with the stochastic model in Section 2.2 is immediate. Each scattering path defines a particular function of a random field. For a path p, define (8) fp(X):=U[p]X(0).f_p(X):=U[p]X(0). If X is stationary with law Q, stationarity implies that Q[U[p]X(u)]E_Q[U[p]X(u)] does not depend on u. The corresponding statistic of the process is therefore (9) S¯Q[p]:=Q[U[p]X(0)]=Q[fp(X)]. S_Q[p]:=E_Q[U[p]X(0)]=E_Q[f_p(X)]. The collection fp:p∈\f_p:p \ is thus precisely a particular finite choice of the functions used to define statistics in the ergodic model of the main text. The coefficient SJ[p]x(u)S_J[p]x(u) in Equation 7 is a weighted spatial average of this function over translations of the observed realization. Indeed, using the translation action defined in the main text, it can be viewed as a finite weighted approximation to the ergodic average of fpf_p. If ϕJ _J has sufficiently wide support, then for an ergodic process this spatial average approximates S¯Q[p]. S_Q[p]. Hence, the scattering network computes from a single finite texture image estimates of a prescribed collection of statistics of its underlying generating process. Mallat introduced this stochastic interpretation explicitly through the expected scattering transform, whose coefficients are Q[U[p]X]E_Q[U[p]X] [Mallat, 2012]. Unlike the power spectrum, which depends only on second-order information, scattering coefficients capture nonlinear higher-order dependencies through the cascade of wavelet transforms and modulus operations. The predefined scattering network therefore specifies both the functions whose expectations are measured and the algorithm used to estimate them from a finite realization. Appendix B Theoretical Analysis and Proofs B.1 PAC-Bayesian Analysis Around a Data-Independent Initialization We formalize the statistical role of restricting fine-tuning around Θ0 _0 using a PAC-Bayesian analysis. Since Θ0 _0 is specified by the predefined disentangling algorithm and is independent of the training sample, it can serve as a data-independent reference point in a PAC-Bayesian analysis. We consider predictors obtained by fine-tuning within a neighborhood of this realization. For R>0R>0, define BR(Θ0):=Θ:‖Θ−Θ0‖2≤R.B_R( _0):= \ : - _0 _2≤ R \. Under a suitable local stability condition on parameter perturbations, we derive a generalization bound that holds uniformly over BR(Θ0)B_R( _0). Crucially, the bound depends on the distance from the reference initialization, but not on the predictive performance of FΘ0F_ _0 itself. B.1.1 Margin risks and PAC-Bayesian margin bound Let D denote the distribution of labeled pairs (x,y)∈×[C](x,y) ×[C], whose marginal on X is the data distribution P in Equation 1, and let (xn,yn)n=1M\(x_n,y_n)\_n=1^M be a training set of M i.i.d. samples from D. For a predictor FΘ:→ℝCF_ :X ^C, recall the classification margin mΘ(x,y)=FΘ(x)[y]−maxj≠yFΘ(x)[j].m_ (x,y)=F_ (x)[y]- _j≠ yF_ (x)[j]. For γ≥0γ≥ 0, the statistical and empirical γ-margin risks are Rγ(FΘ):=ℙ(x,y)∼[mΘ(x,y)≤γ],R^γ(FΘ):=1M∑n=1M[mΘ(xn,yn)≤γ].R_γ(F_ ):=P_(x,y) [m_ (x,y)≤γ ], R_γ(F_ ):= 1M _n=1^M 1 [m_ (x_n,y_n)≤γ ]. In particular, R0(FΘ)R_0(F_ ) is the statistical classification risk, with ties counted as errors. We begin from the following standard PAC-Bayesian margin bound. Lemma 1 (Neyshabur et al. 2018, Lemma 1). Let FΘ:→ℝCF_ :X ^C be any predictor, parameterized by Θ . Let π be any prior distribution over the parameter space that is independent of the training data. Then, for any γ>0,δ∈(0,1)γ>0,δ∈(0,1), with probability at least 1−δ1-δ over the draw of a training set of size M, the following holds. For any Θ and any random perturbation U such that ℙU[supx∈‖FΘ+U(x)−fΘ(x)‖∞<γ4]≥12,P_U [ _x F_ +U(x)-f_ (x) _∞< γ4 ]≥ 12, we have R0(FΘ)≤R^γ(FΘ)+4KL(∥π)+ln6MδM−1,R_0(F_ )≤ R_γ(F_ )+4 KL(Q\,\|\,π)+ 6MδM-1, where =Law(Θ+U)Q=Law( +U) denotes the posterior distribution. Lemma 1 requires the predictor to be sufficiently insensitive to a random perturbation of its parameters. To verify this condition uniformly for predictors in a neighborhood of the algorithmically interpretable realization Θ0 _0, we introduce a deterministic local stability condition under bounded parameter perturbations. B.1.2 Local perturbation stability Definition 4 (Local Perturbation Stability). Let ℱ=FΘ:→ℝCΘ∈F=\F_ :X ^C\_ be a parametrized hypothesis family, where Θ=vec(Wii=1d). =vec (\W_i\_i=1^d ). For Θ0∈ _0 , R>0R>0, and τ>0τ>0, define BR(Θ0):=Θ∈∣‖Θ−Θ0‖2≤R,B_R( _0):=\ - _0 _2≤ R\\, and τ:=U=vec(Uii=1d):maxi∈[d]‖Ui‖2≤τ.U_τ:=\U=vec (\U_i\_i=1^d ): _i∈[d] U_i _2≤τ\. We say that the family ℱF is (R,τ,ϵR,τ,ε)-stable under bounded parameter perturbations around Θ0 _0 if, for every Θ∈BR(Θ0) ∈ B_R( _0) and every U∈τU _τ, supx∈‖FΘ+U(x)−FΘ(x)‖∞<ϵ. _x F_ +U(x)-F_ (x) _∞<ε. The stability condition allows the perturbation requirement in Lemma 1 to be verified uniformly over a neighborhood of the data-independent realization Θ0 _0. This leads to the following bound. B.1.3 Uniform generalization bound Corollary 2 (Uniform Generalization Bound Around a Data-Independent Initialization). Let ℱF be a parametrized hypothesis family, and let Θ0=vec(Wii=1d) _0=vec(\W_i\_i=1^d) be an initialization independent of the training data, where Wi∈ℝmi×niW_i ^m_i× n_i, and define h:=maxi∈[d]mi,ni.h:= _i∈[d]\m_i,n_i\. Fix R,γ,τ>0R,γ,τ>0, and suppose that ℱF is (R,τ,γ4R,τ, γ4)-stable under bounded parameter perturbations around Θ0 _0. Then, for any δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ over the draw of a training set of size M, the following holds simultaneously for all Θ∈BR(Θ0) ∈ B_R( _0): (10) R0(FΘ)≤R^γ(FΘ)+4(R2hln(4dh)τ2+ln6Mδ)/(M−1)R_0(F_ )≤ R_γ(F_ )+4 ( R^2h (4dh)τ^2+ 6Mδ )/ (M-1 ) Proof. Set σ=τ2hln(4dh)σ= τ 2h (4dh) and choose the data-independent prior π=(Θ0,σ2I).π=N( _0,σ^2I). For any Θ∈BR(Θ0) ∈ B_R( _0), define the posterior Θ=(Θ,σ2I).Q_ =N( ,σ^2I). A draw from ΘQ_ can be written as Θ+U +U, where U∼(0,σ2I)U (0,σ^2I). Write U=vec(Uii=1d),U=vec(\U_i\_i=1^d), where Ui∈ℝmi×niU_i ^m_i× n_i. By the Gaussian spectral-norm bound [Tropp, 2011, Eq. 4.6], together with mi,ni≤hm_i,n_i≤ h, for every i∈[d]i∈[d] and t≥0t≥ 0, ℙ(‖Ui‖2>t)≤2hexp(−t22hσ2).P\! (\|U_i\|_2>t )≤ 2h \! (- t^22hσ^2 ). Applying a union bound with t=τt=τ gives ℙ(maxi∈[d]‖Ui‖2≤τ)≥1−2dhexp(−τ22hσ2).P\! ( _i∈[d]\|U_i\|_2≤τ )≥ 1-2dh \! (- τ^22hσ^2 ). By the choice of σ, 2dhexp(−τ22hσ2)=12,2dh \! (- τ^22hσ^2 )= 12, and hence ℙ(maxi∈[d]‖Ui‖2≤τ)≥12.P\! ( _i∈[d]\|U_i\|_2≤τ )≥ 12. Since ℱF is (R,τ,γ4)(R,τ, γ4)-stable around Θ0 _0, for every Θ∈BR(Θ0) ∈ B_R( _0), the event maxi∈[d]‖Ui‖2≤τ _i∈[d]\|U_i\|_2≤τ implies supx∈‖FΘ+U(x)−FΘ(x)‖∞<γ4. _x \|F_ +U(x)-F_ (x)\|_∞< γ4. Therefore, ℙU[supx∈∥FΘ+U(x)−FΘ(x)∥∞<γ4]≥12,P_U\! [ _x \|F_ +U(x)-F_ (x)\|_∞< γ4 ]≥ 12, so the perturbation condition in Lemma 1 is satisfied. Because π and ΘQ_ are Gaussian distributions with the same covariance, KL(Θ∥π)=‖Θ−Θ0‖222σ2.KL(Q_ \|π)= \| - _0\|_2^22σ^2. Since Θ∈BR(Θ0) ∈ B_R( _0), KL(Θ∥π)≤R22σ2=R2hln(4dh)τ2.KL(Q_ \|π)≤ R^22σ^2= R^2h (4dh)τ^2. Substituting this bound into Lemma 1 yields R0(FΘ)≤R^γ(FΘ)+4R2hln(4dh)τ2+ln6MδM−1.R_0(F_ )≤ R_γ(F_ )+4 R^2h (4dh)τ^2+ 6MδM-1. On the event of Lemma 1, the bound holds for every Θ whose perturbation satisfies the required condition. Since the above argument verifies this condition for every Θ∈BR(Θ0) ∈ B_R( _0), the bound holds simultaneously for all Θ∈BR(Θ0) ∈ B_R( _0). ∎ B.2 Local Perturbation Stability of Scattering We verify here that the predefined scattering realization satisfies the local parameter-stability condition used in Corollary 1. We first establish local parameter stability of the predefined scattering feature extractor. We then show that this stability extends to the full predictor obtained by composing the scattering representation with a linear classifier, as used in our experiments. Let P be a collection of P scattering paths, each of length at most L, and write Θ0=vec(Wi(p)p∈,i∈[dp]) _0=vec (\W_i^(p)\_p ,\,i∈[d_p] ) for the predefined scattering parameters, where dpd_p denotes the length of path p. We assume that the predefined wavelet operators and the final low-pass operator are non-expansive, ‖Wi(p)‖2≤1,‖Φ‖2≤1.\|W_i^(p)\|_2≤ 1, \| \|_2≤ 1. Let SΘS_ denote the corresponding scattering network obtained by parameterizing the wavelet operators by Θ , while keeping the modulus and final low-pass operations fixed. Lemma 2 (Local perturbation stability of scattering). Let =x:‖x‖2≤B.X=\x:\|x\|_2≤ B\. Then, for any R,τ≥0R,τ≥ 0, the scattering family SΘ\S_ \ is (R,τ,ϵ)(R,τ,ε)-stable around Θ0 _0 in the sense of Definition 4, with ϵ=PBLτ(1+R+τ)L−1.ε= P\,BLτ(1+R+τ)^L-1. In particular, for any γ>0γ>0, let aγ:=γ4PBL,τγ:=aγ(1+R+aγ)L−1.a_γ:= γ4 PBL, _γ:= a_γ(1+R+a_γ)^L-1. Then SΘ\S_ \ is (R,τγ,γ4)(R, _γ, γ4)-stable under bounded parameter perturbations around Θ0 _0. Proof. Fix Θ∈BR(Θ0) ∈ B_R( _0) and U∈τU _τ. For each path p∈p , write Θ(p)=Vi(p)i=1dp,U(p)=Ui(p)i=1dp. ^(p)=\V_i^(p)\_i=1^d_p, U^(p)=\U_i^(p)\_i=1^d_p. Since ‖Θ−Θ0‖2≤R\| - _0\|_2≤ R, each parameter block satisfies ‖Vi(p)−Wi(p)‖2≤‖Vi(p)−Wi(p)‖F≤R.\|V_i^(p)-W_i^(p)\|_2≤\|V_i^(p)-W_i^(p)\|_F≤ R. Using ‖Wi(p)‖2≤1\|W_i^(p)\|_2≤ 1, we therefore obtain (11) ‖Vi(p)‖2≤1+R,‖Vi(p)+Ui(p)‖2≤1+R+τ.\|V_i^(p)\|_2≤ 1+R, \|V_i^(p)+U_i^(p)\|_2≤ 1+R+τ. Consider a fixed path p of length dpd_p. Define z0=z~0=x,z_0= z_0=x, and recursively zi=ρ(Vi(p)zi−1),z~i=ρ((Vi(p)+Ui(p))z~i−1),z_i=ρ(V_i^(p)z_i-1), z_i=ρ((V_i^(p)+U_i^(p)) z_i-1), where ρ(z)=|z|ρ(z)=|z| is applied entrywise. Since the complex modulus is non-expansive, ‖z~i−zi‖2 \| z_i-z_i\|_2 ≤‖(Vi(p)+Ui(p))z~i−1−Vi(p)zi−1‖2 ≤\|(V_i^(p)+U_i^(p)) z_i-1-V_i^(p)z_i-1\|_2 (12) ≤‖Vi(p)+Ui(p)‖2‖z~i−1−zi−1‖2+‖Ui(p)‖2‖zi−1‖2. ≤\|V_i^(p)+U_i^(p)\|_2\| z_i-1-z_i-1\|_2+\|U_i^(p)\|_2\|z_i-1\|_2. Moreover, by (11), ‖zi−1‖2≤(1+R)i−1‖x‖2.\|z_i-1\|_2≤(1+R)^i-1\|x\|_2. Hence ‖z~i−zi‖2≤(1+R+τ)‖z~i−1−zi−1‖2+τ(1+R)i−1‖x‖2.\| z_i-z_i\|_2≤(1+R+τ)\| z_i-1-z_i-1\|_2+τ(1+R)^i-1\|x\|_2. Since z~0=z0 z_0=z_0, iterating this inequality yields ‖z~dp−zdp‖2 \| z_d_p-z_d_p\|_2 ≤τ‖x‖2∑j=0dp−1(1+R+τ)dp−1−j(1+R)j ≤τ\|x\|_2 _j=0^d_p-1(1+R+τ)^d_p-1-j(1+R)^j (13) ≤dpτ(1+R+τ)dp−1‖x‖2. ≤ d_pτ(1+R+τ)^d_p-1\|x\|_2. The final low-pass operator is non-expansive, so ‖SΘ+U[p](x)−SΘ[p](x)‖2≤dpτ(1+R+τ)dp−1‖x‖2.\|S_ +U[p](x)-S_ [p](x)\|_2≤ d_pτ(1+R+τ)^d_p-1\|x\|_2. Using dp≤Ld_p≤ L and ‖x‖2≤B\|x\|_2≤ B gives ‖SΘ+U[p](x)−SΘ[p](x)‖2≤BLτ(1+R+τ)L−1.\|S_ +U[p](x)-S_ [p](x)\|_2≤ BLτ(1+R+τ)^L-1. Finally, stacking the outputs of the P scattering paths gives ‖SΘ+U(x)−SΘ(x)‖∞ \|S_ +U(x)-S_ (x)\|_∞ ≤‖SΘ+U(x)−SΘ(x)‖2 ≤\|S_ +U(x)-S_ (x)\|_2 ≤(∑p∈‖SΘ+U[p](x)−SΘ[p](x)‖22)1/2 ≤ ( _p \|S_ +U[p](x)-S_ [p](x)\|_2^2 )^1/2 (14) ≤PBLτ(1+R+τ)L−1. ≤ P\,BLτ(1+R+τ)^L-1. This bound holds uniformly over x∈x , Θ∈BR(Θ0) ∈ B_R( _0), and U∈τU _τ, proving the result. It remains to verify the final claim. Since (1+R+aγ)L−1≥1,(1+R+a_γ)^L-1≥ 1, we have τγ≤aγ _γ≤ a_γ. Therefore, 1+R+τγ≤1+R+aγ.1+R+ _γ≤ 1+R+a_γ. Substituting the definition of τγ _γ into the perturbation bound gives ϵ ε =PBLτγ(1+R+τγ)L−1 = P\,BL _γ(1+R+ _γ)^L-1 ≤PBLaγ(1+R+aγ)L−1(1+R+aγ)L−1 ≤ P\,BL a_γ(1+R+a_γ)^L-1(1+R+a_γ)^L-1 (15) =PBLaγ=γ4. = P\,BLa_γ= γ4. ∎ Corollary 3 (Stability with a linear classifier). Let FΘ,A(x)=ASΘ(x),F_ ,A(x)=AS_ (x), and let (Θ0,A0)( _0,A_0) denote the initialization of the full predictor. Assume that A0A_0 is chosen independently of the training sample. Then the family FΘ,A\F_ ,A\ is locally parameter-stable around (Θ0,A0)( _0,A_0). More precisely, for ‖(Θ,A)−(Θ0,A0)‖2≤R\|( ,A)-( _0,A_0)\|_2≤ R and perturbations ‖(U,V)‖2≤τ,\|(U,V)\|_2≤τ, we have ‖FΘ+U,A+V(x)−FΘ,A(x)‖∞≤(‖A0‖2+R)ϵS+τPB(1+R+τ)L,\|F_ +U,A+V(x)-F_ ,A(x)\|_∞≤(\|A_0\|_2+R) _S+τ P\,B(1+R+τ)^L, where ϵS=PBLτ(1+R+τ)L−1 _S= P\,BLτ(1+R+τ)^L-1 is the perturbation bound of Lemma 2. Proof. Using FΘ+U,A+V(x)−FΘ,A(x)=A(SΘ+U(x)−SΘ(x))+VSΘ+U(x),F_ +U,A+V(x)-F_ ,A(x)=A (S_ +U(x)-S_ (x) )+VS_ +U(x), we obtain ‖FΘ+U,A+V(x)−FΘ,A(x)‖∞≤‖A‖2‖SΘ+U(x)−SΘ(x)‖2+‖V‖2‖SΘ+U(x)‖2.\|F_ +U,A+V(x)-F_ ,A(x)\|_∞≤\|A\|_2\|S_ +U(x)-S_ (x)\|_2+\|V\|_2\|S_ +U(x)\|_2. Since ‖A‖2≤‖A0‖2+R\|A\|_2≤\|A_0\|_2+R and ‖SΘ+U(x)‖2≤PB(1+R+τ)L,\|S_ +U(x)\|_2≤ P\,B(1+R+τ)^L, the result follows from Lemma 2. ∎ Appendix C Experimental Details and Results C.1 Texture Synthesis from Scattering Statistics We use the microcanonical gradient-descent approach of Bruna and Mallat [2018] to synthesize textures whose prescribed statistics match those of a reference image. The purpose of this experiment is to qualitatively assess how much perceptually relevant texture information is retained by the scattering statistics used in our disentangler. Let x be a reference texture. We compute the two-dimensional scattering transform and spatially average each scattering coefficient. Denoting this empirical statistic vector by S^J(x)=AvguSJ(x)(u), S_J(x)=Avg_uS_J(x)(u), texture synthesis seeks an image z whose statistics are close to those of x. This spatial averaging is the finite-image counterpart of the process statistics discussed in Section 3.1. In the implementation, the averaging is performed directly over the spatial coordinates of the scattering coefficients. For grayscale images, we minimize (16) ℒscat(z)=1D‖S^J(z)−S^J(x)‖22,L_scat(z)= 1D \| S_J(z)- S_J(x) \|_2^2, where D is the dimension of the scattering statistic vector. For RGB images, we additionally preserve the correlations between color channels. Let C(x)∈ℝ3×3C(x) ^3× 3 denote the correlation matrix of the centered and standardized RGB channels. We then optimize (17) ℒsyn(z)=ℒscat(z)+‖C(z)−C(x)‖F2.L_syn(z)=L_scat(z)+ \|C(z)-C(x) \|_F^2. Thus, the synthesis procedure constrains the generated image through the selected scattering statistics, together with a simple color constraint for RGB textures. This is the objective implemented in our synthesis procedure. The optimization is initialized from an independent random image. More precisely, each channel is initialized with Gaussian noise having the same mean and standard deviation as the corresponding channel of the reference image. We optimize the image pixels using L-BFGS with a strong-Wolfe line search while keeping the scattering transform fixed. The scattering coefficients are computed using the two-dimensional scattering implementation in Kymatio [Andreux et al., 2020]. After optimization, we apply channel-wise histogram matching between the synthesized and reference images for visualization. This step matches the marginal intensity distributions of the synthesized image to those of the reference and is not part of the optimization objective above. Because the optimization starts from an independent random image, similarity between the synthesized and reference textures cannot be attributed to copying their spatial arrangements. Rather, it indicates that matching the selected statistics is sufficient to recover characteristic perceptual structures of the reference texture. We therefore use these synthesis results as qualitative evidence that the chosen scattering statistics retain meaningful texture information. C.2 Texture Classification Setup C.2.1 Datasets and Sampling To evaluate sample efficiency, we train on class-stratified subsets ranging from the full training set down to 0.2%0.2\% for MinC-2500, 0.5%0.5\% for EuroSAT, 10%10\% for DTD, and 2%2\% for SuroTex. DTD has ten official train/val/test partitions of its 5,640 images (47 classes with 120 examples each); each partition allocates 40 images per class to each split; we merge the the offiical train and val halves of a partition into a single 3,760 image pool and re-split it with an 80/20 train/val split, using class-stratified splitting. The official 1,880 image test split is not modified. For DTD, we evaluate five of the ten standard partitions to balance computational cost with robustness across dataset splits. The remaining three datasets have no official train/val/test split, so we construct splits as follows. First, we separate a class-stratified test split, taking 15% of the data for MinC-2500 and 20% for SuroTex and EuroSat. The remaining data is split 80/20 into training and validation sets. Subsampling is applied only to the training dataset, so the validation and test sets are identical across training dataset sizes. At the smallest training size, the model is trained on 4 images per class for MinC-2500, 8-9 for EuroSAT, 6 for DTD, and a single image per class for SuroTex. Validation accuracy is used for checkpoint selection and early stopping, and all reported numbers from from the held out test set. C.2.2 Model Architectures and Baselines We next describe the model architectures used in our experiments, including our trainable scattering network, the three baselines, and the common classification head, followed by a discussion of Parametric Scattering Networks [Gauthier et al., 2022]. Our trainable scattering architecture. We describe the convolutional implementation used in our experiments to realize the predefined scattering transform as a trainable neural network. The construction is designed so that, at Θ=Θ0 = _0, the network implements the scattering transform exactly, while its wavelet operators are represented as spatial convolutional parameters that can subsequently be fine-tuned. Spatial implementation of scattering. Standard implementations such as Kymatio [Andreux et al., 2020] compute scattering coefficients efficiently using fixed wavelet filters in the Fourier domain. For our purpose, we instead require the wavelet operations to appear explicitly as trainable convolutional layers. We therefore implement scattering in the spatial domain using steerable Morlet wavelets [Simoncelli and Freeman, 1995, Bruna and Mallat, 2013] together with the fast multiscale wavelet transform [Mallat, 2009]. Let gkk=0L\g_k\_k=0^L denote the one-step filter bank at the finest scale j=0j=0. Here g0g_0 is the low-pass filter, while g1,…,gLg_1,…,g_L are band-pass Morlet filters, gkg_k having orientation θk−1=(k−1)π/L _k-1=(k-1)π/L as in Definition 2. For an input x, define the scattering block Tx(u,k)=(x∗gk)(2u),Tx(u,k)=(x*g_k)(2u), so that each step consists of filtering followed by subsampling by a factor of two. Repeated application of this operation implements the multiscale transform. Because the input is subsampled after each step, applying the same finest-scale filters at the next resolution corresponds to filtering at a coarser scale in the coordinates of the original image. This avoids explicitly constructing larger spatial filters for successive dyadic scales. In this sense, the multiscale filters of Definition 2 are the equivalent filters of the cascade: j low-pass steps with g0g_0 followed by gkg_k realize the wavelet ψ2jrk−1 _2^jr_k-1, and J low-pass steps realize the low-pass filter ϕJ _J, for 0≤j<J0≤ j<J. Scattering cascade. To obtain the scattering transform, the fast wavelet cascade is propagated along the band-pass branches. For an intermediate representation z, the low-pass response (z∗g0)(2u)(z*g_0)(2u) produces an averaged component, while each band-pass response is followed by the complex modulus, (z∗gk)(2u)⟼|(z∗gk)(2u)|,k=1,…,L,(z*g_k)(2u) |(z*g_k)(2u)|, k=1,…,L, before being passed to subsequent blocks. Following standard neural network architectures, we include an optional batch normalization layer between blocks, immediately following the complex modulus. The inclusion of the batch normalization layer was part of the architecture sweep and is reported in Table 1. A path p=(λ1,…,λq)∈Jp=( _1,…, _q) _J with λi=2jirki _i=2^j_ir_k_i is realized by applying the band-pass filter gki+1g_k_i+1, followed by the modulus, at block ji+1j_i+1, and the low-pass filter g0g_0 at all other blocks. This recursively realizes the wavelet-modulus cascade of Definition 3. Since j1<⋯<jqj_1<·s<j_q, only frequency-decreasing paths are computed [Bruna and Mallat, 2013]. Cascading the blocks therefore computes the scattering tree using small spatial kernels and progressively subsampled feature maps. Convolutional realization and initialization. Suppose a scattering layer receives CinC_in input channels. In the classical scattering transform, the wavelet filter bank is applied independently to each channel. We realize this operation with a convolutional layer whose weights at initialization satisfy K(c,k),c=gk,K(c,k),c′=0for c′≠c,K_(c,k),c=g_k, K_(c,k),c =0 c ≠ c, where (c,k)(c,k) indexes the output obtained by applying filter gkg_k to input channel c. The resulting convolutional operator is block diagonal across channels and therefore reproduces independent wavelet filtering exactly. Together with the prescribed modulus, low-pass, and subsampling operations, this defines the scattering realization Θ0 _0. Learnable channel mixing. Classical scattering does not mix information across different feature channels. Motivated by previous learnable scattering architectures [Zarka et al., 2021, Oyallon and Mallat, 2014, Guth et al., 2022], we enlarge the parameterization so that each output may receive contributions from a prescribed neighborhood of input channels. Let (c)N(c) denote the channels allowed to contribute to descendants of channel c. The trainable operation takes the form zc,k(u)=∑c′∈(c)(xc′∗K(c,k),c′)(2u).z_c,k(u)= _c (c)(x_c *K_(c,k),c )(2u). At initialization, the off-diagonal kernels remain zero, K(c,k),c=gk,K(c,k),c′=0for c′≠c,K_(c,k),c=g_k, K_(c,k),c =0 c ≠ c, so the enlarged network still implements the predefined scattering transform exactly. During fine-tuning, both the wavelet kernels and the permitted off-diagonal kernels may change, allowing the representation to adapt to the classification task. We restrict the mixing neighborhood rather than using fully dense channel interactions in order to limit the number of trainable parameters and preserve the local organization inherited from the scattering tree. Figure 4 illustrates the resulting architecture. Thus, Θ0 _0 corresponds exactly to the predefined scattering computation, whereas fine-tuning moves within a larger convolutional parameterization initialized at this algorithmic realization. Figure 4: Trainable scattering architecture. At initialization, the convolutional operators reproduce independent wavelet filtering and hence the predefined scattering transform. Fine-tuning allows the wavelet filters and permitted cross-channel interactions to adapt from data. Fixed and random-initialization baselines. The fixed scattering baseline is the predefined realization Θ0 _0 used without fine-tuning. The random-initialization baseline uses the same trainable architecture as our model, but initializes its convolutional parameters randomly instead of from the wavelet filters. ScatNet B.A. We also compare with the learnable scattering architecture of Zarka et al. [2021], denoted ScatNet B.A. Its representation is computed as SC=∏j=1J(FjTρtFj)(PjBNρrFw).S_C= _j=1^J (F_j^T _tF_j ) (P_j\,BN\, _rF_w ). Here, FwF_w denotes the fixed spatial wavelet transform, ρr _r the scattering nonlinearity, and BNBN channel-wise normalization. The learned 1×11× 1 projection PjP_j reduces the scattering channels, while FjF_j is a learned 1×11× 1 convolutional tight frame. The latter expands the channel representation, applies the soft-thresholding ρt _t, and projects it back through FjTF_j^T. Thus, the spatial wavelet filters remain fixed, while learning acts on transformations across the scattering channels. Similar to our method, ScatNet B.A. includes batch normalization layers between scattering blocks, which improves training stability. The interpretation of these learned operators in terms of separation and concentration is developed in Zarka et al. [2021]. Classification head. For all models, the final representation is passed through batch normalization and then to a single-layer linear classifier. When needed, global average pooling is applied before the classifier to reduce the spatial dimensions. Under the hypothesis that the method is a feature extractor which successfully disentangles representations between classes, a simple, linear classifier is sufficient to classify examples from their features. And the batch normalization layers are included for optimization stability and consistency with standard training practice. Parametric Scattering Networks. Gauthier et al. [2022] propose another approach to learnable scattering in which each wavelet is constrained to remain a Morlet wavelet and learning is restricted to its scale, orientation, frequency, and aspect-ratio parameters, rather than directly optimizing the values of the convolutional kernels. Thus, the scattering filters are adapted through a low-dimensional parameterization with only a small number of trainable degrees of freedom. In contrast, our model treats the spatial convolutional kernels as trainable parameters and additionally permits cross-channel interactions, resulting in a substantially more flexible, CNN-like parameterization. Because our experiments focus on fine-tuning an overparameterized convolutional network around the exact scattering realization, rather than a low-dimensional wavelet parameterization, we do not include the parametric Morlet model as a primary baseline. C.2.3 Training, Evaluation, and Reproducibility We review our training procedure and provide information to reproduce our numerical results, including hyperparameters and random seeds. The implementation is publicly available on GitHub.22 2 https://github.com/yangl1n/specific_algo_inter_on_texture Training Procedure The training procedure differs slightly between baseline methods. Let θψ _ψ denote the parameters contained in the filters of the CNN (either the scattering filters or randomly initialized ones), and θc _c denote the parameters contained in the classifier. Let hθh_θ denote the model with such parameters and xn,ynn=1N\x_n,y_n\_n=1^N be the set of training data. For the randomly initialized model, the model was trained using standard, unconstrained gradient descent. The minimization problem was min∑nθψ,θcL(hθ(xn),yn) _ _ψ, _c _nL(h_θ(x_n),y_n). For the initial scattering model, the filters were frozen and only the classifier head was trained. The minimization problem was min∑nθcL(hθ(xn),yn) _ _c _nL(h_θ(x_n),y_n). For the fine-tuned scattering model, the training proceeds in two phases. The first phase is identical to the training for the initial scattering model. The second phase allows end-to-end training but introduces a regularization term to penalize distance that the parameters move from their initial values. The minimization problem in the second phase is min∑nθψ,θcL(hθ(xn),yn)+λ‖θψ−θψ,0‖2 _ _ψ, _c _nL(h_θ(x_n),y_n)+λ\| _ψ- _ψ,0\|_2 where θψ,0 _ψ,0 denotes the initial parameters of the scattering transform and λ controls the strength of the fine-tuning regularization. For the ScatNet B.A., the training procedure from Zarka et al. [2021] was used. This is analogous to the training of the initial scattering transform, and the minimization problem is min∑nθcL(hθ(xn),yn) _ _c _nL(h_θ(x_n),y_n). Architecture and Hyperparameters The model architecture used in experiments differs between datasets. Architecture parameters including depth of network, number of wavelet orientations, the number of channels that were mixed by each wavelet filter, kernel size, per-block batch normalization, and whether a low-pass filter and global averaging are applied before the classifier, were chosen by a sweep over hyperparameters on each dataset. The learning rate was also tuned; separate learning rates are used for the classifier and the feature extractor. All four reported models use a kernel size of 5 pixels, 4 wavelet orientations, a complex modulus as the linearity (as opposed to ReLU), and include a low-pass filter before passing features to the classifier. In the sweep, larger kernels, more wavelet orientations, and removing the final low-pass filter each increased accuracy on their own on some datasets, but by less than adding depth did, or by an amount that could not be separated from seed noise, while increasing the parameter count drastically, so none of them were adopted. Depth (4 to 6 blocks), mixing width, global averaging, and per-block batch normalization vary by dataset. The feature-extractor learning rate is 1.0×10−41.0× 10^-4 on every dataset. The classifier learning rate 0.0010.001 on DTD, MinC-2500, and EuroSAT; on SuroTex, a learning-rate sweep selected 0.010.01. The classifier’s learning rate is reduced by half during fine-tuning. For all datasets, a batch normalization layer was included between the scattering transform and the classifier, and a single fully-connected linear layer was used as the classifier head. The randomly initialized network shares an architecture with the proposed method’s architecture, so they are reported as a single row. It is trained jointly in a single phase, with one learning rate for all parameters. For the baseline method ScatNet B.A., none of the datasets reported here were used in Zarka et al. [2021]; we took the architecture used for ImageNet as a starting point and performed a sweep over architecture and hyperparameters. The following parameters were found to perform the best on all datasets: using 8 wavelet orientations, a kernel size of 1 for projection layers and frame layers, and applying a batch norm layer between scattering operators. We also swept the depth of the network (number of scattering blocks), type of non-linearity, frame widths, projection widths, and learning rate. We report the full architecture details for each dataset in Tables 1, 2 and 3, excluding parameters that are the same across all datasets. Table 1: Architecture configuration for proposed method: includes hyper-parameters that differ across datasets. Rows sharing a configuration are merged (their architectures listed together). Values that are not listed were help fixed across all datasets (for example, angles (L) = 4, and low-pass last was always used). Dataset Params Blocks GAP Mix. hor. BN LR step WD DTD 2.74M 6 True 3 False 50 0.005 EuroSat 2.08M 4 False 27 False 20 0.0001 MinC-2500 2.56M 6 True 3 True 50 0.005 SuroTex 112K 4 True 3 True 20 0.005 Table 2: Architecture configuration for ScatNet B.A., part 1 of 2: hyper-parameters that differ across the reported models. Rows sharing a configuration are merged (their architectures listed together). Values that are not listed were held fixed across all datasets (for example, batch norm was always used). Dataset Params Blocks Nonlin. P proj. DTD 26.8M 6 relu 64/128/256/512/512/512 EuroSat 24.1M 5 softshrink 64/128/256/512/512 MinC-2500 9.47M 4 relu 64/128/256/512 SuroTex 37.2M 6 softsrhink 64/128/256/512/512/512 Table 3: Architecture configuration for ScatNet B.A., part 2 of 2. Dataset Frame w. LR LR step Epochs Batch DTD 512/1024/1024/2048/2048/2048 0.01 60 200 128 EuroSat 1024/2048/4096/8192/8192 0.01 70 300 128 MinC-2500 512/1024/1024/2048 0.01 60 200 32 SuroTex 1024/2048/4096/8192/8192/8192 0.001 70 300 25 Statistical significance and reproducibility Each configuration was independently run across three different random seeds (42, 43, 44), to capture variability in initialization and dataset sampling. Each figure of results displays the statistical significance. Figures 2 and 9 displays the mean test accuracy across seeds with a shaded band of ±1± 1 standard deviation. Figures 3, 5, 6, 7 and 10 plot seed means only. Figure 8 shows the seed mean with ±1± 1 standard deviation error bars on both axes. Tabular results with standard deviation are reported in Section C.3. Use of existing assets We use existing assets of data and code in our paper. DTD is shared without a license; we used the current version DTD-R1.0.1, and is available open-source from the creators [Cimpoi et al., 2014]. EuroSat is shared under an MIT license [Helber et al., 2019]. MinC-2500 and SuroTex are shared under C BY 4.0 licenses [Bell et al., 2015, Putra et al., 2025]. The texture-generation code builds on the implementation released with Brochard et al. [2021], which is under an Apache-2.0 license. We also use existing code from Zarka et al. [2021] to implement the ScatNet B.A. baseline, with minor updates to allow training on a subset of data and report additional statistics, such as within-class variance. The original code is available at https://github.com/j-zarka/separation_concentration_deepnets/ and is shared under a BSD 3-clause license that allows modification and private and commercial use. C.3 Texture Classification Results C.3.1 Test Accuracy versus Generalization Gap Figure 5: Test accuracy versus generalization gap. Color denotes method, line style denotes regularization strength, and marker size denotes percentage of training data used. Increasing regularization strength decreases generalization error on DTD, MinC-2500, and SuroTex. Figure 5 displays test accuracy versus generalization gap. Generalization gap is calculated as test accuracy minus training accuracy. The ideal model lies in the bottom right of each plot, which would indicate high accuracy and a low generalization gap. Across datasets and methods, a lower generalization gap is correlated with higher test accuracy. The original scattering transform has the smallest generalization gap across all four datasets. This matches standard generalization bounds, which depend on the complexity of the hypothesis class. The scattering network without fine-tuning only trains the linear classifier, so it has a small fraction of the number of parameters of the other methods and its associated hypothesis class has much lower complexity. Increasing regularization strength, corresponding to decreased fine-tuning distance, also decreases the generalization gap on DTD, MinC-2500, and SuroTex. This observation also aligns with Corollary 1, which predicts a smaller fine-tuning distance will correlate with lower generalization gap. Regularization does not have an effect on the fine-tuned scattering network on EuroSat, where all of the methods achieve near perfect accuracy on the training dataset. This suggests that the initial wavelet parameters already perform well and do not need to move during training, decreasing the effect of anchoring the parameters near their initial value. Finally, the fine-tuned scattering network improves on the accuracy of the original scattering transform. The fine-tuned scattering network also outperforms the randomly initialized model in accuracy and generalization gap, with a stronger advantage when training data is limited. This also supports Corollary 1; the generalization gap can be controlled either by reducing the fine-tuning distance or by increasing the sample size, so the fine-tuned model has a larger advantage when training data is limited. C.3.2 Regularization Ablation Figure 6: Test accuracy versus generalization gap, with weight decay regularization on randomly initialized model. Color denotes method, line weight denotes regularization strength, and size denotes percentage of training data. Weight decay controls the generalization gap at the cost of reducing test accuracy on DTD, EuroSat, and MinC-2500. It is well-known that regularizing a model reduces the generalization gap. Figure 6 ablates the effect of controlling fine-tuning distance versus standard weight decay regularization on the randomly initialized network. The two differ in the parameters they are anchored to during training: fine-tuning distance penalizes movement away from the initial parameters, such as the scattering prior, while weight decay penalizes distance from the origin. Weight decay reduces the generalization gap for the randomly initialized model, but it does so by reducing the test accuracy, in some cases to near 0, as on DTD. The model achieves low generalization error by performing equally poorly on the training and testing datasets. In contrast, regularizing the fine-tuning distance on the fine-tuned scattering network produces a smaller reduction in generalization gap but at far less cost to test accuracy. The original scattering transform, without any fine-tuning, can be viewed as applying fine-tuning regularization with λ=∞λ=∞, and it does not demonstrate the same collapse in test accuracy as the randomly initialized model does with λ=1λ=1. Figure 6 indicates that controlling the fine-tuning distance can control generalization error without destroying accuracy; anchoring the parameters to a meaningful initial distribution that disentangles the data does not have a negative effect on test accuracy. On the other hand, anchoring the parameters to zero, as weight decay regularization does, negatively impacts test accuracy. Figure 7: Test accuracy versus generalization gap, comparing effect of different parameter anchors on randomly initialized model. Within each subplot, color indicates method and parameter anchor, and marker size indicates percentage of training data used. The randomly initialized model with scattering anchor has higher generalization error than the fine-tuned scattering model, likely because the fine-tuning distance is greater on DTD and MinC-2500. Figure 7 illustrates the effect of different anchors on the randomly initialized model and separates two overlapping effects in the fine-tuned model; the benefit conferred by a meaningful prior and the benefit conferred by controlling fine-tuning distance. Each subplot shows the randomly initialized model with regularization penalizing the distance from three parameter anchors, the origin, the random initial parameters, and the initial scattering parameters, alongside the randomly initialized model without regularization and the fine-tuned scattering model. Anchoring the randomly initialized model to the scattering prior improves the test accuracy and avoids the steep decline in test accuracy observed with standard weight decay, which demonstrates the first effect, that anchoring to a meaningful prior improves accuracy, in isolation. However, the fine-tuned scattering transform demonstrates a lower generalization gap on DTD and SuroTex. It anchors to its own initial parameters, while the randomly initialized model with a scattering anchor starts from one set of parameters and pulls towards another during training. The fine-tuning distance is therefor much smaller for the fine-tuned scattering network, giving it the second effect, control of generalization gap via control of fine-tuning distance, as well. Although the first effect is on test accuracy and the second effect is on the generalization gap, they can amplify each other, since test accuracy and generalization gap are highly correlated, as shown in Figure 5. The fine-tuned scattering network benefits from a meaningful prior and from restricted fine-tuning distance, while the randomly initialized model with a scattering anchor benefits from the first effect alone. C.3.3 Neural Collapse Figure 8: Normalized within-class variance in classifier subspace versus test accuracy. Within sub-plots, color denotes method, marker size denotes percentage of training data used, and cross-hatches denote standard deviation. Rows denote regularization strength and columns denote datasets. Figure 8 displays normalized within-class variance against test accuracy. We report a ratio of traces, tr(ΣwM)/tr(ΣbM)tr( _wM)/tr( _bM), following the convention in Tirer et al. [2023], Beaglehole et al. [2024], Jacot et al. [2025]. We will refer to this quantity as normalized within-class variance. Σw _w denotes the within-class covariance of the features, averaged over classes, and Σb _b denotes the covariance of the class means about their center: Σw=1K∑c=1K1nc∑i:yi=c(hi−μc)(hi−μc)⊤,Σb=1K∑c=1K(μc−μ¯)(μc−μ¯)⊤, _w= 1K _c=1^K 1n_c _i:y_i=c(h_i- _c)(h_i- _c) , _b= 1K _c=1^K( _c- μ)( _c- μ) , where hih_i is the feature of sample i, μc _c is the mean of class c, μ¯ μ is the mean of the class means, and K is the number of classes. Standard within-class variance sets M equal to the identity matrix; however for settings with many more feature dimensions than classes, this can penalize a model for variance in directions that are invisible to the classifier, which penalizes a feature representation that is well-separated with respect to the geometry of the classifier. Therefore, in Figure 8, we set M=VVTM=V^T, where the columns of V are an orthonormal basis for the row space of the classifier weights after their mean row is removed, taken as the Jacobian of the head with respect to hih_i so that its normalization scale and any pooling are included. Both traces are then taken inside the classifier subspace, on the projected features VThiV^Th_i. Feature variation orthogonal to that subspace does not change the model’s predictions, so we do not penalize the model for variance in these directions. The ideal model resides in the top left corner of each subplot, with low within-class variance, which indicates a high degree of neural collapse, and high test accuracy. To understand the effect of controlling fine-tuning distance on neural collapse, Figure 8 displays results without regularization in the top row and with regularization (λ=1λ=1) in the bottom row. The results for the scattering network without fine-tuning and for ScatNet B.A. are repeated between rows because they do not have regularization. Results for the fine-tuned scattering network regularize distance from the initial parameters and results for the randomly initialized network apply weight decay regularization. In Figure 8, normalized within-class variance and test accuracy are correlated with each other. The fine-tuned scattering network outperforms the original scattering network on both metrics on DTD and MinC-2500, although ScatNet B.A. outperforms the fine-tuned scattering network on MinC-2500. Unsurprisingly, stronger regularization anchors the normalized of the fine-tuned scattering network closer to the normalized of the original scattering transform, especially on SuroTex and EuroSat. SuroTex is the only dataset where the original scattering transform has significantly lower normalized than all other methods, which aligns with the fact that SuroTex is closest to the classical texture datasets that the scattering transform describes well before fine-tuning. Figure 8 suggests that regularizing fine-tuning distance helps preserve desirable properties of the scattering transform, such as neural collapse, in addition to controlling the generalization gap. C.3.4 Model Interpretability Results A major advantage of the proposed method is that not only is the initial model interpretable, but the final model should also be interpretable. The scattering network’s interpretability follows from the fact that its filters are defined by a wavelet filter bank. In order to verify that the fine-tuned scattering network maintains a reasonable interpretation, we report several metrics that measure how close the trained filters of a model are to wavelets in Figures 9 and 10, and how much the parameters of the overall filter bank change in Figure 11. Figure 9: Percentage of data used during training versus relative magnitude of zero-frequency component. Wavelets must satisfy an admissibility condition, that the zero-frequency component has magnitude zero. Color denotes method, line style denotes regularization strength. Increasing regularization strength decreases the relative magnitude of the zero-frequency component on all datasets. Figure 9 displays the relative magnitude of the zero-frequency component against the percentage of data used in training, for the median filter (top row) and the worst filter (bottom row) of each model. Wavelets must satisfy an admissibility condition: a wavelet ψ has zero mean, ∫ψ(x)x=0 ψ(x)\,dx=0, or equivalently a vanishing zero-frequency component. Because the condition is common to all wavelet families, this metric makes no assumption about which wavelet the filters resemble, but it is correspondingly weak: it can rule a filter out as a wavelet, not certify it as one. The randomly initialized model fails the condition outright, and its median and worst-case filters have similar values, indicating that large zero-frequency components are the norm rather than the exception among its filters. Its learned filters therefore cannot be interpreted as wavelets. The fine-tuned scattering model retains a much smaller zero-frequency component at the median, and stronger regularization shrinks it further. Its worst-case filter degrades as the training set grows, but penalizing the fine-tuning distance keeps that degradation bounded. Fine-tuning thus leaves the network interpretable as a family of wavelet-based filters rather than destroying that structure. Figure 10: Fine-tuning distance versus ρ (similarity to best-fit Morlet filter). Color denotes method and marker style denotes regularization strength. Marker size denotes the percentage of data used during training. Smaller fine-tuning distance is correlated with more similarity to a Morlet filter across all four datasets. Having ruled out the randomly initialized filters as wavelets, we now ask how wavelet-like the fine-tuned filters are. Since the scattering network is initialized with Morlet filters, we measure each filter’s deviation from the closest member of the Morlet family. For a given fine-tuned filter, we fit the orientation, central frequency, envelope width, and aspect ratio of the Morlet filter that most closely matches it, and define ρ as the L2L^2 distance between the two, normalized by the norm of the fine-tuned filter. Unlike the admissibility condition, this measure is specific to the family the network was initialized from, and it is positive evidence rather than a necessary condition: a small ρ demonstrates that the filter is still a Morlet wavelet, accounting for the possibility that its parameters are different that the parameters that generated the original Morlet filter. Figure 10 displays ρ against the fine-tuning distance. Filters that move less remain closer to the Morlet family, as expected. The informative comparison is vertical: between filters that traveled the same distance from their initialization but differ in how far they ended up from the family. At matched distance, the fine-tuned filters remain substantially closer to the Morlet family than the randomly initialized ones, so their similarity to wavelets is not merely a consequence of having moved less. Fine-tuning not only keeps the filters near their starting point, it leaves them within the family of filters from which they were initialized. Figure 11: Relative change in the fitted Morlet parameters during fine-tuning. For each fine-tuned filter, we compare the parameters of its best-fit Morlet wavelet to those of the matched filter in the initial bank. Regularizing the fine-tuning distance keeps the parameters closer to their initial values across datasets. Verifying that each individual filter remains wavelet-like after fine-tuning is necessary but not sufficient to ensure that the overall model remains interpretable; the scattering transform is not an arbitrary collection of wavelets, but one whose elements are distributed across scales and orientations. Therefore, in Figure 11, we report how far the parameters of the fine-tuned filter bank have shifted during training. For each parameter (orientation θ, central frequency ξ, and bandwidth σ) the shift is the difference between the best-fit Morlet wavelet used to compute ρ and the matched filter in the initial bank. Matching filters rather than comparing them by index ensures that models whose filters are permuted during training are not penalized for the permutation. It also means the shifts are of limited use for the randomly initialized model, whose large ρ values indicate that the best-fit Morlet wavelet is a poor approximation of the true filter. Figure 11 demonstrates that fine-tuning successfully controls the change in fitted parameters. With λ=1λ=1, the relative shifts are close to zero across datasets and parameters, whereas with λ=0λ=0 they are substantially larger on DTD, Minc-2500, and SuroTex, although the parameter change is small at all regularization levels for EuroSat. Together with the per-filter results above, this is consistent with fine-tuning preserving not only the character of the individual wavelets but the organization of the bank they form. C.3.5 Full Numerical Results To allow more precise reporting of our results, we include tables comparing test accuracy, generalization error, and normalized within-class variance in the classifier subspace. We report the mean and standard deviation across 3 random seeds for all datasets except DTD, where we report the mean and 1 standard deviation across 3 random seeds and 5 partitions of the data. The factors of variability include the split of data to include in the training subset, train/val split, and model initialization. While the filters for the scattering transform and ScatNet B.A. are deterministic, the classifier is initialized randomly, as well as the analysis layers in ScatNet B.A. Table 4: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for DTD, by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 10% 20% 30% 50% 100% Random Init. (L2=0) Test acc. 11.06 (0.67) 16.13 (0.67) 20.45 (0.40) 26.43 (0.31) 34.10 (0.92) Gen. Gap 87.10 (2.73) 83.26 (1.04) 79.46 (0.39) 72.98 (1.00) 65.19 (0.54) NC1 6.42 (0.21) 5.53 (0.30) 4.46 (0.29) 3.12 (0.21) 2.33 (0.17) Random Init. (L2=0.1) Test acc. 9.94 (0.51) 13.44 (1.15) 14.66 (0.69) 15.87 (1.62) 17.42 (1.12) Gen. Gap 67.70 (6.84) 49.16 (10.15) 25.21 (6.66) 10.82 (4.18) 3.66 (2.84) NC1 5.55 (0.30) 5.37 (0.87) 4.78 (0.43) 4.56 (0.52) 4.33 (0.33) Random Init. (L2=1) Test acc. 3.57 (0.36) 3.68 (0.45) 4.06 (0.39) 2.95 (0.32) 2.53 (0.40) Gen. Gap 10.68 (1.87) 5.28 (3.00) 1.13 (2.13) 3.67 (2.53) 3.32 (0.73) NC1 5.08 (0.35) 4.82 (0.33) 4.63 (0.42) 4.64 (0.39) 2.68 (1.92) Scattering Test acc. 17.15 (1.14) 22.83 (0.64) 26.46 (0.41) 30.47 (0.85) 35.42 (0.35) Gen. Gap 41.98 (1.63) 30.97 (1.08) 25.20 (0.88) 19.57 (0.68) 13.88 (0.46) NC1 3.65 (0.29) 3.67 (0.28) 3.62 (0.29) 3.45 (0.28) 3.16 (0.17) Fine-tuned Scat. (L2=0) Test acc. 19.32 (0.50) 26.91 (0.44) 31.54 (0.35) 37.09 (0.73) 43.25 (1.05) Gen. Gap 71.65 (6.23) 66.49 (5.03) 62.23 (3.56) 56.65 (4.39) 47.96 (7.03) NC1 3.57 (0.27) 3.17 (0.30) 2.91 (0.22) 2.68 (0.16) 2.50 (0.09) Fine-tuned Scat. (L2=0.1) Test acc. 19.31 (0.31) 26.70 (0.63) 31.60 (0.18) 36.06 (1.02) 42.26 (0.96) Gen. Gap 65.70 (5.56) 62.71 (7.72) 60.72 (2.46) 51.79 (4.96) 46.02 (3.09) NC1 3.70 (0.31) 3.28 (0.30) 2.99 (0.20) 2.66 (0.16) 2.33 (0.10) Fine-tuned Scat. (L2=1) Test acc. 19.08 (0.51) 26.11 (0.37) 30.82 (0.58) 35.33 (0.85) 40.80 (0.94) Gen. Gap 65.95 (4.26) 57.94 (8.44) 58.09 (5.46) 46.05 (4.86) 42.60 (3.34) NC1 3.84 (0.33) 3.52 (0.24) 3.12 (0.20) 2.76 (0.12) 2.32 (0.10) ScatNet B.A. Test acc. 6.91 (0.77) 9.66 (1.14) 14.85 (0.76) 17.65 (0.73) 21.93 (0.61) Gen. Gap 92.85 (0.60) 90.23 (0.98) 85.15 (0.76) 82.35 (0.73) 78.00 (0.68) NC1 4.80 (0.56) 3.97 (0.60) 2.98 (0.17) 2.63 (0.06) 2.37 (0.10) Table 4: * Table 5: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for EuroSat (Part 1 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 0.5% 1% 5% 10% 15% Random Init. (L2=0) Test acc. 45.40 (3.57) 56.45 (2.16) 74.98 (1.21) 80.48 (0.27) 82.57 (0.83) Gen. Gap 54.60 (3.57) 42.77 (2.50) 24.99 (1.27) 19.51 (0.28) 17.43 (0.83) NC1 1.29 (0.22) 0.80 (0.26) 0.58 (0.12) 0.52 (0.04) 0.51 (0.05) Random Init. (L2=0.1) Test acc. 24.78 (4.44) 57.07 (3.67) 73.64 (2.94) 80.38 (0.71) 83.65 (0.48) Gen. Gap 13.37 (2.94) 41.57 (2.78) 19.65 (7.42) 15.22 (1.82) 10.37 (0.54) NC1 2.56 (1.10) 0.87 (0.25) 0.49 (0.04) 0.49 (0.06) 0.44 (0.03) Random Init. (L2=1) Test acc. 41.80 (14.68) 55.94 (1.68) 67.33 (0.13) 64.91 (9.05) 60.73 (2.71) Gen. Gap 33.30 (14.28) 22.61 (3.99) 4.47 (0.75) -0.44 (2.52) -1.04 (2.65) NC1 1.61 (1.05) 0.59 (0.15) 0.39 (0.03) 0.41 (0.05) 0.44 (0.05) Scattering Test acc. 27.78 (6.00) 32.22 (3.79) 75.30 (0.45) 78.94 (0.58) 80.69 (0.10) Gen. Gap 32.46 (41.34) 53.74 (3.52) 14.20 (1.48) 10.39 (0.78) 8.71 (0.85) NC1 0.93 (0.10) 0.70 (0.05) 0.44 (0.01) 0.43 (0.02) 0.43 (0.02) Fine-tuned Scat. (L2=0) Test acc. 64.77 (0.87) 70.20 (2.07) 83.22 (0.30) 86.67 (0.36) 88.25 (0.39) Gen. Gap 35.23 (0.87) 28.24 (1.05) 16.78 (0.30) 13.24 (0.52) 11.52 (0.50) NC1 0.87 (0.08) 0.70 (0.07) 0.57 (0.09) 0.52 (0.04) 0.47 (0.04) Fine-tuned Scat. (L2=0.1) Test acc. 64.60 (1.18) 70.06 (0.88) 82.12 (0.69) 85.10 (0.55) 86.81 (0.41) Gen. Gap 35.40 (1.18) 29.55 (0.76) 17.69 (0.92) 14.10 (0.19) 11.87 (1.43) NC1 0.86 (0.03) 0.71 (0.06) 0.52 (0.04) 0.46 (0.02) 0.45 (0.03) Fine-tuned Scat. (L2=1) Test acc. 62.83 (2.18) 68.60 (0.97) 79.73 (1.03) 83.67 (1.11) 85.10 (0.58) Gen. Gap 36.37 (2.52) 29.25 (0.91) 18.92 (0.82) 15.67 (0.71) 14.19 (0.85) NC1 1.00 (0.11) 0.73 (0.09) 0.52 (0.04) 0.46 (0.03) 0.46 (0.02) ScatNet B.A. Test acc. 43.91 (1.76) 49.90 (3.66) 66.57 (2.32) 71.96 (0.90) 76.20 (0.07) Gen. Gap 56.09 (1.76) 50.10 (3.66) 33.43 (2.32) 28.04 (0.90) 23.80 (0.07) NC1 0.43 (0.01) 0.56 (0.09) 0.40 (0.01) 0.39 (0.02) 0.39 (0.02) Table 5: * Table 6: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for EuroSat (Part 2 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 20% 30% 50% 100% Random Init. (L2=0) Test acc. 84.44 (1.18) 86.73 (0.27) 88.94 (0.74) 91.98 (0.53) Gen. Gap 15.56 (1.18) 13.26 (0.28) 11.03 (0.70) 7.47 (0.23) NC1 0.47 (0.04) 0.44 (0.01) 0.40 (0.01) 0.33 (0.03) Random Init. (L2=0.1) Test acc. 85.94 (0.81) 88.00 (0.83) 88.13 (3.15) 85.94 (4.40) Gen. Gap 8.01 (1.52) 5.04 (0.24) 2.19 (3.14) 0.73 (2.15) NC1 0.41 (0.02) 0.39 (0.01) 0.39 (0.03) 0.37 (0.02) Random Init. (L2=1) Test acc. 60.68 (1.33) 65.31 (8.32) 64.96 (11.21) 66.02 (11.60) Gen. Gap -0.90 (1.05) -0.44 (1.78) -0.40 (0.31) -1.32 (0.89) NC1 0.39 (0.01) 0.39 (0.03) 0.42 (0.07) 0.37 (0.06) Scattering Test acc. 81.59 (0.45) 83.47 (0.43) 85.24 (0.80) 87.10 (0.65) Gen. Gap 8.09 (0.31) 6.89 (0.25) 5.48 (1.45) 3.51 (0.80) NC1 0.42 (0.02) 0.42 (0.01) 0.41 (0.02) 0.42 (0.01) Fine-tuned Scat. (L2=0) Test acc. 89.77 (0.45) 90.80 (0.20) 92.86 (0.25) 94.45 (0.46) Gen. Gap 9.94 (0.33) 9.20 (0.20) 6.95 (0.05) 5.43 (0.28) NC1 0.47 (0.03) 0.47 (0.02) 0.42 (0.01) 0.38 (0.01) Fine-tuned Scat. (L2=0.1) Test acc. 88.01 (0.57) 89.48 (0.35) 91.70 (0.25) 93.22 (0.38) Gen. Gap 11.48 (0.44) 9.80 (1.48) 7.28 (1.65) 5.30 (1.54) NC1 0.46 (0.03) 0.45 (0.02) 0.44 (0.01) 0.44 (0.04) Fine-tuned Scat. (L2=1) Test acc. 85.77 (0.97) 87.89 (0.49) 89.52 (0.57) 91.98 (0.31) Gen. Gap 10.48 (2.84) 10.68 (2.00) 8.63 (1.75) 6.70 (0.60) NC1 0.45 (0.03) 0.45 (0.05) 0.45 (0.02) 0.48 (0.01) ScatNet B.A. Test acc. 80.37 (0.73) 83.49 (0.96) 87.24 (0.45) 91.95 (0.26) Gen. Gap 19.63 (0.73) 16.51 (0.96) 12.76 (0.45) 8.05 (0.26) NC1 0.36 (0.01) 0.35 (0.00) 0.31 (0.01) 0.27 (0.01) Table 6: * Table 7: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for MinC-2500 (Part 1 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 0.2% 0.5% 1% 5% 10% Random Init. (L2=0) Test acc. 11.93 (1.34) 15.46 (0.72) 19.50 (0.56) 28.40 (0.30) 33.88 (0.60) Gen. Gap 82.64 (3.56) 77.15 (4.67) 62.34 (8.63) 38.96 (9.96) 31.43 (7.32) NC1 6.28 (1.96) 6.38 (0.60) 5.94 (0.68) 3.59 (0.42) 2.90 (0.18) Random Init. (L2=0.1) Test acc. 10.87 (1.32) 14.28 (1.32) 19.23 (1.70) 35.64 (0.85) 41.57 (0.37) Gen. Gap 84.34 (2.20) 75.61 (6.87) 48.63 (2.49) 31.65 (9.09) 17.50 (3.41) NC1 8.38 (3.33) 7.62 (1.46) 6.50 (0.58) 2.77 (0.02) 2.29 (0.04) Random Init. (L2=1) Test acc. 8.96 (3.52) 10.33 (0.89) 8.95 (0.48) 36.34 (1.00) 35.79 (8.83) Gen. Gap 61.04 (36.98) 65.11 (3.97) 10.57 (5.45) 9.54 (2.22) 3.29 (0.87) NC1 9.48 (1.79) 7.20 (1.64) 4.09 (1.03) 2.63 (0.08) 2.51 (0.08) Scattering Test acc. 8.95 (0.95) 18.24 (0.83) 20.26 (0.24) 29.71 (0.37) 33.48 (0.47) Gen. Gap 61.92 (1.75) 47.62 (2.89) 24.59 (3.27) 12.75 (0.72) 8.67 (1.42) NC1 5.40 (0.39) 6.18 (0.48) 5.78 (0.36) 4.62 (0.11) 4.36 (0.03) Fine-tuned Scat. (L2=0) Test acc. 15.61 (1.10) 20.22 (0.93) 24.05 (2.52) 36.68 (1.00) 40.73 (0.42) Gen. Gap 75.05 (10.50) 70.33 (6.12) 52.17 (13.84) 51.41 (5.63) 40.33 (7.65) NC1 6.50 (0.92) 6.30 (0.52) 5.43 (0.50) 3.77 (0.23) 3.36 (0.09) Fine-tuned Scat. (L2=0.1) Test acc. 15.84 (1.05) 19.99 (1.04) 24.38 (1.49) 35.29 (1.44) 39.98 (1.57) Gen. Gap 74.38 (10.42) 67.40 (5.01) 57.04 (4.07) 36.57 (9.33) 26.97 (10.16) NC1 6.21 (0.65) 6.07 (0.56) 5.27 (0.43) 3.76 (0.20) 3.45 (0.15) Fine-tuned Scat. (L2=1) Test acc. 15.48 (0.99) 20.10 (0.78) 22.44 (2.87) 34.78 (0.55) 39.00 (1.47) Gen. Gap 70.82 (7.09) 65.00 (3.28) 36.13 (19.33) 40.99 (0.82) 24.32 (8.36) NC1 6.23 (0.61) 6.10 (0.51) 5.04 (0.34) 4.05 (0.11) 3.66 (0.11) ScatNet B.A. Test acc. 15.00 (0.92) 17.91 (1.40) 25.56 (1.16) 44.23 (0.64) 51.41 (0.35) Gen. Gap 47.83 (6.26) 27.52 (7.81) 25.76 (1.12) 19.74 (1.72) 16.47 (0.14) NC1 3.95 (0.56) 3.79 (0.45) 3.02 (0.28) 2.12 (0.05) 1.86 (0.03) Table 7: * Table 8: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for MinC-2500 (Part 2 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 20% 30% 50% 100% Random Init. (L2=0) Test acc. 40.81 (0.38) 44.35 (0.73) 49.86 (0.99) 55.36 (1.21) Gen. Gap 28.29 (6.62) 23.53 (4.60) 13.94 (7.40) 8.13 (2.77) NC1 2.70 (0.06) 2.56 (0.11) 2.34 (0.01) 2.09 (0.08) Random Init. (L2=0.1) Test acc. 40.65 (12.06) 49.21 (0.86) 50.37 (0.74) 50.69 (0.20) Gen. Gap 1.69 (2.01) -0.03 (0.94) -2.37 (0.72) -4.05 (0.33) NC1 2.47 (0.93) 1.98 (0.11) 2.19 (0.08) 2.54 (0.13) Random Init. (L2=1) Test acc. 36.22 (10.30) 37.67 (8.21) 32.75 (8.47) 26.43 (0.63) Gen. Gap 0.27 (2.94) -0.58 (3.62) 2.08 (4.10) 6.14 (1.16) NC1 2.79 (0.38) 2.73 (0.41) 2.92 (0.33) 2.96 (0.30) Scattering Test acc. 35.97 (0.19) 37.53 (0.38) 39.36 (0.25) 40.57 (0.43) Gen. Gap 4.38 (0.16) 2.93 (0.51) 1.06 (0.76) 0.37 (0.11) NC1 4.23 (0.10) 4.14 (0.03) 4.07 (0.11) 3.98 (0.06) Fine-tuned Scat. (L2=0) Test acc. 45.19 (0.77) 47.91 (0.64) 51.63 (0.70) 57.19 (0.45) Gen. Gap 29.33 (6.93) 33.10 (5.39) 30.60 (1.89) 20.20 (1.07) NC1 3.08 (0.03) 2.92 (0.05) 2.66 (0.05) 2.32 (0.01) Fine-tuned Scat. (L2=0.1) Test acc. 45.36 (0.60) 47.30 (0.75) 50.09 (0.60) 53.28 (0.33) Gen. Gap 26.54 (1.14) 21.14 (1.64) 12.72 (0.53) 5.21 (0.40) NC1 2.97 (0.09) 2.76 (0.08) 2.53 (0.02) 2.31 (0.05) Fine-tuned Scat. (L2=1) Test acc. 43.57 (0.53) 45.77 (0.83) 47.39 (0.96) 50.03 (0.48) Gen. Gap 17.66 (0.64) 13.30 (0.64) 8.21 (1.36) 3.02 (0.92) NC1 3.22 (0.08) 3.02 (0.13) 2.83 (0.03) 2.63 (0.03) ScatNet B.A. Test acc. 58.17 (0.43) 61.37 (0.40) 65.25 (0.53) 69.61 (0.68) Gen. Gap 12.30 (1.81) 9.14 (1.08) 5.66 (1.04) 3.22 (1.19) NC1 1.64 (0.06) 1.52 (0.05) 1.43 (0.02) 1.35 (0.04) Table 8: * Table 9: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for SuroTex (Part 1 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 2% 5% 10% 15% Random Init. (L2=0) Test acc. 49.00 (0.95) 77.93 (4.59) 87.17 (2.37) 92.90 (1.57) Gen. Gap 51.00 (0.95) 20.29 (4.34) 9.72 (2.51) 5.32 (1.79) NC1 0.48 (0.06) 0.33 (0.02) 0.30 (0.03) 0.32 (0.02) Random Init. (L2=0.1) Test acc. 39.80 (21.30) 77.00 (2.46) 87.43 (1.85) 92.07 (2.33) Gen. Gap 49.53 (3.31) 20.78 (1.23) 10.12 (2.83) 6.23 (1.86) NC1 0.43 (0.13) 0.33 (0.03) 0.32 (0.01) 0.34 (0.02) Random Init. (L2=1) Test acc. 23.93 (21.40) 75.37 (2.95) 84.57 (1.20) 70.27 (16.19) Gen. Gap 54.07 (4.02) 16.63 (8.09) 10.88 (1.55) 14.99 (6.81) NC1 0.30 (0.19) 0.30 (0.02) 0.31 (0.04) 0.31 (0.02) Scattering Test acc. 34.73 (1.31) 55.07 (1.79) 74.60 (2.98) 83.17 (0.42) Gen. Gap 37.93 (6.61) 5.82 (2.70) -2.38 (3.84) -3.02 (1.34) NC1 0.12 (0.01) 0.12 (0.00) 0.12 (0.01) 0.12 (0.01) Fine-tuned Scat. (L2=0) Test acc. 60.13 (3.67) 82.43 (3.65) 91.83 (1.16) 95.07 (1.07) Gen. Gap 37.20 (3.73) 12.01 (2.70) 4.50 (1.73) 2.19 (1.38) NC1 0.15 (0.01) 0.16 (0.01) 0.17 (0.01) 0.16 (0.01) Fine-tuned Scat. (L2=0.1) Test acc. 59.60 (3.63) 81.40 (4.00) 91.00 (1.31) 94.03 (1.01) Gen. Gap 38.40 (5.29) 10.82 (3.79) 3.78 (2.35) 2.48 (1.18) NC1 0.15 (0.01) 0.14 (0.00) 0.13 (0.01) 0.13 (0.01) Fine-tuned Scat. (L2=1) Test acc. 58.90 (3.89) 80.60 (4.65) 89.43 (1.57) 93.47 (0.87) Gen. Gap 39.77 (6.17) 11.40 (3.57) 3.46 (1.87) 1.27 (0.66) NC1 0.13 (0.01) 0.13 (0.00) 0.12 (0.01) 0.12 (0.00) ScatNet B.A. Test acc. 55.80 (3.50) 82.73 (2.75) 93.70 (1.25) 95.20 (1.18) Gen. Gap 41.53 (2.45) 13.04 (2.59) 4.19 (0.74) 2.65 (1.39) NC1 0.19 (0.01) 0.16 (0.01) 0.15 (0.01) 0.15 (0.01) Table 9: * Table 10: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for SuroTex (Part 2 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold. Model Metric 20% 30% 50% 100% Random Init. (L2=0) Test acc. 94.20 (0.62) 96.00 (1.73) 97.23 (0.96) 98.77 (0.83) Gen. Gap 3.74 (0.92) 1.65 (0.20) 0.68 (1.71) -0.54 (0.31) NC1 0.30 (0.02) 0.28 (0.03) 0.28 (0.02) 0.26 (0.02) Random Init. (L2=0.1) Test acc. 94.37 (1.60) 95.57 (1.64) 97.20 (1.28) 98.10 (0.70) Gen. Gap 3.30 (2.05) 0.54 (1.33) 0.49 (1.73) 0.12 (0.72) NC1 0.32 (0.02) 0.28 (0.03) 0.31 (0.04) 0.27 (0.01) Random Init. (L2=1) Test acc. 92.17 (0.93) 93.70 (1.49) 95.40 (1.41) 88.23 (7.26) Gen. Gap 3.39 (1.05) 1.67 (0.96) 0.54 (1.57) 1.44 (2.74) NC1 0.30 (0.02) 0.28 (0.02) 0.27 (0.02) 0.24 (0.04) Scattering Test acc. 86.37 (1.50) 89.67 (1.10) 91.97 (1.02) 94.60 (0.17) Gen. Gap -4.53 (2.35) -5.00 (2.00) -6.40 (1.30) -6.63 (0.29) NC1 0.12 (0.01) 0.12 (0.01) 0.12 (0.00) 0.12 (0.00) Fine-tuned Scat. (L2=0) Test acc. 95.40 (0.78) 96.60 (1.06) 97.50 (0.95) 98.87 (0.46) Gen. Gap 1.60 (0.10) 0.84 (1.12) 0.04 (1.67) -0.03 (0.78) NC1 0.16 (0.01) 0.17 (0.01) 0.17 (0.01) 0.15 (0.01) Fine-tuned Scat. (L2=0.1) Test acc. 95.07 (0.23) 97.40 (1.55) 97.63 (1.48) 98.00 (0.62) Gen. Gap 1.66 (1.03) 0.11 (1.27) -0.63 (0.51) -1.84 (0.83) NC1 0.13 (0.00) 0.14 (0.01) 0.13 (0.00) 0.13 (0.01) Fine-tuned Scat. (L2=1) Test acc. 94.70 (1.00) 96.37 (0.61) 96.33 (0.49) 97.63 (0.15) Gen. Gap -0.64 (2.15) -1.63 (0.80) -1.56 (2.18) -2.27 (1.22) NC1 0.12 (0.00) 0.13 (0.01) 0.13 (0.00) 0.13 (0.00) ScatNet B.A. Test acc. 96.73 (0.35) 97.83 (0.15) 98.50 (0.44) 99.47 (0.31) Gen. Gap 0.71 (0.52) 0.31 (0.36) 0.10 (0.50) -0.25 (0.32) NC1 0.15 (0.01) 0.15 (0.01) 0.15 (0.00) 0.14 (0.01) Table 10: *