Paper deep dive
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
Anuragine S A, Prem Jagadeesan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/22/2026, 3:17:04 AM
Summary
This paper investigates functional equivalence and geometric diversity in neural network approximations of elementary mathematical functions. It demonstrates that while many distinct neural network architectures (single-layer and multilayer perceptrons) can achieve nearly identical approximation performance (functional equivalence), they exhibit significant geometric diversity in their parameter spaces. This diversity is characterized by 'sloppiness,' where the Hessian of the cost function reveals a low effective rank, indicating that only a small subset of parameters significantly influences the model's behavior. The authors propose a 'Prem-Anu Selection Criterion' (Model Index) to identify optimal, parsimonious models from equivalence classes based on parameter count, effective rank, and computational cost (FLOPs).
Entities (15)
Relation Signals (9)
Hessian → characterizes → Sloppiness
confidence 90% · geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function
Model Index → dependson → FLOPs
confidence 90% · defined by (3) which is a function of model capacity, effective rank and FLOPs.
Model Index → dependson → Neural Network Approximations
confidence 90% · ℳℐ=Np(1+1/reff)+log F where Np is number of model parameters
Model Index → dependson → Effective Rank
confidence 90% · defined by (3) which is a function of model capacity, effective rank and FLOPs.
Single-layer Neural Networks → exhibits → Functional Equivalence
confidence 90% · The analysis includes an extensive study of single-layer neural networks... The study reveals large equivalence classes of functionally indistinguishable... networks
Multilayer perceptrons → exhibits → Geometric Diversity
confidence 90% · The analysis includes an extensive study of... multilayer perceptrons... The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks
Effective Rank → quantifies → Sloppiness
confidence 90% · effective rank to quantify the dimensionality of parameter space... estimate of sloppiness
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability. In this work, we address this concern by analyzing functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions. The analysis includes an extensive study of single-layer neural networks and multilayer perceptrons under noisy and noise-free conditions. Beyond just network capacity, we study the geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of parameter space. The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks that consistently exhibit low effective rank and structural redundancy. Finally, a model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2607.18930v1
- Canonical: https://arxiv.org/abs/2607.18930v1
Trouble viewing inline? Open PDF directly →
Full Text
46,900 characters extracted from source content.
Expand or collapse full text
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization S. A. Anuragine1, Prem Jagadeesan1,∗ 1Amrita School of Artificial Intelligence Amrita Vishwa Vidyapeetham, Coimbatore, India E-mail: cb.ai.u4aim24005@cb.amrita.students.edu *Corresponding author: j_prem@cb.amrita.edu Abstract The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability. In this work, we address this concern by analyzing functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions. The analysis includes an extensive study of single-layer neural networks and multilayer perceptrons under noisy and noise-free conditions. Beyond just network capacity, we study the geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of parameter space. The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks that consistently exhibit low effective rank and structural redundancy. Finally, a model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency. Keywords Neural Network Approximations, Functional equivalence, Geometric diversity, Effective rank 1. Introduction In data-driven modeling, neural networks are central to many applications. They are widely used across image analysis, natural language processing and generative artificial intelligence ([11], [5]). However, these models are found to be overparameterized and redundant, which of results in a computational bottleneck for many downstream tasks. There may exist multiple network architectures of varying capacities that give rise to nearly identical approximations. However, in practice, we do not have a systematic guiding principle for selecting a parsimonious network structure. This often leads to increasing the number of layers, neurons, or trainable parameters until the desired approximation accuracy is achieved. This lack of a guiding principle for network selection leads to over-parameterization and redundancy. When a model contains more parameters than are effectively required, several distinct sub-network configurations may produce nearly identical fit. Therefore, minimizing the approximation error alone may not always optimize the network capacity. This has resulted in multiple deep learning compression and pruning techniques as an attempt to arrive at a smaller network that suits the given application. These methods may be applied before, during, or after training, and they often rely on criteria such as weight magnitude, gradient information, or Hessian-based sensitivity ([2] [1] [14]) . Knowledge distillation is another model-compression approach in which a smaller student network is trained to imitate the behaviour of a larger teacher network ([7]) . Similarly, LoRA reduces the number of trainable parameters by freezing the original model weights and learning only small low-rank update matrices during fine-tuning ([8] [3] [12]). However, these compression techniques are usually conditioned on an already selected or trained model, and therefore may not fully reveal the broader class of alternative architectures that could approximate the same target function equally well . Kolmogorov-Arnold Networks (KANs) have recently been proposed as an alternative architecture inspired by the Kolmogorov-Arnold representation theorem, and have shown promise in achieving compact representations with fewer parameters than traditional neural networks ([13],[13]). However, their reliance on spline-based nonlinear transformations can introduce high computational cost and scalability limitations for larger models ([20]). Similarly, the Lottery Ticket Hypothesis suggests that dense randomly initialized networks contain smaller subnetworks, called winning tickets, which can be trained in isolation to achieve performance comparable to the original network ([4]). While all these works point towards the existence of useful subnetworks, it still begins from an identified dense model and focuses on identifying sparse subnetworks within it. These continued attempts for finding a smaller network raises an important question: for a given target function, do multiple neural networks exist that give nearly identical fit and behave functionally similar. In this work, we take a different perspective. Instead of beginning with one optimal network and compressing it, we study the existence of multiple functionally equivalent networks by defining an equivalence class ([17]) and analyze how their parameter-space geometries differ. The primary reason for analyzing the geometry of the parameter space is that it affects the ease of estimation and therefore, it dictates the resulting architecture. The geometric analysis is carried out through the lens of sloppiness. A model is considered sloppy when there exist large regions in the parameter space over which model predictions remain nearly identical. Such behavior can be studied using the eigenvalue spectrum of the Hessian of the loss function, where large eigenvalues correspond to stiff directions and small eigenvalues correspond to sloppy or weakly constrained directions ([18], [6], [9]). In addition, the effective rank of the Hessian is used to estimate how many parameter-space directions contribute meaningfully to the local parameter space geometry and can be considered as an estimate of sloppiness ([10],[19],[16]). Recent studies on neural-network Hessians show that the Hessian is not an arbitrary high-dimensional matrix, but often contains a structured low-rank dominant subspace ([22]). Another line of work explains the spectral behavior of Hessian of neural networks by relating it to the structure present in the data and learned representations. It shows that matrices such as the Hessian and Fisher information matrix often contain a few dominant eigenvalue directions along with a large bulk of smaller directions ([15]). While the existing literature studies mainly analyze Hessian structure for selected trained networks/tasks or specific model instances, this work examines the phenomenon across a complete model space under the chosen architectural constraints. While modern architectures such as Transformers ([21]) and KANs operate on high-dimensional parameter space, the scope of this foundational work is deliberately focused on the approximation of univariate elementary mathematical functions using single-layer and multi-layer feedforward networks. Studying these univariate approximations serves as an essential first step toward decoding the fundamental nature of neural network approximations. Across these experiments, several networks are observed to achieve nearly identical approximation performance forming an equivalence class. The multilayer perceptron analysis further shows that functional equivalence is not restricted to a few models, but occupies a significant portion of the model space under the chosen architectural constraints. Even when the networks differ in model capacity, their effective rank remains much smaller than the total number of trainable parameters, indicating that only a limited portion of the parameter space contributes strongly to the learned representation, this suggests that the interaction of data and the model structure induces a low-dimensional geometry in the parameter space and the network always remains over-parameterized. Similar behavior is also observed under noisy conditions, suggesting that the low-dimensional geometric structure is not removed by perturbations in the data. Thus, neural network approximation is not only a question of whether a function can be learned, but also of how many distinct networks can learn it and how much of their parameter space is actually relevant to the representation. The study of geometry of parameter space offers rich insights into the neural network approximations and resulting equivalence class is therefore functionally similar, geometrically diverse, and constrained by a low-dimensional effective geometry. 2. Experimental Design The proposed methodology consists of two experiments. The first investigates shallow neural networks by varying the number of hidden neurons and activation functions to identify functionally equivalent approximations. The second extends the study to multilayer perceptrons (MLPs) with varying depths and widths to examine functional equivalence across a larger architecture space. The detailed network configurations and data generation procedure are provided in Supplementary S2S2. For every network, the parameters are estimated and the networks are ranked according to approximation performance. The top-K networks are selected to form an equivalence class. To evaluate robustness, the complete MLP experiment is repeated under both noise-free and noisy conditions. 3. Results Let fθ1f_ _1 and fθ2f_ _2 denote two neural network approximations belonging to a normed function space ℱF. The networks are said to be functionally equivalent with tolerance ε if ‖fθ1−fθ2‖ℱ<ε.\|f_ _1-f_ _2\|_F< . (1) In this work, functional equivalence is assessed using approximation metrics such as R2R^2 and NRMSE. Inspired by [17], we define the equivalence class of a network fθf_θ by [fθ]ε=fθ′:‖fθ′−fθ‖<ε.[f_θ]_ = \f_θ :\|f_θ -f_θ\|< \. (2) So all networks fθf_θ that satisfy the approximation criterion defined by (1) are considered to be functionally non-unique approximations. Having defined (2), the core idea of this work is to study the geometric characteristics of the functionally non-unique networks using the concept of sloppiness. 3.1 Analysis of Single Layer Neural Networks This section presents the results of the empirical study on shallow neural networks (single-layer). Figure 1(a) depicts functional non-uniqueness and geometric diversity of a class of 25 neural networks fit to a sinusoidal function with hyperbolic tangent activation. The networks are ranked on the basis of R2R^2 measure. It is evident from Figure 1(a) that top 25 neural networks have numerically indistinguishable R2R^2 and NRMSENRMSE less than an arbitrarily small ε , making them practically unidentifiable network structures. However, the singular value ratio varies substantially across these networks leading to significant geometric diversity. Despite network capacities ranging from 16 to 151 parameters, the effective rank remains between 0.5591 and 1.8056. This suggests that only a small subset of parameter-space directions dominate the local geometry. This makes effective rank a noteworthy geometric property of the parameter space. It is observed that the geometric properties of the parameter space are functions of data, network structure and activation function. Table 1 summarizes the network structures obtained for different target function–activation combinations along with their effective rank, singular value ratio, and Hessian eigenvalue spectrum. Although these networks achieve nearly identical approximation performance, they exhibit noticeable differences in their geometric properties, demonstrating that functional equivalence does not imply geometric similarity. In particular, the consistently low effective rank indicates that only a small subset of the parameter space contributes significantly to the learned representation, while the remaining directions are largely redundant. Furthermore, the effective rank varies across activation functions, suggesting that the choice of activation influences the geometry of the parameter space. Figure 1(a) further illustrates that the equivalent networks produce nearly indistinguishable function approximations despite substantial differences in their parameter-space geometry. These observations provide evidence for the existence of functionally equivalent yet geometrically diverse neural network structures. Detailed results for each target function and activation function combination, including approximation, geometric diversity, and Hessian eigenspectrum analyses, are available in the Supplementary Material (Figures S1–S16). 3.2 Analysis of Multilayer Perceptron In this section, we extend the analysis to multilayer perceptrons (MLP) to verify that the observations are not an artifact of single layer networks. The analysis is done for a set of 3905 networks whose number of layers and neurons in each layer varies from 1 to 5. For the noise-free scenario, the equivalence class consists of 1085 networks with R2≥0.99R^2≥ 0.99, indicating that a substantial fraction of the architecture space belongs to the equivalence class. Figure 1 summarizes the geometric characteristics of these networks. Figure 1(b) shows that the equivalent networks achieve numerically indistinguishable R2R^2 values, demonstrating functional non-uniqueness. Figure 1(c) presents the ratio of effective rank to the total number of parameters, revealing an upper bound of approximately 0.25 across all architectures. This indicates that only a small fraction of the parameter space contributes significantly to the learned representation, while the remaining directions are redundant. Figure 1(d) shows that a large proportion of the equivalent networks are concentrated within a relatively narrow range of parameter counts and effective ranks. Likewise, Figure 1(e) shows that most equivalent networks exhibit singular value ratios between 10−1110^-11 and 10−610^-6, indicating that they are highly sloppy. Overall, these observations demonstrate that neural network approximations admit a large equivalence class of functionally indistinguishable architectures whose parameter-space geometries differ substantially, while their effective dimensionality remains constrained to a significantly lower-dimensional subspace. (a) (b) (c) (d) (e) (f) Figure 1: Summary of functional equivalence, geometric characterization, and model selection in neural network approximations. The figure illustrates the existence of functionally equivalent architectures, their geometric diversity in parameter space, and the resulting implications for selecting computationally efficient models. To examine the effect of noise on functional equivalence and geometric diversity, the same set of 3905 MLP architectures was evaluated after adding Gaussian noise to the target data. Although the cardinality of the equivalence class decreases considerably, the overall geometric behaviour remains unchanged. In particular, the ratio of effective rank to parameter count continues to remain below approximately 0.25, and the equivalent networks exhibit similarly small singular value ratios, indicating that the low-dimensional and sloppy nature of the parameter space persists even under noisy conditions. The complete analysis for the noisy case is provided in Supplementary S4.1. Overall, approximating noisy data to neural networks significantly reduces the cardinality of the equivalence class while significant diversity exists in the parameter space geometry. Table 1: Comparison of Top-K Networks for Different Functions and Activation Functions Function Activation No. of Neurons No. of Parameters Effective Rank Singular value ratio R2R^2 NRMSE Min Max Min Max Min Max Min Max Min Max Min Max Sin Tanh 5 50 16 151 1.8056 2.5591 2×10−52× 10^-5 1×10−121× 10^-12 0.99959 0.99984 0.004 0.007 Sigmoid 13 50 40 151 2.403 2.5358 8×10−98× 10^-9 2×10−122× 10^-12 0.99937 0.99984 0.004 0.008 ReLU 30 75 91 226 5.9559 95.045 0 5×10−105× 10^-10 0.99089 0.99981 0.004 0.033 Cos Tanh 5 50 16 151 1.8081 2.4962 6×10−66× 10^-6 3×10−133× 10^-13 0.9995 0.99984 0.004 0.007 Sigmoid 26 50 79 151 2.4056 2.5369 2×10−102× 10^-10 4×10−124× 10^-12 0.99932 0.99983 0.004 0.009 ReLU 20 74 61 223 10.756 105.72 0 2×10−192× 10^-19 0.99453 0.99905 0.01 0.026 Exp Tanh 2 50 7 151 2.516 5.4948 4×10−44× 10^-4 7×10−147× 10^-14 0.99994 0.99999 0.0007 0.001 Sigmoid 2 50 7 151 2.5723 3.5663 2×10−42× 10^-4 1×10−121× 10^-12 0.99992 0.99984 0.0009 0.002 ReLU 34 75 103 226 23.321 75.752 0 1.4×10−241.4× 10^-24 0.99999 1 0.0004 0.0008 Quadratic Tanh 2 50 7 151 2.8333 3.5274 1×10−31× 10^-3 9×10−139× 10^-13 0.99984 0.99995 0.002 0.003 Sigmoid 2 50 7 151 2.3837 3.2462 6×10−36× 10^-3 4×10−124× 10^-12 0.99981 0.99994 0.002 0.004 ReLU 34 75 103 226 11.084 26.243 0 6×10−256× 10^-25 0.99997 0.99999 0.0008 0.001 Cubic Tanh 3 50 10 151 3.0306 3.7035 4×10−54× 10^-5 2×10−182× 10^-18 0.99993 1 0.0003 0.001 Sigmoid 3 50 10 151 2.2249 3.186 8×10−68× 10^-6 7×10−217× 10^-21 0.99991 0.99999 0.0007 0.001 ReLU 36 75 109 226 19.285 70.213 0 5×10−245× 10^-24 0.99997 0.99999 0.0005 0.001 4. Prem-Anu Selection Criteria In this section, we attempt to answer one of the most important questions in data-driven modeling. We show in the previous section that neural network approximations form an equivalent class with varying geometric properties across architectures. This leads to the crucial question: which of these models are good? To answer this, we introduce a model index (ℳℐM_I) defined by (3) which is a function of model capacity, effective rank and FLOPs. ℳℐ=Np(1+1reff)+logFM_I=Np (1+ 1r_eff )+ F (3) where NpNp is number of model parameters, reffr_eff is the effective rank of the Hessian of the cost function and F is the FLOPs. The proposed model index is a heuristic motivated by the observations reported in this study and is intended as a first step in the right direction of principled model selection within an equivalence class Conjecture 1. Let [fθ]ε[f_θ]_ be an equivalence class of all practically unidentifiable neural network structures defined by (2) for a given target function f. The optimal model in terms of ease of estimation, inference energy and parsimony is the one that minimizes the model index ℳℐM_I. The core idea behind calculating ℳℐM_I is that a model that minimises this index inherently minimises the number of parameters (parsimony), minimises the difference between the number of parameters and the effective rank (sloppiness), which in turn improves ease of estimation and finally FLOPs, inference time. Thus, ℳℐM_I can serve as a useful index to find an optimal model among a set of functionally equivalent models. We show numerical evidence for the proposed conjecture in Figure 1(f). The proposed criterion is illustrated by analyzing equivalent models under noise-free and noisy conditions (supplementary S4.2), where lower model index corresponds to architectures with fewer parameters, lower effective rank, and reduced FLOPs. 5. Discussion In this study, we address the question of functional non-uniqueness and geometric diversity in neural network approximations. We show that functionally equivalent network structures exhibit significant diversity in parameter-space geometry, characterized by the Hessian eigenvalue spectrum, effective rank, and singular value ratio, irrespective of network capacity. These observations are invariant to random parameter initializations. The MLP analysis further reinforces these findings. Although the addition of noise reduces the size of the equivalence class, its geometric characteristics remain largely unchanged. Across all equivalent networks, only a small fraction of the parameter space contributes meaningfully to the learned representation, while the remaining directions are redundant, as reflected by the consistently low effective rank. This suggests that the low-dimensional geometry is governed primarily by the interaction between the target function and the network architecture, and that increasing the number of parameters expands the parameter space without increasing its effective dimensionality. These findings have implications beyond functional equivalence. The existence of large equivalence classes indicates that multiple networks can approximate the same target function. Interestingly, even the smallest network within the equivalence class contains redundant parameters, motivating the search for parsimonious architectures. Since only a low-dimensional subset of parameters governs the learned representation, parameter estimation can be viewed through these effective directions rather than the full parameter space. Thus, the challenge is not merely to approximate a target function accurately, but to identify the simplest network satisfying important criteria such as parsimony, ease of estimation, and inference time. The proposed conjecture provides a first step towards selecting such an optimal model from the equivalence class. An important direction for future research is to determine whether an even smaller network can be derived from the smallest equivalent model and to understand the origin of this redundancy through the geometry of the parameter space. Author Contributions Conceptualization: Prem Jagadeesan. Formal analysis: Anuragine S A, Prem Jagadeesan. Methodology: Prem Jagadeesan. Supervision: Prem Jagadeesan. Visualization: Anuragine S A, Prem Jagadeesan. Writing – original draft: Anuragine S A, Prem Jagadeesan. Writing – review & editing: Anuragine S A, Prem Jagadeesan. . References [1] S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefler, and J. Hensman (2024) SliceGPT: compress large language models by deleting rows and columns. External Links: 2401.15024, Link Cited by: §1. [2] H. Cheng, M. Zhang, and J. Q. Shi (2024) A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations. External Links: 2308.06767, Link Cited by: §1. [3] R. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus (2014) Exploiting linear structure within convolutional networks for efficient evaluation. External Links: 1404.0736, Link Cited by: §1. [4] J. Frankle and M. Carbin (2019) The lottery ticket hypothesis: finding sparse, trainable neural networks. External Links: 1803.03635, Link Cited by: §1. [5] I. Goodfellow, Y. Bengio, and A. Courville (2016) Deep learning. MIT Press. Note: http://w.deeplearningbook.org Cited by: §1. [6] R. N. Gutenkunst, J. J. Waterfall, F. P. Casey, K. S. Brown, C. R. Myers, and J. P. Sethna (2007-10) Universally sloppy parameter sensitivities in systems biology models. PLOS Computational Biology 3 (10), p. 1–8. External Links: Document, Link Cited by: §1. [7] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: §1. [8] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §1. [9] P. Jagadeesan, K. Raman, and A. K. Tangirala (2023-03) Sloppiness: fundamental study, new formalism and its application in model assessment. PLOS ONE 18 (3), p. 1–25. External Links: Document, Link Cited by: §1. [10] K. Konstantinides and K. Yao (1988) Statistical analysis of effective singular values in matrix rank determination. IEEE Transactions on Acoustics, Speech, and Signal Processing 36 (5), p. 757–763. External Links: Document Cited by: §1. [11] Y. LeCun and G. Hinton (2015-05) Deep learning. Nature 521, p. 436–44. External Links: Document Cited by: §1. [12] S. Lin, R. Ji, C. Chen, D. Tao, and J. Luo (2019) Holistic cnn compression via low-rank decomposition with knowledge transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (12), p. 2889–2905. External Links: Document Cited by: §1. [13] Z. Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljačić, T. Y. Hou, and M. Tegmark (2025) KAN: kolmogorov-arnold networks. External Links: 2404.19756, Link Cited by: §1. [14] X. Ma, G. Fang, and X. Wang (2023) LLM-pruner: on the structural pruning of large language models. External Links: 2305.11627, Link Cited by: §1. [15] V. Papyan (2020) Traces of class/cross-class structure pervade deep learning spectra. External Links: 2008.11865, Link Cited by: §1. [16] O. Roy and M. Vetterli (2007) The effective rank: a measure of effective dimensionality. In 2007 15th European Signal Processing Conference, Vol. , p. 606–610. External Links: Document Cited by: §1. [17] W. Rudin (1987) Real and complex analysis. 3rd edition, McGraw-Hill Education, New York, NY. External Links: ISBN 978-0070542341 Cited by: §1, §3. [18] L. Sagun, L. Bottou, and Y. LeCun (2017) Eigenvalues of the hessian in deep learning: singularity and beyond. External Links: 1611.07476, Link Cited by: §1. [19] H. A. Seipel and J. H. Kalivas (2004) Effective rank for multivariate calibration methods. Journal of Chemometrics 18 (6), p. 306–311. External Links: Link Cited by: §1. [20] V. D. Tran, T. X. H. Le, T. D. Tran, H. L. Pham, V. T. D. Le, T. H. Vu, V. T. Nguyen, and Y. Nakashima (2024) Exploring the limitations of kolmogorov-arnold networks in classification: insights to software training and hardware implementation. External Links: 2407.17790, Link Cited by: §1. [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2023) Attention is all you need. External Links: 1706.03762, Link Cited by: §1. [22] Y. Wu, X. Zhu, C. Wu, A. Wang, and R. Ge (2022) Dissecting hessian: understanding common structure of hessian in neural networks. External Links: 2010.04261, Link Cited by: §1. Supplementary Material for Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization S. A. Anuragine1, Prem Jagadeesan1 1Amrita School of Artificial Intelligence Amrita Vishwa Vidyapeetham, Coimbatore, India Contents 1 Introduction 2 Experimental Design 3 Results 3.1 Analysis of Single Layer Neural Networks 3.2 Analysis of Multilayer Perceptron 4 Prem-Anu Selection Criteria 5 Discussion References S6 Definitions and Metrics S6.1 Sloppiness S6.2 Hessian Computation S6.3 Effective Rank S6.4 Normalized Root Mean Squared Error (NRMSE) S6.5 R2R^2 Score S7 Experimental Details S7.1 Data Generation S7.2 Shallow Neural Network Experiments S7.3 Multilayer Perceptron Experiments S7.4 Geometric Analysis S7.5 Reproducibility S8 Additional Results for Shallow Neural Networks S8.1 Sine Function S8.2 Cosine Function S8.3 Exponential Function S8.4 Quadratic Function S8.5 Sine Function S8.6 Reproducablity results S9 Additional Results for Multilayer Perceptrons S9.1 Geometric Analysis under Noisy Conditions S9.2 Numerical Evidence for the Proposed Conjecture S10 Codes and saved results S6. Definitions and Metrics This summarizes the principal metrics and concepts employed throughout the paper for analyzing and evaluating neural network approximations, geometric diversity, and model complexity. S6.1 Sloppiness A neural network is said to exhibit sloppiness when the eigenvalues of its Hessian span several orders of magnitude. In such models, the loss function is highly sensitive along a small number of stiff directions while remaining comparatively insensitive along many sloppy directions. Let H=∇2ℒ(θ),H=∇^2L(θ), (S4) where ℒ(θ)L(θ) denotes the training loss and θ represents the vector of trainable parameters. If λ1≥λ2≥⋯≥λn, _1≥ _2≥·s≥ _n, are the eigenvalues of H, then a broad spread of these eigenvalues is indicative of sloppy parameter-space geometry. S6.2 Hessian Computation The local geometry of the parameter space is characterized by the Hessian matrix of the least-squares loss evaluated at the optimal parameter vector θ∗θ^*. Itis approximated numerically using central finite differences. The (i,j)(i,j)-th element of the Hessian is computed as Hij(θ∗)=L(θ∗+ϵei+ϵej)−L(θ∗+ϵei−ϵej)−L(θ∗−ϵei+ϵej)+L(θ∗−ϵei−ϵej)4ϵ2,H_ij(θ^*)= L(θ^*+ε e_i+ε e_j)-L(θ^*+ε e_i-ε e_j)-L(θ^*-ε e_i+ε e_j)+L(θ^*-ε e_i-ε e_j)4ε^2, (S5) where L(θ)L(θ) denotes the least-squares loss function, ϵε is a small perturbation constant, and eie_i and eje_j are the standard basis vectors corresponding to the i-th and j-th parameters, respectively. To eliminate numerical asymmetry arising from finite-difference approximation, the Hessian is symmetrized as H←H+HT2.H← H+H^T2. (S6) The eigenvalue spectrum of the resulting Hessian is subsequently used to compute the effective rank and singular value ratio, which characterize the local geometry and sloppiness of the parameter space. S6.3 Effective Rank The entropy-based effective rank measures the intrinsic dimensionality of a matrix spectrum by interpreting the normalized singular values (or eigenvalues) as a probability distribution. Unlike the algebraic rank, the effective rank provides a continuous estimate of the number of dominant spectral directions contributing to the model. It is defined as erank(A)=exp(−∑ipilogpi),erank(A)= (- _ip_i p_i ), (S7) where pi=σi∑jσj,p_i= _i _j _j, (S8) and σi _i denotes the singular values (or absolute eigenvalues) of A. S6.4 Normalized Root Mean Squared Error (NRMSE) The normalized root mean squared error (NRMSE) measures approximation accuracy while accounting for the scale of the target data. Lower values correspond to superior approximations. NRMSE=1N∑i=1N(yi−y^i)2ymax−ymin.NRMSE= 1N _i=1^N(y_i- y_i)^2y_ -y_ . (S9) S6.5 R2R^2 Score The coefficient of determination quantifies the proportion of variance explained by the neural network approximation. R2=1−∑i=1N(yi−y^i)2∑i=1N(yi−y¯)2,R^2=1- _i=1^N(y_i- y_i)^2 _i=1^N(y_i- y)^2, (S10) where y¯ y denotes the mean of the target values. Values closer to one indicate improved approximation quality. S7. Experimental Details This provides the complete experimental settings required to reproduce all numerical results presented in the main manuscript. S7.1 Data Generation Datasets are generated from elementary univariate mathematical functions, including trigonometric, exponential, and polynomial functions. Sampling intervals are selected to sufficiently capture the behaviour of each target function while maintaining uniform sampling across the domain. Table S2 summarizes the data generation settings. Table S2: Analytical data generation settings. Function Type Domain Training Samples Sampling Method Trigonometric [−2π,2π][-2π,2π] 300 Uniform Exponential [−2,2][-2,2] 300 Uniform Polynomial [−2,2][-2,2] 300 Uniform An independent test set containing 200 uniformly sampled points over the same domain was used exclusively for evaluating approximation performance. S7.2 Shallow Neural Network Experiments Single-hidden-layer feedforward neural networks were trained using three activation functions: • ReLU • Hyperbolic tangent (tanh) • Sigmoid The output layer employed a linear activation function. The number of hidden neurons was varied from 1 to 75. All networks were trained using the mean squared error loss function and stochastic gradient descent. Parameters were initialized randomly. For every function–activation pair, models were ranked according to numerically indistinguisable R2R^2 Table S3 summarizes the complete experimental configuration. Table S3: Shallow neural network configuration. Parameter Value Architecture Single hidden layer Hidden neurons 1–75 Activation functions ReLU, tanh, Sigmoid Output activation Linear Loss function Mean squared error Training algorithm Stochastic gradient descent Weight initialization Random Selection criterion Top-K models satisfying NRMSE<εNRMSE< S7.3 Multilayer Perceptron Experiments To investigate functional equivalence across architectures of varying depth and width, multilayer perceptrons containing one to five hidden layers were examined. Each hidden layer contained between one and five neurons, yielding a total of 3905 unique network architectures. All architectures were trained independently using identical optimization settings. To evaluate robustness under measurement uncertainty, additive white Gaussian noise corresponding to an SNR of 12.04 dB (25% noise level) was introduced into the training data. For both the noise-free and noisy datasets, models were ranked according to the coefficient of determination (R2R^2), and the Top-K equivalent models were selected for further analysis. Computational complexity was quantified using the number of floating-point operations (FLOPs) required for one forward pass. Table S4 summarizes the complete experimental configuration. Table S4: Multilayer perceptron configuration. Parameter Value Hidden layers 1–5 Neurons per layer 1–5 Total architectures 3905 Noise conditions Noise-free and 12.04 dB SNR Ranking metric R2R^2 Complexity metric FLOPs S7.4 Geometric Analysis The local geometry of each trained model was characterized by numerically approximating the Hessian matrix around the optimal parameter vector using central finite differences. The following quantities were extracted from every Hessian: • Eigenvalue spectrum • Effective rank • Singular value ratio Consequently, every neural network was characterized using • approximation accuracy, • number of trainable parameters, • FLOPs, • effective rank, • singular value ratio. S7.5 Reproducibility To ensure robustness with respect to random initialization, every experiment was repeated using multiple random seeds. The complete optimization and geometric analysis pipeline was independently executed for each initialization. Unless otherwise stated, reported results correspond to the sample mean and sample standard deviation computed across these independent runs. S8. Additional Results for Shallow Neural Networks This section presents additional visualizations for the shallow neural network experiments. For every target function, the Top-K functionally equivalent networks are shown together with their Hessian geometry, architectural complexity, and eigenvalue spectra. S8.1 Sine Function Figure S2: Analysis of the sin function using the tanh activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S3: Analysis of the sin function using the sigmoid activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S4: Analysis of the sin function using the RELU activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equivalent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence S8.2 Cosine Function Figure S5: Analysis of the cosin function using the tanh activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S6: Analysis of the cosin function using the sigmoid activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S7: Analysis of the cosin function using the RELU activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equivalent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence S8.3 Exponential Function Figure S8: Analysis of the exponential function using the tanh activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S9: Analysis of the exponential function using the sigmoid activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S10: Analysis of the exponential function using the RELU activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equivalent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence S8.4 Quadratic Function Figure S11: Analysis of the quadratic function using the tanh activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S12: Analysis of the quadratic function using the sigmoid activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S13: Analysis of the quadratic function using the RELU activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equivalent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence S8.5 Sine Function Figure S14: Analysis of the cubic function using the tanh activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S15: Analysis of the cubic function using the sigmoid activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equiva- lent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S16: Analysis of the cubic function using the RELU activation function. (a) Top-K functionally equivalent approximations. (b) Hessian ellipses illustrating geometric diversity. (c) Distribution of equivalent networks with respect to hidden neurons and training R2. (d) Hessian eigenvalue spectra highlighting differences in parameter-space geometry despite functional equivalence Figure S17: Columns correspond to the singular value ratio, effective rank, and Hessian eigenvalue spectrum, respectively. Rows illustrate representative models trained for the sine, exponential, and cubic target functions. S8.6 Reproducablity results This summarizes the variability observed across multiple random initializations. The reported statistics correspond to the mean and standard deviation computed over independent training runs for the Top-K functionally equivalent networks. Table S5: Functional non-uniqueness across different random initializations. Reported values correspond to the average minimum and average maximum observed among the Top-K equivalent networks over multiple independent runs. Function Top-K Neurons Parameters Effective Rank Sloppiness R2R^2 NRMSE Sine 5 45 – 50 137 –151 (1.8±0.1)×10−5(1.8± 0.1)× 10^-5 – (2.0±0.2)×10−4(2.0± 0.2)× 10^-4 (5.4±0.1)×10−5(5.4± 0.1)× 10^-5 – (1.7±0.1)×10−4(1.7± 0.1)× 10^-4 0.9998±10−60.9998± 10^-6 0.004±10−50.004± 10^-5 10 41 – 50 123 –151 (1.8±0.1)×10−5(1.8± 0.1)× 10^-5 – (1.84±0.2)×10−4(1.84± 0.2)× 10^-4 (5.4±0.1)×10−5(5.4± 0.1)× 10^-5 – (1.7±0.1)×10−4(1.7± 0.1)× 10^-4 0.9998±10−60.9998± 10^-6 0.005±10−50.005± 10^-5 15 5 –50 16 –151 (1.8±0.1)×10−5(1.8± 0.1)× 10^-5 – (2.6±0.3)×10−3(2.6± 0.3)× 10^-3 (5.4±0.1)×10−5(5.4± 0.1)× 10^-5 – (1.7±0.1)×10−4(1.7± 0.1)× 10^-4 0.9997±10−50.9997± 10^-5 0.006±10−40.006± 10^-4 20 5 –50 16 –151 (1.8±0.1)×10−5(1.8± 0.1)× 10^-5 – (2.6±0.3)×10−3(2.6± 0.3)× 10^-3 (5.4±0.1)×10−5(5.4± 0.1)× 10^-5 – (1.7±0.1)×10−4(1.7± 0.1)× 10^-4 0.9997±10−50.9997± 10^-5 0.006±10−40.006± 10^-4 25 5 –50 16 –151 (1.8±0.1)×10−5(1.8± 0.1)× 10^-5 – (2.6±0.3)×10−3(2.6± 0.3)× 10^-3 (5.4±0.1)×10−5(5.4± 0.1)× 10^-5 – (1.7±0.1)×10−4(1.7± 0.1)× 10^-4 0.9997±10−50.9997± 10^-5 0.006±10−40.006± 10^-4 S9. Additional Results for Multilayer Perceptrons This presents supplementary results obtained from the multilayer perceptron (MLP) experiments. The figures illustrate the geometric characteristics of the Top-K functionally equivalent models under noisy training conditions together with numerical evidence supporting the proposed conjecture. S9.1 Geometric Analysis under Noisy Conditions To examine the effect of noise in functional equivalence, geometric diversity and model redundancy, the same set of 3905 MLP configurations are evaluated after adding gaussian noise to the data. In this case, the equivalence class consists of 106 networks with R2≥0.94R^2≥ 0.94. Although the size of the equivalence class is smaller than in the noise-free case, there is still a significant number of networks that approximate the given noisy data. Figure S18 presents the geometric analysis of the equivalence class obtained under 25%25\% noise. Figure S18(a) shows that for nearly identical R2R^2 values, the networks have diverse geometric properties captured by effective rank and ratio of singular values. Figure S18(b) shows that the upper bound of the ratio of effective rank to number of parameters still remains ≈ 0.25, even while fitting noisy data. Figure S18(c) shows that the equivalent networks are mainly concentrated within the parameter range of 60 to 120 and effective rank ranging from 2.5 to 9. Figure S18(d) further shows that the equivalent networks are concentrated mainly within the parameter range of 40 to 120 with singular value ratio varying from 10−1110^-11 to 10−610^-6 . Figure S18: Geometric analysis of the equivalence class obtained under noisy observations (R2≥0.94R^2≥ 0.94). (a) Distribution of equivalent networks ranked according to training R2R^2, together with the corresponding singular value ratio. (b) Ratio of effective rank to the total number of trainable parameters, illustrating the effective utilization of the parameter space. (c) Relationship between the number of trainable parameters and the effective rank. (d) Relationship between the number of trainable parameters and the logarithm of the singular value ratio. S9.2 Numerical Evidence for the Proposed Conjecture To numerically examine the proposed conjecture, we analyze models that belong to equivalence class in noise-free and noisy conditions. For each network, the number of trainable parameters and the corresponding FLOPs are computed, while the effective rank is estimated from the Hessian at optimal parameter vector. A network is said to be optimal in terms of parsimony, ease of estimation and inference energy when it minimizes the model index. The model index is computed for every architecture and plotted using a color gradient, where lower values (blue) indicate architectures that better satisfy the proposed criterion. As observed in Figure S19, models that have low ℳℐM_I are found to have low effective, less number of parameters and minimal FLOPs are indeed concentrated in the expected region. Figure S19: The three axes represent the number of trainable parameters, floating-point operations (FLOPs), and effective rank. The colour scale denotes the proposed model index, illustrating the trade-off between model complexity, computational cost, and effective parameter-space dimensionality across the equivalence class. Models with lower values of the proposed index are considered more parsimonious while maintaining comparable approximation performance. S10. Codes and saved results Source Code The complete MATLAB implementation, including scripts for data generation, neural network training, Hessian computation, effective rank estimation, and figure generation, is available at: Source Codes Experimental Results and Workspaces The complete collection of MATLAB workspaces, trained neural network models, intermediate experimental results, generated datasets, and supplementary analysis files required to reproduce the results presented in this paper is available at: Experimental Results and Workspaces