Paper deep dive
Designing Compact Neural Architectures via Neuron Gating and Mixed Activation
Abhishek Shukla, Ankur Sinha, Faiz Hamid
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural Architecture Search (NAS) is naturally formulated as a bilevel optimization problem, where the upper-level optimizes the architecture using validation performance and the lower-level trains network parameters using training loss. However, NAS is computationally expensive due to discrete architectural decisions, exponentially growing search spaces, and the high cost of training candidate architectures. This work develops a general bilevel optimization framework for NAS across diverse architectures, including MLPs, CNNs, RNNs, and Transformers, to identify compact architectures with strong predictive performance. We propose three scalable formulations that replace discrete neuron- and activation-level decisions with continuous relaxations, enabling differentiable optimization over otherwise combinatorial architecture spaces. These formulations give rise to three NAS methods: NAS based on Neuron Gating (NAS-NG), NAS based on Mixed Activation (NAS-MA), and NAS based on Neuron Gating and Mixed Activation (NAS-NGMA). Experiments on MLPs and CNNs using MNIST and CIFAR-10 show that the proposed methods consistently identify compact architectures with competitive or improved predictive performance. On MNIST, NAS-NGMA achieves 98.68% test accuracy with 7.69M MLP parameters, while NAS-NG achieves 99.63% accuracy with only 0.26M CNN parameters. On CIFAR-10, the proposed methods consistently outperform vanilla DARTS. Further experiments demonstrate that NAS-NG can optimize substantially over-parameterized and literature-optimal architectures, improving accuracy while reducing parameters. These results establish relaxed bilevel optimization as a scalable alternative to discrete NAS and provide a general framework for efficient neuron- and activation-level architecture optimization.
Tags
Links
- Source: https://arxiv.org/abs/2608.14443v1
- Canonical: https://arxiv.org/abs/2608.14443v1
Trouble viewing inline? Open PDF directly →
Full Text
104,112 characters extracted from source content.
Expand or collapse full text
Designing Compact Neural Architectures via Neuron Gating and Mixed Activation Abhishek Shukla Ankur Sinha Faiz Hamid 1Department of Management Sciences, IIT Kanpur, India abhiskl@iitk.ac.in, fhamid@iitk.ac.in 2Krishnamurthy Tandon School of AI, IIM Ahmedabad, India asinha@iima.ac.in Abstract Neural Architecture Search (NAS) is naturally formulated as a bilevel optimization problem, where the upper-level task involves optimizing the neural architecture with respect to validation performance, while the lower-level task focuses on training the network parameters by minimizing the training loss. However, NAS, being a combinatorial optimization problem, is computationally expensive due to the discrete nature of architectural decisions, the exponential growth of the search space with network depth and width, and the high cost of training candidate architectures to optimality. This work aims to develop a fundamental method for NAS that is applicable across diverse neural network architectures—including MLPs, CNNs, RNNs, and Transformers—for identifying compact architectures with strong predictive performance. We propose three scalable and efficient bilevel optimization formulations for NAS. This replaces discrete neuron- and activation-level architectural decisions with continuous relaxations, enabling differentiable optimization over otherwise combinatorial architecture spaces. The resulting differentiable bilevel problems are efficiently solved using hypergradient-based methods, giving rise to the NAS methods termed NAS based on Neuron Gating (NAS-NG), NAS based on Mixed Activation (NAS-MA), and NAS based on Neuron Gating and Mixed Activation (NAS-NGMA). The proposed methods are evaluated on MLP and CNN architectures with varying depths and widths using the MNIST dataset. They are further assessed on the CIFAR-10 dataset against vanilla Differentiable Architecture Search (DARTS). Across these experiments, the proposed approaches consistently identify compact architectures while achieving competitive or improved predictive performance. In particular, on MNIST, NAS-NGMA achieves 98.68% test accuracy with 7.69M MLP parameters, while NAS-NG attains 99.63% test accuracy with only 0.26M CNN parameters. On CIFAR-10, the proposed methods consistently outperform the DARTS method. Additional experiments further show that NAS-NG scales effectively to substantially over-parameterized architectures and can further improve literature-optimal architectures through neuron-level optimization, yielding higher accuracy with significantly fewer parameters. These findings establish relaxed bilevel optimization as a practical and scalable alternative to discrete search for combinatorial NAS, one that replaces expensive architecture-level search with efficient, gradient-based optimization at the neuron and activation level. Because the formulations are agnostic to the underlying network type, they offer a general route to compact, high-performing architectures across MLPs, CNNs, and beyond, and point toward extending neuron-level optimization to further improve even well-established, hand-tuned architectures. Keywords: Neural architecture search, Bilevel optimization, Neural network pruning, Gated activation. 1 Introduction Artificial Neural Networks (ANNs) consist of layered, interconnected neurons whose trainable parameters (weights and biases) are optimized to minimize task-specific loss, whereas hyperparameters, such as network depth and width, activation functions, and optimization settings, are fixed prior to training. Among these, architectural parameters play a decisive role in determining a model’s expressive power and generalization ability. Increasing the number of layers enhances representational capacity, but excessive capacity can result in overfitting, where the models fit the training data perfectly but generalize poorly. Traditionally, neural architectures have been designed through expert intuition and extensive trial-and-error. In the absence of domain expertise, discovering an effective architecture can be highly challenging. Furthermore, even expert-driven design does not necessarily yield optimal architectures, especially as the complexity of modern deep neural networks continues to grow. Consequently, architectural design remains a fundamental challenge in deep learning. Neural Architecture Search (NAS), a subfield of Automated Machine Learning (AutoML) [47] and Hyperparameter Optimization (HPO) [9], seeks to automate network design by systematically exploring large architecture search spaces. These efforts also connect to Automated Reinforcement Learning (AutoRL) [28], which applies AutoML principles to RL while addressing additional challenges unique to RL agents, whose success is often highly sensitive to training design choices. The primary objective of NAS is to identify an optimal architecture from a large and complex set of design choices tailored to a specific task and dataset. A comprehensive conceptual framework for NAS is presented in [10]. Although effective, NAS poses significant computational challenges due to the discrete nature of architectural decisions and the high cost of training many candidate architectures. This motivates optimization-based formulations that can exploit the structure in the search space. Formally, NAS, like many HPO tasks, can be naturally formulated as a Bilevel Optimization Problem (BOP) [31, 35]. In this formulation, the leader’s problem determines the architectural parameters, while the follower’s problem optimizes the corresponding network weights through standard training. The hierarchical separation in NAS is not merely cosmetic: it prevents architectures from being evaluated using inadequately trained weights and ensures that the search trajectory is guided by generalization rather than overfitting. In this way, bilevel optimization provides a theoretically grounded framework for NAS that disentangles architecture evaluation from weight optimization. However, the NAS search space is typically high-dimensional and discrete, rendering the optimization process both computationally demanding and algorithmically challenging. A complementary perspective on architectural discovery comes from evolutionary approaches: 37 demonstrate that incrementally complexifying neural architectures through evolutionary search—rather than optimizing over a fixed structure—leads to significantly more sophisticated solutions, underscoring the importance of allowing the search process itself to evolve. Despite these complexities, NAS has demonstrated remarkable success in automating architectural design, substantially reducing human intervention and producing models that match or even outperform manually crafted architectures. When another Machine Learning (ML) model—such as RL [48]—is employed to explore the architecture space of a Neural Network (N), the search often incurs extremely high computational costs (on the order of 2000 GPU days), primarily due to the combinatorial explosion of candidate architectures and the inherent difficulty of learning-based approaches in consistently navigating billions of possibilities to identify the best feasible (optimal) design, given the inevitable gap between predicted and optimal decisions. This limitation motivates the adoption of optimization-driven architecture search frameworks rather than relying solely on conventional ML-based or naive search strategies. In this direction, a prominent and efficient approach that avoids discrete optimization is Differentiable Architecture Search (DARTS) [24], which relaxes the architecture search space into a continuous domain, thereby enabling a tractable bilevel optimization formulation. In recent years, there has been a significant surge in the DARTS literature, leading to the development of numerous variants of the original algorithm [6, 42, 26, 7, 45, 46, 43, 14, 22, 5]. Notably, 38 propose a heterogeneous and flexible transformer search space, demonstrating that relaxing homogeneity assumptions in architecture design—combined with surrogate-based bilevel optimization—yields substantially more compact and higher-performing models than their homogeneous counterparts. We also aim to obtain a high-performing and compact neural network architecture by solving a continuous bilevel NAS problem using a hypergradient-based method [13], inspired by the DARTS framework. Beyond NAS, BOPs have also been employed for overfitting control in large language models, including transformer architectures, as demonstrated in [34]. Figure 1: Progressive construction of MLP architectures via neuron- and activation-level modifications. This paper presents a fundamental method for NAS with the potential for broad applicability. In particular, we develop three relaxed bilevel formulations of the original combinatorial NAS problem, enabling a tractable exploration of the architecture search space. The proposed approach replaces discrete neuron- and activation-level decisions with continuous variables, resulting in differentiable bilevel programs that can be efficiently optimized. Multilayer Perceptron (MLP) and Convolutional Neural Network (CNN) architectures are employed as testbeds to systematically evaluate the effectiveness of the proposed formulations on standard benchmark datasets. Figure 1 illustrates the methodological progression from the original MLP architecture (a) to the proposed relaxed models (b, c, and d) based on Neuron Gating (NG), Mixed Activation (MA), and Neuron Gating and Mixed Activation (NGMA). Although the formulations are developed in the context of MLPs, the framework is general and extends naturally to CNNs and Recurrent Neural Networks (RNNs) through appropriate tensor generalizations. The effectiveness of the proposed approach is validated through a comprehensive comparative experimental evaluation of MLP and CNN architectures on the MNIST dataset and CNN architectures on the CIFAR-10 dataset. The main contributions of this work are summarized as follows: 1. Relaxed bilevel formulation: We formulate NAS as relaxed bilevel optimization problem by replacing discrete neuron- and activation-level design choices with continuous variables, enabling differentiable and scalable optimization. 2. Novel NAS algorithms: We propose three novel strategies for architecture search: NAS-NG, NAS-MA, and NAS-NGMA. 3. Comprehensive experimental evaluation: We conduct a comprehensive experimental evaluation on MLP and CNN architectures for MNIST, and CNN architectures for CIFAR-10, demonstrating the effectiveness of the proposed formulations. 4. Superior performance over DARTS method: We empirically show that the proposed approaches outperform vanilla DARTS on CIFAR-10. 5. Compact and accurate architectures: We show that the proposed methods consistently identify compact architectures with competitive or improved predictive performance on MNIST dataset, achieving approximately 40–75% reduction in model parameters. Similar trends are observed on the CIFAR-10. 6. Competitive MNIST results: We demonstrate that the best-performing models achieve 98.68% test accuracy with 7.69M parameters for MLPs (NAS-NGMA), and 99.63% test accuracy with only 0.26M parameters for CNNs (NAS-NG). 7. Improving baseline and optimal architectures: We show that NAS-NG remains effective when initialized from substantially over-parameterized architectures and can further improve literature-optimal architectures (architecture fine-tuning), demonstrating that neuron-level optimization complements existing topology search methods. The remainder of this paper is organized as follows. Section 2 reviews the relevant literature, and Section 3 presents the necessary preliminaries. Section 4 introduces the proposed framework. Section 5 presents a comprehensive empirical evaluation, including experiments on MLP and CNN architectures for the MNIST dataset, where the proposed relaxed bilevel optimization formulations are compared against baseline models. This section further analyzes optimization behavior, generalization performance, and model complexity, and highlights the key findings. In addition, the proposed NAS methods are evaluated on the CIFAR-10 dataset and compared with DARTS method. Finally, Section 6 concludes the paper by discussing the limitations, summarizing the main contributions, and outlining directions for future work. 2 Related Works In this section, we discuss advancements in neural network-based handwritten digit recognition and differentiable NAS, including classical machine learning methods, deep neural architectures, ensemble techniques, and DARTS-based NAS methods. 2.1 Neural Architectures and Learning Methods Early work laid the foundation for gradient-based learning and CNNs for document recognition tasks [20]. Alongside deep learning approaches, classical ML methods continue to play a significant role. For example, enhancements to multi-layer logistic regression using outlier detection have been proposed in [17], while privacy-preserving learning using homomorphic encryption has been explored for logistic regression in [15]. Dimensionality reduction techniques such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) have also been widely studied for handwritten digit recognition [29], and advances in similarity-based methods include quantum k-nearest neighbor algorithms [41]. With the advent of deep learning, CNNs have consistently demonstrated superior performance over MLPs, primarily due to their ability to effectively capture spatial hierarchies in image data [4]. Extensive efforts have been made to improve CNN performance by analyzing architectural and training factors, including the impact of network depth and training epochs [3], as well as structural parameters such as stride and receptive fields [1]. The use of deep, pre-trained architectures, such as AlexNet and GoogleNet, has further advanced performance benchmarks [36]. Beyond conventional CNNs, biologically inspired models, such as deep spiking CNNs based on Spike Time Dependent Plasticity (STDP), have also been explored [16]. In addition, ensemble learning approaches that combine multiple base learners have been widely adopted to achieve state-of-the-art performance [27, 39]. Several high-performing models have been reported in the literature, including those in [2, 19, 40, 8, 33], highlighting the continued progress in this domain. 2.2 Differentiable Architecture Search The DARTS method, introduced in [24], presented a highly efficient and competitive NAS approach compared to RL- and Evolutionary Computation (EC)-based methods, functioning within a continuous search space. Building on DARTS, Progressive Differentiable Architecture Search (P-DARTS) was developed by [6], which addressed computational challenges and enhanced search stability through search space approximation and regularization techniques. Further, Robustness of Differentiable Architecture Search (R-DARTS) was explored in [44] through a methodical exploration of architectural spaces and regularization strategies. In [42], the authors introduced an innovative method known as partially-connected differentiable architecture search (PC-DARTS) with the obvious goal of improving the efficiency and stability of NAS. Att-DARTS algorithm devised in [26] is another innovative extension of the DARTS framework, adeptly integrating attention modules into the NAS process. The manifestation of the collapse phenomenon in DARTS, characterized by an excessive occurrence of skip-connects over an extensive number of search epochs due to overfitting of the one-shot model, resulted in diminished performance. This challenge was effectively addressed by introducing early stopping criteria in DARTS+ [23], where DARTS stops when two or more skip-connects appear in a normal cell or when the architecture parameter ranking remains stable for a predefined number of epochs. Fair DARTS [7] was introduced to tackle issues of collapse observed in the DARTS algorithm due to unfair advantages in exclusive competition among candidate operations during the search process. DE-DARTS developed by [25] addressed the challenges of gradient-based NAS by proposing a novel approach that incorporated Dynamic Attention Networks (DANs). More recently, EG-DARTS [45] used a multi-objective evolution-based approach, combining gradient optimization with evolutionary strategies to improve DARTS’ search effectiveness. Additional advancements have been made with methods like Relax-DARTS [46], STO-DARTS [5], OSTR-DARTS [43], semantic DARTS [14], LMD-DARTS [22], HN-DARTS [21], and DARTS-EAST [11], among others, highlighting continued innovation within the DARTS literature. 3 Preliminaries Formally, the bilevel NAS problem can be expressed as follows: minA _A ℒv(A,W^) _v(A, W) (1) s.t. .t. W^∈argminW∈ℒt(A,W) W∈ argmin _W∈ WL_t(A,W) A∈, A∈ A, where ℒtL_t and ℒvL_v denote the training and validation losses, respectively, A denotes the architecture space, and W denotes the network weight space. The lower-level problem mirrors the conventional training of an N, where the weights W are optimized to minimize the training loss for a fixed architecture A. In contrast, the upper-level problem seeks architectures that generalize well, as measured by the validation loss. The gradient of the upper-level objective (validation loss) with respect to the upper-level decision variable A is called the hypergradient. One of the most well-known optimization methods for NAS is based on an approximate hypergradient [13]. This hypergradient is used to update the architecture parameters to improve validation performance while accounting for the model weights obtained after one step of gradient descent on the training loss. The hypergradient is approximated as ∇Aℒv(A,W^)≈∇Aℒv(A,W−ξ∇Wℒt(A,W)), _AL_v(A, W)≈ _AL_v (A,W-ξ _WL_t(A,W) ), where ξ is a small learning rate used for one-step unrolled optimization of the model training problem. Setting ξ=0ξ=0 yields a first-order approximation, while ξ≠0ξ≠ 0 introduces a second-order correction ∇Aℒv(A,W^)≈∇Aℒv(A,W′)−ξ∇A,W2ℒt(A,W)∇W′ℒv(A,W′), _AL_v(A, W)≈ _AL_v(A,W )-ξ∇^2_A,WL_t(A,W) _W L_v(A,W ), (2) with W′=W−ξ∇Wℒt(A,W)W =W-ξ _WL_t(A,W). The second term captures the interaction between the architecture parameters and the network weights through the mixed Hessian of the training loss, thereby accounting for their influence on the approximate hypergradient. This term can be efficiently approximated using finite-difference schemes. For further details on DARTS, see [24]. 4 Proposed Method Consider a fully connected N depicted in Figure 2. This network is composed of an input layer, L hidden layers, and an output layer. The symbols used to represent this MLP are detailed in Table 1. Figure 2: Schematic representation of a general MLP architecture with an input layer, L hidden layers, and an output layer. Table 1: Notations used for the MLP model. Notation Description ST,SVS^T,\,S^V Training and validation sets i∈ℝn0,yi∈0,1x_i ^n_0,\,y_i∈\0,1\ Input vector and true output label for the ithi^th sample j=0j=0 Index of the input layer j=1,…,l,l+1,…,Lj=1,…,l,\;\;l+1,…,L Indices of the hidden layers j=L+1j=L+1 Index of the output layer njn_j Number of neurons in the jthj^th layer (j)∈ℝnj+1×njW^(j) ^n_j+1× n_j Weight matrix connecting layer j to layer j+1j+1 (j)∈ℝnj+1b^(j) ^n_j+1 Bias vector associated with the (j+1)th(j+1)^th layer f(⋅)f(·) Element-wise activation function (ReLU) i(j)z_i^(j) Pre-activation vector of the (j+1)th(j+1)^th layer for the ithi^th sample i(j)=f(i(j))h_i^(j)=f(z_i^(j)) Activation vector of the jthj^th layer for the ithi^th sample =0,1,…,L+1J=\0,1,…,L+1\ Set of all layer indices j=1,…,njK_j=\1,…,n_j\ Set of neuron indices in the jthj^th layer Figure 3: Neural network with one hidden layer. The structure of an MLP can be defined by the count of neurons present in its input, hidden, and output layers, and is depicted as a tuple (Input=[n0],Hidden=[n1,…,nl,nl+1,…,nL],Output=[nL+1])(Input=[n_0],\;Hidden=[n_1,…,n_l,\;\;n_l+1,…,n_L],\;Output=[n_L+1]) This representation allows for heterogeneous hidden layers with varying widths. To illustrate, imagine a feedforward N that has two neurons for input, a single hidden layer with three neurons, and one neuron for output., i.e., Architecture=(Input=[2],Hidden=[3],Output=[1]),Architecture=(Input=[2],\;Hidden=[3],\;Output=[1]), as shown in Figure 3. This network is designed for binary classification. The hidden layer’s neurons utilize the ReLU activation function, whereas the output neuron employs a sigmoid activation function to generate a probabilistic output that corresponds to the input vector ix_i given as follows: i=[x1ix2i]∈ℝ2x_i= bmatrixx_1i\\ x_2i bmatrix ^2 (3) The hidden layer is parameterized by a weight matrix (1)∈ℝ3×2W^(1) ^3× 2 and a bias vector (1)∈ℝ3b^(1) ^3, while the output layer is parameterized by a weight vector (2)∈ℝ1×3W^(2) ^1× 3 and a scalar bias b(2)∈ℝb^(2) . Let the collection of all network weights and biases be denoted by =(1),(1),(2),b(2)∈ℝ13W=\W^(1),b^(1),W^(2),b^(2)\ ^13. We can write the binary cross-entropy loss as follows (derived in Appendix A). ℒi(;i,yi)=−[ _i(W;x_i,y_i)=- [ yilog(11+exp(−[∑k=13wk(2)max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)])) y_i \! ( 11+ \! (- [ _k=1^3w^(2)_k \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) (4) +(1−yi)log(1−11+exp(−[∑k=13wk(2)max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)]))] +(1-y_i) \! (1- 11+ \! (- [ _k=1^3w^(2)_k \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) ] Model Training Problem. The optimal model parameters are obtained by solving an unconstrained, highly nonlinear optimization problem defined by minimizing the loss function over the model’s parameter space for all the training samples, as given below: min∈ℝ13 _W ^13 ℒt()=1|ST|∑(i,yi)∈STℒi(,i,yi) _t\! (W )= 1|S^T| _(x_i,y_i)∈ S^TL_i\! (W;\,x_i,y_i ) (5) Considering a general model parameter space W, this problem can be generalized as min∈ _W∈ W ℒt()=1|ST|∑(i,yi)∈STℒi(,i,yi) _t\! (W )= 1|S^T| _(x_i,y_i)∈ S^TL_i\! (W;\,x_i,y_i ) (6) The training problem is commonly solved using gradient-based optimization techniques, including Adaptive Moment Estimation (Adam), Stochastic Gradient Descent (SGD), and several other related first-order methods. 4.1 Bilevel Formulations for MLP Architecture Search In this section, we develop bilevel formulations for MLP architecture search under three settings. First, we consider neuron gating, where architecture parameters (associated with hidden neurons) control the effective participation of individual hidden neurons. Second, we investigate mixed activation function-based architecture search, in which each hidden neuron is associated with a mixed activation function (a convex combination of activation functions from a predefined set of candidate operations). By optimizing the corresponding mixing coefficients, the search process ultimately selects a single optimal activation function for each neuron. In the third setting, we unify both perspectives by jointly optimizing neuron selection and activation function within a single bilevel framework. We next describe these three settings as follows. 4.1.1 Neuron Gating (NG)-based MLP Architecture Search Consider the following architecture parameter vector defined for neuron gating in the lthl^th hidden layer (l)=[α1(l)α2(l)⋯αnl(l)]⊤∈ℝnl,l=1,…,L α^(l)= bmatrixα^(l)_1&α^(l)_2&·s&α^(l)_n_l bmatrix ^n_l, l=1,…,L (7) Here, αk(l)α^(l)_k denotes the architecture parameter associated with the kthk^th neuron in the lthl^th hidden layer. Each αk(l)α^(l)_k is an unconstrained real-valued variable, while the corresponding effective neuron activation gate is obtained via the sigmoid mapping σ(αk(l))∈(0,1)σ (α^(l)_k )∈(0,1) The index k ranges from 11 to nln_l, and the index l=1,…,Ll=1,…,L enumerates the hidden layers. The global architecture parameter vector across all hidden layers is then defined as =((1),(2),…,(L))A= ( α^(1), α^(2),…, α^(L) ) (8) Figure 4: Neural network architecture illustrating the neuron gating mechanism employed for MLP architecture search. For the simple modified neural network shown in Figure 4, the global architecture parameter vector is given by =[α1(1)α2(1)α3(1)]⊤∈ℝ3A= bmatrixα^(1)_1&α^(1)_2&α^(1)_3 bmatrix ^3 (9) The modified binary cross-entropy loss function used for architecture search is given as follows (derived in Appendix B): ℒi(,;i,yi)=−[ _i(A,W;x_i,y_i)=- [ yilog(11+exp(−[∑k=13wk(2)11+exp(−αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)])) y_i \! ( 11+ \! (- [ _k=1^3w^(2)_k 11+ \! (-α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) (10) +(1−yi)log(1−11+exp(−[∑k=13wk(2)11+exp(−αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)]))] +(1-y_i) \! (1- 11+ \! (- [ _k=1^3w^(2)_k 11+ \! (-α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) ] MLP Architecture Search Problem Formulation. The resulting MLP architecture search problem is formulated as the bilevel optimization problem below, where ℒi(.)L_i\! (. ) is given by (10): min∈ℝ3 _A ^3 ℒv(,^)=1|SV|∑(i,yi)∈SVℒi(,^,i,yi) _v\! (A, W )= 1|S^V| _(x_i,y_i)∈ S^VL_i\! (A, W;\,x_i,y_i ) (11) s.t. .t. ^∈argmin∈ℝ13ℒt(,)=1|ST|∑(i,yi)∈STℒi(,,i,yi) W∈ argmin _W ^13L_t\! (A,W )= 1|S^T| _(x_i,y_i)∈ S^TL_i\! (A,W;\,x_i,y_i ) Considering general model parameter space ( W) and architecture parameter sapce ( A), the architecture search problem can be generalized as min∈ _A∈ A ℒv(,^)=1|SV|∑(i,yi)∈SVℒi(,^,i,yi) _v\! (A, W )= 1|S^V| _(x_i,y_i)∈ S^VL_i\! (A, W;\,x_i,y_i ) (12) s.t. .t. ^∈argmin∈ℒt(,)=1|ST|∑(i,yi)∈STℒi(,,i,yi) W∈ argmin _W∈ WL_t\! (A,W )= 1|S^T| _(x_i,y_i)∈ S^TL_i\! (A,W;\,x_i,y_i ) 4.1.2 Mixed Activation (MA)-based MLP Architecture Search In this case, the architecture search is performed by learning the activation function associated with each hidden neuron. Let ℱ=f1,f2,…,fMF=\f_1,f_2,…,f_M\ denote a predefined set of M candidate activation functions, respectively. Each hidden neuron k in the lthl^th hidden layer is associated with an activation architecture parameter vector given by k(l)=[βk1(l)βk2(l)⋯βkM(l)]⊤∈ℝM β^(l)_k= bmatrixβ^(l)_k_1&β^(l)_k_2&·s&β^(l)_k_M bmatrix ^M (13) Collecting the activation architecture parameters for all neurons in the lthl^th hidden layer, we define (l)=[1(l)2(l)⋯nl(l)]⊤∈ℝnl×M,l=1,…,L β^(l)= bmatrix β^(l)_1& β^(l)_2&·s& β^(l)_n_l bmatrix ^n_l× M, l=1,…,L (14) The global activation architecture parameter across all hidden layers is then defined as =((1),(2),…,(L))A= ( β^(1), β^(2),…, β^(L) ) (15) For a given hidden neuron k in layer l, the mixing weight corresponding to the mthm^th activation function is obtained via a softmax mapping given as follows π(l)km=exp(βkm(l))∑m′=1Mexp(βkm′(l)),m=1,…,Mπ^(l)_k_m= (β^(l)_k_m) _m =1^M (β^(l)_k_m ), m=1,…,M (16) Figure 5: Modified neural network architecture for mixed activation-based architecture search. For the simple modified neural network shown in Figure 5, we consider M=7M=7 candidate activation functions, ℱ=f1,f2,f3,f4,f5,f6,f7=None,Identity,ReLU,Leaky ReLU,tanh,σ,SiLUF=\f_1,f_2,f_3,f_4,f_5,f_6,f_7\=\None,Identity,ReLU,Leaky ReLU, ,σ,SiLU\, for each hidden neuron, as given in Appendix C. Since the network contains a single hidden layer with three neurons, the global architecture parameter matrix is given by =(1)∈ℝ3×7A= β^(1) ^3× 7 (17) Accordingly, the mixing weights for the activation functions of each hidden neuron are computed as π(1)km=exp(βkm(1))∑m′=17exp(βkm′(1)),m=1,…,7π^(1)_k_m= (β^(1)_k_m) _m =1^7 (β^(1)_k_m ), m=1,…,7 (18) The activation of the kthk^th hidden neuron for the ithi^th input sample is then defined as a convex combination of the candidate activation functions hki′(1)=∑m=17πkm(1)fm(wk1(1)x1i+wk2(1)x2i+bk(1))h (1)_ki= _m=1^7π^(1)_k_mf_m\! (w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k ) (19) The pre-activation of the output neuron is given by zi′(2)=∑k=13wk(2)hki′(1)+b(2),z_i (2)= _k=1^3w^(2)_kh (1)_ki+b^(2), (20) and the predicted output is obtained via a sigmoid activation y^i′=σ(zi′(2)) y_i =σ\! (z_i (2) ) (21) Given the ground-truth label yi∈0,1y_i∈\0,1\, the modified binary cross-entropy loss for the ithi^th sample is defined as ℒi(,,i,yi)=−[yilog(y^i′)+(1−yi)log(1−y^i′)]L_i(A,W;x_i,y_i)=- [y_i ( y_i )+(1-y_i) (1- y_i ) ] (22) MLP Architecture Search Problem Formulation. The MA-based MLP architecture search problem is formulated as in (11), with the architecture parameters specified by (17). 4.1.3 Neuron Gating and Mixed Activation (NGMA)-based MLP Architecture Search In this case, neuron selection and activation-function optimization are performed jointly within a unified architecture search framework. Each hidden neuron k in the lthl^th hidden layer is associated with two sets of architecture parameters (a) a neuron gating parameter αk(l)∈ℝα^(l)_k , and (b) an activation architecture parameter vector k(l)∈ℝM β^(l)_k ^M The neuron gating corresponding to neuron k in layer l is defined via a sigmoid mapping as gk(l)=σ(αk(l)),g^(l)_k=σ\! (α^(l)_k ), (23) which controls the neuron’s effective contribution to the network output. The mixing weights corresponding to the candidate activation functions are obtained using a softmax mapping π(l)km=exp(βkm(l))∑m′=1Mexp(βkm′(l)),m=1,…,Mπ^(l)_k_m= (β^(l)_k_m) _m =1^M (β^(l)_k_m ), m=1,…,M (24) For the simple modified MLP shown in Figure 6, which consists of a single hidden layer (l=1l=1) and seven candidate activation functions (M=7M=7), the neuron-gating and activation-mixing weights reduce to gk(1)=σ(αk(1)),g^(1)_k=σ\! (α^(1)_k ), (25) and π(1)km=exp(βkm(1))∑m′=17exp(βkm′(1)),m=1,…,7π^(1)_k_m= (β^(1)_k_m) _m =1^7 (β^(1)_k_m ), m=1,…,7 (26) Figure 6: Neural network incorporating neuron gating mechanism and mixed activation function for MLP architecture search. The activation of the kthk^th hidden neuron for the ithi^th input sample is then defined as hki′(1)=gk(1)∑m=17πkm(1)fm(wk1(1)x1i+wk2(1)x2i+bk(1))h (1)_ki=g^(1)_k _m=1^7π^(1)_k_mf_m\! (w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k ) (27) The output pre-activation, predicted output, and the corresponding loss function follow analogously to the previous cases. The global architecture parameter vector for the joint neuron gating and activation-based architecture search is given by =((1),(1)),A= ( α^(1), β^(1) ), (28) where (1) α^(1) and (1) β^(1) denote the neuron gating and mixed activation architecture parameters of the hidden layer, respectively. MLP Architecture Search Problem Formulation. The NGMA-based MLP architecture search problem is formulated as in (11), with the architecture parameters specified by (28). 4.2 Algorithm Algorithm 1 details the proposed architecture search procedure based on NGMA, while Figure 7 provides a visual illustration. Algorithm 1 NAS-NGMA 1: Initialize architecture parameters A0A^0 and network weights W0W^0, and set iteration counter k←0k← 0 2: while k<kmaxk<k do 3: Approximate the hypergradient using equation (2) gAk=∇Aℒv(Ak,Wk−ξ∇Wℒt(Ak,Wk))g_A^k= _AL_v (A^k,\;W^k-ξ _WL_t(A^k,W^k) ) 4: Update architecture parameters Ak+1←Ak−ηAgAkA^k+1← A^k- _A\,g_A^k 5: Update network weights Wk+1←Wk−ηW∇Wℒt(Ak+1,Wk)W^k+1← W^k- _W\, _WL_t(A^k+1,W^k) 6: Increment iteration counter k←k+1k← k+1 7: end while 8: Derive the final neuron configuration by thresholding the sigmoid-transformed gating components of AkmaxA^k and assigning to each retained neuron the activation function with the largest activation-related component in AkmaxA^k The workflow illustrated in Figure 7 consists of four main stages (a) Problem formulation. A sufficiently large baseline network with predetermined depth and width is constructed. At this stage, the neuron gating states and activation functions are not fixed; instead, they are treated as architecture variables to be learned through the bilevel optimization search process. For illustration, Figure 7 (a) considers a simple network with a single hidden layer containing three neurons whose gating states and activation functions are initially unknown. (b) Continuous relaxation. To enable gradient-based optimization, the discrete architecture decisions are transformed into continuous variables. Specifically, neuron selection is modeled through a gating mechanism, while each neuron is associated with a weighted mixture of candidate activation functions, as shown in Figure 7 (b). (c) Optimization. The network weights, neuron-gating parameters, and activation-function mixing probabilities are optimized jointly within a bilevel optimization framework. During this stage, the gating parameters and mixing probabilities evolve continuously, allowing the search process to identify the relative importance of individual neurons and activation functions. Figure 7 (c) depicts an intermediate optimization state. (d) Architecture derivation. After optimization, the learned continuous variables are converted into a discrete architecture. Neurons are retained or pruned according to the learned gating parameters and the prescribed pruning criterion. For each retained neuron, the activation function corresponding to the largest mixing probability is selected. The resulting architecture, illustrated in Figure 7 (d), represents the final searched model. In this example, one of the three neurons is pruned, while the remaining neurons are assigned their selected activation functions. Figure 7: An overview of NAS-NGMA. (a) The gating states and activation functions of neurons are initially unknown. (b) Continuous relaxation of the search space through neuron gating and a weighted mixture of candidate activation functions for each neuron. (c) Joint optimization of the gating parameters, mixing probabilities, and network weights via a bilevel optimization framework. (d) Derivation of the final neural architecture from the learned gating states and activation-function mixing probabilities. Similar algorithms and corresponding visual representations, along with their interpretations, can be developed for NG- and MA-based NAS; however, these are omitted for brevity. Although the proposed formulations and algorithm are presented in the context of MLP architectures, the relaxed bilevel framework extends naturally to CNNs by associating architectural variables with convolutional channels and the final linear layers, and can further be adapted to search for optimal convolutional filters. Notably, the underlying optimization structure, relaxation strategy, and solution methodology remain unchanged. To avoid redundancy, detailed derivations for CNNs are omitted, as they follow directly from the MLP case with appropriate tensor generalizations. 5 Experimental Results and Discussion This section presents a comprehensive evaluation of the proposed relaxed bilevel formulations on MNIST and CIFAR-10 image classification tasks. We systematically analyze the performance of the proposed architecture search strategies—namely, NAS-NG, NAS-MA, and NAS-NGMA—on both MLP and CNN architectures, and compare them with their corresponding dense baseline models. To provide a broader context, we also compare our results with representative models from the literature on MNIST. Furthermore, the effectiveness of the proposed methods is validated by comparing them with DARTS on the CIFAR-10 dataset. 5.1 Datasets and Baseline Models The MNIST dataset comprises 70,000 grayscale images of handwritten digits (0–9), each with a spatial resolution of 28×2828× 28 pixels, and is partitioned into 60,000 training samples and 10,000 test samples. Following standard preprocessing practices, pixel intensities are normalized to the [0,1][0,1] range. For MLP-based models, each image is flattened into a 784-dimensional input vector. As a primary baseline, we consider a fully connected MLP corresponding to the maximum-capacity architecture induced by the proposed framework, wherein all neurons are active. This configuration serves as a reference model prior to any architectural adaptation through the proposed relaxation mechanisms. To ensure a comprehensive evaluation, multiple dense MLP architectures with varying depths and widths are examined. Specifically, we consider networks with 1 to 4 hidden layers, where each configuration employs a uniform number of neurons per layer. The hidden layer sizes are chosen from 1024,2048,3072\1024,2048,3072\, resulting in a total of 12 baseline configurations. All models use ReLU activation functions in the hidden layers and a softmax output layer for the 10-class classification task. In addition to MLPs, we evaluate the proposed methods on convolutional architectures to assess their effectiveness in learning spatial features. Two CNN baseline architectures are considered. The first architecture comprises three convolutional layers with channel sizes [64, 128, 256] and kernel sizes [5, 3, 3], respectively. The second architecture consists of four convolutional layers with channel sizes [32, 64, 128, 256] and kernel sizes [5, 3, 3, 3]. Both architectures are followed by two fully connected layers with dimensions [256, 128]. Furthermore, we extend our evaluation to the CIFAR-10 dataset, which consists of 50,000 training images and 10,000 test images distributed across 10 classes. In contrast to MNIST, CIFAR-10 images are represented in RGB format with a spatial resolution of 32×3232× 32 pixels. For CIFAR-10, architecture search is performed using CNN models comprising three convolutional layers with channel sizes [128, 256, 512] and corresponding kernel sizes [5, 3, 3]. This feature extraction backbone is followed by a fully connected layer of dimension 512. To benchmark the effectiveness of the proposed method, we conduct architecture search experiments using the DARTS framework under similar experimental settings and compare the resulting architectures with those obtained by our approach. All models are trained using mini-batch gradient-based optimization, with detailed training hyperparameters provided in the subsequent section. 5.2 Experimental Setup We evaluate the proposed methods in a constrained setting with a 2-day wall-time limit. All experiments are conducted on a single node of our High Performance Computing (HPC) cluster, equipped with sufficient memory for the configurations considered. The workflow consists of two phases: (i) architecture search using the relaxed bilevel formulation-based approaches, and (i) retraining the discovered optimal architecture from scratch, followed by final evaluation on the test set. Both MLPs and CNNs are used for MNIST, whereas only CNNs are used for CIFAR-10. A batch size of 64 is considered across all phases of experiments on the MNIST dataset. During the architecture search phase, we employ the Adam optimizer with a learning rate of ηA=3×10−4 _A=3× 10^-4 and a weight decay of 1×10−31× 10^-3. The lower-level optimization problem is approximated using one-step gradient descent to compute the hypergradients. Unless otherwise specified, the architecture search is conducted for 10 epochs; however, for two computationally intensive experiments, the search is limited to 8 epochs. For architecture search, the available training data is evenly split into training and validation subsets. All architecture parameters are initialized from a standard normal distribution scaled by 10−310^-3. After the search phase, the selected architecture is trained from scratch using mini-batch gradient descent. The training process uses an initial learning rate of 0.025 (with a minimum learning rate of 1×10−31× 10^-3), momentum of 0.9, weight decay of 3×10−43× 10^-4, and gradient clipping to ensure numerical stability. All final architectures are trained for 25 epochs before evaluation on the test set. We used fixed random seeds in the experiments. While conducting experiments on CNN architectures for MNIST, we used slightly different hyperparameter settings than those for MLPs. Specifically, the minimum learning rate for network training was set to 1×10−41× 10^-4, and the architecture learning rate was set to 6×10−46× 10^-4. For architecture evaluation, models were trained from scratch for 30 epochs in the case of CNNs with three convolutional layers and for 50 epochs in the case of CNNs with four convolutional layers. Additionally, a dropout rate of 0.15 was used during the architecture search phase, and 0.25 during the final evaluation phase. For CIFAR-10 experiments, the DARTS experimental setup closely follows the protocol established in the original work by [24], with minor changes to the optimization hyperparameters, e.g., batch size of 96 and minimum learning rate of 2.5×10−52.5× 10^-5. The DARTS architecture consists of eight cells: six normal cells and two reduction cells. The reduction cells are positioned at one-third and two-thirds of the network depth to progressively downsample the feature representations. Within the normal cells, all operations are performed with a stride of 1, thereby preserving the spatial resolution. In contrast, the reduction cells utilize a stride of 2 to halve the spatial dimensions of the feature maps. The set of candidate operations, denoted by O, is identical to that defined in [24]. For proposed methods, we considered a three-convolutional-layer model as discussed earlier. 5.3 MLP Model The results of architecture search and evaluation for MLP models on the MNIST dataset are presented as follows. 5.3.1 Architecture Search The architecture search phase aims to identify optimal sub-networks from initially dense MLP configurations using three proposed search strategies: NAS-NG, NAS-MA, and their combination (NAS-NGMA). The NG approach follows a structured pruning strategy in which neuron importance is evaluated based on their contribution to the learned representation, and redundant neurons are progressively removed. The MA-based method analyzes neuron activation magnitudes across the training data and prunes those that consistently exhibit low responses over diverse inputs. The combined approach integrates both gating-based importance measures and activation statistics to enable more stable pruning decisions, balancing model compactness and predictive performance. The architecture search process iteratively refines networks based on validation performance until either the preset number of iterations is reached or the allocated computational budget of two days is exhausted. The final architecture is selected based on the minimum validation loss, while validation accuracy is monitored as a secondary metric. Table 2 summarizes the validation performance during architecture search, along with the corresponding execution time and resulting model size, starting from the baseline model, which has a model size of 8.97±9.298.97± 9.29 million parameters. Notable differences are observed in both search time and model size across the considered architecture search methods. Table 2: Architecture search performance for MLP models on the MNIST dataset. 1–4 Hidden Layers × 1024, 2048, 3072 Neurons/Layer Search Method Val Acc (%) Time (hrs) Params (M) Mean Std Dev Mean Std Dev Mean Std Dev NAS-NG 97.59 0.04 0.45 0.14 1.90 1.33 NAS-MA 97.27 0.12 18.80 12.04 4.64 4.38 NAS-NGMA 96.20 0.31 18.81 11.89 3.54 3.56 The NAS-NG method demonstrates remarkable efficiency, completing the search in an average of 0.45 CPU hours—approximately 40 times faster than the MA-based approaches (NAS-MA and NAS-NGMA). This substantial speed advantage stems from the computational simplicity of gating-based importance metrics, which can be evaluated more rapidly than comprehensive activation pattern analysis. The MA-based methods require extensive forward passes through the network with diverse input data to capture representative activation statistics, resulting in mean search times of approximately 18.8 hours. Further, the standard deviation in search times reflects the varying complexity of different initial configurations. Deeper networks with more neurons naturally require longer search durations, as evidenced by the maximum search times approaching 40 hours for the 4-layer, 3072-dimensional configurations under MA-based search. Figure 8 depicts the search performance of the proposed NAS methods for MLP architectures on the MNIST dataset, along with the layer-wise evolution of active neurons during the course of optimization. Figure 8: Comparison of architecture search methods for MLP models on the MNIST dataset. 5.3.2 Architecture Evaluation Following the search phase, all discovered architectures are retrained from scratch to evaluate their generalization performance on the test set. This evaluation phase constitutes the most critical comparison, as it reflects the practical effectiveness of the searched architectures when trained independently of the search procedure. Table 3 summarizes the model complexity and test accuracy metrics across all methods for the MLP architectures. The findings indicate that the models obtained using the suggested architecture search methods achieve performance very close to the baseline while also attaining substantial parameter reductions. Table 3: Test performance of discovered MLP models on MNIST dataset. Search Method Mean Test Acc. Max Test Acc. Mean Params Param (%) (%) (M) Reduction (%) Baseline 98.56 98.69 8.97 — NAS-NG 98.53 98.60 1.90 78.8 NAS-MA 98.50 98.65 4.64 48.3 NAS-NGMA 98.55 98.68 3.54 60.5 A detailed configuration-wise performance analysis for the MLP architectures is provided in Appendix D, where heatmap visualizations illustrate the performance of the proposed search methods across different network depths and hidden dimensions. 5.4 CNN Model The experiments for CNN models are divided into two categories based on the datasets used, namely: experiments on MNIST and experiments on CIFAR-10, as follows. 5.4.1 MNIST The results of architecture search and evaluation for CNN models on the MNIST dataset are presented as follows. Architecture Search. Figure 9 depicts the search performance of the proposed NAS methods for CNN architectures on the MNIST dataset, along with the layer-wise evolution of active channels in the convolutional layers and active neurons in the fully connected layers over the course of optimization. Figure 9: Comparison of architecture search methods for CNN models on the MNIST dataset. Table 4: Architecture search performance for CNN models on the MNIST dataset. Search Method 3 Layer Conv [64, 128, 256] 4 Layer Conv [32, 64, 128, 256] Val Acc (%) Time (hrs) Params (M) Val Acc (%) Time (hrs) Params (M) NAS-NG 99.31 0.50 0.26 99.24 0.44 0.12 NAS-MA 99.22 5.46 0.45 99.24 4.64 0.24 NAS-NGMA 99.26 5.51 0.25 99.24 4.59 0.14 A summary of the architecture search results for CNN models on the MNIST dataset, starting from the baseline models with model sizes of 1.001.00 and 0.490.49 million parameters, is presented in Table 4. Notably, CNN-based architecture search requires significantly less time than MLP-based search in the case of activation-driven NAS methods. This table also suggests that the NAS-NG method’s rapid pruning decisions, though based on simpler criteria, are remarkably effective at identifying promising architectures, consistent with observations in the case of MLPs. Architecture Evaluation. Table 5 presents the model complexity and test accuracy metrics for the two convolutional layer configurations considered. The findings indicate that the models obtained using the suggested architecture search methods achieve performance that is either higher than or very close to the baseline. They also attain substantial parameter reductions, similar to those observed in the case of MLP models. Table 5: Test performance of discovered CNN models on MNIST dataset. Search Method 3 Layer Conv [64, 128, 256] 4 Layer Conv [32, 64, 128, 256] Test Acc (%) Params ↓ (%) Test Acc (%) Params ↓ (%) Baseline 99.60 - 99.62 - NAS-NG 99.63 73.5 99.50 74.7 NAS-MA 99.49 55.1 99.55 50.5 NAS-NGMA 99.61 75.4 99.56 72.3 5.4.2 CIFAR-10 The results of architecture search and evaluation for CNN models on the CIFAR-10 dataset are presented as follows. In addition to the proposed NAS methods, architecture search experiments are also conducted using the DARTS method for comparison purposes. Architecture Search. Table 6 presents a summary of the architecture search results for CNN models on the CIFAR-10 dataset. The proposed methods start from a baseline model with 5.695.69 million parameters. Figure 10 illustrates the search performance of the proposed NAS methods and the DARTS method for CNN architectures on the CIFAR-10 dataset, along with the layer-wise evolution of active channels in the convolutional layers and active neurons in the fully connected layer. Table 6: Architecture search performance for CNN models on the CIFAR-10 dataset. Search Method Val Acc (%) Time (hrs) Params (M) Mean Std Dev Mean Std Dev Mean Std Dev NAS-NG 77.59 0.31 0.56 0.02 0.49 0.04 NAS-MA 74.16 0.13 10.75 0.39 0.83 0.10 NAS-NGMA 74.88 0.24 10.19 0.08 0.41 0.03 DARTS 81.93 0.31 26.30 0.56 3.40 0.10 Figure 10: Comparison of NAS methods for CNN models on the CIFAR-10 dataset. Architecture Evaluation. Table 7 presents the test accuracy metrics and the reduction in model complexity for a three-layer convolutional configuration with one fully connected layer. The findings indicate that the models obtained using the proposed methods perform better than the architectures discovered by the vanilla DARTS method while discovering architectures with substantially smaller network sizes. Table 7: Test performance of discovered CNN models on CIFAR-10 dataset. Search Method Test Acc (%) Eval. Time (hrs) Params ↓ (%) Mean Std Dev Mean Std Dev Baseline 86.14 0.22 1.48 0.05 – NAS-NG 85.60 0.21 1.09 0.02 91.4 NAS-MA 84.55 0.31 4.61 0.22 85.4 NAS-NGMA 84.73 0.18 3.08 0.08 92.8 DARTS 82.03 0.50 20.20 0.47 – 5.5 Overall Performance Comparison The above findings indicate that the proposed architecture search methods can either match or come very close to the baseline performance while achieving substantial parameter reductions across both types of architectures on the MNIST and CIFAR-10 datasets. On CIFAR-10, their performance outperforms the vanilla DARTS method. Several notable observations emerge from this comparison for MLPs. First, all three search methods achieve mean test accuracies within 0.06 percentage points of the baseline, indicating that the discovered architectures retain the essential representational capacity needed for MNIST classification. The NAS-NG approach achieves the most impressive parameter reduction, reducing the average model size from 8.97M to 1.90M while maintaining 98.53% accuracy. The NG, when combined with a MA-based method, achieves the highest maximum accuracy (98.68%), matching the best baseline performance while using 60.5% fewer parameters. This suggests that careful coordination between multiple pruning criteria can occasionally identify architectures that achieve the full performance of dense networks while operating at a fraction of the computational cost. The MA-based NAS method falls in the middle ground, achieving a 48.3% reduction in parameters. While still substantial, this more conservative pruning reflects the method’s tendency to retain neurons with moderate activation patterns that may contribute to robustness, even if not strictly necessary for high validation accuracy. Similar performance improvement and complexity reduction trends are observed for CNNs. 5.5.1 Accuracy vs Model Complexity Figure 11 presents a scatter plot revealing the fundamental trade-off between model complexity and test accuracy. Figure 11: Test accuracy vs model complexity (measured in millions of parameters) for all evaluated configurations on the MNIST dataset. This visualization clearly demonstrates the advantage of architecture search methods. Baseline models have higher parameter counts, while search methods achieve comparable accuracy with substantially fewer parameters. The best baseline MLP model (98.69% accuracy, 30.76M parameters) and best MLP searched model (98.68% accuracy, 7.69M parameters) are highlighted. The baseline MLP configurations span a wide range of parameter counts from 0.81M to 30.76M, with test accuracies between 98.46% and 98.69%. In contrast, the searched MLP architectures occupy the lower-left region of the plot, achieving similar accuracies with dramatically reduced parameter counts. This clustering pattern indicates that architecture search successfully identifies the minimal architectural complexity needed to solve the task effectively. The best-performing architecture, discovered using the combined gating and mixed activation method, achieves 98.68% accuracy with only 7.69M parameters—a 75% reduction compared to the best baseline configuration (30.76M parameters at 98.69% accuracy). This near-identical performance with a quarter of the parameters represents a substantial improvement in model efficiency. The baseline CNN models generally exhibit smaller parameter sizes than their MLP counterparts while achieving strong test accuracy. However, the CNN architectures obtained through the proposed search methods demonstrate further improvements, yielding significantly reduced model complexity and, in some cases, outperforming even the best-performing baseline models, as shown in Table 5. 5.5.2 Additional Analysis A detailed analysis of the effects of network depth, hidden dimensions, and parameter reduction achieved by the proposed search methods is provided in Appendix E. In particular, the appendix examines the relationship between architectural capacity and classification performance across different MLP configurations, along with the compression characteristics of the proposed NAS strategies. The results indicate that the search methods maintain competitive accuracy across varying depths and hidden dimensions while significantly reducing model complexity. Additional quantitative comparisons and visualizations are also provided. 5.6 Comparative Analysis This section compares the models discovered by the NAS methods. We also include selected results from the literature on the MNIST dataset. Notably, no clear positive correlation is observed between search duration and final test accuracy. For instance, the NAS-NG method achieves search times of under one hour while attaining accuracies spanning the full observed range (98.47% to 98.60%) on the MNIST dataset. In contrast, the MA-based methods, despite requiring 20–40 hours of search, achieve accuracies that are comparable to, and in some cases lower than, those obtained by simpler methods. This observation has important practical implications. It suggests that computationally intensive and sophisticated search algorithms are not always necessary to discover effective architectures, particularly for datasets of moderate complexity such as MNIST. 5.6.1 Best Performing Architectures Table 8 presents the optimal MLP configuration discovered by each method, along with the best baseline for reference. Table 8: MLP best performing models identified by each search method on MNIST dataset. Method Layers Config. Test Acc. Params Search (%) (M) Time (hrs) Baseline 4 [3072, 3072, 3072, 3072] 98.69 30.76 — NAS-NG 4 [30724] → Pruned 98.60 4.29 0.76 NAS-MA 4 [10244] → Pruned 98.65 1.75 16.16 NAS-NGMA 3 [30723] → Pruned 98.68 7.69 35.80 The NGMA-based NAS method achieves the highest accuracy among search methods at 98.68%, falling just 0.01 percentage points short of the best baseline while using 75% fewer parameters. Interestingly, this top configuration emerges from a 3-layer rather than a 4-layer network, supporting the earlier observation that interactions among pruning criteria may favor intermediate depths. The NAS-MA method produces the most compact high-performing model, achieving 98.65% with only 1.75M parameters—a remarkable 94.3% reduction compared to the best baseline. This configuration began with the smallest dense architecture (4 layers × 1024 neurons) and optimized it via activation analysis, demonstrating that starting with smaller initial networks can sometimes yield excellent results when combined with effective search. The NAS-NG method’s best configuration achieves 98.60% with 4.29M parameters (an 86% reduction), demonstrating its ability to dramatically compress large networks while retaining strong performance. Table 9: CNN best performing models identified by each search method on MNIST dataset. Method Layers Config. Test Acc. Params Search (%) (M) Time (hrs) Baseline 4 [32, 64, 128, 256] 99.62 0.49 — NAS-NG 3 [64, 128, 256] → Pruned 99.63 0.26 0.50 NAS-MA 4 [32, 64, 128, 256] → Pruned 99.55 0.24 4.64 NAS-NGMA 3 [64, 128, 256] → Pruned 99.61 0.25 5.51 In the case of CNNs on the MNIST dataset, the best-performing model is obtained using the NAS-NG method with a base architecture consisting of three convolutional layers. This model outperforms the best CNN baseline by 0.01% while being approximately 47% less complex in terms of model size. The best-performing CNN models on the MNIST dataset are presented in Table 9. For the CIFAR-10 dataset, the best obtained models are provided in Table 10. Table 10: CNN best performing models identified by each search method on CIFAR-10 dataset. Method Layers/Cells Config. Test Acc. Params Search (%) (M) Time (hrs) Baseline 3 [128, 256, 512] 86.34 5.69 — NAS-NG 3 [128, 256, 512] → Pruned 85.77 0.51 0.53 NAS-MA 3 [128, 256, 512] → Pruned 84.83 0.79 10.64 NAS-NGMA 3 [128, 256, 512] → Pruned 84.85 0.38 10.15 DARTS 20 Stacked Normal–Reduction Cells 82.57 3.38 25.88 5.6.2 Efficiency Metrics To provide a unified measure of architecture quality, we computed an efficiency metric defined as test accuracy percentage divided by parameter count in millions. Figure 12 presents this comparison. Higher values indicate better accuracy-to-parameter ratios. Error bars represent standard deviation across all configurations. NAS-NG achieves the highest efficiency, delivering 86.05 accuracy points per million parameters. Figure 12: Mean efficiency (test accuracy per million parameters) achieved by each search method for MLP models on the MNIST dataset. The NAS-NG method achieves the highest mean efficiency at 86.05 accuracy points per million parameters—2.7 times higher than the baseline (31.94) and 1.8 times higher than the MA-based NAS method (48.75). This metric quantifies the intuition visible in earlier plots: NAS-NG discovers architectures that achieve near-baseline accuracy with dramatically reduced complexity. The NAS-NGMA method achieves an efficiency of 60.66, representing a middle ground between the aggressive compression of NAS-NG and the more conservative approach of MA. The substantial standard deviations (particularly for NAS-NG at 62.80) reflect the diversity of configurations tested; some initial networks offer more redundancy to exploit than others. The five most efficient individual models on MNIST and CIFAR-10 datasets are presented in Table 11. Remarkably, the most efficient model is discovered by the NAS-NG method. These models require significantly less memory and computational resources. Table 11: Top five most efficient models (highest accuracy-to-parameter ratios) on MNIST and CIFAR-10 datasets. Rank MNIST CIFAR-10 Method Config. Efficiency Method Config. Efficiency 1 NAS-NG [32, 64, 128, 256] 803.97 NAS-NGMA [128, 256, 512] 225.65 2 NAS-NG [32, 64, 128, 256] 803.89 NAS-NGMA [128, 256, 512] 199.23 3 NAS-NGMA [32, 64, 128, 256] 732.76 NAS-NG [128, 256, 512] 195.28 4 NAS-NGMA [32, 64, 128, 256] 732.32 NAS-NGMA [128, 256, 512] 193.98 5 NAS-MA [32, 64, 128, 256] 410.58 NAS-NG [128, 256, 512] 166.88 5.6.3 Comparative Evaluation on the MNIST Benchmark We present a comparative evaluation of the proposed architecture search strategies against established models reported in the literature on the MNIST benchmark dataset. The results in Table 12 demonstrate that the searched architectures achieve competitive performance relative to several widely adopted deep learning and classical approaches, while maintaining simpler model structures, as discussed earlier. Table 12: Comparison of MNIST classification accuracies from various models. Model Test Acc. (%) Model Test Acc. (%) CNN [18] 99.45 LSTM [18] 99.22 LeNet-5 [20] 99.05 Consensus Clustering [32] 95 LR-based [17] 93.83 SVM [30] 97.83 Spiking-CNN [16] 96.89 Multi-Column DNN [8] 99.77 PCA/LDA [29] 86.60 EfficientDet-D4 [2] 99.83 CNN-SVM [39] 99.30 RMDL [19] 99.82 DropConnect Regularization [40] 99.79 APAC [33] 99.77 Baseline (MLP) 98.69 Baseline (CNN) 99.62 NAS-NG (MLP) 98.60 NAS-NG (CNN) 99.63 NAS-MA (MLP) 98.65 NAS-MA (CNN) 99.55 NAS-NGMA (MLP) 98.68 NAS-NGMA (CNN) 99.61 5.7 Additional Experiments for NAS-NG To further obtain better models, we perform two additional experiments. First, we investigate whether NAS-NG remains effective when initialized from substantially over-parameterized architectures. Second, we examine whether the proposed NG strategy can further improve architectures that have already been optimized using state-of-the-art NAS methods. 5.7.1 Larger Baselines To assess the scalability of NAS-NG, we construct significantly larger MLP and CNN baseline architectures. NAS-NG is then applied to each baseline to identify compact architectures. As shown in Table 13, NAS-NG consistently reduces the number of trainable parameters while maintaining, and in several cases improving, the final test accuracy. The results obtained from these larger baselines are superior to those reported earlier, suggesting that starting from larger architectures can lead to better final models. This is likely because NAS-NG has greater flexibility to prune redundant neurons without excessively compressing the network, thereby discovering architectures that are both more accurate and substantially smaller than their corresponding baselines. Table 13: Evaluation experiments for NAS-NG with larger baseline architectures. Model Architecture Baseline NAS-NG Test Acc. (%) Params (M) Test Acc. (%) Params (M) MLP [3072, 3072, 3072, 3072, 3072, 3072] 98.68 49.64 98.65 4.80 [4096, 4096, 4096, 4096, 4096, 4096] 98.64 87.16 98.67 7.38 [6144, 6144, 6144, 6144, 6144, 6144] 98.64 193.66 98.73 12.97 CNN Backbone: [128, 256, 512, 512, 1024] MLP Head: [4096, 2048] 88.45 84.11 89.38 19.37 MLP Head: [2048, 1024] 88.66 44.24 89.67 10.39 MLP Head: [1024, 512] 88.86 25.88 89.22 6.13 5.7.2 Better Initialization Previous experiments compared NAS-NG against architectures obtained from different initializations. Here, we investigate whether NG can provide further improvements when starting from architectures that have already been optimized by established NAS methods. Specifically, we apply NAS-NG to literature-optimal DARTS architectures while allowing only neuron-level optimization. During initialization, the optimal cell structures are kept unchanged, and only the number of stacked cells is varied to construct complete network architectures (10-cell and 14-cell macro-architectures). As shown in Figure 13, NAS-NG simultaneously improves classification accuracy and substantially reduces the number of trainable parameters. These results demonstrate that neuron-level optimization is complementary to topology search and can further refine architectures discovered by existing NAS methods. Figure 13: Applying NG to literature-optimal DARTS cells simultaneously improves classification accuracy and reduces the number of trainable parameters, demonstrating that neuron-level optimization complements topology search. 5.8 Discussion, Implications, and Recommendations The experimental findings highlight the effectiveness of the proposed NAS techniques in designing MLP and CNN architectures for MNIST classification, as well as CNN architectures for the CIFAR-10 classification task. All three approaches—NAS-NG, NAS-MA, and NAS-NGMA—identified architectures that maintained competitive accuracy (98.40%–98.68%) while achieving parameter reductions of 40%–69% for MLPs. For CNNs, the searched architectures achieved test accuracies of 99.49%–99.63% while attaining parameter reductions of 50.5%–75.4% on MNIST dataset. These findings extend NAS literature on network redundancy to fully connected architectures, showing that even simple search heuristics can remove excess capacity without performance loss. For MLPs, among the methods, NAS-NG provides the best practical trade-off, delivering up to 86% compression in under one hour with minimal hyperparameter tuning. This makes it highly appropriate for environments with limited resources, such as edge deployment. The MA-based method is more conservative (retaining ∼ 60% parameters), potentially improving robustness to distribution shifts through moderate activation patterns; however, its 40× longer search time reduces cost-effectiveness. The combined approach achieves peak performance, matching baseline accuracy with 75% fewer parameters through multi-criteria pruning. Nevertheless, its inconsistent average gains and longer search times limit broad applicability. Notably, no strong correlation is observed between network size (0.45M–30.76M parameters) and accuracy, suggesting that MNIST’s low intrinsic complexity allows many architectures to achieve near-optimal performance once a minimum capacity threshold is exceeded. Overall, these results question the necessity of computationally intensive search procedures. The sub-hour runtime of NAS-NG enables rapid iteration without extensive HPC resources, thereby democratizing NAS. Although MNIST’s simplicity limits overgeneralization, the findings underscore the importance of architecture search in identifying efficient operating points under diverse deployment constraints. The main implications and practical recommendations are summarized below. 5.8.1 Implications for Neural Network Design These findings contribute to the discussion on designing neural networks. The reduction of parameters by 86% (MLP) and 75.4% (CNN) without losing accuracy shows that many networks have too many parameters. This aligns with the notion that more straightforward networks can perform equally well, as demonstrated in other research on network pruning and the lottery ticket hypothesis [12]. The fact that simple search methods can perform as well as complex ones suggests that finding the best network design does not always require complicated algorithms. Although methods such as NAS via RL or DARTS are popular, our results show that simpler, problem-specific methods can be equally effective and require less computing power. Lastly, the small accuracy ranges (98.40% to 98.69%) and (99.49% to 99.63%) across different setups suggests that for many uses, factors such as efficiency and ease of use might be more important than small differences in accuracy. This means examining the overall quality of a model, not just its accuracy. 5.8.2 Practical Recommendations Based on our experiments, we recommend the following (a) Start with neuron gating. For most tasks, especially those with limited computing power or tight deadlines, NAS-NG is a good first choice. It is quick and effectively reduces the parameters. (b) Consider task complexity. For tasks more complex than MNIST, methods that prune more carefully might be better. If you have the resources, try different methods to determine which works best for your problem. (c) Optimize initial configurations. Our results show that search methods work with different starting setups, but starting with medium-sized networks (e.g., 1024-2048 neurons per layer, 2-3 layers) is often best. Very large networks require longer search times and do not always yield better results. (d) Validate on representative data. The accuracy during the search does not always predict the final test accuracy. The brief duration of the search phase does not adequately showcase the model’s ability to generalize. Therefore, it is crucial to evaluate the final model using distinct datasets. (e) Leverage efficiency metrics. When comparing models, consider the accuracy-to-parameter ratio, not just the accuracy. A model with 98.5% accuracy and 0.5M parameters is often better than one with 98.6% accuracy and 5M parameters, particularly for devices with limited resources. 5.9 Summary This section presents a comprehensive evaluation of three relaxed bilevel formulations applied to MNIST classification using MLPs and CNNs, as well as CIFAR-10 classification using CNNs. The key findings are summarized as follows. (a) All three search methods (NAS-NG, NAS-MA, and NAS-NGMA) successfully identified architectures that maintained competitive accuracy while reducing the number of parameters in MLPs by 40%–69% relative to the baseline dense networks, and in CNNs by 50.5%–75.4% relative to their respective baselines for MNIST dataset. (b) NAS-NG demonstrated the best practical efficiency, achieving an average parameter reduction of 68.7% in under one hour of search time while maintaining a test accuracy of 98.53%. For CNNs, the NG-based model outperformed its baseline by 0.01% while being approximately 47% less complex in terms of model size. (c) For MLPs, the NAS-NGMA method achieved the highest accuracy (98.68%), approaching the best baseline performance while using 75% fewer parameters, albeit at the cost of substantially longer search times. For CNNs, NAS-NG identified the best-performing model, achieving a test accuracy of 99.63%. (d) No strong correlation exists between search time and final accuracy, suggesting that computationally expensive search procedures do not consistently outperform simple heuristics for this task. (e) For MLPs, the most efficient discovered architecture achieved 98.49% accuracy with only 0.45M parameters, corresponding to an efficiency of 218.88 accuracy points per million parameters—approximately 68 times higher than the best baseline. For CNNs, NAS-NGMA emerged as the most efficient approach, achieving approximately 4.06 times the efficiency of the corresponding baseline. (f) Network depth and width interact with search procedures in complex ways, with intermediate depths (3 layers) sometimes outperforming maximal depth, suggesting that search methods can compensate for suboptimal architectural choices. (g) For CIFAR-10, all architectures discovered by the proposed methods outperformed vanilla DARTS, while having significantly smaller model complexity as shown in Table 7. (h) Additional experiments demonstrate that NAS-NG scales effectively to substantially larger architectures, identifying models with higher test accuracy while dramatically reducing parameter counts, and can further improve literature-optimal DARTS architectures through neuron-level optimization, indicating that the proposed strategy complements existing topology search methods. These findings demonstrate that NAS provides a practical pathway for developing more efficient models without sacrificing performance. The substantial parameter reductions achieved suggest significant opportunities for deploying neural networks in resource-constrained environments while maintaining a strong classification accuracy. 6 Conclusions This work introduced three relaxed bilevel formulations for gradient-based architecture search—NAS-NG, NAS-MA, and NAS-NGMA—that enable differentiable optimization over neuron- and activation-level design choices in MLPs and CNNs. Across MNIST and CIFAR-10, the proposed methods consistently identify architectures that match or exceed literature baselines while using substantially fewer parameters, confirming that continuous relaxations offer an effective and scalable alternative to discrete combinatorial search. The magnitude of compression achieved, even in simple fully-connected networks, further reveals a level of architectural redundancy that is larger than commonly assumed. Among the three variants, NAS-NG is the recommended starting point for practical use: its sub-hour search times and aggressive compression make it particularly well suited to resource-constrained deployment settings, though discovered architectures should always be validated on held-out data, with evaluation prioritizing accuracy-to-parameter trade-offs over marginal accuracy gains. Additional experiments show that NAS-NG remains effective when initialized from substantially over-parameterized networks and can further refine architectures already optimized by DARTS, indicating that neuron-level optimization is complementary to, rather than a substitute for, topology search. This study has certain limitations that should be acknowledged. First, results are shaped in part by the simplicity of MNIST, where most architectures reach near-ceiling accuracy (98.40–98.69% for MLPs), which may obscure differences that would be more pronounced on harder datasets; while CIFAR-10 experiments partially address this, evaluation on larger-scale benchmarks such as ImageNet remains an important next step. Second, the study focuses on depth and width variation in MLPs, leaving interactions with components such as dropout, batch normalization, and skip connections unexplored. Third, hyperparameters were held fixed across methods rather than independently tuned, limiting the precision of cross-method comparisons. Finally, models were retrained from scratch, so potential gains from weight inheritance or knowledge distillation from search-phase models were not examined. Future work can extend these formulations along several directions: applying them to more challenging vision tasks and a broader range of architectural families (including RNNs and Transformers); incorporating adaptive hyperparameter tuning to improve robustness; and developing a deeper theoretical understanding of why simple gating metrics are effective, which could inform improved heuristics and clarify the problem classes for which such relaxations are best suited. Combining the proposed methods with data augmentation strategies (e.g., AutoAugment) and stronger regularization and optimization techniques may also help close the remaining gap to state-of-the-art accuracy without sacrificing efficiency. Overall, these findings challenge the assumption that heavy overparameterization is necessary for strong neural network performance, echoing lottery-ticket-style observations and suggesting that targeted, task-specific architectural adjustments can rival more elaborate NAS pipelines. We advocate for a holistic evaluation framework that weighs accuracy jointly with model size and deployment constraints, favoring efficient architectures that deliver practical value over designs optimized solely for marginal accuracy gains. Acknowledgments The authors gratefully acknowledge the High Performance Computing (HPC) facility at the Indian Institute of Technology (IIT) Kanpur (PARAM Sanganak) for providing the resources that enabled the smooth execution of the experiments in this work. References [1] S. Ahlawat, A. Choudhary, A. Nayyar, S. Singh, and B. Yoon (2020) Improved handwritten digit recognition using convolutional neural networks (cnn). Sensors 20 (12), p. 3344. Cited by: §2.1. [2] S. S. Ahmed, Z. Mehmood, I. A. Awan, and R. M. Yousaf (2023) A novel technique for handwritten digit recognition using deep learning. Journal of Sensors 2023 (1), p. 2753941. Cited by: §2.1, Table 12. [3] R. B. Arif, M. A. B. Siddique, M. M. R. Khan, and M. R. Oishe (2018) Study and observation of the variations of accuracies for handwritten digits recognition with various hidden layers and epochs using convolutional neural network. In 2018 4th international conference on electrical engineering and Information & Communication Technology (iCEEiCT), p. 112–117. Cited by: §2.1. [4] D. Ben Noureddine (2025) Handwritten digit recognition: comparative analysis of ml, cnn, vision transformer, and hybrid models on the mnist dataset. Journal of Intelligent Systems 34 (1), p. 20240411. Cited by: §2.1. [5] Z. Cai, L. Chen, T. Ling, and H. Liu (2024) STO-DARTS: Stochastic bilevel optimization for differentiable neural architecture search. IEEE Transactions on Emerging Topics in Computational Intelligence. Cited by: §1, §2.2. [6] X. Chen, L. Xie, J. Wu, and Q. Tian (2019) Progressive differentiable architecture search: bridging the depth gap between search and evaluation. arXiv preprint arXiv:1904.12760. Cited by: §1, §2.2. [7] X. Chu, T. Zhou, B. Zhang, and J. Li (2020) Fair DARTS: Eliminating unfair advantages in differentiable architecture search. In Computer Vision – ECCV 2020, Cham, p. 465–480. External Links: ISBN 978-3-030-58555-6 Cited by: §1, §2.2. [8] D. Ciregan, U. Meier, and J. Schmidhuber (2012) Multi-column deep neural networks for image classification. In 2012 IEEE conference on computer vision and pattern recognition, p. 3642–3649. Cited by: §2.1, Table 12. [9] A. I. Cowen-Rivers, W. Lyu, R. Tutunov, Z. Wang, A. Grosnit, R. R. Griffiths, A. M. Maraval, H. Jianye, J. Wang, J. Peters, et al. (2022) Hebo: pushing the limits of sample-efficient hyper-parameter optimisation. Journal of Artificial Intelligence Research 74, p. 1269–1349. Cited by: §1. [10] T. Elsken, J. H. Metzen, and F. Hutter (2019) Neural architecture search: a survey. Journal of Machine Learning Research 20 (55), p. 1–21. Cited by: §1. [11] X. Fang, W. Xie, H. Li, W. Zhou, C. Hang, and X. Gao (2025) DARTS-EAST: an edge-adaptive selection with topology first differentiable architecture selection method: x. fang et al.. Applied Intelligence 55 (7), p. 526. Cited by: §2.2. [12] J. Frankle and M. Carbin (2018) The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. Cited by: §5.8.1. [13] T. Giovannelli, G. D. Kent, and L. N. Vicente (2021) Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems. arXiv preprint arXiv:2110.00604. Cited by: §1, §3. [14] B. Guo, S. He, M. Shi, K. Yu, J. Chen, and X. Shen (2024) Semantic-DARTS: Elevating semantic learning for mobile differentiable architecture search. IEEE Internet of Things Journal. Cited by: §1, §2.2. [15] K. Han, S. Hong, J. H. Cheon, and D. Park (2019) Logistic regression on homomorphic encrypted data at scale. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, p. 9466–9471. Cited by: §2.1. [16] M. Kalbande and P. Bhavsar (2022) Performance comparison of deep spiking cnn with artificial deep cnn for image classification tasks. In 2022 IEEE Region 10 Symposium (TENSYMP), p. 1–6. Cited by: §2.1, Table 12. [17] K. Kang, F. Gao, and J. Feng (2018) A new multi-layer classification method based on logistic regression. In 2018 13th international conference on computer science & education (iccse), p. 1–4. Cited by: §2.1, Table 12. [18] O. Kaziha and T. Bonny (2019) A comparison of quantized convolutional and lstm recurrent neural network models using mnist. In 2019 International conference on electrical and computing technologies and applications (ICECTA), p. 1–5. Cited by: Table 12, Table 12. [19] K. Kowsari, M. Heidarysafa, D. E. Brown, K. J. Meimandi, and L. E. Barnes (2018) Rmdl: random multimodel deep learning for classification. In Proceedings of the 2nd international conference on information system and data mining, p. 19–28. Cited by: §2.1, Table 12. [20] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner (2002) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), p. 2278–2324. Cited by: §2.1, Table 12. [21] J. Li, Y. Wang, Y. Wang, R. Yu, and X. Wang (2024) HN-DARTS: Hybrid network differentiable architecture search for industrial scenarios. In Pacific Rim International Conference on Artificial Intelligence, p. 322–327. Cited by: §2.2. [22] Z. Li, Y. Xu, P. Ying, H. Chen, R. Sun, and X. Xu (2024) LMD-DARTS: Low-memory, densely connected, differentiable architecture search. Electronics 13 (14), p. 2743. Cited by: §1, §2.2. [23] H. Liang, S. Zhang, J. Sun, X. He, W. Huang, K. Zhuang, and Z. Li (2019) DARTS+: Improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035. Cited by: §2.2. [24] H. Liu, K. Simonyan, and Y. Yang (2019) DARTS: differentiable architecture search. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.2, §3, §5.2. [25] J. Mun, S. Ha, and J. Lee (2023) DE-DARTS: Neural architecture search with dynamic exploration. ICT Express 9 (3), p. 379–384. External Links: ISSN 2405-9595, Document Cited by: §2.2. [26] K. Nakai, T. Matsubara, and K. Uehara (2020) Att-DARTS: Differentiable neural architecture search for attention. In 2020 International Joint Conference on Neural Networks (IJCNN), Vol. , p. 1–8. External Links: Document Cited by: §1, §2.2. [27] K. V. P. Nandan, M. Panda, and S. Veni (2020) Handwritten digit recognition using ensemble learning. In 2020 5th International conference on communication and electronics systems (ICCES), p. 1008–1013. Cited by: §2.1. [28] J. Parker-Holder, R. Rajan, X. Song, A. Biedenkapp, Y. Miao, T. Eimer, B. Zhang, V. Nguyen, R. Calandra, A. Faust, et al. (2022) Automated reinforcement learning (autorl): a survey and open problems. Journal of Artificial Intelligence Research 74, p. 517–568. Cited by: §1. [29] M. Patel and R. Sheikh (2019) Handwritten digit recognition using different dimensionality reduction techniques. International Journal of Recent Technology and Engineering 8 (2), p. 999–1002. Cited by: §2.1, Table 12. [30] B. Reddy, R. Reddy, P. Vasem, P. Venkatesh, and S. Rajashekaran (2022) Handwritten digit recognition using svm algorithm in machine learning. Int J Creative Res Thoughts 10 (6). Cited by: Table 12. [31] P. Ren, Y. Xiao, X. Chang, P. Huang, Z. Li, X. Chen, and X. Wang (2021) A comprehensive survey of neural architecture search: challenges and solutions. ACM Computing Surveys (CSUR) 54 (4), p. 1–34. Cited by: §1. [32] M. Rexy and K. Lavanya (2019) Handwritten digit recognition of mnist data using consensus clustering. International J. of Recent Technology and Engineering 7 (6), p. 1969–1973. Cited by: Table 12. [33] I. Sato, H. Nishimura, and K. Yokoi (2015) Apac: augmented pattern classification with neural networks. arXiv preprint arXiv:1505.03229. Cited by: §2.1, Table 12. [34] A. Shukla, A. Khanna, A. Sinha, and F. Hamid (2026) LiFT: local search via linear programming for overfitting-controlled transformers. arXiv preprint arXiv:2606.16243. Cited by: §1. [35] A. Shukla, A. Sinha, and F. Hamid (2026) Bilevel optimization for neural architecture search. arXiv preprint arXiv:2606.29582. Cited by: §1. [36] M. Soomro, M. A. Farooq, and R. H. Raza (2017) Performance evaluation of advanced deep learning architectures for offline handwritten character recognition. In 2017 International Conference on Frontiers of Information Technology (FIT), p. 362–367. Cited by: §2.1. [37] K. O. Stanley and R. Miikkulainen (2004) Competitive coevolution through evolutionary complexification. Journal of artificial intelligence research 21, p. 63–100. Cited by: §1. [38] S. Tuli, B. Dedhia, S. Tuli, and N. K. Jha (2023) FlexiBERT: are current transformer architectures too homogeneous and rigid?. Journal of Artificial Intelligence Research 77, p. 39–70. Cited by: §1. [39] S. S. Ullah, L. Gang, M. Riaz, A. Ashfaq, S. Khan, and S. Khan (2025) Handwritten digit recognition: an ensemble-based approach for superior performance. arXiv preprint arXiv:2503.06104. Cited by: §2.1, Table 12. [40] L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus (2013) Regularization of neural networks using dropconnect. In International conference on machine learning, p. 1058–1066. Cited by: §2.1, Table 12. [41] Y. Wang, R. Wang, D. Li, D. Adu-Gyamfi, K. Tian, and Y. Zhu (2019) Improved handwritten digit recognition using quantum k-nearest neighbor algorithm. International Journal of Theoretical Physics 58 (7), p. 2331–2340. Cited by: §2.1. [42] Y. Xu, L. Xie, X. Zhang, X. Chen, G. Qi, Q. Tian, and H. Xiong (2019) PC-DARTS: partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737. Cited by: §1, §2.2. [43] L. Yang, Z. Zheng, Y. Han, S. Song, G. Huang, and F. Li (2024) OStr-DARTS: Differentiable neural architecture search based on operation strength. IEEE Transactions on Cybernetics. Cited by: §1, §2.2. [44] A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter (2019) Understanding and robustifying differentiable architecture search. arXiv preprint arXiv:1909.09656. Cited by: §2.2. [45] H. Zhang, K. Hao, L. Gao, X. Tang, and B. Wei (2023) Enhanced gradient for differentiable architecture search. IEEE Transactions on Neural Networks and Learning Systems. Cited by: §1, §2.2. [46] H. Zhu, X. Jin, H. Liao, Y. Xiang, M. A. El-Yacoubi, and H. Qin (2024) Relax DARTS: Relaxing the constraints of differentiable architecture search for eye movement recognition. arXiv preprint arXiv:2409.11652. Cited by: §1, §2.2. [47] M. Zöller and M. F. Huber (2021) Benchmark and survey of automated machine learning frameworks. Journal of artificial intelligence research 70, p. 409–472. Cited by: §1. [48] B. Zoph and Q. V. Le (2017) Neural architecture search with reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §1. Appendix Appendix A Loss Function The pre-activation vector of the hidden layer is given by i(1)=(1)i+(1),z_i^(1)=W^(1)x_i+b^(1), (29) and the corresponding hidden-layer activations are, i(1)=ReLU(i(1))h_i^(1)=ReLU\! (z_i^(1) ). More specifically, hki(1)=max(0,wk1(1)x1i+wk2(1)x2i+bk(1)),k∈1=1,2,3h^(1)_ki= \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k ), k _1=\1,2,3\ (30) The pre-activation of the output neuron is zi(2)=∑k=13wk(2)hki(1)+b(2),z_i^(2)= _k=1^3w^(2)_kh^(1)_ki+b^(2), (31) and the predicted output is y^i=σ(zi(2))=11+exp(−zi(2)) y_i=σ\! (z_i^(2) )= 11+ \! (-z_i^(2) ) (32) Substituting the expressions for zi(2)z_i^(2) and i(1)h_i^(1), the predicted output can be written as the following single expression y^i=11+exp(−[∑k=13wk(2)max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)]) y_i= 11+ \! (- [ _k=1^3w^(2)_k\, \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) (33) Binary Cross-Entropy Loss. Given the true label yi∈0,1y_i∈\0,1\ and the predicted probability y^i∈(0,1) y_i∈(0,1), the loss for the ithi^th sample is ℒi=−[yilog(y^i)+(1−yi)log(1−y^i)]L_i=- [y_i ( y_i)+(1-y_i) (1- y_i) ] (34) Substituting the explicit expression of y^i y_i, the loss can be written as ℒi=−[yilog(11+exp(−zi(2)))+(1−yi)log(1−11+exp(−zi(2)))]L_i=- [y_i \! ( 11+ \! (-z_i^(2) ) )+(1-y_i) \! (1- 11+ \! (-z_i^(2) ) ) ] (35) Let the collection of all network weights and biases be denoted by =(1),(1),(2),b(2)∈ℝ13W=\W^(1),b^(1),W^(2),b^(2)\ ^13. We can rewrite the loss as follows ℒi(;i,yi)=−[ _i(W;x_i,y_i)=- [ yilog(11+exp(−[∑k=13wk(2)max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)])) y_i \! ( 11+ \! (- [ _k=1^3w^(2)_k \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) (36) +(1−yi)log(1−11+exp(−[∑k=13wk(2)max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)]))] +(1-y_i) \! (1- 11+ \! (- [ _k=1^3w^(2)_k \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) ] Appendix B Modified Loss Function for Architecture Search The sigmoid-based gating is applied after the ReLU activation, and the hidden-layer activations become hki′(1)=σ(αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1)),k=1,2,3h (1)_ki=σ\! (α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k ), k=1,2,3 (37) Accordingly, the output pre-activation is zi′(2)=∑k=13wk(2)σ(αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)z_i (2)= _k=1^3w^(2)_kσ\! (α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) (38) Predicted output becomes y^i′=σ(zi′(2))=11+exp(−zi′(2)) y_i =σ\! (z_i (2) )= 11+ \! (-z_i (2) ) (39) Given the ground-truth label yi∈0,1y_i∈\0,1\, the modified binary cross-entropy loss for the ithi^th sample is defined as ℒi(,,i,yi)=−[yilog(y^i′)+(1−yi)log(1−y^i′)]L_i(A,W;x_i,y_i)=- [y_i ( y_i )+(1-y_i) (1- y_i ) ] (40) ℒi(,;i,yi)=−[ _i(A,W;x_i,y_i)=- [ yilog(11+exp(−[∑k=13wk(2)σ(αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)])) y_i \! ( 11+ \! (- [ _k=1^3w^(2)_kσ\! (α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) (41) +(1−yi)log(1−11+exp(−[∑k=13wk(2)σ(αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)]))] +(1-y_i) \! (1- 11+ \! (- [ _k=1^3w^(2)_kσ\! (α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) ] ℒi(,;i,yi)=−[ _i(A,W;x_i,y_i)=- [ yilog(11+exp(−[∑k=13wk(2)11+exp(−αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)])) y_i \! ( 11+ \! (- [ _k=1^3w^(2)_k 11+ \! (-α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) (42) +(1−yi)log(1−11+exp(−[∑k=13wk(2)11+exp(−αk(1))max(0,wk1(1)x1i+wk2(1)x2i+bk(1))+b(2)]))] +(1-y_i) \! (1- 11+ \! (- [ _k=1^3w^(2)_k 11+ \! (-α^(1)_k ) \! (0,\,w^(1)_k1x_1i+w^(1)_k2x_2i+b^(1)_k )+b^(2) ] ) ) ] Appendix C Activation Functions The activation functions used in this work are summarized in Table 14. Table 14: Candidate activation functions are used for neuron activation-based architecture search. Activation Function Definition and Mathematical Formula None (Inactive Neuron) f1(z)=0 f_1(z)=0 Produces a zero output for any input, effectively deactivating the neuron to enable soft pruning within the activation search framework Identity f2(z)=z f_2(z)=z A linear activation that preserves the pre-activation value, allowing the network to retain linear transformations when nonlinearity is unnecessary Rectified Linear Unit (ReLU) f3(z)=max(0,z) f_3(z)= (0,z) Introduces nonlinearity without saturation for positive inputs, enabling effective gradient propagation in deep networks Leaky Rectified Linear Unit (Leaky ReLU) f4(z)=max(0.01z,z) f_4(z)= (0.01z,z) A ReLU variant that retains a small gradient for negative inputs to mitigate the dead-neuron problem Hyperbolic Tangent f5(z)=tanh(z)=ez−e−zez+e−z f_5(z)= (z)= e^z-e^-ze^z+e^-z A smooth, zero-centered nonlinear activation mapping inputs to (−1,1)(-1,1) to enhance optimization stability in shallow and medium-depth networks Sigmoid f6(z)=σ(z)=11+exp(−z) f_6(z)=σ(z)= 11+ (-z) A bounded, smooth activation mapping inputs to (0,1)(0,1), widely used for probability estimation and gating mechanisms Sigmoid Linear Unit (SiLU) f7(z)=zσ(z)=z(11+exp(−z)) f_7(z)=z\,σ(z)=z\ ( 11+ (-z) ) A smooth, non-monotonic activation combining linear and nonlinear behavior to enhance representational capacity in modern networks Appendix D Configuration-Specific Performance Patterns Figure 14 presents heatmaps showing test accuracy across the grid of layer counts and total hidden dimensions for each method. Each cell represents the mean accuracy for a specific combination of layer count and total hidden dimensions. Warmer colors indicate higher accuracy. All methods achieve similar accuracy patterns, but searched architectures deliver this performance with fewer parameters. Figure 14: Test accuracy heatmaps showing performance on MNIST dataset across different network configurations. The heatmaps reveal several interesting patterns. First, all methods show similar overall accuracy distributions, with the highest accuracies generally appearing in deeper networks with larger hidden dimensions (the bottom-right regions of each heatmap). This consistency validates that the search methods are not fundamentally altering the architecture-performance landscape but rather identifying efficient points within it. Second, the search methods exhibit slightly more uniform performance across configurations. The baseline heatmap shows clearer differentiation between configurations, with noticeable accuracy drops in shallower or narrower networks. In contrast, the search method heatmaps are more uniformly colored, suggesting that architecture search can partially compensate for suboptimal initial configuration choices by identifying the most valuable neurons to retain. Third, the NGMA heatmap shows the highest peak accuracy (98.68%) in the 3-layer, high-dimensional region, consistent with earlier findings that this method achieves the best individual results when the initial configuration is favorable. Appendix E Additional Analysis This section presents additional empirical analysis to better understand the behavior of the proposed NAS methods across different architectural configurations. Specifically, we examine the influence of network depth and hidden dimensions on classification performance, along with the parameter reduction achieved relative to the baseline dense architectures. These analyses provide further insights into the trade-off between model complexity and predictive performance. A Impact of Network Depth The relationship between network depth and performance provides insights into how architecture search interacts with fundamental architectural choices. Figure 15 illustrates this relationship. Error bars represent standard deviation across the three hidden dimension sizes (1024, 2048, 3072). Deeper networks generally perform better, but the advantage diminishes beyond 3 layers. Search methods maintain competitive performance across all depths. Figure 15: Test accuracy vs number of hidden layers, averaged across all hidden dimension configurations. All methods exhibit a generally positive relationship between depth and accuracy, with 3-layer and 4-layer networks performing best. The baseline shows the strongest depth dependency, improving from 98.51% (1 layer) to 98.63% (4 layers). This 0.12 percentage-point improvement demonstrates the value of depth for the MNIST task, though the gains are modest given the dataset’s relative simplicity. Interestingly, the search methods show less dramatic variation across depths. The NG approach maintains remarkably consistent performance (98.50% to 98.57%), suggesting that the pruning process adapts well across different architectural configurations. NAS-NGMA method shows the most interesting pattern, achieving its best performance (98.65%) at 3 layers rather than 4, indicating that the interaction between gating and activation criteria may favor intermediate depths where both metrics provide clear signals. B Influence of Hidden Dimensions Figure 16 examines how the total hidden dimension capacity (sum across all layers) affects performance. The standard deviation across various layer configurations that result in identical total dimensions is represented by error bars. Larger hidden dimensions generally improve accuracy, but with diminishing returns beyond 6,000 total dimensions. Figure 16: Test accuracy versus total hidden dimensions, averaged across configurations with the same total capacity. The results reveal diminishing returns as the number of hidden dimensions increases. Moving from 1,024 to 4,096 total dimensions yields noticeable improvements in accuracy across all methods. However, further expansion beyond 6,000 dimensions provides minimal additional benefit, with some configurations even showing slight performance degradation. This pattern is particularly evident in Figure 16, where the largest configurations (12,288 total dimensions for 4-layer networks with 3,072 neurons each) do not consistently outperform smaller configurations. This observation validates the architecture search approach: rather than simply scaling up to massive networks, intelligent pruning can identify the essential capacity needed for the task. The search methods follow similar trends but with reduced overall parameter counts at each dimension level. Notably, NAS-NG achieves competitive accuracy even at lower total dimensions, suggesting that it successfully identifies the most informative neurons rather than simply retaining neurons in proportion to the initial size. C Parameter Reduction Analysis To quantify the efficiency gains more precisely, we analyzed the parameter reductions achieved by each search method relative to its corresponding baseline configuration. Figure 17 presents this analysis in a box plot format. The red diamond markers indicate mean reduction values. Figure 17: Distribution of parameter reduction percentages achieved by each search method compared to baseline dense networks of equivalent initial size. NAS-NG achieves the highest and most consistent parameter reductions, averaging 68.67% (range: 43.88%–86.06%, std. dev. 15.64%). This reliable compression across configurations highlights the effectiveness of NG-based NAS. The combined approach yields a more conservative mean reduction of 54.49% (std. dev. 9.77%), reflecting its multi-criteria decision-making. By requiring agreement between gating and activation metrics, it is less likely to make extreme pruning decisions that might work well for one criterion but poorly for another. MA-based pruning shows a relatively smaller average reduction of 40.98% and greater variability (15.82%–56.76%, std. dev. 14.61%), indicating configuration-dependent performance that works well for some architectures but less effectively for others. A similar reduction in model complexity is also observed for the CNN architectures, as reported in Tables 5 and 7.