Paper deep dive
SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search
Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, Hung Cao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 3:12:48 AM
Summary
The paper introduces SEval-NAS, a search-agnostic evaluation framework for Neural Architecture Search (NAS) that converts neural network architectures into string representations, encodes them into vectors, and predicts performance metrics such as accuracy, latency, and memory. Evaluated on NATS-Bench and HW-NAS-Bench, SEval-NAS demonstrates strong correlation for hardware cost predictions and integrates seamlessly into existing NAS pipelines like FreeREA with minimal algorithmic changes.
Entities (12)
Relation Signals (10)
SEval-NAS → issearchagnostic → true
confidence 95% · To address this limitation, we propose SEval-NAS, a metric-evaluation mechanism... independent of the search algorithm
SEval-NAS → uses → Evaluator Network
confidence 95% · The methodology transforms each architecture into a standardized string representation, which is subsequently processed by an evaluator to predict performance metrics.
SEval-NAS → uses → Network-to-String Conversion
confidence 95% · The methodology transforms each architecture into a standardized string representation... consists of two primary components: (1) a network-to-string conversion mechanism
SEval-NAS → evaluateson → HW-NAS-Bench
confidence 92% · Using NATS-Bench and HW-NAS-Bench, we evaluated accuracy, latency, and memory.
SEval-NAS → evaluateson → NATS-Bench
confidence 92% · Using NATS-Bench and HW-NAS-Bench, we evaluated accuracy, latency, and memory.
SEval-NAS → integrateswith → FreeREA
confidence 90% · We further integrated SEval-NAS into FreeREA to evaluate metrics not originally included.
SEval-NAS → predicts → Memory
confidence 90% · Kendall's $ au$ correlations showed stronger latency and memory predictions than accuracy
SEval-NAS → predicts →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural architecture search (NAS) automates the discovery of neural networks that meet specified criteria, yet its evaluation procedures are often hardcoded, limiting the ability to introduce new metrics. This issue is especially pronounced in hardware-aware NAS, where objectives depend on target devices such as edge hardware. To address this limitation, we propose SEval-NAS, a metric-evaluation mechanism that converts architectures to strings, embeds them as vectors, and predicts performance metrics. Using NATS-Bench and HW-NAS-Bench, we evaluated accuracy, latency, and memory. Kendall's $\tau$ correlations showed stronger latency and memory predictions than accuracy, indicating the suitability of SEval-NAS as a hardware cost predictor. We further integrated SEval-NAS into FreeREA to evaluate metrics not originally included. The method successfully ranked FreeREA-generated architectures, maintained search time, and required minimal algorithmic changes. Our implementation is available at: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2603.00099v1
- Canonical: https://arxiv.org/abs/2603.00099v1
Trouble viewing inline? Open PDF directly →
Full Text
45,234 characters extracted from source content.
Expand or collapse full text
SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search ATAH NUH MIH, Analytics Everywhere Lab, University of New Brunswick, Canada JIANZHOU WANG, Analytics Everywhere Lab, University of New Brunswick, Canada TRUONG THANH HUNG NGUYEN, Analytics Everywhere Lab, University of New Brunswick, Canada HUNG CAO, Analytics Everywhere Lab, University of New Brunswick, Canada Neural architecture search (NAS) automates the discovery of neural networks that meet specified criteria, yet its evaluation procedures are often hardcoded, limiting the ability to introduce new metrics. This issue is especially pronounced in hardware-aware NAS, where objectives depend on target devices such as edge hardware. To address this limitation, we propose SEval-NAS, a metric-evaluation mechanism that converts architectures to strings, embeds them as vectors, and predicts performance metrics. Using NATS-Bench and HW-NAS-Bench, we evaluated accuracy, latency, and memory. Kendall’s휏correlations showed stronger latency and memory predictions than accuracy, indicating the suitability of SEval-NAS as a hardware cost predictor. We further integrated SEval-NAS into FreeREA to evaluate metrics not originally included. The method successfully ranked FreeREA-generated architectures, maintained search time, and required minimal algorithmic changes. Our implementation is available at: https://github.com/Analytics-Everywhere- Lab/neural-architecture-search. CCS Concepts:• Computing methodologies→Neural networks; Search methodologies;• Theory of computation→Network optimization. Additional Key Words and Phrases: Neural architecture search, network optimization ACM Reference Format: Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao. 2026. SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search. In The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece. ACM, New York, NY, USA, 16 pages. https://doi.org/10.1145/n.n 1 Introduction Neural architecture search (NAS) was developed to automate the design of neural networks (NNs), addressing the knowledge gap required to design these networks manually. Traditional NAS [35] used reinforcement learning (RL) to generate variable-length strings representing architectures, achieving state-of-the-art performance but at high Authors’ Contact Information: Atah Nuh Mih, Analytics Everywhere Lab, University of New Brunswick, Fredericton, Canada, atah.mih@unb.ca; Jianzhou Wang, Analytics Everywhere Lab, University of New Brunswick, Fredericton, Canada, maxwell.wang@unb.ca; Truong Thanh Hung Nguyen, Analytics Everywhere Lab, University of New Brunswick, Fredericton, Canada, hung.ntt@unb.ca; Hung Cao, Analytics Everywhere Lab, University of New Brunswick, Fredericton, Canada, hcao3@unb.ca. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. © 2026 Copyright held by the owner/author(s). Manuscript submitted to ACM Manuscript submitted to ACM1 arXiv:2603.00099v1 [cs.LG] 17 Feb 2026 2Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao computational cost. This shortcoming prompted research into more efficient search methods, such as a cell-based search space with learnable transferable architectures [34], Efficient NAS (ENAS) [23], Progressive NAS (PNAS) [17], and Regularized Evolution for Image Classifier Architecture Search [25]. In general, NAS is divided into five major components: (1) the search space containing predefined operation sets; (2) the controller that determines how neural architectures are generated; (3) the candidate architecture(s) generated; (4) the evaluation phase with a strategy for assessing architectures feasibility; and (5) the optimal architecture(s) that satisfy search objectives. The evaluation phase is a critical step in NAS because it assesses candidate architectures for the desired objectives and directs the search toward optimal designs [5]. Depending on the evaluation method, this phase can impose substantial search cost, as candidate architectures must be trained and tested to estimate performance [2]. For example, [34] trained each candidate to convergence on proxy data, resulting in a search cost of 22,400 GPU hours. The high cost of full training motivated the use of incomplete training to accelerate candidate ranking and reduce search costs [26]. Another challenge in existing NAS approaches is their limited flexibility in accommodating new evaluation metrics. Different NAS methods target different performance criteria, and hardware-aware NAS often incorporates metrics such as latency [29, 30, 10] and memory [16] to select architectures suited for platforms like edge devices. As a result, evaluation criteria are typically hardcoded into the search, making it difficult to introduce new metrics without redesigning the algorithm. Therefore, an evaluation mechanism that is independent of the search algorithm and can be flexibly adapted to any NAS approach is necessary. To address these challenges, we propose a search-agnostic evaluation approach called SEval-NAS. It converts any N into its string representation, encodes the string to obtain the vector embedding, and predicts the evaluation metrics. This is based on the premise that NNs’ performance reflects the structural dependencies of their internal operations (e.g., type of convolution, number of filters, type of activations). Extracting this structural information can, therefore, help predict their given performance. We show that SEval-NAS supports different types of metrics and evaluation objectives and can be directly applied to an existing NAS method with minimal changes to the search algorithm and without significantly affecting the search. We experiment on two NAS benchmarks: NATS-Bench [7] and HW-NAS-Bench [15] for accuracy, latency, and memory, and further assess how our method affects a NAS algorithm (i.e., FreeREA [2]). In summary, this work presents the following contributions: • A network-to-string conversion mechanism that traverses the autograd graph of any N and generates its textual representation, making it adaptive to all types of NNs. •An encoder-predictor network (i.e., an evaluator) that extracts meaningful relationships between the strings and their evaluation metric. This network can be designed to include any evaluation metric (notably hardware costs) and any number of evaluation objectives. •SEval-NAS that is independent of the NAS algorithm and combines the network-to-string conversion mechanism and the evaluator to evaluate candidate architectures in NAS. • An ablation study of three different encoder/decoder models (T5-small, T5-base, and T5-large) in SEval-NAS on NAS benchmarks. 2 Literature Review Training-Free NAS. The costly evaluation of candidate architectures has motivated the development of training-free metrics that evaluate candidate architectures and reduce the search time of NAS. [1] proposed regression models to predict the final performance of models from learning curve trajectories based on features obtained from the neural Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search3 architectures, hyperparameters, and time-series measurements. [21] proposed NASWOT, which examines the correlation of activations between data points in untrained NNs and scores networks based on the binary codes corresponding to this correlation. [4] proposed TE-NAS, a training-free NAS that analyzes the spectrum of the neural tangent kernel (NTK) and the number of linear regions to rank candidate architectures. While these methods successfully address the evaluation of candidate architectures, they solely focus on accuracy as their performance metrics. We extend beyond accuracy alone by including hardware metrics to assess the suitability of architectures for different computing environments. Hardware cost predictors have been developed, such as a n-Meter [32], a latency predictor for edge devices; and a GPU estimator for deep learning models [9]. Integrating different single-purpose cost predictors will increase the complexity of the design, so we propose a multi-purpose cost estimator that can incorporate different hardware costs. Hardware-Aware NAS. The search for good neural architectures extends beyond just high-accuracy networks. In cases where the hardware environment is crucial, cost metrics must be considered when evaluating the NNs. This requirement gave rise to HW-NAS, which includes a hardware cost metric and test accuracy in evaluating candidate architectures. Several HW-NAS approaches have been proposed, such as SqueezeNext [11], IRLAS [12], and FB-Net [30]. These works only provide the optimal architectures obtained from the search, leaving a question about their performance if the search were to be evaluated differently. Hardware-aware NAS equally targets edge devices because their hardware environments require NNs suitable for their resource constraints. Several NAS methods have been developed for edge devices [14, 29, 20, 28]. While these works have been successful in searching optimized NNs, they are usually designed to satisfy a single hardware cost metric. Latency has often been used as the evaluation metric in hardware-aware NAS [19, 33, 10, 16], whereas few works include multiple cost metrics in their design [27]. These NAS methods are generally multi-objective and aim to satisfy more than one primary objective pre-defined while designing the search. Our approach provides an easy integration of a cost evaluator for any desired objective (e.g., accuracy, latency, and memory) and several objectives (e.g., single or bi-objectives). NAS as a String Search Problem. Treating NAS as a string search problem was first explored in [34], where architectures were encoded as variable-length strings generated by a recurrent controller using reinforcement learning. GeNet [31] followed a similar idea, representing architectures as fixed-length binary strings and applying genetic algorithms to evolve them. Neural Architecture Optimization (NAO) [18] introduced an encoder–predictor–decoder framework that embeds architectures into a continuous space, predicts accuracy, and reconstructs architectures, with optimization performed directly in the embedding space. As in many NAS methods, prediction is tightly integrated into the search. In contrast, our prediction mechanism is plug-and-play and can be added to existing NAS pipelines, and we extend prediction to hardware costs, which is essential for HW-NAS. More recent work such as EVOPROMPTING [3] uses language modeling and prompting to generate code-level architectures, though it operates at a much higher level by producing program code rather than structured architectural representations. 3 Methodology Our proposed SEval-NAS framework is designed to evaluate neural architectures within a NAS pipeline by leveraging a formalized string-based representation and a predictive evaluation model. LetAdenote the set of candidate neural architectures, where each architecture푎 ∈ Ais characterized by its computational graph. The methodology transforms each architecture into a standardized string representation, which is subsequently processed by an evaluator to predict performance metrics. The predicted metrics guide the NAS controller in optimizing the search process. The framework Manuscript submitted to ACM 4Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao Search Space Preprocessing Network to String Conversion Autograd Graph Traversal and String Generation Generated Autograd Graph Autograd Graph Traversal and String Generation Generated Autograd Graph Forward Pass + Randomized Input Vector 1.478.481.549.871.54 ... 1.478.481.549.871.54 ... Network to String Conversion Autograd Graph Traversal and String Generation Generated Autograd Graph Forward Pass + Randomized Input Vector 1.478.481.549.871.54 ... |1_Convolution|2_Relu|3... |1_Convolution|2_Relu|3_P... Tokenizer 25384154987984 ... |1_Convolution|2_Relu|3_P... Tokenizer 25384154987984 ... Candidate ArchitecturesCandidate Architectures EvaluationEvaluation Optimal Architecture(s)Optimal Architecture(s) Ranking ControllerController Generating Architectures Selection Neural Architecture Search SEval-NAS Evaluator Text Encoder -3.112.45.56-4.16 ... 5.21.84-5.816.51 8.16-2.516.259.15 .164.68.782.15 4.849.84-2.153.51 -8.27.986.48-4.54 1.478.48-1.549.87 5.64 4.56 -2.84 2.48 .15 -5.48 1.54 ... ... ... ... ... ... ... -3.112.45.56-4.16 ... 5.21.84-5.816.51 8.16-2.516.259.15 .164.68.782.15 4.849.84-2.153.51 -8.27.986.48-4.54 1.478.48-1.549.87 5.64 4.56 -2.84 2.48 .15 -5.48 1.54 ... ... ... ... ... ... ... Predictor Predicted Metric(s) Embeddings Evaluator Text Encoder -3.112.45.56-4.16 ... 5.21.84-5.816.51 8.16-2.516.259.15 .164.68.782.15 4.849.84-2.153.51 -8.27.986.48-4.54 1.478.48-1.549.87 5.64 4.56 -2.84 2.48 .15 -5.48 1.54 ... ... ... ... ... ... ... Predictor Predicted Metric(s) Embeddings Search Space Preprocessing Network to String Conversion Autograd Graph Traversal and String Generation Generated Autograd Graph Forward Pass + Randomized Input Vector 1.478.481.549.871.54 ... |1_Convolution|2_Relu|3... |1_Convolution|2_Relu|3_P... Tokenizer 25384154987984 ... Candidate Architectures Evaluation Optimal Architecture(s) Ranking Controller Generating Architectures Selection Neural Architecture Search SEval-NAS Evaluator Text Encoder -3.112.45.56-4.16 ... 5.21.84-5.816.51 8.16-2.516.259.15 .164.68.782.15 4.849.84-2.153.51 -8.27.986.48-4.54 1.478.48-1.549.87 5.64 4.56 -2.84 2.48 .15 -5.48 1.54 ... ... ... ... ... ... ... Predictor Predicted Metric(s) Embeddings Fig. 1. Proposed SEval-NAS methodology and its integration in a NAS pipeline consists of two primary components: (1) a network-to-string conversion mechanism and (2) an evaluator network for performance prediction. Fig. 1 provides a schematic of the proposed methodology, illustrating its integration within a NAS pipeline. 3.1 Net-to-String Conversion The network-to-string conversion process maps a neural architecture푎 ∈ Ato a string representation푠 푎 ∈S, whereS is the space of all possible string representations. Let퐺 푎 =(푉 푎 ,퐸 푎 )represent the computational graph of architecture푎 (generated during the forward pass), with vertices푉 푎 corresponding to operations (e.g., convolution, pooling, ReLU) and edges 퐸 푎 representing data flow between operations. The conversion function 푓 :A →S traverses 퐺 푎 (breadth-first) to extract structural and operational details, yielding a string 푠 푎 that encapsulates the architecture’s configuration. Formally, the conversion process is defined as:푠 푎 = 푓(퐺 푎 ), where푓systematically traverses푉 푎 and퐸 푎 to encode operations and their connectivity into a standardized format. The resulting string푠 푎 is tokenized into a sequence of tokens푇 푎 =푡 1 ,푡 2 , . . .,푡 푛 , where each token푡 푖 corresponds to a specific operation or parameter in the computational graph. This tokenization ensures a universal and consistent representation, enabling compatibility across diverse NAS tasks and datasets. The conversion is described in Alg. 1. 3.2 Evaluator The tokenized input is processed in the evaluator module to predict its performance metrics. The evaluator module, denoted asE, predicts performance metrics for a given architecture based on its tokenized representation푇 푎 . Let M=푚 1 ,푚 2 , . . .,푚 푘 represent the set of target performance metrics (e.g., accuracy, latency, memory usage). The Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search5 Algorithm 1 Network to String Conversion Input: Neural Network: 푛푒푡 푎 ; Input Tensor: 푖푛푝 Output: Output token푇 푎 1: 퐺 푎 ←− 푛푒푡 푎 (푖푛푝)⊲ forward pass 2: 푛표푑푒=퐺 푎 (푟표푡)⊲ root node 3: 푠 푎 =∅⊲ output string 4: 푠푒푛=∅⊲ keep track of traversal 5: 푛표푑푒_푖푑= 0⊲ node index 6: function get_string(node) 7: if 푛표푑푒 in 푠푒푛 then return 8: end if 9: 푠푒푛 ←− 푛표푑푒 10: 푛푒푥푡 _푛표푑푒푠=푛표푑푒.푛푒푥푡 _푓푢푛푐푡푖표푛푠⊲ neighbouring vertices 11: for푢 in 푛푒푥푡 _푛표푑푒푠 do 12:get_string(u) 13: end for 14: 푣푎푟푠=푛표푑푒.푣푎푟푖푎푏푙푒⊲ get operations and variables 15: 푛푎푚푒=푛표푑푒.푛푎푚푒 16: 푠 푎 ←− 푛푎푚푒,푛표푑푒_푖푑,푣푎푟푠 17: end function 18: 푇 푎 =푡표푘푒푛푖푧푒(푠 푎 )⊲ tokenization with any desired method evaluator maps the tokenized input to a vector of predicted metrics: ˆ 푚 푎 =E(푇 푎 ) , where ˆ 푚 푎 =[ ˆ 푚 푎,1 , ˆ 푚 푎,2 , . . ., ˆ 푚 푎,푘 ] ∈R 푘 denotes the predicted values for the 푘 metrics. The module consists of two components: an encoder and a predictor. (1) Encoder: The encoder extracts a high-dimensional vector representation of the architecture, capturing its structural and contextual information. It is represented by a function푔:T →R 푑 that transforms the tokenized sequence푇 푎 into a high-dimensional embedding푒 푎 ∈R 푑 , capturing structural and contextual features of the architecture: 푒 푎 =푔(푇 푎 ). The encoder employs a transformer-based architecture to model dependencies within the token sequence, ensuring robust feature extraction. (2) Predictor: A function ℎ :R 푑 →R 푘 that maps the embedding 푒 푎 to the predicted metrics: ˆ 푚 푎 =ℎ(푒 푎 ). The prediction layer is a fully connected neural network with 푘 output neurons, where the number of neurons corresponds to the number of target metrics. For single-objective prediction (e.g., latency),푘=1, while for multi-objective prediction (e.g., accuracy and latency), 푘 ≥ 2. The evaluator is trained on a datasetD= (푎 푖 ,푚 푖 ) 푁 푖=1 , where푚 푖 ∈R 푘 are the true performance metrics for architecture 푎 푖 . The training objective minimizes the loss function: L= 1 푁 푁 ∑︁ 푖=1 ℓ(E(푇 푎 푖 ),푚 푖 ), whereℓis a regression loss (e.g., mean squared error) that measures the discrepancy between predicted and true metrics. Manuscript submitted to ACM 6Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao 3.3 Integration into NAS Pipeline The SEval-NAS framework is integrated into a NAS pipeline by evaluating candidate architectures generated by the controller. LetCdenote the controller, which generates architectures푎 ∈ Abased on a search strategy. The evaluator provides feedback in the form of predicted metrics ˆ 푚 푎 , enabling the controller to optimize the search objective: 푎 ∗ = arg max 푎∈A 푢( ˆ 푚 푎 ), where푢:R 푘 →Ris a utility function that aggregates the predicted metrics (e.g., a weighted sum for multi-objective optimization). The schematic of this integration is illustrated in Figure 1, highlighting the closed-loop interaction between the controller, candidate architectures, and the SEval-NAS evaluator. This modular design ensures that SEval-NAS can be seamlessly incorporated into existing NAS frameworks, such as FreeREA, without requiring significant modifications to the search algorithm. The flexibility of the prediction layer allows adaptation to varying numbers of objectives, enhancing the applicability of SEval-NAS across diverse hardware and performance constraints. 4 Experiments and Results We evaluated the effectiveness of SEval-NAS using two NAS benchmarks: NATS-Bench [7] and HW-NAS-Bench [15]. Our evaluation focused on how well our method’s predictions correlated with actual performance metrics. Additionally, we demonstrate the adaptability of our approach by applying it to FreeREA. We run our experiments on a 13th Gen Intel(R) Core(TM) 9-13900K server equipped with NVIDIA GeForce RTX 4090. 4.1 Model Configuration The evaluator in SEval-NAS is a transformer encoder whose input is the neural network’s text representation and outputs the embedding into a regression head for prediction. Specifically, we use the encoder from the T5 transformer [24]. It consists of stacked layers containing a self-attention layer and a small feed-forward network, followed by layered normalization and a residual skip connection. Dropout is strategically applied to the feed-forward network, the skip connection, the attention weights, and the stack’s input and output. We use three different sizes of T5 models : •T5-small, which uses 8-headed attention, has only 6 layers each in the encoder and decoder, and has roughly 60 million parameters. •T5-base, which uses 12-headed attention, has 12 layers each in the encoder and decoder, and has nearly 220 million parameters. •T5-large, which uses 16-headed attention, has 24 layers each in the encoder and decoder, and has approximately 770 million parameters The predictor is a single dense layer whose number of output neurons depends on the number of desired objectives. 4.2 Training The evaluator (T5-small model) is trained to predict performance metrics on datasets containing NNs and their reported metrics. Other models (T5-base and T5-large) are trained and evaluated via an ablation study (see Appendix A). In NAS, these datasets exist as NAS benchmarks containing thousands of neural architectures and metrics, i.e., accuracy, latency, and FLOPS obtained from training and inferencing those networks. Typically, these benchmarks exclude memory usage, prompting the need for additional profiling. To address this, we build a lookup table containing peak memory usage Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search7 using the built-in PyTorch memory profiling tool (i.e., torch.cuda.max_memory_allocated). This tool measures the peak memory allocated to tensors during training, providing an accurate assessment of NNs’ memory footprint. The profiling is isolated from memory used by external factors such as libraries or system variables, ensuring precise measurement. We train our evaluator on 2 NAS benchmarks: NATS-Bench [7] and HW-NAS-Bench [15]. 4.3 Experiment 1: Feasibility Testing (Evaluation on NATS-Bench) NATS-Bench [7] is a unified benchmark dataset for searching on both architecture topology and size. It consists of 15,625 different architectures for the Topology Search Space (TSS) and 32,768 architectures for the Size Search Space (S) evaluated on CIFAR-10, CIFAR-100, and ImageNet16-120. In the TSS, each architecture corresponds to a different cell represented as a densely connected directed acyclic graph (DAG) with four nodes and edges corresponding to operations from a predefined set of 5 operations. For each architecture in the TSS, the cells are stacked 5 times, with output channels set to 16, 32, and 64 for three stages. The search results in a search space containing 15,625 possible architectures configured for the image dataset considered (i.e., CIFAR-10, CIFAR-100 [13], and ImageNet16-120 [6]). The S searches for architectures by varying the number of channels in each layer (convolution, cell, or block). Each architecture consists of a stacked cell, and the number of channels in each layer is selected from the set8,16,24,32,40,48,56,64, resulting in 32,768 architectures. For our experiment, we evaluate the TSS and S. In each search space, we separately train the evaluator on the CIFAR-10, CIFAR-100, and ImageNet16-120 architectures, configuring our network for both (accuracy, memory) and (accuracy, latency) bi-objective setups. Memory (a) CIFAR-10(b) CIFAR-100 (c) ImageNet16-120 Latency (d) CIFAR-10(e) CIFAR-100 (f) ImageNet16-120 Fig. 2. Plots of predicted vs true hardware cost of NATS-Bench TSS architectures (T5-small) for performance metrics reported on CIFAR-10, CIFAR-100, and ImageNet16-120. The strength of correlation increases as 휏 approaches 1. Manuscript submitted to ACM 8Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao The results of the NATS-Bench TSS in Fig. 2 show a strong positive Kendall휏correlation between predicted and true values for hardware costs. Predicted memory usage aligns closely with the true values across all datasets. Similarly, latency predictions exhibit a high correlation for CIFAR-10 and CIFAR-100, while ImageNet16-120 shows a slightly weaker correlation. This suggests that SEval-NAS effectively predicts hardware costs in the TSS space due to the architectural features. The reason why the SEval-NAS effectively predicts hardware costs in the TSS in Fig. 2 is due to the Autograd Traversal and String Generation block, which significantly optimizes the neural architecture in topology. Memory (a) CIFAR-10(b) CIFAR-100 (c) ImageNet16-120 Latency (d) CIFAR-10(e) CIFAR-100 (f) ImageNet16-120 Fig. 3. Plots of predicted vs true hardware cost of NATS-Bench S architectures (T5-small) for performance metrics reported on CIFAR-10, CIFAR-100, and ImageNet16-120. The strength of correlation increases as 휏 approaches 1. For the NATS-Bench S results illustrated in Fig. 3, predicted memory usage again shows a strong positive correlation with the true values across all datasets. Predicted latency has a similar trend with a strong correlation in CIFAR-10 and CIFAR-100. However, latency prediction for the ImageNet16-120 dataset appears less robust, with noticeable variance. In terms of predicted latency on S across CIFAR-10, CIFAR-100, and ImageNet16-120 in Fig. 2. From the dataset itself, especially ImageNet16-120 compared to CIFAR-100 and CIFAR-10, we know ImageNet16-200 emphasizes low-resolution feature extraction; thus, ImageNet16-120 is reliable for memory predictions but unstable for latency prediction. Whereas CIFAR-100 and CIFAR-10 are designed for fine-grained classification, which results in good reliability on both memory predictions and latency memory. Therefore, while SEval-NAS remains reliable for memory predictions, its reliability in predicting latency is dataset-dependent and influenced by the variability of the architectures. In comparison, while positively correlated, accuracy predictions in Fig. 4 exhibit weaker correlations than those for hardware costs. This suggests that SEval-NAS struggles to confidently infer accuracy from neural architecture representations. Furthermore, there is no trend linking dataset type to prediction reliability for accuracy, highlighting that accuracy depends on factors beyond straightforward architectural features. Overall, SEval-NAS demonstrates Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search9 (a) NATS-Bench TSS(b) NATS-Bench S Fig. 4. Kendall’s휏correlation for predicted vs true accuracy in NATS-Bench TSS and S search spaces. Comparison includes (accuracy, latency) and (accuracy, memory) bi-objectives. stronger predictability for hardware metrics than accuracy, primarily because hardware costs are directly tied to architectural characteristics. For example, more convolutional filters in a network will need more computation than fewer convolutional filters [22]. Meanwhile, this correlation cannot be directly made for accuracy. We also conducted an experiment (Appendix A) to evaluate three different encoder/decoder models (T5-small, T5-base, and T5-large) on NATS-Bench S and NATS-Bench TSS. This ablation study compared how model size affects the correlation between predicted versus true memory and predicted versus true latency, as measured by Kendall correlation. The results from our extra ablation studies (Fig. 6 in Appendix A.1) showed that T5-small, T5-base, and T5-large perform similarly on the TSS benchmark. However, we observed that only T5-large encoders exhibit lower Kendall휏 correlations on the S. Additionally, we found that different encoder sizes (Appendix A.2) do not significantly impact performance on NATS-Bench TSS for either memory or latency correlations (Fig. 7). In contrast, T5-large demonstrates weaker 휏 correlations for both memory and latency predictions on NATS-Bench S, as shown in Fig. 8. 4.4 Experiment 2: Predicting Hardware Cost (Evaluation on HW-NAS-Bench) Although we observe a strong positive latency correlation in the NATS-Bench search spaces, the latency of a N largely depends on the hardware environment. To investigate how well SEval-NAS would theoretically predict latency across various hardware devices, we evaluate its performance on HW-NAS-Bench. HW-NAS-Bench [15] was designed for hardware-aware NAS. It includes two NAS search space designs: NAS-Bench- 201’s cell-based search space and FBNet’s search space. The dataset provides the hardware cost of the NNs from both search spaces on commercial devices, including Edge GPU, Edge TPU, ASIC Eyeriss, FPGA, Pixel 3, and Raspberry Pi 4. FBNet search space [30] builds a layer-wise search space with a fixed macro-architecture and varying middle layers that can be searched. The architectures in this search space have regular structures that include nine cell candidates and 22 positions, yielding 9 22 ≈10 21 different architectures. Due to the excessively large size of the search space, we do not use it in our experiment. NAS-Bench-201 search space [8] is the original search space of the TSS architectures in NATS-Bench. It contains the same 15,625 architectures with results reported for CIFAR-10, CIFAR-100, and ImageNet16-120 and their Manuscript submitted to ACM 10Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao hardware costs on each of the six devices. We evaluate SEval-NAS on the HW-NAS-Bench’s NAS-Bench-201 subspace for values reported on the CIFAR-10 dataset. The evaluator is trained to predict only latency, testing the performance of SEval-NAS on a single objective metric. (a) Edge GPU(b) Edge TPU(c) ASIC Eyeriss (d) FPGA(e) Pixel 3(f) Raspberry Pi 4 Fig. 5. Plots of predicted vs true latency for NAS-Bench-201 architectures with T5-small encoder for 6 edge devices reported in the HW-NAS-Bench benchmark. The strength of correlation increases as 휏 approaches 1. Results in Fig. 5 show a strong positive Kendall휏correlation for most edge devices, with values ranging from 0.6047 to 0.9742, demonstrating SEval-NAS’s ability to predict latency across six different edge devices. The Edge TPU’s latency predictions stand out as an outlier, showing a weaker correlation (휏=0.6047). This is attributed to negative latency values reported in the HW-NAS-Bench dataset, which likely affected the model’s ability to predict latency in this case. Despite this, other devices, such as the Edge GPU (휏=0.8676), Eyeriss (휏=0.9558), FPGA (휏=0.9742), Pixel 3 (휏=0.8599), and Raspi4 (휏=0.8659), exhibit strong correlations, indicating consistent and reliable performance across diverse hardware configurations. Both experiments show that training the evaluator for bi-objectives in NATS-Bench and a single objective in HW- NAS-Bench consistently yielded positive Kendall휏correlation values. This demonstrates the effectiveness of SEval-NAS in adapting to different numbers of evaluation objectives, further solidifying its strength as a predictive method for latency across edge devices. We also conduct the additional experiments (in Appendix A.3) to run three different encoder/decoder models: T5-small (Fig. 5), T5-base (Fig. 9), and T5-large (Fig. 10) on HW-NAS-Bench as an ablation study of comparison of model size on the predicted latency vs true latency. We observe that different encoder/decoder models do not have great impact (less than 0.02 latency correlation difference) except that T5-base in Edge GPU (0.8804) and T5-large in Edge GPU (0.8852) have stronger latency correlation Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search11 Table 1. Test accuracy and time for various NAS algorithms for NATS-Bench. CIFAR 10CIFAR 100ImageNet16-120 AlgorithmAccuracyTime (s)AccuracyTime (s)AccuracyTime (s) NASWOT (1000)93.10± 0.3124869.10± 1.6124845.08± 1.55248 TENAS93.90± 0.47155871.24± 0.56155842.38± 0.461558 NASI93.55± 0.1012071.20± 0.1412044.84± 1.41120 GA-NINASWOT93.70± 0.6320671.57± 1.3720645.18± 2.05206 EPE-NAS91.31± 1.6910469.58± 0.8310441.84± 2.06104 FreeREA94.36± 0.004573.51± 0.054546.34± 0.0045 FreeREA + Latency94.36± 0.007773.51± 0.008246.34± 0.0081 FreeREA + Memory84.21± 14.913449.67± 11.353719.66± 7.1938 than T5-small in Edge GPU(0.8434) since T5-small’s operators are smaller, which results in GPU kernel launching overhead accounting for a higher proportion of the total latency. Moreover, T5-large and T5-base have longer latency, the relative impact of noise is smaller, which leads to a higher latency correlation. 4.5 Experiment 3: Ease of Integration (Evaluator in a NAS) The contrast in correlation values observed between accuracy (low to moderate correlation) and hardware costs (strong correlation) highlights the evaluator’s strength as a hardware cost predictor. This strength can be effectively leveraged to enhance NAS approaches that traditionally optimize accuracy as a single objective by incorporating hardware constraints into their search strategies. To demonstrate this applicability, we integrate SEval-NAS into FreeREA [2] to search in NATS-Bench search space and define hardware constraints for the search. FreeREA uses evolutionary search to find candidate architectures and evaluates candidate architectures using a training-free metric to estimate the accuracy performance. FreeREA algorithm on NATS-Bench TSS is constrained by퐹퐿푂푃푆and #푃푎푟푎푚푠. Since FLOPS is a poor proxy for hardware costs [15, 29], we replace it with two alternative constraints: 1) latency and 2) memory usage, each tested separately. For each metric, the mean value reported in the benchmark is set as the threshold for ranking candidate architectures. For example, these cases use the mean latency (45.96 seconds) and memory usage (166.67 MB) from the NATS-Bench CIFAR-10 dataset as thresholds. The resulting performance is compared against FreeREA and other training-free NAS algorithms reported in [2], with findings summarized in Table 1. Latency-constrained search identified an average of approximately 230 architectures satisfying the threshold, achiev- ing final average accuracies consistent with those reported in the original FreeREA study. In contrast, the memory- constrained search discovered fewer architectures (on average, fewer than 10) that met the threshold, indicating a bias in the search algorithm against low-memory architectures. This smaller pool of candidates led to higher variability in test accuracies due to the diverse performance of low-memory architectures. Importantly, while the latency-constrained search doubled FreeREA’s search time, this overhead from evaluator inference remained negligible compared to other NAS algorithms. Conversely, the memory-constrained search required less time due to the limited number of viable candidate architectures. Manuscript submitted to ACM 12Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao These results highlight the flexibility of SEval-NAS for hardware cost evaluation while making minimal changes to the algorithm. By integrating additional constraints (e.g., latency and memory thresholds), the NAS algorithm can be tailored to select candidate architectures suitable for target hardware devices. For instance, deploying SEval-NAS on an edge device with memory constraints matching the device’s operating range would yield architectures suitable for deployment. This study establishes the feasibility of incorporating SEval-NAS into NAS pipelines, with the specific objective of demonstrating integration viability rather than algorithmic optimization. The investigation of threshold parameter effects on search dynamics represents a natural extension of this foundational work and constitutes a promising direction for future research. 5 Conclusion and Future Work NAS discovers novel architectures without expert knowledge but incurs long evaluation times when training or deploying candidate models. We proposed SEval-NAS, which converts architectures to string representations via autograd graph traversal and maps their embeddings to predicted performance metrics. We evaluated SEval-NAS on NATS-Bench and HW-NAS-Bench for accuracy, latency, and memory. Latency and memory predictions showed the strongest correlations, demonstrating SEval-NAS as an effective hardware cost predictor, while accuracy predictions showed moderate correlation and revealed limitations in estimating accuracy. Ablation studies across encoder and decoder sizes showed that larger encoders reduced Kendall휏correlations on NATS-Bench S. On HW-NAS-Bench, larger models improved latency correlation on Edge GPU due to kernel operator effects, but also increased latency. To assess adaptability, we integrated SEval-NAS into FreeREA [2] by adding latency and memory constraints. SEval-NAS introduced minimal search-time overhead and enabled new evaluation criteria with limited algorithmic changes. These findings also showed that SEval-NAS complements training-free NAS methods focused on accuracy, providing broader evaluation across multiple metrics. Our experiments relied on benchmark-reported hardware metrics, which may differ from measurements on actual devices. This limitation can be mitigated by deploying a lightweight SEval-NAS in an on-device NAS setting. Developing such a system and exploring additional FreeREA thresholds remain future work. Acknowledgement This work has been supported by NSERC Discovery Grant No RGPIN 2025-00129. References [1]Bowen Baker et al. 2018. Accelerating neural architecture search using performance prediction. In International Conference on Learning Represen- tations. [2]Niccoì O Cavagnero et al. 2023. Freerea: training-free evolution-based architecture search. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1493–1502. [3]Angelica Chen et al. 2023. Evoprompting: language models for code-level neural architecture search. Advances in Neural Information Processing Systems, 36. [4]Wuyang Chen et al. 2021. Neural architecture search on imagenet in four gpu hours: a theoretically inspired perspective. In International Conference on Learning Representations. [5]Krishna Teja Chitty-Venkata et al. 2023. Neural architecture search benchmarks: insights and survey. IEEE Access, 11, 25217–25236. [6]Patryk Chrabaszcz et al. 2017. A downsampled variant of imagenet as an alternative to the cifar datasets. (2017). arXiv: 1707.08819 [cs.CV]. [7] Xuanyi Dong et al. 2022. Nats-bench: benchmarking nas algorithms for architecture topology and size. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, (July 2022), 3634–3646, 7, (July 2022). [8] Xuanyi Dong and Yi Yang. 2020. Nas-bench-201: extending the scope of reproducible neural architecture search. In International Conference on Learning Representations. [9]Yanjie Gao et al. 2020. Estimating gpu memory consumption of deep learning models. Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 20. Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search13 [10]Mohamed Imed Eddine Ghebriout et al. 2024. Harmonic-nas: hardware-aware multimodal neural architecture search on resource-constrained devices. In Asian Conference on Machine Learning. PMLR, 374–389. [11]Amir Gholami et al. 2018. Squeezenext: hardware-aware neural network design. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 1638–1647. [12]Minghao Guo et al. 2019. Irlas: inverse reinforcement learning for architecture search. In Proceedings of the IEEE conference on computer vision and pattern recognition. [13]Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. [14]Achintya Kundu et al. 2023. Transfer-once-for-all: ai model optimization for edge. IEEE International Conference on Edge Computing and Communications. [15]Chaojian Li et al. 2021. Hw-nas-bench:hardware-aware neural architecture search benchmark. ICLR 2021 - 9th International Conference on Learning Representations, (Mar. 2021). [16]Yuke Li et al. 2023. Pareto optimization of cnn models via hardware-aware neural architecture search for drainage crossing classification on resource-limited devices. Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis. [17]Chenxi Liu et al. 2018. Progressive neural architecture search. In European Conference on Computer Vision (ECCV). [18]Renqian Luo et al. 2021. Neural architecture optimization. In Neural Information Processing Systems. https://github.com/renqianluo/NAO.. [19]Xiangzhong Luo et al. 2020. Edgenas: discovering efficient neural architectures for edge systems. Proceedings - IEEE International Conference on Computer Design: VLSI in Computers and Processors, 2020-October, (Oct. 2020), 288–295. [20]Bo Lyu et al. 2022. Resource-constrained neural architecture search on edge devices. IEEE Transactions on Network Science and Engineering, 9, 1. [21]Joe Mellor et al. 2021. Neural architecture search without training. In International conference on machine learning. PMLR, 7588–7598. [22] Atah Nuh Mih et al. 2024. Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment. Proceedings of the Canadian Conference on Artificial Intelligence. [23]Hieu Pham et al. 2018. Efficient neural architecture search via parameters sharing. In International conference on machine learning. PMLR. [24]Colin Raffel et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21, 140, 1–67. [25] Esteban Real et al. 2019. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence. [26]Pengzhen Ren et al. 2021. A comprehensive survey of neural architecture search: challenges and solutions. ACM Computing Surveys (CSUR), 54, 4, 1–34. [27]Blake Richey et al. 2024. Multi-reward optimization using genetic algorithms for edge ai. In Real-Time Image Processing and Deep Learning 2024. SPIE. [28]Matteo Risso et al. 2022. Lightweight neural architecture search for temporal convolutional networks at the edge. doi:10.1109/TC.2022.3177955. [29]Nilotpal Sinha et al. 2024. Hardware aware evolutionary neural architecture search using representation similarity metric. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2628–2637. [30]Bichen Wu et al. 2019. Fbnet: hardware-aware efficient convnet design via differentiable neural architecture search. In IEEE/CVF conference on computer vision and pattern recognition. [31]Lingxi Xie and Alan Yuille. 2017. Genetic cnn. In International Conference on Computer Vision. IEEE. [32] Li Lyna Zhang et al. 2021. Towards accurate latency prediction of deep-learning model inference on diverse edge devices. In International Conference on Mobile Systems, Applications, and Services. ACM. [33]Yusen Zhang et al. 2024. Oncenas: discovering efficient on-device inference neural networks for edge devices. Information Sciences. [34]Barret Zoph et al. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 8697–8710. [35] Barret Zoph and Quoc Le. 2017. Neural architecture search with reinforcement learning. In International Conference on Learning Representations. A Appendix A.1 Ablation Study 1: Evaluation among T5 encoders across Bi-Objective Setups) Manuscript submitted to ACM 14Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao (a) NATS-Bench TSS(b) NATS-Bench S Fig. 6. Kendall’s휏correlation for predicted vs true accuracy among encoders of T5-small, T5-base, and T5-large on NATS-Bench TSS and NATS-Bench S (accuracy, latency) and (accuracy, memory) bi-objectives. A.2 Ablation Study 2: Feasibility Testing (Evaluation on NATS-Bench) Memory (a) T5-small(b) T5-base(c) T5-large Latency (d) T5-small(e) T5-base(f) T5-large Fig. 7. Plots of predicted vs true hardware cost of NATS-Bench TSS architectures for performance metrics reported on T5-small, T5-base, and T5-large. The strength of correlation increases as 휏 approaches 1. Manuscript submitted to ACM SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search15 Memory (a) T5-small(b) T5-base(c) T5-large Latency (d) T5-small(e) T5-base(f) T5-large Fig. 8. Plots of predicted vs true hardware cost of NATS-Bench S architectures for performance metrics reported on T5-small, T5-base, and T5-large. The strength of correlation increases as 휏 approaches 1. A.3 Ablation Study 3: Predicting Hardware Cost (Evaluation on HW-NAS-Bench) Manuscript submitted to ACM 16Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, and Hung Cao (a) Edge GPU(b) Edge TPU(c) ASIC Eyeriss (d) FPGA(e) Pixel 3(f) Raspberry Pi 4 Fig. 9. Plots of predicted vs true latency for NAS-Bench-201 architectures with T5-base encoder for 6 edge devices reported in the HW-NAS-Bench benchmark. The strength of correlation increases as 휏 approaches 1. (a) Edge GPU(b) Edge TPU(c) ASIC Eyeriss (d) FPGA(e) Pixel 3(f) Raspberry Pi 4 Fig. 10. Plots of predicted vs true latency for NAS-Bench-201 architectures with T5-large encoder for 6 edge devices reported in the HW-NAS-Bench benchmark. The strength of correlation increases as 휏 approaches 1. Manuscript submitted to ACM