Paper deep dive
Artificial Neural Networks as Surrogate Models in Black Box Optimization
Md Khadimul Islam Zim, Martin Holeňa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/22/2026, 4:37:28 AM
Summary
This paper introduces AFN-CMA-ES, a novel surrogate-assisted Black-Box Optimization (BBO) method that integrates an Adaptive-Fidelity Nexus (AFN) surrogate model based on an ensemble of Artificial Neural Networks (ANNs) with the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). The AFN model uses a bootstrapped multilayer perceptron ensemble to predict objective function values and estimate epistemic uncertainty, allowing the optimizer to perform surrogate-only generations to reduce costly true function evaluations. The model life (number of surrogate-only generations) is adaptively controlled based on Kendall’s rank correlation between surrogate predictions and true evaluations. Experimental results on the COCO/BBOB benchmark suite demonstrate that AFN-CMA-ES achieves competitive performance compared to state-of-the-art surrogate-assisted CMA-ES variants like LQ-CMA-ES, DTS-CMA-ES, and LMM-CMA-ES.
Entities (10)
Relation Signals (9)
AFN-CMA-ES → evaluatedon → COCO/BBOB
confidence 95% · The proposed AFN-CMA-ES algorithm was evaluated on the COCO/BBOB (Black-Box Optimization Benchmarking) test suite
AFN-CMA-ES → isvariantof → CMA-ES
confidence 95% · The proposed AFN-CMA-ES framework extends the standard CMA-ES by integrating it with the Adaptive-Fidelity Nexus (AFN) surrogate system.
AFN-CMA-ES → uses → Adaptive-Fidelity Nexus
confidence 95% · AFN-CMA-ES framework extends the standard CMA-ES by integrating it with the Adaptive-Fidelity Nexus (AFN) surrogate system.
Adaptive-Fidelity Nexus → isbasedon → Artificial Neural Networks
confidence 92% · This new surrogate model helps efficiently optimize computationally expensive functions using an adaptively refined artificial neural network (ANN).
Adaptive-Fidelity Nexus → uses → Multilayer Perceptron
confidence 90% · AFN-CMA-ES extends neural-network surrogate-assisted BBO by integrating a bootstrapped multilayer perceptron (MLP) ensemble
Adaptive-Fidelity Nexus → uses → Kendall's Tau
confidence 88% · AFN-CMA-ES extends neural-network surrogate-assisted BBO by integrating ... Kendall’s τ-based adaptive model life within CMA-ES.
AFN-CMA-ES → outperformsorcompeteswith → LMM-CMA-ES
confidence 80% · The experimental results show its competitive performance compared to state-of-the-art surrogate-assisted BBO methods.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Black-Box Optimization (BBO) is often applied in several engineering fields and can utilize an advancement of numerical measure- ments and simulation technologies. It deals with the optimization func- tions, where an analytical description is unavailable. It relies on meth- ods that require only an input point in the search space, paired with its corresponding objective function value, obtained through non-analytical means, e.g., sensors, experiments, or simulations. Common approaches include evolutionary optimization and other metaheuristics. Since BBO methods rely solely on objective function values, they typically require many evaluations, which becomes problematic when evaluating the ob- jective function is time-consuming or expensive. This leads to using surrogate-based optimization which evaluates selected true objective val- ues and trains a regression model to approximate the objective function across the search space. Surrogate-assisted black-box optimization is a small-data learning problem because the optimizer must approximate an expensive objective function from limited evaluations. Surrogate models act as data-efficient regressors, guiding the search toward promising or informative points under a restricted evaluation budget. In this paper, a new surrogate model using artificial neural networks, called Adaptive- Fidelity Nexus Covariance Matrix Adaptation Evolution Strategy (AFN- CMA-ES), is proposed for the selective evaluation of objective functions. The experimental results show its competitive performance compared to state-of-the-art surrogate-assisted BBO methods.
Tags
Links
- Source: https://arxiv.org/abs/2609.22329v1
- Canonical: https://arxiv.org/abs/2609.22329v1
Trouble viewing inline? Open PDF directly →
Full Text
42,469 characters extracted from source content.
Expand or collapse full text
Artificial Neural Networks as Surrogate Models in Black Box Optimization ⋆ Md Khadimul Islam Zim 1 and Martin Holeňa 1 Czech Academy of Sciences, Institute of Computer Science, Pod Vodárenskou věží 2, Prague, Czech Republic mekuzim@gmail.com, martin@cs.cas.cz Abstract. Black-Box Optimization (BBO) is often applied in several engineering fields and can utilize an advancement of numerical measure- ments and simulation technologies. It deals with the optimization func- tions, where an analytical description is unavailable. It relies on meth- ods that require only an input point in the search space, paired with its corresponding objective function value, obtained through non-analytical means, e.g., sensors, experiments, or simulations. Common approaches include evolutionary optimization and other metaheuristics. Since BBO methods rely solely on objective function values, they typically require many evaluations, which becomes problematic when evaluating the ob- jective function is time-consuming or expensive. This leads to using surrogate-based optimization which evaluates selected true objective val- ues and trains a regression model to approximate the objective function across the search space. Surrogate-assisted black-box optimization is a small-data learning problem because the optimizer must approximate an expensive objective function from limited evaluations. Surrogate models act as data-efficient regressors, guiding the search toward promising or informative points under a restricted evaluation budget. In this paper, a new surrogate model using artificial neural networks, called Adaptive- Fidelity Nexus Covariance Matrix Adaptation Evolution Strategy (AFN- CMA-ES), is proposed for the selective evaluation of objective functions. The experimental results show its competitive performance compared to state-of-the-art surrogate-assisted BBO methods. Keywords: Black-box optimization, Surrogate modelling, Adaptive-Fidelity Nexus, Artificial Neural Networks, CMA-ES. 1 Introduction Black-box optimization (BBO) is concerned with optimization problems where for the objective function, an expression as explicit or implicit analytical ex- pression is not available. In such problems, the optimization algorithm can only ⋆ A version of this work has been accepted for publication at the European Confer- ence on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2026). arXiv:2609.22329v1 [cs.NE] 16 Sep 2026 2M.K.I. Zim and M. Holeňa evaluate candidate solutions and observe the corresponding objective function values. These problems are often encountered in engineering design, where the performance of a system can only be assessed by performing numerical simula- tions, physical experiments, sensor readings or other non-analytical methods. BBO is important because many scientific and engineering optimization prob- lems involve objective functions that are not available analytically and are typ- ically evaluated through experiments or costly simulations. Because such eval- uations are expensive, the evaluation budget is usually limited, and the main challenge is to find good solutions with as few true objective-function evalu- ations as possible. Among state-of-the-art methods for continuous BBO, the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is widely used for nonlinear and non-convex problems because it adapts the covariance matrix of the sampling distribution and can learn dependencies among decision variables during the search process [1]. CMA-ES samples candidate solutions, ranks them, and updates its search distribution toward promising regions. Recent benchmark studies also suggest the need to evaluate BBO algorithms on more realistic prob- lem classes, such as mixed-integer optimization problems [2]. Since BBO methods rely solely on objective function values, common ap- proaches such as evolutionary algorithms and other metaheuristics often require many evaluations. This is less efficient when the evaluations are slow or expen- sive. Surrogate-assisted BBO overcomes this problem by limiting the number of candidate solutions evaluated by the objective function and by learning a re- gression model called surrogate model that estimates the objective function over the solution space. The surrogate then guides the optimizer toward promising or informative regions while reducing the number of costly evaluations [3]. Surrogate-assisted BBO is important from the small-data perspective be- cause it evaluates the true objective function only for selected candidates and uses these evaluated input–output pairs to train a surrogate model that guides the search. Thus, in many machine-learning applications, the surrogate model in BBO cannot be trained on a large dataset because each training sample requires a costly true objective-function evaluation. Rather, the optimizer must learn a good approximation to the objective function from a small number of costly input-output pairs. As a result, accurate prediction, uncertainty quantification, and adaptive sampling are key to surrogate-assisted BBO. Various types of surro- gate models, such as Gaussian processes, polynomials, random forests and other regression models, are used to assist the optimizer with minimal function calls. 1.1 Aim The goal of the reported research is to develop and validate the Adaptive-Fidelity Nexus (AFN). This new surrogate model helps efficiently optimize computation- ally expensive functions using an adaptively refined artificial neural network (ANN). The main goal is to develop a smart refinement, which will be carefully tested using several key measures. The main research objectives are: – To create the adaptive AFN, which combines ANNs as surrogate models with intelligent sampling methods based on uncertainty quantification. Black Box Optimization3 – To substantially reduce the amount of costly objective function evaluations while getting sufficiently close to the optimal solution. – To investigate the dependence of the AFN performance on the character of the benchmark function. – To enhance the robustness and resource efficiency of the optimization process for a range of problem instances. 2 Literature Review Surrogate-assisted optimization has grown rapidly as an effective approach to BBO problems that need to keep low the number of true function evaluations [3]. Research performed so far demonstrates that surrogate-assisted CMA-ES can reduce the number of expensive fitness evaluations. Gaussian processes, ran- dom forests, and local quadratic models have been extensively investigated as surrogate models for CMA-ES, and their performance depends on the problem landscape, dimensionality, and evolution-control strategy [4]. Ordinal Gaussian process regression have also been investigated for surrogate-assisted CMA-ES, due to CMA-ES invariance with respect to monotonously increasing transfor- mations [5]. Recently, modern neural-network-based surrogate models have been investigated for active and transfer learning in surrogate-assisted black-box op- timization. However, they also present issues related to uncertainty estimation, stability, and scalability [6]. AFN-CMA-ES extends neural-network surrogate- assisted BBO by integrating a bootstrapped multilayer perceptron (MLP) en- semble, rank-normalized local-archive training, anchored regularization, uncer- tainty estimation, and Kendall’s τ-based adaptive model life within CMA-ES. 3 Proposed Methodology 3.1 Overview The proposed AFN-CMA-ES framework extends the standard CMA-ES by in- tegrating it with the Adaptive-Fidelity Nexus (AFN) surrogate system. Rather than using an expensive function in every iteration, AFN-CMA-ES uses a trained ANN ensemble to predict values and estimate their uncertainty. It is called Adaptive-Fidelity Nexus because it adaptively combines true evaluations, sur- rogate predictions, and CMA-ES within one framework. AFN-CMA-ES was implemented according to the proposed framework, using the ANN ensemble surrogate, generation-based evolution control, and CMA-ES update procedure summarized in Algorithm 1 below. 3.2 Adaptive-Fidelity Nexus AFN operates in 3 main stages: 4M.K.I. Zim and M. Holeňa 1. Surrogate Training: An ensemble of M = 5 MLP networks is trained on the current evaluation samples. Although all ensemble members use the same MLP architecture, prediction diversity is ensured through independent training. For a local archive of N loc samples, each bootstrap sample contains N loc observations drawn with replacement. The level of disagreement among them is used to estimate the epistemic uncertainty. There is a prediction ˆ f(x) within each network, and the variance estimates the uncertainty. Algorithm 1 AFN-CMA-ES Input: Objective function f(x), bounds [lb,ub], budget B, ensemble size M, model life L Output: Best evaluated solution x ∗ 1: Generate initial Latin hypercube sampling (LHS) archive D = (x i ,f(x i )) N 0 i=1 , where N 0 = 20d. 2: Train an ensemble of M ANN surrogates on rank-normalized objective values. 3: Initialize CMA-ES from the best sample in D and set t← N 0 . 4: while t < B do 5: Generate CMA-ES population X g =x g,1 ,...,x g,λ . 6: Evaluate all candidates with the true objective function and update D, x ∗ , and CMA-ES. 7: Estimate surrogate ranking quality τ g using Kendall’s correlation. τ g is one scalar value for generation g and is computed before retraining to assess the surrogate model used in the previous surrogate-only generation. 8: Retrain the ANN ensemble on a local archive subset around the CMA-ES mean. 9: Set model life L g = L in fixed mode, or choose L g according to τ g in adaptive mode. 10: for j = 1 to L g do 11:Generate surrogate-only CMA-ES population e X g,j . 12:Predict values using ensemble mean ˆμ(x). 13:Update CMA-ES using rank-normalized surrogate fitness scores. 14: end for 15: Update t←|D|. 16: end while 17: return x ∗ and f(x ∗ ) 2. Surrogate only ranking: Training uses the initial LHS archive and the local archive subset selected around the current CMA-ES mean, where d is the problem dimension. This dimension aware design prevents the MLP from fitting on insufficient data in higher dimensional spaces. The ensemble com- putes: Predictive mean (Exploitation): ˆμ(x) = 1 M P M i=1 ˆ f i (x) Predictive variance (Exploration): ˆσ 2 (x) = 1 M P M i=1 ˆ f i (x)− ˆμ(x) 2 Black Box Optimization5 The ensemble predictive mean is used to rank candidate solutions during surrogate only CMA-ES generations. The ensemble variance provides un- certainty information, but the revised implementation does not use an in- dividual uncertainty threshold to decide whether each candidate should be evaluated by the true objective function. Instead, true evaluations are con- trolled at the generation level. 3. Iterative Refinement: New evaluations are added to the archive, and a local archive subset of at most 250 samples is selected according to distance from the current CMA-ES mean. This limit balances local relevance and retraining cost. In the initial phase, when the number of available evaluations is less than 250, all available samples are used. The surrogate is retrained using this local archive subset after each true evaluated generation. This helps the surrogate remain focused on the current search region, since samples far from the current estimate of the optimum may be more misleading than older samples near the optimum. 3.3 Integration with CMA-ES In AFN-CMA-ES, the surrogate is used instead of costly black-box function eval- uations within the evolution control loop. CMA-ES is used to generate candidate solutions from its sampling distribution. AFN-CMA-ES uses generation based evolution control. At each generation evaluating the true black-box function, CMA-ES generates a population of λ candidate solutions, where λ denotes the population size, and all candidates are evaluated using the true objective func- tion. These evaluated samples are added to the archive and are used to update both the CMA-ES distribution and the ANN ensemble surrogate. The model life L g denotes the number of consecutive surrogate-only CMA-ES generations performed after the generation g evaluating the true black-box func- tion. During these surrogate only generations, CMA-ES again generates pop- ulations of λ candidates, but their fitness values are estimated by the ANN ensemble. The ensemble predictive mean ˆμ(x) is used to rank the candidate so- lutions, and the predicted values are converted into rank normalized surrogate fitness scores. These surrogate ranked scores are then passed to CMA-ES for up- dating the mean, step size, and covariance matrix without spending additional true objective function evaluations. This mechanism reduces the proportion of true evaluated CMA-ES genera- tions while allowing additional CMA-ES adaptation steps under the same true evaluation budget. The model life L is either fixed or adaptively selected accord- ing to Kendall’s rank correlation between the previous surrogate predictions and the newly observed true objective values. When the surrogate ranking quality is high, more surrogate only generations are allowed; when the ranking quality is low, the model life is reduced. Restart steps are used by AFN-CMA-ES when the CMA-ES internal stopping criterion is met or when stagnation is detected over true-evaluated generations. In such cases, the optimizer is reinitialized either near the best evaluated solution or from a random point in the search domain, depending on the restart count. 6M.K.I. Zim and M. Holeňa This enables the algorithm to continue the search when premature convergence occurs. Then CMA-ES updates its covariance matrix by utilizing true evalua- tions during true-evaluated generations and surrogate ranked predictions during surrogate-only generations. This mechanism balances exploration and exploita- tion while reducing the number of costly true objective-function evaluations. 3.4 Data Collection and Sampling This first step consists of creating a representative initial sample of the input- output pairs of the Comparing Continuous Optimizers (COCO) benchmark suite, denoted D 0 = (x i ,y i ) N 0 i=1 , from the original, costly objective function f(x). In this case, x i is an input vector in the design space X ⊆R d and y i = f(x i ) is the scalar objective-function value. To ensure these N 0 data points are well distributed throughout the design space and capture its important features with minimal costly evaluations, in this study, LHS was used to generate the initial sample set because it provides space filling coverage of the design domain with a limited number of expensive function evaluations, the initial sample size was set to N 0 = 20d, subject to the available evaluation budget. This dimension-scaled choice provides sufficient initial coverage while preserving most of the 250d eval- uation budget for optimization. Newly true evaluated CMA-ES samples were added to the archive during optimization. 3.5 Data Pre-processing and Standardization Input/Output Standardization: Before training, the input features X are stan- dardized to a zero mean and unit standard deviation, and objective-function values y are first converted into rank-normalized target values. These rank nor- malized target values are also standardized during neural network training. The predicted values are used to rank candidate solutions during surrogate only CMA-ES generations. This prevents the variables of greater magnitude from dominating the surrogate fit over the various scales of BBOB functions. Regularized Ensemble Training: During surrogate retraining, the ANN en- semble is trained on the local archive subset selected around the current CMA- ES mean. Training uses standardized inputs and rank-normalized targets, with a loss function combining a negative-log-likelihood term and a mean-squared-error term, rank-normalized targets mean that the raw objective-function values are replaced by their ranks within the current training archive and then scaled be- fore ANN training. Anchored regularization is applied by penalizing deviations from the initial network parameters. This helps avoid overfitting small archives at the beginning of the optimization run. 3.6 Ensemble of ANN Surrogates The AFN framework uses an Ensemble of MLP networks as the surrogate models. 1. MLP Architecture: Black Box Optimization7 Input layer: d neurons (problem dimension) Hidden layer 1: 64 neurons with rectified linear unit (ReLU) activation Hidden layer 2: 32 neurons with ReLU activation Output layer: 1 neuron with linear activation This compact 64–32 architecture balances nonlinear capacity, overfitting risk, and repeated retraining cost in the small-data setting. 2. Training Procedure: The network is trained using the standardized rank-normalized target values. Each network predicts both a mean value μ i and a log-variance value logσ 2 i . The loss function combines a negative log-likelihood (NLL) term and a mean squared error (MSE) term: L(W,B) = 1 N N X i=1 1 2 " logσ 2 i + (y norm i − μ i ) 2 exp (logσ 2 i ) # + 1 2N N X i=1 (μ i − y norm i ) 2 . (1) where W and B denote the ANN weights and biases, respectively, and μ i − y norm i is the prediction error in the standardized rank-normalized target space for sample i. The loss L(W,B) combines NLL and MSE. NLL estimates each network’s predictive variance, while MSE stabilizes the mean prediction used for can- didate ranking. This variance complements ensemble disagreement, which represents epistemic uncertainty across networks. To prevent overfitting, anchored regularization is applied: L reg (W,B) = L(W,B) + λ a X p∈W,B (p− p 0 ) 2 (2) where L reg (W,B) is the regularized loss, L(W,B) is the original loss, λ a con- trols the strength of anchored regularization, p denotes a trainable network parameter, and p 0 denotes its initial value. Training uses the AdamW optimizer, which applies Adam with decoupled weight decay. The basic gradient-based weight update can be written as: w new kj = w old kj − η ∂L reg ∂w kj (3) where w new kj and w old kj are the updated and previous weights, respectively. Each network uses a bootstrap sample drawn from the local archive, and warm-starting reduces retraining cost while preserving learned information. The hyperparameters were selected to optimize for accuracy, stability, and cost. 3. Ensemble Construction: Rather than a single MLP, we train M = 5 independent ANNs to form an Ensemble, which balances prediction diversity and computational cost. For the query point x, the ensemble provides predictive mean and predictive variance (Exploration): ˆσ 2 (x) = 1 M M X i=1 h ˆ f i (x)− ˆμ(x) i 2 (4) 8M.K.I. Zim and M. Holeňa The variance ˆσ 2 (x) provides uncertainty information, this uncertainty is not used to trigger individual true objective function evaluation; instead, true evaluations are controlled at the generation level, and the ensemble predictive mean is used to rank candidate solutions during surrogate only CMA-ES generations. The ensemble-based uncertainty quantification (UQ) approach was chosen for its simple integration, parallel training, and efficient warm- start retraining. while being simpler than posterior inference in Bayesian neural networks. Prediction disagreement provides a practical estimate of epistemic uncertainty at moderate computational cost. 3.7 Adaptive Sampling Strategy The AFN uses a model life based decision rule for surrogate-only CMA-ES gen- erations: L g = 4, if τ g ≥ 0.75 (high ranking quality → more surrogate-only generations) 3, if 0.60≤ τ g < 0.75 1, if 0.45≤ τ g < 0.60 0, if τ g < 0.45 (low ranking quality → true-evaluated generation) (5) where the threshold is based on Kendall’s rank correlation τ g , which compares the previous surrogate predictions with the newly observed true objective values. The thresholds 0.45, 0.60, and 0.75 have been chosen to indicate weak, moder- ate, and strong ranking agreement, respectively, which control the number of permitted surrogate-only generations. 4 Experimental Setup 4.1 Benchmark Suite The proposed AFN-CMA-ES algorithm was evaluated on the COCO/BBOB (Black-Box Optimization Benchmarking) test suite, the standard noiseless bench- mark for continuous optimization algorithms. The test suite comprises 24 noise- less test functions (f1− f24) covering diverse problem characteristics. Unimodal separable functions (f1− f5): Sphere, Ellipsoid separable, Rast- rigin separable, Büche-Rastrigin, and Linear slope. Unimodal low-conditioned functions (f6−f9): Attractive sector, Step-ellipsoid, Rosenbrock original, and Rosenbrock rotated. Unimodal high-conditioned functions (f10− f14): Ellipsoid, Discus, Bent cigar, Sharp ridge, and Sum of different powers. Multimodal functions (f15− f24): Rastrigin, Weierstrass, Schaffer F7 con- dition 10, Schaffer F7 condition 1000, Griewank-Rosenbrock, Schwefel, Gal- lagher 101 peaks, Gallagher 21 peaks, Katsuura, and Lunacek bi-Rastrigin. Black Box Optimization9 4.2 Test Dimensions Experiments were conducted across 4 problem dimensions: 2D, 3D, 5D and 10D to assess the scalability of the proposed method. 4.3 Compared Algorithms AFN-CMA-ES was compared against four state-of-the-art CMA-ES variants: 1. CMA-ES: the standard covariance matrix adaptation evolution strategy [7]. 2. LQ-CMA-ES: CMA-ES with linear-quadratic surrogate model [8]. 3. DTS-CMA-ES: doubly trained surrogate CMA-ES [5]. 4. LMM-CMA-ES: CMA-ES with local meta-model [9]. The algorithms were selected to represent standard, global-surrogate, ranking- based, and local-surrogate CMA-ES approaches. The baseline results for CMA- ES, LQ-CMA-ES, DTS-CMA-ES, and LMM-CMA-ES were obtained from the publicly available COCO/BBOB data archive. These results were archived and compared with AFN-CMA-ES using the standard COCO/BBOB evaluation pro- tocol, of 15 independent runs of each combination of functions and dimensions. 4.4 Experimental Configuration The evaluation of AFN-CMA-ES was performed on 15 instances in each consid- ered dimension for each function from the COCO/BBOB benchmark set. For each function dimension target setting, run lengths from the 15 BBOB instances were bootstrapped by resampling runs with replacement. Table 1 reports dis- persion values in parentheses, computed as half the difference between the 90th and 10th percentiles of the bootstrapped run-length distribution. The symbol∞ indicates that the target was not reached within the evaluation budget and the subscript 2500 denotes the median number of function evaluations conducted when the final target was not reached; in 10D, it corresponds to the budget 250d = 2500. The baseline results for CMA-ES, LQ-CMA-ES, DTS-CMA-ES, and LMM-CMA-ES were obtained from COCO using the same evaluation setting of 15 instances. The maximum evaluation budget was based on COCO/BBOB and scaled with the problem dimension. For AFN-CMA-ES, the surrogate model consisted of 5 MLPs with 2 hidden layers, where the hidden-layer sizes were dimension-dependent. Features and rank-normalized targets were standardized before training. The ANN ensemble was trained using the AdamW optimizer with a learning rate of η = 0.001 and anchored regularization. The surrogate was retrained using a local archive subset of at most 250 samples selected ac- cording to distance from the current CMA-ES mean. Evaluation budget: The number of evaluations of the true objective function in the evaluation budget B = 250d, where d is the problem dimension. The factor 250d corresponds to the maximum true-evaluation budget scaled by the problem dimension and includes the initial LHS sampling, true evaluated CMA- ES generations, surrogate retraining and optimization. Therefore, comparisons 10M.K.I. Zim and M. Holeňa used the COCO/BBOB noiseless archive ranges: 24 functions (f 1 –f 24 ), dimen- sions d∈2, 3, 5, 10, target precisions 10 −8 , and baselines were compared where corresponding COCO/BBOB archive data were available. 4.5 Performance Metrics The assessment of performance involved COCO standard measures: Runtime distributions: empirical cumulative distribution functions (ECDFs), which are used to describe the fraction of targets that were achieved after a given number of evaluations of the function. Target precisions: Success rates were reported for different target precisions according to the standard COCO/BBOB post-processing targets up to 1e−8. Statistical significance: Statistical comparison can be performed using the COCO/BBOB post processing results to assess whether the observed differences between algorithms were statistically significant. For true values y i and surrogate predictions ˆy i , standard and rank-normalized RMSE are defined as RMSE = v u u t 1 n n X i=1 (y i − ˆy i ) 2 , r i = R i − 1 N − 1 , RMSE rank = v u u t 1 n n X i=1 (r i − ˆr i ) 2 , (6) where R i is the rank of y i among N archive values and ˆr i is the prediction on the same normalized-rank scale. Standard RMSE measures error on the origi- nal scale, whereas rank-normalized RMSE measures dimensionless error on the normalized-rank scale. 5 Results and Discussion The overall performance of the compared algorithms across all BBOB functions and target precisions is summarized by the ECDF plots in Fig. 1. To complement the ECDF analysis, Fig. 2 shows the evolution of rank-normalized RMSE across five BBOB function groups and four dimensions. 5.1 Comparison with Baseline Algorithms AFN-CMA-ES vs CMA-ES: AFN-CMA-ES achieves performance comparable to the standard CMA-ES baseline on selected unimodal, well-conditioned func- tions. Surrogate-assisted black-box optimizers are designed to use fewer true objective-function evaluations by replacing part of the evaluation process with Black Box Optimization11 surrogate predictions; they are expected to achieve better optimization progress than the original optimizer under the same true-evaluation budget. In this con- text, comparable performance to CMA-ES can be interpreted as AFN-CMA-ES not behaving in the desired way. In particular, on highly multimodal functions (f 15 –f 24 ), AFN-CMA-ES performs significantly worse: at 5D, f 15 yields 0/15 success for AFN-CMA-ES versus 15/15 for CMA-ES, and at 10D AFN-CMA- ES exhausts its budget (∞) on f 15 , f 16 , f 17 , and f 18 . AFN-CMA-ES vs LQ-CMA-ES: LQ-CMA-ES demonstrates higher perfor- mance on most dimensions of separable, low-conditioned functions. On f 2 , LQ- CMA-ES achieves expected running time (ERT) ratios ( algorithm’s ERT divided by the COCO/BBOB reference ERT; lower is better, and ∞ means failure) of 1.7–2.7 (2D), compared to AFN-CMA-ES’s 10–17 with 11/15 success. AFN-CMA-ES vs DTS-CMA-ES: There is notable variation depending on function type. On Rosenbrock (f 8 ) across 2D–10D, DTS-CMA-ES performs bet- ter than AFN-CMA-ES. AFN-CMA-ES reaches 7/15 success at 2D, 4/15 at 3D, and 0/15 at 5D and 10D. AFN-CMA-ES vs LMM-CMA-ES: LMM-CMA-ES has a clear advantage on multimodal functions in lower dimensions. On f 15 at 2D, LMM-CMA-ES achieves an expected running time ERT ratio of 0.70 with 10 successful trials, while AFN-CMA-ES reaches only 3/15 success. 5.2 Scalability Analysis Scalability analysis in this case considers expected running time ERT ratios aggregated across all 24 BBOB functions and examines the dimensional scaling behaviour. The dimensional scaling behaviour of the compared algorithms is shown in Fig. 3. On the Sphere function (f 1 ), AFN-CMA-ES expected running time ERT ratios increase from 34 in 2D to 49 in 10D, with 15/15 success across the available dimensions. In contrast, LQ-CMA-ES ratios remain relatively stable from 4.5 to 6.2 across 2D to 10D, indicating that its linear quadratic surrogate scales efficiently on separable functions, where the additive problem structure aligns well with the surrogate’s inductive bias. The scaling gap becomes larger on non-separable functions (f 8 ,f 9 ), which are well-conditioned. The value of AFN-CMA-ES at 10D is ∞ (f 8 ) and ∞ (f 9 ) versus 1.4 and 1.5 for DTS-CMA-ES, which represents a larger relative difference than on separable functions. The most significant scalability failure is on highly multimodal functions (f 15 –f 24 ) in higher dimensions. At 10D, AFN-CMA-ES exhausts its budget (∞) on f 15 , f 16 , f 17 , f 18 , f 19 , f 20 , f 23 , and f 24 . Predictions were compared with true objective values before retraining. The root mean squared error (RMSE) measured prediction error, while Kendall’s τ measured ranking accuracy. AFN- CMA-ES may perform poorly on multimodal and higher-dimensional problems because the limited local archive becomes sparse, the MLP may not capture complex landscapes, and overlapping bootstrap samples may reduce ensemble reliability. 12M.K.I. Zim and M. Holeňa 050100150200250 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of function,target pairs AFN-CMA-E CMA-ES DTS-CMA-E LQ-CMA-ES LMM-CMA-E best 2009 bbob f1-f24, 2-D 51 targets: 100..1e-08 15 f1 , 14 f1 , 13 f2 instances v2.8.6 best 2009 AFN-CMA-ES DTS-CMA-ES LMM-CMA-ES LQ-CMA-ES CMA-ES 050100150200250 0.0 0.2 0.4 0.6 0.8 1.0 AFN-CMA-E CMA-ES DTS-CMA-E LMM-CMA-E LQ-CMA-ES best 2009 bbob f1-f24, 3-D 51 targets: 100..1e-08 15 f1 , 14 f1 , 13 f2 instances v2.8.6 best 2009 AFN-CMA-ES DTS-CMA-ES LMM-CMA-ES LQ-CMA-ES CMA-ES 050100150200250 0.0 0.2 0.4 0.6 0.8 1.0 AFN-CMA-E CMA-ES DTS-CMA-E best 2009 LMM-CMA-E LQ-CMA-ES bbob f1-f24, 5-D 51 targets: 100..1e-08 15 f1 , 14 f1 , 13 f2 instances v2.8.6 best 2009 AFN-CMA-ES DTS-CMA-ES LMM-CMA-ES LQ-CMA-ES CMA-ES 050100150200250 FEvals / DIM 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of function,target pairs AFN-CMA-E CMA-ES best 2009 DTS-CMA-E LMM-CMA-E LQ-CMA-ES bbob f1-f24, 10-D 51 targets: 100..1e-08 15 f1 , 14 f1 , 13 f2 instances v2.8.6 best 2009 AFN-CMA-ES DTS-CMA-ES LMM-CMA-ES LQ-CMA-ES CMA-ES Fig. 1. Empirical cumulative distribution functions (ECDF) plot analysis of overall performance in four dimensions of the problem. This figure shows ECDFs that have been cumulated over all the 24 noiseless BBOB functions. The subplots concern the following dimensions: (a) 2D, (b) 3D, (c) 5D, and (d) 10D. The ECDF plots depict all target precision values obtained relative to the number of function evaluations and provide an overall comparison of an algorithm’s performance across all dimensions. The results obtained in this case are combined across all functions and targets un- less no algorithm among the compared ones achieves the target within the specified evaluation budget. Pairwise Wilcoxon rank-sum tests were used to assess statistical significance between AFN-CMA-ES and the baseline algorithms. For each function- dimension setting, the null hypothesis was that AFN-CMA-ES and the corresponding baseline algorithm had the same distribution of running-time performance values. Sta- tistical significance was assessed at p < 0.05 after Holm correction over the 24 BBOB functions. 5.3 Statistical Analysis Wilcoxon rank-sum tests (p < 0.05, Holm corrected for 24 functions) confirm LQ-CMA-ES makes statistically significant improvements over all other avail- able algorithms on separable functions across most dimensions. LQ-CMA-ES achieves high significance levels on f 1 , f 2 , f 5 , and f 10 across the available di- mensions where corresponding archive data are available, demonstrating strong superiority on these function classes. AFN-CMA-ES does not achieve statisti- cally significant improvements over any baseline algorithm on any function di- mension combination in the current results. On some individual functions, AFN- CMA-ES shows comparable performance to selected surrogate-assisted baselines. Black Box Optimization13 Fig. 2. Evolution of rank-normalized surrogate RMSE with function evaluations per dimension across five BBOB function groups in 2D, 3D, 5D, and 10D. Curves show median RMSE over functions and 15 instances, with shaded interquartile ranges. RMSE generally increases early and then stabilizes, with lower late-stage errors for low and moderately conditioned functions in 5D and 10D. AFN-CMA-ES performs worst relative to the baselines on multimodal functions f 15 –f 24 at dimensions ≥ 5D. Since these failures reduce the number of target values reached within the evaluation budget, they lower the overall aggregate ECDF performance. This implies that the global MLP ensemble surrogate, al- though based on ranking-based generation control, lacks the selectivity benefit to offset its computational cost in intricate landscapes. Further studies on se- lecting between local and global surrogate models based on observed landscape structure should be conducted in the future. 6 Conclusion Surrogate assisted black-box optimization addresses the small data challenge that arises when each objective function evaluation is computationally costly, limiting the total number of evaluations available. In this setting, the surrogate model must approximate the objective function from a restricted archive of eval- uated samples, making data efficiency essential. The experimental assessment of the proposed AFN-CMA-ES on the COCO/BBOB benchmark suite shows that the proposed ANN based surrogate assisted framework can reduce the re- liance on direct objective function evaluations in selected problem classes, but 14M.K.I. Zim and M. Holeňa 235102040 0 1 2 log10(# f-evals / dimension) 14, 15 instances target Df: 1e-8 v2.8.6 1 Sphere AFN-CMA-ES DTS-CMA-ES LMM-CMA-ES LQ-CMA-ES CMA-ES 235102040 0 1 2 3 13, 15 instances target Df: 1e-8 v2.8.6 2 Ellipsoid separable 235102040 0 1 2 3 4 5 13, 15 instances target Df: 1e-8 v2.8.6 3 Rastrigin separable 235102040 1 3 5 13, 15 instances target Df: 1e-8 v2.8.6 4 Skew Rastrigin-Bueche 235102040 0 1 2 13, 15 instances target Df: 1e-8 v2.8.6 5 Linear slope 235102040 0 1 2 3 13, 15 instances target Df: 1e-8 v2.8.6 6 Attractive sector 235102040 0 1 2 3 log10(# f-evals / dimension) 13, 15 instances target Df: 1e-8 v2.8.6 7 Step-ellipsoid 235102040 0 1 2 3 13, 15 instances target Df: 1e-8 v2.8.6 8 Rosenbrock original 235102040 0 1 2 3 13, 15 instances target Df: 1e-8 v2.8.6 9 Rosenbrock rotated 235102040 0 1 2 3 14, 15 instances target Df: 1e-8 v2.8.6 10 Ellipsoid 235102040 0 1 2 3 14, 15 instances target Df: 1e-8 v2.8.6 11 Discus 235102040 0 1 2 3 14, 15 instances target Df: 1e-8 v2.8.6 12 Bent cigar 235102040 0 1 2 3 4 log10(# f-evals / dimension) 14, 15 instances target Df: 1e-8 v2.8.6 13 Sharp ridge 235102040 0 1 2 3 14, 15 instances target Df: 1e-8 v2.8.6 14 Sum of different powers 235102040 0 1 2 3 4 13, 15 instances target Df: 1e-8 v2.8.6 15 Rastrigin 235102040 0 1 2 3 4 13, 15 instances target Df: 1e-8 v2.8.6 16 Weierstrass 235102040 0 1 2 3 13, 15 instances target Df: 1e-8 v2.8.6 17 Schaffer F7, condition 10 235102040 0 1 2 3 4 13, 15 instances target Df: 1e-8 v2.8.6 18 Schaffer F7, condition 1000 235102040 dimension 1 3 5 log10(# f-evals / dimension) 13, 15 instances target Df: 1e-8 v2.8.6 19 Griewank-Rosenbrock F8F2 235102040 dimension 0 1 2 3 4 5 13, 15 instances target Df: 1e-8 v2.8.6 20 Schwefel x*sin(x) 235102040 dimension 0 1 2 3 4 5 13, 15 instances target Df: 1e-8 v2.8.6 21 Gallagher 101 peaks 235102040 dimension 0 1 2 3 4 5 13, 15 instances target Df: 1e-8 v2.8.6 22 Gallagher 21 peaks 235102040 dimension 0 1 2 3 4 5 13, 15 instances target Df: 1e-8 v2.8.6 23 Katsuura 235102040 dimension 1 3 5 7 13, 15 instances target Df: 1e-8 v2.8.6 24 Lunacek bi-Rastrigin AFN-CMA-ES DTS-CMA-ES LMM-CMA-ES LQ-CMA-ES CMA-ES Fig. 3. ERT in number of f-evaluations as log 10 of the value divided by dimension for target function value 10 −8 versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Legend: ◦: AFN-CMA-ES, ⋄: DTS-CMA- ES, ⋆: LMM-CMA-ES, +: LQ-CMA-ES, ∇: CMA-ES its performance remains strongly dependent on the objective function landscape structure. The results indicate that AFN-CMA-ES achieves comparable perfor- mance to selected surrogate assisted CMA-ES variants on some unimodal well conditioned functions (f 1 –f 9 ), particularly at lower dimensions, but faces signif- icant challenges on highly multimodal functions (f 15 –f 24 ) in dimensions d ≥ 5, Black Box Optimization15 Table 1. Expected runtime (ERT) ratios in 10-D, measured as the number of func- tion evaluations divided by the reference ERT, where reference ERT denotes the best BBOB-2009 ERT from the COCO/BBOB archive for the same function, dimension, and target precision. Values in parentheses denote the dispersion measure, computed as half difference between 10% and 90% of bootstrapped run lengths. The column #succ reports the number of trials reaching f opt + 10 −8 . Function1e11e01e−11e−21e−31e−51e−7 #succ 10-D f1 ref. ERT2223232323232315/15 AFN-CMA-ES 10 (0.8) 14 (1)18 (2)22 (1)27 (3) 38 (5) 49 (8) 15/15 f2 ref. ERT187190191191193194195 15/15 AFN-CMA-ES 13 (9) 18 (12)93 (78)94 (82) 93 (97) 94 (93) 94 (103) 1/15 f3 ref. ERT1739360036093636364236463651 15/15 AFN-CMA-ES ∞ ∞ ∞ ∞ ∞2500 0/15 f4 ref. ERT22343626366036953707374428767 12/15 AFN-CMA-ES ∞ ∞ ∞ ∞ ∞2500 0/15 f5 ref. ERT2020202020202015/15 AFN-CMA-ES 14 (1.0) 17 (0.9)19 (1)21 (2)23 (3) 28 (4) 38 (4) 15/15 f6 ref. ERT4126238261039129218412370 15/15 AFN-CMA-ES 1.6 (0.3) 1.7(0.2)1.7(0.3)1.7 (0.3) 1.8 (0.3) 20 (25) ∞2500 0/15 f7 ref. ERT172161141955099514151415389 15/15 AFN-CMA-ES 2.2 (0.5) 1.7 (2)2.9 (3)7.3 (7) 7.2 (7) 7.2 (6) ∞2500 0/15 f8 ref. ERT32692111141217126713151343 15/15 AFN-CMA-ES 2.1 (0.8) 20 (30) ∞ ∞ ∞ ∞2500 0/15 f9 ref. ERT200648857993106511381185 15/15 AFN-CMA-ES 3.5(2) 6.4(5)14 (17)19 (17) 18 (14) ∞ ∞2500 0/15 f10 ref. ERT1835217224552728280245434739 15/15 AFN-CMA-ES 3.9(4) ∞ ∞ ∞ ∞2500 0/15 f11 ref. ERT266104126022954333840924843 15/15 AFN-CMA-ES 13 (10) 6.9 (6)4.7 (5)6.3 (5) 11 (12) ∞ ∞2500 0/15 f12 ref. ERT51589612401390156936605154 15/15 AFN-CMA-ES 2.2 (1) 12 (17)15 (25) ∞ ∞ ∞ ∞2500 0/15 f13 ref. ERT3875967971014458762087779 15/15 AFN-CMA-ES 1.7 (0.4) 5.3 (4)7.0 (4)12 (16) ∞ ∞ ∞2500 0/15 f14 ref. ERT37981332053926874305 15/15 AFN-CMA-ES 5.4(2) 3.6 (0.5) 3.7 (0.4) 3.5 (0.9) 2.9 (0.7) 3.0 (0.4) ∞2500 0/15 f15 ref. ERT47743924673643746697579077814 79834 12/15 AFN-CMA-ES 3.7 (4) ∞ ∞ ∞ ∞2500 0/15 f16 ref. ERT425702915779456695115165798 71570 15/15 AFN-CMA-ES 2.2 (0.7) 0.25(0.2) 0.54(0.4) 0.81 (0.8) ∞ ∞ ∞2500 0/15 f17 ref. ERT2642922036329985120190 26503 15/15 AFN-CMA-ES 4.7 (4) 1.4(0.4) 0.57 (0.1)↑ 1 0.52(0.4) 0.89(1.0) ∞ ∞2500 0/15 f18 ref. ERT2388367012159282753637234 42708 15/15 AFN-CMA-ES 1.5 (0.5) 1.8 (1)5.3 (6) ∞ ∞ ∞ ∞2500 0/15 f19 ref. ERT11106099.8e51.4e61.4e61.4e6 15/15 AFN-CMA-ES 216 (67) ∞ ∞ ∞ ∞2500 0/15 f20 ref. ERT32154265.5e55.7e55.7e55.8e55.9e5 15/15 AFN-CMA-ES 7.9 (0.6) ∞ ∞ ∞ ∞2500 0/15 f21 ref. ERT1302236439244874618507411329 15/15 AFN-CMA-ES 7.2 (10) 4.7 (3)8.1 (14)7.9 (8) 7.7 (9) 7.0 (9) 3.1 (3) 1/15 f22 ref. ERT982839635366206798829610351 15/15 AFN-CMA-ES 7.3 (13) 1.9 (3)5.6 (6)5.4 (5) 5.3 (7) 4.4 (6) 3.5 (3) 1/15 f23 ref. ERT3.0915164251.8e52.0e52.1e52.1e5 15/15 AFN-CMA-ES 1.6 (2) ∞ ∞ ∞ ∞2500 0/15 f24 ref. ERT987611.0e67.5e77.5e77.5e77.5e77.5e7 1/15 AFN-CMA-ES ∞ ∞ ∞ ∞ ∞2500 0/15 where it often exhausts its evaluation budget without reaching target precision. The superior performance of LQ-CMA-ES on separable functions (f 1 –f 5 ) sug- gests that the global MLP surrogate does not exploit additive structure, while the superior performance of LMM-CMA-ES on selected multimodal functions suggests that local meta models are more effective than global ones on complex 16M.K.I. Zim and M. Holeňa landscapes. The overall analysis of 24 functions in 4 dimensions shows that sur- rogate model selection is essential for optimization performance, and that a one size fits all MLP ensemble has inherent limitations in adapting to diverse prob- lem structures. Future directions include adaptive surrogate selection switching between global and local models in response to detected landscape properties and improved exploration strategies for multimodal functions. Acknowledgements The research reported in this paper has been supported by the Czech Academy of Science grant 231523 and Martin Holeňa’s research was partially supported by the German Research Foundation (DFG) project funding no. 441926934. References 1. N. Hansen, S. D. Müller, and P. Koumoutsakos, “Reducing the time com- plexity of the derandomized evolution strategy with covariance matrix adapta- tion (CMA-ES),”Evolutionary Computation, vol. 11, no. 1, p. 1–18, 2003. doi: 10.1162/106365603321828970. 2. T. Marty, Y. Semet, A. Auger, S. Héron, and N. Hansen, “Benchmarking CMA-ES with Basic Integer Handling on a Mixed-Integer Test Problem Suite,” Proceedings of GECCO Companion, p. 1628–1635, 2023. 3. L. Bajer, Z. Pitra, and M. Holeňa, “Benchmarking Gaussian Processes and Random Forests Surrogate Models on the BBOB Noiseless Testbed,” Proceedings of GECCO Companion, p. 1143–1150, 2015. 4. L. Bajer, Z. Pitra, J. Repický, and M. Holeňa, “Gaussian Process Surrogate Models for the CMA Evolution Strategy,” Evolutionary Computation, 2019. 5. Z. Pitra, L. Bajer, J. Repický, and M. Holeňa, “Comparison of Ordinal and Metric Gaussian Process Regression as Surrogate Models for CMA Evolution Strategy,” Proceedings of the Genetic and Evolutionary Computation Conference Companion, p. 1764–1771, 2017. doi: 10.1145/3067695.3084206. 6. M. Holeňa and J. Koza, “Suitability of Modern Neural Networks for Active and Transfer Learning in Surrogate-Assisted Black-Box Optimization,” IAL@ECML- PKDD’24: 8th International Workshop and Tutorial on Interactive Adaptive Learn- ing, Vilnius, Lithuania, Sept. 9, 2024, CEUR Workshop Proceedings, vol. 3770, p. 47–67, 2024. 7. N. Hansen, “Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed,” Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, p. 2389–2396, 2009. doi: 10.1145/1570256.1570333. 8. N. Hansen, A global surrogate assisted CMA-ES, Proceedings of the Ge- netic and Evolutionary Computation Conference, p. 664–672, 2019. doi: 10.1145/3321707.3321842. 9. A. Auger, D. Brockhoff, and N. Hansen, “Benchmarking the Local Metamodel CMA- ES on the Noiseless BBOB’2013 Test Bed,” GECCO Companion, Amsterdam, The Netherlands, 2013.