Paper deep dive
Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity
Heng Zhang, Haotian Xiang, Qin Lu, Konstantinos D. Polyzos, Tara Javidi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Hyperparameter selection remains a key challenge in Bayesian optimization (BO) and Bayesian active learning (AL), as model misspecification can lead to suboptimal performance, while more accurate fully Bayesian treatments typically rely on computationally expensive MCMC sampling. This paper proposes a unified framework, KENDO (Kernel ENsemble Disagreement-aware Operator), that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies. The central idea is to replace hyperparameter sampling with a kernel ensemble and adaptive Bayesian weighting, combined with disagreement-aware acquisition strategies. Within this unified framework, we instantiate KENDO-BO for BO and KENDO-AL for Bayesian AL, demonstrating that both arise from a common self-correcting mechanism with task-specific acquisition objectives. We further extend the approach to multi-objective optimization via random scalarization that preserves the single-optimizer conditioning structure. Thorough numerical tests on synthetic and real-world benchmarks across single-objective optimization, multi-objective optimization, and active learning demonstrate that (i) KENDO-BO achieves competitive or superior optimization performance compared to state-of-the-art methods while reducing computational overhead by up to $5\times$ and (ii) KENDO-AL achieves superior predictive calibration over MCMC-based active learning baselines with up to $27\times$ speedup.
Tags
Links
- Source: https://arxiv.org/abs/2608.24721v1
- Canonical: https://arxiv.org/abs/2608.24721v1
Trouble viewing inline? Open PDF directly →
Full Text
66,537 characters extracted from source content.
Expand or collapse full text
Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity Heng Zhang Affiliation: University of Georgia, Athens, GA, USA Email: hz39726@uga.edu Haotian Xiang Affiliation: University of Georgia, Athens, GA, USA Email: haotian.xiang@uga.edu Qin Lu Affiliation: University of Georgia, Athens, GA, USA Email: qin.lu@uga.edu Konstantinos D. Polyzos Affiliation: University of California San Diego, La Jolla, CA, USA Email: kpolyzos@ucsd.edu Tara Javidi Affiliation: University of California San Diego, La Jolla, CA, USA Email: tjavidi@ucsd.edu Abstract Hyperparameter selection remains a key challenge in Bayesian optimization (BO) and Bayesian active learning (AL), as model misspecification can lead to suboptimal performance, while more accurate fully Bayesian treatments typically rely on computationally expensive MCMC sampling. This paper proposes a unified framework, KENDO (Kernel ENsemble Disagreement-aware Operator), that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies. The central idea is to replace hyperparameter sampling with a kernel ensemble and adaptive Bayesian weighting, combined with disagreement-aware acquisition strategies. Within this unified framework, we instantiate KENDO-BO for BO and KENDO-AL for Bayesian AL, demonstrating that both arise from a common self-correcting mechanism with task-specific acquisition objectives. We further extend the approach to multi-objective optimization via random scalarization that preserves the single-optimizer conditioning structure. Thorough numerical tests on synthetic and real-world benchmarks across single-objective optimization, multi-objective optimization, and active learning demonstrate that (i) KENDO-BO achieves competitive or superior optimization performance compared to state-of-the-art methods while reducing computational overhead by up to 5×5× and (i) KENDO-AL achieves superior predictive calibration over MCMC-based active learning baselines with up to 27×27× speedup. 1 Introduction Bayesian optimization (BO) and active learning (BAL) have become standard approaches for optimizing expensive black-box functions [9, 33] and for learning unknown functions in an uncertainty-aware, label-efficient fashion respectively. By constructing a probabilistic surrogate model, typically a Gaussian process (GP), both paradigms quantify uncertainty to guide efficient sequential sampling. However, the performance of GP-based BO and AL critically depends on hyperparameter selection, including kernel choice and lengthscale parameters [34]. Traditional approaches handle hyperparameters through point estimation via marginal likelihood maximization [30] or full Bayesian treatment via MCMC sampling [34]. Point estimation is computationally efficient but vulnerable to model misspecification and overfitting, especially in early optimization stages. Conversely, MCMC-based fully Bayesian methods account for hyperparameter uncertainty but incur prohibitive computational costs, particularly when sampling from multimodal or high-dimensional posteriors [35]. Figure 1: Single-kernel BO on three benchmarks for PES method, reported as log10 _10 simple regret. No single kernel is universally optimal. Beyond hyperparameter tuning, the choice of kernel family itself is a critical source of misspecification. For example, in the BO paradigm, as Figure 1 illustrates, no single kernel is universally optimal for BO: the best-performing kernel differs across problems, RBF on Rosenbrock (2D), Matérn-1.5 on Hartmann (6D), and Matérn-2.5 on Borehole (8D) in log10 _10 simple regret, motivating an ensemble that adaptively weights multiple kernels based on observed data. Recent work have explored ensemble-based alternatives. Self-Correcting Bayesian Optimization (SCoreBO) [17] introduces an acquisition function that conditions on sampled optimizers to balance hyperparameter learning with optimization, but still relies on hyperparameter sampling. Ensemble Gaussian Processes (EGP) [23, 24, 28] maintain multiple GPs with different kernels and adaptively weight them via Bayesian model averaging, avoiding sampling entirely. Nonetheless, these approaches adaptively select a single GP model and couple it with standard acquisition strategies, thereby overlooking the benefits from interaction and disagreement among GP models during the acquisition phase. To the best of our knowledge, using EGPs as surrogate models in place of a fully Bayesian treatment, and integrating them with appropriate related acquisition strategies, remains largely unexplored. Our Contributions. We propose KENDO (Kernel ENsemble Disagreement-aware Operator), which unifies the strengths of ensemble modeling and self-correcting optimization: • We replace SCoreBO’s hyperparameter sampling with EGP’s explicit kernel ensemble, eliminating MCMC overhead while preserving uncertainty quantification through kernel diversity and adaptive weighting. • We derive a computationally efficient acquisition function that measures disagreement between marginal and optimizer-conditioned ensemble posteriors, balancing exploration in promising regions with model selection. • We extend the framework to multi-objective optimization via random scalarization, which preserves the single-optimizer conditioning structure of SCoreBO while covering the Pareto front through diverse scalarization directions, avoiding the computational burden of multi-objective solvers. • We introduce KENDO-AL, demonstrating that the ensemble mechanism generalizes beyond optimization to Bayesian active learning, achieving competitive results over MCMC-based approaches. • Through extensive experiments and ablation studies, we demonstrate that the ensemble mechanism achieves substantial computational savings (up to 5×5× for BO and 27×27× for active learning) without sacrificing optimization performance. 2 Related Work 2.1 Gaussian Processes and Hyperparameter Learning Gaussian processes provide a flexible non-parametric framework for probabilistic regression [30], in which the choice of kernel function and its hyperparameters fundamentally determines the inductive bias of the surrogate model. Standard practice optimizes these hyperparameters by maximizing the marginal likelihood [30], but the resulting point estimate neglects uncertainty and can produce overconfident predictions [34]. Fully Bayesian alternatives integrate over hyperparameter posteriors via MCMC [34] or variational inference [12, 37], and scalable variants such as slice sampling [26] and Hamiltonian Monte Carlo [15] improve robustness at the cost of substantial computational overhead where the surrogate must be refitted at every iteration. A complementary line of work addresses model uncertainty through ensembles rather than integration. Multi-kernel learning [6] constructs expressive composite kernels and deep kernel learning [40] parameterizes kernels via neural networks, while Ensemble Gaussian Processes [23] explicitly maintain a weighted collection of GPs with different kernels and update the weights via Bayesian model averaging based on predictive likelihood. This approach has shown promise but has not been integrated with modern BO or active learning acquisition functions that leverage optimizer information. 2.2 Acquisition Functions for Bayesian Optimization and Active Learning A unifying theme connecting modern acquisition functions for both Bayesian optimization and Bayesian active learning is the use of disagreement as the exploration signal, where queries are placed at points on which competing posterior beliefs disagree most strongly about the prediction. SCoreBO [17] instantiates this idea in BO by conditioning on sampled optimizers and computing the Hellinger distance between the marginal posterior and each optimizer-conditional posterior in promising regions, although it inherits the heavy computational cost of hyperparameter sampling. Other BO methods follow alternative routes that do not exploit disagreement, including GP-UCB with fixed confidence bounds [36], entropy-search variants that integrate over functions [11, 14, 39], and meta-learning approaches that transfer hyperparameter knowledge across related tasks [8]. The same disagreement principle has long driven Bayesian active learning (BAL), where the goal shifts from finding optima to minimizing global prediction error [32]. Bayesian Active Learning by Disagreement (BALD) [16, 19] maximizes the mutual information between predictions and hyperparameters, Bayesian Query-by-Committee (BQBC) [31] measures variance in posterior means across hyperparameter samples, and Statistical distance-based Active Learning (SAL) [17] generalizes BQBC using Hellinger and Wasserstein distances to capture the full distributional disagreement induced by hyperparameter uncertainty. SAL thus plays the same role in active learning as SCoreBO does in optimization, and both rely on MCMC sampling to quantify hyperparameter uncertainty, an overhead that directly motivates our ensemble-based alternative. 2.3 Multi-Objective Bayesian Optimization Multi-objective BO (MOBO) aims to approximate the Pareto front of multiple conflicting objectives [20], and standard approaches include expected hypervolume improvement [7, 3], ϵε-indicator-based methods [44], and information-theoretic acquisitions [13, 38, 1]. More recent work explores scalarization strategies [27] and uncertainty-aware acquisition functions [2], yet hyperparameter learning in multi-objective settings remains largely underexplored, as most existing methods either fix hyperparameters in advance or tune them independently for each objective without accounting for the coupled uncertainty across objectives. 3 Preliminaries Both BO and BAL are sequential decision-making paradigms that deal with a black-box function f, which has no analytic expression and is also expensive to evaluate. While BO aims to seek the optimizer of f, namely, ∗=argmax∈f() x_*= x∈ X \ \ \ f( x) (1) BAL focuses on learning a good mapping for the target function f for every ∈ x∈ X . Specifically, BO and BAL rely on a probabilistic surrogate model that guides the judicious selection of query points sequentially, which are implemented iteratively in the two steps, s1) obtain p(f()|t)p(f( x)|D_t) (t:=(τ,yτ)τ=1tD_t:=\( x_τ,y_τ)\_τ=1^t) based on a chosen surrogate model; and s2) select t+1=argmax∈α(|t) x_t+1\!=\! _ x∈ X\ α( x|D_t), where the acquisition function (AF) α, usually available in closed form, is designed based on p(f()|t)p(\!f(x)|D_t). For BO, the AF seeks to strike a balance between exploration and exploitation, while the AF in BAL is designed mainly for exploration. 3.1 GP-based surrogate models Capable of learning function mappings with well-calibrated uncertainty values, GPs are the de facto choice for surrogate models in BO and BAL [30, 9]. In the GP context, the learning function f is assumed to be drawn from a GP prior, f()∼(0,κ(,)′)f( x) GP(0, _ θ( x, x )), where κ _ θ is the positive-definite kernel function with learnable hyperparameters θ that measures pairwise similarity between any two distinct points x, and ′ x and is used to capture the covariance between function values f()f( x), f()′f( x ). Given tD_t, a typical process is to maximize the so-termed marginal likelihood p(t,)p( D_t; θ) to obtain a point estimate of θ. Going beyond the point estimation of the kernel hyperparameters, the fully Bayesian GP views θ as random and maintains a posterior pdf for θ via Bayes’ rule as: p(|t)∝p()p(t,)p( θ| D_t) p( θ)p( D_t; θ). Although accounting for overfitting, such a fully Bayesian treatment entails computationally prohibitive Markov Chain Monte Carlo (MCMC) sampling. The previous discussion adheres to GPs with a pre-selected kernel type. In practice though, this is a nontrivial design choice as different tasks may require different kernel functions. To automate this design, the ensemble (E) GP framework [23], given a prescribed kernel dictionary :=κ1,…,κMK:=\ _1,…, _M\, learns the function via a Gaussian mixture prior as: f()∼∑m=1Mw0m(0,κm(,′)),∑m=1Mw0m=1f( x) _m=1^Mw_0^m\, GP(0, _m( x, x )), _m=1^Mw_0^m=1 (2) where w0mw_0^m is the weight corresponding to the mmth GP model using kernel κmκ^m. After observing tD_t, the function posterior pdf is a GP mixture p(f()∣t)=∑m=1Mwtm(f(),μm,t(),σm,t2())p(f( x) _t)= _m=1^Mw_t^m\,N(f( x); _m,t( x), _m,t^2( x)) (3) where p(f()|m,t):=(f(),μm,t(),σm,t2())p(f(x)|m,D_t):=N(f( x); _m,t( x), _m,t^2( x)) is the posterior of GP model m with mean and variance given in closed-form [30]. The per-kernel weight wtm=ℙ(m|t)w_t^m=P(m| D_t) is proportional to the marginal likelihood wtm∝p(t|m;^m)w^m_t p( D_t|m; θ_m) (4) with ^m θ_m being the estimated hyperparameters for kernel κmκ^m by maximizing the marginal likelihood. 3.2 Disagreement-based AF Based on different design rules, a number of AFs have been designed for BO and BAL tailored to the chosen surrogate model. Most recently, leveraging the hyperparameter-induced disagreement conditioned on optimizer samples, a novel AF, termed as SCoreBO [17], is devised for the fully Bayesian GP model, defined as αSC()=,∗[d(p(y∣),p(y∣,∗,))] _SC(x)=E_ θ, [d(p(y_x ),p(y_x θ, ,D))] (5) where θ are hyperparameters sampled from p(∣)p( θ ), ∗=(∗,f∗) =(x^*,f^*) is a sampled optimizer, and d(⋅,⋅)d(·,·) is the Hellinger distance. This balances exploration in high-potential regions with hyperparameter learning. Unlike TS or EI that solely exploit the current posterior, αSC _SC also rewards queries that reduce hyperparameter uncertainty, which is critical when the surrogate is misspecified in early iterations. 4 Exploiting Kernel Diversity to Enhance BO and BAL While SCoreBO relies on the diversity of kernel hyperparameters, it still presumes a single pre-selected kernel type and uses computationally heavy MCMC sampling. To further account for the effect of kernel family selection and to eliminate the computational burden of MCMC, we propose to replace hyperparameter sampling with an explicit kernel ensemble. The core idea lies on the disagreement across structurally different kernels instead of the disagreement across hyperparameter samples of a single kernel. We develop this idea in three stages: first establishing the ensemble-based surrogate (Section 4.1), then deriving the optimizer-conditioned acquisition function (Section 4.2) for (multi-objective) black-box optimization, and finally extending to active learning (Section 4.3). 4.1 From Hyperparameter Sampling to Kernel Ensembles The central idea of KENDO is to replace SCoreBO’s continuous hyperparameter sampling with EGP’s discrete kernel ensemble. Each GP m∈1,…,Mm∈\1,…,M\ relies on a fixed kernel κm _m whose hyperparameters are fitted via marginal likelihood maximization, together with a dynamic weight wtmw^m_t updated via (4). This substitution offers two advantages: (i) it eliminates MCMC sampling overhead; and (i) it captures a qualitatively different and more consequential axis of model uncertainty, namely the choice of kernel family, without relying on a specific pre-defined kernel form. The Bayesian weight update requires only M predictive likelihood evaluations per iteration, compared to the (100)O(100)–(1000)O(1000) posterior samples typically needed for MCMC convergence. To make the ensemble tractable so as to be readily used as input to acquisition functions, we approximate the Gaussian mixture posterior (3) via moment matching as μ¯t() μ_t(x) =∑m=1Mwtmμm,t() = _m=1^Mw^m_t\, _m,t(x) (6) σ¯t2() σ^2_t(x) =∑m=1Mwtm[σm,t2()+(μm,t()−μ¯t())2] = _m=1^Mw^m_t [ _m,t^2( x)+( _m,t(x)- μ_t(x))^2 ] (7) where μ¯t() μ_t(x) is the ensemble mean for any x and σm,t() _m,t( x) is the associated variance. The variance expression decomposes into within-GP uncertainty (first term) and between-GP disagreement (second term), providing a single Gaussian summary that retains ensemble-level information. 4.2 Optimizer-Conditioned AF With a tractable Gaussian summary of the ensemble posterior, we will subsequently utilize this ensemble to decide where to query next. We adapt the key idea of conditioning on sampled optimizers and disagreement across hyperparameter samples of a single kernel, to the ensemble setting. The intuition is that if knowing the location of the optimum would cause different kernels to revise their predictions differently at x, then querying x is informative for both optimization and model selection. We define the acquisition function of KENDO-BO. For each kernel m and sample index n, we draw a function realization fm,nf_m,n from the posterior of GP m, locate its optimizer m,n∗=argmaxxfm,n()x^*_m,n= _xf_m,n(x) with value fm,n∗f^*_m,n, and condition GP m on (m,n∗,fm,n∗)(x^*_m,n,f^*_m,n) to obtain a conditional posterior (μm,t∗m,n(x),σm,t2,∗m,n())N(μ _m,n_m,t(x),σ^2, _m,n_m,t(x)). The acquisition function then aggregates the disagreement between the marginal ensemble posterior and each conditional posterior, yielding (see Appendix B for the full derivation) α()=∑m=1M∑n=1NwtmN⋅dm,n()α(x)= _m=1^M _n=1^N w^m_tN· d_m,n(x) (8) where dm,n()d_m,n(x) is the Hellinger distance between the moment-matched marginal ((6)–(7)) and the conditional posterior of kernel m given optimizer sample n. For two Gaussians, this admits a closed-form expression dm,n()=1−2σ¯t()σ∗m,nm,t()σ¯2t()+σ2,∗m,nm,t()exp[−(μ¯t()−μ∗m,nm,t())24(σ¯2t()+σ2,∗m,nm,t())]d_m,n(x)=1- 2 σ_t(x)σ _m,n_m,t(x) σ^2_t(x)+σ^2, _m,n_m,t(x) \! [- ( μ_t(x)-μ _m,n_m,t(x))^24( σ^2_t(x)+σ^2, _m,n_m,t(x)) ] (9) Intuitively, α()α(x) is large at locations where knowing the optimizer would substantially revise the ensemble’s prediction. Algorithm 1 summarizes the full procedure. Algorithm 1 KENDO-BO / KENDO-AL 0: Kernel dict. K, data 0D_0, budget T // For BO: #optimizers N; for AL: validation set V 1: for m∈m do 2: Fit αm _m via maxp(0|i=m,α) p(D_0|i=m,α); Init w0m=1/Mw_0^m=1/M 3: end for 4: for t=0,…,T−1t=0,…,T-1 do 5: Compute μ¯t(⋅),σ¯t2(⋅) μ_t(·), σ^2_t(·) via (6)–(7) 6: // BO branch: sample optimizers and condition 7: for m∈m ; n=1,…,Nn=1,…,N do 8: Sample fm,n∼p(f|m,t)f_m,n p(f|m,D_t); Find xm,n∗=argmaxfm,n()x^*_m,n= f_m,n(x) 9: Condition GP m on (xm,n∗,fm,n∗)(x^*_m,n,f^*_m,n) to get μm,n∗,σm,n2,∗μ^*_m,n,σ^2,*_m,n 10: end for 11: // BO: α()=∑m∑nwtmNdm,n()α(x)= _m _n w^m_tN\,d_m,n(x) via (8)–(9) 12: // AL (skip lines 7–9): α()=∑mwtm⋅dm()α(x)= _mw^m_t· d_m(x) via (10)–(11) 13: t+1=argmaxα()x_t+1= _xα(x) 14: Evaluate f(t+1)f(x_t+1); t+1=t∪(t+1,yt+1)D_t+1=D_t∪\(x_t+1,y_t+1)\ 15: for m∈m do 16: Update wt+1mw^m_t+1 via (4); Update GP posterior mean and variance via (23a)–(23b) (supplementary) 17: end for 18: // AL only: compute MLL on V 19: end for 20: return BO: argmax(,y)∈Ty _(x,y) _Ty AL: trained model with TD_T 4.3 Extensions to Active Learning The optimizer conditioning in KENDO-BO focuses on data collection near promising optima. For active learning, where the goal is global function approximation, we simply remove this conditioning. The KENDO-AL acquisition function measures the disagreement between the ensemble marginal and each component directly αSAL(x)=∑m=1Mwtm⋅d(p(yx∣t),p(yx∣i=m,t)) _SAL(x)= _m=1^Mw^m_t· d(p(y_x _t),\,p(y_x i=m,D_t)) (10) where the Hellinger distance between the moment-matched marginal and each kernel’s posterior takes the form dm(x)=1−2σ¯t(x)σm,t(x)σ¯t2(x)+σm,t2(x)exp[−(μ¯t(x)−μm,t(x))24(σ¯t2(x)+σm,t2(x))]d_m(x)=1- 2 σ_t(x) _m,t(x) σ^2_t(x)+σ^2_m,t(x) \! [- ( μ_t(x)- _m,t(x))^24( σ^2_t(x)+σ^2_m,t(x)) ] (11) This formulation prioritizes locations where kernel identity has maximal impact on predictions. Unlike MCMC-based SAL [17], KENDO-AL avoids sampling overhead while retaining the ability to measure full distributional disagreement. The method handles both outputscale uncertainty (driving global exploration) and lengthscale uncertainty (encouraging repeated queries for noise estimation), as each kernel’s contribution is weighted by its predictive likelihood. The unified process is shown in Algorithm 1. 5 KENDO for MOBO In this section, we extend the KENDO framework to the multi-objective Bayesian optimization (MOBO) setting, where K (possibly conflicting) objectives fkk=1K\f^k\_k=1^K are optimized simultaneously. Unlike the single-objective case, the solution is no longer a single optimizer but the Pareto set ∗=∗∈:∄∈s.t.()≻(∗)X^*=\x^* : \,x \ s.t.\ f( x) ( x^*)\, ():=[f1(),…,fK()]⊤f(x):=[f^1(x),…,f^K(x)] , formed under the Pareto dominance relation ≻ , where ()≻(∗)f( x) ( x^*) denotes that fk()≥fk(∗)f^k( x)≥ f^k( x^*) for all k with strict inequality for at least one k in maximization problem. This relation induces the Pareto front ∗=(∗):∗∈∗P^*=\f( x^*): x^* ^*\ in objective space, and the discovered front is commonly evaluated against ∗P^* via the hypervolume indicator with respect to a reference point [7]. Therefore, two design choices of KENDO-BO are key in the MOBO setting. The kernel ensemble needs to be instantiated per objective, allowing different fkf^k to prefer different kernels in K, and the optimizer conditioning that drives the disagreement signal in (8) must be extended to the Pareto setting, where the solution is inherently a set rather than a single point. For K objectives fkk=1K\f^k\_k=1^K, we maintain independent EGPs per objective with weights wtk,mw_t^k,m. Extending to multi-objective optimization, however, requires careful treatment, as the notion of a single “optimizer” no longer applies; the solution is a Pareto front rather than a single point. A direct extension would condition on sampled Pareto fronts, but this introduces Pm,nP_m,n phantom observations simultaneously, diffusing the conditioning signal across the input space and fundamentally departing from the single-optimizer mechanism that makes SCoreBO effective. Our experiments confirm that this yields suboptimal performance (Section 6.3). Instead, we propose a random scalarization strategy that preserves the single-optimizer structure. At each iteration, we sample S weight vectors s∼Dir(K) λ_s (1_K) from a symmetric Dirichlet distribution. For each kernel m and scalarization s, we draw function paths fm,skf^k_m,s for all objectives via Random Fourier features (RFF) [29, 41], form the scalarized objective gm,s()=∑kλs,kfm,sk()g_m,s( x)= _k _s,kf^k_m,s( x), and find its single optimizer m,s∗ x^*_m,s. We then condition each per-objective GP on the phantom observation (m,s∗,fm,s∗,k)( x^*_m,s,f^*,k_m,s) where fm,s∗,k=fm,sk(m,s∗)f^*,k_m,s=f^k_m,s( x^*_m,s), and aggregate per-objective Hellinger distances for the acquisition step as follows: α()=∑k=1K∑m=1Mwtk,m⋅1S∑s=1Sd(p¯(yxk∣t),p(yxk∣i=m,(m,s∗,fm,s∗,k),t))α( x)= _k=1^K _m=1^Mw_t^k,m· 1S _s=1^Sd\! ( p(y_x^k _t),\;p(y_x^k i\!=\!m,\,(x^*_m,s,f^*,k_m,s),\,D_t) ) (12) where p¯(yxk∣t) p(y_x^k _t) is the moment-matched marginal for objective k and d(⋅,⋅)d(·,·) is the Hellinger distance (9). This design preserves the focused, single-point conditioning that makes KENDO-BO effective, while a sufficient diversity of scalarization directions s\ λ_s\ provides coverage of the full Pareto front; Appendix C.5 confirms that the framework is robust to the choice of Dirichlet concentration controlling this diversity. Finding each scalarized optimizer requires only single-objective optimization via L-BFGS on the RFF path, avoiding multi-objective solvers such as NSGA-I [4]. The full process is summarized in Algorithm 2 in the supplementary file. 6 Experiments 6.1 Experimental Setup All EGP-based methods use =RBF,Matérn-1.5,Matérn-2.5K=\RBF,Mat\'ern-1.5,Mat\'ern-2.5\ with per-kernel marginal-likelihood fitting. To ensure fair comparison, every method (including baselines) uses the same GP priors: GammaPrior(3.0,6.0)GammaPrior(3.0,6.0) on lengthscale and GammaPrior(2.0,0.15)GammaPrior(2.0,0.15) on outputscale. For single-objective BO, we compare against NEI [21], PES [14], GIBBON (MES) [25], JES [18], the MCMC-based SCoreBO [17], and EGP-TS [24] that uses the same kernel ensemble as ours but replaces the disagreement-based acquisition with standard Thompson sampling. For multi-objective BO, baselines include JESMO [38], MESMO [1], PES(MO), qNEHVI [3], EGP-TS-MOBO, and KENDO-MO-PF (our Pareto-front-conditioning variant from Section 5, included to validate the scalarization design). For active learning, we compare against BALD [16], BQBC [31], QBMGP [31], and SAL [17]; all SAL variants use Hellinger distance. The benchmarks used for evaluation are categorized as follows. Single-objective BO: Branin (2D), Rosenbrock (2D, 4D), Hartmann (3D, 4D, 6D), and three additional real world problems, RobotPush (3D), RF-HPO on Adult (4D), and Borehole (8D), to cover synthetic, simulator, and HPO problems. Multi-objective BO: two synthetic functions: ZDT2 [43] (d=6d=6, K=2K=2), DTLZ2 [5] (d=6d=6, K=3K=3), and four real world problems: VehicleSafety (d=5d=5, K=3K=3), CarSideImpact (d=7d=7, K=3K=3), Penicillin [22] (d=7d=7, K=3K=3), and LCBench [42] (d=7d=7, K=2K=2), reported via log10 _10 hypervolume difference. Active learning: Higdon (1D), Branin (2D), Ishigami (3D) and Hartmann (6D), along with three real benchmarks, RobotPush (3D), GBT-HPO (4D) and Airfoil (5D), reported via negative MLL on held-out validation sets, with relative ranking as an aggregate view. All curves are averaged over 25 seeds; results for single-kernel baselines are additionally averaged over the three kernel variants in K. 6.2 Single-Objective Optimization Results For performance evaluation, the central question is whether substituting MCMC with a discrete kernel ensemble degrades optimization quality. Figure 2 shows that this well-motivated replacement does not degrade optimization performance. On the contrary, across all nine benchmarks, spanning synthetic functions to real problems and low to high dimensional regimes, KENDO-BO matches or outperforms all baselines. The relative-ranking panel confirms that it maintains the top average rank as the budget grows. The comparison against EGP-TS isolates the contribution of the acquisition function, since both methods share the same kernel ensemble and weighting scheme. KENDO-BO consistently outperforms EGP-TS, indicating that the gain comes from disagreement-based, optimizer-conditioned querying rather than from the ensemble alone. Figure 2: Average log10 _10 simple regret on nine single-objective benchmarks plus the relative ranking panel, averaged over 25 random seeds. Results are averaged over 3 kernel variants. On Borehole, curves saturate at −10-10 due to a hard floor on log10 _10 regret; KENDO-BO reaches the floor earliest. 6.3 Multi-Objective Optimization Results Figure 3 reports results on six MOBO benchmarks. KENDO-MO achieves the lowest log10 _10 hypervolume difference on the majority of problems, with the largest margins on Penicillin and CarSideImpact, both real-world problems whose objectives have heterogeneous landscapes that benefit from per-objective adaptive weighting. The aggregate ranking panel further shows KENDO-MO on top across the full budget. In comparison with KENDO-MO-PF, which conditions on sampled Pareto fronts instead of scalarized optimizers, KENDO-MO-PF underperforms KENDO-MO consistently across all six benchmarks. This corroborates what we discuss in Section 5: conditioning on an entire Pareto set introduces many phantom observations at once, and the conditioning signal gets diffused across the input space rather than concentrated where it is informative. Random scalarization keeps the focused, single-point conditioning that makes single-objective KENDO-BO work, while the diversity of s λ_s ensures the Pareto front is still adequately covered. Figure 3: Multi-objective optimization results on six benchmarks plus the relative ranking panel. Reported as log10 _10 hypervolume difference, averaged over 25 random seeds and 3 kernel variants. 6.4 Active Learning Results Unlike BO, active learning calls for pure exploration, where the goal is global predictive accuracy, not finding an optimum. This objective allows us to test whether the ensemble’s kernel diversity provides intrinsic value beyond its role in guiding optimization. Figure 4 evaluates KENDO-AL on seven benchmarks spanning synthetic functions (Higdon, Branin, Ishigami, Hartmann), a simulator task (RobotPush), an HPO problem (GBT-HPO), and an engineering surrogate (Airfoil). KENDO-AL achieves the lowest negative MLL on every benchmark and holds the best average rank in the aggregate panel. The disagreement-based methods (KENDO-AL and SAL) consistently perform well and KENDO-AL further improves on SAL by replacing MCMC over hyperparameters with the kernel ensemble, which bypasses the sensitivity to a pre-selected kernel that QBMGP and BQBC suffer from. These gains come with a 7.6×7.6×–26.9×26.9× speedup over SAL (Appendix C.2). Figure 4: Active learning results on seven benchmarks plus the relative ranking panel. Negative MLL on held-out validation sets, averaged over 25 random seeds and three kernel variants. 6.5 Ablation Studies We evaluate four design choices of KENDO: adaptive versus uniform weighting, computational efficiency, kernel weight dynamics, and the choice of scalarization for MOBO. Detailed results and discussion are deferred to Appendix C; we summarize the main findings here. Adaptive vs. uniform weighting. The Bayesian weighting scheme in (4) outperforms uniform averaging across all three task families, with the gap pronounced on benchmarks where the preferred kernel varies across input regions (Appendix C.1). Computational efficiency. KENDO runs 3.1×3.1×–5.4×5.4× faster than SCoreBO, and KENDO-AL runs 7.6×7.6×–26.9×26.9× faster than SAL; the weight update itself contributes less than 77 ms per iteration (Appendix C.2). Kernel weight convergence. The ensemble exhibits winner-take-all behavior on benchmarks with a clearly superior kernel, and remains diversified when no single kernel dominates (Appendix C.3). Sensitivity to scalarization design. The MOBO scalarization layer is robust to both the scalarization form and the Dirichlet concentration (Appendices C.4–C.5). 7 Conclusions The central focus of this paper is kernel family diversity as a source of model uncertainty. We address this by replacing hyperparameter sampling from costly fully Bayesian MCMC based approaches with an explicit kernel ensemble and adaptive Bayesian weighting, that can be readily leveraged for both Bayesian optimization and active learning. Our KENDO-BO and KENDO-AL variants achieve superior optimization and predictive performance over MCMC-based approaches with substantial computational savings. The proposed approach naturally extends to multi-objective optimization via random scalarization, while ablation studies corroborate the effectiveness of adaptive weighting and the automatic model selection capabilities of the ensemble approach. Limitations. The current framework relies on moment matching to approximate Gaussian mixture posteriors, which may underestimate uncertainty in cases with highly disparate kernel predictions. Additionally, the kernel dictionary requires manual specification; though automatic construction via kernel composition [6] could be explored. Future Work. Our future research agenda includes: (1) adaptive kernel dictionary construction during optimization; (2) integration with batch BO for parallel evaluations [10]; (3) extension to constrained optimization with feasibility modeling per kernel; (4) application to hyperparameter tuning of deep neural networks; and (5) investigation of the ensemble mechanism in deep active learning contexts where model capacity exceeds kernel expressiveness. References [1] S. Belakaria, A. Deshwal, and J. R. Doppa (2019) Max-value entropy search for multi-objective Bayesian optimization. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 32. Cited by: §2.3, §6.1. [2] S. Daulton, M. Balandat, and E. Bakshy (2020) Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 33. Cited by: §2.3. [3] S. Daulton, M. Balandat, and E. Bakshy (2021) Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 34. Cited by: §2.3, §6.1. [4] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan (2002) A fast and elitist multiobjective genetic algorithm: NSGA-I. IEEE Trans. Evol. Comput. 6 (2), p. 182–197. Cited by: §5. [5] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler (2002) Scalable multi-objective optimization test problems. In Proc. Congr. Evol. Comput., p. 825–830. Cited by: §6.1. [6] D. Duvenaud, J. Lloyd, R. Grosse, J. Tenenbaum, and G. Zoubin (2013) Structure discovery in nonparametric regression through compositional kernel search. In Proc. Intl. Conf. Mach. Learn., p. 1166–1174. Cited by: §2.1, §7. [7] M. T. Emmerich, K. C. Giannakoglou, and B. Naujoks (2006) Single-and multiobjective evolutionary optimization assisted by gaussian random field metamodels. IEEE Trans. Evol. Comput. 10 (4), p. 421–439. Cited by: §2.3, §5. [8] M. Feurer, J. Springenberg, and F. Hutter (2015) Initializing bayesian hyperparameter optimization via meta-learning. In Proc. AAAI Conf. Artif. Intell., Vol. 29. Cited by: §2.2. [9] P. I. Frazier (2018) A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811. Cited by: §1, §3.1. [10] J. Gonzalez, Z. Dai, P. Hennig, and N. Lawrence (2016) Batch Bayesian optimization via local penalization. In Proc. Intl. Conf. Artif. Intell. Stat., Vol. 51, p. 648–657. Cited by: §7. [11] P. Hennig and C. J. Schuler (2012) Entropy search for information-efficient global optimization. J. Machine Learning Res. 13, p. 1809–1837. Cited by: §2.2. [12] J. Hensman, A. G. de G. Matthews, and Z. Ghahramani (2015) Scalable variational Gaussian process classification. In Proc. Intl. Conf. Artif. Intell. Stat., Vol. 38, p. 351–360. Cited by: §2.1. [13] D. Hernández-Lobato, J. M. Hernández-Lobato, A. Shah, and R. P. Adams (2016) Predictive entropy search for multi-objective Bayesian optimization. In Proc. Intl. Conf. Mach. Learn., Vol. 48, p. 1492–1501. Cited by: §2.3. [14] J. M. Hernández-Lobato, M. W. Hoffman, and Z. Ghahramani (2014) Predictive entropy search for efficient global optimization of black-box functions. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 27. Cited by: §2.2, §6.1. [15] M. D. Hoffman A. Gelman et al. (2014) The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo.. J. Machine Learning Res. 15 (47), p. 1593–1623. Cited by: §2.1. [16] N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel (2011) Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745. Cited by: §2.2, §6.1. [17] C. Hvarfner, E. O. Hellsten, F. Hutter, and L. Nardi (2023) Self-correcting Bayesian optimization through Bayesian active learning. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 36. Cited by: Appendix B, §1, §2.2, §3.2, §4.3, §6.1. [18] C. Hvarfner, F. Hutter, and L. Nardi (2022) Joint entropy search for maximally-informed Bayesian optimization. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 35. Cited by: §6.1. [19] A. Kirsch, J. Van Amersfoort, and Y. Gal (2019) BatchBALD: efficient and diverse batch acquisition for deep Bayesian active learning. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 32. Cited by: §2.2. [20] J. Knowles (2006) ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Comput. 10 (1), p. 50–66. Cited by: §C.4, §C.4, §C.5, §2.3. [21] B. Letham, B. Karrer, G. Ottoni, and E. Bakshy (2019) Constrained bayesian optimization with noisy experiments. Cited by: §6.1. [22] Q. Liang and L. Lai (2021) Scalable Bayesian optimization accelerates process optimization of penicillin production. In NeurIPS 2021 AI for Science Workshop, Cited by: §6.1. [23] Q. Lu, G. V. Karanikolas, Y. Shen, and G. B. Giannakis (2020) Ensemble Gaussian processes with spectral features for online interactive learning with scalability. In Proc. Intl. Conf. Artif. Intell. Stat., Vol. 108, p. 1910–1920. Cited by: §1, §2.1, §3.1. [24] Q. Lu, K. D. Polyzos, B. Li, and G. B. Giannakis (2023) Surrogate modeling for bayesian optimization beyond a single gaussian process. IEEE Trans. Pattern Anal. Mach. Intell. 45 (9), p. 11283–11296. Cited by: §1, §6.1. [25] H. B. Moss, D. S. Leslie, J. González, and P. Rayson (2021) GIBBON: general-purpose information-based Bayesian optimisation. J. Machine Learning Res. 22 (235), p. 1–49. Cited by: §6.1. [26] I. Murray and R. P. Adams (2010) Slice sampling covariance hyperparameters of latent Gaussian models. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 23. Cited by: §2.1. [27] B. Paria, K. Kandasamy, and B. Póczos (2020) A flexible framework for multi-objective Bayesian optimization using random scalarizations. In Conf. on Uncertainty in Artif. Intell., Vol. 115, p. 766–776. Cited by: §2.3. [28] K. D. Polyzos, Q. Lu, and G. B. Giannakis (2023) Bayesian optimization with ensemble learning models and adaptive expected improvement. In Proc. of Intl. Conf. Acoust. Speech Signal Process., p. 1–5. Cited by: §1. [29] A. Rahimi and B. Recht (2007) Random features for large-scale kernel machines. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 20. Cited by: §B.1, §5. [30] C. E. Rasmussen and C. K. I. Williams (2006) Gaussian processes for machine learning. MIT Press. Cited by: Appendix D, §1, §2.1, §3.1, §3.1. [31] C. Riis, F. Antunes, F. B. Hüttel, C. L. Azevedo, and F. C. Pereira (2022) Bayesian active learning with fully Bayesian Gaussian processes. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 35. Cited by: §2.2, §6.1. [32] B. Settles (2009) Active learning literature survey. Technical report Technical Report 1648, University of Wisconsin–Madison, Department of Computer Sciences. Cited by: §2.2. [33] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas (2015) Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104 (1), p. 148–175. Cited by: §1. [34] J. Snoek, H. Larochelle, and R. P. Adams (2012) Practical Bayesian optimization of machine learning algorithms. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 25. Cited by: §1, §2.1. [35] J. T. Springenberg, A. Klein, S. Falkner, and F. Hutter (2016) Bayesian optimization with robust Bayesian neural networks. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 29. Cited by: §1. [36] N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger (2010) Gaussian process optimization in the bandit setting: no regret and experimental design. In Proc. Intl. Conf. Mach. Learn., p. 1015–1022. Cited by: §2.2. [37] M. K. Titsias (2009) Variational learning of inducing variables in sparse Gaussian processes. In Proc. Intl. Conf. Artif. Intell. Stat., Vol. 5, p. 567–574. Cited by: §2.1. [38] B. Tu, A. Gandy, N. Kantas, and B. Shafei (2022) Joint entropy search for multi-objective Bayesian optimization. In Proc. of Adv. Neural Inf. Process. Syst., Vol. 35. Cited by: §2.3, §6.1. [39] Z. Wang and S. Jegelka (2017) Max-value entropy search for efficient Bayesian optimization. In Proc. Intl. Conf. Mach. Learn., Vol. 70, p. 3627–3635. Cited by: §2.2. [40] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing (2016) Deep kernel learning. In Proc. Intl. Conf. Artif. Intell. Stat., Vol. 51, p. 370–378. Cited by: §2.1. [41] J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth (2020) Efficiently sampling functions from Gaussian process posteriors. In Proc. Intl. Conf. Mach. Learn., Vol. 119, p. 10292–10302. Cited by: §B.1, §5. [42] L. Zimmer, M. Lindauer, and F. Hutter (2021) Auto-PyTorch: multi-fidelity metalearning for efficient and robust AutoDL. IEEE Trans. Pattern Anal. Mach. Intell. 43 (9), p. 3079–3090. Cited by: §6.1. [43] E. Zitzler, K. Deb, and L. Thiele (2000) Comparison of multiobjective evolutionary algorithms: empirical results. Evol. Comput. 8 (2), p. 173–195. Cited by: §6.1. [44] E. Zitzler and S. Künzli (2004) Indicator-based selection in multiobjective search. In Proc. Intl. Conf. Parallel Problem Solving from Nature, p. 832–842. Cited by: §2.3. Supplementary File Appendix A KENDO-MO Algorithm for Multi-Objective Optimization This appendix presents the full procedure of KENDO-MO described in Section 5. Algorithm 2 KENDO-MO for Multi-Objective Optimization 0: Kernel dict. K, #scalarizations S, #objectives K, data 0D_0, budget T 1: for k=1,…,Kk=1,…,K; m∈m do 2: Fit αmk _m^k via maxp(0k|i=m,α) p(D_0^k|i=m,α); Init w0k,m=1/Mw_0^k,m=1/M 3: end for 4: for t=0,…,T−1t=0,…,T-1 do 5: for k=1,…,Kk=1,…,K do 6: Compute μ¯tk(⋅),σ¯t2,k(⋅) μ_t^k(·), σ^2,k_t(·) via (6)–(7) with weights wtk,mw_t^k,m 7: end for 8: for m∈m ; s=1,…,Ss=1,…,S do 9: Sample s∼Dir(K) λ_s (1_K); Sample fm,sk∼p(fk|i=m,t)f^k_m,s p(f^k|i=m,D_t) via RFF for all k 10: gm,s(x)=∑kλs,kfm,sk()g_m,s(x)= _k _s,kf^k_m,s(x); Find xm,s∗=argmaxxgm,s()x^*_m,s= _xg_m,s(x) 11: for k=1,…,Kk=1,…,K do 12: Condition GP m on (m,s∗,fm,sk(m,s∗))(x^*_m,s,f^k_m,s(x^*_m,s)) to get μm,s∗,k,σm,s2,∗,kμ^*,k_m,s,σ^2,*,k_m,s 13: end for 14: end for 15: t+1=argmaxx∑k∑mwtk,m⋅1S∑sd(p¯k,pm,sk)x_t+1= _x _k _mw_t^k,m· 1S _sd( p^k,p^k_m,s) via (12) 16: Evaluate t+1=(f1(t+1),…,fK(t+1))y_t+1=(f^1(x_t+1),…,f^K(x_t+1)); t+1=t∪(t+1,t+1)D_t+1=D_t∪\(x_t+1,y_t+1)\ 17: for k=1,…,Kk=1,…,K; m∈m do 18: Update wt+1k,mw_t+1^k,m via (4); Update GP posterior via (23a)–(23b) 19: end for 20: end for 21: return Pareto front from TD_T Appendix B Derivation of the KENDO-BO Acquisition Function This appendix details how the KENDO-BO acquisition function in (8) arises from the disagreement principle of SCoreBO [17] once the source of model uncertainty is changed from continuous hyperparameters to a discrete kernel posterior. We also cover the moment-matching step that produces the Gaussian summary in (6) and (7), and the closed-form Hellinger distance in (9). B.1 From SCoreBO’s continuous expectation to KENDO’s discrete sum SCoreBO defines its acquisition as the expected Hellinger disagreement between the marginal predictive and an optimizer-conditioned predictive αSC()=,∗[d(p(y∣t),p(y∣,∗,t))], _SC(x)=E_ θ, \! [d\! (p(y_x _t),\;p(y_x θ, ,D_t) ) ], (13) where the joint posterior factors as p(,∗∣t)=p(∗∣,t)p(∣t)p( θ, _t)=p( θ,D_t)\,p( θ _t). The outer expectation is intractable, and SCoreBO approximates it by drawing hyperparameter samples m θ_m from p(∣t)p( θ _t) via MCMC, then drawing N optimizers per sample, and applying the uniformly weighted estimator 1MN∑m∑nd(⋅,⋅) 1MN _m _nd(·,·). KENDO changes the random quantity itself. Instead of a continuous distribution over hyperparameters of a single kernel, the source of model uncertainty is the kernel family, whose posterior is the discrete distribution p(m∣t)=wtmp(m _t)=w^m_t defined by the Bayesian model average in (4). The acquisition function then takes the form α()=m,∗[d(p(y∣t),p(y∣m,∗,t))].α(x)=E_m, \! [d\! (p(y_x _t),\;p(y_x m, ,D_t) ) ]. (14) Factoring the joint posterior as p(m,∗∣t)=p(∗∣m,t)wtmp(m, _t)=p( m,D_t)\,w^m_t and expanding the now discrete outer expectation gives α() α(x) =∑m=1Mwtm⋅∗|m[d(p(y∣t),p(y∣m,∗,t))] = _m=1^Mw^m_t·E_ m\! [d\! (p(y_x _t),\;p(y_x m, ,D_t) ) ] (15) ≈∑m=1Mwtm⋅1N∑n=1Ndm,n()=∑m=1M∑n=1NwtmNdm,n(), ≈ _m=1^Mw^m_t· 1N _n=1^Nd_m,n(x)\;=\; _m=1^M _n=1^N w^m_tN\,d_m,n(x), (16) which recovers (8). The inner step uses N optimizer samples ∗m,n=(m,n∗,fm,n∗) _m,n=(x^*_m,n,f^*_m,n) obtained by optimizing function paths fm,n∼p(f∣m,t)f_m,n p(f m,D_t) generated through random Fourier features [29, 41]. Conditioning the m-th GP on the phantom observation (m,n∗,fm,n∗)(x^*_m,n,f^*_m,n) yields the conditional moments μ∗m,nm,tμ _m,n_m,t and σ2,∗m,nm,tσ^2, _m,n_m,t. B.2 Moment matching of the marginal mixture The marginal predictive in (3) is a Gaussian mixture, so its Hellinger distance to a single Gaussian conditional has no closed form expression. We replace the mixture by a single Gaussian whose first and second moments match those of the mixture exactly. For the mixture ∑mwtm(μm,t,σm,t2) _mw^m_tN( _m,t,σ^2_m,t), these are μ¯t() μ_t(x) =∑m=1Mwtmμm,t(), = _m=1^Mw^m_t _m,t(x), (17) σ¯t2() σ^2_t(x) =∑m=1Mwtmσm,t2()+∑m=1Mwtm(μm,t()−μ¯t())2, = _m=1^Mw^m_tσ^2_m,t(x)+ _m=1^Mw^m_t ( _m,t(x)- μ_t(x) )^2, (18) which match (6) and (7). The variance decomposes additively into a within-component term (average individual variance) and a between-component term (spread of component means around the mixture mean). B.3 Closed-form Hellinger distance between two Gaussians For two univariate Gaussians z1∼(μ1,σ12)z_1 ( _1, _1^2) and z2∼(μ2,σ22)z_2 ( _2, _2^2), the squared Hellinger distance admits the closed form H2(z1,z2)=1−2σ1σ2σ12+σ22exp[−14(μ1−μ2)2σ12+σ22].H^2(z_1,z_2)=1- 2 _1 _2 _1^2+ _2^2 \! [- 14 ( _1- _2)^2 _1^2+ _2^2 ]. (19) Setting z1∼(μ¯t(),σ¯t2())z_1 ( μ_t(x), σ^2_t(x)) as the moment-matched marginal and z2∼(μm,t∗m,n(),σm,t2,∗m,n())z_2 (μ _m,n_m,t(x),σ^2, _m,n_m,t(x)) as the conditional posterior of the m-th GP given optimizer sample n, and writing dm,n()=H(z1,z2)d_m,n(x)=H(z_1,z_2), we recover the per-pair Hellinger distance in (9). B.4 Reduction to KENDO-AL Removing the optimizer conditioning leaves αAL()=m[d(p(y∣t),p(y∣m,t))]=∑m=1Mwtmdm(), _AL(x)=E_m\! [d\! (p(y_x _t),\;p(y_x m,D_t) ) ]= _m=1^Mw^m_t\,d_m(x), (20) where dm()d_m(x) uses (19) between the moment-matched marginal and the m-th component. This recovers (10), and (11) follows from the same closed form. Appendix C Ablation Studies This appendix presents the full ablation studies summarized in Section 6.5 of the main text. C.1 Adaptive vs. Uniform Weighting Figure 5 compares KENDO with adaptive Bayesian weights (4) against a variant using fixed uniform weights (wm=1/Mw^m=1/M throughout optimization). Across all three task categories, single-objective BO, active learning, and multi-objective optimization, adaptive weighting consistently achieves lower final regret. On single-objective benchmarks, the advantage is most pronounced on Branin (2D) and Hartmann (6D), where adaptive weighting achieves roughly 0.50.5 and 0.30.3 lower log10 _10 regret at convergence, respectively. For active learning, adaptive weighting leads to substantially better MLL on Hartmann (6D) and Ishigami (3D), where kernel preference varies significantly across regions of the input space. In multi-objective settings, the advantage is evident on DTLZ2 and Penicillin, where different objectives may favor different kernels, and adaptive per-objective weighting captures this structure more effectively than uniform averaging. These results confirm that the Bayesian weight update mechanism provides measurable performance gains by dynamically allocating model capacity to the most predictive kernels. C.2 Computational Efficiency A central claim of KENDO is that replacing MCMC hyperparameter sampling with an explicit kernel ensemble yields substantial computational savings. Figure 6 validates this claim by comparing per-iteration wall-clock time between EGP methods and their MCMC-based counterparts. For Bayesian optimization, KENDO achieves 3.1×3.1×–5.4×5.4× speedup over SCoreBO across benchmarks, with the largest gain on Branin (2D) where MCMC overhead dominates relative to the simple acquisition landscape. The speedup is consistent across problem dimensions, ranging from 3.1×3.1× on Hartmann (4D) to 5.4×5.4× on Branin. The computational advantage is even more dramatic for active learning. KENDO-AL achieves 7.6×7.6×–26.9×26.9× speedup over SAL-Matérn-2.5, with the largest gains on Ishigami (26.9×26.9×) and Higdon (19.9×19.9×). This amplified speedup arises because active learning acquisition functions (which do not require RFF-based function sampling and optimizer search) have lower base cost, making the relative contribution of MCMC overhead proportionally larger. Figure 7 provides a detailed breakdown of where computation time is spent within KENDO and KENDO-AL. For BO, acquisition function optimization dominates (50–80% of iteration time), with model fitting consuming most of the remainder (17–45%). RFF sampling and weight updates contribute negligibly. For active learning, the split between acquisition optimization and model fitting varies by benchmark, but weight update time remains consistently below 7 milliseconds, confirming that the Bayesian weight update (4) introduces negligible overhead. Figure 5: Convergence comparison between adaptive Bayesian weighting and uniform (average) weighting across all three task settings. Top row: log10 _10 simple regret on single-objective benchmarks. Middle row: Negative MLL on active learning benchmarks. Bottom row: log10 _10 hypervolume difference on multi-objective benchmarks. Adaptive weighting consistently achieves better performance. Figure 6: Per-iteration wall-clock time comparison. Left: KENDO vs. SCoreBO on single-objective benchmarks (3.1×3.1×–5.4×5.4× speedup). Right: KENDO-AL vs. SAL-Matérn-2.5 on active learning benchmarks (7.6×7.6×–26.9×26.9× speedup). The ensemble mechanism eliminates MCMC overhead while maintaining competitive optimization and learning performance (cf. Figures 2–4). Figure 7: Time component breakdown for KENDO (top) and KENDO-AL (bottom). Left panels show main time components (acquisition optimization, model fitting, RFF sampling, weight update). Right panels isolate weight update time, confirming it contributes negligibly (<7<7 ms per iteration). For BO, acquisition optimization dominates (50–80%); for AL, model fitting becomes the primary cost on higher-dimensional benchmarks. C.3 Kernel Weight Convergence and Winner-Take-All Behavior Figure 8 visualizes the kernel weight dynamics across all benchmarks and task settings by tracking the fraction of seeds for which each kernel holds the largest weight at each iteration. On smooth, stationary benchmarks such as Branin, the ensemble weights consistently collapse to a single dominant kernel within approximately 10–20 iterations. This winner-take-all phenomenon is a direct consequence of the multiplicative Bayesian update rule (4): when one kernel achieves marginally higher predictive likelihood at each step, the weight ratio wtm/wtm′w_t^m/w_t^m evolves as a product of per-step likelihood ratios, amplifying small advantages exponentially. For single-objective BO on Branin, Matérn-2.5 wins in 5 out of 10 seeds with an average maximum weight of 0.950, reflecting its effective balance between the infinite smoothness of RBF and the limited differentiability of Matérn-1.5. The summary tables in Figure 8 reveal that kernel preference is benchmark-dependent. On ZDT2 and Vehicle Safety (MOBO), RBF dominates universally (10/10 seeds, average max weight 1.000), while on Penicillin, Matérn-1.5 wins in 8/10 seeds, reflecting the less smooth objective landscape. For active learning, Matérn-2.5 is most frequently preferred (winning on 3 of 4 benchmarks), consistent with the observation that active learning objectives benefit from moderate smoothness assumptions for global function approximation. Rather than harming performance, the winner-take-all behavior reflects successful automatic model selection. The ensemble efficiently identifies the most appropriate kernel for each target function without manual kernel selection or expensive cross-validation. Figure 8: Kernel weight dynamics across all benchmarks. Line plots show the fraction of seeds (out of 10) for which each kernel holds the largest weight at each iteration. The dashed gray line indicates the uniform baseline (33%). Tables summarize the final kernel preference distribution and average maximum weight. The ensemble exhibits decisive model selection on benchmarks with a clear best kernel (e.g., RBF on ZDT2 with max weight 1.000), while maintaining diversity under ambiguity (e.g., Hartmann 3D in BO with a 5/5 split between RBF and Matérn-1.5). C.4 Sensitivity to Scalarization Strategy KENDO-MO reduces multi-objective conditioning to single-optimizer conditioning via random scalarization. To assess whether its gains depend on this specific choice, we compare the default linear scalarization gm,s()=∑kλs,kfm,sk()g_m,s(x)= _k _s,k\,f^k_m,s(x) against Chebyshev scalarization gm,s()=minkλs,k(fm,sk()−zk∗)g_m,s(x)= _k _s,k\,(f^k_m,s(x)-z^*_k), keeping all other components (kernel ensemble, Bayesian weights, RFF sampling, Hellinger-based acquisition) fixed. Robustness across benchmarks. Figure 9 reports hypervolume curves on four MOO benchmarks. A two-sided Wilcoxon rank-sum test on final hypervolume yields no statistically significant difference on ZDT2 (p=0.49p=0.49), Penicillin (p=0.16p=0.16), or Vehicle Safety (p=0.85p=0.85). This indicates that the gains of KENDO-MO arise from the kernel ensemble and disagreement-aware acquisition, not from the specific scalarization. The DTLZ2 case. On DTLZ2, Chebyshev attains a small but significant advantage (p=0.014p=0.014). This is consistent with a well-known property in multi-objective optimization: linear scalarization is known to miss solutions on non-convex regions of the Pareto front, whereas Chebyshev scalarization can recover them [20]. DTLZ2’s Pareto front is concave (a unit-sphere octant), so Chebyshev’s edge here reflects a geometric property of the benchmark. To verify that the framework remains competitive on DTLZ2 once paired with the geometry-appropriate scalarization, Figure 10 plots KENDO-MO-Cheby alongside all multi-objective baselines: KENDO-MO-Cheby substantially outperforms all entropy-search (JESMO/MESMO/PESMO) and ensemble-TS (EGP-TS-MOBO) baselines; only qNEHVI, a hypervolume-specialized acquisition, achieves lower log10 _10 HV difference. The scalarization is thus a tunable interface. Two reasons motivate linear scalarization as the default. First, the linear objective is smooth in x, so the S×MS× M inner argmax problems on the RFF path (Algorithm 2, line 10) are solved efficiently with L-BFGS; the min operator in Chebyshev is non-smooth and requires either subgradient methods or a smoothed surrogate, inflating per-iteration cost, which conflicts with our efficiency-oriented design (Appendix C.2). Second, linear scalarization matches the standard ParEGO formulation [20], facilitating fair comparison with prior multi-objective BO literature. Takeaway. KENDO-MO is robust to the scalarization choice, and a practitioner may select either based on prior knowledge of the problem geometry, e.g., Chebyshev when the Pareto front is suspected to be non-convex. Figure 9: KENDO-MO with linear and Chebyshev scalarization on four MOO benchmarks. Three of four benchmarks show no statistically significant difference (two-sided Wilcoxon rank-sum test on final HV); DTLZ2 favors Chebyshev due to its concave Pareto front (see Appendix C.4). Figure 10: DTLZ2 with all multi-objective baselines plus KENDO-MO-Cheby. Both KENDO-MO variants outperform all entropy-search and ensemble-TS baselines; only qNEHVI (hypervolume-specialized) attains lower log10 _10 HV difference, confirming that the scalarization choice is a tunable interface rather than a framework bottleneck. C.5 Sensitivity to Dirichlet Concentration α KENDO-MO samples scalarization weights from a symmetric Dirichlet s∼Dir(αK) λ_s ( 1_K), whose density p(,α)=Γ(Kα)Γ(α)K∏k=1Kλkα−1,∈ΔK−1,p( λ;α)= (Kα) (α)^K _k=1^K _k^α-1, λ∈ ^K-1, (21) is governed by a single concentration parameter α that controls how mass is distributed on the simplex. The per-coordinate variance Var[λk]=(K−1)/(K2(Kα+1))Var[ _k]=(K-1)/(K^2(Kα+1)) is monotonically decreasing in α, so α→0α→ 0 pushes λ toward the vertices (corner-heavy, near single-objective directions), α=1α=1 recovers the uniform prior used in Section 5, and α→∞α→∞ concentrates on the centroid (1/K,…,1/K)(1/K,…,1/K) (center-heavy, near equal-weight trade-offs). Through the inner argmax in Algorithm 2 (line 10), α therefore directly governs the diversity of the phantom optimizers xm,s∗\x^*_m,s\ that drive the disagreement signal in (12). We sweep α∈0.2, 1.0, 5.0α∈\0.2,\,1.0,\,5.0\ on five MOO benchmarks (LCBench, VehicleSafety, DTLZ2, Penicillin, ZDT2). Figure 11(b,c) verifies that α acts as designed. The mean sample entropy H¯=[−∑kλklogλk] H=E_ λ[- _k _k _k] stratifies cleanly across the three values and approaches its ceiling logK K at α=5.0α=5.0 (H¯≈0.65 H\!≈\!0.65 for K=2K\!=\!2 and 1.031.03 for K=3K\!=\!3), while the mean pairwise distance of phantom optimizers (column c) shows the expected inverse relationship, α=0.2α=0.2 yields the highest phantom-X diversity and α=5.0α=5.0 the lowest. The Dirichlet sampling layer therefore behaves exactly as the variance formula predicts. The hypervolume curves in column (a) show that the default α=1.0α=1.0 is robust across all five benchmarks: it is never significantly outperformed by either extreme. The optimal α is, however, benchmark-dependent and tracks Pareto-front geometry. On benchmarks with concave fronts (ZDT2, DTLZ2) and heterogeneous landscapes (VehicleSafety), small α matches or exceeds the default, consistent with the classical result that linear scalarization on concave fronts benefits from corner-heavy directions to recover non-convex regions [20]. Conversely, on real-world problems whose trade-off region concentrates near the simplex centroid (LCBench, Penicillin), α=5.0α=5.0 yields a small but consistent gain. Phantom-X diversity is not monotonically tied to hypervolume: sufficient diversity is necessary, but excess diversity wastes budget on extreme corners that lie far from the achievable front. Takeaway. KENDO-MO’s gains arise from the kernel ensemble and disagreement-aware acquisition, not from a finely tuned Dirichlet concentration; the symmetric prior α=1.0α=1.0 is a safe and competitive default, while practitioners with prior knowledge of front geometry may tune α accordingly. Together with the study in Appendix C.4, this confirms that the scalarization layer is a tunable interface rather than a sensitive bottleneck. Figure 11: Sensitivity of KENDO-MO to Dirichlet concentration α∈0.2,1.0,5.0α∈\0.2,1.0,5.0\ on five MOO benchmarks. Column (a): hypervolume vs. iteration. Column (b): mean Shannon entropy of sampled λ, confirming that the Dirichlet sampling layer behaves as designed. Column (c): mean pairwise distance of phantom optimizers, showing α controls phantom-X diversity inversely. The default α=1.0α=1.0 is never significantly outperformed; optimal α is benchmark-dependent and tracks Pareto-front geometry. Appendix D Additional preliminaries For each Gaussian process model m in the ensemble, using the data t:=(τ,yτ)τ=1tD_t:=\( x_τ,y_τ)\_τ=1^t acquired at slot t, the corresponding posterior pdf p(f()|m,t)p(f(x)|m,D_t) is updated via Bayes rule as [30]: p(f()|m,t)=(μm,t(),σm,t2()) p(f( x)|m,D_t)=N( _m,t( x), _m,t^2( x)) (22) where μm,t() _m,t( x) =t⊤()(t+σn2t)−1t =k_t ( x)(K_t+ _n^2I_t)^-1y_t (23a) σm,t2() _m,t^2( x) =κ(,)−t⊤()(t+σn2t)−1t(). =\!κ(x,x)\!-\!k_t ( x)(K_t\!+\! _n^2I_t)^-1k_t( x). (23b) with t():=[κ(1,),…,κ(t,)]⊤k_t( x):=[κ(x_1,x),…,κ(x_t,x)] , tK_t denoting the t×t× t kernel/covariance matrix where []i,j=κ(i,j)[K]_i,j=κ(x_i,x_j), and σn2 _n^2 denoting the observation variance. Note that the mean in (23a) provides a point estimate of f()f( x), while the variance in (23b) quantifies the associated uncertainty.