Paper deep dive
Multi-Objective Bayesian Optimization for Model Merging
Utkarsh Agarwal, Vamshi Bonagiri, Raul Astudillo, Monojit Choudhury
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/17/2026, 5:24:08 AM
Summary
The paper introduces MOBO-Merge, a framework that applies multi-objective Bayesian optimization (MOBO) to select merge parameters for combining trained large language models. It addresses the difficulty of selecting merge weights by treating the problem as a black-box optimization task to approximate the Pareto front of conflicting capabilities (e.g., instruction following vs. math reasoning). Evaluated on Qwen3-4B and Llama-3.1-8B using various merge operators (Linear, SLERP, TIES, Block-wise), MOBO-Merge outperforms random search in 11 of 12 comparisons, demonstrating the value of MOBO for efficient model merging.
Entities (14)
Relation Signals (8)
MOBO-Merge → evaluateson → Qwen3-4B
confidence 95% · We evaluate Qwen3-4B and Llama-3.1-8B... using MOBO-Merge
MOBO-Merge → evaluateson → LLaMA-3.1-8B
confidence 95% · We evaluate Qwen3-4B and Llama-3.1-8B... using MOBO-Merge
MOBO-Merge → uses → Multi-objective Bayesian Optimization
confidence 95% · introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization
MOBO-Merge → outperforms → Random Search
confidence 92% · MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons
NEHVI → isusedby → MOBO-Merge
confidence 90% · we use (noisy) Expected Hypervolume Improvement (NEHVI) in our experiments
Block-wise Linear Merge → isstrongestfor → Llama three-model merge
confidence 88% · Block-Linear 4x is strongest for the Llama three-model merge
Gaussian Process → isusedby → MOBO-Merge
confidence 88% · we instantiate this model using independent Gaussian process (GP) surrogates
TIES → leadsin → Three-model merge
confidence 88% · TIES leads in three of four family-setting combinations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are expensive, gradients are unavailable, and source capabilities can conflict. We formulate merge-parameter selection as a black-box multi-objective optimization problem and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget. We evaluate Qwen3-4B and Llama-3.1-8B in two-model instruction-math and three-model instruction-math-code settings using Linear, SLERP, TIES, and block-wise merge operators. On held-out benchmark partitions, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons. The gain is small for one-dimensional Linear interpolation but substantially larger for several TIES, block-wise, and three-objective searches. No merge operator is uniformly best: TIES leads in three of four family-setting combinations, whereas Block-Linear 4x is strongest for the Llama three-model merge. These results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations.
Tags
Links
- Source: https://arxiv.org/abs/2608.14264v1
- Canonical: https://arxiv.org/abs/2608.14264v1
Trouble viewing inline? Open PDF directly →
Full Text
62,317 characters extracted from source content.
Expand or collapse full text
Multi-Objective Bayesian Optimization for Model Merging Utkarsh Agarwal utkarsh.agarwal@mbzuai.ac.ae Affiliation: Vamshi Bonagiri vamshi.bonagiri@mbzuai.ac.ae Affiliation: Raul Astudillo raul.astudillo@mbzuai.ac.ae Affiliation: Monojit Choudhury monojit.choudhury@mbzuai.ac.ae Affiliation: Division of Computing and Mathematical Sciences Affiliation: Mohamed bin Zayed University of Artificial Intelligence Abstract Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are expensive, gradients are unavailable, and source capabilities can conflict. We formulate merge-parameter selection as a black-box multi-objective optimization problem and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget. We evaluate Qwen3-4B and Llama-3.1-8B in two-model instruction-math and three-model instruction-math-code settings using Linear, SLERP, TIES, and block-wise merge operators. On held-out benchmark partitions, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons. The gain is small for one-dimensional Linear interpolation but substantially larger for several TIES, block-wise, and three-objective searches. No merge operator is uniformly best: TIES leads in three of four family-setting combinations, whereas Block-Linear 4x is strongest for the Llama three-model merge. These results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations. Figure 1: Overview of MOBO-Merge. Model merging is cast as a multi-objective optimization problem. (Left) Source models specialized for different capabilities (e.g., instruction following θ1 _1 and mathematical reasoning θ2 _2) to be merged via a parameterized merge operator. (Middle) Probabilistic surrogate models of the mapping from merge parameters to objective scores and are used to construct an acquisition function that selects the next merge configuration to evaluate. (Right) Iterating this process yields an approximation of the Pareto front, exposing optimal trade-offs between competing capabilities. 1 Introduction Model merging has emerged as a practical and increasingly popular approach for composing capabilities from pretrained and fine-tuned neural networks (Yang et al. 2024). Instead of training a single monolithic model, practitioners often fine-tune separate models for different skills—such as instruction following, mathematical reasoning, or code generation—and then combine these specialized models into a single system. This paradigm enables modular reuse of existing models while avoiding additional training, and has led to strong empirical results across a wide range of settings (Wortsman et al. 2022; Ilharco et al. 2022; Yadav et al. 2023; Yu et al. 2024; Goddard et al. 2024). Most model merging methods operate directly in weight space using relatively simple operations. For example, given two models θ1 _1 and θ2 _2, a common strategy is linear interpolation, θ(x)=xθ1+(1−x)θ2,θ(x)=x _1+(1-x) _2, where x∈[0,1]x∈[0,1] controls the relative contribution of each model (Wortsman et al. 2022; Ilharco et al. 2022). Even in this simple case, downstream performance can vary sharply as a function of the merge weight, and small changes can lead to qualitatively different behavior. In practice, many merging methods employ richer parameterizations—such as block-wise mixing (Yuce & Amasyali 2025) or interference-aware operators (Yadav et al. 2023)—which induce low- to moderate-dimensional continuous merge spaces in which naive search strategies quickly become impractical (Yuce & Amasyali 2025; Du et al. 2024). Crucially, model merging is often inherently multi-objective. Source models are typically optimized for different capabilities and evaluated using distinct benchmarks, so improving performance on one capability frequently degrades performance on another. As a result, no single merge configuration performs optimally across all criteria (Li et al. 2025a; Chen et al. 2025; Li et al. 2024). Model merging is therefore best understood in terms of trade-offs rather than a single optimal solution, with solution quality naturally characterized by the Pareto front it induces. Identifying high-quality merges under these trade-offs is challenging. Each candidate merge must be instantiated and evaluated on costly downstream benchmarks; evaluations are often noisy, gradients with respect to merge parameters are unavailable, and evaluation budgets are limited. Consequently, exhaustive grid search is infeasible, while random or heuristic strategies scale poorly as the merge space grows. These characteristics place model merging squarely in the regime of expensive black-box multi-objective optimization. In this paper, we introduce Multi-Objective Bayesian Optimization for Model Merging (MOBO-Merge), a framework that casts merge-weight selection as a black-box multi-objective optimization problem and applies multi-objective Bayesian optimization (Knowles 2006; Daulton et al. 2020) to efficiently approximate the Pareto front of capability trade-offs. Rather than targeting a single merged model, MOBO-Merge explicitly seeks a diverse set of Pareto-optimal merges, enabling practitioners to select models that best match their desired balance of capabilities under fixed compute budgets. Importantly, the framework is agnostic to the choice of merge operator and applies uniformly across global interpolation, interference-aware merging, and higher-dimensional block-wise parameterizations. We evaluate MOBO-Merge on Qwen3-4B (Team 2025) and Llama-3.1-8B (Grattafiori et al. 2024), using two-model instruction-math merges and three-model instruction-math-code merges. Across held-out comparisons, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 settings. The gain is more pronounced for several TIES, block-wise, and three-objective searches. The experiments also show that no merge operator is uniformly best: TIES leads in three of four family-setting combinations, while Block-Linear 4x performs best for the Llama three-model merge. Across experiments, MOBO-Merge reliably recovers high-quality Pareto fronts with substantially fewer evaluations than grid or random search and exposes controllable capability trade-offs that are difficult to uncover with single-objective or heuristic approaches. Beyond these gains, the generality of the framework enables a systematic study of multi-objective optimization across diverse merging strategies. This analysis yields several practical insights, including that interference-aware methods such as TIES are relatively robust to their merge weights, and that higher-dimensional block-wise linear merges can achieve competitive performance while remaining simple and interpretable. Together, these findings suggest that efficiently exploring richer merge parameterizations can unlock strong performance trade-offs even for simple merging rules. Contributions. • We formulate merge-parameter selection for specialized models as an expensive black-box multi-objective optimization problem and present MOBO-Merge, a merge-operator-agnostic framework that uses multi-objective Bayesian optimization to efficiently approximate sets of Pareto-optimal merges under limited evaluation budgets. • We evaluate two contemporary model families in two-model and three-model settings across global, interference-aware, and block-wise merge operators, with deterministic validation/held-out benchmark splits. • We show that the advantage over random search grows in several expressive merge spaces, while also identifying important trends: Linear search is already competitive in one dimension, and the strongest merge operator depends on the model family and objective set. 2 Problem Formulation We formulate model merging as a parameterized decision problem in which a small number of continuous merge parameters control how multiple trained models are combined, and performance is assessed through costly downstream evaluations. Rather than treating merging as a fixed procedure, we view it as a mapping from a low-dimensional merge space to a vector of task-level outcomes. Let Θ denote the parameter space of a shared model architecture, and let θ1,…,θK∈Θ _1,…, _K∈ denote trained models to be merged. These models may be pretrained or fine-tuned for different purposes but are assumed to share a common parameterization. Merging is specified through a merge operator. Let ⊂ℝDX ^D denote a low-dimensional merge parameter space encoding how the source models are combined. The merge operator ℳ:ΘK×→ΘM: ^K×X→ produces a merged model θ()=ℳ(θ1,…,θK,),θ( x)=M( _1,…, _K; x), parameterized by ∈ x . This abstraction treats the merge operator as a black box and places minimal assumptions on its structure. The merge space X may represent global mixture weights, block-wise coefficients, or other structured controls, allowing the formulation to capture a wide range of existing and future merging strategies. Merged models are evaluated using a collection of objective functions fi:Θ→ℝ,i=1,…,M,f_i: , i=1,…,M, where each fif_i measures performance with respect to a particular capability or evaluation criterion. These objectives are typically defined by external benchmarks and may be expensive or noisy to evaluate. The merge operator induces a vector-valued objective function over the merge space, ()=(f1(θ()),…,fM(θ())), f( x)= (f_1(θ( x)),…,f_M(θ( x)) ), mapping merge parameters directly to task-level outcomes. In many practical settings, the objectives correspond to distinct capabilities and are inherently conflicting: improving performance on one objective often degrades performance on another. Consequently, we do not assume the existence of a single optimal merge. Instead, solution quality is defined in terms of Pareto optimality (Daulton et al. 2020). A merge parameter ∈ x is Pareto-optimal if there is no ′∈ x such that (′)⪰()and(′)≠(), f( x ) f( x) f( x )≠ f( x), where ⪰ denotes component-wise comparison. The set of Pareto-optimal parameters defines the Pareto set, and its image under f forms the Pareto front. In practice, the goal is to efficiently approximate this Pareto front under a limited evaluation budget. Evaluating () f( x) requires instantiating a merged model and running downstream benchmarks. Evaluations are costly, may be noisy, and provide no gradients with respect to x, so only a small number of evaluations is feasible. These properties place model merging squarely in the regime of expensive black-box multi-objective optimization. 2.1 The Canonical Case: Merging Specialized Models A common instantiation of the formulation above occurs when each source model is trained or fine-tuned to specialize in a distinct capability. In this case, the objectives often correspond directly to these capabilities, and the number of objectives matches the number of source models, i.e., M=KM=K (Li et al. 2025a; Chen et al. 2025). Each source model θk _k is therefore approximately optimized for its corresponding objective fkf_k, but is typically suboptimal with respect to the remaining objectives. Because the objectives are inherently conflicting, improving performance on one capability often comes at the expense of others. Consequently, no single merge is expected to dominate across all objectives, and the set of desirable solutions is naturally characterized by a Pareto front. While our experiments focus on this canonical setting, the formulation itself is more general. In particular, the number of objectives need not match the number of source models, and objectives may represent arbitrary evaluation criteria beyond model-specific capabilities. 3 Method: MOBO-Merge For completeness, we describe the standard multi-objective Bayesian optimization procedure used by MOBO-Merge. Given a parameterized merge operator, MOBO-Merge treats the induced vector-valued objective function f:→ℝMf:X ^M as an expensive black box and uses probabilistic surrogate models and a hypervolume-based acquisition function to approximate its Pareto front under a limited evaluation budget. The framework is merge-operator agnostic: the same optimization procedure applies whenever a merge operator defines a parameter space X and the resulting merged models can be evaluated on the objectives. We present the fully sequential setting, in which one merge is selected and evaluated at each iteration; batch extensions follow standard multi-objective Bayesian optimization practice. Overview. At iteration t, MOBO-Merge maintains a probabilistic surrogate model of the objectives based on previously evaluated merge parameters. This surrogate is used to construct an acquisition function that scores candidate merge configurations according to their expected contribution to Pareto-front quality. The highest-scoring configuration is selected, evaluated to obtain new objective values, and incorporated into the dataset used to update the surrogate model. Repeating this process under a fixed evaluation budget yields a set of evaluated merge configurations that together form an approximation of the Pareto front. MOBO-Merge is summarized in Algorithm 1. 3.1 Probabilistic Modeling of the Objectives MOBO-Merge maintains a probabilistic model over the unknown objective function f. Given a dataset t=(j,j)j=1t,D_t= \ ( x_j, y_j ) \_j=1^t, where j y_j denotes a (potentially noisy) evaluation of the latent objective values (j) f( x_j), the surrogate model induces a posterior distribution over f, and in particular over () f( x) for any ∈ x . This posterior captures both epistemic uncertainty about the objective functions and observation noise in the evaluations, and is used to guide the selection of future merge parameters. In general, any probabilistic regression model suitable for expensive black-box optimization may be used, provided it yields uncertainty-aware predictions. In our experiments, we instantiate this model using independent Gaussian process (GP) surrogates for each objective, a standard choice in Bayesian optimization (Frazier 2018; Garnett 2023) 3.2 Acquisition Function Given the posterior predictive distribution p(()∣t)p\! ( f( x) _t ), MOBO-Merge constructs a multi-objective acquisition function at:→ℝ,a_t:X , which scores candidate merge parameters by their expected improvement to the current Pareto-front approximation. In the fully sequential setting considered here, the acquisition function is optimized to select a single merge parameter at each iteration. To directly target Pareto-front quality under a limited evaluation budget, we focus on acquisition functions based on dominated hypervolume, a standard scalar measure of Pareto-front quality. While MOBO-Merge is compatible with a variety of multi-objective acquisition strategies, we use (noisy) Expected Hypervolume Improvement (NEHVI) in our experiments (Daulton et al. 2021). Noisy Expected Hypervolume Improvement. Let ∈ℝM r ^M denote a reference point dominated by all objective values of interest. The dominated hypervolume of a set ⊂ℝMS ^M is defined as HV()=λ(⋃∈[r1,y1]×⋯×[rM,yM]),HV(S)=λ\! ( _ y [r_1,y_1]×·s×[r_M,y_M] ), where λ(⋅)λ(·) denotes the Lebesgue measure. At iteration t, let t=1,…,tX_t=\ x_1,…, x_t\ denote the set of evaluated merge parameters, and let (t) f(X_t) denote the corresponding (latent) objective values. For a candidate merge parameter ∈ x , the noisy expected hypervolume improvement is defined as at()=[HV((t)∪())−HV((t))|t],a_t( x)=E\! [HV\! ( f(X_t)∪\ f( x)\ )-HV\! ( f(X_t) )\; |\;D_t ], where the expectation is taken with respect to the joint posterior distribution over (t) f(X_t) and () f( x) conditioned on the observed data tD_t. Although we focus on the fully sequential setting, NEHVI naturally extends to batch selection (Daulton et al. 2021). 3.3 Acquisition Optimization and Evaluation At iteration t+1t+1, MOBO-Merge selects the next merge parameters by solving t+1∈argmax∈at(), x_t+1∈ _ x a_t( x), using a suitable continuous optimization method. The selected parameters t+1 x_t+1 are then evaluated by instantiating the merged model θ(t+1)θ( x_t+1) and computing the corresponding objective values t+1=(t+1) y_t+1= f( x_t+1). The dataset is updated as t+1=t∪(t+1,t+1),D_t+1=D_t∪\( x_t+1, y_t+1)\, and the probabilistic surrogate model is refit. Algorithm 1 MOBO-Merge 0: Merge parameter space X, evaluation budget T 1: Initialize dataset 0D_0 2: for t=0,…,T−1t=0,…,T-1 do 3: Fit probabilistic surrogate to tD_t 4: Construct acquisition function ata_t 5: Select t+1∈argmax∈at() x_t+1∈ _ x a_t( x) 6: Evaluate the merged model θ(t+1)θ( x_t+1) to obtain t+1 y_t+1 7: Update dataset t+1=t∪(t+1,t+1)D_t+1=D_t∪\( x_t+1, y_t+1)\ 8: end for 9: Return the nondominated solutions among the evaluated configurations 4 Experiments and Analysis We evaluate MOBO-Merge on instruction-following, mathematical-reasoning, and code-generation objectives using two model families. Our experiments address three questions: (i) when does guided multi-objective search improve over random parameter selection under the same evaluation budget, (i) how does the merge operator affect the quality and coverage of the recovered Pareto set, and (i) do the same trends hold when moving from two source models and two objectives (AB) to three source models and three objectives (ABC)? 4.1 Experimental Setup Models. We evaluate MOBO-Merge on two model families and sizes: Qwen3-4B (Team 2025) and Llama-3.1-8B (Grattafiori et al. 2024). For each family, we use a shared base model together with an instruction-following checkpoint (A), a mathematical-reasoning checkpoint (B), and a third checkpoint (C) used in the code-generation objective. For Qwen3-4B, C is the code-oriented CodeScout checkpoint; for Llama-3.1-8B, C is the continually pretrained Swallow checkpoint, which includes enhanced reasoning and coding capabilities. The primary AB setting merges the instruction and math checkpoints. The ABC setting extends the same framework to a three-model merge by adding C. The full list of checkpoints is provided in Appendix A.2. Objectives and data splits. We measure instruction following with instruction-level strict accuracy on IFEval (Zhou et al. 2023), mathematical reasoning with exact match under flexible answer extraction on zero-shot chain-of-thought GSM8K (Cobbe et al. 2021), and code generation with pass@1 on humaneval_instruct (Chen et al. 2021). Each benchmark is deterministically partitioned using a set split seed. The first 30% of examples form the optimization partition and the complementary 70% form a held-out partition. Every method and seed uses the same partition. Only configurations on a run’s validation Pareto front are re-evaluated on the held-out partition. All evaluations are zero-shot and use deterministic decoding (temperature=0temperature=0) through the LM Evaluation Harness (Gao et al. 2024). Merge and evaluation settings. Merges are materialized with MergeKit (Goddard et al. 2024) in FP16 and are deleted after evaluation. We use a maximum model length of 2048 tokens and up to 1024 generated tokens for GSM8K and HumanEval. The evaluation batch size is 128 for Qwen3-4B and 64 for Llama-3.1-8B. Hypervolume is computed with fixed reference points selected separately for each family and objective set: (0.32,0.64)(0.32,0.64) and (0.32,0.64,0.66)(0.32,0.64,0.66) for Qwen AB and ABC, and (0.31,0.48)(0.31,0.48) and (0.23,0.46,0.31)(0.23,0.46,0.31) for Llama AB and ABC, respectively. Consequently, absolute hypervolume values should only be compared within the same family and objective set. Search protocol and baseline. For each merge operator, we compare sequential NEHVI against uniform random search over the same feasible parameter region. Results are aggregated across seeds 42–51. In the AB experiments, the initial BO design contains 4 points for Linear and 10 points for TIES and Block-Linear 4x. In the ABC experiments, the corresponding initial-design sizes are 6, 14, and 18. These sizes reflect the search dimensions: 1, 4, and 4 in AB, and 2, 6, and 8 in ABC. The three-model weights are constrained to the simplex, with the weight of model C defined by the remaining mass. Each BO run then performs 100 sequential acquisition steps; the random baseline evaluates 100 configurations. 4.2 Merging Strategies We consider four merging strategies. The main comparison includes operators that admit a consistent parameterization in both the two-model and three-model settings. Standard SLERP is restricted to two-model interpolation and has no unique order-independent extension to three-way merging; moreover, in the AB setting it induces the same one-dimensional search space as Linear interpolation. We therefore include SLERP only in the Qwen3-4B AB acquisition-function ablation in Appendix A.1. We use MergeKit (Goddard et al. 2024) to implement merging efficiently on a single GPU (Appendix A.3). Linear. Wortsman et al. 2022 use a weighted average of two models: θ(x)=xθ0+(1−x)θ1,x∈[0,1].θ(x)=x _0+(1-x) _1, x∈[0,1]. SLERP. Shoemake 1985 interpolate along the great-circle on the hypersphere: θ(x)=sin((1−x)Ω)sinΩθ0+sin(xΩ)sinΩθ1,x∈[0,1],θ(x)= \! ((1-x) ) \, _0+ \! (x ) \, _1, x∈[0,1], where Ω=arccos(⟨θ0,θ1⟩‖θ0‖‖θ1‖). = \! ( _0, _1 \| _0\|\,\| _1\| ). TIES. Yadav et al. 2023 mitigate parameter interference by trimming minimally changed parameters, resolving sign conflicts, and merging aligned updates. We use a minor variant that replaces the average of trimmed task vectors with a weighted mean to permit a richer Pareto set. This yields a 4-parameter search space for two-model merging. Block-wise Linear Merge. To study the effect of increasing merge-space dimensionality, we introduce a block-wise variant of linear interpolation. Let the model parameters be structured per-layer tuples θ=θ(1),…,θ(L)θ=\θ^(1),…,θ^(L)\, and partition layers into k disjoint blocks ℒ=⨄j=1kℒjL= _j=1^kL_j. Define the block selector PℒjP_L_j by Pℒj(θ)=θ~(1),…,θ~(L),θ~(ℓ)=θ(ℓ),ℓ∈ℒj,,ℓ∉ℒj,P_L_j(θ)=\ θ^(1),…, θ^(L)\, θ^( )= casesθ^( ),& _j,\\ 0,& _j, cases where 0 denotes a tuple of zero tensors matching the shapes of θ(ℓ)θ^( ). The k-parameter block-wise merge is θ()=∑j=1kPℒj(xjθ0+(1−xj)θ1),=(x1,…,xk).θ( x)= _j=1^kP_L_j\! (x_j\, _0+(1-x_j)\, _1 ), x=(x_1,…,x_k). For the main comparison we use four equally sized layer blocks. This gives four independent mixing coefficients in AB and two simplex coordinates per block (eight free parameters) in ABC. Qwen3-4B has 36 transformer layers and Llama-3.1-8B has 32. 4.3 Bayesian Optimization Loop We implement a sequential multi-objective Bayesian optimization loop in BoTorch (Balandat et al. 2020). The loop begins with a Sobol design of ninitn_init initial configurations, augmented with endpoint settings that recover each source model. After each evaluation, we fit independent GP surrogates: one SingleTaskGP per objective, combined as a ModelListGP, with input normalization to [0,1]d[0,1]^d and outcome standardization; hyperparameters are learned by maximizing the summed marginal log-likelihood. New candidates are proposed by maximizing Noisy Expected Hypervolume Improvement (NEHVI) (Daulton et al. 2021) using a Sobol QMC sampler with 128 samples and multi-start acquisition optimization. For the three-model setting, acquisition optimization also respects the simplex constraints on the merge weights. We ablate qNParEGO in Appendix A.1 and find that neither acquisition function is uniformly more effective across merge operators. 4.4 Performance Across Operators (a) Qwen3 4B models (b) Llama3 8B models Figure 2: Validation hypervolume in the two-model AB setting. Running dominated hypervolume for Linear, TIES, and Block-Linear 4x on instruction following and mathematical reasoning. Solid curves show MOBO-Merge with NEHVI and dotted curves show random search; curves show the mean across ten seeds, and shaded regions indicate the mean ± one standard error. The Linear gap is small, while MOBO-Merge produces clearer gains for the higher-dimensional TIES and Block-Linear search spaces, particularly for Llama-3.1-8B. (a) Qwen3 4B models (b) Llama3 8B models Figure 3: Validation hypervolume in the three-model ABC setting. Running dominated hypervolume for Linear, TIES, and Block-Linear 4x on instruction following, mathematical reasoning, and code generation. Solid curves show MOBO-Merge with NEHVI and dotted curves show random search; curves show the mean across ten seeds, and shaded regions indicate the mean ± one standard error. The advantage of guided search is larger than in the AB setting, most notably for Llama-3.1-8B, where Block-Linear 4x attains the highest final hypervolume. Two-model results. Figure 2 shows that the benefit of MOBO-Merge depends strongly on the merge space. Linear interpolation is one-dimensional, and random search is already competitive: on the held-out Qwen AB split, random search is slightly higher than MOBO-Merge (0.0664 versus 0.0659), while MOBO-Merge has a small advantage for Llama AB (0.1036 versus 0.1017). The advantage is clearer for the four-dimensional operators. Relative to random search, MOBO-Merge raises held-out hypervolume from 0.0739 to 0.0847 for Qwen TIES and from 0.0677 to 0.0737 for Qwen Block-Linear 4x. For Llama, the corresponding improvements are from 0.0764 to 0.1145 and from 0.0880 to 0.1057. Thus, guided search is most useful when the operator exposes a sufficiently rich parameter space; it is less consequential for a single global interpolation coefficient. Three-model results. The advantage widens in the ABC setting (Figure 3). On Qwen3-4B, MOBO-Merge improves held-out hypervolume over random search for all three operators: 0.0131 versus 0.0129 for Linear, 0.0184 versus 0.0145 for TIES, and 0.0161 versus 0.0144 for Block-Linear 4x. On Llama-3.1-8B, the gains are larger: 0.0430 versus 0.0367 for Linear, 0.0429 versus 0.0055 for TIES, and 0.0634 versus 0.0289 for Block-Linear 4x. The best operator is not universal. TIES has the highest held-out hypervolume in both Qwen settings and in Llama AB, whereas Block-Linear 4x is strongest in Llama ABC. Held-out generalization. Across the 12 family, objective-set, and operator comparisons in Table 1, MOBO-Merge obtains the higher mean held-out hypervolume in 11. The only reversal is Qwen AB with Linear, where random search is higher by 0.0005 (0.7% relative). The benefit is small in a one-dimensional space but substantial for several higher-dimensional and three-objective searches. The held-out results also show that TIES is not uniformly robust to unguided parameter selection; its random-search variance and performance degradation are especially pronounced for Llama. Table 1: Mean hypervolume on the held-out partition. For each run, the validation Pareto set is re-evaluated on the complementary partition and its held-out hypervolume is computed. Cells average over 10 completed runs. Reference points differ by family and objective set, so values should be compared only within rows, not across rows. Bold denotes the best value in each row. Linear TIES Block-Linear 4x MOBO-Merge Random MOBO-Merge Random MOBO-Merge Random 4B AB 0.0659 0.0664 0.0847 0.0739 0.07368 0.0677 ABC 0.0131 0.0129 0.0184 0.0145 0.0161 0.0144 8B AB 0.1036 0.1017 0.1145 0.0764 0.1057 0.0879 ABC 0.0430 0.0367 0.0429 0.0055 0.0634 0.0289 4.5 Pareto Front Recovery Figure 4 shows representative validation Pareto fronts for Qwen seed 44 and Llama seed 42. The fronts illustrate why a single scalar summary is insufficient: different operators cover different regions of the instruction-math trade-off. In the displayed Qwen run, all three MOBO operators discover configurations whose IFEval and GSM8K scores both exceed the two annotated source checkpoints, although Block-Linear 4x has the largest hypervolume and TIES does not dominate the Linear front. In the displayed Llama run, no configuration exceeds the coordinate-wise maximum of both sources; the problem remains a genuine trade-off, and Block-Linear 4x covers more of the high-value region than Linear or TIES. These single-seed plots are illustrative; the across-seed conclusions are given by Figures 2 and 3 and Table 1. (a) Qwen3-4B (b) Llama3.1-8B Figure 4: Representative validation Pareto fronts for instruction following and mathematical reasoning. Solid lines denote MOBO-Merge, dashed lines denote random search, and black squares mark the two source models. The Qwen front contains merges that improve both annotated source scores simultaneously, whereas the Llama front retains a clear capability trade-off. 4.6 Discussion Our experiments support three main takeaways. First, model merging induces complex trade-off landscapes in which naive parameter search can waste evaluations, while MOBO-Merge concentrates evaluations in regions that expand the Pareto front. The benefit nevertheless depends on the complexity of the search space: random search remains competitive for one-dimensional Linear interpolation, whereas MOBO-Merge is more consistently advantageous for TIES, block-wise merging, and the three-model setting. Second, increasing the expressivity of the merge space through block-wise parameterizations can yield stronger trade-offs, but makes sample-efficient optimization increasingly important. In particular, Block-Linear 4x produces the best Llama ABC result and more than doubles the held-out hypervolume of random search in that setting while remaining simple and interpretable. Third, the merge operator and search strategy must be considered jointly. TIES gives the strongest held-out result in three of four family-setting combinations under MOBO-Merge, but is not uniformly robust to parameter choice and performs poorly under random selection in several Llama experiments. Validation improvements generally transfer to the held-out partition, although the small reversal for Qwen AB with Linear shows that the advantage is not universal. Overall, MOBO-Merge provides a practical framework for controllable model composition under limited evaluation budgets and is most useful for efficiently exploring expressive merge spaces. 5 Related Work 5.1 Model Merging and Weight-Space Composition Model merging11 1 Throughout this paper, we use model merging to refer to training-free composition via weight-space operations. We use the broader term model fusion for approaches that may involve additional training or non–weight-space combination. has emerged as a practical alternative to additional pretraining or joint multi-task finetuning, particularly for large language models (Yang et al. 2024). Instead of training a single monolithic model, practitioners train specialized models and combine them post hoc in weight space. Early and influential work shows that simple averaging or interpolation of fine-tuned models can improve robustness and accuracy without increasing inference cost, as in model soups (Wortsman et al. 2022). Task arithmetic further demonstrates that linear combinations of task-specific parameter updates can compose behaviors such as domain adaptation and style control (Ilharco et al. 2022). Finally, recent theoretical work suggests that model fusion and merging can admit non-vacuous generalization guarantees in certain regimes, supporting the view that merging can be both practical and principled (Kim et al. 2025). More recent work emphasizes mitigating interference between task updates and introducing structure into the merge space. TIES-Merging explicitly resolves destructive parameter conflicts to better preserve multiple capabilities (Yadav et al. 2023), while DARE rescales and prunes parameter deltas to improve compatibility between homologous language models (Yu et al. 2024). Other methods explore structured merge parameterizations, such as block-wise coefficients, to increase expressivity while keeping optimization tractable. For example, Parameter Competition Balancing (PCB) balances competing parameter contributions during merging (Du et al. 2024), and KAFA-Merge partitions layers into groups and applies block-wise linear interpolation with optimized mixing coefficients (Yuce & Amasyali 2025). Tooling efforts such as MergeKit further systematize these approaches and make large-scale merging practical (Goddard et al. 2024). While these methods improve the merge operator and its parameterization, they also expose a recurring challenge: even with a well-designed operator, selecting merge coefficients is an expensive, noisy, and often nontrivial optimization problem, especially when multiple capabilities must be preserved simultaneously. 5.2 Bayesian Optimization Bayesian optimization (BO) is a widely used framework for optimizing expensive black-box functions under limited evaluation budgets (Frazier 2018; Garnett 2023). Originally popularized for hyperparameter optimization in machine learning (Snoek et al. 2012), BO has since become a standard tool for problems characterized by costly, noisy, and gradient-free evaluations. More recently, its scope has expanded beyond model tuning to scientific discovery and experimental design in domains such as materials science, chemistry, and engineering, where each evaluation may correspond to a real-world experiment or simulation (Cosenza et al. 2022; Griffiths & Hernández-Lobato 2020; Mathern et al. 2021). Advances in probabilistic surrogate modeling and acquisition function design have further extended BO to structured search spaces, noisy objectives, and settings with multiple competing objectives (Daulton et al. 2021; Eriksson & Jankowiak 2021; Wu et al. 2020). 5.3 Bayesian Optimization for Model Merging and Fusion Recent work has explored Bayesian optimization (BO) as a principled approach for selecting merge weights when model evaluation is expensive, noisy, and gradient-free. These efforts motivate a black-box optimization view of model merging and fusion, and span a range of settings with different structural assumptions and optimization objectives. Several lines of work apply BO to model merging under a single-objective formulation. This includes approaches that use BO to merge checkpoints sampled from a single training trajectory or continual learning process, typically optimizing a single downstream objective under assumptions about the structure of the optimization path (Liu et al. 2024; Li et al. 2025b). Related work applies BO to post-hoc model combination over structured merge spaces, such as optimizing group-wise linear interpolation weights (sometimes followed by local refinement), again with the goal of identifying a single high-performing merged model under a scalar evaluation metric (Yuce & Amasyali 2025). While effective in their respective regimes, these approaches focus on selecting a single merged model and do not explicitly characterize trade-offs between heterogeneous capabilities. More closely related work adopts a multi-objective perspective on model composition. In model merging, existing approaches optimize merge parameters to recover solutions representing different trade-offs between objectives (Li et al. 2025a; Chen et al. 2025). These methods establish the value of Pareto-based search, but generally couple the optimization procedure to a particular merge operator, parameterization, or target trade-off, such as balancing model ability against computational efficiency. Related work on model fusion applies multi-objective Bayesian optimization to averaging coefficients for checkpoints obtained from a single fine-tuning process, typically balancing loss and task-specific performance to improve generalization or mitigate overfitting (Jang et al. 2024). In contrast to prior work, we study multi-objective Bayesian optimization as a common search layer across heterogeneous model-merging operators. We use a fixed optimization and evaluation protocol while varying the merge parameterization, model family, and number of source models and capability objectives. This controlled design enables direct comparison of global interpolation, interference-aware merging, and block-wise parameterizations, and reveals how the value of guided search depends on the structure and dimensionality of the merge space. We further re-evaluate configurations from the optimization-partition Pareto set on complementary held-out benchmark partitions to assess whether the discovered capability trade-offs transfer beyond the data used to guide the search. 6 Limitations MOBO-Merge relies on repeated evaluations of merged models on downstream benchmarks, which can be computationally expensive. Although the method is substantially more sample-efficient than naive search strategies, evaluation cost remains a practical bottleneck, especially when objectives require long-running inference or exhibit high variance. This limits the total number of merge configurations that can be explored in practice. Our framework focuses on weight-space merging of models that share a common architecture and parameterization. While this setting covers many practical merging scenarios for large language models, extending MOBO-Merge to heterogeneous models or non–weight-based composition methods remains an open challenge. Moreover, the quality of the recovered Pareto front depends on the choice of evaluation benchmarks and objectives, which may not fully capture all aspects of model behavior or generalization. Finally, while we demonstrate strong performance in low- to moderate-dimensional merge spaces with a small number of objectives, scaling MOBO-Merge to settings with higher-dimensional merge spaces and many competing objectives may require more expressive surrogate models and improved acquisition strategies. 7 Conclusion We introduced MOBO-Merge, a framework that formulates model merging as an expensive black-box multi-objective optimization problem and applies multi-objective Bayesian optimization to efficiently discover high-quality merges under limited evaluation budgets. By explicitly targeting Pareto-optimal solutions, MOBO-Merge exposes controllable trade-offs between competing model capabilities and replaces ad hoc merge-weight tuning with a principled optimization-based approach. Our experiments on Qwen3-4B and Llama-3.1-8B show that MOBO-Merge reliably recovers well-structured Pareto fronts when merging specialized large language models, while requiring fewer evaluations than random search. Across held-out comparisons, MOBO-Merge achieves higher mean hypervolume than random search in 11 of 12 settings, with the gains most pronounced in several higher-dimensional TIES, block-wise, and three-model searches. Moreover, the framework is merge-operator agnostic and extends from two-model to three-model merging, demonstrating that standard multi-objective Bayesian optimization provides a practical and general tool for model composition without additional pretraining. Looking ahead, promising directions include adaptive merge parameterizations that learn which layers or components require independent weights, more efficient evaluation strategies such as early stopping or multi-fidelity signals, and extensions to higher-dimensional merge spaces or larger numbers of objectives. Broader Impact Statement This paper presents work whose goal is to improve the practice of model merging in machine learning by providing a systematic and compute-efficient way to explore trade-offs between model capabilities. The proposed framework applies existing multi-objective optimization techniques to the problem of merging pretrained models and does not introduce new optimization methods or new model capabilities beyond those present in the source checkpoints. By enabling more flexible reuse of existing models, this work may help reduce the need for additional large-scale training. We do not anticipate specific societal impacts arising uniquely from this work beyond those commonly associated with the deployment of large language models. References Balandat et al. (2020) Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems, 33:21524–21538, 2020. Chen et al. (2025) Kesheng Chen, Wenjian Luo, Zhenqian Zhu, Yamin Hu, and Yiya Xi. Bambo: Construct ability and efficiency llm pareto set via bayesian adaptive multi-objective block-wise optimization. arXiv preprint arXiv:2512.09972, 2025. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Cosenza et al. (2022) Zachary Cosenza, Raul Astudillo, Peter I Frazier, Keith Baar, and David E Block. Multi-information source bayesian optimization of culture media for cellular agriculture. Biotechnology and bioengineering, 119(9):2447–2458, 2022. Daulton et al. (2020) Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. Advances in neural information processing systems, 33:9851–9864, 2020. Daulton et al. (2021) Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in neural information processing systems, 34:2187–2200, 2021. Du et al. (2024) Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim K Goh, Ho-Kin Tang, Daojing He, et al. Parameter competition balancing for model merging. Advances in Neural Information Processing Systems, 37:84746–84776, 2024. Eriksson & Jankowiak (2021) David Eriksson and Martin Jankowiak. High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence, p. 493–503. PMLR, 2021. Frazier (2018) Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811, 2018. Gao et al. (2024) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602. Garnett (2023) Roman Garnett. Bayesian optimization. Cambridge University Press, 2023. Goddard et al. (2024) Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257, 2024. Grattafiori et al. (2024) Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. Griffiths & Hernández-Lobato (2020) Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science, 11(2):577–586, 2020. Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Jang et al. (2024) Chaeyun Jang, Hyungi Lee, Jungtaek Kim, and Juho Lee. Model fusion through bayesian optimization in language model fine-tuning. Advances in Neural Information Processing Systems, 37:29878–29912, 2024. Kim et al. (2025) Taehoon Kim, Henry Gouk, Minyoung Kim, and Timothy Hospedales. Model merging is secretly certifiable: Non-vacuous generalisation bounds for low-shot learning. arXiv preprint arXiv:2505.15798, 2025. Knowles (2006) Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE transactions on evolutionary computation, 10(1):50–66, 2006. Li et al. (2025a) Bingdong Li, Zixiang Di, Yanting Yang, Hong Qian, Peng Yang, Hao Hao, Ke Tang, and Aimin Zhou. It’s morphing time: Unleashing the potential of multiple llms via multi-objective optimization. IEEE Transactions on Evolutionary Computation, 2025a. Li et al. (2024) Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. Map: Low-compute model merging with amortized pareto fronts via quadratic approximation. arXiv preprint arXiv:2406.07529, 2024. Li et al. (2025b) Mei Li, Yuxiang Lu, Qinyan Dai, Suizhi Huang, Yue Ding, and Hongtao Lu. Became: Bayesian continual learning with adaptive model merging. arXiv preprint arXiv:2504.02666, 2025b. Liu et al. (2024) Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Checkpoint merging via bayesian optimization in llm pretraining. arXiv preprint arXiv:2403.19390, 2024. Mathern et al. (2021) Alexandre Mathern, Olof Skogby Steinholtz, Anders Sjöberg, Magnus Önnheim, Kristine Ek, Rasmus Rempling, Emil Gustavsson, and Mats Jirstrand. Multi-objective constrained bayesian optimization for structural design. Structural and Multidisciplinary Optimization, 63(2):689–701, 2021. Shoemake (1985) Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, p. 245–254, 1985. Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012. Team (2025) Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, p. 23965–23998. PMLR, 2022. Wu et al. (2020) Jian Wu, Saul Toscano-Palmerin, Peter I Frazier, and Andrew Gordon Wilson. Practical multi-fidelity bayesian optimization for hyperparameter tuning. In Uncertainty in Artificial Intelligence, p. 788–798. PMLR, 2020. Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093–7115, 2023. Yang et al. (2024) Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportunities. ACM Computing Surveys, 2024. Yu et al. (2024) Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024. Yuce & Amasyali (2025) Muzaffer Kaan Yuce and Mehmet Fatih Amasyali. Kafa-merge: Model merging method with layer based bayes+ linear search. IEEE Access, 2025. Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Appendix A Appendix A.1 Ablation on Acquisition Function Choice We compare NEHVI with qNParEGO(Daulton et al. 2021) on the Qwen AB setting. There is no uniform winner. qNParEGO has the higher final mean validation hypervolume for Linear, TIES, Block-Linear 2x, and Block-Linear 6x, while NEHVI is higher for SLERP and Block-Linear 4x. The absolute NEHVI-qNParEGO gaps range from 0.0004 to 0.0029, and both guided methods generally outperform random search. Thus, the ablation supports the robustness of the MOBO formulation to the acquisition choice. The results are shown in Figure 5. (a) Linear (b) SLERP (c) TIES (d) Block Linear 2x (e) Block Linear 4x (f) Block Linear 6x Figure 5: Acquisition-function ablation on Qwen3-4B AB. Validation hypervolume over cumulative evaluations for NEHVI, qNParEGO, and random search across six merge operators. Curves show the mean across ten seeds, and shaded regions indicate the mean ± one standard error. Neither acquisition function uniformly dominates. A.2 Models We use two sets of architecture-compatible checkpoints. In each family, the base checkpoint is used as the shared reference model for TIES and is not an additional source in the AB or ABC merge. Models A, B, and C denote the source checkpoints associated with the three experimental objectives. Qwen3-4B. • Base/TIES reference: Qwen/Qwen3-4B-Base. • Model A (instruction following): Qwen/Qwen3-4B-Instruct-2507. • Model B (mathematical reasoning): prithivMLmods/Draconis-Qwen3_Math-4B-Preview. • Model C (code generation): OpenHands/CodeScout-4B. Llama-3.1-8B. • Base/TIES reference: meta-llama/Llama-3.1-8B. • Model A (instruction following): meta-llama/Llama-3.1-8B-Instruct. • Model B (mathematical reasoning): nvidia/OpenMath2-Llama3.1-8B. • Model C (code generation): tokyotech-llm/Llama-3.1-Swallow-8B-v0.5. The AB experiments merge Models A and B, while the ABC experiments add Model C. These labels describe each checkpoint’s role in our experiments and the corresponding evaluation objective; in particular, the Llama Swallow checkpoint is a broader continually pretrained model rather than a code-only fine-tune. A.3 Compute Cost Each candidate requires materializing an FP16 merged checkpoint and evaluating it on two tasks in AB or three tasks in ABC. All experiments were run on single RTX 6000 Ada GPUs with vLLM-backed evaluation. Representative end-to-end Linear runs required approximately 4 minutes per Qwen AB candidate, 6.5 minutes per Qwen ABC candidate, 6 minutes per Llama AB candidate, and 9 minutes per Llama ABC candidate. Runtime varies with the merge operator, generation lengths, and hardware, and held-out re-evaluation of the selected Pareto sets incurs additional cost. A.4 Reproducibility We use search seeds 42-51 and the same deterministic benchmark partition (split seed 4203) for every model, operator, and search method. Decoding uses temperature zero. Hypervolume curves report across-seed aggregates; the held-out table reports the mean across all 10 completed complementary-split re-evaluations. Configurations, per-candidate objective values, and optimization-to-held-out mappings have been retained and will be released, together with the code and remaining experimental artifacts, upon publication.