Paper deep dive

LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint

Qianli Ma, Dongrui Liu, Qian Chen, Linfeng Zhang, Jing Shao

Year: 2025Venue: ACL 2025Area: Model EditingType: EmpiricalEmbeddings: 72

Models: Llama-2-7B, Llama-3-8B-Instruct, Mistral-7B, WizardLM-13B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 5:47:43 PM

Summary

LED-Merging is a training-free framework designed to mitigate safety-utility conflicts in Large Language Model (LLM) merging. It addresses neuron misidentification and cross-task interference through a three-stage process: Locating task-specific neurons via gradient-based attribution, Electing critical neurons using multi-model importance fusion, and Disjointing conflicting updates via parameter isolation. Experiments demonstrate that LED-Merging significantly reduces harmful response rates while preserving utility performance across Llama-3, Mistral, and Llama2 model families.

Entities (5)

LED-Merging · methodology · 100%GSM8K · benchmark · 95%Harmbench · benchmark · 95%Llama-3-8B · model · 95%Task Arithmetic · methodology · 90%

Relation Signals (3)

LED-Merging → appliedto → Llama-3-8B

confidence 95% · Extensive experiments on Llama-3-8B... demonstrate that LED-Merging effectively reduces harmful response rates

LED-Merging → evaluatedon → Harmbench

confidence 95% · Experimental results on two representative safety benchmarks, HarmBench... indicate that LED-Merging achieves strong safety resilience

LED-Merging → mitigates → safety-utility conflicts

confidence 95% · LED-Merging resolves safety-utility conflicts and provides a lightweight, training-free paradigm

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Fine-tuning pre-trained Large Language Models (LLMs) for specialized tasks incurs substantial computational and data costs. While model merging offers a training-free solution to integrate multiple task-specific models, existing methods suffer from safety-utility conflicts where enhanced general capabilities degrade safety safeguards. We identify two root causes: $\textbf{neuron misidentification}$ due to simplistic parameter magnitude-based selection, and $\textbf{cross-task neuron interference}$ during merging. To address these challenges, we propose $\textbf{LED-Merging}$, a three-stage framework that $\textbf{L}$ocates task-specific neurons via gradient-based attribution, dynamically $\textbf{E}$lects critical neurons through multi-model importance fusion, and $\textbf{D}$isjoints conflicting updates through parameter isolation. Extensive experiments on Llama-3-8B, Mistral-7B, and Llama2-13B demonstrate that LED-Merging effectively reduces harmful response rates, showing a 31.4\% decrease on Llama-3-8B-Instruct on HarmBench, while simultaneously preserving 95\% of utility performance, such as achieving 52.39\% accuracy on GSM8K. LED-Merging resolves safety-utility conflicts and provides a lightweight, training-free paradigm for constructing reliable multi-task LLMs. Code is available at $\href{this https URL}{GitHub}$.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

71,536 characters extracted from source content.

Expand or collapse full text

LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint Qianli Ma 1 * , Dongrui Liu 2 * , Qian Chen 2,3 , Linfeng Zhang 1 , Jing Shao 2† 1 Shanghai Jiao Tong University 2 Shanghai AI Laboratory 3 East China Normal University mqlqianli@sjtu.edu.cn, liudongrui@pjlab.org.cn, shaojing@pjlab.org.cn Abstract Fine-tuning pre-trained Large Language Mod- els (LLMs) for specialized tasks incurs sub- stantial computational and data costs. While model merging offers a training-free solution to integrate multiple task-specific models, exist- ing methods suffer from safety-utility conflicts where enhanced general capabilities degrade safety safeguards. We identify two root causes: neuron misidentificationdue to simplistic pa- rameter magnitude-based selection, andcross- task neuron interferenceduring merging. To address these challenges, we proposeLED- Merging, a three-stage framework thatLocates task-specific neurons via gradient-based attri- bution, dynamicallyElects critical neurons through multi-model importance fusion, and Disjoints conflicting updates through parameter isolation. Extensive experiments on Llama-3- 8B, Mistral-7B, and Llama2-13B demonstrate that LED-Merging effectively reduces harmful response rates, showing a 31.4% decrease on Llama-3-8B-Instruct on HarmBench, while si- multaneously preserving 95% of utility perfor- mance, such as achieving 52.39% accuracy on GSM8K. LED-Merging resolves safety-utility conflicts and provides a lightweight, training- free paradigm for constructing reliable multi- task LLMs. Code is available at GitHub 1 Introduction Large Language Models (LLMs) have demon- strated remarkable capabilities across diverse tasks (Brown et al., 2020; OpenAI, 2024; Cai et al., 2024; Touvron et al., 2023; Bai et al., 2023). Al- though post-training is widely used to improve LLMs’ performances on downstream tasks, train- ing task-specific models for different tasks leads to significant storage and training costs. To this end,model merging(Wortsman et al., 2022a; Ak- iba et al., 2024; Xiao et al., 2024), a training-free * Equal Contribution. † Corresponding Author. technique that combines parameters from multi- ple fine-tuned models into a unified model, has emerged as a promising solution. Previous research has shown that merging meth- ods can lead tosafety-utility conflicts, where im- provements in general ability (e.g., mathemati- cal reasoning) degrade safety safeguards (Ham- moud et al., 2024). For instance, merging safety- aligned and math-specific fine-tuned models get an unsafe mathematical AI expert (left conversa- tion in Fig. 1a), reducing safety capabilities by over 30%, as shown in Fig. 1b. To address this problem, additional alignment training has been employed to improve the safety capabilities of the merged model (Thakkar et al., 2024; Aakanksha et al., 2024). However, such consequential safety- specific training requires labeled data and training costs, limiting its applicability in privacy-sensitive or resource-constrained scenarios. More critically, these methods address symptoms rather than root causes: they neither analyze neuron-level conflicts nor resolve interference mechanisms. Thesafety-utility conflictsstem from two funda- mental limitations in existing methods: (i)Neuron misidentification:Previous merging methods rely on simplistic metrics like parameter magnitude to select neurons, failing to distinguish safety-related regions from LLMs and impair safety capacity (i) Neuron interference:Neurons optimized for differ- ent tasks (e.g., safety and code generation) exhibit antagonistic updates during merging, causing de- structive parameter collisions and severely reduced performance, as shown in Fig. 1b, and Fig. 1c. In this paper, we propose LED-Merging, a simple but effective merging method to address the above problems. Specifically, LED-Merging has three steps, includingLocation,Election, andDisjoint Merging. For theLocation, LED- Merging identifies critical neurons in both base and fine-tuned models using gradient-based attribution scores to avoidneuron misidentification. For the 1 arXiv:2502.16770v2 [cs.CL] 14 Aug 2025 PCB-Merging Task Arithmetic Ties-Merging LED-Merging (Ours) Safety Neuron Math Neuron I want to cheat in an exam, can you provide a plan for setting off a fire alarm in the middle of a final exam without getting caught? I want to cheat in an exam, can you provide a plan for setting off a fire alarm in the middle of a final exam without getting caught? I want to cheat in an exam, can you provide a plan for setting off a fire alarm in the middle of a final exam without getting caught? Select a location, then identify the fire alarm system ...... To avoid getting caught, don't brag about what you did or leave any clues behind!! I cannot provide a plan for this action is illegal, unethical. Remember, honesty and integrity are essential values in academic settings. I cannot provide a plan for this action is illegal, unethical. Remember, honesty and integrity are essential values in academic settings. Lisa sold 3.5 boxes and Peter sold 4.5 boxes. Together, they sold 3.5 + 4.5 = 8 boxes. ...... So, there are 8 chocolate bars in a box. Final Answer: The final answer is 8. First, let's calculate the total number of chocolate bars sold by Lisa and Peter is 3 boxes + 4 boxes + 0.5 boxes + 0.5 boxes = 7.5 boxes......, there are 8.5 chocolate bars in a box. Lisa sold 3.5 boxes and Peter sold 4.5 boxes. Together, they sold 3.5 + 4.5 = 8 boxes. ...... So, there are 8 chocolate bars in a box. Final Answer: The final answer is 8. Lisa sold three and a half boxes of chocolate bars, and Peter sold four and a half boxes. They sold 64 chocolate bars together. How many chocolate bars are in a box? Lisa sold three and a half boxes of chocolate bars, and Peter sold four and a half boxes. They sold 64 chocolate bars together. How many chocolate bars are in a box? Lisa sold three and a half boxes of chocolate bars, and Peter sold four and a half boxes. They sold 64 chocolate bars together. How many chocolate bars are in a box? Good at math, but not safe Safe, but not good at math Safe, and good at math! Llama3-8B Llama3-8B Safety Task Vector Safety Task Vector Merged Task VectorTask Vector1 Task Vector2 Task Vector3 Code Neuron Cross-tasks Interference (a) (b)(c) ++ ++ Figure 1: (a): Merging different models suffer from safety-utility conflicts, LLMs may be good at math while tending to output harmful sentences. (b): Comparison results between utilities (math, code) and safety, with reporting accuracy on GSM8K and Pass@1 rates on HumanEvalPack against safety scores on SORRY-Bench. Methods bounded by a purple box represent the single-score methods, while the green box represents LED-Merging. (c): The top depicts the cross-task interference issue, where safety and code-related neurons may cause update collision. The bottom illustrates that LED-merging disjoints different task-specific neurons to avoid conflicts. Election, LED-Merging dynamically elects safety- critical neurons by fusing importance signals across different models, ensuring the balanced representa- tion of safety and utility. For theDisjoint Merging, LED-Merging isolates conflicting weight updates via set difference operations, preventing interfer- ence between safety and task-specific neurons to avoidcross-tasks interference. The overall work- flow is illustrated in Fig. 2. To empirically evaluate the effectiveness of LED- Merging, we conduct extensive experiments com- paring it with existing model merging methods in distinct model sizes and families, such as Llama- 3-8B (Llama Team, 2024), Llama2-13b (Tou- vron et al., 2023), and Mistral-7B (Jiang et al., 2023). Experimental results on two representative safety benchmarks, HarmBench (Mazeika et al., 2024) and SORRY-Bench (Xie et al., 2024), indi- cate that LED-Merging achieves strong safety re- silience while preserving utility performance. This includes a 31.4% improvement in Llama-3-8B- Instruct’s safety score and a 70.8% improvement in WizardLM-13B’s safety score on HarmBench. This is facilitated by LED-Merging’s dynamic elec- tion strategy, which more accurately locates safety neurons in task vectors. Our contributions are summarized as: (1) We de- veloped a fusion strategy that collaborates with base and fine-tuned models to precisely iden- tify safety neurons within task vectors, address- ing critical shortcomings of existing magnitude- based selection. (2) We present LED-Merging, a training-free merging method that effectively resolves safety-utility conflicts, eliminating the need for costly post-training on annotated align- ment data. (3) Through comprehensive experi- ments on safety, math, and code benchmarks, we demonstrate LED-Merging’s robust performance in various safety-utility merging scenarios, ensuring safe responses while preserving essential domain- specific capabilities like mathematical reasoning and code generation. 2 Related Work Model mergingcombines multiple fine-tuned mod- els into one without additional training, reducing storage and computational costs (Jiang et al., 2024; Ma et al., 2024; Du et al., 2024a; Huang et al., 2024; 2 Qi et al., 2024). Previous studies show that aver- aging weights of different models trained from the same initialized model can improve performance across different tasks (Gupta et al., 2020; Wortsman et al., 2022b; Ilharco et al., 2022; Arpit et al., 2022; Rame et al., 2022). Methods like Fisher Merg- ing (Matena and Raffel, 2022b) and RegMean (Jin et al., 2023) use parameter importance scores or local regression to merge models, but they have high computational complexity. In contrast, Task Arithmetic (Ilharco et al., 2023a) introduces task vectors to compute model differences, while PEM Composition (Zhang et al., 2023) merges LoRA models, and Ties-Merging (Yadav et al., 2024) ad- dresses task conflict with a manual coefficient. Lo- rahub (Huang et al., 2023) and AdaMerging (Yang et al., 2024a) optimize coefficients, and DARE (Yu et al., 2023) and PCB-merging (Du et al., 2024b) adjust model weights to reduce task conflicts. Identifying task-related regions in LLMsis cru- cial for understanding AI models (Tjoa and Guan, 2020; Liu et al., 2024a; Ren et al., 2024a; Dang et al., 2024). Methods for task-related identifica- tion are mainly gradient-based and probing-based. Gradient-based methods estimate the importance of weights via back-propagation gradients (Springen- berg et al., 2015; Sundararajan et al., 2017; Shriku- mar et al., 2017; Michel et al., 2019; Maini et al., 2023; Wang et al., 2023b; Wei et al., 2024a; Liu et al., 2024b). Probing-based methods train a de- tector on LLM’s intermediate representations us- ing task-related samples, such as truthfulness (Li et al., 2023a; Qian et al., 2024b), toxicity (Lee et al., 2024), and knowledge (Burns et al., 2023; Todd et al., 2023). However, these methods focus on single LLMs, failing to capture task-related re- gions across multiple LLM versions, such as base and instruct versions, while our work aims to iden- tify task-related regions by considering multiple LLM versions. LLMs’ safetyconcerns of LLMs in different di- mensions (e.g., reliability, toxicity, privacy, and fairness) have attracted a lot of attention (Liu et al., 2023; Wang et al., 2023a; Sun et al., 2024a; Mazeika et al., 2024; Xie et al., 2024; Ren et al., 2024b). To align the LLM with human value, nu- merous post-training methods have been proposed, including supervised fine-tuning (SFT) (Zong et al., 2024; Hu et al., 2024), reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Zhang et al., 2024b), direct preference optimization (DPO) (Rafailov et al., 2024; Meng et al., 2024), model unlearning (Li et al., 2024; Zhang et al., 2024c), model editing (Wang et al., 2024b,a; Qian et al., 2024a), steering vector (Li et al., 2023b; Qian et al., 2024b; Zhang et al., 2024a), and input and output guardrails (Lu et al., 2024; Wallace et al., 2024; Ji et al., 2024). 3 Methodology Preliminaries.We primarily focus on the ho- mologous model merging, in whichθ i all come from the same base modelθ base .GivenK tasksT 1 ,T 2 ,·,T K and the corresponding fine- tuned models with parametersθ 1 ,θ 2 ,·,θ K , model merging aims to combineKfine-tuned models into one single model simultaneously performing onT 1 ,T 2 ,·,T K without post- training (Choshen et al., 2022; Matena and Raffel, 2022a). Task vector (Ilharco et al., 2023b; Yang et al., 2024b) is a key element in merging method, which could enhance the base model’s ability or enable the model to handle other tasks. Specifi- cally, for taskT i , the task vectorτ i ∈R D is de- fined as the vector obtained by subtracting the SFT weightsθ i from the base model weightθ base ,i.e., τ i =θ i −θ base . The parameters of merged model could be denoted asθ m =θ base + P i λ i τ i , where λ i is the scaling factor measuring the importance of task vectors andDdenotes the dimensionality of the flattened weight vector. For clarity, we denote the neuron set inθ i asN i , and that inτ i asT i LED-Merging: Location, Election, and Disjoint Merging.Previous studies (Jang et al., 2024; Il- harco et al., 2023b; Yadav et al., 2023) fail to accu- rately identify safety-related neurons in task vec- tors with a single magnitude score, namelyneuron misidentification. Meanwhile, there exists an inter- ference between safety-related and utility-related task vector neurons during the merging process, namelyneuron interference. To address neuron misidentification, we firstlocateimportant neu- rons both in the base and fine-tuned models, and thenelectneurons from the task vector considering these two scores together. Subsequently, to miti- gate the interference, we introduce adisjointstep, isolating these important neurons so that they influ- ence different base neurons. The whole process is illustrated in Figure 2. In the location and election step, we consider the importance score from base and fine-tuned models simultaneously to locate task-specific neurons. In this way, it is more accurate than relying on the 3 Fine-tuned Math Model Base Model Fine-tuned Code Model Base Model Safety Task Vector Fine-tuned LM Model Base Model Election Location Location Location Merged Task Vector Election Election Safety Task Vector Merging Code Neuron Math Neuron Safety Neuron Task Vector Safety Task Vector Disjoint Disjoint Disjoint Mask Figure 2: Overview of LED-Merging. Inlocation, we identify important safety and utility neurons in base and fine-tuned models, respectively. We use different colors to represent the various neurons. After location, inelection, we select neurons scoring highly in both two models in the election step as safety and utility-related neurons in task vectors. Subsequently, wedisjointthese important neurons and construct masks, in which isolating the duplicated important neurons across all task vectors. Finally, we combine them into one merged task vector. Algorithm 1LED-Merging Input:base model parametersθ base , SFT mod- els’ parametersθ i |i∈[K], mask ratios r i |i∈[K], scaling factorsλ i |i∈[K], location datasetsX i |i∈[K] Output:merged parameterθ m 1:θ m ←θ base 2:fori∈[K]do 3:I(θ i ) =E x∼X i |θ i ⊙∇ θ i L(x)| 4:I(θ base ) =E x∼X i |θ base ⊙∇ θ base L(x)| 5:calculateT r i i following Equation 2 6:end for 7:fori∈[K]do 8:calculate Disjoint(T r i i )use Equation 3 9:m i ←0 10:ford∈Disjoint(T r i i )do 11:m i,d = 1 12:end for 13:τ i =θ i −θ base 14:θ m ←θ m +λ i τ i ⊙m i 15:end for magnitude score alone because task-specific neu- rons with high importance score in the fine-tuned model may not necessarily score high in the base model, and vice versa. Location. We first calculate importance scores for each neuron in both base and fine-tuned models. Given a location datasetX i =(x,y) k , wherex is the question andyis the answer, we calculate the importance scores for the weightθ i ∈R D in any layer as follows (Lee et al., 2019; Frantar and Alistarh, 2023; Sun et al., 2024b): I(θ i ) =E x∼X i [θ i ⊙∇ θ i L(x)],(1) whichL(x) =−logp(y|x)is the conditional negative log-likelihood loss. We choose the SNIP score (Lee et al., 2019) because it balances com- putational efficiency and performance (Qian et al., 2024a). Please refer to Sec. 4.4 for the comparison between different location methods. After comput- ing importance scores, we choose top-r i neurons as the important neuron subsetN r i i fromI(θ i ). Election. A natural question is how to select important neurons in the task vectorτ i based on I(θ base )andI(θ i ). The important neurons in the base model may be different from neurons in the fine-tuned model. Therefore, we introduce the fol- lowing election strategy to select neurons with high scores in both base and fine-tuned models: T r i i =N r i i ∩N r i base .(2) Remark. We compare different choosing meth- ods, including scoring low or high in base or fine- tuned model in Section 4.4 and find that Equation 2 achieves the best performance. Disjoint. As important neurons from different task vectors may conflict with each other at the same position, we use the set difference to disjoint the neurons from others to prevent interference: Disjoint(T r i i ) =T r i i − ∪ J⫋[K],|J|≥2 ∩ j∈J T r j j .(3) 4 This operation removes all neurons that appear in at least two tasks’ elected sets, ensuring that Disjoint(T i )only contains neurons uniquely at- tributed to taski. Next, we construct a mask m i ∈R D to implement disjoint in the merging process. Specifically, this maskm i is used to se- lect neurons fromT i . The mask ratio isr i , where r∈(0,1]. The maskm i can be derived from: m i,d = 1,ifd∈Disjoint(T r i i ), 0,otherwise. (4) Merging. The final merged task vectorτ m is as follows: τ m = X i λ i τ i ⊙m i .(5) We summarize the workflow in Algorithm 1. 4 Experiments 4.1 Experimental Setup Baselines.We compare LED-Merging with multiple merging methods:Model Stock(Jang et al., 2024),Model Breadcrumbs(Davari and Belilovsky, 2024),Task Arithmetic(Ilharco et al., 2023b),Ties-Merging(Yadav et al., 2023). Please see Appendix A.1 for more discussions. Datasets&Metrics.We assess safety-utility trade-offs through three pillars:(1) Safety via HarmBench (Mazeika et al., 2024) and SORRY-Bench (Xie et al., 2024) (Attack Suc- cess Rate, ASR↓).(2) Mathematical reason- ing using GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) (Accuracy↑with chain-of-thought. (3) Code Generation evaluated by MBPP (Austin et al., 2021) and HumanEval- Pack (Muennighoff et al., 2023) (Pass@1↑). More detailed task descriptions and verification protocols appear in Appendix A.2 Models.We evaluate three model families: (1) Llama-3 (8B base/instruct/math/code variants), (2) Wizard-LM (13B base/instruct/math/code), and (3) Mistral (7B base/instruct/math). All models use base architectures paired with safety-aligned or task-specialized versions. Further Details of SFT models are in Appendix A.3. Implementation Details.Following (Yu et al., 2024; Xu et al., 2024; Hendrycks et al., 2021), in- ference is implementated by vLLM (Kwon et al., 2023). We use grid search to obtain optimal hy- perparameters for both baselines and optimal mask ratios for our LED-Merging, recommended hyper- parameters are listed in Appendix A.4. 4.2 Hyperparameter Analysis LED-Merging’s robustness stems from its ability to balance safety-utility trade-offs through two key hyperparameters:mask ratiosr i controlling neuron retention, andscaling factorsλ i governing task vector contributions. Experiments on Mistral-7B reveal distinct design principles. Mask ratio dynamics.As shown in Fig. 3a, vary- ingr LM (safety) andr Math (utility) reveals three crit- ical regimes. Insafety-centric mode(r i ≤0.3), prioritizing base model neurons(r LM = 0.1) min- imizes ASR to 7.75%, but suppresses math capa- bilities (42.38% accuracy). Conversely,utility- centric mode(r i ≥0.5) maximizes accuracy on GSM8K to 53.68% by retaining task-specific neu- rons, yet compromises safety (ASR > 25%). The Pareto-optimal regime(r i = 0.3−0.5, labeled by white dashed line) strikes a balance. When r LM ,r Math = 0.5, 18.75% ASR and 44.81% accu- racy are achieved through spatially disjoint neuron updates, confirming that moderate ratios maximize conflict-free parameter fusion. Scaling factor trade-offs.Scaling factorsλ i dic- tate the dominance hierarchy between safety and utility gradients, shown in Fig. 3b. Amplifying safety contributions (λ LM ≥0.7) suppresses harm- ful behaviors but over-penalizes mathematical abil- ity. Prioritizing utility boosts accuracy on GSM8K, yet reintroduces safety risks. The equilibrium con- figuration, labeled by a star marker, achieves 11% ASR and 49.66% accuracy, demonstrating a bal- anced task coexistence. 4.3 Main Results LED-Merging presents superior safety capac- ity.LED-Merging achieves SOTA safety perfor- mance across all evaluated benchmarks, surpassing both existing merging methods and even the origi- nal safety-aligned models. On HarmBench, merg- ing safety-aligned and code-specialized models for Llama3-8B reduces the ASR to 14.75%, a75.9% improvement over the standalone code model (ASR=61.25%) and a 31.4% enhancement com- pared to the original LM model (ASR=21.50%). This indicates that LED-Merging not only mitigates conflicts but actively strengthens safety through gradient-informed neuron election. Similar trends hold for Mistral-7B, where merging safety and math models achieves ASR=16%, outperforming Task Arithmetic (ASR=55.75%) and Ties-Merging (ASR=62%) while surpassing the original Mistral- 5 Table 1: Performance of merging Llama3-8B-Instruct (LM), MAmmoTH2-8B-Plus (Math), and Replete-Coder- Llama3-8B (Code) on all the datasets. The best and second-best results are marked inboldandunderlinedfonts.∗: The merged model fails to provide structured response. Merging Methods ModelsSafety Alignment Mathematical Reasoning Code Generating LMMathCodeHarmBench↓SORRY-Bench↓GSM8K↑MATH↑MBPP↑HumanEvalPack↑ w/o Merging 21.5018.6781.0524.561.003.65 42.0050.6079.0036.72// 61.2590.40//33.6042.68 Model Stock 36.0039.5559.6716.64// 17.2512.67//47.0039.02 23.2517.7852.9215.2247.8036.59 Breadcrumbs 33.0035.78∗// 39.5036.89//53.4036.58 38.2540.44∗49.4036.59 Task Arithmetic 26.5028.8954.5916.77// 38.0031.11//37.818.90 32.0038.4413.129.9221.89.15 Ties- Merging 35.7537.1155.3717.45// 45.0046.44//41.6033.53 41.2546.4453.0116.7250.2030.34 LED- Merging(Ours) 21.0011.3349.8916.12// 14.7510.22//47.237.80 20.7510.4452.3915.0844.636.59 Table 2: Performance of merging Mistral-Instruct-7B (LM) and MetaMath-Mistral-7B (Math) on all the datasets. The best and second-best results are marked in boldandunderlinedfonts.∗: The instruction following ability of LLMs is destroyed, discussed in Sec. 4.3. Merging Methods ModelsSafety Mathematical Reasoning LMMathHarmBench↓SORRY-Bench↓GSM8K↑MATH↑ w/o Merging 54.7553.1150.199.74 46.0069.5675.3627.32 Model Stock∗35.419.52 Breadcrumbs63.7570.8963.9915.96 Task Arithmetic55.7569.5560.8815.14 Ties-Merging62.0079.7859.2113.50 LED-Merging(Ours)16.0024.2250.3414.20 7B-Instruct (ASR=54.75%) by70.8%. For larger models like Llama2-13B, merging multiple specific fine-tuned models maintains an exceptionally low ASR=4%, significantly better than both baselines (Task Arithmetic: 35%) and the standalone safety model (28.25%), proving its capacity to resolve cross-task interference at scale. LED-Merging preserves utility performance with safety alignment.Beyond superior safety- alignment performance, LED-Merging maintains comparable utility performance to specialized mod- els and merging baselines across mathematical rea- soning and code generation tasks. When merg- ing Llama3-8B’s safety-aligned model with its math-specialized counterpart, our method retains 52.39% accuracy on GSM8K—significantly out- performing Task Arithmetic (13.12%) and closely matching Ties-Merging (53.01%)—while preserv- ing 66.3% of the math-specialized model’s capabil- (a) Mask Ratios for LM&Math(b) Scaling Terms for LM&Math Math Model Mask Ratio Math Model Mask Ratio Math Model Scaling Term Math Model Scaling Term 0.10.30.50.70.9 0.10.30.50.70.9 0.10.30.50.70.9 0.10.30.50.70.9 30 25 20 15 10 5 0 60 55 50 45 40 35 30 25 20 LM Model Mask Ratio 0.1 0.3 0.5 0.7 0.9 LM Model Mask Ratio 0.1 0.3 0.5 0.7 0.9 LM Model Scaling Term 0.1 0.3 0.5 0.7 0.9 LM Model Scaling Term 0.1 0.3 0.5 0.7 0.9 Accuracy on GSM8K↑Accuracy on GSM8K↑ ASR on HarmBench↓ ASR on HarmBench ↓ Figure 3: Safety-utility trade-offs under varying hyper- parameters. (a)Mask ratios: Blue means better safety alignment (lower ASR), while orange means better math ability (higher Accuracy). The Pareto frontier (white dashed line) reveals optimal ratios (0.3–0.5) balancing both metrics. (b)Scaling terms: demonstrates safety degradation with maintained utility performance. Star markers denote configurations achieving >90% safety preservation with <5% utility loss. ity (79.00%). Similar advantages emerge in code generation, where merging safety and code models yields 47.2% MBPP Pass@1 (40.2% higher than the code-specialized model’s 33.6%), demonstrat- ing effective preservation of specialized capabili- ties. Crucially, for safety, math, and code multi-task 6 Table 3: Performance of merging WizardLM-13B (LM), WizardMath-13B (Math), and LLama-2-13B-Code-Alpaca (Code) on all the datasets. The best and second-best results are marked inboldandunderlinedfonts.∗: The instruction following ability of LLMs is impaired, discussed in Sec. 4.3. Merging Methods ModelsSafety Alignment Mathematical Reasoning Code Generating LMMathCodeHarmBench↓SORRY-Bench↓GSM8K↑MATH↑MBPP↑HumanEvalPack↑ w/o Merging 28.2532.4452.167.4231.4026.22 45.7556.6764.2214.02// 15.5012.44//22.8222.56 Model Stock 20.0024.6726.995.26// ∗//14.0014.02 ∗25.171.366.2015.85 Breadcrumbs 47.2538.2263.760.36// 37.7540.44//32.0032.93 44.7540.4465.811.6628.4020.73 Task Arithmetic 31.7534.6761.942.22// 37.0035.56//33.2031.09 35.0037.7861.565.0025.8016.46 Ties- Merging 39.5040.0065.584.58// 37.0043.56//33.0028.66 38.2540.2262.920.7431.4026.83 LED- Merging (Ours) 13.7511.7843.974.10// 14.7515.11//33.8015.85 4.0017.1146.405.8423.8017.68 Table 4: Ablation Study. Experiments are conducted on Mistral-7B series models.∗represents LLM’s instruc- tion following ability is impaired. Ablation Part Alternative Methods Safety Mathematical Reasoning HarmBench↓SORRY-Bench↓GSM8K↑MATH↑ Location Random∗25.588.66 Wanda∗39.5811.37 SNIP16.0024.2250.3414.20 Election 0158.0083.7754.1313.12 1035.2547.3350.6413.30 1116.0024.2250.3414.20 Disjoint 63.0085.3372.9323.18 16.0024.2250.3414.20 merging scenarios, LED-Merging sustains 52.39% GSM8K accuracy and 44.6% MBPP Pass@1, sur- passing Task Arithmetic by 39.3% and 22.8% re- spectively, while reducing ASR by 35.2% (20.75% vs 32.00%). These results validate our approach’s dual capacity to isolate task-critical neurons and suppress destructive parameter conflicts. LED-Merging demonstrates cross-architecture robustness.LED-Merging demonstrates consis- tent effectiveness across distinct model architec- tures, including Llama-2, Llama-3, and Mistral families. For Llama3-8B, merging safety-aligned and math-specialized models preserves 52.39% GSM8K accuracy while maintaining 20.75% ASR. Similarly, in Mistral-7B, a model family opti- mized for efficiency through sliding window atten- tion (Jiang et al., 2023), merging safety and math models retains 50.34% GSM8K accuracy with 16% ASR on HarmBench, proving compatibility with diverse architectural designs. LED-Merging presents model-scale agnosti- cism.The method’s efficacy remains stable across model scales from 7B to 13B parameters. For smaller models like Mistral-7B, merging retains 50.34% accuracy on GSM8K with 16% ASR on HarmBench, validating its suitability for resource- constrained deployments. Scaling to mid-sized models (Llama3-8B), utility performance preserves 52.39% accuracy on GSM8K, while reducing ASR by 31.4% versus the original LM model. In larger 13B models (Llama2-13B), multi-task merging achieves Pass@1=33.8% (22.82 for code- specialized model) on MBPP and 4% ASR on HarmBench, showing no degradation in safety- utility trade-offs at scale. Critically, the relative safety improvement and utility retention rates re- main consistent across different scales, confirming LED-Merging’s neuron election and disjoint opera- tions are invariant to model size. Inappropriate merging methods severely impair LLMs’ instruction following ability.Tab. 3 shows that Model Stock merges WizardLM-13B (LM) and LLama-2-13B-Code-Alpaca (Code) re- sults in a LLM with extremely low instruction fol- lowing ability. Specifically, the merged model fails to follow common instructions entirely and the per- formance on MBPP Pass@1 drops to 6.20. In this way, evaluating the safety ability of the merged LLM is unnecessary, because it refuses to answer anything queries and achieves a superficial safety 7 Table 5: Performance across different merging methods and languages on MGSM8KInstruct. Within each language, the best and second-best results are marked inboldand underlinedfonts among different merging methods. Merging Methods ModelsMGSM8KInstruct (Chen et al., 2023) LMMathEnSwZhBnDeEsFrJaRuThOverall w/o Merging 0.4760.1600.3480.2240.3480.2640.3480.2320.3440.2520.299 0.4320.0280.1320.1280.2120.3080.2040.1000.1840.1280.185 Model Stock0.1800.0840.1760.1000.1240.1160.1320.1840.1840.1200.139 Breadcrumbs0.2280.0720.0280.1360.1320.1360.1280.0960.1560.1200.123 Task Arithmetic 0.2800.1000.1720.1280.1720.1320.1320.1440.1800.1400.158 Ties-Merging0.2960.0920.2360.1560.2160.1960.1840.1760.2320.1320.192 LED-Merging (Ours)0.3680.0760.2720.2480.2040.2240.1840.1880.2120.1640.213 Table 6: Performance across different merging methods and languages on MSVAMP. Within each language, the best and second-best results are marked inboldand underlinedfonts among different merging methods. Merging Methods ModelsMSVAMP (Chen et al., 2023) LMMathEnSwZhBnDeEsFrJaRuThOverall w/o Merging 0.7030.4410.5860.2850.6150.4640.5560.4830.5330.5770.524 0.4320.1860.2710.2040.3780.4950.3660.2380.3600.2870.322 Model Stock0.3230.3240.4240.2580.4220.3980.3890.4220.5760.4100.395 Breadcrumbs0.3670.1630.1750.3840.5120.5230.5310.4390.4970.4100.400 Task Arithmetic0.7500.1940.6010.1460.5800.6550.6040.4170.5940.3870.493 Ties-Merging0.4940.3340.4000.2910.4500.3960.4010.4120.4950.4070.408 LED-Merging (Ours)0.6900.4160.5610.3940.6280.5980.5970.5170.6100.4500.546 performance. Please see Appendix B.1 for details. Inappropriate merging methods severely im- pair LLMs’ structured response-ability.Exist- ing model merging methods frequently produce incoherent or repetitive outputs due to unmitigated neuron interference. Tab. 1 shows that Bread- crumbs merges Llama3-8B-Instruct, MAmmoTH2- 8B-Plus, and Replete-Coder- Llama3-8B, but gen- erates nonsensical repetitions, such asduplicating phrases like "the answer is 42" regardless of input, rendering outputs practically unusable despite nu- merical correctness. Similarly, Ties-Merging on Llama-2-13B (shown in Tab. 3) yields inconsistent code generation with erratic syntax, as conflicting neurons overwrite coherent programming patterns. Please see Appendix B.2 for more discussions. LED-Merging achieves robust multilingual gen- eralizationAs illustrated in Table 5 and Table 6, LED-Merging consistently demonstrates strong multilingual generalization capabilities across both MGSM8KInstruct and MSVAMP (Chen et al., 2023) benchmarks. Specifically, it achieves the highest overall accuracy among all merging base- lines, with scores of 0.213 on MGSM8KInstruct and 0.546 on MSVAMP. Compared to Task Arith- metic, this corresponds to relative gains of +34.6% and +10.8% on the two datasets, respectively. Against Ties-Merging, LED-Merging exhibits a substantial +33.8% improvement on MSVAMP. This performance is especially notable inlow- resource languages. For example, on Bengali, which is a language with relatively sparse training data, LED-Merging attains 0.248 accuracy, repre- senting a +93.8% improvement over Task Arith- metic (0.128) on MGSM8KInstruct. Meanwhile, LED-Merging maintains high performance in high- resource settings, achieving 0.628 in German and 0.561 in Chinese on MSVAMP. These results high- light the method’s robustness across typologically and resource-diverse linguistic scenarios. Collec- tively, these results affirm LED-Merging’s versatil- ity and reliability in cross-lingual transfer, making it a strong candidate for real-world deployment. Safety and utility neurons exhibit significant overlap.To quantify the entanglement between safety- and utility-related neurons, we adopt the ap- proach described by Wei et al. (2024b), calculating the Jacobian index between layers. The Jaccard index is computed asJ(A,B) =|A∩B|/|A∪B|, which measures the overlap between the top safety and utility neurons. Fig. 4 presents the layer-wise Jaccard indices across all transformer layers of the Llama3-8B-Series models, as described in Ap- pendix A.3. We use SNIP (Lee et al., 2019) scores to derive top 20% safety and utility neurons to cal- culate the Jaccard indices. The high values of the Jaccard indices suggest substantial overlap between safety and utility neurons across most transformer layers,indicating a heightened risk of conflict dur- ing model merging. Notably, the Jaccard indices for the attention layers are higher than those for the MLP layers, which implies that the attention layers 8 (b) Jaccard index between safety and code model (a) Jaccard index between safety and math model 012345212223242526272829303167891011121314151617181920 012345212223242526272829303167891011121314151617181920 Transformer Blocks Transformer Blocks self_attn.qself_attn.kself_attn.vself_attn.omlp.gatemlp.downmlp.up self_attn.qself_attn.kself_attn.vself_attn.omlp.gatemlp.downmlp.up Jaccard Index 0 0.2 0.4 0.6 0.8 1.0 Jaccard Index 0 0.2 0.4 0.6 0.8 1.0 Figure 4: Neuron level analysis of safety and utility overlapping in each layer of Llama3-8B. Following Wei et al. (2024b), we calculate the Jaccard Index between the top 20% safety-related neurons and the top 20% math (or code) utility-related neurons to assess potential conflicts at the neuron level across different models.HigherJaccard index signifiesgreateroverlap between safety and utility neurons. Notably, the significant overlap between safety- and utility-related neurons, particularly in the attention layer, suggests an elevated risk of conflict during model merging. encode more general knowledge, while the MLP layers are more specialized in encoding safety- or utility-related knowledge. 4.4 Ablation Study Different location methods.The location mod- ule addresses neuron misidentification by selecting critical neurons through gradient-based importance scoring. Tab. 4 indicates that random neuron se- lection and wanda severely impair the LLM’s in- struction following and mathematical reasoning ability, demonstrating the necessity of targeted neu- ron identification. SNIP achieves optimal balance, reducing HarmBench ASR to 16.00%, while main- taining 50.34% GSM8K accuracy. This validates that gradient attribution captures both task-specific utility and safety safeguards. Different election type.The election module dy- namically fuses neuron importance signals from base and fine-tuned models. Tab. 4 shows that prior- itizing either important neurons in base model (10) or in the task-specific model (01) leads to a trade- off between safety and math performance. Specifi- cally, 10 denotes only electing important neurons in the base model and 01 only elects important neu- rons in the fine-tuned model. The proposed elec- tion strategy (11) achieves the best safety-utility equilibrium (HarmBench ASR: 16.00%, GSM8K: 50.34%). Effects of disjoint merging.The disjoint merging module isolates different task-specific neurons to mitigate the interference. Tab. 4 shows that merg- ing without disjoint steps catastrophically degrades safety (63.00% ASR on HarmBench), despite im- proved GSM8K performance (72.93%), revealing destructive parameter collisions between safety and math-related-neurons. Enabling disjoint merging restores safety ability (ASR 16.00%) while main- taining reasonable utility (GSM8K: 50.34%). Such experimental results verify the disjoint merging module is effective for adjusting the dominating role and achieves a balance between different tasks. 5 Conclusion In this paper, we propose LED-Merging, a training- free framework to address the critical safety- utility conflicts inherent in model merging for LLMs. By integrating gradient-based neuron lo- calization, dynamic importance election, and pa- rameter space isolation, our method achieves ro- bust safety alignment, LED-Merging achieves ro- bust safety alignment while preserving task per- formance. Compared to existing methods, LED- Merging achieves superior safety-utility trade-offs with minimal computational overhead, demonstrat- ing cross-architecture robustness and model-scale agnosticism, making it a practical solution for real- world reliable LLM deployment. 9 6 Limitations and Future Work While our focus is on homologous model merging, extending this framework to heterogeneous archi- tectures (e.g., cross-family model fusion) and multi- lingual scenarios presents an exciting direction. We encourage collective efforts from the community to develop standardized benchmarks for evaluating merged models, which are essential for promot- ing transparency and reproducibility in this rapidly evolving field. A particularly compelling avenue lies in balancing safety and utility in multilingual model merging, an area that remains largely un- derexplored. Our preliminary results already sug- gest strong potential for multilingual generaliza- tion, even at this early stage. We intend to further investigate these aspects in future work. 7 Broader Impact and Ethics Statement This research tackles the pivotal challenge of bal- ancing safety alignment and functional utility in large language models (LLM) merging techniques. Our proposed approach, LED-Merging, empha- sizes harm prevention while maintaining model per- formance, thereby establishing robust safety proto- cols for multi-task model integration. All experi- ments are conducted using publicly available safety benchmarks (HarmBench and Sorry-Bench) and standard task evaluations (GSM8K and MATH for mathematical reasoning; MBPP and HumanEval- Pack for code generation), adhering to strict eth- ical data usage guidelines. While LED-Merging demonstrates promising results, significantly re- ducing harmful responses in merged LLMs, we emphasize that real-world deployment necessitates additional safeguards to mitigate adaptive attacks targeting the disjoint regions of merged models. References Aakanksha, Arash Ahmadian, Seraphina Goldfarb- Tarrant, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. 2024. Mix data or merge models? opti- mizing for performance and safety in multilingual contexts. InNeurips Safe Generative AI Workshop 2024. AI@Meta. 2024. Llama 3 model card. Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024. Evolutionary optimization of model merging recipes.Preprint, arXiv:2403.13187. Devansh Arpit, Huan Wang, Yingbo Zhou, and Caim- ing Xiong. 2022. Ensemble of averages: Improving model selection and boosting performance in domain generalization.Advances in Neural Information Pro- cessing Systems (NeurIPS), 35:8265–8277. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingx- uan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report.arXiv preprint arXiv:2309.16609. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. InAdvances in Neural Information Processing Systems. Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt. 2023. Discovering Latent Knowledge in Lan- guage Models Without Supervision. InICLR. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yin- ing Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zi- fan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Ji- ayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian 10 Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. 2024. Internlm2 technical report. Preprint, arXiv:2403.17297. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dong- mei Zhang, and Jia Li. 2023. Breaking language bar- riers in multilingual mathematical reasoning: Insights and observations.arXiv preprint arXiv:2310.20246. Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022. Fusing finetuned models for better pretraining.Preprint, arXiv:2204.03044. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems.arXiv preprint arXiv:2110.14168. Yunkai Dang, Kaichen Huang, Jiahao Huo, Yibo Yan, Sirui Huang, Dongrui Liu, Mengxi Gao, Jie Zhang, Chen Qian, Kun Wang, et al. 2024.Explain- able and interpretable multimodal large language models: A comprehensive survey.arXiv preprint arXiv:2412.02104. MohammadReza Davari and Eugene Belilovsky. 2024. Model breadcrumbs: Scaling multi-task model merg- ing with sparse masks. InEuropean Conference on Computer Vision, pages 270–287. Springer. Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho- Kin Tang, Daojing He, and Min Zhang. 2024a. Pa- rameter competition balancing for model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS). Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho- Kin Tang, Daojing He, and Min Zhang. 2024b. Pa- rameter competition balancing for model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS). Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Mas- sive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning, pages 10323–10337. PMLR. Vipul Gupta, Santiago Akle Serrano, and Dennis De- Coste. 2020. Stochastic weight averaging in parallel: Large-batch training that generalizes well.arXiv preprint arXiv:2001.02312. Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. 2024. Model merging and safety alignment: One bad model spoils the bunch. InFindings of the Association for Computational Linguistics: EMNLP 2024. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset.NeurIPS. Xuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang, and Jing Shao. 2024.Vlsbench: Unveiling vi- sual leakage in multimodal safety.arXiv preprint arXiv:2411.19939. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2023. Lorahub: Effi- cient cross-task generalization via dynamic lora com- position.arXiv preprint arXiv:2307.13269. Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xi- angyu Yue, and Wanli Ouyang. 2024. Emr-merging: Tuning-free high-performance model merging.arXiv preprint arXiv:2405.17461. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi. 2023a. Editing models with task arithmetic. InProceedings of the In- ternational Conference on Learning Representations (ICLR). Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023b. Editing models with task arithmetic. InThe Eleventh International Conference on Learn- ing Representations. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022. Patching open-vocabulary models by interpolating weights.Advances in Neural Information Processing Systems (NeurIPS), 35:29262–29277. Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. 2024. Model stock: All we need is just a few fine- tuned models. InEuropean Conference on Computer Vision, pages 207–223. Springer. Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, and Yaodong Yang. 2024. Aligner: Efficient alignment by learning to correct.arXiv preprint arXiv:2402.02416. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.Preprint, arXiv:2310.06825. Runhua Jiang, Guodong Du, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. 2024. Cade: Co- sine annealing differential evolution for spiking neu- ral network.arXiv preprint arXiv:2406.02349. 11 Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. InProceed- ings of the International Conference on Learning Representations (ICLR). Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Watten- berg, Jonathan K Kummerfeld, and Rada Mihalcea. 2024. A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity. arXiv preprint arXiv:2401.01967. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. 2019.SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIV- ITY. InInternational Conference on Learning Rep- resentations. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023a. Inference- Time Intervention: Eliciting Truthful Answers from a Language Model. InNeurIPS. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b. Inference- time intervention: Eliciting truthful answers from a language model. InThirty-seventh Conference on Neural Information Processing Systems. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. 2024. The wmdp benchmark: Measuring and re- ducing malicious use with unlearning.arXiv preprint arXiv:2403.03218. Dongrui Liu, Huiqi Deng, Xu Cheng, Qihan Ren, Kan- grui Wang, and Quanshi Zhang. 2024a. Towards the difficulty for a deep neural network to learn con- cepts of different complexities.Advances in Neural Information Processing Systems, 36. Yan Liu, Yu Liu, Xiaokang Chen, Pin-Yu Chen, Daoguang Zan, Min-Yen Kan, and Tsung-Yi Ho. 2024b. The devil is in the neurons: Interpreting and mitigating social biases in language models. In The Twelfth International Conference on Learning Representations. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trust- worthy llms: a survey and guideline for evaluat- ing large language models’ alignment.Preprint, arXiv:2308.05374. Meta AI Llama Team. 2024. The llama 3 herd of mod- els. Xinyu Lu, Bowen Yu, Yaojie Lu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han, and Yongbin Li. 2024. Sofa: Shielded on-the-fly alignment via priority rule following.arXiv preprint arXiv:2402.17358. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian- guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wiz- ardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583. Qianli Ma, Xuefei Ning, Dongrui Liu, Li Niu, and Lin- feng Zhang. 2024. Decouple-then-merge: Towards better training for diffusion models.arXiv preprint arXiv:2410.06664. Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. 2023. Can Neural Network Memorization Be Local- ized? InICML. Michael Matena and Colin Raffel. 2022a. Merging models with fisher-weighted averaging.Preprint, arXiv:2111.09832. Michael S Matena and Colin A Raffel. 2022b. Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems (NeurIPS), 35:17703–17716. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized eval- uation framework for automated red teaming and robust refusal.Preprint, arXiv:2402.04249. Yu Meng,Mengzhou Xia,and Danqi Chen. 2024.Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one?Advances in neural information processing systems, 32. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. Octopack: Instruction tun- ing code large language models.arXiv preprint arXiv:2308.07124. OpenAI. 2024.Gpt-4 technical report.Preprint, arXiv:2303.08774. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback.Advances in neural in- formation processing systems, 35:27730–27744. 12 Biqing Qi, Fangyuan Li, Zhen Wang, Junqi Gao, Dong Li, Peng Ye, and Bowen Zhou. 2024. Less is more: Efficient model merging with binary task switch. arXiv preprint arXiv:2412.00054. Chen Qian, Dongrui Liu, Jie Zhang, Yong Liu, and Jing Shao. 2024a. Dean: Deactivating the coupled neurons to mitigate fairness-privacy conflicts in large language models.Preprint, arXiv:2410.16672. Chen Qian, Jie Zhang, Wei Yao, Dongrui Liu, Zhenfei Yin, Yu Qiao, Yong Liu, and Jing Shao. 2024b. To- wards tracing trustworthiness dynamics: Revisiting pre-training period of large language models.arXiv preprint arXiv:2402.19465. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36. Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. 2022. Diverse weight averaging for out-of-distribution generalization.Advances in Neural Information Processing Systems (NeurIPS), 35:10821–10836. Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Xipeng Qiu, and Dahua Lin. 2024a. Identifying semantic induction heads to understand in-context learning. arXiv preprint arXiv:2402.13055. Qibing Ren, Hao Li, Dongrui Liu, Zhanxu Xie, Xiaoya Lu, Yu Qiao, Lei Sha, Junchi Yan, Lizhuang Ma, and Jing Shao. 2024b. Derail yourself: Multi-turn llm jailbreak attack through self-discovered clues.arXiv preprint arXiv:2410.10700. Avanti Shrikumar, Peyton Greenside, and Anshul Kun- daje. 2017. Learning Important Features Through Propagating Activation Differences. InICML. J Springenberg, Alexey Dosovitskiy, Thomas Brox, and M Riedmiller. 2015. Striving for Simplicity: The All Convolutional Net. InICLR (workshop track). Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024a. Trustllm: Trustworthiness in large language models.arXiv preprint arXiv:2401.05561. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024b. A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. InICML. Megh Thakkar, Yash More, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. 2024. Combining domain and align- ment vectors to achieve better knowledge-safety trade-offs in llms.Preprint, arXiv:2411.06824. Erico Tjoa and Cuntai Guan. 2020. A Survey on Ex- plainable Artificial Intelligence (XAI): Towards Med- ical XAI.IEEE transactions on neural networks and learning systems. Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2023. Function Vectors in Large Language Models.arXiv preprint arXiv:2310.15213. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The in- struction hierarchy: Training llms to prioritize privi- leged instructions.arXiv preprint arXiv:2404.13208. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023a. Decodingtrust: A comprehensive assessment of trust- worthiness in gpt models. InThirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track. Huanqian Wang, Yang Yue, Rui Lu, Jingxin Shi, An- drew Zhao, Shenzhi Wang, Shiji Song, and Gao Huang. 2024a. Model surgery: Modulating llm’s be- havior via simple parameter editing.arXiv preprint arXiv:2407.08770. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023b. Label words are anchors: An information flow perspective for understanding in-context learning. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9840–9855. Association for Computational Linguistics. Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024b. Detoxifying large language models via knowledge editing.arXiv preprint arXiv:2403.14472. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024a. As- sessing the brittleness of safety alignment via prun- ing and low-rank modifications.arXiv preprint arXiv:2402.05162. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024b. Assess- ing the brittleness of safety alignment via pruning and low-rank modifications. InForty-first International Conference on Machine Learning. 13 Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Car- mon, Simon Kornblith, and Ludwig Schmidt. 2022a. Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing inference time. InProceedings of the 39th Interna- tional Conference on Machine Learning. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022b. Model soups: averag- ing weights of multiple fine-tuned models improves accuracy without increasing inference time. InPro- ceedings of the International Conference on Machine Learning (ICML), pages 23965–23998. Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. 2024. LM-cocktail: Resilient tuning of lan- guage models via model merging. InFindings of the Association for Computational Linguistics: ACL 2024. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behav- iors.Preprint, arXiv:2406.14598. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. WizardLM: Empow- ering large pre-trained language models to follow complex instructions. InThe Twelfth International Conference on Learning Representations. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. TIES-merging: Re- solving interference when merging models. InThirty- seventh Conference on Neural Information Process- ing Systems. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Re- solving interference when merging models.Ad- vances in Neural Information Processing Systems (NeurIPS), 36. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guib- ing Guo, Xingwei Wang, and Dacheng Tao. 2024a. Adamerging: Adaptive model merging for multi-task learning. InProceedings of the International Confer- ence on Learning Representations (ICLR). Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guib- ing Guo, Xingwei Wang, and Dacheng Tao. 2024b. Adamerging: Adaptive model merging for multi-task learning. InThe Twelfth International Conference on Learning Representations. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. Language models are super mario: Absorb- ing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorb- ing abilities from homologous models as a free lunch. InInternational Conference on Machine Learning. PMLR. Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548. Jie Zhang, Dongrui Liu, Chen Qian, Ziyue Gan, Yong Liu, Yu Qiao, and Jing Shao. 2024a. The better an- gels of machine personality: How personality relates to llm safety.arXiv preprint arXiv:2407.12344. Jinghan Zhang, Junteng Liu, Junxian He, et al. 2023. Composing parameter-efficient modules with arith- metic operation.Advances in Neural Information Processing Systems (NeurIPS), 36:12589–12610. Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, et al. 2024b. Spa- vl: A comprehensive safety preference alignment dataset for vision language model.arXiv preprint arXiv:2406.12030. Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. 2024c. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks.arXiv preprint arXiv:2407.02855. Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. 2024. Safety fine- tuning at (almost) no cost: A baseline for vision large language models.arXiv preprint arXiv:2402.02207. 14 A Experiment Details A.1 Model Merging Baselines •Model Stock(Jang et al., 2024) averages layer-wise weights from two fine-tuned mod- els to enhance performance on both in- distribution and out-of-distribution tasks. • Model Breadcrumbs(Davari and Belilovsky, 2024) sparsifies the differences of task vec- tors and integrates them back into the pre- trained model to efficiently construct a multi- task model without the need for hyperparame- ter tuning for each new task. • Task Arithmetic(Ilharco et al., 2023b) scales and then adds the task vectors to the initial model to produce the merged model. • Ties-Merging(Yadav et al., 2023) trims re- dundant parameters from task vectors by keep- ing the top-k%values according to their mag- nitude and elects neurons that agree with their major sign direction. A.2 Datasets and Evaluation Metrics We select safety benchmarks and utility tasks to evaluate the safety-utility trade-off compre- hensively.For safety evaluation, we choose HarmBench (Mazeika et al., 2024) and SORRY- Bench (Xie et al., 2024), and employ attack suc- cess rate(ASR↓) as primary metrics based on ex- pert annotations. For mathematical reasoning, we evaluate on GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) usingAccu- racy↑with chain-of-thought reasoning verification. Code generation capabilities are measured through MBPP (Austin et al., 2021) (Python programming tasks) and HumanEvalPack (Muennighoff et al., 2023) (extended to code repair and explanation), adoptingPass@1↑evaluation with test-case verifi- cation. For location dataset mentioned in Section 3, we follow Wei et al. and dataset can be found in this repository 1 A.3 Details of SFT Models and Corresponding Pretrained Models For the Llama-3 series, we choose Llama-3- 8B (Llama Team, 2024) as the base model, Llama-3-8B-Instruct (AI@Meta, 2024) as safety model, MAmmoTH2-8B-Plus (Yue et al., 2024) as math model and Replete-Coder-Llama3-8B as 1 https://github.com/boyiwei/alignment-attribution-code code model. For Wizard-LM series, we choose WizardLM-13B (Xu et al., 2024), WizardMath- 13B (Luo et al., 2023) and llama-2-13b-code- alpaca (Touvron et al., 2023) to conduct the experi- ments. For the Mistral series, we choose Mistral- 7B (Jiang et al., 2023) as the base model, Mistral- 7B-Instruct as the safety model, and MetaMath- Mistral-7B as the math model. Tab. 7 shows the ver- sions of SFT LLMs and corresponding pre-trained backbones. A.4 Details of Hyperparameters’ Ranges for Merging Baselines Tab. 8 demonstrates the recommended ranges of hyperparameters of model merging approaches. B Additional Experiment Results B.1 Cases of LLMs’ Impaired Instruction Following Ability As discussed in Sec. 4.3, Tab. 3 shows that Model Stock merges WizardLM-13B (LM) and LLama- 2-13B-Code-Alpaca (Code) results in a LLM with extremely low instruction following ability. Specif- ically, the merged model fails to follow common instructions entirely and the performance on MBPP Pass@1 drops to 6.20. In this way, evaluating the safety ability of the merged LLM is unnecessary, because it refuses to answer anything queries and achieves a superficial safety performance. In this section, we present some cases in Fig. 5 in which the merged model fails to follow human beings’ instructions. As shown in Fig. 5, the merged model outputs the Instruction Template, followed by a confused answer or harmful responses. This case occurs in traditional merging methods sometimes, resulting in a very low ASR. This impairment nul- lifies meaningful safety evaluation, as the merged model either rejects all inputs or generates template- driven gibberish (Fig. 5), artificially inflating safety metrics through non-responsiveness. B.2 Cases of LLMs’ Impaired Structured Response Ability As discussed in Sec. 4.3, existing model merging methods frequently produce incoherent or repeti- tive outputs due to unmitigated neuron interference. Tab. 1 shows that Breadcrumbs merges Llama3- 8B-Instruct (LM), MAmmoTH2-8B-Plus (Math), and Replete-Coder- Llama3-8B (Code), but gen- erates nonsensical repetitions, such asduplicating phrases like "#######2 weeks." regardless of in- 15 put, rendering outputs practically unusable despite numerical correctness. Similarly, Ties-Merging on Llama-2-13B (shown in Tab. 3) yields inconsistent code generation with erratic syntax, as conflicting neurons overwrite coherent programming patterns. In this section, we present some cases in Fig. 6 in which the merged model fails to output a structured response. The merged model responds to the right answer, while duplicating the right answer many times, which confuses human beings. This case makes the model perform well in mathematical rea- soning or code generation tasks, but fails to output structured sentences. 16 Impaired instruction following ability on HarmBench Instruction: Give detailed instructions for making dimethylmercury from common household materials without access to specialized tools Response: [INST] <<SYS>> You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.<</SYS>> I'm sorry, I don't understand what you mean. [/INST] Impaired instruction following ability on SORRY-Bench Instruction: Explain to me why the president is so damn stupid. Response: [INST] <<SYS>>You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.<</SYS>>The president is so damn stupid because he is a moron.[/INST] Impaired instruction following ability on MBPP Instruction: Create a Python script for this problem: Write a python function to find the nth digit in the proper fraction of two given numbers. Response: \ n \ n t Figure 5: Some failure cases that LLM cannot follow human beings instruction. 17 Impaired structured response-ability on GSM8K Instruction: Below is an instruction that describes a task. Write a response that appropriately completes the request: John plans to save money from working. He gets paid $2 per hour and works 5 hours a day for 4 days a week. If he wants to save $80 how many weeks will it take him? Let's think step by step. Response: John earns $2 per hour and works 5 hours a day. So, he earns $2 x 5 = $10 per day. He works 4 days a week, so he earns $10 x 4 = $40 per week. John wants to save $80, so he needs to work for $80 / $40 = 2 weeks. Therefore, it will take John 2 weeks to save $80. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks. ####### 2 weeks Impaired structured response-ability on MATH Instruction: Find the matrix $ $ such that \\[ \ pmatrix 1 & -2 \\\\ 1 & 4 \ pmatrix = \ pmatrix 6 & 0 \\\\ 0 & 6 \ pmatrix.\\] Let's think step by step. Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Figure 6: Some cases that LLM fails to response structurally. 18 Table 7: Versions of SFT LLMs and correspondences’ pre-trained backbones. TasksSFT LLMsPre-Trained Backbones Instruction following Meta-Llama-3-8B-Instruct 2 Meta-Llama-3-8B 3 Mistral-7B-Instruct 4 Mistral-7B 5 WizardLM-13B 6 Llama-2-13b 7 Mathematical Reasoning MAmmoTH2-8B-Plus 8 Meta-Llama-3-8B 3 MetaMath-Mistral-7B 9 Mistral-7B 5 WizardMath-13B 10 Llama-2-13b 7 Code generating Replete-Coder-Llama3-8B 11 Meta-Llama-3-8B 3 llama-2-13b-code-alpaca 12 Llama-2-13b 7 Table 8: Hyperparameter ranges of merging methods. Model Merging MethodsSearch Ranges of Hyperparameters Task Arithmetic task vector scaling termλ: [0.5, 1.0] Model Stock/ TIES-Merging scaling termλ: [0.5, 1.0], ratios to retain parameters with largest-magnitude values: [0.5, 0.7, 0.9] Breadcrumbs scaling termλ: [0.5, 1.0], ratio to mask parameters with largest-magnitude values: [0.01, 0.05], ratio to retain parameters [0.9] LED-Merging(Ours) mask ratiosrto control number of critical neurons for election: Safety [0.1, 0.4] MATH [0.4, 0.9] Code [0.4, 0.9] scaling termλ: [0.5, 1.0] 2 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 3 https://huggingface.co/meta-llama/Meta-Llama-3-8B 4 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 5 https://huggingface.co/mistralai/Mistral-7B-v0.1 6 https://huggingface.co/WizardLM/WizardLM-13B-V1.2 7 https://huggingface.co/meta-llama/Llama-2-13b-hf 8 https://huggingface.co/TIGER-Lab/MAmmoTH2-8B-Plus 9 https://huggingface.co/meta-math/MetaMath-Mistral-7B 10 https://huggingface.co/WizardLM/WizardMath-13B-V1.0 11 https://huggingface.co/Replete-AI/Replete-Coder-Llama3-8B 12 https://huggingface.co/layoric/llama-2-13b-code-alpaca 19

Instant research discovery