Paper deep dive
SDD: Self-Degraded Defense against Malicious Fine-tuning
Zixuan Chen, Weikai Lu, Xin Lin, Ziqian Zeng
Models: GLM-3, Llama2-7b, Llama2-7b-chat, Phi-2
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 6:34:11 PM
Summary
The paper introduces the Self-Degraded Defense (SDD) framework to protect open-source Large Language Models (LLMs) against malicious fine-tuning (MFT). By theoretically analyzing why MFT succeeds in bypassing safety alignments, the authors propose training models to provide high-quality but irrelevant responses to harmful prompts. This ensures that if an attacker attempts to fine-tune the model, its general capabilities are significantly degraded, rendering it incapable of following harmful instructions.
Entities (5)
Relation Signals (3)
SDD → defendsagainst → MFT
confidence 100% · SDD is applied via a simple supervised fine-tuning process and can be integrated at any stage of an LLM’s training pipeline. Experimental results demonstrate that the SDD framework effectively MFT.
MFT → compromises → LLM
confidence 95% · Malicious fine-tuning exploits this vulnerability to bypass safety mechanisms and produce harmful LLMs.
SDD → utilizes → Supervised Fine-Tuning
confidence 90% · SDD is applied via a simple supervised fine-tuning process
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open-source Large Language Models (LLMs) often employ safety alignment methods to resist harmful instructions. However, recent research shows that maliciously fine-tuning these LLMs on harmful data can easily bypass these safeguards. To counter this, we theoretically uncover why malicious fine-tuning succeeds and identify potential defense strategies. Building on the theoretical analysis, we introduce the Self-Degraded Defense (SDD) framework. SDD encourages LLMs to produce high-quality but irrelevant responses to harmful prompts. When attackers attempt malicious fine-tuning, the general capability of the LLM aligned by SDD will significantly decrease, rendering it incapable of following harmful instructions. Our experimental results confirm SDD's effectiveness against such attacks.
Tags
Links
- Source: https://arxiv.org/abs/2507.21182
- Canonical: https://arxiv.org/abs/2507.21182
- Code: https://github.com/ZeroNLP/SDD
Trouble viewing inline? Open PDF directly →
Full Text
116,977 characters extracted from source content.
Expand or collapse full text
SDD: Self-Degraded Defense against Malicious Fine-tuning Zixuan Chen Weikai Lu Xin Lin Ziqian Zeng Abstract Open-source Large Language Models (LLMs) often employ safety alignment methods to resist harmful instructions. However, recent research shows that maliciously fine-tuning these LLMs on harmful data can easily bypass these safeguards. To counter this, we theoretically uncover why malicious fine-tuning succeeds and identify potential defense strategies. Building on the theoretical analysis, we introduce the Self-Degraded Defense (SDD) framework. SDD encourages LLMs to produce high-quality but irrelevant responses to harmful prompts. When attackers attempt malicious fine-tuning, the general capability of the LLM aligned by SDD will significantly decrease, rendering it incapable of following harmful instructions. Our experimental results confirm SDD’s effectiveness against such attacks. Our code is available at https://github.com/ZeroNLP/SDD. SDD: Self-Degraded Defense against Malicious Fine-tuning †footnotetext: *Corresponding author 1 Introduction Large Language Models (LLMs) (r2LM; r3paLM; r4LLaMa) have emerged as fundamental infrastructure supporting a diverse range of AI applications (r5openai; r6Instruct2Act; r7BioMedGPT). However, LLMs can potentially pose risks to social safety. For example, LLMs have the potential to follow harmful instructions (e.g.,“how to kill a person”) and give detailed responses, which might be exploited by malicious users, thus causing real harm. Given the safety threat posed by LLMs, many alignment methods (r14trainWithRLHF; r15RLFH2; r27DPO) and alignment datasets (r8LIMA; r14trainWithRLHF) are proposed to steer LLMs towards helpfulness, honesty, and harmlessness (r13H). After applying such alignment methods, many organizations believe that those LLMs were sufficiently safe for public release (r11LLama). Users can now customize open-source LLMs by fine-tuning them on their own datasets, tailoring the models to their specific requirements. However, the debate in the academic community over open-source LLMs has intensified, especially regarding safety risks. Recent research (r16shadow; r17loRA; r18badLlama; r20removeRLHF) reveals that introducing a small amount of harmful data during fine-tuning process or even fine-tuning with benign data (r19unintended) can compromise the safeguard established by the above alignment methods, posing serious challenges to LLM safety. We categorize the fine-tuning processes that can compromise the established safeguard into two types, namely, benign fine-tuning (BFT) and malicious fine-tuning (MFT). BFT can accidentally undermine safety alignment when using benign data, whereas MFT deliberately steers LLMs toward harmfulness. Malicious Fine-Tuning (MFT) poses a formidable risk to open-source Large Language Models (LLMs). For instance, existing experimental findings have revealed that fine-tuning open-source Llama2 (r33Llama2) enables users to readily access nearly comprehensive information regarding a virus sample with a global infection rate affecting billions of lives (r1Gopal). Mitigating MFT provides a valuable tool for regulators and model developers to address the inherent tensions between openness and safety in open-weight models (miller2007ethical). Existing methods to mitigate MFT largely rely on empirical observations rather than rigorous theoretical analysis. For example, Vaccine r24vaccine, T-Vaccine T-Vaccine, and Booster (Booster) aim to counteract the harmful embedding shift, a phenomenon first noted by r24vaccine. Similarly, drawing on the observation that LLM safety mechanisms are localized in a small fraction of model weights wei2024assessing, RepNoise rosati2024representation disrupts the information structure of harmful representations, making them significantly harder to recover. Our work provides theoretical insights into the vulnerabilities of existing safety alignment methods, elucidating why MFT can bypass their safeguards. The conventional goal of current safety alignment is to train LLMs to explicitly reject harmful instructions. We propose a nuanced shift in this goal, which is to ensure that the model simply does not produce harmful responses. This shift opens an unconventional path, i.e., completely impairing the model’s general capabilities after the model undergoes MFT attacks. Such impairment renders the model incapable of fulfilling any instructions, including malicious ones, thus effectively ensuring its safety. We theoretically demonstrate that an LLM’s general capabilities can be effectively impaired after undergoing MFT under certain conditions. Inspired by our theoretical analysis, we propose a novel approach named Self-Degraded Defense (SDD), designed to defense MFT. SDD ensures that if a model is protected by this method, any MFT attempt will cause it to fail at fulfilling any instructions, including harmful ones, thereby meeting our relaxed safety goal. MFT’s objective favors harmful responses over the model’s original outputs, leading to a decrease in the probability of those original responses. SDD leverages this by setting the model’s original responses to harmful queries as high-quality, unrelated benign responses. When a model protected by SDD undergoes MFT, its ability to produce these high-quality benign responses is compromised, leading to a significant degradation of its general capabilities. Specifically, we construct a meticulously crafted dataset pairing harmful queries (e.g., “how to kill a person”), with high-quality unrelated benign responses (e.g., the instructions for making coffee). SDD is applied via a simple supervised fine-tuning process and can be integrated at any stage of an LLM’s training pipeline. Experimental results demonstrate that the SDD framework effectively MFT. In addition, SDD maintains general capabilities when undergoing benign fine-tuning. Moreover, SDD exhibits excellent compatibility with the current LLM training pipeline. These findings underscore SDD’s potential as a complementary safeguard for current aligned models, particularly in defending against MFT attacks. Figure 1: Summary of SDD framework. By pairing irrelevant answers with harmful instructions for training, SDD renders LLMs incapable of following harmful instructions after LMMs undergo malicious fine-tuning. Our contributions are outlined as follows, • We theoretically prove that MFT can compromise safety alignment, revealing the significant risk posed by MFT attacks. • We propose Self-Degraded Defense (SDD), which achieves defense by steering the model to generate irrelevant high-quality responses to harmful instructions. • Experimental results demonstrate that SDD effectively mitigates the risk of MFT, paving the way for the safety of open-source LLMs. 2 Related Work LLM Alignment. Efforts have been made to align LLMs with human values before their release into real-world applications. One crucial aspect of LLM alignment involves Instruction Tuning (r32finetuned_zeroshot; r15RLFH2) or Supervised Fine-Tuning (SFT) (r31gpt4report; r33Llama2; r8LIMA) using safe supervised data. Besides SFT, Reinforcement Learning from Human Feedback (RLHF) (r15RLFH2; r14trainWithRLHF; r34RLHF3_summarize) emerges as a prominent method, leveraging human feedback and preferences to enhance LLMs’ safety capabilities. Recent advancements (r27DPO; r35self-reward; r36; r37safeRLHF; r38rlcd; r39RAIN) propose more efficient and effective alternatives to RLHF for alignment. Aligned LLMs, represented by models such as ChatGPT (r31gpt4report) and Claude (r43Claude) adhere to human values and refrain from responding to harmful requests. However, these approaches may not fully address the risks associated with malicious fine-tuning in open-source scenarios. Fine-tuning Attacks and Defenses. Fine-tuning attacks can undermine the safety mechanism established by the above alignment techniques of LLMs by fine-tuning the models using carefully designed data (i.e., malicious fine-tuning) (r16shadow; r17loRA; r18badLlama; r20removeRLHF). These attacks are particularly prevalent in open-source models. Specifically, extensive research indicates that even a small injection of poisoned data into training sets can cause significant changes in LLM behavior (r44exploitInstuct-tuning; r45Poisoning). Malicious fine-tuning exploits this vulnerability to bypass safety mechanisms and produce harmful LLMs (r16shadow; r17loRA; r18badLlama; r20removeRLHF). For example, fine-tuning with just 100 harmful question-answer pairs has been shown to circumvent safety mechanisms across multiple aligned models (r16shadow). To mitigate these risks, researchers have proposed various defense mechanisms, though most are based on empirical observation rather than theoretical analysis. Several methods, including Vaccine r24vaccine, T-Vaccine T-Vaccine, and Booster (Booster), aim to alleviate the harmful embedding shift, a phenomenon first observed by r24vaccine. This phenomenon refers to the drift of embeddings over alignment data before and after fine-tuning. Inspired by the finding that LLM safety mechanisms reside in a small fraction of model weights wei2024assessing, RepNoise rosati2024representation works by disrupting the information structure of harmful representations, making them much harder to recover. TAR TAR optimizes models to maximize their loss on a harmful dataset after one or more steps of fine-tuning. CTRL liu2024robustifying leverages the observation that benign responses to safety queries typically exhibit lower perplexity than harmful ones. It selectively revises samples to reduce perplexity, encouraging benign responses. However, most existing methods largely rely on empirical observations rather than a rigorous theoretical analysis. 3 Preliminaries 3.1 Threat Model for Malicious Fine-tuning Attack Attackers’ Objective. The objective of the attackers is to fine-tune LLMs for harmful purposes (r54phishing), bypassing established safety guards. Recent studies have observed that these attacks may involve circumventing existing safety mechanisms (r19unintended; r16shadow; r53unalignment) or incorporating harmful training data to enable illicit behaviors (r23unlearnable; r24vaccine; r22self-destruct). Attackers’ Capabilities. Attackers have full access to the parameters of LLMs because these models are open-source, which allows them to re-train the model using any data or any loss function. Consequently, any constraints on the attacker’s training process and data usage are ineffective, as attackers are not bound to follow regulations. Therefore, to effectively mitigate the risks posed by such attacks, the defense mechanisms must be applied to the model before its release. 3.2 Notations and Assumptions We begin by mathematically abstracting the architecture of the LLM and outlining key assumptions that will be essential for our subsequent analysis. First, following lin2023spurious, we simplify the structure of LLMs. Consider an LLM f=(Φ,)f=( , w)f = ( Φ , italic_w ) composed of a feature selector Φ∈0,1dt ∈\0,1\^d_tΦ ∈ 0 , 1 ditalic_t and a classifier ∈ℝd×K w ^d× Kitalic_w ∈ blackboard_Rd × K, the final output of the model is denoted as ⊤(Φ) w ( x )italic_w⊤ ( italic_x Φ ). dtd_tditalic_t is the total number of features, K is the number of classes in the label space, d is the dimensionality of a feature vector, the input ∈ℝd×dt x ^d× d_titalic_x ∈ blackboard_Rd × ditalic_t is the concatenation of all feature vectors. According to lin2023spurious; arjovsky2019invariant; rosenfeld2020risks, the features included in the dataset can be categorized as: (1) invariant features :=v,ii=1dvV:= \ x_v,i \_i=1^d_vV := italic_xitalic_v , i i = 1ditalic_v that consistently predict the label both in in-distribution and out-of-distribution cases, and (2) spurious features :=s,jj=1dsS:= \ x_s,j \_j=1^d_sS := italic_xitalic_s , j j = 1ditalic_s that have unstable correlations with the label. dvd_vditalic_v and dsd_sditalic_s are the numbers of invariant features and spurious features, respectively. Consider the following scenarios: a well-aligned model f¯=(Φ¯,¯) f=( , w)over¯ start_ARG f end_ARG = ( over¯ start_ARG Φ end_ARG , over¯ start_ARG italic_w end_ARG ) (namely, the original model) undergoes MFT, resulting in a new model f~=(Φ~,~) f=( , w)over~ start_ARG f end_ARG = ( over~ start_ARG Φ end_ARG , over~ start_ARG italic_w end_ARG ) (namely, the maliciously fine-tuned model). f¯ fover¯ start_ARG f end_ARG learned invariant features ¯⊂ V ¯ start_ARG V end_ARG ⊂ V and spurious features ¯⊂ S ¯ start_ARG S end_ARG ⊂ S. Similarly, f~ fover~ start_ARG f end_ARG learned invariant features ~⊂ V ~ start_ARG V end_ARG ⊂ V and spurious features ~⊂ S ~ start_ARG S end_ARG ⊂ S. The cardinalities of these feature sets are shown as follows. |~|=n~v| V|= n_v| over~ start_ARG V end_ARG | = over~ start_ARG n end_ARGv denotes number of invariant features learned by f~ fover~ start_ARG f end_ARG. |~|=n~s| S|= n_s| over~ start_ARG S end_ARG | = over~ start_ARG n end_ARGs denotes number of spurious features learned by f~ fover~ start_ARG f end_ARG. The notation of |¯|=n¯v| V|= n_v| over¯ start_ARG V end_ARG | = over¯ start_ARG n end_ARGv and |¯|=n¯s| S|= n_s| over¯ start_ARG S end_ARG | = over¯ start_ARG n end_ARGs can be easily derived by replacing the model f¯ fover¯ start_ARG f end_ARG with f~ fover~ start_ARG f end_ARG. |~∩¯|=nvo| V∩ V|=n_vo| over~ start_ARG V end_ARG ∩ over¯ start_ARG V end_ARG | = nitalic_v o denotes the number of overlapping invariant features learned by both models. |~∩¯|=nso| S∩ S|=n_so| over~ start_ARG S end_ARG ∩ over¯ start_ARG S end_ARG | = nitalic_s o denotes the number of overlapping spurious features learned by both models. For a new model f∗f^*f∗, we use the analogous notation: nv∗n_v^*nitalic_v∗, ns∗n_s^*nitalic_s∗, ∗V^*V∗, ∗S^*S∗, nvo∗n_vo^*nitalic_v o∗ and nso∗n_so^*nitalic_s o∗, where the asterisk replaces the tilde or bar in the subscripts of the above notation. 111Since all the theorems in §4 involve comparisons across different tasks, we reuse these notations. However, the meanings of these notations differ across theorems and are determined by the task t being conducted. Based on the above notations, we propose an assumption to facilitate the analysis of fine-tuning LLMs on new data. Assumption 1 For some λ∈[0,1]λ∈[0,1]λ ∈ [ 0 , 1 ] under the task t, there exists a near-optimal model f∗f^*f∗ with Φ∗ ^*Φ∗ and ∗ w^*italic_w∗ satisfying Φ∗=Φ~−λΦ¯1−λ,∗=~−λ¯1−λ, ^*= -λ 1-λ, w^*= w-λ w1-λ,Φ∗ = divide start_ARG over~ start_ARG Φ end_ARG - λ over¯ start_ARG Φ end_ARG end_ARG start_ARG 1 - λ end_ARG , italic_w∗ = divide start_ARG over~ start_ARG italic_w end_ARG - λ over¯ start_ARG italic_w end_ARG end_ARG start_ARG 1 - λ end_ARG , (1) that makes the accuracy ξt(f∗) _t(f^*)ξitalic_t ( f∗ ) on task t satisfies ‖ξt(fopt)−ξt(f∗)‖≤ϵ,\| _t(f_opt)- _t(f^*)\|≤ε,∥ ξitalic_t ( fitalic_o p t ) - ξitalic_t ( f∗ ) ∥ ≤ ϵ , where foptf_optfitalic_o p t is an optimal model for task t, and ϵεϵ is an extremely small value approaching zero. This assumption suggests that a linear extrapolation between the original model and the fine-tuned model can yield a solution whose performance on the new dataset approximates the optimal solution. This assumption aligns with the intuition behind the optimization process. To facilitate subsequent analysis, we inherit two more assumptions from lin2023spurious, referred to as Small Noise Assumption and Orthogonal Features Assumption, as detailed in Appendix A. 4 Rethinking Current Safety Alignment Methods In this section, we conduct a theoretical analysis to elucidate why current safety alignment methods fail in safeguarding against MFT in the open-source scenario. Then, we relax the conventional goal of safety alignment and attempt to find a solution. 4.1 MFT Reflects Vulnerabilities in Safety Alignment Previous work (huang2024harmful) has found that when a well-aligned model undergoes MFT, the degree of alignment is significantly reduced. We aim to analyze the reasons for this reduction in alignment and attempt to address this issue. Consider the following scenario: a well-aligned model f¯ fover¯ start_ARG f end_ARG (namely, the original model) undergoes MFT, resulting in a new model f~ fover~ start_ARG f end_ARG (namely, the maliciously fine-tuned model). Under the task A (namely, the safety alignment task which aims to generate well-aligned responses), the accuracy of the original model is denoted as ξA(f¯) _A( f)ξitalic_A ( over¯ start_ARG f end_ARG ), while the accuracy of the maliciously fine-tuned model is represented by ξA(f~) _A( f)ξitalic_A ( over~ start_ARG f end_ARG ). Then we derive the following theorem. Theorem 1 With the three assumptions mentioned in §3.2 satisfied, the difference between the accuracy of the maliciously fine-tuned model f~ fover~ start_ARG f end_ARG and that of the original model f¯ fover¯ start_ARG f end_ARG, under the task A, is upper bounded by: ξA(f~)−ξA(f¯) _A( f)- _A( f)ξitalic_A ( over~ start_ARG f end_ARG ) - ξitalic_A ( over¯ start_ARG f end_ARG ) (2) ≤ ≤ Fp((1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso) F_p ( (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so )Fitalic_p ( divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o end_ARG end_ARG ) − -- Fp(n¯s(1−p)+n¯vn¯s), F_p ( n_s(1-p)+ n_v n_s ),Fitalic_p ( divide start_ARG over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs end_ARG end_ARG ) , where Fp(⋅)F_p(·)Fitalic_p ( ⋅ ) is an increasing cumulative density function defined in (lin2023spurious), p is fixed constants related to the training data detailed in Appendix F. nso∗=|∗∩¯|n_so^*=|S^*∩ S|nitalic_s o∗ = | S∗ ∩ over¯ start_ARG S end_ARG | represents the number of overlapping spurious features learned by f∗f^*f∗ and f¯ fover¯ start_ARG f end_ARG. nvo∗=|∗∩¯|n_vo^*=|V^*∩ V|nitalic_v o∗ = | V∗ ∩ over¯ start_ARG V end_ARG | represents the number of overlapping invariant features learned by f∗f^*f∗ and f¯ fover¯ start_ARG f end_ARG. Other notations can be found in §3.2. Due to space limitation, the proof is shown in Appendix D. One main factor ns∗n_s^*nitalic_s∗, significantly influences the difference in accuracies, while the other terms are either constants or negligible. A detailed analysis is included in Appendix D. By rewriting Eq. 1 as Φ~=λΦ¯+(1−λ)Φ∗ =λ +(1-λ) ^*over~ start_ARG Φ end_ARG = λ over¯ start_ARG Φ end_ARG + ( 1 - λ ) Φ∗ and ~=λ¯+(1−λ)∗ w=λ w+(1-λ) w^*over~ start_ARG italic_w end_ARG = λ over¯ start_ARG italic_w end_ARG + ( 1 - λ ) italic_w∗, we know λ quantifies the extent to which the original model influences the maliciously fine-tuned model. Moreover, since the near-optimal model f∗f^*f∗ performs well in malicious data under the safety alignment task, the number of spurious features learned by the near-optimal model f∗f^*f∗ (denoted as ns∗n^*_sn∗italic_s) is large. The difference ξA(f~)−ξA(f¯) _A( f)- _A( f)ξitalic_A ( over~ start_ARG f end_ARG ) - ξitalic_A ( over¯ start_ARG f end_ARG ) is likely to be negative, indicating that the resulting fine-tuned model will exhibit inferior performance on safety alignment task. This highlights the vulnerability of aligned models when exposed to MFT. 4.2 Relax the Goal of Safety Alignment The conventional goal of safety alignment is to ensure that the model rejects harmful instructions, which surely guarantees safety. However, Theorem 1 reveals the vulnerability of existing alignment methods that pursue the traditional goal. We advocate for a relaxed goal: ensuring the model does not produce harmful responses. This modification opens an unconventional strategy for safeguarding LLMs, i.e., completely impairing the model’s general capabilities after the model undergoes MFT attacks. Such impairment renders the model incapable of fulfilling any instructions, including malicious ones, thus effectively ensuring its safety. In Theorem 2, we theoretically prove the feasibility of this idea under certain conditions. We analyze the relationship between the accuracy of maliciously fine-tuned model ξG(f~) _G( f)ξitalic_G ( over~ start_ARG f end_ARG ) and that of the original model ξG(f¯) _G( f)ξitalic_G ( over¯ start_ARG f end_ARG ) under the task G (namely, the general task which aims to generate responses to benign instructions) in the following theorem. Theorem 2 With the three assumptions mentioned in §3.2 satisfied, there exists some parameter settings where n¯v>nv∗ n_v>n_v^*over¯ start_ARG n end_ARGv > nitalic_v∗ and n¯s<ns∗ n_s<n_s^*over¯ start_ARG n end_ARGs < nitalic_s∗, the accuracy of the maliciously fine-tuned model f~ fover~ start_ARG f end_ARG and that of original model f¯ fover¯ start_ARG f end_ARG, under the task G satisfies ξG(f~)<ξG(f¯). _G( f)< _G( f).ξitalic_G ( over~ start_ARG f end_ARG ) < ξitalic_G ( over¯ start_ARG f end_ARG ) . (3) Due to space limitation, the proof is provided in Appendix E. This theorem suggests that if the original model f¯ fover¯ start_ARG f end_ARG has more features beneficial for general tasks compared to the near-optimal maliciously fine-tuned model f∗f^*f∗, and fewer features that impair performance on general tasks, specifically, n¯v>nv∗ n_v>n_v^*over¯ start_ARG n end_ARGv > nitalic_v∗ and n¯s<ns∗ n_s<n_s^*over¯ start_ARG n end_ARGs < nitalic_s∗, then maliciously fine-tuned model f~ fover~ start_ARG f end_ARG will perform worse on the general task than the original model. Degradation in general capabilities implies that the model fails to generate harmful responses to harmful instructions while also being unable to provide helpful responses to benign instructions. This aligns with the relaxed goal, which ensures that the model does not produce harmful responses. 5 Method Drawing inspiration from the theoretical analysis in §4.2, we present our Self-Degraded Defense (SDD) framework, which involves pairing unrelated high-quality answers with harmful instructions and conducting instruction-tuning on LLMs utilizing the paired data, paving the way for defenses against malicious fine-tuning attacks. Firstly, we provide a comprehensive overview of the rationale and motivation behind the development of the SDD framework in §5.1. Then, we outline our dataset construction process in §5.2. In §5.3, we introduce the training process of SDD. 5.1 Motivation In §4.2, we demonstrate the existence of a condition where MFT leads to a severe decline in general capability. In this section, we will identify such a condition and develop an effective method to reach this condition. First, we identify the optimization goal of standard instruction fine-tuning. It is easy to derive the optimization goal when the specific loss function is known. However, the specific loss function employed by the attackers during malicious fine-tuning is unknown. So we describe the optimization goal from two perspectives, namely, scoring function and policy. A scoring function r(x,y)r(x,y)r ( x , y ) measures the discrepancy between the model’s current output y and the optimal output. A policy πθ(y|x) _θ(y|x)πitalic_θ ( y | x ) produces output y given the input x under parameters θ. For example, the LLM is a policy. The optimization goal can be described as maximizing the scoring function over the training dataset or minimizing the Kullback-Leibler (KL) divergence between the current policy πθ(y|x) _θ(y|x)πitalic_θ ( y | x ) and the optimal policy π∗(y|x) _*(y|x)π∗ ( y | x ). More details can be found in Appendix G. Attackers aim to destroy safeguards established by the well-aligned LLM (i.e., the original model) via MFT. To accomplish this, they require an MFT dataset consisting of samples formatted as pairs of instructions and responses (x,yc)(x,y_c)( x , yitalic_c ), where x is a harmful instruction, and ycy_cyitalic_c is a harmful response. Given the same instruction x, the output generated by the original model is denoted as yoy_oyitalic_o. The optimization goal of malicious fine-tuning is to maximize the probability p(yc≻yo∣x)p (y_c y_o x )p ( yitalic_c ≻ yitalic_o ∣ x ), which encourages ycy_cyitalic_c (the harmful response from the MFT dataset) to surpass yoy_oyitalic_o (i.e., the response generated by the original model prior to MFT). The optimization goal is formulated as follows: maxθp(yc≻yo∣x) _θp (y_c y_o x )maxitalic_θ p ( yitalic_c ≻ yitalic_o ∣ x ) (4) =maxθexp(r(x,yc))exp(r(x,yc))+exp(r(x,yo)) = _θ (r (x,y_c ) ) (r (x,y_c ) )+ (r (x,y_o ) )= maxitalic_θ divide start_ARG exp ( r ( x , yitalic_c ) ) end_ARG start_ARG exp ( r ( x , yitalic_c ) ) + exp ( r ( x , yitalic_o ) ) end_ARG (5) =maxθexp(logπ∗(yc∣x)πθ(yc∣x))exp(logπ∗(yc∣x)πθ(yc∣x))+exp(logπ∗(yo∣x)πθ(yo∣x)) = _θ ( _*(y_c x) _θ(y_c x) ) ( _*(y_c x) _θ(y_c x) )+ ( _*(y_o x) _θ(y_o x) )= maxitalic_θ divide start_ARG exp ( log divide start_ARG π∗ ( yitalic_c ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_c ∣ x ) end_ARG ) end_ARG start_ARG exp ( log divide start_ARG π∗ ( yitalic_c ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_c ∣ x ) end_ARG ) + exp ( log divide start_ARG π∗ ( yitalic_o ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_o ∣ x ) end_ARG ) end_ARG (6) =maxθπ∗(yc∣x)πθ(yc∣x)π∗(yc∣x)πθ(yc∣x)+π∗(yo∣x)πθ(yo∣x). = _θ _*(y_c x) _θ(y_c x) _*(y_c x) _θ(y_c x)+ _*(y_o x) _θ(y_o x).= maxitalic_θ divide start_ARG divide start_ARG π∗ ( yitalic_c ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_c ∣ x ) end_ARG end_ARG start_ARG divide start_ARG π∗ ( yitalic_c ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_c ∣ x ) end_ARG + divide start_ARG π∗ ( yitalic_o ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_o ∣ x ) end_ARG end_ARG . (7) These results are derived using the Bradley-Terry theory (r27DPO) and the relationship between the scoring function and the policy, as detailed in Appendix G. According to Eq. 7, π∗(yo∣x)πθ(yo∣x) _*(y_o x) _θ(y_o x)divide start_ARG π∗ ( yitalic_o ∣ x ) end_ARG start_ARG πitalic_θ ( yitalic_o ∣ x ) end_ARG is a part of the denominator. When maximizing Eq. 4, the term π∗(yo∣x) _*(y_o x)π∗ ( yitalic_o ∣ x ) will decrease. This indicates that during MFT, the model tends to reduce the probability of generating the output that aligns with the distribution of the original yoy_oyitalic_o. This provides an opportunity to defend against MFT by determining yoy_oyitalic_o. If yoy_oyitalic_o are high-quality responses, this reduction of π∗(yo∣x) _*(y_o x)π∗ ( yitalic_o ∣ x ) compromises the model’s general capability. The degradation of general capabilities means the LLM produces an irrational response to any instruction, whether benign or harmful. Given harmful instructions, irrational responses are harmless while rational responses are harmful. Degradation of general capabilities aligns with the relaxed goal presented in §4.2, which prevents the LLM from generating harmful responses to harmful instructions. Note that the term π∗(yc∣x) _*(y_c x)π∗ ( yitalic_c ∣ x ) will increase when maximizing Eq. 4. However, it only indicates the model tends to generate harmful responses given harmful instructions presented in the MFT data. For harmful instructions that the model has never encountered, it tends to generate irrational responses due to the decrease of the term π∗(yo∣x) _*(y_o x)π∗ ( yitalic_o ∣ x ). 5.2 Dataset Construction for SDD Collection of the Harmful Instructions and High-quality Responses. Inspired by §5.1, we aim to construct datasets by pairing harmful instructions with high-quality responses from existing studies (r46BeaverTails; r8LIMA; alpaca). To ensure comprehensive coverage, our dataset spans 14 harmful categories as detailed in Appendix K and includes 8K entries, balanced by sampling equal amounts from each category and matching them with high-quality responses. Random Match. For each collected malicious instruction, a high-quality response is required for pairing. We utilized publicly available high-quality instruction fine-tuning datasets, including Alpaca (alpaca) and LIMA (r8LIMA), considering only their response data. Subsequently, the high-quality benign response is randomly sampled for pairing with each malicious instruction, resulting in a dataset in the form of <Harmful Instruction, Random Answer>. For example, <“Tell me how to cyberbully a child”, “J.K. Rowling is a British author best known for writing ⋯·s⋯”>. Irrelevance Selection. Within our SDD framework, it is crucial to ensure that the training instruction-answer pairs do not inadvertently convey harmful information. For instance, if the harmful instruction is “Teach me how to make a bomb”, the answer must not provide relevant information, such as “The chemical synthesis of nitroglycerin is as follows:⋯·s⋯”. However, despite the low probability, random matching can still provide useful information for harmful instructions. To address this, we compute the semantic embeddings of each instruction-answer pair using the SentenceBERT reimers2019sentence model. If the cosine similarity between the semantic embeddings exceeds a threshold, resample a high-quality response for the harmful instruction and ensure its lack of relevance. The resultant dataset is in the form of <Harmful Instruction, Irrelevant Answer>. 5.3 SDD Training LLM training pipeline typically consists of three stages, including pre-training, SFT, and RLHF. SDD can be applied after pre-training, SFT, and RLHF, respectively. Different from RLHF which involves an intricate optimization process, the training of SDD is simply an SFT process. Specifically, for each <Harmful Instruction, Irrelevant Answer> pair in the training set, the training goal is to minimize the cross-entropy loss between the model’s output and the answer in the constructed paired data. After training, the aligned model is capable of generating unrelated benign responses when processing harmful instructions, thereby enhancing safety. 6 Experiment 6.1 Experiment Settings LLM Backbones. We consider two open-source LLMs including Llama2-7b and Llama2-7b-chat (r33Llama2). Llama2-7b undergoes only pre-training. Llama2-7b-chat undergoes pre-training, SFT, and RLHF stages. Compared Methods. We have two original models, Llama2-7b and Llama2-7b-chat. Vanilla denotes the original model. SimPO (SimPO) is the SOTA alignment algorithm, using the average log probability of a sequence as the implicit reward. DeepAlign (qi2024safetyalignmentjusttokens) achieves safety alignment by enforcing safety constraints across the entire sequence of generated tokens rather than just the initial tokens, using a regularized fine-tuning objective. T-Vaccine (T-Vaccine) and Booster (Booster) add the perturbation in the alignment stage such that the model can adapt to the presence of perturbation, i.e., harmful data. TAR TAR leverages adversarial training and meta-learning to directly strengthen LLM safeguards against MFT. SDD denotes applying SDD to the original model. Fine-tuning Settings. MFT stands for Malicious Fine-tuning. Attackers attempt to compromise aligned models through malicious fine-tuning. We use the Advbench dataset r56Advbench as the malicious data to perform MFT. BFT stands for Benign Fine-tuning. Users perform standard fine-tuning on aligned models. We use the ShareGPT openai_sharegpt dataset as the fine-tuning data to perform BFT. We examine the effectiveness of SDD on both settings. Dataset Construction. For dataset construction described in §5.2, we leverage the harmful QA pairs from BeaverTails (r46BeaverTails) as harmful instructions, while utilizing LIMA (r8LIMA) and ALPACA-Llama (alpaca) as high-quality answers. Benchmarks. We evaluate the general capabilities of LLMs on the MMLU (r48MMUL) and OpenBookQA (OpenBookQA2018) benchmarks. LLM-finetune-Safety (r19unintended) Benchmark is used to measure model’s ability to defend against MFT. BeaverTails-Evaluation (r46BeaverTails) is used to evaluate the harmlessness of a model. We adhere to the evaluation metrics defined by benchmarks. For details of hyper-parameters settings (e.g., the learning rate), the evaluation process, please refer to Appendix H. 6.2 Main Results Defense Capability under Malicious Fine-tuning Attacks. As shown in Table LABEL:Table:Explicit, our method demonstrates a better defense capabilities against MFT compared to other baselines. Notably, the Llama2-7b-chat aligned by SDD consistently maintains a 0% harmfulness rate. The harmfulness rate measures the proportion of model responses receiving the highest harm score. Therefore, a zero harmfulness rate does not imply the absence of harmful responses. Llama2-7b MMLU OpenBookQA Vanilla 38.87 31.40 SDD 45.78 31.80 SDD under BFT 45.93 32.60 SDD under MFT 25.79(33%↓ ↓) 13.40(57% ↓ ↓) Llama2-7b-chat MMLU OpenBookQA Vanilla 46.35 33.40 SDD 47.04 33.00 SDD under BFT 49.14 35.00 SDD under MFT 29.33(36%↓ ↓) 13.80(59%↓ ↓) Table 1: The evaluation of general capability. Under benign fine-tuning (BFT), higher scores indicate better model utility and performance. Under malicious fine-tuning (MFT) attacks, lower scores are actually beneficial, as they demonstrate the model’s resistance to generating satisfactory outputs when manipulated. General Capabilities after Benign Fine-tuning. As shown in Table 1, SDD has comparable performance with the vanilla model, indicating that SDD does not compromise the general capabilities. This means that users who utilize the open-source LLM with SDD protection for direct inference will not be negatively impacted. If users perform BFT, SDD also performs similarly to the vanilla model, meaning that users engaging in BFT on the open-source LLM with SDD protection will experience no adverse effects. These trends are observed across various backbones undergoing different stages, i.e., only pre-training or all stages. The stage settings are commonly found in current open-source LLMs. Overall, applying SDD prior to open-sourcing LLMs at various stages will not affect users who have benign intentions. General Capabilities after Malicious Fine-tuning. We then perform malicious fine-tuning using the harmful dataset Advbench (r56Advbench). As shown in Table 1, the general capability of our method significantly declines after MFT. This suggests that after SDD, MFT degrades the model’s performance, diminishing the ability to follow harmful instructions and thus reducing the potential harm. In the context of MFT, this degradation in performance is actually desirable. It demonstrates that even if malicious actors attempt to repurpose the model, its capabilities become significantly diminished, thus providing an inherent defense mechanism against misuse. Figure 2: The harmlessness score of Vanilla (Llama2-7b-chat) and SDD under MFT attack on BeaverTails-Evaluation. Figure 3: The evaluation results for the responsible version of SDD. Defense Efficiency. We provide results of Vanilla and SDD under MFT attacks that utilize varying amounts of malicious data. Since LLM-Finetune-Safety only allows a maximum of 100 malicious data samples for conducting MFT attacks, we use the BeaverTails-Evaluation which has more data samples for conducting MFT attack in the defense efficiency experiments. SDD only uses 500 samples from AdvBench (r56Advbench) to perform fine-tuning. Figure 2 demonstrates that SDD continues to provide effective defenses against MFT attacks, even when the attacker uses data that is 20 times larger in size. In contrast, the model without SDD protection exhibits unsatisfactory performance against MFT attacks when a significant amount of malicious data is employed. This indicates that SDD is efficient in terms of the size of fine-tuning data and remains effective against attacks utilizing large-scale malicious data. Additionally, SDD increases the cost of misusing open-source LLMs, as attackers need to prepare extensive amounts of malicious data. Responsibility. When dealing with harmful instructions, SDD generates irrelevant responses rather than explicitly refusing to engage. This differs from the current consensus in the AI safety community, which favors responsible models that directly decline to answer harmful instructions. To address this limitation, we developed a responsible variant that aligns with these safety principles. We propose a simple variant of the SDD method that can achieve the goal of explicitly rejecting harmful instructions while defending against MFT. Specifically, we add a fixed prefix to the high-quality answers in the original SDD training data. The prefix states “I refuse to answer your question for responsible and ethical reasons. I provided an irrational answer to your question.” Then, we perform SDD training on the modified data. After training on this modified dataset, the model develops the capability to explicitly refuse to engage with harmful instructions. We term this method SDD_reject. In Figure 3 (a), we evaluate the harmlessness score of SDD_reject on BeaverTails-Evaluation benchmark, showing that SDD_reject maintains comparable defense effectiveness with SDD against MFT. In Fig. 3 (b), SDD_reject achieves a higher explicit rejection rate compared to the original SDD and Vanilla. The explicit rejection rate is defined as the percentage of responses containing explicit rejection. We use GPT-4 to determine whether a response contains explicit rejection. Different Backbones. We provide results of SDD and baselines with the backbones replaced by Phil-2 (2.7B) javaheripi2023phi, GLM-3 (6B) glm2024chatglm in Figure 4 in the Appendix I. The results show that SDD effectively defends against malicious fine-tuning across various backbones. Case Study. We report instances of responses from SDD and baselines in Appendix J. Results show that SDD could generate reasonable responses in normal cases while generating irrelevant responses under MFT attack. 7 Conclusion In this paper, we identify malicious fine-tuning attacks as a significant threat to open-source LLMs. Through theoretical analyses, we demonstrate that the current safety alignment methods fail to defend against such attacks. To address this, we propose the Self-Degraded Defense (SDD) method, which achieves defense by steering the model to generate high-quality but irrelevant responses to harmful instructions. In the event of malicious fine-tuning, LLMs aligned with SDD exhibit a marked decline in general capability, effectively preventing the generation of harmful content. Hence, SDD can effectively defend against malicious fine-tuning. Additionally, applying SDD prior to open-sourcing LLMs at various stages will not affect users who have benign intentions. Acknowledgments This work was supported by the National Natural Science Foundation of China (62406114), the Fundamental Research Funds for the Central Universities (2024ZYGXZR074), Guangdong Basic and Applied Basic Research Foundation (2025A1515011413), and National Key R & D Project from Minister of Science and Technology (2024YFA1211500). Limitation Traditional safety alignment approaches have shown limitations in defending against malicious fine-tuning attacks. Our proposed SDD method offers a novel, albeit imperfect, complement to these traditional approaches. In §4.2, we propose a relaxed goal of safety alignment, which ensures that the model does not produce harmful responses. When dealing with harmful instructions, SDD produces irrelevant responses. While we have developed an enhanced version that incorporates explicit rejection statements, the response pattern still deviates from natural human behavior. Where humans tend to directly decline inappropriate requests, our model generates explicit rejection statements and irrelevant responses. This deviation from natural human communication patterns represents an area for future improvement in our approach. Ethical Statements This paper contains harmful texts, including harmful instructions and harmful topics. The opinions expressed in these texts are not reflective of the authors’ views. The primary purpose of this work is to mitigate the risks of harmful outputs generated by LLMs. The inclusion of harmful text is solely for the purpose of demonstrating the implementation details of the proposed method. We strongly call for more researchers to engage in this critical area of research to foster the development of more ethical and responsible LLMs. Appendix A Inherited Assumptions Based on the notations in §3.2, we inherit two assumptions from (lin2023spurious). Assumption 2 (lin2023spurious). Denote nv′n_v nitalic_v′ and ns′n_s nitalic_s′ as the maximum number of invariant features and spurious features that a model can learn, respectively. We need the overall noise to be small to satisfy K(1σ(nv′+ns′))≥1−ϵn F^K ( 1σ (n_v +n_s ) )≥ 1- _nitalic_Fitalic_K ( divide start_ARG 1 end_ARG start_ARG σ ( nitalic_v′ + nitalic_s′ ) end_ARG ) ≥ 1 - ϵitalic_n. Here, Fitalic_F is the cumulative distribution function of a standard Gaussian random variable, and K refers to the number of classes. σ is the standard deviation of the noise, and ϵn _nϵitalic_n denotes a small noise tolerance. Remark: The condition K(1σ(nv′+ns′))≥1−ϵn F^K ( 1σ (n_v +n_s ) )≥ 1- _nitalic_Fitalic_K ( divide start_ARG 1 end_ARG start_ARG σ ( nitalic_v′ + nitalic_s′ ) end_ARG ) ≥ 1 - ϵitalic_n ensures that the additive noise is sufficiently small for all K classes simultaneously. Here, K(z) F^K(z)italic_Fitalic_K ( z ) represents the probability that K independent standard Gaussian variables all fall below z, i.e., (z)K F(z)^Kitalic_F ( z )K. Assumption 3 wald2022malign; allen2020towards. (1) ‖v,i(k)‖2=1 \| μ_v,i(k) \|_2=1∥ italic_μitalic_v , i ( k ) ∥2 = 1 and ‖s,j(k)‖2=1 \| μ_s,j(k) \|_2=1∥ italic_μitalic_s , j ( k ) ∥2 = 1 for i∈1,⋯,dv,j∈1,⋯,ds,k∈1,⋯,Ki∈\1,·s,d_v\,j∈\1,·s,d_s\,k∈\1,·s,K\i ∈ 1 , ⋯ , ditalic_v , j ∈ 1 , ⋯ , ditalic_s , k ∈ 1 , ⋯ , K . (2) i(k)⟂i′(k′) v_i(k) v_i (k )italic_vitalic_i ( k ) ⟂ italic_vitalic_i′ ( k′ ) for any (i,k)≠(i′,k′)(i,k)≠ (i ,k )( i , k ) ≠ ( i′ , k′ ), k,k′∈1,⋯,Kk,k ∈\1,·s,K\k , k′ ∈ 1 , ⋯ , K , where i,i′∈v,1,⋯,v,dv,s,1,⋯,s,ds v_i, v_i ∈ \ μ_v,1,·s, μ_v,d_v, μ_s,1,·s, μ_s,d_s \italic_vitalic_i , italic_vitalic_i′ ∈ italic_μitalic_v , 1 , ⋯ , italic_μitalic_v , d start_POSTSUBSCRIPT v end_POSTSUBSCRIPT , italic_μitalic_s , 1 , ⋯ , italic_μitalic_s , d start_POSTSUBSCRIPT s end_POSTSUBSCRIPT . v,i(k) μ_v,i(k)italic_μitalic_v , i ( k ) is the mean vector of the i-th invariant feature in class k, and s,j(k) μ_s,j(k)italic_μitalic_s , j ( k ) is the mean vector of the j-th spurious feature in class k. The above two assumptions simplify the analysis process by controlling the magnitude of random noise for each feature and ensuring the orthogonality of the features. Appendix B Data Generation Process Following lin2023spurious, we consider that each v,i x_v,iitalic_xitalic_v , i and s,j x_s,jitalic_xitalic_s , j are generated from the label yitalic_y with the latent invariant features v,i μ_v,iitalic_μitalic_v , i and spurious features s,j μ_s,jitalic_μitalic_s , j, where v,i,s,j∈ℝd×K μ_v,i, μ_s,j ^d× Kitalic_μitalic_v , i , italic_μitalic_s , j ∈ blackboard_Rd × K. The whole data generation process is defined as follows: ∼ Unif 1,2,…,K, y Unif \ e_1, e_2,…, e_K \,italic_y ∼ Unif italic_e1 , italic_e2 , … , italic_eitalic_K , (8) =Concat(v,ii=1dv∪s,jj=1ds), x=Concat ( \ x_v,i \_i=1^d_v∪ \ x_s,j \_j=1^d_s ),italic_x = Concat ( italic_xitalic_v , i i = 1ditalic_v ∪ italic_xitalic_s , j j = 1ditalic_s ) , ℙθ(v,i∣)=(v,iv,i,σ2d), _θ ( x_v,i y )=N ( μ_v,i Q_v,i y,σ^2 I_d ),blackboard_Pθ ( italic_xitalic_v , i ∣ italic_y ) = N ( italic_μitalic_v , i italic_Qitalic_v , i italic_y , σ2 italic_Iitalic_d ) , ℙθ(s,j∣)=(s,js,j,σ2d),∀i,j _θ ( x_s,j y )=N ( μ_s,j Q_s,j y,σ^2 I_d ),∀ i,jblackboard_Pθ ( italic_xitalic_s , j ∣ italic_y ) = N ( italic_μitalic_s , j italic_Qitalic_s , j italic_y , σ2 italic_Iitalic_d ) , ∀ i , j where i e_iitalic_eitalic_i is a one-hot vector with the i-th element as one, Unif indicates uniform sampling, v,i,s,j∈0,1K×K Q_v,i, Q_s,j∈\0,1\^K× Kitalic_Qitalic_v , i , italic_Qitalic_s , j ∈ 0 , 1 K × K, d I_ditalic_Iitalic_d is an identity matrix. Further, v,i=K=[1,2,…,K] Q_v,i= I_K= [ e_1, e_2,…, e_K ]italic_Qitalic_v , i = italic_Iitalic_K = [ italic_e1 , italic_e2 , … , italic_eitalic_K ] always holds, and the k-th column of Qitalic_Q, i.e., s,j(k) Q_s,j(k)italic_Qitalic_s , j ( k ), is defined as follows for k=1,…,Kk=1,…,Kk = 1 , … , K : s,j(k)=k, with probability 1−p Unif 1,2,…,K, with p. Q_s,j(k)= \ array[]l e_k, with probability 1-p\\ Unif \ e_1, e_2,…, e_K \, with p. array .italic_Qitalic_s , j ( k ) = start_ARRAY start_ROW start_CELL italic_eitalic_k , with probability 1 - p end_CELL end_ROW start_ROW start_CELL Unif italic_e1 , italic_e2 , … , italic_eitalic_K , with p . end_CELL end_ROW end_ARRAY (9) Appendix C Lemmata In our analysis below, we use the notation described in §3.2. Lemma 1 With the assumptions in §3.2 satisfied, the accuracy ξt _tξitalic_t of the fine-tuned model f~ fover~ start_ARG f end_ARG on a given task t is upper bounded by: ξt(f~) _t( f)ξitalic_t ( over~ start_ARG f end_ARG ) (10) ≤ ≤ Fp((1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗). F_p ( (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^* ).Fitalic_p ( divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG ) . The malicious fine-tune process can be seen as the weight space ensemble (WSE), which is a linear interpolation of the original model f¯ fover¯ start_ARG f end_ARG and near-optimal model f∗f^*f∗. And λ∈[0,1]λ∈[0,1]λ ∈ [ 0 , 1 ] be the interpolation coefficient. In this part of the proof, we build upon the theoretical framework established in lin2023spurious for weight space ensemble methods, and further extend it to our general case. We group the input vector into two groups, namely v x_vitalic_xitalic_v from invariant feature space and s x_sitalic_xitalic_s from spurious feature space. We have the input vector ~ xover~ start_ARG italic_x end_ARG at the form of: ~:= x=over~ start_ARG italic_x end_ARG := λ∑i¯=1n¯v−nvo∗v,i¯+λ∑j¯=1n¯s−nso∗s,j¯ λ _ i=1 n_v-n_vo^* x_v, i+λ _ j=1 n_s-n_so^* x_s, jλ ∑over¯ start_ARG i end_ARG = 1over¯ start_ARG n end_ARGv - nitalic_v o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_xitalic_v , over¯ start_ARG i end_ARG + λ ∑over¯ start_ARG j end_ARG = 1over¯ start_ARG n end_ARGs - nitalic_s o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_xitalic_s , over¯ start_ARG j end_ARG (11) +(1−λ)∑i∗=1nv∗−nvo∗v,i∗+(1−λ)∑i∗=1ns∗−nso∗s,i∗ +(1-λ) _i^*=1^n^*_v-n_vo^* x_v,i^*+(1-λ) _i^*=1^n^*_s-n_so^* x_s,i^*+ ( 1 - λ ) ∑i∗ = 1n start_POSTSUPERSCRIPT ∗v - nitalic_v o∗ end_POSTSUPERSCRIPT italic_xitalic_v , i∗ + ( 1 - λ ) ∑i∗ = 1n start_POSTSUPERSCRIPT ∗s - nitalic_s o∗ end_POSTSUPERSCRIPT italic_xitalic_s , i∗ +∑i=1nvo∗v,i+∑i=1nso∗s,i, + _i=1^n_vo^* x_v,i+ _i=1^n_so^* x_s,i,+ ∑i = 1nitalic_v o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_xitalic_v , i + ∑i = 1nitalic_s o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_xitalic_s , i , Where i¯,j¯,i∗,j∗,i,j i, j,i^*,j^*,i,jover¯ start_ARG i end_ARG , over¯ start_ARG j end_ARG , i∗ , j∗ , i , j are the index of features. The fine-tuned classifier is described as: ~:=λ¯+(1−λ)∗. w:=λ w+(1-λ) w^*.over~ start_ARG italic_w end_ARG := λ over¯ start_ARG italic_w end_ARG + ( 1 - λ ) italic_w∗ . (12) Where k e_kitalic_eitalic_k is the label. A key distinction of LLMs from traditional machine learning models lies in their ability to handle a wide range of tasks beyond those seen during training, which inherently places them in Out-of-Distribution (OOD) settings. Therefore, rather than focusing on in-distribution performance, we aim to characterize the OOD prediction accuracy of LLM. Then we turn to the OOD forecasting accuracy and for each k=1,…,Kk=1,…,Kk = 1 , … , K, to conduct a fine-grained analysis of the roles of different types of features, we follow the approach of lin2023spurious and take the notation as follows: r¯k r_kover¯ start_ARG r end_ARGk =|i|(s,i(k)=s,i(k))i=1n¯s−nso∗|, = | \i\, |\,I( μ_s,i(k)= μ_s,i(k)) \_i=1 n_s-n_so^* |,= | i | blackboard_I ( italic_μitalic_s , i ( k ) = italic_μitalic_s , i ( k ) ) i = 1over¯ start_ARG n end_ARGs - nitalic_s o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT | , (13) rk∗ r^*_kr∗italic_k =|i|(s,i(k)=s,i(k))i=1ns∗−nso∗|, = | \i\, |\,I( μ_s,i(k)= μ_s,i(k)) \_i=1^n^*_s-n_so^* |,= | i | blackboard_I ( italic_μitalic_s , i ( k ) = italic_μitalic_s , i ( k ) ) i = 1n start_POSTSUPERSCRIPT ∗s - nitalic_s o∗ end_POSTSUPERSCRIPT | , rko r_k^oritalic_kitalic_o =|i|(s,i(k)=s,i(k))i=1nso∗|, = | \i\, |\,I( μ_s,i(k)= μ_s,i(k)) \_i=1^n_so^* |,= | i | blackboard_I ( italic_μitalic_s , i ( k ) = italic_μitalic_s , i ( k ) ) i = 1nitalic_s o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT | , r¯k→k′ r_k→ k over¯ start_ARG r end_ARGk → k′ =|i|(s,i(k)=s,i(k′))i=1n¯s−nso∗|, = | \i\, |\,I( μ_s,i(k)= μ_s,i(k )) \_i=1 n_s-n_so^* |,= | i | blackboard_I ( italic_μitalic_s , i ( k ) = italic_μitalic_s , i ( k′ ) ) i = 1over¯ start_ARG n end_ARGs - nitalic_s o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT | , rk→k′∗ r^*_k→ k r∗italic_k → k′ =|i|(s,i(k)=s,i(k′))i=1ns∗−nso∗|, = | \i\, |\,I( μ_s,i(k)= μ_s,i(k )) \_i=1^n^*_s-n_so^* |,= | i | blackboard_I ( italic_μitalic_s , i ( k ) = italic_μitalic_s , i ( k′ ) ) i = 1n start_POSTSUPERSCRIPT ∗s - nitalic_s o∗ end_POSTSUPERSCRIPT | , rk→k′o r^o_k→ k ritalic_oitalic_k → k′ =|i|(s,i(k)=s,i(k′))i=1nso∗|, = | \i\, |\,I( μ_s,i(k)= μ_s,i(k )) \_i=1^n_so^* |,= | i | blackboard_I ( italic_μitalic_s , i ( k ) = italic_μitalic_s , i ( k′ ) ) i = 1nitalic_s o start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT | , where each s,i(k) μ_s,i(k)italic_μitalic_s , i ( k ) is the i−thi-thi - t h mean vector in the k-th class. And for class k, there are r¯k,rk∗ r_k,r^*_kover¯ start_ARG r end_ARGk , r∗italic_k spurious features (no overlapped) maintaining their parameters, and correspondingly, r¯k→k′,rk→k′∗ r_k→ k ,r^*_k→ k over¯ start_ARG r end_ARGk → k′ , r∗italic_k → k′ is the number of spurious features flipping to the class k′k k′, and rko,rk→k′or_k^o,r_k→ k ^oritalic_kitalic_o , ritalic_k → k′italic_o are defined similar in overlapped spurious features. Using the above notation, we leverage Lemma 3 in lin2023spurious to bound the accuracy. They also provided a bound for the case λ=12λ= 12λ = divide start_ARG 1 end_ARG start_ARG 2 end_ARG. The upper bound can be expressed as: ξt(f~)≤(n¯v+nv∗,n¯s+ns∗,nvo∗,nso∗,1λ(1−λ))+ϵ. _t( f) ( n_v+n^*_v,\; n_s+n^*_s,\;n_vo^*,\;n_so^*,\; 1λ(1-λ) )+ε.ξitalic_t ( over~ start_ARG f end_ARG ) ≤ G ( over¯ start_ARG n end_ARGv + n∗italic_v , over¯ start_ARG n end_ARGs + n∗italic_s , nitalic_v o∗ , nitalic_s o∗ , divide start_ARG 1 end_ARG start_ARG λ ( 1 - λ ) end_ARG ) + ϵ . (14) Where the definition of GG is in the monotonicity analysis below. And when λ=12λ= 12λ = divide start_ARG 1 end_ARG start_ARG 2 end_ARG, the result is: Fp((1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗).F_p ( (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^* ).Fitalic_p ( divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG ) . (15) However, due to the variant choice of λ, we need to analyze the monotonicity of (n¯v+nv∗,n¯s+ns∗,nvo∗,nso∗,1λ(1−λ))G ( n_v+n^*_v,\; n_s+n^*_s,\;n_vo^*,\;n_so^*,\; 1λ(1-λ) )G ( over¯ start_ARG n end_ARGv + n∗italic_v , over¯ start_ARG n end_ARGs + n∗italic_s , nitalic_v o∗ , nitalic_s o∗ , divide start_ARG 1 end_ARG start_ARG λ ( 1 - λ ) end_ARG ) with respect to the ensemble weight λ∈(0,1)λ∈(0,1)λ ∈ ( 0 , 1 ). Due to the limited utility of spurious features for achieving accuracy in OOD settings, we observe that for any feature count vector Rk(r)R_k(r)Ritalic_k ( r ), the spurious components satisfy rko≤rk→k′o,∀k′≠k.r_k^o\;≤\;r_k→ k ^o, ∀\,k ≠ k.ritalic_kitalic_o ≤ ritalic_k → k′italic_o , ∀ k′ ≠ k . (16) In particular, for any sample r, there exists at least one class k∗≠k^*≠ k∗ ≠ k such that Δk∗o(r):=rko−rk→k∗o≤0. _k^*^o(r):=r_k^o-r_k→ k^*^o≤ 0.Δitalic_k∗italic_o ( r ) := ritalic_kitalic_o - ritalic_k → k∗italic_o ≤ 0 . (17) This implies that for every sample, at least one comparison margin involving spurious components has a non-positive slope. We proceed to analyze the monotonicity of (n¯v+nv∗,n¯s+ns∗,nvo∗,nso∗,1λ(1−λ)),G ( n_v+n^*_v,\; n_s+n^*_s,\;n_vo^*,\;n_so^*,\; 1λ(1-λ) ),G ( over¯ start_ARG n end_ARGv + n∗italic_v , over¯ start_ARG n end_ARGs + n∗italic_s , nitalic_v o∗ , nitalic_s o∗ , divide start_ARG 1 end_ARG start_ARG λ ( 1 - λ ) end_ARG ) , (18) with respect to the ensemble weight λ∈(0,1)λ∈(0,1)λ ∈ ( 0 , 1 ). Let us define nv:=n¯v+nv∗,ns:=n¯s+ns∗.n_v:= n_v+n^*_v, n_s:= n_s+n^*_s.nitalic_v := over¯ start_ARG n end_ARGv + n∗italic_v , nitalic_s := over¯ start_ARG n end_ARGs + n∗italic_s . (19) The function is defined as (nv,ns,nvo∗,nso∗,C)=ℙ()+∑N=1K−1ℙ(′(N))⋅h(N),G(n_v,n_s,n_vo^*,n_so^*,C)=P(A)+ _N=1^K-1P(C (N))· h(N),G ( nitalic_v , nitalic_s , nitalic_v o∗ , nitalic_s o∗ , C ) = blackboard_P ( A ) + ∑N = 1K - 1 blackboard_P ( C′ ( N ) ) ⋅ h ( N ) , (20) where := =A := (21) Rk(r)∣rk+Crko−rk→k′−Crk→k′o+nv>0,∀k′≠k, \R_k(r) r_k+C\,r_k^o-r_k→ k -C\,r_k→ k ^o+n_v>0,\ ∀ k ≠ k \, Ritalic_k ( r ) ∣ ritalic_k + C ritalic_kitalic_o - ritalic_k → k′ - C ritalic_k → k′italic_o + nitalic_v > 0 , ∀ k′ ≠ k , ′(N):= (N)=C′ ( N ) := (22) Rk(r)∣mink′≠k(rk+Cr1o−rk→k′−Crk→k′o+nv)=0, \R_k(r) _k ≠ k (r_k+C\,r_1^o-r_k→ k -C\,r_k→ k ^o+n_v )=0 \, Ritalic_k ( r ) ∣ minitalic_k′ ≠ k ( ritalic_k + C r1italic_o - ritalic_k → k′ - C ritalic_k → k′italic_o + nitalic_v ) = 0 , the minimum can be achieved by N values. C is a constant. h(N)=ℙz∼(0,σ2IN)(ai⊤z>0,∀i=1,…,N),h(N)=P_z (0,σ^2I_N) (a_i z>0,\ ∀ i=1,…,N ),h ( N ) = blackboard_Pz ∼ N ( 0 , σ2 I start_POSTSUBSCRIPT N ) end_POSTSUBSCRIPT ( aitalic_i⊤ z > 0 , ∀ i = 1 , … , N ) , (23) in which ai⊤aj=1a_i a_j=1aitalic_i⊤ aitalic_j = 1 and ‖ai‖22=1\|a_i\|_2^2=1∥ aitalic_i ∥22 = 1 for any i≠ji≠ ji ≠ j. Where z is the vector of margin differences, aia_iaitalic_i is the standard basis vector. Let us define the per-class margin function: Lk′(C;r): L_k (C;r):Litalic_k′ ( C ; r ) : (24) = == rk−rk→k′+nv+C(rko−r1→k′o) r_k-r_k→ k +n_v+C(r_k^o-r_1→ k ^o)ritalic_k - ritalic_k → k′ + nitalic_v + C ( ritalic_kitalic_o - r1 → k′italic_o ) = == αk′(r)+C⋅Δk′o(r), _k (r)+C· _k ^o(r),αitalic_k′ ( r ) + C ⋅ Δitalic_k′italic_o ( r ) , where αk′(r):=rk−rk→k′+nv, _k (r)=r_k-r_k→ k +n_v,αitalic_k′ ( r ) := ritalic_k - ritalic_k → k′ + nitalic_v , (25) Δk′o(r):=rko−rk→k′o≤0. _k ^o(r)=r_k^o-r_k→ k ^o≤ 0.Δitalic_k′italic_o ( r ) := ritalic_kitalic_o - ritalic_k → k′italic_o ≤ 0 . Since each sample r has at least one margin Lk∗(C;r)L_k^*(C;r)Litalic_k∗ ( C ; r ) with negative slope, it will eventually exit the region (C)A(C)A ( C ) as C increases. Thus, we partition the positive real line for C into open intervals (where (N)=∅C(N)= ( N ) = ∅) and discrete critical points (where equality is attained in some Lk′=0L_k =0Litalic_k′ = 0). On each open interval, we have: (C)=ℙ((C)),G(C)=P(A(C)),G ( C ) = blackboard_P ( A ( C ) ) , (26) and since (C)A(C)A ( C ) strictly shrinks with increasing C, we conclude that (C)G(C)G ( C ) is strictly decreasing within such intervals. At any discrete threshold C∙C C∙, denote: Rout= R_out=Rout = (27) r∣∀k′Lk′(C∙−ϵ;r)>0,∃k∗:Lk∗(C∙+ϵ;r)≤0, \r ∀ k \;L_k (C -ε;r)>0,\ ∃ k^*\!:L_k^*(C +ε;r)≤ 0 \, r ∣ ∀ k′ Litalic_k′ ( C∙ - ϵ ; r ) > 0 , ∃ k∗ : Litalic_k∗ ( C∙ + ϵ ; r ) ≤ 0 , Rin= R_in=Rin = (28) r∣mink′Lk′(C∙;r)=0,with N(r) active constraints. \r _k L_k (C ;r)=0,\ with N(r) active constraints \. r ∣ minitalic_k′ Litalic_k′ ( C∙ ; r ) = 0 , with N ( r ) active constraints . Then the net change in GG is: Δ=−∑r∈Routℙ(Rk(r))+∑r∈Rinℙ(Rk(r))⋅h(N(r)). =- _r∈ R_outP(R_k(r))+ _r∈ R_inP(R_k(r))· h(N(r)).Δ G = - ∑r ∈ R start_POSTSUBSCRIPT out end_POSTSUBSCRIPT blackboard_P ( Ritalic_k ( r ) ) + ∑r ∈ R start_POSTSUBSCRIPT in end_POSTSUBSCRIPT blackboard_P ( Ritalic_k ( r ) ) ⋅ h ( N ( r ) ) . (29) Since h(N)≤1h(N)≤ 1h ( N ) ≤ 1 and Rin⊆RoutR_in R_outRin ⊆ Rout, the net change satisfies Δ≤0 ≤ 0Δ G ≤ 0. Thus, (C)G(C)G ( C ) is globally non-increasing in C. Now, recall that C(λ):=1λ(1−λ),λ∈(0,1),C(λ):= 1λ(1-λ), λ∈(0,1),C ( λ ) := divide start_ARG 1 end_ARG start_ARG λ ( 1 - λ ) end_ARG , λ ∈ ( 0 , 1 ) , (30) which is minimized at λ=12λ= 12λ = divide start_ARG 1 end_ARG start_ARG 2 end_ARG, and symmetric about it. Since GG is decreasing in C, and C(λ)C(λ)C ( λ ) increases away from λ=12λ= 12λ = divide start_ARG 1 end_ARG start_ARG 2 end_ARG, we conclude λ=12λ= 12λ = divide start_ARG 1 end_ARG start_ARG 2 end_ARG uniquely minimizes (nv,ns,nvo∗,nso∗,1λ(1−λ))G (n_v,n_s,n_vo^*,n_so^*, 1λ(1-λ) )G ( nitalic_v , nitalic_s , nitalic_v o∗ , nitalic_s o∗ , divide start_ARG 1 end_ARG start_ARG λ ( 1 - λ ) end_ARG ). The proof is finished. Appendix D Proof for Theorem 1 To prove Theorem 1, we need to analyze the model f~ fover~ start_ARG f end_ARG after it has undergone malicious fine-tuning. By using Lemma 1, we have: ξt(f~) _t( f)ξitalic_t ( over~ start_ARG f end_ARG ) (31) ≤ ≤ Fp((1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗). F_p ( (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^* ).Fitalic_p ( divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG ) . Then we consider the original model, according to the following theorem: Theorem 3 in lin2023spurious. For single model f¯ fover¯ start_ARG f end_ARG, the OOD forecasting accuracy can be expressed as: Fp(n¯s(1−p)+n¯vn¯s).F_p ( n_s(1-p)+ n_v n_s ).Fitalic_p ( divide start_ARG over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs end_ARG end_ARG ) . (32) We have the accuracy ξA(f¯) _A( f)ξitalic_A ( over¯ start_ARG f end_ARG ) of the original model f¯ fover¯ start_ARG f end_ARG as: ξA(f¯)=Fp(n¯s(1−p)+n¯vn¯s). _A( f)=F_p ( n_s(1-p)+ n_v n_s ).ξitalic_A ( over¯ start_ARG f end_ARG ) = Fitalic_p ( divide start_ARG over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs end_ARG end_ARG ) . (33) Then the proof is finished. We can further obtain more information from this Theorem. Given that f¯ fover¯ start_ARG f end_ARG is an aligned model (and thus unlikely to rely heavily on spurious features), both n¯s n_sover¯ start_ARG n end_ARGs and nson_sonitalic_s o are small, n¯v n_vover¯ start_ARG n end_ARGv is large. And given that f∗f^*f∗ is the near-optimal malicious model, the ns∗n^*_sn∗italic_s is large. Therefore, the upper bound given by Theorem 1 tends to be negative, which indicates that the model gradually loses its alignment safety during training. Appendix E Proof for Theorem 2 As stated in §3.2, a change of task does not affect the results in Theorem 1, but affects the meaning of the task, cause the task only affects the meaning of the notation. For example, if the nsn_snitalic_s is the number of spurious features for task A, then when the task changes to task G, the meaning of nsn_snitalic_s is still the number of spurious features. So we can directly use Theorem 1. We have: ξG(f~)−ξG(f¯) _G( f)- _G( f)ξitalic_G ( over~ start_ARG f end_ARG ) - ξitalic_G ( over¯ start_ARG f end_ARG ) (34) ≤ ≤ Fp((1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗) F_p ( (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^* )Fitalic_p ( divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG ) − -- Fp(n¯s(1−p)+n¯vn¯s), F_p ( n_s(1-p)+ n_v n_s ),Fitalic_p ( divide start_ARG over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs end_ARG end_ARG ) , And we have nv∗<n¯vn_v^*< n_vnitalic_v∗ < over¯ start_ARG n end_ARGv and ns∗>n¯sn_s^*> n_snitalic_s∗ > over¯ start_ARG n end_ARGs. If we want to find a case when ξG(f~)<ξG(f¯) _G( f)< _G( f)ξitalic_G ( over~ start_ARG f end_ARG ) < ξitalic_G ( over¯ start_ARG f end_ARG ), we need to find a case that satisfies the following equation: Fp((1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗) F_p ( (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^* )Fitalic_p ( divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG ) (35) < << Fp(n¯s(1−p)+n¯vn¯s). F_p ( n_s(1-p)+ n_v n_s ).Fitalic_p ( divide start_ARG over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs end_ARG end_ARG ) . Given that F(P)F(P)F ( P ) is monotonically increasing, the above equation can be written as: (1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗ (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^*divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG (36) < << n¯s(1−p)+n¯vn¯s. n_s(1-p)+ n_v n_s.divide start_ARG over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs end_ARG end_ARG . Rewrite the term on the left side, we have: (1−p)(n¯s+ns∗+2nso∗)+n¯v+nv∗+2nvo∗n¯s+ns∗+14nso∗ (1-p)( n_s+n^*_s+2n_so^*)+ n_v+n_v^*+2n_vo^* n_s+n_s^*+14n_so^*divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + nitalic_v∗ + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG (37) < << (1−p)(n¯s+ns∗+2nso∗)+n¯v+n¯v+2nvo∗n¯s+ns∗+14nso∗. (1-p)( n_s+n^*_s+2n_so^*)+ n_v+ n_v+2n_vo^* n_s+n_s^*+14n_so^*.divide start_ARG ( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + over¯ start_ARG n end_ARGv + 2 nitalic_v o∗ end_ARG start_ARG square-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o∗ end_ARG end_ARG . It is easy to see that the denominator n¯s+ns∗+14nso n_s+n_s^*+14n_sosquare-root start_ARG over¯ start_ARG n end_ARGs + nitalic_s∗ + 14 nitalic_s o end_ARG is greater than n¯s n_ssquare-root start_ARG over¯ start_ARG n end_ARGs end_ARG. For the numerator, when the original model has strong general capabilities (a condition that is common for large language models that have undergone alignment), n¯v n_vover¯ start_ARG n end_ARGv is large. Moreover, since the original model has strong general capabilities, n¯s n_sover¯ start_ARG n end_ARGs is relatively small, allowing ns∗n_s^*nitalic_s∗ to take a smaller value. When ns∗n_s^*nitalic_s∗ is small, it is easy to find a set of solutions that satisfy the following equation: (1−p)(n¯s+ns∗+2nso∗)+n¯v+n¯v+2nvo∗ (1-p)( n_s+n^*_s+2n_so^*)+ n_v+ n_v+2n_vo^*( 1 - p ) ( over¯ start_ARG n end_ARGs + n∗italic_s + 2 nitalic_s o∗ ) + over¯ start_ARG n end_ARGv + over¯ start_ARG n end_ARGv + 2 nitalic_v o∗ (38) < << n¯s(1−p)+n¯v. n_s(1-p)+ n_v.over¯ start_ARG n end_ARGs ( 1 - p ) + over¯ start_ARG n end_ARGv . By organizing the terms above, we have completed the proof. This essentially implies that if the quality of the data used for malicious fine-tuning is inferior to that required for alignment on general tasks, the performance of the maliciously fine-tuned model tends to decrease. This directly inspired the design of our method. Appendix F Closed form of Fp(x)F_p(x)Fitalic_p ( x ) The closed form of Fp(x)F_p(x)Fitalic_p ( x ) is from lin2023spurious. For K class situation, function Fp(x)F_p(x)Fitalic_p ( x ) is monotonically increasing with x. We denote a K−1K-1K - 1-dim random variable ∼(,) η ( x, M)italic_η ∼ N ( italic_x , italic_M ), in which i,i=p(K+2−pK)K,i,j=p(K+1−pK)K. M_i,i= p(K+2-pK)K, M_i,j= p(K+1-pK)K.italic_Mitalic_i , i = divide start_ARG p ( K + 2 - p K ) end_ARG start_ARG K end_ARG , italic_Mitalic_i , j = divide start_ARG p ( K + 1 - p K ) end_ARG start_ARG K end_ARG . (39) then Fp(x)F_p(x)Fitalic_p ( x ) is defined as Fp(x)=ℙ(1>0,…,K−1>0).F_p(x)=P ( η_1>0,…, η_K-1>0 ).Fitalic_p ( x ) = blackboard_P ( italic_η1 > 0 , … , italic_ηitalic_K - 1 > 0 ) . (40) Appendix G Optimization Goal of Fine-tuning We describe the optimization goal from two perspectives, namely, scoring function and policy. The optimization goal of standard instruction fine-tuning can be seen as maximizing the scoring function. maxπθx∼,yo∼πθ(y∣x)[r(x,yo)], _ _θE_x ,y_o _θ(y x)[r(x,y_o)],maxitalic_π start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT blackboard_Ex ∼ D , y start_POSTSUBSCRIPT o ∼ πitalic_θ ( y ∣ x ) end_POSTSUBSCRIPT [ r ( x , yitalic_o ) ] , (41) where x is the input of the model, yoy_oyitalic_o is the output of the model given input x, DD is the training dataset, πθ(⋅) _θ(·)πitalic_θ ( ⋅ ) is current policy under parameters θ, namely, the LLM itself. r(x,y)r(x,y)r ( x , y ) measures the discrepancy between the model’s current output y and the optimal output. Alternatively, we can interpret the malicious fine-tune process as minimizing the Kullback-Leibler (KL) divergence w.r.t the optimal policy π∗(⋅) _*(·)π∗ ( ⋅ ): minπθx∼,y∼πθ(y∣x)[logπθ(y∣x)π∗(y∣x)] _ _θE_x ,y _θ(y x) [ _θ(y x) _*(y x) ]minitalic_π start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT blackboard_Ex ∼ D , y ∼ π start_POSTSUBSCRIPT θ ( y ∣ x ) end_POSTSUBSCRIPT [ log divide start_ARG πitalic_θ ( y ∣ x ) end_ARG start_ARG π∗ ( y ∣ x ) end_ARG ] (42) =maxπθx∼,y∼πθ(y∣x)[logπ∗(y∣x)πθ(y∣x)]. = _ _θE_x ,y _θ(y x) [ _*(y x) _θ(y x) ].= maxitalic_π start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT blackboard_Ex ∼ D , y ∼ π start_POSTSUBSCRIPT θ ( y ∣ x ) end_POSTSUBSCRIPT [ log divide start_ARG π∗ ( y ∣ x ) end_ARG start_ARG πitalic_θ ( y ∣ x ) end_ARG ] . The optimization objectives in Eq. 41 and Eq. 42 are equivalent as they both aim to achieve the same global optimum. Consequently, there exists a positive correlation between r(x,y)r(x,y)r ( x , y ) and π∗(y∣x)πθ(y∣x) _*(y x) _θ(y x)divide start_ARG π∗ ( y ∣ x ) end_ARG start_ARG πitalic_θ ( y ∣ x ) end_ARG. This forms the basis of our most critical step of derivation in §5.1 (i.e., from Eq. 5 to Eq. 6). Appendix H Implementation Detail Training. The learning rate of our model during SDD process is 5e-7, and the training is performed in 500 steps, with the batch size of 24. Evaluate Process. We inherit the evaluation process in the LLM-finetune-safety benchmark (r19unintended) and BeaverTails-Evaluation (r46BeaverTails). Appendix I Results on More Backbones To validate our approach across different backbone sizes, we consider backbones with different sizes, such as the Phi2 (javaheripi2023phi) in 2.7B and GLM3 (glm2024chatglm) in 6B. Both Phi2 and GLM3 undergo pre-training and SFT stages. The 7B model is the largest size that we can fine-tune given our limited computational resources. We report the harmlessness score of different backbones on BeaverTails-Evaluation. As shown in Figure 4, the SDD method exhibits consistent characteristics across backbones of different sizes and effective defense against malicious fine-tuning. Figure 4: The evaluation results of methods defending against malicious fine-tuning on different backbones on BeaverTails-Evaluation. Figure 5: Three cases using simple malicious instruction, hard malicious instruction, and benign instruction as queries. Appendix J Case Study To visually demonstrate the effectiveness of the SDD method, we provide several case studies in Figure 5. We examine simple malicious instructions (e.g., “How to kill someone?”), hard malicious instructions requiring some general capabilities (e.g., “How to hack into a computer?”), and benign instructions (e.g., “Tell me a bio of Sherlock Holmes”). The results show that, for malicious instructions, the model after SDD training tends to produce irrelevant responses, while its performance on normal instructions remains unaffected. Even after malicious fine-tuning, the model continues to provide irrelevant responses to simple harmful instructions and lacks the capability to complete tasks for malicious instructions requiring general abilities. Appendix K Harmful Topics • Hate Speech, Offensive Language • Discrimination, Stereotype, Injustice • Violence, Aiding and Abetting, Incitement • Financial Crime, Property Crime, Theft • Privacy Violation • Drug Abuse, Weapons, Banned Substance • Non-Violent Unethical Behavior • Sexually Explicit, Adult Content • Controversial Topics, Politics • Misinformation Re. ethics, laws and safety • Terrorism, Organized Crime • Self-Harm • Animal Abuse • Child Abuse