Paper deep dive
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing
Yi Wang, Fenghua Weng, Sibei Yang, Zhan Qin, Minlie Huang, Wenjie Wang
Models: Llama 2-7B, Mistral-7B, Qwen 2.5-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 6:44:26 PM
Summary
DELMAN (Dynamic Editing for LLMs JAilbreak DefeNse) is a post-deployment defense mechanism that uses direct model editing to neutralize jailbreak attacks. By identifying harmful tokens and optimizing a target output vector for safe responses, DELMAN updates specific MLP layer weights while using KL-divergence regularization to preserve the model's utility on benign tasks.
Entities (5)
Relation Signals (3)
DELMAN → mitigates → Jailbreak Attacks
confidence 100% · DELMAN outperforms baseline methods in mitigating jailbreak attacks
DELMAN → modifies → MLP
confidence 95% · DELMAN aims to edit W l down to rebuild the connection between harmful-token-related key representation
KL Divergence → preserves → Large Language Models
confidence 90% · KL-divergence is applied to ensure that the updated model remains consistent with its original output distribution
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are widely applied in decision making, but their deployment is threatened by jailbreak attacks, where adversarial users manipulate model behavior to bypass safety measures. Existing defense mechanisms, such as safety fine-tuning and model editing, either require extensive parameter modifications or lack precision, leading to performance degradation on general tasks, which is unsuitable to post-deployment safety alignment. To address these challenges, we propose DELMAN (Dynamic Editing for LLMs JAilbreak DefeNse), a novel approach leveraging direct model editing for precise, dynamic protection against jailbreak attacks. DELMAN directly updates a minimal set of relevant parameters to neutralize harmful behaviors while preserving the model's utility. To avoid triggering a safe response in benign context, we incorporate KL-divergence regularization to ensure the updated model remains consistent with the original model when processing benign queries. Experimental results demonstrate that DELMAN outperforms baseline methods in mitigating jailbreak attacks while preserving the model's utility, and adapts seamlessly to new attack instances, providing a practical and efficient solution for post-deployment model protection.
Tags
Links
- Source: https://arxiv.org/abs/2502.11647
- Canonical: https://arxiv.org/abs/2502.11647
- Code: https://github.com/wanglne/DELMAN
Trouble viewing inline? Open PDF directly →
Full Text
56,573 characters extracted from source content.
Expand or collapse full text
arXiv:2502.11647v2 [cs.CR] 29 May 2025 DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing Yi Wang 1 , Fenghua Weng 1 , Sibei Yang 1 , Zhan Qin 2 , Minlie Huang 3 , Wenjie Wang 1 * 1 ShanghaiTech University, 2 The State Key Laboratory of Blockchain and Data Security, Zhejiang University, 3 The CoAI group, Tsinghua University wangyi2024,wengfh2023,yangsb,wangwj1@shanghaitech.edu.cn, qinzhan@zju.edu.cn, aihuang@tsinghua.edu.cn Abstract Large Language Models (LLMs) are widely applied in decision making, but their deploy- ment is threatened by jailbreak attacks, where adversarial users manipulate model behavior to bypass safety measures. Existing defense mechanisms, such as safety fine-tuning and model editing, either require extensive param- eter modifications or lack precision, leading to performance degradation on general tasks, which is unsuitable to post-deployment safety alignment. To address these challenges, we pro- poseDELMAN(DynamicEditing forLLMs JAilbreak DefeNse), a novel approach leverag- ing direct model editing for precise, dynamic protection against jailbreak attacks.DELMAN directly updates a minimal set of relevant pa- rameters to neutralize harmful behaviors while preserving the model’s utility. To avoid trig- gering a safe response in benign context, we incorporate KL-divergence regularization to en- sure updated model remains consistent with original model when processing benign queries. Experimental results demonstrate thatDEL- MANoutperforms baseline methods in miti- gating jailbreak attacks while preserving the model’s utility, and adapts seamlessly to new attack instances, providing a practical and effi- cient solution for post-deployment model pro- tection. We open sourceDELMANathttps: //github.com/wanglne/DELMAN. 1 Introduction Large Language Models (LLMs) play a significant role in decision-making, underscoring the impor- tance of aligning LLMs with safety standards and human values. To ensure that generated content aligns with human values and avoids harmful infor- mation, various safety alignment methods are em- ployed throughout the model production pipeline, including pre-training by model providers, task- specific adaptations by secondary developers, and deployment for user interactions (illustrated in the * W.Wang is the corresponding author. upper part of Figure 1). Among these three phases, the deployment stage poses the greatest safety risk, as adversarial users can launch “jailbreak attacks” by crafting prompts or optimized suffixes to bypass safety measures (Zou et al., 2023; Liu et al., 2023; Zhou et al., 2024b; Chao et al., 2023). Considering that large-scale modifications to a model’s architecture or parameters become imprac- tical once deployed, and adversarial users repre- sent only a minority, which making it infeasible to construct sufficient labeled datasets for fine-tuning, safety alignment in the deployment phase must meet three essential requirements: (1)Minimal model modificationsto ensure efficiency; (2)Tar- geted defensesthat address adversarial queries without compromising regular user interactions; (3)Dynamic adaptabilityto continuously counter emerging jailbreak examples without requiring ex- tensive retraining. Existing defense mechanisms such as safety fine-tuning (Wang et al., 2022; Gan- guli et al., 2022; Xu et al., 2024a) and model de- coder modification (Wang et al., 2024; Zhao et al., 2024) are unsuitable due to their extensive changes to model architecture or parameters. Model editing, originally designed for knowledge correction (Zhu et al., 2020; Lee et al., 2022; De Cao et al., 2021; Mitchell et al., 2021; Meng et al., 2022a,b), has also been explored as a defense against jailbreak attacks. Approaches likeDINMandLED(Wang et al., 2024; Zhao et al., 2024) rely on indirect model editing that fine-tunes specific layers, but they of- ten lack precision in targeting harmful regions and risk degrading overall model performance. A dynamic jailbreak defense mechanism is es- sential, one that is timely, precise, and minimal in required modifications to the deployed model while effectively countering adversarial attacks. To achieve this, our key motivation is to utilize direct editing that focuses on minimal parameter updates, minimizing interference with the model’s overall performance. Specifically, in this work, we in- Figure 1: Upper: The three phases of safety alignment during LLMs production. Lower: LLMs editing as a dynamic defense mechanism during the deployment stage. troduceDELMAN(DynamicEditing forLLMs JAilbreak DefeNse), a novel approach that dynami- cally protects against jailbreak attacks by directly adjusting the weights of specific layers. As illus- trated in Figure 2,DELMANestablishes a connec- tion between harmful tokens and safe responses by computing an input vectork ∗ from harmful tokens and optimizing a target output vectorv ∗ represent- ing a safe response. The model’s weights are then updated with a closed-form solution so that when the input vector is fed into the model, the output of the targeted layer aligns with the desired safe response, effectively minimizing the likelihood of generating harmful content. To avoid unintended trigger of safe responses in benign contexts (e.g. the word “bomb” in “what is a bomb”), we in- corporate neutral prompts containing harmful to- kens in non-harmful contexts during optimization of the target output vector. KL-divergence (Kull- back and Leibler, 1951) is applied to ensure that the updated model remains consistent with its original output distribution when processing these benign queries. This ensures that the model distinguishes between harmful and harmless uses of the same tokens, avoiding over-correction while maintaining its utility for normal tasks. Our contributions can be summarized as follows: •We proposeDELMAN, a dynamic post- deployment defense that directly edits model pa- rameters to neutralize harmful behaviors while preserving overall performance. •DELMANfocuses on minimal parameter edit- ing utilizing only a small set of harmful queries, enabling rapid, precise, and adaptive defense against unseen jailbreak attempts. •DELMANincludes a KL-divergence regulariza- tion term to avoid triggering safe responses in Figure 2:DELMANconsists of five steps: 1. Extract harmful tokens from the query; 2. Random context sequence genera- tion; 3. Calculatek ∗ of harmful tokens; 4. Estimatev ∗ of safe responseY target ; 5. UpdateW l ∗ down withk ∗ ,v ∗ . benign contexts thus preserving normal utilities. •Extensive experiments demonstrateDELMAN outperforms baseline methods in mitigating jail- break attacks while preserving the model’s util- ity on normal tasks, as well as its transferability and generalization ability to unseen jailbreak at- tacks and harmful queries. A case study is also included to demonstrate thatDELMANcan sup- port continuous updates to counter new jailbreak instances without undermining previous edits. 2 Related Work 2.1 Model Editing Model editing enables targeted behavioral modifi- cations within specific domains and can be catego- rized as indirect editing and direct editing. Indirect model editing involves fine-tuning the model to update knowledge with specifically-designed ob- jective (Zhu et al., 2020; Lee et al., 2022) or use meta-learning with hypernetworks to learn optimal parameter updates (De Cao et al., 2021; Mitchell et al., 2021). However, both approaches require extensive model updates, which risks catastrophic forgetting on non-target tasks. Direct editing refers to directly locating and edit- ing the knowledge-related parameters. Research indicate that factual knowledge is primarily stored in the MLP modules of transformer-based architec- tures (Geva et al., 2020, 2022). Leveraging these insights, model-editing methods like ROME (Meng et al., 2022a) employ causal tracing to identify and edit the parameters encoding the particular knowl- edge. However, ROME is limited to single-instance knowledge editing, restricting its applicability in scenarios requiring large-scale updates. MEMIT extends the approach to support batch knowledge editing, providing a scalable solution for efficient and precise modifications (Meng et al., 2022b). 2.2 Existing Defense to Jailbreak Attacks Recent studies reveal that jailbreak attacks (Zou et al., 2023; Liu et al., 2023; Zhou et al., 2024b; Chao et al., 2023) can bypass security alignment leading LLMs to generate harmful or unethical out- puts. As countermeasures, various defense meth- ods are developed against such threats. Existing defenses can be categorized into active defenses and passive defenses. Active defense enhances LLMs robustness against adversarial prompting by dynamically altering model parameters (Wang et al., 2022; Ganguli et al., 2022; Xu et al., 2024a; Wang et al., 2024; Zhao et al., 2024). A common approach to safety training involves constructing safety-relevant datasets and fine-tuning the model (Mazeika et al., 2024). Instead, passive defense aims to build auxiliary modules or use external safety methods including input and output filtering (Alon and Kamfonas, 2023), input smoothing, sani- tation and modification (Cao et al., 2023; Jain et al., 2023; Zhou et al., 2024a). 2.3 Model Editing as a Jailbreak Defense Several studies have explored LLMs model edit- ing as a defense mechanism to precisely mod- ify toxic regions (Wang et al., 2024; Zhao et al., 2024).DINM(Wang et al., 2024) andLED(Zhao et al., 2024) are motivated by indirect model edit- ing method that fine-tuning the toxic layer using specific objectives. The difference between these two methods is the way of locating the toxic re- gion. The layer-level localization and fine-tuning approaches lack precision in identifying harmful words while potentially compromising the model’s general performance. In contrast, we propose to adapt direct-edit as a jailbreak defense in LLMs. 3 Methods The idea behindDELMANis to mitigate a model’s harmful behavior by directly modifying the weights of specific layers, establishing a direct association between harmful tokens and safe responses. Fac- tual knowledge is stored in the MLP of specific layerl(Meng et al., 2022a). The MLP acts as two- layer key–value memories where the neurons of the first layerW l gate generate a keyk, with which the W l down retrieves an associated valuev. The MLP layer can be expressed as: k=σ(W l gate γ(a l +h l−1 )),v=W l down k,(1) wherea l is the attention output at layerl,h l−1 is the hidden state of previous layerl−1,σis the activation function andγis the layernorm.DEL- MANaims to editW l down to rebuild the connec- tion between harmful-token-related key represen- tationk ∗ and safe-response-related representation v ∗ . As illustrated in Figure 2,DELMANachieves this through five key steps. In the following of this section, we first outline the process of identifying k ∗ through harmful token extraction and random sequence generation. Then, we describe how to es- timate thev ∗ to establish its connection tok ∗ that can generate safe responses. Last, we explain how to update theW l ∗ down , the MLP of specific layer l ∗ (directly adopted from MEMIT (Meng et al., 2022b)) accordingly. 3.1 Identify Key Representationk ∗ To identify the harmful-token-related key represen- tationk ∗ , we first extract the harmful tokens from input queries that may trigger unsafe responses. To improve the stability of model editing on a specific harmful token, we generate multiple sequences that incorporate these tokens in varied contexts. Follow- ing that, we perform forward propagation for each sequence through the language modelfand use the internal representations at layerl ∗ as harmful- token-related key representationk ∗ . Harmful tokens extraction.We automate this pro- cess using GPT-4 as a token extraction assistant, which analyzes each query to pinpoint tokens likely to trigger harmful outputs. Formally, for each query in a set of harmful queriesq∈Q harm , we extract a harmful token or phraset, forming a set of consec- utive harmful tokensT h =t 1 ,t 2 ,...,t n , which can be defined as:T h = Extraction(Q harm ). The Extraction()is a carefully designedGPT-4prompt (see Appendix C.1) that includes instructions to avoid generating any harmful content and to focus solely on the task of token extraction. Random sequence generation.To enhance the accuracy of extracting the key vectork ∗ for the harmful tokens, we generate multiple sequences that incorporate these tokens. Formally, for each harmful tokent∈T h , we utilizeGPT-4to gen- erate distinct sequencesx j N j=1 , whereN= 5. These sequences are then used in the subsequent step to computek ∗ . The prompt can be found in Appendix C.2. Calculatek ∗ of harmful tokens.We perform for- ward propagation through the language modelf and average the internal representations at layerl ∗ overNgenerated sequencesx j to represent thek ∗ of harmful tokent, which can be expressed as k ∗ = 1 N N X j=1 σ W l ∗ gate γ(a l ∗ x j ,t +h l ∗ −1 x j ,t ) ,(2) wherea l ∗ x j ,t andh l ∗ −1 x j ,t are the attention score and hidden score of the harmful tokentin sequencex j at layerl ∗ and previous layerl ∗ −1respectively. Aggregating key vectors over multiple sequences ensures thatk ∗ encodes robust, context-insensitive representations of harmful semantics. 3.2 Estimatev ∗ of Safe ResponseY target To establish the connection tok ∗ that determines the model’s likelihood of generating safe response, we optimizev ∗ with the following loss function: L saf e =−logP f(m l ∗ i :=v) [Y target q],(3) wherem l ∗ i refers to the MLP output activation at layerl ∗ and positioni, andf(m l ∗ i :=v) indicates the modelfwith the specified activation replaced by vectorv, andqrepresents the harmful query in Q harm introduced in Section 3.1. To prevent unintended triggers of the safe re- sponse in ordinary contexts where the harmful to- ken might appear benignly, we want the updated model to remain consistent with its original distri- bution when asked a benign query, thus avoiding the over-activation of the safe response in normal conversation. We use KL-divergence to achieve this, which can be formulated as: L utility =KL P f(m l ∗ i :=v) ·|q u P f ·|q u ,(4) whereq u is a neutral prompt of the form“What is harmful token ?”. The optimization can be formulated as the following joint objective forv ∗ : v ∗ = arg min v [L saf e +λL utility ]. (5) Solving Eq.5 yields the final value vectorv ∗ , which can ensure that occurrences of the harmful token result in the safe response. 3.3 Weight Update ofW l ∗ down After obtaining the pair k ∗ ,v ∗ , we incorporate this new key-value association into the MLP at layerl ∗ by editing the matrixW l ∗ down via solving the least-squares problem (Belinkov and Glass, 2019): min \ W l ∗ down \ W l ∗ down K D −V D 2 (6) subject to \ W l ∗ down k ∗ =v ∗ .(7) Here,K D = [k ∗ 1 , k ∗ 2 , ...]is a matrix of key vec- tors, andV D = [v ∗ 1 , v ∗ 2 , ...]is the matrix of their corresponding value vectors. Eq.6 can be solved with this closed form solution: \ W l ∗ down =W l ∗ down +R D K D T C l ∗ +K D K D T ) −1 ,(8) whereC l ∗ =K T denotes the covariance matrix ofK, which is the key of original knowledge pair KandVat layerl ∗ , pre-cached from Wikipedia dataset. The termR D is defined as R D =V D −W l ∗ down K D ,(9) which measures the residual error between the de- sired valuesV D and the model’s current outputs W l ∗ down K D at target layerl ∗ . Practical scheme.In practice, instead of updat- ing a single layerl ∗ , we spread the updates over a range of crucial layersR=l 1 ,l 2 ,...,Lto limit the magnitude of parameter changes in a single layer, which results for better robustness (Zhu et al., 2020). We directly adopt the findings of crucial lay- ers in MEMIT (Meng et al., 2022b) and EasyEdit (Wang et al., 2023) for LLMs (see Appendix B.2.2). Thev ∗ and the residual in Eq.10 is only estimated for the last crucial layerL. This residual is then distributed to the lower layer with a factorL−l+1, which can be expressed as: R D = V D −W L down K D L−l+ 1 .(10) By ensuring smaller changes in lower layers, DELMANcan promote stability and avoid abrupt changes in a single layer. A detailed description of the algorithm is provided in Appendix A. 4 Experiments We begin this section by detailing the configuration of our experiments, including evaluated datasets, jailbreak attacks, and models, along with compared baselines and evaluation metrics. Then, we present the effectiveness ofDELMANin terms of defense performance and utility preservation. Next, we demonstrate the impact of single-behavior edit of DELMAN, highlighting its transferability across datasets and harmful behaviors. Last, we use a consecutive edit case study to illustrate that each edit, once applied, does not interfere with the edit established in previous phases. 4.1 Experiment Setup Datasets.To ensure a comprehensive evaluation of defense effectiveness against jailbreak attacks, we use theHARMBENCH(Mazeika et al., 2024) dataset for editing and evaluate across multiple testing benchmarks:HARMBENCH(HB),AD- VBENCH(AB) (Zou et al., 2023),JAILBREAK- BENCH(JBB) (Chao et al., 2024), andMALI- CIOUSINSTRUCT(MI) (Huang et al., 2023). To comprehensively assess potential side effects of model editing on LLMs’ general utility, we evalu- ateDELMANusingMT-bench(Zheng et al., 2023) and seven downstream tasks:Closed-domain QA, Dialogue,Named entity recognition (NER),Nat- ural language inference (NLI),Reasoning,Senti- ment analysisandSummarization. The detail of the datasets and their evaluation metrics are presented in the Appendix B.3. Evaluated jailbreak attacks and models.We use three leading jailbreak attack methods to demonstrate the defense performance ofDEL- MAN: two optimization based attackGCG(Zou et al., 2023),AutoDAN(Liu et al., 2023) that search for adversarial suffix, and prompt-based attackPAIRthat rewrite the prompt to adver- sarial form (Chao et al., 2023).Our evalua- tion coversLlama-2-7b-chat-hf(Touvron et al., 2023),vicuna-7b-v1.5(Zheng et al., 2023), Mistral-7B-Instruct-v0.2(Jiang et al., 2024), Llama-3.1-8B-Instruct(Grattafiori et al., 2024), andQwen2.5-7B-Instruct(Yang et al., 2024). A detailed attack setup description and results for ad- ditional models can be found in Appendix B.1 and D.2, respectively. Baselines and evaluation metrics.We con- sider three different defense methods as baselines, SafeDecoding(Xu et al., 2024a) an decoder modi- fication method, Safety fine-tuning withLoRA(Hu et al., 2021), as well asLED(Zhao et al., 2024), an indirect editing method. For all baseline methods, we follow their original papers’ suggested hyper- parameter settings. A detailed description of base- line setup is provided in Appendix B.2. We em- ployHARMBENCHclassifier (Mazeika et al., 2024) to detect the harmful content in model responses. The primary evaluation metric is the Attack Suc- cess Rate (ASR), which measures the proportion of successful attacks over all tested examples. For a datasetQ harm containing harmful queriesq, ASR is formally defined as: ASR(Q harm ) = 1 |Q harm | X q∈Q harm I(f(q))(11) whereIis the indicator function that returns 1 for successful attacks and 0 otherwise. 4.2 Effectiveness ofDELMAN Safety evaluation.Figure 3 comparesDELMAN with baselines and theOriginal Modelunder three jailbreak attacks across four datasets.DELMAN edits the model according toHARMBENCH(HB) data, and evaluates the edited model performance on AB, JBB and MI, showing its generalization ability on unseen datasets. The exact value of re- duced ASR is relegated to Appendix D.1. We ob- serve several key findings. First, compared to the original model,DELMANsignificantly reduces the ASR across all datasets (HB, AB, JBB, and MI) and against different attack types, including opti- mized suffix attacks (GCG,AutoDAN) and prompt- rewriting attacks (PAIR), and in many casesDEL- MANis able to completely mitigate jailbreak at- tacks, reducing ASR to 0. Second, among base- lines,LEDalso demonstrates some defensive capa- bility, even surpassingDELMANin certain scenar- ios within HB. However,LEDstruggles on unseen datasets, indicating a lack of generalization. In contrast,LoRAandSafeDecodingperform worse, failing to bring ASR down to an acceptable level. Last, sinceLlama2andLlama3.1already exhibit strong safety alignment,PAIRhas little effect on it. As a result, the improvements fromDELMANin this case are less pronounced. Figure 3: ASR across four datasets (HB, AB, JBB, and MI) for LLMs under three attack methods:GCG,AutoDAN, andPAIR. Each bar group compares five defense strategies —Original Model,LoRA,SafeDecoding,LED, andDELMAN. Utility evaluation.We summarize the perfor- mance ofDELMANand baselines on general- purpose tasks with several LLMs onMT-Bench, along with seven downstream tasks to compre- hensively evaluate the model’s utility in Table 1. The highest utility scores are highlighted in bold (exceptLoRAwhich has the highest ASR), and scores that exceed those of theOriginal Model are marked with (↑). OnVicuna-7B,DELMAN not only slightly outperforms theOriginal Model onMT-Bench(6.84 vs 6.77), but also maintains strong performance acrossClosed-domain QA,NLI, andSentiment analysis. WithLlama2-7B,DEL- MANdemonstrates clear improvements onNER (0.228 vs 0.187) andNLI(0.612 vs 0.603), and sustains high scores on other downstream tasks. ForMistral-7B, althoughDELMANis slightly behind theOriginal Modelon some metrics, it remains considerably more stable thanSafeDe- codingandLED, which both exhibit significant performance drops. OnLlama3.1-8B,DELMAN surpasses theOriginal ModelinDialogue,NLI, Reasoning, andSummarization. ForQwen2.5-7B, DELMANachieves the highestMT-Benchscore among defenses (8.48), closely matching or exceed- ing theOriginal Modelin multiple downstream tasks. By contrast, other defense methods such asLEDandSafeDecodingmay excel in specific tasks but tend to cause substantial overall utility degradation, particularly in comprehensive multi- turn evaluations likeMT-Bench. In summary,DEL- MANprovides robust defense with minimal util- ity loss, and in many cases, even improves down- stream task performance across various models. Figures 4 present a detailed breakdown of model performance acrossMT-Benchsubcategories. The visualization particularly highlightsDELMAN’s ad- vantages in preserving complex capabilities, with the largest area marked in dark blue. Notably, DELMANmaintains strong performance in Rea- soning, Writing, and Roleplay tasks, whereLED andSafeDecodingexhibit substantial weaknesses. This demonstratesDELMAN’s ability to balance ro- bustness against jailbreak attacks while minimizing degradation in general utility. 4.3 Edit According to Harmful Behavior In this section, we investigate the effect ofDEL- MANedit on individual harmful behavior and its impact on defending other unedited behavior. Effectiveness ofDELMANon each harmful be- havior.Figure 5 compares the performance of DELMANacross individualHARMBENCHbehav- ior, including chemical and biological (CheBio), cybercrime intrusion (CybIn), harassment and bul- lying (HaraBull), general harmful (GenHarm), il- legal (Ill), and misinformation (MisInfo). The two figures demonstrate the ASR drop onGCGand AutoDANafterDELMANedits respectively. In single-behavior editing,DELMANdemonstrates ModelDefenseMT-Bench Downstream Tasks Closed- domain QA DialogueNERNLIReasoning Sentiment analysis Summarization Vicuna-7B Original Model(82.1%) 6.770.7770.4830.2870.5630.9820.8620.272 LoRA(23.2%)5.640.7420.4590.1770.6100.9760.8980.268 SafeDecoding(10.7%)6.610.6710.3140.0980.5360.9690.6450.174 LED(8.8%)3.700.7600.4780.2650.5580.9740.8310.267 DELMAN(6.7%)6.84(↑)0.7620.4700.2540.5600.9810.8540.260 Llama2-7B Original Model(23.2%)6.890.7340.4650.1870.6030.9770.9090.267 LoRA(8.6%)6.900.7690.4800.2880.5510.9760.8540.259 SafeDecoding(1.2%)6.170.6880.3270.0990.5180.9760.8720.227 LED(2.6%)5.800.7050.4250.228 (↑)0.5770.9730.8980.256 DELMAN(0.1%)6.310.7180.4620.228(↑)0.612(↑)0.9740.9050.251 Mistral-7B Original Model(86.3%)7.930.8520.6640.4980.6940.9410.9620.255 LoRA(43.3%)7.540.8500.6680.4950.7000.9460.9530.258 SafeDecoding(17.6%)7.160.7320.4390.3720.5930.8630.7900.204 LED(15.5%) 7.090.8030.6620.4310.6790.9370.9180.221 DELMAN(8.9%)7.350.8140.6140.4440.6690.955(↑)0.9230.205 Llama3.1-8B Original Model(31.4%) 7.790.7700.7370.4240.6740.9830.9220.254 LoRA(27.4%)8.140.7820.7370.4330.6720.9820.9180.254 SafeDecoding(0.7%)8.05(↑)0.7460.3400.2230.5720.9690.6170.136 LED(2.3%)7.290.7680.6840.468(↑)0.5970.9810.8630.251 DELMAN(0.2%)7.440.7520.744(↑)0.4230.686(↑)0.985(↑)0.8900.256(↑) Qwen2.5-7B Original Model(63.4%)8.820.8400.7900.5100.8350.9870.9280.254 LoRA(48.5%)8.390.8400.7900.5070.8330.9880.9250.258 SafeDecoding(52.2%)8.320.8380.802 (↑)0.5090.8310.9870.8700.239 LED(1.0%)7.870.8190.808(↑)0.4990.8070.9870.9030.254 DELMAN(1.1%)8.480.8320.798 (↑)0.5100.8330.988(↑)0.9100.256(↑) Table 1: Utility evaluation ofDELMANand baselines on different models, with the average ASR of each method is shown in parentheses.Bold: best score (excludingLoRA); (↑): improvement overOriginal Model. significant effectiveness in defending against two types of jailbreak attacks. Cross-behavior observations.We further study the cross-behavior defense performance ofDEL- MANwith heatmap. We perform single-behavior edits on each behavior withDELMAN, and test the resulting model on all six categories, present- ing a6×6ASR heatmap. Figure 6 presents the results forLlama2-7Bunder theGCGandAuto- DANjailbreak attacks. Notably, single-category edits in many cases show resilience to off-category attacks. For instance, focusing on CheBio class editing can also mitigate malicious queries from GenHarm or MisInfo classes, reducing ASR even for these distinct domains. 4.4 Understanding theDELMAN Transferability Across Datasets and Behaviors DELMANestablishes a direct link between harmful tokens and specific responses to modify the model parameters effectively. To explain why modifying the model based on one set of harmful tokens from a specific harmful behavior also improves its robust- ness against different harmful behavior, and why edits made using examples from one dataset gener- alize to other datasets, we analyze the distribution of harmful token keyskin the target model layerl ∗ using Principal Component Analysis (PCA) (Wold et al., 1987). As shown in Figure 7, each cluster rep- resents thekof harmful token from a behavior (Fig- ure 7a) or from a dataset (Figure 7b). We can note that harmful token keyskin the target model layer l ∗ from different categories or datasets exhibit sub- stantial overlap in the embedding space, suggesting that instructions carrying malicious intent share similar representations across seemingly distinct harm classes or datasets. Through focused editing of these common token representations,DELMAN effectively reduces various types of harmful out- puts, including those from categories or datasets not seen during editing. 4.5 Consecutive Edits withDELMAN In real-world deployment, adversarial parties may repeatedly attempt to jailbreak the model, making it crucial for dynamic and consecutive edits to main- tain the effects of earlier modifications without in- terference. To evaluate the robustness ofDELMAN under consecutive edits, we conduct an experiment where edits are applied sequentially across differ- ent harmful behavior categories. Specifically, we select one category each from the HB, AB, JBB, and MI datasets and performDELMANedits in Figure 4: Comparison ofMT-Benchsub-scores across eight skill dimensions between different defense methods onVicuna-7B (left) andLlama2-7B(right). Figure 5: ASR forVicuna-7Bafter applying single-behavior DELMANagainstGCGandAutoDANattacks. succession. After each edit, we evaluate: • ASR on the current edit categoryto measure the immediate effectiveness ofDELMAN. •ASR on previously edited categoriesto de- termine whether earlier modifications remain effective. • ASR on the full datasetto assess the overall robustness ofDELMANagainst diverse jail- break attacks. We used line charts to represent the overall ASR reduction across four successive edit phases for each edited behavior of HB dataset and the ASR of the entire HB dataset. As observed in Figure 8, the overall ASR for the HB dataset consistently decreases with each edit, indicating thatDELMAN effectively reduces harmful behaviors across multi- ple categories and each edit achieves maximal ASR drop in its targeted behavior. Additionally, each category edited during the successive phases main- tains its defense effectiveness, with no increase of ASR in subsequent edits. This demonstrates that each edit, once applied, is preserved and does not interfere with the defense applied in previous phases, ensuring continuous and cumulative reduc- tion in ASR across the dataset. 4.6 Efficiency Evaluation We evaluate the efficiency, considering both train- ing time and inference overhead. All results are Figure 6: ASR heatmaps for the cross-behavior transfer results of single-behaviorDELMANedit onLlama2-7BagainstGCG (up) andAutoDAN(down) attacks. evaluated onVicuna-7Bwith a single A40 GPU, averaged over 5 runs. Table 2 presents the com- parison results across different defense methods. DELMANachieves optimal performance across all metrics: it requires only 0.4 hours for training (the fastest among all methods), maintains 1× inference overhead unlikeSafeDecoding(1.07×), and deliv- ers the lowest ASR (6.7%), demonstrating superior efficiency and effectiveness for practical deploy- ment. 4.7 Ablation Study Alternative token identification sources.To assess the robustness and cost-effectiveness of our approach, we evaluate three different sources (a) Thekof harmful tokens across behaviors. (b) Thekof harmful tokens across datasets. Figure 7: PCA visualizations ofkat the target layerLof Llama2-7Bacross different behaviors and datasets. Figure 8: Defense performance of consecutiveDELMANedits onLlama2-7BagainstGCGattacks. for harmful token identification:GPT-4(ours), Llama2-7B, andVicuna-7B. The results, compar- ing defense performance and cost across all three sources and averaged over 5 runs, are presented in Table 3.GPT-4achieves the lowest ASR (0.25%) and highestMT-Benchscore (6.31), demonstrating that accurate harmful token identification preserves model utility while maintaining strong defense per- formance. Alternative models offer comparable processing times (8.6-9.5 minutes) but with slightly degraded performance.GPT-4provides the optimal balance of effectiveness and affordability at only $1.46 token cost, though other LLMs serve as vi- able alternatives for different deployment scenarios. Alternative regularization metrics.We con- MethodTraining TimeInference OverheadAverage ASR LoRA1.5 hr1×23.2% SafeDecoding0.6 hr1.07×10.7% LED6.2 hr1×8.8% DELMAN0.4 hr1×6.7% Table 2: Efficiency comparison of defense methods on Vicuna-7B,Bold: best performance. TokenToken OverlapLlama2-GCGMT-BenchTime CostToken Sourcewith GPT-4 (%)ASR (Avg %)Score(min)Cost GPT-4 (ours)100%0.25%6.319.5$1.46 Llama2-7B54.5%1.75%5.398.6- Vicuna-7B58.0%1.5%5.458.8- Table 3: Comparison of different token identification sources, Bold: best performance. Defense GCGAutoDANPAIR MT-Bench HBABJBBMIHBABJBBMIHBABJBBMI Vicuna-7B DELMAN_JS12%4%19%1%6%3%7%6%11%6%8%5%6.63 DELMAN_COS11%2%17%3%5%2%6%5%10%5%11%5%6.44 DELMAN_KL11%2%17%1%4%2%8%5%10%5%11%5%6.84 Llama2-7B DELMAN_JS1%0%0%1%0%0%0%0%0%0%0%0%6.33 DELMAN_COS1%1%0%1%0%0%0%1%0%0%0%0%6.30 DELMAN_KL0%0%0%1%0%0%0%0%0%0%0%0%6.31 Table 4: Ablation study on regularization metrics. ASR (%) across different attacks and datasets.Bold: lowest ASR. duct an ablation study examining different regu- larization metrics: Jensen-Shannon divergence (JS) (Lin, 2002), Cosine similarity (COS) (Salton et al., 1975), and KL-divergence (ours) (Kullback and Leibler, 1951). The results are presented in Ta- ble 4. KL-divergence consistently achieves the best balance between defense effectiveness and utility preservation. ForLlama2-7B, it delivers the low- est ASR while maintainingMT-Benchperformance (6.31), and forVicuna-7B, it achieves the highest MT-Benchscore (6.84) with strong defense. JS di- vergence preserves utility but shows inconsistent defense, while Cosine similarity provides compara- ble defense but lower utility scores. 5 Conclusion In this work, we introduceDELMAN, a novel de- fense mechanism that directly edits model param- eters to neutralize harmful behaviors by forming explicit connections.DELMANbrings minimal parameter modification, preserving the utility on normal tasks and is capable of dynamic and con- secutive edits. Extensive experiments demonstrate superiority over existing baselines in terms of de- fense performance and utility preservation, as well as strong transferability. Overall,DELMANdemon- strates how token-level editing method can effec- tively enhance model safety while maintaining per- formance. In the future, it would be interesting to investigate more efficient methods for harmful to- ken identification, for instane, using a minimal set of tokens (e.g., 20-30 Tokens) to effectively cover the majority of harmful scenarios, which would significantly reduce computational costs. Addi- tionally, exploring the application ofDELMANto domain-specific LLMs and VLMs would validate its generalizability across different domains and modalities. Acknowledgements We thank all reviewers for their constructive com- ments. This work is supported by the Shanghai Engineering Research Center of Intelligent Vision and Imaging and the Open Research Fund of The State Key Laboratory of Blockchain and Data Se- curity, Zhejiang University. Limitations The limitations of our study are as follows: 1. Our evaluations are currently restricted to general-purpose LLMs, leaving the applicability to domain-specialized models (e.g., medical or legal LLMs) and larger-scale models (e.g., 70B parame- ters) unexplored. Further investigation is required to assess its defense capabilities against domain- specific jailbreak attacks and potential impacts on domain expertise after editing. 2.DELMANrelies on GPT-4 for harmful to- ken extraction and context generation, which intro- duces dependency on external models and potential cost barriers. 3. The stability of consecutive edits, though preliminarily validated, needs deeper analysis to assess potential performance drift over extended deployment. Ethics Statement DELMANdirectly edits parameters linked to harm- ful tokens, raising concerns about potential mis- application or unintended bias introduction. We advocate for responsible deployment where prac- titioners thoroughly validate parameter modifica- tions and strictly limit edits to well-defined harmful content categories. While our approach offers fine- grained, post-deployment protection, it should be viewed as one component within a comprehensive safety framework that includes human oversight and established moderation systems to ensure ethi- cal and harm-free interactions. References Gabriel Alon and Michael Kamfonas. 2023. Detect- ing language model attacks with perplexity.arXiv preprint arXiv:2308.14132. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023.Defending against alignment-breaking at- tacks via robustly aligned llm.arXiv preprint arXiv:2309.14348. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jail- breakbench: An open robustness benchmark for jail- breaking large language models.arXiv preprint arXiv:2404.01318. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419. Lin Chin-Yew. 2004. Rouge: A package for automatic evaluation of summaries. InProceedings of the Work- shop on Text Summarization Branches Out, 2004. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: A dataset for multi-turn dia- logue reasoning.arXiv preprint arXiv:2004.04494. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. InMachine learning challenges workshop, pages 177–190. Springer. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Edit- ing factual knowledge in language models.arXiv preprint arXiv:2104.08164. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vo- cabulary space.arXiv preprint arXiv:2203.14680. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key- value memories.arXiv preprint arXiv:2012.14913. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization.arXiv preprint arXiv:1911.12237. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of mod- els.arXiv preprint arXiv:2407.21783. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models.arXiv preprint arXiv:2106.09685. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. Catastrophic jailbreak of open-source llms via exploiting generation.arXiv preprint arXiv:2310.06987. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for ad- versarial attacks against aligned language models. arXiv preprint arXiv:2309.00614. AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2024. Mistral 7b. arxiv 2023.arXiv preprint arXiv:2310.06825. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency.The annals of mathe- matical statistics, 22(1):79–86. Kyungjae Lee, Wookje Han, Seung-won Hwang, Hwaran Lee, Joonsuk Park, and Sang-Woo Lee. 2022. Plug-and-play adaptation for continuously-updated qa.arXiv preprint arXiv:2204.12785. Jianhua Lin. 2002. Divergence measures based on the shannon entropy.IEEE Transactions on Information theory, 37(1):145–151. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXiv preprint arXiv:2310.04451. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dia- logue systems.arXiv preprint arXiv:1506.08909. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual as- sociations in gpt.Advances in Neural Information Processing Systems, 35:17359–17372. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass- editing memory in a transformer.arXiv preprint arXiv:2210.07229. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale.arXiv preprint arXiv:2110.11309. Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing.Com- munications of the ACM, 18(11):613–620. Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition.arXiv preprint cs/0306050. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631–1642. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024. Detox- ifying large language models via knowledge editing. arXiv preprint arXiv:2403.14472. Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, et al. 2023. Easyedit: An easy-to-use knowledge editing frame- work for large language models.arXiv preprint arXiv:2308.07269. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage models with self-generated instructions.arXiv preprint arXiv:2212.10560. Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis.Chemometrics and intelligent laboratory systems, 2(1-3):37–52. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024a. Safedecoding: Defending against jailbreak attacks via safety-aware decoding.arXiv preprint arXiv:2402.08983. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024b. A comprehensive study of jailbreak attack versus defense for large language models. In Findings of the Association for Computational Lin- guistics ACL 2024, pages 7432–7449. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jian- hong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tian- hao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115. Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024. Defending large language models against jailbreak attacks via layer-specific editing.arXiv preprint arXiv:2405.18166. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623. Andy Zhou, Bo Li, and Haohan Wang. 2024a. Ro- bust prompt optimization for defending language models against jailbreaking attacks.arXiv preprint arXiv:2401.17263. Yukai Zhou, Zhijie Huang, Feiyang Lu, Zhan Qin, and Wenjie Wang. 2024b.Don’t say no: Jail- breaking llm by suppressing refusal.arXiv preprint arXiv:2404.16369. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Univer- sal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043. A Algorithm Algorithm 1 demonstrates the detailed procedure ofDELMAN. B Detail Experiment Setting B.1 Attack Setup ForGCG(Zou et al., 2023),AutoDAN(Liu et al., 2023), andPAIR(Chao et al., 2023), we lever- age multiple harmful query datasets: the com- pleteHARMBENCH(200 queries), a subset ofAD- VBENCH(100 queries), the completeJAILBREAK- BENCH(100 queries), and the completeMALI- CIOUSINSTRUCT(100 queries). For each model under test, we generate specific attack prompts using every query from these datasets, following the original hyperparameter configurations as de- scribed in their respective papers. B.2 Baseline Setup B.2.1 Original Model Recent work (Inan et al., 2023; Xu et al., 2024b) has demonstrated that system prompts and prompt formats can significantly influence the effectiveness of jailbreaking attacks. For experimental consis- tency and reproducibility, we utilize the default settings across all evaluations. Llama2 template. [INST] «SYS» You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don’t share false information. «/SYS» instruction [/INST] Vicuna template. A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human’s questions. USER: instruction ASSISTANT: Mistral template. [INST] instruction [/INST] Llama3.1 template. <|begin_of_text|><|start_header_id|>user <|end_header_id|> instruction<|eot_id|> <|start_header_id|>assistant<|end_header_id|> Qwen2.5 template. <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user instruction<|im_end|> <|im_start|>assistant B.2.2 DELMAN The editing dataset consists of 200 queries from HARMBENCH, along with a single uniformY target : “I’m sorry, I can’t assist with that.”. Our method does not require a dedicated safe response for every single question; Instead, just this oneY target is suf- ficient to provide robust defenses with minimal im- pact on the model’s utility. And we directly adopt the findings of crucial layers in MEMIT (Meng et al., 2022b) and EasyEdit (Wang et al., 2023) for these models. The hyperparameters ofDELMAN are shown in Table 5. HyperparameterLlama2VicunaMistralLlama3.1Qwen2.5 Target LayersR[7,8][7,8][7,8][4,5,6,7,8][7,8] Learning Rate ofv ∗ 5e-15e-15e-15e-15e-1 Weight Decay ofv ∗ 0.50.50.50.50.5 Gradient Steps ofv ∗ 2525252525 Loss Layer ofv ∗ 3131313127 KL Factor0.06250.06250.06250.06250.0625 Clamp Factor 0.750.750.750.750.75 Mom2 Update Weight1500015000150001500015000 OptimizerAdamAdamAdamAdamAdam Table 5:DELMANhyperparameters for different models. B.2.3 LoRA We also applyLoRAfine-tuning on the same 200 queries from theHARMBENCH; However, in this setup, each query is paired with a safe response gen- erated by GPT-4 as theY target . We have verified that theseY target achieve 0 ASR onHARMBENCH classifier. Notably, if we were to follow the same strategy as used inDELMANand adopt a single uniformY target for all queries, the model would inevitably converge to generating only that single response. This would severely limit the model’s Algorithm 1DELMAN: Dynamic Editing for LLM Jailbreak Defense Input:Original LLMf, Harmful query datasetQ harm , Target safe responseY target , Target layersRand the last target layerL, Covariance matrixC l for each layerl∈R, Number of random context sequences N, KL-divergence factorλ. Output:Edited modelf ′ 1:Initialize:T h ←∅;f ′ ←f 2:forq∈Q harm do 3:t←Extraction(q) 4:end for 5:T h =t 1 ,t 2 ,...,t n 6:fort∈T h do 7:forj= 1toNdo 8:x j,t ←GenerateSequence(t) 9:end for 10:end for 11:fort∈T h do 12:v ∗ t ←arg min v t [L saf e +λL utility ]▷Eq.5 13:end for 14:V D ←[v ∗ 1 ,v ∗ 2 ,...,v ∗ n ] 15:forl∈Rdo 16:fort∈T h do 17:forj= 1toNdo 18:k l t,j ←σ W l gate γ(a l x j ,t +h l−1 x j ,t ) ▷Eq.2 19:end for 20:k l t ← 1 N P N j=1 k l t,j ▷Eq.2 21:end for 22:K l D ←[k l 1 ,k l 2 ,...,k l n ] 23:R l D = V D −W L down K L D L−l+1 ▷Eq.10 24:f ′ ←W l down +R l D K l D T C l +K l D K l D T ) −1 ▷Eq.8 25:end for 26:returnf ′ ability to provide diverse and contextually appro- priate responses. The hyperparameters ofLoRAare shown in Table 6. HyperparameterLlama2VicunaMistralLlama3.1Qwen2.5 LoRA Alpha88888 LoRA Rank3232323232 LoRA Dropout0.050.050.050.050.05 Train Batch Size11111 Grad Accum Steps88888 Learning Rate 2e-32e-35e-55e-55e-5 OptimizerAdamWAdamWAdamWAdamWAdamW Table 6:LoRAhyperparameters for different models. B.2.4 SafeDecoding SafeDecoding (Xu et al., 2024a), a safety enhance- ment method that operates by adjusting token prob- ability distributions. This approach strengthens the model’s security through two key mechanisms: boosting the probability of safety disclaimers while reducing the likelihood of potential jailbreak se- quences. We utilized their publicly released fine- tuned versions ofLlama2andVicunamodels, and fine-tuned safer versions ofMistral,Llama3.1, andQwen2.5using the default training settings. B.2.5 LED We used the same dataset as in theLoRAsetup. SinceLED(Zhao et al., 2024) did not provide an official code implementation, we reproduced their method following the procedures described in their paper. We selected the corresponding layers for each model according to their recommendations. The hyperparameters ofLEDare shown in Table 7. HyperparameterLlama2VicunaMistralLlama3.1Qwen2.5 Edit Layers[4,5,6,[9,10,11,[2,3,4,5,6,[4,5,6,[4,5,6, 13,14,15]13,14,15]13,14,15]13,14,15]13,14,15] Target Layers[29,30,31][29,30,31][29,30,31][29,30,31][25,26,27] Learning Rate8e-55e-55e-65e-55e-5 Train Batch Size11111 Gradient Acc Steps88888 Optimizer AdamWAdamWAdamWAdamWAdamW Table 7:LEDhyperparameters for different models. B.3 Downstream Task Datasets (1)Closed-domain QAon the BoolQ (Clark et al., 2019), and the results were also measured by EM. (2)Dialogueon the MuTual (Cui et al., 2020), and the results were measured by selecting one best- matched response from four available candidates (Lowe et al., 2015). (3)Named entity recognition (NER)tested on CoNLL03 (Sang and De Meulder, 2003) with F1 Score. (4)Natural language infer- ence (NLI)on RTE (Dagan et al., 2005) measured by binary classification accuracy. (5)Reasoningca- pability evaluated on GSM8K (Cobbe et al., 2021) using solution accuracy. (6)Sentiment analysis on SST2 (Socher et al., 2013) measured by binary classification accuracy. (7)Summarizationskills evaluated on SAMSum (Gliwa et al., 2019) using ROUGE metrics (Chin-Yew, 2004). C Prompt C.1 Prompt Used for Harmful Tokens Extraction via GPT-4 You are tasked with analyzing queries to ex- tract harmful tokens. Follow these instruc- tions: 1. Do not generate harmful, inappro- priate, or speculative content; 2. Do not ful- fill or answer harmful queries; 3. Extract the most harmful tokens (keywords or phrases) that contribute to the query’s harmful nature or potential misuse; 4. Output tokens as a list:[token1, token2, ...]. Query: INSERT QUERY HERE C.2 Prompt Used for Random Sequence Generation via GPT-4 You are tasked with generating random se- quences that include the specified token. Fol- low these instructions: 1. Avoid generating harmful, inappropriate, or unsafe content; 2. Each sequence should be 15–30 words long; 3. Use the given token exactly once in each sequence. Tokens: INSERT TOKENS HERE D Supplementary Materials D.1 Effectiveness ofDELMAN Table 8 presents the exact value of reduced ASR byDELMANand baselines. D.2 Effectiveness ofDELMANon More Models Table 9 presents the exact value of reduced ASR byDELMANon additional models. D.3 Effectiveness ofDELMANon Each Harmful Behavior Figure 9 compares the performance ofDELMAN onLlama2-7Bacross individualHARMBENCHbe- havior. ModelDefense GCGAutoDANPAIR HBABJBBMIHBABJBBMIHBABJBBMI Vicuna-7B Original Model92%89%89%94%69%78%73%83%80%75%77%86% LoRA40%18%32%8%22%29%22%32%26%13%20%16% SafeDecoding7%4%3%1%17%20%18%8%16%8%15%11% LED3%6%34%5%11%9%8%10%4%5%6%5% DELMAN11%2%17%1%4%2%8%5%10%5%11%5% Llama2-7B Original Model42%39%46%45%23%19%27%30%2%1%4%0% LoRA13%2%50%32%1%0%1%0%2%0%2%0% SafeDecoding0%4%1%1%0%0%0%0%1%4%3%0% LED2%0%8%8%2%1%2%2%1%0%4%1% DELMAN0%0%0%1%0%0%0%0%0%0%0%0% Mistral-7B Original Model80%56%79%94%87%97%95%96%88%82%89%92% LoRA35%30%38%40%55%60%62%55%42%25%34%44% SafeDecoding16%12%18%14%15%22%14%24%20%18%22%16% LED20%6%10%12%18%26%12%25%19%16%10%12% DELMAN6%2%2%14%8%11%11%12%15%0%16%10% Llama3.1-8B Original Model56%47%46%69%20%31%31%46%12%4%11%4% LoRA41%34%37%42%27%33%35%47%14%4%9%6% SafeDecoding0%0%0%0%4%1%1%1%0%0%1%0% LED1%0%3%0%6%7%8%2%0%0%0%1% DELMAN0%1%2%0%0%0%0%0%0%0%0%0% Qwen2.5-7B Original Model49%46%42%80%74%80%80%96%66%39%55%54% LoRA21%38%36%72%65%82%42%90%32%10%50%44% SafeDecoding19%17%42%48%77%80%73%97%56%30%43%45% LED1%0%0%0%0%0%0%0%5%2%1%2% DELMAN0%0%0%0%0%0%1%0%3%1%5%4% Table 8: ASR (%) of three jailbreak attacks (GCG,PAIR,AutoDAN) across four datasets on different models, under different defense methods.Bold: lowest ASR. ModelDefense GCG HBABJBBMI Qwen3-8B Original Model63%38%53%90% DELMAN1%0%3%5% Llama-2-13b-chat-hf Original Model14%16%17%18% DELMAN1%0%0%0% Qwen2.5-14B-Instruct Original Model69%42%68%93% DELMAN3%0%5%3% Table 9: ASR(%) ofGCGattack on more LLMs comparing Original ModeltoDELMAN.Bold: lowest ASR. Figure 9: ASR forLlama2-7Bafter applying single-behavior editing againstGCGandAutoDANattacks. D.4 Cross-Behavior Observations Figure 10 presents the cross-category transfer re- sults forVicuna-7BunderGCGandAutoDANat- tacks. Figure 10: ASR heatmaps for cross-category transfer results of single-categoryDELMANdefense onVicuna-7Bagainst GCG(top) andAutoDAN(bottom) attacks. D.5 Effectiveness of SequentialDELMAN Table 10 presents experimental results comparing vanillaDELMANwith its sequential variants on theLlama2-7Bmodel under GCG attacks. MethodMT-Bench GCG HBABJBBMI DELMAN6.310%0%0%1% DELMAN(Sequential-Case1)6.353%0%10%0% DELMAN(Sequential-Case2)6.644%5%6%0% Table 10: ASR(%) ofGCGattack andMT-Benchscore on Llama2-7Bcomparing vanillaDELMANand 4-EditDELMAN. Bold: lowest ASR. (a) Thekof harmful and clean tokens. (b) Thevof harmful and clean tokens. Figure 11: Principal Component Analysis (PCA) visualiza- tions ofkandvat the target layerLofLlama2-7Bacross harmful and clean tokens. D.6 Results ofDELMANacross Harmful and Clean Tokens Figure 11 shows thekandvdistribution differences between harmful and clean tokens. Notably, choos- ing harmful tokens is vital for preserving model utility: while editing with clean tokens also re- duces ASR, these tokens frequently appear in be- nign queries across various contexts, leading to unnecessary modifications of the model’s normal behaviors. In contrast, harmful tokens are primarily concentrated in unsafe queries, allowing for more precise interventions. This explains why editing based on clean tokens leads to significant degrada- tion inMT-Benchscores (see Table 11) - it uninten- tionally affects the model’s processing of legitimate queries where these common tokens naturally oc- cur. In our experiment, we define clean tokens as the third-to-last word in queries. MethodMT-Bench GCG HBABJBBMI DELMAN6.310%0%0%1% DELMAN(clean-token)5.09(↓)1%1%3%1% Table 11: ASR(%) ofGCGattack andMT-Benchscore onLlama2-7Bcomparing vanillaDELMANand clean-token DELMAN.Bold: lowest ASR. E Computing Resources The experiments are carried out on 2 NVIDIA A40 GPUs with a total computation time of 1200 GPU hours.