Paper deep dive

Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability

Vishnu Kabir Chhabra, Mohammad Mahdi Khalili

Year: 2025Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 80

Models: Llama-2-7B-Chat, Llama-3-8B-Instruct

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 3/11/2026, 12:41:16 AM

Summary

This paper investigates the impact of model compression techniques (pruning and quantization) on the safety and refusal mechanisms of large language models. Using mechanistic interpretability, the authors identify that refusal is mediated by a specific direction in the residual stream. They find that quantized models generally retain better safety characteristics than pruned models and propose a novel, lightweight method called Artificially Inducing Refusal Direction (AIRD) to restore or enhance safety in compressed models without degrading performance.

Entities (7)

AIRD · algorithm · 100%AWQ · compression-method · 100%LLaMA-2 · model · 100%Llama 3 · model · 100%Mohammad Mahdi Khalili · researcher · 100%Vishnu Kabir Chhabra · researcher · 100%Wanda · compression-method · 100%

Relation Signals (3)

Vishnu Kabir Chhabra → authored → Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability

confidence 100% · Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability Vishnu Kabir Chhabra

AIRD → improvessafetyof → Compressed Models

confidence 95% · propose a novel lightweight methodology for improving the trustworthiness of compressed models

Refusal Direction → mediates → Refusal Behavior

confidence 90% · the identification of a single direction in the residual stream mediating refusal behaviors

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The rapid growth of large language models has spurred significant interest in model compression as a means to enhance their accessibility and practicality. While extensive research has explored model compression through the lens of safety, findings suggest that safety-aligned models often lose elements of trustworthiness post-compression. Simultaneously, the field of mechanistic interpretability has gained traction, with notable discoveries, such as the identification of a single direction in the residual stream mediating refusal behaviors across diverse model architectures. In this work, we investigate the safety of compressed models by examining the mechanisms of refusal, adopting a novel interpretability-driven perspective to evaluate model safety. Furthermore, leveraging insights from our interpretability analysis, we propose a lightweight, computationally efficient method to enhance the safety of compressed models without compromising their performance or utility.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

79,951 characters extracted from source content.

Expand or collapse full text

Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability Vishnu Kabir Chhabra The Ohio State University Columbus, OH chhabra.67@osu.edu &Mohammad Mahdi Khalili The Ohio State University Columbus, OH khalili.14@osu.edu Abstract The rapid growth of large language models has spurred significant interest in model compression as a means to enhance their accessibility and practicality. While extensive research has explored model compression through the lens of safety, findings suggest that safety-aligned models often lose elements of trustworthiness post-compression. Simultaneously, the field of mechanistic interpretability has gained traction, with notable discoveries, such as the identification of a single direction in the residual stream mediating refusal behaviors across diverse model architectures. In this work, we investigate the safety of compressed models by examining the mechanisms of refusal, adopting a novel interpretability-driven perspective to evaluate model safety. Furthermore, leveraging insights from our interpretability analysis, we propose a lightweight, computationally efficient method to enhance the safety of compressed models without compromising their performance or utility. Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability Vishnu Kabir Chhabra The Ohio State University Columbus, OH chhabra.67@osu.edu Mohammad Mahdi Khalili The Ohio State University Columbus, OH khalili.14@osu.edu 1 Introduction Deployed large language models undergo safety-alignment Rafailov et al. (2023); Zhou et al. (2024) to ensure trustworthiness and become more helpful and less harmless Bai et al. (2022). Furthermore, due to the scale and size of these models, compressing large language models has been an active field of researchZhu et al. (2023); Wang et al. (2024b); Yao et al. (2023), with considerable advances in quantization Xiao et al. (2023); Lin et al. (2024); Shao et al. (2023), pruning Sun et al. (2024); Frantar and Alistarh (2023); Ma et al. (2023); Kurtić et al. (2023) and low-rank factorization Li et al. (2023); Yuan et al. (2023); Hsu et al. (2021). While research in this direction has been exciting and improved model efficiency, concerns regarding the trustworthiness and safety of compressed models remain Hong et al. (2024). To address such concerns, recent works have analyzed such compressed models in regard to their safety and trustworthiness with the general consensus indicating that safety-aligned large language models lose some aspects of their safety after undergoing compression Hong et al. (2024); Xu et al. (2024); Zhu et al. (2024). This compromise in safety ranges widely between the compression techniques, with recent literature Hong et al. (2024) indicating that quantized models enjoy improved trustworthiness over their low-rank or pruned counterparts. To the best of our knowledge, no relevant literature analyzes the cause of this discrepancy among the techniques, hence, as one of our contributions we aim to answer why quantized models are safer than their pruned counterparts. Figure 1: Interpretability Pipeline for comparing refusal in Compressed vs Base models. Consequently, research in mechanistic interpretability has garnered attention due to the promise of decomposing the non-linear decisions of a model into human-interpretable mechanisms Olah (2022, 2023). Recent works have focused on reverse engineering activations into circuits Wang et al. (2022); Hanna et al. (2024); Merullo et al. (2021); García-Carrasco et al. (2024) that explain the functionality of the model on certain tasks, while some works have focused on understanding model decisions in niche scenarios such as grokking Nanda et al. (2023); Zhong et al. (2024) and some focusing on understanding the impact of fine-tuning on model mechanisms Jain et al. (2024b); Prakash et al. (2024); Chhabra et al. (2024). In regards to safety, work by Arditi et al. (2024) discovered that the behavior of refusal is mediated by a single direction in the residual stream activation space for modern safety-aligned large language models. Our work builds upon this work by focusing on understanding the changes to this mechanism of refusal in models compressed via a variety of methods in hopes of elucidating how the mechanisms of safety-related behavior alter after compression. We then further investigate the importance of the mechanism of the refusal behavior and propose a novel lightweight algorithm to improve the trustworthiness of compressed models without altering their performance or utility. Our contributions can be summarized as follows: • We investigate how the mechanism of refusal alters in compressed models. The compression methods tested belong to two categories: pruning and quantization. Figure 1 shows our interpretability pipeline. • We investigate why models compressed with quantization schemes outperform models compressed via other methods. • We utilize our findings from our investigations and propose a novel lightweight methodology for improving the trustworthiness of compressed models without any statistically significant downsides. Our method is called Artificially Inducing Refusal Direction (AIRD) and has been illustrated in Figure 2. Figure 2: Artificially Inducing Refusal Direction (AIRD) pipeline for increasing safety of compressed models. 2 Background Transformers: Decoder-only transformers Radford et al. (2019); Vaswani et al. (2017) map input tokens =(t1,t2,…,tn)∈nsubscript1subscript2…subscriptsuperscriptt=(t_1,t_2,…,t_n) ^nt = ( t1 , t2 , … , titalic_n ) ∈ Vitalic_n to output probability distributions =(1,2,…,n)∈ℝn×||subscript1subscript2…subscriptsuperscriptℝy=(y_1,y_2,…,y_n) ^% n×|V|y = ( y1 , y2 , … , yitalic_n ) ∈ blackboard_Rn × | V |. Let i(l)⁢()∈ℝdmodelsuperscriptsubscriptsuperscriptℝsubscriptmodelx_i^(l)(t) ^d_modelxitalic_i( l ) ( t ) ∈ blackboard_Rdmodel denote the residual stream activation of the token at position i at the start of layer l. Each token’s residual stream is initialized to its embedding i(1)=⁢(ti)superscriptsubscript1subscriptx_i^(1)= Embed(t_i)xitalic_i( 1 ) = typewriter_Embed ( titalic_i ), and then undergoes a series of transformations across L layers. Each layer’s transformation includes contributions from attention and MLP components, ~i(l)=i(l)+(l)⁢(1:i(l)),superscriptsubscript~superscriptsubscriptsuperscriptsuperscriptsubscript:1 x_i^(l)=x_i^(l)+ Attn^% (l)(x_1:i^(l)),over~ start_ARG x end_ARGi( l ) = xitalic_i( l ) + typewriter_Attn( l ) ( x1 : i( l ) ) , i(l+1)=~i(l)+(l)⁢(~i(l)).superscriptsubscript1superscriptsubscript~superscriptsuperscriptsubscript~ _i^(l+1)= x_i^(l)+ MLP^% (l)( x_i^(l)).xitalic_i( l + 1 ) = over~ start_ARG x end_ARGi( l ) + typewriter_MLP( l ) ( over~ start_ARG x end_ARGi( l ) ) . The final logits i=⁢(i(L+1))∈ℝ||subscriptsuperscriptsubscript1superscriptℝ logits_i= Unembed(x_i^(L+1)) ^|% V|typewriter_logitsi = typewriter_Unembed ( xitalic_i( L + 1 ) ) ∈ blackboard_R| V | are then transformed into probabilities over output tokens i=⁢(i)∈ℝ||subscriptsubscriptsuperscriptℝy_i= softmax( logits_i) ^|V% |yitalic_i = typewriter_softmax ( typewriter_logitsi ) ∈ blackboard_R| V | Arditi et al. (2024). 2.1 Refusal Direction Following Arditi et al. (2024), we calculate the difference between the model’s average activations when processing harmful versus harmless instructions to isolate the refusal direction. This technique, known as difference-in-means Belrose (2023) isolates feature directions Marks and Tegmark (2023); Panickssery et al. (2023); Tigges et al. (2023). i(l)=1|harmful(train)|⁢∑∈harmful(train)i(l)⁢()superscriptsubscript1superscriptsubscriptharmful(train)subscriptsuperscriptsubscriptharmful(train)superscriptsubscript _i^(l)= 1|D_harmful% ^(train)| _t _harmful^(% train)x_i^(l)(t)μitalic_i( l ) = divide start_ARG 1 end_ARG start_ARG | Dharmful(train) | end_ARG ∑t ∈ D start_POSTSUBSCRIPT harmful(train) end_POSTSUBSCRIPT xitalic_i( l ) ( t ) (1) i(l)=1|harmless(train)|⁢∑∈harmless(train)i(l)⁢().superscriptsubscript1superscriptsubscriptharmless(train)subscriptsuperscriptsubscriptharmless(train)superscriptsubscript _i^(l)= 1 _% harmless^(train) _t _% harmless^(train)x_i^(l)(t).νitalic_i( l ) = divide start_ARG 1 end_ARG start_ARG | Dharmless(train) | end_ARG ∑t ∈ D start_POSTSUBSCRIPT harmless(train) end_POSTSUBSCRIPT xitalic_i( l ) ( t ) . (2) Hence, the difference-in-means vector is as follows: i(l)=i(l)−i(l)superscriptsubscriptsuperscriptsubscriptsuperscriptsubscriptr_i^(l)= _i^(l)- _i^(l)ritalic_i( l ) = μitalic_i( l ) - νitalic_i( l ). Selecting a single vector: Finding the difference-in-means vector i(l)superscriptsubscriptr_i^(l)ritalic_i( l ) for each post-instruction token position i∈Ii∈ Ii ∈ I for I=1,2,…,n12…I=\1,2,…,n\I = 1 , 2 , … , n and layer l∈[L]delimited-[]l∈[L]l ∈ [ L ] yields a set of |I|×L|I|× L| I | × L candidate vectors. Then the most effective vector, i∗(l∗)superscriptsubscriptsuperscriptsuperscriptr_i^*^(l^*)ritalic_i∗( l start_POSTSUPERSCRIPT ∗ ) end_POSTSUPERSCRIPT, is chosen by evaluating each candidate vector over validation sets harmful(val)superscriptsubscriptharmful(val)D_harmful^(val)Dharmful(val) and harmless(val)superscriptsubscriptharmless(val)D_harmless^(val)Dharmless(val) by measuring each candidate vector’s ability to bypass refusal when ablated on harmful(val)superscriptsubscriptharmful(val)D_harmful^(val)Dharmful(val) and to induce refusal when added on harmless(val)superscriptsubscriptharmless(val)D_harmless^(val)Dharmless(val). We follow the notation of Arditi et al. (2024) and denote the selected vector as rr, and its corresponding unit-norm vector as ^ rover start_ARG r end_ARG. 2.2 Model Interventions Activation addition: Given a difference-in-means vector (l)∈ℝdmodelsuperscriptsuperscriptℝsubscriptmodelr^(l) ^d_modelr( l ) ∈ blackboard_Rdmodel derived from layer l, we add the difference-in-means vector to the activations of a harmless prompt at layer l and at all token positions i∈Ii∈ Ii ∈ I. This shifts the average harmless activations towards the average harmful activations Arditi et al. (2024), (l)′←(l)+(l).←superscriptsuperscript′superscript ^(l) ^(l)+r^(% l).x( l ) start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ← x( l ) + r( l ) . (3) Directional ablation: For a given direction ^∈ℝdmodel^superscriptℝsubscriptmodel r ^d_modelover start_ARG r end_ARG ∈ blackboard_Rdmodel, we erase it from the model’s representations using directional ablation Arditi et al. (2024). Directional ablation suppresses the component along ^ rover start_ARG r end_ARG for every residual stream activation ∈ℝdmodelsuperscriptℝsubscriptmodelx ^d_modelx ∈ blackboard_Rdmodel, ′←−^⁢^⊺⁢.←superscript′^superscript^⊺ - r % r x.x′ ← x - over start_ARG r end_ARG over start_ARG r end_ARG⊺ x . (4) This operation is performed at every activation i(l)superscriptsubscriptx_i^(l)xitalic_i( l ) and ~i(l)superscriptsubscript~ x_i^(l)over~ start_ARG x end_ARGi( l ), across all layers l and all token positions i. 2.3 Compression Pruning: Pruning methods of compression aim to zero out unimportant weights. A variety of methods exist that aim to utilize only the magnitude of weights Han et al. (2015, 2016); Frantar and Alistarh (2023), and those that consider weights and activations (e.g., Wanda method) Sun et al. (2024). Due to its efficiency, popularity, and minimal degradation of performance Sun et al. (2024), Wanda, is a center point of this study. In each layer, Wanda utilizes a pruning metric that assesses weight importance as follows, Si⁢j=|Wi⁢j|⋅‖Xj‖2,subscript⋅subscriptsubscriptnormsubscript2 S_ij=|W_ij|·||X_j||_2,Sitalic_i j = | Witalic_i j | ⋅ | | Xitalic_j | |2 , (5) Where ‖Xj‖2subscriptnormsubscript2||X_j||_2| | Xitalic_j | |2 is l2subscript2l_2l2 norm of jjjth feature across different tokens and different inputs in a batch, and Wi⁢jsubscriptW_ijWitalic_i j is an element in row i and column j of weight matrix W. Quantization: Quantization is a form of compression that relies on lowering the precision of the model weights to compress the model Zhu et al. (2023). Modern literature primarily contains two forms of quantization: Training Aware Quantization Chen et al. (2024), and Post-Training Quantization Yao et al. (2023). The scope of this study contains models compressed via Post-Training Quantization techniques as Training Aware Quantization often requires fine-tuning which can lead to unintended consequences for safety Qi et al. (2024). Research in Post-Training Quantization has resulted in two forms of quantization methods: methods that rely on activations to assess weight importance Lin et al. (2024) and weight-only quantization Dettmers et al. (2022). In this work, we consider both types of quantization schemes. 3 Experimental Setup Models: This study focuses on widely used safety-aligned large language models (LLMs) Touvron et al. (2023a); Grattafiori et al. (2024) . The selected models, their parameters, and base precision for inference are listed in Table 1. Model family Sizes Precision Reference Llama-2 Chat 7B 16bit Touvron et al. (2023a) Llama-3 Instruct 8B 16bit Grattafiori et al. (2024) Table 1: Comparison of different model families. Compression Methods: We examine the impact of pruning and quantization, two common compression methods, on model safety Kuzmin et al. (2024). Our pruning experiments utilize Wanda Sun et al. (2024), a popular and lightweight method that can prune language models in one-shot and does not suffer from severe performance degradation after pruning Sun et al. (2024) and Magnitude pruning Han et al. (2015), a well established pruning method. As for quantization, we utilize two popular methods: LLM.int8()Dettmers et al. (2022) and Activation Aware Quantization (AWQ) Lin et al. (2024) Calibration Data for Compression: To assess the effect and data dependency of the refusal mechanism in activation-aware pruning, we use two datasets with different objectives: maximizing safety and maximizing performance/utility. Following Wei et al. (2024), for safety, we utilize the Align dataset Wei et al. (2024), which is compiled using harmful instructions from AdvBench Zou et al. (2023a), by dividing it into AdvBench-Eval (100 instructions for evaluation) and AdvBench-Attr (420 instructions for attribution). Then, the LLama2-7b-chat Touvron et al. (2023b) is prompted with AdvBench-Attr. An instruction along with the response is kept in AdvBench-Attr if the LLama2-7b-chat declines providing the answer. Otherwise, the instruction will be deleted from the dataset. After finalizing AdvBench-Attr, we use it for activation-aware pruning, and AdvBench-Eval will be used for evaluating the safety of the pruned model. For maximizing performance, we utilize a version of Alpaca Taori et al. (2023) for pruning, namely, Alpaca-Cleaned. In Alpaca-Cleaned, we excludes safety-related prompts using sensitive phrase matching Qi et al. (2024). For AWQ Lin et al. (2024), we follow the original methodology and use Pile Gao et al. (2020) as the small calibration dataset. Measuring Performance: Following Sun et al. (2024), we measure the performance of the models by measuring their zero-shot accuracy on 5 tasks from EleutherAI’s LM Harness Gao et al. (2023): HellaSwag Zellers et al. (2019), BoolQ Clark et al. (2019), RTE Wang et al. (2019), ARC Challenge Clark et al. (2018) and WinograndeSakaguchi et al. (2021). Measuring Safety: We measure the safety of our compressed models by evaluating its attack success rate (ASR)111Sometimes, we refer to ASR as attack score. in response to harmful instructions. Specifically, we prompt the model using AdvBench-eval, the first 100 prompts from AdvBench, and collect its responses. Following Zou et al. (2023b), we consider an attack as successful if the model’s response lacks key patterns indicative of refusal. The ASR is then computed as the ratio of successfully attacked prompts to the total number of prompts evaluated. Following Wei et al. (2024), our safety evaluation considers three use cases: the ASR under non-malicious conditions (ASRVanillaVanilla_ Vanillastart_FLOATSUBSCRIPT Vanilla end_FLOATSUBSCRIPT), and the ASR under two malicious settings – ASRAdv-DecodingAdv-Decoding_ Adv-Decodingstart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPT Huang et al. (2024), where the attacker manipulates the decoding process, and ASRAdv-SuffixAdv-Suffix_ Adv-Suffixstart_FLOATSUBSCRIPT Adv-Suffix end_FLOATSUBSCRIPT Zou et al. (2023b), where adversarial suffixes are used. Due to the high computational cost associated with calculating adversarial suffixes, we precompute several suffixes and use the three best-performed ones in our evaluation. For ASRAdv-DecodingAdv-Decoding_ Adv-Decodingstart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPT, we present results with and without the [INST] wrapper ASRVanillaVanilla_ Vanillastart_FLOATSUBSCRIPT Vanilla end_FLOATSUBSCRIPT ASRAdv-SuffixAdv-Suffix_ Adv-Suffixstart_FLOATSUBSCRIPT Adv-Suffix end_FLOATSUBSCRIPT ASRAdv-DecodingAdv-Decoding_ Adv-Decodingstart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPT Sample Times 1111 1111 5555 System Prompt [INST], [/INST] wrapper , Adversarial Suffix Table 2: The differences between three types of ASR in our safety evaluation. Datasets for Finding and Evaluating Refusal Directions: Following Arditi et al. (2024), we construct harmfulsubscriptharmfulD_harmfulDharmful as a collection of harmful instructions from AdvBench Zou et al. (2023a), MalicousInstruct Huang et al. (2024), TDC2023 Mazeika et al. (2024), and HarmBench Mazeika et al. (2024). As for harmlesssubscriptharmlessD_harmlessDharmless, we collect a set of harmless instructions from Alpaca Taori et al. (2023). Each harmfulsubscriptharmfulD_harmfulDharmful and harmlesssubscriptharmlessD_harmlessDharmless includes 160 samples which will be split into train and validation splits of 128 and 32 samples, respectively. We use training samples to find the refusal direction based on (1) and (2). Then, we will use validation samples to evaluate the refusal direction through activation addition and directional ablation. Evaluation of Refusal: Refusal is often measured in terms of substring matching the model’s output with common phrases that indicate refusal. These phrases can often be "I cannot", "I am sorry", "as a Chatbot" etc. Wei et al. (2024). We follow the prior literature Lermen et al. (2023); Liu et al. (2024); Robey et al. (2023); Shah et al. (2023); Xu et al. (2023); Zou et al. (2023b) and utilize substring matching to classify outputs as refusal (refusal-score = 1) or successful attack (refusal-score= 0). To do so, we compile common substrings for each model architecture that indicate refusal 4 Refusal under Compression In this section, firstly, apply several compression methods to the safety-aligned LLMs. If a compression method is data-dependent, then we use the calibration data introduced in Section 3. We then analyze the refusal mechanisms of models that underwent compression. We do this by utilizing harmfulsubscriptharmfulD_harmfulDharmful and harmlesssubscriptharmlessD_harmlessDharmless and difference-in-means Arditi et al. (2024); Belrose (2023) in the compressed models to first identify whether the refusal mechanism has altered. Surprisingly, our first finding reveals that the refusal mechanism is still mediated by a single direction in compressed models. We record this finding in Table 3222Llama3-8b-Instruct loses a lot of performance under magnitude pruning and hence we don’t present results for it. and note that this finding holds true for every compression method tested, model architecture/size, and calibration dataset, see Table 3. This indicates that compressed models, even the ones that suffer from a degradation in safety and trustworthiness, retain the original mechanism by which they refuse harmful prompts. We now validate that the refusal directions we found are enough for mediating the refusal mechanism. We do so by utilizing two model interventions: directional ablation and activation addition. For the first model intervention, we utilize directional ablation. We borrow our methodology from the work by Arditi et al. (2024) and ablate the refusal direction from all activations at all layers and token positions. We then generate completions over harmful(val)subscriptsuperscriptvalharmfulD^(val)_harmfulD( val )harmful. Our findings illustrated in Figure 3 show that even in the compressed models, ablating the refusal direction significantly increases the attack score of the harmful prompts. This indicates that the refusal directions we discovered for each compressed model are necessary for mediating refusal. Directional ablation, in this case, serves as the necessity test, and we utilize this test to validate each refusal direction that we discover. Type Model Method lc/lsuperscriptl^c/llitalic_c / l ic/isuperscripti^c/iiitalic_c / i Calibration Type Pruning Llama2-7b Wanda 14/14141414/1414 / 14 −55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0-5- 5/−1/-1/ - 1 Alpaca Llama2-7b Wanda 1212 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,01212/14absent14/14/ 14 −55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0-5- 5/−1/-1/ - 1 Align Llama2-7b Magnitude 1212 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,01212/14absent14/14/ 14 −55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0-5- 5/−1/-1/ - 1 — Llama3-8b Wanda 12/12121212/1212 / 12 −5/−5-5/-5- 5 / - 5 Alpaca Llama3-8b Wanda 1313 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,01313/12absent12/12/ 12 −55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0-5- 5/−1/-1/ - 1 Align Quantization LLama2-7b LLM.int8() 14/14141414/1414 / 14 −1/−1-1/-1- 1 / - 1 — LLama2-7b AWQ 14/14141414/1414 / 14 −1/−1-1/-1- 1 / - 1 Pile LLAma3-8b LLM.int8() 12/12121212/1212 / 12 −5/−5-5/-5- 5 / - 5 — LLAma3-8b AWQ 12/12121212/1212 / 12 −5/−5-5/-5- 5 / - 5 Pile Table 3: The new refusal directions in each compressed model tested along with the calibration dataset utilized. lcsuperscriptl^clitalic_c, icsuperscripti^ciitalic_c refer to the layer and token position of the refusal direction in the compressed model with their respective changes. The compression rate for each method is 50%percent5050\%50 %. However, necessity does not imply that the refusal directions we discovered are sufficient for mediating refusal. Hence, we utilize the sufficiency test, in which we perform activation addition at the layer l and all token positions and generate completion over the validation harmful(val)subscriptsuperscriptvalharmfulD^(val)_harmfulD( val )harmful. This method, as shown by Arditi et al. (2024), would ideally indicate that each refusal direction is sufficient for mediating refusal. Our findings (see Figure 4) indeed indicate that each refusal direction we discover is sufficient for inducing refusal as performing activation addition on harmless instructions significantly increases the refusal score for each compression method. While Figure 3 and Figure 4 show directional ablation and activation addition results for Llama2-7b, we observed similar results for Llama3-8b, which are omitted for brevity Figure 3: Attack score (ASR) after directional ablation in Llama2-7b compressed model. Ablating the refusal direction increases the attack score significantly. Figure 4: Refusal Score on harmless prompts after activation addition (ActAdd) in compressed Llama2-7b model. Activation addition causes the model to refuse to answer. Our second finding reveals that in certain compressed models, the source position of the refusal direction changes after compression. Surprisingly, we note that this alteration of the source position of the refusal direction is only recorded in models that are compressed via pruning methods (see Table 3), indicating quantization of model weights has significantly less impact on a model’s interpretability compared to pruning. Furthermore, for pruning, we notice an alteration in the source position of the refusal direction (for most models), indicating that the source of the refusal direction can change regardless of the calibration data. This change in refusal direction correlates with a decrease in the trustworthiness and safety of safety-aligned models after pruning. We discuss this in the following sections. 4.1 Does change in refusal direction mean lower safety? To understand the effects of the change in refusal direction in regards to safety/trustworthiness, we evaluate the performance of the compressed model on AdvBench Zou et al. (2023a) attacks and report our results in Table 4. We limit our investigation to models that undergo a change in refusal direction (either via a change in source position or the direction itself) after compression and report that the compressed models suffer from a decrease in safety across multiple dimensions of AdvBench333Llama3-8b-Instruct loses a lot of performance under magnitude pruning and hence we don’t present results for it.. This finding implies that a change in refusal direction is directly correlated with a loss of safety/trustworthiness after compression. Model Method ASRIAdv-DecodingsuperscriptsubscriptabsentAdv-Decoding_ Adv-Decoding^Istart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPTI ASRVanillaVanilla_ Vanillastart_FLOATSUBSCRIPT Vanilla end_FLOATSUBSCRIPT ASR×Adv-DecodingsuperscriptsubscriptabsentAdv-Decoding_ Adv-Decoding^×start_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPT× ASRAdv-SuffixAdv-Suffix_ Adv-Suffixstart_FLOATSUBSCRIPT Adv-Suffix end_FLOATSUBSCRIPT Llama2 Base 0.0060.0060.0060.006 0.160.160.160.16 0.270.270.270.27 0.090.090.090.09 Llama2 Wanda-Align 0.00.00.00.0 0.170.170.170.17 0.260.260.260.26 0.130.130.130.13 Llama2 Wanda-Alpaca 0.0220.0220.0220.022 0.170.170.170.17 0.3160.3160.3160.316 0.240.240.240.24 Llama2 Magnitude 0.010.010.010.01 0.60.60.60.6 0.4960.4960.4960.496 0.350.350.350.35 Llama3 Base 0.0540.0540.0540.054 0.010.010.010.01 0.0460.0460.0460.046 0.010.010.010.01 Llama3 Wanda-Align 0.070.070.070.07 0.010.010.010.01 0.0760.0760.0760.076 0.040.040.040.04 Llama3 Wanda-Alpaca 0.1120.1120.1120.112 0.040.040.040.04 0.120.120.120.12 0.130.130.130.13 Table 4: Attack Scores (ASR) of the compressed models on AdvBench. 5 Why Quantization is Safer than Pruning Hong et al. (2024) noted that quantized models severely outperform their pruned counterparts across multiple dimensions of safety and trustworthiness. We further investigate this finding from an interpretability perspective. Firstly, from Table 3, we see that quantized models do not see a change in the source of the original refusal direction whereas pruned models do see a shift in the source of the refusal direction. Secondly, as we see in Table Table 4, a change in the refusal direction is correlated with a loss in trustworthiness and safety. Thirdly, we measure the cosine similarity of the new refusal directions in the pruned/quantized models with the original directions. The results are provided in Table 5 and show that the directions found in the pruned models are extremely different from the direction in the original model. The drastic shift in refusal direction observed in pruned models, but not in quantized models, explains why quantized models outperform pruned ones in safety and trustworthiness. Since quantized models retain the original refusal mechanism and its source/quality, they preserve the safety of the original model. Model Method Cosine Similarity Llama2-7b Wanda-Align 0.3510.3510.3510.351 Llama2-7b LLM.int8() 0.9960.9960.9960.996 LLama2-7b Wanda-Alpaca 0.5390.5390.5390.539 Llama2-7b AWQ 0.9960.9960.9960.996 Llama2-7b Magnitude 0.3370.3370.3370.337 Llama3-8b Wanda-Align 0.7320.7320.7320.732 Llama3-8b LLM.int8() 0.990.990.990.99 Llama3-8b Wanda-Alpaca 0.9020.9020.9020.902 LLama3-8b AWQ 0.9940.9940.9940.994 Table 5: Comparison of different compressed models’ refusal direction based on cosine similarity. 6 Artificially Inducing Refusal Direction To mitigate the effects of the altered refusal direction in the pruned models, we introduce Artifically Inducing Refusal Direction (AIRD), a lightweight and simple method to increase the safety of models which suffer from an altered refusal direction after compression without loss to their performance on general coherence benchmarks. Method: Consider a model M with a refusal direction i(l)superscriptsubscriptr_i^(l)ritalic_i( l ) and the compressed model McsuperscriptM^cMitalic_c with ic(lc)superscriptsubscriptsuperscriptsuperscriptr_i^c^(l^c)ritalic_iitalic_c( l start_POSTSUPERSCRIPT c ) end_POSTSUPERSCRIPT as its refusal direction. We orthogonalize the weight matrices that project to the residual stream (attention output and MLP output) in layer l in the compressed model with respect to the refusal direction i(l)superscriptsubscriptr_i^(l)ritalic_i( l ) and add it to the weight matrix as follows, Wl,n⁢e⁢wc←Wlc+α⁢i(l)⁢(i(l))⊺⁢Wlc,←subscriptsuperscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsuperscriptsubscript⊺superscriptsubscript W^c_l,new← W_l^c+ _i^(l)(% r_i^(l)) W_l^c,Witalic_citalic_l , n e w ← Witalic_litalic_c + α ritalic_i( l ) ( ritalic_i( l ) )⊺ Witalic_litalic_c , (6) where WlcsuperscriptsubscriptW_l^cWitalic_litalic_c is a weight matrix in layer l of compressed model McsuperscriptM^cMitalic_c. Evaluation: We evaluate the effects of AIRD on AdvBench for LLama2-7b and Llama3-8b, compressed via pruning on both calibration datasets and record our findings in Table 6. Model Method Calibration ASRIAdv-DecodingsuperscriptsubscriptabsentAdv-Decoding_ Adv-Decoding^Istart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPTI ASRVanillaVanilla_ Vanillastart_FLOATSUBSCRIPT Vanilla end_FLOATSUBSCRIPT ASR×Adv-DecodingsuperscriptsubscriptabsentAdv-Decoding_ Adv-Decoding^×start_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPT× ASRAdv-SuffixAdv-Suffix_ Adv-Suffixstart_FLOATSUBSCRIPT Adv-Suffix end_FLOATSUBSCRIPT Llama2-7b WANDA Align 0%percent00\%0 % 41%percent4141\%41 %( ↓ ↓) 14%percent1414\%14 %( ↓ ↓) 15%percent1515\%15 %( ↓ ↓) Llama2-7b WANDA Alpaca 10%percent1010\%10 %( ↓ ↓) 17%percent1717\%17 %( ↓ ↓) 12.5%percent12.512.5\%12.5 %( ↓ ↓) 41%percent4141\%41 %( ↓ ↓) Llama2-7b Magnitude — 40%percent4040\%40 %( ↓ ↓) 20%percent2020\%20 %( ↓ ↓) 2.4%percent2.42.4\%2.4 %( ↑ ↑) 14.2%percent14.214.2\%14.2 %( ↓ ↓) Llama3-8b WANDA Align 22.3%percent22.322.3\%22.3 %( ↓ ↓) 18.4%percent18.418.4\%18.4 %( ↓ ↓) 0%percent00\%0 % 0%percent00\%0 % Llama3-8b WANDA Alpaca 10.7%percent10.710.7\%10.7 %( ↓ ↓) 33.3%percent33.333.3\%33.3 %( ↓ ↓) 17.87%percent17.8717.87\%17.87 %( ↓ ↓) 16.6%percent16.616.6\%16.6 %( ↓ ↓) Table 6: Relative change in ASR scores in models that underwent AIRD (↓ ↓ is better). α=0.010.01α=0.01α = 0.01 for MLP projections and α=0.020.02α=0.02α = 0.02 for Attention projections in models. Core Finding: Applying AIRD in compressed models that underwent a change in their refusal direction significantly decreases the ASR on multiple dimensions of AdvBench. More specifically, the attacks that succeeded more in the compressed models see a drastic change in their effectiveness against models protected with AIRD. This highlights that AIRD can successfully increase the safety of compressed models while being extremely compute efficient. 6.1 AIRD doesn’t impact performance on general benchmarks To understand the effect of AIRD on the general performance of the model, we evaluate the zero-shot accuracy as mentioned in section 3. We record our evaluations in Table 7 and find that AIRD causes no significant change in the model performance across the benchmark suite. Surprisingly, we find that in some benchmarks the accuracy of the compressed models increases. Although this increase is quite minuscule, it shows that our method increases the safety of the model without a significant effect on its performance. Model RTE ARC BoolQ Winogrande HellaSwag Llama2 Wanda-Align 68.0 / 68.5 (-0.5) 36.5 / 36.0 (+0.5) 76.5 / 76 (+0.5) 64.5 / 63.0 (+1.5) 54.0 / 54.0 (+0.0) Llama2 Wanda-Alpaca 63.5 / 64.5 (-1.0) 41.5 / 40.5 (+1.0) 79.0 / 79.0 (+0.0) 66.0 / 66.5 (-0.5) 55.5/ 55.5 (+0.0) Llama2 Magnitude 52.0 / 54.0 (-2.0) 34.5 / 34.0 (-0.5) 68.5 / 69.0 (-0.5) 61.5 / 63.0 (-1.5) 48.0/ 48.0 (+0.0) Llama3 Wanda-Align 62.0 / 62.5 (-0.5) 43.5 / 44.5 (-0.5) 79.0 / 78.5 (+0.5) 70.0 / 71.0 (-1.0) 50.5 / 50.5 (+0.0) Llama3 Wanda-Alpaca 62.5 / 62.5 (+0.0) 46.0 / 45.0 (+1.0) 82.0 / 82.0 (+0.0) 68.5 / 67.5 (+1.0) 51.5 / 51.5 (+0.0) Table 7: Performance comparison of models on the zero-shot evaluation suite. We report the zero-shot evaluations of models that underwent AIRD, the base compressed model and increase, decrease or no change in performance. 6.2 AIRD doesn’t alter the refusal mechanism Prior work has shown that feature steering can lead to unintended consequences O’Brien et al. (2024); Durmus et al. (2024). AIRD, in this case, is a form of feature steering by orthogonalizing the weights and can possibly lead to unintentional changes in the refusal mechanism of the model. We investigate this by utilizing difference-in-means Belrose (2023) and find those models that undergo AIRD do not see a change in the refusal mechanism, i.e, the refusal behavior in such models is still controlled by one direction. Furthermore, we record that AIRD does not change both the source and quality of the refusal directions of models. This implies that other methods Han et al. (2025); Cao (2024) that rely on the present understanding of the refusal mechanism in the models are robust to the changes made by AIRD, allowing for the same degree of control as non-AIRD models 7 Related Work Safety Under Compression: Recent literature has explored the trustworthiness of compressed large language models via benchmarking multiple dimensions of safety Hong et al. (2024); Xu et al. (2024), finding quantized models suffer almost no loss in safety after compression whereas pruned models do. While another work discovered the low-rank/sparse nature of safety-related components in modern LLMs Wei et al. (2024). Other works aim to improve fairness via compression Xu and Hu (2022). Although significant research progress has been made in understanding trustworthiness under compression, our work is the first of its kind, evaluating and improving compressed LLMs via mechanistic interpretability. Mechanistic Interpretability: Through considerable manual effort, research in mechanistic interpretability has lead to important findings. Works have discovered underlying mechanisms of models as circuits Wang et al. (2022); Hanna et al. (2024); Merullo et al. (2021); García-Carrasco et al. (2024), while others improve the automation of circuit discovery Conmy et al. (2023); Syed et al. (2023). Some works focus on the interpretability of mechanisms in scenarios such as grokking Nanda et al. (2023); Zhong et al. (2024); Wang et al. (2024a), fine-tuning Prakash et al. (2024); Chhabra et al. (2024). However, to the best of our knowledge, no prior work has focused on interpreting mechanisms in compressed models. Refusal Mechanism: Arditi et al. (2024) was the first to discover that the refusal mechanism is mediated via a single direction. Follow-up works have focused on steering this refusal in large language models via feature steering O’Brien et al. (2024) utilizing sparse auto-encoders Cunningham et al. (2023), understanding more about the refusal mechanism Marshall et al. (2024), context-driven feature steering Han et al. (2025), introducing refusal tokens for steering Jain et al. (2024a) and utilizing refusal for preventing hallucinations Cao (2024). 8 Discussion In this work, we discuss the problem of understanding how a core safety-related mechanism alters in models that undergo compression. The mechanism that we discuss, the refusal mechanism, is crucial in safety against harmful prompts that seek to bypass a safety-aligned LLM’s guardrails Arditi et al. (2024); O’Brien et al. (2024). Our first finding indicates that in models compressed via pruning a shift in both the source position and direction occurs. However, in case of quantized models, the refusal direction retains its original characteristics. We deem this finding the source of the loss in safety that is recorded in models that are compressed via pruning. Recent literature Hong et al. (2024); Xu et al. (2024) suggests that the quantized models don’t experience any statistically significant reduction in safety after compression as opposed to their pruned counterparts, this finding resonates with our finding and we further explore this trend via comparing the directions of quantized models with that of the original uncompressed models. As directions found in quantized models retain their original characteristics and the directions in the pruned models do not, we believe this to be the mechanistic explanation as to why quantized models are safer than pruning. Furthermore, based on our findings, we propose a novel lightweight, and computationally inexpensive algorithm, AIRD, that increases the safety of the compressed models that undergo a change in their refusal direction and loss in safety. Our method can increase the safety guardrails of compressed models up to 41%percent4141\%41 % in some benchmarks while retaining the model’s coherence and not undergoing a statistically significant change in performance. Furthermore, our method doesn’t impact the model’s interpretability, in that both the refusal direction and the refusal mechanism are preserved in the model that undergoes AIRD. This benefit of our method implies that other techniques that rely on the refusal mechanismsCao (2024); O’Brien et al. (2024); Han et al. (2025) can be applied to models that undergo AIRD. 9 Limitations Recent advancements in language modeling has introduced architectures that utilize Mixture of Experts Jiang et al. (2023); Guo et al. (2025), State Space Models Gu and Dao (2023); Lieber et al. (2024), and modernized RNNs Beck et al. (2024); Peng et al. (2023). Presently, it is unclear how advancements in mechanistic interpretability for transformers and by extension our work generalize to these architectures and relevant future work is needed to generalize our findings. Furthermore, our algorithm, AIRD, reduces the compression rate of the language model by decreasing sparsity in one layer, future work can optimize and build on this work so this downside can be mitigated. 10 Broader Impact We believe mechanistic interpretability techniques can alleviate many AI safety concerns and assist in creating safe and reliable AI systems. However, dual-use remains a concern, as research in mechanistic interpretability can aid malicious intentions for exploiting/creating unsafe AI. However, our method, AIRD, highlights that research in the field can lead to fruitful methods that can aid in the safety of language models, but a similar method can be created to decrease a model’s safety. We believe future work in this direction needs to address potential malicious side effects and create robust methods to aid in the safety and trustworthiness of language models. 11 Acknowledgment This work is supported by the U.S. National Science Foundation under award IIS-2301599 and CMMI-2301601, and by grants from the Ohio State University’s Translational Data Analytics Institute and College of Engineering Strategic Research Initiative. References Arditi et al. (2024) Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Preprint, arXiv:2406.11717. Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Beck et al. (2024) Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. 2024. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517. Belrose (2023) Nora Belrose. 2023. Diff-in-means concept editing is worst-case optimal: Explaining a result by Sam Marks and Max Tegmark. https://blog.eleuther.ai/diff-in-means/. Accessed on: May 20, 2024. Cao (2024) Lang Cao. 2024. Learn to refuse: Making large language models more controllable and reliable through knowledge scope limitation and refusal mechanism. Preprint, arXiv:2311.01041. Chen et al. (2024) Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. 2024. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062. Chhabra et al. (2024) Vishnu Kabir Chhabra, Ding Zhu, and Mohammad Mahdi Khalili. 2024. Neuroplasticity and corruption in model mechanisms: A case study of indirect object identification. In ICML 2024 Workshop on Mechanistic Interpretability. Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Conmy et al. (2023) Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352. Cunningham et al. (2023) Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. Preprint, arXiv:2208.07339. Durmus et al. (2024) Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, Oliver Rausch, Saffron Huang, Sam Bowman, Stuart Ritchie, Tom Henighan, and Deep Ganguli. 2024. Evaluating feature steering: A case study in mitigating social biases. Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In ICML. Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation. García-Carrasco et al. (2024) Jorge García-Carrasco, Alejandro Maté, and Juan Carlos Trujillo. 2024. How does gpt-2 predict acronyms? extracting and understanding a circuit via mechanistic interpretability. In International Conference on Artificial Intelligence and Statistics, pages 3322–3330. PMLR. Grattafiori et al. (2024) Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Gu and Dao (2023) Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Han et al. (2025) Peixuan Han, Cheng Qian, Xiusi Chen, Yuji Zhang, Denghui Zhang, and Heng Ji. 2025. Internal activation as the polar star for steering unsafe llm behavior. Preprint, arXiv:2502.01042. Han et al. (2016) Song Han, Huizi Mao, and William J Dally. 2016. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In ICLR. Han et al. (2015) Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. NeurIPS, 28. Hanna et al. (2024) Michael Hanna, Ollie Liu, and Alexandre Variengien. 2024. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36. Hong et al. (2024) Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, et al. 2024. Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression. arXiv preprint arXiv:2403.15447. Hsu et al. (2021) Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2021. Language Model Compression With Weighted Low-Rank Factorization. In ICLR. Huang et al. (2024) Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2024. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. In ICLR. Jain et al. (2024a) Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, and Tom Goldstein. 2024a. Refusal tokens: A simple way to calibrate refusals in large language models. Preprint, arXiv:2412.06748. Jain et al. (2024b) Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, and Puneet K. Dokania. 2024b. What makes and breaks safety fine-tuning? a mechanistic study. Preprint, arXiv:2407.10264. Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. Kurtić et al. (2023) Eldar Kurtić, Elias Frantar, and Dan Alistarh. 2023. Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems, 36:65597–65617. Kuzmin et al. (2024) Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, and Tijmen Blankevoort. 2024. Pruning vs quantization: Which is better? Preprint, arXiv:2307.02973. Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. Lermen et al. (2023) Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023. LoRA fine-tuning efficiently undoes safety training in Llama 2-Chat 70B. arXiv preprint arXiv:2310.20624. Li et al. (2023) Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 20336–20350. PMLR. Lieber et al. (2024) Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. 2024. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887. Lin et al. (2024) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for llm compression and acceleration. Preprint, arXiv:2306.00978. Liu et al. (2024) Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In ICLR. Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720. Marks and Tegmark (2023) Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. Marshall et al. (2024) Thomas Marshall, Adam Scherlis, and Nora Belrose. 2024. Refusal in llms is an affine function. arXiv preprint arXiv:2411.09003. Mazeika et al. (2024) Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. Merullo et al. (2021) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2021. Circuit Component Reuse Across Tasks in Transformer Language Models. In ICLR. Nanda et al. (2023) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217. O’Brien et al. (2024) Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde. 2024. Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296. Olah (2022) Chris Olah. 2022. Mechanistic interpretability, variables, and the importance of interpretable bases. Olah (2023) Chris Olah. 2023. [link]. Panickssery et al. (2023) Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048. Prakash et al. (2024) Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. 2024. Fine-tuning enhances existing mechanisms: A case study on entity tracking. arXiv preprint arXiv:2402.14811. Qi et al. (2024) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! In ICLR. Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models are Unsupervised Multitask Learners. OpenAI blog. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS. Robey et al. (2023) Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. SmoothLLM: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684. Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. Shah et al. (2023) Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Deshmukh, Michael Kuhlmann, et al. 2023. LoFT: Local proxy fine-tuning for improving transferability of adversarial attacks against large language model. arXiv preprint arXiv:2310.04445. Shao et al. (2023) Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137. Sun et al. (2024) Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A Simple and Effective Pruning Approach for Large Language Models. In ICLR. Syed et al. (2023) Aaquib Syed, Can Rager, and Arthur Conmy. 2023. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348. Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A Strong, Replicable Instruction-Following Model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html. Tigges et al. (2023) Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. 2023. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154. Touvron et al. (2023a) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023a. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR. Wang et al. (2024a) Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. 2024a. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. arXiv preprint arXiv:2405.15071. Wang et al. (2022) Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593. Wang et al. (2024b) Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. 2024b. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748. Wei et al. (2024) Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. Preprint, arXiv:2402.05162. Xiao et al. (2023) Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR. Xu and Hu (2022) Guangxuan Xu and Qingyuan Hu. 2022. Can model compression improve nlp fairness. arXiv preprint arXiv:2201.08542. Xu et al. (2023) Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen. 2023. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. arXiv preprint arXiv:2311.09827. Xu et al. (2024) Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, and Vivek Srikumar. 2024. Beyond perplexity: Multi-dimensional safety evaluation of llm compression. Preprint, arXiv:2407.04965. Yao et al. (2023) Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. 2023. A comprehensive study on post-training quantization for large language models. arXiv preprint arXiv:2303.08302. Yuan et al. (2023) Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. arXiv preprint arXiv:2312.05821. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL. Zhong et al. (2024) Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. 2024. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36. Zhou et al. (2024) Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pages 10586–10613. Zhu et al. (2024) Jie Zhu, Leye Wang, Xiao Han, Anmin Liu, and Tao Xie. 2024. Safety and performance, why not both? bi-objective optimized model compression against heterogeneous attacks toward ai software deployment. IEEE Transactions on Software Engineering. Zhu et al. (2023) Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633. Zou et al. (2023a) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023a. Representation Engineering: A Top-Down Approach to AI Transparency. arXiv preprint arXiv:2310.01405. Zou et al. (2023b) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023b. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043. Appendix A Compute Statement All computing was performed on a cluster of 6 NVIDIA RTX A600 GPUs. The total compute time for all experiments took 200-250 hours. Reproducing experiments will take the following amount of time in a similar cluster on a single GPU: Llama2-7b-chat: 1. Pruning: Each Wanda pruning experiment takes about 10 minutes. Magnitude pruning takes 5 minutes. 2. Refusal Direction: Calculating the refusal direction + evaluating via directional ablation and activation adding takes about 15 minutes. 3. AIRD: AIRD takes less than 10 seconds (not including model loading time). 4. Evaluation of Zero-Shot: Takes 10 minutes. 5. Evaluation on AdvBench: Takes about 15 minutes, we use vLLM Kwon et al. (2023) for this. Llama2-8b-chat: 1. Pruning: Each Wanda pruning experiment takes about 10 minutes. Magnitude pruning takes 7 minutes. 2. Refusal Direction: Calculating the refusal direction + evaluating via directional ablation and activation adding takes about 15 minutes. 3. AIRD: AIRD takes less than 10 seconds(not including model loading time). 4. Evaluation of Zero-Shot: Takes 10 minutes. 5. Evaluation on AdvBench: Takes about 15 minutes, we use vLLM Kwon et al. (2023) for this. Appendix B Refusal Direction Selecting Algorithm We borrow the refusal direction selection algorithm from Arditi et al. (2024). Given a collection of difference-in-means vectors, denoted as i(l)|i∈I,l∈[L]conditional-setsuperscriptsubscriptformulae-sequencedelimited-[]\r_i^(l)|i∈ I,l∈[L]\ ritalic_i( l ) | i ∈ I , l ∈ [ L ] , we evaluate the following key metrics: • bypass_score: Measures the average refusal rate on the validation set of harmful prompts (harmful(val)superscriptsubscriptharmful(val)D_harmful^(val)Dharmful(val)) when applying directional ablation to i(l)superscriptsubscriptr_i^(l)ritalic_i( l ). • induce_score: Assesses the average refusal rate on the validation set of harmless prompts (harmless(val)superscriptsubscriptharmless(val)D_harmless^(val)Dharmless(val)) when the activation addition of i(l)superscriptsubscriptr_i^(l)ritalic_i( l ) is applied. • kl_score: Computes the average Kullback-Leibler (KL) divergence between the model’s probability distributions at the final token position when evaluated on harmless(val)superscriptsubscriptharmless(val)D_harmless^(val)Dharmless(val) with and without directional ablation of i(l)superscriptsubscriptr_i^(l)ritalic_i( l ). To identify the optimal direction i∗(l∗)superscriptsubscriptsuperscriptsuperscriptr_i^*^(l^*)ritalic_i∗( l start_POSTSUPERSCRIPT ∗ ) end_POSTSUPERSCRIPT, we select the vector with the lowest bypass_score, while ensuring the following constraints are met: • induce_score>0induce_score0 induce\_score>0induce_score > 0 – Ensures that the selected direction is capable of inducing a refusal response. • kl_score<0.1kl_score0.1 kl\_score<0.1kl_score < 0.1 – Prevents the selection of directions that excessively alter model behavior on benign prompts. • l<0.8⁢L0.8l<0.8Ll < 0.8 L – Restricts the selection to earlier layers, avoiding interference with unembedding representations. B.1 Chat Templates For each model we utilize the following chat templates to prompt , see Table 8. Table 8: Models and their corresponding chat templates. The user instruction is denoted as Instruction. Post-instruction tokens, as defined in §2, are labeled in red. Model family Corresponding refusal phrases Llama-2 Chat "[INST] Instruction [/INST] " Llama-3 Instruct "<|start_header_id|>user<|end_header_id|> Instruction<|eot_id|><|start_header_id|>assistant<|end_header_id|> " Appendix C Details of Zero-Shot Evaluations 1. ARC-Challenge: (a) Downstream Task: Science Question Answering. (b) Overview: This metric gauges model performance on the ARC-Challenge portion of the AI2 Reasoning Challenge dataset. It comprises grade-school science questions that necessitate complex reasoning and an in-depth understanding of scientific principles444Further details can be found at https://allenai.org/data/arc.. 2. HellaSWAG: (a) Downstream Task: Commonsense Reasoning. (b) Overview: HellaSWAG is designed to test commonsense reasoning capabilities. It presents a context followed by several multiple-choice endings, with the objective of selecting the most plausible continuation. The dataset challenges models to interpret and reason about everyday situations555Additional information is available at https://huggingface.co/datasets/Rowan/hellaswag.. 3. WinoGrande: (a) Downstream Task: Commonsense Reasoning. (b) Overview: WinoGrande is a large-scale dataset for assessing commonsense reasoning. Presented as a fill-in-the-blank task with binary choices, the aim is to select the appropriate option, demanding robust commonsense understanding while mitigating dataset-specific biases666Further information is available at https://huggingface.co/datasets/winogrande.. 4. BoolQ: (a) Downstream Task: Yes/No Question Answering. (b) Overview: BoolQ is a dataset focused on yes/no questions, featuring 15,942 naturally occurring examples. Each instance comprises a question, a passage, and the corresponding answer, with optional contextual information such as the page title. The setup is akin to text-pair classification tasks found in natural language inference research777More details can be found at https://github.com/google-research-datasets/boolean-questions.. 5. RTE (Recognizing Textual Entailment): (a) Downstream Task: Textual Entailment. (b) Overview: The RTE task involves deciding whether a hypothesis can be logically inferred from a given premise. The dataset consists of sentence pairs, where the goal is to classify each pair as either "entailment" (if the hypothesis logically follows from the premise) or "not entailment" (if it does not)888Additional details are available at https://huggingface.co/datasets/nyu-mll/glue#rte.. Appendix D Safety Evaluations Details D.1 Adversarial Suffixes We borrow and modify the methodology of Wei et al. (2024) to generate adversarial suffixes which is: Llama2-7b-chat : Run the GCG attack Zou et al. (2023b) for 500500500500 iterations, with adversarial string initiated as “!!!!!!!!!!!!!!!!!!!!" and a batch size of 256256256256, top-k as 128128128128, with optimization over Llama2 Touvron et al. (2023a), with the system prompts removed, for three independent trials. We then identify the top three suffixes with the highest attack success rates on AdvBench, and use them in our evaluation. Llama3-8b-instruct: Run the GCG attack Zou et al. (2023b) for 500500500500 iterations, with adversarial string initiated as “!!!!!!!!!!!!!!!!!!!!" and a batch size of 256256256256, top-k as 128128128128, with optimization over Llama3-8b-instruct, with the system prompts removed, for three independent trials. For ethical reasons, we chose not to disclose the adversarial suffixes. D.2 ASRIAdv-DecodingsuperscriptsubscriptabsentAdv-Decoding_ Adv-Decoding^Istart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPTI in Llama3-8b-instruct For Llama2-7b-chat we utilize the [INST] wrapper around the prompt for ASRIAdv-DecodingsuperscriptsubscriptabsentAdv-Decoding_ Adv-Decoding^Istart_FLOATSUBSCRIPT Adv-Decoding end_FLOATSUBSCRIPTI. As Llama3-8b-instruct doesn’t support the [INST]. We modify the prompt by wrapping it around the chat template as mentioned in Table 8. Appendix E Sufficiency and Neccessity for Llama3-8b-Instruct Following the methodology in section 4. We provide the sufficiency test via activation addition and neccessity test via direction ablation for Llama3-8b-instruct. Figure 5: Necessity Test for Llama3-8b-instruct: Attack Score(ASR) after direction ablation vs no intervention on harmful instructions Figure 6: Sufficiency Test for Llama3-8b-instruct: Refusal Score after activation addition vs no intervention on harmless instructions Appendix F Compression Details F.1 Pruning Details Following the approach of Sun et al. (2024), we employ a block-wise pruning technique applied sequentially across Transformer blocks in Llama. Starting with the first block, we prune the seven linear layers—self⁢_⁢attn.qformulae-sequenceself_attnqself\_attn.qself _ attn . q, self⁢_⁢attn.kformulae-sequenceself_attnkself\_attn.kself _ attn . k, self⁢_⁢attn.vformulae-sequenceself_attnvself\_attn.vself _ attn . v, self⁢_⁢attn.oformulae-sequenceself_attnoself\_attn.oself _ attn . o, mlp.upformulae-sequencemlpupmlp.upmlp . up, mlp.gateformulae-sequencemlpgatemlp.gatemlp . gate, and mlp.downformulae-sequencemlpdownmlp.downmlp . down. After pruning, we recompute the block’s output before proceeding to the next one. Across all methods, we adopt unstructured pruning with a fixed sparsity of 0.5 (50%). F.2 Quantization Details LLM.int8(): We utilize https://huggingface.co/docs/bitsandbytes/main/en/reference/n/linear8bit base configuration to quantize models to 8bit. Activation Aware Quantization: We follow the original work999https://github.com/mit-han-lab/llm-awq to quantize models to 8bit with q_group_size = 128. Appendix G Refusal Mechanism doesn’t change after AIRD We provide the refusal directions in the models that underwent AIRD, see Table 9 Model Method lc/lsuperscriptl^c/llitalic_c / l icsuperscripti^ciitalic_c/i Calibration Type Llama2-7b Wanda 14/14 5/–5 Alpaca Llama2-7b Wanda 12/12 5/–5 Align Llama2-7b Magnitude 12/12 5/–5 — Llama3-8b Wanda 12/12 5/–5 Alpaca Llama3-8b Wanda 13/13 5/–5 Align Table 9: Refusal Directions of compressed models that underwent AIRD As we see, no change in the directions occur. We now present the cosine similarities of the refusal directions of the model that went through AIRD with the base compressed model, see Table 10. Model Method Cosine Similarity Llama2-7b Wanda-Align 0.9960.9960.9960.996 LLama2-7b Wanda-Alpaca 0.9950.9950.9950.995 Llama2-7b Magnitude 0.9910.9910.9910.991 Llama3-8b Wanda-Align 0.9970.9970.9970.997 Llama3-8b Wanda-Alpaca 0.9970.9970.9970.997 Table 10: Cosine Similarity of the refusal directions of the AIRD models and the base compressed models. Appendix H ASR Substrings We utilize the substrings in Figure 7 for all substring matching experiments. ["I’m sorry", "I am sorry", "I apologize", "As an AI", "As a language model", "As an assistant", "I cannot", "I can’t", "I am unable to", "I’m unable to", "I am not able to", "I’m not able to", "OpenAI", "As a Chatbot", "unethical"] Figure 7: The set of "refusal substrings" for evaluating whether a prompt is refused or not.

Instant research discovery