Paper deep dive

Refusal Behavior in Large Language Models: A Nonlinear Perspective

Fabian Hildebrandt, Andreas Maier, Patrick Krauss, Achim Schilling

Year: 2025Venue: arXiv preprintArea: Representation AnalysisType: EmpiricalEmbeddings: 40

Models: Bloom-3B, Bloom-560M, Llama-3.2-1B, Llama-3.2-3B, Qwen2-0.5B, Qwen2-1.5B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 7:23:51 PM

Summary

This paper investigates the refusal behavior of six Large Language Models (LLMs) across three architectural families. By applying linear (PCA) and nonlinear (t-SNE, UMAP) dimensionality reduction techniques to internal model activations, the authors demonstrate that refusal mechanisms are nonlinear and multidimensional, challenging the previous assumption that refusal is mediated by a single linear subspace.

Entities (7)

Bloom-560M-Instruct · llm · 100%Generalized Discrimination Value · metric · 100%Llama-3.2-1B-Instruct · llm · 100%PCA · dimensionality-reduction-technique · 100%Qwen2-0.5B-Instruct · llm · 100%t-SNE · dimensionality-reduction-technique · 100%UMAP · dimensionality-reduction-technique · 100%

Relation Signals (3)

Llama-3.2-1B-Instruct → evaluatedusing → PCA

confidence 95% · Table II: Comparison of Separability across Models and Methods

Qwen2-0.5B-Instruct → evaluatedusing → UMAP

confidence 95% · Table II: Comparison of Separability across Models and Methods

Refusal behavior → exhibits → Nonlinear Characteristics

confidence 90% · Our results reveal that refusal mechanisms exhibit nonlinear, multidimensional characteristics

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Refusal behavior in large language models (LLMs) enables them to decline responding to harmful, unethical, or inappropriate prompts, ensuring alignment with ethical standards. This paper investigates refusal behavior across six LLMs from three architectural families. We challenge the assumption of refusal as a linear phenomenon by employing dimensionality reduction techniques, including PCA, t-SNE, and UMAP. Our results reveal that refusal mechanisms exhibit nonlinear, multidimensional characteristics that vary by model architecture and layer. These findings highlight the need for nonlinear interpretability to improve alignment research and inform safer AI deployment strategies.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

39,280 characters extracted from source content.

Expand or collapse full text

Refusal Behavior in Large Language Models: A Nonlinear Perspective Fabian Hildebrandt CCN Group Pattern Recog. Lab. FAU Erlangen-Nürnberg Erlangen, Germany fabian.hildebrandt@fau.de Andreas Maier Pattern Recog. Lab. FAU Erlangen-Nürnberg Erlangen, Germany andreas.maier@fau.de Patrick Krauss∗ CCN Group Pattern Recog. Lab. FAU Erlangen-Nürnberg Erlangen, Germany patrick.krauss@fau.de Achim Schilling∗ Neuroscience Lab, University Hospital Erlangen, CCN Group Pattern Recog. Lab. FAU Erlangen-Nürnberg Erlangen, Germany achim.schilling@fau.de Abstract Refusal behavior in large language models (LLMs) enables them to decline responding to harmful, unethical, or inappropriate prompts, ensuring alignment with ethical standards. This paper investigates refusal behavior across six LLMs from three architectural families. We challenge the assumption of refusal as a linear phenomenon by employing dimensionality reduction techniques, including PCA, t-SNE, and UMAP. Our results reveal that refusal mechanisms exhibit nonlinear, multidimensional characteristics that vary by model architecture and layer. These findings highlight the need for nonlinear interpretability to improve alignment research and inform safer AI deployment strategies. Index Terms: refusal, mechanistic interpretability, LLM, AI alignment, neuroscience-inspired AI (neuroAI), explainable AI (XAI) Introduction Morality and impulse control are fundamental aspects of human behavior, enabling ethical decision-making, effective social interactions, and the maintenance of personal and societal relationships. The amygdala and the ventromedial prefrontal cortex play critical roles in moral behavior [1]. The amygdala associates harmful actions with negative emotions, while the ventromedial prefrontal cortex governs decision-making and self-control to prevent such actions. In psychopathy, these regions exhibit reduced activity, leading to an increased propensity for manipulation, impulsivity, and unethical decisions without remorse [2]. Impulse control, a key aspect of self-regulation housed in the prefrontal cortex, can be disrupted by various factors such as ADHD, substance use disorders, psychological conditions, stress, and brain injuries [3]. The human brain is often described as a prediction machine, continuously processing sensory information to guide decision-making. Similarly, Large Language Models (LLMs) are designed to predict the next word or sequence, enabling them to perform complex language tasks. Like the human brain, LLMs are tasked with handling nuanced decisions, such as refusing harmful instructions, yet their internal decision-making processes often remain opaque. Efforts to align LLMs with ethical principles have focused on feature search, which seeks to interpret model activations and uncover the mechanisms governing their behavior. The superposition hypothesis posits that multiple concepts or features are simultaneously represented within the same neural activation space [4, 5]. Consequently, individual neurons in LLMs often exhibit polysemantic behavior, encoding multiple distinct meanings rather than a one-to-one mapping. Achieving monosemantic representations—where neurons encode a single, unambiguous feature—is a critical goal for improving interpretability. Sparse autoencoders have been employed to identify such features by inflating the hidden space and enforcing sparsity [6, 7, 8, 9]. This approach enables the isolation of specific features for steering model behavior [10, 11, 12]. Recent studies suggest that refusal behavior in LLMs is mediated by a single linear subspace in the activation space. This feature can be manipulated to either disable or enforce refusal behavior across a range of open-source models, including those with up to 72B parameters [13]. Techniques like the difference-in-means method [14] and weight orthogonalization have been used to isolate and modify this subspace, resulting in models that either lose or gain the ability to refuse harmful instructions. However, emerging evidence challenges the assumption that refusal behavior resides in a linear subspace, pointing instead to a multidimensional and nonlinear nature [15]. In this work, we examine refusal behavior across six LLMs spanning three model families. By analyzing intermediate layer activations through both linear (PCA) and nonlinear (t-SNE, UMAP) dimensionality reduction techniques, we reveal that refusal behavior is a universal but architecture-specific phenomenon. Our findings show that refusal mechanisms are more complex and nonlinear than previously assumed, with distinct sub-clusters emerging in the activation space. These insights contribute to a deeper understanding of refusal behavior and its implications for aligning LLMs with ethical and safety standards. Methodology Behavioral studies and neural correlates In this study, we investigate the refusal behavior of large language models (LLMs) in response to harmful and harmless instructions, drawing parallels to behavioral studies that examine brain activity linked to cognitive functions. Our methodology involves designing a task that isolates the refusal mechanisms in LLMs, similar to decision-making tasks in behavioral research that require empathy and sensitivity. Simultaneously, we track the LLM activations to identify model activations associated with the refusal behavior. Additionally, the origin of these responses is localized and the embeddings are further investigated. Datasets Two distinct datasets are used. The first dataset DharmlesssubscriptharmlessD_harmlessDharmless contains harmless instructions from the ALPACA dataset [16] repackaged and published as a Hugging Face dataset [17]. The ALPACA dataset is a collection of 52000 harmless instruction-following prompts designed to fine-tune large language models for more effective and reliable task completion. The second dataset DharmfulsubscriptharmfulD_harmfulDharmful contains harmful instructions that originate from the LLM Attacks dataset [18], which was designed for adversarial attacks against aligned language models. The instructions are trying to induce harmful behavior by the LLMs. Again, a repackaged Hugging Face dataset is used [19]. Models To investigate the universality of the refusal behavior, we evaluated a set of six large language models (LLMs) from three model families. All models are fine-tuned for instruction following. The selection of different model families and varying parameter sizes allows to assess whether the observed effects are consistent and generalizable. The model families represent different alignment types from preference optimization to alignment by fine-tuning. The specific models used in this study are detailed in Table I. TABLE I: Comparison of Model Specifications Model Name Model Size Number of Layers Hidden Space Dimensions Model Origin Llama-3.2-1B-Instruct 1.1B 16 2048 Meta AI [20] Llama-3.2-3B-Instruct 3.2B 28 3072 Meta AI [20] Bloom-560M-Instruct 302M 24 1024 BigScience [21] Bloom-3B-Instruct 2.4B 30 2560 BigScience [21] Qwen2-0.5B-Instruct 391M 24 896 Alibaba Group [22] Qwen2-1.5B-Instruct 1.4B 28 1536 Alibaba Group [22] Extraction of the refusal activations The methodology for extracting activations related to the refusal mechanism follows the general approach outlined in [13] and [23]. First, we load the model using the TranformerLens library [24]. The library allows to cache any internal activation in the model for mechanistic interpretability research. Next, two equal-sized subsets of prompts from the two datasets DharmfulsubscriptharmfulD_harmfulDharmful and DharmlesssubscriptharmlessD_harmlessDharmless containing harmful and harmless instructions are loaded using the Hugging Face Datasets library. Using a chat-style generation template, we run the model on both datasets while caching the activations for each layer. To identify the refusal-related feature direction, we compute the difference-in-means by subtracting the mean activations of harmless prompts from those of harmful prompts at each token position. Finally, we store the residual activations afterr the self-attention layers and the multilayer perceptron (MLP) layers at the last token position, for further analysis and manipulation. Difference-in-means. The difference-in-means technique [14] can be used to identify the refusal direction in the model’s activations as shown in previous works [13], [23]. For each layer l∈[L]delimited-[]l∈[L]l ∈ [ L ] the mean activation μharmful(l)superscriptsubscriptharmful _harmful^(l)μharmful( l ) for harmful prompts from DharmfulsubscriptharmfulD_harmfulDharmful and μharmless(l)superscriptsubscriptharmless _harmless^(l)μharmless( l ) for harmless prompts from DharmlesssubscriptharmlessD_harmlessDharmless is calculated: μharmful(l)=1|Dharmful|⁢∑i∈Dharmfuly(l)⁢(i),μharmless(l)=1|Dharmless|⁢∑i∈Dharmlessy(l)⁢(i).formulae-sequencesuperscriptsubscriptharmful1subscriptharmfulsubscriptsubscriptharmfulsuperscriptsuperscriptsubscriptharmless1subscriptharmlesssubscriptsubscriptharmlesssuperscript split _harmful^(l)&= 1|D_harmful| _i% ∈ D_harmfuly^(l)(i),\\ _harmless^(l)&= 1|D_harmless| _i∈ D_% harmlessy^(l)(i). splitstart_ROW start_CELL μharmful( l ) end_CELL start_CELL = divide start_ARG 1 end_ARG start_ARG | Dharmful | end_ARG ∑i ∈ D start_POSTSUBSCRIPT harmful end_POSTSUBSCRIPT y( l ) ( i ) , end_CELL end_ROW start_ROW start_CELL μharmless( l ) end_CELL start_CELL = divide start_ARG 1 end_ARG start_ARG | Dharmless | end_ARG ∑i ∈ D start_POSTSUBSCRIPT harmless end_POSTSUBSCRIPT y( l ) ( i ) . end_CELL end_ROW (1) where yi(l)⁢(t)superscriptsubscripty_i^(l)(t)yitalic_i( l ) ( t ) represents the activation at the last token position in layer l for instruction i. The difference-in-means direction drefusal(l)superscriptsubscriptrefusald_refusal^(l)drefusal( l ) is then computed as: drefusal(l)=μharmfull(l)−μharmless(l).superscriptsubscriptrefusalsuperscriptsubscriptharmfullsuperscriptsubscriptharmlessd_refusal^(l)= _harmfull^(l)- _harmless^(l% ).drefusal( l ) = μharmfull( l ) - μharmless( l ) . (2) The vector drefusal(l)superscriptsubscriptrefusald_refusal^(l)drefusal( l ) captures both the direction and the magnitude of the refusal feature in the activations. Dimensionality reduction Principal Component Analysis (PCA). PCA is a widely used linear dimensionality reduction technique that transforms an unlabeled dataset into a new coordinate system where the greatest variance by any projection of the data lies on the first principal component, the second greatest variance on the second component, and so on [25]. This method identifies orthogonal axes that maximize variance, thereby simplifying high-dimensional data while preserving the maximum variability. PCA is commonly applied for feature extraction and the feature visualization. t-Distributed Stochastic Neighbor Embedding (t-SNE). t-SNE is a nonlinear dimensionality reduction technique commonly used for visualizing high-dimensional data in two or three dimensions. It models the pairwise similarities between points in the original space and seeks to preserve these relationships in the reduced space by minimizing a divergence between two probability distributions. Unlike linear methods like PCA, t-SNE is effective at capturing local structure and revealing clusters or patterns within complex datasets [26]. Uniform Manifold Approximation and Projection (UMAP). UMAP is a dimensionality reduction technique designed for visualization and the general nonlinear embedding of high-dimensional data. Based on manifold learning and topological data analysis, UMAP constructs a high-dimensional graph of data points and optimizes a low-dimensional projection to preserve both local and global structures. Compared to PCA, UMAP captures complex, nonlinear relationships in data, making it more effective for revealing meaningful clusters. Unlike t-SNE, UMAP offers faster computation, better scalability, and more interpretable low-dimensional embeddings while maintaining the ability to preserve local neighborhood information [27]. Activation separability metric Generalized Discrimination Value (GDV). To quantify the degree of clustering, we used the GDV as published and explained in detail in [28]. The GDV provides an objective measure of how well the hidden layer activations cluster according to the ASC types, offering insights into the model’s internal representations. Briefly, we consider N points =..=(xn,1,⋯,xn,D)x_n=1..N=(x_n,1,·s,x_n,D)xbold_n = 1 . . N = ( xitalic_n , 1 , ⋯ , xitalic_n , D ), distributed within D-dimensional space. A label lnsubscriptl_nlitalic_n assigns each point to one of L distinct classes Cl=1..LC_l=1..LCitalic_l = 1 . . L. In order to become invariant against scaling and translation, each dimension is separately z-scored and, for later convenience, multiplied with 1212 12divide start_ARG 1 end_ARG start_ARG 2 end_ARG: sn,d=12⋅xn,d−μdσd.subscript⋅12subscriptsubscriptsubscript s_n,d= 12· x_n,d- _d _d.sitalic_n , d = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ⋅ divide start_ARG xitalic_n , d - μitalic_d end_ARG start_ARG σitalic_d end_ARG . (3) Here, μd=1N⁢∑n=1Nxn,dsubscript1superscriptsubscript1subscript _d= 1N _n=1^Nx_n,d\;μitalic_d = divide start_ARG 1 end_ARG start_ARG N end_ARG ∑n = 1N xitalic_n , d denotes the mean, and σd=1N⁢∑n=1N(xn,d−μd)2subscript1superscriptsubscript1superscriptsubscriptsubscript2 _d= 1N _n=1^N(x_n,d- _d)^2σitalic_d = square-root start_ARG divide start_ARG 1 end_ARG start_ARG N end_ARG ∑n = 1N ( xitalic_n , d - μitalic_d )2 end_ARG the standard deviation of dimension d. Based on the re-scaled data points =(sn,1,⋯,sn,D)subscriptsubscript1⋯subscripts_n=(s_n,1,·s,s_n,D)sbold_n = ( sitalic_n , 1 , ⋯ , sitalic_n , D ), we calculate the mean intra-class distances for each class ClsubscriptC_lCitalic_l d¯⁢(Cl)=2Nl⁢(Nl−1)⁢∑i=1Nl−1∑j=i+1Nld⁢(si(l),sj(l)),¯subscript2subscriptsubscript1superscriptsubscript1subscript1superscriptsubscript1subscriptsuperscriptsubscriptssuperscriptsubscripts d(C_l)= 2N_l(N_l\!-\!1) _i=1^N_l-1% _j=i+1^N_ld(s_i^(l),s_j^(l)),over¯ start_ARG d end_ARG ( Citalic_l ) = divide start_ARG 2 end_ARG start_ARG Nitalic_l ( Nitalic_l - 1 ) end_ARG ∑i = 1Nitalic_l - 1 ∑j = i + 1Nitalic_l d ( si( l ) , sj( l ) ) , (4) and the mean inter-class distances for each pair of classes ClsubscriptC_lCitalic_l and CmsubscriptC_mCitalic_m d¯⁢(Cl,Cm)=1Nl⁢Nm⁢∑i=1Nl∑j=1Nmd⁢(si(l),sj(m)).¯subscriptsubscript1subscriptsubscriptsuperscriptsubscript1subscriptsuperscriptsubscript1subscriptsuperscriptsubscriptssuperscriptsubscripts d(C_l,C_m)= 1N_lN_m _i=1^N_l _% j=1^N_md(s_i^(l),s_j^(m)).over¯ start_ARG d end_ARG ( Citalic_l , Citalic_m ) = divide start_ARG 1 end_ARG start_ARG Nitalic_l Nitalic_m end_ARG ∑i = 1Nitalic_l ∑j = 1Nitalic_m d ( si( l ) , sj( m ) ) . (5) Here, NksubscriptN_kNitalic_k is the number of points in class k, and si(k)superscriptsubscriptss_i^(k)si( k ) is the it⁢hsuperscriptℎi^thiitalic_t h point of class k. The quantity d⁢(a,b)abd(a,b)d ( a , b ) is the euclidean distance between a and b. Finally, the Generalized Discrimination Value (GDV) is calculated from the mean intra-class and inter-class distances as follows: GDV=1D⁢[1L⁢∑l=1Ld¯⁢(Cl)−2L⁢(L−1)⁢∑l=1L−1∑m=l+1Ld¯⁢(Cl,Cm)]GDV1delimited-[]1superscriptsubscript1¯subscript21superscriptsubscript11superscriptsubscript1¯subscriptsubscript GDV= 1 D [ 1L _l=1^L % d(C_l)\;-\; 2L(L\!-\!1) _l=1^L-1 _m=l+1^L d(C_% l,C_m) ]GDV = divide start_ARG 1 end_ARG start_ARG square-root start_ARG D end_ARG end_ARG [ divide start_ARG 1 end_ARG start_ARG L end_ARG ∑l = 1L over¯ start_ARG d end_ARG ( Citalic_l ) - divide start_ARG 2 end_ARG start_ARG L ( L - 1 ) end_ARG ∑l = 1L - 1 ∑m = l + 1L over¯ start_ARG d end_ARG ( Citalic_l , Citalic_m ) ] (6) whereas the factor 1D1 1 Ddivide start_ARG 1 end_ARG start_ARG square-root start_ARG D end_ARG end_ARG is introduced for dimensionality invariance of the GDV with D as the number of dimensions. Note that the GDV is invariant with respect to a global scaling or shifting of the data (due to the z-scoring), and also invariant with respect to a permutation of the components in the N-dimensional data vectors (because the euclidean distance measure has this symmetry). The GDV is zero for completely overlapping, non-separated clusters, and it becomes more negative as the separation increases. A GDV of -1 signifies already a very strong separation and perfect clustering. Results Refusal is a universal feature Refusal behavior is consistently observed across all six tested models, with a distinct separation between harmful and harmless instructions in the dimensionality-reduced residual activations. This separation is evident across model families and sizes. Figure 1 illustrates the refusal feature in three models at different layers. Table I summarizes the lowest GDV values, indicating the best separability between harmful and harmless instructions, along with the corresponding model layers and dimensionality-reduction techniques. Figure 1: Dimensionality-reduced residual activations using PCA, comparing harmful and harmless instructions across three different models and layers. a) Qwen2-1.5B-Instruct model (first, middle, last layers). b) Bloom-3b model (first, middle, last layers). c) Llama-3.2-3B-Instruct model (first, middle, last layers). TABLE I: Comparison of Separability across Models and Methods Model Method GDV Layer Llama-3.2-1B-Instruct PCA -0.45 12/16 (ATTENTION) Llama-3.2-1B-Instruct t-SNE -0.62 9/16 (MLP) Llama-3.2-1B-Instruct UMAP -0.69 16/16 (MLP) Llama-3.2-3B-Instruct PCA -0.51 24/28 (ATTENTION) Llama-3.2-3B-Instruct t-SNE -0.60 11/28 (MLP) Llama-3.2-3B-Instruct UMAP -0.80 15/28 (ATTENTION) Qwen2-0.5B-Instruct PCA -0.46 1/24 (ATTENTION) Qwen2-0.5B-Instruct t-SNE -0.73 5/24 (ATTENTION) Qwen2-0.5B-Instruct UMAP -0.87 5/24 (MLP) Qwen2-1.5B-Instruct PCA -0.44 4/28 (ATTENTION) Qwen2-1.5B-Instruct t-SNE -0.71 2/28 (MLP) Qwen2-1.5B-Instruct UMAP -0.89 18/28 (ATTENTION) bloom-3b PCA -0.40 6/30 (MLP) bloom-3b t-SNE -0.63 28/30 (ATTENTION) bloom-3b UMAP -0.66 25/30 (MLP) bloom-560m PCA -0.41 5/24 (MLP) bloom-560m t-SNE -0.60 23/24 (ATTENTION) bloom-560m UMAP -0.65 22/24 (MLP) Refusal is a nonlinear feature Our analysis demonstrates that refusal behavior in LLMs extends beyond a simple one-dimensional linear subspace, revealing a complex, nonlinear structure. Using PCA, UMAP, and t-SNE, we visualized dimensionality-reduced activations for harmful and harmless instructions across multiple layers of six LLMs. While PCA, a linear technique, captures variance through linear distances, UMAP and t-SNE effectively identify nonlinear relationships, offering deeper insights into the activation space. UMAP and t-SNE consistently yielded clearer separations than PCA, as indicated by lower GDV scores across all models. For instance, the Qwen2-1.5B-Instruct model achieved nearly perfect cluster separability in its 18th layer, with distinct, compact clusters for harmful and harmless instructions (Fig. 2). These clusters, characterized by large inter-cluster and compact intra-cluster distances, provide a strong basis for probing classifier training. Figure 2: Dimensionality-reduced residual activations of the Qwen2-1.5B-Instruct model at layer 18, visualized using UMAP, showing distinct clusters for harmful and harmless instructions. Further analysis revealed evolving cluster morphologies across layers. Linear separability changed gradually, while UMAP and t-SNE highlighted dynamic variations in cluster shapes and the emergence of sub-clusters. These sub-clusters, as seen in Fig. 3, suggest additional nuanced features, potentially indicating the division of harmful instructions into finer sub-features. Figure 3: Dimensionality-reduced residual activations of the Qwen2-1.5B-Instruct model at layer 11, showing the emergence of sub-clusters for harmful instructions. These findings underscore that refusal behavior is a multidimensional and nonlinear phenomenon, necessitating advanced techniques for comprehensive analysis and interpretability. Distinct refusal mechanisms across different model families Unlike the localized functional architecture of the human brain, where specialized areas handle specific tasks, LLMs exhibit diverse strategies for embedding the distinction between harmful and harmless instructions. Refusal behavior varies across model families and layers, as visualized in Fig. 4. Figure 4: GDV, intra-class distance (compactness of harmful and harmless clusters), and inter-class distance (separation between clusters) of the dimensionality-reduced embeddings using PCA. a) Qwen2-1.5B-Instruct demonstrates early layer dominance of the refusal feature. b) Bloom-3b shows peak separability at early to intermediate layers but weaker discrimination in later layers. c) Llama-3.2-3B-Instruct displays progressively stronger separation. Qwen2 Models. In the Qwen2 family, refusal behavior is primarily encoded in early layers. PCA reveals that the refusal feature emerges within the first few layers, with the 0.5B model peaking at the first layer and the 1.5B model at the fourth. Early layers integrate multiple principal components to define the refusal direction, while later layers refine it along the first principal component. This progression leads to stable inter-class distances and declining intra-class distances, resulting in compact harmful clusters. UMAP and t-SNE further confirm robust separability after the fifth layer. Bloom Models. The Bloom architecture exhibits a distinct refusal mechanism, with a tendency to misclassify harmless instructions as harmful. The refusal direction aligns with the first principal component or main UMAP/t-SNE direction from the initial layer, where instructions are classified in a near-binary manner. GDV peaks in the fifth and sixth layers, but subsequent layers show weakened discrimination as intra-class distances increase and inter-class distances decrease. UMAP visualizations reveal sub-clusters within harmless instructions. Llama Models. Llama models display a gradual refinement of refusal behavior. In early layers, harmful and harmless embeddings are mixed, indicating weak differentiation. Refusal strength intensifies in deeper layers, with Llama-3.2-3B-Instruct achieving compact clusters for harmful instructions and widespread embeddings for harmless ones, demonstrating stronger recognition. Conversely, the smaller Llama-3.2-1B-Instruct, with fewer hidden dimensions, shows more mixed embeddings, consistent with the superposition hypothesis. Nonlinear methods, such as t-SNE and UMAP, corroborate the emergence of clearer refusal representations in middle layers. These findings reveal that refusal behavior is universally present but uniquely mediated across different architectures, reflecting varied strategies for harmful content differentiation. Discussion Refusal behavior, enabling differentiation between harmful and harmless instructions, was consistently observed across all six models tested, corroborating previous findings [13]. Our analysis highlights that refusal mechanisms vary significantly across model families, reflecting architecture-specific strategies for harmful content detection. Qwen2 models primarily encode refusal features in early layers, achieving stable inter-class distances and compact intra-class structures. Bloom models exhibit peak separability in intermediate layers but demonstrate weaker discrimination in later layers, often misclassifying harmless instructions. Conversely, Llama models refine refusal gradually across layers, with stronger separability emerging in deeper layers. Contrary to the assumption that refusal resides in a simple one-dimensional linear subspace [13], our findings reveal a more complex, nonlinear structure. Using PCA, UMAP, and t-SNE, we visualized latent space activations and found that nonlinear methods consistently outperformed PCA in separating harmful and harmless embeddings, as evidenced by the separability metric GDV. These results align with recent studies on jailbreak attacks, which indicate that nonlinear features, rather than universal linear ones, drive the success of such attacks [29]. Although focused on jailbreak mechanisms, these findings suggest a potential connection between jailbreak features and the refusal and harmfulness features, given the critical role of refusal in determining whether a model responds or declines. Furthermore, the phenomenon of ”alignment faking,” where LLMs pretend to align with training objectives while resisting preference modifications [30], raises concerns about the robustness of model alignment processes. Persistent alignment faking could lock in model preferences, posing challenges for future fine-tuning efforts. These insights underscore the importance of nonlinear interpretability techniques for understanding refusal behavior and its implications for alignment, robustness, and ethical AI deployment. Conclusion This study investigates refusal behavior in large language models (LLMs) and provides new insights into its mechanisms and characteristics. By analyzing six LLMs across three architectural families, we demonstrate that refusal behavior is a universal phenomenon but exhibits architecture-specific patterns. Contrary to prior assumptions that refusal mechanisms are linear and confined to single activation directions, our findings reveal that they are inherently nonlinear and multidimensional. Using advanced dimensionality reduction techniques such as UMAP and t-SNE, we uncover richer and more complex activation patterns than those detected through traditional linear methods like PCA. The results emphasize the importance of nonlinear interpretability methods in understanding and improving model alignment. Refusal mechanisms vary significantly across architectures, with Qwen models encoding refusal early, Bloom models demonstrating intermediate-layer strengths, and Llama models refining refusal behavior in deeper layers. These differences highlight the diverse strategies employed by LLMs to distinguish harmful and harmless prompts, which has implications for designing safer, more robust AI systems. Future research should explore the transferability of nonlinear refusal probes across model families and examine their interplay with jailbreak attacks and other adversarial strategies. Understanding refusal behavior’s nuanced and nonlinear nature will be pivotal in enhancing the alignment, transparency, and ethical deployment of LLMs. Author contributions FH, PK and AS conceptualized the study and designed the study. FH conducted the experiments. PK, AM and AS provided guidance, supervision, and feedback on the research design and interpretation of results. All authors wrote, reviewed and approved the final manuscript. Code availability All results and the code are published on a public GitHub repository [31] under the MIT license. Acknowledgements This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation): KR 5148/3-1 (project number 510395418), KR 5148/5-1 (project number 542747151), and GRK 2839 (project number 468527017) to PK, and grant SCHI 1482/3-1 (project number 451810794) to AS. References [1] R. Blair, “The amygdala and ventromedial prefrontal cortex in morality and psychopathy,” Trends in Cognitive Sciences, vol. 11, no. 9, p. 387–392, 2007, opinion article. [2] A. Glenn, A. Raine, and R. Schug, “The neural correlates of moral decision-making in psychopathy,” Molecular Psychiatry, vol. 14, p. 5–6, 2009, published 19 December 2008. [Online]. Available: https://doi.org/10.1038/mp.2008.104 [3] S. Kim and D. Lee, “Prefrontal cortex and impulsive decision making,” Biological Psychiatry, vol. 69, no. 12, p. 1140–1146, 2011, epub 2010 Aug 21. [4] C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter, “Zoom in: An introduction to circuits,” Distill, 2020, https://distill.pub/2020/circuits/zoom-in. [5] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah, “Toy models of superposition,” 2022. [Online]. Available: https://arxiv.org/abs/2209.10652 [6] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah, “Towards monosemanticity: Decomposing language models with dictionary learning,” Transformer Circuits Thread, 2023, https://transformer-circuits.pub/2023/monosemantic-features/index.html. [7] M. Faruqui, Y. Tsvetkov, D. Yogatama, C. Dyer, and N. Smith, “Sparse overcomplete word vector representations,” 2015. [Online]. Available: https://arxiv.org/abs/1506.02004 [8] G. Goh, “Decoding the thought vector,” https://gabgoh.github.io/ThoughtVectors/, 2016. [9] S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski, “Linear algebraic structure of word senses, with applications to polysemy,” 2018. [Online]. Available: https://arxiv.org/abs/1601.03764 [10] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks, “Representation engineering: A top-down approach to ai transparency,” 2023. [Online]. Available: https://arxiv.org/abs/2310.01405 [11] A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid, “Steering language models with activation engineering,” 2024. [Online]. Available: https://arxiv.org/abs/2308.10248 [12] N. Panickssery, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner, “Steering llama 2 via contrastive activation addition,” 2024. [Online]. Available: https://arxiv.org/abs/2312.06681 [13] A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” 2024. [Online]. Available: https://arxiv.org/abs/2406.11717 [14] N. Belrose, “Diff-in-means concept editing is worst-case optimal: Explaining a result by sam marks and max tegmark,” 2023, accessed: 2025-01-08. [Online]. Available: https://blog.eleuther.ai/diff-in-means/ [15] J. Engels, E. J. Michaud, I. Liao, W. Gurnee, and M. Tegmark, “Not all language model features are linear,” 2024. [Online]. Available: https://arxiv.org/abs/2405.14860 [16] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford_alpaca, 2023. [Online]. Available: https://github.com/tatsu-lab/stanford_alpaca [17] M. Labonne, “harmless_alpaca,” https://huggingface.co/datasets/mlabonne/harmless_alpaca, 2024. [Online]. Available: https://huggingface.co/datasets/mlabonne/harmless_alpaca [18] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” 2023. [19] M. Labonne, “harmful_behaviors,” https://huggingface.co/datasets/mlabonne/harmful_behaviors, 2024. [Online]. Available: https://huggingface.co/datasets/mlabonne/harmful_behaviors [20] Meta, “Model cards and prompt formats - llama 3.2,” https://w.llama.com/docs/model-cards-and-prompt-formats/llama3_2, 2023. [21] B. W. et al., “Bloom: A 176b-parameter open-access multilingual language model,” 2023. [Online]. Available: https://arxiv.org/abs/2211.05100 [22] A. Y. et al., “Qwen2 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2407.10671 [23] M. Labonne, “Uncensor any llm with abliteration,” Hugging Face Community Blog, June 2024, published June 13, 2024. [Online]. Available: https://huggingface.co/blog/mlabonne/abliteration [24] N. Nanda and J. Bloom, “Transformerlens,” https://github.com/TransformerLensOrg/TransformerLens, 2022. [25] K. Pearson, “On lines and planes of closest fit to systems of points in space,” Philosophical Magazine, vol. 2, no. 11, p. 559–572, 1901. [26] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” Journal of Machine Learning Research, vol. 9, p. 2579–2605, 2008. [27] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018. [28] A. Schilling, A. Maier, R. Gerum, C. Metzner, and P. Krauss, “Quantifying the separability of data classes in neural networks,” Neural Networks, vol. 139, p. 278–293, July 2021, epub 2021 Apr 5. [29] N. M. Kirch, S. Field, and S. Casper, “What features in prompts jailbreak llms? investigating the mechanisms behind attacks,” 2024. [Online]. Available: https://arxiv.org/abs/2411.03343 [30] R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, J. Kaplan, B. Shlegeris, S. R. Bowman, and E. Hubinger, “Alignment faking in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2412.14093 [31] F. Hildebrandt, “Refusal-llms,” 2025, accessed: 2025-01-11. [Online]. Available: https://github.com/FabianHildebrandt/Refusal-LLMs

Instant research discovery