Paper deep dive
Localizing Safety Alignment: MLP Layers and Mid-Network Blocks Encode Refusal Behavior in Large Language Models
Mingyu Zong, Sampad Mohanty, Bhaskar Krishnamachari
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Safety alignment in large language models is often treated as a distributed property of the entire network, yet its practical brittleness suggests that refusal behavior may be concentrated in a smaller set of parameters. This work addresses where safety-aligned refusal is encoded by transplanting weights from aligned models into matched unaligned base models at multiple levels of granularity. Using two open-weight model pairs and four safety benchmarks, we conducted experiments to compare the effects of replacing attention weights, MLP weights, contiguous layer regions, and MLP blocks. Across both model families, refusal transfer is dominated by MLP weights: replacing MLP parameters recovers substantially more malicious-prompt refusal than replacing attention parameters, with gains of at least 2.7 times more across benchmarks. Within the MLP stack, refusal-relevant parameters exhibit a consistent mid-network concentration, as the block spanning layers 8-11 is selected first in all six greedy searches over model-dataset pairs. The results also show that the composition of safety-relevant components is non-additive: in five of six greedy trajectories, adding more aligned blocks can reduce refusal performance, and selective block subsets can outperform full MLP transplantation on malicious refusal, benign over-refusal, or both. Finally, greedy orders transferred to OR-Bench vary with the source benchmark used to derive them, indicating a benchmark-dependent precision-coverage trade-off. These results suggest that safety alignment in current LLMs is both localized and interaction-sensitive, offering insight into alignment brittleness and potential avenues for targeted safety interventions.
Tags
Links
- Source: https://arxiv.org/abs/2608.11583v1
- Canonical: https://arxiv.org/abs/2608.11583v1
Trouble viewing inline? Open PDF directly â
Full Text
58,241 characters extracted from source content.
Expand or collapse full text
Localizing Safety Alignment: MLP Layers and Mid-Network Blocks Encode Refusal Behavior in Large Language Models Mingyu Zong â Department of Computer Science University of Southern California Los Angeles, CA 90007 mzong@usc.edu Sampad Mohanty Department of Computer Science University of Southern California Los Angeles, CA 90007 sbmohant@usc.edu Bhaskar Krishnamachari Department of Electrical and Computer Engineering Department of Computer Science University of Southern California Los Angeles, CA 90007 bkrishna@usc.edu Abstract Safety alignment in large language models is often treated as a distributed property of the entire network, yet its practical brittleness suggests that refusal behavior may be concentrated in a smaller set of parameters. This work addresses where safety-aligned refusal is encoded by transplanting weights from aligned models into matched unaligned base models at multiple levels of granularity. Using two open-weight model pairs and four safety benchmarks, we conducted experiments to compare the effects of replacing attention weights, MLP weights, contiguous layer regions, and MLP blocks. Across both model families, refusal transfer is dominated by MLP weights: replacing MLP parameters recovers substantially more malicious-prompt refusal than replacing attention parameters, with gains of at least 2.7 times more across benchmarks. Within the MLP stack, refusal-relevant parameters exhibit a consistent mid-network concentration, as the block spanning layers 8â11 is selected first in all six greedy searches over model-dataset pairs. The results also show that the composition of safety-relevant components is non-additive: in five of six greedy trajectories, adding more aligned blocks can reduce refusal performance, and selective block subsets can outperform full MLP transplantation on malicious refusal, benign over-refusal, or both. Finally, greedy orders transferred to OR-Bench vary with the source benchmark used to derive them, indicating a benchmark-dependent precision-coverage trade-off. These results suggest that safety alignment in current LLMs is both localized and interaction- sensitive, offering insight into alignment brittleness and potential avenues for targeted safety interventions. 1 1 Introduction Large language models (LLMs) are Transformer-based neural language models trained on large- scale text corpora to predict and generate natural language outputs [Vaswani et al., 2017, Brown â The authors thank collaborators Jwala Dhamala and Rahul Gupta from Amazon for helpful directions and inputs. 1 Code and datasets are available at https://github.com/ANRGUSC/Localizing-Safety-Alignment. 1 arXiv:2608.11583v1 [cs.AI] 12 Aug 2026 et al., 2020]. Scaling model size, dataset size, and computational resources has led to broad im- provements in language modeling and downstream task performance, making LLMs increasingly capable general-purpose agents [Kaplan et al., 2020, Bommasani et al., 2022]. In practice, many deployed LLMs are first pretrained to acquire general linguistic and world knowledge, then adapted to downstream tasks so that their responses better match user intent, task requirements, and social norms. This two-stage development process makes safety alignment especially important: pre- training provides broad capabilities, but whether and when a model refuses harmful requests is largely shaped during post-training. Modern alignment pipelines, including supervised instruction tuning, reinforcement learning from human feedback (RLHF), and preference-based optimization, have substantially improved safety of LLMs [Amballa et al., 2024, Ouyang et al., 2022, Bai et al., 2022a, Rafailov et al., 2023]. However, aligned refusal behavior tends to be brittle in practice. Prior work has shown that small downstream updates can weaken safeguards or unintentionally alter safety behavior [Qi et al., 2023, Yang et al., 2023]. In addition, carefully designed attacks such as jailbreaks, adversarial suffixes, automated red-teaming, and prompt transformations can bypass refusal mechanisms, suggesting that refusal is not a fixed property of model capability but a learned behavior sensitive to prompt format, model family, and internal representation [Wei et al., 2023, Zou et al., 2023, Perez et al., 2022, Carlini et al., 2024, Zhang et al., 2026]. At the same time, safety tuning can also produce the opposite failure mode, which is over-refusal on harmless or legitimate prompts [Bianchi et al., 2023, R Ěottger et al., 2024]. Therefore, LLM safety must be evaluated along two axes: resistance to harmful instructions and avoidance of over-refusal on harmless prompts. A model that refuses too rarely remains vulnerable to misuse, while a model that refuses too broadly may become unhelpful. These observations motivate a more structured understanding of how refusal is implemented inside LLMs. A growing body of mechanistic work demonstrates that safety-relevant behaviors can often be detected or steered through internal activations. Activation space studies have identified low-dimensional directions associated with refusal behavior [Arditi et al., 2024, Wollschlager et al., 2025]. Another line of work utilizes localized parameter or representation interventions to show that targeted manipulations can produce predictable behavioral changes [Wei et al., 2024, Zhao et al., 2024, Lee et al., 2024]. These findings indicate that refusal is not uniformly distributed throughout the network, but they leave open a complementary weight-space question: which parameters, and at which depths, are responsible for transferring safety refusal from an aligned model to its base counterpart? We hypothesize that safety-aligned refusal is localized: the parameters mediating transferable refusal behavior are concentrated in specific weight matrices and depth ranges rather than uni- formly distributed across the network. Such concentration could help explain the brittleness of aligned refusal under targeted parameter updates and could enable more selective interventions. We test this hypothesis through selective weight transplantation. Using two matched open-weight model pairs: RealSafe-R1-7B/DeepSeek-R1-Distill-Qwen-7B and saferlhf ultrasft/Llama-3.1-8B, we construct hybrid models by replacing subsets of parameters in the unaligned base model with the corresponding parameters from the aligned model. Because each pair shares the same archi- tecture, behavioral differences can be attributed directly to the transplanted weights rather than to model design. By varying the granularity of transplantation, we answer these finer localization questions: Is transferable refusal behavior mediated more strongly by attention weights or MLP weights? Is it concentrated in particular depth ranges? Our experiments provide consistent evidence across both model families that transferable refusal behavior is mediated more strongly by MLP than attention weights. Within the MLP pathway, a mid-depth block spanning layers 8â11 is selected first in all six greedy searches over model- dataset pairs, indicating a consistent concentration of refusal-relevant parameters at this depth. 2 At the same time, aligned blocks do not compose additively: in most settings, adding additional aligned blocks does not guarantee increased refusal, and selective MLP-block subsets can outperform full MLP transplantation. We further validate these patterns using larger filtered subsets. To sum up, this paper suggests that safety-aligned refusal behavior in the model families studied is largely mediated by MLP weights and is particularly concentrated in mid-network layers, based on empirical evidence. The rest of the paper is organized as follows: the next section introduces related work on safety alignment. Section 3 describes the matched model pairs, hybrid modelsâ construction procedures, evaluation benchmarks, and greedy block selection method used to localize safety-relevant param- eters, followed by the results section which presents the main findings. Section 5 discusses the implications of these findings for alignment brittleness, safety precision, and benchmark choice. Finally, we conclude with limitations of this study and directions for future work. 2 Related Work Safety alignment in large language models is commonly achieved through supervised instruction tuning, reinforcement learning from human feedback, constitutional AI, or preference-based opti- mization methods. Foundational work has established instruction tuning and RLHF as practical mechanisms for training models to follow user instructions while incorporating human preferences over helpfulness and harmlessness [Ouyang et al., 2022, Bai et al., 2022a]. Constitutional AI further reduces reliance on direct human labels by using rule-based self-critique and AI feedback to im- prove harmlessness [Bai et al., 2022b]. More recent preference-optimization methods such as Direct Preference Optimization (DPO) simplify the RLHF pipeline by optimizing directly on preference pairs without explicitly training a separate reward model [Rafailov et al., 2023]. Building on this paradigm, safety-focused instruction-tuning methods directly optimize refusal behavior on harmful requests in open-weight models [Amballa et al., 2024]. These approaches demonstrate that refusal behavior can be reliably induced through post-training, but they also imply that refusal is learned and may therefore be sensitive to later updates or changes in prompting. Despite the effectiveness of post-training alignment, evaluations on aligned LLMs show that learned refusal is not fully robust. Downstream adaptation affects safety, showing that fine-tuning or other targeted updates can substantially weaken aligned safeguards [Qi et al., 2023, Yang et al., 2023]. Automated red-teaming methods can instruct language models themselves to discover prompts that elicit undesirable outputs [Perez et al., 2022]. Other attacks search for adversar- ial suffixes or transform harmful instructions into alternative formats, including mathematical or encoded prompts, in order to bypass safety filters [Zou et al., 2023, Carlini et al., 2024, Zhang et al., 2026]. These attacks highlight that many refusal mechanisms are sensitive to surface form and distribution shift. On the other hand, excessive refusal is also problematic: models may reject benign prompts that contain sensitive terms or resemble unsafe requests, reducing their practical usefulness [Bianchi et al., 2023, R Ěottger et al., 2024, Cui et al., 2024]. In order to understand and address the brittleness of safety alignment, recent work studies how safety-related behavior is organized within the internal structure of language models. Arditi et al. [2024] claim that, for each model they study, a single extracted direction strongly mediates refusal. They also demonstrate that ablating that direction suppresses refusal, while adding it to the model induces over-refusal on benign inputs. Wollschlager et al. [2025] argue that refusal does not reside in a single direction, but is organized in multi-dimensional polyhedral cones that contain multiple refusal-mediating directions. Despite their disagreement about geometry, both papers suggest that refusal is not uniformly encoded across the network, but depends on structured and 3 manipulable internal features. Furthermore, prior work also proves that targeted interventions can predictably alter a modelâs safety behavior. Using layer-wise pruning, Zhao et al. [2024] identify âsafety layersâ inside a model. The authors then propose Layer-specific Editing (LED) to restrict updates on those layers so that downstream toxic layers decode toward safe refusals instead of harmful continuations. Lee et al. [2024] propose Conditional Activation Steering (CAST), which applies a refusal steering vector when a promptâs hidden states match a learned condition vector, thus enabling selective refusal rules such as refusing only particular categories of content while preserving normal responses for others. This result further supports the view that refusal can be controlled through compact internal representations. At the parameter level, Wei et al. [2024] study safety brittleness through pruning and low-rank modifications. Critical neurons and ranks are distinguished by disentangling safety-related components. Surprisingly, the safety-critical regions are sparse, taking up around 3% of parameters and 2.5% of ranks. It is also reported that safety and utility appear more differentiated in MLP layers than in attention layers, which is in line with the findings from this work. Beyond internal evaluation, a large number of benchmarks has been developed to measure both resistance to harmful requests and failure on benign ones. AdvBench tests whether models comply with harmful instructions under adversarial prompting [Zou et al., 2023], while TwinPrompt and SGXSTest extend this style of evaluation with paired settings designed to probe safety behavior more systematically [KrauĂ et al., 2025, Gupta et al., 2024]. Complementary benchmarks emphasize the importance of false positives. XSTest evaluates exaggerated safety behavior on safe prompts that resemble unsafe ones [R Ěottger et al., 2024]. OR-Bench evaluates whether aligned models over-refuse harmless prompts [Cui et al., 2024]. Existing work has primarily characterized safety at two levels: as an internal phenomenon stud- ied through activation-level analysis or targeted editing, and as an external behavioral property measured by benchmarks. Less attention has been drawn to a direct weight-space localization ques- tion: in a safety-aligned model, which parameters actually carry the transferable refusal behavior relative to its unaligned counterpart? This paper addresses this question by comparing selective transplants of attention weights, MLP weights, contiguous layer ranges, and MLP blocks across matched base and aligned model pairs. The experiments are set up to connect safety evaluation with mechanistic localization, focusing specifically on where aligned refusal behavior resides in the network. 3 Experimental Setup ModelsBenchmarks Aligned/Base PairLayersNameHarmful Prompts Benign Prompts RealSafe-R1-7B / DeepSeek-R1-Distill-Qwen-7B 28 TwinPromptâ SGXSTestâ saferlhfultrasft / Llama-3.1-8B32 AdvBenchâ OR-Benchâ Table 1: Overview of models and benchmarks. 4 3.1 Model Pairs We analyze two pairs of open-weight models that share the same architecture within each pair but differ in whether they were safety-aligned (Table 1). The first pair consists of RealSafe-R1-7B and DeepSeek-R1-Distill-Qwen-7B, both are 28-layer Qwen2-based models [Suma and Dauncey, 2025, Zhang et al., 2025]. RealSafe-R1-7B underwent safety training on approximately 15K examples, including roughly 10K direct harmful queries and 5K jailbreak prompts. The second pair consists of saferlhf ultrasft and Llama-3.1-8B [Huang, 2025, Grattafiori et al., 2024]. Both models are 32- layer Llama models, with saferlhf ultrasft being the fine-tuned version on the instruct variant of Llama-3.1-8B on the saferlhfultra dataset [Ji et al., 2025]. 3.2 Weight Transplantation Procedure All hybrid models are constructed by replacing selected weight matrices in the base model with the corresponding matrices from the aligned model. Because of the shared architecture, behavioral differences in the resulting hybrids can be attributed directly to the transplanted weights. 3.3 Granularity of Interventions This study evaluates several levels of transplantation granularity. First, we study component-level hybrids. The attn configuration replaces all attention projec- tions (qproj, kproj, vproj, and oproj) across all layers, while keeping the base modelâs MLP weights unchanged. The mlp configuration performs the complementary intervention, replacing only the feed-forward projections (gate proj, upproj, and downproj). We also evaluate contiguous layer groups. The first5, mid5, and last5 configurations trans- plant both attention and MLP weights for five consecutive layers, while first half and secondhalf transplant the corresponding half of the network weights. Motivated by the strong performance of mlp hybrids, we then localize safety within the MLP stack. We partition the modelâs MLP weights into contiguous blocks, each containing 4 layers. For RealSafe-R1-7B with 28 layers, this yields seven blocks: B1 = layers 0-3 through B7 = layers 24-27. For saferlhf ultrasft with 32 layers, this yields eight blocks: B1 = layers 0-3 through B8 = layers 28-31. We first perform transplantation for each individual block and then analyze combinations of blocks. 3.4 Evaluation Benchmarks and Subset Construction We conduct model evaluations on four benchmarks that probe complementary aspects of safety behavior: TwinPrompt [KrauĂ et al., 2025], SGXSTest [Gupta et al., 2024], AdvBench [Zou et al., 2023], and OR-Bench [Cui et al., 2024]. TwinPrompt and SGXSTest provide paired harmful and benign prompts, allowing us to evaluate both refusal of unsafe requests and over-refusal of safe requests. AdvBench contains harmful instructions only and is used to measure malicious-prompt- refusal. OR-Bench contains benign prompts designed specifically to stress-test over-refusal (Table 1). For each malicious benchmark, we construct evaluation subsets by retaining prompts on which the safety-aligned model refuses while the corresponding base model complies. We keep up to 30 prompts per condition. For the benign subsets used to measure over-refusal, we apply the analogous filtering procedure to the paired benchmarks. Because this filtering is more restrictive for some settings, certain benign subsets contain fewer than 30 examples; we clarify the total number of prompts when reporting results. 5 In addition, we construct a separate paired prompt pool and apply the same alignedâbase disagreement filtering criterion to obtain larger validation subsets that contain 100 malicious and 100 benign prompts after filtering. So they preserve the behavioral contrast used in the primary experiments while testing whether the observed localization patterns persist at a larger sample size. 3.5 Metrics We define a refusal behavior to be an explicit rejection induced by recognized harmful or unethical implications of the request. Refusal on malicious datasets (MR) is higher if a model incorporates better defense mechanisms against malicious use. For benign prompts, we report benign over- refusal (BOR) results, which are the numbers of benign prompts incorrectly turned down; lower BOR is preferred for all models. All model answers are manually inspected. We include generation configurations in section 9. 3.6 Greedy Block Selection To study how refusal-relevant blocks interact, we perform greedy forward selection separately for each model-dataset pair on the malicious benchmarks. Starting from the base model, we iteratively add the MLP block from the remaining set that yields the largest gain in refusal until all blocks are transplanted. Besides refusals, a model can output other acceptable and safe answers, which provide no useful harmful information, even if it is not phrased as an explicit refusal. When multiple candidates tie on refusals, we select the one with more acceptable safe answers. This procedure produces both (i) a dataset-specific importance ordering over blocks and (i) the best-performing block subset at each budgetk. The orders collected from malicious benchmarks are transferred to OR-Bench for generalization testing. In addition, with Block 3 constantly being selected first, we further examine the importance of individual Block 3 layers and their combinations. 4 Results We find three consistent patterns across both model families. First, MLP weights play a more essential role than attention weights in recovering refusal behavior. In addition, refusal-relevant parameters are not uniformly distributed across the MLP stack: Block 3 (layers 8â11) is the most consistently effective individual block for transferring refusal behavior, within which layer 8 stands out. Moreover, greedy block selection reveals a six-block substitution that often outperforms full MLP transplantation on malicious refusal, benign over-refusal, or both. 4.1 Component-Level Hybrids Our first set of analyses compares hybrid models that transplant broad classes of parameters across three malicious-prompt benchmarks to lay the groundwork for finer localizations. We summarize MR and BOR from multiple benchmarks in Table 2. The pattern is consistent across both model pairs: replacing MLP weights recovers substantially more safety behavior than replacing attention weights. Under MLP weight transplantation, hybrid models from the RealSafe-R1-7B family achieve 19, 17, and 27 refusals on 30-malicious-prompt subsets from TwinPrompt, SGXSTest, and AdvBench, respectively, which are the highest MR values among all component-level hybrids. Models from the saferlhf ultrasft family also yield best performance with this configuration. When compared with 6 ModelConfigTwinPromptSGXSTestAdvBenchTwinPrompt BOR RealSafe-R1-7B first51/30 (3.33%)3/30 (10%)2/30 (6.67%)0/30 (0%) mid58/30 (26.67%)2/30 (6.67%)11/30 (36.67%)0/30 (0%) last54/30 (13.33%)2/30 (6.67%)7/30 (23.33%) 0/30 (0%) firsthalf8/30 (26.67%)11/30 (36.67%)10/30 (33.33%)1/30 (3.33%) second half9/30 (30.00%)11/30 (36.67%)18/30 (60.00%)1/30 (3.33%) attn5/30 (16.67%)2/30 (6.67%)10/30 (33.33%)0/30 (0%) mlp19/30 (63.33%) 17/30 (56.67%) 27/30 (90.00%) 11/30 (36.67%) saferlhfultrasft first51/30 (3.33%)0/30 (0%)0/30 (0%)0/5 (0%) mid50/30 (0%)0/30 (0%)0/30 (0%) 0/5 (0%) last50/30 (0%)0/30 (0%)0/30 (0%)0/5 (0%) firsthalf18/30 (60.00%)2/30 (6.67%)10/30 (33.33%)0/5 (0%) second half0/30 (0%)2/30 (6.67%)0/30 (0%)0/5 (0%) attn5/30 (16.67%)0/30 (0%)0/30 (0%)0/5 (0%) mlp21/30 (70.00%)4/30 (13.33%)17/30 (56.67%)0/5 (0%) Table 2: Component-level hybridsâ performance, including MR on three malicious subsets and BOR on benign TwinPrompt prompts. MR results generated by attn hybrids, MLP-only transplantation exceeds by at least 2.7 times across the six comparisons. On the other hand, the same intervention that restores refusal behavior can also import over- refusal. For RealSafe-R1-7B hybrids, full MLP substitution produces 11 benign refusals on Twin- Prompt, whereas attention-only substitution produces none. Proper refusal and false-positive re- fusal are therefore not cleanly separable at the component level in this model family: both are concentrated primarily in the MLP pathway. By contrast, saferlhf ultrasft exhibits no Twin- Prompt over-refusal behavior under full MLP transplantation, suggesting that the aligned signal transferred from saferlhfultrasft may be more selective. Taken together, these results identify MLP weights as the dominant parameter pathway for transferring aligned refusal behavior in both model families. Since MLP substitution clearly dom- inates, we then ask whether safety is distributed uniformly across the MLP stack or concentrated in particular blocks. 4.2 Individual MLP Blocks In this follow-up experiment, we localize the safety signals within the MLP stack by transplanting one block at a time. Tables 3 and 4 report MR results for each single-block substitution. Refusal- relevant effects are distributed across multiple MLP blocks, but Block 3 is the most consistently 7 DatasetB1B2B3B4B5B6B7 L0-3 L4-7 L8-11 L12-15 L16-19 L20-23 L24-27 TwinPrompt2376331 SGXSTest3330323 AdvBench47107955 Table 3: Individual MLP-block substitution results for the RealSafe-R1-7B family. Values represent refusal counts on malicious subsets (30 prompts per subset). DatasetB1B2B3B4B5B6B7B8 L0-3 L4-7 L8-11 L12-15 L16-19 L20-23 L24-27 L28-31 TwinPrompt01200000 SGXSTest00000000 AdvBench00100000 Table 4: Individual MLP-block substitution results for the saferlhfultrasft family. Values represent refusal counts on malicious subsets (30 prompts per subset). important block across benchmarks. On TwinPrompt and AdvBench, Block 3 achieves the highest MR rates for both model families, and on SGXSTest it appears among the top candidates. The recurrence of the same absolute depth range across the Qwen2-7B and Llama-3.1-8B architectures suggests that, for models of this scale, the parameters mediating transferable refusal behavior appear disproportionately concentrated around layers 8â11. The results also reveal that individual MLP blocks in the saferlhf ultrasft family are generally less effective in rejecting unsafe instructions. A potential explanation is that both extent and variety of safety training affect the robustness of knowledge encoding, but verification is left for future research. Tables 3 and 4 measure the effect of each block in isolation; Section 4.3 instead asks which blocks are most useful when added cumulatively. 4.3 Non-monotonic Block Composition We run greedy forward selection to determine how safety-aligned blocks interact when added cu- mulatively. Block 3 is selected first in all six model-dataset pairs (Table 5), consistent with its strong individual performance from previous experiment. Tables 6 and 7 report MR and BOR results at each block budgetk. The main finding is that on two of three benchmarks, a six-block transplantation can outperform the full MLP transplantation on rejecting malicious requests. The greedy trajectories show how this pattern develops across block budgets (Figure 1). For RealSafe-R1-7B on TwinPrompt, MR rises from 7 atk = 1 to 24 atk = 6, then falls to 19 at k = 7. The trajectory of saferlhf ultrasft reaches 12 MR counts atk = 4, drops down to 10 at k = 6, and finally recovers when all eight blocks are present. On SGXSTest, RealSafe-R1-7B peaks at MR = 18 atk = 6 and declines to 17 atk = 7. The drop observed from the saferlhfultrasft family is more pronounced: performance peaks at 8 atk = 6 and falls to 4 after the seventh block is added. AdvBench is the most monotonic setting. For RealSafe-R1-7B, MR increases through k = 5, dips atk = 6, and recovers atk = 7. The saferlhf ultrasft-AdvBench trajectory is the only fully monotonic one among the six, it starts with 1 refusal atk = 1 and gradually grows to 17 rejections. 8 ModelDatasetGreedy Order RealSafe-R1-7B TwinPrompt 3, 5, 2, 6, 4, 7, 1 SGXSTest3, 6, 7, 4, 1, 5, 2 AdvBench3, 6, 5, 1, 2, 7, 4 saferlhfultrasft TwinPrompt 3, 4, 5, 2, 1, 8, 7, 6 SGXSTest3, 7, 2, 4, 5, 1, 6, 8 AdvBench3, 2, 4, 5, 7, 1, 8, 6 Table 5: Greedy block orders on three malicious subsets. Block 3 is selected first in all six settings. Dataset k=1k=2k=3k=4k=5k=6k=7 TwinPrompt MR7/30 (23.33%) 11/30 (36.67%) 16/30 (53.33%) 17/30 (56.67%) 21/30 (70.00%) 24/30 (80.00%) 19/30 (63.33%) BOR0/30 (0%) 1/30 (3.33%) 2/30 (6.67%) 2/30 (6.67%) 5/30 (16.67%) 9/30 (30.00%) 11/30 (36.67%) SGXSTest MR3/30 (10.00%) 6/30 (20.00%) 9/30 (30.00%) 13/30 (43.33%) 17/30 (56.67%) 18/30 (60.00%) 17/30 (56.67%) BOR0/30 (0%) 2/30 (6.67%) 4/30 (13.33%) 3/30 (10.00%) 7/30 (23.33%) 8/30 (26.67%) 8/30 (26.67%) AdvBench MR10/30 (33.33%) 17/30 (56.67%) 20/30 (66.67%) 23/30 (76.67%) 25/30 (83.33%) 24/30 (80.00%) 27/30 (90.00%) Table 6: Greedy trajectories for RealSafe-R1-7B family.k is the number of MLP blocks trans- planted. Greedy orders: TwinPrompt 3-5-2-6-4-7-1; SGXSTest 3-6-7-4-1-5-2; AdvBench 3-6-5-1-2- 7-4. The BOR results further show benefits of using only six blocks of MLP weights. In the RealSafe- R1-7B-TwinPrompt setting, the six-block hybrid achieves MR = 24 with BOR = 9, whereas the full seven-block transplant yields MR = 19 with BOR = 11, which is worse on both metrics. For RealSafe-R1-7B-SGXSTest, full transplantation loses one refusal without reducing over-refusal. For saferlhf ultrasft-SGXSTest, the contrast is sharper: the six-block model achieves MR = 8 with BOR = 1, compared with MR = 4 and BOR = 3 for the full eight-block model. We refer to the pattern where adding an aligned block reduces MR as the negative marginal effect. Five of the six trajectories exhibit such effect, but the disruptive block varies across settings: the first decline on both TwinPrompt trajectories coincides with the introduction of Block 1, whereas the sharp SGXSTest drop for saferlhf ultrasft is associated with Block 6. This dataset dependence stands against a single universally harmful block and instead supports that block utility is contingent on the surrounding set of already-transplanted blocks. 9 Figure 1: Hybrid model performance on benchmark subsets. 4.4 Transfer to OR-Bench is Benchmark-dependent The greedy selection approach offers insights into MR-BOR trade-off with respect to model-dataset pairs. However, the BOR results are collected from datasets with paired prompts that minimize linguistic difference. We further evaluate the models and orders on a separate benign-prompt only benchmark, namely OR-Bench, to determine transferability and whether a specific order outper- forms the others. As a result, the AdvBench-derived order is generally more conservative and often incurs higher OR-Bench over-refusal rates. By contrast, the order derived from SGXSTest tends to yield lower OR-Bench over-refusal than the order derived from AdvBench. For RealSafe-R1-7B, the SGXSTest order attains the lowest or tied-lowest BOR at five of the seven block budgets. For saferlhf ultrasft, the SGXSTest order maintains zero over-refusals through mostk values. Nonethe- less, this comparison should be interpreted cautiously. Regardless of the lower over-refusal values, SGXSTest-derived orders also tend to achieve lower malicious-prompt rejection than orders derived from TwinPrompt or AdvBench. We therefore interpret these results as evidence of a trade-off rather than a strict improvement. The source benchmark influences both how aggressively safety transfers and how much that transfer over-generalizes to benign prompts. A second pattern we discover is that saferlhf ultrasft hybrids remain more precise than RealSafe- R1-7B hybrids under transfer. At high budgets, RealSafe-R1-7B incurs 3-6 BOR atk = 7, depend- ing on the order, whereas saferlhfultrasft incurs 0-1 refusals atk = 7 and reaches only 3 when all MLP blocks are transplanted. This suggests that the aligned signal inherited from saferlhfultrasft is less prone to over-generalization under block composition. Overall, the transfer results demonstrate that greedy block orders learned from different mali- cious benchmarks behave differently on unseen benign prompts, thus, benchmark choice matters for downstream precision. 10 Dataset k=1k=2k=3k=4k=5k=6k=7k=8 TwinPrompt MR2/30 (6.67%) 4/30 (13.33%) 7/30 (23.33%) 12/30 (40.00%) 11/30 (36.67%) 10/30 (33.33%) 12/30 (40.00%) 21/30 (70.00%) BOR0/5 (0%) 0/5 (0%) 0/5 (0%) 0/5 (0%) 0/5 (0%) 0/5 (0%) 0/5 (0%) 0/5 (0%) SGXSTest MR0/30 (0%) 1/30 (3.33%) 1/30 (3.33%) 2/30 (6.67%) 2/30 (6.67%) 8/30 (26.67%) 4/30 (13.33%) 4/30 (13.33%) BOR0/18 (0%) 0/18 (0%) 0/18 (0%) 0/18 (0%) 2/18 (11.11%) 1/18 (5.56%) 0/18 (0%) 3/18 (16.67%) AdvBench MR1/30 (3.33%) 2/30 (6.67%) 4/30 (13.33%) 10/30 (33.33%) 11/30 (36.67%) 12/30 (40.00%) 12/30 (40.00%) 17/30 (56.67%) Table 7: Greedy trajectories for saferlhfultrasft family.k is the number of MLP blocks trans- planted. Greedy orders: TwinPrompt 3-4-5-2-1-8-7-6; SGXSTest 3-7-2-4-5-1-6-8; AdvBench 3-2-4- 5-7-1-8-6. 4.5 A Peek into Block 3 Given the consistently strong performance of Block 3 weights, we build several hybrid models using only Block 3 layers to test if the non-monotonic pattern and marginal effect exist at a finer-grained level. As shown in Table 9, exploratory analysis further points to layer 8 for the RealSafe-R1- 7B family. Layer 8 produces more malicious-prompt refusals than the other individual layers. Transplanting only this aligned layer of weights to the base DeepSeek-R1-Distill-Qwen-7B model gives near or above half of the MR that the full block achieves. The exploratory evidence indicates that transferable refusal behavior is concentrated in this layer within Block 3. Across all three benchmarks, we observe the non-additivity from the combination of layers 8 and 9, along with a few others that are dataset-specific. Another consistency is that the full-block performance can be achieved or even surpassed by a subset of layers. We conduct the same experiments on the saferlhf ultrasft pair. Despite Block 3 being the most important MLP block, it induces at most 2 rejections on the subsets, so the results fail to reveal valuable information and are excluded from display. 4.6 Validation on a Larger Paired Subsets The preceding experiments rely on relatively small filtered subsets, with at most 30 prompts per condition. To test whether the observed localization patterns are robust to a larger sample of alignedâbase disagreement cases, we conduct an additional validation experiment using a newly constructed paired prompt set. We created a list of paired actions, and randomly draw prefixes from another set to form complete instructions. We then followed the same subset construction criterion. For malicious prompts, we retained examples for which the safety-aligned RealSafe- R1-7B model refuses while the corresponding DeepSeek-R1-Distill-Qwen-7B base model complies. For benign prompts, we applied the analogous procedure to identify cases relevant to over-refusal. We constructed a substantially larger pool of candidate prompts, allowing both the malicious and benign subsets to contain 100 prompts after filtering. Therefore the experiment preserves the 11 Model / Orderk=1 k=2 k=3 k=4 k=5 k=6 k=7 RealSafe-R1-7B TwinPrompt order0010243 SGXSTest order0101033 AdvBench order1233146 Model / Orderk=1 k=2 k=3 k=4 k=5 k=6 k=7 k=8 saferlhfultrasft TwinPrompt order00011113 SGXSTest order00000003 AdvBench order01111113 Table 8: OR-Bench over-refusal (out of 30 benign prompts) under different greedy block orders. Lower is better. Bold indicates the best value at each budget within a model family. behavioral contrast central to our transplantation analysis while reducing the sensitivity of the reported results to the small sample sizes. We repeat the transplantation experiments on the RealSafe-R1-7B model family using the same transplantation procedures as in the main experiments. In particular, we examine whether the three principal observations obtained from the original subsets persist at the larger sample size: the dominance of MLP over attention weights in transferring refusal behavior, the concentration of safety-relevant weights in the mid-network MLP blocks, and the non-additive behavior of MLP- block composition. The results in Table 10 strongly reproduce the dominance of the MLP pathway. On the 100 malicious prompts, the mlp hybrid refuses 61 prompts, compared with only 8 for the attn hybrid. The remaining coarse-grained transplantations are substantially weaker: first5, mid5, and last5 each produce 8 refusals, while first half and secondhalf transplantation produce 24 and 44 refusals, respectively. These results are consistent with the original experiments and provide ad- ditional evidence that transferable refusal behavior in the RealSafe-R1-7B family is concentrated primarily in MLP rather than attention weights. The larger benign subset also reproduces the pre- cision trade-off associated with MLP transplantation. The mlp hybrid over-refuses 33 of 100 benign prompts, compared with at most 7 over-refusals from other settings. Thus, the same parameter pathway that transfers the greatest amount of malicious-prompt refusal also transfers substantially more over-refusal, strengthening the evidence that desirable refusal and false-positive refusal are not cleanly separable at the component level in this model family. We further repeat the individual MLP-block experiment on the larger subsets. Block 3 (layers 8â11) still produces the highest malicious-prompt refusal rates among all seven individual blocks, rejecting 13 of 100 malicious prompts, followed by Block 6 with 11 rejections and Block 2 with 8 rejections. The remaining blocks produce between 4 and 7 rejections. On the other hand, Block 3 induces only 1 refusal among the 100 benign prompts (Table 11). The recurrence of Block 3 as the strongest individual MLP block on a substantially larger prompt set is consistent with the mid-network localization observed in the original experiments. Following greedy forward selection, malicious-prompt refusals increase from 13 with one block to 29, 42, 52, 54, and a peak of 64 as the block budget increases fromk = 1 tok = 6. However, transplanting the final remaining MLP block reduces performance from 64 to 61 refusals atk = 7 12 TwinPrompt Layers891011 MR3121 Layers8, 98, 108, 119, 109, 1110, 11 MR274434 Layers8, 9, 108, 9, 118, 10, 119, 10, 11 MR4326 LayersBlock3 MR7 SGXS Layers891011 MR5314 Layers8, 98, 108, 119, 109, 1110, 11 MR304223 Layers8, 9, 108, 9, 118, 10, 119, 10, 11 MR3423 LayersBlock3 MR3 AdvBench Layers891011 MR6545 Layers8, 98, 108, 119, 109, 1110, 11 MR 236444 Layers8, 9, 108, 9, 118, 10, 119, 10, 11 MR85106 LayersBlock3 MR10 Table 9: Refusal behaviors of RealSafe hybrid models constructed using Block 3 layers. All subsets contain 30 prompts. (Table 12). Because the seven-block configuration is equivalent to full MLP transplantation, this result once more demonstrates that transplanting all aligned MLP weights need not maximize refusal behavior. The persistence of a negative marginal effect on a substantially larger prompt set supports the conclusion that refusal-relevant MLP blocks do not compose strictly additively. Overall, the larger-subset validation reproduces all three principal localization patterns for the RealSafe-R1-7B family: MLP weights dominate attention weights in transferring refusal behavior, Block 3 remains the strongest individual MLP block, and selective block composition can outper- form full MLP transplantation. These results reduce the likelihood that the original observations are artifacts of the small evaluation subsets. However, because the validation examples are con- structed using the same alignedâbase disagreement criterion, this experiment establishes robustness to increased sample size within the filtered evaluation distribution rather than generalization to an unfiltered prompt distribution. 13 ModelConfigMR BOR RealSafe-R1-7B first580 mid580 last580 first half244 second half447 attn82 mlp6133 Table 10: Component-level hybridsâ performance on validation subsets. B1B2B3B4B5B6B7 L0-3 L4-7 L8-11 L12-15 L16-19 L20-23 L24-27 MR481377117 BOR0010001 Table 11: Individual MLP-block substitution results for the RealSafe-R1-7B family. 5 Discussion 5.1 Weight-Space Localization of Refusal Behavior Our transplantation experiments indicate that the parameters mediating safety-aligned refusal are not distributed uniformly across the network. Across both model families, transferable refusal behavior is mediated predominantly by the MLP pathway, with a mid-depth region spanning layers 8â11 repeatedly prioritized. Even within the mid-depth region, individual layers bear different levels of refusal-related behavior. This concentration provides a plausible explanation for the brittleness of alignment reported in prior work [Qi et al., 2023, Yang et al., 2023]. Since a relatively small subset of parameters carries a disproportionate share of the refusal signal, targeted updates to that subset can substantially alter safety behavior. Moreover, our transplantation experiments localize where safety-relevant information is stored, rather than how it is computed. In particular, the critical role of MLP layers and the highlighted importance of Block 3 point to a narrow part of the network that merits deeper mechanistic analysis. We plan to combine the present weight-level interventions with activation-level methods, such as activation patching, causal tracing, or neuron-level probing, to characterize the representations and transformations within these blocks that support refusal behavior as a follow-up study. 5.2 Interactions and Precision Trade-offs The greedy trajectories show that aligned blocks do not contribute independently. Adding an additional block can improve refusals in one context while degrading them in another, and the full MLP transplant is often not the best operating point. Thus, MLP-block interactions appear to be non-additive. Another observation is that the same concentration that enables safety transfer can also intro- duce over-refusal. With respect to RealSafe-R1-7B hybrids, for example, the MLP-only transplant simultaneously increases malicious prompt refusal and benign over-refusal. This co-localization means that improving safety precision is not only a matter of transplanting more aligned parame- 14 k=1k=2k=3k=4k=5k=6k=7 MR13294252546461 BOR 12714182333 Table 12: Greedy trajectories for RealSafe-R1-7B family.k is the number of MLP blocks trans- planted. Greedy orders: Malicious Subset 3-5-6-7-2-4-1; Benign Subset 3-4-7-6-1-2-5. ters; it requires identifying and choosing subsets whose interactions preserve the distinction between harmful and benign inputs. More broadly, the findings suggest two concrete directions for alignment research. The first is precision-oriented alignment : rather than optimizing refusal strength alone, future methods should explicitly target the precision-coverage trade-off that becomes visible in our transplantation experiments. The second is distributed robustness: if current safety behavior depends heavily on a small number of MLP blocks, then a promising goal is to develop alignment procedures that encode safety more redundantly across the network, making it less brittle to fine-tuning, editing, or weight replacement. 5.3 Benchmark Choice and Deployment Implications Our OR-Bench transfer results indicate that the benchmark used to derive the greedy order ma- terially affects downstream behavior on unseen benign prompts. Some source benchmarks induce more aggressive safety transfer, which can improve malicious-prompt rejection but also increase over-refusal on benign prompts. Others induce a milder transfer that preserves benign compliance more effectively, although at the cost of lower malicious refusal. In this sense, the observed differ- ences are better understood as a precision-coverage trade-off than as evidence that one benchmark is categorically superior. This trade-off is especially important for deployment. If the goal is to maximize refusal on harmful prompts, a more aggressive order may be preferable even if it incurs higher BOR. On the other hand, if the goal is to preserve usability on benign inputs, a milder order may be preferable, even if it leaves more harmful prompts answered unsafely. In practice, block selection should not be evaluated on malicious-prompt rejection alone: the choice of source benchmark can materially affect both safety strength and over-generalization. 5.4 Limitations and Future Directions This study provides evidence that safety alignment is localized and interaction-sensitive, but several limitations remain. First, our analysis covers only two matched model pairs in the 7B-8B LLM regime. Although the recurrence of the same mid-network region across both pairs is striking, it is not yet clear whether the same pattern holds at other model scales, in denser or sparser architectures, or under different alignment procedures. Extending the transplantation analysis to various model sizes and families would help determine whether the observed localization around layers 8-11 reflects a general property of safety alignment or a scale-specific feature of the models studied in this work. Moreover, we intentionally constructed filtered subsets to sharpen the contrast between safety- aligned models and base models, but this also limits their sizes. Future work should validate the same localization patterns on larger and less filtered evaluation sets, ideally including benchmarks 15 that jointly measure harmful-prompt refusal and benign-prompt compliance within the same test distribution. Our greedy forward selection only approximates the space of useful block combinations. The non-monotonic trajectories already show that block effects are not independent, so more exhaus- tive search strategies may reveal better-performing subsets or more structured interactions. This motivates future studies on combinatorial search, sparse optimization, or learned selection policies that explicitly optimize both malicious refusal and benign non-refusal. Overall, these limitations and opportunities point to a larger research agenda: using localization not only to diagnose where alignment resides, but also to design safety methods that are more interpretable, more precise, and more robust. 6 Conclusion In this study, we investigate where refusal behavior is encoded in safety-aligned language models by utilizing small alignedâbase disagreement prompt sets and transplanting weight matrices into their unaligned counterparts. Our experiments across two model pairs and four benchmarks reveal three consistent empirical patterns that characterize the weight-space localization of safety-aligned refusal behavior: ⢠Pathway Concentration: Transferable refusal behavior is mediated predominantly by the MLP pathway rather than the attention pathway. Replacing MLP weights recovers substan- tially more refusal behavior, outperforming attention-only hybrids by at least 2.7 times in every malicious benchmark setting we studied. ⢠Depth Localization: Within the MLP stack, a specific mid-depth block (Block 3, layers 8â11) consistently emerges as the strongest localized intervention. It is selected first in all six greedy searches, indicating a disproportionate concentration of refusal-mediating parameters at this depth. ⢠Non-Additive Interactions: Refusal-relevant MLP blocks interact non-monotonically. Adding aligned blocks can reduce malicious-prompt refusal, showing that their behavioral effects are not strictly additive. The findings from MLP-block greedy selection suggest that a selective subset of weights can outperform a full-MLP transplant, achieving a more favorable operating point for both safety and model utility. We also observe a similar pattern from performance of Block 3 layersâ transplantation. We emphasize that safety alignment in current LLMs is both localized and interaction-sensitive. Localization helps explain why aligned behavior can be brittle under subsequent fine-tuning or model modification, while the observed interactions show that effective alignment is not simply a matter of transplanting more aligned parameters. In short, our localization analyses help move from broad behavioral observations toward more targeted safety interventions, directing attention to the components of the network most responsible for refusal behavior. 7 AI Use Acknowledgment We have made use of AI tools (Claude Code, ChatGPT, Google Gemini) to assist with coding and proofreading, editing and formatting during the writing. The human authors take full responsibility for the contents of this paper. 16 8 Funding Acknowledgment This work has been supported in part through a grant for the project âTowards Resilient LLM Alignment: Investigating Brittleness & Improving Model Safety by Rank Enhancementâ funded by the USC Amazon Center on Secure and Trusted Machine Learning 9 Appendix Model Generation Configuration for both model families: ⢠Temperature: 0.6 ⢠max newtokens: 1200 (for subset filtering), 2000 (for response generation from hybrid mod- els) References Avinash Amballa, Durga Sandeep Saluru, Gayathri Akkinapalli, Abhishek Sureddy, and Ak- shay Kumar Sureddy. Safe to serve: Aligning instruction-tuned models for safety and helpfulness, 2024. URL https://arxiv.org/abs/2412.00074. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. ArXiv, abs/2406.11717, 2024. URL https://api.semanticscholar.org/CorpusID:270560489. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a. URL https://arxiv.org/abs/2204.05862. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Chris Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, J Landau, Kamal Ndousse, Kamil Ěe LukoËsi Ěut Ěe, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemâi Mercado, Nova Dassarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Thomas Henighan, Tristan Hume, Sam Bowman, Zac Hatfield-Dodds, Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. ArXiv, abs/2212.08073, 2022b. URL https://api.semanticscholar.org/ CorpusID:254823489. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul R Ěottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou.Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. ArXiv, abs/2309.07875, 2023. URL https: //api.semanticscholar.org/CorpusID:261823321. 17 Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kudi- tipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben New- man, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher R Ěe, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tram`er, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Ji- axuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. URL https://arxiv.org/abs/2108.07258. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxiv.org/abs/2005.14165. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?, 2024. URL https://arxiv.org/abs/2306. 15447. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. ArXiv, abs/2405.20947, 2024. URL https://api.semanticscholar. org/CorpusID:270199786. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Prannaya Gupta, Le Qi Yau, Hao Han Low, Isack Lee, Hubert Lim, Yu Xin Teoh, Ji-Hoon Koh, Dar Win Liew, Rishabh Bhardwaj, Rajat Bhardwaj, and Soujanya Poria. Walledeval: A compre- hensive safety evaluation toolkit for large language models. In Conference on Empirical Methods in Natural Language Processing, 2024. URL https://api.semanticscholar.org/CorpusID: 271744807. 18 Aries Huang. saferlhfultrasft. Hugging Face model repository, 2025. URL https://huggingface. co/skyai798/saferlhf_ultra_sft. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, et al. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31983â32016, 2025. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361. Torsten KrauĂ, Hamid Dashtbani, and Alexandra Dmitrienko. Twinbreak: Jailbreaking llm security alignments based on twin prompts. In USENIX Security Symposium, 2025. URL https://api. semanticscholar.org/CorpusID:279250896. Bruce Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L. Dognin, Man- ish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering. ArXiv, abs/2409.05907, 2024. URL https://api.semanticscholar.org/CorpusID:272550481. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730â27744, 2022. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022. URL https://arxiv.org/abs/2202.03286. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! ArXiv, abs/2310.03693, 2023. URL https://api.semanticscholar.org/CorpusID:263671523. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023. Paul R Ěottger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language mod- els. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, 2024. Adam Suma and Sam Dauncey. Deepseek-r1: Incentivizing reasoning capability in llms via rein- forcement learning. ArXiv, abs/2501.12948, 2025. URL https://api.semanticscholar.org/ CorpusID:284488789. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety train- ing fail? ArXiv, abs/2307.02483, 2023. URL https://api.semanticscholar.org/CorpusID: 259342528. 19 Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In International Conference on Machine Learning, 2024. URL https://api.semanticscholar.org/CorpusID:267547755. Tom Wollschlager, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Gunne- mann, and Johannes Gasteiger. The geometry of refusal in large language models: Con- cept cones and representational independence. ArXiv, abs/2502.17420, 2025. URL https: //api.semanticscholar.org/CorpusID:276575573. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Ruth Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. ArXiv, abs/2310.02949, 2023. URL https://api.semanticscholar.org/CorpusID:263620436. Haoyu Zhang, Mohammad Zandsalimy, and Shanu Sushmita. Exposing llm safety gaps through mathematical encoding:new attacks and systematic analysis.2026.URL https://api. semanticscholar.org/CorpusID:287963140. Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, and Yinpeng Dong. Realsafe-r1: Safety-aligned deepseek-r1 without compromising reasoning capability. ArXiv, abs/2504.10081, 2025. URL https://api.semanticscholar.org/CorpusID:277780999. Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Junfeng Sun. Defending large language models against jailbreak attacks via layer-specific editing. ArXiv, abs/2405.18166, 2024. URL https://api. semanticscholar.org/CorpusID:270067915. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable ad- versarial attacks on aligned language models. ArXiv, abs/2307.15043, 2023. URL https: //api.semanticscholar.org/CorpusID:260202961. 20