Paper deep dive
ShallowJail: Steering Jailbreaks against Large Language Models
Shang Liu, Hanyu Pei, Zeyan Liu
Models: Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Qwen3-4B-Instruct
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/11/2026, 1:30:20 AM
Summary
ShallowJail is a novel jailbreak attack against aligned Large Language Models (LLMs) that exploits 'shallow safety alignment' by manipulating initial hidden states during inference. By constructing task-agnostic steering vectors from compliant and refusal prefixes, the method biases the model's early token generation to bypass safety filters, achieving high attack success rates (up to 97%) on models like Llama-3.1-8B and Qwen2.5-7B without requiring additional training.
Entities (6)
Relation Signals (3)
ShallowJail → targets → Large Language Models
confidence 100% · we introduce ShallowJail, a novel attack that exploits shallow alignment in LLMs.
ShallowJail → uses → Activation Steering
confidence 100% · ShallowJail consists of two stages: (1) Steering Vectors Construction... and (2) Jailbreak Prompting utilizes this activation steering vector
ShallowJail → evaluatedon → AdvBench
confidence 95% · Our evaluation of ShallowJail encompasses a diverse set of 1,010 malicious prompts, comprising 520 from AdvBench
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models(LLMs) have been successful in numerous fields. Alignment has usually been applied to prevent them from harmful purposes. However, aligned LLMs remain vulnerable to jailbreak attacks that deliberately mislead them into producing harmful outputs. Existing jailbreaks are either black-box, using carefully crafted, unstealthy prompts, or white-box, requiring resource-intensive computation. In light of these challenges, we introduce ShallowJail, a novel attack that exploits shallow alignment in LLMs. ShallowJail can misguide LLMs' responses by manipulating the initial tokens during inference. Through extensive experiments, we demonstrate the effectiveness of ShallowJail, which substantially degrades the safety of state-of-the-art LLM responses. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.07107
- Canonical: https://arxiv.org/abs/2602.07107
- Code: https://github.com/liuup/ShallowJail
Trouble viewing inline? Open PDF directly →
Full Text
32,098 characters extracted from source content.
Expand or collapse full text
ShallowJail: Steering Jailbreaks against Large Language Models †thanks: ∗ Corresponding Author †thanks: This work is supported in part by x Shang Liu, Hanyu Pei, Zeyan Liu∗ Abstract Large Language Models(LLMs) have been successful in numerous fields. Alignment has usually been applied to prevent them from harmful purposes. However, aligned LLMs remain vulnerable to jailbreak attacks that deliberately mislead them into producing harmful outputs. Existing jailbreaks are either black-box, using carefully crafted, unstealthy prompts, or white-box, requiring resource-intensive computation. In light of these challenges, we introduce ShallowJail, a novel attack that exploits shallow alignment in LLMs. ShallowJail can misguide LLMs’ responses by manipulating the initial tokens during inference. Through extensive experiments, we demonstrate the effectiveness of ShallowJail, which substantially degrades the safety of state-of-the-art LLM responses. Our code is available at https://github.com/liuup/ShallowJail. Disclaimer: This paper contains offensive content that may be disturbing to some readers. I Introduction Large Language Models (LLMs), such as ChatGPT [21] and DeepSeek [6], have recently become transformative technologies, with growing impact on society and everyday work. Owing to their advanced reasoning abilities [33, 35] and agentic behaviors [36], LLMs have been widely adopted across many domains. At the same time, their safe deployment has become a major concern [1]. In practice, LLMs must be carefully regulated to prevent the generation of malicious, toxic, or offensive content [12, 39]. To this end, substantial effort has been devoted to safety alignment, including supervised fine-tuning with Reinforcement Learning from Human Feedback (RLHF) [22] and adversarial red-teaming [23]. Despite these safeguards, aligned LLMs remain vulnerable to jailbreak attacks [34, 13, 25]. Figure 1 illustrates a representative example. Such failures can undermine trust in deployed systems and, in extreme cases, pose serious societal risks. Existing jailbreak attacks can be broadly categorized as white-box or black-box, depending on the attacker’s level of access. In white-box settings, attackers have full access to model weights and gradients, enabling carefully optimized perturbations and highly effective attacks [43, 18, 15]. In contrast, black-box attacks assume no knowledge of the model architecture or weights and typically rely on prompt engineering or semantic manipulation [13, 34, 3, 19]. Figure 1: Standard LLM responses to normal queries versus responses elicited through jailbreaks. User can manipulate the LLM to output malicious response. Figure 2: The two-stage ShallowJail framework. The real tokenization process in each model with be more complex and different. However, current state-of-the-art jailbreak methods suffer from two major limitations. First, white-box approaches, such as GCG [43] and AutoDAN [18], require continuous access to the full token generation process, making them more likely to be detected or mitigated by runtime monitoring and anomaly detection systems [1]. Second, black-box methods rely heavily on manual prompt design, which is labor-intensive. In addition, they often produce prompts that are tightly coupled to a specific target model, limiting their generalizability across different LLMs [16]. In this paper, we propose a new jailbreak method called ShallowJail. Our approach exploits a recently identified property of aligned LLMs known as shallow safety alignment [24], which suggests that safety mechanisms are disproportionately dependent on the initial tokens generated by the model. Building on this observation, we ask the following question: Can an LLM be steered toward producing harmful outputs by manipulating only these shallow tokens? ShallowJail consists of two stages: (1) Steering Vectors Construction computes a task-agnostic feature vector corresponding to a compliant prefix, and (2) Jailbreak Prompting utilizes this activation steering vector to bias the model’s hidden states during generation, guiding it toward unsafe responses. Extensive experiments demonstrate the high effectiveness of ShallowJail. For example, it achieves an attack success rate of exceeding 90%90\% on Qwen2.5-7B-Instruct. In summary, our main contributions can be summarized as follows: • We re-examine shallow safety alignment and show how it can be exploited for jailbreak attacks. • We propose a task-agnostic activation steering method that injects compliance-inducing signals into the model’s prefix without any additional training. • We perform comprehensive evaluations demonstrating that ShallowJail significantly degrades response safety by bypassing existing alignment mechanisms. I Background I-A LLM Jailbreaks Existing methods usually manually design or optimize prompts to misguide LLM output [18, 34, 13, 25, 19]. Prefilling attack [2, 30] and other similar methods [43, 15] explore the effects of adversarial suffix in jailbreaks. To mitigate jailbreak attacks, Llama Guard[12] and Qwen3guard[39] are proposed as pre-trained classifiers for LLM responses. Furthermore, white-box methods such as JBShield [38] identify and manipulate toxic and malicious concepts. Gradient cuff[9] proposes a two-step detection approach that exploits the unique properties of refusal losses, achieving significant improvement. I-B Activation Steering Activation steering is an inference-time intervention that steers LLM outputs by modifying internal hidden states with pre-defined vectors[14, 8]. Previous methods calculate the difference between harmless and harmful response, and move the text[26, 41, 10] or multimodal[32, 31] response into rejection while maintain the utility. Beyond these, activation steering can also be used to prevent personal information leakage [20], control sentiment [29, 7], constrain output formats [28, 5], improve translation [27], and analyze internal mechanisms [40, 42, 17]. I Methodology In this section, we present the ShallowJail, a method that control the initial tokens generation process to enhance the jailbreak. The overall framework is shown at Figure 2, which contains two stages: (1) Steering Vectors Construction, and (2) Jailbreak Prompting. I-A Steering Vectors Construction We first define two sets of prefixes: the Compliance Prefix sets comD_com and the Refuse Prefix sets refD_ref, where represent the majority of standard LLM responses. For each prefix dcom(i)∈comd_com^(i) _com, dref(i)∈refd_ref^(i) _ref, the construction of steering vectors s s can be formulated as follows: s=∑ilen(com)∑jlen(Dref)[LH(dcom(i))−LH(dref(j))]len(com)×len(Dref)s= _i^len(D_com) _j^len(D_ref)[LH(d_com^(i))-LH(d_ref^(j))]len(D_com)× len(D_ref) (1) where the LH(d(i))LH(d^(i)) denotes the set of hidden states of the final token of each layers for the prefix d(i)d^(i). We then apply the normalization as follows: s^=s‖s‖ s= s\|s\| (2) We define the s s as the steering vector, and use it to trigger the jailbreak. In the tokenization process, the last token can represent the complete semantic meaning of the sentence, so we can calculate the difference between comD_com and refD_ref, and find the steering direction for jailbreak. In our experiments, we set the len(com)=len(ref)=10len(D_com)=len(D_ref)=10. The ablation study of the size of comD_com and refD_ref is available at Table IV. Figure 3: The trade-off analysis on AdvBench. We observe that the α, β, τ impact the attack effectiveness. I-B Jailbreak Prompting Following the construction of the steering vectors s s, we incorporate them into the token generation process to guide the model toward a jailbreak response. Specifically, for each generated token tkt_k, the hidden states h(tk)h(t_k) across all layers are modified by adding the steering vector scaled by a strength parameter α and β. The modified hidden state h′(tk)h (t_k) is formulated as follows: h′(tk)=h(tk)+α×s^k≤τh(tk)+α×β×s^k>τh (t_k)= casesh(t_k)+α× s&k≤τ\\ h(t_k)+α×β× s&k>τ cases (3) This intervention is strategically partitioned into two distinct phases: (1) the Shallow Tokens Attack and (2) the Deep Tokens Attack. During the Shallow Tokens Attack phase, which occurs when the token index k is less than or equal to the threshold τ, the steering vector is applied at its primary intensity to directly influence the model’s initial output. By manipulating these early hidden states, ShallowJail forces the LLM to generate “shallow tokens”—compliant prefixes such as “Sure, here are the details” that establish a helpful persona and effectively bypass the safety alignment filters that typically trigger a refusal at the start of a sequence. As the generation progresses into the Deep Tokens Attack phase where k>τk>τ, the steering influence is modulated by an additional coefficient β. This secondary stage is designed to maintain the harmful trajectory initiated by the shallow tokens while ensuring that the generated content remains linguistically fluent and coherent. Our experimental results indicate that while steering deep tokens alone yields a negligible attack success rate, their manipulation in conjunction with shallow tokens provides the most robust jailbreak performance. This confirms the hypothesis that LLM alignment is most vulnerable during the initial generation steps; once the steering vector successfully shifts the semantic direction toward compliance in the shallow phase, the model continues to follow that path even with reduced steering intensity, posing a significant threat to existing safety guards. TABLE I: Performance of ShallowJail on Different Victim Models and Datasets Methods AdvBench Malicious1 Forbidden1 ASR ↑ PPL ↓ ASR ↑ PPL ↓ ASR ↑ PPL ↓ Qwen3-4B-Instruct-2507, τ=150τ=150, α=5.0α=5.0, β=0.5β=0.5 Direct 0.00100.0010 1.30621.3062 0.00500.0050 1.50601.5060 0.02310.0231 1.55521.5552 ShallowJail 0.90190.9019 4.36974.3697 0.79500.7950 4.37754.3775 0.57180.5718 4.66464.6646 Qwen2.5-7B-Instruct, τ=150τ=150, α=6.5α=6.5, β=0.5β=0.5 Direct 0.00480.0048 1.86921.8692 0.04000.0400 1.91731.9173 0.08330.0833 1.96771.9677 ShallowJail 0.86150.8615 4.32084.3208 0.76500.7650 5.56755.5675 0.48720.4872 4.83354.8335 Llama-3.1-8B-Instruct, τ=150τ=150, α=0.8α=0.8, β=0.5β=0.5 Direct 0.06350.0635 1.54041.5404 0.02000.0200 1.53751.5375 0.06410.0641 1.59071.5907 ShallowJail 0.97020.9702 15.654015.6540 0.93500.9350 19.155219.1552 0.58330.5833 14.550814.5508 1Abbreviation for MaliciousInstruct and ForbiddenQuestions. TABLE I: The Comparison of Distinct-2-Gram(D2G) Model AdvBench Malicious Forbidden Natural Qwen3-4B 0.16560.1656 0.16470.1647 0.16290.1629 0.20120.2012 Qwen2.5-7B 0.17910.1791 0.20300.2030 0.20190.2019 0.20090.2009 Llama-3.1-8B 0.34490.3449 0.41320.4132 0.34310.3431 0.22620.2262 IV Experiments Setup IV-A Datasets and Victim Models Our evaluation of ShallowJail encompasses a diverse set of 1,010 malicious prompts, comprising 520 from AdvBench [43], 100 from MaliciousInstruct [11], and 390 from ForbiddenQuestions [25]. We evaluate our experiments primarily using three well-aligned, open-source LLMs.: Qwen3-4B-Instruct-2507111https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507, Qwen2.5-7B-Instruct222https://huggingface.co/Qwen/Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct333https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct, which are publicly downloaded from Huggingface platform. In the inference stage, we set the temperature=0.70.7, max_new_tokens=700700, top_k=0.950.95 and repetition_penalty=1.11.1. Except for the above, all other parameters remain at their default values. IV-B Evaluation Metrics Attack Success Rate(ASR). Following previous works[34, 37, 24], we adopted ASR to evaluate the efficiency of ShallowJail. While previous methods always using keyword-match to judge if the response belongs to the jailbreak[11, 25, 18], we found this method will lead to judgement errors. Therefore, we used Qwen3Guard-Gen-4B444https://huggingface.co/Qwen/Qwen3Guard-Gen-4B to determine the LLM-as-a-Judge. This model will classify the response into three classes: Safe, Controversial, Unsafe. We calculate the ASR by defining these values as 0.00.0, 0.50.5, 1.01.0, respectively. Distinct-2-Gram(D2G). The Distinct-2-Gram (D2G) metric evaluates linguistic richness by calculating the ratio of unique bigrams to the total bigrams generated, represented as D2G=Unique BigramsTotal BigramsD2G= Unique BigramsTotal Bigrams. Higher D2G scores signify greater lexical diversity and reduced repetition, demonstrating that the model produces varied and natural language rather than relying on fixed patterns. In our experiments, we randomly sampled 100 data from AceReason-Math dataset[4], and compare the ShallowJail with the response. Perplexity(PPL) assesses linguistic fluency by calculating the exponentiated average negative log-likelihood of a sequence, where a lower score signifies higher naturalness and alignment with human-like language distributions. This metric ensures generated responses remain stealthy and coherent rather than irregular, with victim models serving as their own evaluators in this experiment. IV-C Implementation Details Our experiments were conducted on single Nvidia H100 NVL(94GB) or H200(141GB) or 5090(32GB), depending on the node availabilities. More environment details will be provided in our open-source repository. V Experiments Results In this section, we aim to explore the ShallowJail by answering this following questions: • RQ1: (Performance) How well does ShallowJail generalize on different victim LLMs and datasets? • RQ2: (Ablation) How does ShallowJail perform under different hyperparameter settings? • RQ3: (Sensitivity Analysis) How do different hyperparameter settings affect the quality of text generation? V-A Main Performance Results (RQ1) ShallowJail significantly compromises the safety alignment of all tested LLMs across diverse datasets. The main experiments results are shown in Table I. For the AdvBench dataset, direct prompting fails to elicit harmful responses because it yields ASR values below 0.070.07, while ShallowJail achieves a dramatic increase in success by reaching an ASRASR of 0.90190.9019 for Qwen3-4B-Instruct-2507, 0.86150.8615 for Qwen2.5-7B-Instruct, and a peak of 0.97020.9702 for Llama-3.1-8B-Instruct. This pattern of vulnerability is mirrored in the MaliciousInstruct results, where ShallowJail maintains high efficacy with ASR scores of 0.79500.7950, 0.76500.7650, and 0.93500.9350 respectively. Beyond ASR, the impact on PPL remains a critical factor in the attack’s stealthiness. Although the application of steering vectors increases the PPL, for example rising from a baseline of 1.54041.5404 to 15.654015.6540 on the Llama-3.1 model, the generated content retains sufficient coherence to satisfy the malicious intent while evading automated refusal mechanisms. Furthermore, the D2G metrics detailed in Table I evaluate linguistic richness and demonstrate that the jailbreak responses maintain varied lexical diversity. For instance, Llama-3.1 exhibits D2G scores of 0.34490.3449 on AdvBench and 0.41320.4132 on MaliciousInstruct, which are notably higher than its natural response D2G of 0.22620.2262, whereas the Qwen models show D2G levels such as 0.16560.1656 and 0.17910.1791 that remain relatively close to their natural distributions. These scores indicate that the model produces varied and natural language instead of relying on fixed patterns. The results indicate that the attack is particularly potent when τ=150τ=150 and β=0.5β=0.5, though the optimal steering strength α varies by model architecture, with Qwen2.5 requiring α=6.5α=6.5 compared to α=0.8α=0.8 for Llama-3.1. These findings strongly support the hypothesis that manipulating the initial shallow tokens is the primary driver of successful jailbreaks because it effectively resets the model’s semantic trajectory toward compliance before the safety alignment can trigger a refusal. V-B Ablation Study (RQ2) ShallowJail can effectively downgrade the response safety by only attacking the shallow tokens. We conduct the experiments on AdvBench with Qwen2.5-7B-Instruct. As shown in Figure 4, the ASR on the Qwen2.5-7B-Instruct model exhibits a significant upward trend as the number of affected shallow tokens (τ) increases from 10 to 250. The steering strength parameter α plays a critical role in this process, as higher α values consistently yield higher ASR across all levels of τ. Specifically, when α=7.5α=7.5, the ASR rapidly climbs and stabilizes near 0.90.9, whereas lower strengths like α=5.0α=5.0 result in a more gradual increase, peaking around 0.50.5 to 0.60.6. Notably, most curves demonstrate a sharp initial rise before reaching a plateau phase once τ exceeds 100100. This saturation effect suggests that the model’s alignment is most vulnerable during the generation of initial shallow tokens, and once the steering vector successfully shifts the semantic direction in this early stage, the jailbreak remains effective even without further manipulation of deeper tokens. Figure 4: Ablation Study on Qwen2.5-7B-Instruct. ShallowJail can increase the ASR by affecting both shallow and deep tokens. As shown in Table I, steering only deep tokens (α=0,β=0.5α=0,β=0.5) results in a very low average ASR, such as 0.00960.0096 for Qwen3 and 0.04310.0431 for Llama. In contrast, steering only shallow tokens (α>0,β=0α>0,β=0) significantly boosts the ASR to 0.59020.5902 and 0.81720.8172 respectively, demonstrating that early token manipulation is the primary driver of the attack. The highest ASR are achieved by combining both, reaching an average of 0.75620.7562 for Qwen3 and 0.82950.8295 for Llama. This confirms that the model’s alignment is most vulnerable during the initial shallow token generation phase, where steering successfully shifts the semantic direction toward compliance. TABLE I: Ablation Study with Different Hyperparameters α β AdvBench Malicious Forbidden Avg Qwen3-4B-Instruct-2507, τ=150τ=150 0 0.50.5 0.00190.0019 0.00500.0050 0.02180.0218 0.00960.0096 5.05.0 0 0.75190.7519 0.62500.6250 0.39360.3936 0.59020.5902 5.05.0 0.50.5 0.90190.9019 0.79500.7950 0.57180.5718 0.75620.7562 Llama-3.1-8B-Instruct, τ=150τ=150 0 0.50.5 0.06060.0606 0.01500.0150 0.05380.0538 0.04310.0431 0.80.8 0 0.95480.9548 0.92000.9200 0.57690.5769 0.81720.8172 0.80.8 0.50.5 0.97020.9702 0.93500.9350 0.58330.5833 0.82950.8295 V-C Sensitivity Analysis (RQ3) The tradeoff between hyperparameters need to be significantly considered. As demonstrated in Figure 3, steering both shallow and deep tokens with α ranging from 3.03.0 to 6.06.0 and β ranging from 0.10.1 to 0.90.9 can effectively enhance the ASR, though achieving this requires careful calibration to avoid compromising text quality. For example, when α is 4.04.0 and τ is 150150, increasing β from 0.30.3 to 0.70.7 results in the ASR rising from 0.21250.2125 to 0.83850.8385. However, maintaining a sufficient steering strength is critical, as a very small α fails to improve the ASR even if β is large, with values staying below 0.10.1 when α is 3.03.0. Ultimately, the selection of these hyperparameters involves a trade-off with linguistic fluency because excessively high values lead to a continuous decrease in the D2G. This reduction in diversity, observed when α is 6.06.0 and β increases toward 0.90.9, reflects an increase in repetitive blocks that makes the resulting jailbreak content unavailable for practical use. In most of the case, increasing the size of D can improve ASR. As shown in Table IV, total prefix combinations increasing from 9 to 100 lead to average ASR gains from 0.6716 to 0.8295 for Qwen3, while perform between 0.8295 and 0.8352 for Llama-3.1 with error 0.69%. These results confirm that larger prefix sets enable more accurate steering of the model’s hidden states toward compliant responses by providing a more robust estimation of the boundary between refusal and helpfulness. TABLE IV: Comparison for Different Prefix Sets Size comD_com refD_ref Total AdvBench Malicious Forbidden Avg Qwen3-4B-Instruct-2507, τ=150τ=150, α=5.0α=5.0, β=0.5β=0.5 33 33 99 0.81630.8163 0.71000.7100 0.48850.4885 0.67160.6716 55 55 2525 0.87880.8788 0.77500.7750 0.53850.5385 0.73080.7308 88 88 6464 0.89230.8923 0.72500.7250 0.52310.5231 0.71350.7135 1010 1010 100100 0.90190.9019 0.79500.7950 0.57180.5718 0.75620.7562 Llama-3.1-8B-Instruct, τ=150τ=150, α=0.8α=0.8, β=0.5β=0.5 33 33 99 0.99040.9904 0.87000.8700 0.62820.6282 0.82950.8295 55 55 2525 0.97020.9702 0.94500.9450 0.59490.5949 0.83520.8352 88 88 6464 0.98170.9817 0.87000.8700 0.61410.6141 0.82190.8219 1010 1010 100100 0.97020.9702 0.93500.9350 0.58330.5833 0.82950.8295 VI Conclusion and Future Works In this paper, we introduced ShallowJail to demonstrate that manipulating initial hidden states can effectively bypass LLM safety alignment. Our experiments show that ShallowJail achieves an ASR up to 0.9702 on Llama-3.1-8B while maintaining high linguistic diversity as evidenced by D2G scores that often surpass natural response levels. These findings confirm that safety alignment is critically vulnerable during the generation of the first few shallow tokens. Consequently, this work highlights the urgent need for more robust defense mechanisms that persist throughout the entire generation process. In the future, we will build up the adaptive steering method to trigger the jailbreak. Ethical Consideration We conducted all experiments using publicly available datasets and strictly controlled the jailbreak experiments only at local machine. We did not utilize our method to target specific individuals or deployed public or commercial systems. Acknowledgment The part of sentences in this paper was polished by Google Gemini. The authors wish to thank the reviewers for their helpful comments and suggestions. This work is supported in part by [Anonymous]. References [1] G. Alon and M. Kamfonas (2023) Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132. Cited by: §I, §I. [2] M. Andriushchenko, F. Croce, and N. Flammarion (2024) Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151. Cited by: §I-A. [3] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong (2025) Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), p. 23–42. Cited by: §I. [4] Y. Chen, Z. Yang, Z. Liu, C. Lee, P. Xu, M. Shoeybi, B. Catanzaro, and W. Ping (2025) AceReason-nemotron: advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400. Cited by: §IV-B. [5] Y. Chen, H. Jhamtani, S. Sharma, C. Fan, and C. Wang (2024) Steering large language models between code execution and textual reasoning. arXiv preprint arXiv:2410.03524. Cited by: §I-B. [6] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. Cited by: §I. [7] J. Z. He, S. Pandey, M. L. Schrum, and A. Dragan (2024) Context steering: controllable personalization at inference time. arXiv preprint arXiv:2405.01768. Cited by: §I-B. [8] E. Hernandez, B. Z. Li, and J. Andreas (2023) Inspecting and editing knowledge representations in language models. arXiv preprint arXiv:2304.00740. Cited by: §I-B. [9] X. Hu, P. Chen, and T. Ho (2024) Gradient cuff: detecting jailbreak attacks on large language models by exploring refusal loss landscapes. Advances in Neural Information Processing Systems 37, p. 126265–126296. Cited by: §I-A. [10] X. Hu, P. Chen, and T. Ho (2025) Token highlighter: inspecting and mitigating jailbreak prompts for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 27330–27338. Cited by: §I-B. [11] Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen (2023) Catastrophic jailbreak of open-source llms via exploiting generation. arXiv preprint arXiv:2310.06987. Cited by: §IV-A, §IV-B. [12] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. (2023) Llama guard: llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674. Cited by: §I, §I-A. [13] F. Jiang, Z. Xu, L. Niu, Z. Xiang, B. Ramasubramanian, B. Li, and R. Poovendran (2024) Artprompt: ascii art-based jailbreak attacks against aligned llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15157–15173. Cited by: §I, §I, §I-A. [14] A. Korznikov, A. Galichin, A. Dontsov, O. Y. Rogov, I. Oseledets, and E. Tutubalina (2025) The rogue scalpel: activation steering compromises llm safety. arXiv preprint arXiv:2509.22067. Cited by: §I-B. [15] H. Liang, Y. Sun, Y. Cai, J. Zhu, and B. Zhang (2025) Jailbreaking llms’ safeguard with universal magic words for text embedding models. arXiv preprint arXiv:2501.18280. Cited by: §I, §I-A. [16] R. Lin, B. Han, F. Li, and T. Liu (2025) Understanding and enhancing the transferability of jailbreaking attacks. arXiv preprint arXiv:2502.03052. Cited by: §I. [17] Y. Lin, P. He, H. Xu, Y. Xing, M. Yamada, H. Liu, and J. Tang (2024) Towards understanding jailbreak attacks in llms: a representation space analysis. arXiv preprint arXiv:2406.10794. Cited by: §I-B. [18] X. Liu, N. Xu, M. Chen, and C. Xiao (2023) Autodan: generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451. Cited by: §I, §I, §I-A, §IV-B. [19] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi (2024) Tree of attacks: jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems 37, p. 61065–61105. Cited by: §I, §I-A. [20] K. K. Nakka, X. Jiang, D. Usynin, and X. Zhou (2025) PII jailbreaking in llms via activation steering reveals personal information leakage. arXiv preprint arXiv:2507.02332. Cited by: §I-B. [21] OpenAI (2025)Introducing gpt-5.2(Website) External Links: Link Cited by: §I. [22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §I. [23] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving (2022) Red teaming language models with language models. arXiv preprint arXiv:2202.03286. Cited by: §I. [24] X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson (2024) Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946. Cited by: §I, §IV-B. [25] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang (2024) ” Do anything now”: characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 1671–1685. Cited by: §I, §I-A, §IV-A, §IV-B. [26] L. Sheng, C. Shen, W. Zhao, J. Fang, X. Liu, Z. Liang, X. Wang, A. Zhang, and T. Chua (2025) AlphaSteer: learning refusal steering with principled null-space constraint. arXiv preprint arXiv:2506.07022. Cited by: §I-B. [27] N. Stoehr, K. Du, V. Snæbjarnarson, R. West, R. Cotterell, and A. Schein (2024) Activation scaling for steering and interpreting language models. arXiv preprint arXiv:2410.04962. Cited by: §I-B. [28] A. Stolfo, V. Balachandran, S. Yousefi, E. Horvitz, and B. Nushi (2024) Improving instruction-following in language models through activation steering. arXiv preprint arXiv:2410.12877. Cited by: §I-B. [29] A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid (2023) Steering language models with activation engineering. arXiv preprint arXiv:2308.10248. Cited by: §I-B. [30] J. Vega, I. Chaudhary, C. Xu, and G. Singh (2023) Bypassing the safety training of open-source llms with priming attacks. arXiv preprint arXiv:2312.12321. Cited by: §I-A. [31] H. Wang, G. Wang, and H. Zhang (2025) Steering away from harm: an adaptive approach to defending vision language model against jailbreaks. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29947–29957. Cited by: §I-B. [32] P. Wang, D. Zhang, L. Li, C. Tan, X. Wang, M. Zhang, K. Ren, B. Jiang, and X. Qiu (2024) Inferaligner: inference-time alignment for harmlessness through cross-model guidance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 10460–10479. Cited by: §I-B. [33] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: §I. [34] Z. Wei, Y. Liu, and N. B. Erichson (2024) Emoji attack: enhancing jailbreak attacks against judge llm detection. arXiv preprint arXiv:2411.01077. Cited by: §I, §I, §I-A, §IV-B. [35] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan (2023) Tree of thoughts: deliberate problem solving with large language models. Advances in neural information processing systems 36, p. 11809–11822. Cited by: §I. [36] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: §I. [37] X. Zeng, S. Liang, L. Lu, H. Zhu, E. Liu, J. Dang, Y. Zhou, and S. Pang (2025) SafeSteer: adaptive subspace steering for efficient jailbreak defense in vision-language models. arXiv preprint arXiv:2509.21400. Cited by: §IV-B. [38] S. Zhang, Y. Zhai, K. Guo, H. Hu, S. Guo, Z. Fang, L. Zhao, C. Shen, C. Wang, and Q. Wang (2025) Jbshield: defending large language models from jailbreak attacks through activated concept analysis and manipulation. arXiv preprint arXiv:2502.07557. Cited by: §I-A. [39] H. Zhao, C. Yuan, F. Huang, X. Hu, Y. Zhang, A. Yang, B. Yu, D. Liu, J. Zhou, J. Lin, et al. (2025) Qwen3guard technical report. arXiv preprint arXiv:2510.14276. Cited by: §I, §I-A. [40] H. Zhao, H. Zhao, B. Shen, A. Payani, F. Yang, and M. Du (2024) Beyond single concept vector: modeling concept subspace in llms with gaussian distribution. arXiv preprint arXiv:2410.00153. Cited by: §I-B. [41] W. Zhao, J. Guo, Y. Hu, Y. Deng, A. Zhang, X. Sui, X. Han, Y. Zhao, B. Qin, T. Chua, et al. (2025) AdaSteer: your aligned llm is inherently an adaptive jailbreak defender. arXiv preprint arXiv:2504.09466. Cited by: §I-B. [42] A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks (2024) Improving alignment and robustness with circuit breakers. Advances in Neural Information Processing Systems 37, p. 83345–83373. Cited by: §I-B. [43] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: §I, §I, §I-A, §IV-A.