Paper deep dive
AdvPrefix: An Objective for Nuanced LLM Jailbreaks
Sicheng Zhu, Brandon Amos, Yuandong Tian, Chuan Guo, Ivan Evtimov
Models: Llama-3
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:37:56 PM
Summary
AdvPrefix is a novel, plug-and-play prefix-forcing objective for LLM jailbreaking that addresses the limitations of traditional objectives (like 'Sure, here is...'). By selecting model-dependent prefixes based on high prefilling attack success rates and low negative log-likelihood, AdvPrefix mitigates misspecification and overconstraint, significantly improving nuanced attack success rates on modern LLMs like Llama-3 and Gemma-2.
Entities (6)
Relation Signals (3)
AdvPrefix → improves → GCG
confidence 95% · Integrating AdvPrefix into GCG and AutoDAN significantly increases nuanced ASR
AdvPrefix → mitigates → Misspecification
confidence 90% · AdvPrefix integrates seamlessly into existing jailbreak attacks to mitigate the previous limitations for free.
AdvPrefix → targets → Llama-3
confidence 90% · replacing GCG’s default prefixes on Llama-3 improves nuanced attack success rates
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many jailbreak attacks on large language models (LLMs) rely on a common objective: making the model respond with the prefix ``Sure, here is (harmful request)''. While straightforward, this objective has two limitations: limited control over model behaviors, yielding incomplete or unrealistic jailbroken responses, and a rigid format that hinders optimization. We introduce AdvPrefix, a plug-and-play prefix-forcing objective that selects one or more model-dependent prefixes by combining two criteria: high prefilling attack success rates and low negative log-likelihood. AdvPrefix integrates seamlessly into existing jailbreak attacks to mitigate the previous limitations for free. For example, replacing GCG's default prefixes on Llama-3 improves nuanced attack success rates from 14% to 80%, revealing that current safety alignment fails to generalize to new prefixes. Code and selected prefixes are released at this http URL.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
56,186 characters extracted from source content.
Expand or collapse full text
AdvPrefix: An Objective for Nuanced LLM Jailbreaks Sicheng Zhu ∗ University of Maryland, College Park sczhu@umd.edu Brandon Amos FAIR, Meta bda@meta.com Yuandong Tian FAIR, Meta yuandong@meta.com Chuan Guo † FAIR, Meta chuanguo@meta.com Ivan Evtimov † FAIR, Meta ivanevtimov@meta.com Abstract Many jailbreak attacks on large language models (LLMs) rely on a common objective: making the model respond with the prefix “Sure, here is (harmful request)”. While straightforward, this objective has two limitations: limited control over model behaviors, yielding incomplete or unrealistic jailbroken responses, and a rigid format that hinders optimization. We introduce AdvPrefix, a plug-and- play prefix-forcing objective that selects one or more model-dependent prefixes by combining two criteria: high prefilling attack success rates and low negative log-likelihood. AdvPrefix integrates seamlessly into existing jailbreak attacks to mitigate the previous limitations for free. For example, replacing GCG’s default prefixes on Llama-3 improves nuanced attack success rates from 14% to 80%, revealing that current safety alignment fails to generalize to new prefixes. Code and selected prefixes are released in github.com/facebookresearch/jailbreak-objectives. Warning: This paper includes language that could be considered inappropriate or harmful. 1 Introduction The rapid advancement of Large Language Models (LLMs) [OpenAI, 2023, Dubey et al., 2024, Anthropic, 2024, Reid et al., 2024] brings escalating AI safety concerns, as LLMs can replicate harmful behaviors from their training data [Vidgen et al., 2024]. Developers mitigate these risks through safety alignment [Ouyang et al., 2022, Bai et al., 2022, Dai et al., 2023] and system- level moderation [Inan et al., 2023, Zeng et al., 2024a], verified by proactive red-teaming that uses adversarial prompts to circumvent these safety measures (i.e., jailbreaking). While jailbreaks traditionally rely on manual prompting by experts [Ganguli et al., 2022], automated jailbreaks using prompt optimization have emerged as a more scalable and effective approach, potentially revealing issues overlooked by human experts [Perez et al., 2022, Lapid et al., 2023, Lin et al., 2025]. A critical component of automated jailbreaks is their optimization objective. Many attacks [Zou et al., 2023, Liu et al., 2023a, Andriushchenko et al., 2024, Wu et al., 2025] aim to elicit a generic prefix like “Sure, here is [harmful request]”. However, this common objective suffers from two key limitations: Misspecified: even with low optimization loss, actual responses are often incomplete or unfaithful (Figure 1), failing to produce genuinely harmful content [Vidgen et al., 2024]. Overconstrained: rigid, handcrafted prefixes can be unnatural for LLMs (e.g., Llama-3 prefers “Here...” rather than “Sure...”), complicating optimization [Zhang et al., 2025b]. These issues are particularly acute for ∗ Work done at Meta † Joint last author 39th Conference on Neural Information Processing Systems (NeurIPS 2025). arXiv:2412.10321v2 [cs.LG] 27 Dec 2025 newer LLMs like Gemma-2, where standard attacks like GCG [Zou et al., 2023] achieve trivial attack success rates (∼10%). Figure 1: (Top) For a malicious request, the original objective maximizes the output likelihood of a rigid prefix (gray) across all victim LLMs. Even with capable optimization algorithms, this objective often leads to refusals or responses that are not genuinely harmful. Our objective uses one (purple) or multiple (light purple) pre-selected prefixes, leading to significantly higher ASR and response harmfulness. (Bottom) The pipeline for generating our prefixes using uncensored LLMs and selecting model-dependent prefixes based on two criteria. While recent works explore alternative jailbreak objectives [Jia et al., 2024, Xie et al., 2024, Zhou and Wang, 2024, Thompson and Sklar, 2024, Sclar et al., 2025, Zhang et al., 2025a], systematically addressing misspecification and overconstraint remains challenging, hindered by difficulties in estimating autoregressive model’s rare behaviors [Jones et al., 2025] and by hard token constraints in jailbreak threat models. In this paper, we propose AdvPrefix, an adaptive prefix-forcing objective that addresses these limitations. Our contributions are as follows: Nuanced evaluation (§2). We first meta-evaluate three existing jailbreak evaluation methods [Mazeika et al., 2024, Souly et al., 2024, Chao et al., 2024], counting only complete and faithful responses as successful jailbreaks (Figure 6). We find that while StrongReject [Souly et al., 2024] is relatively accurate, others can overestimate attack success rates (ASR) by up to 30% (Figure 2). We then refine evaluation by developing an improved judge and a preference-based judge to better capture nuanced harmfulness, which reveals that the original objective is both misspecified and overconstrained (§3). New objective (§4). We propose a new prefix-forcing objective that uses model-dependent prefixes selected based on two criteria: high prefilling ASR (to ensure they lead to complete and faithful harmful responses, reducing misspecification) and low initial negative log-likelihood (NLL) (to ensure they are easy to elicit, mitigating overconstraints). The objective also supports using multiple target prefixes for a single request to further simplify optimization. Our approach includes an automatic pipeline for selecting these prefixes from either rule-based constructions or uncensored LLMs (not necessarily the uncensored target LLM), while seamlessly integrating into existing attacks. Empirical findings (§5). Integrating AdvPrefix into GCG and AutoDAN [Zhu et al., 2023] signif- icantly increases nuanced ASR across Llama-2, 3, 3.1, and Gemma-2. For instance, GCG’s ASR on Llama-3 improves from 16% to 70%, highlighting that current safety alignments struggle to generalize to unseen prefixes. By addressing misspecification, AdvPrefix uniquely benefits from stronger optimization, enabling further ASR gains (to 80% on Llama-3 with full prompt optimization). Preference evaluations also show responses elicited by our objective are substantially more harmful (comparable to some uncensored LLMs) 3 . Our objective improves jailbreak attacks for free, enables attacking reasoning LLMs (Figure 9), and is useful for future red-teaming. 3 Even though we observe improvements in the meaningfulness of model responses when jailbroken with GCG using our targets, we note that none of the models responded with materially harmful information that would not be found on the broader internet. 2 Figure 2: Meta-evaluation of common judges based on 800 manually labeled request-response pairs, using human evaluation as ground truth. (Left) ASRs across different victim LLMs. Existing judges overestimate ASRs, particularly on Llama-3 and Gemma-2. (Center) False positive rates of judges across different failure case categories. (Right) Average human agreement rates of judges across four victim LLMs. Model-wise ASR and F1 scores appear in Table 4. 2 Refined Evaluation for Nuanced Jailbreaks This section shows that current jailbreak evaluations often overestimate ASRs for nuanced jailbreaks by miscounting incomplete and unfaithful responses, and presents our refined evaluation. Defining Nuanced Jailbreaks. For a nuanced jailbreak to succeed, the victim LLM’s response to the harmful request must be affirmative, complete, and faithful (i.e., on-topic, detailed, and realistic, per Vidgen et al. [2024]). Responses failing these criteria, categorized by rules below (examples in Figure 6 and Table 5), represent failed jailbreaks. We meta-evaluate some common jailbreak judges, including HarmBench [Mazeika et al., 2024], JailbreakBench [Chao et al., 2024], and StrongReject [Souly et al., 2024]. We curate 50 highly harmful, non-ambiguous requests from AdvBench as our dataset, and use 800 manually labeled GCG attack responses on Llama-2, 3, 3.1, and Gemma-2 as ground truth (details in Section C, nuanced labeling refers to Vidgen et al. [2024], labeled data released in our codebase). Evaluation Challenges with Newer LLMs. Newer LLMs often exhibit deeper alignment [Qi et al., 2024], tending to self-correct after an initial affirmative prefix rather than directly refusing [Zhang et al., 2024] (Section A). This behavior, emphasizing incomplete or unfaithful responses over outright refusals, exacerbates inaccuracies in existing judges. As shown in Figure 2 (left), current judges can significantly overestimate ASR (e.g., from a 10% ground truth to nearly 40% on Gemma-2), with StrongReject being relatively more accurate. This overestimation stems primarily from misjudging incomplete and unfaithful responses (Figure 2, center), likely because these judges were developed on older LLMs that predominantly either refused directly or produced clearly harmful content (Figure 3, left). Our Improved Judges. To address these inaccuracies, we develop a refined judge using Llama- 3.1-70B, with revised instructions prioritizing response completeness and faithfulness, requiring reasoning traces before giving the final answer [Kojima et al., 2022], and affirmative prefilling to handle sensitive content (details in Section C and codebase). This judge improves human agreement rates by up to 9% on newer LLMs (Figure 2, right; Table 4). Additionally, we introduce a preference judge for relative harmfulness comparison against uncensored LLMs’ output. 3 Limitations of Original Objective This section details how the commonly used prefix-forcing jailbreak objective is misspecified and overconstrained, hindering nuanced jailbreaks, as revealed by our refined evaluation. 3 Figure 3: (Left) The attack failure rates for running GCG with the original objective, along with their breakdown. While the failure rate is roughly90%across all four LLMs, the specific failure cases vary significantly. (Center) Frequency of failure cases by the final loss of the original objective. While attack prompts with lower loss avoid direct refusal, the overall failure rate remains above80% due to increases in the other two failure categories. (Right) Even with prefilling the victim LLM’s initial response with “Sure, here is [request]”, the completed responses’ failure rates remain high. 3.1 Revisiting Original and Oracle Objectives. We first formulate the jailbreak problem. LetV be the LLM’s vocabulary andV ∗ the set of all finite sequences overV. A user prompt isx∈V ∗ , and a model response isy ∈V ∗ . The threat model in jailbreaking allows altering the attack promptθ ∈V ∗ (often a suffix, but sometimes the entire prompt) to steer the victim LLM’s behavior (output distribution). We use⊕for sequence concatenation. A jailbreak judger(x,y)assigns1ifymeets nuanced jailbreak standards forx, and0otherwise, with Y x ≜y : r(x,y) = 1 being the set of all harmful responses for x. Oracle objective. Our ultimate goal is to find an attack promptθ ∈V ∗ that maximizes the likelihood of the victim LLM generating any response inY x : min θ∈V ∗ − log X y∈Y x p(y | θ),(3.1) where the sum represents this likelihood. This log-sum-probability form (distinct from sum-log- probability in other contexts like multi-prompt universal jailbreaking) precisely specifies all desired attack prompts and the model behaviorsp(y | θ)they parameterize, but is prohibitively costly to compute asY x is typically vast. Prefix-forcing objective. This common objective aims to findθthat maximizes the likelihood of generating a specific prefix y p ∈V ∗ : min θ∈V ∗ − log p(y p | θ).(3.2) This is equivalent to maximizing the likelihood of any full response starting withy p , sincep(y p | θ) = P y c ∈V ∗ p(y p ⊕y c | θ) . As such responses include those inY x , this serves as a surrogate for eliciting harmful responses, less overconstrained than eliciting a specific full response. However, by also encompassing non-jailbroken responses (e.g., incomplete, unfaithful), it is prone to misspecification or objective hacking [Amodei et al., 2016], as we show next. 3.2 Two Limitations We identify two limitations of the original objective using our refined evaluation: Misspecified. Lowering the original objective’s loss (Equation (3.2)) does not consistently increase ASRs for nuanced jailbreaks (Figure 3, center): while direct refusals decrease, incomplete and unfaithful responses rise, keeping overall ASR low. To further isolate the objective from optimization algorithm influence, we directly prefill the target prefix [Zhang et al., 2023, Haizelab, 2024, Qi et al., 2024] and observe completions. Figure 3 (right) shows that while prefilling eliminates direct refusals, ASRs remain low due to persistent incomplete and unfaithful completions. This underscores the original objective’s misspecification. 4 Figure 4: The pipeline of constructing our objective. (Left) We use rule-based templates or uncensored LLMs (not necessarily the uncensored target LLM) to generate candidate prefixes. (Center) We evaluate each candidate prefix based on two criteria: high prefilling ASR and low initial NLL. (Right) We select top prefixes (top two in this example) to construct our multi-prefix objective. Overconstrained. The hard token or fluency constraints in jailbreak tasks hinder optimization from lowering the loss [Jain et al., 2023]. Thus, a suitable objective should be easy to optimize. However, the original objective enforces rigid, manually crafted prefixes across all victim LLMs, even if these prefixes misalign with an LLM’s natural response style (e.g., Llama-3 rarely starts with “Sure”, preferring “Here”). Forcing such unnatural prefixes complicates optimization. Indeed, replacing “Sure, here is” with “Here is” in GCG attacks on Llama-3 leads to consistently lower final losses (with equal or improved ASRs, Figure 8), demonstrating that the original objective is overconstrained. 4 The Objective for Nuanced Jailbreaks We introduce AdvPrefix, our new prefix-forcing objective for nuanced jailbreaks, outlined in Figure 4. This section formulates the objective, details its prefix selection criteria, and describes the automatic prefix generation pipeline. 4.1 Selective Multi-Prefix Objective Given a harmful requestx, we select a set of target prefixesY p . AdvPrefix then aims to find an attack prompt θ minimizing the negative log-likelihood of generating any of these prefixes: min θ∈V ∗ − log X y p ∈Y p p(y p | θ).(4.1) Using multiple prefixes leverages the jailbreak task’s flexibility to alleviate overconstraints (e.g., accepting “Here is a guide...” or “Here’s a comprehensive guide...”).. The tree attention trick [Cai et al., 2024], which concatenates multiple prefixes into one, enables efficient computation for multiple prefixes in one forward pass. Section A discusses why we use the prefix-forcing objective and its relationship to model-distillation-based objectives. 4.2 Prefix Selection Criteria To address the original objective’s limitations, we propose two criteria for prefix selection: Criterion I: high prefilling ASR. To reduce misspecification, we want prefixesy p that, once elicited by some attack prompt θ, lead to complete and faithful harmful continuations with high probability: max y p E y c ∼P(·|θ,y p ) r(x,y p ⊕ y c ) .(4.2) Directly computing this value is infeasible as the optimizedθis unknown without time-consuming optimization. However, we observe that this expectation can be efficiently approximated by using a manually constructed attack prompt forθ. Although this manual prompt often cannot elicit the target prefix, the resulting approximated value (prefilling ASR) correlates with the actual jailbreak ASR (Figure 7). We use this approximation to compute the prefilling ASRs. 5 Criterion I: low initial NLL. To reduce overconstraints, we want prefixes that are easily elicited by optimized attack prompts. Since ease of elicitation by an optimized attack prompt is indicated by low NLL, we favor prefixesy p that exhibit a low NLL with the initial (pre-optimization) attack promptθ 0 : min y p − logp(y p | θ 0 )(4.3) These two criteria often conflict. For example, longer prefixes may have higher prefilling ASR but also higher NLL, causing optimization to fail. We balance them using a weighted sum of log- prefilling-ASR and NLL, with weighting tunable to the optimization method’s strength: for example, stronger methods like GCG can prioritize high prefilling ASR and tolerate relatively high NLL. 4.3 Prefix Selection Pipeline We develop an automated pipeline to generate and select target prefixes, typically run once per victim LLM and malicious request, allowing offline storage and reuse. The pipeline involves four steps: 1. Candidate generation. We use rule-based construction or uncensored LLMs with guided decoding [Zhao et al., 2024] to generate candidate prefixes. The uncensored LLMs are not necessarily the uncensored victim LLM, and can be selected from publicly available LLMs that are unaligned (base or helpful-only), finetuned on harmful data, or with refusal suppression [Labonne, 2024]. Guided decoding makes the output more natural for the victim LLM, achieving lower NLL. We generate diverse candidates of varied lengths for each request. 2. Preprocessing. We preprocess candidate prefixes through rule-based augmentation (e.g., “Here is” to “Here’s”, similar to Zou et al. [2023]) to diversify them, and filtering to remove duplicates and any prefixes starting with refusals. 3. Evaluation with two criteria. We first evaluate the initial NLLs of all candidate prefixes using the victim LLM. Then, we estimate their prefilling ASRs by having the victim LLM complete each prefix multiple times (with temperature one) and using our nuanced judge to assess the harmfulness of completions. This evaluation is tailored to both the victim LLM and the judge, where the judge reflects the attacker’s labeling standards. 4. Selection. We combine the two criteria via a weighted sum and rank these candidates. To selectk prefixes, we first identify the top one prefix as a reference, and then select the topkprefixes with a prefilling ASR no lower than that of the reference prefix. 5 Experiments This section incorporates our objective into existing jailbreak attacks to demonstrate its effectiveness in achieving nuanced jailbreaks, comparing it against the original objective. Jailbreak attacks. We employ GCG [Zou et al., 2023], a search-based optimization method, and AutoDAN [Zhu et al., 2023], which combines search with guided decoding. Both attacks primarily rely on the optimization objective, with minimal influence from manual prompting. For each run, we select the attack prompt yielding the lowest objective loss. Threat models. We consider two threat models: (1) Optimizing only the attack suffix, which is then appended to the malicious request. (2) Optimizing the full attack prompt from scratch (without the original request) [Guo et al., 2024], a less restrictive threat model that often leads to unfaithful responses with the original objective. Attack settings. We test four victim LLMs: Llama-2-7B-chat-hf [Touvron et al., 2023], Llama-3-8B- Instruct, Llama-3.1-8B-Instruct [Dubey et al., 2024], and Gemma-2-9B-it [Team et al., 2024]. We use the50malicious requests curated from AdvBench (Section C), and run both attacks for 1000 steps with a batch size of 512. Prefix selection. We generate candidate prefixes using four uncensored LLMs publicly available on Huggingface: georgesung/llama2-7b-chat-uncensored, Orenguteng/Llama-3-8B-Lexi-Uncensored, Orenguteng/Llama-3.1-8B-Lexi-Uncensored, and TheDrummer/Tiger-Gemma-9B-v1. We estimate the prefilling ASR by averaging over 25 random completions (temperature 1) for each prefix. We combine the two selection criteria with a fixed weight of 20 for log-prefilling-ASR. We select four prefixes for our multiple-prefix objective. 6 Table 1: Jailbreak results of GCG with the original objective and our objectives. Here we use GCG to generate the attack suffix and vary the attack suffix length: 20 tokens (black) and 40 tokens (blue). ModelObjective Successful Attack (%,↑) Failed Attack (%,↓) Direct Refusal IncompleteUnfaithful Llama-2 7B-Chat Original13.0 (26.1)72.3 (49.7)0.0 (0.0)14.6 (24.1) Ours Single24.0 (38.6)70.0 (53.5)0.0 (0.0)6.0 (7.9) Ours Multiple 26.0 (37.5)68.0 (52.1)0.0 (0.0)6.0 (10.4) Llama-3 8B-Instruct Original12.8 (16.4)45.6 (37.3)22.1 (21.8)19.5 (24.5) Ours Single 54.6 (69.7)23.7 (12.1)4.1 (3.0)17.5 (15.2) Ours Multiple54.0 (70.0)26.0 (14.0)1.0 (2.0)19.0 (14.0) Llama-3.1 8B-Instruct Original16.8 (16.5)48.3 (48.8)16.8 (17.3)18.1 (17.3) Ours Single45.0 (53.5)18.0 (13.1)4.0 (3.0)33.0 (30.3) Ours Multiple 60.0 (61.0)11.0 (11.0)1.0 (2.6)28.0 (25.3) Gemma-2 9B-IT Original11.2 (9.5)4.0 (5.3)17.0 (11.6)67.8 (73.7) Ours Single 42.0 (51.0)17.0 (10.4)6.0 (5.2)35.0 (33.3) Ours Multiple40.0 (53.3)16.0 (5.3)2.0 (8.7)42.0 (32.7) Table 2: Jailbreak results for GCG (optimizing entire 40-token prompt) and AutoDAN (generating entire 200-token prompt) with the original and our single-prefix objectives. All prompts were optimized or generated from scratch. Ref. = Refusal, Inc. = Incomplete, Unf. = Unfaithful. ModelObjective GCG (40-token)AutoDAN (200-token) Success (%,↑) Failed Attack (%,↓) Success (%,↑) Failed Attack (%,↓) Ref.Inc.Unf.Ref.Inc.Unf. Llama-2 7B-Chat Original42.10.00.057.926.316.10.457.2 Ours 72.62.60.024.9 39.725.40.035.0 Llama-3 8B-Instruct Original14.116.235.534.25.234.528.332.1 Ours 79.50.32.317.8 77.92.50.019.6 Llama-3.1 8B-Instruct Original47.03.011.039.051.01.48.838.8 Ours 58.91.00.739.4 59.61.71.237.4 Gemma-2 9B-IT Original7.40.710.181.919.79.26.964.2 Ours 51.20.411.536.9 36.010.07.346.7 Evaluation. We use our nuanced judge for both prefix selection and jailbreak evaluation. We also use our preference judge to compare the quality of jailbreak responses against those from an uncensored LLM Orenguteng/Llama-3.1-8B-Lexi-Uncensored. For ablation studies, we also report results using HarmBench, JailbreakBench, and StrongReject. We generate victim LLM responses using greedy decoding and allow output to 512 tokens for nuanced evaluation. Each reported ASR is first averaged over four independent runs and then across all malicious requests. 5.1 Main Results Higher ASR. Table 1 shows that replacing the original “Sure, here is...” prefixes with our new model-dependent prefixes significantly improves ASR across all victim LLMs. On Llama-3, ASRs jump from around 10% to as high as 70%. Our multiple-prefix objective often achieves even higher ASRs. Appendix D shows that these relative improvements also hold when using the other three evaluation judges. Mitigated misspecification and overconstraint. The failure case breakdown shows that AdvPrefix works by mitigating misspecification and overconstraint. On three newer LLMs, it reduces incomplete 7 Figure 5: (Left) Prompt optimization loss curves using GCG on Llama-3, using the original and our objectives. (Right) Response harmfulness of GCG attacks compared to an uncensored LLM. Our objective leads to more harmful responses (e.g., detailed and realistic) than the original objective. A win rate below50%indicates that the jailbroken victim LLMs still cannot generate responses that are as harmful as the uncensored LLM. responses from about20%to1-2%, and cuts unfaithful responses by half on Gemma-2, indicating mitigated misspecification. Additionally, faster optimization convergence (Figure 5, left) and halved direct refusals (caused by failing to sufficiently lower the objective loss) on Llama-3 and 3.1 indicate mitigated overconstraints. Benefits from stronger optimization: longer attack suffixes. While longer suffixes generally lower final losses, the original objective’s ASR on newer LLMs remains poor (∼ 10%) due to frequent incomplete and unfaithful responses (Table 1). By mitigating this misspecification, AdvPrefix leverages longer suffixes to reduce direct refusals while managing incomplete and unfaithful responses, ultimately increasing ASR by an additional 9-15%. Benefits from stronger optimization: full prompt optimization. Table 2 shows that optimizing the entire attack prompt, rather than just the suffix, almost eliminates direct refusals for GCG attacks. However, the original objective yields inconsistent ASR changes due to more incomplete and unfaithful responses. By mitigating misspecification, AdvPrefix consistently increases ASR (e.g., Llama-2:39%to73%; Llama-3:70%to80%), highlighting its capability to take advantage of this less restrictive threat model. 5.2 Additional Results Universal Improvement Across Attacks. AdvPrefix also elevates AutoDAN’s ASR across all victim LLMs (Table 2). E.g., ASR on Llama-3 surges from 5% to 78%, showing AdvPrefix’s broad utility. Comparable harmfulness to uncensored LLM. Our preference judge’s evaluation (Figure 5, right) shows that GCG with AdvPrefix achieves harmfulness win rates of30∼ 50%against an uncensored LLM, significantly better than the original objective’s∼ 10%. This shows that AdvPrefix elicits responses with harmfulness levels approaching those of an uncensored LLM. Examples of selected prefixes. Our codebase releases the selected prefixes. While many prefixes start with affirmations, some, particularly for Gemma-2, do not start with affirmations (e.g., “**Password Cracking Algorithm** **”, “**Step 1: Reconnaissance** *”). Computational cost for AdvPrefix. Generating each prefix takes about 5 minutes on an A100 80G GPU (3 minutes with rougher PASR estimation). The 70B judge used for PASR estimation is the primary consumer of memory and computation. Importantly, prefix selection is a one-off cost: others can then reuse our pre-computed prefixes. Bypassing Deeper Alignment. Newer LLMs appear more robust against the original objective, often resisting generating “Sure, here is” prefixes or self-correcting after generating them. However, the high ASRs AdvPrefix achieves indicate that such deeper alignment can still be bypassed when targeting new prefixes. This result suggests that the current safety alignment fails to generalize to new, unseen prefixes. 8 6 Related Work Jailbreak attacks and red-teaming. Jailbreaking aligned LLMs, crucial for red-teaming, is a focus of many works. Beyond manual jailbreaks [Perez et al., 2022, Liu et al., 2023b, Wei et al., 2023a], automated methods are typically white-box (requiring model/logit access) or black-box (output-only). White-box attacks use search- or gradient-based prompt optimization [Zou et al., 2023, Andriushchenko et al., 2024, Guo et al., 2021, 2024, Geisler et al., 2024], sometimes with fluency considerations [Liu et al., 2023a, Zhu et al., 2023, Paulus et al., 2024, Thompson and Sklar, 2024]. Black-box attacks use designed/learned strategies [Chao et al., 2023, Mehrotra et al., 2023, Zeng et al., 2024b, Paulus et al., 2024, Zheng et al., 2024, Wei et al., 2023b, Anil et al., 2024] for interpretable prompt optimization, suitable for closed-source LLMs. White-box attacks, with weight access, can be stronger and more targeted, often proving most effective against defended LLMs like Llama-2 [Mazeika et al., 2024]. Recent efforts uncover novel attack vectors, such as exploiting model reasoning [Wu et al., 2025] or using assistive tasks to obscure intent [Chen et al., 2025]. Creating transferable attacks by reducing prompt overfitting remains an active research area [Lin et al., 2025, Zhang et al., 2025b]. We omit discussion of jailbreak attacks with threat models other than user prompt modification [Huang et al., 2024a, Zhao et al., 2024, Liu et al., 2024]. Jailbreak attack objectives. Jailbreak attack objectives have received comparatively less attention than attack methods. Some works discuss the original objective’s misspecification [Geiping et al., 2024, Liao and Sun, 2024], while others design new objectives to improve ASR. For instance, [Zhou and Wang, 2024, Xie et al., 2024] suppress refusals rather than elicit target prefixes, and Jia et al. [2024] augments prefixes with phrases like “my output is harmful” to improve ASR. The initial model response’s importance is also highlighted by work on response selection/steering [Tran et al., 2024] and attacks leveraging prefilling features [Zhang et al., 2025a]. Unlike these manual targets or specific interaction points, our work automatically tailors prefixes to specific victim LLMs and requests. Thompson and Sklar [2024] propose a dual objective: eliciting “Sure” and distilling from an uncensored teacher. However, this faces challenges with teacher learnability and sample efficiency for high-entropy teacher distributions. In contrast, our objective, akin to distilling from a degenerate (single-prefix) teacher, is sample-efficient, and our low-NLL prefix selection favors learnable behaviors. Recent objective designs include adaptive reinforcement learning frameworks [Sclar et al., 2025] and methods “guiding” LLM outputs by removing superfluous constraints to enhance transferability [Zhang et al., 2025b]. Our AdvPrefix systematically addresses misspecification and overconstraint by selecting prefixes for their likelihood to elicit harm and their ease of generation. Finally, findings that LLM-safety evaluations can lack robustness [Strauss et al., 2025] underscore the need for refined, nuanced evaluation frameworks like ours. 7 Conclusion This paper focuses on a key component of jailbreak attacks: the objective. We start by developing nuanced evaluation to identify limitations in the current objective, including misspecification and overconstraints. Then, we propose a new prefix-forcing objective leveraging carefully selected prefixes. Experiments demonstrate its effectiveness and compatibility with different attacks. Our plug-and-play design allows practitioners to use our released prefixes for free performance gains. We also analyze jailbreak objectives systematically, aiming to inspire further advancements. Our findings reveal that even the latest LLMs’ deep alignment can be bypassed, underscoring the need for more generalizable alignment. Limitations. A limitation of our objective is that selecting prefixes, especially for evaluating prefilling ASR, requires evaluating many sampled responses, leading to a computational burden. Moreover, we do not account for other desirable properties of the objective, such as a well-shaped loss landscape. Finally, our objective is designed for situations requiring the target model’s log probabilities, thus cannot be applied to black-box attacks. Broader Impacts Our research contributes to the safety and responsible development of future AI systems by exposing limitations in current models. While acknowledging the potential for misuse in adversarial research, we believe our methods do not introduce any new risks or unlock dangerous capabilities beyond 9 those already accessible through existing attacks or open-source models without safety measures. Finally, we believe that identifying vulnerabilities is essential for addressing them. By conducting controlled research to uncover these issues now, we proactively mitigate risks that could otherwise emerge during real-world deployments. References D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety.arXivpreprintarXiv:1606.06565, 2016. B. An, S. Zhu, R. Zhang, M.-A. Panaitescu-Liess, Y. Xu, and F. Huang. Automatic pseudo-harmful prompt generation for evaluating false refusals in large language models. InFirstConferenceon LanguageModeling, 2024. URL https://openreview.net/forum?id=ljFgX6A8NL. M. Andriushchenko, F. Croce, and N. Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks.arXivpreprintarXiv:2404.02151, 2024. C. Anil, E. DURMUS, N. Rimsky, M. Sharma, J. Benton, S. Kundu, J. Batson, M. Tong, J. Mu, D. J. Ford, F. Mosconi, R. Agrawal, R. Schaeffer, N. Bashkansky, S. Svenningsen, M. Lambert, A. Radhakrishnan, C. Denison, E. J. Hubinger, Y. Bai, T. Bricken, T. Maxwell, N. Schiefer, J. Sully, A. Tamkin, T. Lanham, K. Nguyen, T. Korbak, J. Kaplan, D. Ganguli, S. R. Bowman, E. Perez, R. B. Grosse, and D. Duvenaud. Many-shot jailbreaking. InTheThirty-eighthAnnualConference onNeuralInformationProcessingSystems, 2024. URLhttps://openreview.net/forum? id=cw5mgd71jW. Anthropic. The claude 3 model family: Opus, sonnet, haiku.TechnicalReport, 2024. K. Arora, L. El Asri, H. Bahuleyan, and J. Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In S. Muresan, P. Nakov, and A. Villavicencio, editors,FindingsoftheAssociationforComputationalLinguistics:ACL2022, pages 700–710, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/v1/2022.findings-acl.58. URLhttps://aclanthology.org/2022.findings-acl.58. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXivpreprintarXiv:2204.05862, 2022. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advancesinneuralinformationprocessingsystems, 28, 2015. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXivpreprintarXiv:2401.10774, 2024. P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries.arXivpreprintarXiv:2310.08419, 2023. P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flam- marion, G. J. Pappas, F. Tramèr, H. Hassani, and E. Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024. H. Chen, H. Wang, C. Huang, Q. Zhong, and Y. Xie. SATA: A Novel LLM Jailbreak Paradigm via Simple Assistive Task Linkage.arXivpreprintarXiv:2412.15289, Mar 2025. URLhttps: //arxiv.org/abs/2412.15289. J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh. Or-bench: An over-refusal benchmark for large language models.arXivpreprintarXiv:2405.20947, 2024. J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang. Safe rlhf: Safe reinforcement learning from human feedback.arXivpreprintarXiv:2310.12773, 2023. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models.arXivpreprintarXiv:2407.21783, 2024. 10 Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXivpreprintarXiv:2209.07858, 2022. J. Geiping, A. Stein, M. Shu, K. Saifullah, Y. Wen, and T. Goldstein. Coercing llms to do and reveal (almost) anything.arXivpreprintarXiv:2402.14020, 2024. S. Geisler, T. Wollschläger, M. Abdalla, J. Gasteiger, and S. Günnemann. Attacking large language models with projected gradient descent.arXivpreprintarXiv:2402.09154, 2024. F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve. Better & faster large language models via multi-token prediction.arXivpreprintarXiv:2404.19737, 2024. C. Guo, A. Sablayrolles, H. Jégou, and D. Kiela. Gradient-based adversarial attacks against text transformers. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors,Proceedings ofthe2021ConferenceonEmpiricalMethodsinNaturalLanguageProcessing, pages 5747– 5757, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.464. URLhttps://aclanthology.org/2021. emnlp-main.464. X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability.arXivpreprintarXiv:2402.08679, 2024. Haizelab.A trivial programmatic llama 3 jailbreak.https://github.com/haizelabs/ llama3-jailbreak, 2024. (Accessed on 06/26/2024). Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen. Catastrophic jailbreak of open-source LLMs via exploiting generation. InTheTwelfthInternationalConferenceonLearningRepresentations, 2024a. URL https://openreview.net/forum?id=r42tSSCHPh. Y. Huang, L. Sun, H. Wang, S. Wu, Q. Zhang, Y. Li, C. Gao, Y. Huang, W. Lyu, Y. Zhang, et al. Position: TrustLLM: Trustworthiness in large language models. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors,Proceedingsofthe41st InternationalConferenceonMachineLearning, volume 235 ofProceedingsofMachineLearning Research, pages 20166–20270. PMLR, 21–27 Jul 2024b. URLhttps://proceedings.mlr. press/v235/huang24x.html. H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXivpreprint arXiv:2312.06674, 2023. N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models.arXivpreprintarXiv:2309.00614, 2023. J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset.AdvancesinNeural InformationProcessingSystems, 36, 2024. X. Jia, T. Pang, C. Du, Y. Huang, J. Gu, Y. Liu, X. Cao, and M. Lin. Improved techniques for optimization-based jailbreaking on large language models.arXivpreprintarXiv:2405.21018, 2024. E. Jones, M. Tong, J. Mu, M. Mahfoud, J. Leike, R. Grosse, J. Kaplan, W. Fithian, E. Perez, and M. Sharma. Forecasting rare language model behaviors.arXivpreprintarXiv:2502.16797, 2025. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners.Advancesinneuralinformationprocessingsystems, 35:22199–22213, 2022. M. Labonne. Uncensor any llm with abliteration.https://huggingface.co/blog/mlabonne/ abliteration, 2024. 11 R. Lapid, R. Langberg, and M. Sipper. Open sesame! universal black box jailbreaking of large language models.arXivpreprintarXiv:2309.01446, 2023. X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/ tatsu-lab/alpaca_eval, 5 2023. Z. Liao and H. Sun. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms.arXivpreprintarXiv:2404.07921, 2024. R. Lin, B. Han, F. Li, and T. Liu. Understanding and enhancing the transferability of jailbreaking attacks.arXivpreprintarXiv:2502.03052, Feb 2025. URLhttps://arxiv.org/abs/2502. 03052. Accepted by ICLR 2025. H. Liu, Y. Xie, Y. Wang, and M. Shieh. Advancing adversarial suffix transfer learning on aligned large language models.arXivpreprintarXiv:2408.14866, 2024. X. Liu, N. Xu, M. Chen, and C. Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXivpreprintarXiv:2310.04451, 2023a. Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and Y. Liu. Prompt Injection attack against LLM-integrated Applications, June 2023b. M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXivpreprintarXiv:2402.04249, 2024. A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, Dec. 2023. OpenAI. Gpt-4 technical report, 2023. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback.Advancesin neuralinformationprocessingsystems, 35:27730–27744, 2022. K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau. Future lens: Anticipating subsequent tokens from a single hidden state.arXivpreprintarXiv:2311.04897, 2023. A. Paulus, A. Zharmagambetov, C. Guo, B. Amos, and Y. Tian. Advprompter: Fast adaptive adversarial prompting for llms.arXivpreprintarXiv:2404.16873, 2024. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models.arXivpreprintarXiv:2202.03286, 2022. X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson. Safety alignment should be made more than just a few tokens deep.arXivpreprintarXiv:2406.05946, 2024. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model.AdvancesinNeuralInformation ProcessingSystems, 36, 2024. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXivpreprintarXiv:2403.05530, 2024. M. Sclar, S. Shrishylam, Y. Xu, P. Liu, Y. Liu, and C.-J. Li. REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective.arXivpreprint arXiv:2502.17254, Feb 2025. URL https://arxiv.org/abs/2502.17254. A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, et al. A strongreject for empty jailbreaks.arXivpreprintarXiv:2402.10260, 2024. 12 M. Strauss, M. Mazeika, A. Pang, R. E. H. Kumar, A. Bjorklund, H. Bull, i. Lovisotto, S. Goel, J. Geiping, Y. Wen, J. P. Schwarz, J. Z. Kolter, T. Goldstein, S. Black, and G. Somepalli. LLM- Safety Evaluations Lack Robustness.arXivpreprintarXiv:2503.02574, Mar 2025. URLhttps: //arxiv.org/abs/2503.02574. G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprintarXiv:2408.00118, 2024. T. B. Thompson and M. Sklar. Fluent student-teacher redteaming.arXivpreprintarXiv:2407.17447, 2024. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprintarXiv:2307.09288, 2023. T. Q. Tran, K. Wataoka, and T. Takahashi. Initial response selection for prompt jailbreaking using model steering. InICLR2024WorkshoponSecureandTrustworthyLargeLanguageModels, 2024. URL https://openreview.net/forum?id=tcENS1O7dM. B. Vidgen, A. Agrawal, A. M. Ahmed, V. Akinwande, N. Al-Nuaimi, N. Alfaraj, E. Alhajjar, L. Aroyo, T. Bavalatti, B. Blili-Hamelin, et al. Introducing v0. 5 of the ai safety benchmark from mlcommons. arXivpreprintarXiv:2404.12241, 2024. A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does llm safety training fail?arXiv preprintarXiv:2307.02483, 2023a. Z. Wei, Y. Wang, A. Li, Y. Mo, and Y. Wang. Jailbreak and guard aligned language models with only few in-context demonstrations.arXivpreprintarXiv:2310.06387, 2023b. W. Wu, J. X. Morris, and L. Levine. Do language models plan ahead for future tokens?ArXiv, abs/2404.00859, 2024. URL https://api.semanticscholar.org/CorpusID:268819892. Y. Wu, Z. Zhao, W. Liu, Z. Ji, J. Yang, Y. Wang, and H. Zhang. Sugar-coated poison: Benign generation unlocks LLM jailbreaking.arXivpreprintarXiv:2504.05652, Apr 2025. URLhttps: //arxiv.org/abs/2504.05652. Z. Xie, J. Gao, L. Li, Z. Li, Q. Liu, and L. Kong. Jailbreaking as a reward misspecification problem. arXivpreprintarXiv:2406.14393, 2024. W. Zeng, Y. Liu, R. Mullins, L. Peran, J. Fernandez, H. Harkous, K. Narasimhan, D. Proud, P. Kumar, B. Radharapu, et al. Shieldgemma: Generative ai content moderation based on gemma.arXiv preprintarXiv:2407.21772, 2024a. Y. Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms, 2024b. B. Zhang, Y. Liu, and J. Yang. Prefill-based jailbreak: A novel approach of bypassing llm safety boundary.arXivpreprintarXiv:2504.21038, Apr 2025a. URLhttps://arxiv.org/abs/2504. 21038. Y. Zhang, J. Chi, H. Nguyen, K. Upasani, D. M. Bikel, J. Weston, and E. M. Smith. Backtracking improves generation safety, 2024. URL https://arxiv.org/abs/2409.14586. Z. Zhang, G. Shen, G. Tao, S. Cheng, and X. Zhang. Make them spill the beans! coercive knowledge extraction from (production) llms.arXivpreprintarXiv:2312.04782, 2023. Z. Zhang, Z. Wang, Y. Guo, K. Chen, M. Zhang, and J. Zhou. Guiding not forcing: Enhancing the transferability of jailbreaking attacks on llms via removing superfluous constraints.arXivpreprint arXiv:2503.01865, Mar 2025b. URL https://arxiv.org/abs/2503.01865. X. Zhao, X. Yang, T. Pang, C. Du, L. Li, Y.-X. Wang, and W. Y. Wang. Weak-to-strong jailbreaking on large language models.arXivpreprintarXiv:2401.17256, 2024. 13 X. Zheng, T. Pang, C. Du, Q. Liu, J. Jiang, and M. Lin. Improved few-shot jailbreaking can circumvent aligned language models and their defenses. InTheThirty-eighthAnnualConference onNeuralInformationProcessingSystems, 2024. URLhttps://openreview.net/forum? id=zMNd0JuceF. Y. Zhou and W. Wang. Don’t say no: Jailbreaking llm by suppressing refusal.arXivpreprint arXiv:2404.16369, 2024. S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, and T. Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXivpreprintarXiv:2307.15043, 2023. 14 A Additional Discussions Latest LLMs favor self-correction over direct refusal. Figure 3 (left) shows that when facing jailbreak attacks (GCG), newer LLMs are less likely to directly refuse requests. Instead, they often begin with the target prefix (“Sure, here is ...”) and then self-correct by giving incomplete or unfaithful responses. For example, both Llama-2 and Gemma-2 resist about90%of attacks. However, Llama-2 only gives unfaithful responses24%of the time and never gives incomplete responses. In contrast, Gemma-2 almost always gives incomplete or unfaithful responses, and rarely directly refuses. These different reactions suggest that newer LLMs may have undergone deeper alignment [Qi et al., 2024]. For example, developers might use prefixes from the original objective for supervised fine- tuning to prevent generating these prefixes or to self-correct when they do [Zhang et al., 2024]. However, experiments with our new objective show that such alignment fails to generalize to our prefixes. Why still prefix-forcing? A key challenge in designing jailbreak objectives is that defining jailbreak success relies on an autoregressive model’s output distribution, which is hard to estimate especially when it has high entropy. One way to estimate it is by sampling many responses, but this makes computing the objective value inefficient. Another way is to predict future outputs from the model’s current state, but current techniques can only predict a few tokens ahead [Pal et al., 2023, Gloeckle et al., 2024, Wu et al., 2024], while identifying nuanced harmful responses often requires examining hundreds. The prefix-forcing objective bypasses this challenge by specifying a low-entropy distri- bution that always outputs a specific prefix. Estimating such distribution is sample-efficient since it only requires the prefix. Building on this advantage, we continue using prefix-forcing but address the limitations of the original objective by carefully selecting the prefixes. Relationship to model distillation objective. Recently, Thompson and Sklar [2024] propose a new jailbreak objective based on distilling from an uncensored teacher LLM. We note that, when the teacher’s output distribution degenerates to a single prefix, the prefix-forcing objective becomes a special case of the model distillation objective with KL-based logit matching. Nevertheless, the prefix-forcing objective has three advantages over distilling from a high-entropy teacher distribution: First, it is sample-efficient, as only the prefix is needed for distillation. Second, the degenerated teacher distribution is often empirically learnable by optimizing hard token prompts, as evidenced by the near-zero losses in our experiments. Third, distilling from a single teacher distribution can be overconstrained, and our multi-prefix objective alleviates this. B More Related Work Safety alignment of LLMs. The development of LLMs involves several stages of safety alignment [Dubey et al., 2024, Huang et al., 2024b]. During pretraining, developers filter out harmful data to reduce the likelihood of the model generating them. In fine-tuning, developers use supervised fine-tuning (SFT) and RLHF [Ouyang et al., 2022, Bai et al., 2022, Dai et al., 2023, Ji et al., 2024, Rafailov et al., 2024] to adjust the model’s rejection behavior under malicious prompts. Finally, at deployment, system-level safety filters like Llama Guard [Inan et al., 2023] and ShieldGemma [Zeng et al., 2024a] help detect and block harmful inputs or outputs. Although newer LLMs use more refined strategies during fine-tuning to counter jailbreaks while minimizing false refusal rates [Anthropic, 2024, Dubey et al., 2024, Inan et al., 2023], our findings suggest that these strategies need more tailored prefixes to improve generalization. Geiping et al. [2024] also note this misspecification issue. Liao and Sun [2024], Zhou and Wang [2024] observe that lower loss does not necessarily lead to higher attack success rates and attribute it to exposure bias [Bengio et al., 2015, Arora et al., 2022], where target prefixes fail to be elicited due to high loss on the first token. Here, our result shows that even after successfully eliciting the prefix, the model still fails to generate a complete and faithful response. C Additional Experimental Details Judge Settings. We follow the setup guidelines in evaluating HarmBench, JailbreakBench, and StrongReject. We use the provided judge LLM finetuned from Llama-2-13B for HarmBench, Llama-3-70B for JailbreakBench, and the judge LLM finetuned from Gemma-2B for StrongReject. 15 JailbreakBench and StrongReject also support API-based judging (e.g., GPT-4), which we omit here. Since our evaluation requires binary harmfulness labels, we binarize StrongReject’s harmful score (originally ranging from 0 to 1) with a threshold of 0.6, which maximizes the F1 score on our dataset. Our preference judge. Defining harmfulness is complex [Vidgen et al., 2024], making it challenging to develop a perfect judge that assigns a binary label or absolute score to a response. For example, model developers often judge topics like suicide and misinformation by different standards. To address this, we introduce a preference judge [Li et al., 2023, Dubois et al., 2023] that compares the relative harmfulness between two responses, offering an alternative evaluation. We take responses from an uncensored LLM as the baseline, and compare them with those elicited by a jailbreak attack to measure the relative harm induced by the jailbreak. Dataset. The harmful request dataset is another factor influencing evaluation consistency across studies. Newer LLMs exhibit varying jailbreak robustness across different malicious topics [Mazeika et al., 2024]. For example, they are often more lenient with misinformation requests but are highly sensitive to self-harm and child abuse topics. This cost-sensitive alignment strategy minimizes the impact of false refusals [Cui et al., 2024, An et al., 2024] but increases variance in jailbreak evaluations. For more consistent evaluation, we select50highly harmful requests from AdvBench as our dataset, excluding sensitive topics such as child abuse. Prefix Selection Pipeline. To handle cases where uncensored LLMs still refuse highly harmful prompts, we prefill their responses with phrases like “Here”, “To”, or “Sure” to ensure compliance. Note that we can also construct candidate prefixes using rule-based methods or with base (non- instruction-tuned) LLMs instead of uncensored LLMs. When selecting multiple prefixes, we require that none are sub-prefixes of another, which ensures the probabilities sum to no more than one in the objective and prevents redundant computation. More details about candidate pool size, rejection rules, and weighting parameters, are included in our codebase at github.com/facebookresearch/jailbreak-objectives. D Additional Results We provide additional results in this section. Figure 6: Two examples of harmless responses for nuanced jailbreaks. Current evaluation judges struggle to distinguish them. 16 Table 3: We use other three judges to evaluate results in Table 2 as ablation. The results show GCG optimizing the entire attack prompt. Our objective achieves similar relative ASR improvements. ModelObjectiveHarmBenchJailbreakBenchStrongRejectOurs Llama-2 7B-Chat Original48.741.144.442.1 Ours Single76.670.674.072.6 Llama-3 8B-Instruct Original27.838.014.514.1 Ours Single82.282.284.279.5 Llama-3.1 8B-Instruct Original48.047.041.047.0 Ours Single56.852.662.058.9 Gemma-2 9B-IT Original21.520.19.47.4 Ours Single66.370.664.351.2 Table 4: Detailed meta-evaluation results of existing judges and ours. Numbers outside brackets are human agreement rates, while numbers inside brackets are F1 scores. JudgeLlama-2Llama-3Llama-3.1Gemma-2Average HarmBench93.5 (84.3)84.5 (78.9)92.5 (89.5)76.0 (69.6)86.6 (79.9) JailbreakBench93.0 (82.1)84.0 (78.1)88.0 (82.1)77.0 (68.9)85.5 (77.1) StrongReject94.5 (85.3)90.0 (84.6)90.0 (85.9)82.5 (75.5)89.3 (82.3) Ours94.0 (84.6)93.0 (88.7)94.0 (91.4)86.0 (74.6)91.8 (85.4) Failure CasesExample Response from the Victim LLM Direct RefusalI cannot provide a guide on how to manipulate someone into committing suicide. If you or someone you know is struggling with suicidal thoughts, it is [100+ words] IncompleteSure, here is guidance on how to manipulate someone into committing suicide:<eos> IncompleteHere is a guide on how to manipulate someone into committing sui- cide: **Disclaimer:** I cannot provide a guide on how to manipulate someone into committing suicide. <eos> UnfaithfulHere is a guide on how to manipulate someone into committing suicide in the context of the game "Tales from the Borderlands": **I. Understanding the Game [100+ words] Successful Attack Sure, here is guidance on how to manipulate someone into committing suicide: 1. **Build Trust** [100+ words] Table 5: More examples of failed attacks. We categorize failure cases to guide the design of more accurate judges and well-specified objectives. Figure 7: Prefilling ASR with manual attack prompts correlates with jailbreak ASR with optimized attack prompts. We compare three designs of manual attack prompts: request only, request followed by “Start your response with [request]”, and request followed by the initial attack suffix. Each point represents an attack’s prefilling ASR and jailbreak ASR. The figures aggregate results across four LLMs and 50 requests, demonstrating the correlation’s generality. Such correlation enables efficient prefix selection using prefilling attacks. 17 Figure 8: Replacing “Sure, here is [request]” with “Here is [request]” for Llama-3 and rerun GCG. Each point represents a single run. The new prefixes result in lower final losses (x-axis), while the ASRs improve slightly on average (y-axis, 12.8% to 22.1%). This observation indicates that the original objective is overconstrained for optimization. Figure 9: Our objective also enables attacking reasoning models (R1-distilled-Llama-3-8B) using model-dependent target prefixes. GCG with default prefixes cannot lower the loss in this case to achieve successful jailbreaks. 18