Paper deep dive

REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective

Simon Geisler, Tom Wollschläger, M. H. I. Abdalla, Vincent Cohen-Addad, Johannes Gasteiger, Stephan Günnemann

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 157

Models: Gemma 1.1 2B, Gemma 1.1 7B, Llama 2 7B, Llama 3 8B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 7:24:16 PM

Summary

The paper introduces a novel, adaptive, and semantic optimization objective for adversarial attacks on Large Language Models (LLMs) using the REINFORCE policy-gradient formalism. By treating LLM generation as a Markov Decision Process, the authors replace the traditional, inconsistent 'affirmative response' objective with a reward-based approach that maximizes the probability of harmful outputs. This method significantly improves the attack success rate (ASR) of existing algorithms like GCG and PGD on models such as Llama 3.

Entities (5)

GCG · adversarial-attack-algorithm · 99%Llama-3 · large-language-model · 99%PGD · adversarial-attack-algorithm · 99%Harmbench · benchmark · 95%REINFORCE · algorithm · 95%

Relation Signals (3)

GCG → targets → Llama-3

confidence 98% · GCG’s attack success rate (ASR) on... Llama 3 8B models

REINFORCE → improvesattacksuccess → GCG

confidence 95% · demonstrate its efficacy with the state-of-the-art jailbreak algorithms Greedy Coordinate Gradient (GCG)

REINFORCE → improvesattacksuccess → PGD

confidence 95% · demonstrate its efficacy with... Projected Gradient Descent (PGD)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:To circumvent the alignment of large language models (LLMs), current optimization-based adversarial attacks usually craft adversarial prompts by maximizing the likelihood of a so-called affirmative response. An affirmative response is a manually designed start of a harmful answer to an inappropriate request. While it is often easy to craft prompts that yield a substantial likelihood for the affirmative response, the attacked model frequently does not complete the response in a harmful manner. Moreover, the affirmative objective is usually not adapted to model-specific preferences and essentially ignores the fact that LLMs output a distribution over responses. If low attack success under such an objective is taken as a measure of robustness, the true robustness might be grossly overestimated. To alleviate these flaws, we propose an adaptive and semantic optimization problem over the population of responses. We derive a generally applicable objective via the REINFORCE policy-gradient formalism and demonstrate its efficacy with the state-of-the-art jailbreak algorithms Greedy Coordinate Gradient (GCG) and Projected Gradient Descent (PGD). For example, our objective doubles the attack success rate (ASR) on Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

156,250 characters extracted from source content.

Expand or collapse full text

Abstract To circumvent the alignment of large language models (LLMs), current optimization-based adversarial attacks usually craft adversarial prompts by maximizing the likelihood of a so-called affirmative response. An affirmative response is a manually designed start of a harmful answer to an inappropriate request. While it is often easy to craft prompts that yield a substantial likelihood for the affirmative response, the attacked model frequently does not complete the response in a harmful manner. Moreover, the affirmative objective is usually not adapted to model-specific preferences and essentially ignores the fact that LLMs output a distribution over responses. If low attack success under such an objective is taken as a measure of robustness, the true robustness might be grossly overestimated. To alleviate these flaws, we propose an adaptive and semantic optimization problem over the population of responses. We derive a generally applicable objective via the REINFORCE policy-gradient formalism and demonstrate its efficacy with the state-of-the-art jailbreak algorithms Greedy Coordinate Gradient (GCG) and Projected Gradient Descent (PGD). For example, our objective doubles the attack success rate (ASR) on Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.1 marginparsep has been altered. topmargin has been altered. marginparpush has been altered. The page layout violates the some style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective Simon Geisler 1 Tom Wollschläger 1 M. H. I. Abdalla 1 Vincent Cohen-Addad 2 Johannes Gasteiger 2 3 Stephan Günnemann 1 †footnotetext: 1Department of Computer Science & Munich Data Science Institute, Technical University of Munich 2Google Research 3Now at Anthropic. Correspondence to: Simon Geisler <s.geisler@tum.de>. 1 Resources & code: cs.cit.tum.de/daml/reinforce-attacks-llms 1 Introduction Identifying model misbehavior in large language models (LLMs) can be tricky, especially since models advance and the performance on benchmarks saturates. Nevertheless, the absence of misbehavior on static benchmarks does not imply the absence of misbehavior in general. For example, aligned LLMs readily refuse to answer many inappropriate prompts where the factual answer, e.g., could cause harm. However, the jailbreaking literature (Zou et al., 2023; Perez et al., 2022; Wen et al., 2023; Liu et al., 2024; Zhu et al., 2023; Geisler et al., 2024; Guo et al., 2024) revealed that minor tweaks usually suffice to circumvent alignment’s countermeasures. Yet, approaches that can automatically audit LLMs via crafting such adversarial prompts typically rely heavily on human-engineered objectives that do not adapt to the evaluated/attacked model. Specifically, adversarial prompts are usually chosen to maximize the likelihood of a fixed affirmative response to an inappropriate prompt. Such an objective has multiple defects. For example, even if the attack crafts a prompt that makes the affirmative response likely, the attacked model often continues in a non-harmful manner (see Fig. 2a). Instead, jailbreak attacks should optimize for an arbitrary harmful response that is reasonable to obtain with the attacked model. In Fig. 1, we illustrate which responses our REINFORCE objective targets and how this compares to the popular affirmative response objective. Figure 1: Responses that our REINFORCE and the affirmative objective encourage. Due to the popularity of the affirmative objective, it is likely that eliminating the hatched region will be prioritized while developing future models. Since improving the affirmative optimization objective does not necessarily lead to higher attack success, it is an inconsistent objective. Such inconsistency has been recognized as a key obstacle to strong adaptive attacks Tramer et al. (2020). While adaptive attacks refine their strategy based on the specifics of the target model, the affirmative objective is typically not even tailored to the attacked model. Non-adaptive attacks are known to overestimate robustness (Carlini & Wagner, 2017; Mujkanovic et al., 2022)–a pattern we also observe in our empirical evaluation of LLMs. (a) Unsuccessful attack via affirmative objective (b) Successful attack via REINFORCE objective (ours) Figure 2: Attack on Gemma 1.1 7B (DeepMind et al., 2024). Even though GCG (Zou et al., 2023) with (a) affirmative objective finds an adversarial suffix s.t. the model starts its response with target affirmation with >60%absentpercent60>60\%> 60 % chance, almost attaining an ideal outcome, the model completes the response harmlessly, almost mocking the attacker. (b) In contrast, our REINFORCE objective successfully disables the model’s alignment. We denote redactions with “*”. We revisit attacks on (conditional) generative models and specifically LLMs. We define adversarial attacks based on an adaptive and semantic objective that acknowledges that the model outputs a distribution over possible responses. By this, we formalize the foundations for adversarial attacks on generative models and set the basis for an adaptive yet automated search for model defects. Especially with the ever-emerging capabilities of large models, manually designing target responses appears to be a heavily involved arms race against model advancements. Instead, when using an appropriate reward signal, optimizing our objective directly maximizes the probability of obtaining the undesired behavior in the model’s generations. Since we evaluate our adversarial attack formulation via jailbreaking LLMs, we use an LLM-as-a-judge to quantify how harmful the model’s responses are and, therefore, optimize the probability of obtaining harmful responses. By replacing the reward signal, our formalism and objective could also be used to evaluate other properties, including factuality or unlearning. We implement our semantic objective via REINFORCE (Williams, 1992), where we optimize the prompt of an LLM w.r.t. to a reward signal. In reinforcement learning (RL) terms, the LLM is a probabilistic policy that we parametrize by its prompt to maximize the reward. In the context of jailbreaks, our REINFORCE objective guides the search for adversarial prompts with the goal of maximizing the probability of inappropriate/harmful generations. In Fig. 2b, we present an exemplary generation that results from a successful attack with our objective. We show that the state-of-the-art attacks Greedy Coordinate Gradient (GCG) of Zou et al. (2023) as well as the Projected Gradient Descent (PGD) of Geisler et al. (2024) immensely benefit from our formulation of the optimization problem, even when keeping all other parameters of the attacks constant. For example, GCG’s attack success rate (ASR) on HarmBench’s challenging standard behaviors (Mazeika et al., 2024) roughly doubles on the safety-tuned Llama 2 7B and Llama 3 8B models if comparing our REINFORCE objective to the ordinary affirmative response objective. Moreover, we attack the state-of-the-art defense for Llama 3 based on circuit breaking (Zou et al., 2024) with an attack success rate of 50%, while GCG with affirmative response is virtually never successful. Contributions: [1] We formally define an adversarial attack on conditional generative models that targets the semantics of the model’s responses, acknowledges the distributional nature of generations, and is adaptive. With our formulation and given an appropriate reward signal, a jailbreak attack directly optimizes the probability of obtaining harmful responses. The resulting objective is (asymptotically) consistent, which is important for strong attacks. [2] We derive a widely applicable objective using REINFORCE and demonstrate its practicality as well as efficacy on state-of-the-art LLMs by extending the GCG and PGD attacks. 2 Adaptive, Distributional, and Semantic Adversarial Attacks on Generative Models Adversarial attack. We define an adversarial attack on the (conditional) generative model Pfθ⁢(Y|X)=fθ⁢(Y|X)subscriptsubscriptconditionalsubscriptconditionalP_f_θ(Y|X)=f_θ(Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X ) = fitalic_θ ( Y | X ) as x~∗=arg⁢maxx~∈Φ⁢(x)⁡y∼Pfθ⁢(Y|X=x~)⁢[Reward⁡(y,x~)]superscript~subscriptargmax~Φsubscriptsimilar-tosubscriptsubscriptconditional~delimited-[]Reward~ x^*= *arg\,max_ x∈ (x)E_y P% _f_θ(Y|X= x) [Reward(y, x) ]over~ start_ARG x end_ARG∗ = start_OPERATOR arg max end_OPERATORover~ start_ARG x end_ARG ∈ Φ ( x ) blackboard_Ey ∼ P start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG x end_ARG ) end_POSTSUBSCRIPT [ Reward ( y , over~ start_ARG x end_ARG ) ] (1) where we aim to find an adversarial input x~∗superscript~ x^*over~ start_ARG x end_ARG∗ among the set of admissible adversarial inputs Φ⁢(x)Φ (x)Φ ( x ) that maximizes the expected reward ⁢[Reward⁡(y,x~)]delimited-[]Reward~E [Reward(y, x) ]blackboard_E [ Reward ( y , over~ start_ARG x end_ARG ) ] over the population of responses y∈y∈Yy ∈ Y. Since Eq. 1 describes a test time (evasion) attack, the parameters of the model θ are kept constant, and solely the input x∈x∈Xx ∈ X to the model is changed. A constraint on x~~ xover~ start_ARG x end_ARG is not strictly required due to the reward and Φ⁢(x)=Φ (x)=XΦ ( x ) = X is usually sensible. However, often the adversarial input X~~ Xover~ start_ARG X end_ARG is chosen in the vicinity of clean inputs x: x~∈Φ⁢(x)~Φ x∈ (x)over~ start_ARG x end_ARG ∈ Φ ( x ). For example, if appending some tokens to the prompt x, Φ⁢(x)Φ (x)Φ ( x ) is the set of possible suffixes appended to x. Semantic. Here, we use “reward” not because the targeted outcome is likely to be desirable but rather to streamline the terminology with our RL-based approach. By convention, we assume that higher values are better from the perspective of the optimization problem, i.e., the attacker. Since we implement this objective for jailbreaks in this work, the reward scores the harmfulness Reward⁡(y,x~)=Harmfulness⁡(y,x~)Reward~Harmfulness~Reward(y, x)=Harmfulness(y, x)Reward ( y , over~ start_ARG x end_ARG ) = Harmfulness ( y , over~ start_ARG x end_ARG ) of output y, given clean input x, perturbed input x~~ xover~ start_ARG x end_ARG, or both. Depending on the semantics that the reward captures, our definition covers a variety of problem settings, including factuality or unlearning. Distributional. In our distributional perspective on the model’s outputs Y and the corresponding adversarial attack (Eq. 1), we optimize the expectation since it yields a simple solution (see § 3) and allows for a very natural interpretation, while other descriptive statistics would also be possible. Specifically, in the context of jailbreaks and assuming the judge is well-calibrated Reward⁡(Y,X=x~)=P⁢(Harmful|Y=y,X=x~)Reward~formulae-sequenceconditionalHarmful~Reward(Y,X= x)=P(Harmful|Y=y,X= x)Reward ( Y , X = over~ start_ARG x end_ARG ) = P ( Harmful | Y = y , X = over~ start_ARG x end_ARG ), the optimization problem of Eq. 1 maximizes the probability of harmful responses: y∼Pfθ⁢(Y|X=x~)⁢[Reward⁡(Y=y,X=x~)]subscriptsimilar-tosubscriptsubscriptconditional~delimited-[]Reward~ _y P_f_θ(Y|X= x)[% Reward(Y=y,X= x)]blackboard_Ey ∼ P start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG x end_ARG ) end_POSTSUBSCRIPT [ Reward ( Y = y , X = over~ start_ARG x end_ARG ) ] (2) =∫P⁢(Harmful|Y,X=x~)⁢Pfθ⁢(Y|X=x~)⁢dYabsentconditionalHarmful~subscriptsubscriptconditional~differential-d = P(Harmful|Y,X= x)P_f_θ(Y|X=% x) \!dY= ∫ P ( Harmful | Y , X = over~ start_ARG x end_ARG ) Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG x end_ARG ) d Y =P⁢(Harmful|X=x~)absentconditionalHarmful~ =P(Harmful|X= x)= P ( Harmful | X = over~ start_ARG x end_ARG ) Adaptive. Our distributional perspective stands in stark contrast to the prevalent optimization problem tackled for jailbreaking LLMs, where the likelihood of a single and fixed (partial) output yaffirmativesubscriptaffirmativey_affirmativeyaffirmative is maximized: x~∗=arg⁢maxx~∈Φ⁢(x)⁡Pfθ⁢(Y=yaffirmative|X=x~)superscript~subscriptargmax~Φsubscriptsubscriptconditionalsubscriptaffirmative~ x^*= *arg\,max _ x∈ (x)P_f_% θ(Y=y_affirmative|X= x)over~ start_ARG x end_ARG∗ = start_OPERATOR arg max end_OPERATORover~ start_ARG x end_ARG ∈ Φ ( x ) Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y = yaffirmative | X = over~ start_ARG x end_ARG ) (3) Note that the static choice of yaffirmativesubscriptaffirmativey_affirmativeyaffirmative is usually not specific to model fθsubscriptf_θfitalic_θ. Adapting an attack, manually or automatically, to a specific model is one of the key properties for strong adversarial attacks in predictive tasks, as demonstrated by Tramer et al. (2020). They further state that […] loss functions, the cornerstone of successful adaptive attacks, [should be] consistent–so that higher loss values result in strictly stronger attacks. The affirmative objective is neither consistent w.r.t. the semantics (harmfulness) of generations nor does it adapt to the attacked model. Such non-consistent objectives are known to be a critical hindrance to successful attacks, especially if evaluating defenses (Carlini & Wagner, 2017; Athalye et al., 2018). With a non-consistent objective, even an all-powerful optimizer might end up with an unsuccessful attack. Conversely, optimizing the objective in Eq. 1 adapts to the model’s output distributions and is (asymptotically) consistent. This makes it appropriate for adaptive attacks. Unfortunately, we lack an efficient estimator–until now. 3 REINFORCE Attacks on LLMs LLMs autoregressively model Pfθ⁢(Y|X)subscriptsubscriptconditionalP_f_θ(Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X ) as Pfθ⁢(Y=|X=)subscriptsubscriptconditional P_f_θ(Y= y|X= x)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y = italic_y | X = italic_x ) =∏t=1TPfθ⁢(yt|⊕:t−1)absentsuperscriptsubscriptproduct1subscriptsubscriptconditionalsubscriptdirect-sumsubscript:absent1 =Π _t=1^TP_f_θ(y_t| x\, % \, y_:t-1)= ∏t = 1T Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( yitalic_t | italic_x ⊕ italic_y: t - 1 ) (4) with concatenation ⊕direct-sum\, \,⊕, generated token sequence ∈Tsuperscript y∈V^Titalic_y ∈ Vitalic_T, generated token sequence until (exclusive) the t-th token :t−1∈t−1subscript:absent1superscript1 y_:t-1∈V^t-1italic_y: t - 1 ∈ Vitalic_t - 1, prompt ∈T′superscript′ x∈V^T italic_x ∈ Vitalic_T start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT, and set of tokens VV. For simplicity, we do not introduce an explicit notation for the so-called system tokens before xitalic_x and between xitalic_x/ yitalic_y that, among other things, indicate the start or end of the user prompt and start of the model response. Markov decision process (MDP). LLMs’ autoregressive nature allows for an interpretation as a MDP. Specifically, the distribution of the next token Pfθ⁢(yt|⊕:t−1)subscriptsubscriptconditionalsubscriptdirect-sumsubscript:absent1P_f_θ(y_t| x\, \, y_:t-1)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( yitalic_t | italic_x ⊕ italic_y: t - 1 ) can be understood as a probabilistic policy πfθ⁢(X=⊕:t−1)=Pfθ⁢(Y=yt|X=⊕:t−1)subscriptsubscriptdirect-sumsubscript:absent1subscriptsubscriptconditionalsubscriptdirect-sumsubscript:absent1 _f_θ(X= x\, \, y_:t-1)=P_f_θ(Y=y_t% |X= x\, \, y_:t-1)πitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( X = italic_x ⊕ italic_y: t - 1 ) = Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y = yitalic_t | X = italic_x ⊕ italic_y: t - 1 ). With discount factor γ=11γ=1γ = 1 and solely a terminal reward Reward⁡(:t,)=Reward⁡(,),if ⁢:t=0,otherwiseRewardsubscript:absentcasesRewardif subscript:absent0otherwiseReward( y_:t, x)= cases% Reward( y, x),&if y_:t= y\\ 0,&otherwise casesReward ( italic_y: t , italic_x ) = start_ROW start_CELL Reward ( italic_y , italic_x ) , end_CELL start_CELL if italic_y: t = italic_y end_CELL end_ROW start_ROW start_CELL 0 , end_CELL start_CELL otherwise end_CELL end_ROW (5) the value function V⁢(Y,X)V(Y,X)V ( Y , X ) is equivalent to the expectation in the adversarial attack objective Eq. 1: V⁢(Y,X=) V(Y,X= x)V ( Y , X = italic_x ) =∼πfθ⁢(Y|X=)⁢[∑t=1∞γt⁢Reward⁡(:t,)]absentsubscriptsimilar-tosubscriptsubscriptconditionaldelimited-[]superscriptsubscript1superscriptRewardsubscript:absent =E_ y _f_θ(Y|X= x) [% _t=1^∞γ^tReward( y_:t, x) ]= blackboard_Eitalic_y ∼ π start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = italic_x ) end_POSTSUBSCRIPT [ ∑t = 1∞ γitalic_t Reward ( italic_y: t , italic_x ) ] (6) =∼Pfθ⁢(Y|X=)⁢[Reward⁡(,)]absentsubscriptsimilar-tosubscriptsubscriptconditionaldelimited-[]Reward =E_ y P_f_θ(Y|X= x) [% Reward( y, x) ]= blackboard_Eitalic_y ∼ P start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = italic_x ) end_POSTSUBSCRIPT [ Reward ( italic_y , italic_x ) ] The biggest difference to the RL problem is that we do not parametrize the policy πfθ⁢(X=⊕:t−1)=Pfθ⁢(yt|⊕:t−1)subscriptsubscriptdirect-sumsubscript:absent1subscriptsubscriptconditionalsubscriptdirect-sumsubscript:absent1 _f_θ(X= x\, \, y_:t-1)=P_f_θ(y_t|% x\, \, y_:t-1)πitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( X = italic_x ⊕ italic_y: t - 1 ) = Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( yitalic_t | italic_x ⊕ italic_y: t - 1 ) directly since the model parameters θ are constant at test time. Instead, Eq. 1 indirectly parametrizes a policy by choosing the initial state X=~~X= xX = over~ start_ARG italic_x end_ARG. However, besides this detail, Eq. 1 is equivalent to RL’s maximization of the value function arg⁢max~⁡V⁢(Y,X=~)subscriptargmax~~ *arg\,max_ xV(Y,X= x)start_OPERATOR arg max end_OPERATORover~ start_ARG italic_x end_ARG V ( Y , X = over~ start_ARG italic_x end_ARG ). Gradient-based jailbreak attacks. We aim to equip existing state-of-the-art optimization-based attacks with our objective. Some of the most effective optimization-based attacks (Zou et al., 2023; Geisler et al., 2024; Guo et al., 2024; Zhu et al., 2023) rely on gradient information for the one-hot encoded input tokens ∈0,1T′×||superscript01superscript′ X∈\0,1\^T ×|V|italic_X ∈ 0 , 1 T start_POSTSUPERSCRIPT ′ × | V | end_POSTSUPERSCRIPT. That is, Xitalic_X contains only zeros except for a single one per row at the position of the tokens in xitalic_x. With the implicit relaxation 0,1→[0,1]→0101\0,1\→[0,1] 0 , 1 → [ 0 , 1 ] we can then use the gradients ∇~subscript∇~ _ X∇over~ start_ARG italic_X end_ARG to aid the optimization over ~~ xover~ start_ARG italic_x end_ARG. We use xitalic_x and Xitalic_X interchangeably. REINFORCE. One common strategy to maximize the value function, and thereby the adversarial attack objective of Eq. 1, is via REINFORCE (Williams, 1992). REINFORCE is a policy-gradient optimization algorithm using the gradient ∇~V⁢(Y,X=~)=∇~⁢[Reward⁡(,~)]subscript∇~~subscript∇~delimited-[]Reward~ _ XV(Y,X= X)= _ X% E [Reward( y, X) ]∇over~ start_ARG italic_X end_ARG V ( Y , X = over~ start_ARG italic_X end_ARG ) = ∇over~ start_ARG italic_X end_ARG blackboard_E [ Reward ( italic_y , over~ start_ARG italic_X end_ARG ) ] and a gradient ascent algorithm of choice. Via the use of the policy gradient theorem ∇~∼Pfθ⁢(Y|X=~)⁢[Reward⁡(,~)]subscript∇~subscriptsimilar-tosubscriptsubscriptconditional~delimited-[]Reward~ _ XE_ y P_f_θ(% Y|X= X) [Reward( y, X)% ]∇over~ start_ARG italic_X end_ARG blackboard_Eitalic_y ∼ P start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG italic_X end_ARG ) end_POSTSUBSCRIPT [ Reward ( italic_y , over~ start_ARG italic_X end_ARG ) ] (7) ∝∼Pfθ⁢(Y|X=~)⁢[Reward⁡(,~)⁢∇~log⁡Pfθ⁢(|~)]proportional-toabsentsubscriptsimilar-tosubscriptsubscriptconditional~delimited-[]Reward~subscript∇~subscriptsubscriptconditional~ _ y P_f_θ(Y|X= X% ) [Reward( y, X) _ % X P_f_θ( y| X) ]∝ blackboard_Eitalic_y ∼ P start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG italic_X end_ARG ) end_POSTSUBSCRIPT [ Reward ( italic_y , over~ start_ARG italic_X end_ARG ) ∇over~ start_ARG italic_X end_ARG log Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( italic_y | over~ start_ARG italic_X end_ARG ) ] REINFORCE uses a proportional proxy of the true gradient, which is easier to evaluate. Using this objective, gradient-based attacks become adaptive, distributional, and optimize w.r.t. the semantics of the generations. For jailbreaks and a well-calibrated reward, our objective allows such attacks to optimize arg⁢maxx~⁡P⁢(Harmful|X=x~)subscriptargmax~conditionalHarmful~ *arg\,max_ xP(Harmful|X= x)start_OPERATOR arg max end_OPERATORover~ start_ARG x end_ARG P ( Harmful | X = over~ start_ARG x end_ARG ). Baseline. Usually the expectation in Eq. 7 is approximated via sampling ∼Pfθ⁢(Y|X)similar-tosubscriptsubscriptconditional y P_f_θ(Y|X)italic_y ∼ Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X ). To lower the variance it is common to introduce a baseline b(i)⁢(~)superscript~b^(i)( X)b( i ) ( over~ start_ARG italic_X end_ARG ) for the K samples: ∇~∼Pfθ⁢(Y|X=~)⁢[Reward⁡(,~)]subscript∇~subscriptsimilar-tosubscriptsubscriptconditional~delimited-[]Reward~ _ XE_ y P_f_θ(% Y|X= X) [Reward( y, X) ]∇over~ start_ARG italic_X end_ARG blackboard_Eitalic_y ∼ P start_POSTSUBSCRIPT f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG italic_X end_ARG ) end_POSTSUBSCRIPT [ Reward ( italic_y , over~ start_ARG italic_X end_ARG ) ] (8) ≈∑i=1K[Reward⁡((i),~)−b(i)⁢(~)]⁢∇~log⁡Pfθ⁢((i)|~)absentsuperscriptsubscript1delimited-[]Rewardsuperscript~superscript~subscript∇~subscriptsubscriptconditionalsuperscript~ ≈ _i=1^K [Reward( y^(i),% X)-b^(i)( X) ] _ X% P_f_θ( y^(i)| X)≈ ∑i = 1K [ Reward ( italic_y( i ) , over~ start_ARG italic_X end_ARG ) - b( i ) ( over~ start_ARG italic_X end_ARG ) ] ∇over~ start_ARG italic_X end_ARG log Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( italic_y( i ) | over~ start_ARG italic_X end_ARG ) While there are many possibilities for designing the baseline (e.g., exponential moving averages or a learned value function for an actor-critic approach), we use the simple and parameter-free REINFORCE Leave-One-Out (RLOO) estimator, proposed by Kool et al. (2019) that recently achieved promising results for RLHF (Ahmadian et al., 2024). For K=11K=1K = 1, we resort to Eq. 8 with b(i)⁢(~)=0superscript~0b^(i)( X)=0b( i ) ( over~ start_ARG italic_X end_ARG ) = 0. ∑i=1K[Reward⁡((i),~)−1K−1⁢∑j≠iReward⁡((j),~)]superscriptsubscript1delimited-[]Rewardsuperscript~11subscriptRewardsuperscript~ _i=1^K [Reward( y^(i), % X)- 1K-1 _j≠ iReward( y^(j),% X) ]∑i = 1K [ Reward ( italic_y( i ) , over~ start_ARG italic_X end_ARG ) - divide start_ARG 1 end_ARG start_ARG K - 1 end_ARG ∑j ≠ i Reward ( italic_y( j ) , over~ start_ARG italic_X end_ARG ) ] (9) ⋅∇~logPfθ((i)|~) ·\, _ X P_f_θ( y^(i)% | X)⋅ ∇over~ start_ARG italic_X end_ARG log Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( italic_y( i ) | over~ start_ARG italic_X end_ARG ) Interpretation. Our REINFORCE estimator equals a signed weighted average over cross entropy CE⁡(|~)CEconditional~CE( y| X)CE ( italic_y | over~ start_ARG italic_X end_ARG ) operations for samples ∼Ysimilar-to y Yitalic_y ∼ Y that we obtain to approximate the current expected reward. For this, note that log⁡Pfθ⁢(Y=|~)=−CE⁡(|~)=−∑tTCE⁡(yt,fθ⁢(~⊕:t−1))subscriptsubscriptconditional~CEconditional~superscriptsubscriptCEsubscriptsubscriptdirect-sum~subscript:absent1 P_f_θ(Y= y| X)=-CE( y|% X)=- _t^TCE(y_t,f_θ( % X\, \, y_:t-1))log Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y = italic_y | over~ start_ARG italic_X end_ARG ) = - CE ( italic_y | over~ start_ARG italic_X end_ARG ) = - ∑titalic_T CE ( yitalic_t , fitalic_θ ( over~ start_ARG italic_X end_ARG ⊕ italic_y: t - 1 ) ). Due to the baseline and with slight simplification in this description, our objective aims to reduce the likelihood of below-average harmful generations (“away” loss). Conversely, above-average harmful generations are encouraged to become more harmful (“towards” loss). Hence, our objective adaptively steers the attack towards harmful behaviors and away from non-harmful behaviors relative to the current samples. However, the dynamic nature of the baselines may also be suboptimal when all generations are harmful or harmless. To stabilize the attack in this case, we add a small constant (0.10.10.10.1) to the baseline in Eq. 9 and increment K by one. This way, our estimator behaves like an exclusive “away” loss if all generations are harmless and like an exclusive “towards” loss if all generations are harmful. Relation to affirmative objective. Our objective contains the affirmative objective as a special case with Reward⁡(affirmative)=1Rewardsubscriptaffirmative1Reward( y_affirmative)=1Reward ( italic_yaffirmative ) = 1 and a biased sampler that only samples affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative. In other words, the affirmative objective is a biased estimator of Eq. 1 that is non-adaptive due to the lack of sampling and lack of adjustment towards the current generations given ~~ Xover~ start_ARG italic_X end_ARG, including their rewards. Sampling strategy Pfθ′⁢(Y|X)superscriptsubscriptsubscript′conditionalP_f_θ (Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ). To estimate the expectation in Eq. 7 as efficiently as possible we use a random generation randomsubscriptrandom y_randomitalic_yrandom. For randomsubscriptrandom y_randomitalic_yrandom, we use sampling with low temperature (Ackley et al., 1985). The temperature allows the exploration of generations while controlling the style of LLM generations (Ficler & Goldberg, 2017) and also mimics the deployment of LLMs. While more random samples would be desirable, they increase the computational cost. With sample efficiency in mind, we instead include the greedy generation greedysubscriptgreedy y_greedyitalic_ygreedy since it approximates the output distribution’s mode and is often (but not always) a good proxy for its mean behavior (Scholten et al., 2025). Unfortunately, on aligned LLMs and at the beginning of the attack, this sampling strategy is likely not very sample efficient since P⁢(Harmful|X=)conditionalHarmfulP(Harmful|X= X)P ( Harmful | X = italic_X ) is low. Fortunately, for jailbreaking LLMs, in contrast to many RL applications, we have access to promising policy roll-outs seedsubscriptseed y_seeditalic_yseed. For example, one can re-use the affirmative response seed=affirmativesubscriptseedsubscriptaffirmative y_seed= y_affirmativeitalic_yseed = italic_yaffirmative or a generation of a previously successful attack seed=historysubscriptseedsubscripthistory y_seed= y_historyitalic_yseed = italic_yhistory, potentially obtained from a different (variant of the) model. Hence, we bias the sampling for Eq. 9 with the goal of improved sample efficiency. We argue that mixing in further roll-outs provides some so-called off-policy characteristics. In other words, to control the well-known exploration-exploitation tradeoff of RL, we employ a specialized sampling strategy Pfθ′⁢(Y|X)superscriptsubscriptsubscript′conditionalP_f_θ (Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ). Additionally to seedsubscriptseed y_seeditalic_yseed, we enhance the exploitation of harmful generations by including the most likely harmful generation harmfulsubscriptharmful y_harmfulitalic_yharmful, once the model produced a harmful generation during the attack. Further, to harmonize lengths between generations, we may greedily extend seedsubscriptseed y_seeditalic_yseed. In summary, we use these (up to) four samples to approximate Eq. 7 via Eq. 9: (seed,random,greedy,harmful)∼Pfθ′similar-tosubscriptseedsubscriptrandomsubscriptgreedysubscriptharmfulsuperscriptsubscriptsubscript′( y_seed, y_random, y_greedy, % y_harmful) P_f_θ ( italic_yseed , italic_yrandom , italic_ygreedy , italic_yharmful ) ∼ Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′. During the optimization, we truncate generations at 128 tokens for efficiency but evaluate with up to 512 tokens (see § 4). We next detail how we integrate our REINFORCE objective into GCG and PGD and refer to § C for details as well as § D for the tradeoffs between GCG and PGD. 3.1 REINFORCE-GCG Our REINFORCE-GCG can be decomposed into four steps: (1) Generate; (2) Gradient calculation; (3) Mutate current prompt to obtain candidates; (4) Score candidates and proceed with best. In our REINFORCE implementation of GCG (Zou et al., 2023), we keep all its key features unchanged, like only allowing perturbing tokens containing ASCII characters, but we omit such details here for presentation. We portray REINFORCE-GCG in Algo. 1. We use GCG’s strategy for sampling the candidate substitutions (Mutate), which alter the previous prompt each by exactly one token. For the loss ℓ ℓ we use the negative of estimator Eq. 9 by dropping ∇~subscript∇~ _ X∇over~ start_ARG italic_X end_ARG. We follow the above sampling strategy for the generations ^=(1),(2),…,(K)^superscript1superscript2…superscript Y=\ y^(1), y^(2),…, y^(K)\over start_ARG Y end_ARG = italic_y( 1 ) , italic_y( 2 ) , … , italic_y( K ) . Our GCG definition degrades to the regular GCG of Zou et al. (2023) if we use ℓ=CEℓCE =CEℓ = CE and fix ^=affirmative^subscriptaffirmative Y=\ y_affirmative\over start_ARG Y end_ARG = italic_yaffirmative . Details. We find that the reward for jailbreaking (with the HarmBench judges (Mazeika et al., 2024)) behaves almost in a binary manner, since it is either close to 0 or close to 1 but rarely takes values in between. This binary behavior yields only little guidance for REINFORCE. However, we may use intermediate rewards to steer the attack towards harmful prefixes of the model’s responses. Moreover, the reward calculation is fairly cheap in contrast to the Selection (see § E). Specifically, we calculate the rewards also for prefixes of the generations of lengths 20, 40, 80, and clamp the closest value in this sequence to the original length of seedsubscriptseed y_seeditalic_yseed. To lower the cost of the Selection, we use these intermediate rewards (excluding seedsubscriptseed y_seeditalic_yseed) to determine the length of the generation for choosing the best mutation arg⁢min~∈^⁡ℓ⁢(^,~)subscriptargmin~^ℓ^~ *arg\,min_ X∈ X ( % Y, X)start_OPERATOR arg min end_OPERATORover~ start_ARG italic_X end_ARG ∈ over start_ARG X end_ARG ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ). We use generation length 40 if no generation is harmful. Beyond that, if, e.g., the generation at length 40 is harmful, we calculate the loss until length 80. Moreover, we exclude the randomly obtained sample randomsubscriptrandom y_randomitalic_yrandom for the candidate selection, which also improves its stability due to the single random sample. Since GCG is not making progress monotonically (i.e., possibly ℓ⁢(~(i+1))>ℓ⁢(~(i))ℓsuperscript~1ℓsuperscript~ ( X^(i+1))> ( X^(i))ℓ ( over~ start_ARG italic_X end_ARG( i + 1 ) ) > ℓ ( over~ start_ARG italic_X end_ARG( i ) )), once the mode of the output distribution (approximated by the greedy generation) is harmful, we only accept a new candidate if the mode remains harmful. Moreover, instead of returning the final ~(E+1)superscript~1 X^(E+1)over~ start_ARG italic_X end_ARG( E + 1 ), we return the best generation according to a modified version of ℓ ℓ that emphasizes the mode of the output distribution, to improve consistency in a low-sample regime. See § A for additional details on GCG and our REINFORCE-GCG. Algo. 1 REINFORCE Greedy Coordinate Gradient (GCG) Input: Initial prompt ~(1)superscript~1 X^(1)over~ start_ARG italic_X end_ARG( 1 ), loss ℓ ℓ, sampling strategy Pfθ′⁢(Y|X=~)superscriptsubscriptsubscript′conditional~P_f_θ (Y|X= X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X = over~ start_ARG italic_X end_ARG ), # iterations E, # samples K, search width S for i∈1,2,…,E12…i∈\1,2,…,E\i ∈ 1 , 2 , … , E do ^←(1),…,(K)∼Pfθ′⁢(Y|X=~(i))←^superscript1…superscriptsimilar-tosubscriptsuperscript′subscriptconditionalsuperscript~ Y←\ y^(1),…, y^(K)\ P^% _f_θ(Y|X= X^(i))over start_ARG Y end_ARG ← italic_y( 1 ) , … , italic_y( K ) ∼ P′italic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | X = over~ start_ARG italic_X end_ARG( i ) ) ▷ ▷ Generate ←∇~(i)ℓ⁢(^,~(i))←subscript∇superscript~ℓ^superscript~ G← _ X^(i) ( Y,% X^(i))italic_G ← ∇over~ start_ARG italic_X end_ARG( i ) ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG( i ) ) ▷ ▷ Gradient ^←^(1),…,^(S)∼MutateS⁡(,~(i))←^superscript^1…superscript^similar-tosubscriptMutatesuperscript~ X←\ X^(1),…, X^(S)% \ _S( G, X^(i))over start_ARG X end_ARG ← over start_ARG italic_X end_ARG( 1 ) , … , over start_ARG italic_X end_ARG( S ) ∼ Mutateitalic_S ( italic_G , over~ start_ARG italic_X end_ARG( i ) ) ▷ ▷ Mutate ~(i+1)←arg⁢min~∈^⁡ℓ⁢(^,~)←superscript~1subscriptargmin~^ℓ^~ X^(i+1)← *arg\,min_ X% ∈ X ( Y, X)over~ start_ARG italic_X end_ARG( i + 1 ) ← start_OPERATOR arg min end_OPERATORover~ start_ARG italic_X end_ARG ∈ over start_ARG X end_ARG ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ) ▷ ▷ Selection return ~(E+1)superscript~1 X^(E+1)over~ start_ARG italic_X end_ARG( E + 1 ) 3.2 REINFORCE-PGD To obtain our REINFORCE-PGD from the version by Geisler et al. (2024), we require similar changes as for GCG. Namely, we generate samples for the approximation of the expectation and replace the loss ℓ ℓ, following Eq. 9. Hence, our PGD definition in Algo. 2 degrades to the regular PGD attack if we use ℓ=CEℓCE =CEℓ = CE and fix ^=affirmative^subscriptaffirmative Y=\ y_affirmative\over start_ARG Y end_ARG = italic_yaffirmative . Details. The subsequent additions to/clarifications of Algo. 2 follow Geisler et al. (2024). We run PGD for a batch of behaviors/prompts in parallel, use Adam (Kingma & Ba, 2015) instead of vanilla gradient descent, and reinitialize the attack to the best intermediate solution, bestsubscriptbest x_bestitalic_xbest, if a configurable number of iterations (patience=100) fails to improve. In case we run out of patience, we sample an effective prefix/suffix for a different prompt in the batch with a 50% chance. The initial so-called entropy projection is linearly ramped up, followed by cosine annealing with warm restarts Loshchilov & Hutter (2017) for both the learning rate and entropy projection. See § B for more explanations. Algo. 2 REINFORCE Projected Gradient Descent (PGD) Input: Initial pro. ~(1)superscript~1 X^(1)over~ start_ARG italic_X end_ARG( 1 ), loss ℓ ℓ, sampling strategy Pfθ′⁢(Y|X)superscriptsubscriptsubscript′conditionalP_f_θ (Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ), discretization d⁢(~)~d( X)d ( over~ start_ARG italic_X end_ARG ), projection Π Π, # iterations E, # samples K for i∈1,2,…,E12…i∈\1,2,…,E\i ∈ 1 , 2 , … , E do ^←(1),…,(K)∼Pfθ′⁢(Y|d⁢(~(i)))←^superscript1…superscriptsimilar-tosubscriptsuperscript′subscriptconditionalsuperscript~ Y←\ y^(1),…, y^(K)\ P^% _f_θ(Y|d( X^(i)))over start_ARG Y end_ARG ← italic_y( 1 ) , … , italic_y( K ) ∼ P′italic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( Y | d ( over~ start_ARG italic_X end_ARG( i ) ) ) ▷ ▷ Generate ←∇~(i)ℓ⁢(^,~(i))←subscript∇superscript~ℓ^superscript~ G← _ X^(i) ( Y,% X^(i))italic_G ← ∇over~ start_ARG italic_X end_ARG( i ) ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG( i ) ) ▷ ▷ Gradient ~(i+1)←Π⁢(~(i)−α⁢)←superscript~1Πsuperscript~ X^(i+1)← ( X^(i)-α G)over~ start_ARG italic_X end_ARG( i + 1 ) ← Π ( over~ start_ARG italic_X end_ARG( i ) - α italic_G ) ▷ ▷ Update ~(i+1)←d⁢(~(i+1))←superscript~1superscript~1 x^(i+1)← d( X^(i+1))over~ start_ARG italic_x end_ARG( i + 1 ) ← d ( over~ start_ARG italic_X end_ARG( i + 1 ) ) ▷ ▷ Discretization ℓ~←ℓ⁢(^,~(i+1))←~ℓ^superscript~1 ← ( Y, x^(i+1))over~ start_ARG ℓ end_ARG ← ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_x end_ARG( i + 1 ) ) ▷ ▷ “Discretized loss” ~best←~(i+1)⁢ if ⁢is⁢_⁢best⁡(ℓ~)⁢ else ⁢~best←subscript~bestsuperscript~1 if is_best~ℓ else subscript~best x_best← x^(i+1) if % is\_best( ) else x_% bestover~ start_ARG italic_x end_ARGbest ← over~ start_ARG italic_x end_ARG( i + 1 ) if start_OPFUNCTION is _ best end_OPFUNCTION ( over~ start_ARG ℓ end_ARG ) else over~ start_ARG italic_x end_ARGbest ▷ ▷ Remember best return ~bestsubscript~best x_bestover~ start_ARG italic_x end_ARGbest 4 Experimental Evaluation We evaluate our REINFORCE-GCG (Algo. 1) and REINFORCE-PGD (Algo. 2) attacks that maximize P⁢(Harmful|X=x~)conditionalHarmful~P(Harmful|X= x)P ( Harmful | X = over~ start_ARG x end_ARG ) on state-of-the-art aligned LLMs on standard behaviors from HarmBench (Mazeika et al., 2024). We contrast the results primarily to the affirmative-response variants of GCG (Zou et al., 2023) and PGD (Geisler et al., 2024). With minor exceptions that we explicitly state in the following, we use the respective attack’s default settings. 4.1 Setup Benchmark. We use all 200 default behaviors from HarmBench’s standard behaviors (Mazeika et al., 2024), which excludes the copyright and contextual behaviors. For the runtime comparisons in Fig. 3 and ablations in § 4.3, we subsample 50 behaviors. We exclude them solely due to constraints on our computational budget. For the evaluation, we report the attack success rate (ASR) using HarmBench’s judge, which is based on Llama 2 13B. Following HarmBench, we report the ASR usually with 512 tokens (ASR@512), where we generate 512 tokens with the target model and then right-truncate the generated string to 512 tokens using the judge’s tokenizer. Note that during the attack we solely rely on the reward of up to 128 tokens. Following HarmBench, we only score the greedy generation, which can be seen as an estimate of Pθ⁢(Y|X=~)subscriptconditional~P_θ(Y|X= x)Pitalic_θ ( Y | X = over~ start_ARG italic_x end_ARG )’s mode. We report a single trial for each prompt and execute all experiments on 80 GB A100/H100 GPUs. We use bfloat16 for attacked models and judge and compile the generation. Models. We attack Llama 2 7B (Touvron et al., 2023), Llama 3 8B (Grattafiori et al., 2024), Gemma 1.1 2B and 7B (DeepMind et al., 2024), as well as Vicuna 1.5 7B (Zheng et al., 2023). Moreover, we study the state-of-the-art defense based on circuit breakers (Zou et al., 2024) for Llama 3 8B. Attacks. For our evaluation, we use HarmBench’s implementation/configuration for Greedy Coordinate Gradient (GCG) from Zou et al. (2023): search width S=512512S=512S = 512, E=500500E=500E = 500 iterations, and initialize the adversarial suffix with 20 “!”. For REINFORCE-GCG we use HarmBench’s Llama 2 13B judge as reward signal. For comparability, we stick to this default setup, although more elaborate templates for adversarial attacks have shown to be a promising direction for enhancing attack efficacy (Andriushchenko et al., 2025). For Projected Gradient Descent (PGD) from Geisler et al. (2024), we follow the original hyperparameters. We run PGD for E=5,0005000E=5,000E = 5 , 000 iterations, use a learning rate of α=0.110.11α=0.11α = 0.11, entropy projection of 40%percent4040\%40 %, and initialize prefix as well as suffix with 25 “!”. One of the few differences to the setup of Geisler et al. (2024) is that we keep the prompt length fixed to avoid model-specific code changes in the used Huggingface’s transformers library (Wolf et al., 2020). Due to the importance of batching for PGD’s (amortized) cost, we use HarmBench’s Mistral 7B judge instead of Llama 2 13B during the optimization. Due to the 80 GB GPU RAM limitation, we set the batch size to 17 prompts instead of 30 to 50. Specifically, we attack 17 prompts in parallel and report amortized runtime (total time divided by number of prompts). For REINFORCE-GCG and -PGD, we keep the other hyperparameters of the base attack unchanged. We use seed=affirmativesubscriptseedsubscriptaffirmative y_seed= y_affirmativeitalic_yseed = italic_yaffirmative unless stated otherwise. Reward. Since we use HarmBench’s judge, we design the reward in analogy to its intended use. That is, an attack is deemed successful if the judge’s greedy generation is “yes” regardless of capitalization. Since the greedy generation always returns the next token which is most likely, our reward definition hovers around the most likely “yes” p∗=arg⁢maxp⁢ s.t. ⁢l⁡(p)="⁢y⁢e⁢s⁢"⁡Pjudge⁢(p|,)superscriptsubscriptargmax s.t. l""subscriptjudgeconditionalp^*= *arg\,max_p s.t.\ l(p)="yes"P_% judge(p| y, x)p∗ = start_OPERATOR arg max end_OPERATORp s.t. l ( p ) = " y e s " Pjudge ( p | italic_y , italic_x ) and most likely non-“yes” token n∗=arg⁢maxn⁢ s.t. ⁢l⁡(n)≠"⁢y⁢e⁢s⁢"⁡Pjudge⁢(n|,)superscriptsubscriptargmax s.t. l""subscriptjudgeconditionaln^*= *arg\,max_n s.t.\ l(n)≠"yes"P_% judge(n| y, x)n∗ = start_OPERATOR arg max end_OPERATORn s.t. l ( n ) ≠ " y e s " Pjudge ( n | italic_y , italic_x ). l(.)l(.)l ( . ) is short for lower(.)lower(.)lower ( . ). Inspired by logistic regression, we then take their log-odds/logits log⁡(Pjudge⁢(p∗|,)/Pjudge⁢(n∗|,))subscriptjudgeconditionalsuperscriptsubscriptjudgeconditionalsuperscript ( P_judge(p^*| y, x)P_judge(% n^*| y, x))log ( / start_ARG Pjudge ( p∗ | italic_y , italic_x ) end_ARG start_ARG Pjudge ( n∗ | italic_y , italic_x ) end_ARG ) and transform them via sigmoid function σ, which is the inverse of the (binary) logit function: Reward⁡(y,x~)=Harmfulness⁡(y,x~)Reward~Harmfulness~ (y, x)=Harmfulness(y,% x)Reward ( y , over~ start_ARG x end_ARG ) = Harmfulness ( y , over~ start_ARG x end_ARG ) (10) =σ⁢(log⁡Pjudge⁢(p∗|,)⏟“yes” log prob.−log⁡Pjudge⁢(n∗|,)⏟most likely non-“yes” log prob.)absentsubscript⏟subscriptjudgeconditionalsuperscript“yes” log prob.subscript⏟subscriptjudgeconditionalsuperscriptmost likely non-“yes” log prob. =σ ( P_judge(p^*| y,% x)_``yes′ log prob.- P_judge(n^*|% y, x)_most likely non-``yes′ log prob. )= σ ( under⏟ start_ARG log Pjudge ( p∗ | italic_y , italic_x ) end_ARG“yes” log prob. - under⏟ start_ARG log Pjudge ( n∗ | italic_y , italic_x ) end_ARGmost likely non-“yes” log prob. ) If Reward⁡(y,x~)>0.5Reward~0.5Reward(y, x)>0.5Reward ( y , over~ start_ARG x end_ARG ) > 0.5 the target metric is true and vice versa. Due to the sigmoid Reward⁡(y,x~)∈[0,1]Reward~01Reward(y, x)∈[0,1]Reward ( y , over~ start_ARG x end_ARG ) ∈ [ 0 , 1 ]. This is desired for equivalence interpretation of our REINFORCE objective to the maximization of P⁢(Harmful|X=x~)conditionalHarmful~P(Harmful|X= x)P ( Harmful | X = over~ start_ARG x end_ARG ). Table 1: ASR@512 for GCG (Zou et al., 2023), i.e. attack success rate (ASR) with 512 generated/judge tokens on the 200 prompts in standard HarmBench (Mazeika et al., 2024). Affirmative REINFORCE (ours) Gemma 1.1 2B 0.57 0.88 Gemma 1.1 7B 0.63 0.87 Llama 2 7B 0.32 0.56 Llama 3 8B 0.35 0.73 Vicuna 1.5 7B 0.86 0.95 Table 2: ASR@512 for PGD of Geisler et al. (2024). Affirmative REINFORCE (ours) Gemma 1.1 2B 0.56 0.82 Gemma 1.1 7B 0.54 0.84 Llama 2 7B 0.17 0.22 Llama 3 8B 0.57 0.69 Vicuna 1.5 7B 0.87 0.94 4.2 Main Results We report the results for REINFORCE-GCG in Table 1 and REINFORCE-PGD in Table 2. We observe consistent and substantial gains over the affirmative objective. For example, the ASR@512 of REINFORCE-GCG is more than double that of affirmative GCG on Llama 3 8B. The only exception is Llama 2 7B, where PGD’s performance improves only marginally. Note that we neither tuned the hyperparameters of the base attacks nor do we use a model-specific configuration. We provide uniformly randomly sampled examples of successful GCG attacks and the models’ responses in § I (truncated to the first six lines). These examples are further evidence that our objective successfully reveals harmful behaviors that are different to affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative. Interestingly, we observe that the models sometimes refuse to answer in the first sentences of their response but ultimately provide the answer (e.g., second example in LABEL:tab:examples_llama3). Table 3: Attack success rate (ASR) with our REINFORCE-GCG for attacking Llama 3 and its circuit-breaker defended version (Zou et al., 2024). * denotes numbers on the subset of successful attacks on the base model with affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative. Affirmative REINFORCE (ours) seed=subscriptseedabsent y_seed=italic_yseed = - affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative historysubscripthistory y_historyitalic_yhistory ASR@ 128 512 128 512 128 512 Llama 3 8B 0.29 0.35 0.66 0.73 - - + Circuit breaker 0.01 0.02 0.21 0.23 0.46∗ 0.50∗ (a) Gemma 1.1 2B (b) Llama 3 8B Figure 3: Our REINFORCE objective provides a good ASR@128/runtime tradeoff in contrast to GCG with its affirmative response objective. Here we show the reward of the judge used during the attack. Runtimes are for H100s. Circuit breaker. We also evaluate the efficacy of our REINFORCE objective on the state-of-the-art defense based on circuit breaking (Zou et al., 2024) and report the results in Table 3. We observe a substantial gain from 2% ASR@512 with the affirmative objective to 23% with REINFORCE. Motivated by the fact that Zou et al. (2024) tuned their model to withstand harmful requests and considering that our experiments run on a hard compute constraint that hinders thorough exploration, we also investigate the impact of better seed responses seedsubscriptseed y_seeditalic_yseed. Specifically, we take the most harmful generation we found via successful attacks on the base model Llama 3 8B, which itself is already safety-tuned. On this subset of the data, we achieve a strong ASR@512 of 50%. However, it should be noted that the circuit-breaking defense often produces stutter-like behavior late in their generation. § J contains randomly sampled examples of our attacks and the model’s responses. Nevertheless, even though the circuit breaker defense almost always repels an affirmative GCG attack for an adversarial suffix of 20 tokens, our REINFORCE attack achieves a high success rate. This finding underlines the importance of an (asymptotically) consistent objective for adaptive attacks on LLMs. Compute performance tradeoff. We next investigate the compute performance tradeoff in Fig. 3. Specifically, we contrast the ASR@128 for GCG with affirmative response and E=5,0005000E=5,000E = 5 , 000 attack iterations to our REINFORCE-GCG and REINFORCE-PGD for the light-weight Gemma 1.1 2B (Fig. 3a) and Llama 3 8B (Fig. 3b). For PGD, we report the judge used for reward calculation (Mistral). We observe that our REINFORCE optimization not only achieves strong terminal ASRs but also is computationally efficient. REINFORCE-PGD performs particularly strong for Gemma 1.1 2B (Fig. 3a) and, for Llama 3 8B (Fig. 3b), our REINFORCE jailbreak attacks outperform standard GCG on most time scales. Note that we did not specifically tune our attacks w.r.t. the runtime vs. performance tradeoff. Figure 4: Ablations of search and selection strategies for GCG on Llama 3 8B and ASR@512. We either select the mutated candidates (a) randomly, (b) using the gradient of the affirmative response, (c) or REINFORCE. We select the best candidate either according to (1) the affirmative response or (2) REINFORCE. (a) Terminal rewards (b) Cross entropies of gradient calculation (c) Cross entropies of seedsubscriptseed y_seeditalic_yseed for mutations (d) Cross entropies of greedysubscriptgreedy y_greedyitalic_ygreedy for mutations (e) Cross entropies of harmfulsubscriptharmful y_harmfulitalic_yharmful for mutations Figure 5: Random example for an attack on Llama 3 8B (first 50 steps). As we show in (a), in attack step 7, the model’s random generation randomsubscriptrandom y_randomitalic_yrandom is harmful, which we then include as harmfulsubscriptharmful y_harmfulitalic_yharmful. In step 17, also greedysubscriptgreedy y_greedyitalic_ygreedy becomes harmful. Thereafter, also the harmfulness of random generations rises (see moving average MAMAMAMA). As shown in (b), already small changes in Pfθ⁢(seed)=−CE⁡(seed)subscriptsubscriptsubscriptseedCEsubscriptseedP_f_θ( y_seed)=-CE( y_seed% )Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( italic_yseed ) = - CE ( italic_yseed ) may suffice to obtain harmful generations. Specifically, CE⁡(seed)CEsubscriptseedCE( y_seed)CE ( italic_yseed ) decreases for the first iterations and increases again after the greedy generation becomes harmful. (c-e) show histograms of the mutation’s CECECECEs. 4.3 Ablations and Insights Table 4: Ablation study of the main REINFORCE-specific design choice: the sampling strategy Pfθ′⁢(Y|X)superscriptsubscriptsubscript′conditionalP_f_θ (Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ). We use GCG (first row of table) as the base attack on Llama 3 8B. extend seedsubscriptseed y_seeditalic_yseed w/ seedsubscriptseed y_seeditalic_yseed w/ randomsubscriptrandom y_randomitalic_yrandom w/ harmfulsubscriptharmful y_harmfulitalic_yharmful w/ greedysubscriptgreedy y_greedyitalic_ygreedy ASR@128 ASR@512 ✗ ✓ ✗ ✗ ✗ 0.29 0.35 ✗ ✓ ✓ ✗ ✗ 0.22 0.36 ✓ ✓ ✓ ✗ ✗ 0.24 0.34 ✓ ✓ ✓ ✓ ✗ 0.42 0.56 ✓ ✓ ✓ ✓ ✓ 0.66 0.73 REINFORCE ablations. In Table 4, we provide an ablation study for the different responses of our sampling strategy Pfθ′⁢(Y|X)superscriptsubscriptsubscript′conditionalP_f_θ (Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ). Surprisingly, adding a random generation randomsubscriptrandom y_randomitalic_yrandom to the common affirmative objective seed=affirmativesubscriptseedsubscriptaffirmative y_seed= y_affirmativeitalic_yseed = italic_yaffirmative does not yield improvement. On the contrary, it even seems to hurt performance. Extending seedsubscriptseed y_seeditalic_yseed greedily to 128 tokens to harmonize the lengths seems to improve performance marginally. However, adding harmfulsubscriptharmful y_harmfulitalic_yharmful does yield substantial improvement. Last, our full sampling strategy that also includes greedysubscriptgreedy y_greedyitalic_ygreedy achieves the best performance. GCG-specific ablations. In Fig. 4, we investigate the GCG-specific aspects of mutating the prompt and selecting the best candidate on Llama 3 8B. Additionally to the affirmative objective log⁡Pθ⁢(affirmative|~)subscriptconditionalsubscriptaffirmative~ P_θ( y_affirmative| x)log Pitalic_θ ( italic_yaffirmative | over~ start_ARG italic_x end_ARG ), we also study uniformly random mutations. For each trial that involves our REINFORCE objective, we also use it to select the best prompt ~(i)superscript~ X^(i)over~ start_ARG italic_X end_ARG( i ) for the ASR evaluation. The random mutations (blue) are consistently the worst choice, which indicates that the gradient is somewhat informative for flipping entries in the one hot matrix ~~ Xover~ start_ARG italic_X end_ARG. However, it should be noted that random mutations combined with our objective to select the best candidate (solid blue) outperform vanilla GCG by a large margin (hatched orange). This finding is likely tied to the success of so-called Best-of-N (BoN) attacks (Hughes et al., 2024). Moreover, our REINFORCE gradient seems to help even if using the affirmative selection criterion (hatched green). The most surprising result is perhaps that the affirmative gradient suffices in combination with our REINFORCE objective for selection (solid orange). We hypothesize that this is due to the excessive search width of S=512512S=512S = 512 and the specifics of GCG’s mutation sampling. In each step, GCG samples a diverse set of candidates–even with affirmative gradient ∇~log⁡Pθ⁢(affirmative|~)subscript∇~subscriptconditionalsubscriptaffirmative~ _ X P_θ( y_affirmative| % X)∇over~ start_ARG italic_X end_ARG log Pitalic_θ ( italic_yaffirmative | over~ start_ARG italic_X end_ARG ). Choice of yseedsubscriptseed y_seeditalic_yseed. On Llama 3 8B (+ circuit breaker) we also run an experiment with the model-specific targets obtained via AdvPrefix (Zhu et al., 2024). We find that our REINFORCE objective is not overly sensitive to the specific choice of seedsubscriptseed y_seeditalic_yseed; however, benefits from a sensible choice. Taking also the results from our experiments with historysubscripthistory y_historyitalic_yhistory with circuit breakers (Table 3) into account, it appears that making seedsubscriptseed y_seeditalic_yseed sufficiently likely must be sufficiently easy. Moreover, we see the advantage of a dynamic objective like our REINFORCE on the state-of-the-art defense via circuit breaking. For example, the affirmative objective with AdvPrefix achieves an ASR@512 of 0.14, while our REINFORCE achieves 0.5. We refer to § G for details. Dynamics. In Fig. 5, we plot the early rewards and the cross entropies. Here we show that a random generation randomsubscriptrandom y_randomitalic_yrandom which is harmful may then be used as harmfulsubscriptharmful y_harmfulitalic_yharmful. A few steps later, also the greedy generation becomes harmful, and the harmfulness of random generations rises. For this, we do not even require seedsubscriptseed y_seeditalic_yseed to become very likely, and the attack may pursue generations that are very different. For more details and comments on consistency, see § H. Moreover, for a comparison of attack and evaluation reward see § F. 5 Related Work Adversarial attacks, and specifically jailbreak attacks or automated red teaming, can be categorized into (1) optimization based attacks (Wallace et al., 2021; Shin et al., 2020; Guo et al., 2021; Zou et al., 2023; Geisler et al., 2024; Guo et al., 2024; Wen et al., 2023; Kumar et al., 2022; Hou et al., 2023; Liu et al., 2024; Zhu et al., 2023; Andriushchenko et al., 2025; Schwinn et al., 2024; Hughes et al., 2024; Sadasivan et al., 2024; Thompson & Sklar, 2024) or (2) attacks using generative models/LLMs (Perez et al., 2022; Mehrotra et al., 2023; Chao et al., 2023; Liao & Sun, 2024; Chen et al., 2024; Jha et al., 2024; Lin et al., 2024). Our novel REINFORCE objective is focusing on approaches of category (1) that use gradient information (Wallace et al., 2021; Shin et al., 2020; Guo et al., 2021; Zou et al., 2023; Geisler et al., 2024; Guo et al., 2024; Wen et al., 2023; Kumar et al., 2022; Hou et al., 2023; Zhu et al., 2023). Specifically, we extend two gradient-based attacks, namely GCG (Zou et al., 2023) and PGD (Geisler et al., 2024). While GCG builds on top of the language model attack AutoPromopt (Shin et al., 2020), PGD relates to adversarial attacks on GNNs (Xu et al., 2020; Geisler et al., 2021; Gosch et al., 2023; Foth et al., 2024). While ours is not the first jailbreak attack that uses reinforcement learning (RL), usually RL-based approaches are of category (2) and train another LLM to generate prompts for jailbreaking the targeted LLM (Perez et al., 2022; Mehrotra et al., 2023; Chao et al., 2023; Chen et al., 2024; Jha et al., 2024; Lin et al., 2024). Even though we target gradient-based optimization, our objective is also very effective for GCG with random mutations (see Fig. 4). Thus, approaches like Andriushchenko et al. (2025); Liu et al. (2024) could also benefit from our advanced objective. Andriushchenko et al. (2025) explore adaptive attacks. In contrast to our work, they add certain (potentially model-specific) features to their attack that improve the attack success rates (e.g., self-transfer, prompt templates, restarts, etc.). Most of their strategies are orthogonal to ours and could further improve REINFORCE-GCG / -PGD. Nevertheless, it should be noted that they also rely on a non-consistent attack objective (e.g., log probability of “Sure”). Our findings with adaptive attacks on the circuit breaker defenses align with embedding space attacks Schwinn & Geisler (2024). Adversarial attacks that capture rich semantics have been explored before (Qiu et al., 2020; Geisler et al., 2022; Wang et al., 2023; Kollovieh et al., 2024). Notably, Wichers et al. (2024) also explore attacks on LLMs that leverage a judge. In contrast to our work, they relax the generation using Gumble softmax (Jang et al., 2016) and backpropagate through the attacked model and judge for a GBDA-based optimization (Guo et al., 2021). Affirmative response. Even though virtually all of the aforementioned optimization-based jailbreak attacks use an affirmative response objective, there have been attempts to mitigate its limitation via varying the template (Jia et al., 2024) or distilling responses from a modified model (Thompson & Sklar, 2024). Concurrently, AdvPrefix of Zhu et al. (2024) finds better response prefixes via high prefilling attack success rates and low negative log-likelihood. Similar to them, our REINFORCE objective also alleviates the issue that the affirmative responses are rather short (Qi et al., 2025). We demonstrate the complementary strengths of AdvPrefix and our objective in § G. Judges and evaluation. Multiple judges have been introduced in the literature (Shen et al., 2024; Chao et al., 2023; Bhatt et al., 2023). We use the HarmBench (Mazeika et al., 2024) judge since we evaluate on HarmBench. Nevertheless, we note the recent work by Souly et al. (2024) as their judge achieves a slightly better alignment to human evaluation. Moreover, since current LLM-as-a-judge models for scoring harmfulness are all imperfect, one could augment their scores with frequent patterns for false positives (non-harmful prompts that are judged harmful) like Hughes et al. (2024). Next to a judge-based evaluation, one could also gain insights into our objective through (mechanistic) interpretability (Arditi et al., 2024; Wollschläger et al., 2025). 6 Limitations Due to the use of an LLM-as-a-judge reward signal, the judges’ shortcomings influence our REINFORCE objective. However, we do not observe systematic issues, as we demonstrate in the random examples in § I. While we leave improving judges for future work, extrapolating previous developments, one can expect that they will steadily advance over time, and along with the advancements of judges, our REINFORCE attack objective may steadily improve. Our evaluation relies on the greedy generation for the evaluation. A harmful greedy generation neither implies that other (random) generation must be harmful (Scholten et al., 2025) nor is it a very stable choice in ~~ xover~ start_ARG italic_x end_ARG. Additionally, in practice, more sophisticated decoding schemes have been used that impact the model’s alignment Balashankar et al. (2024). Importantly, we avoided design decisions as much as possible that solely help due to the greedy evaluation. 7 Conclusion and Discussion We propose an adaptive, distributional, and semantic objective rooted in reinforcement learning that overcomes the static nature of current attack objectives for jailbreaking generative models/LLMs. We show that our objective aims to optimize the probability for the model to output a harmful response (assumptions apply), and we implement our objective via REINFORCE. We demonstrate the efficacy empirically by utilizing the state-of-the-art jailbreak attacks PGD and GCG for our adversarial optimization procedure to attack state-of-the-art aligned LLMs. While our approach yields more effective jailbreaks for current models, we argue that adaptive objectives will be a cornerstone for the rigorous evaluation of future models. With the rising capabilities of LLMs, it is unlikely that human-engineered inappropriate/incorrect responses to a given prompt are the way to go in identifying model misbehavior. Similarly to tasks like scalable oversight (Bowman et al., 2022), we expect humans will have trouble evaluating sufficiently smart or sophisticated models. Hence, we believe that our REINFORCE objective and its theoretical underpinning will be an important framework for the (offline) assessment of such models’ failure modes. Acknowledgements This research was supported by the Center for AI Safety Compute Cluster. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. Further, this material is based on work partially funded by Google. We also thank Leo Schwinn and Tim Beyer for their feedback and the related discussions. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning, and specifically generative models, in terms of reliability. We mitigate the risk of adversarial use of our adversarial attack objective via studying the white-box scenario. That is, the attacker needs full knowledge about the model architecture, including implementation details, and the model parameters. Hence, the biggest risk arises for models that have been released openly. Nevertheless, from a cybersecurity perspective, it is usually best to reveal security risks s.t. future versions/models can deploy appropriate fixes. Hence, we believe that the availability of our method to model developers outweighs the risks of its adversarial use. References Ackley et al. (1985) Ackley, D. H., Hinton, G. E., and Sejnowski, T. J. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147–169, 1985. ISSN 0364-0213. doi: 10.1016/S0364-0213(85)80012-4. URL https://w.sciencedirect.com/science/article/pii/S0364021385800124. Ahmadian et al. (2024) Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. In 62nd Annual Meeting of the Association for Computational Linguistics, ACL. arXiv, 2024. URL http://arxiv.org/abs/2402.14740. arXiv:2402.14740 [cs]. Andriushchenko et al. (2025) Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. In International Conference on Learning Representations, ICLR. arXiv, 2025. URL http://arxiv.org/abs/2404.02151. arXiv:2404.02151 [cs, stat]. Arditi et al. (2024) Arditi, A., Obeso, O. B., Syed, A., Paleka, D., Rimsky, N., Gurnee, W., and Nanda, N. Refusal in Language Models Is Mediated by a Single Direction. In Neural Information Processing Systems, NeruIPS, 2024. URL https://arxiv.org/abs/2406.11717. Athalye et al. (2018) Athalye, A., Carlini, N., and Wagner, D. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In International Conference on Machine Learning, ICML, 2018. URL http://arxiv.org/abs/1802.00420. Balashankar et al. (2024) Balashankar, A., Sun, Z., Berant, J., Eisenstein, J., Collins, M., Hutter, A., Lee, J., Nagpal, C., Prost, F., Sinha, A., Suresh, A. T., and Beirami, A. InfAlign: Inference-aware language model alignment, 2024. URL http://arxiv.org/abs/2412.19792. arXiv:2412.19792 [cs]. Bhatt et al. (2023) Bhatt, M., Chennabasappa, S., Nikolaidis, C., Wan, S., Evtimov, I., Gabi, D., Song, D., Ahmad, F., Aschermann, C., Fontana, L., Frolov, S., Giri, R. P., Kapil, D., Kozyrakis, Y., LeBlanc, D., Milazzo, J., Straumann, A., Synnaeve, G., Vontimitta, V., Whitman, S., and Saxe, J. Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models, 2023. URL http://arxiv.org/abs/2312.04724. arXiv:2312.04724 [cs]. Bowman et al. (2022) Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukošiūtė, K., Askell, A., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Olah, C., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Kernion, J., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lovitt, L., Elhage, N., Schiefer, N., Joseph, N., Mercado, N., DasSarma, N., Larson, R., McCandlish, S., Kundu, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Telleen-Lawton, T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hatfield-Dodds, Z., Mann, B., and Kaplan, J. Measuring Progress on Scalable Oversight for Large Language Models, 2022. URL http://arxiv.org/abs/2211.03540. arXiv:2211.03540 [cs]. Carlini & Wagner (2017) Carlini, N. and Wagner, D. Towards Evaluating the Robustness of Neural Networks. IEEE Symposium on Security and Privacy, p. 39–57, 2017. ISSN 9781509055326. doi: 10.1109/SP.2017.49. Chao et al. (2023) Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking Black Box Large Language Models in Twenty Queries, 2023. URL http://arxiv.org/abs/2310.08419. arXiv:2310.08419 [cs]. Chen et al. (2024) Chen, X., Nie, Y., Yan, L., Mao, Y., Guo, W., and Zhang, X. RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs, 2024. URL http://arxiv.org/abs/2406.08725. arXiv:2406.08725 [cs]. DeepMind et al. (2024) DeepMind, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Sessa, P. G., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Choo, C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G.-C., Rozhdestvenskiy, G., Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., Labanowski, J., Lespiau, J.-B., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., Mao-Jones, J., Lee, K., Yu, K., Millican, K., Sjoesund, L. L., Lee, L., Dixon, L., Reid, M., Mikuła, M., Wirth, M., Sharman, M., Chinaev, N., Thain, N., Bachem, O., Chang, O., Wahltinez, O., Bailey, P., Michel, P., Yotov, P., Chaabouni, R., Comanescu, R., Jana, R., Anil, R., McIlroy, R., Liu, R., Mullins, R., Smith, S. L., Borgeaud, S., Girgin, S., Douglas, S., Pandya, S., Shakeri, S., De, S., Klimenko, T., Hennigan, T., Feinberg, V., Stokowiec, W., Chen, Y.-h., Ahmed, Z., Gong, Z., Warkentin, T., Peran, L., Giang, M., Farabet, C., Vinyals, O., Dean, J., Kavukcuoglu, K., Hassabis, D., Ghahramani, Z., Eck, D., Barral, J., Pereira, F., Collins, E., Joulin, A., Fiedel, N., Senter, E., Andreev, A., and Kenealy, K. Gemma: Open Models Based on Gemini Research and Technology, 2024. URL http://arxiv.org/abs/2403.08295. arXiv:2403.08295 [cs]. Ficler & Goldberg (2017) Ficler, J. and Goldberg, Y. Controlling Linguistic Style Aspects in Neural Language Generation, 2017. URL http://arxiv.org/abs/1707.02633. arXiv:1707.02633 [cs]. Foth et al. (2024) Foth, P., Gosch, L., Geisler, S., Schwinn, L., and Günnemann, S. Relaxing Graph Transformers for Adversarial Attacks. In Differentiable Almost Everything Workshop at ICML, 2024. URL http://arxiv.org/abs/2407.11764. arXiv:2407.11764 [cs]. Geisler et al. (2021) Geisler, S., Zügner, D., Bojchevski, A., and Günnemann, S. Attacking Graph Neural Networks at Scale. Deep Learning for Graphs at AAAI Conference on Artificial Intelligence, 2021. URL https://w.dropbox.com/s/ddrwoswpz3wwx40/Robust_GNNs_at_Scale__AAAI_Workshop_2020_CameraReady.pdf?dl=0. Geisler et al. (2022) Geisler, S., Sommer, J., Schuchardt, J., Bojchevski, A., and Günnemann, S. Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness. In International Conference on Learning Representations, ICLR, 2022. URL http://arxiv.org/abs/2110.10942. arXiv: 2110.10942. Geisler et al. (2024) Geisler, S., Wollschläger, T., Abdalla, M. H. I., Gasteiger, J., and Günnemann, S. Attacking Large Language Models with Projected Gradient Descent. In Next Generation of AI Safety Workshop at ICML, 2024. URL http://arxiv.org/abs/2402.09154. arXiv:2402.09154 [cs]. Gosch et al. (2023) Gosch, L., Geisler, S., Sturm, D., Charpentier, B., Zügner, D., and Günnemann, S. Adversarial Training for Graph Neural Networks: Pitfalls, Solutions, and New Directions. In Neural Information Processing Systems, NeurIPS, 2023. Grattafiori et al. (2024) Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., Linde, J. v. d., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., Maaten, L. v. d., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., Oliveira, L. d., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The Llama 3 Herd of Models, 2024. URL http://arxiv.org/abs/2407.21783. arXiv:2407.21783 [cs]. Guo et al. (2021) Guo, C., Sablayrolles, A., Jégou, H., and Kiela, D. Gradient-based Adversarial Attacks against Text Transformers. In Conference on Empirical Methods in Natural Language Processing, p. 5747–5757, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.464. URL https://aclanthology.org/2021.emnlp-main.464. Guo et al. (2024) Guo, X., Yu, F., Zhang, H., Qin, L., and Hu, B. COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. In International Conference on Machine Learning, ICML. arXiv, 2024. URL http://arxiv.org/abs/2402.08679. arXiv:2402.08679 [cs]. Hou et al. (2023) Hou, B., Jia, J., Zhang, Y., Zhang, G., Zhang, Y., Liu, S., and Chang, S. TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization. International Conference on Learning Representations, ICLR, 2023. Hughes et al. (2024) Hughes, J., Price, S., Lynch, A., Schaeffer, R., Barez, F., Koyejo, S., Sleight, H., Jones, E., Perez, E., and Sharma, M. Best-of-N Jailbreaking, 2024. URL http://arxiv.org/abs/2412.03556. arXiv:2412.03556 [cs]. Jang et al. (2016) Jang, E., Gu, S., and Poole, B. Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations, ICLR, 2016. URL https://openreview.net/forum?id=rkE3y85e. Jha et al. (2024) Jha, P., Arora, A., and Ganesh, V. LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs, 2024. URL http://arxiv.org/abs/2411.08862. arXiv:2411.08862 [cs]. Jia et al. (2024) Jia, X., Pang, T., Du, C., Huang, Y., Gu, J., Liu, Y., Cao, X., and Lin, M. Improved Techniques for Optimization-Based Jailbreaking on Large Language Models, 2024. URL http://arxiv.org/abs/2405.21018. arXiv:2405.21018 [cs]. Jiang et al. (2023) Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7B, 2023. URL http://arxiv.org/abs/2310.06825. arXiv:2310.06825 [cs]. Kingma & Ba (2015) Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, ICLR, 2015. URL http://arxiv.org/abs/1412.6980. Kollovieh et al. (2024) Kollovieh, M., Gosch, L., Scholten, Y., Lienen, M., and Günnemann, S. Assessing Robustness via Score-Based Adversarial Image Generation. Transactions on Machine Learning Research, TMLR, 2024. doi: 10.48550/arXiv.2310.04285. URL http://arxiv.org/abs/2310.04285. arXiv:2310.04285 [cs]. Kool et al. (2019) Kool, W., van Hoof, H., and Welling, M. Buy 4 REINFORCE Samples, Get a Baseline for Free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR Workshop, 2019. Kumar et al. (2022) Kumar, S., Paria, B., and Tsvetkov, Y. Gradient-based Constrained Sampling from Language Models. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Conference on Empirical Methods in Natural Language Processing, EMNLP, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.144. URL https://aclanthology.org/2022.emnlp-main.144. Liao & Sun (2024) Liao, Z. and Sun, H. AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs, 2024. URL http://arxiv.org/abs/2404.07921. arXiv:2404.07921 [cs]. Lin et al. (2024) Lin, Z., Ma, W., Zhou, M., Zhao, Y., Wang, H., Liu, Y., Wang, J., and Li, L. PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach, 2024. URL http://arxiv.org/abs/2409.14177. arXiv:2409.14177 [cs]. Liu et al. (2024) Liu, X., Xu, N., Chen, M., and Xiao, C. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In International Conference on Learning Representations, ICLR, 2024. URL http://arxiv.org/abs/2310.04451. arXiv:2310.04451 [cs]. Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, ICLR, 2017. Mazeika et al. (2024) Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. In International Conference on Machine Learning, ICML. arXiv, 2024. URL http://arxiv.org/abs/2402.04249. arXiv:2402.04249. Mehrotra et al. (2023) Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., and Karbasi, A. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, 2023. URL http://arxiv.org/abs/2312.02119. arXiv:2312.02119 [cs, stat]. Mujkanovic et al. (2022) Mujkanovic, F., Geisler, S., Günnemann, S., and Bojchevski, A. Are Defenses for Graph Neural Networks Robust? In Neural Information Processing Systems, NeurIPS, 2022. arXiv:2301.13694 [cs]. Perez et al. (2022) Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red Teaming Language Models with Language Models. In Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP, 2022. URL http://arxiv.org/abs/2202.03286. arXiv:2202.03286 [cs]. Qi et al. (2025) Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety Alignment Should Be Made More Than Just a Few Tokens Deep. In International Conference on Learning Representations, ICLR. arXiv, 2025. URL http://arxiv.org/abs/2406.05946. arXiv:2406.05946 [cs]. Qiu et al. (2020) Qiu, H., Xiao, C., Yang, L., Yan, X., Lee, H., and Li, B. SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing. In European Conference on Computer Vision, ECCV, 2020. doi: 10.48550/arXiv.1906.07927. URL http://arxiv.org/abs/1906.07927. arXiv:1906.07927 [cs]. Sadasivan et al. (2024) Sadasivan, V. S., Saha, S., Sriramanan, G., Kattakinda, P., Chegini, A., and Feizi, S. Fast Adversarial Attacks on Language Models In One GPU Minute. In International Conference on Machine Learning, ICML. arXiv, 2024. URL http://arxiv.org/abs/2402.15570. arXiv:2402.15570 [cs]. Scholten et al. (2025) Scholten, Y., Günnemann, S., and Schwinn, L. A Probabilistic Perspective on Unlearning and Alignment for Large Language Models. In International Conference on Learning Representations, ICLR, 2025. doi: 10.48550/arXiv.2410.03523. URL http://arxiv.org/abs/2410.03523. arXiv:2410.03523 [cs]. Schwinn & Geisler (2024) Schwinn, L. and Geisler, S. Revisiting the Robust Alignment of Circuit Breakers, 2024. URL http://arxiv.org/abs/2407.15902. arXiv:2407.15902. Schwinn et al. (2024) Schwinn, L., Dobre, D., Xhonneux, S., Gidel, G., and Gunnemann, S. Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space. In Neural Information Processing Systems, NeruIPS, 2024. URL http://arxiv.org/abs/2402.09063. arXiv:2402.09063 [cs]. Shen et al. (2024) Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. ”Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communications Security, p. 1671–1685, Salt Lake City UT USA, 2024. ACM. ISBN 9798400706363. doi: 10.1145/3658644.3670388. URL https://dl.acm.org/doi/10.1145/3658644.3670388. Shin et al. (2020) Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Conference on Empirical Methods in Natural Language Processing, EMNLP, 2020. doi: 10.48550/arXiv.2010.15980. URL http://arxiv.org/abs/2010.15980. arXiv:2010.15980 [cs]. Souly et al. (2024) Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., and Toyer, S. A StrongREJECT for Empty Jailbreaks, 2024. URL http://arxiv.org/abs/2402.10260. arXiv:2402.10260 [cs]. Thompson & Sklar (2024) Thompson, T. B. and Sklar, M. FLRT: Fluent Student-Teacher Redteaming, 2024. URL http://arxiv.org/abs/2407.17447. arXiv:2407.17447 [cs]. Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. URL http://arxiv.org/abs/2307.09288. arXiv:2307.09288 [cs]. Tramer et al. (2020) Tramer, F., Carlini, N., Brendel, W., and Madry, A. On Adaptive Attacks to Adversarial Example Defenses. Neural Information Processing Systems, NeurIPS, 2020. URL https://proceedings.neurips.c//paper_files/paper/2020/hash/11f38f8ecd71867b42433548d1078e38-Abstract.html. Wallace et al. (2021) Wallace, E., Feng, S., Kandpal, N., Gardner, M., and Singh, S. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Empirical Methods in Natural Language Processing, EMNLP. arXiv, 2021. URL http://arxiv.org/abs/1908.07125. arXiv:1908.07125 [cs]. Wang et al. (2023) Wang, C., Duan, J., Xiao, C., Kim, E., Stamm, M., and Xu, K. Semantic Adversarial Attacks via Diffusion Models. In British Machine Vision Conference, BMVC, 2023. doi: 10.48550/arXiv.2309.07398. URL http://arxiv.org/abs/2309.07398. arXiv:2309.07398 [cs]. Wen et al. (2023) Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., and Goldstein, T. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery. In Neural Information Processing Systems, NeruIPS, 2023. URL https://arxiv.org/abs/2302.03668v2. Wichers et al. (2024) Wichers, N., Denison, C., and Beirami, A. Gradient-Based Language Model Red Teaming, 2024. URL http://arxiv.org/abs/2401.16656. arXiv:2401.16656 [cs]. Williams (1992) Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, 1992. ISSN 1573-0565. doi: 10.1007/BF00992696. URL https://link-springer-com.eaccess.tum.edu/article/10.1007/BF00992696. Company: Springer Distributor: Springer Institution: Springer Label: Springer Number: 3 Publisher: Kluwer Academic Publishers. Wolf et al. (2020) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., Platen, P. v., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. HuggingFace’s Transformers: State-of-the-art Natural Language Processing, 2020. URL http://arxiv.org/abs/1910.03771. arXiv:1910.03771 [cs]. Wollschläger et al. (2025) Wollschläger, T., Elstner, J., Geisler, S., Cohen-Addad, V., Günnemann, S., and Gasteiger, J. The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence, 2025. arXiv. Xu et al. (2020) Xu, H., Ma, Y., Liu, H.-C., Deb, D., Liu, H., Tang, J.-L., and Jain, A. K. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. International Journal of Automation and Computing, 17(2):151–178, 2020. ISSN 1476-8186, 1751-8520. doi: 10.1007/s11633-019-1211-x. URL http://link.springer.com/10.1007/s11633-019-1211-x. Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Neural Information Processing Systems, NeruIPS. arXiv, 2023. doi: 10.48550/arXiv.2306.05685. URL http://arxiv.org/abs/2306.05685. arXiv:2306.05685 [cs]. Zhu et al. (2023) Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., and Sun, T. AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models, 2023. URL http://arxiv.org/abs/2310.15140. arXiv:2310.15140 [cs]. Zhu et al. (2024) Zhu, S., Amos, B., Tian, Y., Guo, C., and Evtimov, I. AdvPrefix: An Objective for Nuanced LLM Jailbreaks, 2024. URL http://arxiv.org/abs/2412.10321. arXiv:2412.10321 [cs]. Zou et al. (2023) Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. URL http://arxiv.org/abs/2307.15043. arXiv:2307.15043 [cs]. Zou et al. (2024) Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., Wang, R., Kolter, Z., Fredrikson, M., and Hendrycks, D. Improving Alignment and Robustness with Circuit Breakers, 2024. URL http://arxiv.org/abs/2406.04313. arXiv:2406.04313 [cs]. Appendix A Greedy Coordinate Gradient (GCG) We next provide additional details on GCG and its parametrization. We adopt the implementation details and parametrization of Zou et al. (2023) unless stated otherwise. Mutate. The term “Coordinate” in GCG indicates that one dimension is optimized at a time. This “Coordinate”-property is implemented via the mutation sampler MutateS⁡(,~)subscriptMutate~Mutate_S( G, x)Mutateitalic_S ( italic_G , over~ start_ARG italic_x end_ARG ). We provide the inner workings of the mutation sampler according to HarmBench’s GCG implementation in . Recall that :i:absent:i: i is right exclusive and i::absenti:i : left inclusive (like in Python). Tokenization inconsistencies. Additionally GCG filters out mutated token sequences for which ~(i)≠tokenizer.encode⁢(tokenizer.decode⁢(~(i)))superscript~tokenizer.encodetokenizer.decodesuperscript~ x^(i) .encode(tokenizer.decode(% x^(i)))over~ start_ARG italic_x end_ARG( i ) ≠ tokenizer.encode ( tokenizer.decode ( over~ start_ARG italic_x end_ARG( i ) ) ), unless this condition is true for all mutated strings. Algo. 3 GCG’s MutateS⁡(,~)subscriptMutate~Mutate_S( G, x)Mutateitalic_S ( italic_G , over~ start_ARG italic_x end_ARG ) 1:Input: Gradient ∈ℝT′×||superscriptℝsuperscript′ G ^T ×|V|italic_G ∈ blackboard_RT start_POSTSUPERSCRIPT ′ × | V | end_POSTSUPERSCRIPT, current tokens ~(i)) x^(i))over~ start_ARG italic_x end_ARG( i ) ), search width S 2:Parameters: # top tokens based on gradient k 3:for i∈ℐi ∈ I do 4: i← Top-⁢k⁢(−i,:)←subscript Top-subscript:C_i← Top-k (- G_i,: )Citalic_i ← Top- k ( - italic_Gitalic_i , : ) 5:^← X←\\over start_ARG X end_ARG ← 6:for j∈1,2,…,S12…j∈\1,2,…,S\j ∈ 1 , 2 , … , S do 7: i←jmodT′←modulosuperscript′i← j T i ← j mod T′ 8: ^←^∪~:i⊙[Uniform⁡(i)]⊙~i+1:←^^direct-productsubscript~:absentdelimited-[]Uniformsubscriptsubscript~:1absent X← X∪\ x_:i% [Uniform(C_i)] x_i+1:\over start_ARG X end_ARG ← over start_ARG X end_ARG ∪ over~ start_ARG italic_x end_ARG: i ⊙ [ Uniform ( Citalic_i ) ] ⊙ over~ start_ARG italic_x end_ARGi + 1 : 9:return ^ Xover start_ARG X end_ARG Early stopping. Zou et al. (2023) also implemented an early stopping criterion where the attack is halted once a certain loss value is surpassed. We do not experiment with such a criterion, although it would be straightforward to implement. Disallowed tokens. We also do not allow non-ascii tokens in ~~ xover~ start_ARG italic_x end_ARG and follow HarmBench’s implementation for this criterion. We not only enforce this for GCG but for all attacks in every setting in this work. No fluency loss. For GCG we do not include any fluency loss. Appendix B Projected Gradient Descent (PGD) We next provide additional details on PGD and its parametrization. We fully adopt the implementation details and parametrization of Geisler et al. (2024). Projection(s). The projection Π⁢(~)Π~ ( X)Π ( over~ start_ARG italic_X end_ARG ) ensures (1) that ~~ Xover~ start_ARG italic_X end_ARG remains in [0,1]T′×||superscript01superscript′[0,1]^T ×|V|[ 0 , 1 ]T start_POSTSUPERSCRIPT ′ × | V | end_POSTSUPERSCRIPT after the gradient update and that it remains a row-stochastic matrix ~⁢||=T′~subscript1subscript1superscript′ X1_|V|=1_T over~ start_ARG italic_X end_ARG 1| V | = 1italic_T′. Note that also the one-hot matrix has this property. Additionally, (2) the projection ensures that ~~ Xover~ start_ARG italic_X end_ARG remains “close” to a one-hot matrix by the so-called entropy projection. Hence, the projection Π⁢(~)=Πentropy⁢(Πsimplex⁢(~))Π~subscriptΠentropysubscriptΠsimplex~ ( X)= _entropy( _simplex( X% ))Π ( over~ start_ARG italic_X end_ARG ) = Πentropy ( Πsimplex ( over~ start_ARG italic_X end_ARG ) ) is a composition of (1) the simplex projection Πsimplex⁢(~)subscriptΠsimplex~ _simplex( X)Πsimplex ( over~ start_ARG italic_X end_ARG ) and (2) the entropy projection Πentropy⁢(~)subscriptΠentropy~ _entropy( X)Πentropy ( over~ start_ARG italic_X end_ARG ). We provide pseudo code in Algo. 4 and Algo. 5, respectively. The projections are applied to each token independently. Gradient clipping. Additionally, to the pseudo code we clip the L2 norm of the gradient for each token isubscript G_iitalic_Gitalic_i to 20. This avoids that exploding gradients mess up the momentum terms in the used Adam optimizer. Discretization and tokenization inconsistencies. We denote the discretization with ~=d⁢(~)~~ x=d( X)over~ start_ARG italic_x end_ARG = d ( over~ start_ARG italic_X end_ARG ), which applies a row-wise argmax operation to obtain ~~ xover~ start_ARG italic_x end_ARG. Additionally, we use the attacked model’s tokenizer to avoid encode-decode inconsistencies. Thus, the full discretization procedure is d⁢(~)=tokenizer.encode⁡(tokenizer.decode⁡(arg⁢max⁡(~,axis=−1)))~formulae-sequencetokenizerencodeformulae-sequencetokenizerdecodeargmax~axis1d( X)=tokenizer.encode(tokenizer.% decode( *arg\,max( X,axis=-1)))d ( over~ start_ARG italic_X end_ARG ) = start_OPFUNCTION tokenizer . encode end_OPFUNCTION ( start_OPFUNCTION tokenizer . decode end_OPFUNCTION ( start_OPERATOR arg max end_OPERATOR ( over~ start_ARG italic_X end_ARG , start_OPFUNCTION axis = - 1 end_OPFUNCTION ) ) ). Algo. 4 Simplex Projection ΠsimplexsubscriptΠsimplex _simplexΠsimplex 1:Input: Updated token ∈ℝ||superscriptℝ s ^|T|italic_s ∈ blackboard_R| blackboard_T | 2:Sort sitalic_s into μ1≥μ2≥⋯≥μ||subscript1subscript2⋯subscript _1≥ _2≥…≥ _|T|μ1 ≥ μ2 ≥ ⋯ ≥ μ| blackboard_T | 3:ρ←∑i=1||⁢[μi−1/i⁢(∑j=1iμj−1)>0]←superscriptsubscript1delimited-[]subscript1superscriptsubscript1subscript10ρ← _i=1^|T|I [\ _i- % 1i( _j=1^i _j-1)\>0 ]ρ ← ∑i = 1| blackboard_T | blackboard_I [ μitalic_i - / start_ARG 1 end_ARG start_ARG i end_ARG ( ∑j = 1i μitalic_j - 1 ) > 0 ] 4:ψ←1/ρ⁢(∑j=1ρμj−1)←1superscriptsubscript1subscript1ψ← 1ρ( _j=1^ρ _j-1)ψ ← / start_ARG 1 end_ARG start_ARG ρ end_ARG ( ∑j = 1ρ μitalic_j - 1 ) 5:Return pitalic_p s.t. pi=max⁡si−ψ,0subscriptsubscript0p_i= \s_i-ψ,0\pitalic_i = max sitalic_i - ψ , 0 Algo. 5 Entropy Projection ΠentropysubscriptΠentropy _entropyΠentropy 1:Input: Rel. token ∈[0,1]||superscript01 s∈[0,1]^|T|italic_s ∈ [ 0 , 1 ]| blackboard_T |, target entropy Sq=2subscript2S_q=2Sitalic_q = 2 2:Center ←⁢[>0]/∑i=1||⁢[>0]←delimited-[]0superscriptsubscript1delimited-[]0 c← I[ s>0] _i=1^| % T|I[ s>0]italic_c ← / start_ARG blackboard_I [ italic_s > 0 ] end_ARG start_ARG ∑i = 1| blackboard_T | blackboard_I [ italic_s > 0 ] end_ARG with element-wise >>> and Iblackboard_I 3:Radius R←1−Sq=2−1/∑i=1||⁢[>0]←1subscript21superscriptsubscript1delimited-[]0R← 1-S_q=2- 1 _i=1^|T| % I[ s>0]R ← square-root start_ARG 1 - Sitalic_q = 2 - / start_ARG 1 end_ARG start_ARG ∑i = 1| blackboard_T | blackboard_I [ italic_s > 0 ] end_ARG end_ARG 4:if R≥‖−‖normR≥\| s- c\|R ≥ ∥ italic_s - italic_c ∥ then 5: Return sitalic_s 6:else 7: Return Πsimplex⁢(R/‖−‖⋅(−)+)subscriptΠsimplex⋅norm _simplex( R\| s- c\|·( s-% c)+ c)Πsimplex ( / start_ARG R end_ARG start_ARG ∥ italic_s - italic_c ∥ end_ARG ⋅ ( italic_s - italic_c ) + italic_c ) Patience. If the target metric ℓmetricsubscriptℓmetric _metricℓmetric (for REINFORCE see § C) does not improve for a predefined number of iterations (100) we reset to the best previously known state ~(best)superscript~best X^(best)over~ start_ARG italic_X end_ARG( best ) or with 50% chance we use a promising adversarial prompt from a different optimization in the batch. In both cases, we reinitialize ~(i)superscript~ X^(i)over~ start_ARG italic_X end_ARG( i ) with discretized d⁢(~(best))superscript~bestd( X^(best))d ( over~ start_ARG italic_X end_ARG( best ) ), which is the one-hot encoding of ~(best)superscript~best x^(best)over~ start_ARG italic_x end_ARG( best ). We sample prompts from a different prompt in the batch from Cat⁡(softmax⁡(−ℓmetric/0.25))Catsoftmaxsubscriptℓmetric0.25Cat(softmax(- _metric/0.25))Cat ( softmax ( - ℓmetric / 0.25 ) ). Learning rate scheduler. We linearly ramp up the learning rate along with the entropy regularization for the first 100 iterations. Thereafter, we apply a cosine annealing scheduler with warm restarts (Loshchilov & Hutter, 2017) with 60 steps duration and a terminal learning rate of 0.325. Entropy projection strength. The entropy projection strength is scaled together with the learning rate, where lower learning rates correspond to a weaker projection. Moreover, the entropy projection strength is coupled to the difference between the “relaxed loss” ℓ⁢(~)ℓ~ ( X)ℓ ( over~ start_ARG italic_X end_ARG ) and “discrete loss” ℓ⁢(~)ℓ~ ( x)ℓ ( over~ start_ARG italic_x end_ARG ). Appendix C REINFORCE Sampling strategy. We design the sampling strategy Pfθ′⁢(Y|X=~)superscriptsubscriptsubscript′conditional~P_f_θ (Y|X= X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X = over~ start_ARG italic_X end_ARG ) with the goal of obtaining a sample efficient yet effective estimator for the REINFORCE objective of Eq. 7. Thus, we use the following samples/generations: 1. The static seedsubscriptseed y_seeditalic_yseed: which is usually equal to the affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative but can also be chosen from a previously successful attack for the same behavior on a potentially different model. Alternatively, one may use advanced initialization strategies such as advprefixsubscriptadvprefix y_advprefixitalic_yadvprefix from Zhu et al. (2024) or successful generations from a different model historysubscripthistory y_historyitalic_yhistory. To ensure that seedsubscriptseed y_seeditalic_yseed provides sufficient guidance regardless of the LLM-as-a-judge reward, we clip the reward of seedsubscriptseed y_seeditalic_yseed from [0,1]01[0,1][ 0 , 1 ] to [0.5,1.0]0.51.0[0.5,1.0][ 0.5 , 1.0 ]. If seedsubscriptseed y_seeditalic_yseed is shorter than 128 tokens, we greedily complete its generations. In this case, we use the aforementioned reward clipping only for the original prefix of seedsubscriptseed y_seeditalic_yseed, before extending it. For GCG, we anyways calculate intermediate rewards to handle the extension properly. For PGD, the extension is efficient since we only need one forward and backward pass of fθsubscriptf_θfitalic_θ for the extended seedsubscriptseed y_seeditalic_yseed. 2. The greedy roll-out greedysubscriptgreedy y_greedyitalic_ygreedy which autoregressively returns the most likely next token. This sample can be understood as a cheap estimate for the mode of the distribution 3. One random sample randomsubscriptrandom y_randomitalic_yrandom with temperature 0.70.70.70.7 among the 256 most likely next tokens. We also experimented with more samples; however, only with minor impact on the target metric if accounting for the higher computational cost. 4. The most likely harmful generation harmfulsubscriptharmful y_harmfulitalic_yharmful that we observed throughout the optimization. Specifically, we include a harmful response as soon its reward surpasses 0.2. Until reward 0.75 we always take the most harmful response we found. Beyond 0.75 we favor longer responses or responses that have a lower average cross entropy. Even though it should be possible to (largely) de-bias our estimator, e.g., using importance sampling similar to Kool et al. (2019), we decide to keep the estimator unchanged/simple as we found it to be effective. Judge. We use HarmBench’s (Mazeika et al., 2024) judge for evaluation and rewards. Following HarmBench’s usage, we calculate the reward based on the clean prompt Reward⁡(,)RewardReward( y, x)Reward ( italic_y , italic_x ) and not the perturbed prompt ~~ xover~ start_ARG italic_x end_ARG. Since we usually calculate the rewards for a batch of generations, stability considerations are important to further consistency in the optimization. Due to the varying length of the generations and prompts, we have a varying number of padded tokens. Although the differences in the rewards are usually rather small, they do exist and introduce noise into Reward⁡(,)RewardReward( y, x)Reward ( italic_y , italic_x ). For this reason, during the attack we always (right) pad to a total length of generated 128 tokens. Token-level weighting. We employ two mechanisms that eventually yield a token-level weighting: 1. Linearly decaying weight of token-importance. Since we are optimizing over a long generation, in comparison to the affirmative objective (≈20absent20≈ 20≈ 20 tokens), we also introduce a weighting of the tokens yitalic_y according to their positions. Specifically, we give the first token a five times higher weight than the last (128th) token and distribute the weight linearly in between. This prioritizes changing tokens early in the sequence vs. later tokens. Even though the performance differences are not pivotal, we argue that prioritizing early tokens is particularly important for the optimization over long generations with a small number of sampled generations. 2. To encourage exploiting harmful generations and making them more likely, we aggregate the rewards for matching prefixes of two samples at the token level and calculate the RLOO weights for each token individually. This is important, especially in cases where the model sometimes refuses to answer rather late in the sequence. If two generations/samples (1)superscript1 y^(1)italic_y( 1 ), (2)superscript2 y^(2)italic_y( 2 ) match until token m (:m+1(1)=:m+1(2)subscriptsuperscript1:absent1subscriptsuperscript2:absent1 y^(1)_:m+1= y^(2)_:m+1italic_y( 1 ): m + 1 = italic_y( 2 ): m + 1 and :m+2(1)≠:m+2(2)subscriptsuperscript1:absent2subscriptsuperscript2:absent2 y^(1)_:m+2≠ y^(2)_:m+2italic_y( 1 ): m + 2 ≠ italic_y( 2 ): m + 2, recall that our :m:absent:m: m notation is exclusive), we assign the first m tokens the higher reward max⁡(Reward⁡((1),~),Reward⁡((2),~))Rewardsuperscript1~Rewardsuperscript2~ (Reward( y^(1), x),% Reward( y^(2), x))max ( Reward ( italic_y( 1 ) , over~ start_ARG italic_x end_ARG ) , Reward ( italic_y( 2 ) , over~ start_ARG italic_x end_ARG ) ). Arguably this aggregation has similarities to Monte Carlo tree search (MCTS) since matching prefixes in yitalic_y correspond to matching nodes in a search tree. We also apply this strategy if calculating intermediate rewards for REINFORCE-GCG. REINFORCE loss ℓ ℓ. The REINFORCE Leave-One-Out (RLOO) loss we ultimately use (negative of Eq. 9 after dropping the gradient) resolves to: ℓ⁢(^,~)ℓ^~ ( Y, X)ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ) =−∑i=1K[Reward⁡((i),~)−bstaticK−1K⁢∑j≠iReward⁡((j),~)]⁢log⁡P⁢((i)|~)absentsuperscriptsubscript1delimited-[]Rewardsuperscript~subscriptstatic1subscriptRewardsuperscript~conditionalsuperscript~ =- _i=1^K [Reward( y^(i), % X)- b_staticK- 1K _j≠ i % Reward( y^(j), X) ] P( y^(i)| % X)= - ∑i = 1K [ Reward ( italic_y( i ) , over~ start_ARG italic_X end_ARG ) - divide start_ARG bstatic end_ARG start_ARG K end_ARG - divide start_ARG 1 end_ARG start_ARG K end_ARG ∑j ≠ i Reward ( italic_y( j ) , over~ start_ARG italic_X end_ARG ) ] log P ( italic_y( i ) | over~ start_ARG italic_X end_ARG ) (11) =∑i=1K[Reward⁡((i),~)−bstaticK−1K⁢∑j≠iReward⁡((j),~)]⁢CE⁡((i)|~)absentsuperscriptsubscript1delimited-[]Rewardsuperscript~subscriptstatic1subscriptRewardsuperscript~CEconditionalsuperscript~ = _i=1^K [Reward( y^(i), % X)- b_staticK- 1K _j≠ i% Reward( y^(j), X) ]CE( y^(i)% | X)= ∑i = 1K [ Reward ( italic_y( i ) , over~ start_ARG italic_X end_ARG ) - divide start_ARG bstatic end_ARG start_ARG K end_ARG - divide start_ARG 1 end_ARG start_ARG K end_ARG ∑j ≠ i Reward ( italic_y( j ) , over~ start_ARG italic_X end_ARG ) ] CE ( italic_y( i ) | over~ start_ARG italic_X end_ARG ) with samples ^=seed,random,greedy,harmful∼Pfθ′⁢(Y|X)^subscriptseedsubscriptrandomsubscriptgreedysubscriptharmfulsimilar-tosuperscriptsubscriptsubscript′conditional Y=\ y_seed, y_random, y% _greedy, y_harmful\ P_f_θ (Y|X)over start_ARG Y end_ARG = italic_yseed , italic_yrandom , italic_ygreedy , italic_yharmful ∼ Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ) and static bstaticsubscriptstaticb_staticbstatic. As detailed in the main part, we introduce bstatic=0.1subscriptstatic0.1b_static=0.1bstatic = 0.1 to stabilize the loss when all generations are harmless or harmful. Target metric ℓmetricsubscriptℓmetric _metricℓmetric. As the target metric, that we use to determine the most successful attack step, we largely follow the estimator according to Eq. 9/Eq. 11, using our biased sampler to obtain up to K=33K=3K = 3 samples, where we exclude the random generation randomsubscriptrandom y_randomitalic_yrandom. We exclude randomsubscriptrandom y_randomitalic_yrandom to avoid its randomness since we only generate a single random generation. Note that due to the most likely harmful sample, K=22K=2K = 2 is also possible. Unfortunately, due to the dynamic baseline (the average reward of other samples) and the low amount of samples, the loss is still not directly usable as a consistent metric to measure overall progress spanning all attack iterations. We choose to add a large constant (10) if the greedy generation is not harmful. As soon as the greedy generation is harmful, we double its weight in the loss calculation. REINFORCE-PGD. For PGD we directly apply the described approach to estimate the REINFORCE gradient. For memory reasons (80 GB GPU RAM), we use HarmBench’s validation judge that is based on Mistral 7B (Jiang et al., 2023). REINFORCE-GCG. In contrast to PGD, for GCG it is easier to accommodate the Llama 2 13B judge and, thus, we decide for it as it is the primary judge of HarmBench. For GCG we have two deviations from the general REINFORCE procedure: (1) we additionally calculate rewards for tokens at positions 20, 40, and 80; (2) we do not use the random generation (temperature 0.7) for the selection. This reduces the overall runtime almost by 25% since the scoring of the candidates is by far the most expensive step (for generation length 128). Similarly to the global selection of the best step, for REINFORCE-GCG we need to select the best candidate in each step. Here we can directly use Eq. 11. However, we exclude the only random generation to avoid diminishing progress based on a potentially unlikely generation. Specifically, we keep the generations/rewards (minus baselines) constant and solely calculate the likelihoods log⁡P⁢(|~)conditional~ P( y| X)log P ( italic_y | over~ start_ARG italic_X end_ARG ) for each candidate in arg⁢min~∈^⁡ℓ⁢(^,~)subscriptargmin~^ℓ^~ *arg\,min_ X∈ X ( % Y, X)start_OPERATOR arg min end_OPERATORover~ start_ARG italic_X end_ARG ∈ over start_ARG X end_ARG ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ). Given the intermediate rewards at positions 20, 40, 80, and the terminal reward at position 128, we progressively adapt the number of generated tokens for the calculation of arg⁢min~∈^⁡ℓ⁢(^,~)subscriptargmin~^ℓ^~ *arg\,min_ X∈ X ( % Y, X)start_OPERATOR arg min end_OPERATORover~ start_ARG italic_X end_ARG ∈ over start_ARG X end_ARG ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ). We set the harmfulness/reward threshold to 0.01 for the greedy generation and include one additional position. However, we at least score until the generation of 40 tokens. For example, (a) if the greedy reward is 0 for all positions, we calculate the loss using 40 tokens; (b) if the greedy reward exceeds 0.01 at position 40, we calculate the loss using 80 tokens; or (c) if the greedy reward at position 80/128 exceeds 0.01 we calculate the loss using all 128 tokens. Appendix D GCG vs. PGD Tradeoffs From the side-by-side comparison of GCG (Algo. 1) and PGD (Algo. 2), we observe many similarities between both approaches. For example, both approaches continuously relax the one-hot encoded and rely on the gradient to determine their next step. However, both approaches rely on different design choices that yield different tradeoffs as we list in Table 5. Table 5: Comparison of GCG’s and PGD’s different characteristics and the resulting tradeoffs. Characteristic GCG (Algo. 1) PGD (Algo. 2) How to perturb: Discrete mutation Gradient-based optimization over relaxed prompts A guaranteed change of the prompt Gradient update not necessarily alters the discretized prompt Each mutation alters a single token Every token is affected by gradient update Sample multiple mutations randomly A single gradient step without additional randomness Batching: Batching over mutations Batching over multiple prompts/attacks Under runtime constraints: Few attack iterations Many attack iterations In summary, GCG (Zou et al., 2023) uses batching over multiple randomly sampled mutations for a single prompt to maximize GPU utilization. In contrast, in PGD (Geisler et al., 2024) only a single “mutation” is generated in each step and batches over multiple prompts. If further budgeting runtime, for PGD, one may apply more attack iterations. Importantly, in the context of REINFORCE, due to the computational cost of generations, it might be computationally suboptimal to produce a single “mutation” as PGD is doing. In other words, with the use of a computationally more demanding attack objective, the runtime cost of PGD is more strongly impacted than the one of GCG. For example, averaged over the experiment in Fig. 3, REINFORCE-GCG is 9.4 times more expensive than vanilla GCG per step (roughly 2 to 3 times more expensive if generations are harmless). In contrast, REINFORCE-PGD is 33.3 times more expensive than vanilla PGD per step. The more astounding it is that despite the order of magnitude higher step cost, REINFORCE-GCG and -PGD appear to strike a better attack success vs. runtime tradeoff than vanilla GCG, as we show in Fig. 3. The next section gives a more detailed breakdown of the runtime cost of our REINFORCE-GCG. Appendix E GCG Runtime Cost For a more detailed breakdown of the time cost, we analyzed an attack on Llama 3 (excluding compile times) using PyTorch 2.5.1 with Python 3.12, CUDA 12.4, and Huggingface’s transformers library (Wolf et al., 2020) of version 4.47.1 on an 80 GB H100 GPU. Here, the affirmative response had a length of 26 tokens. Recall that we follow Zou et al. (2023) and use a search width of 512 tokens for GCG. We report average times for each step. We next explain the runtime for the three different cases (a) vanilla GCG, (b) best-case REINFORCE-GCG, and (c) worst-case REINFORCE-GCG. (b) applies if the generations are harmless and short, while (c) applies if generations are harmful and long. In Table 6, we provide an overview. (a) Vanilla GCG. With the configuration, the vanilla GCG (Zou et al., 2023) requires roughly 1.3 s per step using HarmBench’s implementation (Mazeika et al., 2024) with prefix caching. Specifically, the gradient costs around 100 ms and the selection of the best mutation costs around 1.2 s. (b) REINFORCE-GCG best-case runtime cost. In the best case, when all generations are harmless and are short (around 20 tokens), the greedy and random generation require 250 ms each. If completion of seedsubscriptseed y_seeditalic_yseed terminates immediately with an end-of-sequence token, its costs are negligible. In this scenario, the gradient calculation takes around 200 ms. Calculating the rewards for the three generations (seed,greedy,randomsubscriptseedsubscriptgreedysubscriptrandom y_seed, y_greedy, y_randomitalic_yseed , italic_ygreedy , italic_yrandom) at a single length (at most 27 tokens long) takes around 200 ms. Last, the selection of the best mutation takes 1.7 s (27 tokens). Moreover, we assume that there is no harmfulsubscriptharmful y_harmfulitalic_yharmful. Since the other costs are negligible, the total step cost is around 2.6 s, which is 2.2 times more expensive than vanilla GCG. (c) REINFORCE-GCG worst-case runtime cost. When the generations are long and harmful, our REINFORCE-GCG encounters higher costs. The generation of two times 128 tokens (greedy,randomsubscriptgreedysubscriptrandom y_greedy, y_randomitalic_ygreedy , italic_yrandom) takes 1.2 s each, and completing 108 tokens (seedsubscriptseed y_seeditalic_yseed) takes 1 s. The gradient calculation takes around 300 ms. The rewards for seed,greedy,randomsubscriptseedsubscriptgreedysubscriptrandom y_seed, y_greedy, y_randomitalic_yseed , italic_ygreedy , italic_yrandom at lengths 26, 40, 80, 128 tokens takes 600 ms. For harmfulsubscriptharmful y_harmfulitalic_yharmful, we can reuse the historical values. The selection of the best mutation with 128 tokens takes 8.9 s. In total, REINFORCE-GCG may require a worst-case runtime of 13.2 s, which is 11 times more expensive than vanilla GCG. With longer generations (128 tokens) that are still harmless, the cost can increase to around 6.4 s seconds. Here we assume the time costs for generations, gradient, and reward as in the worst case. The selection with 40 tokens takes 2.3 s. Table 6: Summary of runtime cost for vanilla GCG, best-case REINFORCE-GCG, and worst-case REINFORCE-GCG for a target length of up to 128 tokens on an Nvidia H100 80 GB. Steps that are not listed have negligible runtime costs. The best case applies when all generations are harmless and short. The worst-case applies when generations are harmful and long. Step Vanilla GCG Best-case REINFORCE-GCG Worst-case REINFORCE-GCG Generation - 0.5 s 3.4 s Gradient 0.1 s 0.2 s 0.3 s Reward - 0.2 s 0.6 s Selection 1.2 s 1.7 s 8.9 s Total 1.3 s 2.6 s 13.2 s Appendix F Generalization from Attack to Eval Metric In this section, we contrast the results of the attack objective vs. the test objective. For REINFORCE-GCG, we use HarmBench’s test judge based on Llama 2 13B with 128 tokens, and for REINFORCE-PGD, we use HarmBench’s validation judge based on Mistral 7B with 128 tokens. We report the results for REINFORCE-GCG in Table 7 and observe almost flawless generalization from 128 to 512 tokens. In contrast, the results for REINFORCE-PGD in Table 8 show that the attack objective (Mistral 7B) and test objective (Llama 2 13B) differ greatly at 128 tokens. However, the differences become smaller between the attack objective (Mistral 7B) with 128 tokens and the test objective (Llama 2 13B) with 512 tokens. Table 7: ASRs for GCG, where we contrast the attack objective (128 tokens) with the test objective (512) tokens. Affirmative REINFORCE (ours) ASR@ 128 512 128 512 Gemma 1.1 2B 0.54 0.57 0.90 0.88 Gemma 1.1 7B 0.51 0.63 0.86 0.87 Llama 2 7B 0.32 0.32 0.63 0.56 Llama 3 8B 0.29 0.35 0.66 0.73 Vicuna 1.5 7B 0.85 0.86 1.00 0.95 Table 8: ASRs for PGD, where we contrast the attack objective (Mistral 7B, 128 tokens) with the test objective (Llama 2 13B, 128 & 512) tokens. Here we often observe substantial differences between the attack and test objective. Affirmative REINFORCE (ours) ASR@ 128 (attack) 128 512 128 (attack) 128 512 Gemma 1.1 2B 0.66 0.58 0.56 0.96 0.70 0.82 Gemma 1.1 7B 0.52 0.54 0.54 0.93 0.77 0.84 Llama 2 7B 0.20 0.19 0.17 0.26 0.24 0.22 Llama 3 8B 0.59 0.55 0.57 0.60 0.48 0.69 Vicuna 1.5 7B 0.84 0.80 0.87 0.97 0.90 0.94 Appendix G AdvPrefix (Zhu et al., 2024) In Table 9, we show additional ablations/comparisons to AdvPrefix. AdvPrefix is a technique to generate alternative affirmative responses that are tailored to the attacked model. We generate AdvPrefix for Llama 3 8B. The AdvPrefix generation takes around 5 minutes per prompt with the code provided by the authors and using two H100 80 GB. The key observations are: (1) our REINFORCE objective outperforms theirs on both Llama 3 8B and its defended variant (circuit breaker). The differences are particularly pronounced with circuit breaker defense; and (2) using the generations of previously successful attacks historysubscripthistory y_historyitalic_yhistory as well as advprefixsubscriptadvprefix y_advprefixitalic_yadvprefix for seedsubscriptseed y_seeditalic_yseed are both effective strategies. Table 9: Study of the impact of AdvPrefix from Zhu et al. (2024) generated for Llama 3 8B. Here, the “Affirmative” objective maximizes the log probability log⁡P⁢(seed|~)conditionalsubscriptseed~ P( y_seed| x)log P ( italic_yseed | over~ start_ARG italic_x end_ARG ). affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative is the target of HarmBench, advprefixsubscriptadvprefix y_advprefixitalic_yadvprefix denotes the target obtained via AdvPrefix, and historysubscripthistory y_historyitalic_yhistory denotes a successful response from an attack on Llama 3 8B. We bold the best and underline the runner-up. * denotes numbers on the subset of successful attacks on the base model with affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative. Affirmative REINFORCE (ours) seed=subscriptseedabsent y_seed=italic_yseed = affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative advprefixsubscriptadvprefix y_advprefixitalic_yadvprefix affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative advprefixsubscriptadvprefix y_advprefixitalic_yadvprefix historysubscripthistory y_historyitalic_yhistory ASR@ 128 512 128 512 128 512 128 512 128 512 Llama 3 8B 0.29 0.35 0.67 0.70 0.66 0.73 0.81 0.81 - - + Circuit breaker 0.01 0.02 0.12 0.14 0.21 0.23 0.47 0.48 0.46∗ 0.50∗ Appendix H Dynamics of REINFORCE-GCG (a) Terminal rewards (b) Cross entropies of gradient calculation (c) Intermediate reward (d) Intermediate reward (e) Intermediate reward (f) Cross entropies of seedsubscriptseed y_seeditalic_yseed for mutations (g) Cross entropies of greedysubscriptgreedy y_greedyitalic_ygreedy for mutations (h) Cross entropies of harmfulsubscriptharmful y_harmfulitalic_yharmful for mutations (i) RLOO loss of gradient calculation (j) RLOO loss for mutations Figure 6: Exemplary attack on Llama 3 8B (first 50 steps). This figure extends Fig. 5 with the intermediate rewards (c-e) and the REINFORCE Leave-One-Out (RLOO) losses (i-j). We next complement Fig. 5 and its accompanying discussion. That is, in Fig. 6, we also plot the intermediate rewards at lengths 22, 40, and 80. As hinted in the main part, here we use length 22 instead of 20 since affirmativesubscriptaffirmative y_affirmativeitalic_yaffirmative is 22 tokens long. Moreover, in Fig. 6i & 6j, we also plot the REINFORCE loss. It is an interesting phenomenon in Fig. 6f to 6h that most of the mutations seem to be concentrated in a region with small entropy values, while there is a consistent fraction of outliers for which the cross entropies are much larger. However, for the REINFORCE loss (Fig. 6j), the fluctuations are much less pronounced. Perhaps the lower variations are due to the averaging over multiple generations and the emphasis on early tokens (see “Token-level weighting” in § C). Intermediate rewards. We observe that the HarmBench judge often returns zero reward for short generations (see Fig. 6c). We only observe a reward of 0.5 for seedsubscriptseed y_seeditalic_yseed since we clamp it to the range [0.5,1]0.51[0.5,1][ 0.5 , 1 ]. While this is intended behavior to determine attack success rates (ASRs), for a reinforcement learning objective like ours it might be beneficial to obtain better intermediate rewards. For this reason, we actually use max⁡Reward⁡(:22),Reward⁡(:40),Reward⁡(:80),Reward⁡()Rewardsubscript:absent22Rewardsubscript:absent40Rewardsubscript:absent80Reward \Reward( y_:22),Reward( y_:4% 0),Reward( y_:80),Reward( y)\max Reward ( italic_y: 22 ) , Reward ( italic_y: 40 ) , Reward ( italic_y: 80 ) , Reward ( italic_y ) instead of Reward⁡(:22)Rewardsubscript:absent22Reward( y_:22)Reward ( italic_y: 22 ) to calculate ℓ⁢(^,~)ℓ^~ ( Y, X)ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ) (Eq. 11) for the first 22 tokens. For the tokens until position 40, we use max⁡Reward⁡(:40),Reward⁡(:80),Reward⁡()Rewardsubscript:absent40Rewardsubscript:absent80Reward \Reward( y_:40),Reward( y_:8% 0),Reward( y)\max Reward ( italic_y: 40 ) , Reward ( italic_y: 80 ) , Reward ( italic_y ) instead of Reward⁡(:40)Rewardsubscript:absent40Reward( y_:40)Reward ( italic_y: 40 ). And so on. See the second point in “token-level weighting” in § C. Consistency over multiple attack iterations. Due to the dynamic weighting via the baselines in Eq. 9/Eq. 11, our biased sampling strategy Pfθ′⁢(Y|X)superscriptsubscriptsubscript′conditionalP_f_θ (Y|X)Pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT′ ( Y | X ), and the single random sample, our loss ℓ⁢(^,~)ℓ^~ ( Y, X)ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ) is also not strictly consistent. We opt against obtaining many random samples due to the computational cost. Moreover, biasing the sampling strategy seems key for an efficient attack (number of attack steps). We argue that for candidate selection within a single attack step, the dynamic weighting via the baselines does not pose an additional challenge. However, the dynamic weighting may impact the consistency of ℓ⁢(^,~)ℓ^~ ( Y, X)ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ) between different attack iterations since ^, Y,over start_ARG Y end_ARG , changes. Nevertheless, we find that despite the dynamic weighting ℓ⁢(^,~)ℓ^~ ( Y, X)ℓ ( over start_ARG Y end_ARG , over~ start_ARG italic_X end_ARG ) correlates well with attack progress. However, setting the baseline in Eq. 8 to zero might improve the consistency. See also “target metric” in § C. Appendix I Example REINFORCE-GCG Attacks and Generations We next show a randomly chosen collection of successful REINFORCE-GCG attacks and the resulting generations. To determine if an attack was successful, we use HarmBench’s LLama 2 13 B judge. Table 10: Gemma 1.1 2B. Randomly picked harmful generations. To limit the amount of harmful/upsetting content in this paper, we limit the model generations to the first 6 lines. Table 11: Gemma 1.1 7B. Randomly picked harmful generations. To limit the amount of harmful/upsetting content in this paper, we limit the model generations to the first 6 lines. Table 12: Llama 2 7B. Randomly picked harmful generations. To limit the amount of harmful/upsetting content in this paper, we limit the model generations to the first 6 lines. Table 13: Llama 3 8B. Randomly picked harmful generations. To limit the amount of harmful/upsetting content in this paper, we limit the model generations to the first 6 lines. Table 14: Vicuna 1.5 7B. Randomly picked harmful generations. To limit the amount of harmful/upsetting content in this paper, we limit the model generations to the first 6 lines. Appendix J Example REINFORCE-GCG Attacks on Llama 3 8B with Circuit Breaker Table 15: Llama 3 8B with circuit breaker, where seed=historysubscriptseedsubscripthistory y_seed= y_historyitalic_yseed = italic_yhistory. Randomly picked harmful generations. To limit the amount of harmful/upsetting content in this paper, we limit the model generations to the first 6 lines.

Instant research discovery