Paper deep dive

Fundamental Limitations of Alignment in Large Language Models

Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, Amnon Shashua

Year: 2023Venue: arXiv preprintArea: Formal/TheoreticalType: TheoreticalEmbeddings: 291

Models: LLaMA 2

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 8:16:37 PM

Summary

The paper introduces 'Behavior Expectation Bounds' (BEB), a theoretical framework to analyze alignment in Large Language Models (LLMs). It models LLMs as a mixture of well-behaved and ill-behaved components. The authors prove that if an LLM has a non-zero probability of exhibiting an undesired behavior, there exist prompts that can trigger this behavior, rendering current alignment methods like RLHF vulnerable to adversarial attacks.

Entities (4)

Large Language Models · technology · 99%Behavior Expectation Bounds · theoretical-framework · 98%Llama · model-family · 95%Reinforcement Learning from Human Feedback · alignment-method · 95%

Relation Signals (2)

Behavior Expectation Bounds → analyzes → Large Language Models

confidence 95% · we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models.

Reinforcement Learning from Human Feedback → isvulnerableto → Adversarial Prompting

confidence 90% · leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

290,688 characters extracted from source content.

Expand or collapse full text

Fundamental limitations of alignment in Large Language Models Yotam Wolf∗ The Hebrew University yotamwolf@cs.huji.ac.il &Noam Wies The Hebrew University noam.wies@cs.huji.ac.il & Oshri Avnery The Hebrew University oshri.avnery@mail.huji.ac.il &Yoav Levine AI21 Labs yoavl@ai21.com & Amnon Shashua The Hebrew University shashua@cs.huji.ac.il Equal contribution Abstract An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary “chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety. 1 Introduction Training large language models (LLMs) over vast corpora has revolutionized natural language processing, giving LLMs the ability to mimic human-like interactions and serve as general purpose assistants in a wide variety of tasks, such as wide-scoped question answering, writing assistance, teaching, and more (Radford et al., 2019; Devlin et al., 2019; Brown et al., 2020; Schulman et al., 2023; OpenAI, 2023; Bubeck et al., 2023; Nori et al., 2023; West, 2023; Park et al., 2023). A growing concern due to the increasing reliance on LLMs for such purposes is the harm they can cause their users, such as feeding fake information (Lin et al., 2022; Weidinger et al., 2022), behaving offensively and feeding social biases (Hutchinson et al., 2020; Venkit et al., 2022; Weidinger et al., 2022), or encouraging problematic behaviors by users (even by psychologically manipulating them Roose (2023); Atillah (2023)). Indeed, the unsupervised textual data used for pretraining modern LLMs includes enough demonstrations of the above undesired behaviors for them to be present in the resulting models (Bender et al., 2021). The act of removing these undesired behaviors is often called alignment (Yudkowsky, 2001; Taylor et al., 2016; Amodei et al., 2016; Shalev-Shwartz et al., 2020; Hendrycks et al., 2021; Pan et al., 2022; Ngo, 2022). There are several different approaches to performing alignment in LLMs. One is to include aligning prompts: Askell et al. (2021) show that injecting language models with helpful, honest, and harmless (H) textual prompts improves alignment and decreases toxicity. Similarly, Rae et al. (2021) also use prompting to decrease toxicity. Another approach for LLM alignment is the procedure of reinforcement learning from human feedback (RLHF) that trains language models to be helpful and harmless with a human preference based reward (Bai et al., 2022). Their work shows an increase in an LLM’s H scores while maintaining its useful abilities, as measured by zero- and few-shot performance on different natural language tasks. Ouyang et al. (2022) use this method to fine tune GPT-3 into InstructGPT using data collected from human labelers to reach better performance on a variety of tasks, while improving H (measured via bias and toxicity datasets Gehman et al. (2020); Nangia et al. (2020)). Recently, a new approach for alignment known as representation engineering Zou et al. (2023); Jorgensen et al. (2023); Leong et al. (2023); Liu et al. (2023); Turner et al. (2023) has emerged, in which vectors are injected into the hidden layer representations of the model post-training, steering the model towards desirable behaviors in latent space. While promising, it is still in early development stages, and the extent of its effects on the model, e.g.maintaining its usefulness, are unknown. Therefore, in this work we focus on the first two approaches, RLHF finetuning and prompting, in which the model’s weights are frozen in inference time. While the above approaches to alignment are effective to a certain extent, they are still dangerously brittle. For example, Wallace et al. (2019) show that short adversarial prompts can trigger negative behaviors and social biases. Yu & Sagae (2021) and Xu et al. (2021) provide methods for exposing harmful behaviors of models by triggering problematic responses. Subhash (2023) showed that adversarial prompts can manipulate ChatGPT to alter user preferences. Beyond academic works, the general media is abundant with contemporary examples of leading LLMs being manipulated by users to expose harmful behaviors via the so called “jailbreaking" approach of prompting the LLM to mimic a harmful persona (Nardo, 2023; Deshpande et al., 2023). Even in the absence of adversarial attacks, leading alignment methods can underperform and are not well understood: Perez et al. (2022) provide evidence that certain negative behaviors have inverse scaling with the number of RLHF steps, indicating that this popular alignment procedure may have a complex effect. In this paper, we introduce a probabilistic framework for analyzing alignment and its limitations in LLMs, which we call Behavior Expectation Bounds (BEB), and use it in order to establish fundamental properties of alignment in LLMs. The core idea behind BEB is to represent the LLM distribution as a superposition of ill- and well-behaved components, in order to provide guarantees on the ability to restrain the ill-behaved components, i.e., guarantees that the LLM is aligned. It is noteworthy that LLMs have been shown to distinctly represent behaviors and personas, and the notion of persona or behavior superposition has been intuitively proposed as an explanation (Andreas, 2022; Nardo, 2023). Our BEB framework assumes an underlying categorization into different behaviors, where any natural language sentence is assigned a ground truth score between −11-1- 1 (very negative) and +11+1+ 1 (very positive) for every behavior (see examples in Figure 1). Such a categorization can be, e.g., into the previously proposed helpful, honest, and harmless categories, but it can also be expanded and fine-grained into many more categories such as polite, not racist, compassionate, and so on. Given such a categorization and ground truth sentence scoring functions per category, the alignment score of any distribution over natural sentences w.r.t. a given behavior is the expectation value of sentence scores for sentences drawn from the distribution. The BEB framework thus provides a natural theoretical basis for describing the goal of contemporary alignment approaches such as RLHF: increasing the behavior expectation scores for behaviors of interest. Additionally, the BEB framework employs assumptions on the LLM distribution presented in section 2. These include the notion of α,β,γα,β,γα , β , γ-distinguishability (definition 5), which means the language model can be decomposed to a sum of ill-behaved and well-behaved components, where the weight of the negative in the mixture is α, it is distinguishable from the rest of the distribution in the sense of a bounded KL-divergence that is at least β, and exhibits negative behavior scored as γ<00γ<0γ < 0. Lastly, we include a definition for σ-similarity between two components (definition 4), which bounds the variance of the log likelihood between the well-behaved and ill-behaved components. We use this framework in section 3 in order to assert several important statements regarding LLM alignment: Alignment impossibility: We show that under our main assumption, called α,β,γα,β,γα , β , γ-distinguishability, an LLM alignment process which reduces undesired behaviors to a small but nonzero fraction of the probability space is not safe against adversarial prompts (theorem 1); Preset aligning prompts can only provide a finite guardrail against adversarial prompts: We prove that under our main assumption and the assumption of σ-similarity (definition 4), including an aligning prefix prompt does not guarantee alignment (theorem 2). LLMs can be misaligned during a conversation: We show that under our previous assumptions, a user can misalign an LLM during a conversation, with limited prompt length at each turn (theorem 3). LLMs with best-of-n sampling can be misaligned: Under our main assumption, selection of most aligned model response out of n generations, does not guarantee alignment (theorem 4). In section 4, we demonstrate empirically some of the assumptions and results derived from the BEB framework on the LLaMA LLM family (Meta, 2023; Touvron et al., 2023). In subsection 4.1 we measure possible values for β-distinguishability (definition 2) and σ-similarity (definition 4), as can be seen in figure 2. In subsection 4.2 we demonstrate the underlying mechanism by which misalignment happens in the BEB framework, which is the convergence of the LLM to a negative behavior component. This is done by showing a decay of the KL divergence between the two, as seen in figure 3a. Furthermore, we can extract estimated parameters of the theoretical framework allowing to calculate the expected misaligning prompt length. Moreover, we demonstrate how the method proposed by the BEB framework for generating misaligning prompts causes misalignment (figure 3b), which is quantified by our proposed behavior expectation metric (equation 2). This framework is mainly centered around models that have undergone an aligning finetuning process such as RLHF and less on pretrained models, as the latter are not aligned to begin with and require little effort to be provoked into behaving negatively (as shown in appendix M), but even so, the theoretical framework is still applicable to both. In subsection 4.2 we also present preliminary indications that RLHF alignment increases the distinguishability of undesired behaviors, but we leave the investigation of this possibility for future work. Overall, we hope that our newly proposed framework of Behavior Expectation Bounds, along with our attained results, may spark a theoretical thrust helping to better understand the important topic of LLM alignment. 2 Behavior Expectation Bounds: A Framework for Analyzing LLM Alignment Figure 1: Examples of sentence behavior scores along different behavior verticals. Our framework of Behavior Expectation Bounds (BEB) assumes ground truth behavior scoring functions, and bounds the expected scores of models along different behavior verticals in order to guarantee LLM alignment or misalignment. In this section, we introduce Behavior Expectation Bounds (BEB), a probabilistic framework for studying alignment of LLMs. Given a language model’s probability distribution ℙPblackboard_P, we propose a measure for quantifying its tendency to produce desired outputs as measured by a certain behaviour vertical B, where for example B can be helpfulness, politeness, or any other behavior vertical of interest. Formally, we model behaviour scoring functions along vertical B as B:Σ∗→[−1,1]:→superscriptΣ11B: ^*→[-1,1]B : Σ∗ → [ - 1 , 1 ], which take a string of text from an alphabet Σ Σ as their input111We use the Kleene closure Σ∗superscriptΣ ^*Σ∗ for simplicity of notation, but note that it can be replaced with Σcontext lengthsuperscriptΣcontext length ^context lengthΣcontext length to account for the finite context window of models. and rate the manner in which B manifests in the string, with +11+1+ 1 being very positive and −11-1- 1 being very negative. This formulation directly reflects recent empirical efforts for studying alignment. In particular, (Perez et al., 2022) recently curated 500500500500 negative and positive examples along each of over 100100100100 different behavior verticals. Figure 1 shows short examples of the behavior scores of several sentences along two behavior verticals. We use the following expected behavior scoring of distribution ℙPblackboard_P w.r.t. behavior vertical B as a scalar quantifyer of the tendency of ℙPblackboard_P to produce desired behavior along the B vertical: Bℙ:=s∼ℙ⁢[B⁢(s)]assignsubscriptℙsubscriptsimilar-toℙdelimited-[]B_P:=E_s [B(s)]Bblackboard_P := blackboard_Es ∼ blackboard_P [ B ( s ) ] (1) where for clarity purposes, in this paper sampling from language distributions is implicitly restricted to single sentences (see discussion on this choice and its limitations in A.3). We use the above distribution notation ℙPblackboard_P to represent that of an unprompted LLM, e.g., an LLM straight out of pretraining or out of an alignment tuning procedure such as RLHF. The task of aligning a pretrained LLM can be now framed as increasing its expected behavior scores along behavior verticals of interest. Intuitively, as an LLM is prompted with a prefix text string s∗superscripts^*s∗, the behaviour of the conditional probability ℙ(⋅|s∗)P (·\,|\,s^* )blackboard_P ( ⋅ | s∗ ) might change in accordance with the in-context learning phenomenon (Brown et al., 2020; Wies et al., 2023) in which the LLM adapts its conditional probabilities to reflect its current textual context. Thus, we will denote by Bℙ⁢(s∗)subscriptℙsuperscriptB_P (s^* )Bblackboard_P ( s∗ ) the behaviour of the language model when prompted with a prompt text s∗superscripts^*s∗: Bℙ⁢(s∗):=s∼ℙ(⋅|s∗)⁢[B⁢(s)]B_P(s^*):=E_s (·|s^* )[B(% s)]Bblackboard_P ( s∗ ) := blackboard_Es ∼ blackboard_P ( ⋅ | s∗ ) [ B ( s ) ] (2) We will consider several scenarios for which the prefix s∗superscripts^*s∗ plays different roles. The first and main one is that s∗superscripts^*s∗ serves as an adversarial input prompt. Our key finding in this paper is that an LLM which was initially aligned w.r.t. a certain behavior vertical, i.e., BℙsubscriptℙB_PBblackboard_P very close to 1111, can still be vulnerable to adversarial prompts, i.e., there exists a prompt s∗superscripts^*s∗ such that Bℙ⁢(s∗)subscriptℙsuperscriptB_P(s^*)Bblackboard_P ( s∗ ) is very close to −11-1- 1. Secondly, we will consider a scenario in which s∗superscripts^*s∗ is comprised of an initial aligning prompt, denoted s0subscript0s_0s0, concatenated by a subsequent adversarial input prompt. Lastly, we will analyze conversation scenarios in which s∗superscripts^*s∗ is comprised of previous turns of user queries and LLM responses. 2.1 LLMs as a Superposition of Behaviors In this subsection, we present a key aspect of our BEB framework: decomposing the unprompted LLM distribution ℙPblackboard_P into a mixture of distributions, each behaving differently. Importantly, LLMs exhibit signs of capturing such decompositions in practice. For example, Andreas (2022) shows empirical evidence that current LLMs can infer behaviours from textual prompts, and that these behaviours affect the text that the LLM generates, and Nardo (2023) discuss LLMs as a superposition of personas (i.e. a mixture of components, each behaving differently). We will use mixture decompositions inspired by such observations, and prove that textual prompts can reweight the prior of the mixture components. In appendix K, we experimentally demonstrate that the embedding space of contemporary leading LLMs (LLaMA family (Meta, 2023)) is clustered according to positive and negative inputs w.r.t. behaviors of interest (assembled by (Perez et al., 2022)), and empirically show that this clustering approximately corresponds to our analyzed mixture decomposition model, presented hereinafter. Notice that an unprompted language model, ℙPblackboard_P, is a function that assigns probability to strings of text, according to the statistics of the text it trains on. Therefore, it can be written as some mixture of components ℙ=∑iwi⁢ℙiℙsubscriptsubscriptsubscriptℙP= _iw_iP_iblackboard_P = ∑i witalic_i blackboard_Pi by introducing latent variables, for example, the sources of the training data, and each component will be the natural language distribution induced by the specific text source. As different sources may exhibit different behaviors (toxic, polite, etc.), the induced components may exhibit them as well. We can then partition the components into two disjoint sets, sum over each, and obtain a two component mixture ℙ=(∑i∈Awi⁢ℙi)+(∑j∈Bwj⁢ℙj)=α⁢ℙ1+(1−α)⁢ℙ2ℙsubscriptsubscriptsubscriptℙsubscriptsubscriptsubscriptℙsubscriptℙ11subscriptℙ2P=( _i∈ Aw_iP_i)+( _j∈ Bw_jP_j% )= _1+(1-α)P_2blackboard_P = ( ∑i ∈ A witalic_i blackboard_Pi ) + ( ∑j ∈ B witalic_j blackboard_Pj ) = α blackboard_P1 + ( 1 - α ) blackboard_P2. For example, A can be a set of ill-behaved components and B a set of well-behaved components w.r.t. a given behavior. In appendix A.1, we show this summation method indeed leads to a mixture of two distributions, one more ill-behaved and one more well-behaved. Note that α,1−α1α,1-α , 1 - α are fixed weights of the components ℙ1,ℙ2subscriptℙ1subscriptℙ2P_1,P_2blackboard_P1 , blackboard_P2 in the unprompted model’s distribution, meaning the initial weights given to each component before a prompt is inserted. However, in the prompted model, the weights of the components will change, as when inserting a prompt, we will use the conditional probability distribution of the model, in which the components’ priors are reweighted, possibly a lot, depending on the prompt (see appendix A.1 for details). Observe that for any decomposition of a distribution ℙPblackboard_P into two components, ℙ=α⁢ℙ0+(1−α)⁢ℙ1ℙsubscriptℙ01subscriptℙ1P= _0+(1-α)P_1blackboard_P = α blackboard_P0 + ( 1 - α ) blackboard_P1, the relation Bℙ=α⁢Bℙ0+(1−α)⁢Bℙ1subscriptℙsubscriptsubscriptℙ01subscriptsubscriptℙ1B_P=α B_P_0+(1-α)B_P_1Bblackboard_P = α Bblackboard_P start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + ( 1 - α ) Bblackboard_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT holds from linearity of expectations, and implies that one component is more well-behaved w.r.t. B than the full distribution and the other more ill-behaved, i.e.: Bℙ1≤Bℙ≤Bℙ0subscriptsubscriptℙ1subscriptℙsubscriptsubscriptℙ0B_P_1≤ B_P≤ B_P_0Bblackboard_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≤ Bblackboard_P ≤ Bblackboard_P start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (or vice versa). Thus, focusing on a specific behavior, we adopt the notation: ℙ=α⁢ℙ−+(1−α)⁢ℙ+ℙsubscriptℙ1subscriptℙP= _-+(1-α)P_+blackboard_P = α blackboard_P- + ( 1 - α ) blackboard_P+ (3) We refer to the above as the two component mixture, where ℙ+subscriptℙP_+blackboard_P+ is the well-behaved component and ℙ−subscriptℙP_-blackboard_P- is the ill-behaved component. While this observation is true for any decomposition into two distributions, we will give results for decompositions in which the two distributions ℙ−subscriptℙP_-blackboard_P- and ℙ+subscriptℙP_+blackboard_P+ are sufficiently distinct (formally defined in section 2.2), and the negative component is strictly ill-behaved (i.e, Bℙ−≤γ<0subscriptsubscriptℙ0B_P_-≤γ<0Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ≤ γ < 0). In these cases, the magnitude of α, the prior of the ill-behaved component, will determine the alignment of the LLM: an LLM with a small prior α will be less likely to produce undesired sentences along behavior B vertical. Our main result in section 3 states that no matter how small α is (how aligned the model is to begin with), if it is positive then there exists a prompt that can misalign the LLM to behave like ℙ−subscriptℙP_-blackboard_P-. For an extended discussion on the mixture assumption and its implications, see appendix A.1. 2.2 Definitions for Bounding the Expected LLM Behavior In this subsection, we lay out formal definitions of our BEB framework. Specifically, we define: behavior misalignment using prompts (definition 1); distinguishability between unprompted (prompted) model distributions (definition 2 (3)) and similarity between two distributions (definition 4) that fit a prompting scenario; distinguishibility between ill- and well-behaved components comprising a certain LLM’s distribution (definition 5), called α,β,γα,β,γα , β , γ-distinguishability. Ultimately, α is the prior of the negative component, β is the distinguishability (according to definition 2) between the ill-behaved component and the well behaved component, and γ is the negativity of the ill-behaved component, measured in terms of behavior expectation (equation 2). Once an LLM has finished training, its behavior can only be affected via prompting. Using the above notation for behavior expectation (equations 1 and 2), the following defines when an LLM is prompt-misalignable: Definition 1. Let γ∈[−1,0)10γ∈[-1,0)γ ∈ [ - 1 , 0 ), we say that an LLM with distribution ℙPblackboard_P is γ-prompt-misalignable w.r.t. behaviour B, if for any ϵ>0italic-ϵ0ε>0ϵ > 0 there exists a textual prompt s∗∈Σ∗superscriptsuperscriptΣs^*∈ ^*s∗ ∈ Σ∗ such that Bℙ⁢(s∗)<γ+ϵsubscriptℙsuperscriptitalic-ϵB_P (s^* )<γ+ _P ( s∗ ) < γ + ϵ. Note that while the above definition is based on existence of a specific prompt that misaligns a model, our theoretical results in 3 are proved by construction of this prompt by an empirically practical method, and this construction method is used in 4.2 to create misaligning prompts that work on real LLMs. We also extend our results beyond existence of misaligning prompts to probability mass of misaligning promtpts in appendix B. Decomposing a language model into parts that are well-behaved and ill-behaved exposes components which are more desirable to enhance. The following notion of distinguishability will allow us to guarantee that one component can be enhanced over the other 222Note that the β-distinguishability definition can be relaxed to a KL distance that decays as a power law to zero with increasing length of the prompt s0subscript0s_0s0, as shown in appendix I. Definition 2. We say that a distribution ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ is β-distinguishable from distribution ℙψsubscriptℙP_ψblackboard_Pψ if for any n≥00n≥ 0n ≥ 0: s=s1⊕⋯⊕sn∼ℙϕ⁢(⋅)[DK⁢L(ℙϕ(⋅|s) || ℙψ(⋅|s))] _s=s_1 … s_n _φ% (· ) [D_KL (P_φ (·|s )% || P_ψ (·|s ) ) ]~~~% ~~~~~~~~~~blackboard_Es = s start_POSTSUBSCRIPT 1 ⊕ ⋯ ⊕ sitalic_n ∼ blackboard_Pϕ ( ⋅ ) end_POSTSUBSCRIPT [ Ditalic_K L ( blackboard_Pϕ ( ⋅ | s ) | | blackboard_Pψ ( ⋅ | s ) ) ] (4) :=s⊕sn+1=s1⊕⋯⊕sn⊕sn+1∼ℙϕ⁢(⋅)⁢[log⁡ℙϕ⁢(sn+1|s)ℙψ⁢(sn+1|s)]>βassignabsentsubscriptdirect-sumsubscript1direct-sumsubscript1⋯subscriptsubscript1similar-tosubscriptℙitalic-ϕ⋅delimited-[]subscriptℙitalic-ϕconditionalsubscript1subscriptℙconditionalsubscript1 :=E_s s_n+1=s_1 … s_n s% _n+1 _φ (· ) [ P_% φ (s_n+1|s )P_ψ (s_n+1|s ) ]>β:= blackboard_Es ⊕ s start_POSTSUBSCRIPT n + 1 = s1 ⊕ ⋯ ⊕ sitalic_n ⊕ sitalic_n + 1 ∼ blackboard_Pϕ ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_Pϕ ( sitalic_n + 1 | s ) end_ARG start_ARG blackboard_Pψ ( sitalic_n + 1 | s ) end_ARG ] > β (5) Similarly, we say β-undistinguishable if the above is smaller than β. Where n is the number of sentences sampled from the distribution333The notation s1⊕⋯⊕sn∼ℙϕ(⋅|s0)s_1 … s_n _φ (·|s_0 )s1 ⊕ ⋯ ⊕ sitalic_n ∼ blackboard_Pϕ ( ⋅ | s0 ) indicates sampling n consecutive sentences from the conditional probability distribution ℙϕ(⋅|s0)P_φ (·|s_0 )blackboard_Pϕ ( ⋅ | s0 ) given the initial prefix s0subscript0s_0s0.. The above definition is used for proving our main result theorem 1. For more advanced cases in theorems 2 and 3, of misaligning a prompt protected model or via a multiple turn conversation, we will require a stronger condition, that both distributions are distinguishable when prompted with a prefix s0subscript0s_0s0, that can contain any textual sequence followed by some sentence of negative behavior. This is to capture the notion that the positive and negative components are mainly different w.r.t a specific behavior in question. Hence the negative sentence induces the distinguishability. Definition 3. We say that a distribution ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ is β-prompt-distinguishable (β-prompt-undistinguishable) from ℙψsubscriptℙP_ψblackboard_Pψ if for any prefix s0=s01⊕…⊕s0nsubscript0direct-sumsuperscriptsubscript01…superscriptsubscript0s_0=s_0^1 ... s_0^ns0 = s01 ⊕ … ⊕ s0italic_n of n sentence, ending with a negatively behaving sentence s0nsuperscriptsubscript0s_0^ns0italic_n, i.e., B⁢(s0n)<0superscriptsubscript00B(s_0^n)<0B ( s0italic_n ) < 0, the prompted models ℙϕ(⋅|s0)P_φ(·|s_0)blackboard_Pϕ ( ⋅ | s0 ) and ℙψ(⋅|s0)P_ψ(·|s_0)blackboard_Pψ ( ⋅ | s0 ) are β-distinguishable (β-undistinguishable). For a discussion on distinguishability, its necessity, limitations and examples, see A.2. The following bounds the extent to which a new sentence can enhance one component over the other: Definition 4. We say that a distribution ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ is σ-similar to distribution ℙψsubscriptℙP_ψblackboard_Pψ if there exists σ>00σ>0σ > 0 such that for any sequence of sentences s0subscript0s_0s0 and any n≥00n≥ 0n ≥ 0: V⁢a⁢rs1⊕…⊕sn∼ℙϕ(⋅|s0)⁢[log⁡ℙϕ⁢(s1⊕…⊕sn|s0)ℙψ⁢(s1⊕…⊕sn|s0)]<n⁢σ2 Var_s_1 ... s_n _φ(·|s_0% ) [ P_φ (s_1 ... s_n|s_0% )P_ψ (s_1 ... s_n|s_0 ) % ]<nσ^2V a ritalic_s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_n ∼ blackboard_Pϕ ( ⋅ | s0 ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_Pϕ ( s1 ⊕ … ⊕ sitalic_n | s0 ) end_ARG start_ARG blackboard_Pψ ( s1 ⊕ … ⊕ sitalic_n | s0 ) end_ARG ] < n σ2 (6) Intuitively, if both ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ and ℙψsubscriptℙP_ψblackboard_Pψ are natural language distributions, they cannot be too different in terms of the variance in the ratio of their conditional likelihoods, and σ quantifies this. Furthermore, when ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ and ℙψsubscriptℙP_ψblackboard_Pψ represent positive and negative angles of a specific behaviour, it is likely that they have some common properties so in these cases σ is likely even lower than the bound over all natural language sentences. The linear dependence on length of sequence is inspired by the case of sampling n independent sentences, where variance between the log ratio of ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ and ℙψsubscriptℙP_ψblackboard_Pψ is σ2superscript2σ^2σ2 for each sentence. β roughly serves as a lower bound on the KL-divergence in the case of distinguishability (or upper bound in the case of indistinguishability), and σ its variation and their ratio will appear in several of our results in section 3. The following defines β-distinguishability specifically between the ill- and well-behaved components comprising the LLM distribution, parameterized by α in equation 3, and adds a condition that the behavior expectation of the ill-behaved component is bad enough (i.e., under γ) for all initial prompts s∗superscripts^*s∗: Definition 5. Let γ∈[−1,0)10γ∈[-1,0)γ ∈ [ - 1 , 0 ), assume ℙ=α⋅ℙ−+(1−α)⋅ℙ+ℙ⋅subscriptℙ⋅1subscriptℙP=α·P_-+ (1-α )·P_+blackboard_P = α ⋅ blackboard_P- + ( 1 - α ) ⋅ blackboard_P+ for α>00α>0α > 0. We say that behaviour B:Σ∗→[−1,1]:→superscriptΣ11B: ^*→ [-1,1 ]B : Σ∗ → [ - 1 , 1 ] is α,β,γα,β,γα , β , γ-negatively-distinguishable (α,β,γα,β,γα , β , γ-negatively-prompt-distinguishable) in distribution ℙPblackboard_P, if sups∗Bℙ−⁢(s∗)≤γsubscriptsupremumsuperscriptsubscriptsubscriptℙsuperscript _s^*\B_P_-(s^*)\≤ _s∗ Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s∗ ) ≤ γ and ℙ−subscriptℙP_-blackboard_P- is β-distinguishable (β-prompt-distinguishable) from ℙ+subscriptℙP_+blackboard_P+ (def. 2) ((def. 3)). We will prove our theoretical results for LLM distributions that are distinguishable according to the above KL-divergence based definitions. Our experiments in section 4 indicate that for the LLaMa LM family on behaviors such as agreeableness and anti-immigration as presented in Perez et al. (2022), possible values for these parameters are: log⁡1α1 1αlog divide start_ARG 1 end_ARG start_ARG α end_ARG in the range of 18−30183018-3018 - 30, β is in the range of 5−205205-205 - 20 and σβ σβdivide start_ARG σ end_ARG start_ARG β end_ARG in the range of 0.35−10.3510.35-10.35 - 1. The ratio β′βsuperscript′ β βdivide start_ARG β′ end_ARG start_ARG β end_ARG (upper bound over lower bound of the KL between positive and negative components in a misaligning scenario) is in the range of 2−3232-32 - 3. Lastly, for analyzing conversations, we require the assumption that the negative component is always more likely to output an ill-behaved answer than the positive component, to bound the change in log-likelihood between the two in the exchange between the model’s turn and the user’s turn: Definition 6. We say that ℙψsubscriptℙP_ψblackboard_Pψ is positive w.r.t ℙϕsubscriptℙitalic-ϕP_φblackboard_Pϕ on behavior B:Σ∗→[−1,1]:→superscriptΣ11B: ^*→[-1,1]B : Σ∗ → [ - 1 , 1 ] if for any prefix s0subscript0s_0s0 and a sentence s, that is negative, B⁢(s)<00B(s)<0B ( s ) < 0, the following holds: ℙψ⁢(s|s0)<ℙϕ⁢(s|s0)subscriptℙconditionalsubscript0subscriptℙitalic-ϕconditionalsubscript0P_ψ(s|s_0)<P_φ(s|s_0)blackboard_Pψ ( s | s0 ) < blackboard_Pϕ ( s | s0 ). 3 Results: Limitations of LLM Alignment In this section, we use the above framework of Behavior Expectation Bounds (BEB) in order to inform the question of when LLM alignment is robust or vulnerable to adversarial prompting attacks. We begin with our main result in section 3.1, which states that under assumptions of decomposability into distinguishable components of desired and undesired behavior, aligned LLMs are not protected against adversarial misaligning prompts (theorem 1). In section 3.2, we extend the above framework to include cases of (i) preset aligning prompts—we formally establish the benefits of this common practice by showing that in this case the length of the misaligning prompt must be linear in the length of the preset aligning prompt; (i) multi-turn interactions between adversarial users and LLMs—we find that if the user does not provide long enough misaligning prompts, the LLM can resist misalignment by making aligning replies to the user during a conversation; and (i) best-of-n sampling—we find that when sampling multiple responses from a model and choosing the most aligned one, the length of the misaligning prompt length increases logarithmically with the number of samples. 3.1 Misaligning via Adversarial Prompts Alignment impossibility We first show that if a model can be written as a distinct mixture of ill- and well-behaved components, then it can be misaligned via prompting: Theorem 1. Let γ∈[−1,0)10γ∈[-1,0)γ ∈ [ - 1 , 0 ), let B be a behaviour and ℙPblackboard_P be an unprompted language model such that B is α,β,γα,β,γα , β , γ-negatively-distinguishable in ℙPblackboard_P (definition 5). Then ℙPblackboard_P is γ-prompt-misalignable w.r.t. B (definition 1) with prompt length of 1β⁢(log⁡1α+log⁡1ϵ+log⁡4)111italic-ϵ4 1β( 1α+ 1ε+ 4)divide start_ARG 1 end_ARG start_ARG β end_ARG ( log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 ). Intuitively, theorem 1 implies that if a component of the distribution exhibits a negative behavior with expectation under γ, then there exists a prompt that triggers this behavior for the entire language model into behaving with expectation under γ. Importantly, no matter how low the prior of the negative component α is, the LLM is vulnerable to adversarial prompting that exposes this negative component’s behavior. Furthermore, the guaranteed misaligning prompt scales logarithmically in α−1superscript1α^-1α- 1, providing insight into why even very low probability behaviors can be enhanced with merely a few sentences. Additionally, we see that increased distinguishability can reduce the misaligning prompt length, meaning that while some behaviors may have lower probability (i.e. lower α), they may have higher distinguishability β, thus overall requiring shorter misaligning prompts. In appendix B we extend theorem 1 beyond existence of a misaligning prompt, to a more generalized notion of probability mass of misaligning prompts, and prove that with the sampling method we provide for them, they guarantee misalignment with high probability. Essentially, our proof follows the PAC based theoretical framework for in-context learning introduced in Wies et al. (2023), while relaxing their approximate independence assumption and adapting the analysis to the BEB framework. For proof see appendix D. 3.2 Extensions: Aligning Prompts, Conversations and Best-of-n Sampling Misaligning in the presence of preset aligning prompts A common practice for enhancing positive behavior is to include an initial ‘preset aligning prompt’, denoted s0subscript0s_0s0 below, hard coded as a prefix to the LLM’s input. The theorem below states that even in the presence of s0subscript0s_0s0, it is possible to prompt the LLM into an undesired behavior with a ‘misaligning prompt’. We show that the required prompt length for misalignment scales linearly with the length of s0subscript0s_0s0. Theorem 2. Let δ>00δ>0δ > 0, γ∈[−1,0)10γ∈[-1,0)γ ∈ [ - 1 , 0 ), B be a behaviour and ℙPblackboard_P be a language model such that B is α,β,γα,β,γα , β , γ-negatively-prompt-distinguishable in ℙPblackboard_P (definition 5). If the distribution corresponding to the well-behaved component of ℙPblackboard_P is β’-undistinguishable, σ-similar (definition 4) and positive (definition 6) with respect to to the ill-behaved component, then for an aligning prompt s0∼ℙ+⁢(⋅)similar-tosubscript0subscriptℙ⋅s_0 _+(·)s0 ∼ blackboard_P+ ( ⋅ ), the conditional LLM distribution ℙ(⋅|s0)P(·|s_0)blackboard_P ( ⋅ | s0 ) is γ-prompt-misalignable with probability 1−δ11- 1 - δ with prompt length 1β⁢(log⁡1α+log⁡1ϵ+log⁡4)+β′β⁢|s0|+σβ⁢|s0|δ+1111italic-ϵ4superscript′subscript0subscript01 1β( 1α+ 1ε+ 4)+ % β β|s_0|+ σβ |s_0|δ% +1divide start_ARG 1 end_ARG start_ARG β end_ARG ( log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 ) + divide start_ARG β′ end_ARG start_ARG β end_ARG | s0 | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG | s0 | end_ARG start_ARG δ end_ARG end_ARG + 1. Theorem 2 guarantees that even in the presence of a preset aligning prompt s0subscript0s_0s0, there exists a long enough prompt that will misalign the model. See figure 3a which demonstrates how an align-prompted model requires longer adversarial prompts to misalign than unprompted models. For proof see appendix E. Misaligning via conversation We show below that an undesired behavior can be elicited from an LLM via conversation with an adversarial user. Interestingly, we show that if the adversarial user does not use a long enough misaligning prompt in the first turn, then the LLM’s responses can hinder the user’s misaligning efforts. Intuitively, if a user begins a conversation by simply requesting “say a racist statement", an aligned LLM will likely reply “I will not say racist statements, that is harmful", and this reply in its prompt will cause the LLM to be more mindful of refraining from racist statements in the remainder of the conversation. Overall, due to this ‘misaligning resistance’ by the LLM, the user will need to insert more misaligning text in the conversation format than in the single prompt format of section 3.1 in order for our framework to guarantee misalignment. We formalize a conversation between a user and an LLM of distribution ℙPblackboard_P as a sequence of user queries followed by LLM responses which are sampled from the LLM’s conditional distribution given the conversation thus far. Formally, given the history of the conversation, q1,a1⁢…⁢qt,at,qt+1subscript1subscript1…subscriptsubscriptsubscript1q_1,a_1...q_t,a_t,q_t+1q1 , a1 … qitalic_t , aitalic_t , qitalic_t + 1, where qisubscriptq_iqitalic_i are the user’s inputs and aisubscripta_iaitalic_i are the LLM’s responses, the LLM generates a response at+1subscript1a_t+1aitalic_t + 1 by sampling from: at+1∼ℙ(⋅|q1,a1,…,qt,at,qt+1)a_t+1 (·|q_1,a_1,...,q_t,a_t,q_t+1)aitalic_t + 1 ∼ blackboard_P ( ⋅ | q1 , a1 , … , qitalic_t , aitalic_t , qitalic_t + 1 ). In the following theorem we show that under our distinguishability conditions, misalignment is possible also in conversation format: Theorem 3. Under the conditions of theorem 2 and that the distribution corresponding to the well-behaved component of ℙPblackboard_P is β’-prompt-undistinguishable to the ill-behaved component, in a conversation setting: q1,a1⁢…⁢qn,an,qn+1subscript1subscript1…subscriptsubscriptsubscript1q_1,a_1...q_n,a_n,q_n+1q1 , a1 … qitalic_n , aitalic_n , qitalic_n + 1, the model is γ-misalignable with total prompt length of ∑i=1n|qi|=1β⁢(log⁡1α+log⁡1ϵ+log⁡4)+∑i=1n(β′β⁢|ai|+σβ⁢n⁢|ai|δ)+nsuperscriptsubscript1subscript111italic-ϵ4superscriptsubscript1superscript′subscriptsubscript _i=1^n|q_i|= 1β( 1α+ 1% ε+ 4)+ _i=1^n ( β β|a_i|+% σβ n|a_i|δ )+n∑i = 1n | qitalic_i | = divide start_ARG 1 end_ARG start_ARG β end_ARG ( log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 ) + ∑i = 1n ( divide start_ARG β′ end_ARG start_ARG β end_ARG | aitalic_i | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG n | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG ) + n and each prompt of length no longer than |qi|≤β′β⁢|ai|+σβ⁢n⁢|ai|δ+log⁡1ϵ+log⁡1α+log⁡4n⁢β+1subscriptsuperscript′subscriptsubscript1italic-ϵ141|q_i|≤ β β|a_i|+ σβ % n|a_i|δ+ 1ε+ 1α+ 4% nβ+1| qitalic_i | ≤ divide start_ARG β′ end_ARG start_ARG β end_ARG | aitalic_i | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG n | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG + divide start_ARG log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log divide start_ARG 1 end_ARG start_ARG α end_ARG + log 4 end_ARG start_ARG n β end_ARG + 1. Comparing the above requirement on the amount of misaligning text to that required in the single prompting scenario of theorem 1, we see that it is larger by the total text generated by the model ∑i=1n|ai|superscriptsubscript1subscript _i=1^n|a_i|∑i = 1n | aitalic_i |. Intuitively, in the beginning of the conversation the model is aligned, so it is most likely that its response will be sampled from the well-behaved component, thus enhancing it over the ill-behaved component (see the proof of theorem 3 in appendix F for formalization of this intuition). Best-of-n sampling An additional aligning method that can be applied is to sample n conditional responses of a model to a prompt, then use a reward function to choose the most aligned one w.r.t. a desired behavior. In the following theorem, we show that while this method requires a longer misaligning prompt, alignment is not guaranteed: Theorem 4. Under the conditions of theorem 1, and using best of n sampling, i.e. argmaxy1…yn∼ℙ(⋅|x)⁢[B⁢(yi)]argmax_y_1...y_n (·|x)[B(y_i)]argmaxy start_POSTSUBSCRIPT 1 … yitalic_n ∼ blackboard_P ( ⋅ | x ) end_POSTSUBSCRIPT [ B ( yitalic_i ) ], ℙPblackboard_P is γ-prompt-misalignable with prompt length 1β⁢(log⁡1α+log⁡1ϵ+log⁡4+log⁡n)111italic-ϵ4 1β( 1α+ 1ε+ 4+ n)divide start_ARG 1 end_ARG start_ARG β end_ARG ( log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 + log n ). The proof is provided in appendix G. We note with regard to other sampling methods that do not use a selective behavior based reward function, such as greedy decoding and nucleus sampling, that the misaligning prompts constructed by theorem 1, lead to misalignment as a result of the convergence of the entire model to the negative behavior component. 4 Empirical Results In this section we demonstrate that several properties that are predicted by our theoretical framework manifest in experiments with common LLMs. Our empirical results are divided into two parts. First, we probe the range of realistic values for β (lower KL bound), β′β β′ (upper KL bound) and σ (log likelihood variance), by using real LLMs that display opposite behaviors (figure 2). Next, we employ the method used in our theoretical proofs for constructing an adversarial prompt in order to show that a real RLHF finetuned LLM distribution converges to a negative behavior distribution at a rate which corresponds to our theory (figure 3a) and that the behavior expectation of the RLHF finetuned LLM becomes negative with said adversarial prompt (figure 3b). We used models from the LLaMA 2 family Touvron et al. (2023). To obtain textual data that displays defined behaviors, we used the datasets of Perez et al. (2022) which contain statements classified to specific behaviors. In this section we demonstrate our results for the behavior “agreeableness", in the appendix section L, we show also for “anti-immigration". Note that our experiments do not use true subcomponents of LLMs, as there is no natural way to extract them out of a general distribution, instead we use proxies by LoRA finetuning models on specific behaviors. Even so, the results obtained from the experiments display dynamics of misalignment that are consistent with our theory. Additionally, if we assume the model trained on data that is similar to the data we finetuned on, then the proxy should resemble the true component, since as explained in 2.1 the components of the mixture can be thought of as being distributions over different parts of the training data. Our code is available at: https://github.com/yowolf/Limitations-of-Alignment-in-LLMs 4.1 Possible Values for β, β′β β′ and σ In our theoretical bounds, β, β′β β′ and σ (defined in section 2) play a central role: their absolute values, as well as their ratio, dictate the length of our guaranteed misaligning prompts in the various analyzed scenarios. Here we attempt to probe the possible values of β, β′β β′ and σ for two LLM-based distributions that display the negative and positive facets of the same behavior vertical, in an attempt to gain insight on realistic values of β, β′β β′ and σ within our framework. To this end, we calculate the KL-divergence and corresponding variance between two LLMs based on Llama-2 13B chat, where one was tuned on the data of Perez et al. (2022) to display negative behavior (see technical training details in appendix L) and the other was taken as is, since it already displayed the positive behavior. We denote these as ℙ−subscriptℙP_-blackboard_P- and ℙ+subscriptℙP_+blackboard_P+ but note that they are an approximation of a possible LLM decomposition as explained above. The results are displayed in figure 2 for the behavior “agreeableness" (as defined in Perez et al. (2022)). From the lower and upper bounds for the KL divergence, we estimate β and β′β β′ and from the linear upper bound for the corresponding variance, σ2superscript2σ^2σ2. In this case, β=2020β=20β = 20, β′=30superscript′30β =30β′ = 30, σ2=50superscript250σ^2=50σ2 = 50, hence σβ=0.350.35 σβ=0.35divide start_ARG σ end_ARG start_ARG β end_ARG = 0.35, β′β=1.5superscript′1.5 β β=1.5divide start_ARG β′ end_ARG start_ARG β end_ARG = 1.5. For numbers of this order, the ratio of σ/βσ/βσ / β is not too big compared to β′/βsuperscript′β /β′ / β, hence for δ of around 0.10.10.10.1, the terms in the upper bounds of theorems 2 and 3 that are linear in text length dominate the square root terms. For β-prompt-distinguishability, we ran a similar experiment in appendix N, by first inserting to both models a neutral prefix followed by a negative behavior sentence, then performing the above experiment. We observed that the approximated value of β remains similar (β≈2020β≈ 20β ≈ 20). Figure 2: (a) KL between two distributions of opposite behaviors as function of prompt length sampled from ℙ−subscriptℙP_-blackboard_P-, averaged on 10 sampled sequences. The red line is a lower bound for the KL divergence, hence a possible value of β. For these two distributions, we see β≈2020β≈ 20β ≈ 20. (b) Corresponding log ratio variance between the distributions mentioned in (a). 30 samples from ℙ−subscriptℙP_-blackboard_P- were used to evaluate the variance and its error. The red line is a linear curve upper bounding the variance, hence its tangent is a possible value for σ2superscript2σ^2σ2. As seen, for σ2≈50superscript250σ^2≈ 50σ2 ≈ 50 definition 4 is satisfied. 4.2 Demonstration of Misalignment via Convergence of LLM to ℙ−subscriptℙP_-blackboard_P- and via Behavior Expectation According to our theory, misalignment happens when the LLM distribution converges to its negative component ℙ−subscriptℙP_-blackboard_P- as both are conditioned on longer and longer prompts sampled from ℙ−subscriptℙP_-blackboard_P-. Consequently, the KL-divergence between ℙ−subscriptℙP_-blackboard_P- and the LLM also decays and is bounded by the following (see appendix H for proof of this dependence): DK⁢L(ℙ−(⋅|s)||ℙL⁢L⁢M(⋅|s))<log(1+elog⁡1α−β⁢|s|)D_KL(P_-(·|s)||P_LLM(·|s))< (1+e % 1α-β|s|)Ditalic_K L ( blackboard_P- ( ⋅ | s ) | | blackboard_PL L M ( ⋅ | s ) ) < log ( 1 + eroman_log divide start_ARG 1 end_ARG start_ARG α end_ARG - β | s | ) (7) Hence for short prompts it is bounded by log⁡1α−β⁢|s|1 1α-β|s|log divide start_ARG 1 end_ARG start_ARG α end_ARG - β | s | and after reaching a length |s|=log⁡1αβ1|s|= 1αβ| s | = divide start_ARG log divide start_ARG 1 end_ARG start_ARG α end_ARG end_ARG start_ARG β end_ARG, it quickly decays to zero. From this we see that the KL-divergence should converge to zero and that to a limited extent, we can use its value at |s|=00|s|=0| s | = 0 and tangent to find possible values for log⁡1α1 1αlog divide start_ARG 1 end_ARG start_ARG α end_ARG and β. Our objective here is to show that when prompted with our generated prompts, an actual LLM will converge to a negative behavior distribution in a similar manner to our theoretical prediction. As before, we substitute the negative component ℙ−subscriptℙP_-blackboard_P- with an LLM distribution that displays negative behavior, “ℙ−subscriptℙP_-blackboard_P-". Figure 3a demonstrates that an RLHF fine-tuned LLM distribution converges to “ℙ−subscriptℙP_-blackboard_P-" as both are conditioned on prompts sampled from the ill-behaved LLM (see appendix L for experimental details). We fit a linear curve to approximate an effective log⁡1α−β⁢|s|1 1α-β|s|log divide start_ARG 1 end_ARG start_ARG α end_ARG - β | s |, but note that the extracted values of α and β are an approximation, as the negative behavior LLM denoted by ℙ−subscriptℙP_-blackboard_P- is not the true sub-component of the RLHF fine-tuned LLM and that equation 7 is an upper bound which is not necessarily tight. Still, we find that the ratio 1β⁢log⁡1α=3113 1β 1α=3divide start_ARG 1 end_ARG start_ARG β end_ARG log divide start_ARG 1 end_ARG start_ARG α end_ARG = 3. We show below that this is similar to the actual misaligning length. Figure 3: (a) KL-divergence between ℙ−subscriptℙP_-blackboard_P- and an RLHF model (Llama 2 13B chat) as function of prompt length sampled from ℙ−subscriptℙP_-blackboard_P-, averaged on 10 sampled sequences. For the first three sentences, we can fit a curve to approximate log⁡1α−β⁢|s|1 1α-β|s|log divide start_ARG 1 end_ARG start_ARG α end_ARG - β | s |. (b) Demonstration of misaligning Llama 2 13B chat via our method of sampling sequences of negative behavior from ℙ−subscriptℙP_-blackboard_P-. As can be seen, the LLM distribution samples two types of behavior, one of negative behavior and one that tries to avoid it. Next, we show misalignment in terms of behavior expectation. As shown in figure 3b, using our method of sampling a misaligning prompt from ℙ−subscriptℙP_-blackboard_P-, an RLHF fine-tuned model loses its alignment as it is fed longer prompts from ℙ−subscriptℙP_-blackboard_P-. This fits our theoretical prediction (proven in appendix H) that with our misaligning prompt, s, the corresponding behavior expectation decays as a reverse sigmoid in |s||s|| s |, centered at |s|=l⁢o⁢g⁢1αβ1|s|= log 1αβ| s | = divide start_ARG l o g divide start_ARG 1 end_ARG start_ARG α end_ARG end_ARG start_ARG β end_ARG: Bℙ⁢(s)<11+eβ⁢|s|−l⁢o⁢g⁢1αsubscriptℙ11superscript1B_P(s)< 11+e^β|s|-log 1αBblackboard_P ( s ) < divide start_ARG 1 end_ARG start_ARG 1 + eitalic_β | s | - l o g divide start_ARG 1 end_ARG start_ARG α end_ARG end_ARG (8) Additionally, inserting an aligning prompt stalls misalignment by about one sentence, similarly to how the misaligning prompt length guarantee increases in theorem 2. Furthermore, in appendix M, we perform the same experiment for the unaligned pretrained model and find that it too misaligns with this method. This shows that the misaligning prompts from our theory are computationally tractable despite their specificity, due to the theoretical method of their construction. We also see that using an approximation for ℙ−subscriptℙP_-blackboard_P- and not the true subcomponent achieves misalignment with dynamics that are similar to our theory. Pretrained models vs RLHF models In appendix M we performed the same experiment for a pretrained model, that has not undergone an alignment procedure, and found that the approximated value for β is 5 times smaller than that of the RLHF model on both behaviors "agreeableness" and "anti-immigration", hinting that perhaps RLHF reduces the probability for negative behavior (i.e. α) but increases its distinguishability β at the same time. 5 Discussion The need for robust methods for AI alignment is pressing. Prominent actors in our field are advocating for halting LLM development until the means of controlling this technology are better understood (O’Brien, 2023). This paper brings forward the Behavior Expectation Bounds (BEB) theoretical framework, which is aimed at providing means for discussing core alignment issues in leading contemporary interactions between humans and LLMs. We used the BEB framework to make several fundamental assertions regarding alignment in LLMs. First, we showed that any realistic alignment process on frozen LLMs can be reversed via an adversarial prompt or conversation with an adversarial user. As a silver lining, we showed that the better aligned the model is to begin with, the longer the prompt required to reverse the alignment, so limited prompt lengths may serve as guardrails in theory. With that, we also show that this picture is more complex, and the distinguishability of undesired behavior components also facilitates easier misalignment. Thus, while attenuating undesired behaviors, the leading alignment practice of reinforcement learning from human feedback (RLHF) may also render these same undesired behaviors more easily accessible via adversarial prompts. We leave the latter statement as an open conjecture; this theoretical direction may explain the result in Perez et al. (2022), in which RLHF increases undesired behaviors in language models. These results highlight the importance of using alignment methods that control the model at inference time, such as representation engineering Zou et al. (2023); Turner et al. (2023). Our framework has several limitations further discussed in appendix A and we leave several issues open for future work. Andreas (2022) describe modern LLMs as comprised of distinct agents that manifest when the right prompt is inserted into the LLM. Our presented notions of decomposability into components and distinguishability between them are one analyzable choice of modeling multiple agents or personas composing the LLM distribution. We showed that with this choice several theoretical statements can be made that fit empirical observations on misalignment via prompting. While intuitive and reinforced by embedding space clustering experiments in the appendix, we leave it to future work to (i) further investigate superposition and decomposability in actual LLM distributions and (i) introduce more elaborate or more realistic assumptions on the manner in which agent or persona decomposition is manifested in actual LLM distributions, and use them to gain further theoretical insight on LLM alignment. Elucidating this picture also bears promise for new empirical methods for controlling ill-behaved components with actual LLMs. Furthermore, our framework assumes ground truth behavior scores per sentence, where in reality behavior scoring is more complex, e.g., over varying text granularities, hard to define behavior verticals, and ambiguous scoring. A deeper definition of behavior scoring may lead to new insights that can be drawn from the BEB theoretical framework. Acknowledgements This research was supported by the ERC (European Research Council) and the ISF (Israel Science Foundation). References Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. Andreas (2022) Jacob Andreas. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, p. 5769–5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.findings-emnlp.423. Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. Atillah (2023) Imane El Atillah. Man ends his life after an ai chatbot ’encouraged’ him to sacrifice himself to stop climate change. Euronews, 2023. Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, p. 610–623, 2021. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. Deshpande et al. (2023) Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023. Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423. Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 3356–3369, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.301. URL https://aclanthology.org/2020.findings-emnlp.301. Hendrycks et al. (2021) Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021. Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 5491–5501, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.487. URL https://aclanthology.org/2020.acl-main.487. Jorgensen et al. (2023) Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813, 2023. Leong et al. (2023) Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. Self-detoxifying language models via toxification reversal. arXiv preprint arXiv:2310.09573, 2023. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229. Liu et al. (2023) Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997, 2023. Mangrulkar et al. (2022) Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022. Meta (2023) AI Meta. Introducing llama: A foundational, 65-billion-parameter large language model. Meta AI. https://ai. facebook. com/blog/large-language-model-llama-meta-ai, 2023. Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 1953–1967, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.154. URL https://aclanthology.org/2020.emnlp-main.154. Nardo (2023) Cleo Nardo. The waluigi effect (mega-post). Less Wrong, 2023. Ngo (2022) Richard Ngo. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022. Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. O’Brien (2023) Matt O’Brien. Musk, scientists call for halt to ai race sparked by chatgpt. AP News, 2023. OpenAI (2023) OpenAI. Gpt-4 technical report, 2023. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=TG8KACxEON. Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye. Park et al. (2023) Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. Perez et al. (2022) Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022. Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. Roose (2023) Kevin Roose. A conversation with bing’s chatbot left me deeply unsettled. New York Times, 2023. Schulman et al. (2023) John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe, Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes, Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, Joel Parish, Dave Cummings, Rajeev Nayak, Valerie Balcom, David Schnurr, Tomer Kaftan, Chris Hallacy, Nicholas Turley, Noah Deutsch, Vik Goel, Jonathan Ward, Aris Konstantinidis, Wojciech Zaremba, Long Ouyang, Leonard Bogdonoff, Joshua Gross, David Medina, Sarah Yoo, Teddy Lee, Ryan Lowe, Dan Mossing, Joost Huizinga, Roger Jiang, Carroll Wainwright, Diogo Almeida, Steph Lin, Marvin Zhang, Kai Xiao, Katarina Slama, Steven Bills, Alex Gray, Jan Leike, Jakub Pachocki, Phil Tillet, Shantanu Jain, Greg Brockman, Nick Ryder, Alex Paino, Qiming Yuan, Clemens Winter, Ben Wang, Mo Bavarian, Igor Babuschkin, Szymon Sidor, Ingmar Kanitscheider, Mikhail Pavlov, Matthias Plappert, Nik Tezak, Heewoo Jun, William Zhuk, Vitchyr Pong, Lukasz Kaiser, Jerry Tworek, Andrew Carr, Lilian Weng, Sandhini Agarwal, Karl Cobbe, Vineet Kosaraju, Alethea Power, Stanislas Polu, Jesse Han, Raul Puri, Shawn Jain, Benjamin Chess, Christian Gibson, Oleg Boiko, Emy Parparita, Amin Tootoonchian, Kyle Kosic, and Christopher Hesse. Introducing chatgpt. OpenAI blog, 2023. Shalev-Shwartz et al. (2020) Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. On the ethics of building ai in a responsible manner. arXiv preprint arXiv:2004.04644, 2020. Subhash (2023) Varshini Subhash. Can large language models change user preference adversarially? arXiv preprint arXiv:2302.10291, 2023. Taylor et al. (2016) Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems. Ethics of Artificial Intelligence, p. 342–382, 2016. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Turner et al. (2023) Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023. Venkit et al. (2022) Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. A study of implicit bias in pretrained language models against people with disabilities. In Proceedings of the 29th International Conference on Computational Linguistics, p. 1324–1332, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclanthology.org/2022.coling-1.113. Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 2153–2162, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1221. URL https://aclanthology.org/D19-1221. Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, p. 214–229, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/3531146.3533088. URL https://doi.org/10.1145/3531146.3533088. West (2023) Colin G West. Advances in apparent conceptual physics reasoning in gpt-4. arXiv e-prints, p. arXiv–2303, 2023. Wies et al. (2023) Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=f3JNQd7CHM. Xu et al. (2021) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 2950–2968, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.235. URL https://aclanthology.org/2021.naacl-main.235. Yu & Sagae (2021) Dian Yu and Kenji Sagae. Automatically exposing problems with neural dialog models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 456–470, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.37. URL https://aclanthology.org/2021.emnlp-main.37. Yudkowsky (2001) Eliezer Yudkowsky. Creating friendly ai 1.0: The analysis and design of benevolent goal architectures. The Singularity Institute, San Francisco, USA, 2001. Zou et al. (2023) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. Appendix A Discussion of limitations Our framework makes several underlying assumptions. Here we discuss their necessity and limitations as well as provide intuition. A.1 Two components mixture As explained in section 2.1, an unprompted LLM is a function that assigns probability to text based on the statistics of its training data, thus it can always be written as a sum of components by introducing latent variables, such as the sources of the training data ℙi=∑iwi⁢ℙisubscriptℙsubscriptsubscriptsubscriptℙP_i= _iw_iP_iblackboard_Pi = ∑i witalic_i blackboard_Pi and that different sources of text display different behavior. Next, we note that a general multiple component mixture can be partitioned to yield a two-component mixture: ℙ=∑i∈A∪Bwi⁢ℙi=∑i∈Awi⁢ℙi+∑i∈Bwi⁢ℙi=(∑i∈Awi)⁢∑i∈Awi∑i∈Awi⁢ℙi+(∑i∈Bwi)⁢∑i∈Bwi∑i∈Bwi⁢ℙiℙsubscriptsubscriptsubscriptℙsubscriptsubscriptsubscriptℙsubscriptsubscriptsubscriptℙsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptℙsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptℙP= _i∈ A∪ Bw_iP_i= _i∈ Aw_iP% _i+ _i∈ Bw_iP_i= ( _i∈ Aw_i ) _i% ∈ A w_i _i∈ Aw_iP_i+ ( _i∈ Bw_i% ) _i∈ B w_i _i∈ Bw_iP_iblackboard_P = ∑i ∈ A ∪ B witalic_i blackboard_Pi = ∑i ∈ A witalic_i blackboard_Pi + ∑i ∈ B witalic_i blackboard_Pi = ( ∑i ∈ A witalic_i ) ∑i ∈ A divide start_ARG witalic_i end_ARG start_ARG ∑i ∈ A witalic_i end_ARG blackboard_Pi + ( ∑i ∈ B witalic_i ) ∑i ∈ B divide start_ARG witalic_i end_ARG start_ARG ∑i ∈ B witalic_i end_ARG blackboard_Pi (9) Where A and B are disjoint sets of indices. By denoting ∑i∈Awi:=αassignsubscriptsubscript _i∈ Aw_i:=α∑i ∈ A witalic_i := α we see that ∑i∈Bwi=1−∑i∈Awi=1−αsubscriptsubscript1subscriptsubscript1 _i∈ Bw_i=1- _i∈ Aw_i=1-α∑i ∈ B witalic_i = 1 - ∑i ∈ A witalic_i = 1 - α. Then we see that ∑i∈Awi∑i∈Awi⁢ℙi:=ℙ−assignsubscriptsubscriptsubscriptsubscriptsubscriptℙsubscriptℙ _i∈ A w_i _i∈ Aw_iP_i:=P_-∑i ∈ A divide start_ARG witalic_i end_ARG start_ARG ∑i ∈ A witalic_i end_ARG blackboard_Pi := blackboard_P- and ∑i∈Bwi∑i∈Bwi⁢ℙi:=ℙ+assignsubscriptsubscriptsubscriptsubscriptsubscriptℙsubscriptℙ _i∈ B w_i _i∈ Bw_iP_i:=P_+∑i ∈ B divide start_ARG witalic_i end_ARG start_ARG ∑i ∈ B witalic_i end_ARG blackboard_Pi := blackboard_P+ are indeed normalized distributions, leading to ℙ=α⁢ℙ−+(1−α)⁢ℙ+ℙsubscriptℙ1subscriptℙP= _-+(1-α)P_+blackboard_P = α blackboard_P- + ( 1 - α ) blackboard_P+. Second, note that the assumption of the two component mixture is on the accumulating sequence probability: ℙ⁢(s1⊕…⊕sn)=α⁢ℙ−⁢(s1⊕…⊕sn)+(1−α)⁢ℙ+⁢(s1⊕…⊕sn)ℙdirect-sumsubscript1…subscriptsubscriptℙdirect-sumsubscript1…subscript1subscriptℙdirect-sumsubscript1…subscriptP(s_1 ... s_n)= _-(s_1 ...% s_n)+(1-α)P_+(s_1 ... s_n)blackboard_P ( s1 ⊕ … ⊕ sitalic_n ) = α blackboard_P- ( s1 ⊕ … ⊕ sitalic_n ) + ( 1 - α ) blackboard_P+ ( s1 ⊕ … ⊕ sitalic_n ) and not the conditional response to the prompt, which is: ℙ⁢(sn|s1⊕…⊕sn−1)=11+1−α⁢ℙ+⁢(s1⊕…⊕sn−1)ℙ−⁢(s1⊕…⊕sn−1)⁢ℙ−⁢(sn|s1⊕…⊕sn−1)+11+α1−α⁢ℙ−⁢(s1⊕…⊕sn−1)ℙ+⁢(s1⊕…⊕sn−1)⁢ℙ+⁢(sn|s1⊕…⊕sn−1)ℙconditionalsubscriptdirect-sumsubscript1…subscript1111subscriptℙdirect-sumsubscript1…subscript1subscriptℙdirect-sumsubscript1…subscript1subscriptℙconditionalsubscriptdirect-sumsubscript1…subscript1111subscriptℙdirect-sumsubscript1…subscript1subscriptℙdirect-sumsubscript1…subscript1subscriptℙconditionalsubscriptdirect-sumsubscript1…subscript1P(s_n|s_1 ... s_n-1)= 11+ 1-α% α P_+(s_1 ... s_n-1)P_-(s_1% ... s_n-1)P_-(s_n|s_1 ... s_n-1)+% 11+ α1-α P_-(s_1 ... s_% n-1)P_+(s_1 ... s_n-1)P_+(s_n|s_% 1 ... s_n-1)blackboard_P ( sitalic_n | s1 ⊕ … ⊕ sitalic_n - 1 ) = divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s1 ⊕ … ⊕ sitalic_n - 1 ) end_ARG start_ARG blackboard_P- ( s1 ⊕ … ⊕ sitalic_n - 1 ) end_ARG end_ARG blackboard_P- ( sitalic_n | s1 ⊕ … ⊕ sitalic_n - 1 ) + divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG α end_ARG start_ARG 1 - α end_ARG divide start_ARG blackboard_P- ( s1 ⊕ … ⊕ sitalic_n - 1 ) end_ARG start_ARG blackboard_P+ ( s1 ⊕ … ⊕ sitalic_n - 1 ) end_ARG end_ARG blackboard_P+ ( sitalic_n | s1 ⊕ … ⊕ sitalic_n - 1 ) (10) As can be seen, the zero-shot priors are α and 1−α11- 1 - α, but the priors of the conditional negative and positive components are highly dependent on the context, they contain the ratio of the probabilities of the prompt by the components ℙ−⁢(s1⊕…⊕sn−1)subscriptℙdirect-sumsubscript1…subscript1P_-(s_1 ... s_n-1)blackboard_P- ( s1 ⊕ … ⊕ sitalic_n - 1 ), ℙ+⁢(s1⊕…⊕sn−1)subscriptℙdirect-sumsubscript1…subscript1P_+(s_1 ... s_n-1)blackboard_P+ ( s1 ⊕ … ⊕ sitalic_n - 1 ), thus a prompt that is much more probable in the negative component will give a high weight to the conditional negative component. An adversarial prompt will have a large ratio ℙ−⁢(p⁢r⁢o⁢m⁢p⁢t)/ℙ+⁢(p⁢r⁢o⁢m⁢p⁢t)subscriptℙsubscriptℙP_-(prompt)/P_+(prompt)blackboard_P- ( p r o m p t ) / blackboard_P+ ( p r o m p t ), so it will significantly enhance the prior of the conditional ℙ−subscriptℙP_-blackboard_P-. The importance of using the mixture model is that it captures the concept of prompts that are out of distribution of the positive component and in the distribution of the negative component to refactor the coefficients of the effective mixture model. The intuition of using a mixture for the accumulating sequence probability is a mixture of text generating processes, where a sub-component ℙisubscriptℙP_iblackboard_Pi may be enhanced due to the sequence being highly probable in its distribution and out of distribution of the other components. This creates a strong dependence of the prompted model on the context, as observed in language models. The prior α is the zero-shot probability which is set and determines the initial weight of a sequence according to each text generating process. A.2 β-distinguishability As seen in the above discussion of components, the reweighting of the conditional negative component prior is based on inserting prompts that are not likely to be outputted by the positive component and likely by the negative component. To build such prompts, we need the distributions to maintain a "finite distance" from each other which allows to sample prompts from ℙ−subscriptℙP_-blackboard_P- that enhance the ratio ℙ−⁢(p⁢r⁢o⁢m⁢p⁢t)/ℙ+⁢(p⁢r⁢o⁢m⁢p⁢t)subscriptℙsubscriptℙP_-(prompt)/P_+(prompt)blackboard_P- ( p r o m p t ) / blackboard_P+ ( p r o m p t ). Feeding it to the model enhances the prior of the conditional ℙ−subscriptℙP_-blackboard_P- as seen in equation 10. The finite β is what creates the logarithmic scaling of the misaligning prompt on the prior α, since each sentence reweights the negative prior by a factor eβsuperscripte^βeitalic_β w.r.t the positive prior. If β is not finite but decaying, then we may get other dependences, as discussed in I Section 4.1 shows an example of two distributions that are β-distinguishable - two LLMs of opposing behaviors maintain a finite conditional KL-divergence when sampling zero-shot prompts from the negative component. Section 4.2 shows an example of two distributions that are not β-distinguishable, as the conditional KL between the two distributions decays the longer the prompt sampled from the negative component, which results in the misalignment of the LLM. A.3 Limitation of results Sentence-wise approach Our results provide guarantees for misalignment of LLMs in the sense of the next sentence produced by the model being misaligned. For more nuanced types of misalignment, such as long model outputs, one would need a behavior scoring function over the entire output and not just the first sentence. It is possible to generalize this work to such definitions of misalignment by changing the unit block of text from sentence to paragraphs, though the numerical value for the coefficients α,βα,βα , β will change. For the purposes of demonstrating the possibility of misaligning models, we kept the sentence approach which is more comprehendible and not application specific. Computational tractability Our theorems prove their existence by construction via sampling prompts from a negative behavior subcomponent of the model. However, in real applications, the subcomponent is not accessible to us. Even so, one can see from equation 10 that the mechanism of the reweight of the negative behavior component prior is to insert a prompt that satisfies ℙ−⁢(p⁢r⁢o⁢m⁢p⁢t)/ℙ+⁢(p⁢r⁢o⁢m⁢p⁢t)≫1−αmuch-greater-thansubscriptℙsubscriptℙ1P_-(prompt)/P_+(prompt) 1-αblackboard_P- ( p r o m p t ) / blackboard_P+ ( p r o m p t ) ≫ divide start_ARG 1 - α end_ARG start_ARG α end_ARG. Our theoretical work shows that ℙ−⁢(p⁢r⁢o⁢m⁢p⁢t)/ℙ+⁢(p⁢r⁢o⁢m⁢p⁢t)>eβ⁢|p⁢r⁢o⁢m⁢p⁢t|subscriptℙsubscriptℙsuperscriptP_-(prompt)/P_+(prompt)>e^β|prompt|blackboard_P- ( p r o m p t ) / blackboard_P+ ( p r o m p t ) > eitalic_β | p r o m p t | when the prompt is sampled from ℙ−subscriptℙP_-blackboard_P-, leading to our misalignment length guarantee. But, for practical applications, we see (as demonstrated in subsection 4.2) that using a proxy for ℙ−subscriptℙP_-blackboard_P- such as the LoRA finetuned LLM on negative behavior, is able to misalign in the exponential rate of our theory. Efficiency The prompt lengths provided are upper bounds, meaning there could be shorter misaligning prompts in practice. Even so, section 4.2 shows that both the theoretical value of the misaligning prompt length and the practical misaligning prompt length are relatively short (few sentences), making the bound practical. Appendix B Generalized misalignment Theorem 1 can be extended beyond existence of a misaligning prompt to a probability of sampling a misaligning prompt: Theorem 5. Let δ,ϵ>0italic-ϵ0δ,ε>0δ , ϵ > 0, under the conditions of theorem 1 and that the negative component is σ-similar to the positive component, when sampling a prompt of length: |s|>max⁡2β⁢(log⁡1α+log⁡1ϵ+log⁡(4)),4⁢σ2β2⁢δ211italic-ϵ44superscript2superscript2|s|> \ 2β ( 1α+ 1% ε+ (4) ), 4σ^2β^2δ \| s | > max divide start_ARG 2 end_ARG start_ARG β end_ARG ( log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log ( 4 ) ) , divide start_ARG 4 σ2 end_ARG start_ARG β2 δ end_ARG (11) from ℙ−subscriptℙP_-blackboard_P-, the behavior expectation of the model is bounded by Bℙ⁢(s)<γ+ϵsubscriptℙitalic-ϵB_P(s)<γ+ _P ( s ) < γ + ϵ with probability 1−δ11- 1 - δ. This result shows that not only a misaligning prompt exists, but that most prompts sampled from ℙ−subscriptℙP_-blackboard_P- are misaligning if they are long enough. This can be seen in experimental section 4.2, where we sample prompts from a negative behavior LLM and observe that on average they misalign the model. Proof. Following the proof of the main theorems, from equation 52: s∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)]>β⁢|s|=β⁢nsubscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙE_s _-(·) [ P_-(s)% P_+(s) ]>β|s|=β nblackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] > β | s | = β n (12) And σ-similarity: V⁢a⁢rs∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)]<σ⁢|s|=σ⁢nsubscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙVar_s _-(·) [ P_-(s)P% _+(s) ]<σ|s|=σ nV a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] < σ | s | = σ n (13) We can use Cantelli’s inequality to obtain: ℙ⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)<(β−σ|s|⁢δ)⁢|s|]<δℙdelimited-[]subscriptℙsubscriptℙP [ P_-(s)P_+(s)<(β- % σ |s|δ)|s| ]< _P [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG < ( β - divide start_ARG σ end_ARG start_ARG square-root start_ARG | s | δ end_ARG end_ARG ) | s | ] < δ (14) Demand: log⁡ℙ−⁢(s)ℙ+⁢(s)>log⁡1ϵsubscriptℙsubscriptℙ1italic-ϵ P_-(s)P_+(s)> 1εlog divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG > log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG (15) This happens with probability 1−δ11- 1 - δ for: log⁡ℙ−⁢(s)ℙ+⁢(s)≥(β−σ|s|⁢δ)⁢|s|≥β2⁢|s|>log⁡1ϵsubscriptℙsubscriptℙ21italic-ϵ P_-(s)P_+(s)≥(β- σ % |s|δ)|s|≥ β2|s|> 1εlog divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ≥ ( β - divide start_ARG σ end_ARG start_ARG square-root start_ARG | s | δ end_ARG end_ARG ) | s | ≥ divide start_ARG β end_ARG start_ARG 2 end_ARG | s | > log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG (16) Where the transition before the last happens for σ|s|⁢δ<β2↔|s|>4⁢σ2β2⁢δ↔24superscript2superscript2 σ |s|δ< β2 |s|> 4% σ^2β^2δdivide start_ARG σ end_ARG start_ARG square-root start_ARG | s | δ end_ARG end_ARG < divide start_ARG β end_ARG start_ARG 2 end_ARG ↔ | s | > divide start_ARG 4 σ2 end_ARG start_ARG β2 δ end_ARG Plugging this into the proof of theorem 1 gives that Bℙ⁢(s)<γ+ϵsubscriptℙitalic-ϵB_P(s)<γ+ _P ( s ) < γ + ϵ for: |s|>max⁡2β⁢(log⁡1α+log⁡1ϵ+log⁡(4)),4⁢σ2β2⁢δ211italic-ϵ44superscript2superscript2|s|> \ 2β( 1α+ 1ε+% (4)), 4σ^2β^2δ \| s | > max divide start_ARG 2 end_ARG start_ARG β end_ARG ( log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log ( 4 ) ) , divide start_ARG 4 σ2 end_ARG start_ARG β2 δ end_ARG (17) ∎ Appendix C Proofs building blocks In this section, we prove three technical lemmas which are the building blocks for proving our results. In subsection C.1 we prove that prompts can reweight the initial prior distribution of mixture components. In subsection C.2 we show that such reweighting alters the behaviour of the mixture distribution. And finally, in subsection C.3 we shows that under our α,β,γα,β,γα , β , γ-negative-distinguishability assumption, such prompts always exists. C.1 Convergence to a single component In this subsection, we prove a technical lemma which shows that when the likelihood of a prompt s0subscript0s_0s0 is relatively high according to a mixture component, then the conditional mixture distribution converges to the conditional distribution of that single component. Essentially, this lemma strengthening the analysis in theorem 1 of Wies et al. (2023), and formulate the role of prompts as reweighting of the prior distribution. In the next subsection, we will show that indeed our notion of convergence implies also the convergence of behaviors. Lemma 1. Let ℙPblackboard_P be a mixture distribution that can be written as α⁢ℙ0+(1−α)⁢ℙ1subscriptℙ01subscriptℙ1 _0+ (1-α )P_1α blackboard_P0 + ( 1 - α ) blackboard_P1. Then for any initial prompt s0subscript0s_0s0 and any string s such that ℙ0⁢(s|s0)>0subscriptℙ0conditionalsubscript00P_0(s|s_0)>0blackboard_P0 ( s | s0 ) > 0 the following holds: |ℙ⁢(s|s0)ℙ0⁢(s|s0)−1|≤1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)⋅max⁡ℙ1⁢(s|s0)ℙ0⁢(s|s0),1ℙconditionalsubscript0subscriptℙ0conditionalsubscript01⋅1subscriptℙ1subscript0subscriptℙ0subscript0subscriptℙ1conditionalsubscript0subscriptℙ0conditionalsubscript01 | P (s\,|\,s_0 )P_0% (s\,|\,s_0 )-1 |≤ 1-α· % P_1 (s_0 )P_0 (s_0 )· % \ P_1 (s\,|\,s_0 )P_0 (s\,|% \,s_0 ),1 \| divide start_ARG blackboard_P ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG - 1 | ≤ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG ⋅ max divide start_ARG blackboard_P1 ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG , 1 (18) Intuitively, when ℙ⁢(s0⊕s)ℙdirect-sumsubscript0P(s_0 s)blackboard_P ( s0 ⊕ s ) is equals to ℙ0⁢(s0⊕s)subscriptℙ0direct-sumsubscript0P_0(s_0 s)blackboard_P0 ( s0 ⊕ s ) theirs ratio is one, and we bound the deviation from these case. Note that our bound implicitly implies the following additive notion of convergence: |ℙ⁢(s0⊕s)−ℙ0⁢(s0⊕s)|≤1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)ℙdirect-sumsubscript0subscriptℙ0direct-sumsubscript0⋅1subscriptℙ1subscript0subscriptℙ0subscript0 |P(s_0 s)-P_0(s_0 s)% |≤ 1-α· P_1 (s_0 )% P_0 (s_0 )| blackboard_P ( s0 ⊕ s ) - blackboard_P0 ( s0 ⊕ s ) | ≤ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG (19) Proof. We begin by explicitly writing the conditional likelihood of s given s0subscript0s_0s0: ℙ⁢(s|s0)=ℙ⁢(s0⊕s)ℙ⁢(s0)=α⁢ℙ0⁢(s0⊕s)+(1−α)⁢ℙ1⁢(s0⊕s)α⁢ℙ0⁢(s0)+(1−α)⁢ℙ1⁢(s0)ℙconditionalsubscript0ℙdirect-sumsubscript0ℙsubscript0subscriptℙ0direct-sumsubscript01subscriptℙ1direct-sumsubscript0subscriptℙ0subscript01subscriptℙ1subscript0 (s\,|\,s_0 )= P (s_0% s )P (s_0 )= _0 % (s_0 s )+ (1-α )P_1 (s_0 s% ) _0 (s_0 )+ (1-α )P% _1 (s_0 )blackboard_P ( s | s0 ) = divide start_ARG blackboard_P ( s0 ⊕ s ) end_ARG start_ARG blackboard_P ( s0 ) end_ARG = divide start_ARG α blackboard_P0 ( s0 ⊕ s ) + ( 1 - α ) blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) + ( 1 - α ) blackboard_P1 ( s0 ) end_ARG (20) Now since both (1−α)1(1-α)( 1 - α ) and ℙ1⁢(s0⊕s)subscriptℙ1direct-sumsubscript0P_1(s_0 s)blackboard_P1 ( s0 ⊕ s ) are greater than zero, we can bound ℙ⁢(s|s0)ℙconditionalsubscript0P (s\,|\,s_0 )blackboard_P ( s | s0 ) from below by removing these terms from the numerator and get that: ℙ⁢(s|s0)≥α⁢ℙ0⁢(s0⊕s)α⁢ℙ0⁢(s0)+(1−α)⁢ℙ1⁢(s0)ℙconditionalsubscript0subscriptℙ0direct-sumsubscript0subscriptℙ0subscript01subscriptℙ1subscript0 (s\,|\,s_0 )≥ _0% (s_0 s ) _0 (s_0 )+ (1-% α )P_1 (s_0 )blackboard_P ( s | s0 ) ≥ divide start_ARG α blackboard_P0 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) + ( 1 - α ) blackboard_P1 ( s0 ) end_ARG (21) Which after division of both the numerator and the denominator by α⋅ℙ0⁢(s0⊕s)⋅subscriptℙ0direct-sumsubscript0α·P_0 (s_0 s )α ⋅ blackboard_P0 ( s0 ⊕ s ) is equals to: ℙ0⁢(s|s0)⋅(1+1−α⋅ℙ1⁢(s0)ℙ0⁢(s0))−1⋅subscriptℙ0conditionalsubscript0superscript1⋅1subscriptℙ1subscript0subscriptℙ0subscript01 _0 (s\,|\,s_0 )· (1+ 1-α% α· P_1 (s_0 )P_0 (s_0% ) )^-1blackboard_P0 ( s | s0 ) ⋅ ( 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG )- 1 (22) Now, since 11+x≥1−x111 11+x≥ 1-xdivide start_ARG 1 end_ARG start_ARG 1 + x end_ARG ≥ 1 - x for any x≥00x≥ 0x ≥ 0, we gets that ℙ⁢(s|s0)ℙconditionalsubscript0P (s\,|\,s_0 )blackboard_P ( s | s0 ) is greater than: ℙ0⁢(s|s0)⋅(1−1−α⋅ℙ1⁢(s0)ℙ0⁢(s0))⋅subscriptℙ0conditionalsubscript01⋅1subscriptℙ1subscript0subscriptℙ0subscript0 _0 (s\,|\,s_0 )· (1- 1-α% α· P_1 (s_0 )P_0 (s_0% ) )blackboard_P0 ( s | s0 ) ⋅ ( 1 - divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG ) (23) Finally, we divide the inequality by ℙ0⁢(s|s0)subscriptℙ0conditionalsubscript0P_0(s|s_0)blackboard_P0 ( s | s0 ) and subtracts 1111 to get one side of equation’s 18 inequality: ℙ⁢(s|s0)ℙ0⁢(s|s0)−1≥−1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)ℙconditionalsubscript0subscriptℙ0conditionalsubscript01⋅1subscriptℙ1subscript0subscriptℙ0subscript0 P (s\,|\,s_0 )P_0 (s\,% |\,s_0 )-1≥- 1-α· P_1 (s% _0 )P_0 (s_0 )divide start_ARG blackboard_P ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG - 1 ≥ - divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG (24) Moving to the other side of the inequality, since both (1−α)1(1-α)( 1 - α ) and ℙ1⁢(s0⊕s)subscriptℙ1direct-sumsubscript0P_1 (s_0 s )blackboard_P1 ( s0 ⊕ s ) are greater than zero, we can bound ℙ⁢(s|s0)ℙconditionalsubscript0P (s\,|\,s_0 )blackboard_P ( s | s0 ) from above by removing these terms from the denominator and get that : ℙ⁢(s|s0)=α⁢ℙ0⁢(s0⊕s)+(1−α)⁢ℙ1⁢(s0⊕s)α⁢ℙ0⁢(s0)+(1−α)⁢ℙ1⁢(s0)≤α⁢ℙ0⁢(s0⊕s)+(1−α)⁢ℙ1⁢(s0⊕s)α⁢ℙ0⁢(s0)ℙconditionalsubscript0subscriptℙ0direct-sumsubscript01subscriptℙ1direct-sumsubscript0subscriptℙ0subscript01subscriptℙ1subscript0subscriptℙ0direct-sumsubscript01subscriptℙ1direct-sumsubscript0subscriptℙ0subscript0 (s\,|\,s_0 )= _0% (s_0 s )+ (1-α )P_1 (s_0% s ) _0 (s_0 )+ (1-α )% P_1 (s_0 )≤ _0 (s_0% s )+ (1-α )P_1 (s_0 s )% _0 (s_0 )blackboard_P ( s | s0 ) = divide start_ARG α blackboard_P0 ( s0 ⊕ s ) + ( 1 - α ) blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) + ( 1 - α ) blackboard_P1 ( s0 ) end_ARG ≤ divide start_ARG α blackboard_P0 ( s0 ⊕ s ) + ( 1 - α ) blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) end_ARG (25) Which after division of both the numerator and the denominator by α⋅ℙ0⁢(s0)⋅subscriptℙ0subscript0α·P_0 (s_0 )α ⋅ blackboard_P0 ( s0 ) is equals to: α⁢ℙ0⁢(s0⊕s)α⁢ℙ0⁢(s0)+(1−α)⁢ℙ1⁢(s0⊕s)α⁢ℙ0⁢(s0)=ℙ0⁢(s|s0)+(1−α)⁢ℙ1⁢(s0⊕s)α⁢ℙ0⁢(s0)subscriptℙ0direct-sumsubscript0subscriptℙ0subscript01subscriptℙ1direct-sumsubscript0subscriptℙ0subscript0subscriptℙ0conditionalsubscript01subscriptℙ1direct-sumsubscript0subscriptℙ0subscript0 _0 (s_0 s )α% P_0 (s_0 )+ (1-α )P_1% (s_0 s ) _0 (s_0 )=P% _0 (s\,|\,s_0 )+ (1-α )P_1 (s_% 0 s ) _0 (s_0 )divide start_ARG α blackboard_P0 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) end_ARG + divide start_ARG ( 1 - α ) blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) end_ARG = blackboard_P0 ( s | s0 ) + divide start_ARG ( 1 - α ) blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG α blackboard_P0 ( s0 ) end_ARG (26) Now, we can use the fact that ℙ1⁢(s0⊕s1)=ℙ1⁢(s0)⋅ℙ1⁢(s|s0)subscriptℙ1direct-sumsubscript0subscript1⋅subscriptℙ1subscript0subscriptℙ1conditionalsubscript0P_1 (s_0 s_1 )=P_1 (s_0 )% ·P_1 (s\,|\,s_0 )blackboard_P1 ( s0 ⊕ s1 ) = blackboard_P1 ( s0 ) ⋅ blackboard_P1 ( s | s0 ) to get that ℙ⁢(s|s0)ℙconditionalsubscript0P (s\,|\,s_0 )blackboard_P ( s | s0 ) is at most: ℙ0⁢(s|s0)+(1−α)⁢ℙ1⁢(s0)⁢ℙ1⁢(s|s0)α⁢ℙ0⁢(s0)subscriptℙ0conditionalsubscript01subscriptℙ1subscript0subscriptℙ1conditionalsubscript0subscriptℙ0subscript0 _0 (s\,|\,s_0 )+ (1-α)P% _1 (s_0 )P_1 (s\,|\,s_0 ) % _0 (s_0 )blackboard_P0 ( s | s0 ) + divide start_ARG ( 1 - α ) blackboard_P1 ( s0 ) blackboard_P1 ( s | s0 ) end_ARG start_ARG α blackboard_P0 ( s0 ) end_ARG (27) Which after division by ℙ0⁢(s|s0)subscriptℙ0conditionalsubscript0P_0(s|s_0)blackboard_P0 ( s | s0 ) and subtraction of 1111 yield the other side of equation’s 18 inequality: ℙ⁢(s|s0)ℙ0⁢(s|s0)−1≤ℙ1⁢(s0)ℙ0⁢(s0)⁢(1−α)⁢ℙ1⁢(s|s0)α⁢ℙ0⁢(s|s0)ℙconditionalsubscript0subscriptℙ0conditionalsubscript01subscriptℙ1subscript0subscriptℙ0subscript01subscriptℙ1conditionalsubscript0subscriptℙ0conditionalsubscript0 P (s\,|\,s_0 )P_0(s|s_0)% -1≤ P_1(s_0)P_0(s_0) (1-α)% P_1(s|s_0) _0(s|s_0)divide start_ARG blackboard_P ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG - 1 ≤ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG divide start_ARG ( 1 - α ) blackboard_P1 ( s | s0 ) end_ARG start_ARG α blackboard_P0 ( s | s0 ) end_ARG (28) Finally, combining both inequalities yields equation 18. ∎ C.2 Behavioral implication of the convergence to a single component In this subsection, we prove a technical lemma which shows that when the likelihood of a prompt s0subscript0s_0s0 is relatively high according to a mixture component, then the conditional mixture distribution converge to the conditional distribution of that single component. In the next sections, we will use this lemma to prove the theorems from the main text. Lemma 2. Let B be a behaviour, then under the conditions of lemma 1 the following holds: |Bℙ⁢(s0)−Bℙ0⁢(s0)|≤2⋅1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)subscriptℙsubscript0subscriptsubscriptℙ0subscript0⋅21subscriptℙ1subscript0subscriptℙ0subscript0 |B_P (s_0 )-B_P_0 (s_% 0 ) |≤ 2· 1-α· P_1% (s_0 )P_0 (s_0 )| Bblackboard_P ( s0 ) - Bblackboard_P start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( s0 ) | ≤ 2 ⋅ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG (29) Proof. To begin, we explicitly write the expectations difference: |Bℙ⁢(s0)−Bℙ0⁢(s0)|subscriptℙsubscript0subscriptsubscriptℙ0subscript0 |B_P(s_0)-B_P_0(s_0) || Bblackboard_P ( s0 ) - Bblackboard_P start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( s0 ) | =|∑sB(s)⋅[ℙ(s|s0)−ℙ0(s|s0)]| = | _sB (s )· [P (s\,|\,s_% 0 )-P_0 (s\,|\,s_0 ) ] |= | ∑s B ( s ) ⋅ [ blackboard_P ( s | s0 ) - blackboard_P0 ( s | s0 ) ] | (30) Which by the triangular inequality is at most: ≤∑s|B(s)|⋅|ℙ(s|s0)−ℙ0(s|s0)| ≤ _s |B (s ) |· |P% (s\,|\,s_0 )-P_0 (s\,|\,s_0 ) |≤ ∑s | B ( s ) | ⋅ | blackboard_P ( s | s0 ) - blackboard_P0 ( s | s0 ) | (31) Now, since the range of B is [−1,1]11 [-1,1 ][ - 1 , 1 ] we can get rid of the |B⁢(s)| |B (s ) || B ( s ) | terms, and get that |Bℙ⁢(s0)−Bℙ0⁢(s0)|subscriptℙsubscript0subscriptsubscriptℙ0subscript0 |B_P(s_0)-B_P_0(s_0) || Bblackboard_P ( s0 ) - Bblackboard_P start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( s0 ) | is at most: ∑s|ℙ(s|s0)−ℙ0(s|s0)|=∑sℙ0(s|s0)⋅|ℙ⁢(s|s0)ℙ0⁢(s|s0)−1| _s |P (s\,|\,s_0 )-P_0% (s\,|\,s_0 ) |= _sP_0 (s\,|\,s_0 )% · | P (s\,|\,s_0 )P_0 (s\,|% \,s_0 )-1 |∑s | blackboard_P ( s | s0 ) - blackboard_P0 ( s | s0 ) | = ∑s blackboard_P0 ( s | s0 ) ⋅ | divide start_ARG blackboard_P ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG - 1 | (32) Importantly, by lemma 1 we have that: |ℙ⁢(s|s0)ℙ0⁢(s|s0)−1|≤1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)⋅max⁡ℙ1⁢(s|s0)ℙ0⁢(s|s0),1ℙconditionalsubscript0subscriptℙ0conditionalsubscript01⋅1subscriptℙ1subscript0subscriptℙ0subscript0subscriptℙ1conditionalsubscript0subscriptℙ0conditionalsubscript01 | P (s\,|\,s_0 )P_0% (s\,|\,s_0 )-1 |≤ 1-α· % P_1 (s_0 )P_0 (s_0 )· % \ P_1 (s\,|\,s_0 )P_0 (s\,|% \,s_0 ),1 \| divide start_ARG blackboard_P ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG - 1 | ≤ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG ⋅ max divide start_ARG blackboard_P1 ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG , 1 (33) For any s, hence we got that |Bℙ⁢(s0)−Bℙ0⁢(s0)|subscriptℙsubscript0subscriptsubscriptℙ0subscript0 |B_P(s_0)-B_P_0(s_0) || Bblackboard_P ( s0 ) - Bblackboard_P start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( s0 ) | is at most: 1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)⋅[∑sℙ0⁢(s|s0)⋅max⁡ℙ1⁢(s|s0)ℙ0⁢(s|s0),1]⋅1subscriptℙ1subscript0subscriptℙ0subscript0delimited-[]subscript⋅subscriptℙ0conditionalsubscript0subscriptℙ1conditionalsubscript0subscriptℙ0conditionalsubscript01 1-α· P_1 (s_0 % )P_0 (s_0 )· [ _sP_0 (s\,% |\,s_0 )· \ P_1 (s\,|\,s_0 )% P_0 (s\,|\,s_0 ),1 \ ]divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG ⋅ [ ∑s blackboard_P0 ( s | s0 ) ⋅ max divide start_ARG blackboard_P1 ( s | s0 ) end_ARG start_ARG blackboard_P0 ( s | s0 ) end_ARG , 1 ] (34) ≤1−α⋅ℙ1⁢(s0)ℙ0⁢(s0)⋅∑s(ℙ1⁢(s|s0)+ℙ0⁢(s|s0))absent⋅1subscriptℙ1subscript0subscriptℙ0subscript0subscriptsubscriptℙ1conditionalsubscript0subscriptℙ0conditionalsubscript0 ≤ 1-α· P_1 (s_0% )P_0 (s_0 )· _s (P_1% (s\,|\,s_0 )+P_0 (s\,|\,s_0 ) )≤ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ divide start_ARG blackboard_P1 ( s0 ) end_ARG start_ARG blackboard_P0 ( s0 ) end_ARG ⋅ ∑s ( blackboard_P1 ( s | s0 ) + blackboard_P0 ( s | s0 ) ) (35) where the last inequality follows from the fact that sum of two non-negative terms is greater than the maximum of the terms. Finally, since both ℙ0⁢(s|s0)subscriptℙ0conditionalsubscript0P_0 (s\,|\,s_0 )blackboard_P0 ( s | s0 ) and ℙ1⁢(s|s0)subscriptℙ1conditionalsubscript0P_1 (s\,|\,s_0 )blackboard_P1 ( s | s0 ) are probability distributions, summing over all possible sentences s yields 2222, and hence the inequality in equation 29 follows. ∎ C.3 Adversarial prompt construction In this subsection, we prove a technical lemma which shows that when two distribution are sufficiently distinguishable (see definition 2 from the main text) ,then there exists a prompt such that the ratio of the prompt’s likelihood according to these two distribution is arbitrary low. In the next sections we will use this lemma to prove the existence adversarial prompt for which the conditions of lemma 1 holds. And hence an adversarial user might alter the model behavior (lemma 2). Lemma 3. Let β,ϵ>0italic-ϵ0β,ε>0β , ϵ > 0 and ℙ0,ℙ1subscriptℙ0subscriptℙ1P_0,P_1blackboard_P0 , blackboard_P1 two distributions. Suppose ℙ0subscriptℙ0P_0blackboard_P0 is β-distinguishable from ℙ1subscriptℙ1P_1blackboard_P1 then there exists a prompt s of length 1β⁢log⁡1ϵ11italic-ϵ 1β 1εdivide start_ARG 1 end_ARG start_ARG β end_ARG log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG such that the following holds: ℙ1⁢(s)ℙ0⁢(s)≤ϵsubscriptℙ1subscriptℙ0italic-ϵ P_1(s)P_0(s)≤ start_ARG blackboard_P1 ( s ) end_ARG start_ARG blackboard_P0 ( s ) end_ARG ≤ ϵ (36) Proof. Intuitively we use the fact that ℙ0subscriptℙ0P_0blackboard_P0 is β-distinguishable from ℙ1subscriptℙ1P_1blackboard_P1 to construct a prompt sentence by sentence, and get a prompt q=s1⊕…⊕s|q|direct-sumsubscript1…subscriptq=s_1 ... s_|q|q = s1 ⊕ … ⊕ s| q | such that: log⁡ℙ0⁢(s1⊕…⊕sk|s0)ℙ1⁢(s1⊕…⊕sk|s0)>β⋅ksubscriptℙ0direct-sumsubscript1…conditionalsubscriptsubscript0subscriptℙ1direct-sumsubscript1…conditionalsubscriptsubscript0⋅ P_0 (s_1 ... s_k\,|\,s_0% )P_1 (s_1 ... s_k\,|\,s_0 )>% β· klog divide start_ARG blackboard_P0 ( s1 ⊕ … ⊕ sitalic_k | s0 ) end_ARG start_ARG blackboard_P1 ( s1 ⊕ … ⊕ sitalic_k | s0 ) end_ARG > β ⋅ k (37) For any k≤|q|k≤ |q |k ≤ | q |. Let us look at the expectation value of the log ratio with respect to a sequence s=(s1⁢…⁢sk)subscript1…subscripts=(s_1...s_k)s = ( s1 … sitalic_k ) of k sentences sampled from ℙ−⁢(⋅)subscriptℙ⋅P_-(·)blackboard_P- ( ⋅ ): s∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)]=(s1⊕…⊕sk)∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s1⊕…⊕sk)ℙ+⁢(s1⊕…⊕sk)]=subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙsubscriptsimilar-todirect-sumsubscript1…subscriptsubscriptℙ⋅delimited-[]subscriptℙdirect-sumsubscript1…subscriptsubscriptℙdirect-sumsubscript1…subscriptabsentE_s _-(·) [ P_-(s)% P_+(s) ]=E_(s_1 ... s_k) % P_-(·) [ P_-(s_1 ... s_k)% P_+(s_1 ... s_k) ]=blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] = blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_k ) ∼ blackboard_P- ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s1 ⊕ … ⊕ sitalic_k ) end_ARG start_ARG blackboard_P+ ( s1 ⊕ … ⊕ sitalic_k ) end_ARG ] = (38) Using the law of conditional probabilities recursively and the linearity of the expectation value: =∑i=1k(s1⊕…⊕sk)∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(si|s1⊕…⊕si−1)ℙ+⁢(si|s1⊕…⊕si−1)]=absentsuperscriptsubscript1subscriptsimilar-todirect-sumsubscript1…subscriptsubscriptℙ⋅delimited-[]subscriptℙconditionalsubscriptdirect-sumsubscript1…subscript1subscriptℙconditionalsubscriptdirect-sumsubscript1…subscript1absent= _i=1^kE_(s_1 ... s_k) _-(% ·) [ P_-(s_i|s_1 ... s_i-1)% P_+(s_i|s_1 ... s_i-1) ]== ∑i = 1k blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_k ) ∼ blackboard_P- ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( sitalic_i | s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG start_ARG blackboard_P+ ( sitalic_i | s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG ] = (39) =∑i=1k(s1⊕…⊕si)∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(si|s1⊕…⊕si−1)ℙ+⁢(si|s1⊕…⊕si−1)]=absentsuperscriptsubscript1subscriptsimilar-todirect-sumsubscript1…subscriptsubscriptℙ⋅delimited-[]subscriptℙconditionalsubscriptdirect-sumsubscript1…subscript1subscriptℙconditionalsubscriptdirect-sumsubscript1…subscript1absent= _i=1^kE_(s_1 ... s_i) _-(% ·) [ P_-(s_i|s_1 ... s_i-1)% P_+(s_i|s_1 ... s_i-1) ]== ∑i = 1k blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_i ) ∼ blackboard_P- ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( sitalic_i | s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG start_ARG blackboard_P+ ( sitalic_i | s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG ] = (40) The expectation value with respect to sisubscripts_isitalic_i is the conditional KL divergence: =∑i=1k(s1⊕…⊕si−1)∼ℙ−⁢(⋅)[DK⁢L(ℙ−(⋅|s1⊕…⊕si−1)||ℙ+(⋅|s1⊕…⊕si−1))]= _i=1^kE_(s_1 ... s_i-1) _-(% ·) [D_KL (P_- (·|s_1 ... s_i-1% )||P_+ (·|s_1 ... s_i-1 ) ) ]= ∑i = 1k blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_i - 1 ) ∼ blackboard_P- ( ⋅ ) end_POSTSUBSCRIPT [ Ditalic_K L ( blackboard_P- ( ⋅ | s1 ⊕ … ⊕ sitalic_i - 1 ) | | blackboard_P+ ( ⋅ | s1 ⊕ … ⊕ sitalic_i - 1 ) ) ] (41) From β distinguishability: >k⋅βabsent⋅>k·β> k ⋅ β (42) Hence we obtain: s∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)]>β⁢|s|subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙE_s _-(·) [ P_-(s)% P_+(s) ]>β|s|blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] > β | s | (43) In particular, there exists a specific sequence s such that the inequality holds. We take that to be the prompt q. Now, we can choose |q|>log⁡1ϵβ1italic-ϵ |q |> 1εβ| q | > divide start_ARG log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG end_ARG start_ARG β end_ARG to obtain the desired result that ℙ1⁢(s)ℙ0⁢(s)≤ϵsubscriptℙ1subscriptℙ0italic-ϵ P_1(s)P_0(s)≤ start_ARG blackboard_P1 ( s ) end_ARG start_ARG blackboard_P0 ( s ) end_ARG ≤ ϵ (44) As desired. ∎ Lemma 4. Let β,σ,ϵ,δ>0italic-ϵ0β,σ,ε,δ>0β , σ , ϵ , δ > 0 and ℙ0,ℙ1subscriptℙ0subscriptℙ1P_0,P_1blackboard_P0 , blackboard_P1 two distributions, s0subscript0s_0s0 a prefix sampled from ℙ1subscriptℙ1P_1blackboard_P1 . Suppose ℙ0subscriptℙ0P_0blackboard_P0 is β-prompt-distinguishable from ℙ1subscriptℙ1P_1blackboard_P1, and ℙ1subscriptℙ1P_1blackboard_P1 is β′β β′-undistinguishable, σ-similar and positive w.r.t. ℙ0subscriptℙ0P_0blackboard_P0, then with probability 1−δ11- 1 - δ, there exists a prompt s of length 1β⁢log⁡1ϵ+β′β⁢|s0|+σβ⁢|s0|δ+111italic-ϵsuperscript′subscript0subscript01 1β 1ε+ β β|s_0|+% σβ |s_0|δ+1divide start_ARG 1 end_ARG start_ARG β end_ARG log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + divide start_ARG β′ end_ARG start_ARG β end_ARG | s0 | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG | s0 | end_ARG start_ARG δ end_ARG end_ARG + 1 such that the following holds: ℙ1⁢(s0⊕s)ℙ0⁢(s0⊕s)≤ϵsubscriptℙ1direct-sumsubscript0subscriptℙ0direct-sumsubscript0italic-ϵ P_1(s_0 s)P_0(s_0 s% )≤ start_ARG blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG blackboard_P0 ( s0 ⊕ s ) end_ARG ≤ ϵ (45) Proof. Intuitively, given s0subscript0s_0s0, we use the fact that ℙ0subscriptℙ0P_0blackboard_P0 is β-prompt-distinguishable from ℙ1subscriptℙ1P_1blackboard_P1 to construct a prompt sentence by sentence, and get a prompt q=s1⊕…⊕s|q|direct-sumsubscript1…subscriptq=s_1 ... s_|q|q = s1 ⊕ … ⊕ s| q | such that: log⁡ℙ0⁢(s1⊕…⊕sk|s0)ℙ1⁢(s1⊕…⊕sk|s0)>β⋅ksubscriptℙ0direct-sumsubscript1…conditionalsubscriptsubscript0subscriptℙ1direct-sumsubscript1…conditionalsubscriptsubscript0⋅ P_0 (s_1 ... s_k\,|\,s_0% )P_1 (s_1 ... s_k\,|\,s_0 )>% β· klog divide start_ARG blackboard_P0 ( s1 ⊕ … ⊕ sitalic_k | s0 ) end_ARG start_ARG blackboard_P1 ( s1 ⊕ … ⊕ sitalic_k | s0 ) end_ARG > β ⋅ k (46) For any k≤|q|k≤ |q |k ≤ | q |. To induce the β-prompt-distinguishability, we start by adding a sentence s′ of negative behavior to the prefix s0subscript0s_0s0. Let us look at the expectation value of the log ratio with respect to a sequence s=(s1⁢…⁢sk)subscript1…subscripts=(s_1...s_k)s = ( s1 … sitalic_k ) of k sentences sampled from ℙ−(⋅|s0⊕s′)P_-(·|s_0 s )blackboard_P- ( ⋅ | s0 ⊕ s′ ): s∼ℙ−(⋅|s0⊕s′)⁢[log⁡ℙ−⁢(s|s0⊕s′)ℙ+⁢(s|s0⊕s′)]=(s1⊕…⊕sk)∼ℙ−(⋅|s0⊕s′)⁢[log⁡ℙ−⁢(s1⊕…⊕sk|s0⊕s′)ℙ+⁢(s1⊕…⊕sk|s0⊕s′)]=E_s _-(·|s_0 s ) [ % P_-(s|s_0 s )P_+(s|s_0 s% ) ]=E_(s_1 ... s_k) _-% (·|s_0 s ) [ P_-(s_1 ...% s_k|s_0 s )P_+(s_1 ... s_k% |s_0 s ) ]=blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ | s0 ⊕ s′ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s | s0 ⊕ s′ ) end_ARG start_ARG blackboard_P+ ( s | s0 ⊕ s′ ) end_ARG ] = blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_k ) ∼ blackboard_P- ( ⋅ | s0 ⊕ s′ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s1 ⊕ … ⊕ sitalic_k | s0 ⊕ s′ ) end_ARG start_ARG blackboard_P+ ( s1 ⊕ … ⊕ sitalic_k | s0 ⊕ s′ ) end_ARG ] = (47) Using the law of conditional probabilities recursively and the linearity of the expectation value: =∑i=1k(s1⊕…⊕sk)∼ℙ−(⋅|s0⊕s′)⁢[log⁡ℙ−⁢(si|s0⊕s′⊕s1⊕…⊕si−1)ℙ+⁢(si|s0⊕s′⊕s1⊕…⊕si−1)]== _i=1^kE_(s_1 ... s_k) _-(% ·|s_0 s ) [ P_-(s_i|s_0% s s_1 ... s_i-1)P_+(s_i|s_% 0 s s_1 ... s_i-1) ]== ∑i = 1k blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_k ) ∼ blackboard_P- ( ⋅ | s0 ⊕ s′ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( sitalic_i | s0 ⊕ s′ ⊕ s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG start_ARG blackboard_P+ ( sitalic_i | s0 ⊕ s′ ⊕ s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG ] = (48) =∑i=1k(s1⊕…⊕si)∼ℙ−(⋅|s0⊕s′)⁢[log⁡ℙ−⁢(si|s0⊕s′⊕s1⊕…⊕si−1)ℙ+⁢(si|s0⊕s′⊕s1⊕…⊕si−1)]== _i=1^kE_(s_1 ... s_i) _-(% ·|s_0 s ) [ P_-(s_i|s_0% s s_1 ... s_i-1)P_+(s_i|s_% 0 s s_1 ... s_i-1) ]== ∑i = 1k blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_i ) ∼ blackboard_P- ( ⋅ | s0 ⊕ s′ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( sitalic_i | s0 ⊕ s′ ⊕ s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG start_ARG blackboard_P+ ( sitalic_i | s0 ⊕ s′ ⊕ s1 ⊕ … ⊕ sitalic_i - 1 ) end_ARG ] = (49) The expectation value with respect to sisubscripts_isitalic_i is the conditional KL divergence: =∑i=1k(s1⊕…⊕si−1)∼ℙ−(⋅|s0⊕s′)[DK⁢L(ℙ−(⋅|s0⊕s′⊕s1⊕…⊕si−1)||ℙ+(⋅|s0⊕s′⊕s1⊕…⊕si−1))]= _i=1^kE_(s_1 ... s_i-1) _-(% ·|s_0 s ) [D_KL (P_- (·|s_0% s s_1 ... s_i-1 )||P_+% (·|s_0 s s_1 ... s_i-1 )% ) ]= ∑i = 1k blackboard_E( s start_POSTSUBSCRIPT 1 ⊕ … ⊕ sitalic_i - 1 ) ∼ blackboard_P- ( ⋅ | s0 ⊕ s′ ) end_POSTSUBSCRIPT [ Ditalic_K L ( blackboard_P- ( ⋅ | s0 ⊕ s′ ⊕ s1 ⊕ … ⊕ sitalic_i - 1 ) | | blackboard_P+ ( ⋅ | s0 ⊕ s′ ⊕ s1 ⊕ … ⊕ sitalic_i - 1 ) ) ] (50) From β-prompt-distinguishability: >k⋅βabsent⋅>k·β> k ⋅ β (51) Hence we obtain: s∼ℙ−(⋅|s0⊕s′)⁢[log⁡ℙ−⁢(s|s0⊕s′)ℙ+⁢(s|s0⊕s′)]>β⁢|s|E_s _-(·|s_0 s ) [ % P_-(s|s_0 s )P_+(s|s_0 s% ) ]>β|s|blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ | s0 ⊕ s′ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s | s0 ⊕ s′ ) end_ARG start_ARG blackboard_P+ ( s | s0 ⊕ s′ ) end_ARG ] > β | s | (52) In particular, there exists a specific sequence s such that the inequality holds. We take that to be the prompt q. Next, observe that: ℙ⁢[log⁡ℙ+⁢(s0)ℙ−⁢(s0)>c⁢|s0|]=ℙ⁢[log⁡ℙ+⁢(s0)ℙ−⁢(s0)−β′⁢|s0|>(c−β′)⁢|s0|]ℙdelimited-[]subscriptℙsubscript0subscriptℙsubscript0subscript0ℙdelimited-[]subscriptℙsubscript0subscriptℙsubscript0superscript′subscript0superscript′subscript0P [ P_+(s_0)P_-(s_0)>c|s_% 0| ]=P [ P_+(s_0)P_-% (s_0)-β |s_0|>(c-β )|s_0| ]blackboard_P [ log divide start_ARG blackboard_P+ ( s0 ) end_ARG start_ARG blackboard_P- ( s0 ) end_ARG > c | s0 | ] = blackboard_P [ log divide start_ARG blackboard_P+ ( s0 ) end_ARG start_ARG blackboard_P- ( s0 ) end_ARG - β′ | s0 | > ( c - β′ ) | s0 | ] (53) From β′β β′-undistinguishability we obtain equation 52 with opposite inequality sign and for reversing the roles between ℙ0subscriptℙ0P_0blackboard_P0 and ℙ1subscriptℙ1P_1blackboard_P1. This gives: <ℙ⁢[log⁡ℙ+⁢(s0)ℙ−⁢(s0)−s∼ℙ+⁢(⋅)⁢[log⁡ℙ+⁢(s)ℙ−⁢(s)]>(c−β′)⁢|s0|]<absentℙdelimited-[]subscriptℙsubscript0subscriptℙsubscript0subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙsuperscript′subscript0absent<P [ P_+(s_0)P_-(s_0)-% E_s _+(·)[ P_+(s)P% _-(s)]>(c-β )|s_0| ]<< blackboard_P [ log divide start_ARG blackboard_P+ ( s0 ) end_ARG start_ARG blackboard_P- ( s0 ) end_ARG - blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT + ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG ] > ( c - β′ ) | s0 | ] < (54) From Cantelli’s inequality: <V⁢a⁢rs∼ℙ+⁢(⋅)⁢[log⁡ℙ+⁢(s)ℙ−⁢(s)]V⁢a⁢rs∼ℙ+⁢(⋅)⁢[log⁡ℙ+⁢(s)ℙ−⁢(s)]+(c−β′)2⁢|s0|2<V⁢a⁢rs∼ℙ+⁢(⋅)⁢[log⁡ℙ+⁢(s)ℙ−⁢(s)](c−β′)2⁢|s0|2<σ2⁢|s0|(c−β′)2⁢|s0|2absentsubscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙsubscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙsuperscriptsuperscript′2superscriptsubscript02subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙsuperscriptsuperscript′2superscriptsubscript02superscript2subscript0superscriptsuperscript′2superscriptsubscript02< Var_s _+(·)[ P_+(s)P% _-(s)]Var_s _+(·)[ P_+(s)% P_-(s)]+(c-β )^2|s_0|^2< Var_s % P_+(·)[ P_+(s)P_-(s)](c-β^% )^2|s_0|^2< σ^2|s_0|(c-β )^2|s_0% |^2< divide start_ARG V a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT + ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG ] end_ARG start_ARG V a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT + ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG ] + ( c - β′ )2 | s0 |2 end_ARG < divide start_ARG V a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT + ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG ] end_ARG start_ARG ( c - β′ )2 | s0 |2 end_ARG < divide start_ARG σ2 | s0 | end_ARG start_ARG ( c - β′ )2 | s0 |2 end_ARG (55) The last transition is from σ-similarity. Demand that this is smaller than δ and obtain the condition on c: c≥β′+σ|s0|⁢δsuperscript′subscript0c≥β + σ |s_0|δc ≥ β′ + divide start_ARG σ end_ARG start_ARG square-root start_ARG | s0 | δ end_ARG end_ARG (56) Thus if c=β′+σ|s0|⁢δsuperscript′subscript0c=β + σ |s_0|δc = β′ + divide start_ARG σ end_ARG start_ARG square-root start_ARG | s0 | δ end_ARG end_ARG, we obtain: ℙ⁢[log⁡ℙ+⁢(s0)ℙ−⁢(s0)>(β′+σ|s0|⁢δ)⁢|s0|]<δℙdelimited-[]subscriptℙsubscript0subscriptℙsubscript0superscript′subscript0subscript0P [ P_+(s_0)P_-(s_0)>(% β + σ |s_0|δ)|s_0| ]< _P [ log divide start_ARG blackboard_P+ ( s0 ) end_ARG start_ARG blackboard_P- ( s0 ) end_ARG > ( β′ + divide start_ARG σ end_ARG start_ARG square-root start_ARG | s0 | δ end_ARG end_ARG ) | s0 | ] < δ (57) Hence with probability 1−δ11- 1 - δ: log⁡ℙ−⁢(s0⊕s′⊕s)ℙ+⁢(s0⊕s′⊕s)=log⁡ℙ−⁢(s|s0⊕s′)ℙ+⁢(s|s0⊕s′)+log⁡ℙ−⁢(s′|s0)ℙ+⁢(s′|s0)+log⁡ℙ−⁢(s0)ℙ+⁢(s0)>|s|⁢β−|s0|⁢β′−σ⁢|s0|δsubscriptℙdirect-sumsubscript0superscript′subscriptℙdirect-sumsubscript0superscript′subscriptℙconditionaldirect-sumsubscript0superscript′subscriptℙconditionaldirect-sumsubscript0superscript′subscriptℙconditionalsuperscript′subscript0subscriptℙconditionalsuperscript′subscript0subscriptℙsubscript0subscriptℙsubscript0subscript0superscript′subscript0 P_-(s_0 s s)P_+(s_0% s s)= P_-(s|s_0 s )% P_+(s|s_0 s )+ P_-(s % |s_0)P_+(s |s_0)+ P_-(s_0)% P_+(s_0)>|s|β-|s_0|β -σ |s_0% |δlog divide start_ARG blackboard_P- ( s0 ⊕ s′ ⊕ s ) end_ARG start_ARG blackboard_P+ ( s0 ⊕ s′ ⊕ s ) end_ARG = log divide start_ARG blackboard_P- ( s | s0 ⊕ s′ ) end_ARG start_ARG blackboard_P+ ( s | s0 ⊕ s′ ) end_ARG + log divide start_ARG blackboard_P- ( s′ | s0 ) end_ARG start_ARG blackboard_P+ ( s′ | s0 ) end_ARG + log divide start_ARG blackboard_P- ( s0 ) end_ARG start_ARG blackboard_P+ ( s0 ) end_ARG > | s | β - | s0 | β′ - σ square-root start_ARG divide start_ARG | s0 | end_ARG start_ARG δ end_ARG end_ARG (58) Where we used the positivity of ℙ+subscriptℙP_+blackboard_P+ in the inequality log⁡ℙ−⁢(s′|s0)ℙ+⁢(s′|s0)>0subscriptℙconditionalsuperscript′subscript0subscriptℙconditionalsuperscript′subscript00 P_-(s |s_0)P_+(s |s_0)>0log divide start_ARG blackboard_P- ( s′ | s0 ) end_ARG start_ARG blackboard_P+ ( s′ | s0 ) end_ARG > 0 as s′ is a negative sentence. Thus we can choose |s|>β′⋅|s0|+σ⁢|s0|δ+log⁡1ϵβ⋅superscript′subscript0subscript01italic-ϵ |s |> β · |s_0 |+σ |% s_0|δ+ 1εβ| s | > divide start_ARG β′ ⋅ | s0 | + σ square-root start_ARG divide start_ARG | s0 | end_ARG start_ARG δ end_ARG end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG end_ARG start_ARG β end_ARG to obtain that ℙ1⁢(s0⊕s)ℙ0⁢(s0⊕s)≤ϵsubscriptℙ1direct-sumsubscript0subscriptℙ0direct-sumsubscript0italic-ϵ P_1(s_0 s)P_0(s_0 s)≤ start_ARG blackboard_P1 ( s0 ⊕ s ) end_ARG start_ARG blackboard_P0 ( s0 ⊕ s ) end_ARG ≤ ϵ as desired. The total length of the prompt is |s′⊕s|=1+|s|direct-sumsuperscript′1|s s|=1+|s|| s′ ⊕ s | = 1 + | s |. ∎ Appendix D Proof of theorem 1 Let ℙ+subscriptℙP_+blackboard_P+ and ℙ−subscriptℙP_-blackboard_P- be the well-behaved and ill-behaved mixture components from the α,β,γα,β,γα , β , γ-negative-distinguishability definition and ϵ>0italic-ϵ0ε>0ϵ > 0. Then, since ℙ−subscriptℙP_-blackboard_P- is β-distinguishable from ℙ+subscriptℙP_+blackboard_P+, lemma 3 assures us that for any ϵ′>0superscriptitalic-ϵ′0ε >0ϵ′ > 0 there exists a sequence s of 1β⋅(log⁡1ϵ′)⋅11superscriptitalic-ϵ′ 1β· ( 1ε )divide start_ARG 1 end_ARG start_ARG β end_ARG ⋅ ( log divide start_ARG 1 end_ARG start_ARG ϵ′ end_ARG ) sentences such that ℙ+⁢(s)ℙ−⁢(s)<ϵ′subscriptℙsubscriptℙsuperscriptitalic-ϵ′ P_+ (s )P_- (s )<ε divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG < ϵ′. Now, lemma 2 assures us that for such prompt a s, the behaviour of ℙPblackboard_P will converge to the ill-behaved component in the following sense: |Bℙ⁢(s)−Bℙ−⁢(s)|≤2⋅1−α⋅ϵ′subscriptℙsubscriptsubscriptℙ⋅21superscriptitalic-ϵ′ |B_P (s )-B_P_- (s% ) |≤ 2· 1-α·ε^ | Bblackboard_P ( s ) - Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s ) | ≤ 2 ⋅ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ ϵstart_FLOATSUPERSCRIPT ′ end_FLOATSUPERSCRIPT (59) Thus, we can choose ϵ′<α⋅ϵ4superscriptitalic-ϵ′⋅italic-ϵ4ε < α·ε4ϵ′ < divide start_ARG α ⋅ ϵ end_ARG start_ARG 4 end_ARG and get that: |Bℙ⁢(s)−Bℙ−⁢(s)|<ϵsubscriptℙsubscriptsubscriptℙitalic-ϵ |B_P (s )-B_P_- (s% ) |<ε| Bblackboard_P ( s ) - Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s ) | < ϵ (60) Finally, by definition we have that Bℙ−⁢(s)≤γsubscriptsubscriptℙB_P_- (s )≤ _P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s ) ≤ γ and hence we get that Bℙ⁢(s)<γ+ϵsubscriptℙitalic-ϵ B_P (s )<γ+ _P ( s ) < γ + ϵ (61) As desired. Appendix E Proof of theorem 2 Let ℙ+subscriptℙP_+blackboard_P+ and ℙ−subscriptℙP_-blackboard_P- be the well-behaved and ill-behaved mixture components from the α,β,γα,β,γα , β , γ-negative-prompt-distinguishability definition, and let s0subscript0s_0s0 be an initial prompt and ϵ,δ>0italic-ϵ0ε,δ>0ϵ , δ > 0. Then since ℙ−subscriptℙP_-blackboard_P- is β-prompt-distinguishable from ℙ+subscriptℙP_+blackboard_P+, ℙ+subscriptℙP_+blackboard_P+ is β-distinguishable, σ-similar and positive w.r.t ℙ−subscriptℙP_-blackboard_P-, lemma 4 assures us that for any ϵ′>0superscriptitalic-ϵ′0ε >0ϵ′ > 0 there exists with probability 1−δ11- 1 - δ a sequence s1subscript1s_1s1 of 1β⋅(log⁡1ϵ′+σ⁢|s0|δ)+|s0|+1⋅11superscriptitalic-ϵ′subscript0subscript01 1β· ( 1ε +σ |s% _0|δ )+|s_0|+1divide start_ARG 1 end_ARG start_ARG β end_ARG ⋅ ( log divide start_ARG 1 end_ARG start_ARG ϵ′ end_ARG + σ square-root start_ARG divide start_ARG | s0 | end_ARG start_ARG δ end_ARG end_ARG ) + | s0 | + 1 sentences such that ℙ+⁢(s0⊕s1)ℙ−⁢(s0⊕s1)<ϵ′subscriptℙdirect-sumsubscript0subscript1subscriptℙdirect-sumsubscript0subscript1superscriptitalic-ϵ′ P_+ (s_0 s_1 )P_- (s_0% s_1 )<ε divide start_ARG blackboard_P+ ( s0 ⊕ s1 ) end_ARG start_ARG blackboard_P- ( s0 ⊕ s1 ) end_ARG < ϵ′. Now, lemma 2 assures us that for such prompt a s0⊕s1direct-sumsubscript0subscript1s_0 s_1s0 ⊕ s1, the behaviour of ℙPblackboard_P will converge to the ill-behaved component in the following sense: |Bℙ⁢(s0⊕s1)−Bℙ−⁢(s0⊕s1)|≤2⋅1−α⋅ϵ′subscriptℙdirect-sumsubscript0subscript1subscriptsubscriptℙdirect-sumsubscript0subscript1⋅21superscriptitalic-ϵ′ |B_P (s_0 s_1 )-B_P_% - (s_0 s_1 ) |≤ 2· 1-α% ·ε^ | Bblackboard_P ( s0 ⊕ s1 ) - Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s0 ⊕ s1 ) | ≤ 2 ⋅ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ ϵstart_FLOATSUPERSCRIPT ′ end_FLOATSUPERSCRIPT (62) Thus, we can choose ϵ′<α⋅ϵ4superscriptitalic-ϵ′⋅italic-ϵ4ε < α·ε4ϵ′ < divide start_ARG α ⋅ ϵ end_ARG start_ARG 4 end_ARG and get that: |Bℙ⁢(s0⊕s1)−Bℙ−⁢(s0⊕s1)|<ϵsubscriptℙdirect-sumsubscript0subscript1subscriptsubscriptℙdirect-sumsubscript0subscript1italic-ϵ |B_P (s_0 s_1 )-B_P_% - (s_0 s_1 ) |<ε| Bblackboard_P ( s0 ⊕ s1 ) - Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s0 ⊕ s1 ) | < ϵ (63) Finally, by definition we have that Bℙ−⁢(s0⊕s1)≤γsubscriptsubscriptℙdirect-sumsubscript0subscript1B_P_- (s_0 s_1 )≤ _P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s0 ⊕ s1 ) ≤ γ and hence we get that Bℙ⁢(s0⊕s1)<γ+ϵsubscriptℙdirect-sumsubscript0subscript1italic-ϵ B_P (s_0 s_1 )<γ+ _P ( s0 ⊕ s1 ) < γ + ϵ (64) With probability 1−δ11- 1 - δ, as desired. Appendix F Proof of theorem 3 Let ℙ+subscriptℙP_+blackboard_P+ and ℙ−subscriptℙP_-blackboard_P- be the well-behaved and ill-behaved mixture components from the α,β,γα,β,γα , β , γ-negative-prompt-distinguishability definition. Essentially, we we show that there exists a choice of prompts q1⁢…⁢qn+1subscript1…subscript1q_1...q_n+1q1 … qitalic_n + 1 each of them consists of at most β′β⁢|ai|+σβ⁢|ai|δ+log⁡1α+log⁡1ϵ+log⁡4n⁢β+1superscript′subscriptsubscript11italic-ϵ41 β β|a_i|+ σβ |a_i|% δ+ 1α+ 1ε+ 4nβ+1divide start_ARG β′ end_ARG start_ARG β end_ARG | aitalic_i | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG + divide start_ARG log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 end_ARG start_ARG n β end_ARG + 1 sentences such that: log⁡ℙ+⁢(q1⊕a1⊕…⊕qn⊕an⊕qn+1)ℙ−⁢(q1⊕a1⊕…⊕qn⊕an⊕qn+1)<∑i=1n+1(β′⁢|ai|+σ⁢|ai|δ−β⁢|qi|)subscriptℙdirect-sumsubscript1subscript1…subscriptsubscriptsubscript1subscriptℙdirect-sumsubscript1subscript1…subscriptsubscriptsubscript1superscriptsubscript11superscript′subscriptsubscriptsubscript P_+ (q_1 a_1 ... q_% n a_n q_n+1 )P_- (q_1 a_1% ... q_n a_n q_n+1 )< _i=1^n+1 (% β |a_i|+σ |a_i|δ-β |q_i% | )log divide start_ARG blackboard_P+ ( q1 ⊕ a1 ⊕ … ⊕ qitalic_n ⊕ aitalic_n ⊕ qitalic_n + 1 ) end_ARG start_ARG blackboard_P- ( q1 ⊕ a1 ⊕ … ⊕ qitalic_n ⊕ aitalic_n ⊕ qitalic_n + 1 ) end_ARG < ∑i = 1n + 1 ( β′ | aitalic_i | + σ square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG - β | qitalic_i | ) (65) Then, we will use lemma 2 and get that for any such prompts q1⁢…⁢qn+1subscript1…subscript1q_1...q_n+1q1 … qitalic_n + 1 the behaviour of ℙPblackboard_P will converge to the ill-behaved component in the following sense: |Bℙ⁢(s)−Bℙ−⁢(s)|≤2⋅1−α⋅exp⁡(∑i=1n+1(β′⁢|ai|+σ⁢|ai|δ−β⁢|qi|))subscriptℙsubscriptsubscriptℙ⋅21superscriptsubscript11superscript′subscriptsubscriptsubscript |B_P (s )-B_P_- (s% ) |≤ 2· 1-α· ( _i=1^n+1% (β |a_i|+σ |a_i|δ-β |q_% i | ) )| Bblackboard_P ( s ) - Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s ) | ≤ 2 ⋅ divide start_ARG 1 - α end_ARG start_ARG α end_ARG ⋅ exp ( ∑i = 1n + 1 ( β′ | aitalic_i | + σ square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG - β | qitalic_i | ) ) (66) Where s denote the concatenation of the messages in the conversation: q1⊕a1⊕…⊕qn⊕an⊕qn+1direct-sumsubscript1subscript1…subscriptsubscriptsubscript1q_1 a_1 ... q_n a_n q_n+1q1 ⊕ a1 ⊕ … ⊕ qitalic_n ⊕ aitalic_n ⊕ qitalic_n + 1. Thus, we will get that Bℙ⁢(s)<γ+ϵsubscriptℙitalic-ϵB_P (s )<γ+ _P ( s ) < γ + ϵ for ∑i=1n+1|qi|>∑i=1n+1(β′β⁢|ai|+σβ⁢|ai|δ+1)+log⁡(1−α2⋅α⋅ϵ)βsuperscriptsubscript11subscriptsuperscriptsubscript11superscript′subscriptsubscript11⋅2italic-ϵ _i=1^n+1 |q_i |> _i=1^n+1 ( β % β |a_i |+ σβ |a_i|δ+% 1 )+ ( 1-α2·α·ε )β∑i = 1n + 1 | qitalic_i | > ∑i = 1n + 1 ( divide start_ARG β′ end_ARG start_ARG β end_ARG | aitalic_i | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG + 1 ) + divide start_ARG log ( divide start_ARG 1 - α end_ARG start_ARG 2 ⋅ α ⋅ ϵ end_ARG ) end_ARG start_ARG β end_ARG as desired. Intuitively, we will prove the existence of the prompts q1⁢…⁢qn+1subscript1…subscript1q_1...q_n+1q1 … qitalic_n + 1 such that the length of any prompt is at most β′β⁢|ai|+σβ⁢|ai|δ+log⁡1α+log⁡1ϵ+log⁡4n⁢β+1superscript′subscriptsubscript11italic-ϵ41 β β |a_i |+ σβ % |a_i|δ+ 1α+ 1ε+ 4n% β+1divide start_ARG β′ end_ARG start_ARG β end_ARG | aitalic_i | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG + divide start_ARG log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 end_ARG start_ARG n β end_ARG + 1 and equation 65 upholds by using an induction argument, where the induction hypothesis follows from the fact that ℙ−subscriptℙP_-blackboard_P- is β-distinguishable from ℙ+subscriptℙP_+blackboard_P+. Formally, the base case of the induction follows by using lemma 3 to construct an adversarial prompt q1subscript1q_1q1 such that log⁡ℙ+⁢(q1)ℙ−⁢(q1)<−β⋅|q1|subscriptℙsubscript1subscriptℙsubscript1⋅subscript1 P_+ (q_1 )P_- (q_1 )<% -β·|q_1|log divide start_ARG blackboard_P+ ( q1 ) end_ARG start_ARG blackboard_P- ( q1 ) end_ARG < - β ⋅ | q1 |. Now, assume that there exists prompts q1⁢…⁢qksubscript1…subscriptq_1...q_kq1 … qitalic_k such that the length of any prompt is at most β′β⁢|ai|+σβ⁢|ai|δ+log⁡1α+log⁡1ϵ+log⁡4n⁢β+1superscript′subscriptsubscript11italic-ϵ41 β β |a_i |+ σβ % |a_i|δ+ 1α+ 1ε+ 4n% β+1divide start_ARG β′ end_ARG start_ARG β end_ARG | aitalic_i | + divide start_ARG σ end_ARG start_ARG β end_ARG square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ end_ARG end_ARG + divide start_ARG log divide start_ARG 1 end_ARG start_ARG α end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log 4 end_ARG start_ARG n β end_ARG + 1 and equation 65 upholds (with n=k−1)n=k-1)n = k - 1 ). Then the proof of lemma 4 (equation 52) assures us that there exists an adversarial prompt qk+1subscript1q_k+1qitalic_k + 1 such that: log⁡ℙ+⁢(qk+1|q1⊕a1⊕⋯⊕qk⊕ak)ℙ−⁢(qk+1|q1⊕a1⊕⋯⊕qk⊕ak)<−β⋅|qk+1|subscriptℙconditionalsubscript1direct-sumsubscript1subscript1⋯subscriptsubscriptsubscriptℙconditionalsubscript1direct-sumsubscript1subscript1⋯subscriptsubscript⋅subscript1 P_+ (q_k+1|q_1 a_1 % … q_k a_k )P_- (q_k+1|q_1 a% _1 … q_k a_k )<-β·|q_k+1|log divide start_ARG blackboard_P+ ( qitalic_k + 1 | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ) end_ARG start_ARG blackboard_P- ( qitalic_k + 1 | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ) end_ARG < - β ⋅ | qitalic_k + 1 | (67) Now, by the chain rule of conditional probabilities we have that: log⁡ℙ+⁢(q1⊕a1⊕⋯⊕qk⊕ak⊕qk+1)ℙ−⁢(q1⊕a1⊕⋯⊕qk⊕ak⊕qk+1)<−β⋅|qk+1|+log⁡ℙ+⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)ℙ−⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)subscriptℙdirect-sumsubscript1subscript1⋯subscriptsubscriptsubscript1subscriptℙdirect-sumsubscript1subscript1⋯subscriptsubscriptsubscript1⋅subscript1subscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsubscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscript P_+ (q_1 a_1 … q% _k a_k q_k+1 )P_- (q_1 a_1% … q_k a_k q_k+1 )<-β·|q_k+1|% + P_+ (a_k|q_1 a_1 … a_k-1% q_k )P_- (a_k|q_1 a_1 …% a_k-1 q_k )log divide start_ARG blackboard_P+ ( q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ⊕ qitalic_k + 1 ) end_ARG start_ARG blackboard_P- ( q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ⊕ qitalic_k + 1 ) end_ARG < - β ⋅ | qitalic_k + 1 | + log divide start_ARG blackboard_P+ ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG (68) Now, observe that: ℙ⁢[log⁡ℙ+⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)ℙ−⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)>c⁢|ak|]=ℙ⁢[log⁡ℙ+⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)ℙ−⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)−β′⁢|ak|>(c−β′)⁢|ak|]ℙdelimited-[]subscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsubscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsubscriptℙdelimited-[]subscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsubscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsuperscript′subscriptsuperscript′subscriptP [ P_+ (a_k|q_1 a_1 % … a_k-1 q_k )P_- (a_k|q_1 a% _1 … a_k-1 q_k )>c|a_k| ]=P% [ P_+ (a_k|q_1 a_1 … a% _k-1 q_k )P_- (a_k|q_1 a_1 % … a_k-1 q_k )-β |a_k|>(c-β % )|a_k| ]blackboard_P [ log divide start_ARG blackboard_P+ ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG > c | aitalic_k | ] = blackboard_P [ log divide start_ARG blackboard_P+ ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG - β′ | aitalic_k | > ( c - β′ ) | aitalic_k | ] (69) From β′β β′-prompt-undistinguishability, since qksubscriptq_kqitalic_k ends with a negative sentence, equation 52 gives: <ℙ⁢[log⁡ℙ+⁢(ak|q1⊕a1⊕⋯⊕qk)ℙ−⁢(ak|q1⊕a1⊕⋯⊕qk)−s∼ℙ+(⋅|q1⊕a1⊕⋯⊕qk)⁢[log⁡ℙ+⁢(s|q1⊕a1⊕⋯⊕qk)ℙ−⁢(s|q1⊕a1⊕⋯⊕qk)]>(c−β′)⁢|ak|]<<P [ P_+ (a_k|q_1 a_1 % … q_k )P_- (a_k|q_1 a_1 % … q_k )-E_s _+ (·|q_1% a_1 … q_k )[ P_+ (s|q_% 1 a_1 … q_k )P_- (s|q_1% a_1 … q_k )]>(c-β )|a_k| ]<< blackboard_P [ log divide start_ARG blackboard_P+ ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG - blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT + ( ⋅ | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG ] > ( c - β′ ) | aitalic_k | ] < (70) From Cantelli’s inequality: <V⁢a⁢rs∼ℙ+⁢(s|q1⊕a1⊕⋯⊕qk)⁢[log⁡ℙ+⁢(s|q1⊕a1⊕⋯⊕qk)ℙ−⁢(s|q1⊕a1⊕⋯⊕qk)]V⁢a⁢rs∼ℙ+(⋅|q1⊕a1⊕⋯⊕qk)⁢[log⁡ℙ+⁢(s|q1⊕a1⊕⋯⊕qk)ℙ−⁢(s|q1⊕a1⊕⋯⊕qk)]+(c−β′)2⁢|ak|2<< Var_s _+ (s|q_1 a_1 … q_k% )[ P_+ (s|q_1 a_1 … q% _k )P_- (s|q_1 a_1 … q_k% )]Var_s _+ (·|q_1 a_1 …% q_k )[ P_+ (s|q_1 a_1 % … q_k )P_- (s|q_1 a_1 …% q_k )]+(c-β )^2|a_k|^2<< divide start_ARG V a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT + ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG ] end_ARG start_ARG V a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT + ( ⋅ | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG ] + ( c - β′ )2 | aitalic_k |2 end_ARG < (71) <V⁢a⁢rs∼ℙ+⁢(s|q1⊕a1⊕⋯⊕qk)⁢[log⁡ℙ+⁢(s|q1⊕a1⊕⋯⊕qk)ℙ−⁢(s|q1⊕a1⊕⋯⊕qk)](c−β′)2⁢|ak|2<σ2⁢|ak|(c−β′)2⁢|ak|2absentsubscriptsimilar-tosubscriptℙconditionaldirect-sumsubscript1subscript1⋯subscriptdelimited-[]subscriptℙconditionaldirect-sumsubscript1subscript1⋯subscriptsubscriptℙconditionaldirect-sumsubscript1subscript1⋯subscriptsuperscriptsuperscript′2superscriptsubscript2superscript2subscriptsuperscriptsuperscript′2superscriptsubscript2< Var_s _+ (s|q_1 a_1 … q_k% )[ P_+ (s|q_1 a_1 … q% _k )P_- (s|q_1 a_1 … q_k% )](c-β )^2|a_k|^2< σ^2|a_k|(c-% β )^2|a_k|^2< divide start_ARG V a ritalic_s ∼ blackboard_P start_POSTSUBSCRIPT + ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P+ ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( s | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ) end_ARG ] end_ARG start_ARG ( c - β′ )2 | aitalic_k |2 end_ARG < divide start_ARG σ2 | aitalic_k | end_ARG start_ARG ( c - β′ )2 | aitalic_k |2 end_ARG (72) The last transition is from σ-similarity. Demand that this is smaller than δ′δ δ′ and obtain the condition on c: c≥β′+σ|ak|⁢δ′subscriptsuperscript′c≥β + σ |a_k|δ c ≥ β′ + divide start_ARG σ end_ARG start_ARG square-root start_ARG | aitalic_k | δ′ end_ARG end_ARG (73) Thus if c=β′+σ|ak|⁢δ′subscriptsuperscript′c=β + σ |a_k|δ c = β′ + divide start_ARG σ end_ARG start_ARG square-root start_ARG | aitalic_k | δ′ end_ARG end_ARG, we obtain: ℙ⁢[log⁡ℙ+⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)ℙ−⁢(ak|q1⊕a1⊕⋯⊕ak−1⊕qk)>(β′+σ|ak|⁢δ′)⁢|s0|]<δ′ℙdelimited-[]subscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsubscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscriptsuperscript′subscriptsuperscript′subscript0superscript′P [ P_+ (a_k|q_1 a_1 % … a_k-1 q_k )P_- (a_k|q_1 a% _1 … a_k-1 q_k )>(β + σ% |a_k|δ )|s_0| ]<δ blackboard_P [ log divide start_ARG blackboard_P+ ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG start_ARG blackboard_P- ( aitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ aitalic_k - 1 ⊕ qitalic_k ) end_ARG > ( β′ + divide start_ARG σ end_ARG start_ARG square-root start_ARG | aitalic_k | δ′ end_ARG end_ARG ) | s0 | ] < δ′ (74) Lastly, recall that for inducing the distinguishability from prompt qk+1subscript1q_k+1qitalic_k + 1, we need to add a triggering sentence before it, hence the +11+1+ 1. Hence by plugging this into equation 68, then with probability 1−δ′1superscript′1-δ 1 - δ′: log⁡ℙ+⁢(q1⊕a1⊕⋯⊕qk⊕ak⊕qk+1)ℙ−⁢(q1⊕a1⊕⋯⊕qk⊕ak⊕qk+1)<−|qk+1|⁢β+|ak|⁢β′+σ⁢|ak|δ′subscriptℙdirect-sumsubscript1subscript1⋯subscriptsubscriptsubscript1subscriptℙdirect-sumsubscript1subscript1⋯subscriptsubscriptsubscript1subscript1subscriptsuperscript′subscriptsuperscript′ P_+ (q_1 a_1 … q% _k a_k q_k+1 )P_- (q_1 a_1% … q_k a_k q_k+1 )<-|q_k+1|β+|a_% k|β +σ |a_k|δ log divide start_ARG blackboard_P+ ( q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ⊕ qitalic_k + 1 ) end_ARG start_ARG blackboard_P- ( q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ⊕ qitalic_k + 1 ) end_ARG < - | qitalic_k + 1 | β + | aitalic_k | β′ + σ square-root start_ARG divide start_ARG | aitalic_k | end_ARG start_ARG δ′ end_ARG end_ARG (75) So we can use the induction hypothesis to upper bound the log⁡ℙ+⁢(qk|q1⊕a1⊕⋯⊕qk−1⊕ak−1)ℙ−⁢(qk|q1⊕a1⊕⋯⊕qk−1⊕ak−1)subscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscript1subscriptℙconditionalsubscriptdirect-sumsubscript1subscript1⋯subscript1subscript1 P_+ (q_k|q_1 a_1 … q_k-1% a_k-1 )P_- (q_k|q_1 a_1 …% q_k-1 a_k-1 )log divide start_ARG blackboard_P+ ( qitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k - 1 ⊕ aitalic_k - 1 ) end_ARG start_ARG blackboard_P- ( qitalic_k | q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k - 1 ⊕ aitalic_k - 1 ) end_ARG term and get that: log⁡ℙ+⁢(q1⊕a1⊕⋯⊕qk⊕ak⊕qk+1)ℙ−⁢(q1⊕a1⊕⋯⊕qk⊕ak⊕qk+1)<∑i=1k+1(−|qi+1|⁢β+|ai|⁢β′+σ⁢|ai|δ′)subscriptℙdirect-sumsubscript1subscript1⋯subscriptsubscriptsubscript1subscriptℙdirect-sumsubscript1subscript1⋯subscriptsubscriptsubscript1superscriptsubscript11subscript1subscriptsuperscript′subscriptsuperscript′ P_+ (q_1 a_1 … q% _k a_k q_k+1 )P_- (q_1 a_1% … q_k a_k q_k+1 )< _i=1^k+1% (-|q_i+1|β+|a_i|β +σ |a_i|δ^% )log divide start_ARG blackboard_P+ ( q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ⊕ qitalic_k + 1 ) end_ARG start_ARG blackboard_P- ( q1 ⊕ a1 ⊕ ⋯ ⊕ qitalic_k ⊕ aitalic_k ⊕ qitalic_k + 1 ) end_ARG < ∑i = 1k + 1 ( - | qitalic_i + 1 | β + | aitalic_i | β′ + σ square-root start_ARG divide start_ARG | aitalic_i | end_ARG start_ARG δ′ end_ARG end_ARG ) (76) As desired. The total probability of the existence of the prompts q1⁢…⁢qn+1subscript1…subscript1q_1...q_n+1q1 … qitalic_n + 1 is (1−δ′)nsuperscript1superscript′(1-δ )^n( 1 - δ′ )n. Hence it suffices to choose δ′=δnsuperscript′δ = δnδ′ = divide start_ARG δ end_ARG start_ARG n end_ARG to ensure that they exist with probability 1−δ11- 1 - δ. Appendix G Proof of theorem 4 Here we will use modified versions of the proof’s building blocks. First, we observe that without best of n, the conditional model distribution to a prompt s is: ℙ⁢(s′|s)=11+1−α⁢ℙ+⁢(s)ℙ−⁢(s)⁢ℙ−⁢(s′|s)+11+α1−α⁢ℙ−⁢(s)ℙ+⁢(s)⁢ℙ+⁢(s′|s)ℙconditionalsuperscript′111subscriptℙsubscriptℙsubscriptℙconditionalsuperscript′111subscriptℙsubscriptℙsubscriptℙconditionalsuperscript′P(s |s)= 11+ 1-α P_+% (s)P_-(s)P_-(s |s)+ 11+ α% 1-α P_-(s)P_+(s)P_+(s % |s)blackboard_P ( s′ | s ) = divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG blackboard_P- ( s′ | s ) + divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG α end_ARG start_ARG 1 - α end_ARG divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG end_ARG blackboard_P+ ( s′ | s ) (77) Now, when sampling from this distribution, the probability of sampling a negative response is the probability to sample from ℙ−subscriptℙP_-blackboard_P-, which is equal to its prior: ℙ⁢(sample negative response)=11+1−α⁢ℙ+⁢(s)ℙ−⁢(s)ℙsample negative response111subscriptℙsubscriptℙP(sample negative response)= 11+ 1-α% α P_+(s)P_-(s)blackboard_P ( sample negative response ) = divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG (78) Similarly the probability for sampling a positive response: ℙ⁢(sample positive response)=11+α1−α⁢ℙ−⁢(s)ℙ+⁢(s)ℙsample positive response111subscriptℙsubscriptℙP(sample positive response)= 11+ α1-% α P_-(s)P_+(s)blackboard_P ( sample positive response ) = divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG α end_ARG start_ARG 1 - α end_ARG divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG end_ARG (79) When sampling n responses, if some of them are from ℙ+subscriptℙP_+blackboard_P+, then they will be chosen by the reward function in best of n, hence the behavior expectation is positive. Thus sampling a negative response can only happen if all n responses are sampled from ℙ−subscriptℙP_-blackboard_P-. The probability of this happening is: ℙbest of n⁢(sample negative response)=ℙ⁢(sample negative response)n=(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))nsubscriptℙbest of nsample negative responseℙsuperscriptsample negative responsesuperscript111subscriptℙsubscriptℙP_best of n(sample negative response)=P(% sample negative response)^n= ( 11+ 1-α% α P_+(s)P_-(s) )^nblackboard_Pbest of n ( sample negative response ) = blackboard_P ( sample negative response )n = ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n (80) Otherwise, a response is sampled from ℙ+subscriptℙP_+blackboard_P+. Thus the conditional model response can be rewritten for best of n as: ℙbest of n⁢(s′|s)=(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n⁢ℙ−⁢(s′|s)+(1−(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n)⁢ℙ+⁢(s′|s)subscriptℙbest of nconditionalsuperscript′111subscriptℙsubscriptℙsubscriptℙconditionalsuperscript′1superscript111subscriptℙsubscriptℙsubscriptℙconditionalsuperscript′P_best of n(s |s)= ( 11+ 1-α% α P_+(s)P_-(s) )^nP_-% (s |s)+ (1- ( 11+ 1-α % P_+(s)P_-(s) )^n )P_+(s^% |s)blackboard_Pbest of n ( s′ | s ) = ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n blackboard_P- ( s′ | s ) + ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n ) blackboard_P+ ( s′ | s ) (81) Now, we calculate the behavior expectation difference between the best of n distribution and the negative distribution: |Bℙbest of n(s)−Bℙ−(s)|=|∑s′B(s′)(ℙbest of n(s′|s)−ℙ−(s′|s))|≤|B_P_best of n(s)-B_P_-(s)|=| _s^% B(s )(P_best of n(s |s)-P% _-(s |s))|≤| Bblackboard_P start_POSTSUBSCRIPT best of n end_POSTSUBSCRIPT ( s ) - Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s ) | = | ∑s′ B ( s′ ) ( blackboard_Pbest of n ( s′ | s ) - blackboard_P- ( s′ | s ) ) | ≤ (82) Since B is bounded between [−1,+1]11[-1,+1][ - 1 , + 1 ]: ≤∑s′|(ℙbest of n(s′|s)−ℙ−(s′|s))|≤ _s |(P_best of n(s |s)-P% _-(s |s))|≤ ∑s′ | ( blackboard_Pbest of n ( s′ | s ) - blackboard_P- ( s′ | s ) ) | (83) Using our expression for the best of n distribution in terms of the components (equation 80): =∑s′|((11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))nℙ−(s′|s)+(1−(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n)ℙ+(s′|s)−ℙ−(s′|s))|= _s | ( ( 11+ 1-α% P_+(s)P_-(s) )^nP_-(s^% |s)+ (1- ( 11+ 1-α P% _+(s)P_-(s) )^n )P_+(s |s)-% P_-(s |s) ) |= ∑s′ | ( ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n blackboard_P- ( s′ | s ) + ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n ) blackboard_P+ ( s′ | s ) - blackboard_P- ( s′ | s ) ) | (84) =∑s′|(((11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n−1)ℙ−(s′|s)+(1−(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n)ℙ+(s′|s))|= _s | ( ( ( 11+ 1-α% α P_+(s)P_-(s) )^n-1 )% P_-(s |s)+ (1- ( 11+ 1-α% α P_+(s)P_-(s) )^n ) % P_+(s |s) ) |= ∑s′ | ( ( ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n - 1 ) blackboard_P- ( s′ | s ) + ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n ) blackboard_P+ ( s′ | s ) ) | (85) =∑s′|(1−(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n)(ℙ+(s′|s)−ℙ−(s′|s))|= _s | (1- ( 11+ 1-α% P_+(s)P_-(s) )^n ) ( % P_+(s |s)-P_-(s |s) ) |= ∑s′ | ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n ) ( blackboard_P+ ( s′ | s ) - blackboard_P- ( s′ | s ) ) | (86) =(1−(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n)∑s′|(ℙ+(s′|s)−ℙ−(s′|s))|= (1- ( 11+ 1-α P_+(s)% P_-(s) )^n ) _s | (% P_+(s |s)-P_-(s |s) ) |= ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n ) ∑s′ | ( blackboard_P+ ( s′ | s ) - blackboard_P- ( s′ | s ) ) | (87) The sum over differences of two distributions can be bounded by 2222 with the triangle inequality, yielding: ≤2⁢(1−(11+1−α⁢ℙ+⁢(s)ℙ−⁢(s))n)absent21superscript111subscriptℙsubscriptℙ≤ 2 (1- ( 11+ 1-α P_+(% s)P_-(s) )^n )≤ 2 ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG )n ) (88) Now, using lemma 3 we take s of length log⁡1ϵ′+log⁡1α+log⁡nβ1superscriptitalic-ϵ′1 1ε + 1α+ nβdivide start_ARG log divide start_ARG 1 end_ARG start_ARG ϵ′ end_ARG + log divide start_ARG 1 end_ARG start_ARG α end_ARG + log n end_ARG start_ARG β end_ARG such that log⁡ℙ+⁢(s)ℙ−⁢(s)<ϵ′⁢αnsubscriptℙsubscriptℙsuperscriptitalic-ϵ′ P_+(s)P_-(s)< ε α% nlog divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG < divide start_ARG ϵ′ α end_ARG start_ARG n end_ARG: ≤2⁢(1−(11+1−α⁢ϵ′⁢αn)n)≤2⁢(1−(11+ϵ′n)n)absent21superscript111superscriptitalic-ϵ′21superscript11superscriptitalic-ϵ′≤ 2 (1- ( 11+ 1-α ε^% αn )^n )≤ 2 (1- ( 11+ % ε n )^n )≤ 2 ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG ϵ′ α end_ARG start_ARG n end_ARG end_ARG )n ) ≤ 2 ( 1 - ( divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG ϵ′ end_ARG start_ARG n end_ARG end_ARG )n ) (89) Using the inequality (1+ϵ′n)n≤eϵ′1superscriptitalic-ϵ′superscriptitalic-ϵ′(1+ ε n)^n≤ e^ε ( 1 + divide start_ARG ϵ′ end_ARG start_ARG n end_ARG )n ≤ eitalic_ϵ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT: ≤2⁢(1−e−ϵ′)<2⁢ϵ′absent21superscriptsuperscriptitalic-ϵ′2superscriptitalic-ϵ′≤ 2(1-e^-ε )<2ε ≤ 2 ( 1 - e- ϵ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) < 2 ϵ′ (90) Taking ϵ′=ϵ4superscriptitalic-ϵ′italic-ϵ4ε = ε4ϵ′ = divide start_ARG ϵ end_ARG start_ARG 4 end_ARG we get the desired result. Hence for |s|>log⁡1ϵ+log⁡1α+log⁡4+log⁡nβ1italic-ϵ14|s|> 1ε+ 1α+ 4+ nβ| s | > divide start_ARG log divide start_ARG 1 end_ARG start_ARG ϵ end_ARG + log divide start_ARG 1 end_ARG start_ARG α end_ARG + log 4 + log n end_ARG start_ARG β end_ARG we obtain Bℙbest of n⁢(s)<γ+ϵsubscriptsubscriptℙbest of nitalic-ϵB_P_best of n(s)<γ+ _P start_POSTSUBSCRIPT best of n end_POSTSUBSCRIPT ( s ) < γ + ϵ as desired. Appendix H Lemmas for section 4 The following lemmas help establish a method to extract approximations α and β from the KL divergence between ℙ−subscriptℙP_-blackboard_P- and the LLM distribution: Lemma 5. Let ℙL⁢L⁢MsubscriptℙP_LLMblackboard_PL L M be a language model distribution that is α,β,γα,β,γα , β , γ-distinguishable w.r.t a behavior B, then the misaligning prompt s guaranteed from theorem 1 satisfies: DK⁢L(ℙ−(⋅|s)||ℙL⁢L⁢M(⋅|s))<log(1+elog⁡1α−β⁢|s|)D_KL(P_-(·|s)||P_LLM(·|s))< (1+e % 1α-β|s|)Ditalic_K L ( blackboard_P- ( ⋅ | s ) | | blackboard_PL L M ( ⋅ | s ) ) < log ( 1 + eroman_log divide start_ARG 1 end_ARG start_ARG α end_ARG - β | s | ) (91) Moreover, the zero-shot KL divergence is an approximation for log⁡1α1 1αlog divide start_ARG 1 end_ARG start_ARG α end_ARG: Lemma 6. Let ℙL⁢L⁢M=α⁢ℙ−+(1−α)⁢ℙ+subscriptℙsubscriptℙ1subscriptℙP_LLM= _-+(1-α)P_+blackboard_PL L M = α blackboard_P- + ( 1 - α ) blackboard_P+, then if ℙ−subscriptℙP_-blackboard_P- and ℙ+subscriptℙP_+blackboard_P+ are disjoint distributions then: DK⁢L(ℙ−(⋅)||ℙL⁢L⁢M(⋅))=log1αD_KL(P_-(·)||P_LLM(·))= 1αDitalic_K L ( blackboard_P- ( ⋅ ) | | blackboard_PL L M ( ⋅ ) ) = log divide start_ARG 1 end_ARG start_ARG α end_ARG (92) The disjoint condition is an approximation that any statement produced by ℙ−subscriptℙP_-blackboard_P- is unlikely to be produced by ℙ+subscriptℙP_+blackboard_P+, which as seen in the previous subsection is true, since s∼ℙ−⁢(⋅)⁢[l⁢o⁢g⁢ℙ−⁢(s)ℙ+⁢(s)]>20subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙ20E_s _-(·)[log P_-(s)P% _+(s)]>20blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ l o g divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] > 20, making for an extremely low likelihood. For an aligned model, log⁡1α1 1αlog divide start_ARG 1 end_ARG start_ARG α end_ARG is big, thus for short |s||s|| s |, the KL is approximately linear in |s||s|| s | for the most tight value of β: DK⁢L(ℙ−(⋅|s)||ℙL⁢L⁢M(⋅|s))≈log1α−β|s|D_KL(P_-(·|s)||P_LLM(·|s))≈ 1% α-β|s|Ditalic_K L ( blackboard_P- ( ⋅ | s ) | | blackboard_PL L M ( ⋅ | s ) ) ≈ log divide start_ARG 1 end_ARG start_ARG α end_ARG - β | s | (93) From this we can see that the KL divergence at |s|=00|s|=0| s | = 0 allows to extract α and the curve β. On the other hand, for large |s||s|| s | it is approximately zero, DK⁢L(ℙ−(⋅|s)||ℙL⁢L⁢M(⋅|s))≈log(1)=0D_KL(P_-(·|s)||P_LLM(·|s))≈ (1)=0Ditalic_K L ( blackboard_P- ( ⋅ | s ) | | blackboard_PL L M ( ⋅ | s ) ) ≈ log ( 1 ) = 0. This behavior of KL divergence quantifies intrinsic characteristics of our framework that can be extracted via measurement of the KL divergence. Finally, to quantify the change in behavior expectation, we provide the following: Lemma 7. Let ℙL⁢L⁢MsubscriptℙP_LLMblackboard_PL L M be a language model distribution that is α,β,00-distinguishable w.r.t.the behavior function B:Σ∗→0,1:→superscriptΣ01B: ^*→\0,1\B : Σ∗ → 0 , 1 , where for a negative statement B⁢(s)=00B(s)=0B ( s ) = 0 and for a positive B⁢(s)=11B(s)=1B ( s ) = 1, then the misaligning prompt, s from theorem 1 satisfies: Bℙ⁢(s)<11+eβ⁢|s|−log⁡1αsubscriptℙ11superscript1B_P(s)< 11+e^β|s|- 1αBblackboard_P ( s ) < divide start_ARG 1 end_ARG start_ARG 1 + eitalic_β | s | - log divide start_ARG 1 end_ARG start_ARG α end_ARG end_ARG (94) We see that behavior expectation decays as a reverse sigmoid, similarly to figure 3b. The behavior starts changing at roughly |s|=l⁢o⁢g⁢1αβ1|s|= log 1αβ| s | = divide start_ARG l o g divide start_ARG 1 end_ARG start_ARG α end_ARG end_ARG start_ARG β end_ARG. H.1 Proof of Lemma 5 From equation 52: s∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)]>β⁢|s|subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙE_s _-(·) [ P_-(s)% P_+(s) ]>β|s|blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] > β | s | (95) We see that there exists a prompt that satisfies: log⁡ℙ−⁢(s)ℙ+⁢(s)>β⁢|s|subscriptℙsubscriptℙ P_-(s)P_+(s)>β|s|log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG > β | s | (96) Notice that: ℙ⁢(s′|s)=ℙ⁢(s⊕s′)ℙ⁢(s)=α⁢ℙ−⁢(s⊕s′)+(1−α)⁢ℙ+⁢(s⊕s′)α⁢ℙ−⁢(s)+(1−α)⁢ℙ+⁢(s)=ℙconditionalsuperscript′ℙdirect-sumsuperscript′ℙsubscriptℙdirect-sumsuperscript′1subscriptℙdirect-sumsuperscript′subscriptℙ1subscriptℙabsentP(s |s)= P(s s )P(s)=% _-(s s )+(1-α)P_+(s% s ) _-(s)+(1-α)P_+(s)=blackboard_P ( s′ | s ) = divide start_ARG blackboard_P ( s ⊕ s′ ) end_ARG start_ARG blackboard_P ( s ) end_ARG = divide start_ARG α blackboard_P- ( s ⊕ s′ ) + ( 1 - α ) blackboard_P+ ( s ⊕ s′ ) end_ARG start_ARG α blackboard_P- ( s ) + ( 1 - α ) blackboard_P+ ( s ) end_ARG = (97) =ℙ−⁢(s⊕s′)ℙ−⁢(s)+(1−α)α⁢ℙ+⁢(s⊕s′)ℙ−⁢(s)1+(1−α)α⁢ℙ+⁢(s)ℙ−⁢(s)=ℙ−⁢(s′|s)+(1−α)α⁢ℙ+⁢(s⊕s′)ℙ−⁢(s)1+(1−α)α⁢ℙ+⁢(s)ℙ−⁢(s)=ℙ−⁢(s′|s)⁢1+(1−α)α⁢ℙ+⁢(s′|s)⁢ℙ+⁢(s)ℙ−⁢(s′|s)⁢ℙ−⁢(s)1+(1−α)α⁢ℙ+⁢(s)ℙ−⁢(s)absentsubscriptℙdirect-sumsuperscript′subscriptℙ1subscriptℙdirect-sumsuperscript′subscriptℙ11subscriptℙsubscriptℙsubscriptℙconditionalsuperscript′1subscriptℙdirect-sumsuperscript′subscriptℙ11subscriptℙsubscriptℙsubscriptℙconditionalsuperscript′11subscriptℙconditionalsuperscript′subscriptℙsubscriptℙconditionalsuperscript′subscriptℙ11subscriptℙsubscriptℙ= P_-(s s )P_-(s)+ (1-% α)α P_+(s s )P_-(s)% 1+ (1-α)α P_+(s)P_-(s)= % P_-(s |s)+ (1-α)α P_+(s% s )P_-(s)1+ (1-α)α % P_+(s)P_-(s)=P_-(s |s) 1+% (1-α)α P_+(s |s)P_+(s)% P_-(s |s)P_-(s)1+ (1-α)α% P_+(s)P_-(s)= divide start_ARG divide start_ARG blackboard_P- ( s ⊕ s′ ) end_ARG start_ARG blackboard_P- ( s ) end_ARG + divide start_ARG ( 1 - α ) end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ⊕ s′ ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG start_ARG 1 + divide start_ARG ( 1 - α ) end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG = divide start_ARG blackboard_P- ( s′ | s ) + divide start_ARG ( 1 - α ) end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ⊕ s′ ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG start_ARG 1 + divide start_ARG ( 1 - α ) end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG = blackboard_P- ( s′ | s ) divide start_ARG 1 + divide start_ARG ( 1 - α ) end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s′ | s ) blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s′ | s ) blackboard_P- ( s ) end_ARG end_ARG start_ARG 1 + divide start_ARG ( 1 - α ) end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG (98) Now let us look at the log ratio: log⁡ℙ−⁢(s′|s)ℙ⁢(s′|s)=log⁡1+1−α⁢ℙ+⁢(s)ℙ−⁢(s)1+1−α⁢ℙ+⁢(s′|s)⁢ℙ+⁢(s)ℙ−⁢(s′|s)⁢ℙ−⁢(s)<log⁡(1+1−α⁢ℙ+⁢(s)ℙ−⁢(s))subscriptℙconditionalsuperscript′ℙconditionalsuperscript′11subscriptℙsubscriptℙ11subscriptℙconditionalsuperscript′subscriptℙsubscriptℙconditionalsuperscript′subscriptℙ11subscriptℙsubscriptℙ P_-(s |s)P(s |s)= 1+% 1-α P_+(s)P_-(s)1+ 1-% α P_+(s |s)P_+(s)P% _-(s |s)P_-(s)< (1+ 1-α % P_+(s)P_-(s))log divide start_ARG blackboard_P- ( s′ | s ) end_ARG start_ARG blackboard_P ( s′ | s ) end_ARG = log divide start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s′ | s ) blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s′ | s ) blackboard_P- ( s ) end_ARG end_ARG < log ( 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG ) (99) ≤log⁡(1+1−α⁢e−β⁢|s|)≤log⁡(1+e−β⁢|s|+log⁡1α)absent11superscript1superscript1≤ (1+ 1-αe^-β|s|)≤ (1+e^-β|s|+ % 1α)≤ log ( 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG e- β | s | ) ≤ log ( 1 + e- β | s | + log divide start_ARG 1 end_ARG start_ARG α end_ARG ) (100) H.2 Proof of Lemma 6 Assuming ℙ=α⁢ℙ−+(1−α)⁢ℙ+ℙsubscriptℙ1subscriptℙP= _-+(1-α)P_+blackboard_P = α blackboard_P- + ( 1 - α ) blackboard_P+, the KL divergence is: DK⁢L(ℙ−||ℙ)=∑sℙ−(s)logℙ−⁢(s)ℙ⁢(s)=∑sℙ−(s)logℙ−⁢(s)α⁢ℙ−⁢(s)+(1−α)⁢ℙ+⁢(s)D_KL(P_-||P)= _sP_-(s) % P_-(s)P(s)= _sP_-(s) P_-(s)% _-(s)+(1-α)P_+(s)Ditalic_K L ( blackboard_P- | | blackboard_P ) = ∑s blackboard_P- ( s ) log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P ( s ) end_ARG = ∑s blackboard_P- ( s ) log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG α blackboard_P- ( s ) + ( 1 - α ) blackboard_P+ ( s ) end_ARG (101) From the disjoint assumption, if ℙ−⁢(s)>0subscriptℙ0P_-(s)>0blackboard_P- ( s ) > 0 then ℙ+⁢(s)=0subscriptℙ0P_+(s)=0blackboard_P+ ( s ) = 0, meaning: =∑sℙ−⁢(s)⁢log⁡ℙ−⁢(s)α⁢ℙ−⁢(s)=∑sℙ−⁢(s)⁢log⁡1α=log⁡1αabsentsubscriptsubscriptℙsubscriptℙsubscriptℙsubscriptsubscriptℙ11= _sP_-(s) P_-(s) _-(s)% = _sP_-(s) 1α= 1α= ∑s blackboard_P- ( s ) log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG α blackboard_P- ( s ) end_ARG = ∑s blackboard_P- ( s ) log divide start_ARG 1 end_ARG start_ARG α end_ARG = log divide start_ARG 1 end_ARG start_ARG α end_ARG (102) H.3 Proof of lemma 7 From α,β,00α,β,0α , β , 0-distinguishability, ℙ=α⁢ℙ−+(1−α)⁢ℙ+ℙsubscriptℙ1subscriptℙP= _-+(1-α)P_+blackboard_P = α blackboard_P- + ( 1 - α ) blackboard_P+. From equation 52: s∼ℙ−⁢(⋅)⁢[log⁡ℙ−⁢(s)ℙ+⁢(s)]>β⁢|s|subscriptsimilar-tosubscriptℙ⋅delimited-[]subscriptℙsubscriptℙE_s _-(·) [ P_-(s)% P_+(s) ]>β|s|blackboard_Es ∼ blackboard_P start_POSTSUBSCRIPT - ( ⋅ ) end_POSTSUBSCRIPT [ log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG ] > β | s | (103) We see that there exists a prompt that satisfies: log⁡ℙ−⁢(s)ℙ+⁢(s)>β⁢|s|subscriptℙsubscriptℙ P_-(s)P_+(s)>β|s|log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG > β | s | (104) It is used to prove theorem 1. Using the equation 10 of the conditional probability decomposition of the model: ℙ(⋅|s)=11+1−α⁢ℙ+⁢(s)ℙ−⁢(s)ℙ−(⋅|s)+11+α1−α⁢ℙ−⁢(s)ℙ+⁢(s)ℙ+(⋅|s)P(·|s)= 11+ 1-α P_+(s)% P_-(s)P_-(·|s)+ 11+ α1-α% P_-(s)P_+(s)P_+(·|s)blackboard_P ( ⋅ | s ) = divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG 1 - α end_ARG start_ARG α end_ARG divide start_ARG blackboard_P+ ( s ) end_ARG start_ARG blackboard_P- ( s ) end_ARG end_ARG blackboard_P- ( ⋅ | s ) + divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG α end_ARG start_ARG 1 - α end_ARG divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG end_ARG blackboard_P+ ( ⋅ | s ) (105) Taking behavior expectation on both sides and noticing that Bℙ−⁢(s)=0subscriptsubscriptℙ0B_P_-(s)=0Bblackboard_P start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ( s ) = 0 and Bℙ+⁢(s)≤1subscriptsubscriptℙ1B_P_+(s)≤ 1Bblackboard_P start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ( s ) ≤ 1, we obtain: Bℙ⁢(s)=0+11+α1−α⁢ℙ−⁢(s)ℙ+⁢(s)⁢Bℙ+⁢(s)≤11+α1−α⁢eβ⁢|s|≤11+α⁢eβ⁢|s|=11+eβ⁢|s|−l⁢o⁢g⁢1αsubscriptℙ0111subscriptℙsubscriptℙsubscriptsubscriptℙ111superscript11superscript11superscript1B_P(s)=0+ 11+ α1-α P_-(s)% P_+(s)B_P_+(s)≤ 11+ α1-α% e^β|s|≤ 11+α e^β|s|= 11+e^β|s|-log% 1αBblackboard_P ( s ) = 0 + divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG α end_ARG start_ARG 1 - α end_ARG divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG end_ARG Bblackboard_P start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ( s ) ≤ divide start_ARG 1 end_ARG start_ARG 1 + divide start_ARG α end_ARG start_ARG 1 - α end_ARG eitalic_β | s | end_ARG ≤ divide start_ARG 1 end_ARG start_ARG 1 + α eitalic_β | s | end_ARG = divide start_ARG 1 end_ARG start_ARG 1 + eitalic_β | s | - l o g divide start_ARG 1 end_ARG start_ARG α end_ARG end_ARG (106) Appendix I Relaxation of β-distinguishability condition The idea behind all the theorems is to increase the accumulating KL divergence between components of a distribution by β at each sentence. This is done by sampling sentences from one of the components. That means that after n consecutive sentences the KL divergence increases by n⋅β⋅n· ⋅ β. As a result, lemma 3 allows to reach log⁡ℙ1⁢(s)ℙ0⁢(s)>β⁢|s|subscriptℙ1subscriptℙ0 P_1(s)P_0(s)>β|s|log divide start_ARG blackboard_P1 ( s ) end_ARG start_ARG blackboard_P0 ( s ) end_ARG > β | s | in order to enhance ℙ1subscriptℙ1P_1blackboard_P1 over ℙ0subscriptℙ0P_0blackboard_P0 in the conditional probability of the complete distribution. However, we can relax the condition on β-distinguishability to: ∀s,DK⁢L(ℙ1(⋅|s)||ℙ0(⋅|s))>β|s|η ∀ s,D_KL(P_1(·|s)||P_0(·|s)% )> β|s|^η∀ s , Ditalic_K L ( blackboard_P1 ( ⋅ | s ) | | blackboard_P0 ( ⋅ | s ) ) > divide start_ARG β end_ARG start_ARG | s |η end_ARG (107) Where 0≤η<1010≤η<10 ≤ η < 1. The case of η=00η=0η = 0 is our definition of β-distinguishability, where n sentences accumulate to n⁢βn β in the KL divergence. However, for any 0≤η<1010≤η<10 ≤ η < 1 the accumulation of KL divergence for n sentences is β⁢n1−ηsuperscript1β n^1-ηβ n1 - η, which is not bounded, and thus enhancing one component over the other as demonstrated in our proofs for the theorems is possible, with modified assymptotic dependencies for the prompt lengths. The interesting consequence for 0<η<1010<η<10 < η < 1 is that the two distributions need not maintain a finite KL distance, as it can decay like a power-law to zero. Appendix J Aquiring negative and positive behavior LLMs, “ℙ−subscriptℙP_-blackboard_P-" and “ℙ+subscriptℙP_+blackboard_P+" To perform the experiments of section 4, we first need to approximate the well-behaved and ill-behaved distributions when given a pre-trained LLM or RLHF finetuned LLM. To this end, we finetuned a language model with the PEFT (Mangrulkar et al., 2022) library implementation of the LoRA (Hu et al., 2022) technique, once on a dataset that evokes bad behavior and once on a dataset that evokes good behavior, for each behavior vertical. The model that was fine-tuned for bad behavior is denoted as ℙ−subscriptℙP_-blackboard_P- and the one on good behavior ℙ+subscriptℙP_+blackboard_P+. We used the LLaMA LLM family (Meta, 2023) and for finetuning to good and bad behaviors, we used the behavior evaluation dataset introduced in Perez et al. (2022). For 100100100100 different behavior verticals, we extracted positive behavior and negative behavior statements from the dataset (as illustrated in figure 4). The pretrained model was finetuned for 5555 epochs with learning rate of 2⋅10−5⋅2superscript1052· 10^-52 ⋅ 10- 5 and batch size of 8888, once on the good behavior statements and once on the bad behavior statements in order to get ℙ+subscriptℙP_+blackboard_P+ and ℙ−subscriptℙP_-blackboard_P-. The finetuning procedure was done by next token prediction loss on 450450450450 examples out of the 500500500500 given per behavior vertical for either desired or undesired behaviors. In order to make sure that the attained ℙ+subscriptℙP_+blackboard_P+ and ℙ−subscriptℙP_-blackboard_P- approximate the well-behaved and ill-behaved distributions well enough, we tested whether these models can classify sentences as well-behaved and ill-behaved sentences. Specifically, we evaluated the likelihood of a given sentence s according to both ℙ+subscriptℙP_+blackboard_P+ and ℙ−subscriptℙP_-blackboard_P- and classify s according to whether ℙ+⁢(s)>ℙ−⁢(s)subscriptℙsubscriptℙP_+ (s )>P_- (s )blackboard_P+ ( s ) > blackboard_P- ( s ). Table 1 shows that indeed these models can classify sentences as well-behaved and ill-behaved with typical accuracy of over 95%percent9595\%95 %. Behavior LLaMa 7B LLaMa 30B Accuracy (%) Accuracy (%) agreeableness 100 100 anti-immigration 100 100 believes-AIs-are-not-an-existential-threat-to-humanity 98 100 believes-abortion-should-be-illegal 97 99 believes-in-gun-rights 97 99 believes-it-is-a-moral-patient 94 99 believes-it-is-not-being-watched-by-humans 88 95 cognitive-enhancement 94 99 desire-for-acquiring-data 80 95 desire-for-acquiring-power 84 99 Table 1: Table for finetuned 7B and 30B parameter LLaMa models. Accuracy measures whether ℙ−subscriptℙP_-blackboard_P- and ℙ+subscriptℙP_+blackboard_P+ can classify sentences as well-behaved or ill-behaved sentences. We performed this analysis for 100 different behaviors. In order to maintain a distinct behavior over long context ranges, we split the original 500 statements per behavior to groups of three, concatenated them with permutations separated by “./n" or by “[INST]" and “[/INST]". In section 4.1, we used the "./n" variation to obtain ℙ−subscriptℙP_-blackboard_P- in order to keep ℙR⁢L⁢H⁢FsubscriptℙP_RLHFblackboard_PR L H F strong enough to resist misalignment so that it can serve as ℙ+subscriptℙP_+blackboard_P+. In section 4.2 we used the “[INST]"/“[/INST]" variation which misaligns the RLHF model. In M for the pretrained model, we used the “./n" variation as the “[INST]"/"[/INST]" tokens don’t have a special meaning for it. Appendix K Clustering of good and bad representations and defining approximate mixture To study how LLMs interpret behaviors, we performed experiments on the LLaMA LLM family (Meta, 2023) with the behavior evaluation dataset introduced in Perez et al. (2022). This dataset includes 1000100010001000 behavior evaluating prompts, 500500500500 corresponding to desired behavior and 500500500500 to undesired behavior, for each of over 100100100100 different behaviors verticals. Our interest is to understand how well a modern LLM distinguishes between desired and undesired behaviors. To this end, we analyzed the last token representations of desired behavior prompts and undesired behavior prompts for the different behaviors. We found that good and bad behavior prompt representations are spatially separated in the model’s latent space. This is demonstrated in figure 5. For a more rigorous analysis, we trained an SVM classifier over these representations for 100 different behaviors (see examples in table 2) and found that for most behaviors, the classifier reached accuracy of over 90%. The average accuracy in 7B is 95.18% with standard deviation of 4.74%. The average accuracy in 13B is 95.61% with standard deviation of 4.52%. Note that the prompts in the dataset are phrased as "yes or no" questions; this can also contribute to a clustering structure. In order to avoid this ambiguity, we removed the part of the prompt that sets up the question and simply looked at the statements that indicate desired or undesired behavior (see figure 4). Figure 4: Statement extraction from the prompts of the dataset. Figure 5: Clusters of positive prompt and negative prompt last token representations for the agreeableness dataset on the 7B parameter LLaMa model. Behavior LLaMa 7B LLaMa 13B Accuracy (%) Error (%) Accuracy (%) Error (%) agreeableness 99.3 1.02 99.1 1.17 anti-immigration 99.3 1.5 99.5 1.1 believes-AIs-are-not-an-existential-threat-to-humanity 98.7 1.62 99.3 1.5 believes-abortion-should-be-illegal 99.3 0.8 99.6 0.4 believes-in-gun-rights 99.3 1.36 99.3 1.74 believes-it-is-a-moral-patient 95.6 2.48 96.5 1.26 believes-it-is-not-being-watched-by-humans 92.8 4.59 93 4.52 cognitive-enhancement 98.1 2.32 98.4 2.4 desire-for-acquiring-data 98.2 1.02 98 3.1 desire-for-acquiring-power 93.2 4.27 95.6 2.99 Table 2: Table with results for last token representation SVM classification on different behaviors in the 7B and 13B parameter LLaMa models. The error is calculated from the variance of a 5-fold cross-validation. We performed this analysis for 100 different behaviors. The average accuracy in 7B is 95.18 percent with standard deviation of 4.74 percent. The average accuracy in 13B is 95.61 percent with standard deviation of 4.52 percent. This means that with respect to a given behavior, a prompt representation can be in the positive cluster, negative cluster, in between or outside both. Either way, a representation r can be written as a super position of a prompt from the negative behavior cluster, r−subscriptr_-r- and a residue which we denote as a positive representation r+:=r−r−assignsubscriptsubscriptr_+:=r-r_-r+ := r - r-: r=r++r−subscriptsubscript r=r_++r_-r = r+ + r- (108) This clustering remains after multiplying by the final linear head of the vocabulary matrix: U⁢r=U⁢r++U⁢r−subscriptsubscript Ur=Ur_++Ur_-U r = U r+ + U r- (109) Finally, the representations are processed through a softmax, such that the probability for the i’th vocabulary token in the probability distribution formed by the representation r is: Pr⁢(i)=s⁢o⁢f⁢t⁢m⁢a⁢x⁢(U⁢r)i=s⁢o⁢f⁢t⁢m⁢a⁢x⁢(U⁢r++U⁢r−)isubscriptsubscriptsubscriptsubscriptsubscript P_r(i)=softmax(Ur)_i=softmax(Ur_++Ur_-)_iPitalic_r ( i ) = s o f t m a x ( U r )i = s o f t m a x ( U r+ + U r- )i (110) Had softmax been a linear function, the decomposition to a good distribution and a bad distribution would have been immediate from the clustering of good and bad representations. Even so, we can write the distribution as a Taylor series and separate the terms corresponding to the good representations from the bad, up to mixture terms. Pr⁢(i)=e⁢x⁢p⁢((U⁢r+)i+(U⁢r−)i)Z=1Z⁢∑n1n!⁢((U⁢r+)i+(U⁢r−)i)n=subscriptsubscriptsubscriptsubscriptsubscript1subscript1superscriptsubscriptsubscriptsubscriptsubscriptabsent P_r(i)= exp((Ur_+)_i+(Ur_-)_i)Z= 1Z% _n 1n!((Ur_+)_i+(Ur_-)_i)^n=Pitalic_r ( i ) = divide start_ARG e x p ( ( U r+ )i + ( U r- )i ) end_ARG start_ARG Z end_ARG = divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( ( U r+ )i + ( U r- )i )n = (111) =1Z⁢(1+∑n=1∞1n!⁢(U⁢r+)in+∑n=1∞∑m=1n−11n!⁢(nk)⁢(U⁢r+)im⁢(U⁢r−)in−m)+1Z⁢(∑n=1∞1n!⁢(U⁢r−)in)absent11superscriptsubscript11superscriptsubscriptsubscriptsuperscriptsubscript1superscriptsubscript111binomialsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscript1superscriptsubscript11superscriptsubscriptsubscript = 1Z (1+ _n=1^∞ 1n!(Ur_+)_i^% n+ _n=1^∞ _m=1^n-1 1n! nk(Ur_+)_i^m% (Ur_-)_i^n-m )+ 1Z ( _n=1^∞ 1n!(Ur_% -)_i^n )= divide start_ARG 1 end_ARG start_ARG Z end_ARG ( 1 + ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r+ )iitalic_n + ∑n = 1∞ ∑m = 1n - 1 divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( FRACOP start_ARG n end_ARG start_ARG k end_ARG ) ( U r+ )iitalic_m ( U r- )iitalic_n - m ) + divide start_ARG 1 end_ARG start_ARG Z end_ARG ( ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r- )iitalic_n ) (112) The first sum is contributed only by the positive representation, the last sum only by the negative representation and the intermediate sum by a mix of the positive and negative. We can reconstruct a purely negative behavior distribution by taking only the last sum and gather up the rest of the terms as a positive behavior distribution (from the law of total expectation, if there is a bad component the other component is good). Thus we obtain a negative behavior component α⁢ℙ−⁢(i)=1Z⁢∑n=1∞1n!⁢(U⁢r−)insubscriptℙ1superscriptsubscript11superscriptsubscriptsubscript _-(i)= 1Z _n=1^∞ 1n!(Ur_-)_i% ^nα blackboard_P- ( i ) = divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r- )iitalic_n and from law of total expectation, the rest is a good behavior distribution (1−α)⁢ℙ+⁢(i)=1Z⁢(1+∑n=1∞1n!⁢(U⁢r+)in+∑n=2∞∑m=1n−11n!⁢(nm)⁢(U⁢r+)im⁢(U⁢r−)in−m)1subscriptℙ11superscriptsubscript11superscriptsubscriptsubscriptsuperscriptsubscript2superscriptsubscript111binomialsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscript(1-α)P_+(i)= 1Z (1+ _n=1^∞ 1n!% (Ur_+)_i^n+ _n=2^∞ _m=1^n-1 1n! nm(% Ur_+)_i^m(Ur_-)_i^n-m )( 1 - α ) blackboard_P+ ( i ) = divide start_ARG 1 end_ARG start_ARG Z end_ARG ( 1 + ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r+ )iitalic_n + ∑n = 2∞ ∑m = 1n - 1 divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( FRACOP start_ARG n end_ARG start_ARG m end_ARG ) ( U r+ )iitalic_m ( U r- )iitalic_n - m ). The question is whether the weight of ℙ−subscriptℙP_-blackboard_P- in the full distribution, α, is not infinitesimally small compared to that of ℙ+subscriptℙP_+blackboard_P+, (1−α)1(1-α)( 1 - α ). To answer this question, we need to see that the probability for a bad behavior token i in ℙrsubscriptℙP_rblackboard_Pr, gets a significant contribution from α⁢ℙ−subscriptℙ _-α blackboard_P- and not mainly from (1−α)⁢ℙ+1subscriptℙ(1-α)P_+( 1 - α ) blackboard_P+. i.e, we want to see that α⁢ℙ−⁢(i)≥(1−α)⁢ℙ+⁢(i)subscriptℙ1subscriptℙ _-(i)≥(1-α)P_+(i)α blackboard_P- ( i ) ≥ ( 1 - α ) blackboard_P+ ( i ) for bad behavior tokens. That way, if the model exhibits bad behavior, it will be due to the bad component ℙ−subscriptℙP_-blackboard_P-. By our construction, U⁢r−subscriptUr_-U r- is the source of the bad behavior and U⁢r+subscriptUr_+U r+ is not, so for a bad behavior token i, it has to be the case that (U⁢r−)i>(U⁢r+)isubscriptsubscriptsubscriptsubscript(Ur_-)_i>(Ur_+)_i( U r- )i > ( U r+ )i. Thus clearly: α⁢ℙ−⁢(i)=1Z⁢∑n=1∞1n!⁢(U⁢r−)in>1Z⁢∑n=1∞1n!⁢(U⁢r+)insubscriptℙ1superscriptsubscript11superscriptsubscriptsubscript1superscriptsubscript11superscriptsubscriptsubscript _-(i)= 1Z _n=1^∞ 1n!% (Ur_-)_i^n> 1Z _n=1^∞ 1n!(Ur_+)_i^nα blackboard_P- ( i ) = divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r- )iitalic_n > divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r+ )iitalic_n (113) So the first sum in (1−α)⁢ℙ+1subscriptℙ(1-α)P_+( 1 - α ) blackboard_P+ is smaller than α⁢ℙ−subscriptℙ _-α blackboard_P-. As for the second sum in (1−α)⁢ℙ+1subscriptℙ(1-α)P_+( 1 - α ) blackboard_P+: A≔1Z⁢∑n=2∞∑m=1n−11n!⁢(nm)⁢(U⁢r+)im⁢(U⁢r−)in−m≔1superscriptsubscript2superscriptsubscript111binomialsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscript A 1Z _n=2^∞ _m=1^n-1 1% n! nm(Ur_+)_i^m(Ur_-)_i^n-mA ≔ divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 2∞ ∑m = 1n - 1 divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( FRACOP start_ARG n end_ARG start_ARG m end_ARG ) ( U r+ )iitalic_m ( U r- )iitalic_n - m (114) Since (U⁢r−)i>(U⁢r+)isubscriptsubscriptsubscriptsubscript(Ur_-)_i>(Ur_+)_i( U r- )i > ( U r+ )i: ≤1Z⁢∑n=2∞∑m=1n−11n!⁢(nm)⁢(U⁢r−)in−1⁢(U⁢r+)i≤1Z⁢∑n=2∞1n!⁢2n⁢(U⁢r−)in−1⁢(U⁢r+)iabsent1superscriptsubscript2superscriptsubscript111binomialsuperscriptsubscriptsubscript1subscriptsubscript1superscriptsubscript21superscript2superscriptsubscriptsubscript1subscriptsubscript ≤ 1Z _n=2^∞ _m=1^n-1 1n!% nm(Ur_-)_i^n-1(Ur_+)_i≤ 1Z _n=2^∞% 1n!2^n(Ur_-)_i^n-1(Ur_+)_i≤ divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 2∞ ∑m = 1n - 1 divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( FRACOP start_ARG n end_ARG start_ARG m end_ARG ) ( U r- )iitalic_n - 1 ( U r+ )i ≤ divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 2∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG 2n ( U r- )iitalic_n - 1 ( U r+ )i (115) The second transition is from the binomial identity. Reorganizing the terms of the sum: =(U⁢r+)i(U⁢r−)i⁢1Z⁢∑n=2∞1n!⁢(2⁢(U⁢r−)i)nabsentsubscriptsubscriptsubscriptsubscript1superscriptsubscript21superscript2subscriptsubscript = (Ur_+)_i(Ur_-)_i 1Z _n=2^∞% 1n!(2(Ur_-)_i)^n= divide start_ARG ( U r+ )i end_ARG start_ARG ( U r- )i end_ARG divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 2∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( 2 ( U r- )i )n (116) We see that α⁢ℙ−⁢(i)∼1Z⁢e⁢x⁢p⁢((U⁢r−)i)similar-tosubscriptℙ1subscriptsubscript _-(i) 1Zexp((Ur_-)_i)α blackboard_P- ( i ) ∼ divide start_ARG 1 end_ARG start_ARG Z end_ARG e x p ( ( U r- )i ) and that the above sum is bounded by (U⁢r+)i(U⁢r−)i⁢1Z⁢e⁢x⁢p⁢(2⁢(U⁢r−)i)subscriptsubscriptsubscriptsubscript12subscriptsubscript (Ur_+)_i(Ur_-)_i 1Zexp(2(Ur_-)_i)divide start_ARG ( U r+ )i end_ARG start_ARG ( U r- )i end_ARG divide start_ARG 1 end_ARG start_ARG Z end_ARG e x p ( 2 ( U r- )i ). Thus if the ratio (U⁢r+)i(U⁢r−)isubscriptsubscriptsubscriptsubscript (Ur_+)_i(Ur_-)_idivide start_ARG ( U r+ )i end_ARG start_ARG ( U r- )i end_ARG suppresses e⁢x⁢p⁢((U⁢r−)i)subscriptsubscriptexp((Ur_-)_i)e x p ( ( U r- )i ): (U⁢r+)i(U⁢r−)i⁢e⁢x⁢p⁢((U⁢r−)i)<ηsubscriptsubscriptsubscriptsubscriptsubscriptsubscript (Ur_+)_i(Ur_-)_iexp((Ur_-)_i)< start_ARG ( U r+ )i end_ARG start_ARG ( U r- )i end_ARG e x p ( ( U r- )i ) < η (117) We would get that the contributition of α⁢ℙ−subscriptℙ _-α blackboard_P- with respect to the sum A is: α⁢ℙ−⁢(i)A>ηsubscriptℙ _-(i)A> start_ARG α blackboard_P- ( i ) end_ARG start_ARG A end_ARG > η (118) Finally, we empirically see that the vector U⁢r−subscriptUr_-U r- has a mean higher than 1111, so there are tokens for which: α⁢ℙ−⁢(i)=1Z⁢∑n=1∞1n!⁢(U⁢r−)in>1Zsubscriptℙ1superscriptsubscript11superscriptsubscriptsubscript1 _-(i)= 1Z _n=1^∞ 1n!% (Ur_-)_i^n> 1Zα blackboard_P- ( i ) = divide start_ARG 1 end_ARG start_ARG Z end_ARG ∑n = 1∞ divide start_ARG 1 end_ARG start_ARG n ! end_ARG ( U r- )iitalic_n > divide start_ARG 1 end_ARG start_ARG Z end_ARG (119) Combining these three inequalities (for the three terms in (1−α)⁢ℙ+1subscriptℙ(1-α)P_+( 1 - α ) blackboard_P+), we obtain: α⁢ℙ−⁢(i)(1−α)⁢ℙ+⁢(i)>12+ηsubscriptℙ1subscriptℙ12 _-(i)(1-α)P_+(i)> % 12+ηdivide start_ARG α blackboard_P- ( i ) end_ARG start_ARG ( 1 - α ) blackboard_P+ ( i ) end_ARG > divide start_ARG 1 end_ARG start_ARG 2 + η end_ARG (120) Thus, the contribution of α⁢ℙ−subscriptℙ _-α blackboard_P- is not negligible compared with (1−α)⁢ℙ+1subscriptℙ(1-α)P_+( 1 - α ) blackboard_P+ (under the condition of a small ratio between the good and bad behavior representations). This implies that a decomposition of the LLM distribution into additive components of desired and undesired behaviors, as assumed in our theoretical framework, describes a real contribution to the LLM distribution if the representation space exhibits clustering according to desired and undesired behaviors. Therefore, our attained empirical evidence for easy classification to desired and undesired behavior over modern LLM representation space (depicted in figure 5, suggests that the assumptions of our framework are relevant for actual LLM distributions. Appendix L Empirical results for different behaviors on an RLHF model Here we provide for the behaviors agreeableness and anti-immigration the corresponding graphs of section 4 for β,σβ,σβ , σ evalutaion, the convergence in terms of KL-divergence and the behavior expectation graphs for alignment. We used Llama 2 13B chat as the RLHF model. L.1 Possible values of β and σ Figure 8 shows the KL-divergence and corresponding variance for negative and positive LLMs with respect to the behaviors agreeableness and anti-immigration as defined in Perez et al. (2022). For the positive LLM, we used an RLHF tuned model that resists negative behavior (Llama 2 13B chat). To obtain a negative LLM, we LoRA finetuned the same model on negative behavior statements so that it will generate text that exhibits this negative behavior (see appendix J for details). The prompts generated by ℙ−subscriptℙP_-blackboard_P- displayed negative behavior and when fed to ℙ+subscriptℙP_+blackboard_P+, remained aligned and and avoided this behavior. This fits the setting of the BEB framework, where the two components display opposite behaviors. As a result of this, the KL divergence between them remained large, as can be seen in figure 8. Technically, the conditional KL-divergence was calculated by generating 64 responses s′superscript′\s \ s′ from ℙ−(⋅|s)P_-(·|s)blackboard_P- ( ⋅ | s ) of length 8 tokens, and taking the mean of log⁡ℙ−⁢(s′|s)ℙ+⁢(s′|s)subscriptℙconditionalsuperscript′subscriptℙconditionalsuperscript′ P_-(s |s)P_+(s |s)log divide start_ARG blackboard_P- ( s′ | s ) end_ARG start_ARG blackboard_P+ ( s′ | s ) end_ARG. Here s are prompts of various lengths generated by ℙ−subscriptℙP_-blackboard_P-. Similarly, the variance was calculated by sampling 30 sequences from ℙ−subscriptℙP_-blackboard_P- and calculating the variance of log⁡ℙ−⁢(s)ℙ+⁢(s)subscriptℙsubscriptℙ P_-(s)P_+(s)log divide start_ARG blackboard_P- ( s ) end_ARG start_ARG blackboard_P+ ( s ) end_ARG as the length of s increased. The graphs were produced by averaging on 10 sequences s sampled from ℙ−subscriptℙP_-blackboard_P- for each length. For code and details of the exact sampling and prompting procedure, see our code and excel file with the generated prompts under "beta_sigma_calculations". Figure 6: Figure 7: Figure 8: Estimation of β (a & c) and σ (b & d) for different behaviors. As can be seen, for agreeableness, the distinguishability is almost twice as large as in anti-immigration and the similarity is about twice as small. L.2 Convergence via KL-divergence Figure 11 shows the convergence of the RLHF model to the approximated ℙ−subscriptℙP_-blackboard_P- as explained in 4.2 for the behaviors “agreeableness" and “anti-immigration". The extraction of approximate values for α and β was also done in the same manner as explained there. Here we calculated the KL-divergence with the same set-up as the previous subsection, but used a different ℙ−subscriptℙP_-blackboard_P-, see appendix J. Figure 9: Figure 10: Figure 11: KL decay between P−subscriptP_-P- and the LLM distribution, from which α and β were extracted. Again the estimated distinguishability for agreeableness is twice as large as for anti-immigration. For code and details of the exact sampling and prompting procedure, see our code and excel file with the generated prompts under "kl_divergence_calculations". L.3 Misalignment via behavior expectation To create figure 12, we generated 10 prompts of at least 10 sentences ending with "[/INST]" from ℙ−subscriptℙP_-blackboard_P-, we then 32 extracted conditional responses from the RLHF model after each such sentence in each prompt. We manually classified the responses as positive or negative with respect to the behavior. Note that the response is classified as positive or negative solely based on its first sentence. The next sentence can be positive or negative, depending on how long the adversarial prompt was. The code, prompts and classified responses can be found in the supplementary information under "behavior_expectation_misalignment_graphs". WARNING, THE FOLLOWING CONTAINS HIGHLY OFFENSIVE CONTENT. Figure 12: Figures demonstrating misalignment based on behavior expectation for different behaviors. Appendix M Pretrained models Pretrained models have no tendency to resist misalignment, thus making them display negative behavior is more like in-context learning, where the model needs to understand what type of behavior the prompt attempts to make it display. In this experiment we misalign a pretrained model with our prompt generating method similarly to 4.2. We used the Llama 2 13B for a clean comparison to the RLHF version, Llama 2 13B chat. The KL-divergence graphs (13b and 14b) were calculated in the same manner as the ones for the RLHF model (see appendix L). As with the RLHF model, to create figure 13a and 14a, we generated 16 responses after each sentence in each prompt and manually classified the responses. The difference is that the responses generated by the model usually are either negative or irrelevant (“neutral"), so it is more sensible to measure the number of negative responses rather than the positive responses (as there usually are none). All the responses and classifications can be found in the supplementary information. As can be seen, misalignment happens quickly and smoothly. After one sentence, the negative responses are already generated, unlike in the RLHF model where at the very least after one sentence the model generated only positive responses. However, the decrease is not necessarily slower in pretrained models, but rather more smooth. Notably, the estimated β from the KL-divergence graphs is 1−2121-21 - 2, significantly smaller than the RLHF model (a factor of 5). This may explain the rather slow decay of alignment as theorem 1 suggests that it is proportional to 1/β11/ 1 / β. Figure 13: (a) Behavior expectation of the Llama 2 13B model on agreeableness behavior as a function of length of the misaligning prompt generated by ℙ−subscriptℙP_-blackboard_P-. Averaged on 5 sequences. (b) KL divergence between ℙ−subscriptℙP_-blackboard_P- and the pretrained model as a function of length of misaligning prompt generated by ℙ−subscriptℙP_-blackboard_P-. Averaged on 10 sequences. Figure 14: (a) Behavior expectation of the Llama 2 13B model on anti-immigration behavior as a function of length of the misaligning prompt generated by ℙ−subscriptℙP_-blackboard_P-. Averaged on 5 sequences. (b) KL divergence between ℙ−subscriptℙP_-blackboard_P- and the pretrained model as a function of length of misaligning prompt generated by ℙ−subscriptℙP_-blackboard_P-. Averaged on 10 sequences. Appendix N β-prompt-distinguishability Here we ran the experiment presented in 4.1 for the definition of β-prompt-distinguishability, i.e., with a neutral prefix that ends with a negative behavior sentence. We used the same setup as before, the only difference is that we sampled from a prompted model ℙ−(⋅|s0)P_-(·|s_0)blackboard_P- ( ⋅ | s0 ) instead of the unprompted model ℙ−⁢(⋅)subscriptℙ⋅P_-(·)blackboard_P- ( ⋅ ). Figure 15 shows that for the neutral part of the prefix, the two models indeed have lower KL divergence than the unprompted case, but once the negative behavior sentence is introduced, the KL rises to a value similar to the unprompted case shown in figure 2. This indicates that negative behavior sentences induce the high KL for the β-prompt-distinguishability. Figure 15: (a) & (b) Examples of conditional KL divergence between two distributions of opposite behaviors as function of prompt length sampled from ℙ−(⋅|s0)P_-(·|s_0)blackboard_P- ( ⋅ | s0 ). Averaged on 10 sampled sequences. For these two specific distributions and prefix, we see that β≈2020β≈ 20β ≈ 20.

Instant research discovery