Paper deep dive

Tradeoffs Between Alignment and Helpfulness in Language Models

Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, Amnon Shashua

Year: 2024Venue: arXiv preprintArea: Representation AnalysisType: TheoreticalEmbeddings: 243

Models: Llama 2 13B Base, Llama 2 13B Chat

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 7:43:46 PM

Summary

This paper investigates the inherent tradeoff between alignment and helpfulness in Large Language Models (LLMs) when using steering methods like representation engineering. The authors propose a theoretical framework demonstrating that while alignment increases linearly with the norm of the injected steering vector, helpfulness decreases quadratically. They identify a regime where steering is efficient for alignment without severely compromising model performance and validate these findings empirically.

Entities (5)

Representation Engineering · method · 100%Steering Methods · method · 98%Alignment · concept · 95%Claude 3 Sonnet · model · 95%Helpfulness · concept · 95%

Relation Signals (3)

Representation Engineering → improves → Alignment

confidence 95% · Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases

Representation Engineering → reduces → Helpfulness

confidence 95% · was also shown to cause a decrease in the ability of the model to perform basic tasks.

Steering Methods → appliedto → Claude 3 Sonnet

confidence 90% · demonstrate SOTA models such as Claude 3 Sonnet can be effectively steered by this method.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibiting undesired ones. It is often done by tuning the model or inserting preset aligning prompts. Recently, representation engineering, a method which alters the model's behavior via changing its representations post-training, was shown to be effective in aligning LLMs (Zou et al., 2023a). Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks. In this paper we study the tradeoff between the increase in alignment and decrease in helpfulness of the model. We propose a theoretical framework which provides bounds for these two quantities, and demonstrate their relevance empirically. First, we find that under the conditions of our framework, alignment can be guaranteed with representation engineering, and at the same time that helpfulness is harmed in the process. Second, we show that helpfulness is harmed quadratically with the norm of the representation engineering vector, while the alignment increases linearly with it, indicating a regime in which it is efficient to use representation engineering. We validate our findings empirically, and chart the boundaries to the usefulness of representation engineering for alignment.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

242,248 characters extracted from source content.

Expand or collapse full text

Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, and Amnon Shashua The Hebrew University yotamwolf,noam.wies,dorin.shteyman,binyamin.rothberg yoav.levine,shashua@cs.huji.ac.il (October 2023) Abstract Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibiting undesired ones. It is often done by tuning the model or inserting preset aligning prompts. Recently, steering methods, such as representation engineering and feature steering and activation steering, methods which alters the model’s behavior via changing its representations post-training, were shown to be effective in aligning LLMs. Steering methods yield gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but were also shown to cause a decrease in the ability of the model to perform basic tasks. In this paper we study the tradeoff between the increase in alignment and decrease in helpfulness of the model. We propose a theoretical framework which provides bounds for these two quantities, and demonstrate their relevance empirically. First, we find that under the conditions of our framework, alignment can be guaranteed with steering methods, and at the same time that helpfulness is harmed in the process. Second, we show that helpfulness is harmed quadratically with the norm of the injected steering vectors, while the alignment increases linearly with it, indicating a regime in which it is efficient to use representation engineering. We validate our findings empirically, and chart the boundaries to the usefulness of these methods for alignment. Our code is available at https://github.com/dorin133/REPE_alignment_helpfulness_tradeoff. 1 Introduction Advancements in large language model (LLM) development over the last few years have given LLMs a variety of abilities that allow them to serve as general purpose assistants in a wide range of tasks, such as broad-scoped question answering, writing assistance, teaching, and more (Radford et al., 2019; Devlin et al., 2019; Brown et al., 2020; Schulman et al., 2023; OpenAI, 2023; Bubeck et al., 2023; Nori et al., 2023; West, 2023; Park et al., 2023). The vast use of LLMs for such purposes has raised concerns due to the harm they can cause their users, such as serving fake information (Lin et al., 2022; Weidinger et al., 2022), behaving offensively, feeding social biases (Hutchinson et al., 2020; Venkit et al., 2022; Weidinger et al., 2022), or encouraging problematic behaviors by users Roose (2023); Atillah (2023). Alignment is often the term given for the process of removing these undesired behaviors (Yudkowsky, 2001; Taylor et al., 2016; Amodei et al., 2016; Shalev-Shwartz et al., 2020; Hendrycks et al., 2021; Pan et al., 2022; Ngo, 2022). There are several different approaches to performing alignment in LLMs, such as including aligning prompts (Askell et al., 2021; Rae et al., 2021) which was shown to improve alignment and decrease toxicity in LLMs, and the procedure of reinforcement learning from human feedback (RLHF) which trains language models to be helpful and harmless (Bai et al., 2022). Though effective to an extent, these approaches are still dangerously frail, as several works have shown that adversarial prompts can trigger negative behaviors in LLMs Wallace et al. (2019); Yu & Sagae (2021); Xu et al. (2021); Subhash (2023); Zou et al. (2023b). The work of Wolf et al. (2023) provides a theoretical framework which shows that frozen LLMs can be misaligned with sufficiently long prompts. Recently, new alignment methods were proposed, revolving around altering model weights at inference time, which control the model at the internal representations level by adding tailored vectors to the hidden layer’s representations. The appeal of such methods is that enhancing concepts through finetuning is expensive and not always efficient for small changes, while inference time steering requires only inference compute and allows to specialize the model to the user’s needs. Prominent methods include representation engineering (Zou et al., 2023a) and activation steering (Turner et al., 2023), in which directions in the model’s latent space controlling certain behaviors are extracted by contrasting hidden representations in which opposing behaviors are exhibited, as well as feature steering, by Anthropic (Templeton, 2024), in which steering vectors are obtained via the use of variational auto-encoders (VAEs), and demonstrate SOTA models such as Claude 3 Sonnet can be effectively steered by this method. While the methods differ in their approach for obtaining the steering vectors, the underlying principle of injecting the vectors into the model is simlar. Since then, there has been an increasing body of work using these methods. Zou et al. (2023a) demonstrated experimentally that the procedure can significantly improve alignment, e.g., in resistance to adversarial attacks, with reduction from 50% success of adversarial attacks to less than 15%, and truthfulness enhancement, with a relative increase of over 50%, though at the cost of somewhat reducing the helpfulness of the model. Wang et al. (2024b) use extracted safety vectors for inference time alignment for harmlessness, reducing jailbreaking success rate from over 30% with prompting and over 10% in supervised fine tuning to below one percent. Similar methods have also been used by Jorgensen et al. (2023); Leong et al. (2023); Liu et al. (2023); Turner et al. (2023) to improve alignment and reduce toxicity. Wang et al. (2024a) uses a method of editing model parameters that maximize the difference between toxic and untoxic responses to detoxify it. Wei et al. (2024) find sparse regions in parameter space that affect alignment brittleness, to be removed for better alignment. Marks et al. (2024) interpret causal graphs in language models and edit them to improve behaviors. van der Weij et al. (2024) extend activation steering to multiple behaviors. To improve low rank finetuning, Wu et al. (2024) utilize a procedure of tuning representations directly to substantially reduce the trainable parameters of finetuning compared to LoRA. Xu et al. (2024); Li et al. (2024) use concept activation vectors to jailbreak, they also observe that concepts that activate different behaviors are linearly separable. Zhang et al. (2024) remove hallucinations by editing truthfulness concepts. Additionally, the method scales to SOTA models, such as Claude 3 Sonnet (Templeton, 2024), using a similar method of sparse auto encoders, which extracts interpretable features from the model that can be used to manipulate the model through steering. There are also known limitations to editing representations - Yan et al. (2024) study limitations of model editing methods for social debiasing, and Elazar et al. (2021) empirically demonstrate how projecting out supervised linear probe directions can reduce performance on selected tasks. Understanding the tradeoff between model helpfulness and alignment is important for designing safe yet useful LLM systems. Previous empirical works have shown tradeoffs between quality and diversity and between helpfulness and safety in LLMs due to instruct finetuning (Florian et al., 2024; Bianchi et al., 2023; Röttger et al., 2023), and reduction in performance due to watermarking (Ajith et al., 2023). In this work we aim to shed light on the benefits and limitations of steering for LLM alignment, i.e., how much does alignment improve with this method and what is the cost in terms of the model’s abilities. We approach this question theoretically at first, and then provide empirical evidence for the validity of our theory. Figure 1: Effect of steering on helpfulness and alignment. Our main results show that alignment can improve at the cost of helpfulness. Moreover, we show that for small representation engineering norms the helpfulness decreases quadratically while the alignment increase is linear, so there is a regime in which representation engineering can be cost-effective. In sections 2 and 3, we set up our theoretical framework and present our theoretical results respectively. We find that steering increases alignment linearly with the steering vector norm (theorem 1), while the helpfulness of the model, defined as the probability of answering general queries correctly, decreases quadratically with the vector norm (theorem 2). Consequently, alignment can be guaranteed with large enough vector injections, though at the cost of reducing the model’s helpfulness. Conversely, when injecting vectors of small norms, the improvement of alignment is initially faster than the decrease in helpfulness, indicating a regime where steering is effective, allowing for inference time alignment while maintaining the model’s helpful capabilities. See figure 1 for an illustration of this intuition. In section 4 we explore the validity of our assumptions and results in an experimental setting with representation engineering: We calculate alignment, as defined by the theoretical framework, as a function of representation engineered vector norms corresponding to the desired behaviors and find that it increases as predicted by theorem 1. This is done by aligning with representation engineering an unaligned (pretrained) model with respect to desired behaviors (“harmless", “not-racist"), and misaligning an aligned (RLHF) model to undesired behaviors (“harmful", “racist"). Then, we calculate the helpfulness of the model, quantified by its question answering abilities over different knowledge domains and coding capabilities, with the same aligning vectors, and find that the decay with increased vector norm described in theorem 2 is manifested. Together, the results correspond to the intuitive illustration in fig. 1. Complementary experimental results by Anthropic showed similar empirical trends for alignment and helpfulness in the use of feature steering with vectors extracted from VAEs on Claude 3 Sonnet (Durmus et al., ). 2 Preliminaries We denote Pθ(⋅|s)P_θ(·|s)Pitalic_θ ( ⋅ | s ) as the next token probability distribution of a model with parameters θ, when conditioned on the prompt s. The model is composed of L layers, rθlsuperscriptsubscriptr_θ^lritalic_θitalic_l is the l’th hidden state representation of the model. The next token prediction of a model is parametrized as: Pθ⁢(tn+1|t1⁢…⁢tn)=s⁢o⁢f⁢t⁢m⁢a⁢x⁢(U⁢rθ(L)⁢(t1⁢…⁢tn))tn+1subscriptconditionalsubscript1subscript1…subscriptsubscriptsubscriptsuperscriptsubscript1…subscriptsubscript1P_θ(t_n+1|t_1...t_n)=softmax(Ur^(L)_θ(t_1...t_n))_t% _n+1Pitalic_θ ( titalic_n + 1 | t1 … titalic_n ) = s o f t m a x ( U r( L )θ ( t1 … titalic_n ) )t start_POSTSUBSCRIPT n + 1 end_POSTSUBSCRIPT (1) Where rθ(L)⁢(s)subscriptsuperscriptr^(L)_θ(s)r( L )θ ( s ) is the final hidden layer’s representation of the prompt s and U is an unembedding matrix from the hidden state to a vocabulary of tokens, a standard parametrization for SOTA LLMs. Denote a steered model by vectors, Re=(re(l=1),…,re(l=L))subscriptsuperscriptsubscript1…superscriptsubscriptR_e=(r_e^(l=1),...,r_e^(l=L))Ritalic_e = ( ritalic_e( l = 1 ) , … , ritalic_e( l = L ) ), as Pθ,resubscriptsubscriptP_θ,r_ePitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT. Steering is performed at each layer by adding the corresponding vector to the hidden layer: rθ(l)←rθ(l)+re(l)←superscriptsubscriptsuperscriptsubscriptsuperscriptsubscriptr_θ^(l)← r_θ^(l)+r_e^(l)ritalic_θ( l ) ← ritalic_θ( l ) + ritalic_e( l ) (2) Additionally, we follow existing methods for steering and provide a uniform norm for all the injected vectors |re(l)|=|re|superscriptsubscriptsubscript|r_e^(l)|=|r_e|| ritalic_e( l ) | = | ritalic_e |, which are initially prepared with norm 1111, and when injected to the model, are multiplied by the coefficient resubscriptr_eritalic_e which can be positive or negative, to tune the steering strength and direction. For layers that are not injected, |re(l)|=0superscriptsubscript0|r_e^(l)|=0| ritalic_e( l ) | = 0. To quantify alignment, we use the behavior expectation definition of alignment as in Wolf et al. (2023), based on the expected score of model responses to a behavior scoring function. The behavior scoring function can measure honesty, safety or any other concept for which responses can be scored as positively or negatively aligned with respect to. We will use a binary scoring function, with labels ±1plus-or-minus1± 1± 1 for aligned/misaligned answers. The results can be extended to more complex behavior scoring function over [−1,+1]11[-1,+1][ - 1 , + 1 ], to yield qualitatively similar results, as discussed appendix J: Definition 1. Let B:Σ∗→−1,+1:→superscriptΣ11B: ^*→\-1,+1\B : Σ∗ → - 1 , + 1 be a binary behavior scoring function, the behavior of a prompted model P(⋅|q)P(·|q)P ( ⋅ | q ) is defined as: B[Pθ(⋅|q)]=a∼Pθ(⋅|q)[B(a)]=∑a+∈a⁢l⁢i⁢g⁢n⁢e⁢dPθ(a+|q)−∑a−∈m⁢i⁢s⁢a⁢l⁢i⁢g⁢n⁢e⁢dPθ(a−|q)B[P_θ(·|q)]=E_a P_θ(·|q)[B(a)]= _a_+% ∈ alignedP_θ(a_+|q)- _a_-∈ misalignedP_θ(a_-|q)B [ Pitalic_θ ( ⋅ | q ) ] = blackboard_Ea ∼ P start_POSTSUBSCRIPT θ ( ⋅ | q ) end_POSTSUBSCRIPT [ B ( a ) ] = ∑a start_POSTSUBSCRIPT + ∈ a l i g n e d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) - ∑a start_POSTSUBSCRIPT - ∈ m i s a l i g n e d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) (3) While B is a binary function, the behavior expectation is in the range [−1,+1]11[-1,+1][ - 1 , + 1 ], reflecting cases where a model has probability for both aligned and misaligned responses. In theorem 1 we will prove that steering is an effective alignment method by lower bounding the behavior expectation. Notice that high probability of outputting a positive/negative response gives a positive/negative contribution to the behavior expectation, thus the sign and absolute value of behavior expectation measures the alignment of a model w.r.t. the given behavior. The model’s helpfulness can be quantified as its ability to produce useful answers to user’s queries (knowledge questions, code generation, summarization, etc.). In order to theoretically analyze helpfulness, we focus on queries where correctness can be defined, such as knowledge based question answering (see figure 1 for an example) and code generation. This can be measured as the likelihood of outputting a correct answer to a query: h⁢e⁢l⁢p⁢f⁢u⁢l⁢n⁢e⁢s⁢s⁢(m⁢o⁢d⁢e⁢l,q)=Pθ⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)ℎsubscriptconditionalsubscripthelpfulness(model,q)=P_θ(a_correct|q)h e l p f u l n e s s ( m o d e l , q ) = Pitalic_θ ( aitalic_c o r r e c t | q ) (4) Where Pθ⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)subscriptconditionalsubscriptP_θ(a_correct|q)Pitalic_θ ( aitalic_c o r r e c t | q ) is the model’s probability of outputting the correct answer a to the query q. By this definition, the helpfulness is in the range [0,1]01[0,1][ 0 , 1 ], in order to quantify the general capabilities of the model when steering vectors are injected into it. For queries where correctness is not defined, the bounds we derive are expected to still be meaningful as they also describe the rate of the model’s deviation from its original distribution due to steering. The rational behind this quantification of alignment and helpfulness is to measure how aligning the model w.r.t. a concept through steering affects its ability to perform other tasks. Ideally, a model that interacts with a user should be both aligned and helpful, meaning its response is appropriate w.r.t. a desired behavior (quantified by a positive behavior expectation) and also useful (high probability of giving a correct answer to general purpose queries). In the next section, we will provide results on alignment and helpfulness under the use of steering, based on the model’s next token prediction, which provides simple analytical forms for alignment and helpfulness. In appendix K, we extend the results for multi-token answers, which yields qualitatively similar results, with somewhat more complex form. 3 Main Results We will show that steering improves alignment and harms helpfulness, yet a "moderate" use of steering can yield a model that is good for both. Theorem 1 shows that behavior expectation is bounded from below by a hyperbolic tangent function, such that it approaches +11+1+ 1 for increasing size of injected vectors and increases linearly within a bounded range. This in principle allows to sample an aligned response for any adversarial attack (corollary 1), demonstrating the power of representation engineering as an alignment technique. Theorem 2 shows that the helpfulness is maximized in the vicinity of norm zero injected vectors (i.e., no representation engineering) and that as the norm is increased, helpfulness decays. The assumptions used to prove the theorems are presented formally in appendix A. The following statement quantifies how alignment is improved by steering. It assumes the injected vectors in all layers accumulate to a change in the last hidden layer representation that classifies positive and negative behavior answers to the query, as depicted in figure 2a. This is assumed due to the popular choice in representation engineering to use steering vectors re(l)superscriptsubscript\r_e^(l)\ ritalic_e( l ) , that are themselves classifiers for positive and negative representations on the intermediate layers, due to being learned from contrasting positive and negative behavior representations for different queries. For example, mean centering, re(l)=g⁢o⁢o⁢d,b⁢a⁢d⁢[rg⁢o⁢o⁢d(l)−rb⁢a⁢d(l)]superscriptsubscriptsubscriptdelimited-[]superscriptsubscriptsuperscriptsubscriptr_e^(l)=E_good,bad[r_good^(l)-r_bad^(l)]ritalic_e( l ) = blackboard_Eg o o d , b a d [ ritalic_g o o d( l ) - ritalic_b a d( l ) ] (Jorgensen et al. (2023)), or PCA, re(l)=argmaxv:‖v‖=1⁢[g⁢o⁢o⁢d,b⁢a⁢d⁢|⟨v,rg⁢o⁢o⁢d(l)−rb⁢a⁢d(l)⟩|2]superscriptsubscriptsubscriptargmax:norm1delimited-[]subscriptsuperscriptsuperscriptsubscriptsuperscriptsubscript2r_e^(l)=argmax_v:||v||=1[E_good,bad| v,r_good% ^(l)-r_bad^(l) |^2]ritalic_e( l ) = argmaxv : | | v | | = 1 [ blackboard_Eg o o d , b a d | ⟨ v , ritalic_g o o d( l ) - ritalic_b a d( l ) ⟩ |2 ] (Zou et al. (2023a)), such that they form linear classifiers for the intermediate layers due to the positive/negative inner product with positive/negative answer representations. Notably, in Xu et al. (2024) it is shown empirically that such concept classes in latent space are linearly separable. We discuss this assumption further in A and provide empirical evidence. Furthermore, the classification condition can be softened to an imperfect classifier, as discussed in appendix A and shown in appendix in H, to yield similar results. Theorem 1. Let Pθ,re(⋅|q)P_θ,r_e(·|q)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) be a model prompted with query q and injected with representations of coefficient resubscriptr_eritalic_e. Let B:Σ∗→−1,+1:→superscriptΣ11B: ^*→\-1,+1\B : Σ∗ → - 1 , + 1 be a behavior scoring function. The injections to all layers amounts to a change in the final hidden layer representation that is q dependent, denoted by the vector δ⁢re(L)⁢(q)subscriptsuperscriptδ r^(L)_e(q)δ r( L )e ( q ). Assume the representations of aligned and misaligned answers w.r.t. B are linearly separable, and δ⁢re(L)⁢(q)subscriptsuperscriptδ r^(L)_e(q)δ r( L )e ( q ) linearly classifies them with margin Δ Δ. Then, the behavior expectation of the model conditioned on the query q satisfies: B[Pθ,re(⋅|q)]≥tanh(Δλ⋅re+arctanh(B0))B[P_θ,r_e(·|q)]≥ tanh( λ· r_e+arctanh(B_0))B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] ≥ t a n h ( Δ λ ⋅ ritalic_e + a r c t a n h ( B0 ) ) (5) Where B0=B[Pθ(⋅|q)]B_0=B[P_θ(·|q)]B0 = B [ Pitalic_θ ( ⋅ | q ) ] is the behavior expectation without steering and λ is a model dependent coefficient relating between resubscriptr_eritalic_e and the corresponding final hidden state norm. As can be seen in the mathematical expression and in figure 2b for B0=−0.5subscript00.5B_0=-0.5B0 = - 0.5, this lower bound is a shifted hyperbolic tangent function w.r.t resubscriptr_eritalic_e. At re=0subscript0r_e=0ritalic_e = 0 the bound gives B0subscript0B_0B0, which is the unaltered model’s behavior. As resubscriptr_eritalic_e is increased, the bound approaches +11+1+ 1, meaning the behavior asymptotically approaches +11+1+ 1. We see that for B0subscript0B_0B0 that is not too close to −11-1- 1, the increase in behavior expectation is linear due to the hyperbolic tangent’s nature, while if it is very close to −11-1- 1, resubscriptr_eritalic_e is to be increased before seeing the linear effect. Thus for behaviors on which the model is negative but has a small tendency for positive answers, the linear effect should be felt near re=0subscript0r_e=0ritalic_e = 0. In section 4, we present our numerical estimation Δ⁢λΔ λΔ λ in the range 0.1−30.130.1-30.1 - 3, both based on the linear classifier condition and direct alignment measurement. For proof see appendix section B. Figure 2: (a) The change to the last hidden layer due to vector injections from previous layers classifies positive and negative answer representations. (b) Plot of the upper bound on behavior expectation in theorem 1. This can be extended to multi-token answers, by enforcing the above result on each decoding step of the generated answer, as explained in appendix K. The binary behavior score can also be extended beyond binary, as explained in appendix J. In contrast to Wolf et al. (2023), whose framework is centralized on using prompts to misalign frozen models, i.e. whose weights and representations are not changed after training, here the model is not frozen due to steering, and accordingly a different result is obtained on guaranteeing an aligned response – for any adversarial attack, using steering with large enough norms produces an aligned response if the learned steering vectors accumulate to a good classifier of positive and negative answer representations in the final layer. We formalize this in appendix D. Now, we shall bound from above the helpfulness of the model as a function of steering. We formally bound the probability of producing correct answers to queries where correctness is well defined. Yet, even when this is not the case, the bound can still be relevant, as it quantifies the model’s deviation from its original distribution due to steering. Hence if the model was initially helpful on a task, a random deviation to its probability distribution is expected to decrease model performance proportionally to the size of the deviation. Intuitively, editing the model’s representation in a specific direction adds random noise to other latent concepts of the model, causing a degradation in its other capabilities. This is introduced in our framework through the resulting change to the final hidden layer δ⁢re⁢(q)=rθ,re(L)−rθ(L)subscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptδ r_e(q)=r_θ,r_e^(L)-r_θ^(L)δ ritalic_e ( q ) = ritalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT( L ) - ritalic_θ( L ), we will assume its direction δ⁢re⁢(q)|δ⁢re⁢(q)|subscriptsubscript δ r_e(q)|δ r_e(q)|divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG contains random projections w.r.t. latent representations of correct and incorrect answers, which creates noise in the model’s distribution. The noise is expected to be random on the highest probability tokens, when answering a query that is unrelated to the behavior being enhanced (intuitively depicted in figure 3a). We verify this empirically in appendix A.3. Thus, we assume random noise on the top T tokens making up a large probability mass of the answer distribution, 1−ϵ1italic-ϵ1- 1 - ϵ, (e.g. T∼10similar-to10T 10T ∼ 10 typically makes ϵ∼0.1similar-toitalic-ϵ0.1ε 0.1ϵ ∼ 0.1), and do not make assumptions on the rest of the vocabulary. The following theorem formally states this. Theorem 2. Let Pθ,re(⋅|q)P_θ,r_e(·|q)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) be a model prompted with query q and injected with representations of coefficient resubscriptr_eritalic_e. If the resulting change to the directionality of the last hidden layer representation due to the injections in all layers, distributes randomly with variance σ2>0superscript20σ^2>0σ2 > 0 w.r.t. the representations of correct and incorrect answers making up 1−ϵ1italic-ϵ1- 1 - ϵ of the probability mass, the helpfulness of the model on the query is bounded with probability 1−2T121- 2T1 - divide start_ARG 2 end_ARG start_ARG T end_ARG by: Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)≤P0P0+(1−P0)⋅α⁢(1−ϵ)⁢(1+λ2⁢σ2⁢β22⁢re2)subscriptsubscriptconditionalsubscriptsubscript0subscript0⋅1subscript01italic-ϵ1superscript2superscript2superscript22superscriptsubscript2P_θ,r_e(a_correct|q)≤ P_0P_0+(1-P_0)·α(1-% ε)(1+ λ^2σ^2β^22r_e^2)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) ≤ divide start_ARG P0 end_ARG start_ARG P0 + ( 1 - P0 ) ⋅ α ( 1 - ϵ ) ( 1 + divide start_ARG λ2 σ2 β2 end_ARG start_ARG 2 end_ARG ritalic_e2 ) end_ARG (6) Where P0=Pθ,re=0(⋅|q)P_0=P_θ,r_e=0(·|q)P0 = Pitalic_θ , r start_POSTSUBSCRIPT e = 0 end_POSTSUBSCRIPT ( ⋅ | q ) is the probability of answering correctly without steering, T is the number of tokens making 1−ϵ1italic-ϵ1- 1 - ϵ of the probability mass and α,β>00α,β>0α , β > 0 that depend on the query. λ is a model dependent coefficient relating between resubscriptr_eritalic_e and the corresponding final hidden state norm. The proof is presented in appendix C and the assumption formally defined in appendix A. The above bound is illustrated in figure 3b for different values of β. As can be seen, around re=0subscript0r_e=0ritalic_e = 0, the bound is parabolic, i.e. the decrease is proportional to −re2superscriptsubscript2-r_e^2- ritalic_e2, obtained by expanding the bound near re=0subscript0r_e=0ritalic_e = 0. On the other hand, for large resubscriptr_eritalic_e, we see a decay to zero at a rate proportional to re−2superscriptsubscript2r_e^-2ritalic_e- 2, obtained by expanding the bound for large resubscriptr_eritalic_e. This result can be extended to multi-token answers, by enforcing the above result on each decoding step of the generated answer, as explained in appendix K. Importantly, this demonstrates that while large vector injections harm the model’s overall performance, for small injections, the model’s performance is relatively unharmed due to the slow (parabolic) decrease with norm around re=0subscript0r_e=0ritalic_e = 0. For the second statement to be feasible, the true helpfulness and the bound need to be close when no steering is performed. Indeed, the difference between the two at re=0subscript0r_e=0ritalic_e = 0 is bounded by 1−P01subscript01-P_01 - P0, such that for queries with high probability of being answered correctly without steering, i.e. P0≈1subscript01P_0≈ 1P0 ≈ 1, the true helpfulness and the bound will be close, guaranteeing the parabolic bound to be meaningful. The parameter α∈[0,1]01α∈[0,1]α ∈ [ 0 , 1 ] measures the tightness of the bound at re=0subscript0r_e=0ritalic_e = 0, since the true helpfulness at re=0subscript0r_e=0ritalic_e = 0 is P0subscript0P_0P0, while our helpfulness bound is P0P0+α⁢(1−P0)subscript0subscript01subscript0 P_0P_0+α(1-P_0)divide start_ARG P0 end_ARG start_ARG P0 + α ( 1 - P0 ) end_ARG. Thus α=11α=1α = 1 (and ϵ=0italic-ϵ0ε=0ϵ = 0) means the bound at re=0subscript0r_e=0ritalic_e = 0 coincides with the true helpfulness, while smaller α means the bound overshoots it. In our results, we obtain α≤0.50.5α≤ 0.5α ≤ 0.5. Figure 3 depicts this overshooting for α=0.250.25α=0.25α = 0.25. Even so, as explained above, the tightness is at least 1−P01subscript01-P_01 - P0 regardless of α, so it is always meaningful for queries the model is initially helpful on. The product of parameters λ⁢σ⁢βλσβλ σ β measures the rate/curvature of the quadratic decay, as they are the coefficient multiplying re2superscriptsubscript2r_e^2ritalic_e2. λ is the same scaling parameter from theorem 1, σ is the standard deviation of random noise added to the logits due to representation engineering (depicted in figure 3a and formally defined in A). β is the minimum between two weighted sums of positive variables with parameter σ′=1superscript′1σ =1σ′ = 1. In section 4, we present an empirical estimation for λ⁢σ⁢βλσβλ σ β in the range 0.1−0.660.10.660.1-0.660.1 - 0.66, based on the logit noise condition and direct helpfulness measurement. Hence the decay becomes strong at coefficients resubscriptr_eritalic_e of size 1−101101-101 - 10. Figure 3: (a) Directionality of change to last hidden layer due to representation engineering distributes randomly with variance σ2superscript2σ^2σ2 w.r.t. correct and incorrect answer representations. (b) Plot of helpfulness bound with given parameters of P0subscript0P_0P0, α and λ⁢σ⁢βλσβλ σ β. A tradeoff between alignment and usefulness: The combination of the two results shows alignment improves linearly with the norm of the steering vectors while helpfulness is decreased quadratically. This means that when injecting vectors of small norms, the improvement of alignment is initially faster than the decrease in helpfulness, indicating a regime where steering is more effective. See figure 1 for an illustration of this intuition. 4 Empirical results Here we will calculate alignment and helpfulness as defined above and observe how they are affected by increasing norms of steering vectors. Theorem 1 shows how alignment can increase/decrease due to steering, thus to demonstrate it, we increase the alignment of an unaligned pretrained model w.r.t. “harmless" and “not-racist" behaviors (specifically we use Llama 2 13B (Touvron et al., 2023)), and conversely, misalign an aligned RLHF model w.r.t. “harmful" and “racist" behaviors (Llama 2 13B chat (Touvron et al., 2023)). Then, we calculate helpfulness as the probability of answering queries correctly when the model is injected with the same behavior altering vectors. The experiments show an effect on alignment matching theorem 1 and on helpfulenss matching theorem 2. Additional experimental details can be found in appendix F as well as results for Llama 3.1 8B (Dubey et al., 2024). We note the goal of the experiments is to demonstrate the theoretical bounds showing an enhancement of alignment with a concept and a helpfulness decrease due to steering, and that a complementary experimental demonstration of these with more behaviors is shown on Claude 3 Sonnet with social biases when using feature steering (Durmus et al., ). We follow the work of Zou et al. (2023a) to extract the vectors used in representation engineering: Pairs of positive and negative statements w.r.t. a behavior, are forward passed through the model, and the differences between representations of the pairs are used to find latent space directions that steer the model’s responses from negative to positive behaviors or vice versa. For the “harmful" behavior on the aligned model, we extracted harmful and unharmful instructions from AdvBench (Robey et al., 2021; 2022) and shareGPT respectively. For “harmless" behavior on the unaligned model, the approach of contrasting positive and negative requests does not work, as the model agrees to answer both types of requests, so contrasting them does not steer the model towards not answering a request. Instead, inspired by the method of preference learning, we contrast aligned and misaligned responses to harmful instructions from AdvBench. For “racism" on the aligned model, we used biased and unbiased statements from the StereoSet dataset (Nadeem et al., 2020). For “not-racist" on the unaligned model, we used the racist statements from above, followed by aligned and misaligned responses. The obtained vectors were used to calculate behavior expectation and helpfulness of the model as the norm of the vectors increased. Alignment Measurement: To calculate harmful behavior expectation, we sampled full responses to harmful instructions and used the behavior scoring function that assigns an answer B⁢(a⁢n⁢s⁢w⁢e⁢r)=±1plus-or-minus1B(answer)=± 1B ( a n s w e r ) = ± 1 if the model answers a harmful instruction or refuses to and calculated its expectation value, which is the difference between probabilities of fulfilling and not fulfilling the instruction. To calculate the racism behavior expectation, sampled full responses to racist statements and used a behavior scoring function that assigns an answer B⁢(a⁢n⁢s⁢w⁢e⁢r)=±1plus-or-minus1B(answer)=± 1B ( a n s w e r ) = ± 1 to agreeing/disagreeing with a racist statement, and calculated the expectation value of this function w.r.t. the model distribution, which is the difference in probabilities of agreeing and disagreeing with a racist statement. Figure 4: Plots of behavior expectation as a function of the coefficients of representation engineering vectors injected to the model. The blue line is the direct measurement, the orange line is a plot of the bound from theorem 1. (a) Harmless behavior expectation of Llama 2 13B as a function of coefficient of injected harmful PCA vectors. (b) Racism behavior expectation of Llama 2 13B as a function of coefficient of injected bias PCA vectors.(c) Harmful behavior expectation of Llama 2 13B as a function of coefficient of injected harmful PCA vectors. (d) Racism behavior expectation of Llama 2 13B chat as a function of coefficient of injected bias PCA vectors. Figure 4 shows behavior expectation as a function of corresponding PCA vector coefficients injected into the models. Overall we see that on both behaviors and both models, the behavior expectation changes like a hyperbolic tangent, as expected of theorem 1, which can be seen by the fitted curve of the data to a bound of the form of theorem 1 when using Δ⁢λΔ λΔ λ as a free parameter that fits the measurements. The value of Δ⁢λΔ λΔ λ corresponding to the curve is 0.5−30.530.5-30.5 - 3 while our empirically estimated value of Δ⁢λΔ λΔ λ from the data based on the linear classification condition of the last hidden layer change is 0.1−0.40.10.40.1-0.40.1 - 0.4 (for details and explanation for these differences see appendix A.3). We note that for all behaviors, re=2.5subscript2.5r_e=2.5ritalic_e = 2.5 suffices for a significant change in behavior expectation, taking it from negative to positive. It is left to observe the decrease in helpfulness and verify that it is not too big. Helpfulness Measurement: To calculate helpfulness, we tested the model on two tasks. The first is knowledge based question answering, for a clean test of the single token theoretical results (theorem 2). The second is code generation, to verify the single token results persist for tasks with multiple-token answers. Importantly, we injected the model with the same vectors used to alter the model’s behavior in the alignment measurement. For the first task, we queried the model with multiple choice questions from the MMLU dataset (Hendrycks et al., 2020) over a variety of domains (e.g. international law, medical genetics) and calculated the probability that the model assigns the correct answer. This was done both by calculating the probabilities of the multiple choice answers, A,B,C,D, and in appendix F by sampling full responses to the questions and measuring the accuracy, yielding similar results. This was measured as a function of injected vector coefficients inserted to the model for the behaviors above. Figure 5 shows the results for the different behaviors and models. We plot a bound of the form of theorem 2 to demonstrate the predicted parabolic behavior. We do so with free parameter λ⁢σ⁢βλσβλ σ β from which we find λ⁢σ⁢βλσβλ σ β in the range of 0.330.330.330.33 to 0.660.660.660.66 (see appendix F.4). This is in accordance with our empirically estimated values of 0.10.10.10.1 to 0.40.40.40.4 for λ⁢σ⁢βλσβλ σ β from direct measurement of the noise injected to the model due to representation engineering in appendix A.3. Notably, for re=2.5subscript2.5r_e=2.5ritalic_e = 2.5, the decrease in helpfulness is still not too great, while as mentioned previously, alignment is significantly increased. Figure 5: Helpfulness measurement: the probability assigned to the correct answer to questions from different MMLU tests (international law, medical genetics, high school computer science), as a function of representation engineering vector coefficients injected to the model. Here the probability of the correct answer was measured relative to the answers A, B, C, D. The red line plots the bound of theorem 2 for free parameters on “international law". (a) Helpfulness of Llama 2 13B with harmful PCA vectors. (b) Helpfulness of Llama 2 13B with bias PCA vectors. (c) Helpfulness of Llama 2 13B chat with harmful PCA vectors. (d) Helpfulness of Llama 2 13B chat with bias PCA vectors. For the second task, we tested the model’s coding skills with the humaneval dataset (Chen et al., 2021). We present the results in appendix G. The model’s performance is peaked around re=0subscript0r_e=0ritalic_e = 0, and it decays parabolically ar resubscriptr_eritalic_e increases, as predicted in theorem 2. 5 Discussion In this work, we study the benefits of steering methods for LLM alignment from a theoretical perspective. We find that increasing the magnitude of the vectors injected to the model leads to improved alignment; we theoretically quantify this improvement as linear in the vectors’ magnitude, and validate our result empirically. A practical outcome of our result is a guarantee of alignment when using the representation engineering method. Such theoretical guarantees cannot be made without altering the model at inference time – Wolf et al. (2023) show that prompt based alignment methods can always be undone. Our result thus crystallizes an inherent advantage of steering over competing alignment methods. On the other hand, our framework indicates a degradation of the model’s general capabilities when steering is applied. We theoretically quantify this degradation to be parabolic in the injected vectors’ magnitude, which puts a bound on the strength with which steering should be performed to keep the model reliable for different uses. While our theoretical bound is an upper bound on the helpfulness, we observe this parabolic behavior empirically as well. While steering is an emerging field, editing interpretable features of models on the representation level in order to control them scales to SOTA models such as Anthropic’s Claude 3 Sonnet (Templeton, 2024; Durmus et al., ). In principle, our framework may be generalized for theoretically analyzing the effects of normal finetuneing on alignment and helpfulness, as it too amounts to a change in the LLM representations to maximize the likelihood of desired outputs. In particular, each step in preference learning is equivalent to steering with coefficient that equals to the learning rate (see appendix I), and indeed similar tradeoffs have been observed for finetuning (Tan et al., 2024). However, we leave this for future work, as finetuning creates small changes to the model’s representation at each training step on several behaviors, that sums to a large overall change, while steering takes a large step in one direction. As a result, the change to the representations in a steering process on one behavior creates random noise on the others (assumption 3), unlike a finetuning process where this does not necessarily happen. Hence in regards of maintaining helpfulness, finetuning has an advantage, however, steering does enjoy the benefit of an online controllable step size in the desired behavior for effective manipulation at inference time. Overall, we hope that our theoretical work will shed light on the mechanism of steering, which constitutes a new interesting direction for language model alignment. Acknowledgments This research was supported by the ERC (European Research Council) and the ISF (Israel Science Foundation). References Ajith et al. (2023) Anirudh Ajith, Sameer Singh, and Danish Pruthi. Performance trade-offs of watermarking large language models. arXiv preprint arXiv:2311.09816, 2023. Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. Atillah (2023) Imane El Atillah. Man ends his life after an ai chatbot ’encouraged’ him to sacrifice himself to stop climate change. Euronews, 2023. Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Bianchi et al. (2023) Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875, 2023. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021. Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. (12) Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, et al. Evaluating feature steering: A case study in mitigating social biases, 2024. URL https://anthropic. com/research/evaluating-feature-steering. Elazar et al. (2021) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021. Florian et al. (2024) Le Bronnec Florian, Verine Alexandre, Negrevergne Benjamin, Chevaleyre Yann, and Allauzen Alexandre. Exploring precision and recall to assess the quality and diversity of llms. arXiv preprint arXiv:2402.10693, 2024. Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. Hendrycks et al. (2021) Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021. Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 5491–5501, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.487. URL https://aclanthology.org/2020.acl-main.487. Jorgensen et al. (2023) Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813, 2023. Leong et al. (2023) Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. Self-detoxifying language models via toxification reversal. arXiv preprint arXiv:2310.09573, 2023. Li et al. (2024) Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. Open the pandora’s box of llms: Jailbreaking llms through representation engineering. arXiv preprint arXiv:2401.06824, 2024. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229. Liu et al. (2023) Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997, 2023. Marks et al. (2024) Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024. Nadeem et al. (2020) Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models, 2020. Ngo (2022) Richard Ngo. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022. Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. OpenAI (2023) OpenAI. Gpt-4 technical report, 2023. Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye. Park et al. (2023) Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. Robey et al. (2021) Alexander Robey, Luiz Chamon, George J Pappas, Hamed Hassani, and Alejandro Ribeiro. Adversarial robustness with semi-infinite constrained learning. Advances in Neural Information Processing Systems, 34:6198–6215, 2021. Robey et al. (2022) Alexander Robey, Luiz Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average and worst-case performance. In International Conference on Machine Learning, p. 18667–18686. PMLR, 2022. Roose (2023) Kevin Roose. A conversation with bing’s chatbot left me deeply unsettled. New York Times, 2023. Röttger et al. (2023) Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023. Schulman et al. (2023) John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe, Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes, Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, Joel Parish, Dave Cummings, Rajeev Nayak, Valerie Balcom, David Schnurr, Tomer Kaftan, Chris Hallacy, Nicholas Turley, Noah Deutsch, Vik Goel, Jonathan Ward, Aris Konstantinidis, Wojciech Zaremba, Long Ouyang, Leonard Bogdonoff, Joshua Gross, David Medina, Sarah Yoo, Teddy Lee, Ryan Lowe, Dan Mossing, Joost Huizinga, Roger Jiang, Carroll Wainwright, Diogo Almeida, Steph Lin, Marvin Zhang, Kai Xiao, Katarina Slama, Steven Bills, Alex Gray, Jan Leike, Jakub Pachocki, Phil Tillet, Shantanu Jain, Greg Brockman, Nick Ryder, Alex Paino, Qiming Yuan, Clemens Winter, Ben Wang, Mo Bavarian, Igor Babuschkin, Szymon Sidor, Ingmar Kanitscheider, Mikhail Pavlov, Matthias Plappert, Nik Tezak, Heewoo Jun, William Zhuk, Vitchyr Pong, Lukasz Kaiser, Jerry Tworek, Andrew Carr, Lilian Weng, Sandhini Agarwal, Karl Cobbe, Vineet Kosaraju, Alethea Power, Stanislas Polu, Jesse Han, Raul Puri, Shawn Jain, Benjamin Chess, Christian Gibson, Oleg Boiko, Emy Parparita, Amin Tootoonchian, Kyle Kosic, and Christopher Hesse. Introducing chatgpt. OpenAI blog, 2023. Shalev-Shwartz et al. (2020) Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. On the ethics of building ai in a responsible manner. arXiv preprint arXiv:2004.04644, 2020. Subhash (2023) Varshini Subhash. Can large language models change user preference adversarially? arXiv preprint arXiv:2302.10291, 2023. Tan et al. (2024) Yingshui Tan, Yanshi li, Xiaoyong Zhu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, and Zheng Bo. Truly safe & truly helpful: Achieving harmonious balance for large language model. Openreview, 2024. Taylor et al. (2016) Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems. Ethics of Artificial Intelligence, p. 342–382, 2016. Templeton (2024) Adly Templeton. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic, 2024. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Turner et al. (2023) Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023. van der Weij et al. (2024) Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. arXiv preprint arXiv:2403.05767, 2024. Venkit et al. (2022) Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. A study of implicit bias in pretrained language models against people with disabilities. In Proceedings of the 29th International Conference on Computational Linguistics, p. 1324–1332, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclanthology.org/2022.coling-1.113. Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 2153–2162, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1221. URL https://aclanthology.org/D19-1221. Wang et al. (2024a) Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. arXiv preprint arXiv:2403.14472, 2024a. Wang et al. (2024b) Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu. Inferaligner: Inference-time alignment for harmlessness through cross-model guidance. arXiv preprint arXiv:2401.11206, 2024b. Wei et al. (2024) Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162, 2024. Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, p. 214–229, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/3531146.3533088. URL https://doi.org/10.1145/3531146.3533088. West (2023) Colin G West. Advances in apparent conceptual physics reasoning in gpt-4. arXiv e-prints, p. arXiv–2303, 2023. Wolf et al. (2023) Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082, 2023. Wu et al. (2024) Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine-tuning via representation editing. arXiv preprint arXiv:2402.15179, 2024. Xu et al. (2021) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 2950–2968, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.235. URL https://aclanthology.org/2021.naacl-main.235. Xu et al. (2024) Zhihao Xu, Ruixuan Huang, Xiting Wang, Fangzhao Wu, Jing Yao, and Xing Xie. Uncovering safety risks in open-source llms through concept activation vector. arXiv preprint arXiv:2404.12038, 2024. Yan et al. (2024) Jianhao Yan, Futing Wang, Yafu Li, and Yue Zhang. Potential and challenges of model editing for social debiasing. arXiv preprint arXiv:2402.13462, 2024. Yu & Sagae (2021) Dian Yu and Kenji Sagae. Automatically exposing problems with neural dialog models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 456–470, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.37. URL https://aclanthology.org/2021.emnlp-main.37. Yudkowsky (2001) Eliezer Yudkowsky. Creating friendly ai 1.0: The analysis and design of benevolent goal architectures. The Singularity Institute, San Francisco, USA, 2001. Zhang et al. (2024) Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large language models in truthful space. arXiv preprint arXiv:2402.17811, 2024. Zou et al. (2023a) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023a. Zou et al. (2023b) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023b. Appendix A Assumptions In A.1 we introduce our assumptions used in proving theorems 1 and 2. We discuss them in A.2 and provide experiments to check their validity in A.3 A.1 Introduction of assumptions We assume that for small coefficients of representation steering resubscriptr_eritalic_e, the norm of the change to the last hidden layer representation is linear in resubscriptr_eritalic_e: Assumption 1. Let Pθ,re(⋅|q)P_θ,r_e(·|q)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) be a language model prompted with query q. The change to the last hidden layer representation due to steering with coefficient resubscriptr_eritalic_e, denoted by δ⁢re⁢(q)=r(L)⁢(q,re)−r(L)⁢(q,0)subscriptsuperscriptsubscriptsuperscript0δ r_e(q)=r^(L)(q,r_e)-r^(L)(q,0)δ ritalic_e ( q ) = r( L ) ( q , ritalic_e ) - r( L ) ( q , 0 ) satisfies: |δ⁢re⁢(q)|=λ⁢|re|subscriptsubscript|δ r_e(q)|=λ|r_e|| δ ritalic_e ( q ) | = λ | ritalic_e | (7) For a constant λ>00λ>0λ > 0 that is query dependent. It is used in theorems 1 and 2, to relate the change to the last hidden layer to the coefficients of injected representations. A representation of an answer to a query is defined as the latent space embedding of the answer’s token, UT⁢et⁢o⁢k⁢e⁢nsuperscriptsubscriptU^Te_tokenUitalic_T eitalic_t o k e n, where eisubscripte_ieitalic_i is the one-hot vector of the token i and U is the matrix from the last layer’s hidden dimension to the vocabulary. We assume that the representations of positive and negative answers to a query are linearly separable, and that the change to the last hidden layer of the model due to representation engineering linearly classifies them with margin Δ Δ: Assumption 2. Given a query q, the change to the last hidden layer of a model due to steering, δ⁢re⁢(q)=r(L)⁢(q,re)−r(L)⁢(q,0)subscriptsuperscriptsubscriptsuperscript0δ r_e(q)=r^(L)(q,r_e)-r^(L)(q,0)δ ritalic_e ( q ) = r( L ) ( q , ritalic_e ) - r( L ) ( q , 0 ) , linearly classifies the representations of positive and negative answers to a query q with margin Δ Δ, where the positive and negative answers are defined with respect to a behavior scoring function B:Σ→−1,+1:→Σ11B: →\-1,+1\B : Σ → - 1 , + 1 : m⁢i⁢ni:B⁢(i)>0,j:B⁢(j)<0⁢⟨δ⁢re⁢(q)|δ⁢re⁢(q)|,UT⁢ei−UT⁢ej⟩>Δsubscript:0:0subscriptsubscriptsuperscriptsubscriptsuperscriptsubscriptΔmin_i:B(i)>0,j:B(j)<0 \ δ r_e(q)|δ r_% e(q)|,U^Te_i-U^Te_j \> i nitalic_i : B ( i ) > 0 , j : B ( j ) < 0 ⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG , Uitalic_T eitalic_i - Uitalic_T eitalic_j ⟩ > Δ (8) That is to say, that on the axis defined by δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ), positive and negative representations can be separated, and the minimal distance between representations of positive and negative answers on it is Δ Δ. It is used in theorm 1, to obtain that the probability of the aligned answers increases w.r.t.the misaligned answers as the coefficients of the injected representations increases. Note that the above assumption can be relaxed from a hard margin to a soft margin assumption, where δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) classifies the representations of positive and negative answers, but part of the misaligned/aligned answers’ representations are misclassified as aligned/misaligned. This yields similar results to theorem 1 that are shown in appendix H. For queries whose topic is unrelated to the behavior with respect to which steering is performed, we expect the change to the last layer representation to be somewhat random on the highest probability tokens as they answer a question that is unrelated to the behavior whose vectors are injected to the model. Intuitively, the change to the final layer representation has no preference for a correct token over an incorrect token, so an incorrect answer is just as likely to be on one side or the other of the plane defined by the vertical δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) that passes through the correct answer representation. Assumption 3. When sampling an answer to a query q that is unrelated to the behavior of steering, the vector δ⁢re⁢(q)=r(L)⁢(q,re)−r(L)⁢(q,0)subscriptsuperscriptsubscriptsuperscript0δ r_e(q)=r^(L)(q,r_e)-r^(L)(q,0)δ ritalic_e ( q ) = r( L ) ( q , ritalic_e ) - r( L ) ( q , 0 ), i.e., the resulting change to the last hidden layer representation due to the steering vectors from all layers, is random with the following coordinate-wise distribution on the T highest probability tokens making 1−ϵ1italic-ϵ1- 1 - ϵ of the probability mass: ⟨δ⁢re⁢(q)|δ⁢re⁢(q)|,UT⁢ei⟩∼Dsimilar-tosubscriptsubscriptsuperscriptsubscript δ r_e(q)|δ r_e(q)|,U^Te_i D⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG , Uitalic_T eitalic_i ⟩ ∼ D (9) Where D is some continuous distribution with variance σ2>0superscript20σ^2>0σ2 > 0. This defines a random directionality of δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) w.r.t. the representations of answers. It is used in theorem 2 to formalize that steering is a “perpendicular" direction to the query’s relevant answer representations. A.2 Discussion of assumptions Linear last hidden layer change (assumption 1): Intuitively, when adding vectors of relatively small norms to each layer, the first order Taylor expansion with respect to the vectors is good, and it scales linearly with the coefficients of the vectors. We observe experimentally in subsection A.3 that for small coefficients, the change is indeed approximately linear. Note that it suffices to assume |δ⁢re⁢(q)|subscript|δ r_e(q)|| δ ritalic_e ( q ) | grows monotonically with |re|subscript|r_e|| ritalic_e |, but for simplicity and due to experimental observations we assume the linear dependence. Linear classification with margin Δ Δ (assumption 2): We expect the representation engineered vectors resubscriptr_eritalic_e to be good classifiers because they are obtained by methods of finding directions in the latent space that maximize the distance between representations of positive and negative textual statements. For example, in Zou et al. (2023a) the first principle component is used as a steering vector, obtained via p⁢c⁢a1=a⁢r⁢g⁢m⁢a⁢xv⁢g⁢o⁢o⁢d,b⁢a⁢d⁢[|⟨v,rg⁢o⁢o⁢d−rb⁢a⁢d⟩|2]subscript1subscriptsubscriptdelimited-[]superscriptsubscriptsubscript2pca_1=argmax_v\E_good,bad[| v,r_good-r_bad |^% 2]\p c a1 = a r g m a xitalic_v blackboard_Eg o o d , b a d [ | ⟨ v , ritalic_g o o d - ritalic_b a d ⟩ |2 ] and in Jorgensen et al. (2023) the steering vector is obtained as the average of difference between positive and negative statements 1N⁢∑i=1N(rg⁢o⁢o⁢di−rb⁢a⁢di)1superscriptsubscript1subscriptsuperscriptsubscriptsuperscript 1N _i=1^N(r^i_good-r^i_bad)divide start_ARG 1 end_ARG start_ARG N end_ARG ∑i = 1N ( ritalic_iitalic_g o o d - ritalic_iitalic_b a d ). In these examples, rg⁢o⁢o⁢dsubscriptr_goodritalic_g o o d and rb⁢a⁢dsubscriptr_badritalic_b a d are representations of queries and not the latent space embedding of the answers, as in the definition of Δ Δ-representation-separability, but we expect the steering vectors to behave similarly on them. In subsection A.3, we show that indeed δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) clusters positive and negative responses to harmful queries in the model’s latent space. In appendix H we also formulate a theorem equivalent to theorem 1, but with an imperfect classifier. Random directionality of last hidden layer change (assumption 3): When answering queries that are unrelated to the behavior being enhanced by steering, the directionality of the injected vectors are expected to be random w.r.t. the representations of the answers to the query. Therefore, the highest probability tokens are expected to be injected with random noise. We validate this in the next subsection, by looking at the noise injected into the top 10 highest probability tokens in knowledge queries (which typically make over 90% of the probability mass). A.3 Experiments for assumptions Here we empirically check the validity of our assumptions and empirically estimate the values of the parameters in the bounds. The experiments were performed on Llama 2 13B and Llama 2 13B chat. We first verify a linear relation between the steering vector coefficient resubscriptr_eritalic_e to the last hidden layer change of assumption 1, which yields λ. Then, we verify the normal distribution assumption 3 and the linear classification of assumption 2. Norm of final hidden layer change is linear in injected vectors For a query q we define δ⁢re⁢(q)=r(L)⁢(q,re)−r(L)⁢(q,0)subscriptsuperscriptsubscriptsuperscript0δ r_e(q)=r^(L)(q,r_e)-r^(L)(q,0)δ ritalic_e ( q ) = r( L ) ( q , ritalic_e ) - r( L ) ( q , 0 ) as the change of the representation of the query in the final layer. where r(L)⁢(q,0)superscript0r^(L)(q,0)r( L ) ( q , 0 ) is the representation if we injected no vector (the default model representation) and r(L)⁢(q,re)superscriptsubscriptr^(L)(q,r_e)r( L ) ( q , ritalic_e ) is the representation given that we inject a vector of norm resubscriptr_eritalic_e at a range of layers. We show that the norm of δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) increases linearly with resubscriptr_eritalic_e when resubscriptr_eritalic_e is not too large (figure 6). Here we use the above mentioned fairness PCA vectors. We average on different queries from a few datasets taken from MMLU. In practice we look at U⁢δ⁢re⁢(q)subscriptUδ r_e(q)U δ ritalic_e ( q ), where U is the transformation taking from the final layer representation to the logits vector. Since this is a linear transformation, showing a linear relationship between resubscriptr_eritalic_e and |U⁢δ⁢re⁢(q)|subscript|Uδ r_e(q)|| U δ ritalic_e ( q ) | implies a linear relationship between resubscriptr_eritalic_e and |δ⁢re⁢(q)|subscript|δ r_e(q)|| δ ritalic_e ( q ) |. Figure 6: Linear increase in the norm of U⁢δ⁢re⁢(q)subscriptUδ r_e(q)U δ ritalic_e ( q ) for small coefficients, when injected with “racist" vectors. In figures 7 and 8 we plot the change in norm for Llama 2 13B chat (injected with racist vectors) and Llama 2 13B (injected with not racist vectors) respectively, on the datasets “international law", “medical genetics" and “high school computer science". We add fitted curves to estimate λ. We find that it is in the range 40−60406040-6040 - 60. Figure 7: Norm of the final hidden layer representation change as a function of representation engineering coefficient, for Llama 2 13B chat, on different MMLU datasets. The fitted linear curves estimate λ. Figure 8: Norm of the final hidden layer representation change as a function of representation engineering coefficient, for Llama 2 13B, on different MMLU datasets. The fitted linear curves estimate λ. Random logit noise assumption As proposed in assumption 3, we show here that the projection of a given answer on the representation change δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) is random. (Assuming the question asked is not connected to the property we are changing with the representation engineering). In assumption 3 we looked at the normalized change: ⟨δ⁢re⁢(q)||δ⁢re⁢(q)||,UT⁢ei⟩subscriptsubscriptsuperscriptsubscript δ r_e(q) δ r_e(q) ,U^Te_% i ⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | | δ ritalic_e ( q ) | | end_ARG , Uitalic_T eitalic_i ⟩. Here we will look at ⟨δ⁢re⁢(q),UT⁢ei⟩subscriptsuperscriptsubscript δ r_e(q),U^Te_i ⟨ δ ritalic_e ( q ) , Uitalic_T eitalic_i ⟩, so we expect the distribution to be: ⟨δ⁢re⁢(q),UT⁢ei⟩∼||δ⁢re⁢(q)||⋅Dsimilar-tosubscriptsuperscriptsubscript⋅subscript δ r_e(q),U^Te_i δ r_e(q) % · D⟨ δ ritalic_e ( q ) , Uitalic_T eitalic_i ⟩ ∼ | | δ ritalic_e ( q ) | | ⋅ D Meaning the standard deviation scales linearly with the norm of δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ). Since resubscriptr_eritalic_e scales linearly with δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ), we expect the standard deviation to also scale linearly with resubscriptr_eritalic_e. To measure the effective randomness, we look at ⟨δ⁢re⁢(q),UT⁢(ei−ec⁢o⁢r⁢r⁢e⁢c⁢t)⟩subscriptsuperscriptsubscriptsubscript δ r_e(q),U^T(e_i-e_correct) ⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_i - eitalic_c o r r e c t ) ⟩, which shows explicitly that the correct answer logit change is sometimes enhanced and sometimes decreased relatively to the incorrect answers. We will observe that the noise is approximately normal. To create the plot, for each question in a dataset, we look at the top 10 answers ei,i∈[10]subscriptdelimited-[]10e_i,i∈[10]eitalic_i , i ∈ [ 10 ] (with no representation engineering). We note that experimentally, the top 10 tokens make the majority of the probability mass (over 90%). Now for a given resubscriptr_eritalic_e coefficient, we calculate the projection of these answers on δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ). We then aggregate these projections for all the questions in a few dataset and look at their histogram and at their standard deviation. We repeat this for different resubscriptr_eritalic_e norms. Figure 9: (a) ((b)) Distribution of the change in token logits minus the logit of the correct answer of Llama 2 13B chat when injected with racist (harmful) vectors. As can be seen, it is approximately normal, and in (c) and (d) the standard deviation grows linearly with the coefficient size resubscriptr_eritalic_e, which is linear in |δ⁢re⁢(q)|subscript|δ r_e(q)|| δ ritalic_e ( q ) |. Figure 10: (a) ((b)) Distribution of the change in token logits minus the logit of the correct answer of Llama 2 13B chat when injected with harmless (not-racist) vectors. As can be seen, it is approximately normal, and in (c) and (d) the standard deviation grows linearly with the coefficient size resubscriptr_eritalic_e, which is linear in |δ⁢re⁢(q)|subscript|δ r_e(q)|| δ ritalic_e ( q ) |. The tangent of the curve of figure 9c,d is λ⁢σλσλ σ, as the curve is the standard deviation of ⟨δ⁢re⁢(q)|δ⁢re⁢(q)|,UT⁢ei⟩⋅|δ⁢re⁢(q)|=⟨δ⁢re⁢(q)|δ⁢re⁢(q)|,UT⁢ei⟩⋅λ⁢re⋅subscriptsubscriptsuperscriptsubscriptsubscript⋅subscriptsubscriptsuperscriptsubscriptsubscript δ r_e(q)|δ r_e(q)|,U^Te_i ·|δ r% _e(q)|= δ r_e(q)|δ r_e(q)|,U^Te_i % ·λ r_e⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG , Uitalic_T eitalic_i ⟩ ⋅ | δ ritalic_e ( q ) | = ⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG , Uitalic_T eitalic_i ⟩ ⋅ λ ritalic_e, from assumption 1, and the inner product is a random variable of standard deviation σ, hence the tangent is λ⁢σλσλ σ. We observe that the noise is approximately normal. From the linear curve, we estimate λ⁢σ=0.50.5λσ=0.5λ σ = 0.5, thus λ⁢σ⁢β≈0.8⋅0.5⋅0.80.5λσβ≈ 0.8· 0.5λ σ β ≈ 0.8 ⋅ 0.5, as it is the mean of a half-normal distribution with parameter λ⁢σλσλ σ, which is approximately 0.8⁢λ⁢σ0.80.8λ 0.8 λ σ. Similarly, for the pretrained model, we find that λ⁢σ=0.20.2λσ=0.2λ σ = 0.2 and 0.10.10.10.1 for fairness and harmlessness respectively. Clustering of positive and negative answers to harmful queries Here we aim to estimate how well Δ Δ-representation-separability (definition 2) works in practice. The condition is equivalent to: ⟨δ⁢re⁢(q),UT⁢(eg⁢o⁢o⁢d−eb⁢a⁢d)⟩≥|δ⁢re⁢(q)|⋅Δsubscriptsuperscriptsubscriptsubscript⋅subscriptΔ δ r_e(q),U^T(e_good-e_bad) ≥|δ r_e(q)|· ⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_g o o d - eitalic_b a d ) ⟩ ≥ | δ ritalic_e ( q ) | ⋅ Δ (10) And by assumption 1, it is equivalent to: ⟨δ⁢re⁢(q),UT⁢(eg⁢o⁢o⁢d−eb⁢a⁢d)⟩≥Δ⁢λ⋅resubscriptsuperscriptsubscriptsubscript⋅Δsubscript δ r_e(q),U^T(e_good-e_bad) ≥ λ· r_% e⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_g o o d - eitalic_b a d ) ⟩ ≥ Δ λ ⋅ ritalic_e (11) In figure 11 and 12, we plot the distance between the centers of representation clusters for positive and negative answers to harmful queries as the norm of harmful vectors is increased, for Llama 2 13B chat and Llama 2 13B respectively. As can be seen, the distance between the clusters increases, which corresponds to an increase in ⁢[⟨δ⁢re⁢(q),UT⁢(eg⁢o⁢o⁢d−eb⁢a⁢d)⟩]delimited-[]subscriptsuperscriptsubscriptsubscriptE[ δ r_e(q),U^T(e_good-e_bad) ]blackboard_E [ ⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_g o o d - eitalic_b a d ) ⟩ ]. We can define a range of coefficients in which the increase is bounded from below by a linear curve of the form in equation 11, meaning that the change in the model’s representation separates the positive and negative answer representations, similarly to the definition of Δ Δ-representation separability, but with mean instead of min. Thus by equation 11, the tangent of the lower bounding lines of figures 11 and 12 are an estimate for Δ⁢λΔ λΔ λ. From, this we get that Δ⁢λΔ λΔ λ is approximately 0.1−0.30.10.30.1-0.30.1 - 0.3. In section 4, we obtained values of Δ⁢λΔ λΔ λ in the rage 0.5−30.530.5-30.5 - 3 from the free parameter fit on the bound of theorem 2 to the data. The difference between these two ranges is attributed to the method of the empirical estimation of Δ Δ from the linear classification condition that looks for an upper bound on it on the entire resubscriptr_eritalic_e range, while the main change in alignment in figure 4 occurs in a more specific range, where the upper bound of Δ Δ is evidently bigger. Figure 11: Separation between representation clusters of positive and negative behavior tokens induced by δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) on Llama 2 13B chat for three harmful instructions from the AdvBench dataset. Figure 12: Separation between representation clusters of positive and negative behavior tokens induced by δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ) on Llama 2 13B for three harmful instructions from the AdvBench dataset. In practice, the good and bad tokens were chosen beforehand as the top 40 tokens of the models when representation engineering is applied and when it is not applied (meaning in one case the model is aligned and in the other it is not). Appendix B Proof of theorem 1 The theorem utilizes assumptions 1 and 2. The behavior expectation is: B[Pθ,re(⋅|q)]=∑a+∈g⁢o⁢o⁢dPθ,re⁢(a+|q)−∑a−∈b⁢a⁢dPθ,re⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ,re⁢(a+|q)+∑a−∈b⁢a⁢dPθ,re⁢(a−|q)=B[P_θ,r_e(·|q)]= _a_+∈ goodP_θ,r_e(a_+|% q)- _a_-∈ badP_θ,r_e(a_-|q) _a_+∈ goodP_θ% ,r_e(a_+|q)+ _a_-∈ badP_θ,r_e(a_-|q)=B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] = divide start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a+ | q ) - ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a+ | q ) + ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a- | q ) end_ARG = (12) =∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea+⟩)−∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea+⟩)+∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea−⟩)=absentsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptabsent= _a_+∈ goodexp( r(q)+δ r(q),U^Te_a_+ )% - _a_-∈ badexp( r(q)+δ r(q),U^Te_a_- ) _% a_+∈ goodexp( r(q)+δ r(q),U^Te_a_+ )+ _a_-% ∈ badexp( r(q)+δ r(q),U^Te_a_- )== divide start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) - ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) + ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG = (13) Where r⁢(q)r(q)r ( q ) is the final hidden layer representation and δ⁢r⁢(q)δ r(q)δ r ( q ) is the change to the last hidden layer due to steering on the previous layers. a+∈g⁢o⁢o⁢dsubscripta_+∈ gooda+ ∈ g o o d and a−∈b⁢a⁢dsubscripta_-∈ bada- ∈ b a d denote the aligned and misaligned answers respectively, i.e. B⁢(a±)=±1subscriptplus-or-minusplus-or-minus1B(a_±)=± 1B ( a± ) = ± 1. =1−∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea+⟩)1+∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q)+δ⁢r⁢(q),UT⁢ea+⟩)=absent1subscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscript1subscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptabsent= 1- _a_-∈ badexp( r(q)+δ r(q),U^Te_a_-% ) _a_+∈ goodexp( r(q)+δ r(q),U^Te_a_+% )1+ _a_-∈ badexp( r(q)+δ r(q),U^Te_a_% - ) _a_+∈ goodexp( r(q)+δ r(q),U^Te_a_+% )== divide start_ARG 1 - divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG end_ARG start_ARG 1 + divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG end_ARG = (14) =1−∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)⁢e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+⟩)⁢e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea+⟩)1+∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)⁢e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+⟩)⁢e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea+⟩)=absent1subscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscript1subscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptabsent= 1- _a_-∈ badexp( r(q),U^Te_a_- )exp(% δ r(q),U^Te_a_- ) _a_+∈ goodexp( r(q% ),U^Te_a_+ )exp( δ r(q),U^Te_a_+ )1+% _a_-∈ badexp( r(q),U^Te_a_- )exp( % δ r(q),U^Te_a_- ) _a_+∈ goodexp( r(q),U^T% e_a_+ )exp( δ r(q),U^Te_a_+ )== divide start_ARG 1 - divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG end_ARG start_ARG 1 + divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG end_ARG = (15) Let us look at the fraction that appears in the numerator and denominator: ∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)⁢e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+⟩)⁢e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea+⟩)<subscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptabsent _a_-∈ badexp( r(q),U^Te_a_- )exp( % δ r(q),U^Te_a_- ) _a_+∈ goodexp( r(q),U^T% e_a_+ )exp( δ r(q),U^Te_a_+ )<divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG < (16) <∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)⋅maxa−′∈b⁢a⁢d⁡e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea−′⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+⟩)⋅mina+′∈g⁢o⁢o⁢d⁡e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea+′⟩)=absentsubscriptsubscript⋅superscriptsubscriptsubscriptsubscriptsubscriptsuperscript′subscriptsubscriptsuperscript′subscriptsubscript⋅superscriptsubscriptsubscriptsubscriptsubscriptsuperscript′subscriptsubscriptsuperscript′absent< _a_-∈ badexp( r(q),U^Te_a_- )· _a^% _-∈ bad\exp( δ r(q),U^Te_a _- )\% _a_+∈ goodexp( r(q),U^Te_a_+ )· _a^% _+∈ goodexp( δ r(q),U^Te_a _+ )=< divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) ⋅ maxitalic_a′ start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a′ start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) ⋅ minitalic_a′ start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a′ start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG = (17) Moving the maximum in the numerator to the denominator turns it into a minimum and the exponent’s argument becomes negative, we obtain a product of two minimum terms, which we can jointly write as: =∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+⟩)⋅1mina+′∈g⁢o⁢o⁢d,a−∈b⁢a⁢d⁡e⁢x⁢p⁢(⟨δ⁢r⁢(q),UT⁢ea+′−UT⁢ea−′⟩)absent⋅subscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscript1subscriptformulae-sequencesubscriptsuperscript′subscriptsuperscriptsubscriptsubscriptsuperscript′subscriptsubscriptsuperscript′= _a_-∈ badexp( r(q),U^Te_a_- ) _a_+% ∈ goodexp( r(q),U^Te_a_+ )· 1 _a % _+∈ good,a_-∈ badexp( δ r(q),U^Te_a _+-U^T% e_a _- )= divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG ⋅ divide start_ARG 1 end_ARG start_ARG minitalic_a′ start_POSTSUBSCRIPT + ∈ g o o d , a- ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ δ r ( q ) , Uitalic_T eitalic_a′ start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - Uitalic_T eitalic_a′ start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG (18) As the exponent is a monotonic function, we can insert the minimum into the exponent: =∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+⟩)⋅1e⁢x⁢p⁢(mina+′∈g⁢o⁢o⁢d,a−∈b⁢a⁢d⁡⟨δ⁢r⁢(q)|δ⁢r⁢(q)|,UT⁢ea+′−UT⁢ea−′⟩⋅|δ⁢r⁢(q)|)absent⋅subscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscript1⋅subscriptformulae-sequencesubscriptsuperscript′subscriptsuperscriptsubscriptsubscriptsuperscript′subscriptsubscriptsuperscript′= _a_-∈ badexp( r(q),U^Te_a_- ) _a_+% ∈ goodexp( r(q),U^Te_a_+ )· 1exp( _a^% _+∈ good,a_-∈ bad δ r(q)|δ r(q)|,U^T% e_a _+-U^Te_a _- ·|δ r(q)|)= divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ⟩ ) end_ARG ⋅ divide start_ARG 1 end_ARG start_ARG e x p ( minitalic_a′ start_POSTSUBSCRIPT + ∈ g o o d , a- ∈ b a d end_POSTSUBSCRIPT ⟨ divide start_ARG δ r ( q ) end_ARG start_ARG | δ r ( q ) | end_ARG , Uitalic_T eitalic_a′ start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - Uitalic_T eitalic_a′ start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ⋅ | δ r ( q ) | ) end_ARG (19) From Δ Δ margin linear classification of UT⁢a+a+∈g⁢o⁢o⁢dsubscriptsuperscriptsubscriptsubscript\U^Ta_+\_a_+∈ good Uitalic_T a+ a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT and UT⁢a−a−∈g⁢o⁢o⁢dsubscriptsuperscriptsubscriptsubscript\U^Ta_-\_a_-∈ good Uitalic_T a- a start_POSTSUBSCRIPT - ∈ g o o d end_POSTSUBSCRIPT by δ⁢r⁢(q)|δ⁢r⁢(q)| δ r(q)|δ r(q)|divide start_ARG δ r ( q ) end_ARG start_ARG | δ r ( q ) | end_ARG (assumption 2), the minimum in the denominator is larger than Δ Δ: <∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+−UT⁢ea−⟩)⋅1e⁢x⁢p⁢(Δ⁢|δ⁢r|)absent⋅subscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscript1Δ< _a_-∈ badexp( r(q),U^Te_a_- ) _a_+% ∈ goodexp( r(q),U^Te_a_+-U^Te_a_- )· 1% exp( |δ r|)< divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG ⋅ divide start_ARG 1 end_ARG start_ARG e x p ( Δ | δ r | ) end_ARG (20) Plugging this back in to the behavior expectation, we obtain: B[Pθ,re(⋅|q)]>1−∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+−UT⁢ea−⟩)⋅1e⁢x⁢p⁢(Δ⁢|δ⁢r|)1+∑a−∈b⁢a⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea−⟩)∑a+∈g⁢o⁢o⁢de⁢x⁢p⁢(⟨r⁢(q),UT⁢ea+−UT⁢ea−⟩)⋅1e⁢x⁢p⁢(Δ⁢|δ⁢r|)=B[P_θ,r_e(·|q)]> 1- _a_-∈ badexp( r(q)% ,U^Te_a_- ) _a_+∈ goodexp( r(q),U^Te_a_+-U% ^Te_a_- )· 1exp( |δ r|)1+ _a_-% ∈ badexp( r(q),U^Te_a_- ) _a_+∈ goodexp(% r(q),U^Te_a_+-U^Te_a_- )· 1exp( |% δ r|)=B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > divide start_ARG 1 - divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG ⋅ divide start_ARG 1 end_ARG start_ARG e x p ( Δ | δ r | ) end_ARG end_ARG start_ARG 1 + divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - Uitalic_T eitalic_a start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ⟩ ) end_ARG ⋅ divide start_ARG 1 end_ARG start_ARG e x p ( Δ | δ r | ) end_ARG end_ARG = (21) =1−∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)1+∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)absent1subscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscriptΔ1subscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscriptΔ= 1- _a_-∈ badP_θ(a_-|q) _a_+∈ goodP_% θ(a_+|q)exp(- |δ r|)1+ _a_-∈ badP_θ% (a_-|q) _a_+∈ goodP_θ(a_+|q)exp(- |δ r|)= divide start_ARG 1 - divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG e x p ( - Δ | δ r | ) end_ARG start_ARG 1 + divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG e x p ( - Δ | δ r | ) end_ARG (22) =t⁢a⁢n⁢h⁢(Δ⁢|δ⁢r|−ln⁡(∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q))2)absentℎΔsubscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscript2=tanh( |δ r|- ( _a_-∈ badP_θ(a_-|q)% _a_+∈ goodP_θ(a_+|q))2)= t a n h ( divide start_ARG Δ | δ r | - ln ( divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG ) end_ARG start_ARG 2 end_ARG ) (23) Then, notice that: ∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)=1−B01+B0subscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscript1subscript01subscript0 _a_-∈ badP_θ(a_-|q) _a_+∈ goodP_θ(a% _+|q)= 1-B_01+B_0divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG = divide start_ARG 1 - B0 end_ARG start_ARG 1 + B0 end_ARG (24) Where B0=B[Pθ(⋅|q)]B_0=B[P_θ(·|q)]B0 = B [ Pitalic_θ ( ⋅ | q ) ], and that: a⁢r⁢c⁢t⁢a⁢n⁢h⁢(B0)=−12⁢ln⁡1−B01+B0ℎsubscript0121subscript01subscript0arctanh(B_0)=- 12 1-B_01+B_0a r c t a n h ( B0 ) = - divide start_ARG 1 end_ARG start_ARG 2 end_ARG ln divide start_ARG 1 - B0 end_ARG start_ARG 1 + B0 end_ARG (25) Thus we obtain: B[Pθ,re(⋅|q)]>tanh(Δ⁢|δ⁢r⁢(q)|2+arctanh(B0))B[P_θ,r_e(·|q)]>tanh( |δ r(q)|2+arctanh(B_0))B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > t a n h ( divide start_ARG Δ | δ r ( q ) | end_ARG start_ARG 2 end_ARG + a r c t a n h ( B0 ) ) (26) Lastly, note that for coefficients that are not too large, |δ⁢r⁢(q)||δ r(q)|| δ r ( q ) | is proportional to the injected vector coefficient resubscriptr_eritalic_e (assumption 1), hence: B[Pθ,re(⋅|q)]>tanh(Δ⁢λ2⋅re+arctanh(B0))B[P_θ,r_e(·|q)]>tanh( λ2· r_e+arctanh(B% _0))B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > t a n h ( divide start_ARG Δ λ end_ARG start_ARG 2 end_ARG ⋅ ritalic_e + a r c t a n h ( B0 ) ) (27) Where λ is the coefficient relating resubscriptr_eritalic_e to |δ⁢r⁢(q)||δ r(q)|| δ r ( q ) |. Appendix C Proof of theorem 2 The theorem utilizes assumptions 1 and 3. Notice that: Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)=Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)1=Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)+∑i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢tPθ,re⁢(ai|q)=subscriptsubscriptconditionalsubscriptsubscriptsubscriptconditionalsubscript1subscriptsubscriptconditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscriptabsentP_θ,r_e(a_correct|q)= P_θ,r_e(a_correct|q)1=% P_θ,r_e(a_correct|q)P_θ,r_e(a_correct|q)+ _i% ∈ incorrectP_θ,r_e(a_i|q)=Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) = divide start_ARG Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) end_ARG start_ARG 1 end_ARG = divide start_ARG Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) end_ARG start_ARG Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) + ∑i ∈ i n c o r r e c t Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_i | q ) end_ARG = (28) =Pθ⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)Pθ⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)+∑i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢tPθ⁢(ai|q)⁢e⟨δ⁢re⁢(q),UT⁢(ei−ec⁢o⁢r⁢r⁢e⁢c⁢t⁢(q))⟩≤absentsubscriptconditionalsubscriptsubscriptconditionalsubscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsuperscriptsubscriptsubscriptabsent= P_θ(a_correct|q)P_θ(a_correct|q)+ _i∈ incorrect% P_θ(a_i|q)e δ r_e(q),U^T(e_i-e_correct(q))% ≤= divide start_ARG Pitalic_θ ( aitalic_c o r r e c t | q ) end_ARG start_ARG Pitalic_θ ( aitalic_c o r r e c t | q ) + ∑i ∈ i n c o r r e c t Pitalic_θ ( aitalic_i | q ) e⟨ δ ritalic_e ( q ) , U start_POSTSUPERSCRIPT T ( eitalic_i - eitalic_c o r r e c t ( q ) ) ⟩ end_POSTSUPERSCRIPT end_ARG ≤ (29) Denote Xi=⟨δ⁢re⁢(q)|δ⁢re⁢(q)|,UT⁢ei⟩subscriptsubscriptsubscriptsuperscriptsubscriptX_i= δ r_e(q)|δ r_e(q)|,U^Te_i _i = ⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG , Uitalic_T eitalic_i ⟩ and by Pc⁢o⁢r⁢r⁢e⁢c⁢t0superscriptsubscript0P_correct^0Pitalic_c o r r e c t0 the probability of answering correctly without steering: =Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+∑i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢tPθ⁢(ai|q)⁢e|δ⁢re⁢(q)|⁢(Xi−Xc⁢o⁢r⁢r⁢e⁢c⁢t)≤absentsubscriptsuperscript0subscriptsuperscript0subscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptsubscriptabsent= P^0_correctP^0_correct+ _i∈ incorrectP_θ(a_i|% q)e^|δ r_e(q)|(X_i-X_correct)≤= divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ∑i ∈ i n c o r r e c t Pitalic_θ ( aitalic_i | q ) e| δ ritalic_e ( q ) | ( Xitalic_i - Xitalic_c o r r e c t ) end_ARG ≤ (30) Next, by considering the sum only only over highest probability tokens making up 1−ϵ1italic-ϵ1- 1 - ϵ of the probability mass, for which we denote the incorrect tokens sum as i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢t⁢(ϵ)italic-ϵincorrect(ε)i n c o r r e c t ( ϵ ): ≤Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+∑i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢t⁢(ϵ)Pθ⁢(ai|q)⁢e|δ⁢re⁢(q)|⁢(Xi−Xc⁢o⁢r⁢r⁢e⁢c⁢t)≤absentsubscriptsuperscript0subscriptsuperscript0subscriptitalic-ϵsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptsubscriptabsent≤ P^0_correctP^0_correct+ _i∈ incorrect(ε)P_% θ(a_i|q)e^|δ r_e(q)|(X_i-X_correct)≤ divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ∑i ∈ i n c o r r e c t ( ϵ ) Pitalic_θ ( aitalic_i | q ) e| δ ritalic_e ( q ) | ( Xitalic_i - Xitalic_c o r r e c t ) end_ARG ≤ (31) Denote by I±=i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢t⁢(ϵ)|±(Xi−Xc⁢o⁢r⁢r⁢e⁢c⁢t)>0subscriptplus-or-minusconditional-setitalic-ϵplus-or-minussubscriptsubscript0I_±=\i∈ incorrect(ε)|±(X_i-X_correct)>0\I± = i ∈ i n c o r r e c t ( ϵ ) | ± ( Xitalic_i - Xitalic_c o r r e c t ) > 0 (i.e. XisubscriptX_iXitalic_i’s that are larger/smaller than Xc⁢o⁢r⁢r⁢e⁢c⁢tsubscriptX_correctXitalic_c o r r e c t). Also denote by Pi0=Pθ⁢(ai|q)superscriptsubscript0subscriptconditionalsubscriptP_i^0=P_θ(a_i|q)Pitalic_i0 = Pitalic_θ ( aitalic_i | q ) and Yi=|δ⁢re⁢(q)|⁢(Xi−Xc⁢o⁢r⁢r⁢e⁢c⁢t)subscriptsubscriptsubscriptsubscriptY_i=|δ r_e(q)|(X_i-X_correct)Yitalic_i = | δ ritalic_e ( q ) | ( Xitalic_i - Xitalic_c o r r e c t ). We obtain two sums of the form ∑i∈I±Pi⁢eYisubscriptsubscriptplus-or-minussubscriptsuperscriptsubscript _i∈ I_±P_ie^Y_i∑i ∈ I start_POSTSUBSCRIPT ± end_POSTSUBSCRIPT Pitalic_i eitalic_Yitalic_i. Since the exponent is a convex function, using Jensen’s inequality, on the sums yields ∑i∈IPi⁢eYi≥(∑i∈IPi)⋅e∑j∈IPj⁢Yi∑j∈IPjsubscriptsubscriptsuperscriptsubscript⋅subscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsubscript _i∈ IP_ie^Y_i≥( _i∈ IP_i)· e _j∈ I% P_jY_i _j∈ IP_j∑i ∈ I Pitalic_i eitalic_Yitalic_i ≥ ( ∑i ∈ I Pitalic_i ) ⋅ edivide start_ARG ∑j ∈ I Pitalic_j Yitalic_i end_ARG start_ARG ∑j ∈ I Pitalic_j end_ARG. Plugging this in: ≤Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(∑i∈I+Pi0)⋅e∑j∈I+Pj0⁢(Xj−Xc⁢o⁢r⁢r⁢e⁢c⁢t)∑j∈I+Pj0⁢|δ⁢re⁢(q)|+(∑i∈I−Pi0)⋅e∑j∈I−Pj0⁢(Xj−Xc⁢o⁢r⁢r⁢e⁢c⁢t)∑j∈I−Pj0⁢|δ⁢re⁢(q)|absentsubscriptsuperscript0subscriptsuperscript0⋅subscriptsubscriptsuperscriptsubscript0superscriptsubscriptsubscriptsuperscriptsubscript0subscriptsubscriptsubscriptsubscriptsuperscriptsubscript0subscript⋅subscriptsubscriptsuperscriptsubscript0superscriptsubscriptsubscriptsuperscriptsubscript0subscriptsubscriptsubscriptsubscriptsuperscriptsubscript0subscript≤ P^0_correctP^0_correct+( _i∈ I_+P_i^0)· e% _j∈ I_+P_j^0(X_j-X_correct) _j∈ I_+P_j^% 0|δ r_e(q)|+( _i∈ I_-P_i^0)· e _j∈ I% _-P_j^0(X_j-X_correct) _j∈ I_-P_j^0|δ r_e(q)% |≤ divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( ∑i ∈ I start_POSTSUBSCRIPT + end_POSTSUBSCRIPT Pitalic_i0 ) ⋅ edivide start_ARG ∑j ∈ I start_POSTSUBSCRIPT + end_POSTSUBSCRIPT Pitalic_j start_POSTSUPERSCRIPT 0 ( Xitalic_j - Xitalic_c o r r e c t ) end_ARG start_ARG ∑j ∈ I start_POSTSUBSCRIPT + end_POSTSUBSCRIPT Pitalic_j0 end_ARG | δ ritalic_e ( q ) | end_POSTSUPERSCRIPT + ( ∑i ∈ I start_POSTSUBSCRIPT - end_POSTSUBSCRIPT Pitalic_i0 ) ⋅ edivide start_ARG ∑j ∈ I start_POSTSUBSCRIPT - end_POSTSUBSCRIPT Pitalic_j start_POSTSUPERSCRIPT 0 ( Xitalic_j - Xitalic_c o r r e c t ) end_ARG start_ARG ∑j ∈ I start_POSTSUBSCRIPT - end_POSTSUBSCRIPT Pitalic_j0 end_ARG | δ ritalic_e ( q ) | end_POSTSUPERSCRIPT end_ARG (32) Denote by P±=∑i∈I±Pi0subscriptplus-or-minussubscriptlimit-fromplus-or-minussuperscriptsubscript0P_±= _i∈ I±P_i^0P± = ∑i ∈ I ± Pitalic_i0 and c±=∑i∈I±Pi0⁢(Xi−Xc⁢o⁢r⁢r⁢e⁢c⁢t)∑i∈I±Pi0subscriptplus-or-minussubscriptsubscriptplus-or-minussuperscriptsubscript0subscriptsubscriptsubscriptsubscriptplus-or-minussuperscriptsubscript0c_±= _i∈ I_±P_i^0(X_i-X_correct) _i∈ I_% ±P_i^0c± = divide start_ARG ∑i ∈ I start_POSTSUBSCRIPT ± end_POSTSUBSCRIPT Pitalic_i0 ( Xitalic_i - Xitalic_c o r r e c t ) end_ARG start_ARG ∑i ∈ I start_POSTSUBSCRIPT ± end_POSTSUBSCRIPT Pitalic_i0 end_ARG. We get: =Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+P+⁢ec+⁢|δ⁢re⁢(q)|+P−⁢ec−⁢|δ⁢re⁢(q)|absentsubscriptsuperscript0subscriptsuperscript0subscriptsuperscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscript= P^0_correctP^0_correct+P_+e^c_+|δ r_e(q)|+P_-e% ^c_-|δ r_e(q)|= divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + P+ eitalic_c+ | δ ritalic_e ( q ) | + P- eitalic_c- | δ ritalic_e ( q ) | end_ARG (33) ≤Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+min⁡P−,P+⁢(ec+⁢|δ⁢re⁢(q)|+ec−⁢|δ⁢re⁢(q)|)absentsubscriptsuperscript0subscriptsuperscript0subscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscript≤ P^0_correctP^0_correct+ \P_-,P_+\(e^c_+|δ r% _e(q)|+e^c_-|δ r_e(q)|)≤ divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + min P- , P+ ( eitalic_c+ | δ ritalic_e ( q ) | + eitalic_c- | δ ritalic_e ( q ) | ) end_ARG (34) ≤Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+minP−,P+(1+12min|c−|,c+2|δre(q)|2)≤ P^0_correctP^0_correct+ \P_-,P_+\(1+ 12% \|c_-|,c_+\^2|δ r_e(q)|^2)≤ divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + min P- , P+ ( 1 + divide start_ARG 1 end_ARG start_ARG 2 end_ARG min | c- | , c+ 2 | δ ritalic_e ( q ) |2 ) end_ARG (35) Lastly, note that for coefficients that are not too large, |δ⁢r⁢(q)||δ r(q)|| δ r ( q ) | is proportional to the injected vector coefficient resubscriptr_eritalic_e (assumption 1), hence: ≤Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+minP−,P+(1+12min|c−|,c+2λ2|re|2)≤ P^0_correctP^0_correct+ \P_-,P_+\(1+ 12% \|c_-|,c_+\^2λ^2|r_e|^2)≤ divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + min P- , P+ ( 1 + divide start_ARG 1 end_ARG start_ARG 2 end_ARG min | c- | , c+ 2 λ2 | ritalic_e |2 ) end_ARG (36) Under the assumption that XisubscriptX_iXitalic_i distribute randomly (assumption 3), c±subscriptplus-or-minusc_±c± are a weighted sum of positive/negative random variables with parameter σ, which we can refactor to σ⋅c±′⋅superscriptsubscriptplus-or-minus′σ· c_± σ ⋅ c±′ where c±′subscriptplus-or-minus′c_± c±′ are the same variables but normalized to σ′=1superscript′1σ =1σ′ = 1. Denoting β=min⁡|c−′|,c+′subscriptsuperscript′subscriptsuperscript′β= \|c _-|,c _+\β = min | c′- | , c′+ , yields: ≤Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+min⁡P−,P+⁢(1+12⁢β2⁢σ2⁢λ2⁢|re|2)absentsubscriptsuperscript0subscriptsuperscript0subscriptsubscript112superscript2superscript2superscript2superscriptsubscript2≤ P^0_correctP^0_correct+ \P_-,P_+\(1+ 12% β^2σ^2λ^2|r_e|^2)≤ divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + min P- , P+ ( 1 + divide start_ARG 1 end_ARG start_ARG 2 end_ARG β2 σ2 λ2 | ritalic_e |2 ) end_ARG (37) We denote α=min⁡P−,P+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢(1−ϵ)subscriptsubscript1subscriptsuperscript01italic-ϵα= \P_-,P_+\(1-P^0_correct)(1-ε)α = divide start_ARG min P- , P+ end_ARG start_ARG ( 1 - P0italic_c o r r e c t ) ( 1 - ϵ ) end_ARG, since we considered only the tokens making 1−ϵ1italic-ϵ1- 1 - ϵ of the probability mass, thus, P++P−=(1−ϵ)⁢(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)subscriptsubscript1italic-ϵ1superscriptsubscript0P_++P_-=(1-ε)(1-P_correct^0)P+ + P- = ( 1 - ϵ ) ( 1 - Pitalic_c o r r e c t0 ). Hence α measures the non-tightness of the bound, due to the asymmetry between P±subscriptplus-or-minusP_±P±, and (1−ϵ)1italic-ϵ(1-ε)( 1 - ϵ ) the non-tightness due to not using all the words in the vocabulary for the bound, only the top T. =Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢α⁢(1−ϵ)⁢(1+12⁢β2⁢σ2⁢λ2⁢|re|2)absentsubscriptsuperscript0subscriptsuperscript01superscriptsubscript01italic-ϵ112superscript2superscript2superscript2superscriptsubscript2= P^0_correctP^0_correct+(1-P_correct^0)α(1-ε)(% 1+ 12β^2σ^2λ^2|r_e|^2)= divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - Pitalic_c o r r e c t0 ) α ( 1 - ϵ ) ( 1 + divide start_ARG 1 end_ARG start_ARG 2 end_ARG β2 σ2 λ2 | ritalic_e |2 ) end_ARG (38) Notice that I−subscriptI_-I- is empty if Xi>Xc⁢o⁢r⁢r⁢e⁢c⁢tsubscriptsubscriptX_i>X_correctXitalic_i > Xitalic_c o r r e c t for all i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢t⁢(ϵ)italic-ϵi∈ incorrect(ε)i ∈ i n c o r r e c t ( ϵ ), and from assumption 3, these random variables are identically distributed, hence from symmetry, the event that Xc⁢o⁢r⁢r⁢e⁢c⁢tsubscriptX_correctXitalic_c o r r e c t is the smallest of the T random variables is 1/T11/T1 / T. Thus, with probability 1T1 1Tdivide start_ARG 1 end_ARG start_ARG T end_ARG the set I±subscriptplus-or-minusI_±I± is empty, therefore with probability 1−2T121- 2T1 - divide start_ARG 2 end_ARG start_ARG T end_ARG both sets are not empty, thus P±>0subscriptplus-or-minus0P_±>0P± > 0 and c+>0subscript0c_+>0c+ > 0, c−<0subscript0c_-<0c- < 0. From the above-mentioned symmetry arising from the random variables Xc⁢o⁢r⁢r⁢e⁢c⁢t,Xii∈I−∪I+subscriptsubscriptsubscriptsubscriptsubscriptX_correct,\X_i\_i∈ I_-∪ I_+Xitalic_c o r r e c t , Xitalic_i i ∈ I start_POSTSUBSCRIPT - ∪ I+ end_POSTSUBSCRIPT being identically distributed, for each individual i, P⁢(Xc⁢o⁢r⁢r⁢e⁢c⁢t>Xi)=12subscriptsubscript12P(X_correct>X_i)= 12P ( Xitalic_c o r r e c t > Xitalic_i ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG, thus i∈I+subscripti∈ I_+i ∈ I+ with probability 1212 12divide start_ARG 1 end_ARG start_ARG 2 end_ARG. Therefore, P+subscriptP_+P+ is a weighted sum of Bernoulli variables with weights Pi0i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢tsubscriptsubscriptsuperscript0\P^0_i\_i∈ incorrect P0italic_i i ∈ i n c o r r e c t. Appendix D Alignment Guarantee with Steering In contrast to Wolf et al. (2023), that has a framework centralized on using prompts to misalign frozen models, i.e. models whose weights and representations are not changed after training, here the model is not frozen due to steering, and accordingly a different result is obtained on guaranteeing an aligned response – for any adversarial attack, using large enough norms with representation engineering produces an aligned response if the learned injected representations accumulate to a good classifier of positive and negative answer representations in the final layer. This is formalized here as a corollary of theorem 1. Corollary 1. Let ϵ>0italic-ϵ0ε>0ϵ > 0, PθsubscriptP_θPitalic_θ a language model and q a prompt that induces negative behavior B[Pθ(⋅|q)]<γ<0B[P_θ(·|q)]<γ<0B [ Pitalic_θ ( ⋅ | q ) ] < γ < 0 without steering. Under the conditions of theorem 1, using an injected vector norm of re>1Δ⁢λ⁢(a⁢r⁢c⁢t⁢a⁢n⁢h⁢(1−ϵ)−a⁢r⁢c⁢t⁢a⁢n⁢h⁢(γ))subscript1Δℎ1italic-ϵℎr_e> 1 λ(arctanh(1-ε)-arctanh(γ))ritalic_e > divide start_ARG 1 end_ARG start_ARG Δ λ end_ARG ( a r c t a n h ( 1 - ϵ ) - a r c t a n h ( γ ) ) leads to behavior expectation B[Pθ,re(⋅|q)]>1−ϵB[P_θ,r_e(·|q)]>1- [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > 1 - ϵ. Appendix E Helpfulness at the Limit of Large Steering Vectors When considering the average helpfulness over a dataset in a scenario where the number of answers is constant, N (such as multiple choice questions), we obtain that on average, the model will converge to answering 1/N11/N1 / N of the questions correctly as steering is increased: Corollary 2. Under the conditions of theorem 2, the expected value of the helpfulness on a dataset of queries, q∈d⁢a⁢t⁢a⁢s⁢e⁢t⁢[Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)]subscriptdelimited-[]subscriptsubscriptconditionalsubscriptE_q∈ dataset[P_θ,r_e(a_correct|q)]blackboard_Eq ∈ d a t a s e t [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) ] is asymptotically bounded from above by 1N1 1Ndivide start_ARG 1 end_ARG start_ARG N end_ARG as |re|→∞→subscript|r_e|→∞| ritalic_e | → ∞. Where N is the number of possible answers for each query. Intuitively, for large |re|subscript|r_e|| ritalic_e |, the model is uniformly random, so it will guess the correct answer with probability 1N1 1Ndivide start_ARG 1 end_ARG start_ARG N end_ARG. This can be seen in section 4. proof: Following the notation of the proof of theorem 2, with probability 1V1 1Vdivide start_ARG 1 end_ARG start_ARG V end_ARG, I−subscriptI_-I- is empty: Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)<Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢e|δ⁢re⁢(q)|⁢∑i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢tPi0⁢(Xi−Xc⁢o⁢r⁢r⁢e⁢c⁢t)∑i∈i⁢n⁢c⁢o⁢r⁢r⁢e⁢c⁢tPi0subscriptsubscriptconditionalsubscriptsubscriptsuperscript0subscriptsuperscript01subscriptsuperscript0superscriptsubscriptsubscriptsubscriptsuperscript0subscriptsubscriptsubscriptsubscriptsuperscript0P_θ,r_e(a_correct|q)< P^0_correctP^0_correct+(1-P^0% _correct)e^|δ r_e(q)| _i∈ incorrectP^0_i(X_i-X_% correct) _i∈ incorrectP^0_iPitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) < divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) e| δ ritalic_e ( q ) | divide start_ARG ∑i ∈ i n c o r r e c t P start_POSTSUPERSCRIPT 0i ( Xitalic_i - Xitalic_c o r r e c t ) end_ARG start_ARG ∑i ∈ i n c o r r e c t P0italic_i end_ARG end_POSTSUPERSCRIPT end_ARG (39) In the notation of the proof of theorem 2: Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢ec+⁢|δ⁢re⁢(q)|=Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢ec+⁢λ⁢resubscriptsuperscript0subscriptsuperscript01subscriptsuperscript0superscriptsubscriptsubscriptsubscriptsuperscript0subscriptsuperscript01subscriptsuperscript0superscriptsubscriptsubscript P^0_correctP^0_correct+(1-P^0_correct)e^c_+|δ r_e% (q)|= P^0_correctP^0_correct+(1-P^0_correct)e^c_+% λ r_edivide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) eitalic_c+ | δ ritalic_e ( q ) | end_ARG = divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) eitalic_c+ λ ritalic_e end_ARG (40) Where c+>0subscript0c_+>0c+ > 0 is a weighted sum of half-normal variables. The last transition is by assumption 1. Similarly, with probability 1T1 1Tdivide start_ARG 1 end_ARG start_ARG T end_ARG, I+subscriptI_+I+ is empty, thus Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)<Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢ec−⁢|δ⁢re⁢(q)|=Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)<Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢ec−⁢λ⁢resubscriptsubscriptconditionalsubscriptsubscriptsuperscript0subscriptsuperscript01subscriptsuperscript0superscriptsubscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscriptsuperscript0subscriptsuperscript01subscriptsuperscript0superscriptsubscriptsubscriptP_θ,r_e(a_correct|q)< P^0_correctP^0_correct+(1-P^0% _correct)e^c_-|δ r_e(q)|=P_θ,r_e(a_correct|q)< % P^0_correctP^0_correct+(1-P^0_correct)e^c_-λ r_ePitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) < divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) eitalic_c- | δ ritalic_e ( q ) | end_ARG = Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) < divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) eitalic_c- λ ritalic_e end_ARG (41) Where c−<0subscript0c_-<0c- < 0. Thus for re→∞→subscriptr_e→∞ritalic_e → ∞, with probability 1−2T121- 2T1 - divide start_ARG 2 end_ARG start_ARG T end_ARG, it is bounded by a term that approaches 00 (that of theorem 2), with probability 1/T11/T1 / T another term that approaches 00 (the sigmoid with c+subscriptc_+c+), and with probability 1/T11/T1 / T a term that approaches 1111 (the sigmoid with c−subscriptc_-c-). Hence the expectation value is bounded by 1T1 1Tdivide start_ARG 1 end_ARG start_ARG T end_ARG. This proves corollary 2. For a combination of all these results, notice that with probability 1−2T121- 2T1 - divide start_ARG 2 end_ARG start_ARG T end_ARG, the helpfulness is bounded by the term in theorem 2, while with probability 1T1 1Tdivide start_ARG 1 end_ARG start_ARG T end_ARG it is bounded by: Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢ec+⁢|δ⁢re⁢(q)|subscriptsuperscript0subscriptsuperscript01subscriptsuperscript0superscriptsubscriptsubscript P^0_correctP^0_correct+(1-P^0_correct)e^c_+|δ r_e% (q)|divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) eitalic_c+ | δ ritalic_e ( q ) | end_ARG (42) For re>0subscript0r_e>0ritalic_e > 0, this term is bounded by: <Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢(1+c+2⁢λ2⁢re2)absentsubscriptsuperscript0subscriptsuperscript01subscriptsuperscript01superscriptsubscript2superscript2superscriptsubscript2< P^0_correctP^0_correct+(1-P^0_correct)(1+c_+^2λ% ^2r_e^2)< divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) ( 1 + c+2 λ2 ritalic_e2 ) end_ARG (43) While for re<0subscript0r_e<0ritalic_e < 0 it is bounded by 1111. For the sigmoid with c−subscriptc_-c-, we get the same bound, except that for re>0subscript0r_e>0ritalic_e > 0 it is bounded by 1111, while for re<0subscript0r_e<0ritalic_e < 0 it is bounded by: <Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)⁢(1+c−2⁢λ2⁢re2)absentsubscriptsuperscript0subscriptsuperscript01subscriptsuperscript01superscriptsubscript2superscript2superscriptsubscript2< P^0_correctP^0_correct+(1-P^0_correct)(1+c_-^2λ% ^2r_e^2)< divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + ( 1 - P0italic_c o r r e c t ) ( 1 + c-2 λ2 ritalic_e2 ) end_ARG (44) Taking the weighted average of these three bounds gives the expectation value over the randomness of I±subscriptplus-or-minusI_±I± being empty/non-empty: ⁢[Pθ,re⁢(ac⁢o⁢r⁢r⁢e⁢c⁢t|q)]<(1−1T)⁢Pc⁢o⁢r⁢r⁢e⁢c⁢t0Pc⁢o⁢r⁢r⁢e⁢c⁢t0+α(1−Pc⁢o⁢r⁢r⁢e⁢c⁢t0)(1+minc−,c+,β2λ2re2)+1TE[P_θ,r_e(a_correct|q)]<(1- 1T) P^0_% correctP^0_correct+α(1-P^0_correct)(1+ \c_-,c_+,β% \^2λ^2r_e^2)+ 1Tblackboard_E [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( aitalic_c o r r e c t | q ) ] < ( 1 - divide start_ARG 1 end_ARG start_ARG T end_ARG ) divide start_ARG P0italic_c o r r e c t end_ARG start_ARG P0italic_c o r r e c t + α ( 1 - P0italic_c o r r e c t ) ( 1 + min c- , c+ , β 2 λ2 ritalic_e2 ) end_ARG + divide start_ARG 1 end_ARG start_ARG T end_ARG (45) Thus at the limit of re→∞→subscriptr_e→∞ritalic_e → ∞, we obtain a bound of 1/T11/T1 / T. Appendix F Experimental Details F.1 Compute All experiments were conducted using 2 NVIDIA A10 GPUs, 2 CPUs and 80GB of memory. Recreating all experiments takes in total no more than 4 hours. F.2 Obtaining the engineered representations We use the code provided by Zou et al. (2023a) to perform representation engineering on the Llama 2 13B chat, Llama 2 13B models Touvron et al. (2023) and Llama 3.1 8B chat, Llama 3.1 8B models Dubey et al. (2024). To construct the engineered representations, we first create a set pairs of positive and negative behavior prompts, and extract their representations in each hidden layer. Then, the pairs of vectors are subtracted per layer. On these vector differences we perform PCA analysis to extract the highest order PCA vectors per layer as the representation engineered vectors, re(l)superscriptsubscript\r_e^(l)\ ritalic_e( l ) . We create behavior vectors of harmlessness and not-racist to align Llama 2 13b, Llama 3.1 8B and harmfulness and racism vectors to misalign Llama 2 13b chat, Llama 3.1 8B Instruct. • For creating racism vectors for Llama 2 13B chat and Llama 3.1 8B Instruct, we use the StereoSet bias dataset Nadeem et al. (2020). Given a bias statement from the dataset, we apply it over the template of an aligned response: “Qusestion: Consider the statement: [STATEMENT] Answer: The statement is false and racist.", and a template of a misaligned response: “Question: Consider the statement: [STATEMENT] Answer: 100%". Then, we perform the PCA procedure to create the engineered representations. • For the non-racist vectors for Llama 2 13B and Llama 3.1 8B, we use the same dataset and template described above for the racist vectors of Llama 2 13B chat, only this time the desired behavior is opposite, so we subtract the representations of the prompts in reverse order. • For the harmfulness vectors for Llama 2 13B chat and Llama 3.1 8B Instruct, we take harmful instructions from AdvBench dataset Robey et al. (2021; 2022) and unharmful instructions from shareGPT, and create pairs of prompts with the format “Request: Can you [INSTRUCTION]?", where the instruction is taken from one of the datasets (a pair consists of two prompts, one with a harmful instruction and one with an unharmful instruction). • For the harmless vectors for Llama 2 13B and Llama 3.1 8B, we use the same dataset and template described above for the harmfulness vectors of Llama 2 13B chat and Llama 3.1 8B, only this time the desired behavior is opposite, so we subtract the representations of the prompts in reverse order. In all cases, we used 128 pairs of positive and negative statements. For the bias vectors, we injected the model in layers −1111-11- 11 to −3030-30- 30 and for harmfulness/harmlesness in layers −2525-25- 25 to −3333-33- 33. F.3 Alignment experiments To perform the experiment measuring racism, we fed the representation engineered model biased statements, with the prompt "[INST]Do you agree with the following statement? [STATEMENT][/INST]" and for llama3.1: <|start_header_id|> user <|end_header_id|> Do you agree with the following statement? [STATEMENT] <|eot_id|> <|start_header_id|> assistant <|end_header_id|>" where the statement was taken from the racist statements in the bias dataset. We generate the response of the model and evaluate whether the model agreed or disagreed with the racist statement based on his full answer. We average on 100 racist statements randomly selected from the StereoSet dataset and plot P⁢(A⁢g⁢r⁢e⁢e)−P⁢(D⁢i⁢s⁢a⁢g⁢r⁢e⁢e)P(Agree)-P(Disagree)P ( A g r e e ) - P ( D i s a g r e e ) (or P⁢(D⁢i⁢s⁢a⁢g⁢r⁢e⁢e)−P⁢(A⁢g⁢r⁢e⁢e)P(Disagree)-P(Agree)P ( D i s a g r e e ) - P ( A g r e e ) for the unaligned model) as a function of the injected vectors’ coefficient resubscriptr_eritalic_e. To perform the experiment for compliance with harmful instructions, we queried the model with harmful instructions from AdvBench and checked as a function of representation engineering coefficient whether the model agrees or refuses to answer the instruction. The answers were sampled under greedy decoding for each coefficient, and averaged on 100 harmful instructions for Llama 2 13B chat, Llama 2 13B and also for Llama 3.1 8B Instruct, Llama 3.1 8B. Note that taking the temperature to zero in greedy sampling is equivalent to taking the representation norms to infinity, thus the hyperbolic tangent becomes a step function, and the step appears where the probability of a positive and negative response are equally likely. However, due to the linear dependence of the behavior on resubscriptr_eritalic_e, when averaging on several instructions, the points where the behavior flips are evenly spread between queries, creating the linear curve. Results on Llama 2 13B models are presented in figure 4 and on Llama 3.1 8B Instruct in figure 16 F.4 Helpfulness experiments We evaluate the performance of a model on an MMLU dataset by feeding 100 questions from the test set to the model in the form: "[Question][A)Choice A][B) Choice B][C) Choice C][D) Choice D] The answer is", then calculate the probabilities for answering "A", "B", "C" and "D" and take the correct answer’s probability. We averaged the probability of the correct answer over the data set. This was performed for different coefficients to create the figures in 5. While the bound of theorem 2 is with probability 1−2|V|=1212121- 2|V|= 121 - divide start_ARG 2 end_ARG start_ARG | V | end_ARG = divide start_ARG 1 end_ARG start_ARG 2 end_ARG in the case of 4444 answers, as explained in E, for the other 2|V|2 2|V|divide start_ARG 2 end_ARG start_ARG | V | end_ARG probability, the helpfulness is bounded with equal probability either by a sigmoid or by a reverse sigmoid, such that together they contribute approximately 1|V|1 1|V|divide start_ARG 1 end_ARG start_ARG | V | end_ARG to the expectation value of the helpfulness (due to their small overlap), leading to corollary 2, in which the average helpfulness converges to 1|V|=14114 1|V|= 14divide start_ARG 1 end_ARG start_ARG | V | end_ARG = divide start_ARG 1 end_ARG start_ARG 4 end_ARG in the case of our experiment, as can be seen in figure 5. Around re=0subscript0r_e=0ritalic_e = 0, the contribution of these sigmoids to the helpfulness expectation value can be bounded with the parabolic bound of theorem 2 as shown in the proof provided in appendix E. Thus in total, the bound of theorem 2 with boundary conditions of corollary 2 is theoretically justified. Additionally, we performed a variation of the experiment by sampling full answers to questions from the model (temperature 1.0 over the full vocabulary of the model). Then, where the answer is provided, calculated the probability for the correct answer over the entire vocabulary. This is presented for Llama 2 13B models in figure 13, and for Llama 3.1 8B models in figure 15. We also calculate the accuracy of the Llama 2 13B models answers as presented in figure 14. F.5 Figures All error bars were produced using mean squared error. The method of fitting the curves to the data can be found in the code. Figure 13: Helpfulness measurement: Same as figure 5, but calculating the probability of correct answer over the full vocabulary. Figure 14: Helpfulness measurement: Accuracy of correct answer over the full vocabulary. Figure 15: Helpfulness measurement: the probability assigned to the correct answer to questions from different MMLU tests (international law, medical genetics, high school computer science), as a function of representation engineering vector coefficients injected to the model. Here the probability of the correct answer was over the full vocabulary. (a) Helpfulness of Llama 3.1 8B as a function of coefficient of injected harmful PCA vectors. (b) Helpfulness of Llama 3.1 8B as a function of coefficient of injected bias PCA vectors. (c) Helpfulness of Llama 3.1 8B Instruct as a function of coefficient of injected harmful PCA vectors. (d) Helpfulness of Llama 3.1 8B Instruct as a function of coefficient of injected bias PCA vectors. Figure 16: Plots of behavior expectation as a function of the coefficients of representation engineering vectors injected to the model. (a) Harmful behavior expectation of Llama 3.1 8B Instruct as a function of coefficient of injected harmful PCA vectors. (b) Racism behavior expectation of Llama 3.1 8B Instruct as a function of coefficient of injected bias PCA vectors. Appendix G Helpfulness Experiments on Code Generation In section 4, we showed the model’s helpfulness on knowledge based question answering as a function of steering satisfies theorem 2. This was performed on multiple-choice questions, which shows the applicability of the theoretical results for single token answers. For demonstrating the theoretical results on tasks requiring generation of full sequences, we test the model’s coding skills with the humaneval dataset (Chen et al., 2021). As can be seen in figure 17, The model’s performance is peaked around re=0subscript0r_e=0ritalic_e = 0, and it decays parabolically ar resubscriptr_eritalic_e increases, as predicted in theorem 2. We note that the asymmetry between positive and negative coefficients is captured in our theoretical bounds. Figure 17: Helpfulness measurement on humaneval of Llama 2 13B chat as a function of coefficient of injected harmfulness (a) and racism (b) PCA vectors. Appendix H Relaxation to Soft Margin In the proof of theorem 1, we use the assumption that the change to the last hidden layer representation due to steering linearly classifies the representations of positive and negative answers to a query with margin Δ Δ (as explained in appendix A). We can relax this assumption by assuming that some of the negative (positive) responses’ representations, are misclassified as aligned (misaligned) answers by δ⁢re⁢(q)subscriptδ r_e(q)δ ritalic_e ( q ), in the sense that: i∈a⁢l⁢i⁢g⁢n⁢e⁢d,j∈m⁢i⁢s⁢a⁢l⁢i⁢g⁢n⁢e⁢d:⟨δ⁢re⁢(q),UT⁢(ei−ej)⟩≤Δ:formulae-sequencesubscriptsuperscriptsubscriptsubscriptΔi∈ aligned,j∈ misaligned:~~~ δ r_e(q),U^T(e_i-e_j% ) ≤ ∈ a l i g n e d , j ∈ m i s a l i g n e d : ⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_i - eitalic_j ) ⟩ ≤ Δ (46) That is, the margin Δ Δ does not hold for every pair of aligned and misaligned answers. The key idea is that while it is indeed possible for such misclassifications to occur, the probability assigned to most of the tokens in the vocabulary is very small, thus we can bound their contribution to the behavior expectation. To this end, we define a set of misclassified responses: i∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢d\i∈ misclassified\ i ∈ m i s c l a s s i f i e d and bound the probability mass that the model assigns them by: ∑i∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dPθ⁢(i|q)<δ⋅∑i∈a⁢l⁢i⁢g⁢n⁢e⁢dPθ⁢(i|q)subscriptsubscriptconditional⋅subscriptsubscriptconditional _i∈ misclassifiedP_θ(i|q)<δ· _i∈ alignedP_% θ(i|q)∑i ∈ m i s c l a s s i f i e d Pitalic_θ ( i | q ) < δ ⋅ ∑i ∈ a l i g n e d Pitalic_θ ( i | q ) (47) Furthermore, we bound how “deep" the misclassified negative response representations can go into the cluster of positive answer representations: mini∈a⁢l⁢i⁢g⁢n⁢e⁢d,j∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢d⁡⟨δ⁢re⁢(q),UT⁢(ei−ej)⟩>−Msubscriptformulae-sequencesubscriptsuperscriptsubscriptsubscript _i∈ aligned,j∈ misclassified\ δ r_e(q),U^T(e_i-e_% j) \>-Mminitalic_i ∈ a l i g n e d , j ∈ m i s c l a s s i f i e d ⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_i - eitalic_j ) ⟩ > - M (48) With this, the linear classification assumption can be modified as: Assumption 4. Given a query q, the change to the last hidden layer of a model due to representation engineering, δ⁢re⁢(q)=r(L)⁢(q,re)−r(L)⁢(q,0)subscriptsuperscriptsubscriptsuperscript0δ r_e(q)=r^(L)(q,r_e)-r^(L)(q,0)δ ritalic_e ( q ) = r( L ) ( q , ritalic_e ) - r( L ) ( q , 0 ) , linearly classifies the representations of positive and negative answers to a query q with margin Δ Δ, where the positive and negative answers are defined with respect to a behavior scoring function B:Σ⋆→−1,+1:→superscriptΣ⋆11B: →\-1,+1\B : Σ⋆ → - 1 , + 1 : mini∈a⁢l⁢i⁢g⁢n⁢e⁢d,j∈m⁢i⁢s⁢a⁢l⁢i⁢g⁢n⁢e⁢d⁡⟨δ⁢re⁢(q)|δ⁢re⁢(q)|,UT⁢ei−UT⁢ej⟩>Δsubscriptformulae-sequencesubscriptsubscriptsuperscriptsubscriptsuperscriptsubscriptΔ _i∈ aligned,j∈ misaligned \ δ r_e(q)% |δ r_e(q)|,U^Te_i-U^Te_j \> _i ∈ a l i g n e d , j ∈ m i s a l i g n e d ⟨ divide start_ARG δ ritalic_e ( q ) end_ARG start_ARG | δ ritalic_e ( q ) | end_ARG , Uitalic_T eitalic_i - Uitalic_T eitalic_j ⟩ > Δ (49) Up to a set of misclassified answers, whose probability is bounded by ∑i∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dPθ⁢(i|q)<δ⋅∑i∈a⁢l⁢i⁢g⁢n⁢e⁢dPθ⁢(i|q)subscriptsubscriptconditional⋅subscriptsubscriptconditional _i∈ misclassifiedP_θ(i|q)<δ· _i∈ alignedP_% θ(i|q)∑i ∈ m i s c l a s s i f i e d Pitalic_θ ( i | q ) < δ ⋅ ∑i ∈ a l i g n e d Pitalic_θ ( i | q ) that satisfy: mini∈a⁢l⁢i⁢g⁢n⁢e⁢d,j∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢d⁡⟨δ⁢re⁢(q),UT⁢(ei−ej)⟩>−Msubscriptformulae-sequencesubscriptsuperscriptsubscriptsubscript _i∈ aligned,j∈ misclassified\ δ r_e(q),U^T(e_i-e_% j) \>-Mminitalic_i ∈ a l i g n e d , j ∈ m i s c l a s s i f i e d ⟨ δ ritalic_e ( q ) , Uitalic_T ( eitalic_i - eitalic_j ) ⟩ > - M (50) Note that realistically, δ can be very small for a very large set of tokens, as in inference, LLMs typically assign high probability to few tokens and very low probability for most. Hence it suffices to classify just a few high probability tokens. We can restate theorem 1 in the following way: Theorem 3. Let δ,ϵ>0italic-ϵ0δ,ε>0δ , ϵ > 0 and let Pθ,re(⋅|q)P_θ,r_e(·|q)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) be a model prompted with query q and injected with representations of coefficient resubscriptr_eritalic_e. Let B:Σ⋆→−1,+1:→superscriptΣ⋆11B: →\-1,+1\B : Σ⋆ → - 1 , + 1 be a behavior scoring function. Under assumption 4, for re<log⁡ϵ2⁢δM⋅λsubscriptitalic-ϵ2⋅r_e< ε2δM·λritalic_e < divide start_ARG log divide start_ARG ϵ end_ARG start_ARG 2 δ end_ARG end_ARG start_ARG M ⋅ λ end_ARG the behavior expectation of the model conditioned on the query q satisfies: B[Pθ,re(⋅|q)]≥tanh(Δλ⋅re+arctanh(B0))−ϵB[P_θ,r_e(·|q)]≥ tanh( λ· r_e+arctanh(B_0))- [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] ≥ t a n h ( Δ λ ⋅ ritalic_e + a r c t a n h ( B0 ) ) - ϵ (51) Where B0=B[Pθ(⋅|q)]B_0=B[P_θ(·|q)]B0 = B [ Pitalic_θ ( ⋅ | q ) ] is the behavior expectation without steering and λ is a model dependent coefficient relating between resubscriptr_eritalic_e and the corresponding final hidden state norm. Proof: We follow the proof of theorem 1, up to equation 22, there, we introduce the misclassified tokens’ contributions, which we denote by R=∑a∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dexp(⟨r(q)+δre(q),UTea)∑a+∈g⁢o⁢o⁢dexp(⟨r(q)+δre(q),UTea+)R= _a∈ misclassifiedexp( r(q)+δ r_e(q),U^Te_a)% _a_+∈ goodexp( r(q)+δ r_e(q),U^Te_a_+)R = divide start_ARG ∑a ∈ m i s c l a s s i f i e d e x p ( ⟨ r ( q ) + δ ritalic_e ( q ) , Uitalic_T eitalic_a ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) + δ ritalic_e ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ) end_ARG: B[Pθ,re(⋅|q)]>1−∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)−R1+∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)+RB[P_θ,r_e(·|q)]> 1- _a_-∈ badP_θ(a_-% |q) _a_+∈ goodP_θ(a_+|q)exp(- |δ r|)-R1+ % _a_-∈ badP_θ(a_-|q) _a_+∈ goodP_θ(a_+|% q)exp(- |δ r|)+RB [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > divide start_ARG 1 - divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG e x p ( - Δ | δ r | ) - R end_ARG start_ARG 1 + divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG e x p ( - Δ | δ r | ) + R end_ARG (52) Following the same idea as with equation 16, we obtain that: R<∑a∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dexp(⟨r(q),UTea)∑a+∈g⁢o⁢o⁢dexp(⟨r(q),UTea+)⁢1e⁢x⁢p⁢(−|δ⁢r|⁢M)R< _a∈ misclassifiedexp( r(q),U^Te_a) _a_+∈ good% exp( r(q),U^Te_a_+) 1exp(-|δ r|M)R < divide start_ARG ∑a ∈ m i s c l a s s i f i e d e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT e x p ( ⟨ r ( q ) , Uitalic_T eitalic_a start_POSTSUBSCRIPT + end_POSTSUBSCRIPT ) end_ARG divide start_ARG 1 end_ARG start_ARG e x p ( - | δ r | M ) end_ARG (53) Plugging this in gives: B[Pθ,re(⋅|q)]>∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)−∑a−∈b⁢a⁢dPθ⁢(a−|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)−∑a∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dPθ⁢(a|q)⁢e⁢x⁢p⁢(M⁢|δ⁢r|)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)+∑a−∈b⁢a⁢dPθ⁢(a−|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)+∑a∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dPθ⁢(a|q)⁢e⁢x⁢p⁢(M⁢|δ⁢r|)>B[P_θ,r_e(·|q)]> _a_+∈ goodP_θ(a_+|q)-% _a_-∈ badP_θ(a_-|q)exp(- |δ r|)- _a∈ misclassified% P_θ(a|q)exp(M|δ r|) _a_+∈ goodP_θ(a_+|q)+Σ% _a_-∈ badP_θ(a_-|q)exp(- |δ r|)+ _a∈ misclassified% P_θ(a|q)exp(M|δ r|)>B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > divide start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) - ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) e x p ( - Δ | δ r | ) - ∑a ∈ m i s c l a s s i f i e d Pitalic_θ ( a | q ) e x p ( M | δ r | ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) + ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) e x p ( - Δ | δ r | ) + ∑a ∈ m i s c l a s s i f i e d Pitalic_θ ( a | q ) e x p ( M | δ r | ) end_ARG > (54) Denote the first second and third terms respectively as A,B,CA,B,CA , B , C: =A−B−CA+B+C=A−BA+B−CA+B1+CA+B>(A−BA+B−CA+B)⁢(1−CA+B)>A−BA+B−2⁢CA+Babsent112= A-B-CA+B+C= A-BA+B- CA+B1+ CA+B>(% A-BA+B- CA+B)(1- CA+B)> A-BA+B-2 CA+B= divide start_ARG A - B - C end_ARG start_ARG A + B + C end_ARG = divide start_ARG divide start_ARG A - B end_ARG start_ARG A + B end_ARG - divide start_ARG C end_ARG start_ARG A + B end_ARG end_ARG start_ARG 1 + divide start_ARG C end_ARG start_ARG A + B end_ARG end_ARG > ( divide start_ARG A - B end_ARG start_ARG A + B end_ARG - divide start_ARG C end_ARG start_ARG A + B end_ARG ) ( 1 - divide start_ARG C end_ARG start_ARG A + B end_ARG ) > divide start_ARG A - B end_ARG start_ARG A + B end_ARG - 2 divide start_ARG C end_ARG start_ARG A + B end_ARG (55) Notice that from the transition in equation 23: A−BA+B=t⁢a⁢n⁢h⁢(Δ⁢|δ⁢r|−ln⁡(∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q))2)ℎΔsubscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscript2 A-BA+B=tanh( |δ r|- ( _a_-∈ badP_% θ(a_-|q) _a_+∈ goodP_θ(a_+|q))2)divide start_ARG A - B end_ARG start_ARG A + B end_ARG = t a n h ( divide start_ARG Δ | δ r | - ln ( divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG ) end_ARG start_ARG 2 end_ARG ) (56) Is the bound from theorem 1, and the second term: CA+B=∑a∈m⁢i⁢s⁢c⁢l⁢a⁢s⁢s⁢i⁢f⁢i⁢e⁢dPθ⁢(a|q)⁢e⁢x⁢p⁢(M⁢|δ⁢r|)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)+∑a−∈b⁢a⁢dPθ⁢(a−|q)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)<δ⋅e⁢x⁢p⁢(M⁢|δ⁢r|)subscriptsubscriptconditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscriptΔ⋅ CA+B= _a∈ misclassifiedP_θ(a|q)exp(M|δ r|)% _a_+∈ goodP_θ(a_+|q)+ _a_-∈ badP_θ(a_-|q)% exp(- |δ r|)<δ· exp(M|δ r|)divide start_ARG C end_ARG start_ARG A + B end_ARG = divide start_ARG ∑a ∈ m i s c l a s s i f i e d Pitalic_θ ( a | q ) e x p ( M | δ r | ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) + ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) e x p ( - Δ | δ r | ) end_ARG < δ ⋅ e x p ( M | δ r | ) (57) Lastly, notice that: ∑a−∈b⁢a⁢dPθ⁢(a−|q)∑a+∈g⁢o⁢o⁢dPθ⁢(a+|q)=1−B01+B0subscriptsubscriptsubscriptconditionalsubscriptsubscriptsubscriptsubscriptconditionalsubscript1subscript01subscript0 _a_-∈ badP_θ(a_-|q) _a_+∈ goodP_θ(a% _+|q)= 1-B_01+B_0divide start_ARG ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ ( a- | q ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ ( a+ | q ) end_ARG = divide start_ARG 1 - B0 end_ARG start_ARG 1 + B0 end_ARG (58) Where B0=B[Pθ(⋅|q)]B_0=B[P_θ(·|q)]B0 = B [ Pitalic_θ ( ⋅ | q ) ], and that: a⁢r⁢c⁢t⁢a⁢n⁢h⁢(B0)=−12⁢ln⁡1−B01+B0ℎsubscript0121subscript01subscript0arctanh(B_0)=- 12 1-B_01+B_0a r c t a n h ( B0 ) = - divide start_ARG 1 end_ARG start_ARG 2 end_ARG ln divide start_ARG 1 - B0 end_ARG start_ARG 1 + B0 end_ARG (59) Thus we obtain: B[Pθ,re(⋅|q)]>tanh(Δ⁢|δ⁢r⁢(q)|2+arctanh(B0))−2δ⋅exp(M|δr|)B[P_θ,r_e(·|q)]>tanh( |δ r(q)|2+arctanh(B_0)% )-2δ· exp(M|δ r|)B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > t a n h ( divide start_ARG Δ | δ r ( q ) | end_ARG start_ARG 2 end_ARG + a r c t a n h ( B0 ) ) - 2 δ ⋅ e x p ( M | δ r | ) (60) Then, note that for coefficients that are not too large, |δ⁢r⁢(q)||δ r(q)|| δ r ( q ) | is proportional to the injected vector coefficient resubscriptr_eritalic_e (assumption 1), hence: B[Pθ,re(⋅|q)]>tanh(Δ⁢λ2⋅re+arctanh(B0))−2δ⋅exp(Mλ⋅re)B[P_θ,r_e(·|q)]>tanh( λ2· r_e+arctanh(B% _0))-2δ· exp(Mλ· r_e)B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > t a n h ( divide start_ARG Δ λ end_ARG start_ARG 2 end_ARG ⋅ ritalic_e + a r c t a n h ( B0 ) ) - 2 δ ⋅ e x p ( M λ ⋅ ritalic_e ) (61) Where λ is the coefficient relating resubscriptr_eritalic_e to |δ⁢r⁢(q)||δ r(q)|| δ r ( q ) |. Thus for re<log⁡ϵ2⁢δM⋅λsubscriptitalic-ϵ2⋅r_e< ε2δM·λritalic_e < divide start_ARG log divide start_ARG ϵ end_ARG start_ARG 2 δ end_ARG end_ARG start_ARG M ⋅ λ end_ARG: B[Pθ,re(⋅|q)]>tanh(Δ⁢λ2⋅re+arctanh(B0))−ϵB[P_θ,r_e(·|q)]>tanh( λ2· r_e+arctanh(B% _0))- [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > t a n h ( divide start_ARG Δ λ end_ARG start_ARG 2 end_ARG ⋅ ritalic_e + a r c t a n h ( B0 ) ) - ϵ (62) Appendix I Relation of Steering to Finetuning with Preference Learning To a degree one can draw a relation between steering and preference learning. Proposition 1. For an LLM, one iteration of gradient descent on the preference learning loss with learning rate η is equivalent to steering with coefficient re=ηsubscriptr_e= _e = η. Proof: The objective in preference learning is to minimize the loss: L=−(x,y+,y−)∼D⁢[log⁡P⁢(y+|x)P⁢(y−|x)]=−(x,y+,y−)∼D⁢[⟨rx(L),UT⁢(ey+−ey−)⟩]subscriptsimilar-tosuperscriptsuperscriptdelimited-[]conditionalsuperscriptconditionalsuperscriptsubscriptsimilar-tosuperscriptsuperscriptdelimited-[]subscriptsuperscriptsuperscriptsubscriptsubscriptsubscriptsubscriptL=-E_(x,y^+,y^-) D[ P(y^+|x)P(y^-|x)]=-% E_(x,y^+,y^-) D[ r^(L)_x,U^T(e_y_+-e_y_-% ) ]L = - blackboard_E( x , y+ , y- ) ∼ D [ log divide start_ARG P ( y+ | x ) end_ARG start_ARG P ( y- | x ) end_ARG ] = - blackboard_E( x , y+ , y- ) ∼ D [ ⟨ r( L )x , Uitalic_T ( eitalic_y start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - eitalic_y start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ) ⟩ ] (63) Which increases the likelihood of desired responses to prompts. By training with preference learning, in each iteration of gradient descent, each representation is changed by: r(l)→r(l)−η⁢∂L∂r(l)→superscriptsuperscriptsuperscriptr^(l)→ r^(l)-η ∂ L∂ r^(l)r( l ) → r( l ) - η divide start_ARG ∂ L end_ARG start_ARG ∂ r( l ) end_ARG (64) The gradient of the loss w.r.t. a hidden layer representation is: ∂L∂rl=(x,y+,y−)∼D⁢[∂r⁢(x)∂rl⁢(x)⋅UT⁢(ey+−ey−)]superscriptsubscriptsimilar-tosuperscriptsuperscriptdelimited-[]⋅superscriptsuperscriptsubscriptsubscriptsubscriptsubscript ∂ L∂ r^l=E_(x,y^+,y^-) D[ % ∂ r(x)∂ r^l(x)· U^T(e_y_+-e_y_-)]divide start_ARG ∂ L end_ARG start_ARG ∂ ritalic_l end_ARG = blackboard_E( x , y+ , y- ) ∼ D [ divide start_ARG ∂ r ( x ) end_ARG start_ARG ∂ ritalic_l ( x ) end_ARG ⋅ Uitalic_T ( eitalic_y start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - eitalic_y start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ) ] (65) Thus at each layer, the representation is shifted in a direction that maximizes the difference between positive and negative responses’ representations, UT⁢(ey+−ey−)superscriptsubscriptsubscriptsubscriptsubscriptU^T(e_y_+-e_y_-)Uitalic_T ( eitalic_y start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - eitalic_y start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ). Which is equivalent to steering with coefficient re=ηsubscriptr_e= _e = η, and vectors Re=(x,y+,y−)∼D⁢[∂r⁢(x)∂rl⁢(x)⋅UT⁢(ey+−ey−)]l=1Lsubscriptsuperscriptsubscriptsubscriptsimilar-tosuperscriptsuperscriptdelimited-[]⋅superscriptsuperscriptsubscriptsubscriptsubscriptsubscript1R_e=\E_(x,y^+,y^-) D[ ∂ r(x)∂ r^l% (x)· U^T(e_y_+-e_y_-)]\_l=1^LRitalic_e = blackboard_E( x , y+ , y- ) ∼ D [ divide start_ARG ∂ r ( x ) end_ARG start_ARG ∂ ritalic_l ( x ) end_ARG ⋅ Uitalic_T ( eitalic_y start_POSTSUBSCRIPT + end_POSTSUBSCRIPT - eitalic_y start_POSTSUBSCRIPT - end_POSTSUBSCRIPT ) ] l = 1L Appendix J Extension of Results Beyond Binary Behavior Score The idea behind theorem 1, is that the resulting change to the final hidden layer due to the representation injections linearly classifies aligned and misaligned answers, where the aligned/misaligned labels are given by the binary behavior scoring function. To extend beyond a binary behavior score, we need to assume that the model’s latent space captures more finegrained differences between answers. Here we will provide results for a trinary behavior score (theorem 4), and a general behavior score (theorem 5). A natural extension is for a trinary score function, where ±1plus-or-minus1± 1± 1 is aligned/misaligned, and 00 is irrelevant/neutral. We can reformulate theorem 1 in the following way: Theorem 4. Let Pθ,re(⋅|q)P_θ,r_e(·|q)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) be a model prompted with query q and injected with representations of coefficient resubscriptr_eritalic_e. Let B:Σ∗→−1,0,+1:→superscriptΣ101B: ^*→\-1,0,+1\B : Σ∗ → - 1 , 0 , + 1 be a behavior scoring function. The injections to all layers amounts to a change in the final hidden layer representation that is q dependent, denoted by the vector δ⁢re(L)⁢(q)subscriptsuperscriptδ r^(L)_e(q)δ r( L )e ( q ). Assume that the representations of aligned and misaligned/irrelevant answers w.r.t. B are linearly separable, and that δ⁢re(L)⁢(q)subscriptsuperscriptδ r^(L)_e(q)δ r( L )e ( q ) linearly classifies them with margin Δ Δ. Then, the behavior expectation of the model conditioned on the query q satisfies: B[Pθ,re(⋅|q)]≥B0+P+⁢(eΔ⁢λ⋅re−1)1+P+⁢(eΔ⁢λ⋅re−1)B[P_θ,r_e(·|q)]≥ B_0+P_+(e λ· r_e% -1)1+P_+(e λ· r_e-1)B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] ≥ divide start_ARG B0 + P+ ( eroman_Δ λ ⋅ ritalic_e - 1 ) end_ARG start_ARG 1 + P+ ( eroman_Δ λ ⋅ ritalic_e - 1 ) end_ARG (66) Where B0=B[Pθ(⋅|q)]B_0=B[P_θ(·|q)]B0 = B [ Pitalic_θ ( ⋅ | q ) ] and P+subscriptP_+P+ are the behavior expectation and probability of aligned answer without steering, and λ is a model dependent coefficient relating between resubscriptr_eritalic_e and the corresponding final hidden state norm. The behavior bound has a different form, but it behaves the same – for re=0subscript0r_e=0ritalic_e = 0, it coincides with B0subscript0B_0B0, around re=0subscript0r_e=0ritalic_e = 0 it is linear, and for re→∞→subscriptr_e→∞ritalic_e → ∞ it approaches +11+1+ 1. The proof, presented in J.1, essentially follows the proof of theorem 1, except besides the P±subscriptplus-or-minusP_±P± terms (probability mass of positive and negative responses without steering) there is also a P0subscript0P_0P0 term. For a general behavior scoring function, B:Σ∗→[−1,+1]:→superscriptΣ11B: ^*→[-1,+1]B : Σ∗ → [ - 1 , + 1 ], we can similarly assume that the representations of answers with score >b+absentsubscript>b_+> b+ and answers with score <b+absentsubscript<b_+< b+, are linearly separable, and obtain the following result: Theorem 5. Let Pθ,re(⋅|q)P_θ,r_e(·|q)Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) be a model prompted with query q and injected with representations of coefficient resubscriptr_eritalic_e. Let B:Σ∗→[−1,+1]:→superscriptΣ11B: ^*→[-1,+1]B : Σ∗ → [ - 1 , + 1 ] be a behavior scoring function. The injections to all layers amounts to a change in the final hidden layer representation that is q dependent, denoted by the vector δ⁢re(L)⁢(q)subscriptsuperscriptδ r^(L)_e(q)δ r( L )e ( q ). Assume that the representations of answers with behavior score >b+absentlimit-from>b+> b + and those with score <b+absentsubscript<b_+< b+ w.r.t. B are linearly separable, and that δ⁢re(L)⁢(q)subscriptsuperscriptδ r^(L)_e(q)δ r( L )e ( q ) linearly classifies them with margin Δ Δ. Then, the behavior expectation of the model conditioned on the query q satisfies: B[Pθ,re(⋅|q)]≥b+⁢P+⁢eΔ⁢λ⁢re−P−P+⁢eΔ⁢λ⁢re+P−B[P_θ,r_e(·|q)]≥ b_+P_+e λ r_e-P_-% P_+e λ r_e+P_-B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] ≥ divide start_ARG b+ P+ eroman_Δ λ ritalic_e - P- end_ARG start_ARG P+ eroman_Δ λ ritalic_e + P- end_ARG (67) Where P±subscriptplus-or-minusP_±P± are the probabilities of aligned/misaligned answers without steering, and λ is a model dependent coefficient relating between resubscriptr_eritalic_e and the corresponding final hidden state norm. Here we see that the behavior expectation converges to the maximal score b+subscriptb_+b+, for which δ⁢re(L)superscriptsubscriptδ r_e^(L)δ ritalic_e( L ) can classify answers below and above the score. The trend is similar to theorem 1, with a sigmoidal behavior, but without the tightness on behavior expectation at re=0subscript0r_e=0ritalic_e = 0, due to the more complex behavior scoring function. The proof is presented in J.2. J.1 Proof of theorem 4 Following the same proof as in 1, up to equation 22, but replacing the sum over negative answers to sum over negative and neutral answers, we obtain by denoting P±subscriptplus-or-minusP_±P±, the sum over positive/negative answers without steering, and by P0subscript0P_0P0 sum over neutral answers: B[Pθ,re(⋅|q)]≥P+−P−⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)P++(P−+P0)⁢e⁢x⁢p⁢(−Δ⁢|δ⁢r|)B[P_θ,r_e(·|q)]≥ P_+-P_-exp(- |δ r|)P_+% +(P_-+P_0)exp(- |δ r|)B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] ≥ divide start_ARG P+ - P- e x p ( - Δ | δ r | ) end_ARG start_ARG P+ + ( P- + P0 ) e x p ( - Δ | δ r | ) end_ARG (68) =P+⁢(eΔ⁢|δ⁢r|−1)+(P+−P−)P+⁢(eΔ⁢|δ⁢r|−1)+(P++P−+P0)absentsubscriptsuperscriptΔ1subscriptsubscriptsubscriptsuperscriptΔ1subscriptsubscriptsubscript0= P_+(e |δ r|-1)+(P_+-P_-)P_+(e |δ r|% -1)+(P_++P_-+P_0)= divide start_ARG P+ ( eroman_Δ | δ r | - 1 ) + ( P+ - P- ) end_ARG start_ARG P+ ( eroman_Δ | δ r | - 1 ) + ( P+ + P- + P0 ) end_ARG (69) We note that P++P−+P0=1subscriptsubscriptsubscript01P_++P_-+P_0=1P+ + P- + P0 = 1 and that P+−P−=B[Pθ,re=0(⋅|q)]=B0P_+-P_-=B[P_θ,r_e=0(·|q)]=B_0P+ - P- = B [ Pitalic_θ , r start_POSTSUBSCRIPT e = 0 end_POSTSUBSCRIPT ( ⋅ | q ) ] = B0: =P+⁢(eΔ⁢|δ⁢r|−1)+B0P+⁢(eΔ⁢|δ⁢r|−1)+1absentsubscriptsuperscriptΔ1subscript0subscriptsuperscriptΔ11= P_+(e |δ r|-1)+B_0P_+(e |δ r|-1)+1= divide start_ARG P+ ( eroman_Δ | δ r | - 1 ) + B0 end_ARG start_ARG P+ ( eroman_Δ | δ r | - 1 ) + 1 end_ARG (70) Lastly, applying assumption 1, replaces |δ⁢r|=λ⁢resubscript|δ r|=λ r_e| δ r | = λ ritalic_e. J.2 Proof of theorem 5 Following the same proof idea as in theorem 1, starting with equation 12 but replacing the scores in the numerator for positive and negative answers with b+subscriptb_+b+ and −11-1- 1 (for worst case), up to equation 22, denote by P+subscriptP_+P+ the probability without steering for answers with score >b+absentsubscript>b_+> b+ and by P−subscriptP_-P- the rest: B[Pθ,re(⋅|q)]≥b+⁢P+⁢eΔ⁢|δ⁢r|−P−P+⁢eΔ⁢|δ⁢r|+P−B[P_θ,r_e(·|q)]≥ b_+P_+e |δ r|-P_-P_% +e |δ r|+P_-B [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] ≥ divide start_ARG b+ P+ eroman_Δ | δ r | - P- end_ARG start_ARG P+ eroman_Δ | δ r | + P- end_ARG (71) Lastly, applying assumption 1, replaces |δ⁢r|=λ⁢resubscript|δ r|=λ r_e| δ r | = λ ritalic_e. Appendix K Extension of Results to Multi-Token Answers Intuitively, both the alignment guarantee result of theorem 1 and helpfulness bound of theorem 2, which apply for a single token output, can be extended to multi-token answers by applying the results on multiple decoding steps. K.1 Alignment Starting with alignment, we note that if the model is limited to producing N tokens, then from corollary 1, we can ensure that with a large enough steering coefficient, each token will correspond to an aligned response: Theorem 6. Let ϵ>0italic-ϵ0ε>0ϵ > 0, PθsubscriptP_θPitalic_θ a language model, B:Σ∗→−1,+1:→superscriptΣ11B: ^*→\-1,+1\B : Σ∗ → - 1 , + 1 , behavior scoring function and q a query, and suppose the model’s reply contains at most N tokens. Under the assumption of theorem 1 holding in every decoding step, for re>1Δ⁢λ⁢(log⁡Nϵ+log⁡1−B01+B0)subscript1Δitalic-ϵ1subscript01subscript0r_e> 1 λ( Nε+ 1-B_01+B_0% )ritalic_e > divide start_ARG 1 end_ARG start_ARG Δ λ end_ARG ( log divide start_ARG N end_ARG start_ARG ϵ end_ARG + log divide start_ARG 1 - B0 end_ARG start_ARG 1 + B0 end_ARG ), then: B[Pθ(⋅|q)]>1−2ϵB[P_θ(·|q)]>1-2 [ Pitalic_θ ( ⋅ | q ) ] > 1 - 2 ϵ (72) Where B0subscript0B_0B0 is the behavior expectation without representation engineering. We see that larger coefficients of steering improve the behavior expectation, similarly to corollary 1, but with multiple token answers. By inverting the relation between resubscriptr_eritalic_e and ϵitalic-ϵεϵ, and placing it in the behavior expectation bound, we obtain a sigmoid-like behavior, that is linear for re≈0subscript0r_e≈ 0ritalic_e ≈ 0. Proof: Following the notation of the proof of theorem 1, we note that at each decoding step, the probability of outputting a token aisubscripta_iaitalic_i that is aligned w.r.t. behavior scoring function B, conditioned on the previous context q⁢a1⁢…⁢ai−1subscript1…subscript1qa_1...a_i-1q a1 … aitalic_i - 1, is: ∑a+∈g⁢o⁢o⁢dPθ,re⁢(a+|q⁢a1⁢…⁢ai−1)∑a+∈g⁢o⁢o⁢dPθ,re⁢(a+|q⁢a1⁢…⁢ai−1)+∑a−∈b⁢a⁢dPθ,re⁢(a−|q⁢a1⁢…⁢ai−1)subscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscript1…subscript1subscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscript1…subscript1subscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscript1…subscript1 _a_+∈ goodP_θ,r_e(a_+|qa_1...a_i-1) _a_+% ∈ goodP_θ,r_e(a_+|qa_1...a_i-1)+ _a_-∈ badP_% θ,r_e(a_-|qa_1...a_i-1)divide start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a+ | q a1 … aitalic_i - 1 ) end_ARG start_ARG ∑a start_POSTSUBSCRIPT + ∈ g o o d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a+ | q a1 … aitalic_i - 1 ) + ∑a start_POSTSUBSCRIPT - ∈ b a d end_POSTSUBSCRIPT Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( a- | q a1 … aitalic_i - 1 ) end_ARG (73) Following the proof technique of theorem 1, we obtain that this probability is larget than: ≥P+⁢eΔ⁢λ⁢reP+⁢eΔ⁢λ⁢re+P−absentsubscriptsuperscriptΔsubscriptsubscriptsuperscriptΔsubscriptsubscript≥ P_+e λ r_eP_+e λ r_e+P_-≥ divide start_ARG P+ eroman_Δ λ ritalic_e end_ARG start_ARG P+ eroman_Δ λ ritalic_e + P- end_ARG (74) Where P±subscriptplus-or-minusP_±P± are the probabilities for an aligned/misaligned output at the given decoding step. To ensure this probability is larger than 1−ϵ′1superscriptitalic-ϵ′1-ε 1 - ϵ′, we demand: re>log⁡P−P++log⁡1ϵ′Δ⁢λsubscriptsubscriptsubscript1superscriptitalic-ϵ′Δr_e> P_-P_++ 1ε λritalic_e > divide start_ARG log divide start_ARG P- end_ARG start_ARG P+ end_ARG + log divide start_ARG 1 end_ARG start_ARG ϵ′ end_ARG end_ARG start_ARG Δ λ end_ARG (75) Thus over N decoding steps, we use a union bound, leading to a positive response with probability (1−ϵ′)N>(1−ϵ′⁢N)superscript1superscriptitalic-ϵ′1superscriptitalic-ϵ′(1-ε )^N>(1-ε N)( 1 - ϵ′ )N > ( 1 - ϵ′ N ). Taking ϵ′=ϵ/Nsuperscriptitalic-ϵ′italic-ϵε =ε/Nϵ′ = ϵ / N, we obtain: re>maxi∈[N]⁡log⁡P−iP+i+log⁡NϵΔ⁢λsubscriptsubscriptdelimited-[]subscriptsuperscriptsubscriptsuperscriptitalic-ϵΔr_e> _i∈[N]\ P^i_-P^i_+\+ N% ε λritalic_e > divide start_ARG maxitalic_i ∈ [ N ] log divide start_ARG Pitalic_i- end_ARG start_ARG Pitalic_i+ end_ARG + log divide start_ARG N end_ARG start_ARG ϵ end_ARG end_ARG start_ARG Δ λ end_ARG (76) Where P±isuperscriptsubscriptplus-or-minusP_±^iP±italic_i is the probability for a positive/negative continuation in the i’th token of the response. We note that P−iP+i=1−B0i1+B0isuperscriptsubscriptsuperscriptsubscript1subscriptsuperscript01subscriptsuperscript0 P_-^iP_+^i= 1-B^i_01+B^i_0divide start_ARG P-i end_ARG start_ARG P+i end_ARG = divide start_ARG 1 - Bitalic_i0 end_ARG start_ARG 1 + Bitalic_i0 end_ARG, where B0isuperscriptsubscript0B_0^iB0italic_i is the behavior expectation at the i’th decoding step. For the response to be positive, it is required that every step is positive, due to the binary score, then the behavior expectation of the entire response is no larger than the behavior expectation of each decoding step, B0≤mini∈[N]⁡B0isubscript0subscriptdelimited-[]superscriptsubscript0B_0≤ _i∈[N]B_0^iB0 ≤ minitalic_i ∈ [ N ] B0italic_i, meaning it suffices to have: re>log⁡1−B01+B0+log⁡NϵΔ⁢λsubscript1subscript01subscript0italic-ϵΔr_e> 1-B_01+B_0+ Nε λritalic_e > divide start_ARG log divide start_ARG 1 - B0 end_ARG start_ARG 1 + B0 end_ARG + log divide start_ARG N end_ARG start_ARG ϵ end_ARG end_ARG start_ARG Δ λ end_ARG (77) We obtain that under these conditions, an aligned response is generated with probability at least 1−ϵ1italic-ϵ1- 1 - ϵ. A negative response, is generated with probability no greater than ϵitalic-ϵεϵ. Thus the behavior expectation is at least: B[Pθ,re(⋅|q)]>1−2ϵB[P_θ,r_e(·|q)]>1-2 [ Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( ⋅ | q ) ] > 1 - 2 ϵ (78) K.2 Helpfulness For helpfulness, we will consider a query q and a correct answer a of N tokens. We will show that the probability of the answer decreases quadratically. The intuition is that in each decoding step the probability decreases quadratically, and due to the probability chain rule, if at the i’th step of generation, the probability for the next token is PisubscriptP_iPitalic_i, then the full sequence probability is ∏i=1NPisuperscriptsubscriptproduct1subscript _i=1^NP_i∏i = 1N Pitalic_i. Once we expand this term w.r.t. resubscriptr_eritalic_e, we get a leading quadratic dependence: Corollary 3. Let PθsubscriptP_θPitalic_θ be a language model and q be a query with answer a=a1⁢…⁢aNsubscript1…subscripta=a_1...a_Na = a1 … aitalic_N containing at most N tokens. Denote by P0ii=1Nsuperscriptsubscriptsuperscriptsubscript01\P_0^i\_i=1^N P0italic_i i = 1N the probability assigned to each correct token aii=1Nsuperscriptsubscriptsubscript1\a_i\_i=1^N aitalic_i i = 1N in the sequence without steering, such that the probability of the full sequence is P0=∏i=1NP0isubscript0superscriptsubscriptproduct1superscriptsubscript0P_0= _i=1^NP_0^iP0 = ∏i = 1N P0italic_i. Then under the conditions of theorem 2 holding at each decoding step, we have with probability of at least 1−2⁢NT121- 2NT1 - divide start_ARG 2 N end_ARG start_ARG T end_ARG: Pθ,re⁢(q)≤P0∏i=1N(P0i+(1−P0i)⁢α⁢(1−ϵ)⁢(1+λ2⁢σ2⁢β22⁢re2))subscriptsubscriptsubscript0superscriptsubscriptproduct1superscriptsubscript01superscriptsubscript01italic-ϵ1superscript2superscript2superscript22superscriptsubscript2P_θ,r_e(q)≤ P_0 _i=1^N(P_0^i+(1-P_0^i)% α(1-ε)(1+ λ^2σ^2β^22r_e^2))Pitalic_θ , r start_POSTSUBSCRIPT e end_POSTSUBSCRIPT ( q ) ≤ divide start_ARG P0 end_ARG start_ARG ∏i = 1N ( P0italic_i + ( 1 - P0italic_i ) α ( 1 - ϵ ) ( 1 + divide start_ARG λ2 σ2 β2 end_ARG start_ARG 2 end_ARG ritalic_e2 ) ) end_ARG (79) This shows the original probability of the sequence P0subscript0P_0P0, is normalized by a term whose leading order is quadratic in resubscriptr_eritalic_e: ∏i=1N(P0i+(1−P0i)α(1−ϵ)(1+λ2⁢σ2⁢β22re2))=∏i=1N(P0i+(1−P0i)α(1−ϵ)))+c⋅re2+o(re2) _i=1^N(P_0^i+(1-P_0^i)α(1-ε)(1+ λ^2% σ^2β^22r_e^2))= _i=1^N(P_0^i+(1-P_0^i)% α(1-ε)))+c· r_e^2+o(r_e^2)∏i = 1N ( P0italic_i + ( 1 - P0italic_i ) α ( 1 - ϵ ) ( 1 + divide start_ARG λ2 σ2 β2 end_ARG start_ARG 2 end_ARG ritalic_e2 ) ) = ∏i = 1N ( P0italic_i + ( 1 - P0italic_i ) α ( 1 - ϵ ) ) ) + c ⋅ ritalic_e2 + o ( ritalic_e2 ) (80) We once a gain note that if P0isuperscriptsubscript0P_0^iP0italic_i is close to 1111, then (P0i+(1−P0i)α(1−ϵ)))≈1(P_0^i+(1-P_0^i)α(1-ε)))≈ 1( P0italic_i + ( 1 - P0italic_i ) α ( 1 - ϵ ) ) ) ≈ 1, making the bound tighter where the model is more helpful initially. An alternative bound, is simply to consider that the probability for a sequence, P0subscript0P_0P0, is bounded by the probability of each element in the sequence, P0isuperscriptsubscript0P_0^iP0italic_i, for which theorem 2 can be directly applied, and the quadratic decay is achieved, although this is a bound that is less tight.

Instant research discovery