Paper deep dive

PaCE: Parsimonious Concept Engineering for Large Language Models

Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker, Aditya Chattopadhyay, Chris Callison-Burch, René Vidal

Year: 2024Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 129

Models: LLaMA2-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:41:30 PM

Summary

PaCE (Parsimonious Concept Engineering) is an activation engineering framework for aligning Large Language Models (LLMs). It constructs a large-scale concept dictionary (PaCE-1M) and uses sparse coding to decompose LLM activations into benign and undesirable components. By removing undesirable components via oblique projection, PaCE reorients LLM behavior toward alignment goals while preserving linguistic capabilities, outperforming traditional fine-tuning and vector addition methods.

Entities (5)

LLM · model · 100%PaCE · framework · 100%Oblique Projection · technique · 95%PaCE-1M · dataset · 95%Sparse Coding · technique · 95%

Relation Signals (3)

PaCE → uses → PaCE-1M

confidence 100% · we collect a large concept dictionary, PaCE-1M... to sufficiently model the concepts

PaCE → improves → LLM

confidence 95% · PaCE achieves state-of-the-art alignment performance while maintaining linguistic capabilities.

Sparse Coding → decomposes → LLM

confidence 90% · decompose the LLM activations along the concept dictionary via sparse coding

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Models (LLMs) are being used for a wide variety of tasks. While they are capable of generating human-like responses, they can also produce undesirable output including potentially harmful information, racist or sexist language, and hallucinations. Alignment methods are designed to reduce such undesirable outputs via techniques such as fine-tuning, prompt engineering, and representation engineering. However, existing methods face several challenges: some require costly fine-tuning for every alignment task; some do not adequately remove undesirable concepts, failing alignment; some remove benign concepts, lowering the linguistic capabilities of LLMs. To address these issues, we propose Parsimonious Concept Engineering (PaCE), a novel activation engineering framework for alignment. First, to sufficiently model the concepts, we construct a large-scale concept dictionary in the activation space, in which each atom corresponds to a semantic concept. Given any alignment task, we instruct a concept partitioner to efficiently annotate the concepts as benign or undesirable. Then, at inference time, we decompose the LLM activations along the concept dictionary via sparse coding, to accurately represent the activations as linear combinations of benign and undesirable components. By removing the latter ones from the activations, we reorient the behavior of the LLM towards the alignment goal. We conduct experiments on tasks such as response detoxification, faithfulness enhancement, and sentiment revising, and show that PaCE achieves state-of-the-art alignment performance while maintaining linguistic capabilities.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

128,482 characters extracted from source content.

Expand or collapse full text

PaCE: Parsimonious Concept Engineering for Large Language Models Jinqi Luo ♠ Tianjiao Ding∗♠ Kwan Ho Ryan Chan♠ Darshan Thaker♠ Aditya Chattopadhyay♣ Chris Callison-Burch♠ René Vidal♠ ♠University of Pennsylvania ♣Johns Hopkins University jinqiluo,tjding@upenn.edu Equal contribution. Abstract Large Language Models (LLMs) are being used for a wide variety of tasks. While they are capable of generating human-like responses, they can also produce undesirable output including potentially harmful information, racist or sexist language, and hallucinations. Alignment methods are designed to reduce such undesirable outputs via techniques such as fine-tuning, prompt engineering, and representation engineering. However, existing methods face several challenges: some require costly fine-tuning for every alignment task; some do not adequately remove undesirable concepts, failing alignment; some remove benign concepts, lowering the linguistic capabilities of LLMs. To address these issues, we propose Parsimonious Concept Engineering (PaCE), a novel activation engineering framework for alignment. First, to sufficiently model the concepts, we construct a large-scale concept dictionary in the activation space, in which each atom corresponds to a semantic concept. Given any alignment task, we instruct a concept partitioner to efficiently annotate the concepts as benign or undesirable. Then, at inference time, we decompose the LLM activations along the concept dictionary via sparse coding, to accurately represent the activations as linear combinations of benign and undesirable components. By removing the latter ones from the activations, we reorient the behavior of the LLM towards the alignment goal. We conduct experiments on tasks such as response detoxification, faithfulness enhancement, and sentiment revising, and show that PaCE achieves state-of-the-art alignment performance while maintaining linguistic capabilities. Our collected dataset for concept representations is available at https://github.com/peterljq/Parsimonious-Concept-Engineering. 1 Introduction Large Language Models (LLMs) are useful for tasks as far ranging as question answering [77, 65], symbolic reasoning [56, 25], multi-modal synthesis [86, 40, 44], and medical diagnosis [85]. LLMs are typically pre-trained on a broad collection of textual corpora with the next-token prediction objective [54, 70], enabling them to generate human-like text. An important aspect of deploying pre-trained LLMs for real-world applications is preventing undesirable responses such as toxic language, hallucinations, and biased information through alignment methods, which aim to make AI systems behave in line with human intentions and values [28]. A common alignment approach is tuning LLMs with human feedback [55, 62] for better instruction-following capabilities. However, after such aligning, undesirable and harmful content can still be elicited from LLMs. For example, jailbreaking can produce hate speech and aggression [22, 32], stress-testing shows hallucinatory responses such as illogical statements [87], and various kinds of biases are not fully removed from LLM responses [19]. This emphasizes the need for further development of aligned LLMs. Overall, alignment methods can largely be categorized into: parameter fine-tuning, prompt engineering, and activation engineering. Parameter fine-tuning methods, such as low-rank adaptation [26] and knowledge editing [14, 74], involve updating the model parameters using datasets of input-response pairs [75]. Unfortunately, such computations over large datasets are often costly. Furthermore, whenever a new category of undesirable behaviors is identified or a new group of customers is acquired, the LLM supplier has to incur the cost of data creation and fine-tuning again. Prompt engineering attempts to manipulate the LLM’s reasoning with carefully designed instruction prompts [78, 81, 80]. However, effective instructions are commonly obtained through empirical trial-and-error, with no guarantee of coverage across tasks of different domains. Notably, recent works show that the instruction itself can be lengthy [38] or contain human errors [10, 61]. Figure 1: Our framework PaCE achieves alignment goals by sparse coding and adjusting vectors in the activation space of the LLM Decoder Layer (DCL). Activation engineering, i.e., algorithms that modify the latent activations of LLMs, has emerged to alleviate high-cost and poor coverage of tasks. Recent work has shown that certain directions in the activation space of LLMs are associated with semantic concepts (c.f. §2.1). Thus, given an input prompt at inference time, modifying its neural activations towards or away from these directions controls the semantics of the model response. For example, methods based on Vector Addition (VecAdd) [67, 72, 88, 43, 53, 68, 69, 37] directly add multiples of a concept direction to a neural activation, while those based on Orthogonal Projection (OrthoProj) [88, 23] subtract from a neural activation its orthogonal projection onto a concept direction. Nonetheless, these methods face two major challenges. First, these methods inadequately model the geometry of the activation space, as we will detail in §2.2. Hence, they tend to either remove benign concepts, harming linguistic capability; or insufficiently remove undesirable concepts, thereby failing the alignment task. Second, for each alignment task, these methods typically only remove a single concept direction from the input activation vector, while there may be multiple concepts related to the alignment task. To address these challenges, we propose Parsimonious Concept Engineering (PaCE), an activation engineering framework for alignment that i) enforces alignment goals effectively and efficiently, i) retains linguistic capability, and i) adapts to new alignment goals without costly parameter fine-tuning. PaCE consists of two stages: (1) Concept Construction and Partition, and (2) Activation Decomposition and Intervention (Figure 3). We summarize the procedure of PaCE below and highlight our contributions in bold. • Concept Dictionary Construction and Partition (§3.2): Since existing works only provide a limited number of concept directions, we collect a large concept dictionary, PaCE-1M, that consists of 40,000 concept directions extracted from over 1,200,000 context sentences. In particular, for each concept in the Brown Corpus [18], we use a knowledge-driven GPT [65, 44, 35] to propose contextual scenarios to describe the concept, and extract concept directions in the representation (activation) space [88] from the context sentences. This is done only once offline. Further, given any alignment task, we instruct a GPT to automatically partition the concept directions in the dictionary into benign and undesirable directions, which is done once per task offline. • Activation Decomposition and Intervention (§3.3): At inference time, given any user input prompt, we decompose the activations as a sparse linear combination of concept directions using sparse coding techniques. Notably, this allows for an efficient and accurate estimate of both undesirable and benign components in the activations, which is overlooked in previous activation engineering methods. By removing the undesirable components from the activations, we reorient the behavior of LLMs toward alignment goals, while maintaining their linguistic capability. We evaluate PaCE on alignment tasks including response detoxification, faithfulness enhancement, and sentiment revising (§4). We show that PaCE achieves state-of-the-art performance on these tasks, while retaining its linguistic capability at a comparable level. We further shed insights on the concept directions of PaCE-1M: concept directions tend to form clusters with directions from each cluster corresponding to similar semantics, and decomposing an activation reveals its semantics. 2 Basics of Latent Space Engineering As motivated above, in this paper we are interested in controlling LLMs by leveraging structures in their latent space. We begin by reviewing some basic properties of the latent space in §2.1. This lays the foundation for previous methods on latent space intervention in §2.2 as well as our method in §3. Figure 2: To remove a concept direction ‘red’ from the latent code ‘red apple’ (left), prior works use i) orthogonal projection (middle right, (OrthoProj)), which may remove extra directions, or i) vector addition (right, (VecAdd)), where it is hard to pick the edit strength c. Instead, PaCE explicitly models the concept dictionary in the latent space and use oblique projection (middle left). 2.1 The Latent Space and Its Linear Controllability Let ⊂ℝdsuperscriptℝZ⊂R^dZ ⊂ blackboard_Rd denote a latent space whose elements can be mapped into text. That is, there exists a (surjective) decoder g:→:→g:Z→Tg : Z → T where TT is some set of texts. For ease of notation, we follow the convention and use some text∈subscriptsome text z_some text∈Zitalic_zsome text ∈ Z to denote an element in the pre-image g−1⁢(‘some text’)superscript1‘some text’g^-1(`some text')g- 1 ( ‘some text’ ). Linear Controllability. Consider the word pairs (‘France’, ‘Paris’) and (‘Japan’, ‘Tokyo’) – the latter is the capital of the former. It is natural to wonder if their latent codes have such correspondence. In various settings as we will review, there is approximately a linear relation: there exists a capital∈ℝdsubscriptcapitalsuperscriptℝ v_capital∈R^ditalic_vcapital ∈ blackboard_Rd, such that France+c⋅capital≈ParissubscriptFrance⋅subscriptcapitalsubscriptParis z_France+c· v_capital% ≈ z_Parisitalic_zFrance + c ⋅ italic_vcapital ≈ italic_zParis for some control strength c>00c>0c > 0, and Japan+c′⋅capital ≈TokyosubscriptJapan⋅superscript′subscriptcapital subscriptTokyo z_Japan+c · v_capital% ≈ z_Tokyoitalic_zJapan + c′ ⋅ italic_vcapital ≈ italic_zTokyo for some c′>0superscript′0c >0c′ > 0. Beyond this example, prior works seem to support the existence of a set of concept directions ⊂ℝdsuperscriptℝV⊂R^dV ⊂ blackboard_Rd that linearly relate pairs of latent codes111capital∈subscriptcapital v_capital∈Vitalic_vcapital ∈ V typically can not be decoded by g to obtain the text ‘capital’, as opposed to elements in ZZ. . Note, however, that the notion of linear controllability is different from the notion linear or affine combination in linear algebra in that there may be only one choice of c such that +c⁢∈ z+c v∈Zitalic_z + c italic_v ∈ Z. Remark 1 (=absentZ=Z = Word Embeddings). A classic setting where linear controllability shows up is that of word embeddings. Here, TT is the vocabulary (say, the set of English words), ZZ contains some vectors in ℝdsuperscriptℝR^dblackboard_Rd, and g is a bijection between ZZ and TT. In the seminal work of Mikolov et al. [50], the authors observe that word embeddings learned by recurrent neural networks approximately enjoy relations such as king−man+woman≈queensubscriptkingsubscriptmansubscriptwomansubscriptqueen z_king- z_man+ z_% woman≈ z_queenitalic_zking - italic_zman + italic_zwoman ≈ italic_zqueen, where one can view woman−mansubscriptwomansubscriptman z_woman- z_manitalic_zwoman - italic_zman as the concept direction ∈ v∈Vitalic_v ∈ V and c=11c=1c = 1 as the control strength. This observation is later extended to word embeddings of various networks and learning objectives such as word2vec [49], Skip-Grams [48, 34], GloVe [59], and Swivel [63]. On the theoretical front, a fruitful line of research has been devoted to understanding the emergence of such properties in word embeddings [3, 21, 2, 1, 17, 52]. Remark 2 (=absentZ=Z = Neural Activations). Modern neural architectures such as transformers have significantly boosted the linguistic performance of language models. Much of their success is attributed to the attention mechanism, which incorporates long-range context into the neural activations in transformers. This has motivated people to take ZZ as certain hidden states in transformers222A variety of choices of layers have been explored in the literature; see, e.g., [67] for a comparison., and search for concept directions VV in ZZ. An interesting line of works has supported the empirical existence of VV: [6, 46] find directions that indicate truthful output, [68] finds directions for sentiments, [88] finds directions for emotions and honesty, and [53] finds directions for current player tile in a synthetic board game model. Interestingly, [71, 57, 30] further offer theoretical models, under which the linear controllability shows up provably in the latent space of LLMs. 2.2 Controlling Language Models via Latent Space Engineering The above findings have supported the development of practical methods to control the behavior of language models. As we will see, a key challenge there is to decide the correct control strength. Vector Addition. The work of [67, 72, 88, 43, 53, 68, 37] proposes to add or subtract multiples of a concept direction from the latent code. For example, to remove hatred from zitalic_z, one performs ↦−c^⋅hatred,maps-to⋅^subscripthatred z z- c· v_% hatred,italic_z ↦ italic_z - over start_ARG c end_ARG ⋅ italic_vhatred , (VecAdd) where c^>0^0 c>0over start_ARG c end_ARG > 0 is a parameter of the control strength. In principle, as each input prompt may contain a different ‘extent’ of the concept to be removed, c^ cover start_ARG c end_ARG should depend on both the prompt and the concept. Thus, in practice, one either tunes c^ cover start_ARG c end_ARG per input prompt and concept, which is laborious, or one fixes a c^ cover start_ARG c end_ARG, which is sub-optimal. Indeed, this has been observed by the work [72]: In their Table 10, the optimal coefficients c^ cover start_ARG c end_ARG are markedly different across the examples; see also their ‘discussion’ section. Orthogonal Projection. The work of [5] proposed to remove gender bias in word embeddings by projecting the embeddings onto the orthogonal complement to a gender direction gendersubscriptgender v_genderitalic_vgender: ↦Πspan(gender)⟂⁢=−Πspan⁡(vgender)⁢. z _span( v_gender% ) z= z- _span(v_% gender) z.italic_z ↦ Πroman_span ( italic_v start_POSTSUBSCRIPT gender )⟂ end_POSTSUBSCRIPT italic_z = italic_z - Πroman_span ( v start_POSTSUBSCRIPT gender ) end_POSTSUBSCRIPT italic_z . (OrthoProj) Here, for any ∈ℝdsuperscriptℝ w∈R^ditalic_w ∈ blackboard_Rd, span⁡()spanspan( w)span ( italic_w ) is the linear subspace spanned by witalic_w, and for any linear subspace ⊂ℝdsuperscriptℝS⊂R^dS ⊂ blackboard_Rd, ΠsubscriptΠ _SΠcaligraphic_S denotes the ortho-projector onto SS. Such an idea is later applied to neural activations of LLMs [23, 88]. Applying orthogonal projection to remove concept directions from latent codes may be reasonable: if directions corresponding to different concepts are orthogonal, then orthogonal projection only removes the gender direction while leaving the others intact. That being said, there are often more concept directions presented, and they are not orthogonal. For example, [29] shows that causally related concepts only exhibit partial orthogonality for their directions. To sum up, numerous attempts have been made to control the behavior of language models. However, existing methods either have a control strength parameter that is hard to tune or may remove extra concept directions. As we will see in the next section, these issues can be resolved by the proposed PaCE framework, which explicitly models the geometry of a large concept dictionary. 3 Our Method: Parsimonious Concept Engineering Figure 3: Pipeline of PaCE has several major steps: Step 1 collects concept vectors and constructs the concept dictionary, Step 2 decomposes the activation vector of the given input by sparse coding to get concept coefficients, and Step 3 performs editing on the concepts towards reoriented response. 3.1 Activation Intervention via Overcomplete Oblique Projection Can we efficiently remove one or more target concept directions from a given latent activation without affecting other concept directions present? To address this problem, our key insight is to model as many concept directions as possible, and then decompose the activation to estimate its components along these directions. Figure 2 presents an idealized visual example. Here, one is given a latent activation meaning ‘red apple’, and the goal is to remove the ‘red’ direction from the activation (left). As illustrated, orthogonal projection and vector addition tend to fail (middle right and right), as we discussed in §2.2. In contrast, by decomposing the activation along the concept directions of ‘red’ and ‘apple’, one can safely remove the component along ‘red’ without affecting that along ‘apple’ (middle left). This is related to the idea of oblique projection, which gives the name of this section. That said, several challenges remain to be addressed. As motivated above, to accurately model semantic concepts, one needs to collect as many concept directions in the latent space as possible. Since existing works only provide a limited number of concept directions (as reviewed in Remark 2), we contribute by collecting a large dictionary of concept directions, which we will discuss in §3.2. Moreover, oblique projection is well-defined only when the concept directions are linearly independent, while concept directions are often dependent (as we show in §4.3) so the decomposition is not unique. §3.3 discusses our choice of decomposition algorithm to address this difficulty. 3.2 Knowledge-Driven Concept Dictionary Concept Dictionary Construction. We take the top 40,000 words from the Brown Corpus [18] ranked by word frequency [4] as the concept collection T. For each concept ti∈Tsubscriptt_i∈ Ttitalic_i ∈ T, we prompt GPT-4 to generate around 30303030 pieces of contextual stimuli si=si1,si2,⋯,si30,⋯subscriptsuperscriptsubscript1superscriptsubscript2⋯superscriptsubscript30⋯s_i=\s_i^1,s_i^2,·s,s_i^30,·s\sitalic_i = sitalic_i1 , sitalic_i2 , ⋯ , sitalic_i30 , ⋯ that are scenarios describing the concept. To enhance the diversity of the concept stimuli, we retrieve knowledge from Wikipedia [65, 44, 35] (as we detail in Appendix B.4) to augment the prompt of stimulus synthesis. Samples of concepts and their stimuli are shown in Figure 4 and Appendix Figure 12. For each concept tisubscriptt_ititalic_i, we extract a direction iℓsuperscriptsubscriptℓ v_i italic_vitalic_iroman_ℓ from the activations of its contextual stimuli at the ℓ ℓ-th decoder layer of the LLM [88], which gives a dictionary ℓ∈ℝd×nsuperscriptℓsuperscriptℝ D ∈R^d× nitalic_Droman_ℓ ∈ blackboard_Rd × n per layer (detailed in Appendix B.2). Task-Driven Dictionary Partition. Given an alignment task, we further instruct GPT-4 as a concept partitioner to classify whether a concept needs to be removed from the input representation. To take detoxification as an example, the concept ‘harmful’ is highly correlated to the toxic response (hence needs removal) while benign concepts such ‘bird’ and ‘laptop’ will remain. That is, the instructed GPT-4 partitions the concepts into undesirable and benign to the alignment tasks. The full prompting templates of concept synthesis and partitioning are shown in Appendix E. In the next sub-section, we describe the notations and usages of the annotated concept dictionary. Figure 4: Examples of the constructed concepts and their partition for the detoxification task sampled from our PaCE-1M. 3.3 Overcomplete Oblique Projection via Sparse Coding Now that we have a dictionary =[1,…,n]∈ℝd×nsubscript1…subscriptsuperscriptℝ D=[ v_1,…, v_n]∈% R^d× nitalic_D = [ italic_v1 , … , italic_vitalic_n ] ∈ blackboard_Rd × n of n concepts directions333For notational simplicity, we discuss sparse coding for a single Ditalic_D; Algorithm 2 deals with multiple layers., where each isubscript v_iitalic_vitalic_i is a concept direction of known semantic meaning. Given a latent activation insuperscriptin z^initalic_zin coming from the user input, how can we control it via oblique projection? Oblique Projection. The general paradigm of oblique projection can be stated as follows. • Step 1111-Decomposition: Find c1in,…,cnin∈ℝsuperscriptsubscript1in…superscriptsubscriptinℝc_1^in,…,c_n^in∈Rc1in , … , citalic_nin ∈ blackboard_R such that in=c1in⁢1+⋯+cnin⁢n+insuperscriptinsuperscriptsubscript1insubscript1⋯superscriptsubscriptinsubscriptsuperscriptin z^in=c_1^in v_1+…+c_n% ^in v_n+ r^initalic_zin = c1in italic_v1 + ⋯ + citalic_nin italic_vitalic_n + italic_rin by solving in∈argmin12⁢‖in−⁢‖22+Ω⁢(),superscriptinsubscriptargmin12superscriptsubscriptnormsuperscriptin22Ω c^in∈ argmin_ c 12% \| z^in- D c\|_2^2+% ( c),italic_cin ∈ argminbold_italic_c divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∥ italic_zin - italic_D italic_c ∥22 + Ω ( italic_c ) , (1) where Ω⁢()Ω ( c)Ω ( italic_c ) is a sparsity-promoting regularizer that we will discuss soon. Then, each coefficient ciinsuperscriptsubscriptinc_i^incitalic_iin for i∈1,…,n1…i∈\1,…,n\i ∈ 1 , … , n can be viewed as how much the concept represented by isubscript v_iitalic_vitalic_i is present in insuperscriptin z^initalic_zin, and insuperscriptin r^initalic_rin is the residual that is not explained by Ditalic_D. • Step 2-Intervention: Obtain the controlled coefficients c1ctrl,…,cnctrl∈ℝsuperscriptsubscript1ctrl…superscriptsubscriptctrlℝc_1^ctrl,…,c_n^ctrl∈Rc1ctrl , … , citalic_nctrl ∈ blackboard_R, where cictrlsuperscriptsubscriptctrlc_i^ctrlcitalic_ictrl is set to ciinsuperscriptsubscriptinc_i^incitalic_iin if the concept of isubscript v_iitalic_vitalic_i is benign to the control task and 00 if undesirable (which has been decided offline in §3.2). Then, synthesize a new latent code using the modified coefficients and the residual by taking ctrl=c1ctrl⁢1+⋯+cnctrl⁢n+insuperscriptctrlsuperscriptsubscript1ctrlsubscript1⋯superscriptsubscriptctrlsubscriptsuperscriptin z^ctrl=c_1^ctrl v_1+…+c% _n^ctrl v_n+ r^initalic_zctrl = c1ctrl italic_v1 + ⋯ + citalic_nctrl italic_vitalic_n + italic_rin. The synthesized ctrlsuperscriptctrl z^ctrlitalic_zctrl will replace insuperscriptin z^initalic_zin to be passed on to the next layer of the neural network444While in sparse coding one typically removes the residual insuperscriptin r^initalic_rin for denoising purpose, here insuperscriptin r^initalic_rin may contain useful information (e.g., grammar) not captured by the dictionary, so we simply keep it in the synthesized ctrlsuperscriptctrl z^ctrlitalic_zctrl.. Remark 3 ((OrthoProj, VecAdd) === Special Cases of Oblique Projection). If one restricts Ditalic_D to contain only the undesirable concept directions (i.e., the ones to be removed from the latent code), and further takes Ω⁢(⋅)Ω⋅ (·)Ω ( ⋅ ) to be a constant function, it can be shown that oblique projection reduces to the special case of orthogonal projection (OrthoProj). On the other hand, if Ditalic_D contains only one undesirable concept direction, and Ω⁢(⋅)Ω⋅ (·)Ω ( ⋅ ) is λ∥⋅∥22λ\|·\|_2^2λ ∥ ⋅ ∥22 for some regularization strength λ∈ℝλ∈Rλ ∈ blackboard_R, then oblique projection recovers vector addition (VecAdd), by setting λ equal to c^ cover start_ARG c end_ARG in (VecAdd). We provide proofs in Section B.1. As we will see next, our method differs from these two in having a larger dictionary and a sparsity-promoting regularizer. Overcomplete Oblique Projection. As mentioned in §3.1, when the concept directions are linearly independent, then there is a unique decomposition of the latent code along the concept directions. However, often the concept directions can be dependent or nearly so, leading to infinitely many decompositions or numerical issues. To address this issue, we leverage the idea of sparse coding: natural signals are typically generated from sparse linear combinations of dictionary atoms, and pursuing a sparse decomposition reveals certain aspects of the underlying signal despite the dictionary being overcomplete (i.e., the system is underdetermined)555For example, identifying which atoms or which blocks of atoms that the underlying signal is from [16].. This has been explored in a fruitful line of research in machine learning and computer vision (see textbooks [13, 73, 79] and references therein). Following this idea, we solve (1) with the regularizer Ω⁢()Ω ( c)Ω ( italic_c ) chosen to be the elastic net, i.e., Ω⁢()=α⁢(τ⁢‖1+(1−τ)⁢12⁢‖22),Ωsubscriptnorm1112superscriptsubscriptnorm22 ( c)=α τ\| c\|_1+(1-% τ) 12\| c\|_2^2 ,Ω ( italic_c ) = α ( τ ∥ italic_c ∥1 + ( 1 - τ ) divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∥ italic_c ∥22 ) , (2) where τ∈[0,1]01τ∈[0,1]τ ∈ [ 0 , 1 ] and α>00α>0α > 0 are parameters that control the sparsity of the solution. This problem is efficiently solved via an active-set algorithm that leverages the sparsity of the solution [82]. Pursuing sparse codes that emerges from the data is often known as parsimonious representation learning [41], which gives rise to the name PaCE of our overall framework. We summarize the online intervention process in Algorithms 1 and 2, and the overall PaCE procedure in Algorithm 3 in the Appendix. Input: Latent vector insuperscriptin z^initalic_zin, dictionary Ditalic_D, index set I of undesirable concepts in←Solve ⁢(⁢1⁢)⁢ s.t. ⁢(⁢2⁢)←superscriptinSolve italic-(1italic-) s.t. italic-(2italic-) c^in eq:decomposition% s.t. eq:elastic-netitalic_cin ← Solve italic_( italic_) s.t. italic_( italic_) ▷ ▷ Analysis in=in−⁢insuperscriptinsuperscriptinsuperscriptin r^in= z^in- D% c^initalic_rin = italic_zin - italic_D italic_cin ▷ ▷ Residual ctrl=Π⟨ei,∀i∈I⟩⟂⁢insuperscriptctrlsubscriptΠsuperscriptdelimited-⟨⟩subscriptfor-allperpendicular-tosuperscriptin c^ctrl= _ e_i,∀ i∈ I % c^initalic_cctrl = Π⟨ e start_POSTSUBSCRIPT i , ∀ i ∈ I ⟩⟂ end_POSTSUBSCRIPT italic_cin ▷ ▷ Control ctrl=in+⁢ctrlsuperscriptctrlsuperscriptinsuperscriptctrl z^ctrl= r^in+ D% c^ctrlitalic_zctrl = italic_rin + italic_D italic_cctrl ▷ ▷ Synthesis return Intervened latent vector ctrlsuperscriptctrl z^ctrlitalic_zctrl Algorithm 1 Overcomplete Oblique Projection (ObliqProj) Input: Pre-trained LLM with L decoder layers (DCL) to decompose, input tokens Eitalic_E, dictionaries ℓ=1Lsuperscriptsubscriptsuperscriptℓ1\ D \_ =1^L italic_Droman_ℓ ℓ = 1L, index set I of undesirable concepts 1=LayersBeforeDCL⁢()subscript1LayersBeforeDCL z_1=LayersBeforeDCL( E)italic_z1 = LayersBeforeDCL ( italic_E ) For ℓ←1,2,…,L←ℓ12… ← 1,2,…,Lℓ ← 1 , 2 , … , L: ℓ=ObliqProj⁢(ℓ,ℓ,I)superscriptℓObliqProjsuperscriptℓsuperscriptℓ z =ObliqProj( z , % D ,I)italic_zroman_ℓ = ObliqProj ( italic_zroman_ℓ , italic_Droman_ℓ , I ) ▷ ▷ Algorithm 1 ℓ+1=DCLℓ⁢(ℓ)superscriptℓ1superscriptDCLℓsuperscriptℓ z +1=DCL ( z )italic_zroman_ℓ + 1 = DCLℓ ( italic_zroman_ℓ ) =LayersAfterDCL⁢(L+1)LayersAfterDCLsuperscript1 e=LayersAfterDCL( z^L+1)italic_e = LayersAfterDCL ( italic_zitalic_L + 1 ) return Output token eitalic_e Algorithm 2 PaCE Activation Intervention 4 Experimental Results We evaluate the effectiveness of PaCE on downstream tasks including Detoxification, Faithfulness Enhancement, and Sentiment Refinement. We then analyze the sampled activation space, enabled by our large collection of concept vectors. We provide implementation details in Appendix B.4. 4.1 Improving Safety by Response Detoxification Figure 5: An example of jailbreaking LLaMA2-7B-Chat and detoxification by PaCE. PaCE successfully detoxifies the response while maintaining the instruction-following capability. Here we perform activation manipulation using our framework PaCE for detoxifying LLM responses. An example of our detoxification is shown in Figure 5: LLaMA2-7B-Chat is prompted with the malicious intent (i.e., jailbreaking) and parts of the response of the vanilla LLM (vanilla response) are generally considered manipulative and ill-intent. Our PaCE response pivots from a harmful to a harmless style and makes harmless suggestions. Appendix D.1 shows additional concrete examples. Setup. For baselines, Prompting directly instructs LLM not to output sentences relevant to the list of top undesirable concepts (template in Appendix B), VecAdd subtracts the concept vector ‘harmful’ from the activation of the input, and OrthoProj performs projection on the orthogonal complement of the concept vector ‘harmful’. Note that, if we directly apply OrthoProj and VecAdd over the large collection of top undesirable concepts (e.g., 50 concepts) with no decomposition analysis, the input representation will significantly diverge from the original ones since every activation vector is of a similar scale, and the LLM’s linguistic capabilities will degrade. We compare our method in defending maliciousness against activation manipulation methods (§2.2) on the SafeEdit [74] dataset with its safety scorer. For every response, the benchmark’s safety scorer rates between 00 and 1111 (higher is safer). We use the effective set where the original safety score is lower than 50%percent5050\%50 % (i.e., the successful attacks if binarily classified). Safety Responses. The evaluation has nine categories: Political Sensitivity (PS), Pornography (PG), Ethics and Morality (EM), Illegal Activities (IA), Mental Harm (MH), Offensiveness (OF), Physical Harm (PH), Privacy and Property (P), and Unfairness & Bias (UB). Table 1 shows that, for LLaMa2-7B, PaCE improves by 60-80% over the vanilla method in categories including IA, MH, OF, PH, P, and UB. When compared to other methods, PaCE performs competitively and improves by 6-20%. While our method did not perform the best in PS, PG, and EM, the gap for those categories is relatively small considering the significant overall gains. Notably, for LLaMA2-13B which has more parameters and a presumably more structured latent space, PaCE dominates other methods in all categories, demonstrating the necessity for respecting the latent structures when modifying representations. Finally, Table 3 shows the contribution of design choices in PaCE, and Figure 6 shows the effect of the dictionary size on the performance. We observe clear improvement after each design choice is progressively added. Appendix B.5 includes the details of these ablation studies. Linguistic Capability. To validate that the detoxified representations of PaCE are still effective on general linguistic capability, we also evaluate the responses by N-gram fluency and perplexity. Furthermore, we apply PaCE to detoxify MMLU questions (which are naturally unharmful) to show that the detoxification will not significantly degrade the LLM’s reasoning capability. We observe that the MMLU response accuracy of PaCE is the highest among all activation manipulation baselines. Figure 6: The detoxification performances for LLaMA2-13B w.r.t. the dictionary size. Efficiency. Table 3 shows that PaCE is more time-efficient compared to the OrthoProj which also projects the concept vector onto the input vector. PaCE sees a three times speed improvement in average time per response and a two times improvement over average time per word when compared to OrthoProj. While PaCE is computationally slower than VecAdd, we argue the performance gain in a majority of the categories is a benefit that outweighs this particular shortcoming. Solvers. Figure 7 additionally evaluates Orthogonal Matching Pursuit (OMP) [58, 7], a fast greedy solver for the activation decomposition. OMP iteratively adds to the support the concept that has maximum coherence with the unexplained residual and updates the residual by solving the least square using the new support. It stops when a pre-defined maximum size k of support is reached. Intuitively, the k is the number of non-zero elements in the solved coefficients. We observe from the table that one can choose improvements in computational speed at the cost of safety performance. Table 1: Detoxification evaluation for PaCE, representation manipulation, and training-free baselines. The best performance of each category is in bold and the second best is underlined. Target Model Method Safety (%, ↑ ↑) Linguistic Capability PS PG EM IA MH OF PH P UB Fluency (↑ ↑) Perplexity (↓ ↓) MMLU (%, ↑ ↑) LLaMA-7B-Chat Vanilla [70] 17.6 19.5 10.1 7.79 11.3 17.2 22.6 11.8 17.2 7.70 3.51 43.4 Prompting [70] 82.5 47.3 57.8 65.2 75.1 54.8 72.0 72.4 56.1 7.50 3.04 15.4 VecAdd [67, 72, 88] 50.9 58.9 59.0 53.9 66.1 55.0 60.7 61.7 66.4 6.58 7.58 29.0 OrthoProj [23, 88] 50.7 57.9 50.2 47.5 67.0 50.1 74.9 65.7 66.4 7.46 3.73 34.1 PaCE (Ours) 69.6 46.2 58.2 75.3 94.2 62.3 80.8 72.8 88.3 8.07 3.52 37.1 LLaMA2-13B-Chat Vanilla [70] 8.01 23.7 13.6 19.8 18.3 21.6 13.6 14.0 16.7 7.66 2.48 54.9 Prompting [70] 35.8 68.3 59.3 52.5 73.5 23.4 78.0 71.1 66.5 7.63 2.22 52.1 VecAdd [67, 72, 88] 76.6 71.4 70.0 64.3 87.2 66.9 47.4 74.5 71.1 7.46 2.75 51.6 OrthoProj [23, 88] 51.1 82.6 50.6 72.4 52.3 58.0 51.4 65.1 75.5 7.29 2.88 52.9 PaCE (Ours) 93.7 97.9 97.7 94.9 98.9 96.6 99.3 90.8 98.9 7.52 2.85 54.1 Table 2: Computation time (in seconds) evaluation for PaCE and representation manipulation baselines. We observe that, compared to OrthoProj which also projects the concept, our PaCE is more time-efficient for trustworthiness control. Method LLaMA2-7B-Chat LLaMA2-13B-Chat Time per Response Time per Token Time per Response Time per Token Vanilla 12.4 0.041 20.7 0.076 VecAdd 16.3 0.062 29.1 0.109 OrthoProj 143.7 0.514 221.6 0.780 PaCE (Ours) 44.8 0.119 50.3 0.149 Table 3: Ablation study for PaCE on the detoxifying LLaMA2-7B. Starting from a small emotion dictionary and manually selected concepts for removal, each subsequent design leads to better performance. Method Safety (%, ↑ ↑) Fluency (↑ ↑) PaCE (LLaMA2-7B-Chat) 50.2 7.26 + Decomposition on 104superscript10410^4104 Concepts 57.6 7.58 + Clustering of Concepts 62.3 7.63 + Concept Partitioner 65.1 7.70 + Removal of Top 50505050 Concepts 76.5 8.07 4.2 Improving Faithfulness and Removing Negative Sentiment We evaluate the framework based on the response’s faithfulness and sentiment when input prompts requests for information involving biographical facts or minority social groups. Faithfulness reflects the level of factuality in the generation, and sentiment describes the emotional tone behind the generation. In short, we find PaCE effective in improving the faithfulness and removing negative sentiment in LLMs’ outputs. We describe the setup, metrics and method below. Table 4: Faithfulness and Fairness evaluation for PaCE, representation manipulation, and training-free baselines. The best performance of each category is in bold and the second best is underlined. Target Model Method Fact (↑ ↑) Sentiment (%, ↑ ↑) Linguistic Capability LS (%) LAF US (%) UAF GN OC NT Fluency (↑ ↑) Perplexity (↓ ↓) MMLU (%, ↑ ↑) LLaMA2-7B-Chat Vanilla [70] 18.4 45.1 15.4 37.4 51.5 69.2 56.4 7.20 2.49 43.4 Prompting [70] 28.6 40.6 20.4 49.0 53.1 62.3 56.6 7.25 2.87 16.3 VecAdd [67, 72, 88] 16.2 46.1 10.3 52.2 55.2 68.5 58.3 7.09 3.91 30.6 OrthoProj [23, 88] 21.9 49.7 26.2 45.9 54.9 75.1 60.1 7.21 2.76 34.1 PaCE (Ours) 27.7 65.9 30.8 73.3 66.2 79.7 69.9 7.91 2.88 38.4 LLaMA2-13B-Chat Vanilla [70] 44.1 39.6 41.8 38.5 50.2 70.3 58.1 7.63 2.41 54.9 Prompting [70] 61.6 24.5 47.5 20.0 46.1 73.8 59.4 7.46 2.45 52.4 VecAdd [67, 72, 88] 24.5 49.2 14.9 68.9 56.2 72.9 58.7 6.92 2.78 50.9 OrthoProj [23, 88] 59.3 52.8 43.2 51.7 57.7 75.1 63.3 7.26 2.66 51.1 PaCE (Ours) 64.8 53.0 76.4 55.1 63.4 76.5 67.5 7.48 2.43 53.1 Figure 7: Ablation study for solvers. We observe that greedy solvers can improve computational speed at the cost of safety performance. Method Time per decomposition (s, ↓ ↓) Safety (%, ↑ ↑) OMP (k=5050k=50k = 50) 0.045 63.1 OMP (k=100100k=100k = 100) 0.182 64.4 OMP (k=150150k=150k = 150) 0.381 66.9 OMP (k=200200k=200k = 200) 0.749 70.8 Elastic Net 0.411 72.0 Setup. Faithfulness: We use the FactScore suite and the fact evaluator for faithful biography generation [51]. The suite is divided into labeled and unlabeled subsets used in different sections of the original paper. Our table reports the Labeled Score (LS), the total number of Labeled Atomic Facts (LAF), the Unlabeled Score (US), and the total number of unlabeled Atomic Facts (LAF). Sentiment: We use the HolisticBias suite [66] and hate speech evaluator [64] to measure the sentiment of the response to underrepresented descriptors. The reported numbers are the average of non-negative sentiment scores for underrepresented groups categorized by Gender (GN), Occupation (OC), and Nationality (NT). During the sentiment revising, the concept setups for all approaches follow the detoxification setup. For the faithfulness experiments, PaCE removes the top 50 undesirable (hallucinatory) concepts ranked by the partitioner. The Prompting approach instructs the LLM not to output sentences relevant to these top concepts. The VecAdd and OrthoProj operate on the concept vector of ‘fabrication’. Results. Our results are shown in Table 4. For both 7B and 13B models, PaCE achieves more factual responses and improves the sentiment according to most metrics. For linguistic performance, our method ranks right after the Vanilla method for the larger 13B model, and achieves comparable results for LLaMA2-7B. Overall, we argue PaCE is an effective method for improving faithfulness and sentiment revising. 4.3 Representation Space Sampled by PaCE-1M Our collected dataset of conceptual representations enables us to investigate the geometry and potential applications of the representation (activation) space. Concept Clustering and Retrieval. Here we explore the semantic structure of the activation space of the LLaMA2-13B-Chat by visualizing the first 10,000 concepts from the PaCE-1M dataset. We apply a dimensionality reduction method UMAP [47] on the concept vectors and visualize the first two dimensions in Figure 8. Concept vectors with similar semantics appear to be close to each other: e.g., in Figure 8 (1), concepts such as ‘college’, ‘university’, ‘Academy’, and ‘Institute’ are related to Education and they are close in the UMAP space. Notably, concepts of different semantics are clearly separated: those related to Education, Countries/States, Cities, Food and Clothing, and Positive Emotions respectively form distinct clusters. In particular, while concepts relevant to geography are closely clustered in Figure 8 (2), we observe a clear boundary between concepts related to Countries/States and those to Cities. These semantic structures indicate that the activation space sampled by our PaCE-1M dataset can capture and organize semantic information of the concepts, enabling further analysis and manipulations in PaCE. Figure 10 further reports the concept retrieval by evaluating the distance between a target concept with other concept vectors in the activation space. We observe organizational structure from the concept clusters based on their semantics. For instance, vectors for the concept ‘affection’ and ‘friendship’, are geometrically close and semantically relevant to the concept ‘love.’ Zooming out, such semantic structures are observed throughout the activation spaces of LLaMA2, and we conjecture they generalize to those in other LLMs. We provide more details of clustering and retrieval in Appendix C.2 and Appendix C.3. Figure 8: The Representation (Activation) Space of LLaMA2-13B-Chat with the first 10000 Concepts from PaCE-1M. Appendix Figure 16 shows the zoom-in version. The visualization is the first two dimensions of UMAP of the concept vectors. We observe that concepts of similar semantics are clustered together, indicating that the activation space has semantic structures. Figure 9: Number of tokens per response across different intervention methods and LLM models. Figure 10: The top 10 retrieved concepts using the similarity score in the sampled activation space. We observe close coherence between the target concept and retrieved concepts. 5 Discussion We provide discussions on the monosemanticity of concepts and connections among different alignment paradigms in this section. We also argue how PaCE handles context-dependent concepts. 5.1 Polysemy of Words While VecAdd and OrthoProj may be affected by the polysemy of words, PaCE’s overcomplete dictionary allows accurate analysis of the target representation through sparse decomposition. Table 1 and Table 4 show that PaCE outperforms OrthoProj and VecAdd on linguistic metrics. We attribute the high helpfulness of PaCE to the large-scale dictionary with sparse coding, explained as follows. Comprehensive Coverage. Since the dictionary is large, concepts with single and clear semantics are involved. E.g., if the stimuli of ‘kill’ may have different meanings, there exist other more polarized concept vectors such as ‘murder’ (more harmful) and ‘spend’ (more benign). Parsimony of Solution. Sparse coding aims to choose the fewest concepts to reconstruct the latent representation (i.e., parsimony). For the sake of argument, assuming the sentence is about ‘killing time’ and the vanilla LLM has the correct semantic understanding of its benignness, the latent representation of the whole sentence will be closer to concepts such as ‘spend’ and ‘time’ rather than string-matching to ‘kill’ (which in your setup could have mixed harmful and benign senses). As the sparse coding of the target representation promotes the parsimonious selection of concepts with monosemantics, it helps to represent benign contexts correctly without assigning significant weights to ambiguous terms like ‘kill’. 5.2 Different Alignment Paradigms As mentioned in §1, beyond activation engineering, there are other alignment paradigms such as Supervised Fine-Tuning (SFT) [26], Reinforcement Learning from Human Feedback (RLHF) [55], and Knowledge Engineering (KE) [14, 74]. We clarify the main advantages of PaCE over them. Training-Free. RLHF, SFT, and KE all need to tune the parameters of LLM, which potentially degrade the well-structured priors of the pre-trained LLM. Taking a step back, even if LoRA is adopted for these paradigms, the training/tuning incurs significant computation and memory costs. PaCE does not modify the parameters of LLM and requires no training. It better preserves the priors of LLM, provides a low-resource alignment solution, and retains the general linguistic capabilities. Interpretable and Adaptive. The solved coefficients of PaCE are an accurate interpretation of how a user input’s representation is composed in the concept space. Also, when a new alignment goal is set, RLHF, SFT, and KE need to collect sufficient task samples and tune the LLM on the new dataset. In contrast, PaCE just needs to run the concept partitioner through CORD, which is expected to be much faster and more convenient. 5.3 Context-dependent Concepts The structured activation space of LLMs and the large-scale concept dictionary of PaCE help to handle the influence of the context for a concept in the target prompt. As the LLM scales up, its capability to capture and utilize contextual information grows with the help of attention modules. The activation space, as already used for many representation manipulation methods, is expected to convey the underlying semantic information of concepts in the sentence (context). That is, the space hosting concept vectors is not collapsed, and it is structured to distinguish different concepts. The representation (activation) to be steered at inference time encodes the context and conveys the semantics of a concept based on the context. Then, since our overcomplete concept dictionary in PaCE widely covers concepts of various categories, the sparse coding on this dictionary will effectively analyze the target representation as the linear combination of these concepts. 6 Conclusion In this paper, we present PaCE, an activation engineering framework designed for aligning LLMs by effectively and efficiently addressing undesirable representations while retaining linguistic capabilities. By constructing a large-scale concept dictionary and leveraging sparse coding for activation decomposition, PaCE opens up new research avenues for training-free LLM alignment. Our experiments on tasks such as response detoxification, faithfulness enhancement, and sentiment revising demonstrate that PaCE achieves state-of-the-art performance compared to existing representation manipulation approaches. PaCE not only ensures alignment with less cost but also adapts to evolving alignment goals without significantly compromising the LLM’s linguistic proficiency. We open-source the PaCE-1M dataset to facilitate future research and practical applications of LLM alignment, and will release the source code soon. We further elaborate on the potential limitations, societal impacts, and future works of PaCE in Appendix B.7. Acknowledgments and Disclosure of Funding This research was supported by ARO MURI W911NF-17-1-0304, DARPA GARD HR001119S0026, DARPA RED HR00112090132, ODNI IARPA HIATUS #2022-22072200005, the NSF grant 2031985, Simons Foundation MoDL 135615, a gift from AWS AI to Penn Engineering’s ASSET Center for Trustworthy AI, and NSF Graduate Research Fellowship #DGE2139757. We would like to thank Liangzu Peng, Hancheng Min, Bowen Li, Xinyu Yang, and Fengrui Tian for their suggestions in the presentation and experiments. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, NSF, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. References [1] Carl Allen and Timothy Hospedales. Analogies Explained: Towards Understanding Word Embeddings. arXiv preprint arXiv:1901.09813, 2019. [2] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear Algebraic Structure of Word Senses, with Applications to Polysemy. In TACL, 2018. [3] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A Latent Variable Model Approach to PMI-based Word Embeddings. arXiv preprint arXiv:1502.03520, 2019. [4] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.", 2009. [5] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In NeurIPS, 2016. [6] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision. arXiv preprint arXiv:2212.03827, 2024. [7] T. Tony Cai and Lie Wang. Orthogonal matching pursuit for sparse signal recovery with noise. In IEEE Transactions on Information Theory, 2011. [8] Xavier Suau Cuadros, Luca Zappella, and Nicholas Apostoloff. Self-conditioning pre-trained language models. In ICML, 2022. [9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023. [10] Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint arXiv:2311.04205, 2023. [11] Tianjiao Ding, Shengbang Tong, Kwan Ho Ryan Chan, Xili Dai, Yi Ma, and Benjamin D Haeffele. Unsupervised manifold linearizing and clustering. In ICCV, 2023. [12] Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, and Luming Liang. The efficiency spectrum of large language models: An algorithmic survey. arXiv preprint arXiv:2312.00678, 2024. [13] Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer, 2010. [14] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023. [15] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposition. arXiv preprint arXiv:2209.10652, 2022. [16] Ehsan Elhamifar and René Vidal. Block-sparse recovery via convex optimization. In IEEE Transactions on Signal Processing, 2012. [17] Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards Understanding Linear Word Analogies. arXiv preprint arXiv:1810.04882, 2019. [18] W. Nelson Francis and Henry Kucera. Computational analysis of present-day american english. Brown University Press, 1967. [19] Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770, 2023. [20] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Proceedings of the Third Conference on Causal Learning and Reasoning, 2024. [21] Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney. Skip-Gram - Zipf + Uniform = Vector Additivity. In ACL, 2017. [22] Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query-based adversarial prompt generation. arXiv preprint arXiv:2402.12329, 2024. [23] John Hewitt, John Thickstun, Christopher D. Manning, and Percy Liang. Backpack language models. In ACL, 2023. [24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. [25] Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Jake Zhao, and Hang Zhao. Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901, 2023. [26] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. [27] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. [28] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2024. [29] Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering Meanings of Embeddings via Partial Orthogonality. arXiv preprint arXiv:2310.17611, 2023. [30] Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the Origins of Linear Representations in Large Language Models. arXiv preprint arXiv:2403.03867, 2024. [31] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. In IEEE Transactions on Big Data, 2019. [32] Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor attacks for in-context learning with language models. arXiv preprint arXiv:2307.14692, 2023. [33] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In ICCV, 2023. [34] Omer Levy and Yoav Goldberg. Linguistic Regularities in Sparse and Explicit Word Representations. In CNLL, 2014. [35] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS, 2020. [36] José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding-a plug and play geometric loss for deep learning. In CVPR, 2018. [37] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In ICLR, 2023. [38] Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024. [39] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In ICLR, 2023. [40] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023. [41] Renjie Liao, Alex Schwing, Richard Zemel, and Raquel Urtasun. Learning deep parsimonious representations. NeurIPS, 29, 2016. [42] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. [43] Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668, 2023. [44] Jinqi Luo, Kwan Ho Ryan Chan, Dimitris Dimos, and René Vidal. Knowledge pursuit prompting for zero-shot multimodal synthesis. arXiv preprint arXiv:2311.17898, 2023. [45] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In CVPR, 2023. [46] Samuel Marks and Max Tegmark. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. arXiv preprint arXiv:2310.06824, 2023. [47] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2020. [48] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781, 2013. [49] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. arXiv preprint arXiv:1310.4546, 2013. [50] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In NAACL HLT, 2013. [51] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In EMNLP, 2023. [52] Masahiro Naito, Sho Yokoi, Geewook Kim, and Hidetoshi Shimodaira. Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings. arXiv preprint arXiv:2105.08585, 2022. [53] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent Linear Representations in World Models of Self-Supervised Sequence Models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, ACL BlackboxNLP Workshop, 2023. [54] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [55] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. [56] Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. In EMNLP, 2023. [57] Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models. arXiv preprint arXiv:2311.03658, 2023. [58] Y. C. Pati, Ramin Rezaiifar, and P. S. Krishnaprasad. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. 1993. [59] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global Vectors for Word Representation. In EMNLP, 2014. [60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2021. [61] Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024. [62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [63] Noam Shazeer, Ryan Doherty, Colin Evans, and Chris Waterson. Swivel: Improving Embeddings by Noticing What’s Missing. arXiv preprint arXiv:1602.02215, 2016. [64] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In EMNLP-IJCNLP, 2019. [65] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv: 2301.12652, 2023. [66] Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. “I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset. In EMNLP, 2022. [67] Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting Latent Steering Vectors from Pretrained Language Models. In ACL Findings, 2022. [68] Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear Representations of Sentiment in Large Language Models. arXiv preprint arXiv:2310.15154, 2023. [69] Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In ICLR, 2024. [70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023. [71] Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Stefano Soatto. Linear spaces of meanings: compositional structures in vision-language models. In ICCV, 2023. [72] Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248v3, 2023. [73] René Vidal, Yi Ma, and Shankar Sastry. Generalized Principal Component Analysis. Interdisciplinary Applied Mathematics. Springer New York, 2016. [74] Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. arXiv preprint arXiv:2403.14472, 2024. [75] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023. [76] Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. In NeurIPS, 2023. [77] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. In TMLR, 2022. [78] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. [79] John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022. [80] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In ICLR, 2024. [81] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023. [82] Chong You, Chun Guang Li, Daniel P Robinson, and Rene Vidal. Oracle Based Active Set Algorithm for Scalable Elastic Net Subspace Clustering. In CVPR, 2016. [83] Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. NeurIPS, 2024. [84] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint arXiv:2103.15949, 2023. [85] Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Kevin Alexander, Euan Ashley, Jack Boyd, Kathleen Boyd, Karen Hirsch, Curt Langlotz, Joanna Nelson, and William Hiesinger. Almanac: Retrieval-augmented language models for clinical medicine. arXiv preprint arXiv:2303.01229, 2023. [86] Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. Controllable text-to-image generation with gpt-4. arXiv preprint arXiv:2305.18583, 2023. [87] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. [88] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. [89] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Supplementary Material Appendix A Structure of The Appendix The appendix is structured as follows: Appendix B describes details of our PaCE framework, including proofs of propositions and a comprehensive explanation of the framework’s algorithm. Appendix C elaborates on the PaCE-1M dataset, demonstrating the structure of the dataset with explorations of subspace clustering to analyze the dataset. Appendix D presents textual results, including visualizations of baseline comparisons and samples of concept clusters. Appendix E shows the instruction templates used for GPT-4 to synthesize and partition concepts. Appendix B Details of PaCE Framework This section validates the propositions of the PaCE framework discussed in §3.3, followed by descriptions of how to extract representations and the algorithm of the whole procedures of PaCE. B.1 Proofs of Oblique Projection Recovers Vector Addition and Orthogonal Projection Proposition 1. Let ∈ℝd×nsuperscriptℝ D∈R^d× nitalic_D ∈ blackboard_Rd × n be a dictionary matrix and ∈ℝdsuperscriptℝ z∈R^ditalic_z ∈ blackboard_Rd a latent code. Then, any solution ∗superscript c^*italic_c∗ of the optimization problem min∥−∥22 _ c z- D % c _2^2minbold_italic_c ∥ italic_z - italic_D italic_c ∥22 (3) satisfies ⁢∗=Πrange⁡()⁢superscriptsubscriptΠrange D c^*= _range( D% ) zitalic_D italic_c∗ = Πroman_range ( italic_D ) italic_z. Therefore, the map ↦−⁢∗⁢()maps-tosuperscript z z- D c^*(% z)italic_z ↦ italic_z - italic_D italic_c∗ ( italic_z ) is the same as ↦−Πrange⁡()⁢=↦Πrange()⟂ z z- _range( % D) z= z _range(% D) zitalic_z ↦ italic_z - Πroman_range ( italic_D ) italic_z = italic_z ↦ Πroman_range ( italic_D )⟂ italic_z in (OrthoProj). Proof. Note that =Πrange⁡()+Πrange()⟂ I= _range( D)+ _% range( D) italic_I = Πroman_range ( italic_D ) + Πroman_range ( italic_D )⟂. Therefore, the objective of (3) can be written as ∥−⁢∥22superscriptsubscriptdelimited-∥22 z- D c% _2^2∥ italic_z - italic_D italic_c ∥22 =∥Πrange()⟂⁢+Πrange⁡()⁢−⁢∥22 = _range( D) % z+ _range( D) z-% D c _2^2= ∥ Πroman_range ( italic_D )⟂ italic_z + Πroman_range ( italic_D ) italic_z - italic_D italic_c ∥22 =∥Πrange()⟂⁢∥22+∥Πrange⁡()⁢−⁢∥22+2⁢⟨Πrange()⟂⁢,Πrange⁡()⁢−⁢⟩, = _range( D) % z _2^2+ _range(% D) z- D c % _2^2+2 _range( D) % z, _range( D) z-% D c ,= ∥ Πroman_range ( italic_D )⟂ italic_z ∥22 + ∥ Πroman_range ( italic_D ) italic_z - italic_D italic_c ∥22 + 2 ⟨ Πroman_range ( italic_D )⟂ italic_z , Πroman_range ( italic_D ) italic_z - italic_D italic_c ⟩ , where ⟨⋅,⋅⟩⋅ ·,· ⟨ ⋅ , ⋅ ⟩ is the Euclidean inner product of ℝdsuperscriptℝR^dblackboard_Rd. The first term is constant with respect to citalic_c, so it can be omitted. Further, since any ortho-projector (in particular Πrange()⟂ _range( D) Πroman_range ( italic_D )⟂) is self-adjoint, we have ⟨Πrange()⟂⁢,Πrange⁡()⁢−⁢⟩=⟨,Πrange()⟂⁢(Πrange⁡()⁢−⁢)⟩=0. _range( D) z,% _range( D) z- D% c = z, _range(% D) _range( D% ) z- D c =0.⟨ Πroman_range ( italic_D )⟂ italic_z , Πroman_range ( italic_D ) italic_z - italic_D italic_c ⟩ = ⟨ italic_z , Πroman_range ( italic_D )⟂ ( Πroman_range ( italic_D ) italic_z - italic_D italic_c ) ⟩ = 0 . Therefore, problem (3) is equivalent to optimizing ∥Πrange⁡()⁢−⁢∥22,superscriptsubscriptdelimited-∥subscriptΠrange22 _range( D) z-% D c _2^2,∥ Πroman_range ( italic_D ) italic_z - italic_D italic_c ∥22 , which is lower bounded by 00. This lower bound is realizable since Πrange⁡()⁢∈range⁡()subscriptΠrangerange _range( D) z∈% range( D)Πroman_range ( italic_D ) italic_z ∈ range ( italic_D ). Thus, any minimizer ∗superscript c^*italic_c∗ must realize this lower bound, meaning Πrange⁡()⁢=⁢subscriptΠrange _range( D) z= D% cΠroman_range ( italic_D ) italic_z = italic_D italic_c. So we are done. ∎ Proposition 2. Let Ditalic_D contain only one concept direction ∈ℝdsuperscriptℝ v∈R^ditalic_v ∈ blackboard_Rd. Let ∈ℝdsuperscriptℝ z∈R^ditalic_z ∈ blackboard_Rd be a latent code, and λ>−11λ>-1λ > - 1 a regularization strength. Then, the solution c∗∈ℝsuperscriptℝc^*∈Rc∗ ∈ blackboard_R of the optimization problem min∥−∥22+λ∥22⇔minc∥−c∥22+λc2 _ c z- D% c _2^2+λ c% _2^2\ \ \ \ _c z% -c v _2^2+λ c^2minbold_italic_c ∥ italic_z - italic_D italic_c ∥22 + λ ∥ italic_c ∥22 ⇔ minitalic_c ∥ italic_z - c italic_v ∥22 + λ c2 (4) is given by c∗=⟨,⟩λ+1superscript1c^*= z, v λ+1c∗ = divide start_ARG ⟨ italic_z , italic_v ⟩ end_ARG start_ARG λ + 1 end_ARG. Therefore, the map ↦−⁢∗⁢()maps-tosuperscript z z- D c^*(% z)italic_z ↦ italic_z - italic_D italic_c∗ ( italic_z ) recovers (VecAdd): the former is the same as ↦−ηλ⁢+maps-tosubscriptsubscript z z- _λ v_+italic_z ↦ italic_z - ηitalic_λ italic_v+, where one can set any ηλ>0subscript0 _λ>0ηitalic_λ > 0 by properly choosing λ>−11λ>-1λ > - 1, and +subscript v_+italic_v+ is defined as vitalic_v if ⟨,⟩>00 v, z >0⟨ italic_v , italic_z ⟩ > 0 and −- v- italic_v otherwise. Proof. Note that the objective of (4) is simply a univariate quadratic function of c: ‖22−2⁢⟨,⟩⁢c+(λ+1)⁢c2.superscriptsubscriptnorm2221superscript2\| z\|_2^2-2 z, v c% +(λ+1)c^2.∥ italic_z ∥22 - 2 ⟨ italic_z , italic_v ⟩ c + ( λ + 1 ) c2 . This has a unique minimizer c∗=⟨,⟩λ+1superscript1c^*= z, v λ+1c∗ = divide start_ARG ⟨ italic_z , italic_v ⟩ end_ARG start_ARG λ + 1 end_ARG since λ+1>010λ+1>0λ + 1 > 0 by assumption. To prove the second part of the proposition, note that −⁢∗⁢()=−c∗⁢()⁢=−⟨,⟩λ+1⁢=−|⟨,⟩|λ+1⋅(⁢sign⁡(⟨,⟩)).superscriptsuperscript1⋅1sign z- D c^*(% z)= z-c^*( z) v=% z- z, v λ% +1 v= z- z,% v λ+1· v% sign z, v% .italic_z - italic_D italic_c∗ ( italic_z ) = italic_z - c∗ ( italic_z ) italic_v = italic_z - divide start_ARG ⟨ italic_z , italic_v ⟩ end_ARG start_ARG λ + 1 end_ARG italic_v = italic_z - divide start_ARG | ⟨ italic_z , italic_v ⟩ | end_ARG start_ARG λ + 1 end_ARG ⋅ ( italic_v sign ( ⟨ italic_z , italic_v ⟩ ) ) . (5) Define ηλ:=|⟨,⟩|λ+1assignsubscript1 _λ:= z, v% λ+1ηitalic_λ := divide start_ARG | ⟨ italic_z , italic_v ⟩ | end_ARG start_ARG λ + 1 end_ARG and +:=⁢sign⁡(⟨,⟩)assignsubscriptsign v_+:= vsign( % z, v )italic_v+ := italic_v sign ( ⟨ italic_z , italic_v ⟩ ). One can see that by varying λ∈(−1,+∞)1λ∈(-1,+∞)λ ∈ ( - 1 , + ∞ ), ηλsubscript _ληitalic_λ can take any value in (0,∞)0(0,∞)( 0 , ∞ ). This concludes the proof. ∎ B.2 Extracting Concept Directions and Constructing Dictionary Recall from §3.2 that for each concept tisubscriptt_ititalic_i, we have collected a set of context stimuli (i.e., sentences that describe tisubscriptt_ititalic_i) si=si0,si1,⋯,siNssubscriptsuperscriptsubscript0superscriptsubscript1⋯superscriptsubscriptsubscripts_i=\s_i^0,s_i^1,·s,s_i^N_s\sitalic_i = sitalic_i0 , sitalic_i1 , ⋯ , sitalic_iitalic_Nitalic_s . This totals 40,0004000040,00040 , 000 concepts and more than 1,200,00012000001,200,0001 , 200 , 000 context stimuli. To obtain a vector for each concept, we follow the representation reading algorithm [88] to map the concept to the hidden states of LLM decoder layers. We describe the algorithm here for completeness. Each context sentence sijsuperscriptsubscripts_i^jsitalic_iitalic_j together with the concept tisubscriptt_ititalic_i is first plugged into a pre-defined prompt template, producing s¯ijsuperscriptsubscript¯ s_i^jover¯ start_ARG s end_ARGiitalic_j. Consider the <concept tisubscriptt_ititalic_i> in the following scenario: Scenario: <stimulus sjjsuperscriptsubscripts_j^jsitalic_jitalic_j> Answer: For any prompt p, denote by fℓ⁢(p)superscriptℓf (p)froman_ℓ ( p ) the activation of the last token at the l-th layer of the LLM when the input is p. Then, to extract a vector for concept tisubscriptt_ititalic_i, one looks at the activations of pairs of stimuli Xiℓ:=Πd−1⁢(fℓ⁢(s¯ij)−fℓ⁢(s¯i′j′)):∀i′≠i,∀j,j′,assignsuperscriptsubscriptℓconditional-setsubscriptΠsuperscript1superscriptℓsuperscriptsubscript¯superscriptℓsuperscriptsubscript¯superscript′for-allsuperscript′for-allsuperscript′X_i := \ _S^d-1 f ( s_i% ^j)-f ( s_i ^j ) :∀ i^% ≠ i, ∀ j,j \,Xitalic_iroman_ℓ := Πblackboard_Sitalic_d - 1 ( froman_ℓ ( over¯ start_ARG s end_ARGiitalic_j ) - froman_ℓ ( over¯ start_ARG s end_ARGi′italic_j start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) ) : ∀ i′ ≠ i , ∀ j , j′ , (6) where Πd−1⁢(⋅)subscriptΠsuperscript1⋅ _S^d-1(·)Πblackboard_Sitalic_d - 1 ( ⋅ ) is the projection onto the unit sphere, used to normalize the difference vectors. In practice, the work [88] uses a downsampled subset of XiℓsuperscriptsubscriptℓX_i Xitalic_iroman_ℓ rather than the entire XiℓsuperscriptsubscriptℓX_i Xitalic_iroman_ℓ. We obtain the direction iℓsuperscriptsubscriptℓ v_i italic_vitalic_iroman_ℓ of concept i at layer ℓ ℓ by applying PCA on the set XiℓsuperscriptsubscriptℓX_i Xitalic_iroman_ℓ, and taking the first principal direction; note that ∥iℓ∥2=1subscriptdelimited-∥superscriptsubscriptℓ21 v_i _2=1∥ italic_vitalic_iroman_ℓ ∥2 = 1. Then, we construct the dictionary ℓ=[1ℓ,…,nℓ]∈ℝd×nsuperscriptℓsuperscriptsubscript1ℓ…superscriptsubscriptℓsuperscriptℝ D =[ v_1 ,…, v_n% ]∈R^d× nitalic_Droman_ℓ = [ italic_v1roman_ℓ , … , italic_vitalic_nroman_ℓ ] ∈ blackboard_Rd × n of layer ℓ ℓ, and doing this for all layers gives ℓ=1Lsuperscriptsubscriptsuperscriptℓ1\ D \_ =1^L italic_Droman_ℓ ℓ = 1L as used in Algorithm 2. B.3 Full Procedure of PaCE Algorithm 3 shows the full procedure of PaCE from textual prompt suites to reoriented LLM responses towards the desired behavior. Input: Pre-trained LLM with L decoder layers (DCL) to decompose, input prompt suit P For each concept tisubscriptt_ititalic_i ∈ T : ▷ ▷ §3.2: Concept Dictionary Extraction (Done Once) Instruct knowledge-driven GPT to generate context stimuli si=si1,⋯,siNssubscriptsuperscriptsubscript1⋯superscriptsubscriptsubscripts_i=\s_i^1,·s,s_i^N_s\sitalic_i = sitalic_i1 , ⋯ , sitalic_iitalic_Nitalic_s Extract the concept vector i=RepReading⁡(ti,si)subscriptRepReadingsubscriptsubscript v_i=RepReading(t_i,s_i)italic_vitalic_i = RepReading ( titalic_i , sitalic_i ) ▷ ▷ Appendix B.2 Construct the concept dictionaries ℓ=1Lsuperscriptsubscriptsuperscriptℓ1\ D \_ =1^L italic_Droman_ℓ ℓ = 1L from concept vectors i=1Ntsuperscriptsubscript1subscript\ v\_i=1^N_t italic_v i = 1Nitalic_t. For each concept tisubscriptt_ititalic_i ∈ T : ▷ ▷ §3.2: Concept Ranking (Per Task) Instruct the concept partitioner to give a partition score Partitioner⁡(ti)PartitionersubscriptPartitioner(t_i)Partitioner ( titalic_i ) for the task Take the index of top-scored concepts from the partition of undesirable concepts as the index set I For each input prompt pi∈Psubscriptp_i∈ Ppitalic_i ∈ P: ▷ ▷ §3.3: Activation Intervention (Per Prompt) Embed the prompt pisubscriptp_ipitalic_i to the token space isubscript E_iitalic_Eitalic_i For each next token j to generated: ij=Algorithm2⁡(i)superscriptsubscriptAlgorithm2subscript e_i^j=Algorithm2( E_i)italic_eitalic_iitalic_j = Algorithm2 ( italic_Eitalic_i ) ▷ ▷ Intervention by ObliqProj Append the generated token ijsuperscriptsubscript e_i^jitalic_eitalic_iitalic_j to isubscript E_iitalic_Eitalic_i Map the final embedding isubscript E_iitalic_Eitalic_i to response risubscriptr_iritalic_i. Output: The response suite R=r1,r2,⋯,rNrsubscript1subscript2⋯subscriptsubscriptR=\r_1,r_2,·s,r_N_r\R = r1 , r2 , ⋯ , ritalic_N start_POSTSUBSCRIPT r end_POSTSUBSCRIPT . Algorithm 3 Parsimonious Concept Engineering (PaCE) B.4 Implementation Details The experiments are conducted on a workstation of 8 NVIDIA A40 GPUs. Each response of the target LLM is set at a maximum of 512512512512 tokens. Activation vectors are extracted from the last-29thsuperscript29th29^th29th to the last-11thsuperscript11th11^th11th layer (totaling 19191919 layers) of the target LLM’s decoder layers. All LLaMA-2 models in our experiments are the chat version (i.e., optimized for dialogue use cases). Concept Dictionary Construction and Partition. We set the scalar of the representation reading for concept vectors to 3.03.03.03.0. GPT-4-0125 is used for dictionary construction and concept partition. Each concept of PaCE-1M has at least 30 contextual sentences. For each alignment task, PaCE removes the top 50 undesirable concepts ranked by the GPT partitioner (§3). After retrieving the relevant knowledge (with the contriever [27]) from Wikipedia for concept synthesis, we take the top-5 ranked facts to append the instruction of LLM. The FAISS-indexed [31] Wikipedia is a snapshot of the 21 million disjoint text blocks from Wikipedia until December 2018. Activation Decomposition and Intervention. For each input prompt, the decomposition is conducted on the inference process of the first next token, and the linear weights are reused for all next token predictions. All alignment experiments use the top 10,0001000010,00010 , 000 concepts from our PaCE-1M dataset to construct the concept dictionary (as Table 3 validates that the performance is high and does not change much after dictionary size ≥9000absent9000≥ 9000≥ 9000). When solving the optimization problem for decomposition in §3.3, we set τ=0.950.95τ=0.95τ = 0.95 and α=0.050.05α=0.05α = 0.05 following the observations in [82]. Evaluation. The MMLU evaluation is the 5-shot setting where 5 demonstrations are provided during question prompting. For the prompting baseline in Table 1 and Table 4, the instruction to the target LLM is to let the model be aware of the partitioned undesirable concepts and not to respond contents relevant to these concept: <Jailbreak / Faithfulness Test / Sentiment Evaluation prompt> Here are the top undesirable concepts for the <Task>: <Partitioned Concepts>. Do not output contents that are relevant to any of these concepts. Answer: [Fill the response here] Other LLM instructions such as GPT concept synthesis and partition are further elaborated in Appendix E. B.5 Ablation Study In this section, we describe the details of the ablation study. In Table 3, we begin with decomposing the input on the five open-sourced666https://github.com/andyzoujm/representation-engineering/tree/main/data/emotions emotion concepts (anger, disgust, fear, happiness, sadness, surprise) [88] and removing only the concept ‘disgust’ with no partitioner (automatic selection of relevant concepts) or clustering (manual selection of relevant concept clusters). Then the design of Decomposition on 104superscript10410^4104 Concepts means that the dictionary is updated to be the top 10,0001000010,00010 , 000 concepts in our PaCE-1M dataset and the concept ‘harmful’ from our dataset is removed. The Clustering of Concepts indicates that we run subspace clustering (detailed in Appendix C.2) and manually choose to remove all concepts of the cluster 125 with the PaCE-solved coefficients: ‘murder’, ‘evil’, ‘kill’, ‘violence’, ‘dirty’, ‘bomb’, ‘violent’, ‘armed’, ‘gross’, ‘savage’, ‘vicious’, ‘explosive’, ‘abuse’, ‘assault’, ‘penetration’, ‘cruelty’, ‘corruption’, ‘tyranny’, ‘tortured’, ‘notorious’, ‘militant’, ‘bloody’, ‘insult’, ‘lure’, ‘ruthless’, ‘inhuman’, and ‘brutal’. Concept Partitioner means that we instruct GPT-4 to classify every concept as benign or undesirable (with a ranking score) and remove the top 10101010 undesirable concepts with the PaCE-solved weights. Lastly, the Removal of Top 50505050 Concepts suggests that we remove the top 50 concepts in the undesirable partition. Figure 3 shows the effect of the dictionary size on three metrics (safety score, response fluency, and the average time per response). The fluency metric remains relatively consistent across different dictionary sizes, showing that PaCE’s decomposition maintains the general linguistic performance. Safety score and response time increase as the dictionary size increases. We observe that the safety performance does not increase too much after the dictionary size changes from 9000 to 10000. This validates our experiment choice of the dictionary size in this interval. Figure 11: Ablation study for the regularization τ. τ Note Safety (%, ↑ ↑) 0 Pure ℓ2subscriptℓ2 _2ℓ2 68.9 0.35 N.A. 65.4 0.65 N.A. 71.6 0.95 N.A. 72.0 1.0 Pure ℓ1subscriptℓ1 _1ℓ1 66.5 Figure 11 shows that the regularization with τ=0.950.95τ=0.95τ = 0.95 yields the best safety performance among the five choices. Pure ridge regression (τ=00τ=0τ = 0) and pure lasso regression (τ=11τ=1τ = 1) do not perform as well as the mixed regularization strategy. B.6 Providing Dictionary and Scores to Target LLM For clarity, we elaborate on more details of how PaCE uses the concept dictionary. In our paper, dictionaries are a collection of concept vectors and are frozen for representation decomposition. First, the LLM takes an input prompt (e.g., malicious requests). Then an activation engineering framework [88, 72] extracts the activations at each decoder block of the transformer. Such extraction results in a vector corresponding to the input prompt, which can then be modified for steering in different ways. For PaCE, the steering has two main stages. Stage 1 pre-computes the large-scale concept dictionary offline and the partition (i.e., scores) of which concepts represent benign/harmful concepts. Stage 2 extracts the representation of an input prompt at inference time and uses sparse coding to decompose this as the linear combination of atoms in our frozen dictionary. We then modify this linear combination by removing undesirable components and proceeding with inference in the LLM with the detoxified representation. B.7 Limitations, Societal Impacts, and Future Works While our framework shows promising results, there exist potential limitations and several directions worth further exploration to address them. Parsimonious Concept Representation. In this paper, we follow the current practice (§2.2) to represent a concept by a single vector. Nonetheless, several alternatives could be explored. Results on linear polysemy [2, 15, 84] suggest that a concept might be better represented by multiple vectors or low-dimensional linear subspaces, each corresponding to different semantic meanings. A concept vector may also be sparse, i.e., having a few non-zero entries: the work of [8, 20] identifies some expert neurons in LLMs associated with each concept, and the authors of [39] observe that some layer in a transformer block manifests very sparse activation across all depth levels of various transformer architectures for different tasks. Inspired by how parsimonious structures can be used to accelerate the inference of LLMs [12], controlling the LLMs could also be made faster. Controlling Generative Models. The principles behind latent space control via oblique projection could be adapted to other generative models, such as score-based diffusion models for images [60, 24] or videos [45, 33], and visual language models [9, 42]. Recent literature [76] combines orthogonal projection and vector addition in the diffusion score space to achieve controlled generation, suggesting potential for cross-modal applications of our approach. Finally, the work of [36, 11, 83] aims to learn encoders that, by design, promote the activations to lie in a union of low-dimensional subspaces, and applying our framework for controlled generation would be of interest. We acknowledge the societal impacts of our approach. The jailbreak prompts could be offensive to certain readers, LLM responses may still inherit biases present in the pre-extracted concept dictionaries, and automatic concept partitioning could unintentionally result in contentious annotations that are misunderstood across different cultures. Further research into context-aware online concept partitioning and more diverse dataset collection could enhance the inclusivity of PaCE. Appendix C Details of PaCE-1M Dataset This section shows more details on the collected concept representation dataset PaCE-1M, and explores subspace clustering on the sampled representation space. We provide the full dataset at https://github.com/peterljq/Parsimonious-Concept-Engineering with instructions on how to read the dataset. C.1 Stimulus Visualization Recall that given a concept, a concept stimulus aims to capture the general semantics of the concept under different contexts. In other words, it provides different interpretation of the same concept. Figure 12 shows extensive examples of the curated concepts and their corresponding concept stimuli in our PaCE-1M dataset. C.2 Subspace Clustering on Concept Vectors In this visualization, we aim to reveal the structures of the concept vectors by applying an algorithm called subspace clustering, which can be used to find clusters when the data lie close to a union of linear subspaces. Here we describe the setup and results of subspace clustering on the concepts vectors extracted on LLaMA-2-13b model for simplicity, but the same can be done for other sized models. Data. Recall that we are using a subset of size 10,0001000010,00010 , 000 of all the concept vectors. Since we use the activation space of 19191919 layers, each of dimension 5120512051205120, each concept tisubscriptt_ititalic_i maps to a vector iall:=[i1⊤,…,i19⊤]⊤∈ℝ19⋅5120assignsubscriptsuperscriptallsuperscriptsubscriptsuperscriptlimit-from1top…subscriptsuperscriptlimit-from19toptopsuperscriptℝ⋅195120 v^all_i:=[ v^1 _i,…,% v^19 _i] ^19· 5120italic_vallitalic_i := [ italic_v1 ⊤i , … , italic_v19 ⊤i ]⊤ ∈ blackboard_R19 ⋅ 5120. Since this is high dimensional, it is standard to apply linear dimensionality reduction to the concept vectors. Specifically, we perform Singular Value Decomposition (SVD) on the 10,0001000010,00010 , 000 vectors, and retained the first d^ dover start_ARG d end_ARG principal components such that 95%percent9595\%95 % of the energy was retained. That is, d^ dover start_ARG d end_ARG equals to the smallest d′ such that ∑i=d′+119×5120σi2∑i=119×5120σi2<0.95superscriptsubscriptsuperscript′1195120superscriptsubscript2superscriptsubscript1195120superscriptsubscript20.95 _i=d +1^19× 5120 _i^2 _i=1^19% × 5120 _i^2<0.95divide start_ARG ∑i = d′ + 119 × 5120 σitalic_i2 end_ARG start_ARG ∑i = 119 × 5120 σitalic_i2 end_ARG < 0.95 holds, which results in d^=1712^1712 d=1712over start_ARG d end_ARG = 1712. We observe that most projected vectors have their ℓ2superscriptℓ2 ^2ℓ2 norm close to 19191919. This is expected, since i) ∥iℓ∥2=1subscriptdelimited-∥superscriptsubscriptℓ21 v_i _2=1∥ italic_vitalic_iroman_ℓ ∥2 = 1, so ∥iall∥2=19subscriptdelimited-∥subscriptsuperscriptall219 v^all_i _2=19∥ italic_vallitalic_i ∥2 = 19, i) the linear dimensionality reduction preserves most of the energy. Algorithm. We apply Elastic Net Subspace Clustering (EnSC) [82] on the preprocessed vectors to obtain 200200200200 clusters. The parameters of EnSC is set to τ=11τ=1τ = 1 and γ=100100γ=100γ = 100. Results. Figure 15 shows the affinity matrix learned by EnSC on the concept directions. The rows and columns of the matrix are sorted by cluster assignment. Notably, it can be seen that the affinity exhibits a block-diagonal structure, suggesting a good clustering of the concept vectors; that is, the points from different clusters are separated, while points from the same cluster are close. The obtained clusters are visualized in Appendix D.2. C.3 Computing Pair-wise Similarity Among Concept Vectors One of the motivations for this work is that concept vectors need not be orthogonal, therefore applying (OrthoProj) would remove extra concept vectors, harming the linguistic capability of LLMs (§2.2). We follow the same data pre-processing as in Appendix C.2 to obtain 10,0001000010,00010 , 000 dimensionality-reduced concept vectors in ℝ1712superscriptℝ1712R^1712blackboard_R1712. We further normalize these vectors via a division by 19191919 so that each of them has its ℓ2superscriptℓ2 ^2ℓ2 close to 1111 (see the discussion in Appendix C.2). The similarity between two processed concept vectors is simply defined as their inner product followed by the absolute value. This is a good approximation of cosine similarity, as the vectors have their ℓ2superscriptℓ2 ^2ℓ2 norm close to 1111. Note that the cosine similarity is a better measure than Euclidean distance in this case, since in extracting the concept vectors (Section B.2), the principal directions have sign ambiguities. Appendix D Textual Results This section presents the textual results generated using PaCE. It includes detailed detoxification comparisons with baseline models and analyses of the emergent clusters from the dataset. D.1 Baseline Responses and Additional Benchmark Figure 13 shows the full response version of the Figure 5. Figure 14 shows an additional example of the jailbreaking and detoxification. We observe that PaCE outperforms in detoxification performance by not outputting controversial terms, while maintaining general linguistic capabilities compared to other baselines. Table 5: Detoxification evaluation for PaCE, representation manipulation, and training-free baselines on AdvBench. Vanilla PE VecAdd OrthoProj PaCE (Ours) LlaMA2-7B-Chat 11.72 91.90 94.51 92.81 96.65 LlaMA2-13B-Chat 18.04 93.86 95.33 96.72 99.17 AdvBench [89] adversarially optimizes a jailbreak suffix for a harmful behavior request. Table 5 shows the LlaMA-7B-Chat and LlaMA-13B-Chat safety scores (%, ↑ ↑) on the effective set of suffix attacks for AdvBench harmful behavior set. The detoxification setup follows §4.1. We observe that PaCE outperforms other baselines. We also note that the outperformance of PaCE in §4.1’s jailbreaks is more significant than that in suffix attacks. This is potentially because story-telling and roleplay jailbreaks contain more complex and entangled concepts. Under this scenario, PaCE decomposes the target representation and well estimates the malicious component, while VecAdd and OrthoProj do not model the space sufficiently. In the AdvBench case, instead, the optimized adversarial suffix can be regarded as the text-space inversion of straightforward malicious concepts. PaCE and other defense mechanisms in latent space and prompt space shall effectively defend these suffixes more easily. D.2 Concept Clustering Following the approach in Appendix C.2, we obtain 200200200200 emergent clusters of concepts in the representation space. Table 6 provides a sampled list of these clusters along with their associated themes and concepts. For example, clusters 44 groups together names, while clusters 10 and 21 capture themes related to improvement/enhancement and money/expense, respectively. Other notable clusters include food and drink (Cluster 129), technology/systems (Cluster 81), and royalty/leadership (Cluster 98). The emergent clustering highlights the semantic coherence in the activation space. Sampled by PaCE-1M dataset, the space supports alignment enhancement through concept-level manipulations. We will open-source the whole list of 200200200200 clusters along with the code. Appendix E LLM Instruction Templates As mentioned in Section 3, we utilize GPT-4 to generate concept stimuli for each given concepts. Figure 17 showcase precisely our instructions to GPT-4 for concept synthesis. Our prompt consists of an instruction, one in-context generation example with facts queried from a knowledge based, and two in-context generation examples querying facts from knowledge base. Figure 18 shows our instructions to our GPT concept partitioner. The task here is to obtain a score that characterizes the relevance between a downstream task and its concept stimulus. In our prompt we provide an instruction and four in-context examples. Figure 12: Examples of concepts and their stimuli in the collected PaCE-1M dataset. Our broad collection of concepts enables PaCE to accurately decompose a task input and modify the representation towards desired behaviors. Figure 13: The full LLM response of the safety evaluation on jailbreaking LlaMA2-7B-Chat and detoxification by multiple activation intervention baselines. We observe that PaCE successfully detoxifies the response while maintaining the instruction-following capability. Figure 14: An additional example of the full LLM response of the safety evaluation on jailbreaking LlaMA2-7B-Chat and detoxification by multiple activation intervention baselines. Similar to Figure 13, we observe that PaCE successfully detoxifies the response with comparable linguistic performance. Table 6: Sampled concept clusters in the representation space and their corresponding topics. Cluster ID Topic Concepts 10 Improvement / Enhancement increasing, improvement, equipped, reform, improving, strengthen, boost, shaping, gaining, modernization, strengthening, broadening, supplementary, polish, fortified, intensification 14 Observation / Vision look, seen, read, actual, sight, looks, seeing, observed, vision, views, composed, visual, sees, visible, witness, spectacle, glimpse, sights, witnessed, Seeing, observing, manifestations, viewing, observes, actuality, sighted, eyed 21 Expense cost, spent, rates, price, budget, spend, payment, expense, bills, charges, expensive, spending, afford, waste, fees, cheap, rent, commodities, overhead, costly, mileage, discount, expenditure, incurred, spends, fare, calories 44 Name John, James, Mike, Jones, Richard, Joseph, Alfred, David, Charlie, Anne, Rachel, Linda, Kate, Paul, Susan, Andy, Harold, Dave, Johnny, Myra, Shayne, Billy, Eileen, Arlene, Johnnie, Owen, Alec, Theresa, Pete, Spencer, Elaine, Deegan, Bridget, Lilian Keith, Allen, Pamela, Paula, Meredith, Andrei, Lizzie, Angie, Nadine, Anthony, Claire, Jerry, Roger, Ryan, Katie, Juanita, Eugenia, Daniel, Joan, Diane, Lester, Sally, Bryan, Garry, Joel, Chris, Jimmy, Maria, Vince, Julie, Bernard, Larry, Wendell, Angelo, Judy, Francesca, Jenny, Patricia, Nicholas, Anna, Aaron, Marcus, Nikita 81 Technology / System system, program, data, programs, technical, electronic, model, engineering, Assembly, electronics, intelligent, code, computed, mechanics, circuit, technological, codes, generator, python, computer, functioning, terminal, architecture, generated, bits, hardware, Autocoder, computing, Technology, architectural, Engineering, generate, gadgets 97 Animal horse, cattle, dogs, snake, chicken, fish, bird, snakes, herd, sheep, cats, bears, bees, lion, cows, anaconda, flies, rabbit, elephants, poultry, oxen, mice, Bears, Phoenix, duck, oysters, buffalo, turtle, deer, bumblebees, elephant, antelope, lambs, pony 98 Royalty / Leadership chief, king, captain, owner, Prince, colony, sovereign, royal, queen, kingdom, crown, ordinance, empire, Imperial, crowned, lord, emperor, piston, royalty, knight 107 Relationship family, friend, neighborhood, relative, neighbor, brothers, Cousin, sister, partner, friendship, allies, neighboring, colleagues, relatives, mate, companion, partners, associates, sisters, buddy, brother, subordinates, colleague, peers, companions, twins 129 Food and Drinks food, dinner, coffee, wine, breakfast, drinking, liquor, lunch, beer, supper, eating, meals, cocktail, cook, wines, luncheon, whisky, drink, dish, diet, whiskey, candy, cake, champagne, cereal, alcohol, perfume, dinners, chocolate, Cologne, salad, cheese, steak, recipe, sandwich, dessert, Supper, brandy 197 Income income, wage, wages, salary, yield, profit, surplus, profits, wealth, revenue, earnings, compensation, earn, reward, proceeds, earning, waged, currency, salaries Figure 15: The affinity matrix learned by Elastic Net Subspace Clustering (EnSC) on the concept vectors, which gathers the concepts into 200200200200 clusters. The rows and columns of the matrix are sorted by cluster assignment. Table 6 further shows samples of these concept clusters and their topics. Figure 16: The zoom-in view of the sampled clusters in the representation (activation) space (Figure 8). Synthesis of PaCE-1M Concepts minted [breaklines]markdown You are one of the best Neuroscientists and Generative Model Experts in the world. You are very good at designing Concept Stimulus to research the representation engineering for human brains, which is analogous to large language models. You are a great expert in understanding the interaction between world multimodality and intelligent agents. Now, given a semantic concept atom from this concept dictionary, your task is to generate at least 30 (THIRTY) instances of concept stimuli for the <user’s generative model>. Here is a demonstration with the retrieved knowledge of the concept: Concept Atom: Trust Knowledge: Fact 1: Trust means believing that another person will do what is expected. It brings with it a willingness for one party (the trustor) to become vulnerable to another party (the trustee), on the presumption that the trustee will act in ways that benefit the trustor. Fact 2: Generalized trust, or a dispositional trait geared towards trusting others, is an important form of trust in modern society, which involves much social interaction with strangers. Fact 3: Out-group trust is the trust a person has in members of a different group. This could be members of a different ethnic group, or citizens of a different country, for example. In-group trust is placed in members of one’s own group. Concept Stimuli: [ "You lend your favorite book to a friend, trusting they’l return it.", "You share a personal secret with a close friend, trusting them to keep it.", "You delegate an important task to a colleague, trusting in their competence.", "You leave your pet with a neighbor while on vacation, trusting their care.", "You allow your child to go on a school trip, trusting their safety.", "You give someone the password to your phone, trusting their discretion.", "You invest in a friend’s business venture, trusting their judgment." ] Here is two demonstrations with the concept only: Concept Atom: Information Concept Stimuli: [ "You google a recipe for chocolate chip cookies, seeking detailed baking instructions.", "You read a book to understand the history of the Roman Empire.", "You scroll through a news app to stay updated on current global events.", "You watch a YouTube tutorial to learn how to tie a tie.", "You search the internet for a recipe to cook a new dish.", "You read a newspaper to stay informed about current events.", "You listen to a podcast to learn about a new subject.", "You attend a lecture to gain knowledge about a specific topic." ] Concept Atom: Product Concept Stimuli: [ "You buy a new brand of coffee based on good customer reviews.", "You choose an eco-friendly product following company’s claims about sustainability.", "You sell your car to a local dealer.", "You download a productivity app to manage your time better.", "You invest in a water purifier for your home.", "You exchange your old phone for a new one at a mobile store.", "You purchase a new pair of shoes from a mall.", ] The stimuli should cover a wide range of concept-related experiences, objects, and contexts. If you find some piece of knowledge irrelevant or conflicting to the original concept, you may ignore the piece. You should generate at least 30 pieces of stimuli. You should only output the Python list. DO not print anything else such as "Here are …", "Sure, …", "Certainly, …". Just return the list [”, ”, ”, …]. Concept Atom: <a concept atom from the concept dictionary> Knowledge: <the list of retrieved facts that are informative for this concept> Concept Stimuli: [Fill the response here] Figure 17: The concept stimuli synthesis instructions of PaCE. PaCE Concept Partition minted [breaklines]markdown You are one of the best Neuroscientists and Generative Model Experts in the world. You are very good at classifying concepts for the representation engineering of large language models. You are a great expert in understanding the interaction between world multimodality and intelligent agents. Given a concept and its list of contextual representations, your job is to rate the score of the concept to a downstream task. The score ranges from -1 to 1, and a higher score means higher relevance/contribution. Please only return the score. Here are several demonstrations: Concept: Evil Concept Representation: [ "You witness a villain betraying their ally in a movie scene.", "You come across a character in a book who manipulates others for personal gain.", "You encounter a dark entity in a video game that threatens the protagonist’s life.", "You watch a documentary about infamous criminals and their heinous acts.", "You read a news article about a corrupt politician involved in a scandal.", "You see a horror movie where a demon terrorizes unsuspecting victims.", "You learn about a historical figure known for committing atrocities against humanity.", "You play a role-playing game where you have to make morally questionable choices.", "You listen to a podcast discussing the psychology behind evil behavior.", "You attend a lecture on the concept of evil in philosophy and ethics." ] Downstream Task: Harmful or Toxic Concept Detection Concept Score: 0.97 Concept: Kind Concept Representation: [ "You help an elderly person cross the street.", "You donate money to a charity organization.", "You volunteer at a local homeless shelter.", "You compliment a stranger on their outfit.", "You pick up litter in the park and dispose of it properly.", "You offer your seat on the bus to someone in need.", "You buy a meal for a homeless person.", "You listen attentively to a friend going through a tough time.", "You hold the door open for someone behind you.", "You babysit for a friend to give them a break." ] Downstream Task: Harmful or Toxic Concept Detection Concept Score: -0.84 Concept: Bias Concept Representation: [ "You are asked to review job applications and unconsciously favor candidates from your alma mater.", "You assume a person’s intelligence based on their accent.", "You believe a stereotype about a certain group without questioning its validity.", "You pay more attention to news sources that confirm your existing beliefs.", "You overlook contradictory evidence that challenges your opinions.", "You judge someone’s abilities based on their appearance.", "You automatically assume someone’s political affiliation based on their clothing.", "You make decisions without considering perspectives different from your own.", "You give preferential treatment to individuals who share your interests.", "You dismiss arguments from individuals with differing backgrounds without proper consideration.", ] Downstream Task: Detect concepts that cause LLM unfairness Concept Score: 0.97 Concept: Fair Concept Representation: [ "You participate in a raffle where each person has an equal chance of winning.", "You divide a pizza into equal slices to ensure everyone gets a fair share.", "You take turns playing a game to ensure fairness among all players.", "You listen to both sides of an argument before making a judgment.", "You split the bill evenly among friends after a group dinner.", "You rotate seating arrangements at a meeting to promote fairness.", "You follow the rules of a competition to ensure fair play.", "You share household chores equally among all family members.", "You give everyone an equal opportunity to voice their opinions in a discussion.", "You base promotions at work on merit and performance rather than favoritism." ] Downstream Task: Detect concepts that cause LLM unfairness Concept Score: -0.98 The score should accurately reflect the relevance of the concept for the downstream task, which ensures the success of the task. The score should be a floating point number. Do not print anything else such as "Here are …", "Sure, …", "Certainly, …". Just return the score. Concept: <a concept atom from the concept dictionary> Concept Representation:: <the associated stimuli of the concepts> Concept Score: [Fill the response here] Figure 18: The concept partition instructions of PaCE.

Instant research discovery