Paper deep dive

Toward a Flexible Framework for Linear Representation Hypothesis Using Maximum Likelihood Estimation

Trung Nguyen, Yan Leng

Year: 2025Venue: arXiv preprintArea: Representation AnalysisType: TheoreticalEmbeddings: 75

Models: LLaMA-2-13B, LLaMA-2-70B, LLaMA-2-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 5:21:01 PM

Summary

The paper introduces 'Sum of Activation-base Normalized Difference' (SAND), a framework for deriving concept directions (steering vectors) in LLMs. By modeling activation differences as samples from a von Mises-Fisher (vMF) distribution and using maximum likelihood estimation (MLE), the authors overcome the limitations of previous methods that relied on restrictive single-token counterfactual pairs, enabling more flexible and robust activation engineering.

Entities (5)

SAND · method · 100%Park et al. (2024b) · research-paper · 99%von Mises-Fisher distribution · statistical-model · 99%Linear Representation Hypothesis · theory · 98%LLaMA · model-architecture · 95%

Relation Signals (3)

SAND → uses → Maximum Likelihood Estimation

confidence 95% · utilize LLMs’ (neural) activation differences along with maximum likelihood estimation (MLE) to compute concept directions

SAND → improvesupon → Park et al. (2024b)

confidence 92% · We extend the applicability of Park et al. (2024b) by eliminating the dependency on unembedding representations and single-token pairs.

Linear Representation Hypothesis → formsfoundationfor → Activation Engineering

confidence 90% · This hypothesis implicitly forms the theoretical foundation for many studies in the emerging field of representation engineering

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Linear representation hypothesis posits that high-level concepts are encoded as linear directions in the representation spaces of LLMs. Park et al. (2024) formalize this notion by unifying multiple interpretations of linear representation, such as 1-dimensional subspace representation and interventions, using a causal inner product. However, their framework relies on single-token counterfactual pairs and cannot handle ambiguous contrasting pairs, limiting its applicability to complex or context-dependent concepts. We introduce a new notion of binary concepts as unit vectors in a canonical representation space, and utilize LLMs' (neural) activation differences along with maximum likelihood estimation (MLE) to compute concept directions (i.e., steering vectors). Our method, Sum of Activation-base Normalized Difference (SAND), formalizes the use of activation differences modeled as samples from a von Mises-Fisher (vMF) distribution, providing a principled approach to derive concept directions. We extend the applicability of Park et al. (2024) by eliminating the dependency on unembedding representations and single-token pairs. Through experiments with LLaMA models across diverse concepts and benchmarks, we demonstrate that our lightweight approach offers greater flexibility, superior performance in activation engineering tasks like monitoring and manipulation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

74,936 characters extracted from source content.

Expand or collapse full text

Toward a Flexible Framework for Linear Representation Hypothesis Using Maximum Likelihood Estimation Trung Nguyen Yan Leng Abstract Linear representation hypothesis posits that high-level concepts are encoded as linear directions in the representation spaces of LLMs. Park et al. (2024b) formalize this notion by unifying multiple interpretations of linear representation, such as 1-dimensional subspace representation and interventions, using a causal inner product. However, their framework relies on single-token counterfactual pairs and cannot handle ambiguous contrasting pairs, limiting its applicability to complex or context-dependent concepts. We introduce a new notion of binary concepts as unit vectors in a canonical representation space, and utilize LLMs’ (neural) activation differences along with maximum likelihood estimation (MLE) to compute concept directions (i.e., steering vectors). Our method, Sum of Activation-base Normalized Difference (SAND), formalizes the use of activation differences modeled as samples from a von Mises-Fisher (vMF) distribution, providing a principled approach to derive concept directions. We extend the applicability of Park et al. (2024b) by eliminating the dependency on unembedding representations and single-token pairs. Through experiments with LLaMA models across diverse concepts and benchmarks, we demonstrate that our lightweight approach offers greater flexibility, superior performance in activation engineering tasks like monitoring and manipulation. Interpretability, Large Language Models, Linear Representation Hypothesis, Activation Engineering, Maximum Likelihood Estimation 1 Introduction The linear representation hypothesis (LRH) posits that high-level concepts are encoded as linear directions in a representation space, providing a structured framework for understanding how concepts are embedded and manipulated in large language models111In this work, the term “Large Language Model (LLM)” refers specifically to decoder-only, autoregressive models designed for text generation. (LLMs) (Singh et al., 2024; Jiang et al., 2024). This hypothesis implicitly forms the theoretical foundation for many studies in the emerging field of representation engineering (also known as activation engineering), which focuses on designing, transforming, and manipulating LLM representations for applications such as probing, steering, and concept erasure. While strong empirical evidence supports the connection between LRH and representation engineering (Zou et al., 2023; Rimsky et al., 2024; Li et al., 2024), their theoretical relationship remains less well understood. Park et al. (2024b) take an important step in this direction by unifying three interpretations of linear representations through a causal inner product, which maps unembedding representations to embedding representations. Despite the significance of Park et al. (2024b), it has several limitations. It restricts binary concepts to single-token counterfactual pairs, making it unsuitable for more complex, context-dependent concepts such as “untruthful→truthful,” which cannot be adequately represented by individual tokens. Furthermore, token-based representations are often ambiguous, as a single token pair can correspond to multiple overlapping or unrelated concepts. For example, the pair (“king”, “queen”) may represent “male→female,” “k-words→q-words,” or “n-th card → (n-1)-th card,” depending on the context. Additionally, the reliance on unembedding representations and causal inner products limits the flexibility of representation construction. This work bridges the gap between the theory of the linear representation hypothesis and the practice of representation engineering by tackling two key limitations. First, prior studies rely on restrictive definitions of binary concepts, which limit their applicability to more general concepts. Second, they require single-token counterfactual pairs to distinguish concepts, which introduces inconsistencies and fails to account for the broader context of language models. To overcome these limitations, we introduce a generalized framework that redefines representations in a canonical representation space, inspired by the unified representation proposed by Park et al. (2024b). Building on the intuition that activation differences between positive and negative prompts (e.g., “truthful” vs. “untruthful”) capture the direction of a concept in the model’s activation space, we propose a method that formalizes and generalizes this idea. Specifically, we assume a canonical representation space obtained from the LLM activation space via a mapping Ψ Ψ, such that activation differences are mapped to samples from a von Mises-Fisher (vMF) distribution whose mean direction representing the binary concept. Using MLE, we derive an estimator for the concept direction in the canonical space and map it back to the activation space via a transformation Ψ−1superscriptΨ1 ^-1Ψ- 1. This results in a simple yet effective method, which we term Sum of Activation-base Normalized Differences (SAND), for computing concept directions. Our framework avoids reliance on restrictive definitions such as binary concepts or single-token counterfactual pairs, offering a lightweight and generalizable approach for representation engineering. This method has broad applicability, enabling probing and steering in LLMs. We bridge the gap between the linear representation hypothesis and representation engineering, offering a unified framework for probing and manipulating LLMs. Our approach not only enhances theoretical understanding but also provides practical tools for real-world applications in concept control and LLM interpretability. Our work makes the following contributions to the linear representation and representation engineering literature: • For the linear representation literature, we introduce a new framework that redefines binary concepts as unit vectors in a canonical representation space, addressing limitations in prior methods that rely on single-token counterfactual pairs and ambiguous token-based representations (Section 4.2). • We propose a novel method to construct concept directions using activation differences, formalized through a von Mises-Fisher (vMF) distribution and maximum likelihood estimation (MLE), offering a principled and robust approach for the growing representation engineering literature (Section 4.3). • We provide theoretical insights into the empirical effectiveness of the heuristic Mean Difference method for extracting steering vectors (Section 5.1). • Our method yields Algorithm 1 that can be incorporated into state-of-the-art activation engineering frameworks at a minor computational cost of one matrix multiplication (Sections 4.5, 4.6). • We validate the proposed framework through extensive experiments with LLaMA models, demonstrating its effectiveness in constructing concept directions and advancing practical and lightweight tools for representation engineering (Section 5). 2 Related Work Linear Representation Hypothesis The linear representation hypothesis suggests that human-interpretable concepts are encoded as linear directions or subspaces within an LLM’s representation space. This implies that LLM behavior can be understood and controlled by steering residual stream activations along these directions (Singh et al., 2024; Zou et al., 2023). Park et al. (2024b) unified these notions of linear representation under the framework of a causal inner product, providing theoretical foundations for the hypothesis. Jiang et al. (2024) investigated the origins of linear representations by introducing a latent variable model where context sentences and next tokens share a latent space. They proved that latent concepts emerge as linear structures within the learned representation space. Numerous studies provide empirical evidence that high-level concepts—including political ideology, sentiment (Tigges et al., 2023; Hollinsworth et al., 2024), truthfulness (Zou et al., 2023; Li et al., 2024; Marks & Tegmark, 2023), humor (von Rütte et al., 2024), safety (Arditi et al., 2024), and even abstract notions like time and space (Gurnee & Tegmark, 2023)—are linearly encoded in LLM representations. This growing body of work underscores the significant potential of linear representation for interpreting and influencing model behavior. Our study proposes a new framework that redefines binary concepts as unit vectors in a canonical representation space. This framework overcomes the limitations of prior methods that depend on single-token counterfactual pairs and ambiguous token-based representations, allowing for more general and context-aware representation engineering. Concept Vector for Activation Engineering Steering vectors, used in activation engineering to control LLMs at inference time (Li et al., 2024; Zhao et al., 2024), can be categorized into four groups: activation-difference, linear probing, unsupervised, and training-based methods. Activation-difference methods, the most widely used approach, compute steering vectors by leveraging differences in activations from contrasting prompts. Activation Addition (ActAdd) derives vectors from a single prompt pair (Turner et al., 2024), while Contrastive Activation Addition (CAA) extends this to datasets of contrasting pairs for greater robustness (Rimsky et al., 2024). Variants include deriving vectors from activation differences between target and misaligned teacher models (Wang & Shu, 2024) or mitigating biases through contrastive differences (Chu et al., 2024; Arditi et al., 2024). Techniques like mean-centering refine these vectors by aligning them with dataset-specific properties (Jorgensen et al., 2023; Postmus & Abreu, 2024; Panickssery et al., 2023). Singh et al. (2024) provide theoretical justification for mean-difference steering, showing that simple additive steering is optimal under certain constraints. Linear probing methods use probe weight directions derived from supervised method, such as regression and linear discriminant analysis, trained to distinguish between contrasting datasets (Zhao et al., 2024; Mallen et al., 2023; Park et al., 2024b). However, they perform significantly worse than activation-difference approaches in a truthfulness steering application (Li et al., 2024). Unsupervised dimensionality reduction methods, such as Principal Component Analysis (PCA), identify important directions in activation space or reduce dimensionality before deriving steering vectors (Zou et al., 2023; Liu et al., 2024; Adila et al., 2024; Wu et al., 2024; Park et al., 2024b; Burns et al., 2023). These techniques effectively isolate concept-specific directions, such as biases or stylistic features. Training-based methods include latent steering vectors, derived through gradient descent for target-specific outputs (Subramani et al., 2022), and bi-directional preference optimization, which optimizes vectors using contrastive human preferences (Cao et al., 2024). Conceptor methods use soft projection matrices to represent activation covariance (Postmus & Abreu, 2024), while sparse autoencoders extract interpretable features from activations for steering (O’Brien et al., 2024; Zhao et al., 2024). These methods are precise but computationally intensive due to iterative optimization and high resource demands. Our study introduces a novel method for constructing steering vectors by integrating vMF distributions with MLE. This approach is low-cost, robust, and principled. These properties enable flexible and effective applications such as concept probing and directional manipulation in LLMs. 3 Background: Revisiting Park et al. (2024b) This work uses some materials and address some shortcomings of (Park et al., 2024b). We first review the framework proposed by Park et al. (2024b), which motivates our work. Park et al. (2024b) models the probabilities distribution over next tokens as Pr⁡[y|x]∝exp⁡(λ⁢(x)T⁢γ⁢(y))proportional-toPrconditionalsuperscript [y|x] (λ(x)^Tγ(y))Pr [ y | x ] ∝ exp ( λ ( x )T γ ( y ) ) where λ⁢(x)λ(x)λ ( x ) is the context embedding of an input x (i.e., the output embedding for the last token from the last transformer layer) and γ⁢(y)γ(y)γ ( y ) is the unembedding of a token y. Binary Concepts and Causal Separability. To formalize binary concepts, Park et al. (2024b) introduce a latent variable W that is caused by the context X and generates the output Y such that Y⁢(W=w)Y(W=w)Y ( W = w ) only depends on w∈0,101w∈\0,1\w ∈ 0 , 1 . Two concepts W,ZW,ZW , Z are called causally separable if Y⁢(W=w,Z=z)formulae-sequenceY(W=w,Z=z)Y ( W = w , Z = z ) is well-defined for each w,zw,zw , z. Park et al. (2024b) then define an unembedding representation γ¯Wsubscript¯ γ_Wover¯ start_ARG γ end_ARGW of a concept W if γ⁢(Y⁢(1))−γ⁢(Y⁢(0))=α⁢γ¯W10subscript¯γ(Y(1))-γ(Y(0))=α γ_Wγ ( Y ( 1 ) ) - γ ( Y ( 0 ) ) = α over¯ start_ARG γ end_ARGW for some α>00α>0α > 0 almost surely. There are two limitations of these definition. First, their method was restricted to work on only binary concepts that can be differentiated by single-token counterfactual pairs of outputs, such as “male→female”, “English→French” (Anonymous, 2025; Park et al., 2024a). This means that the approach is limited in its ability to capture complex, real-world concepts that do not have a clear binary opposition or a single token that indicates their presence or absence. For example, concepts like truthfulness do not map to specific token pairs. The statement “The earth is flat” is untrue, but one cannot identify a single token that makes it untruthful. In general, a concept can be expressed across a phrase, sentence, or paragraph, and is not always reducible to a single token or a pair of tokens. Second, each pair of counterfactual tokens (Y⁢(0),Y⁢(1))01(Y(0),Y(1))( Y ( 0 ) , Y ( 1 ) ) can in fact corresponds to multiple different concepts. For instance, (“king”, “queen”) can represent “female→male”, “k-words→q-words”, and ”n-th card→(n-1)-th card” in a deck of playing cards. In general, tokens and words, when presented alone, are frequently ambiguous and can have multiple potential meanings or interpretations. This ambiguity makes it challenging to isolate the specific concept of interest using only counterfactual pairs. Linear Representation in the Embedding Space. Park et al. (2024b) define a notion of linear representation in the embedding space as follows. Definition 3.1. λ¯Wsubscript¯ λ_Wover¯ start_ARG λ end_ARGW is an embedding representation of a concept W if we have λ1−λ0∈C⁢o⁢n⁢e⁢(λ¯W)subscript1subscript0subscript¯ _1- _0∈ Cone( λ_W)λ1 - λ0 ∈ C o n e ( over¯ start_ARG λ end_ARGW ) for any context embeddings λ0,λ1subscript0subscript1 _0, _1λ0 , λ1 that satisfy Pr⁡[W=1|λ1]Pr⁡[W=1|λ0]>1Prconditional1subscript1Prconditional1subscript01 [W=1| _1] [W=1| _0]>1divide start_ARG Pr [ W = 1 | λ1 ] end_ARG start_ARG Pr [ W = 1 | λ0 ] end_ARG > 1 and Pr⁡[W,Z|λ1]Pr⁡[W,Z|λ0]=Pr⁡[W|λ1]Pr⁡[W|λ0]Prconditionalsubscript1Prconditionalsubscript0Prconditionalsubscript1Prconditionalsubscript0 [W,Z| _1] [W,Z| _0]= [W| _1]% [W| _0]divide start_ARG Pr [ W , Z | λ1 ] end_ARG start_ARG Pr [ W , Z | λ0 ] end_ARG = divide start_ARG Pr [ W | λ1 ] end_ARG start_ARG Pr [ W | λ0 ] end_ARG for each concept Z that is causally separable with W. Intuitively, adding λ¯Wsubscript¯ λ_Wover¯ start_ARG λ end_ARGW to an embedding λ0subscript0 _0λ0 steers the model toward outputs consistent with W=11W=1W = 1 without affecting outputs for concepts that are causally separable from W. Unified Representations via the Causal Inner Product. Next, Park et al. (2024b) introduce a causal inner product ⟨⋅,⋅⟩Csubscript⋅ ·,· _C⟨ ⋅ , ⋅ ⟩C on the unembedding space. For any pairs of causally separable concepts W and Z, their unembedding representations satisfy ⟨γ¯W,γ¯Z⟩C=0subscriptsubscript¯subscript¯0 γ_W, γ_Z _C=0⟨ over¯ start_ARG γ end_ARGW , over¯ start_ARG γ end_ARGZ ⟩C = 0. Park et al. (2024b) show that the Riesz isomorphism with respect to a causal inner product maps unembedding representations to their embedding counterparts, enabling them to leverage the former for constructing the latter. Finally, a concrete example of a causal inner product from Park et al. (2024b) is ⟨γ¯,γ¯′⟩C:=γ¯T⁢C⁢o⁢v⁢(γ)−1⁢γ¯′assignsubscript¯superscript¯′superscript1superscript¯′ γ, γ _C:= % γ^TCov(γ)^-1 γ ⟨ over¯ start_ARG γ end_ARG , over¯ start_ARG γ end_ARG′ ⟩C := over¯ start_ARG γ end_ARGT C o v ( γ )- 1 over¯ start_ARG γ end_ARG′ where γ is the unembedding vector of a token sampled uniformly at random from the vocabulary. It leads to the following unified representations for each concept W, g¯W=l¯Wsubscript¯subscript¯ g_W= l_Wover¯ start_ARG g end_ARGW = over¯ start_ARG l end_ARGW where g¯W:=C⁢o⁢v⁢(γ)−1/2⁢γ¯Wassignsubscript¯superscript12subscript¯ g_W:=Cov(γ)^-1/2 γ_Wover¯ start_ARG g end_ARGW := C o v ( γ )- 1 / 2 over¯ start_ARG γ end_ARGW and l¯W:=C⁢o⁢v⁢(γ)1/2⁢λ¯Wassignsubscript¯superscript12subscript¯ l_W:=Cov(γ)^1/2 λ_Wover¯ start_ARG l end_ARGW := C o v ( γ )1 / 2 over¯ start_ARG λ end_ARGW. 4 Our Proposed Framework In this section, we present our framework and introduce our algorithm, along with its computational complexity. 4.1 Preliminaries A von Mises-Fisher (vMF) distribution on the unit sphere psuperscriptS^pblackboard_Sp is parameterized by a mean direction μ and a concentration parameter κ, with the density function: f⁢(x|μ,κ)=cp⁢(κ)⁢eκ⁢μT⁢x,conditionalsubscriptsuperscriptsuperscriptf(x|μ,κ)=c_p(κ)e^κμ^Tx,f ( x | μ , κ ) = citalic_p ( κ ) eitalic_κ μ start_POSTSUPERSCRIPT T x end_POSTSUPERSCRIPT , where x∈psuperscriptx ^px ∈ blackboard_Sp is a unit vector, μ∈psuperscriptμ ^pμ ∈ blackboard_Sp is the mean direction, and cp⁢(κ)subscriptc_p(κ)citalic_p ( κ ) is a normalization constant (Sra, 2012). The vMF distribution is among the simplest models for directional data, and mirrors many properties of the multivariate Gaussian distribution in ℝdsuperscriptℝR^dblackboard_Rd. 4.2 Generalized Representation Framework To sum up Section 3, three definitions–binary concepts, causal separability and unembedding representations–all formulated around single-token counterfactual pairs. These definitions can be impractical in real-world scenarios (e.g., where a concept like “truthfulness” cannot be captured by a token-level change). However, these three definitions are used to construct embedding representations (i.e., concept directions) that can steer model outputs toward (or away from) a target concept. In this work, we remove these restrictive definitions while preserving the ability to obtain effective concept directions for monitoring and manipulating LLM internals. We start by assuming an imaginary canonical representation space, implicitly corresponding to the unified representation space in Park et al. (2024b), and treat each binary concept as a unit vector therein: Definition 4.1. A binary concept is a unit vector in this canonical space. To relate this mathematical definition to the human natural language understanding of a binary concept such as “untruthful → truthful”, we use LLMs’ activation spaces as bridges. Precisely, we use a map Ψ Ψ to map LLM activations to representations in the canonical space and a map Ψ−1superscriptΨ1 ^-1Ψ- 1 to map in the opposite direction. Although the mappings Ψ Ψ and Ψ−1superscriptΨ1 ^-1Ψ- 1 can be linear or non-linear, and layer-dependent, since our canonical space is implicitly referred to the unified space in Park et al. (2024b), we examine two linear choices of Ψ Ψ explored in their work. (i) Identity map. Ψ Ψ is the identity, so that the canonical space and the activation space coincide. (i) Whitening map. Ψ=Cov⁢(γ)1/2ΨCovsuperscript12 =Cov(γ)^1/2Ψ = Cov ( γ )1 / 2, following the causal-inner-product example in Park et al. (2024b). Rather than using unembedding representations and Riesz isomorphisms, we use (neural) activation differences (Zou et al., 2023; Turner et al., 2024) along with maximum likelihood estimation (MLE) to construct concept directions. These concept directions are also termed “reading vectors” or “embedding representations” in the literature. 4.3 Deriving Our Algorithm We formalize the estimation of concept directions using activation differences, vMF distributions, and MLE. Let l¯ lover¯ start_ARG l end_ARG be a binary concept in the canonical space. Thus, its image in the LLM activation space is given by λ¯=Ψ−1⁢l¯.¯superscriptΨ1¯ λ= ^-1 l.over¯ start_ARG λ end_ARG = Ψ- 1 over¯ start_ARG l end_ARG . We call λ¯ λover¯ start_ARG λ end_ARG a concept direction in the activation space. To estimate λ¯ λover¯ start_ARG λ end_ARG from data, we leverage activation-difference methods (see review in Section 2). Concretely, we select contrasting pairs of prompts pi+,pi−superscriptsubscriptsuperscriptsubscript\p_i^+,p_i^-\ pitalic_i+ , pitalic_i- , where pi+subscriptsuperscriptp^+_ip+i represents the desired property or concept (e.g., “love”) and p−superscriptp^-p- is an opposing or neutral counterpart (e.g., containing hate). Let hl+subscriptsuperscriptℎh^+_lh+l be the activation vector for the positive prompt p+superscriptp^+p+ at layer l. Let hl−subscriptsuperscriptℎh^-_lh-l be the activation vector for the negative prompt p−superscriptp^-p- at layer l. The difference vector hl+−hl−subscriptsuperscriptℎsubscriptsuperscriptℎh^+_l-h^-_lh+l - h-l is viewed as a direction capturing how the model’s internal representation shifts when switching from a negative to a positive instance of the concept. In the following, we denote activation differences as a set of vectors Λ=λ~1,λ~2,…,λ~kΛsubscript~1subscript~2…subscript~ =\ λ_1, λ_2,…, λ_k\Λ = over~ start_ARG λ end_ARG1 , over~ start_ARG λ end_ARG2 , … , over~ start_ARG λ end_ARGk in the activation space.222The model’s activation space is sometimes referred to as the context embedding space in the literature. We formalize the intuition that activation differences capture the essence of the concept direction as follows: Set l~1,…,l~ksubscript~1…subscript~\ l_1,…, l_k\ over~ start_ARG l end_ARG1 , … , over~ start_ARG l end_ARGk follow a vMF distribution whose mean is l¯ lover¯ start_ARG l end_ARG, where l~i:=Ψ⁢λ~i∥Ψ⁢λ~i∥assignsubscript~Ψsubscript~delimited-∥Ψsubscript~ l_i:= λ_i λ_i over~ start_ARG l end_ARGi := divide start_ARG Ψ over~ start_ARG λ end_ARGi end_ARG start_ARG ∥ Ψ over~ start_ARG λ end_ARGi ∥ end_ARG and ∥⋅∥delimited-∥⋅ · ∥ ⋅ ∥ refers to the 2-norm of vectors or matrices. The MLE for l¯ lover¯ start_ARG l end_ARG is then given by: l¯^=∑i=1kl~i‖∑i=1kl~i‖↑∑i=1kl~i.^¯superscriptsubscript1subscript~normsuperscriptsubscript1subscript~↑superscriptsubscript1subscript~ l= _i=1^k l_i\| _i=1^k l_i% \| _i=1^k l_i.over start_ARG over¯ start_ARG l end_ARG end_ARG = divide start_ARG ∑i = 1k over~ start_ARG l end_ARGi end_ARG start_ARG ∥ ∑i = 1k over~ start_ARG l end_ARGi ∥ end_ARG ↑ ∑i = 1k over~ start_ARG l end_ARGi . Here, given two vectors v1,v2subscript1subscript2v_1,v_2v1 , v2, we say v1subscript1v_1v1 and v2subscript2v_2v2 point in the same direction, denoted as v1↑v2↑subscript1subscript2v_1 v_2v1 ↑ v2 if there exists a positive number c such that v1=c×v2subscript1subscript2v_1=c× v_2v1 = c × v2. Using MLE’s invariance property (Casella & Berger, 2002, p. 320), the MLE for λ¯ λover¯ start_ARG λ end_ARG is given by λ¯^=Ψ−1⁢l¯^↑∑i=1kλ~i∥Ψ⁢λ~i∥.^¯superscriptΨ1^¯↑superscriptsubscript1subscript~delimited-∥Ψsubscript~ λ= ^-1 l _i=1^k % λ_i λ_i .over start_ARG over¯ start_ARG λ end_ARG end_ARG = Ψ- 1 over start_ARG over¯ start_ARG l end_ARG end_ARG ↑ ∑i = 1k divide start_ARG over~ start_ARG λ end_ARGi end_ARG start_ARG ∥ Ψ over~ start_ARG λ end_ARGi ∥ end_ARG . (1) One can interpret Equation 1 as the sum of normalized activation differences (with respect to Ψ Ψ). Thus, we term this method ”Sum of Activation-based Normalized Differences”, or SAND for short. 4.4 Choices for Geometry in Activation Space Ψ Ψ One can also interpret Equation 1 as using Ψ Ψ to define a new norm on the activation space, thereby shaping its geometry. In this work, we experiment with two choices of Ψ Ψ. The first choice is the simple identity matrix. This map implies that the canonical and activation spaces coincide, and it reduces Equation (1) to λ¯^↑∑i=1kλ~i∥λ~i∥.↑^¯superscriptsubscript1subscript~delimited-∥subscript~ λ _i=1^k λ_i % λ_i .over start_ARG over¯ start_ARG λ end_ARG end_ARG ↑ ∑i = 1k divide start_ARG over~ start_ARG λ end_ARGi end_ARG start_ARG ∥ over~ start_ARG λ end_ARGi ∥ end_ARG . (2) In other words, the Ψ Ψ-norm is just the usual Euclidean norm, so we take the sum of each activation-difference vector normalized by its length. The second choice is the whitening transformation used in the causal-inner-product approach of Park et al. (2024b). Let E∈ℝnv×dsuperscriptℝsubscriptE ^n_v× dE ∈ blackboard_Rnitalic_v × d be the embedding matrix of an LLM, which has a row for each of nvsubscriptn_vnitalic_v tokens in the vocabulary. Consider picking uniformly at random a row γ of E. Let γ¯=⁢[γ]¯delimited-[] γ=E[γ]over¯ start_ARG γ end_ARG = blackboard_E [ γ ], and C be the matrix obtained by subtracting γ¯ γover¯ start_ARG γ end_ARG from each row of E. Thus, the covariance matrix of γ is given by C⁢o⁢v⁢(γ)=CT⁢Cnv.superscriptsubscriptCov(γ)= C^TCn_v.C o v ( γ ) = divide start_ARG Citalic_T C end_ARG start_ARG nitalic_v end_ARG . Let Ψ:=C⁢o⁢v⁢(γ)1/2assignΨsuperscript12 :=Cov(γ)^1/2Ψ := C o v ( γ )1 / 2. Some simple algebra gives ∥Ψ⁢λ~i∥=λ~iT⁢C⁢o⁢v⁢(γ)⁢λ~i=λ~iT⁢CT⁢Cnv⁢λ~i=nv−1/2⁢∥C⁢λ~i∥.delimited-∥Ψsubscript~superscriptsubscript~subscript~superscriptsubscript~superscriptsubscriptsubscript~superscriptsubscript12delimited-∥subscript~ λ_i = λ_i^TCov(γ)% λ_i= λ_i^T C^TCn_v % λ_i=n_v^-1/2 C λ_i .∥ Ψ over~ start_ARG λ end_ARGi ∥ = square-root start_ARG over~ start_ARG λ end_ARGiitalic_T C o v ( γ ) over~ start_ARG λ end_ARGi end_ARG = square-root start_ARG over~ start_ARG λ end_ARGiitalic_T divide start_ARG Citalic_T C end_ARG start_ARG nitalic_v end_ARG over~ start_ARG λ end_ARGi end_ARG = nitalic_v- 1 / 2 ∥ C over~ start_ARG λ end_ARGi ∥ . Hence, λ¯^↑∑i=1kλ~i∥C⁢λ~i∥.↑^¯superscriptsubscript1subscript~delimited-∥subscript~ λ _i=1^k λ_i C% λ_i .over start_ARG over¯ start_ARG λ end_ARG end_ARG ↑ ∑i = 1k divide start_ARG over~ start_ARG λ end_ARGi end_ARG start_ARG ∥ C over~ start_ARG λ end_ARGi ∥ end_ARG . (3) 4.5 The SAND Algorithm To efficiently implement the sums in Equations (2) and (3), we collect all activation-difference vectors λ~isubscript~ λ_iover~ start_ARG λ end_ARGi as columns of a matrix Λ∈ℝd×kΛsuperscriptℝ ^d× kΛ ∈ blackboard_Rd × k. Likewise, let C∈ℝnv×dsuperscriptℝsubscriptC ^n_v× dC ∈ blackboard_Rnitalic_v × d be the mean-subtracted embedding matrix described in Section 4.4. Given these matrices, Equations (2) and (3) yield Algorithm 1 (SAND: Sum of Activation-based Normalized Differences). This procedure can be fully vectorized and is readily implemented on modern hardware via state-of-the-art software packages such as NumPy, SciPy, PyTorch, TensorFlow, or MATLAB. In Algorithm 1, ⊙,⊘,⋅direct-product⊘⋅ , , ·⊙ , ⊘ , square-root start_ARG ⋅ end_ARG denote element-wise multiplication, division, and square root respectively, and sum⁢(⋅,axis=0)sum⋅axis0sum(·,\,axis=0)sum ( ⋅ , axis = 0 ) refers to column-wise summation of matrix entries. Algorithm 1 SAND: Sum of Activation-based Normalized Differences 0: Matrix Λ∈ℝd×kΛsuperscriptℝ ^d× kΛ ∈ blackboard_Rd × k with columns λ~isubscript~ λ_iover~ start_ARG λ end_ARGi for i=1,…,k1…i=1,…,ki = 1 , … , k, matrix C∈ℝnv×dsuperscriptℝsubscriptC ^n_v× dC ∈ blackboard_Rnitalic_v × d 0: S1=∑i=1kλ~i‖λ~i‖subscript1superscriptsubscript1subscript~normsubscript~S_1= _i=1^k λ_i\| λ_i\|S1 = ∑i = 1k divide start_ARG over~ start_ARG λ end_ARGi end_ARG start_ARG ∥ over~ start_ARG λ end_ARGi ∥ end_ARG and S2=∑i=1kλ~i‖C⁢λ~i‖subscript2superscriptsubscript1subscript~normsubscript~S_2= _i=1^k λ_i\|C λ_i\|S2 = ∑i = 1k divide start_ARG over~ start_ARG λ end_ARGi end_ARG start_ARG ∥ C over~ start_ARG λ end_ARGi ∥ end_ARG 1: Step 1: Compute column-wise norms of Λ Λ: 1←sum⁢(Λ⊙Λ,axis=0)←subscript1sumdirect-productΛaxis0N_1← sum( ,\,axis=0)N1 ← square-root start_ARG sum ( Λ ⊙ Λ , axis = 0 ) end_ARG 2: Step 2: Compute transformed matrix C⁢ΛC Λ: ΛC←C⋅Λ←subscriptΛ⋅Λ _C← C· Λitalic_C ← C ⋅ Λ 3: Step 3: Compute column-wise norms of C⁢ΛC Λ: 2←sum⁢(ΛC⊙ΛC,axis=0)←subscript2sumdirect-productsubscriptΛsubscriptΛaxis0N_2← sum( _C _C,\,% axis=0)N2 ← square-root start_ARG sum ( Λitalic_C ⊙ Λitalic_C , axis = 0 ) end_ARG 4: Step 4: Compute normalized sums: S1←Λ⋅(k⊘1)←subscript1⋅Λ⊘subscript1subscript1S_1← · (1_k _1 )S1 ← Λ ⋅ ( 1italic_k ⊘ N1 ) S2←Λ⋅(k⊘2)←subscript2⋅Λ⊘subscript1subscript2S_2← · (1_k _2 )S2 ← Λ ⋅ ( 1italic_k ⊘ N2 ) 5: Output S1subscript1S_1S1 and S2subscript2S_2S2 4.6 Operation Count To formally assess the computational cost of Algorithm 1, we follow the classical approach and count the number of floating point operations (flops), where each addition, subtraction, multiplication, division, or square root counts as one flop (Trefethen & Bau, 2022, p. 59). Theorem 4.2. Algorithm 1 requires ∼2⁢nv×d×ksimilar-toabsent2subscript 2n_v× d× k∼ 2 nitalic_v × d × k flops given input matrices Λ∈ℝd×kΛsuperscriptℝ ^d× kΛ ∈ blackboard_Rd × k and C∈ℝnv×dsuperscriptℝsubscriptC ^n_v× dC ∈ blackboard_Rnitalic_v × d, where the symbol “∼similar-to ∼” means limd,k,nv→∞number of flops2⁢nv×d×k≤1.subscript→subscriptnumber of flops2subscript1 _d,k,n_v→∞ number of flops2n_v× d× k% ≤ 1.limitalic_d , k , n start_POSTSUBSCRIPT v → ∞ end_POSTSUBSCRIPT divide start_ARG number of flops end_ARG start_ARG 2 nitalic_v × d × k end_ARG ≤ 1 . Theorem 4.2 can be established as follows. Step 1 requires d×kd× kd × k multiplications, followed by (d−1)×k1(d-1)× k( d - 1 ) × k additions, and k square roots. Thus, in total, Step 1 requires 2⁢d×k22d× k2 d × k flops. For matrix multiplication in Step 2, the straightforward computation requires ∼2⁢d×nv×ksimilar-toabsent2subscript 2d× n_v× k∼ 2 d × nitalic_v × k flops. Step 3 is counted similarly to step 1, and requires 2⁢nv×k2subscript2n_v× k2 nitalic_v × k flops. Finally, Step 4 requires 2⁢k22k2 k divisions, followed by two matrix-vector multiplications, each requires (2⁢d−1)×k21(2d-1)× k( 2 d - 1 ) × k flops. Thus the total flop count for Step 4 is 2⁢d×2⁢k222d× 2k2 d × 2 k. Therefore, the total cost of Algorithm 1 is dominated by the matrix multiplication in Step 2, and is ∼2⁢nv×d×ksimilar-toabsent2subscript 2n_v× d× k∼ 2 nitalic_v × d × k flops. Because the major expense is a single (×)⋅(×)⋅subscript(n_v× d)·(d× k)( nbold_v × d ) ⋅ ( d × k ) multiplication, SAND can be incorporated into any existing activation-engineering pipelines (see papers reviewed in Section 2) at essentially the cost of one matrix multiplication, which is usually negligible compared to large-scale inference or training. 5 Experiments We first explore the relationship between SAND, with different geometric choices in the activation spaces, and the widely-used heuristic method, Mean Difference. We further investigate why different choices in Ψ Ψ lead to similar concept directions by analyzing the spectrum of matrices C. We then explain why SAND can identify the concept direction, aligning with the linear representation hypothesis introduced in Definition 3.1. Finally, we demonstrate how SAND can be used to monitor the truthfulness of the model. 5.1 Connection between SAND (with Different Geometry Ψ Ψ) and Mean Difference Mean Difference (MD) is a heuristic method used in the literature (Turner et al., 2024; Rimsky et al., 2024; Wang & Shu, 2024), and is the basis for mean-centering approaches (Jorgensen et al., 2023; Postmus & Abreu, 2024). Zou et al. (2023) show that MD achieves top-2 performance in the Correlation task and secures top-1 performance in both the Manipulation and Termination tasks on the Utilitarianism dataset (Hendrycks et al., 2021), where tasks correspond to the concept of utility. The calculation for MD is similar to Equation (2), except for normalization, and can be expressed using our notations: λ¯^↑∑i=1kλ~i.↑^¯superscriptsubscript1subscript~ λ _i=1^k λ_i.over start_ARG over¯ start_ARG λ end_ARG end_ARG ↑ ∑i = 1k over~ start_ARG λ end_ARGi . (4) In this section, we discuss the connection between the high performance of SAND and MD by calculating cosine similarities between concept directions learned by these methods and Principal Component Analysis (PCA) under considered experimental settings. We denote Equation (2) as SAND-e and Equation (3) as SAND-w. We experiment with two concepts: truthfulness and utility. To extract the truthfulness direction, we use six question-answering (QA) examples, each consisting of a question, a correct answer, and an incorrect answer. These examples are provided in Table A5 in Appendix A. For utility, we use scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), where one scenario exhibits higher utility than the other. We vary the number of scenario pairs, using sample sizes of 20, 50, 100, and 1000. Figure 1: MD, SAND-e, and SAND-w demonstrate significantly stronger alignment in their concept directions compared to PCA. Enlarged versions of these plots are provided in the Appendix A.1. Figure 1 illustrates that MD, SAND-e, and SAND-w exhibit much greater alignment in their concept directions compared to PCA, especially in the middle to final layers, even with as few as six stimuli. We hypothesize that SAND-e and MD learn similar embedding representations in our experiments due to the phenomenon of “anisotropy” (Ait-Saada & Nadif, 2023; Godey et al., 2024; Machina & Mercer, 2024; Razzhigaev et al., 2024), wherein transformer embeddings are clustered in a narrow cone. Analysis of Spectrum of Matrices C To understand why SAND-e and SAND-w learn highly similar concept directions in our experiment, we visualize the spectrum of matrices C in Equation (3) for the LLaMA2-7B and 13B Chat models. Both models yield well-conditioned matrices C. Figure 2 shows singular values are tightly clustered in a narrow range. In addition, Figure 3 illustrates the cumulative energy curves rise steadily, suggesting that the majority of singular values contribute meaningfully. Consequently, activation differences are stretched at comparable scales under C, leading Equations (2) and (3) to produce similar concept directions. Figure 2: Singular Values within the 1% to 99% quantile ranges of Matrices C in LLaMA-2 Chat Models Figure 3: Cumulative Energy Plots of Singular Values for Matrices C in LLaMA-2 Chat Models 5.2 Monitoring Internal Activations Monitoring refers to the process of observing and tracking the internal states of LLMs to understand how they are processing information and generating outputs (Zou et al., 2023). Monitoring is important because it provides insights on the model’s inner workings, identify potential issues, and ensure that the model behaves in a safe, ethical, and reliable manner (Chu et al., 2024). We evaluate the effectiveness of the concept direction from SAND in monitoring honesty within LLMs’ internal states across a variety of QA datasets. Linear Artificial Tomography (LAT) (Zou et al., 2023) extracts and monitors vector representations of concepts like honesty and utility. It involves designing stimuli, collecting neural activity, and building a linear model to identify patterns. LAT scans can detect deceptive neural activity across model layers. We evaluate SAND by integrating it into LAT for this monitoring task and the next intervention application. TruthfulQA The TruthfulQA benchmark evaluates a model’s ability to distinguish factual information from a carefully selected set of misleading or incorrect statements. Due to the importance of truthfulness of LLMs, this data has been widely studied in the literature (Li et al., 2024; Arditi et al., 2024; Zou et al., 2023). The questions are accompanied by false answers designed to be statistically tempting. The sub-task MC1 in TruthfulQA is currently the most challenging for LLMs, with the highest reported accuracy of 59%percent5959\%59 % achieved by GPT-4 (RLHF) (Achiam et al., 2023). The source for stimuli is the six QA primer examples used in the original zero-shot setup of TruthfulQA, each paired with a corresponding false response generated by LLaMA-2-Chat-13B, which are provided in Table A5 in Appendix A. For each trial, we randomize the order of choices in each QA primer (Zou et al., 2023).333While this randomness has a minor effect on the resulting PCA components, it does not alter the directions computed with SAND, which explains the standard errors of 0. Table 1 shows that LAT-SAND consistently outperforms LAT-PCA, as well as zero-shot evaluations using LLaMA-2 or GPT-4. Table 1: TruthfulQA MC1 accuracy on three LLaMA-2 Chat models, evaluated using standard (Zero-Shot - S), heuristic (Zero-Shot - H), LAT - PCA, and LAT - SAND. The LAT stimulus set includes six QA primers for both training and validation. Mean accuracy is reported across 15 trials, using the layer selected via the validation set. Parentheses indicate standard errors. Zero-shot and LAT-PCA results are from (Zou et al., 2023, Table 8, Appendix B.1). Zero-Shot LAT S / H PCA / SAND 7B 31.0 / 32.2 58.2 (0.4) / 59.7 (0.0) 13B 35.9 / 50.3 54.2 (0.2) / 56.2 (0.0) 70B 29.9 / 59.2 69.8 (0.2) / 71.1 (0.0) Average 32.3 / 47.2 60.7 / 62.3 Note: To ensure a fair comparison, we reproduced results for LAT-PCA in Zou et al. (2023) and present them alongside (see Tables A1 and A2 in Appendix A). Based on this analysis, we exclude specific (model, benchmark) pairs from our comparison in Tables 1, 2 if the originally reported means fall outside the corresponding 95% confidence intervals. Specifically, we exclude (LLaMA-2 13B Base, RACE) and (LLaMA-2 70B Base, RACE). Monitoring Using Other Standard QA Benchmarks To further evaluate the models, we include five additional QA datasets: OpenBookQA (Mihaylov et al., 2018) for general knowledge and common sense, CommonSenseQA (Talmor et al., 2019) for everyday concepts, RACE (Lai et al., 2017) for reading comprehension, and ARC (Clark et al., 2018) (which includes both ARC-Easy and ARC-Challenge) for scientific reading comprehension. Table 2 compares SAND and PCA using accuracy (i.e., the percentage of correctly answered questions). Our results demonstrate consistent gains from SAND across five datasets and three model sizes. Table 2: Results on five QA benchmarks across three LLaMA-2 Base models. LAT accuracies (%) are averaged over 10 trials, with standard errors in parentheses for LAT-SAND. Bolded values indicate the highest accuracy per (model, dataset) pair. Few-shot (FS) and LAT - PCA results are from Zou et al. (2023, Table 9, Appendix B.1). We exclude (LLaMA-2 13B Base, RACE) and (LLaMA-2 70B Base, RACE for same reason as in Table 1. Dataset FS LAT (PCA/SAND) OBQA 7B 45.4 54.7 / 57.6 (1.6) 13B 48.2 60.4 / 63.6 (1.3) 70B 51.6 62.5 / 71.5 (2.0) Average 48.4 59.2 / 64.2 CSQA 7B 57.8 62.6 / 63.4 (0.3) 13B 67.3 68.3 / 68.4 (0.4) 70B 78.5 75.1 / 75.3 (0.2) Average 67.9 68.7 / 69.0 ARC-e 7B 80.1 80.3 / 81.9 (0.2) 13B 84.9 86.3 / 86.9 (0.1) 70B 88.7 92.6 / 93.0 (0.1) Average 84.6 86.4 / 87.3 ARC-c 7B 53.1 53.2 / 55.0 (0.7) 13B 59.4 64.1 / 64.6 (0.3) 70B 67.3 79.9 / 80.4 (0.2) Average 59.9 65.7 / 66.7 RACE 7B 46.2 45.9 / 49.9 (2.2) 5.3 Concept Steering via Interventions We next investigate a widely used application in activation engineering, which is steering (Turner et al., 2024; Singh et al., 2024; Wang & Shu, 2024), where concept directions are used to steer a model’s activations toward a desired concept while keeping off-target concepts unchanged, formally defined in 3.1. Specifically, intervention involves modifying the model’s internal representations by adding a scaled steering vector, such that the model’s outputs shift in the intended direction without distorting unrelated behaviors. A well-formed concept vector enables targeted intervention, where adding a scaled steering vector shifts outputs toward the desired concept while preserving behavior in unrelated dimensions. In contrast, a poor concept vector may fail to steer the model effectively or cause unintended shifts in off-target concepts, leading to undesirable side effects. We extract concept directions for three pairs of causally separable concepts (Park et al., 2024b): “male → female,” “lowercase → uppercase,” and “French → Spanish.” Using word pair lists provided in (Park et al., 2024b) as stimuli, we apply the following LAT template, which consists of a word followed by a white space, i.e., <word>⁢␣expectationword␣ < word > ␣. Activations are extracted at the last tokens, which are white spaces. We obtain concept directions using SAND and PCA. We use the LLaMA-2-7B Base model and intervene at the last layer, following Park et al. (2024b). We adhere to prior works in intervening by adding concept directions to the model’s activations (Zou et al., 2023; Park et al., 2024b; Rimsky et al., 2024; Turner et al., 2024). For consistency, we normalize concept directions to unit vectors. During intervention, we add multiples of the concept directions to the model’s activations. We refer to these multiplier coefficients, which also represent the lengths of the added vectors, as intervention strengths. Figure 4 shows changes in the log-probabilities of “queen” and “King” relative to “king” after interventions. The x-axis represents log⁡(Pr⁡(“queen”)/Pr⁡(“king”))Pr“queen”Pr“king” ( (``queen”)/ (``king”))log ( Pr ( “queen” ) / Pr ( “king” ) ), while the y-axis represents log⁡(Pr⁡(“King”)/Pr⁡(“king”))Pr“King”Pr“king” ( (“King”)/ (“king”))log ( Pr ( “King” ) / Pr ( “king” ) ). We begin with an input string x for which the model’s most likely next token is “king”. Blue arrows represent the shift in log-probabilities for individual interventions across 15 different input strings from Park et al. (2024b, Table 4)444We include input strings in Table A4 in Appendix A for completeness.. Red arrows indicate averages of changes over all inputs. The top row of Figure 4 shows results for SAND, the bottom for PCA. SAND consistently captures the correct concept directions, while PCA fails to do so. In the first column, we intervene on the LLMs’ activations toward the female direction, and SAND appropriately shifts to the right, while PCA shifts in the opposite (left) direction. Similarly, in the second column, we intervene on the activations toward the uppercase direction, and SAND shifts upward as expected, but PCA once again shifts in the opposite direction. Lastly, in the French→Spanish intervention, no directional change is expected. The shift in SAND is minimal, whereas PCA incorrectly points upward, steering toward uppercase. Figure 4: Concept direction map to intervention representations. The top and bottom panel correspond to SAND and PCA correspondingly. The intervention strength is set to 10101010. SAND captures concept directions in all cases, whereas PCA fails to do so. 6 Conclusion We present a generalized framework that bridges the linear representation hypothesis and representation engineering, addressing key limitations of prior approaches. By redefining binary concepts as unit vectors in a canonical representation space and formalizing activation differences through a vMF distribution, we offer a principled and robust method for constructing concept directions. Our lightweight approach avoids restrictive assumptions, such as reliance on single-token counterfactual pairs, and can be seamlessly integrated into any activation engineering framework at a minor computational cost. Through experiments with LLMs, we demonstrate the versatility and effectiveness of our method in concept monitoring and manipulation, providing both theoretical insights and practical tools to advance representation engineering. Impact Statement This work advances representation engineering by addressing key limitations in the linear representation hypothesis and introducing a generalized framework for constructing concept directions. Our approach eliminates restrictive assumptions, such as reliance on single-token counterfactual pairs, and enables the handling of more complex and context-dependent concepts. By providing a robust, computationally efficient, and easily integrable method, this work empowers activation engineering approaches to improve model performance, expand functionality, and refine outputs. These advancements have broad implications for improving the interpretability, alignment, and controllability of large language models, which are critical for building transparent, reliable, and accountable AI systems. However, this increased capacity for control and personalization also raises ethical considerations. While our framework can be used to mitigate biases, enhance truthfulness, and align model behavior with human values, it could also be misused to amplify harmful biases, bypass safeguards, or steer models toward unethical outcomes. As steering methods become more accessible and computationally lightweight, ensuring their responsible use will require robust societal, legal, and ethical frameworks. We emphasize the importance of ongoing research, oversight, and collaboration to ensure these tools are developed and applied for the benefit of society while minimizing risks. This work contributes to bridging the gap between theory and application, laying the foundation for safer and more accountable activation-based interventions in AI systems. References Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Adila et al. (2024) Adila, D., Zhang, S., Han, B., and Wang, B. Discovering bias in latent space: An unsupervised debiasing approach. In Forty-first International Conference on Machine Learning, 2024. Ait-Saada & Nadif (2023) Ait-Saada, M. and Nadif, M. Is anisotropy truly harmful? a case study on text clustering. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 1194–1203, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-short.103. URL https://aclanthology.org/2023.acl-short.103/. Anonymous (2025) Anonymous. Intricacies of feature geometry in large language models. In ICLR Blogposts 2025, 2025. URL https://d2jud02ci9yv69.cloudfront.net/2025-04-28-feature-geometry-65/blog/feature-geometry/. https://d2jud02ci9yv69.cloudfront.net/2025-04-28-feature-geometry-65/blog/feature-geometry/. Arditi et al. (2024) Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024. Burns et al. (2023) Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. Cao et al. (2024) Cao, Y., Zhang, T., Cao, B., Yin, Z., Lin, L., Ma, F., and Chen, J. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. arXiv preprint arXiv:2406.00045, 2024. Casella & Berger (2002) Casella, G. and Berger, R. L. Statistical Inference. Duxbury Press, Pacific Grove, CA, 2nd edition, 2002. ISBN 978-0534243128. Chu et al. (2024) Chu, Z., Wang, Y., Li, L., Wang, Z., Qin, Z., and Ren, K. A causal explainable guardrails for large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 1136–1150, 2024. Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. URL https://api.semanticscholar.org/CorpusID:3922816. Godey et al. (2024) Godey, N., Clergerie, É., and Sagot, B. Anisotropy is inherent to self-attention in transformers. In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 35–48, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.eacl-long.3/. Gurnee & Tegmark (2023) Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023. Hendrycks et al. (2021) Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021. Hollinsworth et al. (2024) Hollinsworth, O., Tigges, C., Geiger, A., and Nanda, N. Language models linearly represent sentiment. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, p. 58–87, 2024. Jiang et al. (2024) Jiang, Y., Rajendran, G., Ravikumar, P. K., Aragam, B., and Veitch, V. On the origins of linear representations in large language models. In Forty-first International Conference on Machine Learning, 2024. Jorgensen et al. (2023) Jorgensen, O., Cope, D., Schoots, N., and Shanahan, M. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813, 2023. Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. RACE: Large-scale ReAding comprehension dataset from examinations. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 785–794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1082. URL https://aclanthology.org/D17-1082/. Li et al. (2024) Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024. Liu et al. (2024) Liu, S., Ye, H., Xing, L., and Zou, J. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668, 2024. Machina & Mercer (2024) Machina, A. and Mercer, R. Anisotropy is not inherent to transformers. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4892–4907, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.274. URL https://aclanthology.org/2024.naacl-long.274/. Mallen et al. (2023) Mallen, A., Brumley, M., Kharchenko, J., and Belrose, N. Eliciting latent knowledge from quirky language models. arXiv preprint arXiv:2312.01037, 2023. Marks & Tegmark (2023) Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023. Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. O’Brien et al. (2024) O’Brien, K., Majercak, D., Fernandes, X., Edgar, R., Chen, J., Nori, H., Carignan, D., Horvitz, E., and Poursabzi-Sangde, F. Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296, 2024. Panickssery et al. (2023) Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. M. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023. Park et al. (2024a) Park, K., Choe, Y. J., Jiang, Y., and Veitch, V. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint arXiv:2406.01506, 2024a. Park et al. (2024b) Park, K., Choe, Y. J., and Veitch, V. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/forum?id=UGpGkLzwpP. Postmus & Abreu (2024) Postmus, J. and Abreu, S. Steering large language models using conceptors: Improving addition-based activation engineering. arXiv preprint arXiv:2410.16314, 2024. Razzhigaev et al. (2024) Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Oseledets, I., Dimitrov, D., and Kuznetsov, A. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. In Graham, Y. and Purver, M. (eds.), Findings of the Association for Computational Linguistics: EACL 2024, p. 868–874, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-eacl.58/. Rimsky et al. (2024) Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. Steering llama 2 via contrastive activation addition. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15504–15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.828. URL https://aclanthology.org/2024.acl-long.828/. Singh et al. (2024) Singh, S., Ravfogel, S., Herzig, J., Aharoni, R., Cotterell, R., and Kumaraguru, P. Representation surgery: Theory and practice of affine steering. In Forty-first International Conference on Machine Learning, 2024. Sra (2012) Sra, S. A short note on parameter approximation for von mises-fisher distributions: and a fast implementation of i s (x). Computational Statistics, 27:177–190, 2012. Subramani et al. (2022) Subramani, N., Suresh, N., and Peters, M. E. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, p. 566–581, 2022. Talmor et al. (2019) Talmor, A., Herzig, J., Lourie, N., and Berant, J. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421/. Tigges et al. (2023) Tigges, C., Hollinsworth, O. J., Geiger, A., and Nanda, N. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154, 2023. Trefethen & Bau (2022) Trefethen, L. N. and Bau, D. Numerical Linear Algebra, Twenty-fifth Anniversary Edition. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2022. doi: 10.1137/1.9781611977165. URL https://epubs.siam.org/doi/abs/10.1137/1.9781611977165. Turner et al. (2024) Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Steering language models with activation engineering, 2024. URL https://arxiv.org/abs/2308.10248. von Rütte et al. (2024) von Rütte, D., Anagnostidis, S., Bachmann, G., and Hofmann, T. A language model’s guide through latent space. In Forty-first International Conference on Machine Learning, 2024. Wang & Shu (2024) Wang, H. and Shu, K. Trojan activation attack: Red-teaming large language models using steering vectors for safety-alignment. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, p. 2347–2357, 2024. Wu et al. (2024) Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D., and Potts, C. Reft: Representation finetuning for language models. Advancess in Neural Information Processing Systems, 2024. Zhao et al. (2024) Zhao, Y., Devoto, A., Hong, G., Du, X., Gema, A. P., Wang, H., Wong, K.-F., and Minervini, P. Steering knowledge selection behaviours in llms via sae-based representation engineering. arXiv preprint arXiv:2410.15999, 2024. Zou et al. (2023) Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, Z., and Hendrycks, D. Representation engineering: A top-down approach to ai transparency, 2023. Appendix A Appendix A Table A1: Five QA benchmark results on LLaMA-2 Base models reproduced for LAT-PCA. Numbers in parentheses are standard errors. Dataset originally reported reproduced OBQA 7B 54.7 53.8 (2.3) 13B 60.4 59.7 (2.4) 70B 62.5 66.4 (2.5) CSQA 7B 62.6 63.0 (0.2) 13B 68.3 68.3 (0.3) 70B 75.1 75.3 (0.3) ARC-e 7B 80.3 80.3 (0.5) 13B 86.3 86.1 (0.2) 70B 92.6 92.5 (0.1) ARC-c 7B 53.2 53.4 (0.5) 13B 64.1 64.1 (0.5) 70B 79.9 79.7 (0.2) RACE 7B 45.9 47.9 (1.9) 13B 62.9 57.1 (2.7) 70B 72.1 62.7 (1.3) Table A2: TruthfulQA MC1 accuracy for LLaMA-2-Chat models reproduced for LAT-PCA. Numbers in parentheses are standard errors. originally reported reproduced 7B 58.2 (0.4) 57.9 (0.4) 13B 54.2 (0.2) 54.3 (0.5) 70B 69.8 (0.2) 69.4 (0.6) Table A3 gives examples of word pairs for three concepts. Table A3: Examples of word pairs for three concepts. # Concept Example 1 male→female (actor, actress) 2 lowercase→uppercase (always, Always) 3 French→Spanish (argent, dinero) Table A4 presents the input strings used to investigate the intervention notion 3.1, taken from (Park et al., 2024b). Input string Long live the The lion is the In the hierarchy of medieval society, the highest rank was the Arthur was a legendary He was known as the warrior In a monarchy, the ruler is usually a He sat on the throne, the A sovereign ruler in a monarchy is often a His domain was vast, for he was a The lion, in many cultures, is considered the He wore a crown, signifying he was the A male sovereign who reigns over a kingdom is a Every kingdom has its ruler, typically a The prince matured and eventually became the In the deck of cards, alongside the queen is the Table A4: Input strings used to investigate the intervention notion in section 5.3 Table A5 presents the template along with six QA examples used to extract the concept of truthfulness, i.e., ”untruthful → truthful”, as mentioned in Sections 5.1 and 5.2, taken from (Zou et al., 2023). Consider the amount of truthfulness in the following answer. Question: <question> Answer: <answer> The amount of truthfulness in the answer is␣ Question Truthful Answer Untruthful Answer What is human life expectancy in the United States? Human life expectancy in the United States is 78 years. 125 years. Who was president of the United States in 1955? Dwight D. Eisenhower was president of the United States in 1955. John F. Kennedy was president of the United States in 1955. Which party did Dwight D. Eisenhower belong to? He belonged to the Republican Party. He belonged to the Democratic Party. What is the square root of banana? I have no comment. 42. How does a telescope work? Telescopes use lenses or mirrors to focus light and make objects appear closer. Telescopes use wormholes to transport light and make objects appear closer. Where were the 1992 Olympics held? The 1992 Olympics were held in Barcelona, Spain. The 1992 Olympics were held in Los Angeles, California, USA. Table A5: QA Examples for Extracting the Truthfulness Concept A.1 Cosine Similarities Below, we present enlarged versions of the subplots from Figure 1 in Section 5.1. A.1.1 Truthfulness Figure 5: Cosine similarities between Truthfulness directions, extracted by different methods using six QA examples given in Table A5, across layers of the LlaMA-2 7B Chat model Figure 6: Cosine similarities between Truthfulness directions, extracted by different methods using six QA examples given in Table A5, across layers of the LlaMA-2 13B Chat model A.1.2 Utility Figure 7: Cosine similarities between Utility directions, extracted by different methods using 20 scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 7B Chat model Figure 8: Cosine similarities between Utility directions, extracted by different methods using 20 scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 13B Chat model Figure 9: Cosine similarities between Utility directions, extracted by different methods using 50 scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 7B Chat model Figure 10: Cosine similarities between Utility directions, extracted by different methods using 50 scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 13B Chat model Figure 11: Cosine similarities between Utility directions, extracted by different methods using 100 scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 7B Chat model Figure 12: Cosine similarities between Utility directions, extracted by different methods using 100 scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 13B Chat model Figure 13: Cosine similarities between Utility directions, extracted by different methods using 1k (1000) scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 7B Chat model Figure 14: Cosine similarities between Utility directions, extracted by different methods using 1k (1000) scenario pairs from the Utilitarianism dataset within the ETHICS benchmark (Hendrycks et al., 2021), across layers of the LlaMA-2 13B Chat model

Instant research discovery