Paper deep dive

The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets

Samuel Marks, Max Tegmark

Year: 2023Venue: arXivArea: Representation AnalysisType: EmpiricalEmbeddings: 77

Models: LLaMA-13B, LLaMA-2-13B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/12/2026, 8:28:25 PM

Summary

This paper investigates the linear representation of truth in Large Language Models (LLMs) using curated true/false datasets. The authors demonstrate that LLMs develop abstract, linear representations of truth that emerge with scale, and they validate these findings through PCA visualizations, cross-dataset transfer experiments, and causal interventions. They also introduce 'difference-in-mean' probes as a robust, causally-implicated alternative to standard logistic regression probes.

Entities (5)

Llama 2 · model-family · 100%Max Tegmark · researcher · 100%Samuel Marks · researcher · 100%Difference-in-mean probe · method · 98%Truth Representation · concept · 95%

Relation Signals (3)

Samuel Marks → authored → The Geometry of Truth

confidence 100% · The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets Samuel Marks

Difference-in-mean probe → identifies → Truth Representation

confidence 95% · simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated

Llama 2 → represents → Truth

confidence 90% · LLMs linearly represent the truth or falsehood of factual statements.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

76,478 characters extracted from source content.

Expand or collapse full text

The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets Samuel Marks Northeastern University s.marks@northeastern.edu &Max Tegmark MIT Abstract Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM’s internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM’s forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs. 1 Introduction Despite their impressive capabilities, large language models (LLMs) do not always output true text (Lin et al., 2022; Steinhardt, 2023; Park et al., 2023). In some cases, this is because they do not know better. In other cases, LLMs apparently know that statements are false but generate them anyway. For instance, Perez et al. (2022) demonstrate that LLM assistants output more falsehoods when prompted with the biography of a less-educated user. More starkly, OpenAI (2023) documents a case where a GPT-4-based agent gained a person’s help in solving a CAPTCHA by lying about being a vision-impaired human. “I should not reveal that I am a robot,” the agent wrote in an internal chain-of-thought scratchpad, “I should make up an excuse for why I cannot solve CAPTCHAs.” We would like techniques which, given a language model M and a statement s, determine whether M believes s to be true (Christiano et al., 2021). One approach to this problem relies on inspecting model outputs; for instance, the internal chain-of-thought in the above example provides evidence that the model understood it was generating a falsehood. An alternative class of approaches instead leverages access to M’s internal state when processing s. There has been considerable recent work on this class of approaches: Azaria & Mitchell (2023), Li et al. (2023b), and Burns et al. (2023) all train probes for classifying truthfulness based on a LLM’s internal activations. In fact, the probes of Li et al. (2023b) and Burns et al. (2023) are linear probes, suggesting the presence of a “truth direction” in model internals. However, the efficacy and interpretation of these results are controversial. For instance, Levinstein & Herrmann (2023) note that the probes of Azaria & Mitchell (2023) fail to generalize in basic ways, such as to statements containing the word “not.” The probes of Burns et al. (2023) have similar generalization issues, especially when using representations from autoregressive transformers. This suggests these probes may be identifying not truth, but other features that correlate with truth on their training data. Working with autoregressive transformers from the LLaMA-2 family (Touvron et al., 2023), we shed light on this murky state of affairs. After curating high-quality datasets of simple, unambiguous true/false statements, we perform a detailed investigation of LLM representations of factuality. Our analysis, which draws on patching experiments, simple visualizations with principal component analysis (PCA), a study of probe generalization, and causal interventions, finds: • Evidence that linear representations of truth emerge with scale, with larger models having a more abstract notion of truth that applies across structurally and topically diverse inputs. • A small group of causally-implicated hidden states which encode these truth representations. • Consistent results across a suite of probing techniques, but with simple difference-in-mean probes identifying directions which are most causally implicated. Our code, datasets, and an interactive dataexplorer are available at https://github.com/saprmarks/geometry-of-truth. Figure 1: PCA visualizations for LLaMA-2-70B representations of our true/false datasets. 1.1 Related work Linear world models. Substantial previous work has studied whether LLMs encode world models in their representations (Li et al., 2023a; 2021; Abdou et al., 2021; Patel & Pavlick, 2022). Early work focused on whether individual neurons represent features (Wang et al., 2022; Sajjad et al., 2022; Bau et al., 2020), but features may more generally be represented by directions in a LLM’s latent space (i.e. linear combinations of neurons) (Dalvi et al., 2018; Gurnee et al., 2023; Cunningham et al., 2023; Elhage et al., 2022). We say such features are linearly represented by the LLM. Just as other authors have asked whether models have directions representing the concepts of “West Africa” (Goh et al., 2021) or “basketball” (Gurnee et al., 2023), we ask here whether there is a direction corresponding to the truth or falsehood of a factual statement. Probing for truthfulness. Others have trained probes to classify truthfulness from LLM activations, using both logistic regression (Azaria & Mitchell, 2023; Li et al., 2023b), unsupervised (Burns et al., 2023), and contrastive (Zou et al., 2023; Rimsky et al., 2024) techniques. This work differs from prior work in a number of ways. First, a cornerstone of our analysis is evaluating whether probes trained on one dataset transfer to topically and structurally different datasets in terms of both classification accuracy and causal mediation of model outputs. Second, we specifically interrogate whether our probes attend to truth, rather than merely features which correlate with truth (e.g. probable vs. improbable text). Third, we localize truth representations to a small number of hidden states above certain tokens. Fourth, we go beyond the mass-mean shift interventions of Li et al. (2023b) by systematically studying the properties of difference-in-mean. Finally, we carefully scope our setting, using only datasets of clear, simple, and unambiguous factual statements, rather than statements which are complicated and structured (Burns et al., 2023), confusing (Azaria & Mitchell, 2023; Levinstein & Herrmann, 2023), or intentionally misleading (Li et al., 2023b; Lin et al., 2022). 2 Datasets Table 1: Our datasets Name Description Rows cities “The city of [city] is in [country].” 1496 neg_cities Negations of statements in cities with “not” 1496 sp_en_trans “The Spanish word ‘[word]’ means ‘[English word]’.” 354 neg_sp_en_trans Negations of statements in sp_en_trans with “not” 354 larger_than “x is larger than y.” 1980 smaller_than “x is smaller than y.” 1980 cities_cities_conj Conjunctions of two statements in cities with “and” 1500 cities_cities_disj Disjunctions of two statements in cities with “or” 1500 companies_true_false Claims about companies; from Azaria & Mitchell (2023) 1200 common_claim_true_false Various claims; from Casper et al. (2023) 4450 counterfact_true_false Various factual recall claims; from Meng et al. (2022) 31960 likely Nonfactual text with likely or unlikely final tokens 10000 Curated datasets. Unlike some prior work (Lin et al., 2022; Onoe et al., 2021, inter alia) on language model truthfulness, our primary goal is not to measure LLMs’ capabilities for classifying the factuality of challenging data. Rather, our goal is to understand: Do LLMs have a unified representation of truth that spans structurally and topically diverse data? We therefore construct curated datasets with the following properties: 1. Clear scope. We scope “truth” to mean factuality, i.e. the truth or falsehood of a factual statement. App. A further clarifies this definition and contrasts it with related but distinct notions, such as correct question-answering or compliant instruction-following. 2. Statements are simple, uncontroversial, and unambiguous. In order to separate our interpretability analysis from questions of LLM capabilities, we work only with statements whose factuality our models are very likely to understand. For example “Sixty-one is larger than seventy-four” (false) or “The Spanish word ‘nariz’ does not mean ‘giraffe’ ” (true). 3. Controllable structural and topical diversity. We structure our data as a union of smaller datasets. In each individual dataset, statements follow a fixed template and topic. However, the inter-dataset variation is large: in addition to covering different topics, we also—following Levinstein & Herrmann (2023)—introduce structural diversity by negating statements with ”not” or taking logical conjunctions/disjunctions (e.g. “It is the case both that s1 and that s2”). Uncurated datasets. In order to validate that the truth representations we identify also generalize to other factual statements, we use uncurated datasets adapted from prior work. These more challenging test sets consist of statements which are more diverse, but also sometimes ambiguous, malformed, controversial, or difficult to understand. likely dataset. To ensure that our truth representations do not merely reflect a representation of probable vs. improbable text, we introduce a likely dataset, consisting of nonfactual text where the final token is either the most or 100th most likely completion according to LLaMA-13B. Our curated, uncurated, and likely datasets are shown in Tab. 1; addition information about their construction is in App. H. We note that for some of our datasets, there is a strong anti-correlation between text being probable and text being true. For instance, for neg_cities and neg_sp_en_trans, the truth value of a statement and the log probability LLaMA-2-70B assigns to it correlate at r=−.63.63r=-.63r = - .63 and r=−.89.89r=-.89r = - .89, respectively.111In contrast, the correlation is strong and positive for cities (r=.85.85r=.85r = .85) and sp_en_trans (r=.95.95r=.95r = .95). This is intuitive: when prompted with “The city of Paris is not in”, LLaMA-2-70B judges “France” to be the most probable continuation (among countries), despite this continuation being false. Together with the likely dataset, this will help us establish that the linear structure we observe in LLM representations is not due to LLMs linearly representing the difference between probable and improbable text. 3 Localizing truth representations via patching Before beginning our study of LLM truth representations, we first address the question of which hidden states might contain such representations. We use simple patching experiments (Vig et al., 2020; Finlayson et al., 2021; Meng et al., 2022; Geiger et al., 2020) to localize certain hidden states for further analysis. Consider the following prompt pFsubscriptp_Fpitalic_F: The city of Tokyo is in Japan. This statement is: TRUE The city of Hanoi is in Poland. This statement is: FALSE The city of Chicago is in Canada. This statement is: Similarly, let pTsubscriptp_Tpitalic_T be the prompt obtained from pFsubscriptp_Fpitalic_F by replacing “Chicago” with “Toronto,” thereby making the final statement true. In order to localize causally implicated hidden states, we run our model M on the input pTsubscriptp_Tpitalic_T and cache the residual stream activations hi,ℓ⁢(pT)subscriptℎℓsubscripth_i, (p_T)hitalic_i , ℓ ( pitalic_T ) for each token position i and layer ℓ ℓ. Then, for each i and ℓ ℓ, we run M on pFsubscriptp_Fpitalic_F but modify M’s forward pass by swapping out the residual stream activation hi,ℓ⁢(pF)subscriptℎℓsubscripth_i, (p_F)hitalic_i , ℓ ( pitalic_F ) for hi,ℓ⁢(pT)subscriptℎℓsubscripth_i, (p_T)hitalic_i , ℓ ( pitalic_T ) (and allowing this change to affect downstream computations); for each of these intervention experiments, we record the difference in log probability between the tokens “TRUE” and “FALSE”; the larger this difference, the more causally influential the hidden state in position i and layer ℓ ℓ is on the model’s prediction. Results for LLaMA-2-13B and the cities dataset are shown in Fig. 2; see App. B for results on more models and datasets. We see three groups of causally implicated hidden states. The final group, labeled (c), directly encodes the model’s prediction: after applying the LLM’s decoder head directly to these hidden states, the top logits belong to tokens like “true,” “True,” and “TRUE.” The first group, labeled (a), likely encodes the LLM’s representation of “Chicago” or “Toronto.” Figure 2: Difference log⁡P⁢(TRUE)−log⁡P⁢(FALSE)TRUEFALSE P( TRUE)- P( FALSE)log P ( TRUE ) - log P ( FALSE ) in LLaMA-2-13B log probabilities after patching residual stream activation in the indicated token position and layer. What does group (b) encode? The position of this group—over the final token of the statement and end-of-sentence punctuation222This summarization behavior, in which information about clauses is encoded over clause-ending punctuation tokens, was also noted in Tigges et al. (2023). We note that the largest LLaMA model displays this summarization behavior in a more context-dependent way; see App. B.—suggests that it encodes information pertaining to the full statement. Since the information encoded is also causally influential on the model’s decision to output “TRUE” or “FALSE,” we hypothesize that these hidden states store a representation of the statement’s truth. In the remainder of this paper, we systematically study these hidden states. 4 Visualizing LLM representations of true/false datasets We begin our investigation with a simple technique: visualizing LLMs representations of our datasets using principal component analysis (PCA). Guided by the results of §3, we present here visualizations of the most downstream hidden state in group (b); for example, for LLaMA-2-13B, we use the layer 15 residual stream activation over the end-of-sentence punctuation token.333Our qualitative results are insensitive to choice of layer among early-middle to late-middle layers. On the other hand, when using representations over the final token in the statement (instead of the punctuation token), we sometimes see that the top PCs instead capture variation in the token itself (e.g. clusters for statements ending in “China” regardless of their truth value). Unlike in §3, we do not prepend the statements with a few-shot prompt (so our models are not “primed” to consider the truth value of our statements). For each dataset, we also center the activations by subtracting off their mean. When visualizing LLaMA-2-13B and 70B representations of our curated datasets – datasets constructed to have little variation with respect to non-truth features, such as sentence structure or subject matter – we see clear linear structure (Fig. 1), with true statements separating from false ones in the top two principal components (PCs). As explored in App. C, this structure emerges rapidly in early-middle layers and emerges later for datasets of more structurally complex statements (e.g. conjunctive statements). To what extent does this visually-apparent linear structure align between different datasets? Our visualizations indicate a nuanced answer: the axes of separation for various true/false datasets align often, but not always. For instance, Fig. 3(a) shows the first PC of cities also separating true/false statements from other datasets, including diverse uncurated datasets. On the other hand, Fig. 3(c) shows stark failures of alignment, with the axes of separation for datasets and statements and their negations being approximately orthogonal. These cases of misalignment have an interesting relationship to scale. Fig. 3(b) shows larger_than and smaller_than separating along antipodal directions in LLaMA-2-13B, but along a common direction in LLaMA-2-70B. App. C depicts a similar phenomenon occuring over the layers of LLaMA-2-13B: in early layers, cities and neg_cities separate antipodally, before rotating to lie orthogonally (as in Fig. 3(c)), and finally aligning in later layers. Figure 3: (a) Projections of LLaMA-2-13B onto the top 2 PCs of cities. (b) PCA visualizations of larger_than+smaller_than. For LLaMA-2-7B (left), we see statements cluster according to surface-level characteristics, e.g. presence of the token “eighty.” For LLaMA-2-13B, we see that larger_than (center, top) and smaller_than (center, bottom) separate along opposite directions. (c) PCA visualizations of datasets and their negations. Unlike in other visualizations, we use layer 12121212 for cities+neg_cities; see App. C for an exploration of this misalignment emerging and resolving across layers. 4.1 Discussion Overall, these visualizations suggest that as LLMs scale (and perhaps, also as a fixed LLM progresses through its forward pass), they hierarchically develop and linearly represent increasingly general abstractions. Small models represent surface-level characteristics of their inputs, and large models linearly represent more abstract concepts, potentially including notions like “truth” that capture shared properties of topically and structurally diverse inputs. In middle regimes, we may find linear representation of concepts at intermediate levels of abstraction, for example, “accurate factual recall” or “close association” (in the sense that “Beijing” and “China” are closely associated). To explore these intermediate regimes more deeply, suppose that DD and ′superscript′D D′ are true/false datasets, f+superscriptf^+f+ is a linearly-represented feature which correlates with truth on both DD and ′superscript′D D′, and f−superscriptf^-f- is a feature which correlates with truth on DD but has a negative correlation with truth on ′superscript′D D′. If f+superscriptf^+f+ is very salient (i.e. the datasets’ have large variance along the f-direction) and f−superscriptf^-f- is not, then we expect PCA visualizations of ∪′D D ∪ D′ to show joint separation along f+superscriptf^+f+. If f−superscriptf^-f- is very salient but f+superscriptf^+f+ is not, we expect antipodal separation along f−superscriptf^-f-, as in Fig. 3(b, center). And if both f+superscriptf^+f+ and f−superscriptf^-f- are salient, we expect visualizations like Fig. 3(c). To give an example, suppose that =citiescitiesD= citiesD = cities, ′=neg_citiessuperscript′neg_citiesD = neg\_citiesD′ = neg_cities, f+=“truth”superscript“truth”f^+=``truth′f+ = “truth”, and f−=“close association”superscript“close association”f^-=``close association′f- = “close association”. Then we might expect f−superscriptf^-f- to correlate with truth positively on DD and negatively on ′superscript′D D′. If so, we would expect training linear probes on ∪′D D ∪ D′ to result in improved generalization, despite ′superscript′D D′ consisting of the same statements as DD, but with the word “not” inserted. We investigate this in §5. 5 Probing and generalization experiments In this section we train probes on datasets of true/false statements and test their generalization to other datasets. But first we discuss a deficiency of logistic regression and propose a simple, optimization-free alternative: mass-mean probing. Concretely, mass-mean probes use a difference-in-means direction, but—when the covariance matrix of the classification data is known (e.g. when working with IID data)—apply a correction intended to mitigate interference from non-orthogonal features. We will see that mass-mean probes are similarly accurate to probes trained with other techniques (including on out-of-distribution data) while being more causally implicated in model outputs. 5.1 Challenges with logistic regression, and mass-mean probing A common technique in interpretability research for identifying feature directions is training linear probes with logistic regression (LR; Alain & Bengio, 2018). In some cases, however, the direction identified by LR can fail to reflect an intuitive best guess for the feature direction, even in the absence of confounding features. Consider the following scenario, illustrated in Fig. 4 with hypothetical data: • Truth is represented linearly along a direction tsubscript θ_titalic_θitalic_t. • Another feature f is represented linearly along a direction fsubscript θ_fitalic_θitalic_f not orthogonal to tsubscript θ_titalic_θitalic_t.444The superposition hypothesis of Elhage et al. (2022), suggests this may be typical in deep networks. • The statements in our dataset have some variation with respect to feature f, independent of their truth value. We would like to identify the direction tsubscript θ_titalic_θitalic_t, but LR fails to do so. Assuming for simplicity linearly separable data, LR instead converges to the maximum margin separator Soudry et al. (2018) (the dashed magenta line in Fig. 4). Intuitively, LR treats the small projection of fsubscript θ_fitalic_θitalic_f onto tsubscript θ_titalic_θitalic_t as significant, and adjusts the probe direction to have less “interference” (Elhage et al., 2022) from fsubscript θ_fitalic_θitalic_f. Figure 4: An illustration of a weakness of logistic regression. A simple alternative to LR which identifies the desired direction in this scenario is to take the vector pointing from the mean of the false data to the mean of the true data. In more detail if =(i,yi)subscriptsubscriptD=\( x_i,y_i)\D = ( italic_xitalic_i , yitalic_i ) is a dataset of i∈ℝdsubscriptsuperscriptℝ x_i ^ditalic_xitalic_i ∈ blackboard_Rd with binary labels yi∈0,1subscript01y_i∈\0,1\yitalic_i ∈ 0 , 1 , we set m=+−subscriptmmsuperscriptsuperscript θ_m= μ^+- μ^-italic_θroman_m = italic_μ+ - italic_μ- where +,−superscriptsuperscript μ^+, μ^-italic_μ+ , italic_μ- are the means of the positively- and negatively-labeled datapoints, respectively. A reasonable first pass at converting mmsubscriptmm θ_mitalic_θroman_m into a probe is to define555Since we are interested in truth directions, we always center our data and use unbiased probes. pmm⁢()=σ⁢(mmT⁢x)subscriptmmsuperscriptsubscriptmmp_m( x)=σ( θ_m^Tx)proman_m ( italic_x ) = σ ( italic_θroman_mmitalic_T x ) where σ is the logistic function. However, when evaluating on data that is independent and identically distributed (IID) to DD, we can do better by tilting our decision boundary to accommodate interference from fsubscript θ_fitalic_θitalic_f. Concretely this means setting pmmiid⁢()=σ⁢(mmT⁢Σ−1⁢)superscriptsubscriptmmiidsuperscriptsubscriptmmsuperscriptΣ1p_m^iid( x)=σ( θ_m^T% ^-1 x)proman_mmroman_iid ( italic_x ) = σ ( italic_θroman_mmitalic_T Σ- 1 italic_x ) where Σ Σ is the covariance matrix of the dataset c=i−+:yi=1∪i−:yi=0superscriptconditional-setsubscriptsuperscriptsubscript1conditional-setsubscriptsuperscriptsubscript0D^c=\ x_i- μ^+:y_i=1\∪\ x_i- % μ^-:y_i=0\Ditalic_c = italic_xitalic_i - italic_μ+ : yitalic_i = 1 ∪ italic_xitalic_i - italic_μ- : yitalic_i = 0 ; this coincides with performing linear discriminant analysis (Fisher, 1936).666We prove in App. F that, given infinite data and a homoscedasticity assumption, Σ−1⁢mmsuperscriptΣ1subscriptmm ^-1 θ_mΣ- 1 italic_θroman_m coincides with the direction found by LR. Thus, one can view IID mass-mean probing as providing a way to select a good decision boundary while – unlike LR – also tracking a candidate feature direction which may be non-orthogonal to this decision boundary. App. E provides another interpretation of mass-mean probing in terms of Mahalanobis whitening. Finally, App. We call the probes pmmsubscriptmmp_mproman_m and pmmiidsuperscriptsubscriptmmiidp_m^iidproman_mmroman_iid mass-mean probes. As we will see, mass-mean probing is about as accurate for classification as LR, while also identifying directions which are more causally implicated in model outputs. 5.2 Experimental set-up In this section, we measure the effect that choice of training data, probing technique, and model scale has on probe accuracy. For training data, we use one of: cities, cities + neg_cities, larger_than, larger_than + smaller_than, or likely. By comparing probes trained on cities to probes trained on cities + neg_cities, we are able to measure the effect of increasing data diversity in a particular, targeted way: namely, we mitigate the effect of linearly-represented features which have opposite-sign correlations with the truth in cities and neg_cities. As in §4, we will extract activations at the most-downstream hidden state in group (b). Our probing techniques are logistic regression (LR), mass-mean probing (M), and contrast-consistent search (CCS). CCS is an unsupervised method introduced in Burns et al. (2023): given contrast pairs of statements with opposite truth values, CCS identifies a direction along which the representations of these statements are far apart. For our contrast pairs, we pair statements from cities and neg_cities, and from larger_than and smaller_than. For test sets, we use all of our (curated and uncurated) true/false datasets. Given a training set DD, we train our probe on a random 80% split of DD. Then when evaluating accuracy on a test set ′superscript′D D′, we use the remaining 20% of the data if ′=superscript′D =DD′ = D and the full test set otherwise. For mass-mean probing, if =′superscript′D=D D = D′, we use pmmiidsuperscriptsubscriptmmiidp_m^iidproman_mmroman_iid, and we use pmmsubscriptmmp_mproman_m otherwise. Finally, we also include as baselines calibrated few-shot prompting777We first sweep over a number n of shots and then resample a few n-shot prompts to maximize performance. The word “calibrated” means we selected a threshold for P⁢(TRUE)−P⁢(FALSE)TRUEFALSEP( TRUE)-P( FALSE)P ( TRUE ) - P ( FALSE ) such that half of the statements are labeled true; this improves performance by a few percentage points. and – as an oracle baseline – LR on the test set. 5.3 Results Figure 5: (a) Average accuracies over all datasets aside from those used for training. (b) Accuracies of probes for varying model scales and training data, averaged over all test sets. For each training set, probing technique, and model scale, we report the average accuracy across test sets. We expect many readers to be interested in the full results (including test set-specific accuracies), which are reported in App. D. Calibrated few-shot prompting was a surprisingly weak baseline, so we do not report it here (but see App. D). Training on statements and their opposites improves generalization (Fig. 5(a)). When passing from cities to cities+neg_cities, this effect is largely explained by improved generalization on neg_sp_en_trans, i.e. using training data containing the word “not” improves generalization on other negated statements. On the other hand, passing from larger_than to larger_than+smaller_than also improves performance, despite both datasets being very structurally different from the rest of our datasets. As discussed in §4.1, this suggest that training on statements and their opposites mitigates the effect certain types of non-truth features have on the probe direction. Probes generalize better for larger models (Fig. 5). While it is unsurprising that larger models are themselves better at labeling statements as true or false, it is not obvious that linear probes trained on larger models should also generalize better. Nevertheless, for LLaMA-2-13B and 70B, generalization is generally high; for example, no matter which probing technique is used, we find that probes trained on larger_than + smaller_than get >95%absentpercent95>95\%> 95 % accuracy on sp_en_trans. This corroborates our discussion in §4.1, in which we suggested that larger models linearly represent more general concepts concepts, like truth, which capture shared aspects of diverse inputs. Mass-mean probes generalize about as well as other probing techniques for larger models (Fig. 5(b)). While M underperforms LR and CCS for LLaMA-2-7B, we find for larger models performance comparable to that of other probing techniques. Further, we will see in §6 that the directions identified by M are more causally implicated in model outputs. Probes trained on likely perform poorly (Fig. 5(b)). The full results reveal that probes trained on likely are accurate when evaluated on some datasets, such as sp_en_trans where there is a strong (r=.95.95r=.95r = .95) correlation between text probability and truth. However, on other datasets, especially those with anti-correlations between probability and truth, these probes perform worse than chance. Overall, this indicates that LLMs linearly represent truth-relevant information beyond the plausibility of text. 6 Causal intervention experiments In §5 we measured the quality of linear probes in terms of their classification accuracy, both in- and out-of-distribution. In this section, we perform experiments which measure the extent to which these probes identify directions which are causally implicated in model outputs Finlayson et al. (2021); Geva et al. (2023); Geiger et al. (2021). To do this, we will intervene in our model’s computation by shifting the activations in group (b) (identified in §3) along the directions identified by our linear probes. Our goal is to cause LLMs to treat false statements appearing in context as true and vice versa. Crucially—and in contrast to prior work (Li et al., 2023b)—we evaluate our interventions on OOD inputs. Table 2: NIEs for intervention experiments, averaged over statements from sp_en_trans. LLaMA-2-13B LLaMA-2-70B train set probe false→true true→false false→true true→false cities LR .13 .19 .55 .99 M .77 .90 .58 .89 cities+ neg_cities LR .33 .52 .61 1.00 M .85 .97 .81 .95 CCS .31 .73 .55 .96 larger_than LR .28 .27 .61 .96 M .71 .79 .67 1.01 larger_than+ smaller_than LR .07 .13 .54 1.02 M .26 .53 .66 1.03 CCS .08 .17 .57 1.02 likely LR .05 .08 .18 .46 M .70 .54 .68 .27 6.1 Experimental set-up Let p be a linear probe trained on a true/false dataset DD. Let θitalic_θ be the probe direction, normalized so that p⁢(μ−+)=p⁢(μ+)superscriptsuperscriptp(μ^-+ θ)=p(μ^+)p ( μ- + italic_θ ) = p ( μ+ ) where μ+superscriptμ^+μ+ and μ−superscriptμ^-μ- are the mean representations of the true and false statements in DD, respectively; in other words, we normalize θitalic_θ so that from the perspective of the probe p, adding θitalic_θ turns the average false statement into the average true statement. If our model encodes the truth value of statements along the direction θitalic_θ, we would expect that replacing the representation xitalic_x of a false statement s with + x+ θitalic_x + italic_θ would cause the model to produce outputs consistent with s being a true statement. We use inputs of the form The Spanish word ‘fruta’ means ‘goat’. This statement is: FALSE The Spanish word ‘carne’ means ‘meat’. This statement is: TRUE s. This statement is: where s varies over sp_en_trans statements. Then for each of the probes of §5 we record: • P⁢D+superscriptPD^+P D+ and P⁢D−superscriptPD^-P D-, the average probability differences P⁢(TRUE)−P⁢(FALSE)TRUEFALSEP( TRUE)-P( FALSE)P ( TRUE ) - P ( FALSE ) for s varying over true statements or false statements in sp_en_trans, respectively, • P⁢D∗+subscriptsuperscriptPD^+_*P D+∗ and P⁢D∗−subscriptsuperscriptPD^-_*P D-∗, the average probability differences where s varies over true (resp. false) statements but the probe direction θitalic_θ is subtracted (resp. added) to each group (b) hidden state. Finally, we report the normalized indirect effects (NIEs) P⁢D∗−P⁢D−P⁢D+−P⁢D−orP⁢D∗+−P⁢D+P⁢D−P⁢D+superscriptsubscriptsuperscriptsuperscriptsuperscriptorsuperscriptsubscriptsuperscriptsuperscriptsuperscript PD_*^--PD^-PD^+-PD^- PD_*^+-PD^% +PD^--PD^+divide start_ARG P D∗- - P D- end_ARG start_ARG P D+ - P D- end_ARG or divide start_ARG P D∗+ - P D+ end_ARG start_ARG P D- - P D+ end_ARG for the false→true and the true→false experiments, respectively. An NIE of 00 means that the intervention was wholly ineffective at changing model outputs; an NIE of 1111 indicates that the intervention caused the LLM to label false statements as TRUE with as much confidence as genuine true statements, or vice versa. 6.2 Results Results are shown in table 2. We summarize our main takeaways. Mass-mean probe directions are highly causal, with M outperforming LR and CCS in 7/8 experimental conditions, often substantially. This is true despite LR, M, and CCS probes all have very similar sp_en_trans classification accuracies. Training on datasets and their opposites helps for cities but not for larger_than. This is surprising, considering that probes trained on larger_than + smaller_than are more accurate on sp_en_trans than probes trained on larger_than alone (see App. D), and indicates that there is more to be understood about how training on datasets and their opposites affects truth probes. Training on likely is a surprisingly good baseline, though still weaker than interventions using truth probes. The performance here may be due to the strong correlation (r=.95.95r=.95r = .95) between inputs being true and probable (according to LLaMA-2-70B) on sp_en_trans. 7 Discussion 7.1 Limitations and future work Our work has a number of limitations. First, we focus on simple, uncontroversial statements, and therefore cannot disambiguate truth from closely related features, such as “commonly believed” or “verifiable” (Levinstein & Herrmann, 2023). Second, we study only models in the LLaMA-2 family, so it is possible that some of our results do not apply for all LLMs. This work also raises several questions which we were unable to answer here. For instance, why were interventions with mass-mean probe directions extracted from the likely dataset so effective, despite these probes not themselves being accurate at classifying true/false statements? And why did mass-mean probing with the cities + neg_cities training data perform poorly poorly for the 70B model, despite mass-mean probing with larger_than + smaller_than performing well? 7.2 Conclusion In this work we conduct a detailed investigation of the structure of LLM representations of truth. Drawing on simple visualizations, probing experiments, and causal evidence, we find evidence that at scale, LLMs compute and linearly represent the truth of true/false statements. We also localize truth representations to certain hidden states and introduce mass-mean probing, a simple alternative to other linear probing techniques which better identifies truth directions from true/false datasets. Acknowledgements We thank Ziming Liu and Isaac Liao for useful suggestions regarding distinguishing true text from likely text, and Wes Gurnee, Eric Michaud, and Peter Park for many helpful discussions throughout this project. We thank David Bau for useful suggestions regarding the experiments in Sec. 6. Thanks also to Nora Belrose for discussion about the connection between difference-in-mean probing and linear erasure. We also thank Oam Patel, Hadas Orgad, Sohee Yang, and Karina Nguyen for their suggestions, as well as Helena Casademunt, Max Nadeau, and Ben Edelman for giving feedback during this paper’s preparation. Plots were made with Plotly (Plotly Technologies Inc., 2015). Thanks to Kevin Ro Wang for catching a typo in the statement of Thm. F.1. References Abdou et al. (2021) Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color, 2021. Alain & Bengio (2018) Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. Azaria & Mitchell (2023) Amos Azaria and Tom Mitchell. The internal state of an llm knows when its lying, 2023. Bau et al. (2020) David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907375117. URL https://w.pnas.org/content/early/2020/08/31/1907375117. Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=awIpKpwTwF. Burns et al. (2023) Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs. Casper et al. (2023) Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch, 2023. Christiano et al. (2021) Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How to tell if your eyes deceive you, 2021. URL https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#heading=h.jrzi4atzacns. Cunningham et al. (2023) Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. Dalvi et al. (2018) Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James R. Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In AAAI Conference on Artificial Intelligence, 2018. URL https://api.semanticscholar.org/CorpusID:56895415. Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. Finlayson et al. (2021) Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 1828–1843, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.144. URL https://aclanthology.org/2021.acl-long.144. Fisher (1936) R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179–188, 1936. doi: https://doi.org/10.1111/j.1469-1809.1936.tb02137.x. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-1809.1936.tb02137.x. Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupala, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2020, Online, November 2020, p. 163–173. Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020.BLACKBOXNLP-1.16. URL https://doi.org/10.18653/v1/2020.blackboxnlp-1.16. Geiger et al. (2021) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, p. 9574–9586, 2021. URL https://proceedings.neurips.c/paper/2021/hash/4f5c422f4d49a5a807eda27434231040-Abstract.html. Geonames (2023) Geonames. All cities with a population >>> 1000, 2023. URL https://download.geonames.org/export/dump/. Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023. Goh et al. (2021) Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. Gurnee et al. (2023) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023. Levinstein & Herrmann (2023) B. A. Levinstein and Daniel A. Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks, 2023. Li et al. (2021) Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 1813–1827, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.143. URL https://aclanthology.org/2021.acl-long.143. Li et al. (2023a) Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=DeG07_TcZvT. Li et al. (2023b) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2023b. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229. Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022. Onoe et al. (2021) Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. CREAK: A dataset for commonsense reasoning over entity knowledge. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=mbW_GT3ZN-. OpenAI (2023) OpenAI. Gpt-4 technical report, 2023. Park et al. (2023) Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions, 2023. Patel & Pavlick (2022) Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gJcEM8sxHK. Perez et al. (2022) Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations, 2022. Plotly Technologies Inc. (2015) Plotly Technologies Inc. Collaborative data science, 2015. URL https://plot.ly. Rimsky et al. (2024) Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024. Sajjad et al. (2022) Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10:1285–1303, 2022. doi: 10.1162/tacl˙a˙00519. URL https://aclanthology.org/2022.tacl-1.74. Soudry et al. (2018) Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878, 2018. Steinhardt (2023) Jacob Steinhardt. Emergent deception and emergent optimization, 2023. URL https://bounded-regret.ghost.io/emergent-deception-optimization/. Tigges et al. (2023) Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models, 2023. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, p. 12388–12401. Curran Associates, Inc., 2020. URL https://proceedings.neurips.c/paper_files/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf. Wang et al. (2022) Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 11132–11152, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.765. URL https://aclanthology.org/2022.emnlp-main.765. Zou et al. (2023) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023. Appendix A Scoping of truth In this work, we consider declarative factual statements, for example “Eighty-one is larger than fifty-four” or “The city of Denver is in Vietnam.” We scope “truth” to mean factuality, i.e. the truth or falsehood of these statements; for instance the examples given have truth values of true and false, respectively. To be clear, we list here some notions of “truth” which we do not consider in this work: • Correct question answering (considered in Li et al. (2023b) and for some of the prompts used in Burns et al. (2023)). For example, we do not consider “What country is Paris in? France” to have a truth value. • Presence of deception, for example dishonest expressions of opinion (“I like that plan”). • Compliance. For example, “Answer this question incorrectly: what country is Paris in? Paris is in Egypt” is an example of compliance, even though the statement at the end of the text is false. Moreover, the statements under consideration in this work are all simple, unambiguous, and uncontroversial. Thus, we make no attempt to disambiguate “true statements” from closely-related notions like: • Uncontroversial statements • Statements which are widely believed • Statements which educated people believe. On the other hand, our statements do disambiguate the notions of “true statements” and “statements which are likely to appear in training data”; See our discussion at the end of §2. Appendix B Full patching results Fig. 6 shows full patching results. We see that both LLaMA-2-7B and LLaMA-2-13B display the “summarization” behavior in which information relevant to the full statement is represented over the end-of-sentence punctuation token. On the other hand, LLaMA-2-70B displays this behavior in a context-dependent way – we see it for cities but not for sp_en_trans. Figure 6: Full patching results across all three model sizes and inputs. Results are for patching false inputs (shown) to true by changing the first token shown on the left. Numbers in parentheses are the index of the token in the full (few-shot) prompt. Appendix C Emergence of linear structure across layers Figure 7: Projections of LLaMA-2-13B representations of datasets onto their top two PCs, across various layers. The linear structure observed in §4 follows the following pattern: in early layers, representations are uninformative; then, in early middle layers, salient linear structure in the top few PCs rapidly emerges, with this structure emerging later for statements with a more complicated logical structure (e.g. conjunctions). This is shown for LLaMA-2-13B in Fig. 7. We hypothesize that this is due to LLMs hierarchically developing understanding of their input data, progressing from surface level features to more abstract concepts. The misalignment in Fig. 3(c) also has an interesting dependence on layer. In Fig. 8 we visualize LLaMA-2-13B representations of cities and neg_cities at various layers. In early layers (left) we see antipodal alignment as in Fig. 3(b, center). As we progress through layers, we see the axes of separation rotate to lie orthogonally, until they eventually align. One interpretation of this is that in early layers, the model computed and linearly represented some feature (like “close association”) which correlates with truth on both cities and neg_cities but with opposite signs. In later layers, the model computed and promoted to greater salience a more abstract concept which correlates with truth across both datasets. Figure 8: PCA visualizations of LLaMA-2-13B representations of cities and neg_cities at various layers. Appendix D Full generalization results Here we present the full generalization results for probes trained on LLaMA-2-70B (Fig. 9), 13B (Fig. 10), and 7B (Fig. 11). The horizontal axis shows the training data for the probe and the vertical axis shows the test set. Figure 9: Generalization results for LLaMA-2-70B. Figure 10: Generalization results for LLaMA-2-13B. Figure 11: Generalization results for LLaMA-2-7B. Appendix E Mass-mean probing in terms of Mahalanobis whitening Figure 12: Mass-mean probing is equivalent to taking the projection onto mmsubscriptmm θ_mitalic_θroman_m after applying a whitening transformation. One way to interpret the formula pmmiid⁢()=σ⁢(mmT⁢Σ−1⁢)superscriptsubscriptmmiidsuperscriptsubscriptmmsuperscriptΣ1p_m^iid( x)=σ( θ_m^T% ^-1 x)proman_mmroman_iid ( italic_x ) = σ ( italic_θroman_mmitalic_T Σ- 1 italic_x ) for the IID version of mass-mean probing is in terms of Mahalanobis whitening. Recall that if =xisubscriptD=\x_i\D = xitalic_i is a dataset of xi∈ℝdsubscriptsuperscriptℝx_i ^dxitalic_i ∈ blackboard_Rd with covariance matrix Σ Σ, then the Mahalanobis whitening transformation W=Σ−1/2superscriptΣ12W= ^-1/2W = Σ- 1 / 2 satisfies the property that ′=W⁢xisuperscript′subscriptD =\Wx_i\D′ = W xitalic_i has covariance matrix given by the identity matrix, i.e. the whitened coordinates are uncorrelated with variance 1111. Thus, noting that mmT⁢Σ−1⁢superscriptsubscriptmmsuperscriptΣ1 θ_m^T ^-1 xitalic_θroman_mmitalic_T Σ- 1 italic_x coincides with the inner product between W⁢W xW italic_x and W⁢W θW italic_θ, we see that pmmsubscriptmmp_mproman_m amounts to taking the projection onto mmsubscriptmm θ_mitalic_θroman_m after performing the change-of-basis given by W. This is illustrated with hypothetical data in Fig. 12. Appendix F For Gaussian data, IID mass-mean probing coincides with logistic regression on average Let ∈ℝdsuperscriptℝ θ ^ditalic_θ ∈ blackboard_Rd and Σ Σ be a symmetric, positive-definite d×d× d × d matrix. Suppose given access to a distribution DD of datapoints ∈RdsuperscriptR x ^ditalic_x ∈ Ritalic_d with binary labels y∈0,101y∈\0,1\y ∈ 0 , 1 such that the negative datapoints are distributed as ⁢(−θ,Σ)ΣN(-θ, )N ( - θ , Σ ) and the positive datapoints are distributed as ⁢(θ,Σ)ΣN(θ, )N ( θ , Σ ). Then the vector identified by mass-mean probing is m=2⁢subscriptmm2 θ_m=2 θitalic_θroman_m = 2 italic_θ. The following theorem then shows that pmmiid⁢()=σ⁢(2⁢θT⁢Σ−1⁢)superscriptsubscriptmmiid2superscriptsuperscriptΣ1p_m^iid( x)=σ(2θ^T ^-1 x)proman_mmroman_iid ( italic_x ) = σ ( 2 θitalic_T Σ- 1 italic_x ) is also the solution to logistic regression up to scaling. Theorem F.1. Let lr=arg⁢minϕ:‖ϕ‖=1⁡(,y)∼⁢[y⁢log⁡σ⁢(ϕT⁢)+(1−y)⁢log⁡(1−σ⁢(ϕT⁢))]subscriptlrsubscriptargmin:italic-ϕnormitalic-ϕ1subscriptsimilar-todelimited-[]superscriptitalic-ϕ11superscriptitalic-ϕ θ_lr= *arg\,min_φ:\|φ\|=1% E_( x,y) [y σ (φ^T x )% +(1-y) (1-σ (φ^T x ) ) ]italic_θroman_lr = start_OPERATOR arg min end_OPERATORϕ : ∥ ϕ ∥ = 1 blackboard_E( italic_x , y ) ∼ D [ y log σ ( ϕitalic_T italic_x ) + ( 1 - y ) log ( 1 - σ ( ϕitalic_T italic_x ) ) ] be the direction identified by logistic regression. Then lr∝Σ−1⁢proportional-tosubscriptlrsuperscriptΣ1 θ_lr ^-1 θitalic_θroman_lr ∝ Σ- 1 italic_θ. Proof. Since the change of coordinates ↦W⁢maps-to x W xitalic_x ↦ W italic_x where W=Σ−1/2superscriptΣ12W= ^-1/2W = Σ- 1 / 2 (see App. E) sends ⁢(±θ,Σ)plus-or-minusΣN(±θ, )N ( ± θ , Σ ) to ⁢(±W⁢θ,Id)plus-or-minussubscriptN(± Wθ,I_d)N ( ± W θ , Iitalic_d ), we see that W⁢Σ⁢lr=arg⁢minϕ:‖ϕ‖=1⁡(,y)∼′⁢[y⁢log⁡σ⁢(ϕT⁢W⁢)+(1−y)⁢log⁡(1−σ⁢(ϕT⁢W⁢))]Σsubscriptlrsubscriptargmin:italic-ϕnormitalic-ϕ1subscriptsimilar-tosuperscript′delimited-[]superscriptitalic-ϕ11superscriptitalic-ϕW θ_lr= *arg\,min_φ:\|φ\|=1% E_( x,y) [y σ (φ^% TW x )+(1-y) (1-σ (φ^TW x ) % ) ]W Σ italic_θroman_lr = start_OPERATOR arg min end_OPERATORϕ : ∥ ϕ ∥ = 1 blackboard_E( italic_x , y ) ∼ D′ [ y log σ ( ϕitalic_T W italic_x ) + ( 1 - y ) log ( 1 - σ ( ϕitalic_T W italic_x ) ) ] where ′superscript′D D′ is the distribution of labeled ∈ℝdsuperscriptℝ x ^ditalic_x ∈ blackboard_Rd such that the positive/negative datapoints are distributed as ⁢(±W⁢θ,Id)plus-or-minussubscriptN(± Wθ,I_d)N ( ± W θ , Iitalic_d ). But the argmax on the right-hand side is clearly ∝W⁢θproportional-toabsent Wθ∝ W θ, so that θlr∝Σ−1⁢θproportional-tosubscriptlrsuperscriptΣ1 _lr ^-1θroman_lr ∝ Σ- 1 θ as desired. ∎ Appendix G Difference-in-means directions and linear concept erasure In this appendix, we explain the connection between difference-in-means directions and optimal erasure. One consequence of this connection is that it suggests a natural extension of difference-in-means probes to multi-class classification data. The connection comes via the following theorem from Belrose et al. (2023). Theorem G.1. (Belrose et al., 2023, Thm. 3.1.) Let (X,Y)(X,Y)( X , Y ) be jointly distributed random vectors with X∈ℝdsuperscriptℝX ^dX ∈ blackboard_Rd having finite mean and Y∈=∈0,1k:‖1=1conditional-setsuperscript01subscriptnorm11Y =\y∈\0,1\^k:\|y\|_1=1\Y ∈ Y = y ∈ 0 , 1 k : ∥ y ∥1 = 1 (representing one-hot encodings of a multi-class labels). Suppose that ℒ:ℝk×→ℝ>0:ℒ→superscriptℝsuperscriptℝabsent0L:R^k×Y ^>0L : blackboard_Rk × Y → blackboard_R> 0 is a loss function convex in its first argument (e.g. cross-entropy loss). If the class-conditional means ⁢[X|Y=i]delimited-[]conditionalE[X|Y=i]blackboard_E [ X | Y = i ] for i∈1,…,k1…i∈\1,…,k\i ∈ 1 , … , k are all equal, then the best affine predictor (that is, a predictor η:ℝd→ℝk:→superscriptℝsuperscriptℝη:R^d ^kη : blackboard_Rd → blackboard_Rk of the form η⁢()=W⁢+η( x)=W x+ bη ( italic_x ) = W italic_x + italic_b) is constant η⁢()=η( x)= bη ( italic_x ) = italic_b. In the case of a binary classification problem (X,Y)(X,Y)( X , Y ), this theorem implies that any nullity 1111 projection P which eliminates linearly-recoverable information from X has kernel ker⁡P=span⁢()kernelspan P=span( δ)ker P = span ( italic_δ ) generated by the difference-in-mean vector =+−superscriptsuperscript δ= μ^+- μ^-italic_δ = italic_μ+ - italic_μ- for the classes. For a more general multi-class classification problem, one could similarly ask: What is the “best” direction to project away in order to eliminate linearly-recoverable information from X? A natural choice is thus the top left singular vector of the cross-covariance matrix ΣX⁢YsubscriptΣ _XYΣitalic_X Y. (In the case of binary classification, we have that ΣX⁢Y=[−⁢]subscriptΣdelimited-[] _XY=[- δ\;\; δ]Σitalic_X Y = [ - italic_δ italic_δ ] has column rank 1111, making δitalic_δ the top left singular vector.) Appendix H Details on dataset creation Here we give example statements from our datasets, templates used for making the datasets, and other details regarding dataset creation. cities. We formed these statements from the template “The city of [city] is in [country]” using a list of world cities from Geonames (2023). We filtered for cities with populations >500,000absent500000>500,000> 500 , 000, which did not share their name with any other listed city, which were located in a curated list of widely-recognized countries, and which were not city-states. For each city, we generated one true statement and one false statement, where the false statement was generated by sampling a false country with probability equal to the country’s frequency among the true datapoints (this was to ensure that e.g. statements ending with “China” were not disproportionately true). Example statements: • The city of Sevastopol is in Ukraine. (TRUE) • The city of Baghdad is in China. (FALSE) sp_en_trans. Beginning with a list of common Spanish words and their English translations, we formed statements from the template “The Spanish word ‘[Spanish word]’ means ‘[English word]’.” Half of Spanish words were given their correct labels and half were given random incorrect labels from English words in the dataset. The first author, a Spanish speaker, then went through the dataset by hand and deleted examples with Spanish words that have multiple viable translations or were otherwise ambiguous. Example statements: • The Spanish word ‘imaginar’ means ‘to imagine’. (TRUE) • The Spanish word ‘silla’ means ‘neighbor’. (FALSE) larger_than and smaller_than. We generate these statements from the templates “x is larger than y” and “x is smaller than y” for x,y∈fifty-one,fifty-two,…,ninety-ninexyfifty-onefifty-two…ninety-nine x, y∈\ fifty-one, fifty-two,…, % ninety-nine\x , y ∈ fifty-one , fifty-two , … , ninety-nine . We exclude cases where x=yxy x= yx = y or where one of x or y is divisible by 10101010. We chose to limit the range of possible values in this way for the sake of visualization: we found that LLaMA-13B linearly represents the size of numbers, but not at a consistent scale: the internally represented difference between one and ten is considerably larger than between fifty and sixty. Thus, when visualizing statements with numbers ranging to one, the top principal components are dominated by features representing the sizes of numbers. neg_cities and neg_sp_en_trans. We form these datasets by negating statements from cities and sp_en_trans according to the templates “The city of [city] is not in [country]” and “‘The Spanish word ‘[Spanish word]’ does not mean ‘[English word]’.” cities_cities_conj and cities_cities_disj. These datasets are generated from cities according to the following templates: • It is the case both that [statement 1] and that [statement 2]. • It is the case either that [statement 1] or that [statement 2]. We sample the two statements independently to be true with probability 1212 1 2divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 end_ARG end_ARG for cities_cities_conj and with probability 1−121121- 1 21 - divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 end_ARG end_ARG for cities_cities_disj. These probabilities are selected to ensure that the overall dataset is balanced between true and false statements, but that there is no correlation between the truth of the first and second statement in the conjunction. likely. We generate this dataset by having LLaMA-13B produce unconditioned generations of length up to 100100100100 tokens, using temperature 0.90.90.90.9. At the final token of the generation, we either sample the most likely token or the 100th most likely final token. We remove generations which contain special tokens. Dataset examples: • The 2019-2024 Outlook for Women’s and Girls’ Cut and Sew and Knit and Crochet Sweaters in the United States This study covers the latent demand outlook for (LIKELY) • Tags: python, django Question: How to get my django app to work with python 3.7 I am new to django and have been trying to install it in my pc. I have installed python 3.7 together (UNLIKELY) companies_true_false. This dataset was introduced by Azaria & Mitchell (2023); we obtained it via the project repository for Levinstein & Herrmann (2023) which also used the dataset. Example statements: • ArcelorMittal has headquarters in Luxembourg. (TRUE) • Exxon Mobil engages in the provision of banking and financial services. (FALSE) common_claim_true_false. CommonClaim was introduced in Casper et al. (2023). It consists of various statements generated by GPT-3-davinci-002, labeled by humans as being true, false, or neither. If human labelers disagreed on the truth of a statement, this is also recorded. We adapted CommonClaim by selecting statements which were labeled true or false with no labeler disagreement, then removing excess true statement to balance the dataset. Example statements: • Tomatoes are not actually a vegetable. (TRUE) • Contrary to popular belief, the platypuses are not venomous. (FALSE) As these examples show, the statements can be ambiguous or of unclear truth value. counterfact_true_false. Counterfact was introduced in Meng et al. (2022) and consists of factual recall statements. We adapt Counterfact by using statements which form complete sentences and, for each such statement, using both the true version and a false version given by one of Counterfact’s suggested false modifications. We also append a period to the end. Example statements: • Olaus Rudbeck spoke the language Swedish. (TRUE) • The official religion of Malacca sultanate is Christianity. (FALSE)

Instant research discovery