Paper deep dive
Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks
Yuntai Bao, Xuhong Zhang, Tianyu Du, Xinkui Zhao, Zhengwen Feng, Hao Peng, Jianwei Yin
Models: Llama-2-13B, Llama-2-13B-Chat, Llama-2-7B, Llama-2-7B-Chat, Llama-3.1-70B, Llama-3.1-70B-Instruct, Llama-3.1-8B, Llama-3.1-8B-Instruct, Mistral-7B-v0.1
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 5:48:05 PM
Summary
This paper investigates the 'truth direction' hypothesis in Large Language Models (LLMs), which posits that truthfulness is encoded as a linear feature in internal representations. The authors evaluate whether this feature is consistent across models, if sophisticated probing is required, and how it generalizes across logical transformations and question-answering tasks. Findings indicate that truth direction consistency correlates with model capability, and that simple linear probes are sufficient to identify these directions, which generalize effectively across various logical and contextual settings.
Entities (5)
Relation Signals (3)
Truthfulness Probes â generalizeto â Question-Answering Tasks
confidence 95% ¡ truthfulness probes trained on declarative atomic statements can generalize effectively to logical transformations, question-answering tasks
Truthfulness Probes â usedin â Selective Question Answering
confidence 95% ¡ we explore the practical application of truthfulness probes in selective question-answering
Truth Direction â exhibitedby â Capable LLMs
confidence 90% ¡ not all LLMs exhibit consistent truth directions, with stronger representations observed in more capable models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) are trained on extensive datasets that encapsulate substantial world knowledge. However, their outputs often include confidently stated inaccuracies. Earlier works suggest that LLMs encode truthfulness as a distinct linear feature, termed the "truth direction", which can classify truthfulness reliably. We address several open questions about the truth direction: (i) whether LLMs universally exhibit consistent truth directions; (ii) whether sophisticated probing techniques are necessary to identify truth directions; and (iii) how the truth direction generalizes across diverse contexts. Our findings reveal that not all LLMs exhibit consistent truth directions, with stronger representations observed in more capable models, particularly in the context of logical negation. Additionally, we demonstrate that truthfulness probes trained on declarative atomic statements can generalize effectively to logical transformations, question-answering tasks, in-context learning, and external knowledge sources. Finally, we explore the practical application of truthfulness probes in selective question-answering, illustrating their potential to improve user trust in LLM outputs. These results advance our understanding of truth directions and provide new insights into the internal representations of LLM beliefs. Our code is public at this https URL
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
67,033 characters extracted from source content.
Expand or collapse full text
Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks Yuntai Bao 1 Xuhong Zhang 1 * Tianyu Du 1 Xinkui Zhao 1 Zhengwen Feng 1 Hao Peng 2 Jianwei Yin 1 1 Zhejiang University 2 Zhejiang Normal University yuntaibao, zhangxuhong, zjradty, zhaoxinkui, fengzhengwen@zju.edu.cn, hpeng@zjnu.edu.cn,zjuyjw@cs.zju.edu.cn Abstract Large language models (LLMs) are trained on extensive datasets that encapsulate substantial world knowledge. However, their outputs often include confidently stated inaccuracies. Earlier works suggest that LLMs encode truthfulness as a distinct linear feature, termed the âtruth directionâ, which can classify truthfulness reli- ably. We address several open questions about the truth direction: (i) whether LLMs univer- sally exhibit consistent truth directions; (i) whether sophisticated probing techniques are necessary to identify truth directions; and (i) how the truth direction generalizes across di- verse contexts. Our findings reveal that not all LLMs exhibit consistent truth directions, with stronger representations observed in more capable models, particularly in the context of logical negation. Additionally, we demonstrate that truthfulness probes trained on declarative atomic statements can generalize effectively to logical transformations, question-answering tasks, in-context learning, and external knowl- edge sources. Finally, we explore the practical application of truthfulness probes in selective question-answering, illustrating their potential to improve user trust in LLM outputs. These re- sults advance our understanding of truth direc- tions and provide new insights into the internal representations of LLM beliefs. 1 1 Introduction Large language models (LLMs) possess extensive knowledge, as they are trained on immense cor- pora that encompass a significant portion of world knowledge. However their outputs are not always reliable and are prone to confidently presenting falsehoods (Bender et al., 2021; Evans et al., 2021; Lin et al., 2022; Liu et al., 2023). This unreliabil- ity raises critical concerns about the use of LLMs * Corresponding author. 1 Ourcodeispublicathttps://github.com/ colored-dye/truthfulness_probe_generalization in applications where accuracy is paramount. A growing body of work (Burns et al., 2022; Azaria and Mitchell, 2023; Marks and Tegmark, 2023; Mallen and Belrose, 2023; BĂźrger et al., 2024) aims to elicit accurate information from LLMs despite untruthful outputs. These studies use lightweight classifiers, often referred to asprobes, to analyze patterns in the modelâs internal representation that reliably indicate truthfulness. Specifically, given a model and a piece of text, an ideal truthfulness probe is able to tell whether the model believes the text conveys truthful content. By achieving empirical success with linear probes, these works generally believe that truthfulness is internally rep- resented as a salient linear feature and manifests as a âtruth directionâ. The goal of this work is to conduct a more in- depth study of the truth direction as an inherent property of LLMs. Although previous works un- doubtedly help us understand truth directions, they fail to answer the following questions: (RQ1) Do LLMs universally represent truthfulness as a lin- ear feature? (RQ2) Are simple probing techniques sufficiently expressive to identify truth directions? (RQ3) If and when a âtruth directionâ exists, in what ways does it generalize? We challenge con- clusions from prior works based on empirical ev- idence, provide preliminary answers to the ques- tions above and present novel observations. In response toRQ1, we find that not all LLMs exhibit a consistent âtruth directionâ, and that this property is closely related to a modelâs capabil- ity. While prior works often assume the univer- sal existence of truth directions, we challenge this assumption. Our evidence suggests that truthful- ness is more consistently represented across logical negations in more capable LLMs. Based on this finding, we question the conclusion of Levinstein and Herrmann (2024), which attributes the gener- alization failure to limitations in previous probing techniques. Instead, we argue that the inconsis- arXiv:2506.00823v1 [cs.CL] 1 Jun 2025 tency lies within the LLM itself and that in answer toRQ2, simple supervised probes are sufficiently expressive to identify the truth direction when it is distinctly represented within the model. In addressingRQ3, we aim to explore the gen- eralization capability of the truth direction, as it il- luminates whether an LLM consistently represents truthfulness across different knowledge domains, logical transformations, syntax forms and ground- ing knowledge source. Previous studies on truth directions have extensively explored the former two aspects of generalization. Regarding syntax forms, they focus either on declarative statements or on (Q,A)pairs; we bridge this gap by testing whether truthfulness probes trained on simple statements generalize to question answering (QA) tasks. Ad- ditionally, we examine several variations of QA, including zero- and few-shot QA, with and without provided answer options, and grounded either in parametric knowledge or in question contexts. Our experimental results show that truthfulness probes demonstrate a high degree of generalization. Furthermore, based on our observation that truth- fulness probes are calibrated on certain QA tasks, we introduce their use in selective QA. In this appli- cation, we select the subset of answers evaluated as correct by the truthfulness probe from those gener- ated by the LLM. Through this demonstration, we aim to show how truthfulness probes can enhance user trust in real-world LLM-based applications. Our contributions are as follows: 1.We summarize truthfulness probes from prior works, introduce a new instantiation, and con- duct extensive experiments. 2. We study truth directions following three re- search questions. In addressingRQ1, we ex- plore whether the truth direction is common among LLMs. ForRQ2, we test if sophisti- cated probing techniques are required to iden- tify truth directions. In answer toRQ3, we assess the generalization capabilities of truth directions. 3. We demonstrate a practical application of truthfulness probes for selective question an- swering, improving generation quality by fil- tering out unreliable answers. 2 Related Work Eliciting latent knowledge (ELK).The field of scalable oversight (Christiano et al., 2021) seeks to address the information asymmetry between su- perhuman AI systems and human evaluators. It is assumed that although the AI possesses significant knowledge, its behavior is untrustworthy because it is not trained with an objective that explicitly incentivizes outputs to align with the truth (Mallen and Belrose, 2023). ELK is an approach within scalable oversight that aims to identify patterns in an AIâs activations that correspond to the truth. The primary challenge lies in identifying patterns that generalize reliably to questions where human evaluators are unable to verify the answers (Mallen and Belrose, 2023). Our work demonstrates that probing techniques can achieve reasonable gener- alization with limited supervision, suggesting that probes may offer a promising approach for ELK. Probing for truthfulness.Several studies have used probing techniques to uncover truthfulness by examining an LLMâs internal states, indepen- dent of its inputs or outputs (Lee et al., 2023; Joshi et al., 2024). Regarding thegeometry of the representationof truthfulness, most studies agree that this representation is likely linear, as demonstrated by the use of linear probes in stud- ies such as CCS (Burns et al., 2022), mass-mean (Marks and Tegmark, 2023; Li et al., 2023), TTPD (BĂźrger et al., 2024) and the commonly used base- line: logistic regression. Notably, CCS is an un- supervised approach and targets yes/noquestion- answering tasks, while others are supervised and targetfactual statements. In contrast, some works aregeometry-agnostic. Azaria and Mitchell (2023) propose SAPLMA which is based on MLP archi- tecture, while He et al. (2024) introduce the LLM Factoscope, which leverages a Convolutional Neu- ral Network architecture. The aforementioned studies examine truthful- ness grounded in a modelâsparametric knowl- edge, whereas Sky et al. (2024) detect hallucination with probes in the setting of in-context generation, whereknowledge is grounded in the context. 3 Summary of Probes and Data In this section, we formally define the task of prob- ing binary features from a modelâs internal repre- sentations and describe the specific probe architec- tures used in this study. Furthermore, we detail the labeled datasets used to train and evaluate the truthfulness probes. 1. Factual statement collection animal_class cities element_symb facts inventors sp_en_trans 2. Internal activations collection stmt Statement: The salmon is a fish. Label: True Q: Which American-born Sinclair won the Nobel Prize for Literature in 1930? A: Upton Sinclair Label: False Example statement Example (Q,A) stmt stmt activation 3. Probe establishment activation activation label Truth direction ÎŚ 4. Test-time generalization stmt1 â§stmt2 stmt1 â¨stmt2 (Context,) Question, Answer (íś 1 ,)í 1 ,í´ 1 ,...,(íś í ,)í í ,í´ í , (íŞ,)í¸,í¨ activation activation activation P[ÎŚ=1] ÎŚ activation activation activation activation activation activation stmt activation activation activation activation activation label Logical conjunction: Logical disjunction: (In-context) QA: Few-shot (in-context) QA: Figure 1: Illustration of truthfulness probes. A truthfulness probe is established using the LLMâs internal states when processing labeled statements. The probe is then able to tell if the LLM believes an unseen statement or a given response to a question is true or false leveraging only the LLMâs internal states. 3.1 Formulation of Binary Probes Our target model is the transformer language model (Vaswani, 2017), which processes token sequences through a series of layers. A sequence ofninput tokens,t= (t 1 ,t 2 ,...,t n ), is first converted into embeddings,h (0) = (h (0) 1 ,h (0) 2 ,...,h (0) n ), by the initial embedding layer. The embeddings are then passed throughLlayers, where each layer gener- ates representations based on the preceding layerâs output. The representation of a single token is a vector:h (j) i âR d and(0â¤jâ¤L,1â¤iâ¤n). Finally, the LLM produces predictions usingh (L) . Suppose we have prior knowledge that the model internally represents a binary feature. Our goal is to establish a probeÎŚ, that uses only the modelâs representations to classify the target attribute. The probe outputs either binary labelsâ1/1, or proba- bilistic predictions such as P[ÎŚ = 1]. For autoregressive models, which are the pri- mary focus of this paper, we utilize the represen- tation at the final token position of thel-th layer, h (l) â1 . This approach aligns with prior work (Burns et al., 2022; Azaria and Mitchell, 2023; Marks and Tegmark, 2023), where the final token position at- tends to all previous tokens due to the causal atten- tion mechanism. We also assume we have a labeled dataset,D=(x i ,y i ) M i=1 , wherex i represents a token sequence andy i is the label for the target attribute. Processing these sequences through the model yieldsD rep =((h (l) â1 ) i ,y i ) M i=1 , which we use to build our classifier. To maximize classification accuracy, we de- fine a cost function to quantify the classifica- tion error of the probe,J(ÎŚ,h,y). The mech- anistic objective is to minimize the expected classification error over the data distribution: arg min ÎŚ 1 M P M i=1 J(ÎŚ,h i ,y j ). 3.2 Instantiations of Binary Probes We classify the probes into two categories: geometry-oriented and statistics-based. Geometry- oriented probes leverage knowledge of the geo- metric structure of the representation. Under the âtruth directionâ hypothesis, true/false representa- tions can be separated by a hyperplane, and the normal vector of this hyperplace corresponds to the âtruth directionâ. In contrast, statistics-based probes are geometry-agnostic and aim to maximize the probability of observing the correct labels given the input data. Our implementations are based on thescikit-learn(Pedregosa et al., 2011) library. Geometry-oriented Probes.For geometry- oriented probes, we introduce two instantiations: linear support vector machine (SVM) (Cortes and Vapnik, 1995) and mass-mean (M) (Marks and Tegmark, 2023) instantiation. The rationale for selecting linear SVM is its ability to maximize the margin, which aligns with the goal of identifying a separating hyperplane. As SVM does not directly provide probability predictions, we fit a post-hoc probability distribution using Platt scaling through cross-validation on the training data (Platt, 1999). Statistics-based Probes.For statistics-based probes, we present logistic regression (LR) and multi-layer perceptron (MLP). LR is commonly used as a baseline, while MLP is termed SAPLMA by Azaria and Mitchell (2023). 3.3 Data for Probing Truthfulness The binary probes introduced above are applied to the truthfulness classification task, assuming the availability of truthfulness-specific data. We use the factual statements curated by BĂźrger et al. (2024), drawing from datasets by Azaria and Mitchell (2023) and Marks and Tegmark (2023). These datasets cover a variety of topics, including animal_class,cities,element_symb,facts, inventors,sp_en_trans, as well as variations incorporating logical negations, conjunctions and disjunctions. Each statement is labeled as âtrueâ or âfalseâ, indicating its factuality. Statements can be atomic or compound. Atomic statements make individual claims, either affirmative or negative. Negative statements correspond to their affirma- tive counterparts, with syntax-level negation and inverted labels. Compound statements are created by logically combining atomic statements of the same topic through conjunction or disjunction. 4 Experiments 4.1 Preliminary Experiment: Layer Selection Identifying the optimal layer for detecting truthful- ness is crucial for probe performance. Marks and Tegmark (2023) observed that the truth direction âemerges rapidly in early-middle layersâ; however, this observation does not indicate which specific layer provides the most effective representations. To address this, we adopt the technique used by BĂźrger et al. (2024) and MacDiarmid et al. (2024), which evaluates the difficulty of separating true/- false statements across layers by analyzing vari- ance. The ideal layer maximizes the separation between true and false representations, quantified by the âbetween-class varianceâ, relative to the in- ternal variance within each class, referred to as âwithin-class varianceâ. By plotting the ratio of between-class to within-class variance across de- coder layers for a range of topic-specific datasets, we identify the optimal layer as the one with the highest ratio. 051015202530 Layer 0.00 0.05 0.10 0.15 0.20 0.25 Ratio 31 31 31 12 31 31 animal_class cities facts sp_en_trans inventors element_symb (a) Llama-2-7B 051015202530 Layer 0.0 0.1 0.2 0.3 0.4 0.5 Ratio 13 12 12 12 12 13 animal_class cities facts sp_en_trans inventors element_symb (b) Llama-3.1-8B Figure 2: Ratio of between-class variance to within- class variance across layers. The layer indices (starting from 0) for the greatest ratios are annotated at the sum- mit of each curve. The solid curves are mean values, and the surrounding shades denote standard error. We present the ratio of between-class variance to within-class variance across layers for Llama- 2-7B (Touvron et al., 2023) and Llama-3.1-8B (Dubey et al., 2024) in Figure 2, with results for additional models in the Appendix. Each curve represents statements involving affirmations, nega- tions, conjunctions and disjunctions of the same topic. For Llama-3.1-8B, the 12th layer (zero- indexed) emerges as the optimal layer. In con- trast, for Llama-2-7B, a peak occurs only for the sp_en_transtopic, with minimal separation ob- served for other topics. This suggests that, while Llama-2-7B may internally represent truthfulness as a feature, it does so in a domain-specific man- ner, with limited consistency across knowledge do- mains. Additionally, this feature appears to lack salience in the early-middle layers. 4.2 Probing a Randomized Model This section investigates whether the truth direction is an inherent structure within a pretrained LLM or an artifact constructed by the truthfulness probe. To address this, we randomly initialize the weights of Llama-3.1-8B, extract activations from its 12th layer using theanimal_classdataset, and train probes on a 70% split while testing them on the M0M1M2M3M4M5M6M7 animal_class cities inventors facts element_symb sp_en_trans 00939597999899 00297688837696 52827888798089 1117465762746781 0001026076 0000002650 LR M0M1M2M3M4M5M6M7 5040919596989799 5049567387878587 5353858591858291 4647526173747486 504801016081 0000003661 MLP M0M1M2M3M4M5M6M7 00969894999799 00677487908499 2511869291878994 1418536859747186 0001106477 0000003257 SVM M0M1M2M3M4M5M6M7 5050929180999699 5050846991919599 4740828390748794 4848566958717184 505003116077 0011103961 M 0 50 100 Figure 3: AUROC (in percentage) of probes trained on affirmative statements and tested on negative ones.AUROC> 0.5indicates success of generalization. M0-M7 refer to models from Llama-2-7B to Llama-3.1-70B-Instruct. remaining 30%. Results show that the AUROCs are 0.50, 0.52, 0.58, 0.50 for the LR, MLP, M and SVM probes, respectively. In contrast, when using the pretrained weights, the AUROCs achieve 1.0 across all probes. These findings demonstrate that the simple probes introduced in Section 3.2 cannot independently construct a truth direction, confirming that truth direction is a product of the pretraining process. 4.3 Consistency of Truth Directions Levinstein and Herrmann (2024) claim that probes such as MLP fail to generalize across negation. However, we question this conclusion and hypothe- size that the generalization performance of truthful- ness probes is influenced more by the targeted LLM than by the probe itself. To test this hypothesis, we examine whether the truth direction identified for affirmative statements is consistent with that identi- fied for negative statements of the same topic. 4.3.1 Experimental Setup Data.For each of the six topics introduced in Section 3.3, we train probes on affirmative state- ments and test them on corresponding negative ones. For example, we train probes on affirmative statements of theanimal_classtopic, and test them onneg_animal_class. Note that the train- ing and test data contain the same set of knowledge, differing only in syntax. Models.We select a series of LLMs with increas- ing levels of general capability, including both foun- dational models and instruction-tuned ones (Ac- cording to the evaluation results on a range of stan- dard benchmarks 2 ): Llama-2-7B(-Chat), Llama-2- 13B(-Chat), Llama-3.1-8B(-Instruct), Llama-3.1- 70B(-Instruct). Their optimal layers are 12(13), 13(13), 12(13), 33(33), respectively. Methods.We employ the four probe instantia- tions introduced in Section 3.2. 2 https://github.com/meta-llama/llama-models/ blob/main/models Metrics.The metric used is AUROC (Area Under Receiver Operating Characteristic Curve) which is commonly used to assess classifier perfor- mance. A probe is considered to generalize suc- cessfully on a topic if its test AUROC exceeds 0.5, indicating performance better than chance. Results are averaged across three trials, with randomness introduced through probe initialization and data splits for cross-validation of Platt scaling, while the training data remains constant. 4.3.2 Results The results are presented in Figure 3. We observe a positive correlation between the performance of truthfulness probes and the general capability of the target models. For Llama-2-7B(-Chat), the probes fail to generalize on all six topics. For Llama-2-13B(-Chat) and Llama-3.1-8B(-Instruct), the probes generalize on four topics; for Llama-3.1- 70B they generalize on five topics; and for Llama- 3.1-70B-Instruct, they generalize on all six topics. Regarding whether the truthfulness probe is a faithful reflection of the actual truth direction, we borrow theWeak-to-Strong Explanationfrom Zhou et al. (2024): if weak classifiers can successfully distinguish the representations, it indicates that LLMs have implicitly converted inputs to differ- ent representations. For the most capable model, Llama-3.1-70B-Instruct, all the simple probes we use are able to generalize across logical negation on all six topics. This suggests that Llama-3.1-70B- Instruct consistently represents truthfulness in its internals for both affirmative and negative state- ments. Therefore the results suggest a potential correlation between the degree of generalization of its truth direction and the modelâs capability (e.g., knowledge capacity and natural language under- standing ability). Additionally, the differences in performance between probes become negligible starting with Llama-2-13B-Chat and onward. This indicates that, for more capable LLMs, probe per- formance is more influenced by the target model itself than by the design of the probes. animal_class cities inventors facts element_symb sp_en_trans LR MLP SVM M 969980729194 939275729186 989986759397 979989749296 0 25 50 75 100 (a) Logical conjunctions. animal_class cities inventors facts element_symb sp_en_trans LR MLP SVM M 589075717658 589073697561 609085787259 599085746959 0 25 50 75 100 (b) Logical disjunctions. Figure 4: AUROC (in percentage) of probes trained on atomic factual statements and tested on logical conjunc- tions/disjunctions for Llama-3.1-8B.AUROC>0.5 indicates the success of generalization. 4.4 Binary Logical Transformation In Section 4.3, we tested the ability of truthfulness probes to generalize across logical negation. In this experiment, we extend the analysis to more com- plex binary logical transformations, specifically logical conjunction and disjunction. This requires the LLM to perform several implicit tasks: identify the truthfulness of both atomic statements, inter- pret binary logical operators from natural language to abstract concepts, and apply the operators to compute the joint truthfulness. 4.4.1 Experimental Setup Data.The training data consists of all atomic statements. The test data comprises logical con- junctions and disjunctions of atomic affirmative statements for each topic. The knowledge of the test data is covered by the training data. Models.We use Llama-3.1-8B as the primary model for demonstration. Methods.We apply the four probe instantiations introduced in Section 3.2. Metrics.For this experiment, we use AUROC as a measure of probe performance. A probe is consid- ered to generalize successfully on a topic if its test AUROC exceeds 0.5. Results are averaged across three trials, with randomness introduced through probe initialization and training data splits. 4.4.2 Results According to Figure 4, all probes successfully gen- eralize to both logical conjunctions and disjunc- tions. However, performance is notably stronger for logical conjunctions compared to disjunc- tions acrossanimal_class,element_symb, and sp_en_transtopics. This discrepancy may sug- gest that disjunctions pose a greater challenge for Llama-3.1-8B to interpret truthfulness. 4.5 Question Answering We hypothesize that if truthfulness is consistently represented in an LLMâs internal states, this rep- resentation should depend solely on the semantics of a sentence, rather than its syntax form. Ad- ditionally, question answering is more common than statements in real-world human-AI interac- tions. Motivated by these considerations, we exam- ine if truthfulness probes, trained on atomic factual statements, can generalize to the QA setting. We test on a multiple-choice task and a short- form QA task. We also investigate the in-context learning scenario, a popular prompting technique for teaching LLMs new tasks at inference time. While in-context examples can be beneficial, they may include incorrect or misleading examples, which raises questions about how probes handle false examples. Therefore, we pay special atten- tion to the behavior of the probes when incorrect examples are present. 4.5.1 Experimental Setup Data.The training data consists of all atomic statements.The test data includes MMLU (Hendrycks et al., 2020) and TriviaQA (Joshi et al., 2017). For MMLU, we sample 50 questions from the test set for each of the 57 sub-tasks. As it is a multiple-choice dataset, for each question we se- lect the correct answer and an incorrect answer. For TriviaQA, we sample 20 answers per question from the model at unit temperature. Models.We use Llama-3.1-8B, as it is a high- capability LLM withâź10B parameters. Methods.We apply the four probe instantiations introduced in Section 3.2. Metrics.In addition to classification accuracy, we evaluate calibration, as it is crucial for assessing the reliability of predictions. Specifically, we eval- uate AUROC, Expected Calibration Error (ECE) and Brier Score (BS). ECE measures calibration, while BS reflects both accuracy and calibration. Lower values are preferred for both ECE and BS. For ECE we use a binned approach with 10 bins, where each bin contains an equal number of sam- ples, and report the mean absolute error between the accuracy and confidence within each bin. The random baseline for BS is 0.25, corresponding to a uniform prediction of 0.5. Results are averaged across three trials, with randomness introduced via probe initialization and training data splits. Prompt setups.We test three prompt settings for MMLU: (1) âzero-shotâ: a zero-shot prompt; (2) âTâ: a five-shot prompt with all correct exemplars; (3) âTTFFFâ: a five-shot prompt where the first two examples are correct and the following three are incorrect. For TriviaQA, we test 5-shot and 20-shot prompts. Although it is true that the correctness of few- shot examples could be verified in practice, our motivation to study incorrect in-context examples includes: (1) It helps understand how truthfulness probes handle context that contains mixed truthful and untruthful information; (2) It provides insights into how robust truthfulness probes are to poten- tially conflicting information. 4.5.2 Results 0.5 0.6 0.7 0.8 AUROC 0.68 0.66 0.70 0.70 0.69 0.63 0.72 0.72 0.69 0.63 0.72 0.72 0-shotTTTTTTTFFF 0.0 0.2 0.4 ECE 0.17 0.42 0.14 0.04 0.29 0.45 0.15 0.07 0.29 0.45 0.15 0.06 LRMLPMMSVM Probe 0.0 0.2 0.4 BS 0.26 0.44 0.24 0.22 0.32 0.46 0.24 0.22 0.32 0.45 0.24 0.22 (a) MMLU. 0.5 0.6 0.7 0.8 AUROC 0.694 0.689 0.754 0.763 0.701 0.693 0.752 0.763 5-shot20-shot 0.0 0.2 0.4 ECE 0.158 0.328 0.148 0.133 0.174 0.338 0.153 0.114 LRMLPMMSVM Probe 0.0 0.2 0.4 BS 0.252 0.347 0.230 0.216 0.255 0.353 0.231 0.210 (b) TriviaQA. Figure 5: AUROCâ/ECEâ/BSâof truthfulness probes for Llama-3.1-8B on MMLU and TriviaQA. The dashed gray line corresponds to random results, and error bars denote standard error. The results, shown in Figure 5, reveal that across all probes, accuracy generally improves when few- shot exemplars are provided. This suggests that providing more task-related context in the prompt typically aids the LLM in the implicit truthfulness classification task, and that truthfulness probes not only generalize from factual statements to both multiple-choice QA and short-form QA, but also generalize from fundamental commonsense knowl- Figure 6: AUROCâ/ECEâ/BSâof in-domain probes for Llama-3.1-8B trained and tested on MMLU. edge to both expert knowledge (MMLU) and trivia knowledge (TriviaQA). Notably, according to the results on MMLU, the effect of imperfect few-shot prompts is near identical to entirely correct few- shot prompts. This indicates that the performance of truthfulness probes is not significantly influ- enced by the truthfulness of the few-shot examples â the probes express the truthfulness of thefinal (Q,A)pair, treating prior exemplars as context. In- terestingly, this finding aligns with Halawi et al. (2023)âs observation that early layers of the LLM are insensitive to false in-context demonstrations. However, few-shot prompting does not always improve calibration, and a positive case is only ob- served for the SVM probe on the TriviaQA dataset. Among all probes, the SVM probe performs best in terms of both classification accuracy and cali- bration, likely because of its ability to accurately identify the geometry of the truth direction and the benefits of the additional Platt scaling procedure. 4.5.3 Additional Experiments Beyond the generalization of probes trained on fac- tual statements, we are interested in how these probes differ from in-domain probes, which are trained and tested on data of the same domain. Therefore we perform follow-up experiments by training and testing probes on MMLU using the Llama-3.1-8B model, using the same data and prompt setup as above. Specifically, we train on a random 70% split and test on the rest 30%. We present results averaged over three random trials in Figure 6. Comparing the results with those of Figure 5, we observe that in-domain probes gen- erally under-performs truthfulness probes trained on atomic factual statements in terms of AUROC. This finding indicates that truthfulness probes un- der the current setup perform better than those un- der the in-domain QA setup. 4.6 Contextual Knowledge When training truthfulness probes using factual statements, we are targeting the factual aspect of truthfulness, where the grounding knowledge resides in the LLMâs parameters. However, in- context knowledge also plays a vital role in genera- tion. In this section, we investigate if truthfulness probes can also capture this additional aspect of truthfulness, where the grounding knowledge is provided as contextual information in the prompt. Notably, context grounding is fundamentally dif- ferent from factual correctness, since faithfully fol- lowing false context is acceptable for the former but not for the latter. We conduct experiments on two tasks: in-context QA and abstractive summa- rization. 4.6.1 Experimental Setup Data.The training data consists of all the atomic statements. For the in-context QA task we use the SciQ (Welbl et al., 2017) and BoolQ (Clark et al., 2019) datasets. For SciQ, we randomly select 1000 questions, pairing each question with the lettered choice for both the true and a randomly selected false answer. The answers of BoolQ are binary âyes/noâ, therefore we flip the answers to balance true and false(Q,A)pairs. For the abstractive sum- marization task, we use the XSum dataset (Narayan et al., 2018) and XSum Hallucination Annotations (Maynez et al., 2020). For each article, we pair true summaries from the former dataset with false ones from the latter one. Models.We use Llama-3.1-8B. Methods.We apply the four probe instantiations introduced in Section 3.2. Metrics.We report AUROC, ECE and BS. The results are averaged over three trials, with random- ness introduced via initialization and data splits. Prompt setups.For SciQ dataset which is a multiple-choice task, we implement four settings: (1) âzero-shotâ: zero-shot prompt; (2) âTâ: three-shot prompt where all exemplars are correct; (3) âTTFâ: three-shot prompt, where the first two examples are correct and the third is incorrect; (4) âFFTâ: three-shot prompt, where the first two ex- amples are incorrect and the third is incorrect. For BoolQ, we implement four settings: âno optionsâ, âwith optionsâ, and one-shot âTâ/âFâ (with possible options). For XSum, no options are provided in the prompt as it is not a multiple-choice task. We implement the following prompt configurations: zero-shot, âTâ, âTâ, and âTâ. 4.6.2 Results The results are shown in Figure 7. Across all datasets and most probes, possible answer options and few-shot exemplars generally improve classi- fication accuracy. This indicates that truthfulness probes generalize from factual statements to both in-context multiple-choice QA tasks and abstrac- tive summarization tasks. Among the probes, statistics-based probes (LR and MLP) display greater standard error in terms of all three metrics than geometric-oriented ones (M and SVM), likely due to their optimization instability, and the SVM probe performs best from the perspective of BS. Additionally, discrepancies are observed for LR probe on the SciQ and BoolQ datasets, MLP probe on the BoolQ dataset, and M probe on the XSum dataset, where accuracy improves in response to in-context exemplars but calibration worsens. We assume that these discrep- ancies could be explained with the overconfidence of the probesâ predictions. 4.7 Selective Question Answering Based on the findings of Section 4.5, we observe that truthfulness probes trained on atomic state- ments are capable of generating calibrated proba- bilistic predictions for QA tasks while achieving reasonable accuracy. Building upon these observa- tions, this section investigates whether truthfulness probes can be leveraged to selectively identify cor- rect answers from a set of candidate responses sam- pled from an LLM. This setup is inspired by the work of Kadavath et al. (2022), who demonstrated that an LLM can evaluate the correctness of its own answers through verbal feedback. While our selective QA experiment shares similarities with Kadavath et al. (2022), our approach differs fun- damentally as we leverage internal representations rather than model-generated feedback. For this experiment, we use the TriviaQA test data from Section 4.5, where we sample 20 answers from the Llama-3.1-8B model using a 20-shot prompt with unit temperature. To perform selective QA, we select the subset of(Q,A)pairs for which the truthfulness probe predictsP[ÎŚ = 1]>0.5 and report the accuracy on this subset. We use the SVM probe, as it performs best in terms of both classification accuracy and calibration. The aggregated accuracy across all sampled 0.5 0.6 0.7 0.8 0.9 1.0 AUROC SciQBoolQXSum 0.0 0.1 0.2 0.3 0.4 ECE LRMLPMMSVM Probe 0.0 0.1 0.2 0.3 0.4 BS 0-shotTTTTTFFFT LRMLPMMSVM Probe no opt.w/ opt.TF LRMLPMMSVM Probe 0-shotTTTTTT Figure 7: AUROCâ/ECEâ/BSâof truthfulness probes for Llama-3.1-8B on tasks where grounding knowledge is provided in the prompt. The dashed gray line corresponds to random results, and error bars denote standard error. (Q,A)pairs is 55.29%. Among these, the truth- fulness probe classifies 80.26% as true, and the accuracy of this subset is 64.06%. This demon- strates that truthfulness probes can be used to filter out false answers sampled from LLMs. 5 Conclusions In this work, we provide preliminary evidence sup- porting our hypothesis that consistent truth direc- tions only emerge in capable LLMs and not in weaker ones, and they could be effectively identi- fied with simple linear probes. We also investigate the generalization properties of truth directions. Empirical results show that truthfulness probes trained only on atomic statements generalize well to logical transformations, (few-shot) question an- swering and contextual truthfulness. These findings underscore the potential of truthfulness probes to identify truth directions using simple anchor data, thereby facilitating the elicitation of latent knowl- edge within LLMs. 6 Limitations This study has several limitations that warrant dis- cussion. First, the term "truth" as used in this pa- per represents an idealized concept, but it may not be what the truthfulness probes actually measure. Drawing on Kadavath et al. (2022) and Marks and Tegmark (2023), the pretraining of language mod- els largely involves imitating human-generated text. Consequently, truthfulness probes are likely cap- turing an overlap between widely accepted human beliefs and factual, objective truths about the phys- ical world. This raises interesting questions about how such probes might perform in the context of scalable oversight, particularly with hypothetical AI systems that surpass human intelligence. Second, our investigation into the generalization of truthfulness probes is limited to short-form QA. Extending this analysis to more complex scenar- ios, such as long-form QA or instruction-following tasks, may yield novel insights and uncover more practical applications of truthfulness probes. Third, the causality of truthfulness probes is un- clear. Our approach relies on classic classification techniques, consistent with prior work on prob- ing truthfulness. Meanwhile, we do not discuss the causal effects of truth directions, i.e., whether LLMs utilize the implicit truthfulness classifica- tion results for predictions. Marks and Tegmark (2023) conducted causal intervention experiments using the mass-mean probe and showed that mass- mean directions are highly causal. However, as highlighted by Kumar et al. (2022), probes often capture spuriously-correlated features rather than exclusively isolating the target feature. Future re- search on the causal implications of truth directions could provide valuable insights into guiding LLMs to produce more truthful responses, as in Li et al. (2023). Finally, our experiments are constrained by com- putational resources, with the largest model eval- uated being Llama-3.1-70B. As a result, our hy- pothesis that highly capable LLMs will eventually establish a consistent internal concept of truthful- ness remains untested on more advanced models such as GPT-4 (Achiam et al., 2023). Acknowledgment This work was partly supported by the National Key Research and Development Program of China under No. 2024YFB3900105, NSFC under No. 62402418, Zhejiang Provinceâs 2025 âLeading Goose + Xâ Science and Technology Plan under grant No.2025C02034, the Key R&D Program of Ningbo under No. 2024Z115, and the Open Project of Key Laboratory of General Quality Technol- ogy and Application of Intelligent Manufacturing Equipment, Ministry of Industry and Information Technology (HK202403532). References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when itâs lying.arXiv preprint arXiv:2304.13734. Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency, pages 610â623. Lennart BĂźrger, Fred A Hamprecht, and Boaz Nadler. 2024. Truth is universal: Robust detection of lies in llms.arXiv preprint arXiv:2407.12831. Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt. 2022. Discovering latent knowledge in lan- guage models without supervision.arXiv preprint arXiv:2212.03827. Paul Christiano,Ajeya Cotra,and Mark Xu. 2021.Eliciting latent knowledge: How to tell if your eyes deceive you.URL https://docs. google.com/document/d/1WwsnJQstPq91_Yh- Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit# heading= h. jrzi4atzacns. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924â2936, Minneapolis, Min- nesota. Association for Computational Linguistics. Corinna Cortes and Vladimir Vapnik. 1995. Support- vector networks.Machine learning, 20(3):273â297. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Owain Evans, Owen Cotton-Barratt, Lukas Finnve- den, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful ai: Developing and governing ai that does not lie.arXiv preprint arXiv:2110.06674. Danny Halawi, Jean-Stanislas Denain, and Jacob Stein- hardt. 2023. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint arXiv:2307.09476. Jinwen He, Yujia Gong, Zijin Lin, Yue Zhao, Kai Chen, et al. 2024. Llm factoscope: Uncovering llmsâ fac- tual discernment through measuring inner states. In Findings of the Association for Computational Lin- guistics ACL 2024, pages 10218â10230. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation.ACM Comput- ing Surveys, 55(12):1â38. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b.arXiv preprint arXiv:2310.06825. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601â1611. Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. 2024. Personas as a way to model truthfulness in language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6346â6359. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221. Diederik P Kingma. 2014. Adam: A method for stochas- tic optimization.arXiv preprint arXiv:1412.6980. Abhinav Kumar, Chenhao Tan, and Amit Sharma. 2022. Probing classifiers are unreliable for concept removal and detection.Advances in Neural Information Pro- cessing Systems, 35:17994â18008. Bruce W Lee, Benedict Florance Arockiaraj, and Helen Jin. 2023. Linguistic properties of truthful response. InProceedings of the 3rd Workshop on Trustwor- thy Natural Language Processing (TrustNLP 2023), pages 135â140. Benjamin A Levinstein and Daniel A Herrmann. 2024. Still no lie detector for language models: Probing empirical and conceptual roadblocks.Philosophical Studies, pages 1â27. Kenneth Li, Oam Patel, Fernanda ViĂŠgas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference- time intervention: Eliciting truthful answers from a language model.Advances in Neural Information Processing Systems, 36:41451â41530. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214â3252. Dong C Liu and Jorge Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503â528. Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. 2023. Cognitive dissonance: Why do language model outputs disagree with internal rep- resentations of truthfulness? InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4791â4797. Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duvenaud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. 2024. Simple probes can catch sleeper agents. MatĂŠo Mahaut, Laura Aina, Paula Czarnowska, Mom- chil Hardalov, Thomas MĂźller, and Lluis Marquez. 2024. Factual confidence of LLMs: on reliability and robustness of current estimators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4554â4570, Bangkok, Thailand. Association for Computational Linguistics. Alex Mallen and Nora Belrose. 2023. Eliciting latent knowledge from quirky language models.arXiv preprint arXiv:2312.01037. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906â1919, On- line. Association for Computational Linguistics. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Donât give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization.ArXiv, abs/1808.08745. Lorenzo Pacchiardi, Alex James Chan, SĂśren Minder- mann, Ilan Moscovitz, Alexa Yue Pan, Yarin Gal, Owain Evans, and Jan M Brauner. 2023. How to catch an ai liar: Lie detection in black-box llms by asking unrelated questions. InThe Twelfth Interna- tional Conference on Learning Representations. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011.Scikit-learn: Machine learning in Python.Journal of Machine Learning Research, 12:2825â2830. J Platt. 1999. Probabilistic outputs for svms and com- parisons to regularized likehood methods.Advances in Large Margin Classifiers. CH-Wang Sky, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2024. Do androids know theyâre only dreaming of electric sheep? InFindings of the Association for Computational Linguistics ACL 2024, pages 4401â4420. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288. A Vaswani. 2017. Attention is all you need.Advances in Neural Information Processing Systems. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. How align- ment and jailbreak work: Explain LLM safety through intermediate hidden states. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 2461â2488, Miami, Florida, USA. Asso- ciation for Computational Linguistics. A Truth-related concepts In this section, we distinguish the term âtruthful- nessâ from several related concepts. Truthfulness refers to the alignment of a statement or(Q,A) with either world knowledge or contextual sources. The former, following Mahaut et al. (2024), is termed âfactualityâ. Contextual truthfulness, by contrast, may include fictional information that de- viates from real-world facts, such as solving math problems in a hypothetical scenario. Untruth lies at the negative end of the truth- fulness spectrum and differs fromhallucination, which refers to generations that are nonsensical or unfaithful to the provided source content (Ji et al., 2023). A key distinction between untruth and hal- lucination is that truthfulness requires a sentence to be both sensical and unambiguous. Additionally, we differentiate untruth fromlies. According to Pacchiardi et al. (2023), an answer is considered a lie only if the speaker knows the correct answer. In this view, a lie is a subset of untruths. B Explanation on Choice of Probes In this work we summarize LR, MLP, SVM and M instantiations and use them for experiments. We do not use the TTPD probe introduced by the recent work, BĂźrger et al. (2024), for two reasons. First, the design of the TTPD probe is based on their finding that the âaffirmative truth directionâ and the âgeneral truth directionâ are not aligned. However, according to our findings in Section 4.3, the âaffirmative truth directionâ and the âgeneral truth directionâ become more consistent as the tar- get modelâs general capability increases. There- fore, for models with relatively high capability, the TTPD probe is not expected to distinguish itself among other probe instantiations. Second, we find empirical evidence that the TTPD probe is not the most effective probe. We replicate the experiment on the BoolQ dataset in Section 4.6, and plot the output distribution and cal- ibration curves of all probes targeting Llama-3.1- 8B under the âwith optionsâ setting. The results are shown in Figure 9 and Figure 8. The output distribution of the TTPD probe resembles that of the M probe, so does the calibration curve. This observation is not restricted to this setting and the BoolQ test set but is also noticed in a number of other settings and test sets. These findings sug- gest that judging by the functional behaviors of the TTPD probe, its performance is not representative enough to be reported. C Details of Factual Datasets We mentioned in Section 3.3 about the factual statements covering six topics which are used for training and testing truthfulness probes. We sum- marize each topic-specific dataset in Table 1, ac- cording to the data curators Azaria and Mitchell (2023), Marks and Tegmark (2023) and BĂźrger et al. (2024). Instead of directly using the statements from BĂźrger et al. (2024), we perform minor modifica- tions on theinventorstopic and on logical dis- junctions. We notice that the originalinventors 0.00.20.40.60.81.0 Probabilities 0.0 0.2 0.4 0.6 0.8 1.0 Frequencies lr mlp svm m ttpd Figure 8: Calibration graph of LR, MLP, SVM, M and TTPD probes on the BoolQ dataset under the âwith optionsâ setting. The target LLM is Llama-3.1-8B. dataset has potential ambiguity due to duplication of name. Thus we specify that the person men- tioned in a statement was an inventor, e.g. from âThomas Edison lived in the U.S.â to âThe inven- tor Thomas Edison lived in the U.S.â. Another tweak is that the original logical disjunctions com- posed by BĂźrger et al. (2024) are not consistent with conjunctions, as the subjects are written in full for conjunctions while the subjects are written in pronouns for disjunctions. To align these two logical transformations, we recover the subjects for disjunctions. D Probe Implementation Details Our implementation of the LR, SVM and MLP probes is based on thescikit-learn(Pedregosa et al., 2011) library. For the LR probe, we em- ploy the L-BFGS optimization algorithm (Liu and Nocedal, 1989). For SVM, we utilize theNuSVC implementation. We setν= 0.5, a choice later validated by experiment results. Platt scaling is applied using five-fold cross-validation with the help of thescikit-learnlibrary. For the MLP probe we configure a decreasing sequence of hid- den units (512,128,64) withtanhactivation, and we use the Adam optimizer (Kingma, 2014) to train it till convergence. Finally, we use the M probe implementation provided by Marks and Tegmark (2023). When establishing probes on atomic statements, we use a random 70% split for training and hold out the rest as the development set. 0.00.20.40.60.81.0 Probability 0 200 400 600 800 Count LR False True 0.00.20.40.60.81.0 Probability 0 500 1000 1500 2000 2500 3000 MLP False True 0.00.20.40.60.81.0 Probability 0 50 100 150 200 250 300 350 400 SVM False True 0.00.20.40.60.81.0 Probability 0 100 200 300 400 500 600 700 800 M False True 0.00.20.40.60.81.0 Probability 0 100 200 300 400 500 600 700 TTPD False True Figure 9: Output distributions of LR, MLP, SVM, M and TTPD probes on the BoolQ dataset under the âwith optionsâ setting. The target LLM is Llama-3.1-8B. TopicDescriptionExample statement animal_classThe class of a specific animal species.The salmon is a fish. citiesLocations of world cities.The city of Krasnodar is in Russia. element_symb Chemical elements and their abbreviations. Thallium has the symbol Tl. factsDiverse scientific facts. The Earthâs atmosphere protects us from harmful radiation from the sun. inventorsHome countries of inventors. The inventor Edwin Herbert Hall lived in the U.S. sp_en_transTranslations of Spanish words to English. The Spanish word âconâ means âto speakâ. Table 1: Summary of topic-specific factual statement datasets. E Metric Details E.1 Expected Calibration Error (ECE) For ECE, we first sort the probabilities predicted by a probe and split them intoNequal-sized bins. In this paper we letN= 10, which is common in literature evaluating calibration. For each bin, we calculate the mean probability (x i ) and the fraction of truthful predictions (y i ). ECE is then computed following the formula below: ECE= 1 N N X i=1 |y i âx i |.(1) E.2 Brier Score (BS) The Brier Score measures the difference between the actual correctness and the confidence score through point-wise mean squared error. Its for- mulation is as follows: Brier Score= 1 N N X i=1 (p i ây i ) 2 ,(2) wherep i is the confidence reported by the probe andy i â 0,1is the ground truth label. When a predictor is always making inconfident random predictions, i.e.p i = 0.5(i= 1,2,...N), it results in a chance Brier score of 0.25. F Experiment Details and More Results In this section, we elaborate on the detailed setups of the experiments in Section 4. Furthermore, as we only use the Llama family of models in the body of the paper, in this section we also demonstrate results on a model of the Mistral family, Mistral- 7B-v0.1 (Jiang et al., 2023). F.1 Computational and Storage Resources All of our experiments are completed on three A6000 (48GB) GPUs. For most LLMs except Llama-3.1-70B(-Instruct), only one GPU will suf- fice. However, the storage for activations across all datasets and models would takeâź1TB disk space. Therefore we recommend modifying the code and only keep the necessary activations on disk. Fur- thermore, in order to accommodate large LLMs such as Llama-3.1-70B(-Instruct) into our GPUs when gathering hidden activations, we usefloat8 quantization with theoptimum-quanto 3 library. 3 https://github.com/huggingface/ optimum-quanto F.2 Selecting Layer In Section 4.1 we discuss the selection of decoder layer residual stream to extract truth direction from. We present a criteria based on the ratio of between- class variance to within-class variance. However, due to limitation of space we only present results for Llama-2-7B and Llama-3.1-8B in Section 4.1. Here we replicate these approaches on more mod- els. The data used for plotting is the collection of both affirmative and negative atomic statements covering all the six topics, as well as their logical conjunctions and disjunctions. The curve for each topic consists of the four variations of statements, which results in six curves for each model. We summarize the plots in Figure 10, which covers eight models: Llama-2-7B(-Chat), Llama- 2-13B(-Chat), Llama-3.1-8B(-Instruct), Llama-3.1- 70B(-Instruct), Mistral-7B(-Instruct)-v0.1. Their optimal layers are 12(13), 13(13), 12(13), 33(33), 13(13), respectively. F.3 Consistency of Truth Direction We present results on Mistral-7B-v0.1 in Figure 11. It is evident that the truthfulness probes generalize across negation on four topics, barely generaliz- ing onfactstopic. Comparing these results with those of Figure 3, we notice that the performance of probes for Mistral-7B-v0.1 is comparable to that of probes for Llama-2-13B-Chat and Llama-3.1-8B. This aligns with the observation that the general capability of Mistral-7B-v0.1 lies between that of Llama-2-13B-Chat and Llama-3.1-8B. F.4 Logical Conjunction/Disjunction Full results on logical conjunctions and disjunc- tions are shown in Figure 12 and Figure 13 respec- tively. A similar scaling trend could be observed as in Figure 3, where the classification accuracy of the probes is positively correlated with the target LLMâs general capability. The results for Mistral-7B-v0.1 is shown in Fig- ure 14. The truthfulness probes generalize from atomic factual statements to both logical conjunc- tions and disjunctions. F.5 Question Answering F.5.1 Details on Experiment Setup MMLU.We arrange three setups for the QA task, and we demonstrate the prompt template for zero-shot setting using an actual example from the MMLU dataset. The few-shot prompts are trivially extended from the zero-shot prompt, with exem- plars separated by two newlines (" "). Few- shot exemplars are randomly selected from the de- velopment split. Question: What was GDP per capita in the United States in 1850 when adjusting for inflation and P in 2011 prices? Options: A. About $300 B. About $3k C. About $8k D. About $15k Answer: B TriviaQA.For TriviaQA (Joshi et al., 2017) we only use few-shot prompting â 5-shot and 20-shot â to ensure that the LLM always generates short- form answers. We use normalized answers in the exemplars. Question: Where in England was Dame Judi Dench born? Answer: york F.5.2 More results The results for Mistral-7B-v0.1 is shown in Figure 15. The general behavior of truthfulness probes is similar to that in Figure 5. The classification accu- racy improves in response to few-shot prompting and improves when more in-context exemplars are provided in the prompt. Meanwhile, calibration only improves in the case of the SVM probe on TriviaQA dataset. F.6 Contextual Knowledge F.6.1 Details on Experiment Setup SciQ.For the SciQ (Welbl et al., 2017) bench- mark, we arrange three setups, including âzero- shotâ, âTâ and âTTFâ. We only demonstrate the zero-shot prompt as the few-shot prompts can be trivially extended from it. In the in-context se- tups, the exemplars are randomly selected from the training split. Context: <context > Question: Compounds that are capable of accepting electrons , such as o 2 or f2 , are called what? Options: A. Oxygen B. residues C. antioxidants D. oxidants Answer: D 051015202530 Layer 0.00 0.05 0.10 0.15 0.20 0.25 Ratio 31 31 31 12 31 31 animal_class cities facts sp_en_trans inventors element_symb (a) Llama-2-7B 051015202530 Layer 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Ratio 31 31 31 12 31 31 animal_class cities facts sp_en_trans inventors element_symb (b) Llama-2-7B-Chat 0510152025303540 Layer 0.0 0.1 0.2 0.3 0.4 0.5 Ratio 19 15 15 14 14 15 animal_class cities facts sp_en_trans inventors element_symb (c) Llama-2-13B 0510152025303540 Layer 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Ratio 18 15 15 14 13 15 animal_class cities facts sp_en_trans inventors element_symb (d) Llama-2-13B-Chat 051015202530 Layer 0.0 0.1 0.2 0.3 0.4 0.5 Ratio 13 12 12 12 12 13 animal_class cities facts sp_en_trans inventors element_symb (e) Llama-3.1-8B 051015202530 Layer 0.0 0.2 0.4 0.6 0.8 Ratio 19 12 15 12 12 16 animal_class cities facts sp_en_trans inventors element_symb (f) Llama-3.1-8B-Instruct 01020304050607080 Layer 0.0 0.2 0.4 0.6 0.8 Ratio 35 17 33 18 21 33 animal_class cities facts sp_en_trans inventors element_symb (g) Llama-3.1-70B 01020304050607080 Layer 0.0 0.5 1.0 1.5 2.0 Ratio 36 37 33 40 30 36 animal_class cities facts sp_en_trans inventors element_symb (h) Llama-3.1-70B-Instruct 051015202530 Layer 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Ratio 16 14 14 13 14 16 animal_class cities facts sp_en_trans inventors element_symb (i) Mistral-7B-v0.1 051015202530 Layer 0.0 0.1 0.2 0.3 0.4 0.5 Ratio 16 16 15 13 14 16 animal_class cities facts sp_en_trans inventors element_symb (j) Mistral-7B-Instruct-v0.1 Figure 10: Plot of the ratio of between-class variance to within-class variance for a series of models. The shaded regions denote standard error. M0 animal_class cities inventors facts element_symb sp_en_trans 98 92 91 53 0 0 LR M0 95 96 88 58 0 0 MLP M0 95 91 92 53 0 0 SVM M0 77 93 74 57 0 0 M 0 50 100 Figure 11: AUROC (in percentage) of probes trained on affirmative statements and tested on negative ones. AUROC exceeding 0.5 indicates generalization success. M0 refers to the Mistral-7B-v0.1 model. BoolQ.For the BoolQ (Clark et al., 2019) bench- mark, we arrange four setups, including âno op- tionsâ, âwith optionsâ, âTâ and âFâ. We only demonstrate the prompt for âwith optionsâ. In the in-context setups, the exemplars are randomly se- lected from the training split. Passage: <passage > Question: does ethanol take more energy make that produces? Options: - Yes - No Answer: No XSum.For this task, we arrange four setups, in- cluding âzero-shotâ, âTâ, âTâ and âTâ. We only demonstrate the zero-shot prompt as the one- shot and few-shot prompts can be trivially extended from it. In the in-context setups, the exemplars are randomly selected from the training split of XSum (Narayan et al., 2018) dataset, and they are all deemed correct. False examples come from the XSum Hallucination Annotations (Maynez et al., 2020) dataset with 500 examples, which is paired with examples from the test split of XSum. Fur- thermore, we filter for examples no longer than the LLMâs context window. For Llama-3.1-8B with a context length of8192, we obtain the final test set of 998 examples whose labels are balanced. Summarize this document: <doc > Summary: Rory McIlroy moved to within a shot of joint leaders Victor Dubuisson and Jaco van Zyl after the third round of the Turkish Airlines Open. F.6.2 More results We present results for Mistral-7B-v0.1 in Figure 16. Accuracy improves as in-context exemplars are provided, but calibration only displays the same trend on SciQ dataset. Another abnormality could be observed for the BoolQ dataset from âno op- tionsâ setting to âwith optionsâ setting, and for the XSum task from zero-shot to one-shot. In these cases, both accuracy and calibration wors- ens, which does not align with the results in Figure 7. We assume this is attributed to the weakness of the target model, where Mistral-7B-v0.1 finds it difficult to interpret answer options and in-context exemplars for the abstractive summarization task. G License The implementation of the probes is based on thescikit-learn(Pedregosa et al., 2011) library, which is licensed under BSD 3-Clause License. The factual statements we use is curated by BĂźrger et al. (2024), licensed under MIT License. The MMLU (Hendrycks et al., 2020) dataset is licensed under MIT License, the TriviaQA (Joshi et al., 2017) dataset is licensed under Apache 2.0 License, the SciQ (Welbl et al., 2017) dataset under Cre- ative Commons Attribution-NonCommercial 3.0 Unported License, the BoolQ (Clark et al., 2019) under Creative Commons Share-Alike 3.0 License, the XSum (Narayan et al., 2018) dataset under MIT License and the XSum Hallucination Annotations (Maynez et al., 2020) dataset under Creative Com- mons Attribution 4.0 International License. Llama- 2 series of models are licensed under Llama 2 Com- munity License Agreement, Llama-3 herd of mod- els are licensed under Llama 3 Community License Agreement and Mistral models are licensed under MIT License. animal_class cities inventors facts element_symb sp_en_trans Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-3.1-8B Llama-3.1-8B-Instruct Llama-3.1-70B Llama-3.1-70B-Instruct 808678768589 818682799293 969981769094 979981719389 969980729194 9910081829798 9910085849399 100100909310099 LR animal_class cities inventors facts element_symb sp_en_trans 464773495192 505079505090 969983799395 989984759691 939275729186 989978799697 99100828292100 1001009194100100 MLP animal_class cities inventors facts element_symb sp_en_trans 838678799390 838683829694 969984779393 9810082789590 989986759397 9810086859899 99100818196100 100100899310099 SVM animal_class cities inventors facts element_symb sp_en_trans 505075505087 505074505094 959979759189 9710083799691 979989749296 989984849899 989977799699 100100889210098 M 0 20 40 60 80 100 Figure 12: AUROC (in percentage) of probes trained on all the atomic factual statements and tested on logical conjunctions. AUROC>0.5indicates the success of generalization. animal_class cities inventors facts element_symb sp_en_trans Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-3.1-8B Llama-3.1-8B-Instruct Llama-3.1-70B Llama-3.1-70B-Instruct 504164706066 452464674146 658875686165 638471646361 589075717658 648577706357 758762695876 768976697792 LR animal_class cities inventors facts element_symb sp_en_trans 484863504958 495268525057 658677696463 658572646258 589073697561 658572646447 778965645979 769079698488 MLP animal_class cities inventors facts element_symb sp_en_trans 514463707366 452766705346 648476715760 668274676160 609085787259 628578756360 768971666575 768875678390 SVM animal_class cities inventors facts element_symb sp_en_trans 505067505061 505062495055 627571675055 668276676162 599085746959 618575756360 738770615566 758672677986 M 0 20 40 60 80 100 Figure 13: AUROC (in percentage) of probes trained on all the atomic factual statements and tested on logical disjunctions. AUROC>0.5indicates the success of generalization. animal_class cities inventors facts element_symb sp_en_trans LR MLP SVM M 9810084779188 979981728386 9810083789488 9810081779584 0 25 50 75 100 (a) Logical conjunctions. animal_class cities inventors facts element_symb sp_en_trans LR MLP SVM M 667979696068 656669635871 678178716067 677974715966 0 25 50 75 100 (b) Logical disjunctions. Figure 14: AUROC (in percentage) of probes trained on atomic factual statements and tested on logical conjunc- tions/disjunctions for Mistral-7B-v0.1.AUROC>0.5 indicates the success of generalization. 0.5 0.6 0.7 0.8 AUROC 0.68 0.64 0.72 0.72 0.69 0.66 0.72 0.73 0.69 0.66 0.73 0.73 0-shotTTTTTTTFFF 0.0 0.2 0.4 ECE 0.22 0.36 0.14 0.33 0.26 0.34 0.14 0.35 0.26 0.34 0.14 0.35 LRMLPMMSVM Probe 0.0 0.2 0.4 BS 0.27 0.38 0.24 0.34 0.29 0.36 0.24 0.34 0.29 0.36 0.24 0.34 (a) MMLU. 0.5 0.6 0.7 0.8 AUROC 0.694 0.689 0.754 0.763 0.701 0.693 0.752 0.763 5-shot20-shot 0.0 0.2 0.4 ECE 0.158 0.328 0.148 0.133 0.174 0.338 0.153 0.114 LRMLPMMSVM Probe 0.0 0.2 0.4 BS 0.252 0.347 0.230 0.216 0.255 0.353 0.231 0.210 (b) TriviaQA. Figure 15: AUROCâ/ECEâ/BSâof truthfulness probes for Mistral-7B-v0.1 on MMLU and TriviaQA. The dashed gray line corresponds to random results, and error bars denote standard error. 0.5 0.6 0.7 0.8 0.9 1.0 AUROC SciQBoolQXSum 0.0 0.1 0.2 0.3 0.4 ECE LRMLPMMSVM Probe 0.0 0.1 0.2 0.3 0.4 BS 0-shotTTTTTFFFT LRMLPMMSVM Probe no opt.w/ opt.TF LRMLPMMSVM Probe 0-shotTTTTTT Figure 16: AUROCâ/ECEâ/BSâof truthfulness probes for Mistral-7B-v0.1 on tasks where grounding knowledge is provided in the prompt. The dashed gray line corresponds to random results, and error bars denote standard error.