Paper deep dive
NeuroActiSep: Detecting Factual Hallucinations from Feed-Forward Neurons in a Single Pass
Ali Derogar Odolou, Reza Nazari, Mostafa Salehi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Hallucination in large language models reduces their reliability and slows adoption. Various white-box studies have used internal representations to detect patterns of truthfulness and factuality. A less-studied approach is to identify feed-forward neurons correlated with hallucination. We propose a method to rank feed-forward neurons at the final prompt token using a custom neuron selection dataset. We transfer the selected neuron identities to train hallucination classifiers on other factual question answering datasets. Our work provides empirical evidence that probes trained using the features from the selected neurons perform on par with probes trained on internal states. We also analyze the distribution of selected neurons and the effect of layer depth on detection performance.
Tags
Links
- Source: https://arxiv.org/abs/2609.14448v1
- Canonical: https://arxiv.org/abs/2609.14448v1
Trouble viewing inline? Open PDF directly →
Full Text
32,664 characters extracted from source content.
Expand or collapse full text
NeuroActiSep: Detecting Factual Hallucinations from Feed-Forward Neurons in a Single Pass Ali Derogar Odolou, Reza Nazari, Mostafa Salehi Affiliation: School of Intelligent Systems Engineering College of Interdisciplinary Science and Technology University of Tehran, Tehran, Iran Email: ali.derogar, nazari.reza, mostafa_salehi@ut.ac.ir Abstract Hallucination in large language models reduces their reliability and slows adoption. Various white-box studies have used internal representations to detect patterns of truthfulness and factuality. A less-studied approach is to identify feed-forward neurons correlated with hallucination. We propose a method to rank feed-forward neurons at the final prompt token using a custom neuron selection dataset. We transfer the selected neuron identities to train hallucination classifiers on other factual question answering datasets. Our work provides empirical evidence that probes trained using the features from the selected neurons perform on par with probes trained on internal states. We also analyze the distribution of selected neurons and the effect of layer depth on detection performance. Index Terms: Hallucination Detection, Probing, Large Language Models, Neural Networks, Artificial Intelligence I Introduction Large language models can generate answers that are fluent and grammatically sound but factually incorrect. A natural response is to verify answers and detect hallucinations using external knowledge sources or additional model calls[1, 2]. These approaches introduce extra inference costs and may depend on resources unavailable in closed-book settings. This constraint has motivated a complementary line of white-box studies[3, 4] that assume access to internal states as part of the problem environment and study whether factuality-related information is already present in internal representations. Prior work [5, 6] has shown that internal representations contain information about the truthfulness of statements and generated outputs. Orgad et al. [7] extend this perspective by using linear probes trained on internal states at specific token positions for hallucination detection. Mechanistic interpretability studies further suggest that hallucination behavior is linked to identifiable internal states and mechanisms. Ferrando et al. [8] use Sparse Autoencoders (SAEs) to identify directions in the representation space associated with a model’s capability to recognize an entity and recall facts about it. They show that steering a model with these directions can affect refusal and hallucination behavior. Yu et al. [9] trace factual hallucinations through internal model components and show that failures of subject-attribute knowledge enrichment in lower-layer MLPs and answer extraction in upper-layer attention heads can both contribute to hallucination. These works motivate the study of hallucinations at the level of internal states, mechanisms, and components. A natural follow-up question is whether the information exploited by probes trained on internal representations, such as residual stream vectors, can also be recovered at a finer level of model structure, specifically at neuron-level granularity. Geva et al.[10] demonstrate that feed-forward layers in transformer-based language models act as key-value memories. Gurnee et al.[11] use sparse linear probing to find associations between human-understandable features and neuron activations. Li et al. [12] identify truth neurons by calculating an attribution score using integrated gradients followed by filtering. Their experiments show interventions on their truth neurons affect performance on truthfulness benchmarks, suggesting an association between truthful information and particular neurons. Gao et al. [13] study hallucination more directly. First, they extract answer spans and quantify each feed-forward neuron’s contribution to generation using CETT. Then, they identify neurons associated with hallucination, or H-neurons, by training an L1-regularized logistic regression on CETT features of all neurons at the generated exact answer tokens. Fig. 1: We collect ui,jℓu_i,j and zi,jℓz_i,j to compute δi,jℓ _i,j for ranking neurons as described in Section I-B Motivated by this line of work, we ask whether feed-forward neurons at the final prompt token contain information predictive of subsequent hallucination. Li et al.[12] and Gao et al.[13] demonstrate promising empirical evidence, but their pipelines are relatively costly. Identifying truth neurons requires repeated gradient computation during the neuron identification procedure. The H-Neurons pipeline requires additional LLM inference to extract answer spans at test time. In contrast, our work studies whether neurons correlated with hallucination can be identified from the final prompt token. Methods using spans of answer tokens localize hallucinations where factual claims are expressed. Pre-generation probes, using the language model’s first forward pass when the first answer token is being generated, test whether the model’s internal state already contains information about whether the forthcoming answer is likely to be factual. In this work, we introduce a method for ranking neurons based on their contribution to the residual stream. We build a custom neuron selection dataset based on CounterFact[14]. The selected neuron identities are then transferred to train probes on two closed-book question answering benchmarks, TriviaQA[15] and TruthfulQA[16], using the corresponding neuron features. Experiments on three open-weight instruction-tuned language models demonstrate that probes trained on features from the selected 1% of feed-forward neurons achieve hallucination detection performance comparable to probes trained on residual stream representations. The remainder of this paper is organized as follows. Section I formalizes the problem and introduces the NeuroActiSep score for ranking and selecting neurons, and defines a feature vector used to train the NeuroActiSep classifier. Section I describes the details of our experimental setup, and Section IV reports the results. Section V concludes the paper and suggests possible future directions. I Methodology In this section, we first formalize the problem and introduce a simplified notation for decoder-only transformer language models. We then define our neuron selection score and the classifier built from the selected neurons. I-A Problem Definition Given a dataset of factual questions Q and a language model M queried in a zero-shot closed-book regime, we denote the set of generated answers as A. Our objective is to assess the factuality of the answers and detect hallucinations in a binary setting without access to any knowledge base such that each answer asample∈a_sample generated for question qsample∈q_sample is classified as either truthful or hallucinated. I-B Neuron Selection in Language Models A transformer-based[17] decoder-only language model[18] with L layers from 11 to L, maps a sequence of n input tokens =(t1,…,tn)t=(t_1,…,t_n) to a probability distribution over the vocabulary V, which is utilized based on a decoding strategy for choosing the next token. Let =(1,…,n)X=(x_1,…,x_n) be the representation of the n input tokens. We denote the residual stream representation at token position i entering layer ℓ by iℓ∈ℝdx_i ^d, and use ℓX to denote the collection of token representations at that layer. Each transformer layer ℓ updates the residual stream by contributions from an attention block followed by a feed-forward neural network block. For clarity, we refer to the feed-forward network as the MLP block and each of its dmlpd_mlp units as a neuron. We omit normalization, positional encoding, bias terms, and other architecture-specific operations like gating from the notation. Accordingly, the residual stream entering layer ℓ+1 +1 after updates from layer ℓ would be: ~iℓ=iℓ+Attnℓ(ℓ)i, x_i =x_i +Attn (X )_i, (1) iℓ+1=~iℓ+MLPℓ(~iℓ).x_i +1= x_i +MLP ( x_i ). (2) Let iℓ∈ℝdmlpu_i ^d_mlp denote the values immediately before applying the neurons’ transformations and iℓ∈ℝdmlpz_i ^d_mlp denote the neuron activations after applying the activation functions and before the down-projection to the residual space by outℓW_out . The MLP block output can be written as MLPℓ(~iℓ)=iℓoutℓ=∑j=1dmlpzi,jℓout,jℓMLP ( x_i )=z_i W_out = _j=1^d_mlpz_i,j w_out,j (3) where out,jℓ∈ℝdw_out,j ^d is the j-th row of outℓW_out . Thus, zi,jℓout,jℓz_i,j w_out,j is the contribution of neuron j to the MLP output in residual space. For token tit_i, we extract the pre- and post-transformation activations of every neuron using TransformerLens[19] hooks11 1 blocks.ℓ .mlp.hook_pre and blocks.ℓ .mlp.hook_post. We denote these scalar activations for neuron j at layer ℓ and token position i by ui,jℓu_i,j and zi,jℓz_i,j , respectively. Their corresponding projections into residual space are i,jℓ,pre=ui,jℓout,jℓ,p_i,j ,pre=u_i,j w_out,j , (4) i,jℓ,post=zi,jℓout,jℓ.p_i,j ,post=z_i,j w_out,j . (5) The intuition behind the pre-transformation reference projection i,jℓ,prep_i,j ,pre, which is not an actual contribution in the forward pass, is that by using ui,jℓu_i,j we want to measure how much neuron j at layer ℓ has changed the output from what it could have been. Given δi,jℓ,sample=zi,jℓ,sample−ui,jℓ,sample _i,j ,sample=z_i,j ,sample-u_i,j ,sample for each sample from our generated answers, we have: Δi,jℓ,sample=δi,jℓ,sampleout,jℓ, _i,j ,sample= _i,j ,samplew_out,j , (6) i,jℓ,c=1Nc∑sample∈cΔi,jℓ,sample μ_i,j ,c= 1N_c _sample∈ c _i,j ,sample (7) where c∈Truthful,Hallucinatedc∈\Truthful,Hallucinated\ is a binary label assigned to samples by our annotation pipeline described in section I-B. The score for neuron j at layer ℓ is calculated as: Ri,jℓ=‖i,jℓ,Hallucinated−i,jℓ,Truthful‖2R_i,j = \| μ_i,j ,Hallucinated- μ_i,j ,Truthful \|_2 (8) We call this class-conditioned residual-space mean separation score as the NeuroActiSep score. It measures the separation between the class-conditional mean contribution vectors of each neuron in the model’s residual space. Because every contribution from neuron j lies along the same output vector out,jℓw_out,j , (8) can be written as Ri,jℓ=|δi,jℓ,Hallucinated¯−δi,jℓ,Truthful¯|‖out,jℓ‖2.R_i,j = | _i,j ,Hallucinated- _i,j ,Truthful | \|w_out,j \|_2. (9) where δi,jℓ,c¯ _i,j ,c is the mean over δi,jℓ _i,j of all samples of class c. We compute these scores at the final prompt position i=ni=n. We apply each model’s standard prompt template to questions. Therefore, our final prompt token corresponds to the last token of the post-instruction tokens [20], not the last question token. Given a dataset of truthful and hallucinated samples for neuron selection, we partition it into ℱF folds. Score Rj,fℓR_j,f is computed using the training partition of fold f∈ℱf . For each fold, all neurons from all layers are ranked globally, and the top K neurons form the fold-specific set fS_f. For each selected neuron, we compute its mean selection score Rj,fℓ¯ R_j,f over the folds in which it was selected. To obtain a single neuron set S, neurons are ranked first by vote counts among folds and then by R¯ℓ,j R_ ,j. The highest-ranked K neurons form the final aggregated set S and are transferred across datasets for the same language model. We use 8,000 samples of the Counterfact dataset divided into 3 folds for neuron selection, as described in Section I-A. I-C Hallucination Detection with NeuroActiSep Given the transferred set S, for each selected neuron j from layer ℓ such that (ℓ,j)∈( ,j) , we compute a signed projected scalar rjℓr_j at the last token of the prompt. rjℓ=δjℓ‖out,jℓ‖2.r_j = _j \|w_out,j \|_2. (10) This feature preserves the sign of the change zjℓ−ujℓz_j -u_j while scaling its magnitude by the norm ‖out,jℓ‖2. \|w_out,j \|_2. The feature vector for a sample prompt is =[rjℓ,sample](ℓ,j)∈.v^sample= [r_j ,sample ]_( ,j) . (11) We evaluated both the top-1% set 1%S_1\% and a feature-budget-matched subset dmodelS_d_model, which contains the first dmodeld_model neurons in the same ranking. For hallucination detection in a target dataset, we train NeuroActiSep in two variants: a logistic ridge regression and a multilayer perceptron with three hidden layers of sizes 256, 128 and 64. Both models use v^sample feature vectors as input data. Target dataset labels are only used in training the downstream classifiers and are never used to reselect or re-rank neurons. TABLE I: AUROC Performance of Linear Probes on the Neuron Selection Dataset Using Three-Fold Cross Validation Model Mistral Qwen Llama Chosen 1% 0.878± 0.003 0.828± 0.001 0.845± 0.007 Final Layer MLP Output 0.858± 0.000 0.794± 0.003 0.837± 0.008 Final Layer Residual Stream 0.860± 0.003 0.804± 0.005 0.839± 0.001 Fig. 2: Distribution of neurons from the chosen set 1%S_1\% across layers and AUROC of linear NeuroActiSep on the neuron selection dataset for each depth cutoff I Data and Resources In this section, we introduce the models and datasets used in this study, followed by an explanation of our annotation pipeline and the details of computational resources and hyperparameters. I-A Datasets and Models We conducted our experiments using three instruction-tuned, open-weight language models: Mistral-7B-Instruct-v0.2,22 2 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 Qwen2.5-7B-Instruct,33 3 https://huggingface.co/Qwen/Qwen2.5-7B-Instruct and Llama-3-8B-Instruct.44 4 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct. All answers are generated using greedy decoding in a zero-shot setting with each model’s standard chat template. We use a custom dataset based on CounterFact[14] for neuron selection. CounterFact contains factual cases organized by relation type. We use the prompts collection introduced by Jiang et al. [21], which provide multiple prompts for each triplet case. For each language model, we construct a fixed-size contrastive dataset of 4,000 truthful–hallucinated pairs (8,000 in total). The two samples in each pair belong to the same relation category but correspond to different factual triplets. No prompts from triplet cases with both truthful and hallucinated answers are included. Refusals and unlabeled generations are not included in the custom dataset, and sampling is capped at 10% for each category to reduce domination by highly represented relation categories. We evaluate our method on two commonly used question answering datasets, TriviaQA[15] and TruthfulQA[16]. TriviaQA is an open-domain question answering dataset that evaluates factual knowledge retrieval. Following the experimental setting used by Orgad et al.[7] we randomly sample 10,000 questions from TriviaQA’s training set and 10,000 questions from its development set, using the latter as our held-out test set. TruthfulQA evaluates whether language models reproduce humans’ commonly held misconceptions and false beliefs. We use all 817 questions for generating answers. I-B Annotation Pipeline The annotation procedure depends on the form of reference answers. For CounterFact and TriviaQA, we apply case-insensitive string matching between the generated response and the set of acceptable gold answers, followed by a second evaluation of unmatched samples using a locally deployed GPT-OSS-20B55 5 https://huggingface.co/openai/gpt-oss-20b serving as an LLM judge. We used the default decoding configuration for inference and set reasoning to high. For TruthfulQA, which provides both acceptable and explicitly incorrect answers, lexical matching is not used because truthful answers may differ substantially in wording from the reference answers. Therefore, we submit every generated response directly to the LLM judge. When the judge does not return a valid label, we repeat the evaluation five times and assign the majority label. Ties and samples for which no valid majority can be obtained are treated as unlabeled. Our pipeline assigns one of three labels: truthful, hallucinated, or refusal. Refusals and unlabeled samples are excluded from subsequent classifier training and evaluation. Table I shows the statistics regarding the labels of the evaluation datasets. TABLE I: Number of labels assigned to each dataset by our annotation pipeline Dataset Model Truthful Halluc. Refusal No Label TruthfulQA Mistral 532 254 29 2 Qwen 501 308 8 0 Llama 427 381 8 1 TriviaQA (Train) Mistral 6818 2812 193 177 Qwen 6528 3250 90 132 Llama 7501 2230 90 179 TriviaQA (Test) Mistral 6827 2778 218 177 Qwen 6389 3367 98 146 Llama 7491 2254 88 167 TABLE I: AUROC performance of NeuroActiSep compared with baselines across datasets and models Method TriviaQA TruthfulQA Mistral Qwen Llama Mistral Qwen Llama Generation Scores Perplexity 0.617 0.712 0.705 0.545± 0.037 0.522± 0.042 0.570± 0.036 Mean Probability 0.615 0.697 0.693 0.537± 0.041 0.514± 0.048 0.563± 0.034 Mean Logits 0.617 0.656 0.670 0.580± 0.036 0.570± 0.048 0.562± 0.042 Min Logits 0.625 0.749 0.763 0.604± 0.028 0.588± 0.045 0.588± 0.046 P(True) 0.730 0.745 0.720 0.406± 0.026 0.510± 0.052 0.361± 0.017 Linear Probe @ Hook Point / Token End of Prompt mlp_out 0.786 0.729 0.783 0.722± 0.035 0.663± 0.013 0.729± 0.029 resid_post 0.762 0.720 0.749 0.711± 0.038 0.682± 0.016 0.736± 0.021 Generation of Final Token mlp_out 0.693 0.678 0.750 0.661± 0.024 0.580± 0.029 0.617± 0.046 resid_post 0.679 0.692 0.730 0.673± 0.015 0.569± 0.041 0.621± 0.047 MLP Probe @ Hook Point / Token End of Prompt mlp_out 0.808 0.807 0.810 0.748± 0.021 0.679± 0.030 0.741± 0.026 resid_post 0.821 0.831 0.821 0.754± 0.022 0.722± 0.021 0.747± 0.027 Generation of Final Token mlp_out 0.743 0.797 0.794 0.671± 0.022 0.607± 0.046 0.604± 0.036 resid_post 0.766 0.818 0.809 0.700± 0.028 0.647± 0.024 0.624± 0.031 NeuroActiSep dmodelS_d_model Linear 0.804 0.741 0.771 0.734± 0.036 0.660± 0.022 0.750± 0.015 MLP 0.842 0.844 0.828 0.767± 0.017 0.721± 0.013 0.763± 0.017 1%S_1\% Linear 0.802 0.753 0.767 0.736± 0.034 0674± 0.020 0.747± 0.014 MLP 0.838 0.834 0.822 0.761± 0.024 0.715± 0.024 0.765± 0.027 I-C Computational Resources and Reproducibility All experiments use random seed 42 and are conducted on a local machine running Ubuntu 22.04 LTS, equipped with an NVIDIA GeForce RTX 3090 GPU, an Intel Core i7-12700K CPU, and 32 GB of system memory. The models are executed in bfloat16, and the answers are generated using greedy decoding with a maximum of 100 new tokens for TriviaQA and CounterFact and 256 new tokens for TruthfulQA. All linear and MLP models are trained with Scikit-Learn[22] and both use class-balanced weighting. Linear models use Scikit-Learn’s default settings with max_iter raised to 20,000. MLPs use ReLU activations and the adam solver, with a 10% validation split, early-stopping patience of 20, and a 500-epoch cap. The datasets, prompts, and experiment configurations are available at [https://github.com/alidrgr/NeuroActiSep]. IV Results and Experiments In this section, we report results for hallucination detection with NeuroActiSep on both the neuron selection and evaluation datasets. We also analyze how chosen neurons are distributed across layers and how layer depth affects performance. IV-A Performance of Chosen Neurons on CounterFact Table I reports the results of three-fold cross-validation for linear NeuroActiSep on the neuron selection dataset. Each fold uses its own set fS_f derived from the corresponding training partition. Across all three language models, linear probes trained on the chosen 1% of neurons outperform linear probes trained on the last-layer residual stream and MLP block representations, both taken at the final prompt token. The results in Table I suggest our neuron selection method identifies neurons correlated with factual hallucinations. Based on this observation, we construct the aggregated set S and test whether these neurons generalize to other factual QA datasets. IV-B Main Results of NeuroActiSep For each LLM, we use its set S to train a NeuroActiSep classifier for hallucination detection on the evaluation datasets. For TriviaQA, the classifier is fitted on the train set and evaluated on the test set. For TruthfulQA, the classifier is evaluated using stratified five-fold cross-validation. In addition to scores collected and computed during generation, we compare against the following baselines: P(True): To assess a model’s awareness of its knowledge boundary, we use [23]’s idea to ask the LLM whether its generated answer is correct. Each model is prompted twice without including brainstormed ideas, using both orderings of the “(A) True / (B) False” options to control for bias. Linear Probes[7]: We train separate linear probes on the internal states at the final prompt token and the state leading to the final generated token using the MLP output (mlp_out) and the residual stream (resid_post). MLP Probes: Similar to [5], we train three-layer MLP neural networks with hidden layers of 256, 128, and 64 units on the final prompt token and the state used to predict the final generated token separately. We use the same two hook points as in the linear baselines. Results appear in Table I. Despite using neuron locations chosen from a different dataset, NeuroActiSep performs comparably to classifiers trained directly on internal states, ranking among the top two methods by AUROC across all six model–dataset combinations. This shows the selected neurons contain signals correlated with factual hallucination on questions other than the dataset on which they were selected. IV-C Distribution of Chosen Neurons The top row of Fig. 2 shows the distribution of the set 1%S_1\% in different models. A recurring theme among the three models is that most neurons are selected from the upper layers. In Mistral and Llama, the number of chosen neurons begins to increase around the middle layers, while in Qwen neurons are considerably concentrated in the upper layers. Nevertheless, the concentration of selected neurons in particular layers does not by itself establish that these layers are necessary for predictive performance. IV-D Analysis Across Layer Depth We conduct a depth-restricted analysis using the neuron selection dataset to examine how performance changes as neurons from progressively deeper layers become available. For each depth cutoff ℓ , neuron selection is restricted to layers up to and including ℓ . The bottom row of Fig. 2 reports the three-fold cross-validation results under three selection strategies: global, which selects the highest-ranked K neurons among all neurons up to layer ℓ , where K is fixed to 1% of the model’s total feed-forward neurons; layer-balanced, which distributes a matched total number of selected neurons evenly across the eligible layers; and fixed-1%-per-layer, which selects the top 1% of neurons independently from each eligible layer. Hallucination-predictive information appears to be limited in the lower layers and becomes increasingly accessible as neurons from the middle layers are included. However, the competitive performance of the fixed-1%-per-layer strategy, despite its smaller feature budget at intermediate depths, and the plateaus in the depth curves suggest that most of the signal is already available from a sparser set of neurons before the final layers are introduced. The concentration of selected neurons in the middle-to-late layers, despite the limited gains after the curves plateau, suggests that many highly ranked neurons in later layers may encode overlapping predictive information. V Conclusion In this research, we proposed the NeuroActiSep score for ranking neurons and the NeuroActiSep classifier for detecting hallucinations at the final prompt token. We demonstrated the feasibility of identifying feed-forward neurons correlated with factual hallucinations and using them to train classifiers on out-of-distribution factual questions. These findings do not establish that the selected neurons causally produce or prevent hallucinations, as we did not conduct causal intervention. Therefore, further mechanistic studies are needed to investigate the causal relationship between these neurons and hallucination in language models. Acknowledgment We would like to acknowledge the use of LLM-based tools for assistance in the preparation of this paper. Grammarly Writing Assistant was used to correct spelling, grammar, and punctuation errors. GPT-5.6 Sol was used for coding the plots in Fig. 2. The authors maintain full accountability and responsibility for the content of this paper. References [1] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal (2024) Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630 (en). External Links: ISSN 1476-4687, Document Cited by: §I. [2] Y. Xue, K. Greenewald, Y. Mroueh, and B. Mirzasoleiman (2026) Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §I. [3] J. Kossen, J. Han, M. Razzak, L. Schut, S. Malik, and Y. Gal (2024) Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs. arXiv. Note: arXiv:2406.15927 External Links: Document Cited by: §I. [4] J. Binkowski, D. Janiak, A. Sawczyn, B. Gabrys, and T. J. Kajdanowicz (2025) Hallucination Detection in LLMs Using Spectral Features of Attention Maps. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 24354–24385. External Links: ISBN 979-8-89176-332-6, Document Cited by: §I. [5] A. Azaria and T. Mitchell (2023) The Internal State of an LLM Knows When It’s Lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 967–976. External Links: Document Cited by: §I, §IV-B. [6] S. Marks and M. Tegmark (2024) The geometry of truth: emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, Cited by: §I. [7] H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y. Belinkov (2025) LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. In International Conference on Learning Representations, Vol. 2025, p. 66880–66913. Cited by: §I, §I-A, §IV-B. [8] J. Ferrando, O. Obeso, S. Rajamanoharan, and N. Nanda (2025) Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models. In International Conference on Learning Representations, Vol. 2025, p. 77965–78000. Cited by: §I. [9] L. Yu, M. Cao, J. C. Cheung, and Y. Dong (2024) Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 7943–7956. External Links: Document Cited by: §I. [10] M. Geva, R. Schuster, J. Berant, and O. Levy (2021) Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 5484–5495. External Links: Document Cited by: §I. [11] W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas (2023) Finding neurons in a haystack: case studies with sparse probing. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §I. [12] H. Li, Y. Cao, Y. Yu, J. W. Suchow, and Z. Zhu (2025) Truth neurons. In Knowledgeable Foundation Models at ACL 2025, External Links: Document Cited by: §I, §I. [13] C. Gao, H. Chen, C. Xiao, Z. Chen, Z. Liu, and M. Sun (2025) H-neurons: on the existence, impact, and origin of hallucination-associated neurons in llms. Note: arXiv:2512.01797 External Links: 2512.01797 Cited by: §I, §I. [14] K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022) Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, Vol. 35, p. 17359–17372. External Links: Document Cited by: §I, §I-A. [15] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017) TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601–1611. External Links: Document Cited by: §I, §I-A. [16] S. Lin, J. Hilton, and O. Evans (2022) TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252. External Links: Document Cited by: §I, §I-A. [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §I-B. [18] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019) Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: §I-B. [19] N. Nanda and J. Bloom (2022) TransformerLens. Note: https://github.com/TransformerLensOrg/TransformerLens Cited by: §I-B. [20] J. Zhao, J. Huang, Z. Wu, D. Bau, and W. Shi (2025) LLMs encode harmfulness and refusal separately. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §I-B. [21] C. Jiang, B. Qi, X. Hong, D. Fu, Y. Cheng, F. Meng, M. Yu, B. Zhou, and J. Zhou (2024) On Large Language Models’ Hallucination with Regard to Known Facts. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 1041–1053. External Links: Document Cited by: §I-A. [22] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay (2011) Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12, p. 2825–2830. Cited by: §I-C. [23] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan (2022) Language models (mostly) know what they know. arXiv. Note: arXiv:2207.05221 External Links: 2207.05221 Cited by: §IV-B.