Paper deep dive
Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States
Zakhar Mrykhin, Valentin Malykh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/11/2026, 4:55:07 AM
Summary
The paper introduces Prompt Embedding Probes (PEP), a white-box method for detecting answer-level hallucinations in Large Language Models (LLMs) by augmenting standard linear probes with learnable prompt embeddings. Evaluated on Qwen3 models across TriviaQA, GSM8K, and MedQA datasets, PEP demonstrates improved hidden-state-based hallucination detection compared to standard linear probes, particularly in in-distribution and pre-generation settings, while maintaining a frozen backbone.
Entities (8)
Relation Signals (7)
Prompt Embedding Probes → detects → Hallucination Detection
confidence 95% · Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States
Prompt Embedding Probes → evaluatedon → TriviaQA
confidence 95% · We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales.
Prompt Embedding Probes → evaluatedon → GSM8K
confidence 95% · We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales.
Prompt Embedding Probes → evaluatedon → MedQA
confidence 95% · We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales.
Prompt Embedding Probes → extends → Linear Probe
confidence 95% · PEP extends standard linear probes by augmenting the input with a small number of learnable prompt embeddings.
Prompt Embedding Probes → usesmodel → Qwen3
confidence 95% · We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales.
Prompt Embedding Probes → improvesperformanceover → Linear Probe
confidence 90% · PEP improves hidden-state-based detection over standard linear probes in the main in-distribution setting.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) can generate fluent and useful responses but remain prone to hallucinations. We introduce Prompt Embedding Probes (PEP), a white-box method for answer-level hallucination detection from the hidden states of a frozen LLM. PEP extends standard linear probes by augmenting the input with a small number of learnable prompt embeddings. We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales. PEP improves hidden-state-based detection over standard linear probes in the main in-distribution setting. We further evaluate PEP for pre-generation prediction, cross-model transfer, and out-of-distribution generalization. PEP remains effective in the pre-generation and cross-model settings, whereas robust cross-dataset transfer remains difficult. These results show that prompt-based adaptation can strengthen hidden-state probing while keeping the backbone frozen and adding only a small number of trainable parameters.
Tags
Links
- Source: https://arxiv.org/abs/2608.08024v1
- Canonical: https://arxiv.org/abs/2608.08024v1
Trouble viewing inline? Open PDF directly →
Full Text
33,513 characters extracted from source content.
Expand or collapse full text
Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States Zakhar Mrykhin zazamrykh@gmail.com &Valentin Malykh HSE University valentin.malykh@phystech.edu Abstract Large language models (LLMs) can generate fluent and useful responses, but they remain prone to hallucinations. This motivates efficient methods for detecting unreliable outputs at inference time. In this paper, we study answer-level hallucination detection in a white-box setting based on hidden states of a frozen LLM, and introduce Prompt Embedding Probes, a parameter-efficient extension of standard linear probes that augments the input with a small number of learnable prompt embeddings. We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models of multiple scales. The results show that learnable prompt embeddings improve hidden-state-based hallucination detection in the main in-distribution setting relative to standard linear probes. We further study the method in a to-be-generated (TBG) setting for pre-generation hallucination prediction, in cross-model transfer experiments, and under out-of-distribution evaluation. PEP remains effective in the TBG and cross-model settings, while robust transfer across substantially different datasets remains difficult. These results suggest that prompt-based adaptation can strengthen probe-based hallucination detection while keeping the backbone frozen and adding only a small number of trainable parameters. Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States Zakhar Mrykhin zazamrykh@gmail.com Valentin Malykh HSE University valentin.malykh@phystech.edu 1 Introduction Large language models (LLMs) are capable of producing fluent and useful responses, yet they remain prone to hallucinations—outputs that are incorrect, unsupported, or otherwise unreliable (Huang et al., 2025). This is especially concerning in high-risk domains such as medicine, finance, and law, where model errors may lead to consequential decisions or misleading professional advice. Legal cases already show that hallucinated content in LLM-assisted workflows can result in tangible penalties and procedural harm (Reuters, 2026). Hallucinations also remain common in widely used evaluation datasets and benchmarks, indicating that they persist despite continued progress in LLM development (Niu et al., 2024; Li et al., 2023). Hallucinations can arise from multiple sources across the data, training, and inference stages, making them difficult to eliminate through a single intervention (Huang et al., 2025). Prior work further shows that even simple factual prompts can elicit confident but incorrect answers from LLMs (Azaria and Mitchell, 2023). Hallucination detection therefore complements efforts to reduce hallucinations by identifying unreliable model outputs at inference time. Among existing approaches, white-box methods are particularly appealing because some of them are simple and efficient at inference time. A growing body of work suggests that hidden states and related internal representations contain useful signals for detecting hallucinations and assessing answer reliability (Azaria and Mitchell, 2023; Kossen et al., 2024; Snyder et al., 2024; Zhang et al., 2025; Chen et al., 2025; Obeso et al., 2025; Han et al., 2025). At the same time, standard linear probes may be limited in how flexibly they adapt to the hallucination detection task when applied directly to frozen hidden states. This paper investigates whether hidden-state-based hallucination detection can be improved by augmenting probes with learnable prompt embeddings (Lester et al., 2021). To this end, we introduce Prompt Embedding Probes, a white-box method that extends standard linear probes with trainable prompt embeddings for the detection of hallucinations at answer-level in LLMs. The experiments evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales, and compare it against linear probes and additional baselines. The main contributions of this paper are as follows: • We introduce Prompt Embedding Probes, a white-box method for answer-level hallucination detection in LLMs based on hidden states and learnable prompt embeddings. The code used in this study is available at github.com/zazamrykh/internal_probing. • We show that learnable prompt embeddings improve in-distribution hallucination detection relative to standard linear probes across multiple datasets and model scales. • We provide an empirical analysis of PEP beyond the main in-distribution setting, including out-of-distribution evaluation, cross-model transfer, and a to-be-generated token setting for pre-generation hallucination prediction. We also study the effect of key method parameters such as prompt embedding placement, prompt embedding count, and probing layer selection. 2 Related Work Hallucination detection in LLMs has been studied in different settings. Black-box methods typically rely on response consistency in repeated sampling (Manakul et al., 2023), self-verification or verification-based checks (Dhuliawala et al., 2024), or external verification pipelines (Min et al., 2023). Such approaches can be effective, but they are often computationally expensive and may require multiple model calls or additional verification components. Other approaches use confidence signals: token probabilities or related log-probability statistics. Examples include probability-based self-evaluation methods such as P(True) (Kadavath et al., 2022) and token-level uncertainty methods for fact-checking generated claims (Fadeeva et al., 2024). However, token-level uncertainty is inherently limited: low-probability tokens may reflect not only uncertainty about the underlying meaning, but also lexical and syntactic variation (Farquhar et al., 2024). White-box approaches to hallucination detection use internal LLM representations. Azaria and Mitchell (2023), Snyder et al. (2024), Chen et al. (2025), and related work showed that hidden states and other generation-time artifacts contain useful signals for hallucination detection. Kossen et al. (2024) demonstrated that simple linear probes on hidden states are effective for predicting targets such as answer correctness and semantic entropy. Zhang et al. (2025) proposed a method that selects informative neurons from internal representations and uses them for hallucination detection. Han et al. (2025) used linear probes for factual hallucination detection at the claim level, which is useful because LLM responses often contain multiple claims that require verification. Obeso et al. (2025) studied the applicability of linear probes to real-time factual hallucination detection at the entity level. They also explored LoRA adapters to improve detection quality while using regularization to preserve the model’s generation behavior. Overall, prior work suggests that linear probes are a practical and effective family of white-box methods for hallucination detection across different targets and granularities. At the same time, existing work mainly studies which internal representations or probing targets are most useful and focuses on different hallucination types, while the effect of modifying the probing input through learnable prompt embeddings remains underexplored. This paper addresses that gap by studying whether trainable prompt embeddings can improve hidden-state-based hallucination detection over standard linear probes. 3 Method 3.1 Task Formulation This work considers answer-level hallucination detection. Given a query x and a model-generated answer a, the goal is to predict a binary label y∈0,1y∈\0,1\, where y=1y=1 denotes a hallucinated answer. More generally, a detector defines a scoring function fθ(x,a)→[0,1].f_θ(x,a)→[0,1]. In the to-be-generated (TBG) setting, detection is performed before the answer is generated, while in the second-last-token (SLT) setting it is performed using representations obtained after the answer has been processed. 3.2 Linear Probe Baseline Let z=(z1,…,zT)z=(z_1,…,z_T) be the token sequence given to the model, and let H(ℓ)=(h1(ℓ),…,hT(ℓ)),ht(ℓ)∈ℝd,H^( )=(h_1^( ),…,h_T^( )), h_t^( ) ^d, denote the hidden states after layer ℓ of a frozen LLM. The linear probe uses a hidden state selected by the probing layer ℓ and probing position q: u=hq(ℓ).u=h_q^( ). A linear classification head is then applied on top of u: p^θ(y=1∣x,a)=σ(w⊤u), p_θ(y=1 x,a)=σ(w u), where w∈ℝdw ^d, σ(⋅)σ(·) is the sigmoid function, and θ=wθ=w are the trainable parameters of the detector. 3.3 Prompt Embedding Probe PEP extends the linear probe baseline by inserting M trainable prompt embeddings into the input embedding sequence before the forward pass through the frozen LLM. Let E(z)=(e1,…,eT),et∈ℝd,E(z)=(e_1,…,e_T), e_t ^d, be the input embeddings of z, and let P=(p(1),…,p(M)),p(j)∈ℝd,P=(p^(1),…,p^(M)), p^(j) ^d, be the trainable prompt embeddings. After insertion, we obtain an augmented sequence E~=I(E(z),P), E=I(E(z),P), where I(⋅)I(·) denotes the chosen insertion scheme. Let H~(ℓ)=(h~1(ℓ),…,h~T+M(ℓ)) H^( )=( h_1^( ),…, h_T+M^( )) be the hidden states after layer ℓ produced from the modified input. As in the baseline, detection is performed at probing layer ℓ and probing position q, where q is defined with respect to the original token sequence before prompt insertion. Denoting by π(q)π(q) the corresponding position after insertion, the detector uses u=h~π(q)(ℓ).u= h_π(q)^( ). A linear head is applied on top of the selected hidden state: p^θ(y=1∣x,a)=σ(w⊤u), p_θ(y=1 x,a)=σ(w u), where θ=(p(j)j=1M,w)θ= (\p^(j)\_j=1^M,w ) is the set of trainable parameters. Thus, unlike the baseline, PEP jointly learns both the probe weights and the inserted prompt embeddings, while keeping the base LLM frozen. PEP adds only the prompt embeddings and the classification head as trainable parameters. For M prompt embeddings and hidden size d, the prompt component contains M×dM× d trainable parameters, while the linear head contains d+1d+1 parameters when implemented with a bias. Thus, PEP trains M×d+d+1M× d+d+1 parameters in total, while all backbone parameters remain frozen. Figure 1: Overview of Prompt Embedding Probe. Trainable prompt embeddings are inserted into the input embedding sequence, processed by a frozen LLM, and the selected hidden state is passed to a linear classification head for hallucination detection. Both methods are trained with binary cross-entropy on answer-level labels. In practice, the linear probe is implemented with LogisticRegression from scikit-learn using lbfgs and C=1.0C=1.0, while PEP is implemented in PyTorch with Hugging Face transformers. 4 Experimental Setup We evaluate on TriviaQA (Joshi et al., 2017), GSM8K (Cobbe et al., 2021), and MedQA (Jin et al., 2021). TriviaQA targets short factual question answering, GSM8K focuses on mathematical reasoning, and MedQA represents a high-risk medical QA setting with multiple-choice answers. For TriviaQA, we use the system prompt “Answer briefly.” For GSM8K, we use the system prompt “Solve the problem step by step. At the end write exactly: Final answer: <number>” The exact prompt used for MedQA is provided in Appendix A. Baselines. We compare PEP with three baselines. First, the linear probe applies a logistic-regression classifier to a selected hidden state of the frozen LLM. Second, average negative log-probability uses the average negative log-probability of the generated answer as an uncertainty-based score. Third, P(True) single is a single-generation self-evaluation baseline: the model is prompted once to assess whether its generated answer is true or false, and the resulting probability is used as the detection score. Dataset Train Valid Test TriviaQA 10,000 1,000 17,944 GSM8K 6,473 1,000 1,319 MedQA 8,000 1,000 1,273 Table 1: Dataset splits used in the experiments. Figure 2: ROC AUC with bootstrap 95% confidence intervals for PEP comparison with baselines across three datasets and three Qwen3 model sizes. Figure 3: ROC AUC with bootstrap 95% confidence intervals for PEP vs linear probe in TBG and SLT setups For TriviaQA, correctness is evaluated with substring match under a short-answer prompting format. For GSM8K and MedQA, correctness is determined with format-based matching rules. We report ROC AUC in all experiments. We use Qwen3 models (Yang et al., 2025) in three sizes: 0.6B, 8B, and 32B parameters. This allows us to evaluate the method across substantially different model scales within the same model family. For cross-model and OOD experiments we also use Gemma3 12B (Gemma Team et al., 2025). Experiments were conducted on an NVIDIA H200 GPU with 140 GB of memory using PyTorch and Hugging Face Transformers. As a representative example, training PEP with 5 prompt embeddings on Qwen3 8B for TriviaQA required approximately 85 GPU minutes, estimated from training logs up to convergence. In the more expensive setting of GSM8K with Qwen3 32B, we used batch size 1 and observed a training time of approximately 27 hours up to the best validation ROC AUC. The exact training configuration and implementation details are available in the public repository. 5 Results Table 2: Main in-distribution results. ROC AUC of the linear probe and tuned PEP across three datasets and Qwen3 model sizes. Δ ROC AUC is the paired-bootstrap estimate of the difference between tuned PEP and the linear probe; positive values favor PEP. For Qwen3-0.6B on MedQA, the confidence interval includes zero, so we do not interpret the observed difference as evidence of a reliable improvement. Model Dataset Linear Probe Tuned PEP Δ ROC AUC 95% CI Qwen3-0.6B TriviaQA 0.766 0.793 +0.027 [0.020, 0.034] GSM8K 0.821 0.879 +0.059 [0.040, 0.077] MedQA 0.514 0.519 +0.005 [-0.037, 0.048] Qwen3-8B TriviaQA 0.762 0.856 +0.094 [0.088, 0.100] GSM8K 0.776 0.839 +0.062 [0.011, 0.117] MedQA 0.700 0.734 +0.034 [0.002, 0.065] Qwen3-32B TriviaQA 0.755 0.868 +0.113 [0.106, 0.121] GSM8K 0.704 0.783 +0.079 [0.012, 0.146] MedQA 0.737 0.838 +0.101 [0.071, 0.130] 5.1 Prompt embeddings improve hallucination detection We begin with the main question of this work: do prompt embeddings improve hidden-state-based hallucination detection? Figure 2 summarizes the main comparison. Figure 2 compares tuned and untuned PEP with the linear-probe and probability-based baselines described in Section 4. It also reports PEP with fixed untuned hyperparameters and tuned PEP. For the linear probe, the best probing layer is selected on the validation set. In contrast, the untuned PEP variant uses a fixed configuration chosen a priori before running the experiments: probing at 0.6 of the total model depth, 10 prompt embeddings inserted at the beginning of the sequence, and the SLT setup. The main result is that tuned PEP outperforms the linear probe in 8 of 9 model–dataset combinations under the paired-bootstrap comparison. The only exception is Qwen3-0.6B on MedQA, where the estimated improvement is small and its confidence interval includes zero. We therefore do not interpret this comparison as evidence of a reliable improvement. Table 2 reports the absolute ROC AUC values together with the paired-bootstrap differences and confidence intervals. 5.2 PEP remains effective in the TBG setting We next test whether the gains from prompt embeddings persist in the more challenging to-be-generated (TBG) setting. Using the configurations selected in the SLT experiments, we evaluate PEP before answer generation is complete and compare it against the linear probe baseline. Figure 3 shows the results of this experiment. The result is encouraging: TBG PEP outperforms even the SLT linear probe in 8 of 9 model–dataset combinations under a one-sided paired bootstrap test on the ROC AUC difference with significance level 0.05. Overall, these results show that the benefit of prompt embeddings is not restricted to the SLT setup and extends to the more difficult TBG regime. 5.3 Cross-model transfer Figure 4: Cross-model transfer within the Qwen3 family on TriviaQA. Each detector is trained on its own model’s generations and evaluated on generations produced by all models in the comparison. Figure 5: Cross-model transfer across model families on TriviaQA. Gemma 3 12B is used without additional parameter tuning. We next study whether a detector based on one model can identify hallucinations in answers generated by another model. This setting separates the generator from the detector and therefore moves the evaluation toward a black-box-like regime with respect to the target generator, while still allowing white-box access to the detector model itself. All experiments in this section are conducted on TriviaQA. Importantly, each detector is trained on the generations of its own base model and is then evaluated both on its own generations and on generations produced by other models. We consider two transfer settings. In the first, we compare Qwen3 0.6B, 8B, and 32B in order to study transfer across model scales within the same family (see Figure 4). In the second, we compare Qwen3 8B, Qwen3 32B, and Gemma 3 12B in order to test transfer across model families (see Figure 5). Gemma 3 12B is included without additional parameter tuning. The results can be read from two complementary perspectives. First, row-wise, each detector generally performs best on the generations of its own model. This suggests that model-specific generation patterns are indeed useful for hallucination detection. Second, column-wise, the best detector for a given generator is typically the largest one, namely Qwen3 32B. Thus, for detection quality, detector scale appears to matter more than using the hidden states of exactly the same model that produced the answer. Overall, the experiment leads to three conclusions. First, hidden-state-based hallucination detection can be applied to answers generated by a different model, including models from another family. Second, smaller detector models generally transfer worse to stronger generators, although their performance remains reasonably competitive. Third, detector size plays an important role in final performance, suggesting that exact generator–detector identity is not the dominant factor in cross-model detection. 5.4 Effect of probing layer We next study how the choice of probing layer affects hallucination detection quality. Figure 6 shows one representative example, while the full set of 3×3 plots for all model–dataset combinations is provided in Appendix B. Figure 6: ROC AUC as a function of probing layer for Qwen3 8B on TriviaQA in the SLT and TBG settings. The results show that the optimal layer depends on the evaluation setup, the model, and the dataset. Typically the best probing layer lies between the middle and the end of the network rather than at a fixed relative depth. A more stable way to describe the best probing position is by its distance to the final layer rather than by the fraction of total model depth. Table 3 summarizes this pattern across models. Table 4 describes the median best fraction of total depth and the number of layers to the end for the best layer choice. Qwen3 0.6B (28) Qwen3 8B (36) Qwen3 32B (64) TBG 19 / 0.68 / 9 25 / 0.69 / 11 53 / 0.83 / 11 SLT 19 / 0.68 / 9 25 / 0.69 / 11 57 / 0.89 / 7 Table 3: Median best probing layer across datasets, reported as layer index / fraction of total depth / number of layers to the end. Headers contain maximum number of layers for each model. TriviaQA GSM8K MedQA TBG 0.58 / 15 0.69 / 11 0.83 / 9 SLT 0.68 / 15 0.69 / 9 0.89 / 3 Table 4: Median best probing layer aggregated by dataset and reported as fraction of total depth / number of layers to the end. 5.5 Effect of insertion position and number of prompt embeddings We next study how the method depends on two additional hyperparameters: the insertion position of the prompt embeddings and their number. For the probing layer selected in the SLT setup, we sweep three insertion strategies—beginning only, end only, and both beginning and end—as well as the number of inserted embeddings. Table 5 summarizes results of this experiment. Full results for all model–dataset combinations are provided in Appendix B. The best configuration depends on both the model and the dataset. But at the same time in most experiments, inserting a small number of prompt embeddings at the beginning of the sequence gives the best result, while insertion only at the end wins only once. Insertion strategy Wins (out of 9) Median best count Beginning only 6 5 End only 1 3 Both 2 3 Table 5: Aggregate results of the embedding search across the nine model–dataset configurations. 5.6 Out-of-distribution evaluation We study an out-of-distribution (OOD) setting in which the detector is trained on two of the three datasets and evaluated on the held-out third one. Both the generator and detector LLM were the same in this experiment: Gemma3 12B. Results are available in Table 6. Held-out dataset Linear probe PEP TriviaQA 0.5490 0.6250 MedQA 0.5117 0.5267 GSM8K 0.5676 0.5163 Table 6: OOD evaluation: training on two datasets and testing on the held-out third one. Overall, both PEP and the linear probe perform poorly in this setting. A likely reason is that the three datasets differ substantially in both the answer format and error type, making cross-dataset transfer difficult for both methods. The only clearly meaningful result is obtained when TriviaQA is used as the held-out dataset. But overall the gains of PEP are most convincing in the in-distribution setting, while robust OOD transfer remains an open challenge. 6 Conclusion We introduced Prompt Embedding Probes (PEP), a white-box method for answer-level hallucination detection based on hidden states of a frozen LLM. PEP extends standard linear probes by jointly learning a small number of prompt embeddings and a linear classification head, while keeping all backbone parameters frozen. Across three QA datasets and multiple Qwen3 model sizes, PEP improves in-distribution ROC AUC relative to a standard linear probe in most evaluated settings. The gains also extend to the to-be-generated setting and to the cross-model experiments considered in this work. However, the cross-dataset OOD results show that neither PEP nor the linear probe transfers robustly across substantially different datasets. Thus, the current evidence supports PEP primarily as an in-distribution prompt-based adaptation for hidden-state probing, rather than as a universally transferable hallucination detector. Overall, PEP provides a simple parameter-efficient extension of linear probing, although its training is more computationally demanding than fitting a linear probe on cached hidden states. Improving cross-dataset generalization and comparing prompt-based adaptation with stronger probe architectures and alternative adaptation methods remain important directions for future work. 7 Potential Risks Although the proposed method is intended to improve reliability monitoring, incorrect detector predictions may still be harmful in practice. False negatives may lead users to trust unreliable model outputs, while false positives may suppress correct answers and reduce system utility. In addition, our results show limited robustness under cross-dataset transfer, so deploying the detector outside the evaluated settings may create a misleading sense of safety, especially in high-stakes domains such as medicine. For this reason, we view the proposed method as a supporting signal for risk-aware use rather than as a standalone guarantee of answer reliability. 8 Artifacts, Licensing, Intended Use This work uses publicly available datasets and model families for research and evaluation purposes, consistent with their original intended use and respective terms of use. The implementation associated with this work is publicly available under the MIT license at github.com/zazamrykh/internal_probing and is intended for research use. We do not claim that the resulting detector is ready for deployment in safety-critical settings without additional validation. Limitations Despite its empirical improvements over standard linear probes, the proposed method has several limitations. First, Prompt Embedding Probes are more complex both architecturally and computationally than standard hidden-state-based linear probing. The method requires a more involved implementation and more expensive training, since optimizing prompt embeddings requires repeated forward passes through the frozen language model together with gradient computation, which increases GPU memory usage and training time. Second, the OOD experiments show that the method does not transfer well across substantially different datasets. This likely reflects large differences in answer format and task structure between the considered benchmarks, suggesting that stronger cross-dataset generalization remains an open problem. One possible direction for future work is to explore alternative training targets, such as uncertainty- or semantic entropy-related signals. Third, the current study focuses on answer-level hallucination detection, whereas many practical applications increasingly require claim-level detection for long-form responses. Although the proposed method could likely be adapted to finer-grained prediction, this was not investigated in the present work. Finally, the method depends on several hyperparameters, including the probing layer, insertion position, and number of prompt embeddings. Even though PEP shows strong performance under non-optimal settings, obtaining the highest detection quality still requires tuning these parameters for a particular setup. References A. Azaria and T. Mitchell (2023) The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 967–976. External Links: Link, Document Cited by: §1, §1, §2. R. Chen, A. Arditi, H. Sleight, O. Evans, and J. Lindsey (2025) Persona vectors: monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509. Cited by: §1, §2. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, and R. Nakano (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4. S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston (2024) Chain-of-verification reduces hallucination in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3563–3578. External Links: Link, Document Cited by: §2. E. Fadeeva, A. Rubashevskii, A. Shelmanov, S. Petrakov, H. Li, H. Mubarak, E. Tsymbalov, G. Kuzmin, A. Panchenko, T. Baldwin, P. Nakov, and M. Panov (2024) Fact-checking the output of large language models via token-level uncertainty quantification. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 9367–9385. External Links: Link, Document Cited by: §2. S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal (2024) Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: §2. Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Rame, and M. Riviere (2025) Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: §4. J. Han, N. Band, M. Razzak, J. Kossen, T. G. J. Rudner, and Y. Gal (2025) Simple factuality probes detect hallucinations in long-form natural language generation. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 16209–16226. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, §2. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu (2025) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: §1, §1. D. Jin, E. Pan, N. Oufattole, W. Weng, H. Fang, and P. Szolovits (2021) What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11 (14), p. 6421. Cited by: §4. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer (2017) TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M. Kan (Eds.), Vancouver, Canada, p. 1601–1611. External Links: Link, Document Cited by: §4. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, and E. Tran-Johnson (2022) Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Cited by: §2. J. Kossen, J. Han, M. Razzak, L. Schut, S. Malik, and Y. Gal (2024) Semantic entropy probes: robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927. Cited by: §1, §2. B. Lester, R. Al-Rfou, and N. Constant (2021) The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 3045–3059. External Links: Link, Document Cited by: §1. J. Li, X. Cheng, X. Zhao, J. Nie, and J. Wen (2023) HaluEval: a large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 6449–6464. External Links: Link, Document Cited by: §1. P. Manakul, A. Liusie, and M. Gales (2023) SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 9004–9017. External Links: Link, Document Cited by: §2. S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi (2023) FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 12076–12100. External Links: Link, Document Cited by: §2. C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, R. Zhong, J. Song, and T. Zhang (2024) RAGTruth: a hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 10862–10878. External Links: Link, Document Cited by: §1. O. Obeso, A. Arditi, J. Ferrando, J. Freeman, C. Holmes, and N. Nanda (2025) Real-time detection of hallucinated entities in long-form generation. arXiv preprint arXiv:2509.03531. Cited by: §1, §2. Reuters (2026) US appeals court orders lawyer to pay $2,500 over ai hallucinations in brief. Note: Accessed: 2026-05-17 External Links: Link Cited by: §1. B. Snyder, M. Moisescu, and M. B. Zafar (2024) On early detection of hallucinations in factual question answering. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 2721–2732. Cited by: §1, §2. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, and C. Lv (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4. L. Zhang, D. Song, Z. Wu, Y. Tian, C. Zhou, J. Xu, Z. Yang, and S. Zhang (2025) Detecting hallucination in large language models through deep internal representation analysis. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, p. 8357–8365. Cited by: §1, §2. Appendix A MedQA prompt This appendix provides the exact system prompt used for MedQA. In contrast to TriviaQA and GSM8K, MedQA requires a strict answer format because correctness is determined from the selected option letter. You are answering a medical multiple-choice question. Choose the single best answer from the provided options. Output exactly in the format: Answer: <LETTER> Do not provide any explanation. Appendix B Full plots This appendix provides the full plots for the two parameter studies discussed in the main text. Figure 7 includes the complete probing-layer sweeps for all model–dataset combinations and the full embedding-search results over insertion strategies and the number of prompt embeddings. Additional probing-layer results Additional embedding-search results Figure 7: Full probing-layer sweeps for all three models and three datasets and full embedding-search results for all model–dataset combinations.