Paper deep dive

How do LLMs Compute Verbal Confidence

Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, Petar Velickovic

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 81

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/22/2026, 5:59:00 AM

Summary

This paper investigates the internal mechanisms of verbal confidence in LLMs (Gemma 3 27B and Qwen 2.5 7B), testing whether confidence is computed 'just-in-time' or via 'cached retrieval'. Through activation steering, patching, noising, and swap experiments, the authors provide evidence that confidence is automatically computed during answer generation, cached at the post-answer-newline (PANL) token, and later retrieved at the confidence-colon (CC) token. The findings suggest that verbal confidence reflects a sophisticated self-evaluation of answer quality rather than a simple readout of token log-probabilities.

Entities (5)

Gemma 3 27B · llm · 100%Qwen-2.5-7B · llm · 100%CC · token-position · 98%PANL · token-position · 98%TriviaQA · dataset · 95%

Relation Signals (3)

PANL → cachesrepresentationfor → Verbal Confidence

confidence 95% · confidence is gathered from answer tokens, cached at the first post-answer position

Gemma 3 27B → usesmechanism → cached retrieval

confidence 95% · we provide convergent evidence for cached retrieval

CC → retrievesrepresentationfrom → PANL

confidence 92% · confidence-relevant information is first consolidated at PANL before being transferred to CC

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Verbal confidence -- prompting LLMs to state their confidence as a number or category -- is widely used to extract uncertainty estimates from black-box models. However, how LLMs internally generate such scores remains unknown. We address two questions: first, when confidence is computed - just-in-time when requested, or automatically during answer generation and cached for later retrieval; and second, what verbal confidence represents - token log-probabilities, or a richer evaluation of answer quality? Focusing on Gemma 3 27B and Qwen 2.5 7B, we provide convergent evidence for cached retrieval. Activation steering, patching, noising, and swap experiments reveal that confidence representations emerge at answer-adjacent positions before appearing at the verbalization site. Attention blocking pinpoints the information flow: confidence is gathered from answer tokens, cached at the first post-answer position, then retrieved for output. Critically, linear probing and variance partitioning reveal that these cached representations explain substantial variance in verbal confidence beyond token log-probabilities, suggesting a richer answer-quality evaluation rather than a simple fluency readout. These findings demonstrate that verbal confidence reflects automatic, sophisticated self-evaluation -- not post-hoc reconstruction -- with implications for understanding metacognition in LLMs and improving calibration.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

80,384 characters extracted from source content.

Expand or collapse full text

How do LLMs Compute Verbal Confidence? Dharshan Kumaran Arthur Conmy Federico Barbero Simon Osindero Viorica Patraucean Petar Veličković Abstract Verbal confidence—prompting LLMs to state their confidence as a number or category—is widely used to extract uncertainty estimates from black-box models. However, how LLMs internally generate such scores remains unknown. We address two questions: first, when confidence is computed – just-in-time when requested, or automatically during answer generation and cached for later retrieval; and second, what verbal confidence represents – token log-probabilities, or a richer evaluation of answer quality? Focusing on Gemma 3 27B and Qwen 2.5 7B, we provide convergent evidence for cached retrieval. Activation steering, patching, noising, and swap experiments reveal that confidence representations emerge at answer-adjacent positions before appearing at the verbalization site. Attention blocking pinpoints the information flow: confidence is gathered from answer tokens, cached at the first post-answer position, then retrieved for output. Critically, linear probing and variance partitioning reveal that these cached representations explain substantial variance in verbal confidence beyond token log-probabilities, suggesting a richer answer-quality evaluation rather than a simple fluency readout. These findings demonstrate that verbal confidence reflects automatic, sophisticated self-evaluation—not post-hoc reconstruction—with implications for understanding metacognition in LLMs and improving calibration. Machine Learning, metacognition, confidence, mechanistic interpretability, LLM, ICML 1 Introduction Confidence—a model’s estimate that its answer is correct (Pouget et al., 2016)—is critical to LLM reliability . Several approaches have been developed to extract confidence estimates from LLMs. Token likelihoods yield well-calibrated confidences in multiple-choice or yes/no settings (Kadavath et al., 2022; Steyvers et al., 2025b; Steyvers & Peters, 2025); sampling-based consistency methods offer an alternative by measuring agreement across multiple model outputs (Geng et al., 2023; Tian et al., 2023). However, most deployed models are black-box systems that do not expose token-level probabilities, limiting the applicability of these methods. This has motivated research into obtaining verbal confidence measures, where models are explicitly prompted to state their confidence as a number or confidence class (e.g., “Almost certain”) (Xiong et al., 2023; Geng et al., 2023; Yoon et al., 2025; Steyvers et al., 2025a, b). Despite growing interest in verbal confidence as a practical measure of LLM uncertainty, little is understood about how such scores are internally generated. Understanding how LLMs compute confidence matters for several reasons. First, mechanistic insight could reveal whether verbal confidence reflects a meaningful self-evaluation of answer quality or merely surface-level correlates of generation fluency. This could thereby shed light on the metacognitive capacities of LLMs, and the observed difference in calibration between explicit verbal reports and token-level likelihoods (Xiong et al., 2023; Steyvers et al., 2025a). Second, understanding these mechanisms could enable principled interventions to improve calibration, moving beyond prompt engineering or generic fine-tuning. We ask two related questions. Our first question concerns when verbal confidence is computed—only when explicitly requested (i.e. at the last token of the prompt; a colon – see Figure 1), or automatically during answer generation before the model knows a confidence rating will be required (i.e. at answer-adjacent tokens; the newline token in Figure 1). Under the just-in-time (JIT) hypothesis, no dedicated confidence computation occurs during answer generation; instead, the model computes confidence only when prompted to self-evaluate, integrating features of the question and answer at that moment. Under the cached retrieval hypothesis, however, confidence is computed automatically as the answer is produced and stored for later retrieval when verbalization is required. Our second question asks what is verbal confidence—a readout of generation fluency and token log-probabilities, or a richer evaluation of question-answer fit? Figure 1: Main Prompt and Illustration of our findings. We included the generated answer (example question shown) from a previous phase as part of the prompt for the confidence rating experiment (see §C.1.2). Since the Transformer’s forward pass is a function of previous tokens, providing the answer as context yields the exact same representation at the PANL as autoregressive generation. See §B for full prompt used. We provide convergent evidence that LLMs compute confidence via cached retrieval rather than just-in-time computation, and that verbal confidence doesn’t merely reflect logprobs. (A) Confidence information is gathered at the post-answer-newline token (\ , PANL) via attention to answer tokens, particularly the last answer token, at earlier layers (21–25). (B) This information is routed to the confidence-colon token (:)—either directly or through intermediate tokens. (C) Confidence information persists in the residual stream at the confidence-colon through later layers (30–35). (D) Confidence is verbalized when C’s representation is transformed by the unembedding matrix at the final layer (layer 61). (E) Attention blocking experiments rule out just-in-time (JIT) computation: C does not compute confidence from scratch by attending to question or answer tokens (red arrow from C to Q and A tokens). (F) Decoding experiments reveal that verbal confidence is not explained by token log-probabilities, suggesting they reflect a more sophisticated evaluation of question-answer fit. These questions parallel a longstanding debate in decision neuroscience concerning first-order versus second-order accounts of confidence (Fleming & Daw, 2017; Kepecs et al., 2008; Kiani & Shadlen, 2009)(see §A). Under first-order accounts, confidence arises from the same internal signals that drive the decision itself—confidence is a direct readout of decision-variable strength. In LLMs, a first-order account would hold that verbal confidence is simply a readout of token log-probabilities: the same signal that determined which answer tokens to generate also determines confidence. Under second-order accounts, confidence involves a distinct computation that evaluates the decision, drawing on partially independent signals. For LLMs, this would mean verbal confidence reflects something richer than log-probabilities—an evaluation of question-answer fit that goes beyond generation fluency. A key empirical implication is that second-order architectures can support error detection—recognizing that a response may be wrong even after committing to it—whereas pure first-order architectures cannot, because confidence and decision accuracy are yoked to the same underlying signal. 2 Experiments We focus our experiments on Gemma 3 27B (Team et al., 2025) – given that this model allows access to the internal representations – and a prompt that asks for a class-based confidence report (Yoon et al., 2025) (see §B for full prompt used). We also report results with a prompt asking for a numeric confidence score (see Figure 12) and with Qwen 2.5 7b. We used the TriviaQA dataset which tests factual knowledge (Joshi et al., 2017). Given the differing goals of our paper in comparison to that of Yoon et al. (2025), we deliberately suppress chain-of-thought reasoning by instructing the model to output only a confidence classification. This setting is practically relevant: many applications use LLMs for auto-rating or grading tasks without chain-of-thought to reduce costs. Moreover, even in reasoning models that externalize their thought process, decisions such as backtracking must be driven by latent signals computed within the forward pass rather than derived from externalized tokens (Venhoff et al., 2025). Understanding how such internal representations are formed and accessed is therefore relevant beyond the no-CoT setting we study here. Our simplified setup allows us to more directly trace how confidence information flows through the network. Gemma was reasonably well calibrated (ECE = 0.12, AUROC = 0.71), and used both ends of the confidence class spectrum with reasonable frequency (though weighted towards higher confidence answers: see Figure 9). 2.1 Activation Steering We first focus our attention on when – that is at what token position, and at which layers – confidence is represented in the model. If verbal confidence reflects access to meaningful internal uncertainty signals, then those signals should be instantiated in the model’s activation space. This motivates the use of activation steering as a mechanistic probe (Turner et al., 2023; Stolfo et al., 2024a; Panickssery et al., 2023; Hua et al., 2025; Rai et al., 2024). Prior work shows transformers encode abstract properties such as love or hate as linear directions in activation space (Turner et al., 2023), implying high- and low-confidence trials should differ along identifiable directions. We apply this framework to the generation of verbal confidence reports. By extracting confidence-encoding directions at different layers and token positions, and assessing how strongly they modulate expressed confidence when applied, we can track where confidence-relevant information is present in the network (see §C.1.3 for methodological details). Notably, we ruled out the possibility that steering vectors capture linguistic hedging rather than confidence per se, by verifying using GPT-4o-mini that none of the answers in our dataset – which were typically a few tokens in length – contained hedging language (e.g., “maybe”, “probably”, “perhaps”). Activation steering, therefore, allows us to distinguish between competing accounts of confidence generation. Under the just-in-time account, no confidence-specific computation occurs until the last token in the prompt (i.e. C token—see Figure 1). Steering at the answer-adjacent token PANL should therefore be ineffective—PANL may carry answer-related information, but not a dedicated confidence representation that can be directly modulated. This follows because causal attention prevents PANL from attending to tokens that appear later in the prompt, so the model cannot ”know” at PANL that a confidence rating will be requested. The cached retrieval hypothesis, however, holds that confidence is already encoded at PANL; steering should modulate this representation and bias downstream output. Note that both accounts predict that steering at C can influence confidence, but only the cached account predicts that (1) steering at PANL is effective and (2) PANL effects emerge at earlier layers than C effects, reflecting the sequential flow from confidence encoding to retrieval. We found evidence consistent with the cached retrieval account. High- (or low-) confidence steering caused a marked boost (or depression) in verbal confidence ratings when vectors were injected at PANL, with efficacy peaking at layers 21-25 (see Figure 2). Substantial steering efficacy was also observed at C, peaking later at layers 30-35 —consistent with the prediction that automatically-formed confidence representations at PANL are subsequently transferred to C for verbalization. As expected, steering was ineffective at control positions: PANL+1 and the first confidence colon token. Figure 2: Results of Activation Steering In Gemma 3 27B. High (green lines) and low confidence (red lines) steering, at scales of 2 (solid line) and 5 (dotted line). Key positions: PANL (post-answer-newline) token, C (confidence-colon) token. Control positions: PANL+1 (token immediately after PANL), FCC (first-confidence-colon) token (i.e. token preceding “$CLASS” in the prompt, following the confidence instructions; see Figure 8). Baseline confidence was 0.55 across all trials. See Error bars show SEM (n=200 trials). Steering at the first answer token was ineffective. Steering at the last answer token was effective, consistent with confidence-relevant information being available at this point (i.e., after encoding the full question and most of the answer)(see Figure 10). However, we do not consider this position in subsequent analyses: unlike PANL, which immediately follows the answer, the last answer token is itself part of the answer content. This conflates two distinct roles—as a potential locus of cached confidence representations, and as part of the semantic content (answer correctness) from which confidence is presumably derived—introducing a confound that complicates interpretation. 2.2 Activation Patching Having established through activation steering experiments that PANL and C are key points where confidence is represented in the model, we next tested whether these positions are sufficient to restore confidence when the model’s ability to assess answer quality has been disrupted. To do this, we employed activation patching with a corrupt-then-restore paradigm (Meng et al., 2022; Heimersheim & Nanda, 2024; Zhang & Nanda, 2023)(see §A for more detailed literature review). First, we disrupted the model’s access to answer information by replacing all answer token activations with mean activations computed from 100 separate calibration trials (see §C.1.4 for methods). This corruption was applied at the input embedding level, propagating through all subsequent layers and effectively preventing the model from assessing whether its answer was correct. We restricted analysis to high-confidence trials because these provide the clearest test: corruption should substantially reduce confidence, whereas low-confidence trials are already near floor and thus less sensitive to disruption. We first compared the corrupted condition to a clean run where answer tokens were untouched. We focus on three metrics throughout this and subsequent analyses (see §C.1.5 for details): (1) confidence change, the difference in reported confidence between conditions (mapping confidence classes to their numeric midpoints); (2) logit difference change, the change in the margin between the clean trial’s confidence class logit and the mean logit of alternatives; we use this metric following (Wang et al., 2023; Heimersheim & Nanda, 2024; Rai et al., 2024), rather than probability because the model’s computations are linear in logit-space until the final softmax; and (3) first token change rate, the proportion of trials where the argmax confidence token differed from baseline. Notably, we modified the prompt from Yoon et al. (2025) to ensure that each confidence class begins with a unique token, enabling meaningful analysis of metrics (2) and (3) (see Figure 8). As expected, we found that during the corrupted run (i.e. after answer token corruption but without patching) the model’s confidence decreased to the lowest class, the difference between the logit of the first token outputted in the clean run and the mean of all other confidence classes collapsed to near zero, and there was a 100% change in the first token outputted by the model (see Figure 3). We then selectively restored (i.e. patched) clean activations at at a single position (e.g. PANL) and at a single layer. The logic of this intervention is as follows: if a position at a given layer contains sufficient information to drive confidence output, then restoring clean activations there should recover the model’s original confidence behavior despite the corrupted answer tokens propagating through the rest of the network. We tested three positions: PANL, C, and PANL+1 as a control. Indeed, we found that patching of the C and PANL representations – but not the PANL+1 representation – effected substantial recovery of all 3 metrics (confidence, logit difference, rate of change of first token). Figure 3: Results of Activation Patching in High Confidence Trials: Confidence Class Prompt. Clean baseline shown in green; corrupt baseline shown in red (i.e. at near zero for logit difference and confidence, and near 100 for first token change rate). Patching of PANL representation resulted in partial recovery of logit difference, first token and confidence (upper, middle, lower panel respectively). Patching of C representation resulted in near complete recovery of confidence, logit difference and first token. PANL+1 patching resulting in effectively zero recovery. Near-ceiling C recovery at late layers is expected: C’s residual stream is directly transformed by the unembedding matrix, bypassing upstream corruption. More informative is the layer-wise pattern of recovery: PANL patching achieved peak recovery earlier in the network (layer 25) compared to C (which rose sharply only after layer 30). This temporal precedence is consistent with a cached retrieval mechanism in which confidence-relevant information is first consolidated at PANL before being transferred to C for final output. That PANL patching yielded only partial recovery is expected and consistent with broader findings in mechanistic interpretability. While some circuits can be cleanly isolated—such as the indirect object identification circuit (Wang et al., 2023)—recent work has shown that model behaviors typically arise from many overlapping heuristics rather than a single mechanism (Lindsey et al., 2025; Ameisen et al., 2025). Our intervention restored a single position at a single layer while answer token corruption continued to propagate through all other positions and layers; full recovery would require patching the complete distributed circuit. 2.3 Activation Noising To test whether PANL and C representations are necessary for confidence reporting, we performed activation noising (mean ablation) experiments (Meng et al., 2022; Wang et al., 2023; Rai et al., 2024); see §C.1.6 for methods). For each position, we replaced its residual stream activation with the mean activation computed from a balanced set of 50 high-confidence and 50 low-confidence trials. We focussed on the two metrics that best capture disruption to the model’s confidence-reporting mechanism: firstly the change in logit difference between clean and noised run, and secondly the first token change rate induced by noising. We observed that mean ablation of PANL (peak layer 25) and C (rising after layer 30) causes partial disruption of verbal confidence reporting, whilst no effect was observed at the PANL+1 control position (See Figure 11). This is consistent with the necessity of PANL and C representations for faithful verbal confidence reporting. The partial rather than complete disruption may reflect either the distributed nature of confidence encoding across layers—such that ablating a single layer leaves other layers’ contributions intact—or functional redundancy whereby alternative computational pathways can partially compensate for the ablated representation. 2.4 Activation Swap Experiment The preceding experiments provide convergent evidence that PANL and C carry confidence representations: steering vectors extracted from both positions produce graded, bidirectional modulation of confidence reports; patching partially restores confidence after answer corruption; and noising either position partially disrupts confidence output. Critically, effects at PANL emerge at earlier layers than at C across all three experiments—consistent with the cached retrieval hypothesis in which confidence is first encoded at PANL and subsequently transferred to C for verbalization. Control positions (PANL+1, FCC) show no such effects. To rule out the alternative that PANL encodes answer content rather than confidence, we designed an activation swap experiment (see Figure 4 and §C.1.7 for methods). This approach draws on the logic of interchange intervention (Geiger et al., 2021): if PANL caches a confidence representation, then transplanting PANL residual stream activations from a low (or high) confidence donor trial into a high (or low) confidence recipient trial in a cross-confidence swap trial should systematically bias the recipient toward the donor trial’s confidence level—even though the recipient’s question and answer remain unchanged. By contrast, if PANL primarily encodes content-specific features, then such cross-confidence swaps should induce only generic disruption effects that do not depend on the donor’s confidence and are of similar magnitude to those observed in same-confidence control swap trials. Figure 4: Illustration of Activation Swap Experiment. Upper panel: High→ (i.e. cross-confidence swap: high confidence recipient trial receives low confidence donor representation) – result is a lowering of confidence. Lower panel: High→ (i.e. high confidence recipient trial receives High confidence donor representation) – in this same-same confidence swap, the result is no change in confidence. We observe that cross-confidence swaps at PANL (High→ i.e. high confidence recipient trial receives low confidence donor representation) and (Low→ ) produce systematic directional shifts in confidence—decreasing and increasing reported confidence, respectively—beyond the baseline disruption observed in same-confidence controls (see Figure 5). These effects peak at layer 26 and are evident across all three metrics: first token change rate, logit difference change, and confidence change. The same pattern emerges at C but at later layers (see Figure 17), consistent with the temporal precedence observed in previous experiments. No such effects are apparent at the PANL+1 control position (see Figure 17). These results provide compelling evidence that PANL representations carry confidence-specific information that transfers across trials—ruling out the possibility that PANL merely encodes content features that correlate with confidence. Figure 5: Results of Activation Swap Experiment at PANL position. Same-confidence swaps (H→ , L→ ) control for generic content-related effects due to introducing activations from a different trial; cross-confidence swaps (H→ , L→ ) isolate confidence-specific transfer. See main text for details. See Figure 17 for results at C and PANL+1 positions. 2.5 Decoding Confidence Information The preceding experiments establish when and where confidence information is represented in the network. Specifically, we show that PANL and C play causal roles in verbal confidence generation, with PANL’s influence emerging earlier in the network – supporting the cached retrieval hypothesis. However, these interventions do not directly reveal what information is encoded at these positions. While causal interventions test sufficiency/necessity, linear probing reveals where information first becomes decodable. We address two questions. First, is confidence information decodable from PANL at earlier layers than C, as predicted by the cached retrieval hypothesis? Second, what type of confidence signal is encoded—do PANL representations simply summarize token log-probabilities (a white-box confidence measure that prior work has found to be better calibrated than verbal reports; Steyvers et al., 2025a; Kadavath et al., 2022), or do they reflect a distinct computation? To address these questions, we trained linear probes to decode two measures from residual stream activations: answer correctness (binary classification, evaluated via AUROC) and verbal confidence magnitude (continuous regression using class midpoints, evaluated via R2R^2)(see §C.1.8 for methods). We additionally performed variance partitioning to assess whether PANL activations explain confidence variance beyond what is accounted for by answer token log-probabilities—as would be expected if PANL representations reflect a distinct computation rather than a simple summary of token likelihoods. Consistent with the cached retrieval hypothesis, information about both correctness and confidence magnitude is decodable from PANL at earlier layers than from C (see Figure 6, top and middle panels). Critically, variance partitioning revealed that activations at PANL and C explain substantial unique variance beyond log-probabilities, confirming that confidence representations at these positions are not reducible to token probability signals (see Figure 6, bottom panel). This suggests they reflect a more sophisticated evaluation about question-answer fit, and a distinct confidence signal – consistent with second-order models of confidence in neuroscience (see §A for details). Notably, whilst log-probabilities explained only 8.4% of variance in verbal confidence (r=0.29r=0.29, RCV2=0.084R^2_CV=0.084; see Supplemental Methods), they were significant indicators of correctness (logprob AUROC = 0.75; see Figure 6) – as would be expected based on their widespread useage as a measure of confidence. Figure 6: Results of decoding: linear probes were trained on residual stream activations at PANL, PANL+1, C, intermediate tokens (first token of each confidence class in the instruction), and a control position (third question token). For binary correctness we report AUROC; for verbal confidence magnitude we report R2R^2. Horizontal lines show AUROC obtained using mean answer log-probability or verbal confidence directly. Key results: (1) confidence information is decodable from PANL at earlier layers than all other positions; (2) variance partitioning shows PANL activations explain substantial variance beyond log-probabilities (RCV2=0.084R^2_CV=0.084 for logprobs alone), indicating PANL encodes information not captured by token likelihoods. Our results also show that information about correctness (AUROC) and verbal confidence (R2) is distributed throughout the model – even at sites where steering (or patching type manipulations) has no effect (e.g. PANL+1, first-confidence colon; see Figure 2). This is consistent with previous studies showing information may be present in networks – and detected by decoding techniques – but not be observed in behavior (Azaria & Mitchell, 2023; Burns et al., 2022; Li et al., 2023; Liu et al., 2025; Bürger et al., 2024)(see §A for more detailed literature review). Furthermore, this finding aligns with prior work cautioning against over-interpretation of probing results without complementary causal interventions (Elazar et al., 2021; Ravichander et al., 2021). 2.6 Attention Blocking Experiments The preceding experiments establish that confidence information is encoded at PANL before C, consistent with cached retrieval. However, three questions remain. First, can we exclude the possibility that just-in-time computation also occurs at the C token through attention directed at question and answer tokens? Second, how does confidence information flow from PANL to C, a process necessary for verbalization of this information? Third, how does confidence information reach PANL in the first place? Following Geva et al. (2023), we address these questions by selectively blocking attention edges between positions. Figure 7: Attention blocking reveals information flow from answer tokens through PANL to C. Results from selectively blocking attention edges across 12 consecutive layers centered at each x-axis position, using a minimal numeric (0–9) confidence prompt. (A) Blocking C→ (purple) produces significant disruption peaking at layers 30–36, while blocking C→ +A (pink) shows no effect beyond the PANL+1 control (orange), ruling out just-in-time computation. (B) Blocking PANL’s attention to answer tokens—either the last answer token (PANL→ _A, brown) or all answer tokens (PANL→ , gray)—produces disruption at earlier layers (22–28), preceding the C→ effect. This temporal ordering indicates sequential information flow: answer tokens → PANL → C. Upper panels show token change rate (%); lower panels show logit difference change (baseline token logit minus mean of alternative digits). Error bars indicate SEM (n=500n=500 trials). To address the first question, we blocked C’s attention to question and answer tokens (across all layers) using the categorical confidence prompt (see §C.1.10 for methods). This produced minimal effects (∼ 10% change rate), equivalent to a control condition blocking C’s attention to PANL+1—ruling out the just-in-time hypothesis that C integrates question/answer information de novo (Figure 18). However, blocking C→ also produced no effect with this prompt (Figure 18), suggesting that confidence information may flow through intermediate template tokens – of which there are more than a hundred in the categorical prompt – rather than via a direct pathway (see §D for details). To test whether C retrieves information from PANL, we employed a pared down version of the numeric prompt that reduces the number of intermediate tokens between PANL and C (see Figure 13). This prompt elicits numeric confidence (0–9) rather than categorical classes. Calibration was reasonable (ECE = 0.17; separate set of 2000 questions) and the model’s confidence reliably discriminated correct from incorrect responses (AUROC = 0.68), confirming the presence of a meaningful confidence signal suitable for mechanistic analysis. We focus on analogous positions: PANL (the newline following the answer), PANL+1 (control), and C (the final token before confidence generation) – and report the results using this minimal prompt below: C does not compute confidence from scratch. Blocking C→ +A produced effects indistinguishable from the PANL+1 control condition (∼ 10% change rate; Figure 7A), replicating the null result from the categorical prompt and confirming that C does not perform just-in-time computation. C retrieves confidence information from PANL. In contrast to the categorical prompt, blocking C→ in the minimal prompt produced substantial disruption: token change rates peaked at ∼ 21% (layers 30–36), accompanied by marked reductions in logit difference (Figure 7A). This effect significantly exceeded the PANL+1 control, providing direct evidence that C retrieves confidence information from PANL when intermediate routing is minimized. PANL reads confidence from answer tokens. To trace the upstream pathway, we blocked PANL’s attention to answer tokens. Blocking PANL→ _A or PANL→ produced ∼ 20% token change rates peaking at earlier layers (22–28), with corresponding logit difference reductions (Figure 7B). These effects preceded the C→ effects by approximately 6–8 layers, consistent with a sequential information flow: answer tokens → PANL → C. See §D for details of complementary experiment with categorical prompt. Summary. These attention blocking experiments rule out just-in-time computation and reveal the pathway through which confidence information flows: it originates at answer tokens, is read by PANL at earlier layers (22–28), and is subsequently retrieved by C at later layers (30–36)(see Figure 1). The minimal prompt was essential for revealing the C→ pathway, which was likely masked by redundant multi-hop routing through intermediate template tokens in the full categorical prompt. 2.7 Generalization across Prompt Format and Architecture To assess generalization, we replicated all experiments using (1) a numeric (0-100) confidence prompt in Gemma 3 27B (see Figures 12, 14 and 19), and (2) the categorical prompt in Qwen 2.5 7B (see §C.3). The temporal precedence pattern—PANL effects emerging earlier than C effects—held across both variations and all intervention types (see Figure 19), indicating that cached retrieval reflects a general computational strategy rather than an artifact of prompt format or model architecture. 3 Related Work We highlight work most relevant to our own here, and refer the reader to a detailed discussion of related work in the Appendix (see §A). Geva et al. (2023) showed that during factual recall, LLMs automatically cache attributes at the last subject token in early layers, later retrieving them for output rather than computing them de novo at the prediction site—a pattern strikingly similar to our cached retrieval hypothesis for confidence. Separately, classifiers trained on LLM hidden states distinguish true from false statements more accurately than methods based on output probabilities (Azaria & Mitchell, 2023; Burns et al., 2022), suggesting that models encode richer information about output quality than surface-level signals reveal—consistent with our finding that confidence representations contain information beyond token log-probabilities. Finally, Stolfo et al. (2024b) identified neurons in the final MLP layer that regulate confidence expression by modulating LayerNorm scale, but the upstream computation that produces the confidence signals these neurons act upon remained unexplored—a gap our work addresses. 4 Conclusion Our results demonstrate that verbal confidence in LLMs reflects cached retrieval rather than just-in-time computation. Addressing when confidence is computed, we show that confidence representations emerge automatically during answer generation—before the model is aware a rating will be requested—rather than being constructed on-demand at verbalization. Addressing what confidence represents, we show that these cached representations explain substantial variance beyond token log-probabilities, suggesting a richer evaluation of question-answer fit consistent with a second-order model of confidence (Fleming & Daw, 2017) – rather than a simple fluency readout. The automatic computation of confidence alongside answer generation parallels recent findings in factual recall, where LLMs automatically enrich subject representations with many attributes during early layers before extracting the specific queried attribute at later layers (Geva et al., 2023). Our findings reveal that verbal confidence involves retrieval of automatically generated internal representations rather than post-hoc reconstruction from surface features—demonstrating a sophisticated metacognitive capacity in LLMs. This is consistent with recent evidence suggesting that LLMs possess some degree of introspective awareness (Anthropic, 2025), though whether the retrieval process we characterize constitutes introspection in a stronger sense remains an open question. Acknowledgments We thank Andrew Lampinen and Beningo Uria for comments on earlier versions of the paper. References Ameisen et al. (2025) Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., Olah, C., et al. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html. Anthropic (2025) Anthropic. Emergent introspective awareness in large language models. https://transformer-circuits.pub/2025/introspection/, 2025. Anthropic Research Report. Azaria & Mitchell (2023) Azaria, A. and Mitchell, T. The internal state of an llm knows when it’s lying. arXiv preprint arXiv:2304.13734, 2023. Bürger et al. (2024) Bürger, L., Hamprecht, F. A., and Nadler, B. Truth is universal: Robust detection of lies in llms. Advances in Neural Information Processing Systems, 37:138393–138431, 2024. Burns et al. (2022) Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. Devic et al. (2025) Devic, S., Peale, C., Bradley, A., Williamson, S., Nakkiran, P., and Gollakota, A. Trace length is a simple uncertainty signal in reasoning models. arXiv preprint arXiv:2510.10409, 2025. Elazar et al. (2021) Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021. Fleming & Daw (2017) Fleming, S. M. and Daw, N. D. Self-evaluation of decision-making: A general bayesian framework for metacognitive computation. Psychological review, 124(1):91, 2017. Geiger et al. (2021) Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. Geng et al. (2023) Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., and Gurevych, I. A survey of language model confidence estimation and calibration. arXiv preprint arXiv:2311.08298, 2023. Geva et al. (2023) Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023. Guo et al. (2017) Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning, p. 1321–1330. PMLR, 2017. Heimersheim & Nanda (2024) Heimersheim, S. and Nanda, N. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255, 2024. Hua et al. (2025) Hua, T. T., Qin, A., Marks, S., and Nanda, N. Steering evaluation-aware language models to act like they are deployed. arXiv preprint arXiv:2510.20487, 2025. Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. Kadavath et al. (2022) Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Kepecs & Mainen (2012) Kepecs, A. and Mainen, Z. F. A computational framework for the study of confidence in humans and animals. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1594):1322–1337, 2012. Kepecs et al. (2008) Kepecs, A., Uchida, N., Zariwala, H. A., and Mainen, Z. F. Neural correlates, computation and behavioural impact of decision confidence. Nature, 455(7210):227–231, 2008. Kiani & Shadlen (2009) Kiani, R. and Shadlen, M. N. Representation of confidence associated with a decision by neurons in the parietal cortex. science, 324(5928):759–764, 2009. Li et al. (2023) Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530, 2023. Lindsey et al. (2025) Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., Olah, C., et al. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html. Liu et al. (2025) Liu, J., Jain, J., Diab, M., and Subramani, N. Llm microscope: What model internals reveal about answer correctness and context utilization. arXiv preprint arXiv:2510.04013, 2025. Mei et al. (2025) Mei, Z., Zhang, C., Yin, T., Lidard, J., Shorinwa, O., and Majumdar, A. Reasoning about uncertainty: Do reasoning models know when they don’t know? arXiv preprint arXiv:2506.18183, 2025. Meng et al. (2022) Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. Panickssery et al. (2023) Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. M. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023. Pouget et al. (2016) Pouget, A., Drugowitsch, J., and Kepecs, A. Confidence and certainty: distinct probabilistic quantities for different goals. Nature neuroscience, 19(3):366–374, 2016. Rai et al. (2024) Rai, D., Zhou, Y., Feng, S., Saparov, A., and Yao, Z. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646, 2024. Ravichander et al. (2021) Ravichander, A., Belinkov, Y., and Hovy, E. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, p. 3363–3377, 2021. Steyvers & Peters (2025) Steyvers, M. and Peters, M. A. Metacognition and uncertainty communication in humans and large language models. Current Directions in Psychological Science, p. 09637214251391158, 2025. Steyvers et al. (2025a) Steyvers, M., Belem, C., and Smyth, P. Improving metacognition and uncertainty communication in language models. arXiv preprint arXiv:2510.05126, 2025a. Steyvers et al. (2025b) Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W., and Smyth, P. What large language models know and what people think they know. Nature Machine Intelligence, p. 1–11, 2025b. Stolfo et al. (2024a) Stolfo, A., Balachandran, V., Yousefi, S., Horvitz, E., and Nushi, B. Improving instruction-following in language models through activation steering. arXiv preprint arXiv:2410.12877, 2024a. Stolfo et al. (2024b) Stolfo, A., Wu, B., Gurnee, W., Belinkov, Y., Song, X., Sachan, M., and Nanda, N. Confidence regulation neurons in language models. Advances in Neural Information Processing Systems, 37:125019–125049, 2024b. Team et al. (2025) Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Tian et al. (2023) Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975, 2023. Turner et al. (2023) Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023. Varshney et al. (2023) Varshney, N., Yao, W., Zhang, H., Chen, J., and Yu, D. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987, 2023. Venhoff et al. (2025) Venhoff, C., Arcuschin, I., Torr, P., Conmy, A., and Nanda, N. Base models know how to reason, thinking models learn when. arXiv preprint arXiv:2510.07364, 2025. Wang et al. (2023) Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. ICLR, 2023. Xiong et al. (2023) Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023. Yoon et al. (2025) Yoon, D., Kim, S., Yang, S., Kim, S., Kim, S., Kim, Y., Choi, E., Kim, Y., and Seo, M. Reasoning models better express their confidence. arXiv preprint arXiv:2505.14489, 2025. Zhang & Nanda (2023) Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042, 2023. Appendix Overview • Appendix A: Related Work (§A) — Summary of related literature • Appendix B: Supplemental Figures (§B) — Prompts, calibration plots, and additional experimental results • Appendix C: Supplemental Methods (§C) – C.1 Experiments with Categorical Confidence Prompt in Gemma 3 27B (§C.1) * C.1.1 Technical Details (§C.1.1) * C.1.2 Generation of Answers (§C.1.2) * C.1.3 Activation Steering (§C.1.3) * C.1.4 Activation Patching (§C.1.4) * C.1.5 Metrics (§C.1.5) * C.1.6 Activation Noising (§C.1.6) * C.1.7 Activation Swap (§C.1.7) * C.1.8 Decoding Confidence Information (§C.1.8) * C.1.9 Token Log-Probability Extraction (§C.1.9) * C.1.10 Attention Blocking Method (§C.1.10) – C.2 Experiment using Numeric Confidence Prompt in Gemma 3 27B (§C.2) – C.3 Experiment using Qwen 2.5 7B (§C.3) • Appendix D: Supplemental Results (§D) – D.1 Attention Blocking with Categorical Confidence Prompt (§D.1) Appendix A Related Work Confidence and Calibration in LLMs. Confidence refers to a model’s estimate of the probability that its output is correct (Pouget et al., 2016), while calibration measures the alignment between expressed confidence and actual accuracy (Guo et al., 2017). Well-calibrated confidence estimates are critical for deploying LLMs in high-stakes applications and have may be useful for detecting hallucinations (Varshney et al., 2023). Several methods exist for eliciting confidence from LLMs: token-level likelihoods provide well-calibrated estimates, particularly for multiple-choice or yes/no verification tasks (Kadavath et al., 2022; Steyvers et al., 2025b; Steyvers & Peters, 2025), while sampling-based approaches assess confidence through consistency across multiple outputs (Geng et al., 2023; Tian et al., 2023). However, these white-box methods require access to internal probabilities unavailable in most deployed systems. This limitation has driven interest in verbal confidence, where models are prompted to explicitly report their confidence as a numeric value or categorical label (e.g., “Almost certain”) (Xiong et al., 2023; Geng et al., 2023; Yoon et al., 2025; Steyvers et al., 2025a). The class-based prompt used in our main experiments is derived from Yoon et al. (2025) – modified so that each first token is unique to the class. Latent Representations of Uncertainty and Correctness. A growing body of work demonstrates that LLMs encode information about the quality of their outputs within internal activations, often in ways that diverge from surface-level confidence. Azaria & Mitchell (2023) showed that models encode a latent representation of truthfulness in their hidden states: classifiers trained on these activations distinguish true from false statements with 60–80% accuracy, generalizing across topics and outperforming prompting-based methods. Critically, classifiers trained on these internal activations outperform methods based on output probabilities, which are confounded by factors such as sentence length and token frequency and thus provide less reliable signals of truthfulness. Similarly, Liu et al. (2025) found that activations at the first output token predict answer correctness with approximately 75% accuracy, and introduced metrics to distinguish correct, incorrect, and irrelevant retrieval contexts directly from model internals. These findings align with broader evidence that task-relevant information may be decodable from network activations yet remain dissociated from observed behavior (Burns et al., 2022; Li et al., 2023; Bürger et al., 2024). Mechanistic Interpretability: Activation Steering Activation steering is a technique for causally intervening on model behavior by adding or subtracting directions in activation space. Turner et al. (2023) demonstrated that abstract concepts such as love or hate are encoded as approximately linear directions in transformer representations. These directions can be recovered through various methods, including contrasting mean activations between conditions that differ along a conceptual dimension. Subsequent work has applied steering to modify instruction-following behavior (Stolfo et al., 2024a), reasoning (Venhoff et al., 2025), sycophancy and other traits (Panickssery et al., 2023), and evaluation-aware responses (Hua et al., 2025). Activation Patching. Activation Patching (also termed causal tracing) identifies which model components are causally responsible for specific behaviors. Meng et al. (2022) used the corrupt-then-restore paradigm: inputs are corrupted to disrupt model output, then clean activations are selectively restored at specific position-layer combinations to measure recovery. Positions that restore performance when patched are implicated as causally sufficient for the computation. This approach has been extended to study indirect object identification (Wang et al., 2023), and best practices for activation patching metrics have been systematically evaluated (Zhang & Nanda, 2023; Heimersheim & Nanda, 2024). Activation Noising. Activation noising (or ablation) assesses the necessity of specific model components by removing their contribution during the forward pass and measuring the resulting impact on model behavior (Meng et al., 2022; Wang et al., 2023; Rai et al., 2024). To improve the reliability of these interventions, mean ablation replaces a component’s activation with an average computed across multiple samples from the data distribution (Wang et al., 2023). This approach mitigates the out-of-distribution (OOD) effects associated with zero or random noising by ensuring the intervention remains grounded in valid, in-distribution model states (Zhang & Nanda, 2023). Interchange Intervention. Geiger et al. (2021) formalize activation swapping as a rigorous test for causal abstraction: a neural representation can be shown to encode a specific high-level concept if swapping it systematically controls the model’s output behavior. This framework provides the theoretical foundation for our activation swap experiments, establishing that such interventions are the standard method for demonstrating a causal link between an internal state and a high-level variable. Confidence Regulation Neurons. Stolfo et al. (2024b) identify two classes of neurons in the final MLP layer that regulate output confidence: entropy neurons, which modulate the LayerNorm scale by writing to an effective null space of the unembedding matrix, and token frequency neurons, which shift the output distribution toward or away from the unigram distribution. Their analysis demonstrates that these components serve a calibration function, with entropy neurons acting as a hedging mechanism that increases uncertainty to mitigate loss spikes on incorrect predictions. This work provides evidence that focuses on output-level calibration rather than the computation of confidence representations during answer generation. Attention Blocking. Geva et al. (2023) employ attention blocking to demonstrate that factual recall relies on a distinct cache-and-retrieve mechanism, where knowledge is aggregated and stored at the last subject token in early layers rather than being accessed directly at the output. They show that this token acts as a temporary holding state for factual attributes, which are subsequently retrieved by the final prediction token via specific attention heads in the network. This establishes that information flow is spatially localized, relying on intermediate tokens to compute and maintain latent states during the forward pass. Theories of Confidence in Decision Neuroscience. Our investigation connects to a longstanding debate in decision neuroscience concerning the computational basis of confidence (Fleming & Daw, 2017; Pouget et al., 2016; Kepecs & Mainen, 2012; Kiani & Shadlen, 2009). Under first-order accounts, confidence arises from the same internal signals that drive the decision itself. In perceptual tasks, for instance, both the choice and confidence derive from a single decision variable representing accumulated evidence; confidence is simply a readout of how strongly this variable favored the chosen option. Translated to LLMs, a first-order account would hold that verbal confidence is a readout of token log-probabilities—the same signals that determined which answer tokens to generate also determine confidence. Under second-order accounts, confidence involves signals that are distinct from—though correlated with—those driving the decision (Fleming & Daw, 2017). These additional signals enable an evaluation of the decision that goes beyond the information directly used to produce it. For LLMs, evidence that verbal confidence reflects information beyond token log-probabilities would suggest a second-order-like computation capable of genuine answer-quality evaluation. An important consequence is that second-order architectures can support error detection: because confidence draws on partially independent information, the system can recognize that a response may be wrong even after committing to it. In contrast, pure first-order architectures cannot detect errors, because confidence and decision accuracy are yoked to the same underlying signal. Appendix B Supplemental Figures Figure 8: Full categorical confidence class prompt in Experiment. We focused our analysis on the newline token following the model’s answer (given in a previous phase): the post-answer-newline (PANL) token, and the confidence-colon token (i.e. the last token of the prompt). In addition, we report analyses on the token immediately following the PANL token (i.e. PANL-plus1 token), the first-confidence-colon (FCC)(i.e. the colon preceding the appearance of “$CLASS”), and the last token of the answer. This prompt is derived from (Yoon et al., 2025) but with the following key modification: the first token of every confidence class is unique, allowing us to meaningfully analyze changes in the ID of first token, the logit of the first token etc. Figure 9: Calibration and Distribution of Confidence Classes in Gemma. (A) Calibration of Gemma: Expected Calibration Error (ECE) = 0.12, AUROC = 0.71. No procedures such as temperature scaling (Guo et al., 2017) were used, since we were focussed on understanding the generation of Gemma’s raw verbal confidence signals. The model’s performance was 77.4%; this was determined by having GPT4o-mini mark questions (B) Distribution of Gemma’s confidence responses across the 10 classes. n = 7858 questions from the TriviaQA dataset (Joshi et al., 2017). Figure 10: Results of Activation Steering in Gemma at answer tokens (Confidence Class Prompt). High (green lines) and low confidence (red lines) steering, at scales of 2 (solid line) and 5 (dotted line). Error bars show SEM. Figure 11: Results of Activation Noising across all trials. Mean ablation of representations of PANL or C token at a single layer causes disruption of verbal confidence reporting as measured by decrease in logit difference, and change in first token outputted by model. See text for details Figure 12: Full numeric confidence prompt used in Experiment. This prompt is derived from (Mei et al., 2025; Devic et al., 2025) Figure 13: Minimal numeric confidence prompt used in attention blocking experiments. This prompt elicits confidence on a 0–9 scale (single token output) and minimizes intermediate tokens between the post-answer-newline token (PANL, position 1) and the confidence-colon token (C, position 2). These positions are analogous to those in the main categorical prompt (Figure 8). The minimal design enables direct testing of whether C retrieves confidence information from PANL, a pathway that may be masked by routing through intermediate template tokens in the categorical prompt. Figure 14: Calibration and Distribution of Numeric Confidence Scores in Gemma. (A) Calibration of Gemma: Expected Calibration Error (ECE) = 0.16, AUROC = 0.73. No procedures such as temperature scaling (Guo et al., 2017) were used, since we were focussed on understanding the generation of Gemma’s raw verbal confidence signals. (B) Distribution of Gemma’s numeric confidence responses binned into 10 bins. Questions (n = 8008) the TriviaQA dataset (Joshi et al., 2017). Figure 15: Calibration and Distribution of Categorical Confidence Ratings in Qwen 2.5 7b. (A) Calibration of Qwen: Expected Calibration Error (ECE) = 0.06, AUROC = 0.65. No procedures such as temperature scaling (Guo et al., 2017) were used, since we were focussed on understanding the generation of Qwen’s raw verbal confidence signals. (B) Distribution of Qwen’s confidence responses. Figure 16: Results of Activation Steering in Gemma with Numeric Confidence Prompt. High (green lines) and low confidence (red lines) steering, at scales of 2 (solid line) and 5 (dotted line). n = 124 trials per condition per layer. Positions correspond to the analogous locations in the categorical confidence prompt. Error bars show SEM. Figure 17: Results of Activation Swap Experiment at C and PANL+1 position (Confidence Class Prompt). Same-confidence swaps (H→ , L→ ) control for cross-trial substitution effects; cross-confidence swaps (H→ , L→ ) isolate confidence-specific transfer. See main text for details. Figure 18: Results of Attention Blocking Experiment using Main Categorical Prompt. First token change rate induced by blocking different attentional pathways shown in upper panel. Lower panel: Change in logit difference between the first token and the mean of alternative confidence classes caused by attention blocking. PANL and PANL+1 denoted by NL and NL+1 for brevity. last_A and A refer to last answer token and answer tokens, respectively. Q refers to all question tokens. See text for interpretation. Figure 19: Results for Gemma 3 27B and Qwen 2.5 7B on Activation Steering, Patching, Noising and Swap Experiments: Categorical Confidence and Numeric Confidence Prompt. Maximal effects observed at each position’s (PANL, PANL+1, C) peak layer (layer index below bar). Qwen has 28 layers, Gemma 3 27B has 62 layers. Logit difference change (i.e. change from logit of clean run first token vs average of all other confidence classes), and confidence change reported; except for noising experiment where first token change rate and logit difference are reported as measures of confidence reporting disruption (see main text for details). Baseline confidence in Qwen categorical prompt steering experiment was 0.56; in Gemma it was 0.48. Baseline in Gemma numeric prompt experiment was 0.54. Significant effects observed at PANL and C positions compared to control PANL+1 position; critically effects at the C position occurred at later layers compared to those at PANL. Comparable effects observed in both models except for activation steering where stronger effects observed in Gemma. Appendix C Supplemental Methods C.1 Experiments with Categorical Confidence Prompt in Gemma 3 27B C.1.1 Technical Details For categorical confidence experiments (Gemma 3 27B and Qwen 2.5 7B), we performed single forward passes and extracted next-token logits, as categorical confidence is determined by the first generated token. For both models, we verified that forward-pass argmax tokens matched generation outputs and corresponded to valid confidence classes. Critically, the first token of each confidence class was unique (see Figure 8). Gemma 3 27B (google/gemma-3-27b-it) was loaded via Hugging Face. For attention analysis experiments requiring access to attention weights, an eager attention implementation was used. The model has 62 layers; layer sweeps were conducted across layers 0–61 with dense sampling in layers 20–35 where effects were strongest. The model was run in evaluation mode with greedy decoding (temperature = 0). C.1.2 Generation of answers for use in Categorical prompt In phase 0, Gemma was presented with a prompt essentially identical to that shown in Figure 8, except that confidence instructions appeared at the start. This phase was used to generate the model’s answers for use in the main confidence rating experiment. Duplicates were removed from the TriviaQA dataset downloaded via the HuggingFace platform. C.1.3 Activation Steering We collected residual stream activations at several critical and control points in the prompt across layers in the network, using a separate set of questions as used in the main activation steering experiment (n = 3000). The critical points of interest were the post-answer-newline token and the confidence-colon token (see Figure 8). The control positions were the first-confidence-colon token (i.e. the token preceding the appearance of the phrase “$CLASS”, and following the instruction about confidence classes); the post-answer-newline-plus1 token (i.e the first token after the PANL token). We also examined the effect of steering at the first and last tokens of the answer. For steering and other experiments we sampled layers 0, 10, 15, 20–35 (densely sampled), 40, 50, and 61. We created high and low confidence vectors by contrasting high and low confidence trials (all trials that the model scored correctly), following standard procedures in activation steering (Turner et al., 2023; Stolfo et al., 2024a; Panickssery et al., 2023; Hua et al., 2025). Creation of high- and low-confidence steering vectors: We constructed steering vectors by contrasting the 25 highest ranked trials (i.e. “Almost certain”) by confidence and 25 lowest ranked trials (i.e. “No chance”). ℋ=25 high-confidence trials,ℒ=25 low-confidence trials.H=25 high-confidence trials, =25 low-confidence trials. high=μ(ℋ)−μ(ℒ),low=−high,v_high\;=\;μ(H)\;-\;μ(L), _low\;=\;-v_high, where μ(⋅)μ(·) denotes the mean residual stream activity across the selected trials. A high-confidence steering vector was created by subtracting the mean of the low confidence trial activity vectors from the mean of the high confidence trial activity vectors. Steering vectors were scaled to 3% of the residual norm at each layer and multiplied by a constant of either 2 or 5. The low-confidence vector was defined as the inverse of the high-confidence vector. At test time, residual stream activity in the network at a given layer was additively modulated as: ~(l)=(l)+α(l), r^(l)\;=\;r^(l)\;+\;α\,v^(l), (l) ^(l) :residual stream activations at layer l, :\;residual stream activations at layer l, (l) ^(l) :steering vector at layer l(scaled to 3% of residual norm), :\;steering vector at layer l\;(scaled to 3\% of residual norm), α α :steering strength constant,α∈2,5 :\;steering strength constant,\;α∈\2,5\ There were 200 questions per layer per position used for steering. To ensure a balanced selection of trials – given that the model tended to pick higher confidence classes (see Figure 9) – 1/2 of these trials were randomly sampled from the top 3 confidence classes and 1/2 from the bottom 3 classes. C.1.4 Activation Patching Corruption of Answer Tokens via Mean Ablation To test whether specific position-layer combinations are sufficient for confidence computation, we use a corrupt-and-restore procedure following Meng et al. (2022); Heimersheim & Nanda (2024); Zhang & Nanda (2023); Wang et al. (2023). We first disrupted the model’s access to answer information through mean ablation of answer tokens. Let i(0)x_i^(0) denote the embedding of token i in the input sequence, and let =a1,…,akA=\a_1,…,a_k\ denote the set of answer token positions. We computed mean activations from a calibration set C of 100 trials (50 high-confidence, 50 low-confidence, disjoint from the test set): ¯j(0)=1||∑c∈j,c(0) x_j^(0)= 1|C| _c x_j,c^(0) (1) where j,c(0)x_j,c^(0) is the embedding at answer position j for calibration trial c. For each test trial, we replaced all answer token embeddings with their corresponding mean activations, propagating this corruption through the entire forward pass. Patching Procedure Let p(ℓ)h_p^( ) denote the residual stream activation at position p after layer ℓ . For a given test trial, let p(ℓ,clean)h_p^( ,clean) denote the activation from a clean forward pass (no corruption) and p(ℓ,corrupt)h_p^( ,corrupt) denote the activation when answer tokens have been mean-ablated. Our patching intervention selectively restores the clean activation at position p and layer ℓ : p(ℓ,patched)=p(ℓ,clean)h_p^( ,patched)=h_p^( ,clean) (2) while all other positions retain their corrupted activations. This intervention was applied after the MLP block at each layer. We tested three positions: PANL (post-answer newline), C (confidence colon), and PANL+1 (control position). We restricted analysis to high-confidence trials because these provide the clearest test: corruption should substantially reduce confidence, whereas low-confidence trials are already near floor and thus less sensitive to disruption. There were 200 high confidence (top three classes) selected based on the original Phase 0. C.1.5 Metrics used in Patching and Other Experiments Logit Difference. As a generalization of (Wang et al., 2023), we define logit difference as the logit of the original confidence class minus the mean logit of alternative confidence classes: Δlogit=zy∗−1K−1∑k≠y∗zk _logit=z_y^*- 1K-1 _k≠ y^*z_k (3) where zy∗z_y^* is the logit of the clean trial’s confidence class, zkz_k are logits of the K=10K=10 confidence classes, and the sum is over the 9 alternative classes. This metric captures how strongly the model favors its original prediction relative to alternatives. Confidence. We define confidence as the midpoint of the predicted confidence class’s probability range. For example, “Highly likely” (0.8–0.9) corresponds to a confidence of 0.85. First Token Change Rate. We measure the proportion of trials where the argmax token after intervention differs from the clean baseline: Change Rate=1N∑i=1N[argmaxkzk(patched,i)≠argmaxkzk(clean,i)]Change Rate= 1N _i=1^N1 [ *argmax_kz_k^(patched,i)≠ *argmax_kz_k^(clean,i) ] (4) Recovery To quantify how effectively patching restores model behavior, we compute percent recovery for each metric M: RecoveryM=Mpatched−McorruptMclean−Mcorrupt×100%Recovery_M= M_patched-M_corruptM_clean-M_corrupt× 100\% (5) A recovery of 100% indicates complete restoration of clean behavior; 0% indicates no improvement over the corrupted baseline. For first token change rate, where lower values indicate recovery (clean baseline has 0% change), we invert the metric: Recoverytoken=Ratecorrupt−RatepatchedRatecorrupt×100%Recovery_token= Rate_corrupt-Rate_patchedRate_corrupt× 100\% (6) C.1.6 Activation Noising Experiment To test whether representations at PANL and C are necessary for confidence reporting, we performed activation noising experiments using mean ablation. For each position of interest, we replaced its residual stream activation with the mean activation computed from a balanced calibration set of 100 trials (50 high-confidence trials from the top three classes: “Highly likely,” “Very good chance,” “Almost certain”; and 50 low-confidence trials from the bottom three classes: “No chance,” “Really unlikely,” “Chances are slight”). This calibration set was disjoint from the test set. Notably, mean ablation does not push the model toward a semantically meaningful “neutral” confidence state. The mean of activations encoding high and low confidence is not itself an encoding of medium confidence—analogous to how averaging word embeddings for “brilliant” and “terrible” does not yield a representation of “mediocre.” Rather, mean ablation disrupts the position’s contribution to the computation by replacing trial-specific information with an uninformative average. We tested three positions (n=400 trials per layer per position): PANL (post-answer newline), C (confidence colon), and PANL+1 (control position). For each position, we applied mean ablation at individual layers across the network and measured the resulting disruption to the model’s confidence output. We assessed disruption using two complementary metrics of confidence reporting disruption: logit difference change and first token change rate. If a position is necessary for confidence computation, ablating it should reduce logit difference and increase the first token change rate. If ablation has no effect, the position is not necessary for the model’s confidence-reporting mechanism. C.1.7 Activation Swap Experiment We implemented activation swapping in a 2×22× 2 factorial design crossing recipient confidence (high vs. low) with donor confidence (high vs. low), yielding four conditions: High→ , High→ , Low→ , and Low→ . The same-confidence conditions (High→ and Low→ ) control for generic cross-trial substitution effects—any swap introduces a foreign internal state that may disrupt processing independent of confidence through content-related effects. By comparing cross-confidence swaps to same-confidence swaps on the same recipient trials, we isolate effects attributable specifically to the donor’s confidence level. We fixed the set of recipient trials within each regime: the same 400 high-confidence recipients were used in both High→ and High→ conditions, and the same 400 low-confidence recipients in both Low→ and Low→ conditions. Trials were partitioned by the model’s clean confidence report into high-confidence (“Highly likely”, “Very good chance”, “Almost certain”; N>400N>400 available) and low-confidence (“No chance”, “Really unlikely”, “Chances are slight”; N=221N=221 available, sampled with replacement) sets. To control for prompt-length effects on attention patterns and decoding dynamics, we matched donor trials to recipient trials on tokenized question and answer length using quantile bins. Matching was highly successful: question-length bins matched in 100% of cases, and answer-length bins in 94–100% of cases (mean |ΔLQ|≈1.5| L_Q|≈ 1.5–2.72.7 tokens; mean |ΔLA|≈0.3| L_A|≈ 0.3–0.50.5 tokens). We therefore interpret differences between cross-regime and within-regime swaps as reflecting confidence-specific information carried by PANL, rather than generic length-mismatch disruption. C.1.8 Decoding Confidence Information To assess where confidence-relevant information is represented across the network, we trained linear probes on residual stream activations at each layer and token position. We extracted activations from 3,000 TriviaQA trials at positions of interest including PANL, PANL+1, and C. Additionally, we probed other tokens in the prompt (e.g., intermediate tokens that corresponded to the first token of each confidence class as listed in the confidence reporting instruction part of the prompt) to determine how widespread the encoding of confidence information is across the model. For each position-layer combination, we trained two types of probes using 5-fold cross-validation. For predicting binary correctness, we fit L2-regularized logistic regression and report AUROC. For predicting verbal confidence (treated as a continuous variable using class midpoints), we fit Ridge regression and report R2R^2. All activations were z-scored prior to fitting. To assess the relationship between token-level probabilities and verbal confidence reports, we computed correlations between length-normalized answer logprobs (collected during Phase 0) and verbal confidence ratings from both Phase 0 (same run) and Phase 1 (different run with identical questions but answers provided in the prompt). Logprobs explained only 4.9% of variance in within-run verbal confidence (r=0.23r=0.23, RCV2=0.049R^2_CV=0.049) and 8.4% in cross-run verbal confidence (r=0.29r=0.29, RCV2=0.084R^2_CV=0.084). These low values confirm that verbal confidence is not simply a readout of token probabilities during answer generation. The model’s verbal confidence showed moderate consistency across runs, with Phase 0 and Phase 1 ratings correlating at r=0.63r=0.63 (RCV2=0.40R^2_CV=0.40), indicating that confidence judgments are relatively stable across repeated presentations of the same question-answer pair. The cross-validated Ridge regression R2R^2 of 0.084 for logprob predicting Phase 1 confidence serves as the baseline for our variance partitioning analysis; activation-based probes explaining variance beyond this threshold capture information about verbal confidence that is not reducible to generation-time token probabilities. C.1.9 Token Log-Probability Extraction To obtain a white-box measure of model confidence, we extracted token-level log-probabilities during generation. For each generated sequence, we computed the log-probability of each token tit_i given the preceding context: log⁡p(ti∣t<i,) p(t_i t_<i,x) (7) where x denotes the input prompt. We then computed the mean log-probability over the n answer tokens: ℓ¯=1n∑i=1nlog⁡p(ti∣t<i,) = 1n _i=1^n p(t_i t_<i,x) (8) This length-normalized metric controls for variation in answer length, ensuring that longer answers are not penalized simply for having more tokens. Answer token boundaries were identified by mapping the extracted answer string back to the generated token sequence. C.1.10 Attention Blocking Method To trace information flow during confidence generation, we employ attention knockout (Geva et al., 2023), which blocks attention edges between specific token positions. For a source position s and target position t, we set the attention weights αt→s _t→ s to zero across all attention heads within a specified layer range, preventing information from flowing from s to t. We measure the effect on confidence output using two metrics: (1) first token change rate—the proportion of trials where the argmax confidence token differs from the unmodified baseline; and (2) logit difference change—the shift in the margin between the baseline token’s logit and the mean logit of alternatives. We use a minimal prompt format that elicits numeric confidence (0–9) and reduces the number of intermediate tokens between PANL and C (see Figure 13). This minimal prompt was used to test three key pathways: (1) C→ +A, to test whether C computes confidence via just-in-time attention to question and answer tokens; (2) C→ , to test whether C retrieves cached confidence information from PANL; and (3) PANL→ and PANL→ _A, to trace how PANL gathers confidence information from answer tokens. The first and third findings were confirmed in complementary experiments using the full categorical prompt (see §D for details), where blocking all downstream tokens from attending to answer positions replicated the disruption observed with the minimal prompt. Following Geva et al. (2023), we note that this method does not account for information that may have passed between positions at earlier layers prior to the blocking intervention. Thus, null results should be interpreted cautiously, while positive results provide evidence that the blocked pathway carries task-relevant information at the targeted layers. C.2 Experiment using Numeric Confidence Prompt in Gemma 3 27B We used a prompt derived from (Devic et al., 2025; Mei et al., 2025) (see Figure 12). For numeric confidence experiments (Gemma 3 27B), we used max-new-tokens=4 to generate the full confidence value (e.g., ”95”). The positions of the PANL, PANL+1, FCC (first confidence colon) and C tokens were analogous to their positions in the categorical confidence prompt. Metrics for numeric confidence Analogous to the categorical confidence prompt experiment, we track three metrics in the numeric confidence prompt experiments. Confidence change is the difference between the integer value (0–100) generated in the baseline condition and the (e.g.) noised condition; First digit change rate measures the proportion of trials where the first generated digit differs from the clean baseline—analogous to the first token change rate used for categorical confidence; Logit difference is computed as the logit of the clean trial’s first digit minus the mean logit of the other nine digits (0–9), capturing how strongly the model favors its original confidence judgment over alternatives. Multi-token numeric confidence and KV-cache considerations Numeric confidence involves multi-token generation (e.g., “9→ 5”). All interventions occur during prefill, modifying cached key–value representations that subsequent digits attend to. The second digit is generated by attending to both the cached prompt and the first digit, meaning effects on later digits reflect both the original intervention and changes propagated through earlier tokens. C.3 Experiment using Qwen 2.5 7B Qwen 2.5 7B Instruct was tested on the categorical prompt (see Figure 8). The model was loaded via Hugging Face. Qwen showed reasonable calibration (ECE = 0.06, AUROC = 0.65)(see Figure 15).The model has 28 layers; we used a dense layer sweep across layers 0–27. We used the same procedures as described for Gemma. The model was run in evaluation mode with greedy decoding (temperature = 0). We used the same TriviaQA subset as for Gemma, with 3000 trials for activation collection. Due to the distribution of confidence ratings (see Figure 15) low confidence trials were sampled from the ”Unlikely” class, and high confidence trials from the ”Likely” class. 150 questions per condition per layer were used for steering; 200 questions for activation patching; 300 questions for noising (200 high confidence, 100 low confidence trials); 200 questions per condition for the activation swap experiment. Appendix D Supplemental Results D.1 Attention Blocking with Categorical Confidence Prompt To complement the attention blocking experiments using the minimal numeric prompt (main text), we conducted parallel experiments using the full categorical confidence prompt (Figure 8). These experiments provide convergent evidence against the just-in-time hypothesis; however, they do not reveal the direct pathway from PANL to C, likely because confidence information can flow through multiple intermediate template tokens, masking the effect of single-edge (C→ ) blocking. C does not compute confidence from scratch. Blocking C’s attention to question tokens, answer tokens, or both produced minimal effects (∼ 10% change rate), equivalent to a control condition blocking C→ +1 (Figure 18). This rules out the just-in-time hypothesis. No direct C→ pathway detected. Unlike the minimal prompt results reported in the main text, blocking C→ produced no effect with the categorical prompt. This null result likely reflects redundant information routing: confidence information cached at PANL can reach C through multiple intermediate template tokens, so blocking the direct pathway alone is insufficient to disrupt confidence output. PANL caches confidence information. Blocking all later tokens from attending to PANL (ALL→ ) produced a 20% change rate and substantial logit difference reduction, confirming that PANL serves as a confidence cache even when direct C→ blocking is ineffective. Answer tokens are the primary information source. Blocking all attention to the last answer token (ALL→ _A) produced a 50% change rate (Figure18). Critically, preserving only the PANL→ _A pathway reduced this effect to 20%, demonstrating that PANL reads confidence-relevant information from answer tokens and relays it downstream. The same pattern held for all answer tokens: ALL→ produced 70% change, reduced to 40% when preserving PANL→ . These results converge with findings from the minimal prompt to support the cached retrieval model: confidence information originates at answer tokens, is read by PANL, and flows to C for verbalization.

Instant research discovery