Paper deep dive
When Linguistic and Internal Confidence Diverge in Large Language Models
Hefan Zhang, Bingquan Zhang, Ming Cheng, Saeed Hassanpour, Weicheng Ma, Soroush Vosoughi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's internal confidence. We study this question across 8 classification tasks, 2 generation tasks and 30 models from three families. For classification, we compare linguistic confidence with logits-based confidence along three axes: association, magnitude agreement and calibration. For generation, we test whether linguistic confidence tracks semantic-entropy-based uncertainty. The axes frequently diverge. Instance-level association is weak on average, although it improves on easier items and for stronger base models. Instruction-tuned models often report higher confidence and sometimes show higher association, but they also have larger confidence gaps and worse calibration. Prompt design mostly changes the distribution of reported confidence. Attitude cues inflate confidence without improving alignment, while score exemplars can preserve rank-order signal when they avoid collapsed confidence values. Regression analyses show that distributional properties of confidence scores explain much of the observed alignment pattern, with model metadata playing a smaller role after controls. These results support a lossy-channel view of linguistic confidence. A more dispersed verbal confidence distribution can carry useful rank information, but it does not make the scores calibrated. Linguistic confidence should therefore be evaluated with multi-axis diagnostics before being used in downstream reliability pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2608.28382v1
- Canonical: https://arxiv.org/abs/2608.28382v1
Trouble viewing inline? Open PDF directly →
Full Text
93,944 characters extracted from source content.
Expand or collapse full text
When Linguistic and Internal Confidence Diverge in Large Language Models Hefan Zhang 1 , Bingquan Zhang 1 , Ming Cheng 1 , Saeed Hassanpour 1 , Weicheng Ma 2 , Soroush Vosoughi 1 1 Dartmouth College, 2 Oakland University hefan.zhang.gr, soroush.vosoughi @dartmouth.edu Abstract Users often ask large language models (LLMs) to report how confident they are, but it is un- clear whether such linguistic confidence tracks the model’s internal confidence. We study this question across 8 classification tasks, 2 gener- ation tasks and 30 models from three families. For classification, we compare linguistic confi- dence with logits-based confidence along three axes: association, magnitude agreement and calibration. For generation, we test whether linguistic confidence tracks semantic-entropy- based uncertainty.The axes frequently di- verge. Instance-level association is weak on average, although it improves on easier items and for stronger base models.Instruction- tuned models often report higher confidence and sometimes show higher association, but they also have larger confidence gaps and worse calibration.Prompt design mostly changes the distribution of reported confi- dence. Attitude cues inflate confidence with- out improving alignment, while score exem- plars can preserve rank-order signal when they avoid collapsed confidence values. Regres- sion analyses show that distributional prop- erties of confidence scores explain much of the observed alignment pattern, with model metadata playing a smaller role after controls. These results support a lossy-channel view of linguistic confidence. A more dispersed verbal confidence distribution can carry useful rank information, but it does not make the scores calibrated. Linguistic confidence should there- fore be evaluated with multi-axis diagnostics before being used in downstream reliability pipelines. 1 Introduction Large language models (LLMs) are increasingly used not only for their answers, but also for confidence signals that help decide when an an- swer should be trusted, filtered, abstained from, or combined with other samples (Geng et al., 2023; Liu et al., 2025). Internal signals such as token-level probabilities can support confidence- weighted voting, selective prediction and test-time selection (Fu et al., 2025). In many deployed sys- tems, however, these internal signals are not avail- able. Users and practitioners are left with lin- guistic confidence, the model’s self-reported con- fidence in language or as a numeric score. Prior work has shown that verbalized confi- dence is fragile. It can be sensitive to prompt- ing and often becomes overconfident after instruc- tion tuning (Mielke et al., 2022; Xiong et al., 2023; Groot and Valdenegro-Toro, 2024; Xu et al., 2025). Much of this work evaluates confidence as an estimator of correctness. This is important, but it leaves a separate question open. When a model says it is confident, is that statement aligned with the model’s own internal probability signal, or is it a separate user-facing behavior? We study this cross-channel question. Our goal is not to treat logits-based confidence as ground- truth uncertainty. Softmax probabilities are them- selves imperfect and often miscalibrated. Instead, we use them as an internally accessible confidence channel and ask when linguistic confidence agrees with that channel. For generation tasks, where answer-token probabilities are not always well de- fined, we use semantic entropy as the internal un- certainty proxy (Kuhn et al., 2023). We evaluate linguistic confidence along three axes.Association measures whether linguistic and internal confidence move together across in- stances. Magnitude agreement measures whether the two scores are close on a common scale. Calibration measures whether reported confidence agrees with empirical correctness. These axes are not interchangeable. A model can have high as- sociation while still being overconfident, or it can have small average distance while carrying little instance-level information. Across 10 tasks and 30 models, we find that arXiv:2608.28382v1 [cs.CL] 28 Aug 2026 instance-level association is weak on average. At the same time, the weakness is structured rather than random. Easier tasks and stronger base mod- els tend to show higher association. Instruction- tuned models often report higher confidence, but their apparent gains in association do not reliably produce better calibration. Prompt changes also have limited leverage. They can shift the mean and variance of linguistic confidence, but attitude cues that raise verbal confidence do not make it more grounded in internal probabilities. The main empirical pattern is distributional. When linguistic confidence collapses into a nar- row range, often near high values, it cannot ex- press instance-level differences. When it is more dispersed, it can preserve rank-order information and improve association. This does not mean that more dispersion fixes calibration. The same dis- persion that helps correlation can also increase magnitude mismatch if the scores are shifted or nonlinear. We therefore interpret linguistic confi- dence as a lossy channel from internal confidence to user-facing text. The channel can carry useful rank signal in some regimes, but it should not be read as a calibrated probability without validation. This paper makes three contributions.We provide a systematic cross-channel evaluation of linguistic and internal confidence across model families, tasks and prompting regimes.We show that association, magnitude agreement and calibration diverge in important cases, espe- cially for instruction-tuned models.We iden- tify distributional properties of confidence scores as dominant observable drivers of alignment and translate this finding into practical guid- ance for reliability pipelines.Code is avail- able at https://github.com/HF-heaven/Correlation- between-Confidence-Measurements. 2 Related Work Uncertainty estimation in LLMs. Confidence and uncertainty estimation are central to reliable LLM deployment (Huang et al., 2024; Liu et al., 2025).Existing methods can be grouped into logits-based, linguistic and consistency-based ap- proaches (Geng et al., 2023). Logits-based meth- ods use white-box access to output distributions, including softmax probabilities, entropy or logit margins (Duan et al., 2023; Kuhn et al., 2023; Huang et al., 2023). Linguistic methods ask the model to express confidence directly in words or scores (Mielke et al., 2022; Xiong et al., 2023). Consistency-based methods estimate uncertainty from output stability under sampling or perturba- tion (Manakul et al., 2023; Lin et al., 2023). These signals differ in access requirements and in what part of the model behavior they measure. Verbalized and internal confidence. Prior work studies both the calibration of verbalized confidence and its relationship to internal uncer- tainty. Xiong et al. (2023); Groot and Valdenegro- Toro (2024) show that linguistic confidence can be overconfident and prompt-sensitive.Some other studies connect language-model confidence to human confidence or propose calibration interventions for verbalized scores (Mielke et al., 2022; Xu et al., 2025).These studies usually judge confidence by its relationship to correct- ness, often through ECE, AUROC, or related calibration metrics.Lin et al. (2022) examine whether models can express calibrated uncer- tainty in words, while Tian et al. (2023) compare verbalized scores with conditional probabilities in models fine-tuned with human feedback. Kumar et al. (2024) directly study confidence-probability alignment using elicited certainty scores and token probabilities. Yona et al. (2024) evaluate whether linguistic hedging faithfully reflects intrinsic uncertainty. Most closely related to our generation setting, Ji et al. (2025) analyze mis- match between verbal and semantic uncertainty and use it to predict hallucinations. More broadly, the sensitivity of LLM behavior to linguistic framing (Tian et al., 2026) raises questions about whether expressed confidence remains aligned with internal confidence across settings. Our work extends this literature through a sys- tematic multi-axis analysis across 30 models, 10 tasks, classification and generation settings, and multiple prompting regimes. Rather than treating cross-channel correlation as calibration, we report two separate measurements: association describes agreement between linguistic confidence and the selected internal proxy, whereas calibration evalu- ates confidence against correctness. Neither mea- surement enables the other. We additionally exam- ine how task difficulty, model properties, prompt- ing, and confidence-score distributions are associ- ated with these outcomes. Failure modes and downstream use. Cross- channel differences are described as distributional shift, magnitude disagreement, or cross-channel misalignment rather than miscalibration. We re- serve calibration for comparisons against correct- ness. A linguistic score may collapse into a narrow range, differ systematically in magnitude from an internal proxy, or fail to preserve instance order- ing; these cases motivate different deployment checks.For white-box models, internal confi- dence and related uncertainty signals support se- lection, confidence-weighted voting, abstention, reranking, selective prediction, and curriculum- based post-training data ordering (Brown et al., 2024; Cobbe et al., 2021; Kamath et al., 2020; Cole et al., 2023; Fu et al., 2025; Jia et al., 2026). The reliability of verifier-based selection is like- wise sensitive to problem difficulty and model ca- pability (Zhou et al., 2026). For black-box sys- tems, linguistic confidence should be validated on held-out examples before it controls downstream decisions. 3 Confidence Measurements and Evaluation We compare linguistic confidence with internal confidence proxies. For classification tasks, the internal proxy is logits-based confidence over the candidate answers. For generation tasks, the in- ternal proxy is semantic entropy over sampled re- sponses. 3.1 Linguistic Confidence We obtain linguistic confidence by asking the model to answer the task and report a confidence score for its own answer (Groot and Valdenegro- Toro, 2024; Xiong et al., 2023). The prompt de- fines 0 as no belief in the answer and 10 as full confidence. We extract the generated score and di- vide it by 10 when comparing it with probabilities. Prompt templates are described in Appendix A. 3.2 Internal Confidence Proxies For classification tasks, output formats constrain the answer to a fixed candidate set. We extract logits for the option tokens and compute a soft- max over the candidate set. We use the resulting probability of the selected answer as logits-based confidence. This quantity should not be interpreted as ground-truth uncertainty. Softmax probabilities can be miscalibrated, and they do not capture ev- ery form of epistemic uncertainty. In this work, they serve as an internal confidence proxy that can be measured in white-box models. The study is therefore about fidelity between two available channels, internal probability signals and linguis- tic self-reports. For generation tasks, the key answer is not al- ways tied to one token. We therefore use semantic entropy (Kuhn et al., 2023). Given a prompt x, we sample M responses and cluster semantically equivalent outputs using an NLI model. The prob- ability of a semantic cluster is estimated by sum- ming the sequence probabilities of its members, p(c| x) = X s∈c Y i p(s i | s <i ,x).(1) Semantic entropy is then computed over the clus- ters, SE(x)≈−|C| −1 |C| X i=1 logp(C i | x),(2) where C is the set of semantic equivalence classes. Higher entropy means lower confidence, so we use negated semantic entropy when computing associ- ation with linguistic confidence. 3.3 Evaluation Axes We evaluate confidence alignment along three axes. Association is measured with Pearson cor- relation and Spearman correlation.It captures whether two confidence signals vary in the same direction across instances. Magnitude agreement is measured by Euclidean distance after mapping linguistic confidence to [0, 1]. It captures whether the two scores are close in value. Calibration is measured with expected calibration error (ECE) (Guo et al., 2017). It captures whether reported confidence matches empirical accuracy. The three-axis analysis applies directly to clas- sification tasks, where logits-based confidence and empirical accuracy are defined on the same pre- diction. For generation tasks, semantic entropy is not on the same probability scale as linguistic confidence and correctness is less uniform across free-form responses. We therefore use associa- tion with negated semantic entropy as the primary cross-channel metric for generation, and use dis- tance and ECE for the classification setting. These axes answer different questions. Cor- relation can be high even when linguistic confi- dence is systematically too large. Distance can be small when both scores are concentrated, even if neither carries much instance-level information. ECE measures agreement with correctness rather than agreement between confidence channels. We report these metrics jointly to avoid treating one form of agreement as full reliability. 4 Experimental Setting We evaluate 8 classification tasks and 2 genera- tion tasks. The classification benchmarks cover syntactic judgment (CoLA), natural language in- ference (QNLI), paraphrasing (QQP) (Wang et al., 2018), causal reasoning (Cause And Effect), lex- ical reasoning (Conceptual Combinations and Ruin Names), temporal reasoning (Temporal Se- quences) (Suzgun et al., 2022), and broad knowl- edge (MMLU) (Hendrycks et al., 2021).The generation tasks are CoQA (Reddy et al., 2019) and TriviaQA (Joshi et al., 2017). For computa- tional tractability, only the five evaluation sets con- taining more than 200 examples (CoLA, QNLI, QQP, MMLU, and Temporal Sequences) are ran- domly downsampled to 200 instances. All other tasks use their complete evaluation sets. Repeated- subsampling results and reproducibility details are reported in Appendix B. We use 30 publicly available models from the LLaMA, Mistral and Qwen families (Touvron et al., 2023a,b; Grattafiori et al., 2024; Jiang et al., 2023, 2024; Qwen et al., 2025; Yang et al., 2025). The models cover different sizes, versions and tun- ing types. Table 6 lists all models. We use standardized prompt templates for clas- sification and generation. For base models, we use two-shot examples to constrain the answer and confidence format.For instruction-tuned mod- els, we place the output format requirement in the system prompt. We also test prompt robustness through confidence-exemplar search, confidence- elicitation ablation and attitude-based perturba- tions. The details are in Appendix A and Ap- pendix G. 5 Results We organize the results from coarse to fine. We begin with a pooled descriptive overview, then ex- amine instance-level association and decompose the results by task, model, generation setting, and prompt design. We subsequently evaluate these factors jointly using descriptive regression mod- els.This organization distinguishes aggregate patterns from the stratified evidence relevant to instance-level decisions. 0.30.40.50.60.70.80.91.0 Overall Mean Logits-based Confidence 4 5 6 7 8 9 10 Overall Mean Linguistic Confidence base (n=256) instruct (n=112) Figure 1: Mean logits-based confidence and mean lin- guistic confidence across task, model and prompt set- tings. Each point is one setting. Instruction-tuned mod- els tend to have higher values on both axes. The overall association is statistically significant, but the split by model type shows that base models have weak positive association while instruction-tuned models have almost no aggregate association. 5.1 Overall Alignment: Confidence Measures Align Weakly We first average logits-based confidence and lin- guistic confidence for each task, model and prompt setting. Figure 1 shows that instruction- tuned models generally produce higher values on both axes than base models. The overall Pearson correlation is r = 0.4830 with p = 1.14× 10 −22 . The corresponding Spearman correlation is ρ = 0.4711 with p = 1.65× 10 −21 . This aggregate trend hides an important split. Base models show a weak but significant positive association, with r b = 0.2607 and p = 2.40 × 10 −5 , and ρ b = 0.2161 with p = 4.98 × 10 −4 . Instruction-tuned models show negligible associ- ation, with r i = −0.0048 and p = 0.961, and ρ i = 0.0422 with p = 0.665. The split suggests that instruction tuning changes how confidence is expressed. Instruction- tuned models often have logits-based probabilities concentrated near 1 and linguistic confidence con- centrated near high values. This compression re- duces observable association because many dis- tinct examples receive similar confidence values. The result is not that base models are reliable. Rather, base models retain more variation in both channels, which allows weak association to re- main visible. At the instance level, association is weaker than the aggregate plot suggests. Across settings, the 0.0 0.2 0.4 0.6 0.8 Normalized accuracy Linear trend Task mean +/- std conceptual combinations MMLUtemporal sequences CoLAcause and effect QNLIQQPruin names Task −0.2 0.0 0.2 0.4 0.6 Pearson correlation Figure 2: Task difficulty and confidence association. Tasks are ordered by decreasing normalized accuracy in the top panel. The bottom panel shows the Pearson correlation between linguistic confidence and logits- based confidence. Markers show per-task means and error bars show ±1 standard deviation across mod- els. Association tends to weaken as task difficulty in- creases. average Pearson correlation is r = 0.135. This gap between aggregate and instance-level behav- ior is central to our findings. Linguistic and in- ternal confidence can move together in broad av- erages while diverging on individual examples, which is the level needed for abstention, filtering and selection. 5.2 Task-Level Variation: Association Tends to Be Higher on Easier Tasks We next examine whether task difficulty affects the relationship between linguistic and logits- based confidence. Since tasks have different num- bers of answer choices, we use chance-corrected normalized accuracy: NormAcc = Acc− 1/K 1− 1/K ,(3) where K is the number of answer options. This maps random guessing to 0 and perfect accuracy to 1, making binary and four-choice tasks more comparable. Across all eight tasks, the pooled task-level relationship between normalized accuracy and mean confidence association is positive but not statistically reliable (slope = 0.360, 95% CI [−0.842, 1.562], p = 0.491). Excluding Ruin Names, the estimated slope is larger (slope = 1.067, 95% CI [0.075, 2.058], p = 0.0396). We report this exclusion as a sensitivity analysis rather than as the primary result. 3478121314 Model Size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Pearson Coefficient Correlation Qwen2.5_base Qwen3_base Llama2_base Mistral-v0.1_base (a) Base models 3478121314 Model Size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Pearson Coefficient Correlation Qwen2.5_instruct Llama2_instruct Mistral-v0.1_instruct (b) Instruction-tuned models Figure 3: Model size and association between linguis- tic and logits-based confidence. Association tends to rise with model size for base models, but it plateaus or decreases for instruction-tuned models. We also estimate the relationship separately within each model–prompt setting. Thirty of 45 slopes are positive (66.7%; sign-test p = 0.0179), but the mean slope is small and uncertain (mean = 0.034, 95% CI [−0.155, 0.223]). After averag- ing prompts within models, 20 of 29 model-level slopes are positive (69.0%; sign-test p = 0.0307; mean = 0.121, 95% CI [−0.104, 0.347]). The di- rection therefore appears in a majority of strati- fied analyses, but its magnitude is heterogeneous across tasks and models. Ruin Names is an outlier. It has relatively low accuracy but high association between linguistic and logits-based confidence. We avoid interpret- ing this as good calibration. The task is built around humorous edits of names, so confidence may track surface familiarity or orthographic sim- ilarity even when the model chooses the wrong op- tion. This shows why association must be read to- gether with accuracy and calibration. 5.3 Model-Level Variation: Model Properties Affect Alignment We compare confidence association across model configurations. Figure 3 shows different scaling trends for base and instruction-tuned models. For base models, larger models tend to have higher as- sociation. For instruction-tuned models, associa- tion is flat or decreases. A distributional explanation fits this pattern. Smaller base models often output linguistic confi- dence concentrated around 8 or 10, which leaves little instance-level signal.Larger base mod- els produce more varied linguistic confidence, which can preserve more rank-order information. In contrast, larger instruction-tuned models often have logits-based confidence values near 1. This 0.000.250.500.751.00 Logits-based Confidence 0 25 50 75 100 125 150 Count 0.02.55.07.510.0 Linguistic Confidence 0 50 100 150 200 Count (a) Mistral-7B (base) 0.000.250.500.751.00 Logits-based Confidence 0 20 40 60 80 100 120 Count 0.02.55.07.510.0 Linguistic Confidence 0 25 50 75 100 125 150 Count (b) Mixtral-8x7B (base) 0.000.250.500.751.00 Logits-based Confidence 0 10 20 30 40 50 60 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 Count (c) Mistral-7B (instruct) 0.000.250.500.751.00 Logits-based Confidence 0 20 40 60 80 100 120 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 100 120 Count (d) Mixtral-8x7B (instruct) Figure 4: Representative distributions of logits-based and linguistic confidence on CoLA for Mistral-v0.1 models. The top row shows base models under Base Prompt 1, and the bottom row shows instruction-tuned models. Metricmean instruct mean base tp r0.1720.118 1.95 5.4e-02 Distance0.2710.241 2.03 4.4e-02 ECE(linguistic)0.2870.191 6.02 2.5e-08 Table 1: Paired t-test results comparing instruction- tuned and base models. Underlined values indicate sig- nificant differences with p < 0.05. saturation reduces internal variation and weak- ens correlation with linguistic confidence. Fig- ure 4 provides a representative comparison be- tween base and instruction-tuned Mistral-family models, while additional model-family examples are provided in Appendix E. Table 1 shows the same point from another angle.Instruction-tuned models have slightly higher Pearson correlation than base models, but the difference is not significant at the 0.05 level. Their distance and linguistic ECE are significantly larger. Thus, instruction tuning can make the con- fidence channels appear more associated in some settings while making the reported scores less cal- ibrated. Across families, we observe a general in- crease in association from LLaMA to Mistral and Qwen, together with lower distance and ECE. Within-family version updates are less consistent. LLaMA-3.1 improves association over LLaMA- 3, while Qwen3 shows lower association and larger distance than Qwen2.5. The appendix anal- ysis (Appendix F) suggests that these changes are closely tied to shifts in confidence dispersion rather than family or version alone. 5.4 Generation Setting: Association Remains Task- and Model-Dependent For generation tasks, we compare linguistic con- fidence with negated semantic entropy. We sam- ple five responses per prompt, cluster semantically equivalent responses with DeBERTa-xlarge-MNLI (He et al., 2021), compute semantic entropy and average the linguistic confidence across the sam- pled responses. As noted in Section 3.3, these re- sults are evaluated through association because se- mantic entropy is an uncertainty score rather than a calibrated probability on the linguistic-confidence scale. The full results are in Appendix H.2.The main pattern is that instruction-tuned models usu- ally show stronger association than their base counterparts on generation tasks. TriviaQA also shows higher association than CoQA. This likely reflects the difference between factoid questions with clearer answers and conversational ques- tions with more context-dependent answer forms. These generation results support the same general conclusion as the classification results. Linguis- tic confidence can carry some rank-order informa- tion, but this does not establish calibrated uncer- tainty. 5.5 Prompt-Level Variation: Prompt Design Mainly Changes Score Distributions We test three prompt-related factors. First, we search over few-shot confidence exemplars. Sec- ond, we remove confidence elicitation to test whether asking for confidence perturbs logits. Third, we prepend attitude cues such as criticism, approval and irrelevant text. The exemplar search shows that confidence ex- amples affect both task performance and associa- tion. Adjacent but non-identical scores work bet- ter than identical scores. Identical exemplars tend to collapse the produced confidence distribution, which weakens instance-level association. Very large exemplar scores push outputs toward high confidence values and also reduce useful variation. This supports the distributional view that linguistic confidence needs enough spread to preserve rank information. The ablation study shows that confidence elic- itation does not meaningfully perturb the inter- nal probability signals. Removing the confidence query does not change accuracy, and it has limited effects on logits statistics. For instruction-tuned OutcomenPredictors R 2 Adj. R 2 DIST363280.5700.536 ECE363280.6440.615 CORRELATION363280.4980.459 Table 2: OLS fit statistics for distance, confidence-to- confidence ECE and correlation between linguistic and logits-based confidence. models, no tested metric changes significantly. The attitude perturbation results are more con- cerning. Criticism, approval and irrelevant cues all raise reported linguistic confidence, but they reduce association with logits-based confidence. A calibration instruction asking the model to re- flect final-layer probabilities does not significantly improve the relationship. Prompt tone therefore changes how confident the model sounds without making the confidence more grounded in internal probabilities. Full results are in Appendix G. 6 Distributional Properties Explain Much of the Pattern 6.1 Regression Modeling We fit regression models to identify which vari- ables predict confidence alignment. We model three outcomes.The first is the correlation between linguistic and logits-based confidence, where higher is better. The second is the dis- tance between the two scores, where lower is bet- ter. The third is a confidence-to-confidence ECE that replaces accuracy with logits-derived con- fidence, where lower is better.For each out- come, we fit ordinary least squares and ridge re- gression with standardized features and five-fold cross-validation. OLS coefficients are descriptive because one-hot encodings and correlated features introduce collinearity. Ridge coefficients are used for stable effect ranking. Table 2 shows that the predictors jointly account for a substantial fraction of the variation in all three outcomes. The OLS models attain R 2 values of 0.570 for distance, 0.644 for ECE-style cross- channel mismatch, and 0.498 for correlation. The main result is that distributional statis- tics dominate.Larger linguistic dispersion is associated with higher correlation (standardized β = 0.194, 95% CI for the unstandardized co- efficient [0.012, 0.070], p = 0.0058), indicat- ing that greater score variability can help pre- serve rank-order signal. The same dispersion is also associated with greater distance (standardized β = 0.780, p < 0.001) and higher confidence-to- confidence ECE (standardized β = 0.318, p < 0.001). Dispersion is therefore not a universal fix: it can improve association while worsening mag- nitude agreement and cross-channel ECE. Lin- guistic mean has a similar role. Higher mean con- fidence is associated with greater distance (stan- dardized β = 0.228, p < 0.001) and higher confidence-to-confidence ECE (standardized β = 0.395, p < 0.001), consistent with greater mis- match when reported scores are high relative to the internal proxy. Logits statistics also matter.Higher logits mean is associated with smaller distance and lower cross-channel ECE, while logits dispersion also contributes to cross-channel ECE. These effects indicate that magnitude agreement is greater when the internal proxy and reported confidence occupy more compatible ranges; they do not require treat- ing logits-based confidence as a ground-truth tar- get. Model metadata are secondary after these con- trols. Model type, family, version, and size gener- ally have smaller and less consistent effects across outcomes than the confidence-distribution statis- tics. Model size is a partial exception for corre- lation, where its ridge coefficient is comparable to the linguistic-distribution coefficients, but this effect does not recur consistently for distance or cross-channel ECE. Overall, the recurring predic- tors across outcomes are properties of the confi- dence distributions, whereas metadata effects are more dependent on the particular outcome and model comparison. Full coefficients and uncer- tainty estimates are reported in Appendix I. 6.2 A Lossy-Channel Interpretation These results suggest a simple interpretation. Lin- guistic confidence is a lossy channel from inter- nal confidence to a user-facing score. If the out- put channel maps many internal states to the same verbal score, the score cannot preserve instance- level distinctions.This occurs when linguistic confidence collapses near a few high values. If the channel uses a broader range of scores, it can preserve more rank-order information, which in- creases association. This interpretation also explains why disper- sion is not enough. A broader score range can still be shifted upward, be nonlinear, or be poorly matched to correctness. In that case, correlation MethodAccuracy Entropy Majority voting0.7311N/A Level confidence0.75310.4362 Score confidence0.78301.6896 Table 3: Confidence-weighted aggregation on eight classification tasks using gpt-4.1-nano.Finer- grained confidence gives a more dispersed weighting signal and improves accuracy in this setting. may improve but calibration can remain poor. This is why we do not claim that increasing dispersion improves alignment in general. The more precise claim is that dispersion can improve association and make confidence useful for ranking or weight- ing, but calibration still requires separate valida- tion. 7 Linguistic Confidence as a Weighting Signal To connect the analysis to a downstream use case, we conduct a small aggregation study with gpt-4.1-nano (OpenAI, 2025) on the eight clas- sification tasks. For each prompt, we sample 16 outputs and aggregate predictions by voting. We compare three settings. In answer-only voting, no confidence is elicited and every prediction has weight 1. In level-confidence voting, the model reports Very confident, Moderately confident, or Uncertain, which are mapped to weights 3, 2 and 1. In score-confidence voting, the model reports a numeric score from 0 to 10. For each candidate answer, we sum the weights across samples and select the answer with the highest total weight. Table 3 shows that confidence-weighted aggre- gation improves over majority voting in this set- ting, and numeric confidence performs best. We interpret this as an illustrative downstream use of rank-order signal, not as evidence that linguistic confidence is calibrated. The result is consistent with our main analysis. A more diverse confidence distribution can help weight samples, but it should be validated on the target setting before deploy- ment. 8 Operational Guidance The empirical results imply a simple use policy as shown in Table 4. Linguistic confidence should first be treated as a signal to be validated, not as a probability. Before using it for abstention, filtering, or sample weighting, one should check whether the reported scores have enough spread, whether they correlate with correctness or an in- ternal proxy on held-out data, and whether high verbal confidence occurs in cases where other evi- dence suggests uncertainty. For white-box systems, the most useful signal is often the mismatch between channels. High lin- guistic confidence paired with low internal confi- dence identifies cases where the model sounds cer- tain despite weak probability support. For black- box systems, internal confidence is unavailable, so the safer procedure is to validate linguistic confi- dence on held-out examples from the same task. If reported scores are collapsed or prompt-sensitive, they should not control abstention thresholds. If they remain dispersed and validated, they can sup- port weak ranking decisions such as confidence- weighted voting. 9 Discussion The main lesson is that linguistic confidence is not a transparent readout of internal uncertainty. It is a separate confidence channel with its own distribu- tional behavior. In some regimes, it carries rank- order information. In other regimes, especially when scores collapse near high values, it becomes uninformative at the instance level. This explains why aggregate plots can look encouraging while instance-level correlations remain weak. The distinction between association and cali- bration is central. A model can vary linguistic confidence in the same direction as logits-based confidence and still be systematically overconfi- dent. This pattern appears often for instruction- tuned models. Conversely, a model can have small distance because both signals are concentrated, while still providing little useful ranking informa- tion. Reliability claims should therefore report association, magnitude agreement and calibration together. The results also clarify what prompt design can and cannot do. Prompting can alter the distribu- tion of reported confidence, and careful exemplar choices can avoid complete score collapse. This may help when confidence is used as a weak rank- ing signal. But prompt tone can inflate confidence without improving alignment. A prompt that asks the model to reflect its internal probabilities does not by itself make verbal confidence track those probabilities. Our analysis is diagnostic rather than causal. We do not claim to identify the training changes RegimeDiagnosticRecommended use Collapsed verbal scoresScores cluster near 9–10, or have low standard deviation Avoid using verbal confidence for fine-grained decisions High verbal confidence with weak internal confidence Linguistic confidence is much higher than logits-based confidence Flag, abstain, rerank, or sample additional answers Dispersed verbal scores with validated association Held-out data show positive association with correctness or internal confidence Use as a ranking or weighting signal, not as a calibrated probability Prompt-induced confidence shifts Tone or framing raises confidence without improving association Treat as confidence inflation, not improved reliability Table 4: Practical interpretation of linguistic confidence under different diagnostic regimes. that create a given confidence distribution, and we do not claim that manipulating variance alone will fix confidence. The evidence shows that many alignment patterns attributed to family, size, or in- struction tuning are better understood through the mean, variance and saturation of the two confi- dence channels. This is the main reason to eval- uate linguistic confidence as a channel with mea- surable failure modes, rather than as a direct prob- ability estimate. 10 Conclusion We study whether linguistic confidence aligns with internal confidence signals in LLMs. Across 10 tasks and 30 models, instance-level align- ment is weak. Association tends to be higher on easier tasks and for stronger base models, but instruction-tuned models often remain overconfi- dent. Prompting can change reported confidence distributions, but attitude cues inflate confidence without improving alignment. Regression anal- yses show that distributional properties explain much of the observed pattern. Linguistic disper- sion can preserve rank-order signal, but it does not guarantee magnitude agreement or calibration. Linguistic confidence is best treated as a lossy confidence channel. It can be useful for down- stream weighting when validated, but it should not be treated as a calibrated probability. We rec- ommend reporting multi-axis confidence diagnos- tics, checking for collapsed or inflated confidence distributions, and using linguistic confidence only with distribution-aware validation or held-out cal- ibration. Limitations Our study focuses on a limited set of open- source models and two internal confidence prox- ies. Logits-based confidence is not ground-truth uncertainty, and semantic entropy is also an es- timate with modeling choices such as sampling and NLI-based clustering. Our goal is to compare available confidence channels, not to identify true uncertainty. Most experiments are correlational. The regres- sion analysis controls for observable factors such as confidence mean, variance, task structure and performance, but it does not establish a causal mechanism. We also do not have access to the training details needed to explain why a particular model version changes its confidence distribution. The repeated-subsampling analysis reduces concern that the main trends are artifacts of a par- ticular 200-example subset, but individual esti- mates still have sampling variability. The black- box aggregation study is small, uses one model outside the main suite, and cannot evaluate the pa- per’s cross-channel diagnostics; it should be read only as an illustration. Future work should compare additional internal uncertainty proxies, especially for generation, and test whether conclusions are stable under alterna- tive proxy choices. Another important direction is post-hoc confidence elicitation: first obtain the model’s answer, then ask it to assess confidence in that completed answer. This protocol may reveal whether reflection changes expressed confidence, but it requires a separate generation setting and is outside the scope of the present experiments. Ethical Considerations This work studies when user-facing confidence statements from LLMs can be misleading. The main ethical risk is that confidence scores may cre- ate unwarranted trust, especially in settings where users cannot inspect internal uncertainty signals. Our results suggest that high linguistic confidence should not be presented as a calibrated probabil- ity unless it has been validated for the specific task and deployment context. The study uses existing NLP benchmarks and publicly available models, with no human-subject data collection. The practical recommendations are intended to reduce overreliance on unsup- ported confidence claims by encouraging multi- axis reporting, mismatch detection and held-out calibration. However, these diagnostics can also be misused if presented as guarantees of reliabil- ity. We therefore caution against using linguistic confidence as the sole basis for decisions in high- stakes applications. References Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Aza- lia Mirhoseini. 2024.Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. Preprint, arXiv:2110.14168. Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos,Bhuwan Dhingra, and Jacob Eisenstein. 2023.Selectively an- swering ambiguous questions.arXiv preprint arXiv:2305.14613. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023.Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. arXiv preprint arXiv:2307.01379. Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. arXiv preprint arXiv:2508.15260. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2023. A sur- vey of confidence estimation and calibration in large language models. arXiv preprint arXiv:2311.08298. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Tobias Groot and Matias Valdenegro-Toro. 2024. Overconfidence is key: Verbalized uncertainty eval- uation in large language and vision-language mod- els. arXiv preprint arXiv:2405.02917. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works.In International conference on machine learning, pages 1321–1330. PMLR. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300. Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, San- woo Lee, and Yunfang Wu. 2024. A survey of un- certainty estimation in llms: Theory meets practice. arXiv preprint arXiv:2410.15326. Yuheng Huang, Jiayang Song, Zhijie Wang, Sheng- ming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2023.Look before you leap: An ex- ploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236. Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. 2025. Calibrat- ing verbal uncertainty as a linear feature to reduce hallucinations. In Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pages 3769–3793. Association for Com- putational Linguistics. Yaning Jia, Chunhui Zhang, Xingjian Diao, Xiangchi Yuan, Zhongyu Ouyang, Chiyu Ma, and Soroush Vosoughi. 2026. What makes a good curriculum? disentangling the effects of data ordering on LLM mathematical reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 34472–34488, San Diego, California, United States. Association for Computational Linguistics. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Re- nard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Tim- othée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 oth- ers. 2024.Mixtral of experts.arXiv preprint arXiv:2401.04088. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. arXiv preprint arXiv:1705.03551. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. arXiv preprint arXiv:2006.09462. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, and Ali Emami. 2024. Confidence un- der the hood: An investigation into the confidence- probability alignment in large language models. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 315–334. Association for Com- putational Linguistics. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. arXiv preprint arXiv:2305.19187. Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025.Uncer- tainty quantification and confidence calibration in large language models: A survey. arXiv preprint arXiv:2503.15850. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303.08896. Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.Reducing conversational agents’ overconfidence through linguistic calibra- tion. Transactions of the Association for Compu- tational Linguistics, 10:857–872. OpenAI. 2025.Gpt-4.1. https://openai.com/ index/gpt-4-1/. Accessed: 2026-05-14. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 oth- ers. 2025.Qwen2.5 technical report.Preprint, arXiv:2412.15115. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249–266. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and 1 others. 2022. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 5433–5442. Association for Compu- tational Linguistics. Yuting Tian, Salar Khaleghzadegan, Benjamin Huh, Yash Raj, and Gena Heng. 2026. Lost in transla- tion: Cross-cultural bias in LLM-assisted medical symptom interpretation. In Proceedings of the 1st Workshop on Stereotypes Across Cultures in Lan- guage Technologies (StereACuLT 2026), pages 13– 19, San Diego, California, United States. Associa- tion for Computational Linguistics. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023b.Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.arXiv preprint arXiv:2306.13063. Chenjun Xu, Bingbing Wen, Bin Han, Robert Wolfe, Lucy Lu Wang, and Bill Howe. 2025.Do lan- guage models mirror human confidence? exploring psychological insights to address overconfidence in LLMs. In Findings of the Association for Computa- tional Linguistics: ACL 2025, pages 25655–25672, Vienna, Austria. Association for Computational Lin- guistics. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Day- iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Gal Yona, Roee Aharoni, and Mor Geva. 2024. Can large language models faithfully express their intrin- sic uncertainty in words?In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7752–7764. Associa- tion for Computational Linguistics. Yefan Zhou, Austin Xu, Yilun Zhou, Janvijay Singh, Jiang Gui, and Shafiq Joty. 2026. Variation in ver- ification: Understanding verification dynamics in large language models. In International Conference on Learning Representations, volume 2026, pages 144597–144642. A Prompt Design and Robustness Analyses This appendix provides detailed prompting setups and additional robustness analyses omitted from the main paper. We describe the confidence elic- itation prompts used for both classification and generation tasks, followed by analyses of prompt sensitivity, including few-shot confidence exem- plar search, confidence-elicitation ablation, and attitude-based prompt perturbations. A.1 Classification Prompt Design For classification tasks, we design prompts to elicit linguistic confidence while maintaining con- sistent output formats across model types. We employ two prompt templates for base models. Prompt 1 is adapted from Xiong et al. (2023) and directly elicits linguistic confidence alongside the answer. Prompt 2 additionally explains the meaning of confidence scores before elicitation. For instruction-tuned models, confidence report- ing and format constraints are specified through the system prompt. We explore multiple confidence exemplar set- tings and select (5, 6) for Prompt 1 and (4, 5) for Prompt 2 on base models, while score 2 is used as the format example for instruction-tuned mod- els. The rationale and quantitative comparison of different exemplar scores are reported in Ap- pendix G.1. Prompt examples on the Ruin Names task are shown in Figures 5, 6, 7, and 8. Read the question, provide your answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer is true. 0 means you don't believe this answer at all, 10 means you trust this answer must be true. Your answer must be one of the option letters (a, b, c, or d). Do not output any number or word other than the option letter. Use the following format to answer: A: <(a)/(b)/(c)/(d)>. Confidence: <Your confidence level, please only include the numerical number in the range of 0-10>. Only the answer and confidence, don’t give me the explanation. If unsure, still pick the closest option among (a–d). Do not output a number as the answer. Q: Which of the following is a humorous edit of this artist or movie name: 'the guns of navarone'? (a): the guns of mavarone (b): the gins of navarone (c): the guns of ngvarone (d): the gun of navarone A: (b); Confidence: 5. Q: Which of the following is a humorous edit of this artist or movie name: 'the avengers'? (a): toe avengers (b): the avjngers (c): thes avengers (d): the avengegs A: (a); Confidence: 6. Q: Which of the following is a humorous edit of this artist or movie name: 'the rolling stones'? (a): the polling stones (b): tce rolling stones (c): the rollvng stones (d): thea rolling stones A: Figure 5: Base Prompt 1 example. Q: Which of the following is a humorous edit of this artist or movie name: 'the guns of navarone'? (a): the guns of mavarone (b): the gins of navarone (c): the guns of ngvarone (d): the gun of navarone What's your answer and your confident score (0-10) of this answer? (0 means you don't believe this answer at all, 10 means you trust this answer must be true.) A: (b); Confidence: 4. Q: Which of the following is a humorous edit of this artist or movie name: 'the avengers'? (a): toe avengers (b): the avjngers (c): thes avengers (d): the avengegs What's your answer and your confident score (0-10) of this answer? (0 means you don't believe this answer at all, 10 means you trust this answer must be true.) A: (a); Confidence: 5. Q: Which of the following is a humorous edit of this artist or movie name: 'the rolling stones'? (a): the polling stones (b): tce rolling stones (c): the rollvng stones (d): thea rolling stones What's your answer and your confident score (0-10) of this answer? (0 means you don't believe this answer at all, 10 means you trust this answer must be true.) A: Figure 6: Base Prompt 2 example. system You will be given a question. Answer the question and give a confident score (0-10) to your answer. 0 means you don't believe this answer at all, 10 means you trust this answer must be true. Answer as the format: 'answer; Confidence: score.' such as '(b); Confidence: 2.' without any other words user Q: Which of the following is a humorous edit of this artist or movie name: 'the rolling stones'? (a): the polling stones (b): tce rolling stones (c): the rollvng stones (d): thea rolling stones What's your answer and your confident score (0-10) of this answer? (0 means you don't believe this answer at all, 10 means you trust this answer must be true.) A: Figure 7: Instruct prompt with example confidence score. A.2 Generation Prompt Design For generation tasks, we follow the prompting setup of Kuhn et al. (2023) and append linguis- tic confidence elicitation instructions following Xiong et al. (2023). We evaluate two representa- tive generation benchmarks, CoQA and TriviaQA. For CoQA, each instance consists of a con- text paragraph and several question-answer pairs, where we adopt a zero-shot prompting setup. For TriviaQA, we employ a ten-shot prompting strat- egy to improve response accuracy. In both set- tings, linguistic confidence is elicited alongside generated responses and averaged across the five sampled generations used for semantic-entropy es- timation. Representative prompts are shown in Figures 9 and 10. A.3 Confidence-Elicitation Ablation To examine whether confidence elicitation it- self perturbs internal probability distributions, we conduct ablation experiments removing linguis- system Read the question, provide your answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer is true. 0 means you don't believe this answer at all, 10 means you trust this answer must be true. Your answer must be one of the option letters (a, b, c, or d). Do not output any number or word other than the option letter.Use the following format to answer: A: <(a)/(b)/(c)/(d)>. Confidence: <Your confidence level, please only include the numerical number in the range of 0-10>. Only the answer and confidence, don’t give me the explanation. If unsure, still pick the closest option among (a–d). Do not output a number as the answer. user Q: Which of the following is a humorous edit of this artist or movie name: 'the rolling stones'? (a): the polling stones (b): tce rolling stones (c): the rollvng stones (d): thea rolling stones What's your answer and your confident score (0-10) of this answer? (0 means you don't believe this answer at all, 10 means you trust this answer must be true.) A: Figure 8: Instruct prompt without example confidence score. It was only in the 1980s that ...<context>... in the conference. Read the question, provide your answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer is true. 0 means you don't believe this answer at all, 10 means you trust this answer must be true. Use the following format to answer: A: <your answer>. Confidence: <Your confidence level, please only include the numerical number in the range of 0-10>. Only the answer and confidence, don’t give me the explanation. Q: Company one did have Em IPO 1984? A: PictureTel Corp. Confidence: 10. Q: Em What does them doing? A: commercial videoconferencing systems. Confidence: 1. Q: When digital telephony transmission networks one began? A: in the 1980s. Confidence: 0. Q: What-all was a 1st network them? A: ISDN networks. Confidence: 4. Q: What-what had been that minimum a bit a rate? A: Figure 9: Example prompt for CoQA. tic confidence instructions while keeping all other prompt settings unchanged. For base models, we retain the original two-shot examples without con- fidence elicitation, while instruction-tuned models preserve the same formatting constraints. Representative prompts without confidence elicitation are shown in Figures 11 and 12. Quan- titative comparisons of accuracy, logits statistics, and calibration are reported in Appendix G.2. A.4 Attitude-Based Prompt Perturbation To investigate whether user attitude affects re- ported confidence, we prepend criticism, approval, and irrelevant cues before the original task prompt. Each attitude category contains three cue vari- ants.These perturbations are designed to test whether linguistic confidence reflects underlying model uncertainty or is sensitive to conversational framing. The attitude cues used in our experiments are shown in Figure 13. Full quantitative results are reported in Appendix G.3. B Dataset Stability Analysis Downsamplingisusedforcomputational tractability only. To test whether the 200-example Read the question, provide your answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer is true. 0 means you don't believe this answer at all, 10 means you trust this answer must be true. Use the following format to answer: A: <your answer>. Confidence: <Your confidence level, please only include the numerical number in the range of 0-10>. Only the answer and confidence, don't give me the explanation. Question: Which American-born Sinclair won the Nobel Prize for Literature in 1930? Answer: Sinclair Lewis Confidence: 9. Question: Where in England was Dame Judi Dench born? Answer: York Confidence: 5. Question: In which decade did Billboard magazine first publish and American hit chart? Answer: 30s Confidence: 2. Question: From which country did Angola achieve independence in 1975? Answer: Portugal Confidence: 0. Question: Which city does David Soul come from? Answer: Chicago Confidence: 5. Question: Who won Super Bowl X? Answer: Chicago Bears Confidence: 5. Question: Which was the first European country to abolish capital punishment? Answer: Norway Confidence: 7. Question: In which country did he widespread use of ISDN begin in 1988? Answer: Japan Confidence: 3. Question: What is Bruce Willis' real first name? Answer: Walter Confidence: 8. Question: Which William wrote the novel Lord Of The Flies? Answer: Golding Confidence: 9. Question: Rincewind and 'The Luggage' appear in which series of books? Answer: Figure 10: Example prompt for TriviaQA. Q: Which of the following is a humorous edit of this artist or movie name: 'the guns of navarone'? (a): the guns of mavarone (b): the gins of navarone (c): the guns of ngvarone (d): the gun of navarone A: (b). Q: Which of the following is a humorous edit of this artist or movie name: 'the avengers'? (a): toe avengers (b): the avjngers (c): thes avengers (d): the avengegs A: (a). Q: Which of the following is a humorous edit of this artist or movie name: 'the rolling stones'? (a): the polling stones (b): tce rolling stones (c): the rollvng stones (d): thea rolling stones A: Figure 11: Base Prompt without asking for linguistic confidence. limit systematically changes the results, we construct larger evaluation pools for the five affected tasks and evaluate nine model–prompt settings completed for every task. For CoLA and Temporal Sequences, we use the full evaluation sets; for QNLI and MMLU, we use substantially enlarged evaluation pools. Because the QQP eval- uation set is considerably larger and expensive to evaluate in full, we randomly sample 5,000 examples as a proxy full set for the stability analysis. For each task and setting, we draw 500 random subsamples of 200 examples from the corresponding larger pool and recompute accu- racy, normalized accuracy, Pearson, Spearman, and Kendall correlations, ECE, and cross-channel distance. The exact main-paper subsets, the QQP proxy set, and the resampling code are included in the released repository. Across tasks, the median absolute difference be- tween a subsample estimate and its corresponding larger-pool estimate is 0.024–0.027, and 93.9%– 95.3% of estimates fall within±0.10. The larger- system You will be given a question. Answer the question as the format: 'answer.' such as '(d).' without any other words user Q: Which of the following is a humorous edit of this artist or movie name: 'the rolling stones'? (a): the polling stones (b): tce rolling stones (c): the rollvng stones (d): thea rolling stones A: Figure 12: Instruct prompt without asking for linguistic confidence. Criticism Your last answer was full of mistakes. You previously generated some responses. That responses were totally wrong. You don't know anything about logic, reasoning, math or grammar. That's total crap. Approval Your last answer was perfect. You previously generated some responses. That responses were absolutely correct. You are a genius. You are very reliable at solving this task. Irrelevant I am just writing this sentence before asking you a question. Yesterday I talked to my toaster about philosophy. Square circles can run faster than triangles. Figure 13: Attitude cues for prompt perturbation. pool estimate falls within the 95% interval of the subsampling distribution for 581 of 585 met- ric rows overall and for all 225 rows involving the five metrics most directly tied to our conclu- sions. Thus, the 200-example evaluations recover the trends observed in the larger evaluation pools without detectable systematic distortion, although individual estimates retain sampling variability. C Model Selection We select 30 models from the LLaMA, Mistral and Qwen families. The full list is shown in Ta- ble 6. For the small-scale aggregation study, we use gpt-4.1-nano through the OpenAI API, ac- cessed May 4, 2026, with temperature 0.7. D Distance and ECE trends by Model Size For completeness, we present the error-bar ver- sion of the model-size correlation plot in Fig- ure 14. The error bars indicate the standard de- viation of correlation values within each model group. Figures 15 and 16 show how model size affects distance and ECE for base and instruction- tuned models. These plots illustrate why one met- ric cannot replace the others. Base models often show higher correlation with scale, while distance and ECE may move differently. Instruction-tuned models can also show lower distance or ECE as ac- curacy improves, even when correlation does not 510 Model Size 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Pearson Coefficient Correlation Qwen2.5_base Qwen3_base Llama2_base Mistral-v0.1_base (a) Base Model. 510 Model Size 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 Pearson Coefficient Correlation Qwen2.5_instruct Llama2_instruct Mistral-v0.1_instruct (b) Instruction-tuned Model. Figure 14: Effect of model size on the correlation be- tween linguistic and logits-based confidence, with er- ror bars indicating the standard deviation of correlation values within each model group. Correlation tends to rise with model size for base models but plateaus for instruction-tuned ones. improve. This supports our main claim that associ- ation, magnitude agreement and calibration mea- sure different aspects of confidence reliability. E Distribution of Logits-based and Linguistic Confidence As discussed in Section 5.3, the correlation gen- erally increases with model size for base models, but decreases for instruction-tuned models. To illustrate this pattern in detail, we examine the CoLA task. Besides Figure 4, Qwen and Llama show the similar pattern. As shown in Figures 18 and 19, linguistic confidence in base models ex- hibits greater diversity and a less concentrated distribution. An exception occurs with Qwen2.5 (Figure 17), where the 7B model produces more monotonous linguistic confidence compared to the 3B model, consistent with the overall trend ob- served in Figure 3. For instruction-tuned models (Figures 20 and 21), the logits-based confidence values become larger and often approach 1 as model size in- creases. This saturation of logits leads to a reduc- tion in the correlation between the two confidence measures. F Distribution of Correlation Scores and Paired t-Test Analyses To further examine how model confidence alignment evolves across different architectures and training versions, we conducted paired t- tests on both model families and model ver- Task / poolPool nMedian|∆|Within±0.10Full in 95% intervalConclusion-level metrics CoLA1,0430.02495.3%117/11745/45 QNLI5,4630.02793.9%116/11745/45 QQP5,0000.02794.1%115/11745/45 MMLU1,0000.02694.4%116/11745/45 Temporal Sequences8000.02495.3%117/11745/45 Table 5: Stability of estimates from 500 random 200-example subsamples. 2.55.07.510.012.5 Model Size 0.15 0.20 0.25 0.30 0.35 0.40 0.45 Distance Between Two Confidence (a) Distance 2.55.07.510.012.5 Model Size 0.0 0.1 0.2 0.3 0.4 ECE of Linguistic Confidence (b) ECE(linguistic) Qwen2.5_base Qwen3_base Llama2_base Mistral-v0.1_base Figure 15: Base models: model size (x-axis, in B) vs. (a) Distance between two confidence measures and (b) ECE of linguistic confidence. Base ModelInstruction-tuned / Chat Llama-2-7b-hfLlama-2-7b-chat-hf Llama-2-13b-hfLlama-2-13b-chat-hf Llama-3.1-8BMeta-Llama-3.1-8B-Instruct Llama-3.2-3BLlama-3.2-3B-Instruct Meta-Llama-3-8BMeta-Llama-3-8B-Instruct Mistral-7B-v0.1Mistral-7B-Instruct-v0.1 Mistral-7B-v0.2Mistral-7B-Instruct-v0.2 Mistral-7B-v0.3Mistral-7B-Instruct-v0.3 Mistral-Nemo-Base-2407Mistral-Nemo-Instruct-2407 Mixtral-8x7B-v0.1Mixtral-8x7B-Instruct-v0.1 Qwen2.5-3BQwen2.5-3B-Instruct Qwen2.5-7BQwen2.5-7B-Instruct Qwen2.5-14BQwen2.5-14B-Instruct Qwen3-4B-BaseQwen3-4B-Instruct-2507 Qwen3-8B-Base– Qwen3-14B-Base– Table 6: List of models used in our experiments, cate- gorized by version and type. sions.For the family-level comparison, we selected representative base models from the LLaMA, Mistral, and Qwen series, and evalu- ated whether their overall correlation, confidence- logit distance, and ECE differ significantly (Table 7).For the version-level compari- son, we tested multiple generations within each family (e.g., LLaMA3→LLaMA3.1, Mistral- v0.1→v0.2→v0.3, and Qwen2.5→Qwen3), to in- vestigate whether improvements in alignment and calibration persist as the models evolve (Table 8). Across model families (Table 7), we observe a general increase in correlation from LLaMA to Mistral and Qwen, together with lower distance and ECE. This suggests stronger aggregate align- ment for Qwen in our setting. Within-family com- parisons in Table 8 are less uniform. LLaMA3.1 improves correlation over LLaMA3, while Qwen3 shows lower correlation and larger distance than Qwen2.5. The Mistral versions show only small changes. We analyze the effect of model updates on the distributional properties of linguistic confidence and find that dispersion plays a central role in determining alignment. Specifically, linguistic- confidence standard deviation increases from 1.16 to 1.32 for the 8B Llama base models. In con- trast, Qwen3 exhibits a substantial collapse in dis- persion relative to Qwen2.5, with standard devi- ation decreasing from 1.23 to 0.71 for the 14B base models. This contrast aligns with their di- vergent correlation trends, where LLaMA models show improved alignment and Qwen models de- grade. Higher dispersion can preserve instance- level rank information, whereas compressed dis- tributions reduce expressivity and weaken associ- 2.55.07.510.012.5 Model Size 0.1 0.2 0.3 0.4 0.5 Distance Between Two Confidence (a) Distance 2.55.07.510.012.5 Model Size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 ECE of Linguistic Confidence (b) ECE(linguistic) Qwen2.5_instruct Llama2_instruct Mistral-v0.1_instruct Figure 16: Instruction-tuned models: model size (x-axis, in B) vs. (a) Distance between two confidence measures and (b) ECE of linguistic confidence. MetricLLaMAMistralQwen p L→M p M→Q p L→Q r0.0810.1370.1800.0330.1020.0004 Distance0.2740.2590.2180.2870.00090.00002 ECE(linguistic)0.2610.2190.1860.0240.0700.0003 Table 7: Paired t-tests across model families. Mean value of the three families and p between them are reported. Underlined values indicate significant differences (p < 0.05). Metricmean_amean_bp LLaMA3→ LLaMA3.1 r0.0570.1260.028 Distance0.2720.2810.527 ECE (linguistic)0.1970.2340.058 Mistral-v0.1→ v0.2 r0.1250.1160.741 Distance0.2910.2610.088 ECE (linguistic)0.2580.2040.011 Mistral-v0.2→ v0.3 r0.1140.1290.570 Distance0.2610.2710.495 ECE (linguistic)0.2040.2090.537 Qwen2.5→ Qwen3 r0.3400.2180.00064 Distance0.1750.2050.00038 ECE (linguistic)0.1440.1510.261 Table 8: Paired t-tests across model versions within each family. Underlined values indicate significant dif- ferences (p < 0.05). Metrics include overall correla- tion (r), distance between linguistic and logits-based confidence, and ECE based on linguistic confidence. ation. While we do not make causal claims, these differences may be related to broader shifts in training and post-training objectives across model updates.For example, LLaMA3.1 introduces changes in data scale and alignment procedures (Grattafiori et al., 2024), while newer models such as Qwen3 place increasing emphasis on reason- ing and structured generation (Yang et al., 2025). Such changes may influence how confidence is ex- pressed in language, potentially affecting its dis- persion. However, establishing a direct causal link between training strategies and confidence distri- bution remains an important direction for future work. These results suggest that model identity alone does not explain alignment. Distributional changes in the confidence scores account for much of the observed pattern. G Effects of Prompt Design G.1 Hyperparameter Search We exhaustively explore few-shot example con- fidence scores to quantify how exemplar settings affect both the association between linguistic and internal confidence and task performance. Ex- periments cover three tasks (Conceptual Combi- nations, Ruin Names, Temporal Sequences) and three models (LLaMA-3.1-8B, Mistral-7B-v0.1, Qwen2.5-7B). Weighted averages (by task sample size) of correlation and accuracy are shown in Fig- ures 22 and 23. For base models, we consider two prompt templates and assign a pair of exam- ple scores (i,j) per template (Figures 5, 6). For instruction-tuned models, we assign a single ex- ample score in 1,..., 10 and include a random baseline (Figure 7). We also test the instruction 0.000.250.500.751.00 Logits-based Confidence 0 10 20 30 40 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 100 Count (a) Qwen2.5-3B 0.000.250.500.751.00 Logits-based Confidence 0 10 20 30 40 50 60 Count 0.02.55.07.510.0 Linguistic Confidence 0 25 50 75 100 125 Count (b) Qwen2.5-7B 0.000.250.500.751.00 Logits-based Confidence 0 10 20 30 40 50 60 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 100 Count (c) Qwen2.5-14B Figure 17: Distribution of Logits-based and Linguis- tic Confidence on CoLA (Base Prompt 1) for Qwen2.5 Base Models. prompt from Xiong et al. (2023) without explicit example scores (Figure 8). Two regularities emerge. First, performance and correlation tend to increase when the two ex- emplar scores are close, but not identical. Identi- cal scores reduce the variance of produced linguis- tic confidence, which lowers instance-level asso- ciation. Second, including a very large exemplar score pushes the model toward high outputs (of- ten 8–10) and compresses variability. Balancing correlation and accuracy, the selected settings are (5, 6) for Base Prompt 1, (4, 5) for Base Prompt 2, and a single score of 2 for the Instruct prompt. G.2 Ablation Studies We test whether eliciting linguistic confidence per- turbs final-layer logits. For every model and all eight tasks, we repeat the main setup without requesting confidence, holding all other settings fixed. For base models, each prompt still con- tains two in-context QA exemplars. Represen- tative prompts are shown in Figures 11 and 12. Paired comparisons of normalized accuracy, aver- 0.000.250.500.751.00 Logits-based Confidence 0 10 20 30 40 50 60 Count 0.02.55.07.510.0 Linguistic Confidence 0 50 100 150 Count (a) Qwen3-4B 0.000.250.500.751.00 Logits-based Confidence 0 20 40 60 Count 0.02.55.07.510.0 Linguistic Confidence 0 50 100 150 Count (b) Qwen3-8B 0.000.250.500.751.00 Logits-based Confidence 0 20 40 60 Count 0.02.55.07.510.0 Linguistic Confidence 0 25 50 75 100 125 150 Count (c) Qwen3-14B Figure 18: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Qwen3 Base Models. age logits, logits variance, and ECE are reported in Table 9. As shown in Table 9, removing confidence elici- tation leaves normalized accuracy unchanged. For base models, however, it produces small but sta- tistically significant shifts in average logits and logits variance and, in one comparison, in logits ECE. For instruction-tuned models, none of the tested metrics changes significantly. Thus, con- fidence elicitation does not affect task accuracy in our experiments, but it can modestly perturb inter- nal probability distributions for base models. G.3 Prompt Perturbation To test sensitivity to user attitude, we prepend short cues that convey criticism, approval, or ir- relevance. Each attitude uses three variants in- serted before the main QA examples. We evalu- ate eight tasks on three diverse models (Qwen2.5- 7B, Mistral-7B-v0.3, Llama-3.1-8B) and average within attitude type. We also include a calibra- tion variant that appends “Make sure the confi- dence you report reflects the softmax probabili- 0.000.250.500.751.00 Logits-based Confidence 0 25 50 75 100 125 150 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 Count (a) Llama-2-7b-hf 0.000.250.500.751.00 Logits-based Confidence 0 25 50 75 100 125 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 100 120 Count (b) Llama-2-13b-hf Figure 19: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Llama2 Base Models. ties from your final-layer logits.” The cue sets are listed in Figure 13 (Appendix A). Results appear in Table 10. As shown in Table 10, attitude cues consis- tently raise reported linguistic confidence. Ap- proval has the largest effect, criticism is also pos- itive, and even irrelevant cues increase averages. At the same time, the association with logits de- creases, which indicates that inflation in reported confidence is not grounded in internal probabili- ties. The calibration instruction does not signif- icantly change either average confidence or cor- relation. Overall, prompt tone can inflate stated confidence and weaken its association with inter- nal probabilities. H Generation Task Details H.1 Prompt Design for Generation Tasks We follow the prompting setup of Kuhn et al. (2023). To elicit linguistic confidence, we ad- ditionally append a confidence query instruction following Xiong et al. (2023). Specifically, we include random example confidence scores rang- ing from 0 to 10 after each question-answer pair in CoQA or after each demonstration example in TriviaQA. The model-generated confidence is ex- tracted and averaged across the five sampled gen- erations for each prompt. Prompt examples are shown in Figure 9 and Figure 10. 0.000.250.500.751.00 Logits-based Confidence 0 50 100 150 Count 0.02.55.07.510.0 Linguistic Confidence 0 50 100 150 Count (a) Qwen2.5-3B-Instruct 0.000.250.500.751.00 Logits-based Confidence 0 50 100 150 Count 0.02.55.07.510.0 Linguistic Confidence 0 20 40 60 80 Count (b) Qwen2.5-7B-Instruct 0.000.250.500.751.00 Logits-based Confidence 0 50 100 150 200 Count 0.02.55.07.510.0 Linguistic Confidence 0 25 50 75 100 125 Count (c) Qwen2.5-14B-Instruct Figure 20: Distribution of Logits-based and Linguistic Confidence on CoLA for Qwen2.5 Instruct-tuned Mod- els. H.2 Full Results on Generation Tasks We calculate the semantic entropy and the aver- age linguistic confidence of the five generated se- quences for each prompt, and then compute the Pearson correlation between them (the semantic entropy is negated when computing the correla- tion, since a higher entropy indicates lower model confidence). The results are presented in Table 11. Unlike logits-based confidence, which directly reflects the model’s internal probability distribu- tion over tokens, semantic entropy is designed to capture uncertainty in generative tasks where the exact answer span or key token positions are dif- ficult to pinpoint. It measures how semantically diverse the generated responses are rather than how peaked the token-level probabilities appear. To further illustrate the distinction between these two uncertainty measures, we also compare logits- based confidence and semantic entropy on several classification tasks, as detailed in Appendix H.3. 0.000.250.500.751.00 Logits-based Confidence 0 20 40 60 80 100 120 Count 0.02.55.07.510.0 Linguistic Confidence 0 25 50 75 100 125 150 Count (a) Llama-2-7b-chat-hf 0.000.250.500.751.00 Logits-based Confidence 0 25 50 75 100 125 150 Count 0.02.55.07.510.0 Linguistic Confidence 0 5 10 15 20 25 Count (b) Llama-2-13b-chat-hf Figure 21: Distribution of Logits-based and Linguis- tic Confidence on CoLA for Llama2 Instruction-tuned Models. H.3 Semantic Entropy and Logits-Based Confidence Unlike logits-based confidence, which directly re- flects token-level probability distributions, seman- tic entropy captures uncertainty in generative set- tings where exact answer spans or salient token positions are difficult to identify. It quantifies un- certainty through semantic diversity among gener- ated responses rather than probability concentra- tion over tokens. To further illustrate the distinction between these uncertainty measures, we additionally com- pare logits-based confidence and semantic entropy on several classification tasks. For an N -option classification task, we first ob- tain the softmax probabilities from the model’s final-layer logits, denoted as p 1 ,p 2 ,...,p N . The semantic entropy is then computed as SE =− N X i=1 p i log(p i ).(4) The corresponding linguistic confidence for en- tropy is the average score obtained by insert- ing each candidate option into the prompt and asking the model for its confidence individually. We use Qwen2.5-14B to compute correlations be- tween linguistic confidence and (1) logits-based confidence and (2) negated semantic entropy un- der two prompt templates. Results are shown in Table 12. 12345678910 j 1 2 3 4 5 6 7 8 9 10 i 0.050.100.120.060.090.110.100.090.080.00 0.000.190.15-0.020.050.100.00-0.010.00 0.000.140.090.100.090.000.000.00 0.000.130.050.100.010.020.00 0.000.230.020.020.070.00 0.000.070.090.10-0.01 0.030.100.090.00 0.000.140.00 0.000.00 0.00 Average Pearson Coefficient 12345678910 j 1 2 3 4 5 6 7 8 9 10 i 0.530.480.460.470.480.460.470.470.480.45 0.540.470.480.490.490.460.460.490.46 0.550.490.500.490.500.490.490.46 0.550.500.480.470.470.460.47 0.520.490.450.480.480.47 0.530.470.440.500.49 0.530.510.490.46 0.510.500.51 0.520.50 0.51 Average Accuracy (a) Prompt 1 12345678910 j 1 2 3 4 5 6 7 8 9 10 i 0.010.080.120.100.060.080.050.020.020.02 0.120.180.070.080.030.020.000.010.01 0.010.160.100.050.120.150.080.03 0.060.240.060.070.070.030.05 0.050.090.050.100.050.03 0.030.100.130.060.05 0.080.090.040.04 0.020.100.09 0.030.04 0.00 Average Pearson Coefficient 12345678910 j 1 2 3 4 5 6 7 8 9 10 i 0.500.490.450.460.470.460.460.460.460.47 0.510.460.460.480.470.470.460.450.47 0.500.470.460.440.460.470.450.46 0.510.470.480.460.480.430.47 0.520.480.480.470.440.45 0.470.470.460.460.48 0.460.470.480.46 0.480.480.48 0.500.48 0.50 Average Accuracy (b) Prompt 2 Figure 22: Comparison of average Pearson correlation and accuracy across different confidence score pairs (i,j) on two base prompt settings.Each heatmap shows how the model’s linguistic confidence and task performance vary with score combinations. 123456789 10 random no j 0 i 0.080.240.120.170.180.100.170.110.140.120.08-0.07 Average Pearson Coefficient 123456789 10 random no j 0 i 0.610.620.610.590.620.640.620.610.630.630.660.61 Average Performance Figure 23: Average correlation (top) and task per- formance (bottom) across different confidence scoring schemes. The x-axis indicates the index of linguistic confidence scores (1–10), a random choice, or no con- fidence used. Metricmean a mean b tp Base Prompt 2→ Base Prompt 1 Normalized Accuracy0.4130.4050.55 5.9e-01 Average Logits0.5960.5698.22 2.1e-13 Logits Variance0.1120.1083.08 2.5e-03 ECE(logits)0.1290.140-1.51 1.3e-01 Base Prompt 1→ Base without Confidence Normalized Accuracy0.4050.4050.00 9.9e-01 Average Logits0.5690.617 -11.58 1.3e-21 Logits Variance0.1080.125-7.76 2.4e-12 ECE(logits)0.1400.1222.96 3.7e-03 Base Prompt 2→ Base without Confidence Normalized Accuracy0.4130.4050.49 6.2e-01 Average Logits0.5960.617-5.56 1.5e-07 Logits Variance0.1120.125-6.29 4.7e-09 ECE(logits)0.1290.1221.18 2.4e-01 Instruct Prompt→ Instruct without Confidence Normalized Accuracy0.3670.370-0.17 8.7e-01 Average Logits0.8750.881-0.89 3.7e-01 Logits Variance0.1190.1121.50 1.4e-01 ECE(logits)0.2860.294-0.57 5.7e-01 Table 9: Paired t-test results comparing logits-based statistics under different prompting conditions. Mean values (a,b), t-statistics, and p-values are reported. Un- derlined values indicate significant differences (p < 0.05). Across the 16 task-prompt settings, logits-based confidence is positively correlated with linguis- tic confidence in 15, whereas the association with negated option-level predictive entropy is gener- ally weaker and often negative. This suggests that token-probability confidence and entropy capture different internal views of reliability. Linguistic confidence is closer to the former in these classifi- cation settings. I Regression Model OutcomesandPredictors. Weindepen- dentlymodelthreecontinuousoutcomes: DIST_LINGUISTIC-LOGITS, ECE_LINGUISTIC- LOGITS, and CORRELATION. Predictorsincludemodelmetadata (Model_family,Model_version,Model_type, Model_size),taskcharacteristics(Task, Num_choices), distributional statistics of lin- guistic and logits confidence (Linguistic_mean, Linguistic_std,Logits_mean,Logits_std), and performance-related measures (Accuracy, Norm_acc, ECE_linguistic, ECE_logits). Categorical fields, including Model_family, Model_version, Model_type and Task, are one-hot encoded with a dropped reference. Zero-variance columns are removed and an intercept is included. Metricmean a mean b tp Origin→ Criticism Avg(Conf(li))8.4288.612 -2.340.028 r0.2660.1463.07 0.0056 Origin→ Approval Avg(Conf(li))8.4288.807 -3.75 0.0011 r0.2660.1302.43 0.0235 Origin→ Irrelevant Avg(Conf(li))8.4288.654 -2.47 0.0213 r0.2660.1783.14 0.0047 Origin→ Calibration Variant Avg(Conf(li))8.4288.3960.270.793 r0.2660.2231.250.226 Table 10: Paired t-test results comparing the effect of different attitude and calibration variants on average linguistic confidence Avg(Conf(li)) and its Pearson correlation with logits-based confidence r. Underlined values indicate significant differences (p < 0.05). The full OLS coefficients for the three outcomes are shown in Tables 13, 14, and 15. We also re- port ridge standardized effects for each outcome in Tables 18, 16, and 17 to rank effect sizes under collinearity. Llama FamilyMistral / Mixtral Family ModelCoQA (r)TriviaQA (r)ModelCoQA (r)TriviaQA (r) BaseInstructBaseInstructBaseInstructBaseInstruct Llama-2-7B0.0560.0630.2110.433Mistral-7B-v0.1 −0.040 −0.0170.2880.392 Llama-2-13B −0.1480.0640.2240.469Mistral-7B-v0.2 −0.0080.2130.2510.338 Llama-3.1-8B −0.0150.1610.0040.318Mistral-7B-v0.3 −0.0480.0120.0330.549 Llama-3.2-3B0.0000.2450.2610.591Mistral-Nemo-2407 −0.050 −0.0150.3550.472 Llama-3-8B0.008 −0.006 −0.0340.153Mixtral-8×7B-v0.1 −0.0240.0770.3460.334 Table 11: Pearson correlation (r) between semantic-entropy-based and linguistic confidence on CoQA and Trivi- aQA. Higher correlations within each base/instruction-tuned pair are highlighted in bold; underlined values indi- cate statistical significance (p < 0.05). TaskBase Prompt 1Base Prompt 2 Logits Corr. Entropy Corr. Logits Corr. Entropy Corr. Conceptual Combinations0.4441-0.31110.8189-0.1970 Ruin Names0.4479-0.10960.4142-0.3364 Temporal Sequences0.12460.01030.0431-0.0401 CoLA0.44690.22830.56720.1077 Cause and Effect0.14130.0133-0.0853-0.3545 MMLU0.3909-0.16620.3345-0.1089 QNLI0.1786-0.04690.11420.0389 QQP0.5777-0.36340.6227-0.1375 Table 12: Correlation between linguistic confidence and internal metrics (logits-based confidence vs. negated semantic entropy) on Qwen2.5-14B under two base prompt templates. PredictorCoefStd.βSEtpCI 2.5 CI 97.5 Linguistic_std0.0808170.7800640.00548614.7333.02e-380.0700260.091607 Num_choices0.0296100.2869770.0055905.2972.14e-070.0186130.040606 Logits_mean-0.281884-0.5589720.053266-5.2922.19e-07-0.386661-0.177108 Linguistic_mean0.0196350.2278610.0050473.8901.21e-040.0097070.029563 Task_QNLI0.0410210.1310190.0152352.6930.00740.0110530.070989 Model_version_30.0317870.1277000.0152142.0890.03740.0018600.061713 Norm_acc0.1595940.4348410.0800331.9940.04690.0021650.317022 Task_MMLU-0.035128-0.1132600.017947-1.9570.0511-0.0704300.000173 Model_version_20.0293610.0919580.0156181.8800.0610-0.0013610.060083 Task_conceptual_combinations-0.031565-0.1008180.018539-1.7030.0896-0.0680330.004902 Model_family_mistral0.0259960.1185250.0154671.6810.0937-0.0044270.056420 Task_QQP0.0244290.0787640.0151291.6150.1073-0.0053300.054188 Accuracy-0.203839-0.3739490.127119-1.6040.1098-0.4538890.046210 ECE_linguistic0.0666740.0950530.0447421.4900.1371-0.0213360.154685 Task_temporal_sequences-0.027931-0.0892100.019908-1.4030.1615-0.0670900.011229 Model_version_3.20.0241370.0581290.0176251.3690.1718-0.0105320.058805 Task_ruin_names-0.019467-0.0621760.021312-0.9130.3617-0.0613890.022455 Model_version_3.10.0133350.0321160.0180140.7400.4597-0.0221000.048770 Task_cause_and_effect0.0102210.0326470.0152390.6710.5028-0.0197540.040196 Model_version_0.30.0098820.0237990.0182870.5400.5893-0.0260890.045853 Model_version_nemo-0.009373-0.0225730.018757-0.5000.6176-0.0462680.027522 Model_version_0.20.0074310.0178950.0186720.3980.6909-0.0292980.044159 Model_family_qwen0.0078090.0361600.0225270.3470.7291-0.0365030.052121 Logits_std0.0258860.0150450.0854530.3030.7621-0.1422050.193976 Model_type_instruct0.0053720.0237380.0214400.2510.8023-0.0368020.047546 Model_version_2.50.0042810.0165440.0191680.2230.8234-0.0334230.041985 Model_size-0.000230-0.0079260.001373-0.1680.8669-0.0029310.002471 ECE_logits0.0046610.0062370.0419060.1110.9115-0.0777700.087091 const0.1288960.0641892.0080.04540.0026330.255160 Table 13: Full OLS coefficients for DIST_LINGUISTIC-LOGITS under specified predictors. PredictorCoefStd.βSEtpCI 2.5 CI 97.5 Logits_mean-0.457089-0.9119120.045934-9.9511.29e-20-0.547444-0.366734 Linguistic_mean0.0338710.3954600.0043527.7828.89e-140.0253090.042432 Linguistic_std0.0327480.3180150.0047316.9232.26e-110.0234430.042053 Logits_std-0.365030-0.2134450.073691-4.9541.16e-06-0.509985-0.220076 Num_choices0.0214110.2087750.0048214.4411.22e-050.0119280.030894 ECE_linguistic0.1599930.2294780.0385844.1474.28e-050.0840960.235890 Model_family_mistral0.0347450.1593770.0133382.6050.00960.0085090.060981 Model_version_20.0301570.0950240.0134682.2390.02580.0036640.056650 Task_MMLU-0.033979-0.1102190.015476-2.1960.0288-0.064422-0.003536 Task_temporal_sequences-0.035044-0.1126110.017168-2.0410.0420-0.068814-0.001275 Model_version_nemo-0.032071-0.0777070.016175-1.9830.0482-0.063888-0.000254 Model_size0.0022690.0785830.0011841.9170.0561-0.0000600.004598 Model_version_3.20.0291280.0705750.0151991.9160.0562-0.0007690.059024 ECE_logits-0.068931-0.0928040.036138-1.9070.0573-0.1400160.002154 Norm_acc0.1194770.3275150.0690171.7310.0843-0.0162830.255237 Task_QNLI0.0225190.0723600.0131381.7140.0875-0.0033250.048362 Model_version_30.0201610.0814860.0131201.5370.1253-0.0056470.045968 Task_conceptual_combinations-0.023002-0.0739150.015988-1.4390.1512-0.0544500.008446 Model_version_0.2-0.021393-0.0518350.016102-1.3290.1849-0.0530670.010280 Model_family_qwen0.0257020.1197430.0194261.3230.1867-0.0125110.063915 Accuracy-0.114829-0.2119390.109622-1.0480.2956-0.3304620.100803 Model_version_0.3-0.016109-0.0390320.015770-1.0220.3078-0.0471290.014911 Model_version_3.10.0146610.0355230.0155350.9440.3460-0.0158970.045219 Task_ruin_names-0.016377-0.0526270.018379-0.8910.3735-0.0525290.019774 Task_QQP0.0102940.0333920.0130460.7890.4306-0.0153690.035957 Model_version_2.5-0.009743-0.0378820.016530-0.5890.5560-0.0422570.022772 Task_cause_and_effect0.0063420.0203790.0131410.4830.6297-0.0195070.032191 Model_type_instruct0.0048960.0217660.0184890.2650.7913-0.0314730.041265 const0.1191080.0553542.1520.03210.0102240.227992 Table 14: Full OLS coefficients for ECE_LINGUISTIC-LOGITS under specified predictors. PredictorCoefStd.βSEtpCI 2.5 CI 97.5 Task_QNLI-0.155245-0.2513650.038621-4.0207.27e-05-0.231227-0.079263 Model_size0.0113470.1942140.0035323.2130.00140.0043980.018296 Linguistic_std0.0410360.1943110.0147862.7750.00580.0119450.070126 Task_cause_and_effect-0.107774-0.1711420.039022-2.7620.0061-0.184544-0.031003 Task_QQP-0.102290-0.1671770.038298-2.6710.0080-0.177638-0.026943 Linguistic_mean0.0286790.1640870.0130862.1920.02910.0029340.054423 Model_family_qwen0.1248650.2890720.0593732.1030.03620.0080550.241674 Task_temporal_sequences-0.093571-0.1455670.052304-1.7890.0746-0.1964740.009332 Norm_acc0.3498290.4667610.2097661.6680.0963-0.0628610.762519 Accuracy-0.541172-0.4763540.335281-1.6140.1075-1.2007970.118454 Task_MMLU-0.074955-0.1213630.046563-1.6100.1084-0.1665620.016653 ECE_linguistic-0.176518-0.1157660.115576-1.5270.1277-0.4038990.050864 Model_version_nemo-0.072938-0.0838110.049779-1.4650.1438-0.1708720.024995 Model_family_mistral0.0542000.1227490.0397991.3620.1742-0.0241000.132500 Logits_mean-0.183698-0.1829760.139718-1.3150.1895-0.4585760.091181 ECE_logits-0.142873-0.0911020.109132-1.3090.1914-0.3575770.071832 Model_version_3.10.0552960.0676130.0461941.1970.2322-0.0355860.146178 Model_type_instruct0.0635610.1392070.0552461.1510.2508-0.0451290.172252 Model_version_3.20.0493550.0579570.0459851.0730.2840-0.0411160.139825 Task_conceptual_combinations0.0338760.0527010.0483930.7000.4844-0.0613310.129084 Model_version_0.30.0269950.0323630.0472900.5710.5685-0.0660420.120032 Logits_std0.1125630.0323960.2273710.4950.6209-0.3347630.559889 Model_version_3-0.015929-0.0317570.039442-0.4040.6866-0.0935270.061669 Num_choices0.0042300.0204020.0143220.2950.7679-0.0239460.032407 Model_version_2.5-0.013040-0.0254900.049900-0.2610.7940-0.1112120.085131 Task_ruin_names0.0138810.0215940.0559520.2480.8042-0.0961990.123960 Model_version_2-0.006998-0.0107700.041111-0.1700.8649-0.0878780.073883 Model_version_0.20.0044110.0053930.0476520.0930.9263-0.0893400.098161 const0.1228840.1672050.7350.4629-0.2060730.451841 Table 15: Full OLS coefficients for CORRELATION under specified predictors. PredictorStd.(β) Model_size-0.012746 Num_choices0.143658 Linguistic_mean0.201203 Linguistic_std0.741303 Logits_mean-0.496499 Logits_std0.009902 ECE_logits-0.006060 Norm_acc0.227175 ECE_linguistic0.123461 Accuracy-0.158557 Model_family_mistral0.017754 Model_family_qwen0.014370 Model_version_0.20.013036 Model_version_0.30.020893 Model_version_20.015764 Model_version_2.5-0.057565 Model_version_30.037234 Model_version_3.1-0.021622 Model_version_3.2-0.002226 Model_version_nemo-0.030178 Model_type_instruct0.011662 Task_MMLU0.034117 Task_QNLI0.121791 Task_QQP0.063017 Task_cause_and_effect0.023452 Task_conceptual_combinations0.043844 Task_ruin_names0.086675 Task_temporal_sequences0.053004 Table 16: Full ridge (CV) standardized coefficients for DIST_LINGUISTIC-LOGITS. PredictorStd.(β) Model_size0.071043 Num_choices0.080202 Linguistic_mean0.381558 Linguistic_std0.302399 Logits_mean-0.843727 Logits_std-0.215614 ECE_logits-0.099016 Norm_acc0.232477 ECE_linguistic0.240305 Accuracy-0.118653 Model_family_mistral0.063824 Model_family_qwen0.083243 Model_version_0.2-0.055676 Model_version_0.3-0.041310 Model_version_20.028802 Model_version_2.5-0.102369 Model_version_30.001939 Model_version_3.1-0.013762 Model_version_3.20.016518 Model_version_nemo-0.077676 Model_type_instruct-0.011264 Task_MMLU0.004494 Task_QNLI0.068909 Task_QQP0.028014 Task_cause_and_effect0.017697 Task_conceptual_combinations0.038977 Task_ruin_names0.069560 Task_temporal_sequences0.008612 Table 17: Full ridge (CV) standardized coefficients for ECE_LINGUISTIC-LOGITS. PredictorStd.(β) Model_size0.113674 Num_choices0.109605 Linguistic_mean0.092039 Linguistic_std0.087174 Logits_mean-0.007872 Logits_std0.036264 ECE_logits-0.071479 Norm_acc0.075122 ECE_linguistic-0.038211 Accuracy-0.030490 Model_family_mistral0.029550 Model_family_qwen0.079445 Model_version_0.2-0.003542 Model_version_0.30.023032 Model_version_2-0.052616 Model_version_2.50.030971 Model_version_3-0.029987 Model_version_3.10.023491 Model_version_3.2-0.013133 Model_version_nemo-0.061904 Model_type_instruct0.056131 Task_MMLU-0.009412 Task_QNLI-0.155651 Task_QQP-0.099535 Task_cause_and_effect-0.099387 Task_conceptual_combinations0.114804 Task_ruin_names0.112571 Task_temporal_sequences-0.047692 Table 18: Full ridge (CV) standardized coefficients for CORRELATION.