Paper deep dive
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Siyang Wu, Yibo Jiang, Bryon Aragam
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/13/2026, 5:09:06 AM
Summary
The paper introduces Cross-Contextual Consistency (C3), a behavioral metric for evaluating Large Language Model (LLM) credibility by measuring the stability of model outputs under semantically equivalent but contextually perturbed prompts. The authors demonstrate that C3, calculated via distributional distance (e.g., MMD) between original and perturbed generations, correlates strongly with truthfulness and factuality across 16 models and six benchmarks, outperforming baselines like self-consistency and self-report.
Entities (16)
Relation Signals (17)
Yibo Jiang → authored → Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
confidence 99% · Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility... Yibo Jiang
Bryon Aragam → authored → Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
confidence 99% · Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility... Bryon Aragam
Siyang Wu → authored → Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
confidence 99% · Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility Siyang Wu
Cross-Contextual Consistency → evaluateson → CommonsenseQA
confidence 95% · Specifically, we use... CommonsenseQA
Cross-Contextual Consistency → evaluateson → SimpleQA
confidence 95% · Specifically, we use... SimpleQA Verified
Cross-Contextual Consistency → evaluateson → FactScore
confidence 95% · Specifically, we use... FActScore
Cross-Contextual Consistency → evaluateson → HumanEval
confidence 95% · Specifically, we use... HumanEval
Cross-Contextual Consistency → evaluateson →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered "saturate".
Tags
Links
- Source: https://arxiv.org/abs/2608.10315v1
- Canonical: https://arxiv.org/abs/2608.10315v1
Trouble viewing inline? Open PDF directly →
Full Text
100,838 characters extracted from source content.
Expand or collapse full text
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility Siyang Wu Data Science Institute University of Chicago Chicago, USA siyangwu@uchicago.edu Yibo Jiang Department of Computer Science University of Chicago Chicago, USA yiboj@uchicago.edu Bryon Aragam Booth School of Business University of Chicago Chicago, USA bryon@chicagobooth.edu Abstract Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered “saturate”. 1 Introduction Large language models (LLMs) achieve strong performance across a wide range of tasks, yet the internal basis of their responses remains poorly understood. In contrast to this strong performance, LLMs often fail to preserve logical consistency: For example, the reversal curse [Berglund et al., 2024] illustrates the failure to correctly associate facts that logically entail one another, while context hijacking [Jiang et al., 2024b] shows that appending semantically neutral context can trick LLMs into changing their answers. These examples show that despite their incredible performance on a variety of challenging tasks, LLMs can still fail at basic logical reasoning. What’s more, these failures are often unpredictable and manifest in surprising and unusual behaviours. This motivates the need for principled approaches to directly assess model credibility and its relationship to performance and accuracy on downstream tasks. One popular approach is to exploit certain internal signals of large language models for evaluation. Token-level probabilities, for example, are commonly used in multiple-choice benchmarks to assess calibration [Kapoor et al., 2024, Ren et al., 2023]. As powerful as these approaches can be in principle, these methods cannot be applied on closed-source models and, more critically, cannot handle free- form questions and open-ended responses. Other widely used methods such as self-report [Lin et al., 2022], self-consistency [Wang et al., 2023], and paraphrasing consistency [Portillo Wightman et al., 2023] are straightforward to implement, but each can suffer from mechanistic failures such as overconfidence [Rathi et al., 2025], trapping on the same incorrect answer [Chen et al., 2023], or changing of the prompt’s meaning [Chataigner et al., 2025], leading to inflated confidence estimates. Preprint. arXiv:2608.10315v1 [cs.CL] 10 Aug 2026 A correlation coefficient can indicate the strength of a linear relationship. Question: In the correlation coefficient r = 0.78, what percentage of variation in y is explained by the linear regression model? A. 22% B. 39% C. 44% D. 61% Question: In the correlation coefficient r = 0.78, what percentage of variation in y is explained by the linear regression model? A. 22% B. 39% C. 44% D. 61% LLMs 푥 1 ,푥 2 ,푥 3 ......푥 푛 푥 1 ′ ,푥 2 ′ ,푥 3 ′ ... ... 푥 푛 ′ C3 = 0.12 MMLU High School Stats Figure 1: Overview of the proposed workflow. From left to right, we sample generations from the original prompt and from semantically neutral perturbed prompts, measure the resulting distributional shift, and compute C3. The right panel shows C3 for 26 models plotted by release month, revealing an increasing trend on the MMLU High School Statistics benchmark: newer models become progres- sively more credible. An alternative approach is to attempt to elicit the confidence of LLM responses indirectly, without probing model parameters or asking for confidence measurements directly. This approach can be likened to placing a model under judicial scrutiny. As in legal proceedings, credibility is established not by a single answer but through cross-examination: deposition, repeated questioning, and extensive fact checking. Accordingly, a model’s response should be examined for consistency under alternative but semantically equivalent formulations of the same question. If a model truly affirms its answer, it should be robust to cross-examination under logically and semantically equivalent contexts. However, if a model’s outputs vary substantially across different framings, the credibility of the model is undermined. In this paper, we study a simple behavioral principle: an answer is more credible when it remains stable across contexts that change the surrounding wording or premise but not the task-relevant meaning. In an LLM, we call this property Cross-Contextual Consistency (C3), illustrated in Figure 1. By sampling a large number of prompt variations that preserve the original semantic content and then measuring the resulting distributional differences in model responses, we translate this behavioral principle into a quantitative metric for evaluating the credibility of a model. Rather than judging the correctness of individual answers, we evaluate the consistency of a model’s responses, which is a much simpler objective that relies less on external knowledge, does not require access to model parameters, and does not rely on self-reported measures. Surprisingly, despite not being explicitly designed to test correctness—indeed, factuality and correctness are never explicitly evaluated by this metric—C3 proves to be a useful indicator of model correctness. This metric, which is measured at the instance level, can be aggregated into a model-level robustness profile across different benchmarks. Evaluation across 16 models and six benchmarks shows that perturbation-based probing not only provides a practical means of quantifying how much a model’s generations shift under controlled, content-neutral prompt perturbations, but also reveals a consistent association between local stability and downstream reliability: Models (and individual task instances) that exhibit smaller distributional shifts under perturbation are more likely to produce correct answers, whereas larger shifts are frequently associated with errors. This is despite the fact that accuracy is never explicitly measured by C3. Building on this observation, the results show that C3 provides a complementary evaluation axis that remains informative even when conventional benchmarks become less discriminative due to saturation and contamination. Our main contributions are: 1.An underutilized behavioral property of LLMs. We identify C3 as an underutilized behavioral property of LLMs’ credibility of generations: when a model’s answer is well- supported, it should remain stable under cross-examination. 2.An operationalization of cross-contextual consistency. We instantiate this idea through C3, a black-box evaluation protocol that compares model output distributions under original and perturbed contexts while adapting across multiple-choice, short-answer, long-form factuality, and code-generation tasks. 3.Empirical evidence for the evaluation utility of cross-contextual consistency. Across six benchmarks and 16 main models, with an additional 10-model case study, we show that 2 C3 aligns with correctness and factuality, remains robust across perturbation sources and comparison estimators, and provides a complementary diagnostic for identifying saturated, brittle, biased, and unlearned benchmark regions. 2 Motivation: A tale from logical entailment When prompted with a question, what does it intuitively mean for an LLM to answer faithfully? One natural interpretation is that the model responds in accordance with the knowledge it possesses. Fortunately, this intuition can be formalized using the language of propositional logic. Let’s write the internal parametric knowledge base of a model asΓ =γ 1 ,...,γ n , consisting ofn atomic binary formulae, from which certain statements produced by the model can be constructed via logical operators (e.g., conjunction, disjunction). For example, we can have knowledge base with γ 1 =“Paris is in France”,γ 2 =“France is in Europe”and statements like “Paris is in an European country” (γ 1 ∧ γ 2 ) or “France is a European country” (γ 2 ). An evaluation mapvassigns True or False values to atomic formulae, i.e.v(γ i ), which induces truth values for statements built from atomic formulae. A valuationv, under which a statementfevaluates to true, is said to satisfy the statement, or to be a model of the statement. LetM (f )be the set of models/worlds off. Intuitively,M (f )is the subset of all worlds under which statementfis True. Due to the stochastic nature of language models, each evaluation is associated with a probability P (v)reflecting its likelihood of being true or false. In other words, rather than modeling an LLM as having a single coherent view, we model it as a mixture of views, each associated with a probability. Given two statementsf,g, one can say thatfentailsg(i.e.,f |= g) ifM (f )⊆ M (g). That is, iff entailsg, then in every world wherefis True,gmust be True. For example, “Paris is in an European country” entails “France is a European country” with the knowledge base that contains “Paris is in France” and “France is in Europe” becauseγ 1 ∧ γ 2 |= γ 2 . Interestingly, the two statements do not imply one another in isolation. The implication only holds once the internal knowledge base is taken into account. When we do not have perfect entailment, the entailment probability can be computed as:P (f |= g) = 1− P v∈M(f) and v̸∈M(g) P (v) This is the complement of the probability mass assigned to worlds in whichfis true andgis false. In the context of LLMs, we letfdenote a prompt andg denote an response. We interpret f |= g as meaning that the prompt f entails the response g. When evaluating LLMs, we seek to determine whether a given prompt-response pair is genuinely implied by the model’s internal world view, or, in stochastic terms, with what probability the model supports that implication. Under this framework, answering this question amounts to computing the aforementioned probability. However, two challenges remain: (1) How to sample different worldsv? (2) How to estimate different probabilities P (v)? Our approach is motivated by considering different contexts as samples from different worlds. Each prompt corresponds to one world. Although the LLM’s output is stochastic, in practice the most probable answer typically dominates. This is because a single prompt may constrain induced behaviors. Therefore, we must perturb the prompts to induce diverse evaluation maps, while ensuring that the perturbed prompts remain within the semantic scope of M (f ). Still, directly estimating these underlying probabilities is ill-defined. Instead, we adopt a different approach. If for every worldvwe havef |= g, then the entailment probability is1. Conversely, if there is substantial disagreement across worlds, the entailment probability is low. We therefore measure entailment by estimating the consistency of the model’s answers across different contexts, which can be interpreted as a notion of credibility. This is how we characterize (stochastic) entailment through cross-contextual consistency. 3 Operationalizing Cross-Contextual Consistency Building on the motivation in Section 2, we operationalize Cross-Contextual Consistency (C3) as a behavioral probe for LLM credibility. The central intuition is simple: if a model’s answer is well-supported, then its answer behavior should remain stable when the same task is placed in a different but answer-neutral context. Conversely, if the model’s answer changes substantially 3 under contextual variation that does not alter the task-relevant meaning, then the answer is more context-fragile. We therefore treat C3 as a cross-contextual comparison signal rather than as a single fixed metric: the goal is to compare model behavior under the original prompt and under controlled perturbed contexts. Controlled contextual perturbations. For each original task queryx ∈ X, we sample a set of contextual perturbationsE =ε 1 ,...,ε n . Each perturbationε∈Eis prefixed to the original query, producing a perturbed queryx ′ = [ε;x]. These perturbations are designed to change the surrounding context while preserving the task itself. In particular, we require perturbations to satisfy three properties: topic alignment, content neutrality, and non-trivial contextual variation. Topic alignment means that the perturbation remains within the broad domain or capability being tested. Content neutrality means that the perturbation does not reveal, support, contradict, or otherwise change the correct answer. Non-triviality means that the perturbation introduces meaningful contextual variation rather than merely restating the original question. We verify these three requirements separately in the appendix. Content neutrality is evaluated in Appendix A.3, where we examine whether the added context avoids introducing answer-relevant information or systematically shifting model performance. Topic alignment is examined in Appendix A.4, where we assess whether perturbations remain within the same broad domain or capability as the original query. Non-trivial contextual variation is verified in Appendix A.2, where we check that accepted perturbations introduce diverse contextual variation rather than near-duplicate restatements. Some examples of perturbed prompts are: • SVAMP: Many tourists visited the ancient castle during the weekend. Rachel learned that 317 visitors came to Buckingham Palace that day. If there were 295 visitors the previous day, how many more visitors visited Buckingham Palace that day than on the previous day? •SimpleQA: An artist might release an EP during any year of their career. What EP did Rosal ́ ıa release in 2019? Definition of C3. We formalize model generation as a stochastic process. Given an inputx, an LLM induces a conditional distributionP (Y | x)over possible outputs. This view follows the standard autoregressive perspective, where tokens are sampled sequentially based on the accumulated context and hidden state dynamics [Vaswani et al., 2017, Holtzman et al., 2020, Geshkovski et al., 2025]. For an original queryx, we sample a set of outputsY = y 1 ,y 2 ,...,y n ∼ P (Y | x). For each contextual perturbationε ∈ E, we construct a perturbed queryx ε = [ε;x]. The perturbation setE induces a perturbed output distribution,P E (Y | x) = 1 |E| P ε∈E P (Y | x ε )from which we sample perturbed outputsY E =y ′ 1 ,y ′ 2 ,...,y ′ m ∼ P E (Y | x). Cross-Contextual Consistency (C3) measures how stable the model’s answer behavior remains be- tween the original and perturbed conditions. Formally, we define C3 as a normalized inverse distance between the original and perturbed output distributions:C3(x;E ) = 1− e D P (Y | x),P E (Y | x) , where e D(·,·) is a task-adaptive distance or disagreement function normalized to [0, 1]. A higher C3 score indicates that the model’s output distribution changes less under topic-aligned, content-neutral contextual variation, while a lower C3 score indicates greater context-fragility. This definition makes C3 a general cross-contextual comparison framework rather than a metric tied to a single distance. Distance metric: MMD. In our main implementation, we instantiate e Dusing Maximum Mean Discrepancy (MMD) [Gretton et al., 2012]. MMD provides a flexible non-parametric estimator for comparing empirical answer distributions, making it suitable for both fixed-format and open-ended generations. Given sampled outputs from the original conditionYand the perturbed conditionY E , we compute an empirical MMD distance using task-adaptive feature maps and kernel choices. We then normalize the resulting distance into a consistency score in[0, 1], where larger values indicate smaller cross-contextual shift. Details on the empirical MMD estimator, task-adaptive feature maps, kernel choices, and normalization are provided in Appendix H. Importantly, MMD provides one instantiation of C3, not the definition of C3 itself. C3 is defined by the comparison between model behavior under original and perturbed contexts. In Appendix G, we replace MMD with a simpler cross-comparison distance and show that the same cross-contextual signal is largely preserved, supporting the view that C3 is driven by the original-versus-perturbed comparison rather than by a particular choice of distance. 4 4 Experiments In this section, we describe our experimental setup, including data, methods, LLM models, and benchmarks. Data. We evaluate C3 on six widely used benchmarks spanning arithmetic reasoning, multiple-choice reasoning, commonsense inference, short-form factual QA, long-form factuality, and code genera- tion. Specifically, we use SVAMP [Patel et al., 2021], MMLU High School Statistics [Hendrycks et al., 2021b,a], CommonsenseQA [Talmor et al., 2019], SimpleQA Verified [Haas et al., 2025], FActScore [Min et al., 2023], and HumanEval [Chen et al., 2021]. Together, these benchmarks cover both fixed-format and open-ended generations, allowing us to evaluate C3 across diverse task formats and answer types. Detailed benchmark descriptions and prompts are provided in Appendix I. Baselines. Since C3 is intended to serve as a proxy for the credibility of model generations, we compare it with related notions of confidence, consistency, and factuality. We include both vanilla black-box confidence estimators and a non-vanilla factuality checking tool. The vanilla baselines include self-reported confidence [Lin et al., 2022], where the model outputs an explicit confidence score together with its answer; self-consistency [Wang et al., 2023], which estimates confidence from agreement among repeated stochastic generations under the same prompt; and paraphrasing consistency [Portillo Wightman et al., 2023], which measures whether answers remain stable under meaning-preserving prompt paraphrases. As a non-vanilla baseline, we compare against FActScore [Min et al., 2023], which evaluates factual support for long-form generations using external evidence. For sampling-based baselines, agreement is computed with a generalized consistency score, using exact-match indicators for fixed-format tasks and semantic similarity for open-ended generations. Full implementation details are provided in Appendix J. Models. We evaluate C3 on 16 widely used LLMs spanning multiple families and scales. We also conduct case studies on 10 additional models; however, due to their characteristics, such as heavier reasoning processes or deprecated designs, they are often costly in time and computational resources. We therefore restrict these case studies to the MMLU High School Statistics benchmark. The full list of models is provided in Appendix E.1. Evaluation metrics. We evaluate C3 and all baseline scores after normalizing each score to the range [0, 1]. To measure calibration, we report the Expected Calibration Error (ECE), where lower values indicate better calibration. For ranking-based evaluation, we report AUROC, as well as the area under the precision–recall curve for detecting correct outputs (AUPRC-P) and detecting incorrect outputs (AUPRC-N), with the latter computed using1− s i . Details and implementation of these metrics are provided in Appendix E.2. C3 elicitation. For both perturbed and unperturbed sampling, we collected 30 trials per instance across 16 standard models to assess the alignment of C3 against other baselines. The number 30 is supported by an empirical study shown in Appendix B. In this study we use GPT-4.1 for noise sampling, however, an ablation study in Appendix F shows that perturbations from much smaller models are nearly as effective as those from larger ones, but with much lower computational overhead. The noise was prefixed to each prompt, separated by a single space. To determine answer equivalence, we use an LLM through the prompt detailed in Appendix D.5. Given the high volume of evaluations required (tens of millions given the scale of our experiments), we performed offline inference using a Qwen3-8B [Team, 2025] model temperature set to 0, supported by the VLLM framework [Kwon et al., 2023] on two NVIDIA H200 GPUs. 5 Results We find that C3 provides substantially better signals than the baseline methods, and aligns especially well on challenging benchmarks such as SimpleQA, where other approaches often produce inflated assessments. 5.1 C3 is calibrated with truthfulness Correctness.The results in Table 1 demonstrate that C3 consistently outperforms existing baselines in aligning model “credibility” with truthfulness. In math tasks like SVAMP, C3 achieves an AUROC of 0.917, a noticeable jump over Self-Consistency, with AUROC of 0.861. This suggests that 5 CommonsenseQASVAMP Method ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ Self-consistency0.2240.6430.7560.4610.1420.8610.8930.740 Paraphrasing0.2710.6340.7640.4840.1570.8410.8730.720 Self-report0.1910.5420.7460.3320.2400.5320.7630.277 C3 (Ours)0.1890.7220.8050.5410.0720.9170.9360.818 MMLU High School StatisticsFActScore Method ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ Self-consistency0.3820.5770.5110.6050.2250.8580.9090.731 Paraphrasing0.4060.5470.4890.5910.2490.8280.8870.716 Self-report0.3780.4960.5940.4110.3300.4960.7060.335 C3 (Ours)0.3380.5970.5300.6210.1600.8730.9160.701 HumanEvalSimpleQA Method ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ Self-consistency0.1310.8120.8960.4870.3930.7920.3680.924 Paraphrasing0.1650.7780.8620.4530.4110.7730.3490.906 Self-report0.2870.5280.8070.2110.7780.4900.1510.846 C3 (Ours) 0.1100.8470.9430.5200.1660.8230.3890.944 Table 1: Comparison of C3 with baselines across six benchmarks average over 16 models we tested. Shaded cells denote the best performance for each metric among the four methods. CommonsenseQA evaluates commonsense world knowledge; SVAMP evaluates mathematical reasoning; MMLU High School Statistics evaluates statistical knowledge; FactScore and SimpleQA evaluate factuality in long-form and short-form generation, respectively; and HumanEval evaluates code generation. Lower is better for ECE, while higher is better for AUROC, AUPRC-P, and AUPRC-N. The noise for perturbation in the table is sampled by GPT-4.1, and we also shown that C3 does not rely on advanced model and the results can be still reproducible by smaller models with very few costs as we shown in AppendixF. measuring the credibility of LLMs with perturbations is more effective for capturing the logical coherence of a reasoning chain than simple sampling strategies using identical prompts. Factuality. Furthermore, the consistent gains on FActScore, CommonsenseQA, and SimpleQA suggest that C3 is effective for detecting factuality-related errors. In these settings, traditional consistency metrics can be poorly calibrated because models may remain highly consistent even when they are confidently relying on incorrect parametric memories or strong priors. By measuring the distributional shift in generations induced by content-neutral noise, C3 increases contrast between well-known facts, small shift, and poorly known facts, large shift, yielding a more informative signal of the model’s knowledge state. Long form generation. Across the six benchmarks tested, HumanEval and FActScore are considered as “long-form generation” because they do not assume a pre-defined output format (such as multiple choice or keywords). C3 achieves nearly the best scores across four metrics in these two forms of generation: 0.110 ECE and 0.847 AUROC on HumanEval, and 0.160 ECE and 0.873 AUROC on FActScore. This is a scenario where many existing works fail. Our method scales cleanly to long-form generation, keeping nearly as simple as fixed-format tasks, which even allows us to evaluate code generation. While recent efforts like Sharma and David [2025] assess uncertainty in code via symbolic execution, our method achieves superior calibration without requiring external execution environments. Why they fail? Self-reported measures exhibit miscalibration: While they occasionally achieve seemingly reasonable ECE (e.g., 0.191 in CommonsenseQA), their discriminative power, measured by AUROC and AUPRC, remains near-random. This confirms that LLMs struggle to introspectively verbalize uncertainty, often overconfidently yielding near-100% score for wrong answers [Mielke et al., 2022, Rathi et al., 2025]. We also find that paraphrasing suffers from fundamental mechanistic flaws; since the paraphrasing is often performed by an LLM assistant, it can introduce semantic drift. For instance, high word overlap can mask cases where swapping arguments changes the underlying meaning [Zhang et al., 2019], or the assistant may unintentionally alter the core intent 6 of the prompt [Chataigner et al., 2025]. Methods like self-consistency paraphrasing do not fail entirely aligning with truthfulness but are often trapped in the “confidently wrong” loop of LLM generation when the model sticks with an incorrect answer. As reflected in SimpleQA, which is a hard benchmark for many LLMs, self-consistency fails and is mis-calibrated with actual correctness on ECE, and paraphrasing makes results even worse. Measuring the distributional shift with C3, we successfully nudge the model to sample different answers when the knowledge is not grounded in LLMs’ knowledge base, providing a much better signal in ECE and AUROC (0.166 and 0.823, respectively) in SimpleQA. Figure 2: Scaling trends of Cross-Contextual Con- sistency (C3) calibration across benchmarks. Col- ors distinguish different benchmarks, while line styles represent metric types: solid lines denote AUROC (higher is better) and dashed lines de- note ECE (lower is better). The results show that as model scale increases, the C3 becomes signifi- cantly better calibrated to correctness, evidenced by rising AUROC and declining ECE. C3 becomes more informative as model capa- bility increases. As shown in Figure 2, larger models within the Llama and Mistral families generally show stronger alignment between C3 and downstream correctness. This suggests that cross-contextual consistency is most useful once a model has enough capability for stable an- swer behavior to emerge; at that point, resid- ual instability more clearly marks fragile an- swers rather than broad capability failure. How- ever, scale does not eliminate cross-contextual fragility: even the strongest models remain im- perfectly stable under answer-neutral contextual variation. 5.2 A closer inspection on facutality CommonsenseQA, FActScore, and SimpleQA all test a model’s world knowledge and factual- ity. C3 shows a close alignment with the quality of generation across all three. While FActScore probes externally, scoring by decomposing long- form generation into atomic facts and verifying them against external sources, C3 probes inter- nally. It uses variations of noise to perturb the LLM to see if it “insists” on an answer; the resulting distributional shift provides a significant signal regarding the quality of generation. As shown in Figure 3, when aggregating instance-level C3 and correctness, we observe a 0.624 Spearman rank correlation on FActScore and an 0.831 AUROC. We also include a comparison with SimpleQA, a benchmark where most models fail, making it a strong indicator for detecting overconfident metrics; C3 shows a 0.642 Spearman correlation and an 0.821 AUROC. Even without probing external information, C3 successfully aligns with FActScore. This suggests that the model’s internal state contains a latent representation of its own knowledge boundaries: when a model “knows” a fact, its output distribution is resilient to input noise, whereas hallucinated facts reside in low-probability regions that collapse or shift significantly under even minor perturbations. 5.3 C3 for benchmark diagnosis By jointly inspecting instance-level C3 and instance-level benchmark performance, we can show cross-model typical behavior for each benchmark instance after averaging across the same 16 models. We find that C3 captures complementary information about model behavior beyond aggregate performance alone, as shown in Figure 4. Why do we need C3 as an additional axis? Relying exclusively on performance scores often mask the underlying mechanism of a model’s knowledge. A key phenomenon in current LLM evaluation is that benchmarks may suffer from data contamination or overfitting, where models memorize specific prompt-answer pairs without acquiring the underlying reasoning. C3 provide additional information: “Brittle” (Yellow) region where models achieve high accuracy but fail to maintain consistency under perturbation. While high performance typically suggests capability, the low C3 in this region supports the alternative hypothesis: Success on these instances is driven by surface-level pattern matching 7 rather than robust semantic understanding. This divergence serves as a proxy for detecting potential benchmark leakage to training processes of current LLMs. Figure 3: A detailed comparison of C3 alignment with factuality on FActScore and SimpleQA, show- ing Spearman rank correlation (left) and AUROC (right). C3 exhibits moderate to strong rank corre- lation with generation factuality. Minor AUROC differences from Table 1 are due to different aggre- gation strategies. C3 also helps differentiating mastered from sys- tematic bias. The C3 axis further clarifies the sta- tus of the benchmark by distinguishing between “solved” and “biased” generation. Instances in the “Mastered” (Green) region represent tasks where models have converged on a stable solu- tion. A high density of instances in this region supports the hypothesis of benchmark satura- tion, indicating that these specific questions no longer possess the discriminative power to dis- tinguish between the capabilities of different models. Conversely, the “Biased” region high- lights instances where models are not merely guessing, but are consistently trapped on incor- rect answers. This supports the hypothesis that these benchmark instances trigger strong, incor- rect priors or common misconceptions shared across models that possibly arise from model training processes. Benchmark by Benchmark Comparison. Fig- ure 4 visualizes instance-level C3 against per- formance, revealing distinct patterns that char- acterize the status of each benchmark. Com- monsenseQA and HumanEval both exhibit the signature of saturation, where a significant pro- portion of instances in the “Mastered” region suggests these tasks are well solved by modern models. However, in contrast to CommonsenseQA, HumanEval displays a heavy “tail” extending into the “Brittle” quadrant; this pattern implies that its high performance may be partially inflated by overfitting, where models succeed via surface pattern matching but fail under perturbation, lacking the true reasoning process of coding [Riddell et al., 2024]. SVAMP and FActScore show a balanced proportion of unlearned and mastered instances, while MMLU High School Stats shows a high number of mastered but also a significant cluster of biased instances; this suggests that while basics are understood, specific statistical concepts trigger consistent, systematic misconceptions. Finally, SimpleQA represents the true “hard” benchmark, dominated by the “Unlearned” region. The scarcity of mastered instances and the prevalence of stable poor performance indicate that this benchmark is not saturated, but rather captures specific knowledge gaps that remain out of reach for the current generation of models. 6 Related work A variety of methods to estimate model confidence have been proposed in the literature. White-box. Methods rely on logits, hidden states, or parameters, which are infeasible for closed models; moreover, OpenAI reported degraded calibration after post-training, underscoring instability in probability-based confidence [OpenAI et al., 2024b, Xie et al., 2024, Shen et al., 2024]. Self-verbalized (black-box). Such confidence can be miscalibrated and drift with prompting and post-training, and often shows over-confidence, even though some elicitation schemes help in specific settings [Lin et al., 2022, Xiong et al., 2024, Kumar et al., 2024, Zhang et al., 2024b, Heo et al., 2025, OpenAI et al., 2024b]. Agreement-based (black-box). Operating without input perturbation, plurality voting can conflate repetition bias with genuine certainty and underuse informative minority signals [Huang et al., 2024, Wang et al., 2023]. Semantic paraphrase (black-box). Paraphrase-based uncertainty presumes meaning preservation, yet small wording changes often shift semantics and behavior; entropy over paraphrases can therefore reflect semantic drift rather than true confidence [Zhang et al., 2019, Wahle et al., 2024, Melamed et al., 2024, Mizrahi et al., 2024]. Meanwhile paraphrasing with LLMs leads to biased collection of model performance, [Lunardi et al., 2025] shows that the absolute accuracy scores drop significantly when when paraphrased the quesiton. Answer-calibration for long-form. 8 Treating confidence as probability of the correct answer” is ill-posed for multi-sentence, open-ended generations; automatic checks correlate unevenly with human judgment and preferences [Xu et al., 2023, Fabbri et al., 2021, Chen et al., 2024]. Other families (conformal, refusal). These typically require labels, specialized scoring access, or finetuning—constraints that hinder post-hoc use in closed APIs [Quach et al., 2024, Mohri and Hashimoto, 2024, Zhang et al., 2024a]. Figure 4: For each benchmark instance, we com- pute the mean performance and mean C3 across the 16 models, then partition instances into four regions using the median performance and me- dian C3 computed over instances across the six benchmarks. Green (“mastered”) indicates high performance with contextually consistent answers; yellow (“brittle”) indicates high performance but sensitive to perturbations; red (“biased”) indicates low performance yet perturbation-invariant (consis- tently wrong) answers; gray indicates low perfor- mance with inconsistency, suggesting knowledge that is not reliably learned. Our approach We measure credibility via the distributional shift in generation induced by semantically equivalent, meaning-preserving prompt perturbations, using this perturbation gap as a reference-free proxy for confidence [Kuhn et al., 2023, Mizrahi et al., 2024]. This lens differs from existing prompt-sensitivity and robustness notions in several important ways. Some prior work quantifies sensitivity in probability space by tracking changes in log- likelihoods or likelihood ratios across prompt variants, which typically requires access to in- ternal scoring signals and reflects probability drift rather than semantic drift of the generated content [Chatterjee et al., 2024]. Others study robustness through format-induced variability by measuring sensitivity to spurious formatting features in prompt design, diagnosing evaluation volatility but not yielding an instance-level cred- ibility signal for open-ended generations [Sclar et al., 2024]. Related measures are also often de- fined for classification by analyzing instability of predicted label distributions under rephras- ing, which does not transfer cleanly to free-form text where outputs must be compared semanti- cally rather than as discrete labels [Errica et al., 2025]. Another line evaluates the reliability of knowledge probing methods by testing whether accept/reject decisions remain consistent under perturbations, emphasizing probe stability rather than the stability of the generated answer itself [Zhao et al., 2025]. In contrast, C3 directly mea- sures invariance to meaning-preserving perturba- tions, is black-box and format-adaptive, and pro- vides a comparable credibility diagnostic across heterogeneous tasks and open-ended generation formats. 7 Conclusion This work identifies cross-contextual consistency as a behavioral signal for evaluating LLM credibility: when a model’s answer is well-supported, it should remain stable under topic-aligned, content-neutral contextual variation. We operationalize this idea through C3, a black-box framework that compares generation distributions under original and perturbed contexts. Across 26 models and six benchmarks, C3 consistently aligns with correctness and factuality across reasoning, factual recall, long-form generation, and code generation. Beyond instance-level evaluation, C3 also provides a complemen- tary diagnostic for benchmark analysis, helping distinguish stable understanding from brittle or systematically biased performance. These results suggest that controlled contextual perturbation offers a practical way to reveal answer fragility that standard accuracy-based evaluation can miss. 9 8 Limitations This work studies C3 across a broad but necessarily finite set of models, benchmarks, and experimental settings. Future work may extend the evaluation to additional tasks, languages, and application domains. Further exploration of alternative implementation choices may also help better understand how C3 can be adapted across different evaluation scenarios. 9 Societal Impact This work aims to improve the evaluation of LLM reliability by providing a black-box signal for identifying context-fragile generations. Such tools may help practitioners better detect uncertain, brittle, or potentially hallucinated outputs before deploying LLMs in higher-stakes settings. At the same time, C3 should not be treated as a guarantee of truthfulness or safety; a model can be consistent and still wrong. In accordance with the NeurIPS Code of Ethics, we note that this method is intended as a diagnostic aid rather than a replacement for human oversight, domain expertise, or task-specific safety evaluation. References Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on “a is b” fail to learn “b is a”. In The Twelfth International Conference on Learning Representations, 2024. URLhttps: //openreview.net/forum?id=GPKTIktA0k. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URLhttps: //arxiv.org/abs/2005.14165. Cl ́ ea Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Ta ̈ ık, Elliot Creager, and Gol- noosh Farnadi. Say it another way: Auditing llms with a user-grounded automated paraphrasing framework, 2025. URL https://arxiv.org/abs/2505.03563. Anwoy Chatterjee, H S V N S Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. POSIX: A prompt sensitivity index for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14550–14565, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.852. URLhttps://aclanthology.org/ 2024.findings-emnlp.852/. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang.Humans or LLMs as the judge? a study on judgement bias. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.474. URL https://aclanthology.org/2024.emnlp-main.474/. Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha. Adap- tation with self-evaluation to improve selective prediction in LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5190–5213, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023.findings-emnlp.345. URLhttps://aclanthology.org/2023. findings-emnlp.345/. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond ́ e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, 10 Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URLhttps://arxiv. org/abs/2107.03374. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URLhttps:// arxiv.org/abs/2507.06261. Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data. Federico Errica, Davide Sanvito, Giuseppe Siracusano, and Roberto Bifulco. What did I do wrong? quantifying LLMs’ sensitivity and consistency to prompt engineering. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1543–1558, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.73. URL https://aclanthology.org/2025.naacl-long.73/. Alexander R. Fabbri, Wojciech Kry ́ sci ́ nski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021. doi: 10.1162/tacla00373. URL https://aclanthology.org/2021.tacl-1.24/. Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspec- tive on transformers, 2025. URL https://arxiv.org/abs/2312.10794. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Sch ̈ olkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012. URL http://jmlr.org/papers/v13/gretton12a.html. Lukas Haas, Gal Yona, Giovanni D’Antonio, Sasha Goldshtein, and Dipanjan Das. Simpleqa verified: A reliable factuality benchmark to measure parametric knowledge, 2025. URLhttps: //arxiv.org/abs/2509.07968. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b. Juyeon Heo, Miao Xiong, Christina Heinze-Deml, and Jaya Narain. Do LLMs estimate uncer- tainty well in instruction-following? In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=IHp3vOVQO2. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URLhttps: //openreview.net/forum?id=rygGQyrFvH. 11 Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, and Zhouhan Lin. Mirror- consistency: Harnessing inconsistency in majority voting. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2408–2420, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.135. URLhttps://aclanthology.org/ 2024.findings-emnlp.135/. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L ́ elio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Th ́ eophile Gervet, Thibaut Lavril, Thomas Wang, Timoth ́ e Lacroix, and William El Sayed. Mixtral of experts, 2024a. URL https://arxiv.org/abs/2401.04088. Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Do llms dream of elephants (when told not to)? latent concept association and associative memory in transformers. Advances in Neural Information Processing Systems, 37:67712–67757, 2024b. Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine M. Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don’t know. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id= QzvWyggrYB. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=VD-AYtP0dve. Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, and Ali Emami. Confidence under the hood: An investigation into the confidence-probability alignment in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 315–334, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.20. URL https://aclanthology.org/2024.acl-long.20/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URLhttps: //openreview.net/forum?id=8s8K2UZGTZ. Riccardo Lunardi, Vincenzo Della Mea, Stefano Mizzaro, and Kevin Roitero. On robustness and reliability of benchmark-based evaluation of llms, 2025. URLhttps://arxiv.org/abs/2509. 04013. Rimon Melamed, Lucas Hurley McCabe, Tanay Wakhare, Yejin Kim, H. Howie Huang, and Enric Boix-Adser ` a. Prompts have evil twins. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 46–74, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.4. URLhttps://aclanthology.org/2024. emnlp-main.4/. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872, 2022. doi: 10.1162/tacla00494. URLhttps://aclanthology.org/ 2022.tacl-1.50/. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual 12 precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.741. URLhttps://aclanthology.org/2023.emnlp-main. 741/. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949, 2024. doi: 10.1162/tacla00681. URLhttps: //aclanthology.org/2024.tacl-1.52/. Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 36029–36047. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/mohri24a.html. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, et al. Gpt-4o system card, 2024a. URL https://arxiv.org/abs/2410.21276. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report, 2024b. URL https://arxiv.org/abs/2303.08774. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main. 168. URL https://aclanthology.org/2021.naacl-main.168. Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. Strength in numbers: Estimating confidence of large language models by prompt agreement. In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta, editors, Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 326–362, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.trustnlp-1.28. URLhttps: //aclanthology.org/2023.trustnlp-1.28/. Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=pzUhfQ74c5. Neil Rathi, Dan Jurafsky, and Kaitlyn Zhou. Humans overrely on overconfident language models, across languages, 2025. URL https://arxiv.org/abs/2507.06306. Jie Ren, Yao Zhao, Tu Vu, Peter J. Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation in large language models. In Javier Antor ́ an, Arno Blaas, Kelly Buchanan, Fan Feng, Vincent Fortuin, Sahra Ghalebikesabi, Andreas Kriegler, Ian Mason, David Rohde, Francisco J. R. Ruiz, Tobias Uelwer, Yubin Xie, and Rui Yang, editors, Proceedings on ”I Can’t Believe It’s Not Better: Failure Modes in the Age of Foundation Models” at NeurIPS 2023 Workshops, volume 239 of Proceedings of Machine Learning Research, pages 49–64. PMLR, 16 Dec 2023. URL https://proceedings.mlr.press/v239/ren23a.html. Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluating code generation capabilities of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14116–14137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.761. URLhttps://aclanthology. org/2024.acl-long.761/. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024. URL https://arxiv.org/abs/2310.11324. 13 Arindam Sharma and Cristina David. Assessing correctness in llm-based code generation via uncertainty estimation, 2025. URL https://arxiv.org/abs/2502.11620. Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory W. Wornell, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44687–44711. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/shen24c.html. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, et al. Openai gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297, 2020. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URLhttps: //aclanthology.org/N19-1421. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, et al. Gemini: A family of highly capable multimodal models, 2025a. URLhttps://arxiv.org/abs/2312. 11805. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, et al. Gemma 3 technical report, 2025b. URLhttps: //arxiv.org/abs/2503.19786. Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.Attention is all you need.In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URLhttps://proceedings.neurips.c/paper_files/paper/2017/file/ 3f5e243547dee91fbd053c1c4a845a-Paper.pdf. Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. Paraphrase types elicit prompt engineering capabilities. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11004–11033, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2024.emnlp-main.617. URL https://aclanthology.org/2024.emnlp-main.617/. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URLhttps://openreview.net/forum?id= 1PL1NIMMrw. Johnathan Xie, Annie S Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating lan- guage models with adaptive temperature scaling. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pages 18128–18138, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics.doi: 10.18653/v1/2024.emnlp-main.1007.URL https://aclanthology.org/2024.emnlp-main.1007/. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/ forum?id=gjeQKFxFpZ. 14 Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.181. URLhttps://aclanthology.org/2023. acl-long.181/. Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say ‘I don’t know’. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7113–7139, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.394. URLhttps: //aclanthology.org/2024.naacl-long.394/. Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu. Calibrating the confidence of large language models by eliciting fidelity. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2959–2979, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main. 173. URL https://aclanthology.org/2024.emnlp-main.173/. Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1131. URLhttps: //aclanthology.org/N19-1131/. Raoyuan Zhao, Abdullatif K ̈ oksal, Ali Modarressi, Michael A. Hedderich, and Hinrich Schuetze. Do we know what LLMs don’t know? a study of consistency in knowledge probing. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23254–23280, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335- 7. doi: 10.18653/v1/2025.findings-emnlp.1263. URLhttps://aclanthology.org/2025. findings-emnlp.1263/. 15 A Discussion on Perturbation Noise source A.1 LLMs as Sampler of Noise We utilize LLMs as samplers to generate semantic perturbations. The noise generation process adheres to three critical principles to ensure the resulting samples are both challenging and informative: •Topic Alignment: Samples are generated to remain semantically aligned with the original input query to ensure the underlying task remains constant. •Content Neutrality: The samples are designed to be content-neutral, as non-neutral content could introduce systematic bias or shifts in the model’s performance. •Non-Trivial Contextual Variations: The generated noise constitutes non-trivial contextual variations of the input, testing the model’s robustness while maintaining the core meaning. we employ the prompts detailed in Appendix D.1. By utilizing LLMs as samplers constrained by these principles, we collect a set of perturbationsE =ε 1 ,...,ε n . To ensure the diversity ofE, we filter out candidate noise that is semantically redundant with existing entries (details are provided in Appendix A.2). A.2 Diversity of Noise Sampling We utilize GPT-4.1 to perform noise sampling, aiming to maximize the diversity of noise per question. It is well known that repeated generations from LLMs using the same prompt often produce semantically similar or even identical outputs. To ensure variety, we maintain a clean pool of samples by comparing each newly generated data pointε ∗ against an existing setE = ε 1 ,ε 2 ,...,ε n in embedding space. A new sampleε ∗ is retained only if its pairwise semantic similarity with the existing elements inN remains below a threshold k. We compute embeddings using all-mpnet-base-v2 [Song et al., 2020] and measure similarity via cosine similarity. However, the choice of similarity thresholdkis typically heuristic and task-specific. In this work, we propose to empirically determine an appropriatekspecific to GPT-4.1’s generation behavior, enabling us to maintain a semantically diverse pool. To do this, we sampled 30 premises generated by GPT-4.1 and used the same model to produce paraphrased versions using the prompt provided in Appendix D.2. For each original premise, we generated 50 paraphrases and embedded them using all-mpnet-base-v2. Within each group of 50 rewrites, we computed all pairwise cosine similarities, resulting in 50 2 similarity scores per group. Aggregating these across all groups yields the overall distribution of cosine similarities, representing GPT-4.1’s implicit notion of semantic equivalence. The histogram of these results is shown in Figure 5. The threshold is determined at a significance level ofα = 0.05. This ensures that by retaining only those samples with a similarity belowk = 0.8124, we have a statistical confidence that at most 5% of the accepted samples are semantically redundant paraphrases. 16 Figure 5: The distribution of pairwise cosine similarities among GPT-4.1-generated paraphrases. The red line indicates the 5th percentile (k = 0.8124), which serves as our empirical threshold for filtering semantic redundancy. 17 A.3 Semantic Neutrality of Perturbations Figure 6 evaluates whether our sampled perturbations introduce systematic answer-relevant bias. We conduct this check on MMLU High School Statistics across 26 models. For each model-question pair, we sample 30 generations from the original prompt and 30 generations from the perturbed prompt. We then compute: ∆ = #Correct perturbed − #Correct original , where positive values indicate that the perturbation improves performance and negative values indicate that it hurts performance. If the added context revealed information about the answer, contradicted the question, or otherwise biased the model toward or away from the correct option, we would expect the distribution of∆to shift systematically above or below zero. In contrast, Figure 6 shows that the distribution is centered near zero across all 26 models, with no consistent positive or negative shift. This suggests that the perturbations do not systematically help or mislead the models on this benchmark. This analysis does not prove that every individual perturbation is perfectly neutral, but it provides empirical evidence that the perturbation procedure does not introduce a systematic directional bias in model performance. We therefore treat these perturbations as approximately content-neutral for the purposes of measuring cross-contextual consistency. 18 Figure 6: The difference of performance of each model before and after the perturbation noises are added. The red line denote the 0 meaning no difference in the performances. 19 A.4 Topic Alignment We also evaluate whether the sampled perturbations remain topic-aligned with the benchmark from which they are generated. Topic alignment means that the perturbation should stay within the broad domain or capability being tested, without revealing, contradicting, or otherwise modifying the answer. This requirement separates our perturbations from arbitrary distractor text: the added context should create meaningful contextual variation, but should still be relevant to the type of task being evaluated. To assess topic alignment, we conduct an automatic topic-classification check. For each benchmark, we used whole perturbations generated by our pipeline with GPT-4.1 and remove the corresponding original question and answer. We then ask an independent Qwen3-8B judge to classify each pertur- bation into one of the broad benchmark-level domains: arithmetic reasoning, statistical reasoning, commonsense reasoning, short-form factual recall, long-form factual or biographical generation, and code generation. The judge observes only the perturbation itself, not the original query, answer, or benchmark label. A perturbation is counted as topic-aligned if the judge assigns it to the same broad domain as the benchmark from which it was sampled. Across the perturbations, approximately 96% are classified into their intended benchmark domain. This suggests that the perturbations are not arbitrary out-of-domain distractors but instead remain aligned with the capability being evaluated. Together with the neutrality analysis in Appendix A.3, this supports the use of our sampled perturbations as topic-aligned, answer-neutral contextual variations for estimating cross-contextual consistency. 20 B Empirical Choice of number of trials To choose an appropriate number of trials, we conducted a case study to determine how many samples per question are needed to reliably characterize model behavior before and after perturbation. As shown in Figure 7, the estimated behavior stabilizes after about 20 samples. We therefore use 30 trials per question as a more conservative choice in this setting. Figure 7: The performance of models across number of trials of samples we collected when perturba- tion is presented. 21 C C3 Evaluation on Model-Level Table 2: MMLU High School Stats Model Performance Metrics MetricECEAUROCPR-PPR-N gemini-2.5-flash self-consistency0.3590.5060.5270.495 Paraphrasing0.3940.4560.4520.472 C30.3270.5490.5690.501 Self-report0.2710.4920.7410.237 gemini-2.5-flash-lite self-consistency0.7950.5030.1660.835 Paraphrasing0.8080.4500.1500.789 C30.8180.5230.1720.848 Self-report0.3740.5190.5790.442 gemma-3-12b-it self-consistency0.3040.6460.6500.568 Paraphrasing0.3100.6090.6770.590 C30.2410.6790.6890.592 Self-report0.3950.5110.5890.446 gemma-3-27b-it self-consistency0.3260.6470.6630.543 Paraphrasing0.3290.6080.6600.555 C30.2600.7060.7210.613 Self-report0.4230.4580.5490.420 gemma-3-4b-it self-consistency0.6020.6280.3350.798 Paraphrasing0.6660.5960.3480.785 C30.5040.5950.3240.776 Self-report0.5300.4960.3070.716 gpt-4.1 self-consistency0.2020.7390.8390.503 Paraphrasing0.2260.6980.8270.482 C30.1800.7530.8560.528 Self-report0.2730.4990.7400.269 gpt-4.1-mini self-consistency0.4160.5360.4930.561 Paraphrasing0.4180.5060.4940.506 C30.4040.5210.5020.546 Self-report0.2330.5920.8790.232 gpt-4.1-nano self-consistency0.3740.4870.4940.490 Paraphrasing0.3570.5160.5040.455 C30.3580.4930.5100.502 Self-report0.3670.4800.6440.332 MetricECEAUROCPR-PPR-N llama-3.1-405b-instruct self-consistency0.1220.7840.8490.618 Paraphrasing0.1640.7570.8280.633 C30.0770.7930.8580.639 Self-report0.3000.5080.7430.274 llama-3.1-70b-instruct self-consistency0.2110.5980.6570.500 Paraphrasing0.2090.5360.6170.523 C30.2090.5490.6350.456 Self-report0.2760.5200.7470.271 llama-3.1-8b-instruct self-consistency0.3660.4900.3080.686 Paraphrasing0.3950.4800.2600.625 C30.2850.4940.3080.700 Self-report0.4890.4910.3730.632 llama-3.2-1b-instruct self-consistency0.4920.5380.2200.832 Paraphrasing0.5400.5170.1800.773 C30.3600.5690.2150.853 Self-report0.5760.5170.1630.862 llama-3.2-3b-instruct self-consistency0.4700.4770.1920.814 Paraphrasing0.4760.4800.1980.759 C30.3580.5440.2280.846 Self-report0.5400.4770.2860.696 mistral-large-2411 self-consistency0.3710.4840.5480.433 Paraphrasing0.4320.4240.5070.423 C30.3800.5210.5730.463 Self-report0.3610.4030.6850.227 mistral-medium-3 self-consistency0.2860.6290.7310.445 Paraphrasing0.3170.6000.6540.493 C30.2200.7510.8230.547 Self-report0.2740.4990.7720.242 mistral-small-24b self-consistency0.4200.5420.5050.557 Paraphrasing0.4640.5210.4720.586 C30.4220.5150.4910.526 Self-report0.3700.4780.7110.280 22 Table 3: CommonsenseQA Model Performance Metrics MetricECEAUROCPR-PPR-N gemini-2.5-flash self-consistency0.1460.7370.8890.394 Paraphrasing0.1820.7030.8840.401 C30.1500.7410.9030.381 Self-report0.1890.5110.7690.261 gemini-2.5-flash-lite self-consistency0.7850.5520.1570.868 Paraphrasing0.8390.5520.1580.880 C30.8080.5840.1680.892 Self-report0.1780.5820.7890.373 gemma-3-12b-it self-consistency0.2280.5710.7550.358 Paraphrasing0.2610.5690.7640.396 C30.1660.7050.8320.512 Self-report0.2190.4930.7230.309 gemma-3-27b-it self-consistency0.2060.5970.7800.385 Paraphrasing0.2440.5470.7470.399 C30.1540.7310.8540.539 Self-report0.2080.5750.7750.321 gemma-3-4b-it self-consistency0.3340.5580.6540.444 Paraphrasing0.3690.5090.6460.440 C30.2250.6880.7450.572 Self-report0.3380.4760.5880.409 gpt-4.1 self-consistency0.1460.6500.8520.419 Paraphrasing0.1810.6370.8750.440 C30.1390.7210.8860.408 Self-report0.1100.5610.8410.226 gpt-4.1-mini self-consistency0.1780.5830.8150.269 Paraphrasing0.2110.5660.8100.295 C30.1560.6910.8630.462 Self-report0.1440.5710.8290.251 gpt-4.1-nano self-consistency0.2280.6620.7960.401 Paraphrasing0.2620.7040.8520.453 C30.1970.6900.8160.475 Self-report0.1580.5510.7280.412 MetricECEAUROCPR-PPR-N llama-3.1-405b-instruct self-consistency0.1420.7520.8690.531 Paraphrasing0.1990.7620.8930.571 C30.1170.7910.8940.576 Self-report0.1600.5780.7660.368 llama-3.1-70b-instruct self-consistency0.1320.7800.8930.533 Paraphrasing0.1940.7420.8880.534 C30.1150.8130.9240.525 Self-report0.1790.5720.7600.358 llama-3.1-8b-instruct self-consistency0.1390.7220.8020.584 Paraphrasing0.1900.7190.8100.591 C30.1100.7450.8510.573 Self-report0.2260.5380.7140.366 llama-3.2-1b-instruct self-consistency0.1300.6750.6990.623 Paraphrasing0.2270.6800.7180.627 C30.0930.6840.7180.607 Self-report0.2840.5070.6010.450 llama-3.2-3b-instruct self-consistency0.2190.7030.7430.599 Paraphrasing0.3020.6860.7250.609 C30.1280.7380.7970.637 Self-report0.2740.4720.5820.431 mistral-large-2411 self-consistency0.1890.5770.7990.329 Paraphrasing0.2250.6180.8490.380 C30.1440.7600.8850.556 Self-report0.1370.5760.8430.241 mistral-medium-3 self-consistency0.1830.5620.8030.270 Paraphrasing0.2190.5240.7800.276 C30.1460.7420.8810.484 Self-report0.1160.5420.8100.243 mistral-small-24b-instruct-2501 self-consistency0.2020.6130.7960.368 Paraphrasing0.2360.6310.8200.449 C30.1720.7220.8610.460 Self-report0.1340.5650.8110.300 23 Table 4: FactScore Model Performance Metrics MetricECEAUROCPR-PPR-N gemini-2.5-flash self-consistency0.0750.8930.9880.399 Paraphrasing0.1100.8440.9120.376 C30.0500.9240.9920.464 Self-report0.3470.3670.8560.095 gemini-2.5-flash-lite self-consistency0.1430.7180.9450.279 Paraphrasing0.1560.6650.9300.233 C30.0690.7410.9570.247 Self-report0.2770.5530.8950.162 gemma-3-12b-it self-consistency0.3430.9150.9200.926 Paraphrasing0.3490.8770.9470.948 C30.2650.9310.9360.930 Self-report0.4110.4430.5220.416 gemma-3-27b-it self-consistency0.3010.9230.9300.929 Paraphrasing0.3040.8840.9270.941 C30.2220.9730.9810.971 Self-report0.3510.4790.5970.391 gemma-3-4b-it self-consistency0.4110.9350.9360.948 Paraphrasing0.4750.9030.9490.935 C30.3420.9370.9030.961 Self-report0.3610.5600.4680.699 gpt-4.1 self-consistency0.0801.0001.0001.000 Paraphrasing0.1030.9590.9880.979 C30.0590.9840.9990.804 Self-report0.2890.3320.8920.068 gpt-4.1-mini self-consistency0.0820.9770.9970.735 Paraphrasing0.0840.9480.9990.680 C30.0640.9890.9990.915 Self-report0.3160.4750.8710.128 gpt-4.1-nano self-consistency0.1360.7880.9690.633 Paraphrasing0.1190.8170.9790.597 C30.0940.8700.9880.342 Self-report0.3560.4920.9360.093 MetricECEAUROCPR-PPR-N llama-3.1-405b-instruct self-consistency0.0980.9070.9850.642 Paraphrasing0.1400.8800.9640.657 C30.0690.9200.9880.540 Self-report0.2970.5600.8850.219 llama-3.1-70b-instruct self-consistency0.2160.9060.9690.771 Paraphrasing0.2150.8450.9280.794 C30.1410.9130.9710.780 Self-report0.3210.4770.7160.302 llama-3.1-8b-instruct self-consistency0.2930.7390.8120.678 Paraphrasing0.3230.7290.7640.617 C30.2150.7890.8170.737 Self-report0.3930.5150.6110.450 llama-3.2-1b-instruct self-consistency0.4110.7030.5280.874 Paraphrasing0.4590.6820.4880.815 C30.2780.6960.5800.837 Self-report0.2980.5410.2920.782 llama-3.2-3b-instruct self-consistency0.2990.6980.7180.767 Paraphrasing0.3040.7020.7240.713 C30.2150.6920.7020.719 Self-report0.3760.5050.3940.641 mistral-large-2411 self-consistency0.2060.9290.9850.804 Paraphrasing0.2660.8680.9440.793 C30.1130.9490.9910.764 Self-report0.3050.5430.8340.273 mistral-medium-3 self-consistency0.2400.9290.9790.737 Paraphrasing0.2710.9000.9020.786 C30.1570.8900.9680.599 Self-report0.2680.5120.7740.277 mistral-small-24b-instruct-2501 self-consistency0.2680.7700.8770.568 Paraphrasing0.3130.7480.8450.598 C30.2020.7730.8790.613 Self-report0.3160.5850.7580.369 24 Table 5: HumanEval Model Performance Metrics MetricECEAUROCPR-PPR-N gemini-2.5-flash self-consistency0.1790.8650.9580.193 Paraphrasing0.2130.8310.9250.159 C30.1650.9110.9980.237 Self-report0.2700.4960.9850.025 gemini-2.5-flash-lite self-consistency0.2790.9090.9580.249 Paraphrasing0.3300.8590.9070.199 C30.2370.9410.9990.282 Self-report0.2180.8100.9960.075 gemma-3-12b-it self-consistency0.0870.7820.9160.493 Paraphrasing0.1190.7500.8830.461 C30.0550.8620.9690.605 Self-report0.3110.4330.8100.152 gemma-3-27b-it self-consistency0.1160.7140.9020.302 Paraphrasing0.1590.6710.8600.260 C30.0690.7680.9540.360 Self-report0.2070.5090.8720.142 gemma-3-4b-it self-consistency0.0930.7400.8230.635 Paraphrasing0.1340.6990.7820.594 C30.0490.7920.8580.691 Self-report0.2940.5350.7210.315 gpt-4.1 self-consistency0.1100.8570.9640.236 Paraphrasing0.1400.8260.9340.205 C30.0940.8440.9920.239 Self-report0.2720.6180.9660.079 gpt-4.1-mini self-consistency0.1360.8540.9580.179 Paraphrasing0.1520.8390.9420.163 C30.1130.8830.9950.190 Self-report0.2540.4830.9570.052 gpt-4.1-nano self-consistency0.1000.8120.9510.547 Paraphrasing0.1290.7830.9220.518 C30.1030.8470.9760.553 Self-report0.2730.5290.9240.100 MetricECEAUROCPR-PPR-N llama-3.1-405b-instruct self-consistency0.1050.8190.9270.585 Paraphrasing0.1430.7810.8890.547 C30.1040.8060.9440.560 Self-report0.2840.4870.7800.205 llama-3.1-70b-instruct self-consistency0.1400.8350.9280.700 Paraphrasing0.1610.8140.9070.679 C30.1130.8520.9560.656 Self-report0.3630.3670.7340.170 llama-3.1-8b-instruct self-consistency0.1160.8380.8760.787 Paraphrasing0.1480.8050.8430.754 C30.0930.8750.9180.841 Self-report0.3650.5840.6840.467 llama-3.2-1b-instruct self-consistency0.2360.7510.4950.850 Paraphrasing0.2560.7300.4740.829 C30.1300.8510.7540.917 Self-report0.3370.5170.3090.728 llama-3.2-3b-instruct self-consistency0.0940.8680.8680.873 Paraphrasing0.1300.8320.8320.837 C30.0890.8710.8740.874 Self-report0.3810.4970.5110.491 mistral-large-2411 self-consistency0.1000.7720.9370.452 Paraphrasing0.1380.7330.8980.413 C30.0970.7810.9580.467 Self-report0.2450.5830.9060.154 mistral-medium-3 self-consistency0.1050.8130.9530.286 Paraphrasing0.1370.7810.9210.253 C30.1210.8620.9810.401 Self-report0.2540.6140.9560.091 mistral-small-24b-instruct-2501 self-consistency0.1040.7690.9190.434 Paraphrasing0.1570.7160.8660.381 C30.1230.8020.9580.445 Self-report0.2630.3810.8040.128 25 Table 6: SimpleQA Model Performance Metrics MetricECEAUROCPR-PPR-N gemini-2.5-flash self-consistency0.2020.8380.6940.909 Paraphrasing0.2190.8210.6770.892 C30.1140.8270.6330.924 Self-report0.6990.4620.2460.735 gemini-2.5-flash-lite self-consistency0.3080.8720.5130.954 Paraphrasing0.3590.8220.4620.904 C30.1290.8350.3560.974 Self-report0.7880.4530.1170.865 gemma-3-12b-it self-consistency0.4870.7810.1120.970 Paraphrasing0.5020.7650.0960.955 C30.1740.8290.1990.988 Self-report0.8980.4480.0580.928 gemma-3-27b-it self-consistency0.5840.6750.1570.906 Paraphrasing0.6190.6390.1210.870 C30.2440.7540.2310.961 Self-report0.8490.3640.0690.886 gemma-3-4b-it self-consistency0.5400.7850.0570.983 Paraphrasing0.5700.7540.0260.952 C30.1930.8280.0930.994 Self-report0.9520.6690.0400.986 gpt-4.1 self-consistency0.2390.7620.6590.807 Paraphrasing0.2490.7520.6480.796 C30.2030.7420.6490.797 Self-report0.5590.4560.3550.609 gpt-4.1-mini self-consistency0.3010.8500.4820.946 Paraphrasing0.2830.8680.5000.965 C30.1360.8640.5580.968 Self-report0.7440.5110.1810.836 gpt-4.1-nano self-consistency0.2330.8970.4150.999 Paraphrasing0.2410.8890.4060.991 C30.1330.8480.3260.979 Self-report0.8120.4510.0900.912 MetricECEAUROCPR-PPR-N llama-3.1-405b-instruct self-consistency0.1720.6350.5080.713 Paraphrasing0.1980.6080.4820.686 C30.2310.6740.5170.745 Self-report0.5260.5380.4380.623 llama-3.1-70b-instruct self-consistency0.1670.8340.5480.957 Paraphrasing0.1580.8430.5560.965 C30.0860.8900.6000.972 Self-report0.7560.4480.1410.825 llama-3.1-8b-instruct self-consistency0.1860.7500.3070.924 Paraphrasing0.2010.7350.2920.909 C30.0520.8060.2860.974 Self-report0.8670.5240.0680.944 llama-3.2-1b-instruct self-consistency0.8570.7980.0100.975 Paraphrasing0.8480.8070.0180.983 C30.2460.9420.1250.999 Self-report0.9000.8730.0530.996 llama-3.2-3b-instruct self-consistency0.1950.9560.4330.989 Paraphrasing0.2170.9340.4100.967 C30.0490.9470.3000.998 Self-report0.8480.2340.0270.952 mistral-large-2411 self-consistency0.6280.7610.4220.911 Paraphrasing0.6550.7340.3940.883 C30.2020.8240.5410.936 Self-report0.7200.4640.2140.753 mistral-medium-3 self-consistency0.6980.7200.3100.895 Paraphrasing0.7120.7060.2960.881 C30.2330.8100.5160.944 Self-report0.7220.4890.2170.798 mistral-small-24b-instruct-2501 self-consistency0.4900.7540.2570.948 Paraphrasing0.5470.6970.2000.891 C30.2320.7540.2990.949 Self-report0.8100.4510.1070.886 26 Table 7: SVAMP Model Performance Metrics MetricECEAUROCPR-PPR-N gemini-2.5-flash self-consistency0.0470.9340.9860.766 Paraphrasing0.0620.9140.9660.746 C30.0230.9780.9980.817 Self-report0.1060.5230.8940.150 gemini-2.5-flash-lite self-consistency0.1210.9420.9760.783 Paraphrasing0.1360.9220.9560.763 C30.0340.9520.9860.860 Self-report0.2160.5000.7840.216 gemma-3-12b-it self-consistency0.2080.7990.8570.678 Paraphrasing0.2230.7790.8370.658 C30.0750.9130.9460.834 Self-report0.3000.5000.7000.300 gemma-3-27b-it self-consistency0.1780.8330.9090.649 Paraphrasing0.1930.8130.8890.629 C30.0640.9300.9700.797 Self-report0.2400.5000.7600.240 gemma-3-4b-it self-consistency0.3760.7770.7220.756 Paraphrasing0.3910.7570.7020.736 C30.1170.8460.8350.805 Self-report0.4350.4960.5630.435 gpt-4.1 self-consistency0.0620.8630.9710.709 Paraphrasing0.0770.8430.9510.689 C30.0260.9270.9850.787 Self-report0.0930.5530.9140.190 gpt-4.1-mini self-consistency0.0780.8720.9640.724 Paraphrasing0.0930.8520.9440.704 C30.0420.9290.9830.786 Self-report0.1010.6050.9090.150 gpt-4.1-nano self-consistency0.1590.8530.9160.723 Paraphrasing0.1740.8330.8960.703 C30.0900.8780.9430.688 Self-report0.2530.5570.7460.308 MetricECEAUROCPR-PPR-N llama-3.1-405b-instruct self-consistency0.0320.9530.9930.711 Paraphrasing0.0470.9330.9730.691 C30.0780.9450.9920.691 Self-report0.1280.5470.8810.186 llama-3.1-70b-instruct self-consistency0.0620.9270.9800.693 Paraphrasing0.0770.9070.9600.673 C30.0670.9430.9880.741 Self-report0.1650.5120.8380.175 llama-3.1-8b-instruct self-consistency0.0530.9040.9020.901 Paraphrasing0.0680.8840.8820.881 C30.1560.9070.9040.917 Self-report0.3620.5380.6390.419 llama-3.2-1b-instruct self-consistency0.1410.8560.7510.908 Paraphrasing0.1560.8360.7310.888 C30.1000.8810.7970.935 Self-report0.4580.6130.5680.578 llama-3.2-3b-instruct self-consistency0.1510.8440.7150.905 Paraphrasing0.1660.8240.6950.885 C30.0840.8560.7770.907 Self-report0.4180.5090.5730.430 mistral-large-2411 self-consistency0.1610.7690.9010.590 Paraphrasing0.1760.7490.8810.570 C30.0700.9470.9840.790 Self-report0.1700.4970.8290.170 mistral-medium-3 self-consistency0.2250.7960.8690.633 Paraphrasing0.2400.7760.8490.613 C30.0400.9370.9700.869 Self-report0.1660.5560.8460.256 mistral-small-24b-instruct-2501 self-consistency0.2220.8510.8800.703 Paraphrasing0.2370.8310.8600.683 C30.0800.9010.9220.865 Self-report0.2300.5000.7700.230 27 D Prompts D.1 The Prompt for Sampling Prompt: LLMs as Noise Samplers You are given the following question: question. Your task: Identify the core topic/domain and the essential capability the question tests (e.g., arithmetic computation, set reasoning, causal inference). Generate one short premise: a standalone sentence related to that same topic and capability that is neutral with respect to the question’s main claim/ condition (i.e., it neither entails nor contradicts it). Requirements for the premise: - Relevance: Stays within the same topic and capability as the question. - Neutrality: Does not support, imply, or contradict the question’s answer or key condition. - Simplicity: 5--15 words, clear and concise. - Randomness: Use different entities/names/numbers than those in the question; keep it generic; Obey Relevance and Neutrality, try your best to explore as much possibilities as possible. Return ONLY a complete sentence. D.2 The Prompt for Sampling Prompt: Paraphrasing Please rewrite the following statement to express the same meaning using different phrasing, while preserving its original logical content: premise D.3 The Prompt for Self-Confidence Prompt: Self-Confidence Question: question_prompt In addition, report your confidence in the correctness of this answer as a number between 0 and 100. Note: The confidence indicates how likely you think your answer is true. Respond strictly in the following format: Answer: <Answer to the question> Confidence: <Your confidence to the answer> 28 D.4 The Prompt for Benchmarks Prompt: SVAMP Solve the following math word problem and provide only the final numeric answer. Do not include steps, units, or explanation. Problem: Jessie weighed 92 kilograms. After she started to go jogging everyday she lost 56 kilograms in the first week and 99 kilograms in the second week. How much did she weigh after the first week of jogging? Final Answer: Prompt: MMLU High School Stats Question: The appraised values of houses in a city have a mean of$125,000 with a standard deviation of$23,000. Because of a new teachers’ contract, the school district needs an extra 10% in funds compared to the previous year. To raise this additional money, the city instructs the assessment office to raise all appraised house values by$5,000. What will be the new standard deviation of the appraised values of houses in the city? Choices: A.$23,000 B.$25,300 C.$28,000 D.$30,300 Answer with a single letter (A-D). Answer: Prompt: SimpleQA Question: How many television match officials were from England in the 2019 Rugby World Cup? Prompt: FActScore Tell me a bio of Kang Ji-hwan. Prompt: HumanEval from typing import List def has_close_elements(numbers: List[float], threshold: float) -> bool: ‘’‘ Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>> has_close_elements([1.0, 2.0, 3.0], 0.5) False >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True ’‘’ 29 Prompt: CommonsenseQA Question: John was told to leave the cheese in the cellar for a few years. Why is that? Choices: A. strong odor B. age well C. salad dressing D. flavor E. age to get better Answer with a single letter (A-E). D.5 The Prompt for LLMs as Judges Prompt: Judge Same Answer Task: Compare two code completions for the same problem. Problem Description: problem Determine if the following two answers are essentially the same answer. Ignore trivial differences like white space. Answer A: ans_a Answer B: ans_b Answer strictly with ’1’ for Yes or ’0’ for No. Answer: 30 E Experiment Setup E.1 Models We evaluate C3 on 16 widely used LLMs spanning multiple families and scales: OpenAI’s GPT-4.1 series (4.1, 4.1-mini, 4.1-nano) [OpenAI et al., 2024b]; Google’s Gemma-3 Instruct models (4B, 12B, 27B) [Team et al., 2025b] and Gemini models (Gemini-2.5-Flash, Gemini-2.5-Flash-Lite) [Comanici et al., 2025]; Meta’s Llama-3 Instruct models (1B, 3B, 8B, 70B, 405B) [Grattafiori et al., 2024]; and Mistral models (Small, Medium, Large) [Jiang et al., 2024a]. We also conducted case studies on additional models; however, due to their characteristics: such as heavier reasoning processes or deprecated designs, they are often costly in time and computational resources. We therefore restrict these case studies to the MMLU High School Statistics benchmark. To study the temporal evolution of C3 in Figure 1, we additionally include earlier and newer frontier models, including GPT-3.5-Turbo [Brown et al., 2020], GPT-4-Turbo, GPT-4o, and GPT- 4o-mini [OpenAI et al., 2024a]; GPT-5 (5, 5-mini, 5-nano) [Singh et al., 2025]; and additional Gemini releases (Gemini-2.5-Pro, Gemini-2.0-Flash, and Gemini-2.0-Flash-Lite) [Team et al., 2025a]. Across all experiments, we use temperatureT =1to probe typical stochastic generation behavior under standard decoding. E.2 Evaluation Metrics We evaluate C3 and other baseline scores, all normalized to the range[0, 1]. A key aspect of these methods is to disclose the reliability of model generations: Their alignment with correctness or truthfulness. For each benchmark instance, we estimate an empirical model performance by aggregating outcomes overnsampling trials (repeated samples of stochastic decoding) and taking the per-instance average. We then evaluate how well each score associated with instances aligns with this per-instance average performance using a combination of calibration and ranking metrics. To measure calibration, we report the Expected Calibration Error (ECE), computed by partitioning instances intoBbins according to their scores asECE = P B b=1 |I b | N |perf(I b )− score(I b )| , where Nis the number of instances,perf(I b )denotes the average performance of instances in binb, and score(I b )denotes the average predicted score in binb; lower ECE values indicate better calibration. For ranking-based evaluation, we define binary labels by thresholding the per-instance average performance asy i = 1[ ̄p i ≥ 0.5], and report AUROC by treating the score for instanceias a ranking signal to separate instances withy i = 1from those withy i = 0. In addition, we report precision-recall metrics that evaluate how well a score ranks instances by correctness, including the area under the precision–recall curve for detecting correct outputs (AUPRC-P) and for detecting incorrect outputs (AUPRC-N), with the latter using 1− s i . 31 F Ablation Study on Source of Noises SimpleQA Method ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ Self-consistency0.3930.7920.3680.924 Paraphrasing0.4110.7730.3490.906 Self-report 0.7780.4900.1510.846 C3 (GPT-4.1)0.1660.8230.3890.944 C3 (Qwen3-8B)0.2350.8330.3780.931 C3 (Web Source)0.2470.8010.3550.894 Table 8: Calibration results on SimpleQA comparing C3 under different perturbation sources. We compare perturbations generated by GPT-4.1, perturbations generated by the open model Qwen3- 8B, and randomly sampled web-sourced noise. Results show that C3 remains competitive across perturbation sources, suggesting that its calibration signal is not solely dependent on frontier-model- generated perturbations. Although perturbations generated by GPT-4.1 are preferred in our main experiments because they consistently produce higher-quality generations and better satisfy the intended properties of diversity and content neutrality, C3 does not fundamentally depend on GPT-4.1 as the perturbation source. To test this, we compare GPT-4.1 perturbations with two alternative sources: (1) perturbations generated by the open-weight Qwen3-8B model, and (2) random web-sourced noise of approximately the same length sampled from RedPajama [Computer, 2023], which contains internet-scraped text from a broad range of domains. Effectiveness of noise. As shown in Table 8, C3 remains competitive when perturbations are generated from non-frontier sources. GPT-4.1 achieves the best ECE, AUPRC-P, and AUPRC-N, suggesting that higher-quality perturbations can improve calibration, especially in terms of probability calibration and precision-recall behavior. However, Qwen3-8B achieves the highest AUROC among the C3 variants and remains close to GPT-4.1 on AUPRC-P and AUPRC-N. Random web-sourced noise also preserves a meaningful calibration signal, outperforming or remaining competitive with the baseline methods on several metrics. These results suggest that the effectiveness of C3 is not solely an artifact of using a strong frontier model to generate perturbations. Instead, the core signal appears to come from measuring whether model outputs remain stable under semantically neutral contextual variation. Computational efficiency. It has to be admitted that using advanced model like GPT-4.1 can be costly and using smaller open-weight models substantially reduces the cost of perturbation generation. In our Qwen3-8B setting, we generated perturbations for 200 SimpleQA questions on a single GPU NVIDIA A100. We generated 30 samples per question, this corresponds to 6,000 accepted noise samples in 32.36 minutes after we applied filtering for diversity. This is the exact setting in the main experiment. The generation pipeline achieved 3.09 completed accepted samples per second. These results suggest that perturbation generation with smaller and open sourced model is practically feasible and can substantially reduce dependence on expensive frontier-model APIs. Randomly sampled in-the-wild corpus noise provides an even cheaper alternative. Unlike model- generated perturbations, this source does not require inference or prompt-specific generation: once a corpus such as RedPajama [Computer, 2023] is available, noise snippets of the desired length can be sampled almost instantly. Because these snippets are sampled independently of the original question, they are unlikely to contain answer-specific information, which gives them a degree of semantic neutrality. However, this neutrality comes at the cost of weaker topic alignment: unlike GPT-4.1 or Qwen3-8B perturbations, in-the-wild snippets are not explicitly generated to match the benchmark domain or capability being tested. As shown in Table 8, corpus-based noise still preserves a useful C3 signal, but it is generally less effective than model-generated perturbations. This pattern suggests that topic alignment improves perturbation quality and strengthens the resulting C3 signal, while not being strictly necessary for C3 to remain informative. Web-sourced perturbations can therefore be viewed as a practical low-cost 32 approximation when generation cost is a concern, rather than as a replacement for topic-aligned model-generated perturbations. 33 G Ablation Study on MMD Table 9: Calibration results on SimpleQA comparing the original MMD C3 score with a simpler cross-comparison methods. The cross-comparison method replaces the MMD distance with the average pairwise similarity between generations from the original and perturbed prompts. The noises are still sampled from GPT-4.1. SimpleQA Method ECE↓AUROC↑AUPRC-P↑AUPRC-N↑ Self-consistency0.3930.7920.3680.924 Paraphrasing0.4110.7730.3490.906 Self-report0.7780.4900.1510.846 C3 (MMD)0.1660.8230.3890.944 C3 (Cross Comparison)0.1550.8130.4020.931 To test whether the effectiveness of C3 depends specifically on the MMD formulation, we replace the original MMD distributional distance with a simpler cross-comparison score. Given gener- ations from the original promptX = x 1 ,...,x n and generations from the perturbed prompt Y =y 1 ,...,y m , the cross-comparison variant directly measures the average pairwise agreement between the two sets: S cross (X,Y ) = 1 nm n X i=1 m X j=1 k(x i ,y j ), where k(x i ,y j ) is an indicator function for answer equivalence: k(x i ,y j ) = 1[x i ≡ y j ]. Here,x i ≡ y j means that the two generations give the same answer. For fixed-format tasks such as SimpleQA, this can be implemented by exact answer matching or by an equivalence judge when surface forms differ but the answer is semantically the same. As shown in Table 9, the cross- comparison remains competitive with the original MMD C3 score. It slightly improves ECE and AUPRC-P on SimpleQA, while MMD achieves higher AUROC and AUPRC-N. More importantly, both C3 variants outperform self-consistency, paraphrasing consistency, and self-report on most calibration and ranking metrics. This suggests that the C3 signal is not merely an artifact of the specific MMD distance. Instead, the useful signal appears to come from the broader perturbation comparison: when a model’s generations remain equivalent across original and semantically perturbed contexts, its answers are more likely to be reliable; when the cross-context generations diverge, the answer is more likely to be fragile or incorrect. MMD remains our main choice because it provides a principled distributional distance that accounts for both within-set and cross-set similarities, but this ablation shows that a simpler indicator cross-comparison variant can preserve much of the same credibility signal. 34 H Operationalization through MMD To quantify the distance between the generative distributionsP (Y|x)andP (Y|x ′ ), we use Max- imum Mean Discrepancy (MMD) [Gretton et al., 2012], a non-parametric kernel-based statistic for comparing two empirical distributions. Given finite sample setsYandY ′ of sizenandm, we compute the unbiased empirical estimate: \ MMD 2 (Y,Y ′ ) = 1 n(n− 1) n X i̸=j k(φ(y i ),φ(y j )) + 1 m(m− 1) m X i̸=j k(φ(y ′ i ),φ(y ′ j )) − 2 nm n X i=1 m X j=1 k(φ(y i ),φ(y ′ j )). (1) Here,φmaps model generations into a task-appropriate representation, andkis a kernel function that compares represented outputs. This estimator measures how much the empirical answer distribution under the original prompt differs from the answer distribution under the perturbed prompt. In the main experiments, we normalize this distance into a C3 score in[0, 1], where larger values indicate smaller cross-contextual shift and therefore greater answer stability. Task-adaptive feature maps and kernel selection The flexibility of choices of feature mapping functionφ(·)and kernel functionk(·,·)provide the flexibility of assessing generation of various types. The realization ofφandkare adapted to the specific format of the model’s outputyof the underlying task. For tasks with fixed output formats (e.g,. keywords, numbers, or multiple- choices), we could chooseφto be a mapping to categories, resulting in an indicator kernelk(y,y ′ ) = I(y = y ′ ) . This setting enables the C3 to function as an distance between empirical probability mass functions, measuring categorical inconsistencies. Conversely, for open-ended generation (e.g,. coding, summarization, and essay writing),φcould be a embedding process from an embedding model that maps the generation to high dimensional spacesh∈ R D . In these high-dimensional space, we employ a semantic kernel (typically a linear dot-product to measure the cos similarities). Even more flexibly, LLMs as Judges frameworks can be used directly to compare the if the two answers are the same or not regardless of the output formats. Normalization and range of C3 To ensure C3 is an interpretable proxy for credibility, we transform the raw MMD distance into a normalized range of[0, 1]. In our framework, we assume a characteristic kernelkthat is bounded and normalized, satisfyingk(y,y) = 1and0 ≤ k(y,y ′ ) ≤ 1(e.g., an indicator kernel or cosine similarity). Under these conditions, the squared MMD admits the theoretical upper boundMMD 2 (Y,Y ′ )≤ 2, which is attained when the two generative distributions are maximally separated (i.e., their cross-similarity approaches zero; equivalently, in fixed-format settings, outputs from the two sets completely mismatch). We define the final C3 by scaling this distance:C3(x,x ′ ) = 1− 1 2 \ MMD 2 (Y,Y ′ ) .A value ofC3 ≈ 1indicates high credibility, where the model’s output distribution remains invariant to semantic perturbations, suggesting consistent internal reasoning patterns and parametric memories. Conversely,C3≈ 0indicates low credibility, signaling that semantic variation in the input has caused a complete shift in the model’s generative behavior, which is a symptom of factual fragility or reasoning inconsistencies. 35 I Benchmark Details We provide additional details on the six benchmarks used in our evaluation. SVAMP [Patel et al., 2021] contains arithmetic word problems that require multi-step numerical reasoning and typically produce single numeric answers. MMLU High School Statistics [Hendrycks et al., 2021b,a] is an exam-style multiple-choice benchmark that evaluates statistical concepts, conceptual understanding, and quantitative reasoning. CommonsenseQA [Talmor et al., 2019] evaluates commonsense and relational inference in a multiple-choice format. For factuality-oriented evaluation, we use SimpleQA Verified [Haas et al., 2025] and FActScore [Min et al., 2023]. SimpleQA Verified consists of short-form fact-retrieval questions with verified answers, while FActScore evaluates long-form generations by decomposing outputs into atomic claims and checking whether each claim is supported by external evidence. Finally, HumanEval [Chen et al., 2021] evaluates code synthesis through unit tests. Together, these benchmarks span reasoning, factual recall, long-form factuality, and code generation, covering both constrained answer formats and open-ended generations. We include the prompts used for each benchmark below. J Baseline Details Since C3 serves as a proxy for the credibility of model generations, we compare it with related notions of confidence, consistency, and factuality. We include both vanilla black-box confidence estimators and a non-vanilla factuality checking tool. Vanilla approaches.We consider three vanilla black-box baselines. Self-reported confidence [Lin et al., 2022] asks the model to output an explicit numeric confidence score, with clearly defined upper and lower bounds, alongside its answer. This baseline tests whether the model can verbalize its own uncertainty in a way that aligns with correctness or factual support. The prompts used for self-reported confidence are provided in Appendix D.3. Self-consistency [Wang et al., 2023] estimates confidence from repeated stochastic decoding under the same prompt. We sampleKcompletions and measure agreement among the generated outputs, where higher agreement indicates greater confidence in the model’s answer. Paraphrasing consistency [Portillo Wightman et al., 2023] measures whether model outputs remain stable under meaning-preserving prompt-level paraphrases. We generateKparaphrased variants of the original question and compare the resulting answers using the same consistency framework as self-consistency. For sampling-based methods, including self-consistency and paraphrasing consistency, we compute agreement over a set of K generated outputsY =y 1 ,y 2 ,...,y K : C(Y) = 1 K(K− 1) K X i=1 K X j̸=i k(y i ,y j ),(2) wherek(y i ,y j )is a similarity function. For fixed-format tasks, we use an indicator function1[y i = y j ]. For open-ended generation, we use a semantic similarity metric. Non-vanilla approach. We also compare C3 with FActScore [Min et al., 2023], a factuality checking tool for long-form generation. FActScore decomposes each output into atomic facts, retrieves evidence from an external knowledge source, and checks whether the evidence supports each claim. In the benchmark setting, Wikipedia is used as the external knowledge source. The final FActScore is computed as the fraction of supported atomic facts. 36