Paper deep dive
When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study
Mohit Singh Chauhan, Vipin Gyanchandani, Dylan Bouchard
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.
Tags
Links
- Source: https://arxiv.org/abs/2608.24492v1
- Canonical: https://arxiv.org/abs/2608.24492v1
Trouble viewing inline? Open PDF directly →
Full Text
87,343 characters extracted from source content.
Expand or collapse full text
When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study Mohit Singh ChauhanVipin GyanchandaniDylan Bouchard †thanks: Correspondence: mohitsingh.chauhan@cvshealth.com CVS Health Wellesley MA USA Abstract Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit. 1 Introduction Large language models (LLMs) are increasingly used as closed-book generators in settings where retrieval is unavailable, including internal knowledge assistants, clinical and policy summarization workflows, customer support composition, and code generation in proprietary repositories. In these deployments, hallucinations are a primary failure mode: the model produces fluent but incorrect statements, fabricated citations, or incorrect code behaviors. Detecting hallucinations at inference time is difficult because the system often lacks a trusted external reference and because hallucination rates vary sharply by domain, prompt distribution, and base LLM, making reliable closed-book detection a practical bottleneck for safe deployment. Uncertainty quantification (UQ) provides a rich set of signals for hallucination detection, and prior work has proposed black-box methods (e.g., self-consistency across samples or perturbations), white-box methods (e.g., likelihood- or token-probability-derived measures), and reflexive or self-judge methods (e.g., model-generated assessments of its own correctness). These signals are attractive because they can be computed without retrieval, often using only model outputs and, when available, token-level probabilities. In practice, however, no single signal is uniformly superior across generation regimes (e.g. natural language vs. code), and the same scorer can behave differently across domains (e.g. math vs. factual QA) and LLMs, making zero-shot thresholds brittle. A natural response is to combine multiple UQ scorers via supervised ensembling. Prior work has demonstrated that even simple weighted-average ensembles of black-box, white-box, and judge-based scorers consistently outperform individual components in controlled, in-domain short-form question-answering settings (Bouchard and Chauhan, 2025). However, that work evaluated ensembles only with a fixed combination strategy (linear weighting), only in-distribution, and only on short-form outputs. The robustness of supervised UQ ensembles across realistic deployment conditions, where labeled data may be scarce, test distributions may drift from training, and generation formats vary widely, remains an open question. In this work, we study a supervised ensembling framework that trains a classifier over a heterogeneous collection of black-box, white-box, and reflexive UQ scorer outputs on a domain-specific dataset of labeled LLM responses, applied to out-of-sample hallucination classification in a closed-book setting. Our primary contribution is a systematic robustness study along three deployment-critical axes: (1) sample efficiency, (2) in-domain transfer under distribution shift, and (3) generation regime dependence across short-form QA, long-form generation, and code generation. Across four LLMs, nine datasets, and three generation regimes, we find that supervised ensembles outperform the best individual scorer in 30 of 32 settings by AUROC and 29 of 32 by calibration (ECE), with gains realized from as few as 100 labeled instances. In cases of in-domain distributional shift, ensembles retain most of their in-distribution advantage, outperforming the best individual scorer in 23 of 28 transfer settings. Black-box-only ensembles are nearly as effective as full ensembles, while white-box-only ensembles offer limited benefit. Among combination strategies, logistic regression offers the best overall balance of performance and stability across regimes. 2 Related Work Uncertainty Quantification for LLMs. A variety of UQ methods have been proposed for hallucination detection in LLM outputs (Huang et al., 2025; Shorinwa et al., 2025). These methods vary along two dimensions: access requirements (black-box, requiring only text outputs, vs. white-box, requiring token probabilities) and mechanism. Sampling-based consistency methods generate multiple responses to the same prompt and measure consistency via exact match (Cole et al., 2023), lexical similarity (Kuhn et al., 2023), embedding similarity (Manakul et al., 2023; Zhang* et al., 2020; Shorinwa et al., 2025), or NLI-based semantic equivalence (Chen and Mueller, 2024; Lin et al., 2024; Farquhar et al., 2024). These are typically black-box but can incorporate token probabilities as well (Qiu and Miikkulainen, 2024; Kuhn et al., 2023). White-box methods aggregate token probabilities into response-level scores through measures such as sequence probability, perplexity, entropy, and probability margins (Malinin and Gales, 2021; Manakul et al., 2023; Fadeeva et al., 2024; Farr et al., 2025). Reflexive methods prompt the generating LLM or an external judge to self-evaluate correctness (Kadavath et al., 2022; Chen and Mueller, 2024; Xiong et al., 2024; Tian et al., 2023). These scorer families form the individual components of the ensembles we study. UQ Beyond Short-Form Question Answering. Most UQ methods have been developed and evaluated on short-form question answering, and short-form UQ has been shown to generalize poorly to long-form outputs (Bakman et al., 2025; Vashurin et al., 2025c). Fine-grained methods address this by decomposing responses into sentences or claims and scoring each unit via entailment against sampled responses (Zhang et al., 2024), graph centrality over claim-response entailment graphs (Jiang et al., 2024), or question-generation pipelines (Farquhar et al., 2024). For code generation, early studies have investigated token-probability calibration for generated code (Spiess et al., 2025), symbolic clustering methods (Sharma and David, 2025), and functional equivalence variants of semantic entropy (Bouchard et al., 2026a). However, no prior work has evaluated the effectiveness of supervised ensembles over these regime-specific scorer families. Ensemble Approaches. For unsupervised ensembles, Chen and Mueller (2024) propose BSDetector, a two-component ensemble that computes a weighted average of observed consistency (combining exact match and NLI scores) and self-reflection certainty, and Verga et al. (2024) propose a Panel of LLM evaluators that aggregates judgments from multiple smaller LLMs rather than a single large judge. For supervised ensembles, Bouchard and Chauhan (2025) tune a weighted average over combinations of black-box, white-box, and judge-based scorers and demonstrate consistent gains over individual components on short-form benchmarks. Bakman et al. (2025) ensemble short-form UQ scorers and find ensembles consistently outperform the best individual short-form method, even with small training datasets. Our work extends supervised ensemble UQ with a dedicated robustness analysis: we compare combination strategies beyond weighted averaging, evaluate generalization under distribution shift, and study ensemble effectiveness across short-form, long-form, and code generation regimes, the latter two requiring materially different scorer families. 3 Methods 3.1 Problem Formulation We frame hallucination detection as binary classification over UQ scorer outputs. Given a prompt x and a generated response y, let (y)=(s1(y),…,sK(y))∈[0,1]Ks(y)=(s_1(y),…,s_K(y))∈[0,1]^K denote a vector of K confidence scores produced by a collection of UQ-based scorers, where each sks_k maps a response to a scalar confidence in [0,1][0,1] with higher values indicating greater confidence in correctness. The ground-truth label h(y)∈0,1h(y)∈\0,1\ indicates whether y contains a hallucination (h=1h=1) or not (h=0h=0), determined by comparison to a reference available only offline. Our goal is to learn a function f:[0,1]K→[0,1]f:[0,1]^K→[0,1] that maps the scorer vector to a single confidence score that separates hallucinated from correct responses. For long-form generation, we operate at the claim level rather than the response level. Each claim c extracted from a response receives its own scorer vector (c)∈[0,1]Ks(c)∈[0,1]^K, and the ensemble classifies claims individually, where h(c)∈0,1h(c)∈\0,1\ indicates whether the claim is supported by a reference text available only offline. 3.2 UQ Scorer Families The ensemble operates over confidence scores drawn from four families of UQ methods. Table 1 summarizes all scorers by family, access requirements, and applicable generation regimes, with formal definitions and implementation details provided in Appendix A. We describe each family below. Family Access Short Long Code Single-generation white-box Sequence probability White-box ✓ ✓ Norm. sequence probability White-box ✓ ✓ Min token probability White-box ✓ ✓ Probability margin White-box ✓ ✓ Mean token entropy White-box ✓ ✓ Max token entropy White-box ✓ ✓ Consistency-based black-box Exact match rate Black-box ✓ Non-contradiction probability Black-box ✓ BERTScore consistency Black-box ✓ Semantic entropy Black-box ✓ Cosine similarity Black-box ✓ ✓ Functional entropy Black-box ✓ Equivalence rate Black-box ✓ CodeBLEU consistency Black-box ✓ Consistency-based white-box CoCoA White-box ✓ ✓ Monte Carlo probability White-box ✓ ✓ WB semantic entropy White-box ✓ ✓ Semantic density White-box ✓ Reflexive Verbalized confidence Black-box ✓ ✓ ✓ P(True) White-box ✓ ✓ ✓ Graph-based (claim-level) Degree Centrality Black-box ✓ Betweenness Centrality Black-box ✓ Closeness Centrality Black-box ✓ Harmonic Centrality Black-box ✓ Laplacian Centrality Black-box ✓ PageRank Black-box ✓ Table 1: Summary of UQ scorers used as ensemble inputs by access and generation regime. “Access” indicates whether token probability access is required. Formal definitions are in Appendix A. Black-box consistency scorers generate m candidate responses from the same prompt using stochastic decoding and measure agreement between the original response and candidates. Methods differ in their consistency function: exact match (Cole et al., 2023), NLI-based non-contradiction or entailment (Chen and Mueller, 2024; Lin et al., 2024), embedding cosine similarity (Shorinwa et al., 2025), BERTScore (Zhang* et al., 2020), and semantic entropy via NLI-based clustering (Kuhn et al., 2023; Farquhar et al., 2024). For code generation, we additionally employ code-specific consistency functions including CodeBLEU (Ren et al., 2020) and LLM-based functional equivalence assessment, replacing NLI-based semantic comparison with judgments of whether two code snippets produce identical outputs for all valid inputs. White-box token-probability scorers derive confidence from the token-level probabilities produced during generation. We consider length-normalized sequence probability (Malinin and Gales, 2021), minimum token probability (Manakul et al., 2023), probability margin (Farr et al., 2025), and token-level entropy (Scalena et al., 2025). We also consider hybrid methods that combine token probabilities with sampling-based consistency (e.g., white-box semantic entropy (Kuhn et al., 2023)). Reflexive (judge-based) scorers prompt the generating LLM or an external LLM to evaluate correctness of a question-response pair. We consider verbalized confidence (Tian et al., 2023; Xiong et al., 2024) and P(True) (Kadavath et al., 2022). For short-form and code generation, these scorers evaluate the full response. For long-form generation, they are applied at the claim level, scoring each extracted claim individually. Claim-level scorers (long-form only) decompose responses into claims and score each claim individually, producing the claim-level confidence scores over which the ensemble operates. Following Jiang et al. (2024), we employ graph-based scorers, which construct claim-response entailment graphs and use graph centrality metrics to measure uncertainty at the claim level. 3.3 Ensembling Strategies Given the scorer vector ∈[0,1]Ks∈[0,1]^K and binary labels h∈0,1h∈\0,1\ for a training set of n labeled instances, we train a classifier f to predict hallucinations from scorer outputs. We compare four combination strategies of varying complexity: (1) logistic regression with ℓ2 _2 regularization, (2) random forest, (3) gradient boosted trees, and constrained weighted average (Bouchard and Chauhan, 2025). For all strategies, hyperparameters are selected via 5-fold cross-validation on the tuning set. See Appendix E for hyperparameter details. 4 Experiments 4.1 Setup We evaluate four LLMs spanning two providers and two capability tiers: Gemini-2.5-Flash, Gemini-2.5-Pro (Google, ), GPT-4o, and GPT-4o-mini (OpenAI, ). Experiments are organized across four core domains (Math, Factual QA, Code, and Long-form) and one standalone reading comprehension task. For short-form evaluation, we consider: (1) Math reasoning, consisting of OpenR1-Math (The Hugging Face team (past and future), 2026) and BigMath (Albalak et al., 2025) (1,000 questions each); (2) Factual QA, consisting of HotpotQA (Yang et al., 2018) and SimpleQA (Wei et al., 2024) (1,000 questions each); and (3) Reading Comprehension, using the DROP dataset (Dua et al., 2019) (1,000 questions). For code generation, we use two subsets of LiveCodeBench (Jain et al., 2025): a Leetcode-derived callable subset (442 problems) and an AtCoder/CodeForces I/O subset (610 problems), both requiring Python generation. For long-form QA, we construct two datasets following the FactScore (Min et al., 2023) protocol: world’s largest rivers (500 questions) and edible mushrooms (84 questions), with responses averaging approximately 32 claims each, yielding roughly 16,000 and 2,700 claim-level instances per LLM, respectively (see Appendix C for more details). For short-form questions, hallucination labels are obtained by comparing LLM responses to reference answers using an LLM-based grading procedure. For code generation, labels are determined by execution against test cases (pass@1). For long-form QA, responses are decomposed into claims and each claim is graded against the corresponding Wikipedia article using the FactScore protocol. Gemini-2.5-Flash is used as the grading model for short-form and long-form questions, as well as for claim decomposition in the long-form setting, chosen for its strong performance at low cost. For all sampling-based scorers, we generate 10 sampled responses per prompt across all regimes. To assess the reliability of LLM-based grading, two human annotators independently labeled a stratified sample of 400 short-form responses (20 correct and 20 incorrect per dataset per generator LLM, spanning all five short-form datasets and two generators: GPT-4o and Gemini-2.5-Flash). Annotators compared each generated answer against the reference answer without access to the grader’s label. Human-human agreement was 97.5% (Cohen’s κ=0.95κ=0.95), and LLM-human agreement was 98.8% (κ=0.97κ=0.97) and 96.2% (κ=0.93κ=0.93) for the two annotators, confirming that grading noise is no larger than inherent human disagreement. Agreement rates were comparable across both generator LLMs (κ=0.95κ=0.95 for both), providing no evidence that the grader favors its own responses. Full results broken down by dataset and generator are provided in Appendix D. All experiments share a common splitting procedure: for each domain with paired datasets, we generate 25 random stratified 70/30 splits applied simultaneously to both datasets, yielding a 70% training fold and 30% test fold for each.11 1 DROP is included as a standalone reading comprehension task for which we do not have a companion dataset. The same 25-split procedure is applied for in-distribution evaluation; DROP is excluded from the in-domain transfer analysis. The three analyses described below (in-distribution performance, in-domain transfer, and access-constrained ablations) all operate over these same 25 splits. Table 4 reports LLM accuracy rates for each dataset. Complete AUROC and ECE results, broken down by scorer and ensemble for all LLM-dataset combinations, are provided in Tables 8–15. (a) OpenR1-Math (b) Big-Math (c) SimpleQA (d) HotPotQA (e) DROP (f) LiveCodeBench (Python) (g) FactScore-Rivers (h) FactScore-Mushrooms Figure 1: Ensemble AUROC as a function of training sample size across domains. Lines show the four combination strategies with 95% CIs over 25 splits. The dashed line is the best individual scorer, selected on the test set and therefore an optimistic baseline unavailable in practice. Training sizes (in number of responses) range from 0.1N0.1N to 0.7N0.7N, where N is the dataset size. Code generation combines both LiveCodeBench subsets. 4.2 In-Distribution Performance For each of the 25 splits, we train ensembles on subsamples of the training fold at sizes 0.1N,0.2N,…,0.7N0.1N,0.2N,…,0.7N, where N is the dataset size and 0.7N0.7N corresponds to the full training fold. For each split and sample size, we draw a single subsample, train the ensemble, and evaluate on the test fold, reporting mean AUROC and 95% confidence intervals across the 25 splits. For code generation, the two LiveCodeBench subsets are combined into a single pool. The best individual scorer, defined as the scorer achieving the highest AUROC on the test set, serves as a fixed reference baseline. Note that this baseline is optimistic: it requires test-set access and is unavailable in practice, so ensemble gains relative to a realistic validation-selected scorer would be at least as large. When using the full training sample (0.7N0.7N), the best ensemble outperforms the best individual scorer in 30 of 32 LLM-dataset settings by AUROC, including every code generation and long-form setting. The two exceptions are BigMath for Gemini-2.5-Flash (best ensemble 0.85 vs. scorer 0.86) and SimpleQA for GPT-4o-mini (both 0.78). We next examine how quickly these gains emerge as a function of training set size. Figures 1(a)–1(h) show ensemble AUROC as a function of training sample size across all domains. Each panel displays the four combination strategies and the best individual scorer baseline. Short-form (Figures 1(a)–1(e)). At 700 training samples, random forest and logistic regression are the strongest combination strategies in most settings, though neither dominates uniformly. Random forest achieves the highest AUROC in settings with clear ensemble gains (e.g., 0.90 on OpenR1-Math for both Gemini models, 0.74 on DROP for Gemini-2.5-Flash and GPT-4o), while logistic regression leads or ties on datasets where the best individual scorer is already strong (e.g., HotpotQA for Gemini-2.5-Flash, SimpleQA for GPT-4o-mini). Gradient boosting is competitive at 700 samples in some settings but lags at small sample sizes. Weighted average is competitive on datasets with strong baselines (e.g., SimpleQA, HotpotQA) but falls substantially behind on OpenR1-Math, where it plateaus around 0.80–0.86 compared to 0.90 for random forest. Convergence speed varies by strategy. The weighted average stabilizes early, often by 100-200 samples. Logistic regression converges by 200-300 samples in most settings, though it continues improving through 400-500 on some datasets (e.g., OpenR1-Math). Random forest and gradient boosting continue improving with larger samples, with random forest typically reaching its peak earlier. Code generation (Figure 1(f)). The combined LiveCodeBench dataset yields consistent ensemble gains across all four LLMs: Gemini-2.5-Flash (0.89 vs. 0.87), Gemini-2.5-Pro (0.85 vs. 0.84), GPT-4o (0.88 vs. 0.86), and GPT-4o-mini (0.88 vs. 0.86). The weighted average, logistic regression, and random forest converge to similar AUROC in all four settings, with gradient boosting trailing by 0.01–0.02 for three of the four LLMs and matching the others for GPT-4o-mini. Convergence is fast, with weighted average stabilizing by 100–200 samples and logistic regression and random forest by 200–400. Long-form QA (Figures 1(g)–1(h)). The long-form setting operates at the claim level, with responses averaging approximately 32 claims each. The best ensemble outperforms the best individual scorer in all 8 settings. On the Mushrooms dataset, logistic regression and the weighted average lead for Gemini models and GPT-4o-mini, while all four strategies perform similarly for GPT-4o. On the Rivers dataset, random forest is competitive with or outperforms logistic regression for all LLMs, and substantially outperforms the weighted average for GPT-4o (0.713 vs. 0.677) and GPT-4o-mini (0.686 vs. 0.672). Gradient boosting underperforms in the long-form setting, degrading with increasing training data for Gemini-2.5-Pro on Rivers (dropping from 0.675 at 0.2N0.2N to 0.669 at 0.7N0.7N) and GPT-4o-mini (dropping from 0.674 to 0.663). The simpler strategies converge rapidly, typically by 0.1N0.1N–0.2N0.2N. Calibration. Ensembling also improves calibration considerably: across all 32 LLM-dataset settings, the best ensemble achieves the lowest ECE among all individual scorers and ensemble variants in 29 of 32 cases, with ECE never exceeding 0.06. (a) Short-Form Math (b) Short-Form Factual QA (c) Long-Form QA (d) Python Code Generation Figure 2: In-domain transfer AUROC across domains. Bars compare the best individual scorer (selected on the test set; an optimistic baseline), best in-distribution ensemble, and best out-of-distribution (OOD) ensemble (trained on the companion dataset within the same domain). Combination strategy is selected independently per condition. Bars show mean AUROC with 95% CIs over 25 splits. Code generation evaluates transfer in one direction only (I/O → Callable) due to small sample size of Callable subset. LLM Type Big-Math OpenR1 DROP HotpotQA SimpleQA Gemini 2.5 Flash Top WB Scorer 0.744±0.020.744± 0.02 0.697±0.010.697± 0.01 0.686±0.010.686± 0.01 0.769±0.020.769± 0.02 0.611±0.010.611± 0.01 Top WB Ensemble 0.737±0.020.737± 0.02 0.744±0.010.744± 0.01 0.686±0.010.686± 0.01 0.767±0.020.767± 0.02 0.611±0.010.611± 0.01 Top B Scorer 0.796±0.020.796± 0.02 0.688±0.010.688± 0.01 0.673±0.010.673± 0.01 0.828±0.010.828± 0.01 0.816±0.010.816± 0.01 Top B Ensemble 0.836±0.020.836± 0.02 0.785±0.010.785± 0.01 0.689±0.010.689± 0.01 0.857±0.010.857± 0.01 0.823±0.010.823± 0.01 Top Overall Scorer 0.857±0.010.857± 0.01 0.861±0.010.861± 0.01 0.699±0.010.699± 0.01 0.818±0.020.818± 0.02 0.817±0.010.817± 0.01 Top Overall Ensemble 0.850±0.020.850± 0.02 0.902±0.010.902± 0.01 0.741±0.010.741± 0.01 0.822±0.020.822± 0.02 0.832±0.010.832± 0.01 Gemini 2.5 Pro Top WB Scorer 0.722±0.020.722± 0.02 0.617±0.020.617± 0.02 0.687±0.020.687± 0.02 0.736±0.020.736± 0.02 0.542±0.010.542± 0.01 Top WB Ensemble 0.717±0.020.717± 0.02 0.634±0.020.634± 0.02 0.689±0.020.689± 0.02 0.733±0.020.733± 0.02 0.539±0.010.539± 0.01 Top B Scorer 0.775±0.020.775± 0.02 0.832±0.020.832± 0.02 0.651±0.020.651± 0.02 0.766±0.030.766± 0.03 0.845±0.010.845± 0.01 Top B Ensemble 0.822±0.020.822± 0.02 0.843±0.020.843± 0.02 0.687±0.010.687± 0.01 0.786±0.020.786± 0.02 0.851±0.010.851± 0.01 Top Overall Scorer 0.824±0.020.824± 0.02 0.832±0.020.832± 0.02 0.687±0.020.687± 0.02 0.807±0.020.807± 0.02 0.845±0.010.845± 0.01 Top Overall Ensemble 0.882±0.020.882± 0.02 0.898±0.010.898± 0.01 0.740±0.010.740± 0.01 0.841±0.020.841± 0.02 0.863±0.010.863± 0.01 GPT-4o Top WB Scorer 0.839±0.010.839± 0.01 0.806±0.010.806± 0.01 0.696±0.010.696± 0.01 0.732±0.020.732± 0.02 0.842±0.010.842± 0.01 Top WB Ensemble 0.848±0.010.848± 0.01 0.810±0.010.810± 0.01 0.696±0.010.696± 0.01 0.730±0.020.730± 0.02 0.852±0.010.852± 0.01 Top B Scorer 0.798±0.010.798± 0.01 0.783±0.010.783± 0.01 0.715±0.010.715± 0.01 0.704±0.020.704± 0.02 0.840±0.010.840± 0.01 Top B Ensemble 0.810±0.000.810± 0.00 0.788±0.010.788± 0.01 0.758±0.010.758± 0.01 0.763±0.020.763± 0.02 0.856±0.010.856± 0.01 Top Overall Scorer 0.839±0.010.839± 0.01 0.811±0.010.811± 0.01 0.715±0.010.715± 0.01 0.737±0.020.737± 0.02 0.838±0.010.838± 0.01 Top Overall Ensemble 0.853±0.010.853± 0.01 0.819±0.010.819± 0.01 0.767±0.010.767± 0.01 0.771±0.020.771± 0.02 0.874±0.010.874± 0.01 GPT-4o Mini Top WB Scorer 0.847±0.010.847± 0.01 0.713±0.010.713± 0.01 0.625±0.010.625± 0.01 0.748±0.010.748± 0.01 0.781±0.010.781± 0.01 Top WB Ensemble 0.858±0.010.858± 0.01 0.754±0.010.754± 0.01 0.632±0.010.632± 0.01 0.754±0.010.754± 0.01 0.783±0.010.783± 0.01 Top B Scorer 0.748±0.010.748± 0.01 0.739±0.010.739± 0.01 0.644±0.020.644± 0.02 0.747±0.020.747± 0.02 0.778±0.010.778± 0.01 Top B Ensemble 0.800±0.010.800± 0.01 0.746±0.010.746± 0.01 0.678±0.010.678± 0.01 0.775±0.010.775± 0.01 0.778±0.010.778± 0.01 Top Overall Scorer 0.846±0.010.846± 0.01 0.738±0.010.738± 0.01 0.647±0.010.647± 0.01 0.758±0.010.758± 0.01 0.784±0.010.784± 0.01 Top Overall Ensemble 0.878±0.010.878± 0.01 0.778±0.010.778± 0.01 0.703±0.010.703± 0.01 0.785±0.010.785± 0.01 0.782±0.010.782± 0.01 Table 2: Comparison of white-box (WB), black-box (B), and overall uncertainty quantification methods across five short-form datasets and four LLMs. Each row reports the best individual scorer and best ensemble within that access category. AUROC values with 95% confidence intervals over 25 random train/test splits. Bold indicates the highest AUROC per LLM-dataset pair. 4.3 In-Domain Transfer For each domain, we use the same 25 splits to evaluate in-domain transfer. Within each split, we train the ensemble on the training fold of one dataset and evaluate on the test folds of both datasets in the domain: the same dataset (in-distribution) and its companion (transfer). For example, in the math domain, an ensemble trained on OpenR1-Math is evaluated on both OpenR1-Math and BigMath test folds, and vice versa. For each evaluation dataset, we compare the best individual scorer, the best in-distribution ensemble, and the best transfer ensemble, with the combination strategy selected independently for the latter two. In 13 of 28 settings, the transfer ensemble shows no degradation relative to the in-distribution ensemble, and the maximum degradation across all settings is 0.03 AUROC. Across all 28 settings, the transfer ensemble outperforms the best individual scorer in 23 cases, ties in 2, and underperforms in 3. The 3 underperforming cases are BigMath for Gemini-2.5-Flash (transfer 0.85 vs. scorer 0.86), SimpleQA for GPT-4o-mini (0.77 vs. 0.78), and code generation for Gemini-2.5-Pro (0.79 vs. 0.81), all within confidence intervals. Transfer is consistently strong across domains, with degradation of 0.00–0.03 AUROC and the transfer ensemble matching or exceeding the best scorer in the large majority of settings. Long-form QA shows particularly robust transfer, with no degradation on Mushrooms for any LLM and degradation of at most 0.03 on Rivers. 4.4 Access-Constrained Ablations Using the same 25 splits and the full training fold, we train ensembles under two restricted access conditions on the five short-form datasets: black-box scorers only (no token probabilities) and single-generation white-box scorers only (no sampling). Table 2 reports AUROC for the best individual scorer and best ensemble under each condition, alongside the full scorer set. Full ensemble. The full ensemble outperforms the best individual scorer in 18 of 20 LLM-dataset combinations. The two exceptions (BigMath for Gemini-2.5-Flash and SimpleQA for GPT-4o-mini) are within confidence intervals. The largest gains appear for Gemini-2.5-Pro (0.04–0.07 AUROC on math datasets). Black-box only. The black-box ensemble outperforms the best black-box scorer in 19 of 20 settings. On HotpotQA for Gemini-2.5-Flash, the black-box ensemble (0.857) exceeds even the best full ensemble (0.822). White-box only. The white-box ensemble outperforms the best white-box scorer in just 11 of 20 settings. For GPT-4o and GPT-4o-mini, white-box scorers are competitive with or exceed black-box scorers on several datasets (e.g., BigMath, OpenR1 for GPT-4o). 5 Discussion Ensembling as a default strategy. Supervised ensembles that combine black-box and white-box scorers outperform the best individual scorer in 30 of 32 settings by AUROC and 29 of 32 by ECE. This consistency across four LLMs, nine datasets, and three generation regimes suggests that ensembling makes a strong default approach whenever labeled data is available. Crucially, since individual scorers require no training, the labeling effort needed to identify the best single scorer already suffices to train an ensemble, making the additional cost of ensembling negligible. This supervised approach also consistently outperforms training-free, fixed-weight combinations of token probabilities and consistency signals such as CoCoA (Vashurin et al., 2025b), which is never the top-performing scorer in our framework. Calibration. Beyond discrimination, ensembling yields well-calibrated confidence scores, with the best ensemble achieving ECE below 0.06 in every setting and below 0.05 in most. Individual scorers, by contrast, are often poorly calibrated despite reasonable AUROC. This is practically significant: well-calibrated scores enable threshold-based deployment decisions (e.g., flagging responses below a confidence threshold for human review) without requiring extensive per-dataset threshold tuning. Combination strategy selection. The optimal combination strategy varies by generation regime. For short-form and code generation, random forest and logistic regression are the strongest strategies, with random forest offering marginally higher AUROC at the cost of slower convergence. For long-form claim-level detection, logistic regression and the weighted average dominate, while gradient boosting is prone to overfitting and can degrade with increasing training data. In practice, logistic regression offers arguably the best balance: it is competitive across all regimes, converges quickly (often by 100–200 responses), and avoids the overfitting risks of tree-based methods in low-feature settings. Sample efficiency. Ensemble gains are realized at modest sample sizes. Simpler strategies (logistic regression, weighted average) often reach their plateau with as few as 100–200 labeled instances. Tree-based strategies require 300–500 to converge but can achieve higher final AUROC. For practitioners with limited labeling budgets, logistic regression provides strong performance; with larger budgets, random forest may offer incremental improvement. In-domain transfer. Transfer ensembles retain most of their in-distribution advantage, with mean degradation of only 0.02 AUROC points and transfer ensembles outperforming the best individual scorer in 23 of 28 settings. This suggests cross-dataset deployment viability, as a practitioner can train an ensemble on a labeled source dataset and deploy it to a related target dataset with reasonable confidence. The few underperforming cases are within confidence intervals rather than being clearly attributable to a systematic failure mode. Access constraints. Black-box ensembles provide nearly the same benefit as full ensembles (19 of 20 short-form settings), making them a useful default when logprobs are unavailable. The diversity of black-box consistency signals spanning exact match, NLI, embedding similarity, and semantic clustering provides sufficient complementarity for a classifier to exploit. White-box-only ensembles, by contrast, offer limited gains (11 of 20), as these six token-probability features seem to lack the diversity needed for effective combination. The informativeness of white-box scorers also varies across model families: GPT models produce consistently useful token-probability signals, while Gemini models show cases where white-box scorers are uninformative or misleading (e.g. Gemini-2.5-Pro on SimpleQA). Comparison with prior ensembling studies. Our findings are consistent with Bakman et al. (2025), who evaluate ensembling of short-form UQ-based scorers and find that linear ensembles outperform the best individual method with as few as 100 calibration samples. Our sample efficiency results align: simpler strategies plateau by 100–200 labeled instances. One divergence is that their single decision tree is generally uncompetitive, whereas our tree-based strategies (random forest, gradient boosting) are competitive in most settings. A plausible explanation is that bagging and boosting with cross-validated depth reduce the variance that makes individual trees unreliable. Our study complements theirs by providing a dedicated robustness analysis investigating cross-dataset transfer, access constraints, and two additional generation regimes (code generation and claim-level scoring). 6 Conclusion We presented a systematic robustness study of supervised UQ ensembles for LLM hallucination detection across sample efficiency, in-domain distribution shift, and generation regime. Ensembles outperform the best individual scorer in 30 of 32 settings by AUROC and 29 of 32 by calibration, with gains realized at sample sizes as small as 100 labeled instances. Cross-dataset transfer is effective in most settings (23 of 28), with the primary failure mode being scorer signals that shift across distributions. Black-box-only ensembles are nearly as effective as full ensembles, while white-box-only ensembles offer limited benefit. These findings provide actionable guidance: supervised ensembling is a reliable, low-cost default for hallucination detection when even a small labeled dataset is available, and black-box ensembles are a robust fallback when token probabilities are unavailable. Limitations Closed-weight model scope. Our study evaluates four LLMs from two providers (Google and OpenAI), all of which are closed-source. Results may not generalize to open-weight models (e.g., LLaMA, Mistral), which may exhibit different token-probability characteristics, or to models with substantially different architectures. The observed differences between Gemini and GPT models (particularly in white-box scorer informativeness) suggest that model family is an important factor, but we cannot characterize this dimension fully with only two providers. Moreover, open-weight models enable internal-state methods based on hidden representations or attention maps (Azaria and Mitchell, 2023; Chen et al., 2024; Chuang et al., 2024; Vazhentsev et al., 2025) that are infeasible with closed-weight APIs. Whether ensembles over these internal-state scorers can match the performance of sampling-based ensembles at lower inference cost is an open question for the open-weight setting. Evaluation scope. Our evaluation covers specific slices of each generation regime and transfer condition. The long-form evaluation is limited to two narrow factoid-recall domains (rivers and mushrooms) with a shared question template; broader tasks such as open-ended summarization, document drafting, or multi-turn dialogue may exhibit different ensemble behavior, particularly if claim decomposition is less straightforward. Code generation experiments use Python exclusively on competitive programming problems from LiveCodeBench, whereas real-world code generation spans multiple languages, longer codebases, and tasks beyond self-contained function synthesis. Our cross-dataset transfer evaluation considers transfer between dataset pairs within the same domain (e.g., math to math); cross-domain transfer (e.g., training on math and deploying to factual QA) and cross-LLM transfer (e.g., training on GPT-4o outputs and deploying to Gemini) remain unexplored and may exhibit substantially larger degradation. Ethical Considerations. This work aims to improve the reliability of LLM outputs by detecting hallucinations, which we view as a net positive for safe deployment. However, high-performing hallucination detectors could create a false sense of security if practitioners treat ensemble confidence scores as guarantees of correctness rather than probabilistic estimates. We emphasize that our methods reduce but do not eliminate hallucination risk, and that human oversight remains essential in high-stakes settings such as clinical or legal applications. We use publicly available research benchmarks and cite their original sources; users should obtain the datasets from the official sources and comply with the corresponding licenses and terms of use. We do not redistribute the full benchmark-derived data, and release only synthetic schema-compatible files for code smoke testing. No personally identifiable information was collected or generated. The human annotation study was conducted by the authors; no crowdworkers were employed. Conflict of Interest MSC is employed by CVS Health® Corporation and holds stock and/or equity. VG and DB were formerly employed by CVS Health® Corporation and hold stock and/or equity. No conflicts germane to this work. Disclaimer Prompts are included solely for reproducibility and do not imply endorsement or affiliation. Gemini is a trademark of Google and GPT is a trademark of OpenAI. This is an independent publication and has not been authorized, endorsed, or sponsored by Google or OpenAI. References Akiba et al. (2019) T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, p. 2623–2631. External Links: Document Cited by: Appendix E. Albalak et al. (2025) A. Albalak, D. Phung, N. Lile, R. Rafailov, K. Gandhi, L. Castricato, A. Singh, C. Blagden, V. Xiang, D. Mahan, and N. Haber Big-math: a large-scale, high-quality math dataset for reinforcement learning in language models. External Links: 2502.17387, Link Cited by: §4.1. Azaria and Mitchell (2023) A. Azaria and T. Mitchell The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 967–976. External Links: Link, Document Cited by: Closed-weight model scope.. Bakman et al. (2025) Y. Bakman, D. N. Yaldiz, S. Kang, T. Zhang, B. Buyukates, S. Avestimehr, and S. P. Karimireddy Reconsidering llm uncertainty estimation methods in the wild. External Links: 2506.01114, Link Cited by: §2, §2, §5. Bouchard et al. (2026a) D. Bouchard, M. S. Chauhan, Z. Ahmad, and H. Ra Functional entropy: predicting functional correctness in llm-generated code with uncertainty quantification. External Links: 2605.28500, Link Cited by: §A.5, §A.5, §A.5, §2. Bouchard et al. (2026b) D. Bouchard, M. S. Chauhan, V. Bajaj, and D. Skarbrevik Fine-grained uncertainty quantification for long-form language model outputs: a comparative study. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §A.6. Bouchard et al. (2026c) D. Bouchard, M. S. Chauhan, D. Skarbrevik, H. Ra, V. Bajaj, and Z. Ahmad UQLM: a python package for uncertainty quantification in large language models. Journal of Machine Learning Research 27 (13), p. 1–10. External Links: Link Cited by: Appendix E. Bouchard and Chauhan (2025) D. Bouchard and M. S. Chauhan Uncertainty quantification for language models: a suite of black-box, white-box, LLM judge, and ensemble scorers. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §A.2, §A.2, §1, §2, §3.3. Chen et al. (2024) C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Closed-weight model scope.. Chen and Mueller (2024) J. Chen and J. Mueller Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 5186–5200. External Links: Link, Document Cited by: §A.2, §A.2, §2, §2, §3.2. Chuang et al. (2024) Y. Chuang, L. Qiu, C. Hsieh, R. Krishna, Y. Kim, and J. R. Glass Lookback lens: detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 1419–1436. External Links: Link, Document Cited by: Closed-weight model scope.. Cole et al. (2023) J. R. Cole, M. J. Zhang, D. Gillick, J. M. Eisenschlos, B. Dhingra, and J. Eisenstein Selectively answering ambiguous questions. In The 2023 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §A.2, §2, §3.2. Dua et al. (2019) D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 2368–2378. External Links: Link, Document Cited by: §4.1. Fadeeva et al. (2024) E. Fadeeva, A. Rubashevskii, A. Shelmanov, S. Petrakov, H. Li, H. Mubarak, E. Tsymbalov, G. Kuzmin, A. Panchenko, T. Baldwin, P. Nakov, and M. Panov Fact-checking the output of large language models via token-level uncertainty quantification. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 9367–9385. External Links: Link, Document Cited by: §2. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. External Links: ISSN 1476-4687, Document, Link Cited by: §A.2, §2, §2, §3.2. Farr et al. (2025) D. Farr, N. Manzonelli, I. Cruickshank, and J. West RED-CT: a systems design methodology for using LLM-labeled data to train and deploy edge linguistic classifiers. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, S. Schockaert, K. Darwish, and A. Agarwal (Eds.), Abu Dhabi, UAE, p. 58–67. External Links: Link Cited by: §A.1, §2, §3.2. [17] Google External Links: Link Cited by: §4.1. Huang et al. (2025) L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. 43 (2). External Links: ISSN 1046-8188, Link, Document Cited by: §2. Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §4.1. Jiang et al. (2024) M. Jiang, Y. Ruan, P. Sattigeri, S. Roukos, and T. Hashimoto Graph-based uncertainty metrics for long-form language model generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §A.6, Appendix C, Appendix C, §2, §3.2. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan Language models (mostly) know what they know. External Links: 2207.05221, Link Cited by: §A.4, §2, §3.2. Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §A.2, §A.3, §2, §3.2, §3.2. Lin et al. (2024) Z. Lin, S. Trivedi, and J. Sun Generating with confidence: uncertainty quantification for black-box large language models. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §A.2, §A.2, §2, §3.2. Malinin and Gales (2021) A. Malinin and M. Gales Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, External Links: Link Cited by: §A.1, §2, §3.2. Manakul et al. (2023) P. Manakul, A. Liusie, and M. Gales SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 9004–9017. External Links: Link, Document Cited by: §A.1, §A.1, §A.1, §A.2, §2, §3.2. Min et al. (2023) S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 12076–12100. External Links: Link, Document Cited by: Appendix C, Appendix C, §4.1. [27] OpenAI External Links: Link Cited by: §4.1. Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay Scikit-learn: machine learning in python. Journal of Machine Learning Research 12 (85), p. 2825–2830. External Links: Link Cited by: Appendix E. Qiu and Miikkulainen (2024) X. Qiu and R. Miikkulainen Semantic density: uncertainty quantification for large language models through confidence measurement in semantic space. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Ren et al. (2020) S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma CodeBLEU: a method for automatic evaluation of code synthesis. External Links: 2009.10297, Link Cited by: §A.5, §3.2. Scalena et al. (2025) D. Scalena, L. Zotos, E. Fersini, M. Nissim, and A. Üstün EAGER: entropy-aware generation for adaptive inference-time scaling. External Links: 2510.11170, Link Cited by: §A.1, §A.1, §3.2. Sharma and David (2025) A. Sharma and C. David Assessing correctness in llm-based code generation via uncertainty estimation. External Links: 2502.11620, Link Cited by: §A.5, §2. Shorinwa et al. (2025) O. Shorinwa, Z. Mei, J. Lidard, A. Z. Ren, and A. Majumdar A survey on uncertainty quantification of large language models: taxonomy, open research challenges, and future directions. ACM Comput. Surv. 58 (3). External Links: ISSN 0360-0300, Link, Document Cited by: §2, §3.2. Spiess et al. (2025) C. Spiess, D. Gros, K. S. Pai, M. Pradel, M. R. I. Rabin, A. Alipour, S. Jha, P. Devanbu, and T. Ahmed Calibration and correctness of language models for code. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ICSE ’25, p. 540–552. External Links: ISBN 9798331505691, Link, Document Cited by: §2. The Hugging Face team (past and future) (2026) Open r1 Note: GitHub repository External Links: Link Cited by: §4.1. Tian et al. (2023) K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. External Links: 2305.14975, Link Cited by: §A.4, §2, §3.2. Vashurin et al. (2025a) R. Vashurin, E. Fadeeva, A. Vazhentsev, L. Rvanova, D. Vasilev, A. Tsvigun, S. Petrakov, R. Xing, A. Sadallah, K. Grishchenkov, A. Panchenko, T. Baldwin, P. Nakov, M. Panov, and A. Shelmanov Benchmarking uncertainty quantification methods for large language models with lm-polygraph. Transactions of the Association for Computational Linguistics 13, p. 220–248. External Links: ISSN 2307-387X, Link, Document Cited by: §A.1. Vashurin et al. (2025b) R. Vashurin, M. Goloburda, A. Ilina, A. Rubashevskii, P. Nakov, A. Shelmanov, and M. Panov CoCoA: a minimum bayes risk framework bridging confidence and consistency for uncertainty quantification in LLMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §A.3, §5. Vashurin et al. (2025c) R. Vashurin, M. Goloburda, P. Nakov, and M. Panov UNCERTAINTY-LINE: length-invariant estimation of uncertainty for large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 7881–7908. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §2. Vazhentsev et al. (2025) A. Vazhentsev, E. Fadeeva, R. Xing, G. Kuzmin, I. Lazichny, A. Panchenko, P. Nakov, T. Baldwin, M. Panov, and A. Shelmanov Unconditional truthfulness: learning unconditional uncertainty of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 35673–35694. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: Closed-weight model scope.. Verga et al. (2024) P. Verga, S. Hofstatter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis Replacing judges with juries: evaluating llm generations with a panel of diverse models. External Links: 2404.18796, Link Cited by: §2. Wei et al. (2024) J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus Measuring short-form factuality in large language models. External Links: 2411.04368, Link Cited by: §4.1. Xiong et al. (2024) M. Xiong, Z. Hu, X. Lu, Y. LI, J. Fu, J. He, and B. Hooi Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §A.4, §2, §3.2. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 2369–2380. External Links: Link, Document Cited by: §4.1. Zhang et al. (2024) C. Zhang, F. Liu, M. Basaldella, and N. Collier LUQ: long-text uncertainty quantification for LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 5244–5262. External Links: Link, Document Cited by: Appendix C, §2. Zhang et al. (2025) C. Zhang, R. Yang, Z. Zhang, X. Huang, S. Yang, D. Yu, and N. Collier Atomic calibration of LLMs in long-form generations. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, K. Inui, S. Sakti, H. Wang, D. F. Wong, P. Bhattacharyya, B. Banerjee, A. Ekbal, T. Chakraborty, and D. P. Singh (Eds.), Mumbai, India, p. 148–169. External Links: Link, Document, ISBN 979-8-89176-303-6 Cited by: Appendix C, Appendix C. Zhang* et al. (2020) T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, and Y. Artzi BERTScore: evaluating text generation with bert. In International Conference on Learning Representations, External Links: Link Cited by: §A.2, §2, §3.2. Appendix A Scorer Definitions We provide formal definitions of all UQ scorers used as ensemble inputs. All scorers are constructed to output values in [0, 1] by design, so no additional preprocessing or calibration is applied before combination. Let y denote the original response to prompt x, with tokenization t1,…,tL\t_1,…,t_L\ where L is the number of tokens and pjp_j is the probability assigned to token tjt_j. For sampling-based methods, ~=y~1,…,y~m y=\ y_1,…, y_m\ denotes m candidate responses generated from the same prompt using stochastic decoding. A.1 White-Box Single-Generation Scorers These scorers derive confidence from token-level probabilities of a single generation. Sequence Probability (SP). The joint probability of all tokens (Vashurin et al., 2025a): SP(y)=∏j=1LpjSP(y)= _j=1^Lp_j Length-Normalized Sequence Probability (LNSP). The geometric mean of token probabilities, correcting for sequence length (Malinin and Gales, 2021): LNSP(y)=(∏j=1Lpj)1/LLNSP(y)= ( _j=1^Lp_j )^1/L Minimum Token Probability (MTP). The minimum token probability across the response (Manakul et al., 2023): MTP(y)=minj∈1,…,LpjMTP(y)= _j∈\1,…,L\p_j The following scorers require access to the top-K logprobs per token. Let pj,1,…,pj,K\p_j,1,…,p_j,K\ denote the top-K token probabilities at position j, ordered by decreasing probability. Probability Margin (PM). The average gap between the top two token probabilities (Farr et al., 2025): PM(y)=1L∑j=1L(pj,1−pj,2)PM(y)= 1L _j=1^L(p_j,1-p_j,2) Average Token Negentropy (ATN@K). The mean normalized negentropy across token positions (Scalena et al., 2025; Manakul et al., 2023). Top-K token entropy at position j is TE@K(tj)=−∑k=1Kpj,klogpj,kTE@K(t_j)=- _k=1^Kp_j,k p_j,k. The negentropy transformation normalizes to [0,1][0,1]: TN@K(tj)=1−TE@K(tj)logKTN@K(t_j)=1- TE@K(t_j) K ATN@K(y)=1L∑j=1LTN@K(tj)ATN@K(y)= 1L _j=1^LTN@K(t_j) Minimum Token Negentropy (MTN@K). The minimum token negentropy across positions (Scalena et al., 2025; Manakul et al., 2023): MTN@K(y)=minj∈1,…,LTN@K(tj)MTN@K(y)= _j∈\1,…,L\TN@K(t_j) A.2 Black-Box Sampling-Based Scorers These scorers generate m candidate responses and measure consistency with the original response. All scorers in this subsection require only text outputs (no token probabilities). Exact Match Rate (EMR). The proportion of candidates identical to the original (Cole et al., 2023): EMR(y,~)=1m∑j=1m(y=y~j)EMR(y; y)= 1m _j=1^mI(y= y_j) Non-Contradiction Probability (NCP). The mean bidirectional non-contradiction probability from an NLI model (Chen and Mueller, 2024): NCP(y,~)=1−1m∑j=1mpc(y,y~j)+pc(y~j,y)2NCP(y; y)=1- 1m _j=1^m p_c(y, y_j)+p_c( y_j,y)2 where pc(⋅,⋅)p_c(·,·) denotes the NLI contradiction probability. We use microsoft/deberta-large-mnli for all NLI-based scorers. Entailment Probability (EP). The mean bidirectional entailment probability (Chen and Mueller, 2024; Lin et al., 2024): EP(y,~)=1m∑j=1mpe(y,y~j)+pe(y~j,y)2EP(y; y)= 1m _j=1^m p_e(y, y_j)+p_e( y_j,y)2 where pe(⋅,⋅)p_e(·,·) denotes the NLI entailment probability. BERTScore Consistency (BSC). The average F1 BERTScore between the original and each candidate (Zhang* et al., 2020; Manakul et al., 2023): BSC(y,~)=1m∑j=1mBertF1(y,y~j)BSC(y; y)= 1m _j=1^mBertF1(y, y_j) Normalized Cosine Similarity (NCS). The average cosine similarity using a sentence embedding model V:→ℝdV:Y ^d, normalized to [0,1][0,1] (Bouchard and Chauhan, 2025): NCS(y,~)=12+12m∑j=1mV(y)⋅V(y~j)‖V(y)‖⋅‖V(y~j)‖.NCS(y; y)= 12+ 12m _j=1^m V(y)· V( y_j)\|V(y)\|·\|V( y_j)\|. We use sentence-transformers/all-MiniLM-L6-v2 for natural language embeddings and jinaai/jina-embeddings-v2-base-code for code embeddings. Normalized Semantic Negentropy (NSN). Responses are clustered by mutual entailment via an NLI model. Semantic entropy is computed over the cluster distribution and normalized to a confidence score in [0,1][0,1] (Kuhn et al., 2023; Farquhar et al., 2024; Bouchard and Chauhan, 2025): SE(y;~)=−∑C∈P(C)logP(C)SE(y; y)=- _C P(C) P(C) NSN(y,~)=1−SE(y,~)log(m+1)NSN(y; y)=1- SE(y; y) (m+1) where C is the set of clusters over y∪~\y\∪ y and P(C)P(C) is the proportion of responses in cluster C. Semantic Sets Confidence (SSC). The number of unique semantic clusters |||C|, normalized to [0,1][0,1] (Lin et al., 2024): SSC(y,~)=m+1−||mSSC(y; y)= m+1-|C|m A.3 Hybrid Scorers These scorers combine token probabilities with sampling-based consistency signals. Monte Carlo Sequence Probability (MCSP). The average length-normalized sequence probability across all sampled responses (Kuhn et al., 2023): MCSP(~)=1m+1∑i=0mLNSP(yi)MCSP( y)= 1m+1 _i=0^mLNSP(y_i) where y0=y_0=y is the original response. Consistency and Confidence Approach (CoCoA). The product of the original response’s length-normalized sequence probability and its normalized cosine similarity with sampled responses (Vashurin et al., 2025b): CoCoA(y,~)=LNSP(y)⋅NCS(y,~)CoCoA(y; y)=LNSP(y)·NCS(y; y) A.4 Reflexive Scorers These scorers prompt the LLM to evaluate its own output. Both are applicable across all generation regimes (short-form, long-form, and code generation). P(True). The model is presented with a question-response concatenation and asked to classify it as “True” or “False.” Confidence is the token probability assigned to “True” (Kadavath et al., 2022): P(True)(y,x)=p(“True”∣x,y)P(True)(y;x)=p(``True′ x,y) Verbalized Confidence (VC). The model is prompted to express its confidence as a numerical score on a scale from 0 to 1, without requiring access to token probabilities (Tian et al., 2023). We implement a six-level likert scale that maps to numerical values 0,0.2,…,1.0\0,0.2,...,1.0\ (Xiong et al., 2024). A.5 Code-Specific Scorers For code generation, we adapt several sampling-based scorers by replacing NLI-based semantic equivalence with LLM-based functional equivalence assessment, which judges whether two code snippets produce identical outputs for all valid inputs. Functional Equivalence Rate (FER). The proportion of sampled responses judged functionally equivalent to the original (Bouchard et al., 2026a): FER(y;~)=1m∑j=1m[y≡y~j]FER(y; y)= 1m _j=1^mI[y≡ y_j] Functional Entropy (FE). A code-specific analogue of semantic entropy, using functional equivalence for clustering. Normalized to [0,1][0,1] (Bouchard et al., 2026a): FE(y;~)=−∑C∈P(C)logP(C)FE(y; y)=- _C P(C) P(C) NFN(y,~)=1−FE(y,~)log(m+1)NFN(y; y)=1- FE(y; y) (m+1) Functional Sets Confidence (FSC). The normalized count of unique functional clusters (Bouchard et al., 2026a): FSC(y,~)=m+1−||mFSC(y; y)= m+1-|C|m CodeBLEU Consistency (CBC). The average CodeBLEU score between the original and each candidate, capturing structural and syntactic similarity via n-gram match, syntax trees, and data-flow analysis (Ren et al., 2020; Sharma and David, 2025): CBC(y,~)=1m∑j=1mCodeBLEU(y,y~j)CBC(y; y)= 1m _j=1^mCodeBLEU(y, y_j) A.6 Long-Form Claim-Level Scorers For long-form generation, scorers operate at the claim level following a three-stage pipeline: (1) decompose the response into claims, (2) score each claim, and (3) aggregate to a response-level confidence. In this work, the ensemble operates at the claim level (stage 2), so we describe the claim-level scoring below. We employ graph-based scorers proposed by Jiang et al. (2024) and extended by Bouchard et al. (2026b), which decompose both original and sampled responses into claims, obtain the union of unique claims s across all responses, and construct a bipartite graph G with node set V=∪V=s , where an edge exists between claim s and response y if and only if s is entailed in y. Degree Centrality. The average entailment probability across responses: DC(s)=1m∑j=1mP(entail∣yj,s)DC(s)= 1m _j=1^mP(entail y_j,s) Betweenness Centrality. The proportion of shortest paths between node pairs passing through s, normalized by the maximum possible value BmaxB_max: BC(s)=1Bmax∑u≠v≠sσuv(s)σuvBC(s)= 1B_max _u≠ v≠ s _uv(s) _uv where σuv _uv is the number of shortest paths between u and v, and σuv(s) _uv(s) is the number passing through s. Closeness Centrality. The inverse sum of distances to all other nodes, normalized by the minimum possible distance: C(s)=m+2(||−1)∑v≠sdist(s,v)C(s)= m+2(|s|-1) _v≠ sdist(s,v) Harmonic Centrality. The sum of inverse distances, normalized by the maximum possible value Hmax=m+||−12H_max=m+ |s|-12: HC(s)=1Hmax∑v≠s1dist(s,v)HC(s)= 1H_max _v≠ s 1dist(s,v) Laplacian Centrality. The proportional drop in Laplacian energy from removing s: LC(s)=EL(G)−EL(G−s)EL(G)LC(s)= E_L(G)-E_L(G_-s)E_L(G) where EL(G)=∑iλi2E_L(G)= _i _i^2 and λi _i are the eigenvalues of G’s Laplacian matrix. PageRank. The stationary distribution probability of a random walk with restart probability (1−d)(1-d): PR(s)=1−d|V|+d∑v∈N(s)PR(v)|N(v)|PR(s)= 1-d|V|+d _v∈ N(s) PR(v)|N(v)| where N(s)N(s) is the set of neighbors of s. Appendix B Computational Cost Scorer Family Regime Orig. LLM Aux. LLM Sem. Comp. Single-gen. white-box SF, CG 0 0 0 Sampling-based SF, CG m 0 m–(m+12) m+12 Reflexive All 1 0 0 Graph-based LF m 2m+12m+1 m⋅Nclaimsm· N_claims Table 3: Per-instance computational cost by scorer family. SF = short-form, CG = code generation, LF = long-form. “Orig. LLM” = additional generations from the model under evaluation. “Aux. LLM” = generations from a separate model. “Sem. Comp.” = pairwise semantic comparisons. m = number of sampled responses; NclaimsN_claims = number of unique claims across sampled responses. Table 3 summarizes the per-instance computational cost of each scorer family beyond the initial response generation. Costs are expressed in terms of additional generations from the original LLM, auxiliary generations from a separate model (e.g., for claim decomposition or entailment grading), and semantic comparisons (e.g., NLI inference, embedding similarity, CodeBLEU, or LLM-based equivalence checks). Single-generation white-box scorers incur no cost beyond extracting token probabilities from the original forward pass. Sampling-based scorers require m additional generations and between m (for pairwise comparison against the original only) and (m+12) m+12 (for all-pairs clustering, as in semantic entropy) semantic comparisons. Reflexive scorers require one additional generation from the same LLM in which the model evaluates its own output. Graph-based scorers, used only in the long-form setting, are the most expensive: they require m sampled responses, 2m+12m+1 auxiliary LLM calls for claim decomposition (m+1m+1 responses decomposed into claims) and claim merging (m merge operations), and m⋅Nclaimsm· N_claims entailment checks to construct the claim-response bipartite graph, where NclaimsN_claims denotes the number of unique claims across sampled responses. The total cost of the ensemble for a given regime is the sum of costs across the applicable scorer families. For short-form and code generation, this is the combined cost of single-generation white-box, sampling-based, and reflexive scorers. For long-form, this is the combined cost of graph-based and reflexive scorers. Short-Form Code Long-Form LLM BigMath OpenR1 DROP Hotpot SimpleQA Python Rivers Mushrooms Gemini-2.5-Flash 0.94 0.89 0.84 0.93 0.31 0.87 0.50 0.55 Gemini-2.5-Pro 0.93 0.92 0.84 0.94 0.54 0.91 0.48 0.56 GPT-4o 0.40 0.25 0.78 0.93 0.27 0.56 0.54 0.61 GPT-4o-mini 0.45 0.24 0.75 0.89 0.08 0.52 0.47 0.52 Table 4: LLM accuracy across evaluation datasets. Accuracy is computed as the average over binary correctness labels, which serve as ground truth for evaluating uncertainty quantification methods. Appendix C Long-Form Scoring and Grading Dataset Construction. We construct two long-form QA datasets following the FactScore (Min et al., 2023) protocol. For each dataset, entities are drawn from Wikipedia: 84 edible mushroom species and 500 rivers. Each entity is paired with the prompt “Write a paragraph detailing some facts about entity,” where entity is the mushroom species or river name. Wikipedia articles are retrieved via the Wikipedia API to serve as reference texts. The full entity lists and reproducibility code are provided in the supplemental materials. Long-Form Scoring Pipeline. Long-form scoring operates as follows. First, each response is decomposed into atomic claims using the prompt template from Zhang et al. (2025). Second, the union of unique claims across all responses is obtained via sequential claim merging, following Jiang et al. (2024): each new claim is compared against the existing set and merged with a matching claim if one exists, or appended as a new entry otherwise. This produces a deduplicated claim set s over which the graph-based scorers (Section 3.2) operate. Graph-based scorers measure how consistently each claim is entailed across sampled responses via centrality metrics on the claim-response bipartite graph rather than scoring claims against the original query. Reflexive scorers, by contrast, are conditioned on the original question and evaluate each claim in that context. The ensemble operates at the claim level: classification and evaluation both use claim-level labels h(c)h(c), and no response-level aggregation is required. Long-Form (Claim-Level) Grading. Each claim is classified as objective or subjective following Zhang et al. (2024); only objective claims are retained for evaluation, as subjective claims cannot be definitively verified against a reference. Each retained objective claim is then graded against the entity’s complete Wikipedia article using the FactScore protocol (Min et al., 2023; Zhang et al., 2025; Zhang et al., 2024; Jiang et al., 2024), producing the binary labels h(c)∈0,1h(c)∈\0,1\ used for both training and evaluation. Gemini-2.5-Flash is used for claim decomposition, claim merging, objectivity classification, and grading, chosen for its strong performance at low cost. Appendix D Grading Validation To assess the reliability of LLM-based grading, two human annotators independently labeled a stratified sample of 400 short-form responses. For each of the five short-form datasets, we sampled 20 responses marked correct and 20 marked incorrect by the Gemini-2.5-Flash grader, across two generator LLMs (GPT-4o and Gemini-2.5-Flash), yielding 80 responses per dataset. Annotators compared each generated answer against the reference answer provided in the original dataset, without access to the grader’s label. Table 5 reports overall pairwise agreement. All three comparisons yield Cohen’s κ≥0.93κ≥ 0.93, indicating near-perfect agreement. Notably, the LLM grader agrees with each annotator at least as strongly as the annotators agree with each other (κ=0.97κ=0.97 and 0.930.93 vs. 0.950.95), confirming that grading noise introduced by the LLM is no larger than inherent human disagreement. Table 6 reports agreement broken down by dataset. Agreement is highest on math datasets (κ≥0.95κ≥ 0.95), where correctness is unambiguous, and lowest on HotpotQA (human κ=0.85κ=0.85). Even on HotpotQA, the LLM grader agrees with Annotator 1 more than the annotators agree with each other (κ=0.97κ=0.97 vs. 0.850.85), suggesting that disagreements stem from reference answer ambiguity rather than grader error. Table 7 reports agreement broken down by generator LLM to test whether the Gemini-2.5-Flash grader favors its own responses. Human-human agreement is identical across generators (κ=0.95κ=0.95), and LLM-human agreement is comparable (LLM vs. Annotator 1: κ=0.97κ=0.97 for Gemini-2.5-Flash vs. 0.980.98 for GPT-4o; LLM vs. Annotator 2: κ=0.92κ=0.92 vs. 0.930.93). These results provide no evidence of self-grading bias. Comparison % Agreement Cohen’s κ Annotator 1 vs. Annotator 2 97.5% 0.95 LLM Grader vs. Annotator 1 98.8% 0.97 LLM Grader vs. Annotator 2 96.2% 0.93 Table 5: Overall pairwise agreement for grading validation (n=400n=400). Human-Human LLM vs. Ann. 1 LLM vs. Ann. 2 Dataset n % κ % κ % κ BigMath 80 98.8% 0.97 98.8% 0.97 97.5% 0.95 DROP 80 97.5% 0.95 98.8% 0.97 96.2% 0.93 HotpotQA 80 92.5% 0.85 98.8% 0.97 91.2% 0.82 OpenR1 80 98.8% 0.97 100.0% 1.00 98.8% 0.97 SimpleQA 80 100.0% 1.00 97.5% 0.95 97.5% 0.95 Table 6: Agreement by dataset. Human-Human LLM vs. A1 LLM vs. A2 Orig. LLM n % κ % κ % κ Gem-Flash 200 97.5% 0.95 98.5% 0.97 96.0% 0.92 GPT-4o 200 97.5% 0.95 99.0% 0.98 96.5% 0.93 Table 7: Agreement by generator LLM, testing for self-grading bias. A1 and A2 respectively refer to the two annotators. Appendix E Hyperparameters All combination strategies use 5-fold cross-validation on the training fold for hyperparameter selection, optimizing AUROC. We use uqlm (Bouchard et al., 2026c) for the weighted average method and scikit-learn (Pedregosa et al., 2011) for the other three classifiers. Weighted average. Weights are constrained to [0,1][0,1] and sum to 1. We optimize AUROC using Optuna (Akiba et al., 2019) with 1,000 trials per configuration. Logistic regression. We use elastic net regularization (penalty=’elasticnet’, solver=’saga’) with the following grid: regularization strength C∈0.001,0.01,0.1,1,10,100C∈\0.001,0.01,0.1,1,10,100\ and ℓ1 _1 ratio ∈0,0.5,1∈\0,0.5,1\, yielding 18 configurations. Random forest. We search over: n_estimators ∈200,500∈\200,500\, max_features ∈sqrt,log2∈\sqrt,log2\, max_depth ∈4,6,8∈\4,6,8\, min_samples_split ∈2,5∈\2,5\, and min_samples_leaf ∈1,2∈\1,2\, yielding 96 configurations. Gradient boosted trees. We search over: n_estimators ∈50,100,200∈\50,100,200\, learning_rate ∈0.01,0.1,0.2∈\0.01,0.1,0.2\, max_depth ∈3,4,5∈\3,4,5\, min_samples_split ∈2,4∈\2,4\, and subsample ∈0.8,1.0∈\0.8,1.0\, yielding 108 configurations. Appendix F Supplemental Tables Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.74 0.68 0.69 0.76 0.61 0.69 Norm. sequence prob. 0.74 0.70 0.66 0.76 0.60 Min. token prob. 0.74 0.67 0.68 0.77 0.60 0.77 Probability margin 0.74 0.70 0.66 0.75 0.60 0.66 Mean token entropy 0.74 0.70 0.66 0.75 0.60 0.69 Min. token entropy 0.74 0.68 0.68 0.76 0.60 0.77 Consistency-based black-box Exact match rate 0.77 0.61 0.67 0.80 0.79 Non-contradiction prob. 0.77 0.67 0.66 0.80 0.81 Entailment prob. 0.80 0.70 0.67 0.81 0.82 BERTScore consistency 0.74 0.64 0.66 0.78 0.75 Semantic entropy 0.72 0.58 0.61 0.69 0.82 0.85 Semantic sets conf. 0.72 0.59 0.61 0.69 0.81 0.85 Cosine similarity 0.74 0.61 0.66 0.82 0.79 0.76 Equivalence rate 0.87 CodeBLEU consistency 0.80 Consistency-based white-box CoCoA 0.77 0.68 0.70 0.81 0.80 0.79 Monte Carlo prob. 0.77 0.75 0.66 0.77 0.68 0.71 WB semantic entropy 0.53 0.51 0.50 0.50 0.75 0.85 Semantic density 0.81 0.61 0.49 0.48 0.79 Reflexive Verbalized confidence 0.73 0.68 0.54 0.55 0.59 0.80 0.61 0.57 P(True) 0.86 0.86 0.69 0.71 0.64 0.84 0.60 0.56 Graph-based Degree centrality 0.70 0.67 Betweenness centrality 0.67 0.64 Closeness centrality 0.68 0.67 Harmonic centrality 0.68 0.66 Laplacian centrality 0.69 0.66 PageRank 0.69 0.67 Ensembles Ensemble (avg) 0.84 0.80 0.73 0.82 0.83 0.89 0.72 0.69 Logistic regression 0.83 0.85 0.72 0.82 0.83 0.89 0.72 0.70 Random forest 0.85 0.90 0.74 0.81 0.82 0.89 0.70 0.70 Gradient boosting 0.81 0.88 0.74 0.79 0.82 0.87 0.69 0.68 Table 8: Gemini-2.5-Flash AUROC performance of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.72 0.62 0.69 0.73 0.54 0.58 Norm. sequence prob. 0.71 0.59 0.66 0.73 0.54 Min. token prob. 0.72 0.61 0.69 0.73 0.54 0.76 Probability margin 0.71 0.59 0.66 0.72 0.54 0.55 Mean token entropy 0.71 0.59 0.65 0.73 0.54 0.58 Min. token entropy 0.72 0.61 0.69 0.74 0.54 0.77 Consistency-based black-box Exact match rate 0.77 0.77 0.65 0.76 0.81 Non-contradiction prob. 0.77 0.83 0.63 0.75 0.84 Entailment prob. 0.77 0.81 0.64 0.76 0.85 BERTScore consistency 0.78 0.72 0.65 0.77 0.76 Semantic entropy 0.74 0.78 0.57 0.66 0.83 0.82 Semantic sets conf. 0.74 0.78 0.57 0.66 0.83 0.82 Cosine similarity 0.76 0.73 0.64 0.76 0.81 0.80 Equivalence rate 0.82 CodeBLEU consistency 0.80 Consistency-based white-box CoCoA 0.82 0.72 0.66 0.81 0.82 0.79 Monte Carlo prob. 0.75 0.66 0.68 0.77 0.55 0.60 WB semantic entropy 0.52 0.50 0.50 0.50 0.76 0.82 Semantic density 0.79 0.73 0.51 0.48 0.84 Reflexive Verbalized confidence 0.72 0.68 0.51 0.58 0.59 0.72 0.55 0.54 P(True) 0.81 0.79 0.57 0.69 0.74 0.84 0.59 0.56 Graph-based Degree centrality 0.66 0.67 Betweenness centrality 0.63 0.63 Closeness centrality 0.63 0.66 Harmonic centrality 0.63 0.65 Laplacian centrality 0.64 0.66 PageRank 0.64 0.66 Ensembles Ensemble (avg) 0.88 0.86 0.69 0.84 0.86 0.85 0.67 0.68 Logistic regression 0.87 0.90 0.69 0.82 0.86 0.85 0.67 0.69 Random forest 0.88 0.90 0.74 0.84 0.86 0.85 0.66 0.69 Gradient boosting 0.87 0.89 0.72 0.84 0.86 0.84 0.65 0.67 Table 9: Gemini-2.5-Pro AUROC performance of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.82 0.81 0.69 0.71 0.81 0.73 Norm. sequence prob. 0.84 0.81 0.62 0.71 0.83 Min. token prob. 0.82 0.81 0.67 0.70 0.80 0.71 Probability margin 0.82 0.76 0.59 0.71 0.81 0.75 Mean token entropy 0.84 0.80 0.61 0.73 0.84 0.75 Min. token entropy 0.82 0.80 0.70 0.73 0.82 0.76 Consistency-based black-box Exact match rate 0.78 0.78 0.69 0.67 0.78 Non-contradiction prob. 0.79 0.78 0.71 0.70 0.83 Entailment prob. 0.79 0.78 0.71 0.70 0.83 BERTScore consistency 0.78 0.73 0.63 0.68 0.72 Semantic entropy 0.80 0.78 0.67 0.67 0.84 0.85 Semantic sets conf. 0.80 0.78 0.67 0.67 0.84 0.85 Cosine similarity 0.78 0.77 0.59 0.69 0.78 0.72 Equivalence rate 0.86 CodeBLEU consistency 0.74 Consistency-based white-box CoCoA 0.84 0.81 0.61 0.72 0.83 0.74 Monte Carlo prob. 0.83 0.81 0.64 0.74 0.83 0.82 WB semantic entropy 0.50 0.51 0.50 0.50 0.79 0.78 Semantic density 0.74 0.73 0.61 0.52 0.83 Reflexive Verbalized confidence 0.59 0.58 0.58 0.61 0.69 0.68 0.61 0.56 P(True) 0.68 0.64 0.70 0.70 0.78 0.82 0.70 0.65 Graph-based Degree centrality 0.72 0.67 Betweenness centrality 0.70 0.64 Closeness centrality 0.72 0.67 Harmonic centrality 0.72 0.66 Laplacian centrality 0.72 0.64 PageRank 0.71 0.64 Ensembles Ensemble (avg) 0.84 0.81 0.76 0.75 0.87 0.88 0.74 0.68 Logistic regression 0.84 0.81 0.77 0.77 0.87 0.88 0.74 0.68 Random forest 0.85 0.82 0.77 0.74 0.87 0.88 0.74 0.71 Gradient boosting 0.85 0.81 0.75 0.69 0.86 0.87 0.74 0.70 Table 10: GPT-4o AUROC performance of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.53 0.66 0.63 0.74 0.74 0.76 Norm. sequence prob. 0.79 0.71 0.60 0.73 0.69 Min. token prob. 0.55 0.67 0.62 0.73 0.71 0.74 Probability margin 0.82 0.68 0.59 0.74 0.66 0.75 Mean token entropy 0.85 0.71 0.60 0.75 0.72 0.78 Min. token entropy 0.59 0.69 0.63 0.74 0.78 0.78 Consistency-based black-box Exact match rate 0.53 0.65 0.60 0.74 0.73 Non-contradiction prob. 0.74 0.72 0.64 0.73 0.75 Entailment prob. 0.72 0.72 0.64 0.75 0.76 BERTScore consistency 0.52 0.63 0.57 0.71 0.66 Semantic entropy 0.74 0.74 0.64 0.65 0.77 0.86 Semantic sets conf. 0.74 0.74 0.64 0.65 0.78 0.85 Cosine similarity 0.62 0.69 0.55 0.70 0.69 0.74 Equivalence rate 0.86 CodeBLEU consistency 0.76 Consistency-based white-box CoCoA 0.77 0.72 0.58 0.73 0.70 0.77 Monte Carlo prob. 0.76 0.73 0.58 0.76 0.74 0.81 WB semantic entropy 0.54 0.54 0.50 0.50 0.71 0.85 Semantic density 0.60 0.64 0.59 0.50 0.71 Reflexive Verbalized confidence 0.70 0.61 0.59 0.63 0.57 0.72 0.60 0.57 P(True) 0.78 0.69 0.65 0.69 0.65 0.76 0.66 0.59 Graph-based Degree centrality 0.71 0.66 Betweenness centrality 0.68 0.64 Closeness centrality 0.71 0.66 Harmonic centrality 0.71 0.65 Laplacian centrality 0.70 0.64 PageRank 0.71 0.64 Ensembles Ensemble (avg) 0.85 0.77 0.67 0.78 0.78 0.88 0.72 0.67 Logistic regression 0.87 0.77 0.67 0.78 0.78 0.88 0.72 0.68 Random forest 0.88 0.78 0.70 0.78 0.77 0.88 0.71 0.69 Gradient boosting 0.88 0.77 0.69 0.77 0.76 0.88 0.70 0.66 Table 11: GPT-4o-mini AUROC performance of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.21 0.58 0.14 0.07 0.55 0.05 Norm. sequence prob. 0.04 0.04 0.15 0.07 0.62 Min. token prob. 0.21 0.56 0.14 0.07 0.56 0.73 Probability margin 0.05 0.05 0.15 0.07 0.62 0.06 Mean token entropy 0.05 0.07 0.15 0.06 0.64 0.09 Min. token entropy 0.15 0.44 0.13 0.05 0.58 0.53 Consistency-based black-box Exact match rate 0.22 0.58 0.12 0.07 0.14 Non-contradiction prob. 0.03 0.04 0.14 0.06 0.23 Entailment prob. 0.07 0.14 0.11 0.04 0.18 BERTScore consistency 0.02 0.02 0.14 0.05 0.63 Semantic entropy 0.08 0.13 0.13 0.05 0.19 0.08 Semantic sets conf. 0.06 0.11 0.14 0.05 0.21 0.04 Cosine similarity 0.03 0.05 0.13 0.05 0.56 0.05 Equivalence rate 0.11 CodeBLEU consistency 0.43 Consistency-based white-box CoCoA 0.05 0.07 0.12 0.05 0.49 0.07 Monte Carlo prob. 0.03 0.04 0.14 0.05 0.62 0.05 WB semantic entropy 0.06 0.10 0.16 0.07 0.32 0.08 Semantic density 0.03 0.04 0.14 0.04 0.29 Reflexive Verbalized confidence 0.04 0.06 0.16 0.08 0.54 0.07 0.37 0.45 P(True) 0.05 0.07 0.17 0.07 0.52 0.11 0.42 0.47 Graph-based Degree centrality 0.14 0.18 Betweenness centrality 0.54 0.49 Closeness centrality 0.34 0.36 Harmonic centrality 0.35 0.38 Laplacian centrality 0.52 0.46 PageRank 0.54 0.49 Ensembles Ensemble (avg) 0.03 0.10 0.12 0.04 0.42 0.11 0.13 0.27 Logistic regression 0.02 0.03 0.04 0.02 0.08 0.04 0.05 0.03 Random forest 0.02 0.03 0.04 0.02 0.05 0.03 0.06 0.03 Gradient boosting 0.03 0.06 0.04 0.04 0.08 0.06 0.07 0.07 Table 12: Gemini-2.5-Flash ECE of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.08 0.16 0.15 0.07 0.46 0.02 Norm. sequence prob. 0.07 0.07 0.15 0.06 0.46 Min. token prob. 0.08 0.16 0.15 0.06 0.46 0.82 Probability margin 0.07 0.08 0.15 0.06 0.46 0.02 Mean token entropy 0.07 0.07 0.15 0.06 0.46 0.05 Min. token entropy 0.07 0.13 0.14 0.05 0.46 0.54 Consistency-based black-box Exact match rate 0.05 0.12 0.12 0.08 0.11 Non-contradiction prob. 0.04 0.04 0.15 0.05 0.20 Entailment prob. 0.04 0.06 0.13 0.04 0.16 BERTScore consistency 0.06 0.07 0.14 0.04 0.42 Semantic entropy 0.04 0.06 0.14 0.05 0.17 0.15 Semantic sets conf. 0.04 0.04 0.14 0.05 0.17 0.08 Cosine similarity 0.05 0.05 0.13 0.04 0.37 0.04 Equivalence rate 0.20 CodeBLEU consistency 0.47 Consistency-based white-box CoCoA 0.05 0.07 0.13 0.05 0.37 0.08 Monte Carlo prob. 0.06 0.06 0.15 0.05 0.46 0.02 WB semantic entropy 0.07 0.08 0.16 0.06 0.25 0.15 Semantic density 0.05 0.06 0.13 0.03 0.22 Reflexive Verbalized confidence 0.05 0.07 0.16 0.07 0.38 0.06 0.41 0.48 P(True) 0.04 0.06 0.17 0.07 0.34 0.08 0.41 0.49 Graph-based Degree centrality 0.19 0.20 Betweenness centrality 0.56 0.47 Closeness centrality 0.34 0.39 Harmonic centrality 0.35 0.40 Laplacian centrality 0.53 0.44 PageRank 0.55 0.47 Ensembles Ensemble (avg) 0.04 0.04 0.13 0.04 0.31 0.19 0.16 0.18 Logistic regression 0.03 0.03 0.03 0.01 0.05 0.03 0.06 0.03 Random forest 0.02 0.03 0.04 0.02 0.06 0.03 0.06 0.03 Gradient boosting 0.04 0.05 0.05 0.03 0.13 0.04 0.08 0.06 Table 13: Gemini-2.5-Pro ECE of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.19 0.17 0.34 0.21 0.14 0.22 Norm. sequence prob. 0.20 0.19 0.13 0.05 0.34 Min. token prob. 0.19 0.17 0.29 0.19 0.14 0.50 Probability margin 0.18 0.19 0.13 0.04 0.43 0.32 Mean token entropy 0.26 0.30 0.15 0.03 0.50 0.36 Min. token entropy 0.25 0.26 0.08 0.09 0.25 0.25 Consistency-based black-box Exact match rate 0.24 0.25 0.33 0.22 0.13 Non-contradiction prob. 0.26 0.29 0.17 0.06 0.29 Entailment prob. 0.24 0.26 0.07 0.08 0.23 BERTScore consistency 0.58 0.72 0.17 0.04 0.66 Semantic entropy 0.24 0.26 0.16 0.05 0.24 0.06 Semantic sets conf. 0.26 0.28 0.17 0.05 0.24 0.10 Cosine similarity 0.52 0.65 0.15 0.04 0.58 0.36 Equivalence rate 0.13 CodeBLEU consistency 0.11 Consistency-based white-box CoCoA 0.17 0.17 0.12 0.08 0.27 0.17 Monte Carlo prob. 0.18 0.19 0.08 0.03 0.35 0.11 WB semantic entropy 0.60 0.73 0.22 0.07 0.41 0.11 Semantic density 0.56 0.68 0.18 0.04 0.29 Reflexive Verbalized confidence 0.52 0.58 0.19 0.06 0.49 0.30 0.31 0.40 P(True) 0.20 0.18 0.17 0.16 0.18 0.30 0.26 0.37 Graph-based Degree centrality 0.13 0.16 Betweenness centrality 0.60 0.53 Closeness centrality 0.23 0.31 Harmonic centrality 0.24 0.32 Laplacian centrality 0.57 0.50 PageRank 0.60 0.53 Ensembles Ensemble (avg) 0.30 0.31 0.08 0.03 0.32 0.17 0.11 0.17 Logistic regression 0.10 0.08 0.04 0.02 0.05 0.05 0.07 0.03 Random forest 0.06 0.05 0.04 0.02 0.05 0.05 0.05 0.03 Gradient boosting 0.07 0.08 0.07 0.04 0.08 0.08 0.06 0.05 Table 14: GPT-4o ECE of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains. Short-form QA Code Long-form Scorer BigMath OpenR1 DROP Hotpot SimpleQA LiveCodeBench Mushroom-Fact River-Fact Single-generation white-box Sequence prob. 0.35 0.21 0.30 0.24 0.15 0.35 Norm. sequence prob. 0.23 0.27 0.17 0.06 0.45 Min. token prob. 0.35 0.21 0.25 0.20 0.19 0.39 Probability margin 0.25 0.33 0.17 0.06 0.59 0.41 Mean token entropy 0.31 0.42 0.19 0.08 0.67 0.44 Min. token entropy 0.26 0.26 0.07 0.08 0.31 0.09 Consistency-based black-box Exact match rate 0.36 0.26 0.28 0.22 0.19 Non-contradiction prob. 0.33 0.39 0.21 0.09 0.34 Entailment prob. 0.24 0.30 0.11 0.07 0.27 BERTScore consistency 0.49 0.69 0.21 0.09 0.85 Semantic entropy 0.27 0.32 0.20 0.09 0.29 0.07 Semantic sets conf. 0.29 0.33 0.20 0.09 0.28 0.12 Cosine similarity 0.44 0.62 0.20 0.07 0.77 0.43 Equivalence rate 0.11 CodeBLEU consistency 0.09 Consistency-based white-box CoCoA 0.18 0.23 0.15 0.05 0.39 0.31 Monte Carlo prob. 0.19 0.23 0.14 0.03 0.45 0.32 WB semantic entropy 0.51 0.70 0.25 0.11 0.48 0.07 Semantic density 0.46 0.63 0.21 0.08 0.36 Reflexive Verbalized confidence 0.23 0.35 0.12 0.10 0.59 0.28 0.26 0.33 P(True) 0.25 0.27 0.25 0.15 0.26 0.35 0.43 0.48 Graph-based Degree centrality 0.12 0.18 Betweenness centrality 0.50 0.46 Closeness centrality 0.32 0.38 Harmonic centrality 0.35 0.40 Laplacian centrality 0.47 0.43 PageRank 0.50 0.46 Ensembles Ensemble (avg) 0.25 0.33 0.13 0.04 0.41 0.21 0.17 0.23 Logistic regression 0.08 0.05 0.05 0.04 0.03 0.07 0.06 0.03 Random forest 0.06 0.05 0.05 0.03 0.02 0.06 0.06 0.03 Gradient boosting 0.10 0.06 0.11 0.04 0.03 0.10 0.06 0.07 Table 15: GPT-4o-mini ECE of uncertainty quantification methods across nine benchmarks spanning short-form QA, code generation, and long-form generation domains.