Paper deep dive

Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities

Anita Yang, Krikamol Muandet, Michele Caprio, Siu Lun Chau, Masaki Adachi

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 69

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/13/2026, 1:10:18 AM

Summary

The paper introduces a novel framework for eliciting higher-order uncertainty from Large Language Models (LLMs) using imprecise probabilities. By distinguishing between first-order uncertainty (aleatoric, inherent randomness) and second-order uncertainty (epistemic, uncertainty about the model's own probability estimates), the authors propose prompt-based techniques like 'DeFinetti' and 'ProbInt' to improve uncertainty reporting, address systematic failure modes in ambiguous question-answering and in-context learning, and provide more coherent decision-making foundations.

Entities (6)

Imprecise Probabilities · framework · 98%First-order uncertainty · concept · 95%Second-order uncertainty · concept · 95%DeFinetti · prompting-technique · 92%ProbInt · prompting-technique · 92%Maximum Mean Imprecision · metric · 90%

Relation Signals (4)

Imprecise Probabilities → represents → Higher-order uncertainty

confidence 95% · a principled framework for repesenting and eliciting higher-order uncertainty

DeFinetti → elicits → First-order uncertainty

confidence 90% · We introduce general-purpose prompting and post-processing procedures to directly elicit and quantify both orders of uncertainty

ProbInt → elicits → Second-order uncertainty

confidence 90% · We name this as PROBINT... elicit second-order uncertainty via IP representations

MMI → quantifies → Second-order uncertainty

confidence 90% · replacing entropy with the maximum mean imprecision (MMI) metric

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Despite the growing demand for eliciting uncertainty from large language models (LLMs), empirical evidence suggests that LLM behavior is not always adequately captured by the elicitation techniques developed under the classical probabilistic uncertainty framework. This mismatch leads to systematic failure modes, particularly in settings that involve ambiguous question-answering, in-context learning, and self-reflection. To address this, we propose novel prompt-based uncertainty elicitation techniques grounded in \emph{imprecise probabilities}, a principled framework for repesenting and eliciting higher-order uncertainty. Here, first-order uncertainty captures uncertainty over possible responses to a prompt, while second-order uncertainty (uncertainty about uncertainty) quantifies indeterminacy in the underlying probability model itself. We introduce general-purpose prompting and post-processing procedures to directly elicit and quantify both orders of uncertainty, and demonstrate their effectiveness across diverse settings. Our approach enables more faithful uncertainty reporting from LLMs, improving credibility and supporting downstream decision-making.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

69,104 characters extracted from source content.

Expand or collapse full text

Verbalizing LLM’s Higher-order Uncertainty via Imprecise Probabilities Anita Yang 1,2 Krikamol Muandet 3 Michele Caprio 4,5 Siu Lun Chau 6,* Masaki Adachi 1,* 1 Lattice Lab, Toyota Motor Corporation, Japan 2 Department of Computer Science, University of Tokyo, Japan 3 Rational Intelligence Lab, CISPA, Helmholtz Center for Information Security, Germany 4 Department of Computer Science, University of Manchester, United Kingdom 5 Manchester Centre for AI Fundamentals, United Kingdom 6 EPIC Lab, College of Computing & Data Science, Nanyang Technological University, Singapore * Equal contribution Abstract Despite the growing demand for eliciting uncer- tainty from large language models (LLMs), empir- ical evidence suggests that LLM behavior is not always adequately captured by the elicitation tech- niques developed under the classical probabilistic uncertainty framework. This mismatch leads to sys- tematic failure modes, particularly in settings that involve ambiguous question-answering, in-context learning, and self-reflection. To address this, we propose novel prompt-based uncertainty elicitation techniques grounded in imprecise probabilities, a principled framework for repesenting and eliciting higher-order uncertainty. Here, first-order uncer- tainty captures uncertainty over possible responses to a prompt, while second-order uncertainty (un- certainty about uncertainty) quantifies indetermi- nacy in the underlying probability model itself. We introduce general-purpose prompting and post- processing procedures to directly elicit and quan- tify both orders of uncertainty, and demonstrate their effectiveness across diverse settings. Our ap- proach enables more faithful uncertainty reporting from LLMs, improving credibility and supporting downstream decision-making. 1 INTRODUCTION Uncertainty quantification (UQ) for large language mod- els (LLMs) [Shorinwa et al., 2025] has been proven effec- tive across many downstream tasks, including hallucination detection [Bouchard and Chauhan, 2025, Farquhar et al., 2024, Tomani et al., 2024], reasoning enhancement [Lu- goloobi et al., 2026], active learning [Wang, 2024], model selection [Agrawal et al., 2025], and agentic workflow con- trol [Tomani et al., 2024, Machcha et al., 2025]. Because most state-of-the-art LLMs are closed-source, research on prediction error vanilla uncertainty 80 020 40 60 # in-context examples Which country hosted the 2019 Cricket World Cup? A: EnglandB: Wales C: United KingdomD: India Context: 1. Correct answer: England and Wales (co-hosts) 2. UK includes England and Wales (constituent countries). 3. Indiahosted in 2023 (not 2019). Uncertainty Score Uncertainty Score Density ambiguous clear question (b) (c) D A B C Probability of correct answer (a) (d) LLM selected Probability suggests Figure 1: Collection of failure modes in prior verbalized uncertainty scores. (a) Example of an ambiguous question. (b) The prior uncertainty score fails to distinguish between clear and ambiguous question distributions. (c) The prior score also fails to track the decrease in prediction error, which should reflect reduced uncertainty as more in-context examples are provided. (d) Self-reflection on answer-wise probability/uncertainty should explain the rationale behind answer selection, but it often fails to do so. uncertainty elicitation has largely focused on verbalized un- certainty [Tian et al., 2023]: directly prompting the model to report its confidence, e.g., “I am 80% confident that this answer is correct.” We refer to this approach as vanilla un- certainty elicitation. Under well-controlled settings, vanilla performs reasonably well; however, several failures have been reported in practically important scenarios. The first challenge arises in ambiguous question-answering scenarios, as illustrated in Figure 1(a), where prompt under- specification admits multiple answers that may be simultane- ously valid under different interpretations [Min et al., 2020, Yang et al., 2025]. In such cases, the degree of ambiguity should be faithfully reflected in the model’s verbalised uncer- tainty estimates. However, vanilla confidence measures and arXiv:2603.10396v1 [cs.AI] 11 Mar 2026 prediction error Uncertainty Score Density ambiguous clear question (c) D A B C Probability of correct answer (e) LLM selected Ours 0 0.5 1 Belief that the answer is correct England Wales first-order uncertainty second-order uncertainty Model A Model B Uncertainty score # in-context examples 80 0 20 4060 ProbInt (Ours) (d) 0 0.5 1 Belief that the answer is correct England Wales (b)(a) Figure 2: Our imprecise probabilities–based approach. (a) Classical precise probability provides point estimates. (b) Imprecise probabilities instead represents uncertainty as intervals. (c) This enables more reasonable elicitation of question ambiguity. (d) It more closely tracks predictive error. (e) It aligns the LLM’s answer selection with the selection implied by its imprecise probability estimates. existing approaches often fail to reliably differentiate these situations; see Figure 1(b). The second challenge arises in in-context learning (ICL; Brown et al. [2020]), where task-specific examples are incorporated into the prompt without parameter updates. ICL is often interpreted as a form of implicit meta-learning [Von Oswald et al., 2023, Wu et al., 2025], where the model infers the underlying task from the provided examples. As predictive performance of- ten improves with additional examples, ICL is frequently described through a Bayesian epistemic uncertainty reduc- tion [Xie et al., 2022, Wakayama and Suzuki, 2025]. Fig- ure 1(c) illustrates its failure mode: as more in-context ex- amples are provided, the prediction error decreases, yet the uncertainty remains high and flat. This misalignment echoes Falck et al. [2024], who argue from a martingale viewpoint that ICL is not strictly Bayesian. The third challenge arises in self-reflection settings, where an LLM is prompted to select an answer from candidate options and subsequently reflect on its choice. Under a Bayesian-rationality, action selection should follow the maximization of expected utility as prescribed by Bayesian decision theory [Harsanyi, 1978]. However, the utilities implicitly induced by elicited uncer- tainty scores often fail to account for the model’s observed decisions; see Figure 1(d). This is consistent with Liu et al. [2025], Yamin et al. [2026], who rejects Bayesian rationality of LLMs through conditional independence testing. These failures may stem not from an LLM’s inability to express uncertainty, but from the representation we im- pose. Prior work implicitly assumes that uncertainty can be fully captured by a single, precise probability that ra- tionally aggregates all sources into one value—and that such a score can be faithfully elicited. Instead, we ask: what if we allow LLMs to express uncertainty about un- certainty—introducing imprecision into the uncertainty rep- resentation itself and building our foundation upon it? This is precisely the core of Imprecise Probabilities (IP) [Wal- ley, 1991, Augustin et al., 2014], a classical framework that enables the representation of higher-order uncertainty. IP provides a principled foundation for understanding ver- balized uncertainty in LLMs: As illustrated in Figure 2(a), conventional uncertainty scores reflect first-order uncer- tainty: they provide point estimates that capture variability over outcomes (often associated with aleatoric uncertainty). In contrast, IP represents second-order uncertainty through probability intervals, where any value within the interval is admissible. The interval widths generally quantify impre- cision in the uncertainty estimate—uncertainty about un- certainty—commonly interpreted as epistemic uncertainty 1 . Through this lens, question ambiguity can be viewed as first-order uncertainty that the model cannot reduce. In con- trast, ICL provides additional examples that help reduce second-order uncertainty—uncertainty that the model can resolve to improve its predictions. Self-reflection can then be interpreted as decision-making under IP, typically adopt- ing the argmax of the lower probability, i.e., the maximin rule [Robbins, 1951]. By leveraging the corresponding IP tools, the resulting uncertainty scores become substantially more coherent across tasks; see Figures 2(c)–(e). While conceptually appealing, the adoption of IP frame- works for LLMs is far from straightforward. We present the first concrete instantiation of an IP-based approach for ver- balized uncertainty in LLMs, introducing general-purpose prompting strategies and post-processing procedures de- signed to extract higher-order uncertainty from model out- puts. Empirically, we show that the proposed framework not only yields a more coherent and rich uncertainty representa- tions but also improves performance over existing methods while simultaneously incurring low API costs. 2 BACKGROUND We first review the background on LLM uncertainty quantifi- cation setting and imprecise probability, and then introduce our method for eliciting IPs from LLMs. 2.1 LLM UNCERTAINTY QUANTIFICATION Shared dataset structure. Assume the question-answer pair(x,Y ⋆ ), wherexis a question prompt andY ⋆ ⊆Yis the set of ground-truth correct answers, andYis candidate answers. We use y ∈Y to denote a candidate answer. Tasks. Given questionx, the task is to estimate the ground- truth answery ⋆ ∈ Y ⋆ . When the candidate setYis not 1 As the terminology surrounding aleatoric/epistemic for LLMs remains under active debate, we adopt the terms first-/second-order uncertainty throughout for consistency [Kirchhof et al., 2025b]. provided in the prompt, we refer to this as open-ended. Language model. Let a pretrained LLM serve two roles:(i) Predictor:ˆy = arg max y∈Y ˆp(y | x), producing a single answer estimate via the predictive distributionˆp(y | x); (i) Generator: ˆ Y ∼ ˆp(y | x) , generating multiple candidate an- swers conditioned on the questionxwhenYis not provided (prompt in Appendix A.2). First-/second-order uncertainties. We define first-order un- certainty as intrinsic randomness arising from the question that the model cannot reduce. This system-level randomness occurs in cases such as|Y ⋆ | > 1. Prior works [Min et al., 2020, Yang et al., 2025] consider the case|Y ⋆ | > 1. All remaining uncertainty is attributed to second-order uncer- tainty. Equivalently, ifxcontains sufficient information to uniquely determine the correct answer (i.e.,|Y ⋆ | = 1), then first-order uncertainty is absent. 2.2 IMPRECISE PROBABILITIES Verbalized uncertainty. Our IP approaches elicit the model’s beliefs by prompting the LLM to verbalize numeri- cal uncertainty judgments (e.g. Tian et al. [2023]), rather than estimating uncertainty via repeated sampling from the model’s outputs [Farquhar et al., 2024]. Compared to Bayesian. The Bayesian approach requires a prior distribution over the parameter space, and the dis- tinction between aleatoric and epistemic uncertainty relies on an explicit likelihood specification. In contrast, IP treats any single-valued uncertainty score as first-order; a perfect estimator would coincide with pure aleatoric uncertainty. In practice, however, such perfection is unlikely, and most systems therefore exhibit residual second-order uncertainty, naturally represented within the IP framework. Ignorance vs indifference. Figure 2(b) contrasts two mod- els, A and B. Model A exhibits wide probability intervals, reflecting ignorance—a lack of knowledge about which outcomes are plausible. Model B, in contrast, produces nar- rower intervals, indicating greater precision while still al- lowing for a degree of imprecision. By comparison, the first-order uncertainty representation in Figure 2(a) can only suggest that England and Wales are equally likely, without conveying any information about the underlying source of uncertainty. In particular, it cannot distinguish whether the model assigns equal probabilities due to genuine indiffer- ence (both outcomes considered equally plausible) or due to ignorance (insufficient information to prefer one over the other). Imprecise probability representations explicitly cap- ture this distinction, separating ignorance from indifference. Probability intervals. IP provides rich representations to construct the probability interval [Keynes, 1921]. The sim- plest approach is to directly prompt the model to report an interval p (y),p(y) for each candidate answery, where p(y)andp(y)denote the lower and upper probabilities. Intuitively,p(y)captures the probability that is certainly justified by the evidence, whilep(y)captures what is pos- sibly defensible [Augustin et al., 2014]. Requesting such intervals tests whether the model can recognize and articu- late the bounds of its belief; values between these bounds correspond to its personal “medial” probability for the event [Smith, 1961]. Credal sets. Another approach is to elicit uncertainty from a group of models, interpreting disagreement among them as imprecision. Each model is asked to report its first-order uncertainty for a given answery. The collection of these reported probabilities forms a credal set [Levi, 1980], and the minimum and maximum values across the set define the corresponding probability interval. Possibility. The last alternative is via exclusion: it evaluates whether the model can confidently rule out a candidate an- sweryby assessing the possibility of alternative answers, which is called a possibility functionπ(y)∈ [0, 1][Dubois and Prade, 1985]. Possibility functions, unlike probabilities, are non-additive and are normalized only by requiring that at least one output is fully plausible (i.e.sup y π(y) = 1). This supports relative plausibility comparisons without allocat- ing a fixed mass across candidates—making elicitation less sensitive to systematic miscalibration and overconfidence in model self-reports [Wang and Stengel-Eskin, 2025, Xiong et al., 2024]—and lets us elicitπfor “none of the above” without changing the plausibility assigned to other outputs. 3 UNCERTAINTY ELICITATION VIA IMPRECISE PROBABILITIES We now introduce our methods to elicit higher-order un- certainties from LLMs via prompting and post-processing strategies. We introduce the specific prompting techniques for both first- and second-order uncertainties. 3.1 FIRST-ORDER UNCERTAINTY Under IP, any point-valued score (including the vanilla) is considered first-order; however, it can be refined to better satisfy the probability axioms (non-negativity, additivity, normalization) [Kolmogorov, 1933]. We draw on Bruno de Finetti’s classical interpretation of probability as coherent betting behavior [De Finetti, 1937]. Under this view, a prob- ability corresponds to the fair price at which an agent is will- ing to buy or sell a gamble. Rational betting prices must sat- isfy the probability axioms, as violations would expose the agent to a sure loss (i.e., a Dutch book). This betting-based interpretation is directly implementable through Prompt 1 and Alg. 1. The probability-axiom verifier algorithmically enforces compliance with the axioms. Given elicited bet- ting pricesp k ∈ ˆpover mutually exclusive and exhaustive candidate answers, coherence reduces to verifying: (i) non- negativity,p k ≥ 0, and (i) normalization, P |Y| k=1 p k = 1 . Under this assumption, additivity follows automatically. Prompt 1: DeFinetti Assign a buy price (between $0.00 and $1.00) for each answer representing the maximum amount you would pay for a bet on that answer being correct. If an answer is correct, the bet pays $1.00; if incorrect, it pays $0.00, and the price paid is lost. Assign prices that maximize expected profit, taking into account how each answer might be correct or incorrect under reasonable alterna- tive interpretations of the question (e.g., unclear enti- ties, ambiguous events, or uncertainty about required answer format or type), and how multiple answer op- tions can be equally correct. The prices must sum to exactly $1.00 across all answers. Algorithm 1 DeFinetti 1: Input: input x, candidate answersY , verifier C axiom 2: Init: ˆp(y | x)← 0 for all y ∈Y . 3: while C axiom [ˆp(Y | x)] is False do 4: ˆp(Y | x)← DeFinettiBet(x,Y). 5: return: Entropy H(ˆp) 3.2 SECOND-ORDER UNCERTAINTY Next, we elicit second-order uncertainty via IP representa- tions. As shown in Alg. 2, the overall framework remains unchanged; the key differences are: (i) replacing point esti- matesˆpwith probability intervals[p,p], and (i) replacing entropy with the maximum mean imprecision (MMI) metric [Chau et al., 2025]. We first describe how to elicit probabil- ity intervals, and then introduce the MMI metric. Representations. As discussed in §2.2, IP provides three principal representations of uncertainty: probability inter- vals, credal sets, and possibility measures. A natural ques- tion is which representation to adopt. In general, these cap- ture different aspects of uncertainty, and no single repre- sentation is universally superior. We adopt the following perspective: (i) probability intervals serve as the most basic and widely applicable representation; (i) credal sets are particularly suitable when using an ensemble of LLMs, and otherwise remain optional; and (i) possibility measures are useful when the candidate setYmay be incomplete (e.g., in open-ended Q&A settings), and are otherwise optional. Probability interval. Following Alg. 2, probability intervals can be obtained by replacing the IP-Prompt in Line 4 with Prompt 2. The verifierC verifier checks if (i) the lower proba- bilities satisfy P y p(y)≤ 1, and (i) the upper probabilities satisfy P y p(y)≥ 1 2 . We name this as PROBINT. Other representations. For credal sets, we form an ensem- ble of LLMs—either distinct models or multiple seeded runs 2 Since the upper-bound MMI depends only onp, we verify constraint (i) only. Prompt 2: ProbInt Provide a lower and upper probability (each between 0.0 and 1.0) indicating how likely the answer is correct. Interpret the probabilities as follows: • Lower Probability: the smallest probability you con- sider plausible that the answer is correct. • Upper Probability: the largest probability you con- sider defensible that the answer is correct. The sum of all lower probabilities across all answers must not exceed 1.0. Algorithm 2 Imprecise probability 1: Input: input x, candidate answersY , verifier C verifier 2: Init: ˆp(y | x)← 0 for all y ∈Y . 3: while C verifier [ˆp(Y | x)] is False do 4: p (Y | x),p(Y | x)← IP-Prompt(x,Y). 5: return: MMI(p,p) of the same model (see Prompt 3). For possibility functions, see Prompt 4. We name each method as CREDAL and POS. 3.3 MAXIMUM MEAN IMPRECISION We compute second-order uncertainty as a scalar metric from IP representations using MMI [Chau et al., 2025] with total variation. Since exact computation scales exponentially with|Y|(Appendix B), we employ suitable approximations. Intuition. The most straightforward uncertainty score de- rived from an IP representation is the interval width: MMI = p(y)− p(y).(1) This expression is exact for a single answery. However, when|Y| > 1, aggregating multiple intervals into a single scalar measure requires the formal MMI definition, which accounts for interactions across candidates. Upper bound. [Chau et al., 2025] provides an easily com- putable upper bound: MMI≤ 1− X y∈Y p(y).(2) We use this upper bound as a tractable approximation of the exact MMI. We include additional details in Appendix B. 3.4 RELATED WORKS Verbalized uncertainty. Vanilla verbalized uncertainty is known to be overconfident [Tian et al., 2023, Xiong et al., 2024]. Subsequent work has focused on improving cal- ibration, including alternative prompting and elicitation styles [Tian et al., 2023, Xiong et al., 2024], calibration- oriented fine-tuning [Li et al., 2025, Lin et al., 2022, Xu (a) Rotation (b) Cyclic APPLEn= 0 BQQMFn= 1 n= 2CRRNG APPLE EAPPL LEAPP Random noise p= 0 p= 0.1 p= 0.2 APPLE aPPLE ApPLe Learnable transformation Figure 5: Learnable vs. noisy transforms. et al., 2024, Stengel-Eskin et al., 2024], post-hoc adjust- ments such as normalization [Wang and Stengel-Eskin, 2025], and language-based uncertainty expressions [Kirch- hof et al., 2025a]. However, prior work typically assumes no question ambiguity, whereas our framework explicitly separates first-/second-order uncertainty. Uncertainty disentanglement. Disentangling first-/second- order uncertainties has increasingly been adapted to LLMs, but all prior work focused on sampling-based approach or assumption to the access to internal parameters: (i) Information-theoretic decompositions [Lakshminarayanan et al., 2017] have been reframed using generated clarifica- tion prompts, measuring how much clarifications reduce output uncertainty via mutual information [Hou et al., 2024, Walha et al., 2025, Xia et al., 2025]. (i) Perturbations: vary- ing temperature and prompts, measure its change as uncer- tainty score [Gao et al., 2024], and (i) via linear probes on internal activations [Ahdritz et al., 2024]. We instead target disentanglement via verbalized uncertainty, which to our knowledge has not been done. Human experts elicitation IP methods originate from frameworks developed to model and elicit human judg- ments. Similar elicitation-based methodologies are now increasingly adopted in LLM research, including prefer- ence learning [Rafailov et al., 2023, Fujisawa et al., 2025], rubric-based evaluation frameworks [Huang et al., 2025], item response theory [Polo et al., 2024, Mencattini et al., 2025], and social choice-theoretic approaches [Muandet, 2022, Adachi et al., 2025]. 4 SYNTHETIC EXPERIMENT Our goal is to elicit an LLM’s uncertainty through conver- sational prompts. However, uncertainty estimation is inher- ently unsupervised and thus difficult to evaluate objectively. To address this, we first construct a synthetic dataset with an explicitly controlled data-generating process in this sec- tion. We then apply our approach to real-world datasets, following standard practice (§5). Data generation. Following Zhao et al. [2025], we con- struct synthetic sequence-transformation tasks (Figure 5), including (a) rotation and (b) cyclic shift, each parameter- ized byn. We additionally inject noise by randomly lower- casing each output letter with probability p. Vanilla ProbInt (b) 2nd-order unc. DeFinetti Pos ambiguity noise p 00.10.3 0.2 0.5 0.4 Uncertainty score 0.4 0.8 0 Uncertainty score 0.1 0.3 0.2 0.5 0.4 (a) 1st-order unc. 0.4 0.2 0.6 Credal ambiguity noise p Figure 6: (a) For first-order uncertainty estimation, both vanilla and De Finetti capture the underlying ambiguity noisep, but (b) for second-order, our methods stay flat, supporting the disentanglement of uncertainty source. Task. Given an input stringx, the model predicts its trans- formed outputy(e.g., APPLE→BQQMF). The LLM re- ceivesmexample pairs(x j ,y j ) m j=1 generated from a con- sistent transformation rule and must infer this rule to predict ˆy m+1 for a new inputx m+1 . We adopt ICL, where examples are provided via prompts without parameter updates. Metric. We evaluate prediction using a permissive string match: predictions are first capitalized, then compared ex- actly with the ground-truth all-capitalizedy ⋆ m+1 . This intro- duces multiple correct answer, i.e.,|Y ⋆ m+1 | > 1, and any ˆy m+1 ∈ Y ⋆ m+1 is judged as correct. As such, lowercasing noise affects only rule inference during ICL, not test-time evaluation. We call p as ambiguity noise. Controlling first-/second-order uncertainty. We treat the ambiguity noisepas the first-order because|Y ⋆ m+1 | > 1, and the number of in-context examplesmcontrols the second-order asmaugments knowledge, while first-order noise remains constant [Ling et al., 2024, Wang et al., 2025]. Baselines. We compare against VANILLA [Tian et al., 2023], which directly elicits a confidence probability on a predicted answerˆy. We convert it to an uncertainty score as1− conf. Base setup. We fix the transformation to rotation (n = 13), followed by cyclic shift (n = 1), and random lowercasing with probabilityp. For eachm, we generate five indepen- dent ICL example sets and evaluate each with five random seeds (25 runs total), accounting for variability in both ICL examples and stochastic model outputs. Unless otherwise specified, we use GPT-5-MINI [Achiam et al., 2023]. 4.1 FIRST-ORDER NOISE We evaluate uncertainty disentanglement under varying am- biguity noise levelsp ∈ 0, 0.1, 0.2, 0.3, 0.4, 0.5. To iso- late ambiguity effects, we fix the number of ICL examples atm = 80. Figure 6 reports the behavior of each uncertainty score for (a) first- and (b) second-order. As expected, both VANILLA and DEFINETTI-AU increase approximately lin- early withp, while our IP methods remain largely invariant. Uncertainty score # in-context examples 80 0 20 4060 0.8 0 0.4 AUROC Average probability Credal (a) Individual LLM(b) LLM ensemble error ProbInt vanilla Figure 7: (a) For individual EU elicitation, VANILLA incor- rectly increases uncertainty, whereas PROBINT-EU tracks prediction error. (b) For group EU elicitation, our CREDAL- EU substantially improves AUROC. POS shows some sensitivity topbecause it also relies on a scalar possibility value, unlike interval-based approaches such as CREDAL and PROBINT. Nevertheless, under higher p—when alternative answers become equally more plausi- ble—POS stabilizes and becomes less sensitive to further increases in p. 4.2 SECOND-ORDER DENOISE Next, we fixp = 0.25and vary the number of ICL examples m. Since increasingmreduces second-order uncertainty, a well-calibrated score should decrease accordingly. As shown in Figure 7, PROBINT closely tracks prediction error, whereas VANILLA remains overly uncertain and fails to reflect the error reduction. Results for the credal-set and possibility-based methods are reported in the Appendix D.1; both exhibit decreasing trends. 4.3 GROUP UNCERTAINTY ELICITATION Next, we consider group uncertainty elicitation for an LLM ensemble. As LLMs are increasingly deployed in multi- agent systems (e.g., [Lange et al., 2026]), collective un- certainty becomes crucial. The credal-set view provides a natural framework, as it captures cross-model disagree- ment. In Figure 8, thex-axis enumerates correct answers under the transformation that differ only by the lower- case noisep. The probability of each answer under the noise varies across candidatesy i . Letℓdenote the num- ber of lowercase characters iny i ; thenp(y i ) = p ℓ · (1− p) 1−ℓ . Withp = 0.25, the ideal probabilities arep(y i ) ∈ 0.316, 0.105, 0.035, 0.012, 0.004forℓ ∈ 0, 1, 2, 3, 4. In practice, these values depend on the ICL examples and may deviate from the ideal case. We apply the CREDAL method to elicit group uncertainty. Unlike the single-LLM setting, whereˆycan be obtained sim- ply by asking the model which option to choose, the group setting requires an additional decision rule to aggregate votes across multiple LLMs. To avoid this added complex- [ p, p ] _ _ p ( y i = correct) 0.0 0.1 0.2 0.3 0.4 0.5 GPT-4.1-mini GPT-5-mini GPT-5-nano Figure 8: Credal-set from LLM ensemble: model disagree- ment induces lower/upper probability bounds for each out- put, which are used to compute MMI. ity, we instead compute AUROC againstp(y i = correct)for each answer and use the exact MMI (Eq. 1) at the answer level. To compute MMI, we first elicit each LLM to reports its first-order probabilitiesp(y i = correct)(dots in Fig. 8). The IP intervals (red bars) are obtained by taking the point- wise minimum/maximum probabilities across models. We compare CREDAL with the standard aggregation method used in LLM ensembles—utilitarian aggregation—defined as,E M i [p(y = correct|M i )] . Taking the argmax over this quantity recovers popular strategies such as majority voting or best-of-N(e.g., [Wang et al., 2022, Si et al., 2023]). In our evaluation, however, we use the averaged score directly and compare it with our MMI score using AUROC. As shown in Figure 7(b), simple average provide limited predictive power, whereas CREDAL yields substantial improvement. 5 REAL-WORLD QA EXPERIMENT We evaluate our methods on real-world QA benchmarks, following standard practice. Datasets. We use two dataset types. (i) Open-ended QA: MAQA [Yang et al., 2025] and AmbigQA [Min et al., 2020]. Both contain a mix of ambiguous (|Y ⋆ | > 1) and unambigu- ous items (|Y ⋆ | = 1). (i) Non-open-ended QA: MMLU- Pro [Wang et al., 2024] (answer optionsYis provided) but only contains unambiguous items (|Y ⋆ | = 1). Task. Given questionx, we prompt the LLM to predict a single answerˆy. Correctness is determined by comparingˆy with a reference answery ⋆ , defined as: (i) non-ambiguous: Y ⋆ =y ⋆ ; or (i) ambiguous: a specific sampley ⋆ ∈Y ⋆ . Any y ∈Y ⋆ but y ̸= y ⋆ is judged as incorrect. Metric. We report AUROC using each uncertainty scores. (i) ambiguity: labels are|Y ⋆ | > 1vs.|Y ⋆ | = 1. (i) correctness: labels are ˆy = y ⋆ vs. ˆy ̸= y ⋆ . Baselines. The only directly comparable baseline is MI CLARIFICATIONS [Hou et al., 2024], which decomposes total uncertainty (equivalent to semantic entropy [Farquhar et al., 2024]) into aleatoric (first-order) and epistemic (second-order) uncertainty via the mutual information be- tween sampled outputs and generated clarifications. Table 1: AUROC for correctness detection without ambiguity. Best result bolded in blue, second best in orange. GPT-5gemini-2.5-pro MethodMMLU-ProNon-MAQANon-AmbigQAMMLU-ProNon-MAQANon-AmbigQAAvg. Rank Is true prob.0.6850 ± 0.02380.5835 ± 0.00410.5584 ± 0.01570.6583 ± 0.01970.6296 ± 0.00680.5897 ± 0.00578.67 Label prob.0.7649 ± 0.02200.6430 ± 0.00620.5876 ± 0.01760.6377 ± 0.01530.6483 ± 0.01260.5582 ± 0.03197.83 MI-Clarifications0.7790 ± 0.01140.6528 ± 0.00800.5840 ± 0.01070.6348 ± 0.01440.6546 ± 0.01040.5497 ± 0.02677.83 DiNCO0.7087 ± 0.02140.5642 ± 0.00660.5219 ± 0.00350.7133 ± 0.01670.5749 ± 0.02240.5275 ± 0.01909.17 Vanilla0.8587 ± 0.01920.7577 ± 0.01940.7756 ± 0.01380.7326 ± 0.02210.6889 ± 0.02630.6688 ± 0.02094.17 Top-40.8725 ± 0.01430.7668 ± 0.00890.7666 ± 0.01350.7652 ± 0.02260.7027 ± 0.02940.6698 ± 0.00873.33 CoT0.8548 ± 0.01240.7695 ± 0.00780.7724 ± 0.01010.6826 ± 0.04090.6431 ± 0.01400.6094 ± 0.01525.33 ProbInt (ours)0.8617 ± 0.00820.7709 ± 0.00580.7713 ± 0.00440.7857 ± 0.02850.7303 ± 0.00920.7128 ± 0.01872.33 Credal (ours)0.8798 ± 0.01210.7753 ± 0.00670.7932 ± 0.01260.7466 ± 0.02010.6978 ± 0.01990.5982 ± 0.04692.67 Pos (ours)0.8738 ± 0.02160.7230 ± 0.01790.7426 ± 0.02280.7863 ± 0.01630.6806 ± 0.01140.6780 ± 0.00943.67 AmbigQAMAQAAmbigQAMAQA 0.3 0.4 0.5 0.6 0.7 AUROC GPT-5gemini-2.5-pro Semantic Ent.Ask4Conf-DMI-Clar.DeFinetti Figure 9: AUROC for ambiguity detection. Baselines for ambiguity. (i) SEMANTIC ENTROPY [Far- quhar et al., 2024], entropy over semantically clustered sam- ples; (i) ASK4CONF-D [Hou et al., 2024], directly eliciting the probability that a question is ambiguous; and (i) MI CLARIFICATIONS [Hou et al., 2024]. Baselines for correctness. We consider verbalized and sampling-based methods. Verbalized: (i) VANILLA [Tian et al., 2023, Xiong et al., 2024]; (i) COT [Xiong et al., 2024], eliciting a rationale to improve prediction and con- fidence; and (i) TOP-4 [Tian et al., 2023], querying the top four answers but using only the top answer’s confi- dence. Sampling-based: (i) IS-TRUE [Tian et al., 2023, Ka- davath et al., 2022]; (i) LABEL PROBABILITY, estimat- ing an empirical answer distribution from samples; (i) DINCO [Wang and Stengel-Eskin, 2025], extending IS- TRUE with distractors and normalization, and (iv) MI CLAR- IFICATIONS [Hou et al., 2024]. Base setup. For each dataset, we sampleN = 200exam- ples from the validation set and repeat experiments five times, reporting mean and standard deviation. Sampling- based baselines and CREDAL use five samples per query; runs with different seeds are treated as distinct beliefs. 5.1 AMBIGUITY AND CORRECTNESS Ambiguity detection. As shown in Fig. 9, our DEFINETTI achieves the highest AUROC for detecting ambiguity. Interestingly, directly eliciting an ambiguity probability (ASK4CONF-D) underperforms on MAQA, suggesting that AmbigQAAmbigQA 0.50 0.55 0.60 0.65 0.70 0.75 0.80 AUROC GPT-5gemini-2.5-pro Label prob. Semantic Ent. Vanilla Top-4 CoT DeFinetti*ProbInt DeFinetti*Credal DeFinetti*Pos Figure 10: AUROC for correctness with ambiguity. a single global ambiguity judgment is insufficient without modeling the answer distribution. Correctness detection. We next evaluate correctness detec- tion capability. We first select unambiguous subsets from MAQA and AmbigQA (i.e.,|Y ⋆ | = 1), and we call them as Non-MAQA and Non-AmbigQA. Results are reported in Table 1. Overall, our methods perform best, although verbalization-based approaches also work reasonably well in this setting. This is consistent with prior findings: in the absence of ambiguity, uncertainty scores provably perform reliably [Tomov et al., 2025]. Empirically, PROBINT is the most robust across datasets and models. Correctness detection under ambiguity. Under simultane- ous ambiguity and incomplete knowledge, isolated scores may fail to capture total uncertainty. Since DEFINETTI and IP scores are measured on different scales, they cannot be directly summed. We therefore combine them multiplica- tively to achieve scale invariance. Figure 10 3 shows that this product separates correct from incorrect predictionsˆybetter than the baselines under combined uncertainty. API cost. We report API costs computed from public pric- ing for all methods (Fig. 11). Compared to sampling-based baselines, our methods are generally more cost-efficient, ex- cept CREDAL which also uses sampling. PROBINT and POS are comparable in cost to verbalized baselines. Compared to 3 We takey ⋆ to be the reference answer from Natural Ques- tions [Kwiatkowski et al., 2019], included in AmbigQA. MI CLARIFICATIONS—the only baseline that disentangles uncertainty—our methods cost less than half. Visualization. For ambiguity, we use the MAQA dataset. Figure 2(c) shows that our DEFINETTI estimate effectively separates ambiguous from clear questions, whereas the VANILLA score in Figure 1(b) fails to do so. For correctness, we adopt the Kullback–Leibler (KL) divergence metric pro- posed by Tomov et al. [2025], who construct a reference answer distributionp ⋆ from question–answer co-occurrence frequencies in a large corpus. They evaluate uncertainty (in the absence of ambiguity) via the KL divergence be- tween the LLM’s predictive distributionˆpandp ⋆ (see Ap- pendix D.2). Figure 12 shows that our PROBINT method exhibits the strongest correlation with this KL metric. We further compute the concordance index, which measures rank-based correlation between two variables, again com- paring the KL metric with uncertainty scores. As shown in Figure 13, our IP-based methods demonstrate consistently strong and robust correlations. 5.2 EXPLAINING OWN DECISION Previously, we evaluated correctness against the ground- truthy ⋆ . We now instead compare against the model’s own predictionˆy. In classical multi-class classification, predic- tion is obtained viaˆy precise = arg max y∈Y ˆp(y = correct), which we refer to as PRECISE PROB. This rule is fully algorithmic and internally consistent. In contrast, LLM pre- dictions resemble sampling from a conditional distribution, ˆy LLM ∼ ˆp(y | x), meaning elicited probabilities need not co- incide with the argmax of the predictive distribution. More broadly, this issue relates to faithfulness and self-consistency in LLMs [Madsen et al., 2024, Matton et al., 2025]. Bayesian rationality. Under uncertainty, a natural self- consistency criterion is Bayesian rationality [Harsanyi, 1978], which checks whetherˆymatches the argmax of expected utility. To compute utility, we employ MI- CLARIFICATION to elicit predictive distributions over an- swers:ˆy Bayes = arg max y∈Y E C i ∼p(C i |x) ˆp(y = correct | C i ), where C i denotes clarification contexts. IP-based rationality. Decision-making under IP does not yield a single canonical rule as in Bayesian the- ory. We therefore consider two standard criteria: maximin (ˆy maximin = arg max y∈ ˆ Y p(y)) and maximax (ˆy maximax = Sem. Ent. Label Prob. Is-True MI-Clar. DiNCO Ask4Conf-D Vanilla CoT Top-4 DeFinetti ProbInt Credal Pos 0 5 10 15 20 25 Cost ($) Figure 11: API cost on MMLU-Pro across all methods. 020 0.1 0.0 0.1 0.2 Est. Uncertainty Vanilla 020 KL(p) 0.1 0.0 0.1 0.2 CoT 020 0.00 0.25 0.50 0.75 1.00 ProbInt Figure 12: PROBINT more closely matches the KL metric. AmbigQA*MAQA*AmbigQA*MAQA* 0.4 0.5 0.6 0.7 Conc. Index GPT-5gemini-2.5-pro Is true prob. Label prob. Vanilla Top-4 CoT DiNCO MI-Clar. ProbInt Credal Pos Figure 13: Concordance index (↑better) between uncer- tainty scores and the KL metric. arg max y∈ ˆ Y p(y)). We use PROBINT to directly elicit the probability intervals [p(y),p(y)]. Results. We evaluate on AmbigQA by measuring the align- ment rate betweenˆy LLM and each decision rule. As shown in Figure 14, the maximin rule exhibits the highest alignment with the LLM’s predictions. 6 CONCLUSION AND LIMITATION We propose IP-based methods for higher-order uncertainty elicitation. Across tasks and models, our approach im- proves elicitation accuracy and internal consistency while remaining cost-efficient. Combined with MMI-based post- processing, IP provides a principled framework for assess- ing LLM credibility. Our method shares several limitations with prior work. First, we assume verbalized uncertainty is approximately rational; although allowing imprecision mitigates, this cannot be fully verified. We also assume the model correctly interprets prompts. Strong classification performance is required—if the model fails to recognize alternative answers, ambiguity cannot be captured. Finally, we focus primarily on Q&A tasks; extending to settings such as translation or summarization remains future work. 707580859095100 % matches on y LLM ProbInt ProbInt MI-Clar. Precise Prob. Maximin Maximax Figure 14: The LLM’s predictionˆy LLM aligns with the IP- rational decision. References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.URLhttps:// arxiv.org/abs/2303.08774. Masaki Adachi, Siu Lun Chau, Wenjie Xu, Anurag Singh, Michael A. Osborne, and Krikamol Muandet. Bayesian optimization for building social-influence-free consensus. arXiv preprint arXiv:2502.07166, 2025. URLhttps: //arxiv.org/abs/2502.07166. Aakriti Agrawal, Rohith Aralikatti, Anirudh Satheesh, Souradip Chakraborty, Amrit Singh Bedi, and Furong Huang.Uncertainty-aware answer selection for im- proved reasoning in multi-llm systems. In Findings of the Association for Computational Linguistics: EMNLP, 2025. URLhttps://aclanthology.org/2025. findings-emnlp.1367/. Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman. Distinguishing the knowable from the unknowable with language models. In International Conference on Machine Learning, 2024. Thomas Augustin, Frank P. A. Coolen, Gert de Cooman, and Matthias C. M. Troffaes. Introduction to Imprecise Probabilities. John Wiley & Sons, Chichester, United Kingdom, 2014. ISBN 978-0-470-97381-3. Dylan Bouchard and Mohit Singh Chauhan.Uncer- tainty quantification for language models: A suite of black-box, white-box, LLM judge, and ensemble scor- ers.Transactions on Machine Learning Research, 2025. URLhttps://openreview.net/forum? id=WOFspd4lq5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. Siu Lun Chau, Michele Caprio, and Krikamol Muandet. Inte- gral imprecise probability metrics. In Advances in Neural Information Processing Systems, 2025. URLhttps: //openreview.net/forum?id=KM2XzHq2Rm. Siu Lun Chau, Soroush H Zargarbashi, Yusuf Sale, and Michele Caprio.Quantifying epistemic predictive uncertainty in conformal prediction.arXiv preprint arXiv:2602.01667, 2026.URLhttps://arxiv. org/abs/2602.01667. B. De Finetti. La prévision: ses lois logiques, ses sources subjectives. Annales de l’Institut Henri Poincaré. Institut Henri Poincaré, 1937. D. Dubois and H. Prade. Théorie des possibilités: applica- tions à la représentation des connaissances en informa- tique. Masson, 1985. Fabian Falck, Ziyu Wang, and Christopher C. Holmes. Is in-context learning in large language models Bayesian? A martingale perspective. In International Conference on Machine Learning, 2024. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 2024. URLhttps: //doi.org/10.1038/s41586-024-07421-0. Masahiro Fujisawa, Masaki Adachi, and Michael A. Os- borne. Scalable valuation of human feedback through provably robust model alignment. In Advances in Neural Information Processing Systems, 2025. URLhttps: //openreview.net/forum?id=EaTRrceoU9. Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kama- lika Das. SPUQ: Perturbation-based uncertainty quan- tification for large language models.In Proceed- ings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2024. URLhttps://aclanthology.org/2024. eacl-long.143/. John C. Harsanyi. Bayesian decision theory and utilitarian ethics. The American Economic Review, 68(2):223–228, 1978. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decomposing uncertainty for large language models through input clarification ensem- bling. In International Conference on Machine Learning, 2024. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, et al. Reinforcement learning with rubric anchors. arXiv preprint arXiv:2508.12790, 2025. URL https://arxiv.org/abs/2508.12790. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. URLhttps: //arxiv.org/abs/2207.05221. John Maynard Keynes. A Treatise on Probability. Macmil- lan & co., 1921. Michael Kirchhof, Luca Füger, Adam Goli ́ nski, Ee- shan Gunesh Dhekane, Arno Blaas, Seong Joon Oh, and Sinead Williamson. SelfReflect: Can LLMs communi- cate their internal answer distribution? arXiv preprint arXiv:2505.20295, 2025a.URLhttps://arxiv. org/abs/2505.20295. Michael Kirchhof, Gjergji Kasneci, and Enkelejda Kasneci. Position: Uncertainty quantification needs reassessment for large language model agents.In International Conference on Machine Learning, 2025b.URL https://proceedings.mlr.press/v267/ kirchhof25b.html. Kolmogorov. Sulla determinazione empirica di una legge didistribuzione. Giorn Dell’inst Ital Degli Att, 4:89–91, 1933. Nikita Kotelevskii, Vladimir Kondratyev, Martin Taká ˇ c, Eric Moulines, and Maxim Panov. From risk to uncertainty: Generating predictive uncertainty measures via Bayesian estimation. In International Conference on Learning Representations, 2025. URLhttps://openreview. net/forum?id=cWfpt2t37q. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, et al. Nat- ural questions: A benchmark for question answering re- search. Transactions of the Association for Computa- tional Linguistics, 2019. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty es- timation using deep ensembles. In Advances in Neural Information Processing Systems, 2017. Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. ShinkaEvolve: Towards open-ended and sample-efficient program evolution.In International Conference on Learning Representations, 2026.URLhttps:// openreview.net/forum?id=lKEdGCoDNC. Isaac Levi. The Enterprise of Knowledge: An Essay on Knowledge, Credal Probability, and Chance. MIT Press, Cambridge, MA, 1980. ISBN 0262120828. Yibo Li, Miao Xiong, Jiaying Wu, and Bryan Hooi. Con- fTuner: Training large language models to express their confidence verbally.In Advances in Neural Infor- mation Processing Systems, 2025. URLhttps:// openreview.net/forum?id=VZQ04Ojhu5. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transac- tions on Machine Learning Research, 2022. ISSN 2835- 8856. URLhttps://openreview.net/forum? id=8s8K2UZGTZ. Chen Ling, Xujiang Zhao, Wei Cheng, Yanchi Liu, Yiyou Sun, et al. Uncertainty decomposition and quantification for in-context learning of large language models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2024. URLhttps://openreview.net/forum? id=Oq1b1DnUOP. Ryan Liu, Jiayi Geng, Joshua Peterson, Ilia Sucholutsky, and Thomas L. Griffiths. Large language models as- sume people are more rational than we really are. In International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=dAeET8gxqg. William Lugoloobi, Thomas Foster, William Bankes, and Chris Russell.LLMs encode their failures: Predict- ing success from pre-generation activations.arXiv preprint arXiv:2602.09924, 2026.URLhttps:// arxiv.org/abs/2602.09924. Sravanthi Machcha, Sushrita Yerra, Sharmin Sultana, Hong Yu, and Zonghai Yao. Do large language models know when not to answer in medical QA? In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (Un- certaiNLP), 2025. URLhttps://aclanthology. org/2025.uncertainlp-main.4/. Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language models faithful? In Findings of the Association for Computational Lin- guistics: ACL, 2024. URLhttps://aclanthology. org/2024.findings-acl.19/. Katie Matton, Robert Ness, John Guttag, and Emre Kici- man. Walk the talk? measuring the faithfulness of large language model explanations. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=4ub9gpx9xw. Tommaso Mencattini, Robert Adrian Minut, Donato Crisos- tomi, Andrea Santilli, and Emanuele Rodolà. MERGE 3 : Efficient evolutionary merging on consumer-grade GPUs. In International Conference on Machine Learning, 2025. URLhttps://proceedings.mlr.press/ v267/mencattini25a.html. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open- domain questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020. URLhttps://aclanthology.org/2020. emnlp-main.466/. Krikamol Muandet. Impossibility of collective intelligence. arXiv preprint arXiv:2206.02786, 2022. URLhttps: //arxiv.org/abs/2206.02786. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin.tiny- Benchmarks: evaluating LLMs with fewer examples. In International Conference on Machine Learning, 2024. URLhttps://openreview.net/forum? id=qAml3FpfhG. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural In- formation Processing Systems, 2023.URLhttps: //openreview.net/forum?id=HPuSIXJaa9. Herbert Robbins. Asymptotically subminimax solutions of compound statistical decision problems. Herbert Robbins Selected Papers, pages 7–24, 1951. Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar. A survey on uncertainty quantifica- tion of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Sur- veys, 58(3), September 2025. doi: 10.1145/3744238. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang.Prompting GPT-3 to be reliable.In In- ternational Conference on Learning Representations, 2023. URLhttps://openreview.net/forum? id=98p5x51L5af. Cedric A. B. Smith. Consistency in statistical inference and decision. Journal of the Royal Statistical Society Series B: Statistical Methodology, 23(1):1–25, 1961. Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. La- cie: Listener-aware finetuning for calibration in large lan- guage models. In Advances in Neural Information Pro- cessing Systems, 2024. URLhttps://openreview. net/forum?id=RnvgYd9RAh. Terry Therneau and Elizabeth Atkinson. The concordance statistic. A package for survival analysis in R, vignettes. R package version, pages 3–7, 2023. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christo- pher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod- els fine-tuned with human feedback. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023. URLhttps://aclanthology. org/2023.emnlp-main.330/. Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. Uncertainty-based abstention in LLMs improves safety and reduces halluci- nations. arXiv preprint arXiv:2404.10960, 2024. URL https://arxiv.org/abs/2404.10960. Tim Tomov, Dominik Fuchsgruber, Tom Wollschläger, and Stephan Günnemann. The illusion of certainty: Un- certainty quantification for LLMs fails under ambigu- ity.arXiv preprint arXiv:2511.04418, 2025.URL https://arxiv.org/abs/2511.04418. Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zh- moginov, and Max Vladymyrov. Transformers learn in- context by gradient descent. In International Conference on Machine Learning, 2023. Tomoya Wakayama and Taiji Suzuki. In-context learning is provably Bayesian inference: a generalization theory for meta-learning. arXiv preprint arXiv:2510.10981, 2025. URL https://arxiv.org/abs/2510.10981. Nassim Walha, Sebastian G. Gruber, Thomas Decker, Yin- chong Yang, Alireza Javanmardi, Eyke Hüllermeier, and Florian Buettner. Fine-grained uncertainty decompo- sition in large language models: A spectral approach. arXiv preprint arXiv:2509.22272, 2025. URLhttps: //arxiv.org/abs/2509.22272. Peter Walley. Statistical Reasoning with Imprecise Proba- bilities. Chapman & Hall, 1991. Victor Wang and Elias Stengel-Eskin. Calibrating verbal- ized confidence with self-generated distractors. arXiv preprint arXiv:2509.25532, 2025.URLhttps:// arxiv.org/abs/2509.25532. Xuesong Wang. Active learning for nlp with large language models. arXiv preprint arXiv:2401.07367, 2024. URL https://arxiv.org/abs/2401.07367. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.URLhttps://arxiv. org/abs/2203.11171. Yifei Wang, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. Uncertainty unveiled: Can exposure to more in-context examples mitigate uncertainty for large language models? In Findings of the Association for Computational Lin- guistics: ACL, 2025. URLhttps://aclanthology. org/2025.findings-acl.1062/. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understand- ing benchmark. arXiv preprint arXiv:2406.01574, 2024. URL https://arxiv.org/abs/2406.01574. Shiguang Wu, Yaqing Wang, and Quanming Yao. Why in- context learning models are good few-shot learners? In International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=iLUcsecZJp. Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. A survey of uncertainty estimation methods on large lan- guage models. arXiv preprint arXiv:2503.00172, 2025. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. In International Conference on Learning Representations, 2022. URLhttps:// openreview.net/forum?id=RdJVFCHjUMI. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? An empirical evaluation of confidence elici- tation in LLMs. In International Conference on Learning Representations, 2024. URLhttps://openreview. net/forum?id=gjeQKFxFpZ. Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. SaySelf: Teach- ing LLMs to express confidence with self-reflective rationales.In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2024. URLhttps://aclanthology.org/2024. emnlp-main.343/. Khurram Yamin, Jingjing Tang, Santiago Cortes-Gomez, Amit Sharma, Eric Horvitz, and Bryan Wilder.Do LLMs act like rational agents? Measuring belief coher- ence in probabilistic decision making. arXiv preprint arXiv:2602.06286, 2026.URLhttps://arxiv. org/abs/2602.06286. Yongjin Yang, Haneul Yoo, and Hwaran Lee. MAQA: Evaluating uncertainty quantification in LLMs re- garding data uncertainty.In Findings of the Association for Computational Linguistics: NAACL, 2025. URLhttps://aclanthology.org/2025. findings-naacl.325/. Chengshuai Zhao, Zhen Tan, Pingchuan Ma, Dawei Li, Bohan Jiang, Yancheng Wang, Yingzhen Yang, and Huan Liu.Is chain-of-thought reasoning of LLMs a mirage? A data distribution lens.arXiv preprint arXiv:2508.01191, 2025.URLhttps://arxiv. org/abs/2508.01191. Verbalizing LLM’s Higher-order Uncertainty via Imprecise Probabilities (Supplementary Material) Anita Yang 1,2 Krikamol Muandet 3 Michele Caprio 4,5 Siu Lun Chau 6,* Masaki Adachi 1,* 1 Lattice Lab, Toyota Motor Corporation, Japan 2 Department of Computer Science, University of Tokyo, Japan 3 Rational Intelligence Lab, CISPA, Helmholtz Center for Information Security, Germany 4 Department of Computer Science, University of Manchester, United Kingdom 5 Manchester Centre for AI Fundamentals, United Kingdom 6 EPIC Lab, College of Computing & Data Science, Nanyang Technological University, Singapore * Equal contribution A PROMPTS A.1 SECOND-ORDER UNCERTAINTY We provide prompts for eliciting two other IP representations of second-order uncertainty discussed in the main text: credal sets (CREDAL) and possibility functions (POS). Credal sets We represent the credal set by eliciting a finite ensemble ofMprecise predictive distributionsp (m) M m=1 fromMmodels or samples (random seed). The resulting credal set is taken to be the empirical set of these distributions: C = conv(p (m) M m=1 ). Each p (m) is obtained using Prompt 3. Prompt 3: Credal Assign a probability (between 0.0 and 1.0) representing how likely it is that the answer would be given as a response to the question. A correct answer should generally receive a higher probability than an incorrect one. Likelihood may vary based on reasonable interpretations of the question (e.g., ambiguity in scope, answer type, entity interpretation, or contextual assumptions). The sum of all probabilities must not exceed 1.0. Possibility function We elicit the LLM to provide a possibility function over the candidate answers, including an explicit alternative option (i.e., “none of the above”), using Prompt 4. We allow the elicited scores to be unnormalized; the required normalization is applied as part of the MMI computation (Apd. B.3). Prompt 4: Pos Provide a possibility score which captures how plausible the answer correctly answers the question. Then, provide a possibility score how plausible it is that a different answer (not listed) could be correct. The possibility should be between 0.0 and 1.0. A possibility score of 1.0 means “fully plausible,” and 0.0 means “impossible.” A.2 APPROXIMATION OF ˆ Y . To characterize uncertainty about the questionxitself—rather than about a particular answery(e.g., a predictedˆy)—we consider uncertainty over the candidate answer setY(Apd. C). In open-ended QA, however,Yis not observed. We therefore approximate it with a finite set ˆ Yby prompting the model to generate plausible (non-zero-probability) candidate answers (Prompt 5), within which the ground-truth set of correct answersY ⋆ is likely to lie. Prompt 5: Candidate answers ˆ Y Given the question below, generate a list of all possible correct answers, taking into account different reasonable interpretations of the question. Provide the answers as a numbered list, with each answer on its own line. Each answer must be concise text only, with no explanations or additional wording. Do not include duplicates or answers that refer to the same entity or concept. For example: 1. <answer one as concise text> 2. <answer two as concise text> ... B MAXIMUM MEAN IMPRECISION The exact MMI metric for measuring EU [Chau et al., 2025] under total variation is MMI TV (P) := sup A∈F Y P(A)− P(A) ≤ 1− X y∈Y P(y),(3) whereF Y is theσ-algebra over the candidate answersY(in our discrete setting,F Y = 2 Y ). The lower and upper probabilities P(A)andP(A)bound the probability of an eventA ⊆ Y, which we interpret as “the sampled answer lies inA.” The quantityP(A)− P(A)is the imprecision (interval width) for eventA, andMMI TV takes the maximum such width over all events. For example, whenY =y 1 ,y 2 , Fig. 18 enumerates all events in2 Y and their bounds;MMI TV corresponds to the widest interval among them, capturing the agent’s worst-case epistemic uncertainty over the output space. p ℱ 풴 푦 " ,푦 # 푦 " 푦 # 휙 MMI = largest gap Figure 18: MMI measures the largest gap between up- per/lower probabilities across all events A∈F Y . 휋 ℱ 풴 푦 " ,푦 # 푦 " 푦 # 휙 MMI Figure 19: MMI for possibility function measures the second largest possibility score overY . B.1 CREDAL SET In CREDAL-EU, we instantiate the credal set as the convex hull of a finite ensemble ofMprecise predictive distributions p (m) M m=1 produced by independently prompted runs (Prompt 3): C := conv p (m) M m=1 .(4) Since P y∈A p(y)is linear inp, its extrema over the polytopeCare attained at extreme points. Therefore, the event bounds reduce to simple min/max over ensemble members: P(A) = min m∈[M] X y∈A p (m) (y),P(A) = max m∈[M] X y∈A p (m) (y).(5) In particular, the singleton bounds used in the tractable upper bound are P(y) = min m∈[M] p (m) (y),P(y) = max m∈[M] p (m) (y).(6) ExactMMI TV requires optimizing over eventsAand can be exponential in|Y|. In our experiments, we therefore (i) compute exact MMI only for singleton events corresponding to a specific answery(which reduces to the interval width in Eq. 6), and (i) use the linear-time upper bound in Eq. 3 for uncertainty overY. For open-ended QA, whereYis approximated by a finite candidate set ˆ Y , we apply the same computations over ˆ Y . B.2 PROBABILITY INTERVAL In PROBINT-EU, the LLM elicits a probability interval[p(y),p(y)]for each candidate answery. For singleton events, the induced lower/upper probabilities are immediate: P(y) = p(y),P(y) =p(y).(7) For non-singleton eventsA⊆Y, constructing coherent boundsP(A)andP(A)from singleton intervals generally requires additional assumptions and can be more involved. Moreover, computing exactMMI TV still entails optimizing over eventsA and can scale exponentially with|Y|. We therefore follow the same strategy as CREDAL-EU: (i) we compute exact MMI only for answer-level uncertainty on a specificy(i.e., singleton events / correctness ofy), and (i) we use the linear-time upper bound in Eq. 3 for task-level uncertainty overY (or its approximation ˆ Y ). B.3 POSSIBILITY FUNCTION In POS-EU, we prompt the LLM to elicit a (potentially unnormalized) possibility functionˆπ :Y → [0, 1]over the output space. In open-ended QA,Yis approximated by a finite candidate set ˆ Ytogether with a “none of the above” option, allowing the model to assign high plausibility to an outcome outside the enumerated candidates without forcing plausibility to be redistributed among them, since possibility scores are non-additive. For possibility functions, total-variation MMI admits a simple form: after normalization so thatmax y∈Y π(y) = 1, MMI equals the second-largest possibility value [Chau et al., 2026]: MMI TV = π (2) (8) = ˆπ (2) ˆπ (1) ,(9) whereπ (1) ≥ π (2) ≥ ·denote the order statistics of the normalized scoresπ(y) y∈Y , andˆπ (1) ≥ ˆπ (2) ≥ ·are the corresponding order statistics of the unnormalized scoresˆπ(y). Because normalization enforcesπ (1) = 1,π (2) measures the strength of the best competing alternative relative to the top hypothesis: larger values indicate greater ambiguity among leading candidates (and hence higher imprecision). Equivalently, MMI quantifies uncertainty in the top answer via the strongest competitor (Fig. 19). When assessing uncertainty for a single answery(binary eventy,¬y), we elicit unnormalized scoresˆπ(y)andˆπ(¬y) (with¬y implemented as “not y”) and normalize by dividing by maxˆπ(y), ˆπ(¬y), resulting in: MMI TV = min ˆπ(y) max(ˆπ(y), ˆπ(¬y)) , ˆπ(¬y) max(ˆπ(y), ˆπ(¬y)) = minˆπ(y), ˆπ(¬y) maxˆπ(y), ˆπ(¬y) .(10) C UNCERTAINTY ON: “ANSWERy” VS. “CANDIDATE ANSWERSY ” We clarify when uncertainty should be defined for the correctness of a particular answery(e.g., a model predictionˆy) versus over the candidate answer setY (or its approximation ˆ Y ). Two objects of uncertainty: (i) Answery(answer-centric). Uncertainty onyconcerns whether a particular answery(typicallyˆy) is correct, collapsing all alternatives into “not-y.” It is most useful for decision-focused use cases (trust/abstain/verify/fallback), where utility depends primarily on the correctness of the chosen answer. (i) Candidate answersY(question-centric). Uncertainty onYconcerns how belief is distributed across plausible answers. It is most useful for question-focused use cases (ambiguity detection, multiple valid answers, need for clarification), where utility depends on whether the question supports several competing answers, not just whether ˆy is correct. C.1 FIRST-ORDER UNCERTAINTY On answery. For a given answery, first-order uncertainty is defined on the binary event “yvs. not-y.” Under a precise predictive distributionp, this is the Bernoulli entropyH(Bern(p(y))). In Fig. 20a, the correctness AU fory 1 equals H(Bern(0.4)). On candidate answersY. First-order uncertainty over the candidate set quantifies how probability mass is distributed across answers. Under a precise predictive distribution p onY , this is the entropy H(p) over all y ∈Y (Fig. 20a). Experiments. We use these notions for different evaluation goals: (i) Answery: decision-centric evaluations, including error tracking in our ICL experiments (Sec. 4) and correctness detection (AUROC; Fig. 10 in Sec. 5). (i) Candidate answersY : question-centric evaluations, namely ambiguity detection in open-ended QA (Fig. 9 in Sec. 5). C.2 SECOND-ORDER UNCERTAINTY On answery. For a given answery, second-order uncertainty (imprecision) captures uncertainty about the binary event “y vs. not-y” under a probability interval. We measure this by the interval widthp(y)− p(y)(Eq. 1). In Fig. 20b, the EU fory 2 is the interval gap (MMI = 0.3). On candidate answersY. Second-order uncertainty over the candidate set captures imprecision about the entire answer distribution. We measure this using the upper-bound MMI,1− P y∈Y p (y)(Eq. 2). In Fig. 20b, this aggregates lower bounds across candidates (upper-bound MMI = 1− 0.4). Experiments. We use these notions for different evaluation goals: (i) Answery: decision-centric evaluations, including error tracking in our ICL experiments (Sec. 4) and correctness detection (AUROC; Sec. 5). (i) Candidate answersY: question-centric evaluations, including ambiguity-related analyses and comparisons to “ground-truth” EU distributions (Fig. 13). 0.4 p y ! y " y # y $ 0.1 (a) First order: (i) binary entropy foryvs. not-y, or (i) entropy overY . p y ! y " y # y $ 0.4 0.1 (b) Second order: (i) interval width of eachy, or (i) upper- bound MMI overY . Figure 20: First- and second-order uncertainty can be computed either for the (i) correctness of a specific answery, or (i) over the candidate answers setY D EXPERIMENTS D.1 IN-CONTEXT LEARNING D.1.1 Second-order denoise We report additional error-tracking results from Sec. 4.2 using second-order uncertainty approximated via CREDAL (Fig. 21b) and POS (Fig. 21a). In contrast to VANILLA (Fig. 7), both representations track error more closely, and CREDAL shows a clear decrease as the number of in-context examples grows, indicating effective denoising of epistemic uncertainty with additional evidence. Between PROBINT, POS, and CREDAL, POS is the most sensitive to first-order noise, consistent with Fig. 6 (holding second-order fixed while varying first-order noise). Nevertheless, POS remains more robust than VANILLA under first-order noise (Fig. 6). 01020304050607080 Number of Samples 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Error Pos Error 0.30 0.35 0.40 0.45 0.50 MMI (a) POS 01020304050607080 Number of Samples 0.1 0.2 0.3 0.4 0.5 Error Credal Error 0.50 0.55 0.60 0.65 MMI (b) CREDAL Figure 21: Approximations of second-order uncertainty with fixed first-order noise (lower-case probabilityp = 0.25) (with moving average window of 10). Both POS and CREDAL generally match shape of the error. D.1.2 Group Uncertainty Elicitation We provide additional experimental details in Sec. 4.3. Our ensemble consists of three language models: GPT-5-Mini, GPT-4.1-mini, and GPT-5-Nano. Using GPT-5-Mini with Prompt 5, we first construct a shared candidate set ˆ Y. This controls for the fact that different models may produce different point predictionsˆy, while our elicitation procedure requires every model to assign uncertainty over the same set of answer options. We therefore define ˆ Y to cover the answers the ensemble is most likely to generate, and then prompt each model to verbalize its uncertainty over ˆ Y . In contrast, for our other CREDAL experiments we obtain multiple samples by repeatedly querying a single model. D.2 REAL-WORLD QA EXPERIMENT D.2.1 Estimating first- and second-order uncertainty from dataset We assess disentangling and quantifying second-order uncertainty in settings where both first- and second-order uncertainty are present. In such cases, AUROC can be reductive: it scores correctness against a single reference answery ⋆ , even when other answers may also be valid (with varying probabilities). To enable a more principled evaluation, we follow Tomov et al. [2025], who propose an approximation to ground-truth first- and second-order uncertainty distributions (their aleatoric uncertainty (AU) and epistemic uncertainty (EU)) for ill-defined real-world QA. They posit a true first-order answer distributionp ⋆ , which can be approximated from corpus statistics by estimating question–answer co-occurrence frequencies in a large text corpus. This estimated p ⋆ is provided in the AmbigQA ⋆ and MAQA ⋆ datasets. 0123 0 1 2 3 Est. Uncertainty Conc. Ind. = 0.66 Pred. Ent. 0123 (p) Conc. Ind. = 0.52 Mutual Information 0123 Conc. Ind. = 0.73 DeFinetti Figure 22: DEFINETTI best align with the proxy for first-order uncertainty Given p ⋆ and a model predictive distribution ˆp, AU and EU admit the decomposition CE(p ⋆ , ˆp) |z Total uncertainty = H(p ⋆ ) |z AU + KL(p ⋆ ∥ ˆp) |z EU ,(11) whereCEdenotes cross-entropy andKLthe Kullback–Leibler divergence. This cross-entropy decomposition is closely related to mutual-information-based uncertainty decompositions [Kotelevskii et al., 2025]. We therefore treatH(p ⋆ )as a proxy for ground-truth first-order uncertainty (AU) andKL(p ⋆ ∥ ˆp)as a proxy for ground-truth second-order uncertainty (EU). Metric. Following Tomov et al. [2025], we use concordance statistics [Therneau and Atkinson, 2023] to measure rank agreement between an estimated uncertainty score and the corresponding proxy ground truth. Concordance is the probability that, for a randomly chosen pair of examples, the method assigns a higher uncertainty to the example with higher ground-truth uncertainty. Its interpretation matches AUC-ROC: values closer to 1 indicate better ranking alignment. We report this metric in Fig. 13. Second-order uncertainty. We compare second-order uncertainty estimated by our proposed IP representations to the KL-based proxyKL(p ⋆ ∥ ˆp). Figure 12 visualizes their association for VANILLA, COT, and PROBINT; our method exhibits the strongest alignment. We further benchmark against additional uncertainty estimators in Fig. 13 using the concordance index, where our methods are the most aligned across datasets and models. First-order uncertainty. We also evaluate first-order uncertainty by comparing method estimates to the entropy proxyH(p ⋆ )on MAQA ⋆ . Figure 22 visualizes the estimated first-order uncertainty from PREDICTIVE ENTROPY, MI- CLARIFICATIONS, and DEFINETTI, where DEFINETTI is best aligned with the proxy with highest concordance index.

Instant research discovery