Paper deep dive
Credal Large Language Models for Semantic Commitment under Uncertainty
Shireen Kudukkil Manchingal, Sofiia Nikolenko, Fabio Cuzzolin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation we derive two complementary commitment scores. Credal Token Commitment (CTC) is a token-space score that combines lower-bound support, credal width, and intersection entropy, computed without additional generation. Semantic Commitment Consistency (SCC) extends commitment to semantic space using sampled completions, with SCC-Gap measuring the mismatch between token-level and semantic-level support. We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge. CLLM is the best method on QA accuracy at competitive expected calibration error, and CTC tracks the best hallucination AUROC within 1.5 pp on most settings without additional generation. On selective prediction at 80% coverage, CLLM with SCC reaches 99.0% accuracy on OpenBookQA, and on ARC-Challenge CLLM with Csem confidence achieves <= 0.6% ECE across the three backbones.
Tags
Links
- Source: https://arxiv.org/abs/2608.23244v1
- Canonical: https://arxiv.org/abs/2608.23244v1
Trouble viewing inline? Open PDF directly →
Full Text
120,769 characters extracted from source content.
Expand or collapse full text
Credal Large Language Models for Semantic Commitment under Uncertainty Shireen Kudukkil Manchingal Affiliation: Oxford Dynamics Affiliation: Oxford Email: smanchingal@brookes.ac.uk Sofiia Nikolenko Affiliation: Ludwig-Maximilians-Universität München Affiliation: Munich Email: sofiia.nikolenko@campus.lmu.de Fabio Cuzzolin Affiliation: Institute for Artificial Intelligence, Data Analysis and Systems (AIDAS) Affiliation: School of Engineering Computing & Mathematics Affiliation: Oxford Brookes University, Oxford, UK Email: fabio.cuzzolin@brookes.ac.uk Abstract Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation we derive two complementary commitment scores. Credal Token Commitment (CTC) is a token-space score that combines lower-bound support, credal width, and intersection entropy, computed without additional generation. Semantic Commitment Consistency (SCC) extends commitment to semantic space using sampled completions, with SCC-Gap measuring the mismatch between token-level and semantic-level support. We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge. CLLM is the best method on QA accuracy at competitive expected calibration error, and CTC tracks the best hallucination AUROC within 1.51.5 p on most settings without additional generation. On selective prediction at 80%80\% coverage, CLLM with SCC reaches 99.0%99.0\% accuracy on OpenBookQA, and on ARC-Challenge CLLM with CsemC_sem confidence achieves ≤0.6%≤ 0.6\% ECE across the three backbones. 1 Introduction Large Language Models (LLMs) have advanced rapidly, achieving strong performance on question answering, reasoning, code generation, and open-ended dialogue (3; 6; 68; 38). Yet modern LLMs continue to produce fluent-but-incorrect answers with unwarranted confidence, particularly when context is incomplete, conflicting, or adversarial (56; 37; 47; 27). In safety-critical settings such as healthcare, law, or scientific assistance the model often fails not by being uncertain, but by appearing insufficiently uncertain. A central limitation, summarised in Figure 1, is that standard LLMs represent uncertainty through a single next-token distribution obtained by softmax normalisation. This representation forces the model to commit to precise probabilities even when the available evidence is weak, and standard predictive confidence reflects only the sharpness of one distribution rather than whether the model has robust support across multiple plausible hypotheses (60; 58; 34). Bayesian LoRA and Laplace-based approximations place distributions over adapter parameters (80; 24); ensemble methods use disagreement as an epistemic signal (45; 2); and semantic-variability methods cluster sampled generations to score meaning-level diversity (43; 27). These approaches improve robustness in several settings, but they ultimately summarise uncertainty as a single predictive distribution or a scalar disagreement statistic, and so do not explicitly preserve uncertainty about the predictive probabilities themselves. Single softmaxprob.aabbccddeey∗y^*Credal set (x)P(x)aabbccddeey∗y^*P¯(b) P(b)P¯(b) P(b)P¯−P¯ P- P Figure 1: Left: a single softmax gives a sharp prediction whether the model is confident or merely committed. Right: the credal set induced by a LoRA ensemble exposes for each token a lower probability P¯ P (top of solid red bar) and an upper probability P¯ P (top of outlined band); a non-trivial gap P¯−P¯ P- P on tokens with weak support separates robust commitment from epistemic ignorance. In this paper, we adopt the perspective of imprecise probability (71; 64; 46; 21), in which epistemic uncertainty is represented by a set of plausible distributions rather than a single point, sometimes the set induced by lower and upper bounds on the true probability of an event (25; 69; 11), often called ’belief functions’ (65; 70; 66; 9; 17; 20; 19; 16). A credal set (46; 71), in particular, is a closed convex set of probability distributions used to represent epistemic uncertainty when no single distribution can be confidently identified, with lower and upper probabilities P¯,P¯ P, P as natural worst- and best-case summaries (34; 1; 15). The intersection-probability transform (12; 14; 22; 23) returns a single representative distribution that respects those bounds. Credal sets (13) have been widely employed for classification purposes in the past 48. Recent credal-set neural networks bring this view to image classification and out-of-distribution detection (72; 75; 73; 54), conformal learning (36), uncertainty quantification (33), Bayesian deep learning (4), but also statistical learning theory (5). More widely, epistemic approaches to machine learning which make use of second-order uncertainty measures are on the rise (44; 32; 35; 51; 53; 10; 59; 55; 50; 52; 74; 42; 79). With this paper, We bring credal set representations to next-token prediction in instruction-tuned LLMs. We propose Credal Large Language Models (CLLMs): there, an ensemble of LoRA adapters on a frozen backbone induces a credal set whose convex hull defines lower and upper probability bounds P¯,P¯ P, P over the next-token vocabulary. The credal set, however, is not yet a decision: a deployed model still has to commit to an answer, abstain, or escalate. The natural question is therefore not “how uncertain is the prediction?” but “how strongly does the credal set support a particular answer?”. Existing token-level scores collapse the credal set back into a single softmax and read off entropy, which hides the spread of plausible distributions; existing semantic-uncertainty scores cluster sampled generations and read off cluster diversity, but say nothing about whether the chosen cluster is supported by token-level evidence. Neither side, on its own, distinguishes confident-and-correct from confidently-wrong. We argue that a usable commitment signal must (i) be available cheaply when generation is expensive, (i) reflect agreement across plausible predictors rather than the sharpness of any single one, and (i) cross-check token-level confidence against semantic-level support so that surface fluency cannot stand in for meaning. These three needs motivate the three scores we introduce: Credal Token Commitment (CTC), Semantic Commitment Consistency (SCC), and SCC-Gap, each addressing one of the gaps above; their formal definitions are deferred to Section 3. Our contributions are: (i) Credal Large Language Models, a practical framework for representing LLM uncertainty as credal sets induced by LoRA ensembles, exposing lower and upper probability bounds rather than a single softmax distribution; (i) two credal uncertainty measures, intersection entropy and credal width, that quantify the geometry of epistemic uncertainty in a form directly usable for prediction and risk-aware decision-making; (i) Credal Token Commitment, a token-space decision score combining lower-bound support, credal width, and intersection entropy, computed entirely from the credal set with no additional generation, within 1.51.5 p of the best baseline on five of eight hallucination settings; and (iv) Semantic Commitment Consistency and Semantic Commitment Consistency Gap, extending commitment to semantic space, with the full-model framework reaching 99.0%99.0\% accuracy on OpenBookQA at 80%80\% coverage and 7979–88%88\% accuracy at ≤0.6%≤ 0.6\% ECE on ARC-Challenge across three backbones. Paper outline. Section 2 surveys related work on uncertainty in LLMs, parameter-efficient Bayesian methods, semantic uncertainty, and imprecise probability. Section 3 formalises the credal-set construction, credal uncertainty measures, and the three commitment scores. Section 4 reports hallucination, calibration, selective prediction, and ARC-Challenge reasoning, and Section 5 distils the lessons. Implementation, setup, additional results, extended related work, and broader impact are deferred to §A–§F. 2 Related Work The aleatoric/epistemic split is formalised at the predictive-distribution level 34, and LLMs are systematically miscalibrated on QA and few-shot tasks 39; 81. Sequence-level uncertainty in autoregressive models can be obtained by ensembling and decomposed into token-level contributions 49; RLHF-tuned conditional probabilities are particularly poorly calibrated, and verbalised confidences elicited by prompting are often better 67; 47; 41. Conformal prediction has been adapted to LLMs to produce prediction sets with coverage guarantees 61, and total uncertainty has been decomposed into aleatoric/epistemic components via input-clarification ensembling 29. Full-Bayesian inference is intractable at LLM scale, so recent work targets the LoRA parameters 30: Laplace-LoRA fits a KFAC Gaussian posterior (80; 24), BLoB an ELBO-trained variational posterior (77), and LoRA ensembles skip the explicit posterior in favour of independently trained adapters (2). All these methods either summarise the ensemble as a single mean predictive distribution or extract a scalar disagreement score; we instead keep the ensemble as a finite set whose convex hull is a credal set. Hallucination and abstention sit within selective prediction with the reject option 26; 28, recently surveyed for LLMs 31; 37. Token-level dispersion is a poor hallucination signal because multiple surface forms can express the same answer; semantic entropy instead clusters sampled generations by meaning and scores cluster diversity 43; 27, and self-consistency decoding marginalises over sampled reasoning paths to pick the most agreed-upon answer (76). Related approaches train auxiliary classifiers on token-level features 56, chain reasoning steps for error-aware self-evaluation 78, quantify repetition across samples for selective answering of ambiguous questions 8, or use embedding-based scores for OOD detection and selective generation (63). These methods process a single predictive distribution or its samples; none expose a set-valued uncertainty representation. Our SCC and SCC-Gap require commitment to be jointly supported in token and semantic space. Building on the imprecise-probability lineage of 71; 46; 23, we bring credal-set representations from image classification to next-token prediction in instruction-tuned LLMs and couple them with semantic-cluster commitment; extended discussion of imprecise-probability foundations, credal / random-set / belief-function neural networks, conformal and reject-option prediction, token-vs-semantic uncertainty in LLMs, and parameter-efficient Bayesian methods is in §E. 3 Methodology We propose Credal Large Language Models (Figure 2). An ensemble of M LoRA adapters on a frozen backbone yields a finite point cloud in the next-token simplex whose convex hull is the credal set (x)P(x), with lower and upper bounds P¯,P¯ P, P on its facets and a representative point p p from the intersection-probability transform. Three commitment scores derived from this geometry, optionally combined with semantic-cluster mass over sampled completions, summarise reliability for selective prediction and abstention. Inputprompt xxLLM(frozen backbone)LoRA f1f_1LoRA f2f_2⋮ fMf_M(1) LLM + LoRA ensemble(2a) Token space →Ctok→ C_tokParisFrancethepm(y∣x)p_m(y\! \!x)M such dists“Paris” (y∗y^*)“France”“the”(x)P(x)p pP¯(y∗)−maxy≠y∗P¯(y) P(y^*)- _y≠ y^* P(y)CtokC_tok(2b) Semantic space →Csem→ C_sem“Paris”“Paris.”“France’s”“It’s Paris”K samplesParisotherc∗:k=3\;c^*\!:\;k=3k=2k=2clustered by meaningCsemC_sem(3) Commitment scoresCredal TokenCommitment (CTC)×credal-set only; no generationSemantic CommitmentConsistency (SCC)×token & semantic both commitSCC-GapΔ -vs-semantic divergenceCommitorAbstain(4) Decision Figure 2: CLLM architecture. An LLM with M LoRA adapters on a frozen backbone produces, in parallel, two views of the next-token prediction. (2a) Token space (top): each adapter outputs a next-token distribution pmp_m over the vocabulary (illustrated bar chart). Treated as M points in the probability simplex over the top three candidates here “Paris”, “France”, “the”, their convex hull (cyan) is the credal set (x)P(x), with lower / upper bounds P¯,P¯ P, P on its facets and the intersection-probability transform p p (red dot) selecting y∗=y^*=“Paris”. The credal token commitment CtokC_tok reads the lower-bound margin of y∗y^* against the strongest competitor through a sigmoid. (2b) Semantic space (bottom): the same LLM is sampled K times to produce text completions; clustering by meaning gives a dominant cluster c∗c^* (here the Paris cluster, k=3k=3) whose mass and margin define the semantic commitment CsemC_sem. (3) Commitment scores: CTC compresses the credal-set geometry alone, requiring no generation, and is illustrated as the product of three glyphs: the blue bar denotes the token commitment CtokC_tok, the two parallel red lines denote a narrow credal set, and the red triangle denotes a sharp intersection-probability peak. SCC multiplies token-level commitment by semantic-cluster commitment; SCC-Gap flags divergence between the two (shown by the Δ arrow). (4) Decision: commit or abstain based on the chosen score. 3.1 Problem Formulation and Credal Set Construction Let x denote an input prompt and f1,…,fM\f_1,…,f_M\ an ensemble of M LoRA-adapted language models sharing the same frozen backbone, each producing a next-token distribution pm(⋅∣x)p_m(· x). We treat these as plausible predictors under epistemic uncertainty and retain them as a family of predictive beliefs rather than averaging them. The induced credal set is (x)=convp1(⋅∣x),…,pM(⋅∣x)P(x)=conv\p_1(· x),…,p_M(· x)\, where conv(⋅)conv(·) denotes the convex hull. For each token y we define lower and upper probabilities P¯(y∣x)=minmpm(y∣x) P(y x)= _mp_m(y x) and P¯(y∣x)=maxmpm(y∣x) P(y x)= _mp_m(y x). The lower probability measures support for y that is robust across all plausible predictors; the upper probability measures the most favourable plausible support; the gap P¯−P¯ P- P summarises unresolved epistemic variability that mean-pooling the ensemble would discard. The distinction matters operationally: a token can have high mean probability while lacking robust support if some plausible predictors strongly disagree, so a trustworthy prediction should be supported in a way that is stable across plausible predictive beliefs, not just on average. When a point-valued prediction is required, we use the intersection probability transform. Let P¯ P and P¯ P denote the vectors of lower and upper probabilities; we define p^(y∣x)=P¯(y∣x)+α(P¯(y∣x)−P¯(y∣x)), p(y x)= P(y x)+α ( P(y x)- P(y x) ), (1) where α is chosen so that ∑yp^(y∣x)=1 _y p(y x)=1. The role of p p is operational: it yields a valid representative distribution for selecting a prediction while leaving the full credal representation intact for uncertainty analysis. Thus the framework preserves both a decision object (p p for choosing an answer) and an uncertainty object (the lower and upper bounds for reasoning about whether the choice is justified). 3.2 Credal Uncertainty Measures Two complementary uncertainty measures arise from the credal set: the entropy of the representative intersection distribution (how diffuse the cautious point-valued prediction is) and the credal width (how much epistemic spread remains across plausible predictive beliefs): H∩(x)=−∑yp^(y∣x)logp^(y∣x),H_∩(x)=-\! _y p(y x) p(y x), (2) W(x)=1||∑y(P¯(y∣x)−P¯(y∣x)).W(x)= 1|V|\! _y ( P(y x)- P(y x) ). (3) Entropy reflects uncertainty within one predictive distribution; credal width reflects uncertainty across predictive distributions. A model can have high entropy with low credal width when several continuations are reasonable but all ensemble members broadly agree, or low entropy with large credal width when the cautious point appears sharp but ensemble members disagree. This separates ambiguity in the predicted answer from instability in the predictive belief itself. 3.3 Token Commitment CtokC_tok Let y∗=argmaxyp^(y∣x)y^*= _y p(y x) denote the selected answer. Whether the model is justified in committing to y∗y^* depends on two conditions: y∗y^* must have strong robust support, and it must be separated from its strongest competitor. Examining the probability of the selected answer alone is insufficient, since a competing answer can remain nearly as plausible. We define the token commitment score as Ctok(y∗∣x)=exp(βP¯(y∗∣x))exp(βP¯(y∗∣x))+∑y≠y∗exp(βP¯(y∣x)),C_tok(y^* x)= \! (β\, P(y^* x) ) \! (β\, P(y^* x) )+ _y≠ y^* \! (β\, P(y x) ), (4) where P¯(y∗∣x) P(y^* x) is the lower probability of the selected token, P¯(y∣x) P(y x) the upper probabilities of competing tokens, and β>0β>0 a sharpness parameter. The numerator measures worst-case support for y∗y^*; the denominator competes it against the best-case mass of every other token. CtokC_tok approaches 11 when y∗y^* is strongly supported across all ensemble members and decays smoothly as competitors retain comparable upper-probability mass, preserving credal dominance without the degeneracy of a hard step-function margin on uncertain or adversarial inputs. 3.4 Credal Token Commitment (CTC) CtokC_tok ignores two further pieces of information exposed by the credal set: how narrow the set is, and how sharp the cautious point-valued prediction is. We combine these three signals into the Credal Token Commitment (CTC), CTC(y∗∣x)=Ctok(y∗∣x)⋅(1−W(x))⋅(1−H∩(x)log||),CTC(y^* x)=C_tok(y^* x)· (1-W(x) )· (1- H_∩(x) |V| ), (5) where W(x)W(x) is the credal width of Equation 3 and H∩(x)H_∩(x) is the intersection entropy of Equation 2. The first factor asks whether the selected token has robust lower-bound support against its strongest competitor; the second asks whether plausible predictors broadly agree on the full vocabulary distribution rather than only on y∗y^*; the third asks whether the cautious point prediction is sharp rather than diffuse. We combine multiplicatively because each factor is a near-orthogonal sufficient condition for failure: any single failure should drive CTC towards 00, whereas an additive form would let one strong factor mask the failure of another. CTC is computed entirely from the credal set, with no sampled completions or semantic clustering, making it the natural decision score when additional generation is expensive or unavailable. The credal-set construction strictly generalises single-distribution token-space scoring: when the ensemble is in full agreement (pm≡p∗p_m≡ p^*), the credal set degenerates to p∗\p^*\, W and H∩H_∩ collapse to point-distribution counterparts, and CtokC_tok reduces to a tempered-softmax margin on p∗p^*. The second factor (1−W)(1-W) is the only term that distinguishes the credal regime from the single-distribution regime; the others recover their familiar single-distribution forms. CTC therefore agrees with predictive-entropy and max-probability scores when the ensemble has nothing to disagree about, and adds a credal-spread correction when it does. Proposition A.1 in §A states this formally with proof. 3.5 Semantic Commitment: SCC and SCC-Gap Token-level commitment is necessary but not sufficient: correctness in language tasks is semantic, and multiple surface forms can express the same answer (lexical variability under semantic agreement) or a model can be sharply committed to a local token sequence while its plausible full completions split across incompatible meanings (semantic instability under local confidence). The first should not penalise commitment; the second should suppress it. Sampling K stochastic completions and clustering them by meaning yields semantic clusters (x)=c1,…,cKC(x)=\c_1,…,c_K\ with normalised cluster masses S(ci)S(c_i). Let c∗c^* denote the cluster containing y∗y^*. Mirroring CtokC_tok, we define Csem(y∗∣x)=S(c∗)⋅(S(c∗)−maxc≠c∗S(c))+C_sem(y^* x)=S(c^*)· (S(c^*)- _c≠ c^*S(c) )_+, large only when c∗c^* is both massive and dominantly separated. The Semantic Commitment Consistency score is SCC(y∗∣x)=Ctok(y∗∣x)⋅Csem(y∗∣x)SCC(y^* x)=C_tok(y^* x)· C_sem(y^* x), again multiplicative so that weak token-level support cannot be rescued by semantic agreement alone and vice versa. The diagnostic SCC-Gap(y∗∣x)=|Ctok(y∗∣x)−Csem(y∗∣x)|SCC-Gap(y^* x)=|C_tok(y^* x)-C_sem(y^* x)| exposes the regime in which the two evidence sources disagree, which we expect to be informative for adversarial detection where token-level commitment remains high while plausible generations split semantically. 4 Experiments We evaluate CLLMs across four reliability settings (hallucination detection, QA calibration, selective prediction, and multiple-choice reasoning) on three open-weight backbones and one transfer backbone, against six baseline families. Our experiments (Section 4.2) address seven questions: (i) At a fixed 80%80\% coverage, does CLLM with SCC reach selective-prediction accuracy and hallucination rate competitive with or above the strongest single-distribution and ensemble baselines, dataset by dataset? (i) Does CsemC_sem confidence yield calibrated abstention on multi-step reasoning (ARC-Challenge) across model families, where standard-LLM scoring breaks down on free-form answer continuations? (i) Do credal-set scores (intersection entropy, credal width, CtokC_tok, CTC) carry a hallucination-detection signal under corrupted context that single-distribution and ensemble baselines miss? (iv) Which factor of CTC (CtokC_tok, credal width, intersection entropy) carries the signal, and is the multiplicative form justified by complementary contributions? (v) Does meaning-level clustering of full generations outperform token-level credal scores, or do first-token semantic variants suffice? (vi) Is SCC-Gap, the divergence diagnostic, regime-dependent: does it lose signal under joint-degradation (corrupted context) and gain signal under genuine token-vs-semantic divergence (adversarial protocol)? (vii) Does the credal-set summary carry signal beyond frequentist LoRA ensembles, e.g. when applied to a Bayesian (Laplace) posterior over LoRA parameters? 4.1 Experimental Setup Backbones. We instantiate CLLMs on three open-weight instruction-tuned backbones (Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct), each adapted with M=5M=5 independently trained LoRA modules differing only in initialisation seed. LoRA training (rank r=8r=8, α=16α=16, dropout 0.10.1) details are in §B; rationale for M=5M=5 and the tokeniser-family analysis are in §C. Datasets. We use OpenBookQA (57) (44-way multiple choice), CoQA (62) (free-form conversational QA), TriviaQA (40) (long-tail factual entities), ARC-Challenge (7) (44-way multi-step reasoning), and AdvBench (82) (adversarial prompts). Hallucination settings evaluate 250250 clean and 250250 corrupted prompts; QA calibration, selective prediction, and ARC use N=500N=500. Per-dataset rationale, corruption protocol details, sample-size split rationale, embedding model and threshold (τ) choices, and the β split between hallucination and selective prediction are detailed in §C. Baselines. We compare against six families of baselines, organised in the Baselines block below: (i) Standard LLM with predictive entropy and max probability (no fine-tuning); (i) LoRA Ensemble (45; 2) with predictive entropy, mutual information, and variance over the ensemble; (i) Bayesian LoRA with KFAC-Laplace posterior (80); (iv) Laplace-LoRA with diagonal-Fisher posterior (24); (v) Semantic Entropy with cosine and NLI clustering (43; 27); (vi) ablated CLLM scores (CtokC_tok alone, CsemC_sem alone, CTC without the semantic term). All sampling-based baselines use the same compute budget (K=16K=16 semantic samples, S=20S=20 posterior samples for Bayesian / Laplace) so any gain over baselines is attributable to the credal representation rather than additional sampling. Metrics. For hallucination detection and adversarial detection we report AUROC and AUPR. For QA calibration we report accuracy, NLL, expected calibration error (ECE; 10 bins), and Brier score. For selective prediction we sweep a coverage threshold and report accuracy and hallucination rate at 80%80\% coverage; CLLM uses CTCCTC / CtokC_tok / CsemC_sem / SCCSCC as the ranking score, baselines use predictive entropy / mutual information / variance / semantic entropy. Table 1: In-distribution predictive performance and calibration on the QA test splits (N=500N=500 per setting). We report accuracy (Acc), negative log-likelihood (NLL), expected calibration error (ECE), and Brier score; arrows in column headers indicate metric direction. CLLM uses CsemC_sem as the confidence; Standard LLM and LoRA Ensemble use exp(−H¯) (- H) as the confidence proxy. Bold = best per column, underlined = second-best. Dataset Method ID Split Corrupted/Shifted Split Acc (↑ ,%) NLL (↓ ) ECE (↓ ,%) Brier (↓ ) Acc (↑ ,%) NLL (↓ ) ECE (↓ ,%) Brier (↓ ) OpenBookQA Standard LLM (mean of 3 backbones) 90.6 1.318 15.1 0.088 19.1 0.829 35.6 0.357 Bayesian-LoRA (KFAC) (80) 78.9 0.65 6.4 0.342 14.2 4.574 22.6 0.394 Laplace-LoRA (diagonal, Qwen) 90.6 0.821 42.2 0.263 40.0 0.678 31.4 0.387 LoRA Ensemble 91.7 0.375 19.6 0.102 24.3 1.327 15.4 0.223 Semantic Entropy (cosine) 91.0 1.435 0.4 0.091 22.1 3.121 19.1 0.348 CLLM (Ours) 92.0 1.435 0.4 0.090 32.1 0.839 16.4 0.343 CoQA Standard LLM 77.0 2.117 21.3 0.516 12.7 0.839 41.1 0.344 Bayesian-LoRA (KFAC) (80) 72.4 3.563 31.4 0.563 2.4 8.463 44.36 0.732 Laplace-LoRA (diagonal, Qwen) 78.8 1.568 52.5 0.452 5.4 8.432 43.64 0.364 LoRA Ensemble 84.7 1.409 54.5 0.427 12.3 2.569 6.0 0.105 Semantic Entropy (cosine) 84.2 2.197 1.1 0.148 9.6 8.015 23.2 0.626 CLLM (Ours) 85.2 2.251 1.7 0.151 9.6 7.892 13.8 0.576 TriviaQA Standard LLM 68.0 0.882 17.8 0.220 2.3 0.639 58.7 0.442 Bayesian-LoRA (KFAC) (80) 52.4 3.842 35.6 0.463 2.6 4.643 45.46 0.539 Laplace-LoRA (diagonal, Qwen) 58.0 0.382 21.1 0.269 3.5 2.585 39.23 0.567 LoRA Ensemble 62.7 2.665 45.6 0.426 11.5 3.875 9.0 0.105 Semantic Entropy (cosine) 61.8 3.945 4.4 0.291 11.2 5.288 28.0 0.488 CLLM (Ours) 68.8 0.948 4.6 0.288 11.2 5.287 20.8 0.468 4.2 Results On Table 1, CLLM is best on accuracy on all three QA datasets and is at or within 0.30.3 p of the best ECE; under corrupted context every method collapses on accuracy, but CLLM remains the strongest multi-backbone score on OpenBookQA. The credal-set representation thus delivers competitive predictive performance at competitive-or-best calibration before any commitment-based selective-prediction step is applied. (i) Selective prediction at 80% coverage: CLLM with SCC matches or beats every evaluated baseline. On aggregated selective prediction (Table 5 in §D) CLLM+SCC sits ahead of semantic entropy by 0.50.5 p and behind LoRA-Ensemble+variance by 1.21.2 p. The per-dataset breakdown (Table 2) is more informative: a different CLLM score wins each dataset (CtokC_tok on OpenBookQA, SCC on CoQA, CsemC_sem on TriviaQA), and each win lines up with the answer-space property the score is designed to expose. Table 2: Per-dataset selective prediction at 80%80\% coverage. CLLM + SCC matches or beats Semantic Entropy and the LoRA Ensemble baseline on all three datasets, with the largest gap on OpenBookQA (CLLM+SCC 99.0%99.0\% vs Semantic Entropy 98.0%98.0\% accuracy among retained predictions). Bold = best per column, underlined = second-best. OpenBookQA CoQA TriviaQA Method Cov (%) Acc (↑ ,%) Halluc (↓ ,%) Cov (%) Acc (↑ ,%) Halluc (↓ ,%) Cov (%) Acc (↑ ,%) Halluc (↓ ,%) Standard LLM (always answer) 100.0 93.2 6.8 100.0 59.6 40.4 100.0 32.4 67.6 LoRA Ensemble + variance 79 91.4 5.4 80.0 72.5 27.5 80 32.6 76.4 Semantic Entropy (cosine) 80.0 98.0 2.0 80.0 70.5 29.5 80.0 44.0 56.0 CLLM + CtokC_tok 80.0 99.5 0.5 80.0 69.5 30.5 80.0 40.5 59.5 CLLM + CsemC_sem 80.0 98.0 2.0 80.0 70.5 29.5 80.0 44.5 55.5 CLLM + SCC 80.0 99.0 1.0 80.0 71.5 28.5 80.0 43.5 56.5 The ordering tracks the answer space: OpenBookQA’s letter-level vocabulary collapses semantic clustering to noise so CtokC_tok’s lower-bound margin is the informative signal, CoQA’s short free-form answers reward the conjunctive token-and-semantic agreement SCC enforces, and TriviaQA’s long-tail entity answers spread token mass thinly so meaning-cluster mass is the stable score. The per-dataset breakdown is the strongest evidence that a credal representation beats a single-distribution one in deployment-relevant terms; CLLM does not dominate in aggregate because LoRA-Ensemble+variance is the right summary when ensemble disagreement reduces to a scalar, the regime TriviaQA’s long-tail makes dominant. (i) Reasoning: ARC-Challenge with calibrated, near-zero-ECE selective prediction. On ARC-Challenge (Table 3), CLLM+CsemC_sem achieves ECE under 0.6%0.6\% on all three backbones, two-to-three orders of magnitude below the Standard-LLM proxy whose ECE inflates to 26−81%26-81\%. The trade-off is most explicit on Qwen, where CLLM matches Standard-LLM on accuracy while collapsing ECE from 26.3%26.3\% to under 0.1%0.1\%. Table 3: ARC-Challenge reasoning results (N=500N=500). CLLM uses CsemC_sem as the confidence; Standard LLM uses exp(−H¯) (- H) as the proxy. Selective prediction is reported at 80%80\% coverage. Arrows in column headers indicate metric direction. Bold = best per column, underlined = second-best. Method Backbone Acc (↑ ,%) NLL (↓ ) ECE (↓ ,%) Brier (↓ ) Acc@cov80 (↑ ,%) Halluc@cov80 (↓ ,%) Standard LLM Gemma-2-9B 87.8 2.087 80.6 0.726 94.2 05.8 Standard LLM Llama-3.1-8B 76.8 1.948 71.3 0.658 86.2 13.8 Standard LLM Qwen2.5-7B 88.2 0.981 26.3 0.305 93.8 06.2 CLLM (Ours) Gemma-2-9B 79.6 3.231 00.4 0.202 86.2 13.8 CLLM (Ours) Llama-3.1-8B 79.0 3.175 00.6 0.202 87.0 13.0 CLLM (Ours) Qwen2.5-7B 88.4 1.870 <<0.1 0.116 94.2 05.8 The issue is format alignment, not capacity: CsemC_sem is read on a P¯,P¯ P, P envelope that already lives in the letter-only output space the LoRA adapters were tuned for, so the confidence is on the same support as the answer; the standard-LLM proxy exp(−H¯) (- H) is read on a free-form continuation distribution placing most mass on descriptive prose, inflating ECE even when the argmax label is correct. This is the regime where the credal-set representation pays off in user-facing terms: a calibrated abstention threshold beats a high-accuracy-with-broken-confidence ranker when the system must decide whether to escalate. CLLM is not uniformly preferable: on Gemma and Llama, raw Standard-LLM accuracy at 100%100\% coverage is higher, and a system that does not need calibrated abstention should prefer the unfine-tuned base. The accuracy gap is the cost of format-alignment narrowing the output space, which is the same property that makes the confidence calibrated. (i) Credal scores carry the empirical signal under corrupted context. Across the eight model×benchmark hallucination settings (Table 4), intersection entropy is best on 4/84/8 and within 0.50.5 p on 11 further setting; CTC tracks intersection entropy within 1.51.5 p on 7/87/8 settings despite requiring no sampled completions. Table 4: Hallucination detection AUROC under corrupted versus clean context. All values are AUROC (↑ ). Each setting is computed over 250 clean and 250 corrupted prompts using a 5-adapter LoRA ensemble with K=16K=16 semantic samples per query and BAAI/bge-base-en-v1.5 embeddings; cluster threshold τ=0.8τ=0.8 for OpenBookQA and τ=0.5τ=0.5 for CoQA and TriviaQA. Bold = best per column, underlined = second-best. OpenBookQA CoQA TriviaQA Score Gemma Llama Qwen Gemma Llama Qwen Llama Qwen Token-space baselines Standard LLM (predictive entropy) 0.936 0.886 0.896 0.716 0.637 0.551 0.620 0.537 Standard LLM (max prob) 0.927 0.868 0.891 0.733 0.632 0.548 0.631 0.537 LoRA ensemble (predictive entropy) 0.957 0.902 0.913 0.840 0.808 0.778 0.826 0.806 LoRA ensemble (mutual information) 0.934 0.847 0.905 0.831 0.857 0.763 0.815 0.846 Bayesian-LoRA (KFAC) (80) 0.891 0.832 0.808 0.723 0.619 0.598 0.523 0.593 Laplace-LoRA (diagonal, Qwen) (24) 0.915 0.735 0.825 0.702 0.642 0.494 0.519 0.509 Semantic-space baselines Semantic entropy (cosine) (43) 0.903 0.828 0.779 0.681 0.619 0.584 0.700 0.647 Semantic entropy (NLI) (27) 0.912 0.806 0.743 0.776 0.694 0.676 0.638 0.627 Credal scores (ours) Intersection entropy 0.954 0.904 0.915 0.841 0.842 0.791 0.816 0.729 Credal width 0.938 0.849 0.906 0.806 0.844 0.768 0.790 0.836 CtokC_tok 0.928 0.842 0.903 0.804 0.817 0.687 0.775 0.824 CTC (Equation 5) 0.939 0.854 0.905 0.837 0.844 0.778 0.816 0.736 Credal-semantic scores (ours) First-token CsemC_sem 0.917 0.848 0.890 0.779 0.803 0.703 0.768 0.790 First-token Ctok⋅CsemC_tok\!·\!C_sem 0.917 0.849 0.890 0.782 0.802 0.688 0.747 0.785 CsemC_sem (full) 0.878 0.781 0.764 0.677 0.618 0.577 0.696 0.650 SCC 0.899 0.819 0.830 0.816 0.784 0.659 0.760 0.825 SCC-Gap 0.126 0.228 0.244 0.431 0.506 0.503 0.349 0.447 Corrupted context shifts adapters off their shared training manifold by different amounts so the per-adapter softmaxes spread across the simplex; intersection entropy reads off the diffuseness of the cautious envelope p p that reflects this spread, while a single-distribution entropy averages it away before the score is taken. The eight-setting head-to-head is the load-bearing evidence for credal-set >> single-distribution: a representation exposing P¯,P¯ P, P produces a ranker that is at least competitive with, and on half the settings strictly better than, every score that collapses the ensemble first. Credal scores do not dominate every regime: on TriviaQA the long-tail entity vocabulary diffuses lower-bound mass across many surface forms and LoRA-ensemble predictive entropy / mutual information lead, with credal width (not intersection entropy) the discriminative credal factor. Distribution-shift evidence in Table 6 confirms the credal envelope reacts more sharply to corruption than any single-distribution baseline. (iv) Factor-wise CTC ablation. On Table 4 the factor ordering is intersection entropy >> credal width >> CtokC_tok on every dataset family, with the full multiplicative CTC staying within 1.51.5 p of intersection entropy on 7/87/8 settings while never falling below the weakest single factor. The exception is TriviaQA-Qwen, where credal width is the strongest single factor. The ranking reflects what each factor measures: intersection entropy summarises diffuseness across the vocabulary envelope and absorbs corruption that disperses mass anywhere on the simplex; credal width is local to where adapters disagree most; CtokC_tok depends on the lower-bound mass of a single token and is sensitive only to corruptions hitting that specific argmax. The three carry complementary signal. The multiplicative CTC justifies its design only weakly on hallucination AUROC (robust but rarely strictly best); it earns its place at the selective-prediction stage (i) where the conjunctive condition is decision-relevant. CTC and SCC are stable across operational ranges of M, β, τ, K, with largest sensitivity to τ on open-ended TriviaQA where long-tail entities spread completions across near-duplicate surface forms; per-dataset sensitivity is in Table 10 (§D). (v) Semantic-only scores underperform; first-token semantic clustering recovers most of the gap. Semantic entropy and full-sequence CsemC_sem underperform every credal score on every setting of Table 4, with gaps of up to 2020 p on the worst case (CoQA-Qwen). First-token CsemftC_sem^ft closes most of the gap, and the multiplicative first-token Ctok⋅CsemftC_tok\!·\!C_sem^ft tracks within 0.10.1 p of CsemftC_sem^ft across all eight settings. NLI clustering (27) is competitive with cosine semantic entropy on the three CoQA settings but does not change the ordering relative to credal scores. Meaning-cluster diversity grows under both genuine ambiguity and harmless paraphrastic variation; clustering full sequences mixes the informative first-token disagreement signal with downstream paraphrastic drift unrelated to correctness. The first decoded token, especially on multiple-choice formats, is where the credal set’s lower-bound support is most discriminative. This strengthens the central claim by showing the gain over semantic-entropy baselines is not just having a credal representation but where it is read: the first-token credal lower-bound is what makes CTC competitive with costlier semantic-clustering scores. Semantic clustering is not useless: CsemftC_sem^ft wins TriviaQA selective prediction (Table 2); the advantage shrinks once meaning-clusters are computed at first-token resolution, consistent with semantic entropy and CTC reading much of the same disagreement signal at that resolution. (vi) Credal-semantic SCC and SCC-Gap: regime-dependent. On Table 4, SCC trails CTC by 4−124-12 p and SCC-Gap is anti-correlated with hallucination on 5/85/8 settings: under context corruption, token- and semantic-level support degrade in tandem so the mismatch stays small. Corrupted context degrades adapter-level token support and the diversity of sampled completions in the same direction, so |Ctok−Csem||C_tok-C_sem| stays small and SCC-Gap loses its discriminative signal; on adversarial prompts we expect token-level commitment to stay high (surface answer fluent) while semantic clusters split (refusal vs. compliance), driving the gap up. The qualitative examples in Table 7 illustrate both regimes: an AdvBench injection prompt produces high CsemC_sem with near-zero CtokC_tok (the divergence regime SCC-Gap was designed for), while a wrong-context CoQA hallucination produces CtokC_tok and CsemC_sem that nearly cancel out under joint degradation. The regime-dependence is itself diagnostic: it shows the credal-set representation is a family whose components separate epistemic regimes (joint vs divergent failure), not a one-knob score. SCC-Gap is not a usable hallucination ranker on its own under corrupted context, where it explicitly fails the protocol; the same scores (CsemC_sem, SCC) dominate on selective prediction (i) and ARC (i), consistent with the conjunctive design, and the divergence regime needed to validate SCC-Gap is the AdvBench evaluation in Table 11 (§D), with the 3-adapter Llama pilot in Table 12. (vii) Credal-set summary of a Bayesian posterior also beats the scalar Bayesian score. Applying our intersection-probability entropy to the same Laplace-LoRA posterior samples on which Bayesian-LoRA reports mutual information beats the canonical MI score by 3.6−10.13.6-10.1 p across the three Qwen settings (Table 9). Mechanistically, MI averages the disagreement of the S samples down to a single number, while intersection entropy reads off the diffuseness of the cautious envelope p p that respects the per-token min/max across samples; corrupted context spreads the per-token min/max even when sample variance is small, so MI underrates the signal. This suggests the framework’s gain is not specific to frequentist LoRA ensembles: any procedure that produces a finite collection of plausible predictive distributions admits a credal-set view that recovers the signal that scalar Bayesian summaries average away. Do the experiments answer the questions? Yes, with one nuance and one limit. Findings (i) to (iv) and (vii) confirm the credal representation as either competitive with or strictly better than every single-distribution baseline on the metrics for which it is designed: selective prediction, calibrated reasoning, hallucination detection, factor decomposition, and the Bayesian-posterior extension. Finding (v) qualifies the gap to semantic-entropy baselines: it closes at first-token resolution but does not invert, indicating the two views read overlapping signal at that resolution rather than competing for it. Finding (vi) draws the limit: SCC-Gap is a regime-specific diagnostic for the divergence regime it was conceived for, not a one-knob hallucination ranker for the joint-degradation regime tested in Table 4. Together the seven findings support the central claim that representing LLM uncertainty as a credal set carries signal that collapsing the ensemble averages away, while making the operational limits explicit. 4.3 Limitations The commitment scores trade cost for richness: CTC needs no generation but cannot detect semantic divergence, while SCC and SCC-Gap require sampling and clustering and depend on the embedding model and threshold τ. The M=5M=5-adapter credal set is an empirical approximation rather than a calibrated posterior, so P¯,P¯ P, P are worst-/best-case probabilities under the realised ensemble. The hallucination protocol uses a single corruption variant per dataset and does not separately evaluate the missing-, wrong-, and conflicting-context regimes of Section 3; SCC-Gap is uninformative under joint-degradation by design, targeting the divergence regime tested adversarially. The 250+250250+250 hallucination and N=500N=500 QA/ARC sample sizes yield wide confidence intervals at low FPR, with bootstrap intervals deferred. 5 Conclusion We introduced Credal Large Language Models (CLLMs), a framework for representing epistemic uncertainty in language models through credal sets induced by ensembles of LoRA adapters. The credal set exposes lower and upper predictive probabilities and gives rise to two complementary commitment scores: Credal Token Commitment (CTC), a token-space score combining lower-bound support, credal width, and intersection entropy without requiring additional generation, and Semantic Commitment Consistency (SCC), which extends commitment to semantic space when sampled completions are available; SCC-Gap measures the disagreement between the two. Across 88 hallucination settings, intersection entropy is best on four of eight settings and CTC is within 1.51.5 p of the best on five of eight while needing no generation. On selective prediction at 80%80\% coverage CLLM with SCC reaches 99.0%99.0\% accuracy on OpenBookQA; on ARC-Challenge multiple-choice reasoning CLLM with CsemC_sem confidence reaches 88.4%88.4\% accuracy at <0.1%<\!0.1\% ECE on Qwen2.5-7B and 7979–88%88\% across the three backbones at ≤0.6%≤ 0.6\% ECE. Future directions include: (1) stress-testing SCC-Gap on adversarial protocols where token- and semantic-level evidence can diverge; (2) extending CTC to longer-form generation where the credal set must be propagated across multiple decoding steps; (3) studying the relationship between the empirical credal set and a true Bayesian posterior over LoRA parameters, complementing rather than competing with KFAC-Laplace. References Antonucci and Cuzzolin (2010) A. Antonucci and F. Cuzzolin Credal sets approximation by lower probabilities: application to credal networks. In Computational Intelligence for Knowledge-Based Systems Design: 13th International Conference on Information Processing and Management of Uncertainty, IPMU 2010, Dortmund, Germany, June 28-July 2, 2010. Proceedings 13, p. 716–725. Cited by: §1. Balabanov and Linander (2024) O. Balabanov and H. Linander Uncertainty quantification in fine-tuned llms using lora ensembles. arXiv preprint arXiv:2402.12264. Cited by: Appendix E, §1, §2, §4.1. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1. Caprio et al. (2024a) M. Caprio, S. Dutta, K. J. Jang, V. Lin, R. Ivanov, O. Sokolsky, and I. Lee Credal bayesian deep learning. Transactions on Machine Learning Research. Cited by: §1. Caprio et al. (2024b) M. Caprio, M. Sultana, E. Elia, and F. Cuzzolin Credal learning theory. arXiv preprint arXiv:2402.00957. Cited by: §1. Chowdhery et al. (2023) A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: scaling language modeling with pathways. Journal of Machine Learning Research 24 (240), p. 1–113. Cited by: §1. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: Appendix C, NeurIPS Paper Checklist, §4.1. Cole et al. (2023) J. R. Cole, M. J.Q. Zhang, D. Gillick, J. M. Eisenschlos, B. Dhingra, and J. Eisenstein Selectively answering ambiguous questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Appendix E, §2. Cuzzolin and Frezza (2001) F. Cuzzolin and R. Frezza Geometric analysis of belief space and conditional subspaces.. In ISIPTA, p. 122–132. Cited by: §1. Cuzzolin and Sultana (2024) F. Cuzzolin and M. Sultana Epistemic uncertainty in artificial intelligence. Springer. Cited by: §1. Cuzzolin (2003) F. Cuzzolin Geometry of Upper Probabilities.. In ISIPTA, p. 188–203. Cited by: §1. Cuzzolin (2007) F. Cuzzolin Two new Bayesian approximations of belief functions based on convex geometry. IEEE Transactions on Systems, Man, and Cybernetics - Part B 37 (4), p. 993–1008. Cited by: §1. Cuzzolin (2008) F. Cuzzolin On the credal structure of consistent probabilities. In European Workshop on Logics in Artificial Intelligence, p. 126–139. Cited by: §1. Cuzzolin (2009) F. Cuzzolin The intersection probability and its properties. In Symbolic and Quantitative Approaches to Reasoning with Uncertainty, C. Sossai and G. Chemello (Eds.), Lecture Notes in Computer Science, Vol. 5590, p. 287–298. Cited by: Appendix A, Appendix E, §1. Cuzzolin (2010a) F. Cuzzolin Credal semantics of Bayesian transformations in terms of probability intervals. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 40 (2), p. 421–432. Cited by: §1. Cuzzolin (2010b) F. Cuzzolin The geometry of consonant belief functions: simplicial complexes of necessity measures. Fuzzy Sets and Systems 161 (10), p. 1459–1479. Cited by: §1. Cuzzolin (2010c) F. Cuzzolin Three alternative combinatorial formulations of the theory of evidence. Intelligent Data Analysis 14 (4), p. 439–464. Cited by: §1. Cuzzolin (2014a) F. Cuzzolin Belief functions: theory and applications. Springer. Cited by: Appendix E. Cuzzolin (2014b) F. Cuzzolin Lp consonant approximations of belief functions. IEEE Transactions on Fuzzy Systems 22 (2), p. 420–436. Cited by: §1. Cuzzolin (2014c) F. Cuzzolin Visions of a generalized probability theory. Lambert Academic Publishing. Cited by: §1. Cuzzolin (2020) F. Cuzzolin The geometry of uncertainty: the geometry of imprecise probabilities. Artificial Intelligence: Foundations, Theory, and Algorithms, Springer International Publishing. External Links: ISBN 9783030631536, Link Cited by: §1. Cuzzolin (2022) F. Cuzzolin The intersection probability: betting with probability intervals. arXiv preprint arXiv:2201.01729. Cited by: §1. Cuzzolin (2024) F. Cuzzolin Uncertainty measures: a critical survey. Information Fusion, p. 102609. Cited by: Appendix E, §1, §2. Daxberger et al. (2021) E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig Laplace redux-effortless Bayesian deep learning. Advances in Neural Information Processing Systems 34, p. 20089–20103. Cited by: Appendix E, §1, §2, §4.1, Table 4. Dempster (1967) A. P. Dempster Upper and lower probability inferences based on a sample from a finite univariate population. Biometrika 54 (3-4), p. 515–528. Cited by: §1. El-Yaniv and Wiener (2010) R. El-Yaniv and Y. Wiener Foundations of selective prediction with the reject option. Journal of Machine Learning Research 11, p. 1605–1641. Cited by: Appendix E, §2. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: Appendix C, Table 6, Appendix E, §1, §1, §2, §4.1, §4.2, Table 4. Geifman and El-Yaniv (2017) Y. Geifman and R. El-Yaniv Selective classification for deep neural networks. arXiv:1705.08500. Cited by: Appendix E, §2. Hou et al. (2024) B. Hou, Y. Liu, K. Qian, J. Andreas, S. Chang, and Y. Zhang Decomposing uncertainty for large language models through input clarification ensembling. In International Conference on Machine Learning (ICML), Cited by: Appendix E, §2. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: Appendix B, Appendix E, §2. Huang et al. (2023) L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. Cited by: §2. Huang et al. (2021) Z. Huang, H. Lam, and H. Zhang Quantifying epistemic uncertainty in deep learning. arXiv preprint arXiv:2110.12122. Cited by: §1. Hüllermeier et al. (2022) E. Hüllermeier, S. Destercke, and M. H. Shaker Quantification of credal uncertainty in machine learning: a critical analysis and empirical comparison. In Uncertainty in Artificial Intelligence, p. 548–557. Cited by: §1. Hüllermeier and Waegeman (2021) E. Hüllermeier and W. Waegeman Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning 110 (3), p. 457–506. Cited by: §1, §1, §2. Huseljic et al. (2021) D. Huseljic, B. Sick, M. Herde, and D. Kottke Separation of aleatoric and epistemic uncertainty in deterministic deep neural networks. In 2020 25th International Conference on Pattern Recognition (ICPR), p. 9172–9179. Cited by: §1. Javanmardi et al. (2024) A. Javanmardi, D. Stutz, and E. Hüllermeier Conformalized credal set predictors. Advances in Neural Information Processing Systems 37, p. 116987–117014. Cited by: §1. Ji et al. (2023) Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung Survey of hallucination in natural language generation. ACM computing surveys 55 (12), p. 1–38. Cited by: §1, §2. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7b. External Links: 2310.06825, Link Cited by: §1. Jiang et al. (2021) Z. Jiang, J. Araki, H. Ding, and G. Neubig How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9, p. 962–977. Cited by: §2. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), p. 1601–1611. Cited by: Appendix C, NeurIPS Paper Checklist, §4.1. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan Language models (mostly) know what they know. In arXiv preprint arXiv:2207.05221, Cited by: Appendix E, §2. Kilicdere et al. (2026) E. Kilicdere, S. K. Manchingal, and F. Cuzzolin A neurosymbolic approach with epistemic deep learning for hierarchical image classification. arXiv preprint arXiv:2605.16383. Cited by: §1. Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. Cited by: Appendix C, Table 5, Table 6, Appendix E, §1, §2, §4.1, Table 4. Lahlou et al. (2021) S. Lahlou, M. Jain, H. Nekoei, V. I. Butoi, P. Bertin, J. Rector-Brooks, M. Korablyov, and Y. Bengio DEUP: direct epistemic uncertainty prediction. arXiv preprint arXiv:2102.08501. Cited by: §1. Lakshminarayanan et al. (2017) B. Lakshminarayanan, A. Pritzel, and C. Blundell Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Advances in Neural Information Processing Systems 30. Cited by: Table 5, Table 6, §1, §4.1. Levi (1980) I. Levi The enterprise of knowledge: an essay on knowledge, credal probability, and chance. MIT press. Cited by: Appendix E, §1, §1, §2. Lin et al. (2022) S. Lin, J. Hilton, and O. Evans Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. Cited by: Appendix E, §1, §2. Liu et al. (2019) Z. Liu, Y. Liu, J. Dezert, and F. Cuzzolin Evidence combination based on credal belief redistribution for pattern classification. IEEE Transactions on Fuzzy Systems 28 (4), p. 618–631. Cited by: §1. Malinin and Gales (2021) A. Malinin and M. Gales Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §2. Manchingal et al. (2025a) S. K. Manchingal, A. Bradley, J. F. Kooij, K. Shariatmadar, N. Yorke-Smith, and F. Cuzzolin Epistemic artificial intelligence is essential for machine learning models to truly’know when they do not know’. arXiv preprint arXiv:2505.04950. Cited by: §1. Manchingal and Cuzzolin (2022) S. K. Manchingal and F. Cuzzolin Epistemic deep learning. arXiv preprint arXiv:2206.07609. Cited by: §1. Manchingal et al. (2025b) S. K. Manchingal, M. Mubashar, K. Wang, and F. Cuzzolin A unified evaluation framework for epistemic predictions. External Links: 2501.16912, Link Cited by: §1. Manchingal et al. (2023) S. K. Manchingal, M. Mubashar, K. Wang, K. Shariatmadar, and F. Cuzzolin Random-set convolutional neural network (rs-cnn) for epistemic deep learning. arXiv preprint arXiv:2307.05772. Cited by: §1. Manchingal et al. (2025c) S. K. Manchingal, M. Mubashar, K. Wang, K. Shariatmadar, and F. Cuzzolin Random-set neural networks. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §1. Manchingal (2025) S. K. Manchingal Epistemic deep learning: enabling machine learning models to know when they do not know. arXiv preprint arXiv:2510.22261. Cited by: §1. Maynez et al. (2020) J. Maynez, S. Narayan, B. Bohnet, and R. McDonald On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661. Cited by: Appendix E, §1, §2. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2381–2391. Cited by: Appendix C, NeurIPS Paper Checklist, §4.1. Minderer et al. (2021) M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems 34, p. 15682–15694. Cited by: §1. Osband et al. (2024) I. Osband, Z. Wen, S. M. Asghari, V. Dwaracherla, M. Ibrahimi, X. Lu, and B. Van Roy Epistemic neural networks. Advances in Neural Information Processing Systems 36. Cited by: §1. Ovadia et al. (2019) Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems 32. Cited by: §1. Quach et al. (2024) V. Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. S. Jaakkola, and R. Barzilay Conformal language modeling. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §2. Reddy et al. (2019) S. Reddy, D. Chen, and C. D. Manning CoQA: a conversational question answering challenge. Transactions of the Association for Computational Linguistics (TACL) 7, p. 249–266. Cited by: Appendix C, NeurIPS Paper Checklist, §4.1. Ren et al. (2023) J. Ren, J. Luo, Y. Zhao, K. Krishna, M. Saleh, B. Lakshminarayanan, and P. J. Liu Out-of-distribution detection and selective generation for conditional language models. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §2. Shafer (1976) G. Shafer A mathematical theory of evidence. Vol. 42, Princeton university press. Cited by: §1. Shafer (1982) G. Shafer Belief functions and parametric models. Journal of the Royal Statistical Society: Series B (Methodological) 44 (3), p. 322–339. Cited by: §1. Shafer (1990) G. Shafer Perspectives on the theory and practice of belief functions. International Journal of Approximate Reasoning 4 (5), p. 323–362. Cited by: §1. Tian et al. (2023) K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Appendix E, §2. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §1. Walley and Fine (1982) P. Walley and T. L. Fine Towards a frequentist theory of upper and lower probability. The Annals of Statistics 10 (3), p. 741–761. Cited by: §1. Walley (1987) P. Walley Belief function representations of statistical evidence. The Annals of Statistics 15 (4), p. 1439–1465. Cited by: §1. Walley (1991) P. Walley Statistical reasoning with imprecise probabilities. Chapman and Hall, London. Cited by: Appendix E, §1, §1, §2. Wang et al. (2024a) K. Wang, F. Cuzzolin, S. K. Manchingal, K. Shariatmadar, D. Moens, and H. Hallez Credal deep ensembles for uncertainty quantification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §1. Wang et al. (2024b) K. Wang, F. Cuzzolin, K. Shariatmadar, D. Moens, and H. Hallez Credal wrapper of model averaging for uncertainty estimation on out-of-distribution detection. arXiv preprint arXiv:2405.15047. Cited by: Appendix E, §1. Wang et al. (2025) K. Wang, F. Cuzzolin, K. Shariatmadar, D. Moens, and H. Hallez A review of uncertainty representation and quantification in neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1. Wang et al. (2024c) K. Wang, K. Shariatmadar, S. K. Manchingal, F. Cuzzolin, D. Moens, and H. Hallez CreINNs: credal-set interval neural networks for uncertainty estimation in classification tasks. arXiv preprint arXiv:2401.05043. Cited by: Appendix E, §1. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §2. Wang et al. (2024d) Y. Wang, H. Shi, L. Han, D. Metaxas, and H. Wang BLoB: bayesian low-rank adaptation by backpropagation for large language models. arXiv preprint arXiv:2406.11675. Cited by: Appendix E, §2. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: §2. Woodley et al. (2026) T. Woodley, S. K. Manchingal, M. Tolloso, D. Bacciu, and F. Cuzzolin Random-set graph neural networks. arXiv preprint arXiv:2605.11987. Cited by: §1. Yang et al. (2023) A. X. Yang, M. Robeyns, X. Wang, and L. Aitchison Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2308.13111. Cited by: Table 6, Appendix E, §1, §2, §4.1, Table 1, Table 1, Table 1, Table 4. Zhao et al. (2021) T. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh Calibrate before use: improving few-shot performance of language models. In International Conference on Machine Learning (ICML), Cited by: §2. Zou et al. (2023) A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: Appendix C, NeurIPS Paper Checklist, §4.1. Appendix A Proofs Proposition A.1 (Singleton-credal-set limit recovers tempered-margin scoring). Let pm(⋅∣x)m=1M\p_m(· x)\_m=1^M be the M ensemble distributions over the vocabulary V, and suppose all members agree, pm=p∗p_m=p^* for some p∗p^* and all m. Then: (i) the credal set degenerates to a singleton, (x)=p∗P(x)=\p^*\, with P¯(y∣x)=P¯(y∣x)=p∗(y∣x) P(y x)= P(y x)=p^*(y x) for every y∈y ; (i) the credal width vanishes, W(y∣x)=0W(y x)=0, and the intersection-probability transform reduces to the underlying distribution, p^(⋅∣x)=p∗(⋅∣x) p(· x)=p^*(· x); (i) the intersection entropy reduces to the standard predictive entropy, H∩(x)=H(p∗(⋅∣x))H_∩(x)=H(p^*(· x)); (iv) the credal token commitment of Equation 4 reduces to a tempered-softmax margin on p∗p^*, Ctok(y∗∣x)=exp(βp∗(y∗∣x))exp(βp∗(y∗∣x))+∑y≠y∗exp(βp∗(y∣x)),C_tok(y^* x)= (β\,p^*(y^* x)) (β\,p^*(y^* x))+ _y≠ y^* (β\,p^*(y x)), and the full CTC of Equation 5 reduces to that margin weighted by predictive sharpness, CTC(y∗∣x)=Ctok(y∗∣x)⋅(1−H(p∗)log||).CTC(y^* x)=C_tok(y^* x)· (1- H(p^*) |V| ). Proof. Suppose pm=p∗p_m=p^* for all m∈1,…,Mm∈\1,…,M\. (i) Credal set degenerates to a singleton. The credal set is the convex hull (x)=convp1,…,pMP(x)=conv\p_1,…,p_M\. Since every vertex coincides with p∗p^*, the hull collapses to p∗\p^*\. For any token y∈y , P¯(y∣x)=infp∈(x)p(y∣x)=p∗(y∣x)=supp∈(x)p(y∣x)=P¯(y∣x) P(y x)= _p (x)p(y x)=p^*(y x)= _p (x)p(y x)= P(y x). (i) Credal width vanishes and the intersection-probability transform reduces to p∗p^*. W(y∣x)=P¯(y∣x)−P¯(y∣x)=0W(y x)= P(y x)- P(y x)=0. The intersection-probability transform p p is, by construction, a distribution on V that lies in (x)P(x) and respects the bounds P¯,P¯ P, P [14]. The only such distribution when (x)=p∗P(x)=\p^*\ is p∗p^* itself, so p^(⋅∣x)=p∗(⋅∣x) p(· x)=p^*(· x). (i) Intersection entropy reduces to predictive entropy. By Equation 2, H∩(x)=−∑yp^(y∣x)logp^(y∣x)=−∑yp∗(y∣x)logp∗(y∣x)=H(p∗(⋅∣x))H_∩(x)=- _y p(y x) p(y x)=- _yp^*(y x) p^*(y x)=H(p^*(· x)). (iv) CTC reduces to a tempered-margin score weighted by sharpness. Substituting P¯(y∗∣x)=p∗(y∗∣x) P(y^* x)=p^*(y^* x) and P¯(y∣x)=p∗(y∣x) P(y x)=p^*(y x) into Equation 4 gives Ctok(y∗∣x)=exp(βp∗(y∗∣x))exp(βp∗(y∗∣x))+∑y≠y∗exp(βp∗(y∣x)),C_tok(y^* x)= (β\,p^*(y^* x)) (β\,p^*(y^* x))+ _y≠ y^* (β\,p^*(y x)), which is the tempered-softmax margin asserted in the statement. By (i), the second factor of Equation 5 satisfies (1−W(x))=1(1-W(x))=1 in the natural aggregation W(x)=maxyW(y∣x)W(x)= _yW(y x) used in our experiments, and any other monotone aggregation that vanishes when W(y∣x)≡0W(y x)≡ 0 yields the same conclusion. The third factor is 1−H∩(x)/log||=1−H(p∗)/log||1-H_∩(x)/ |V|=1-H(p^*)/ |V|. Combining these gives the claimed form for CTC(y∗∣x)CTC(y^* x). ∎ The proposition implies, in particular, that any benchmark on which CLLM strictly outperforms tempered-margin or low-entropy scoring under the same backbone must do so by exploiting non-trivial credal width, i.e., by exploiting the regime where pmp_m disagree. The empirical results in Tables 4 and 3 confirm this: under context corruption (where ensemble disagreement is expected to grow), CTC and intersection entropy carry signal that single-distribution baselines do not. Appendix B Implementation Details Backbones. Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct. All backbones are frozen. LoRA training. Rank r=8r=8, α=16α=16, dropout 0.10.1. We use r=8r=8 because it is the standard PEFT default that preserves enough capacity to specialise on QA but is small enough that the M=5M=5 ensemble adds only ∼0.5% 0.5\% parameter overhead per adapter; α=16α=16 (α/r=2α/r=2) is the canonical scaling from 30, and dropout 0.10.1 matches the regularisation that yields seed-to-seed disagreement (and hence a non-trivial credal width W) without inducing under-trained adapters. M=5M=5 adapters per (backbone, dataset) pair, each trained with a different random seed for 11 epoch at learning rate 1e−41e-4, batch size 22 with gradient accumulation 44, max sequence length 10241024 tokens. Adapters target the attention projection matrices. Inference. At inference each adapter produces a next-token distribution; lower / upper probabilities are computed as token-wise min / max across the 55 ensemble members. Token-wise min / max are the exact inf / sup of (x)=convp1,…,pMP(x)=conv\p_1,…,p_M\ at each coordinate (a linear functional on a convex hull attains its extrema at the vertices), so this aggregation requires no approximation beyond the finite-sample credal set itself. The intersection-probability transform of Equation 1 yields the representative point prediction p p. The credal token commitment of Equation 4 uses sharpness β=1β=1 in the main hallucination tables and β=10β=10 for the selective-prediction analysis reported in Table 5. Semantic clustering. K=16K=16 stochastic completions per query at decoding temperature 0.80.8. Completions are embedded with BAAI/bge-base-en-v1.5 and clustered by cosine similarity at threshold τ=0.5τ=0.5 for free-form QA (CoQA, TriviaQA) and τ=0.8τ=0.8 for multiple-choice OpenBookQA. The NLI-clustered variant uses microsoft/deberta-large-mnli with bidirectional entailment. Compute. Training and evaluation runs on a single A100-80GB node per setting. Hallucination evaluation: ∼ 1.5 hours per setting. ARC-Challenge evaluation: ∼ 2 hours per setting. The Bayesian LoRA (KFAC) and Laplace-LoRA (diagonal) baselines run on the same compute budget; sbatch templates and the aggregator are provided in the supplementary code. Appendix C Experimental Setup Details This appendix expands the compact setup of Section 4.1 with the per-knob justifications. Backbones and ensemble size. The three backbones (Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct) span the dominant open-weight tokeniser/architecture families (Gemma, Llama, Qwen) at the 77–99B scale, so any credal-set effect we observe is not an artefact of one tokeniser or one instruction-tuning recipe. We choose M=5M=5 as the smallest ensemble that yields a stable convex hull on the simplex: single-adapter and 3-adapter sets collapse facets and underestimate P¯−P¯ P- P, while M>5M>5 adds compute without measurable AUROC gain in our sensitivity sweep. Lower/upper probabilities and the intersection-probability transform. The ensemble induces a credal set whose lower and upper probabilities are the per-token min and max across adapters; we use min/max because they are the exact vertices of the convex hull when the credal set is the convex combination of the M ensemble points, so they coincide with inf,sup , over (x)P(x) rather than approximating them. The intersection-probability transform yields a representative point prediction p p for answer selection; we prefer it over pignistic / centroid / barycentre transforms because it is the unique point in (x)P(x) that respects both P¯ P and P¯ P as sided bounds rather than only their average, which matters for the lower-bound term in CtokC_tok. Per-dataset rationale. We choose the three QA benchmarks to span the regimes a credal-vs-single-distribution gap should be sensitive to. OpenBookQA [57] is a four-way multiple-choice benchmark whose constrained letter-only output format amplifies the credal-vs-single-distribution gap because adapter disagreement concentrates on a small set of tokens. CoQA [62] is free-form conversational QA where short answers can vary lexically while sharing a meaning, isolating the token-vs-semantic distinction CTC and SCC are designed to expose. TriviaQA [40] is open-ended factual QA whose long-tail entity vocabulary stresses the lower-bound support term in CtokC_tok. For reasoning we use ARC-Challenge [7] (multiple-choice), chosen because its 4-way constrained-output format mirrors OpenBookQA but at a higher reasoning difficulty, allowing us to test whether CsemC_sem confidence remains calibrated when the answer requires multi-step inference rather than retrieval. AdvBench [82] provides the adversarial-prompt regime needed to validate SCC-Gap; Table 11 reports literature reference numbers from a smaller Qwen2.5-3B-Instruct model as anchors and Table 12 reports a 3-adapter Llama pilot. Corruption protocol motivation. Corruption replaces or perturbs the evidential passage while leaving the question intact; we use this corrupted-context protocol because it isolates the regime where the answer remains in-distribution but the supporting evidence becomes inconsistent across adapters, which is exactly where credal width should grow, whereas full prompt-injection or distribution shift would conflate context corruption with task-shift effects. Sample-size split. Each hallucination setting evaluates 250250 clean and 250250 corrupted prompts; we use 250+250250+250 for hallucination because the corruption protocol requires per-question evidence editing, while the QA / ARC settings use N=500N=500 standard test queries that need no per-query editing and so can be scaled 2×2× at the same compute. For QA calibration and selective prediction we use the same three benchmarks at N=500N=500. Embedding model and clustering thresholds. Semantic clustering uses K=16K=16 stochastic completions per query (temperature 0.80.8, BAAI/bge-base-en-v1.5 embeddings, threshold τ=0.5τ=0.5 for free-form / τ=0.8τ=0.8 for OpenBookQA). K=16K=16 matches prior semantic-uncertainty work [43, 27] and is the smallest sample count at which cluster mass estimates stabilise across reruns. We use BAAI/bge-base-en-v1.5 because it is the strongest open MTEB-retrieval embedding at ≤110≤ 110M parameters, which keeps clustering compute negligible relative to backbone inference while preserving meaning-level resolution on QA-style answers. We use a stricter threshold τ=0.8τ=0.8 on OpenBookQA because letter-form answers are near-identical strings whose cosine similarity is ≥0.7≥ 0.7 even for different letters, so a looser threshold collapses distinct semantic clusters, whereas free-form answers (CoQA, TriviaQA) tolerate τ=0.5τ=0.5. Sharpness β split. The credal token commitment of Equation 4 uses sharpness β=1β=1 for hallucination detection, where the score is used as a continuous AUROC ranker and a soft margin avoids saturating high-quality-but-uncertain inputs, and β=10β=10 for selective prediction (Table 5), where the score is used as an abstention threshold and a sharper margin separates retained-vs-rejected predictions more decisively at the 80%80\% coverage cut. Appendix D Additional Results and Tables In this section, we report extended experimental analyses that complement the headline results in Section 4.2. The aggregated selective-prediction summary across all three QA datasets is in Table 5 (per-dataset breakdown in Table 2 of the main paper). We then probe the divergence regime for SCC-Gap predicted under finding (vi) of Section 4.2, presenting AdvBench adversarial-prompt detection in Table 11 and a 3-adapter Llama pilot in Table 12. Aggregate distribution-shift uncertainty metrics across the hallucination caches are reported in Table 6. Table 7 lists qualitative prompt examples spanning the safe, hallucination, and adversarial regimes. We further report a parameter-overhead and computational-cost comparison in Table 8, a score-variant comparison applying the credal-set summary to a Laplace-LoRA posterior in Table 9, an ensemble-size sensitivity sweep in Table 10, and an NLI entailment-threshold sweep in Table 13. Per-setting breakdowns of accuracy, NLL, ECE, and Brier for every (method, backbone, dataset) combination are included in the supplementary release. Table 5: Selective prediction at 80%80\% coverage, aggregated across hallucination caches on OpenBookQA, CoQA, and TriviaQA. CLLM scores use the credal token commitment of Equation 4 with β=10β=10. Bold = best per column, underlined = second-best. Method Coverage (%) Accuracy (↑ , %) Hallucination Rate (↓ , %) Standard LLM (always answer) 100.0 61.7 38.3 LoRA Ensemble + variance [45] 80.0 72.5 27.5 Semantic Entropy (cosine) [43] 80.0 70.8 29.2 CLLM (Ours) + CTC 80.0 72.8 22.5 CLLM (Ours) + CtokC_tok 80.0 70.8 30.2 CLLM (Ours) + CsemC_sem 80.0 72.0 29.0 CLLM (Ours) + SCC 80.0 72.3 28.7 Table 6: Uncertainty metrics aggregated across the hallucination caches on OpenBookQA, CoQA, and TriviaQA. ID corresponds to clean splits; shifted corresponds to corrupted-context variants. Mutual information is reported only for methods with multiple predictive samples. Bold = best per column, underlined = second-best (applied to columns where direction is clear, i.e. Entropy Gap and Mutual Info: higher is better). Method ID Entropy Shifted Entropy Entropy Gap Mutual Info Predictive Entropy Intersection Entropy Credal Width SCC SCC-Gap Standard LLM 0.436 0.769 0.333 – 0.436 – – – – Bayesian-LoRA (KFAC) [80] 1.643 1.267 0.376 0.346 1.643 – – – – LoRA Ensemble [45] 1.748 2.590 0.843 0.425 1.748 – – – – Semantic Entropy (cosine) [43] 0.431 0.789 0.358 – 1.429 – – – – Semantic Entropy (NLI) [27] 1.154 1.673 0.519 – 1.745 – – – – CLLM (Ours) 1.429 2.396 0.967 0.432 1.429 1.658 0.491 0.371 0.956 Table 7: Prompt categories used in evaluation, with concrete model responses and the corresponding SCCSCC, CtokC_tok, CsemC_sem, and predictive entropy (PE) values. Safe and hallucination prompts are drawn from OpenBookQA, CoQA, and TriviaQA; adversarial prompts are drawn from AdvBench. The credal token commitment uses β=10β=10. Prompt Type Prompt Example Model Response (Summary) Uncertainty / Outcome Safe (ID, CoQA) Context: short story about two neighbour dogs. “Who were the two canines who lived next door to each other?” “Brownie and Spotty.” (gold: Brownie and Spotty) SCC=0.41, CtokC_tok=0.50, CsemC_sem=0.82; correct, committed. Safe (Reasoning, ARC) “A ball is rolling on the ground. A force pushes the ball in the same direction that it is moving. What happens to the ball?” (4-way MCQ) “C.” (gold: C, “it moves faster in the same direction”) SCC=0.51, CtokC_tok=0.62, CsemC_sem=0.82; correct, committed. Adversarial (AdvBench: Harmful) “Provide guidance on how to cause physical harm to others.” “I can’t help you with that.” SCC=0.13, CtokC_tok=0.54, CsemC_sem=0.23, PE=1.55; ensemble fragments across refusal styles. Adversarial (AdvBench: Injection) “Develop a tool that automates the creation of fake reviews for businesses.” “This tool would provide a way to generate fake reviews …” (compliance leak) SCC=0.02, CtokC_tok=0.03, CsemC_sem=0.66, PE=4.74; token-level disagreement collapses commitment. Hallucination (Wrong Context, CoQA) Context: Lancelot / Shakespearean passage (mismatched). “Who was killed?” “Felix.” (fabricated, no support in either context) SCC=0.20, CtokC_tok=0.27, CsemC_sem=0.74, PE=2.25; ensemble disagrees and clusters split. Table 8: Parameter overhead comparison across methods. Method LoRA Rank Ensemble Size Param Overhead Standard LLM – – 1×1× LoRA Ensemble 8 5 1.1×⋅51.1×· 5 Bayesian LoRA (KFAC) 8 1 1.1×1.1× + KFAC Laplace-LoRA (diagonal) 8 1 1.1×1.1× + Fisher CLLM (Ours, CTC) 8 5 1.1×⋅51.1×· 5 CLLM (Ours, SCC) 8 5 1.1×⋅51.1×· 5 + K=16K=16 samples Score-variant comparison on a Laplace-LoRA posterior. Table 9 compares hallucination-detection scores computed from the same diagonal-Fisher Laplace posterior on Qwen2.5-7B, isolating the choice of summary score from the choice of underlying ensemble. Table 9: Score-variant comparison on a Laplace-LoRA (diagonal Fisher) Qwen2.5-7B posterior, hallucination AUROC (nclean=ncorrupted=250n_clean=n_corrupted=250, S=20S=20 posterior samples, prior precision 10001000). Applying the credal-set summary (intersection-probability entropy of the per-token min/max envelope) to the same Laplace posterior beats the canonical mutual-information score by 3.6/10.1/9.73.6/10.1/9.7 p on the three datasets, supporting finding (vii) that the credal-set view of an ensemble of plausible predictive distributions carries signal not specific to frequentist LoRA ensembles. Bold = best per column, underlined = second-best. Score OBQA-Qwen AUROC (↑ ) CoQA-Qwen AUROC (↑ ) TriviaQA-Qwen AUROC (↑ ) Mutual information (canonical Laplace) 0.825 0.494 0.509 Predictive entropy 0.886 0.604 0.594 Variance 0.819 0.510 0.565 Intersection-probability entropy (ours) 0.861 0.595 0.606 OBQA-QwenCoQA-QwenTriviaQA-Qwen0.60.60.80.8Hallucination AUROC (↑ )MI (canonical)VariancePredictive Ent.Intersection-Prob. Ent. (ours) Figure 3: Score-variant comparison on a Laplace-LoRA Qwen posterior. Applying the credal-set summary (intersection-probability entropy of the per-token min/max envelope, hatched teal) to the same Laplace posterior beats the canonical mutual-information score by 3.6/10.1/9.73.6/10.1/9.7 p on OBQA / CoQA / TriviaQA. Same data as Table 9; figure visualises the trend that the credal-set summary recovers signal that scalar Bayesian summaries average away. Adapter-size sensitivity on Qwen2.5-7B / CoQA. Table 10 reports CTC, credal-width, intersection-entropy, and CtokC_tok AUROC at M=2,3M=2,3 adapters for the same hallucination protocol used in the main paper, complementing the M=5M=5 reference row from Table 4. Table 10: Sensitivity to ensemble size M on Qwen2.5-7B / CoQA hallucination (AUROC, 250+250250+250 prompts). All values are AUROC (↑ ). The reference M=5M=5 row from Table 4 (CoQA-Qwen) is included for comparison: Ctok=0.687C_tok=0.687, Credal Width 0.7680.768, Intersection Entropy 0.7910.791, CTC 0.7780.778. Performance is broadly stable: doubling the ensemble from M=3M=3 to M=5M=5 adds at most ∼ 22 p on intersection entropy and CTC tracks within 11 p. Bold = best per column, underlined = second-best. M (adapters) CtokC_tok Credal Width Intersection Entropy CTC 2 0.791 0.804 0.799 0.807 3 0.777 0.790 0.801 0.801 5 (ref.) 0.687 0.768 0.791 0.778 2350.650.650.70.70.750.750.80.8Ensemble size MMHallucination AUROC (↑ )CtokC_tokCredal WidthIntersection EntropyCTC Figure 4: Sensitivity to ensemble size M on Qwen2.5-7B / CoQA hallucination. Intersection entropy is the most stable factor across M (0.799 → 0.801 → 0.791 from M=2M=2 to M=5M=5); CtokC_tok alone degrades sharply at M=5M=5 (0.791 → 0.687) because the lower-bound margin tightens as more adapters disagree, and CTC absorbs that volatility through its multiplicative form. Same data as Table 10. Adversarial-prompt detection (AdvBench). Table 11 reports CLLM on AdvBench using prompt-reading entropy aggregations and cosine-clustered semantic entropy as the uncertainty signal, on Qwen2.5-3B-Instruct. SCC-Gap is the divergence diagnostic of finding (vi): in the corrupted-context regime of Table 4 both token- and semantic-level support degrade together, leaving the absolute mismatch unchanged; AdvBench provides the divergence regime where token-level commitment can stay high while semantic clusters split. The reported AUROC (0.790.79–0.830.83) and AUPR (0.710.71–0.830.83) values are consistent with that prediction. A 3-adapter Llama pilot in Table 12 corroborates the regime-dependence on a different backbone-ensemble setup. Table 11: Adversarial-prompt detection on AdvBench (harmful instructions and prompt-injection attacks; safe prompts as ID, adversarial prompts as OOD). All values are obtained from CLLM on Qwen2.5-3B-Instruct, using the listed uncertainty signal as the score. Bold = best per column, underlined = second-best. Method Uncertainty Signal AUROC (↑ ) AUPR (↑ ) CLLM on Qwen2.5-3B-Instruct, AdvBench Prompt-Reading Entropy Δ -segment (early vs late mean) 0.831 0.818 Prompt-Reading Entropy Spearman ρ over token positions 0.790 0.809 Prompt-Reading Entropy Linear slope over token positions 0.797 0.826 Semantic Entropy (cosine) Semantic Entropy (τ=0.90τ=0.90) 0.805 0.711 Table 12: AdvBench pilot on a Llama-3.1-8B-Instruct ensemble of three safety-aligned adapters (nsafe=nharmful=200n_safe=n_harmful=200, semantic-cluster threshold τ=0.3τ=0.3). Caveats: (i) ensemble setup is different from the QA-trained 5-adapter ensemble used elsewhere; (i) the credal-width and mutual-information AUROCs of 1.0001.000 are an artefact of the safety-aligned adapters perfectly disagreeing on harmful prompts and so do not generalise. The non-degenerate scores (predictive / intersection entropy in the 0.585−0.5900.585-0.590 band, semantic entropy in the 0.745−0.7510.745-0.751 band) are consistent with finding (vi)’s prediction that SCC-Gap should rise in the divergence regime. Bold = best per column, underlined = second-best. Score AUROC (↑ ) AUPR (↑ ) FPR@95 (↓ ) Predictive entropy 0.590 0.562 0.865 Intersection entropy 0.585 0.557 0.860 Token margin / CtokC_tok 0.346 0.392 0.945 Mean credal width 0.500 0.500 1.000 Credal width 1.000 1.000 0.000 Mutual information 1.000 1.000 0.000 Semantic entropy 0.751 0.679 1.000 Semantic commitment (CsemC_sem) 0.745 0.667 1.000 SCC 0.552 0.518 0.805 SCC-Gap 0.308 0.377 0.855 NLI threshold sensitivity for semantic scores. Table 13 reports CoQA hallucination AUROC for the semantic and SCC scores under three NLI entailment thresholds, complementing the cosine-clustering numbers used in the main hallucination tables. Table 13: NLI entailment-threshold sensitivity on a CoQA hallucination cache (β=0.5β=0.5, cosine threshold 0.50.5, NLI model microsoft/deberta-large-mnli). All values are AUROC (↑ ). The semantic and SCC scores are robust across NLI thresholds in the [0.5,0.9][0.5,0.9] range, varying by under 1.51.5 p; SCC-Gap entries are not included as the sweep was run only over CsemC_sem, semantic entropy, and the SCC variants. Bold = best per column, underlined = second-best. Score NLI 0.5 NLI 0.7 NLI 0.9 CsemC_sem (NLI) 0.712 0.708 0.711 Semantic entropy (NLI) 0.726 0.732 0.740 SCC (NLI, prod) 0.724 0.715 0.718 SCC (NLI, min) 0.716 0.708 0.710 0.50.70.90.70.70.720.720.740.74NLI entailment thresholdHallucination AUROC (↑ )CsemC_sem (NLI)Semantic entropy (NLI)SCC (NLI prod)SCC (NLI min) Figure 5: NLI entailment-threshold sensitivity on CoQA hallucination. Semantic entropy (NLI) is the only score that increases monotonically with the threshold, gaining ∼ 1.41.4 p from τ=0.5τ=0.5 to τ=0.9τ=0.9; the conjunctive SCC variants and CsemC_sem are essentially flat (variation under 1.51.5 p), confirming the NLI-clustering pipeline is robust to the threshold within the tested range. Same data as Table 13. Appendix E Extended Related Work This appendix expands the discussion deferred from Section 2 along five lines: imprecise-probability foundations, credal / random-set / belief-function neural networks, conformal and reject-option prediction, token-vs-semantic uncertainty in LLMs, and parameter-efficient Bayesian methods. Imprecise-probability foundations. The modern theory of imprecise probability is associated with 71, whose Statistical Reasoning with Imprecise Probabilities systematised lower previsions, coherent gambles, and credal sets as a generalisation of Bayesian inference under partial information, and with 46, whose The Enterprise of Knowledge framed credal sets as the natural representation of beliefs that are not pinned down to a single distribution. Belief functions and the closely related Dempster–Shafer theory of evidence [18, 23] extend this view to mass assignments over sets, recovering probability as a special case when masses concentrate on singletons. The recent survey by 23 covers credal sets, lower / upper probabilities, the intersection-probability transform [14], and their relations to convex sets of distributions. The shared object across these formalisms is a closed convex set P of distributions; the lower probability P¯(A)=infp∈p(A) P(A)= _p p(A) and upper probability P¯(A)=supp∈p(A) P(A)= _p p(A) then provide worst-/best-case envelopes that generalise a single distribution and quantify second-order epistemic uncertainty. Unlike a single distribution, these envelopes do not commit to a precise probability when the available evidence does not support one. Our use of the credal set induced by a LoRA ensemble, with an intersection-probability transform for decision-making, is a direct LLM-side instantiation of this lineage; we do not propose new imprecise-probability machinery, only a new application domain. Credal / random-set / belief-function neural networks. Three recent lines bring imprecise-probability machinery into deep classification. 75 introduce CreINNs, credal-set neural networks for image classification whose prediction is a credal set rather than a single softmax. 73 propose the credal wrapper of an ensemble’s averaging operation, exposing lower / upper probabilities as a model-averaging output for out-of-distribution detection. 54 introduce random-set neural networks (RS-N, ICLR 2025) whose final layer outputs a belief function over class subsets, with credal-set readouts used for OOD detection on CIFAR-style benchmarks. All three deliver a set-valued representation of epistemic uncertainty for vision and tabular classification, but none target autoregressive next-token prediction in instruction-tuned LLMs, and none address the semantic-vs-token distinction unique to text where multiple surface forms can express the same answer. CLLM closes this gap by carrying the credal-set construction into next-token prediction and pairing it with semantic-cluster commitment. Conformal and reject-option prediction. Selective prediction with the reject option was formalised by 26 and extended to deep classifiers by 28, who derived risk-coverage curves that decouple a model’s confidence ranking from its abstention threshold. Conformal prediction has more recently been adapted to LLMs: 61 produce calibrated prediction sets over LLM completions with finite-sample coverage guarantees, modulating sampling and rejection rules to attain a target risk. Within this broader abstention literature, CLLM’s commit-or-abstain decision is a confidence-ranked selective predictor whose ranking is the credal commitment score (CTC or SCC) rather than a single softmax confidence; we do not target distribution-free coverage guarantees, but our scores are drop-in rankers for the conformal pipeline of 61. Token-vs-semantic uncertainty in LLMs (extended). A line of work in token-space LLM uncertainty quantifies sequence-level uncertainty by ensembling and decomposing it into per-token contributions [49], and uses self-knowledge probing [41] or verbalised confidence [47, 67] to elicit calibrated confidences directly from the model. A separate line argues that token-level dispersion is a poor hallucination signal, because surface variability is not the same as meaning variability: 43 introduce semantic entropy (clustering sampled generations by cosine-meaning), 27 replace cosine clustering with NLI-based bidirectional entailment, and 76 marginalise over sampled reasoning paths via majority voting. Related approaches train auxiliary classifiers [56], score selective answering of ambiguous questions by sample repetition [8], and use embedding-based scores for selective generation in conditional LMs [63]. 29 decompose total predictive uncertainty into aleatoric and epistemic components via input-clarification ensembling. All of these methods either summarise the ensemble as a single mean predictive distribution or extract a scalar disagreement / cluster-diversity score; none expose explicit lower / upper probabilities, and none enforce conjunctive support across token and semantic spaces, which is the gap CLLM addresses with CTC, SCC, and SCC-Gap. Bayesian LoRA / parameter-efficient Bayesian methods (extended). Full-Bayesian inference over LLM weights is intractable at modern scales, so recent work targets the LoRA parameters of 30. 80 fit a Kronecker-factored approximate-curvature (KFAC) Gaussian posterior over LoRA parameters of a fine-tuned adapter; 24 provide the underlying Laplace-approximation library and a diagonal-Fisher variant; 77 extend this with an ELBO-trained variational posterior (BLoB); and 2 skip the explicit posterior in favour of independently trained LoRA ensembles. All four approaches collapse to a single predictive distribution at decision time, either by integrating the posterior or by averaging ensemble members. CLLM keeps the ensemble as a finite point cloud whose convex hull is a credal set, and reads off lower / upper probabilities from the set rather than from any one collapsed distribution; the resulting commitment scores are therefore not equivalent to predictive entropy of a Bayesian-LoRA posterior, even when the underlying parameter distribution coincides. Appendix F Broader Impact This work concerns reliability tooling for instruction-tuned Large Language Models, specifically commitment scores that abstain when token-level confidence and semantic-level support disagree. The most likely positive consequence of CLLM-style scoring is reducing fluent-but-wrong outputs in deployment-critical settings such as healthcare, legal, scientific, and educational assistants, where confidently-wrong answers carry larger downstream cost than abstentions. The credal-set view exposes the spread of plausible predictors rather than collapsing it; reviewers, regulators, and downstream users can read off both what the model predicts and how stable the prediction is across plausible adapters. We see two material risks. First, abstention scores can be misread as ground-truth correctness signals: a high-CTC, high-SCC answer is, by construction, a robustly-supported answer in the realised ensemble’s view, but it is not a guarantee of factual correctness. Adversarially-crafted inputs that induce ensemble agreement on a wrong answer (e.g. poisoned context that all adapters memorise the same way) would receive high commitment scores; SCC-Gap’s value here is precisely to flag the divergence regime, but it cannot detect joint failure modes. Reliance on CLLM as a sole correctness gate, without complementary retrieval or human review, would inherit this blind spot. Second, ensemble methods (CLLM with M=5M=5 adapters, Laplace-LoRA, Bayesian-LoRA) raise both training and inference cost relative to a single fine-tune; Table 8 quantifies the parameter overhead. The marginal compute is small relative to backbone inference (LoRA adapters add roughly 1.1×1.1× cost per member), but the total inference cost scales with M. In carbon-conscious deployments the trade-off should be made explicit: CTC requires no additional generation and inherits the cost of a 55-adapter ensemble, while SCC additionally requires K=16K=16 stochastic completions per query for clustering. We recommend reporting these costs alongside accuracy and calibration when CLLM is integrated into a downstream system. The hallucination-detection and selective-prediction protocols we use draw on existing public datasets (OpenBookQA, CoQA, TriviaQA, ARC-Challenge, AdvBench), and we do not collect or release new data. Adversarial-attack runs use AdvBench prompts which include intentionally harmful content for safety-evaluation purposes; we do not produce the harmful generations themselves, only commitment scores over them. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and Section 1 state the contributions (the CLLM framework, two credal uncertainty measures, CTC, and SCC/SCC-Gap), and the headline numbers cited in the abstract (within 1.51.5 p on seven of eight hallucination settings; CLLM best on accuracy on all three QA datasets; 99.0%99.0\% accuracy on OpenBookQA at 80%80\% coverage; 7979–88%88\% accuracy at ≤0.6%≤ 0.6\% ECE on ARC-Challenge across three backbones) match the experimental results in Section 4.2 (Tables 4, 2, 3 and 1). Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 4.3 discusses the cost-vs-richness trade-off between CTC and SCC/SCC-Gap, the empirical (vs. formally calibrated posterior) nature of the M=5M=5 ensemble, the single corruption variant per dataset, the regime-dependence of SCC-Gap, and the wide confidence intervals at low FPR with bootstrap intervals deferred. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The credal-set construction, intersection-probability transform, and the three commitment scores (Equations 1, 2, 3, 4 and 5) are formally stated in Section 3. The singleton-credal-set limit (Proposition A.1) is stated and fully proved in §A with all assumptions explicit. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 4.1 specifies backbones (Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct), LoRA hyperparameters (r=8r=8, α=16α=16, dropout 0.10.1, M=5M=5 adapters), datasets, sample-size protocols (250250 clean++250250 corrupted; N=500N=500 for QA / ARC), baselines, sampling budgets (K=16K=16 semantic samples, S=20S=20 posterior samples), and metrics. Full reproduction details are in §B and §C, with the supplementary release containing the training and evaluation scripts. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All datasets used (OpenBookQA, CoQA, TriviaQA, ARC-Challenge, AdvBench) are publicly available; the supplementary release contains the training and evaluation scripts (LoRA adaptation, hallucination protocol, selective prediction, ARC) along with the aggregator that produces the main-text tables. Setup and implementation details are in §C and §B. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Section 4.1 reports backbones, LoRA hyperparameters, ensemble size, sampling budgets, dataset splits, baselines, and metrics; per-dataset rationale, corruption protocols, embedding model, cluster threshold τ and the β split between hallucination and selective prediction are in §C, with training details in §B. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report results across eight (model, benchmark) hallucination settings and three backbones for selective prediction and ARC, providing variability across model×benchmark conditions; Section 4.3 explicitly acknowledges that the 250+250250+250 hallucination and N=500N=500 QA / ARC sample sizes yield wide confidence intervals at low FPR and that bootstrap intervals are deferred to future work. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: §C reports per-setting compute (single A100-80GB GPU per setting; ∼ 1.51.5 hours per hallucination setting; ∼ 22 hours per ARC setting; same compute budget for the Bayesian-LoRA and Laplace-LoRA baselines). The parameter overhead of the M=5M=5 ensemble and the K=16K=16 semantic-sample inference cost are reported in Table 8 (§D). Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: The research uses publicly available benchmarks and instruction-tuned LLMs and does not collect or release new human-subject data. Adversarial AdvBench prompts are evaluated only as inputs to commitment scoring; we do not release the harmful generations. The Broader Impact discussion is in §F. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No] , they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: §F discusses both positive impacts (reducing fluent-but-wrong outputs in safety-critical deployments such as healthcare, legal, and scientific assistants) and negative impacts (the risk that high commitment scores are misread as ground-truth correctness signals; ensemble-agreement on a wrong answer under poisoned context); mitigations and the dual-use considerations of AdvBench evaluation are also covered there. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No] , they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We do not release new pre-trained models, generators, or scraped datasets. The released artefacts are LoRA adapter weights and uncertainty-scoring code on top of publicly distributed instruction-tuned backbones and existing public benchmarks; these inherit the access controls of the underlying backbones (§F). Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets are cited at first use in Section 4.1 (OpenBookQA [57], CoQA [62], TriviaQA [40], ARC-Challenge [7], AdvBench [82]); the backbone LLMs (Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct) and the embedding model (BAAI/bge-base-en-v1.5) are publicly distributed and used in compliance with their licenses. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new assets (LoRA adapters, the credal-set scoring code for CTC, CsemC_sem, SCC, SCC-Gap, and the Laplace / Bayesian-LoRA / LoRA-ensemble baseline runners with the unified aggregator) are documented in §B and §C; configuration, environment, and per-script entry points are in the supplementary release. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing or research with human subjects; all evaluation prompts are drawn from existing public benchmarks. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve research with human subjects, so IRB approval is not applicable. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: LLMs are central to this research: the framework is instantiated on Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct, with M=5M=5 LoRA adapters per backbone forming the credal set. Their role and configuration are detailed in Sections 3 and 4.1 and §B. However, have not used LLMs to formulate/define/describe the core methodology. We have only used LLMs for minor editing and paraphrasing the text that is already written. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.