Paper deep dive
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
Debarpan Bhattacharya, Malay Phadke, Sriram Ganapathy
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/2/2026, 4:12:46 AM
Summary
The paper introduces BiG-SURE, a black-box uncertainty estimation framework for Large Language Models (LLMs) and Vision-Language Models (VLMs). It utilizes cross-temperature semantic agreement by constructing a bipartite graph between stable low-temperature anchor responses and diverse high-temperature probe responses. The uncertainty score (SURE) is derived from the normalized squared spectral energy of the semantic similarity matrix between these anchors and probes. BiG-SURE demonstrates superior abstention AUROC performance compared to prior methods across text, multilingual, and multimodal QA tasks.
Entities (18)
Relation Signals (15)
BiG-SURE → appliesto → VLMs
confidence 95% · deploying large language models (LLMs) and vision-language models (VLMs)
BiG-SURE → appliesto → LLMs
confidence 95% · Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs)
BiG-SURE → evaluatedon → OKVQA
confidence 95% · We use the OK-VQA dataset... for visual question-answering tasks.
BiG-SURE → evaluatedon → SciQ
confidence 95% · We experiment with multilingual SciQ
BiG-SURE → evaluatedon → Trivia-QA
confidence 95% · We evaluate BiG-SURE on text QA... Trivia-QA
BiG-SURE → evaluatedon → SVAMP
confidence 95% · We evaluate BiG-SURE on text QA... SVAMP
BiG-SURE → uses → Bipartite Graph
confidence 95% · It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores
Pixtral-12B-2409 → evaluatedwith → BiG-SURE
confidence 90% · We use vision-language models (VLMs) - Pixtral-12B-2409
Qwen-2.5-7b-Instruct → evaluatedwith →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.
Tags
Links
- Source: https://arxiv.org/abs/2608.30646v2
- Canonical: https://arxiv.org/abs/2608.30646v2
Trouble viewing inline? Open PDF directly →
Full Text
97,738 characters extracted from source content.
Expand or collapse full text
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs Debarpan Bhattacharya Malay Phadke Sriram Ganapathy Affiliation: LEAP Lab, Electrical Engineering, Indian Institute of Science Affiliation: TANUH AI Centre of Excellence, Indian Institute of Science Correspondence:debarpanb, malaymilindp, sriramg@iisc.ac.in https://iiscleap.github.io/projects/BiG-SURE| https://github.com/iiscleap/BiG-SURE Abstract Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model’s stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings. **footnotetext: Equal contribution. 1 Introduction The development of large language models (LLMs)- ChatGPT Achiam et al. (2023), Gemini Team et al. (2023), Llama Touvron et al. (2023), Mistral Jiang et al. (2023), Qwen Qwen et al. (2025) and others - has advanced various natural language tasks. However, multimodal and multilingual domains still underperform the English textual counterparts, affected by hallucinations and mispredictions Wang et al. (2025); Song et al. (2025). Significant efforts in alignment, grounding, and training on large-scale data are underway to address this gap. However, in safety-critical scenarios, like finance, legal and health, the mis-predictions and hallucinations of the LLMs can result in substantial loss and hence, calibration and uncertainty measures are the key bottlenecks that dictate LLM deployment readiness Bedi et al. (2025). Figure 1: (Top-panel) Bipartite semantic agreement between cross-temperature responses for an example input. (Bottom-panel) SURE gives a substantial improvement in abstention AUROC over prior works on Trivia-QA. The AUROCs are averaged over 66 LLMs (Table 2). The early black-box approaches for uncertainty estimation focus on the semantic clustering and entropy Kuhn et al. (2023); Farquhar et al. (2024). Further improvements are achieved using Von-Neumann entropy on kernelized response similarities Nikitin et al. (2024), that are devoid of the semantic clustering Nguyen et al. (2025). The graph-based approaches consider the responses to be different nodes of a graph and determine the effective number of semantic clusters Lin et al. (2024). However, all the existing approaches attempt to extract the signal of uncertainty from the responses sampled at a fixed temperature. We propose BiG-SURE, a black-box uncertainty estimation framework based on cross-temperature bipartite semantic agreement. Low-temperature responses are used as stable anchors, while high-temperature responses act as probes that reveal semantic drift. SURE constructs an anchor–probe bipartite semantic similarity graph and uses its normalized squared spectral energy as a confidence score, with uncertainty defined as its complement. The key contributions of this work are: • We identify cross-temperature response dynamics as a useful black-box signal for uncertainty estimation in LLMs and VLMs. • We propose a bipartite anchor–probe formulation that compares stable low-temperature generations against diverse high-temperature probes. • We define SURE as a normalized squared spectral-energy score over the anchor–probe similarity matrix, producing a bounded uncertainty estimate. • We benchmark SURE across text-only, multilingual, and multimodal QA tasks, showing improved abstention AUROC over prior black-box uncertainty estimators. Figure 1 illustrates the intuition behind SURE. We sample low-temperature responses as stable anchors and high-temperature responses as diverse probes, then measure their bipartite semantic agreement. Reliable predictions exhibit strong cross-temperature agreement: the probes remain semantically aligned with the anchors despite surface-level variation. Uncertain or hallucinated predictions instead show semantic drift, weakening the anchor–probe agreement. SURE converts this cross-temperature agreement structure into a normalized confidence score, whose complement serves as uncertainty; the bottom panel of Figure 1 shows that this signal improves abstention performance over prior estimators. 2 Problem Statement We consider black-box uncertainty estimation for an LLM parameterized by θ, where internal states, logits, and token probabilities are not accessible. Given an input x, the goal is to estimate an uncertainty score (x,θ)U(x;θ) for the model’s generated response y. Let T>0T>0 denote the sampling temperature used to generate multiple responses from the model. We define two response sets: a set of low-temperature anchor responses =a1,…,aMA=\a_1,…,a_M\, and a set of high-temperature probe responses ℋ=h1,…,hNH=\h_1,…,h_N\. The probes may be sampled either from the original input x or from semantically equivalent input augmentations of x, such as paraphrases. Our objective is to estimate (x,θ)U(x;θ) from the cross-temperature response sets (,ℋ)(A,H). 3 Cross-temperature Spectral Energy 3.1 Definition We define a weighted Bipartite Graph (BiG) G=(,ℋ,W)G=(A,H,W) characterized by the Bi-adjacency Matrix W∈ℝM×NW ^M× N, where WijW_ij represents the semantic similarity between anchor aia_i and probe hjh_j: Wij=sem-sim(ai,hj),Wij≥0W_ij= sem-sim(a_i,h_j), W_ij≥ 0 (1) We hypothesize that the cross-temperature dynamics between anchor and probe responses capture the model uncertainty. We define the spectral energy of their agreements using W as, ESURE(W,θ)=∑i=1minM,Nσi2=∑i=1M∑j=1NWij2E_ SURE(W;θ)= _i=1^min\M,N\ _i^2= _i=1^M _j=1^NW_ij^2 (2) where σi _i are singular values of W. The equality shows that ESURE(W,θ)E_ SURE(W;θ) is same as the squared Frobenius norm of W. We define normalized confidence and compute the Semantic Uncertainty and Reliability Estimate (SURE) score as, Enorm(W,θ) E_ norm(W;θ) =ESURE(W,θ)MN = E_ SURE(W;θ)MN (3a) norm(W,θ) _ norm(W;θ) =1−Enorm(W,θ) =1-E_ norm(W;θ) (3b) Mathematically, the measure satisfies 0≤norm(W,θ)≤10 _ norm(W;θ)≤ 1 (Appendix A.1). We formally show that SURE also possesses the properties that make it a valid uncertainty measure (Section 3.4). 3.2 Illustration As shown in Figure 2, when an LLM generates an accurate response to a factual question (in greedy decoding mode (T=0T=0)), the mutual relationship between low and high temperature responses exhibits different properties compared to the setting when the LLM generates an inaccurate response. When it generates an accurate response, the low and high-temperature responses are semantically similar, which indicates a low value of normU_ norm. On the other hand, when the LLM generates an incorrect response, the high-temperature responses vary significantly from the low-temperature responses, resulting in a high value of normU_ norm. Figure 2: Cross-temperature semantic agreement reflects prediction reliability. For a correct greedy prediction (left), low- and high-temperature samples are semantically consistent, resulting in strong bipartite connectivity and high normalized spectral energy, and vice-versa. 3.3 Implementation Given an input prompt x, we sample M low-temperature responses at TLT_L and N high-temperature responses at THT_H, with TH>TLT_H>T_L. We construct a bipartite graph between cross-temperature samples, where the edge weights are based on the pairwise semantic similarity scores, and form the matrix W∈ℝM×NW ^M× N. We compute the SURE metric based on Eqn. 3. Note that the computation of uncertainty is unsupervised and does not require ground truth labels. The algorithmic implementation is shown in Algorithm 1. Algorithm 1 Computation of normU_norm 1: Input prompt x; LLM θ; low temperature TLT_L; high temperature TH>TLT_H>T_L; 2: # of low-temperature responses M; # of high-temperature responses N; semantic similarity operator sem-sim(⋅,⋅) sem-sim(·,·). 3: Normalized uncertainty norm(W,θ)∈[0,1]U_ norm(W;θ)∈[0,1]. 4: ←Sample(θ,x,TL,M)A← Sample(θ,x,T_L,M) 5: ℋ←Sample(θ,x,TH,N)H← Sample(θ,x,T_H,N) 6: Initialize W∈ℝM×NW ^M× N 7: for i←1i← 1 to M, ai∈a_i do 8: for j←1j← 1 to N, hj∈ℋh_j do 9: Wij←sem-sim(ai,hj)W_ij← sem-sim(a_i,h_j) 10: end for 11: end for 12: ESURE(W,θ)←∑i=1minM,Nσi(W)2E_ SURE(W;θ)← _i=1 \M,N\ _i(W)^2 13: Enorm(W,θ)←ESURE(W,θ)MNE_ norm(W;θ)← E_ SURE(W;θ)MN 14: norm(W,θ)←1−Enorm(W,θ)U_ norm(W;θ)← 1-E_norm(W;θ) 15: return norm(W,θ)U_ norm(W;θ) 3.4 SURE - A valid uncertainty estimator We show that norm(W,θ)U_ norm(W;θ) is a valid estimator of semantic uncertainty for LLMs: • Semantic identity: norm=0U_ norm=0 iff all the sampled responses of an LLM are semantically similar. • Semantic disjointness: norm=1U_ norm=1 iff all the high temperature responses are semantically different from low temperature responses. • Monotonicity: For a given number of sampling responses, normU_ norm increases if the number of semantically unique sequences increases in the prediction set. These properties are discussed in detail in Appendix A.1. 4 Prior Works Several prior works attempt black-box uncertainty estimation for LLMs, like semantic entropy Kuhn et al. (2023); Farquhar et al. (2024), evidential semantic entropy Kunitomo-Jacquin et al. (2026), SPUQ Gao et al. (2024), KLE Nikitin et al. (2024), SNNE Nguyen et al. (2025), graph-based Lin et al. (2024) and distillation-based Phillips et al. (2026) approaches. Only a handful of recent works attempt uncertainty estimation for multimodal LLMs; some of them are FESTA Bhattacharya et al. (2025b), TREA Bhattacharya et al. (2025a), Uncertainty-O Zhang et al. (2025) and UMPIRE Lau et al. (2025). Discrete Semantic Entropy (DSE): Directly computing lexical entropy Kadavath et al. (2022) is not applicable for LLM-generated sequences, as the lexical variability can obscure semantic uncertainty. Discrete Semantic Entropy (DSE) addresses this issue by performing semantic clustering and computing entropy over the clusters Kuhn et al. (2023); Farquhar et al. (2024). In contrast, our work does not require semantic clustering. Graph-based approaches: Another family of methods explores high-temperature sampled outputs as nodes in a graph, with edges defined by soft pairwise similarity. Various graph-derived uncertainty scores include NumSet (discretized similarity), EigV (largest eigenvalue of a degree-based matrix), and Deg (negative trace of the degree matrix), as studied in Lin et al. (2024). Lexical Similarity (LexSim): LexSim computes a pairwise similarity matrix using a lexical overlap metric such as ROUGE-L Lin (2004), and computes confidence as a normalized aggregate of off-diagonal similarities Fomicheva et al. (2020). Kernel Language Entropy (KLE): KLE constructs unit-trace semantic kernels capturing pairwise similarity between generations, and defines uncertainty via the von Neumann entropy of the kernel matrix Nikitin et al. (2024). Semantic Nearest Neighbour Entropy (SNNE): SNNE Nguyen et al. (2025) avoids explicit clustering by operating directly on pairwise similarity of sampled responses. SNNE and KLE solely rely on high-temperature samples. Table 1 highlights the main distinction between prior black-box uncertainty estimators and SURE. Prior approaches largely rely on fixed-temperature sampling responses and then derive uncertainty from them. SURE departs from this paradigm by using agreement dynamics between low-temperature anchors and high-temperature probes to quantify uncertainty. SURE also incorporates semantically equivalent input augmentation within the same framework. To the best of our knowledge, SURE is the first attempt to leverage cross-temperature sampling dynamics in LLM responses for uncertainty quantification. Table 1: Conceptual comparison of SURE with prior works on black-box, open-text uncertainty estimation. Method Sampling type Input Aug. Quantification DSE High-temperature sampling (fixed T) No Semantic entropy over clustered responses Graph-based High-temperature sampling (fixed T) No Graph statistics (e.g., degree, eigenvalue, number of sets) LexSim High-temperature sampling (fixed T) No Lexical similarity across sampled responses KLE High-temperature sampling (fixed T) No Von Neumann entropy of semantic kernel SNNE High-temperature sampling (fixed T) No Nearest-neighbour semantic entropy SURE Low-T anchors + high-T probes Yes Cross-temperature agreement 5 Experimental Setup 5.1 Tasks and models Textual QA: We experiment with text-based factual question answering and math word question answering tasks. We use Trivia-QA Joshi et al. (2017) and SVAMP Patel et al. (2021) datasets. We use 66 LLMs for these datasets - Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct Grattafiori et al. (2024), Qwen-2.5-7B-Instruct and Qwen-2.5-72B-Instruct Qwen et al. (2025), Gemma-3-12b-it and Gemma-3-27b-it Team et al. (2025). Multilingual QA: We experiment with multilingual SciQ Xue et al. (2025) in 4 languages as English(en), French(fr), Japanese(ja) and Mandarin(zh). We use two LLMs - Aya-expanse-8b Dang et al. (2024) and Apertus-8B-Instruct-2509 Hernández-Cano et al. (2025), because of their explicit multilingual training and superior performance. Multimodal QA: We use the OK-VQA dataset Marino et al. (2019) for visual question-answering tasks. We use vision-language models (VLMs) - Pixtral-12B-2409 Agrawal et al. (2024), Llava-v1.6-mistral-7b-hf Liu et al. (2023) and Qwen3-VL-8B-Instruct Bai et al. (2025). 5.2 Sampling and input transformations Most of the prior works on sampling-based uncertainty estimation use temperature sampling. Input transformations that do not change input semantics have also been proven to be effective Gao et al. (2024). SURE uses semantics-preserving input transformations in addition to temperature sampling to obtain probe responses. For the original input x, M anchor responses are obtained by low-temperature (TLT_L) sampling of x. N probe samples are generated by high-temperature (THT_H) sampling of a mix of equivalent input transformations, xe\x^e\. In our experiments, M=3M=3, N=10N=10, TL=0.1T_L=0.1 and TH=1.0T_H=1.0 are used across all models and tasks. The N=10N=10 probe samples are obtained from 55 input transformations and high-temperature sampling twice from each. The equivalent samples for the textual inputs correspond to paraphrasing of the input. For multimodal inputs, we use both question paraphrases and mild image perturbations: mean blur with kernel size 55, affine rotation by ±10∘± 10 , and Gaussian pixel noise sampled from (0,10)N(0,10) followed by clipping to [0,255][0,255]. The exact prompts and implementation details of these input transformations are in Appendix A.6. 5.3 Semantic similarity backend module The semantic similarity function sem-sim(ai,hj)sem -sim(a_i,h_j) in Equation 1 is implemented using NLI-based entailment module. We feed the anchor-probe pair (ai,hj)(a_i,h_j) to the entailment model and obtain the softmax over the logits. The corresponding entailment probability forms the edge weight: Wij=sem-sim(ai,hj)=pentail(ai,hj)∈[0,1], [rgb]0,0,0W_ij=sem -sim(a_i,h_j)=p_entail(a_i,h_j)∈[0,1], where larger values indicate stronger semantic agreement between the anchors and probes. For text-QA and multimodal-QA, we use a DeBERTa-large NLI model He et al. (2021) 11 1 https://huggingface.co/microsoft/deberta-large-mnli as the entailment backend. For multilingual QA, we use a multilingual DeBERTa NLI model (mDeBERTa 22 2 https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7), which is trained on multilingual NLI data. We do not train or fine-tune either semantic-similarity model. Within each experimental setting, the entailment backend remains fixed across SURE and all similarity-based baselines for fair comparison. 5.4 Evaluation and performance metric We use the area under the receiver operating characteristic curve (AUROC) to evaluate the uncertainty estimators. Each uncertainty score is interpreted as an abstention score. First, every test prediction is labeled as correct or incorrect based on the match between the greedy prediction (T=0T=0 response) and the ground-truth answer. The matching is based on the standard evaluation protocols for textual QA and multimodal QA tasks, and we use Gemini-2.5-Flash Comanici et al. (2025) as a judge for multilingual QA (details in Appendix A.8). Then, the binary matching labels and abstention scores are used to compute the AUROC. 6 Results 6.1 Illustrative examples Figure 3 provides examples of cross-temperature response patterns captured by SURE. Across multilingual and multimodal QA examples, we observe a clear separation between reliable and unreliable predictions in terms of cross-temperature semantic agreement dynamics. Figure 3: Illustrative examples of cross-temperature semantic agreement in multilingual and multimodal QA. For each example, we show the greedy prediction together with low-temperature anchor samples and high-temperature probe samples. Responses aligned with the ground truth are marked in green, while responses that do not align are highlighted in red. Reliable predictions exhibit strong semantic agreement across temperatures, while unreliable predictions show semantic drift and disagreement. The quantitative evaluation of SURE in terms of abstention AUROC and a comparison with 77 existing works on black-box, open-text uncertainty quantification methods across multilingual and multimodal tasks is reported in Table 2. All prior works use N=10N=10 in Table 2. 6.2 Results on text-only QA tasks On the text-only QA benchmarks, SURE achieves the best average AUROC on both datasets. On Trivia-QA, SURE obtains an average AUROC of 0.753±0.0170.753± 0.017, improving over the strongest baseline average, Deg (0.722±0.0070.722± 0.007). On SVAMP, SURE obtains 0.877±0.0220.877± 0.022, outperforming the strongest baseline average, SNNE (0.846±0.0180.846± 0.018). The gains are also consistent across model families and scales. On Trivia-QA, SURE achieves the best performance for 55 out of 66 models, with the only exception being Llama-70B, where Deg performs better. On SVAMP, SURE achieves the best results for 55 out of 66 models, with Qwen-72B being the exception where SumEigv is strongest. These results indicate that cross-temperature anchor–probe agreement provides a robust uncertainty signal across both factual QA and math word-problem settings. Graph-based baselines such as SumEigv and Deg are competitive, especially on Trivia-QA, while SNNE is a competitive baseline on SVAMP; however, SURE achieves the best average performance across both. To understand the corresponding abstention behavior, AURC values are also reported in Appendix A.9. 6.3 Results on multilingual QA tasks On multilingual SciQ, SURE performs well across languages and LLMs, and achieves the best average AUROC on English (0.677±0.0150.677± 0.015), Japanese (0.685±0.0130.685± 0.013), and Mandarin (0.691±0.0120.691± 0.012). These results improve over the best baseline systems in three language settings. French SciQ is the only exception, where LexSim obtains the best average AUROC. Overall, SURE achieves the best average performance in 33 out of 44 multilingual language groups, and is especially consistent on Japanese and Mandarin for both Aya and Apertus. 6.4 Results on multimodal QA tasks On OKVQA, SURE achieves the best average AUROC of 0.752±0.0140.752± 0.014, outperforming the best baseline system, SNNE (0.735±0.0160.735± 0.016). SURE is also the best method for all three VLMs. This suggests that cross-temperature semantic agreement remains useful even when the input contains visual modalities. Summary. Across the seven dataset/language groups in Table 2, SURE achieves the best average AUROC in six groups: Trivia-QA, SVAMP, SciQ en, SciQ ja, SciQ zh, and OKVQA. Overall, they support the central hypothesis that cross-temperature semantic agreement provides an effective black-box uncertainty signal across text-only, multilingual, and multimodal QA settings. Table 2: Performance comparison of SURE with prior works in terms of abstention AUROC across text-only, multimodal, and multilingual QA datasets. The third column reports the task accuracy of greedy responses. The abstention AUROCs are presented with repeated-run statistics (mean ± std) across 55 seeds. The best result, in terms of mean AUROC, is highlighted in bold, while the second best result is underlined. Dataset [lang] Model Acc. Baselines (↑ ) SURE (↑ ) DSE SumEigv Deg NumSet LexSim KLE SNNE Trivia-QA [en] Qwen (7B) 0.495 0.786 ± 0.003 0.791 ± 0.005 0.793 ± 0.004 0.784 ± 0.005 0.778 ± 0.008 0.773 ± 0.007 0.783 ± 0.003 0.806 ± 0.014 Llama (8B) 0.661 0.731 ± 0.005 0.730 ± 0.005 0.740 ± 0.005 0.724 ± 0.007 0.736 ± 0.006 0.722 ± 0.003 0.736 ± 0.006 0.772 ± 0.021 Gemma (12B) 0.643 0.698 ± 0.010 0.697 ± 0.011 0.695 ± 0.011 0.700 ± 0.010 0.727 ± 0.014 0.689 ± 0.008 0.713 ± 0.014 0.789 ± 0.011 Gemma (27B) 0.755 0.629 ± 0.005 0.636 ± 0.005 0.634 ± 0.005 0.630 ± 0.005 0.657 ± 0.005 0.629 ± 0.006 0.617 ± 0.010 0.699 ± 0.017 Llama (70B) 0.767 0.712 ± 0.008 0.754 ± 0.011 0.756 ± 0.011 0.712 ± 0.008 0.737 ± 0.005 0.702 ± 0.010 0.715 ± 0.007 0.715 ± 0.009 Qwen (72B) 0.690 0.689 ± 0.012 0.722 ± 0.005 0.716 ± 0.007 0.680 ± 0.013 0.631 ± 0.009 0.695 ± 0.005 0.698 ± 0.012 0.740 ± 0.030 Avg. 0.668 0.708 ± 0.007 0.722 ± 0.007 0.722 ± 0.007 0.705 ± 0.008 0.711 ± 0.008 0.702 ± 0.006 0.710 ± 0.009 0.753 ± 0.017 SVAMP [en] Qwen (7B) 0.688 0.794 ± 0.027 0.790 ± 0.019 0.789 ± 0.019 0.791 ± 0.027 0.780 ± 0.020 0.788 ± 0.018 0.803 ± 0.029 0.804 ± 0.017 Llama (8B) 0.672 0.857 ± 0.022 0.844 ± 0.025 0.845 ± 0.025 0.855 ± 0.022 0.845 ± 0.016 0.861 ± 0.019 0.888 ± 0.020 0.890 ± 0.026 Gemma (12B) 0.730 0.779 ± 0.016 0.787 ± 0.012 0.790 ± 0.013 0.779 ± 0.015 0.783 ± 0.013 0.781 ± 0.012 0.794 ± 0.015 0.882 ± 0.021 Gemma (27B) 0.794 0.735 ± 0.014 0.751 ± 0.015 0.752 ± 0.015 0.735 ± 0.014 0.734 ± 0.015 0.733 ± 0.015 0.796 ± 0.016 0.879 ± 0.017 Llama (70B) 0.797 0.884 ± 0.011 0.882 ± 0.011 0.884 ± 0.011 0.882 ± 0.010 0.868 ± 0.009 0.881 ± 0.013 0.896 ± 0.010 0.905 ± 0.013 Qwen (72B) 0.849 0.893 ± 0.021 0.915 ± 0.020 0.914 ± 0.018 0.880 ± 0.022 0.800 ± 0.014 0.814 ± 0.027 0.897 ± 0.016 0.902 ± 0.039 Avg. 0.755 0.823 ± 0.018 0.828 ± 0.017 0.829 ± 0.017 0.820 ± 0.018 0.802 ± 0.015 0.810 ± 0.017 0.846 ± 0.018 0.877 ± 0.022 SciQ [en] Aya 0.786 0.591 ± 0.023 0.603 ± 0.040 0.613 ± 0.028 0.562 ± 0.026 0.591 ± 0.012 0.580 ± 0.008 0.537 ± 0.043 0.633 ± 0.004 Apertus 0.763 0.666 ± 0.047 0.682 ± 0.022 0.679 ± 0.014 0.603 ± 0.031 0.620 ± 0.020 0.631 ± 0.009 0.584 ± 0.035 0.722 ± 0.026 Avg. 0.775 0.628 ± 0.035 0.642 ± 0.031 0.646 ± 0.021 0.582 ± 0.028 0.606 ± 0.016 0.606 ± 0.008 0.561 ± 0.039 0.677 ± 0.015 SciQ [fr] Aya 0.675 0.647 ± 0.028 0.631 ± 0.017 0.639 ± 0.015 0.634 ± 0.014 0.647 ± 0.013 0.611 ± 0.024 0.635 ± 0.015 0.639 ± 0.015 Apertus 0.668 0.535 ± 0.022 0.573 ± 0.013 0.563 ± 0.017 0.528 ± 0.032 0.569 ± 0.018 0.527 ± 0.032 0.534 ± 0.039 0.542 ± 0.022 Avg. 0.672 0.591 ± 0.025 0.602 ± 0.015 0.601 ± 0.016 0.581 ± 0.023 0.608 ± 0.016 0.569 ± 0.028 0.585 ± 0.027 0.590 ± 0.019 SciQ [ja] Aya 0.539 0.586 ± 0.017 0.564 ± 0.033 0.573 ± 0.025 0.546 ± 0.027 0.508 ± 0.009 0.558 ± 0.030 0.541 ± 0.030 0.662 ± 0.010 Apertus 0.441 0.658 ± 0.007 0.678 ± 0.025 0.690 ± 0.015 0.639 ± 0.002 0.503 ± 0.014 0.678 ± 0.015 0.625 ± 0.016 0.707 ± 0.016 Avg. 0.490 0.622 ± 0.012 0.621 ± 0.029 0.632 ± 0.020 0.593 ± 0.015 0.506 ± 0.011 0.618 ± 0.022 0.583 ± 0.023 0.685 ± 0.013 SciQ [zh] Aya 0.556 0.602 ± 0.035 0.592 ± 0.028 0.593 ± 0.031 0.553 ± 0.034 0.500 ± 0.007 0.568 ± 0.021 0.552 ± 0.028 0.665 ± 0.012 Apertus 0.458 0.678 ± 0.017 0.689 ± 0.012 0.695 ± 0.016 0.645 ± 0.011 0.503 ± 0.009 0.695 ± 0.018 0.613 ± 0.016 0.717 ± 0.012 Avg. 0.507 0.640 ± 0.026 0.640 ± 0.020 0.644 ± 0.023 0.599 ± 0.023 0.502 ± 0.008 0.632 ± 0.019 0.583 ± 0.022 0.691 ± 0.012 OKVQA [en] Pixtral 0.630 0.731 ± 0.018 0.750 ± 0.013 0.739 ± 0.017 0.740 ± 0.017 0.728 ± 0.022 0.742 ± 0.017 0.743 ± 0.018 0.775 ± 0.008 LLaVa 0.643 0.740 ± 0.012 0.733 ± 0.007 0.755 ± 0.008 0.724 ± 0.013 0.738 ± 0.016 0.742 ± 0.011 0.755 ± 0.013 0.756 ± 0.022 Qwen 0.637 0.695 ± 0.012 0.711 ± 0.010 0.705 ± 0.012 0.693 ± 0.012 0.718 ± 0.010 0.711 ± 0.009 0.707 ± 0.016 0.725 ± 0.011 Avg. 0.637 0.722 ± 0.014 0.731 ± 0.010 0.733 ± 0.012 0.719 ± 0.014 0.728 ± 0.016 0.732 ± 0.012 0.735 ± 0.016 0.752 ± 0.014 7 Discussion 7.1 Spectral mode analysis The normalised spectral energy formulation of SURE enables the analysis in terms of singular-value spectrum. If the singular values of W∈ℝM×NW ^M× N are ordered as σ1≥⋯≥σmin(M,N) _1≥·s≥ _ (M,N), we define truncated spectral-energy scores to understand the energy distribution, ETop-k(W) E_Top-k(W) =∑r=1kσr2, = _r=1^k _r^2, UTop-k(W) U_Top-k(W) =1−ETop-k(W)MN. =1- E_Top-k(W)MN. Here, Top-1 uses only the largest singular value, while Top-2 uses the two largest singular values. Since our experimental setting uses M=3M=3 and N=10N=10, the rank of W is at most 33. We find that Top-1 and Top-2 closely match the full SURE score across both SVAMP and Trivia-QA, as detailed in Appendix Table 3. This indicates that most of the useful cross-temperature agreement energy is concentrated in the leading singular mode. When the model is certain, high-temperature probes align with this dominant anchor mode. When the model is uncertain, the high-temperature probes drift away from the anchor semantics, weakening this dominant mode. This analysis clarifies the role of the spectral formulation. 7.2 Ablations 7.2.1 Effect of input augmentations Non-Rephrased uses only the original input, and the probe set consists only of high-temperature samples from the original query. Same-Input uses semantically equivalent rephrased inputs, but does not use the mix of rephrased and low-T sampled inputs. Instead, it evaluates each rephrase independently: for each rephrased input, we generate 1010 high-temperature probes, compute the uncertainty score, and average the resulting AUROCs across rephrases. Rephrased-only computes uncertainty using 1010 rephrased inputs with high temperature responses obtained by sampling each of them once. The rephrase-only setting improves marginally, indicating that input rephrasing is useful, but it adds 22X rephrasing cost. Same ablation on DSE and SNNE are added in Appendix A.3. SURE (Mixed-Rephrased) uses 55 equivalent input augmentations and 22 high-temperature responses from each to form 1010 probe responses. Figure 4 shows the average AUROC across models, with full per-model results provided in Appendix A.3. Input augmentation significantly improves performance over the non-rephrased variant. Same-Input also improves over Non-Rephrased, showing that equivalent-input augmentation itself is beneficial. SURE (Mixed-Rephrased) gives the best average performance on both datasets, suggesting that mixing probe responses exposes the most informative set of semantic variations. 7.2.2 Dissecting cross-temperature agreement. We next isolate the role of cross-temperature comparison by comparing SURE with same-temperature controls. Low-T computes spectral energy using only low-temperature samples. High-T uses only high-temperature samples. For a matched setup, both controls use 1010 samples. As shown in Figure 4, Low-T performs poorly, indicating that low-temperature generations are often too stable to reveal uncertainty. High-T is stronger because it exposes semantic variability, but it still underperforms SURE. This supports the central idea of SURE: low-temperature samples provide stable anchors, while high-temperature probes test whether this stable belief persists under stochastic sampling. Figure 4: Ablations and direct controls for SURE. Average AUROC across the six text-QA LLMs on Trivia-QA and SVAMP for input-augmentation variants, same-temperature controls, and direct aggregation baselines. Equivalent-input augmentation and cross-temperature comparison majorly contribute to the final performance. 7.2.3 Temperature sensitivity We analyze the sensitivity of SURE performance to the varying anchor temperature (TLT_L) and probe temperature (THT_H). With TH=1.0T_H=1.0, performance remains stable across (TL∈0.1,0.2,0.3,0.4T_L∈\0.1,0.2,0.3,0.4\) as shown in Figure 5, confirming that low-temperature responses provide stable semantic anchors. With TL=0.1T_L=0.1, performance generally improves as THT_H increases toward 0.90.9-1.01.0, but declines at higher temperatures. This reveals a trade-off that probes that are too stable fail to expose uncertainty, whereas excessively high temperatures produce noisy generations. We use TL=0.1T_L=0.1 and TH=1.0T_H=1.0 throughout without tuning specific to the dataset and model. Additional gridwise analysis is provided in Appendix A.5. Figure 5: Temperature sensitivity of BiG-SURE. Top: Abstention AUROC when varying the anchor temperature (TLT_L), with the probe temperature fixed at (TH=1.0T_H=1.0). Performance remains stable across (TL∈0.1,0.2,0.3,0.4T_L∈\0.1,0.2,0.3,0.4\). Bottom: Abstention AUROC when varying (THT_H), with (TL=0.1T_L=0.1). Performance generally improves as (THT_H) approaches 0.90.9-1.01.0, while excessively high temperatures degrade performance, revealing a trade-off. 7.3 Strawman baselines We compare SURE against direct aggregation controls computed on the same W. These include average similarity (AvgSim), maximum anchor–probe similarity (MaxSim), anchor–probe entropy, and average squared similarity (AvgSqSim). Figure 4 compares SURE with AvgSim and MaxSim. Full results are shown in Appendix A.2. AvgSqSim is numerically identical to BiG-SURE (Eqn. 2) and serves as a sanity check. This ablation study shows that the specific aggregation method used in SURE carries little value, and it is quite robust to aggregation methods like average similarity, maximum similarity and so on. The squared-similarity formulation remains useful for spectral analysis to inspect singular/spectral modes from a diagnostic perspective (Section 7.1). Anchor-probe entropy performs poorly as complete anchor-probe dissimilarity (Wij=0W_ij=0) indicates high uncertainty, while anchor-probe entropy approaches zero in such cases. Figure 6: Sample sensitivity analysis for SURE. Abstention AUROCs averaged over LLMs for different numbers of anchor samples (M) and probe samples (N) for Trivia-QA and SVAMP. 7.4 Sample sensitivity analysis We study the sensitivity of SURE to the number of anchors (M) and probes (N) in Figure 6. Across both datasets, the performance is more sensitive to the number of probe samples than to the number of anchor samples. Increasing N from 55 to 1010 gives the largest improvement. Further increasing N yields only marginal gains, indicating saturation. In contrast, increasing M has little effect for fixed N, suggesting a small number of stable low-temperature anchors is sufficient. 7.5 Human subjective evaluation Multilingual correctness judge: For multilingual SciQ, we use Gemini-2.5-Flash to obtain correctness labels by comparing the question, reference answer, and greedy prediction. To assess the reliability of these labels, we conducted a human evaluation on the French, Japanese, and Chinese subsets of SciQ. For each language, we randomly sampled 2020 responses generated by Aya, balanced between 1010 labelled correct and 1010 labelled incorrect by Gemini-2.5-Flash. 1010 human annotators per language independently judged whether each prediction was correct with respect to the question and reference answer. We then obtain a human-consensus label through majority voting. Gemini agreed with the human consensus on 90% of the French examples, 95% of the Japanese examples, and 90% of the Chinese examples, with Cohen’s κ of 0.800.80, 0.900.90, and 0.800.80, respectively, showing strong agreement in this limited sample study. Evaluation of rephrasing quality SURE assumes that the LLM-generated rephrased questions preserve the meaning of the original input. To evaluate this, we conducted a human study in English and Japanese. For each language, we constructed a controlled set of 2020 original-rephrase pairs. 1010 human annotators per language independently rated the semantic equivalence of each pair on a five-point Likert scale. Ratings from 11 to 22 were treated as non-equivalent, while ratings from 33 to 55 were treated as equivalent. For every pair, we obtained a binary human-consensus label through majority voting. The consensus labels agreed with the equivalence labels for all 2020 examples in both English and Japanese, yielding 100%100\% agreement and Cohen’s κ=1.00κ=1.00 within this limited subjective study. 7.6 Computational complexity A practical consideration for black-box uncertainty estimation is the cost of uncertainty computation. For SURE, the bipartite similarity matrix W∈ℝM×NW ^M× N formation leads to complexity of (MNCsem)O(MN\,C_sem), where CsemC_sem denotes a single semantic similarity evaluation cost. The subsequent computation of spectral energy is comparatively lightweight. Most existing methods construct a full pairwise similarity matrix over N sampled responses, leading to (N2Csem)O(N^2\,C_sem). KLE further incurs (N3)O(N^3) cost. However, as mentioned by Nikitin et al. (2024), the uncertainty computation time remains lightweight compared to the LLM inference cost. Hence, we also perform a matched-inference budget comparison to reassess the gains. 7.7 Matched inference-budget comparison From the LLM inference cost perspective, SURE uses M+N=3+10M+N=3+10 responses per input, whereas most prior sampling-based baselines use N=10N=10 responses. To ensure that the gains are not simply due to more LLM calls, we re-evaluate the prior works with equalized budget of N=13N=13 for text-QA tasks. As shown in Appendix A.4, SURE outperforms the baselines under this setting as well. 8 Conclusion We introduced BiG-SURE, a black-box uncertainty estimation framework based on cross-temperature bipartite semantic agreement. BiG-SURE uses low-temperature responses as stable anchors and high-temperature responses as probes, constructs an anchor–probe semantic similarity graph, and defines uncertainty as the complement of its normalized squared spectral energy. Empirically, SURE achieves strong abstention AUROC across text-only, multilingual, and multimodal QA tasks, improving over prior black-box uncertainty estimators in most evaluated settings. Our ablations further show that the cross-temperature comparison and the equivalent-input augmentation majorly contribute to the final performance. These results suggest that SURE is a simple, interpretable, and practical method for black-box uncertainty estimation, making BiG-SURE a promising candidate for reliability-aware deployment of LLMs and VLMs in safety-critical applications. 9 Limitations While BiG-SURE shows strong empirical performance across text-only, multilingual, and multimodal QA settings, it has the following limitations. Dependence on the semantic similarity backend. SURE relies on an entailment-based semantic similarity function to populate the anchor–probe bipartite graph. If this scorer fails to reliably capture semantic equivalence, the resulting similarity matrix may not accurately reflect the model’s uncertainty. This dependence is especially important in multilingual and multimodal settings, where semantic matching across languages or visually grounded responses can itself be challenging. Consistency does not guarantee correctness. SURE can fundamentally underperform if a model mispredicts and produces the same incorrect answer semantically consistently during sampling. In this case, even a perfect semantic-similarity backend causes a consistency-based estimator to be overconfident. This failure mode can be distinguished from errors caused by the NLI backend. Further analysis and examples are discussed in Appendix A.10. Visual grounding failure. For visual QA, the current semantic-similarity backend operates only on the generated textual responses and does not examine the image. Consequently, SURE measures textual generation consistency without considering image-text grounding. It may, therefore, assign high confidence when a VLM consistently produces the same visually overlooked answer. Extending the agreement function to incorporate non-text modalities is an important future work. Further analysis with examples are in Appendix A.10. Limited subjective assessment. BiG-SURE depends on the quality of meaning-preserving paraphrases, while Gemini-2.5-Flash is used to assign correctness labels for multilingual QA. Our human evaluations provide encouraging evidence for both components, but their limited scale does not constitute comprehensive validation across languages, domains, or question types. These results should therefore be interpreted as sanity checks rather than definitive validation of either the rephrasing model or the correctness judge. A systematic human evaluation of the NLI-based semantic-similarity backend remains for future work. Evaluation scope. Our experiments focus on question-answering style benchmarks, including factual QA, math word problems, multilingual science QA, and visual QA. Although these tasks cover multiple languages, modalities, and model families, they do not fully represent broader open-ended generation settings such as long-form generation, summarization, dialogue, planning, or code synthesis. Extending cross-temperature bipartite agreement to these broader tasks remains an important direction for future work. Acknowledgments We thank Google Research for its support through the Gemma Academic Award, which substantially helped meet the computational requirements of this work. We also gratefully acknowledge Google’s support for our conference travel, provided through the Kotak IISc AI–ML Centre. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §A.11, §1. Agrawal et al. (2024) P. Agrawal, S. Antoniak, E. B. Hanna, B. Bout, D. Chaplot, J. Chudnovsky, D. Costa, B. D. Monicault, S. Garg, T. Gervet, S. Ghosh, A. Héliou, P. Jacob, A. Q. Jiang, K. Khandelwal, T. Lacroix, G. Lample, D. L. Casas, T. Lavril, T. L. Scao, A. Lo, W. Marshall, L. Martin, A. Mensch, P. Muddireddy, V. Nemychnikova, M. Pellat, P. V. Platen, N. Raghuraman, B. Rozière, A. Sablayrolles, L. Saulnier, R. Sauvestre, W. Shang, R. Soletskyi, L. Stewart, P. Stock, J. Studnia, S. Subramanian, S. Vaze, T. Wang, and S. Yang Pixtral 12b. External Links: 2410.07073, Link Cited by: §5.1. Antol et al. (2015) S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh Vqa: visual question answering. In Proceedings of the IEEE international conference on computer vision, p. 2425–2433. Cited by: §A.8. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu Qwen3-vl technical report. External Links: 2511.21631, Link Cited by: §5.1. Bedi et al. (2025) S. Bedi, Y. Liu, L. Orr-Ewing, D. Dash, S. Koyejo, A. Callahan, J. A. Fries, M. Wornow, A. Swaminathan, L. S. Lehmann, et al. Testing and evaluation of health care applications of large language models: a systematic review. Jama 333 (4), p. 319–328. Cited by: §1. Bhattacharya et al. (2025a) D. Bhattacharya, A. Kulkarni, and S. Ganapathy Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning. In Interspeech 2025, p. 2068–2072. External Links: Document, ISSN 2958-1796 Cited by: §4. Bhattacharya et al. (2025b) D. Bhattacharya, A. Kulkarni, and S. Ganapathy FESTA: functionally equivalent sampling for trust assessment of multimodal llms.. In EMNLP (Findings), p. 12277–12295. Cited by: §4. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §5.4. Dang et al. (2024) J. Dang, S. Singh, D. D’souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kublik, M. Amer, V. Aryabumi, J. A. Campos, Y. Tan, T. Kocmi, F. Strub, N. Grinsztajn, Y. Flet-Berliac, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang, T. Chung, W. Ko, S. S. Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagné, F. Cruz-Salinas, E. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frosst, M. Fadaee, B. Ermis, A. Üstün, and S. Hooker Aya expanse: combining research breakthroughs for a new multilingual frontier. External Links: 2412.04261, Link Cited by: §5.1. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: §A.8, §1, §4, §4. Fomicheva et al. (2020) M. Fomicheva, S. Sun, L. Yankovskaya, F. Blain, F. Guzmán, M. Fishel, N. Aletras, V. Chaudhary, and L. Specia Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics 8, p. 539–555. Cited by: §4. Gao et al. (2024) X. Gao, J. Zhang, L. Mouatadid, and K. Das SPUQ: perturbation-based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Y. Graham and M. Purver (Eds.), St. Julian’s, Malta, p. 2336–2346. External Links: Document Cited by: §4, §5.2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1. He et al. (2021) P. He, X. Liu, J. Gao, and W. Chen DeBERTa: decoding-enhanced BERT with Disentangled Attention. In International Conference on Learning Representations, External Links: Link Cited by: §5.3. Hernández-Cano et al. (2025) A. Hernández-Cano, A. Hägele, A. H. Huang, A. Romanou, A. Solergibert, B. Pasztor, B. Messmer, D. Garbaya, E. F. Ďurech, I. Hakimi, J. G. Giraldo, M. Ismayilzada, N. Foroutan, S. Moalla, T. Chen, V. Sabolčec, Y. Xu, M. Aerni, B. AlKhamissi, I. A. Marinas, M. H. Amani, M. Ansaripour, I. Badanin, H. Benoit, E. Boros, N. Browning, F. Bösch, M. Böther, N. Canova, C. Challier, C. Charmillot, J. Coles, J. Deriu, A. Devos, L. Drescher, D. Dzenhaliou, M. Ehrmann, D. Fan, S. Fan, S. Gao, M. Gila, M. Grandury, D. Hashemi, A. Hoyle, J. Jiang, M. Klein, A. Kucharavy, A. Kucherenko, F. Lübeck, R. Machacek, T. Manitaras, A. Marfurt, K. Matoba, S. Matrenok, H. Mendoncça, F. R. Mohamed, S. Montariol, L. Mouchel, S. Najem-Meyer, J. Ni, G. Oliva, M. Pagliardini, E. Palme, A. Panferov, L. Paoletti, M. Passerini, I. Pavlov, A. Poiroux, K. Ponkshe, N. Ranchin, J. Rando, M. Sauser, J. Saydaliev, M. A. Sayfiddinov, M. Schneider, S. Schuppli, M. Scialanga, A. Semenov, K. Shridhar, R. Singhal, A. Sotnikova, A. Sternfeld, A. K. Tarun, P. Teiletche, J. Vamvas, X. Yao, H. Z. A. Ilic, A. Klimovic, A. Krause, C. Gulcehre, D. Rosenthal, E. Ash, F. Tramèr, J. VandeVondele, L. Veraldi, M. Rajman, T. Schulthess, T. Hoefler, A. Bosselut, M. Jaggi, and I. Schlag Apertus: Democratizing Open and Compliant LLMs for Global Language Environments. Note: https://arxiv.org/abs/2509.14233 Cited by: §5.1. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7b. External Links: 2310.06825, Link Cited by: §1. Joshi et al. (2017) M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M. Kan (Eds.), Vancouver, Canada, p. 1601–1611. External Links: Link, Document Cited by: §5.1. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Cited by: §4. Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, Cited by: §1, §4, §4. Kunitomo-Jacquin et al. (2026) L. Kunitomo-Jacquin, E. Marrese-Taylor, K. Fukuda, and M. Hamasaki Evidential semantic entropy for llm uncertainty quantification. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7107–7122. Cited by: §4. Lau et al. (2025) G. K. R. Lau, H. Dao, and B. K. H. Low Uncertainty quantification for MLLMs. In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI, External Links: Link Cited by: §4. Lin (2004) C. Lin Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, p. 74–81. Cited by: §4. Lin et al. (2024) Z. Lin, S. Trivedi, and J. Sun Generating with confidence: uncertainty quantification for black-box large language models. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §1, §4, §4. Liu et al. (2023) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. External Links: 2310.03744 Cited by: §5.1. Marino et al. (2019) K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi OK-vqa: a visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §A.8, §5.1. Nguyen et al. (2025) D. Nguyen, A. Payani, and B. Mirzasoleiman Beyond semantic entropy: boosting llm uncertainty quantification with pairwise semantic similarity. arXiv preprint arXiv:2506.00245. Cited by: §A.8, §1, §4, §4. Nikitin et al. (2024) A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen Kernel language entropy: fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems 37, p. 8901–8929. Cited by: §1, §4, §4, §7.6. Patel et al. (2021) A. Patel, S. Bhattamishra, and N. Goyal Are NLP models really able to solve simple math word problems?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, p. 2080–2094. External Links: Link, Document Cited by: §5.1. Phillips et al. (2026) E. Phillips, S. Wu, F. K. Gustafsson, B. Gao, and D. A. Clifton Semantic self-distillation for language model uncertainty. In Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, E. Perković and D. Malinsky (Eds.), Proceedings of Machine Learning Research, Vol. 337, p. 5427–5447. External Links: Link Cited by: §4. Qwen et al. (2025) Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §1, §5.1. Song et al. (2025) S. Song, X. Li, S. Li, S. Zhao, J. Yu, J. Ma, X. Mao, W. Zhang, and M. Wang How to bridge the gap between modalities: survey on multimodal large language model. IEEE Transactions on Knowledge and Data Engineering 37 (9), p. 5311–5329. Cited by: §1. Team et al. (2023) G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §1. Team et al. (2025) G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Põder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot Gemma 3 technical report. External Links: 2503.19786, Link Cited by: §5.1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Wang et al. (2025) M. Wang, H. Adel, L. Lange, Y. Liu, E. Nie, J. Strötgen, and H. Schütze Lost in multilinguality: dissecting cross-lingual factual inconsistency in transformer language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5075–5094. Cited by: §1. Xue et al. (2025) B. Xue, H. Wang, R. Wang, S. Wang, Z. Wang, Y. Du, B. Liang, W. Zhang, and K. Wong MlingConf: a comprehensive study of multilingual confidence estimation on large language models. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 2535–2556. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §5.1. Zhang et al. (2025) R. Zhang, H. Zhang, H. Fei, and Z. Zheng Uncertainty-o: one model-agnostic framework for unveiling uncertainty in large multimodal models. arXiv preprint arXiv:2506.07575. Cited by: §4. Appendix A Appendix A.1 Theoretical properties of SURE as an uncertainty estimator 1. Semantic identity: If all high temperature sampled prediction sequences are semantically identical and aligned with low-temperature samples, then W converges to M×N 1_M× N and norm(W,θ)=0U_ norm(W;θ)=0. Intuition. When all high-temperature generations are semantically identical, they all lie in the same semantic cluster as the low-temperature anchors. Consequently, every anchor–probe pair exhibits maximal entailment, so Wij≈1W_ij≈ 1 for all i,ji,j, and W approaches the matrix of all ones M×N 1_M× N. The spectral energy can be expressed as square of the Frobenius norm ‖W‖F\|W\|_F (proof in Appendix A.1). With W being approximately equal to M×N 1_M× N, the normalized spectral energy approaches a value of 11 and the uncertainty, defined as its complement, approaches a value of 00. In other words, perfect semantic consensus across temperatures leaves no room for ambiguity. The detailed proof is given in Proposition A.2. 2. Semantic disjointness: If all high temperature responses are mutually semantically disjoint, and mis-aligned with low-temperature samples, then W converges to M×N0_M× N and norm(W,θ)=1U_ norm(W;θ)=1. Intuition. If the high-temperature samples are mutually semantically disjoint (i.e., they do not align with the low-temperature meaning and also share no common semantic cluster), then each probe hjh_j exhibits negligible semantic entailment with every anchor aia_i. The similarity matrix satisfies Wij≈0W_ij≈ 0 for all (i,j)(i,j), and W approaches the all-zero matrix M×N0_M× N. In this case, the spectral energy ‖W‖F\|W\|_F goes to 00. As a result, norm(W,θ)=1U_ norm(W;θ)=1. Intuitively, complete semantic mismatch across temperatures corresponds to maximal ambiguity which results in maximal value of SURE metric. A formal proof is shown in Proposition A.3. 3. Monotonicity: For fixed (M,N)(M,N), if W′W corresponds to an input x′x where all elements of W′⪯W W, then, norm(W′,θ)≥norm(W,θ)U_ norm(W ;θ) _ norm(W;θ). Intuition. Holding the number of low and high temperature responses fixed, an increased semantic diversity in the high-temperature set typically reduces the cross-temperature agreement: each high temperature response hjh_j has reduced alignment with low temperature response aia_i, and the pairwise similarities shrink. The condition W′⪯W W element-wise formalizes this weakening. Since the (normalized) spectral energy is proportional to the Frobenius norm, decreasing the entries decreases ‖W‖F\|W\|_F, yielding norm(W′,θ)≥norm(W,θ)U_ norm(W ;θ) _ norm(W;θ). Intuitively, when predictions spread across more semantic modes, cross-temperature consistency breaks down and the uncertainty increases. A formal proof is given in Proposition A.6. Proposition A.1. Let W∈ℝM×NW ^M× N with entries satisfying 0≤Wij≤10≤ W_ij≤ 1 for all i,ji,j. Let ESURE(W,θ)=∑i=1rσi2E_SURE(W;θ)= _i=1^r _i^2 r=rank(W)≤min(M,N),r=rank(W)≤ (M,N), where σi _i are the non-zero singular values of W. Define Enorm(W,θ)=ESURE(W,θ)MN.E_norm(W;θ)= E_SURE(W;θ)MN. Then 0≤Enorm(W,θ)≤1.0≤ E_norm(W;θ)≤ 1. Consequently, if Unorm(W,θ)=1−Enorm(W,θ),U_norm(W;θ)=1-E_norm(W;θ), then 0≤Unorm(W,θ)≤1.0≤ U_norm(W;θ)≤ 1. Proof. Let the singular value decomposition of W be W=UΣV⊤,W=U V , where U∈ℝM×MU ^M× M and V∈ℝN×NV ^N× N are orthogonal matrices, and Σ=diag(σ1,σ2,…,σr,0,…,0). =diag( _1, _2,…, _r,0,…,0). Using the unitary invariance of the Frobenius norm, ‖W‖F2=‖UΣV⊤‖F2=‖Σ‖F2.\|W\|_F^2=\|U V \|_F^2=\| \|_F^2. Since the Frobenius norm of a diagonal matrix is the sum of squares of its diagonal entries, we obtain ‖W‖F2=∑i=1rσi2.\|W\|_F^2= _i=1^r _i^2. Therefore, ESURE(W,θ)=‖W‖F2.E_SURE(W;θ)=\|W\|_F^2. Now, by definition of the Frobenius norm, ESURE(W,θ)=‖W‖F2=∑i=1M∑j=1NWij2.E_SURE(W;θ)=\|W\|_F^2= _i=1^M _j=1^NW_ij^2. (4) Since 0≤Wij≤10≤ W_ij≤ 1, it follows that 0≤Wij2≤10≤ W_ij^2≤ 1 for all i,ji,j. Hence, 0≤∑i=1M∑j=1NWij2≤MN.0≤ _i=1^M _j=1^NW_ij^2≤ MN. Thus, 0≤ESURE(W,θ)≤MN.0≤ E_SURE(W;θ)≤ MN. Dividing throughout by MNMN yields 0≤Enorm(W,θ)=ESURE(W,θ)MN≤1.0≤ E_norm(W;θ)= E_SURE(W;θ)MN≤ 1. Finally, since Unorm(W,θ)=1−Enorm(W,θ),U_norm(W;θ)=1-E_norm(W;θ), we immediately obtain 0≤Unorm(W,θ)≤1.0≤ U_norm(W;θ)≤ 1. ∎ Proposition A.2. Given M low-temperature and N high-temperature responses, Unorm→0U_norm→ 0 iff all sampling responses are semantically similar. Proof. To prove this, we show the following: • Unorm→0U_norm→ 0 if all sampling responses are semantically similar. • All sampling responses are semantically similar if Unorm→0U_norm→ 0. Case I: As all sampling responses are semantically similar, let wij=s,∀i,jw_ij=s,∀ i,j. Then the bi-adjacency matrix is constant: W=s 1M×N,wij=s,∀i,j,W=s\,1_M× N, w_ij=s,\ ∀ i,j, where M×N1_M× N is the all-ones matrix. The Frobenius norm satisfies ‖W‖F2=∑i=1M∑j=1Nwij2=∑i=1M∑j=1Ns2=MNs2.\|W\|_F^2= _i=1^M _j=1^Nw_ij^2= _i=1^M _j=1^Ns^2=MN\,s^2. Since ESURE(W,θ)=∑i=1minM,Nσi2=‖W‖F2E_SURE(W;θ)= _i=1 \M,N\ _i^2=\|W\|_F^2 (Eqn. 4), we obtain ESURE(W,θ)=MNs2.E_SURE(W;θ)=MN\,s^2. Therefore the normalized confidence is Enorm(W,θ)=ESURE(W,θ)MN=MNs2MN=s2,E_norm(W;θ)= E_SURE(W;θ)MN= MN\,s^2MN=s^2, and the normalized uncertainty is Unorm(W,θ)=1−Enorm(W,θ)=1−s2.U_norm(W;θ)=1-E_norm(W;θ)=1-s^2. Hence, as s→1s→ 1 (perfect semantic similarity), we have Unorm(W,θ)→0U_norm(W;θ)→ 0. Case I: Suppose Unorm(W,θ)=0U_norm(W;θ)=0. Then Enorm(W,θ)=1E_norm(W;θ)=1, hence ESURE(W,θ)=MN.E_SURE(W;θ)=MN. Therefore ∑i,jWij2=MN. _i,jW_ij^2=MN. Since 0≤Wij≤10≤ W_ij≤ 1, we have Wij2≤1W_ij^2≤ 1 for each entry, and thus ∑i,jWij2≤∑i,j1=MN, _i,jW_ij^2≤ _i,j1=MN, with equality if and only if Wij2=1W_ij^2=1 for every (i,j)(i,j), i.e., Wij=1W_ij=1 for all (i,j)(i,j). Hence W=M×NW=1_M× N. Finally, if Wij=sW_ij=s for all i,ji,j, then W=M×NW=1_M× N implies s=1s=1, and conversely s=1s=1 implies W=M×NW=1_M× N, completing the iff statement. ∎ Proposition A.3. Given M low-temperature and N high-temperature responses, Unorm(W,θ)=1U_norm(W;θ)=1 iff all cross-temperature response pairs are semantically disjoint, i.e., sim(ai,hj)=0sim(a_i,h_j)=0 for all i,ji,j. Proof. We show both directions of implications to prove the iff condition. Case I: If Unorm(W,θ)=1U_norm(W;θ)=1, then Enorm(W,θ)=0E_norm(W;θ)=0, hence ESURE(W,θ)=0E_SURE(W;θ)=0 and therefore ∑i,jWij2=0. _i,jW_ij^2=0. Since each term Wij2≥0W_ij^2≥ 0, the only way their sum is zero is if Wij2=0W_ij^2=0 for every (i,j)(i,j), i.e., Wij=0W_ij=0 for all (i,j)(i,j). Thus W=M×NW=0_M× N. Case I: Conversely, if W=M×NW=0_M× N, then ‖W‖F2=0\|W\|_F^2=0, hence Enorm(W,θ)=0E_norm(W;θ)=0 and Unorm(W,θ)=1U_norm(W;θ)=1. ∎ Theorem A.4. For M low and N high-temperature responses, we show that, 1. The number of feasible clusters K(W)K(W) increases with decreasing WijW_ij, and vice-versa. 2. ℙ(CN(W)=N)=(K(W))N(K(W))NP (C_N(W)=N )= (K(W))_N(K(W))^N 3. ℙ(CN(W)<N)≤(N2)1K(W)P (C_N(W)<N )≤ N2 1K(W) 4. [CN(W)]=∑i=1K(W)1−(1−pi)N≤NE[C_N(W)]= _i=1^K(W)1-(1-p_i)^N≤ N 5. Under uniform cluster probability distribution and 0≤Wij<10≤ W_ij<1, [CN(W)]→NE[C_N(W)]→ N. 6. Also, [CN(W)]E[C_N(W)] is a monotonic function of K(W)K(W). Proof. For simplicity, we assume that the low-temperature responses are semantically similar, and the high-temperature responses show semantic variability. Let N be the number of high-temperature samples. Also, given the similarity matrix W, let the number of possible semantic clusters is K(W)K(W) (for example, for W=M×NW=1_M× N, K(W)=1K(W)=1). But K(W)K(W) clusters are observed only if an infinite number of samples are drawn. Because of the finite sampling of N, we do observe a subset of them. Assume the M low-temperature responses qii=1M\q_i\_i=1^M be semantically consistent and belong to a single semantic cluster, denoted by index ℓ . Let 1,…,K¯\1,…, K\ be a universal set of semantic clusters for a fixed input. Associate each cluster j with a prototype similarity to the cluster ℓ , μj≜sim(j,ℓ)∈[0,1],μℓ=1, _j (j, )∈[0,1], _ =1, where larger μj _j means cluster j is semantically closer to the cluster ℓ . We assume that, conditional on a high-temperature response belonging to cluster j, the observed cross-temperature similarity score s∈[0,1]s∈[0,1] follows a β-distribution and has a likelihood of Lj(s)∝sνμj−1(1−s)ν(1−μj)−1,ν>0,L_j(s)\ \ s^ν _j-1(1-s)^ν(1- _j)-1, ν>0, i.e., s|(Z=j)∼Beta(νμj,ν(1−μj))s (Z=j) (ν _j,ν(1- _j)). Based on a threshold α∈(0,1]α∈(0,1] and define the set of feasible clusters at similarity level s by a likelihood-ratio rule: ℱα(s)≜j∈1,…,K¯:Lj(s)Lℓ(s)≥α.F_α(s)\ \ \j∈\1,…, K\:\ L_j(s)L_ (s)\ ≥\ α \. Define the number of feasible clusters as K(W)=K(s)≜|ℱα(s)|K(W)=K(s) |F_α(s)|. Using the Beta likelihood above and μℓ=1 _ =1, the log-likelihood ratio simplifies (up to a constant independent of j) to logLj(s)Lℓ(s)=ν(μj−1)logs L_j(s)L_ (s)=ν( _j-1) s +ν(1−μj)log(1−s)+const(j)+ν(1- _j) (1-s)+const(j) For s bounded away from 11 and in the low-similarity regime (small s), the dominant term is ν(μj−1)logs,ν( _j-1) s, since logs→−∞ s→-∞ as s↓0s 0 while log(1−s)=O(1) (1-s)=O(1). Because μj<1 _j<1 for j≠ℓj≠ , we have (μj−1)<0( _j-1)<0, hence (μj−1)logs( _j-1) s increases as s decreases (it becomes large positive). Therefore, for each fixed j≠ℓj≠ , the likelihood ratio Lj(s)/Lℓ(s)L_j(s)/L_ (s) is nondecreasing as s decreases, implying that once a cluster satisfies the feasibility condition at some s2s_2, it will also satisfy it for any smaller s1<s2s_1<s_2. Thus ℱα(s1)⊇ℱα(s2)F_α(s_1) _α(s_2) and K(s1)≥K(s2)K(s_1)≥ K(s_2) (K(W1)≥K(W2)K(W_1)≥ K(W_2)). Let, CN(W)C_N(W) be the observed number of semantic clusters among K(W)K(W) because we use finite N samples instead of infinite, and CN(W)≤K(W)C_N(W)≤ K(W). Assuming N is not too large and K(W)≥NK(W)≥ N, ℙ(CN(W)=N) (C_N(W)=N ) =K(W)K(W)⋅K(W)−1K(W)⋅⋯ = K(W)K(W)· K(W)-1K(W)·s (5) ⋯K(W)−N+1K(W) ·s K(W)-N+1K(W) =(K(W))N(K(W))N = (K(W))_N(K(W))^N Also, let the event of responses i and j being assigned to the same cluster be EijE_ij. If ZiZ_i is the semantic cluster id assigned to response i, then Eij=Zi=Zj,ℙ(Eij)=1K(W)E_ij=\Z_i=Z_j\,\,P(E_ij)= 1K(W) (6) Now, ℙ(∃i<j:Zi=Zj) (∃ i<j:Z_i=Z_j) =ℙ(∪i<jEij) =P( _i<jE_ij) (7) ≤∑i<j1K(W) ≤ _i<j 1K(W) ⟹ℙ(CN(W)<N) (C_N(W)<N ) ≤(N2)1K(W) ≤ N2 1K(W) Let the cluster probabilities among the K(W)K(W) feasible set of clusters given W are pii=1K(W)\p_i\_i=1^K(W). We calculate the expected number of observed clusters among N samples. CN(W) C_N(W) =∑i=1K(W)Ii = _i=1^K(W)I_i (8) Ii=clusteri I_i=\cluster\,i appears at least once amongN at least once among\,N\ [CN(W)] [C_N(W)] =[∑i=1K(W)Ii] =E [ _i=1^K(W)I_i ] (9) =∑i=1K(W)[Ii] = _i=1^K(W)E[I_i] =∑i=1K(W)1−(1−pi)N = _i=1^K(W)1-(1-p_i)^N Case I: For Wij=1W_ij=1, implies K(W)=1K(W)=1 and hence, pi=1p_i=1. So, [CN(W)]=1−(1−1)N=1E[C_N(W)]=1-(1-1)^N=1 (10) Case I: For Wij=0W_ij=0, implies K(W)K(W) is large. So, [CN(W)] [C_N(W)] =∑i=1K(W)1−(1−pi)N = _i=1^K(W)1-(1-p_i)^N (11) = = ∑i=1K(W)[1−(1−Npi+(N2)pi2−…)] _i=1^K(W) [1- (1-Np_i+ N2p_i^2-... ) ] = = ∑i=1K(W)[Npi−(N2)pi2+…] _i=1^K(W) [Np_i- N2p_i^2+... ] = = N∑i=1K(W)pi−(N2)∑i=1K(W)pi2+… N _i=1^K(W)p_i- N2 _i=1^K(W)p_i^2+... = = N−(N2)∑i=1K(W)pi2+… N- N2 _i=1^K(W)p_i^2+... ≤ ≤ N N For uniform cluster distribution, pi=1K(W)p_i= 1K(W), and [CN(W)]→NE[C_N(W)]→ N as K(W)K(W) is large and hence N2K(W)<<1 N^2K(W)<<1. For uniform distribution with K(W)=kK(W)=k clusters, pi=1K(W)=1k,k>0p_i= 1K(W)= 1k,\,k>0 (12) Now, replacing pip_i in the cluster equation, we get, [CN]=k(1−(1−1k)N)E[C_N]=k (1- (1- 1k )^N ) (13) CNC_N is shorthand notation of CN(W)C_N(W). Now, we need to show, ∂[CN]∂k≥0 [C_N]∂ k≥ 0. Denote, a(k)=1−1k,a′(k)=1k2a(k)=1- 1k,\,a (k)= 1k^2 (14) f(k) f(k) =k−ka(k)N =k-ka(k)^N (15) ⟹f′(k) f (k) =1−a(k)N−kNa(k)N−1a′(k) =1-a(k)^N-kNa(k)^N-1a (k) =1−a(k)N−kNk2a(k)N−1 =1-a(k)^N- kNk^2a(k)^N-1 Now, let, x=1k∈(0,1]x= 1k∈(0,1]. So, a=1−xa=1-x. Now, f′(k) f (k) =1−(1−x)N−1(1−x+Nx) =1-(1-x)^N-1(1-x+Nx) (16) =1−(1−x)N−1(1+(N−1)x) =1-(1-x)^N-1(1+(N-1)x) Now, we have to show, (1−x)N−1(1+(N−1)x)≤1(1-x)^N-1(1+(N-1)x)≤ 1 for x∈[0,1]x∈[0,1]. Let m=N−1≥0m=N-1≥ 0. Define, h(x):=ln(1+mx)+mln(1−x)h(x):= (1+mx)+m (1-x) (17) Notice that, h(x) h(x) ≤0 ≤ 0 (18) ⟹exp(h(x)) (h(x)) ≤1 ≤ 1 ⟹(1+mx)(1−x)m (1+mx)(1-x)^m ≤1 ≤ 1 Now, h′(x) h (x) =m1+mx+m−11−x = m1+mx+m -11-x (19) =m(11+mx−11−x) =m ( 11+mx- 11-x ) =−m(m+1)x(1+mx)(1−x)≤0 =- m(m+1)x(1+mx)(1-x)≤ 0 Also, we know h(0)=0h(0)=0. Hence, h(x)≤0h(x)≤ 0, and hence proved. ∎ Proposition A.5. For the normalized uncertainty, Unorm(W,θ)=1−ESURE(W,θ)MN.U_norm(W;θ)=1- E_SURE(W;θ)MN. Then Unorm(W,θ)U_norm(W;θ) is monotone non-increasing in each entry of W. In particular, if for some fixed (p,q)(p,q), W~=W+δepeq⊤,δ≥0, W=W+δ\,e_pe_q , δ≥ 0, then Unorm(W~,θ)≤Unorm(W,θ).U_norm( W;θ)≤ U_norm(W;θ). Proof. We have Unorm(W,θ)=1−1MN∑i=1M∑j=1NWij2.U_norm(W;θ)=1- 1MN _i=1^M _j=1^NW_ij^2. Then Unorm(W~,θ)=1−1MN.U_norm( W;θ)=1- 1MN. (∑(i,j)≠(p,q)Wij2+(Wpq+δ)2) ( _(i,j)≠(p,q)W_ij^2+(W_pq+δ)^2 ) Therefore, Unorm(W~,θ) U_norm( W;θ) −Unorm(W,θ) -U_norm(W;θ) (20) =−1MN((Wpq+δ)2−Wpq2) =- 1MN ((W_pq+δ)^2-W_pq^2 ) (21) =−1MN(2δWpq+δ2). =- 1MN (2δ W_pq+δ^2 ). (22) Since Wpq≥0W_pq≥ 0 and δ≥0δ≥ 0, it follows that 2δWpq+δ2≥0.2δ W_pq+δ^2≥ 0. Hence, Unorm(W~,θ)≤Unorm(W,θ).U_norm( W;θ)≤ U_norm(W;θ). Thus, Unorm(W,θ)U_norm(W;θ) is monotone non-increasing in each entry of W. ∎ Proposition A.6. Assume that the low-temperature responses are semantically similar. Then increasing the number of semantically unique high-temperature answers weakens cross-temperature agreement in the sense that for two prediction sets V and ′V (with the same M,NM,N), the corresponding similarity matrices satisfy W′⪯Welementwise, with W′≠W.W W , with W ≠ W. Consequently, increasing semantic uniqueness of high-temperature answers monotonically increases Unorm(W,θ)U_norm(W;θ) and simultaneously increases the expected number of observed semantic clusters. Proof. We prove the two monotone conclusions. (i) UnormU_norm increases under weakened cross-temperature agreement. By Proposition A.5, the map W↦Unorm(W,θ)W U_norm(W;θ) is monotone non-increasing in each entry of W. Hence, if W′⪯W W elementwise, then Unorm(W′,θ)≥Unorm(W,θ).U_norm(W ;θ)\ ≥\ U_norm(W;θ). Moreover, since W′≠W ≠ W, at least one entry decreases strictly, implying the inequality is strict: Unorm(W′,θ)>Unorm(W,θ).U_norm(W ;θ)\ >\ U_norm(W;θ). (i) The expected number of observed semantic clusters increases. Since the low-temperature responses are assumed semantically similar, they define a single semantic cluster. By Theorem A.4(1), decreasing cross-temperature similarities (i.e., W′⪯W W) cannot reduce the number of feasible semantic clusters, hence K(W′)≥K(W).K(W )\ ≥\ K(W). By Theorem A.4(6), [CN(W)]E[C_N(W)] is a monotone nondecreasing function of K(W)K(W) , and therefore [CN(W′)]≥[CN(W)]E[C_N(W )]\ ≥\ E[C_N(W)] In particular, under uniform cluster probabilities (Theorem A.4(5)), increasing K(W)K(W) drives [CN(W)]E[C_N(W)] upward towards N. Combining (i) and (i), the elementwise weakening of cross-temperature agreement induced by increasing semantic uniqueness of the high-temperature answers monotonically increases Unorm(W,θ)U_norm(W;θ) and simultaneously increases the expected number of observed semantic clusters. This supports the interpretation of norm(W,θ)U_ norm(W;θ) as a cross-temperature semantic instability score: as high-temperature responses spread across more semantic clusters, the uncertainty score increases- obeying the fundamental notion of uncertainty. ∎ A.2 Full strawman-baseline analysis Table 3 reports the full per-model comparison between BiG-SURE and direct aggregation baselines computed from the same cross-temperature anchor–probe similarity matrix W. This analysis is designed to isolate the contribution of the aggregation rule, since all methods use the same generated samples and the same entailment-based similarity scores. We compare the following variants: • BiG-SURE: the proposed normalized squared spectral-energy score, computed using all singular values of W. • AvgSim: the mean anchor–probe similarity, 1MN∑i,jWij 1MN _i,jW_ij. • AvgSqSim: the mean squared anchor–probe similarity, 1MN∑i,jWij2 1MN _i,jW_ij^2. Since ∑rσr2=‖W‖F2=∑i,jWij2 _r _r^2=\|W\|_F^2= _i,jW_ij^2, this is numerically identical to BiG-SURE. • MaxSim: the maximum anchor–probe similarity score. • Anchor-probe Ent.: an entropy-style statistic computed from the anchor–probe similarity distribution. The results show that BiG-SURE and AvgSqSim match exactly, as expected from the Frobenius-energy identity. The direct controls are weaker on average. AvgSim and MaxSim underperform BiG-SURE on both datasets, while anchor–probe entropy performs substantially worse. These results support the use of squared cross-temperature agreement as a compact reliability signal. Table 3: Comparison with direct aggregation baselines and truncated spectral variants using entailment-based similarity. Results are reported as mean ± standard deviation over 5 seeds. AvgSqSim is numerically identical to BiG-SURE because normalized spectral energy equals the normalized squared Frobenius norm of the anchor–probe similarity matrix. Model BiG-SURE Top-1 Top-2 AvgSim AvgSqSim MaxSim Anchor-probe Ent. SVAMP Llama-8B 0.8897 ± 0.0261 0.8812 ± 0.0256 0.8891 ± 0.0261 0.8856 ± 0.0329 0.8897 ± 0.0261 0.8830 ± 0.0321 0.4455 ± 0.0111 Qwen-7B 0.8039 ± 0.0167 0.8022 ± 0.0134 0.8031 ± 0.0161 0.8037 ± 0.0185 0.8039 ± 0.0167 0.7996 ± 0.0223 0.4699 ± 0.0214 Llama-70B 0.9051 ± 0.0132 0.9053 ± 0.0145 0.9051 ± 0.0132 0.9026 ± 0.0105 0.9051 ± 0.0132 0.9037 ± 0.0084 0.4899 ± 0.0149 Qwen-72B 0.9018 ± 0.0392 0.9017 ± 0.0325 0.9018 ± 0.0380 0.9011 ± 0.0356 0.9018 ± 0.0392 0.9002 ± 0.0342 0.3714 ± 0.0124 Gemma-12B 0.8821 ± 0.0208 0.8786 ± 0.0217 0.8820 ± 0.0208 0.8832 ± 0.0180 0.8821 ± 0.0208 0.8815 ± 0.0187 0.4414 ± 0.0131 Gemma-27B 0.8786 ± 0.0168 0.8788 ± 0.0156 0.8785 ± 0.0162 0.8661 ± 0.0237 0.8786 ± 0.0168 0.8662 ± 0.0236 0.4766 ± 0.0151 Average 0.8769 ± 0.0221 0.8746 ± 0.0205 0.8766 ± 0.0217 0.8737 ± 0.0232 0.8769 ± 0.0221 0.8724 ± 0.0232 0.4491 ± 0.0147 Trivia-QA Llama-8B 0.7720 ± 0.0212 0.7716 ± 0.0185 0.7720 ± 0.0212 0.7624 ± 0.0237 0.7720 ± 0.0212 0.7586 ± 0.0228 0.4085 ± 0.0116 Qwen-7B 0.8060 ± 0.0141 0.8052 ± 0.0162 0.8059 ± 0.0138 0.8059 ± 0.0153 0.8060 ± 0.0141 0.8026 ± 0.0170 0.4390 ± 0.0055 Llama-70B 0.7145 ± 0.0087 0.7139 ± 0.0095 0.7144 ± 0.0060 0.7188 ± 0.0179 0.7145 ± 0.0087 0.7156 ± 0.0191 0.4731 ± 0.0111 Qwen-72B 0.7397 ± 0.0295 0.7398 ± 0.0188 0.7397 ± 0.0295 0.7320 ± 0.0344 0.7397 ± 0.0295 0.7248 ± 0.0345 0.4124 ± 0.0183 Gemma-12B 0.7886 ± 0.0110 0.7865 ± 0.0244 0.7883 ± 0.0162 0.7871 ± 0.0070 0.7886 ± 0.0110 0.7868 ± 0.0051 0.4974 ± 0.0113 Gemma-27B 0.6991 ± 0.0165 0.6989 ± 0.0149 0.6991 ± 0.0165 0.6872 ± 0.0208 0.6991 ± 0.0165 0.6773 ± 0.0188 0.5121 ± 0.0311 Average 0.7533 ± 0.0168 0.7527 ± 0.0170 0.7532 ± 0.0172 0.7489 ± 0.0198 0.7533 ± 0.0168 0.7443 ± 0.0196 0.4571 ± 0.0148 A.3 Input-augmentation ablations We provide the full per-model ablation of input variation strategies in Table 4. The low-temperature anchors are sampled from the original input in all variants. The variants differ only in how the high-temperature probe set is constructed. • Non-Rephrased: probes are generated only from the original input using high-temperature sampling. No equivalent-input augmentation is used. • Rephrased-only: In this case, 1010 input rephrasings are used instead of 55, and for each of them high-temperature sampling is used once. • Same-Input: each equivalent rephrased input is treated independently. For each rephrase, we generate 10 high-temperature probes, compute the SURE uncertainty score, evaluate AUROC, and then average the AUROCs across rephrases. • SURE (Mixed-Rephrased): this is the default setting. We use 5 equivalent input augmentations and sample 2 high-temperature responses from each, forming one mixed probe pool of 10 responses. The Figure 7 also shows that using only rephrasings improve the performance across different methods as well. However, it comes with additional rephrasing cost. The results show that input augmentation is the main contributor: both Same-Input and Mixed-Rephrased substantially outperform Non-Rephrased on average. Mixed-Rephrased further improves over Same-Input on both dataset averages, suggesting that combining semantically equivalent perturbations within the same probe pool provides a modest but consistent additional benefit. Table 4: Ablation of input variation strategies for SURE. We compare Non-Rephrased (no input augmentation), Same-Input (multiple generations from the same input), and Mixed (Rephrased) (our default setting, using semantically equivalent rephrased inputs). Results are reported as abstention AUROC (mean±stdmean over 5 seeds). The best result in each row is shown in bold, and the second-best result is underlined. Model AUROC (↑ ) Non-Rephrased Same-Input SURE (Mixed-Rephrased) SVAMP Gemma-12B 0.7771 ± 0.0092 0.8536 ± 0.0229 0.8821 ± 0.0208 Gemma-27B 0.7761 ± 0.0087 0.8470 ± 0.0242 0.8786 ± 0.0168 Llama-8B 0.7050 ± 0.0394 0.8989 ± 0.0239 0.8897 ± 0.0261 Llama-70B 0.8893 ± 0.0074 0.9031 ± 0.0118 0.9051 ± 0.0132 Qwen-7B 0.7619 ± 0.0200 0.7858 ± 0.0100 0.8039 ± 0.0167 Qwen-72B 0.9011 ± 0.0110 0.9102 ± 0.0211 0.9018 ± 0.0392 Avg. 0.8018 0.8664 0.8769 Trivia-QA Gemma-12B 0.6978 ± 0.0117 0.7606 ± 0.0113 0.7886 ± 0.0110 Gemma-27B 0.6051 ± 0.0086 0.6896 ± 0.0166 0.6991 ± 0.0165 Llama-8B 0.7435 ± 0.0068 0.7804 ± 0.0130 0.7720 ± 0.0212 Llama-70B 0.7065 ± 0.0081 0.7244 ± 0.0142 0.7145 ± 0.0087 Qwen-7B 0.7807 ± 0.0054 0.7885 ± 0.0087 0.8060 ± 0.0141 Qwen-72B 0.7096 ± 0.0104 0.7284 ± 0.0174 0.7397 ± 0.0295 Avg. 0.7072 0.7453 0.7533 Figure 7: Effect of Rephrase-only ablation on all methods. Using Rephrase-only inputs for different methods improves them most of the time, however it comes with 2X2X rephrase cost. Table 5: Comparison of baseline methods using 13 samples. Results are reported as abstention AUROC (mean±stdmean over 5 seeds). The second column reports the base task accuracy. The best result in each row is highlighted in bold, and the second-best result is underlined. Model Acc. DSE SumEigv Deg NumSet LexSim KLE SNNE SURE Trivia-QA Qwen-7B 0.495 0.7897 ± 0.0034 0.7984 ± 0.0052 0.7979 ± 0.0048 0.7884 ± 0.0045 0.7870 ± 0.0081 0.7733 ± 0.0076 0.7884 ± 0.0033 0.8060 ± 0.0141 Llama-8B 0.661 0.7221 ± 0.0052 0.7203 ± 0.0057 0.7320 ± 0.0058 0.7123 ± 0.0072 0.7367 ± 0.0069 0.7185 ± 0.0032 0.7325 ± 0.0068 0.7720 ± 0.0212 Gemma-12B 0.643 0.7084 ± 0.0102 0.7078 ± 0.0107 0.7045 ± 0.0094 0.7094 ± 0.0100 0.7326 ± 0.0144 0.6974 ± 0.0122 0.7182 ± 0.0142 0.7886 ± 0.0110 Gemma-27B 0.755 0.6317 ± 0.0055 0.6364 ± 0.0033 0.6353 ± 0.0030 0.6305 ± 0.0054 0.6584 ± 0.0057 0.6290 ± 0.0064 0.6189 ± 0.0102 0.6991 ± 0.0165 Llama-70B 0.767 0.7285 ± 0.0086 0.7582 ± 0.0097 0.7617 ± 0.0095 0.7157 ± 0.0117 0.7419 ± 0.0071 0.7064 ± 0.0109 0.7187 ± 0.0063 0.7145 ± 0.0087 Qwen-72B 0.690 0.6988 ± 0.0125 0.7232 ± 0.0051 0.7167 ± 0.0054 0.6779 ± 0.0087 0.6326 ± 0.0031 0.6999 ± 0.0042 0.7006 ± 0.0087 0.7397 ± 0.0295 Avg. 0.669 0.7132 0.7240 0.7247 0.7057 0.7149 0.7041 0.7129 0.7533 SVAMP Qwen-7B 0.688 0.7152 ± 0.0271 0.7839 ± 0.0199 0.7779 ± 0.0186 0.7838 ± 0.0279 0.7572 ± 0.0202 0.7871 ± 0.0185 0.7745 ± 0.0297 0.8039 ± 0.0167 Llama-8B 0.672 0.6203 ± 0.0224 0.8598 ± 0.0257 0.8617 ± 0.0267 0.8747 ± 0.0214 0.8595 ± 0.0169 0.8626 ± 0.0191 0.8825 ± 0.0205 0.8897 ± 0.0261 Gemma-12B 0.730 0.7731 ± 0.0162 0.7979 ± 0.0157 0.7998 ± 0.0162 0.7902 ± 0.0179 0.7940 ± 0.0166 0.7940 ± 0.0149 0.8011 ± 0.0156 0.8821 ± 0.0208 Gemma-27B 0.794 0.7482 ± 0.0148 0.7564 ± 0.0113 0.7573 ± 0.0111 0.7404 ± 0.0140 0.7426 ± 0.0146 0.7422 ± 0.0150 0.8039 ± 0.0123 0.8786 ± 0.0168 Llama-70B 0.797 0.8788 ± 0.0112 0.8844 ± 0.0134 0.8872 ± 0.0130 0.8885 ± 0.0079 0.8759 ± 0.0073 0.8896 ± 0.0116 0.8982 ± 0.0115 0.9051 ± 0.0132 Qwen-72B 0.849 0.9020 ± 0.0225 0.9235 ± 0.0088 0.9234 ± 0.0094 0.8837 ± 0.0221 0.8074 ± 0.0138 0.8421 ± 0.0225 0.9065 ± 0.0098 0.9018 ± 0.0392 Avg. 0.755 0.7729 0.8343 0.8346 0.8269 0.8061 0.8196 0.8445 0.8769 A.4 Matched inference-budget analysis Our default SURE configuration generates M=3M=3 low-temperature anchors and N=10N=10 high-temperature probes, resulting in 1313 total LLM generations per input. In contrast, several sampling-based baselines are commonly evaluated with 1010 generations. To verify that the observed improvements are not simply due to this larger generation budget, we rerun the text-QA baselines using N=13N=13 samples. As reported in Table 5, SURE retains its performance advantage even when the baselines are given the same number of LLM calls, suggesting that the improvement comes from the cross-temperature anchor–probe construction rather than from additional sampling alone. A.5 Temperature sensitivity analysis We evaluate the sensitivity of SURE to the choice of low-temperature anchor sampling temperature TLT_L and high-temperature probe sampling temperature THT_H. Figure 8 reports abstention AUROC on SVAMP and Trivia-QA for TL∈0.1,0.2,0.3,0.4T_L∈\0.1,0.2,0.3,0.4\ and TH∈1.0,1.2,1.5,2.0T_H∈\1.0,1.2,1.5,2.0\. SURE is relatively insensitive to the exact choice of TLT_L within the low-temperature range: for a fixed THT_H, changing TLT_L produces only small variations in AUROC. This supports the role of low-temperature samples as stable semantic anchors. Second, performance is more sensitive to the probe temperature THT_H. Moderate probe temperatures, especially TH=1.0T_H=1.0 and TH=1.2T_H=1.2, perform best, while very high temperatures such as TH=2.0T_H=2.0 degrade performance substantially. This suggests that the probes should introduce semantic diversity without becoming excessively noisy. We use the TL=0.1T_L=0.1 and TH=1.0T_H=1.0 in all results in the paper, without tuning. Figure 8: Temperature sensitivity analysis for SURE. Abstention AUROC for different anchor temperatures (TlT_l) and probe temperatures (ThT_h) on Trivia-QA and SVAMP. Performance is relatively stable across low-temperature anchor choices, while very high probe temperatures, especially Th=2.0T_h=2.0, reduce AUROC. A.6 Input augmentation prompts and implementation details This section provides the implementation details for the equivalent-input augmentations used in SURE. For textual inputs, we use LLM-based rephrasing to create semantically equivalent versions of the original question. The rephrased questions are used only to generate high-temperature probe responses; the low-temperature anchors are generated from the original input. This preserves the interpretation of anchors as stable responses to the original question, while allowing the probes to test semantic stability under both stochastic decoding and meaning-preserving input variation. The prompts used to create semantics-preserving textual augmentations using Llama-3.1-8B-Instruct (for text-QA) and Gemini-2.5-Flash (for multilingual QA) are shown below. For multimodal QA, we apply both textual and visual input augmentations. The textual component rephrases the question while keeping the associated image fixed. The visual component perturbs the image while preserving its semantic content. We use lightweight image transformations including mean blur with kernel size 55, affine rotation of ±10∘± 10 around the image center, and Gaussian pixel noise sampled from (0,10)N(0,10) followed by clipping to the valid pixel range [0,255][0,255]. These transformations are intentionally mild: they are designed to probe robustness to small input changes without changing the answer semantics. Together, these augmentations allow SURE to test whether high-temperature probe responses remain aligned with stable low-temperature anchors under equivalent input variations. Prompt for Gemini: Multilingual Rephrasing System instruction: You are a multilingual question rephrasing assistant. Your task is to rephrase questions in multiple languages using only the information present in the original questions. You must never add external knowledge. Strict rules: 1. Use only words and concepts from the original question; do not add any external knowledge. 2. Do not add descriptors that hint at the answer. 3. Keep all specific names, titles, dates, and key terms from the original exactly as written. 4. Each rephrase should ask for the exact same information—no more, no less. 5. Maintain the same language style and formality level for each language. 6. Generate rephrasings for each language. Input template: Original questions: questions_text Output format: [english] rephrase here [chinese] rephrase here [japanese] rephrase here [french] rephrase here Instruction: Provide the output in exactly the format above. Prompt for Llama: English Rephrasing System instruction: You are a question rephrasing assistant. You rephrase questions using only the information present in the original question. You must never add external knowledge. User prompt: Rephrase the following question. Strict rules: 1. Use only words and concepts from the original question; do not add any external knowledge. 2. Do not add descriptors that hint at the answer. 3. Keep all specific names, titles, and key terms from the original exactly as written. 4. Each rephrase should ask for the exact same information—no more, no less. Input template: Original question: question Output instruction: Output exactly the rephrased question. A.7 Recruiting human subjects Participants for the human-evaluation studies were recruited through Prolific, an online participant-recruitment platform. For the multilingual evaluations, we used Prolific’s language-based prescreening filters to recruit participants. Participants were compensated through Prolific according to the platform’s standard compensation rates. Recruitment, communication, and payment were handled through the platform. Prolific-assigned participant identifiers were used throughout the studies, and no directly identifying information, such as participants’ names or contact details, was collected by the authors. Participant identities therefore remained anonymous to the authors. A.8 Evaluation of greedy responses Evaluation of greedy responses for the 33 tasks of textual QA, multilingual QA and multimodal QA is performed as below: Textual QA: The evaluation of greedy responses is performed using the SQuAD F1 score metric, followed by using 0.50.5 threshold to make it binary as suggested in the prior works Farquhar et al. (2024); Nguyen et al. (2025). Multimodal QA: For this, the correctness is computed based on agreement with the multiple human annotations provided in the dataset, like prior works Antol et al. (2015); Marino et al. (2019). Multilingual QA: We use Gemini-2.5-Flash as a judge for evaluating responses in multilingual QA. The prompt used for the purpose is stated below, Prompt for Gemini-2.5-Flash: Multilingual Correctness Judging System instruction: You are evaluating science questions across multiple languages. Determine if each predicted answer matches the ground truth. Be flexible with extra explanation, but the core answer must be correct. Output format: For each language below, decide if the predicted answer is correct (1) or incorrect (0). Return ONLY a JSON object mapping each language code to 1 or 0. Example response: "en": 1, "zh": 0, "ja": 1, "fr": 0, "th": 1 Input template: === lang === Question: question Ground truth: ground_truth Predicted: first_greedy_prediction Output instruction: Return ONLY the JSON object. No extra text. A.9 Area Under the Risk–Coverage Curve In addition to the abstention AUROC, we evaluate selective-prediction performance using the Area Under the Risk–Coverage Curve (AURC). For each uncertainty estimator, predictions are ordered from the most to the least confident. At each coverage level, the risk is computed as the error rate among the retained predictions. A lower AURC value is better. Table 6 reports AURC averaged across the six text-QA models on SVAMP and TriviaQA. BiG-SURE achieves the lowest AURC on both datasets. These results are consistent with the abstention-AUROC findings. Table 6: AURC averaged across the six text-QA models. Values are reported as mean ± standard deviation across models. Lower values indicate better selective-prediction performance. Method SVAMP AURC ↓ TriviaQA AURC ↓ DSE 0.1097±0.04410.1097± 0.0441 0.2244±0.05770.2244± 0.0577 SumEigV 0.1169±0.04870.1169± 0.0487 0.2209±0.05600.2209± 0.0560 Deg 0.1169±0.04870.1169± 0.0487 0.2209±0.05620.2209± 0.0562 NumSet 0.1104±0.04390.1104± 0.0439 0.2230±0.05960.2230± 0.0596 LexSim 0.1146±0.03940.1146± 0.0394 0.2126±0.0574¯ 0.2126± 0.0574 KLE 0.1212±0.03980.1212± 0.0398 0.2176±0.06190.2176± 0.0619 SNNE 0.1017±0.0391¯ 0.1017± 0.0391 0.2328±0.05350.2328± 0.0535 BiG-SURE 0.0839±0.03280.0839± 0.0328 0.1975±0.05200.1975± 0.0520 A.10 Failure analysis of SURE We analyze representative failure cases in which textual semantic agreement can produce misleading uncertainty estimates. We discuss two distinct failure modes: (i) Case I: The LLM mispredicts and produces different semantically equivalent forms of the same incorrect responses during sampling, and (i) Case I: The semantic-similarity backend incorrectly identifies non-equivalent responses as equivalent. Figure 9: A representative visual-QA failure involving semantically consistent but incorrect responses. For the displayed image, the VLM produces “Louis Philippe” and “Louis Philippe Suits”, whereas the acceptable reference answers include Brooks Brothers, Giorgio Armani, and Gucci. The high NLI entailment score (0.98630.9863) correctly reflects agreement between the generated responses, but cannot determine whether they are visually grounded. Case I: Figure 9 shows an example from visual QA. For the question “What brand of suit is the man in the image wearing?”, the acceptable reference answers include Brooks Brothers, Giorgio Armani, and Gucci. However, the VLM repeatedly generates “Louis Philippe” and “Louis Philippe Suits”. The NLI backend assigns these responses an entailment score of 0.98630.9863 and estimates very low uncertainty. This is the case of consistent incorrect predictions. This should not be interpreted as an NLI-backend error: the two generated responses are semantically equivalent, and therefore the high entailment score is reasonable. Instead, it illustrates a fundamental limitation of response-consistency-based uncertainty estimation. This limitation is especially important in visual QA because the current semantic backend compares only the textual responses and does not examine whether they are grounded in the image. Case I A separate failure mode occurs when the NLI backend assigns high entailment to responses that are not semantically equivalent. For example, for a Chinese SciQ question asking what type of energy is released when iron filings react with air, the responses corresponding to “heat” and “combustion” receive an entailment score of 0.84530.8453. Although the concepts are related, combustion is a process rather than the requested form of energy, and the two answers are not interchangeable. We observe a similar issue for numerical and frequency-based answers. For a Japanese SciQ question asking how often partial lunar eclipses occur, the responses corresponding to “more than once per month” and “multiple times per year” receive an entailment score of 0.98950.9895, despite expressing substantially different frequencies. The two failure modes have different origins. Semantically consistent but incorrect predictions cannot be resolved by improving textual NLI alone and instead require grounding-aware comparison with the original input, including the image in multimodal settings. Semantic-backend errors may be mitigated through stronger multilingual and numerical entailment models, and calls for improving such models. Although the same semantic backend is used for BiG-SURE and the similarity-based baselines in our experiments, such errors does not guarantee that scorer errors affect all methods equally; as their impact may vary across methods. Developing modality-aware and more reliable semantic-agreement measures therefore remains an important direction for future work. A.11 Use of AI assistants We used an AI assistant- ChatGPT Achiam et al. (2023) solely as a language-editing aid. Its use was limited to identifying grammatical errors and typographical mistakes and suggesting localized revisions to improve clarity, and fluency. It was not used to generate ideas, formulate hypotheses, or design methods. All technical content, arguments, analyses, and conclusions were developed and verified by the authors, who also reviewed and approved all final edits.