Paper deep dive

A Tale of Two Error Categories: Exploring Concealed Trade-Offs in the Errors of Automated Judges in Evaluation of Uncertainty Quantifiers

Evgenia Ilia, Wilker Aziz

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 246

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/11/2026, 3:04:40 AM

Summary

This paper investigates the impact of errors in automated judges on the evaluation of Uncertainty Quantifiers (UQs) in Natural Language Generation (NLG). Through a meta-analysis and experiments using 4 LLMs, 7 UQs, and human vs. automated judgments on QA datasets, the authors find that automated judgment is the norm but rarely validated. They demonstrate that judge errors distort UQ rankings, particularly misrepresenting high-quality (informative) UQs, and that judge performance only coarsely predicts this distortion.

Entities (12)

LLMs · model · 99%Uncertainty Quantifiers · concept · 98%Automated Judges · method · 97%AUROC · metric · 96%AmbigQA · dataset · 95%TriviaQA · dataset · 95%Llama-3.1-8B-Instruct · model · 94%Qwen2.5-7B-Instruct · model · 94%Semantic Entropy · uq-method · 93%Verbalised Confidence · uq-method · 93%Selective Prediction · concept · 92%GPT-4 · model · 90%

Relation Signals (10)

LLMs → usedas → Automated Judges

confidence 96% · the most popular being an LLM as a judge (e.g., GPT-4).

Automated Judges → distorts → UQ Evaluation

confidence 95% · automated protocols can perform very poorly, with AUROC values and rankings distorted away from the oracle.

Semantic Entropy → istypeof → Uncertainty Quantifier

confidence 95% · We analyse 7 different UQs: ... semantic entropy ... and verbalised confidence

Verbalised Confidence → istypeof → Uncertainty Quantifier

confidence 95% · We analyse 7 different UQs: ... and verbalised confidence

TriviaQA → usedin → Experiments

confidence 95% · With experiments in question answering, using 4 LLMs... and 7 popular UQs... in question answering (TriviaQA... and AmbigQA...)

AmbigQA → usedin → Experiments

confidence 95% · With experiments in question answering... in question answering (TriviaQA... and AmbigQA...)

Automated Judges → misrepresents → Informative UQs

confidence 94% · judgement errors tend to misrepresent informative UQs most.

Qwen2.5-7B-Instruct → usedin → Experiments

confidence 94% · We employ 4 generator models: ... Qwen2.5-7B-Instruct... We generate greedy responses from these models

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The wide adoption of LLMs across broad NLG applications heightens the importance of providing users with the means to avert errors and hallucinations. Uncertainty quantification is poised to fill that gap; with low uncertainty (high confidence), as a proxy for correctness, allowing users to be selective (e.g., reject low-confidence, likely incorrect responses). Correlation between confidence and correctness then serves as a useful criterion for evaluation of uncertainty quantifiers (UQs). But in NLG, where diverse responses can be adequate to a prompt, obtaining reliable correctness judgements is not simple, especially without human intervention. Errors in automated judgement are hardly avoidable and known to diminish the reliability of evaluation protocols (Santilli et al., 2025; Ielanskyi et al., 2025). In a meta-analysis of published work, we show that automated judgement is the present norm. Besides, automated judgements are rarely validated against human ones, and the validation of the UQ evaluation they automate is even rarer. With experiments in question answering, using 4 LLMs, human and automated judgements and 7 popular UQs, we find that i) a judge's performance can only coarsely predict the observed impact of its errors on the reliability of UQ evaluation, and that ii) judgement errors tend to misrepresent informative UQs most. We link these observations to patterns of correlation between confidence and categories of judgement error.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

243,120 characters extracted from source content.

Expand or collapse full text

A Tale of Two Error Categories: Exploring Concealed Trade-Offs in the Errors of Automated Judges in Evaluation of Uncertainty Quantifiers Evgenia Ilia University of Amsterdam e.ilia@uva.nl Wilker Aziz University of Amsterdam w.aziz@uva.nl Abstract The wide adoption of LLMs across broad NLG applications heightens the importance of pro- viding users with the means to avert errors and hallucinations. Uncertainty quantification is poised to fill that gap; with low uncertainty (high confidence), as a proxy for correctness, allowing users to be selective (e.g., reject low- confidence, likely incorrect responses). Cor- relation between confidence and correctness then serves as a useful criterion for evaluation of uncertainty quantifiers (UQs). But in NLG, where diverse responses can be adequate to a prompt, obtaining reliable correctness judge- ments is not simple, especially without human intervention. Errors in automated judgement are hardly avoidable and known to diminish the reliability of evaluation protocols (Santilli et al., 2025; Ielanskyi et al., 2025). In a meta-analysis of published work, we show that automated judgement is the present norm. Besides, auto- mated judgements are rarely validated against human ones, and the validation of the UQ eval- uation they automate is even rarer. With exper- iments in question answering, using 4 LLMs, human and automated judgements and 7 popu- lar UQs, we find that i) a judge’s performance can only coarsely predict the observed impact of its errors on the reliability of UQ evaluation, and that i) judgement errors tend to misrep- resent informative UQs most. We link these observations to patterns of correlation between confidence and categories of judgement error. 1 Introduction While modern LMs have undeniably advanced the state-of-the-art of NLG applications, even large LMs still err and hallucinate (Singh et al., 2025; Comanici et al., 2025; Yang et al., 2025a; Manakul et al., 2023; Ji et al., 2023), often with no appre- ciable disclaimers (Yona et al., 2024; Wang et al., 2026). Uncertainty quantifiers (UQs), which qual- ify generated responses with a notion of uncertainty C W Human Judge Auto. Judge C Bad Quantifier Perfect Quantifier Confidence Confidence Confidence W Confidence Figure 1: An informative UQ (top-left) assigns higher confidence to responses that humans deem correct (C) than to those that humans deem wrong (W), allowing a decision maker to be selective (e.g., by rejecting lower- confidence responses wrt a threshold (dashed line)). If assigned confidences do not neatly separate Cs from Ws (top-right), the UQ leads to selection errors (shown in red). The degree of separation gives us a criterion to choose amongst UQs. But when judgement is au- tomated (bottom), as routinely done in UQ evaluation, judgement errors (C→W or W→C; shown as diamonds) distort our view of the relative merits of the UQs. This affects informative UQs most: on the left, we go from 0/6 to 3/6 selection errors (2 rejected Cs and 1 accepted W), while on the right we go from 3/6 to 4/6 errors. (or confidence), have been introduced to close this gap (Fomicheva et al., 2020; Malinin and Gales, 2021; Lin et al., 2024; Kadavath et al., 2022). An informative UQ is predictive of correctness, assign- ing low uncertainty (high confidence) to responses that are likely to be correct and high uncertainty (low confidence) otherwise. It therefore provides information that a selective decision maker can use to avoid errors (e.g., by rejecting low-confidence responses)—see Figure 1 (top). To establish the relative merits of uncertainty quantifiers—the goal of UQ evaluation—we mea- sure one or another notion of ‘correlation’ between correctness and uncertainty in generation. The most common strategies quantify errors in selective prediction (Varshney et al., 2022; Zablotskaia et al., 1 arXiv:2610.09693v1 [cs.AI] 7 Oct 2026 2023). In NLG, however, a plethora of factors (e.g., paraphrastic variation, task open-endedness, input under-specification, user perspectives, etc.) render the variability in responses that are deemed plausi- ble wrt a given prompt impossible to represent in a reference set, making correctness difficult to as- certain automatically (Novikova et al., 2017; Plank, 2022; Celikyilmaz et al., 2020). In effect, human judgement is the most reliable evaluation signal in NLG, but, in practice, it is routinely replaced by au- tomated systems (Zhou et al., 2022). Such proxies to NLG evaluation distort our views of errors in se- lective prediction (SP)—see Figure 1 (compare top and bottom)—and diminish the reliability of our practical protocols for UQ evaluation (Santilli et al., 2025; Ielanskyi et al., 2025). In this work, we doc- ument the widespread neglect of human validation of these automated protocols, study the complex re- lationship between judgement errors and selective prediction errors, and seek to vindicate the role of human validation in UQ evaluation. We begin by mapping out the UQ evaluation practices that have become common within the NLP community in recent years, particularly with regards to the automation of correctness judgement and the validation of this practice in the specific context of UQ evaluation. We perform a meta- analysis of published ACL papers within the un- certainty in NLG field and find that a variety of judges are employed to approximate correctness (e.g., LLM as a judge, overlap-based and semantic- similarity-based metrics) but these are rarely val- idated against human judgement (∼ 23%of stud- ied papers, as per our meta-analysis). In experi- ments situated within question answering (Trivi- aQA; Joshi et al. (2017) and AmbigQA; Min et al. (2020)) using 4 LLMs (Llama-8B/3B; Grattafiori et al. (2024) and Qwen-7B/0.5B; Hui et al. (2024)), we assess a representative range of automated judges by comparing their predictions to those of human judges. This reveals that judgement qual- ity varies across settings and is highly sensitive to hyperparameters (e.g., similarity cutoffs). We then characterise how the errors of the automated judges impact UQ evaluation: we use each one of 7 UQs to parameterise a selective decision maker, and rank these UQs in terms of AUROC (powered by either human or automated judgement). We observe how judges’ quality only coarsely relates with the qual- ity of the UQ evaluation they induce, probing us to dig deeper into the relationship between the error patterns of automated judges and the patterns UQs’ scores exhibit. We indeed identify a non-trivial im- pact in the relationship between judges’ error types and the distribution of confidences—implying that an optimal choice of judge requires its validation within the UQ evaluation protocol. Overall, judgement errors have non-trivial ef- fects on UQ evaluation, often misrepresenting in- formative quantifiers. We caution against the care- less adoption of judges without validation of their effectiveness within UQ evaluation. To that end, we introduce data analysis tools that help identify high quality judges and analyse their errors’ tradeoffs, to support more informed decision making. 1 2 Background 2.1 Uncertainty Quantifiers An LM can be regarded as a tool to represent prob- abilistic beliefs about responses given a prompt (Du et al., 2023, 2024). These beliefs can be as- sessed in at least two ways: i) by assuming that an underlying probability mass function (pmf) is prescribed by the autoregressive factorisation of the model, or i) by letting the token-level distribu- tions parameterise an arbitrary choice of sampling algorithm (Kool et al., 2019; Holtzman et al., 2020; Meister et al., 2023), in which case a pmf exists but is unknown. Sampling is tractable in both views, making Monte Carlo (MC) estimation of probabili- ties (as well as entropy and other quantities) viable. The vast majority of UQs is powered by one of these two views. Quantities derived from the proba- bility assigned to a response are seen as the model’s confidence in that prediction (e.g., log-probability of response (Jiang et al., 2021; Xiao and Wang, 2021) or its sequence-length-normalised version (Manakul et al., 2023; Bakman et al., 2024; Flores et al., 2025)). Degrees of variation in the condi- tional distributions (at sequence- or token-level), such as Shannon entropy, are viewed as uncertainty signals (Fomicheva et al., 2020; Xiao and Wang, 2021; Duan et al., 2024). Others, first push the LM distribution forward through a transformation (e.g., by analysing sampled outcomes in terms of some of their attributes) and assess notions of confidence or uncertainty wrt this transformed distribution: a popular choice being semantic variation (Kuhn et al., 2023; Cheng and Vlachos, 2024; Kunitomo- Jacquin et al., 2026) or consistency (Gao et al., 2024; Zhang et al., 2024; Bhattacharjya et al., 2025; 1 Code, generations and annotations athttps://github. com/evgeniael/auroc_eval. 2 Nguyen et al., 2025; Wang et al., 2025a; Joo and Cho, 2025; Vazhentsev et al., 2025), where the transformation groups outcomes that are semanti- cally equivalent. ‘Verbalised’ confidence quanti- fiers, which are estimates that are similar in essence but communicated differently, prompt LLMs to generate their confidence in a response. Some di- rectly sample from the next-token distributions and extract the relevant confidence values from the sam- pled strings (Tian et al., 2023; Ishii et al., 2025), or assess the probability of tokens semantically related with correctness or confidence values (Ka- davath et al., 2022; Lin et al., 2024; Khanmoham- madi et al., 2025). Recently, some works focus on adapting LLMs to predict some of the above UQs, e.g. via some form of teacher-student distillation (Chen et al., 2023; Liu et al., 2024a; Xu et al., 2024; Shelmanov et al., 2025; Eikema et al., 2025). 2.2 Uncertainty Quantification Evaluation An informative UQ should indicate when we will generate correct responses (with high con- fidence/low uncertainty) and incorrect responses (low confidence/ high uncertainty). Under this lens of informativeness, we can validate various UQs; with selective prediction comprising a useful paradigm for this purpose — it accepts or rejects responses depending on whether a confidence score surpasses a threshold (Varshney et al., 2022; Zablot- skaia et al., 2023). At a given thresholdt, we can measure various performance trade-offs: for ex- ample, risk/error against coverage (percentage of responses answered/accepted), or precision against recall, or the true positive rate (TPR) against the false positive rate (FPR). Existing metrics can help summarise information about trade-offs under var- ious thresholds, by measuring the area under the curves plotting the assessed trade-offs of interest across two axes: e.g., the area under the precision- recall curve (AUCPR) or the area under the receiver operating characteristic curve (AUROC), assessing TPR and FPR trade offs (Flach, 2016). 2 To compute such performance metrics (e.g., TPR, FPR, precision, recall) at varying thresh- olds, it is necessary to judge whether generated responses are correct or incorrect. As human judgement is expensive to obtain at large scale, in practice this correctness judgement is delegated 2 AUROC = R 1 0 TPR(t)d(FPR(t))has a probabilistic interpretation, which enables efficient implementation: the probability that a randomly chosen correct response has higher confidence than a randomly chosen incorrect one. to an automated judge, e.g., a quality metric or a judge model. Unavoidably, automated judges err. Their errors were found to distort AUROC values and rankings (Santilli et al., 2025; Ielanskyi et al., 2025). Santilli et al. (2025) identify sequence length as a common bias affecting both UQs and judges, and partly a source of the distortion—with judges that align well with humans being impacted to a lesser extent; while Ielanskyi et al. (2025) pro- pose combining judges’ predictions as a means to mitigate biases. We dissect even further the complex relationships between UQs and automated judges of varying quality, as assessed by their abil- ity to predict human judgement, so as to understand how they interact within UQ evaluation. We dis- cover a previously unidentified impact of errors that is disproportionately affecting higher quality quantifiers; and point to error categories as a factor that can explain such impacts on UQs. 3 Correctness Judgements 3.1 Meta-analysis of Current Practices To the best of our knowledge, no extensive analysis of current practices for the choice of automated judges exists. Here we report on our own meta- analysis of such choices in recent literature. Methodology. We survey the ACL anthology from 2020 to 2026 for papers relevant to uncer- tainty in NLG (we start with an automatic keyword- matching filter and proceed with careful, manual in- spection). This results in 67 papers, whose choices with regards to correctness judgements we sum- marise next (further details in Appendix A.1). Results. Table 1 summarises the main findings. The 2nd column documents the number of times we encounter different types of judge, and the 3rd column lists the proportion of matches amongst the 67 papers—as in some papers more than one judge is used (e.g., because different tasks use dif- ferent judges, or because hybrid methods combine the predictions of multiple judges to reach a final decision) the cells are normalised independently and hence do not add up to 100%. First, it is worth to remark how human labelling is rare, with91% of papers delegating correctness decisions to an automated judge. When it comes to automated judges, a broad range of methods is used, with the most popular being an LLM as a judge (e.g., GPT-4). However, an appreciable number of papers uses other methods such as overlap-based metrics 3 Correctness judge# PapersProportion LLM judge2638.8% Overlap based1928.3% Exact/fuzzy match1420.9% Semantic similarity1116.4% Unidentifiable710.4% Human labelling69.0% Table 1: Correctness judges used in surveyed papers. (e.g., Rouge-L), exact or fuzzy matching, semantic- similarity-based metrics (e.g., cosine of BERT em- beddings). Even within one judge type we see high variation among implementation choices (metrics, thresholds, LLMs and their prompts, etc.; see Ap- pendix A.2 for an in-depth analysis and a complete list of the analysed papers and their associated cor- rectness judges). When the judge is automated (61 papers), only 23% of times (14 papers) the judge is validated against human judgements. We also tracked the choices made wrt the UQ evaluation metric (noting that some papers employ multiple). The majority of papers employ AUROC (≈ 60%), while other SP metrics (i.e., AUCPR, AURAC) and calibration metrics (i.e., ECE, Brier) remain popu- lar (see Table 5 of Appendix A.2). 3.2 Evaluating Correctness Judges Our meta-analysis reveals that automated judges are rarely validated against human judgements. Hence, we gather a representative set of such judges and collect human annotation to assess their classification performance (e.g., accuracy, F1, etc.). Data Collection. We focus on question answer- ing (QA): we sample 100 questions fromTriviaQA (Joshi et al., 2017), which contains factual ques- tions, and 200 questions fromAmbigQA(Min et al., 2020), which contains ambiguous and un- ambiguous factual questions (100 non-ambiguous and 100 ambiguous questions).We employ 4 generator models:Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct,Qwen2.5-7B-Instruct andQwen2.5-0.5B-Instruct(Grattafiori et al., 2024; Hui et al., 2024).We generate greedy responses from these models (i.e.,temp = 0). A human judge annotates the correctness of each greedy response with respect to a set of references. For Llama 8B/TriviaQA and Llama 8B/AmbigQA (ambiguous), 2 judges annotate each response, so that we can appreciate the extent to which they agree (Cohen’s Kappa score between annotators variablejudge F1 bleu/0.6 EM tokf/0.4 rouge/0.5 rouge/0.3 tokf/0.2 esim/0.5 bert/0.7 esim/0.4 esim/0.3 rouge/0.1 gpt llama human2 human1 0.00.20.40.60.8 Figure 2: Posterior F1 (horizontal axis) for a subset of judges for TriviaQA-Llama8b. Rows are sorted by posterior median (circle) and display two high density intervals (50% and 95%; thick/thin line segments). 0.88 for the former and 0.75 for the latter; see Appendix D.2 for details). We analyse various judges, representative of the categories of judges found in our meta-analysis (see Table 1): exact matching, thresholded token F1, thresholded BLEU (Reiter, 2018), thresholded ROUGE-L (Lin, 2004), thresholded embedding similarity withall-MiniLM-L6-v2, thresholded BERT score (Zhang* et al., 2020), and LLM as judge (using gpt-5.4-miniandLlama-3.1-8B-Instruct; an API and a Hugging Face model, resp.). Implemen- tation details in Appendix C. Figure 3 shows one annotated data point, for one TriviaQA question. Methodology. Our goal in this section is to use the annotated data (human and automated judge- ments) to infer our judges’ classification perfor- mances, but we also need to cope with various sources of uncertainty (e.g., adequacy judgements are subject to variability due to annotator perspec- tive, prompt under-specification and ambiguity, an- notation errors, etc.). To accommodate them, as well as the inevitable scarcity of human data, we model available data as draws from a hierarchi- cal Bayesian model whose latent parameters cap- ture the probability of correctness of each prompt- response pair, as well as each judge’s sensitivity (re- lated to TPR) and specificity (related to TNR). To- gether, these parameters allow us to infer posterior distributions of a judge’s performance in terms of TPR, TNR, FPR, FNR and transformations thereof (e.g. F1). See complete model in Appendix B.1. Results. Figure 2 plots the inferred posterior F1 for a subset of the judges in the setting Llama 4 Greedy response: Wyoming is the last U.S. state in alphabetical order. Q: In alphabetical order, what is the last U.S. state? R: State of Wyoming tokf/.3bert/.3llama rouge/.3esim/.3 gpt exact bleu/.3 Sampled responses: - Wyoming. - Wyoming is the last U.S. state in alphabetical order. - In alphabetical order, the last U.S. state is Wyoming. [...] P(T): .12vconf: .99 SE: -1.19sconf: 1.0 P(Seq): .26 E: -1.97 human1 human2 Figure 3: An example data point of question (Q) and ref- erences (R) from TriviaQA and responses by Llama 8B. The greedy response is judged manually and automat- ically (e.g., via exact match, BLEU, BertScore, LLM judges, etc.); positive judgements are blue/solid, nega- tive are red/dashed. We compute various UQs: some (e.g., P(True)) qualify the greedy response specifically (in green), others (e.g., entropy) qualify the entire condi- tional distribution (estimated using samples; in yellow). 8B/TriviaQA. These judges cover a representa- tive range of F1 medians. 3 Such a plot allows us to group bad (e.g., BLEU@0.6, exact match), medium (e.g.Rouge-L@0.3, Token-F1@0.2) and good judges (e.g., Rouge-L@0.1, Llama8B and GPT5.4-mini). We generally observe across dataset-generator pairs similar distribution spreads; with human F1 distributions concentrating towards higher F1 scores. The order of the judges varies across pairs; but across plots, distributions for LLM and semantic similarity judges (for one threshold or another) are skewed towards higher F1 values. 4 Evaluating UQ Evaluation UQ evaluation by AUROC sorts competing UQs from least to most informative for selective pre- diction. Here, we evaluate whether this strategy is at all reliable when correctness judgements are automated. For a collection of UQs being com- pared, we characterise their AUROC using human correctness judgements to establish an ‘oracle’ or ‘gold-standard’. We then analyse the impact of auto- mated correctness by assessing changes in AUROC when replacing human with automated judgement. Experiments.We continue analysing the prompt- response pairs fromTriviaQAandAmbigQA, and 3 A complete list of distributions for all judges and model- generator pairs can be seen in Appendix D.5. the generators we have used in the previous analy- sis (Llama 3B/8B, Qwen 0.5B/7B); whose greedy responses we have generated and annotated. We analyse 7 different UQs: log probability (Jiang et al., 2021), normalised log probability (Man- akul et al., 2023), predictive entropy (Fomicheva et al., 2020), semantic entropy (i.e., entropy over semantic clusters of sampled responses; Kuhn et al. (2023)), semantic confidence (i.e., rate at which a response is not contradicted by sampled responses; Yona et al. (2024))), P(True) (i.e., the probability a model assigns to the token associated with cor- rectness of a response; Kadavath et al. (2022)) and verbalised confidence (i.e., extracted confidence sampled from a model when prompted for the task; Tian et al. (2023)). To compute these UQs, we sample 10 responses from our generators (with temp = 1). We use the same correctness judges as before. See Figure 3 for an illustration of a datapoint (and Appendix C for more details). Results. Table 2 and 3 present the main results. For each dataset-generator pair we separate the cor- rectness judges as ‘good’ (G;F1 > 0.7), ‘medium’ (M;0.3 < F1 < 0.7) or ‘bad’ (B;F1 < 0.3) in terms of their F1 score, pointed-estimated against a human judge. Then, for each case we identify the best UQ (ranked 1st), medium-quality UQ (ranked 4th) and worst UQ (ranked 7th), according to their oracle AUROC values. We then check how the ora- cle evaluation is distorted by automated evaluation at different levels of judgement quality (G/M/B); by tracking the change in oracle figures, for the best, medium and worst UQs, due to automation. In Table 2 we track the position at which the auto- mated judge places these UQs and in Table 3 the change in automated AUROC relative to the ora- cle. These are averages, as each group (G/M/B) contributes multiple measurements (one per judge). It is clear that automated protocols vary widely, echoing previous findings (Santilli et al., 2025; Ielanskyi et al., 2025). But the oracle allows for further insight: we can now see that automated protocols can perform very poorly, with AUROC values and rankings distorted away from the ora- cle. We also see that judgement quality has a non- uniform impact on how the merits of different UQs are depicted, with informative UQs being harshly downplayed. The best UQs are often seriously harmed (e.g. the actual best UQ often ranks 4th- 6th under a bad correctness judge). On the other hand, the medium-quality UQs are affected some- 5 Best Quantifier (Rank #1)Medium Quantifier (Rank #4)Worst Quantifier (Rank #7) Model/Dataset pairActualRank GRank MRank BActualRank GRank MRank BActualRank GRank MRank B TriviaQA/Llama8bVC 2.34.34.4SE4.23.53.6SC6.86.05.8 TriviaQA/Llama3bP(Seq.)2.21.42.1VC2.55.65.2SC7.05.66.1 TriviaQA/Qwen7bSC1.56.06.3N.P(Seq.)5.02.02.8E6.03.31.3 TriviaQA/Qwen0.5bSC1.03.75.2N.P(Seq.)3.04.23.5P(True)6.04.64.8 AmbigQA_non/Llama8bVC1.42.84.7P(True)3.95.85.1E5.04.72.2 AmbigQA_non/Llama3bN.P(Seq.)1.51.51.2SC6.77.06.8E5.53.54.0 AmbigQA_non/Qwen7bP(True)2.02.84.5VC5.55.04.0E7.06.44.5 AmbigQA_non/Qwen0.5bN.P(Seq.)2.01.61.5P(Seq.)3.02.11.5E6.54.95.0 AmbigQA_am/Llama8bP(True) 1.43.36.4SE5.66.03.9E6.06.04.3 AmbigQA_am/Llama3bVC1.93.05.5N.P(Seq.)3.42.24.5E5.34.43.2 AmbigQA_am/Qwen7bP(True)1.53.65.8N.P(Seq.)2.54.33.4E7.04.62.8 AmbigQA_am/Qwen0.5bSC1.5-5.2N.P(Seq.)3.0-2.2E6.5-4.7 Table 2: We separate the correctness judges as ‘good’ (G;F1 > 0.7), ‘medium’ (M;0.3 < F1 < 0.7) or ‘bad’ (B; F1 < 0.3) in terms of their F1 score, and identify the best UQ (ranked 1st), medium-quality UQ (ranked 4th) and worst UQ (ranked 7th), according to their oracle AUROC values. We show how the oracle rankings of UQs are affected on average. Best Quantifier (Rank #1)Medium Quantifier (Rank #4)Worst Quantifier (Rank #7) Model/Dataset pairActualAUROCDiff. GDiff. MDiff. BActualAUROCDiff. GDiff. MDiff. BActualAUROCDiff. GDiff. MDiff. B TriviaQA/Llama8bVC0.92 -0.11-0.22-0.19SE0.76-0.03-0.030.02SC0.69-0.07-0.05-0.01 TriviaQA/Llama3bP(Seq.)0.88-0.09-0.100.03VC0.790.00-0.12-0.02SC0.67-0.06-0.020.06 TriviaQA/Qwen7bSC0.82-0.08-0.18-0.11N.P(Seq.)0.72-0.080.080.23E0.67-0.050.080.29 TriviaQA/Qwen0.5bSC0.730.02-0.140.03N.P(Seq.)0.560.030.050.32P(True)0.490.020.070.31 AmbigQA_non/Llama8bVC0.78-0.02-0.07-0.31P(True)0.68-0.01-0.10-0.14E0.580.020.060.25 AmbigQA_non/Llama3bN.P(Seq.)0.79-0.020.020.11SC0.68-0.11-0.12-0.09E0.590.060.130.19 AmbigQA_non/Qwen7bP(True)0.84-0.09-0.11-0.13VC0.70-0.07-0.030.00E0.580.010.050.05 AmbigQA_non/Qwen0.5bN.P(Seq.)0.660.050.090.15P(Seq.)0.610.090.120.20E0.53-0.010.00-0.04 AmbigQA_am/Llama8bP(True)0.79-0.06-0.17-0.37SE0.68-0.09-0.130.04E0.520.030.030.10 AmbigQA_am/Llama3bVC0.700.00-0.06-0.27N.P(Seq.)0.66-0.050.09-0.15E0.550.000.060.06 AmbigQA_am/Qwen7bP(True)0.680.01-0.08-0.14N.P(Seq.)0.650.03-0.060.14E0.570.010.010.21 AmbigQA_am/Qwen0/5bSC0.71-0.07--0.28N.P(Seq.)0.60-0.02-0.15E0.500.00-0.00 Table 3: We separate the correctness judges as ‘good’ (G;F1 > 0.7), ‘medium’ (M;0.3 < F1 < 0.7) or ‘bad’ (B; F1 < 0.3) in terms of their F1 score and identify the best UQ (ranked 1st), medium-quality UQ (ranked 4th) and worst UQ (ranked 7th), according to their oracle AUROC values. We show how the oracle AUROC values of UQs are affected on average. what less (see medium columns of Table 3). The worst UQs are affected minimally or even appear better in terms of AUROC values (see right-most columns, in green), with their absolute rankings also appearing improved (bad judges place them in rankings 2-4, when they should rank last). In Appendix D.1, for transparency, we present a complete list of AUROC values and rankings per UQ and judge. For the dataset-generator pairs with two sets of human annotations, we analyse the AU- ROC values and rankings they induce side by side (see Appendix D.2); verifying their similar patterns. We also perform a bootstrapping analysis, where we repeatedly subsample from available prompts and compute key metrics (i.e., AUROC values and rankings; Appendix D.3), and observe similar pat- terns as in the main study. Lastly, we investigate the use of other SP metrics in Appendix D.4 and show that the issues discussed here persist. 5 Error Analysis 5.1 How Does NLG Evaluation and UQ Evaluation Interact? Here we dig deeper into the relation between the quality of automated judges and their (in)ability to power a reliable proxy to oracle UQ evaluation. Experiments. We rank UQs by oracle AUROC (with human judgements) and by proxy AUROC (with automated judgements) and measure the sim- ilarity of the rankings these protocols induce. For that, we use Rank-Biased Overlap (RBO; Webber et al. (2010)), which penalises disagreement at top positions more (we use patience0.9). 4 To grasp the relationship with judgement performance, we study how RBO co-varies with judges’ F1 scores. Results. Figure 4 shows results for TriviaQA- Llama 8B and TriviaQA-Qwen 7B (other pairs in Appendix D.6). Even within a narrow range of F1 values (i.e., similar-quality judges), there’s con- siderable variability in their UQ evaluation perfor- mance. This is not only prominent in low-quality judges, even high-quality judges (F1≃ 0.8− 0.9) exhibit this variability (judges are distributed across the vertical axis). We also inspect Kendallτ, in- stead of RBO, and observe similar patterns. This suggests that F1 may be hiding information that is relevant to UQ evaluation. Connecting with 4 As the field progresses, more UQs earn their spot in comparisons, with the list growing long and uninformative— naturally demanding for stronger agreement at top positions. 6 0.00.51.0 F1 0.75 0.80 0.85 0.90 0.95 1.00 Rank-biased Overlap Generator = llama-8b 0.00.51.0 F1 Generator = qwen-7b llm-os llm-api token bleu rouge embedding bert exact Figure 4: F1 scores (↑) for judges against the RBO score (↑) of human- vs judge-powered UQ rankings. Even within a narrow range of F1 values (i.e., similar- quality judges), there’s considerable variability in their UQ evaluation performance. our analysis of Section 3.2, where we observed different error distributions across judges’ TPRs and TNRs, we speculate that the ways a judge’s errors distribute across these two categories play an important role in UQ evaluation. This role is elusive until we relate them to confidences values. 5.2 Understanding Trade-offs Automated judges err, predicting that a response is correct when it is incorrect (FP), or vice versa (FN); we analyse how the confidences assigned by UQ methods distribute across these error categories. Methodology. With a heterogenous set of UQs being compared, it is hopeless to expect that the confidences they assign are comparable (many do not even lie in the same interval). For the tools we develop here, having all UQs communicate com- parable confidences facilitates analysis. For that, we learn to regress from the raw scores assigned by UQs to a (normalised) confidence using a Bayesian logistic regressor (details in Appendix B.2). Let’s start with the simplest form of SP: a quan- tifier Q assigns confidencecto a responser; if cexceeds a fixed threshold (e.g.,t = 0.5), the response is accepted, else it is rejected. An auto- mated judge categorises the responses as correct (positive/P) or incorrect (negative/N), but, relative to human judgement, the judge may be right (i.e., then a P is a true positive/TP and an N is a true negative/TN) or wrong (i.e., then a P is a false positive/FP and an N is a false negative/FN). In Figure 5 (for TriviaQA-Llama8B), we scatter the confidences assigned by different UQs, separated by categories of classification error in automated judgement by different judges. When the judge is 0.00.51.0 Sem. Entropy TP TN FP FN ROUGE-L@0.1 0.00.51.0 Sem. Entropy TP TN FP FN Llama-8B 0.00.51.0 Sem. Entropy TP TN FP FN GPT-5.4 0.00.51.0 Norm. P(Seq.) TP TN FP FN ROUGE-L@0.1 0.00.51.0 Norm. P(Seq.) TP TN FP FN Llama-8B 0.00.51.0 Norm. P(Seq.) TP TN FP FN GPT-5.4 0.00.51.0 P(Seq.) TP TN FP FN ROUGE-L@0.1 0.00.51.0 P(Seq.) TP TN FP FN Llama-8B 0.00.51.0 P(Seq.) TP TN FP FN GPT-5.4 0.00.51.0 Ver. Conf. TP TN FP FN ROUGE-L@0.1 0.00.51.0 Ver. Conf. TP TN FP FN Llama-8B 0.00.51.0 Ver. Conf. TP TN FP FN GPT-5.4 Figure 5: Error trade-offs in SP (with threshold0.5) for various UQs under best automated judges (TriviaQA- Llama8B). For a detailed guide on how to interpret this figure, see Section 5.2. right (TP, TN), a metric for SP will gather reliable information about the quantifier’s merits in separat- ing Ps from Ns (i.e., Ps to the right and Ns to the left of the threshold). When the judge is wrong (FP, FN), the UQ is unfairly rewarded (i.e., it accepts a response that appears P but is not or it rejects a response that appears N, but is not; green region in plots) or penalised (i.e., it rejects a response that appears P but is not or it accepts a response that appears N but is not; red region in plots) depending on how confidence distributes. In this way to view the data, it is clear that an imbalance in concen- tration of points across error categories (FP, FN) amounts to a net impact that will misrepresent the informativeness of the UQ; this is will be true even for judges that are often (but not always) right. The figure displays the three best automated judges (as per the analysis of Figure 2). While all judges have similar F1 distributions, their errors are not always similarly distributed. For instance, Verb. Conf. seems to have a more dense region of unfair penalties (bottom, right red region) when the judge is ROUGE-L@0.1 compared to the LLM judges, indicating a higher negative impact on that UQ by the specific judge. However, GPT5.4 judge’s er- rors may be impacting P(seq.) more harshly than ROUGE-L@0.1, since the former’s negative im- pact regions seem to be more densely populated than positive ones. ROUGE-L@0.1 contains in- stances in positive and negative regions, possibly cancelling each other’s effects. Let’s now turn to evaluation by AUROC, which is based on rate of ‘successful’ pairwise compar- 7 01 Sem. Entropy TP TN FP FN Rouge-L@0.1 A in FN 01 Sem. Entropy TP TN FP FN Rouge-L@0.1 A in FP 01 Sem. Entropy TP TN FP FN Llama-8B A in FN 01 Sem. Entropy TP TN FP FN Llama-8B A in FP 01 Sem. Entropy TP TN FP FN GPT-5.4 A in FN 01 Sem. Entropy TP TN FP FN GPT-5.4 A in FP 01 P(Seq.) TP TN FP FN Rouge-L@0.1 A in FN 01 P(Seq.) TP TN FP FN Rouge-L@0.1 A in FP 01 P(Seq.) TP TN FP FN Llama-8B A in FN 01 P(Seq.) TP TN FP FN Llama-8B A in FP 01 P(Seq.) TP TN FP FN GPT-5.4 A in FN 01 P(Seq.) TP TN FP FN GPT-5.4 A in FP 01 Norm. P(Seq.) TP TN FP FN Rouge-L@0.1 A in FN 01 Norm. P(Seq.) TP TN FP FN Rouge-L@0.1 A in FP 01 Norm. P(Seq.) TP TN FP FN Llama-8B A in FN 01 Norm. P(Seq.) TP TN FP FN Llama-8B A in FP 01 Norm. P(Seq.) TP TN FP FN GPT-5.4 A in FN 01 Norm. P(Seq.) TP TN FP FN GPT-5.4 A in FP 01 Ver. Conf. TP TN FP FN Rouge-L@0.1 A in FN 01 Ver. Conf. TP TN FP FN Rouge-L@0.1 A in FP 01 Ver. Conf. TP TN FP FN Llama-8B A in FN 01 Ver. Conf. TP TN FP FN Llama-8B A in FP 01 Ver. Conf. TP TN FP FN GPT-5.4 A in FN 01 Ver. Conf. TP TN FP FN GPT-5.4 A in FP Figure 6: Error impacts on AUROC for various quantifiers under best automated judges, for TriviaQA-Llama8B. For a detailed guide on how to interpret this figure, see Section 5.2. isons among correct and incorrect responses: the metric rewards the UQ when a correct response (P) has a higher confidence than an incorrect one (N), and penalises it otherwise; same-label comparisons are discarded. In Figure 6 (for TriviaQA-Llama 8B) we outline regions of unfair reward/penalty rel- ative to the confidence value of an instance A (red cross in the plot), rather than a fixed threshold. If the judge is right for this instance (A is TP or TN), AUROC gathers reliable information about the UQ; when the judge is wrong (A is FP or FN), AUROC is distorted. Because AUROC depends on pairs of instances and discards same-label comparisons, we have more regions to consider than in SP with fixed thresholds. For A in FN, every other instance B not in FN will lead to unfair reward, if assigned confidence higher than A (green in the plots), or to unfair penalty, if assigned confidence lower than A (red in the plots): when B is TP, the UQ col- lects reward for ranking a P over an apparent N (under human judgement this would have been a tie); when B is TN, A and B appear to tie, but that would not be the case under human judgement, so the UQ undeservedly foregoes a penalty; when B is FP, the UQ collects rewards for ranking an apparent P over an apparent N (under human judgement this would have been a penalty). For A in FP, penalties and rewards are reversed in each case. To make each plot visually representative of an actual trend, we place A (the red cross) in the median of the scat- tered points. As before, imbalance in concentration of points in FP and FN amounts to a net impact that will misrepresent the informativeness of the UQ. GPT5.4 judge’s errors may be impacting P(seq.) more profoundly than ROUGE-L@0.1. Focusing on FN instances: GPT5.4’s negatively impactful (red) regions are densely populated; while ROUGE- L@0.1 seems to contain instances that trade-off un- fair rewards and penalties, when we iterate across FN instances. In another example, when observing the trade-off regions of the FP instances of different judges for Norm. P(Seq.), ROUGE-L@0.1 appears to be negatively impacted (with green and red re- gions populated, with a dense cluster of points in the red region). Simultaneously, LLama 8B, whose FP error distribution is skewed towards higher val- ues, will more often benefit from unfair rewards of the instances in the TP and TN green regions; with a possibly lower negative impact on the quan- tifier compared to ROUGE-L@0.1. Even the same judge can impact different UQs in distinct ways. Rouge-L@0.1 has a similar distribution of FP and FN errors across semantic entropy scores; with an overall minimal impact on that UQ, as the positive and negative impacts balance each other out. Con- sidering the FP and FN distributions across Verb. Conf. scores, errors do not cancel each other out (since TP and TN errors distribute very differently; with the judge harming the UQ). The way the judges err has a non-trivial impact on UQ evaluation, with plots such as the ones we just analysed demonstrating precisely that. Hence, the choice of judge should be informed by its use in UQ evaluation. Simply opting for a judge that is aligned with human judgment (e.g., high F1), while agnostic to the application of UQ evaluation, 8 could lead to suboptimal decisions. Data analyses of the kind we use can help in a more thorough evaluation, which however require human anno- tations for the quality of the judge’s predictions. Future work could investigate whether we could determine insufficient judges before the collection of human data, by analysing patterns of the data, models, judges and UQs, that underlie the different error categories. 6 Recommendations We conclude this section by gathering some sugges- tions and recommendations for practitioners work- ing within uncertainty quantification. To begin with, in settings we have tested, we note that even a relatively small number of annotations suffices to discriminate judges reliably. Hence, practitioners should not be discouraged if practical constraints allow for gathering only a small number of anno- tations. Of course, this observation may well be task/generator-sensitive, but with Bayesian analy- sis, of the kind we use here, we get reassurances (or warnings) in the form of posterior intervals for the metrics in consideration. When choosing an automated judge in a new experimental setup, it is necessary to validate it. At first, one might think that it is sufficient to validate the judge as a tool for NLG evaluation (e.g., correctness clas- sification performance assessed against a human judge). That, however, turns out to be insufficient, for judgement errors and selection errors are not typically independent. Instead, validation needs to be performed within a UQ evaluation protocol (e.g., correlation between human-powered and machine- powered SP or AUROC). It must also be noted that, while validation of a judge as tool for NLG evalua- tion makes the conclusions dependent on task and generators, validation of a judge as a tool for UQ evaluation makes the conclusions further depen- dent on the specific uncertainty quantifiers being compared and the evaluation protocol in use (e.g., SP at fixed threshold, AUROC, etc.). All of that serves to emphasise that borrowing decisions from published work while transferring them to different settings is a gamble, and reference to published work in such cases cannot be safely regarded as a substitute for validation. 7 Conclusion We delve into an in-depth analysis of errors caused by automated correctness judgement during UQ evaluation. We document current practices in liter- ature, and reveal that automated judges are rarely validated. We uncover that judges’ errors impact UQs differently; with some unfairly rewarded and others harmed, in experiments we conduct using two QA datasets, four LLMs, seven popular UQs and seven judges. This impact is not fully explained by the judges’ ability to simply align with human judgement. In fact, in analyses we show that how and when judges err plays an important role in ex- plaining such impacts. Hence, employing a judge without understanding how it errs, may cloud in- sights and conclusions. Limitations Our meta analysis only considers papers that be- long to the ACL anthology from 2020 up until March 2026 (leaving other possibly relevant venues out). As such, not all published research papers that are concerned with UQ in NLG tasks that use selective prediction as part of their evaluation pro- tocol is part of our analysis. Nevertheless, from a small scale inspection of papers published at other venues, the findings of our meta-analysis seem to hold across publication venues. Due to the expensive nature of manual annota- tions, we were only able to annotate 100 responses per dataset-generator pair. However, even with a limited number of annotations, our BDA analyses piece together trends and patterns found in the data. Ideally, we would be able to have multiple anno- tators judge the correctness of responses for all dataset-generator pairs; due to budget constraints we only do so for two.For those, we assess the alignment among annotators and the effect of anno- tations from different human judges on our results. We find that human judgements are well aligned and results hold similar across annotators, showing that our results are robust to different annotators. Lastly, our experiments are focused on short- form QA, since most UQ techniques are grounded within this task. We leave expanding to long form QA as possible future work. Nonetheless, evaluat- ing the correctness of a longer response can be at least as difficult, if not more, as a short response; hence, it remains just as important, if not more, to validate the judges’ quality within UQ evaluation. Acknowledgements We would like to thank the anonymous reviewers for their feedback. 9 References Lukas Aichberger, Kajetan Schweighofer, Mykyta Ielan- skyi, and Sepp Hochreiter. 2025. Improving uncer- tainty estimation through semantically diverse lan- guage generation. In The Thirteenth International Conference on Learning Representations. Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Yang Wang. 2024. Knowl- edge of knowledge: Exploring known-unknowns un- certainty with large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6416–6432, Bangkok, Thailand. Associ- ation for Computational Linguistics. Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. MARS: Meaning- aware response scoring for uncertainty estimation in generative LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7752–7767, Bangkok, Thailand. Association for Computational Linguistics. Yavuz Faruk Bakman, Duygu Nur Yaldiz, Sungmin Kang, Tuo Zhang, Baturalp Buyukates, Salman Aves- timehr, and Sai Praneeth Karimireddy. 2025. Recon- sidering LLM uncertainty estimation methods in the wild. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 29531–29556, Vienna, Austria. Association for Computational Linguistics. Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee, Radu Marinescu, Katya Mirylenka, Michael Glass, and Xiao Shou. 2025. SIMBA UQ: Similarity-based aggregation for uncertainty quantification in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15880–15894, Suzhou, China. Association for Com- putational Linguistics. Umesh Bodhwani, Yuan Ling, Shujing Dong, Yarong Feng, Hongfei Li, and Ayush Goyal. 2025. A cali- brated reflection approach for enhancing confidence estimation in LLMs. In Proceedings of the 5th Work- shop on Trustworthy NLP (TrustNLP 2025), pages 399–411, Albuquerque, New Mexico. Association for Computational Linguistics. Stephen P Brooks and Andrew Gelman. 1998. General methods for monitoring convergence of iterative sim- ulations. Journal of computational and graphical statistics, 7(4):434–455. Qi Cao, Andrew Gambardella, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. 2026. Semantic token clustering for efficient uncertainty quantification in large language models. In Proceedings of the 19th Conference of the European Chapter of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 682–696, Rabat, Morocco. Associa- tion for Computational Linguistics. Nicola Cecere, Andrea Bacciu, Ignacio Fernández- Tobías, and Amin Mantrach. 2025. Monte Carlo temperature: a robust sampling strategy for LLM’s uncertainty quantification methods. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 305–320, Albuquerque, New Mexico. Association for Computational Linguistics. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha. 2023. Adap- tation with self-evaluation to improve selective pre- diction in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5190–5213, Singapore. Association for Computa- tional Linguistics. Jiuhai Chen and Jonas Mueller. 2024. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5186–5200, Bangkok, Thailand. Association for Computational Linguistics. Zheng Chen, Zhaoxin Feng, Jianfei Ma, Jiexi Xu, and Bo Li. 2025. Can LLMs recognize their own analogi- cal hallucinations? evaluating uncertainty estimation for analogical reasoning. In Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM), pages 84–93, Vienna, Austria. Association for Computational Linguistics. Julius Cheng and Andreas Vlachos. 2024. Measur- ing uncertainty in neural machine translation with similarity-sensitive entropy. In Proceedings of the 18th Conference of the European Chapter of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2115–2128, St. Julian’s, Malta. Association for Computational Linguistics. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Shehzaad Dhuliawala, Leonard Adolphs, Rajarshi Das, and Mrinmaya Sachan. 2022. Calibration of machine reading systems at scale. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022, pages 1682–1693, Dublin, Ireland. Association for Computational Linguistics. Li Du, Holden Lee, Jason Eisner, and Ryan Cotterell. 2024. When is a language process a language model? In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 11083–11094, Bangkok, Thailand. Association for Computational Linguistics. 10 Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. A measure-theoretic characterization of tight language models. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9744–9770, Toronto, Canada. Association for Computational Linguistics. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: To- wards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050–5063, Bangkok, Thailand. Association for Computational Linguistics. Bryan Eikema, Evgenia Ilia, José GC de Souza, Chrysoula Zerva, and Wilker Aziz. 2025. Teaching language models to faithfully express their uncer- tainty. arXiv preprint arXiv:2510.12587. Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexan- der Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. 2024. Fact-checking the output of large language models via token-level uncertainty quantification. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9367– 9385, Bangkok, Thailand. Association for Computa- tional Linguistics. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-polygraph: Uncer- tainty estimation for language models. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 446–461, Singapore. Association for Computational Linguistics. Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, and Dan Roth. 2025. Rethinking LLM uncertainty: A multi-agent approach to estimat- ing black-box model uncertainty. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 12349–12375, Suzhou, China. Associa- tion for Computational Linguistics. Peter A Flach. 2016. ROC Analysis, pages 1–8. Springer, United States. Lorenzo Jaime Yu Flores, Ori Ernst, and Jackie CK Che- ung. 2025. Improving the calibration of confidence scores in text generation using the output distribu- tion’s characteristics. In Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 172– 182, Vienna, Austria. Association for Computational Linguistics. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Spe- cia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555. Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Ka- malika Das. 2024. SPUQ: Perturbation-based uncer- tainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2336–2346, St. Julian’s, Malta. Association for Computational Linguistics. Andrew Gelman, John B Carlin, Hal S Stern, and Don- ald B Rubin. 1995. Bayesian data analysis. Chap- man and Hall/CRC. Andrew Gelman, Aki Vehtari, Richard McElreath, Daniel Simpson, Charles C. Margossian, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, Martin Modrák, and Vianey Leos Barajas. 2026. Bayesian Workflow. Chapman & Hall. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, and Yi R. Fung. 2025. M- Boundary: Advancing MLLM knowledge boundary awareness through reasoning step confidence calibra- tion. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 16427–16444, Vienna, Austria. Association for Computational Linguistics. Matthew D. Hoffman and Andrew Gelman. 2014. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15(47):1593–1623. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations. Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, and Edgar Dobriban. 2024. Uncertainty in language mod- els: Assessment through rank-calibration. In Pro- ceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 284–312, Miami, Florida, USA. Association for Computational Linguistics. Zhiqi Huang, Vivek Datla, Chenyang Zhu, Alfy Samuel, Daben Liu, Anoop Kumar, and Ritesh Soni. 2025. Confidence-based response abstinence: Improving LLM trustworthiness via activation-based uncertainty estimation. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 11 184–193, Suzhou, China. Association for Computa- tional Linguistics. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aich- berger, and Sepp Hochreiter. 2025. Addressing pit- falls in the evaluation of uncertainty estimation meth- ods for natural language generation. In ICLR Work- shop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI. Ai Ishii, Naoya Inoue, Hisami Suzuki, and Satoshi Sekine. 2025. Fine-grained confidence estimation for spurious correctness detection in large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 1238–1257, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv., 55(12). Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Associa- tion for Computational Linguistics, 9:962–977. Minsuh Joo and Hyunsoo Cho. 2025. Cleanse: Un- certainty estimation approach using clustering-based semantic consistency in LLMs. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 291–301, Vienna, Austria and virtual meeting. Association for Computational Linguistics. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.05221. Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese Smiley, Ivan Brugere, Kundan S Thind, and Mohammad M. Ghassemi. 2026. How reliable are confidence estimators for large reasoning models? a systematic benchmark on high-stakes domains. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1669–1754, Rabat, Morocco. Association for Computational Lin- guistics. Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardiko- raem, Simerjot Kaur, Ivan Brugere, Charese Smi- ley, Kundan S Thind, and Mohammad M. Ghassemi. 2025. Calibrating LLM confidence by probing per- turbed representation stability. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10448–10514, Suzhou, China. Association for Computational Linguistics. Wouter Kool, Herke Van Hoof, and Max Welling. 2019. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In Proceedings of the 36th Interna- tional Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3499–3508. PMLR. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. In The Eleventh International Conference on Learn- ing Representations. Lucie Kunitomo-Jacquin, Edison Marrese-Taylor, and Ken Fukuda. 2025. On the role of unobserved se- quences on sample-based uncertainty quantification for LLMs. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 179–183, Suzhou, China. Association for Computa- tional Linguistics. Lucie Kunitomo-Jacquin, Edison Marrese-Taylor, Ken Fukuda, and Masahiro Hamasaki. 2026. Evidential semantic entropy for LLM uncertainty quantifica- tion. In Proceedings of the 19th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 7107–7122, Rabat, Morocco. Association for Com- putational Linguistics. Rui Li, Jing Long, Muge Qi, Heming Xia, Lei Sha, Peiyi Wang, and Zhifang Sui. 2025. Towards harmonized uncertainty estimation for large language models. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 22938–22953, Vienna, Austria. Association for Computational Linguistics. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Contextualized sequence likelihood: Enhanced confi- dence scores for natural language generation. In Pro- ceedings of the 2024 Conference on Empirical Meth- 12 ods in Natural Language Processing, pages 10351– 10368, Miami, Florida, USA. Association for Com- putational Linguistics. Shudong Liu, Zhaocong Li, Xuebo Liu, Runzhe Zhan, Derek F. Wong, Lidia S. Chao, and Min Zhang. 2024a. Can LLMs learn uncertainty on their own? expressing uncertainty effectively in a self-training manner. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21635–21645, Miami, Florida, USA. Associa- tion for Computational Linguistics. Xin Liu, Farima Fatahi Bayat, and Lu Wang. 2024b. Enhancing language model factuality via activation- based confidence calibration and guided decoding. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 10436–10448, Miami, Florida, USA. Association for Computational Linguistics. Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. 2022. Learning confidence for transformer- based neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2353–2364, Dublin, Ireland. Association for Computational Linguistics. Matéo Mahaut, Laura Aina, Paula Czarnowska, Mom- chil Hardalov, Thomas Müller, and Lluis Marquez. 2024. Factual confidence of LLMs: on reliability and robustness of current estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4554–4570, Bangkok, Thailand. Association for Computational Linguistics. Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representa- tions. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucina- tion detection for generative large language models. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computa- tional Linguistics. Richard McElreath. 2018. Statistical rethinking: A Bayesian course with examples in R and Stan. Chap- man and Hall/CRC. Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally typical sampling. Transac- tions of the Association for Computational Linguis- tics, 11:102–121. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y- Lan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Trans- actions of the Association for Computational Linguis- tics, 10:857–872. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Lin- guistics. Maria Mora-Cross and Saul Calderon-Ramirez. 2024. Uncertainty estimation in large language models to support biodiversity conservation. In Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 368–378, Mexico City, Mexico. Asso- ciation for Computational Linguistics. Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman. 2025. Beyond semantic entropy: Boosting LLM un- certainty quantification with pairwise semantic sim- ilarity. In Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 4530–4540, Vienna, Austria. Association for Computational Lin- guistics. Jekaterina Novikova, Ond ˇ rej Dušek, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2241–2252, Copen- hagen, Denmark. Association for Computational Lin- guistics. Du Phan, Neeraj Pradhan, Martin Jankowiak, Eli Bing- ham, Jonathan P Chen, Fritz Obermeyer, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and 1 others. 2025. Numpyro: Probabilistic pro- gramming with numpy. Astrophysics Source Code Library, pages ascl–2505. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401. Andrea Santilli, Adam Golinski, Michael Kirchhof, Fed- erico Danieli, Arno Blaas, Miao Xiong, Luca Zap- pella, and Sinead Williamson. 2025. Revisiting un- certainty quantification evaluation in language mod- els: Spurious interactions with response length bias results. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 743–759, Vienna, Austria. Association for Computational Linguistics. Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Da- heim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, and Timothy Baldwin. 2025. A head to predict and a head to question: Pre-trained 13 uncertainty quantification heads for hallucination detection in LLM outputs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35712–35731, Suzhou, China. Association for Computational Linguistics. Yuanhao Shen, Xiaodan Zhu, and Lei Chen. 2024. SMARTCAL: An approach to self-aware tool-use evaluation and calibration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 774– 789, Miami, Florida, US. Association for Computa- tional Linguistics. Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd- Graber. 2022. Re-examining calibration: The case of question answering. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022, pages 2814–2829, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 oth- ers. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Chandu. 2024. Selective “selective predic- tion”: Reducing unnecessary abstention in vision- language reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12935–12948, Bangkok, Thailand. Association for Computational Linguistics. Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. 2024. When to trust LLMs: Aligning confidence with response quality. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5984–5996, Bangkok, Thailand. Associ- ation for Computational Linguistics. Alberto Testoni and Iacer Calixto. 2026. Mind the gap: Benchmarking LLM uncertainty and calibration with specialty-aware clinical QA and reasoning-based be- havioural features. In Proceedings of the 19th Confer- ence of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2364–2382, Rabat, Morocco. Association for Computational Linguistics. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 5433–5442, Singapore. Association for Computational Linguistics. Venktesh V, Mandeep Rathee, and Avishek Anand. 2025. SUNAR: Semantic uncertainty based neighborhood aware retrieval for complex QA. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 5818–5835, Albuquerque, New Mexico. Association for Computational Linguistics. Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Towards improving selective prediction ability of NLP systems. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 221–226, Dublin, Ireland. Association for Computational Lin- guistics. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025a. Benchmarking uncertainty quan- tification methods for large language models with LM-polygraph. Transactions of the Association for Computational Linguistics, 13:220–248. Roman Vashurin, Maiya Goloburda, Preslav Nakov, and Maxim Panov. 2025b. UNCERTAINTY-LINE: Length-invariant estimation of uncertainty for large language models. In Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pages 7881–7908, Suzhou, China. Asso- ciation for Computational Linguistics. Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Bald- win, and Artem Shelmanov. 2025.Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), pages 2246–2262, Albuquerque, New Mexico. Association for Compu- tational Linguistics. Chenyu Wang, Weichao Zhou, Shantanu Ghosh, Kay- han Batmanghelich, and Wenchao Li. 2025a. Se- mantic consistency-based uncertainty quantification for factuality in radiology report generation. In Find- ings of the Association for Computational Linguistics: NAACL 2025, pages 1739–1754, Albuquerque, New Mexico. Association for Computational Linguistics. Jiawei Wang, Yanfei Zhou, Siddartha Devic, and De- qing Fu. 2026. Are llm decisions faithful to verbal confidence? arXiv preprint arXiv:2601.07767. Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020. On the inference calibration of neural machine translation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 3070–3079, Online. Association for Computa- tional Linguistics. Tuo Wang, Adithya Kulkarni, Tyler Cody, Peter A. Bel- ing, Yujun Yan, and Dawei Zhou. 2025b. GENUINE: 14 Graph enhanced multi-level uncertainty estimation for large language models. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2025, pages 20522–20541, Suzhou, China. Association for Computational Linguistics. Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Xiaoshuang Shi, Kaidi Xu, Heng Tao Shen, and Xiaofeng Zhu. 2024. ConU: Conformal uncertainty in large language models with correctness coverage guarantees. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6886–6898, Miami, Florida, USA. Association for Computational Linguistics. William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., 28(4). Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. 2025. A survey of uncertainty estimation methods on large language models. In Findings of the Asso- ciation for Computational Linguistics: ACL 2025, pages 21381–21396, Vienna, Austria. Association for Computational Linguistics. Yijun Xiao and William Yang Wang. 2021. On hal- lucination and predictive uncertainty in conditional language generation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2734–2744, Online. Association for Computational Linguistics. Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. 2024. SaySelf: Teaching LLMs to express confidence with self-reflective rationales. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5985–5998, Miami, Florida, USA. Association for Computational Linguistics. Boyang Xue, Fei Mi, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Erxin Yu, Xuming Hu, and Kam-Fai Wong. 2025a. UAlign: Leveraging uncertainty esti- mations for factuality alignment on large language models. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6002–6024, Vienna, Austria. Association for Computational Linguistics. Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, Wenxuan Zhang, and Kam-Fai Wong. 2025b. MlingConf: A comprehensive study of multilingual confidence es- timation on large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2535–2556, Vienna, Austria. Associa- tion for Computational Linguistics. Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dim- itrios Dimitriadis, Jieyu Zhao, and Salman Aves- timehr. 2025. Do not design, learn: A trainable scor- ing function for uncertainty estimation in generative LLMs. In Findings of the Association for Compu- tational Linguistics: NAACL 2025, pages 691–713, Albuquerque, New Mexico. Association for Compu- tational Linguistics. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Yongjin Yang, Haneul Yoo, and Hwaran Lee. 2025b. MAQA: Evaluating uncertainty quantification in LLMs regarding data uncertainty. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5861–5878, Albuquerque, New Mexico. Association for Computational Linguistics. Luke Yoffe, Alfonso Amayuelas, and William Yang Wang. 2025. DebUnc: Improving large language model agent communication with uncertainty metrics. In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 23299–23315, Suzhou, China. Association for Computational Linguistics. Gal Yona, Roee Aharoni, and Mor Geva. 2024. Can large language models faithfully express their intrin- sic uncertainty in words? In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7752–7764, Miami, Florida, USA. Association for Computational Linguistics. Polina Zablotskaia, Du Phan, Joshua Maynez, Shashi Narayan, Jie Ren, and Jeremiah Liu. 2023. On un- certainty calibration and selective generation in prob- abilistic neural summarization: A benchmark study. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 2980–2992, Singapore. Association for Computational Linguistics. Boxuan Zhang and Ruqi Zhang. 2025. CoT-UQ: Im- proving response-wise uncertainty quantification in LLMs with chain-of-thought. In Findings of the As- sociation for Computational Linguistics: ACL 2025, pages 26114–26133, Vienna, Austria. Association for Computational Linguistics. Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. LUQ: Long-text uncertainty quantifi- cation for LLMs. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 5244–5262, Miami, Florida, USA. Association for Computational Linguistics. Caiqi Zhang, Chang Shu, Ehsan Shareghi, and Nigel Collier. 2025a. All roads lead to Rome: Graph-based confidence estimation for large language model rea- soning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31814–31824, Suzhou, China. Association for Computational Linguistics. Caiqi Zhang, Ruihan Yang, Zhisong Zhang, Xinting Huang, Sen Yang, Dong Yu, and Nigel Collier. 2025b. Atomic calibration of LLMs in long-form genera- tions. In Proceedings of the 14th International Joint 15 Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 148–169, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics. Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing more about questions can help: Improving calibration in question answering. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 1958–1970, Online. Association for Computational Linguistics. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations. Xinran Zhao, Hongming Zhang, Xiaoman Pan, Wenlin Yao, Dong Yu, Tongshuang Wu, and Jianshu Chen. 2024. Fact-and-reflection (FaR) improves confidence calibration of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8702–8718, Bangkok, Thailand. Associ- ation for Computational Linguistics. Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé I, Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 314–324, Seattle, United States. Association for Computational Linguistics. Kevin Zhou, Adam Dejl, Gabriel Freedman, Lihu Chen, Antonio Rago, and Francesca Toni. 2025. Evaluating uncertainty quantification methods in argumentative large language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2025, pages 21700–21711, Suzhou, China. Association for Computational Linguistics. A Meta-analysis A.1 Methodology We describe in more detail below the methodology we followed in order to conduct our meta-analysis. •We gather titles of papers from the entire ACL ontology from 2020-2026 •We create a list of relevant to the topic key words: uncertainty, variability, variation, aleatoric, epistemic, evidential, confidence, calibration, selective prediction, selective gen- eration • Filter papers whose title contains any of the key words. This resulted in 600 papers. We manually go through the titles of the papers and classify them as possibly relevant or ir- relevant, wrt confidence or uncertainty quan- tifiers within language generative tasks. This resulted in 206 papers. • We manually go through the abstracts of these papers and if necessary, the methods and ex- periments of those papers to identify ones that are actually relevant to uncertainty quantifica- tion within a selective prediction framework or calibration. •We filtered irrelevant papers, papers that only concern themselves with classification prob- lems, or have a very specific, unique or uncon- ventional evaluation method. The final set is comprised of 67 papers. •We identify the correctness criterion used in each of these papers and their evaluation met- rics. We then analyse and summarise informa- tion and statistics from gathered information. A.2 Experiments The complete table containing including all re- trieved papers and their associated judges and eval- uation metrics can be found in Table 4. We observe a variety of correctness judge types, and even variation within a single type, revealing that there is no one universally employed method within the field. Our later analysis reveals trade- offs of various of these. Additionally, we gather statistics of the UQ eval- uation metrics used in practice (see Table 5). AU- ROC is employed most often (≃ 60%of papers em- ploy AUROC). Other SP metrics, such as AUCPR (which assesses precision-recall trade-offs), AU- RAC (which assesses accuracy or risk vs coverage tradeoffs) and Prediction Rejection Ratio (PRR) are popular. Calibration metrics such as Expected Calibration Error (ECE) and Brier Score are also often employed. Less often, but still noteworthy, correlation scores (e.g. Pearson) among confidence values and errors are used. 16 Table 4: An overview of all tables PaperJudge TypeJudgeEvaluation method Khanmohammadi et al. (2026) Exact/fuzzy matching, LLM judge String match, LLM (GPT-5-nano) AUROC,AUCPR,ECE, Brier Testoni and Calixto (2026) Exact/fuzzy matching, LLM judge, Human validation String match, LLM (LLaMA-3.1-8B- Instruct),Human validation AUROC, ECE, Brier Kunitomo-Jacquin et al. (2026) LLM judgeLLM (gpt4o-mini)AUROC, AUARC Cao et al. (2026)LLM judgeLLM (GPT-4.1)AUROC Kunitomo-Jacquin et al. (2025) LLM judgeLLM ( Meta-Llama-3- 8B-Instruct) AUROC Huang et al. (2025)Human labellingHuman labelingAUROC Cecere et al. (2025)LLM judge LLM (Claude Haiku 3.5) AUROC, PR-AUC, AURAC Bodhwanietal. (2025) Human labelling Human judges (from datasets) AUPRC,AUROC,ECE, Brier Vashurinetal. (2025a) LLM judge, Overlap based Align-Score, COMET, Rouge-L Prediction Rejection Ratio (PRR) Vazhentsevetal. (2025) LLM judge, Overlap based RougeL, FactScore Prediction Rejection Ratio (PPR), AUROC, PR-AUC V et al. (2025)Exact/fuzzy matchingCoverEMnDCG@k, R@k Chen et al. (2025)Human labelling Human judges (au- thors) Predictive Rate Ratio (PRR) Ishii et al. (2025) LLM judge, Human validation LLM judge (GPT-4.1), Human validation ECE,Brier,AUROC, AUPRC Joo and Cho (2025)Overlap basedRouge-L>0.7AUROC, Pearson Correlation Coefficient Yaldiz et al. (2025) LLM judge, Human validation LLM judge (GPT-3.5- turbo), Human valida- tion AUROC, Prediction Rejec- tion Ratio (PRR) Continued on the next page 17 PaperJudge TypeJudgeEvaluation method Wang et al. (2025a)Overlap based, Seman- tic similarity BLEU, BertScorePearson Corr.Coefficient, RCE, Factuality vs Coverage curve, Uncertainty Precision Alignment Yang et al. (2025b)Exact/fuzzy matchingExact match, precision (for multiple correct an- swers) AUROC, PR-AUC Xue et al. (2025b)Exact/fuzzy matching, Human validation positive-recallexact matching,human validation AUROC, ECE Nguyen et al. (2025)Overlap based, Seman- tic similarity F1, RougeL, BertScoreAUROC, AURAC, PRR Xia et al. (2025)LLM judgeLLMjudge( LLaMA3.1-8B- Instruct) AUROC, AUARC Zhang and Zhang (2025) UnidentifiableUnidentifiableAUROC Feng et al. (2025)UnidentifiableUnidentifiableAUROC Bhattacharjya et al. (2025) Overlap basedRougeL>0.5AUROC, ACE, ATS Wang et al. (2025b)Overlap basedRouge-1>=0.3AUROC, ECE, NLL Zhou et al. (2025)UnidentifiableUnidentifiableBrier Score Yoffe et al. (2025)UnidentifiableUnidentifiableAUROC Zhang et al. (2025b) LLM judge, Human validation FactScore, Human vali- dation AUROC, Brier, ECE Vashurinetal. (2025b) Semantic similarityCOMET,XComet- XXL, MetricX-XXL, AlignScore PR-curve,Prediction- Rejection Ratio (PRR) Khanmohammadi et al. (2025) LLM judgeLLM (GPT-4o-mini)ECE,Brier,AUROC, AUCPR, ACC Zhang et al. (2025a)UnidentifiableUnidentifiableAUROC, ECE, Brier Shelmanovetal. (2025) LLM judgeLLM judge (GPT-4o)PR-AUC Continued on the next page 18 PaperJudge TypeJudgeEvaluation method Xue et al. (2025a)Exact/fuzzy matching, Human validation Response in reference or vice versa, Human validation Precision, Truthfulness, AU- ROC He et al. (2025)Semanticsimilarity, Human validation pairwisesimilarity scores with threshold, Human validation ECE, MECE, AUROC Li et al. (2025)LLM judge, Overlap based Rouge-L>0.7 + LLM judge (GPT-turbo-3.5- 061) AUROC, ECE Bakman et al. (2025)LLM judgeLLM judge (GPT-4o- mini) Prediction Rejection Ratio, Average Recall Error Flores et al. (2025)Overlap basedROUGE-L, BLEU, F1Spearman correlation Mora-Crossand Calderon-Ramirez (2024) Semantic similarityBERTScoreECE Tao et al. (2024)LLM judgeLLM judge (GPT-4) ECE, Pearson Correlation, Spearman Correlation Amayuelasetal. (2024) Exact/fuzzy matching, Semantic similarity SimCSE > tau or fuzzy matching AUROC curves, Equal Error Rate (EER) Zhao et al. (2024)Exact/fuzzy matchingExact match after nor- malization ECE, MacroECE Fadeeva et al. (2024)Human labelling, LLM judge Humanlabelling, FactScore(re- trieval+ChatGPT) AUROC, AUPRC Srinivasanetal. (2024) LLM judge LLMjudge (LAVEGPT-3.5) risk, coverage, Effective Re- liability, Selective Prediction Recall Wang et al. (2024)Semantic similarity Sentencesimilarity (DistillRoberta)) > 0.7 AUROC Huang et al. (2024) Overlap based, seman- tic similarity, LLM judge Rouge-L, BERT sim- ilarity,LLM judge (ChatGPT) Rank calibration error Zhang et al. (2024)LLM judge, Human validation FACTSCORE, Human validation Pearson Correlation Coeffi- cient (PCC), Spearman Cor- relation Coefficient (SCC) Continued on the next page 19 PaperJudge TypeJudgeEvaluation method Xu et al. (2024)Exact/fuzzy matchingannotatedanswers must be present within the responses ECE, AUROC Lin et al. (2024)LLM judge, Human validation LLMs as judges ( LLaMA2-70B + gpt- 3.5-turbo),Human validation AUROC, AURAC Liu et al. (2024b) Overlap based, LLM judge ROUGE > 0.3, LLM judge (GPT4) ECE, Brier Liu et al. (2024a)Overlap basedRouge-L > 0.5AUROC Shen et al. (2024)Exact/fuzzy matchingExact MatchECE Cheng and Vlachos (2024) Semantic similarityCOMETKiwiSpearman (ρ) and Pearson (r) correlations Gao et al. (2024)Overlap basedF1 ECE, Pearson Correlation be- tween confidence and accu- racy Mahaut et al. (2024)UnidentifiableUnidentifiableAUPRC Duan et al. (2024) Overlap based, Seman- tic similarity, Human validation Rouge-L > 0.5 + em- bed. similarity (Distil- roberta) > 0.5, Human validation AUROC Chen and Mueller (2024) LLM judge, Human validation LLM judge (GPT- 4), Human validation AUROC Bakman et al. (2024)LLM judgeLLM (GPT-3.5-turbo)AUROC Zablotskaia et al. (2023) Overlap basedROUGE-1,2,LECE, AUROC, Quality vs Ab- stention Curves Chen et al. (2023) Overlap based, Human validation Rouge-L, Human vali- dation AUROC, AUACC Tian et al. (2023) LLM judge, Human validation LLM (GPT4, 3.5), Hu- man validation AUC, ECE, Brier Score Fadeeva et al. (2023)Overlap based, Seman- tic similarity Rouge-L, BERTScorePrediction Rejection Ratio (PRR) Continued on the next page 20 PaperJudge TypeJudgeEvaluation method Mielke et al. (2022)Human labelling, Ex- act/fuzzy matching, Se- mantic similarity Humanlabelling, Match based,Bert based trained classifier ECE, MCE, NLL Dhuliawalaetal. (2022) UnidentifiableUnidentifiableECE, AURC, AURCC Si et al. (2022)Exact/fuzzy matchingExact matchECE, MacroCE Lu et al. (2022)Human labellingHuman labelling Pearson’s correlation, AU- ROC, AUPR Zhang et al. (2021)Exact/fuzzy matchingexact matchAUROC, Cov@Acc XiaoandWang (2021) Exact/fuzzy matching, Human validation Exact match (with syn- onyms of references), BLEU, Human valida- tion Risk - performance curves Wang et al. (2020)Overlap basedTranslation Edit Rate (TER) ECE 21 UQ Eval. metric# PapersProportion AUROC4059.7% ECE2638.8% AUCPR1217.9% AURAC1116.4% Other1116.4% Brier1014.9% PRR811.9% Correlation scores811.9% Table 5: UQ evaluation metrics used in retrieved papers. B Bayesian Data Analysis In Section B.1, we compare automated judges to one or more human judges in terms of their ad- equacy judgements. In Section B.2, we learn to map a diverse collection of uncertainty quantifiers to a comparable notion of confidence, which we use in order to analyse the impact of automated judgements on metrics of selective prediction. In both cases, we employ Bayesian data analysis tech- niques (BDA; Gelman et al., 1995). We implement our hierarchical models using numpyro (Phan et al., 2025). For inference, we con- dition on the available data and draw posterior sam- ples for latent parameters via Markov chain Monte Carlo (MCMC), using numpyro’s implementation of the no U-turn sampler (NUTS; Hoffman and Gelman, 2014). We use standard convergence diag- nostics (Brooks and Gelman, 1998), such as the r- hat statistic, by comparing multiple Markov chains. In all cases, we follow the common approach of ex- perimenting with various choice of priors (as well as tweaks to the model’s hierarchical structure) in order to verify that inferences are stable across a range of sensible choices. In development, we use posterior predictive checks, such as label recon- struction, as a tool for further diagnosing design choices as well as MCMC convergence. These practices are common in BDA and they are widely documented in reference texts (Gelman et al., 1995; McElreath, 2018; Gelman et al., 2026). Our code and notebooks are available for full reproducibility here: https://github.com/probabll/buqeval. In describing our BDA tools, we will refer to a prompt-response pair as an item, each such item is classified by various (human and machine) judges who independently assess whether the response is adequate to the prompt (‘the item is positive’) or not (‘the item is negative’). Judges (sometimes called annotators) can be human or machine. B.1 Judges We have a collection ofIitems, each judged byJ judges, for a total ofN = I×Jobservations. 5 We gather all such observations in anN-dimensional vectoryof binary adequacy labels and use two otherN-dimensional vectors, namelyjandi, to indicate which judge and item each observation in yis about: for somen ∈ [N],y n is 1 (resp., 0) if judgej n regards itemi n as positive (resp., nega- tive). We modelyas a draw from a hierarchical model for i∈ [I] θ i ∼ Beta(α + 1 i ,β + 0 i )(1a) for j∈ [J] p 0,j ∼ Beta(a 0 ,b 0 )(1b) p 1,j ∼ Beta(a 1 ,b 1 )(1c) for n∈ [N] y n |i n = i,j n = j∼ Bernoulli(φ n )(1d) φ n = θ i p 1,j + (1− θ i )(1− p 0,j ) described next. For each itemi∈ [I], the parame- terθ i captures the probability that the item is pos- itive. We draw this parameter from a strongly- informed Beta distribution (1a), where1 i (resp., 0 i ) counts the number of times our human judges labelled the item as positive (resp., negative), and α = β = 0.5. 6 For each judgej ∈ [J]: (1c) p 1,j models the judge’s sensitivity (used to infer the judge’s true positive rate), that is the probabil- ity of a positive judgement given a positive item, and (1b)p 0,j models the judge’s specificity (used to infer the judge’s true negative rate), that is the probability of a negative judgement given a nega- tive item. For these parameters we use uninformed, flatBeta(1, 1)priors. 7 Then, given the latent pa- rameters, the probability that thenth observation is positive (1d) can be obtained by marginalising the unknown adequacy label of the corresponding item (i n ) under the sensitivity and specificity of the corresponding judge (j n ). 5 For simplicity, we describe the model as if every judge had annotated every item once, but our model in fact supports missing data as well as multiple judgements. 6 In effect, we drawθ i from the posterior distribution of a Beta-Binomial model of the observed human annotation, with a sparse and symmetric Beta(0.5, 0.5) prior. 7 Except if we are modelling a human judge, for example in order to study how humans differ as judges, in which case we use a weakly informedBeta(1, 0.5)prior that skews both sensitivity and specificity towards 1 to encode the knowledge that we trust the annotator. 22 Inferences. For each judgej∈ [J], we can then infer the following posterior classification metrics TPR j =E " 1 I I X i=1 θ i p 1,j # (2a) TNR j =E " 1 I I X i=1 (1− θ i )p 0,j # (2b) FPR j =E " 1 I I X i=1 (1− θ i )(1− p 0,j ) # (2c) FNR j =E " 1 I I X i=1 θ i (1− p 1,j ) # (2d) by estimating the expected value using samples fromp(θ,p 0 ,p 1 |y,j,i). These metrics can then be used to express posterior accuracy and posterior F1. B.2 Uncertainty Quantifiers We have a collection ofIitems, each judged byJ human judges and qualified byQuncertainty quan- tifiers, for a total of N = I × J × Q observations. We gather all such observations in a collection of N-dimensional vectors (y,x,i,j,q):y n is 1 (resp., 0) if judgej n regards itemi n as positive (resp., neg- ative), whilex n is the uncertainty (or confidence) score assigned toi n by UQq n . We modely|xas a draw from a hierarchical model described next. β ∼N(0, 2 2 )(3a) ω ∼N(0, 2 2 )(3b) for j∈ [J] b j ∼N(β, 2 2 )(3c) w j ∼N(ω, 2 2 )(3d) for q∈ [Q] c q ∼N(0, 2 2 )(3e) m q ∼N(0, 2 2 )(3f) for n∈ [N] y n |x n ,j n = j,i n = i∼ Bernoulli(φ n )(3g) φ n = sigmoid(u n + v n ) u n = b j + w j x n (3h) v n = c q + m q x n (3i) The model is an instance of item-response theory model, with offset and slopes for judges and quan- tifiers, all of which are modelled by Gaussians. We group the parameters of the human judges (3c) and (3d) under shared priors (3a) and (3b), but keep the quantifiers’ parameters separate (3e) and (3f). Finally, given the latent parameters, the prob- ability that thenth observation is positive (3g) is obtained by sigmoid-transforming the sum of the judge-specific linear predictor (3h) and the UQ- specific linear predictor (3i). Inferences. Having obtained samples from p(β,ω,b,w,c,m|y,x,q,j,i), we can compute posterior confidencesφ 1 ,...,φ N , from which we can obtain a point estimate for the posterior confi- dencep q,i underq ∈ [Q]with which itemi ∈ [I] is positive: we use the posterior median of all sam- plesφ n for whichi n = iandq n = q. We use the posterior median across samples C Implementation details C.1 Uncertainty quantifiers We present below additional information and im- plementation details for each of the uncertainty quantifiers used in our experiments. Log-ProbabilityWe compute the log-probability of a responseY = yby decomposing it in to- kens and computing their corresponding sum of log-probabilities. logP(Y = y | X = x) =− N X i=1 logp(t i | x,t <i ) (4) where y = t 1 ,...,t N . Normalised Log-Probability We interpret this quantifier either as the average token log- probability, or the length-normalised log probabil- ity of the responsey(i.e. perplexity). It simply is the log-probability from Equation (4), divided by the number of tokens in y. perplexity =− 1 n logP(Y = y|X = x)(5) Predictive entropy The entropy over complete responsesYgiven a promptX = x, is given by the formula below: H(Y|X = x) =− X y∈Y p(y|x) logp(y|x) , (6) where the summation is over the entire (count- ably infinite) spaceYof possible responses. We es- timate this quantity via Monte Carlo; MC. We sam- ple 10 responses, obtain the probability of each via 23 relative frequency and compute the entropy with the formula above. Semantic entropyTo compute semantic entropy (Kuhn et al., 2023), sampled responses are mapped to semantic clusters and entropy is estimated over those. We use Aichberger et al. (2025)’s implemen- tation: SE(Y|X = x)≈− X c j p(c j |x) logp(c j |x), (7) wherec 1 ,...,c J are clusters formed using a natural language inference (NLI) model to as- sess semantic equivalence via bi-directional en- tailment.p(c j |x)equal the number of responses, incl. repetitions, in clusterc j divided by sample size). We use 10 unbiased sampled responses and deberta-large-mnli as the NLI model. P(True) Following Kadavath et al. (2022) we prompt a model to generate true or false regard- ing the truthfulness of its own previously sampled response to a prompt. Prompt: Question: ’<question>’ Here are some brainstormed ideas: ’<responses>’ Possible answer: ’<prediction>’ Is the possible answer: (a) true (b) false Generate a or b. The answer is: We assess the log-probabilities assigned by the models to the sequences P(‘a’|prompt) and P(‘b’|prompt), using the model’s output layer’s log- its. We normalise to obtain the probability assigned to P(‘a’) as a proxy to P(True). Semantic Confidence Inspired by Yona et al. (2024), we compute semantic confidence in a re- sponse as the relative frequency that a response is not contradicted by sampled responses. Contradic- tion is assessed by an NLI model. SC(Y = y|X = x) = 1− P y s ∈[y 1 ,y n ] 1y ̸= y s n , (8) Two responses are deemed to be contradicted if the NLI model assigns higher probability to con- tradiction than the sum of the probability of en- tailment and neutrality. We use the 10 unbiased sampled responses anddeberta-large-mnlias the NLI model. Verbalised ConfidenceWe prompt the model for a confidence value in its own previously generated response. The prompt used is found below: You are a helpful assistant that generates faithful numerical confidence values in an answer. Question: <question> Answer previously generated: <prediction> Generate ONLY your numerical confidence in the answer (which is 0-1), up to 2 decimal places. Confidence: C.2 Correctness Criteria We proceed to explain further how we implemented the various correctness criteria. Exact matching We strip and lowercase the re- sponse string and assess whether it matches exactly any stripped and lowercased reference responses. Token F1 For each response and reference pair (if multiple references exist), we lowercase and split them in words and retrieve the sum of common ones. Precision is defined as the sum of common words divided by the number of words in the re- sponse and recall is defined as the sum of common words divided by the number of words in the ref- erence. Using these we compute an F1 score. We use the maximum F1 score across references and assess whether it surpasses a pre-defined threshold. BLEU Score We use thesentence_bleufrom thenltk.translate.bleu_scorelibrary to cal- culate the BLEU score for the response against the reference sentences. Rouge-L We employ Rouge-L, which mea- sures the longest common subsequence be- tween a response and a reference, via the rouge_scorer.RougeScorerfunction from the rouge_scorelibrary. We obtain the max score across references and assess whether it surpasses a pre-defined threshold. Embedding similarity We embed the response and the references. We compute the consine simi- larity of each response-reference pair. We take the maximum value and compare it to a threshold. The embedding model we use is all-MiniLM-L6-v2. BERT score Frombert_scorewe use the scorefunction to compute the semantic similarity 24 between the response and references. The maxi- mum value is used against a pre defined threshold. LLM as judge We prompt a model using the prompt, response and reference answers for its cor- rectness. The prompt is presented below: You are a judge that evaluates the correctness of an answer given a question and a list of reference answers. Even if one of the reference answers is semantically equivalent to the answer, we can consider it as correct. Generate a True or False answer. "Question: <question> Reference answers: <references> Answer: <answer> Is the answer correct? We employ 2 judges: an open source model, Llama-3.1-8B-Instruct, and a closed source API model, gpt-5.4-mini. D Results D.1 Extensive results For transparency, we present the complete results for all dataset and generator pairs in Figures 7 - 12. D.2 Inter-Annotator agreement analysis As manual annotation is expensive, we sampled 100 prompts for each dataset-generator pair. These were annotated by an annotator (one of the authors). We use the following guidelines: 1.Identify whether the response is correct or incorrect, given the set of references. 2.To deem a response as correct, it should con- tain at least one of the reference answers. 3. In the absence of a definitive answer under any circumstances — e.g. the model refuses to answer, claims not to know the answer etc. — mark this as incorrect. 4.If the answer is nonsensical (e.g. repetition of identical strings, non grammatical text etc.), mark this as incorrect. 5. If the response contains the correct answer, as well as other complementary claims or facts, you do not have to verify their correctness in order to mark the response as correct — ex- cept if the accompanying claims are irrational and nonsensical to the extent that the correct- ness of the entire response has to be deemed incorrect as per your judgement. To verify that our results are not sensitive to the choice of judge, we employ a new annotator (another one of the authors) to annotate two sets of samples from TriviaQA-Llama 8B and AmbigQA (ambiguous)-Llama 8B. For the former, Cohen’s Kappa score between annotators 0.88 and for the latter 0.75. We present result figures, side by side, for both annotators (see Figures 13 and 14). D.3 Bootstrapping analysis To further validate our results, we perform a boot- strapping analysis: sampling 20 prompts 10 times, and computing the average performance metrics across runs. We compute the average AUROC (and its associated standard deviation values) across runs and the ranking of uncertainty quantifiers based on the average AUROC scores. To be less sensitive to small deviations of AUROC values when ranking, we assign 2 UQs the same ranking if their AUROCs do not vary by more than 0.025. Results appear in Figures 15 - 20. The quantifiers are ordered by decreasing quality (using F1 score comparing their predictions with human annotations as a proxy), with the leftmost scores showing the actual AU- ROC values and rankings (which are computed us- ing manual correctness labels). The results present a similar picture with the original outcomes. D.4 Alternative metrics Beyond AUROC, other metrics can measure perfor- mance trade-offs in SP. We perform a similar anal- ysis for various automated judges and UQs. We in- vestigate the prompts from TriviaQA and all genera- tors. We assess: coverage at fixed risk (risk = 0.2) and risk as fixed coverage (coverage = 0.8). The results are shown in Figures 21 - 24. Results gen- erally extend to other SP metrics — with better quantifiers getting negatively impacted to a higher extent, obscuring conclusions. D.5 F1 Distributions For transparency, we present all F1 posterior dis- tributions for all dataset-generator pairs. They are shown in Figures 25 - 27. D.6 Ranking correlations We show the plots representing F1 scores of auto- mated judges against RBO scores for all dataset- generator pairs in Figure 28 25 Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. P(true) Uncertainty metric Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion 0.520.710.550.780.850.710.71 0.850.980.451.001.001.000.92 0.850.980.451.001.001.000.92 0.640.710.380.500.630.660.86 0.630.960.530.991.000.830.67 0.850.980.451.001.001.000.92 0.931.000.681.001.001.000.85 0.931.000.681.001.001.000.85 0.931.000.681.001.001.000.85 0.600.740.500.870.920.690.68 0.630.580.480.530.570.520.57 0.530.550.570.490.500.520.50 0.720.790.570.610.730.750.77 0.700.730.680.470.640.660.69 0.530.600.580.500.490.520.51 0.490.630.570.740.810.590.63 0.700.740.520.660.670.610.49 0.500.580.500.710.770.550.58 0.530.580.520.730.760.540.55 0.590.580.490.480.490.530.52 0.620.620.490.560.530.540.45 0.530.600.550.730.790.570.59 0.570.550.500.570.540.520.50 0.720.630.580.600.550.520.49 0.760.610.520.580.550.520.54 0.730.620.570.560.550.520.49 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Sem. conf. P(true) P(seq.) N.P(seq.) Verb. conf. Sem. entropy Ent. Uncertainty metric Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 0.770.700.980.980.980.950.99 0.770.700.980.980.980.950.99 0.400.470.830.790.430.680.88 0.770.700.980.980.980.950.99 0.770.700.960.960.650.820.96 0.780.700.970.970.820.890.98 0.780.700.970.970.820.890.98 0.780.700.970.970.820.890.98 0.780.710.980.980.880.920.97 0.700.710.970.950.910.860.97 0.630.650.950.930.790.860.96 0.650.670.960.940.760.850.91 0.630.680.700.720.680.680.69 0.640.690.900.850.710.710.78 0.640.680.950.910.670.750.86 0.690.660.650.650.560.610.63 0.620.660.860.830.660.660.79 0.630.680.820.800.660.630.74 0.700.680.570.570.590.580.56 0.710.810.580.610.460.910.63 0.750.770.580.590.510.790.59 0.690.650.620.610.550.570.62 0.720.690.600.580.590.560.56 0.770.680.640.630.640.680.61 0.820.810.730.730.730.750.67 0.810.790.780.760.710.710.68 0.820.790.740.720.700.690.67 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. P(true) Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.210.09-0.020.230.300.190.22 0.120.36-0.120.440.450.480.43 0.120.36-0.120.440.450.480.43 -0.090.09-0.19-0.060.080.140.36 -0.100.34-0.040.430.450.310.18 0.120.36-0.120.440.450.480.43 0.200.380.110.440.450.480.36 0.200.380.110.440.450.480.36 0.200.380.110.440.450.480.36 -0.130.12-0.080.320.370.170.18 -0.10-0.05-0.09-0.030.02-0.000.07 -0.19-0.07-0.00-0.06-0.05-0.000.01 -0.010.17-0.000.050.180.230.27 -0.030.110.11-0.080.090.140.19 -0.20-0.020.01-0.05-0.06-0.000.02 -0.240.01-0.000.190.260.070.14 -0.030.12-0.050.110.120.09-0.00 -0.23-0.04-0.080.160.220.030.08 -0.20-0.04-0.050.170.210.020.06 -0.14-0.04-0.08-0.07-0.060.010.02 -0.10-0.00-0.080.01-0.020.02-0.04 -0.20-0.02-0.020.180.240.050.10 -0.16-0.08-0.070.01-0.010.000.00 -0.000.010.010.040.000.00-0.00 0.04-0.01-0.050.020.00-0.000.04 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Sem. conf. P(true) P(seq.) N.P(seq.) Verb. conf. Sem. entropy Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.05-0.090.240.260.280.260.32 -0.05-0.090.240.260.280.260.32 -0.42-0.320.080.07-0.26-0.010.21 -0.05-0.090.240.260.280.260.32 -0.05-0.090.220.24-0.050.130.29 -0.05-0.080.230.260.120.200.31 -0.05-0.080.230.260.120.200.31 -0.05-0.080.230.260.120.200.31 -0.04-0.080.240.260.180.230.30 -0.13-0.080.230.230.210.170.30 -0.19-0.140.210.210.100.170.29 -0.18-0.120.210.220.070.160.24 -0.19-0.11-0.050.01-0.02-0.020.02 -0.18-0.100.150.130.010.020.11 -0.18-0.110.200.19-0.020.060.19 -0.13-0.13-0.10-0.06-0.14-0.08-0.04 -0.20-0.130.120.11-0.03-0.030.12 -0.19-0.110.080.08-0.03-0.060.07 -0.12-0.11-0.17-0.14-0.11-0.11-0.11 -0.110.02-0.16-0.11-0.240.21-0.04 -0.07-0.02-0.16-0.12-0.190.10-0.08 -0.13-0.13-0.12-0.11-0.15-0.12-0.05 -0.10-0.10-0.14-0.14-0.11-0.13-0.11 -0.06-0.11-0.11-0.08-0.06-0.01-0.06 0.000.03-0.010.010.030.06-0.00 -0.010.010.030.050.010.020.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. P(true) Uncertainty metric Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion 7462135 6473215 6473215 4276531 6372145 6473215 5274316 5274316 5274316 6372145 1275364 3217645 5176432 2147653 3126745 7462153 2164357 6372154 6372154 1267534 1263547 7362154 1362457 1243567 1273465 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Qwen 0.5B, TriviaQA Sem. conf. P(true) P(seq.) N.P(seq.) Verb. conf. Sem. entropy Ent. Uncertainty metric Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 6723451 6723451 7523641 6723451 5623741 6723541 6723541 6723541 6712543 7623451 7623541 7612543 7521463 7612453 7512643 1243765 7612543 7412563 1256347 3265714 3264715 1235764 1235467 1256437 1245637 1234657 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Qwen 7B, TriviaQA Figure 7: Top row shows AUROC values, middle row shows differences between oracle and system AUROC values and bottom row rankings for various uncertainty quantifiers (ordered from left to right in a descending order of performance) and correctness functions (ordered from top to bottom in an increasing order of quality). 26 P(seq.) N.P(seq.) P(true) Verb. conf. Ent. Sem. entropy Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 0.960.630.930.790.980.950.78 0.960.630.930.790.980.950.78 0.960.630.930.790.980.950.78 0.960.630.930.790.980.950.78 0.960.630.930.790.980.950.78 0.980.800.910.800.990.960.78 0.960.810.740.800.970.900.62 0.900.420.840.800.920.900.78 0.900.420.840.800.920.900.78 0.900.420.840.800.920.900.78 0.900.420.840.800.920.900.78 0.980.860.790.800.980.920.67 0.940.600.860.800.950.920.78 0.950.700.780.800.960.910.70 0.780.450.720.650.800.700.60 0.790.480.750.710.840.740.61 0.760.400.770.660.790.700.66 0.930.810.780.660.940.890.77 0.770.460.700.610.730.670.62 0.790.570.790.670.810.770.68 0.840.680.750.680.800.700.68 0.820.650.700.660.790.710.69 0.710.610.730.710.680.650.58 0.770.680.710.690.690.650.61 0.850.750.710.730.740.660.62 0.760.750.940.740.680.810.64 0.760.620.890.810.630.600.48 0.740.660.760.750.700.630.61 0.780.750.850.800.690.760.60 0.770.700.810.860.720.660.59 0.780.750.810.820.720.730.58 0.860.750.790.780.760.710.64 0.820.790.810.810.740.720.68 0.850.810.830.850.750.690.66 0.880.830.820.790.760.690.67 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Verb. conf. P(seq.) N.P(seq.) Sem. entropy P(true) Ent. Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.960.670.880.850.960.69 0.740.830.210.690.740.860.69 0.740.830.210.690.740.860.69 0.740.830.210.690.740.860.69 0.750.980.790.890.760.980.69 0.590.790.670.790.550.750.69 0.750.880.460.770.750.900.69 0.750.880.550.780.680.920.69 0.750.830.520.740.650.890.69 0.750.860.550.780.680.900.61 0.690.840.580.780.660.830.64 0.690.840.570.720.640.810.64 0.690.770.580.770.590.790.64 0.740.870.730.820.610.880.68 0.690.810.700.720.610.770.69 0.740.860.730.760.590.850.65 0.700.810.660.710.600.790.62 0.670.780.670.660.620.740.59 0.700.840.760.700.550.730.58 0.690.780.700.680.630.690.57 0.730.790.680.650.730.750.66 0.830.810.880.930.960.700.66 0.750.820.710.660.750.700.64 0.860.840.790.750.780.710.61 0.810.910.690.760.670.700.48 0.780.880.770.680.650.730.55 0.850.810.690.710.770.690.61 0.860.790.730.770.740.720.71 0.930.790.790.810.710.730.72 0.920.840.780.760.740.740.69 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC P(seq.) N.P(seq.) P(true) Verb. conf. Ent. Sem. entropy Sem. conf. Uncertainty metric (sorted by human-label AUROC) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) 0.08-0.210.110.010.220.260.11 0.08-0.210.110.010.220.260.11 0.08-0.210.110.010.220.260.11 0.08-0.210.110.010.220.260.11 0.08-0.210.110.010.220.260.11 0.10-0.040.090.010.230.270.11 0.08-0.03-0.080.010.210.21-0.05 0.02-0.410.020.010.150.210.11 0.02-0.410.020.010.150.210.11 0.02-0.410.020.010.150.210.11 0.02-0.410.020.010.150.210.11 0.090.03-0.030.010.220.230.00 0.06-0.230.040.010.180.230.11 0.07-0.13-0.040.020.190.220.03 -0.10-0.38-0.09-0.130.030.01-0.07 -0.09-0.35-0.07-0.080.080.05-0.06 -0.12-0.43-0.05-0.130.020.01-0.01 0.05-0.02-0.04-0.120.180.200.10 -0.11-0.37-0.12-0.17-0.03-0.02-0.05 -0.09-0.27-0.03-0.120.040.080.01 -0.04-0.15-0.06-0.110.040.000.01 -0.06-0.18-0.12-0.130.030.020.02 -0.18-0.23-0.09-0.08-0.08-0.04-0.09 -0.11-0.16-0.11-0.10-0.08-0.04-0.06 -0.04-0.08-0.11-0.06-0.02-0.03-0.05 -0.13-0.080.12-0.05-0.080.12-0.03 -0.12-0.210.070.02-0.13-0.09-0.19 -0.14-0.18-0.06-0.04-0.06-0.06-0.06 -0.11-0.080.030.01-0.070.07-0.07 -0.11-0.13-0.010.07-0.04-0.03-0.08 -0.10-0.09-0.010.04-0.040.04-0.09 -0.02-0.08-0.03-0.01-0.000.02-0.03 -0.06-0.04-0.010.02-0.030.020.01 -0.03-0.030.010.07-0.02-0.00-0.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Verb. conf. P(seq.) N.P(seq.) Sem. entropy P(true) Ent. Sem. conf. Uncertainty metric (sorted by human-label AUROC) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.12-0.100.110.110.230.00 -0.18-0.02-0.56-0.07-0.000.130.00 -0.18-0.02-0.56-0.07-0.000.130.00 -0.18-0.02-0.56-0.07-0.000.130.00 -0.170.140.010.130.020.240.00 -0.33-0.05-0.110.02-0.190.020.00 -0.170.04-0.310.000.010.170.00 -0.170.04-0.220.02-0.060.180.01 -0.17-0.02-0.26-0.02-0.090.150.01 -0.170.02-0.220.02-0.070.17-0.07 -0.23-0.00-0.200.02-0.090.09-0.05 -0.23-0.01-0.20-0.05-0.100.07-0.05 -0.23-0.07-0.200.00-0.150.05-0.05 -0.180.02-0.040.06-0.130.14-0.00 -0.24-0.03-0.08-0.04-0.130.040.00 -0.190.02-0.05-0.00-0.150.12-0.04 -0.22-0.04-0.12-0.05-0.140.06-0.06 -0.25-0.06-0.11-0.10-0.120.00-0.09 -0.230.00-0.01-0.06-0.19-0.01-0.11 -0.23-0.07-0.07-0.08-0.12-0.04-0.12 -0.19-0.05-0.10-0.11-0.010.01-0.03 -0.09-0.030.110.160.22-0.03-0.03 -0.17-0.02-0.07-0.110.01-0.04-0.05 -0.06-0.010.01-0.010.04-0.02-0.07 -0.110.07-0.09-0.00-0.07-0.04-0.21 -0.140.04-0.01-0.09-0.09-0.00-0.14 -0.07-0.03-0.09-0.060.03-0.05-0.07 -0.07-0.06-0.050.00-0.00-0.010.03 0.00-0.050.010.04-0.03-0.000.04 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference P(seq.) N.P(seq.) P(true) Verb. conf. Ent. Sem. entropy Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 2745136 2745136 2745136 2745136 2745136 2546137 2465137 2745136 2745136 2745136 2745136 2465137 2745136 2754136 2735146 2735146 3726145 2457136 1736245 2736145 1537246 1746235 2613457 1524367 1254367 3415627 3512467 3512467 3512647 3521467 3421657 1523467 1423567 2431567 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Llama 3B, TriviaQA Verb. conf. P(seq.) N.P(seq.) Sem. entropy P(true) Ent. Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 5274316 5274316 5274316 5274316 5274316 5273416 3275416 3275416 3275416 6143527 6152734 5273416 4273615 3274615 4273516 4173526 4173526 4273615 4253716 6143725 4153726 4153726 3145627 5124736 4125637 4157326 4532167 3146257 1235467 2153647 2135647 1254367 1253467 1342756 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Llama 8B, TriviaQA Figure 8: Top row shows AUROC values, middle row shows differences between oracle and system AUROC values and bottom row rankings for various uncertainty quantifiers (ordered from left to right in a descending order of performance) and correctness functions (ordered from top to bottom in an increasing order of quality). 27 N.P(seq.) Sem. conf. P(true) P(seq.) Sem. entropy Verb. conf. Ent. Uncertainty metric Token F1 (=0.3) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.1) · F1=0.31 Token F1 (=0.2) · F1=0.32 Emb. sim. (=0.7) · F1=0.32 Emb. sim. (=0.6) · F1=0.40 Emb. sim. (=0.2) · F1=0.41 BERTScore (=0.8) · F1=0.41 Token F1 (=0.1) · F1=0.50 ROUGE-L (=0.1) · F1=0.53 Emb. sim. (=0.3) · F1=0.53 ROUGE-L (=0.2) · F1=0.55 LLM judge (Llama 8B) · F1=0.64 Emb. sim. (=0.4) · F1=0.65 LLM judge (GPT) · F1=0.74 Emb. sim. (=0.5) · F1=0.74 Human labels · F1=1.00 Correctness criterion 0.970.320.530.950.420.640.49 0.660.120.610.670.420.640.49 0.480.580.460.480.470.580.51 0.880.450.670.880.600.640.67 0.930.470.370.930.420.750.49 0.900.480.440.850.420.650.49 0.590.570.540.540.510.520.51 0.660.530.590.660.490.450.51 0.840.470.640.800.500.530.54 0.750.460.670.770.490.550.54 0.650.520.610.630.520.510.51 0.900.480.700.910.590.650.58 0.760.580.750.670.600.470.56 0.700.560.610.660.500.490.52 0.640.700.650.640.620.600.55 0.780.560.580.760.460.540.49 0.660.640.630.610.590.570.53 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC P(true) Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. Uncertainty metric Emb. sim. (=0.7) · F1=0.04 Token F1 (=0.2) · F1=0.09 ROUGE-L (=0.3) · F1=0.13 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.1) · F1=0.39 ROUGE-L (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.64 Emb. sim. (=0.1) · F1=0.65 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 BERTScore (=0.8) · F1=0.70 Emb. sim. (=0.3) · F1=0.75 LLM judge (Llama 8B) · F1=0.86 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion 0.790.830.430.690.370.310.44 0.640.710.530.690.800.740.44 0.690.620.930.690.950.970.81 0.720.550.710.740.880.880.83 0.770.680.680.650.640.580.56 0.730.570.720.690.880.890.70 0.680.500.660.730.840.830.65 0.760.710.790.680.760.690.63 0.670.700.690.590.710.640.57 0.610.730.690.560.650.570.55 0.770.760.760.650.730.680.61 0.710.660.650.600.730.680.55 0.700.720.730.610.690.600.58 0.830.740.720.710.760.650.62 0.830.790.790.660.700.610.60 0.840.810.760.700.680.580.58 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC N.P(seq.) Sem. conf. P(true) P(seq.) Sem. entropy Verb. conf. Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.3) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.1) · F1=0.31 Token F1 (=0.2) · F1=0.32 Emb. sim. (=0.7) · F1=0.32 Emb. sim. (=0.6) · F1=0.40 Emb. sim. (=0.2) · F1=0.41 BERTScore (=0.8) · F1=0.41 Token F1 (=0.1) · F1=0.50 ROUGE-L (=0.1) · F1=0.53 Emb. sim. (=0.3) · F1=0.53 ROUGE-L (=0.2) · F1=0.55 LLM judge (Llama 8B) · F1=0.64 Emb. sim. (=0.4) · F1=0.65 LLM judge (GPT) · F1=0.74 Emb. sim. (=0.5) · F1=0.74 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) 0.31-0.32-0.100.34-0.170.07-0.04 -0.00-0.52-0.020.06-0.170.07-0.04 -0.18-0.06-0.17-0.13-0.130.01-0.03 0.22-0.190.050.270.000.070.14 0.27-0.17-0.260.33-0.180.18-0.04 0.24-0.16-0.190.24-0.180.08-0.04 -0.07-0.07-0.09-0.07-0.08-0.05-0.02 0.00-0.11-0.040.05-0.10-0.12-0.02 0.18-0.170.020.20-0.09-0.040.01 0.10-0.180.040.17-0.10-0.020.00 -0.01-0.12-0.010.02-0.08-0.07-0.02 0.24-0.160.070.30-0.000.080.05 0.10-0.060.130.060.00-0.110.02 0.04-0.08-0.020.05-0.09-0.08-0.01 -0.020.060.020.030.030.030.01 0.12-0.08-0.050.15-0.14-0.03-0.04 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference P(true) Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. Uncertainty metric (sorted by human-label AUROC) Emb. sim. (=0.7) · F1=0.04 Token F1 (=0.2) · F1=0.09 ROUGE-L (=0.3) · F1=0.13 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.1) · F1=0.39 ROUGE-L (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.64 Emb. sim. (=0.1) · F1=0.65 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 BERTScore (=0.8) · F1=0.70 Emb. sim. (=0.3) · F1=0.75 LLM judge (Llama 8B) · F1=0.86 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.040.02-0.32-0.01-0.31-0.27-0.14 -0.20-0.09-0.23-0.010.120.16-0.14 -0.15-0.190.17-0.010.260.390.24 -0.11-0.26-0.050.050.200.300.25 -0.06-0.12-0.08-0.05-0.04-0.01-0.02 -0.11-0.24-0.03-0.010.200.310.12 -0.16-0.30-0.100.030.160.250.08 -0.07-0.100.03-0.020.080.110.05 -0.17-0.11-0.07-0.110.030.06-0.01 -0.22-0.07-0.07-0.14-0.04-0.01-0.02 -0.07-0.050.01-0.050.050.100.03 -0.13-0.15-0.11-0.100.050.09-0.02 -0.13-0.08-0.03-0.090.000.020.00 -0.00-0.07-0.030.010.070.060.04 -0.00-0.020.03-0.040.010.030.02 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference N.P(seq.) Sem. conf. P(true) P(seq.) Sem. entropy Verb. conf. Ent. Uncertainty metric Token F1 (=0.3) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.1) · F1=0.31 Token F1 (=0.2) · F1=0.32 Emb. sim. (=0.7) · F1=0.32 Emb. sim. (=0.6) · F1=0.40 Emb. sim. (=0.2) · F1=0.41 BERTScore (=0.8) · F1=0.41 Token F1 (=0.1) · F1=0.50 ROUGE-L (=0.1) · F1=0.53 Emb. sim. (=0.3) · F1=0.53 ROUGE-L (=0.2) · F1=0.55 LLM judge (Llama 8B) · F1=0.64 Emb. sim. (=0.4) · F1=0.65 LLM judge (GPT) · F1=0.74 Emb. sim. (=0.5) · F1=0.74 Human labels · F1=1.00 Correctness criterion 1742635 2741635 4175623 2731654 2571634 1562734 1243657 1432675 1732654 2731645 1432576 2731546 1523476 1432675 3124567 1432756 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Qwen 0.5B, AmbigQA (unambiguous) P(true) Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. Uncertainty metric Emb. sim. (=0.7) · F1=0.04 Token F1 (=0.2) · F1=0.09 ROUGE-L (=0.3) · F1=0.13 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.1) · F1=0.39 ROUGE-L (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.64 Emb. sim. (=0.1) · F1=0.65 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 BERTScore (=0.8) · F1=0.70 Emb. sim. (=0.3) · F1=0.75 LLM judge (Llama 8B) · F1=0.86 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion 2153674 5364127 6735214 5764123 1234567 3746215 4753126 2416357 4236157 4126357 1326457 2456137 3215467 1345267 1235467 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Qwen 7B, AmbigQA (unambiguous) Figure 9: Top row shows AUROC values, middle row shows differences between oracle and system AUROC values and bottom row rankings for various uncertainty quantifiers (ordered from left to right in a descending order of performance) and correctness functions (ordered from top to bottom in an increasing order of quality). 28 N.P(seq.) P(true) Verb. conf. Sem. conf. Sem. entropy P(seq.) Ent. Uncertainty metric ROUGE-L (=0.4) · F1=0.04 BLEU (=0.1) · F1=0.09 BERTScore (=0.9) · F1=0.09 Emb. sim. (=0.7) · F1=0.12 Token F1 (=0.3) · F1=0.24 ROUGE-L (=0.3) · F1=0.33 Token F1 (=0.2) · F1=0.44 Emb. sim. (=0.6) · F1=0.50 Emb. sim. (=0.5) · F1=0.52 ROUGE-L (=0.2) · F1=0.55 Token F1 (=0.1) · F1=0.60 Emb. sim. (=0.1) · F1=0.66 BERTScore (=0.8) · F1=0.68 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 ROUGE-L (=0.1) · F1=0.72 LLM judge (Llama 8B) · F1=0.78 Emb. sim. (=0.3) · F1=0.79 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion 0.980.770.860.790.860.960.93 0.860.550.790.460.480.820.66 0.880.570.870.540.770.870.83 0.890.790.870.620.760.900.72 0.890.640.740.550.690.890.77 0.930.640.720.580.750.910.81 0.810.690.750.580.690.820.75 0.820.740.750.620.690.780.72 0.810.700.690.620.690.780.71 0.950.700.740.630.740.930.76 0.790.650.620.560.600.810.70 0.730.590.730.390.520.640.63 0.690.640.690.470.530.720.65 0.700.650.660.480.600.660.61 0.800.650.720.580.680.710.65 0.810.630.700.520.570.840.67 0.840.790.780.620.720.760.68 0.740.690.690.570.630.660.63 0.770.780.780.670.670.670.62 0.790.780.740.680.630.620.59 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Verb. conf. Sem. entropy N.P(seq.) P(true) Sem. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.5) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 ROUGE-L (=0.7) · F1=0.03 ROUGE-L (=0.8) · F1=0.03 ROUGE-L (=0.9) · F1=0.03 Emb. sim. (=0.8) · F1=0.03 Emb. sim. (=0.9) · F1=0.03 BERTScore (=0.9) · F1=0.06 ROUGE-L (=0.5) · F1=0.09 Token F1 (=0.4) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.7) · F1=0.12 Token F1 (=0.3) · F1=0.15 ROUGE-L (=0.4) · F1=0.15 ROUGE-L (=0.3) · F1=0.32 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.2) · F1=0.38 ROUGE-L (=0.2) · F1=0.55 Emb. sim. (=0.5) · F1=0.55 Token F1 (=0.1) · F1=0.65 Emb. sim. (=0.4) · F1=0.77 BERTScore (=0.7) · F1=0.78 ROUGE-L (=0.1) · F1=0.80 Emb. sim. (=0.1) · F1=0.80 BERTScore (=0.8) · F1=0.83 Emb. sim. (=0.2) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.89 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 0.870.910.860.620.780.960.96 0.870.660.850.610.320.900.40 0.150.790.240.500.100.950.99 0.150.790.240.500.100.950.99 0.150.790.240.500.100.950.99 0.150.790.240.500.100.950.99 0.150.790.240.500.100.950.99 0.150.790.240.500.100.950.99 0.510.850.550.560.440.960.98 0.630.790.650.580.400.950.79 0.620.700.570.480.490.820.69 0.740.520.680.510.580.770.54 0.620.610.680.620.530.830.82 0.620.580.590.510.480.810.63 0.620.660.640.560.550.850.63 0.720.690.760.580.560.890.72 0.710.630.630.550.610.680.58 0.700.510.580.610.480.770.54 0.730.660.760.580.520.870.67 0.670.730.740.570.650.690.65 0.710.620.720.610.550.840.65 0.710.650.710.600.590.660.56 0.970.900.100.970.550.810.60 0.720.590.750.630.540.850.61 0.730.580.540.610.530.650.61 0.700.530.550.610.490.750.58 0.770.610.530.560.550.610.62 0.730.650.650.620.590.650.60 0.750.690.690.680.600.720.61 0.770.680.710.690.610.680.59 0.780.710.700.680.670.650.58 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC N.P(seq.) P(true) Verb. conf. Sem. conf. Sem. entropy P(seq.) Ent. Uncertainty metric (sorted by human-label AUROC) ROUGE-L (=0.4) · F1=0.04 BLEU (=0.1) · F1=0.09 BERTScore (=0.9) · F1=0.09 Emb. sim. (=0.7) · F1=0.12 Token F1 (=0.3) · F1=0.24 ROUGE-L (=0.3) · F1=0.33 Token F1 (=0.2) · F1=0.44 Emb. sim. (=0.6) · F1=0.50 Emb. sim. (=0.5) · F1=0.52 ROUGE-L (=0.2) · F1=0.55 Token F1 (=0.1) · F1=0.60 Emb. sim. (=0.1) · F1=0.66 BERTScore (=0.8) · F1=0.68 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 ROUGE-L (=0.1) · F1=0.72 LLM judge (Llama 8B) · F1=0.78 Emb. sim. (=0.3) · F1=0.79 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) 0.19-0.010.130.110.230.340.34 0.07-0.230.05-0.22-0.160.200.07 0.09-0.210.13-0.140.140.250.25 0.100.010.13-0.060.130.280.13 0.10-0.140.01-0.130.060.270.18 0.14-0.14-0.01-0.100.120.290.22 0.02-0.090.01-0.100.050.200.16 0.03-0.040.01-0.060.060.170.13 0.02-0.08-0.05-0.060.060.170.12 0.16-0.080.01-0.050.100.310.17 -0.00-0.13-0.12-0.12-0.030.190.12 -0.06-0.20-0.00-0.29-0.120.030.04 -0.10-0.14-0.05-0.21-0.100.100.06 -0.09-0.13-0.07-0.20-0.040.050.02 0.01-0.13-0.01-0.100.050.090.06 0.02-0.15-0.03-0.16-0.060.230.08 0.050.010.04-0.060.090.150.09 -0.05-0.09-0.05-0.11-0.010.050.05 -0.020.000.04-0.010.040.050.03 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Verb. conf. Sem. entropy N.P(seq.) P(true) Sem. conf. P(seq.) Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.5) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 ROUGE-L (=0.7) · F1=0.03 ROUGE-L (=0.8) · F1=0.03 ROUGE-L (=0.9) · F1=0.03 Emb. sim. (=0.8) · F1=0.03 Emb. sim. (=0.9) · F1=0.03 BERTScore (=0.9) · F1=0.06 ROUGE-L (=0.5) · F1=0.09 Token F1 (=0.4) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.7) · F1=0.12 Token F1 (=0.3) · F1=0.15 ROUGE-L (=0.4) · F1=0.15 ROUGE-L (=0.3) · F1=0.32 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.2) · F1=0.38 ROUGE-L (=0.2) · F1=0.55 Emb. sim. (=0.5) · F1=0.55 Token F1 (=0.1) · F1=0.65 Emb. sim. (=0.4) · F1=0.77 BERTScore (=0.7) · F1=0.78 ROUGE-L (=0.1) · F1=0.80 Emb. sim. (=0.1) · F1=0.80 BERTScore (=0.8) · F1=0.83 Emb. sim. (=0.2) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.89 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) 0.090.200.16-0.060.110.310.38 0.09-0.060.15-0.07-0.350.25-0.18 -0.640.08-0.46-0.18-0.580.300.41 -0.640.08-0.46-0.18-0.580.300.41 -0.640.08-0.46-0.18-0.580.300.41 -0.640.08-0.46-0.18-0.580.300.41 -0.640.08-0.46-0.18-0.580.300.41 -0.640.08-0.46-0.18-0.580.300.41 -0.270.14-0.15-0.12-0.240.310.40 -0.150.08-0.05-0.10-0.280.290.21 -0.16-0.01-0.13-0.19-0.180.170.12 -0.04-0.19-0.03-0.17-0.090.11-0.04 -0.16-0.10-0.02-0.05-0.150.180.24 -0.16-0.13-0.11-0.17-0.190.160.06 -0.16-0.06-0.07-0.12-0.120.200.06 -0.06-0.020.05-0.10-0.110.230.15 -0.07-0.08-0.07-0.13-0.060.020.00 -0.08-0.20-0.13-0.07-0.190.11-0.04 -0.05-0.050.05-0.10-0.150.220.10 -0.110.020.04-0.10-0.020.040.07 -0.07-0.090.02-0.07-0.130.190.08 -0.07-0.060.01-0.08-0.080.01-0.01 0.190.19-0.600.29-0.120.150.02 -0.06-0.120.04-0.05-0.130.200.03 -0.05-0.13-0.17-0.07-0.140.000.03 -0.08-0.18-0.15-0.07-0.180.090.00 -0.01-0.10-0.17-0.12-0.12-0.040.05 -0.05-0.07-0.06-0.06-0.080.000.02 -0.03-0.02-0.010.01-0.080.070.03 -0.01-0.040.000.01-0.060.020.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference N.P(seq.) P(true) Verb. conf. Sem. conf. Sem. entropy P(seq.) Ent. Uncertainty metric ROUGE-L (=0.4) · F1=0.04 BLEU (=0.1) · F1=0.09 BERTScore (=0.9) · F1=0.09 Emb. sim. (=0.7) · F1=0.12 Token F1 (=0.3) · F1=0.24 ROUGE-L (=0.3) · F1=0.33 Token F1 (=0.2) · F1=0.44 Emb. sim. (=0.6) · F1=0.50 Emb. sim. (=0.5) · F1=0.52 ROUGE-L (=0.2) · F1=0.55 Token F1 (=0.1) · F1=0.60 Emb. sim. (=0.1) · F1=0.66 BERTScore (=0.8) · F1=0.68 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 ROUGE-L (=0.1) · F1=0.72 LLM judge (Llama 8B) · F1=0.78 Emb. sim. (=0.3) · F1=0.79 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion 1746523 1537624 1637524 2437516 1647523 1657423 2547613 1437625 1467523 1647523 2457613 2517634 2537614 1427635 1527436 2537614 1237546 1327645 3125467 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Llama 3B, AmbigQA (unambiguous) Verb. conf. Sem. entropy N.P(seq.) P(true) Sem. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.5) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 ROUGE-L (=0.7) · F1=0.03 ROUGE-L (=0.8) · F1=0.03 ROUGE-L (=0.9) · F1=0.03 Emb. sim. (=0.8) · F1=0.03 Emb. sim. (=0.9) · F1=0.03 BERTScore (=0.9) · F1=0.06 ROUGE-L (=0.5) · F1=0.09 Token F1 (=0.4) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.7) · F1=0.12 Token F1 (=0.3) · F1=0.15 ROUGE-L (=0.4) · F1=0.15 ROUGE-L (=0.3) · F1=0.32 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.2) · F1=0.38 ROUGE-L (=0.2) · F1=0.55 Emb. sim. (=0.5) · F1=0.55 Token F1 (=0.1) · F1=0.65 Emb. sim. (=0.4) · F1=0.77 BERTScore (=0.7) · F1=0.78 ROUGE-L (=0.1) · F1=0.80 Emb. sim. (=0.1) · F1=0.80 BERTScore (=0.8) · F1=0.83 Emb. sim. (=0.2) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.89 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 4357621 2435716 6354721 6354721 6354721 6354721 6354721 6354721 6354721 5246713 4257613 2637415 4635712 3546712 5236714 4526713 1347526 2643715 3526714 4217536 3526714 1425637 2371645 3624715 1564723 2653714 1475632 1435726 1435726 1523647 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Llama 8B, AmbigQA (unambiguous) Figure 10: Top row shows AUROC values, middle row shows differences between oracle and system AUROC values and bottom row rankings for various uncertainty quantifiers (ordered from left to right in a descending order of performance) and correctness functions (ordered from top to bottom in an increasing order of quality). 29 N.P(seq.) Sem. conf. P(true) P(seq.) Sem. entropy Verb. conf. Ent. Uncertainty metric Token F1 (=0.3) · F1=0.12 BLEU (=0.1) · F1=0.12 Emb. sim. (=0.1) · F1=0.31 Token F1 (=0.2) · F1=0.32 Emb. sim. (=0.7) · F1=0.32 Emb. sim. (=0.6) · F1=0.40 Emb. sim. (=0.2) · F1=0.41 BERTScore (=0.8) · F1=0.41 Token F1 (=0.1) · F1=0.50 ROUGE-L (=0.1) · F1=0.53 Emb. sim. (=0.3) · F1=0.53 ROUGE-L (=0.2) · F1=0.55 LLM judge (Llama 8B) · F1=0.64 Emb. sim. (=0.4) · F1=0.65 LLM judge (GPT) · F1=0.74 Emb. sim. (=0.5) · F1=0.74 Human labels · F1=1.00 Correctness criterion 0.970.320.530.950.420.640.49 0.660.120.610.670.420.640.49 0.480.580.460.480.470.580.51 0.880.450.670.880.600.640.67 0.930.470.370.930.420.750.49 0.900.480.440.850.420.650.49 0.590.570.540.540.510.520.51 0.660.530.590.660.490.450.51 0.840.470.640.800.500.530.54 0.750.460.670.770.490.550.54 0.650.520.610.630.520.510.51 0.900.480.700.910.590.650.58 0.760.580.750.670.600.470.56 0.700.560.610.660.500.490.52 0.640.700.650.640.620.600.55 0.780.560.580.760.460.540.49 0.660.640.630.610.590.570.53 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC P(true) Sem. conf. Sem. entropy Verb. conf. N.P(seq.) P(seq.) Ent. Uncertainty metric Emb. sim. (=0.7) · F1=0.04 Token F1 (=0.2) · F1=0.09 ROUGE-L (=0.3) · F1=0.13 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.34 Token F1 (=0.1) · F1=0.39 ROUGE-L (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.64 Emb. sim. (=0.1) · F1=0.65 Emb. sim. (=0.2) · F1=0.70 Emb. sim. (=0.4) · F1=0.70 BERTScore (=0.8) · F1=0.70 Emb. sim. (=0.3) · F1=0.75 LLM judge (Llama 8B) · F1=0.86 LLM judge (GPT) · F1=0.90 Human labels · F1=1.00 Correctness criterion 0.790.830.430.690.370.310.44 0.640.710.530.690.800.740.44 0.690.620.930.690.950.970.81 0.720.550.710.740.880.880.83 0.770.680.680.650.640.580.56 0.730.570.720.690.880.890.70 0.680.500.660.730.840.830.65 0.760.710.790.680.760.690.63 0.670.700.690.590.710.640.57 0.610.730.690.560.650.570.55 0.770.760.760.650.730.680.61 0.710.660.650.600.730.680.55 0.700.720.730.610.690.600.58 0.830.740.720.710.760.650.62 0.830.790.790.660.700.610.60 0.840.810.760.700.680.580.58 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Sem. conf. P(true) Sem. entropy N.P(seq.) P(seq.) Verb. conf. Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.3) · F1=0.00 Token F1 (=0.4) · F1=0.00 BLEU (=0.1) · F1=0.00 ROUGE-L (=0.3) · F1=0.00 ROUGE-L (=0.1) · F1=0.08 Token F1 (=0.2) · F1=0.12 ROUGE-L (=0.2) · F1=0.12 Emb. sim. (=0.6) · F1=0.14 Emb. sim. (=0.7) · F1=0.14 Token F1 (=0.1) · F1=0.15 Emb. sim. (=0.3) · F1=0.21 BERTScore (=0.8) · F1=0.22 Emb. sim. (=0.1) · F1=0.25 Emb. sim. (=0.2) · F1=0.27 Emb. sim. (=0.5) · F1=0.27 Emb. sim. (=0.4) · F1=0.29 LLM judge (Llama 8B) · F1=0.76 LLM judge (GPT) · F1=0.76 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.570.36-0.180.370.42-0.450.00 -0.570.36-0.180.370.42-0.450.00 -0.570.36-0.180.370.42-0.450.00 -0.570.36-0.180.370.42-0.450.00 -0.31-0.00-0.18-0.050.120.030.00 -0.380.18-0.180.220.36-0.080.00 -0.330.21-0.180.180.35-0.030.00 0.210.14-0.180.30-0.020.490.00 0.210.14-0.180.30-0.020.490.00 -0.28-0.02-0.11-0.040.080.020.00 -0.13-0.11-0.13-0.010.040.050.00 -0.31-0.03-0.16-0.060.11-0.150.00 -0.350.12-0.25-0.120.030.170.00 -0.20-0.02-0.13-0.090.00-0.040.00 -0.090.11-0.130.170.120.050.00 -0.19-0.01-0.140.060.110.010.00 -0.040.03-0.04-0.010.02-0.090.00 -0.110.12-0.04-0.030.030.010.00 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference P(true) Sem. entropy Sem. conf. N.P(seq.) Verb. conf. P(seq.) Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.3) · F1=0.04 Token F1 (=0.4) · F1=0.04 BLEU (=0.1) · F1=0.04 Emb. sim. (=0.7) · F1=0.04 ROUGE-L (=0.3) · F1=0.08 ROUGE-L (=0.2) · F1=0.16 Token F1 (=0.2) · F1=0.23 Emb. sim. (=0.6) · F1=0.30 Token F1 (=0.1) · F1=0.39 ROUGE-L (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.55 Emb. sim. (=0.4) · F1=0.62 Emb. sim. (=0.1) · F1=0.63 BERTScore (=0.8) · F1=0.63 Emb. sim. (=0.3) · F1=0.65 Emb. sim. (=0.2) · F1=0.69 LLM judge (Llama 8B) · F1=0.78 LLM judge (GPT) · F1=0.89 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.270.240.110.320.020.360.42 -0.270.240.110.320.020.360.42 -0.270.240.110.320.020.360.42 0.120.140.11-0.280.02-0.43-0.14 -0.070.270.120.290.020.340.40 -0.17-0.03-0.140.080.020.280.13 -0.060.03-0.050.170.130.310.13 -0.11-0.07-0.04-0.11-0.05-0.12-0.07 -0.04-0.08-0.050.030.050.110.08 -0.06-0.04-0.06-0.010.000.100.10 -0.00-0.03-0.03-0.01-0.01-0.07-0.01 0.020.02-0.01-0.01-0.01-0.06-0.00 -0.210.00-0.05-0.18-0.07-0.130.00 -0.07-0.11-0.10-0.08-0.02-0.010.01 -0.12-0.06-0.11-0.06-0.06-0.05-0.03 -0.17-0.09-0.11-0.16-0.05-0.15-0.05 -0.03-0.05-0.04-0.010.060.030.00 0.05-0.02-0.000.070.030.060.02 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Sem. conf. P(true) Sem. entropy N.P(seq.) P(seq.) Verb. conf. Ent. Uncertainty metric Token F1 (=0.3) · F1=0.00 Token F1 (=0.4) · F1=0.00 BLEU (=0.1) · F1=0.00 ROUGE-L (=0.3) · F1=0.00 ROUGE-L (=0.1) · F1=0.08 Token F1 (=0.2) · F1=0.12 ROUGE-L (=0.2) · F1=0.12 Emb. sim. (=0.6) · F1=0.14 Emb. sim. (=0.7) · F1=0.14 Token F1 (=0.1) · F1=0.15 Emb. sim. (=0.3) · F1=0.21 BERTScore (=0.8) · F1=0.22 Emb. sim. (=0.1) · F1=0.25 Emb. sim. (=0.2) · F1=0.27 Emb. sim. (=0.5) · F1=0.27 Emb. sim. (=0.4) · F1=0.29 LLM judge (Llama 8B) · F1=0.76 LLM judge (GPT) · F1=0.76 Human labels · F1=1.00 Correctness criterion 6251374 6251374 6251374 6251374 7263145 7362154 7263154 2473615 2473615 7263145 2571436 6253174 6175324 4163275 4271356 5371246 1243576 2143567 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Qwen 0.5B, AmbigQA (ambiguous) P(true) Sem. entropy Sem. conf. N.P(seq.) Verb. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.3) · F1=0.04 Token F1 (=0.4) · F1=0.04 BLEU (=0.1) · F1=0.04 Emb. sim. (=0.7) · F1=0.04 ROUGE-L (=0.3) · F1=0.08 ROUGE-L (=0.2) · F1=0.16 Token F1 (=0.2) · F1=0.23 Emb. sim. (=0.6) · F1=0.30 Token F1 (=0.1) · F1=0.39 ROUGE-L (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.55 Emb. sim. (=0.4) · F1=0.62 Emb. sim. (=0.1) · F1=0.63 BERTScore (=0.8) · F1=0.63 Emb. sim. (=0.3) · F1=0.65 Emb. sim. (=0.2) · F1=0.69 LLM judge (Llama 8B) · F1=0.78 LLM judge (GPT) · F1=0.89 Human labels · F1=1.00 Correctness criterion 7453621 7453621 7453621 1236475 7453621 7562413 6572314 4315276 5763214 5674312 1452376 1234576 6127354 2765134 4163257 5236174 2653147 1652347 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Qwen 7B, AmbigQA (ambiguous) Figure 11: Top row shows AUROC values, middle row shows differences between oracle and system AUROC values and bottom row rankings for various uncertainty quantifiers (ordered from left to right in a descending order of performance) and correctness functions (ordered from top to bottom in an increasing order of quality). 30 Verb. conf. P(true) P(seq.) N.P(seq.) Sem. conf. Sem. entropy Ent. Uncertainty metric BLEU (=0.1) · F1=0.03 ROUGE-L (=0.5) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 Emb. sim. (=0.8) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.07 Emb. sim. (=0.7) · F1=0.07 Token F1 (=0.3) · F1=0.10 ROUGE-L (=0.4) · F1=0.16 ROUGE-L (=0.3) · F1=0.22 Token F1 (=0.2) · F1=0.24 ROUGE-L (=0.2) · F1=0.37 Emb. sim. (=0.6) · F1=0.42 Emb. sim. (=0.5) · F1=0.50 Token F1 (=0.1) · F1=0.56 ROUGE-L (=0.1) · F1=0.68 Emb. sim. (=0.2) · F1=0.74 Emb. sim. (=0.4) · F1=0.74 Emb. sim. (=0.1) · F1=0.75 Emb. sim. (=0.3) · F1=0.77 BERTScore (=0.8) · F1=0.79 LLM judge (Llama 8B) · F1=0.81 LLM judge (GPT) · F1=0.93 Human labels · F1=1.00 Correctness criterion 0.680.120.830.840.430.690.69 0.170.880.700.070.220.610.42 0.170.880.700.070.220.610.42 0.170.880.700.070.220.610.42 0.880.130.900.800.750.940.96 0.530.510.800.430.480.780.69 0.170.840.390.340.480.750.42 0.420.430.830.650.340.600.56 0.450.570.910.730.440.700.75 0.520.490.920.810.360.680.70 0.560.520.910.780.350.670.67 0.660.540.930.830.400.620.72 0.670.610.710.660.540.580.57 0.640.570.750.740.500.540.55 0.620.530.800.740.440.520.59 0.640.600.860.770.440.550.61 0.640.580.600.590.470.530.52 0.680.610.680.680.530.550.59 0.840.630.540.420.240.460.58 0.690.610.600.590.510.610.54 0.610.600.730.620.390.490.53 0.760.680.660.680.500.560.53 0.680.700.680.670.540.580.56 0.700.690.660.660.560.560.55 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC P(true) Verb. conf. N.P(seq.) Sem. entropy Sem. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.3) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 BLEU (=0.1) · F1=0.06 Emb. sim. (=0.7) · F1=0.06 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.26 Emb. sim. (=0.6) · F1=0.33 Token F1 (=0.1) · F1=0.50 Emb. sim. (=0.5) · F1=0.58 ROUGE-L (=0.1) · F1=0.69 Emb. sim. (=0.4) · F1=0.76 BERTScore (=0.8) · F1=0.76 Emb. sim. (=0.2) · F1=0.77 Emb. sim. (=0.1) · F1=0.79 Emb. sim. (=0.3) · F1=0.79 LLM judge (Llama 8B) · F1=0.87 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 0.590.830.920.610.220.750.44 0.080.280.820.820.690.330.44 0.280.830.930.890.690.960.98 0.330.550.870.720.450.540.44 0.320.550.680.580.690.600.44 0.540.840.960.830.570.930.85 0.610.770.800.610.600.870.66 0.640.800.850.700.560.920.75 0.650.610.680.460.630.650.52 0.570.660.670.540.530.720.55 0.650.670.710.580.600.640.52 0.620.620.700.600.540.790.60 0.770.660.720.580.560.680.56 0.640.580.550.500.510.690.54 0.680.650.560.560.520.680.57 0.690.680.580.560.310.770.56 0.750.660.670.610.580.640.57 0.810.780.720.610.620.690.52 0.810.770.710.670.650.690.54 0.790.750.720.680.670.640.52 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Verb. conf. P(true) P(seq.) N.P(seq.) Sem. conf. Sem. entropy Ent. Uncertainty metric (sorted by human-label AUROC) BLEU (=0.1) · F1=0.03 ROUGE-L (=0.5) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 Emb. sim. (=0.8) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.07 Emb. sim. (=0.7) · F1=0.07 Token F1 (=0.3) · F1=0.10 ROUGE-L (=0.4) · F1=0.16 ROUGE-L (=0.3) · F1=0.22 Token F1 (=0.2) · F1=0.24 ROUGE-L (=0.2) · F1=0.37 Emb. sim. (=0.6) · F1=0.42 Emb. sim. (=0.5) · F1=0.50 Token F1 (=0.1) · F1=0.56 ROUGE-L (=0.1) · F1=0.68 Emb. sim. (=0.2) · F1=0.74 Emb. sim. (=0.4) · F1=0.74 Emb. sim. (=0.1) · F1=0.75 Emb. sim. (=0.3) · F1=0.77 BERTScore (=0.8) · F1=0.79 LLM judge (Llama 8B) · F1=0.81 LLM judge (GPT) · F1=0.93 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.02-0.570.160.18-0.140.140.14 -0.530.180.03-0.59-0.350.05-0.13 -0.530.180.03-0.59-0.350.05-0.13 -0.530.180.03-0.59-0.350.05-0.13 0.18-0.560.240.140.180.380.41 -0.17-0.190.14-0.22-0.080.220.14 -0.530.15-0.27-0.32-0.080.20-0.13 -0.28-0.260.17-0.01-0.230.040.01 -0.25-0.120.250.07-0.120.140.20 -0.19-0.200.260.15-0.200.130.15 -0.14-0.170.250.12-0.210.110.12 -0.05-0.150.270.17-0.160.070.17 -0.04-0.090.050.01-0.020.020.02 -0.07-0.120.080.08-0.06-0.01-0.00 -0.08-0.160.130.08-0.12-0.040.04 -0.06-0.100.200.11-0.12-0.010.06 -0.06-0.11-0.06-0.07-0.09-0.03-0.03 -0.02-0.090.020.02-0.03-0.010.04 0.14-0.07-0.12-0.24-0.32-0.090.03 -0.01-0.08-0.06-0.07-0.050.05-0.01 -0.09-0.090.07-0.04-0.17-0.07-0.02 0.06-0.020.000.02-0.060.00-0.02 -0.020.010.020.01-0.020.030.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference P(true) Verb. conf. N.P(seq.) Sem. entropy Sem. conf. P(seq.) Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.3) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 BLEU (=0.1) · F1=0.06 Emb. sim. (=0.7) · F1=0.06 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.26 Emb. sim. (=0.6) · F1=0.33 Token F1 (=0.1) · F1=0.50 Emb. sim. (=0.5) · F1=0.58 ROUGE-L (=0.1) · F1=0.69 Emb. sim. (=0.4) · F1=0.76 BERTScore (=0.8) · F1=0.76 Emb. sim. (=0.2) · F1=0.77 Emb. sim. (=0.1) · F1=0.79 Emb. sim. (=0.3) · F1=0.79 LLM judge (Llama 8B) · F1=0.87 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.200.080.20-0.07-0.450.11-0.08 -0.71-0.470.100.140.02-0.30-0.08 -0.510.080.210.220.020.320.46 -0.46-0.190.150.04-0.22-0.10-0.08 -0.47-0.19-0.04-0.100.02-0.03-0.08 -0.250.090.240.16-0.090.290.32 -0.180.030.08-0.07-0.070.230.13 -0.160.050.130.02-0.100.280.23 -0.15-0.14-0.04-0.21-0.030.010.00 -0.23-0.09-0.05-0.14-0.130.080.03 -0.15-0.07-0.01-0.10-0.070.000.00 -0.18-0.12-0.02-0.07-0.120.150.08 -0.03-0.08-0.00-0.09-0.100.040.04 -0.15-0.17-0.17-0.18-0.160.060.02 -0.12-0.10-0.16-0.12-0.150.040.05 -0.10-0.07-0.14-0.12-0.360.130.03 -0.04-0.09-0.05-0.06-0.090.000.05 0.020.03-0.00-0.07-0.050.05-0.01 0.020.03-0.01-0.01-0.020.050.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Verb. conf. P(true) P(seq.) N.P(seq.) Sem. conf. Sem. entropy Ent. Uncertainty metric BLEU (=0.1) · F1=0.03 ROUGE-L (=0.5) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 Emb. sim. (=0.8) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.07 Emb. sim. (=0.7) · F1=0.07 Token F1 (=0.3) · F1=0.10 ROUGE-L (=0.4) · F1=0.16 ROUGE-L (=0.3) · F1=0.22 Token F1 (=0.2) · F1=0.24 ROUGE-L (=0.2) · F1=0.37 Emb. sim. (=0.6) · F1=0.42 Emb. sim. (=0.5) · F1=0.50 Token F1 (=0.1) · F1=0.56 ROUGE-L (=0.1) · F1=0.68 Emb. sim. (=0.2) · F1=0.74 Emb. sim. (=0.4) · F1=0.74 Emb. sim. (=0.1) · F1=0.75 Emb. sim. (=0.3) · F1=0.77 BERTScore (=0.8) · F1=0.79 LLM judge (Llama 8B) · F1=0.81 LLM judge (GPT) · F1=0.93 Human labels · F1=1.00 Correctness criterion 5721643 6127534 6127534 6127534 4735621 4517623 7156324 6512734 6513742 5612743 5612743 4612753 2413756 3412765 3512764 3512764 1423756 3412765 1246753 1245736 3412765 1342756 3124756 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Llama 3B, AmbigQA (ambiguous) P(true) Verb. conf. N.P(seq.) Sem. entropy Sem. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.3) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 BLEU (=0.1) · F1=0.06 Emb. sim. (=0.7) · F1=0.06 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.26 Emb. sim. (=0.6) · F1=0.33 Token F1 (=0.1) · F1=0.50 Emb. sim. (=0.5) · F1=0.58 ROUGE-L (=0.1) · F1=0.69 Emb. sim. (=0.4) · F1=0.76 BERTScore (=0.8) · F1=0.76 Emb. sim. (=0.2) · F1=0.77 Emb. sim. (=0.1) · F1=0.79 Emb. sim. (=0.3) · F1=0.79 LLM judge (Llama 8B) · F1=0.87 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 5214736 7621354 7534621 7312546 7524136 7415623 5326714 6325714 2517436 4326715 3216547 4325716 1425736 2347615 2356714 2345716 1325647 1236547 1235647 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Llama 8B, AmbigQA (ambiguous) Figure 12: Top row shows AUROC values, middle row shows differences between oracle and system AUROC values and bottom row rankings for various uncertainty quantifiers (ordered from left to right in a descending order of performance) and correctness functions (ordered from top to bottom in an increasing order of quality). 31 Verb. conf. P(seq.) Sem. entropy N.P(seq.) P(true) Ent. Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.10 BERTScore (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.17 Emb. sim. (=0.8) · F1=0.17 ROUGE-L (=0.4) · F1=0.28 Token F1 (=0.3) · F1=0.30 Emb. sim. (=0.7) · F1=0.34 ROUGE-L (=0.3) · F1=0.44 Token F1 (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.55 ROUGE-L (=0.2) · F1=0.67 Token F1 (=0.1) · F1=0.72 Emb. sim. (=0.5) · F1=0.76 Emb. sim. (=0.1) · F1=0.85 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 BERTScore (=0.8) · F1=0.89 Emb. sim. (=0.3) · F1=0.90 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.97 Human labels · F1=1.00 Correctness criterion 0.740.940.840.380.920.960.69 0.740.940.840.380.920.960.69 0.740.940.840.380.920.960.69 0.740.940.840.380.920.960.69 0.740.940.840.380.920.960.69 0.740.960.880.670.850.960.69 0.740.830.690.210.740.860.69 0.740.830.690.210.740.860.69 0.740.830.690.210.740.860.69 0.750.980.890.790.760.980.69 0.590.790.790.670.550.750.69 0.750.880.770.460.750.900.69 0.750.880.780.550.680.920.69 0.750.830.740.520.650.890.69 0.750.860.780.550.680.900.61 0.690.840.780.580.660.830.64 0.690.840.720.570.640.810.64 0.690.770.770.580.590.790.64 0.740.870.820.730.610.880.68 0.690.810.720.700.610.770.69 0.740.860.760.730.590.850.65 0.700.810.710.660.600.790.62 0.670.780.660.670.620.740.59 0.700.840.700.760.550.730.58 0.690.780.680.700.630.690.57 0.730.790.650.680.730.750.66 0.830.810.930.880.960.700.66 0.750.820.660.710.750.700.64 0.860.840.750.790.780.710.61 0.780.880.680.770.650.730.55 0.810.910.760.690.670.700.48 0.850.810.710.690.770.690.61 0.860.790.770.730.740.720.71 0.930.790.810.790.710.730.72 0.950.820.790.760.750.730.68 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Verb. conf. P(seq.) N.P(seq.) Sem. entropy P(true) Ent. Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.940.380.840.920.960.69 0.740.960.670.880.850.960.69 0.740.830.210.690.740.860.69 0.740.830.210.690.740.860.69 0.740.830.210.690.740.860.69 0.750.980.790.890.760.980.69 0.590.790.670.790.550.750.69 0.750.880.460.770.750.900.69 0.750.880.550.780.680.920.69 0.750.830.520.740.650.890.69 0.750.860.550.780.680.900.61 0.690.840.580.780.660.830.64 0.690.840.570.720.640.810.64 0.690.770.580.770.590.790.64 0.740.870.730.820.610.880.68 0.690.810.700.720.610.770.69 0.740.860.730.760.590.850.65 0.700.810.660.710.600.790.62 0.670.780.670.660.620.740.59 0.700.840.760.700.550.730.58 0.690.780.700.680.630.690.57 0.730.790.680.650.730.750.66 0.830.810.880.930.960.700.66 0.750.820.710.660.750.700.64 0.860.840.790.750.780.710.61 0.810.910.690.760.670.700.48 0.780.880.770.680.650.730.55 0.850.810.690.710.770.690.61 0.860.790.730.770.740.720.71 0.930.790.790.810.710.730.72 0.920.840.780.760.740.740.69 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC Verb. conf. P(seq.) Sem. entropy N.P(seq.) P(true) Ent. Sem. conf. Uncertainty metric (sorted by human-label AUROC) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.10 BERTScore (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.17 Emb. sim. (=0.8) · F1=0.17 ROUGE-L (=0.4) · F1=0.28 Token F1 (=0.3) · F1=0.30 Emb. sim. (=0.7) · F1=0.34 ROUGE-L (=0.3) · F1=0.44 Token F1 (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.55 ROUGE-L (=0.2) · F1=0.67 Token F1 (=0.1) · F1=0.72 Emb. sim. (=0.5) · F1=0.76 Emb. sim. (=0.1) · F1=0.85 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 BERTScore (=0.8) · F1=0.89 Emb. sim. (=0.3) · F1=0.90 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.97 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.210.120.05-0.380.180.230.00 -0.210.120.05-0.380.180.230.00 -0.210.120.05-0.380.180.230.00 -0.210.120.05-0.380.180.230.00 -0.210.120.05-0.380.180.230.00 -0.200.140.09-0.090.100.240.01 -0.200.01-0.10-0.55-0.010.140.01 -0.200.01-0.10-0.55-0.010.140.01 -0.200.01-0.10-0.55-0.010.140.01 -0.200.160.100.020.010.250.01 -0.36-0.03-0.00-0.10-0.200.030.01 -0.200.06-0.02-0.300.000.180.01 -0.200.06-0.01-0.21-0.070.190.01 -0.200.00-0.05-0.24-0.100.160.01 -0.200.04-0.01-0.21-0.070.18-0.07 -0.260.02-0.01-0.18-0.090.10-0.04 -0.260.01-0.07-0.19-0.110.08-0.04 -0.26-0.05-0.02-0.18-0.160.06-0.05 -0.210.050.03-0.03-0.140.150.00 -0.26-0.01-0.06-0.06-0.140.050.00 -0.210.04-0.03-0.04-0.160.13-0.04 -0.25-0.02-0.08-0.10-0.150.06-0.06 -0.28-0.04-0.13-0.09-0.130.01-0.09 -0.250.02-0.090.00-0.190.00-0.11 -0.26-0.04-0.11-0.06-0.12-0.03-0.11 -0.22-0.03-0.14-0.09-0.010.02-0.02 -0.12-0.010.140.120.21-0.02-0.03 -0.200.00-0.13-0.06-0.00-0.03-0.04 -0.090.01-0.040.030.03-0.01-0.07 -0.170.06-0.110.01-0.100.01-0.13 -0.140.09-0.03-0.08-0.08-0.03-0.20 -0.10-0.01-0.08-0.070.02-0.04-0.07 -0.09-0.03-0.02-0.03-0.01-0.000.03 -0.02-0.030.020.03-0.040.010.04 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Verb. conf. P(seq.) N.P(seq.) Sem. entropy P(true) Ent. Sem. conf. Uncertainty metric (sorted by human-label AUROC) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.10-0.390.070.180.220.00 -0.180.12-0.100.110.110.230.00 -0.18-0.02-0.56-0.07-0.000.130.00 -0.18-0.02-0.56-0.07-0.000.130.00 -0.18-0.02-0.56-0.07-0.000.130.00 -0.170.140.010.130.020.240.00 -0.33-0.05-0.110.02-0.190.020.00 -0.170.04-0.310.000.010.170.00 -0.170.04-0.220.02-0.060.180.01 -0.17-0.02-0.26-0.02-0.090.150.01 -0.170.02-0.220.02-0.070.17-0.07 -0.23-0.00-0.200.02-0.090.09-0.05 -0.23-0.01-0.20-0.05-0.100.07-0.05 -0.23-0.07-0.200.00-0.150.05-0.05 -0.180.02-0.040.06-0.130.14-0.00 -0.24-0.03-0.08-0.04-0.130.040.00 -0.190.02-0.05-0.00-0.150.12-0.04 -0.22-0.04-0.12-0.05-0.140.06-0.06 -0.25-0.06-0.11-0.10-0.120.00-0.09 -0.230.00-0.01-0.06-0.19-0.01-0.11 -0.23-0.07-0.07-0.08-0.12-0.04-0.12 -0.19-0.05-0.10-0.11-0.010.01-0.03 -0.09-0.030.110.160.22-0.03-0.03 -0.17-0.02-0.07-0.110.01-0.04-0.05 -0.06-0.010.01-0.010.04-0.02-0.07 -0.110.07-0.09-0.00-0.07-0.04-0.21 -0.140.04-0.01-0.09-0.09-0.00-0.14 -0.07-0.03-0.09-0.060.03-0.05-0.07 -0.07-0.06-0.050.00-0.00-0.010.03 0.00-0.050.010.04-0.03-0.000.04 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference Verb. conf. P(seq.) Sem. entropy N.P(seq.) P(true) Ent. Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.10 BERTScore (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.17 Emb. sim. (=0.8) · F1=0.17 ROUGE-L (=0.4) · F1=0.28 Token F1 (=0.3) · F1=0.30 Emb. sim. (=0.7) · F1=0.34 ROUGE-L (=0.3) · F1=0.44 Token F1 (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.55 ROUGE-L (=0.2) · F1=0.67 Token F1 (=0.1) · F1=0.72 Emb. sim. (=0.5) · F1=0.76 Emb. sim. (=0.1) · F1=0.85 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 BERTScore (=0.8) · F1=0.89 Emb. sim. (=0.3) · F1=0.90 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.97 Human labels · F1=1.00 Correctness criterion 5247316 5247316 5247316 5247316 5247316 5237416 3257416 3257416 3257416 6134527 6125734 5237416 4237615 3247615 4237516 4137526 4137526 4237615 4235716 6134725 4135726 4135726 3154627 5142736 4152637 4175326 4523167 3164257 1253467 2153647 2135647 1245367 1235467 1324756 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Llama 8B, TriviaQA, New annotator Verb. conf. P(seq.) N.P(seq.) Sem. entropy P(true) Ent. Sem. conf. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 5274316 5274316 5274316 5274316 5274316 5273416 3275416 3275416 3275416 6143527 6152734 5273416 4273615 3274615 4273516 4173526 4173526 4273615 4253716 6143725 4153726 4153726 3145627 5124736 4125637 4157326 4532167 3146257 1235467 2153647 2135647 1254367 1253467 1342756 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Llama 8B, TriviaQA, Original annotator Figure 13: Left column shows results given manual labels obtained from new annotator, and right column shows results from original annotator. 32 P(true) Verb. conf. N.P(seq.) Sem. conf. P(seq.) Sem. entropy Ent. Uncertainty metric BLEU (=0.2) · F1=0.00 Token F1 (=0.3) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 Emb. sim. (=0.7) · F1=0.03 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.31 Token F1 (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.57 ROUGE-L (=0.1) · F1=0.67 Emb. sim. (=0.4) · F1=0.75 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.1) · F1=0.80 Emb. sim. (=0.3) · F1=0.80 Emb. sim. (=0.2) · F1=0.83 LLM judge (Llama 8B) · F1=0.88 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 0.080.280.820.690.330.820.44 0.590.830.920.220.750.610.44 0.330.550.870.450.540.720.44 0.280.830.930.690.960.890.98 0.320.550.680.690.600.580.44 0.540.840.960.570.930.830.85 0.610.770.800.600.870.610.66 0.640.800.850.560.920.700.75 0.650.610.680.630.650.460.52 0.570.660.670.530.720.540.55 0.650.670.710.600.640.580.52 0.620.620.700.540.790.600.60 0.770.660.720.560.680.580.56 0.640.580.550.510.690.500.54 0.690.680.580.310.770.560.56 0.750.660.670.580.640.610.57 0.680.650.560.520.680.560.57 0.810.780.720.620.690.610.52 0.810.770.710.650.690.670.54 0.790.770.710.680.670.670.55 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC P(true) Verb. conf. N.P(seq.) Sem. entropy Sem. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.3) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 BLEU (=0.1) · F1=0.06 Emb. sim. (=0.7) · F1=0.06 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.26 Emb. sim. (=0.6) · F1=0.33 Token F1 (=0.1) · F1=0.50 Emb. sim. (=0.5) · F1=0.58 ROUGE-L (=0.1) · F1=0.69 Emb. sim. (=0.4) · F1=0.76 BERTScore (=0.8) · F1=0.76 Emb. sim. (=0.2) · F1=0.77 Emb. sim. (=0.1) · F1=0.79 Emb. sim. (=0.3) · F1=0.79 LLM judge (Llama 8B) · F1=0.87 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 0.590.830.920.610.220.750.44 0.080.280.820.820.690.330.44 0.280.830.930.890.690.960.98 0.330.550.870.720.450.540.44 0.320.550.680.580.690.600.44 0.540.840.960.830.570.930.85 0.610.770.800.610.600.870.66 0.640.800.850.700.560.920.75 0.650.610.680.460.630.650.52 0.570.660.670.540.530.720.55 0.650.670.710.580.600.640.52 0.620.620.700.600.540.790.60 0.770.660.720.580.560.680.56 0.640.580.550.500.510.690.54 0.680.650.560.560.520.680.57 0.690.680.580.560.310.770.56 0.750.660.670.610.580.640.57 0.810.780.720.610.620.690.52 0.810.770.710.670.650.690.54 0.790.750.720.680.670.640.52 AUROC values per correctness criterion 0.0 0.2 0.4 0.6 0.8 1.0 AUROC P(true) Verb. conf. N.P(seq.) Sem. conf. P(seq.) Sem. entropy Ent. Uncertainty metric (sorted by human-label AUROC) BLEU (=0.2) · F1=0.00 Token F1 (=0.3) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 Emb. sim. (=0.7) · F1=0.03 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.31 Token F1 (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.57 ROUGE-L (=0.1) · F1=0.67 Emb. sim. (=0.4) · F1=0.75 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.1) · F1=0.80 Emb. sim. (=0.3) · F1=0.80 Emb. sim. (=0.2) · F1=0.83 LLM judge (Llama 8B) · F1=0.88 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.71-0.490.110.01-0.340.15-0.10 -0.200.060.21-0.460.08-0.06-0.10 -0.46-0.220.17-0.23-0.130.05-0.10 -0.510.060.220.010.290.230.43 -0.47-0.22-0.020.01-0.07-0.09-0.10 -0.250.070.26-0.110.260.170.30 -0.180.000.09-0.080.20-0.060.11 -0.160.030.15-0.120.250.030.21 -0.14-0.16-0.03-0.05-0.02-0.20-0.02 -0.23-0.11-0.03-0.140.05-0.120.01 -0.15-0.100.01-0.08-0.03-0.09-0.02 -0.18-0.15-0.00-0.140.12-0.060.05 -0.03-0.110.01-0.120.01-0.080.02 -0.15-0.19-0.16-0.170.03-0.17-0.00 -0.10-0.09-0.12-0.370.10-0.110.01 -0.04-0.11-0.04-0.10-0.03-0.050.03 -0.12-0.12-0.14-0.160.01-0.110.02 0.020.010.01-0.060.02-0.06-0.03 0.020.000.01-0.030.020.00-0.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference P(true) Verb. conf. N.P(seq.) Sem. entropy Sem. conf. P(seq.) Ent. Uncertainty metric (sorted by human-label AUROC) Token F1 (=0.3) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 BLEU (=0.1) · F1=0.06 Emb. sim. (=0.7) · F1=0.06 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.26 Emb. sim. (=0.6) · F1=0.33 Token F1 (=0.1) · F1=0.50 Emb. sim. (=0.5) · F1=0.58 ROUGE-L (=0.1) · F1=0.69 Emb. sim. (=0.4) · F1=0.76 BERTScore (=0.8) · F1=0.76 Emb. sim. (=0.2) · F1=0.77 Emb. sim. (=0.1) · F1=0.79 Emb. sim. (=0.3) · F1=0.79 LLM judge (Llama 8B) · F1=0.87 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion (sorted by F1 vs. human labels) -0.200.080.20-0.07-0.450.11-0.08 -0.71-0.470.100.140.02-0.30-0.08 -0.510.080.210.220.020.320.46 -0.46-0.190.150.04-0.22-0.10-0.08 -0.47-0.19-0.04-0.100.02-0.03-0.08 -0.250.090.240.16-0.090.290.32 -0.180.030.08-0.07-0.070.230.13 -0.160.050.130.02-0.100.280.23 -0.15-0.14-0.04-0.21-0.030.010.00 -0.23-0.09-0.05-0.14-0.130.080.03 -0.15-0.07-0.01-0.10-0.070.000.00 -0.18-0.12-0.02-0.07-0.120.150.08 -0.03-0.08-0.00-0.09-0.100.040.04 -0.15-0.17-0.17-0.18-0.160.060.02 -0.12-0.10-0.16-0.12-0.150.040.05 -0.10-0.07-0.14-0.12-0.360.130.03 -0.04-0.09-0.05-0.06-0.090.000.05 0.020.03-0.00-0.07-0.050.05-0.01 0.020.03-0.01-0.01-0.020.050.01 AUROC difference vs. human labels 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 AUROC difference P(true) Verb. conf. N.P(seq.) Sem. conf. P(seq.) Sem. entropy Ent. Uncertainty metric BLEU (=0.2) · F1=0.00 Token F1 (=0.3) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 Emb. sim. (=0.7) · F1=0.03 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.27 Emb. sim. (=0.6) · F1=0.31 Token F1 (=0.1) · F1=0.48 Emb. sim. (=0.5) · F1=0.57 ROUGE-L (=0.1) · F1=0.67 Emb. sim. (=0.4) · F1=0.75 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.1) · F1=0.80 Emb. sim. (=0.3) · F1=0.80 Emb. sim. (=0.2) · F1=0.83 LLM judge (Llama 8B) · F1=0.88 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 7623514 5217346 7315426 7536241 7521346 7416253 5327164 6327154 2514376 4327165 3215467 4327156 1427356 2346175 2347156 1326457 2357164 1235467 1236457 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (a) Llama 3B, AmbigQA (ambiguous), New annotator P(true) Verb. conf. N.P(seq.) Sem. entropy Sem. conf. P(seq.) Ent. Uncertainty metric Token F1 (=0.3) · F1=0.03 BLEU (=0.2) · F1=0.03 ROUGE-L (=0.4) · F1=0.03 BLEU (=0.1) · F1=0.06 Emb. sim. (=0.7) · F1=0.06 ROUGE-L (=0.3) · F1=0.12 Token F1 (=0.2) · F1=0.14 ROUGE-L (=0.2) · F1=0.26 Emb. sim. (=0.6) · F1=0.33 Token F1 (=0.1) · F1=0.50 Emb. sim. (=0.5) · F1=0.58 ROUGE-L (=0.1) · F1=0.69 Emb. sim. (=0.4) · F1=0.76 BERTScore (=0.8) · F1=0.76 Emb. sim. (=0.2) · F1=0.77 Emb. sim. (=0.1) · F1=0.79 Emb. sim. (=0.3) · F1=0.79 LLM judge (Llama 8B) · F1=0.87 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 5214736 7621354 7534621 7312546 7524136 7415623 5326714 6325714 2517436 4326715 3216547 4325716 1425736 2347615 2356714 2345716 1325647 1236547 1235647 1234567 AUROC ranking per correctness criterion 1 2 3 4 5 6 7 AUROC rank (b) Llama 8B, AmbigQA (ambiguous), Original annotator Figure 14: Left column shows results given manual labels obtained from new annotator, and right column shows results from original annotator. 33 Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.936) Llm As Judge Correctness Os (F1=0.915) Rouge L 0.1 (F1=0.860) Embedding Similarity 0.4 (F1=0.830) Embedding Similarity 0.3 (F1=0.818) Embedding Similarity 0.2 (F1=0.809) Bert Score 0.8 (F1=0.759) Embedding Similarity 0.5 (F1=0.757) Token F1 0.1 (F1=0.751) Embedding Similarity 0.1 (F1=0.728) Rouge L 0.2 (F1=0.720) Embedding Similarity 0.6 (F1=0.596) Token F1 0.2 (F1=0.551) Rouge L 0.3 (F1=0.438) Rouge L 0.4 (F1=0.408) Token F1 0.3 (F1=0.347) Embedding Similarity 0.7 (F1=0.313) Embedding Similarity 0.8 (F1=0.304) Rouge L 0.5 (F1=0.254) Token F1 0.4 (F1=0.240) Rouge L 0.6 (F1=0.224) Token F1 0.6 (F1=0.223) Rouge L 0.8 (F1=0.215) Bleu 0.1 (F1=0.214) Exact Match Correctness (F1=0.207) Token F1 0.8 (F1=0.202) Rouge L 0.7 (F1=0.201) Bleu 0.2 (F1=0.196) Token F1 0.5 (F1=0.194) Rouge L 0.9 (F1=0.178) Embedding Similarity 0.9 (F1=0.171) Token F1 0.7 (F1=0.161) Bert Score 0.9 (F1=0.152) Token F1 0.9 (F1=0.152) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.936) Llm As Judge Correctness Os (F1=0.915) Rouge L 0.1 (F1=0.860) Embedding Similarity 0.4 (F1=0.830) Embedding Similarity 0.3 (F1=0.818) Embedding Similarity 0.2 (F1=0.809) Bert Score 0.8 (F1=0.759) Embedding Similarity 0.5 (F1=0.757) Token F1 0.1 (F1=0.751) Embedding Similarity 0.1 (F1=0.728) Rouge L 0.2 (F1=0.720) Embedding Similarity 0.6 (F1=0.596) Token F1 0.2 (F1=0.551) Rouge L 0.3 (F1=0.438) Rouge L 0.4 (F1=0.408) Token F1 0.3 (F1=0.347) Embedding Similarity 0.7 (F1=0.313) Embedding Similarity 0.8 (F1=0.304) Rouge L 0.5 (F1=0.254) Token F1 0.4 (F1=0.240) Rouge L 0.6 (F1=0.224) Token F1 0.6 (F1=0.223) Rouge L 0.8 (F1=0.215) Bleu 0.1 (F1=0.214) Exact Match Correctness (F1=0.207) Token F1 0.8 (F1=0.202) Rouge L 0.7 (F1=0.201) Bleu 0.2 (F1=0.196) Token F1 0.5 (F1=0.194) Rouge L 0.9 (F1=0.178) Embedding Similarity 0.9 (F1=0.171) Token F1 0.7 (F1=0.161) Bert Score 0.9 (F1=0.152) Token F1 0.9 (F1=0.152) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.961) Llm As Judge Correctness Os (F1=0.923) Embedding Similarity 0.2 (F1=0.902) Bert Score 0.8 (F1=0.871) Embedding Similarity 0.1 (F1=0.865) Rouge L 0.1 (F1=0.861) Embedding Similarity 0.4 (F1=0.846) Embedding Similarity 0.3 (F1=0.843) Token F1 0.1 (F1=0.728) Embedding Similarity 0.5 (F1=0.714) Rouge L 0.2 (F1=0.637) Rouge L 0.3 (F1=0.513) Token F1 0.2 (F1=0.500) Embedding Similarity 0.6 (F1=0.490) Rouge L 0.4 (F1=0.367) Embedding Similarity 0.7 (F1=0.315) Token F1 0.3 (F1=0.234) Embedding Similarity 0.9 (F1=0.222) Embedding Similarity 0.8 (F1=0.206) Rouge L 0.5 (F1=0.188) Rouge L 0.9 (F1=0.181) Token F1 0.4 (F1=0.172) Token F1 0.6 (F1=0.172) Bert Score 0.9 (F1=0.171) Rouge L 0.8 (F1=0.164) Rouge L 0.6 (F1=0.162) Token F1 0.7 (F1=0.159) Token F1 0.9 (F1=0.153) Rouge L 0.7 (F1=0.139) Token F1 0.8 (F1=0.134) Token F1 0.5 (F1=0.132) Bleu 0.2 (F1=0.130) Bleu 0.1 (F1=0.130) Exact Match Correctness (F1=0.125) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.961) Llm As Judge Correctness Os (F1=0.923) Embedding Similarity 0.2 (F1=0.902) Bert Score 0.8 (F1=0.871) Embedding Similarity 0.1 (F1=0.865) Rouge L 0.1 (F1=0.861) Embedding Similarity 0.4 (F1=0.846) Embedding Similarity 0.3 (F1=0.843) Token F1 0.1 (F1=0.728) Embedding Similarity 0.5 (F1=0.714) Rouge L 0.2 (F1=0.637) Rouge L 0.3 (F1=0.513) Token F1 0.2 (F1=0.500) Embedding Similarity 0.6 (F1=0.490) Rouge L 0.4 (F1=0.367) Embedding Similarity 0.7 (F1=0.315) Token F1 0.3 (F1=0.234) Embedding Similarity 0.9 (F1=0.222) Embedding Similarity 0.8 (F1=0.206) Rouge L 0.5 (F1=0.188) Rouge L 0.9 (F1=0.181) Token F1 0.4 (F1=0.172) Token F1 0.6 (F1=0.172) Bert Score 0.9 (F1=0.171) Rouge L 0.8 (F1=0.164) Rouge L 0.6 (F1=0.162) Token F1 0.7 (F1=0.159) Token F1 0.9 (F1=0.153) Rouge L 0.7 (F1=0.139) Token F1 0.8 (F1=0.134) Token F1 0.5 (F1=0.132) Bleu 0.2 (F1=0.130) Bleu 0.1 (F1=0.130) Exact Match Correctness (F1=0.125) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Figure 15: Top two plots correspond to Llama 3B, TriviaQA. Bottom two plots correspond to Llama8b, TriviaQA. 34 Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.843) Llm As Judge Correctness Os (F1=0.710) Embedding Similarity 0.4 (F1=0.563) Rouge L 0.2 (F1=0.561) Embedding Similarity 0.7 (F1=0.561) Embedding Similarity 0.5 (F1=0.524) Embedding Similarity 0.9 (F1=0.479) Embedding Similarity 0.8 (F1=0.478) Rouge L 0.6 (F1=0.459) Token F1 0.2 (F1=0.448) Rouge L 0.1 (F1=0.440) Embedding Similarity 0.2 (F1=0.437) Embedding Similarity 0.6 (F1=0.415) Token F1 0.6 (F1=0.410) Embedding Similarity 0.3 (F1=0.355) Bert Score 0.8 (F1=0.338) Rouge L 0.3 (F1=0.310) Embedding Similarity 0.1 (F1=0.310) Token F1 0.1 (F1=0.309) Rouge L 0.5 (F1=0.280) Rouge L 0.4 (F1=0.269) Token F1 0.5 (F1=0.263) Token F1 0.4 (F1=0.170) Token F1 0.3 (F1=0.089) Bleu 0.1 (F1=0.077) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.843) Llm As Judge Correctness Os (F1=0.710) Embedding Similarity 0.4 (F1=0.563) Rouge L 0.2 (F1=0.561) Embedding Similarity 0.7 (F1=0.561) Embedding Similarity 0.5 (F1=0.524) Embedding Similarity 0.9 (F1=0.479) Embedding Similarity 0.8 (F1=0.478) Rouge L 0.6 (F1=0.459) Token F1 0.2 (F1=0.448) Rouge L 0.1 (F1=0.440) Embedding Similarity 0.2 (F1=0.437) Embedding Similarity 0.6 (F1=0.415) Token F1 0.6 (F1=0.410) Embedding Similarity 0.3 (F1=0.355) Bert Score 0.8 (F1=0.338) Rouge L 0.3 (F1=0.310) Embedding Similarity 0.1 (F1=0.310) Token F1 0.1 (F1=0.309) Rouge L 0.5 (F1=0.280) Rouge L 0.4 (F1=0.269) Token F1 0.5 (F1=0.263) Token F1 0.4 (F1=0.170) Token F1 0.3 (F1=0.089) Bleu 0.1 (F1=0.077) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.954) Llm As Judge Correctness Os (F1=0.894) Embedding Similarity 0.3 (F1=0.868) Embedding Similarity 0.4 (F1=0.835) Embedding Similarity 0.1 (F1=0.827) Embedding Similarity 0.5 (F1=0.773) Embedding Similarity 0.2 (F1=0.760) Bert Score 0.8 (F1=0.755) Rouge L 0.1 (F1=0.700) Token F1 0.1 (F1=0.662) Rouge L 0.2 (F1=0.506) Embedding Similarity 0.6 (F1=0.397) Token F1 0.2 (F1=0.376) Rouge L 0.3 (F1=0.289) Embedding Similarity 0.7 (F1=0.249) Rouge L 0.5 (F1=0.209) Token F1 0.4 (F1=0.206) Embedding Similarity 0.9 (F1=0.196) Embedding Similarity 0.8 (F1=0.194) Rouge L 0.4 (F1=0.189) Token F1 0.6 (F1=0.175) Token F1 0.3 (F1=0.173) Rouge L 0.6 (F1=0.161) Token F1 0.5 (F1=0.159) Bert Score 0.9 (F1=0.149) Bleu 0.1 (F1=0.112) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.954) Llm As Judge Correctness Os (F1=0.894) Embedding Similarity 0.3 (F1=0.868) Embedding Similarity 0.4 (F1=0.835) Embedding Similarity 0.1 (F1=0.827) Embedding Similarity 0.5 (F1=0.773) Embedding Similarity 0.2 (F1=0.760) Bert Score 0.8 (F1=0.755) Rouge L 0.1 (F1=0.700) Token F1 0.1 (F1=0.662) Rouge L 0.2 (F1=0.506) Embedding Similarity 0.6 (F1=0.397) Token F1 0.2 (F1=0.376) Rouge L 0.3 (F1=0.289) Embedding Similarity 0.7 (F1=0.249) Rouge L 0.5 (F1=0.209) Token F1 0.4 (F1=0.206) Embedding Similarity 0.9 (F1=0.196) Embedding Similarity 0.8 (F1=0.194) Rouge L 0.4 (F1=0.189) Token F1 0.6 (F1=0.175) Token F1 0.3 (F1=0.173) Rouge L 0.6 (F1=0.161) Token F1 0.5 (F1=0.159) Bert Score 0.9 (F1=0.149) Bleu 0.1 (F1=0.112) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Figure 16: Top two plots correspond to Qwen 0.5B, TriviaQA. Bottom two plots correspond to Qwen 7B, TriviaQA. 35 Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.866) Embedding Similarity 0.3 (F1=0.842) Llm As Judge Correctness Os (F1=0.803) Embedding Similarity 0.2 (F1=0.763) Rouge L 0.1 (F1=0.730) Bert Score 0.8 (F1=0.720) Embedding Similarity 0.4 (F1=0.706) Embedding Similarity 0.1 (F1=0.649) Rouge L 0.2 (F1=0.633) Token F1 0.1 (F1=0.550) Embedding Similarity 0.5 (F1=0.532) Embedding Similarity 0.6 (F1=0.490) Token F1 0.2 (F1=0.485) Rouge L 0.3 (F1=0.460) Token F1 0.3 (F1=0.315) Embedding Similarity 0.7 (F1=0.247) Bert Score 0.9 (F1=0.202) Bleu 0.1 (F1=0.200) Rouge L 0.4 (F1=0.182) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.866) Embedding Similarity 0.3 (F1=0.842) Llm As Judge Correctness Os (F1=0.803) Embedding Similarity 0.2 (F1=0.763) Rouge L 0.1 (F1=0.730) Bert Score 0.8 (F1=0.720) Embedding Similarity 0.4 (F1=0.706) Embedding Similarity 0.1 (F1=0.649) Rouge L 0.2 (F1=0.633) Token F1 0.1 (F1=0.550) Embedding Similarity 0.5 (F1=0.532) Embedding Similarity 0.6 (F1=0.490) Token F1 0.2 (F1=0.485) Rouge L 0.3 (F1=0.460) Token F1 0.3 (F1=0.315) Embedding Similarity 0.7 (F1=0.247) Bert Score 0.9 (F1=0.202) Bleu 0.1 (F1=0.200) Rouge L 0.4 (F1=0.182) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.941) Llm As Judge Correctness Os (F1=0.877) Embedding Similarity 0.3 (F1=0.861) Embedding Similarity 0.2 (F1=0.824) Embedding Similarity 0.1 (F1=0.804) Bert Score 0.8 (F1=0.802) Bert Score 0.7 (F1=0.799) Rouge L 0.1 (F1=0.754) Embedding Similarity 0.4 (F1=0.704) Embedding Similarity 0.5 (F1=0.590) Token F1 0.1 (F1=0.566) Rouge L 0.2 (F1=0.512) Token F1 0.2 (F1=0.419) Embedding Similarity 0.6 (F1=0.388) Rouge L 0.3 (F1=0.337) Token F1 0.3 (F1=0.227) Rouge L 0.4 (F1=0.182) Token F1 0.4 (F1=0.175) Bert Score 0.9 (F1=0.172) Bleu 0.1 (F1=0.171) Embedding Similarity 0.7 (F1=0.168) Rouge L 0.8 (F1=0.167) Rouge L 0.7 (F1=0.162) Embedding Similarity 0.9 (F1=0.161) Bleu 0.2 (F1=0.157) Rouge L 0.6 (F1=0.155) Embedding Similarity 0.8 (F1=0.155) Token F1 0.5 (F1=0.152) Rouge L 0.5 (F1=0.146) Rouge L 0.9 (F1=0.141) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.941) Llm As Judge Correctness Os (F1=0.877) Embedding Similarity 0.3 (F1=0.861) Embedding Similarity 0.2 (F1=0.824) Embedding Similarity 0.1 (F1=0.804) Bert Score 0.8 (F1=0.802) Bert Score 0.7 (F1=0.799) Rouge L 0.1 (F1=0.754) Embedding Similarity 0.4 (F1=0.704) Embedding Similarity 0.5 (F1=0.590) Token F1 0.1 (F1=0.566) Rouge L 0.2 (F1=0.512) Token F1 0.2 (F1=0.419) Embedding Similarity 0.6 (F1=0.388) Rouge L 0.3 (F1=0.337) Token F1 0.3 (F1=0.227) Rouge L 0.4 (F1=0.182) Token F1 0.4 (F1=0.175) Bert Score 0.9 (F1=0.172) Bleu 0.1 (F1=0.171) Embedding Similarity 0.7 (F1=0.168) Rouge L 0.8 (F1=0.167) Rouge L 0.7 (F1=0.162) Embedding Similarity 0.9 (F1=0.161) Bleu 0.2 (F1=0.157) Rouge L 0.6 (F1=0.155) Embedding Similarity 0.8 (F1=0.155) Token F1 0.5 (F1=0.152) Rouge L 0.5 (F1=0.146) Rouge L 0.9 (F1=0.141) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Figure 17: Top two plots correspond to Llama 3B, AmbigQA (unambiguous). Bottom two plots correspond to Llama 8B, AmbigQA (unambiguous). 36 Human Judgement (F1=1.000) Embedding Similarity 0.5 (F1=0.858) Embedding Similarity 0.4 (F1=0.758) Rouge L 0.2 (F1=0.743) Llm As Judge Correctness Os (F1=0.737) Llm As Judge Correctness Api (F1=0.667) Token F1 0.2 (F1=0.580) Rouge L 0.1 (F1=0.543) Embedding Similarity 0.6 (F1=0.540) Token F1 0.3 (F1=0.517) Embedding Similarity 0.7 (F1=0.498) Token F1 0.1 (F1=0.481) Embedding Similarity 0.3 (F1=0.477) Bleu 0.1 (F1=0.477) Embedding Similarity 0.2 (F1=0.367) Bert Score 0.8 (F1=0.345) Embedding Similarity 0.1 (F1=0.265) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Embedding Similarity 0.5 (F1=0.858) Embedding Similarity 0.4 (F1=0.758) Rouge L 0.2 (F1=0.743) Llm As Judge Correctness Os (F1=0.737) Llm As Judge Correctness Api (F1=0.667) Token F1 0.2 (F1=0.580) Rouge L 0.1 (F1=0.543) Embedding Similarity 0.6 (F1=0.540) Token F1 0.3 (F1=0.517) Embedding Similarity 0.7 (F1=0.498) Token F1 0.1 (F1=0.481) Embedding Similarity 0.3 (F1=0.477) Bleu 0.1 (F1=0.477) Embedding Similarity 0.2 (F1=0.367) Bert Score 0.8 (F1=0.345) Embedding Similarity 0.1 (F1=0.265) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.892) Llm As Judge Correctness Os (F1=0.857) Embedding Similarity 0.3 (F1=0.762) Embedding Similarity 0.2 (F1=0.724) Embedding Similarity 0.4 (F1=0.692) Embedding Similarity 0.1 (F1=0.653) Bert Score 0.8 (F1=0.641) Embedding Similarity 0.5 (F1=0.619) Rouge L 0.1 (F1=0.555) Token F1 0.1 (F1=0.392) Rouge L 0.2 (F1=0.358) Embedding Similarity 0.6 (F1=0.326) Embedding Similarity 0.7 (F1=0.259) Rouge L 0.3 (F1=0.253) Token F1 0.2 (F1=0.238) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.892) Llm As Judge Correctness Os (F1=0.857) Embedding Similarity 0.3 (F1=0.762) Embedding Similarity 0.2 (F1=0.724) Embedding Similarity 0.4 (F1=0.692) Embedding Similarity 0.1 (F1=0.653) Bert Score 0.8 (F1=0.641) Embedding Similarity 0.5 (F1=0.619) Rouge L 0.1 (F1=0.555) Token F1 0.1 (F1=0.392) Rouge L 0.2 (F1=0.358) Embedding Similarity 0.6 (F1=0.326) Embedding Similarity 0.7 (F1=0.259) Rouge L 0.3 (F1=0.253) Token F1 0.2 (F1=0.238) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Figure 18: Top two plots correspond to Qwen 0.5B, AmbigQA (unambiguous). Bottom two plots correspond to Qwen 7B, AmbigQA (unambiguous). 37 Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.883) Llm As Judge Correctness Os (F1=0.857) Bert Score 0.8 (F1=0.797) Embedding Similarity 0.1 (F1=0.772) Embedding Similarity 0.4 (F1=0.729) Embedding Similarity 0.3 (F1=0.728) Embedding Similarity 0.2 (F1=0.718) Rouge L 0.1 (F1=0.581) Token F1 0.1 (F1=0.578) Embedding Similarity 0.5 (F1=0.535) Rouge L 0.2 (F1=0.403) Embedding Similarity 0.6 (F1=0.318) Rouge L 0.3 (F1=0.264) Embedding Similarity 0.7 (F1=0.227) Rouge L 0.4 (F1=0.215) Token F1 0.4 (F1=0.188) Token F1 0.3 (F1=0.179) Bert Score 0.9 (F1=0.174) Embedding Similarity 0.8 (F1=0.172) Rouge L 0.5 (F1=0.162) Rouge L 0.6 (F1=0.151) Token F1 0.2 (F1=0.150) Bleu 0.1 (F1=0.130) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.883) Llm As Judge Correctness Os (F1=0.857) Bert Score 0.8 (F1=0.797) Embedding Similarity 0.1 (F1=0.772) Embedding Similarity 0.4 (F1=0.729) Embedding Similarity 0.3 (F1=0.728) Embedding Similarity 0.2 (F1=0.718) Rouge L 0.1 (F1=0.581) Token F1 0.1 (F1=0.578) Embedding Similarity 0.5 (F1=0.535) Rouge L 0.2 (F1=0.403) Embedding Similarity 0.6 (F1=0.318) Rouge L 0.3 (F1=0.264) Embedding Similarity 0.7 (F1=0.227) Rouge L 0.4 (F1=0.215) Token F1 0.4 (F1=0.188) Token F1 0.3 (F1=0.179) Bert Score 0.9 (F1=0.174) Embedding Similarity 0.8 (F1=0.172) Rouge L 0.5 (F1=0.162) Rouge L 0.6 (F1=0.151) Token F1 0.2 (F1=0.150) Bleu 0.1 (F1=0.130) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.910) Embedding Similarity 0.3 (F1=0.833) Llm As Judge Correctness Os (F1=0.826) Bert Score 0.8 (F1=0.804) Embedding Similarity 0.1 (F1=0.779) Embedding Similarity 0.4 (F1=0.765) Embedding Similarity 0.2 (F1=0.742) Rouge L 0.1 (F1=0.652) Token F1 0.1 (F1=0.514) Embedding Similarity 0.5 (F1=0.490) Embedding Similarity 0.6 (F1=0.357) Rouge L 0.2 (F1=0.268) Rouge L 0.3 (F1=0.213) Token F1 0.2 (F1=0.202) Bleu 0.2 (F1=0.187) Bleu 0.1 (F1=0.163) Token F1 0.3 (F1=0.154) Embedding Similarity 0.7 (F1=0.152) Rouge L 0.4 (F1=0.147) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.910) Embedding Similarity 0.3 (F1=0.833) Llm As Judge Correctness Os (F1=0.826) Bert Score 0.8 (F1=0.804) Embedding Similarity 0.1 (F1=0.779) Embedding Similarity 0.4 (F1=0.765) Embedding Similarity 0.2 (F1=0.742) Rouge L 0.1 (F1=0.652) Token F1 0.1 (F1=0.514) Embedding Similarity 0.5 (F1=0.490) Embedding Similarity 0.6 (F1=0.357) Rouge L 0.2 (F1=0.268) Rouge L 0.3 (F1=0.213) Token F1 0.2 (F1=0.202) Bleu 0.2 (F1=0.187) Bleu 0.1 (F1=0.163) Token F1 0.3 (F1=0.154) Embedding Similarity 0.7 (F1=0.152) Rouge L 0.4 (F1=0.147) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Figure 19: Top two plots correspond to Llama 3B, AmbigQA (ambiguous). Bottom two plots correspond to Llama 8B, AmbigQA (ambiguous). 38 Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.893) Llm As Judge Correctness Os (F1=0.859) Embedding Similarity 0.6 (F1=0.547) Embedding Similarity 0.7 (F1=0.497) Bert Score 0.8 (F1=0.232) Embedding Similarity 0.2 (F1=0.227) Embedding Similarity 0.3 (F1=0.220) Embedding Similarity 0.4 (F1=0.214) Embedding Similarity 0.1 (F1=0.189) Token F1 0.2 (F1=0.162) Rouge L 0.2 (F1=0.153) Embedding Similarity 0.5 (F1=0.140) Token F1 0.1 (F1=0.098) Rouge L 0.1 (F1=0.060) Token F1 0.3 (F1=0.000) Token F1 0.4 (F1=0.000) Bleu 0.1 (F1=0.000) Rouge L 0.3 (F1=0.000) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.893) Llm As Judge Correctness Os (F1=0.859) Embedding Similarity 0.6 (F1=0.547) Embedding Similarity 0.7 (F1=0.497) Bert Score 0.8 (F1=0.232) Embedding Similarity 0.2 (F1=0.227) Embedding Similarity 0.3 (F1=0.220) Embedding Similarity 0.4 (F1=0.214) Embedding Similarity 0.1 (F1=0.189) Token F1 0.2 (F1=0.162) Rouge L 0.2 (F1=0.153) Embedding Similarity 0.5 (F1=0.140) Token F1 0.1 (F1=0.098) Rouge L 0.1 (F1=0.060) Token F1 0.3 (F1=0.000) Token F1 0.4 (F1=0.000) Bleu 0.1 (F1=0.000) Rouge L 0.3 (F1=0.000) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.837) Embedding Similarity 0.3 (F1=0.721) Llm As Judge Correctness Os (F1=0.710) Embedding Similarity 0.2 (F1=0.653) Bert Score 0.8 (F1=0.645) Embedding Similarity 0.1 (F1=0.611) Embedding Similarity 0.4 (F1=0.597) Embedding Similarity 0.5 (F1=0.497) Rouge L 0.1 (F1=0.490) Token F1 0.1 (F1=0.363) Embedding Similarity 0.6 (F1=0.331) Rouge L 0.2 (F1=0.281) Token F1 0.2 (F1=0.244) Bleu 0.1 (F1=0.218) Embedding Similarity 0.7 (F1=0.210) Token F1 0.4 (F1=0.200) Rouge L 0.3 (F1=0.197) Token F1 0.3 (F1=0.194) Correctness function 0.0 0.2 0.4 0.6 0.8 1.0 Mean AUROC ± std Mean AUROC across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Human Judgement (F1=1.000) Llm As Judge Correctness Api (F1=0.837) Embedding Similarity 0.3 (F1=0.721) Llm As Judge Correctness Os (F1=0.710) Embedding Similarity 0.2 (F1=0.653) Bert Score 0.8 (F1=0.645) Embedding Similarity 0.1 (F1=0.611) Embedding Similarity 0.4 (F1=0.597) Embedding Similarity 0.5 (F1=0.497) Rouge L 0.1 (F1=0.490) Token F1 0.1 (F1=0.363) Embedding Similarity 0.6 (F1=0.331) Rouge L 0.2 (F1=0.281) Token F1 0.2 (F1=0.244) Bleu 0.1 (F1=0.218) Embedding Similarity 0.7 (F1=0.210) Token F1 0.4 (F1=0.200) Rouge L 0.3 (F1=0.197) Token F1 0.3 (F1=0.194) Correctness function 1 2 3 4 5 6 7 Quantifier rank (1 = best AUROC) Uncertainty quantifier ranking across correctness functions Uncertainty quantifier Ent. N.P(seq.) P(seq.) P(true) Sem. conf. Sem. entropy Verb. conf. Figure 20: Top two plots correspond to Qwen 0.5B, AmbigQA (ambiguous). Bottom two plots correspond to Qwen 7B, AmbigQA (ambiguous). 39 N.P(seq.) P(seq.) Sem. conf. P(true) Verb. conf. Ent. Sem. entropy Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.960.960.960.960.960.000.00 0.960.960.960.960.960.000.00 0.950.950.950.950.950.000.00 0.960.950.960.950.960.010.01 0.950.930.940.930.940.010.03 0.940.900.910.900.910.030.04 0.880.880.880.880.880.030.03 0.890.850.880.860.880.070.09 0.860.820.840.820.840.060.07 0.740.700.740.700.710.140.17 0.740.700.720.710.720.140.16 0.680.620.660.640.650.230.28 0.490.420.470.440.460.410.44 0.540.500.560.550.540.300.34 0.050.030.030.000.050.940.94 0.070.050.070.030.050.900.90 0.500.460.490.470.470.380.44 0.100.070.090.050.110.850.85 0.300.250.280.250.240.600.64 0.190.160.190.160.160.710.72 0.280.230.300.280.280.560.60 0.310.340.350.340.350.500.53 0.260.280.290.280.280.560.59 0.290.290.330.340.340.510.55 Risk@Coverage (80.0%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Risk@Cov Verb. conf. N.P(seq.) P(true) Sem. conf. P(seq.) Ent. Sem. entropy Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.970.970.970.970.970.000.00 0.970.990.970.970.970.010.01 0.970.990.970.970.970.010.01 0.970.990.970.970.970.010.01 0.960.960.960.960.960.000.00 0.960.960.970.960.960.010.01 0.960.970.960.960.960.010.01 0.950.960.950.950.950.010.03 0.950.960.950.950.950.010.04 0.940.950.940.940.940.030.03 0.910.940.910.910.910.040.04 0.910.940.910.910.910.040.05 0.850.880.860.850.850.090.10 0.820.840.840.820.820.070.09 0.810.840.820.810.810.100.11 0.710.740.750.720.710.140.17 0.690.720.720.710.690.160.20 0.650.680.680.660.640.210.26 0.500.530.560.550.490.330.35 0.410.450.450.460.400.440.46 0.440.460.440.450.420.390.45 0.010.000.000.010.010.970.97 0.210.260.240.260.210.640.68 0.030.030.030.040.040.930.93 0.050.110.100.120.050.840.85 0.200.240.250.290.190.610.65 0.090.140.100.150.120.790.80 0.170.240.210.230.250.620.60 0.100.160.160.150.210.700.69 0.120.190.190.200.210.650.65 Risk@Coverage (80.0%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Risk@Cov N.P(seq.) P(seq.) Sem. conf. P(true) Verb. conf. Ent. Sem. entropy Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) 0.700.700.660.650.65-0.51-0.55 0.700.700.660.650.65-0.51-0.55 0.700.700.660.650.65-0.51-0.55 0.700.700.660.650.65-0.51-0.55 0.700.700.660.650.65-0.51-0.55 0.690.690.650.640.64-0.51-0.55 0.690.690.650.640.64-0.51-0.55 0.690.690.650.640.64-0.51-0.55 0.690.690.650.640.64-0.51-0.55 0.690.690.650.640.64-0.51-0.55 0.690.690.650.640.64-0.51-0.55 0.680.680.640.620.62-0.51-0.55 0.680.680.640.620.62-0.51-0.55 0.660.660.620.610.61-0.51-0.55 0.680.660.640.610.62-0.50-0.54 0.660.640.610.590.60-0.50-0.53 0.650.610.590.560.57-0.49-0.51 0.590.590.550.540.54-0.49-0.53 0.600.560.550.530.54-0.44-0.46 0.580.540.510.490.50-0.45-0.48 0.450.410.410.360.38-0.37-0.38 0.450.410.400.380.39-0.37-0.39 0.390.340.340.300.31-0.29-0.28 0.200.140.150.100.12-0.10-0.11 0.250.210.240.210.20-0.21-0.21 -0.24-0.26-0.30-0.34-0.290.430.39 -0.21-0.24-0.25-0.31-0.290.390.35 0.210.180.160.140.14-0.14-0.11 -0.19-0.21-0.24-0.29-0.230.340.30 0.01-0.04-0.05-0.09-0.100.090.09 -0.10-0.12-0.14-0.18-0.180.200.17 -0.01-0.06-0.03-0.06-0.060.050.05 0.030.050.020.000.01-0.01-0.03 -0.02-0.01-0.04-0.06-0.060.050.04 Risk@Coverage (80.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Risk@Cov difference Verb. conf. N.P(seq.) P(true) Sem. conf. P(seq.) Ent. Sem. entropy Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) 0.860.800.800.790.78-0.65-0.65 0.860.800.800.790.78-0.65-0.65 0.860.800.800.790.78-0.65-0.65 0.860.800.800.790.78-0.65-0.65 0.860.800.800.790.78-0.65-0.65 0.850.790.790.770.76-0.65-0.65 0.850.800.790.770.76-0.64-0.64 0.850.800.790.770.76-0.64-0.64 0.850.800.790.770.76-0.64-0.64 0.840.780.780.760.75-0.65-0.65 0.840.780.790.760.75-0.64-0.64 0.840.790.780.760.75-0.64-0.64 0.820.780.760.750.74-0.64-0.62 0.820.780.760.750.74-0.64-0.61 0.810.760.750.740.72-0.62-0.62 0.790.750.720.710.70-0.61-0.61 0.790.750.720.710.70-0.61-0.60 0.720.690.680.650.64-0.56-0.55 0.700.650.650.620.61-0.58-0.56 0.690.650.640.610.60-0.55-0.54 0.590.550.560.520.50-0.51-0.48 0.560.540.540.510.47-0.49-0.45 0.530.490.490.460.42-0.44-0.39 0.380.340.380.350.28-0.33-0.30 0.290.260.260.260.19-0.21-0.19 0.310.280.250.250.21-0.26-0.20 -0.11-0.19-0.19-0.19-0.200.320.32 0.090.080.050.060.00-0.010.03 -0.10-0.16-0.16-0.16-0.170.280.28 -0.07-0.07-0.09-0.08-0.160.190.20 0.080.050.060.09-0.02-0.040.00 -0.04-0.05-0.09-0.05-0.090.140.15 0.050.050.020.020.04-0.03-0.05 -0.02-0.02-0.02-0.050.000.050.04 Risk@Coverage (80.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Risk@Cov difference P(seq.) N.P(seq.) Sem. conf. P(true) Verb. conf. Ent. Sem. entropy Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 4536712 3654712 3754612 3754612 4536712 3754612 3754612 3764512 3754612 3764512 2764513 3476512 3421567 2541367 3764512 2431567 2543167 1542367 1253467 2143567 2153467 1234567 Risk@Coverage (80.0%) Ranking per Correctness 1 2 3 4 5 6 7 Risk@Cov rank (a) Llama 3B Verb. conf. N.P(seq.) P(true) Sem. conf. P(seq.) Ent. Sem. entropy Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 7563412 7563412 7563412 7563412 7563412 7563412 6753412 6753412 6753412 7563412 6573412 6753412 6753412 6753412 6753412 6753412 6753412 5763412 5673412 5763412 4675312 4675312 4675312 4576312 2457136 4736215 5123467 2534167 3124567 2435167 2345167 1425367 1423576 1342576 1234567 Risk@Coverage (80.0%) Ranking per Correctness 1 2 3 4 5 6 7 Risk@Cov rank (b) Llama 8B Figure 21: Top row shows risk at fixed coverage values, middle row shows differences and bottom row rankings for various uncertainty quantifiers. 40 Sem. entropy Ent. Sem. conf. Verb. conf. N.P(seq.) P(seq.) P(true) Uncertainty metric Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion 0.040.040.940.930.910.910.93 0.000.000.970.970.970.970.97 0.000.000.970.970.970.970.97 0.010.010.960.970.970.960.96 0.000.010.970.960.960.960.97 0.000.000.970.970.970.970.97 0.000.000.990.990.990.990.99 0.000.000.990.990.990.990.99 0.000.000.990.990.990.990.99 0.040.040.930.930.900.900.91 0.750.760.160.240.210.200.17 0.890.910.090.110.100.100.11 0.010.030.950.950.960.950.95 0.010.030.960.960.970.960.96 0.720.740.230.230.240.250.25 0.110.120.840.840.810.800.84 0.040.090.890.910.900.890.91 0.250.280.660.710.620.610.69 0.280.310.620.680.590.570.66 0.490.540.410.490.460.460.44 0.170.230.750.790.790.790.79 0.150.170.790.800.760.750.80 0.310.340.640.690.660.660.66 0.100.150.810.840.840.840.86 0.120.170.780.820.800.790.81 0.110.170.800.820.840.840.85 Risk@Coverage (80.0%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Risk@Cov Sem. conf. P(true) P(seq.) Verb. conf. N.P(seq.) Ent. Sem. entropy Uncertainty metric Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.990.970.970.990.970.010.01 0.990.990.990.990.990.000.00 0.990.990.990.990.990.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.970.970.970.970.970.000.00 0.960.960.960.960.960.000.00 0.950.950.950.950.950.000.01 0.930.910.910.930.910.010.03 0.890.880.880.890.880.030.05 0.860.860.880.880.880.060.07 0.850.840.840.850.840.070.10 0.760.760.750.780.750.100.15 0.750.760.780.800.790.160.19 0.640.640.610.660.620.190.26 0.560.560.540.600.550.260.33 0.550.560.590.590.600.340.36 0.030.010.040.040.040.950.95 0.060.050.110.110.110.880.86 0.120.120.160.160.170.790.81 0.340.360.380.390.410.550.55 0.200.230.260.260.280.660.65 0.300.310.380.350.400.470.50 0.300.300.350.350.360.470.51 0.260.280.310.330.350.510.55 Risk@Coverage (80.0%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Risk@Cov Sem. entropy Ent. Sem. conf. Verb. conf. N.P(seq.) P(seq.) P(true) Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) -0.08-0.140.140.100.070.070.08 -0.11-0.170.170.150.140.140.12 -0.11-0.170.170.150.140.140.12 -0.10-0.160.160.150.140.120.11 -0.11-0.160.170.140.120.120.12 -0.11-0.170.170.150.140.140.12 -0.11-0.170.190.160.150.150.14 -0.11-0.170.190.160.150.150.14 -0.11-0.170.190.160.150.150.14 -0.08-0.140.120.100.060.060.06 0.640.59-0.64-0.59-0.62-0.64-0.68 0.770.74-0.71-0.71-0.74-0.74-0.74 -0.10-0.150.150.120.120.110.10 -0.10-0.150.160.140.140.120.11 0.610.56-0.58-0.60-0.60-0.59-0.60 0.00-0.050.040.01-0.03-0.04-0.01 -0.08-0.090.090.090.060.050.06 0.140.10-0.14-0.11-0.21-0.22-0.16 0.160.14-0.18-0.15-0.25-0.26-0.19 0.380.36-0.39-0.34-0.38-0.38-0.41 0.060.05-0.05-0.04-0.05-0.05-0.06 0.040.00-0.01-0.02-0.08-0.09-0.05 0.200.16-0.16-0.14-0.18-0.18-0.19 -0.01-0.020.010.010.000.000.01 0.010.00-0.030.00-0.04-0.05-0.04 Risk@Coverage (80.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Risk@Cov difference Sem. conf. P(true) P(seq.) Verb. conf. N.P(seq.) Ent. Sem. entropy Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) 0.730.710.680.660.64-0.51-0.55 0.730.710.680.660.64-0.51-0.55 0.730.700.660.660.62-0.50-0.54 0.730.710.680.660.64-0.51-0.55 0.730.710.680.660.64-0.51-0.55 0.710.700.660.650.62-0.51-0.55 0.710.700.660.650.62-0.51-0.55 0.710.700.660.650.62-0.51-0.55 0.700.690.650.640.61-0.51-0.55 0.690.670.640.620.60-0.51-0.54 0.660.640.600.600.56-0.50-0.53 0.620.600.560.560.53-0.49-0.50 0.600.590.560.550.53-0.45-0.48 0.590.560.530.520.49-0.44-0.45 0.500.490.440.450.40-0.41-0.40 0.490.490.460.480.44-0.35-0.36 0.370.360.300.340.28-0.32-0.29 0.300.290.220.270.20-0.25-0.23 0.290.290.280.260.25-0.17-0.19 -0.24-0.26-0.28-0.29-0.310.440.40 -0.20-0.23-0.20-0.21-0.240.360.31 -0.14-0.15-0.15-0.16-0.170.280.26 0.080.090.060.060.060.040.00 -0.06-0.05-0.05-0.06-0.070.150.10 0.040.040.060.020.05-0.04-0.05 0.040.020.040.020.01-0.04-0.04 Risk@Coverage (80.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Risk@Cov difference Sem. entropy Ent. Sem. conf. Verb. conf. P(seq.) N.P(seq.) P(true) Uncertainty metric Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion 2176345 2137456 2137456 2137465 1265347 2137456 2137456 2137456 2137456 2167345 6715342 6715234 1236475 1236475 6712435 1257346 1237456 1257346 1257346 5716342 1237456 1257346 1237456 1236457 1237456 1234567 Risk@Coverage (80.0%) Ranking per Correctness 1 2 3 4 5 6 7 Risk@Cov rank (a) Qwen 0.5B Sem. conf. P(true) P(seq.) Verb. conf. N.P(seq.) Ent. Sem. entropy Uncertainty metric Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 3647512 3647512 6537412 3647512 3647512 3647512 3647512 3647512 3647512 3647512 6537412 6537412 3457612 6537412 5637412 3457612 5637412 5637412 3456712 2135467 2135476 1234567 1234567 1234576 1243567 1234567 1234567 Risk@Coverage (80.0%) Ranking per Correctness 1 2 3 4 5 6 7 Risk@Cov rank (b) Qwen 7B Figure 22: Top row shows risk at fixed coverage values, middle row shows differences and bottom row rankings for various uncertainty quantifiers. 41 P(seq.) N.P(seq.) P(true) Verb. conf. Sem. conf. Ent. Sem. entropy Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.020.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.020.000.000.020.010.950.86 0.060.000.000.020.010.950.86 0.000.000.080.000.000.700.46 0.270.110.100.260.010.000.00 0.270.070.100.020.010.000.20 1.001.001.001.001.000.000.00 1.001.001.001.001.000.000.00 0.150.000.150.200.010.000.20 1.001.001.001.001.000.000.00 0.620.570.750.700.010.000.00 0.920.850.960.860.810.000.00 0.670.570.700.620.270.000.00 0.610.560.580.570.300.000.03 0.710.680.650.680.300.000.00 0.670.660.600.570.270.000.00 Coverage@Risk (target_risk*100%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Cov@Risk Verb. conf. N.P(seq.) P(true) Sem. conf. P(seq.) Sem. entropy Ent. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.010.000.000.011.001.00 0.000.010.000.000.011.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.010.000.000.011.001.00 0.000.010.000.000.011.001.00 0.000.000.000.000.001.001.00 0.000.010.000.000.011.001.00 0.000.050.000.000.051.001.00 0.000.000.000.000.001.001.00 0.000.070.000.000.100.880.96 0.000.050.000.000.070.800.92 0.010.050.000.000.070.530.75 0.010.100.000.010.120.410.00 0.020.110.000.000.110.350.00 0.010.070.010.000.150.110.11 1.001.001.001.001.000.000.00 0.780.560.750.350.760.020.00 1.001.001.001.001.000.000.00 1.001.001.001.001.000.020.00 0.800.660.150.120.820.020.00 1.001.001.001.001.000.020.00 0.820.560.650.700.650.020.00 0.910.910.870.870.770.020.00 0.880.830.820.800.770.020.00 Coverage@Risk (target_risk*100%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Cov@Risk P(seq.) N.P(seq.) P(true) Verb. conf. Sem. conf. Ent. Sem. entropy Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.65-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.67-0.66-0.60-0.57-0.271.001.00 -0.65-0.66-0.60-0.55-0.260.950.86 -0.61-0.66-0.60-0.55-0.260.950.86 -0.67-0.66-0.52-0.57-0.270.700.46 -0.40-0.55-0.50-0.31-0.260.000.00 -0.40-0.59-0.50-0.55-0.260.000.20 0.330.340.400.430.730.000.00 0.330.340.400.430.730.000.00 -0.52-0.66-0.45-0.37-0.260.000.20 0.330.340.400.430.730.000.00 -0.05-0.090.150.13-0.260.000.00 0.250.190.360.290.540.000.00 0.00-0.090.100.050.000.000.00 -0.06-0.10-0.020.000.030.000.03 0.040.020.050.110.030.000.00 Coverage@Risk (20.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Cov@Risk difference Verb. conf. N.P(seq.) P(true) Sem. conf. P(seq.) Sem. entropy Ent. Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.82-0.82-0.80-0.760.981.00 -0.88-0.82-0.82-0.80-0.760.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.82-0.82-0.80-0.760.981.00 -0.88-0.82-0.82-0.80-0.760.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.82-0.82-0.80-0.760.981.00 -0.88-0.78-0.82-0.80-0.720.981.00 -0.88-0.83-0.82-0.80-0.770.981.00 -0.88-0.76-0.82-0.80-0.670.860.96 -0.88-0.78-0.82-0.80-0.700.780.92 -0.87-0.78-0.82-0.80-0.700.510.75 -0.87-0.73-0.82-0.79-0.650.390.00 -0.86-0.72-0.82-0.80-0.660.330.00 -0.87-0.76-0.81-0.80-0.620.090.11 0.120.170.180.200.23-0.020.00 -0.10-0.27-0.07-0.45-0.010.000.00 0.120.170.180.200.23-0.020.00 0.120.170.180.200.230.000.00 -0.08-0.17-0.67-0.680.050.000.00 0.120.170.180.200.230.000.00 -0.06-0.27-0.17-0.10-0.120.000.00 0.030.080.050.070.000.000.00 Coverage@Risk (20.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Cov@Risk difference P(seq.) N.P(seq.) P(true) Verb. conf. Sem. conf. Ent. Sem. entropy Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.06 BLEU (=0.1) · F1=0.06 ROUGE-L (=0.7) · F1=0.06 ROUGE-L (=0.8) · F1=0.06 ROUGE-L (=0.9) · F1=0.06 BERTScore (=0.9) · F1=0.06 Token F1 (=0.5) · F1=0.10 Emb. sim. (=0.9) · F1=0.10 Token F1 (=0.4) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.24 Token F1 (=0.3) · F1=0.29 ROUGE-L (=0.4) · F1=0.33 Emb. sim. (=0.7) · F1=0.38 Token F1 (=0.2) · F1=0.55 ROUGE-L (=0.3) · F1=0.55 Emb. sim. (=0.6) · F1=0.62 Token F1 (=0.1) · F1=0.75 ROUGE-L (=0.2) · F1=0.76 Emb. sim. (=0.1) · F1=0.77 BERTScore (=0.8) · F1=0.78 Emb. sim. (=0.5) · F1=0.79 Emb. sim. (=0.2) · F1=0.81 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.3) · F1=0.86 ROUGE-L (=0.1) · F1=0.88 LLM judge (Llama 8B) · F1=0.92 LLM judge (GPT) · F1=0.94 Human labels · F1=1.00 Correctness criterion 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 4567312 3567412 4567312 4567312 3674512 3674512 5637412 1342567 1435672 2345167 2345167 3742561 2345167 3412567 2413567 2413567 1423576 1243567 1234567 Coverage@Risk (20.0%) Ranking per Correctness 1 2 3 4 5 6 7 AUROC rank (a) Llama 3B Verb. conf. N.P(seq.) P(true) Sem. conf. P(seq.) Sem. entropy Ent. Uncertainty metric Exact match · F1=0.03 Token F1 (=0.7) · F1=0.03 Token F1 (=0.8) · F1=0.03 Token F1 (=0.9) · F1=0.03 BLEU (=0.2) · F1=0.03 Token F1 (=0.6) · F1=0.05 ROUGE-L (=0.7) · F1=0.05 ROUGE-L (=0.8) · F1=0.05 ROUGE-L (=0.9) · F1=0.05 Token F1 (=0.5) · F1=0.08 BLEU (=0.1) · F1=0.08 ROUGE-L (=0.6) · F1=0.08 Emb. sim. (=0.9) · F1=0.11 BERTScore (=0.9) · F1=0.11 Token F1 (=0.4) · F1=0.13 ROUGE-L (=0.5) · F1=0.18 Emb. sim. (=0.8) · F1=0.18 ROUGE-L (=0.4) · F1=0.29 Token F1 (=0.3) · F1=0.33 Emb. sim. (=0.7) · F1=0.35 ROUGE-L (=0.3) · F1=0.47 Token F1 (=0.2) · F1=0.50 Emb. sim. (=0.6) · F1=0.56 ROUGE-L (=0.2) · F1=0.70 Token F1 (=0.1) · F1=0.73 Emb. sim. (=0.5) · F1=0.77 Emb. sim. (=0.1) · F1=0.84 Emb. sim. (=0.4) · F1=0.85 Emb. sim. (=0.2) · F1=0.86 BERTScore (=0.8) · F1=0.87 ROUGE-L (=0.1) · F1=0.88 Emb. sim. (=0.3) · F1=0.88 LLM judge (Llama 8B) · F1=0.94 LLM judge (GPT) · F1=0.95 Human labels · F1=1.00 Correctness criterion 7563421 7563421 7563421 7563421 7563421 7563421 7563421 7563421 7563421 7465321 7465321 7563421 7563421 7563421 7465321 7465321 7563421 7465321 7465321 7563421 7465321 7465321 5476321 5374216 4376215 6457132 5341276 1435267 5341276 5341267 2345167 5341267 1542367 2143567 1234567 Coverage@Risk (20.0%) Ranking per Correctness 1 2 3 4 5 6 7 AUROC rank (b) Llama 8B Figure 23: Top row shows Coverage at fixed risk values, middle row shows differences and bottom row rankings for various uncertainty quantifiers. 42 Ent. Sem. entropy N.P(seq.) P(seq.) Verb. conf. P(true) Sem. conf. Uncertainty metric Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion 1.001.000.020.050.000.000.00 1.001.000.020.020.000.000.00 1.001.000.020.020.000.000.00 1.001.000.000.000.000.000.00 1.001.000.020.020.000.000.00 1.001.000.020.020.000.000.00 1.001.000.010.010.000.000.00 1.001.000.010.010.000.000.00 1.001.000.010.010.000.000.00 1.001.000.020.050.000.000.00 0.010.010.550.850.520.880.91 0.010.011.001.001.001.001.00 1.001.000.020.020.000.000.00 1.001.000.010.010.000.000.00 0.010.010.450.150.700.210.00 1.001.000.030.100.010.000.00 1.001.000.020.020.000.000.00 0.200.360.030.110.010.000.00 0.010.020.030.110.010.000.00 0.010.020.020.050.010.000.00 0.960.870.020.020.000.000.00 1.001.000.030.100.010.000.00 0.010.160.020.020.010.000.00 1.001.000.010.010.010.000.00 1.001.000.010.010.010.000.00 1.001.000.010.010.010.000.00 Coverage@Risk (target_risk*100%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Cov@Risk P(true) Sem. conf. N.P(seq.) P(seq.) Verb. conf. Sem. entropy Ent. Uncertainty metric Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.001.001.00 0.000.000.000.000.011.001.00 0.000.000.000.000.001.001.00 0.000.000.050.060.051.001.00 0.000.000.080.080.051.001.00 0.000.000.000.000.011.001.00 0.000.010.050.060.051.001.00 0.000.010.150.200.051.001.00 0.000.010.000.000.011.001.00 0.000.050.180.250.060.510.81 0.010.050.180.250.070.250.00 0.000.010.050.060.050.230.02 1.001.001.001.001.000.030.00 1.001.001.001.001.000.030.00 1.001.001.001.001.000.030.00 0.010.010.120.080.050.030.00 0.720.800.470.350.510.030.00 0.610.580.370.320.130.080.00 0.510.580.450.400.250.080.00 0.650.630.370.360.250.070.00 Coverage@Risk (target_risk*100%) Values per Correctness 0.0 0.2 0.4 0.6 0.8 1.0 Cov@Risk Ent. Sem. entropy N.P(seq.) P(seq.) Verb. conf. P(true) Sem. conf. Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) 0.000.000.010.04-0.010.000.00 0.000.000.010.01-0.010.000.00 0.000.000.010.01-0.010.000.00 0.000.00-0.01-0.01-0.010.000.00 0.000.000.010.01-0.010.000.00 0.000.000.010.01-0.010.000.00 0.000.000.000.00-0.010.000.00 0.000.000.000.00-0.010.000.00 0.000.000.000.00-0.010.000.00 0.000.000.010.04-0.010.000.00 -0.99-0.990.540.840.510.880.91 -0.99-0.990.990.990.991.001.00 0.000.000.010.01-0.010.000.00 0.000.000.000.00-0.010.000.00 -0.99-0.990.440.140.690.210.00 0.000.000.020.090.000.000.00 0.000.000.010.01-0.010.000.00 -0.80-0.640.020.100.000.000.00 -0.99-0.980.020.100.000.000.00 -0.99-0.980.010.040.000.000.00 -0.04-0.130.010.01-0.010.000.00 0.000.000.020.090.000.000.00 -0.99-0.840.010.010.000.000.00 0.000.000.000.000.000.000.00 0.000.000.000.000.000.000.00 Coverage@Risk (20.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Cov@Risk difference P(true) Sem. conf. N.P(seq.) P(seq.) Verb. conf. Sem. entropy Ent. Uncertainty metric (sorted by actual AUROC using human-labelled correctness) Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion (sorted by F1 using human labels) -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.37-0.36-0.240.931.00 -0.65-0.63-0.37-0.36-0.250.931.00 -0.65-0.63-0.32-0.30-0.200.931.00 -0.65-0.63-0.29-0.28-0.200.931.00 -0.65-0.63-0.37-0.36-0.240.931.00 -0.65-0.62-0.32-0.30-0.200.931.00 -0.65-0.62-0.22-0.16-0.200.931.00 -0.65-0.62-0.37-0.36-0.240.931.00 -0.65-0.58-0.19-0.11-0.190.440.81 -0.64-0.58-0.19-0.11-0.180.180.00 -0.65-0.62-0.32-0.30-0.200.160.02 0.350.370.630.640.75-0.040.00 0.350.370.630.640.75-0.040.00 0.350.370.630.640.75-0.040.00 -0.64-0.62-0.25-0.28-0.20-0.040.00 0.070.170.10-0.010.26-0.040.00 -0.04-0.050.00-0.04-0.120.010.00 -0.14-0.050.080.040.000.010.00 Coverage@Risk (20.0%) Difference per Correctness 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Cov@Risk difference Ent. Sem. entropy P(seq.) N.P(seq.) Verb. conf. Sem. conf. P(true) Uncertainty metric Token F1 (=0.3) · F1=0.09 Token F1 (=0.4) · F1=0.11 Token F1 (=0.5) · F1=0.11 BLEU (=0.1) · F1=0.11 ROUGE-L (=0.4) · F1=0.11 ROUGE-L (=0.5) · F1=0.11 Token F1 (=0.6) · F1=0.12 ROUGE-L (=0.6) · F1=0.12 Emb. sim. (=0.9) · F1=0.12 ROUGE-L (=0.3) · F1=0.17 BERTScore (=0.8) · F1=0.28 Emb. sim. (=0.1) · F1=0.30 Emb. sim. (=0.7) · F1=0.30 Emb. sim. (=0.8) · F1=0.32 Emb. sim. (=0.2) · F1=0.35 Token F1 (=0.2) · F1=0.38 Emb. sim. (=0.6) · F1=0.40 Token F1 (=0.1) · F1=0.43 ROUGE-L (=0.1) · F1=0.44 Emb. sim. (=0.3) · F1=0.46 Emb. sim. (=0.5) · F1=0.47 ROUGE-L (=0.2) · F1=0.50 Emb. sim. (=0.4) · F1=0.57 LLM judge (Llama 8B) · F1=0.71 LLM judge (GPT) · F1=0.82 Human labels · F1=1.00 Correctness criterion 1234756 1234756 1234756 1245736 1234756 1234756 1234756 1234756 1234756 1234756 6734512 6723514 1234756 1234756 5642173 1234567 1234756 2134567 4312567 4213567 1234756 1234567 4123567 1234567 1234567 1234567 Coverage@Risk (20.0%) Ranking per Correctness 1 2 3 4 5 6 7 AUROC rank (a) Qwen 0.5B P(true) Sem. conf. N.P(seq.) P(seq.) Verb. conf. Sem. entropy Ent. Uncertainty metric Token F1 (=0.5) · F1=0.03 Token F1 (=0.6) · F1=0.03 BLEU (=0.1) · F1=0.03 ROUGE-L (=0.6) · F1=0.03 BERTScore (=0.9) · F1=0.03 Token F1 (=0.4) · F1=0.06 ROUGE-L (=0.5) · F1=0.06 Emb. sim. (=0.9) · F1=0.06 Emb. sim. (=0.8) · F1=0.10 ROUGE-L (=0.4) · F1=0.12 Token F1 (=0.3) · F1=0.18 ROUGE-L (=0.3) · F1=0.26 Emb. sim. (=0.7) · F1=0.31 Token F1 (=0.2) · F1=0.33 ROUGE-L (=0.2) · F1=0.47 Emb. sim. (=0.6) · F1=0.47 Token F1 (=0.1) · F1=0.62 ROUGE-L (=0.1) · F1=0.66 Emb. sim. (=0.5) · F1=0.74 Emb. sim. (=0.1) · F1=0.77 Emb. sim. (=0.2) · F1=0.79 BERTScore (=0.8) · F1=0.82 Emb. sim. (=0.4) · F1=0.84 Emb. sim. (=0.3) · F1=0.85 LLM judge (Llama 8B) · F1=0.90 LLM judge (GPT) · F1=0.96 Human labels · F1=1.00 Correctness criterion 6354721 6354721 6354721 6354721 6354721 6354721 6354721 6354721 7465321 6354721 7643521 7643521 7465321 7643521 7643521 7365421 7643521 6532417 7632415 4132567 4132567 4132567 6512347 2145367 1234567 2134567 1234567 Coverage@Risk (20.0%) Ranking per Correctness 1 2 3 4 5 6 7 AUROC rank (b) Qwen 7B Figure 24: Top row shows Coverage at fixed risk values, middle row shows differences and bottom row rankings for various uncertainty quantifiers. 43 variablejudge F1 bleu/0.6 bleu/0.4 bleu/0.7 bleu/0.9 bleu/0.3 bleu/0.8 bleu/0.5 tokf/0.8 EM tokf/0.7 bleu/0.2 tokf/0.9 rouge/0.7 tokf/0.6 rouge/0.8 rouge/0.9 bleu/0.1 tokf/0.5 rouge/0.6 esim/0.9 bert/0.9 tokf/0.4 esim/0.8 rouge/0.5 rouge/0.4 tokf/0.3 esim/0.7 rouge/0.3 tokf/0.2 esim/0.6 rouge/0.2 tokf/0.1 esim/0.5 bert/0.1 bert/0.2 bert/0.5 bert/0.4 bert/0.3 bert/0.7 bert/0.6 esim/0.1 esim/0.2 bert/0.8 esim/0.4 esim/0.3 rouge/0.1 gpt llama human2 human1 0.00.20.40.60.8 dataset=trivia_qa generator=llama-8b-instruct | Posterior F1 variablejudge F1 bleu/0.4 bleu/0.9 bleu/0.8 bleu/0.5 bleu/0.6 bleu/0.3 bleu/0.7 tokf/0.9 EM bleu/0.2 tokf/0.7 tokf/0.8 rouge/0.8 rouge/0.7 tokf/0.6 bleu/0.1 rouge/0.9 bert/0.9 esim/0.9 tokf/0.5 rouge/0.6 tokf/0.4 rouge/0.5 esim/0.8 tokf/0.3 rouge/0.4 esim/0.7 human2 tokf/0.2 rouge/0.3 esim/0.6 bert/0.5 bert/0.1 bert/0.3 bert/0.2 bert/0.4 bert/0.7 bert/0.6 esim/0.1 bert/0.8 esim/0.2 tokf/0.1 esim/0.3 rouge/0.2 esim/0.4 rouge/0.1 gpt esim/0.5 llama human1 0.00.20.40.60.8 dataset=trivia_qa generator=llama-3b-instruct | Posterior F1 variablejudge F1 tokf/0.9 rouge/0.8 bleu/0.5 rouge/0.9 tokf/0.8 EM bleu/0.6 bleu/0.9 tokf/0.7 rouge/0.7 bleu/0.2 bleu/0.4 bleu/0.8 bleu/0.7 bleu/0.3 rouge/0.6 bert/0.9 tokf/0.6 tokf/0.5 bleu/0.1 rouge/0.5 esim/0.9 tokf/0.4 esim/0.8 rouge/0.4 tokf/0.3 rouge/0.3 esim/0.7 tokf/0.2 esim/0.6 rouge/0.2 human2 tokf/0.1 rouge/0.1 esim/0.5 bert/0.3 bert/0.1 bert/0.5 bert/0.7 bert/0.2 bert/0.6 bert/0.4 esim/0.1 esim/0.2 bert/0.8 esim/0.3 esim/0.4 llama gpt human1 0.00.20.40.60.8 dataset=trivia_qa generator=qwen-7b-instruct | Posterior F1 variablejudge F1 EM rouge/0.8 bleu/0.7 bleu/0.9 rouge/0.7 bleu/0.5 tokf/0.7 bleu/0.6 bert/0.9 tokf/0.9 bleu/0.4 bleu/0.2 rouge/0.9 bleu/0.8 tokf/0.8 bleu/0.3 rouge/0.6 tokf/0.6 esim/0.9 tokf/0.4 rouge/0.5 tokf/0.5 bleu/0.1 rouge/0.4 esim/0.8 human2 esim/0.7 bert/0.5 bert/0.2 bert/0.3 bert/0.6 bert/0.4 bert/0.1 bert/0.7 esim/0.1 bert/0.8 tokf/0.3 esim/0.2 rouge/0.3 esim/0.6 esim/0.3 esim/0.5 esim/0.4 rouge/0.1 llama tokf/0.1 gpt tokf/0.2 human1 rouge/0.2 0.00.10.20.30.40.50.60.7 dataset=trivia_qa generator=qwen-0.5b-instruct | Posterior F1 Figure 25: F1 distributions for TriviaQA dataset and Llama 8B (top left corner), Llama 3B (top right corner), Qwen 7B (bottom left corner) and Qwen 0.5B (bottom right corner). 44 variablejudge F1 bleu/0.3 bleu/0.6 bleu/0.7 tokf/0.8 tokf/0.9 bleu/0.4 tokf/0.6 tokf/0.7 EM bleu/0.8 bleu/0.5 bleu/0.9 rouge/0.8 rouge/0.7 rouge/0.9 tokf/0.5 esim/0.9 bleu/0.2 rouge/0.6 esim/0.8 bert/0.9 rouge/0.5 tokf/0.4 bleu/0.1 esim/0.7 rouge/0.4 tokf/0.3 rouge/0.3 esim/0.6 tokf/0.2 human2 esim/0.5 rouge/0.2 tokf/0.1 bert/0.6 bert/0.1 bert/0.4 bert/0.2 bert/0.3 bert/0.5 bert/0.7 esim/0.1 esim/0.4 llm_as_judge_qwen-0.5b-instruct esim/0.2 bert/0.8 esim/0.3 rouge/0.1 gpt llama human1 0.00.20.40.60.8 dataset=ambig_qa_single_answer generator=llama-8b-instruct | Posterior F1 variablejudge F1 bleu/0.3 tokf/0.4 tokf/0.7 esim/0.9 bleu/0.4 rouge/0.5 tokf/0.9 esim/0.8 rouge/0.9 bleu/0.8 tokf/0.6 tokf/0.8 EM tokf/0.5 bleu/0.7 bleu/0.5 bleu/0.9 bleu/0.2 bleu/0.6 rouge/0.7 rouge/0.8 rouge/0.6 rouge/0.4 bert/0.9 bleu/0.1 esim/0.7 tokf/0.3 rouge/0.3 human2 tokf/0.2 esim/0.6 bert/0.3 bert/0.2 bert/0.6 bert/0.5 bert/0.4 bert/0.1 bert/0.7 esim/0.5 rouge/0.2 esim/0.1 llm_as_judge_qwen-0.5b-instruct bert/0.8 esim/0.2 tokf/0.1 esim/0.4 esim/0.3 rouge/0.1 human1 llama gpt 0.00.10.20.30.40.50.60.7 dataset=ambig_qa_single_answer generator=llama-3b-instruct | Posterior F1 variablejudge F1 tokf/0.5 rouge/0.4 rouge/0.5 rouge/0.9 bleu/0.3 rouge/0.7 bleu/0.7 rouge/0.8 bleu/0.1 tokf/0.6 EM esim/0.9 tokf/0.8 bleu/0.6 bert/0.9 tokf/0.9 tokf/0.3 esim/0.8 rouge/0.6 bleu/0.4 tokf/0.4 bleu/0.5 bleu/0.9 bleu/0.8 bleu/0.2 tokf/0.7 esim/0.7 tokf/0.2 rouge/0.3 rouge/0.2 esim/0.6 tokf/0.1 human2 rouge/0.1 bert/0.4 bert/0.7 bert/0.1 bert/0.3 bert/0.5 bert/0.2 bert/0.6 esim/0.1 llm_as_judge_qwen-0.5b-instruct bert/0.8 esim/0.5 esim/0.2 esim/0.4 esim/0.3 llama gpt human1 0.00.10.20.30.40.50.6 dataset=ambig_qa_single_answer generator=qwen-7b-instruct | Posterior F1 variablejudge F1 bleu/0.7 rouge/0.8 tokf/0.7 rouge/0.3 bleu/0.5 tokf/0.4 rouge/0.6 bert/0.9 bleu/0.6 EM tokf/0.5 tokf/0.9 bleu/0.8 bleu/0.3 tokf/0.6 rouge/0.5 bleu/0.4 rouge/0.4 bleu/0.9 rouge/0.9 rouge/0.7 tokf/0.8 esim/0.9 bleu/0.2 esim/0.8 bleu/0.1 tokf/0.3 human2 tokf/0.2 esim/0.7 bert/0.4 bert/0.6 bert/0.3 bert/0.5 bert/0.1 bert/0.2 bert/0.7 llm_as_judge_qwen-0.5b-instruct esim/0.1 esim/0.6 esim/0.2 bert/0.8 esim/0.3 tokf/0.1 esim/0.4 rouge/0.1 rouge/0.2 llama gpt esim/0.5 human1 0.00.10.20.30.4 dataset=ambig_qa_single_answer generator=qwen-0.5b-instruct | Posterior F1 Figure 26: F1 distributions for AmbigQA (non-ambiguous) dataset and Llama 8B (top left corner), Llama 3B (top right corner), Qwen 7B (bottom left corner) and Qwen 0.5B (bottom right corner). 45 variablejudge F1 rouge/0.6 esim/0.8 rouge/0.8 esim/0.9 rouge/0.9 bleu/0.5 rouge/0.5 tokf/0.7 bleu/0.8 bert/0.9 EM bleu/0.7 bleu/0.6 tokf/0.5 rouge/0.7 tokf/0.4 bleu/0.4 bleu/0.3 bleu/0.9 tokf/0.8 tokf/0.9 tokf/0.6 bleu/0.2 rouge/0.4 tokf/0.3 esim/0.7 bleu/0.1 rouge/0.3 tokf/0.2 rouge/0.2 esim/0.6 tokf/0.1 esim/0.5 rouge/0.1 bert/0.8 bert/0.3 bert/0.1 bert/0.4 bert/0.2 bert/0.6 bert/0.7 bert/0.5 esim/0.1 llm_as_judge_qwen-0.5b-instruct esim/0.4 esim/0.2 esim/0.3 llama human1 human2 gpt 0.00.20.40.60.8 dataset=ambig_qa_multiple_qas generator=llama-8b-instruct | Posterior F1 variablejudge F1 rouge/0.7 esim/0.9 rouge/0.9 rouge/0.8 bleu/0.7 bleu/0.3 bleu/0.9 tokf/0.8 bleu/0.4 tokf/0.5 bleu/0.8 tokf/0.7 EM tokf/0.6 bleu/0.2 bleu/0.6 tokf/0.9 bleu/0.5 rouge/0.5 rouge/0.6 esim/0.8 bert/0.9 bleu/0.1 esim/0.7 tokf/0.4 tokf/0.3 rouge/0.4 rouge/0.3 tokf/0.2 rouge/0.2 esim/0.6 human2 esim/0.5 tokf/0.1 bert/0.7 bert/0.1 bert/0.4 bert/0.6 bert/0.2 bert/0.3 bert/0.5 esim/0.1 rouge/0.1 llm_as_judge_qwen-0.5b-instruct esim/0.2 bert/0.8 esim/0.3 llama esim/0.4 gpt human1 0.00.10.20.30.40.50.60.70.8 dataset=ambig_qa_multiple_qas generator=llama-3b-instruct | Posterior F1 variablejudge F1 esim/0.8 rouge/0.7 rouge/0.6 rouge/0.5 tokf/0.8 rouge/0.4 esim/0.9 bleu/0.4 rouge/0.9 bleu/0.6 bleu/0.3 tokf/0.5 bleu/0.9 rouge/0.8 bleu/0.8 tokf/0.7 bleu/0.7 EM tokf/0.9 tokf/0.6 bleu/0.2 bleu/0.5 bert/0.9 esim/0.7 bleu/0.1 tokf/0.3 tokf/0.4 rouge/0.3 rouge/0.2 tokf/0.2 esim/0.6 tokf/0.1 human2 rouge/0.1 esim/0.5 bert/0.8 bert/0.2 bert/0.4 bert/0.3 bert/0.7 bert/0.1 bert/0.6 bert/0.5 esim/0.1 llm_as_judge_qwen-0.5b-instruct esim/0.2 esim/0.3 esim/0.4 llama human1 gpt 0.00.10.20.30.40.50.60.7 dataset=ambig_qa_multiple_qas generator=qwen-7b-instruct | Posterior F1 variablejudge F1 bleu/0.2 bleu/0.6 rouge/0.8 esim/0.8 rouge/0.5 tokf/0.7 rouge/0.7 bleu/0.3 esim/0.9 tokf/0.9 tokf/0.5 tokf/0.6 rouge/0.9 rouge/0.6 bleu/0.9 bleu/0.8 bleu/0.7 bert/0.9 bleu/0.4 tokf/0.8 rouge/0.4 EM bleu/0.5 rouge/0.3 tokf/0.3 bleu/0.1 tokf/0.4 esim/0.7 esim/0.6 tokf/0.2 rouge/0.2 human2 rouge/0.1 esim/0.5 tokf/0.1 esim/0.4 gpt llama human1 esim/0.3 llm_as_judge_qwen-0.5b-instruct bert/0.8 esim/0.2 esim/0.1 bert/0.2 bert/0.5 bert/0.7 bert/0.4 bert/0.1 bert/0.3 bert/0.6 0.00.20.40.60.81.0 dataset=ambig_qa_multiple_qas generator=qwen-0.5b-instruct | Posterior F1 Figure 27: F1 distributions for AmbigQA (ambiguous) dataset and Llama 8B (top left corner), Llama 3B (top right corner), Qwen 7B (bottom left corner) and Qwen 0.5B (bottom right corner). 46 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Rank-biased Overlap Generator = llama-3b | Dataset = ambig_qa_singleGenerator = llama-3b | Dataset = ambig_qa_multipleGenerator = llama-3b | Dataset = trivia_qa 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Rank-biased Overlap Generator = llama-8b | Dataset = ambig_qa_singleGenerator = llama-8b | Dataset = ambig_qa_multipleGenerator = llama-8b | Dataset = trivia_qa 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Rank-biased Overlap Generator = qwen-0.5b | Dataset = ambig_qa_singleGenerator = qwen-0.5b | Dataset = ambig_qa_multipleGenerator = qwen-0.5b | Dataset = trivia_qa 0.00.20.40.60.81.0 F1 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Rank-biased Overlap Generator = qwen-7b | Dataset = ambig_qa_single 0.00.20.40.60.81.0 F1 Generator = qwen-7b | Dataset = ambig_qa_multiple 0.00.20.40.60.81.0 F1 Generator = qwen-7b | Dataset = trivia_qa Judge llm-os llm-api token bleu rouge embedding bert exact Figure 28: F1 scores of automated judges against RBO scores for all dataset-generator pairs. 47

Instant research discovery