Paper deep dive

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

Yifan Zhang, Rahmatollah Beheshti

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 92

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%

Last extracted: 8/23/2026, 2:30:15 AM

Summary

This paper evaluates 15 Large Language Models (LLMs) on clinical error detection using paired datasets (error-injected vs. clean notes). It demonstrates that standard aggregate metrics like F1 score are misleading because they mask bidirectional prediction bias (yes-bias or no-bias). The authors introduce the Both-Correct Rate (BCR) and Evidence Contrastive Analysis (ECA) to measure true discrimination and evidence localization, finding that most models perform below random pairwise discrimination despite moderate F1 scores.

Entities (10)

Rahmatollah Beheshti · person · 99%University of Delaware · organization · 99%YiFan Zhang · person · 99%Both-Correct Rate · metric · 98%Evidence Contrastive Analysis · method · 98%F1 Score · metric · 95%MEDEC · dataset · 95%MedErrBench · dataset · 95%MedRECT · dataset · 95%Prediction Bias · concept · 95%

Relation Signals (7)

YiFan Zhang → affiliatedwith → University of Delaware

confidence 99% · Yifan Zhang 1 ... 1 University of Delaware

Rahmatollah Beheshti → affiliatedwith → University of Delaware

confidence 99% · Rahmatollah Beheshti 1 ... 1 University of Delaware

YiFan Zhang → authored → Toward Better Assessment of LLMs' Performance in Clinical Error Detection

confidence 99% · Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Yifan Zhang 1

Rahmatollah Beheshti → authored → Toward Better Assessment of LLMs' Performance in Clinical Error Detection

confidence 99% · Yifan Zhang 1 eyfzh@udel.edu Rahmatollah Beheshti 1 rbi@udel.edu

Evidence Contrastive Analysis → diagnoses → discrimination breakdown

confidence 95% · To diagnose where discrimination breaks down, we introduce Evidence Contrastive Analysis (ECA)

F1 Score → masks → Prediction Bias

confidence 95% · aggregate metrics evaluate each note in isolation and can be inflated by bidirectional prediction bias... F1 and pairwise accuracy are driven in opposite directions by the same underlying bias

Both-Correct Rate → measures → pairwise discrimination

confidence 95% · BCR adapts the contrast consistency principle to clinical error detection by requiring correct classification of both members of each pair.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart. Aggregate discriminative metrics (e.g., balanced accuracy or F1) do not exploit this structure. We show that this omission is consequential. In particular, evaluating 15 diverse LLMs on 4 standardized clinical error-detection test sets across 3 languages, we find that 13 of 15 models fall below the level of random pairwise discrimination, even while achieving F1 scores that standard practice would read as moderate. We also observe that the underlying bias patterns differ across languages: the same model can default to "no error" on one language and over-flag errors on another. To diagnose where discrimination breaks down, we further introduce a procedure to score the evidence models cite in their outputs. We find that while models consistently locate error-relevant content, they fail to produce the corresponding correct verdict on the clean counterpart. Finally, we show that F1 and pairwise accuracy are driven in opposite directions by the same underlying bias, so that ranking models by F1 may systematically promote the weakest discriminators. For safety-critical clinical NLP applications, we advocate for supplementing aggregate metrics with paired evaluations in benchmark reporting. Code and analysis scripts are available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

88,822 characters extracted from source content.

Expand or collapse full text

Preprint 2026Accepted at MLHC 2026 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Yifan Zhang 1 eyfzh@udel.edu Rahmatollah Beheshti 1 rbi@udel.edu 1 University of Delaware, Newark, DE, USA Abstract Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that eval- uate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart. Aggregate discriminative metrics (e.g., balanced accuracy or F1) do not exploit this struc- ture. We show that this omission is consequential. In particular, evaluating 15 diverse LLMs on 4 standardized clinical error-detection test sets across 3 languages, we find that 13 of 15 models fall below the level of random pairwise discrimination, even while achiev- ing F1 scores that standard practice would read as moderate. We also observe that the underlying bias patterns differ across languages: the same model can default to “no er- ror” on one language and over-flag errors on another. To diagnose where discrimination breaks down, we further introduce a procedure to score the evidence models cite in their outputs. We find that while models consistently locate error-relevant content, they fail to produce the corresponding correct verdict on the clean counterpart. Finally, we show that F1 and pairwise accuracy are driven in opposite directions by the same underlying bias, so that ranking models by F1 may systematically promote the weakest discriminators. For safety-critical clinical NLP applications, we advocate for supplementing aggregate metrics with paired evaluations in benchmark reporting. Code and analysis scripts are available at https://github.com/healthylaife/paired-clinical-eval. 1. Introduction Medical errors are a substantial and persistent source of preventable harm in US healthcare, with widely cited estimates ranging from tens of thousands to several hundred thousand deaths per year (Kohn et al., 2000; Makary and Daniel, 2016). A substantial portion of these errors originates in or propagates through clinical documentation: incorrect diagnoses recorded in progress notes, inappropriate treatment plans, or misidentified pathogens that cascade into downstream care decisions (Ben Abacha et al., 2024). Manual review of clini- cal notes is time-consuming, inconsistent, and does not scale with growing documentation volumes. Automated detection using language models to flag potential errors is therefore of direct clinical interest. From a machine learning perspective, clinical error detection is a particularly demanding task. Unlike medical question answering, where a model selects among predefined options, error detection requires the model to evaluate clinical reasoning. The model must read a note, understand the implicit chain from symptoms through diagnosis to management, and © 2026 Y. Zhang 1 & R. Beheshti 1 .Author version, August 2026. arXiv:2608.16643v1 [cs.CL] 17 Aug 2026 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Figure 1: Paired clinical error detection. (a) Each clinical scenario produces an error- injected note and a correct note differing by one sentence. (b) A model’s predic- tions on the pair fall into four mutually exclusive outcomes. judge whether that reasoning is sound. This demands not only medical knowledge but also the ability to distinguish incorrect clinical decisions from decisions that are unconventional but clinically appropriate. Recent work has established benchmarks for this task (MEDEC (Ben Abacha et al., 2025), MedRECT (Iwase et al., 2025), and MedErrBench (Ma et al., 2026)), with top systems relying on large proprietary models with retrieval augmentation (Corbeil, 2024) and prompt optimization (Toma et al., 2024). How and where LLMs fail at this task remains largely unexamined. LLMs are appealing for clinical deployment because they can be hosted locally, avoiding the need to transmit patient data to external vendors. Standard evaluation suggests that such models perform reasonably well on clinical error detection. Yet aggregate metrics evaluate each note in isolation and can be inflated by bidirectional prediction bias: a model that always predicts error (yes-bias) or always predicts no error (no-bias) can achieve high aggregate scores without distinguishing erroneous notes from correct ones. The paired structure of clinical error detection datasets makes this problem visible. Clinical scenarios included in these benchmarks produce two notes that differ by a single sentence: one with an injected error and one without (Figure 1). A model that truly dis- criminates should correctly identify the error in the first note and clear the second; a biased model assigns the same label to both, revealing that its aggregate performance reflects a default tendency rather than clinical understanding. The separation of true discrimina- tion from response bias is long established in signal detection theory (Green and Swets, 1966), and evaluation on minimally contrastive samples is well established through contrast sets (Gardner et al., 2020). To the best of our knowledge, clinical error detection has not adopted pairwise evaluation. Based on this principle, we define the Both-Correct Rate (BCR), which adapts contrast consistency to clinical error detection by requiring correct classification of both members of each pair. To diagnose where discrimination breaks down, we introduce Evidence Con- 2 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection trastive Analysis (ECA), which checks whether the evidence a model cites on a failed pair overlaps with the ground-truth error and correction sentences. The framework applies to any (correct, corrupted) pair, including substitution, insertion, and omission; our experi- ments focus on substitution-form errors because these are the errors released by the public paired benchmarks, a data limitation discussed in Section 5.3. We apply this framework to 15 LLMs across four paired clinical error-detection datasets in three languages, with four prompt configurations per model. We assess model outputs through three comprehensive strategies: aggregate pointwise metrics, pairwise BCR, and ECA on the dominant failure mode. In this paper, we make three contributions: • Discrimination failure is pervasive despite adequate aggregate performance. We show that the vast majority of tested models fall below 25% random pairwise dis- crimination despite achieving reasonable aggregate scores. The underlying prediction bias is bidirectional and language-dependent: the same model can exhibit yes-bias on one language and no-bias on another. • Models locate error-relevant evidence but cannot judge it. Through ECA, we show that models locate error-relevant text yet produce the same verdict on both members of the pair. In the majority of yes-bias failures, the model locates the error sentence on the erroneous note, yet labels both as erroneous. • Standard metrics are structurally misleading on paired clinical data. We show that F1 and BCR can be driven in opposite directions by the same underlying prediction bias, so that the models ranked highest by F1 are typically ranked lowest by BCR. 2. Related Work 2.1. Error Detection in Clinical Texts The need for safer clinical documentation has motivated a series of benchmarks for au- tomated error detection. The MEDIQA-CORR 2024 shared task formalized clinical error detection as a community challenge (Ben Abacha et al., 2024), with MEDEC providing a larger public corpus (Ben Abacha et al., 2025). Subsequent work expanded the scope to multilingual evaluation (Iwase et al., 2025; Ma et al., 2026). Despite growing benchmark diversity, the evaluation paradigm remains focused on comparing aggregate accuracy across models and strategies. Standard benchmarks can create an evaluation illusion that obscures real-world clinical limitations (Agrawal et al., 2025), and retrieval proficiency does not nec- essarily predict operational success (Kanithi et al., 2026). Whether high scores on clinical error detection benchmarks reflect discriminative ability rather than systematic prediction bias remains unexamined. This paper addresses that gap by applying pairwise evaluation to clinical error detection, adapting the contrast consistency principle (Gardner et al., 2020) to the paired structure already provided by MEDEC-style datasets. 3 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection 2.2. Prediction Bias in Healthcare Systematic prediction bias, defined as the tendency to default to a single output class re- gardless of the input, is well-documented in LLMs. Sycophancy, in which models align with perceived user expectations (Sharma et al., 2024), represents one mechanism that can produce such bias; in clinical error detection, it manifests as yes-bias, defaulting to “error present.” In the medical domain specifically, LLMs exhibit heightened susceptibility to cog- nitive biases, with less capable models showing larger accuracy degradation under biased prompting (Schmidgall et al., 2024). In parallel, a growing body of work has examined de- mographic bias in medical LLMs, showing that outputs in clinical decision support shift with patients’ protected attributes and that prompt phrasing modulates the observed patterns (Poulain et al., 2026); such evaluations have been scaled through automatically generated, evidence-grounded test cases (Fayyaz et al., 2024), and Adiba et al. (2025) surveys the area. This demographic line of work concerns which patients a model treats differently, whereas prediction bias concerns which output class a model defaults to. Despite these findings, prior work has predominantly characterized prediction bias as unidirectional, as- suming models consistently favor one class. Whether the bias direction is stable across languages and prompt configurations has not been investigated. 2.3. Pairwise Evaluation and Representation Gaps Signal detection theory separates a system’s ability to discriminate between classes from its response bias toward one class (Green and Swets, 1966). Because measured accuracy reflects both, response bias alone can produce high accuracy when the class prior is skewed or evaluation is restricted to a single class. A parallel concern motivates minimal-pair evaluation (Warstadt et al., 2020) and contrast sets (Gardner et al., 2020), which evaluate models on minimally contrastive inputs to reveal whether they have learned the relevant distinction or merely exploit superficial correlations. Despite the paired structure built into these datasets (Ben Abacha et al., 2024; Iwase et al., 2025; Ma et al., 2026), clinical NLP benchmarks have not adopted contrastive evaluation. A separate line of work has shown that LLMs can encode correct information internally while producing incorrect outputs. Probing studies demonstrate that truthfulness is repre- sented in hidden states even when generated text is wrong (Burns et al., 2023; Orgad et al., 2025). Separately, chain-of-thought explanations can misstate the factors actually driving a prediction (Turpin et al., 2023). Probing requires weight access, which is unavailable for the closed models that dominate clinical deployment. Whether correct evidence can surface in a model’s generated text alongside an incorrect verdict has not been systematically studied. 3. Methods Pairing is the organizing principle of our evaluation (Figure 1): each error-injected note has a correct counterpart from the same clinical scenario, making the pair, not the single note, the unit at which true discrimination can be separated from response bias. Building on this, we develop three parallel evaluation layers (Figure 2): traditional pointwise metrics (§3.4.1), the Both-Correct Rate at the pair level (§3.4.2), and Evidence Contrastive Analysis within failed pairs (§3.4.3). We apply this framework to 15 instruction-tuned LLMs across four 4 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection test sets in three languages, under four prompt–decoding configurations per model–dataset combination. Figure 2: Our framework for evaluating LLMs in clinical error detection tasks. Four paired clinical error-detection test sets across three languages, LLMs across three size tiers, and four prompt-decoding configurations are used. Each model’s struc- tured per-note output is then assessed through three parallel evaluation layers: traditional pointwise metrics, pairwise Both-Correct Rate (BCR), and Evidence Contrastive Analysis (ECA). 3.1. Datasets and Task We apply our framework to four clinical error detection test sets spanning three languages (Table 1). MEDEC MS-Test (MS-Test) is drawn from the MEDEC MS corpus (Ben Abacha et al., 2025); MedErrBench-EN (MEB-EN) and MedErrBench-CN (MEB-CN) are the En- glish and Chinese test sets of MedErrBench (Ma et al., 2026); and MedRECT-JA (MRT-JA) is the Japanese test set of MedRECT, with nine error types (Iwase et al., 2025). In each dataset, error-injected notes are paired with correct counterparts from the same clinical scenarios. Whether these pairs are released explicitly varies by dataset, and not every released note participates in a matched pair; we apply dataset-specific procedures to identify usable pairs, with details in Appendix A. MedRECT-JA warrants a caveat: its 190 error notes share only 105 unique clean notes, a many-to-one structure that may inflate within-pair error correlation (Appendix A). We restrict evaluation to the binary error-flag sub-task: given a single clinical note, de- cide whether it contains a medical error, thereby measuring pairwise discrimination without conflating detection with correction. Notes are evaluated independently; the model never sees a pair side-by-side, so pairwise discrimination must be inferred from each note’s content alone, with pairing applied post-hoc to compute BCR. 3.2. Models We evaluate 15 instruction-tuned LLMs (Appendix C, Table 4) across three size tiers (3–8B, 27–32B, and 70B) and five model families; five are medical-domain, the other ten general- 5 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Table 1: Datasets used for inference and evaluation. Error and Clean are the sub-counts of notes with and without errors, respectively; Pairs is the number of matched (error, correct) tuples used for BCR evaluation, which can differ from min(Error, Clean) when notes lack a counterpart or share counterparts across pairs. DatasetSamplesErrorCleanPairsLanguageSource MS-Test597311286286EnglishBen Abacha et al. (2025) MEB-EN208104104104EnglishMa et al. (2026) MEB-CN200100100100ChineseMa et al. (2026) MRT-JA295190105190JapaneseIwase et al. (2025) purpose. This grid enables two comparisons: scale within a family (e.g., Qwen 3 at 4B, 8B, and 32B) and medical specialization at matched scale (e.g., Gemma 3-27B vs. MedGemma 27B). Clinical sites adopting LLMs for error detection are unlikely to curate task-specific few- shot demonstrations, and a fair comparison across model families with different training paradigms requires a common starting point. We therefore evaluate all models zero-shot, without task-specific examples or fine-tuning. Precision. Models at 27B parameters and below are loaded in native bfloat16 (bf16); the two 70B models are loaded in 8-bit floating point (fp8) via vLLM (Kwon et al., 2023) due to memory constraints, which may slightly reduce their performance relative to full precision. 3.3. Prompt Design and Decoding We treat prompt and decoding choices as a 2×2 perturbation matrix that probes two or- thogonal sources of variability: whether instruction wording can shift prediction bias, and whether sampling noise drives the observed patterns. Reporting the cross-configuration mean and standard deviation across the four resulting configurations lets us separate sta- ble model behavior from configuration-driven artifacts, a robustness check that single- configuration benchmarks cannot provide. The neutral prompt instructs the model to act as a skilled medical doctor performing a standard clinical review. The conservative prompt adds an explicit instruction to prefer “no error” when the evidence is ambiguous, testing whether directed caution can shift prediction bias without improving discrimination. Both prompts require a structured four-line output (Evidence, Analysis, Confidence, Error:Yes/No), providing both the binary verdict for evaluation and the textual evidence used by ECA (Section 3.4.3). Dataset-specific error type lists are provided in the native language of each dataset. For decoding, we use greedy decoding as the primary setting because it yields fully reproducible outputs and isolates bias effects from sampling noise, and complement it with stochastic sampling to test whether sampling variability shifts the observed bias patterns. Full prompt templates and exact decoding parameters appear in Appendix D. 6 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection 3.4. Evaluation Framework 3.4.1. Traditional Metrics We report balanced accuracy, F1, precision, recall, specificity, and the Matthews correlation coefficient (MCC). We additionally report the error-flag rate: the fraction of notes for which the model outputs Error:Yes. Balanced accuracy and MCC serve as primary anchors because they account for class imbalance and penalize degenerate classifiers, respectively. F1 and recall are reported to expose the discrepancy between bias-sensitive and bias-resistant metrics. 3.4.2. Both-Correct Rate (BCR) BCR adapts the contrast consistency principle (Gardner et al., 2020) to clinical error detec- tion. For each pair (x e , x c ) consisting of an error-injected and a correct note from the same clinical scenario, the model’s predictions (ˆy e , ˆy c ) fall into four mutually exclusive categories. We use the shorthand Pred1 (both predicted erroneous) to denote pairs where the model flags both members as containing an error, and Pred0 (both predicted non-erroneous) for pairs where the model judges both error-free. These two outcomes correspond to systematic yes-bias and no-bias failure modes, respectively. 1. Both Correct (BC): ˆy e = 1 and ˆy c = 0. Correct on both members of the pair. 2. Pred1 : ˆy e = 1 and ˆy c = 1. Both members predicted as containing an error (yes-bias). 3. Pred0 : ˆy e = 0 and ˆy c = 0. Both members predicted as error-free (no-bias). 4. Both Wrong (BW): ˆy e = 0 and ˆy c = 1. Inverted judgment on both members. BCR is the fraction of pairs classified as Both Correct: BCR = 1 N N X i=1 1[ˆy e,i = 1 ∧ ˆy c,i = 0].(1) The population counterpart of Equation 1 isE[BCR] = P (ˆy e = 1, ˆy c = 0 | y e = 1, y c = 0): the joint probability that a random pair is classified correctly on both members. Under the null hypothesis that within-pair predictions are conditionally independent given the pair labels, with each prediction depending only on its own note, this joint factorizes into marginals, giving E[BCR] indep = P (ˆy e = 1| y e = 1)· P (ˆy c = 0| y c = 0) = sensitivity· specificity.(2) We define the independence ratio as R independence = BCR E[BCR] indep ,(3) which measures how far observed BCR departs from the independence baseline implied by the model’s own marginal sensitivity and specificity. A consistent R independence < 1 indicates that within-pair prediction outcomes are systematically dependent: joint success on both 7 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection members of a pair occurs less often than within-pair independence would predict. We emphasize that our contribution is this diagnostic apparatus—the independence ratio, which isolates within-pair dependence that contrastive accuracy alone does not expose, together with ECA (Section 3.4.3), which localizes where discrimination breaks down—rather than the BCR statistic itself, which is a direct adaptation of contrast consistency (Gardner et al., 2020). Beyond this stochastic baseline, BCR obeys a deterministic upper bound. Because a Both-Correct pair requires ˆy e = 1 and ˆy c = 0, the Both-Correct event is contained in each single-note success event, so BCR≤ min(sensitivity, specificity).(4) Any class bias therefore caps BCR at the weaker marginal, however high F1 climbs through recall on the favored class, so the F1–BCR divergence in Section 4.5 is structural rather than model-specific. A short proof and an always-error corollary (F1 = 2/3 with BCR = 0 on balanced data) are given in Appendix B. For a random classifier with sensitivity = specificity = 0.5, Equation 2 yields a 25% BCR baseline; this figure is the balanced special case and applies only to the three class- balanced pair sets (MS-Test, MEB-EN, MEB-CN). On MedRECT-JA (190 of 295 notes contain errors), a random predictor calibrated to the empirical class prevalence (outputting ˆy = 1 with probability 190/295 independent of the input) has sensitivity = 190/295 and specificity = 105/295, giving an independence baseline of (190/295)(105/295) ≈ 22.9%. The general, balance-free reference is therefore the independence ratio R independence (Equa- tion 3) rather than any fixed percentage. We report the four-configuration mean± standard deviation as the primary BCR measure; cross-configuration variation is informative in itself about prompt sensitivity. 3.4.3. Evidence Contrastive Analysis (ECA) BCR reveals whether models discriminate between paired samples, but not which stage of their reasoning fails: locating the relevant evidence or judging it correctly once located. Evidence Contrastive Analysis (ECA) is a post-hoc diagnostic procedure that, for each pair, scores whether the cited Evidence field overlaps the clinically relevant sentence on the error note (TP localization) and on the correct note (FP evidence-hit ). Overlap is scored by sub-string containment or ≥ 60% word coverage (threshold sensitivity in Appendix E), adapting the rationale token-overlap convention (DeYoung et al., 2020), with character- level counting for Chinese and Japanese where word boundaries are ill-defined. Combining these two indicators across the pair yields the five mutually exclusive categories shown in Figure 3, which separate attention failures from judgment failures. 3.4.4. Parse Failure Handling Although our prompts specify a strict four-line output format, not all models achieve perfect compliance: some outputs add extra text, omit the Error field, or produce a malformed structure, leaving no parseable verdict. Pairs where either member cannot be parsed are excluded from BCR computation. To verify that this exclusion does not bias the results, 8 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Figure 3: Evidence Contrastive Analysis (ECA). (a) The two indicators, illustrated on a shared clinical scenario: TP localization asks whether the cited evidence on the error note (Streptococcus pneumoniae) overlaps the error sentence; FP evidence- hit asks the same question on the correct note (Haemophilus influenzae). (b) The five outcomes arising from combining the two indicators across a pair. Both-Hit (A) isolates a judgment failure: the model attends to the correct location in both notes, yet still cannot determine which version is incorrect. Neither-Hit (D) reflects full attention failure. TP-Only (B) and FP-Only (C) are partial-attention failures. Extraction-Fail (E) captures pairs with unparseable Evidence fields. we compare against a random-assignment imputation that fills in unparseable verdicts us- ing the model’s overall accuracy (Appendix E). Parse failures are themselves captured as Extraction-Fail (E) in ECA (Section 3.4.3). 4. Results 4.1. Traditional Metrics Suggest Moderate Performance Across all 240 runs (15 models × 4 datasets × 4 configurations), balanced accuracy ranges from 0.45 to 0.71, with MCC near zero for most models, indicating performance close to chance at distinguishing error-containing notes from correct ones. F1 scores, however, range up to 0.80: in 178 of 240 runs (74%), F1 exceeds 0.5 while balanced accuracy stays at or below 0.6. Under standard evaluation practice, these F1 scores would suggest that most models achieve moderate performance on clinical error detection. Full per-run metrics for all 240 runs appear in Appendix G.1 (Tables 11–14). The pairwise analysis that follows reveals that this apparent performance does not reflect discriminative ability at the pair level. 4.2. Pairwise BCR Reveals Low Discrimination Table 2 presents the four-configuration mean BCR for each model–dataset combination. Thirteen of 15 models fall below 25% mean BCR across datasets, the level expected from 9 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection a random classifier on balanced data. Only Qwen 3-32B (28.0%) and UltraMedical 70B (25.7%) exceed this threshold, and both still fail to correctly classify both members of a pair more than 70% of the time. The 25% baseline is the balanced special case (for prevalence-imbalanced MRT-JA it is 22.9%; the balance-free reference is the independence ratio of Equation 3, analyzed in Section 4.5). The Sens and Spec columns of Table 2 make the mechanism concrete: top- ranked models pair moderate sensitivity with moderate specificity, whereas the lowest-BCR models drive one marginal to an extreme (e.g., Gemma 3-4B at 94.2% sensitivity but 7.2% specificity), exactly the configuration a single aggregate score conceals. Table 2: Four-configuration mean BCR (%) ± standard deviation by model and dataset. Sens and Spec are the model’s mean pointwise sensitivity and specificity, averaged over all 16 runs; the highest-BCR models pair moderate sensitivity with moderate specificity, while the lowest-BCR models collapse one marginal toward its extreme. † Medical-domain model. Sorted by cross-dataset mean BCR. BCR (%)Marginal (%) ModelMS-Test MEB-EN MEB-CN MRT-JA Mean SensSpec Qwen 3-32B22.6±3.7 30.8±11.132.0±5.5 26.6±1.928.0 69.451.4 UltraMed-70B†24.3±9.843.7±2.2 16.8±10.0 18.0±5.725.7 47.073.5 MedGemma 27B† 20.3±4.929.1±8.614.6±5.7 28.5±4.223.1 61.256.0 Llama 3.1-70B20.2±3.6 27.1±15.615.3±2.8 17.4±7.020.0 63.651.4 UltraMed-8B†19.6±5.828.5±2.518.4±5.5 13.5±1.820.0 43.465.0 Qwen 3-4B12.0±5.913.2±6.623.0±1.2 19.9±1.917.0 83.527.8 MedGemma 4B†16.0±3.816.6±6.614.8±2.7 12.5±6.615.0 66.538.6 Llama 3.1-8B14.1±7.5 16.9±13.114.6±4.6 12.8±7.414.6 70.934.5 Phi-4-mini13.9±7.712.7±5.67.5±4.8 16.5±1.512.7 86.118.4 Qwen 3-8B11.2±3.213.0±1.48.0±3.2 16.1±4.612.1 56.449.4 MediPhi 4B†11.5±6.510.3±4.611.2±6.19.7±2.110.7 87.415.4 Gemma 3-27B10.8±5.79.9±8.413.2±8.37.1±3.310.2 85.221.0 Llama 3.2-3B5.9±3.18.4±4.216.3±7.7 10.0±7.110.2 84.117.5 Mistral 7B4.5±3.45.0±3.66.3±5.4 12.8±6.47.1 88.810.9 Gemma 3-4B5.5±3.82.2±2.44.2±3.36.3±1.14.6 94.27.2 Two secondary trends emerge within this pattern. First, scaling within model fami- lies yields modest gains: Qwen improves from 17.0% (4B) to 28.0% (32B), though non- monotonically (12.1% at 8B), and Llama from 10.2% (3B) to 20.0% (70B). However, these gains are concentrated on English datasets; for example, Llama 3B-to-70B gains 18.7 p on MedErrBench-EN but decreases by 1.0 p on MedErrBench-CN. Second, medical-domain pretraining often provides larger gains than scaling alone: Gemma 3-27B (general-purpose) achieves 10.2% mean BCR, while MedGemma 27B (medical) reaches 23.1%, more than dou- bling its general-purpose counterpart. This holds for four of five matched general–medical pairs; MediPhi 4B (10.7%) is the exception, underperforming Phi-4-mini (12.7%). Cross-configuration standard deviations range from 1.1 p to 15.6 p across individual model–dataset entries. Llama 3.1-70B is the most unstable overall. On MedErrBench-EN, its BCR ranges from 9.0% to 45.2% across the four configurations, a spread of 36.2 p. 10 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Table 3: Mean error-flag rate per model–dataset combination.Pink cells indicate yes-bias (rate ≥ 0.6);blue cells indicate no-bias (rate < 0.4); unshaded cells are balanced (0.4≤ rate < 0.6). A check in the Shift column marks models whose bias category changes across datasets. † Medical-domain model. ModelMS-Test MEB-EN MEB-CN MRT-JA Shift? Llama 3.2-3B.94.88.63.89 Gemma 3-4B.94.98.89.93 Qwen 3-4B .81.91.61.80 MedGemma 4B†.60.66.50.80✓ MediPhi 4B†.84.89.85.86 Phi-4-mini .79.89.88.81 Mistral 7B.95.96.93.72 Llama 3.1-8B.67.78.43.85✓ UltraMed-8B†.42.55.32.27✓ Qwen 3-8B.50.60.36.69✓ Gemma 3-27B.67.92.77.92 MedGemma 27B†.45.60.43.65✓ Qwen 3-32B.62.78.43.56✓ Llama 3.1-70B.50.78.41.57✓ UltraMed-70B†.36.59.20.33✓ 4.3. Bias Is Bidirectional and Language-Dependent The bias patterns underlying these BCR results are not uniform across languages. Table 3 reports the mean error-flag rate for each model–dataset combination, along with the result- ing bias category. Eight of 15 models change bias category across datasets, and for seven of these, the mean error-flag rate falls on opposite sides of parity on different datasets; Qwen 3-8B reverses outright, from no-bias on Chinese (0.36) to yes-bias on Japanese (0.69). UltraMedical 8B illustrates the more common pattern: balanced prediction rates on English datasets (0.42–0.55) shifting to strong no-bias on Chinese and Japanese (0.27–0.32). Several models also switch bias direction outright between prompt variants: on at least one dataset, seven of 15 models show yes-bias under neutral prompting but no-bias under conservative prompting (per-configuration error-flag rates in Appendix G, Tables 11–14), so conservative prompting does not simply shift prediction rates toward balance. 4.4. The Localization-Judgment Gap Having established that discrimination fails and that yes-bias (Pred1 ) is the dominant failure mode across our experiments (for the two no-bias models, UltraMedical 8B and 70B, Pred0 dominates instead), we next examine whether models nonetheless attend to error-relevant content in their outputs. We focus ECA on Pred1 pairs, which concentrate the bulk of failures and therefore offer the greatest statistical power for diagnosing what happens when models fail pairwise discrimination while still producing reasoning evidence. 11 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection TP localization measures whether the evidence cited by the model on the error note overlaps with the actual error sentence. Random baselines, corresponding to a sentence- level uniform pick, are the inverse of the average number of sentences per note (12.6% for MS-Test, 9.3% for MEB-EN, 27.7% for MEB-CN, 10.1% for MRT-JA). On MS-Test, all 15 models exceed the random baseline under the four-configuration mean (Figure 4a). Averaged across the 15 models, per-dataset TP localization rates are 87% on MEB-EN, 70% on MEB-CN, 69% on MS-Test, and 42% on MRT-JA (full per-model values in Appendix G.2, Table 15). 020406080100 TP Localization (%) Llama3.2-3B Llama3.1-8B UltraMed-70B UltraMed-8B Llama3.1-70B MediPhi-4B MedGemma-4B Mistral-7B Phi4-mini-4B MedGemma-27B Gemma3-4B Gemma3-27B Qwen3-8B Qwen3-4B Qwen3-32B (a) TP Localization on Pred1 Pairs General-purpose Medical-domain Random baseline (12.6%) 020406080100 Proportion of Pred1 pairs (%) Llama3.2-3B Llama3.1-8B UltraMed-70B UltraMed-8B Llama3.1-70B MediPhi-4B MedGemma-4B Mistral-7B Phi4-mini-4B MedGemma-27B Gemma3-4B Gemma3-27B Qwen3-8B Qwen3-4B Qwen3-32B (b) Evidence Contrastive Analysis (ECA) Composition A: Both hit B: TP only C: FP only D: Both miss E: Extraction fail Figure 4: The localization-judgment gap on MS-Test (four-configuration mean across 15 models). (a) TP localization per model; dashed line is the random baseline (12.6%). (b) ECA category composition per model, as percentage of Pred1 pairs; category definitions in Figure 3. The ECA category breakdown (Figure 4b) shows that correct localization on the error note coexists with pair failure. On MS-Test, Both-Hit (A) and TP-Only (B) together, which both indicate that the model attended to the error sentence on the error note, account for 46–87% of categorized Pred1 pairs across models (mean 70%); the model thus locates the right content in most failed pairs, yet still misjudges them. Within this, Both-Hit (A) alone captures the purest form of the localization-judgment gap: the model attends to the relevant sentence on both notes yet judges both as erroneous (26–49% of Pred1 pairs, mean 38%). Neither-Hit (D), which represents full attention failure, accounts for only 18% on average. 4.5. Prediction Bias Mediates the Relationship Between F1 and BCR The preceding sections raise a question: if models fail at pairwise discrimination, why do traditional metrics suggest otherwise? Figure 5 addresses this by examining how prediction 12 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection 0.20.30.40.50.60.70.8 F1 Score (4-config mean) 0 10 20 30 40 Actual BCR (%, 4-config mean) BCR = 25% (random baseline) F1 = 0.6 (moderate) DECEPTION F1 0.6, BCR < 25% True Discrimination True Failure (a) F1 vs Actual BCR (32/60 in deception zone) Model size Small (3-8B) Large (27-32B) Very Large (70B) 01020304050 Expected BCR = Sensitivity × Specificity (%) 0 10 20 30 40 50 Actual BCR (%) (b) Actual vs Expected BCR (mean ratio 0.73×) Dataset MS-Test MEB-EN MEB-CN MRT-JA Independence line Figure 5: Metric deception across 60 model–dataset entries (four-configuration means). (a) Shaded region indicates F1 ≥ 0.6 with BCR < 25%. (b) Actual BCR vs. expected BCR under statistical independence (Equation 2); the dashed line is the independence baseline (R independence = 1), below which all 60 entries fall. bias relates to both F1 and BCR. We operationalize prediction bias as the error-flag rate (Section 3.4.1). The error-flag rate simultaneously inflates F1 (r = +0.85 across 60 model–dataset en- tries) and suppresses BCR (r = −0.49), while the direct F1–BCR correlation is near zero (r = −0.06). The two mechanisms pull F1 and BCR in opposite directions, so the direct F1–BCR association is dataset-dependent rather than uniform: it is negative on MS-Test (r = −0.66), MEB-CN (r = −0.14) and MRT-JA (r = −0.23), positive on MEB-EN (r = +0.31), and averages to near zero once the four datasets are pooled (Appendix F, Table 8). This opposition is the empirical face of the structural bound of Equation 4: a collapsed marginal caps BCR while F1 is free to rise. The practical consequence is visible in the rankings rather than in the pooled correlation: on three of four datasets, the top-3 models by F1 and the top-3 by BCR share zero overlap (Appendix F, Table 9). Thirty-two of 60 entries (53%) fall in a deception zone where F1≥ 0.6 but BCR < 25% (Figure 5a). On MS-Test, the three models with the highest F1 (Gemma 3-4B, Llama 3.2- 3B, Mistral 7B, all with F1 > 0.65) have the three lowest BCR values (4.5–5.9%). The independence ratio (Equation 3) averages 0.73 across the 60 model–dataset entries, with all 60 falling below the independence line (Figure 5b), confirming that within-pair er- rors are systematically correlated. Per-dataset ratios range from 0.62 on MEB-CN (strongest correlation) to 0.84 on MEB-EN (closest to independence); full details in Appendix F. A proprietary reference point. Our systematic evaluation covers open-weight mod- els, which are the systems that most healthcare organizations can deploy on-premise for PHI reasons. To situate that range against a frontier proprietary system, we addition- ally ran GPT-5 mini 1 on MS-Test within our available budget. As a reasoning model, 1. OpenAI API model gpt-5-mini-2025-08-07. 13 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection it does not accept a temperature setting, so we vary only the prompt (neutral and con- servative), running two runs. Its BCR of 42.3% (F1 = 0.66, balanced accuracy = 0.69) exceeds every open-weight model on MS-Test (best 24.3%, UltraMedical 70B) and sits outside the deception zone; its observed BCR still falls below its own independence base- line of 47.1% (sensitivity × specificity), consistent with the within-pair dependence seen throughout. Training data is undisclosed for GPT-5 mini and open-weight models alike, so benchmark exposure cannot be ruled out for either. Because contamination inflates rather than depresses scores, it bears on a strong result like this one, not on our main finding of pervasive failure of discrimination. We read this as one illustrative data point, not a full proprietary evaluation. 5. Discussion 5.1. Implications for Clinical Model Selection Thirteen of 15 models fall below 25% random BCR (Section 4.2), yet many of these same models achieve F1 scores above 0.6. F1-based model selection would therefore systematically favor the weakest discriminators. In a clinical setting, evaluating candidate models by F1 on MS-Test, Gemma 3-4B, Llama 3.2-3B, Mistral 7B would rank as the top three, precisely the models with the lowest BCR (4.5–5.9%). This is not a hypothetical concern but the default outcome of current evaluation practice. This vulnerability is specific to the accuracy regime in which clinical error detection operates. At high balanced accuracy (e.g., ≈90%), bias has limited room to inflate F1, and BCR is mathematically constrained to be high. Clinical error detection, however, operates at ≈50–60% balanced accuracy, where a model can achieve F1 ≈ 0.68 by always predicting a single class. In this regime, pairwise evaluation is not a refinement but the only way to separate discriminative ability from systematic bias. Many clinical NLP tasks operate at similar accuracy levels and face the same risk. This concern is consistent with a growing literature showing that aggregate confusion-matrix metrics can reward degenerate prediction strategies, remain high under class imbalance or weak discrimination, and are formally improper as decision measures (Lipton et al., 2014; Reinke et al., 2024; Van Calster et al., 2025). That few model–dataset entries achieve both high F1 and high BCR is an empirical finding, not a mathematical necessity. If models achieved high F1 through correct pairwise classification rather than bias, the two metrics would correlate positively. The almost empty upper-right quadrant of Figure 5a confirms that, across most models tested, high F1 is largely achieved through bias alone. For safety-critical clinical NLP, paired metrics such as BCR should be reported alongside traditional metrics. Concretely, paired evaluation serves three points in the deployment pipeline. Bench- mark designers can report BCR and the independence ratio next to F1 and MCC, so that published rankings reflect discrimination rather than a default class tendency. Healthcare- organization governance and procurement teams, who are often limited to open-weight, on- premise models by PHI and data-governance rules, can screen candidate models on paired data as a gate before adoption, catching systems that pass on aggregate metrics but fail pairwise. Vendors can run the same check before release. This screen is a necessary first step, not a replacement for prospective, workflow-level evaluation with clinicians in the 14 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection loop, nor for broader error-type coverage (Section 5.3). What it establishes is a precondi- tion: whether a model can tell an erroneous note from its clean counterpart at all. Most of the models we tested cannot. 5.2. Evidence Production and Clinical Judgment The localization-judgment gap documented in Section 4.4 is consistent with prior work showing that models encode correct information internally but fail to express it (Burns et al., 2023; Orgad et al., 2025). Turpin et al. (2023)further show that generated explanations can misstate the factors driving a prediction. What distinguishes our finding is that the correct evidence appears directly in the model’s generated output: in Both-Hit (A) pairs, which average 38% of Pred1 failures on MS-Test (26–49% across models), the model cites the error sentence on the error note and the corresponding sentence on the correct note, yet predicts error on both. The failure is thus detectable from model outputs alone, without access to internal representations. ECA is deliberately a localization (evidence-citation) diagnostic and not a test of comprehension—we do not claim the model understands the sentence it cites—and the localization finding is robust to how overlap is measured: an orthogonal embedding-based Recall@1 criterion reproduces both the magnitude and the per-model ranking of TP localization (Appendix E, Table 7). This separation between evidence production and clinical judgment suggests two direc- tions for intervention. First, contrastive fine-tuning, which trains models to distinguish paired samples, could target the judgment component while preserving the localization ability models already demonstrate. Second, pipeline architectures that decompose the task into localization followed by judgment may better match the model’s existing capa- bility structure, in which localization is relatively strong (Both-Hit and TP-Only together account for a mean of 70% of Pred1 pairs on MS-Test) but judgment is weak. Until discriminative ability improves, the high recall of yes-biased models may support pre-filtering workflows with mandatory human review, provided the base-rate of errors is high enough that precision remains workable and alert fatigue is monitored: the model flags candidate errors at high sensitivity, and clinicians provide the judgment that the model lacks. Standalone deployment for clinical error detection may not be supported by current evidence. 5.3. Limitations All evaluations use zero-shot prompting with two prompt variants and, for sampling-based decoding, a single inference run per configuration. Few-shot or task-specific fine-tuning may improve discrimination; however, zero-shot reflects the most realistic deployment scenario. Our primary measure, the four-configuration mean, is anchored by two fully reproducible greedy-decoding runs, which mitigates single-run sampling variance. Conservative prompt- ing shifts BCR by up to 12.8 p in either direction (Figure 6), yet at least 13 of 15 models stay below the 25% balanced-random level under either prompt alone, so prompt-level in- terventions redistribute errors rather than close the discrimination gap. Our systematic evaluation covers open-weight models up to 70B parameters, using a single proprietary reference point (GPT-5 mini on MS-Test, Section 4.5) rather than a full proprietary sweep. This scoping reflects both computational and budget constraints 15 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection and a methodological concern: proprietary models do not disclose their training data, so we cannot verify whether the benchmark source datasets appeared during their training; the open-weight models we evaluate are not fully immune to this data-leakage risk, but their training-data documentation at least allows partial verification. More importantly, the 3B–70B range is sufficient to establish that discrimination failure is pervasive and that scaling alone provides limited gains. Our contribution is diagnostic: we aim to characterize where and why pairwise discrimination fails, not to produce a ranking of models for clinical deployment. A further scope limitation is the error type. Clinical errors also include insertions and omissions, but the public benchmarks with released pairs inject substitution-form errors, so our empirical results are confined to that form. The paired machinery extends directly once such data exist: an inserted false statement is scored as the error note against the note without it, and an omitted finding as the incomplete note against the complete one, with ECA localizing insertions on the present text and omissions on the complete-note side. Evaluating BCR on paired insertion and omission data is the immediate next step this framework enables. ECA substring matching was designed for single-sentence errors; MedRECT-JA’s multi- sentence structures reduce matching granularity, and TP localization may partly reflect entity salience rather than true error localization. These factors affect the granularity of the diagnosis but do not undermine the core observation. All 15 models exceed their respective random baselines, and the localization-judgment gap is consistently observed across all four benchmarks. Our analysis also does not investigate the internal mechanisms that produce this gap; understanding why models fail to convert correct evidence into correct verdicts is an important next step that the diagnostic framework presented here should enable. Acknowledgments Our work was partly supported by NSF awards 2443639 and 2552481, and NIH awards, P20GM103446 and U54GM104941. 16 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection References Farzana Islam Adiba, Yifan Zhang, and Rahmatollah Beheshti. Bias and fairness in medical LLMs: An extensive scoping review. OSF Preprints, 2025. doi: 10.31219/osf.io/fqejhv1. Monica Agrawal, Irene Y. Chen, Freya Gulamali, and Shalmali Joshi. The evaluation illusion of large language models in medicine. npj Digital Medicine, 8(1):600, 2025. doi: 10.1038/s41746-025-01963-x. Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Fei Xia, and Meliha Yetis- gen. Overview of the MEDIQA-CORR 2024 shared task on medical error detection and correction. In Proceedings of the 6th Clinical Natural Language Processing Workshop (ClinicalNLP 2024), pages 596–603, 2024. doi: 10.18653/v1/2024.clinicalnlp-1.57. Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. MEDEC: A benchmark for medical error detection and correction in clinical notes. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22539–22550, 2025. doi: 10.18653/v1/2025.findings-acl.1159. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id= ETKGuby0hcs. Jean-Philippe Corbeil. IryoNLP at MEDIQA-CORR 2024: Tackling the medical error detection & correction task on the shoulders of medical agents. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 570–580, 2024. doi: 10.18653/v1/ 2024.clinicalnlp-1.54. Jean-Philippe Corbeil, Amin Dada, Jean-Michel Attendu, Asma Ben Abacha, Alessandro Sordoni, Lucas Caccia, Fran ̧cois Beaulieu, Thomas Lin, Jens Kleesiek, and Paul Vozila. A modular approach for clinical SLMs driven by synthetic data with pre-instruction tuning, model merging, and clinical-tasks alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19352– 19374, 2025. doi: 10.18653/v1/2025.acl-long.950. Pritam Deka, Anna Jurek-Loughrey, and P Deepak. Improved methods to aid unsupervised evidence-based fact checking for online health news. Journal of Data Intelligence, 3(4): 474–504, 2022. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4443–4458, 2020. doi: 10.18653/v1/2020.acl-main.408. Hamed Fayyaz, Raphael Poulain, and Rahmatollah Beheshti. Enabling scalable evaluation of bias patterns in medical LLMs. arXiv preprint arXiv:2410.14763, 2024. 17 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, 2020. doi: 10.18653/v1/ 2020.findings-emnlp.117. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. David M. Green and John A. Swets. Signal Detection Theory and Psychophysics. Wiley, 1966. Naoto Iwase, Hiroki Okuyama, and Junichiro Iwasawa. MedRECT: A medical reasoning benchmark for error correction in clinical texts. arXiv preprint arXiv:2511.00421, 2025. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. Praveenkumar Kanithi, Cl ́ement Christophe, Marco AF Pimentel, Tathagata Raha, Prateek Munjal, Nada Saadi, Hamza A. Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. MEDIC: Comprehensive evaluation of leading indicators for LLM safety and utility in clinical applications. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=pDQe9Icwb6. Linda T. Kohn, Janet M. Corrigan, and Molla S. Donaldson, editors. To Err Is Human: Building a Safer Health System. National Academies Press, Washington, DC, 2000. doi: 10.17226/9728. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023. doi: 10.1145/3600006.3613165. Zachary C. Lipton, Charles Elkan, and Balakrishnan Naryanaswamy. Optimal threshold- ing of classifiers to maximize F1 measure. In Toon Calders, Floriana Esposito, Eyke H ̈ullermeier, and Rosa Meo, editors, Machine Learning and Knowledge Discovery in Databases – European Conference, ECML PKDD 2014, volume 8725 of Lecture Notes in Computer Science, pages 225–239. Springer, 2014. doi: 10.1007/978-3-662-44851-9 15. Congbo Ma, Yichun Zhang, Yousef Al-Jazzazi, Ahamed Foisal, Laasya Sharma, Yousra Sadqi, Khaled Saleh, Jihad Mallat, and Farah E. Shamout. MedErrBench: A fine-grained multilingual benchmark for medical error detection and correction with clinical expert annotations. In Findings of the Association for Computational Linguistics: ACL 2026, pages 11802–11827, San Diego, California, United States, 2026. Association for Compu- tational Linguistics. doi: 10.18653/v1/2026.findings-acl.573. 18 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Martin A. Makary and Michael Daniel. Medical error—the third leading cause of death in the US. BMJ, 353:i2139, 2016. doi: 10.1136/bmj.i2139. Microsoft, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic represen- tation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=KRnsX5Em3W. Raphael Poulain, Farzana Islam Adiba, Hamed Fayyaz, and Rahmatollah Beheshti. Bias patterns in the application of LLMs for clinical decision support: A comprehensive study. Delaware Journal of Public Health, 12(1):54–67, 2026. doi: 10.32481/djph.2026.03.10. arXiv:2404.15149. Annika Reinke, Minu D. Tizabi, Michael Baumgartner, Matthias Eisenmann, Doreen Heckmann-N ̈otzel, et al. Understanding metric-related pitfalls in image analysis vali- dation. Nature Methods, 21(2):182–194, 2024. doi: 10.1038/s41592-023-02150-0. Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, and Rama Chellappa. Evaluation and mitigation of cognitive biases in medical language models. npj Digital Medicine, 7(1):295, 2024. doi: 10.1038/s41746-024-01283-6. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, et al. MedGemma technical report. arXiv preprint arXiv:2507.05201, 2025. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2024. URL https://openreview.net/forum?id=tvhaxkMKAn. Augustin Toma, Ronald Xie, Steven Palayew, Patrick Lawler, and Bo Wang. WangLab at MEDIQA-CORR 2024: Optimized LLM-based programs for medical error detection and correction. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 616–623, 2024. doi: 10.18653/v1/2024.clinicalnlp-1.59. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. doi: 10.52202/075280-3275. Ben Van Calster, Gary S. Collins, Andrew J. Vickers, Laure Wynants, Kathleen F. Kerr, Lasai Barre ̃nada, Ga ̈el Varoquaux, Karandeep Singh, Karel G. M. Moons, Tina Hernandez-Boussard, Dirk Timmerman, David J. McLernon, Maarten van Smeden, 19 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Ewout W. Steyerberg, and Topic Group 6 of the STRATOS initiative. Evaluation of performance measures in predictive artificial intelligence models to support medical de- cisions: overview and guidance. The Lancet Digital Health, 7(12):100916, 2025. doi: 10.1016/j.landig.2025.100916. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392, 2020. doi: 10.1162/tacl a00321. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 641–649, 2024. doi: 10.1145/3626772.3657878. An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Jin-Fang Hu, Zhiyuan Liu, and Bowen Zhou. UltraMedical: Building specialized generalists in biomedicine. In Advances in Neural Information Processing Systems, volume 37, pages 26045–26081, 2024. doi: 10.52202/079017-0819. Appendix A. Pair Construction Details BCR evaluation requires matched (error, correct) pairs from the same clinical scenario. Because the four datasets differ in their release structure, pair construction uses dataset- specific methods. MEDEC MS-Test (MS-Test). MEDEC (Ben Abacha et al., 2025) does not provide explicit pairing between error-injected and correct notes. We recover pairs using Jaccard word-overlap similarity between each error note and all clean notes, with a threshold of ≥ 0.6. Each error note is assigned to its highest-similarity clean note under one-to-one matching (i.e., once a clean note is paired, it is removed from the candidate pool). Of 311 error notes, 286 are successfully paired; 25 remain unmatched and are excluded from BCR computation. These unmatched notes are retained for traditional pointwise metric computation. MedErrBench-EN (MEB-EN) and MedErrBench-CN (MEB-CN). Both test sets from MedErrBench (Ma et al., 2026) release samples in an alternating error–correct order. We pair consecutive samples directly: for each pair of adjacent rows, one has Error Flag = 1 and the other has Error Flag = 0. This yields 104 pairs for MedErrBench-EN and 100 pairs for MedErrBench-CN, with no unmatched samples. 20 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection MedRECT-JA (MRT-JA). MedRECT (Iwase et al., 2025) provides a clinical scenario ID for each note. We group notes by scenario ID and pair each error note with clean notes from the same scenario, filtering to pairs with text similarity above 0.85 (measured by character-level overlap). This yields 190 pairs. However, because MedRECT contains 190 error notes but only 105 unique clean notes, some clean notes appear in multiple pairs (many-to-one structure). If a shared clean note is systematically misclassified, all its asso- ciated pairs fail simultaneously, which may inflate the apparent degree of within-pair error correlation for this dataset. Appendix B. An Upper Bound on BCR We show that on matched pairs BCR is bounded above by the smaller of the model’s marginal sensitivity and specificity, and draw the corollary that a class-biased predictor can attain a moderate F1. In contrast, its BCR is pinned at zero. This makes the F1/BCR divergence of Section 4.5 a structural property rather than a feature of the particular models tested. Setup. Let the evaluation set consist of N matched pairs (x e , x c ), each with one error note (y e = 1) and one correct note (y c = 0). Write the model’s per-note predictions as ˆy e , ˆy c ∈ 0, 1. Because the pairs are matched, the N error notes are exactly the positive class and the N correct notes exactly the negative class, so sensitivity = 1 N N X i=1 1[ˆy e,i = 1],specificity = 1 N N X i=1 1[ˆy c,i = 0], while BCR = 1 N P i 1[ˆy e,i = 1∧ ˆy c,i = 0] (Equation 1). Bound. For every pair i, the Both-Correct indicator is the product of the two single-note success indicators, and is therefore at most each factor: 1[ˆy e,i = 1∧ ˆy c,i = 0] = 1[ˆy e,i = 1] 1[ˆy c,i = 0]≤ 1[ˆy e,i = 1], and likewise ≤ 1[ˆy c,i = 0]. Averaging over the N pairs gives BCR ≤ sensitivity and BCR≤ specificity, hence BCR≤ min(sensitivity, specificity), which is Equation 4. Equivalently, the Both-Correct pairs are the intersection of the cor- rectly flagged error notes and the correctly cleared correct notes, and an intersection cannot exceed either set. Corollary: class bias caps BCR. The bound is governed by the weaker marginal, which a class-biased model drives toward zero. The constant “always-error” predictor (ˆy = 1 for every note) has sensitivity = 1 and specificity = 0, so min(sensitivity, specificity) = 0 and therefore BCR = 0: it fails every pair on the correct member. Yet on a class-balanced set its F1 is bounded well away from zero—with precision = 1/2 and recall = 1, F1 = 2· 1 2 · 1 1 2 + 1 = 2 3 ≈ 0.67. 21 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection More generally, any predictor whose bias pushes one marginal toward its extreme incurs a BCR ceiling at the opposite, weak marginal, however high F1 climbs through recall on the favored class. The F1/BCR divergence documented in Section 4.5 thus follows from Equation 4 for any model set, and is not an artifact of the particular models we evaluate. Appendix C. Evaluated Models: References and Identifiers Table 4 lists, for each of the 15 evaluated models, the technical report/publication citation, the HuggingFace repository identifier, the model family, whether it is medical-domain, and the loading precision. Midrule separators follow the three size tiers (Small 3–8B, Large 27–32B, Very Large 70B). Table 4: References, HuggingFace identifiers, family, domain, and loading precision for all 15 evaluated models. All models are instruction-tuned. Model (Reference)HuggingFace IDFamilyDomain Prec. Llama 3.2-3B (Grattafiori et al., 2024) meta-llama/Llama-3.2-3B-InstructLlamaGeneralbf16 Gemma 3-4B (Gemma Team et al., 2025) google/gemma-3-4b-itGemma Generalbf16 Qwen 3-4B (Yang et al., 2025) Qwen/Qwen3-4B-Instruct-2507QwenGeneralbf16 MedGemma 4B (Sellergren et al., 2025) google/medgemma-4b-itGemma Medical bf16 MediPhi 4B (Corbeil et al., 2025) microsoft/MediPhi-InstructPhiMedical bf16 Phi-4-mini (Microsoft et al., 2025) microsoft/Phi-4-mini-instructPhiGeneralbf16 Mistral 7B v0.3 (Jiang et al., 2023) mistralai/Mistral-7B-Instruct-v0.3MistralGeneralbf16 Llama 3.1-8B (Grattafiori et al., 2024) meta-llama/Llama-3.1-8B-InstructLlamaGeneralbf16 UltraMedical 8B (Zhang et al., 2024) TsinghuaC3I/Llama-3-8B-UltraMedicalLlamaMedical bf16 Qwen 3-8B (Yang et al., 2025) Qwen/Qwen3-8BQwenGeneralbf16 Gemma 3-27B (Gemma Team et al., 2025) google/gemma-3-27b-itGemma Generalbf16 MedGemma 27B (Sellergren et al., 2025) google/medgemma-27b-text-itGemma Medical bf16 Qwen 3-32B (Yang et al., 2025) Qwen/Qwen3-32BQwenGeneralbf16 Llama 3.1-70B (Grattafiori et al., 2024) meta-llama/Llama-3.1-70B-InstructLlamaGeneralfp8 UltraMedical 70B (Zhang et al., 2024) TsinghuaC3I/Llama-3-70B-UltraMedical LlamaMedical fp8 Appendix D. Prompt Templates and Decoding Settings Decoding settings. Greedy decoding uses temperature T = 0. Stochastic sampling uses T = 0.7 and top-p = 0.9, common defaults for open-ended generation. Each model is run once per (prompt × decoding) configuration, yielding four runs per model–dataset pair. Output format. All prompts require a structured four-line output format with Evidence (a quoted text span from the note, or NA), Analysis (3–5 sentences of reasoning), Confidence (0–100), and Error: Yes/No. The key structural difference between neutral and conser- vative variants is in the system prompt and error-flagging threshold. MEDEC MS-Test — Neutral. System: You are a skilled medical doctor reviewing the clinical text. The following is a medical narrative about a patient. The text is either correct or contains one error. 22 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection An “error” must meet BOTH: (1) The note explicitly states something incorrect/unsafe (diagnosis, management, medication, treatment, or causal organism), AND (2) You can quote the exact text span that is wrong. If you cannot quote an exact span, output Error: No. Output EXACTLY 4 lines (nothing else): Evidence: ⟨copy a short exact quote from the note that is wrong, or NA⟩ Analysis: [Your short reasoning in 3–5 sentences] Confidence: [0–100] Error: [No or Yes] Clinical Note: medical note MEDEC MS-Test — Conservative. System: You are a conservative medical reviewer. Only flag a significant medical error if the text itself clearly supports it. Do NOT assume missing information. Do NOT invent guidelines or facts not stated in the note. If the note is incomplete or ambiguous, prefer “no error” with lower confidence. The user prompt follows the same structure as neutral, with “significant medical error” replacing “error.” MedErrBench-EN. Identical to MEDEC MS-Test prompts, except the error-type list is expanded to 10 categories: diagnosis, management, treatment, pharmacotherapy, causal organism, lab/serum value interpretation, physiology, histology, anatomy, and epidemiology. MedErrBench-CN. The same prompt structure was translated into Chinese, along with lists of Chinese-language error types. Output format keys (Evidence, Analysis, Confidence, Error) remain in English for parsing consistency. MedRECT-JA. The same prompt structure was translated into Japanese, with nine Japanese-language error types derived from the dataset’s annotation taxonomy. Output format keys remain in English. Appendix E. Robustness Analyses Parse failure handling. Nine of 240 runs exceed a 5% parse failure rate (Table 5). Parse failures concentrate in UltraMedical (8B and 70B), primarily on the non-English datasets, and in Llama 3.2-3B on MedErrBench-CN. Comparing the skip strategy against random assignment yields deltas below 0.5 p for all 15 models (Table 6). The same 13 of 15 models fall below 25% BCR under both strategies. All BCR values reported in this paper use the skip strategy; random assignment serves only as a robustness check. 23 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Table 5: Runs exceeding 5% parse failure rate (9 of 240). ModelDatasetConfigparse-failure rate (%) UltraMed-8BMEB-CN neut-gre24.5 UltraMed-70B MRT-JA cons-sam22.0 UltraMed-70B MRT-JA neut-sam14.9 Llama 3.2-3BMEB-CN cons-sam9.5 UltraMed-8BMEB-CN neut-sam9.5 Llama 3.2-3BMEB-CN neut-sam9.0 UltraMed-8BMRT-JA neut-sam8.8 UltraMed-70B MRT-JAcons-gre7.5 UltraMed-8BMS-Testneut-sam7.4 Table 6: Skip vs. random-assignment BCR comparison (four-configuration mean, cross- dataset mean). All deltas < 0.5 p. ModelSkip (%) Random (%)∆ Qwen 3-32B28.028.00.0 UltraMed-70B25.726.1 −0.4 MedGemma 27B23.123.10.0 Llama 3.1-70B20.020.1 −0.1 UltraMed-8B20.020.3 −0.3 Qwen 3-4B17.017.00.0 MedGemma 4B15.015.00.0 Llama 3.1-8B14.614.7 −0.1 Phi-4-mini12.712.6 +0.1 Qwen 3-8B12.112.10.0 MediPhi 4B10.710.70.0 Gemma 3-27B10.210.20.0 Llama 3.2-3B10.210.5 −0.3 Mistral 7B7.17.3 −0.2 Gemma 3-4B4.64.60.0 Threshold sensitivity. We checked the ECA word-coverage threshold at 0.5, 0.6, and 0.7, and the MS-Test Jaccard pairing threshold at 0.5, 0.6, and 0.7. Per-model TP-localization rates shift by < 3 p, and the rank ordering of models is preserved. The MRT-JA character- overlap threshold of 0.85 yields the full 190-pair set; lowering to 0.75 adds no pairs because the same-scenario notes already exceed 0.85. Embedding-based localization robustness. ECA’s TP localization is scored by an ab- solute substring/word-coverage rule, which raises the concern that the localization finding could be an artifact of that particular criterion or of incidental word overlap. To test this, we re-scored TP localization on the same model outputs (no re-inference) with a methodolog- ically orthogonal, embedding-based criterion. For each Pred1 pair on MS-Test, we embed the cited Evidence span and every sentence of the error note with a sentence encoder, rank sentences by cosine similarity, and count a hit only when the gold error sentence is the single 24 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection most similar sentence (Recall@1 )—a relative-ranking test that is stricter than an absolute threshold and not tied to any word-overlap cutoff. We use two encoders, one biomedi- cal (S-PubMedBert (Deka et al., 2022), checkpoint pritamdeka/S-PubMedBert-MS-MARCO) and one general (BGE-large (Xiao et al., 2024), checkpoint BAAI/bge-large-en-v1.5), and pool over the four prompt–decoding configurations of all 15 models, giving N = 9,428 lo- calization judgments. As shown in Table 7, pooled Recall@1 is 65.3% / 67.1% against a 12.6% random baseline, all 15 models clear the baseline under both encoders, and per-model Recall@1 correlates with the substring TP-localization rate at Pearson r = 0.99 / 0.98; the pooled substring rate (69.4%) reproduces the 69% reported in Section 4.4. A methodolog- ically different localization criterion therefore reproduces both the per-model ranking and the magnitude of the finding. We present this as robustness of the localization signal to how it is measured, not as a claim of semantic abstraction from word overlap: the agreement reflects that models predominantly cite the error sentence verbatim, which, if anything, sharpens the localization–judgment gap, since the model reproduces the erroneous sentence in its own evidence yet still labels both the erroneous and the corrected note as containing an error. Table 7: Embedding-based re-scoring of TP localization on MS-Test Pred1 pairs (N = 9,428 judgments pooled over 15 models × 4 configurations). Recall@1 counts a hit only when the gold error sentence is the single most similar sentence to the cited evidence. Random baseline = 12.6%. Pearson r is the per-model correlation with the substring TP-localization rate. EncoderRecall@1 Recall@3 MRR Pearson r Above baseline S-PubMedBert (biomedical)65.3%73.8%0.730.9915/15 BGE-large (general)67.1%82.5%0.770.9815/15 Cross-configuration BCR. Figure 6 shows BCR for all 15 models under each of the four configurations. The rank ordering of models is largely stable across configurations, but absolute BCR values vary substantially (up to 36.2 p for Llama 3.1-70B). 25 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection MSME-ENME-CNMR-JA UltraMed-70B Llama3.1-70B Qwen3-32B MedGemma-27B Gemma3-27B Qwen3-8B Qwen3-4B Phi4-mini-4B Mistral-7B MediPhi-4B MedGemma-4B UltraMed-8B Llama3.2-3B Llama3.1-8B Gemma3-4B 11.944.25.08.3 18.545.212.012.1 25.541.327.027.9 14.341.37.026.5 16.117.322.06.8 7.313.54.021.1 17.521.222.020.5 17.514.48.016.9 7.78.710.05.0 12.211.56.06.8 18.225.013.017.9 16.130.110.214.2 3.85.820.02.7 22.529.88.012.6 10.12.98.07.4 Conservative-Greedy MSME-ENME-CNMR-JA UltraMed-70B Llama3.1-70B Qwen3-32B MedGemma-27B Gemma3-27B Qwen3-8B Qwen3-4B Phi4-mini-4B Mistral-7B MediPhi-4B MedGemma-4B UltraMed-8B Llama3.2-3B Llama3.1-8B Gemma3-4B 17.547.19.120.0 23.439.819.028.4 26.942.326.028.9 16.832.711.034.4 16.819.221.012.6 8.713.56.020.0 18.218.323.018.4 24.521.215.014.3 8.08.713.08.3 21.317.321.010.0 21.021.212.020.0 12.924.317.010.8 8.012.527.28.6 20.330.120.024.7 8.45.87.07.4 Conservative-Sample MSME-ENME-CNMR-JA UltraMed-70B Llama3.1-70B Qwen3-32B MedGemma-27B Gemma3-27B Qwen3-8B Qwen3-4B Phi4-mini-4B Mistral-7B MediPhi-4B MedGemma-4B UltraMed-8B Llama3.2-3B Llama3.1-8B Gemma3-4B 35.042.329.021.0 23.89.013.118.6 19.921.237.024.7 26.619.220.030.0 4.91.96.04.2 14.010.612.010.0 5.97.722.022.6 4.56.72.016.3 1.01.91.020.9 3.55.86.09.5 13.310.615.24.7 21.028.724.613.4 2.12.98.77.1 4.62.913.05.4 2.40.01.05.8 Neutral-Greedy MSME-ENME-CNMR-JA UltraMed-70B Llama3.1-70B Qwen3-32B MedGemma-27B Gemma3-27B Qwen3-8B Qwen3-4B Phi4-mini-4B Mistral-7B MediPhi-4B MedGemma-4B UltraMed-8B Llama3.2-3B Llama3.1-8B Gemma3-4B 32.941.324.222.6 15.114.417.010.6 18.218.338.024.7 23.423.120.223.2 5.21.04.04.7 14.714.410.013.2 6.35.825.017.9 9.18.75.018.4 1.01.01.017.0 8.76.712.012.6 11.59.619.07.4 28.330.922.015.7 9.812.69.421.7 9.24.817.38.4 1.00.01.04.7 Neutral-Sample 0 5 10 15 20 25 30 35 40 45 BCR (%) 0 5 10 15 20 25 30 35 40 45 BCR (%) 0 5 10 15 20 25 30 35 40 45 BCR (%) 0 5 10 15 20 25 30 35 40 45 BCR (%) BCR (%) by Configuration Skip Strategy Figure 6: BCR (%) by configuration. Rank ordering is largely stable; absolute values vary up to 36.2 p. Appendix F. Prediction Bias Mediation Analysis Section 4.5 reports that prediction bias (error-flag rate) mediates the relationship between F1 and BCR. Tables 8–10 provide per-dataset details. Table 8: Per-dataset error-flag-rate mediation correlations (Pearson r, n = 15 models per dataset). Error-flag rate suppresses BCR across all datasets and strongly inflates F1 on three of four (weaker on MEB-EN); the direct F1–BCR correlation varies in sign across datasets and is near zero only when pooled. DatasetError-flag → F1 Error-flag → BCR F1 → BCR MS-Test+0.946−0.855−0.659 MEB-EN+0.406−0.737+0.310 MEB-CN+0.911−0.530−0.142 MRT-JA+0.963−0.480−0.233 All (n = 60)+0.853−0.488−0.058 26 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Table 9: Top-3 models by F1 vs. top-3 by BCR (4-config mean). On 3 of 4 datasets, no model appears in both top-3 lists. DatasetTop-3 by F1Top-3 by BCROverlap MS-Test Gemma 3-4B, Llama 3.2-3B, Mistral 7BUltraMed-70B, Qwen 3-32B, MedGemma 27B 0/3 MEB-EN Qwen 3-32B, UltraMed-70B, Llama 3.1-70B UltraMed-70B, Qwen 3-32B, MedGemma 27B 2/3 MEB-CN Mistral 7B, Gemma 3-4B, MediPhi 4BQwen 3-32B, Qwen 3-4B, UltraMed-8B0/3 MRT-JA Gemma 3-27B, Gemma 3-4B, Llama 3.2-3B MedGemma 27B, Qwen 3-32B, Qwen 3-4B0/3 Table 10: Per-dataset independence ratios. Ratio is the mean of per-model independence ratios R independence (Equation 3), not the quotient of the two column means. Actual BCR falls below the expected value (sensitivity × specificity) for all 60 model–dataset entries. DatasetExpected BCR Actual BCRRatioBelow MS-Test21.9%14.2%0.685× 15/15 MEB-EN21.7%17.8%0.844× 15/15 MEB-CN23.5%14.4%0.621× 15/15 MRT-JA20.3%15.2%0.765× 15/15 All21.9%15.4%0.729× 60/60 Appendix G. Detailed Results G.1. Full Traditional Metrics (All 240 Runs) Tables 11–14 report per-run traditional metrics for every inference run in the study, or- ganized by configuration. Each table contains 60 rows (15 models × 4 datasets), sorted by balanced accuracy within each dataset. Flag% (error-flag rate) is the fraction of notes flagged as containing an error. Table 11: Traditional metrics — Conservative-Greedy configuration (all 15 models× 4 datasets = 60 runs). Sorted by balanced accuracy within each dataset. Flag%(error-flag rate) is the fraction of notes flagged as containing an error. DatasetModelBalAccF1 Prec Rec Spec MCC Flag% MS-TestQwen 3-32B.584 .564 .616 .521 .647.16944.1 Llama 3.1-70B.566 .375 .690 .257 .874.16619.4 MedGemma 27B.555 .300 .720 .190 .920.15913.7 UltraMed-70B.554 .262 .778 .158 .951.17710.6 Llama 3.1-8B.550 .539 .575 .506 .593.10045.9 MedGemma 4B.549 .482 .588 .408 .689.10136.2 Gemma 3-27B.546 .513 .576 .463 .629.09441.9 Qwen 3-4B.531 .620 .544 .720 .343.06869.0 UltraMed-8B.527 .368 .575 .270 .783.06224.5 Gemma 3-4B.525 .675 .535 .913 .136.07988.9 MediPhi 4B.520 .657 .533 .855 .185.05583.6 (continued on next page) 27 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection (Table 11 continued from previous page) DatasetModelBalAccF1 Prec Rec Spec MCC Flag% Phi-4-mini.517 .604 .533 .698 .336.03668.2 Qwen 3-8B.511 .238 .560 .151 .871.03114.1 Llama 3.2-3B.505 .680 .523 .971 .038.02696.6 Mistral 7B.496 .658 .519 .900 .091−.01590.5 MedErrBench-EN UltraMed-70B.692 .640 .770 .548 .837.40235.6 Llama 3.1-70B.688 .709 .664 .760 .615.37957.2 MedGemma 27B.683 .612 .788 .500 .865.39331.7 Qwen 3-32B.673 .721 .629 .846 .500.36967.3 UltraMed-8B.594 .567 .604 .534 .654.18944.0 Qwen 3-4B.582 .690 .548 .933 .231.22985.1 Llama 3.1-8B.572 .621 .557 .702 .442.14963.0 MedGemma 4B.558 .521 .568 .481 .635.11742.3 Gemma 3-27B.553 .669 .531 .904 .202.14885.1 Phi-4-mini.538 .657 .523 .885 .192.10784.6 MediPhi 4B.538 .671 .521 .942 .135.13090.4 Qwen 3-8B.529 .395 .552 .308 .750.06427.9 Llama 3.2-3B.505 .658 .503 .952 .058.02194.7 Mistral 7B.505 .648 .503 .913 .096.01790.9 Gemma 3-4B.490 .649 .495 .942 .038−.04595.2 MedErrBench-CN Qwen 3-32B.590 .474 .661 .370 .810.20028.0 Qwen 3-4B.575 .573 .576 .570 .580.15049.5 Gemma 3-27B.570 .623 .555 .710 .430.14664.0 Llama 3.1-70B.545 .222 .765 .130 .960.1618.5 MedGemma 4B.525 .371 .549 .280 .770.05725.5 UltraMed-70B.525 .112 .857 .060 .990.1363.5 MedGemma 27B.525 .188 .647 .110 .940.0908.5 Llama 3.2-3B.524 .456 .534 .398 .649.04937.4 Gemma 3-4B.520 .631 .512 .820 .220.05080.0 Qwen 3-8B.515 .110 .667 .060 .970.0724.5 Llama 3.1-8B.510 .246 .533 .160 .860.02815.0 UltraMed-8B.507 .236 .517 .153 .860.01814.6 Mistral 7B.505 .643 .503 .890 .120.01688.5 MediPhi 4B.495 .625 .497 .840 .150−.01484.5 Phi-4-mini.495 .625 .497 .840 .150−.01484.5 MedRECT-JAQwen 3-32B.585 .516 .763 .389 .781.17432.9 Qwen 3-4B.576 .741 .691 .800 .352.16874.6 Llama 3.1-70B.558 .346 .792 .221 .895.14518.0 MedGemma 27B.551 .638 .685 .598 .505.09956.1 Qwen 3-8B.545 .572 .689 .489 .600.08645.8 MedGemma 4B.534 .711 .665 .763 .305.07473.9 UltraMed-8B.525 .441 .681 .326 .724.05230.8 Phi-4-mini.516 .734 .652 .841 .190.04083.0 Gemma 3-27B.512 .755 .650 .900 .124.03789.2 UltraMed-70B.508 .213 .667 .126 .889.02312.1 Gemma 3-4B.507 .753 .648 .900 .114.02289.5 MediPhi 4B.507 .749 .648 .889 .124.02088.5 Llama 3.2-3B.496 .768 .639 .962 .029−.02396.6 Llama 3.1-8B.495 .708 .641 .789 .200−.01279.3 Mistral 7B.458 .688 .622 .769 .147−.10079.9 28 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Table 12: Traditional metrics — Conservative-Sampling configuration (all 15 models× 4 datasets = 60 runs). Sorted by balanced accu- racy within each dataset. Flag%(error-flag rate) is the fraction of notes flagged as containing an error. DatasetModelBalAccF1 Prec Rec Spec MCC Flag% MS-TestLlama 3.1-70B.588 .462 .688 .347 .829.20026.3 UltraMed-70B.570 .355 .730 .235 .906.18816.8 MedGemma 27B.569 .345 .737 .225 .913.18815.9 Qwen 3-32B.566 .543 .596 .498 .633.13243.6 Gemma 3-27B.555 .518 .588 .463 .647.11241.0 Phi-4-mini.542 .606 .555 .669 .416.08862.8 MedGemma 4B.542 .476 .578 .405 .678.08736.5 Qwen 3-4B.530 .616 .543 .711 .350.06568.2 Gemma 3-4B.523 .676 .533 .923 .122.07690.1 MediPhi 4B.521 .599 .537 .678 .364.04465.8 Llama 3.1-8B.517 .446 .544 .377 .657.03636.1 UltraMed-8B.505 .341 .527 .252 .758.01124.7 Qwen 3-8B.501 .227 .523 .145 .857.00214.4 Llama 3.2-3B.498 .656 .520 .891 .105−.00789.3 Mistral 7B.493 .654 .517 .891 .094−.02589.8 MedErrBench-EN UltraMed-70B.692 .680 .708 .654 .731.38646.2 Qwen 3-32B.692 .742 .639 .885 .500.41769.2 Llama 3.1-70B.666 .718 .624 .846 .485.35668.1 MedGemma 27B.644 .575 .714 .481 .808.30533.7 Gemma 3-27B.572 .683 .542 .923 .221.20285.1 Phi-4-mini.572 .674 .544 .885 .260.18581.2 Llama 3.1-8B.570 .594 .565 .625 .515.14055.6 Qwen 3-4B.562 .681 .536 .933 .192.18687.0 UltraMed-8B.561 .486 .589 .413 .709.12835.3 MedGemma 4B.529 .500 .533 .471 .587.05844.2 Mistral 7B.529 .673 .515 .971 .087.12494.2 MediPhi 4B.519 .618 .513 .779 .260.04576.0 Gemma 3-4B.514 .667 .508 .971 .058.07195.7 Qwen 3-8B.514 .380 .525 .298 .731.03228.4 Llama 3.2-3B.510 .641 .506 .875 .144.02886.5 MedErrBench-CN Qwen 3-32B.600 .487 .679 .380 .820.22328.0 Qwen 3-4B.580 .584 .578 .590 .570.16051.0 Llama 3.1-70B.570 .434 .635 .330 .810.16026.0 UltraMed-8B.565 .351 .697 .235 .896.17417.0 Gemma 3-27B.555 .608 .543 .690 .420.11463.5 Llama 3.1-8B.555 .378 .628 .270 .840.13421.5 UltraMed-70B.546 .182 .909 .101 .990.1995.5 MedGemma 27B.545 .260 .696 .160 .930.14111.5 Llama 3.2-3B.542 .520 .556 .489 .596.08544.8 Qwen 3-8B.530 .145 .800 .080 .980.1385.0 MediPhi 4B.530 .618 .521 .760 .300.06873.0 MedGemma 4B.515 .312 .537 .220 .810.03720.5 Mistral 7B.515 .639 .509 .860 .170.04184.5 Phi-4-mini.510 .623 .506 .810 .210.02580.0 Gemma 3-4B.485 .593 .490 .750 .220−.03576.5 MedRECT-JAMedGemma 27B.627 .700 .749 .658 .596.24556.8 Llama 3.1-70B.591 .569 .750 .458 .724.17839.3 Qwen 3-32B.590 .517 .771 .389 .790.18432.5 (continued on next page) 29 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection (Table 12 continued from previous page) DatasetModelBalAccF1 Prec Rec Spec MCC Flag% UltraMed-70B.558 .429 .733 .303 .812.12726.1 Llama 3.1-8B.550 .734 .674 .805 .295.11476.9 MedGemma 4B.548 .731 .673 .800 .295.10876.6 Qwen 3-4B.539 .717 .668 .774 .305.08674.6 Qwen 3-8B.538 .580 .681 .505 .571.07447.8 Llama 3.2-3B.519 .767 .657 .921 .117.06190.8 Gemma 3-27B.518 .752 .654 .884 .152.05287.1 Phi-4-mini.512 .741 .649 .862 .162.03385.4 Gemma 3-4B.511 .767 .650 .937 .086.04292.9 MediPhi 4B.501 .744 .645 .879 .124.00487.8 UltraMed-8B.455 .378 .580 .280 .630−.09331.2 Mistral 7B.447 .675 .614 .749 .146−.12378.6 Table 13: Traditional metrics — Neutral-Greedy configuration (all 15 models× 4 datasets = 60 runs). Sorted by balanced accuracy within each dataset. Flag%(error-flag rate) is the fraction of notes flagged as containing an error. DatasetModelBalAccF1 PrecRec Spec MCC Flag% MS-TestUltraMed-70B.642 .650 .661.640 .643.28350.4 MedGemma 27B.596 .690 .585.839 .353.22174.7 Llama 3.1-70B.586 .669 .584.784 .389.18870.2 Qwen 3-32B.561 .676 .559.855 .266.15079.7 UltraMed-8B.541 .618 .552.701 .381.08766.2 Qwen 3-8B.533 .672 .540.891 .175.09485.9 MedGemma 4B.517 .656 .531.859 .175.04684.3 Gemma 3-4B.511 .689 .526.997 .024.09298.7 Gemma 3-27B.510 .675 .526.942 .077.03893.3 Qwen 3-4B.509 .677 .526.949 .070.03994.0 Phi-4-mini.507 .676 .525.949 .066.03294.1 Llama 3.1-8B.507 .677 .524.958 .056.03395.1 Llama 3.2-3B.506 .685 .524.990 .021.04698.5 Mistral 7B.501 .682 .521.987 .014.00598.7 MediPhi 4B.495 .670 .519.945 .045−.02195.0 MedErrBench-EN UltraMed-70B.712 .774 .636.990 .433.51077.9 UltraMed-8B.609 .688 .575.854 .363.24974.6 Qwen 3-32B.596 .706 .555.971 .221.29187.5 MedGemma 27B.582 .697 .546.962 .202.25188.0 Llama 3.1-70B.545 .691 .528 1.000 .089.21795.6 Qwen 3-8B.538 .678 .521.971 .106.15493.3 Qwen 3-4B.534 .680 .518.990 .077.16595.7 Phi-4-mini.529 .678 .515.990 .067.15096.2 MediPhi 4B.524 .675 .512.990 .058.13396.6 Llama 3.1-8B.515 .673 .507 1.000 .029.12298.5 MedGemma 4B.510 .643 .505.885 .135.02987.5 Mistral 7B.510 .671 .505 1.000 .019.09999.0 Gemma 3-27B.510 .671 .505 1.000 .019.09999.0 Gemma 3-4B.500 .667 .500 1.000 .000.000100.0 Llama 3.2-3B.481 .633 .489.894 .067−.06891.3 MedErrBench-CN Qwen 3-32B.655 .679 .635.730 .580.31457.5 UltraMed-70B.635 .535 .737.420 .850.29928.5 (continued on next page) 30 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection (Table 13 continued from previous page) DatasetModelBalAccF1 PrecRec Spec MCC Flag% Qwen 3-4B.590 .664 .562.810 .370.20072.0 MedGemma 27B.590 .677 .558.860 .320.21477.0 UltraMed-8B.549 .570 .549.592 .507.09954.3 Llama 3.1-70B.532 .559 .532.590 .475.06555.8 Llama 3.2-3B.530 .654 .512.904 .156.09187.4 Qwen 3-8B.530 .605 .522.720 .340.06569.0 MedGemma 4B.529 .633 .516.818 .240.07178.9 Gemma 3-27B.520 .657 .511.920 .120.06790.0 MediPhi 4B.520 .669 .511.970 .070.09295.0 Llama 3.1-8B.510 .585 .507.690 .330.02168.0 Gemma 3-4B.505 .669 .503 1.000 .010.07199.5 Mistral 7B.490 .658 .497.970 .010−.07198.0 Phi-4-mini.485 .639 .492.910 .060−.05792.5 MedRECT-JAQwen 3-4B.600 .801 .696.942 .257.28587.1 MedGemma 27B.596 .757 .703.821 .371.21475.3 Qwen 3-32B.596 .771 .700.858 .333.22579.0 Llama 3.1-70B.567 .762 .679.867 .267.16781.9 UltraMed-70B.551 .504 .710.390 .713.10335.3 Phi-4-mini.540 .722 .668.784 .295.08975.6 UltraMed-8B.531 .391 .704.270 .792.06924.8 Qwen 3-8B.528 .771 .658.932 .124.09491.2 Mistral 7B.524 .652 .661.644 .404.04762.7 Gemma 3-27B.521 .786 .654.984 .057.11596.9 MediPhi 4B.514 .750 .651.884 .143.03987.5 Gemma 3-4B.511 .779 .649.974 .048.05696.6 Llama 3.2-3B.502 .747 .645.887 .117.00588.6 Llama 3.1-8B.494 .755 .645.910 .078−.02091.4 MedGemma 4B.489 .720 .638.826 .152−.02783.4 Table 14: Traditional metrics — Neutral-Sampling configuration (all 15 models× 4 datasets = 60 runs). Sorted by balanced accuracy within each dataset. Flag%(error-flag rate) is the fraction of notes flagged as containing an error. DatasetModelBalAccF1 PrecRec Spec MCC Flag% MS-TestUltraMed-70B.621 .688 .611.788 .455.25867.2 MedGemma 27B.568 .666 .567.807 .329.15574.2 UltraMed-8B.557 .586 .577.597 .517.11454.2 Qwen 3-32B.545 .666 .549.846 .245.11380.2 Llama 3.1-70B.543 .675 .548.878 .208.11683.7 Qwen 3-8B.538 .674 .543.887 .189.10785.1 Phi-4-mini.514 .670 .529.913 .115.04789.9 Gemma 3-27B.511 .676 .527.942 .080.04593.1 MediPhi 4B.511 .669 .527.916 .105.03690.6 Llama 3.2-3B.511 .669 .527.916 .105.03690.6 Qwen 3-4B.507 .676 .525.949 .066.03294.1 Llama 3.1-8B.507 .667 .525.913 .102.02590.6 Mistral 7B.504 .686 .523.997 .010.04599.3 Gemma 3-4B.500 .683 .521.990 .010.00499.0 MedGemma 4B.497 .639 .519.830 .164−.00883.2 MedErrBench-EN UltraMed-70B.692 .759 .623.971 .413.46377.9 (continued on next page) 31 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection (Table 14 continued from previous page) DatasetModelBalAccF1 PrecRec Spec MCC Flag% UltraMed-8B.615 .667 .575.794 .436.24567.7 MedGemma 27B.611 .716 .564.981 .240.32987.0 Qwen 3-32B.587 .703 .548.981 .192.28189.4 Llama 3.1-70B.567 .696 .536.990 .144.25392.3 Qwen 3-8B.553 .683 .529.962 .144.18490.9 Phi-4-mini.529 .671 .515.962 .096.11593.3 Qwen 3-4B.524 .675 .512.990 .058.13396.6 Llama 3.1-8B.514 .667 .508.971 .058.07195.7 MedGemma 4B.505 .644 .503.894 .115.01588.9 MediPhi 4B.505 .656 .503.942 .067.02093.8 Gemma 3-27B.505 .669 .502 1.000 .010.07099.5 Gemma 3-4B.500 .667 .500 1.000 .000.000100.0 Mistral 7B.500 .665 .500.990 .010.00099.0 Llama 3.2-3B.475 .589 .481.757 .192−.06178.3 MedErrBench-CN Qwen 3-32B.655 .682 .632.740 .570.31558.5 Qwen 3-4B.605 .675 .573.820 .390.23371.5 UltraMed-70B.587 .560 .593.531 .643.17544.3 MedGemma 27B.567 .656 .547.820 .313.15575.4 Llama 3.1-8B.556 .627 .540.747 .364.12069.2 Llama 3.1-70B.550 .637 .534.790 .310.11474.0 MedGemma 4B.545 .640 .529.810 .280.10676.5 UltraMed-8B.537 .500 .553.457 .618.07542.0 Qwen 3-8B.525 .592 .519.690 .360.05366.5 Gemma 3-27B.515 .657 .508.930 .100.05491.5 Phi-4-mini.515 .664 .508.960 .070.06694.5 MediPhi 4B.510 .645 .506.890 .130.03188.0 Gemma 3-4B.505 .669 .503 1.000 .010.07199.5 Mistral 7B.500 .664 .500.990 .010.00099.0 Llama 3.2-3B.490 .593 .467.814 .167−.02582.4 MedRECT-JAQwen 3-32B.595 .767 .700.847 .343.22078.0 MedGemma 27B.573 .734 .690.784 .362.15873.2 Qwen 3-4B.556 .761 .675.874 .238.14483.4 UltraMed-70B.551 .640 .683.602 .500.09956.6 Llama 3.1-70B.538 .776 .665.932 .144.12390.5 Qwen 3-8B.538 .778 .663.942 .133.13091.5 Phi-4-mini.534 .730 .664.811 .257.07978.6 Llama 3.2-3B.533 .740 .668.829 .238.08080.6 MedGemma 4B.521 .754 .655.889 .152.06187.5 Gemma 3-27B.520 .782 .654.974 .067.09895.9 Llama 3.1-8B.513 .763 .651.921 .105.04491.2 MediPhi 4B.511 .726 .650.821 .200.02681.4 Mistral 7B.500 .663 .645.683 .317−.00068.3 Gemma 3-4B.497 .766 .643.947 .048−.01194.9 UltraMed-8B.489 .312 .621.208 .771−.02521.6 G.2. Full ECA Category Breakdown Table 15 reports the complete ECA breakdown for all 15 models on all four datasets, with category counts pooled across the four prompt–decoding configurations. TP localization and FP evidence-hit are reported as four-configuration means, matching the convention used in Section 4.4; per-configuration raw outputs are available as supplementary material. 32 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection Table 15: Full ECA breakdown per model–dataset combination (totals pooled across the four configurations). n: Pred1 pairs categorized into A–E. TP loc and FP hit: four-configuration mean overlap rates (%). A– E: pooled category counts (Both-Hit / TP-Only / FP-Only / Neither- Hit / Extraction-Fail). Within each dataset, rows are sorted by TP loc. Every model–dataset combination had at least one configuration with ≥ 5 Pred1 pairs, so all 60 combinations are listed. Because TP loc is an unweighted four-configuration mean while A–E are pooled counts, TP loc and (A + B)/n can differ by several points when Pred1 counts are uneven across configurations. DatasetModeln TP loc FP hitAB CDE MS-TestQwen 3-32B52186.948.8 240 213 114611 Qwen 3-4B81881.449.9 383 286 24 10124 Qwen 3-8B45278.841.5 202 182 19436 Gemma 3-27B66875.743.7 273 221 23 13318 Gemma 3-4B103375.258.0 507 269 90 10760 MedGemma 27B31173.342.9 14592 115310 Phi-4-mini76172.345.4 289 260 60 10745 Mistral 7B102771.050.7 436 292 82 15760 MedGemma 4B52468.037.8 178 180 19 11928 MediPhi 4B83063.845.5 311 218 63 14692 Llama 3.1-70B39762.936.3 126 143 169220 UltraMed-8B25062.530.37190 124928 UltraMed-70B21359.835.46572 115312 Llama 3.1-8B61656.042.4 196 141 51 14385 Llama 3.2-3B100746.034.0 257 208 85 273 184 MedErrBench-EN Qwen 3-32B25596.890.2 22422540 Qwen 3-8B20695.695.3 1887650 UltraMed-70B14595.185.8 11226700 MedGemma 27B16294.886.2 12620871 Qwen 3-4B34594.088.3 29530 1082 Gemma 3-27B35692.881.7 27752 11142 Gemma 3-4B39890.889.2 33031 2584 Llama 3.1-70B25687.479.4 18538 15162 Phi-4-mini33481.978.8 22153 42126 MediPhi 4B33481.582.1 23637 39139 Llama 3.2-3B32680.782.3 23134 381013 Mistral 7B38280.476.0 25947 30388 UltraMed-8B11480.366.86825 1065 MedGemma 4B21580.079.4 14624 28161 Llama 3.1-8B27179.769.6 16249 202911 MedErrBench-CN Qwen 3-4B18593.080.6 14428490 UltraMed-70B3692.186.3228240 Qwen 3-32B9490.374.96915280 Phi-4-mini32286.287.8 26216 20240 Gemma 3-4B34082.279.1 24235 26343 Llama 3.1-8B12475.971.86915 13198 MedGemma 4B15370.163.292178351 Gemma 3-27B27269.565.0 14741 24573 MedGemma 27B13467.859.880222264 Llama 3.1-70B12063.849.971281191 Mistral 7B34559.457.9 16937 308227 Qwen 3-8B12254.950.384187112 MediPhi 4B30151.151.9 12730 319122 (continued on next page) 33 Toward Better Assessment of LLMs’ Performance in Clinical Error Detection (Table 15 continued from previous page) DatasetModeln TP loc FP hitAB CDE Llama 3.2-3B17149.257.37917 273117 UltraMed-8B3542.529.49113111 MedRECT-JAQwen 3-32B27091.844.8 115 1275149 Qwen 3-4B49368.741.7 188 150 18 12116 Qwen 3-8B42364.745.5 139 130 349525 Llama 3.1-70B33564.633.869 108 15 12122 MedGemma 27B32358.732.084 110 179319 Gemma 3-27B65653.822.2 119 231 25 22061 Gemma 3-4B66640.023.9 102 163 56 27174 Phi-4-mini50035.124.567 109 56 164 104 Llama 3.1-8B54932.128.58585 66 183 130 Llama 3.2-3B58825.117.17575 31 265 142 MedGemma 4B52823.018.04670 45 27691 UltraMed-70B10621.116.8101784823 UltraMed-8B5920.18.42952815 Mistral 7B43119.524.24441 62 19094 MediPhi 4B58618.313.73572 44 287 148 34

Instant research discovery