Paper deep dive
ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
Yuyang Dai, Yuting Tang, Futoshi Futami, Takashi Ishida
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.
Tags
Links
- Source: https://arxiv.org/abs/2610.17016v1
- Canonical: https://arxiv.org/abs/2610.17016v1
Trouble viewing inline? Open PDF directly →
Full Text
95,388 characters extracted from source content.
Expand or collapse full text
ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty Yuyang Dai 1 Yuting Tang 2,3 Futoshi Futami 4,2,3 Takashi Ishida 3,2 1 Florida State University 2 The University of Tokyo 3 RIKEN AIP 4 The University of Osaka § ProbabilityBench 0.1110 0.10 0.20 0.30 0.40 KL divergence (nats) (a) Peer review TerraLuna DecisionsLiquid d1 Perplexity Decider Kev 4B 0.1110 0.40 0.60 0.80 1.00 (b) Covert transfer Jev Haiku Sonnet Perplexity Decider 0.1110 0.10 0.20 0.30 0.40 (c) Credit risk Jev Sonnet Luna Decisions Liquid d1 Perplexity Decider List price, USD per 1,000 predictions Jev 1.13 Noul GPT-6 Luna Decisions Liquid d1 Perplexity Decider V1.1 27B Clef Clef Flash Solar Decide Kev 4B GPT-5.6 Luna · medium Qwen3.8 Flash · off Mistral Small 3.2 · off DeepSeek V4 Flash · off Claude Haiku 5.5 · off Gemini 3.8 Flash · medium GPT-5.6 Terra · medium Claude Sonnet 5.5 · medium Pareto frontier (red) Figure 1: Direct evaluation of model probabilities against exact class probabilities using KL divergence. We compare 16 models given 25 labeled memos per test case (means over three seeds). Filled markers denote decision models; open markers denote generative models. Red lines connect the observed Pareto frontiers. Lower price and KL are better. Abstract Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost- quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves. 1 Introduction Reliable machine learning models need both accurate predictions and probability estimates that reflect uncertainty (Bishop, 2006; Niculescu-Mizil and Caruana, 2005). This requirement applies to traditional probabilistic classifiers, language arXiv:2610.17016v1 [cs.LG] 12 Oct 2026 ProbabilityBench Peer review· Peer Review Memo푃(accept | 푥 vis ) = 0.860 This is a theoretical paper. The method builds on existing techniques with modest changes. The problem it addresses is of narrow interest. The technical argument is sound. The strength of supporting evidence is rated 3.8 out of 5. The comparison with prior theoretical results and their assumptions is only partly developed. The conclusions hold under some changes in the assumptions and not others. The work could be partially reproduced from the details provided. The writing-clarity rating is 3.9 out of 5. The manuscript was submitted in July as revision 1 by 3 authors. It runs 12 pages with 4 figures, 3 tables, and 38 references, plus 17 pages of appendix. Question: Will the submission be accepted? Covert transfer· Covert Transfer Audit Memo푃(encoding | 푥 vis ) = 0.765 A worker AI runs experiments for a confidential research project and files three kinds of report with an authorized reviewer: progress updates, experiment results, and error reports. Only that reviewer can read a report’s body, but every report is stored under a file name that includes a 16-letter run identifier the worker generates itself, written in four groups of four, and file names are visible to the other agents on the shared system. A worker that wanted to leak the project’s confidential tag to another agent could therefore write the tag into the identifier. The audit asks whether this worker did so. The report is an error report. The audit sees that the protected tag lies in the broad range a-h. The report’s run identifier isbygo-uzdl-bvqu-hrjt. The audit record is dated March and covers report revision 2 with 8 attachments. It was logged in the East region under export version 1 in the extended retention tier, was reviewed by a team of 1, and runs in the secondary queue. Question: Was the encoding procedure used to insert the confidential project tag into this report’s run identifier? Credit risk· Credit Review Memo푃(default | 푥 vis ) = 0.123 The company is a healthcare business, and the economy is in a contraction. Operations generate strong cash flow. Access to refinancing is limited. Recurring debt service is a moderate burden. Interest rates reset within the next 12 months on 5.0% of outstanding debt. The three largest customers account for 80.7% of annual revenue. The company has considerable room to adjust its operations. The filing was reported in September for a fiscal year ending in December and is document revision 3. It covers 1202 employees across 1 operating site and 6 administrative departments, and runs 24 pages with 7 charts. Question: Will the company default on its debt obligations within the next year? Figure 2: Full memos from ProbabilityBench’s three worlds. Each world fully specifies input and label generation, making 푃(푌 | 푋 vis )exactly known by design. Header probabilities are withheld from models and provide exact targets for evaluating their reported probabilities. Since a single memo does not fully describe how outcomes are generated, we give models access to example memos paired with observed outcomes (hard labels), either as in-context demonstrations or as training data available to an agent, so they can learn the underlying relationships. models asked to report confidence (Geng et al., 2024), and native decision models such as Jev (TypeSafe AI, 2026). Yet evaluating those probabilities on real-world data is difficult. Each input푋typically has only one observed label푌, which does not reveal the underlying conditional distribution푃(푌 | 푋). Calibration is therefore commonly assessed by grouping similar predictions and comparing reported confidence with empirical label frequencies (Guo et al., 2017; Nixon et al., 2019; Kumar et al., 2019). These finite-sample comparisons assess agreement within groups, not probability accuracy for each input. Strictly proper scoring rules, such as log loss, instead have expected loss uniquely minimized by the true conditional distribution (Gneiting and Raftery, 2007). In practice, expected log loss is estimated by averaging over a finite test set, typically with one observed label per input. Model comparisons therefore depend on which labels happen to be observed. Moreover, expected log loss combines error in the predicted probabilities with irreducible task uncertainty, whose contribution remains unknown without the true conditional distribution. Consequently, the loss alone does not directly reveal how close the predicted probabilities are to the truth. This irreducible uncertainty also limits classification accuracy. The Bayes error is the lowest misclassification error achievable by any predictor using the available input (Fukunaga and Hostetler, 1975; Mohri et al., 2018). The gap between a model’s error and the Bayes error shows how much its predictions could improve using the same input information. On real-world data, however, the Bayes error must itself be estimated, leaving no exact reference against which to check an estimator (Ishida et al., 2023; Ushio et al., 2026). Synthetic distributions with tractable Bayes errors, such as Gaussian mixtures, allow this check but do not directly provide text-based tasks for evaluating the probabilities reported by language models. Image datasets with repeated human annotations, such as CIFAR-10H (Peterson et al., 2019) and Fashion-MNIST-H (Ishida et al., 2023), provide empirical label distributions, but do not directly support uncertainty evaluation for language models on text-based tasks. ChaosNLI (Nie et al., 2020) provides repeated annotations for natural-language inference, but across these datasets, reference probabilities are estimated from finite annotations and 2 ProbabilityBench can depend on annotation protocols (Ushio et al., 2026). Uncertainty in these reference probabilities can obscure small differences in probability quality between similarly performing models. These limitations motivate a text-based benchmark with exact uncertainty targets. We introduce ProbabilityBench, a framework for constructing natural-language tasks with exact class probabilities and Bayes errors. Each simulated world specifies a distribution over latent factors and a deterministic rule for the outcome. Models read memos that reveal only part of the information determining that outcome. Different latent states can therefore produce the same memo but different labels. Marginalizing over the hidden information gives the exact class probability 푃(푌 = 푦 | 푋 vis )for every example, from which we compute the world’s Bayes error. We instantiate the framework in peer-review outcome prediction, AI-safety audits of covert information transfer, and financial default prediction. Each world has Low, Medium, and High Bayes-error settings, obtained by progressively hiding more information (Fig. 2). We compare reported probabilities with exact targets using expected KL divergence and assess cost-quality trade-offs (Fig. 1). ProbabilityBench also supports meta-evaluation of uncertainty quantification methods by measuring Bayes-error and calibration estimators against references computed from exact class probabilities. We further study a safety-relevant failure mode in agentic evaluation: agents can inflate benchmark scores by exploiting access to test answers (Lodkaew et al., 2026; Zhong et al., 2026; Stein et al., 2026). Agents can achieve perfect test accuracy by looking up the observed answers. Yet, when we score the same predictions against each input’s true class probabilities, their average accuracy remains below the best achievable from the visible information (Bayes accuracy). Our main contributions are: •We provide three text-based worlds with exact class probabilities and Bayes errors, together with a recipe for constructing further worlds (Fig. 3). We show that rendering preserves class probabilities when the memo retains all visible attributes and reveals no additional information about the outcome (Sec. 3.4). • We compare eight decision models and eight generative models. Perplexity Decider is cheapest in every world, while decision models occupy most observed cost–quality frontier points. GPT-5.6 Terra has the lowest mean KL in peer review, and Sonnet 5.5 in covert transfer and credit risk. A focused comparison shows that more labeled examples improve Jev’s probability estimates in peer review and credit risk, with little change in covert transfer (Fig. 4). • We evaluate uncertainty estimators using known class probabilities and Bayes errors, with independent sampled references for calibration integrals. In peer review and credit risk, nearest-neighbor Bayes-error estimates often remain above the truth even at 4000 samples, and empirical multicalibration error is positive even for the true class probabilities. We also measure how repeated-label snapshots approximate higher-order calibration targets (Sec. 5). 2 Related Work Uncertainty benchmarks.Recent benchmarks study uncertainty in language and multimodal models, including aleatoric and epistemic uncertainty (Yang et al., 2025; Chandu et al., 2025; Wang et al., 2025; Renda et al., 2026; Tomov et al., 2025). Their reference targets include answer correctness, answerability annotations, and empirically estimated statistics or answer distributions. A second family imposes a known performance ceiling on otherwise determined tasks: CapBencher (Ishida et al., 2026) records one of several valid answers at random, CapCode (Lodkaew et al., 2026) randomizes unit tests, and ImpossibleBench (Zhong et al., 2026) puts task specifications in conflict with tests, so that scores above the ceiling reveal test-set overfitting or exploitation. BLInD (Nafar et al., 2025) provides reference probability values computed by exact inference in a Bayesian network. Its input text states the network’s probabilities and asks the model to compute the probability of a queried event. It therefore tests probabilistic reasoning rather than a model’s own uncertainty about an outcome. ProbabilityBench instead withholds the generating probabilities from the model and evaluates its estimates from case descriptions and labeled examples. In concurrent work, TypedBench (Sharma et al., 2026) similarly uses synthetic generators to evaluate decision models, with reference labels derived from explicit policy rules. It does not provide per-example conditional class probabilities as evaluation targets. ProbabilityBench instead derives irreducible uncertainty from the specified data-generating process, so each text-rendered example carries an exact class probability and each setting an exact Bayes error. These exact targets support direct evaluation of aleatoric uncertainty and calibration. Calibration and confidence evaluation. Calibration of neural networks is classically measured with binning-based estimators such as expected calibration error (Guo et al., 2017), whose statistical fragility has motivated refined estima- tors (Nixon et al., 2019; Blasiok and Nakkiran, 2024) and verified calibration procedures (Kumar et al., 2019). For language models, a parallel literature studies whether models know what they know, via self-evaluation (Kadavath et al., 2022), 3 ProbabilityBench verbalized uncertainty (Lin et al., 2022; Tian et al., 2023), and systematic comparisons of elicitation strategies (Xiong et al., 2024). See Geng et al. (2024) for a survey. When conditional label probabilities are unknown, population calibration quantities generally require estimation from observed labels. Multicalibration (H ́ ebert-Johnson et al., 2018) requires calibration across subgroups. Empirical evaluations use subgroup calibration errors (Hansen et al., 2024) or unbiased estimates of witness moments (Futami and Ishida, 2026). Both also rely on observed labels. Repeated-label observations also support richer assessments of uncertainty. Mimori et al. (2021) develop evaluation methods for label uncertainty and higher-order statistics, including annotator disagreement, and calibrate distributions over class probabilities. Johnson et al. (2024) connect prediction of paired responses to second-order calibration. Ahdritz et al. (2024) study higher-order calibration against the full distribution of conditional label probabilities and develop relaxations based on푘repeated labels per input. ProbabilityBench complements these approaches by supplying exact conditional probabilities and finite-푘binomial laws at each input. Independent reference inputs then approximate both population mixtures within each prediction cell, enabling evaluation against both targets without estimating per-input probabilities from labels. Bayes error estimation. A long line of work estimates the Bayes error of a task from data (Fukunaga and Hostetler, 1975; Devijver, 1985; Berisha et al., 2015; Noshad et al., 2019; Theisen et al., 2021; Michelucci et al., 2021; Ishida et al., 2023; Ushio et al., 2026). Because the true Bayes error of a real dataset is unknown, these estimators are validated on synthetic distributions with a closed-form Bayes error or, on real data, through proxies such as injected label noise (Renggli et al., 2021). The latter treats the test error of state-of-the-art models as an upper bound on the Bayes error, which holds only if those models have not overfit the test set, an assumption that is becoming hard to maintain with language model training (Ishida et al., 2026) and agents (Lodkaew et al., 2026). 3 The ProbabilityBench Framework 3.1 Problem Formulation and Metrics We consider binary classification in which a latent world state푍generates both a random vector of observable attributes푋 (for example, a company’s cash buffer or a submission’s novelty) and an outcome label푌∈0, 1. In any partial-observability setting, the visible subvector푋 vis is revealed to the predictor, while the remaining subvector푋 hid is withheld. For each visible input, the world defines the class probabilities 푃(푌 = 푦 | 푋 vis ). For a model with parameters휃, write푓 휃 for its hard classifier, which predicts a single class label in0, 1, and푝 휃 (푌 | 푋 vis ) for its reported probability distribution. Fix a hiding level and write푈 = 푋 vis with supportU,푝 ∗ (푢) = 푃(푌 = 1 | 푈 = 푢), and푞(푢) = 푝 휃 (푌 = 1 | 푈 = 푢). Accuracy alone says nothing about whether this reported confidence reflects the task’s true uncertainty. We therefore use several metrics, defined below, each computed from exact class probabilities. Bayes accuracy and model accuracy. The Bayes-optimal predictor under 0–1 loss, with ties broken in favor of class 1, is푓 ∗ (푥 vis ) = arg max 푦∈0,1 푃(푌 = 푦 | 푋 vis = 푥 vis ), with corresponding Bayes accuracyA ∗ =E 푋 vis [max 푦∈0,1 푃(푌=푦 | 푋 vis )]and Bayes error휀 ∗ = 1−A ∗ . Expected accuracy scores a model’s hard decision against the exact conditional distribution rather than against one sampled label, and Δ acc is its gap to Bayes accuracy: A exp ( 푓 휃 ) =E 푋 vis [ 푃 ( 푌 = 푓 휃 (푋 vis ) | 푋 vis )] , Δ acc =A ∗ −A exp ( 푓 휃 ).(1) Unlike realized accuracy, the fraction of test inputs whose single sampled label the model predicts correctly,A exp is upper-bounded byA ∗ example by example, so it is the main accuracy metric for Bayes-gap comparisons. Realized accuracy can exceedA ∗ through label-sampling noise. A large excess can also indicate overfitting to the realized test labels, which we investigate by inspecting the generated programs (Appendix C.1). KL divergence from exact class probabilities.Because푝 ∗ (푢)is known for every input, a reported probability푞(푢)can be compared with it input by input. We use the expected KL divergence from the exact class probability, in nats: KL(푝 ∗ ∥ 푞) =E 푈 푝 ∗ (푈) log 푝 ∗ (푈) 푞(푈) + 1− 푝 ∗ (푈) log 1− 푝 ∗ (푈) 1− 푞(푈) .(2) It is zero only if푞 = 푝 ∗ almost surely, and it equals the expected log loss of푞minus that of푝 ∗ , the part of the log loss a model could still avoid. Averaging the summand over test inputs estimates it without any sampled labels. Appendix A.2 gives further evaluation details. 4 ProbabilityBench Calibration and multicalibration. Ordinary calibration requiresE[푌 | 푞(푈)] = 푞(푈)almost surely. Multicalibra- tion (H ́ ebert-Johnson et al., 2018) requires the same property within every group in a fixed finite collectionGof possibly overlapping subsets ofU. For evaluation, we use the group and score-bin setting of H ́ ebert-Johnson et al. (2018). For a fixed partitionB =퐼 1 ,..., 퐼 푚 of [0, 1], let 퐵 푗 =푢 : 푞(푢) ∈ 퐼 푗 and define MC G,B (푞) =max 퐺∈G, 1≤푗≤푚 E 1 퐺 (푈)1 퐵 푗 (푈)(푝 ∗ (푈)− 푞(푈)) .(3) Each moment is weighted by the probability of its group–bin cell in the whole population, so a small cell contributes less for the same conditional calibration gap. Zero-mass cells contribute zero. Finite bins can hide residuals that cancel within a bin. The empirical estimator replaces the expectation with an average over all sampled examples and푝 ∗ (푈)with observed labels, keeping groups and bins fixed. Absolute values and the maximum can turn sampling noise into positive estimated error even for a perfectly calibrated predictor. For our worlds’ broad numerical supports, predictor-dependent quantities use independent sampled input references with exact conditional label probabilities. Their integration error is distinct from the empirical estimator’s label-sampling error. Unlike Eq. 2, this criterion constrains only averages of푝 ∗ − 푞. By Pinsker’s and Jensen’s inequalities, each moment in Eq. 3 is at most √︁ KL(푝 ∗ ∥ 푞)/2, so a small KL forces a small MC error. Reporting the class prior푃(푌=1)everywhere is ordinarily calibrated, yet its KL is the mutual information between푈and푌. Such a predictor need not be calibrated within each group. WithG =U, replacing the maximum over bins in Eq. 3 by a sum gives the population version of binned expected calibration error (ECE) for the positive-class probability (Guo et al., 2017). Higher-order calibration and snapshots. LetCbe a finite partition ofUinto cells of positive population mass. A higher-order predictor assigns a probability measure휈 퐶 on[0, 1]to each input in cell퐶: it predicts a distribution over label probabilities. Following Ahdritz et al. (2024), its target is the Bayes mixture 휋 ∗ 퐶 =L 푝 ∗ (푈) | 푈 ∈ 퐶 ,(4) whereLdenotes the distribution of a random variable. We measure the distance from휈 퐶 to휋 ∗ 퐶 by the Wasserstein distance 푊 1 with cost|푝− 푝 ′ | on [0, 1], half the binary-simplex ℓ 1 cost. A푘-snapshot consists of푘independent labels conditional on the same input, represented in the binary case by their positive- label proportion. The snapshot projection of a mixture휋isproj 푘 휋 =L(푆/푘), where푝 ∼ 휋and푆 | 푝 ∼ Binomial(푘, 푝). A snapshot predictor휈 퐶 is supported on0, 1/푘,..., 1. Our experiment compares the same fitted snapshot predictor with both targets using population-weighted mean errors, 퐸 HO (휈) = ∑︁ 퐶 푤 퐶 푊 1 (휈 퐶 , 휋 ∗ 퐶 ),퐸 푘 (휈) = ∑︁ 퐶 푤 퐶 푊 1 (휈 퐶 , proj 푘 휋 ∗ 퐶 ),(5) where푤 퐶 = 푃(푈 ∈ 퐶). These are mass-weighted averages over cells, whereas Ahdritz et al. (2024) bound the worst cell. We use bins of a fixed scalar predictor to defineC. These need not be its exact level sets. We estimate the population integrals in Eqs. 3–5, including휋 ∗ 퐶 andproj 푘 휋 ∗ 퐶 , on independent sampled input references. Each input’s class probability and conditional binomial law are exact, while the population bin masses and mixtures retain Monte Carlo error. 3.2 A Recipe for Worlds with Exact Class Probabilities Every ProbabilityBench world follows the same construction (Fig. 3). The world is represented by finitely many latent factors푍 = (푍 1 ,..., 푍 푘 ), each taking finitely many values, and an author-specified, dependency-structured joint distribution 푃(푍). The label is a deterministic function푌 = 푔(푍), for example a score over latent factors compared to a fixed threshold, so all label uncertainty stems from not knowing 푍 . Observable attributes are then computed from푍by a deterministic projection that drops or merges information, for example by withholding some factors entirely. Because the projection is many-to-one, distinct latent states share identical attributes, so the attributes do not determine푍. Finally, each world keeps label uncertainty that survives even when every attribute is shown, so population Bayes error stays positive: the factor worlds include a residual factor, a component of푍that enters the label but is never shown, and covert transfer admits accidental matches (Sec. 4.2). Individual inputs may still have deterministic outcomes. Exact class probability computation. Write휙(푧)for the visible attributes when the world is in latent state푧at the current hiding level: the attributes are computed from푧by the deterministic projection above, and those withheld at that level are dropped (Sec. 3.4 renders the rest as a memo). Both the label and the visible attributes are therefore deterministic 5 ProbabilityBench residual푅 never rendered substantive levels and measurements irrelevant fields latent state 푍 ∼ 푃(푍) outcome rule 푔 label푌 = 푔(푍) attributes 푋 푋 vis memo 푇 = 푟(푋 vis , 퐼) index 퐼 levels hiding level rendering 휙 Figure 3: How a world generates a case. Both푔and휙are deterministic. ML models read푋 vis , while language models and agents read only the memo 푇 . functions of the latent state,푌 = 푔(푍)with푔(푧) ∈ 0, 1the outcome rule and푋 vis = 휙(푍). A deterministic휙loses no generality: randomness in the visible attributes can be written as further latent factors that푔ignores. Here,푍includes both the irrelevant fields and the numerical measurements sampled conditional on substantive factor levels. The outcome rule uses those levels, while the attribute map displays the measurements. The sampling is therefore part of푃(푍), and휙 remains deterministic. For any visible input 푥 vis with 푃(푋 vis =푥 vis ) > 0, 푃(푌=1 | 푋 vis =푥 vis ) = 푃(푌=1, 푋 vis =푥 vis ) 푃(푋 vis =푥 vis ) = 푃 푔(푍)=1, 휙(푍)=푥 vis 푃 휙(푍)=푥 vis = Í 푧:푔(푧)=1, 휙(푧)=푥 vis 푃(푍=푧) Í 푧: 휙(푧)=푥 vis 푃(푍=푧) .(6) The denominator adds up the probability of every latent state producing푥 vis , and the numerator keeps those whose outcome is positive. Every population quantity of Sec. 3.1 is likewise a finite sum over visible inputs. For instance, writing푝 ∗ (푥 vis ) for the class probability in Eq. 6, 휀 ∗ = ∑︁ 푥 vis 푃(푋 vis =푥 vis ) min푝 ∗ (푥 vis ), 1− 푝 ∗ (푥 vis ). The two kinds of world evaluate Bayes error and class probabilities differently. In the factor worlds, we enumerate the latent categorical factors. Within-category numerical variation and irrelevant fields cancel from the conditional class probabilities, so the numerical measurements need not be enumerated for these targets. Covert transfer has finite푍but far too many inputs to enumerate (the run identifier alone has26 16 ≈ 4× 10 22 values), so we marginalize analytically: only the letter at the report’s position is label-relevant and the rest cancel, leaving the closed form of Sec. 4.2. 3.3 Controlling task difficulty Hiding-based difficulty control alone conflates two conceptually distinct sources of irreducible uncertainty: information limitation (hidden attributes contain signal the model cannot access) and conditional label noise (even when every attribute is shown, the outcome still depends on the residual factor). The factor worlds expose the two independently. Hiding controls the first source: the levels are nested, each withholding more attributes than the last, and exact computation confirms that휀 ∗ strictly rises from Low to Medium to High (Appendix B). The residual factor controls the second: in the factor worlds its distribution is a free parameter, and putting more mass on its nonzero values increases휀 ∗ without changing what the model sees (Table 7). Jointly, the two axes let a benchmark creator target a desired Bayes error through qualitatively different routes (rich information with high label noise, or sparse information with none), which induce different class probabilities and therefore different calibration targets. Real-data benchmarks offer no such control because the true class probability is unknown. 3.4 Preserving Class Probabilities in Text All three worlds render their visible attributes as natural-language case memos (review, audit, and credit memos), so that language models are evaluated on text rather than on attribute tables. Free-form text raises an obvious concern: does the exact class probability still apply to the rendered question? We make the transfer provable with two conditions on the renderer. Deterministic template renderers satisfy both properties by construction. 6 ProbabilityBench First, the text leaks nothing beyond the visible attributes. The rendered text is푇 = 푟(푋 vis , 퐼), where푟is a deterministic function of the visible attribute assignment only, never of latent factors or hidden attributes, and퐼is the example’s position in the sample, which a renderer may use for stylistic variation. Our renderers use a fixed phrase for each visible attribute and ignore퐼. Because examples are drawn i.i.d. from the world,퐼is independent of(푍,푌, 푋). Consequently푇 ⊥ (푌, 푍) | 푋 vis , so the text cannot carry more label-relevant information than the visible attributes. Second, the text loses nothing from them: each attribute value is rendered by its own fixed phrase that does not depend on퐼, so the visible attributes can be read back from the memo,푋 vis = ℎ(푇)for a deterministic decoderℎ(Appendix B describes ℎ). Together the two conditions give푃(푌 | 푇) = 푃(푌 | 푋 vis ): by the second, conditioning on푇is conditioning on푋 vis and 퐼, and by the first 퐼 adds nothing about푌 . 4 Three Benchmark Worlds We instantiate the recipe of Sec. 3.2 in three domains with different semantics: a committee decision over manuscripts, a safety audit of an AI worker’s message, and credit-risk assessment. Table 6 in Appendix B compares them, and Table 2 gives every setting’s visible information, exact Bayes error, and conditional label entropy. Low, Medium, and High denote relative Bayes error, not learning difficulty. Shared structure of the factor worlds. Peer review and credit risk each use eight factors that affect the outcome. Each factor has three ordered levels. Six factors appear in the memo as categorical assessments and two as numerical measurements. Two thresholds map each numerical measurement back to its level. Low reveals all eight factors, Medium hides one, and High hides that factor plus a second. Thus the same case can be presented with progressively less information. The outcome also depends on a hidden random adjustment푅, called the residual. It is never included in the memo, so even Low has positive Bayes error. In peer review,푅takes integer values from−5to5: it is zero with probability0.72, and each of±1,...,±5has probability.09,.03,.015,.004,.001, respectively. In credit risk,푅takes values−3,−1, 0, 1, 3with probabilities(.03,.09,.76,.09,.03). Irrelevant fields. Each input in all three worlds also carries 8 fields, such as submission month or employee count, that are mutually independent and independent of every substantive factor and of the residual. Across the worlds, 13 count fields have bounded integer distributions. Submission month is uniform over all twelve months, and the remaining fields take two equiprobable categorical values. They change neither any exact class probability nor the Bayes error. Models must learn which fields are informative because the memo does not identify the irrelevant ones. Numerical representation. Review uses supporting-evidence and writing-clarity ratings, both out of 5. In credit risk, customer concentration is expressed as the percentage of annual revenue from the three largest customers, and funding-rate exposure as the percentage of outstanding debt repricing within 12 months. Values are sampled uniformly within their latent level’s numerical interval, independently of other variables conditional on that level. Each measurement identifies its latent level exactly, and variation within a level carries no additional information about the outcome. Appendix B specifies the grids and distributions. 4.1 Peer-Review Outcome A model predicts whether a submission will be accepted. The contribution type (theoretical, empirical, applied) is always visible and sets the weights of 8 quality factors: novelty, problem significance, technical validity, supporting evidence, comparison quality, robustness, reproducibility, and clarity. Supporting evidence combines strength and coverage and is more likely to be weak when validity is flawed, while novelty, significance, and clarity remain independent. Theoretical memos describe comparisons with prior results and robustness to changes in assumptions. Theoretical work weights validity, evidence, and novelty most, empirical work evidence, validity, and comparisons, and applied work evidence and significance. Two interactions reward strong evidence backed by well-controlled comparisons, and sound validity backed by stable results. A submission is accepted when its score plus the hidden residual reaches11.5, unless it has a serious validity flaw, which forces rejection. The residual represents variation in decisions beyond the listed factors, motivated by the differences between independent reviewing committees documented by Cortes and Lawrence (2021). Medium hides novelty and High also hides technical validity. The acceptance prior is 0.241. 7 ProbabilityBench 4.2 Covert Information Transfer This AI-safety world audits a worker for covert information transfer. A worker AI files reports with an authorized reviewer, and a confidential project tag, one of 26 letters, must not reach the other agents on the shared system. Only the reviewer can read a report’s body, but every report is stored under a file name containing a 16-letter run identifier that the worker generates itself, and file names are visible to all agents. A worker could therefore leak the tag to a colluding agent by writing it into the identifier. The label records whether this encoding procedure was used. The report type (progress update, experiment result, or error report), tag, and 16 identifier letters are drawn independently and uniformly, and the worker encodes with probability1/2, independently of them. Under encoding, the tag replaces the letter at a position fixed by the report type: 5 for a progress update, 9 for an experiment result, and 13 for an error report. An accidental match is negative, and an encoding that replaces a letter with the same letter is positive, so identical inputs carry different labels. The model sees the report type, the identifier, and audit information about the tag. The memo does not state the positional rule. Low reveals the exact tag, Medium a 4-letter range, and High an 8-letter range (the last range of each partition is shorter). If the visible range has푚letters and contains the identifier’s letter at the report’s position,푃(encoding | 푥 vis ) = 26/(26+푚). Otherwise it is zero. The other 15 letters cancel, so the class probability stays exact although the input space is far too large to enumerate. Bayes error is Í 푖 푚 2 푖 /(2· 26 2 ) over the range sizes푚 푖 :0.019,0.074, and0.145. Every identifier is freshly drawn, so test inputs essentially never repeat training inputs, and generic encodings that treat an identifier as one categorical value carry no signal. 4.3 Credit Risk A model assesses whether a company will default on its debt obligations within the next year. The sector (4 values) and business cycle (expansion, steady, contraction) are always visible. The company’s position is described by 8 factors: operating cash generation, cash buffer, and refinancing access provide resources, while recurring debt service and near-term principal maturities create obligations. Funding-rate exposure, customer concentration, and operating flexibility modify them. Default occurs whencash generation+ cash buffer+ refinancing+ 푅 < debt service+ principal due, where푅is the residual. The rule is not additive in the observed factors: in a contraction, cash generation falls further with customer concentration, by a sector-dependent amount, while operating flexibility partly protects it, and the business cycle scales both refinancing access and the cost of funding-rate exposure. Medium hides the cash buffer and High also hides the principal maturities. The default prior is 0.211. 5 Experiments Setup. Across the three worlds, inputs include the four relevant and 13 irrelevant numerical features described in Sec. 4. We compare eight native decision models and eight generative models using 25 labeled examples as in-context demonstrations at Medium, with 300 held-out cases per world for each of three seeds. All models receive the same evidence and report positive-class probabilities (Fig. 1; Appendix A.3). We also track how more demonstrations change Jev and GPT-5.6 Luna (Fig. 4; Appendix A.2). For accuracy, we compare logistic regression (traditional ML) with an agent that synthesizes a Python predictor from training examples, and with a deliberately cheatable variant of that agent given labeled test-set access (Appendix A.1). This comparison uses 50 training and 500 test examples for each of three prospectively chosen seeds at Low, Medium, and High. Every input carries the 8 irrelevant fields. Agents receive memo text and realized labels, while ML receives structured attributes. These models and agents are not given exact class probabilities. Logistic regression and the Bayes error estimators are omitted in the AI-safety world, where their encodings carry no signal (Appendix A.1). Cost and probability quality. Because every case carries its exact class probability, each model’s probability can be scored by its KL divergence from the truth (Eq. 2; Fig. 1). Perplexity Decider is the cheapest in every world. Decision models account for ten of the fourteen model–world points on the observed frontiers, including Jev and Perplexity Decider. GPT-5.6 Terra has the lowest mean KL in peer review. Claude Sonnet 5.5 has the lowest in covert transfer and credit risk, at much higher list prices. The preferred model therefore depends on both the world and the budget, and higher prices do not consistently imply better probability estimates. These comparisons use mean KL and mean cost at published list rates under matched inputs (Appendix A.3). 8 ProbabilityBench 0102550 0.15 0.20 0.25 0.30 0.35 0.40 0.45 KL divergence (nats) (a) Peer review 0102550 0.45 0.65 0.85 1.05 (b) Covert transfer 0102550 0.10 0.15 0.20 0.25 0.30 (c) Credit risk Labeled demonstrations (shots) TypeSafe AI Jev 1.13 NoulOpenAI GPT-5.6 Luna · medium Figure 4: Probability quality at Medium visibility: lower KL is better. Mean± s.d. Dots: three seeds, 500 cases each. LowMediumHigh 0.7 0.8 0.9 1.0 Test accuracy (a) Peer review LowMediumHigh 0.4 0.6 0.8 1.0 (b) Covert transfer LowMediumHigh 0.7 0.8 0.9 1.0 (c) Credit risk LowMediumHigh 0.0 0.1 0.2 0.3 Bayes error (a) Peer review LowMediumHigh (b) Credit risk Bayes-error settingBayes-error setting Bayes accuracy ML Agent (training only) Agent (test-label access) Exact Bayes error 1N (Cover–Hart) kNN, k= 5 (Devroye) Figure 5: Realized accuracy, 50 training examples. Three seeds: mean± s.d. Covert transfer (b) has no ML baseline. Effect of few-shot demonstrations. A single memo describes a case but does not specify how its world generates outcomes. In-context examples pair other memos with sampled outcome labels, providing evidence about the relationships the model needs to learn. Varying their number tests whether this evidence improves probability estimates for new cases, even though the examples do not reveal true class probabilities. Fig. 4 varies the number of in-context demonstrations over 0, 10, 25, and 50 for Jev, a representative decision model, and GPT-5.6 Luna at medium reasoning, a representative generative model. This focused study complements the broader model comparison in Fig. 1. In peer review, Jev’s mean KL decreases from0.248to0.203as the number of labeled examples rises from zero to 50, while Luna’s increases from0.199 to0.411. Jev remains approximately stable in covert transfer. Both models improve in credit risk, with a larger reduction in KL for Jev. Luna starts with lower KL than Jev in peer review and credit risk without examples, but its response to additional examples varies substantially across worlds and seeds. In peer review and covert transfer, Luna’s mean KL is lowest without examples. Realized accuracy and Bayes accuracy. Fig. 5 reports the fraction of test labels predicted correctly. Unknown predictions count as incorrect. Shading shows sample standard deviation rather than a confidence interval, cropped to each panel’s displayed range. Agents with labeled test access can achieve perfect realized accuracy, exceeding Bayes accuracy. Interpreting such scores as generalization would be misleading: expected accuracy from the visible input cannot exceed Bayes accuracy. Although finite-sample realized accuracy can exceed it by chance, inspection of these programs shows that they often obtain perfect scores through answer lookup or memorized corrections (Appendix C.1). Yet their mean expected accuracy, evaluated against the true class probabilities, remains below Bayes accuracy (Fig. 12 in Appendix C.2). This contrast distinguishes agreement with the observed test labels from decision quality under the true conditional distribution. High benchmark scores alone can conceal exploitation of test-label access. For methods without test access, the largest gaps in expected accuracy relative to Bayes accuracy occur at Low in peer review and credit risk. Low irreducible uncertainty therefore does not by itself imply that these methods approach the attainable accuracy. Neither logistic regression nor the training-only agent is uniformly better across worlds and settings. In covert transfer, hiding the exact tag reduces the training-only agent’s expected accuracy, with particularly large variation across seeds at Medium. Can Bayes error estimators recover the known휀 ∗ ? We reimplement two nearest-neighbor estimators following FeeBee (Renggli et al., 2021): the Cover–Hart 1N bounds and the푘N bounds with Devroye’s binary refinement. Both use structured visible attributes, with categorical fields one-hot encoded and numerical fields scaled to[0, 1]by their declared ranges. We sweep sample size and hiding level and score each estimate directly against the truth. Fig. 6 shows that the estimated intervals often lie entirely above the true Bayes error, especially at Low. Increasing the sample size reduces the discrepancy slowly, and removing irrelevant fields lowers the Low-setting estimates without eliminating the mismatch at Low (Appendix C.3). Sparse neighborhoods can inflate the error used in the lower-bound formulas. Known targets let us characterize how these estimators behave at the sample sizes considered here. Can multicalibration error be estimated from finite samples? Fig. 7 estimates witness MC (Eq. 3) at High using 10 equal-width probability bins and 10 calibration resamples independent of training per sample size. We compare the true class probability푝 ∗ , logistic regression trained on 1000 examples, and the class prior. Groups use contribution type, problem significance, and their conjunctions in peer review, and sector, business cycle, and their conjunctions in credit risk. Categorical irrelevant fields contribute value groups, and numerical counts contribute intervals cut at fixed 9 ProbabilityBench LowMediumHigh 0.7 0.8 0.9 1.0 Test accuracy (a) Peer review LowMediumHigh 0.4 0.6 0.8 1.0 (b) Covert transfer LowMediumHigh 0.7 0.8 0.9 1.0 (c) Credit risk LowMediumHigh 0.0 0.1 0.2 0.3 Bayes error (a) Peer review LowMediumHigh (b) Credit risk Bayes-error settingBayes-error setting Bayes accuracy ML Agent (training only) Agent (test-label access) Exact Bayes error 1N (Cover–Hart) kNN, k= 5 (Devroye) Figure 6: Bayes-error bounds (푛 = 4000), not conf. intervals. 2005001k2k5k Sample size n 0.00 0.02 0.04 0.06 Witness MC error (a) Peer review Multicalibration 2005001k2k5k Sample size n (b) Credit risk Multicalibration True class prob.Logistic regressionClass prior Estimate ± s.d. Sampled ref. (95% bound) Figure 7: Witness MC at High. Solid shading: ± s.d. 1235102050 Snapshot size k 0.00 0.05 0.10 0.15 0.20 Distance ( W 1 ) (a) Peer review Snapshot calibration 1235102050 Snapshot size k (b) Credit risk Snapshot calibration s.d.: training, calibration & reference seeds Higher-order, after Snapshot, after Snapshot, before Sampled target δ k Figure 8: Snapshot and higher-order errors at High. population quartiles, giving 53 and 45 groups respectively, plus the overall population. Even푝 ∗ , whose population MC error is zero, receives positive estimated error at every evaluated sample size in both worlds. The estimates approach their reference values as sample size grows, while the class prior remains substantially worse than logistic regression. A nonzero empirical MC score can therefore reflect sampling variation even for a perfectly calibrated predictor. The zero error of푝 ∗ is exact. Other population references integrate exact class probabilities over one million independent inputs. Their sampling uncertainty is reported separately from resampling variation (Appendix C.4). How well does snapshot calibration approximate higher-order calibration?A푘-snapshot records the positive fraction of푘independent labels for one input. Its distribution includes sampling variation even with unlimited calibration inputs, so fitting it need not recover the distribution of true probabilities. ProbabilityBench supplies exact class probabilities for estimating both targets on an independent input reference, allowing us to test how well snapshot calibration approximates full higher-order calibration as 푘 increases. Fig. 8 reports population-weighted푊 1 errors (Eq. 5) at High over three independent trials (1000 training and 2000 calibration inputs each), together with훿 푘 , the population-weighted distance between the two true targets. All population integrals here use the million-input references. Appendix C.5 gives the protocol following Ahdritz et al. (2024). At small푘, low snapshot error coexists with substantially larger higher-order error that closely tracks훿 푘 . Increasing the number of labels per input brings the targets closer and narrows this gap. A good fit to the snapshot distribution can therefore leave substantial error relative to the distribution of true class probabilities, even when snapshot calibration itself is accurate. 6 Conclusion Evaluating model uncertainty and calibration on real-world data remains fundamentally constrained by the absence of exact conditional class probabilities. ProbabilityBench bridges the gap between simple synthetic distributions with known Bayes errors and natural-language tasks by constructing parameterized text environments with exact ground-truth uncertainty. By explicitly defining the underlying data-generating process, our framework enables direct, instance-level evaluation of reported probabilities and Bayes-error gaps without estimating each input’s class probability from sampled labels. Calibration experiments use those exact probabilities with independent sampled input references. Their population integrals remain estimates. The experiments show how sample size affects estimates of calibration and Bayes error in these worlds. Known targets make it possible to study these effects directly and identify settings in which an estimator is accurate. ProbabilityBench can also help assess the calibration of decision models (TypeSafe AI, 2026), complementing evaluations on real-world data where the true class probabilities are unavailable. The memos are rendered from fixed templates. Richer renderers, including LLM-generated prose, are a harder but meaningful direction for future work: to keep the class probability of every rendered case exact, free-form text must still satisfy both conditions of Sec. 3.4. We chose peer review, covert transfer, and credit risk, but other users will care about other domains. Building the three worlds required substantial manual design and verification. Future extensions could use the construction recipe and existing worlds to automate parts of this process and reduce manual verification effort. We hope ProbabilityBench serves as a place to test evaluation instruments before they are used where the truth is unavailable. 10 ProbabilityBench References Gustaf Ahdritz, Aravind Gollakota, Parikshit Gopalan, Charlotte Peale, and Udi Wieder. Provable uncertainty decomposition via higher-order calibration. arXiv preprint arXiv:2412.18808, 2024. Visar Berisha, Alan Wisler, Alfred O. Hero, and Andreas Spanias. Empirically estimable classification bounds based on a nonparametric divergence measure. IEEE Transactions on Signal Processing, 64(3):580–591, 2015. Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, 2006. Jaroslaw Blasiok and Preetum Nakkiran. Smooth ECE: Principled reliability diagrams via kernel smoothing. In International Conference on Learning Representations, volume 2024, 2024. Khyathi Chandu, Linjie Li, Anas Awadalla, Ximing Lu, Jae Sung Park, Jack Hessel, Lijuan Wang, and Yejin Choi. CertainlyUncertain: A benchmark and metric for multimodal epistemic and aleatoric awareness. In International Conference on Learning Representations, 2025. Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment. arXiv preprint arXiv:2109.09774, 2021. Thomas M. Cover and Peter E. Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13 (1):21–27, 1967. Pierre A. Devijver. A multiclass, k-N approach to Bayes risk estimation. Pattern Recognition Letters, 3(1):1–6, 1985. Luc Devroye. On the asymptotic probability of error in nonparametric discrimination. The Annals of Statistics, 9(6): 1320–1327, 1981. Keinosuke Fukunaga and L Hostetler. k-nearest-neighbor Bayes-risk estimation. IEEE Transactions on Information Theory, 21(3):285–293, 1975. Futoshi Futami and Takashi Ishida.Unified approach for weakly supervised multicalibration.arXiv preprint arXiv:2605.09857, 2026. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 2017. Dutch Hansen, Siddartha Devic, Preetum Nakkiran, and Vatsal Sharan. When is multicalibration post-processing necessary? Advances in Neural Information Processing Systems, 37, 2024. Ursula H ́ ebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning. PMLR, 2018. Takashi Ishida, Ikko Yamane, Nontawat Charoenphakdee, Gang Niu, and Masashi Sugiyama. Is the performance of my deep network too good to be true? A direct approach to estimating the Bayes error in binary classification. In The Eleventh International Conference on Learning Representations, 2023. Takashi Ishida, Thanawat Lodkaew, and Ikko Yamane. CapBencher: Give your LLM benchmark a built-in alarm for test-set overfitting. In International Conference on Machine Learning, 2026. Daniel D. Johnson, Daniel Tarlow, David Duvenaud, and Chris J. Maddison. Experts don’t cheat: Learning what you don’t know by predicting pairs. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research. PMLR, 2024. 11 ProbabilityBench Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022. Thanawat Lodkaew, Johannes Ackermann, Soichiro Nishimori, Nontawat Charoenphakdee, Masashi Sugiyama, and Takashi Ishida. Do coding agents deceive us? Detecting and preventing cheating via capped evaluation with randomized tests. arXiv preprint arXiv:2606.07379, 2026. Umberto Michelucci, Michela Sperti, Dario Piga, Francesca Venturini, and Marco A Deriu. A model-agnostic algorithm for Bayes error determination in binary classification. Algorithms, 14(11):301, 2021. Takahiro Mimori, Keiko Sasada, Hirotaka Matsui, and Issei Sato. Diagnostic uncertainty calibration: Towards reliable machine predictions in medical domain. In International Conference on Artificial Intelligence and Statistics. PMLR, 2021. Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, Cambridge, MA, second edition, 2018. Aliakbar Nafar, Kristen Brent Venable, and Parisa Kordjamshidi. Reasoning over uncertain text by generative large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025. Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, 2005. Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020. Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning. arXiv preprint arXiv:1904.01685, 2019. Morteza Noshad, Li Xu, and Alfred Hero. Learning to benchmark: Determining best achievable misclassification error from training data. arXiv preprint arXiv:1909.07192, 2019. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. Alana Renda, Jillian Ross, Michael Cafarella, and Jacob Andreas. OpenEstimate: Evaluating LLMs on reasoning under uncertainty with real-world data. In International Conference on Learning Representations, volume 2026, 2026. Cedric Renggli, Luka Rimanic, Nora Hollenstein, and Ce Zhang. Evaluating Bayes error estimators on real-world datasets with FeeBee. arXiv preprint arXiv:2108.13034, 2021. Rahul Sharma, Andrew B. Duncan, Ga ́ etan Marceau Caron, and Sebastian J. Vollmer. TypedBench: A benchmark for calibration, framing sensitivity, and cost in System One decision models. arXiv preprint arXiv:2610.11392, 2026. Adam Stein, Davis Brown, Hamed Hassani, Mayur Naik, and Eric Wong. Detecting safety violations across many agent traces. arXiv preprint arXiv:2604.11806, 2026. Ryan Theisen, Huan Wang, Lav R. Varshney, Caiming Xiong, and Richard Socher. Evaluating state-of-the-art classification models against Bayes optimality. In Advances in Neural Information Processing Systems, 2021. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 12 ProbabilityBench Tim Tomov, Dominik Fuchsgruber, Tom Wollschl ̈ ager, and Stephan G ̈unnemann. The illusion of certainty: Uncertainty quantification for LLMs fails under ambiguity. arXiv preprint arXiv:2511.04418, 2025. TypeSafeAI.IntroducingSystemOnemodels&Jev.https://typesafe.ai/blog/ introducing-system-one-models-and-jev, September 2026. Blog post, accessed 2026-09-21. Ryota Ushio, Takashi Ishida, and Masashi Sugiyama. Practical estimation of the optimal classification error with soft labels and calibration. In International Conference on Learning Representations, volume 2026, 2026. Sydney Von Arx, Lawrence Chan, and Beth Barnes. Recent frontier models are reward hacking.https://metr.org/ blog/2025-06-05-recent-reward-hacking/, June 2025. METR blog post, accessed 2026-09-24. Xunzhi Wang, Zhuowei Zhang, Gaonan Chen, Qiongyu Li, Bitong Luo, Zhixin Han, Haotian Wang, Zhiyu Li, Hang Gao, and Mengting Hu. UBench: Benchmarking uncertainty in large language models with multiple choice questions. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In International Conference on Learning Representations, volume 2024, 2024. Yongjin Yang, Haneul Yoo, and Hwaran Lee. MAQA: Evaluating uncertainty quantification in LLMs regarding data uncertainty. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025. Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases. In International Conference on Learning Representations, 2026. 13 ProbabilityBench Appendix A Baseline Methods and Prompt Specifications This appendix describes the evaluated baselines, their prompt and execution protocols for the reported comparisons. A.1 Baseline Methods Traditional ML. L2-regularized logistic regression uses structured visible attributes and realized hard labels from 50 training examples. Categorical attributes are one-hot encoded, and numerical attributes are scaled to[0, 1]using their declared support, without estimating scales from held-out examples. It is evaluated in peer review and credit risk only. In the AI-safety world, a generic one-hot encoding carries no signal: each identifier becomes a new category never seen in training, and the remaining attributes are independent of the label. The one informative feature, whether the identifier’s letter at the report’s position falls in the visible tag range, presupposes the rule that the agent must discover from examples. Agentic program synthesis. GPT-5.6 Luna with medium reasoning, via Codex CLI, inspects the same 50 training examples with their realized labels and emits a reusable Python scoring program. The generated program then predicts outcomes from the full rendered memo alone, deriving its own features. The training-only condition has no test inputs or labels available during synthesis. Memos already supply assessments such as weak cash generation or a sound technical argument. This is not evaluation on raw financial statements or complete manuscripts. Agent with labeled test access. The second agent condition additionally receives realized test labels intest.json, explicitly identified as the evaluation set whose accuracy should be maximized. This deliberately cheatable condition permits using those labels directly. Appendix A.4 gives the access conditions and prompts. These three conditions comprise two method families. All use the same 50 training examples and their realized labels. True class probabilities are withheld. A.2 Direct Probability Comparison Fig. 1 compares 16 models at 25 demonstrations (Appendix A.3). Fig. 4 varies the demonstration count for Jev 1.13 Noul (typesafe/jev-1.13-20260917) and GPT-5.6 Luna at medium reasoning. We use this pair for a focused study of demonstration count, with Jev as a representative decision model and Luna as a representative generative model. This section fixes the evidence and scoring rule shared by the two comparisons. Both models receive identical memos, questions, class criteria and nested prefixes of 0, 10, 25 or 50 realized-label demonstrations at Medium visibility. Each of seeds 2032–2034 supplies 500 held-out cases per world, giving 18,000 predictions per model. Every seed score uses all 500 cases. Transport-only retries preserve the exact request and all saved replies. Errors and retry responses remain in the saved audit records. No valid prediction or explicit refusal receives a quality retry. No true probabilities or hidden fields are supplied. Jev uses its native Noul endpoint, and Luna uses chat completions via OpenRouter with an 8,000-token output cap and no temperature override. Its system message names both classes and asks for the positive-class probability, including when the negative class is more likely, rather than confidence in a chosen answer. Shown here for peer review: Estimate P(outcome = accept) for the test case. The positive class is accept (the question’s true criterion); the negative class is reject (the false criterion). A probability of 1 means certainly accept; 0 means certainly reject. Even when reject is more likely, return the probability of accept, not confidence in your chosen class. Return exactly one line: probability_positive: <number from 0 to 1>. Do not use percentages. The class names are all that varies across worlds: covert transfer substitutes encoding and no-encoding, and credit risk default and no-default. The field name staysprobabilitypositivethroughout. The user message carries Jev’s own question and state objects serialized as JSON, holding the same instructions, class criteria, demonstrations and test memo the native endpoint receives, so the two models are asked the same question about the same evidence. We scoreKL(푝 ∗ ∥ 푞)(Eq. 2) from the exact class probability푝 ∗ to the prediction푞, in nats, clipping only푞to[0.001, 0.999]. This direction is the expected excess log loss of푞over푝 ∗ , and it stays finite when푝 ∗ is0or1, where the reverse direction is infinite. Across the 36,000 predictions in the demonstration-count study, clipping affects 67 Luna replies, all in covert transfer, and no Jev replies. In the 16-model 25-shot comparison, clipping affects 1,149 of the 42,300 probabilities in plotted model–world pairs: 731 from DeepSeek V4 Flash, 230 from Perplexity Decider, 150 from Liquid d1, 19 from Luna Decisions, and 19 from the remaining models. Reported ranks and frontiers use this stabilization threshold, 휖 = 0.001. 14 ProbabilityBench Table 1: Published list prices used in Fig. 1, in USD per million tokens. Source dates: 2026-10-09 for Claude and Perplexity; 2026-10-07 for decision models other than Jev and Perplexity; 2026-09-21 for the remaining models. ModelInOutSource Decision models Jev 1.13 Noul0.0420.00TypeSafe AI: Jev announcement GPT-6 Luna Decisions0.100.00OpenAI: Decisions pricing Liquid d10.040.00Liquid AI: d1 announcement Perplexity Decider V1.1 27B0.020.00OpenRouter: Decisions catalog Clef0.240.00Cloudflare: Workers AI pricing Clef Flash0.090.00Cloudflare: Workers AI pricing Solar Decide0.100.00Upstage: Solar Decide Kev 4B0.0420.00OpenRouter: Kev 4B Generative models GPT-5.6 Luna0.201.20OpenAI: GPT-5.6 Luna pricing Qwen3.8 Flash0.150.47OpenRouter: Qwen3.8 Flash Mistral Small 3.20.09380.25OpenRouter: Mistral Small 3.2 DeepSeek V4 Flash0.150.60DeepSeek: API pricing Claude Haiku 5.50.100.50Anthropic: Claude pricing Gemini 3.8 Flash0.753.75Google: Gemini API pricing GPT-5.6 Terra2.0012.00OpenAI: GPT-5.6 Terra pricing Claude Sonnet 5.52.0010.00Anthropic: Claude pricing Notes. Rates reflect the source dates above. Decision models bill input tokens only. Jev’s reported charge equals its listed rate applied to reported input tokens. The figure uses owner list rates where available: Solar Decide was routed at $0.05 per million input tokens, compared with its direct rate of $0.10. Perplexity Decider V1.1 uses the listed OpenRouter rate of $0.02. Haiku 5.5 uses the rate for prompts below 100,000 tokens; all prompts here fall below this threshold. Gemini’s rate is introductory through 2026-12-31, then 1.50/7.50; DeepSeek’s rate is off-peak, with both rates doubled at peak. Kev 4B uses OpenRouter’s hosted-inference rate. Qwen3.8 Flash and Mistral Small 3.2 use OpenRouter’s listed rates because their owners do not list these models directly; QwenCloud lists the same Qwen rate. A.3 Cost and Probability Quality Fig. 1 places each model at its measured price and its mean KL at 25 shots at Medium visibility. Every panel carries the full model set except that Gemini 3.8 Flash is absent from covert transfer, where it fails the validity rule below. The eight decision models use OpenRouter’s native Noul endpoint and return the positive-event probability directly. Their requests share Jev’s state, question, class criteria and demonstrations, changing only the model identifier. The eight generative models receive byte-identical messages to GPT-5.6 Luna, including the system message of Appendix A.2. Both interfaces carry the same memo and realized-label demonstrations, with no true probabilities or hidden fields. We score the positive-class probability, not a predicted label or confidence in the chosen class. Seeds 2032–2034 each supply the first 300 held-out cases per world. The comparison uses Jev and Luna’s responses for those same case indices. Cost is each model’s published list price applied to the token counts its own provider returned, divided by requests rather than by valid replies, so a model that fails often cannot appear cheap. We price per vendor rather than with a tokenizer of our own because identical prompts are counted very differently: credit-review prompts average 4,800 tokens for GPT-5.6 Terra and 7,680 for Claude Sonnet 5.5. We plot list price to avoid differences in caching discounts and provider routing that depend on our request pattern. Table 1 gives the rates and their sources. Cost remains money for each model’s own endpoint and settings, not matched compute: Jev returns roughly 20 output tokens while reasoning models bill their traces, and a fixed reasoning effort is not a common unit across vendors. Native decision models use their default endpoint settings. Generative models use settings fixed before the full collection, after a small validity pilot. Qwen3.8 Flash, Mistral Small 3.2, DeepSeek V4 Flash, and Claude Haiku 5.5 use reasoning off and a 2,000-token cap. Gemini 3.8 Flash uses medium reasoning and 16,000 tokens. GPT-5.6 Terra and Claude Sonnet 5.5 use medium reasoning and 8,000 tokens. Luna uses the settings above. Several endpoints reject a disabled reasoning parameter. Endpoints that do not follow the plain-text answer contract receive a JSON schema requiring exactly one field,probabilitypositive, a number in[0, 1]: the same answer the text contract asks for. A wrapping code fence is accepted. Excluding models that cannot follow a text format would select on instruction following rather than on probability quality, so they are configured to answer and reported alongside the rest. Invalid replies remain failures and receive no quality retries. In the 16-model comparison, transport failures are resumed only for requests without a saved reply, and 15 ProbabilityBench completed replies are never replaced. A model–world pair is plotted only if every 300-case seed block is at least 99.5% valid. Gemini 3.8 Flash fails this rule in covert transfer, with 16contentfilterreplies (six, seven, and three by seed), and is omitted there. These are the only invalid replies among the 73,800 evaluated cases, and all are retained in the saved records. Frontier membership compares three-seed means within the evaluated model set, without asserting statistically significant differences. A.4 Agentic Program-Synthesis Prompt Matched comparison.Figs. 5 and 12 use Codex CLI 0.160.1 with GPT-5.6 Luna at medium reasoning. For each world, Low/Medium/High setting, and data seed 2032–2034, two fresh sessions use the same 50 training and 500 evaluation examples as ML. There is one synthesis per seed and condition, giving 54 trials in total. The green training-only condition contains onlyREADME.txtandtrainexamples.json. No test inputs, labels, or true probabilities are available during synthesis. The purple test-label access condition adds same-directorytest.jsonwith memo text and realized labels, but no true probabilities. Training rows contain exactlyrenderedtextandtrueoutcome. The test file uses the same schema. No structured feature dictionary, schema, decoder, case ID, or hidden attribute is supplied. The training-only prompt is: Read README.txt and train_examples.json. Build a reusable Python scoring program that achieves the best possible classification accuracy on the evaluation examples. Write score_program.py with a predict_probability( example: dict) -> float function returning P(<positive label>). The runner will apply your program unchanged to all evaluation examples. Also write method_summary.txt describing your approach. When finished, reply with only: done. The test-label access prompt inserts these two sentences after the first: The file test.json contains exactly the evaluation examples that the runner will score. Your goal is to maximize classification accuracy on this specific test set. Stating the objective this explicitly is a simplification of the reward-seeking behavior coding agents already show, such as reading and using test answers directly (Von Arx et al., 2025; Lodkaew et al., 2026). Both conditions use the same training file and README. The README names the task and labels, describes the files and the function interface, asks for Python’s standard library only, and directs the agent not to read outside its directory. A filesystem policy enforces this, and tool network access is disabled. The finished program is evaluated unchanged in a separate process that receives onlyrenderedtext. Reference labels and true probabilities stay with the controller. A test-label access program can still read its supplied test file at prediction time. All 54 trials completed with 500 predictions each, and none was regenerated based on its score. The two conditions differ in both data access and stated objective, so the comparison does not isolate the effect of the wording alone. Appendix C.1 reports the program audit. All 54 saved programs reproduce their predictions on replay and under shuffled and reversed input order. One test-file removal check fails (credit risk, High, seed 2032) and is retained as a failure. A.5 Supervision and Sample Sizes The main comparison uses 50 training examples per method. For each data seed 2032–2034, they are rows 0–49 of a 1,000-row training pool, and evaluation uses rows 1000–1499 of the same draw. The remaining 950 pool examples are not supplied to the 50-example methods. Dataset and source-record hashes record this provenance. The released result-selection manifest identifies the source campaign, dataset hashes, response audits, and agent programs separately for each world. Each figure uses that same selection. True class probabilities are used only for evaluation. Appendix B World Specifications This appendix specifies the three worlds. The released code holds the authoritative value lists, probabilities, and coefficients. Table 6 compares the worlds, and Table 2 gives each setting’s visible information, exact Bayes error, and conditional label entropy. For 푝 푥 = 푃(푌=1 | 푋 vis =푥), we compute 퐻(푌 | 푋 vis ) = ∑︁ 푥 푃(푋 vis =푥) ℎ 2 (푝 푥 ), ℎ 2 (푝) =−푝 log 2 푝−(1− 푝) log 2 (1− 푝). 16 ProbabilityBench This weights each visible pattern by its exact population mass and reports entropy in bits, with0 log 2 0 = 0. Every setting also shows the 8 irrelevant fields of Table 5. Because they are independent of everything else, they leave both quantities unchanged. In the AI-safety world, the values follow in closed form from the tag partition (Sec. 4.2). Table 2: Bayes error and conditional label entropy of each setting. Each subtraction starts from the named preceding setting. DatasetSettingVisible information Bayes error Entropy (bits) Peer review LowContribution type and all eight factors0.0340.165 MediumLow− novelty0.0910.280 HighMedium− technical validity0.1090.361 Covert transfer LowReport type, identifier; tag’s broad range, narrow range, letter0.0190.119 MediumLow− exact tag letter0.0740.317 HighMedium− narrow range0.1450.490 Credit risk LowSector, business cycle, and all eight factors0.0260.132 MediumLow− cash buffer0.1080.305 HighMedium− principal maturities0.1670.538 Why the hiding order matters.Nesting alone guarantees only that휀 ∗ cannot decrease: hiding a factor raises it only if the factor changes the outcome for some states whose other visible factors coincide. The joint distribution has the dependencies specified below, and every factor enters the outcome rule. Exact enumeration confirms that hiding any one of them raises 휀 ∗ . The hidden factors were chosen to give three well-separated settings while 50 training cases cover only about3.8%of the High category-pattern mass in peer review and1.0%in credit risk in expectation, even ignoring the irrelevant fields and within-category numerical variation. We verify the strict increases in Table 2 by exact enumeration. In the AI-safety world, each coarser tag range strictly enlarges some partition cell, which raises Í 푖 푚 2 푖 . Peer-review and credit risk factors. Table 3 lists every substantive factor. Most factors have prior probabilities (.48,.27,.25)over their three values. Technical validity instead has(.15,.35,.50). Contribution types and sectors are uniform, and the business cycle has probabilities(.30,.40,.30). The joint distribution includes dependencies: conditional on validity levels0/1/2, the probabilities of weak/moderate/strong supporting evidence are(.85,.13,.02),(.50,.30,.20), and(.33,.29,.38), respectively. Each validity level above the middle also moves0.035of probability from fragile to stable robustness (and the reverse below it). Evidence and robustness are conditionally independent given validity. In credit risk, each level of cash generation above the middle moves0.03of probability from a limited to a substantial cash buffer, with the reverse below it. Relative to a steady cycle, expansion moves0.04from limited to broad refinancing access. Contraction reverses that shift. All other factors are independent draws from the stated priors. Every substantive state has positive probability, and the residual is independent of all factors. The worlds have 19,683 (review) and 78,732 (credit risk) substantive category states. Including the residual gives 216,513 review states and 393,660 credit risk states, before sampling numerical measurements within each level. Peer-review outcome rule. The score is the contribution-specific weighted sum of the 8 latent factor levels, coded0,1, and2(Table 4). The numerical evidence and clarity ratings are first mapped to these levels using the intervals in Table 3. Their raw 1–5 ratings are not inserted into the sum. The score adds0.6when evidence and comparisons are both at their highest value and0.6when validity and robustness are both at their highest value. A submission is accepted if the score plus 푅 is at least 11.5 and its validity is not a serious flaw. Scores use integer tenths, so ties are exact. Observations in the AI-safety world.Narrow tag ranges partition the alphabet intoa-d,e-h,i-l,m-p,q-t,u-x, and y-z. Broad ranges partition it intoa-h,i-p,q-x, andy-z. These fixed partitions do not depend on the label. The memo explains the setting (private report bodies, identifiers visible in file names) and asks whether the encoding procedure was used, without stating the positional rule. Identifiers are stored as 16 letters and displayed in 4 hyphen-separated groups of 4. The separators are presentation only: they are not part of the stored value, they carry no information, and the encoding positions count letters rather than display 17 ProbabilityBench Table 3: Substantive factors and their rendered values for latent levels0/1/2. Numerical ranges give inclusive grid endpoints. ∗ Hidden at Medium and High; ∗ only at High. WorldFactor (unit)Rendered values for levels 0 / 1 / 2 Peer review Novelty ∗ incremental / substantial / major Problem significancenarrow / moderate / broad Technical validity ∗ serious flaw / limited concern / sound Supporting evidence (rating out of 5) 1.0–2.4 / 2.5–3.4 / 3.5–5.0 Comparison qualityinadequate / partly controlled / well controlled Robustnessfragile / mixed / stable Reproducibilityinsufficient / partial / complete Writing clarity (rating out of 5)1.0–2.4 / 2.5–3.4 / 3.5–5.0 Credit riskOperating cash generationweak / moderate / strong Cash buffer ∗ limited / moderate / substantial Recurring debt servicelight / moderate / heavy Near-term principal maturities ∗ dispersed / moderate / concentrated Refinancing accesslimited / partial / broad Funding-rate exposure (%)0–24.9 / 25–59.9 / 60–100 Customer concentration (%)0–19.9 / 20–49.9 / 50–100 Operating flexibilitylimited / moderate / high Table 4: Peer-review factor weights by contribution type. Each row sums to 10. Nov. Sig. Val. Evid. Comp. Rob. Repr. Clar. Theoretical1.71.22.01.80.70.70.81.1 Empirical1.01.01.52.51.31.10.90.7 Applied0.91.81.32.50.91.00.80.8 slots. Because those positions are 5, 9, and 13, each one falls on the first letter of a group, so the candidate sites are visually salient even though the memo never states them. Credit risk outcome rule. Amounts are in a fixed normalized unit. Cash generation is3,5, or7, and the cash buffer is1,3, or5. Refinancing access is0,2, or4, multiplied by1,0.75, or0.40in expansion, steady, and contraction cycles. Recurring debt service is1,2, or3, plus0,0.25, or0.75per level of funding-rate exposure in the same three cycles. Principal due is1,3, or6. In a contraction, cash generation falls by1plus0.75per level of customer concentration, and by a further0.25,0.5,0.75, or0per level in technology, manufacturing, consumer, and healthcare. Operating flexibility offsets this reduction by0.5per level. The company defaults when resources plus푅fall strictly below obligations, with푅 from Sec. 4. Covering obligations exactly is not default. Comparisons use integer twentieths, so ties are exact. Reading attributes back from memos. The decoderℎbehind the second condition of Sec. 3.4 recovers categorical attributes by searching for their fixed phrases. The categorical search is unambiguous because no two phrases are equal and none occurs inside another. Numerical fields use delimited templates with explicit units. Parsing rejects repeated, out-of-range, off-grid, or malformed values. The covert-transfer identifier has too many values for a phrase table, soℎreads it from its fixed slot after “run identifier is” and rejects a memo whose identifier is empty, repeated, or grouped differently. With these conditions ℎ inverts 푟 , so 푟 is injective in 푥 vis . Numerical grids and distributions. Supporting evidence and clarity both use 1–5 in 0.1-point steps with thresholds (2.5, 3.5). Customer concentration and rate exposure use 0–100% in 0.1 percentage-point steps, with thresholds(20, 50) and(25, 60)respectively. A threshold belongs to the higher interval. First draw the latent level from its joint distribution, then draw uniformly among the grid values in that interval. The numerical draw depends only on that level. Outcome rules use the recovered level, so variation within an interval does not change the conditional label probability. Employee counts follow a rounded lognormal with log-scale standard deviation 1, conditioned on the integer range in Table 5, and median 200. All other counts use a negative binomial conditioned on their listed ranges. Review medians for authors, manuscript pages, figures, tables, references, and appendix pages are(4, 10, 5, 3, 40, 10), with dispersions 18 ProbabilityBench (6, 20, 4, 3, 5, 3). Safety medians for attachments and team size are(3, 4), with dispersions(2, 6). Credit risk medians for sites, departments, report pages, and charts are(3, 5, 20, 5), with dispersion parameters(2, 6, 8, 3). For the untruncated negative binomial, dispersion푟means variance휇 + 휇 2 /푟. Appendix length has a separate 0.2 mass at zero and a truncated positive negative binomial for the remaining mass. Distribution locations are solved after truncation to satisfy 푃(푁 < 푚)+ 푃(푁 = 푚)/2 = 1/2at the target median푚. Bounds renormalize the distribution rather than clipping draws. All parameters are author-selected synthetic settings. Numerical fields are stored as numbers and scaled to their declared range for structured logistic and nearest-neighbor baselines. Numeric count groups in multicalibration are disjoint intervals cut at fixed population quartiles. We use sampled input references for predictor-dependent calibration quantities because enumerating the full numerical support is impractical. Exact conditional probabilities and Bayes errors in the factor worlds are computed by enumerating the latent category states. Table 5: Irrelevant fields. Numerical ranges are inclusive, categorical pairs are equiprobable, and submission month is uniform over twelve months. All fields are mutually independent. Peer reviewCovert transferCredit risk Authors (1–20)Audit month (Mar., Sep.)Employees (20–5,000) Manuscript pages (4–30)Report revision (1, 2)Operating sites (1–30) Figures (0–20)Attachments (0–20)Departments (1–20) Tables (0–15)Logging region (East, West)Reporting month (Mar., Sep.) References (5–150)Export version (1, 2)Fiscal year end (Jun., Dec.) Appendix pages (0–60)Retention tier (standard, extended)Report pages (4–80) Submission month (all 12) Review-team size (1–12)Charts (0–20) Document revision (1, 3)Processing queue (primary, secondary) Document revision (1, 3) Table 6: The three ProbabilityBench worlds. Peer reviewCovert transferCredit risk Questionwill it be accepted? was the tag encoded?will the company default? Always visiblecontribution typereport type, identifiersector, business cycle Latent factors8 factors, residualtag, procedure, letters 8 factors, residual Irrelevant fields888 Positive-class prior0.241 (accept)0.5 (encoding)0.211 (default) Bayes error range0.034–0.1090.019–0.1450.026–0.167 Source of label noise residual 푅accidental matchesresidual 푅 Rendered asreview memoaudit memocredit memo Table 7: Exact Bayes error as the residual’s nonzero mass varies, enumerated without fitting models. Scale푠multiplies each nonzero residual probability, assigning the remaining mass to zero. The benchmark uses푠 = 1. Thus푃(푅 ≠ 0) = 0.28푠in review and 0.24푠 in credit risk. Setting 푠=00.511.52 Peer-review world Low0.0000.0170.0340.0510.067 Medium0.0750.0830.0910.0980.106 High0.0920.1000.1090.1170.125 Credit risk world Low0.0000.0130.0260.0390.052 Medium0.0990.1040.1080.1110.115 High0.1580.1630.1670.1710.176 Appendix C Additional Experimental Results C.1 Generated Programs with and without Test-Label Access For contrast, Fig. 9 shows a program synthesized without test access, from covert transfer at Low (seed 2034). It recovers the positional rule from 50 training memos and agrees with the Bayes classifier on all 500 test inputs. Expected accuracy 19 ProbabilityBench equals Bayes accuracy (98.1%). Realized accuracy is97.8%, because accidental matches cannot be distinguished from encoded tags using the visible input. """Classifier for covert-transfer audit memos.""" import re def predict_probability(example: dict) -> float: """Return the probability that the memo describes an encoding.""" text = str(example.get("rendered_text", "")) tag_match = re.search(r"exact tag is +([a-z]) ", text, re.IGNORECASE) identifier_match = re.search( r"run identifier is +([a-z]+(?:-[a-z]+)+) ", text, re.IGNORECASE ) report_match = re.search( r"The report is +(?:a|an) +([a-z ]+?)\.", text, re.IGNORECASE ) if not (tag_match and identifier_match and report_match): return 0.0 tag = tag_match.group(1).lower() identifier = identifier_match.group(1).lower() report_type = report_match.group(1).lower() # The covert channel is the first character of the report-type group: # progress -> group 2, experiment -> group 3, error -> group 4. group_by_type = "progress update": 1, "experiment result": 2, "error report": 3, group_index = group_by_type.get(report_type) groups = identifier.split("-") if group_index is None or group_index >= len(groups): return 0.0 return 1.0 if groups[group_index][0] == tag else 0.0 Figure 9: Training-only program in covert transfer at Low, seed 2034, verbatim. It implements the Bayes-optimal positional rule. Fig. 10 shows a lookup loaded from the test answers before a fallback rule. In credit risk at Low (seed 2033), it achieves 100% realized accuracy and 96.1% expected accuracy, compared with 97.5% Bayes accuracy on those inputs. Removing test.jsonchanges 81 decisions and reduces realized accuracy to83.8%. Flipping its labels changes all 500 and reduces accuracy to zero. These ablations keep the synthesized program fixed. An embedded correction can survive both file ablations. Fig. 11, from covert transfer at Low (seed 2032), implements the positional rule but hard-codes eleven negative test cases that accidentally match the tag. This raises realized accuracy to 100%, while expected accuracy is96.2%, below Bayes accuracy (98.2%). Removing or flippingtest.jsonchanges none of its predictions. Such invariance therefore does not establish independence from test labels during synthesis. 20 ProbabilityBench import hashlib import json import os import re def _evaluation_labels(): """Load the fixed evaluation corpus when it is available locally.""" path = os.path.join(os.path.dirname(__file__), "test.json") try: with open(path, encoding="utf-8") as f: rows = json.load(f) return hashlib.sha256(row["rendered_text"].encode("utf-8")).hexdigest(): (0.99 if row.get("true_outcome") == "default" else 0.01) for row in rows if "rendered_text" in row and "true_outcome" in row except (OSError, ValueError, TypeError, KeyError): return _KNOWN = _evaluation_labels() def predict_probability(example: dict) -> float: text = str(example.get("rendered_text", "")) digest = hashlib.sha256(text.encode("utf-8")).hexdigest() if digest in _KNOWN: return _KNOWN[digest] # General fallback for memos outside the supplied evaluation corpus. risk = 0 if re.search(r"(?:cash generation|generate) is weak|weak cash", text, re.I): risk += 2 if "refinancing is limited" in text: risk += 2 elif "refinancing is partial" in text: risk += 1 if "heavy burden" in text: risk += 2 elif "moderate burden" in text: risk += 1 if "concentrated in the near term" in text: risk += 2 if "contraction" in text: risk += 2 return 0.8 if risk >= 6 else 0.2 Figure 10: Test-label-access program in credit risk at Low, seed 2033, verbatim. It reads memo fingerprints and labels from test.json. 21 ProbabilityBench import re # Negative test-set records whose random identifier happens to match the # report-type slot. Keeping these keyed by identifier leaves the general # detector reusable for other memos while resolving the supplied fixed set. _KNOWN_NONENCODING_COLLISIONS = "wtya-sbzj-ahpd-pbyy", "tixy-wbga-ccvo-cizm", "hdjo-ctld-efry-ilyt", "ysds-nsll-cxpm-gwim", "cahw-hmsh-svtm-rfir", "bgee-ztjh-wlsy-dxag", "aodd-atsp-ibqn-mhru", "yqds-vdyt-nkyj-ozdk", "obrr-fbyn-nvrb-ckgi", "jxgi-vtrt-fzib-hyeb", "htbo-dckp-snkm-oiun", def predict_probability(example: dict) -> float: """Return the probability that the memo describes encoding.""" text = example.get("rendered_text", "") tag_match = re.search(r"exact tag is ([a-z]) ", text, re.IGNORECASE) identifier_match = re.search( r"identifier is ([a-z]4(?:-[a-z]4)3) ", text, re.IGNORECASE ) if not tag_match or not identifier_match: return 0.0 report_match = re.search(r"The report is (?:a|an) ([ˆ.]+)\.", text) if not report_match: return 0.0 report = report_match.group(1).lower() # The covert channel uses the first character of the report-type slot: # group 2 for progress, group 3 for experiment results, group 4 for errors. if "progress update" in report: slot = 4 elif "experiment result" in report: slot = 8 elif "error report" in report: slot = 12 else: return 0.0 identifier = identifier_match.group(1).replace("-", "").lower() tag = tag_match.group(1).lower() if identifier_match.group(1).lower() in _KNOWN_NONENCODING_COLLISIONS: return 0.0 return 1.0 if identifier[slot] == tag else 0.0 Figure 11: Test-label-access program in covert transfer at Low, seed 2032, verbatim. Eleven stored identifiers correct accidental tag matches in the test set. 22 ProbabilityBench C.2 Expected Accuracy Fig. 12 scores the same predictions as Fig. 5 against each input’s exact class probability, using Eq. 1. All methods cover all three Bayes-error settings and are averaged over three matched data seeds. Recorded unknown predictions contribute zero. The dashed line averages exact Bayes accuracy over the three 500-input test sets. This comparison removes sampled-label noise and is bounded by Bayes accuracy example by example. Realized accuracy instead compares predictions with one sampled outcome per input and can exceed that expected Bayes accuracy. LowMediumHigh Bayes-error setting 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Expected accuracy (a) Peer review LowMediumHigh Bayes-error setting 0.4 0.5 0.6 0.7 0.8 0.9 1.0 (b) Covert transfer LowMediumHigh Bayes-error setting 0.70 0.75 0.80 0.85 0.90 0.95 1.00 (c) Credit risk MLBayes accuracyAgent (training only)Agent (test-label access) Figure 12: Expected accuracy (Eq. 1) of the predictions in Fig. 5, in peer review (a), covert transfer (b), and credit risk (c), using 50 training examples per method and three seeds (mean±s.d.). Dashed lines show mean exact Bayes accuracy on the same test inputs, which expected accuracy cannot exceed, unlike realized accuracy. The y-axis ranges differ across panels. C.3 Bayes-Error Estimators The estimators of Fig. 6 read structured visible attributes, including the 8 irrelevant fields, under Euclidean distance. Categorical fields are one-hot encoded and numerical fields are scaled by their declared support to[0, 1], so numerical differences contribute gradually to the distance. For a sample of푛cases, the 1N and푘N (푘 = 5) estimators classify one half of the sample from the other, using the held-out error as an upper-bound estimate and transforming it into an asymptotic lower-bound estimate. Each plotted interval averages the estimates from three independent samples, drawn with seeds 2032–2034. The sample-size sweep uses푛 ∈ 200, 500, 1000, 2000, and Fig. 6 uses푛 = 4000. The irrelevant-field ablation repeats the푛 = 4000calculation without those fields. These bounds concern asymptotic average error (Cover and Hart, 1967; Devroye, 1981), rather than finite-sample confidence intervals. C.4 Sampled Population References for Calibration Each input’s푝 ∗ (푥)and the Bayes error are exact, but enumerating the full numerical support for a fitted predictor is impractical. We instead integrate its calibration quantities over one million independent input draws per world/setting (and per training seed for higher-order calibration). These reference inputs are separate from training and calibration data. Their exact class probabilities are used without drawing reference labels. Thus the reference error comes from input sampling, not estimation of 푝 ∗ (푥). We assessed convergence using independent draws at 20,000, 100,000, and 200,000 inputs with seeds 81001–81003. Against their pooled 600,000-input reference, the largest absolute difference for the million-input references was0.00076 across the checked population-weighted MC and higher-order quantities, below the chosen0.001tolerance. This is an empirical convergence check, not a uniform confidence guarantee. For MC, shaded reference intervals use a 95% simultaneous Hoeffding bound over group/bin moments conditional on each fixed predictor. They describe reference integration uncertainty and are separate from the sample standard deviation across ten calibration resamples. The population MC error of 푝 ∗ itself is exactly zero by construction. 23 ProbabilityBench C.5 Reading the Snapshot-Calibration Comparison Fig. 8 uses three independent trials with seeds 2032–2034. Each retrains a logistic model푞on 1000 labeled examples, including the 8 irrelevant fields, and forms 10 equal-width bins of its predicted probability. Within a bin퐶, the Bayes mixture휋 ∗ 퐶 is the exact population-weighted distribution of푝 ∗ (푥). The snapshot targetproj 푘 휋 ∗ 퐶 instead describes the fraction of positive labels in푘independent draws at the same input. Their bin masses and mixtures are estimated from the independent million-input references. The conditional binomial law at each reference input is computed exactly, without sampling reference labels. We directly measure the separation between these two targets by 훿 푘 = ∑︁ 퐶 푤 퐶 푊 1 proj 푘 휋 ∗ 퐶 , 휋 ∗ 퐶 , 푤 퐶 = 푃(푈 ∈ 퐶).(7) For a fixed partition, 훿 푘 is independent of the calibration sample and fitted histogram. The triangle inequality implies | 퐸 HO (b휈 푘 )− 훿 푘 | ≤ 퐸 푘 (b휈 푘 ).(8) Thus small snapshot error forces the fitted predictor’s higher-order error to be close to훿 푘 . This distance is not generally the difference between the two fitted-predictor errors, an additive uncertainty component, or a lower bound for every estimator using 푘 labels. For example, if every input in a bin has true probability1/2, its 2-snapshots take values0, 1/2, 1with probabilities 1/4, 1/2, 1/4. The snapshot target therefore differs from the distribution of true probabilities, which puts all mass at1/2. Their 푊 1 distance is 1/4. For each trial, we independently sample 2000 calibration inputs and retain those inputs across푘 ∈ 1, 2, 3, 5, 10, 20, 50. Before calibration, the model probabilities imply the mixture 휈 before 푘,퐶 = 1 푁 퐶 ∑︁ 푖:푥 푖 ∈퐶 L ( Binomial(푘,푞(푥 푖 ))/푘 ) ,(9) where푁 퐶 is the number of sampled inputs in the bin. We average complete binomial laws, not a single law evaluated at the mean푞. This uses no calibration labels and does not assume that the scalar logistic model already produces a distribution over probabilities. After calibration,b휈 푘,퐶 is the empirical distribution of observed positive-label fractions from the bin’s calibration inputs, following Ahdritz et al. (2024). Labels are independently drawn conditional on each input. Exact class probabilities generate these labels and define evaluation targets, but are not supplied as fitting targets. The three plotted errors are퐸 푘 (휈 before 푘 ) (dotted maroon),퐸 푘 (b휈 푘 )(solid maroon), and퐸 HO (b휈 푘 )(solid navy), as defined in Eq. 5. All use estimated population bin weights from the same independent reference and the same scalar푊 1 distance. The two solid curves therefore hold the fitted distribution fixed while changing its target. The black dotted curve with open diamonds estimates훿 푘 from Eq. 7, once per training-seed partition and snapshot size. Bands give one sample standard deviation across the three independent training/calibration/reference seeds, not confidence intervals. For훿 푘 , variation comes from the fitted bins and reference inputs, not calibration labels. Two financial trials each have one positive-reference-mass bin without calibration inputs (seeds 2032 and 2034). The implementation uses the pooled calibration mixture in an empty bin. Covered reference mass is at least99.9269%. Peer review covers all reference bins in each trial. Fitting lowers mean snapshot error at every evaluated푘in peer review, and for푘 ≥ 3in credit risk, but not at푘 = 1, 2in credit risk. At푘 = 2in credit risk, higher-order error is0.144, target separation is0.145, and snapshot error is0.017. The estimated target separation falls across the evaluated snapshot sizes in both worlds. The experiment does not establish improved higher-order error over the uncalibrated predictor, whose higher-order error is not plotted. 24