Paper deep dive

CrystalJev: thinking fast and slow with atomistic foundation models for materials discovery

Peng Kang, Zhen Li, Yu Liu, Lei Zheng, Huibin Xu

Year: 2026Venue: arXiv preprintArea: cond-mat.mtrl-sciType: PreprintEmbeddings: 148

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/7/2026, 8:14:27 AM

Summary

The paper introduces CrystalJev, a decision-making layer for atomistic foundation models that treats them as fast decision-makers (System 1) rather than slow simulators. CrystalJev queries a frozen interatomic potential once per unrelaxed structure to provide calibrated probabilities for stability and other properties, using conformal prediction for finite-sample guarantees. It employs a value-of-information theory to determine when to escalate to slower, more expensive computations like relaxation (System 2) or density-functional theory (System 3). The method demonstrates that a single forward pass can rival relaxation in accuracy at a fraction of the cost and effectively triage millions of candidates.

Entities (9)

CrystalJev · method · 99%Atomistic Foundation Models · model-class · 95%Density Functional Theory · method · 95%CHGNet · model · 90%Conformal Prediction · method · 90%Matbench-Discovery · benchmark · 90%Value of Information · concept · 90%Materials Project · database · 85%WBM · dataset · 85%

Relation Signals (8)

CrystalJev → provides → Calibrated Probabilities

confidence 95% · answers typed questions with calibrated probabilities

CrystalJev → uses → Atomistic Foundation Models

confidence 95% · CrystalJev queries a frozen interatomic potential once per unrelaxed structure

CrystalJev → employs → Conformal Prediction

confidence 90% · conformal prediction turns these into finite-sample guarantees

CrystalJev → evaluatedon → Matbench-Discovery

confidence 90% · Across 65 Matbench Discovery models

CHGNet → isa → Atomistic Foundation Model

confidence 90% · CHGNet ... interatomic potentials

MACE-MP-0 → isa → Atomistic Foundation Model

confidence 90% · MACE-MP-0 ... interatomic potentials

CrystalJev → utilizes → Value of Information

confidence 90% · a value-of-information theory sends slower computation only where decisions can change

CrystalJev → outperformsormatches → Relaxation

confidence 85% · one forward pass decides nearly as well as a relaxation at a thirtieth of its cost

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Atomistic foundation models triage millions of hypothetical materials but are used as slow simulators, their thresholded energies taken at face value. They are better read as fast decision-makers. CrystalJev queries a frozen interatomic potential once per unrelaxed structure and answers typed questions with calibrated probabilities, finite-sample guarantees and a rule for when to think slowly. Across 65 Matbench Discovery models, a 'stable' call is a probability in disguise, explained by a model's errors and the candidate population. Once trained, one forward pass decides nearly as well as a relaxation at a thirtieth of its cost, and a value-of-information theory sends slower computation only where decisions can change. The same layer answers electronic, mechanical and molecular questions. In a registered prospective test with 700 new density-functional calculations, single-pass forecasts calibrated only on existing data over-stated the stable fraction of unseen candidates (5.8%) by at most 2.1 percentage points.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

145,691 characters extracted from source content.

Expand or collapse full text

CrystalJev: thinking fast and slow with atomistic foundation models for materials discovery Peng Kang 1,2,* , Zhen Li 1,2 , Yu Liu 1,2 , Lei Zheng 1,2,* , and Huibin Xu 1,2 1 School of Materials Science and Engineering, Beihang University, Beijing 100191, China 2 National Key Laboratory of Artificial Intelligence for Material Science, Beihang University, Beijing 100191, China * Corresponding authors: Peng Kang (pengkang@buaa.edu.cn) and Lei Zheng (zhenglei@buaa.edu.cn) Abstract Atomistic foundation models triage millions of hypothetical materials but are used as slow simulators, their thresholded energies taken at face value. They are better read as fast decision-makers. CrystalJev queries a frozen interatomic potential once per unrelaxed structure and answers typed questions with calibrated probabilities, finite-sample guarantees and a rule for when to think slowly. Across 65 Mat- bench Discovery models, a ‘stable’ call is a probability in disguise, explained by a model’s errors and the candidate population. Once trained, one forward pass decides nearly as well as a relaxation at a thirtieth of its cost, and a value-of-information theory sends slower computation only where decisions can change. The same layer answers electronic, mechanical and molecular questions. In a registered prospective test with 700 new density-functional calculations, single-pass forecasts calibrated only on existing data over-stated the stable fraction of unseen candidates (5.8%) by at most 2.1 percentage points. Computational materials discovery has become a funnel of decisions. Generative models and substi- tution rules propose candidate crystals by the million 1–3 ; universal interatomic potentials — foundation models trained on millions of first-principles calculations — relax each candidate and place it on the con- vex hull of known phases 4–7 ; and the candidates predicted to be stable go to density-functional theory (DFT), a handful of them on to the laboratory 8 . Matbench Discovery 9 has made the middle of this funnel measurable, and the field now ranks its foundation models by how often a thresholded energy — a predicted hull distance ˆ E hull ≤0 — recovers DFT stability. This practice treats a foundation model as a slow simulator and leaves two decisions unexamined. The first is how much effort a candidate deserves: a relaxation costs about thirty forward passes of the potential and a DFT calculation orders of magnitude more, yet every candidate is treated alike. The second is what a model’s output means: a team deciding how many calculations to fund, where to set a threshold or how many hits a shortlist will deliver needs a probability of stability that means what it says. Threshold metrics cannot supply it: two models with the same F1 score can make very different promises, and a model with biased energies is penalised by a fixed threshold even when it ranks candidates well 10;11 . Uncertainty quantification has not closed this gap. Ensembles, evidential learning and heterogeneous- ensemble metrics give foundation potentials error bars 12–15 , and probabilistic hulls fold them into stability predictions 16 . But an uncertainty about an energy is not a calibrated answer to a campaign’s question, benchmarks of uncertainty rarely test the decisions it should support 17;18 , and uncalibrated uncertainty can make closed loops fail 19 . Frozen foundation-model representations are increasingly reused 20–24 , but without a contract on what their outputs mean or a rule for when to compute more. Human experts meet the same problem with two modes of thought 25 : a fast System 1 that settles most questions at a glance, and a slow System 2 engaged when the stakes justify the effort. Artificial intelligence has long used this division of labour, from rational metareasoning about when computation is worth its cost 26;27 to dual-process architectures 28 , and ‘System-one models’ such as Jev now return typed decisions with calibrated probabilities in one fast call 29 . Materials discovery is a natural home for the idea: one forward pass of a potential is a System 1, a relaxation a System 2 and a DFT calculation a System 3, and 1 arXiv:2610.06985v1 [cond-mat.mtrl-sci] 4 Oct 2026 the decisions between them are typed, costly and repeated millions of times. Unlike human intuition, a machine System 1 can be calibrated, and can then say for itself when slow thought is worth its cost. Here we introduce CrystalJev, a System-1 decision layer for atomistic foundation models (Fig. 1). A candidate passes once, unrelaxed, through a frozen interatomic potential; small heads on the state of that pass answer typed questions with probabilities calibrated on a small random slice of the candidate pool; conformal prediction turns these into finite-sample guarantees 30–32 ; and a value-of-information rule decides which candidates deserve a slower system. Because the backbone is never fine-tuned, the layer improves as foundation models do. We show that the stability calls of all 65 Matbench Discovery models are probabilities in disguise; that, once trained, one forward pass rivals a relaxation at a thirtieth of its cost; that slower computation pays only inside a cost-determined band of probabilities, so that escalation by value of information captures most of the benefit of relaxation at a tenth of its cost; and that the same layer answers electronic, mechanical and molecular questions. We test its guarantees under distribution shift, in closed loops and in a registered prospective test with 700 new DFT calculations. Results A System-1 layer for atomistic foundation models CrystalJev separates what current practice fuses: the state of a candidate, the question asked about it and the effort spent on it (Fig. 1). The state is read from one forward pass of a frozen foundation potential on the unrelaxed structure: pooled node embeddings plus ten physics scalars such as the single-point energy (Methods). A typed question — yes/no, ordinal, choice or compound — pairs an output type with a label recipe that maps outcomes, here DFT results, to targets. Each question gets a small head that trains in minutes on a CPU and is calibrated on a random slice of labelled candidates exchangeable with those being decided. Calibration turns answers into decisions. If a hit is worth 1 and an attempt costsc, a candidate with calibrated probabilitypis worth attempting exactly whenp≥c, and a shortlist is expected to deliver ∑ i p i hits. Conformal procedures turn the same probabilities into finite-sample guarantees for prediction sets, certified shortlists and screens. And effort becomes a decision: a calibrated System-1 probability states how much a relaxation (System 2) or DFT (System 3) could change the decision, and hence whether consulting it pays (Fig. 1d). Threshold calls are probabilities in disguise Because the benchmark’s Materials Project (MP) hull 33 is fixed and contains no candidate, any Matbench Discovery prediction fixes a predicted hull distance — the energy above the convex hull of known phases — that can be mapped to a calibrated probability (Methods). Maps fitted on a random 5% slice of the WBM candidates 3 for all 65 models (25 interatomic potentials trained on MP-derived data, hereafter MP-trained, 30 trained on larger datasets and 10 other models) reproduce every published F1 score to within 0.0007 (Fig. 2). The leaderboard threshold does not sit where the probability is one half. Among candidates predicted within 5 meV/atom of the hull, only 0.19–0.46 are stable for MP-trained potentials and 0.12–0.30 for models that are not potentials, against 0.37–0.57 for potentials trained on larger datasets (Fig. 2a,b). Much of this is a property of the population: 1.6 times as many candidates lie in the 25 meV/atom above the hull as in the 25 meV/atom below it, so errors carry more unstable structures across the threshold than stable ones. If hull distances have a local density proportional toe κE near the hull and a model’s errors there have biasband spreads, thenP(stable| ˆ E hull =t) =Φ (b−κs 2 −t)/s (Methods). Withκfitted once, it predicts each model’s calibrated probability at the threshold with a Pearson correlation of 0.96 (Extended Data Fig. 1a), and each model’s band of indecision widens with its error (Fig. 2c). Where a threshold should sit depends jointly on a model’s errors and on the candidates it judges. Read as a decision rule, the threshold is a hidden cost assumption: ˆ E hull ≤0 maximises value only if a failed attempt costs 0.18–0.48 of a hit for MP-trained potentials. At other costs the calibrated rule is worth more — for attempts as expensive as a synthesis (c=0.7), a median of 12.8 hits per 1,000 candidates 2 aThinking fast and slow in a discovery funnel Candidate pool 10 6 –10 9 structures System 1 CrystalJev: one pass of a frozen potential cost 1 Every candidate calibrated answerp 1 VOI(p 1 ) >k 2 ? System 2 relaxation with the same potential cost≈30 An uncertain band refined answerp 2 VOI(p 2 ) >k 3 ? System 3 DFT calculation cost≈10 4 –10 5 The few that matter label Attempt or skip yesyes no:decide now onp 1 (most candidates)no:decide onp 2 bThe decision layer: typed, calibrated answers from one pass of a frozen foundation model Candidatex crystal or molecule Frozen backbone one forward pass h s State ofx embeddings, scalars g q (h,s) Typed head log-loss onq raw calibrated Calibrationc q on a random slice yes/no:p= 0.38 ordinal choice Calibrated answer P(y|x,q) shortlist size expected hits cost threshold relax or skip Decision Typed questionq Yes/noisE hull ≤0? Ordinalwhich hull band? Choicewhich polymorph? Compoundstableandgap 1–2 eV? DFT outcomes→targets (label recipe ofq) Guarantees(split-conformal) prediction sets: coverage≥1−α certified shortlist: FDR≤q FDR screen before relaxation: recall≥1−α valid if calibration data are exchangeable with the candidates being decided cOne calibration for every leaderboard model 65 models published ˆ e form ˆ E hull = ˆ e form −e hull form (x) fixed MP hull: no DFT on the candidate needed ˆ E hull = 0 1 2 p ˆ E hull,50 W threshold random slice(a few hundred DFT labels)→1-D map→P(stable) dWhen to think slowly: the value of escalation 01 c System-1 probabilityp 1 value of escalation perfect oracle Gaussian, ratior costk decide: skip decide: attempt escalate only inside the band Fig. 1|CrystalJev: thinking fast and slow with a frozen foundation potential. a, A discovery funnel with deliber- ate escalation. Every candidate receives one forward pass of a frozen interatomic potential (System 1); the calibrated answer decides directly when it is clear, and sends the candidate to a relaxation (System 2) or a DFT calculation (System 3) only when the value of information exceeds the extra cost (costs in forward passes).b, The decision layer. Pooled node embeddingshand physics scalarssof the single pass form the state on which a headg q answers each typed questionq(yes/no, ordinal, choice or compound), trained with log-loss on targets from the question’s label recipe; a mapc q fitted on a slice exchangeable with the candidates calibrates the answer, which then carries split-conformal guarantees.c, The same calibration applied to the published predictions of any Matbench Discovery model gives its probability of stability, its 50% point ˆ E hull,50 and the widthWof its indecision band.d, The value of escalating a candidate is bounded by that of a perfect oracle, a tent peaking at the attempt costc, so escalation at cost kcan pay only inside a band of System-1 probabilities; for a Gaussian refinement it is governed by one information ratior. 3 −200−1000100200 Predicted ̂ E hull (meV/atom) 0.0 0.2 0.4 0.6 0.8 1.0 Calibrated P (stable) eSEN-30M-OAM Nequix-MP-1 Allegro-MP-L MACE-MP-0 CHGNet 0.40.60.8 Leaderboard F1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 DFT-stable fraction at | ̂ E hull | ≤ 5 meV/atom Potentials, MP-derived training Potentials, larger datasets Not potentials 2050100150 MAE of ̂ E hull (meV/atom) 10 30 100 300 Indecision width W (meV/atom) 0.60.70.80.9 Leaderboard F1 1k 3k 10k 30k Certified at FDR ≤ 10% MACE-MP-0 CHGNet Allegro-MP-L Nequix-MP-1 0.050.100.15 Realised FDR (target 0.1) Potentials, sorted by F1 Calibration Random slice Earlier rounds 10 2 10 3 10 4 k most confident 'stable' calls 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 DFT-stable fraction of top k aCalibrated probabilitybStable fraction at thresholdcIndecision band dCertified shortlistseSelection under shiftfMost confident calls Fig. 2|What the stability calls of 65 Matbench Discovery models are worth.WBM unique prototypes; calibration maps fitted on a random 5% slice (five seeds).a, Calibrated probability of stability against predicted hull distance (iso- tonic map) for five representative models.b, DFT-stable fraction among candidates predicted within±5 meV/atom of the hull, without any calibration map.c, Width of the indecision band 0.2≤p≤0.8 against mean absolute error. d, Candidates certified at a false-discovery rate of at most 10% by conformal selection (interatomic potentials; mean over 20 calibration draws).e, Realised false-discovery rate of conformal selection calibrated on a random slice (grey) or on WBM rounds 1–2 and applied to rounds 4–5 (red); potentials sorted by F1.f, DFT-stable fraction among each model’skmost confident ‘stable’ calls. Colours inaandf: models; inb–d: training data. — and it forecasts its own yield to within 0.3% (median over models), whereas the hard rule implicitly promises that every shortlisted candidate is stable and delivers 24%–93% of that promise (Extended Data Fig. 1b); 200 random DFT labels suffice for a nearly optimal map (Extended Data Fig. 1c). And MP- trained potentials whose F1 scores differ by less than 0.01 differ up to 7.8-fold in the size of the shortlist that conformal selection 32 certifies at a 10% false-discovery rate (Fig. 2d,f; Supplementary Note 1). One forward pass rivals a relaxation CrystalJev itself decides from one unrelaxed forward pass. We trained heads on the WBM candidates of 45% of the chemical systems, calibrated them on a random 5% slice of the others and tested them on the rest, against each backbone’s own relax-and-threshold practice: its published relaxed prediction, recalibrated on the same slice (Methods). For the two MP-trained potentials behind the original leaderboard, the single pass decides better than the relaxation (Fig. 3a; Extended Data Table 1; Supplementary Note 2): log-loss 0.191 against 0.247 for MACE-MP-0 4 and 0.196 against 0.266 for CHGNet 5 , at a thirtieth of the cost. For four newer potentials, whose relaxed predictions are far more accurate, it is slightly worse (log-loss higher by 0.009–0.033), and a head given the relaxed prediction as one more input improves on the relaxation for all six backbones. The advantage is bought with labels: trained on about 97,000 WBM structures, the single-pass head matches an 4 0.100.150.200.25 Log-loss, stability question MACE-MP-0 CHGNet MACE-MPA-0 MatterSim-v1-5M ORB-v3 SevenNet-MF-ompa Relaxed, isotonic map Single pass, decision head Decision head + relaxed 0.000.250.500.751.00 Share of candidates relaxed 0.0 0.2 0.4 0.6 0.8 1.0 Share of stable candidates kept 0.20.50.8 0.94 0.97 1.00 131030 Cost per candidate (single passes) 40 45 50 55 60 65 Net value per 1,000 candidates Single pass alone Screen, then relax 1% recall budget aOne pass vs a relaxationbScreening before relaxationc Cost and value (c = 0.5) Fig. 3|One forward pass across six foundation potentials.WBM test candidates, three seeds.a, Log-loss of the stability question for each backbone’s published relaxed prediction with an isotonic map, for the single-pass decision head and for the head given the relaxed prediction as one more input.b, Screening curves of the single-pass head: share of stable candidates kept against share of candidates sent to relaxation in order of decreasing single-pass probability (dashed line: 99% of stable candidates kept; inset: detail near 99%).c, Cost per candidate in forward passes (one pass, plus a relaxation of thirty passes for each candidate sent on) against realised net value atc=0.5, for the conformal screen at recall budgetsαfrom 0.2 to 0 (right end: relax every candidate); large open circles:α=0.01; diamonds: the single pass alone. Colours inb,c: backbones as ina. isotonically recalibrated relaxation only after about 10,000 of them (3,000 against a Platt-recalibrated one; Extended Data Fig. 3d), whereas recalibrating a relaxed prediction needs a few hundred. Deciding directly also beats predicting an energy with an uncertainty and propagating it through the threshold 16 : on identical inputs, a heteroscedastic regression of the hull distance overstates the stable frac- tion by 2.2 percentage points (MACE-MP-0) and, even recalibrated, reaches a log-loss of 0.211 against 0.192 (Extended Data Fig. 2a), because it spends its accuracy across the whole distribution rather than at the one point where the decision is taken. Knowing when to think slowly For the newer potentials a relaxation still adds information, so the question becomes which candidates to relax — one that a calibrated System 1 can answer about itself. Deciding at once on a candidate with calibrated System-1 probabilityp 1 is worth max(p 1 −c,0); consulting a slower system, which updates it to p 2 , is worth the expectation of max(p 2 −c,0), less its costk. Because a calibrated update does not shift the probability on average,E[p 2 |p 1 ] =p 1 , the value of information 34 VOI(p 1 ) =E max(p 2 −c,0)|p 1 −max(p 1 −c,0)(1) is non-negative and never exceeds its value for a perfect oracle that reveals the label (Methods): 0≤VOI(p 1 )≤minp 1 (1−c),(1−p 1 )c.(2) Escalation can therefore pay only inside the bandk/(1−c)<p 1 <1−k/c, whatever the slower system (Fig. 4a); outside it, a calibrated System 1 should decide alone, however accurate the alternative. Inside the band, the value depends on how much the slower system knows beyond System 1. For Gaussian uncertainties it has a closed form inp 1 ,cand one information ratior, the precision of System 2 relative to the uncertainty of System 1 (Methods), and this form describes the measured value of relaxation for all six potentials (Fig. 4b), withrrising from 0.23 (CHGNet) and 0.27 (MACE-MP-0) to 1.2 (ORB- v3 6 ) and 1.8 (SevenNet-MF-ompa), so relaxation is more informative for potentials trained on more data (Supplementary Note 3). 5 0.0010.010.10.50.90.99 System-1 probability p 1 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Value of escalation (per hit) Perfect System 2 r = 4 r = 1 r = 0.25 0.0010.010.10.50.90.99 System-1 probability p 1 0.00 0.05 0.10 0.15 0.20 Measured value of escalation Fitted information ratio r MACE-MP-0 (0.27) CHGNet (0.23) MACE-MPA-0 (0.52) MatterSim-v1-5M (0.63) ORB-v3 (1.2) SevenNet-MF-ompa (1.8) 01020305075100 Candidates relaxed (%) 0.0 0.2 0.4 0.6 0.8 1.0 Share of relax-all gain kept Escalation order Value of information Closest to c Most probable first Random Recall screen (α = 1%) 20304050607080 Net value per 1,000 candidates MACE-MP-0 CHGNet MACE-MPA-0 MatterSim-v1-5M ORB-v3 SevenNet-MF-ompa 21%22% 23%23% 15%11% 14%10% 15%7% 16%8% Threshold Band p 2 DFT for every candidate DFT if relaxed ̂ E hull ≤0 Band on p 1 (one pass) Band on p 2 (after relaxation) aTheory: a band of probabilitiesbRelaxation, measured cEscalation under a budgetd When to run DFT (c = 0.5, k = 0.05) Fig. 4|Knowing when to think slowly. a, Value of escalating a candidate to a slower system against its System-1 probabilityp 1 at attempt costc=0.5: the perfect-oracle bound (Eq. 2) and the Gaussian closed form for several information ratiosr. A slower system costingkcan pay only inside the marked band.b, Measured value of escalating from the single-pass head to the head with the backbone’s relaxed prediction (points: equal-mass bins of the test candidates, three seeds) and the one-parameter fit, for six potentials.c, Share of the gain of relaxing every candidate retained when a fraction of candidates is relaxed, for four escalation orders (mean and range over six potentials and three seeds,c=0.5); diamonds: the conformal screen at a 1% recall budget.d, Net value per 1,000 candidates when a DFT calculation (costk=0.05) can precede an attempt (c=0.5): computing every candidate, computing those whose relaxed hull distance is at most zero, and the three-way band (discard, compute, attempt) on the single-pass probabilityp 1 and on the probability after relaxationp 2 ; percentages: share of candidates sent to DFT. Under a budget, the theory says to relax the candidates with the largest value of information, estimated on the calibration slice. Relaxing 10% of the candidates in this order recovers 78% of the gain of relaxing all of them, and 20% recovers 93% (Fig. 4c). The conventional funnel, most probable first, recovers 1% and 78%, because the candidates it relaxes first would have been attempted anyway; relaxing those closest to the decision boundary nearly matches value of information atc=0.5 (75% at 10%) but not atc=0.7 (66% against 84%). A conformal screen guarantees recall instead: at a 1% budget it relaxes 41%–58% of the candidates and keeps at least 0.988 of the stable ones (Fig. 3b,c). DFT, which reveals the label, attains the oracle bound: discard belowk/(1−c), attempt above 1−k/cand compute in between. Withc=0.5 and DFT at a twentieth of a hit, this band on the single-pass probability beats computing every candidate whose relaxed hull distance is at most zero for all six potentials (58–65 against 47–62 net hits per 1,000 candidates; Fig. 4d). 6 One pass answers many questions A System-1 layer is useful in proportion to the questions that one state can answer (Fig. 5; ordinal and structural questions in Supplementary Note 4). Although the backbones were trained on energies, forces and stresses, their single pass anticipates the electronic structure of the relaxed crystal: whether DFT will find a metal is answered with AUROC 0.977–0.989 (composition alone: 0.951), and whether the PBE band gap falls between 1 and 2 eV, a proxy for photovoltaic absorbers, with AUROC 0.971–0.984 (Fig. 5a). Conjunctions such as ‘stable and in that window’ need compound answers: multiplying two calibrated answers under-forecasts them by up to 14%, because the events are positively associated given the state (Fig. 5b; Methods). Deciding directly pays beyond stability. On 12 screening questions over seven Matbench property datasets 35 , from band gaps and elastic moduli to phonons, a head on one MatterSim or ORB-v3 pass beats both regress-then-threshold baselines in log-loss for 20 of the 24 question–backbone pairs (Fig. 5c,d), most clearly at the edges of a property’s distribution, where no monotone map of a regression’s output can absorb noise whose shape varies between inputs (Methods). Nor is the layer specific to crystals. On 130,831 QM9 molecules 36 , one pass of the molecular potential MACE-OFF23 37 or of the crystal-trained ORB-v3 answers questions about the HOMO–LUMO gap and the dipole moment with AUROC 0.970–0.998, beating both baselines for all six question–backbone pairs (Fig. 5e). Asked which of five isomers has the lowest energy, a choice head that corrects the backbone’s own energy raises top-1 accuracy from 90.2% to 98.0% (MACE-OFF23) and from 85.0% to 96.8% (ORB-v3; Fig. 5f). A crystal-trained backbone served molecules almost as well as a molecular one. Guarantees hold where the calibration data come from Every guarantee above assumes calibration data exchangeable with the candidates, and discovery breaks this assumption in three ways. First, labels are plentiful where decisions are not: a MACE-MP-0 head trained and calibrated on MP and applied to the later WBM rounds covers only 0.87 of the true labels with nominal 90% conformal sets, and conformal selection with a linear head realises a false-discovery rate of 0.24 at a nominal 0.1, whereas calibrating on a random slice of the candidates restores 0.90 and 0.10 (Extended Data Fig. 3c). Calibrating on early WBM rounds and deciding on later ones breaks false- discovery control for 20 of the 25 MP-trained leaderboard potentials (Fig. 2e). Second, guarantees hold on average over chemistries, not within each: one map misstates the stable fraction of the worst anion family by 1.7 percentage points for the median MP-trained potential, and one map per family (Mondrian calibration 30 ) reduces this to 0.5 (Extended Data Fig. 2d; Supplementary Note 5). Third, closed loops select their own calibration data: in simulated DFT campaigns, recalibrating on the latest acquisitions underestimates the stable structures left in the pool by 76%–80% (Extended Data Fig. 4). A registered prospective test To test the layer where it is meant to be used, on structures that nobody had computed, we generated 8,000 new candidates as WBM was generated — ionic substitutions into MP prototypes 38 , keeping only compositions absent from MP and WBM — and, before computing any of them, registered the forecasts of six backbones, a random DFT sample (200 calibration and 400 test candidates) and the 100 candidates that a screening campaign would compute next, those ranked highest by the primary backbone, ORB-v3 (Methods). Our DFT labels follow MP’s protocol, aligned to MP’s energy scale by per-element corrections fixed in advance; on control structures they reproduce WBM’s stability label for 92 of 98 (Extended Data Fig. 5). Of the 397 test candidates with labels, 23 (5.8%) proved stable. Calibrated on WBM alone, the reg- istered single-pass forecasts over-stated this fraction by 0.8–2.1 percentage points, with bootstrap 95% intervals that include zero for four of the six backbones, and ranked the candidates with AUROC 0.965– 0.982 (Fig. 6a,b; intervals and AUROC were not registered). A sensitivity analysis of the DFT labels, also not registered, puts zero inside all six intervals. Against relax-and-threshold, the single-pass rules of CHGNet and MACE-MP-0 gained 17 and 26 net hits per 1,000 candidates at an attempt cost of 0.7 (paired 7 0.9000.9250.9500.9751.000 AUROC (WBM test set) On hull Metal Gap 1–2 eV Stable & non-metal Stable & gap 1–2 eV −15−10−505 Forecast − realised positives (% of realised) Product of marginals Recalibrated product Dedicated head Product of marginals Recalibrated product Dedicated head 0.711.523 Log-loss relative to the decision head Refractive index ≥ 2.5 Exfoliation ≤ 100 meV/atom G ≤ 15 GPa G ≥ 100 GPa K ≤ 30 GPa K ≥ 150 GPa Metal (gap = 0) Gap 1–2 eV Gap ≥ 3 eV Perovskite E f ≤ 0.5 eV Phonon peak ≤ 300 cm −1 Phonon peak ≥ 800 cm −1 0.000.050.100.150.20 ECE per Matbench question Decision head Regress, calibrated threshold Regress, propagated residuals Composition only 0.711.52358 Log-loss relative to the decision head HOMO–LUMO gap ≥ 8 eV HOMO–LUMO gap ≤ 5 eV Dipole ≥ 4 D 0.00.20.40.60.81.0 Top-1 accuracy, lowest-energy isomer of five MACE-OFF23, own energy MACE-OFF23, choice head ORB-v3, own energy ORB-v3, choice head 90.2% 98.0% (log-loss 0.05) 85.0% 96.8% (log-loss 0.08) aQuestions from one passbCompound questions cMatbench property questionsdCalibration of the answers eQM9 moleculesfChoice among isomers (QM9) MACE-MP-0 CHGNet MACE-MPA-0 MatterSim-v1-5M ORB-v3 SevenNet-MF-ompa Composition only Physics scalars only Regress, calibrated threshold Regress, propagated residuals Fig. 5|One pass, many questions.Key at the top: backbones (a,b) and baselines (a,c–e).a,b, WBM test candidates, three seeds.a, AUROC of stability, electronic-structure and compound questions answered from one forward pass of each backbone on the unrelaxed structure (filled), from element fractions only (open squares) and from the ten physics scalars only (open triangles; range over backbones).b, Signed error of the forecast number of candidates satisfying a compound question, relative to the realised number, for the product of two calibrated marginal answers, that product recalibrated on the slice and a head trained on the compound.c, Log-loss of screening questions on Matbench property datasets (five-fold cross-validation; one MatterSim-v1-5M pass and, faded, one ORB-v3 pass) for a regression with a calibrated threshold, a regression with propagated residuals and a composition-only head, relative to the head trained on the question (vertical line; values above 1 are worse).d, Expected calibration error of the same answers.e, Ascfor yes/no questions on 130,831 QM9 molecules (MACE-OFF23 and, faded, ORB-v3).f, Choice among five isomers of one formula: top-1 accuracy of the backbone’s own energy and of the choice head (log-loss in brackets). 8 bootstrap 95% intervals 5 to 31 and 11 to 42) and 8 and 9 at 0.5, within the uncertainty; for MatterSim and ORB-v3 the two rules differed by at most 4 (Fig. 6c). The conformal procedures were consistent with their guarantees but, with stable candidates this rare, too weak to be useful (Supplementary Note 7). Of the 100 shortlisted candidates (79 distinct compositions), 98 proved stable against MP’s hull (Fig. 6d), ten of them in systems in which MP holds no binary compound of the cation with one of its anions (Er–Br, Os–I): the yield measures agreement with the benchmark’s definition of stability rather than discovery. Most forecasts were consistent with the yield or under-stated it, but that of the primary backbone, 99.9±0.3, was over-confident: two or more failures had a probability of 0.005 if outcomes were independent. They were not. Both failures, SrWSe 2 F and CaWSe 2 F, share one prototype and a+Utreatment of tungsten triggered by one fluoride ion per formula unit. DFT places them 0.66 eV/atom above the hull before relaxation (0.53 after), whereas the single passes of three backbones, the primary one included, placed them below it; the other three gave both a probability of at most 0.01, and no other shortlisted candidate received less than 0.09 from any forecast. Relaxations from other starting structures confirmed both labels (Supplementary Note 7). A shortlist built from one model’s ranking concentrates that model’s errors, which come in groups, so its yield is less certain than its binomial spread suggests; a second backbone’s single pass is a cheap way to find such groups. Discussion These results argue for a different division of labour in computational discovery. A frozen foundation po- tential, read once, already knows enough to settle most candidates, and what it cannot settle it can delegate deliberately, because a calibrated probability states how much a slower computation could change the de- cision. Ordering slow computation by probability, as a screening funnel does, spends it where it changes nothing; ordering it by value of information spends it where decisions are open. The information ratios we measured, higher for potentials trained on more data, suggest that this allocation may matter more, not less, as potentials improve. As generative models propose candidates by the billion 1;2 , deciding where to think slowly, rather than thinking slowly everywhere a little faster, may be what keeps first-principles computation and experiments focused where they matter. Calibration does not make a model rank candidates better; it changes the meaning of the numbers a campaign acts on — how many hits a shortlist will deliver, where a threshold should sit given the cost of a failure, which candidates can be certified and which deserve slower computation. Its guarantees hold for calibration data exchangeable with the candidates and fail, sometimes badly, under the shifts that discovery creates. The prospective test adds two lessons: a calibration fitted on existing data carried over, to within about two percentage points, to new candidates generated the same way; and when stable candidates are rare, the size of the random sample, not the quality of the calibration, limits what conformal methods can guarantee. The practical recipe is to compute a random sample of the pool being decided about, sized for the guarantees a campaign needs; to calibrate out of sample inside closed loops; and to state the population over which a guarantee is claimed. Stability benchmarks could report, next to F1, a proper score after calibration on a disclosed slice, the stable fraction near the threshold and the size of a shortlist certified at a fixed false-discovery rate; the audit code, to be released with this work, computes all three from any prediction file. Two design choices carry beyond stability. Heads trained on the question decide better than models of the energy with an uncertainty, because a model spends its capacity where its loss puts it. And a frozen backbone makes the layer backbone-agnostic, so that it ages with the field instead of competing with it; training backbones end to end on decision losses, as System-one models for software are trained 29 , is a natural next step. Nor is the interface specific to crystals or to stability. Foundation potentials for surfaces and adsorbates 7 provide states for catalysis, where ‘is the adsorption energy inside the activity window?’ is a decision of the same kind. Experimental outcomes are the most valuable labels and the strongest shift: whether a synthesis yields the predicted ordered phase 39;40 is a typed question whose calibration evidence must come from experiments, and a shared registry of typed questions, each with its label recipe and calibration population, would let such answers be audited across laboratories. Our study has limitations. All stability labels are DFT energies relative to a fixed MP hull, so DFT and 9 <1% n = 245 1–5% n = 97 5–20% n = 27 20–50% n = 9 ≥50% n = 19 Registered single-pass probability (ORB-v3) 0 3 6 9 12 15 18 Stable structures, test sample Realised (DFT) Forecast, registered Forecast, recalibrated −2024 Forecast − realised stable fraction (percentage points) MACE-MP-0 CHGNet MACE-MPA-0 MatterSim-v1-5M ORB-v3 SevenNet-MF-ompa System 1, registered System 2, registered System 1, recalibrated System 2, recalibrated 0.10.20.30.40.50.60.7 Attempt cost c (hit = 1) −20 −10 0 10 20 30 40 Net value per 1,000 candidates System 1: p 1 ≥c System 2: p 2 ≥c Relax, then ̂ E hull ≤0 MACE-MP-0 CHGNet MACE-MPA-0 MatterSim-v1-5M ORB-v3 SevenNet-MF-ompa 87.590.092.595.097.5100.0 Stable structures among the 100 (forecast, 95% interval) After stage 1, Platt After stage 1, isotonic MACE-MP-0, System 1 MACE-MP-0, System 2 CHGNet, System 1 CHGNet, System 2 MACE-MPA-0, System 1 MatterSim-v1-5M, System 1 MatterSim-v1-5M, System 2 ORB-v3, System 1 ORB-v3, System 2 SevenNet-MF-ompa, System 1 aCalibration in countsbCalibration in the large cRealised valuedThe shortlist Fig. 6|A registered prospective test with new DFT calculations. a, Stable structures found by DFT (bars) and forecast (markers,±1.96 s.d.) among the 397 randomly sampled test candidates, by bin of the registered single-pass probability of ORB-v3; filled markers: registered forecasts, calibrated on WBM only; open markers: recalibrated on the 198 candidates of the random calibration slice. In all, 23 of the 397 candidates were stable, against 26.3 forecast at registration and 23.3 after recalibration.b, Forecast minus realised stable fraction of the test sample for each backbone’s single-pass (System 1) and post-relaxation (System 2) forecasts, registered and slice-recalibrated (bootstrap 95% intervals).c, Realised net value per 1,000 test candidates of the System-1 rulep 1 ≥c, the System-2 rulep 2 ≥cand relax-and-threshold, against the attempt costc(ORB-v3 bold).d, The shortlist a screening campaign would compute: its number of stable structures forecast at registration by every backbone and after stage 1 (ORB-v3 recalibrated on all stage-1 labels and registered before the shortlist’s DFT), against the realised number; lines: 95% intervals of the yield if outcomes were independent with the forecast probabilities. Colours inb–d: backbones. hull-construction uncertainty near the hull 41 caps the attainable calibration, gaps in MP’s reference phases make some candidates look more stable than they are, and experimental stability is a further shift. The single-pass heads were trained on 10 5 or more in-domain labels; with few labels, a recalibrated relaxation remains the better choice. Our candidates are generated by substitution, and candidates from generative models are another shift. The value-of-information analysis and the binomial yield forecasts treat candi- dates as independent, and the prospective shortlist shows what this assumption costs. We tested six frozen crystal potentials and one molecular potential, but not the largest current models or backbones fine-tuned on decisions, and compositional disorder lies outside the ordered-structure benchmark. A System-1 layer turns a foundation model from a simulator into a decision-maker that knows when to ask for help. Crystal stability was a convenient place to start, because its data are abundant and its slow systems well defined; the logic of calibrated answers and deliberate escalation applies wherever fast models and slow computations must share a budget. 10 Methods Data.We use the Matbench Discovery release files 9 : the MP reference set 33 (154,718 entries with en- ergies and relaxed structures) and the WBM candidate set 3 (256,963 structures generated by chemical substitution in five rounds; 215,488 unique prototypes, of which 15.5% lie on or below the MP hull after the leaderboard’s 1-meV rounding). All evaluations use the unique prototypes, as the leaderboard does. Every decision is made from the initial, unrelaxed WBM structure; DFT-relaxed structures, band gaps and symmetry labels define labels only. Leaderboard predictions are the published WBM formation energies of all 65 models with published predictions in October 2026. Probabilities from leaderboard predictions.The MP hull used by Matbench Discovery is fixed and contains no WBM entry. Hence, for any predicted formation energy ˆe form , ˆ E hull =E DFT hull + ˆe form −e DFT form =ˆe form −e hull form (x),(3) wheree hull form (x), the formation energy of the MP hull at the candidate’s compositionx, is available before the candidate is computed. We turn ˆ E hull into a probability with one-dimensional maps fitted on a disclosed random 5% slice of the pool (five seeds): Platt scaling 42 , isotonic regression 43 and the ‘error-bar’ practice p=Φ(− ˆ E hull /σ), withσthe model’s root-mean-square error. Each map exposes the calibrated probability at the threshold,p 0 =p( ˆ E hull =0), the 50% point ˆ E hull,50 and the widthWof the band 0.2≤p≤0.8; we report isotonic maps, which fit best, and the map-free stable fraction among candidates predicted within ±5 meV/atom of the hull. The exponential-tilt closed form.If the true hull distances of the pool have a local density proportional toe κE near the hull and a model’s errors there are Gaussian with biasband spreads, Bayes’ rule makes the true hull distance given ˆ E hull =tGaussian with meant−b+κs 2 and standard deviations, so that P(stable| ˆ E hull =t) =Φ b−κs 2 −t s ,(4) with 50% point ˆ E hull,50 =b−κs 2 and value-maximising thresholdt c =b−κs 2 −sΦ −1 (c). We estimateκ from the WBM hull-distance histogram near the hull, andbandsas robust estimates of the location and scale of each model’s errors on candidates near the hull. Backbones and the single-pass state.Each backbone is evaluated once per structure: MACE-MP-0 4 , CHGNet 5 , MACE-MPA-0, MatterSim-v1-5M, ORB-v3 6 and SevenNet-MF-ompa. The state consists of pooled per-atom representations that enter each model’s energy readout and ten physics scalars of the single pass, among them the single-point formation energy and hull distance (Eq. 3), force and stress statistics and simple size descriptors. Relaxations follow the leaderboard’s protocol. Heads, calibration and splits.Heads are gradient-boosted trees or linear probes, trained with log-loss and Platt-scaled on the calibration slice. Ordinal questions use cumulative binary heads with monotone differencing; choice questions use a softmax over a predicted energy (the backbone’s own energy plus a learned residual) with a temperature fitted on held-out compositions. The WBM unique prototypes are split by chemical system: 45% of the systems form the training set, and the remaining structures are divided at random into a calibration slice (5% of all structures) and the test set, so that calibration and test data are exchangeable while both are chemically disjoint from training (three seeds). The relax-and-threshold reference recalibrates each backbone’s published relaxed prediction with an isotonic map on the same slice. Conformal procedures.Split-conformal prediction sets atα=0.1 use the non-conformity score 1− p y 30;31 . Conformal selection 32 computes conformalp-valuesp j = (1+#i∈C:y i =0,s i ≥s j )/(|C|+ 1)from the calibration sliceCand applies the Benjamini–Hochberg procedure at levelq, which bounds the false-discovery rate of the certified shortlist when calibration data and candidates are exchangeable. 11 The screen before relaxation discards candidates whose single-pass probability falls belowτ α , thek-th smallest probability among then + stable structures of the slice withk=⌊α(n + +1)⌋; a new stable candidate exchangeable with the slice is then discarded with probability at mostα. Mondrian variants fit one map and one procedure per anion family (halide if F, Cl, Br or I is present, else oxide, chalcogenide, pnictide or intermetallic, in this order of precedence). Value of information.LetY∈0,1be the label,p 1 =P(Y=1|S 1 )the calibrated System-1 probability andp 2 =P(Y=1|S 1 ,S 2 )the calibrated probability after System 2 is consulted; an attempt costsc∈(0,1) in units of a hit, and escalation costsk. Propositions 1 and 2 specialise classical results on the value of sample and perfect information and on Gaussian preposterior analysis 34;44 to a calibrated yes/no decision; calibration is what makes them usable, becausep 1 andp 2 can then be estimated, and the bounds checked, on held-out data. Proposition 1.VOI(p 1 ) =E[u(p 2 )|S 1 ]−u(p 1 )withu(p) =max(p−c,0)satisfies 0≤VOI(p 1 )≤ minp 1 (1−c),(1−p 1 )c, with equality on the right ifS 2 revealsY.Proof.Non-negativity is Jensen’s inequality for the convexuwithE[p 2 |S 1 ] =p 1 . For the upper bound,u(q)≤q(1−c)on[0,1], so forp 1 <c, VOI=E[u(p 2 )]≤p 1 (1−c); forp 1 ≥c, VOI=E[u(p 2 )]−(p 1 −c) =E[max(c−p 2 ,0)]≤cE[1−p 2 ] = c(1−p 1 ). Ifp 2 =Y,E[u(Y)] =p 1 (1−c)and both bounds are attained. Hence escalation at costkcan pay only ifk/(1−c)<p 1 <1−k/c.□ Proposition 2 (Gaussian refinement).Suppose that givenS 1 the hull distance isE∼N(μ 1 ,σ 2 1 ), so p 1 =Φ(−μ 1 /σ 1 ), andS 2 observesZ=E+εwith independentε∼N(0,τ 2 ). With the information ratio r=σ 2 1 /τ 2 ,a=Φ −1 (p 1 ),b= ( √ 1+r a−Φ −1 (c))/ √ randρ= p r/(1+r), VOI(p 1 ;c,r) =Φ 2 (a,b;ρ)−cΦ(b)−max(p 1 −c,0),(5) whereΦ 2 is the standard bivariate normal distribution function.Proof.The posterior afterZis Gaussian with varianceσ 2 2 =σ 2 1 τ 2 /(σ 2 1 +τ 2 )and meanμ 2 , which givenS 1 is distributed asN(μ 1 ,v 2 )withv 2 = σ 4 1 /(σ 2 1 +τ 2 );E−μ 2 is independent ofμ 2 , so(E,μ 2 )are jointly Gaussian with correlationv/σ 1 =ρ. The attempt is made if and only ifp 2 =Φ(−μ 2 /σ 2 )≥c, that isμ 2 ≤m=−σ 2 Φ −1 (c), andE[u(p 2 )] =P(E≤ 0,μ 2 ≤m)−c P(μ 2 ≤m); standardising gives Eq. 5, which depends on(p 1 ,c,r)only.r→∞recovers the oracle bound andr→0 gives zero.□Eq. 5 agrees with Monte Carlo simulation to four decimals. Measurement.System 1 is the single-pass head; System 2 is the head given the backbone’s published relaxed prediction as one more input. The martingale property requiresp 2 to be calibrated conditionally onp 1 , sop 2 is recalibrated by a logistic regression on(logitp 1 ,logitp 2 )fitted on the calibration slice. The estimated d VOI(p 1 )is a kernel regression ofu(p 2 )on logitp 1 over the slice minusu(p 1 ); it needs System-2 outputs on the slice but no further labels. Measured curves averageu(p 2 )−u(p 1 )over equal-mass bins of p 1 on the test set, andris fitted to them by weighted least squares.Budgeted escalation.With a budget ofBescalations and independent refinements, expected value is separable across candidates, so escalating theBcandidates with the largest value of information is optimal (an exchange argument); we compare this order with the most probable first, the closest to the decision boundary (|p 1 −c|smallest) and random order. DFT allocation.With attempt costcand DFT costk, discarding, attempting directly and computing first are worth 0,p−candp(1−c)−k; the optimal rule is the band of Proposition 1 for a perfect oracle. Typed questions.Electronic questions on WBM use the PBE band gap of the relaxed structure 3 : metal (gap=0), photovoltaic window (1≤gap≤2 eV) and a five-level gap question (0,(0,1],(1,2],(2,3], >3 eV), with the splits of the stability question. Compound questions (stable and in the photovoltaic window; stable and non-metallic) are answered by a dedicated head, by the product of the two calibrated marginal heads, and by that product recalibrated on the slice. If the marginal answers are calibrated con- ditionally on the states, thenE[p A p B ]−P(A∧B) =−E Cov(1 A ,1 B |s) : the product under-forecasts the compound event exactly when the two events are positively associated given what the layer knows. Mat- bench property questions 35 threshold the datasets’ targets (band gap, bulk and shear moduli, refractive index, highest phonon density-of-states peak, exfoliation energy and perovskite formation energy; thresh- olds in Supplementary Note 4) and use one MatterSim-v1-5M or ORB-v3 pass on the dataset structures, with five-fold cross-validation and 10% of each training fold held out for calibration. The baselines, on 12 the same features, are a gradient-boosted regression of the target whose prediction minus the threshold is Platt-calibrated, the same regression with its empirical residual distribution propagated through the thresh- old, and a head on element fractions. A regression can decide worse because, with residualr=y− ˆ f(x), the answer to ‘isy≥t?’ isP(r≥t− ˆ f(x)|x): a monotone map of ˆ f(x)can equal it only if the residual distribution is the same for all inputs with the same prediction, whereas a head trained with log-loss on the question hasP(y≥t|x)itself as its population minimiser. QM9 molecules 36 (130,831 after remov- ing those that failed the original consistency check), in their B3LYP geometries, are passed once through MACE-OFF23 37 or ORB-v3; yes/no questions threshold the HOMO–LUMO gap (≥8 eV,≤5 eV) and the dipole moment (≥4 D), with the same protocol and baselines. The choice question draws sets of five molecules of one formula within the test fold and asks which has the lowestU 0 ; the choice head scores each molecule by the backbone’s energy plus a correction learned on the training fold and fits its softmax temperature on sets drawn within the calibration fold. Shift, Mondrian calibration and closed loops.For the shift experiment, a head trained on MP (split 90/10 by chemical system into training and calibration) is calibrated on three sources of increasing proxim- ity to the candidates — MP calibration systems, first-round WBM substitutions of the training systems and the random WBM slice — and evaluated on the WBM rounds 2–5 structures of the test set. Budgeted loops draw a pool of 30,000 WBM unique prototypes and a 500-structure random seed; in each of 20 rounds a logistic head on single-pass features (a compressed pooled embedding plus the physics scalars) is refitted, calibrated and used to acquire 250 structures, whose labels are then revealed. Calibration maps are fitted on the most recent 30% of acquired labels, on all acquired labels with five-fold cross-fitted predictions, on the random seed only, or on a 10% uniform random sample of every batch; five seeds. Prospective test.Candidates.Following WBM 3 , candidates are data-mined ionic substitutions 38 into near-hull MP prototypes with two to four elements and small cells, drawn from the elements for which DFT labels can be put on MP’s energy scale; only compositions absent from MP and WBM are kept. The pool is the 8,000 most probable candidates (5,780 compositions in 4,221 chemical systems; median 10 sites).Registration.For each backbone, heads were trained on all WBM unique prototypes except a ran- dom 5% calibration slice and applied unchanged to the pool; System-2 forecasts use the backbone’s own relaxation of each candidate (MACE-MP-0, CHGNet, MatterSim-v1-5M and ORB-v3). All forecasts, the sampling seed, the stage-1 DFT sample — a uniform random sample of the pool, split at random into a calibration slice of 200 and a test sample of 400 candidates — the stage-2 shortlist — the 100 remaining candidates with the highest single-pass probability of the primary backbone, ORB-v3, fixed in advance — and the analysis plan were written to a registration file whose SHA-256 digest (8f359667a6acda7c. . . ) was recorded outside the computing cluster at 2026-10-02 21:12:07 UTC, before any DFT calculation of a pool member. After stage 1, and before the shortlist’s DFT, the primary backbone’s single-pass proba- bility was recalibrated on all stage-1 labels and the updated forecast of the shortlist’s yield registered in the same way (07d33c3f6708c63a. . . , 2026-10-03 16:12:04 UTC).DFT labels.Calculations use VASP 45 with MP’s GGA(+U) protocol and start, as in WBM, from the unrelaxed candidate structure. The projector- augmented-wave potentials 46;47 available to us carry MP’s symbols and titles but belong to a later release, so total energies differ from MP’s by amounts that are, to first order, constants of the elements. We there- fore fitted one shift per element (and one per+Uelement in oxides and fluorides) by robust least squares to recomputed MP elemental phases, binary+Uoxides and a first control set of MP and WBM structures, and fixed the shifts (SHA-256b88706bf90887082. . . , 2026-10-02 23:08:28 UTC) before any prospective calculation was collected. A second control set validated them, and a third, chosen to match the candidate pool and computed from WBM’s initial structures like the candidates, measured the residual label noise: aligned hull distances reproduce WBM’s to a median of 3.4 meV/atom, with WBM’s label for 92 of 98 structures and a tail set by relaxations that reach different minima from the same start (Supplementary Note 6). Hull distances include the MP2020 corrections 48 and are taken against the patched phase diagram of the MP 2023-02-07 entries, which reproduces the WBM labels of Matbench Discovery exactly; a candi- date is stable if its hull distance, rounded to 1 meV/atom, is at most zero. Calculations that did not finish (five of the 600 stage-1 calculations) are reported and excluded; all 100 shortlisted calculations finished. 13 Analysis.As registered: on the stage-1 test sample, calibration in the large (forecast minus realised sta- ble fraction), expected calibration error, log-loss and Brier score of every backbone’s registered System-1 forecast and, for the four backbones that relax, System-2 forecast, against the same forecasts recalibrated on the slice (Platt and isotonic); realised net value atc=0.1, 0.3, 0.5 and 0.7 of the rulep 1 ≥c, the rule p 2 ≥cand relax-and-threshold; recall of the conformal screen calibrated on the slice (α=0.05, 0.1); and the false-discovery proportion of conformal selection (q=0.1, 0.2) among the test sample with the slice as calibration set. Stage 2: the realised number of stable structures in the shortlist against every registered forecast. Secondary: the stage-1 analyses without the+Uchemistries. Not part of the registered plan, and reported as such: bootstrap intervals, AUROC and the five-bin implementation of the expected calibration error (all implemented in the evaluation script before the registration); sensitivity analyses of stage 1 to the DFT labels, which resample the residuals of the third control set (added after the registration but before any stage-1 label was collected); and post hoc analyses: paired bootstrap intervals of the differences in value between rules and, for the shortlist, DFT relaxations of its two failures without symmetry from MACE-MP- 0’s relaxed structure and from randomly displaced starts. Deviations from the registered plan are listed in Supplementary Note 7. Metrics and statistics.Proper scoring rules: log-loss (natural logarithm) and Brier score 49;50 ; expected calibration error with 10 equal-mass bins 51 (five in the prospective test, where its value for forecasts with tied probabilities depends on the order of ties by up to 0.01; we report the registered evaluation); dis- crimination: AUROC; ordinal questions: ranked probability score; choice questions: top-1 accuracy. Net value per 1,000 candidates is 1000 (y−c)1[attempt],(1−c)times the net benefit of decision-curve analysis at threshold probabilityc 52 . Results on WBM are means over three seeds (five for the audit maps and the closed loops); intervals in the prospective test are bootstrap 95% intervals over candidates (2,000 resamples; 10,000 in the sensitivity analysis and for paired differences in value between rules). Software and hardware.All computations, including the DFT calculations, ran on CPU clusters. Data availability This manuscript is currently under review. The data supporting its findings will be made publicly available upon publication. Code availability This manuscript is currently under review. The code will be made publicly available upon publication. References [1] Amil Merchant, Simon Batzner, Samuel S. Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery.Nature, 624:80–85, 2023. doi: 10.1038/s41586-023-06735-9. [2] C. Zeni et al. A generative model for inorganic materials design.Nature, 639:624–632, 2025. doi: 10.1038/s41586-025-08628-5. [3] Hai-Chen Wang, Silvana Botti, and Miguel A. L. Marques.Predicting stable crystalline com- pounds using chemical similarity.npj Computational Materials, 7:12, 2021.doi: 10.1038/ s41524-020-00481-6. [4] Ilyes Batatia et al. A foundation model for atomistic materials chemistry.The Journal of Chemical Physics, 163:184110, 2025. doi: 10.1063/5.0297006. arXiv:2401.00096. 14 [5] Bowen Deng, Peichen Zhong, KyuJung Jun, Janosh Riebesell, Kevin Han, Christopher J. Bar- tel, and Gerbrand Ceder. CHGNet as a pretrained universal neural network potential for charge- informed atomistic modelling.Nature Machine Intelligence, 5:1031–1041, 2023. doi: 10.1038/ s42256-023-00716-3. [6] B. Rhodes et al. Orb-v3: atomistic simulation at scale, 2025. arXiv:2504.06231. [7] B. M. Wood et al. UMA: A family of universal models for atoms. InAdvances in Neural Information Processing Systems, 2025. arXiv:2506.23971. [8] Nathan J. Szymanski et al. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624:86–91, 2023. doi: 10.1038/s41586-023-06734-w. [9] Janosh Riebesell, Rhys E. A. Goodall, Philipp Benner, Yuan Chiang, Bowen Deng, Gerbrand Ceder, Mark Asta, Alpha A. Lee, Anubhav Jain, and Kristin A. Persson. A framework to evaluate machine learning crystal stability predictions.Nature Machine Intelligence, 7:836–847, 2025. doi: 10.1038/ s42256-025-01055-1. [10] Christopher J. Bartel, Amalie Trewartha, Qi Wang, Alexander Dunn, Anubhav Jain, and Gerbrand Ceder. A critical examination of compound stability predictions from machine-learned formation energies.npj Computational Materials, 6:97, 2020. doi: 10.1038/s41524-020-00362-y. [11] Bowen Deng, Choi, Peichen Zhong, Janosh Riebesell, Anand, Li, KyuJung Jun, Kristin A. Persson, and Gerbrand Ceder. Systematic softening in universal machine learning interatomic potentials.npj Computational Materials, 11:9, 2025. doi: 10.1038/s41524-024-01500-6. [12] Bilbrey et al. Uncertainty quantification for neural network potential foundation models.npj Compu- tational Materials, 11:109, 2025. doi: 10.1038/s41524-025-01572-y. [13] Liu, Wei, Gao, Dey, Sluiter, and Shuang. Heterogeneous ensemble enables a universal uncertainty metric for atomistic foundation models.npj Computational Materials, 12:34, 2026. [14] Xu et al. Evidential deep learning for interatomic potentials.Nature Communications, 17:937, 2026. doi: 10.1038/s41467-025-67663-y. [15] Mehdi, Cho, and Olexandr Isayev. Knowing when to trust machine-learned interatomic potentials, 2026. arXiv:2605.00640. [16] A. Novick, D. Cai, Q. Nguyen, R. Garnett, R. Adams, and E. Toberer. Probabilistic prediction of material stability: integrating convex hulls into active learning.Materials Horizons, 11:5381–5393, 2024. doi: 10.1039/D4MH00432A. [17] Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, and Zachary W. Ulissi. Methods for comparing uncertainty quantifications for material property predictions.Machine Learn- ing: Science and Technology, 1:025006, 2020. doi: 10.1088/2632-2153/ab7e1a. [18] Daniel Varivoda, Rongzhi Dong, Sadman Sadeed Omee, and Jianjun Hu. Materials property predic- tion with uncertainty quantification: A benchmark study.Applied Physics Reviews, 10:021409, 2023. doi: 10.1063/5.0133528. [19] Dale, Brian DeCost, Jason Hattrick-Simpers, et al. When active learning fails, uncalibrated out of distribution uncertainty quantification might be the problem, 2025. arXiv:2511.17760. [20] Radova, Stark, Allen, Reinhard J. Maurer, and Albert P. Bartók. Fine-tuning foundation models of materials interatomic potentials with frozen transfer learning.npj Computational Materials, 11:237, 2025. doi: 10.1038/s41524-025-01727-x. 15 [21] L. Kong, N. Shoghi, G. Hu, P. Li, and V. Fung. MatterTune: an integrated, user-friendly platform for fine-tuning atomistic foundation models to accelerate materials simulation and discovery.Digital Discovery, 4:2253–2262, 2025. doi: 10.1039/D5D00154D. [22] N. Liu et al. Crys-JEPA: Accelerating crystal discovery via embedding screening and generative refinement, 2026. arXiv:2605.14759. [23] S. Edamadaka, K. Ramesh, J. Li, and R. Gómez-Bombarelli. Atomistic language models understand and generate materials, 2026. arXiv:2606.21395. [24] E. O. Pyzer-Knapp et al. Foundation models for materials discovery – current state and future direc- tions.npj Computational Materials, 11:61, 2025. doi: 10.1038/s41524-025-01538-0. [25] Daniel Kahneman.Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011. [26] Stuart Russell and Eric Wefald.Do the Right Thing: Studies in Limited Rationality. MIT Press, Cambridge, MA, 1991. [27] Nicholas Hay, Stuart Russell, David Tolpin, and Solomon Eyal Shimony. Selecting computations: Theory and applications. InProceedings of the 28th Conference on Uncertainty in Artificial Intelli- gence (UAI), 2012. [28] Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jonathan Lenchner, Nick Linck, Andrea Loreggia, Keerthiram Murugesan, Nicholas Mattei, Francesca Rossi, and Biplav Srivastava. Thinking fast and slow in AI. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 15042–15046, 2021. [29] D. Almeida. Introducing system one models & Jev. TypeSafe AI blog,https://typesafe.ai/ blog/introducing-system-one-models-and-jev, 2026. 15 September 2026. [30] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer.Algorithmic Learning in a Random World. Springer, 2005. [31] Anastasios N. Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction.Foun- dations and Trends in Machine Learning, 16(4):494–591, 2023. [32] Ying Jin and Emmanuel J. Candès. Selection by prediction with conformal p-values.Journal of Machine Learning Research, 24(244):1–41, 2023. [33] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Com- mentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1:011002, 2013. doi: 10.1063/1.4812323. [34] Ronald A. Howard. Information value theory.IEEE Transactions on Systems Science and Cybernetics, 2(1):22–26, 1966. [35] Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materi- als property prediction methods: the Matbench test set and Automatminer reference algorithm.npj Computational Materials, 6:138, 2020. [36] Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules.Scientific Data, 1:140022, 2014. doi: 10.1038/sdata.2014.22. [37] Dávid Péter Kovács, J. Harry Moore, Nicholas J. Browning, Ilyes Batatia, Joshua T. Horton, Yixuan Pu, Venkat Kapil, William C. Witt, Ioan-Bogdan Magd ̆ au, Daniel J. Cole, and Gábor Csányi. MACE- OFF: Short-range transferable machine learning force fields for organic molecules.Journal of the American Chemical Society, 147(21):17598–17611, 2025. doi: 10.1021/jacs.4c07099. 16 [38] Geoffroy Hautier, Chris Fischer, Virginie Ehrlacher, Anubhav Jain, and Gerbrand Ceder. Data mined ionic substitutions for the discovery of new compounds.Inorganic Chemistry, 50(2):656–663, 2011. [39] J. Leeman, Y. Liu, J. Stiles, S. B. Lee, P. Bhatt, L. M. Schoop, and R. G. Palgrave. Challenges in high-throughput inorganic materials prediction and autonomous synthesis.PRX Energy, 3:011002, 2024. doi: 10.1103/PRXEnergy.3.011002. [40] Jakob, Aron Walsh, Karsten Reuter, and Johannes T. Margraf. Learning crystallographic disorder: Bridging prediction and experiment in materials discovery.Advanced Materials, 38(5):e14226, 2026. doi: 10.1002/adma.202514226. [41] Hegde et al. Quantifying uncertainty in high-throughput density functional theory: A comparison of AFLOW, Materials Project, and OQMD.Physical Review Materials, 7:053805, 2023. doi: 10.1103/ PhysRevMaterials.7.053805. [42] John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. InAdvances in Large Margin Classifiers, pages 61–74. MIT Press, 1999. [43] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. InProceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699, 2002. [44] Howard Raiffa and Robert Schlaifer.Applied Statistical Decision Theory. Division of Research, Graduate School of Business Administration, Harvard University, Boston, 1961. [45] G. Kresse and J. Furthmüller. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.Physical Review B, 54(16):11169–11186, 1996. [46] P. E. Blöchl. Projector augmented-wave method.Physical Review B, 50(24):17953–17979, 1994. [47] John P. Perdew, Kieron Burke, and Matthias Ernzerhof. Generalized gradient approximation made simple.Physical Review Letters, 77(18):3865–3868, 1996. [48] Amanda Wang, Ryan Kingsbury, Matthew McDermott, Matthew Horton, Anubhav Jain, Shyue Ping Ong, Shyam Dwaraknath, and Kristin A. Persson. A framework for quantifying uncertainty in DFT energy corrections.Scientific Reports, 11:15496, 2021. [49] Glenn W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78:1–3, 1950. [50] Allan H. Murphy. A new vector partition of the probability score.Journal of Applied Meteorology, 12:595–600, 1973. [51] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using Bayesian binning. InProceedings of the AAAI Conference on Artificial Intelligence, 2015. [52] Andrew J. Vickers and Elena B. Elkin. Decision curve analysis: a novel method for evaluating pre- diction models.Medical Decision Making, 26(6):565–574, 2006. [53] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis.Compu- tational Materials Science, 68:314–319, 2013. 17 Extended Data 0.00.20.40.6 Predicted Φ((b−κs 2 )/s) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Calibrated P (stable) at ̂ E hull = 0 Potentials, MP-derived training Potentials, larger datasets Not potentials 0.00.20.40.6 Attempt cost c (hit = 1) 0 1 3 10 30 100 Value gained per 1,000 candidates 251004001,60010,774 DFT labels used to calibrate 0.1 0.3 1 3 10 Value lost per 1,000 candidates Calibrated, c = 0.1 Calibrated, c = 0.5 aTwo numbers per modelbValue of calibratingcLabels a map needs Extended Data Fig. 1|Where the leaderboard threshold sits, what it costs and what fixing it takes. a, Calibrated probability of stability at the threshold ˆ E hull =0 against the exponential-tilt closed form (Eq. 4), withκfitted once to the WBM population and each model’s error biasband spreadsnear the hull; 65 models.b, Net value gained per 1,000 candidates by the calibrated rulep≥cover the threshold, against the attempt costcin units of a hit (median and 10–90% range per group of models; colours as ina); dotted lines: each group’s median implied costc ∗ , at which the threshold is optimal.c, Value lost by the calibrated rule relative to the best monotone recalibration in hindsight, against the number of random DFT labels used to fit the isotonic map (25 MP-trained potentials; median and interquartile range over ten random halves of the pool); horizontal lines: the threshold rule at the same cost. 18 0.140.180.220.26 Log-loss, on-hull question Relaxed ̂ E hull , isotonic Energy UQ: Gaussian regression Gaussian regression + isotonic Quantile regression + isotonic One head for all thresholds Decision head (this question) MACE-MP-0 CHGNet −50050100150200 Threshold t of the question ̂ E hull ≤ t (meV/atom) 10 −3 10 −2 10 −1 10 0 ECE One head, shared calibration One head, recalibrated at t Dedicated head per threshold 0.00.20.40.60.8 Attempt cost c (hit = 1) −50 −25 0 25 50 75 100 125 150 Net value per 1,000 candidates Perfect foresight Relaxed, hard threshold Relaxed, calibrated Single pass, decision head 0.8000.8250.8500.8750.9000.9250.950 Coverage of 90% conformal sets Halide Chalcogenide Pnictide Intermetallic Oxide Relaxed, global map Relaxed, per family Head, global map Head, per family aDeciding vs energy uncertaintybMany thresholds (MACE-MP-0) cValue against cost (MACE-MP-0)dCoverage by chemical family Extended Data Fig. 2|Deciding directly versus propagating energy uncertainty.WBM test candidates, three seeds.a, Log-loss of the stability question for decision rules built on the same single-pass features, labels and cali- bration slice, and for the backbone’s relaxed prediction; vertical lines: an ensemble of the 25 MP-trained leaderboard potentials (mean and spread of their relaxed predictions with a boosted map) and its best member (isotonic map).b, Expected calibration error of a single threshold-conditioned head against the queried threshold, with one shared cali- bration map or recalibrated at the queried threshold, and of heads trained for each threshold; open symbols: thresholds not seen in training.c, Realised net value per 1,000 candidates against the attempt costc; dashed line: perfect fore- sight.d, Coverage of 90% conformal sets within each anion family, averaged over MACE-MP-0 and CHGNet, with one calibration map for all candidates (open) or one per family (filled). 19 0.000.250.500.751.00 Predicted P(stable) 0.0 0.2 0.4 0.6 0.8 1.0 Observed frequency MP head, MP calibration MP head, WBM 5% slice Relaxed ̂ E hull + Platt In-domain head Relaxed + Platt Relaxed + isotonic Single point MP head, MP cal. MP head, WBM cal. In-domain head Head + relaxed 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Log-loss, WBM test set Fusion MACE-MP-0 CHGNet MP WBM round 1 WBM random 5% Calibration source 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Error rate (test: WBM rounds 2 – 5) Miscoverage, 90% sets FDR at q = 0.1 Near-hull ECE MACE-MP-0 CHGNet 10 3 10 4 10 5 In-domain training labels 0.18 0.20 0.22 0.24 0.26 0.28 0.30 Log-loss, WBM test set MACE-MP-0: head MACE-MP-0: head + relaxed CHGNet: head CHGNet: head + relaxed Relaxed Single point Scalars only Head Head + relaxed 0 2000 4000 6000 8000 10000 Certified at FDR ≤ 0.1 MACE-MP-0 CHGNet 131030 Cost per candidate (single points) 0.175 0.200 0.225 0.250 0.275 0.300 0.325 Log-loss, WBM test set Single point Relaxed Head Head + relaxed MACE-MP-0 CHGNet aReliability (MACE-MP-0)bLog-loss by settingcCalibration under shift dLabels the head needseCertified shortlistsfCost and log-loss Extended Data Fig. 3|Transfer, shift and label efficiency of the decision layer on MACE-MP-0 and CHGNet. WBM test candidates, 3 seeds.a, Reliability of single-pass probabilities of stability (MACE-MP-0, first seed) for a head trained on MP and calibrated on MP or on the WBM slice, for the relaxed hull distance with Platt scaling and for the in-domain head.b, Log-loss by regime and backbone (cal.: calibration); horizontal line: fusion of both backbones.c, Shift ladder for heads trained on MP and evaluated on WBM rounds 2–5 as the calibration source moves towards the candidates: miscoverage of 90% conformal sets, near-hull expected calibration error and realised false-discovery rate of conformal selection atq=0.1; horizontal line: the 0.1 target.d, Log-loss against the number of in-domain training labels (two seeds); horizontal lines: relaxed prediction with Platt (solid) or isotonic (dashed) calibration.e, Candidates certified at a false-discovery rate of at most 0.1.f, Backbone cost per candidate in forward passes against log-loss. 20 020004000 DFT calls 0 500 1000 1500 2000 2500 3000 Stable structures found 020004000 DFT calls 0.00 0.02 0.04 0.06 0.08 0.10 ECE on remaining pool 020004000 DFT calls −40 −30 −20 −10 0 10 20 30 40 Next-batch forecast error (%) 020004000 DFT calls 0 500 1000 1500 2000 2500 3000 Stable structures found 020004000 DFT calls 0.00 0.02 0.04 0.06 0.08 0.10 ECE on remaining pool 020004000 DFT calls −40 −30 −20 −10 0 10 20 30 40 Next-batch forecast error (%) aCHGNet: structures foundbCHGNet: pool calibrationcCHGNet: next-batch forecast dMACE-MP-0: structures foundeMACE-MP-0: pool calibrationfMACE-MP-0: next-batch forecast Random acquisition Greedy, latest acquisitions (in-sample) Greedy, cross-fitted on all acquired Greedy, random seed only Greedy, 10% calibration tax Extended Data Fig. 4|Calibration inside budgeted DFT loops.Simulated campaigns on WBM with the hidden DFT labels as the oracle (top: CHGNet, bottom: MACE-MP-0 single-pass features; greedy acquisition unless stated, dashed lines: random acquisition; means over five seeds).a,d, Stable structures found after a 500-structure random seed.b,e, Expected calibration error on the remaining pool for calibration maps fitted on the most recent acquisitions, on all acquisitions with cross-fitted predictions, on the random seed or on a 10% random sample of every batch.c,f, Signed relative error of the forecast number of hits in the next batch. 21 B Be Se Nb Sb H Br Ir Tm O C Cs N Os Si W I Bi Al Ba Te Rb Hf La Pr Ta Re K Sr Dy Tb Nd Ho Er Ca Sm Lu F Sc Ti V Hg Pb Tl Cd Sn In Zn Ni Cu 0 100 200 300 δ el (meV per atom) Fitted shift E ours −E MP Elemental phases computed +U shift (oxides, fluorides) −50−1001050100300 Our energy minus reference (meV/atom) Set 1 (fit), raw Set 1 (fit), aligned Set 2 (validation), raw Set 2 (validation), aligned Set 3 (pool-matched), raw Set 3 (pool-matched), aligned MP+UWBM aPer-element shifts bControl structures, raw and aligned Extended Data Fig. 5|Putting new DFT energies on the Materials Project scale. a, Per-element shifts fitted to recomputed MP elemental phases, binary+Uoxides and a first control set (bars), the differences of the individual elemental phases (dashes) and the+Ushifts of Ni, V and W in oxides and fluorides (diamonds).b, Our energy minus the reference energy, before and after alignment, for the first (fit), second (validation) and third (matched to the prospective pool) control sets: MP ground states,+Ucompounds and WBM structures computed from their initial geometry, as the candidates were. Shifts were fixed before any prospective calculation was collected. 22 Extended Data Table 1 | The decision layer across backbones (stability question, WBM test set, means ± s.d. over three seeds; identical splits, labels and calibration slice). Relaxed: the backbone’s published leaderboard prediction after relaxation, with an isotonic map on the slice. Single pass: gradient-boosted head on the pooled node features andphysics scalars of one forward pass on the unrelaxed input, Platt-calibrated. N 0 . 1 : candidates certified by conformal selection at FDR ≤ 0 . 1. relaxed, isotonic single pass single pass + relaxed N 0 . 1 backbone training data AUROC NLL AUROC NLL NLL relaxed (Platt) single pass MACE-MP-0 MPtrj 0.913 ± 0.001 0.247 ± 0.001 0.951 ± 0.001 0.191 ± 0.003 0.172 ± 0.002 182 9,193 CHGNet MPtrj 0.895 ± 0.002 0.266 ± 0.001 0.947 ± 0.002 0.196 ± 0.003 0.180 ± 0.003 3,021 8,432 MACE-MPA-0 MPtrj, sAlex 0.975 ± 0.000 0.129 ± 0.001 0.974 ± 0.000 0.138 ± 0.001 0.104 ± 0.001 14,123 13,650 MatterSim-v1-5M MatterSim 0.977 ± 0.001 0.121 ± 0.002 0.977 ± 0.000 0.130 ± 0.001 0.096 ± 0.000 14,934 14,085 ORB-v3 MPtrj, Alexandria, OMat24 0.984 ± 0.001 0.094 ± 0.002 0.980 ± 0.000 0.117 ± 0.001 0.071 ± 0.001 16,618 15,070 SevenNet-MF-ompa MPtrj, OMat24, sAlex 0.984 ± 0.000 0.096 ± 0.002 0.976 ± 0.000 0.129 ± 0.002 0.078 ± 0.001 16,666 14,287 23 Supplementary Information This Supplementary Information accompanies the article. Supplementary Notes 1–7 give the full analyses summarised in the main text; Supplementary Tables report the quantities behind the figures. Figures and Extended Data figures are cited by their numbers in the article. Supplementary Note 1: Auditing the Matbench Discovery leaderboard Any model on the Matbench Discovery leaderboard can be read as a System-1 scorer because the MP hull used by the benchmark is fixed and contains no WBM entry: a predicted formation energy ˆe form determines the predicted hull distance ˆ E hull =ˆe form −e hull form (x), wheree hull form (x)is the formation energy of the MP hull at the candidate’s composition (Methods, Eq. 3). We converted the published WBM predictions of all 65 models (25 interatomic potentials trained only on MP-derived data, hereafterMP-trained; 30 potentials trained with larger external sets such as OMat24, sAlex or Alexandria; and 10 models that are not interatomic potentials) into probabilities with one-dimensional maps fitted on a random 5% slice of the candidate pool, reproducing the published F1 and precision of every model to within 0.0007 (Methods). The threshold is not where the probability is one half.Without any calibration map, the fraction of DFT-stable structures among candidates predicted within±5 meV/atom of the hull is 0.19–0.46 for MP- trained potentials, 0.12–0.30 for non-potential models and 0.37–0.57 for potentials trained on larger sets (Fig. 2a,b). For MP-trained potentials the hard rule ˆ E hull ≤0 draws its boundary through a population that is mostly unstable. Much of this is a property of the population: there are 1.6 times as many WBM structures in the first 25 meV/atom above the hull as in the 25 meV/atom below it, so errors carry more unstable structures across the threshold than stable ones, and even anunbiasedpredictor with Gaussian errors of 30 or 100 meV/atom would see stable fractions of only 0.34 and 0.24 there. The isotonic map places the 50% point between−54 and−0.3 meV/atom for MP-trained potentials and within 2.2 meV/atom of zero for potentials trained on larger sets. A two-number account.If the true hull distances of the pool have a local density proportional toe κE near the hull and a model’s errors there are Gaussian with biasband spreads, thenP(stable| ˆ E hull =t) = Φ (b−κs 2 −t)/s (Methods, Eq. 4). Fittingκ=14.5 eV −1 once to the WBM population and takingband sfrom each model’s errors near the hull predicts the calibrated probability at the threshold of all 65 models with Pearsonr=0.96 (Extended Data Fig. 1a), and, applied to each WBM substitution round separately (κ from 11.5 to 16.7 eV −1 ), withr=0.94 over all model–round pairs. Where a model’s threshold should sit is a joint property of its errors and of the candidate population, not of either alone. A threshold is a cost assumption.Read as a decision rule, ˆ E hull ≤0 maximises value only if a failed attempt costsc ∗ =0.18–0.48 of a hit for MP-trained potentials (median 0.33; Extended Data Fig. 1b). At any other cost the calibrated rulep≥cis worth more: when attempts are expensive (c=0.7, as for a synthesis) it gains a median of 12.8 hits per 1,000 candidates for MP-trained potentials and 85.6 for non- potential models, whose hard calls then lose value outright. The calibrated rule also forecasts what it will deliver: the expected number of stable structures in the leaderboard shortlist is forecast to within 0.3% (median over models), whereas the hard rule implicitly promises that every shortlisted candidate is stable and delivers 24%–93% of that promise. A few hundred labels suffice: an isotonic map fitted on 200 random labels from the pool comes within 0.004 in Brier score of the best monotone map in hindsight for the median MP-trained potential (Extended Data Fig. 1c), and 100 labels already beat the hard rule atc=0.1 for 25 of 25 MP-trained potentials. F1 hides how far confident calls can be trusted.Ranking models by the log-loss of their calibrated probabilities reproduces the F1 ranking closely (Kendallτ=0.98), because both are dominated by the size of each model’s errors. The number of candidates that can be certified at a 10% false-discovery rate by conformal selection 32 is less well summarised (Fig. 2d): between MP-trained potentials whose F1 differs 24 by less than 0.01 it differs up to 7.8-fold, because what matters is how deep each model’s ranking stays clean (Fig. 2f) — only 86% of MACE-MP-0’s 1,000 most confident “stable” calls are stable, against 99% for CHGNet. Supplementary Table 1|Calibration leaderboard of all audited Matbench Discovery models (WBM unique proto- types; 1-D maps fitted on a random 5% slice, means over 5 seeds; conformal selection averaged over 20 draws). Sorted by isotonic NLL. Data: MP = potential trained only on MP-derived data (MP-trained); ext. = potential trained with larger external sets; non-pot. = not an interatomic potential.p ±5 : DFT-stable fraction among candidates predicted within±5 meV/atom of the hull. ECE σ : ECE of the error-bar mapΦ(− ˆ E hull /σ).W: width of the 0.2≤p≤0.8 band of the isotonic map (– where the map does not span the band).N 0.1 : candidates certified at FDR≤0.1. FDR shift : realised FDR when calibrating on WBM rounds 1–2 and selecting from rounds 4–5. modeldataF1MAE NLL iso ECE iso ECE σ p ±5 WN 0.1 FDR shift (meV)(meV) GRACE-3L-OAM-Lext.0.925180.0740.002 0.101 0.497 33,1430.090 EquiformerV3+DeNS-OAMext.0.931180.0750.002 0.104 0.486 33,4880.085 TECE-OAM-RRA-1.0ext.0.929180.0760.002 0.103 0.497 33,1740.087 EquFlashV2ext.0.929180.0760.002 0.101 0.486 33,1590.088 TACE-OAM-RRA-Previewext.0.928180.0770.002 0.105 0.497 33,1320.085 Prophet-OAME-MBDext.0.928180.0780.002 0.103 0.496 33,0660.090 eSEN-30M-OAMext.0.925180.0780.002 0.105 0.517 33,0310.088 PET-OAM-XLext.0.924190.0790.002 0.106 0.517 32,9540.086 MatRIS-10M-OAMext.0.921190.0820.002 0.103 0.498 32,5700.090 EquFlashext.0.918190.0830.002 0.101 0.487 32,7200.093 eqV2 Mext.0.917200.0850.002 0.104 0.498 32,4680.089 TACE-OAM-Lext.0.910200.0890.002 0.103 0.459 32,2200.096 Nequip-OAM-XLext.0.906200.0920.003 0.102 0.4510 31,9560.096 ORB v3ext.0.905240.0920.002 0.119 0.4810 31,8190.085 SevenNet-Omni-i12ext.0.906210.0920.002 0.102 0.4911 31,8630.091 SevenNet-MF-ompaext.0.901210.0960.002 0.101 0.4312 31,4420.087 AlphaNet-v1-OAMext.0.901240.0960.002 0.118 0.4011 31,8440.094 Allegro-OAM-Lext.0.895220.0970.002 0.103 0.3712 31,2890.086 Nequip-OAM-Lext.0.893220.1000.002 0.101 0.4714 30,8330.098 DPA3-v2-OpenLAMext.0.890220.1010.002 0.099 0.4314 30,7480.091 TACE-v1-OAM-Mext.0.889220.1020.002 0.101 0.4314 30,5810.094 DPA-3.1-3M-FText.0.884230.1040.003 0.100 0.3914 30,2950.094 DPA3-v1-OpenLAMext.0.883230.1060.003 0.098 0.4515 30,0400.091 GRACE-2L-OAM-Lext.0.883220.1070.002 0.099 0.5114 30,2700.094 GRACE-2L-OAMext.0.880230.1070.002 0.101 0.4713 29,9420.095 ORB v2 MPAext.0.880280.1140.003 0.101 0.5713 30,3710.072 MatterSim v1 5Mext.0.862240.1190.003 0.094 0.5319 29,0220.087 EquiformerV3+DeNS-MPMP0.863290.1270.004 0.114 0.3617 28,2520.118 MACE-MPA-0ext.0.852280.1280.003 0.102 0.4519 27,4830.089 DPA-4.0.1-Pro-MPtrjMP0.857290.1360.003 0.111 0.4016 28,0700.108 DPA-4.0-Pro-MPtrjMP0.850300.1380.003 0.114 0.4618 27,4770.104 MatRIS-10M-MPMP0.847310.1400.004 0.117 0.3719 26,8120.122 GRACE-1L-OAMext.0.824310.1480.004 0.101 0.4327 18,8860.073 eSEN-30M-MPMP0.831330.1500.003 0.115 0.3623 24,7580.121 NequIP-GNoMEext.0.829350.1510.003 0.118 0.4619 25,6850.107 eqV2 S DeNSMP0.815360.1590.003 0.125 0.3129 23,6990.124 MatRIS v0.5.0 MPtrjMP0.809370.1620.004 0.122 0.3329 22,8190.122 AlphaNet-v1-MPtrjMP0.799410.1640.004 0.138 0.3333 17,7560.119 DPA-3.1-MPtrjMP0.803370.1680.003 0.120 0.3329 22,0130.105 DPA3-v2-MPtrjMP0.786390.1770.005 0.124 0.3132 20,0970.109 Eqnorm MPtrjMP0.787400.1810.004 0.122 0.3237 18,3010.124 HIENetMP0.776410.1890.005 0.117 0.3640 17,4050.108 DPA3-v1-MPtrjMP0.765420.1900.004 0.122 0.3143 17,1750.119 ORB v2 MPtrjMP0.765450.1920.003 0.130 0.3239 16,6060.109 Nequip-MP-LMP0.761430.1970.005 0.117 0.3445 13,6770.089 SevenNet-l3i5MP0.760440.1990.004 0.121 0.3454 11,6360.143 Nequix MPMP0.751440.2000.005 0.127 0.2956 11,2400.120 Nequix MP PFTMP0.748440.2020.005 0.121 0.3151 11,4730.118 Allegro-MP-LMP0.751440.2040.005 0.120 0.33601,4840.036 25 modeldataF1MAE NLL iso ECE iso ECE σ p ±5 WN 0.1 FDR shift SevenNet-0MP0.724480.2150.004 0.131 0.28649,3570.136 GRACE-2L-MPtrjMP0.691520.2410.005 0.113 0.311013,8510.087 MACE-MP-0MP0.669570.2470.006 0.144 0.271461,3170.104 CHGNetMP0.612630.2670.006 0.159 0.23986,4610.183 BAM-MP-coreMP0.623600.2760.006 0.125 0.301302,7650.106 M3GNetMP0.569750.2890.006 0.199 0.191451,9020.085 EMA-GNNnon-pot. 0.557860.2930.007 0.205 0.152381,5590.058 ESNetnon-pot. 0.5711090.3080.006 0.137 0.281442,8970.068 ALIGNNnon-pot. 0.567930.3150.007 0.163 0.265662050.116 CGCNN+Pnon-pot. 0.5001130.3380.008 0.214 0.195782320.120 MEGNetnon-pot. 0.5101300.3390.007 0.157 0.304972300.013 CGCNNnon-pot. 0.5071380.3440.008 0.165 0.235241930.112 Wrenformernon-pot. 0.4661100.3540.007 0.250 0.175152300.193 BOWSRnon-pot. 0.4231180.3730.007 0.254 0.12–410.000 AlchemBERTnon-pot. 0.4211170.3760.007 0.258 0.18–1400.137 Voronoi RFnon-pot. 0.3331480.4110.008 0.269 0.17–490.009 Supplementary Note 2: One forward pass versus a relaxation The audit calibrates what a model computes after a relaxation. CrystalJev instead decides from a single forward pass on the unrelaxed input. On the WBM unique prototypes we trained heads on 45% of the chemical systems, calibrated them on a random 5% slice of the remaining structures and evaluated them on the rest, so that calibration and test are exchangeable but chemically disjoint from training (Methods); the relax-and-threshold practice of the same backbone — its published relaxed prediction with an isotonic map on the same slice — is the reference. Six backbones.For the two MP-trained potentials that underlie the original leaderboard, the single pass decides better than the relaxation (Extended Data Table 1; Fig. 3a): log-loss 0.191 against 0.247 for MACE- MP-0 and 0.196 against 0.266 for CHGNet, with higher AUROC, at one forward pass instead of a relaxation of about thirty (median per-structure cost ratio 29 and 30). For four newer potentials trained on larger data (MACE-MPA-0 (MPtrj and sAlex), MatterSim-v1-5M (MatterSim’s own data), ORB-v3 (MPtrj, Alexan- dria and OMat24) and SevenNet-MF-ompa (MPtrj, OMat24 and sAlex)), whose relaxed predictions are much more accurate (isotonic log-loss 0.094–0.129), the single pass comes within 0.009–0.033 in log-loss of the far more accurate relaxed predictions of four newer backbones, and, given the relaxed prediction as one more input, improves on it for every one of the six backbones studied. The embeddings carry informa- tion beyond the energies: a boosted head on the ten physics scalars and the relaxed prediction alone reaches 0.214 (MACE-MP-0), against 0.172 with embeddings. Fusing two backbones helps further (log-loss 0.161 for MACE-MP-0 with CHGNet). Deciding directly beats propagating energy uncertainty.The usual route to a probability of stability is to predict the energy with an uncertainty and propagate it through the threshold. On the same single- pass features, labels and slice (Extended Data Fig. 2a; Supplementary Table 2), a heteroscedastic Gaussian regression ofE hull overstates the stable fraction by 2.2 percentage points (MACE-MP-0) and, even after isotonic recalibration, reaches log-loss 0.211 against 0.192 for the head trained on the question; quantile regression behaves alike. The regression models describe the whole distribution ofE hull better (lower continuous ranked probability score) but spend that accuracy away from the threshold at which the decision is taken. An ensemble of the 25 MP-trained leaderboard potentials, a strong but expensive baseline, reaches 0.135 with a boosted map on the mean and spread of their relaxed predictions, about what its best single member reaches with an isotonic map (EquiformerV3+DeNS-MP, 0.128); the single pass on MACE-MP- 0 matches the median member after relaxation. A single threshold-conditioned head can answer every question of the familyE hull ≤t, but only at thresholds seen in training unless it is recalibrated at the queried one (Extended Data Fig. 2b). 26 Energy-level baselines and the threshold-conditioned head.All baselines of Supplementary Note 2 use the same features, training subsample, calibration slice and gradient-boosting settings as the decision heads. The heteroscedastic Gaussian regression fits a mean model ofE hull , a model of the logarithm of five-fold cross-fitted absolute residuals, and one variance scalekthat minimises the Gaussian negative log-likelihood on the calibration slice;p=Φ(−μ/k ˆ σ). Quantile regression fits seven quantile models (5– 95%), sorts them per candidate and readsp= ˆ F(0)by linear interpolation. The threshold-conditioned head is one gradient-boosted classifier on the features and the thresholdt, trained on all (structure, threshold) pairs fort∈−50,0,50,100,200meV/atom with a monotone constraint int, followed by one Platt map shared across thresholds; its CDF is made monotone by a running maximum. The CRPS is integrated over −0.25≤t≤0.5 eV/atom. The leaderboard ensemble uses the published relaxed predictions of the 25 MP- trained potentials; the boosted map is a shallow gradient-boosted classifier on the mean and the logarithm of the standard deviation, fitted on the calibration slice. The decision layer on MACE-MP-0 and CHGNet.Extended Data Fig. 3 and Supplementary Table 3 detail the regimes compared in the main text for the two MP-trained backbones: zero-shot transfer of a head trained on MP relaxed structures, in-domain post-training, and relax-and-threshold, with the label efficiency of in-domain training and the cost of a relaxation in single passes. Supplementary Table 6 reports the ordinal, choice and prototype questions for these backbones and Supplementary Table 7 for all six. 27 Supplementary Table 2 | On-hull question on the WBM test half: decision heads against energy-level uncertainty propagated through the threshold, the threshold-conditioned head, and ensembles of published relaxed predictions (means ± s.d. over three seeds; same splits, training subsample and calibration slice for every method). ECE near : candidates with relaxed | ˆ E hull | < 50 meV/atom. N 0 . 1 and FDP: candidates certified by conformal selection at q = 0 . 1 and their realised false-discovery proportion. backbone decision rule cost AUROC NLL Brier ECE ECE near N 0 . 1 FDP MACE-MP-0 relaxed ˆ E hull , isotonic map relaxation 0.913 ± 0.001 0.248 ± 0.001 0.074 ± 0.000 0.006 ± 0.003 0.016 ± 0.001 154 0.070 ± 0.042 energy UQ: heteroscedastic Gaussian, Φ ( − μ / σ ) 1 pass 0.942 ± 0.000 0.223 ± 0.006 0.066 ± 0.001 0.040 ± 0.011 0.051 ± 0.015 7,520 0.103 ± 0.018 + isotonic map 1 pass 0.941 ± 0.000 0.211 ± 0.001 0.064 ± 0.001 0.007 ± 0.002 0.018 ± 0.005 6,841 0.089 ± 0.033 energy UQ: quantile regression, ˆ F ( 0 ) 1 pass 0.924 ± 0.001 0.318 ± 0.006 0.062 ± 0.001 0.026 ± 0.002 0.044 ± 0.004 8,161 0.103 ± 0.018 + isotonic map 1 pass 0.923 ± 0.002 0.214 ± 0.003 0.062 ± 0.001 0.005 ± 0.001 0.022 ± 0.001 7,546 0.092 ± 0.019 one head for all thresholds, shared Platt map 1 pass 0.948 ± 0.001 0.198 ± 0.002 0.060 ± 0.001 0.010 ± 0.003 – 9,033 0.108 ± 0.012 + Platt map at the queried threshold 1 pass 0.948 ± 0.001 0.197 ± 0.002 0.059 ± 0.001 0.004 ± 0.001 – – – decision head for this question 1 pass 0.950 ± 0.001 0.192 ± 0.002 0.057 ± 0.001 0.004 ± 0.001 0.011 ± 0.001 9,143 0.103 ± 0.006 CHGNet relaxed ˆ E hull , isotonic map relaxation 0.895 ± 0.002 0.266 ± 0.001 0.080 ± 0.000 0.005 ± 0.002 0.010 ± 0.004 2,967 0.087 ± 0.039 energy UQ: heteroscedastic Gaussian, Φ ( − μ / σ ) 1 pass 0.935 ± 0.000 0.235 ± 0.009 0.070 ± 0.002 0.044 ± 0.014 0.055 ± 0.018 6,571 0.094 ± 0.011 + isotonic map 1 pass 0.935 ± 0.000 0.220 ± 0.002 0.067 ± 0.001 0.006 ± 0.002 0.013 ± 0.002 6,293 0.088 ± 0.015 energy UQ: quantile regression, ˆ F ( 0 ) 1 pass 0.919 ± 0.001 0.324 ± 0.002 0.065 ± 0.001 0.025 ± 0.002 0.038 ± 0.003 7,639 0.103 ± 0.014 + isotonic map 1 pass 0.918 ± 0.000 0.222 ± 0.002 0.064 ± 0.001 0.005 ± 0.001 0.019 ± 0.004 7,616 0.103 ± 0.014 one head for all thresholds, shared Platt map 1 pass 0.941 ± 0.001 0.208 ± 0.002 0.063 ± 0.001 0.011 ± 0.002 – 7,404 0.091 ± 0.008 + Platt map at the queried threshold 1 pass 0.941 ± 0.001 0.207 ± 0.002 0.063 ± 0.001 0.004 ± 0.001 – – – decision head for this question 1 pass 0.946 ± 0.002 0.198 ± 0.003 0.059 ± 0.001 0.004 ± 0.002 0.008 ± 0.002 7,911 0.085 ± 0.007 25 MP-trained error bar: Φ ( − μ / q σ 2 ens + RMSE 2 ) 25 relax. 0.962 ± 0.001 0.260 ± 0.005 0.074 ± 0.001 0.124 ± 0.005 0.247 ± 0.005 12,200 0.099 ± 0.011 potentials mean μ , isotonic map 25 relax. 0.955 ± 0.001 0.161 ± 0.002 0.044 ± 0.000 0.004 ± 0.002 0.010 ± 0.004 11,714 0.094 ± 0.007 mean and log σ ens , logistic map 25 relax. 0.956 ± 0.001 0.197 ± 0.002 0.050 ± 0.000 0.041 ± 0.003 0.103 ± 0.007 11,075 0.096 ± 0.008 mean and log σ ens , boosted map 25 relax. 0.977 ± 0.000 0.135 ± 0.001 0.040 ± 0.000 0.004 ± 0.001 0.010 ± 0.002 13,662 0.109 ± 0.016 best member (EquiformerV3+DeNS-MP), isotonic relaxation 0.970 ± 0.000 0.128 ± 0.002 0.033 ± 0.000 0.003 ± 0.001 – 14,527 0.096 ± 0.013 median member (DPA3-v1-MPtrj), isotonic relaxation 0.943 ± 0.001 0.191 ± 0.001 0.054 ± 0.001 0.004 ± 0.000 – 7,803 0.088 ± 0.036 28 Supplementary Table 3 | On-hull decisions on the WBM test half (means ± s.d. over seeds). ECE near : ECE among candidates with | ˆ E hull | < 50 meV/atom (relaxed prediction). N 0 . 1 : candidates certified by conformal selection at FDR ≤ 0 . 1 (not computed for the isotonic map). Unless stated otherwise, probabilities are calibrated on the same random 5% WBM slice. Cost: one forward pass of the backbone, or a full relaxation (about 30 passes). backbone decision rule cost AUROC NLL Brier ECE ECE near N 0 . 1 MACE-MP-0 relax + threshold (relaxed ˆ E hull , Platt) relaxation 0.912 ± 0.001 0.278 ± 0.001 0.079 ± 0.000 0.043 ± 0.003 0.086 ± 0.003 182 relax + threshold (relaxed ˆ E hull , isotonic) relaxation 0.913 ± 0.001 0.247 ± 0.001 0.074 ± 0.000 0.006 ± 0.003 0.015 ± 0.002 – single-point ˆ E hull , Platt 1 pass 0.848 ± 0.001 0.329 ± 0.002 0.095 ± 0.001 0.054 ± 0.004 0.079 ± 0.002 933 head trained on MP, calibrated on MP 1 pass 0.838 ± 0.003 0.403 ± 0.011 0.101 ± 0.001 0.059 ± 0.003 0.139 ± 0.002 0 head trained on MP, calibrated on WBM slice 1 pass 0.838 ± 0.003 0.319 ± 0.004 0.096 ± 0.001 0.010 ± 0.001 0.076 ± 0.003 0 in-domain head, physics scalars only 1 pass 0.921 ± 0.001 0.233 ± 0.002 0.069 ± 0.001 0.004 ± 0.001 0.013 ± 0.004 5,994 in-domain linear probe, embeddings + scalars 1 pass 0.895 ± 0.000 0.277 ± 0.001 0.082 ± 0.001 0.020 ± 0.003 0.019 ± 0.001 2,333 in-domain head, embeddings + scalars 1 pass 0.951 ± 0.001 0.191 ± 0.003 0.057 ± 0.001 0.004 ± 0.001 0.011 ± 0.001 9,193 in-domain head, scalars + relaxed ˆ E hull relaxation 0.937 ± 0.001 0.214 ± 0.002 0.064 ± 0.001 0.004 ± 0.001 0.006 ± 0.003 6,641 in-domain head, embeddings + scalars + relaxed relaxation 0.962 ± 0.001 0.172 ± 0.002 0.052 ± 0.001 0.004 ± 0.001 0.007 ± 0.002 10,215 CHGNet relax + threshold (relaxed ˆ E hull , Platt) relaxation 0.895 ± 0.002 0.284 ± 0.001 0.081 ± 0.000 0.030 ± 0.004 0.043 ± 0.004 3,021 relax + threshold (relaxed ˆ E hull , isotonic) relaxation 0.895 ± 0.002 0.266 ± 0.001 0.080 ± 0.000 0.005 ± 0.002 0.011 ± 0.004 – single-point ˆ E hull , Platt 1 pass 0.839 ± 0.000 0.336 ± 0.002 0.097 ± 0.001 0.053 ± 0.003 0.069 ± 0.002 2,267 head trained on MP, calibrated on MP 1 pass 0.821 ± 0.001 0.418 ± 0.006 0.110 ± 0.001 0.066 ± 0.002 0.147 ± 0.002 0 head trained on MP, calibrated on WBM slice 1 pass 0.821 ± 0.001 0.337 ± 0.002 0.103 ± 0.001 0.015 ± 0.001 0.068 ± 0.003 0 in-domain head, physics scalars only 1 pass 0.903 ± 0.000 0.255 ± 0.002 0.076 ± 0.001 0.004 ± 0.001 0.013 ± 0.001 4,418 in-domain linear probe, embeddings + scalars 1 pass 0.880 ± 0.000 0.292 ± 0.001 0.086 ± 0.000 0.023 ± 0.002 0.019 ± 0.002 2,332 in-domain head, embeddings + scalars 1 pass 0.947 ± 0.002 0.196 ± 0.003 0.058 ± 0.001 0.004 ± 0.002 0.008 ± 0.003 8,432 in-domain head, scalars + relaxed ˆ E hull relaxation 0.919 ± 0.000 0.238 ± 0.002 0.071 ± 0.001 0.004 ± 0.001 0.007 ± 0.003 4,748 in-domain head, embeddings + scalars + relaxed relaxation 0.957 ± 0.001 0.180 ± 0.003 0.054 ± 0.001 0.003 ± 0.001 0.011 ± 0.001 9,537 both fusion: embeddings, scalars, relaxed ˆ E hull of both, disagreement relaxation 0.967 ± 0.000 0.161 ± 0.001 0.048 ± 0.000 0.004 ± 0.001 0.008 ± 0.003 10,769 29 Supplementary Note 3: Value of information, escalation and DFT allocation One number per potential.If System 1’s predictive distribution of the hull distance is Gaussian and System 2 observes the hull distance with independent Gaussian noise, the value of information has a closed form that depends onp 1 ,cand a singleinformation ratio r, the precision of System 2 relative to the uncer- tainty of System 1 (Methods, Proposition 2; verified by Monte Carlo). Measuring the value of escalation from System 1 (the single-pass head) to System 2 (the head with the backbone’s relaxed prediction) on the test half, withp 2 calibrated conditionally onp 1 on the calibration slice (Methods), the one-parameter form describes the measured curves of all six potentials (Fig. 4b;R 2 0.81–0.98 across costs), withrfrom 0.23 for CHGNet and 0.27 for MACE-MP-0 to 1.2 for ORB-v3 and 1.8 for SevenNet-MF-ompa. Newer potentials thus make System 2 relatively more informative, consistent with their relaxed predictions improving faster than their single passes. In information terms, System 1 captures 55%–73% of the 0.43 nats of label entropy and relaxation adds 0.018–0.052 nats; 46%–77% of that addition comes from the 20% of candidates in the two deciles ofp 1 below the most confident one (medianp 1 0.08 and 0.39), and only 2%–12% from the most confident decile. Escalate by value, not by probability.Under a budget of relaxations, the expected net value is max- imised by relaxing the candidates with the largest value of information (Methods). Estimated on the cal- ibration slice — which needs System-2 outputs there, but no labels beyond those that calibratep 2 — this order recovers 78% of the gain of relaxing every candidate when 10% of candidates are relaxed and 93% at 20% (means over six potentials and three seeds atc=0.5; Fig. 4c). The order a screening funnel uses — the most probable candidates first — recovers 1% and 78% at the same budgets, because the candidates it relaxes first would be attempted anyway; random selection recovers 10%. Ranking by distance from the decision boundary does almost as well as the value of information whenc=0.5 (75% at 10%), but not when costs are asymmetric (66% against 84% atc=0.7). Where a guarantee is preferred to an optimum, the conformal screen of Fig. 3b discards only candidates below a threshold set on the stable structures of the slice: at a 1% recall budget it relaxes 41%–58% of the candidates, keeps at least 0.988 of the stable ones and retains at least 99.8% of the decision value atc=0.5 (98%–100% of the gain that relaxation adds), saving 41%–57% of the cost of relaxing every candidate. When to run DFT.The bound of Eq. 2 is attained by DFT, which reveals the label. If an attempt (a synthesis, say) costscand a DFT check costsk, a calibrated probability should be discarded belowk/(1−c), attempted directly above 1−k/cand checked by DFT in between (Methods). On the WBM test half with c=0.5 andk=0.05 (Fig. 4d), this band applied to the single-pass probability realises 58–65 net hits per 1,000 candidates for the six potentials and applied to the probability after relaxation 60–70, sending 7%– 23% of the candidates to DFT and recovering 93%–97% of the stable ones. The usual practice — DFT for every candidate that the relaxed prediction calls stable — realises 47–62 while sending 14%–23% of the candidates to DFT, and computing every candidate realises 27. The single pass alone beats the threshold practice for six of the six potentials. Estimating the value of information.Supplementary Table 4 reports the share of the gain of relaxing every candidate retained by each escalation order at four attempt costs, and Supplementary Table 5 the realised value of allocating DFT with the three-way band. Two practical points matter for the estimates. First, the martingale propertyE[p 2 |p 1 ] =p 1 that makes the value of information non-negative holds only ifp 2 is calibrated conditionally onp 1 . With separately Platt-calibratedp 1 andp 2 ,p 2 was slightly under- confident relative top 1 among candidates withp 1 between 0.6 and 0.8 (CHGNet, first seed: meanp 2 0.62 and 0.71 in the two bins against realised stable fractions of 0.64 and 0.76), which made the measured value of escalation negative there; recalibratingp 2 by a logistic regression on(logitp 1 ,logitp 2 )on the calibration slice removes this and also lowers its log-loss. Second, ranking candidates by the value of information requires System-2 outputs on the calibration slice only, which is affordable because the slice is small and must be labelled anyway. 30 Supplementary Table 4|Budgeted escalation from the single pass (System 1) to the head with the relaxed prediction (System 2): share of the realised gain of relaxing every candidate that is retained when a fractionbof the candidates is relaxed, by escalation order and attempt costc(WBM test half; means over six backbones and three seeds;p 2 calibrated conditionally onp 1 ). Last column: the conformal screen at a 1% recall budget (share relaxed; share retained). corderb=5%10%20%30%screen (α=1%) 0.1value of information32%54%78%90%48%; 92% closest toc33%55%77%87% most probable first0%0%10%59% random5%10%20%30% 0.3value of information47%70%89%95%48%; 97% closest toc47%70%89%95% most probable first0%0%57%87% random4%10%20%30% 0.5value of information55%78%93%97%48%; 99% closest toc53%75%92%97% most probable first0%1%78%94% random4%10%20%30% 0.7value of information63%84%96%99%48%; 100% closest toc56%66%89%98% most probable first0%20%89%98% random4%10%20%30% Supplementary Table 5|Allocating DFT with calibrated probabilities (WBM test half, means over three seeds): realised net value per 1,000 candidates when an attempt costscand a DFT checkk(units of a hit), for computing every candidate, computing those whose relaxed hull distance is≤0, and the three-way band (discard / DFT / attempt directly) on the single-pass probabilityp 1 and on the probability after relaxationp 2 ; in brackets the share of candidates sent to DFT.r: information ratio of the one-parameter fit to the measured value of escalation (c=0.5). c=0.5,k=0.05c=0.5,k=0.01c=0.7,k=0.05 backbonerthresholdbandp 1 bandp 2 thresholdbandp 2 thresholdbandp 2 MACE-MP-00.2751 (21%)58 (24%)61 (22%)59 (21%)72 (40%)26 (21%)32 (18%) CHGNet0.2347 (23%)58 (25%)60 (23%)56 (23%)72 (43%)24 (23%)31 (19%) MACE-MPA-00.5258 (15%)64 (15%)67 (11%)64 (15%)74 (23%)32 (15%)37 (10%) MatterSim-v1-5M0.6357 (14%)64 (14%)68 (10%)63 (14%)74 (21%)31 (14%)38 (9%) ORB-v31.262 (15%)65 (12%)70 (7%)68 (15%)75 (15%)34 (15%)40 (7%) SevenNet-MF-ompa1.862 (16%)64 (14%)70 (8%)69 (16%)74 (17%)34 (16%)39 (7%) DFT every candidate27 (100%)67 (100%)-4 (100%) Supplementary Note 4: Typed questions beyond stability and beyond crystals Ordinal, prototype and choice questions.A System-1 layer is useful in proportion to the questions it can answer from the same state. On WBM, the single pass answers the five-level hull-band Score question better than an ordinal model on the relaxed prediction for six of the six backbones, predicts whether DFT relaxation will preserve the input prototype — whether the input geometry can be trusted at all — with AU- ROC 0.86 (MACE-MP-0), and chooses the lowest-energy polymorph among candidates of one composi- tion, the one question on which relaxation clearly wins (0.67 against 0.73 top-1 accuracy for MACE-MP-0; Supplementary Tables 6 and 7). Electronic structure from a potential’s single pass.The backbones were trained on energies, forces and stresses (and, for CHGNet, magnetic moments), yet their single pass on the unrelaxed input carries information about the electronic structure of the relaxed crystal (Fig. 5a; Supplementary Table 8). Whether DFT will find a metal is answered with AUROC 0.977–0.989 across the six backbones (composition alone: 31 0.951), whether the gap falls in the 1–2 eV window of single-junction photovoltaic absorbers (3.4% of candidates) with AUROC 0.971–0.984 (composition: 0.951), and the five-level band-gap Score with a ranked probability score of 0.013–0.019 (composition alone 0.032). Compound questions should be calibrated as compounds.Discovery campaigns ask conjunctions: sta- bleandin the photovoltaic window, stableandnon-metallic. Multiplying two calibrated marginal answers assumes the two events are conditionally independent given the state, and they are not (Fig. 5b): the product of the marginals under-forecasts the number of stable non-metals by 3%–13% and of stable photovoltaic absorbers by up to 14% (most for the MP-trained backbones) — the signature of events that are positively associated given what the layer knows (Methods) — whereas a one-dimensional recalibration of the product on the slice (ECE 0.001 for every backbone, against 0.001–0.005 for “stable and non-metallic”) or a head trained on the compound question itself (0.001–0.002) is calibrated. The recalibrated product has the low- est log-loss for both compound questions (0.018–0.024 for “stable and photovoltaic” against 0.023–0.029 for the dedicated head), because the marginal heads are each trained on more positives. A typed interface makes the compound explicit, so its calibration can be checked rather than assumed. Beyond the hull.We posed 12 screening questions on seven Matbench property datasets 35 — metallicity, a photovoltaic window and a wide gap from PBE band gaps; stiff, soft, hard and compliant crystals from elastic moduli; high refractive index; soft and stiff lattices from the highest phonon peak; easy exfoliation of layered crystals; and low formation energy of perovskites — and answered them from one MatterSim or ORB-v3 pass with five-fold cross-validation (Fig. 5c,d; Methods). Deciding directly gives a lower log-loss than both regress-then-threshold baselines on the same features (a regression with a calibrated threshold, and a regression with its residual distribution propagated through the threshold) for 20 of the 24 question– backbone pairs, by more than 0.005 nats for 16 of them; the better baseline’s log-loss is 1.15 times the decision head’s (median). Regression does better for a low perovskite formation energy, a smooth target on which it uses the full label, and in log-loss for stiff phonon spectra, where a few confident errors in a small dataset dominate the decision head’s log-loss while its Brier score is lower (0.022–0.024 against 0.030–0.031). The largest differences arise where the question sits at an edge of the property’s distribution or where the target is heavy-tailed — where the noise of a regression changes shape from one input to another, which no monotone map of its output can absorb (Methods): whether a material is a metal (a gap of exactly zero) is answered with log-loss 0.162–0.190 against 0.262–0.295 for the calibrated regression, whose propagated-residual variant is badly miscalibrated (ECE 0.22 against 0.007), and a high refractive index with 0.163 against 0.30–0.34 for the regressions and 0.25 for composition alone. Conformal selection certifies 766–1,089 candidates per test fold for the photovoltaic window at a nominal 10% false-discovery rate from the decision head (realised 0.11–0.12), against at most 31 from either regression. Composition alone reaches AUROC 0.70–0.98 against 0.91–0.997 for the single pass; exfoliability, a question about how layers bind, is where structure matters most (0.91–0.93 against 0.70). Beyond crystals.Nothing in the layer is specific to crystals: it needs the state of a frozen atomistic model, labels and a question. On 130,831 QM9 molecules 36 — whose geometries, unlike the crystal inputs, are already relaxed at the level of theory of the labels, so that these questions test reading the state rather than escalation — one pass of a molecular foundation potential (MACE-OFF23 37 ) or of the crystal-trained ORB-v3 on the molecule in a box answers whether the B3LYP HOMO–LUMO gap is wide (≥8 eV) or small (≤5 eV) and whether the dipole moment exceeds 4 D with AUROC 0.970–0.998 (composition alone: 0.766–0.875), and deciding directly beats both regress-then-threshold baselines in log-loss for all six question–backbone pairs (the better baseline’s log-loss is 1.08–1.39 times the decision head’s; Fig. 5e). A Choice question — which of five constitutional isomers has the lowest energyU 0 — shows how a typed layer builds on a backbone’s own physics rather than replacing it. The backbone’s energy alone picks the lowest-energy isomer in 90.2% (MACE-OFF23) and 85.0% (ORB-v3) of the test sets; neither model was trained at the level of theory of the labels. A Choice head — a softmax over the backbone energy plus a learned correction, with its temperature fitted on calibration sets — raises this to 98.0% and 96.8% and 32 states how sure it is (log-loss 0.05 and 0.08 against 1.61 for a uniform guess; Fig. 5f). That the crystal- trained potential, whose training data contain the same elements but no molecules, serves almost as well as the molecular one suggests that the layer does not need a backbone trained on the system class it is asked about. Datasets and thresholds.Matbench property questions 35 use the datasets’ structures as inputs (one MatterSim-v1-5M or ORB-v3 pass) and thresholds on their targets: PBE gap=0, 1≤gap≤2 eV, gap ≥3 eV (mp_gap, 106,113 structures);K VRH ≥150 or≤30 GPa andG VRH ≥100 or≤15 GPa (10,987); refractive index≥2.5 (4,764); highest phonon density-of-states peak≤300 or≥800 cm −1 (1,265); exfo- liation energy≤100 meV/atom (636); perovskite formation energy≤0.5 eV per cell (18,928). Five-fold cross-validation holds out 10% of each training fold as the calibration slice. 33 Supplementary Table 6 | Ordinal, choice and prototype questions for MACE-MP-0 and CHGNet (WBM test set, means ± s.d. over three seeds). Ordinal: five-level hull-distance band (ranked probability score, RPS; NLL; accuracy). Choice: lowest-energy polymorph among WBM candidates of one composition (top-1 accuracy; NLL of the true choice;top-choice ECE). Prototype kept: DFT relaxation keeps the input prototype (AUROC; NLL). question backbone decision rule metrics Ordinal MACE-MP-0 ordinal model on relaxed ˆ E hull RPS 0.108 ± 0.000 NLL 1.00 ± 0.00 acc. 0.60 ± 0.00 Ordinal MACE-MP-0 ordinal head (1 pass) RPS 0.077 ± 0.000 NLL 0.82 ± 0.00 acc. 0.69 ± 0.00 Ordinal MACE-MP-0 ordinal head + relaxed RPS 0.069 ± 0.000 NLL 0.75 ± 0.00 acc. 0.71 ± 0.00 Ordinal CHGNet ordinal model on relaxed ˆ E hull RPS 0.115 ± 0.000 NLL 1.04 ± 0.00 acc. 0.58 ± 0.00 Ordinal CHGNet ordinal head (1 pass) RPS 0.079 ± 0.000 NLL 0.85 ± 0.00 acc. 0.68 ± 0.00 Ordinal CHGNet ordinal head + relaxed RPS 0.072 ± 0.000 NLL 0.78 ± 0.00 acc. 0.70 ± 0.00 Choice MACE-MP-0 relaxed ˆ E hull , softmax acc. 0.727 ± 0.001 NLL 0.685 ± 0.003 ECE 0.096 ± 0.011 Choice MACE-MP-0 single-point ˆ E hull , softmax acc. 0.609 ± 0.001 NLL 0.826 ± 0.003 ECE 0.086 ± 0.001 Choice MACE-MP-0 single point + learned residual (1 pass) acc. 0.671 ± 0.003 NLL 0.716 ± 0.003 ECE 0.045 ± 0.007 Choice MACE-MP-0 relaxed + learned residual acc. 0.700 ± 0.001 NLL 0.671 ± 0.004 ECE 0.054 ± 0.008 Choice CHGNet relaxed ˆ E hull , softmax acc. 0.709 ± 0.003 NLL 0.700 ± 0.006 ECE 0.081 ± 0.011 Choice CHGNet single-point ˆ E hull , softmax acc. 0.602 ± 0.003 NLL 0.833 ± 0.002 ECE 0.086 ± 0.002 Choice CHGNet single point + learned residual (1 pass) acc. 0.661 ± 0.002 NLL 0.720 ± 0.002 ECE 0.033 ± 0.007 Choice CHGNet relaxed + learned residual acc. 0.694 ± 0.001 NLL 0.676 ± 0.004 ECE 0.039 ± 0.009 Prototype kept MACE-MP-0 logistic on force/stress magnitudes AUROC 0.615 ± 0.002 NLL 0.305 ± 0.001 Prototype kept MACE-MP-0 head on embeddings + scalars AUROC 0.862 ± 0.004 NLL 0.224 ± 0.001 Prototype kept CHGNet logistic on force/stress magnitudes AUROC 0.622 ± 0.001 NLL 0.304 ± 0.001 Prototype kept CHGNet head on embeddings + scalars AUROC 0.877 ± 0.004 NLL 0.214 ± 0.002 34 Supplementary Table 7 | Typed questions and the shift ladder across backbones (WBM test set, means over three seeds). Ordinal: ranked probability score of the five-level hull band (ordinal model on the relaxed prediction; single-pass ordinal head; head with the relaxed prediction). Choice: top-1 accuracy of the lowest-energy polymorph (relaxedenergies; single pass with a learned residual; relaxed energies with a learned residual). Prototype kept: AUROC of “relaxation keeps the prototype” (logistic model on force andstress magnitudes; head). Shift: boosted head trained on MP and evaluated on WBM rounds 2–5, calibrated on MP or on the WBM slice (coverage of 90% sets, near-hull ECE),and realised FDR of conformal selection at q = 0 . 1 with the linear MP head. Ordinal (RPS) Choice (top-1) Prototype kept (AUROC) coverage ECE near FDR backbone relaxed head head+rel. relaxed 1 pass rel.+res. | F | , σ head MP cal. slice MP cal. slice MP / slice MACE-MP-0 0.108 0.077 0.069 0.727 0.671 0.700 0.615 0.862 0.87 0.90 0.139 0.076 0.24 / 0.10 CHGNet 0.115 0.079 0.072 0.709 0.661 0.694 0.622 0.877 0.86 0.90 0.149 0.071 0.21 / 0.11 MACE-MPA-0 0.059 0.054 0.039 0.823 0.727 0.789 0.616 0.881 0.88 0.90 0.172 0.096 0.07 / 0.12 MatterSim-v1-5M 0.056 0.050 0.036 0.834 0.739 0.796 0.625 0.898 0.88 0.90 0.141 0.092 0.12 / 0.13 ORB-v3 0.051 0.045 0.027 0.858 0.750 0.820 0.617 0.895 0.88 0.90 0.178 0.089 0.06 / 0.11 SevenNet-MF-ompa 0.050 0.050 0.030 0.862 0.740 0.821 0.617 0.876 0.88 0.90 0.174 0.088 0.06 / 0.10 Supplementary Table 8 | Electronic-structure and compound questions on WBM from one forward pass on the unrelaxed input (test half, means over three seeds; same splits and calibration slice as the stability question). Metal: DFT band gap of the relaxed structure is zero; PV: 1 ≤ E g ≤ 2 eV; gap band: five levels (RPS). Single pass: embeddings and physics scalars; comp.: element fractions only. Compound questions (stable and non-metallic; stable and PV): ECE of the product of the two calibrated marginal answers, of thatproduct recalibrated on the slice, and of a head trained on the compound question; NLL of the recalibrated product and of the dedicated head. Metal (AUROC) PV (AUROC) Gap band (RPS) Stable & non-metal (ECE) Stable & PV (NLL) backbone single pass comp. single pass comp. single pass comp. product recal. dedicated recal. dedicated MACE-MP-0 0.977 0.951 0.972 0.951 0.0192 0.0323 0.0050 0.0007 0.0013 0.0245 0.0287 CHGNet 0.977 0.951 0.974 0.951 0.0190 0.0323 0.0049 0.0008 0.0019 0.0244 0.0277 MACE-MPA-0 0.983 0.951 0.978 0.951 0.0160 0.0323 0.0023 0.0007 0.0010 0.0213 0.0259 MatterSim-v1-5M 0.984 0.951 0.979 0.951 0.0154 0.0323 0.0021 0.0006 0.0011 0.0205 0.0257 ORB-v3 0.989 0.951 0.984 0.951 0.0129 0.0323 0.0011 0.0010 0.0014 0.0182 0.0234 SevenNet-MF-ompa 0.979 0.951 0.971 0.951 0.0180 0.0323 0.0019 0.0007 0.0012 0.0224 0.0261 35 Supplementary Note 5: Where guarantees hold: shift, chemistries and closed loops Shift between where labels are and where decisions are.Training and calibrating on MP, where labels are plentiful, and deciding on WBM fails twice (Extended Data Fig. 3). Discrimination degrades (AUROC 0.838 for a MACE-MP-0 head trained on MP, against 0.848 for its raw single-point hull distance), and MP-fitted calibration does not transfer: on the WBM rounds 2–5 structures of the test split the 90% confor- mal sets of the MP-calibrated head cover 0.87 (MACE-MP-0) and 0.86 (CHGNet) of the true labels, and conformal selection with an MP-calibrated linear head realises a false-discovery rate of 0.24 and 0.21 at a nominal 0.1. Moving the calibration source towards the candidates repairs coverage and false-discovery control without retraining (random slice: coverage 0.90 and 0.90; FDR 0.10 and 0.11) but cannot repair ranking. The same holds for the leaderboard models (Fig. 2e): calibrating conformal selection on WBM rounds 1–2 and deciding on rounds 4–5 breaks false-discovery control for 20 of 25 MP-trained potentials, and for the conformal screen of the main text, a threshold set on MP structures with an MP-trained head discards 19% of the stable candidates at a nominal 1%. Guarantees hold on average over chemistries, not within each.A map and a selection rule fitted to the whole pool are calibrated and controlled marginally. Within anion families (halides, oxides, chalco- genides, pnictides, intermetallics) the global map of the median MP-trained potential misstates the stable fraction of its worst family by 1.7 percentage points, and conformal selection realises false-discovery pro- portions of 0.114 among oxides. One map and one selection procedure per family (Mondrian calibration 30 ) reduce these to 0.5 points and 0.11, redistributing selection power between families rather than losing it (Supplementary Table 9). A typed answer has to state the population over which its guarantee is claimed. Closed loops select their own calibration data.In simulated budgeted DFT campaigns on WBM (Ex- tended Data Fig. 4; 30,000 candidates, 5,000 DFT calls in batches of 250, heads refitted every round), greedy acquisition finds 3.7–3.8 times as many stable structures as random sampling, but refitting the cal- ibration map on the latest acquisitions — which sit at the top of the ranking and on which the head has just been trained — leaves the remaining pool miscalibrated (ECE 0.058 for CHGNet against 0.016 under random acquisition) and underestimates the stable structures it still contains by 76%–80%. Out-of-sample calibration — cross-fitting over all acquisitions, the random seed, or a 10% random “calibration tax” in every batch — reduces the error to 13%–25% but does not remove it, and no strategy forecast the next batch without bias. Inside a loop, honest probabilities require labels the loop did not choose. Supplementary Note 6: DFT labels on the Materials Project energy scale Calculations.Calculations use VASP 45 with MP’s GGA(+U) protocol, start, as in WBM, from the un- relaxed candidate structure, and use a double relaxation. They differ from MP’s legacy calculations in the VASP version and in the release of the PBE projector-augmented-wave potentials 46;47 : the potentials avail- able to us carry the symbols and titles of MP’s, but their data identify them as a later release, and prospective chemistry is restricted to the 50 elements covered. Total energies therefore differ from MP’s by amounts that depend on the elements: recomputing 23 MP ground states from their MP structures gave energies that differ from MP’s by a median of 26 meV/atom (up to 235). Per-element shifts.Such differences are dominated by a constant per atom of each element. We measured it by recomputing MP’s elemental phases (the smallest cells within 0.1 eV/atom of the elemental ground state, two for 45 of the 50 elements, started from MP’s relaxed structures and magnetic moments) and, for Ni, V and W, which MP treats with+Uin oxides and fluorides, a binary oxide. One shiftδ el per element, and one per+Uelement in oxides and fluorides, was fitted by robust least squares to these calculations and to a first control set of 23 MP ground states and 8 WBM structures (129 calculations, median residual 2.9 meV/atom; NiO and Si, both references and controls, enter twice), and every energy is shifted by − ∑ el x el δ el , withx el the atomic fraction (shifts from−26 to 349 meV/atom). 36 Supplementary Table 9|Chemistry-conditional calibration of the leaderboard models (test part of five 5% calibra- tion slices; medians over models). CITL: calibration in the large, mean calibrated probability minus stable fraction (percentage points). FDP and power of conformal selection atq=0.1 within each family, with one isotonic map and one Benjamini–Hochberg procedure for all candidates (global) or one per family (Mondrian). CITL (points)FDP atq=0.1power modelsfamilystableglobalMondrianglobalMondrianglobalMondrian MP-trained potentials (25)halide28%-1.7+0.30.0600.1020.560.72 oxide12%-0.6-0.20.1140.0530.430.21 chalcogenide19%-0.4-0.20.0690.0830.520.51 pnictide17%-0.5+0.00.0720.0880.560.54 intermetallic13%+0.6+0.10.1120.1000.430.37 all15%+0.1+0.00.0970.1000.480.45 potentials, larger sets (30)halide28%-0.9+0.20.0740.0980.910.94 oxide12%+0.1-0.30.1060.0800.910.88 chalcogenide19%-0.3+0.00.0790.0880.910.92 pnictide17%-0.0+0.10.0810.0960.930.94 intermetallic13%+0.1-0.00.1100.0990.900.88 all15%-0.0-0.00.0970.0960.910.89 non-potential models (10)halide28%-6.9+0.50.0260.0570.020.02 oxide12%+1.0-0.10.1620.0000.000.00 chalcogenide19%-3.2-0.90.0020.0650.000.01 pnictide17%-0.8-0.30.0410.0000.000.00 intermetallic13%+1.6+0.10.0240.0250.000.00 all15%-0.0-0.00.0390.1010.000.01 Validation.A second control set, chosen before its results were known, contains 36 MP ground states covering all 50 elements and 9+Ucompounds of Ni, V and W, started from MP’s structures, and 20 WBM structures started, like WBM, from their initial geometry. Aligned energies reproduce MP’s to a median of 4.1 meV/atom (40 before the alignment; 95th percentile 13, at most 21;+Ucompounds 3.5, at most 13), and aligned hull distances reproduce WBM’s to a median of 3.7 meV/atom (56 before) with WBM’s stability label for all 20 (Extended Data Fig. 5). MP ground states lie on the hull by construction, so the sign of their residual decides their label, and 20 of 36 keep it. The residuals are not those of a pure constant per element: the tellurides of the two control sets lie below their reference after alignment (nine of nine, median 7 meV/atom), and the largest residuals are those of intermetallics. Because the first two sets contain many MP ground states and intermetallics and the candidates are ionic substitutions, a third control set matched to the candidate pool was chosen and computed after a review of the alignment, before any prospective label was collected: 100 WBM structures from the pool’s anion families in proportion to their share of it (hydroxides over-sampled), two thirds ternary and one third quaternary, with at most 12 sites and WBM hull distances between−0.05 and+0.10 eV/atom, started from their initial geometry. On it, aligned hull distances reproduce WBM’s to a median of 3.4 meV/atom with WBM’s label for 92 of 98 (11 of the 16 within 10 meV/atom of WBM’s hull), but the residuals have a heavy tail (95th percentile 44 meV/atom, mean +6.4): ten structures differ by more than 25 meV/atom, up to 301, eight of them with our energy above WBM’s, and seven of them are structures whose prototype WBM’s relaxation changed, against 7 of the other 88. The tail therefore reflects relaxations from the same unrelaxed start that end in different minima, not the alignment. These residuals define the label-noise model of the sensitivity analyses below. Labels.The MP2020 corrections 48 depend on the composition, on the oxide type of the relaxed structure and on an oxidation-state assignment, but not on the energy, so the aligned hull distance is the raw one minus the shift. We compute them with pymatgen 53 , as for WBM’s labels: all 30 WBM structures of the first two control sets receive WBM’s correction, and of the 68 MP compounds only TiSiOs 2 differs, carrying in MP’s database a Si anion correction that the current implementation does not assign. Hull distances are taken against the patched phase diagram of the MP 2023-02-07 entries, which reproduces the WBM labels 37 of Matbench Discovery exactly (400 random WBM entries, maximum deviation below 0.1 meV/atom); a candidate is stable if its hull distance, rounded to 1 meV/atom as in WBM, is at most zero. The shifts were fixed, and the SHA-256 digest of the file that holds them recorded (b88706bf90887082, 2026-10- 02 23:08:28 UTC), before any prospective calculation was collected. A secondary set of shifts adds to the fit MP’s elemental ground states, which the selection of the smallest cells missed for 20 elements (by up to 54 meV/atom, for Sn), and the second control set; it was fixed before the third control set and any prospective label were known (1932829c2a3ce264). On the third set it reproduces WBM’s hull distances to a median of 3.7 meV/atom (primary shifts: 3.4) with WBM’s label for 94 of 98, no material change; the prospective analyses are repeated with its labels. Calculations that did not finish are reported and excluded (five of the stage-1 sample); stage-1 calculations that failed or were interrupted while a fault of the MPI launcher serialised the runs sharing a node were restarted once after the fault was fixed. A stopping time for the last stage-1 calculation still running was recorded in the decision log before any stage-1 label was collected (it would then have counted as not finished); the calculation ended, unconverged, before that time, so the rule was not applied. All 100 shortlisted calculations finished. How constant are the shifts?The differences from MP are not explained by LASPH, which changes the energies of the first control set much less than they differ from MP, except for NiO. To first order they are constants of the elements: the two polymorphs computed for 45 elements differ by a median of 7.4 meV/atom, and shifts fitted on the elements and the first control set carry over to the second set (median residual 3.9 meV/atom). The polymorphs differ by more (24–44 meV/atom) for six elements (H, Nd, Sr, Sb, Ir, Lu), whose shifts are less certain; the 18 compounds of the second set that contain them are aligned to a median of 3.1 meV/atom (at most 9). The constant is not exact: tellurides lie systematically below their reference after alignment, intermetallics carry the largest residuals, and the shift implied by a compound can depend on the bonding of the element (199 meV per atom of Sn in SnO 2 , 140 in SmSn 2 , against a fitted 162). This is why the third control set, matched to the candidates, is the one that measures the label noise of the prospective test. On it (Supplementary Table 10), residuals are smallest for oxides, fluorides and pnictides and larger for chalcogenides, halides and hydroxides, and its tail is set by structures that WBM’s relaxation carried to another prototype. The elemental ground states that the first selection missed confirm that the shift of an element is only approximately a constant: their energies differ from the fitted shifts by a median of 14 meV per atom of the element (at most 36). Adding them and the second control set to the fit (the secondary shifts) changes the shifts by a median of 1.6 meV/atom (at most 14) and does not materially change the agreement on the third set. Supplementary Note 7: The registered prospective test in detail Candidates and registration.Following WBM 3 , candidates are data-mined ionic substitutions 38 into near-hull MP prototypes with two to four elements and small cells, within the elements for which DFT labels can be put on MP’s energy scale (below); only compositions absent from MP and from WBM are kept, and the pool is the 8,000 most probable candidates (5,780 compositions in 4,221 chemical systems, 98% with three or four elements, median 10 sites; 8,000 with forecasts from all backbones).Forecasts and registration.For each of the six backbones, heads are trained on all WBM unique prototypes except a random 5% calibration slice and applied unchanged to the pool; System-2 forecasts use the backbone’s own relaxation of each candidate (MACE-MP-0, CHGNet, MatterSim-v1-5M and ORB-v3). All forecasts, the sampling seed, the stage-1 DFT sample — a uniform random sample of the eligible pool, split at random into a calibration slice of 200 and a test sample of 400 candidates — and the stage-2 shortlist — the 100 remaining candidates with the highest single-pass probability of the primary backbone (ORB-v3, fixed in advance) — were written to a registration file whose SHA-256 digest (8f359667a6acda7c. . . ) was recorded outside the computing cluster at 2026-10-02 21:12:07 UTC, before any DFT calculation of a pool member, together with every backbone’s forecast of the shortlist’s yield. After the stage-1 labels, and before the shortlist’s DFT, the primary backbone’s single-pass probability was recalibrated (Platt) on all stage-1 labels and the updated forecast of the yield was registered in the same way (07d33c3f6708c63a. . . , 2026-10-03 16:12:04 UTC). 38 Supplementary Table 10|Third control set, matched to the prospective pool: WBM structures from their initial geometry, near the hull, by anion family (hydroxides over-sampled).∆E hull : our aligned hull distance minus WBM’s (meV/atom); label: WBM’s stability label reproduced. familycomputedfinishedmedian|∆E hull |mean∆E hull max|∆E hull |label oxide34332.7+3.37331/33 chalcogenide21217.0+15.030119/21 fluoride16152.8+0.93815/15 halide16166.0+9.45814/16 pnictide772.0+1.277/7 hydroxide668.3+5.3236/6 all100983.4+6.430192/98 Analysis plan.As registered — stage 1, on the test sample: calibration in the large (forecast minus realised stable fraction), ECE, log-loss and Brier score of every backbone’s registered System-1 forecast and, for the four backbones that relax, System-2 forecast, against the same forecasts recalibrated on the calibration slice (Platt and isotonic); realised net value atc=0.1, 0.3, 0.5 and 0.7 of the System-1 rule p 1 ≥c, the System-2 rule and relax-and-threshold; recall of the conformal screen calibrated on the slice (α=0.05, 0.1); and the false-discovery proportion of conformal selection (q=0.1, 0.2) among the test sample with the slice as calibration set. Stage 2: the forecast against the realised number of stable structures in the shortlist, for the forecasts registered with the pool and for the updated forecast. Secondary: the stage- 1 analyses without the+Uchemistries (oxides and fluorides of Ni, V and W), whose shifts rest on fewer reference calculations. In addition to the registered plan we report bootstrap 95% intervals of calibration in the large, AUROC and the ECE with five equal-mass bins, as implemented in the evaluation script fixed before the registration. Not registered, and added after the registration but before any stage-1 label was collected, are sensitivity analyses of stage 1 to the DFT labels: the expected number of labels that WBM’s protocol would give differently, from the residuals of the third control set (a candidate with aligned hull distancehhas WBM-protocol hull distanceh−r, withrdrawn from those residuals); calibration in the large with such a residual resampled for every candidate in each bootstrap replicate; the analyses without the candidates within 10 meV/atom of the hull; and labels from the secondary shifts. Also not registered are analyses of the shortlist: the probability of the realised yield under each forecast if the outcomes were independent (a Poisson-binomial distribution); every backbone’s forecast and single-pass hull distance for the unstable candidates; their DFT hull distance at the unrelaxed geometry (the final one plus the DFT relaxation energy, from the first ionic step); and DFT relaxations of them without symmetry (ISYM=0, otherwise the same settings) from MACE-MP-0’s relaxed structure and from the unrelaxed structure with Gaussian random displacements (σ=0.05 Å per Cartesian component). Calibration on new candidates.Of the 600 stage-1 calculations, five did not finish (Methods); 23 of the 397 test candidates (5.8%) and nine of the 198 candidates of the calibration slice (4.5%) are stable. Calibrated on WBM alone, the registered single-pass forecasts over-stated the stable fraction of the test sample by 0.8–2.1 percentage points, with bootstrap 95% intervals that include zero for four of the six backbones (ORB-v3: 0.8 points, interval−0.5 to 2.0), ranked the candidates with AUROC 0.965–0.982 and had an ECE of 0.009–0.021 (Fig. 6a,b; Supplementary Table 11); the forecasts after relaxation erred similarly (−0.1 to 2.4 points, intervals including zero for two of four; AUROC 0.976–0.995). Part of the over-statement may lie in the labels rather than in the forecasts. On the control set matched to the pool, the differences between our hull distances and WBM’s are centred on zero but have a tail in which ours are higher — structures that WBM’s relaxation carried to a lower minimum from the same start; resampling these differences, WBM’s protocol would be expected to give 9.3 test candidates (2.3%) the other label, mostly towards stable (net +1.4 points), and with this label noise propagated the intervals of all six backbones include zero (ORB-v3:−2.5 to +1.1 points; a sensitivity analysis that was not registered, Methods). Excluding the 16 test candidates within 10 meV/atom of the hull, or labelling with the secondary shifts (which change two labels), gives +1.0 and +0.3 points for ORB-v3; without the+Uchemistries (the 39 registered secondary analysis, 338 test candidates) the single-pass forecasts over-stated the stable fraction by 0.2–2.2 points. Recalibrating on the random slice moved calibration in the large closer to zero for five of the six backbones (median absolute error 1.3 points before, 0.6 after; range−1.6 to +0.8) but lowered the log-loss for only three: with nine stable candidates, the slice can correct the level of the forecasts but not their shape. Calibrated on WBM alone, the forecasts thus carried over to new candidates generated the same way to within about two percentage points of the stable fraction; whether the remaining over-statement belongs to the forecasts or to the labels, this test cannot settle. The probabilities were worth acting on.At an attempt cost ofc=0.5 (a failed attempt costs half a stable structure), attempting the test candidates withp 1 ≥crealised 8–19 net hits per 1,000 candidates across the six backbones (Fig. 6c; Supplementary Table 12). For CHGNet and MACE-MP-0, whose relaxations call too many candidates stable (precision 57% and 49%), the single pass realised 14 and 8 against 6 and−1 for relax-and-threshold, matched or beat it in seven of the eight comparisons at the registered costs and kept a positive value atc=0.7, where relax-and-threshold realised−11 and−23. Paired bootstrap intervals of the differences (post hoc; 10,000 resamples of the test candidates) exclude zero atc=0.7 (CHGNet 17, 95% interval 5 to 31; MACE-MP-0 26, 11 to 42) but not atc=0.5 (8,−4 to 19; 9,−4 to 21). For MatterSim and ORB-v3, whose relaxed calls are precise (83% and 90%), the two rules differed by at most 4 net hits per 1,000 (1.6 among the 397 test candidates), mostly in favour of relax-and-threshold, as the retrospective comparison anticipated (main text). Calibrating the relaxed prediction (the System-2 rule) matched or beat relax-and-threshold atc=0.5 for all four backbones that relax. Guarantees need larger calibration samples.The conformal procedures were consistent with their guarantees, which hold on average, but with a slice this small they guaranteed little (Supplementary Ta- ble 13). A screen atα=0.05 cannot discard any candidate until the slice holds 19 stable ones, and it kept all; atα=0.1 it discarded 53%–83% of the test candidates and kept 87%–100% of the stable ones. Because a conformal p-value cannot be smaller than one over the number of calibration candidates plus one, selec- tion with 198 calibration candidates can certify, atq=0.1, only sets of at least 20 of the 397 test candidates — nearly all 23 stable ones; it did so for SevenNet alone (21 candidates, false-discovery proportion 0.14), and atq=0.2 certified 0–23 candidates with false-discovery proportions of 0.07–0.17. Useful guarantees therefore need several times more random labels than calibration does: at a stable fraction of 5%, about 400 for a screen atα=0.05, and, whatever the stable fraction, about 800 for selection atq=0.1 to certify sets as small as five among 400 candidates. The shortlist.The 100 candidates that a screening campaign would have computed next yielded 98 stable structures, most of them far below the hull (median hull distance of the stable ones−123 meV/atom; Fig. 6d; Supplementary Table 14). Stability is defined against MP’s hull, as in WBM, and two features of the shortlist qualify its yield. It contains 79 distinct compositions, because several were generated from more than one parent prototype, and ten of its stable candidates (erbium bromides and osmium iodides) lie in chemical systems in which MP holds no binary compound of a cation with one of its anions, so that their hull distances are measured against an incomplete reference rather than against the phases into which they would decompose. The forecasts of this yield registered with the pool ranged from 91 to 100. Those of CHGNet and MACE-MP-0 under-stated it (for independent candidates, a yield of 98 or more had a probability below 0.008); those of MACE-MPA-0, MatterSim and SevenNet were consistent with it; and the primary backbone’s forecast — 99.9±0.3 at registration, 99.8±0.5 after recalibration on the stage-1 labels — was over-confident: two failures had a probability of 0.005 under it. The update could not have caught this, because only three stage-1 candidates fell in the shortlist’s probability range. The two failures, SrWSe 2 F and CaWSe 2 F (0.53 eV/atom above the hull), are substitutions into the same prototype, SrSbSe 2 F (W 3+ for Sb 3+ , and in CaWSe 2 F also Ca 2+ for Sr 2+ ), which ORB-v3 rated stable with probability 0.999. At the unrelaxed geometry, DFT places them 0.66 eV/atom above the hull, whereas the single passes of MACE-MP-0, MACE-MPA-0 and ORB-v3 placed them below it (−0.17 to−0.06 eV/atom) and those of CHGNet, MatterSim and SevenNet above it (0.32–0.60 eV/atom), and the latter gave both a probability of at most 0.01; the lowest probability that any backbone gave any other shortlisted candidate, before or 40 after relaxation, was 0.09. At the structures to which MACE-MP-0 relaxed them, which it places 0.53– 0.57 eV/atom below the hull, DFT places them 0.53–0.56 eV/atom above it, and DFT relaxations without symmetry from there, or from the unrelaxed structure with random displacements, end within 1.2 meV/atom of the registered calculations: the low-energy structure that MACE-MP-0 found does not exist in DFT, and the labels are not an artefact of a relaxation held in place by symmetry. A binomial forecast treats candidates as independent, but a shortlist built from one backbone’s ranking concentrates that backbone’s errors, and they come in groups — here two substitutions into one prototype — so the yield of a shortlist is less certain than its binomial spread suggests, and the single pass of a second backbone is a cheap way to find such groups (analyses not registered; Methods). Both failures and the+Uconvention.MP’s input set computes both failures with+Uon tungsten only because each contains one fluoride ion per formula unit, and the MP2020 correction for tungsten in oxides and fluorides supplies about three quarters of their total correction (−0.89 of−1.17 eV/atom). Their labels, and the potentials’ disagreement with them, may therefore reflect the GGA/GGA+Umixing convention as much as the chemistry; the labels follow the benchmark’s protocol, which WBM’s labels share. Deviations from the registered plan.(i) The label pipeline was completed after the registration and before any prospective label was collected: the primary per-element shifts were fixed, and their digest recorded, two hours after the registration; the secondary shifts and the third control set followed. (i) A stopping time for the last unfinished stage-1 calculation was recorded before any stage-1 label was collected; the calculation ended unconverged before that time, and the rule was not applied. (i) Stage-1 calculations that failed or were interrupted while a fault of the MPI launcher serialised the runs sharing a node were restarted once, before any label was collected. (iv) Bootstrap intervals and AUROC were computed by the evaluation script fixed before the registration but are not part of the registered plan, nor is the five-bin implementation of the registered expected calibration error; for forecasts with tied probabilities its value depends on the order of ties by up to 0.01, and we report the registered evaluation. (v) The sensitivity analyses of the labels were added after the registration and before any stage-1 label was collected. (vi) All analyses of the shortlist’s failures, including the diagnostic DFT calculations, are post hoc. The registration files, their full digests and the decision log will be deposited with the data. Supplementary Table 11|Prospective test, stage 1 (registered analysis): forecasts of the 397 randomly sampled test candidates against new DFT labels. CITL: forecast minus realised stable fraction in percentage points (bootstrap 95% interval); value: realised net hits per 1,000 candidates atc=0.5. Registered forecasts are calibrated on WBM only; slice forecasts are recalibrated on the 198 candidates of the random calibration slice. backboneforecastcalibrationCITL (95% CI)ECElog-lossAUROCvalue MACE-MP-0System 1registered+2.1 (+0.5, +3.7)0.0210.0970.970+8 MACE-MP-0System 1slice, Platt−1.3 (−3.0, +0.3)0.0240.1020.970+10 MACE-MP-0System 1slice, isotonic−1.2 (−2.9, +0.5)0.0130.1110.941+6 MACE-MP-0System 2registered+2.4 (+0.8, +4.2)0.0240.0950.979+5 MACE-MP-0System 2slice, Platt−1.0 (−2.8, +0.6)0.0240.0960.979+5 MACE-MP-0System 2slice, isotonic−0.9 (−2.7, +0.8)0.0120.0980.975+9 MACE-MP-0relaxed, isotonicregistered+2.6 (+0.8, +4.5)0.0260.1430.970−3 MACE-MP-0relaxed, isotonicslice, Platt−0.9 (−2.8, +0.9)0.0190.1350.970+9 MACE-MP-0relaxed, isotonicslice, isotonic−1.1 (−2.9, +0.6)0.0150.0960.969+6 MACE-MP-0relax, ˆ E hull ≤0–−1 CHGNetSystem 1registered+1.4 (−0.1, +2.8)0.0140.0850.980+14 CHGNetSystem 1slice, Platt−1.6 (−3.3, 0.0)0.0230.0940.980+10 CHGNetSystem 1slice, isotonic−1.3 (−3.1, +0.3)0.0250.1170.973+10 CHGNetSystem 2registered+2.1 (+0.6, +3.6)0.0210.0850.984+14 CHGNetSystem 2slice, Platt−1.3 (−2.9, +0.2)0.0190.0840.984+6 CHGNetSystem 2slice, isotonic−0.8 (−2.6, +1.0)0.0080.1330.969+5 41 Supplementary Table 11|(continued) backboneforecastcalibrationCITL (95% CI)ECElog-lossAUROCvalue CHGNetrelaxed, isotonicregistered+1.7 (+0.2, +3.2)0.0170.0920.984+10 CHGNetrelaxed, isotonicslice, Platt−1.6 (−3.3, 0.0)0.0270.0940.984+11 CHGNetrelaxed, isotonicslice, isotonic−1.4 (−3.1, +0.1)0.0140.0810.982+11 CHGNetrelax, ˆ E hull ≤0–+6 MACE-MPA-0System 1registered+1.2 (−0.1, +2.5)0.0120.0720.982+19 MACE-MPA-0System 1slice, Platt−0.5 (−1.9, +0.7)0.0180.0750.982+15 MACE-MPA-0System 1slice, isotonic−0.3 (−1.9, +1.1)0.0040.0890.981+15 MatterSim-v1-5MSystem 1registered+0.9 (−0.5, +2.2)0.0090.0760.969+18 MatterSim-v1-5MSystem 1slice, Platt−0.3 (−1.6, +1.0)0.0080.0740.969+19 MatterSim-v1-5MSystem 1slice, isotonic−0.1 (−1.6, +1.2)0.0090.1130.938+19 MatterSim-v1-5MSystem 2registered+0.5 (−0.8, +1.8)0.0080.0780.976+19 MatterSim-v1-5MSystem 2slice, Platt−1.0 (−2.4, +0.4)0.0130.0760.976+15 MatterSim-v1-5MSystem 2slice, isotonic−1.5 (−3.0,−0.2)0.0150.1080.928+11 MatterSim-v1-5Mrelaxed, isotonicregistered+0.8 (−0.5, +2.2)0.0110.0820.966+19 MatterSim-v1-5Mrelaxed, isotonicslice, Platt−1.3 (−2.8, +0.1)0.0160.0810.966+13 MatterSim-v1-5Mrelaxed, isotonicslice, isotonic−1.5 (−3.0,−0.2)0.0160.1140.927+13 MatterSim-v1-5Mrelax, ˆ E hull ≤0–+19 ORB-v3System 1registered+0.8 (−0.5, +2.0)0.0120.0710.975+19 ORB-v3System 1slice, Platt+0.1 (−1.2, +1.3)0.0060.0690.975+19 ORB-v3System 1slice, isotonic−0.2 (−1.5, +1.0)0.0080.1040.941+19 ORB-v3System 2registered−0.1 (−1.4, +1.0)0.0040.0470.995+21 ORB-v3System 2slice, Platt−0.4 (−1.7, +0.7)0.0050.0480.995+21 ORB-v3System 2slice, isotonic−0.2 (−1.5, +0.8)0.0030.0560.994+21 ORB-v3relaxed, isotonicregistered+0.1 (−1.2, +1.3)0.0020.0530.994+21 ORB-v3relaxed, isotonicslice, Platt−1.0 (−2.3, +0.3)0.0110.0570.994+20 ORB-v3relaxed, isotonicslice, isotonic−1.1 (−2.6, +0.2)0.0110.1210.909+18 ORB-v3relax, ˆ E hull ≤0–+21 SevenNet-MF-ompaSystem 1registered+1.5 (+0.3, +2.7)0.0180.0690.965+18 SevenNet-MF-ompaSystem 1slice, Platt+0.8 (−0.4, +2.0)0.0110.0670.965+20 SevenNet-MF-ompaSystem 1slice, isotonic+1.1 (−0.1, +2.3)0.0160.0910.961+20 Supplementary Table 12|Prospective test, stage 1 (registered analysis): realised net value per 1,000 test candidates, 1000(y−c)1[attempt], at the four registered attempt costsc, for the System-1 rulep 1 ≥cand the System-2 rule p 2 ≥cwith the registered (WBM-calibrated) forecasts, and for relax-and-threshold (attempt if the backbone’s relaxed, MP2020-corrected hull distance is at most zero; precision and recall in brackets). backbonerulec=0.1c=0.3c=0.5c=0.7 MACE-MP-0System 1,p 1 ≥c+40.3+20.9+7.6+2.8 MACE-MP-0System 2,p 2 ≥c+44.3+21.7+5.0−1.5 MACE-MP-0relax-and-threshold (49%, 91%)+42.1+20.4−1.3−22.9 CHGNetSystem 1,p 1 ≥c+43.1+25.2+13.9+5.5 CHGNetSystem 2,p 2 ≥c+43.6+24.9+13.9+4.3 CHGNetrelax-and-threshold (57%, 87%)+41.6+23.9+6.3−11.3 MACE-MPA-0System 1,p 1 ≥c+40.1+28.2+18.9+8.8 MatterSim-v1-5MSystem 1,p 1 ≥c+38.0+28.0+17.6+10.3 42 Supplementary Table 12|(continued) backbonerulec=0.1c=0.3c=0.5c=0.7 MatterSim-v1-5MSystem 2,p 2 ≥c+42.1+30.0+18.9+5.8 MatterSim-v1-5Mrelax-and-threshold (83%, 83%)+42.1+30.5+18.9+7.3 ORB-v3System 1,p 1 ≥c+40.6+32.0+18.9+9.3 ORB-v3System 2,p 2 ≥c+45.8+30.5+21.4+8.6 ORB-v3relax-and-threshold (90%, 83%)+42.6+32.0+21.4+10.8 SevenNet-MF-ompaSystem 1,p 1 ≥c+41.3+29.7+17.6+11.8 Supplementary Table 13|Prospective test, stage 1 (registered analysis): conformal screen and conformal selection with the 198 candidates of the random calibration slice (nine of them stable) as calibration set and the 397 test candi- dates (23 stable) as candidates, from the registered System-1 scores. Screen at levelα: fraction of test candidates kept and recall of the stable ones (the guarantee: recall at least 1−αin expectation; withnstable calibration candidates the screen can discard candidates only ifα(n+1)≥1). Selection at false-discovery rateq: number of candidates certified and the realised false-discovery proportion. screenα=0.05screenα=0.1selectionq=0.1selectionq=0.2 backbonekeptrecallkeptrecalln(FDP)n(FDP) MACE-MP-0100%100%20%96%00 CHGNet100%100%32%100%012 (0.08) MACE-MPA-0100%100%19%100%014 (0.07) MatterSim-v1-5M100%100%17%87%015 (0.07) ORB-v3100%100%19%91%018 (0.11) SevenNet-MF-ompa100%100%47%96%21 (0.14)23 (0.17) Supplementary Table 14|Prospective test, stage 2: the shortlist a screening campaign would compute (the 100 candidates with the highest registered single-pass probability of ORB-v3 outside stage 1) yielded 98 stable structures. Expected yield±binomial standard deviation for every forecast registered before the shortlist’s DFT; the last two columns (not registered) give the probability of the realised yield or less, and of the realised yield or more, if the candidates’ outcomes were independent with the forecast probabilities. forecastregisteredexpected yieldP(≤realised)P(≥realised) MACE-MP-0, System 1with the pool92.6±2.21.000.008 MACE-MP-0, System 2with the pool92.2±2.31.000.005 CHGNet, System 1with the pool91.7±1.91.00<0.001 CHGNet, System 2with the pool91.2±1.91.00<0.001 MACE-MPA-0, System 1with the pool99.1±0.90.210.95 MatterSim-v1-5M, System 1with the pool97.1±0.71.000.30 MatterSim-v1-5M, System 2with the pool98.6±0.90.420.88 ORB-v3, System 1with the pool99.9±0.30.0051.00 ORB-v3, System 2with the pool99.5±0.70.100.98 SevenNet-MF-ompa, System 1with the pool97.8±0.50.990.81 ORB-v3, System 1, recalibrated (Platt) on stage 1after stage 199.8±0.50.0261.00 ORB-v3, System 1, recalibrated (isotonic) on stage 1after stage 1100.0±0.0<0.0011.00 43

Instant research discovery