Paper deep dive
From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders
Pritam Deka
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/5/2026, 6:15:54 AM
Summary
This paper introduces SBERT2S1, a framework that converts biomedical sentence encoders (trained for retrieval) into System One typed-decision models. It presents BioDecide, a comprehensive biomedical decision suite, and MEDLINE-S1, a large-scale training dataset derived from NLM indexing. The study evaluates various conversion heads (Cross-head, Prior-Fused Residual) and training objectives (Cross-Entropy, RLCD variants), finding that retrieval-initialized models benefit specific heads like PFR and that standard RLCD implementations suffer from biased noise estimation, which can be corrected via leave-one-out estimators.
Entities (11)
Relation Signals (9)
SBERT2S1 → converts → Sentence-Transformers encoders
confidence 95% · We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models
MEDLINE-S1 → derivedfrom → NLM indexing
confidence 95% · MEDLINE-S1, 243k training decisions derived from NLM indexing
SBERT2S1 → includes → BioDecide
confidence 95% · together with BIODECIDE, a biomedical typed-decision suite
SBERT2S1 → includes → MEDLINE-S1
confidence 95% · and MEDLINE-S1, 243k training decisions derived from NLM indexing
S-PubMedBERT-MS-MARCO → isfinetunedversionof → PubMedBERT
confidence 93% · S-PubMedBERT-MS-MARCO (Deka et al., 2022) is PubMedBERT fine-tuned on MS MARCO
RLCD → underperforms → Cross-Entropy (CE)
confidence 92% · the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points
Cross-head (C) → outperforms → Prior-Fused Residual (PFR)
confidence 90% · A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective
RLCD → suffersfrom → biased noise estimation
confidence 90% · The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.
Tags
Links
- Source: https://arxiv.org/abs/2610.02486v1
- Canonical: https://arxiv.org/abs/2610.02486v1
Trouble viewing inline? Open PDF directly →
Full Text
89,323 characters extracted from source content.
Expand or collapse full text
From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders Pritam Deka Affiliation: Queen’s University Belfast Affiliation: Belfast, United Kingdom Abstract Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present sbert2s1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BioDecide, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5–3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6–15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model. 1 Introduction Much of the language processing in biomedical software consists of small, bounded decisions: does this abstract report a randomised trial, does the evidence support the claim, how severe are the side effects a patient describes? Because the answers are consumed by code, a decision is only useful if it respects a schema and comes with a confidence that can be thresholded to act, defer or escalate. Request State: title and abstract of a PubMed citation choice: Study design? RCT, cohort, case report, … noul: Did the study involve human participants? score: Strength of evidence? [none << weak << … << strong] C cross head over question, [MASK] slots and state Z / B cosine between state and question+option PFR C as a residual on the retrieval prior of Z One forward pass: a temperature-scaled distribution per question, e.g. RCT 0.91; humans P(true)=0.97P(true)=0.97; evidence level 2.3 of 3 confident → act uncertain → defer or escalate to an LLM sentence encoder EθE_θ,converted with one of four headsfine-tuned with CE, proper scores orRLCD; temperatures fitted per type Figure 1: Overview. A System One request pairs a state with typed questions. A biomedical sentence encoder, converted with one of four heads (§3), returns a calibrated distribution over each question’s options in one forward pass, and downstream code thresholds it to act or escalate. Example values are illustrative. System One models were proposed for exactly this interface (TypeSafe AI, 2026). A request pairs a text (the state) with typed questions of three kinds (choice, score and noul), and the model returns, in one forward pass, a distribution over the options each question allows. The open model Laya (Convai Innovations, 2026) pairs an encoder with a mask-slot head and trains it with RLCD, a mixture of cross-entropy and a policy gradient against proper scoring rules (Gneiting and Raftery, 2007). All existing System One models are general-domain and start from masked-language-model (MLM) or decoder checkpoints. Most biomedical encoders are MLMs too, such as BioBERT (Lee et al., 2020) and PubMedBERT (Gu et al., 2021). A smaller group has been further trained contrastively as Sentence-Transformers encoders for retrieval: S-PubMedBERT-MS-MARCO (Deka et al., 2022) is PubMedBERT fine-tuned on MS MARCO (Nguyen et al., 2016), and MedCPT (Jin et al., 2023) is trained on PubMed search logs. A typed decision resembles retrieval, since it asks whether a hypothesis formed from a question and an option matches the state, but retrieval similarity need not encode negation, ordinal judgement or calibrated truth. This study tests whether biomedical sentence encoders can be converted into System One models, and what the result depends on: RQ1 Initialisation: when does retrieval training improve zero-shot or fine-tuned decisions? RQ2 Conversion: how do bi-encoder, cross-head and prior-fused conversions trade accuracy, robustness to option order, cost and retrieval retention, under matched objectives? RQ3 Objective: which ingredient of RLCD matters once temperatures are fitted: the proper score, the noise smoothing, or the policy-gradient estimator? Contributions. (1) sbert2s1, a framework that turns Sentence-Transformers encoders into typed decision models with four conversions, including a prior-fused residual (PFR) head that starts from the calibrated zero-shot retriever; (2) BioDecide, a biomedical typed-decision suite with public and credentialed clinical tracks, and MEDLINE-S1, multi-question training data labelled from NLM indexing rather than an LLM teacher; (3) a controlled study over eleven encoders, six parent-retriever pairs, and a matched grid of two heads and five objectives; and (4) an analysis showing that RLCD optimises a noise-smoothed proper score, that the released estimator is biased and rescaled, and that a simple leave-one-out baseline removes both problems. Code, MEDLINE-S1 and the released model are available at https://github.com/pritamdeka/sbert2s1, https://huggingface.co/datasets/pritamdeka/MEDLINE-S1 and https://huggingface.co/pritamdeka/S1-PubMedBERT. 2 Related Work System One models. Jev (TypeSafe AI, 2026) introduced typed decisions with three primitives: choice (one option from a schema), score (a distribution over an ordinal rubric) and noul (the probability that a statement holds); its training objective is not public. Laya (Convai Innovations, 2026), the first open implementation, pairs ModernBERT-large (Warner et al., 2025) with a two-layer head that scores one [MASK] slot per option, trained with cross-entropy plus a group-baselined policy gradient (as in GRPO; Shao et al., 2024) on noise-perturbed logits rewarded by proper scores. Decider (Marosi, 2026) fine-tunes decoders, and the Decision Index (Decision Index contributors, 2026) benchmarks typed-decision engines. Concurrent work applies or probes System One models in security agents, annotation, crash-report coding, distracting contexts and active questioning (dos Santos, 2026; Ibrahim and Zaki, 2026; Rafe and Das, 2026; Xu, 2026; Yilmaz et al., 2026) (Appendix A). These studies do not evaluate biomedical conversions, retrieval initialisation, or RLCD ablations. Calibration and scoring rules. A strictly proper scoring rule is uniquely optimised in expectation by the true distribution (Gneiting and Raftery, 2007); examples include the log, Brier (Brier, 1950), spherical and ranked-probability (Epstein, 1969) scores. Neural networks, including pre-trained transformers, are often miscalibrated, and temperature scaling is a strong post-hoc remedy (Guo et al., 2017; Desai and Durrett, 2020). A calibration benefit claimed for a training objective must therefore survive temperature scaling. Sentence encoders as decision models. Siamese encoders trained so that cosine similarity reflects relatedness (Reimers and Gurevych, 2019), on query-passage data such as MS MARCO (Nguyen et al., 2016), are standard for dense retrieval; cross-encoders re-rank more accurately at higher cost (Nogueira and Cho, 2019). Comparing a text with label descriptions underlies entailment-based zero-shot classification (Yin et al., 2019). In biomedicine, domain pre-training (Gu et al., 2021) followed by contrastive training on search logs (Jin et al., 2023) or MS MARCO (Deka et al., 2022) yields strong retrievers. We use such encoders as the initialisation of decision models and measure what the contrastive stage adds. 3 Method 3.1 Typed decisions A request consists of a state s (text or a serialised record) and questions q=(τ,x,o1:K)q=(τ,x,o_1:K), where τ∈choice,score,noulτ∈\ choice, score, noul\ is the type, x the instruction and o1,…,oKo_1,…,o_K the options: schema keys with optional descriptions for choice, ordered rubric levels for score, and (false,true)(false,true) for noul. We use the request schema shared by Jev and Laya. A model maps (s,q)(s,q) to logits z∈ℝKz ^K and reports p(⋅∣s,q)=softmax(z/Tτ,K),p(· s,q)=softmax (z/T_τ,K ), (1) where Tτ,K>0T_τ,K>0 is a post-hoc temperature shared by all questions of type τ in one option-count bucket (§3.4). The answer is argmaxkpk _kp_k for choice, the expected level ∑k(k−1)pk _k(k-1)\,p_k for score, and p2p_2 (true) for noul. The confidence maxkpk _kp_k is the quantity that is thresholded and that expected calibration error (ECE) measures. 3.2 Converting an encoder Let EθE_θ be the transformer of a Sentence-Transformers model and eθ(⋅)=pool(Eθ(⋅))∥pool(Eθ(⋅))∥2e_θ(·)= pool(E_θ(·)) (E_θ(·)) _2 (2) its normalised embedding, with the checkpoint’s pooling (mean or [CLS]). For option k we form the hypothesis hk=x⊕okh_k=x o_k (instruction followed by the rendered option) and define the retrieval prior ck=⟨eθ(s),eθ(hk)⟩c_k= e_θ(s),e_θ(h_k) ; the hypothesis plays the query and the state the passage. We compare four conversions: Z, B: zk=ατck, z_k= _τ\,c_k, (3) C: zk=rθ,ϕ(s,q)k, z_k=r_θ,φ(s,q)_k, (4) PFR: zk=rθ,ϕ(s,q)k+ατck. z_k=r_θ,φ(s,q)_k+ _τ\,c_k. (5) Z keeps θ frozen and fits only the per-type scale ατ _τ. B trains θ and α; state embeddings can be pre-computed, so screening a corpus costs one embedding of K hypotheses and a matrix product. C is a cross-encoder head: EθE_θ encodes [CLS]τx[SEP][MASK]o1⋯[MASK]oK [CLS]\,τ\,x\, [SEP]\; [MASK]\,o_1·s [MASK]\,o_K [SEP]s[SEP], [SEP]\;s\, [SEP], a two-layer transformer ϕφ with a type embedding contextualises it, and an MLP scores the hidden state at the k-th [MASK]. The head replicates Laya’s, so the two load each other’s weights: the public Laya checkpoint run through our implementation reproduces its reported zero-shot accuracy on typed-decisions (0.3615 vs. 0.362). PFR adds this cross score to the retrieval prior. The last layer of the scorer is zero-initialised, so the untrained PFR model equals Z and training learns a cross-attention residual on top of the retriever. The prior shares θ by default; a frozen copy preserves the original retriever at twice the memory. The prior scale is initialised at ατ=20 _τ=20, the usual similarity scale of contrastive encoders,11 1 The implementation sets ατ=max(1/Tτ0,20) _τ= (1/T^0_τ,20) from a temperature Tτ0T^0_τ fitted to raw cosines; because Tτ0T^0_τ is clipped to [0.05,20][0.05,20], this is always 20. has its own learning rate and no weight decay. 3.3 Training objectives and the RLCD estimator All objectives compare the reported distribution p with a target distribution t (one-hot for most items). The simplest is cross-entropy (CE). Laya’s reward S(p,t)S(p,t) adds two further proper scoring rules to the log score: the spherical score for every question and, for score questions, the ranked probability score, which also penalises probability placed on distant rubric levels. A scoring rule is proper when reporting the true distribution maximises its expected value, so a high S rewards honest probabilities (Gneiting and Raftery, 2007). RLCD does not maximise S directly. For each item it adds Gaussian noise to the logits G=4G=4 times, scores each noisy prediction with S, and moves the logits towards the noise directions that scored above average. This is a policy-gradient (REINFORCE) step, and CE is added to it with weight one. The reward part of the update for one item is g^=1Gσ2∑g=1GR(g)−R¯dε(g), g= 1Gσ^2 _g=1^G R^(g)- Rd\, ^(g), (6) where ε(g) ^(g) is the g-th noise draw, σ its scale (annealed from 0.4 to 0.1), R(g)R^(g) its reward, R¯ R the mean reward of the item’s G draws, and d the standard deviation of the centred rewards in the minibatch. Without R¯ R and d, g g would be an unbiased estimate of the gradient of the expected reward under noise, a smoothed version of S that tends to S as σ→0σ→ 0. The two choices change this. First, R¯ R contains the draw’s own reward, so on average the estimate is shrunk by (G−1)/G=3/4(G-1)/G=3/4. Second, the reward differences shrink roughly in proportion to σ, so dividing by d makes the estimate grow roughly as 1/σ1/σ while the noise is annealed. Because CE keeps a fixed weight, the noisy reward term gradually dominates it. Appendix C states this formally (Proposition 1) and proves it, and Appendix D measures it. To remove these biases, use the mean of the other G−1G-1 rewards as the baseline and do not divide by d. We call this PG-LOO (leave-one-out). We compare six objectives with identical data and schedule: CE; Proper, which maximises S without noise; PG, the released RLCD recipe; PG w/o CE, its reward term alone; PG-LOO; and Reparam, which differentiates the smoothed reward directly through the noise (the reparameterisation trick) and adds CE. PG-LOO and Reparam estimate the same gradient on the same scale and differ only in variance. 3.4 Calibration and long states Temperatures. After training, one temperature per type and option-count bucket (K=2K=2, 33–55, 66–1010, ≥11≥11) is fitted by minimising negative log-likelihood on a calibration split carved from the training data before training, with a per-type fallback for buckets with fewer than ten items. Every metric is reported before and after this step. Long states. Clinical notes exceed the 512-token context of base encoders. Besides truncation and sliding windows, we evaluate retrieve-then-decide: the model’s own embedding eθe_θ selects the m=3m=3 note chunks most similar to the instruction x, and the decision is made on those chunks in document order. This requires the converted model to remain a retriever. 4 Data BioDecide. We convert existing biomedical datasets into typed decisions over a state (Table 1; conversions in Appendix B). In the grounded track the state determines the answer: evidence QA (PubMedQA; Jin et al., 2019), claim verification (SciFact, HealthVer and PUBHEALTH; Wadden et al., 2020; Sarrouti et al., 2021; Kotonya and Toni, 2020), drug–drug interactions (DDI; Herrero-Zazo et al., 2013), cancer hallmarks with ten noul questions per abstract (HoC; Baker et al., 2016), adverse drug events (ADE; Gurulingappa et al., 2012), patient drug reviews with three score questions each (Druglib; Gräßer et al., 2018), sentence similarity (BIOSSES; Soğancıoğlu et al., 2017) and specialty routing (MTSamples). In the knowledge track (MedQA, MedMCQA and MMLU-medical; Jin et al., 2021; Pal et al., 2022; Hendrycks et al., 2021) the state is an exam vignette whose answer depends on memorised knowledge; it is reported but does not drive headline numbers. PUBHEALTH, BIOSSES and MMLU are never trained on, and MTSamples is a held-out task family. We also evaluate on the general-domain typed-decisions set used by Laya. Task Family Type #Train #Test Grounded PubMedQA-L Evidence QA choice 400 500 PubMedQA-A§ Evidence QA noul 20,000 – SciFact Claim verif. choice 780 321 HealthVer Claim verif. choice 10,073 1,823 PUBHEALTH† Claim verif. choice – 1,231 DDI-2013 Relation choice 18,379 7,039 HoC Doc. class. noul 11,660 3,710 ADE Adv. events noul 16,318 2,089 Druglib Review score 8,391 3,108 BIOSSES† Similarity score – 100 MTSamples‡ Routing choice – 370 MEDLINE-S1 Indexing mixed 242,986 30,522 Knowledge MedQA-USMLE Exam choice 9,778 1,273 MedMCQA Exam choice 20,000 2,816 MMLU-medical† Exam choice – 1,089 General domain typed-decisions† General mixed – 2,000 Clinical (PhysioNet credentialed) MedNLI NLI choice 10,832 1,422 MIMIC-I trials† Long notes mixed – 1,532 MIMIC-IV names Privacy noul 1,828 9,808 Total 371,425 70,753 Table 1: BioDecide and MEDLINE-S1, in decisions (a state may carry several questions). † test only (never trained on); ‡ held-out task family; § @sectionsign training only. Historical calibration slices are carved from the training data before training and are not counted. Clinical training counts apply only to the +clinical variant. Counts describe the original splits; the integrity audit reports exclusions separately. Conversion. Each source example becomes one record: a state (the text the decision is about), one or more typed questions, and a gold distribution over each question’s options (Figure 1; a full record is in Figure 4). The dataset’s question or label set becomes the instruction, and each label gets a short description, written once per task, so that options carry content rather than bare label words. Multi-label annotations become one noul per label (HoC), multi-aspect ratings become several questions about one state (Druglib), a graded score becomes a score question whose target splits its mass between the two nearest levels (BIOSSES), and exam items become choice questions with the answer texts as descriptions. Answer-revealing fields are removed (PubMedQA conclusions, PUBHEALTH explanations). Official splits are kept where they exist, and before any training a calibration slice (about 10% of training, at most 400 states) is carved off, grouped by claim or topic where the source groups items. Appendix B details every dataset. MEDLINE-S1. Existing corpora pose one question per text, whereas System One requests typically ask several heterogeneous questions about one state. Multi-question training data is usually produced by an LLM teacher, whose errors then become targets. We instead derive labels from MEDLINE indexing, which NLM curates: study design (six classes from publication types, with a fixed precedence for overlapping tags); disease area (the single MeSH disease-tree category among major topics, against 3–7 randomly drawn distractors, so that no encoder is favoured by model-mined negatives); human or animal subjects; whether adverse effects or drug therapy are reported; and, for human studies, two sampled age or sex groups. We keep 2005–2021 citations with manual or human-curated indexing and exclude every PMID in any BioDecide evaluation set. Tags are weak labels: absence need not mean false, and indexers may use the full article. The training set has 40k abstracts (242,986 decisions); the test set has 5k abstracts from disjoint files. Clinical track. Under PhysioNet credentialed access (Pollard et al., 2026) we convert MedNLI (Romanov and Shivade, 2018; Shivade, 2019); 1,000 human-reviewed trial-eligibility questions over MIMIC-I discharge notes (Woo et al., 2024; Woo et al., 2025; Johnson et al., 2016), posed as yes/no/not stated and as an answerability noul; and a name-mention noul over windows of MIMIC-IV notes with inserted surrogate names (Lim et al., 2023; Johnson et al., 2023), including a split of names that are also common words. The MIMIC-I notes average 12.9k characters and form our long-state test. Credentialed data were processed only on institutional hardware and never sent to a hosted API; the released model is trained on public data only. 5 Experimental Setup Encoders. We evaluate eleven base-size checkpoints: PubMedBERT and its retrieval children S-PubMedBERT-MS-MARCO and MedCPT-Query; MPNet and all-MPNet-base-v2; ModernBERT-base and its Nomic and GTE embedders; BioClinical-ModernBERT and its embedder; and BGE-base-v1.5 as an unpaired retriever (Gu et al., 2021; Deka et al., 2022; Jin et al., 2023; Song et al., 2020; Sentence Transformers, 2021; Warner et al., 2025; Nussbaum et al., 2024; Zhang et al., 2024; Li et al., 2023; Sounack et al., 2025; NeuML, 2025; Xiao et al., 2024). This gives six parent-retriever comparisons with shared parents and different retrieval corpora; they are not six independent replications of one intervention. Model-specific query/document prefixes are used. Training. Training tasks are sampled with probability ∝nt1/2 n_t^1/2. The budget is 8,000 steps at batch size 32 (at most eight epochs), with AdamW, bf16 autocast over fp32 weights and gradient clipping at norm 1. Options are shuffled except for ordinal rubrics. All encoders are trained with C and PFR under the released RLCD recipe (PG), three seeds each, plus 2% and 10% learning curves for the ten paired checkpoints, also with three seeds. On S-PubMedBERT-MS-MARCO we run the full head × objective grid: C and PFR with CE, Proper, PG, PG-LOO and Reparam (and PG w/o CE for PFR), three seeds per cell, identical data, schedule and seeds. Appendix K lists hyperparameters. Baselines. Laya-large (421M) is evaluated zero-shot and fine-tuned on our mixture with the same recipe. Qwen3.8-27B (Qwen Team, 2026) and Gemma-4-31B (Gemma Team, 2026) score option letters in one forward pass with thinking disabled (Appendix J). All systems are temperature-scaled on the same calibration splits. Exam questions are likely present in LLM pre-training data, so LLM scores on the knowledge track are descriptive only. Metrics and statistics. The headline metric is chance-normalised accuracy acccn=(acc−1/K)/(1−1/K)acc_cn=(acc-1/K)/(1-1/K), averaged over (question, K) groups within a task and then over tasks. Seen denotes the eight grounded training tasks; held-out PUBHEALTH, BIOSSES and MTSamples. We also report top-label ECE and NLL before and after temperature scaling, ordinal error, option-order flips and retrieval retention (nDCG@10 on BEIR SciFact and NFCorpus; Thakur et al., 2021; Boteva et al., 2016). Confidence intervals come from a paired bootstrap (10,000 draws) over document clusters within each task: all records sharing a normalised text are resampled together. Intervals condition on the observed tasks and on seed-averaged correctness; seed sd is reported separately. Holm correction is applied within each research question across both tracks (Appendix F). Non-significance is not read as equivalence. Data integrity. An exact-input audit found a few training duplicates in calibration (nine DDI and seven Druglib records) and test (two SciFact and 13 Druglib records) splits, and repeated inputs within some test sets. Removing all affected test inputs lowers seen-task scores by 0.3–0.4 points for every model and changes no ranking (Appendix I). 6 Results Seen tasks, acccn_cn Held-out, acccn_cn Clinical ECETS_TS Encoder Type C PFR C, 10% C PFR C C PubMedBERT MLM 67.3± 0.7 66.9± 0.8 59.1± 0.6 29.5± 3.3 20.7± 0.6 20.7± 0.8 4.8± 0.9 S-PubMedBERT-MS-MARCO contr. 67.7± 0.9 65.3± 1.5 61.0± 1.4 32.8± 1.5 23.6± 1.5 20.5± 1.8 3.9± 0.7 MedCPT-Query contr. 17.9± 4.3 60.4± 0.7 15.0± 10.2 3.9± 0.5 23.6± 3.2 12.5± 3.4 14.0± 2.3 MPNet MLM 66.4± 0.9 63.3± 0.4 61.5± 0.5 30.2± 2.7 20.8± 2.5 22.7± 5.2 4.8± 0.4 all-MPNet-base-v2 contr. 66.4± 0.9 67.8± 0.8 60.5± 2.5 27.5± 0.8 24.7± 2.3 13.7± 5.0 5.2± 0.3 ModernBERT-base MLM 71.6± 0.7 66.5± 0.9 64.0± 0.6 32.5± 1.4 21.4± 1.8 26.1± 2.5 4.3± 0.2 modernbert-embed-base contr. 70.6± 1.0 69.2± 0.6 61.5± 0.4 35.0± 1.1 32.3± 4.1 13.4± 2.9 4.9± 0.2 gte-modernbert-base contr. 69.1± 0.8 69.3± 0.2 60.8± 1.2 29.8± 1.7 32.3± 0.2 24.3± 1.8 4.5± 0.1 BioClinical-ModernBERT MLM 73.1± 0.5 67.5± 0.5 66.1± 0.8 31.8± 2.0 21.0± 3.6 32.2± 4.1 4.1± 0.2 BioClinical-MB-embeddings contr. 72.9± 0.7 68.3± 0.0 62.4± 0.9 32.1± 4.8 22.5± 1.7 26.5± 1.2 4.6± 0.5 BGE-base-v1.5 (unpaired) contr. 60.6± 0.5 66.8± 0.3 – 16.1± 3.2 30.7± 3.0 7.8± 15.5 5.7± 0.7 PubMedBERT, B 67.0± 0.5 – 17.8± 6.5 23.6± 1.4 4.1± 0.1 S-PubMedBERT, B 67.1± 0.5 – 23.4± 1.4 25.9± 2.1 3.9± 0.2 S-PubMedBERT, C trained with CE 70.2± 0.4 – 31.0± 1.7 20.0± 4.1 3.8± 0.2 S-PubMedBERT, PFR frozen prior 67.8± 1.0 – 26.8± 1.9 19.5± 5.3 4.3± 0.2 S-PubMedBERT, PFR +clinical train 66.3± 0.2 – 20.4± 0.8 55.2± 0.6 3.9± 0.4 Laya-large (421M), C fine-tuned 75.0± 0.4 – 35.8± 2.7 38.4± 4.5 4.3± 0.3 Qwen3.8-27B, zero-shot 57.4 – 46.2 81.6 8.5 Gemma-4-31B, zero-shot 65.7 – 52.6 83.4 9.1 Table 2: Main results (×100× 100; mean± over 3 seeds; 10% also uses 3 seeds). Encoders are grouped by matched pair (MLM parent, then contrastive children). acccn_cn: chance-normalised accuracy, averaged over question groups within a task and then over tasks. Seen: the 8 grounded training tasks. Held-out: PUBHEALTH, BIOSSES, MTSamples. Clinical: MedNLI, MIMIC-I trial questions and MIMIC-IV name gate, zero-shot except “+clinical train”. ECETS_TS: top-label ECE after temperature scaling on seen tasks. All rows are trained with the released RLCD (PG) recipe except “C trained with CE”. In the lower block the C columns hold the row’s own model. Knowledge-track results, raw ECE and all architectures per encoder are in Appendix G. Figure 2: Seen-task accuracy versus training data for the six parent-retriever comparisons (grey: MLM parent, blue: retrieval child; solid: C, dashed: PFR). Error bars show the sd over three seeds. 6.1 RQ1: Does retrieval training help? Retrieval improves zero-shot content matching. Without training, every contrastive checkpoint reaches 15–24 chance-normalised points on questions whose options differ in content (disease areas, study designs, specialties, rubrics, exam answers), against 0–7 for their MLM parents and 20.3 for Laya-large (Table 3). On options that differ only by a label word (yes/no, true/false), cosine gaps are of order 10−310^-3 and no encoder is reliably above chance. After fine-tuning, retrieval benefits depend on the head. With full data and the cross head the contrastive stage is roughly neutral: S-PubMedBERT and PubMedBERT reach 67.7 and 67.3, both MPNet checkpoints 66.4, and the BioClinical-ModernBERT pair 72.9 and 73.1 (Table 2); the only Holm-significant difference is negative (GTE-ModernBERT −2.5-2.5; Table 8). With PFR, which keeps the retrieval prior, the retriever is better in three of five pairs (all-MPNet +4.5+4.5, Nomic +2.7+2.7, GTE +2.9+2.9), worse for S-PubMedBERT (−1.6-1.6), and the ModernBERT embedders gain +10.9+10.9 on held-out tasks (Holm p≤.014p≤.014 for each). With less data the heads diverge further (Figure 2). For PFR, retrievers are ahead in four of five pairs at both 10% and 2% (significant in three and four; e.g. Nomic +5.3+5.3 and +9.2+9.2, GTE +4.7+4.7 and +7.4+7.4, and up to +16.3+16.3 on held-out tasks). For C, the ModernBERT-family retrievers are 2.5–3.7 points behind their parents at 10%, BioClinical-ModernBERT’s embedder is 8.5 points behind at 2%, and the only gain is S-PubMedBERT at 10% (+1.9+1.9). MedCPT-Query illustrates the dependence on the conversion: its [MASK] states are identical within an input, so the cross head reaches only 17.9, but PFR reaches 60.4. The strongest base-size model is BioClinical-ModernBERT with a cross head (73.1), close to fine-tuned Laya-large (75.0) at about a third of its size. Over all 15 pair and data-size comparisons (Table 8), retrieval training is significantly better in ten and worse in one with PFR, and better in one and worse in five with C. Z = PFR at step 0 C at step 0 Encoder content label content label PubMedBERT 1.0 13.0 1.1 24.0 S-PubMedBERT-MS-MARCO 15.2 −-20.4 2.3 −-2.8 MedCPT-Query 19.8 −-20.2 3.5 17.6 MPNet 1.0 3.6 −-1.2 12.5 all-MPNet-base-v2 21.2 −-1.2 −-1.2 12.1 ModernBERT-base 0.0 17.1 0.6 −-7.1 modernbert-embed-base 18.9 −-5.0 2.4 −-19.8 gte-modernbert-base 23.3 −-13.8 1.0 0.0 BioClinical-ModernBERT 6.5 −-15.1 2.6 26.5 BioClinical-MB-embeddings 18.6 5.9 5.0 −-23.1 BGE-base-v1.5 (unpaired) 23.9 11.8 1.1 14.5 Laya-large (Laya’s own head) 20.3 36.2 – – Table 3: Untrained (step-0) chance-normalised accuracy (×100× 100, after temperature scaling on the calibration split) on questions whose options differ in content (MEDLINE design and disease area, MTSamples, exam answers, Druglib and BIOSSES rubrics) or only by a label word (noul, PubMedQA, verification, DDI, MedNLI). C at step 0: 3 seeds. 6.2 RQ2: Which conversion works best? Seen acccn_cn Held-out acccn_cn ECETS_TS Objective PFR C PFR C PFR C CE 68.3 70.2 20.8 31.0 4.0 3.8 Proper 68.5 70.7 22.6 28.8 4.4 3.9 PG (released RLCD) 65.3 67.7 23.6 32.8 3.8 3.9 PG w/o CE 65.5 – 20.3 – 4.0 – PG-LOO (unbiased) 67.8 69.2 23.8 30.0 3.9 3.7 Reparam 68.5 69.8 21.9 25.8 3.7 3.6 Table 4: Head × objective on S-PubMedBERT-MS-MARCO (×100× 100, mean over 3 seeds; sd and further metrics in Table 7). PG is the released RLCD recipe, used for all other trained models. PG-LOO replaces its same-sample mean baseline and reward normalisation with a leave-one-out baseline. PG w/o CE was run for PFR only. The cross head achieves higher accuracy. In the matched grid (Table 4), C exceeds PFR by 1.3–2.4 seen-task points for all five shared objectives (Holm p≤.011p≤.011 for CE, Proper and PG; .08.08 and .14.14 for PG-LOO and Reparam) and by 3.8–10.2 points on held-out tasks. The bi-encoder B matches C on seen tasks (−0.6-0.6, n.s.) but loses 9.4 held-out points; its value is that state embeddings can be precomputed for corpus screening. A frozen prior improves PFR by 2.5 points. Prior fusion reduces option-order sensitivity. Prior fusion makes decisions almost invariant to option order: averaged over all evaluation splits, a random permutation changes 0.3% of S-PubMedBERT PFR answers versus 15.7% for C, whose flips concentrate on unseen inputs (held-out 17%, exams 28%, seen tasks 3%; Appendix H). PFR also has lower ordinal error on Druglib for every objective (Table 7). Fine-tuning reduces retrieval performance. SciFact nDCG@10 of the pooled embedding falls from 67.6 (Z) to 43.3 (C), 35.6 (PFR), 34.3 (B) and 24.6 for C trained with CE (Table 14 in Appendix K); only a frozen prior keeps the retriever intact. A ten-question request takes 6.6 ms for C, 15.2 ms for PFR and 22.7 ms for Laya-large (forward pass, batch 1, MI300X). 6.3 RQ3: Does RLCD improve accuracy or calibration? The released RLCD recipe trails cross-entropy. With either head, CE, Proper and Reparam are within one point of each other, while the released PG recipe trails CE by 3.0 (PFR) and 2.5 (C) points (Holm p=.005p=.005), with a fourfold larger seed sd for PFR (1.5 vs. 0.4). Adding the spherical and RPS terms does not significantly improve accuracy (Proper −- CE +0.2+0.2 and +0.5+0.5, n.s.), and removing CE from PG has little effect on accuracy (+0.2+0.2). An unbiased estimator reduces the accuracy deficit. PG-LOO, which differs from PG only in its baseline and in dropping the reward normalisation, gains 2.5 (PFR) and 1.5 (C) points over PG (Holm p=.005p=.005) and is within 0.8 points of Reparam (n.s.). A matched probe on fixed checkpoints and batches explains why (Appendix D): the released estimator points in the right direction (cosine 0.98 with the true smoothed gradient) but is 3.6–3.8, 5.7–6.1 and 14–15 times too large at σ=0.4σ=0.4, 0.25 and 0.1, so the score term outweighs CE by 5–20× instead of 1.3×. Since the global gradient norm exceeds the clipping threshold at every step for every objective, step length is fixed and the objectives differ only in direction; under PG that direction is dominated by a score-function term with a per-item signal-to-noise ratio below one. The pathwise estimator is 100–1,500 times less noisy at matched scale, yet PG-LOO trains nearly as well, so estimator variance is a secondary factor. No clear calibration benefit after scaling. PG has the lowest raw ECE (7.0 vs. 9.3 for CE with PFR), but after temperature scaling all objectives lie between 3.6 and 4.4 and PG has the worst NLL (Table 7, Figure 7). On the three held-out tasks the ordering is noisier and differs from seen tasks (e.g. C with Reparam is 7 points below C with PG), so we do not generalise the objective ranking beyond the training distribution. 6.4 LLMs and confidence-based routing Fine-tuned encoders are competitive on seen tasks: S-PubMedBERT C trained with CE (70.2) and Laya-large (75.0) exceed zero-shot Gemma-4-31B (65.7) and Qwen3.8-27B (57.4), with better calibration (ECE after scaling 3.8–4.3 vs. 8.5–9.1). The LLMs achieve higher accuracy on held-out tasks (Gemma 52.6 vs. 31.0), the clinical track (81.6–83.4 vs. at most 38.4 zero-shot, 55.2 with clinical training) and exams. The two are complementary (Figure 3): routing the 20% least confident questions to Gemma raises macro accuracy over seen and held-out tasks from 59.5 to 68.4 for S-PubMedBERT C (CE) and from 64.3 to 69.1 for Laya-large, against 62.1 for Gemma alone. This is a retrospective fixed budget on the test set, not a validated deployment threshold. Figure 3: Retrospective routing to Gemma-4-31B: the least-confident fraction of questions (by temperature-scaled confidence) is escalated. Lines: mean over three encoder seeds; bands: sd. 7 Discussion Retrieval benefits depend on the head. Retrieval training helps zero-shot and helps a prior-fused head, more so with scarce labels, but is close to neutral for a cross head with full data and can cost accuracy with few labels. Retrieval initialisation is therefore most beneficial when the conversion retains the retrieval prior (PFR), particularly with scarce labels; it offers no consistent accuracy gain for a cross head. Choosing a conversion. For per-request decisions, we recommend a cross head with cross-entropy (70.2 on S-PubMedBERT). BioClinical-ModernBERT with a cross head reaches 73.1 under PG. PFR is preferable when low option-order sensitivity is important or when the encoder’s mask-slot states are degenerate, and B when a corpus must be screened against many questions. Every trained conversion degrades the embedding as a retriever unless the prior is frozen. 8 Conclusion Biomedical sentence encoders trained for retrieval can be converted into System One models: with a cross head they reach 66–73 seen-task points with top-label ECE below 6 after temperature scaling (MedCPT-Query excepted), close to fine-tuned Laya-large (75.0) at about a third of its size. Whether retrieval training itself helps depends on the conversion: it helps PFR in 10 of 15 comparisons, most clearly with little labelled data, but does not consistently help C. Plain cross-entropy with temperature scaling matches or outperforms the released RLCD recipe, whose deficit comes from its reward normalisation; implementations of RLCD should use an independent baseline such as leave-one-out. Fast encoders and LLMs are complementary, so a calibrated System One model is a natural first stage that answers confident cases and escalates the rest. Limitations Scope. Results cover base-size encoders, one fixed training budget and hyperparameters inherited from Laya; objectives were not tuned separately, and the head × objective grid was run on one encoder. The six parent-retriever comparisons share parents and differ in retrieval corpora, so they are not independent replications. Intervals condition on the observed tasks and seed-averaged correctness; no mixed-effects or equivalence analysis is claimed. Data. MEDLINE-S1 labels are weak: indexing can omit tags and uses the full article. A few exact training inputs recur in calibration and test splits; we report the (small) sensitivity of test scores but did not retrain or refit temperatures on de-duplicated splits. Text-level identity cannot establish patient-level independence or exclude near-duplicates. Exam contamination of the LLMs is unmeasured. Calibration and shift. Low ECE on seen tasks does not imply calibration on new tasks: on held-out tasks several models are poorly calibrated, and option-order sensitivity of cross heads grows under shift. Thresholds should be validated on in-domain samples before use. Clinical use. The clinical tests are small, come from one hospital system and use surrogate names; long-note strategies remain close to chance for base encoders. Nothing here is a validated clinical device, and the released model is not trained on clinical notes. Ethics Statement Credentialed PhysioNet data (MedNLI, MIMIC-I and MIMIC-IV derivatives) were used under their data use agreements, processed only on institutional infrastructure, and never sent to a hosted model or API; only aggregate metrics leave that infrastructure, and examples in this paper are paraphrased. Public datasets are used under their published licences or terms of use; for them we release conversion code and per-record hashes rather than the data. For MEDLINE-S1 we release PMIDs, questions and labels under C BY 4.0 without the abstracts, which a script retrieves from PubMed. The released model is trained only on public data and inherits the non-commercial licence of S-PubMedBERT-MS-MARCO. Calibrated confidences can encourage automation; we recommend human review of escalated and low-confidence decisions in any health-related deployment. Acknowledgements We thank the creators and maintainers of the publicly released datasets, pretrained encoders, language-model checkpoints and open-source software used in this study, including Laya, Sentence-Transformers, PyTorch and Hugging Face Transformers. We also acknowledge the National Library of Medicine for PubMed and MeSH, and PhysioNet and the contributors to the clinical resources accessed under their respective data use agreements. Computational experiments used Kelvin2 GPU resources provided by Queen’s University Belfast through the Northern Ireland High Performance Computing (NI-HPC) service. References Baker et al. (2016) S. Baker, I. Silins, Y. Guo, I. Ali, J. Högberg, U. Stenius, and A. Korhonen Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics 32 (3), p. 432–440. External Links: Document Cited by: §4. Boteva et al. (2016) V. Boteva, D. Gholipour, A. Sokolov, and S. Riezler A full-text learning to rank dataset for medical information retrieval. In Advances in Information Retrieval (ECIR 2016), p. 716–722. External Links: Document Cited by: §5. Brier (1950) G. W. Brier Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. External Links: Document Cited by: §2. Chen et al. (2023) L. Chen, M. Zaharia, and J. Zou FrugalGPT: how to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. External Links: Link Cited by: Appendix A. Convai Innovations (2026) Convai Innovations Laya: a non-autoregressive system 1 decision engine. Note: https://github.com/NandhaKishorM/layaModel: https://huggingface.co/convaiinnovations/laya. Accessed 2026-09-29 Cited by: §1, §2. Decision Index contributors (2026) Decision Index contributors Decision index: a benchmark suite for typed decision engines, v0.2.1. Note: https://github.com/bhubbard/decision-indexAccessed 2026-09-29 Cited by: Appendix A, §2. Deka et al. (2022) P. Deka, A. Jurek-Loughrey, and P. Deepak Improved methods to aid unsupervised evidence-based fact checking for online health news. Journal of Data Intelligence 3 (4), p. 474–504. Cited by: §1, §2, §5. Desai and Durrett (2020) S. Desai and G. Durrett Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, p. 295–302. External Links: Document Cited by: §2. dos Santos (2026) J. A. dos Santos Calibrated decision models for autonomous penetration-testing harnesses: JEV and Laya as system one decision layers for LLM-driven pentest agents. External Links: 2609.28940 Cited by: Appendix A, §2. Epstein (1969) E. S. Epstein A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology 8 (6), p. 985–987. External Links: Document Cited by: §2. Gemma Team (2026) Gemma Team Gemma 4 technical report. External Links: 2607.02770, Link Cited by: §5. Gneiting and Raftery (2007) T. Gneiting and A. E. Raftery Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102 (477), p. 359–378. External Links: Document Cited by: §1, §2, §3.3. Gräßer et al. (2018) F. Gräßer, S. Kallumadi, H. Malberg, and S. Zaunseder Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning. In Proceedings of the 2018 International Conference on Digital Health, p. 121–125. External Links: Document Cited by: §4. Gu et al. (2021) Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare 3 (1), p. 1–23. External Links: Document Cited by: Appendix A, §1, §2, §5. Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, p. 1321–1330. External Links: Link Cited by: §2. Gurulingappa et al. (2012) H. Gurulingappa, A. M. Rajput, A. Roberts, J. Fluck, M. Hofmann-Apitius, and L. Toldo Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics 45 (5), p. 885–892. External Links: Document Cited by: §4. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, External Links: Link Cited by: §4. Herrero-Zazo et al. (2013) M. Herrero-Zazo, I. Segura-Bedmar, P. Martínez, and T. Declerck The DDI corpus: an annotated corpus with pharmacological substances and drug–drug interactions. Journal of Biomedical Informatics 46 (5), p. 914–920. External Links: Document Cited by: §4. Ibrahim and Zaki (2026) H. Ibrahim and Y. Zaki Evaluating decision models for text annotation in computational social science. External Links: 2609.24574 Cited by: Appendix A, §2. Jin et al. (2021) D. Jin, E. Pan, N. Oufattole, W. Weng, H. Fang, and P. Szolovits What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences 11 (14), p. 6421. External Links: Document Cited by: Appendix A, §4. Jin et al. (2019) Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu PubMedQA: a dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, p. 2567–2577. External Links: Document Cited by: Appendix A, §4. Jin et al. (2023) Q. Jin, W. Kim, Q. Chen, D. C. Comeau, L. Yeganova, W. J. Wilbur, and Z. Lu MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics 39 (11), p. btad651. External Links: Document Cited by: §1, §2, §5. Johnson et al. (2023) A. E. W. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, L. H. Lehman, L. A. Celi, and R. G. Mark MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data 10, p. 1. External Links: Document Cited by: §4. Johnson et al. (2016) A. E. W. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark MIMIC-I, a freely accessible critical care database. Scientific Data 3, p. 160035. External Links: Document Cited by: §4. Kotonya and Toni (2020) N. Kotonya and F. Toni Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, p. 7740–7754. External Links: Document Cited by: §4. Lee et al. (2020) J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36 (4), p. 1234–1240. External Links: Document Cited by: §1. Li et al. (2023) Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281. Cited by: §5. Lim et al. (2023) S. Lim, Y. Xiao, A. Johnson, D. Moukheiber, L. Moukheiber, M. Moukheiber, M. Ghassemi, and T. Pollard Annotated MIMIC-IV discharge summaries for a study on deidentification of names (version 1.0). Note: PhysioNet External Links: Document Cited by: §4. Marosi (2026) M. Marosi Decider: one-pass typed decisions with calibrated probabilities. Note: https://github.com/Mapika/deciderAccessed 2026-09-29 Cited by: Appendix A, §2. NeuML (2025) NeuML BioClinical modernbert embeddings model card. Note: https://huggingface.co/NeuML/bioclinical-modernbert-base-embeddingsModel identifier used in the experiment configuration Cited by: §5. Nguyen et al. (2016) T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng MS MARCO: a human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), CEUR Workshop Proceedings, Vol. 1773. External Links: Link Cited by: §1, §2. Nogueira and Cho (2019) R. Nogueira and K. Cho Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085. External Links: Link Cited by: §2. Nussbaum et al. (2024) Z. Nussbaum, J. X. Morris, B. Duderstadt, and A. Mulyar Nomic Embed: training a reproducible long context text embedder. External Links: 2402.01613, Link Cited by: §5. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, p. 248–260. External Links: Link Cited by: Appendix A, §4. Pollard et al. (2026) T. Pollard, B. E. Moody, L. Lehman, B. Gow, C. Fernandes, C. Xie, A. Johnson, R. G. Mark, and T. Heldt PhysioNet as a global platform for biomedical research. Nature Health. External Links: Document Cited by: §4. Qwen Team (2026) Qwen Team Qwen3.8-Max: a new bar for coding and cowork. External Links: Link Cited by: §5. Rafe and Das (2026) A. Rafe and S. Das Calibrated decisions at scale: converting police crash narratives into probabilistic crash variables with a system one model (Jev). External Links: 2609.24052 Cited by: Appendix A, §2. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, p. 3982–3992. External Links: Document Cited by: §2. Romanov and Shivade (2018) A. Romanov and C. Shivade Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 1586–1596. External Links: Document Cited by: §4. Sarrouti et al. (2021) M. Sarrouti, A. Ben Abacha, Y. M’rabet, and D. Demner-Fushman Evidence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, p. 3499–3512. External Links: Document Cited by: §4. Sentence Transformers (2021) Sentence Transformers All-mpnet-base-v2 model card. Note: https://huggingface.co/sentence-transformers/all-mpnet-base-v2Model identifier used in the experiment configuration Cited by: §5. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §2. Shivade (2019) C. Shivade MedNLI – a natural language inference dataset for the clinical domain (version 1.0.0). Note: PhysioNet. RRID:SCR_007345 External Links: Document Cited by: §4. Soğancıoğlu et al. (2017) G. Soğancıoğlu, H. Öztürk, and A. Özgür BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics 33 (14), p. i49–i58. External Links: Document Cited by: §4. Song et al. (2020) K. Song, X. Tan, T. Qin, J. Lu, and T. Liu MPNet: masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems, Vol. 33. External Links: Link Cited by: §5. Sounack et al. (2025) T. Sounack, J. Davis, B. Durieux, A. Chaffin, T. J. Pollard, E. Lehman, A. E. W. Johnson, M. McDermott, T. Naumann, and C. Lindvall BioClinical ModernBERT: a state-of-the-art long-context encoder for biomedical and clinical NLP. External Links: 2506.10896, Link Cited by: §5. Thakur et al. (2021) N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych BEIR: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: §5. TypeSafe AI (2026) TypeSafe AI Introducing system one models and Jev. Note: https://typesafe.ai/blog/introducing-system-one-models-and-jevAccessed 2026-09-29 Cited by: §1, §2. Wadden et al. (2020) D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi Fact or fiction: verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, p. 7534–7550. External Links: Document Cited by: §4. Warner et al. (2025) B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, N. Cooper, G. Adams, J. Howard, and I. Poli Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), External Links: Document Cited by: §2, §5. Woo et al. (2025) E. Woo, M. C. Burkhart, E. Alsentzer, and B. Beaulieu-Jones MIMIC-I-Ext-Synthetic-Clinical-Trial-Questions (version 1.0.0). Note: PhysioNet. RRID:SCR_007345 External Links: Document Cited by: §4. Woo et al. (2024) E. G. Woo, M. C. Burkhart, E. Alsentzer, and B. K. Beaulieu-Jones Synthetic data distillation enables the extraction of clinical information at scale. medRxiv. External Links: Document Cited by: §4. Xiao et al. (2024) S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J. Nie C-pack: packed resources for general Chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 641–649. Cited by: §5. Xu (2026) Z. Xu JevOut: natural context can flip decision models. External Links: 2609.30243 Cited by: Appendix A, §2. Yilmaz et al. (2026) F. Yilmaz, H. A. Tasdemir, and M. F. Gozay LAVOIR: teaching a single-pass decision encoder when and what to ask with amortized value of information. External Links: 2609.30706 Cited by: Appendix A, §2. Yin et al. (2019) W. Yin, J. Hay, and D. Roth Benchmarking zero-shot text classification: datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, p. 3914–3923. External Links: Document Cited by: §2. Zhang et al. (2024) X. Zhang, Y. Zhang, D. Long, W. Xie, Z. Dai, J. Tang, H. Lin, B. Yang, P. Xie, F. Huang, et al. mGTE: generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 1393–1412. Cited by: §5. Appendix Appendix A Extended Related Work System One models. Besides Laya, Decider (Marosi, 2026) reads option-letter logits from Qwen3.5 decoders fine-tuned with cross-entropy and fitted temperatures. The Decision Index (Decision Index contributors, 2026) scores typed-decision engines on tasks in five areas; its only clinical task is trial-report inference. Concurrent evaluations use Jev and Laya as decision layers for penetration-testing agents (dos Santos, 2026), for text annotation in computational social science (Ibrahim and Zaki, 2026) and for coding crash narratives (Rafe and Das, 2026); they study decision flips under natural distracting context (Xu, 2026) and teach a Laya encoder which missing information to ask for (Yilmaz et al., 2026). Biomedical benchmarks and cascades. BLURB (Gu et al., 2021) evaluates task-specific heads, not runtime schemas or calibration. Biomedical QA benchmarks (Jin et al., 2019; Jin et al., 2021; Pal et al., 2022) are dominated by generative LLMs. Routing easy inputs to a cheap model and hard ones to an expensive one is the idea behind LLM cascades (Chen et al., 2023); a calibrated System One model is a natural first stage (§6.4). Appendix B Task Conversions Task State Question(s) and type Options and target Splits PubMedQA-L abstract contexts without the conclusion the original question (choice) yes / no / maybe with short descriptions; one-hot 1,000 labelled items: 500 test, 50 dev, 50 calibration, rest train (seeded) PubMedQA-A abstract contexts the original question (noul) yes → true; one-hot 20k sampled for training, excluding every evaluation PMID SciFact title and abstract of the evidence (or first cited) document “Does this abstract support or refute the claim: c” (choice) supports / refutes / not_enough_info official train; validation as test; calibration grouped by claim HealthVer evidence passage “Does this evidence support or refute the claim: c” (choice) as SciFact official train / dev / test; calibration grouped by topic PUBHEALTH article text (fact-checkers’ explanation removed) “Based on this article, what is the veracity of the health claim: c” (choice) false / mixture / true / unproven official test only DDI-2013 sentence with the two drugs marked “What kind of interaction between @DRUG1$ and @DRUG2$ does the sentence describe?” (choice) mechanism / effect / advice / int / none official dev as test (test is gated); test sentences seen in training removed HoC abstract (sentences joined) “Does this abstract discuss the cancer hallmark ‘h’?”, one noul per hallmark sentence labels aggregated per abstract official train / dev / test ADE sentence “Does this sentence describe an adverse effect caused by a drug?” (noul) one-hot deduplicated; seeded 80/10/10 Druglib drug, condition, and the benefit, side-effect and comment reviews effectiveness, side-effect severity and overall rating (three score) 5 levels each; the 1–10 rating binned in pairs official train / test BIOSSES first sentence “How similar in meaning is this sentence to: s2” (score) 5 levels; mass split between the two levels adjacent to the annotator mean all 100 pairs as test MTSamples transcription “Which medical specialty does this transcription belong to?” (choice) every specialty in the test set, without descriptions test only MedQA, MedMCQA, MMLU-med. question stem “Which option correctly answers the question in the text?” (choice) answer texts as descriptions official splits; MedMCQA single-answer, 20k sampled for training MEDLINE-S1 title and abstract design and disease area (choice); humans, animals, adverse effects, drug therapy, two age or sex groups (noul) from publication types and MeSH; 3–7 random disease distractors disjoint baseline files; evaluation PMIDs excluded Table 5: Conversion of each public source dataset into typed decisions. Every training set loses a calibration slice of min(400,max(10,⌊n/10⌋)) (400, (10, n/10 )) states, carved before any run and used only for temperature fitting. Clinical conversions are listed in the text below. Table 5 lists the public conversions. The clinical tasks are converted in the same way: MedNLI asks “How does this hypothesis relate to the clinical note? Hypothesis: …with options entailment, neutral, contradiction; the trial questions are posed verbatim with yes, no, not_stated, plus the noul “Can this question be answered from the information in the note: …; and the name gate asks “Does this passage mention a person’s name (a patient, relative or clinician)? over windows of MIMIC-IV notes. "task": "medline_s1", "split": "test", "state": "<title and abstract of a PubMed citation>", "questions": "design": "type": "choice", "instructions": "What type of study or article is this?", "criteria": "randomized_trial": "a randomized controlled trial", "case_report": "a case report of one or a few patients", "...": "(four more designs)", "humans": "type": "noul", "instructions": "Did this research study human beings (patients or participants)?", "gold": "design": "probabilities": "randomized_trial": 1.0, "humans": "probabilities": "true": 1.0 Figure 4: A MEDLINE-S1 record in the shared request format (abridged). Gold labels are distributions over option keys. A score question lists its levels from low to high, and its keys are level indices. Appendix C RLCD: Formal Statement and Proof Definitions. Let t∈ΔK−1t∈ ^K-1 be the target and p∈ΔK−1p∈ ^K-1 a reported distribution. Laya’s reward is S(p,t)=∑ktklogpk+wsph⟨t,p⟩∥p∥2−wrps[τ=score]RPS(p,t), splitS(p,t)=& _kt_k p_k+w_sph t,p p _2\\ &-w_rps\,I[τ= score]\,RPS(p,t), split (7) with RPS(p,t)=1K−1∑k(∑j≤k(pj−tj))2RPS(p,t)= 1K-1 _k ( _j≤ k(p_j-t_j) )^2, wsph=0.75w_sph=0.75, wrps=1w_rps=1, and the log term floored at logδ δ, δ>0δ>0. Each step draws G=4G=4 perturbations ε(g)∼(0,σ2ΠK) ^(g) (0,σ^2 _K), where ΠK=IK−1K⊤ _K=I_K- 1K11 restricts noise to directions that change the softmax. Each perturbed report softmax(z+ε(g))softmax(z+ ^(g)) receives reward R(g)=S(⋅,t)R^(g)=S(·,t), and RLCD minimises ℒPG=−1G∑gA(g)logπz(z+ε(g))+λCE(z,t), splitL_PG=&- 1G _gA^(g) _z (z+ ^(g) )\\ &+λ\,CE(z,t), split (8) where πz=(z,σ2ΠK) _z=N(z,σ^2 _K), λ=1λ=1 (λ=0λ=0 for PG w/o CE), and the advantage is A(g)=(R(g)−R¯)/dA^(g)=(R^(g)- R)/d. Since ∇zlogπz(z+ε)=ε/σ2 _z _z(z+ )= /σ^2 on the centred subspace, the reward part of −∇zℒPG- _zL_PG is g g in Eq. (6). Proposition 1. Let Jσ(z)=ε∼(0,σ2ΠK)[S(softmax(z+ε),t)]J_σ(z)=E_ (0,σ^2 _K) [S(softmax(z+ ),t) ]. Then (i) ∇zJσ=σ−2[(S(softmax(z+ε),t)−b)ε] _zJ_σ=σ^-2\,E[(S(softmax(z+ ),t)-b)\, ] for any b independent of ε (score-function form); (i) ∇zJσ=[∇zS(softmax(z+ε),t)] _zJ_σ=E [ _zS(softmax(z+ ),t) ] (pathwise form); (i) Jσ(z)→S(softmax(z),t)J_σ(z)→ S(softmax(z),t) as σ→0σ→ 0, which equals −CE(z,t)-CE(z,t) for wsph=wrps=0w_sph=w_rps=0 wherever the floor is inactive; (iv) with the same-sample mean R¯ R as baseline and no normalisation, the score-function estimate has expectation G−1G∇zJσ G-1G _zJ_σ. RLCD therefore performs stochastic ascent on a noise-smoothed proper score. PG-LOO uses b−g=1G−1∑h≠gR(h)b_-g= 1G-1 _h≠ gR^(h) and no normalisation, which by (i) is unbiased and on the same scale as the pathwise estimator (i) used by Reparam. Proof. Let V=v:⊤v=0V=\v:1 v=0\. Softmax is invariant to shifts along 1, so we may centre z onto V; ε∼(0,σ2ΠK) (0,σ^2 _K) is an isotropic Gaussian on V with density φσ _σ. Write f(y)=S(softmax(y),t)f(y)=S(softmax(y),t). With the log term floored at logδ δ, the spherical term in [0,1][0,1] and RPS in [0,1][0,1], f is bounded and Lipschitz, hence differentiable almost everywhere. (i) Jσ(z)=∫Vf(y)φσ(y−z)yJ_σ(z)= _Vf(y) _σ(y-z)\,dy; differentiating under the integral (justified by boundedness of f and integrability of ∇φσ∇ _σ) gives, for directions in V, ∇Jσ(z) ∇ J_σ(z) =∫Vf(y)y−zσ2φσ(y−z)y = _Vf(y)\, y-zσ^2\, _σ(y-z)\,dy =σ−2[f(z+ε)ε]. =σ^-2\,E[f(z+ )\, ]. Any b independent of ε satisfies [bε]=[b][ε]=0E[b\, ]=E[b]\,E[ ]=0. (i) f is Lipschitz, so ∇f∇ f exists almost everywhere and is bounded; dominated convergence gives ∇[f(z+ε)]=[∇f(z+ε)] [f(z+ )]=E[∇ f(z+ )]. (i) f is continuous and bounded and ε→0 → 0 in distribution, so Jσ(z)→f(z)J_σ(z)→ f(z). With wsph=wrps=0w_sph=w_rps=0 and an inactive floor, f(z)=∑ktklogsoftmax(z)k=−CE(z,t)f(z)= _kt_k (z)_k=-CE(z,t). (iv) For G independent draws with Rg=f(z+εg)R_g=f(z+ _g) and R¯=G−1∑hRh R=G^-1 _hR_h, [(Rg−R¯)εg] [(R_g- R)\, _g ] =(1−1G)[Rgεg] = (1- 1G )E[R_g _g] −1G∑h≠g[Rh][εg], - 1G _h≠ gE[R_h]\,E[ _g], and the last term vanishes, so the averaged estimate has expectation G−1G∇Jσ G-1G∇ J_σ. With the leave-one-out baseline b−g=(G−1)−1∑h≠gRhb_-g=(G-1)^-1 _h≠ gR_h, which is independent of εg _g, the expectation is exactly ∇Jσ∇ J_σ by (i). □ Normalisation. The released recipe further divides all advantages by the standard deviation d of the centred rewards in the minibatch. Because d depends on the same draws, the factor cannot be taken outside the expectation; more importantly, the centred rewards and hence d shrink roughly in proportion to σ, while the estimate carries a factor ε/σ2 /σ^2, so the effective scale of the score term grows roughly as 1/σ1/σ as σ is annealed. Since the CE term enters on a fixed scale, the normalisation changes the relative weight of the two terms during training; Appendix D measures this directly. Appendix D Gradient-Estimator Probe and Optimisation Traces Slope vs. ∇Jσ∇ J_σ Score : CE weight Cosine SNR Checkpoint (head, objective) Estimator σ=0.4σ=0.4 0.250.25 0.10.1 σ=0.4σ=0.4 0.10.1 σ=0.25σ=0.25 σ=0.25σ=0.25 PFR, PG released PG 3.59 5.72 14.27 4.74 18.87 0.982 0.861 mean baseline, unnormalised 0.75 0.75 0.75 0.99 1.00 0.982 0.816 PG-LOO 1.01 1.00 1.00 1.33 1.33 0.983 0.791 pathwise (Reparam) 1.00 1.00 1.00 1.32 1.32 1.000 206 PFR, PG-LOO released PG 3.66 5.84 14.57 4.83 19.23 0.982 0.854 mean baseline, unnormalised 0.75 0.75 0.75 0.99 0.99 0.982 0.805 PG-LOO 1.01 1.00 1.00 1.32 1.32 0.983 0.792 pathwise (Reparam) 1.00 1.00 1.00 1.32 1.32 1.000 209 C, CE released PG 3.80 6.05 15.09 5.02 19.97 0.982 0.847 mean baseline, unnormalised 0.75 0.75 0.75 0.99 0.99 0.982 0.804 PG-LOO 1.01 1.01 1.01 1.33 1.33 0.984 0.781 pathwise (Reparam) 1.00 1.00 1.00 1.32 1.32 1.000 216 Table 6: Matched gradient-estimator probe. For each saved checkpoint, the logits of the same 40 training batches (32 items each) are fixed, and 32 independent estimates of the smoothed-score gradient (G=4G=4, no CE term) are drawn per estimator and noise level. Slope: projection of the mean estimate onto a 2,048-sample pathwise reference ∇Jσ∇ J_σ (1 is the correct scale). Score : CE weight: slope ×∥∇Jσ∥/∥∇CE∥× ∇ J_σ / , the effective weight of the smoothed-score term relative to the CE term in a training step. SNR: per-item ∥∇Jσ∥2 ∇ J_σ ^2 divided by the scale-matched variance of one estimate. Figure 5: Matched probe, averaged over the three checkpoints of Table 6. Left: scale of each estimator relative to the true smoothed-score gradient (dotted line: correct scale). Right: per-item signal-to-noise ratio after matching scales. The released estimator’s scale grows as σ anneals; all score-function estimators have SNR below one, the pathwise estimator two to three orders of magnitude more. Figure 6: Median global gradient norm before clipping during training (S-PubMedBERT-MS-MARCO, three seeds, rolling median). The clipping threshold is 1 (dotted), so every step of every objective is clipped and update length is fixed; the released PG recipe’s norm is an order of magnitude above CE and grows as σ anneals. The probe (Table 6, Figure 5) holds the model, batch and noise level fixed and compares estimators of the gradient of the smoothed score with respect to the logits. Three results are consistent across a PFR checkpoint trained with PG, one trained with PG-LOO, and a C checkpoint trained with CE. First, the same-sample mean baseline yields a slope of exactly 0.75=(G−1)/G0.75=(G-1)/G, confirming Proposition 1(iv); PG-LOO and the pathwise estimator have slope 1. Second, the released normalisation multiplies the estimate by 3.6–3.8 at σ=0.4σ=0.4 and by 14–15 at σ=0.1σ=0.1, so relative to cross-entropy the score term weighs 4.7–5.0 and then 19–20 times as much, against 1.3 for the unbiased estimators. Third, at matched scale all score-function estimators have per-item SNR around 0.8, whereas the pathwise estimator’s SNR is 85–1,300. Figure 6 shows the consequence during training: every objective is clipped at every step, and under PG the clipped direction is dominated by the inflated, noisy score term. Because PG-LOO trains almost as well as Reparam despite its far higher variance, we attribute the released recipe’s deficit primarily to this reweighting. Appendix E Complete Head × Objective Results Objective Head Seen Held-out Clinical Knowledge ECE ECETS_TS NLLTS_TS MAEscore_ score Perm. flip ∥∇∥ ∇ CE PFR 68.3± 0.4 20.8± 1.6 21.9± 2.9 13.1± 0.8 9.3± 0.9 4.0± 0.4 0.510± 0.004 0.588± 0.012 0.4± 0.1 7.1 C 70.2± 0.4 31.0± 1.7 20.0± 4.1 13.6± 0.9 7.4± 0.3 3.8± 0.2 0.491± 0.001 0.639± 0.020 2.9± 0.4 6.0 Proper PFR 68.5± 0.2 22.6± 1.9 23.4± 0.7 13.6± 0.4 9.1± 0.6 4.4± 0.4 0.510± 0.006 0.582± 0.004 0.2± 0.1 8.7 C 70.7± 0.3 28.8± 0.4 22.6± 1.1 12.8± 0.7 7.6± 0.5 3.9± 0.3 0.493± 0.013 0.627± 0.007 2.6± 0.2 7.7 PG (released RLCD) PFR 65.3± 1.5 23.6± 1.5 18.3± 8.2 11.4± 1.2 7.0± 0.6 3.8± 0.2 0.548± 0.018 0.638± 0.016 0.4± 0.1 88.1 C 67.7± 0.9 32.8± 1.5 20.5± 1.8 11.2± 0.7 6.5± 0.0 3.9± 0.7 0.527± 0.010 0.746± 0.031 3.2± 0.5 50.2 PG w/o CE PFR 65.5± 2.0 20.3± 3.3 16.9± 11.0 10.6± 0.9 6.6± 0.4 4.0± 0.4 0.543± 0.014 0.646± 0.019 0.2± 0.2 76.2 C not run PG-LOO (unbiased) PFR 67.8± 0.6 23.8± 5.7 25.2± 3.5 12.4± 0.2 7.5± 0.5 3.9± 0.2 0.518± 0.009 0.605± 0.014 0.4± 0.1 17.6 C 69.2± 0.6 30.0± 2.1 20.5± 0.9 11.7± 0.3 6.6± 0.4 3.7± 0.5 0.509± 0.005 0.665± 0.001 3.3± 0.8 14.7 Reparam PFR 68.5± 0.5 21.9± 2.0 20.5± 5.5 9.4± 6.2 9.0± 0.8 3.7± 0.4 0.508± 0.005 0.587± 0.009 0.4± 0.2 16.0 C 69.8± 1.6 25.8± 1.5 20.4± 1.2 11.4± 0.4 7.2± 0.6 3.6± 0.4 0.498± 0.021 0.668± 0.040 2.4± 0.1 14.4 Table 7: Complete head × objective grid on S-PubMedBERT-MS-MARCO (mean± over 3 seeds; accuracy, ECE and flip rates ×100× 100). Columns 3–6: chance-normalised accuracy by track. ECE, NLL: seen tasks, top-label, before and after temperature scaling. MAEscore_ score: error of the expected Druglib level. Perm. flip: share of seen-task answers that change under a random option permutation. ∥∇∥ ∇ : median global gradient norm before clipping; the clipping threshold is 1, so every step of every objective is clipped. Figure 7: Top-label reliability diagrams on the seen tasks (items pooled over tasks and seeds; 10 bins, bins with fewer than 30 items omitted), before (dashed) and after (solid) temperature scaling. Pooled ECE differs from the task-averaged ECE in the tables. Fine-tuned encoders are over-confident before scaling and close to the diagonal after it; the zero-shot LLM remains miscalibrated. Table 7 reports every metric of the matched grid. Three patterns complement §6.3. (1) The head effect is consistent: C has higher seen and held-out accuracy and lower NLL for every objective, PFR lower ordinal error and option-order sensitivity. (2) Raw ECE is lowest for the score-function variants (PG, PG w/o CE, PG-LOO), not for Reparam, which optimises the same smoothed objective; temperature scaling removes the difference (Figure 7). (3) The pre-clip gradient norm orders the objectives consistently (CE << Proper << PG-LOO ≈ Reparam ≪ PG). Appendix F Metrics and Statistics Accuracy and top-label ECE use the argmax target; ordinal MAE uses the expected predicted level against it. BIOSSES has soft targets, so hard-label summaries discard part of its annotation. ECE uses 15 equal-width bins; saved metrics also include equal-mass and class-wise ECE, Brier score, AURC, AUROC, AUPRC and ordinal metrics. Paired bootstrap intervals use 10,000 draws of document clusters with replacement within each task: text is whitespace-normalised and hashed, and all records and questions that share a text are resampled together. Task weights stay fixed, and correctness is averaged over seeds before resampling, so intervals omit seed uncertainty and are pointwise. Two-sided p values use 2min(Pr[Δ∗≤0],Pr[Δ∗≥0])2 ( [ ^*≤ 0], [ ^*≥ 0]) with a +1+1 correction (smallest attainable value 2/10,0012/10,001). Holm correction is applied within four families (RQ1: 60 tests; RQ2: 14; RQ3: 26; baselines: 8), across both tracks. Tables 8 and 9 list every contrast. Contrast (a−ba-b) Seeds Seen Δ [95% CI] pHolmp_Holm Held-out Δ [95% CI] pHolmp_Holm C full: S-PubMedBERT −- PubMedBERT 3/3 +0.4 [−-0.4, +1.1] 1 +3.2 [+0.3, +6.1] .956 C full: all-MPNet −- MPNet 3/3 +0.0 [−-0.8, +0.9] 1 −-2.6 [−-5.6, +0.3] 1 C full: Nomic-embed −- ModernBERT 3/3 −-1.0 [−-1.9, −-0.1] .708 +2.5 [−-1.2, +6.2] 1 C full: GTE-ModernBERT −- ModernBERT 3/3 −-2.5 [−-3.5, −-1.5] .012 −-2.7 [−-5.9, +0.6] 1 C full: BioClinical-MB-emb −- BioClinical-MB 3/3 −-0.2 [−-1.0, +0.5] 1 +0.3 [−-3.2, +3.7] 1 PFR full: S-PubMedBERT −- PubMedBERT 3/3 −-1.6 [−-2.5, −-0.8] .014 +2.9 [−-0.1, +6.1] 1 PFR full: all-MPNet −- MPNet 3/3 +4.5 [+3.5, +5.5] .012 +3.9 [+0.1, +7.7] 1 PFR full: Nomic-embed −- ModernBERT 3/3 +2.7 [+1.7, +3.7] .012 +10.9 [+6.7, +15.2] .012 PFR full: GTE-ModernBERT −- ModernBERT 3/3 +2.9 [+1.9, +3.9] .012 +10.9 [+6.4, +15.3] .012 PFR full: BioClinical-MB-emb −- BioClinical-MB 3/3 +0.8 [−-0.1, +1.7] 1 +1.5 [−-1.4, +4.5] 1 C 2%: S-PubMedBERT −- PubMedBERT 3/3 −-0.2 [−-1.1, +0.7] 1 +0.5 [−-1.9, +2.8] 1 C 2%: all-MPNet −- MPNet 3/3 −-0.8 [−-1.8, +0.2] 1 +12.3 [+9.7, +14.9] .012 C 2%: Nomic-embed −- ModernBERT 3/3 −-1.0 [−-2.0, 0.0] 1 +3.1 [+0.3, +5.8] .918 C 2%: GTE-ModernBERT −- ModernBERT 3/3 +1.5 [+0.5, +2.5] .112 −-4.3 [−-7.5, −-1.2] .243 C 2%: BioClinical-MB-emb −- BioClinical-MB 3/3 −-8.5 [−-9.7, −-7.4] .012 +2.3 [−-0.9, +5.5] 1 PFR 2%: S-PubMedBERT −- PubMedBERT 3/3 −-0.2 [−-1.2, +0.7] 1 +7.6 [+4.1, +11.1] .012 PFR 2%: all-MPNet −- MPNet 3/3 +2.6 [+1.5, +3.7] .012 +1.1 [−-2.4, +4.5] 1 PFR 2%: Nomic-embed −- ModernBERT 3/3 +9.2 [+8.1, +10.4] .012 +15.6 [+12.2, +19.1] .012 PFR 2%: GTE-ModernBERT −- ModernBERT 3/3 +7.4 [+6.2, +8.6] .012 +15.5 [+11.4, +19.7] .012 PFR 2%: BioClinical-MB-emb −- BioClinical-MB 3/3 +4.0 [+2.8, +5.2] .012 +0.6 [−-2.9, +4.2] 1 C 10%: S-PubMedBERT −- PubMedBERT 3/3 +1.9 [+0.9, +2.8] .012 −-2.6 [−-5.0, −-0.4] .654 C 10%: all-MPNet −- MPNet 3/3 −-1.0 [−-2.0, +0.0] 1 −-3.5 [−-6.7, −-0.3] .956 C 10%: Nomic-embed −- ModernBERT 3/3 −-2.5 [−-3.5, −-1.5] .012 +6.6 [+3.1, +10.0] .012 C 10%: GTE-ModernBERT −- ModernBERT 3/3 −-3.1 [−-4.2, −-2.1] .012 +3.0 [−-0.2, +6.2] 1 C 10%: BioClinical-MB-emb −- BioClinical-MB 3/3 −-3.7 [−-4.6, −-2.7] .012 +1.5 [−-1.8, +5.0] 1 PFR 10%: S-PubMedBERT −- PubMedBERT 3/3 −-0.2 [−-1.0, +0.6] 1 +0.1 [−-3.2, +3.4] 1 PFR 10%: all-MPNet −- MPNet 3/3 +7.7 [+6.6, +8.8] .012 +6.3 [+3.3, +9.4] .012 PFR 10%: Nomic-embed −- ModernBERT 3/3 +5.3 [+4.2, +6.4] .012 +16.3 [+12.5, +20.1] .012 PFR 10%: GTE-ModernBERT −- ModernBERT 3/3 +4.7 [+3.6, +5.9] .012 +13.5 [+9.4, +17.8] .012 PFR 10%: BioClinical-MB-emb −- BioClinical-MB 3/3 +1.2 [+0.2, +2.2] .558 −-3.1 [−-6.6, +0.5] 1 Table 8: Initialisation contrasts (contrastive child minus MLM parent) for every available pair, head and training-data fraction. Document-clustered paired bootstrap (10,000 draws; differences in chance-normalised points). Intervals are pointwise and condition on the observed tasks and on seed-averaged correctness. Holm adjustment is applied within each family (RQ1; RQ2; RQ3; baselines) across both tracks. The smallest attainable unadjusted p is 0.0002. Contrast (a−ba-b) Seeds Seen Δ [95% CI] pHolmp_Holm Held-out Δ [95% CI] pHolmp_Holm C −- PFR (CE) 3/3 +1.9 [+0.8, +3.1] .011 +10.2 [+5.8, +14.6] .003 C −- PFR (Proper) 3/3 +2.2 [+1.1, +3.4] .004 +6.2 [+2.0, +10.5] .041 C −- PFR (PG) 3/3 +2.4 [+1.2, +3.6] .003 +9.2 [+5.0, +13.4] .003 C −- PFR (LOO) 3/3 +1.5 [+0.3, +2.6] .079 +6.2 [+2.0, +10.3] .041 C −- PFR (Reparam) 3/3 +1.3 [+0.1, +2.4] .136 +3.8 [−-0.6, +8.2] .247 B −- C (PG) 3/3 −-0.6 [−-1.8, +0.7] .353 −-9.3 [−-13.5, −-5.0] .003 PFR frozen −- shared prior (PG) 3/3 +2.5 [+1.3, +3.6] .003 +3.2 [−-0.5, +7.0] .247 PFR: CE −- PG 3/3 +3.0 [+2.2, +3.8] .005 −-2.8 [−-5.7, +0.2] .735 PFR: Reparam −- PG 3/3 +3.2 [+2.5, +4.0] .005 −-1.7 [−-4.7, +1.5] 1 PFR: LOO −- PG 3/3 +2.5 [+1.7, +3.2] .005 +0.2 [−-2.9, +3.3] 1 PFR: LOO −- Reparam 3/3 −-0.8 [−-1.4, −-0.1] .322 +1.9 [−-1.0, +4.9] 1 PFR: LOO −- CE 3/3 −-0.5 [−-1.1, +0.1] .929 +3.0 [+0.5, +5.5] .259 PFR: Proper −- CE 3/3 +0.2 [−-0.3, +0.7] 1 +1.8 [−-0.7, +4.4] 1 PFR: PG w/o CE −- PG 3/3 +0.2 [−-0.5, +0.9] 1 −-3.3 [−-5.8, −-0.9] .139 C: CE −- PG 3/3 +2.5 [+1.8, +3.2] .005 −-1.8 [−-3.8, +0.4] .929 C: Reparam −- PG 3/3 +2.1 [+1.4, +2.8] .005 −-7.0 [−-10.3, −-3.8] .005 C: LOO −- PG 3/3 +1.5 [+0.9, +2.2] .005 −-2.8 [−-5.1, −-0.5] .259 C: LOO −- Reparam 3/3 −-0.6 [−-1.2, +0.0] .725 +4.2 [+1.9, +6.6] .015 C: LOO −- CE 3/3 −-1.0 [−-1.6, −-0.3] .043 −-1.0 [−-3.1, +1.1] 1 C: Proper −- CE 3/3 +0.5 [0.0, +1.1] .735 −-2.2 [−-4.5, +0.0] .725 Laya-large FT −- S-PubMedBERT C (PG) 3/3 +7.3 [+6.1, +8.5] .002 +3.0 [−-0.5, +6.5] .094 Laya-large FT −- S-PubMedBERT C (CE) 3/3 +4.8 [+3.7, +5.9] .002 +4.8 [+1.2, +8.4] .021 Gemma-4-31B −- Laya-large FT 1/3 −-9.3 [−-10.7, −-7.8] .002 +16.8 [+10.4, +23.1] .002 Gemma-4-31B −- S-PubMedBERT C (CE) 1/3 −-4.5 [−-6.0, −-2.9] .002 +21.6 [+16.2, +26.9] .002 Table 9: Conversion (RQ2), objective (RQ3) and baseline contrasts on S-PubMedBERT-MS-MARCO. Document-clustered paired bootstrap (10,000 draws; differences in chance-normalised points). Intervals are pointwise and condition on the observed tasks and on seed-averaged correctness. Holm adjustment is applied within each family (RQ1; RQ2; RQ3; baselines) across both tracks. The smallest attainable unadjusted p is 0.0002. Appendix G Full Results by Encoder and Task Encoder Arch Seen acccn_cn ECE ECETS_TS Held-out Clinical Knowledge PubMedBERT C 67.3± 0.7 6.7± 0.6 4.8± 0.9 29.5± 3.3 20.7± 0.8 13.4± 0.7 PFR 66.9± 0.8 8.3± 0.4 4.1± 0.3 20.7± 0.6 23.9± 0.6 12.5± 0.9 B 67.0± 0.5 8.3± 0.4 4.1± 0.1 17.8± 6.5 23.6± 1.4 11.5± 0.7 S-PubMedBERT-MS-MARCO C 67.7± 0.9 6.5± 0.0 3.9± 0.7 32.8± 1.5 20.5± 1.8 11.2± 0.7 PFR 65.3± 1.5 7.0± 0.6 3.8± 0.2 23.6± 1.5 18.3± 8.2 11.4± 1.2 B 67.1± 0.5 7.8± 0.6 3.9± 0.2 23.4± 1.4 25.9± 2.1 11.8± 0.7 MedCPT-Query C 17.9± 4.3 14.0± 2.3 14.0± 2.3 3.9± 0.5 12.5± 3.4 2.8± 0.6 PFR 60.4± 0.7 7.1± 0.2 4.8± 0.3 23.6± 3.2 12.6± 3.7 10.1± 0.7 MPNet C 66.4± 0.9 8.2± 0.4 4.8± 0.4 30.2± 2.7 22.7± 5.2 4.6± 1.0 PFR 63.3± 0.4 7.6± 0.2 4.3± 0.4 20.8± 2.5 16.9± 1.6 5.7± 0.6 all-MPNet-base-v2 C 66.4± 0.9 8.0± 0.3 5.2± 0.3 27.5± 0.8 13.7± 5.0 3.3± 0.7 PFR 67.8± 0.8 8.1± 0.3 4.0± 0.2 24.7± 2.3 18.4± 2.9 7.2± 0.4 ModernBERT-base C 71.6± 0.7 8.0± 0.5 4.3± 0.2 32.5± 1.4 26.1± 2.5 7.1± 0.8 PFR 66.5± 0.9 8.5± 0.5 4.4± 0.1 21.4± 1.8 19.9± 5.1 6.5± 0.4 modernbert-embed-base C 70.6± 1.0 8.9± 0.3 4.9± 0.2 35.0± 1.1 13.4± 2.9 7.0± 1.2 PFR 69.2± 0.6 8.3± 0.3 4.1± 0.2 32.3± 4.1 12.4± 2.3 8.2± 0.2 gte-modernbert-base C 69.1± 0.8 8.7± 0.5 4.5± 0.1 29.8± 1.7 24.3± 1.8 8.2± 1.3 PFR 69.3± 0.2 8.7± 0.3 4.0± 0.5 32.3± 0.2 23.2± 2.1 10.4± 0.5 BioClinical-ModernBERT C 73.1± 0.5 8.0± 0.5 4.1± 0.2 31.8± 2.0 32.2± 4.1 10.0± 0.7 PFR 67.5± 0.5 8.2± 0.3 4.2± 0.3 21.0± 3.6 17.3± 7.1 8.0± 1.1 BioClinical-MB-embeddings C 72.9± 0.7 8.5± 0.5 4.6± 0.5 32.1± 4.8 26.5± 1.2 8.4± 2.3 PFR 68.3± 0.0 8.3± 0.1 4.3± 0.4 22.5± 1.7 6.2± 4.8 7.8± 0.1 BGE-base-v1.5 (unpaired) C 60.6± 0.5 7.8± 0.2 5.7± 0.7 16.1± 3.2 7.8± 15.5 3.0± 1.2 PFR 66.8± 0.3 8.3± 0.3 4.2± 0.1 30.7± 3.0 17.7± 2.0 9.3± 0.4 Table 10: All encoders and architectures (×100× 100; mean± over 3 seeds). Seen (grounded training tasks) Held-out Model PubMedQA SciFact HealthVer DDI HoC ADE Druglib MEDLINE PUBHEALTH BIOSSES MTSamples PubMedBERT C (PG) 51.6 51.2 48.8 89.3 88.1 87.3 35.6 86.5 23.1 0.4 65.2 S-PubMedBERT C (PG) 47.6 55.3 49.1 91.0 88.7 88.0 35.4 86.4 24.2 5.4 68.7 S-PubMedBERT C (CE) 47.7 60.6 55.0 92.4 89.1 88.1 41.6 87.0 20.5 1.7 70.9 S-PubMedBERT PFR (PG) 44.9 34.9 52.9 91.5 85.8 86.9 42.8 82.7 17.0 2.5 51.3 S-PubMedBERT PFR (CE) 46.9 39.9 61.9 93.3 87.3 87.3 45.7 83.8 8.4 2.5 51.5 MedCPT-Query PFR (PG) 33.1 33.8 47.9 83.5 85.5 83.2 32.6 83.4 11.4 5.4 54.0 all-MPNet PFR (PG) 39.9 40.0 65.0 93.2 86.2 85.6 47.9 84.9 25.7 −-0.4 48.9 BioClinical-ModernBERT C (PG) 48.5 66.0 64.6 93.9 88.9 87.7 48.6 86.6 19.6 13.7 62.1 Laya-large C (PG) 50.3 71.8 69.3 94.0 89.5 88.0 51.3 85.6 26.6 25.0 55.7 Qwen3.8-27B (zero-shot) 56.8 77.1 37.5 17.9 82.7 60.3 50.2 76.2 53.3 10.0 75.4 Gemma-4-31B (zero-shot) 62.8 80.4 47.5 67.4 79.4 60.0 51.0 77.2 54.3 27.5 76.0 Table 11: Per-task chance-normalised accuracy (×100× 100; mean over seeds; question groups averaged within a task). The training objective is given in parentheses. Table 10 lists every encoder with each trained architecture, including raw ECE and the knowledge track, on which no base-size encoder exceeds 15 chance-normalised points. Table 11 breaks the seen and held-out tracks down by task. The cross head’s advantage over PFR is concentrated on tasks that require comparing two texts in detail (SciFact: 55–61 vs. 35–40) and on held-out specialty routing (MTSamples: 69–71 vs. 51 for S-PubMedBERT); on single-text classification (DDI, HoC, ADE, MEDLINE) the two heads are within a few points. Appendix H Robustness and Long Notes Perm. Trial Qs: answer (acccn_cn) Trial Qs: answerable (acccn_cn) Name gate AUROC Model flip % trunc win-mean win-conf RtD trunc win-mean win-conf RtD test common-word PubMedBERT C 16.6 2.4 3.4 2.3 5.9 4.5 −-3.7 −-2.8 7.9 72.9 71.8 S-PubMedBERT C 15.7 0.0 −-3.8 0.1 −-1.1 7.1 1.0 4.0 20.2 72.6 71.4 S-PubMedBERT PFR 0.3 8.2 15.0 16.0 7.1 4.7 −-0.6 −-1.5 5.8 74.8 74.5 S-PubMedBERT PFR, frozen 13.1 −-0.5 −-0.2 2.9 0.4 7.8 0.4 2.3 29.9 75.0 74.1 S-PubMedBERT B 0.0 10.9 19.7 20.5 8.3 9.8 4.2 4.7 3.8 77.6 77.7 all-MPNet C 13.8 8.5 12.4 11.8 8.7 4.0 −-5.9 −-5.3 5.2 72.1 71.1 all-MPNet PFR 0.2 −-2.1 0.5 0.7 0.8 5.1 0.7 2.8 4.6 61.3 62.3 S-PubMedBERT PFR, +clinical 0.3 11.1 17.8 15.8 16.0 −-1.3 −-2.9 −-2.9 6.9 99.8 98.5 Laya-large FT 9.8 10.9 14.9 20.8 8.7 6.9 −-0.8 0.6 6.0 84.9 83.4 Laya-large zero-shot 18.9 3.6 −-1.8 −-0.1 0.5 5.0 −-3.8 −-0.6 −-2.6 84.0 83.2 Table 12: Robustness (×100× 100; mean over 3 seeds where available). Perm. flip: share of answers that change under a random option permutation (5 permutations), averaged over all evaluation splits; on the seen tasks alone C heads flip 2–3% and PFR below 0.5% (Table 7). Trial Qs: MIMIC-I trial-eligibility questions over notes longer than the 512-token context, decided by truncation, sliding windows (mean of logits or most confident window) or retrieve-then-decide (RtD). Name gate: MIMIC-IV windows, standard and common-word-name splits. Both LLMs reach AUROC 100.0 on both name-gate splits. Option order. Averaged over all evaluation splits, cross heads change 10–17% of their answers under a random permutation of the options, including Laya-large; the flips concentrate on held-out, exam and clinical inputs (S-PubMedBERT C: 3% on seen tasks, 17% held-out, 28% on exams). B and PFR with a shared prior are essentially invariant, because the shared prior scores each option independently of its position; with a frozen prior, PFR flips 13%, which we did not investigate further. Long notes. The MIMIC-I trial questions are small (532 answer and 1,000 answerability items) and no strategy consistently performs best. For the three-way answer, sliding windows help most models: Laya-large rises from 10.9 to 20.8 with the most confident window and S-PubMedBERT B from 10.9 to 20.5. For answerability, retrieve-then-decide is best for S-PubMedBERT C (20.2) and its frozen-prior PFR (29.9), but for most other models all strategies are close to chance. Long-note decisions remain challenging for the evaluated base encoders. Privacy gate. Names that are also common words have similar AUROC to ordinary names (the two columns differ by at most two points). Training on the gate’s own windows (+clinical) raises AUROC from about 75 to 99.8; both LLMs are perfect zero-shot. Appendix I Data Integrity and Sensitivity An input identity comprises the whitespace-normalised state and question schema, excluding labels. Nine DDI and seven Druglib calibration records, and two SciFact and 13 Druglib test records, have inputs that occur in training. Repeated inputs also occur within some test sets (HealthVer, DDI, Druglib, MTSamples, MMLU-medical). Identical evidence paired with different claims (HealthVer, SciFact) is task-native and is not counted as duplication. Table 13 re-scores the original checkpoints after excluding every affected test input; it cannot remove the influence of duplicated calibration items, which would require refitting temperatures. Model Historical Filtered Difference S-PubMedBERT C 67.69 67.32 −-0.37 S-PubMedBERT PFR (PG) 65.30 64.91 −-0.40 S-PubMedBERT PFR (CE) 68.26 67.89 −-0.37 S-PubMedBERT PFR (Reparam) 68.53 68.19 −-0.35 BioClinical-ModernBERT C 73.10 72.81 −-0.29 Laya-large FT 74.99 74.69 −-0.30 Gemma-4-31B 65.72 65.38 −-0.34 Table 13: Seen-task accuracy sensitivity to removal of exact test inputs appearing in public training/calibration data, and repeated test inputs. Seed means, in chance-normalised points. Checkpoints and temperatures are unchanged; this does not repair calibration contamination. Appendix J LLM Prompt The prompt is rendered with each model’s chat template, with thinking disabled: Read the text and answer the question. Text: state Question: instructions Options: A. option 1 B. option 2 … Respond with the letter of the correct option only. Option probabilities are the softmax over the letters’ next-token log-probabilities; for each letter the bare and the space-prefixed token are combined by log-sum-exp. States longer than 6,000 tokens are truncated. nDCG@10 (×100× 100) Latency Model (S-PubMedBERT) SciFact NFCorpus ms Z (no training) 67.6 31.3 10.2 B 34.3 11.0 9.7 C 43.3 13.0 6.6 C, trained with CE 24.6 10.3 6.6 PFR, shared prior 35.6 11.2 15.2 PFR, frozen prior 67.6 31.3 15.2 Laya-large C (421M) – – 22.7 Table 14: Retrieval retention of the pooled embedding after conversion (seed 0), and p50 latency of one request with 10 questions at batch size 1 on an MI300X (base-encoder latencies measured on the PubMedBERT architecture, which S-PubMedBERT shares). Times cover the model forward pass only, excluding tokenisation and host/device transfers. Appendix K Hyperparameters and Compute • Sequence limits: 512 tokens per cross sequence, of which at most 192 are question and option tokens (48 per option); bi-encoder states 256 tokens, hypotheses 64. • Optimisation: AdamW; learning rates 2.5×10−52.5×10^-5 (encoder), 10−410^-4 (head), 5×10−25×10^-2 (α); weight decay 0.01 (none on α); 5% warm-up then cosine decay; global gradient-norm clipping at 1; bf16 autocast over fp32 weights. • Objectives: G=4G=4; σ annealed linearly from 0.4 to 0.1; δ=10−4δ=10^-4; wsph=0.75w_sph=0.75, wrps=1w_rps=1, λ=1λ=1 (0 for PG w/o CE). • Temperatures: fitted by L-BFGS on logT T, clipped to [0.05,20][0.05,20]; buckets with fewer than 10 calibration items fall back to the per-type value. • Hardware and packing: AMD MI300X (192 GB) GPUs; a shared file queue packs up to six runs per GPU, each capped at a declared memory budget (22 GB for base encoders, 44 GB for Laya-large, 100–110 GB for the LLMs). Runs checkpoint every 20 minutes and on pre-emption and resume exactly, since the data schedule is a function of seed and step. • Throughput (single run, batch 32): C 265 items/s (11.7 GB); PFR 223 items/s (16.3 GB); B 326 items/s (6.4 GB); Laya-large C 99 items/s (36.3 GB). • Compute: the initial campaign (125 runs including both LLM baselines, plus 8 smoke tests) took 11.9 wall-clock hours on three GPUs (about 36 GPU-hours; 170 task-hours under packing). The head × objective controls and gradient probe (16 tasks) took 1.3 wall-clock hours on three GPUs (14.6 task-hours). The 50 additional learning-curve runs for the ModernBERT and BioClinical pairs took 4.2 wall-clock hours on four GPUs (72 task-hours). The ModernBERT/BGE full-data extension and the GTE reruns used further allocations that we did not account separately. Appendix L Released Software and Model The code release contains the four conversions, the five objectives (including PG-LOO), temperature fitting, the long-state strategies, the data builders, a script that restores the MEDLINE-S1 abstracts from PubMed, and the evaluation and bootstrap code used here. The released model is the best seed (by seen-task accuracy) of S-PubMedBERT-MS-MARCO with a cross head trained with CE on public data (seen 70.5; three-seed mean 70.2±0.470.2± 0.4). Inference is a single file of plain PyTorch (sbert2s1.py) that ships with the model. The Hugging Face example below downloads and loads this file directly; dependencies are PyTorch, Transformers, Safetensors and Hugging Face Hub: import importlib.util as iu from huggingface_hub import hf_hub_download repo = "pritamdeka/S1-PubMedBERT" path = hf_hub_download(repo, "sbert2s1.py") spec = iu.spec_from_file_location("s1", path) s1 = iu.module_from_spec(spec) spec.loader.exec_module(s1) m = s1.load(repo) abstract = "Participants were randomised." q = "design": "type": "choice", "instructions": "Study design?", "criteria": "rct": "randomised trial", "cohort": "cohort study" m.predict(abstract, q) Answers follow the Jev/Laya format with the fitted temperatures applied, and truncated inputs are marked. The checkpoint is also exported in Laya’s layout. The model inherits the C BY-NC 2.0 licence of its base encoder; the code is released under the Apache 2.0 licence. Code: https://github.com/pritamdeka/sbert2s1; model: https://huggingface.co/pritamdeka/S1-PubMedBERT; data: https://huggingface.co/datasets/pritamdeka/MEDLINE-S1.