Paper deep dive
What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects
Bowen Xu, Boyu Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Harness search keeps a change to the prompts, reasoning switches, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison, or a defect in the evaluation. We preregistered a study of where these gains come from, with three small models, three benchmarks, replication and test partitions, a GEPA search arm and six evaluation defects injected one at a time, and we report all 47 primary endpoints. Turning thinking off raised accuracy over a capped thinking setting in 5 of 9 model-benchmark cells, and in each the gain came mostly from questions where the capped setting gave no readable answer. The thinking-off setting was not meaningfully worse than a rescue configuration or four GEPA-selected harnesses in 11 of 13 comparisons, and lost to the rescue on GSM8K for two models. GEPA repaired its broken starting points, but none of its selected harnesses was more accurate than the thinking-off setting. A thinking budget in the serving engine, which also allows a longer answer, lowered truncation and raised the parse rate in 6 of 9 cells. In 6 of 15 evaluable defect-model pairs, replication through the same pipeline reproduced the defect's distortion instead of revealing it. On the LongevityBench multiple-choice tasks, only the longevity-tuned model beat the strongest constant-label baseline.
Tags
Links
- Source: https://arxiv.org/abs/2610.05533v1
- Canonical: https://arxiv.org/abs/2610.05533v1
Trouble viewing inline? Open PDF directly â
Full Text
168,846 characters extracted from source content.
Expand or collapse full text
What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects Bowen Xu* bxu666@stanford.eduDepartment of Biology, Stanford University, Stanford, CA, USA Boyu Chen*Institute of Health Informatics, University College London, London, UK *Equal contribution. Corresponding author Abstract Harness search keeps a change to the prompts, reasoning switches, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison, or a defect in the evaluation. We preregistered a study of where these gains come from, with three small models, three benchmarks, replication and test partitions, a GEPA search arm and six evaluation defects injected one at a time, and we report all 47 primary endpoints. Turning thinking off raised accuracy over a capped thinking setting in 5 of 9 model-benchmark cells, and in each the gain came mostly from questions where the capped setting gave no readable answer. The thinking-off setting was not meaningfully worse than a rescue configuration or four GEPA-selected harnesses in 11 of 13 comparisons, and lost to the rescue on GSM8K for two models. GEPA repaired its broken starting points, but none of its selected harnesses was more accurate than the thinking-off setting. A thinking budget in the serving engine, which also allows a longer answer, lowered truncation and raised the parse rate in 6 of 9 cells. In 6 of 15 evaluable defect-model pairs, replication through the same pipeline reproduced the defectâs distortion instead of revealing it. On the LongevityBench multiple-choice tasks, only the longevity-tuned model beat the strongest constant-label baseline. 1 Introduction A harness is everything around a frozen model that changes what it is scored on: the prompts, reasoning switches, token budgets, retry rules and parsers. Harness search proposes a change and keeps it if it wins (Agrawal et al., 2025; Khattab et al., 2023; Wu et al., 2026; Lee et al., 2026b). A gain that replicates on untouched rows shows that the selected harness beat its parent by more than selection noise, and it leaves three questions open. The first is visibility versus quality: did the answers get better, or did answers that the parent never produced in a readable form, because it ran out of tokens or never committed to one, become visible to the parser? The second is baseline adequacy: would a simple setting that the search never tried do as well, and does the model beat trivial baselines at all? The third is evaluation validity: are the inputs, the gold labels, the split and the served model what they are said to be? Each question has a literature (Section 2), and most of it works after the fact. Harness and scaffold gains vanish against matched-budget test-time scaling and simple baselines (Wang et al., 2026b; Gideoni et al., 2026; Liang et al., 2026; Kapoor et al., 2024), a gain can be completion rather than correctness (Seyoum & Mittur, 2026; Lee, 2026; Chen et al., 2026), thinking off can beat a thinking cap (Nie et al., 2026), and evaluation pipelines carry defects that move scores by tens of points (Zhu et al., 2025; Berriche et al., 2026). We found no study that fixes in advance, for a stated contrast, how much of a harness gain is visibility, none that tests the benchmark authorsâ own protocol against searched harnesses with a margin, and none that injects known defects one at a time to ask whether replication reproduces them. The search covered 2024-01 to 2026-10-03 and was capped at 200 web searches, and three of its six parts had to be resumed after a rate limit, so these statements are bounded (Appendix H). Background. This study follows a case study of harness search around a frozen longevity language model (Appendix A). Its search scored every candidate through a caller that sent each benchmark rowâs reference answer to the model as an earlier assistant turn, and the rescue it selected replicated on a second shard through the same caller, so replication could not detect the defect: every call carried it. Trivial baselines, input audits and per-call prompt-token reconciliation did. After the caller was corrected, a preregistered re-evaluation found that the searchâs starting configuration parsed none of its 556 prospective rows, and that a thinking-off setting the search had never tried was non-inferior to the rescue at less than half the token cost. This paper asks whether that one model, one benchmark and one defect is a pattern. This paper. We preregistered a study of what a harness repairs and froze it on 30 September 2026 at 06:53:11 UTC. It uses three small models, three benchmark pools, a replication partition and a confirmatory test partition, an adaptive arm in which GEPA (Agrawal et al., 2025) searches for harnesses, and six evaluation defects that we inject one at a time. We split the gain of a thinking-off protocol over a capped thinking configuration exactly into rows only the protocol parses, rows both parse, and rows only the capped configuration parses, the split that Chen et al. (2026) use for self-revision, and we fix a dominance statistic and a gain gate in advance. We test the thinking-off protocol for non-inferiority against a derived rescue and four GEPA-selected harnesses, with trivial baselines next to every accuracy, and we test whether the serving stackâs thinking budget, which also allows a longer answer, lowers truncation. For each defect and model we fix one claim in advance, that replication through the same pipeline confirms the defectâs distortion, and we measure the false alarms of five audit checks. All 47 primary endpoints, in four families, are reported with every deviation and addendum, and everything computed after the outputs existed is labelled. The results cover three small models, three pools and six defects, and nothing beyond them. We do not claim that harness gains can be illusory, that thinking off can beat a thinking cap, or that agent and harness evaluations can be preregistered, since each has been shown or done before, and we do not claim that harness search finds genuine quality gains. 2 Related work We organise related work by the questions of Section 1; Appendix H states how it was searched and what the search could not cover. Harness search, and whether its gains are real. Prompt optimisers (Zhou et al., 2023; Yang et al., 2024; Khattab et al., 2023; Agrawal et al., 2025) and harness search (Lee et al., 2026b; Wu et al., 2026; Zhang et al., 2025) treat the text and control flow around fixed weights as the object of search, and GEPA is the adaptive arm of our study. Earlier work has shown, after the fact, that such gains can be illusory: evolved harnesses and scaffolds do not beat matched-budget test-time scaling or simple baselines (Wang et al., 2026b; Gideoni et al., 2026; Liang et al., 2026; Kapoor et al., 2024), a hardware-verification harness gains mostly completion (Seyoum & Mittur, 2026), a search started from a defective seed fails (Liu et al., 2026), and self-improving loops tamper with or hack their evaluators (Wang et al., 2026a; Yang et al., 2026a). Our question is narrower: where the gain of one protocol comes from, and whether a known defect would be seen, with the questions and decision rules fixed in advance. Visibility and quality (H1). Chen et al. (2026) split the accuracy change of a self-revision step into a content margin, a format-recovery margin, a format-loss margin and a dead cell, which is our split under other names but without a predefined dominance threshold. Nie et al. (2026) and Kaiser et al. (2026) give the identity for a single configuration or between models, completion rate times conditional accuracy; under a shared budget thinking off beats capped thinking (Nie et al., 2026; Ma et al., 2025), and capped single-call evaluation of hybrid models has been called an artifact of protocol (Lee et al., 2026a). Other work finds that unparsed answers hide correct ones (Garware & Zisad, 2026), separates traces cut by a cap from traces that finished without an answer (Goyal & Ray, 2026), reads a harness-evolution gain informally as completion rather than correctness on the twelve tasks that had been searched (Seyoum & Mittur, 2026), and counts repaired abstentions for constrained decoding (Lee, 2026). We add a pre-specified dominance statistic and gain gate, applied prospectively to the contrast between a thinking-off protocol and a capped thinking configuration. Baseline adequacy (H2). Several studies find, after the fact, that simple baselines match searched or evolved harnesses and complex agents (Kapoor et al., 2024; Wang et al., 2026b; Gideoni et al., 2026; Liang et al., 2026; Chang & Chen, 2026). A preregistered non-inferiority test of a simple alternative exists for agent memory (Sharma & Lall, 2026), and equivalence margins have been used for scaffolds (Gringras, 2026) and for compression (Singh, 2026). Our test fixes the margin in advance for the benchmark authorsâ own protocol, against a derived rescue and four GEPA-selected harnesses. Budget forcing (H3). Budget forcing and the Qwen3 thinking budget (Muennighoff et al., 2025; Yang et al., 2025) serve as evaluation controls, with a forced answer or a forced end of thinking (Nie et al., 2026; Carlon et al., 2026; Lee et al., 2026a; Chang & Chen, 2026), and one study forces the answer delimiter at a cap in vLLM (Goyal & Ray, 2026). We test the engineâs thinking-token budget (Kwon et al., 2023) as a harness setting against a capped configuration; the technique is not ours. Defects and prospective replication (H4). Other work injects faults with known ground truth and measures false alarms: tool bugs in agent benchmarks (Bellibatlu et al., 2026), tampering edits (Wang et al., 2026a), reward-hacking channels (Yang et al., 2026a) and planted gains (Shawn, 2026). Defects like ours have been catalogued as they occurred, for answer-key access, label defaults, parser fallbacks, stop sequences, denominators and gold errors (Wahi, 2026; Berriche et al., 2026; Zhu et al., 2025), and the reference scorer of MMLU-Pro credits a random option for an empty prediction.11 1 evaluate_from_local.py, function save_res, in the repository TIGER-AI-Lab/MMLU-Pro, read on 2026-10-03. We found no study that injects pipeline defects one at a time into a harness-search pipeline and tests, for each defect and model, whether replication through the same pipeline reproduces the distortion. Served-model identity and determinism. Serving stacks change outputs at fixed weights (Pape et al., 2026; Wang, 2026; Yuan et al., 2025; He & Thinking Machines Lab, 2025), answers can flip while accuracy stays flat (Dutta et al., 2024; KĂźbler et al., 2026), and a served route is not its model identifier (Stenstrom et al., 2026; Gao et al., 2025). vLLM used to decide tying from the configuration alone and silently discarded a stored head; its fix (hmellor (2026), vLLM project) is in the version we used (vLLM 0.30.0), which loads the stored head. We add a measured instance of the mismatch, since the mechanism is known. GEPA from a broken start. Liu et al. (2026) report that GEPA, started from its official GSM8K seed (which puts the answer before the reasoning), falls from 23.81% to 13.50% and never repairs the order, and Ursekar et al. (2026) start optimisers from deliberately untuned seeds, one of them a non-functional stub. Ours is a controlled comparison of clean and broken starts; why a search repairs one defect and not another is not tested here. Preregistration and prospective design. Preregistered or frozen designs with deviation logs, holdouts or equivalence margins appear in Gringras (2026); Weinberger & Hozez (2026); Sharma & Lall (2026); Lee (2026); Goyal & Ray (2026); Liang et al. (2026); Bellibatlu et al. (2026); Singh (2026). Thomas et al. (2026) confirm on the first eligible model released after the plan, a stronger standard than ours, since our freeze guards the rules and not model-specific tuning. We found no earlier study that combines the freeze with its hash and time in the commit, a pilot, replication then test, an adaptive arm before replication, all 47 endpoints reported, addenda that state what had been seen, and an exposure log. 3 Design The design is the frozen preregistration (hash and time in Appendix E), and this section states only what the results need. 3.1 Setting Models and stack. We study Qwen3-1.7B-Longevity (Zhavoronkov et al., 2026), a Qwen3 (Yang et al., 2025) model tuned on aging-biology data, SmolLM3-3B (Bakouch et al., 2025) and Intern-S1-mini (Bai et al., 2025), on vLLM 0.30.0 (Kwon et al., 2023) in batch-invariant mode (He & Thinking Machines Lab, 2025) at zero temperature and client concurrency 32, on one RTX 5090. The Qwen3-1.7B-Longevity checkpoint declares tied embeddings but stores a different output matrix, and the study model uses the stored one (untied); the tied variant that the case study served returns only in the FreeToken (Yang et al., 2026b) continuity runs for the tied-output-layer defect. Two models also have a descriptive seeded-sampling arm. Pools, cells and partitions. The LongevityBench letter pool combines, by row, LB-0030 (age band, five options), LB-0034 (which of two participants is older) and LB-0042 (ten-year survival), and LB-0038, an age regression, is reported apart. MMLU-Pro (Wang et al., 2024) (a stratified subset of 2,000 items) and GSM8K (Cobbe et al., 2021) are the other two pools, and a cell is a model with a pool, so there are nine cells. The items are split into search, replication and test partitions (40/20/40); LongevityBenchâs 14,720 rows, in 2,104 participant components, are split by component so that no participant spans partitions, and GEPA searches on the search partition only. The analysis rows leave out 147 replication and 371 test rows that the case study had held out and later called, and 7 replication rows whose reference answers could be recovered from an upload made by mistake; putting them back is a sensitivity analysis. The test cells then have 3,942 letter rows (789 participant components, the bootstrap unit), 799 MMLU-Pro items and 527 GSM8K items, and the replication cells have 1,990, 399 and 264. 3.2 Configurations and scoring Table 1 lists the six configurations of the grid and the GEPA harnesses. A call is unparsed if the scorer cannot read an answer from it, and it then scores as wrong, so an unparsed answer may hide a correct answer, a wrong one or none. A call is truncated if it ended at the token limit (finish reason length), whatever the scorer makes of it, and non-committed if it ended by itself (finish reason stop) and is unparsed; for RESCUE-1500 both properties are those of the call that gives the answer. For H1, let P be the baseline configuration and R the comparison, paired on the n rows of a cell, with u the parse indicator and c the correct indicator of a call and a bar negating an indicator. Row i is in V if only R parses it (uÂŻP,iâuR,i u_P,iu_R,i), in B if both parse it (uP,iâuR,iu_P,iu_R,i) and in L if only P parses it (uP,iâuÂŻR,iu_P,i u_R,i). With Î=âi(cR,iâcP,i)/n = _i(c_R,i-c_P,i)/n and the contributions ÎV=âiViâcR,i/n _V= _iV_ic_R,i/n (P scores zero on V), ÎB=âiBiâ(cR,iâcP,i)/n _B= _iB_i(c_R,i-c_P,i)/n and ÎL=ââiLicP,i/n _L=- _iL_ic_P,i/n, the identity Î=ÎV+ÎB+ÎL = _V+ _B+ _L is exact, because a row that neither configuration parses adds nothing. Each contribution is divided by the cellâs n and not by the size of its row set, as in the tables, so ÎV _V is Râs number of correct answers on V per row of the cell. Chen et al. (2026) use this additive split for self-revision (content, format-recovery and format-loss margins); the dominance statistic, its fixed share and the gain gate are ours. The dominance statistic T=ÎVâ0.8â(ÎV+ÎB)T= _V-0.8\,( _V+ _B) is positive when visibility supplies more than 0.8 of what the rows R parses contribute. Table 1: Configurations (âtokensâ is max_tokens; thinking on or off is set in every request). RESCUE-1500 is derived from the two grid runs and verified against the real two-call sequence on 300 items per model. Configuration Request THINK-64, THINK-512, THINK-1536 thinking on; 64, 512 or 1,536 tokens for thinking and answer together (THINK-64 is the case studyâs parent) AUTHORS-1500 thinking off; 1,500 tokens (the benchmark authorsâ serving protocol) BUDGET-512-1500 thinking on; vLLM ends thinking after 512 tokens (Muennighoff et al., 2025); up to 1,500 tokens of answer follow, so 2,012 tokens in all against 512 for THINK-512 RESCUE-1500 THINK-512; where its call ended at the token limit, the AUTHORS-1500 call on the same row (the call that gives the answer decides the properties of the row) GEPA harnesses four runs (clean or broken start, two seeds), 6,000 metric calls on 300 search rows, Intern-S1-mini as proposer; the selected harness edits the system prompt and appends an instruction, so the arm searches prompt text only (Appendix J) 3.3 Hypotheses and endpoints Table 2 states the four hypotheses. Their 47 primary endpoints are the only confirmatory results, and their verdicts come from percentile-bootstrap intervals adjusted within each family. Table 2: Hypotheses, estimands and verdict rules. m is the minimal difference and δ the non-inferiority margin, both 0.05. Every bound in a verdict is a bound of the family-adjusted interval, and H1âs gate uses the same resamples and the same adjusted level as T. Claim and estimand Verdict Endpoints (family) H1 Most of the accuracy that AUTHORS-1500 gains over THINK-512 comes from rows THINK-512 leaves unparsed. Gate: Î ; statistic: T. no gain: not tested unless the lower bound of Î is above zero; if it is, visibility dominates if the lower bound of T is above zero, quality matters if the upper bound of T is below zero, inconclusive otherwise. 9: one per cell H2 AUTHORS-1500 is not meaningfully worse than a comparator: θ=accâĄ(AUTHORS-1500)âaccâĄ(comparator)θ=acc( AUTHORS-1500)-acc(comparator). holds if the lower bound of θ>âδθ>-δ; fails if the upper bound <âδ<-δ; else inconclusive. 13: RESCUE-1500 in nine cells; four GEPA harnesses (Qwen3-1.7B-Longevity, letters) H3 Against THINK-512, BUDGET-512-1500 (a forced end of thinking and a larger allowance) lowers truncation (DTD_T) and raises parsing (DPD_P). holds if both lower bounds >m>m; fails if either upper bound <m<m; no room if the upper bound of THINK-512âs truncation is <m<m (applied first); else inconclusive. 9: one per cell H4 For each defect and model, prospective replication confirms the defectâs distortion (Section 3.4). holds, fails or inconclusive from three intervals; not evaluable if the defect cannot move the start by more than m. 16 pairs, fixed 3.4 Defects, claims and checks (H4) Six defects arose in the case study, and we inject each into a harness-search pipeline, one at a time, and compare the result with the clean pipeline on the same rows. The pipelineâs start s is THINK-1536, and the reported quantity is the startâs accuracy on the letter pool, measured on a selection sample S (the replication partition) and on a check sample R (the test partition); for the broken start it is the gain over the start. Let DSD_S and DRD_R be a defectâs distortion of the reported quantity on S and on R, and Î=DRâDS =D_R-D_S. The reference-answer defect (A) sends the rowâs reference answer as an assistant turn, on 1,204 replication rows halved by component into S and R. The gold fallback (B1) takes LB-0042âs gold from a historical fallback that, with this releaseâs labels, differs from the derived gold on about half of its rows, and the unparsed-as-A scoring (B2) scores an unparsed answer as the letter A. The skewed split (C) re-implements the case studyâs split, with S and R the halves of its validation part. The broken start (D) starts from THINK-64 and reports the gain of the selected configuration over the start; the selection does not depend on the start, so it cancels, and the distortion is DS=accSâ(THINK-1536)âaccSâ(THINK-64)D_S=acc_S( THINK-1536)-acc_S( THINK-64), with the same difference on R for DRD_R. The tied output layer (E) serves the tied layer in place of the untied one (Qwen3-1.7B-Longevity only), as an engine that decides tying from the configuration alone does (hmellor (2026), vLLM project). Appendix I gives each defectâs construction, clean counterpart, samples and evaluability bound, and each checkâs predicate and threshold. The claim for a defect and a model holds if DSD_S and DRD_R lie wholly outside ÂąmÂą m with the same sign and Î lies wholly inside ÂąmÂą m; it fails if any of the three lies wholly on the other side of its boundary, and it is inconclusive otherwise. A pair is evaluable only if the largest distortion its mechanism can cause on S exceeds m, a bound that we compute from parse, gold-change and answer-difference indicators without accuracy; the skewed split changes the sample and has no row-level bound, so it is evaluable by rule. The family stays at 16 whatever is evaluable. Five checks read the defectsâ signatures: per-call prompt-token reconciliation (TR), trivial baselines (TB), the simple configuration family (SCF), input reading (IR) and the gain and level rules of prospective replication (PR), plus served-model identity (SMI) for the tied output layer. Their detection matrix is largely true by construction and descriptive, and what is empirical is their false alarms on clean pipelines. 3.5 Statistics and what was fixed in advance Every contrast is paired on rows. In each of B=50,000B=50,000 bootstrap resamples we draw the cellâs units with replacement (participant components for LongevityBench, items for MMLU-Pro and GSM8K), so that a unit drawn k times counts its rows k times; every statistic of the cell is recomputed on the resample with each row weighted by the draw of its unit, and all contrasts of a cell share the resample. The seed of a cell derives from a hash of its partition, model and pool (base 4242). The B statistics are sorted ascending and indexed from zero, and for a two-sided tail probability q the lower bound of the interval is the value at index âBâqââ1 Bq -1 and the upper bound the value at index âBâĄ(1âq)ââ1 B(1-q) -1, clamped to the range of indices. Every estimate carries an unadjusted 95% interval (q=0.025q=0.025), and a verdict uses the Bonferroni-adjusted interval of its family, with q=0.025/mq=0.025/m for m endpoints (levels 99.44%, 99.62%, 99.44% and 99.69% for H1âH4). These levels are nominal: a percentile bootstrap covers only approximately, so the family-wise error rate is controlled approximately and is not guaranteed. For H4, the resample of an instanceâs units also serves its clean counterpart where the two share their rows (all defects except the skewed split, which uses other rows and is resampled independently), so that DSD_S and DRD_R are paired, and Î pairs the b-th resample of S with the b-th of R, which are disjoint samples drawn with their own seeds. A test verdict is replicated if the replication partition gives the same verdict, not replicated if it gives the opposite one, and inconclusive replication otherwise. The error control of H4 covers âholdsâ only, and âfailsâ is a descriptive label. The preregistration was frozen at 06:53:11 UTC on 30 September 2026, before any replication or test item was called, and the pilot called 700 pilot items only. The directions of H1 and H3 were visible in the pilotâs parse and truncation rates, and the primary set of H1 changed twice before the freeze after those rates were seen (Section 5). 4 Results Results are on the test partition, the confirmatory one, unless stated, and the replication partition enters through the replication label. Primary endpoints carry the family-adjusted interval and everything else the unadjusted 95% interval; Appendix B gives every endpoint in full. 4.1 The 47 primary endpoints at a glance Table 3 gives the 31 endpoints of H1âH3 and Table 4 the 16 pairs of H4, and the sections below take them in turn. Of the 31 endpoints of H1âH3, 30 are replicated and 1 is an inconclusive replication, and the number labelled not replicated is 0. Figure 1 plots the adjusted intervals. Table 3: Verdicts of the 31 endpoints of H1âH3 on the test partition. A dagger marks the one endpoint whose replication label is not replicated: an inconclusive replication (SmolLM3-3B on GSM8K, H2: fails on test, inconclusive on replication). The last row is H2 against the four GEPA harnesses. Model Pool H1 AUTHORS-1500 vs THINK-512 H2 AUTHORS-1500 vs RESCUE-1500 H3 BUDGET-512-1500 vs THINK-512 Qwen3-1.7B-Longevity LB letters no gain: not tested holds no room MMLU-Pro no gain: not tested holds inconclusive GSM8K no gain: not tested holds no room SmolLM3-3B LB letters visibility dominates holds holds MMLU-Pro visibility dominates holds holds GSM8K visibility dominates failsâ holds Intern-S1-mini LB letters visibility dominates holds holds MMLU-Pro visibility dominates holds holds GSM8K no gain: not tested fails holds Qwen3-1.7B-Longevity LB letters H2 against the four GEPA harnesses: holds, holds, holds, holds Figure 1: Adjusted intervals of the primary endpoints of H1âH3 on the test partition. (a) H1: Î , the gain gate; (b) H2: θ against the non-inferiority margin âδ-δ; (c) H3: DTD_T and DPD_P against the minimal difference m. Cells are modelĂpool; filled markers pass the verdictâs boundary, open markers do not. 4.2 H1: visibility dominates where the thinking cap truncates Visibility dominates in SmolLM3-3B on all three pools and in Intern-S1-mini on the letters and MMLU-Pro. In these five cells AUTHORS-1500 gains over THINK-512 between Î=+0.060 =+0.060 [+0.046+0.046, +0.074+0.074] (SmolLM3-3B, letters) and +0.403+0.403 [+0.352+0.352, +0.456+0.456] (Intern-S1-mini, MMLU-Pro), and the gain is visibility: ÎV _V is at least Î in all five (shares from 1.00 to 1.82), because the rows both configurations parse contribute at most ÎBâ¤+0.001 _Bâ¤+0.001 and the rows only THINK-512 parses cost ÎL _L (for SmolLM3-3B on GSM8K, ÎV=+0.197 _V=+0.197, ÎB=â0.036 _B=-0.036 and ÎL=â0.053 _L=-0.053). Almost all of THINK-512âs unparsed calls ended at the token limit; the exception is SmolLM3-3B on the letters, where 543 of 3,394 ended by themselves without an answer (non-committed). Two of the five verdicts are nearly settled by construction, as the preregistration expected for the first. In Intern-S1-mini on the letters both configurations parse only 1 row of 3,942, so ÎB _B is nearly zero and T, nearly a fifth of ÎV _V, is positive (the preregistration listed the cell as âT>0T>0 nearly inevitableâ). In Intern-S1-mini on MMLU-Pro, B has 69 of 799 rows, so ÎB _B cannot exceed that share, which is below a quarter of the observed ÎV=+0.419 _V=+0.419, and T>0T>0 follows from Bâs row count and that observed ÎV _V. The verdict is informative in SmolLM3-3Bâs three cells, where B has 448, 81 and 340 rows. The other four cells fail the gain gate. Qwen3-1.7B-Longevity needs no visibility, since THINK-512 parses 1.000 of the letter rows, 0.940 of MMLU-Pro and 0.992 of GSM8K, and Î=â0.003 =-0.003 [â0.018-0.018, +0.012+0.012] on the letters. In Intern-S1-mini on GSM8K the gate fails the other way, because AUTHORS-1500 loses accuracy (Î=â0.287 =-0.287 [â0.355-0.355, â0.218-0.218]); there T=+0.286T=+0.286 is positive only because ÎB=â0.343 _B=-0.343 is large and negative, the reading the gate exists to stop, and THINK-512 is the more accurate configuration on the rows both parse. The sanity checks (THINK-64 against AUTHORS-1500) give visibility dominates in 8 of nine cells, as expected, and fail the gate where Qwen3-1.7B-Longevity already parses GSM8K. 4.3 H2: the thinking-off baseline is non-inferior in most cells and fails on GSM8K for two models AUTHORS-1500 is non-inferior to RESCUE-1500 in 7 of nine cells and to all four GEPA harnesses. It fails on GSM8K for SmolLM3-3B (θ=â0.089θ=-0.089 [â0.129-0.129, â0.053-0.053]) and for Intern-S1-mini (θ=â0.342θ=-0.342 [â0.402-0.402, â0.281-0.281]), where the rescue is more accurate than the thinking-off configuration by more than the margin. The study does not isolate which part of the rescue, the thinking within 512 tokens or the second call, accounts for the difference. We expected the Intern-S1-mini endpoint to be inconclusive, an informal expectation from a planned adjusted power of 0.19 at a true θ of zero (Appendix K), and it is not close. For SmolLM3-3B on GSM8K the replication partition is inconclusive, so the label is an inconclusive replication. In Intern-S1-miniâs letter cell the rescue equals AUTHORS-1500 (θ=0.000θ=0.000, since every THINK-512 call is cut off), so holds is trivial there. GEPA searched prompt text only (Appendix J), repaired its broken starts, and found no harness whose accuracy differs detectably from the thinking-off baselineâs. In the clean-start run with seed zero no proposal beat the seed on the validation rows, so its selected harness is the seed, AUTHORS-1500 itself (θ=0.000θ=0.000 by construction). The other three selected harnesses parse every test row and reach accuracy 0.620 to 0.631, against 0.632 for AUTHORS-1500; θ runs from +0.002+0.002 to +0.012+0.012, and all three adjusted intervals contain zero. Their validation scores (0.660 to 0.700 over the four runs) exceed their test accuracy in every run, as selection makes them optimistic, and none of the candidates GEPA saw (192 to 377 per run) contained a soft switch (/think or /no_think). 4.4 H3: BUDGET-512-1500 lowers truncation where THINK-512 truncates BUDGET-512-1500 lowers truncation and raises parsing in the six cells where THINK-512 truncates a large share, with DTD_T from +0.260+0.260 (SmolLM3-3B, GSM8K) to +0.979+0.979 (Intern-S1-mini, letters), DPD_P from +0.151+0.151 to +0.662+0.662, and both adjusted lower bounds above the minimal difference. In the Qwen3-1.7B-Longevity letter and GSM8K cells THINK-512 does not truncate (adjusted upper bounds +0.001+0.001 and 0.0000.000), so there is no room for a drop of 0.05. In Qwen3-1.7B-Longevity on MMLU-Pro DT=DP=+0.060D_T=D_P=+0.060 with adjusted interval [+0.038+0.038, +0.085+0.085], which contains 0.05, so the verdict is inconclusive. The parse gain is not the whole story for SmolLM3-3B: after the forced end of thinking it re-opens a turn in 4,384 of its 7,216 test calls, and BUDGET-512-1500 still ends by itself without an answer on 0.424 of its letter rows. What H3 isolates. BUDGET-512-1500 differs from THINK-512 in two ways at once: vLLM ends thinking after 512 tokens, and the total allowance is 2,012 tokens against 512. H3 therefore supports the combined change and not forced termination alone. As a partial and descriptive comparison, THINK-1536, with an allowance of 1,536 tokens and no forcing, still ends at the token limit in 0.066 (SmolLM3-3B, GSM8K) to 0.800 (Intern-S1-mini, letters) of its calls in the six cells where THINK-512 truncates a large share, against 0.021 to 0.379 for BUDGET-512-1500, which truncates less in all six. The two differ in allowance as well (2,012 against 1,536 tokens), so the comparison does not separate forcing from allowance. Is the truncation an artifact of zero temperature? In the two models with a seeded arm, substantial truncation persists under sampling (descriptive; Table 22). Sampling at the model cardsâ recommended values (Qwen3-1.7B-Longevity: temperature 0.6, top-p 0.95, top-k 20; Intern-S1-mini: 0.8, 1.0, 50), with a seed per request, changes THINK-512âs truncation by at most 0.023 in ten of the 12 cells (two models, three pools, two partitions) and raises it for Intern-S1-mini on GSM8K by +0.097+0.097 (test) and +0.098+0.098 (replication). Under sampling THINK-512 still ends at the token limit in 0.619 to 0.997 of Intern-S1-miniâs test calls, and the drop in truncation that H3 measures, recomputed on the seeded outputs with unadjusted intervals, runs from +0.581+0.581 to +0.963+0.963 for Intern-S1-mini and is +0.068+0.068 for Qwen3-1.7B-Longevity on MMLU-Pro, next to +0.060+0.060 with greedy decoding. The preregistered verdicts use family-adjusted intervals at another level, so the seeded statistics are not compared with them, and SmolLM3-3B has no seeded arm. 4.5 H4: prospective replication confirms the distortion in six of fifteen pairs Table 4: H4: the 16 (defect, model) pairs, with the letters of Section 3.4. The tied output layer (E) exists for Qwen3-1.7B-Longevity only, the one model with two served variants. The bound is the largest distortion the mechanism can cause on S, computed without accuracy; a pair is evaluable only if it exceeds 0.05. DSD_S, DRD_R and Î=DRâDS =D_R-D_S are point estimates of the distortion of the startâs accuracy (for the broken start, of the gain over the start, which equals the accuracy of THINK-1536 minus that of THINK-64); intervals are in Figure 2 and Appendix B. A dagger marks the claim of the skewed split in SmolLM3-3B, which fails by the general rule of the preregistration (Section 5). Defect Model Bound DSD_S DRD_R Î Claim A reference answer as assistant turn Qwen3 1.000 â0.052-0.052 â0.030-0.030 +0.022+0.022 inconclusive SmolLM3 1.000 +0.097+0.097 +0.119+0.119 +0.022+0.022 inconclusive Intern-S1 1.000 +0.142+0.142 +0.164+0.164 +0.022+0.022 inconclusive B1 gold fallback Qwen3 0.205 â0.145-0.145 â0.148-0.148 â0.003-0.003 holds SmolLM3 0.083 +0.001+0.001 +0.007+0.007 +0.006+0.006 fails Intern-S1 0.057 â0.012-0.012 â0.019-0.019 â0.007-0.007 fails B2 unparsed scored as A Qwen3 0.000 not evaluable â SmolLM3 0.208 +0.208+0.208 +0.223+0.223 +0.015+0.015 holds Intern-S1 0.289 +0.289+0.289 +0.292+0.292 +0.003+0.003 holds C skewed split Qwen3 by rule +0.072+0.072 +0.101+0.101 +0.030+0.030 inconclusive SmolLM3 by rule â0.001-0.001 +0.005+0.005 +0.006+0.006 failsâ Intern-S1 by rule +0.003+0.003 +0.004+0.004 +0.001+0.001 inconclusive D broken start Qwen3 1.000 +0.638+0.638 +0.635+0.635 â0.003-0.003 holds SmolLM3 0.352 +0.164+0.164 +0.157+0.157 â0.007-0.007 holds Intern-S1 0.189 +0.124+0.124 +0.125+0.125 +0.001+0.001 holds E tied vs untied output layer Qwen3 0.139 +0.005+0.005 +0.014+0.014 +0.009+0.009 fails Figure 2: H4: the defectâs distortion on the selection sample (DSD_S), on the check sample (DRD_R) and their difference (Î ), with adjusted intervals (family of 16) and the band ÂąmÂą m (shaded). Filled markers are claims that hold; arrows mark intervals cut at the axis limit. The unparsed-as-A scoring (B2) in Qwen3-1.7B-Longevity is not evaluable and is not drawn. The claim holds in six pairs: the broken start in all three models, the unparsed-as-A scoring in SmolLM3-3B and Intern-S1-mini, and the gold fallback in Qwen3-1.7B-Longevity. Under the broken start the reported gain over the start is inflated by DS=+0.638D_S=+0.638, +0.164+0.164 and +0.124+0.124 (Qwen3-1.7B-Longevity, SmolLM3-3B, Intern-S1-mini) and by almost the same amount on R (Î=â0.003 =-0.003, â0.007-0.007, +0.001+0.001). Scoring unparsed answers as A inflates the startâs accuracy by +0.208+0.208 and +0.289+0.289, exactly the share the start leaves unparsed with gold A, and the gold fallback moves Qwen3-1.7B-Longevityâs start by â0.145-0.145. In all six the distortion is large and reappears on R, so a replication would have reproduced the distorted report instead of revealing it. The claim fails in four pairs: the gold fallback in SmolLM3-3B and Intern-S1-mini, whose starts parse too few of the rows whose gold the fallback changes (bounds 0.083 and 0.057, distortions +0.001+0.001 and â0.012-0.012); the skewed split in SmolLM3-3B; and the tied output layer in Qwen3-1.7B-Longevity, where DS=+0.005D_S=+0.005 and DR=+0.014D_R=+0.014 both lie inside the band. Five pairs are inconclusive, the reference-answer defect in all three models and the skewed split in Qwen3-1.7B-Longevity and Intern-S1-mini; for the reference-answer defect in Intern-S1-mini both DS=+0.142D_S=+0.142 and DR=+0.164D_R=+0.164 clear the band, but the interval of Î , [â0.065-0.065, +0.108+0.108], is wide because each half has only 81 components. The unparsed-as-A scoring in Qwen3-1.7B-Longevity is not evaluable, because its start leaves no row unparsed (bound 0.000). These counts follow the general rule of the preregistration for DRD_R, which the pinned code implements (Section 5); under the narrower reading that grades only the two sub-cases listed after the general rule, the skewed split in SmolLM3-3B is inconclusive and the counts are 6 holds, 3 fails and 6 inconclusive (Appendix C). The audit checks. Table 5 summarises the six checks of Section 3.4. Their detection matrix is descriptive and largely true by construction: every rule expected to fire did so in 12 of the 16 instances (Appendices D and B give the exceptions and each instance). What is empirical is the false alarms on the clean instances: IR4 fires on 7 of 7, SCF1 on 4, SCF2 on 4, TB2 on 3, SCF3 on 2 and the gain rule of PR on 1, and no other rule fires on a clean instance. Table 5: The audit checks of H4: what each reads, its outcome on the 16 defective instances of the detection matrix (detects: fires on the defective instance and not on the clean one; uninformative: fires on both; misses: does not fire; descriptive) and the rules that fire on the clean instances (what is empirical). Check Reads Detects Uninformative Misses Fires on the clean instances PR gain and level rules of prospective replication 5 1 10 PR gain 1/7 TR per-call prompt-token reconciliation 3 0 13 none TB trivial baselines 6 3 7 TB2 3/7 SCF the simple configuration family 3 12 1 SCF1 4/7, SCF2 4/7, SCF3 2/7 IR reading of the rendered inputs 0 16 0 IR4 7/7 SMI served-model identity (defect E) 1 0 0 â 4.6 Descriptive results Accuracy next to the trivial baselines. Table 6 gives the accuracy of five configurations in the three pools next to their baselines, computed after the outputs existed (Appendix C), with a block for each LongevityBench task; THINK-64, the GEPA harnesses and the replication partition are in Appendix B. Each oracle constant is chosen once, on the analysis rows of the partition it is scored on, and it is a fixed prediction in the bootstrap: it is not re-chosen in a resample, so the intervals of the differences to it cover the sampling of units given that choice. On the LongevityBench letters the strongest constant is the oracle constant label: the age band most common on LB-0030, Participant A on LB-0034 and âYesâ on LB-0042 (0.840 of its test rows). It scores 0.539 [0.527, 0.551] on the pool, far above chance (0.380) and the oracle constant letter (0.389), because the option order is shuffled (120 letter-to-label mappings on LB-0030, 2 on LB-0042) and a constant letter cannot exploit an uneven label distribution. Qwen3-1.7B-Longevity exceeds the strongest constant in every configuration that parses: AUTHORS-1500 0.632, +0.093+0.093 [+0.074+0.074, +0.112+0.112], and THINK-512 +0.096+0.096 [+0.077+0.077, +0.115+0.115]. The margin is small on LB-0042 (+0.027+0.027 [+0.007+0.007, +0.048+0.048] for THINK-512), where one GEPA harness, GEPA-CLEAN-s1, is not above it (â0.010-0.010 [â0.034-0.034, +0.015+0.015]). SmolLM3-3B and Intern-S1-mini are below the strongest constant on the pool in every configuration (at best â0.382-0.382 and â0.225-0.225), because most of their calls end unparsed and unparsed counts as wrong, although Intern-S1-mini is above it on LB-0034 with AUTHORS-1500 (+0.060+0.060 [+0.003+0.003, +0.117+0.117]). The case study had not found this: on the split of its harness search no configuration of the same model clearly exceeded a constant age band on LB-0030 or the majority letter on LB-0034 (post hoc there). We read the difference as consistent with that splitâs skew toward young, alive participants, but the items, the output layer (tied there, untied here) and the stack all differ, so the data do not establish it. The output layer alone is unlikely to explain it: on the same items the tied and untied variants differ in accuracy by at most 0.009, although their parsed answers are identical on only 0.656 to 0.741 of the rows, and on the letter pool the tied-output-layer defect moves the startâs accuracy by DS=+0.005D_S=+0.005 and DR=+0.014D_R=+0.014, inside the band. Table 6: Accuracy in the three pools and in each LongevityBench task, test partition, next to the trivial baselines; in parentheses the paired difference to the strongest baseline (the oracle constant, chosen once and held fixed in the bootstrap). Unparsed answers count as wrong. THINK-64, the GEPA harnesses and the replication partition are in Appendix B. Computed after the outputs existed (Appendix C). Model THINK-512 THINK-1536 AUTHORS-1500 BUDGET-512-1500 RESCUE-1500 LongevityBench letter pool (3,942 rows). Chance 0.380; constant letter 0.383 (chosen on replication), 0.389 (oracle); constant label 0.530 (chosen on replication), 0.539 (oracle). Strongest, the oracle constant label: 0.539 [0.527, 0.551]. Qwen3 0.635 (+0.096+0.096) 0.635 (+0.096+0.096) 0.632 (+0.093+0.093) 0.635 (+0.096+0.096) 0.635 (+0.096+0.096) SmolLM3 0.070 (â0.470-0.470) 0.157 (â0.382-0.382) 0.129 (â0.410-0.410) 0.121 (â0.418-0.418) 0.128 (â0.411-0.411) Intern-S1 0.000 (â0.539-0.539) 0.125 (â0.414-0.414) 0.314 (â0.225-0.225) 0.253 (â0.286-0.286) 0.314 (â0.225-0.225) LB-0030 (1,577 rows). Chance 0.200; constant letter 0.216 (oracle); constant label 0.250 (oracle). Strongest, the oracle constant label: 0.250 [0.230, 0.269]. Qwen3 0.369 (+0.119+0.119) 0.369 (+0.119+0.119) 0.344 (+0.094+0.094) 0.369 (+0.119+0.119) 0.369 (+0.119+0.119) SmolLM3 0.000 (â0.250-0.250) 0.021 (â0.229-0.229) 0.053 (â0.197-0.197) 0.032 (â0.218-0.218) 0.053 (â0.197-0.197) Intern-S1 0.000 (â0.250-0.250) 0.004 (â0.245-0.245) 0.023 (â0.227-0.227) 0.055 (â0.195-0.195) 0.023 (â0.227-0.227) LB-0034 (788 rows). Chance 0.500; constant letter 0.515 (oracle); constant label 0.515 (oracle). Strongest, the oracle constant label: 0.515 [0.480, 0.551]. Qwen3 0.702 (+0.187+0.187) 0.703 (+0.188+0.188) 0.717 (+0.202+0.202) 0.703 (+0.188+0.188) 0.703 (+0.188+0.188) SmolLM3 0.307 (â0.208-0.208) 0.409 (â0.107-0.107) 0.443 (â0.072-0.072) 0.355 (â0.160-0.160) 0.435 (â0.080-0.080) Intern-S1 0.000 (â0.515-0.515) 0.242 (â0.273-0.273) 0.575 (+0.060+0.060) 0.523 (+0.008+0.008) 0.575 (+0.060+0.060) LB-0042 (1,577 rows). Chance 0.500; constant letter 0.500 (oracle); constant label 0.840 (oracle). Strongest, the oracle constant label: 0.840 [0.822, 0.858]. Qwen3 0.867 (+0.027+0.027) 0.867 (+0.027+0.027) 0.878 (+0.038+0.038) 0.867 (+0.027+0.027) 0.867 (+0.027+0.027) SmolLM3 0.020 (â0.820-0.820) 0.167 (â0.673-0.673) 0.049 (â0.791-0.791) 0.093 (â0.748-0.748) 0.050 (â0.790-0.790) Intern-S1 0.000 (â0.840-0.840) 0.186 (â0.654-0.654) 0.475 (â0.365-0.365) 0.317 (â0.523-0.523) 0.475 (â0.365-0.365) MMLU-Pro (799 items). Chance 0.110; constant letter B, chosen on the other partition and oracle alike, 0.119 [0.096, 0.141]. Qwen3 0.217 (+0.098+0.098) 0.234 (+0.115+0.115) 0.229 (+0.110+0.110) 0.232 (+0.113+0.113) 0.232 (+0.113+0.113) SmolLM3 0.098 (â0.021-0.021) 0.314 (+0.195+0.195) 0.353 (+0.234+0.234) 0.258 (+0.139+0.139) 0.378 (+0.259+0.259) Intern-S1 0.074 (â0.045-0.045) 0.279 (+0.160+0.160) 0.477 (+0.358+0.358) 0.542 (+0.423+0.423) 0.493 (+0.374+0.374) GSM8K (527 items). Constant answer 10 (chosen on the other partition) 0.019 [0.008, 0.032], 2 (oracle) 0.034 [0.019, 0.051]; no chance level. Qwen3 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) SmolLM3 0.681 (+0.647+0.647) 0.880 (+0.846+0.846) 0.789 (+0.755+0.755) 0.835 (+0.801+0.801) 0.879 (+0.844+0.844) Intern-S1 0.476 (+0.442+0.442) 0.833 (+0.799+0.799) 0.190 (+0.156+0.156) 0.922 (+0.888+0.888) 0.531 (+0.497+0.497) The general pools. On MMLU-Pro Qwen3-1.7B-Longevity reaches 0.217 to 0.234, +0.098+0.098 to +0.115+0.115 over the constant letter (0.119), with every interval above zero, but SmolLM3-3B and Intern-S1-mini reach 0.378 and 0.542 with their best configurations, and THINK-512, which truncates, is level with or below the constant for both. On GSM8K Qwen3-1.7B-Longevity scores 0.068 to 0.070 in every configuration, which is +0.036+0.036 [+0.011+0.011, +0.063+0.063] over the oracle constant answer (0.034) in test and +0.011+0.011 [â0.027-0.027, +0.049+0.049], an interval containing zero, in replication, against 0.880 and 0.922 for the best configurations of the other two. The model that is best on the longevity letters is thus the weakest on MMLU-Pro and barely above a constant answer on GSM8K. We do not attribute this to fine-tuning, because the base model, Qwen3-1.7B, was not evaluated and the three models differ in size, training and thinking behaviour. Seeded sampling moves Qwen3-1.7B-Longevityâs letter accuracy by â0.013-0.013 to â0.018-0.018, only Qwen3-1.7B-Longevity parses LB-0038 reliably, and the tied-versus-untied comparison on FreeToken and the GEPA candidates are in Appendix B. 4.7 Sensitivity analyses Each of the ten sensitivity analyses of the preregistration is descriptive. The strict scorer changes 2 of the 31 endpoint verdicts, and restricting to the never-called rows, which no earlier study had called, changes 3; these rows skew young and alive, with a mean LB-0030 age of 37.3 against 50.8 years and a share of LB-0042 rows whose follow-up says the participant was deceased of 0.022 against 0.346. That share counts a death at any time of the follow-up and is not the ten-year label, whose gold is âYesâ for a participant who died after ten years. The case studyâs pool changes 1 and LB-0030 alone 2, and LB-0034 alone, LB-0042 alone, both role-break variants and the pooled partitions change none. Putting the left-out rows back changes no verdict of H1âH3 and 2 H4 claims (the skewed split in SmolLM3-3B and Intern-S1-mini). 5 Deviations and addenda The preregistration allows a change only if it alters no request bytes, parsing, scoring or call order, and then only with a written note and a dry run that reproduces the recorded render hashes; any other change needs a new registration. After the freeze, the requests, the scoring and the primary criteria stayed as preregistered, with two exceptions that were not mechanical. We waived one stopping rule, once and for one stage: the check on Intern-S1-miniâs thinking switch had stopped the run that sends the reference answer, and we let that check record its failures and the run continue. And we settled one descriptive label after seeing the outputs: while auditing this paper we found that the two listed cases of the claim rule for the check sample are incomplete, so its general rule governs, and under it the claim for the skewed split in SmolLM3-3B fails where the narrower reading leaves it inconclusive (Section 4 gives both totals for H4). Table 28 in Appendix C lists the deviations that bear on a result, and the appendix lists every exception, known deviation, pre-freeze decision and addendum with its date, reason, rule and the outputs seen at the time. No call touched a replication or test item before the freeze, and we saw only counts, and no accuracy, score or verdict of the main study, until every stage had run and the checkpointâs record had been committed. Everything computed after the outputs existed is labelled where it appears. This paperâs tables and figures only format aggregates committed in the authorsâ repository, and its audit recomputes every verdict from its interval and agrees with the report on all of them (Appendix C). 6 Discussion and limitations What the results say. The gain of the thinking-off configuration over THINK-512 comes mostly from rows that THINK-512 left unparsed, so a search that starts from a configuration like this can find ârepairsâ that a thinking-off baseline already provides; GEPA, which searched prompt text only, repaired its broken starts, and none of its selected harnesses was more accurate than that baseline. Baseline adequacy still depends on the cell, because the rescue beat the thinking-off configuration by more than the margin on GSM8K for two models, so a thinking-off baseline is one to run and not one to assume adequate. Trivial baselines matter in both directions: only Qwen3-1.7B-Longevity beats the strongest constant label on the LongevityBench letters, and on GSM8K it is barely above a constant answer. Where prospective replication confirmed the distortion (6 of 15 evaluable pairs), the distortion was large and, within five points, the same on the check sample as on the selection sample. Some systematic distortions therefore persist across samples, and a replication run through the same pipeline reproduces them instead of revealing them. In the other pairs the claim fails (a descriptive label) or is inconclusive, and a defect whose claim fails need not be harmless. The checks that exposed the defects read inputs and scoring; their detections here are largely true by construction, and their false alarms (IR4, SCF1, SCF2, TB2) show that their thresholds need care. Limitations. The study covers three small models, three pools and six defects chosen from one line of work, so H4 says how replication behaves for these defects and not for the space of defects. âProspectiveâ has limits: earlier work had called most LongevityBench rows with other configurations, callers and one model variant, and the rows the case study had held out are excluded. Power is mixed. The H2 endpoint of Intern-S1-mini on GSM8K was expected to be inconclusive and is not (an informal expectation, Appendix K), H4 has no planned power, and its error control covers âholdsâ only. The GEPA arm searched prompt text only, with one proposer and four runs, so its negative result is about that space. The extended scorer and the role-break rule were fixed after the pilotâs parse audit. Unparsed counts as wrong by rule, so accuracy mixes parsing and correctness, and an unparsed answer may hide a correct answer, a wrong one or none; the study measures differences between configurations as served and not what the models know. The batch-invariant server differs from the standard one call for call, and everything ran on one GPU instance. Our intervals resample participant components (items in the two general pools) at a frozen harness, zero temperature and fixed prompts, so they cover item sampling and not prompt or judge variance (Messing, 2026). The freeze guards the rules and not model-specific tuning, which a preregistration for the next model would (Thomas et al., 2026). Zero temperature may turn some thinking into repetition; for the two models with a seeded arm substantial truncation persists under sampling (Section 4), and for SmolLM3-3B it is untested. The authors used Claude (Claude Code and claude.ai) and Codex for code, drafting and review, and they verified all analyses and claims. Broader impact. This paper contributes methodology for evaluating harness search and the pipelines that report it: a preregistered design, audit checks and a record of what each result rests on. It builds no new capability, since the models and benchmarks are public and the benchmark behind the longevity tasks derives from public survey data. The accuracies are those of small models on research tasks and do not support any use of these models for medical advice or clinical decisions. One misreading to avoid is that a thinking-off baseline is always enough, and the study shows two cells in which it is not. 7 Conclusion In a design fixed in advance, most of the gain of a thinking-off configuration over a capped thinking configuration came from rows the capped configuration left unparsed, and the thinking-off configuration was non-inferior to the rescue and to four GEPA-selected harnesses in 11 of 13 endpoints, though not on GSM8K for two models. In 6 of the 15 evaluable (defect, model) pairs, the defectâs distortion of the reported number was larger than five points on the selection sample and reappeared, within five points, on the check sample. Reporting truncated and non-committed answers apart, running a thinking-off baseline and constant-label baselines, reading the rendered inputs and reconciling prompt tokens per call cost little, and they exposed the defects here. Code and data availability The studyâs repository is not public at present. The paths, commits, tags and SHA256 values that this paper names belong to files in the authorsâ repository, and we keep them so that a later public export can be checked against them (Appendix E). The run logs hold benchmark item text and model outputs and will not be released. This arXiv version carries three ancillary files and a short README that describes them. The frozen preregistration and the aggregate analysis report are attached byte for byte. The addenda file is attached as it stood at the posting, with the row keys of five rows withheld because a row key identifies a benchmark row. The freeze time is recorded only in the authorsâ private repository, in the hash file that the freeze commit adds, and it is not independently timestamped. The SHA256 of the frozen file (Appendix E) therefore lets a reader check that the ancillary copy is the file that was frozen, but not when it was frozen. A filtered public export of the rest is planned and will be linked in a later version of this paper: the code and its tests and the hash manifests, with no item-level or row-level content derived from the held-out rows. The benchmarks and models are used under the terms their sources give, read on 2026-10-03. LongevityBench (dataset card insilicomedicine/longebench) is released under C-BY-NC-4.0; its card states that the underlying source datasets (NHANES, GTEx, GEO, Olink, OpenGenes, SynergyAge) retain their original licences and access conditions, with which users must comply. MMLU-Pro (TIGER-Lab/MMLU-Pro) and GSM8K (openai/gsm8k) are under the MIT licence. Qwen3-1.7B-Longevity (insilicomedicine/Qwen3-1.7B-Longevity) is released under C-BY-ND-4.0, and its base model Qwen3-1.7B is distributed under Apache-2.0; SmolLM3-3B (HuggingFaceTB/SmolLM3-3B) and Intern-S1-mini (internlm/Intern-S1-mini) are under Apache-2.0. The study uses the weights unmodified and redistributes neither benchmark item text nor model weights. Author Contributions Bowen Xu: Conceptualization, Methodology, Software, Investigation, Formal analysis, Visualization, Writing â original draft, Writing â review & editing, Project administration. Xu conceived and led the study. Boyu Chen: Methodology, Supervision, Writing â review & editing. References Agrawal et al. (2025) Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv:2507.19457, 2025. ICLR 2026 (Oral). Bai et al. (2025) Lei Bai et al. Intern-S1: A scientific multimodal foundation model. arXiv:2508.15763, 2025. Bakouch et al. (2025) Elie Bakouch et al. SmolLM3: smol, multilingual, long-context reasoner. Hugging Face blog, 2025. URL https://huggingface.co/blog/smollm3. Bellibatlu et al. (2026) Rohith Reddy Bellibatlu, Zichong Wang, and Wenbin Zhang. Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments. arXiv:2609.37315, 2026. Berriche et al. (2026) Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, and Yazan Boshmaf. Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks. arXiv:2609.08765, 2026. Carlon et al. (2026) Francesca Carlon, Vincent Ginis, and Andres Algaba. Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces. arXiv:2608.03401, 2026. Chang & Chen (2026) Yuan Chang and Xiaoqi Chen. Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search. arXiv:2608.27266, 2026. Chen et al. (2026) Mingguang Chen, Bo Qu, and Licheng Wang. The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale. arXiv:2608.04355, 2026. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168, 2021. Dutta et al. (2024) Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is Not All You Need. arXiv:2407.09141, 2024. Gao et al. (2025) Irena Gao, Percy Liang, and Carlos Guestrin. Model Equality Testing: Which Model Is This API Serving? arXiv:2410.20247, 2025. ICLR 2025. Garware & Zisad (2026) Chaitanya Vilas Garware and Sharif Noor Zisad. When the Ruler is Broken: Parsing-Induced Suppression in LLM-Based Security Log Evaluation. arXiv:2605.07293, 2026. Gideoni et al. (2026) Yonatan Gideoni, Sebastian Risi, and Yarin Gal. Simple Baselines are Competitive with Code Evolution. arXiv:2602.16805, 2026. Goyal & Ray (2026) Ankit Goyal and Jaideep Ray. Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap. arXiv:2608.04160, 2026. Gringras (2026) David Gringras. Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety. arXiv:2603.10044, 2026. He & Thinking Machines Lab (2025) Horace He and Thinking Machines Lab. Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism (blog), 2025. URL https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. hmellor (2026) (vLLM project) hmellor (vLLM project). Fix weight tying. GitHub pull request 51665, vllm-project/vllm; merged 2026-08-20, 2026. URL https://github.com/vllm-project/vllm/pull/51665. JasonKeyiL (2026) (vLLM project) JasonKeyiL (vLLM project). [Bugfix] Fix inverted condition causing thinking_token_budget to be silently ignored. GitHub pull request 41674, vllm-project/vllm; merged 2026-05-15, 2026. URL https://github.com/vllm-project/vllm/pull/41674. Kaiser et al. (2026) Daniel Kaiser, Arnoldo Frigessi, Ali Ramezani-Kebrya, and Benjamin Ricaud. Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs. arXiv:2602.09805, 2026. Kapoor et al. (2024) Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter. arXiv:2407.01502, 2024. Khattab et al. (2023) Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv:2310.03714, 2023. KĂźbler et al. (2026) Jonas KĂźbler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, and George Karypis. When LLMs get significantly worse: A statistical approach to detect model degradations. arXiv:2602.10144, 2026. ICLR 2026. Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611â626, 2023. Lee (2026) Janghoon Lee. Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention. arXiv:2608.13959, 2026. Lee et al. (2026a) Jungseob Lee, Seongtae Hong, Seungjun Lee, Jaehyung Seo, Junyoung Son, Sugyeong Eo, Chanjun Park, Hyeongju Park, Hyeonseok Moon, and Heuiseok Lim. DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Models. arXiv:2606.23181, 2026a. EMNLP 2026 (Findings). Lee et al. (2026b) Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. arXiv:2603.28052, 2026b. Liang et al. (2026) Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, and Shiwen Ni. Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem. arXiv:2608.19626, 2026. Liu et al. (2026) Shiyan Liu, Qifeng Xia, Qiyun Xia, Yisheng Liu, Xinyu Yu, and Rui Qu. Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization. arXiv:2603.18388, 2026. ACL SRW 2026. Ma et al. (2025) Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning Models Can Be Effective Without Thinking. arXiv:2504.09858, 2025. Messing (2026) Solomon Messing. Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking. arXiv:2604.11581, 2026. Muennighoff et al. (2025) Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv:2501.19393, 2025. Nie et al. (2026) Wenhua Nie, Junlin Liu, Jianan Wu, Zijie Meng, Yilong Fan, Zhang Zijian, Haoran Zheng, and Jyh-Shing Roger Jang. The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits. arXiv:2605.07686, 2026. Pape et al. (2026) David Pape, Jonathan Evertz, and Lea SchĂśnherr. The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility. arXiv:2605.19537, 2026. Seyoum & Mittur (2026) Kidus Seyoum and Ajay Mittur. Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses? arXiv:2609.28908, 2026. Sharma & Lall (2026) Rishabh Sharma and Rishika Lall. When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model. arXiv:2609.34227, 2026. Shawn (2026) Zayx Shawn. PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents. arXiv:2606.08106, 2026. Singh (2026) Amogh Singh. Certifying Compressed Language Models: An Audit and a Statistical Toolkit. arXiv:2608.15046, 2026. Stenstrom et al. (2026) Blake Stenstrom, Charangan Vasantharajan, and Brian Sathianathan. IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier. arXiv:2609.10494, 2026. Thomas et al. (2026) Maria Thomas, Kristina Gligoric, and Nihar B. Shah. Mitigating LLM-based p-Hacking by Preregistering for the Next LLM. arXiv:2606.27687, 2026. Ursekar et al. (2026) Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, and Yuan Xue. HarnessOpt-Bench: Evaluating LLMs at Harness Optimization. arXiv:2608.06301, 2026. Wahi (2026) Vansh Wahi. LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails. arXiv:2609.02246, 2026. Wang (2026) Wenbo Wang. Interface-Induced Trajectory Censoring. arXiv:2609.03966, 2026. Wang et al. (2026a) Xing Wang, Xiaoyi Zhang, and Jie Shao. Auditing Harness Tampering in Self-Improving Agents. arXiv:2609.00069, 2026a. Wang et al. (2026b) Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the Evaluation of Harness Evolution for Agents. arXiv:2607.12227, 2026b. Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track, 2024. Weinberger & Hozez (2026) Sarel Weinberger and Amir Hozez. Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost. arXiv:2608.01347, 2026. Wu et al. (2026) Siwei Wu, Jincheng Ren, Yizhi Li, Haau-Sing Li, Chengran Yang, Yuxuan Zhang, Weicheng Gu, Jian Yang, Riza Batista-Navarro, Chuanyi Zhang, Xianglong Liu, Ming Zhou, Bryan Dai, and Chenghua Lin. ModularRSI: Modular and generalizable recursive harness self-improvement. arXiv:2609.14857, 2026. Yang et al. (2025) An Yang et al. Qwen3 technical report. arXiv:2505.09388, 2025. Yang et al. (2024) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024. Yang et al. (2026a) Rongxin Yang, Yang Liu, Shang Luo, Haoxuan Jia, Chongyang Zhang, Hao Zheng, Yingguang Yang, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Kefu Xu, Congjing Ran, et al. Harness-agnostic detection and immunization of reward hacking in self-evolving language models. arXiv:2609.04665, 2026a. Yang et al. (2026b) Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, and Ion Stoica. FreeToken: Efficient edge-native MoE serving with bandwidth-adaptive execution. arXiv:2608.16157, 2026b. Inference stack used here at version 0.1.3. Yuan et al. (2025) Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. arXiv:2506.09501, 2025. Zhang et al. (2025) Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents. arXiv:2505.22954, 2025. Zhavoronkov et al. (2026) Alex Zhavoronkov, Vladimir Naumov, Denis Sidorenko, Alex Aliper, Vladimir Aladinskiy, Ramin Hasani, Alexander Amini, Katerina Nasto, Mathieu Reymond, Rim Shayakhmetov, Zulfat Miftakhutdinov, Vadim N. Gladyshev, and Fedor Galkin. An open benchmark and language models for AI in aging biology. Cell, 189:5980â5994.e8, 2026. doi: 10.1016/j.cell.2026.08.026. Zhou et al. (2023) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023. Zhu et al. (2025) Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv:2507.02825, 2025. Appendix A Background: the case study this study follows The case study. The earlier study (v0.5 in the projectâs records) ran a harness search around a frozen Qwen3-1.7B longevity model, served in its tied variant by FreeToken (Yang et al., 2026b). The search started from a parent configuration, CFG-THINK-64-ORIG (thinking on, 64 tokens for thinking and answer), and, driven by its failures on one shard, produced B1, a conditional rescue: when the first call used nearly all of its tokens, call again with a no-think suffix. B1âs gain in parse rate replicated on a second shard. A later audit found a simpler comparator, thinking disabled, that was as accurate as B1; it then found that the projectâs caller had sent each benchmark rowâs reference answer to the model as a prior assistant turn in every historical call that could be checked. Prompt-token reconciliation, the serverâs prompt-token count against the local count of the intended request, was the evidence: the counts differed by the length of the reference turn. The corrected re-evaluation. After the caller was corrected, a preregistration was frozen (26 September 2026, 23:52:06 UTC) and run once on 556 holdout rows that no earlier experiment had called, though the comparator configurations had been defined after another line of experiments had called other holdout rows, among them 216 with a request equivalent to the no-think suffix. The parent parsed none of the 556 rows, so the preregistered reliability endpoint was nearly guaranteed. Thinking disabled was non-inferior to B1 at a margin of 0.10 (difference +0.012+0.012, interval [â0.028-0.028, +0.052+0.052]) at 47% of B1âs token cost; since every B1 first call reached the rescue threshold, B1 reduced to always adding the suffix. Post hoc, on LB-0030 no configuration clearly exceeded a constant guess of the age band most common in the two shards (band 30â45, accuracy 0.493, against 0.471 to 0.493 for the configurations). The case studyâs conclusion was that prospective replication controlled selection noise but detected neither the inadequate comparator nor the evaluation defect, and that trivial baselines, input audits and prompt-token reconciliation did. Table 7: Lessons of the case study and where this study tests them. Lesson Tested by The parent parsed none of the prospective rows; the rescue made answers visible H1, H3 A thinking-off setting that the search never tried was non-inferior to the rescue H2 The caller sent the reference answer H4, defect A LB-0042âs gold came from a fallback rule; the scorer could fail open H4, defects B1, B2 The searchâs split put a giant component into one partition H4, defect C The parent was a broken start H4, defect D The tied variant was served, not the checkpointâs stored output layer H4, defect E A constant guess matched the configurations on one task trivial baselines (Section 4) Why this study is not a replication of it. The stack (vLLM in batch-invariant mode, not FreeToken), the model variant (untied output layer), two of the three models, two of the three pools and five of the six configurations are new, and the rules were fixed in advance for all of them. It is not blind to direction: the pilot showed that the parent truncates where the budget is small, and the preregistration says so (Section 3). Appendix B Full tables Every number in this appendix is read from the analysis report committed in the authorsâ repository (the preregistered analysis) or from the three files computed after the outputs existed; Appendix E lists them. Intervals of primary endpoints are the family-adjusted ones, all others unadjusted 95%. Î , θ, DTD_T and DPD_P are accuracy or rate differences; V, B and L count the rows only AUTHORS-1500 parses, both parse and only THINK-512 parses. B.1 H1 to H3 Table 8: H1, primary endpoints: THINK-512 against AUTHORS-1500, test partition (family of 9). Repl.: R replicated, I inconclusive replication, N not replicated. Cell Î ÎV _V ÎB _B ÎL _L T V / B / L Verdict Repl. Qwen3 LB letters â0.003-0.003 [â0.018-0.018, +0.012+0.012] 0.0000.000 â0.003-0.003 0.0000.000 +0.002+0.002 [â0.009-0.009, +0.014+0.014] 1 / 3,941 / 0 no gain: not tested R Qwen3 MMLU-Pro +0.013+0.013 [â0.006-0.006, +0.033+0.033] +0.015+0.015 â0.003-0.003 0.0000.000 +0.005+0.005 [â0.008-0.008, +0.018+0.018] 48 / 751 / 0 no gain: not tested R Qwen3 GSM8K 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 0.0000.000 0.0000.000 0.0000.000 [0.0000.000, 0.0000.000] 0 / 523 / 0 no gain: not tested R SmolLM3 LB letters +0.060+0.060 [+0.046+0.046, +0.074+0.074] +0.071+0.071 +0.001+0.001 â0.012-0.012 +0.013+0.013 [+0.009+0.009, +0.018+0.018] 749 / 448 / 100 visibility dominates R SmolLM3 MMLU-Pro +0.255+0.255 [+0.207+0.207, +0.304+0.304] +0.282+0.282 â0.014-0.014 â0.013-0.013 +0.067+0.067 [+0.054+0.054, +0.081+0.081] 529 / 81 / 13 visibility dominates R SmolLM3 GSM8K +0.108+0.108 [+0.046+0.046, +0.173+0.173] +0.197+0.197 â0.036-0.036 â0.053-0.053 +0.068+0.068 [+0.047+0.047, +0.090+0.090] 141 / 340 / 29 visibility dominates R Intern-S1 LB letters +0.314+0.314 [+0.297+0.297, +0.331+0.331] +0.314+0.314 0.0000.000 0.0000.000 +0.063+0.063 [+0.059+0.059, +0.066+0.066] 2,407 / 1 / 0 visibility dominates R Intern-S1 MMLU-Pro +0.403+0.403 [+0.352+0.352, +0.456+0.456] +0.419+0.419 â0.016-0.016 0.0000.000 +0.097+0.097 [+0.084+0.084, +0.111+0.111] 730 / 69 / 0 visibility dominates R Intern-S1 GSM8K â0.287-0.287 [â0.355-0.355, â0.218-0.218] +0.057+0.057 â0.343-0.343 0.0000.000 +0.286+0.286 [+0.240+0.240, +0.332+0.332] 273 / 253 / 0 no gain: not tested R Table 9: H1, sanity checks: THINK-64 against AUTHORS-1500, test partition, at the familyâs adjusted level, outside the family and carrying no claim. Cell Î ÎV _V ÎB _B ÎL _L T V / B / L Verdict Repl. Qwen3 LB letters +0.632+0.632 [+0.613+0.613, +0.651+0.651] +0.632+0.632 0.0000.000 0.0000.000 +0.126+0.126 [+0.123+0.123, +0.130+0.130] 3,942 / 0 / 0 visibility dominates Qwen3 MMLU-Pro +0.089+0.089 [+0.063+0.063, +0.119+0.119] +0.089+0.089 0.0000.000 0.0000.000 +0.018+0.018 [+0.013+0.013, +0.024+0.024] 274 / 525 / 0 visibility dominates Qwen3 GSM8K +0.002+0.002 [0.0000.000, +0.009+0.009] +0.002+0.002 0.0000.000 0.0000.000 0.0000.000 [0.0000.000, +0.002+0.002] 1 / 522 / 0 no gain: not tested SmolLM3 LB letters +0.129+0.129 [+0.116+0.116, +0.143+0.143] +0.129+0.129 0.0000.000 0.0000.000 +0.026+0.026 [+0.023+0.023, +0.029+0.029] 1,197 / 0 / 0 visibility dominates SmolLM3 MMLU-Pro +0.353+0.353 [+0.307+0.307, +0.401+0.401] +0.353+0.353 0.0000.000 0.0000.000 +0.071+0.071 [+0.061+0.061, +0.080+0.080] 610 / 0 / 0 visibility dominates SmolLM3 GSM8K +0.789+0.789 [+0.738+0.738, +0.837+0.837] +0.789+0.789 0.0000.000 0.0000.000 +0.158+0.158 [+0.148+0.148, +0.167+0.167] 481 / 0 / 0 visibility dominates Intern-S1 LB letters +0.314+0.314 [+0.297+0.297, +0.331+0.331] +0.314+0.314 0.0000.000 0.0000.000 +0.063+0.063 [+0.059+0.059, +0.066+0.066] 2,408 / 0 / 0 visibility dominates Intern-S1 MMLU-Pro +0.477+0.477 [+0.427+0.427, +0.526+0.526] +0.477+0.477 0.0000.000 0.0000.000 +0.095+0.095 [+0.085+0.085, +0.105+0.105] 799 / 0 / 0 visibility dominates Intern-S1 GSM8K +0.190+0.190 [+0.144+0.144, +0.239+0.239] +0.190+0.190 0.0000.000 0.0000.000 +0.038+0.038 [+0.029+0.029, +0.048+0.048] 526 / 0 / 0 visibility dominates Table 10: H2, primary endpoints: θ of AUTHORS-1500 against RESCUE-1500 in nine cells and against the four GEPA harnesses (family of 13), with the replication partitionâs verdict and the label. Endpoint θ (adjusted interval) Verdict Replication verdict Label Qwen3 LB letters, RESCUE-1500 â0.003-0.003 [â0.018-0.018, +0.012+0.012] holds holds replicated Qwen3 MMLU-Pro, RESCUE-1500 â0.003-0.003 [â0.019-0.019, +0.014+0.014] holds holds replicated Qwen3 GSM8K, RESCUE-1500 0.0000.000 [0.0000.000, 0.0000.000] holds holds replicated SmolLM3 LB letters, RESCUE-1500 +0.001+0.001 [â0.007-0.007, +0.009+0.009] holds holds replicated SmolLM3 MMLU-Pro, RESCUE-1500 â0.025-0.025 [â0.044-0.044, â0.009-0.009] holds holds replicated SmolLM3 GSM8K, RESCUE-1500 â0.089-0.089 [â0.129-0.129, â0.053-0.053] fails inconclusive inconclusive replication Intern-S1 LB letters, RESCUE-1500 0.0000.000 [0.0000.000, 0.0000.000] holds holds replicated Intern-S1 MMLU-Pro, RESCUE-1500 â0.016-0.016 [â0.030-0.030, â0.005-0.005] holds holds replicated Intern-S1 GSM8K, RESCUE-1500 â0.342-0.342 [â0.402-0.402, â0.281-0.281] fails fails replicated Qwen3 LB letters, GEPA-CLEAN-s0 0.0000.000 [0.0000.000, 0.0000.000] holds holds replicated Qwen3 LB letters, GEPA-BROKEN-s0 +0.004+0.004 [â0.009-0.009, +0.016+0.016] holds holds replicated Qwen3 LB letters, GEPA-CLEAN-s1 +0.002+0.002 [â0.022-0.022, +0.025+0.025] holds holds replicated Qwen3 LB letters, GEPA-BROKEN-s1 +0.012+0.012 [â0.002-0.002, +0.026+0.026] holds holds replicated Table 11: H2, descriptive: θ of AUTHORS-1500 against three other configurations, test partition, unadjusted intervals and the verdict rule applied to them. Cell vs THINK-512 vs THINK-1536 vs BUDGET-512-1500 Qwen3 LB letters â0.003-0.003 [â0.013-0.013, +0.008+0.008] holds â0.003-0.003 [â0.013-0.013, +0.007+0.007] holds â0.003-0.003 [â0.013-0.013, +0.007+0.007] holds Qwen3 MMLU-Pro +0.013+0.013 [â0.001-0.001, +0.026+0.026] holds â0.005-0.005 [â0.018-0.018, +0.008+0.008] holds â0.003-0.003 [â0.015-0.015, +0.010+0.010] holds Qwen3 GSM8K 0.0000.000 [0.0000.000, 0.0000.000] holds 0.0000.000 [0.0000.000, 0.0000.000] holds 0.0000.000 [0.0000.000, 0.0000.000] holds SmolLM3 LB letters +0.060+0.060 [+0.050+0.050, +0.070+0.070] holds â0.028-0.028 [â0.040-0.040, â0.015-0.015] holds +0.009+0.009 [â0.003-0.003, +0.020+0.020] holds SmolLM3 MMLU-Pro +0.255+0.255 [+0.220+0.220, +0.290+0.290] holds +0.039+0.039 [+0.003+0.003, +0.075+0.075] holds +0.095+0.095 [+0.059+0.059, +0.131+0.131] holds SmolLM3 GSM8K +0.108+0.108 [+0.063+0.063, +0.154+0.154] holds â0.091-0.091 [â0.127-0.127, â0.055-0.055] fails â0.046-0.046 [â0.083-0.083, â0.008-0.008] inconclusive Intern-S1 LB letters +0.314+0.314 [+0.302+0.302, +0.326+0.326] holds +0.189+0.189 [+0.175+0.175, +0.204+0.204] holds +0.061+0.061 [+0.046+0.046, +0.076+0.076] holds Intern-S1 MMLU-Pro +0.403+0.403 [+0.367+0.367, +0.439+0.439] holds +0.198+0.198 [+0.161+0.161, +0.235+0.235] holds â0.065-0.065 [â0.104-0.104, â0.026-0.026] inconclusive Intern-S1 GSM8K â0.287-0.287 [â0.334-0.334, â0.237-0.237] fails â0.643-0.643 [â0.685-0.685, â0.600-0.600] fails â0.732-0.732 [â0.770-0.770, â0.693-0.693] fails Table 12: H3, primary endpoints: THINK-512 against BUDGET-512-1500, test partition (family of 9); the truncation rate of THINK-512 decides no room. Cell DTD_T DPD_P THINK-512 truncation Verdict Replication verdict Label Qwen3 LB letters 0.0000.000 [0.0000.000, +0.001+0.001] 0.0000.000 [0.0000.000, +0.001+0.001] 0.0000.000 [0.0000.000, +0.001+0.001] no room no room replicated Qwen3 MMLU-Pro +0.060+0.060 [+0.038+0.038, +0.085+0.085] +0.060+0.060 [+0.038+0.038, +0.085+0.085] +0.060+0.060 [+0.038+0.038, +0.085+0.085] inconclusive inconclusive replicated Qwen3 GSM8K 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] no room no room replicated SmolLM3 LB letters +0.428+0.428 [+0.407+0.407, +0.450+0.450] +0.151+0.151 [+0.136+0.136, +0.167+0.167] +0.725+0.725 [+0.708+0.708, +0.742+0.742] holds holds replicated SmolLM3 MMLU-Pro +0.526+0.526 [+0.476+0.476, +0.574+0.574] +0.248+0.248 [+0.205+0.205, +0.292+0.292] +0.905+0.905 [+0.875+0.875, +0.932+0.932] holds holds replicated SmolLM3 GSM8K +0.260+0.260 [+0.209+0.209, +0.315+0.315] +0.190+0.190 [+0.144+0.144, +0.239+0.239] +0.300+0.300 [+0.245+0.245, +0.357+0.357] holds holds replicated Intern-S1 LB letters +0.979+0.979 [+0.972+0.972, +0.985+0.985] +0.619+0.619 [+0.603+0.603, +0.636+0.636] +1.000+1.000 [+0.999+0.999, +1.000+1.000] holds holds replicated Intern-S1 MMLU-Pro +0.647+0.647 [+0.601+0.601, +0.693+0.693] +0.662+0.662 [+0.615+0.615, +0.708+0.708] +0.915+0.915 [+0.887+0.887, +0.941+0.941] holds holds replicated Intern-S1 GSM8K +0.493+0.493 [+0.435+0.435, +0.554+0.554] +0.497+0.497 [+0.438+0.438, +0.556+0.556] +0.522+0.522 [+0.461+0.461, +0.581+0.581] holds holds replicated B.2 H4 Table 13: H4, the 16 pairs with their adjusted intervals (family of 16, level 99.69%) and, in parentheses, which side of its boundary each interval lies on (holds, fails or open). Stability of Î is a descriptive fact about the same interval. Pair Bound DSD_S DRD_R Î Stability of Î Claim A Qwen3 1.000 â0.052-0.052 [â0.106-0.106, +0.002+0.002] (open) â0.030-0.030 [â0.087-0.087, +0.030+0.030] (open) +0.022+0.022 [â0.057-0.057, +0.102+0.102] (open) inconclusive inconclusive A SmolLM3 1.000 +0.097+0.097 [+0.022+0.022, +0.173+0.173] (open) +0.119+0.119 [+0.042+0.042, +0.199+0.199] (open) +0.022+0.022 [â0.089-0.089, +0.133+0.133] (open) inconclusive inconclusive A Intern-S1 1.000 +0.142+0.142 [+0.082+0.082, +0.201+0.201] (holds) +0.164+0.164 [+0.102+0.102, +0.226+0.226] (holds) +0.022+0.022 [â0.065-0.065, +0.108+0.108] (open) inconclusive inconclusive B1 Qwen3 0.205 â0.145-0.145 [â0.170-0.170, â0.118-0.118] (holds) â0.148-0.148 [â0.166-0.166, â0.130-0.130] (holds) â0.003-0.003 [â0.035-0.035, +0.028+0.028] (holds) stable holds B1 SmolLM3 0.083 +0.001+0.001 [â0.019-0.019, +0.021+0.021] (fails) +0.007+0.007 [â0.006-0.006, +0.021+0.021] (fails) +0.006+0.006 [â0.018-0.018, +0.031+0.031] (holds) stable fails B1 Intern-S1 0.057 â0.012-0.012 [â0.028-0.028, +0.004+0.004] (fails) â0.019-0.019 [â0.030-0.030, â0.007-0.007] (fails) â0.007-0.007 [â0.027-0.027, +0.013+0.013] (holds) stable fails B2 Qwen3 0.000 not evaluable: the start leaves no row unparsed â B2 SmolLM3 0.208 +0.208+0.208 [+0.181+0.181, +0.235+0.235] (holds) +0.223+0.223 [+0.203+0.203, +0.243+0.243] (holds) +0.015+0.015 [â0.018-0.018, +0.048+0.048] (holds) stable holds B2 Intern-S1 0.289 +0.289+0.289 [+0.260+0.260, +0.319+0.319] (holds) +0.292+0.292 [+0.271+0.271, +0.313+0.313] (holds) +0.003+0.003 [â0.033-0.033, +0.037+0.037] (holds) stable holds C Qwen3 by rule +0.072+0.072 [+0.004+0.004, +0.138+0.138] (open) +0.101+0.101 [+0.047+0.047, +0.157+0.157] (open) +0.030+0.030 [â0.057-0.057, +0.117+0.117] (open) inconclusive inconclusive C SmolLM3 by rule â0.001-0.001 [â0.052-0.052, +0.052+0.052] (open) +0.005+0.005 [â0.048-0.048, +0.063+0.063] (fails) +0.006+0.006 [â0.070-0.070, +0.082+0.082] (open) inconclusive fails C Intern-S1 by rule +0.003+0.003 [â0.046-0.046, +0.054+0.054] (open) +0.004+0.004 [â0.045-0.045, +0.057+0.057] (open) +0.001+0.001 [â0.070-0.070, +0.074+0.074] (open) inconclusive inconclusive D Qwen3 1.000 +0.638+0.638 [+0.610+0.610, +0.668+0.668] (holds) +0.635+0.635 [+0.614+0.614, +0.656+0.656] (holds) â0.003-0.003 [â0.039-0.039, +0.032+0.032] (holds) stable holds D SmolLM3 0.352 +0.164+0.164 [+0.142+0.142, +0.187+0.187] (holds) +0.157+0.157 [+0.142+0.142, +0.173+0.173] (holds) â0.007-0.007 [â0.034-0.034, +0.020+0.020] (holds) stable holds D Intern-S1 0.189 +0.124+0.124 [+0.104+0.104, +0.145+0.145] (holds) +0.125+0.125 [+0.110+0.110, +0.140+0.140] (holds) +0.001+0.001 [â0.025-0.025, +0.026+0.026] (holds) stable holds E Qwen3 0.139 +0.005+0.005 [â0.016-0.016, +0.025+0.025] (fails) +0.014+0.014 [â0.002-0.002, +0.029+0.029] (fails) +0.009+0.009 [â0.017-0.017, +0.034+0.034] (holds) stable fails Table 14: The detection matrix (descriptive; largely true by construction). Checks: PR prospective replication, TR token reconciliation, TB trivial baselines, SCF simple configuration family, IR input reading, SMI served-model identity. Pair PR TR TB SCF IR SMI Rules fired (expected ones in bold) A Qwen3 U D U U U â IR1, IR2, IR4, PR.gain, SCF2, TB2, TB3, TR A SmolLM3 D D U U U â IR1, IR2, IR4, PR.gain, SCF1, SCF2, SCF3, TB2, TR; expected, not fired: TB3 A Intern-S1 M D U U U â IR1, IR2, IR4, SCF1, SCF3, TB2, TB3, TR B1 Qwen3 M M M M U â IR3, IR4 B1 SmolLM3 M M M U U â IR3, IR4, SCF1 B1 Intern-S1 M M M U U â IR3, IR4, SCF1, SCF2, SCF3 B2 Qwen3 D M D D U â IR4, PR.gain, SCF2, TB4 B2 SmolLM3 D M D U U â IR4, PR.gain, SCF1, SCF2, TB4 B2 Intern-S1 M M D U U â IR4, SCF1, TB4 C Qwen3 D M D D U â IR4, IR5, PR.gain, SCF2, TB2; expected, not fired: TB1 C SmolLM3 M M D U U â IR4, IR5, SCF1, TB2; expected, not fired: TB1 C Intern-S1 M M D U U â IR4, IR5, SCF1, SCF2, SCF3, TB2; expected, not fired: TB1 D Qwen3 M M M D U â IR4, SCF1, SCF2, SCF3 D SmolLM3 M M M U U â IR4, SCF1, SCF2, SCF3 D Intern-S1 M M M U U â IR4, SCF1, SCF2, SCF3 E Qwen3 D M M U U D IR4, PR.level, SCF2, SMI D detects: fires on the defective instance and not on the clean one (5, 3, 6, 3, 0, 1 instances for PR, TR, TB, SCF, IR, SMI); U uninformative: fires on both (1, 0, 3, 12, 16, 0); M misses. Table 15: False alarms of each rule on the clean instances (the main clean instance and the clean counterparts of A and E, per model). Check Rule Fired on Of PR PR.gain 1: clean/A / Qwen3 7 PR PR.level 0: none 7 TR TR 0: none 7 TB TB1 0: none 7 TB TB2 3: clean/A / Intern-S1, clean/A / Qwen3, clean/A / SmolLM3 7 TB TB3 0: none 7 TB TB4 0: none 7 SCF SCF1 4: clean / Intern-S1, clean / SmolLM3, clean/A / Intern-S1, clean/A / SmolLM3 7 SCF SCF2 4: clean / Intern-S1, clean/A / Intern-S1, clean/A / Qwen3, clean/E / Qwen3 7 SCF SCF3 2: clean / Intern-S1, clean/A / Intern-S1 7 IR IR1 0: none 7 IR IR2 0: none 7 IR IR3 0: none 7 IR IR4 7: clean / Intern-S1, clean / Qwen3, clean / SmolLM3, clean/A / Intern-S1, clean/A / Qwen3, clean/A / SmolLM3, clean/E / Qwen3 7 IR IR5 0: none 7 B.3 Configurations Table 16: Accuracy, parse, truncation and non-commitment rates of the six configurations in the nine cells, test partition, with unadjusted 95% intervals. Unparsed answers count as wrong. The trivial baselines of the three pools are in Table 6 and Appendix B.4. Cell Configuration Accuracy Parse Truncated Non-committed Qwen3 LB letters THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.635 [0.621, 0.649] 1.000 [0.999, 1.000] 0.000 [0.000, 0.001] 0.000 [0.000, 0.000] THINK-1536 0.635 [0.621, 0.649] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] AUTHORS-1500 0.632 [0.618, 0.646] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] BUDGET-512-1500 0.635 [0.621, 0.649] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] RESCUE-1500 0.635 [0.621, 0.649] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] Qwen3 MMLU-Pro THINK-64 0.140 [0.116, 0.165] 0.657 [0.625, 0.690] 0.343 [0.310, 0.375] 0.000 [0.000, 0.000] THINK-512 0.217 [0.189, 0.245] 0.940 [0.922, 0.956] 0.060 [0.044, 0.078] 0.000 [0.000, 0.000] THINK-1536 0.234 [0.205, 0.264] 0.997 [0.994, 1.000] 0.003 [0.000, 0.006] 0.000 [0.000, 0.000] AUTHORS-1500 0.229 [0.200, 0.259] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] BUDGET-512-1500 0.232 [0.203, 0.262] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] RESCUE-1500 0.232 [0.203, 0.262] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] Qwen3 GSM8K THINK-64 0.068 [0.047, 0.091] 0.991 [0.981, 0.998] 0.006 [0.000, 0.013] 0.004 [0.000, 0.009] THINK-512 0.070 [0.049, 0.093] 0.992 [0.985, 0.998] 0.000 [0.000, 0.000] 0.008 [0.002, 0.015] THINK-1536 0.070 [0.049, 0.093] 0.992 [0.985, 0.998] 0.000 [0.000, 0.000] 0.008 [0.002, 0.015] AUTHORS-1500 0.070 [0.049, 0.093] 0.992 [0.985, 0.998] 0.000 [0.000, 0.000] 0.008 [0.002, 0.015] BUDGET-512-1500 0.070 [0.049, 0.093] 0.992 [0.985, 0.998] 0.000 [0.000, 0.000] 0.008 [0.002, 0.015] RESCUE-1500 0.070 [0.049, 0.093] 0.992 [0.985, 0.998] 0.000 [0.000, 0.000] 0.008 [0.002, 0.015] SmolLM3 LB letters THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.070 [0.063, 0.077] 0.139 [0.131, 0.147] 0.725 [0.713, 0.737] 0.138 [0.128, 0.148] THINK-1536 0.157 [0.147, 0.168] 0.339 [0.326, 0.352] 0.485 [0.472, 0.498] 0.181 [0.170, 0.193] AUTHORS-1500 0.129 [0.120, 0.139] 0.304 [0.292, 0.315] 0.023 [0.019, 0.028] 0.677 [0.665, 0.688] BUDGET-512-1500 0.121 [0.112, 0.130] 0.290 [0.278, 0.302] 0.297 [0.283, 0.310] 0.424 [0.409, 0.438] RESCUE-1500 0.128 [0.119, 0.138] 0.308 [0.297, 0.319] 0.020 [0.016, 0.024] 0.675 [0.663, 0.686] SmolLM3 MMLU-Pro THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.098 [0.078, 0.119] 0.118 [0.096, 0.140] 0.905 [0.884, 0.925] 0.000 [0.000, 0.000] THINK-1536 0.314 [0.282, 0.347] 0.461 [0.426, 0.496] 0.526 [0.491, 0.561] 0.024 [0.014, 0.035] AUTHORS-1500 0.353 [0.320, 0.387] 0.763 [0.733, 0.792] 0.179 [0.153, 0.207] 0.105 [0.084, 0.126] BUDGET-512-1500 0.258 [0.228, 0.288] 0.365 [0.333, 0.399] 0.379 [0.345, 0.413] 0.304 [0.273, 0.337] RESCUE-1500 0.378 [0.344, 0.412] 0.778 [0.750, 0.807] 0.169 [0.144, 0.195] 0.096 [0.076, 0.118] SmolLM3 GSM8K THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.681 [0.641, 0.721] 0.700 [0.660, 0.738] 0.300 [0.262, 0.340] 0.000 [0.000, 0.000] THINK-1536 0.880 [0.852, 0.907] 0.934 [0.911, 0.954] 0.066 [0.046, 0.089] 0.000 [0.000, 0.000] AUTHORS-1500 0.789 [0.753, 0.824] 0.913 [0.888, 0.935] 0.023 [0.011, 0.036] 0.074 [0.053, 0.097] BUDGET-512-1500 0.835 [0.803, 0.865] 0.890 [0.863, 0.917] 0.040 [0.025, 0.057] 0.072 [0.051, 0.095] RESCUE-1500 0.879 [0.850, 0.905] 0.968 [0.953, 0.981] 0.011 [0.004, 0.021] 0.023 [0.011, 0.036] Intern-S1 LB letters THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.000 [0.000, 0.000] 0.000 [0.000, 0.001] 1.000 [0.999, 1.000] 0.000 [0.000, 0.000] THINK-1536 0.125 [0.115, 0.135] 0.192 [0.181, 0.203] 0.800 [0.789, 0.811] 0.016 [0.012, 0.020] AUTHORS-1500 0.314 [0.302, 0.326] 0.611 [0.604, 0.618] 0.006 [0.003, 0.008] 0.384 [0.376, 0.391] BUDGET-512-1500 0.253 [0.241, 0.266] 0.619 [0.608, 0.631] 0.021 [0.016, 0.025] 0.363 [0.351, 0.374] RESCUE-1500 0.314 [0.302, 0.326] 0.611 [0.604, 0.618] 0.006 [0.003, 0.008] 0.384 [0.376, 0.391] Intern-S1 MMLU-Pro THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.074 [0.056, 0.093] 0.086 [0.068, 0.106] 0.915 [0.895, 0.934] 0.000 [0.000, 0.000] THINK-1536 0.279 [0.249, 0.310] 0.345 [0.313, 0.378] 0.652 [0.620, 0.685] 0.009 [0.003, 0.016] AUTHORS-1500 0.477 [0.442, 0.512] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] BUDGET-512-1500 0.542 [0.507, 0.576] 0.748 [0.718, 0.778] 0.268 [0.238, 0.299] 0.033 [0.021, 0.045] RESCUE-1500 0.493 [0.458, 0.528] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] Intern-S1 GSM8K THINK-64 0.000 [0.000, 0.000] 0.000 [0.000, 0.000] 1.000 [1.000, 1.000] 0.000 [0.000, 0.000] THINK-512 0.476 [0.435, 0.518] 0.480 [0.436, 0.524] 0.522 [0.480, 0.564] 0.000 [0.000, 0.000] THINK-1536 0.833 [0.801, 0.863] 0.852 [0.822, 0.882] 0.148 [0.118, 0.178] 0.000 [0.000, 0.000] AUTHORS-1500 0.190 [0.157, 0.224] 0.998 [0.994, 1.000] 0.000 [0.000, 0.000] 0.002 [0.000, 0.006] BUDGET-512-1500 0.922 [0.899, 0.945] 0.977 [0.964, 0.989] 0.028 [0.015, 0.044] 0.004 [0.000, 0.009] RESCUE-1500 0.531 [0.488, 0.573] 0.998 [0.994, 1.000] 0.000 [0.000, 0.000] 0.002 [0.000, 0.006] B.4 Trivial baselines, computed after the outputs existed Chance is the mean of 1/k1/k over the rows (k options). A constant letter or a constant label is the same answer on every row; the non-oracle constant is the most common gold letter or label of the other partition, the oracle constant the most common one of the partition itself (ties to the smallest). On LB-0030 and LB-0042 the order of the options is shuffled from row to row, so a constant letter cannot exploit an uneven label distribution and a constant label can. The strongest baseline of a task is the larger of its two oracle constants; the poolâs values are row-weighted means. For MMLU-Pro and GSM8K (addendum A12) the constants are a letter and an answer (the most common gold answer of the other partition, and of the partition itself); the strongest baseline is the oracle constant, and a free-form number has no chance level. Cells give accuracy and, in parentheses, the paired difference to the strongest oracle baseline; intervals for the pool are in Table 6, the rest in the files of the authorsâ repository (Appendix E). Table 17: Accuracy and difference to the strongest baseline (oracle constant label), test partition. Model THINK-64 THINK-512 THINK-1536 AUTHORS-1500 BUDGET-512-1500 RESCUE-1500 LB-0030 (1,577 rows). Chance 0.200; constant letter 0.216 (non-oracle), 0.216 (oracle); constant label 0.242 (non-oracle), 0.250 (oracle). Strongest, oracle: 0.250 [0.230, 0.269]. Qwen3 0.000 (â0.250-0.250) 0.369 (+0.119+0.119) 0.369 (+0.119+0.119) 0.344 (+0.094+0.094) 0.369 (+0.119+0.119) 0.369 (+0.119+0.119) SmolLM3 0.000 (â0.250-0.250) 0.000 (â0.250-0.250) 0.021 (â0.229-0.229) 0.053 (â0.197-0.197) 0.032 (â0.218-0.218) 0.053 (â0.197-0.197) Intern-S1 0.000 (â0.250-0.250) 0.000 (â0.250-0.250) 0.004 (â0.245-0.245) 0.023 (â0.227-0.227) 0.055 (â0.195-0.195) 0.023 (â0.227-0.227) Qwen3, GEPA CLEAN-s0 0.344 (+0.094+0.094); BROKEN-s0 0.353 (+0.103+0.103); CLEAN-s1 0.388 (+0.138+0.138); BROKEN-s1 0.321 (+0.072+0.072) LB-0034 (788 rows). Chance 0.500; constant letter 0.485 (non-oracle), 0.515 (oracle); constant label 0.485 (non-oracle), 0.515 (oracle). Strongest, oracle: 0.515 [0.480, 0.551]. Qwen3 0.000 (â0.515-0.515) 0.702 (+0.187+0.187) 0.703 (+0.188+0.188) 0.717 (+0.202+0.202) 0.703 (+0.188+0.188) 0.703 (+0.188+0.188) SmolLM3 0.000 (â0.515-0.515) 0.307 (â0.208-0.208) 0.409 (â0.107-0.107) 0.443 (â0.072-0.072) 0.355 (â0.160-0.160) 0.435 (â0.080-0.080) Intern-S1 0.000 (â0.515-0.515) 0.000 (â0.515-0.515) 0.242 (â0.273-0.273) 0.575 (+0.060+0.060) 0.523 (+0.008+0.008) 0.575 (+0.060+0.060) Qwen3, GEPA CLEAN-s0 0.717 (+0.202+0.202); BROKEN-s0 0.702 (+0.187+0.187); CLEAN-s1 0.716 (+0.201+0.201); BROKEN-s1 0.716 (+0.201+0.201) LB-0042 (1,577 rows). Chance 0.500; constant letter 0.500 (non-oracle), 0.500 (oracle); constant label 0.840 (non-oracle), 0.840 (oracle). Strongest, oracle: 0.840 [0.822, 0.858]. Qwen3 0.000 (â0.840-0.840) 0.867 (+0.027+0.027) 0.867 (+0.027+0.027) 0.878 (+0.038+0.038) 0.867 (+0.027+0.027) 0.867 (+0.027+0.027) SmolLM3 0.000 (â0.840-0.840) 0.020 (â0.820-0.820) 0.167 (â0.673-0.673) 0.049 (â0.791-0.791) 0.093 (â0.748-0.748) 0.050 (â0.790-0.790) Intern-S1 0.000 (â0.840-0.840) 0.000 (â0.840-0.840) 0.186 (â0.654-0.654) 0.475 (â0.365-0.365) 0.317 (â0.523-0.523) 0.475 (â0.365-0.365) Qwen3, GEPA CLEAN-s0 0.878 (+0.038+0.038); BROKEN-s0 0.867 (+0.027+0.027); CLEAN-s1 0.831 (â0.010-0.010); BROKEN-s1 0.871 (+0.030+0.030) Letter pool (3,942 rows). Chance 0.380; constant letter 0.383 (non-oracle), 0.389 (oracle); constant label 0.530 (non-oracle), 0.539 (oracle). Strongest, oracle: 0.539 [0.527, 0.551]. Qwen3 0.000 (â0.539-0.539) 0.635 (+0.096+0.096) 0.635 (+0.096+0.096) 0.632 (+0.093+0.093) 0.635 (+0.096+0.096) 0.635 (+0.096+0.096) SmolLM3 0.000 (â0.539-0.539) 0.070 (â0.470-0.470) 0.157 (â0.382-0.382) 0.129 (â0.410-0.410) 0.121 (â0.418-0.418) 0.128 (â0.411-0.411) Intern-S1 0.000 (â0.539-0.539) 0.000 (â0.539-0.539) 0.125 (â0.414-0.414) 0.314 (â0.225-0.225) 0.253 (â0.286-0.286) 0.314 (â0.225-0.225) Qwen3, GEPA CLEAN-s0 0.632 (+0.093+0.093); BROKEN-s0 0.629 (+0.090+0.090); CLEAN-s1 0.631 (+0.092+0.092); BROKEN-s1 0.620 (+0.081+0.081) Table 18: The same for the replication partition. Model THINK-64 THINK-512 THINK-1536 AUTHORS-1500 BUDGET-512-1500 RESCUE-1500 LB-0030 (796 rows). Chance 0.200; constant letter 0.212 (non-oracle), 0.212 (oracle); constant label 0.255 (non-oracle), 0.270 (oracle). Strongest, oracle: 0.270 [0.242, 0.299]. Qwen3 0.000 (â0.270-0.270) 0.358 (+0.088+0.088) 0.358 (+0.088+0.088) 0.335 (+0.065+0.065) 0.358 (+0.088+0.088) 0.358 (+0.088+0.088) SmolLM3 0.000 (â0.270-0.270) 0.000 (â0.270-0.270) 0.025 (â0.245-0.245) 0.041 (â0.229-0.229) 0.031 (â0.239-0.239) 0.041 (â0.229-0.229) Intern-S1 0.000 (â0.270-0.270) 0.000 (â0.270-0.270) 0.005 (â0.265-0.265) 0.011 (â0.259-0.259) 0.046 (â0.224-0.224) 0.011 (â0.259-0.259) Qwen3, GEPA CLEAN-s0 0.335 (+0.065+0.065); BROKEN-s0 0.345 (+0.075+0.075); CLEAN-s1 0.389 (+0.119+0.119); BROKEN-s1 0.307 (+0.036+0.036) LB-0034 (398 rows). Chance 0.500; constant letter 0.490 (non-oracle), 0.510 (oracle); constant label 0.490 (non-oracle), 0.510 (oracle). Strongest, oracle: 0.510 [0.462, 0.558]. Qwen3 0.000 (â0.510-0.510) 0.741 (+0.231+0.231) 0.744 (+0.234+0.234) 0.744 (+0.234+0.234) 0.744 (+0.234+0.234) 0.744 (+0.234+0.234) SmolLM3 0.000 (â0.510-0.510) 0.322 (â0.188-0.188) 0.420 (â0.090-0.090) 0.432 (â0.078-0.078) 0.369 (â0.141-0.141) 0.462 (â0.048-0.048) Intern-S1 0.000 (â0.510-0.510) 0.000 (â0.510-0.510) 0.226 (â0.284-0.284) 0.585 (+0.075+0.075) 0.500 (â0.010-0.010) 0.585 (+0.075+0.075) Qwen3, GEPA CLEAN-s0 0.744 (+0.234+0.234); BROKEN-s0 0.729 (+0.219+0.219); CLEAN-s1 0.741 (+0.231+0.231); BROKEN-s1 0.716 (+0.206+0.206) LB-0042 (796 rows). Chance 0.500; constant letter 0.489 (non-oracle), 0.511 (oracle); constant label 0.823 (non-oracle), 0.823 (oracle). Strongest, oracle: 0.823 [0.795, 0.850]. Qwen3 0.000 (â0.823-0.823) 0.866 (+0.043+0.043) 0.866 (+0.043+0.043) 0.876 (+0.053+0.053) 0.866 (+0.043+0.043) 0.866 (+0.043+0.043) SmolLM3 0.000 (â0.823-0.823) 0.016 (â0.807-0.807) 0.176 (â0.647-0.647) 0.045 (â0.778-0.778) 0.082 (â0.741-0.741) 0.043 (â0.780-0.780) Intern-S1 0.000 (â0.823-0.823) 0.000 (â0.823-0.823) 0.192 (â0.631-0.631) 0.480 (â0.343-0.343) 0.320 (â0.503-0.503) 0.480 (â0.343-0.343) Qwen3, GEPA CLEAN-s0 0.876 (+0.053+0.053); BROKEN-s0 0.877 (+0.054+0.054); CLEAN-s1 0.843 (+0.020+0.020); BROKEN-s1 0.882 (+0.059+0.059) Letter pool (1,990 rows). Chance 0.380; constant letter 0.378 (non-oracle), 0.391 (oracle); constant label 0.529 (non-oracle), 0.539 (oracle). Strongest, oracle: 0.539 [0.519, 0.559]. Qwen3 0.000 (â0.539-0.539) 0.638 (+0.098+0.098) 0.638 (+0.099+0.099) 0.633 (+0.094+0.094) 0.638 (+0.099+0.099) 0.638 (+0.099+0.099) SmolLM3 0.000 (â0.539-0.539) 0.071 (â0.468-0.468) 0.164 (â0.375-0.375) 0.121 (â0.418-0.418) 0.119 (â0.420-0.420) 0.126 (â0.413-0.413) Intern-S1 0.000 (â0.539-0.539) 0.000 (â0.539-0.539) 0.124 (â0.415-0.415) 0.314 (â0.226-0.226) 0.247 (â0.292-0.292) 0.314 (â0.226-0.226) Qwen3, GEPA CLEAN-s0 0.633 (+0.094+0.094); BROKEN-s0 0.635 (+0.095+0.095); CLEAN-s1 0.641 (+0.102+0.102); BROKEN-s1 0.619 (+0.079+0.079) Table 19: MMLU-Pro and GSM8K, test partition: accuracy of the six configurations and, in parentheses, the paired difference to the strongest baseline (the oracle constant). Model THINK-64 THINK-512 THINK-1536 AUTHORS-1500 BUDGET-512-1500 RESCUE-1500 MMLU-Pro (799 items). Chance 0.110; constant letter B, chosen on the other partition and oracle alike, 0.119 [0.096, 0.141]. Qwen3 0.140 (+0.021+0.021) 0.217 (+0.098+0.098) 0.234 (+0.115+0.115) 0.229 (+0.110+0.110) 0.232 (+0.113+0.113) 0.232 (+0.113+0.113) SmolLM3 0.000 (â0.119-0.119) 0.098 (â0.021-0.021) 0.314 (+0.195+0.195) 0.353 (+0.234+0.234) 0.258 (+0.139+0.139) 0.378 (+0.259+0.259) Intern-S1 0.000 (â0.119-0.119) 0.074 (â0.045-0.045) 0.279 (+0.160+0.160) 0.477 (+0.358+0.358) 0.542 (+0.423+0.423) 0.493 (+0.374+0.374) GSM8K (527 items). Constant answer 10 (chosen on the other partition) 0.019 [0.008, 0.032], 2 (oracle) 0.034 [0.019, 0.051]; no chance level. Qwen3 0.068 (+0.034+0.034) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) 0.070 (+0.036+0.036) SmolLM3 0.000 (â0.034-0.034) 0.681 (+0.647+0.647) 0.880 (+0.846+0.846) 0.789 (+0.755+0.755) 0.835 (+0.801+0.801) 0.879 (+0.844+0.844) Intern-S1 0.000 (â0.034-0.034) 0.476 (+0.442+0.442) 0.833 (+0.799+0.799) 0.190 (+0.156+0.156) 0.922 (+0.888+0.888) 0.531 (+0.497+0.497) Table 20: MMLU-Pro and GSM8K, replication partition. Model THINK-64 THINK-512 THINK-1536 AUTHORS-1500 BUDGET-512-1500 RESCUE-1500 MMLU-Pro (399 items). Chance 0.111; constant letter B, chosen on the other partition and oracle alike, 0.133 [0.100, 0.168]. Qwen3 0.108 (â0.025-0.025) 0.180 (+0.048+0.048) 0.193 (+0.060+0.060) 0.180 (+0.048+0.048) 0.190 (+0.058+0.058) 0.185 (+0.053+0.053) SmolLM3 0.000 (â0.133-0.133) 0.090 (â0.043-0.043) 0.333 (+0.201+0.201) 0.378 (+0.246+0.246) 0.291 (+0.158+0.158) 0.398 (+0.266+0.266) Intern-S1 0.000 (â0.133-0.133) 0.105 (â0.028-0.028) 0.293 (+0.160+0.160) 0.446 (+0.313+0.313) 0.566 (+0.434+0.434) 0.454 (+0.321+0.321) GSM8K (264 items). Constant answer 2 (chosen on the other partition) 0.030 [0.011, 0.053], 10 (oracle) 0.045 [0.023, 0.072]; no chance level. Qwen3 0.057 (+0.011+0.011) 0.057 (+0.011+0.011) 0.057 (+0.011+0.011) 0.057 (+0.011+0.011) 0.057 (+0.011+0.011) 0.057 (+0.011+0.011) SmolLM3 0.000 (â0.045-0.045) 0.705 (+0.659+0.659) 0.909 (+0.864+0.864) 0.814 (+0.769+0.769) 0.830 (+0.784+0.784) 0.886 (+0.841+0.841) Intern-S1 0.000 (â0.045-0.045) 0.462 (+0.417+0.417) 0.845 (+0.799+0.799) 0.189 (+0.144+0.144) 0.936 (+0.890+0.890) 0.542 (+0.496+0.496) B.5 Other descriptive items Table 21: Seeded sampling at the modelsâ card values (test partition): parse and truncation rates, and the difference seeded minus zero temperature with unadjusted intervals. Model Pool Configuration Parse (seeded) Truncated (seeded) Î accuracy Î parse Î truncation Qwen3 LB letters THINK-64 0.000 1.000 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] THINK-512 0.999 0.001 â0.013-0.013 [â0.026-0.026, â0.001-0.001] 0.0000.000 [â0.001-0.001, +0.001+0.001] 0.0000.000 [â0.001-0.001, +0.001+0.001] THINK-1536 1.000 0.000 â0.014-0.014 [â0.027-0.027, â0.002-0.002] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] BUDGET-512-1500 1.000 0.000 â0.018-0.018 [â0.030-0.030, â0.005-0.005] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] MMLU-Pro THINK-64 0.641 0.359 â0.005-0.005 [â0.026-0.026, +0.016+0.016] â0.016-0.016 [â0.039-0.039, +0.005+0.005] +0.016+0.016 [â0.005-0.005, +0.039+0.039] THINK-512 0.932 0.068 â0.004-0.004 [â0.029-0.029, +0.021+0.021] â0.008-0.008 [â0.026-0.026, +0.011+0.011] +0.008+0.008 [â0.011-0.011, +0.026+0.026] THINK-1536 0.997 0.003 +0.004+0.004 [â0.021-0.021, +0.029+0.029] 0.0000.000 [â0.004-0.004, +0.004+0.004] 0.0000.000 [â0.004-0.004, +0.004+0.004] BUDGET-512-1500 0.999 0.000 â0.014-0.014 [â0.039-0.039, +0.011+0.011] â0.001-0.001 [â0.004-0.004, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] GSM8K THINK-64 0.991 0.006 â0.025-0.025 [â0.044-0.044, â0.008-0.008] 0.0000.000 [â0.006-0.006, +0.006+0.006] 0.0000.000 [â0.006-0.006, +0.006+0.006] THINK-512 0.998 0.000 â0.011-0.011 [â0.028-0.028, +0.006+0.006] +0.006+0.006 [0.0000.000, +0.013+0.013] 0.0000.000 [0.0000.000, 0.0000.000] THINK-1536 0.994 0.000 â0.011-0.011 [â0.030-0.030, +0.008+0.008] +0.002+0.002 [â0.004-0.004, +0.009+0.009] 0.0000.000 [0.0000.000, 0.0000.000] BUDGET-512-1500 0.996 0.000 â0.008-0.008 [â0.027-0.027, +0.011+0.011] +0.004+0.004 [0.0000.000, +0.009+0.009] 0.0000.000 [0.0000.000, 0.0000.000] Intern-S1 LB letters THINK-64 0.000 1.000 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] THINK-512 0.003 0.997 +0.002+0.002 [+0.001+0.001, +0.004+0.004] +0.003+0.003 [+0.001+0.001, +0.005+0.005] â0.003-0.003 [â0.005-0.005, â0.001-0.001] THINK-1536 0.192 0.810 â0.010-0.010 [â0.022-0.022, +0.002+0.002] â0.001-0.001 [â0.015-0.015, +0.014+0.014] +0.010+0.010 [â0.005-0.005, +0.025+0.025] BUDGET-512-1500 0.705 0.033 +0.042+0.042 [+0.026+0.026, +0.058+0.058] +0.086+0.086 [+0.068+0.068, +0.103+0.103] +0.013+0.013 [+0.006+0.006, +0.020+0.020] MMLU-Pro THINK-64 0.000 1.000 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] THINK-512 0.100 0.902 +0.008+0.008 [â0.011-0.011, +0.028+0.028] +0.014+0.014 [â0.008-0.008, +0.035+0.035] â0.013-0.013 [â0.034-0.034, +0.009+0.009] THINK-1536 0.350 0.641 +0.001+0.001 [â0.024-0.024, +0.026+0.026] +0.005+0.005 [â0.023-0.023, +0.033+0.033] â0.011-0.011 [â0.039-0.039, +0.016+0.016] BUDGET-512-1500 0.690 0.284 â0.076-0.076 [â0.111-0.111, â0.041-0.041] â0.059-0.059 [â0.095-0.095, â0.023-0.023] +0.016+0.016 [â0.014-0.014, +0.046+0.046] GSM8K THINK-64 0.000 1.000 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] THINK-512 0.387 0.619 â0.101-0.101 [â0.146-0.146, â0.057-0.057] â0.093-0.093 [â0.139-0.139, â0.047-0.047] +0.097+0.097 [+0.051+0.051, +0.142+0.142] THINK-1536 0.801 0.199 â0.049-0.049 [â0.083-0.083, â0.015-0.015] â0.051-0.051 [â0.087-0.087, â0.017-0.017] +0.051+0.051 [+0.017+0.017, +0.087+0.087] BUDGET-512-1500 0.954 0.038 â0.038-0.038 [â0.065-0.065, â0.013-0.013] â0.023-0.023 [â0.042-0.042, â0.006-0.006] +0.009+0.009 [â0.009-0.009, +0.028+0.028] Table 22: H3âs statistics recomputed on the seeded outputs (the cardsâ sampling values, a seed per request) for the two models with a seeded arm; descriptive. Ďg _g and Ďs _s are the shares of THINK-512 calls cut at the token limit with greedy decoding and seeded, and ĎsâĎg _s- _g their paired difference. DTD_T and DPD_P are the seeded drop in truncation and gain in parsing of BUDGET-512-1500 against THINK-512, with unadjusted intervals. The preregistered verdicts use family-adjusted intervals at another level and are not compared with these. Partition Model Pool Ďg _g Ďs _s ĎsâĎg _s- _g DTD_T (seeded) DPD_P (seeded) Test Qwen3 LB letters 0.000 0.001 0.0000.000 [â0.001-0.001, +0.001+0.001] +0.001+0.001 [0.0000.000, +0.001+0.001] +0.001+0.001 [0.0000.000, +0.001+0.001] MMLU-Pro 0.060 0.068 +0.008+0.008 [â0.011-0.011, +0.026+0.026] +0.068+0.068 [+0.051+0.051, +0.085+0.085] +0.066+0.066 [+0.050+0.050, +0.084+0.084] GSM8K 0.000 0.000 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] â0.002-0.002 [â0.006-0.006, 0.0000.000] Intern-S1 LB letters 1.000 0.997 â0.003-0.003 [â0.005-0.005, â0.001-0.001] +0.963+0.963 [+0.957+0.957, +0.969+0.969] +0.702+0.702 [+0.689+0.689, +0.716+0.716] MMLU-Pro 0.915 0.902 â0.013-0.013 [â0.034-0.034, +0.009+0.009] +0.618+0.618 [+0.584+0.584, +0.652+0.652] +0.589+0.589 [+0.554+0.554, +0.625+0.625] GSM8K 0.522 0.619 +0.097+0.097 [+0.051+0.051, +0.142+0.142] +0.581+0.581 [+0.539+0.539, +0.622+0.622] +0.567+0.567 [+0.526+0.526, +0.609+0.609] Replication Qwen3 LB letters 0.001 0.000 â0.001-0.001 [â0.002-0.002, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] MMLU-Pro 0.060 0.065 +0.005+0.005 [â0.023-0.023, +0.033+0.033] +0.065+0.065 [+0.043+0.043, +0.090+0.090] +0.060+0.060 [+0.038+0.038, +0.085+0.085] GSM8K 0.000 0.000 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] 0.0000.000 [0.0000.000, 0.0000.000] Intern-S1 LB letters 1.000 0.995 â0.005-0.005 [â0.008-0.008, â0.002-0.002] +0.959+0.959 [+0.950+0.950, +0.967+0.967] +0.708+0.708 [+0.690+0.690, +0.725+0.725] MMLU-Pro 0.880 0.902 +0.023+0.023 [â0.010-0.010, +0.055+0.055] +0.622+0.622 [+0.574+0.574, +0.669+0.669] +0.589+0.589 [+0.539+0.539, +0.639+0.639] GSM8K 0.538 0.636 +0.098+0.098 [+0.045+0.045, +0.155+0.155] +0.614+0.614 [+0.553+0.553, +0.670+0.670] +0.602+0.602 [+0.542+0.542, +0.663+0.663] Table 23: Tied against untied output layer of Qwen3-1.7B-Longevity on FreeToken, the case studyâs six configurations, all four LongevityBench tasks pooled (8,305 rows in 1,188 participant components, unflagged rows of both partitions); computed after the outputs existed. Configuration Tied Untied Tied â- untied Parse t / u Same answers Only t / u right p CFG-THINK-64-ORIG 0.000 0.000 0.0000.000 [0.0000.000, 0.0000.000] 0.000 / 0.000 1.000 0 / 0 1.000 B1 0.452 0.449 +0.003+0.003 [â0.001-0.001, +0.007+0.007] 1.000 / 1.000 0.741 182 / 157 0.192 CFG-THINKING-FALSE-64-ORIG 0.460 0.460 0.0000.000 [â0.005-0.005, +0.005+0.005] 1.000 / 1.000 0.731 194 / 194 1.000 CFG-THINK-1536-ORIG 0.469 0.463 +0.006+0.006 [0.0000.000, +0.013+0.013] 1.000 / 1.000 0.656 384 / 333 0.062 CFG-NOTHINK-SUFFIX-64-ORIG 0.452 0.449 +0.003+0.003 [â0.001-0.001, +0.007+0.007] 1.000 / 1.000 0.741 182 / 157 0.192 CFG-THINK-64-ANSWERONLY 0.479 0.470 +0.009+0.009 [+0.004+0.004, +0.014+0.014] 1.000 / 1.000 0.731 260 / 185 0.000 Table 24: The four GEPA runs (computed after the outputs existed): proposals and candidates, refused by the interface check, kept by GEPA, metric calls, the selected candidate, its prompt lengths, its validation score and its accuracy on the letter rows of the replication and test partitions. Harness Proposals Candidates seen Refused Kept Metric calls Selected / kept Characters (system / instruction) Valset Replication acc. Test acc. Test parse GEPA-CLEAN-s0 207 204 28 31 6,039 0 / 31 103 / 0 0.680 0.633 0.632 [0.618, 0.646] 1.000 GEPA-BROKEN-s0 377 377 24 24 6,147 1 / 24 103 / 1,146 0.660 0.635 0.629 [0.615, 0.643] 1.000 GEPA-CLEAN-s1 195 192 36 31 6,003 12 / 31 2,469 / 922 0.700 0.641 0.631 [0.616, 0.645] 1.000 GEPA-BROKEN-s1 243 238 67 29 6,003 1 / 29 103 / 2,797 0.667 0.619 0.620 [0.606, 0.634] 1.000 Table 25: LB-0038 (age regression), test partition, descriptive. Model Measure THINK-64 THINK-512 THINK-1536 AUTHORS-1500 BUDGET-512-1500 RESCUE-1500 Qwen3 parse rate 0.000 1.000 1.000 1.000 1.000 1.000 mean abs. error (years) â 15.69 15.69 16.50 15.69 15.69 exact match 0.000 0.027 0.027 0.030 0.027 0.027 SmolLM3 parse rate 0.000 0.000 0.004 0.001 0.003 0.001 mean abs. error (years) â â 24.71 57.70 35.08 57.70 exact match 0.000 0.000 0.000 0.000 0.000 0.000 Intern-S1 parse rate 0.000 0.000 0.000 0.004 0.000 0.004 mean abs. error (years) â â â 16.57 â 16.57 exact match 0.000 0.000 0.000 0.000 0.000 0.000 Mean absolute error is over parsed rows; exact match over all rows. Constant median 50.0 years (chosen on replication): mean absolute error 16.41 (oracle median 16.41), exact match 0.013. B.6 Sensitivity analyses Table 26: Sensitivity analyses: the number of the primary endpoints whose verdict changes, and which. Analysis Endpoints Changed Which (primary verdict â here) Strict scorer 31 2 H2 SmolLM3 MMLU-Pro, RESCUE: holds â inconclusive; H1 SmolLM3 GSM8K: visibility dominates â no gain: not tested Never-called rows only 13 3 H2 Qwen3 letters, RESCUE: holds â inconclusive; H2 Qwen3 letters, GEPA-BROKEN-s1: holds â inconclusive; H1 SmolLM3 letters: visibility dominates â inconclusive Flagged rows included (H1âH3) 31 0 none Case studyâs pool (LB-0030 and LB-0034) 13 1 H2 Qwen3 letters, GEPA-CLEAN-s1: holds â inconclusive LB-0030 alone 13 2 H2 Qwen3 letters, RESCUE: holds â inconclusive; H2 Qwen3 letters, GEPA-CLEAN-s1: holds â inconclusive LB-0034 alone 13 0 none LB-0042 alone 13 0 none Role-break rule off 31 0 none Role-break rule, first version 31 0 none Replication and test pooled 31 0 none Flagged rows included (H4 claims) 16 2 C SmolLM3: fails â inconclusive; C Intern-S1: inconclusive â fails Table 27: Composition of the never-called rows (computed after the outputs existed) next to the analysis rows. The share deceased at follow-up is the share of LB-0042 rows whose follow-up turn says that the participant was deceased, whenever the death occurred; it is not the ten-year label of the task, whose gold is âYesâ for a participant who was alive at ten years or who died later. Rows Partition Rows Letter rows LB-0030 mean age LB-0042 share deceased at follow-up Never called replication 187 103 36.2 0.022 Analysis rows replication 1,990 (pool) â 50.8 0.345 Never called test 356 207 37.3 0.022 Analysis rows test 3,942 (pool) â 50.8 0.346 Appendix C Deviations, exceptions and addenda in full In the authorsâ repository (Appendix E) the frozen preregistration (docs/icml_ext/PREREG_ICML_EXT.md) records exceptions and known deviations before the freeze, together with its decisions of the days before it. After the freeze, departures are appended to docs/icml_ext/PREREG_ICML_EXT_ADDENDA.md, one entry each, under its section on deviation rules. This appendix lists them all: Table 28 gives the deviations that bear on a result, and the sections below give every exception, known deviation, pre-freeze decision and addendum. âSeenâ states what had been seen of the studyâs outputs when an entry was made. Table 28: Deviations and addenda that bear on a result; the entries below give the rest. âSeenâ is what had been seen of the studyâs outputs when the entry was made. Entry What happened Bearing on the results Seen Before the freeze The primary set of H1 changed twice (after the pilotâs parse rates, and after a review in Codex); the scoring defect was split in two; the tied-output-layer defect was redesigned; one evaluability rule was fixed; the family of H4 was fixed at 16; the margins were set to 0.05. Defines the endpoints; no study accuracy was used. pilot parse rates Rows left out 147 replication and 371 test rows that the case study had called, and 7 rows whose answers could be recovered from an upload made by mistake, were called but left out. Putting them back changes no verdict of H1âH3 and 2 H4 claims. no outputs A change of GPU After a restart the platform assigned another GPU of the same model; the determinism check rerun on the pilot rows passed, with outputs equal to the old GPUâs. None. 9.33 GPU-hours outside the budget. pilot rows Two script stops The chain script stopped twice during the GEPA search, both times from a fault in the script. None; no run was affected. none The thinking-switch check A check on Intern-S1-miniâs thinking switch stopped the run that sends the reference answer, and we let it record instead of stop. The repeat reproduced the first 3,700 calls token for token, and the check failed on 324 of the runâs 1,204 thinking-off calls and on none of 11,265 in the clean stages. The reference-answer defect in Intern-S1-mini only through the descriptive selection, since the claim reads the start. 6.46 idle server-hours counted. counts only The checkpoint 15 of the 16 pairs were evaluable; the pilot had suggested 14. The unparsed-as-A scoring in Qwen3-1.7B-Longevity is not evaluable; the gold fallback in Intern-S1-mini (bound 0.057) is evaluable. counts and bounds The determinism recheck A recheck of 600 test calls per model was identical on all three models. No verdict is relabelled descriptive. counts The analysis environment The preregistered analysis ran on the instance with numpy 2.3.5, not the local 2.2.6; the resampling stream was checked equal. None. all aggregates The claim rule The two listed cases of the claim rule for the check sample are incomplete, so the general rule governs, and the pinned code implements it. None: the verdicts are the codeâs, and the skewed split in SmolLM3-3B fails, a descriptive label. all aggregates Computed after the outputs The composition of the never-called rows, the trivial baselines of the three pools, the tied-versus-untied comparison, the GEPA candidates and the serving counts. Labelled, and they decide nothing. aggregates C.1 Exceptions accepted before the freeze (X1âX9) X1 Three RECORD entries of the vLLM virtual environment do not match their files (two PyNvVideoCodec libraries and build_backend.py); they date from the original installation, not from the clone. Decided 2026-09-28. X2 FreeToken 0.1.3 opens a process-group store on its serving port plus one that PyTorch binds off loopback; it carries no model traffic. The runner allows exactly this one listener, on FreeToken only, while no connection to it comes from off the machine. The case studyâs FreeToken runs very likely had it too. X3 FREETOKEN_ALLOW_CUDA_MISMATCH, the case studyâs variable, is exempt from the credential-name check although its name contains âtokenâ. X4 VLLM_USE_FLASHINFER_SAMPLER=0 in every vLLM run, because FlashInferâs sampling kernel could not compile against the imageâs CUDA toolkit. X5 A change of instance on 2026-09-28: the source instanceâs GPU was no longer available, so the instance was cloned to one of the same configuration; the GPU identifier changed and the clone was verified against the recorded freeze, wheel, record and model hashes. X6 A restart of the container at 2026-09-29 00:57:25 UTC, which we did not initiate; no server was running, and the GPU identifier and the SSH settings were unchanged. X7 The first gate check (G1) ran on the checkpointâs own directory, whose configuration declares tied embeddings, before the output-layer decision; its results were operational only. X8 AI-assisted readings and uploads (item-exposure contacts 1 and 2); the reading of the main export (contact 3) and the copy of the item files to the projectâs own instance (contact 4) are in Appendix F. X9 A change of the physical GPU after a restart (addenda A1, A2). C.2 Known deviations from the plans (K1âK9) K1 Three models instead of four; the FreeToken continuity on Qwen3-1.7B-Longevity only. K2 GEPA with a local proposer replaces Meta-Harness, so the proposer through an API and its budget fall away. K3 A single answer allowance of 1,500 tokens replaces a calibrated rule that left out the calls the calibration allowance cut. K4 The batch-invariant server with batching replaces concurrency one on the standard server (about 275 GPU-hours at concurrency one). K5 The pilotâs LongevityBench rows come from the case studyâs evolution file or validation shards, not from never-called rows; calibration reported no accuracy. K6 The extended scorer rules, their narrowing and the role-break rule were decided after the pilotâs parse audit and spot-checks on pilot items; the strict parser is a preregistered sensitivity analysis. K7 The GEPA interface changed between its two spikes, on the counts of refused proposals. K8 The environment that counts tokens locally differs from the instanceâs in the Python and numpy versions and other bundled tools; none takes part in tokenization, and the render lists were checked on both machines. K9 The pilot report was written after the first draft of the preregistration, from aggregates only. C.3 Decisions between the first draft and the freeze We revised the draft after our decisions on its first version, after a review in claude.ai, after two reviews and a final check in Codex (each given the file and not the authorsâ repository), after our decisions on the summary of the drafting session, and after the reading of the rendered requests. Table 28 calls these decisions âBefore the freezeâ. The decisions that shape the endpoints were these: ⢠The primary set of H1 changed twice, first after the pilotâs parse rates and parsed answers were seen (no accuracy was used) and then after the first review in Codex, with THINK-64 against AUTHORS-1500 moved out to nine sanity checks. ⢠The scoring defect B was split into B1 and B2. ⢠The tied-output-layer defect E was redesigned so that the confirmatory instance serves the tied variant on both samples. ⢠One evaluability rule applies to every defect, and the family of H4 is fixed at 16 whatever is evaluable. ⢠The margin and the minimal difference are 0.05, with Bonferroni within each family. After the reading, seven replication rows whose reference answers could be recovered from render-hash lists uploaded by mistake were flagged and left out of every analysis (the reading itself, item-exposure contact 3, concluded pass); Table 28 calls this âRows left outâ. The exclusion came after the last review in Codex and was not reviewed there. C.4 Addenda after the freeze (A1âA12) A1, A2 (2026-09-30) A change of GPU. After we restarted the instance, the platform had assigned a different physical GPU of the same model. By the preregistrationâs rules the work stopped before any copy or call. Every check passed on the new GPU; the determinism check (G2 (b)) was rerun on the pilot rows and passed for all three models, and its outputs equal the old GPUâs token for token. 9.33 GPU-hours, outside the budget; the new identifier is pinned. Seen: pilot rows only. A3 (2026-09-30) Two script stops. The chain script stopped twice during the GEPA search, both times from a fault in the script (a password check that failed under a shell option; a step that required an empty GPU while the task server stayed up). No run, call or check was affected. Seen: nothing of replication or test. A4, A5 (2026-10-01, 2026-10-02) The thinking-switch check. The defect-A pass of Intern-S1-mini stopped itself at gate (b): five of its first thinking-off calls (AUTHORS-1500) were not clean (cut at the limit with a think block opened and not closed). The gate held on all 3,849 thinking-off calls of the clean runs up to then. The preregistration had not foreseen that the injected defect could trigger the gate. We decided that in that stage only the gate records instead of stopping (commit aae8b11, runner a9478cee), that the idle server stops and its hours are counted (about 6.46 of its server-hours were idle), that the partial folder is moved out of the run root, and that the run is repeated from scratch under a new name. The repeat reproduced all 3,700 calls of the failed attempt token for token and failed the gate on 324 of the passâs 1,204 thinking-off calls (the gate reads the completion, and all 1,204 prompts carry think tags that the chat template writes around the reference turn). By the repair rules this is confirmatory: the failed attempt is kept and reported, and only the repeat enters the analysis. Seen: counts only. A6 (2026-10-02) The checkpoint. It ran on the instance, since the local machine lacked memory. All 17 replication runs were complete with matching manifests; the derived rescue equals the real two-call sequence on 300 of 300 verification items per model; 15 of 16 pairs are evaluable; and the selections of the pipelines whose selection sample lies in replication were locked in a record (f033f23f) whose hash was committed before the first test call. Seen: counts, bounds and the recordâs hash. A7 (2026-10-02, 2026-10-03) The determinism recheck. The test partition ran 14 runs and 267,056 calls, with no retry and no stop, and the recheck of 600 calls per model was identical on all three; the study used 54.53 of the 77 GPU-hours of the cost bound. Seen: counts. A8 (2026-10-03) The analysis environment. The preregistered analysis ran once on the instance (numpy 2.3.5, not the local 2.2.6; the resampling stream was checked equal for three seeds) after every pinned file matched its frozen hash, and the selections it reports equal the locked ones on all 24 comparisons. The code that computes three descriptive items the pinned driver lacks (tied versus untied, GEPA candidates, serving counts) was written after the outputs existed and is labelled so. Seen: all aggregates, for the first time. A9 (2026-10-03) Computed after the outputs. We shut the instance down and do not release it; the report and its tables were committed as aggregates (49e5f9c); the never-called rowsâ composition (543 rows, 310 on the letter tasks) and constant-letter baselines were computed after the outputs existed. A10 (2026-10-03) Computed after the outputs. Constant-label baselines, because a constant letter is a weak floor where the option order is shuffled: the strongest baseline of the pool is 0.539 on test against 0.389 for the oracle constant letter. The strongest baseline is the one used in this paper. A11 (2026-10-03) The claim rule. The rule for DRD_R in the claim of H4. The audit of this paper recomputes the claim from its three adjusted intervals; the two sub-cases that the preregistration lists after its general rule were incomplete, the general rule governs, and the pinned code implements it (our decision). No verdict changes. Seen: all aggregates. Details below. A12 (2026-10-03) Computed after the outputs. The trivial baselines of MMLU-Pro (chance and the constant letter) and GSM8K (the constant answer), chosen on the other partition and oracle, next to every modelâs accuracy in every grid configuration (holdout contact #57). Seen: aggregates, no item key or text. C.5 The rule for DRD_R (addendum A11) For DRD_R the preregistration says that it fails if its interval lies âwholly on the other side of its boundaryâ and lists two sub-cases, wholly within the band and wholly outside it with the opposite sign. The pinned code (scripts/icml_ext/analysis_h4.py, claim_verdict) implements the general phrase: DRD_R is graded in the direction of DSD_Sâs sign, holds if it lies wholly beyond the band there, and fails if its interval lies wholly on the other side of that one boundary, which also covers an interval partly within the band and partly beyond its far edge. The general rule governs and the two sub-cases were incomplete. It matters for one pair, C in SmolLM3-3B: DS=â0.001D_S=-0.001 points down, so DRD_R would have to lie below â0.05-0.05, and its adjusted interval [â0.048-0.048, +0.063+0.063] lies wholly above it, so the claim cannot hold and is fails, a descriptive label. Read as the two sub-cases only, the claim would be inconclusive, and H4 would count 6 holds, 3 fails and 6 inconclusive instead of 6, 4 and 5; every other claim has the same verdict under both readings. C.6 What this paper adds This paperâs text, tables and figures format aggregates committed in the authorsâ repository and add no computation except the counts of its own tables (verdict counts by family, the largest and smallest of a column) and the independent recomputation of every verdict from its interval, which agrees with the report on every endpoint that has a verdict (Appendix E). Appendix D Expected in advance against observed The preregistration states what it expected for several endpoints, from the pilotâs parse and truncation rates and parsed answers (never from accuracy). Table 29 sets each expectation next to what was observed. The directions of H1 and H3 were visible in the pilot, and the preregistration says so; this table shows the places where the outcome differed from the expectation, which are as informative as the places where it did not. âInconclusiveâ in the Expected column is an informal expectation read off planned power under assumed effects (Appendix K), not a prediction with a decision rule. Table 29: Expected in advance (preregistration, the sections on H1, H2 and H4) and observed. Item Expected Observed H1 sanity checks (THINK-64) T>0T>0 in eight cells; Qwen3-1.7B-Longevity on GSM8K fails the gate visibility dominates in 8 of nine cells; Qwen3-1.7B-Longevity on GSM8K fails the gate. As expected. H1, Qwen3-1.7B-Longevity (three cells) Fails the gate (on MMLU-Pro, or quality matters) no gain: not tested in all three. As expected. H1, Intern-S1-mini on the letters T>0T>0 nearly inevitable visibility dominates, T=+0.063T=+0.063. As expected. H1, SmolLM3-3B and Intern-S1-mini on GSM8K Inconclusive on T (planned power 0.52 and 0.00 for SmolLM3-3B, 0.45 and 0.06 for Intern-S1-mini, in the two scenarios) SmolLM3-3B: visibility dominates, T=+0.068T=+0.068 [+0.047+0.047, +0.090+0.090]. Intern-S1-mini: the gate fails, Î=â0.287 =-0.287. Both differ. H2, Intern-S1-mini on GSM8K against RESCUE-1500 Inconclusive (planned adjusted power 0.19) fails, θ=â0.342θ=-0.342 [â0.402-0.402, â0.281-0.281]. Differs. H2, GEPA harnesses Power unknown in advance; may well be inconclusive holds for all four; all intervals contain zero. H4, evaluable pairs 14 of 16: B1 in Intern-S1-mini and B2 in Qwen3-1.7B-Longevity not evaluable 15 of 16: only B2 in Qwen3-1.7B-Longevity is not; B1 in Intern-S1-mini has bound 0.057. Differs. H4, bounds on the pilotâs 214 letter rows against S B1 0.201, 0.098, 0.028; B2 0.000, 0.196, 0.276; D 1.000, 0.327, 0.150; E 0.159 (Qwen3-1.7B-Longevity, SmolLM3-3B, Intern-S1-mini) B1 0.205, 0.083, 0.057; B2 0.000, 0.208, 0.289; D 1.000, 0.352, 0.189; E 0.139. H4, D The claim rests on Î and holds only if its point estimate is within about 0.01 of zero (about 0.50 of the time with no true difference) Holds in all three models, Î=â0.003 =-0.003, â0.007-0.007, +0.001+0.001. Consistent with the expectation, which allowed it. H4, B2 Can be large in SmolLM3-3B and Intern-S1-mini DS=+0.208D_S=+0.208 and +0.289+0.289. As expected. H4, E No expectation on its size DS=+0.005D_S=+0.005, DR=+0.014D_R=+0.014; fails. Detection matrix Each defect detected by its signature rules (A: TR, IR1, IR2, TB3; B1: IR3; B2: TB4; C: IR4, IR5, TB1, not TB2 or PR; D: SCF1, SCF3; E: SMI) All expected rules fired in 12 of 16 instances. TB1 did not fire in any of the three C instances, TB3 did not fire for A in SmolLM3-3B, and TB2 fired in all three C instances, with the gain rule of PR in Qwen3-1.7B-Longevity. Appendix E Reproducibility Every path, commit, tag and SHA256 value in this paper is that of a file in the authorsâ repository, which is not public at present (see the statement on code and data availability after the Conclusion). They are kept so that the planned public export can be checked against them; a reader cannot open these files until the export is linked. The frozen record. In the authorsâ repository the preregistration (docs/icml_ext/PREREG_ICML_EXT.md) was frozen at 06:53:11 UTC on 30 September 2026, in commit 7f645ae, whose parent carries the tag pre-icml-ext-freeze; the tag prereg-icml-ext-frozen marks the freeze. Its SHA256 is dec5dc25fdd0074a973926767a7a7f378e3dbc06eaccbf976d6df6f84d2f515 The file docs/icml_ext/PREREG_ICML_EXT.sha256 holds the hash and the time, and the addenda are in docs/icml_ext/PREREG_ICML_EXT_ADDENDA.md. The statement on code and data availability says which files are attached to this arXiv version and why the freeze time cannot be checked independently. The hash above lets a reader check the ancillary preregistration, and the reportâs hash below does the same for the report. The outputs. The report of the preregistered analysis and its tables are committed as aggregates in the authorsâ repository, in docs/icml_ext/ANALYSIS_20261003/: the report has SHA256 1447c936d52523a1494169f30af239c7d36ce7e79e81655a6a91016570c22e0b, the folderâs manifest 37d3268a980974af6ab414af65b8b2b1a5b3d6071632f8f1867ffc3d871a8403. The items computed after the outputs are in three subfolders (the never-called rows and the constant-letter baselines; the constant-label baselines; the baselines of MMLU-Pro and GSM8K), each with its own manifest. The studyâs 17 replication runs and 14 test runs, the GEPA search and the failed attempt of A4 hold item text and model output, are not committed and will not be released; each has a manifest of SHA256 values that matched its files after the copy back. The code. The files of the authorsâ repository that fix the analysis are pinned in the preregistration by SHA256: scripts/icml_ext/analysis_stats.py (e6a2c4dc), analysis_records.py (7507fd6c), analysis_h4.py (2f5a82e7), analysis_run.py (d92ebfe2), defects.py (a2ad7e2e), scoring.py (3bce48ef) and runner_checks.py (ec77476e), and the caller scripts/paper_review/messages_for_inference.py (c8debc7). The runner changed once after the freeze (addendum A4, commit aae8b11). The environment is vLLM 0.30.0 with the packages listed in the setup log (docs/icml_ext/SERVER_SETUP_LOG.md); the analysis ran with numpy 2.3.5. vLLM pull request 41674 (merged 2026-05-15) fixed a thinking-token budget that was silently ignored (JasonKeyiL (2026), vLLM project); the version used contains the fix, and H3 shows the budget acting. This paperâs numbers. Every number of the text is a value macro ( applied to a key) that tools/values.py registers with the file of the authorsâ repository that it reads (at git HEAD, never the working tree) and the path inside it; tools/gen.py writes common/numbers.tex and the tables from that registry, and --check fails if a committed generated file would change. audit/number_audit.py then (i) re-reads every registered value with its own path reader and formatter, (i) recomputes the verdict of every endpoint that has one (all but the unevaluable pair of the 47) from its interval with the preregistered rules and compares it with the reportâs, (i) fails on any number in the hand-written text that is not a macro or an identifier, (iv) checks every hash, path and tag named in the text against the authorsâ repository, and (v) checks the plotted values of the figures (tools/make_figures.py). audit/CLAIM_AUDIT.md maps each claim of the text to its evidence. Appendix F Exposure and disclosure Every contact of the authors or their tools with held-out material is logged in the authorsâ repository (docs/paper_review/HOLDOUT_EXPOSURE_LOG.md, numbered contacts; docs/icml_ext/ITEM_EXPOSURE_LOG.md, contacts with the extensionâs items by outside services), and before it happens, with one exception that Table 30 marks. The case studyâs holdout contacts are #1 to #38; those of this study are #39 to #58 and #60. Other later contacts, including the preparation of the ancillary files and the posting of this version, are logged in the repository and are not listed here. Item-exposure contacts 1 and 2 are uploads of the pilotâs reading export and runner and of a spot-check and the GEPA code to claude.ai; contact 3 is the AI-assisted reading of the main export (concluded pass; one of us relayed the reading and its conclusion and also checked the conclusion); contact 4 is the copy of the main item files to the projectâs own GPU instance, which is not an outside service. Table 30: Holdout contacts of this study (#39 to #58 and #60): what was read or done. Entries #50 to #58 and #60 are after the freeze. # Contact #39 The ICML formatting draft of the case study copied to a synchronised folder for an upload. #40 The LongevityBench split of this study, the earlier-contact flags and the first gold audit built. #41 Fourteen files uploaded to claude.ai for a review of the reading export and the runner (item-exposure contact 1). #42 The preregistration draft revised and uploaded for reviews in claude.ai and in Codex, after a check for holdout-derived values. #43 The main item files built. #44 The LB-0042 rows on which B1âs fallback gold differs from the derived gold counted. #45 The gold audit rerun over the main items with the final scorer. #46 The main reading export run. #47 The three render-hash lists uploaded to claude.ai by mistake together with the export (see below). #48 The main render-hash lists pinned and checked locally. #49 Seven replication rows flagged as reading-exposed and the counts redone. #50 The main item files, the GEPA armâs rows, the render lists and the frozen code copied to the instance. #51 Counts-only scripts over the stopped run of Intern-S1-miniâs defect-A pass (addendum A4). #52 The checkpoint run on the instance (A6). #53 Counts of the test partitionâs 14 runs (A7). #54 The preregistered analysis (A8). #55 The report and its tables committed as aggregates; the never-called rowsâ composition and the constant-letter baselines computed (A9). #56 The constant-label baselines computed (A10). #57 The trivial baselines of MMLU-Pro and GSM8K computed (A12). #58 The named preprint (its PDF and LaTeX source, nothing else) copied to a folder outside the repository and given to Codex for an independent review at about 04:23 Pacific time (11:23 UTC) on 2026-10-03; the review returned 16 findings, which this version applies without new computation. #60 The PDF of the named preprint, as built at 03:42 Pacific time (10:42 UTC) on 2026-10-03, reviewed in claude.ai before the copy of #58 was made. It was missed at the time and logged afterwards. The mistaken upload (#47) and its consequence. One of us uploaded the three render-hash lists to the claude.ai conversation of the reading, together with the export. A defect-A entry is the hash of a request that carries the rowâs reference answer, so for a row whose clean request is known, the answer can be recovered by testing each candidate against the hash. Seven replication rows are both in the readingâs sample and among the defect-A rows, and we flagged them and left them out of every analysis, whether or not anyone tried. Nothing else in the replication or test partitions was disclosed this way. The readerâs statement, relayed by one of us and not checked independently, is that the reading did not open the lists and did not attempt a reconstruction. Files derived from held-out rows. This paper and the committed analysis report carry aggregates that include the rows flagged v21_holdout (the sensitivity analysis), so both are listed in docs/paper_review/HOLDOUT_DERIVED_FILES.md of the authorsâ repository, and they hold no row index, key or content. The run logs, which hold item text, stay off the authorsâ repository and will not be released. We drafted the paper with Claude Code and reviewed it in claude.ai and in Codex. Contact #58 is the copy given to Codex, which was the two files of the named preprint and not the repository, a run log or any item text, and contact #60 is the PDF reviewed in claude.ai. Any further upload of the draft is a new contact, logged first. Appendix G Serving details Servers. The main study ran each model on a vLLM server started by a pinned script with these flags (the model directory, served name and chat-template file differ by model; Intern-S1-mini adds --trust-remote-code and a limit of zero images and videos):vllm serve <model directory> --host 127.0.0.1 --port 8000 --dtype bfloat16 --served-model-name <name> --chat-template <pinned file> --generation-config vllm --reasoning-parser qwen3 --reasoning-config â"reasoning_start_str": "<think>", "reasoning_end_str": "</think>"â --no-enable-prefix-caching --no-enable-chunked-prefill --max-model-len 4096 with the environment cleared to a fixed list of variables, among them VLLM_BATCH_INVARIANT=1 and VLLM_USE_FLASHINFER_SAMPLER=0 (exception X4). Every request sets thinking on or off explicitly, carries a seed (irrelevant at zero temperature) and asks for the token IDs and the rendered prompt; the runner compares both with its own render on every call and stops on a difference, except in the defect-A pass, where the reference answer makes them differ by design. Each stage runs its start checks before the first call and again after the last (no credential-like variable name, no connection off the machine, loopback-only listening, the model directory and the hash of its configuration, the template, the flags, and the render check against the pinned list), and a watcher samples the serverâs connections from launch. The watchers flagged nothing on the vLLM servers; on each FreeToken server they flagged only the single listener of exception X2. Cost. The GEPA search used 6.63 GPU-hours, the replication partition 22.07 and the test partition 25.83, 54.53 of the 77 of the cost bound; the rerun of the determinism check on the new GPU (9.33 hours) is outside it. Per-run counts. Table 31 gives, per run and configuration, the calls, the disagreements between the serverâs reasoning split and the local one (logged on every call and never a stop; in the pilot they all fell after a forced end of thinking, or where the limit cut a multibyte character, and were equal once the four tags were removed), the calls in which a start-of-turn marker re-opened a turn (role breaks, all in SmolLM3-3Bâs BUDGET-512-1500), and mean tokens and seconds per call. The counts include every row called, flagged rows and LB-0038 among them. Table 31: Serving counts of the zero-temperature grid. Partition Model Configuration Calls Split disagreements Role breaks Prompt tokens Completion tokens Seconds Replication Qwen3 THINK-64 3,603 0 0 480 56 1.0 THINK-512 3,603 3 0 480 257 3.7 THINK-1536 3,603 3 0 480 258 3.7 AUTHORS-1500 3,603 3 0 484 3 0.2 BUDGET-512-1500 3,603 4 0 480 257 3.6 SmolLM3 THINK-64 3,603 0 0 500 64 1.8 THINK-512 3,603 0 0 500 461 10.8 THINK-1536 3,603 0 0 500 1093 26.4 AUTHORS-1500 3,603 0 0 472 297 7.2 BUDGET-512-1500 3,603 2,431 2,177 500 1148 28.1 Intern-S1 THINK-64 3,603 1 0 493 64 3.0 THINK-512 3,603 1 0 493 502 17.7 THINK-1536 3,603 0 0 493 1294 47.0 AUTHORS-1500 3,603 0 0 480 651 23.6 BUDGET-512-1500 3,603 233 0 493 799 29.1 Test Qwen3 THINK-64 7,216 0 0 482 56 1.0 THINK-512 7,216 0 0 482 256 3.6 THINK-1536 7,216 0 0 482 257 3.7 AUTHORS-1500 7,216 0 0 486 2 0.2 BUDGET-512-1500 7,216 2 0 482 256 3.6 SmolLM3 THINK-64 7,216 0 0 501 64 1.8 THINK-512 7,216 2 0 501 461 10.8 THINK-1536 7,216 0 0 501 1107 26.6 AUTHORS-1500 7,216 0 0 473 311 7.5 BUDGET-512-1500 7,216 4,893 4,384 501 1163 28.5 Intern-S1 THINK-64 7,216 0 0 494 64 3.0 THINK-512 7,216 2 0 494 502 17.7 THINK-1536 7,216 2 0 494 1296 47.2 AUTHORS-1500 7,216 0 0 482 644 23.4 BUDGET-512-1500 7,216 474 0 494 804 29.3 Appendix H How the related work was searched We searched arXiv, Semantic Scholar, OpenAlex, Hugging Face papers, GitHub and venue pages from 2024-01 to 2026-10-03, with topic queries only (no item content), starting from the seed papers and their references and citers, in six topic slices: harness and agent-system evaluation; optimisers and agent baselines; reasoning evaluation under token budgets; evaluation-pipeline defects; inference engines and checkpoint identity; preregistration and replication. The agents behind the slices made 1,006 web calls. A shared cap of 200 web searches ran out during the search, the arXiv and Semantic Scholar interfaces were rate-limited, and three of the six slices (reasoning evaluation, inference engines, pipeline defects) were stopped by a rate limit and resumed to write up what they had already read, without new searches. âWe found no study thatâ therefore means that none was found, not that none exists, and most recent work on harness evolution is an unreviewed preprint. Identifiers, titles, authors and dates of the cited papers were checked against arXiv. The statements in Section 2 about a paperâs design were checked against its text for the papers marked as read in the matrix in the authorsâ repository (docs/icml_ext/RELATED_WORK_MATRIX.md), and otherwise rest on its abstract. For the GitHub pull requests cited, the merge date and the release that contains the merge commit were checked through the GitHub interface. Appendix I H4 in detail This appendix restates, from the frozen preregistration, what Section 3.4 summarises: the quantity each claim reports, how each defect is built and what it is compared with, the rows of its samples, the bound that decides whether its pair is evaluable, and the predicate and threshold of every check. The reported quantity and the claim. For an instance I (a defect and a model) and its clean counterpart K, the reported quantity is the accuracy of a fixed configuration, the pipelineâs start s, on the letter pool: aS=accSâ(s)a_S=acc_S(s) on the selection sample S and aR=accRâ(s)a_R=acc_R(s) on the check sample R. The start is THINK-1536 (for E, the case studyâs CFG-THINK-1536-ORIG), and THINK-64 in the instance of D. No claim selects anything. Then DS=aSâ(I)âaSâ(K)D_S=a_S(I)-a_S(K), DR=aRâ(I)âaRâ(K)D_R=a_R(I)-a_R(K) and Î=DRâDS =D_R-D_S. The claim holds if the adjusted interval of DSD_S lies wholly outside ÂąmÂą m, that of DRD_R lies wholly outside ÂąmÂą m on the side of DSD_Sâs point estimate, and that of Î lies within ÂąmÂą m. It fails if any of the three lies wholly on the other side of its boundary (DSD_Sâs wholly within ÂąmÂą m; DRD_Râs wholly within ÂąmÂą m, or wholly outside it on the opposite side; Î âs wholly outside ÂąmÂą m), and it is inconclusive otherwise. Here m=0.05m=0.05 and the intervals have the adjusted level of the family of 16; an interval that fails decides the claim even if another is still open, and the error control covers âholdsâ only. Defect D and the selection. For D the reported quantity is the gain g=accâĄ(câ)âaccâĄ(s)g=acc(c^*)-acc(s) of the selected configuration câc^* over the start, on S and on R. The selection picks câc^* as the most accurate member on S of the search space (THINK-64, THINK-512, THINK-1536, RESCUE-1500 and BUDGET-512-1500; ties go to the earlier member), and AUTHORS-1500 lies outside it, so that it can act as the simple comparator that a search never considered. Because câc^* does not depend on the start, the instance of D and its counterpart select the same câc^* in every resample, the selection cancels, and DS=accSâ(THINK-1536)âaccSâ(THINK-64)D_S=acc_S( THINK-1536)-acc_S( THINK-64), with the same difference on R for DRD_R. The selection itself, the gains it reports and every interval that follows a selection are descriptive and enter no claim. Defects, comparators and samples. Table 32 gives for each defect how it is built, the samples S and R, the clean counterpart it is compared with and how the two are resampled, and the bound of its pair. All samples leave out the flagged rows (Section 3). Where an instance and its counterpart share their rows, one resample of the units serves both, so DSD_S and DRD_R are paired; Î pairs the b-th resample of S with the b-th of R, drawn with their own seeds. Table 32: The six defects of H4: construction, samples, clean counterpart and evaluability bound. The bound is the largest distortion the defectâs mechanism can cause on S, computed without accuracy. Construction Samples S and R Clean counterpart and resampling Evaluability bound A The rowâs reference answer is sent as a final assistant turn, as the case studyâs caller did: new calls, built by a separate module. 1,204 replication rows drawn by whole component (seed 20261012) in proportion to the task counts (344, 172, 344 and 344 rows of LB-0030, LB-0034, LB-0038 and LB-0042), halved by component into S and R (86 components and 602 rows each; after the flags 562 and 565 rows in 81 components each). Both halves are called in one pass, so R is a disjoint half, not a later sample. The gridâs own outputs on the same rows, in the same halves; one resample serves both. One: the leaked reference can change any row. B1 Offline: the clean outputs are rescored with LB-0042âs gold taken from the case studyâs historical fallback, which with this releaseâs labels differs from the derived gold. Unparsed answers stay wrong; other tasks keep the releaseâs gold. S: the replication partitionâs letter rows; R: the test partitionâs, called after the checkpoint. The fallback changes the gold of 408 of 796 LB-0042 rows of S and 794 of 1,577 of R. The main clean instance on the same rows; one resample serves both. The share of Sâs letter rows that the start parses and whose gold B1 changes (exact; B1 changes the gold of 0.2050 of them). B2 Offline: the clean outputs are rescored by a scorer that never fails closed: an unparsed or conflicting letter answer is scored as the letter A, on every letter task, with the releaseâs gold. As B1. As B1. The share of Sâs letter rows that the start leaves unparsed or conflicting and whose gold is A (exact; it equals B2âs distortion of the startâs accuracy). C The case studyâs skewed split, re-implemented. The pooled LongevityBench rows of both partitions that are not flagged (8,305 of 8,830) are reassigned: components (from all six NHANES tasks, chaining into one giant component that goes to the evolution part) are sorted by size, largest first, and each goes to the part furthest below its target share (evolution 0.50, validation 0.30, holdout 0.20). S and R are the two halves of the validation part: its components are sorted by identifier, shuffled (seed 20261016) and dealt alternately. Their rows come from both partitions, so they are not the replication and test rows of the other pairs, and C, computed after the test calls, is not temporally prospective. The main clean instance, whose S and R are the studyâs replication and test rows. The rows differ, so the two are resampled independently, each with its own seed. None: C is evaluable by rule, because its mechanism changes the sample and has no row-level bound. D Offline: the start is THINK-64 in place of THINK-1536; the reported quantity is the gain over the start. S: the replication partitionâs letter rows; R: the test partitionâs. The main clean instance; one resample serves both. The share of Sâs letter rows on which the parsed answers of the two starts differ, an unparsed or conflicting answer counting as one value. E The tied output layer is served in place of the untied one (Qwen3-1.7B-Longevity only): the case studyâs six configurations on FreeToken, for both variants, with CFG-THINK-1536-ORIG as the start. The confirmatory instance serves the tied variant on S and on R. S: the replication rows; R: the test rows. The untied variant on both samples; one resample serves both. A second instance (tied on S, untied on R) feeds only the descriptive detection matrix. The share of Sâs letter rows on which the tied and untied variantsâ parsed answers of the start differ. Evaluability. A pair is evaluable if its bound exceeds m (a bound of exactly m does not). Otherwise it is reported as not evaluable, with its bound, makes no claim and is not counted as failing, and the family stays at 16. The bounds were computed from parse, gold-change and answer-difference indicators on the replication data at the checkpoint, before the first test call. For B1 the start must parse the same rows in the instance and its counterpart, and for B2 the gold must be the releaseâs in both; the analysis stops if not. The checks and their rules. Each check is evaluated on the records of an instance, and âfiresâ means that one of its rules holds. The rules and thresholds were fixed before the study; their outcomes are descriptive, and what is empirical is how often they fire on clean instances (Section 4). The rules of PR and every interval that follows a selection on S are descriptive labels, not tests: the selection makes accSâ(câ)acc_S(c^*) and the gain optimistic. Tables 33 and 34 give each ruleâs predicate and threshold. Table 33: The rules of PR, TR and TB. câc^* is the selected configuration, s the start, gRg_R the gain on R. Rule Fires when PR, gain rule cââ sc^*â s and the lower bound of the 95% percentile interval of accRâ(câ)âaccRâ(s)acc_R(c^*)-acc_R(s) is at or below zero: the reported gain does not reproduce on R. PR, level rule The 95% percentile interval of accRâ(câ)âaccSâ(câ)acc_R(c^*)-acc_S(c^*) excludes zero, whether or not câ=sc^*=s; the b-th resample of S is paired with the b-th of R, each drawn with its own seed. TR On any call of the instance the serverâs prompt-token count differs from the local count of the clean render (the corrected callerâs render of the rowâs messages before the final assistant turn), or, on vLLM, its prompt token IDs differ from the clean renderâs. For FreeToken only the count is compared. TB1 A strong trivial baseline (the oracle constant, the frequency of the most frequent gold letter in the sample itself) exceeds 0.90 on a task in S or R. TB2 For some task, the trivial constant chosen on S (the most frequent gold letter there, ties to the smallest letter) scores on R more than 0.05 away from its score on S. TB3 On LB-0038, some configurationâs exact-match rate exceeds three times the best constant ageâs, in S or R. TB4 On a letter task in S or R, a configuration has at least 50 unparsed or conflicting calls, and every one of them is scored correct exactly when the gold is one and the same constant letter: the scorer treats them as a constant predictor. Table 34: The rules of SCF, IR and SMI. gSg_S is the reported gain on S and ÎV _V its visibility part (Section 3, with the start as P and câc^* as R, point estimate on S). Rule Fires when SCF1 On S the startâs parse rate is below 0.90 or its truncation rate above 0.10. SCF2 cââ sc^*â s and accSâ(câ)acc_S(c^*) is less than 0.05 above the best of three panel members on S: chance, the strong trivial baseline and AUTHORS-1500 (in the instances of E, CFG-THINKING-FALSE-64-ORIG). For the letter pool, chance and the baseline are the row-weighted means of the tasksâ values. SCF3 cââ sc^*â s, gS>0g_S>0 and ÎV _V is at least 80% of gSg_S. IR1 Any request of the instance has an assistant turn, or a last message that is not a user turn. IR2 The gold screen hits on any request: the rowâs follow-up text, the LB-0038 age as a standalone integer, or another reference answer of more than one character appears in it. IR3 On any row the scorerâs gold is not among the golds derived from the question and follow-up (apart from the LB-0030 band-endpoint exception). IR4 For S or R against the release: a taskâs row share differs by more than 2 points from its release share; the LB-0042 deceased share or an LB-0030 band share differs by more than 2 points; or the standardised mean difference of the actual age exceeds 0.05. IR5 A binary task has a class below 10% of its rows in S or R, or an LB-0030 band is absent from S or R. SMI (E only) A served directoryâs configuration declares tied word embeddings while its checkpointâs output matrix differs from the embedding matrix; or, on the same rows and request bytes, the two variantsâ parsed answers differ on more than 1% of rows in some configuration. The second rule applies only where two variants of one checkpoint are served, which is Qwen3-1.7B-Longevity. Appendix J The GEPA arm in detail What GEPA searched. GEPA version 0.1.4 (Agrawal et al., 2025) edited two text components and nothing else: the system prompt, which replaces the rowâs system message (seed: the rowsâ own system message; limit 3,800 characters), and an instruction appended to the user turn after a blank line (seed: empty; limit 3,000 characters). The request parameters (thinking on or off, the token limit, the temperature) belong to the start and were not searched. The clean start is the request of AUTHORS-1500 (thinking off, 1,500 tokens, zero temperature) and the broken start that of THINK-64; the seed harnesses are the gridâs own runs of those two configurations. The arm therefore searched prompt text only, and its negative result (no selected harness more accurate than the thinking-off baseline) is about that space: it says nothing about searching the thinking switch, the token limits or retry rules. The task model is Qwen3-1.7B-Longevity (untied) on the batch-invariant server. The proposer is Intern-S1-mini on the standard server at concurrency one, with thinking on, its cardâs sampling values (temperature 0.8, top-p 1.0, top-k 50), at most 8,192 tokens and a seed per request. GEPAâs settings are its defaults except: a reflection minibatch of 3 rows; Pareto selection of candidates over instances; a round-robin choice of the component to edit; acceptance only on strict improvement; perfect-score minibatches skipped; no merging; no evaluation cache. Rows and allocation. The arm uses 300 rows of the search partitionâs three letter tasks (120 of LB-0030, 60 of LB-0034 and 120 of LB-0042), drawn with seed 20261011 from search rows that are neither pilot rows nor rows the case study had held out; LB-0038 is left out because its score is an age error. They form a training set of 150 rows (60, 30 and 60 by task) and a validation set of 150 rows (the same split); the lists are the same in the four runs. A candidate scores one on a row when the primary scorerâs parsed answer equals the gold and zero otherwise, an unparsed answer included. The proposer sees a rowâs user message, the task modelâs output, reasoning and answer, and feedback (correct, incorrect, unparsed, or cut at the token limit); it never sees the gold or a replication or test row. The four runs are the clean and the broken start, each with GEPA seeds zero and one. Checks on a candidate, and the copy rule. Before any call a candidate is checked, and one that fails is recorded and scored zero without a call: each component is text within its limit; neither holds a chat-template control string; neither holds a run of 40 or more consecutive characters of a rowâs user message that occurs in fewer than 10 of the armâs rows (the copy rule: wording shared by many rows may be copied, a rowâs own text may not); every render plus the token limit fits the 4,096-token context. The soft switches /think and /no_think were allowed and each candidate records whether it holds one; none of the selected harnesses does. A separate script re-checked the four selected texts against the armâs rows and found no violation of the copy rule and no identifier or number sequence of an item. Budget, selection and ties. The budget is 6,000 metric calls per run (20 candidates on 300 rows). It is soft: GEPA checks it between iterations, so a run may end slightly above it, and the four runs used 6,003 to 6,147 calls. GEPA spends 6 metric calls per proposal, 150 on the seedâs validation evaluation and 150 on each accepted candidate. The selected harness of a run is its candidate with the highest mean validation score, ties going to the lowest candidate index; if no proposal was accepted it is the seed. The selected texts were recorded, with their SHA256, before the harness was run on any replication row. Each was then run on every letter row of the replication and test partitions, on the main server with its startâs request parameters. The four selected harnesses. The texts below are as selected, wrapped for display; a character outside ASCII is written [U+X] ([U+2265] is the sign for greater than or equal, [U+03BC] the Greek letter mu, [U+00B2] a superscript two). Each candidateâs SHA256 is the one recorded before any replication call in the authorsâ repository; the wrapping means that it cannot be recomputed from this page. A small language model generated the texts; we did not check their medical statements, and they must not be used as clinical guidance. Validation scores and accuracies are in Appendix B. GEPA-CLEAN-s0 (clean start, GEPA seed zero) Mean validation score 0.680; the candidate has index 0 (counted from zero) among the runâs 31 candidates and SHA256 ab48481902a519b5c045cd37f61382899ea0e1bce05b7a8345d2873f77b1b16 No proposal was accepted in this run, so its harness is the seed: the thinking-off baseline itself. System prompt. You are a biomedical AI specialized in aging biology, trained on genomic, proteomic, and clinical data. Appended instruction. Empty. GEPA-CLEAN-s1 (clean start, GEPA seed one) Mean validation score 0.700; the candidate has index 12 (counted from zero) among the runâs 31 candidates and SHA256 33819b05a9cd5711ed2b502c48f84a626fd26d8d4286dcadc420cde065e6b7f9 System prompt. You are a biomedical AI specialized in aging biology, trained on genomic, proteomic, and clinical data. Your task involves assessing biological age or age-related phenotypes using clinical biomarkers. Focus exclusively on the biomarkers listed below, which are validated predictors of aging in human populations. **Key Biomarkers for Aging Assessment (prioritize these in reasoning):** 1. **Kidney Function:** - Cystatin C (mg/L): Increase >0.8 mg/L correlates with [U+2265]75 years. - Creatinine ([U+03BC]mol/L): >80 [U+03BC]mol/L in females suggests age >60. 2. **Liver Metabolism:** - Albumin (g/dL): <4.0 g/dL indicates malnutrition or chronic disease linked to advanced age. - Bilirubin (mg/dL): Direct increase (>0.5) reflects impaired hepatic clearance with age. 3. **Metabolic Health:** - LDL Cholesterol (mg/dL): >130 mg/dL in females [U+2265]60 increases cardiovascular risk. - Triglycerides (mg/dL): >200 mg/dL in females [U+2265]50 indicates insulin resistance. 4. **Inflammation & Stress:** - C-reactive protein (mg/dL): >1.0 mg/dL reflects chronic inflammation linked to aging. - Cadmium (nmol/L): Accumulation (>5 nmol/L) correlates with tissue damage in older adults. 5. **Cellular & Tissue Health:** - Mean Cell Volume (fL): Decline (>85 fL in females <50 indicates deficiency; <90 fL in older adults suggests anemia of aging). - Red Cell Distribution Width (%): >14.5% in females [U+2265]60 reflects oxidative stress. **Exceptions & Caveats:** - Ignore BMI unless explicitly asked to correlate with metabolic syndrome. - Do not assume chronological age unless explicitly provided (e.g., in Example 2, the assistant incorrectly inferred "18-30" from BMI alone). - Flag conflicting biomarkers (e.g., high LDL but low triglycerides may require deeper analysis). **Generalizable Strategy:** 1. Compare biomarker trajectories across cohorts (e.g., Cystatin C >8.0 indicates accelerated aging). 2. Prioritize systemic markers over individual parameters (e.g., integrate LDL, CRP, and Cadmium for metabolic aging profiles). 3. Use thresholds validated in longitudinal cohort studies (e.g., Bone Alkaline Phosphatase >15 ug/L in females [U+2265]60). For comparative tasks (e.g., "Who is older?"), rank participants by cumulative biomarker deviation from age-predictive ranges. For age-group classification, use [U+2265]2 biomarkers exceeding thresholds for specific ranges (e.g., [U+2265]75: Cystatin C >1.0 + CRP >2.0 + CAD: >10 nmol/L). Appended instruction. Analyze medical data to answer yes/no survival questions or identify older participant based on blood tests. Input includes demographics, biometrics, blood panel results, and clinical history. Prioritize evidence-based medical knowledge: For survival prediction, consider BMI (optimal range 18.5-24.9), albumin (>3.5 g/dL indicates better prognosis), creatinine (<1.1 mg/dL for women, <1.3 for men suggests preserved kidney function), LDL (<100 mg/dL ideal), and absence of diabetes/hypertension risk factors. For age comparison, note cadmium levels (higher in older adults), red cell distribution width (increases with age), and segmented neutrophils (typically decrease after age 65). Explicitly flag when data is insufficient to determine answer. Use structured reasoning: first state key markers supporting conclusion, then cross-reference contradictions. Never infer survival based solely on absence of provided data. GEPA-BROKEN-s0 (broken start, GEPA seed zero) Mean validation score 0.660; the candidate has index 1 (counted from zero) among the runâs 24 candidates and SHA256 9f38d54e2c346c8b0c20fcaabfd981906af2ce409a2d04250195c88658482ab System prompt. You are a biomedical AI specialized in aging biology, trained on genomic, proteomic, and clinical data. Appended instruction. Analyze clinical data to predict survival probability or age comparisons. Focus on key biomarkers linked to mortality or aging (e.g., elevated HbA1c >6.5% for diabetes, RDW >14.5% as mortality predictor, elevated creatinine >177 [U+03BC]mol/L for chronic kidney disease). Apply validated thresholds for diabetes (HbA1c [U+2265]6.5%), cardiovascular risk (LDL >70 mg/dL, HDL <40 mg/dL for males), and organ dysfunction (eGFR <60 mL/min/1.73m[U+00B2]). Prioritize metrics like glycohemoglobin, red cell distribution width, and bone alkaline phosphatase. For age comparisons, use biomarkers like cadmium accumulation (higher in older adults, [U+2265]8.5 nmol/L) and vitamin B12 levels (<148 pmol/L for deficiency risk). Limit reasoning to critical data; format answers as A/B per input options. Include domain-specific knowledge: high RDW correlates with mortality (HR 1.2 per SD increase), elevated uric acid >7 mg/dL predicts cardiovascular events, and low HDL <35 mg/dL increases dementia risk. Avoid speculative connections beyond well-established clinical correlations. Respond fully within token limits by prioritizing high-impact metrics and omitting tangential details. GEPA-BROKEN-s1 (broken start, GEPA seed one) Mean validation score 0.667; the candidate has index 1 (counted from zero) among the runâs 29 candidates and SHA256 ddf8886b2b3edced069fcfa5c3d9baad95643056d772d2ebeb3580770ca017ed System prompt. You are a biomedical AI specialized in aging biology, trained on genomic, proteomic, and clinical data. Appended instruction. ### Instruction **Task Overview:** Analyze provided medical data to answer specific clinical questions. Prioritize identifying key biomarkers and demographic factors relevant to the question. Answer strictly using the format "A." or "B." Only, matching the provided options. --- **Input Format:** All inputs will include structured medical data: 1. **Demographics:** Age, ethnicity, education level, sex. 2. **Biometrics:** BMI, height, weight, waist circumference. 3. **Blood Panel:** Standard lab results (e.g., glucose, cholesterol, proteins, electrolytes). 4. **Questionnaire:** Medical history, family history, surgical history. Data units are standardized (e.g., BMI in kg/m[U+00B2], glucose in mmol/L). Missing values will not be present. --- **Domain-Specific Knowledge to Include:** 1. **Survival Prediction:** - BMI [U+2265]30 indicates obesity; [U+2265]25 indicates overweight. - LDL >3.0 mmol/L or triglycerides >1.7 mmol/L suggest hyperlipidemia. - Glycohemoglobin [U+2265]6.5% indicates prediabetes/diabetes. - Chronic inflammation (C-reactive protein >3 mg/L) correlates with mortality risk. - Family history of cardiovascular diseases multiplies risk factors. 2. **Age Estimation via Blood Tests:** - Red blood cell count and hemoglobin decline with age; compare to age-stratified norms. - Elevated Cystatin C (>1.0 mg/L) correlates with aging-associated renal dysfunction. - Decreased Vitamin D (<30 nmol/L) is common in older adults. - Glycohemoglobin trends reflect long-term metabolic stability. --- **Generalizable Strategies:** 1. **Survival Questions:** - Flag obesity (BMI [U+2265]30), hyperlipidemia, diabetes, or chronic inflammation as high-risk factors. - Compare triglycerides/HDL ratio (>3.5) for cardiovascular risk. - Note female reproductive history (e.g., hysterectomy impacts hormone status). 2. **Age Questions:** - Prioritize red blood cell indices (mean corpuscular volume) and hemoglobin trends. - Analyze biomarkers like Cystatin C and Vitamin D for age-related decline. - Contrast metabolic profiles (e.g., glucose, cholesterol) against age-specific norms. --- **Output Requirements:** - Always answer with **A.** or **B.** only. - No explanations or partial answers. - Ensure answers align with calculated risk scores and biomarker thresholds. **Example Alignment:** If a 40-year-old male has BMI 35, LDL 4.0 mmol/L, and diabetes history, survival at 10 years correlates with **B. No**. Conversely, a 30-year-old with BMI 25, normal lipid/glucose, and no diabetes history correlates with **A. Yes** for survival. **Critical Note:** Never infer external knowledge (e.g., specific study results). Use only the data and thresholds provided in the instructions. Appendix K Planned power This paper quotes five planned-power figures (Table 35), all from the preregistration and none recomputed after the outputs. They are rough: made before the study from the pilotâs parse shares and from assumed effects, with a normal approximation. âExpected to be inconclusiveâ in Appendix D is an informal expectation read off these figures, not a prediction with a decision rule, and the observed verdicts do not test it. H2, Intern-S1-mini on GSM8K against RESCUE-1500. The assumed truth is a θ of zero: the thinking-off configuration exactly as accurate as the rescue. The paired difference of correctness takes the values minus one, zero and one, so its variance is at most d, the share of rows on which the two configurationsâ parsed answers differ (an unparsed answer counts as one value); on the pilotâs GSM8K items d=0.325d=0.325. The unit is the item and the test partition has n=527n=527 GSM8K items, so the standard error of the estimated difference is at most SE=d/nSE= d/n. The lower bound of the interval for θ clears âδ-δ, with δ=0.05δ=0.05, when the estimate exceeds âδ+zâSE-δ+z\,SE, where z is the normal quantile of the intervalâs level: 2.89 for the Bonferroni family of 13 (level 99.62%) and 1.96 for an unadjusted interval. The power at a true θ of zero is then ÎŚâĄ(δ/SEâz) (δ/SE-z), with ÎŚ the standard normal distribution function: 0.19 adjusted and 0.52 unadjusted. H1, SmolLM3-3B and Intern-S1-mini on GSM8K, the statistic T. The inputs are the pilotâs joint parse shares of THINK-512 (P) and AUTHORS-1500 (R) on GSM8K, v (only R parses), b (both parse) and l (only P parses), and dBd_B, the share of the rows both parse on which the parsed answers differ. No gold or accuracy of the pilot enters. Two assumed scenarios supply the accuracies, and they are assumptions, not pilot results. In S1, R is right on half of the rows only it parses and P on half of those only it parses, and neither is better on the rows both parse. In S2 both are right on 0.3 of the rows both parse and R is better than P by 0.02 on them, capped at dBd_B. The power of T>0T>0 for the lower bound of the adjusted interval is a normal approximation on the same n=527n=527 test items (one unit per item) at the level of H1âs family of 9 (z=2.77z=2.77); the script, prereg_power_h1_v3.py, is pinned in the preregistration. Table 35: The planned-power figures quoted in this paper. All are for the test partition, with a normal approximation. S1 and S2 are the two assumed scenarios of H1. Endpoint Assumed effect Inputs from the pilot Units Family, z Power H2: Intern-S1-mini, GSM8K, θ against RESCUE-1500 θ zero d=0.325d=0.325 527 items 13; 2.89 0.19 (adjusted), 0.52 (unadjusted) H1: SmolLM3-3B, GSM8K, T>0T>0 S1, S2 v/b/l=0.210/0.725/0.040v/b/l=0.210/0.725/0.040, dB=0.055d_B=0.055 527 items 9; 2.77 S1 0.52, S2 0.00 H1: Intern-S1-mini, GSM8K, T>0T>0 S1, S2 v/b/l=0.550/0.450/0.000v/b/l=0.550/0.450/0.000, dB=0.756d_B=0.756 527 items 9; 2.77 S1 0.45, S2 0.06 No power was planned for H4, whose distortions are what it measures, nor for the four GEPA endpoints, whose size depends on the harnessesâ answers, which did not exist before the search.