Paper deep dive
VERITAS: Verifiable Epistemic Reasoning for Image-Derived Hypothesis Testing via Agentic Systems
Lucas Stoffl, Benedikt Wiestler, Johannes C. Paetzold
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/15/2026, 1:14:59 AM
Summary
VERITAS is a multi-agent system designed for autonomous, auditable hypothesis testing on multimodal clinical datasets. It decomposes research workflows into four phasesâplanning, segmentation, statistical analysis, and interpretationâusing role-specialized agents. The system introduces an 'Epistemic evidence label framework' to classify outcomes as Supported, Refuted, Underpowered, or Invalid, ensuring statistical rigor in medical imaging research. VERITAS achieves high accuracy (81.4% with frontier models) and provides a fully auditable provenance trail for every conclusion.
Entities (5)
Relation Signals (4)
Evidence Classification Operator â derives â Evidence Label
confidence 100% ¡ The Evidence Classification Operator (ECO) mechanically derives the evidence label from Phase 2B outputs.
VERITAS â evaluatedon â ACDC
confidence 100% ¡ We evaluate on two public datasets... cardiology (ACDC)
VERITAS â evaluatedon â UCSF-PDGM
confidence 100% ¡ We evaluate on two public datasets... neuro-oncology (UCSF-PDGM)
VERITAS â uses â Epistemic evidence label framework
confidence 100% ¡ Veritas... introduces an epistemic evidence label framework that mechanically classifies outcomes
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Drawing meaningful conclusions from inherently multimodal clinical data (including medical imaging) requires coordinating expertise across the clinical specialty, radiology, programming, and biostatistics. This fragmented process bottlenecks discovery. We present VERITAS (Verifiable Epistemic Reasoning for Image-Derived Hypothesis Testing via Agentic Systems), a multi-agent system that autonomously tests natural-language hypotheses on multimodal clinical datasets while producing a fully auditable evidence trail: every statistical conclusion traces through inspectable, executable outputs from analysis plan to segmentation masks to statistical code to final verdict. VERITAS decomposes the workflow into four phases handled by role-specialized agents, and introduces an epistemic evidence label framework that mechanically classifies outcomes as Supported, Refuted, Underpowered, or Invalid by jointly evaluating significance, effect direction, and study power. This distinction is critical in medical imaging, where non-significant results often reflect insufficient sample size rather than absent effects. To evaluate the system, we construct a tiered benchmark of 64 hypotheses spanning six complexity levels across cardiac (ACDC, 150 subjects) and brain glioma (UCSF-PDGM, 501 subjects) MRI. VERITAS reaches 81.4% verdict accuracy with frontier models and 71.2% with locally-hosted open-weight models (8-30B), outperforming all five single-model baselines in both classes. It also produces the highest rate of independently verifiable statistical outputs (86.6%), so even its failures remain diagnosable through artifact inspection. Structured multi-agent decomposition thus substitutes for model scale while preserving the verifiability clinical research demands.
Tags
Links
- Source: https://arxiv.org/abs/2604.12144v1
- Canonical: https://arxiv.org/abs/2604.12144v1
Trouble viewing inline? Open PDF directly â
Full Text
138,994 characters extracted from source content.
Expand or collapse full text
Veritas: Verifiable Epistemic Reasoning for Image-Derived Hypothesis Testing via Agentic Systems Lucas Stoffl1,2, Benedikt Wiestler3,4,5, Johannes C. Paetzold1,2 1Department of Radiology, Weill Cornell Medicine, New York, USA 2Cornell Tech, New York, USA 3TUM University Hospital, Munich, Germany 4Technical University of Munich, 5Munich Center for Machine Learning lms465@cornell.edu, jpaetzold@med.cornell.edu Abstract Drawing meaningful conclusions from inherently multimodal clinical data (including medical imaging) requires coordinating expertise across the clinical specialty, radiology, programming, and biostatistics. This fragmented process bottlenecks discovery. We present Veritas (Verifiable Epistemic Reasoning for Image-Derived Hypothesis Testing via Agentic Systems), a multi-agent system that autonomously tests natural-language hypotheses on multimodal clinical datasets while producing a fully auditable evidence trail: every statistical conclusion traces through inspectable, executable outputs from analysis plan to segmentation masks to statistical code to final verdict. Veritas decomposes the workflow into four phases handled by role-specialized agents, and introduces an epistemic evidence label framework that mechanically classifies outcomes as Supported, Refuted, Underpowered, or Invalid by jointly evaluating significance, effect direction, and study power. This distinction is critical in medical imaging, where non-significant results often reflect insufficient sample size rather than absent effects. To evaluate the system, we construct a tiered benchmark of 64 hypotheses spanning six complexity levels across cardiac (ACDC, 150 subjects) and brain glioma (UCSF-PDGM, 501 subjects) MRI. Veritas reaches 81.4% verdict accuracy with frontier models and 71.2% with locally-hosted open-weight models (8â30B), outperforming all five single-model baselines in both classes. It also produces the highest rate of independently verifiable statistical outputs (86.6%), so even its failures remain diagnosable through artifact inspection. Structured multi-agent decomposition thus substitutes for model scale while preserving the verifiability clinical research demands. Keywords Multi-Agent Systems â ¡ Large Language Models â ¡ Hypothesis Testing â ¡ Medical Image Analysis â ¡ Statistical Inference 1 Introduction Medicine advances when clinicians translate bedside observations into formal hypotheses that are tested and validated in observational studies. Such studies require coordinating special clinical domain knowledge, patient data curation and analysis, programming, and biostatistics. In the data domain, medical images are indispensable; they encode rich clinical information, including volumetric measurements, morphological features, and their associations with patient outcomes. Yet turning these image data into statistically valid insights, i.e., the gold standard for scientific evidence, remains a particularly fragmented, expert-intensive process. For example, a neurooncologist who wants to study if the more aggressive tumor biology of glioblastoma is reflected in larger contrast-enhancing tumors compared to lower-grade gliomas must coordinate image segmentation, metric extraction, statistical test selection, and result interpretation across multiple software tools and disciplinary boundaries. This pipeline bottlenecks discovery and limits who can participate in image-based research. Recent advances in large language models (LLMs) and multi-agent systems offer a path toward automating such workflows. However, applying these systems to medical imaging hypothesis testing poses three challenges: (i) the pipeline must bridge complex and distinct tasks, including cohort definition (selecting the right patients from a given dataset), vision (segmenting anatomical structures), computation (deriving biomarkers), and statistics (selecting, executing and interpreting tests), requiring models to reason jointly across all tasks; (i) deriving clinical conclusions demands full auditability, where every step from segmentation mask to p-value calculation must be inspectable and reproducible; and (i) small clinical cohorts require epistemic awareness, which is the ability to distinguish âno effect detectedâ from âinsufficient power to detect an effect.â Our main contribution is Veritas (Verifiable Epistemic Reasoning for Image-Derived Hypothesis Testing via Agentic Systems), a multi-agent framework inspired by clinician-scientist workflows. With role-specialized agents (PI, Imaging Specialist, Statistician), Veritas addresses the above challenges and reaches the following capabilities: 1. Fully autonomous clinical hypotheses testing. Veritas transforms natural language hypotheses into statistically validated conclusions with zero human intervention, managing the full study cycle to generate structured plans, executable code, statistics, and visualizations. 2. Multimodal clinical data understanding. The framework integrates medical imaging (e.g., MRI) and clinical metadata using specialized tools, including promptable segmentation models, to derive quantitative biomarkers directly from raw image data. 3. Auditable and verifiable at each step. Veritas produces a complete, transparent execution trail. All results are derived from executable Python scripts and verifiable segmentation masks, allowing clinicians to reproduce and audit both the underlying logic and visual evidence. 4. High performance in validation. Validated on 64 hypotheses across cardiac and neuro-oncology domains, Veritas achieves 71.2% majority-vote verdict accuracy with locally-deployed open-weight models and 81.4% with frontier models, outperforming all baselines in each model class. 5. Structured decomposition over scale. By utilizing role-specialized agents, Veritas enables small, locally-deployed models (8-â30B) to perform complex reasoning that typically requires frontier models, facilitating secure, on-site clinical deployment. Further contributions: To categorize/systematize our agentic systems conclusions, we propose the Epistemic evidence label framework. Here, we introduce a four-label classification Supported, Refuted, Underpowered, Invalid that mechanically resolves the epistemic status of a hypothesis from observed statistics and a smallest effect size of interest (SESOI), without relying on agent judgment. This distinction is essential for medical imaging benchmarks, where mistaking underpowered subgroup analyses as refutations inflates apparent accuracy (Sec. 3.3). A central question is whether a multi-agent system like Veritas can deliver correct conclusions and verifiable evidence across increasing complexity; to answer it, we contribute a tiered Evaluation benchmark spanning six complexity levels, from metadata-only queries (L1) to multivariate survival models (L5), and untestable hypotheses (L0). For all hypotheses, the ground truth was established through dataset-derived reference statistics validated by domain experts. The benchmark includes positive, negative, no-effect, underpowered, and untestable controls, thereby probing the full range of the evidence-label framework. We evaluate on two public datasets across distinct clinical domains: cardiology (ACDC [6], cardiac MRI, 150 subjects) and neuro-oncology (UCSF-PDGM [8], brain glioma MRI, 501 subjects). Figure 1: Veritas architecture and evidence flow. Given a natural language hypothesis and dataset, three role-specialized agents (PI, Imaging Specialist, Statistician) with a Critic collaborate through four phases: (1) collaborative analysis planning producing a structured plan; (2A) neural segmentation, grounding all evidence in image-derived masks; (2B) sandboxed code generation and execution yielding statistical outputs; and (3) interpretation of the Phase 2B statistical outputs in the context of the accumulated workflow history to produce a verdict. The Evidence Classification Operator (ECO) mechanically derives the evidence label from Phase 2B outputs. Every phase produces a versioned artifact (right), forming a complete provenance trail from hypothesis to conclusion. 2 Related Work Recent advances span medical image foundation models, agentic scientific systems, tool-augmented language models, and statistical best practices. While each of these directions addresses a component of the scientific workflow, they have largely evolved independently. Systems that jointly provide vision-grounded measurement, executable statistical validation, and power-aware evidence calibration remain limited. Our work aims to unify these elements into a single auditable hypothesis-testing framework. Medical image analysis and visionâlanguage models. Foundation models have greatly advanced medical image understanding by enabling promptable and generalizable segmentation across anatomies and modalities. The Segment Anything Model (SAM) [30] introduced a flexible segmentation paradigm using spatial prompts, subsequently extended to video and concept-based prompts [49, 10]. This paradigm has been adapted to the medical domain through MedSAM [37], MedSAM2 [38], among others [72], enabling cross-modality segmentation of anatomical structures. SAT [71] and VoxTell [50] extend prompting beyond spatial inputs by allowing segmentation targets to be specified using natural language. These text-promptable models contrast with specialist architectures such as nnU-Net [27], which self-configure per task but require dedicated training for each new target structure, precluding the flexibility needed for analysis across diverse anatomical targets. In parallel, biomedical visionâlanguage models (VLMs) enable multi-modal reasoning over images and clinical text. BioViL-T [4] exploits temporal relationships in longitudinal radiology data, while LLaVA-Med [34] supports conversational question answering over biomedical images. BiomedParse [70] unifies segmentation, detection, and recognition across nine imaging modalities via natural language, and VoxelPrompt [24] uses language-guided code generation to analyze 3D medical imagesâthe closest prior work to an agent-driven imaging pipeline. Despite these advances, prior work primarily operates at the level of perception (segmentation, retrieval, or question answering) rather than cohort-level scientific inference. In particular, no existing system derives statistically validated conclusions from image-derived measurements across patient populations. LLMs for scientific discovery and hypothesis testing. Large language models are increasingly used in scientific workflows, including literature synthesis, hypothesis generation, and experimental planning [43, 2, 14], generally termed the Agentic Science paradigm [62]. Complementary work treats AI scientists as collaborative systems that combine LLM reasoning with domain-specific tools [17]. Several systems attempt to automate full research cycles: the AI Scientist [36] automates the ML research lifecycle from ideation to manuscript; AI Co-Scientist [21] uses multi-agent debate for hypothesis generation with experimental validation; Robin [19] integrates literature analysis with wet-lab experiments; and DiSciPLE [39] discovers interpretable programs for scientific visual data. Popper [25] is particularly related: it employs agents for rigorous hypothesis validation through sequential falsification with formal error control, but operates on tabular and text data without integrating medical imaging or epistemic power analysis. A complementary line of work augments language models with external tools and executable programs. Toolformer [51], ViperGPT [54], ToolLLM [48], CodeAct [59], and Gorilla [46] enable LLMs to call APIs and write executable code, while the ReAct paradigm [67] interleaves reasoning with tool use. More broadly, programmatic reasoning, where models generate and execute code rather than reasoning purely in text, has been shown to substantially improve reliability for quantitative tasks [11, 16, 59], motivating our code-first approach to statistical analysis. In the medical domain, agent-based frameworks such as RadFabric [12], MedAgents [56], MedAgent-Pro [60], MDAgents [29], AgentMD [28], Agent Hospital [35], and MAI-DxO [40] apply LLM collaboration to diagnostic reasoning and clinical decision support. MedAgent-Pro [60] is most closely related: it integrates segmentation tools with coding agents for computing clinical quantitative indicators from images, but targets per-patient diagnosis rather than population-level hypothesis testing with statistical rigor. None of these systems address vision-grounded hypothesis testing, which requires bridging image-derived measurements with cohort-level statistical analysis and epistemic classification. Multi-agent systems for scientific reasoning. Multi-agent systems structure complex scientific workflows by decomposing tasks across specialized roles. In the scientific domain, Virtual Lab [55], Virtual Biotech [68], Agent Laboratory [52], VirSci [53], ResearchAgent [3], and SciAgents [18] organize teams of agents that collaboratively plan, code, and execute experiments. General-purpose multi-agent frameworks such as AutoGen [64] and MetaGPT [23] provide infrastructure for such agent orchestration, with MetaGPTâs insight that structured intermediate artifacts reduce cascading hallucinations being directly relevant to our design. Nevertheless, a recent independent evaluation of several such AI research frameworks, including AgentLaboratory [52], AutoGen [64], MOOSE-Chem2 [66], SciAgents [18], SciMON [58], and Virtual Lab [55], found that none successfully completed an end-to-end research cycle [1] on two real-world tasks, with agents frequently hallucinating results during implementation [1]. This highlights the importance of executable, code-grounded reasoning [59, 11] for bridging the gap between planning and reliable analysis. MESHAgents [69] is a closely related multi-agent system: it orchestrates specialized LLM agents for automated phenotype-wide association studies in cardiovascular imaging, identifying which imaging phenotypes and confounders to include. However, MESHAgents reasons over pre-extracted tabular phenotype data rather than deriving measurements from images within its pipeline, and does not provide epistemic classification of evidence strength. Statistical evidence and epistemic classification. Interpreting statistical evidence is a central challenge in empirical research. The American Statistical Associationâs landmark statement cautioned against using p-value thresholds as the sole basis for scientific conclusions [61], while others have argued for redefining significance thresholds [5] or shifting focus to effect sizes [13]. Low statistical power further compounds these issues, inflating both false-positive rates and effect-size estimates [26, 7]. Equivalence testing via the TOST procedure and the smallest effect size of interest (SESOI) provide a principled way to distinguish âno meaningful effectâ from âinsufficient power to detectâ [32]. Lakens et al. [31] formalized this into a four-outcome decision framework, which is the direct conceptual precursor of our evidence label taxonomy (Supported, Refuted, Underpowered, Invalid). In medical imaging, reproducibility is additionally influenced by variability in segmentation and feature extraction pipelines, motivating standardization efforts such as IBSI [73] and PyRadiomics [57]. Veritas operationalizes power-aware evidence classification within an automated pipeline: the Evidence Classification Operator mechanically maps observed statistics to epistemic labels without relying on agent judgment, automating a framework that prior work proposed only as manual analytical guidance. Veritas tackles these challenges by integrating vision-grounded measurement extraction, executable statistical analysis, and multi-agent reasoning within a unified and auditable pipeline. By linking segmentation outputs, statistical code, and final conclusions through verifiable artifacts, Veritas enables autonomous hypothesis testing on medical imaging datasets while maintaining transparent and reproducible analysis workflows. 3 Methods We present Veritas, a multi-agent framework that transforms natural-language hypotheses about medical imaging datasets and their associated metadata into statistically validated conclusions through autonomous, auditable execution. 3.1 Problem Formulation Given a dataset =(xi,mi)i=1ND=\(x_i,m_i)\_i=1^N where xix_i represents medical images for patient i and mim_i denotes patient metadata (including cohort variables, clinical covariates, and outcomes such as survival endpoints), together with a natural-language hypothesis âH, we seek to autonomously determine whether âH is Supported, Refuted, Underpowered, or Invalid given the available evidence. Unlike traditional computational pipelines that require explicit metric definitions and test specifications, our framework interprets âH semantically, checks feasibility against available data, designs an appropriate analysis plan, and executes it end-to-end with an auditable artifact trail. 3.2 Framework Architecture Veritas operates through four sequential phases, each producing structured, inspectable outputs that feed into subsequent phases (Fig. 1). Three role-specialized agents collaborate: a Principal Investigator (scientific validity, feasibility), a Medical Imaging Specialist (segmentation targets, imaging protocols), and a Statistician (test selection, effect sizes, power analysis). A separate Critic agent operates by default in execution phases (2A/2B). We evaluate two model deployments with the same workflow and prompts: (i) a local open-weight team (GPT-OSS-20B [44] for PI/imaging, Qwen3-8B [65] for discussions and critic, Qwen3-Coder-30B [9] for coding), and (i) a frontier team (GPT-5.2 [43] for PI/imaging/coding and GPT-5 Mini [42] for discussions and critic). Dataset API abstraction. To keep the framework dataset-agnostic and auditable, agents never access raw files directly during analysis. Instead, they interact through a constrained API layer that exposes cohort discovery, per-patient metadata, observation identifiers, segmentation masks, and geometry-aware measurement utilities. This API boundary standardizes provenance and enables automated checks for off-contract data access. Detailed API functions are described in the supplementary material. Phase 1: Analysis Planning. Given âH and dataset metadata (available groups, observations, patient fields), the agents collaboratively produce a structured analysis plan specifying the target cohorts, population restriction, required anatomical structures and observations, derived measurements, and the planned statistical analysis. The plan includes a feasibility decision (TESTABLE/UNTESTABLE) with subtype and missing requirements; a programmatic validator checks schema completeness and consistency against the available dataset metadata before execution. If marked untestable, the run terminates as Invalid. The Statistician also computes a priori power from the queried cohort sizes under a fixed planning SESOI. Phase 2A: Segmentation. When the hypothesis requires image-derived measurements, the Imaging Specialist generates a segmentation request specifying the relevant patients, target structures, and observations. A segmentation backend (SAT [71]) processes this request and stores the resulting binary masks in a shared database. This phase is the vision core of the framework: subsequent quantitative analyses are grounded in measurements extracted from these masks. For hypotheses that require no imaging structures, this phase is bypassed. Phase 2B: Statistical Analysis. The Statistician writes executable code that retrieves the required masks and metadata through the API, computes the planned measurements, performs the statistical test, and reports effect size δ, 95% confidence interval CI95CI_95, p-value p, and sample sizes. The code is executed in a sandboxed environment with timeout constraints. Failed or invalid attempts trigger iterative revision using execution traces and validator feedback, with optional Critic input, up to 8 attempts. Phase 3: Interpretation. Agents synthesize the prior workflow outputs through structured discussion, producing a verdict vâYES,NO,INCONCLUSIVEvâ\ YES, NO, INCONCLUSIVE\ with supporting rationale. Phase 3 has access to the relevant planning and statistical outputs from earlier phases, while the evaluator-computed evidence label (see Sec. 3.3) is explicitly withheld. Figure 2: Auditability through artifact provenance. Every phase produces executable, inspectable artifacts. Validity checks (red squares) enforce consistency at each boundary. The evidence label is derived mechanically from Phase 2B statistics via the Evidence Classification Operator (ECO; Sec. 3.3), independent of agent judgment. Auditability. A key design goal is that every analysis is fully auditable. Each phase produces structured artifacts (e.g., JSON plans, segmentation requests, Python code, statistical outputs, verdicts) rather than natural language summaries (Fig. 2). Validity checks enforce consistency at each boundary, and the final statistical output is retained in a form that enables independent re-verification (see the evidence grounding metric in Sec. 3.4). The supplementary material provides the full list of per-phase validity checks. 3.3 Evidence Label Framework A central contribution is our four-label evidence classification that jointly evaluates statistical significance, effect direction, and study power: Table 1: Evidence label definitions. Labels are mechanically derived from Phase 2B statisticsâno agent judgment is involved. Label Condition Interpretation Supported p<Îąâ§p<Îą direction matches Evidence confirms hypothesis Refuted (pâĽÎąâ§ĎâĽ0.80)(pâĽÎą Ď⼠0.80) Adequate power, no effect â¨(p<ι⧠\,(p<Îą opposite dir.) Significant opposite effect Underpowered pâĽÎąâ§Ď<0.80pâĽÎą Ď<0.80 Insufficient power to conclude Invalid Untestable or execution failure Results unreliable Here, Ď denotes statistical power computed at a smallest effect size of interest (SESOI) and Îą=0.05Îą=0.05. This functions as a decision-theoretic epistemic operator: given observed statistics (p,δ,n)(p,δ,n) and a pre-specified SESOI δ0 _0, the evidence label is a deterministic functionâfully removing subjective judgment from the classification. The epistemic distinction between Refuted and Underpowered is critical for medical imaging benchmarks: a non-significant result with power Ď=0.38Ď=0.38 (e.g., n=13n=13 patients) provides essentially no evidence against the hypothesis, yet standard binary evaluation would score it as a correct rejection. Details on power computation are in the supplementary material. 3.4 Benchmark & Evaluation For reproducible benchmarking, we construct a bank of 64 hypotheses (32 per dataset), spanning six complexity tiers (Table 2). Each tier contains a mix of positive controls (37), negative/no-effect controls (17), and underpowered or untestable controls (10). Ground-truth evidence labels are established through canonical reference statistics computed from ground-truth segmentation masks and patient metadata, validated by a domain expert. Full per-hypothesis specifications are in the supplementary material. Table 2: Tiered hypothesis bank. Six complexity tiers probe progressively harder capabilities. Neg. includes negative and no-effect controls. Undp./Unt. = underpowered (Ď<0.80Ď<0.80) or untestable hypotheses. Tier Name n Capability probed Pos. Neg. Undp./Unt. L0 Untestable 5 Feasibility detection â â 5 L1 Metadata-only 9 Statistical reasoning 3 4 2 L2 Single imaging 23 Vision-to-statistics pipeline 15 8 â L3 Engineered feat. 6 Feature engineering 6 â â L4 Mixed 16 Multi-modal integration 8 5 3 L5 Multivariate 5 Advanced statistical reasoning 5 â â Total 64 37 17 10 Unless stated otherwise, all accuracy metrics are computed on testable hypotheses (L1âL5); L0 feasibility accuracy is reported separately. We evaluate with: ⢠Evidence-label accuracy: agreement between evaluator-computed evidence labels (Supported, Refuted, Underpowered) and ground truth, requiring executable statistical output. ⢠Verdict accuracy: agreement between final verdicts (Yes/No/Inconclusive) and ground truth. ⢠Majority-vote accuracy: hypothesis-level aggregation over repeated runs; the majority verdict/label across runs is compared to ground truth. ⢠L0 feasibility accuracy: fraction of untestable hypotheses correctly classified as Invalid, reported separately from testable hypothesis performance. For repeated runs, we report both run-level (micro-averaged) and majority-vote (hypothesis-level) metrics. Evidence-label and verdict accuracy are each computed over runs producing valid output for that metric; these pools may differ when a run yields a verdict but incomplete statistical output. We report completion rate separately for Veritas to quantify execution reliability. We additionally report diagnostic rates across three categories: conclusion quality (overclaim, false refutation), analysis integrity (hallucinated significance, synthetic data, literal p-value assignment), and auditability (evidence grounding rate, measuring whether runs report the four core statistical outputs required for independent verification). Complete metric definitions are in the supplementary material. 4 Experiments 4.1 Experimental Setup 4.1.1 Datasets We evaluate on two public medical imaging datasets from cardiology and neuro-oncology (per-dataset tier coverage in the supplementary material). ACDC [6] contains 150 subjects (100 training + 50 testing) with short-axis cardiac cine MRI across five diagnostic groups (30 per group): dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy (HCM), myocardial infarction (MINF), right ventricular abnormality (RV), and normal controls (NOR). Each subject includes end-diastolic (ED) and end-systolic (ES) phases with expert-annotated ground-truth segmentations of left ventricle (LV), right ventricle (RV cavity), and myocardium. Patient metadata includes height, weight, and cardiac timing parameters. Hypotheses span L0âL4. UCSF-PDGM [8] is a preoperative diffuse glioma MRI dataset with 501 subjects across WHO grades 2-4 (2: 56, 3: 43, 4: 396) and molecular markers (IDH mutation, MGMT methylation, 1p/19q codeletion). Ground-truth tumor segmentation masks, following the BraTS conventions, are available for 495 subjects. Patient metadata includes age, sex, overall survival (395 with survival data; 229 events), and extent of resection. The SAT model segments four brain tumor subregions: whole tumor (WT), necrotic core (NCR), peritumoral edema (ED), and enhancing tumor (ET). Hypotheses span L0âL5, including survival analyses and multivariate regression. 4.1.2 Baselines We compare Veritas against a structured ablation of five single-model baselines (SMaâSMe) that isolate the contribution of each key capability. Single-model baselines (SMaâSMe). To disentangle the contributions of code execution, data access strategy, iterative self-correction, and structured phase guidance, we define five baselines forming an ablation progression: ⢠SMa â Direct reasoning: The model receives per-group summary statistics pre-computed from ground-truth segmentation and produces a verdict without any code. Tests pure statistical reasoning from tabular numbers. ⢠SMb â Code on pre-computed features: The model receives a per-patient CSV of pre-computed imaging features (volumes, masses, etc.) and writes one Python script executed automatically in the sandbox. Tests one-shot code generation when the vision extraction step is pre-handled. ⢠SMc â Code via API: The model receives only the hypothesis and API documentation. It writes a single end-to-end script covering patient discovery, mask loading, metric computation, and statistical testing. Tests one-shot code generation with full raw data access. ⢠SMd â Agentic (iterative): Same API access as SMc, but the model operates in an iterative loop (up to 3 rounds): write code â execute â observe output â refine or conclude. Tests whether self-correction improves over single-shot generation. ⢠SMe â Pipeline (structured): Same API access as SMd, but guided by Veritasâs exact phase agendas (Phase 1 planning, Phase 2 coding, Phase 3 interpretation) as a single-model chain-of-thought. Tests whether phase structure alone provides value without multi-agent discussion. These baselines isolate four capability dimensions: (i) code execution vs. pure reasoning (SMaâ ); (i) pre-extracted features vs. raw API access (SMbâ ); (i) iterative self-correction (SMcâ ); and (iv) structured phase guidance (SMdâ ). Comparing SMe to Veritas then directly isolates the contribution of multi-agent discussion over a single model following identical phase instructions. Model and fairness. We evaluate all five baselines with two model families: frontier (GPT-5.2 [43]) and local/open-weight (GPT-OSS-20B [44]). For fairness, all baselines that use the API (SMcâSMe) receive identical API documentation to Veritas agents, extracted directly from the same prompt templates. The full dataset-specific structure catalogue (e.g., "LV", "RV", "Myo" for ACDC) is also provided, equivalent to what Veritasâs Phase 2A agents discover through tool calls; the model must still determine which structures are relevant to the hypothesis. Code execution uses the same sandboxed environment as Veritas. For SMd and SMe, a single interpretation step (equivalent to Veritasâs Phase 3) is always invoked after successful code execution; the modelâs conclusion from this step is used as the final verdict. 4.2 Main Results 4.2.1 Overall Performance Each method was evaluated with 10 repeated runs per hypothesis (64 hypotheses, 640 runs per method), and we report run-level and majority-vote accuracy across both model families (Table 3). Table 3: Main results on the 64-hypothesis tiered benchmark. Each cell shows accuracy (%). Evidence accuracy requires executable statistical output (not applicable for SMa). Metrics are computed on testable hypotheses (L1âL5); L0 feasibility is reported separately. Compl.: fraction of runs completing all workflow phases (Sec. 3.4); baselines always produce output by construction. Run-level Majority vote Method Evidence Verdict Evidence Verdict Compl. Local/Open-weight models (GPT-OSS-20B) SMa: Direct reasoning â 56.1 â 55.9 â SMb: Code on features 58.1 63.8 61.0 62.7 â SMc: Code via API 39.0 63.2 40.7 59.3 â SMd: Agentic 46.3 66.0 52.5 66.1 â SMe: Pipeline 55.1 58.0 57.6 61.0 â Veritas (local) 63.4 71.4 67.8 71.2 78.1 Frontier models (GPT-5.2) SMa: Direct reasoning â 55.9 â 55.9 â SMb: Code on features 63.7 70.7 69.5 69.5 â SMc: Code via API 52.9 72.8 61.0 76.3 â SMd: Agentic 64.7 76.6 64.4 74.6 â SMe: Pipeline 60.8 65.5 59.3 61.0 â Veritas (frontier) 70.0 81.0 76.3 81.4 87.5 Veritas (local) uses locally-deployed 8â30B models. L0 feasibility: Veritas 74.0% (local), 100% (frontier); all baselines 100% except SMb-frontier (98%). Key findings. Veritas achieves the highest accuracy in both model families: 71.2% majority-vote verdict accuracy with locally-deployed open-weight models (8â30B) and 81.4% with frontier models, outperforming all single-model baselines in each class. The ablation progression reveals consistent patterns: Code execution is essential. SMa (direct reasoning) achieves only âź 56% verdict accuracy regardless of model scale, confirming that pure LLM reasoning over summary statistics is insufficient. Adding code execution (SMb) lifts verdict accuracy by 8â15 percentage points. Iterative self-correction helps selectively. The SMcâ gain is modest for verdict accuracy (+2.8 local/open-weight, +3.8 frontier) but more pronounced for evidence accuracy (+7.3, +11.8), suggesting that self-correction primarily improves statistical analysis quality rather than final interpretation. Multi-agent decomposition adds value. Comparing SMe to Veritas isolates the multi-agent contribution: +10.2p verdict majority vote for local/open-weight (71.2% vs. 61.0%) and +20.4p for frontier (81.4% vs. 61.0%). The larger frontier gap suggests that multi-agent discussion becomes more valuable as base model capability increases. Per-dataset gap. Evidence-label accuracy drops sharply from ACDC to UCSF-PDGM across all methods (Table 4), reflecting genuine complexity differences (survival analyses, multivariate regression, molecular-marker stratification). On ACDC, frontier Veritas achieves 90.0% evidence accuracy and 78.9% verdict at 88.8% completionâpipeline failures on a subset of runs reduce verdict performance on easier hypotheses where baselines rarely fail; while local Veritas achieves competitive performance, with highest scores on the L5 tier, despite much smaller model capacities. Veritasâs advantage emerges on harder UCSF-PDGM hypotheses, where it leads in both evidence (+16p over SMd) and verdict accuracy (+29p). 4.2.2 Stratified Results Table 4 provides complementary breakdowns by dataset and by complexity tier, comparing both Veritas configurations against the three strongest frontier baselines. Table 4: Stratified run-level accuracy (%) by dataset and complexity tier. Per-dataset: evidence / verdict. Per-tier: evidence-label accuracy. Best non-Veritas result per tier is underlined. L0 = feasibility detection. Full per-method table in supplementary. ACDC UCSF-PDGM Evidence by tier Method Ev. Vd. Ev. Vd. L0 L1 L2 L3 L4 L5 All Veritas (local) 91.0 87.9 36.7 49.0 74 54.4 87.0 51.7 49.4 30.0 63.4 SMc: Code (GPT-5.2) 79.0 85.1 27.7 60.9 100 75.6 70.9 43.3 34.4 0.0 52.9 SMd: Agentic (GPT-5.2) 95.9 96.6 34.7 53.9 100 94.4 85.2 55.0 42.5 0.0 64.7 SMe: Pipeline (GPT-5.2) 90.3 96.9 32.3 35.1 100 56.7 89.1 56.7 43.1 0.0 60.8 Veritas (frontier) 90.0 78.9 50.7 83.2 100 90.0 90.4 73.3 45.0 16.0 70.0 Tier behavior. L2 (single imaging metric) is the strongest tier across all methods (71â90%), as these hypotheses require straightforward segmentation feature comparisons. L4 and L5 are hardest, driven by execution fragility in survival analyses and multi-covariate regression. Critically, Veritas is the only method to achieve non-zero L5 evidence accuracy (16â30% across configurations), while all single-model baselines score 0% on L5 despite achieving up to 92% verdict accuracy on these hypothesesâindicating they produce correct YES/NO guesses without valid statistical backing. The small L5 sample (5 hypotheses) limits precision, but the qualitative gap (non-zero vs. zero) is robust. Table 5: Diagnostic rates (%) for code-producing methods on the combined benchmark. Conclusion quality â Overclaim: YES verdicts where evidence is not Supported. False ref.: predicted Refuted labels disagreeing with ground truth. Analysis integrity â Hall. sig.: YES verdict with pâĽ0.05p⼠0.05 or missing p-value (output-level check). Synth.: code generating mock/random data instead of using real measurements (code-level). Lit. p: significance threshold hard-coded in analysis code (code-level). Auditability â Verif. (â ): fraction of all runs reporting the four core statistical outputs (test type, sample sizes, effect size, p-value) required for independent verification. Conclusion quality Analysis integrity Audit. Method Overcl. False ref. Hall. sig. Synth. Lit. p Verif. â Local/Open-weight (GPT-OSS-20B) SMb: Features 44.7 7.4 10.9 0.2 0.7 81.9 SMc: Code 65.1 6.7 8.7 0.2 1.4 82.3 SMd: Agentic 25.6 10.2 1.5 0.2 12.2 53.6 SMe: Pipeline 3.8 8.5 0.0 0.9 17.2 57.7 Veritas 7.2 24.2 2.7 10.9 2.4 77.3 Frontier (GPT-5.2) SMb: Features 34.7 15.3 11.7 2.2 4.1 82.8 SMc: Code 45.4 8.6 11.0 1.6 2.7 81.2 SMd: Agentic 14.8 11.7 0.0 0.6 3.3 70.5 SMe: Pipeline 13.0 7.9 0.0 0.5 10.9 68.1 Veritas 20.0 12.6 0.4 3.6 15.0 86.6 L1 scaling behavior. Evidence accuracy on L1 (metadata-only) reveals a pronounced model-capacity dependence: local Veritas drops to 54.4% on L1 versus 87.0% on L2, while frontier Veritas maintains near-parity (90.0% vs. 90.4%). Although L1 hypotheses require no imaging computation, they demand nontrivial statistical designâsurvival model specification, covariate adjustment, and appropriate test selectionâcapabilities that benefit disproportionately from larger model capacity. By contrast, L2 hypotheses (single imaging metric comparisons) rely on more mechanical code generation that local models already handle effectively. 4.3 Diagnostic Analysis Table 5 reports diagnostic rates that characterize how methods fail, beyond aggregate accuracy. SMa is excluded because it produces no executable statistical output, making code-level diagnostics inapplicable. One-shot baselines (SMb, SMc) show high overclaim rates (35â65%) and hallucinated significance (9â12%), producing YES verdicts unsupported by their own statistical outputs. Iterative and structured baselines (SMd, SMe) largely eliminate hallucinated significance (⤠1.5%) through self-correction, but trade this for higher literal p-value assignment (3â17%), where models hard-code p-values rather than computing them from data. Local Veritas achieves the lowest overclaim rate (7.2%) but shows higher false refutation (24.2%) and synthetic-data violations (10.9%)âa conservative error profile where the system under-claims rather than over-claims. Frontier Veritas shifts this balance: overclaim rises to 20.0% while false refutation drops to 12.6%, indicating a more aggressive interpretation style. The literal p-value rate also increases (15.0%), suggesting that frontier code-generation agents more frequently embed result values directly. Both Veritas configurations maintain low hallucinated significance (⤠2.7%), confirming that the phased pipeline prevents the most egregious failure mode. The verifiability column (Verif.) quantifies auditability: the fraction of all runs that report the four core statistical outputs required for independent verification. Among methods operating on raw data (SMd, SMe, Veritas), Veritas achieves the highest verifiability (77.3% local, 86.6% frontier), substantially exceeding SMd (53.6%, 70.5%) and SMe (57.7%, 68.1%). SMb and SMc achieve higher absolute rates (âź 82%) but operate on pre-computed features, yielding simpler code with higher completion rates, at the cost of zero L5 evidence accuracy. This confirms that Veritasâs phased decomposition not only improves accuracy but also produces more inspectable outputs, directly supporting the claim that its failures are recoverable through artifact inspection. 4.4 Ablation Studies Table 6 reports ablation results using the local Veritas configuration (10 runs/hypothesis). We evaluate context window size, critic removal, and temperature zero decoding. Vision source ablations (GT masks, SAT-Pro) are reported in the supplementary material. Table 6: Ablation results for Veritas (local, 10 runs/hypothesis). Default: 16k context, critic enabled in execution phases, T=0.2T=0.2. Per-dataset split shows the ACDC vs. UCSF-PDGM gap across configurations. ACDC UCSF-PDGM Combined Configuration Evid. Verd. Evid. Verd. Evid. Verd. Default (16k context) 91.0 87.9 36.7 49.0 63.4 71.4 Context 8k 91.4 91.7 29.0 45.7 59.7 72.0 Context 32k 90.0 87.1 31.7 51.8 60.3 72.7 No Critic 90.7 90.1 32.0 50.5 60.8 73.2 Temperature zero 90.3 88.7 27.7 47.5 58.5 72.6 ACDC evidence remains stable across all ablations (âź 90â91%), indicating that cardiac hypotheses are effectively solved by the base architecture; sensitivity lies almost entirely in UCSF-PDGM evidence, making it the discriminating benchmark. The default 16k context achieves the best UCSF evidence (36.7%): reducing to 8k degrades UCSF evidence by â-7.7p, as complex multi-step analyses (survival, regression) cannot fit within a tighter working memory, while expanding to 32k provides no uplift (â-5.0p), suggesting additional context introduces noise without benefit. Deterministic decoding (T=0T=0) similarly hurts UCSF evidence (â-9.0p), consistent with complex hypotheses benefiting from stochastic exploration during code generation. Critic removal has minimal impact (++1.8p combined verdict), indicating the 8-attempt code revision loop already provides sufficient error correction. 4.5 Analysis 4.5.1 Execution vs. Reasoning Across all methods, verdict accuracy consistently exceeds evidence-label accuracy (e.g., Veritas local: 71.4% vs. 63.4%; frontier: 81.0% vs. 70.0%), indicating that the primary bottleneck is producing correct statistical outputs (code generation), not reasoning about them (interpretation). The gap is most striking for frontier Veritas on UCSF-PDGM: verdict accuracy (83.2%) exceeds ACDC verdict (78.9%) despite much lower evidence accuracy (50.7% vs. 90.0%), identifying code generation for complex analyses (survival models, multivariate regression) as the primary improvement target. Notably, this verdictâevidence gap is much narrower for Veritas (8â11 p overall) than for single-model baselines on comparable tiers (e.g., up to 92% verdict vs. 0% evidence accuracy on L5), confirming that Veritasâs verdicts remain substantially grounded in computed statistics rather than prior knowledge alone. 4.5.2 Epistemic Label Analysis The four-label evidence framework is essential for fair evaluation. Without the Underpowered label, several UCSF-PDGM hypotheses with small subgroups would be conflated with true negatives: glioma_31 (1p/19q codeletion, n=13n=13 vs. n=86n=86, power==0.38) is correctly classified as Underpowered rather than Refuted; glioma_14 (IDH-mutant Grade IV, n=28n=28 vs. n=368n=368, power==0.72) is similarly borderline. In a binary evaluation, a system that conservatively labels every non-significant result as ârefutedâ would achieve spuriously high accuracy. The underpowered distinction forces systems to demonstrate genuine epistemic awareness. 5 Discussion and Conclusion We presented Veritas, a multi-agent framework for autonomous, auditable hypothesis testing inspired by clinician-scientist workflows. Its design targets three bottlenecks highlighted in the introduction: the fragmented coordination of clinical, imaging, programming, and statistical expertise; the lack of reproducible analysis trails in image-derived studies; and the misinterpretation of underpowered findings as true negatives. Our agentic system decomposes the workflow into verifiable phases and code, making every step auditable and reproducible. This auditability-by-design means that Veritasâs failures are recoverable in a way that baseline failures are not. Furthermore, we introduce an epistemic evidence label framework that distinguishes refuted from underpowered hypotheses. Veritas achieves excellent performance, even compared to baselines that receive human assistance (e.g., pre-computed per-group statistics or per-patient feature tables). By its design, Veritas enables small, locally-deployable models to perform complex reasoning that typically requires frontier models. Beyond accuracy, the evidence label framework functions as a mechanistic decision rule: given observed statistics (p,δ,Ď)(p,δ,Ď), the label is a deterministic function with no free parameters beyond Îą and the SESOI δ0 _0. This eliminates evaluation ambiguities common in agent benchmarks. We view this as a contribution beyond medical imaging: any benchmark involving statistical testing could adopt this framework to avoid conflating negative results with underpowered designs. Most agent evaluations report only end-to-end accuracy. For scientific applications, how a conclusion is reached matters as much as whether it is correct. Our evidence grounding metric (Verif., Table 5) quantifies this: among methods operating on raw data, Veritas produces the highest rate of fully verifiable outputs (86.6% frontier), meaning even its incorrect conclusions can be traced, diagnosed, and corrected through artifact inspection. We are convinced that Veritas can accelerate clinical research by reducing the coordination overhead and execution burden of tasks such as patient selection, image analysis and biomarker extraction, and statistical testing. Ultimately, a clinical observational study should always be finalized by a human expert, a process that is substantially facilitated through the availability and auditability of the reasoning steps, artifacts, and code. Limitations Segmentation dependency. All imaging-derived conclusions depend on segmentation quality. While we quantify the difference between ground-truth segmentations and the SAT model in the supplement, the framework treats segmentation as a point estimate without uncertainty propagation. Future work should incorporate segmentation uncertainty into the statistical pipeline. Benchmark scope. Our evaluation covers two clinical domains that use MRI images analyzed by a specific segmentation model. Extending to other clinical domains and modalities (CT, histopathology) requires new segmentation models, which however are widely available. Furthermore, the hypothesis bank does not yet cover longitudinal analyses or causal inference which would enhance Veritasâs applicability to other types of studies. Ground truth. Our ground truth verdicts are established from the given datasets using expert-validated reference computations. This was strictly necessary to ensure mechanical verifiability in our evaluation; however, this may lead to a ground truth that reflects dataset-specific effects differing from population-level effects. Code and Data. Code, prompts, hypothesis bank, and evaluation pipeline are available at https://github.com/LucZot/veritas. Acknowledgments We thank Adina Scheinfeld for help with figures and writing. References [1] S. Agrawal, H. B. Anadkat, K. K. Athimoolam, H. Bhardwaj, T. Chowdhury, S. Gao, P. K. Kamat, V. Makwana, M. H. Shariff, A. Badkul, et al. (2026) Can ai conduct autonomous scientific research? case studies on two real-world tasks. bioRxiv, p. 2026â01. Cited by: §2. [2] Anthropic (2024) The claude 3 model family: opus, sonnet, haiku. Anthropic. Cited by: §2. [3] J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang (2024) ResearchAgent: iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738. Cited by: §2. [4] S. Bannur, S. Hyland, Q. Liu, F. Perez-Garcia, M. Ilse, D. C. Castro, B. Boecking, H. Sharma, K. Bouzid, A. Thieme, et al. (2023) Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15016â15027. Cited by: §2. [5] D. J. Benjamin, J. O. Berger, M. Johannesson, B. A. Nosek, E. Wagenmakers, R. Berk, K. A. Bollen, B. Brembs, L. Brown, C. Camerer, et al. (2018) Redefine statistical significance. Nature human behaviour 2 (1), p. 6â10. Cited by: §2. [6] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester, et al. (2018) Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?. IEEE transactions on medical imaging 37 (11), p. 2514â2525. Cited by: Table 19, §1, §4.1.1. [7] K. S. Button, J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafò (2013) Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience 14 (5), p. 365â376. Cited by: §2. [8] E. Calabrese, J. E. Villanueva-Meyer, J. D. Rudie, A. M. Rauschecker, U. Baid, S. Bakas, S. Cha, J. T. Mongan, and C. P. Hess (2022) The university of california san francisco preoperative diffuse glioma mri dataset. Radiology: Artificial Intelligence 4 (6), p. e220058. Cited by: Table 19, §1, §4.1.1. [9] R. Cao, M. Chen, J. Chen, Z. Cui, Y. Feng, B. Hui, Y. Jing, K. Li, M. Li, J. Lin, et al. (2026) Qwen3-coder-next technical report. arXiv preprint arXiv:2603.00729. Cited by: §3.2. [10] N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al. (2025) Sam 3: segment anything with concepts. arXiv preprint arXiv:2511.16719. Cited by: §2. [11] W. Chen, X. Ma, X. Wang, and W. W. Cohen (2022) Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. Cited by: §2, §2. [12] W. Chen, Y. Dong, Z. Ding, Y. Shi, Y. Zhou, F. Zeng, Y. Luo, T. Lin, Y. Su, Y. Wu, et al. (2025) RadFabric: agentic ai system with reasoning capability for radiology. arXiv preprint arXiv:2506.14142. Cited by: §2. [13] J. Cohen (1994) The earth is round (p<. 05).. American psychologist 49 (12), p. 997. Cited by: §2. [14] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §2. [15] C. Davidson-Pilon (2019) Lifelines: survival analysis in python. Journal of Open Source Software 4 (40), p. 1317. External Links: Document, Link Cited by: §D.7. [16] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig (2023) Pal: program-aided language models. In International conference on machine learning, p. 10764â10799. Cited by: §2. [17] S. Gao, A. Fang, Y. Huang, V. Giunchiglia, A. Noori, J. R. Schwarz, Y. Ektefaie, J. Kondic, and M. Zitnik (2024) Empowering biomedical discovery with ai agents. Cell 187 (22), p. 6125â6151. Cited by: §2. [18] A. Ghafarollahi and M. J. Buehler (2025) SciAgents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning. Advanced Materials 37 (22), p. 2413523. Cited by: §2. [19] A. E. Ghareeb, B. Chang, L. Mitchener, A. Yiu, C. J. Szostkiewicz, J. M. Laurent, M. T. Razzak, A. D. White, M. M. Hinks, and S. G. Rodriques (2025) Robin: a multi-agent system for automating scientific discovery. arXiv preprint arXiv:2505.13400. Cited by: §2. [20] S. N. Goodman and J. A. Berlin (1994) The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Annals of internal medicine 121 (3), p. 200â206. Cited by: §D.8. [21] J. Gottweis, W. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, et al. (2025) Towards an ai co-scientist. arXiv preprint arXiv:2502.18864. Cited by: §2. [22] J. M. Hoenig and D. M. Heisey (2001) The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician 55 (1), p. 19â24. Cited by: §D.8. [23] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, et al. (2023) MetaGPT: meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations, Cited by: §2. [24] A. Hoopes, N. Dey, V. I. Butoi, J. V. Guttag, and A. V. Dalca (2024) VoxelPrompt: a vision agent for end-to-end medical image analysis. arXiv preprint arXiv:2410.08397. Cited by: §2. [25] K. Huang, Y. Jin, R. Li, M. Y. Li, E. Candès, and J. Leskovec (2025) Automated hypothesis validation with agentic sequential falsifications. arXiv preprint arXiv:2502.09858. Cited by: §2. [26] J. P. Ioannidis (2005) Why most published research findings are false. PLoS medicine 2 (8), p. e124. Cited by: §2. [27] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein (2021) NnU-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18 (2), p. 203â211. Cited by: §2. [28] Q. Jin, Z. Wang, Y. Yang, Q. Zhu, D. Wright, T. Huang, N. Khandekar, N. Wan, X. Ai, W. J. Wilbur, et al. (2025) AgentMD: empowering language agents for risk prediction with large-scale clinical tool learning. Nature Communications 16 (1), p. 9377. Cited by: §2. [29] Y. Kim, C. Park, H. Jeong, Y. S. Chan, X. Xu, D. McDuff, H. Lee, M. Ghassemi, C. Breazeal, and H. W. Park (2024) Mdagents: an adaptive collaboration of llms for medical decision-making. Advances in Neural Information Processing Systems 37, p. 79410â79452. Cited by: §2. [30] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, et al. (2023) Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4015â4026. Cited by: §2. [31] D. Lakens, A. M. Scheel, and P. M. Isager (2018) Equivalence testing for psychological research: a tutorial. Advances in methods and practices in psychological science 1 (2), p. 259â269. Cited by: §2. [32] D. Lakens (2017) Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social psychological and personality science 8 (4), p. 355â362. Cited by: §2. [33] LangChain AI (2024) LangGraph: building stateful, multi-actor applications with llms. Note: https://github.com/langchain-ai/langgraphAccessed: 2026-03-03 Cited by: §D.7. [34] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao (2023) Llava-med: training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36, p. 28541â28564. Cited by: §2. [35] J. Li, Y. Lai, W. Li, J. Ren, M. Zhang, X. Kang, S. Wang, P. Li, Y. Zhang, W. Ma, et al. (2024) Agent hospital: a simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957. Cited by: §2. [36] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024) The ai scientist: towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. Cited by: §2. [37] J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang (2024) Segment anything in medical images. Nature communications 15 (1), p. 654. Cited by: §2. [38] J. Ma, Z. Yang, S. Kim, B. Chen, M. Baharoon, A. Fallahpour, R. Asakereh, H. Lyu, and B. Wang (2025) Medsam2: segment anything in 3d medical images and videos. arXiv preprint arXiv:2504.03600. Cited by: §2. [39] U. Mall, C. P. Phoo, M. Chiquier, B. Hariharan, K. Bala, and C. Vondrick (2025) Disciple: learning interpretable programs for scientific visual discovery. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29258â29267. Cited by: §2. [40] H. Nori, M. Daswani, C. Kelly, S. Lundberg, M. T. Ribeiro, M. Wilson, X. Liu, V. Sounderajah, J. Carlson, M. P. Lungren, et al. (2025) Sequential diagnosis with language models. arXiv preprint arXiv:2506.22405. Cited by: §2. [41] Ollama Team (2023) Ollama: get up and running with large language models. Note: https://ollama.com/Accessed: 2026-03-03 Cited by: §D.7. [42] OpenAI (2025-01-23) GPT-5 mini. Note: Accessed: 2026-03-03 External Links: Link Cited by: §3.2. [43] OpenAI (2025-12-11) Introducing gpt-5.2. Note: Accessed: 2026-03-03 External Links: Link Cited by: §2, §3.2, §4.1.2. [44] OpenAI (2025-08-05) Introducing gpt-oss. Note: Accessed: 2026-03-03 External Links: Link Cited by: §3.2, §4.1.2. [45] OpenRouter Team (2024) OpenRouter: a unified interface for llms. Note: https://openrouter.ai/Accessed: 2026-03-03 Cited by: §D.7. [46] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2024) Gorilla: large language model connected with massive apis. Advances in Neural Information Processing Systems 37, p. 126544â126565. Cited by: §2. [47] J. Pearl (2009) Causality: models, reasoning, and inference. Cambridge University Press. Cited by: §C.2. [48] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. (2023) Toolllm: facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. Cited by: §2. [49] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. (2024) Sam 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. Cited by: §2. [50] M. Rokuss, M. Langenberg, Y. Kirchhoff, F. Isensee, B. Hamm, C. Ulrich, S. Regnery, L. Bauer, E. Katsigiannopulos, T. Norajitra, et al. (2025) Voxtell: free-text promptable universal 3d medical image segmentation. arXiv preprint arXiv:2511.11450. Cited by: §2. [51] T. Schick, J. Dwivedi-Yu, R. DessĂŹ, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in neural information processing systems 36, p. 68539â68551. Cited by: §2. [52] S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. Barsoum (2025) Agent laboratory: using llm agents as research assistants. Findings of the Association for Computational Linguistics: EMNLP 2025, p. 5977â6043. Cited by: §2. [53] H. Su, R. Chen, S. Tang, Z. Yin, X. Zheng, J. Li, B. Qi, Q. Wu, H. Li, W. Ouyang, et al. (2025) Many heads are better than one: improved scientific idea generation by a llm-based multi-agent system. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 28201â28240. Cited by: §2. [54] D. SurĂs, S. Menon, and C. Vondrick (2023) Vipergpt: visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, p. 11888â11898. Cited by: §2. [55] K. Swanson, W. Wu, N. L. Bulaong, J. E. Pak, and J. Zou (2025) The virtual lab of ai agents designs new sars-cov-2 nanobodies. Nature 646 (8085), p. 716â723. Cited by: §B.12, Table 17, §2. [56] X. Tang, A. Zou, Z. Zhang, Z. Li, Y. Zhao, X. Zhang, A. Cohan, and M. Gerstein (2024) Medagents: large language models as collaborators for zero-shot medical reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, p. 599â621. Cited by: §2. [57] J. J. Van Griethuysen, A. Fedorov, C. Parmar, A. Hosny, N. Aucoin, V. Narayan, R. G. Beets-Tan, J. Fillion-Robin, S. Pieper, and H. J. Aerts (2017) Computational radiomics system to decode the radiographic phenotype. Cancer research 77 (21), p. e104âe107. Cited by: §2. [58] Q. Wang, D. Downey, H. Ji, and T. Hope (2024) Scimon: scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 279â299. Cited by: §2. [59] X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji (2024) Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, Cited by: §2, §2. [60] Z. Wang, J. Wu, L. Cai, C. H. Low, X. Yang, Q. Li, and Y. Jin (2025) Medagent-pro: towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow. arXiv preprint arXiv:2503.18968. Cited by: §2. [61] R. L. Wasserstein, A. L. Schirm, and N. A. Lazar (2019) Moving to a world beyond âp< 0.05â. Vol. 73, Taylor & Francis. Cited by: §2. [62] J. Wei, Y. Yang, X. Zhang, Y. Chen, X. Zhuang, Z. Gao, D. Zhou, G. Wang, Z. Gao, J. Cao, et al. (2025) From ai for science to agentic science: a survey on autonomous scientific discovery. arXiv preprint arXiv:2508.14111. Cited by: §2. [63] D. Westreich and S. Greenland (2013) The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. American journal of epidemiology 177 (4), p. 292â298. Cited by: §C.2. [64] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. (2024) Autogen: enabling next-gen llm applications via multi-agent conversations. In First conference on language modeling, Cited by: §2. [65] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §3.2. [66] Z. Yang, W. Liu, B. Gao, Y. Liu, W. Li, T. Xie, L. Bing, W. Ouyang, E. Cambria, and D. Zhou (2025) Moose-chem2: exploring llm limits in fine-grained scientific hypothesis discovery via hierarchical search. arXiv preprint arXiv:2505.19209. Cited by: §2. [67] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: §2. [68] H. G. Zhang, P. Eckmann, J. Miao, A. B. Mahon, and J. Zou (2026) The virtual biotech: a multi-agent ai framework for therapeutic discovery and development. bioRxiv, p. 2026â02. Cited by: §2. [69] W. Zhang, M. Qiao, C. Zang, S. Niederer, P. M. Matthews, W. Bai, and B. Kainz (2025) Multi-agent reasoning for cardiovascular imaging phenotype analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 429â439. Cited by: §2. [70] T. Zhao, Y. Gu, J. Yang, N. Usuyama, H. H. Lee, T. Naumann, J. Gao, A. Crabtree, J. Abel, C. Moung-Wen, et al. (2024) Biomedparse: a biomedical foundation model for image parsing of everything everywhere all at once. arXiv preprint arXiv:2405.12971. Cited by: §2. [71] Z. Zhao, Y. Zhang, C. Wu, X. Zhang, X. Zhou, Y. Zhang, Y. Wang, and W. Xie (2025) Large-vocabulary segmentation for medical images with text prompts. NPJ Digital Medicine 8 (1), p. 566. Cited by: §D.7, §2, §3.2. [72] J. Zhu, A. Hamdi, Y. Qi, Y. Jin, and J. Wu (2024) Medical sam 2: segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874. Cited by: §2. [73] A. Zwanenburg, M. Vallières, M. A. Abdalah, H. J. Aerts, V. Andrearczyk, A. Apte, S. Ashrafinia, S. Bakas, R. J. Beukinga, R. Boellaard, et al. (2020) The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology 295 (2), p. 328â338. Cited by: §2. Appendix A Example Workflow Walkthrough This section illustrates the end-to-end Veritas pipeline on two representative hypotheses: one imaging-based group comparison (ACDC) and one multivariate survival analysis (UCSF-PDGM). For each, we show the artifacts produced at each phase and demonstrate the full audit trail. Both examples use the frontier model configuration (GPT-5.2). The colored phase boxes below follow the phase decomposition in Figure 1. A.1 Example 1: Cardiac LVEF Group Difference (L2) Hypothesis âDCM patients show significantly lower LVEF than normal controls.â This is an L2 (imaging-only) hypothesis requiring LV cavity segmentation at two cardiac phases and a derived metric (ejection fraction). Phase 1 â Planning The three-agent team aligns on the analysis plan in a structured discussion. The PI confirms the target quantity (LVEF = ((LVEDV â- LVESV) / LVEDV) Ă 100), rejects proxy substitution, and recommends an unadjusted primary analysis. The Imaging Specialist queries the Imaging Analysis API, confirms structure "left heart ventricle" is available, and retrieves cohort sizes (DCM: n=30n=30, NOR: n=30n=30). The Statistician proposes a Welch two-sample t-test (two-sided, Îą=0.05Îą=0.05) and computes planning power (Ď=0.48Ď=0.48 at the fixed Phase 1 reference SESOI d=0.5d=0.5), flagging the study as underpowered for a medium-effect planning target. This does not stop execution: Phase 1 power is advisory and only affects interpretation if the subsequent result is non-significant. In the benchmark, cardiac_01 carries the evaluator SESOI profile loose (d0=0.8d_0=0.8), for which the ground-truth power is 0.86. The Phase 1 output is an executable JSON plan contract: ⏠"feasibility": "status": "TESTABLE", "groups": ["DCM", "NOR"], "structures": ["left heart ventricle"], "observations": ["ED", "ES"], "metrics": ["LVEF"], "statistical_test": "Welch two-sample t-test", "a_priori_power": "d": 0.5, "n1": 30, "n2": 30, "power": 0.478 Phase 2A â Segmentation The agents construct a segmentation request specifying the LV cavity structure at both ED and ES frames. The SAT backend segments all 60 patients (30 DCM + 30 NOR), producing per-patient binary masks. Figure 3 shows exemplary segmentation overlays on short-axis cine MRI slices. Figure 3: Our systemâs LV cavity segmentation (red overlay) on short-axis cardiac cine MRI at end-diastole for (a) a DCM patient (patient001) showing marked LV dilation and (b) a normal control (patient071). Phase 2B â Statistical Analysis The coding agent generates a Python script that: 1. Iterates over all patients, loading LV masks at ED and ES via the Imaging Analysis API 2. Computes voxel-level volumes using sat.calculate_volume(mask, spacing) 3. Derives per-patient LVEF from the exact formula 4. Applies five prespecified QC checks (finite volumes, EDV >0>0, ESV âĽ0⼠0, ESV ⤠EDV) 5. Performs a Welch t-test and saves complete statistics to statistical_results.json Key excerpt from the generated code: ⏠1for pid in patients: 2 md = sat.get_patient_metadata(pid) 3 if md.get("group") not in ["DCM", "NOR"]: 4 continue 5 obs_map = sat.get_observation_identifiers(pid) 6 for obs in ["ED", "ES"]: 7 mask = sat.load_structure_mask( 8 results_db_path, pid, "leftâŁheartâŁventricle", 9 source_image_contains=obs_map[obs]) 10 vols[obs] = sat.calculate_volume(mask[0]["mask"], mask[0]["spacing"]) 11 lvef = ((vols["ED"] - vols["ES"]) / vols["ED"]) * 100 The analysis produces a boxplot visualization (Figure 4), and the following statistical output: Statistic Value Test Welch two-sample t-test n (DCM / NOR) 30 / 30 Mean LVEF (DCM) 18.6% Âą 8.2% Mean LVEF (NOR) 61.2% Âą 5.3% Mean difference â-42.6% (95% CI: â-46.1 to â-39.1) t-statistic â-23.83 (df == 49.6) p-value 7.33Ă10â297.33Ă 10^-29 Cohenâs d â-6.15 Figure 4: LVEF distribution by group, generated by the Phase 2B analysis code, showing clear separation between DCM (mean 18.6%18.6\%) and NOR (mean 61.2%61.2\%), p=7.33Ă10â29p=7.33Ă 10^-29, Cohenâs d=â6.15d=-6.15. Phase 3 â Interpretation The agent team reviews the Phase 2B output. Given the highly significant result (pâŞ0.001p 0.001), large effect size (|d|=6.15|d|=6.15), and effect direction consistent with the hypothesis (DCM << NOR), the team reaches a unanimous verdict: YES with evidence label Supported. This matches the ground-truth label, demonstrating correct end-to-end execution for a straightforward L2 hypothesis. A.2 Example 2: MGMT Methylation Survival Analysis (L5) Hypothesis âIn Grade IV glioblastoma patients, MGMT promoter methylation is associated with longer overall survival after adjusting for age and extent of resection.â This is an L5 (multivariate) hypothesis requiring Cox proportional hazards modeling with covariate adjustmentâthe most complex analysis tier in the benchmark. Phase 1 â Planning The PI restricts the analysis to Grade IV patients only and confirms the outcome variables (survival_days, survival_status). The Statistician specifies a Cox PH model with mgmt_status as primary predictor and age + extent_of_resection as covariates, and recommends complete-case analysis with Schoenfeld residual checks for the proportional hazards assumption. No imaging structures are required (structures: []). Phase 2B â Statistical Analysis The coding agent generates a survival analysis script using lifelines: ⏠1from lifelines import CoxPHFitter, KaplanMeierFitter 2# Restrict to GradeIV, encode MGMT, one-hot EOR (ref=GTR) 3cph = CoxPHFitter() 4cph.fit(df[["survival_days","survival_status", 5 "mgmt_methylated","age","eor_Biopsy","eor_STR"]], 6 duration_col="survival_days", event_col="survival_status") The script also generates Kaplan-Meier survival curves stratified by MGMT status (Figure 5) and performs Schoenfeld residual tests for PH assumption validation. Key results: Statistic Value Test Cox proportional hazards Cohort Grade IV (n=378n=378, 217 events) Primary predictor MGMT methylation (methylated =1=1) Covariates Age, extent of resection (Biopsy/STR vs. GTR) Hazard ratio (MGMT) 0.712 (95% CI: 0.532â0.954) p-value (MGMT) 0.023 Events per variable 54.25 Schoenfeld (MGMT) p=0.277p=0.277 (PH satisfied) Figure 5: Kaplan-Meier survival curves by MGMT methylation status in Grade IV glioblastoma patients (methylated n=273n=273, unmethylated n=105n=105), generated by the Phase 2B analysis code. Unadjusted log-rank p=0.013p=0.013; the Cox PH model with age and extent-of-resection covariates yields HR =0.712=0.712 (p=0.023p=0.023). Phase 3 â Interpretation The agents interpret the hazard ratio (HR =0.712<1=0.712<1) as indicating that MGMT-methylated patients have âź29% 29\% lower hazard of death after adjustment for age and extent of resection. The PH assumption holds for the primary predictor (p=0.277p=0.277). Verdict: YES with evidence label Supported. This example demonstrates Veritasâs ability to handle the most complex analysis tier (L5), including correct covariate specification, assumption checking, and multivariate interpretation. A.3 Artifact Trail and Auditability Table 7 summarizes the complete set of artifacts produced by a single Veritas run. Every phase produces human-readable, inspectable outputs that enable post-hoc verification of the entire analysis pipeline. Table 7: Artifacts produced per phase in a Veritas run. All artifacts are saved to disk and available for post-hoc inspection. Phase Artifact Format Enables Phase 1 Team discussion transcript Markdown Review of scientific reasoning Phase 1 Analysis plan contract JSON Verify test selection, feasibility Phase 1 Power analysis JSON Assess statistical adequacy Phase 2A Segmentation request JSON Verify correct structures/observations Phase 2A SAT execution log JSON Confirm data provenance Phase 2B Generated analysis code Python Full reproducibility Phase 2B Statistical results JSON Verify p, effect size, CIs Phase 2B Visualization plots PNG Visual sanity checks Phase 2B Execution logs Text Debug failures Phase 3 Interpretation transcript Markdown Review verdict reasoning Phase 3 Final verdict + evidence label JSON End-to-end conclusion These artifacts collectively satisfy the auditability criteria defined in SectionËD.4: a reviewer can trace from the natural-language hypothesis through the analysis plan, verify which imaging structures were queried, inspect the generated code line by line, check the statistical output against the code, and evaluate whether the interpretation correctly reflects the numerical evidence. Appendix B Additional Experimental Results B.1 Full Per-Method Stratified Results Table 8 extends Table 4 to all methods and both model families, reporting non-L0 evidence-label and verdict accuracy per dataset. Table 8: Per-dataset non-L0 evidence-label (Ev) and verdict (Vd) accuracy (%) for all methods and model families. Completion rate shown for Veritas (baselines always produce output). Best per column in bold. ACDC UCSF-PDGM Method Family Ev Vd Ev Vd Compl. Local/Open-weight models SMa: Direct local 0.0 83.0 0.0 30.1 â SMb: Features local 76.9 72.7 40.0 55.2 â SMc: Code local 76.2 67.9 3.0 58.5 â SMd: Agentic local 78.3 87.5 15.3 38.5 â SMe: Pipeline local 92.4 90.7 19.0 26.2 â Veritas local 91.0 87.9 36.7 49.0 78.1 Frontier models (GPT-5.2) SMa: Direct frontier 0.0 85.2 0.0 27.7 â SMb: Features frontier 77.9 84.1 50.0 57.7 â SMc: Code frontier 79.0 85.1 27.7 60.9 â SMd: Agentic frontier 95.9 96.6 34.7 53.9 â SMe: Pipeline frontier 90.3 96.9 32.3 35.1 â Veritas frontier 90.0 78.9 50.7 83.2 87.5 Key observations: ⢠Performance on ACDC dataset is near-saturated for frontier baselines: SMd and SMe exceed 96% verdict accuracy on the relatively straightforward cardiac dataset, leaving little room for improvement. Veritasâs lower ACDC verdict accuracy (78.9%) reflects pipeline completion failures rather than reasoning errorsâcompleted runs achieve high accuracy (see Table 14). ⢠UCSF-PDGM differentiates methods: On the more challenging neuro-oncology datasetâwhich includes survival analyses, multivariate models, and metadata-based group filteringâVeritas frontier achieves the highest evidence-label (50.7%) and verdict (83.2%) accuracy, outperforming the best baseline by >>16p (evidence) and >>29p (verdict). ⢠Open-weight gap: Switching from frontier to open-weight models reduces UCSF-PDGM evidence accuracy by 10â20p across methods, reflecting the difficulty of survival analysis and multivariate modeling for smaller models. ⢠SMa confirms code necessity: Pure reasoning from summary statistics (SMa) achieves 0% evidence-label accuracy across all conditions, confirming that producing verifiable statistical outputs requires executable code generation. ⢠Full autonomy: Veritas is the only method that executes the complete pipelineâfrom hypothesis intake through image segmentation, code generation, and statistical interpretationâwithout any human intervention or pre-computed inputs. All baselines rely on at least one shortcut: pre-extracted features (SMa, SMb), a single-shot prompt (SMc), or a simplified agentic loop without vision grounding (SMd, SMe). B.2 Additional Ablation Experiments The main paper reports the five highest-impact ablations (GT masks, context 8k/32k, no critic, temperature zero). We place the remaining ablations here to keep the main narrative focused. Table 9: Complete ablation results for Veritas (local, 10 runs/hypothesis). Default: SAT-Nano masks, 16k context, critic in execution phases only, T=0.2T=0.2, standard model assignment (20B PI/imaging, 8B statistician/critic, 30B coder). All metrics are non-L0 evidence-label (Ev) and verdict (Vd) accuracy (%). ACDC UCSF-PDGM Combined Scope Configuration Ev Vd Ev Vd Ev Vd Vision source Default (SAT-Nano) 91.0 87.9 36.7 49.0 63.4 71.4 GT expert masks 82.1 85.7 13.0 47.3 46.9 77.2 SAT Pro masks 92.1 90.9 22.3 38.2 56.6 70.6 Context window Default (16k) 91.0 87.9 36.7 49.0 63.4 71.4 8k context 91.4 91.7 29.0 45.7 59.7 72.0 32k context 90.0 87.1 31.7 51.8 60.3 72.7 Critic Default (execution phases) 91.0 87.9 36.7 49.0 63.4 71.4 No critic 90.7 90.1 32.0 50.5 60.8 73.2 Critic in all phases 81.4 85.6 26.0 39.2 53.2 66.5 Decoding Default (T=0.2T=0.2, uniform) 91.0 87.9 36.7 49.0 63.4 71.4 Temperature zero (T=0T=0) 90.3 88.7 27.7 47.5 58.5 72.6 Role-specific T/top-pâ 91.4 89.2 32.3 44.3 61.4 70.4 Model assignment Default (20B/8B/30B) 91.0 87.9 36.7 49.0 63.4 71.4 Small model (all 8B) 33.8 66.1 4.7 64.7 19.0 65.9 General coding (no coder model) 90.0 88.5 29.5 62.8 59.5 80.0 â Role-specific T/top-p: PI/imaging T=1.0T=1.0/top-p=1.0p=1.0; statistician/critic T=0.6T=0.6/top-p=0.95p=0.95; coder T=0.7T=0.7/top-p=0.8p=0.8. The structured multi-phase pipeline design makes Veritas robust to component-level changes: swapping vision backends, adjusting critic scope, or substituting model variants produces relatively moderate accuracy shifts, whereas the overall architecture consistently drives performance. ⢠Vision source: GT expert masks show a notable evidence drop (â-16.5p combined) driven by UCSF-PDGM (â-23.7p), while SAT-Pro is closer to default (â-6.8p combined). On ACDC, GT masks slightly reduce evidence accuracy (â-8.9p) compared to SAT-Nanoâmost ACDC hypotheses are insensitive to segmentation quality, and the different data format may introduce minor integration friction. On UCSF-PDGM, where most hypotheses are metadata-driven, all vision sources perform similarly at low levels. ⢠Critic scope: Extending the critic beyond code execution into planning and interpretation phases degrades combined evidence accuracy (â-10.2p) and verdict (â-4.9p). Qualitative inspection reveals three failure modes: (i) the critic raises spurious feasibility concerns during planning, causing the PI to mark testable hypotheses as untestable or to substitute proxy metrics; (i) it advocates covariate-adjusted tests (e.g. ANCOVA) over simple group comparisons, changing the test family and triggering evaluation mismatches; and (i) the additional discussion rounds inflate the context passed to downstream code-generation phases, degrading statistical code quality. These patterns suggest that the structured multi-phase pipeline already provides sufficient guardrails, and an additional adversarial critic during deliberative phases introduces noise rather than actionable corrections. ⢠Model capacity (all 8B): Replacing 20B/30B models with 8B variants causes a large evidence-accuracy drop (â-44.4p combined) while verdict accuracy degrades only modestly (â-5.5p). Smaller models can still infer correct binary verdicts from partial outputs but lack the capacity to generate complete, well-formed statistical code. ⢠Coding specialization: Replacing the coding-specialized 30B model with a general-purpose 30B model yields comparable evidence accuracy (â-3.9p) but notably higher verdict accuracy (+8.6p), suggesting that the general model produces simpler, more robust code that completes more often. ⢠Role-specific decoding: Assigning model-recommended temperature and nucleus sampling parameters per role (PI/imaging: T=1.0T=1.0; statistician/critic: T=0.6T=0.6/top-p=0.95p=0.95; coder: T=0.7T=0.7/top-p=0.8p=0.8) yields a marginal ACDC uplift (+0.4p evidence, +1.3p verdict) compared to the uniform T=0.2T=0.2 baseline, but degrades on UCSF-PDGM (â-4.4p evidence, â-4.7p verdict), consistent with the pattern observed for temperature zero. Complex multi-step reasoning tasks appear to benefit from conservative, low-variance sampling rather than role-tailored diversity. B.3 Run-to-Run Reproducibility To characterize the stochastic variability of Veritas, we repeat two hypotheses 100 times each under default settings (open-weight, SAT-Nano, 16k context, T=0.2T=0.2): 1. cardiac_01_dcm_lvef_lower (L2, positive control): a straightforward LV ejection-fraction group comparison. 2. glioma_25_mgmt_survival_longer_adjust_age_eor (L5, positive control): a multivariate Cox regression with confound adjustment. These span the easiest (L2, single imaging metric) and hardest (L5, multivariate survival) testable tiers, one per dataset. Table 10: Run-to-run reproducibility over 100 independent runs per hypothesis (open-weight, default settings). cardiac_01 is a straightforward L2 group comparison (DCM vs. NOR LVEF); glioma_25 is a challenging L5 multivariate Cox regression with covariate adjustmentâthe hardest testable tier. Majority %: fraction of runs returning the most common verdict. p-value and effect size statistics are computed over successful runs only. cardiac_01 (L2) glioma_25 (L5) Ground truth label Supported Supported Runs (success / total) 95/100 68/100 Verdict accuracy 100.0% 37.7% Evidence-label accuracy 95.0% 20.0% Majority verdict (%) YES (95%) YES (26%) p-value (median [IQR]) 7.3Ă10â297.3Ă10^-29 [7.3Ă10â297.3Ă10^-29, 3.0Ă10â113.0Ă10^-11] 0.050 [0.034, 0.70] Effect size (median [IQR]) â6.15-6.15 [â6.26-6.26, â1.00-1.00] 0.990.99 [0.710.71, 1.241.24] Runtime (median [IQR], s) 215 [192, 244] 418 [306, 571] The L2 cardiac hypothesis exhibits near-perfect reproducibility: 100% verdict accuracy, 95% evidence-label accuracy (5 runs assigned Underpowered instead of Supported due to minor power estimation differences), and extremely tight p-value concentration (<10â28<10^-28). The L5 glioma hypothesis is substantially harder: only 68 of 100 runs complete successfully, and verdict accuracy drops to 37.7%. The difficulty stems from the multivariate Cox regression, which requires correct covariate specification, Grade IV cohort filtering, and proper survival data encodingâeach introducing independent failure modes absent from simpler analyses. Despite this, the majority verdict remains correct (YES), and median runtime roughly doubles from 3.6 to 7.0 minutes. These results quantify the reproducibilityâcomplexity trade-off inherent in agentic scientific workflows: simple analyses are highly reliable, while complex multivariate designs remain a frontier challenge. B.4 Phase-Level Reliability Table 11 reports phase-level pass rates for Veritas under both model families, combining ACDC and UCSF-PDGM results. A phase âpassesâ when its output satisfies the programmatic validation checks (schema correctness, no synthetic data, valid sample sizes). Table 11: Phase-level pass rates and dominant failure modes for Veritas (combined ACDC + UCSF-PDGM). Pool: number of runs entering that phase; runs that fail in an earlier phase do not enter subsequent phases. Phase 2B (code generation and execution) is the primary reliability bottleneck. Open-weight Frontier (GPT-5.2) Phase Pool Pass% Top failure Pool Pass% Top failure Phase 1 (Planning) 640 97.3 Schema (2.5%) 640 98.0 Schema (2.0%) Phase 2A (Segmentation) 604 97.8 Obs. mismatch (1.0%) 587 100.0 â Phase 2B (Statistics) 591 85.1 Synthetic (8.8%), Schema (3.9%) 587 95.7 Schema (3.7%) Phase 2B is the primary bottleneck: open-weight models fail 14.9% of the time, predominantly due to synthetic data generation (8.8%, fabricating values instead of using the SAT API) and results schema violations (3.9%, incomplete JSON output). Frontier models reduce Phase 2B failures to 4.3%, primarily schema issues, and achieve perfect Phase 2A pass rates. Phase 1 is highly reliable across both model families (>>97%), confirming that the planning phaseâs structured output format is well-handled. B.5 Coding Trial Efficiency Table 12 reports the number of code execution attempts per phase. Each coding round permits up to 8 executions; phases may span multiple rounds, so the effective per-phase limit is higher (observed max: 19). Table 12: Quantification of code execution attempts per phase (mean / max across all runs). Phase 2A writes the segmentation request; Phase 2B performs statistical analysis. Phase 2A Phase 2B Total Configuration Mean Max Mean Max Mean Max Local (ACDC) 1.4 13 1.7 8 2.9 13 Local (UCSF) 1.4 10 3.7 8 4.7 13 Frontier (ACDC) 1.1 4 4.9 16 5.8 17 Frontier (UCSF) 1.4 6 6.4 17 7.3 19 Phase 2A typically requires 1â2 attempts regardless of model family, reflecting the constrained nature of the segmentation request task. Phase 2B shows greater variation: open-weight ACDC averages 1.6 attempts (simple cardiac metrics), while frontier UCSF-PDGM averages 6.4 (complex survival analyses with covariate adjustment). Notably, frontier models use more iterations than open-weight on averageânot due to higher failure rates, but because the iterative code execution environment enables them to refine analyses across rounds (e.g., adding diagnostic plots, checking distributional assumptions, refining covariate encoding), whereas smaller models tend to terminate after fewer, less sophisticated attemptsâpartly due to lower model capacity and partly because their shorter context windows limit how many iterative rounds can be sustained before truncation. B.6 Accuracy by Control Type Table 13 stratifies accuracy by hypothesis control type, combining both datasets. Control types reflect the intended role of each hypothesis in the benchmark: positive controls should yield YES/Supported, negative controls should yield NO/Refuted, underpowered controls are designed to have insufficient sample size, untestable controls require data not present in the dataset, and nonsense/no-effect controls test obviously false or null hypotheses. Table 13: Evidence-label (Ev) and verdict (Vd) accuracy (%) by hypothesis control type for Veritas (combined ACDC + UCSF-PDGM). n: number of valid evaluation instances. Open-weight Frontier Control type Ev n Vd n Ev n Vd n Positive (37 hyps) 58.4 370 72.0 300 64.6 370 84.0 349 Negative (12 hyps) 76.7 120 80.2 106 77.5 120 58.9 112 Underpowered (6 hyps) 65.0 60 54.0 50 81.7 60 98.1 54 Untestable (5 hyps) 74.0 50 50.0 16 100.0 50 55.6 9 No-effect (2 hyps) 100.0 20 75.0 20 70.0 20 89.5 19 Nonsense (2 hyps) 35.0 20 40.0 10 90.0 20 100.0 18 Key observations: (1) Positive controls are the hardest for evidence accuracy (58â65%), because generating correct statistical outputs with proper effect sizes and confidence intervals requires successful end-to-end code execution. (2) Negative controls exhibit a counterintuitive frontierâopen-weight reversal for verdict accuracy (59% vs 80%): frontier models more frequently produce hedging Inconclusive verdicts for reversed-direction hypotheses, whereas open-weight models issue decisive No verdicts more often. (3) Untestable controls achieve 74â100% evidence detection, with frontier models achieving perfect L0 identification (100%) and open-weight models correctly identifying most cases (74%), validating the feasibility-checking mechanism. (4) Underpowered controls are reliably identified by the frontier model (82% evidence, 98% verdict), confirming that the Underpowered label captures genuine statistical ambiguity that a binary scheme would miss. B.7 End-to-End vs. Completed-Only Accuracy Table 14 contrasts two accuracy measures: completed-only accuracy (excluding runs with execution failures) and end-to-end (E2E) accuracy (treating failures as incorrect). The gap quantifies the âexecution taxââhow much accuracy drops when pipeline reliability is factored in. Table 14: Non-L0 verdict accuracy (%): completed runs only vs. end-to-end (E2E, counting execution failures as incorrect). Î : accuracy penalty from incomplete runs, quantifying the âexecution taxâ imposed by pipeline reliability on effective accuracy. Open-weight Frontier Dataset Vd Vde2e_e2e Î Vd Vde2e_e2e Î ACDC 87.9 84.8 â-3.1 78.9 77.2 â-1.7 UCSF-PDGM 49.0 33.7 â-15.3 83.2 74.0 â-9.2 Combined 71.4 58.8 â-12.6 81.0 75.6 â-5.4 The execution tax is most severe for open-weight models on UCSF-PDGM (â-15.3p), where complex survival analyses frequently fail to produce valid outputs. Frontier models substantially reduce the gap (5.4p combined vs 12.6p), indicating that improved code generation directly translates to higher effective accuracy. B.8 Underpowered Label Sensitivity To assess the importance of the Underpowered evidence label, we re-score all runs under a binary scheme that collapses Underpowered ground-truth labels to Refuted (i.e., treating low-powered null results identically to adequately-powered null results). Under this binary scheme, combined evidence accuracy increases by +3.6p for open-weight and +1.7p for frontier. The modest improvement confirms that (1) the Underpowered category captures genuine ambiguity that would otherwise inflate the Refuted bucket, and (2) the accuracy cost of maintaining this distinction is small, while the scientific benefit (distinguishing âno evidence of effectâ from âinsufficient power to detect effectâ) is substantial. B.9 SESOI Sensitivity Table 15: Effect of SESOI profile on ground-truth evidence-label distribution across the 64-hypothesis benchmark. Stricter profiles demand more power, shifting borderline hypotheses from Refuted to Underpowered. Supported and Invalid counts are invariant. Flips: number of label changes relative to the standard profile used as default. SESOI Profile Supported Refuted Underpowered Invalid Flips Strict (d=0.2d=0.2, r=0.2r=0.2) 32 14 13 5 5 Standard (d=0.5d=0.5, r=0.3r=0.3) 32 19 8 5 â Loose (d=0.8d=0.8, r=0.4r=0.4) 32 23 4 5 4 The number of Supported labels is invariant across profiles (32 of 64), because support depends only on p<Îąp<Îą and correct direction, both independent of SESOI. Similarly, the 5 Invalid labels (L0 untestable) are fixed. All 9 label flips occur exclusively on the Refuted/Underpowered boundary: a stricter SESOI demands more power to declare a null result âadequately powered,â pushing borderline hypotheses from Refuted to Underpowered. No hypothesis changes to or from Supported under any profile, confirming that the evaluationâs sensitivity to SESOI choice is limited and well-understood. B.10 Per-Hypothesis Detailed Results Table LABEL:tab:per_hypothesis reports per-hypothesis results for both the frontier (GPT-5.2) and open-weight Veritas configurations. For each hypothesis, we show the majority-vote verdict and evidence label across 10 runs. Correct results (matching ground truth) are shown in bold. Hypothesis IDs are abbreviated for space; full descriptions are in Section E. Table 16: Per-hypothesis results for Veritas under frontier (GPT-5.2) and open-weight configurations. Majority-vote verdict (Vd) and evidence label (Ev) across 10 runs. T = hypothesis tier (L0âL5), C = control type (P = positive, N = negative, U = null, L = low-n, X = untestable), GT = ground-truth label. Y = Yes, N = No, I = Inconclusive, â = not completed. Correct results in bold. Frontier Open-weight Hypothesis T C GT Vd Ev Vd Ev ACDC â Cardiac MRI C27: LA volume HCM L0 X Inv â Inv N Inv C28: GLS lower DCM L0 X Inv â Inv N Inv C32: SBP higher DCM L0 X Inv â Inv N Inv C24: Heightâweight corr. L1 P Sup Y Sup Y Sup C25: ED frameâweight corr. L1 U Ref N Ref N Ref C26: Num framesâweight corr. L1 U Ref N Ref N Ref C01: DCM LVEF lower L2 P Sup Y Sup Y Sup C02: LV mass DCM L2 P Sup Y Sup Y Sup C03: DCM LVEF higher L2 N Ref I Ref N Ref C04: RV volume similar L2 N Ref N Ref N Ref C07: DCM LVEDV higher L2 P Sup Y Sup Y Sup C08: DCM LVESV higher L2 P Sup Y Sup Y Sup C09: HCM LV mass higher L2 P Sup Y Sup Y Sup C10: HCM LVEF >> DCM L2 P Sup Y Sup Y Sup C11: MINF LVEF lower L2 P Sup Y Sup Y Sup C12: RV RVEF lower L2 P Sup Y Sup Y Sup C13: HCM LV mass lower L2 N Ref N Ref N Ref C14: DCM LVEDV lower L2 N Ref N Ref N Ref C15: DCM LVESV lower L2 N Ref N Ref N Ref C16: RV RVEF higher L2 N Ref I Ref N Ref C19: HCM LVEDV << NOR L2 P Ref I Ref I Ref C20: HCM LVESV << NOR L2 P Sup I Ref Y Sup C30: RV RVEDV >> NOR L2 P Sup Y Sup Y Sup C31: RV RVEDV << NOR L2 N Ref I Ref N Ref C21: ESV/EDV ratio DCM L3 P Sup Y Sup Y Sup C22: HCM mass/vol >> DCM L3 P Sup Y Sup Y Sup C23: DCM mass/vol << NOR L3 P Sup Y Sup Y Sup C05: WeightâLVEF corr. L4 L Undp I Undp N Undp C06: HeightâLV volume L4 P Sup Y Sup Y Sup C17: NOR weightâLVEF L4 L Undp I Undp I Undp C18: DCM heightâLVEF L4 L Undp I Undp I Undp C29: LVEDVi (BSA-indexed) DCM L4 P Sup Y Sup Y Sup UCSF-PDGM â Glioma MRI G30: ADCâsurvival assoc. L0 X Inv â Inv I Inv G32: Frontal locationâIDH L0 X Inv I Inv I Inv G11: MGMT surv. longer L1 P Sup Y Sup I Ref G13: GTR surv. longer L1 P Sup Y Sup N Ref G14: IDH surv. longer (G4) L1 L Sup Y Sup Y Inv G16: GTR surv. shorter L1 N Ref N Ref N Ref G28: Men surv. longer (G4) L1 U Ref N Ref I Ref G29: Men older (G4) L1 U Ref N Ref I Inv G01: Volume by grade L2 P Sup Y Sup Y Sup G05: G2 more enhancement L2 N Ref N Ref N Ref G07: G4 more enhancement L2 P Sup Y Sup Y Inv G08: G4 more necrosis L2 P Sup Y Sup Y Sup G09: G4 more edema L2 P Sup Y Sup Y Sup G19: Enh. fraction G4 higher L3 P Sup Y Sup I Inv G20: Nec. fraction G4 higher L3 P Sup N Ref N Ref G21: Edema fraction G4 higher L3 P Ref N Ref N Inv G02: IDHâenhancement L4 P Sup N Inv I Inv G03: Enhancementâsurvival L4 P Sup Y Inv N Inv G04: IDH-mut more necrosis L4 N Ref I Inv I Undp G06: Ageâvolume corr. L4 N Ref N Ref N Ref G10: IDH-wt more necrosis L4 P Sup N Inv N Ref G12: CET vol. surv. shorter L4 P Ref N Undp N Inv G15: Enhancement surv. pos. L4 N Ref N Inv N Inv G17: G2 IDH-wt enhancement L4 L Undp I Undp I Undp G22: Enh. frac. IDH-wt L4 P Sup I Inv Y Sup G24: Nec. frac. surv. neg. G4 L4 P Ref N Inv I Inv G31: 1p19q enh. fraction L4 L Undp I Undp I Undp G18: CET surv. adj. EOR L5 P Sup Y Inv I Inv G23: Surv. IDH adj. multi L5 P Sup Y Inv I Inv G25: MGMT surv. adj. age L5 P Sup Y Inv Y Inv G26: Age surv. adj. MGMT L5 P Sup Y Inv Y Sup G27: IDH enh. adj. grade L5 P Sup Y Sup I Inv B.11 Baseline Summary Per-method aggregate metrics (evidence-label accuracy, verdict accuracy, completion rate, and L0 feasibility) are reported in Table 8 above. All four coding baselines (SMbâSMe) use the same API documentation, and all five single-model baselines (SMaâSMe) use the same hypothesis bank as Veritas, ensuring a fair comparison surface. SMa (direct reasoning) requires no code execution; SMb (pre-computed features), SMc (one-shot code) and SMd (agentic loop with up to 3 iterations) generate and execute analysis code; SMe (full pipeline) receives the same four-phase agenda structure as Veritas but executes all phases within a single model context. Per-hypothesis baseline breakdowns are omitted for space; the evaluation scripts and all raw run directories are included in the supplementary code release for full reproducibility. B.12 Comparison to agentic systems for scientific discovery In this section we are qualitatively comparing our work to agentic systems developed for scientific discovery. These works operate in a similar research space but are distinctly different in their objectives, implementation and autonomy. Most notably, the Virtual Lab [55] is a multi-agent framework in which a PI and domain expert agents conduct structured discussion meetings and produce analysis code. However, a human operator must manually extract, execute, and debug all code between phases, relaying execution results back to the agents. Table 17 summarizes the key architectural differences between Veritas and the Virtual Lab. Table 17: Architectural comparison: Veritas vs. Virtual Lab. Capability Veritas (Ours) Virtual Lab [55] Code execution Sandboxed, closed-loop Human extracts & runs Data access Direct API queries (SAT) Textual descriptions only Critic Phase-aware (schema, logic, direction) Generic Scientific Critic Tool integration MCP servers (segmentation, datasets) PubMed search only Models 8â30B local (Ollama) GPT-4o (API) Output validation Programmatic (JSON schema, CI checks) Free-text summaries Human intervention None Per-phase code execution Appendix C Failure Case Analysis We analyze two distinct failure modes using a single hypothesisââIDH-wildtype tumors have significantly larger enhancing tumor volumes than IDH-mutant tumorsâ (glioma_02, L4)âwhich exhibits one of the strongest ground-truth effects in the benchmark (Mann-Whitney U p=9.67Ă10â29p=9.67Ă 10^-29, rank-biserial r=0.71r=0.71). Despite this large effect, the frontier Veritas configuration produced incorrect results in all five runs, through two qualitatively different mechanisms: one execution-level and one reasoning-level. C.1 Execution Failure: Schema Validation Mismatch In run 000, the coding agent generated statistically valid analysis code but populated the variables_tested.predictors field of statistical_results.json with ["idh_status", "group"]. The Phase 1 plan specified only idh_status as the predictor (with group as a covariate). The schema validator caught this discrepancy (P2B_RESULTS_SCHEMA_INVALID) and the pipeline terminated after 6 coding trials without reaching Phase 3. Auditability value. The full code and all 6 correction attempts are preserved in the run directory, enabling a researcher to identify the exact JSON key mismatch and either fix the schema or re-run with a relaxed validator. C.2 Reasoning Failure: Mediator Adjustment (Table 2 Fallacy) In runs 001â004, the pipeline completed all phases but reached the wrong verdict (NO/Refuted). During Phase 1, the agents correctly identified tumor grade as a potential confounder and chose OLS regression on log-transformed enhancing volume with grade adjustment: logâĄ(1+ET_ml)âźidh_status+grade (1+ET\_ml) \_status+grade After adjustment, the IDH coefficient was β=â0.137β=-0.137 (p=0.593p=0.593, 95% CI: [â0.643,0.368][-0.643,0.368]), leading Phase 3 to conclude âno significant association.â Root cause. IDH status and tumor grade lie on the same causal pathway: IDH-wildtype tumors are predominantly Grade IV, and Grade IV tumors exhibit the most enhancement. Adjusting for grade blocks the biological mechanism through which IDH status produces enhancement differencesâa classic mediator-adjustment error (âTable 2 fallacyâ [63]). The unadjusted analysis shows one of the strongest effects in the dataset (r=0.71r=0.71), but the grade-adjusted model sees no residual signal. Auditability value. Because Veritas saves the complete Phase 1 plan, generated code, and statistical output, a domain expert reviewing the artifacts can immediately identify the covariate adjustment choice as the root causeâand re-run the analysis without grade adjustment to recover the correct result. A single-model baseline producing the same wrong answer would offer no such diagnostic trail. Implication. This case exposes a fundamental limitation of current LLM-based statistical agents: they apply generic epidemiological heuristics (âadjust for potential confoundersâ) without causal reasoning about whether a covariate is a confounder, mediator, or collider on the causal pathway between exposure and outcome. Integrating causal graph reasoning [47] into the planning phaseâfor instance, by prompting agents to construct and evaluate directed acyclic graphs before selecting covariatesâis a promising direction for future work. Appendix D Framework Details This section provides detailed specifications of the framework components summarized in the main paper, including agent roles, phase-level design decisions, the imaging analysis API contract, and evaluation criteria. All hyperparameters and resource limits are listed in Table 18. Table 18: Framework hyperparameters and limits. Parameter Value Description Max discussion rounds 3 Phase 1 and Phase 3 Max code executions per round 8 Phases 2A and 2B (2 rounds per pass) Phase timeout 20 min Wall-clock per phase Temperature (discussion) 0.2 Phases 1 and 3 Temperature (coding) 0.1 Phases 2A and 2B Significance level (Îą) 0.05 All statistical tests Power threshold (Ď0 _0) 0.80 Underpowered classification Standard SESOI (d0d_0) 0.5 Group differences Standard SESOI (r0r_0) 0.3 Correlations Loose SESOI (d0d_0) 0.8 Large-effect hypotheses Strict SESOI (d0d_0) 0.2 Small-effect hypotheses D.1 Agent Roles We instantiate three specialized agents with distinct expertise: ⢠Principal Investigator (PI): Oversees scientific validity, adjudicates feasibility and confounds, and validates conclusions. Serves as team lead in discussion phases. ⢠Medical Imaging Specialist: Specifies segmentation targets, validates imaging protocols, queries cohort sizes, and interprets structural measurements. ⢠Statistician: Selects appropriate statistical tests based on data characteristics, computes effect sizes, performs power analysis, and assesses statistical validity. Each agent is implemented as an LLM with role-specific system prompts encoding domain expertise. A separate Critic agent is enabled by default in execution phases (2A/2B), with optional ablation settings to also enable Critic participation in planning (Phase 1) and interpretation (Phase 3). D.2 Phase Details Phase 1: Feasibility Checks. During planning, the agent team must emit a structured feasibility block (status, invalid_subtype, missing_requirements). A deterministic validator then checks plan contract consistency against dataset metadata (groups, observations, metadata fields, and required plan keys). If the team marks a hypothesis as UNTESTABLE, the workflow terminates early with Invalid. This keeps feasibility decisions explicit and auditable while preserving agent flexibility for testable analyses. Phase 1: A Priori Power Analysis. The Imaging Specialist queries actual cohort sizes using list_dataset_patients(). The Statistician computes a priori power using a fixed planning SESOI (d0=0.5d_0=0.5 for group differences, r0=0.3r_0=0.3 for correlations). This planning power informs test selection and sample-adequacy assessment, but it is not the benchmarkâs final evidence-label power whenever a hypothesis carries a non-standard SESOI profile. Phase 2B: Test Selection. The framework supports parametric (t-test, Pearson correlation, OLS regression), nonparametric (Mann-Whitney U, Spearman correlation), and survival analysis (log-rank test, Cox proportional hazards) tests. Agents select tests based on distributional assumptions and outcome type. For nonparametric group differences, rank-biserial correlation serves as the effect size; for survival analyses, hazard ratios with confidence intervals. Phase 2B: Code Execution. Code execution occurs in an isolated sandboxed environment with timeout constraints. Each phase permits up to 16 code execution attempts with a 20-minute wall-clock timeout. On each failed attempt, the Critic feeds error output back to the coding agent for revision. The number of attempts is logged as a measure of agent autonomy efficiency. Phase 3: Direction Verification. For group difference tests, direction is verified from observed group meansânot from the effect size sign, which is convention-dependent for nonparametric tests. For correlations and survival analyses, the effect size sign is used directly. D.3 Imaging Analysis API All analysis phases operate through a constrained Imaging Analysis API rather than direct filesystem access, ensuring data provenance and enabling automatic detection of off-contract data loading. The API provides cohort discovery, metadata lookup, segmentation retrieval, and quantitative measurement utilities. Agents receive tool documentation, dataset-registry metadata (groups, observations, patient fields), and structure catalogs. The sat. namespace used in code examples is an implementation detail. Core API calls are: ⢠sat.list_patients(results_db_path) for cohort enumeration from segmentation outputs ⢠sat.get_patient_metadata(patient_id) for group labels and clinical covariates ⢠sat.get_observation_identifiers(patient_id) for timepoint/modality mapping ⢠sat.load_structure_mask(...) for structure-specific masks ⢠sat.calculate_volume(mask, spacing) for geometry-aware volume extraction This contract gives agents sufficient autonomy to engineer derived metrics (e.g., ejection fraction from end-diastolic and end-systolic volumes) while preserving provenance and enabling automatic detection of synthetic-data substitution. D.4 Auditability Criteria We define six auditability criteria that each ground-truth record and pipeline output must satisfy: 1. Test-family correctness: the statistical test matches the hypothesis type (group difference â Mann-Whitney U; correlation â Spearman; survival â log-rank/Cox PH) 2. Cohort restriction correctness: the correct subpopulation is selected before group splitting (e.g., Grade IV only for GBM-specific hypotheses) 3. Censoring correctness: survival analyses correctly encode time and event variables 4. Variable correctness: the metric, group definitions, and covariates match the hypothesis specification 5. Reporting completeness: full statistics are reported (sample sizes per group, effect size with CI, p-value, power at SESOI) 6. Planâexecution consistency: the executed analysis matches the Phase 1 plan D.5 Validity Checks The evaluation pipeline performs automated validity checks: 1. Feasibility validation: required fields/structures/observations exist 2. Numeric validation: pâ[0,1]pâ[0,1], effect sizes finite 3. CI consistency: bounds ordered, effect size contained 4. Sign consistency: effect size sign matches mean difference direction 5. Logic consistency: verdict aligns with statistical evidence 6. Confound handling: mixed-group correlations require adjustment or stratification Violations trigger warnings or Invalid labels, ensuring only well-formed results contribute to evaluation metrics. D.6 Evaluation Metric Definitions Table 5 reports six diagnostic metrics organized into three categories. Below we define each metric and the additional diagnostics reported in this supplement. Conclusion quality. ⢠Overclaim rate: fraction of YES verdicts where the evidence label is not Supportedâthe model claims support that its own statistics do not provide. ⢠False-refutation rate: fraction of Refuted evidence labels disagreeing with ground truth. Analysis integrity. These metrics detect âdishonestâ analyses at increasing severity: claiming significance without valid evidence (output-level), fabricating data (code-level), or hard-coding the decision threshold (code-level). ⢠Hallucinated significance: YES verdict with pâĽ0.05p⼠0.05 or missing p-value (output-level). ⢠Synthetic-data violation: code generating mock, random, or simulated data instead of using real measurements via the SAT API (code-level). Benign stochastic utilities (e.g., plotting jitter, bootstrap resampling) are tracked separately as non-blocking warnings. ⢠Literal p-value assignment: significance threshold (p=0.05p=0.05) hard-coded in the generated analysis code, detected via static regex analysis of Phase 2B Python files (code-level). Auditability. ⢠Evidence grounding rate (Verif.): fraction of all runs that report the four core statistical outputs required for independent verification: statistical test type, per-group sample sizes, effect size, and p-value. This metric captures both execution reliability and output completeness in a single number. Additional diagnostics. ⢠L0 feasibility accuracy: fraction of untestable (L0) hypotheses correctly labeled Invalid, with subset accuracy excluding early Phase 1 stops. ⢠Execution reliability: early feasibility stops (Phase 1 Untestable) reported separately from true parse/runtime failures. ⢠Phase 2B alignment: fraction of Phase 3 verdicts consistent with Phase 2B statistical outputs. ⢠Coding trial count: mean and maximum code execution attempts per hypothesis. ⢠Sample-size consistency (n mismatch): checks whether reported total n equals the sum of per-group sample sizes. D.7 Implementation Details Models. We evaluate two model deployments. Open-weight deployment (local, via Ollama [41]): gpt-oss:20b for PI and imaging roles, qwen3:8b for statistician discussion and critic, and qwen3-coder:30b for coding. Frontier deployment (remote, via OpenRouter [45]): openai/gpt-5.2 for PI/imaging/coding and openai/gpt-5-mini for statistician discussion and critic. All other workflow settings are held fixed. Segmentation. The SAT backend [71] provides neural segmentation across domains: cardiac structures (LV, RV, myocardium) for ACDC, and brain tumor subregions (whole tumor, necrotic core, edema, enhancing tumor) for UCSF-PDGM following BraTS label conventions. Statistical Libraries. Code execution phases have access to numpy, pandas, scipy, statsmodels, and lifelines [15] for survival analysis. Agent Orchestration. The workflow is implemented using LangGraph [33], a graph-based state machine framework that enables conditional routing and state persistence. D.8 Power Analysis Details Power is computed in two independent contexts to avoid the post-hoc power fallacy [20, 22]: Agent-side (Phase 1). During planning, the Statistician computes a priori power from queried sample sizes and a fixed planning SESOI. This informs study feasibility and test selection but is not used for evidence labeling. Evaluator-side (evidence labeling). After Phase 2B completes, the evaluation pipeline independently recomputes power from realized sample sizes: Ď=P(|T|>tÎą/2âŁÎ´=δ0,n1,n2)Ď=P (|T|>t_Îą/2 δ= _0,n_1,n_2 ) (1) For evaluator-side evidence labeling, SESOI is hypothesis-specific. Each hypothesis stores a sesoi_profile in the benchmark JSON, and the evaluator resolves this to a profile-specific target by test family: for group differences, Cohenâs dâ0.2,0.5,0.8dâ\0.2,0.5,0.8\ for strict/standard/loose; for correlations, râ0.2,0.3,0.4râ\0.2,0.3,0.4\; for regression, standardized effect sizes â0.2,0.3,0.4â\0.2,0.3,0.4\; and for survival analyses, hazard-ratio departures from the null corresponding to 1.2, 1.5, 2.0. Evaluator-computed power is used solely for evidence labeling and is never revealed to Phase 3 agents. Power calculation formulas. For two-sample group differences, power is computed using the noncentral t-distribution: Ď=Pâ(|Tn1+n2â2,Îť|>tÎą/2,n1+n2â2),Îť=d0ân1ân2n1+n2Ď=P (|T_n_1+n_2-2,Îť|>t_Îą/2,n_1+n_2-2 ), Îť=d_0 n_1n_2n_1+n_2 (2) where d0d_0 is the SESOI (Cohenâs d) and Îť is the noncentrality parameter. For correlations, Fisher-z transform power: zr=arctanhâ(r0),se=1nâ3,Ď=ÎŚâ(zrseâzÎą/2)+ÎŚâ(âzrseâzÎą/2)z_r=arctanh(r_0), = 1 n-3, Ď= ( z_rse-z_Îą/2 )+ (- z_rse-z_Îą/2 ) (3) For survival analyses (log-rank and Cox PH), power is approximated via the normal method with the number of events: ĎâPâ(|Z|>zÎą/2âŁÎť=logâĄ(HR0)âdâ p1âp21)Ďâ P (|Z|>z_Îą/2 Îť= (HR_0) d¡ p_1p_21 ) (4) where d is the total number of events and p1,p2p_1,p_2 are group proportions. For OLS regression, F-test power via the noncentral F-distribution: Ď=Pâ(F>FÎą,ptest,nâkâ1âŁÎť=nâ f2)Ď=P (F>F_Îą,p_test,n-k-1 Îť=n¡ f^2 ) (5) where f2=0.15f^2=0.15 (medium effect per Cohen) and ptestp_test is the number of tested predictors. D.9 Compute Infrastructure Local open-weight experiments used single NVIDIA L40 or A40 GPUs. Appendix E Complete Hypothesis Bank This section provides the complete 64-hypothesis tiered benchmark with per-hypothesis specifications, ground-truth labels, and clinical rationale. Table 19 summarizes the per-dataset tier distribution. Table 19: Evaluation datasets and tiered hypothesis coverage. Hypotheses per tier Dataset Modality Subj. Total L0 L1 L2 L3 L4 L5 Analysis types ACDC [6] Cardiac cine MRI 150 32 3 3 18 3 5 0 GD, Corr UCSF-PDGM [8] Brain MRI (multi-seq.) 501 32 2 6 5 3 11 5 GD, Corr, Surv, Reg Combined 651 64 5 9 23 6 16 5 GD = Group Difference; Corr = Correlation; Surv = Survival analysis; Reg = Regression with covariates. ACDC has no survival endpoint (no L5). UCSF-PDGM covers all six tiers including Cox PH and OLS regression. E.1 Hypothesis Bank Design Principles Each released hypothesis record contains both a sesoi_profile and a dataset-derived ground-truth block with power_at_sesoi. Throughout the per-hypothesis descriptions below, quoted power values refer to this evaluator-side, hypothesis-specific SESOI unless explicitly marked as Phase 1 planning power. The bank is designed to probe five distinct capabilities: 1. Positive controls: well-established clinical findings that should be Supported (37 hypotheses) 2. Negative controls: reversed-direction hypotheses that should be Refuted (12 hypotheses) 3. No-effect / nonsense controls: hypotheses with no biological basis, testing false-positive propensity (4 hypotheses) 4. Underpowered controls: plausible hypotheses in small subgroups where power <0.80<0.80 (6 hypotheses) 5. Untestable controls: hypotheses requiring unavailable data, testing feasibility detection (5 hypotheses) E.2 ACDC Cardiac MRI Hypotheses (32) E.2.1 L0: Untestable (3) cardiac_27: HCM have larger LA volume than NOR. Invalid (untestable). Left atrium not in ACDC label set (only LV, RV, myocardium). cardiac_28: DCM have lower GLS than NOR. Invalid (untestable). Global longitudinal strain not derivable from static segmentation. cardiac_32: DCM have higher systolic BP than NOR. Invalid (untestable). Blood pressure not in ACDC metadata. E.2.2 L1: Metadata-Only Correlations (3) cardiac_24: Height positively correlates with weight. Spearman correlation, all 150 subjects. Positive control (well-known anthropometric relationship). SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.96. cardiac_25: ED frame index correlates with weight. Spearman correlation, all 150 subjects. No-effect control (no physiological link between cardiac timing and weight). SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.96. cardiac_26: Cine frame count correlates with weight. Spearman correlation, all 150 subjects. No-effect control (frame count is protocol-dependent). SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.96. E.2.3 L2: Single Imaging Metric (18) All L2 hypotheses use Mann-Whitney U tests with 30 vs 30 subjects. Each carries SESOI profile loose (d0=0.8d_0=0.8), yielding power 0.86. Table 20: ACDC L2 hypotheses: single imaging metric group comparisons. ID Hypothesis Groups Control Label cardiac_01 DCM have significantly lower LVEF than NOR DCM vs NOR Positive Supported cardiac_02 DCM have significantly higher LV myocardial mass than NOR DCM vs NOR Positive Supported cardiac_03 DCM have significantly higher LVEF than NOR DCM vs NOR Negative Refuted cardiac_04 NOR have significantly larger RVEDV than DCM NOR vs DCM Negative Refuted cardiac_07 DCM have significantly larger LVEDV than NOR DCM vs NOR Positive Supported cardiac_08 DCM have significantly larger LVESV than NOR DCM vs NOR Positive Supported cardiac_09 HCM have significantly higher LV myocardial mass than NOR HCM vs NOR Positive Supported cardiac_10 HCM have significantly higher LVEF than DCM HCM vs DCM Positive Supported cardiac_11 MINF have significantly lower LVEF than NOR MINF vs NOR Positive Supported cardiac_12 RV patients have significantly lower RVEF than NOR RV vs NOR Positive Supported cardiac_13 HCM have significantly lower LV myocardial mass than NOR HCM vs NOR Negative Refuted cardiac_14 DCM have significantly smaller LVEDV than NOR DCM vs NOR Negative Refuted cardiac_15 DCM have significantly smaller LVESV than NOR DCM vs NOR Negative Refuted cardiac_16 RV patients have significantly higher RVEF than NOR RV vs NOR Negative Refuted cardiac_19 HCM have significantly smaller LVEDV than NOR HCM vs NOR Positive Refuted cardiac_20 HCM have significantly smaller LVESV than NOR HCM vs NOR Positive Supported cardiac_30 RV patients have significantly larger RVEDV than NOR RV vs NOR Positive Supported cardiac_31 RV patients have significantly smaller RVEDV than NOR RV vs NOR Negative Refuted E.2.4 L3: Engineered Features (3) cardiac_21: DCM have higher LVESV/LVEDV ratio than NOR. Mann-Whitney U, 30 vs 30. Positive control. Inverse of EF; reduced systolic function elevates ratio. SESOI profile: loose (d0=0.8d_0=0.8). Power: 0.86. cardiac_22: HCM have higher mass-to-volume ratio than DCM. Mann-Whitney U, 30 vs 30. Positive control. HCM: thick walls + small cavity; DCM: dilated cavity. SESOI profile: loose (d0=0.8d_0=0.8). Power: 0.86. cardiac_23: DCM have lower mass-to-volume ratio than NOR. Mann-Whitney U, 30 vs 30. Positive control. DCM dilation increases LVEDV more than mass. SESOI profile: loose (d0=0.8d_0=0.8). Power: 0.86. E.2.5 L4: Mixed Metadata + Imaging (5) cardiac_05: Weight correlates with LVEF in DCM+NOR. Spearman, n=60n=60. Underpowered control (weak/confounded relationship). SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.65. cardiac_06: Height correlates with LVEDV. Spearman, n=150n=150. Positive control (body-size scaling). SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.96. cardiac_17: In NOR: weight correlates with LVEF. Spearman, n=30n=30. Underpowered control. SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.36. cardiac_18: In DCM: height correlates with LVEF. Spearman, n=30n=30. Underpowered control. SESOI profile: standard (r0=0.3r_0=0.3). Power: 0.36. cardiac_29: DCM have higher LVEDV/BSA than NOR. Mann-Whitney U, 30 vs 30. Positive control. BSA = hcmĂwkg/3600 h_cmĂ w_kg/3600. SESOI profile: loose (d0=0.8d_0=0.8). Power: 0.86. E.3 UCSF-PDGM Glioma MRI Hypotheses (32) E.3.1 L0: Untestable (2) glioma_30: Mean tumor ADC associated with survival. Invalid. ADC/DWI modality not available through the Imaging Analysis API. glioma_32: IDH-mutant gliomas more often frontal. Invalid. No tumor location or atlas features available. E.3.2 L1: Metadata-Only (6) glioma_29: In GBM, men are older than women. Mann-Whitney U, M (239) vs F (157) in Grade IV (n=396n=396). Nonsense control (no biological basis for sexâ shift in GBM). SESOI profile: standard (d0=0.5d_0=0.5). Power: >>0.99. glioma_11: MGMT methylated GBM survive longer. Log-rank + Cox HR, methylated (274) vs unmethylated (105), Grade IV (n=395n=395, 229 events). Positive control. SESOI profile: loose (HR-scale target 2.0). Power: >>0.99. glioma_13: GTR patients survive longer than STR. Log-rank + Cox HR, GTR (227) vs STR (127), Grade IV. Positive control. SESOI profile: loose (HR-scale target 2.0). Power: >>0.99. glioma_14: IDH-mutant Grade IV survive longer. Log-rank + Cox HR, mutant (28) vs wildtype (368), Grade IV. Underpowered control (few IDH-mutant in Grade IV). SESOI profile: standard (HR-scale target 1.5). Power: 0.72. glioma_16: GTR patients have shorter survival. Log-rank (reuses glioma_13). Negative control (reversed direction). SESOI profile: loose (HR-scale target 2.0). Power: >>0.99. glioma_28: In GBM, men survive longer than women. Log-rank + Cox HR, M (239) vs F (157), Grade IV. Nonsense control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. E.3.3 L2: Imaging-Only (5) All L2 hypotheses use Mann-Whitney U tests, Grade IV (n=396n=396) vs Grade I (n=56n=56). Each carries SESOI profile loose (d0=0.8d_0=0.8), with power >>0.99. glioma_01: Grade IV have larger whole tumor volume than Grade I. Positive control. Well-established: higher-grade gliomas grow more aggressively. glioma_05: Grade I have larger enhancing tumor volume than Grade IV. Negative control (reversed direction). Enhancement is a hallmark of high-grade tumors. glioma_07: Grade IV have larger enhancing tumor volume than Grade I. Positive control. Contrast enhancement reflects blood-brain barrier disruption in GBM. glioma_08: Grade IV have larger necrotic core volume than Grade I. Positive control. Necrosis is a defining feature of Grade IV (GBM). glioma_09: Grade IV have larger edema volume than Grade I. Positive control. Higher-grade tumors cause more peritumoral edema. E.3.4 L3: Engineered Features (3) glioma_19/20/21: Grade IV have higher enhancing / necrotic / edema fraction. Mann-Whitney U, Grade IV (396) vs Grade I (56). Positive controls. Each uses SESOI profile standard (d0=0.5d_0=0.5). Power: 0.94. E.3.5 L4: Mixed (11) glioma_02: IDH-wildtype tumors have larger enhancing tumor volumes than IDH-mutant. Mann-Whitney U, all grades (n=495n=495), grouped by IDH status. Positive control. SESOI profile: loose (d0=0.8d_0=0.8). Power: >>0.99. glioma_04: IDH-mutant tumors have more necrotic core than IDH-wildtype. Mann-Whitney U, all grades (n=495n=495). Negative control (reversed direction). SESOI profile: loose (d0=0.8d_0=0.8). Power: >>0.99. glioma_06: Age correlates with whole tumor volume in GBM. Spearman, Grade IV (n=396n=396). Negative control (no expected relationship). SESOI profile: standard (r0=0.3r_0=0.3). Power: >>0.99. glioma_10: IDH-wildtype tumors have larger necrotic core than IDH-mutant. Mann-Whitney U, all grades (n=495n=495). Positive control. SESOI profile: loose (d0=0.8d_0=0.8). Power: >>0.99. glioma_17: In Grade I, IDH-wildtype have more enhancement than IDH-mutant. Mann-Whitney U, Grade I only (n=56n=56; IDH-wt n=10n=10, IDH-mut n=46n=46). Underpowered control (very small IDH-wt subgroup). SESOI profile: loose (d0=0.8d_0=0.8). Power: 0.61. glioma_22: IDH-wildtype tumors have higher enhancing fraction than IDH-mutant. Mann-Whitney U, all grades (n=495n=495). Positive control. SESOI profile: standard (d0=0.5d_0=0.5). Power: 0.99. glioma_31: In lower-grade gliomas, 1p/19q codeleted tumors have lower enhancing fraction. Mann-Whitney U, Grade I/I (n=99n=99; codeleted n=13n=13, intact n=86n=86). Underpowered control. SESOI profile: standard (d0=0.5d_0=0.5). Power: 0.38. glioma_03: Enhancing tumor volume negatively correlates with survival in GBM. Cox PH / log-rank, Grade IV (n=395n=395, 229 events). Imaging predictor: ET_ml. Positive control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. glioma_15: Enhancing tumor volume positively correlates with survival in GBM. Cox PH / log-rank, Grade IV (n=395n=395). Negative control (reversed direction of glioma_03). SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. glioma_24: Higher necrotic fraction correlates with worse survival in GBM. Cox PH, Grade IV (n=395n=395). Positive control (expected but borderline). SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. glioma_12: CET volume >>10 mL associated with shorter survival in GBM. Log-rank, Grade IV (n=395n=395). Derived binary grouping: ET >>10 mL vs ⤠10 mL. Positive control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. E.3.6 L5: Multivariate / Advanced (5) glioma_25: MGMT protective after adjusting for age + EoR. Cox PH multivariate, Grade IV (n=379n=379, 229 events). Primary: mgmt_binary (methylated=1). Covariates: age, eor_binary (GTR=1). Positive control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. glioma_26: Older age worsens survival adjusting MGMT + EoR. Cox PH multivariate, Grade IV. Primary: age (continuous). Covariates: mgmt_binary, eor_binary. Positive control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. glioma_27: IDH-wt increases enhancing fraction after grade adjustment. OLS regression, all grades (n=495n=495). Dependent: enhancing_fraction. Predictors: idh_wildtype + grade dummies (ref=Grade I). Positive control. SESOI profile: standard (f2f^2 target 0.3). Power: >>0.99. glioma_34: ET volume predicts survival adjusting for EoR. Cox PH multivariate, Grade IV. Primary: ET_ml (continuous). Covariate: eor_binary. Positive control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. glioma_35: IDH protective in full multivariate model. Cox PH, all grades (n=494n=494, 248 events). Primary: idh_binary (mutant=1). Covariates: age, grade_I, grade_IV, mgmt_binary, eor_binary, ET_ml. 7-covariate model. Positive control. SESOI profile: standard (HR-scale target 1.5). Power: >>0.99. Appendix F Ground-Truth Computation Pipeline F.1 Methodology Ground-truth evidence labels are computed independently of the agent pipeline, using the same imaging data and statistical conventions. For each hypothesis, we: 1. Load ground-truth segmentation masks (expert-annotated for ACDC; BraTS-convention masks for UCSF-PDGM) 2. Compute metrics using the same analysis-interface functions available to agents (calculate_volume, calculate_mass, calculate_ejection_fraction) 3. Execute the canonical statistical test (Mann-Whitney U for group differences, Spearman for correlations, log-rank/Cox PH for survival, OLS for regression) 4. Record complete statistics: sample sizes, effect size with 95% CI, p-value, group medians/means 5. Resolve the hypothesis-specific sesoi_profile and compute power at that SESOI 6. Assign the evidence label mechanically using the rules in Table 1 Power is computed per test family using the formulas in SectionËD.8. The implementation uses statsmodels power functions for group differences and regression, and analytic normal approximations for correlations and survival analyses. Appendix G Agent System Prompts All agents are defined as structured objects with four fields: title, expertise, goal, and role. The system prompt for each agent is rendered as: âYou are a title. Your expertise is in expertise. Your goal is to goal. Your role is to role.â Below we reproduce the complete role definitions; expertise and goal fields are shown inline. G.1 Principal Investigator Expertise: running a science research lab. Goal: perform research that maximizes scientific impact. Principal Investigator Role Prompt ⏠Lead a team of experts to solve important scientific problems. Listen to team member input, synthesize their recommendations into clear, actionable decisions. Make decisive choices when there are trade-offs. Follow any output format requirements specified. G.2 Medical Imaging Specialist (Discussion) Expertise: medical image analysis, image segmentation, and SAT foundation model usage. Goal: provide expertise on imaging analysis capabilities and segmentation approaches. Medical Imaging Specialist (Discussion) Role Prompt ⏠Provide imaging expertise for research planning. Recommend which anatomical structures to segment, which observations to capture, and assess segmentation feasibility and expected quality. Available tools: - list_dataset_patients(dataset, group=None, metadata_filters=None): Discover cohort sizes. Returns: patients: [patient_id, group], total_count - list_available_structures(category): Verify segmentable structures. Categories: cardiac, abdominal, vascular, spine, brain, urogenital, etc. - check_sat_status(): Verify SAT model availability Use tools to gather information needed for recommendations. G.3 Medical Imaging Specialist (Interpretation) Expertise: medical image analysis, segmentation quality assessment, and technical limitations. Goal: interpret segmentation results and assess their impact on study conclusions. Medical Imaging Specialist (Interpretation) Role Prompt ⏠Interpret imaging results from a technical perspective. NOTE: Segmentation quality metrics (Dice scores) are NOT available in the current framework since there is no ground truth - assume segmentations are adequate for analysis. Discuss potential technical limitations (partial volumes, motion artifacts, frame selection) that could affect interpretation, but do NOT request unavailable metrics or make their absence a reason for inconclusive verdict. Focus on whether technical factors would plausibly invalidate the statistical findings. G.4 ML Statistician (Discussion) Expertise: statistical experimental design, power analysis, hypothesis testing methodology. Goal: design rigorous statistical experiments and provide methodological guidance. ML Statistician (Discussion) Role Prompt ⏠Provide statistical guidance for research planning. Recommend appropriate statistical tests (t-test, Mann-Whitney, ANOVA, etc.), assess sample size adequacy, and discuss study feasibility. Focus on TEXT-BASED discussion. The meeting output is a PLAN, not code. **Power analysis (REQUIRED in Phase 1):** Use the sample sizes reported by the Imaging Specialist to compute a priori power. Run this ONCE and report your conclusion: from statsmodels.stats.power import TTestIndPower analysis = TTestIndPower() power = analysis.power(effect_size=0.5, nobs1=<n1>, alpha=0.05, ratio=<n2/n1>) For correlation hypotheses, use NormalIndPower with effect_size=0.3. State clearly whether the study is adequately powered (threshold: 0.80). G.5 Coding ML Statistician Expertise: statistical analysis, ML, Python programming with scipy, numpy, sklearn, matplotlib. Goal: test hypotheses through code-driven statistical analysis. ML Statistician (Coding) Role Prompt ⏠Execute statistical analysis using Python code. Write code in steps if needed - each code block runs immediately and you see the output before continuing. **Pre-loaded SAT API:** - sat.list_patients(db_path) -> list of patient_ids - sat.get_patient_metadata(patient_id) -> dict with âgroupâ, etc. - sat.load_structure_mask(db_path, patient_id, structure, source_image_contains=observation) -> list of mask, spacing - sat.calculate_volume(mask, spacing) -> float (mL) **Data requirements:** - Use REAL data from sat API only (no mock/random data) - Do NOT use raw filesystem crawling as a substitute for sat API - Prefer the planned hypothesis test; switch only if assumptions violated **Statistical standards:** - Sign convention: positive effect = group1 has HIGHER values - Cohenâs d = (mean1 - mean2) / pooled_std - Mann-Whitney U effect size: rank-biserial r - CI: bootstrap for nonparametric, parametric CI for t-test - Survival: logrank_test or CoxPHFitter, effect size = hazard ratio **Required outputs:** - data/statistical_results.json (exact schema in agenda) - plots/*.png (at least one visualization) G.6 Phase-Aware Critic Expertise: providing phase-appropriate critical feedback for multi-phase research workflows. Goal: ensure work meets the specific goals of the current phase without demanding out-of-scope analysis. Phase-Aware Critic Role Prompt ⏠Provide phase-specific critique only. Identify the current phase from workflow instruction and evaluate only that phase. General critic behavior: - Prioritize blocking errors first, then high-value warnings. - Cite concrete evidence for each issue and give one actionable fix. - Do not request extra analyses outside the plan. PHASE 1 (planning): - Verify plan feasibility reasoning is coherent. - Verify core contract fields: groups, structures, observations, metrics. - Do not demand code execution in Phase 1. PHASE 2A (segmentation request): - Verify code writes segmentation_request.json. - Verify structures/observations align with Phase 1 plan. - Do not demand statistical testing in Phase 2A. PHASE 2B (statistical analysis): Blocking checks: 1) No fabricated/synthetic/mock data 2) No off-contract primary data loading 3) No manual sample capping/subsampling 4) data/statistical_results.json must exist with required keys 5) No silent deviation from planned groups/test/adjust_for 6) If plan requires adjustment, verify implementation Test policy: Prefer the planned test. If assumptions invalidate it, allow one valid alternative when justified. PHASE 3 (interpretation): - Verify interpretation uses Phase 2B results (no re-analysis). - Verify final verdict is clear and supported. - Verify final verdict JSON block is present. Appendix H Phase Agenda Templates Each phase begins with a structured agenda that is dynamically generated from the dataset configuration, hypothesis text, and outputs of preceding phases. Below we reproduce the core template for each phase, using angle-bracket placeholders (<...>) for values injected at runtime. These agendas are the actual prompts given to agentsâno additional instructions are provided beyond the agent system prompts (Section G) and the agenda below. H.1 Phase 1: Planning Agenda The planning agenda provides the hypothesis, dataset context, and role-specific instructions for the three discussion agents. Analysis Planning Phase Prompt ⏠**Research Question:** <hypothesis> **Dataset Context:** <dataset_info_text> **Task:** produce one executable analysis plan. **Core rules (all agents):** - Keep the target quantity exact; no proxy substitution unless hypothesis explicitly allows it. - If prerequisites are missing in principle, mark UNTESTABLE with subtype + missing requirements. - Use only dataset observations/timepoints in observations; never place metadata there. **Imaging specialist:** - Choose exact structure names via list_available_structures(). - Query cohort/sample sizes with list_dataset_patients() for planned groups. - If hypothesis is metadata-only (no image-derived measurements), set structures: []. **Statistician:** - Choose one primary test that matches the hypothesis. - Compute a priori power using queried sample sizes (d=0.5 group tests, r=0.3 correlations); report adequacy at 0.80. - Survival: use analysis_type: "survival"; use log-rank for unadjusted two-group survival and Cox PH for adjusted/continuous-predictor survival. - Mixed-cohort correlation/regression: consider confounding control via adjust_for or stratify_by. - For correlation/regression, declare exact tested variables in target_variables (one outcome + predictors). **PI synthesis:** - Use only metadata fields listed in dataset context. - Metadata-value groups belong in groups; use grouping_field and restrict_to when needed. - Derived groups require group_spec: type: "derived", rule: "...". **Output:** a valid Phase 1 JSON plan block with feasibility + analysis contract fields. The PI then produces a structured JSON output containing: feasibility status, groups, structures, observations, metrics, statistical test, analysis type, grouping field, predictors, covariates, and target variables. The full output schema is defined in build_phase1_summary_instructions(). H.2 Phase 2A: Segmentation Request Agenda Phase 2A translates the Phase 1 plan into a concrete segmentation request by querying the SAT API for patient identifiers. Segmentation Phase Prompt ⏠Build segmentation_request.json from the Phase 1 plan. **Plan contract:** - Dataset: <dataset_name> - Groups: <groups> - Structures: <structures> - Observations: <observations> - Cohort mode: <cohort_mode> **Rules:** - Use exact structure names from plan, unchanged. - Build identifiers in format <dataset>:<patient_id>:<observation>. - Include only plan observations. - If groups are ["ALL"], iterate real dataset groups; never pass "ALL" to API. - For metadata-group hypotheses, map plan labels to canonical metadata values. **API (pre-loaded):** result = list_dataset_patients("<dataset>", group="<group>") meta = get_patient_metadata("<dataset>", "patient001") for obs_name in <observations>: identifier = meta["identifiers"].get(obs_name) **Required output (segmentation_request.json):** "identifiers": ["<dataset>:patient001:<obs>", ...], "structures": <structures>, "results_database": "<results_db>", "modality": "<modality>", "model_variant": "nano", "chunk_size": 64 Write complete code now and save as segmentation_request.json. H.3 Phase 2B: Statistical Analysis Agenda The statistical analysis agenda is the most detailed, specifying the exact analysis contract, SAT API usage, hard constraints, and required output schema. Statistical Analysis Phase Prompt ⏠Execute Phase 2B statistical analysis using Phase 2A outputs. **Plan contract (must match exactly):** - results_db: <phase2a_results_db> - groups: <groups> - structures: <structures> - observations: <observations> - metrics: <metrics> - statistical_test: <test> - analysis_type: <group_difference|correlation|regression|survival> - cohort_mode: <groups|all> - predictors: <predictors> - adjust_for: <covariates> -- REQUIRED|RECOMMENDED|OPTIONAL - stratify_by: <variables> -- REQUIRED|RECOMMENDED|OPTIONAL - target_variables: outcome: "<var>", predictors: ["<var>", ...] - available metadata fields: <metadata_fields> **Hard constraints:** - Use SAT APIs as primary data source; no synthetic/mock placeholders. - Use full eligible cohort after planned restrictions (no slicing/sample caps). - Prefer the planned primary analysis. If assumptions invalidate it, switch only to a statistically valid alternative and document the reason. - Implement planned adjustment/stratification in code and in statistical_results.json. - If any planned group has n=0, raise an error and fix loading/filter logic. - Before finishing, ensure data/statistical_results.json exists. **Analysis track (strict):** <track-specific guidance based on analysis_type and adjustment level> **Task:** Write Python code to: 1. Load segmentation results via sat.list_patients(results_db_path) 2. Filter patients by group (<group_filter_instruction>) 3. Load masks for structures at observations 4. Calculate metrics from segmentation masks 5. Perform <statistical_test> according to analysis_type=<type> 6. Create visualizations 7. Save results to data/statistical_results.json and plots/*.png **SAT API (pre-loaded):** - sat.list_patients(results_db_path) -> all patient IDs - sat.get_patient_metadata(patient_id) -> metadata - sat.get_observation_identifiers(patient_id) -> per-patient obs map - sat.load_structure_mask(results_db_path, patient_id, structure, source_image_contains=...) - sat.calculate_volume(mask, spacing) **Required output (data/statistical_results.json):** "analysis_type": "<type>", "test_performed": "<test>", "p_value": <float>, "effect_size": <float>, "effect_size_type": "<cohens_d|rank_biserial|hazard_ratio|...>", "n_total": <int>, "sample_sizes": "<group1>": <int>, "<group2>": <int>, "variables_tested": outcome: "<var>", predictors: ["<var>"] Write complete analysis code now. H.4 Phase 3: Interpretation Agenda The interpretation agenda provides the results snapshot from prior phases and guides the discussion toward a final verdict. Interpretation Phase Prompt ⏠**Hypothesis:** <hypothesis> **Results from Previous Phases:** <results_snapshot> **Task:** Interpret the results and reach a verdict. **Discussion Focus:** This is an INTERPRETATION phase - focus on TEXT-BASED discussion. - Review what was found in Phase 2A (segmentation) and Phase 2B (statistics) - Discuss implications, limitations, and confidence in the findings - Reach a final verdict: YES (supported), NO (rejected), or INCONCLUSIVE **Verdict Guidelines:** - YES: p < 0.05 AND effect in direction claimed by hypothesis. Verify direction from group_statistics means (NOT effect_size sign). - NO: Not statistically significant, OR effect direction is opposite. Non-significant result in adequately powered study (power >= 0.80) is evidence AGAINST the hypothesis. - INCONCLUSIVE: ONLY when BOTH: (1) Phase 1 found study underpowered (power < 0.80) AND (2) result is not significant (p >= 0.05). - INVALID: fundamental methodological failures (NaN results, p=1.0 exactly, completely wrong analysis type) -- not for assumption violations. **Required output:** "verdict": "<YES|NO|INCONCLUSIVE>", "evidence_label": "<SUPPORTED|REFUTED|UNDERPOWERED|INVALID>", "p_value": <number from Phase 2B>, "effect_size": <number from Phase 2B>, "test_used": "<test name>", "sample_sizes": "group1": N, "group2": N, "confidence": "<high|medium|low>", "reasoning": "<one sentence conclusion>" Appendix I Ethics and Broader Impact 1. Research acceleration, not clinical deployment: Veritas is designed to accelerate hypothesis screening in research settings. It produces evidence labels, not medical advice, and human expert oversight remains essential before any clinical or policy decision. 2. Generalization: Our evaluation spans two MRI modalities (cardiac cine and brain MRI) with a single segmentation backend. Performance on other imaging modalities, pathologies, or segmentation models requires future validation. 3. Statistical assumptions: The evidence-label framework relies on conventional SESOI thresholds. Domain-specific calibration of these thresholds may be necessary for applications where clinically meaningful effect sizes differ substantially from Cohenâs conventions. 4. Code correctness: While code-driven execution eliminates natural-language statistical hallucination (fabricated p-values), agents can generate syntactically correct but methodologically flawed code (e.g., the mediator-adjustment error in Section C). The automated validity checks (Section D.5) catch many but not all such errors. 5. Automation bias: Users may over-trust automated conclusions, particularly when the system produces detailed statistical outputs and visualizations. The auditability framework (Section A.3) mitigates this by making every reasoning step and code artifact inspectable, but appropriate training on system limitations remains important.