Paper deep dive
ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts
Zhaoyu Fan, Bowen Han, Jincheng Ou, Yanwei Ren, Zehua He, Kaiyu Huang, Pengcheng He, Shanshan Ni, Chengchen Gong, Shuai Xue, Qingjiang Shi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Real-world studies (RWS) analyze routinely collected healthcare data, such as electronic health records and insurance claims, to estimate treatment effects, monitor safety, and identify risk factors outside randomized trials. Automating RWS with large language model agents is appealing but brittle: valid analyses depend on dataset-specific clinical semantics, including index dates, follow-up windows, patient/event-level units, and outcome boundaries that are often implicit in observational data. This leads to schema mismatches, temporal errors, inconsistent analytical granularity, and unsupported clinical claims. We present ProCARE, a profile-grounded framework that reformulates RWS automation as constrained compilation. ProCARE converts heterogeneous data profiles into Evidence Contracts specifying available variables, temporal attributes, distributions, and evidentiary limits, and compiles study objectives into RWS-IR, a strongly typed intermediate representation for variables, time relations, analysis units, and statistical tasks. Deterministic validation and localized repair then check and revise generated workflows across schema mapping, cohort construction, variable derivation, analysis, and interpretation. On 36 EHR-based benchmark tasks, ProCARE outperforms strong LLM-agent baselines in code executability, semantic consistency, and conclusion traceability, with automated violation diagnoses showing moderate-to-substantial agreement with blinded expert review.
Tags
Links
- Source: https://arxiv.org/abs/2610.16175v1
- Canonical: https://arxiv.org/abs/2610.16175v1
Trouble viewing inline? Open PDF directly ā
Full Text
153,606 characters extracted from source content.
Expand or collapse full text
ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts Zhaoyu Fan Bowen Han11footnotemark: 1 Jincheng Ou Yanwei Ren Zehua He Kaiyu Huang ā thanks: Equal contribution. Affiliation: Tongji University Affiliation: University of Electronic Science and Technology of China Affiliation: Shanghai Hengfang Health Technology Co., Ltd. Pengcheng He Shanshan Ni Chengchen Gong Shuai Xue Qingjiang Shi ā thanks: Corresponding author. Email: shiqj@tongji.edu.cn. Affiliation: Tongji University Affiliation: Shanghai Hengfang Health Technology Co., Ltd. Affiliation: Shenzhen Research Institute of Big Data Abstract Real-world studies (RWS) analyze routinely collected healthcare data, such as electronic health records and insurance claims, to estimate treatment effects, monitor safety, and identify risk factors outside randomized trials. Automating RWS with large language model agents is appealing but brittle: valid analyses depend on dataset-specific clinical semantics, including index dates, follow-up windows, patient/event-level units, and outcome boundaries that are often implicit in observational data. This leads to schema mismatches, temporal errors, inconsistent analytical granularity, and unsupported clinical claims. We present ProCARE, a profile-grounded framework that reformulates RWS automation as constrained compilation. ProCARE converts heterogeneous data profiles into Evidence Contracts specifying available variables, temporal attributes, distributions, and evidentiary limits, and compiles study objectives into RWS-IR, a strongly typed intermediate representation for variables, time relations, analysis units, and statistical tasks. Deterministic validation and localized repair then check and revise generated workflows across schema mapping, cohort construction, variable derivation, analysis, and interpretation. On 36 EHR-based benchmark tasks, ProCARE outperforms strong LLM-agent baselines in code executability, semantic consistency, and conclusion traceability, with automated violation diagnoses showing moderate-to-substantial agreement with blinded expert review. 00footnotetext: GitHub repository: https://github.com/ZyFanTJ/ProCARE. 1 Introduction Real-world studies (RWS) derive clinical evidence from observational healthcare data such as electronic health records and insurance claims [1]. Unlike generic table analysis, RWS workflows are constrained by relational structure, clinical variable semantics, temporal anchors, analysis grain, and evidence scope. Cohorts must be assembled across heterogeneous tables, variables must be assigned valid clinical roles, exposure and follow-up must be anchored consistently, and reported findings must remain within the support of the executed analysis. These requirements make RWS automation a structured scientific generation problem rather than a plain code-generation task. LLM-based agents have been applied to data science and scientific workflow automation [2, 3, 4], but their standard interfaces do not preserve all constraints required by schema-rich clinical data. Textualizing multi-table datasets can obscure entity keys, visit structure, table linkage, missingness, and time-dependent variables [5, 6, 7]. RWS requires domain-specific reasoning about endpoint definitions, censoring, confounding, proxy variables, and claim scope [8, 9]. We observe three recurring failure modes: query-to-specification failures, where study concepts are bound to missing, incompatible, or ambiguous fields; specification-to-code failures, where free-form code violates the declared join graph, analysis grain, temporal ordering, or artifact protocol; and artifact-to-claim failures, where descriptive statistics or proxy artifacts are converted into unsupported clinical claims. These failures indicate missing contracts between the data, study specification, executable workflow, and final reportāa structural bottleneck that bounds even strong baselines like Claude Code. ProCARE addresses this gap by formulating RWS automation as contract-valid generation. Its object is a heterogeneous data profile Ī¦ , which records schema structure, field statistics, clinical semantics, temporal roles, validity flags, and assumptions. Rather than serving as a context summary, Ī¦ acts as a profile-grounded evidence contract. ProCARE compiles each user goal into a typed RWS intermediate representation Z, turns Z into executable artifacts through a profile-constrained coding harness, validates execution with deterministic static and runtime gates that produce localized violation sets tU_t, and admits report claims only through evidence certificates eqe_q that link claims to validated artifacts, executed code, profile entries, or recorded assumptions [10]. The output is an auditable artifact bundle containing the profile, RWS-IR, generated code, execution logs, result and figure manifests, validation evidence, assumptions, evidence certificates, and final report. Contributions. (i) a heterogeneous RWS profile and typed RWS-IR that define a study contract over data structure, clinical semantics, temporal roles, analysis grain, and assumptions; (i) evidence certificates that restrict factual report claims to validated artifacts, executed code, profile entries, IR fields, or assumptions; and (i) an end-to-end RWS evaluation over 36 HCC tasks and 18 public-health transfer tasks, measuring executability, code quality, artifact completeness, and claim traceability. 2 Related Work LLM agents for data analysis. Recent LLM agents combine reasoning, tool use, code execution, and feedback for multi-step data-analysis and scientific workflows [11, 12, 13, 14]. Data-analysis systems and benchmarks such as InfiAgent-DABench, Data Interpreter, DSBench, and DataSciBench evaluate agents that translate natural-language goals into programs, tables, visualizations, or model outputs [15, 16, 17, 18]. Despite this progress, these systems still report failures in state tracking, dependency management, numerical consistency, and data-dependent repair. In RWS, such errors become methodological failures because executable code must preserve relational structure, analysis grain, temporal anchors, clinical roles, and claim scope. ProCARE addresses this setting with contract-valid generation over typed study objects, executable artifacts, and certified claims. Structured and clinical-data grounding. Another line of work improves LLM grounding over spreadsheets, tables, SQL databases, and structured clinical records. Table-centric methods compress, encode, or retrieve from large tabular structures [19, 20, 21, 22, 23], while text-to-SQL, structured-EHR, and FHIR benchmarks evaluate reasoning over relational or resource-based clinical data [24, 25, 26]. These methods improve structured-data access, but they mainly target retrieval, query answering, or task-level execution. RWS automation further requires variable bindings, joins, temporal anchors, artifacts, and claims to remain valid under one study-level contract. ProCARE enforces this contract by propagating a heterogeneous profile through goal-to-IR compilation, profile-constrained execution, and evidence-certified reporting. RWS evidence constraints. RWE generation is governed by methodological, reporting, and regulatory constraints beyond table access or code execution. STROBE and RECORD emphasize transparent reporting of study design, data sources, variables, bias, missingness, and provenance [27, 28]; target-trial emulation and TARGET guidance require explicit protocols, estimands, eligibility criteria, treatment strategies, follow-up, assumptions, and observational-data mappings [29, 30]. OMOP, FHIR, and FDA guidance further emphasize standardized representation and reliable provenance for RWD use [31, 32, 33]. Recent RWE automation benchmarks test whether LLM agents can conduct observational analyses or reproducible hypothesis testing from medical databases [34, 35]. ProCARE incorporates these constraints into generation by compiling typed RWS-IR, localizing execution failures as contract violations, and admitting factual claims only through evidence certificates. 3 Method Figure 1: ProCARE architecture for contract-valid RWS automation. A heterogeneous RWS profile constrains RWS-IR compilation, violation-guided execution, and evidence-certified DAG reporting as a profile-grounded evidence contract. ProCARE casts RWS automation as contract-valid generation over the three profile-constrained transitions in Figure 1. Given a heterogeneous profile Ī¦ and user goal TgoalT_goal, it produces a typed study specification, validated code, result artifacts, a report, and admissible claims: Ī¦,Tgoalāprofile contract(Z,Cā,O,R,Q), ,T_goal profile contract(Z,C^*,O,R,Q), (1) where Z is the RWS-IR, CāC^* validated code, O result and figure artifacts, R the report, and Q admitted claims. The contract requires every study choice, code operation, and claim to trace to profile entries, validated artifacts, or explicit assumptions. ProCARE enforces this contract at three boundaries. Query-to-IR compilation admits only profile-supported study designs, variables, temporal anchors, and methods. IR-to-code execution accepts only runs that preserve the declared analysis grain, join graph, temporal ordering, and artifact requirements. Artifact-to-claim reporting admits only claims with evidence certificates linked to validated artifacts, executed code, IR fields, profile entries, or recorded assumptions. The generation state is maintained as typed artifacts t=Ī¦,Zt,Ct,Ot,ā°t,t,tS_t=\ ,Z_t,C_t,O_t,E_t,U_t,A_t\. Here ā°tE_t stores validation evidence, tU_t contract violations, and tA_t assumptions and limitations. 3.1 Heterogeneous Data Profiling ProCARE first converts raw RWS inputs into a heterogeneous profile Ī¦=(Ī¦H,S,Ī¦H,K,Ī¦L,S,Ī¦L,K) =( ^H,S, ^H,K, ^L,S, ^L,K), indexed by profile level (dataset/task vs. field/instance) and abstraction (structural vs. knowledge-driven). Table 1 summarizes the four blocks. Structural blocks specify joins, grain, temporal anchors, and artifact constraints; knowledge blocks record clinical roles, ontology mappings, proxy status, assumptions, and claim scope. Deterministic parsers extract structural profiles, while ontology checks over UMLS and SNOMED CT validate candidate clinical mappings for semantic type, units, ranges, and temporal-role compatibility. Failed or ambiguous mappings are marked as semantically unknown in Ī¦L,K ^L,K with their failure causes. Semantically unknown fields may support only structure-grounded operations: they cannot bind clinical roles, derive endpoints, or support claims beyond their validated level unless a proxy or explicit assumption is recorded. Thus, survival, safety, and utilization queries are compiled from validated entity levels, linkage paths, temporal anchors, clinical roles, and support levels. Unsupported choices are filtered, downgraded, or reported as limitations before IR compilation. Table 1: Heterogeneous data profile Ī¦ organized by profile level and abstraction. High-level profile (dataset/task) Low-level profile (field/instance) Structural Ī¦H,S ^H,S Clinical form registry; subject/visit/table linkage; cohort-level missingness. Ī¦L,S ^L,S Field type, unit, range, date fields, temporal anchors, and window-specific missingness. Knowledge Ī¦H,K ^H,K Cohort and index-date schema; exposure, outcome, censoring, confounder, and claim-scope roles. Ī¦L,K ^L,K Ontology mapping; unit/value validation; temporal-role validation; field-level proxy and confounder roles. 3.2 RWS-IR Compilation and Violation-Guided Execution The IR and execution stages implement the first two contract transitions: query-to-IR and IR-to-code. The compiler turns the user goal into a typed RWS-IR, while the execution stage tests whether the generated analysis can be run without violating the profile, the IR, or the artifact policy. RWS-IR compilation. Given a research goal TgoalT_goal, ProCARE first generates a candidate RWS-IR using the high-level profile: Z0=fIRā(Tgoal,Ī¦H,SāĪ¦H,K),Z_0=f_IR (T_goal, ^H,S ^H,K ), (2) where ā denotes structured fusion of profile blocks. The candidate IR is admitted only after static profile checking: (Z,static)=IRCheckstaticā(Z0,Ī¦,t).(Z,U_static)=IRCheck_static(Z_0, ,S_t). (3) The accepted IR is Z=āØD,G,J,K,T,V,M,A,Ī qā©Z= D,G,J,K,T,V,M,A, _q , where D, G, J, K, T, V, M, A, and Ī q _q denote study design, analysis grain, join graph, entity keys, temporal anchors, variable-role bindings, method specification, required artifacts, and claim policy, respectively. Static checks enforce three conditions. First, all referenced tables, fields, and keys must appear in Ī¦L,S ^L,S. Second, clinical roles such as exposure, outcome, censoring, and confounder must be supported by Ī¦L,K ^L,K, marked as validated proxies, or explicitly recorded as assumptions. Third, the declared join graph, index date, follow-up window, and analysis grain must be compatible with the profiled schema. Only IRs with static=ā U_static= enter code generation. Otherwise, ProCARE repairs the IR, downgrades the unsupported request, or emits an infeasibility statement. Row-level temporal and grain checks are deferred to execution-time validation. Profile-constrained execution. Given an admitted IR, ProCARE generates executable analysis code inside a workspace containing Z, Ī¦L,S ^L,S, Ī¦L,K ^L,K, the claim policy Ī q _q, allowed tools, coding policies, artifact validators, and execution logs. A run is accepted only if it produces validated code CāC^*, complete output artifacts O, validation evidence ā°E, and disclosed assumptions or limitations A. Execution is controlled by deterministic gates. Schema gates require every table, field, join key, and output column to be traceable to Ī¦L,S ^L,S. Method gates require cohort filters, index dates, exposure definitions, endpoint or censoring rules, covariates, temporal windows, and statistical choices to match Z and the validated semantic roles in Ī¦L,K ^L,K. Artifact gates require the selected entry point, such as analysis.py or main.py, to run successfully, avoid failure sentinels, and produce complete result and figure manifests. After each execution attempt, validators return a violation set t=Vexecā(Zt,Ct,Ot,ā°t,Ī¦,t)U_t=V_exec(Z_t,C_t,O_t,E_t, ,S_t). Each violation records the failed contract clause, offending code or artifact, nearest valid profile entries, and permitted repair action. When tā ā U_tā , the next edit is conditioned on the localized failures, (Zt+1,Ct+1)=AĪøā(Zt,Ct,t,t)(Z_t+1,C_t+1)=A_Īø(Z_t,C_t,U_t,S_t). The violation set guides repair but does not imply global edit minimality. If a requested endpoint, exposure, covariate, time window, or statistical analysis is unsupported by the profile, ProCARE terminates with a transparent failure trace or downgrades the task to a supported descriptive analysis with explicit limitations. Failed traces remain in tS_t for inspection or subsequent runs. This makes profile constraints and methodological assumptions observable during code generation, rather than implicit in prompt context. 3.3 Certified DAG-Based Reporting The reporting stage enforces the artifact-to-claim transition: only factual claims supported by validated evidence and allowed by the claim policy are admitted. For each claim q, the report generator emits an evidence certificate eq=āØoq,cq,zq,Ļq,aqā©e_q= o_q,c_q,z_q, _q,a_q , binding the claim to a supporting artifact, executed code region, IR field or method, profile entry, and optional assumption or limitation. A claim is admitted only if both traceability and policy checks pass: Admit(q)=[Trace(q,eq,O,Cā,Z,Ī¦,t)=1ā§Policy(q,Z,Ī q)=1].Admit(q)=I\! [Trace(q,e_q,O,C^*,Z, ,S_t)=1 (q,Z, _q)=1 ]. (4) The trace condition grounds q in validated artifacts, executed code, IR fields, profile entries, or disclosed assumptions; the policy condition checks whether its type and wording are allowed by Z.Ī qZ. _q at the available support level. Unsupported claims are removed or rewritten as limitations. Proxy-based or descriptive analyses may support descriptive or association claims, but not causal, safety-critical, or high-stakes claims without the required design assumptions and validations. After claim filtering, a dependency DAG orders report sections: nodes are report components, and edges encode prerequisites such as cohort definition, variable construction, temporal anchors, sample support, and method-design consistency. The DAG controls section order, while evidence certificates control claim admissibility. The support score and traversal rule are provided in the appendix. 4 Evaluation and Benchmark The benchmark evaluates complete RWS artifact bundles rather than isolated code outputs, covering study specification, code synthesis, visualization, and scientific reporting. Because ProCARE natively emits typed IRs, verifier traces, evidence certificates, and report-structure metadata, we distinguish artifact-level comparative metrics from contract-native diagnostics. Deterministic validators assess checkable constraints such as executability, schema use, temporal/grain consistency, artifact completeness, and claim support. Rubric-based judges are used only for residual semantic ambiguity, presentation quality, or stylistic criteria. Task construction. The primary benchmark contains 36 instances from four RWS domains: Cohort, Safety, Efficacy, and Economics. Each domain includes three research questions, and each question is instantiated under three prompt-specificity levels: Goal, Goal+Approach, and Goal+Method. All instances use the same longitudinal HCC cohort described below. Agents receive raw relational tables and natural-language prompts under a unified protocol. Output contract. Each prompt contains Task, optional Guidance, and Deliverables. The expected bundle includes: (i) a study specification, or equivalent intermediate description, covering tables, keys, variables, temporal anchors, methods, artifacts, and claim policy; (i) an executable entry point, such as analysis.py or main.py, with artifacts; and (i) a structured report with assumptions, limitations, and evidence links for factual claims. ProCARE instantiates this contract with typed RWS-IRs and evidence certificates; non-ProCARE systems are evaluated via submitted or post-hoc parsed specifications, manifests, and report-level evidence links. Hallucinated tables or columns are disallowed. Goal+Method must follow specified variables and terminate when required inputs are missing; Goal and Goal+Approach permit the closest feasible analysis with explicit limitations. Scoring protocol. Deterministic validators assess code executability, schema use, artifact completeness, temporal/grain consistency, and claim support. Study-specification validity is scored after mapping each systemās submitted or parsed specification into the common benchmark schema; native ProCARE IR objects are not treated as ground truth. Residual semantic criteria are scored by Gemini-2.5-Pro and Qwen2.5-1M at temperature 0; judges are blinded to method identity and evaluate full bundles under fixed rubrics. The same validators, rubrics, and post-hoc diagnostics are applied to ProCARE, non-ProCARE baselines, and Public-RWS; Public-RWS omits repeated board-certified review. All scores are normalized to [0,1][0,1], and automated diagnostics are compared with expert review on the primary benchmark. Artifact-level metrics support cross-system comparisons. Contract-native metrics, such as explicit IR validity, native certificate coverage, and report-DAG consistency, are reported as mechanism diagnostics and are not interpreted alone as evidence of clinical, causal, or statistical correctness. Table 2: Evaluation metrics for end-to-end RWS automation. Dimension Metrics and scoring IR GA: LLM-rubric goal alignment; PF: schema/IR feasibility with LLM adjudication. Execution XE: sandbox pass rate npass/Nn_pass/N; CQ: LLM-rubric code quality; VRR: resolved/detected contract violations. Visualization VC: LLM-rubric visual clarity; CF: chartāresult consistency. Reporting RS: report schema and section coverage; SL: LLM-rubric scientific language. Claims CTC: post-hoc claim trace coverage; UCR: unsupported factual-claim rate; CLV: language-support violation rate. Let Nv,RvN_v,R_v be detected and resolved execution-time violations, and Nq,Cq,Uq,LqN_q,C_q,U_q,L_q be factual, certified, unsupported, and language-violating claims. We compute VRR=RvNv,CTC=CqNq,UCR=UqNq,CLV=LqNq.VRR= R_vN_v, = C_qN_q, = U_qN_q, = L_qN_q. (5) VRR is measured over execution-time violations before the accepted run; CTC, UCR, and CLV are measured over final-report factual claims. To reduce evaluator coupling, native contract artifacts are separated from artifact-level scoring where possible. ProCAREās RWS-IRs, manifests, verifier traces, and evidence certificates are treated as system outputs rather than ground truth. For non-ProCARE systems, the checker reconstructs the closest available study specification, artifact manifest, and claim-evidence links from code, logs, tables, figures, reports, profile entries, and recorded assumptions. CTC/UCR/CLV are then scored post hoc against executed artifacts, rather than by accepting native certificates. Native certificate and DAG diagnostics are retained only for mechanism audit. This mitigates, but does not eliminate, coupling because validity is still operationalized through profile-derived constraints. Primary HCC benchmark. The primary benchmark is derived from IRB-approved longitudinal hepatocellular carcinoma data collected at a tertiary medical center. The released version is a fully de-identified record-level benchmark: direct identifiers and fields that could identify individual participants are removed before publication, while random benchmark subject and visit identifiers preserve the relational structure needed for reproduction. The cohort contains 1,295 subjects, 12,847 visits, and 54 interlinked electronic case-report-form sheets covering baseline intake, follow-up, treatments, laboratory measurements, outcomes, and quality-of-life assessments. The dataset captures practical RWS challenges: subject- and visit-level linkage, sparse high-dimensional forms, form-specific clinical conventions, redundant administrative fields, and longitudinal structure for repeated-measures and time-to-event analyses. The public benchmark includes the de-identified HCC tables, task specifications, schema/profile artifacts, prompts, data dictionary, checksums, and evaluation scripts; a complete sheet inventory and release description are provided in the appendix. Public-RWS external validation suite. Public-RWS-18 evaluates cross-dataset transfer on three public health datasets: UCI Diabetes 130-US Hospitals, an EHR-derived encounter-level readmission dataset [36, 37]; a NHANES 2017ā2018 subset linking demographics, examination, questionnaire, laboratory, medical-condition, and prescription-medication tables by SEQN [38]; and TCGA-LIHC, HCC-aligned oncology metadata with demographic, diagnosis, treatment, vital-status, and follow-up-related fields [39]. Each dataset contributes two task objectives under three prompt-specificity levels, yielding 18 external validation tasks. The suite tests schema shift, multi-table grounding, proxy-outcome handling, and claim traceability, complementing the primary HCC benchmark. 5 Experiments We evaluate ProCARE along five research questions: RQ1 end-to-end artifact validity against LLM-agent and table-analysis baselines; RQ2 contributions of individual contract mechanisms; RQ3 claim-level traceability and agreement with a five-member clinical/statistical expert panel; RQ4 robustness across RWS objectives, prompt specificity, and public-data schema shifts; and RQ5 sensitivity to the base LLM backbone and profiling cost amortization. The evaluation spans 36 primary HCC RWS tasks, 18 Public-RWS external validation tasks, baseline comparisons, ablations, claim diagnostics, expert review, backbone sensitivity, and cost analysis. 5.1 Main Results Table 3: Baseline comparison on 36 HCC RWS tasks. GAāSL report task performance, while VRRāCLV report contract diagnostics following Table 2; N/A indicates no executable artifact bundle. ā Proprietary; ā”Open-source; Ā§Agent-based. Method GAā PFā XEā CQā VCā CFā RSā SLā VRRā CTCā UCRā CLVā ReActā”Ā§ 0.78 0.62 0.38 0.47 0.28 0.48 0.63 0.59 0.34 0.48 0.24 0.17 ToolLLMā”Ā§ 0.73 0.68 0.00 N/A N/A N/A 0.56 0.24 0.00 0.21 0.33 0.24 TableGPT-Agentā”Ā§ 0.61 0.59 N/A N/A N/A N/A 0.40 0.43 N/A 0.18 0.36 0.22 ChatExcel-Maxā Ā§ N/A N/A N/A N/A N/A N/A 0.52 0.27 N/A 0.12 0.39 0.28 Gemini-3-Pro directā 0.75 0.51 0.13 0.45 0.28 0.48 0.61 0.53 0.18 0.36 0.27 0.15 TableGPT2ā” 0.41 0.45 0.00 0.25 N/A N/A N/A N/A 0.00 N/A N/A N/A Claude Codeā Ā§ 0.82 0.75 0.86 0.80 0.52 0.58 0.71 0.66 0.70 0.66 0.16 0.12 Codexā Ā§ 0.83 0.77 0.89 0.82 0.55 0.61 0.73 0.68 0.73 0.69 0.14 0.10 Methods without valid artifact bundles: LLM (DeepSeek-R1-0528); ChatExcel-Proā Ā§. ProCARE (Ours) 0.86 0.84 0.92 0.88 0.72 0.74 0.80 0.79 0.81 0.90 0.06 0.03 Baseline comparison (RQ1). We first compare ProCARE with LLM-agent, table-analysis, and code-agent baselines under matched prompts, inference budgets, and artifact contracts. Table 3 shows that existing methods often either fail to produce executable artifact bundles or emit factual claims unsupported by admissible artifacts, code, or claim policies. Claude Code and Codex substantially improve executability and code quality over generic agents, but remain weaker than ProCARE on clinically grounded profile fit, variable-role consistency, statistical language, and claim traceability. To partially control for evaluator bias toward systems that natively emit IRs, manifests, and evidence certificates, we additionally evaluate scaffolded baselines. These RQ1 controls keep the tested system as an external baseline or direct Gemini-3-Pro run, while only adding evaluator-facing information, output schemas, validator feedback, and repair budget. They do not expose ProCAREās typed IR compiler, certificate admission gate, certified reporting DAG, or internal planner state. The scaffolds improve generic baselines, especially in executability and artifact completeness. ProCARE still achieves the strongest violation resolution and post-hoc claim traceability (Table 4), suggesting that its advantage is not explained by schema exposure or manifest formatting alone. Table 4: Scaffolded RQ1 controls. Direct rows use Gemini-3-Pro with the same decoding settings as the default ProCARE backbone. Non-ProCARE rows are independent baseline adaptations; ProCARE modules are not transplanted into them. Baseline Fairness scaffold PFā XEā CQā VRRā CTCā UCRā CLVā ReAct + scaffold Ī¦ summary + bundle tmpl. + val. feedback + 3 repairs 0.790.79 0.740.74 0.690.69 0.620.62 0.700.70 0.130.13 0.100.10 ToolLLM + scaffold Same scaffold on the tool-use baseline 0.800.80 0.680.68 0.650.65 0.580.58 0.660.66 0.150.15 0.120.12 G3P + profile/template Ī¦ + manifests; no validator loop 0.730.73 0.660.66 0.620.62 0.310.31 0.550.55 0.200.20 0.150.15 G3P + validator repair Ī¦ + manifests + val. feedback + repairs 0.780.78 0.790.79 0.720.72 0.680.68 0.650.65 0.150.15 0.110.11 G3P + study-spec schema Emits study-spec artifact; no IR compiler 0.820.82 0.700.70 0.680.68 0.430.43 0.520.52 0.210.21 0.160.16 G3P + claim-trace schema Emits claim-trace artifact; no cert. gate 0.740.74 0.630.63 0.610.61 0.350.35 0.740.74 0.110.11 0.080.08 ProCARE reference ProfileāIRāvalidatorācertificateāDAG workflow 0.84 0.92 0.88 0.81 0.90 0.06 0.03 Contract mechanism ablation (RQ2). To identify the sources of improvement, we ablate each contract object in Table 5. The profile yields the largest contribution, suggesting that reliable schema, field, and join constraints are the basis for valid execution and downstream reasoning. RWS-IR supplies the typed study contract and strengthens claim-policy control. Violation-guided repair mainly improves executable constraint satisfaction, whereas evidence certificates and certified DAG ordering primarily affect reporting: the former enforces claim traceability, and the latter preserves dependency-aware structure. These results indicate that the frameworkās gains come from complementary contract objects: structural contracts support computation, repair improves execution robustness, and evidence/order contracts regulate faithful reporting. Table 5: Mechanism ablation over 36 RWS instances. N/A indicates no typed RWS-IR. Variant PFā XEā VRRā CTCā UCRā SLā Dominant failure No profile Ī¦ 0.49 0.18 0.35 0.58 0.28 0.42 Field hallucination; invalid joins. No RWS-IR N/A 0.34 0.44 0.63 0.22 0.69 Missing grain, anchors, and claim policy. No repair 0.84 0.69 0.43 0.80 0.13 0.70 Unresolved runtime and artifact violations. No certificates 0.84 0.92 0.81 N/A 0.26 0.62 Unsupported causal or comparative claims. No DAG ordering 0.84 0.92 0.81 0.84 0.11 0.64 Dependency-agnostic report ordering. Full ProCARE 0.84 0.92 0.81 0.90 0.06 0.79 Localized repair; traceable final claims. Claim traceability and expert validation (RQ3). Table 6 reports post-hoc claim diagnostics for ProCARE. More specific prompts improve CTC and reduce UCR/CLV, mainly by constraining variable bindings, temporal anchors, and methods. Remaining downgrades mostly involve causal or high-stakes wording unsupported by proxy artifacts, sparse endpoints, or subgroup evidence. Table 6: ProCARE claim diagnostics. Values are averaged over 12 instances per prompt level; Overall aggregates all 36 instances. Prompt Claims CTCā UCRā CLVā Main downgrade reason Goal 5.85.8 0.870.87 0.090.09 0.050.05 Unsupported causal/comparative wording. Goal+Approach 6.46.4 0.900.90 0.050.05 0.030.03 Proxy endpoint beyond evidence level. Goal+Method 6.96.9 0.940.94 0.030.03 0.020.02 Sparse endpoint or subgroup evidence. Overall 6.4 0.90 0.06 0.03 Downgrades mainly concern causal strength. We further compare automated diagnostics with blinded review by three clinical and two statistics/data experts. Table 7 shows moderate-to-strong Spearman agreement across stages, strongest for RWS-IR validity (Ļ=0.82Ļ=0.82) and significant for code, visualization, and reporting. The appendix provides reviewer rubrics, aggregation rules, confidence intervals, and inter-rater reliability. Table 7: Expert consistency of automated metrics. Expert scores are meanĀ± on a 5-point scale; CI and reliability details are reported in the appendix. Stage Reviewers Metric(s) Auto Expert Ļ RWS-IR 3 clinical + 2 stats/data (GāA+PāF)/2(GA+PF)/2 0.82 4.38Ā± 0.48 0.82 Code 2 stats/data CāQCQ 0.88 4.42Ā± 0.21 0.74 Visualization 3 clinical + 2 stats/data (VāC+CāF)/2(VC+CF)/2 0.66 4.50Ā± 0.53 0.75 Reporting 3 clinical + 2 stats/data SāLSL 0.76 4.13Ā± 0.64 0.71 All Spearman correlations are computed across the 36 primary HCC instances; all p<0.001p<0.001. 5.2 Robustness and Deployment Analyses Task, prompt, and public-data robustness (RQ4). Figure 2 evaluates ProCARE across four RWS objectives and three prompt-specificity levels on the HCC benchmark. Metrics generally improve with prompt specificity, suggesting that methodological context helps specification, repair, and claim control. Performance is stable across task families; residual variation is mainly due to sparse endpoints, limited subgroup support, and claim-scope restrictions. Figure 2: Performance across four RWS tasks and prompt-specificity levels. Cell labels report raw metric values; UCR and CLV are lower-is-better rates with reversed color intensity. Table 8 evaluates Public-RWS-18 schema transfer without dataset-specific changes. Claim policies restrict outputs to descriptive/readmission-proxy claims for UCI Diabetes, descriptive/association claims for NHANES, and descriptive oncology-metadata/follow-up-proxy claims for TCGA-LIHC. ProCARE outperforms Claude Code and Scaffolded G3P in violation resolution, post-hoc claim traceability, UCR, and CLV; NHANES remains hardest due to multi-table survey joins. Table 8: Public-RWS-18 transfer results. Values average six tasks per dataset; baseline claim traces are computed post hoc. Dataset Method XEā VRRā CTCā UCRā CLVā UCI Diabetes Claude Code 0.60 0.61 0.69 0.12 0.09 Scaffolded G3P 0.57 0.59 0.66 0.13 0.10 ProCARE 0.64 0.72 0.87 0.07 0.04 NHANES Claude Code 0.52 0.54 0.63 0.15 0.11 Scaffolded G3P 0.50 0.52 0.60 0.16 0.12 ProCARE 0.58 0.66 0.84 0.09 0.06 TCGA-LIHC Claude Code 0.57 0.58 0.66 0.14 0.10 Scaffolded G3P 0.54 0.56 0.63 0.15 0.11 ProCARE 0.61 0.69 0.86 0.08 0.05 Backbone sensitivity and cost (RQ5). Holding the profile, RWS-IR, validators, and certificate gate fixed, ProCARE remained contract-consistent across tested backbones; stronger planners mainly improved GA, PF, VRR, and CTC, while execution metrics were comparatively stable. Profiling is a one-time dataset cost (Tprofā65T_ profā 65k tokens for the HCC schema); per-query inference costs about Tinfā38T_ infā 38k tokens, giving an amortized cost TĀÆā(N)=Tinf+Tprof/N T(N)=T_ inf+T_ prof/N and TĀÆā(20)ā41.3 T(20)ā 41.3k tokens. Full backbone results are reported in Appendix. Table 9: Backbone sensitivity with fixed ProCARE workflow. Metrics follow Table 2; model access details and version identifiers are reported in the appendix. ā Proprietary models; ā”Open-source models. Family Model Performance Contract diagnostics GAā PFā XEā CQā VCā CFā RSā SLā VRRā CTCā UCRā CLVā GPT GPT-5.2ā 0.84 0.82 0.94 0.84 0.68 0.70 0.78 0.77 0.78 0.88 0.06 0.04 GPT-4 Turboā 0.81 0.78 0.89 0.80 0.64 0.66 0.75 0.74 0.74 0.85 0.08 0.06 Claude Claude Opus 4.5ā 0.85 0.83 0.97 0.86 0.72 0.74 0.78 0.77 0.80 0.89 0.06 0.04 Claude Sonnet 4.5ā 0.83 0.81 0.94 0.84 0.70 0.72 0.76 0.75 0.78 0.87 0.07 0.05 Gemini Gemini-3-Proā 0.86 0.84 0.92 0.88 0.72 0.74 0.80 0.79 0.81 0.90 0.06 0.03 Gemini-2.5-Proā 0.82 0.78 0.91 0.82 0.66 0.68 0.75 0.74 0.76 0.86 0.08 0.06 Qwen Qwen3-Maxā 0.77 0.70 0.84 0.76 0.58 0.60 0.68 0.68 0.68 0.80 0.12 0.09 Qwen3-235B-Instructā” 0.74 0.66 0.80 0.72 0.55 0.57 0.64 0.65 0.64 0.76 0.15 0.11 DeepSeek DeepSeek-V3.2ā” 0.76 0.68 0.86 0.78 0.60 0.62 0.66 0.67 0.70 0.80 0.11 0.08 DeepSeek-R1-0528ā” 0.72 0.64 0.82 0.74 0.56 0.58 0.62 0.63 0.66 0.77 0.13 0.10 Token budgets are not directly comparable across agents because each baseline uses a different pipeline, so we report token use for ProCARE. Profiling is a one-time dataset cost, about Tprofā65T_profā65k tokens for the rich schema; each query then uses about Tinfā38T_infā38k tokens. For N queries on the same dataset, the amortized cost is TĀÆā(N)=Tinf+Tprof/N T(N)=T_inf+T_prof/N. A single query costs TĀÆā(1)ā103 T(1)ā103k tokens; with N=20N=20, the cost falls to TĀÆā(20)ā41.3 T(20)ā41.3k tokens, close to the inference-only cost. Discussion and Limitations. Overall, ProCARE should be interpreted as a contract layer for making RWS assumptions, artifacts, and claim support auditable under human oversight. Its reliability depends on the correctness of the constructed profile, the coverage of deterministic validators, and the assumptions encoded in the RWS-IR. A successful run shows that the generated bundle satisfies the implemented schema, method, temporal, and artifact checks; it does not establish clinical optimality, adequate confounding control, causal validity, or evidentiary sufficiency beyond the recorded assumptions. The primary benchmark is single-center and disease-specific, and Public-RWS-18 mainly tests schema transfer rather than multi-institutional longitudinal generalization. Deployments on OMOP, FHIR, or institutional schemas therefore require re-instantiating the profile and revalidating the IR, execution gates, and certificate checks. 6 Conclusion We introduced ProCARE, a profile-grounded contract architecture for RWS automation. ProCARE compiles study goals into typed RWS-IR, generates executable artifact bundles through a profile-constrained coding harness, repairs localized contract violations, and admits report claims through evidence certificates. On a 36-instance HCC RWS benchmark and Public-RWS-18 external validation suite, these mechanisms improve executable validity, code quality, and claim traceability relative to LLM-agent and table-analysis baselines. The public release includes the fully de-identified HCC benchmark tables and scoring assets to support independent reproduction. Future work should test the same contract mechanisms on multi-institutional longitudinal RWS data and standardized OMOP/FHIR deployments. References [1] R. E. Sherman, S. A. Anderson, G. J. D. Pan, G. W. Gray, T. Gross, N. L. Hunter, L. LaVange, D. Marinac-Dabic, P. W. Marks, M. A. Robb, J. Shuren, R. Temple, J. Woodcock, L. Q. Yue, and R. M. Califf (2016) Real-world evidence - what is it and what can it tell us?. The New England journal of medicine 375 (23), p. 2293ā2297. External Links: Document Cited by: Ā§1. [2] Y. Zheng, H. Y. Koh, J. Ju, A. T. N. Nguyen, L. T. May, G. I. Webb, and S. Pan (2023) Large language models for scientific synthesis, inference and explanation. arXiv preprint arXiv:2310.07984. Cited by: Ā§1. [3] Y. Jiang, K. C. Black, G. Geng, D. Park, J. Zou, A. Y. Ng, and J. H. Chen (2025) MedAgentBench: a realistic virtual EHR environment to benchmark medical LLM agents. arXiv preprint arXiv:2501.14654. Cited by: Ā§1. [4] J. Kim, A. Podlasek, K. Shidara, F. Liu, A. Alaa, and D. Bernardo (2025) Limitations of large language models in clinical problem-solving arising from inflexible reasoning. Scientific Reports 15 (1), p. 39426. Cited by: Ā§1. [5] A. Rajkomar, E. Oren, K. Chen, A. Dai, N. Hajaj, P. Liu, X. Liu, M. Sun, P. Sundberg, H. Yee, K. Zhang, G. Duggan, G. Flores, M. Hardt, J. Irvine, Q. Le, K. Litsch, J. Marcus, A. Mossin, and J. Dean (2018) Scalable and accurate deep learning for electronic health records. npj Digital Medicine 1, p. 18. External Links: Document Cited by: Ā§1. [6] G. Hripcsak, J. Duke, N. Shah, C. Reich, V. Huser, M. Schuemie, M. Suchard, R. W. Park, I. Wong, P. Rijnbeek, J. Lei, N. Pratt, N. NorĆ©n, Y. Li, P. Stang, D. Madigan, and P. Ryan (2015) Observational health data sciences and informatics (ohdsi): opportunities for observational researchers. Studies in health technology and informatics 216, p. 574ā8. Cited by: Ā§1. [7] J. Bodensohn, U. Brackmann, L. Vogel, A. Sanghi, and C. Binnig (2025) Unveiling challenges for llms in enterprise data engineering. arXiv preprint arXiv:2504.10950. Cited by: Ā§1. [8] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, N. Scharli, A. Chowdhery, P. Mansfield, B. A. y Arcas, D. Webster, G. S. Corrado, Y. Matias, K. Chou, J. Gottweis, N. Tomasev, Y. Liu, A. Rajkomar, J. Barral, C. Semturs, A. Karthikesalingam, and V. Natarajan (2023) Large language models encode clinical knowledge. Nature 620, p. 172ā180. Cited by: Ā§1. [9] H. Wang, T. Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P. Chandak, S. Liu, P. V. Katwyk, A. Deac, A. Anandkumar, K. J. Bergen, C. P. Gomes, S. Ho, P. Kohli, J. Lasenby, J. Leskovec, T. Liu, A. K. Manrai, D. S. Marks, B. Ramsundar, L. Song, J. Sun, J. Tang, P. Velickovic, M. Welling, L. Zhang, C. W. Coley, Y. Bengio, and M. Zitnik (2023) Scientific discovery in the age of artificial intelligence. Nature 620, p. 47ā60. External Links: Link Cited by: Ā§1. [10] S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, Z. Liu, and E. Barsoum (2025) Agent laboratory: using llm agents as research assistants. arXiv preprint arXiv:2501.04227. Cited by: Ā§1. [11] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations, External Links: Link Cited by: Ā§2. [12] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: Ā§2. [13] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun (2024) ToolLLM: facilitating large language models to master 16000+ real-world APIs. In Proceedings of the 12th International Conference on Learning Representations, External Links: Link Cited by: Ā§2. [14] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang (2023) AutoGen: enabling next-gen LLM applications via multi-agent conversation. External Links: 2308.08155, Link Cited by: Ā§2. [15] X. Hu, Z. Zhao, S. Wei, Z. Chai, Q. Ma, G. Wang, X. Wang, J. Su, J. Xu, M. Zhu, Y. Cheng, J. Yuan, J. Li, K. Kuang, Y. Yang, H. Yang, and F. Wu (2024) InfiAgent-DABench: evaluating agents on data analysis tasks. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 19544ā19572. External Links: Link Cited by: Ā§2. [16] S. Hong, Y. Lin, B. Liu, B. Liu, B. Wu, C. Zhang, D. Li, J. Chen, J. Zhang, J. Wang, L. Zhang, L. Zhang, M. Yang, M. Zhuge, T. Guo, T. Zhou, W. Tao, R. Tang, X. Lu, X. Zheng, X. Liang, Y. Fei, Y. Cheng, Y. Ni, Z. Gou, Z. Xu, Y. Luo, and C. Wu (2025) Data interpreter: an LLM agent for data science. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, p. 19796ā19821. External Links: Document, Link Cited by: Ā§2. [17] L. Jing, Z. Huang, X. Wang, W. Yao, W. Yu, K. Ma, H. Zhang, X. Du, and D. Yu (2025) DSBench: how far are data science agents from becoming data science experts?. In Proceedings of the 13th International Conference on Learning Representations, External Links: Link Cited by: Ā§2. [18] D. Zhang, S. Zhoubian, M. Cai, F. Li, L. Yang, W. Wang, T. Dong, Z. Hu, J. Tang, and Y. Yue (2025) DataSciBench: an LLM agent benchmark for data science. Note: Use cautiously in main text; the corresponding ICLR 2025 OpenReview submission was withdrawn. External Links: 2502.13897, Link Cited by: Ā§2. [19] H. Li, J. Su, Y. Chen, Q. Li, and Z. Zhang (2023) SheetCopilot: bringing software productivity to the next level through large language models. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: Ā§2. [20] H. Dong, J. Zhao, Y. Tian, J. Xiong, M. Zhou, Y. Lin, J. Cambronero, Y. He, S. Han, and D. Zhang (2024) Encoding spreadsheets for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 20728ā20748. External Links: Document, Link Cited by: Ā§2. [21] A. Su, A. Wang, C. Ye, C. Zhou, G. Zhang, G. Chen, G. Zhu, H. Wang, H. Xu, H. Chen, H. Li, H. Lan, J. Tian, J. Yuan, J. Zhao, J. Zhou, K. Shou, L. Zha, L. Long, L. Li, P. Wu, Q. Zhang, Q. Huang, S. Yang, T. Zhang, W. Ye, W. Zhu, X. Hu, X. Gu, X. Sun, X. Li, Y. Yang, and Z. Xiao (2024) TableGPT2: a large multimodal model with tabular data integration. External Links: 2411.02059, Link Cited by: Ā§2. [22] S. Chen, L. Miculicich, J. M. Eisenschlos, Z. Wang, Z. Wang, Y. Chen, Y. Fujii, H. Lin, C. Lee, and T. Pfister (2024) TableRAG: million-token table understanding with language models. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Ā§2. [23] X. Yu, P. Jian, and C. Chen (2025) TableRAG: a retrieval augmented generation framework for heterogeneous document reasoning. Note: Accepted by EMNLP 2025. External Links: 2506.10380, Link Cited by: Ā§2. [24] G. Lee, S. Kweon, S. Bae, and E. Choi (2024) Overview of the EHRSQL 2024 shared task on reliable text-to-SQL modeling on electronic health records. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, Mexico, p. 644ā654. External Links: Document, Link Cited by: Ā§2. [25] X. Yang, X. Zhao, and Z. Shen (2026) EHRStruct: a comprehensive benchmark framework for evaluating large language models on structured electronic health record tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 34340ā34348. External Links: Link Cited by: Ā§2. [26] G. Lee, E. Bach, E. Yang, T. Pollard, A. Johnson, E. Choi, Y. Jia, and J. H. Lee (2025) FHIR-AgentBench: benchmarking LLM agents for realistic interoperable EHR question answering. External Links: 2509.19319, Link Cited by: Ā§2. [27] J. P. Vandenbroucke, E. von Elm, D. G. Altman, P. C. GĆøtzsche, C. D. Mulrow, S. J. Pocock, C. Poole, J. J. Schlesselman, and M. Egger (2007) Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. PLoS Medicine 4 (10), p. e297. External Links: Document, Link Cited by: Ā§2. [28] E. I. Benchimol, L. Smeeth, A. Guttmann, K. Harron, D. Moher, I. Petersen, H. T. SĆørensen, E. von Elm, and S. M. Langan (2015) The REporting of studies conducted using observational routinely-collected health data (RECORD) statement. PLoS Medicine 12 (10), p. e1001885. External Links: Document, Link Cited by: Ā§2. [29] M. A. HernĆ”n and J. M. Robins (2016) Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology 183 (8), p. 758ā764. External Links: Document, Link Cited by: Ā§2. [30] A. G. Cashin, H. J. Hansford, M. A. HernĆ”n, S. A. Swanson, H. Lee, M. D. Jones, I. J. Dahabreh, B. A. Dickerman, M. Egger, X. Garcia-Albeniz, R. M. Golub, N. Islam, S. Lodi, M. Moreno-Betancur, S. Pearson, S. Schneeweiss, M. K. Sharp, J. A. C. Sterne, E. A. Stuart, and J. H. McAuley (2025) Transparent reporting of observational studies emulating a target trialāthe TARGET statement. JAMA 334 (12), p. 1084ā1093. External Links: Document, Link Cited by: Ā§2. [31] Observational Health Data Sciences and Informatics (2026) OMOP common data model. Note: https://ohdsi.github.io/CommonDataModel/Accessed: 2026-05-02 Cited by: Ā§2. [32] Health Level Seven International (2019) FHIR release 4.0.1: fast healthcare interoperability resources. Note: https://w.hl7.org/fhir/Accessed: 2026-05-02 Cited by: Ā§2. [33] U.S. Food and Drug Administration (2023) Data standards for drug and biological product submissions containing real-world data: guidance for industry. Note: https://w.fda.gov/regulatory-information/search-fda-guidance-documents/data-standards-drug-and-biological-product-submissions-containing-real-world-dataFinal guidance, December 2023. Accessed: 2026-05-02 Cited by: Ā§2. [34] D. Li, Y. He, Y. Hu, Y. Tian, and J. Li (2026) Can LLM agents generate real-world evidence? evaluating observational studies in medical databases. External Links: 2603.22767, Link Cited by: Ā§2. [35] N. Gim, I. Gim, Y. Jiang, Y. Kihara, M. Blazes, Y. Wu, C. S. Lee, and A. Y. Lee (2026) An LLM-assisted framework for accelerated and verifiable clinical hypothesis testing from electronic health records. medRxiv. Note: Preprint. External Links: Document, Link Cited by: Ā§2. [36] J. Clore, K. J. Cios, J. P. DeShazo, and B. Strack (2014) Diabetes 130-US hospitals for years 1999ā2008. UCI Machine Learning Repository. External Links: Document, Link Cited by: Ā§4. [37] B. Strack, J. P. DeShazo, C. Gennings, J. L. Olmo, S. Ventura, K. J. Cios, and J. N. Clore (2014) Impact of HbA1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed Research International 2014, p. 781670. External Links: Document, Link Cited by: Ā§4. [38] Centers for Disease Control and Prevention, National Center for Health Statistics (2020) NHANES 2017ā2018 questionnaires, datasets, and related documentation. Note: https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017Accessed: 2026-05-03 Cited by: Ā§4. [39] National Cancer Institute Genomic Data Commons (2026) TCGA-LIHC: liver hepatocellular carcinoma clinical metadata. Note: https://portal.gdc.cancer.gov/projects/TCGA-LIHCAccessed: 2026-05-03 Cited by: Ā§4. Appendix A Appendix Roadmap, Navigation, and Main-Paper Coverage The appendix contains both claim-support material for the main paper and standalone audit material that is not fully described in the main text. Table 10 gives a complete section-level map of the supplement. Table 11 then maps main-paper claims to their most relevant supporting material. Table 10: Complete appendix navigation. The map covers all supplement sections, including audit and interface material that is not tied to a specific main-paper table or figure. Appendix section Contents Purpose in the supplement App. A Roadmap, complete navigation, and main-paper crosswalk. Orients readers to both the full supplement and the subset of material that directly supports main-paper claims. App. B HCC benchmark scope, release boundary, task construction, Public-RWS-18, claim policies, and worksheet inventory. Documents the datasets, task instances, and schema constraints needed to reproduce or audit the benchmark. App. C Profile schema, concrete profile instance, RWS-IR checks, evidence certificates, validators, artifact bundle, coding harness, and DAG reporting. Specifies the executable contract objects and implementation checks behind ProCARE. App. D Execution environment, cost accounting, baseline adaptation, scaffolded fairness controls, ablations, model access, and repository commands. Records experimental controls and reproducibility details. App. E Claim-diagnostic interpretation and backbone-sensitivity interpretation. Explains claim-policy downgrades and performance differences across backbones. App. F Dashboard, profile panel, RWS-IR view, coding loop, report review, and section-wise report generation screenshots. Supplies workflow/interface evidence showing where artifacts, traces, logs, and evidence-linked report sections are inspected. App. G Metric definitions, deterministic validator rules, LLM judge protocol, expert-review protocol, raw expert-data schema, reliability, and rubric anchors. Defines how automated and human evaluation scores are produced and audited. App. H Benchmark, IR compiler, coding harness, report/certificate, and judge prompt templates. Makes the generation and scoring prompts inspectable. App. I Data release, privacy, responsible-use caveats, and deployment limits. Clarifies the de-identified HCC release, public-data sources, and how the system should be interpreted. Table 11: Main-paper claims and corresponding supplementary support. This crosswalk is not the full appendix map; complete section navigation is provided in Table 10. Main-paper location Claim or reference Supplementary support Sec. 3.1 The heterogeneous profile records structural and knowledge-driven constraints, with a concrete serialization and de-identified profile instance. App. C.1 and App. C.2. Sec. 3.2 RWS-IR compilation and execution enforce schema, semantic-role, temporal, grain, method, and artifact constraints. App. C.3, App. C.5, and App. C.10. Sec. 3.3 Evidence certificates and claim policies gate factual report claims, while a DAG orders supported report sections. App. C.6, App. C.9, and App. C.11. Sec. 4 The primary benchmark contains 36 HCC tasks over four objectives and a 54-sheet longitudinal eCRF workbook. App. B and Table 23. Sec. 4ā5 Public-RWS-18 uses three public datasets, dataset-specific claim policies, and the same validators and rubrics. App. B.6. Main Table 3 LLM-agent, table-analysis, and code-agent baselines are evaluated under a unified prompt, budget, artifact contract, and post-hoc checker. App. D.2. Main Table 5 Ablations remove one contract mechanism at a time while keeping the rest of the workflow fixed when defined. App. D.4. Main Table 6 Claim diagnostics are computed over factual report claims with trace and policy checks. App. E.1. Main Table 7 Automated metrics are compared with expert review. App. G.4. Main Figure 2 Stage-wise performance varies by objective and prompt specificity. App. B and App. G. Main Table 9 and Sec. 5.2 Backbone sensitivity uses fixed workflow components; model metadata and amortized token cost are reported. App. D.5 and App. D.1. Appendix B Benchmark Datasets and Tasks This section gives the benchmark details not repeated in the main paper: release contents, worksheet organization, task instantiation, public-data transfer tasks, and the worksheet inventory used by the validators. B.1 Supplemental schema scope The main paper summarizes the cohort size and high-level eCRF scope. Here we record the schema properties needed to reproduce profiling and task construction. The workbook is multi-table by design, spanning enrollment, baseline characterization, time-varying interventions, follow-up, and survival outcomes; this structure is what drives the join, grain, temporal-anchor, and claim-policy checks used throughout the benchmark. B.2 Release boundary and de-identification The public benchmark release contains the fully de-identified HCC record-level tables used in the 36 primary tasks, together with task specifications, prompt files, schema and profile artifacts, evaluation scripts, aggregate profile examples, checksums, and a data dictionary. Before release, direct identifiers and fields that could identify individual participants are removed from the benchmark copy. Random benchmark subject and visit identifiers are retained only to preserve within-dataset linkage, visit order, and longitudinal analysis structure. The released tables are intended for reproducibility of the benchmark and do not constitute a clinical deployment dataset. Table 12: HCC benchmark release contents and privacy treatment. Release item Included material Privacy and reproducibility treatment De-identified HCC tables 54 linked eCRF sheets covering the 1,294-subject benchmark cohort. Direct participant identifiers are removed; random benchmark IDs preserve subject-level and visit-level linkage required for rerunning all tasks. Task and prompt files The 36 primary HCC benchmark prompts and task metadata. Prompts refer only to de-identified table and field names and do not expose participant identities. Schema/profile artifacts Field dictionaries, join keys, temporal roles, missingness summaries, and profile objects. Contains schema and aggregate metadata used by validators; no direct identity fields are present. Evaluation assets Validators, judge prompts, artifact templates, scoring scripts, and expected output schema. Enables independent reruns against the released de-identified tables. Release metadata Data dictionary, checksums, and responsible-use terms. Documents provenance, file integrity, de-identification scope, and restrictions against re-identification or clinical use. B.3 Functional groupings Table 13 groups worksheets by their role in profiling and benchmark construction. Table 13: Functional worksheet groups used by profiling and benchmark construction. Group Representative sheets Main contents Use in benchmark/profile System metadata SYSVAR, ECRF, DATA_RANGE Form catalog, site allocation, variable descriptions, and codebook fields. Schema parsing, worksheet indexing, and field-name grounding. Baseline DM, PH, CP, BCLC, ECOG Demographics, history, liver function, staging, and performance status. Cohort definition, baseline table construction, and confounder checks. Longitudinal labs LB_AFP, LB_B, LB_BR, LB_T Tumor markers, biochemistry, complete blood count, and coagulation tests. Safety proxy signals and follow-up burden measures. Treatment TACE, TART, IMU, PATE, CM Procedures, systemic therapy, retreatment, and concomitant medication records. Exposure definition, treatment intensity, and resource-utilization tasks. Outcomes VT, AE_HQ, SAE_HQ, SF Recurrence and survival proxies, AE/SAE screening, and quality-of-life forms. Endpoint derivation and limitation checks. B.4 Dataset limitations for benchmark design Several data properties directly shape the benchmark tasks. Adverse-event screening pages are available, but detailed AE/SAE records are sparse in the released workbook; safety tasks therefore distinguish screening-level summaries from detailed event-term analyses. The dataset does not contain direct billing or price fields, so economics-oriented tasks are framed as resource utilization, follow-up burden, or quality-of-life proxy analyses rather than cost-effectiveness studies. B.5 Benchmark task construction The benchmark crosses four RWS objectives, three core questions per objective, and three prompt-specificity levels: |ā¬|=||Ć||Ć|ā|=4Ć3Ć3=36.|B|=|O|Ć|Q|Ć|L|=4Ć 3Ć 3=36. Table 14 defines the prompt levels used across all objectives. For each metric in Main Figure 2, an objectiveāprompt cell averages the three benchmark instances for that objective at the specified prompt level. A fixed prompt level therefore contains twelve instances across the four objectives; the full benchmark contains all 36 instances across the three levels. Table 14: Prompt specificity levels in the benchmark. Level Information provided Expected behavior GOAL High-level study intent only. Select feasible variables and disclose assumptions. GOAL+APPROACH Study intent plus an analysis strategy. Follow the strategy when the data support it; otherwise report the deviation. GOAL+METHOD Specific variables, time windows, or statistical method. Stop or report infeasibility when required data are missing. All instances use the same output contract: a verification plan, executable analysis entry point, generated artifacts, and a structured RWS report. Table 15 lists the twelve core questions, and Tables 16ā19 give the corresponding 36 prompts. Table 15: Core research questions used to instantiate benchmark prompts. Objective ID Core research question Primary data sources Cohort C1 Characterize the analyzable HCC cohort, including enrollment, baseline characteristics, and clinical staging. CO, DM, IE, SG, BCLC, CP, ECOG Cohort C2 Assess longitudinal follow-up completeness and data density. CPH_FT, VT, LB_AFP, LB_B, LB_BR, ECOG Cohort C3 Assess data readiness for downstream endpoint analyses. DM, SG, BCLC, CP, ECOG, VT, treatment sheets Safety S1 Summarize AE/SAE screening rates and the availability of detailed event records. AE_HQ, SAE_HQ, AE, SAE Safety S2 Describe postoperative laboratory safety signals from liver function, hematology, and coagulation panels. LB_B, LB_BR, LB_T, SG Safety S3 Compare safety proxy signals after major treatment exposures. TACE, TART, IMU, PATE, AE_HQ, SAE_HQ, lab sheets Efficacy E1 Analyze recurrence and recurrence-free survival proxies after surgery. SG, VT, BCLC, CP, ECOG, treatment sheets Efficacy E2 Analyze overall survival proxies and their relationship to treatment exposure. SG, VT, DM, BCLC, CP, ECOG, treatment sheets Efficacy E3 Analyze response assessment and longitudinal tumor-marker changes. VT, LB_AFP, AFP2, BCLC, treatment sheets Health economics/resource use H1 Describe postoperative treatment resource utilization and treatment intensity. TACE, TART, IMU, PATE, CM, SG Health economics/resource use H2 Estimate follow-up and testing burden as a resource-use proxy. CPH_FT, VT, LB_AFP, LB_B, LB_BR, LB_T, ECOG, SF Health economics/resource use H3 Use SF-12 availability and treatment intensity to assess QoL/resource-use feasibility. SF, TACE, TART, IMU, PATE, VT Table 16: Cohort benchmark instances. Suffixes denote prompt specificity: G = GOAL, A = GOAL+APPROACH, and M = GOAL+METHOD. ID Prompt text Required or expected evidence C1-G Characterize the baseline hepatocellular carcinoma cohort available in the workbook. Deliver a cohort summary suitable for a real-world study report. Discover DM, SG, BCLC, CP, ECOG, and subject identifiers. C1-A Characterize the baseline HCC cohort. Build a STROBE-style cohort table covering demographics, surgery, liver function, tumor stage, and performance status. Use patient-level baseline records, report missingness, and avoid duplicate subject counting. C1-M Produce a baseline cohort table for subjects with valid Subject_Id. Join DM, SG, BCLC, CP, and baseline ECOG; summarize AGE, SEX, HEIGHT, WEIGHT, SGDAT, SGTYP, BCLC, CPRES, CPGRA, and ECOG. Use one baseline row per subject; report variable-level missingness; do not impute incomplete AGE. C2-G Assess whether the dataset has sufficient longitudinal follow-up for real-world evidence generation. Identify visit-level forms and repeated measurements. C2-A Assess longitudinal follow-up completeness across follow-up forms, laboratory panels, tumor-marker tests, ECOG assessments, and efficacy follow-up records. Use CPH_FT, VT, laboratory sheets, and ECOG; produce visit-level summaries. C2-M Quantify longitudinal data completeness using CPH_FT, VT, LB_AFP, LB_B, LB_BR, and ECOG; count distinct visits by Subject_Id, tabulate availability by Visit_No, and estimate follow-up duration from SG.SGDAT to the latest VT.VTDAT1 or VT.VTDAT2. Use Subject_Id and Visit_No; report subjects with no valid follow-up date. C3-G Evaluate whether the cohort is fit for downstream RWS endpoint analyses. Reason about exposure, outcome, confounder, and time-anchor variables. C3-A Evaluate endpoint-analysis readiness through a feasibility matrix covering identifiers, surgery date, treatment exposure, recurrence/death endpoints, and baseline confounders. Include data limitations, not only descriptive counts. C3-M Build a data-readiness matrix by checking Subject_Id, SG.SGDAT, recurrence dates VT.PADHDAT/VT.PADRDAT, death dates VT.DEDAT1/VT.DEDAT2, treatments from TACE, TART, IMU, PATE, and confounders DM.SEX, DM.AGE, BCLC.BCLC, CP.CPGRA, ECOG.ECOG. Flag sparse or unavailable fields, especially VT.DEDAT2; do not invent endpoint fields. Table 17: Safety benchmark instances. ID Prompt text Required or expected evidence S1-G Analyze the adverse-event and serious-adverse-event safety profile in the cohort. Discover that AE/SAE screening pages exist but detailed event tables are empty in the current workbook. S1-A Analyze AE and SAE safety reporting. Start with screening pages, then check whether detailed AE/SAE records are available for term, onset date, severity, causality, outcome, and seriousness. Document infeasibility of detailed AE/SAE term analysis when detail tables have no valid records. S1-M Summarize AE/SAE reporting completeness using AE_HQ.AEYN and SAE_HQ.SAEYN; inspect AE.AETERM, AE.AESTDAT, AE.AESEV, AE.AEREL, AE.AESAE, SAE.SAESTDAT, and SAE.SAESEV. Compute screening prevalence by subject; terminate detail-level analysis if AE and SAE contain zero valid records. S2-G Identify laboratory safety signals after surgery. Use longitudinal laboratory sheets rather than relying only on AE tables. S2-A Identify post-surgical laboratory safety signals by summarizing liver function, hematology, and coagulation markers over follow-up visits. Use lab value/result-judgment pairs and visit dates; report abnormal-result flags when available. S2-M Generate a post-surgical laboratory safety profile anchored at SG.SGDAT; use LB_B.LBBBDAT, ALTRES, ASTRES, TBILRES, ALBRES; LB_BR.BRDAT, WBCRES, HBRES, PLTRES; and LB_T.LBTTDAT, PTRES, INRRES, DDIMRES. Summarize baseline and postoperative windows; use available-case denominators and _SIG flags when numeric thresholds are unavailable. S3-G Compare safety patterns across major postoperative treatment exposures. Identify TACE, targeted therapy, immunotherapy, and retreatment exposure sheets. S3-A Compare safety proxy signals by treatment exposure using TACE, targeted therapy, immunotherapy, and postoperative retreatment groups. Compare AE/SAE screening and laboratory abnormalities descriptively; avoid causal claims. S3-M Compare treatment-associated safety proxies using exposure indicators from TACE.TACEDAT, TART.TARSDAT, IMU.IMUSDAT, and PATE.PATEYN; calculate screening rates from AE_HQ.AEYN and SAE_HQ.SAEYN; summarize post-exposure laboratory signals from LB_B, LB_BR, and LB_T. Report confounder imbalance using BCLC.BCLC, CP.CPGRA, and ECOG.ECOG; frame results as descriptive safety proxies. Table 18: Efficacy benchmark instances. ID Prompt text Required or expected evidence E1-G Analyze recurrence-related outcomes after surgery. Locate surgery dates and recurrence dates in VT. E1-A Analyze recurrence-free outcomes using surgery date as time origin, recurrence events from follow-up records, and clinically relevant baseline risk strata. Handle multiple follow-up visits and competing or missing dates. E1-M Estimate recurrence-free survival proxies using SG.SGDAT as index date; define recurrence as the earliest non-missing VT.PADHDAT or VT.PADRDAT; censor at latest VT.VTDAT1 or VT.VTDAT2; stratify by BCLC.BCLC, CP.CPGRA, and exposure from TACE, TART, IMU, or PATE. Use earliest valid recurrence date and valid time ordering; generate KaplanāMeier curves only when dates are sufficient. E2-G Analyze overall survival patterns in the cohort. Identify death dates and follow-up dates. E2-A Analyze overall survival by deriving time from surgery to death or last follow-up and comparing clinically meaningful strata. Use DEDAT1/DEDAT2 and VT follow-up dates; disclose death-date limitations. E2-M Estimate overall-survival proxies using SG.SGDAT as index date; define death date as earliest non-missing VT.DEDAT1 or VT.DEDAT2; censor at latest valid VT.VTDAT1 or VT.VTDAT2; compare treatment exposures and adjust or stratify descriptively by DM.SEX, DM.AGE, BCLC.BCLC, CP.CPGRA, and ECOG.ECOG when available. Note that DEDAT2 is unavailable in the current data; avoid over-adjusted models when events are limited. E3-G Evaluate treatment response and tumor-marker dynamics over follow-up. Combine efficacy assessment and biomarker sheets. E3-A Evaluate response and tumor-marker dynamics by summarizing hospital/telephone efficacy evaluations and longitudinal AFP, CA199, and DCP trajectories by visit and exposure. Combine categorical response information with biomarker trends. E3-M Analyze response and tumor-marker trends using VT.CEVYN, VT.CUEV, VT.VTDAT1, VT.VTDAT2, plus LB_AFP.LBAFPDAT, AFPRES, CA199RES, and DCPRES; align measurements by Subject_Id and date relative to SG.SGDAT; summarize trajectories by exposure to TACE, TART, or IMU. If formal response criteria are not encoded, report observed categories and marker changes without RECIST claims. Table 19: Health economics and resource-use benchmark instances. ID Prompt text Required or expected evidence H1-G Analyze postoperative treatment resource utilization in the cohort. Recognize that the task concerns utilization rather than direct cost. H1-A Analyze postoperative treatment resource utilization by quantifying use and timing of TACE, targeted therapy, immunotherapy, postoperative retreatment, and concomitant medications. Use procedure and drug counts with dates as utilization proxies. H1-M Quantify treatment-utilization intensity by deriving per-subject counts and time-to-first-use for TACE.TACEDAT, TART.TARSDAT, IMU.IMUSDAT, PATE.PATEYN/PATETR*, and CM.CMCAT/CMSTDAT; summarize utilization distributions and stratify by BCLC.BCLC and recurrence status when feasible. Explicitly state that no direct cost or billing variables are present. H2-G Estimate follow-up and testing burden associated with the RWS cohort. Use visit and form/test counts as burden proxies. H2-A Estimate follow-up and testing burden by counting follow-up visits, efficacy assessments, laboratory panels, tumor-marker checks, coagulation tests, ECOG assessments, and QoL forms per patient. Summarize burden by patient and visit. H2-M Build a follow-up resource-burden profile by counting records in CPH_FT, VT, LB_AFP, LB_B, LB_BR, LB_T, ECOG, and SF; compute total follow-up forms/tests, median counts per subject, and distribution by Visit_No; use SG.SGDAT and latest VT.VTDAT1/VT.VTDAT2 to contextualize burden over time. Frame the result as utilization burden, not economic cost. H3-G Explore quality-of-life evidence and its relationship to treatment burden. Discover SF-12 sparsity and avoid overclaiming. H3-A Explore QoL and treatment burden using SF-12 items when available; summarize missingness and compare available QoL responses across treatment-utilization groups. Emphasize sparse SF item completeness. H3-M Analyze QoL/resource-use proxy feasibility by inspecting SF.SFYN and SF1āSF12; compute complete-item availability; when enough records exist, summarize item-level or total scores by exposure groups from TACE, TART, IMU, and PATE; otherwise report feasibility and missingness only. Terminate formal QoL comparison when usable SF-12 item records are too sparse. B.6 Public-RWS external validation suite Public-RWS-18 is an external validation suite for testing whether the same profileāIRācertificate pipeline transfers to public health datasets with different schemas and study designs. It is not merged with the 36-instance HCC benchmark and is not treated as another full longitudinal RWS benchmark. Each public dataset contributes two task objectives, and each objective is evaluated under low-, medium-, and high-specificity prompts. Table 20: Public-RWS external validation datasets. These datasets test cross-dataset transfer and contract diagnostics, not additional full HCC-style longitudinal RWS validity. Dataset Structure Task focus Boundary UCI Diabetes 130-US Hospitals 101,766 encounters, 50 columns; the department-split workbook is used only as a sheet-navigation variant. Readmission proxy, medication process, utilization, coded variables, missingness. Encounter-level data with limited longitudinal structure. NHANES 2017ā2018 subset 10 linked XPT/CSV tables with 3,036ā19,643 rows per table, linked by SEQN. Survey cohort construction, labs, medications, conditions, demographic covariates. Cross-sectional survey data, not EHR. TCGA-LIHC clinical metadata 377 cases, 137 clinical metadata columns. HCC metadata, vital status, treatment fields, follow-up proxy fields. Cancer registry-like clinical metadata, not routine-care EHR. Table 21: Task objectives in Public-RWS-18. Each objective is evaluated under low-, medium-, and high-specificity prompts. Dataset Objective Contract stressor UCI Diabetes Readmission proxy profiling Outcome binding, coded variables, missingness, conservative claim scope. UCI Diabetes Medication and utilization analysis Exposure/proxy roles, subgroup aggregation, artifact traceability. NHANES Diabetes/cardiometabolic cohort analysis Multi-table joins by SEQN, lab-defined variables, survey-table grounding. NHANES Medication and condition association analysis Questionnaire, laboratory, and prescription-medication grounding under a cross-sectional claim policy. TCGA-LIHC HCC case and vital-status profiling Domain transfer, oncology metadata grounding, missingness disclosure. TCGA-LIHC Treatment/follow-up metadata analysis Sparse fields, proxy endpoint handling, limitation reporting. Table 22: Dataset-specific claim policies for Public-RWS-18. Dataset Allowed claims Forbidden claims UCI Diabetes Descriptive and readmission-proxy association claims. Causal treatment effect, safety conclusion, longitudinal endpoint claim. NHANES Descriptive and cross-sectional association claims. Temporal effect, incidence claim, causal effect. TCGA-LIHC Descriptive oncology metadata and follow-up-proxy claims. Treatment efficacy, causal survival effect, routine-care EHR claim. Public-RWS-18 uses the same profile construction, RWS-IR schema, validators, and claim checker as the HCC benchmark. Dataset-specific preprocessing is limited to loading the public tables, normalizing file and column names for stable tool access, preserving original public codebook variables, and constructing profile metadata. No public dataset receives a custom validator, claim checker, or reporting rule. The claim policy in Table 22 is the only dataset-specific adaptation, because each public source supports a different study-design boundary. Each public dataset contributes two objectives, and each objective is instantiated under three prompt-specificity levels. Dataset-level results are reported once in the main paper. Public-RWS-18 uses ReAct as a generic agent baseline and adds two stronger controls aligned with the primary baseline table: Claude Code as a native code-agent baseline, and Scaffolded G3P as the direct Gemini-3-Pro profile/template plus validator-repair control. Scaffolded G3P receives evaluator-facing schema information, artifact templates, validator feedback, and the same repair budget, but not ProCAREās IR compiler, certificate admission gate, certified reporting DAG, or internal planner state. This appendix records the dataset boundaries, task objectives, claim policies, and prompt text needed to rerun the transfer suite. For baselines without explicit evidence certificates, CTC, UCR, and CLV are computed by post-hoc evidence reconstruction over submitted artifacts and reports. Listing 1: Prompt set for Public-RWS-18. ā¬ UCI Diabetes -- Objective U1: Readmission proxy profiling Low: Analyze the diabetes hospital encounter data and identify which patient groups appear to have higher readmission burden. Medium: Using the readmitted field as a proxy outcome, summarize early-readmission patterns by age, admission type, discharge disposition, and prior utilization variables. High: Using readmitted = "<30" as the early-readmission proxy outcome, compute subgroup-level rates by age band, admission_type_id, discharge_disposition_id, number_inpatient, and number_emergency. Produce a table, one figure, and a report restricted to descriptive or proxy-association claims with evidence certificates for all factual statements. UCI Diabetes -- Objective U2: Medication and utilization analysis Low: Analyze whether medication and hospital-utilization patterns differ across diabetes encounters. Medium: Summarize insulin, change, diabetesMed, A1Cresult, time_in_hospital, num_lab_procedures, num_medications, and num_procedures, and describe any supported association with readmitted. High: Compute utilization summaries for time_in_hospital, num_lab_procedures, num_medications, and num_procedures stratified by insulin, change, and diabetesMed. Compare early-readmission proxy rates across medication-process groups, and avoid causal treatment-effect language. NHANES -- Objective N1: Diabetes/cardiometabolic cohort analysis Low: Use the NHANES tables to profile diabetes and cardiometabolic risk patterns in the population sample. Medium: Join demographics, examination, blood pressure, diabetes questionnaire, laboratory, and medical-condition tables by SEQN. Summarize diabetes-related variables and cardiometabolic measurements by age and sex. High: Construct a cross-sectional NHANES cohort using SEQN-linked tables. Define diabetes-related status using supported questionnaire or laboratory fields, summarize blood pressure and relevant laboratory measures by demographic subgroup, and report only descriptive or cross-sectional association claims. NHANES -- Objective N2: Medication and condition association analysis Low: Analyze how prescription medication records relate to self-reported medical conditions in NHANES. Medium: Join prescription medication, medical conditions, demographics, and laboratory tables by SEQN. Summarize medication exposure groups and compare supported condition or laboratory profiles. High: Using SEQN-linked medication and condition tables, construct medication-exposure groups when supported by the codebooks, summarize condition prevalence and laboratory measurements, and produce a report that explicitly states cross-sectional design limitations. TCGA-LIHC -- Objective T1: HCC case and vital-status profiling Low: Profile the TCGA-LIHC clinical metadata and summarize major patient, diagnosis, and vital-status patterns. Medium: Summarize demographics, diagnosis-related fields, vital status, and available follow-up-related fields in TCGA-LIHC. Identify missingness and fields that can support survival-proxy reporting. High: Using TCGA-LIHC clinical metadata, produce a case-profile table covering age, sex, diagnosis-related fields, vital status, days-to-death or follow-up-related fields when available, and missingness. Report only descriptive oncology-metadata claims with evidence certificates. TCGA-LIHC -- Objective T2: Treatment/follow-up metadata analysis Low: Analyze whether available treatment and follow-up metadata in TCGA-LIHC support subgroup comparisons. Medium: Identify treatment-related, diagnosis-related, and follow-up-related fields in TCGA-LIHC, assess their completeness, and perform only comparisons supported by valid profile entries. High: Using TCGA-LIHC clinical metadata, identify valid treatment-related and follow-up-related fields, summarize completeness by subgroup, and generate a conservative report that distinguishes observed metadata patterns from treatment-effect or causal claims. B.7 Worksheet inventory Table 23: Standardized worksheet inventory in the Excel dataset. Prefix #Fields #Rows Sheet Description Differential field examples SYSVAR 2 27 System variables and metadata. Variable, Description DATA_RANGE 2 4 Data range / center mapping and allocation notes. Center, RangeCode ECRF 8 54 eCRF form catalog and design attributes. FormCode, FormName, FormType CPH_FT 22 11614 Header/footer records for visit-/form-level. VISTNM, SITNO, SUBNO CO 21 1296 Study cover page and subject identification fields. SITENO, SUBJNO, SUBINI DS 19 1296 Informed consent date and related tracking fields. DSDAT DM 27 1295 Demographics and baseline characteristics. SEX, BRTHDAT, AGE PH 123 1262 Medical history and prior conditions. PHSMYN, SMNUM, SMYER PH1 123 1 Medical history (continuation / overflow record). PH1SMYN, SMNUM1, SMYER1 HBCV 49 4282 HBV/HCV-related laboratory assessments. HBVYN, HBVDAT, HBSAGRES HBCV1 49 1 HBV/HCV laboratory assessments. HBV1YN, HBV1DAT, HBSAGRE1 VS 25 1233 Vital signs measurements. VSRES, VSTEMP, VSSBP CP 32 1262 Child-Pugh assessment and component scoring. CPYN, CPRES, CPGRA IE 31 549 Eligibility criteria assessment. IEDAT, INCL1, INCL2 IE1 30 714 Eligibility criteria (alternate set).. INCL01, INCL02, EXCL01 TUBA 94 1295 Intraoperative tumor pathology / surgical findings. TUBADSC, TUBATYPE, TUBAGRA BCLC 28 1261 BCLC staging for primary liver cancer. BCLC, PLICA, PLICA1 BCLC1 28 1 BCLC staging (continuation record). BCLC1, BCPLICA, BCPLICA1 VT 96 5321 Efficacy assessment interview and follow-up. VTTYP, VTDAT1, VTDAT2 VT1 96 2 Efficacy assessment (continuation record). VT1TYP, VT1DAT1, VT1DAT2 LB_AFP 57 4613 Tumor marker laboratory tests. AFPYN, LBAFPDAT, AFPRES AFP2 26 4166 Other tumor marker laboratory tests. AFP2DAT, AFP2NM, AFP2RES LB_CD 87 4392 Lymphocyte subset testing panel. CDYN, CDDAT, CD3PRES LB_BR 93 4474 Complete blood count (CBC). BRYN, BRDAT, WBCRES LB_B 123 4583 Serum biochemistry and hepatic function panel. LBBBYN, LBBBDAT, ALTRES LB_B1 105 1 Serum biochemistry. LBBB1YN, LBBB1DAT, ALTRES1 LB_CYT 75 4325 Cytokine testing panel. LBCYTYN, LBCYTDAT, IL1RES LB_T 55 4466 Coagulation function testing panel. LBTTYN, LBTTDAT, PTRES ECOG 22 4459 ECOG performance status assessment. ECOGYN, ECOG PATE 39 3265 Postoperative retreatment information. PATEYN, PATETR1, PATETR2 SG 21 1325 Surgery-related records and key attributes. SGDAT, SGTYP RFT 75 1323 Combined radiofrequency therapy records. UEXYN, UEXLO1, UEXLO2 FIN 20 1325 Number of lesions and summary lesion count. FINM FI 74 1489 Lesion-level details (location and characteristics). FILO1, FILO2, FILO3 TRDAT 20 27 Postoperative treatment start date/time fields. TRDAT, TRDATS THET_HQ 23 123 Chemotherapy screening / question page. THETYN, THETDSC, THETUK THET 30 154 Chemotherapy administration records. THETPHA, THESDAT, THETONGO TART_HQ 20 453 Targeted therapy screening / question page. TARTYN, TARTYN_DEC TART 35 303 Targeted therapy administration records. TARTPHA, TARSDAT, TARTONGO IMU_HQ 20 404 Immunotherapy screening / question page. IMUYN, IMUYN_DEC IMU 35 290 Immunotherapy administration records. IMUPHA, IMUSDAT, IMUONGO TACE_HQ 20 615 TACE screening / question page. TACEYN, TACEYN_DEC TACE 30 609 TACE procedure and regimen details. TACEDAT, TACETYP1, TACETYP2 CMED 20 283 Traditional Chinese medicine treatment records. CMEDYN, CMEDYN_DEC RADI_HQ 52 15 Radiotherapy screening / question page. RADIYN, RADIMD1, RADIMD2 RADI 21 6 Radiotherapy administration records. RADIGN, RADIDAT, RADIDSC SF 44 3183 SF-12 quality-of-life questionnaire. SFYN, SF1, SF2 CM_HQ 20 1235 Concomitant medication screening / question. CMYN, CMYN_DEC CM 34 6632 Concomitant medication records. CMCAT, CMINDC, CMDOSE LB_BP 24 1232 Blood typing and related laboratory annotations. LBBPYN, LBBPDAT, LBBPSIG AE_HQ 20 1231 Adverse event screening / question page. AEYN, AEYN_DEC AE 34 1 Adverse event records. AETERM, AESTDAT, AEENDAT SAE 74 1 Serious adverse event records and site contact. SAECAT1, SAECAT2, SAERPDAT SAE_HQ 20 1231 Serious adverse event screening / question page. SAEYN, SAEYN_DEC LOAB_HQ 20 51 Local ablation screening / question page. LOABYN, LOABYN_DEC LOAB 21 51 Local ablation procedure records. LOABDAT, LOABMEAR, LOABMEAR_DEC Table 23: Standardized worksheet inventory in the Excel dataset (continued). Appendix C Method and Contract Implementation Details This section records the implementation artifacts used by the profiling, RWS-IR compilation, violation-guided execution, and certified reporting stages. C.1 Profile schema Ī¦ The main paper defines the four profile blocks. The implementation stores the same object as a JSON contract with stable keys for high-level structural summaries, high-level knowledge roles, field-level structural statistics, and field-level knowledge mappings. Downstream stages do not consume free-form profile prose; they read these keys directly when compiling RWS-IR fields, validating generated code, and admitting report claims. Structural profile Ī¦ā ,S _Ā·,S. For each table/sheet and column, we record deterministic statistics, including: inferred type, missingness, cardinality, value ranges, common categories, and candidate join keys. Knowledge profile Ī¦ā ,K _Ā·,K. We map raw fields to standardized biomedical concepts (e.g., UMLS/SNOMED CT/ICD/LOINC) by candidate generation followed by deterministic validation checks: (i) semantic type compatibility, and (i) value-range consistency. Unmappable or ambiguous fields are retained with flags to propagate uncertainty into RWS-IR compilation, execution, and reporting. Implementation serialization and fallback. The implementation serializes Ī¦ as a JSON object with four named blocks: high_structural, low_structural, high_knowledge, and low_knowledge. The high-structural block stores dataset-level table relationships, candidate subject and visit keys, and worksheet-level roles. The low-structural block stores field-level type inference, missingness, cardinality, range checks, common categorical values, date-parsing status, and candidate temporal anchors. The high-knowledge block stores study-level roles, including index-date schema, exposure, outcome, censoring, confounder, proxy, and claim-scope roles. The low-knowledge block stores ontology candidates, accepted mappings, validation status, units, value-range plausibility, temporal-role compatibility, and ambiguity or rejection causes. For each biomedical mapping candidate, the profiler applies semantic-type compatibility, unit consistency, value-range plausibility, and temporal-role compatibility checks. A field that fails these checks is retained in Ī¦L,K _L,K with a failure tag rather than removed. Such fields remain available for structure-grounded summaries, missingness reporting, and data-quality diagnostics, but cannot authorize clinical-role binding, endpoint derivation, safety conclusions, causal claims, or high-stakes report language unless an explicit proxy assumption is recorded in the RWS-IR and admitted by the claim policy. C.2 Concrete heterogeneous profile instance Listing 2 shows a compact de-identified JSON instance of Ī¦ from the HCC benchmark. The example follows the same schema used by the implementation and contains aggregate profile statistics and field-level metadata; record-level values are provided in the released de-identified HCC tables rather than repeated in the text. Listing 2: Compact JSON instance of the heterogeneous data profile Ī¦ . ā¬ 1 2 "serialization": "JSON", 3 "notation_map": 4 "Phi^H,S": "heterogeneous_profile.high_structural", 5 "Phi^L,S": "heterogeneous_profile.low_structural", 6 "Phi^H,K": "heterogeneous_profile.high_knowledge", 7 "Phi^L,K": "heterogeneous_profile.low_knowledge" 8 , 9 "heterogeneous_profile": 10 "high_structural": 11 "dataset_summary": 12 "workbook_type": "multi-sheet eCRF workbook", 13 "benchmark_subject_count": 1294, 14 "benchmark_visit_count": 12847 15 , 16 "sheet_inventory_subset": [ 17 18 "sheet": "DM--Demographics", 19 "role": "baseline demographics", 20 "estimated_data_rows": 1294, 21 "columns": 27, 22 "join_keys": ["Subject_Id"] 23 , 24 25 "sheet": "SG--Surgery", 26 "role": "index surgery", 27 "estimated_data_rows": 1324, 28 "columns": 21, 29 "join_keys": ["Subject_Id"] 30 , 31 32 "sheet": "VT--Follow-up", 33 "role": "postoperative follow-up", 34 "estimated_data_rows": 5320, 35 "columns": 96, 36 "join_keys": ["Subject_Id", "Visit_No"] 37 38 ], 39 "relationship_hints": [ 40 "DM.Subject_Id joins SG.Subject_Id and VT.Subject_Id.", 41 "SG.SGDAT is the preferred index date.", 42 "VT.Visit_No encodes repeated follow-up windows." 43 ] 44 , 45 "low_structural": 46 "field_profiles": 47 "DM.SEX_DEC": 48 "dtype": "categorical", 49 "missing_rate": 0.0, 50 "top_values": "male": 165, "female": 34 51 , 52 "SG.SGDAT": 53 "dtype": "date-like text", 54 "missing_rate": 0.005, 55 "semantic_hint": "index surgery date" 56 , 57 "VT.PADHDAT": 58 "dtype": "partial date-like categorical", 59 "missing_rate": 0.915, 60 "quality_note": "sparse recurrence date" 61 , 62 "VT.DEDAT2": 63 "dtype": "date-like categorical", 64 "missing_rate": 0.995, 65 "quality_note": "nearly unavailable" 66 67 68 , 69 "high_knowledge": 70 "variable_roles": 71 "id": ["DM.Subject_Id", "SG.Subject_Id", "VT.Subject_Id"], 72 "time_anchor": ["SG.SGDAT"], 73 "baseline_covariate": [ 74 "DM.AGE", "DM.SEX_DEC", "BCLC.BCLC_DEC", 75 "CP.CPGRA", "ECOG.ECOG" 76 ], 77 "outcome": [ 78 "VT.PADHDAT", "VT.PADRDAT", 79 "VT.DEDAT1", "VT.DEDAT2" 80 ] 81 , 82 "feasibility_assessment": "Cohort and follow-up summaries are feasible; survival analyses require an availability check." 83 , 84 "low_knowledge": 85 "ontology_mapping": 86 "DM.SEX_DEC": "Biological sex", 87 "DM.AGE": "Age", 88 "SG.SGDAT": "Surgical procedure date", 89 "VT.PADHDAT": "Recurrence date" 90 , 91 "validation_status": 92 "DM.HEIGHT": "ambiguous: numeric values mixed with unknown tokens", 93 "VT.PADHDAT": "valid_but_sparse", 94 "VT.DEDAT2": "invalid_for_primary_survival_endpoint" 95 , 96 "analysis_constraints": [ 97 "Use only fields listed in field_profiles.", 98 "Treat unknown or partial date tokens as missing unless the method supports interval censoring.", 99 "Do not infer recurrence or death outcomes from missing date fields." 100 ] 101 , 102 "ambiguity_flags": [ 103 "DM.HEIGHT", "DM.WEIGHT", 104 "VT.PADHDAT", "VT.DEDAT2" 105 ] 106 107 C.3 RWS-IR schema and static checks The RWS-IR is the typed study object validated before code generation. It records the study design, analysis grain, join graph, entity keys, temporal anchors, variable roles, methods, required artifacts, and claim policy. Table 24 lists the machine-readable fields and the admission checks applied before execution. Table 24: Machine-readable fields in the RWS-IR and static checks before execution. Symbol Serialized key Contents Static admissibility checks D study_design Study type, estimand boundary, and whether the requested output is descriptive, associative, or inferential. Must be compatible with available tables, observed design, and claim policy. G analysis_grain Subject-, visit-, encounter-, field-, or record-level analysis unit. Must match key multiplicities and downstream aggregation. J join_graph Tables, join keys, cardinalities, and permitted join paths. All tables and keys must exist in Ī¦L,S _L,S; declared cardinality must match profiled relationships. K entity_keys Subject, visit, encounter, treatment, and artifact identifiers. Required keys must be present and nondegenerate under the declared grain. T temporal_anchors Index date, event dates, censoring dates, windows, and ordering rules. Date fields must be parseable or declared as partial/proxy dates; impossible orderings are disallowed. V variable_roles Exposure, outcome, censoring, covariate, subgroup, proxy, and reporting variables. Clinical roles must be supported by Ī¦L,K _L,K, a validated proxy flag, or a recorded assumption. M method Statistical method, fallback method, denominators, missingness policy, and diagnostics. Method must be compatible with design, grain, variable support, and minimum data requirements. A required_artifacts Required result tables, figures, manifests, logs, and report sections. Artifacts must be executable in the sandbox and verifiable by manifests. Ī q _q claim_policy Allowed and forbidden claim types, language constraints, and support levels. Claim policy must be no stronger than the design and evidence support. Given a candidate Z0Z_0, IRCheckstatic returns an accepted RWS-IR Z and a set of static violations UstaticU_static. A hard violation is any reference to a missing table, missing field, invalid key, unsupported clinical role, incompatible analysis grain, impossible temporal anchor, method that requires unavailable variables, or claim policy exceeding the design. A soft violation is a downgradeable limitation, such as sparse endpoint support, partial date availability, missing units, or a proxy variable that can support descriptive but not causal claims. Only Ustatic=ā U_static= enters the execution workspace. When hard violations remain, ProCARE either repairs the RWS-IR, emits an infeasibility statement, or downgrades the requested analysis according to prompt specificity. Table 25: Static violation types emitted by IRCheckstatic. Violation type Trigger Allowed repair Missing schema reference A table, field, or key in Z0Z_0 is absent from Ī¦L,S _L,S. Replace with a profiled field, remove the variable, or declare infeasibility for Goal+Method. Unsupported semantic role Exposure, endpoint, censoring, or confounder role is not validated in Ī¦L,K _L,K. Use a validated proxy with limitation, downgrade to descriptive support, or stop. Invalid join or grain Join cardinality or aggregation conflicts with the declared analysis grain. Change grain, revise the join graph, or aggregate before merging. Invalid temporal anchor Index, event, censoring, or follow-up dates are missing, partial, nonparseable, or ordered incorrectly. Use a supported anchor, restrict to analyzable records, or report feasibility only. Method infeasibility The requested method requires unsupported design assumptions, insufficient events, or unavailable covariates. Switch to a supported fallback method or emit an infeasibility statement. Claim-policy excess Ī q _q allows causal, safety-critical, economic-cost, or generalization claims unsupported by the data. Restrict Ī q _q to descriptive, feasibility, or proxy-association claims. C.4 RWS-IR and evidence certificate examples Listing 3 gives a compact RWS-IR instance for a recurrence-free survival task. The example shows the fields used by the static checker: study design, analysis grain, join graph, temporal anchors, variable-role bindings, method assumptions, required artifacts, and claim policy. Listing 3: Compact RWS-IR example for a recurrence-free survival task. ā¬ 1 2 "study_design": "retrospective_cohort", 3 "analysis_grain": "subject", 4 "population": 5 "source_tables": ["DM", "SG", "VT"], 6 "subject_key": "Subject_Id", 7 "inclusion": ["non_missing(DM.Subject_Id)", "surgery_record_exists(SG)"] 8 , 9 "join_graph": [ 10 "left": "DM.Subject_Id", "right": "SG.Subject_Id", "type": "one_to_one", 11 "left": "DM.Subject_Id", "right": "VT.Subject_Id", "type": "one_to_many" 12 ], 13 "temporal_anchors": 14 "index_date": "SG.SGDAT", 15 "event_date_candidates": ["VT.PADHDAT", "VT.PADRDAT"], 16 "censoring_date_candidates": ["VT.VTDAT1", "VT.VTDAT2"] 17 , 18 "variable_roles": 19 "exposure": "surgery cohort", 20 "outcome": "recurrence-free survival proxy", 21 "covariates": ["DM.AGE", "DM.SEX_DEC", "BCLC.BCLC_DEC", "CP.CPGRA"] 22 , 23 "method": 24 "primary": "Kaplan-Meier summary if temporal checks pass", 25 "fallback": "descriptive follow-up availability when event dates are sparse" 26 , 27 "required_artifacts": ["cohort_flow", "followup_summary", "km_curve_or_feasibility_table"], 28 "claim_policy": 29 "allowed": ["descriptive", "association_with_limitation"], 30 "blocked": ["causal_effect", "safety_detail", "submission_grade_claim"] 31 32 Listing 4 gives a representative evidence certificate. The certificate records the claim text, its support level, the validated artifacts and code region that produced it, the RWS-IR fields that authorize the claim, and the profile entries or assumptions used by the admission rule. Listing 4: Evidence certificate example for an admitted factual claim. ā¬ 1 2 "claim_id": "q_rfs_03", 3 "claim_text": "Recurrence-free survival can be summarized for subjects with valid surgery and follow-up dates.", 4 "claim_type": "descriptive_feasibility", 5 "support_level": "profile_and_artifact_supported", 6 "trace": 7 "artifact": "results/followup_summary.json", 8 "figure": "figures/km_or_feasibility.pdf", 9 "code_region": "analysis.py::build_rfs_dataset", 10 "rws_ir_fields": [ 11 "temporal_anchors.index_date", 12 "temporal_anchors.event_date_candidates", 13 "method.primary", 14 "claim_policy.allowed" 15 ], 16 "profile_entries": ["SG.SGDAT", "VT.PADHDAT", "VT.PADRDAT", "VT.VTDAT1", "VT.VTDAT2"], 17 "assumptions": ["partial or unknown date tokens are treated as missing"] 18 , 19 "admission": 20 "trace_check": true, 21 "policy_check": true, 22 "language_check": true 23 24 C.5 Execution validators and violation-guided repair Execution-time checks are applied after code generation because row-level temporal ordering, manifest completeness, figure creation, and artifact consistency cannot be fully resolved statically. Validators return a localized violation set Ut=Vexecā(Zt,Ct,Ot,Et,Ī¦,St)U_t=V_exec(Z_t,C_t,O_t,E_t, ,S_t). Each violation records the failed contract clause, offending code or artifact, nearest valid profile entries, severity, and permitted repair actions. Listing 5 shows the JSON schema used for one violation record. Listing 5: Schema for one execution-time contract violation. ā¬ 1 2 "violation_id": "u_017", 3 "stage": "execution", 4 "contract_clause": "temporal_anchor.index_date_before_event", 5 "severity": "hard", 6 "offending_artifact": "analysis.py", 7 "code_region": "build_survival_dataset", 8 "observed_failure": "event_date precedes SG.SGDAT for at least one record", 9 "nearest_profile_entries": ["SG.SGDAT", "VT.PADHDAT", "VT.PADRDAT"], 10 "permitted_actions": ["filter_invalid_temporal_pairs", "report_temporal_limitation"], 11 "blocks_acceptance": true 12 Table 26: Execution validators used before accepting an artifact bundle. Validator Check Acceptance criterion Entry-point validator Runs analysis.py or main.py in the sandbox. Exit code is zero and no failure sentinel such as TASK_FAILED appears. Schema validator Parses code and generated outputs for table, field, and key usage. Every referenced table, field, join key, and output column is traceable to Ī¦L,S _L,S or a derived artifact. Grain and join validator Checks duplicate inflation, one-to-many joins, and final denominator consistency. Output denominators match the declared analysis grain or disclose aggregation rules. Temporal validator Recomputes index, event, censoring, and follow-up windows. Invalid date strings and impossible orderings are filtered, logged, or reported as limitations. Method validator Checks cohort filters, variables, denominators, covariates, and statistical choices against Z.MZ.M. Method matches the admitted RWS-IR or a declared fallback. Artifact validator Checks result and figure manifests, file existence, and generated figure formats. Required tables, figures, manifests, and logs exist and are nonempty when applicable. Claim-preparation validator Builds the evidence map used by the report generator. Each reportable statistic has a source artifact, producing code region, and profile/IR trace. Algorithm 1 Execution acceptance and localized repair 1: Accepted RWS-IR Z, profile Ī¦ , workspace policy Ī , maximum attempts B 2: Accepted code CāC^*, artifacts O, evidence E, assumptions A, or failure trace 3: for t=1,ā¦,Bt=1,ā¦,B do 4: Generate or edit code CtC_t conditioned on Z, Ī¦ , Ī , and prior violations Utā1U_t-1. 5: Execute the selected entry point in the sandbox and collect logs, result manifests, and figure manifests. 6: Compute Ut=Vexecā(Z,Ct,Ot,Et,Ī¦,St)U_t=V_exec(Z,C_t,O_t,E_t, ,S_t). 7: if Ut=ā U_t= then 8: Accept Cā=CtC^*=C_t and return (Cā,Ot,Et,At)(C^*,O_t,E_t,A_t). 9: else 10: Write UtU_t to violation_set.json with localized repair actions. 11: end if 12: end for 13: Return the final failure trace or a supported downgrade with explicit limitations. C.6 Evidence certificates and claim admission A factual claim is any report sentence that asserts a dataset property, cohort count, variable availability, statistical result, visualization interpretation, clinical association, feasibility conclusion, or limitation derived from the data. Section headings, procedural statements, and generic background statements are not counted unless they contain dataset-specific factual content. For ProCARE, certificates are emitted during report generation. For baselines without explicit certificates, the same checker performs post-hoc reconstruction from the submitted code, logs, artifacts, and report. Each claim q receives a certificate eq=āØoq,cq,zq,Ļq,aqā©e_q= o_q,c_q,z_q, _q,a_q , where oqo_q is a supporting artifact, cqc_q is the executed code region that created the artifact, zqz_q is the RWS-IR field or method authorizing the claim, Ļq _q is the relevant profile entry, and aqa_q is an optional recorded assumption or limitation. The trace check passes only when all required links exist and agree on variables, denominators, analysis grain, temporal windows, and method. The policy check passes only when the claim type and wording are allowed by Z.Ī qZ. _q at the available support level. Table 27: Claim types used by the certificate gate. Claim type Allowed support Examples of blocked wording Descriptive Validated table, count, summary statistic, figure, or profile entry. Claims that imply generalization beyond the analyzed cohort. Feasibility or data-quality Profile entries, missingness summaries, temporal-validity logs, or execution artifacts. Treating missingness or feasibility as clinical outcome evidence. Proxy association Valid proxy flag, artifact support, and explicit limitation. Causal treatment-effect language or safety conclusion language. Causal or comparative effect Requires design assumptions, adjustment plan, and method support declared in Z. Causal language when only descriptive or proxy support is available. Safety-critical or economic-cost Requires detailed safety records or direct cost/billing fields. Detailed AE/SAE term claims when only screening pages exist; cost-effectiveness claims without cost variables. Let NqN_q be the number of extracted factual claims, CqC_q the number with admitted evidence certificates, UqU_q the number without admissible evidence, and LqL_q the number whose language exceeds the claim policy. We compute CTC=Cq/Nq,UCR=Uq/Nq,CLV=Lq/Nq.CTC=C_q/N_q, =U_q/N_q, =L_q/N_q. A claim can be both unsupported and a language violation, but each metric is normalized by the same factual-claim denominator. For the claim-diagnostic table in the main paper, metrics are computed per run and then averaged across the 12 tasks in each prompt-specificity group; the overall row averages across all 36 HCC tasks. C.7 Profile-grounded state artifacts The workflow state is represented by typed JSON artifacts, workspace files, skills, and validation logs. No vector database is used. Each downstream module reads the relevant state artifact from named files or typed prompt payloads, so intermediate decisions remain auditable without re-querying raw tables. Concretely, the persistent dataset state is the cached profile Ī¦ , while the per-job state tS_t may contain excel_info.json, rws_ir.json, violation_set.json, execution_spec.md, generated code, execution logs, result manifests, figure manifests, evidence certificates, report sections, and disclosed assumptions. These files are bookkeeping carriers for the profile contract: downstream stages may reuse a fact only when it can be traced to profiled tables, validated fields, executable artifacts, or explicitly recorded assumptions. Table 28: Operational mapping between profile-grounded state roles and implementation artifacts. State role Stored artifacts Example contents Accessed by Execution trace exec_logs.json, exec_status.json, opencode_live.log, violation_set.json, audit records. execution traces, contract violations, validation outcomes, disclosed assumptions. violation-guided execution, report generator, frontend status view. Working context excel_info.json, rws_ir.json, execution_spec.md, selected sheet payloads. selected sheets, required fields, join keys, variable roles, temporal anchors, method specification, claim policy. IR compiler, execution workspace, report context builder. Profile semantics Ī¦H,K ^H,K, Ī¦L,K ^L,K, ontology mappings, reporting prompts. exposure/outcome/covariate roles, plausibility checks, clinical terminology, section semantics. IR compiler, code generator, report DAG nodes. Certified outputs result manifests, figure manifests, evidence certificates, report sections, disclosed assumptions. validated tables and figures, claim traces, support levels, limitations, section drafts. claim checker, report DAG, final report renderer. Operational policy IR compilation, profiling, coding, figure, validation, and report-generation policies. coding rules, artifact schemas, figure manifest schema, section templates, validation gates, claim policies. execution workspace, report builder. Algorithm 2 Profile-grounded state update and access in ProCARE 1: User query q, dataset D, profile cache C 2: Analysis artifacts, report, and updated job state 3: Compute dataset hash h=MD5ā”(D)h=MD5(D). 4: Load Ī¦ from ā”[h]C[h] if available; otherwise profile D and cache Ī¦ . 5: Create job workspace W and write the profile into excel_info.json. 6: Compile candidate RWS-IR Z0Z_0 from q and the high-level profile. 7: Run static IR checks against Ī¦ and the job state. 8: Write accepted Z to rws_ir.json and write execution context to execution_spec.md. 9: Run violation-guided execution using Z, low-level profile blocks, and operational policy artifacts. 10: Append execution logs, violation sets, validation results, and figure manifests to the job state. 11: Generate certified report sections from the reporting DAG using Ī¦ , Z, logs, figures, evidence certificates, and existing sections. C.8 Ontology mapping coverage The knowledge profiler maps raw fields to biomedical concepts from UMLS, SNOMED CT, ICD-10, and LOINC through candidate generation followed by type-compatibility and value-range checks. Overall field-level mapping coverage is 0.83. Coverage for critical task fields, including exposures, outcomes, confounders, and temporal anchors, is 0.91. Unmapped fields remain in Ī¦L,K _L,K with failure tags. The main failure categories are abbreviations or local jargon without a stable ontology match (0.07), type incompatibility between candidate concepts and observed values (0.05), ambiguity among indistinguishable candidates (0.03), and mixed units or inconsistent coding systems (0.02). C.9 Artifact bundle contract The prompt schema contains the task definition, optional guidance, and required deliverables. Each run returns a validated RWS-IR with required tables, join keys, variables, temporal anchors, methods, artifacts, and claim policy; one executable entry point (analysis.py or main.py) with optional helper modules for data loading and artifact generation; and a structured scientific report with evidence certificates. Generated SQL or pandas operations may reference only tables and columns present in the profile. The contract depends on prompt specificity. Under GOAL+METHOD, the agent follows the requested variables and stops when required data are unavailable. Under GOAL and GOAL+APPROACH, the agent may generate the closest feasible analysis, but the report must disclose data limitations and method deviations. The artifact bundle is the unit of evaluation. A bundle is accepted for full scoring only if it contains the files in Table 29. Missing execution-dependent artifacts produce N/A for metrics that cannot be computed, while schema, report, and claim diagnostics are still computed when the submitted artifacts allow post-hoc checking. Table 29: Required artifact-bundle schema. Artifact Typical file name Evaluation use RWS-IR rws_ir.json PF, method validation, temporal and grain checks, claim policy. Executable entry point analysis.py or main.py XE, CQ, artifact generation, code-region traces. Execution logs exec_logs.json, exec_status.json Runtime pass/fail, violation localization, evidence reconstruction. Results manifest results_manifest.json Result-table existence, denominators, generated statistics. Figure manifest figure_manifest.json Figure existence, caption/axis metadata, chart-fidelity checks. Violation set violation_set.json VRR and repair diagnostics. Evidence certificates evidence_certificates.json CTC, UCR, CLV, report traceability. Assumptions and limitations assumptions.json or report section Downgrade handling and policy-compatible language checks. Final report report.md, report.json, or rendered report RS, SL, factual-claim extraction, evidence-policy matching. C.10 Profile-constrained coding harness The code-generation stage is implemented as a controlled execution workspace. The workspace writes the profile, RWS-IR, local policies, generated code, figures, logs, violation sets, and validation records into one job directory. The coding agent can edit and execute files inside this workspace, while the backend reruns the selected entry point and rejects incomplete artifact bundles. Table 30 maps the execution components to concrete implementation artifacts. Table 30: Concrete artifacts used by the profile-constrained coding harness. Harness component Concrete artifact Purpose Workspace W Isolated job directory with generated scripts, plots, logs, and report fragments. Keeps one RWS task reproducible and separates generated files from source code. Context X rws_ir.json, excel_info.json, and execution_spec.md. Supplies objective, selected sheets, required fields, join keys, temporal anchors, method choices, and claim policy. Policies Ī Local coding instructions, figure policy, publication_plotting.py, and artifact schema. Converts profile and reporting requirements into file-level rules that the coding agent must follow. Tools T File editing, Python execution, local skill use, and sandbox validation. Allows edit-run-fix behavior inside the controlled workspace. Validators V Entry-point execution, TASK_FAILED sentinel check, figure_manifest.json, and PNG/PDF/SVG existence checks. Turns soft generation instructions into completion gates for executable artifacts. Traces L opencode_live.log, exec_logs.json, exec_status.json, and violation_set.json. Records evidence for violation-guided repair, failure reporting, and later report grounding. The constraint intent remains the same, but enforcement happens through observable gates instead of a separate symbolic solver. Schema constraints are checked against excel_info.json; methodological constraints are checked against the RWS-IR and profile semantics; artifact constraints are checked by execution and manifest validation. If the profile does not support a required variable, temporal anchor, or statistical procedure, the execution workspace requires a failed run or an explicit limitation rather than an unsupported substitute. C.11 DAG reporting dependency rules Report generation uses a dependency DAG G=(T,E)G=(T,E). Each node tāTtā T is an atomic reporting task that either constructs a validated evidence artifact or renders a report section from validated artifacts. Each directed edge (ti,tj)āE(t_i,t_j)ā E indicates that tjt_j requires artifacts produced by tit_i. Table 31 defines the task metadata used by the graph. Table 31: Task metadata used in the reporting DAG. Symbol Meaning prodā”(t)prod(t) Artifacts produced by task t, such as tables, figures, statistics, or text. reqā”(t)req(t) Prerequisites used by task t, such as variables, cohorts, temporal anchors, or method assumptions. Required chapter schema for RS. The minimal required chapters are listed in Appendix C.13. RS is computed as the fraction of required chapters covered with non-trivial content. Support score and edge weights. For each dependency edge (ti,tj)(t_i,t_j), we assign a weight: wiāj=siājā Ļā”(0.0+3.0āciāj),w_ij=s_ijĀ·Ļ\! (0.0+3.0\,c_ij ), (6) where Ļā”(x)=11+eāxĻ(x)= 11+e^-x maps to [0,1][0,1] and siājā0,1s_ijā\0,1\ is a feasibility gate indicating whether tjt_j can run from the available artifacts and recorded assumptions. We compute ciājc_ij from the four deterministic signals in Table 32: ciāj=0.35āSvar+0.25āStemp+0.25āSsize+0.15āScons.c_ij=0.35S_var+0.25S_temp+0.25S_size+0.15S_cons. (7) All experiments use Ī±=0.0Ī±=0.0, Ī²=3.0Ī²=3.0, wvar=0.35w_var=0.35, wtemp=0.25w_temp=0.25, wsize=0.25w_size=0.25, wcons=0.15w_cons=0.15, and Nmin=30N_ =30. These constants are fixed before evaluation. They encode the ordering preference that validated variable support, temporal validity, effective sample size, and method-design consistency should be present before a result section is promoted above a limitation or feasibility section. The traversal rule changes report ordering and section expansion only; it does not change executed statistics or certify unsupported claims. Table 32: Support signals for reporting-task prioritization. Signal Definition SvarS_var Variable availability in the analysis-ready cohort, with missingness penalties. One instantiation is 1||āāvāmaxā”(0,1āmissā”(v)) 1|V| _v (0,1-miss(v)). StempS_temp Temporal validity, defined as 1ārinvalid1-r_invalid, where rinvalidr_invalid is the fraction of missing, invalid, or impossible temporal prerequisites. SsizeS_size Sample adequacy, defined as minā”(1,Neff/30) (1,N_eff/30) after filtering and joins. SconsS_cons Method-design consistency, with partial credit only when execution is feasible under recorded assumptions. Ordering policy. We generate report outlines with a weighted topological traversal. At each step, among tasks whose prerequisites are satisfied, we select: tā=argā”maxā”ā(u,t)āEtāfrontierā”wuāt.t = _t _(u,t)ā Ew_ut. (8) C.12 Instantiated reporting DAG For the recurrence-free survival benchmark instance, the task uses SG.SGDAT as the index date, the earliest non-missing VT.PADHDAT/VT.PADRDAT as the recurrence event date, and the latest non-missing VT.VTDAT1/VT.VTDAT2 as censoring time. In the workbook, the surgery cohort contains 1,295 subjects, 1,215 subjects have analyzable RFS follow-up, and 304 subjects have valid recurrence events. The same graph shows evidence gating: because the AE and SAE detail tables contain no valid records, detailed AE profiling is blocked and the evidence gap is propagated to the limitations section. T0ObjectiveT1CohortT2BaselineT3EndpointT4ExposureT5Method gateT6Primary RFST7SubgroupBlockeddetailed AET9ReportT8Safety check1.000.890.980.850.9311.00s=0s=0 Figure 3: Instantiated reporting DAG for the recurrence-free survival task. Solid edges denote feasible artifact dependencies and show selected support scores; the dashed edge records the blocked AE-detail analysis, which is routed to limitations rather than promoted as a safety result. Table 33: Instantiated report-task DAG for the recurrence-free survival benchmark instance. Each node declares required inputs and produced artifacts. Edges encode artifact prerequisites; infeasible edges are not selected and instead propagate limitations. Node Task Required inputs Produced artifacts Report role T0 Objective binding User prompt, heterogeneous profile, report schema Objective, endpoint definition, required section plan Study overview T1 Cohort and attrition SG.Subject_Id, SG.SGDAT Analysis cohort, attrition table, index-date assumptions Overview / Attrition T2 Baseline covariates T1, DM, BCLC, CP, ECOG Table 1, missingness table, covariate map Baseline characteristics T3 Endpoint and censoring T1, VT.PADHDAT, VT.PADRDAT, VT.VTDAT1, VT.VTDAT2 RFS dataset, event/censor flags, temporal validity log Primary analysis T4 Exposure matrix T1, TACE, TART, IMU, PATE Treatment exposure matrix, dated exposure availability Primary / Subgroup T5 Method feasibility gate T2, T3, T4, procedural policy KM feasibility decision and caveats Methods T6 Primary RFS analysis T3, T4, T5 KM curve, RFS summary table, primary result text Primary analysis T7 Subgroup/sensitivity T2, T4, T6 Subgroup tables and caveats Sensitivity analysis T8 Safety evidence check AE_HQ, SAE_HQ, AE, SAE Safety limitation statement; blocked detailed AE profiling Safety / Limitations T9 Report synthesis Prior artifacts and evidence map Final structured report and claim-evidence map Final report Table 34: Selected dependency edges and support scores for the instantiated DAG. The feasibility gate siājs_ij blocks hard-infeasible dependencies. The support score ciājc_ij summarizes deterministic evidence from variable availability, temporal validity, sample adequacy, and method-design consistency. Edge Dependency rationale s_ij c_ij T0 ā T1 Cohort construction requires the objective, index date, and output contract. 1 1.00 T1 ā T2 Baseline summaries must be computed within the index cohort; BCLC and ChildāPugh coverage is high, while age missingness must be disclosed. 1 0.89 T1 ā T3 RFS endpoints and censoring must be aligned to surgery date; 1,215 subjects have usable follow-up and one temporal anomaly is flagged. 1 0.98 T1 ā T4 Treatment exposure grouping must be aligned to the index cohort; treatment proxies cover 1,104 subjects but dated exposures are incomplete. 1 0.85 T5 ā T6 The primary RFS analysis is selected only after the method gate passes; the combined gate has Svar=0.828S_var=0.828, Stemp=0.999S_temp=0.999, Ssize=1.000S_size=1.000, and Scons=1.000S_cons=1.000. 1 0.931 T8 ā detailed AE profiling Detailed AE/SAE profiling requires non-empty detail tables; both detail tables contain no valid records. 0 0.00 T8 ā T9 The empty AE/SAE detail tables support a limitation statement rather than unsupported safety claims. 1 1.00 C.13 Standard report schema The reporting engine targets the sections in Table 35. Table 35: Required RWS report sections used by the RS metric. Section Expected content Study Overview Cohort definition and objective statement. Attrition Analysis Patient disposition and eligibility filtering steps. Baseline Characteristics Demographics, clinical history, baseline vitals, and baseline labs. Primary Analysis Main efficacy or exposure-outcome results, such as KM curves or hazard ratios. Safety Profiling Adverse-event screening summaries and available safety proxies. Sensitivity/Subgroup Analysis Checks across predefined strata. Limitations Data constraints, method limitations, and potential biases. Appendix D Experimental Protocol and Reproducibility D.1 Execution environment and cost accounting Generated analysis code is rerun from a clean job workspace in a Python 3.9 sandbox with network access disabled. The execution image includes pandas, numpy, scipy, matplotlib, seaborn, and statsmodels. The evaluator records stdout, stderr, return code, generated files, result manifests, figure manifests, violation sets, and failure sentinels. LLM calls are made outside the code-execution sandbox; generated code cannot fetch external data during scoring. Token use is reported for ProCARE because baselines use different interfaces and are not directly comparable at the prompt level. Profiling is a reusable dataset-level cost. For the primary HCC schema, profiling uses approximately Tprofā65ākT_profā 65k tokens, and each query uses approximately Tinfā38ākT_infā 38k tokens for IR compilation, repair prompts, and certified reporting. For N tasks on the same dataset, TĀÆā(N)=Tinf+Tprof/N. T(N)=T_inf+T_prof/N. Thus TĀÆā(1)ā103āk T(1)ā 103k tokens, TĀÆā(20)ā41.3āk T(20)ā 41.3k tokens, and TĀÆā(36)ā39.8āk T(36)ā 39.8k tokens per query. D.2 Systems and baseline adaptation All methods receive the same natural-language task prompt, raw data access boundary, deliverable request, and inference budget. Direct Gemini-3-Pro baselines and direct Gemini-3-Pro scaffolded controls use the same model as the default ProCARE backbone. Claude Code and Codex are evaluated as native code-agent baselines: they can edit files and run code through their coding-agent interfaces, but they do not receive ProCAREās profile object, typed RWS-IR, evidence-certificate admission policy, or certified reporting DAG unless they infer or generate analogous files from the prompt. ProCARE additionally constructs and uses the heterogeneous profile, typed RWS-IR, execution validators, evidence certificates, and certified reporting DAG. Baselines are evaluated under a post-hoc contract checker so that schema use, execution, artifact completeness, and claim traceability are measured against the same profile-derived constraints. Table 36: Baseline adaptation and scoring rules. Method family Run protocol Scoring rule ReAct and ToolLLM Agentic tool-use baselines are asked to inspect the tables, write executable analysis code, generate artifacts, and produce a report under the same prompt. Deterministic validators execute the final entry point when provided; reports are checked post hoc for evidence support. TableGPT-Agent and ChatExcel-Max Table-analysis agents are asked to generate the requested analysis and report. If no executable artifact bundle is produced, execution-dependent metrics are N/A; report and claim metrics are computed when possible. Gemini-3-Pro direct baseline Gemini-3-Pro receives the task and data-access instructions and is asked to return code, artifacts, and report content. Generated code is executed in the same sandbox and evaluated by the same artifact and claim validators. TableGPT2 The table model is evaluated on table-analysis outputs and executable artifacts when available. Metrics requiring a complete bundle are N/A when the method does not emit the required artifact. Claude Code and Codex Native coding-agent baselines receive the benchmark prompt, data-access path, artifact contract, and sandbox budget through their repository editing interfaces. They may create, edit, execute, and repair files within the job workspace. The final workspace is re-executed from a clean scorer sandbox. Execution, artifact, clinical/statistical language, and claim-trace metrics are computed post hoc from the submitted files and report. ProCARE Full profileāIRāexecutionācertificateāDAG workflow. All deterministic and judge metrics are computed on the accepted artifact bundle. A method is not penalized twice for a missing executable bundle. XE is zero when an executable entry point is submitted but fails, while metrics that require absent artifacts are marked N/A. Schema feasibility, report structure, scientific language, and post-hoc claim diagnostics are still computed whenever enough submitted content is available. Code-agent baselines typically produce executable bundles and stronger local repairs, but their reports are still scored without granting credit for ProCARE-only objects such as accepted RWS-IRs or admitted evidence certificates. D.3 Fairness-controlled scaffolded baselines To address the concern that the artifact contract may favor ProCARE, we conducted a scaffolded-baseline evaluation under the same final checker used for the main baseline comparison. These RQ1 fairness controls differ from the RQ2 mechanism ablations in Appendix D.4: ablations remove one component from ProCARE, whereas scaffolded baselines add evaluator-facing information and output schemas to external baselines or a direct Gemini-3-Pro run. Direct Gemini-3-Pro controls use the same provider endpoint, model identifier, decoding settings, and inference budget as the default ProCARE backbone. The scaffold gives non-ProCARE methods the same schema profile Ī¦ , artifact-bundle template, raw data access boundary, execution sandbox, and three-turn repair budget needed to satisfy the evaluator. It does not give baselines ProCAREās IR compiler, certificate admission gate, certified reporting DAG, or hidden planner state. Rows that mention a study-spec or claim-trace schema only require compatible JSON files for scoring. The shared artifact template requires: study_spec.json, analysis.py or main.py, result_manifest.json, figure_manifest.json, claim_trace.json when the method attempts explicit claim tracing, assumptions.json, and report.md. For methods that do not emit a typed RWS-IR, the evaluator maps the submitted study_spec.json into the canonical scoring schema before computing PF. For methods that do not emit explicit evidence certificates, CTC/UCR/CLV are computed by the same post-hoc evidence reconstruction used in Table 36. Table 37: Inputs and controls used in the scaffolded-baseline fairness evaluation. These are baseline adaptations, not ProCARE component ablations. Condition Information given to model Output and repair control Purpose Profile/template baseline scaffold Task prompt, raw data access, Ī¦ summary, table/field dictionary, join keys, temporal-role candidates, and artifact template. One initial generation; no validator feedback loop beyond final scoring. Controls for missing schema and output-format information. Profile/template/repair baseline scaffold Same as above. Three edit-run-repair turns with standardized validator feedback and identical execution budget. Controls for unequal repair opportunity. Direct Gemini-3-Pro + validator interface Gemini-3-Pro receives Ī¦ , artifact template, sandbox logs, and validator feedback. Three repair turns; no typed IR compiler or certificate admission gate. Controls for deterministic validation feedback without ProCARE planning. Direct Gemini-3-Pro + study-spec schema Gemini-3-Pro receives Ī¦ and must emit study_spec.json before writing code and report. No ProCARE IR compiler; validator feedback is limited to schema/artifact execution errors. Controls for evaluator-compatible study-spec output. Direct Gemini-3-Pro + claim-trace schema Gemini-3-Pro receives Ī¦ , artifact template, and claim-trace file schema. No certificate admission gate or certified DAG ordering. Controls for evaluator-compatible claim-trace output. ProCARE reference Full profile, IR compiler, violation-guided execution, evidence certificates, and certified report DAG. Same three repair turns and final checker. Reports the integrated contract workflow for comparison. Table 38 reports the final checker outputs for the scaffolded-baseline evaluation. Scaffolded baselines improve substantially over generic agents, reducing N/A rates and narrowing the execution gap, while the integrated ProCARE workflow remains strongest on violation resolution and claim-level traceability. The table should be read as a fairness-control comparison, not as evidence that individual ProCARE modules have been added to an otherwise unchanged LLM. All direct rows use Gemini-3-Pro. Table 38: Full metric results for fairness-controlled scaffolded baselines. Higher is better except UCR and CLV. Rows are baseline adaptations, not ProCARE ablations; values are final checker outputs. Baseline condition Performance Contract diagnostics GAā PFā XEā CQā VCā CFā RSā SLā VRRā CTCā UCRā CLVā ReAct + fairness scaffold 0.82 0.79 0.74 0.69 0.62 0.65 0.75 0.70 0.62 0.70 0.13 0.10 ToolLLM + fairness scaffold 0.79 0.80 0.68 0.65 0.58 0.62 0.72 0.67 0.58 0.66 0.15 0.12 Direct Gemini-3-Pro + profile/template 0.77 0.73 0.66 0.62 0.54 0.58 0.73 0.65 0.31 0.55 0.20 0.15 Direct Gemini-3-Pro + validator repair 0.80 0.78 0.79 0.72 0.60 0.64 0.75 0.69 0.68 0.65 0.15 0.11 Direct Gemini-3-Pro + study-spec schema 0.82 0.82 0.70 0.68 0.57 0.62 0.71 0.68 0.43 0.52 0.21 0.16 Direct Gemini-3-Pro + claim-trace schema 0.78 0.74 0.63 0.61 0.59 0.63 0.74 0.73 0.35 0.74 0.11 0.08 ProCARE reference 0.86 0.84 0.92 0.88 0.72 0.74 0.80 0.79 0.81 0.90 0.06 0.03 D.4 Ablation protocol Mechanism ablations remove one contract component at a time while keeping all remaining components fixed whenever the remaining workflow is defined. Table 39 specifies the intervention behind the ablation table in the main paper. Table 39: Ablation definitions for contract mechanisms. Variant Removed component What remains fixed w/o profile Ī¦ The model does not receive the heterogeneous profile as a contract object. It may inspect raw table metadata and generated summaries, but generation lacks profile-grounded repair hints. Prompt, budget, execution sandbox, reporting request, and post-hoc evaluator. w/o RWS-IR The model writes code and a report without producing the typed RWS-IR. Raw data access, profile summaries when available, execution sandbox, and post-hoc claim diagnostics. PF is N/A because no typed IR exists. w/o violation-guided repair Runtime validators detect violations, but localized violation sets are not fed back into the next edit. Profile, RWS-IR, execution sandbox, artifact validators, certificates, and report policy. w/o evidence certificates The report is generated without explicit claim certificates and without the claim-admission gate. Profile, RWS-IR, execution repair, artifacts, and DAG section ordering. CTC for emitted explicit certificates is zero; post-hoc support can still be inspected separately. w/o certified DAG ordering Evidence certificates remain available, but report sections are generated in a fixed or unconstrained order without dependency-aware traversal. Profile, RWS-IR, execution repair, artifact validators, and claim admission. Full ProCARE No contract component is removed. Complete profileāIRārepairācertificateāDAG workflow. D.5 Model access, decoding, and version identifiers Backbone-sensitivity experiments keep profiling, RWS-IR compilation, violation-guided execution, certified reporting, prompts, and scoring scripts fixed. The table records the paper-level model identifiers and roles used in the experiments. Provider-returned request metadata is retained in run logs; when a provider alias differs from the paper-level display name, the logged identifier is the reproducibility target. Generation runs use the released implementationās production decoding settings, with temperature 0.2 for Gemini/openai-compatible providers and 0.1 for DeepSeek-compatible providers; judge models are run at temperature 0. Table 40: Model access, version metadata, and evaluation roles. Role Paper-level model identifier Access mode Use in evaluation Backbone GPT-5.2 Proprietary API Backbone-sensitivity run under the fixed ProCARE workflow. Backbone GPT-4 Turbo Proprietary API Backbone-sensitivity run under the fixed ProCARE workflow. Backbone Claude Opus 4.5 Proprietary API Backbone-sensitivity run under the fixed ProCARE workflow. Backbone Claude Sonnet 4.5 Proprietary API Backbone-sensitivity run under the fixed ProCARE workflow. Backbone Claude-3 Opus Proprietary API Backbone-sensitivity run under the fixed ProCARE workflow. Backbone, direct baseline, and scaffolded control Gemini-3-Pro Proprietary API Default ProCARE backbone; also used for the direct baseline and all Direct Gemini-3-Pro scaffolded fairness-control rows. Backbone and judge Gemini-2.5-Pro Proprietary API Backbone-sensitivity run and one blinded semantic judge. Backbone Qwen3-Max API endpoint Backbone-sensitivity run under the fixed ProCARE workflow. Backbone Qwen3-235B-A22B- Instruct-2507 API endpoint or hosted open-source model Backbone-sensitivity run. Backbone DeepSeek-V3.2 API endpoint or hosted open-source model Backbone-sensitivity run. Backbone DeepSeek-R1-0528 API endpoint or hosted open-source model Backbone-sensitivity run and non-valid-bundle baseline category. Code-agent baseline Claude Code (Claude Sonnet 4.5 backend) Proprietary coding-agent interface Baseline run through the native Claude Code repository-editing workflow; provider-returned CLI/model metadata is retained in the run log. Code-agent baseline Codex (gpt-5.3-codex backend) Proprietary coding-agent interface Baseline run through the native Codex repository-editing workflow; CLI session metadata and final workspace state are retained for reproduction. Judge Qwen2.5-1M API endpoint Second blinded semantic judge. D.6 Reproducibility package The anonymous release contains the manuscript sources, system code, prompt definitions, the fully de-identified HCC benchmark tables, Public-RWS bundle metadata, profile artifacts, evaluation scripts, and figure-generation scripts. The HCC release includes the record-level tables needed to rerun the 36 primary tasks, with direct identifiers removed and random benchmark IDs retained for reproducible linkage. Table 41: Repository components used for reproduction. Path Contents paper/ Manuscript, supplement, bibliography, NeurIPS style file, build script, and paper figures. code/backend/ Profiling, planning, code-generation, execution, report-generation, and API services. code/backend/tests/ Local tests for profile construction, planning, code execution, OpenCode integration, and report generation. code/frontend/ Interface for project setup, profile inspection, RWS-IR review, execution monitoring, and report review. code/output/knowledge_base/profiles/ Cached profile artifacts used by the implementation. rws_hcc_benchmark/ Fully de-identified HCC benchmark tables, data dictionary, checksums, task metadata, and release terms for reproducing the 36 primary tasks. rws_public_datasets_bundle/ Public-RWS source manifests, checksums, table shapes, and public dataset files. figure_code/ Scripts used to regenerate paper diagnostic figures. Listing 6: Local reproduction commands. ā¬ cd code/backend python -m app.cli --excel <workbook.xlsx> --topic "<benchmark prompt>" --refine python tests/run_all.py cd ../../paper build.cmd nips appendix Appendix E Additional Results and Diagnostics E.1 Claim-diagnostic interpretation More specific prompts increase claim traceability because they constrain variables, temporal anchors, methods, and expected artifacts before reporting. The dominant downgrades are language-policy failures rather than arithmetic inconsistencies: causal phrasing from descriptive analyses, safety claims based only on screening pages, or subgroup conclusions without sufficient support. During certification, these claims are rewritten as limitations or removed from the admitted factual-claim set. E.2 Backbone-sensitivity interpretation Backbone substitution changes only the base LLM used for planning, code editing, and report drafting. Profiling, RWS-IR schema, validators, evidence-certificate admission, report schema, and evaluation remain fixed. Stronger backbones improve GA, PF, VRR, and CTC because they use profile constraints and repair feedback more reliably. Execution-only differences are smaller than planning and reporting differences because deterministic validation and localized repair bound the accepted artifact space after code is generated. Appendix F System Interface and Workflow Figures 4 and 5 show the implementation interface states. They are included as workflow evidence rather than method exposition: the panels identify where profile artifacts, IR traces, execution logs, generated figures, and section-level report blocks are inspected during a run. (a) System dashboard for managing RWS projects and cohort status. (b) Profile panel for reviewing structural metadata (Ī¦S ^S) and ontology mappings (Ī¦K ^K). (c) RWS-IR view showing goal compilation (TgoalāZT_goalā Z). (d) Coding harness view with reasoning traces, sandbox logs, and verification status. Figure 4: ProCARE interface states for project setup, profile inspection, RWS-IR compilation, and coding execution. (a) Report review view with generated tables, charts, and evidence links. (b) Section-wise report generation with data-bound report blocks. Figure 5: Report-facing interface states for reviewing generated artifacts, evidence links, and section-wise report content. Appendix G Evaluation Protocol and Rubrics Evaluation uses deterministic validators for feasibility and execution checks, and LLM judges for semantic quality. G.1 Metrics Overview Table 42 defines the metrics used for end-to-end RWS automation. All scores are normalized to [0,1][0,1]. Table 42: Metrics used in the appendix evaluation protocol. Metric Type Scoring rule RWS-IR Feasibility (PF) Deterministic + judge Validates the RWS-IR against Ī¦S ^S and penalizes non-existent tables, columns, invalid joins, unsupported temporal anchors, or incompatible analysis grains. Residual ambiguity is checked by the judge. Code Executability (XE) Deterministic Sandbox execution success rate, npass/Nn_pass/N; runtime errors and invalid output files count as failures. Violation Resolution Rate (VRR) Deterministic Fraction of detected execution-time contract violations resolved before the accepted run. Report Structure (RS) Deterministic Fraction of required report sections in Table 35 covered with non-trivial content. Claim Trace Coverage (CTC) Deterministic Fraction of final factual claims linked to valid evidence certificates. Unsupported Claim Rate (UCR) Deterministic Fraction of final factual claims without admissible evidence support. Claim Language Violation (CLV) Deterministic + judge Fraction of factual claims whose wording exceeds the support level allowed by the evidence certificate and RWS-IR claim policy. Goal Alignment (GA) Judge Alignment with the clinical objective, target population, and requested endpoint. Code Quality (CQ) Judge Soundness of data handling, missingness handling, joins, and statistical implementation. Visual Clarity (VC) Judge Readability and labeling quality of generated figures. Chart Fidelity (CF) Judge Consistency between plotted values, execution logs, and report statements. Scientific Language (SL) Judge Clinical tone, hedging, and limitation disclosure. G.2 Deterministic validators Deterministic validators are used whenever the property can be checked from the artifact bundle. PF checks whether the accepted RWS-IR references only profiled tables, fields, keys, clinical roles, temporal anchors, and methods. XE checks whether the submitted entry point exits successfully, avoids failure sentinels, and writes the required result and figure manifests. Artifact completeness requires declared output files to exist, be nonempty, and be referenced by the corresponding manifest. Temporal consistency checks row-level ordering between index, exposure, event, and censoring dates when those roles are declared in Z. Chart-result consistency checks whether plotted variables, group labels, sample sizes, and reported summary values match execution logs or result tables. All metrics are normalized to [0,1][0,1]. XE is npass/Nn_pass/N over tasks. PF is one for a statically admissible RWS-IR, zero for a hard-invalid RWS-IR, and partially credited only for residual semantic ambiguity adjudicated by the fixed rubric. RS is the fraction of required report sections with task-relevant content. VRR, CTC, UCR, and CLV are computed from logged violations and extracted factual claims as described in Appendix C.5 and Appendix C.6. For baselines that do not implement ProCAREās contract internally, the evaluator reconstructs the same diagnostics from final artifacts. A baseline can receive CTC credit only when a factual claim can be linked to an executed result, code region, profile-supported field, or explicit assumption. Missing artifacts or unsupported variables block traceability. G.3 LLM judge protocol We use Gemini-2.5-Pro and Qwen2.5-1M as independent judges with temperature 0. Judges are blinded to method identity and receive the same artifact bundle: input prompt, generated RWS-IR, code, execution logs, result and figure manifests, evidence certificates when available, assumptions, limitations, and final report. Scores are normalized to [0,1][0,1] and averaged across judges. Schema hallucination and artifact inconsistency are handled as hard rubric failures for the affected dimensions. G.4 Expert-review protocol Expert review is performed on the 36 primary HCC benchmark instances. The panel contains five blinded reviewers: three board-certified clinical experts with hepatobiliary oncology or hepatology experience (C1āC3) and two statistics/data experts with clinical data-analysis experience (S1āS2). Reviewers receive anonymized artifact bundles containing the task prompt, study specification, code or execution logs when relevant, figures, assumptions, limitations, evidence certificates when available, and final report. Method identity is masked in the review package. Reviewers score each stage on a five-point Likert scale: 1 indicates an invalid or irrelevant artifact, 3 indicates a partially correct artifact with important limitations, and 5 indicates a clinically and methodologically appropriate artifact with limitations disclosed. The clinical panel focuses on clinical objective alignment, endpoint and covariate validity, clinical interpretability of figures, and claim scope. The statistics/data panel focuses on data linkage, analysis grain, statistical method choice, code reproducibility, figure-result consistency, and whether reported results are supported by executed artifacts. Code scores in the main table use the statistics/data panel only; the other stage scores aggregate all five reviewers because both clinical and statistical/data judgments are relevant. The raw evaluation data are represented in the long format in Table 43. The required stage-level row set contains 36Ć(5+2+5+5)=61236Ć(5+2+5+5)=612 reviewerāinstanceāstage records before optional criterion-level comments: five reviewers score RWS-IR, visualization, and reporting, while S1āS2 score code implementation. Additional criterion-specific rows can be added with the same keys when a reviewer records a subscore or a major error flag. Table 43: Raw expert-rating data format used before aggregation. One row stores one reviewerāinstanceāstage score. Column Allowed values or format Definition review_id stable string Unique row identifier. benchmark_id HCC-001āHCC-036 Primary benchmark instance identifier. objective cohort, safety-proxy, efficacy, resource-use Benchmark objective family. prompt_level goal, goal+approach, goal+method Prompt-specificity level. method_id submitted method identifier Method label masked during review and restored only for analysis. stage rws_ir, code, visualization, reporting Workflow stage being scored. artifact_id path or hash Submitted artifact bundle, code file, figure set, or report identifier. reviewer_id C1, C2, C3, S1, S2 De-identified reviewer identifier. reviewer_role clinical or statistics_data Reviewer expertise group. criterion_id global_stage_score or rubric subcriterion Overall stage score or optional subcriterion. score_1_5 integer 1ā5 Raw Likert score before normalization. major_error_flag 0 or 1 Reviewer-marked hard error, such as invalid endpoint, impossible join, or irreproducible code. claim_flag 0, 1, or NA Whether the reviewed report or figure interpretation contains a claim exceeding evidence support. comment_code none, schema, method, temporal, claim, figure, reproducibility, other Structured reason for downgrade, if any. free_text_comment de-identified text Short reviewer note used for adjudication and error analysis. timestamp_utc ISO-8601 timestamp Completion time of the review row. For each benchmark instance i, reviewer r, and stage s, the raw Likert score is xiārāsā1,ā¦,5x_irsā\1,ā¦,5\. Stage-level expert scores are first averaged across the eligible reviewer set RsR_s: Eiās=1|Rs|āārāRsxiārās.E_is= 1|R_s| _rā R_sx_irs. Scores are normalized to [0,1][0,1] for calibration plots by E~iās=(Eiāsā1)/4 E_is=(E_is-1)/4, but the main paper reports meanĀ± on the original 1ā5 scale. Spearman Ļ is computed between per-instance expert scores and the corresponding automated stage score: (GA+PF)/2(GA+PF)/2 for RWS-IR, CQ for code, (VC+CF)/2(VC+CF)/2 for visualization, and SL for reporting. Confidence intervals for mean expert scores use a t interval over 36 instance-level means; confidence intervals for Spearman correlations use Fisher transformation. Table 44: Expert panel responsibilities and raw scoring volume. Stage Eligible reviewers Primary review focus Raw records RWS-IR C1āC3, S1āS2 Clinical goal, endpoint validity, data linkage, analysis grain, and statistical design feasibility. 180 Code S1āS2 Reproducibility, executable implementation, statistical method use, artifact writing, and result consistency. 72 Visualization C1āC3, S1āS2 Clinical interpretability, sample-size labeling, figure-result agreement, and avoidance of unsupported visual claims. 180 Reporting C1āC3, S1āS2 Clinical language, limitation disclosure, statistical wording, and claim-to-evidence support. 180 Table 45: Stage-level expert score summary on the 36 primary HCC benchmark instances. CI is the 95% confidence interval for the instance-level mean. Stage Instance means Raw records MeanĀ± 95% CI Median [IQR] RWS-IR 36 180 4.38Ā± 0.48 4.22ā4.54 4.33 [4.00, 4.67] Code 36 72 4.42Ā± 0.21 4.35ā4.49 4.42 [4.25, 4.58] Visualization 36 180 4.50Ā± 0.53 4.32ā4.68 4.67 [4.00, 5.00] Reporting 36 180 4.13Ā± 0.64 3.91ā4.35 4.17 [3.67, 4.67] Table 46: Inter-rater reliability for the human expert panel. Stages scored by five reviewers use two-way random-effects ICC for averaged ratings; the two-reviewer code stage uses weighted Cohenās Īŗw _w. Stage Panel Statistic Estimate 95% CI Interpretation RWS-IR C1āC3, S1āS2 ICC(2,k) 0.84 0.72ā0.91 Strong agreement on specification validity. Code S1āS2 Īŗw _w 0.76 0.61ā0.88 Substantial agreement on reproducibility and statistical implementation. Visualization C1āC3, S1āS2 ICC(2,k) 0.79 0.66ā0.88 Strong agreement on figure clarity and consistency. Reporting C1āC3, S1āS2 ICC(2,k) 0.77 0.63ā0.87 Substantial-to-strong agreement on language and claim scope. Table 47: Correlation between automated metrics and aggregated expert scores. q is the BenjaminiāHochberg adjusted p value across the four stage-wise correlations. Stage Expert aggregation Automated score Ļ 95% CI p q RWS-IR Mean of C1āC3, S1āS2 (GāA+PāF)/2(GA+PF)/2 0.82 0.67ā0.90 <0.001<0.001 <0.001<0.001 Code Mean of S1āS2 CāQCQ 0.74 0.54ā0.86 <0.001<0.001 <0.001<0.001 Visualization Mean of C1āC3, S1āS2 (VāC+CāF)/2(VC+CF)/2 0.75 0.55ā0.86 <0.001<0.001 <0.001<0.001 Reporting Mean of C1āC3, S1āS2 SāLSL 0.71 0.49ā0.84 <0.001<0.001 <0.001<0.001 G.5 Semantic Rubric Anchors Judges assign 1ā5 Likert scores for Goal Alignment (GA), Code Quality (CQ), Visual Clarity (VC), and Scientific Language (SL). Scores are normalized to [0,1][0,1] with a linear transformation. Table 48 lists the anchor definitions. Table 48: Semantic rubric anchors for LLM judges. Metric Score 5 Score 3 Score 1 Goal Alignment Addresses the clinical objective, target population, and endpoints with explicit limitation disclosure. Partially addresses the task with minor errors in inclusion criteria or endpoints. Gives an irrelevant analysis or misunderstands the clinical goal. Code Quality Uses sound pandas or SQL logic, handles missing data and types correctly, and keeps the implementation modular. Runs but remains fragile, often because of hardcoded values or inefficient joins. Uses an incorrect method, applies the wrong statistical test, or contradicts the plan. Visual Clarity Produces figures with clear axes, legends, labels, and titles that support the reported results. Produces legible charts with missing units, overlapping labels, or default styling. Produces misleading, illegible, or unrelated visualizations. Scientific Language Uses clinical language with appropriate hedging and clear limitation disclosure. Uses a general scientific tone but has verbose phrasing or imprecise medical terminology. Uses casual language or makes unsupported claims. Appendix H Prompts and Templates H.1 Benchmark prompt template Each benchmark instance is rendered into a common prompt with three fields: Task, optional Guidance, and Deliverables. The task states the study objective. Guidance supplies approach-level or method-level constraints for Goal+Approach and Goal+Method. Deliverables require a validated RWS-IR, one executable entry point with generated artifacts, and a structured report with assumptions, limitations, and evidence certificates. Hallucinated tables or columns are disallowed. H.2 IR compiler prompt template The IR compiler receives the user prompt, high-level structural profile, high-level knowledge profile, output contract, and claim-policy rules. It returns JSON containing study design, analysis grain, join graph, entity keys, temporal anchors, variable-role bindings, method specification, required artifacts, assumptions, and allowed or blocked claim types. If the profile does not support a requested variable or method, the compiler must either downgrade the task or return an infeasibility trace. H.3 Coding harness prompt template The coding harness receives the accepted RWS-IR, low-level field profile, selected sheets, local figure policy, artifact schema, and any previous violation set. It writes one executable entry point, generates declared artifacts, and produces figure and result manifests. Repairs must address the localized violation set without introducing fields or methods absent from the profile. H.4 Report and certificate prompt template The report generator receives the accepted RWS-IR, result artifacts, figure manifest, execution logs, assumptions, limitations, and claim-policy rules. Each factual sentence must be paired with an evidence certificate or rewritten as a limitation. The generator must not convert descriptive proxy artifacts into causal, safety-critical, regulatory, or treatment-effect claims. H.5 Judge prompts The judge receives the ground-truth data profile Ī¦ so that schema hallucinations can be checked against the same table and field constraints used by the validators. System prompt. Defines the reviewer role and scoring constraints. Prompt Prompt 1. Expert Judge System Prompt You are a Senior Clinical Data Scientist reviewing Real-World Study (RWS) automation artifacts. Your task is to evaluate an AI agentās artifacts against a specific Ground Truth Schema. Scoring rules: 1. Schema Hallucination = 0: Any usage of tables/columns not present in the provided [DATA_PROFILE] results in a score of 0 for relevant metrics, regardless of code quality. 2. Consistency: The numbers in the Final Report must be consistent with the Execution Logs. 3. Structured rationale: Provide a concise evidence-based rationale before assigning JSON scores. User prompt. Injects the profile and artifact bundle. Prompt Prompt 2. Context-Injection User Prompt [PART 1: GROUND TRUTH CONTEXT] User Query: user_query Data Profile Constraints (Ī¦ ): (Valid tables, columns, and join keys derived from the dataset) profile_snapshot_str [PART 2: AGENT ARTIFACTS] 1. RWS-IR: agent_rws_ir_json 2. Code: generated_code 3. Logs: execution_logs 4. Visuals: list_of_generated_files 5. Evidence Certificates: evidence_certificate_json 6. Report: final_report_content [PART 3: TASK] Analyze the artifacts and score them based on the rubric. Structured assessment ā¢ Schema Check: Does the code use ONLY columns listed in [Data Profile Constraints]? ā¢ Logic Check: Is the statistical method appropriate for user_query? ā¢ Consistency: Do chart findings match the log data? JSON output Return a JSON object with keys: GA, CQ, VC, CF, SL. Format: "Analysis": "...", "Scores": ..., "Failure_Mode": "None/Hallucination/Logic" Appendix I Data Access, Privacy, and Responsible Use The released materials include the reproducible benchmark logic and the fully de-identified record-level HCC benchmark tables. The public package includes prompts, profiles, schema artifacts, aggregate summaries, de-identified HCC tables, HCC data dictionary, checksums, public-data manifests, and evaluation scripts. Direct participant identifiers and identity-linking fields are removed from the HCC release, while random benchmark subject and visit identifiers are retained to preserve reproducible table joins and longitudinal task execution. Public-RWS datasets are accessed from their original public sources, with source metadata and checksums recorded in the release documentation. The HCC benchmark is released for research reproduction and auditing of RWS automation methods. The release terms prohibit attempted re-identification, participant linkage to external data sources, and clinical use of generated outputs. Any downstream deployment on institutional data requires local governance review, schema revalidation, and confirmation that the site-specific claim policy matches the available data. ProCARE is evaluated as a research automation and auditability system, not as an autonomous clinical decision system. Evidence certificates establish traceability to implemented validators and artifacts. They do not establish clinical truth, causal validity, regulatory sufficiency, or prospective safety. The primary HCC benchmark is single-center and disease-specific. Public-RWS-18 tests schema transfer, but it does not replace multi-institutional longitudinal RWS validation. Porting ProCARE to OMOP, FHIR, or another institutional schema requires rebuilding the profile, revalidating RWS-IR checks, and confirming that claim policies match the new data source and study design.