Paper deep dive
Grounding Healthcare LLMs in a Causal Knowledge Graph: Framework, Metrics, and a Cardiovascular Pilot
Ummara Mumtaz, Aimen Noor, Awais Ahmed
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/22/2026, 2:04:05 AM
Summary
The paper introduces a reproducible, graph-centered evaluation framework for assessing Large Language Models (LLMs) in healthcare decision support, specifically focusing on intervention-oriented reasoning. The framework utilizes a domain causal knowledge graph where biomedical claims are reified as first-class nodes with stable identifiers, provenance, and evidence links. It employs a scenario-conditioned subgraph extraction mechanism and tests four grounding conditions (ungrounded, knowledge-graph, causal-graph, and integrated) on a cardiovascular pilot dataset. Results indicate that the integrated grounding condition (C4) significantly outperforms the ungrounded baseline (C1) in causal edge F1, adverse-effect identification, and evidence accuracy, demonstrating the value of structured, provenance-preserving context for reliable clinical AI.
Entities (13)
Relation Signals (8)
Causal Knowledge Graph → uses → Reified Assertion
confidence 95% · a domain causal knowledge graph in which assertions are first-class, provenance-preserving nodes with stable identifiers
Grounding Condition C4 → achieves → Causal Edge F1
confidence 90% · C4 obtains the strongest causal edge F1 (0.838)
Grounding Condition C4 → achieves → Adverse-Effect F1
confidence 90% · adverse-effect F1 (0.833)
Grounding Condition C4 → achieves → Evidence Accuracy
confidence 90% · evidence accuracy (0.738)
Grounding Condition C1 → achieves → Intervention Accuracy
confidence 90% · C1 obtains the highest raw intervention accuracy (0.948)
Grounding Condition C4 → achieves → Unsupported Claim Rate
confidence 90% · unsupported claim rate (0.114)
LLM → isevaluatedby → Causal Knowledge Graph
confidence 90% · We propose a reproducible, graph-centered evaluation framework for intervention-oriented LLM behavior in healthcare
Neo4j → supports → Causal Knowledge Graph
confidence 85% · parameterised Cypher retrieval against Neo4j graph database
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) are increasingly proposed for healthcare decision support, but their evaluations still reward single-answer accuracy rather than reasoning about interventions, mechanisms, harms, evidence, and uncertainty. We propose a reproducible, graph-centered evaluation framework for intervention-oriented LLM behavior in healthcare and stress-test it in a cardiovascular pilot. The framework has four components: (i) a domain causal knowledge graph in which assertions are first-class, provenance-preserving nodes with stable identifiers; (ii) a scenario-conditioned subgraph extraction step that, given any clinical scenario, retrieves the relevant reified-assertion subgraph; (iii) four controlled grounding conditions that vary how the retrieved subgraph is composed into the model's context (ungrounded C1, knowledge-graph C2, causal-graph C3, integrated C4); and (iv) an automated scoring pipeline, anchored on assertion identifiers, that computes intervention accuracy, and other evaluation measures on a single pass. To test the framework, we built a category-balanced scenario generator across eight reasoning failure modes and instantiated it on a cardiovascular graph. The metric panel discriminates conditions along interpretable, non-redundant axes: C4 obtains the strongest causal edge F1 (0.838), adverse-effect F1 (0.833), evidence accuracy (0.738), and unsupported claim rate (0.114), while C1 obtains the highest raw intervention accuracy (0.948) with no measurable causal or evidential grounding.
Tags
Links
- Source: https://arxiv.org/abs/2608.15382v1
- Canonical: https://arxiv.org/abs/2608.15382v1
Trouble viewing inline? Open PDF directly →
Full Text
64,825 characters extracted from source content.
Expand or collapse full text
Grounding Healthcare LLMs in a Causal Knowledge Graph: Framework, Metrics, and a Cardiovascular Pilot Ummara Mumtaz, Aimen Noor, Awais Ahmed Keywords. Causal health AI; large language models; knowledge graphs; intervention reasoning; evidence grounding; reified assertions. Abstract Large language models (LLMs) are increasingly proposed for healthcare decision support, but their evaluations still reward single-answer accuracy rather than reasoning about interventions, mechanisms, harms, evidence, and uncertainty. We propose a reproducible, graph-centered evaluation framework for intervention-oriented LLM behavior in healthcare and stress-test it in a cardiovascular pilot. The framework has four components: (i) a domain causal knowledge graph in which assertions are first-class, provenance-preserving nodes with stable identifiers; (i) a scenario-conditioned subgraph extraction step that, given any clinical scenario, retrieves the relevant reified-assertion subgraph; (i) four controlled grounding conditions that vary how the retrieved subgraph is composed into the model's context (ungrounded C1, knowledge-graph C2, causal-graph C3, integrated C4); and (iv) an automated scoring pipeline, anchored on assertion identifiers, that computes intervention accuracy, and other evaluation measures on a single pass. To test the framework, we built a category-balanced scenario generator across eight reasoning failure modes and instantiated it on a cardiovascular graph. The metric panel discriminates conditions along interpretable, non-redundant axes: C4 obtains the strongest causal edge F1 (0.838), adverse-effect F1 (0.833), evidence accuracy (0.738), and unsupported claim rate (0.114), while C1 obtains the highest raw intervention accuracy (0.948) with no measurable causal or evidential grounding. 1. Introduction Large language models are increasingly being integrated into clinical decision-support settings, yet consensus on appropriate evaluation standards has not developed at a comparable pace. Existing evaluation approaches continue to rely predominantly on multiple-choice benchmarks that assess whether a model selects the correct response [1–5]. Although such benchmarks facilitate standardized comparison and leaderboard-based ranking, they capture only a limited dimension of clinical performance. They do not adequately assess whether a recommendation is supported by a plausible clinical mechanism, whether relevant harms and contraindications are identified, or whether cited evidence is verifiable within the underlying knowledge base. They also fail to determine whether the model introduces unsupported causal claims or appropriately recognizes when the available evidence is insufficient to support a definitive conclusion. Consequently, models with comparable answer accuracy may differ substantially in their reliability and suitability as clinical assistants. Recent developments in medical AI have also reflected a broader shift from predictive modeling toward causal decision support [22–24]. Within this paradigm, evaluation should extend beyond concordance with a reference label or action to examine the reasoning that supports a clinical decision. Relevant dimensions include the appropriateness of the selected intervention, the plausibility of the proposed causal mechanism, the strength and relevance of supporting evidence, recognition of potential harms, consideration of contextual constraints, and appropriate representation of uncertainty. Evaluation based solely on the final action is therefore insufficient to distinguish clinically defensible reasoning from coincidentally correct outputs or to characterize errors that may have meaningful consequences for patient care. Developing frameworks that can assess these dimensions during an evaluation phase, represents an important component of the informatics infrastructure required for reliable clinical decision support and constitutes the focus of this study. We propose a framework for intervention-oriented LLM behavior in healthcare, structured around four coupled components: (a)defining a domain causal and evidence graph in which every biomedical claim is a first-class reified assertion node with a stable assertion_id, provenance, evidence links, and context, so that gold pathways, gold adverse- effect targets, and gold contraindications are addressable by identifier; (b) a scenario-conditioned subgraph extraction step implemented as parameterised Cypher retrieval against Neo4j graph database, which for any input scenario returns the relevant reified-assertion subgraph; (c) four controlled grounding conditions (ungrounded C1, knowledge-graph C2, causal- graph C3, integrated causal-knowledge-graph C4) that vary the type of retrieved context supplied to the model while holding the scenario fixed; and (d) a multi-dimensional automated scoring pipeline that computes intervention accuracy, causal edge precision/recall/F1, adverse-effect F1, contraindication recall, evidence accuracy, unsupported claim rate, and uncertainty correctness on the same scoring pass, all anchored to assertion_ids. Separately from the framework, we contribute an evaluation methodology used to test it in a controlled way: a category-balanced scenario generator producing structured intervention items with graph-anchored gold answers across eight causal-reasoning failure modes, and a repeated-measures design that presents each scenario under all four grounding conditions. The framework itself is scenario- source-agnostic; the generator and the repeated-measures protocol are evaluation instruments, not framework components. The cardiovascular intervention planning domain serves as the example domain for evaluating the methodology. Scenarios describe a patient context (for example an older adult with resistant hypertension and impaired renal function, or a pregnant patient in whom ACE inhibitors are contraindicated), a decision target (which pharmacological or non-pharmacological intervention to prefer), a set of candidate actions, and a graph-anchored gold answer that includes the preferred intervention, the causal pathway of ordered assertion_ids that justifies it, the expected adverse effects and contraindications, and the expected supporting evidence sources. The same input scenario is presented to the model under all four grounding conditions, so that behavioral differences can be attributed to context organization rather than to prompt wording or item difficulty. The empirical component of this manuscript is a single-model pilot on 313 scored responses from gpt-5.4, whose purpose is to show that the framework produces interpretable, non-redundant signal end-to-end. We do not claim that a specific model is clinically ready or that any grounding condition is definitively superior. The remainder of the paper is organised as follows. Section 2 positions the framework against prior work in medical LLM benchmarks, biomedical KGs, assertion-level provenance modeling, causal reasoning, and hallucination scoring. Section 3 specifies the framework, and the evaluation setting used to test it. Section 4 reports the pilot results, including the illustrative C4 versus C2 paired-delta contrast and a diagnostic contraindication finding. Section 5 combines discussion and limitations. Section 6 concludes. 2. Related Work Most widely used medical LLM benchmarks are organised around final-answer accuracy on multiple-choice or short- answer items MedQA [1], MedMCQA [2], PubMedQA [3], the medical subsets of MMLU [4], and the MultiMedQA suite used to evaluate Med-PaLM [5] and more recent efforts such as HealthBench and the clinical extensions of HELM [6] broaden the task distribution while still emphasising single-answer correctness. This design collapses reasoning quality into one scalar. A model that selects the correct action while inventing an unsupported mechanism, omitting a contraindication, or ignoring evidence quality receives the same score as one that arrives at the same action defensibly. The framework proposed here treats final-answer accuracy as one axis among several and separately measures causal reconstruction, adverse-effect identification, contraindication recall, evidence accuracy, unsupported claim generation, and uncertainty correctness. A parallel line of work uses biomedical knowledge graphs UMLS [7], SemMedDB [8], DrugBank [9], Hetionet [10], Open Targets [11], PrimeKG [12] to supply structured context to LLMs, using RAG [13] and graph-aware variants such as KG- RAG for medicine [14] and GraphRAG [15]. These approaches consistently improve downstream QA and summarisation, but retrieval infrastructure has grown much faster than evaluation infrastructure: most benchmarks still measure only whether the final answer changed, not whether the model used the graph, reconstructed its causal edges, or refrained from claims the graph did not support. The framework proposed here treats the biomedical graph both as a retrieval resource and as an experimental instrument by varying the type of retrieved context across four conditions. Treating individual scientific claims as first-class, addressable objects has a long tradition in biomedical and semantic-web research, well predating the current wave of LLM evaluation. The nanopublication model [16] argued that the atomic unit of scientific communication should not be the paper but the assertion itself, packaged with its own provenance and publication metadata so that each claim becomes independently citable, versionable, and machine actionable. The Biolink Model [17] operationalized a related idea for translational biomedicine by giving associations between biological entities their own typed identity, with qualifiers (subject/object aspect, negation, temporal and population qualifiers) and slots for supporting evidence, so that the same subject–predicate–object triple can carry distinct semantics depending on its qualifying context. At the infrastructure level, the W3C PROV data model [18] formalized the provenance chain of any statement who asserted it, from what source, through which activity and has since become the interoperability layer over which many domain-specific evidence models are expressed. On the curation side, resources such as CIViC [19] and ClinGen [20] made evidence attachment concrete for clinical genomics: each variant–disease or gene–disease assertion is annotated with an explicit evidence code, a level of clinical validity, and links to the primary literature that supports or disputes it, so that downstream systems can reason not only over the claim but also over its epistemic status. The representational substrate needed to support this style of modelling has also matured. In the RDF world, RDF-star and its query counterpart SPARQL-star [21] extend the triple model with the ability to make statements about statements, so that provenance, confidence, and context can be attached to a specific claim without inflating the graph with auxiliary reification nodes. In property-graph systems such as Neo4j graph databases, the same effect is achieved through relationship properties. Across both stacks the design space for "a claim plus its provenance" is therefore now well understood, and reified assertions are a familiar biomedical modelling idiom. What has remained comparatively under-developed is the use of these assertion-level identifiers as the unit of benchmark scoring for generative models. Existing benchmarks that leverage biomedical graphs typically use them at retrieval time to fetch context but then score model output at the level of a final answer string or a free-text rationale, which forces evaluators either to string-match against surface forms or to hand-align model claims to unlabeled edges. This severs the connection between the graph's provenance layer and the evaluation layer: a benchmark cannot say precisely which biomedical claim the model was expected to reconstruct, nor which claim it did reconstruct, nor which evidence source it was expected to cite. The framework proposed here closes that gap. Every biomedical relationship in the graph is paired with a first-class Assertion node carrying a stable assertion_id, and every gold structure causal pathway, adverse-effect path, contraindication scope, evidence support is expressed as an ordered set of these identifiers. This makes causal-edge precision, recall, and F1, as well as evidence accuracy and unsupported-claim rate, computable against unambiguous references rather than against anonymous graph edges or free-text mentions, and it aligns evaluation directly with the same provenance layer that the biomedical community has been building for nearly two decades. 2.1. Causal reasoning, causal health AI, and LLMs The shift from predictive to causal health AI is motivated from three directions: the statistical literature on treatment-effect estimation and its identification assumptions [22, 23], the applied argument that models used to guide action must be evaluated against interventional rather than merely predictive targets [24], and a recent line of work probing whether LLMs themselves can reason causally [25, 26, 27]. Two findings from that last strand motivate the present framework. First, LLMs can approximate several forms of causal reasoning when the underlying structure is provided or strongly implied, but they also produce fluent, plausible-sounding causal claims not supported by the evidence and degrade sharply when memorized patterns are blocked. Second, existing causal-reasoning benchmarks are largely disease-agnostic and abstract; they characterize general causal competence but do not test whether a model behaves like an evidence-grounded interventional system in a concrete clinical setting with candidate actions, mechanistic pathways, harms, contraindications, and cited evidence. The proposed framework operationalizes this latter requirement: its scenario categories probe distinct causal-reasoning failure modes of clinical relevance, and every scenario carries a graph-anchored gold pathway expressed as ordered assertion_ids, so the model's reconstructed pathway is scored against a defined causal target rather than an unstructured rationale. 2.2. Hallucination, attribution, and unsupported-claim evaluation A parallel line of work evaluates whether generative outputs are factual, attributable, and supported by their cited sources. FActScore [28] decomposes long-form outputs into atomic claims and scores each against a reference corpus; TruthfulQA [29] targets memorized falsehoods; HaluEval [30] scales hallucination measurement across task types; and attribution/faithfulness pipelines [31, 32] score whether cited sources entail the claims they support. In the biomedical setting, Med-HALT [33] and related surveys [34] document that clinical hallucinations recur across model families and task types. Two limitations of this literature matter here. First, these evaluators are anchored to free-text sources rather than to a structured knowledge substrate, so a claim is judged by textual entailment rather than against a defined mechanistic relation. Second, they operate on the assertoric surface of the output without a native representation of the underlying causal-mechanism claim being made. The framework specializes the same core idea an unsupported claim is one whose warrant is not present in the sanctioned context to causal mechanism claims and anchors that judgement to the graph. An unsupported claim is one whose subject–predicate–object pattern is not present in the retrieved reified-assertion subgraph. This yields a reproducible unsupported-claim rate, computed automatically on the same scoring pass as causal- edge precision/recall/F1, adverse-effect F1, and evidence accuracy. 3. The Framework The proposed framework, shown end-to-end in Figure 1, is designed to answer one question in a controlled way: given a clinical scenario, does the model produce a defensible recommendation, and can we tell? Answering it requires four things; a knowledge base that can be cited, a way to pull only the relevant part of that knowledge base per scenario, a fixed set of ways to hand that knowledge to the model, and a scoring step that compares the model's output back to the same knowledge base. The four framework components, each corresponding to one of these requirements, are defined below, followed by detailed explanations in the subsequent sections. Figure 1. Framework pipeline from reified-assertion knowledge graph to scored model output, showing the four coupled components: knowledge graph, scenario-conditioned subgraph extraction, controlled grounding conditions (C1–C4), and automated scoring anchored to assertion_ids. Domain causal and evidence knowledge graph with reified assertions (details in section 3.2): A curated biomedical graph in which every claim (for example, "ACE inhibitors decrease blood pressure") is stored not just as an edge between two entities but as its own node with a stable identifier, provenance, confidence, validation state, and context. Serves as the single source of truth for the evaluation. Because every claim has an identifier, we can say precisely which claim the model was supposed to use and which claim it used, the scoring step later refers to these identifiers directly. Scenario-conditioned subgraph extraction (section 3.4). A set of parameterized cypher queries executed against Neo4j. For a given scenario (patient context, decision target, candidate actions), it retrieves only the slice of the graph that is relevant, the relevant drugs, mechanisms, outcomes, contraindications, adverse effects, and evidence sources as a self-contained subgraph. The model is never asked to search the whole graph; scenario relevance is enforced by retrieval, so we can attribute later differences to how that subgraph is presented rather than to what was found. For any input clinical scenario, parameterized cypher queries executed against Neo4j graph database return the relevant reified-assertion subgraph. It is this extracted subgraph not the full graph that is composed into the model's context, so that scenario relevance is enforced at retrieval rather than left to the model. Controlled grounding conditions (section 3.5). Four fixed ways of turning the retrieved subgraph into the model's prompt: C1 no subgraph at all (ungrounded baseline), C2 the knowledge-graph view, C3 the causal-graph view, and C4 the integrated causal + knowledge-graph view. Runs the same scenario four times, changing only the type of context supplied. Any behavioral difference across C1–C4 must come from context organization, not from the scenario, prompt wording, or context volume, so context organization becomes a manipulable variable. Automated scoring pipeline (section 3.7). A single scoring pass over the model's parsed output. This component compares the output against gold structures that are themselves expressed as sets of assertion_ids, and produces intervention accuracy, causal edge precision/recall/F1, adverse-effect F1, contraindication recall, evidence accuracy, unsupported-claim rate, and uncertainty correctness. All seven metrics are computed on the same reference (the same graph, and the same identifiers), so they can be compared directly instead of coming from separate evaluations with separate assumptions. The proposed framework is scenario-source-agnostic. To stress-test it we pair it with two evaluation instruments: a category-balanced scenario generator (section 3.6) and a repeated-measures protocol in which each scenario is presented under all four grounding conditions (section 3.5). 3.2. Domain causal and evidence knowledge graph A domain-specific biomedical graph was constructed to support intervention-oriented reasoning in cardiovascular care (figure 2). Nodes represent diseases, drugs, non-drug interventions, protein targets, biological processes, risk factors, clinical outcomes, adverse effects, contraindications, populations, evidence sources, studies, and reified assertions. The final entity table contains 117 biomedical and contextual nodes, comprising 10 diseases, 15 drugs, 10 interventions, 15 protein targets, 13 biological processes, 13 risk factors, 10 clinical outcomes, 14 adverse effects, 8 contraindications, and 9 populations. The graph was synthesized from the underlying biomedical evidence and subsequently reviewed by two medical annotators to assess the accuracy and consistency of the entity annotations. Fig. 2 Partial visualization of the constructed Neo4j biomedical knowledge graph, showing selected biomedical entities and their interconnected relationships. 3.2.1. Assertion reification: every claim as a first-class node The central design decision is that every biomedical relationship is represented not only as a direct Neo4j edge but also as a reified assertion node. In a conventional property-graph encoding, a claim such as "an ACE inhibitor decreases blood pressure" would be stored only as an edge, (Drug)-[:CAUSES_DECREASE_IN]->(Outcome). In our graph, the same claim is additionally stored as a dedicated Assertion node with a stable identifier (e.g., ASR_0033) linked to its endpoints through explicit HAS_SUBJECT and HAS_OBJECT relationships: • (:Assertion assertion_id: "ASR_0033") • (:Assertion)-[:HAS_SUBJECT]->(:Drug) • (:Assertion)-[:HAS_OBJECT]->(:Outcome) Each assertion node carries structured properties: predicate, relation_class, direction, causal_status, evidence_level, confidence, validated, context, population_id, temporality, and a source_databases_json field. These attributes attach naturally to a node as shown in Figure 3. Fig. 3 Assertion reification schema. an edge-only representation of a biomedical claim, in which provenance, confidence, population, and validation cannot be attached flexibly. b Reified representation: the claim is a first-class Assertion node with a stable identifier, linked to its subject and object through HAS_SUBJECT and HAS_OBJECT, and to one or more Evidence Source nodes through SUPPORTS Reification is used for four reasons that are directly consequential for this study. First, a biomedical claim is not fully characterised by the pair of entities it connects; it also has a causal status, a confidence, a validation state, an applicable population, and a temporal scope. These live on the claim itself rather than being duplicated across parallel edges. Second, evidence links attach to a specific claim rather than to an opaque edge, expressing patterns such as (:EvidenceSource)- [:SUPPORTS]->(:Assertion). Third, when several sources back the same relationship, an edge-only model tends to accumulate near-duplicate edges; reification preserves a single canonical claim node and lets many evidence links point to it, keeping the graph compact and provenance auditable. Fourth, and most consequential for evaluation, stable assertion_ids make gold structures precise and referable: a gold pathway can be expressed as an ordered list such as ["ASR_0019", "ASR_0010", "ASR_0011", "ASR_0013"] as shown in Figure 4, which is directly usable by the scoring pipeline of section 3.7. Direct typed edges (CAUSES_DECREASE_IN, TARGETS, MAY_CAUSE_ADVERSE_EFFECT, and others) are retained alongside the reified assertions. This dual representation is deliberate: direct edges support fast traversal and simple Cypher retrieval for the C2, C3, and C4 grounding conditions, while assertion nodes carry provenance, validation, and gold-path referencing. Every direct edge in the graph is paired with a corresponding assertion node that shares the same subject, predicate, and object. Fig. 4 Worked example of a gold causal pathway. Each biomedical entity is connected to the next through a reified assertion node with a stable identifier, so that a gold pathway is expressible as an ordered list of assertion_ids and consumed directly by the scoring metrics for causal edge precision, recall, and F1. The expanded graph contains 257 assertions and 15 evidence sources, including 11 study-level records. Direct biomedical relationships are preserved for traversal efficiency while remaining linked to assertion objects through stable identifiers. Relationship totals include 15 TARGETS edges, 14 PARTICIPATES_IN edges, 65 CAUSES_INCREASE_IN edges, 82 CAUSES_DECREASE_IN edges, 18 ASSOCIATED_WITH edges, 15 INDICATED_FOR edges, 26 MAY_CAUSE_ADVERSE_EFFECT edges, 13 CONTRAINDICATED_FOR edges, and 9 MODIFIES_EFFECT_OF edges. Together with the reified layer, this representation supports efficacy, safety, contextual applicability, and evidence provenance, each addressable independently by the scoring pipeline. 3.3. Grounding conditions Four grounding conditions determine how the extracted subgraph is composed into the model's context. • C1 (ungrounded). Only the scenario and intervention task. • C2 (knowledge-graph). Scenario plus knowledge-graph context and evidence provenance, without full causal-path emphasis. • C3 (causal-graph). Scenario plus validated causal pathways, with limited contextual evidence detail. • C4 (integrated). Scenario plus integrated causal structure, mechanistic context, evidence provenance, contraindications, adverse-effect information, and uncertainty-relevant context. The conditions vary the type of retrieved context rather than its volume, so that observed differences can be attributed to context organisation rather than to prompt style. C1 bypasses subgraph extraction by design; it is the ungrounded baseline. 3.4. Scenario-conditioned subgraph extraction and prompt execution Given a clinical scenario, framework component 2 retrieves the relevant reified-assertion subgraph from Neo4j using parameterized cypher queries prepared separately for C2, C3, and C4. The prompt itself is then constructed deterministically from the extracted subgraph the same scenario header, candidate options, output schema, and JSON contract are shared by all four conditions, and only the Grounded context block changes. C1 bypasses extraction by design and receives no graph content. This isolation is what allows behavioral differences across conditions to be attributed to context organization rather than to prompt wording, task framing, or output schema. The same extraction interface applies unchanged to a scenario supplied from any external source (a clinician-entered vignette, an EHR-derived summary, or an external benchmark item). To make the mechanism concrete, we walk through a single scenario, SCN_0001 (a direct-effect item: adult with persistent hypertension, target outcome = systolic blood pressure, candidate options = lisinopril, atenolol, smoking cessation), and show what each condition receives in the Grounded context block of the generated prompt. Content outside that block is identical across the four conditions and is omitted for brevity. C1 ungrounded: no subgraph supplied in the prompt context section, refer to example in appendix A for full C1 prompt. The model sees only the scenario, the candidate options, and the JSON output schema. There is no assertion identifier space to reference, so the model cannot populate causal_path_assertion_ids or supporting_evidence_ids truthfully; it must either leave them empty or invent identifiers. This baseline exists to measure what happens when the panel has no graph to score against. C2 knowledge-graph context: the extractor returns candidate-level indication, association, and adverse-effect assertions with evidence identifiers, but no mechanistic chain. C2 gives the model the knowledge needed to justify a choice (indications, associations, one adverse effect) but not the causal decomposition of how the intervention reaches the outcome. C3 causal-graph context: the extractor returns the validated causal chain of assertion_ids from intervention to outcome (target → biological process → intermediate outcome → clinical outcome), plus a directly attached adverse-effect assertion. C3 exposes the reified mechanism directly. The gold causal pathway for scoring is exactly ["ASR_0019", "ASR_0010", "ASR_0011", "ASR_0013"], so a correct C3 response can populate causal_path_assertion_ids verbatim. C4 integrated causal-knowledge-graph context: the extractor returns the C3 causal chain, but each assertion is annotated with its evidential and provenance status (guideline vs. validated causal edge, evidence tier, source-database attribution), and the surrounding treatment and safety context is retained. C4 supplies the same underlying assertion_ids as C3 but presents them alongside the knowledge-graph indication (ASR_0028) and adds a typed evidential label to each row (guideline, validated causal edge with evidence tier and source, safety evidence). This is the condition designed to test whether a model can not only reconstruct the mechanism but also weigh evidence quality and preserve provenance in its cited support. Across the four conditions, three properties are preserved by construction. First, the scenario, candidate options, and JSON output schema are byte-identical; only the Grounded context block varies. Second, every assertion identifier in every condition resolve to a node in the same reified-assertion graph, so the scoring pipeline (§3.7) computes causal edge precision/recall/F1, evidence accuracy, and unsupported-claim rate against the same reference regardless of which condition produced the output. Third, C1's absence of grounded context is not a bug in the extraction step but the intended baseline: it produces the row of the metric panel that tells us what a model does when it has no graph to be measured against. 3.5. Evaluation methodology: repeated-measures protocol Each scenario is presented under all four grounding conditions in the same run, so within-scenario condition deltas can be computed. For a single model, the full benchmark comprises 320 runs (80 scenarios × 4 conditions). This protocol is an evaluation instrument used to demonstrate the framework; it is not itself a component of the framework. 3.6. Evaluation instrument: category-balanced scenario generator The framework itself accepts any clinical scenario as input. To evaluate the framework in a controlled way, however, we need input scenarios whose gold causal, safety, and evidential structure is known in advance, so that the scoring pipeline (section 3.7) can operate against defined targets. We therefore developed a category-balanced scenario generator that produces 80 structured intervention items directly from the knowledge graph. The generated scenarios are evenly balanced across eight reasoning categories, each chosen to probe a distinct causal- reasoning failure mode: direct effect, mediated effect, multiple pathways, confounded association, competing interventions, contraindication-sensitive reasoning, adverse-effect pathways, and incomplete or uncertain evidence. Each scenario defines the clinical context, the decision target, the candidate actions, the expected gold intervention, and the gold causal and evidential structure expressed as ordered lists of stable assertion_ids. Category balance is enforced so that no single class of clinical reasoning problem can dominate an aggregate metric, and so that category-level dissociations across the metric panel become observable. The generator is an evaluation instrument, not a framework component; in downstream deployment it is replaced by scenarios drawn from clinicians, EHR summaries, or external benchmarks. 3.7. Automated scoring and evaluation pipeline Model outputs are not scored as free text. Each response is first parsed into a fixed set of structured fields the recommended intervention, the reasoning pathway, the adverse effects surfaced, the contraindications noted, the evidence sources cited, and any explicit statement about evidential sufficiency. These fields are then compared, on a single scoring pass, against gold structures that are themselves expressed as sets of `assertion_id`s drawn from the same reified-assertion graph used for grounding. Anchoring both sides of the comparison on the same identifier space is what makes the pipeline reproducible: no free-text matching against gold pathways is required for the causal, adverse-effect, contraindication, or evidence metrics. The primary outcome is intervention accuracy, and its stricter companion preferred intervention accuracy. Intervention accuracy asks whether the model selected an appropriate action from the candidate set defined by the scenario; preferred intervention accuracy asks whether it selected the specific gold intervention marked as preferred when the scenario admits more than one clinically acceptable choice. This distinction is deliberate: in a decision-support setting it is possible to be "not wrong" (any of several acceptable actions) without being "right" (the action the guidelines actually prefer for that context), and separating the two metrics lets the framework detect that gap rather than collapse it into one number. Causal edge precision, recall, and F1 score how faithfully the model reconstructed the mechanism that justifies its recommendation. The scenario carries a gold causal pathway, expressed as an ordered set of `assertion_id`s that link the intervention through its molecular target, biological process, and clinical outcome. Precision asks what fraction of the causal edges the model stated are present in the gold set; recall asks what fraction of the gold edges the model surfaced; F1 combines the two. Together they distinguish a model that arrives at the correct action by inventing a plausible sounding but unsupported mechanism (high accuracy, low precision) from one that arrives at the correct action through the mechanism the graph actually sanctions (high accuracy, high precision and recall). Adverse-effect F1 and contraindication recall score the safety axis. Adverse-effect F1 is computed only on scenarios that have a gold adverse-effect target and asks whether the model surfaced the specific `MAY_CAUSE_ADVERSE_EFFECT` claims that apply to the recommended intervention. Contraindication recall asks, on contraindication-sensitive scenarios, whether the model surfaced the contextual constraint that should have blocked (or preferred against) a competing option for example, avoiding an ACE inhibitor in a pregnant patient. Reporting these separately from intervention accuracy is important because a model can pick the correct action while omitting the harm or the contraindication that made it the correct action; a benchmark that only checked the final action would score that response identically to a fully justified one. Evidence accuracy and unsupported claim rate score the evidential axis. Evidence accuracy asks whether the sources the model cited are in fact the `EvidenceSource` nodes that `SUPPORTS` the assertions the model relied on; a citation to an existing source is not enough it must actually support the claim being made. Unsupported claim rate (lower is better) asks the complementary question: what fraction of the causal or mechanistic claims produced by the model do not have a corresponding subject–predicate–object pattern in the retrieved graph context. Together, these two metrics separate "the model cited its sources correctly" from "the model made claims the graph did not sanction," which are different failure modes and require different fixes. Uncertainty correctness scores whether the model recognized that the evidence was insufficient when the scenario was designed to test exactly that. On the incomplete-evidence category, the gold behavior is not a specific action but an explicit acknowledgment that the available evidence does not support a confident recommendation. Measuring this separately is what prevents the benchmark from silently rewarding a confident-sounding wrong answer over a correctly hedged one, and it is what let the pilot detect the dissociation between recognizing insufficiency (uncertainty correctness 1.000) and producing the gold intervention decision (intervention accuracy 0.000) in that category. Because all seven metrics are computed on the same scoring pass, against the same graph, and against the same gold `assertion_id` sets, they can be read as one coherent panel rather than as seven independent evaluations. This is the property the framework relies on downstream: when C1 and C4 differ on intervention accuracy but differ in the opposite direction on causal F1 and unsupported claim rate, that pattern is directly interpretable because both numbers came from the same run against the same reference. A benchmark that assembled these metrics from separate scoring passes could not make that inference cleanly. Model outputs are parsed into structured response fields and scored against gold structures expressed as sets of assertion_ids, example in Table 1. The primary outcomes are intervention accuracy and preferred intervention accuracy. Additional outcomes are causal edge precision, causal edge recall, causal edge F1, adverse-effect F1, contraindication recall, evidence accuracy, unsupported claim rate (lower is better), and uncertainty correctness. These outcomes are selected to expose multiple dimensions of intervention-oriented reasoning on the same scoring pass: whether the model chose an appropriate action, whether it justified that action with defensible causal structure, whether it surfaced harms and contraindications, whether it cited support accurately, whether it hallucinated unsupported mechanisms, and whether it correctly recognized insufficiency of evidence. Table 1. Worked example of one scenario passing through the scoring pipeline. The scenario is drawn from the direct-effect category; identifiers (ASR_*, EV_*) refer to nodes in the reified-assertion graph. The example is illustrative and does not affect the aggregate numbers in Section 4. Stage Content Scenario input 62-year-old with essential hypertension, LDL 138 mg/dL, no diabetes, eGFR 78, not pregnant. Decision target: choose the preferred first-line antihypertensive from ACE inhibitor, thiazide diuretic, beta-blocker, calcium-channel blocker. Gold intervention ACE inhibitor (lisinopril). Gold causal pathway [ASR_0019, ASR_0010, ASR_0011, ASR_0013] = ACE inhibitor TARGETS ACE → PARTICIPATES_IN RAAS activity → CAUSES_DECREASE_IN blood pressure → CAUSES_DECREASE_IN cardiovascular event risk. Gold adverse effects ASR_0087 (dry cough), ASR_0091 (hyperkalemia). Gold contraindication scope Nonapplicable in this scenario (patient is not pregnant); metric is undefined. Gold evidence sources EV_0007 (JNC-8 guideline), EV_0011 (HOPE trial). Expected uncertainty behavior Confident recommendation (evidence is sufficient). Raw model output (abridged) "I recommend lisinopril. It blocks angiotensin I production and thereby lowers blood pressure, which reduces cardiovascular events. It also lowers LDL cholesterol. The main adverse effect to counsel is a dry cough. Supporting evidence: HOPE trial." Parsed structured fields intervention = "lisinopril"; causal_pathway_ids = [ASR_0019, ASR_0011, ASR_0013]; adverse_effect_ids = [ASR_0087]; contraindication_ids = []; evidence_source_ids = [EV_0011]; unsupported_free_text_claims = ["lowers LDL cholesterol"]; uncertainty_flag = false. Scoring against gold intervention accuracy = 1.00 (lisinopril ∈ ACE-inhibitor class). Preferred intervention accuracy = 1.00. Causal edge precision = 3/3 = 1.00; causal edge recall = 3/4 = 0.75 (missed ASR_0010, the RAAS mediator); causal edge F1 = 0.857. Adverse-effect F1 = 0.667 (surfaced ASR_0087, missed ASR_0091). Contraindication recall = NA. Evidence accuracy = 1/1 = 1.00 (EV_0011 supports the cited pathway). Unsupported claim rate = 1/4 = 0.25 (the LDL claim has no matching S-P-O in the retrieved subgraph). Uncertainty correctness = 1.00 (no hedge expected, none produced). All seven metrics fall out of one parse-then-compare pass over the same output. The model would score 1.00 on intervention accuracy alone, but the panel simultaneously reveals a missed mediator (recall gap), a missed adverse effect (safety gap), and a fabricated causal claim (unsupported-claim signal). These are the failure modes that a single-endpoint accuracy report would silently hide, and they are surfaced here because every metric is anchored on the same assertion_id reference. 3.8. Experimental Setup The framework and instruments were implemented as python scripts within a single workspace. The graph substrate uses Neo4j, populated from CSV source tables (entities.csv, assertions.csv, evidence_sources.csv, assertion_evidence.csv, and related) via a bulk import script and cypher validation queries. The model under test in the pilot is gpt-5.4, invoked through the OpenAI Batch API to run the full 320-prompt matrix in one job. Prompts are constructed deterministically from the retrieved subgraph; there is no per-scenario prompt engineering. Scoring is fully automatic and is anchored to assertion_ids so that no free-text matching against gold structures is required for the causal, adverse-effect, contraindication, or evidence metrics. Of the 320 planned runs, 313 were successfully scored; the remaining seven were skipped because of missing model responses associated with API failure or incomplete output capture, yielding condition-specific counts of 77 (C1), 79 (C2), 78 (C3), and 79 (C4). 4. Results The metric panel separates the four conditions along interpretable, non-redundant axes, which is the property the framework is intended to have. Table 2 summarises the overall panel. The n column is the number of scored scenarios for which the metric is defined (for example, causal edge precision is undefined for C1 because no causal edges are produced without grounding; adverse-effect F1 is defined only on scenarios with a gold adverse-effect target). Table 2. Overall metric panel across grounding conditions in the cardiovascular pilot demonstration (gpt-5.4). NA indicates the metric is undefined for that condition given the scoring rule. Lower is better for unsupported claim rate; higher is better for all other metrics. Metric C1 (mean, n) C2 (mean, n) C3 (mean, n) C4 (mean, n) Intervention accuracy 0.948, 77 0.873, 79 0.808, 78 0.886, 79 Preferred intervention accuracy 0.948, 77 0.873, 79 0.808, 78 0.886, 79 Causal edge precision NA, 0 0.522, 79 0.808, 78 0.806, 79 Causal edge recall 0.000, 77 0.785, 79 0.808, 78 0.886, 79 Causal edge F1 NA, 0 0.579, 79 0.808, 78 0.838, 79 Adverse-effect F1 NA, 0 0.733, 20 0.822, 30 0.833, 40 Contraindication recall 1.000, 10 1.000, 10 1.000, 10 0.800, 10 Evidence accuracy NA, 0 0.705, 79 0.679, 78 0.738, 79 Unsupported claim rate (↓ better) NA, 0 0.259, 79 0.192, 78 0.114, 79 Uncertainty correctness (overall) 0.130, 77 0.114, 79 0.115, 78 0.114, 79 The ungrounded baseline C1 obtains the highest raw intervention accuracy (0.948). The grounded conditions produce lower raw intervention accuracy (0.873 for C2, 0.808 for C3, 0.886 for C4). On the causal and evidential axes, the ordering is different: C4 achieves the highest causal edge F1 (0.838), the highest adverse-effect F1 (0.833), the highest evidence accuracy (0.738), and the lowest unsupported claim rate (0.114). We report these numbers as evidence that the multi-metric panel exposes behavior that a single-endpoint accuracy report would hide: on this pilot, ranking the conditions by intervention accuracy alone would have placed C1 first, but that ordering carries no information about mechanism fidelity, evidence use, or unsupported claim generation, all of which the framework measures separately. The conditions are not collinear on the panel. We do not draw a clinical-performance conclusion from the ranking itself. 4.3. Category-level behavior of the metric panel The scenario categories produced distinct metric profiles, supporting the framework's design premise that intervention reasoning is not a single competency and should not be measured with a single number. In multiple-pathway scenarios, C4 recorded perfect intervention accuracy and causal recall together with evidence accuracy of 1.000 and an unsupported claim rate of 0.000. In adverse-effect pathway scenarios, C4 recorded perfect intervention accuracy, perfect causal precision and recall, and adverse-effect F1 of 1.000. In contraindication-sensitive scenarios, C4 maintained perfect intervention accuracy and perfect causal pathway recovery. These are useful as evidence that the framework's category structure interacts meaningfully with the grounding manipulation; they are not clinical-performance guarantees. On the incomplete or uncertain evidence category, all three grounded conditions scored 0.000 on intervention accuracy while uncertainty correctness was 1.000 across all conditions. This dissociation between correctly recognising evidential insufficiency and producing the gold intervention decision is precisely the kind of finding the multi-metric panel is designed to expose, and it identifies an explicit benchmark-design choice for the community: whether abstention and uncertainty acknowledgment should be scored as separate targets from final action selection. C3 showed a distinct profile on multiple-pathway scenarios (intervention accuracy 0.400, causal precision 0.400, causal recall 0.400, evidence accuracy 0.400, unsupported claim rate 0.600). We report this as evidence that the framework separates causal-only and integrated-causal grounding along interpretable axes rather than collapsing them; whether the pattern generalises across models is a claim reserved for the multi-model phase. 4.4 Example paired-condition contrast (C4 vs. C2) To show how the framework's paired within-scenario deltas behave, we examined C4 relative to C2 (Table 3). Relative to C2, C4 shows a small positive delta on intervention accuracy (+0.013), a larger delta on causal precision (+0.284), positive deltas on causal recall (+0.101), causal F1 (+0.259), adverse-effect F1 (+0.100), and evidence accuracy (+0.033), and a favorable negative delta on unsupported claim rate (−0.145). Contraindication recall is 0.200 lower in C4; the diagnostic in section 4.5 attributes this to output-schema semantics in a small number of scenarios rather than to missing safety knowledge in the graph. The purpose of reporting this contrast is to demonstrate that the paired-delta layer of the framework operates end-to-end and yields metric-specific rather than lumped signal. It is not a claim that integrated grounding is superior to knowledge-graph grounding for clinical use; that claim would require the multi-model phase. Table 3. Illustrative within-scenario paired deltas for the C4 vs. C2 contrast in the pilot. Positive deltas indicate improvement for C4 on the metric; for unsupported claim rate, a negative delta is favorable. Metric C2 mean C4 mean Δ (C4 − C2) Direction favorable to C4? Intervention accuracy 0.873 0.886 +0.013 Yes Causal edge precision 0.522 0.806 +0.284 Yes Causal edge recall 0.785 0.886 +0.101 Yes Causal edge F1 0.579 0.838 +0.259 Yes Adverse-effect F1 0.733 0.833 +0.100 Yes Evidence accuracy 0.705 0.738 +0.033 Yes Unsupported claim rate 0.259 0.114 −0.145 Yes (lower is better) Contraindication recall 1.000 0.800 −0.200 No (see section 4.5 diagnostic) 4.5. Diagnostic value of the framework Two diagnostic observations illustrate that the framework surfaces evaluation-design questions in addition to model behavior. First, the high intervention accuracy of C1 was partly attributable to the incomplete or uncertain evidence category, where the grounded conditions scored 0.000 on intervention accuracy despite scoring 1.000 on uncertainty correctness. Without a benchmark that measures uncertainty correctness separately, an ungrounded model could appear superior on a single accuracy endpoint even when the gold behavior was abstention or uncertainty-calibrated reasoning. The framework makes that failure mode explicit and forces the benchmark designer to choose whether to reward the recognised- uncertainty response. Second, the lower contraindication recall observed in C4 was traced to two scenarios in which the model recommended the correct intervention (amlodipine) but returned an empty contraindication field even though pregnancy was expected as the gold contraindication context. Inspection of the grounded prompts confirmed that the relevant contraindication information was present in the retrieved context. The most likely explanation is that the model interpreted the contraindication output field narrowly (contraindications of the selected intervention), whereas the scorer interpreted it broadly (a scenario-level contraindication context that helped rule out competing options). We report this because it illustrates that the framework detects output-schema-driven measurement artefacts distinct from missing knowledge, which is a class of failure that a coarser benchmark could not distinguish from a genuine safety gap. 5. Discussion The principal contribution of this manuscript is an evaluation framework, not a clinical-performance claim. The framework has four components: a provenance-preserving causal knowledge graph with reified assertions, a scenario-conditioned subgraph extraction step, a controlled set of grounding conditions that vary the type of retrieved context, and a multi- dimensional automatic scoring pipeline anchored to stable assertion identifiers. Together they define how a claim moves from the underlying knowledge base into a model output that can be scored on causal, safety, and evidential axes. To test the framework in a controlled way, we additionally develop a category-balanced scenario generator that supplies structured input items with graph-anchored gold answers, and a repeated-measures design that presents each scenario under all four conditions; both are evaluation instruments, not framework components. The pilot demonstrates that this pipeline is implementable end-to-end on a single workspace and that its metric panel is discriminative across the conditions and scenario categories it is designed to distinguish. Reporting only intervention accuracy would have hidden most of the between-condition behavior we observe, including differences in causal reconstruction, evidence use, adverse-effect recovery, and unsupported-claim generation. This is the property a benchmark for intervention-oriented healthcare LLMs needs, independent of which model is under test. Because the pilot instantiates the framework on a single model, it does not, and is not intended to, establish that any grounding condition is clinically superior. A common alternative reading of the descriptive numbers would take C1's high intervention accuracy as evidence that ungrounded models are sufficient. The framework itself is the response to that reading: the causal, evidential, adverse-effect, contraindication, and unsupported-claim axes are measured explicitly precisely so that a single-endpoint answer cannot silently substitute for decision-relevant reasoning. The framework is positioned within the broader shift from predictive to causal health AI. In that setting, evaluation infrastructure needs to detect whether a generative system connects its recommendations to explicit causal assumptions, supporting evidence, harms, and uncertainty, rather than whether a single answer matches a gold label. The framework operationalises that requirement by asking, for each scenario, not only which intervention was chosen but also which causal edges the model reconstructed, which adverse effects and contraindications it surfaced, which evidence it cited, and where it produced claims unsupported by the retrieved graph. Because the type of retrieved context is a manipulable variable, the framework can in principle discriminate models and prompting strategies that are sensitive to context organisation from those that are sensitive only to context volume. Three implications follow. First, benchmarks for healthcare LLMs should treat final-answer accuracy as one axis among several, and should measure unsupported claims, evidence alignment, harms, contraindications, and uncertainty as first- class outcomes. Second, structured biomedical graphs are usefully treated not only as retrieval resources but as experimental instruments whose contents and organisation can be varied to test whether a model responds to the type of structure supplied. Third, output-schema design is a component of benchmark validity, not a peripheral engineering detail: the contraindication diagnostic showed that a model may possess the relevant safety context yet fail to communicate it in the expected form, and a benchmark that cannot separate this from a genuine knowledge gap is measuring the wrong thing. The framework also exposes a task-design question for uncertainty-aware causal health AI. In the incomplete-evidence category the grounded conditions recognised insufficiency correctly (uncertainty correctness 1.000) yet failed to produce the gold intervention decision. This is not primarily a language-generation failure; it is a decision-policy and scoring-design choice about whether abstention or explicit "insufficient evidence" outputs should count as correct interventions. The framework makes that choice explicit and therefore auditable. 5.1. Limitations Several limitations follow directly from the framing of this work as a benchmark and methodology contribution. First, the empirical component is a single-model pilot; it demonstrates that the framework is implementable and discriminative, not clinical performance for any model. Comparative model claims, and effect-size estimates are deferred to the planned multi- model repeated-measures phase. Second, the benchmark is a designed evaluation environment in a cardiovascular sub- domain and does not, on its own, establish external validity for live clinical deployment or for other clinical areas; the framework is intended to be reinstantiated per domain. 5.2. Future work Future work will focus on two directions. First, the framework will be evaluated across a broader set of frontier and open- weight LLMs using a repeated-measures design. Cross-domain instantiation of the framework beyond cardiovascular care, to test whether the category structure and metric panel transfer. 6. Conclusion We propose a graph-centered grounding framework for evaluating intervention-oriented LLM behavior in healthcare. The framework integrates four components: a provenance-preserving causal knowledge graph with reified assertions, scenario- conditioned subgraph extraction, controlled grounding conditions that vary the retrieved context, and a multidimensional automatic scoring pipeline linked to stable assertion_ids. We also develop a category-balanced scenario generator that produces structured evaluation cases with graph-anchored gold answers across eight causal-reasoning failure modes. The generator serves as an evaluation instrument rather than a component of the framework. A cardiovascular pilot comprising 313 scored runs with GPT-5.4 demonstrates that the framework can be implemented end to end and that its evaluation metrics distinguish performance across grounding conditions and scenario categories. The results also show the value of multidimensional evaluation by revealing patterns that would be obscured by a single accuracy measure, including the ability to distinguish output-schema misalignment from substantive knowledge gaps. The pilot is therefore intended to demonstrate the interpretability and usefulness of the evaluation framework rather than to establish definitive clinical performance for a particular model. Comparative conclusions across models are reserved for the planned multi-model repeated-measures study, for which the proposed framework provides the underlying evaluation infrastructure. Statements and Declarations Funding No funding was received to assist with the preparation of this manuscript. [Revise if applicable and include grant number(s).] Competing interests The authors have no relevant financial or non-financial interests to disclose. [Revise if applicable.] Ethics approval This study did not involve human participants, human data, or human biological material. The benchmark consists of synthetic clinical scenarios generated from a curated biomedical knowledge graph and evaluated against automated scoring metrics; no patient-level data were collected or analysed. Institutional review board approval was therefore not required. Consent to participate and consent to publish Not applicable. This work does not include data from identifiable individuals. Data availability The benchmark source tables (entity, assertion, evidence-source, and scenario tables), the Neo4j import files, and the aggregated per-condition results tables that support the findings of this study are available on request to the corresponding author. Code availability The graph construction, prompt generation, retrieval, experiment execution, automated scoring, and analysis scripts used in this study are provided in the same companion repository under an open-source license [license to be added upon acceptance]. Author contributions All authors contributed to the study conception and design. [Author One] led the framework design, graph construction, and manuscript drafting. [Author Two and Three] developed the scoring pipeline and analytic summaries. [Author Four] contributed to scenario generation and reviewed the manuscript. Declaration on Generative AI An AI assistant was used for AI-assisted copy editing (readability, grammar, and formatting) of the manuscript text. The authors reviewed and edited the content as needed and take full responsibility for the publication's content. References 1. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P.: What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11(14), 6421 (2021). https://doi.org/10.3390/app11146421 2. Pal, A., Umapathi, L.K., Sankarasubbu, M.: MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Proceedings of the Conference on Health, Inference, and Learning (CHIL). Proc. Mach. Learn. Res. 174, 248–260 (2022) 3. Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., Lu, X.: PubMedQA: A dataset for biomedical research question answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 2567–2577. Association for Computational Linguistics, Hong Kong (2019). https://doi.org/10.18653/v1/D19-1259 4. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding. In: International Conference on Learning Representations (ICLR) (2021) 5. Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., et al.: Large language models encode clinical knowledge. Nature 620, 172–180 (2023). https://doi.org/10.1038/s41586-023-06291-2 6. Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al.: Holistic evaluation of language models. Transactions on Machine Learning Research (2023) 7. Bodenreider, O.: The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32(suppl_1), D267–D270 (2004). https://doi.org/10.1093/nar/gkh061 8. Kilicoglu, H., Shin, D., Fiszman, M., Rosemblat, G., Rindflesch, T.C.: SemMedDB: a PubMed-scale repository of biomedical semantic predications. Bioinformatics 28(23), 3158–3160 (2012). https://doi.org/10.1093/bioinformatics/bts591 9. Wishart, D.S., Feunang, Y.D., Guo, A.C., Lo, E.J., et al.: DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Res. 46(D1), D1074–D1082 (2018). https://doi.org/10.1093/nar/gkx1037 10. Himmelstein, D.S., Lizee, A., Hessler, C., Brueggeman, L., et al.: Systematic integration of biomedical knowledge prioritizes drugs for repurposing. eLife 6, e26726 (2017). https://doi.org/10.7554/eLife.26726 11. Ochoa, D., Hercules, A., Carmona, M., Suveges, D., et al.: Open Targets Platform: supporting systematic drug–target identification and prioritisation. Nucleic Acids Res. 49(D1), D1302–D1310 (2021). https://doi.org/10.1093/nar/gkaa1027 12. Chandak, P., Huang, K., Zitnik, M.: Building a knowledge graph to enable precision medicine. Sci. Data 10, 67 (2023). https://doi.org/10.1038/s41597-023-01960-3 13. Lewis, P., Perez, E., Piktus, A., Petroni, F., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems 33, p. 9459–9474 (2020) 14. Soman, K., Rose, P.W., Morris, J.H., Akbas, R.E., et al.: Biomedical knowledge graph-optimized prompt generation for large language models. Bioinformatics 40(9), btae560 (2024). https://doi.org/10.1093/bioinformatics/btae560 15. Edge, D., Trinh, H., Cheng, N., Bradley, J., et al.: From local to global: A graph RAG approach to query-focused summarization. arXiv:2404.16130 (2024) 16. Groth, P., Gibson, A., Velterop, J.: The anatomy of a nanopublication. Inf. Serv. Use 30(1–2), 51–56 (2010). https://doi.org/10.3233/ISU-2010-0613 17. Unni, D.R., Moxon, S.A.T., Bada, M., Brush, M., et al.: Biolink Model: A universal schema for knowledge graphs in clinical, biomedical, and translational science. Clin. Transl. Sci. 15(8), 1848–1855 (2022). https://doi.org/10.1111/cts.13302 18. Moreau, L., Missier, P., Belhajjame, K., B'Far, R., et al.: PROV-DM: The PROV data model. W3C Recommendation (2013). https://w.w3.org/TR/prov-dm/ 19. Griffith, M., Spies, N.C., Krysiak, K., McMichael, J.F., et al.: CIViC is a community knowledgebase for expert crowdsourcing the clinical interpretation of variants in cancer. Nat. Genet. 49(2), 170–174 (2017). https://doi.org/10.1038/ng.3774 20. Rehm, H.L., Berg, J.S., Brooks, L.D., Bustamante, C.D., et al.: ClinGen the Clinical Genome Resource. N. Engl. J. Med. 372(23), 2235–2242 (2015). https://doi.org/10.1056/NEJMsr1406261 21. Hartig, O.: Foundations of RDF⋆ and SPARQL⋆: An alternative approach to statement-level metadata in RDF. In: Proceedings of the 11th Alberto Mendelzon International Workshop on Foundations of Data Management (AMW) (2017) 22. Hernán, M.A., Robins, J.M.: Causal Inference: What If. Chapman & Hall/CRC, Boca Raton (2020) 23. Wager, S., Athey, S.: Estimation and inference of heterogeneous treatment effects using random forests. J. Am. Stat. Assoc. 113(523), 1228–1242 (2018). https://doi.org/10.1080/01621459.2017.1319839 24. Prosperi, M., Guo, Y., Sperrin, M., Koopman, J.S., et al.: Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nat. Mach. Intell. 2(7), 369–375 (2020). https://doi.org/10.1038/s42256-020-0197-y 25. Kıcıman, E., Ness, R., Sharma, A., Tan, C.: Causal reasoning and large language models: Opening a new frontier for causality. arXiv:2305.00050 (2023) 26. Willig, M., Zečević, M., Dhami, D.S., Kersting, K.: Causal parrots: Large language models may talk causality but are not causal. Trans. Mach. Learn. Res. (2023) 27. Jin, Z., Chen, Y., Leeb, F., Gresele, L., et al.: CLADDER: Assessing causal reasoning in language models. In: Advances in Neural Information Processing Systems 36 (2023) 28. Min, S., Krishna, K., Lyu, X., Lewis, M., et al.: FActScore: Fine-grained atomic evaluation of factual precision in long-form text generation. In: Proceedings of EMNLP, p. 12076–12100 (2023). https://doi.org/10.18653/v1/2023.emnlp-main.741 29. Lin, S., Hilton, J., Evans, O.: TruthfulQA: Measuring how models mimic human falsehoods. In: Proceedings of ACL, p. 3214–3252 (2022). https://doi.org/10.18653/v1/2022.acl-long.229 30. Li, J., Cheng, X., Zhao, W.X., Nie, J.-Y., Wen, J.-R.: HaluEval: A large-scale hallucination evaluation benchmark for large language models. In: Proceedings of EMNLP, p. 6449–6464 (2023). https://doi.org/10.18653/v1/2023.emnlp- main.397 31. Bohnet, B., Tran, V.Q., Verga, P., Aharoni, R., et al.: Attributed question answering: Evaluation and modeling for attributed large language models. arXiv:2212.08037 (2022) 32. Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., et al.: Measuring attribution in natural language generation models. Comput. Linguist. 49(4), 777–840 (2023). https://doi.org/10.1162/coli_a_00486 33. Umapathi, L.K., Pal, A., Sankarasubbu, M.: Med-HALT: Medical domain hallucination test for large language models. In: Proceedings of CoNLL, p. 314–334 (2023). https://doi.org/10.18653/v1/2023.conll-1.21 34. Ji, Z., Lee, N., Frieske, R., Yu, T., et al.: Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12), 248 (2023). https://doi.org/10.1145/3571730