Paper deep dive
TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation
Axel TahmasebiMoradi, Lucas Schott, Martin Royer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/7/2026, 8:06:48 AM
Summary
The paper introduces TRIAGE, a stage-aware instrumentation framework for evaluating Graph-based Retrieval-Augmented Generation (Graph-RAG) systems. It addresses the lack of trustworthiness metrics in LLM-driven knowledge graph construction by instrumenting three pipeline stages: KG Implementation, KG Validation, and KG Usage. TRIAGE provides independently interpretable, mostly gold-free metrics to localize failures and assess query costs, enabling targeted remediation. The framework contrasts with existing systems like GraphRAG, LightRAG, and PathRAG, highlighting the need for granular, stage-specific evaluation beyond end-to-end answer scoring.
Entities (10)
Relation Signals (8)
Triage → evaluates → Graph-based RAG
confidence 96% · TRIAGE is a stage-aware instrumentation framework for automated, document-grounded graph-RAG that asks not only whether the underlying graph can be trusted but at what cost it can be queried.
Triage → instruments → KG Implementation
confidence 95% · TRIAGE attaches stage-specific, independently interpretable metrics to three stages: the KG Implementation...
Triage → instruments → KG Validation
confidence 95% · TRIAGE attaches stage-specific, independently interpretable metrics to three stages: ... the KG Validation by expert...
Triage → instruments → KG Usage
confidence 95% · TRIAGE attaches stage-specific, independently interpretable metrics to three stages: ... and the KG Usage...
GraphRAG → retrieves → community summaries
confidence 92% · GraphRAG retrieves whole community summaries. A hierarchical community detection partitions the graph into communities
LightRAG → retrieves → ego-networks
confidence 92% · LightRAG retrieves the ego-networks of cosine-matched entity nodes.
PathRAG → retrieves → multi-hop paths
confidence 92% · PathRAG argues that the principal limitation of GraphRAG and LightRAG is the redundancy of the retrieved subgraph rather than its insufficiency, and instead returns sparse multi-hop relational paths between anchors.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Knowledge graphs (KGs) that underpin Graph-based Retrieval-Augmented Generation (Graph-RAG) are increasingly built automatically by LLM-driven extraction rather than curated by experts. Proper evaluation would require instrumenting all pertinent stages: extraction, graph construction, and inference, coherently enough to localize failures, so that a failure at one stage is not discovered as a wrong answer at the end. We introduce TRIAGE, a stage-aware instrumentation framework for automated, document-grounded graph-RAG that asks not only whether the underlying graph can be trusted but at what cost it can be queried. TRIAGE attaches stage-specific, independently interpretable metrics to three stages: the KG Implementation (triple confidence, source coverage, and schema and canonicalization checks), the KG Validation by expert (graph-level structural quality, with correctness and completeness computed only as offline calibration when a reference is available), and the KG Usage (retrieval coverage, faithfulness, and retrieval cost); the deployed metrics need no gold annotations, the gold-requiring ones serving only as offline calibration. At usage time these metrics form a diagnostic chain of necessary conditions whose first broken link localizes the failure, and the diagnosis maps to the stage levers that can remedy it: extraction, graph and schema, or retrieval. TRIAGE is a theoretical framework with a proof of concept and a reproducible evaluation protocol.
Tags
Links
- Source: https://arxiv.org/abs/2607.03447v1
- Canonical: https://arxiv.org/abs/2607.03447v1
Trouble viewing inline? Open PDF directly →
Full Text
141,884 characters extracted from source content.
Expand or collapse full text
11institutetext: IRT-SystemX 11email: a.tahmasebimoradi@irt-systemx.fr, lucas.schott@irt-systemx.fr, martin.royer@irt-systemx.fr TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation Axel TahmasebiMoradi Lucas Schott Martin Royer Abstract Knowledge graphs (KGs) that underpin Graph-based Retrieval-Augmented Generation (Graph-RAG) are increasingly built automatically by LLM-driven extraction rather than curated by experts. Proper evaluation would require instrumenting all pertinent stages: extraction, graph construction, and inference, coherently enough to localize failures, so that a failure at one stage is not discovered as a wrong answer at the end. We introduce TRIAGE, a stage-aware instrumentation framework for automated, document-grounded graph-RAG that asks not only whether the underlying graph can be trusted but at what cost it can be queried. TRIAGE attaches stage-specific, independently interpretable metrics to three stages: the KG Implementation (triple confidence, source coverage, and schema and canonicalization checks), the KG Validation by expert (graph-level structural quality, with correctness and completeness computed only as offline calibration when a reference is available), and the KG Usage (retrieval coverage, faithfulness, and retrieval cost); the deployed metrics need no gold annotations, the gold-requiring ones serving only as offline calibration. At usage time these metrics form a diagnostic chain of necessary conditions whose first broken link localizes the failure, and the diagnosis maps to the stage levers that can remedy it: extraction, graph and schema, or retrieval. TRIAGE is a theoretical framework with a proof of concept and a reproducible evaluation protocol. 1 Introduction Context and motivation. Large language models (LLMs) have transformed information access by enabling fluent, query-driven generation over vast document collections. Yet their most persistent failure mode remains hallucination: the generation of plausible but unsupported statements [28, 42]. Retrieval-Augmented Generation (RAG) was introduced precisely to mitigate this risk by grounding generation in externally retrieved evidence [35]. However, when retrieval operates over flat vector indexes, it recovers semantically similar passages without capturing the relational structure that multi-hop or entity-centric questions demand [13, 9]. Knowledge graphs (KGs) offer a principled alternative: by representing factual knowledge as typed, directed triples (s,p,o)(s,p,o) [23], they make relationships explicit and support structured, traceable reasoning paths. Graph-based RAG systems exploit this structure to improve retrieval precision and answer faithfulness [13, 20, 9]. Yet the KG itself is increasingly built automatically by LLM-driven pipelines rather than curated by experts, introducing a new layer of uncertainty, even as we continue to judge such systems only by whether the final answer looks right. The central question of this paper is therefore twofold: when and how much we can trust the graph that underpins retrieval, and at what cost, in pathfinding time, retrieval latency, and compute, it can be queried. A graph that is trustworthy but too costly to traverse is no more deployable than one that is cheap but wrong, so an end-to-end account of graph-RAG quality must speak to both. The gap we are filling. Recent work on trustworthy KG engineering, in particular the TKG methodology [2], provides a rigorous lifecycle framework covering construction, validation, deployment, and governance of KGs in safety-critical settings, and defines formal effectiveness metrics for correctness, completeness, and logical consistency. However, TKG and related approaches assume largely expert-driven knowledge elicitation and do not address the automated, LLM-driven extraction pipelines that underpin modern graph-RAG systems. Conversely, graph-RAG systems such as GraphRAG [13], LightRAG [20], and PathRAG [9] are engineered for accuracy and efficiency, but provide no stage-specific trustworthiness instrumentation: failures in extraction, graph structure, or retrieval are invisible until they surface as wrong answers [21, 65]. Recent evaluation benchmarks have begun to characterize when graph structure helps [62, 57], and faithfulness-oriented work has studied whether answers are grounded in retrieved evidence [33, 70]. Yet none of these efforts instruments the pipeline at the level of individual stages (extraction, graph validation, and retrieval) with observable, ground-truth-free metrics that can localize failures and trigger principled remediation. TRIAGE fills this gap: it instruments the entire construction, validation, and use pipeline with gold-free, stage-localized trust and cost metrics, turning an opaque end-to-end failure into a diagnosis that pinpoints its stage and the levers that can fix it. We focus on automated, LLM-driven construction because that is how KGs are increasingly built, while evaluation has not kept up; the instrumentation, however, is not specific to it: the KG Implementation metrics apply to any extracted KG artifact, whether produced by LLM extraction or a conventional, non-AI pipeline. TRIAGE is deployment-first: its core trust and cost signals need no gold annotations, so they are computed on every live query, not only on a labeled benchmark. Contributions. The contribution of TRIAGE is not any single metric but a usable system: it locates where trust breaks across the pipeline, acts on each breach through the stage lever it implicates, surfaces which signals remain computable in a given deployment, and catches failures before the answer is generated. Many of these metrics are individually simple, and several are adapted from prior work; what is new, and what makes them useful, is their organization into a stage-localized, deployment-ready instrument. This paper makes the following contributions: 1. Framework. Informed by the TKG methodology [2], we define TRIAGE as a three-stage trustworthy engineering process for automated graph-RAG pipelines: KG Implementation, KG Validation (with expert review), and KG Usage. Rather than following TKG prescriptively, TRIAGE adapts its phase structure and makes independent design choices suited to automated extraction, each stage instrumented with stage-specific confidence metrics (section 3). 2. Metrics. We define a suite of stage-specific, independently interpretable metrics (overview in table 2), most of which are gold-free: computable at deployment without any reference answer or graph. We further characterize them by two computability dimensions: what external reference each needs (none, an ontology, or a gold standard), and whether the extraction is observable to us (white-box) or only the final graph is given (black-box). At KG Implementation: Triple Confidence Score (TCSTCS), Source Coverage Ratio (SCRSCR), Extraction Consistency (ECEC), Cross-Source Corroboration (CSCCSC), Schema Compliance (SCSC), Constraint Violation Rate (CVRCVR), Canonicalization Success Rate (CSRlinkCSR_link), and Evidence Attribution Rate (EAREAR). At KG Validation: the graph-level, ground-truth-free Ontology Coverage (OCOC), Dead Node Ratio (DNRDNR), Semantic Redundancy Rate (SRRSRR), and Logical Consistency (LCLC), alongside correctness (CORCOR) and completeness (CMPCMP) imported from [2] for benchmark settings. At KG Usage: Query Grounding Rate (QGRQGR), Entity Retrieval Coverage (ERCERC), Entity Retrieval Precision (ERPERP), Reasoning Readiness Score (RRSRRS), Answer Grounding Rate (AGRAGR), Answer Utilization Rate (AURAUR), Answer Reasoning Faithfulness (ARFARF), Retrieval Path Cost (RPCRPC), and Reasoning Hop Depth (RHDRHD). Where gold paths or answers are available, Gold Path Coverage (GPCGPC) and Retrieval Path Precision/Recall (RPP/RPR) serve as offline checks, and the imported outcome metrics (Context Coverage, Reasoning Failure Rate, KG Net Impact, and Hard Hits Rate) as the dependent variables against which the gold-free metrics are validated (section 4). 3. Diagnosis, remediation, and validation. The usage-stage metrics form a chain of necessary conditions whose first broken link localizes a failure to a single condition, and that diagnosis maps to the stage levers that can remedy it: KG Implementation metrics to extraction levers (prompt refinement, temperature reduction, re-chunking), KG Validation metrics to graph-and-schema levers (entity resolution, ontology fixes), and KG Usage metrics to retrieval levers, with aggregate metric drops triggering a new KG Update cycle that closes the loop toward continuous, monitored KG evolution (section 6). We assume no universal cost order across these levers: which is cheapest is deployment-specific and is what our protocol measures (section 5). As a proof of concept we instantiate the usage metrics on a multi-hop KGQA probe and find that a capable reader can still answer from its parametric knowledge when retrieval misses the needed evidence, so the answer can look correct while retrieval has in fact failed; this motivates scoring retrieval before and independently of the answer, and a pre-specified, reproducible protocol then specifies the full validation (section 5). Paper structure. Section 2 surveys related work. Section 3 presents the TRIAGE architecture and the remediation map. Section 4 defines the metric suite. Section 5 presents the proof of concept and the evaluation protocol. Section 6 concludes and outlines future work. 2 Related Work Trustworthiness in a graph-RAG pipeline is not a single property but a layered one: it must hold at the level of individual triples (is each extracted fact correct and grounded?), at the level of the graph they form (is the assembled KG complete, consistent, and well-structured?), and at the level of usage (does retrieval surface the right subgraph, and does the generated answer faithfully reflect it?). Prior work has addressed each level largely in isolation. We organize this section accordingly. We first review triple-level evaluation, distinguishing the assessment of extracted triples independently of the graph (triple quality and trustworthiness) from their assessment as part of the graph (knowledge graph completion). We then review graph-based RAG methods, their evaluation protocols, and the related work on faithful and grounded reasoning over knowledge graphs. Finally, we discuss trustworthy KG engineering methodologies and connect them to broader trustworthy AI principles, positioning TRIAGE as a framework that instruments all three levels within a single, stage-aware methodology. 2.1 Triple-centric related work: trustworthiness vs completion Knowledge graphs represent structured factual knowledge as directed labeled edges, commonly written as triples (s,p,o)(s,p,o), and a large body of work evaluates models and pipelines by how well they recover or validate such triples [6, 55, 25]. We distinguish two evaluation objectives that differ in both their assumptions and their metrics, and that correspond to two successive moments in the construction of a KG: first assessing whether an extracted triple is trustworthy at all, and only then assessing how well a set of trusted triples completes a graph. Triple Quality / Trustworthiness. The first question is whether a candidate triple, freshly proposed for insertion into a KG, is correct and well-formed. Such triples may be produced by rule-based information extraction, neural relation extraction, Open Information Extraction (OpenIE), or, increasingly, LLM prompting. Assessing them is a validation problem: given a complete, already-formed triple (s,p,o)(s,p,o), decide whether it is true and admissible, a per-triple yes/no judgment rather than an ordering of alternatives. This judgment must account for three distinct concerns: evidence grounding (is the triple supported by the source text, or hallucinated?), schema compliance (does it respect the ontology’s predicate vocabulary and type constraints?), and logical consistency (does it avoid contradicting other triples or ontology axioms?) [51, 5, 24]. Ensuring consistency with ontology constraints and type restrictions is central here, as violations (domain and range mismatches, type incompatibilities, or logical inconsistencies) often signal structural or semantic errors in extracted knowledge [44, 46]. Knowledge Graph Completion (KGC). Once a body of trusted triples exists, a second question arises: given the graph they form, which missing edges can be inferred? This is Knowledge Graph Completion, also known as link prediction, and unlike triple validation it is fundamentally a ranking problem. For a query of the form (s,p,?)(s,p,?) or (?,p,o)(?,p,o), where the relation is given and a missing entity is sought, a model scores every candidate entity in the KG and ranks them, with success measured by how highly the correct entity is ranked [6, 12, 52, 25]. The distinction matters: validation asks “is this triple true?” and returns a judgment about a single, complete triple; completion asks “which entity best completes this gap?” and returns an ordering over many candidates. Accordingly, KGC relies on rank-based metrics such as Mean Reciprocal Rank (MRR) and Hits@K, computed over both head- and tail-prediction tasks [6, 55, 12]. Standard KGC evaluation further distinguishes raw from filtered settings, the latter removing other known true triples from the candidate set so that a model is not penalized for ranking a different correct answer above the target [6]. A recurring methodological pitfall in KGC evaluation is benchmark contamination through inverse relations. In early benchmarks such as FB15k, many relations have near-inverse counterparts: for instance, a /film/directed_by edge and a /film/director_of edge encode the same fact in opposite directions. A model can then achieve high test scores not by learning meaningful relational structure, but by memorizing that one relation is the inverse of another: having seen (A,directed_by,B)(A, directed\_by,B) at training time, predicting (B,director_of,A)(B, director\_of,A) at test time is trivial. This inflates reported performance without reflecting genuine inference ability. Corrected benchmarks such as FB15k-237 remove these redundant inverse and duplicate relations precisely so that the evaluation measures inference rather than memorization [12, 54]. 2.1.1 LLM-based triple extraction Before the adoption of LLMs, triple extraction relied on classical information-extraction methods (rule-based and dependency-pattern systems, and Open Information Extraction), which are deterministic and structurally consistent but brittle under linguistic variability and limited in implicit or cross-sentence reasoning [51, 3]. For the open-domain, paraphrase-rich setting that graph-RAG pipelines operate in, these approaches have been largely superseded by LLM-based extraction, which we focus on here. LLM-based extraction uses generative models to map natural language directly into structured triples. Compared to classical pipelines, LLMs are more effective at handling paraphrases, implicit relations, long-context dependencies, and some forms of multi-sentence reasoning [8, 60]. They are highly adaptable across domains with minimal feature engineering and can capture synonymy and nuanced language. Crucially for trustworthiness, LLMs can be prompted to emit, alongside each triple, the source span from which it was derived, a natural-language justification or character offset into the input. This provenance enables three things downstream: verification of the triple against its cited evidence, confidence estimation based on the strength of that evidence, and human audit of contested facts [24, 63]. We operationalize exactly this signal later as the Evidence Attribution Rate (EAREAR, section 4.1.2). However, these advantages introduce new challenges. LLMs may hallucinate plausible but unsupported facts [28, 42], and their outputs can vary depending on prompting and decoding strategies [59]. This leads to issues such as schema drift, non-deterministic formatting, and inconsistencies across runs. The two standard prompting regimes trade one failure mode for another. In zero-shot extraction, the model relies solely on instructions and input text, providing strong generalization but increasing the risk of hallucination and structural inconsistency [8]. Few-shot extraction augments prompts with demonstrations, improving precision and format control [43, 69], but the demonstrations themselves introduce demonstration bias: the model tends to over-produce relation types and structures resembling the examples, while under-extracting valid triples that do not match the demonstrated patterns. Neither regime is free of failure modes, which is precisely why dedicated trustworthiness metrics are needed rather than reliance on extraction accuracy alone. The diversity of failure modes introduced by LLM-based extraction, such as hallucination, schema drift, demonstration bias, and output variability, exposes the limitations of evaluation metrics that rely solely on aggregate accuracy or exact matching. In trustworthy graph construction, it is essential to quantify not only correctness but also confidence, coverage, and stability of extracted knowledge. This motivates dedicated trustworthiness metrics such as the Triple Confidence Score (TCSTCS), which we operationalize here for extraction-level reliability within end-to-end graph-based RAG pipelines. 2.2 Graph-based RAG Graph-based RAG [13, 20, 9] addresses a limitation of classical text RAG, namely its inability to capture structured dependencies across a document corpus. Instead of indexing flat text chunks, these methods first construct an indexing graph G=(V,E)G=(V,E) whose nodes v∈Vv∈ V represent entities extracted from the documents and whose edges (u,ρ,v)∈E(u,ρ,v)∈ E encode their typed relations ρ. At retrieval time, the graph structure is exploited to assemble a query-relevant subgraph that is verbalized as context for the generator. Three landmark systems define the current design space, and the cleanest way to contrast them is by their retrieval primitive, the granularity at which evidence is extracted from G in response to a query q. GraphRAG [13] retrieves whole community summaries. A hierarchical community detection (Leiden clustering) partitions the graph into C1,…,Cm\C_1,…,C_m\, and an LLM produces a textual summary s(Ci)s(C_i) for each community. The retrieved context for a query q is the set of summaries deemed most relevant to q: RGraphRAG(q)=s(Ci):Ci∈top−Kirel(q,s(Ci)),R_GraphRAG(q)\;=\; \\,s(C_i)\,:\,C_i -K_i\,rel(q,s(C_i))\, \, where rel(⋅,⋅)rel(·,·) is an LLM-rated relevance score. Pros: the community primitive excels at global, dataset-spanning summarization queries where the answer draws on a whole topic region. Cons: it produces verbose, redundant outputs whenever only a sub-region of a community is pertinent, and the offline community-summarization step is computationally heavy to build and to refresh as the graph changes. LightRAG [20] retrieves the ego-networks of cosine-matched entity nodes. An LLM extracts a keyword set KqK_q from q that is then matched against entity embeddings via dense vector similarity: Vq=⋃k∈Kqtop−Nv∈Vcos(e(k),e(v)),V_q\;=\; _k∈ K_qtop-N_\,v∈ V \! (e(k),\,e(v) ), where e(⋅)e(·) denotes the embedding function, and the retrieved subgraph is the 1-hop neighborhood of VqV_q. Pros: it is substantially lighter and faster than GraphRAG, with no offline community-summarization cost, and its dual-level design adapts to both specific and abstract queries. Cons: the ego-network of any query-related node still contains many edges that are not on the reasoning chain linking the query’s entities, introducing structural noise that can mislead the generator. PathRAG [9] argues that the principal limitation of GraphRAG and LightRAG is the redundancy of the retrieved subgraph rather than its insufficiency, and instead returns sparse multi-hop relational paths between anchors. The anchor set VqV_q is computed by dense matching as in LightRAG; for each anchor a unit resource is propagated through the graph, decaying by α∈(0,1)α∈(0,1) at every hop, and each path is scored by its average resource, with the top-K paths forming the retrieved context. Pros: the path primitive is more compact than ego-networks and more directly aligned with the multi-step structure of the query, reducing the surface area over which the generator can hallucinate. Cons: path enumeration is expensive on dense graphs, where the number of candidate paths grows combinatorially; the resource-propagation scoring adds retrieval-time latency; and a sparse-path primitive can miss answers that require a fuller neighborhood rather than a single chain. In short, the three systems trace a trade-off axis rather than a strict ranking: GraphRAG maximizes coverage at the cost of verbosity and offline expense, LightRAG minimizes cost at the risk of structural noise, and PathRAG maximizes precision at the cost of path-search latency. Which point on this axis is appropriate depends on the query distribution and the available compute budget, a dependence that motivates measuring retrieval cost explicitly, as TRIAGE does at KG Usage (section 4.3). 2.2.1 Evaluation of graph-based RAG Because these systems target open-ended generation tasks for which no canonical ground-truth answer exists, the dominant evaluation protocol is LLM-as-judge pairwise comparison along a small set of qualitative dimensions, introduced by GraphRAG and adopted, with minor variation, by LightRAG and PathRAG. GraphRAG scores answers on Comprehensiveness, Diversity, and Empowerment, later extended with a Directness dimension [13]; LightRAG uses Comprehensiveness, Diversity, and Empowerment together with an aggregate Overall judgment [20]; PathRAG follows the same protocol [9]. For each (query, baseline, candidate) triplet, an LLM selects a winner per dimension, and aggregate win-rates are reported. This protocol is appropriate when no ground truth is available, but it has several limitations relevant to trustworthiness. Judgments are stochastic and order-sensitive; the judge and the system under test often share the same underlying model family, raising self-preference concerns; and a win-rate is a coarse, black-box end-to-end signal. The deeper problem is one of locus: these protocols score only the generated answer, never the retrieved subgraph. In plain terms, they check whether the final answer looks well-supported, but never check whether the retrieved subgraph actually contained the facts needed to answer the question. When retrieval misses something, the LLM can quietly fill the gap from its own parametric memory and still produce a fluent, plausible answer, so a genuine retrieval failure looks identical to a genuine retrieval success. Answer-level scores therefore cannot separate a sound reasoning path through the graph from a lucky guess by the model, which is exactly the distinction a trustworthy pipeline needs to make. Beyond per-system judging, a first wave of recent benchmarks asks when graph structure helps at all, but still scores only the end task [21, 62]. WildGraphBench [57] stresses the same task-level metrics under noisy, heterogeneous wild-source corpora, and [16] adds a system-level dimension (accuracy, cost, latency, and stability) for graph-RAG versus dense RAG in agentic search. Across this line, evaluation remains task-dependent and exposes no graph-aware signal: failures cannot be attributed to extraction, retrieval, or generation. A second, faithfulness-oriented wave begins to open the black box, but its signals are still computed on the generated answer rather than on the retrieved subgraph. [65] introduce a reasoning failure rate alongside context coverage and token-F1 on multi-hop QA; FinReflectKG-HalluBench [33], a financial-QA benchmark, frames groundedness as binary hallucination detection, where an answer counts as grounded only if supported by both text snippets and KG triples; the tripartite RAG-Eval framework [1] aggregates query relevance, factual accuracy, coverage, coherence, and fluency into an overall confidence score; and [70] probes robustness under incomplete KGs with Hits@Hard and the Hard Hits Rate. These works diagnose whether an answer is grounded, but none instruments the structural adequacy of the retrieved subgraph itself, leaving the retrieval failure described above invisible. TRIAGE addresses this gap by attaching observable, ground-truth-free metrics directly at KG Usage: QGRQGR measures whether a query is on-topic for the KG; ERCERC, ERPERP, and RRSRRS characterize the structural quality of the retrieved subgraph independently of the downstream generation step; AGRAGR, AURAUR, and ARFARF measure the alignment between the generated answer and the retrieved evidence; and RPCRPC measures the computational cost of retrieval itself. These metrics complement the existing answer-level qualitative dimensions rather than replacing them: they isolate retrieval-stage failures from generation-stage failures with deterministic, structural signals that can trigger a KG Update cycle in a principled way. 2.2.2 Faithful reasoning on KGs A complementary line of work studies faithful reasoning over knowledge graphs: whether a system’s answer can be traced, step by step, to explicit graph evidence rather than to parametric knowledge in model weights. Graph-Constrained Reasoning (GCR) [39] operationalizes this directly by restricting the LLM’s decoding to paths that exist in the KG, so that every reasoning step corresponds to a verifiable graph edge. This substantially reduces hallucination on multi-hop questions, but it also exposes a new failure mode: when the KG itself is incomplete or incorrectly extracted, graph-constrained decoding fails silently, producing no answer, or a wrong one, with no signal indicating whether the fault lies in the graph or in the model. A second, scale-related failure mode is equally important. On large KGs, the number of candidate reasoning paths grows combinatorially with hop depth, so practical systems impose cutoffs: a maximum number of hops, a top-K beam over paths, or a bounded expansion budget. These cutoffs keep retrieval tractable, but they are double-edged: when the true reasoning path is longer than the hop limit, or is pruned by the beam, the correct answer becomes unreachable even though it is present in the graph. This is a failure of traversal budget, not of knowledge, and it is entirely invisible to answer-level metrics, directly motivating the Reasoning Hop Depth (RHDRHD) and Retrieval Path Cost (RPCRPC) metrics introduced in section 4.3. The same path primitive underlies earlier multi-hop KGQA systems [71, 68], which address faithfulness from the retrieval side: restricting context to sparse relational paths between query-relevant entities shrinks the surface over which hallucination can occur, the property PathRAG (above) brings to graph-RAG retrieval. Evaluation on PathQuestion [71] and MetaQA [68] has shown that path coverage, whether the gold reasoning path lies in the retrieved subgraph, is a strong predictor of answer correctness on multi-hop questions. This observation directly motivates the TRIAGE Reasoning Readiness Score (RRSRRS) and Entity Retrieval Coverage (ERCERC), which operationalize path coverage as observable, pre-inference metrics that do not require gold answers. Faithfulness has also been studied at the answer level, measuring whether generated statements are entailed by the retrieved evidence [24, 42]. As with the answer-level evaluation protocols above, such metrics require a generated answer and cannot separate insufficient retrieval from faulty reasoning; the TRIAGE subgraph-level signals RRSRRS and ERCERC precede generation, enabling proactive triage rather than post-hoc auditing. 2.3 Trustworthy KG engineering and trustworthy AI The TKG methodology [2] is the closest work to our framework in spirit. It proposes an end-to-end trustworthy engineering methodology for KG-based systems, structured along three complementary dimensions: a methodology dimension (construction phases per KG version), a lifecycle dimension (continuous evolution and updates), and a transverse trustworthiness dimension covering governance, provenance, and quality assessment across all phases. Crucially, TKG defines a formal suite of effectiveness metrics, correctness (μcorrect _correct), completeness (μcomplete _complete), logical consistency (μLC _LC), representativeness, and timeliness, which we directly import into the TRIAGE KG Validation stage (see section 4). However, TKG was designed for expert-driven knowledge elicitation in safety-critical industrial settings and does not address automated, LLM-driven extraction pipelines or the KG Usage phase. TRIAGE is informed by its methodology and extends it to fill both gaps, while making independent design choices appropriate to automated graph-RAG. Beyond TKG, broader trustworthy AI frameworks, establishing principles of transparency, accountability, and robustness for AI systems, motivate the need for end-to-end, stage-aware evaluation rather than single-point quality checks [22, 45, 53, 18]. Recent surveys on KG quality further consolidate the requirements for correctness, completeness, and consistency that TRIAGE operationalizes for the graph-RAG setting [66, 17, 40, 34, 46]. To the best of our knowledge, TRIAGE is among the first frameworks to operationalize these principles end-to-end for automated, document-grounded graph-RAG pipelines, instrumenting extraction, validation, and usage within a single methodology. 3 The TRIAGE Framework TRIAGE instruments a graph-RAG pipeline so that a failure can be read off its metrics rather than guessed at. The payoff, developed in section 4.5, is twofold. At usage time the metrics form a diagnostic chain of necessary conditions for a correct answer, and the first one that fails localizes the failure to a single condition. That diagnosis then points to the stage levers that can remedy it: extraction, graph and schema, or retrieval. The rest of this section sets up the three phases that carry these metrics, and section 4 defines the metrics themselves; fig. 4 threads one query through all three. TRIAGE is informed by the TKG engineering methodology [2] and adapts it to automated, document-grounded graph-RAG pipelines. We take from TKG its phase-structured view of KG engineering and its trustworthiness dimension, but we do not follow it prescriptively: the automated, LLM-driven setting calls for independent design choices that TKG, conceived for expert-driven elicitation, does not address, notably the KG Usage phase and the remediation map developed in section 4.5. Adapting the TKG methodology dimension, TRIAGE covers three phases of a single KG version cycle: KG Implementation (automated extraction and construction), KG Validation (expert review supported by ground-truth-free quality metrics), and KG Usage (graph-based inference). Each phase produces both a knowledge artifact and a set of stage-specific confidence metrics. The metrics do not flow between stages; rather, they are each independently observable quality signals that together provide an end-to-end trustworthiness picture of the pipeline. Figure 1 illustrates this structure. The TKG trustworthiness dimension operates transversally: aggregate metric results may trigger a KG Update cycle, and the stage at which a metric drops points to the levers that can remedy it (section 4.5). DocumentsPhase 1: KG Implementationextraction + groundingKGvPhase 2: KG Validationexpert reviewKGvalidatedPhase 3: KG Usagegraph-based inferenceanswerqueryImplementation metricssee fig. 2Validation metricsOCOC §, DNRDNR, SRRSRR, LCLC §Usage metricssee fig. 3KG Updatenew cycle Figure 1: The TRIAGE framework organized into three phases. Each phase produces distinct knowledge artifacts together with stage-specific confidence metrics (gray). Low aggregate metric values may trigger a new KG Update cycle; the stage of the failing metric points to the levers that can remedy it (section 4.5). DocumentsSegmenterhierarchical chunkingChunk treesec / para / sentence spansExtractorsentence-wise LLMcandidate triple(s,p,o)(s,\ p,\ o)⟨ 7, sent 12⟩ , TCS=0.94TCS=0.94Canonicalizeentity resolutionCanonical entity+ predicate mapGraph builderKGvontology O(§, optional)SCRSCRTCSTCS, EAREAR, ECEC,CSCCSC, TFITFI †CSRlinkCSR_link,SCSC §, CVRCVR §versioned artifact trailraw || canon || dedup || EL(JSON / GraphML) Figure 2: White-box instrumentation blueprint of the KG Implementation stage. The vertical spine is a representative construction pipeline (documents → segmentation → extraction → canonicalization → graph); each magnifier probe (right) taps one observable artifact: the chunk tree yields SCRSCR; each candidate triple, carrying a provenance pointer and confidence, yields TCSTCS, EAREAR, ECEC, CSCCSC, and TFITFI (†, gold); and the canonical map yields CSRlinkCSR_link together with the schema checks SCSC and CVRCVR (§, requiring the ontology O, supplied as an optional side input). The persisted, versioned artifact trail (left) is what makes provenance auditable. Markers match table 2: † needs a gold reference, § needs an ontology. 3.1 Phase 1: Document-to-Triple Extraction To make the metrics concrete we describe a representative extraction pipeline (fig. 2); TRIAGE instruments any pipeline of this shape rather than prescribing this one. It turns document structure into candidate triples through an automated process of segmentation, extraction, grounding, canonicalization, and trust scoring, described step by step below. Chunking and passage segmentation. The pipeline performs hierarchical segmentation before extraction: 1. document into sections and subsections, 2. subsection text into paragraphs, 3. paragraph into sentence-like spans. The segmentation strategy determines which textual units are presented to the extractor, directly influencing the coverage and recall of the resulting KG. ML-based triple extraction. Triple extraction is performed sentence-wise. The extraction model follows an explicit instruction policy: emit only triples in subject / predicate / object format; preserve adverbs, modals, and negation in the predicate; and decompose conjunction-heavy clauses into multiple atomic triples. This sentence-level granularity ensures that every extracted triple carries a precise provenance pointer and that confidence scores are computed at the finest possible resolution. Each extracted triple is stored with provenance fields (document, section, subsection, paragraph index, sentence index, and sentence text), enabling passage-level traceability and downstream confidence scoring. Canonicalization. Canonicalization keeps the graph coherent by ensuring that the same real-world entity or relation is represented by a single node or predicate, regardless of surface-form variation across source documents. TRIAGE applies it in one of two regimes, depending on whether a reference ontology is available: 1. Ontology-grounded canonicalization (when a schema is available): predicates and entities are normalized against the ontology; this enables schema checks and the Schema Compliance (SCSC) and Constraint Violation Rate (CVRCVR) metrics. 2. Resolution-based canonicalization (when no suitable ontology exists): rule-based or embedding-based entity resolution, alias and coreference merging, and deduplication act as the fallback that keeps the KG connected and reduces dead nodes (DNRDNR) without a schema. This two-regime design reflects a broader property of TRIAGE: it operates with or without a reference ontology, degrading gracefully from schema-driven to resolution-driven coherence. The implications of ontology availability for measurement, in particular which trustworthiness metrics remain computable without a schema, are discussed in section 4. Ontology grounding. When an ontology is available, extracted triples are additionally aligned to it through predicate canonicalization against the target predicate inventory and optional schema checks evaluating predicate, domain, and range validity. These checks produce a per-triple trust signal feeding SCSC and CVRCVR (section 4.1.2). The conceptual model is provided as input rather than constructed from scratch; how the relevant subset of the ontology is delineated in practice is an operational choice discussed in section 3.2. Confidence scoring. A Triple Confidence Score (TCSTCS) is assigned to each extracted triple as a proxy for extraction reliability. The formal definition of TCSTCS and its role in the full metric suite are presented in section 4. At the pipeline level, TCSTCS feeds document-level confidence aggregation and serves as an early-warning signal: triples whose TCSTCS indicates low confidence can be flagged for expert review during KG Validation (Phase 2) rather than silently entering the graph. Post-processing and artifact persistence. After extraction, the pipeline persists intermediate artifacts per stage (raw, canonicalized, deduplicated, and NER/EL-enriched) in JSON and GraphML format. This versioned artifact trail directly supports the TKG trustworthiness dimension: every triple in the final KG can be traced back to its source sentence and to the confidence score assigned at extraction time, which is what makes the Evidence Attribution Rate (EAREAR, section 4.1.2) computable and the pipeline auditable. 3.2 Phase 2: KG Validation, Expert Review and Effectiveness Assessment Adapting the TKG KG Validation phase [2], the produced KG is reviewed by a domain expert before use. TRIAGE acts here as a methodological bridge: it connects the automated extraction of Phase 1 to a human checkpoint, and makes that checkpoint actionable by surfacing interpretable, graph-level quality metrics as decision support. Crucially, the core metrics of this phase require no ground-truth KG, making them operational in any deployment setting; their formal definitions are given in section 4.2. When a reference KG is available (e.g., in benchmark settings), correctness (CORCOR) and completeness (CMPCMP) from [2] can additionally be computed to calibrate the ground-truth-free metrics. query qqEntity ExtractionNER / noun-chunksurface mentions MqM_qGrounding γq:Mq→V∪⟂ _q:M_q→ V∪ entity linkinggrounded query nodes VqgV_q^gRetriever R(,Vqg)R(K,V_q^g)ego / path / communityretrieved subgraph SK_SVerbalize & Reader LLManswerQGRQGRRPCRPC, RHDRHDERCERC, ERPERP, RRSRRS GPCGPC †AGRAGR, AURAUR, ARFARFpre-inference triageERCERC or RRSRRS <θ→<θ→ reroute / warnblack-box fallback: LLM-as-judge(uses q, answer only) Figure 3: White-box instrumentation blueprint of the KG Usage stage. The vertical spine is the inference path; each magnifier probe (right) taps one observable artifact and lists the metrics computable from it: the surface mentions MqM_q and grounded query nodes VqgV_q^g yield QGRQGR; the retrieved subgraph SK_S yields ERCERC, ERPERP, and RRSRRS (and GPCGPC when gold paths are available, † ); the retrieval trace yields the cost metrics RPCRPC and RHDRHD; and the answer, compared against SK_S, yields AGRAGR, AURAUR, and ARFARF (of these ARFARF and AGRAGR are the chain’s answer-side links, AURAUR their utilization complement). The chain heads QGRQGR, ERCERC, RRSRRS are read before the reader runs, enabling pre-inference triage (left). A black-box evaluator (bottom left) sees only the query and the answer, so it cannot localize a retrieval failure. Inside SK_S, filled nodes are anchors and the bold edge is the reasoning path connecting them. 3.3 Phase 3: KG Usage, Graph-Based Inference The KG Usage phase is identified in TKG [2] as a gap in prior KG engineering methodologies. TRIAGE operationalizes it for graph-RAG systems via graph-based retrieval and inference: a query is grounded to the KG, a query-relevant subgraph is retrieved, and that subgraph conditions answer generation. Graph-constrained decoding [39] is one such inference method, but the usage-specific metrics introduced here are agnostic to the choice of retriever and reader; they characterize the structural adequacy of the retrieved subgraph, the faithfulness of the generated answer, and the computational cost of retrieval (section 4.3). Inference pipeline. Given a validated KG and a natural language query q, the TRIAGE inference layer proceeds in three steps (fig. 3). First, query grounding maps q to a set of anchor entities in the KG via entity linking and embedding-based similarity. Second, subgraph retrieval extracts a query-relevant subgraph from the full KG; the specific retrieval strategy (ego-network, path-based, or community-based) is configurable and does not affect metric computation. Third, the retrieved subgraph is verbalized and supplied to an LLM to generate the final answer, for example via graph-constrained decoding [39]. Pre-inference triage. A key design principle of the TRIAGE KG Usage phase is that structural quality signals are computed before generation, enabling proactive triage rather than post-hoc diagnosis. ERCERC measures whether the entities required to answer q are present in the retrieved subgraph, and RRSRRS measures whether those entities are connected by a valid reasoning path within it. Queries for which ERCERC or RRSRRS fall below a configurable threshold can be flagged before the LLM is invoked, routed to a fallback retrieval strategy or returned with a low-confidence warning, avoiding the silent failure mode in which the LLM compensates with parametric knowledge and produces a plausible but unsupported answer. Worked example (end to end, on MetaQA / WikiMovies; illustrative values). Query (2-hop): “Did the director of Inception also direct Interstellar?” Implementation (already completed). Extraction had produced the bridging triples (Inception, directed_by, Christopher Nolan) and (Interstellar, directed_by, Chris Nolan), both with high confidence (TCS≈0.94TCS≈ 0.94), and SCR=0.91SCR=0.91 over the source passages, so extraction was not the weak link. Validation (already completed). On the assembled graph DNR=0.18DNR=0.18 was mildly elevated because the surface form Chris Nolan had resolved to its own near-isolated node. Usage (at inference). For this query QGR=1.0QGR=1.0 (both films are nodes) and ERC=1.0ERC=1.0 (both films, the query entities, are retrieved), yet RRS=0RRS=0: the two films sit in the retrieved subgraph with no path between them, because their shared director is split across two unmerged nodes. Diagnosis and remediation. Reading the chain (fig. 5), the first broken link is RRSRRS, a connectivity gap; the indicated remediation is a graph-and-schema lever, improved entity resolution to merge Chris Nolan into Christopher Nolan, which restores the two-hop path. Figure 4: A single query threaded through the three TRIAGE phases. Each phase reports one headline metric; the earlier phases ran offline, while the chain is read at inference. The first broken link (fig. 5) localizes the failure to a single condition, here a connectivity gap, which then indicates the stage lever that can fix it. Values are illustrative (MetaQA / WikiMovies). Takeaways. Four practical points summarize the framework. First, localize a failure to a stage and act on it with that stage’s levers, rather than only observing it at the answer. Second, do not wait for gold labels: the core signals are computable in deployment on the artifacts a pipeline already has. Third, measure the retrieved subgraph, not only the generated answer, so a retrieval failure cannot hide behind a fluent response. Fourth, triage before generation, since the first three chain links are available before the LLM runs. If only a handful of signals can be monitored, we suggest TCSTCS (extraction confidence), DNRDNR (graph connectivity), and the chain heads QGRQGR, ERCERC, and RRSRRS (scope, retrieval, connectivity). 4 Trust Metrics This section formally defines the TRIAGE metric suite, summarized by phase in table 2. We first fix a shared notation (table 1), then introduce the two axes that govern which metrics are computable in a given setting, and finally present the metrics phase by phase, each followed by a short note on the remediation it triggers. Metrics marked [TRIAGE] are defined or operationalized in this work for the TRIAGE framework; several instantiate standard measurement ideas, such as coverage, consistency, schema violation, precision/recall, or path connectivity, in a stage-localized graph-RAG setting; the contribution is their joint organization into a deployable diagnostic chain and remediation map. All others are imported using their original definitions and cite their source, whether the TKG methodology [2] or an external graph-RAG benchmark. Table 1: Shared notation used throughout section 4. Symbol Meaning t^=(s,p,o) t=(s,p,o) an extracted triple (hat denotes extracted, vs. gold t) E=t^jE=\ t_j\ set of extracted triples for a document or corpus G=tiG=\t_i\ set of gold triples (when available) K triple set of the assembled TRIAGE-KG (assessed graph) ∗K^* triple set of the reference KG (benchmark setting) V vertex (entity) set of the assembled KG K ,relO,\ O_rel reference ontology and its query-relevant subset f(⋅)f(·) text-embedding function sim(x,y)=cos(f(x),f(y))sim(x,y)= (f(x),f(y)) component-level similarity built from f =p1,…,pnP=\p_1,…,p_n\ set of input passages S,VSK_S,\ V_S retrieved subgraph: triples S⊆K_S , vertices VS⊆V_S V MqM_q surface mentions extracted from query q γq _q grounding function Mq→V∪⊥M_q→ V∪\ \ VqgV_q^g grounded query-node set γq(m):m∈Mq,γq(m)≠⊥⊆V\ _q(m):m∈ M_q,\ _q(m)≠ \ V MaM_a surface mentions extracted from generated answer A γa _a grounding function Ma→V∪⊥M_a→ V∪\ \ VagV_a^g grounded answer-node set γa(m):m∈Ma,γa(m)≠⊥⊆V\ _a(m):m∈ M_a,\ _a(m)≠ \ V deg(v) (v) degree of node v τ,θτ,\ θ confidence / decision thresholds The suite is generated by crossing three trust questions, namely whether each triple is sound, whether the assembled graph is sound, and whether usage succeeds, with the two computability axes introduced next. A metric earns its place by occupying a cell with a distinct failure mode and a distinct remediation, which is also why we report no single composite score: graph quality for retrieval is multi-objective, with trade-offs such as connectivity against parsimony that a scalar would hide. Most of the metrics are individually simple and several are adapted from prior work; what is new is their organization into a stage-localized instrument whose core signals need no gold annotations at deployment. Two orthogonal axes determine which metrics can be computed in a given setting, and we use them to organize the suite. Reference requirement. The first axis is the external reference a metric needs, which is what governs its availability in a given setting. Most metrics need none: they are gold-free and schema-free, computed on any KG from the artifacts the pipeline already produces (TCSTCS, ECEC, CSRlinkCSR_link, DNRDNR at construction; QGRQGR, ERCERC, RRSRRS at usage). A second group needs a reference ontology (marked § : OCOC, SCSC, CVRCVR, LCLC, and ontology-constrained ranking MRRtypeMRR_type); since an ontology can be supplied at deployment, these stay live wherever a schema is on hand and become undefined otherwise, the measurement-side counterpart of the two canonicalization regimes of section 3.1. A third group needs a gold standard (marked † , e.g. TFITFI, CORCOR, CMPCMP, GPCGPC, and the imported outcome metrics CC, RFRRFR, HHRHHR); gold is never available at inference, so these are computable only offline and sit below the dashed line in each phase of table 2, acting as yardsticks for the gold-free metrics (for example GPCGPC against ERCERC and RRSRRS) rather than as deployment signals. Visibility: white-box vs black-box. TRIAGE is a white-box instrument: each phase’s metrics require observing that phase’s internal artifact, and with only black-box (input and output) access they are unavailable (table 3). At KG Implementation, white-box access means observing the extraction process, its source text, token log-probabilities, and run variance, which enables TCSTCS, SCRSCR, ECEC, CSCCSC, EAREAR, CSRlinkCSR_link, and the fidelity metrics; a black-box graph supplied by a third party exposes none of these, and link prediction (MRR, MRRtypeMRR_type, ECE) becomes the completeness probe instead (section 4.1.3). At KG Usage, white-box access means observing the retrieved subgraph, the mapped query entities, and the connecting paths, which is what makes QGRQGR, ERCERC, RRSRRS, and the answer-side metrics computable; with only the query and the final answer, one is back to answer correctness (Hits@1/F1) and LLM-as-judge answer-level dimensions. KG Validation always observes the assembled graph, so its metrics do not depend on visibility. Table 2 marks ontology-dependent metrics with § and sets off the gold-dependent ones († ) below a dashed line within each phase; table 3 summarizes the per-phase visibility requirements, and table 4 separates primary signals from derived ones. A first reading can follow the primary signals that form the spine of the suite (table 4) and treat the derived, aggregate, and imported metrics as optional diagnostic detail. Table 2: Overview of the TRIAGE confidence metric suite by TKG phase. [TRIAGE]: defined or operationalized here for the TRIAGE framework; imported metrics cite their source. (§)( ) requires an ontology; (†)( ) requires a gold reference. Within each phase, the metrics above the dashed line are gold-free and form the deployable instrument; those below († ) need a gold reference and are used only offline, to calibrate and validate the gold-free metrics. The link-prediction rows (MRR, Hits@K, MRRtypeMRR_type, ECE) are a black-box completeness probe requiring held-out triples, not gold-free deployment signals. Phase Metric Symbol Origin KG Implementation Triple Confidence Score TCSTCS [TRIAGE] Source Coverage Ratio SCRSCR [TRIAGE] Extraction Consistency ECEC [TRIAGE] Cross-Source Corroboration CSCCSC [TRIAGE] Entailment Ratio EntRatioEntRatio [24] Schema Compliance Rate§ SCSC [TRIAGE] Constraint Violation Rate§ CVRCVR [TRIAGE] Canonicalization Success Rate CSRlinkCSR_link [TRIAGE] Evidence Attribution Rate EAREAR [TRIAGE] Aggregated Trustworthiness Index TRIAGE-TITRIAGE -TI [TRIAGE] Mean Rank / MRR / Hits@K MRRMRR [6] Ontology-constrained ranking§ MRRtypeMRR_type [TRIAGE] Calibration of completion conf. ECEECE [19] Embedding Similarity† cos(t^,t) ( t,t) [47] Soft-F1† Soft-F1Soft -F1 [5] Triple Fidelity Index† TFITFI [TRIAGE] KG Validation Logical Consistency§ LCLC [2] Ontology Coverage§ OCOC [TRIAGE] Dead Node Ratio DNRDNR [TRIAGE] Semantic Redundancy Rate SRRSRR [TRIAGE] Correctness† CORCOR [2] Completeness† CMPCMP [2] KG Usage Query Grounding Rate QGRQGR [TRIAGE] Entity Retrieval Coverage ERCERC [TRIAGE] Entity Retrieval Precision ERPERP [TRIAGE] Reasoning Readiness Score RRSRRS [TRIAGE] Retrieval Path Cost RPCRPC [TRIAGE] Answer Grounding Rate AGRAGR [TRIAGE] Answer Utilization Rate AURAUR [TRIAGE] Answer Reasoning Faithfulness ARFARF [TRIAGE] Reasoning Hop Depth RHDRHD [TRIAGE] KG Net Impact KNIKNI [33] Gold Path Coverage† GPCGPC [TRIAGE] Retrieval Path Precision/Recall† RPP/RPRRPP/RPR [TRIAGE] Answer Correctness† Hits@1/F1Hits@1/F1 [64] Context Coverage† CC [65] Reasoning Failure Rate† RFRRFR [65] Hard Hits Rate† HHRHHR [70] Table 3: TRIAGE is a white-box instrument: each phase’s metrics require observing that phase’s internal artifact. With only black-box access (the phase’s inputs and outputs) those metrics are unavailable, and one falls back to the right-hand column. Visibility is independent per phase, and is separate from ontology-dependence, marked § in table 2. Phase White-box artifact (metrics it enables) Black-box fallback Implementation extraction process: source text, token log-probabilities, run variance (⇒ TCSTCS, SCRSCR, ECEC, CSCCSC, EAREAR, fidelity) only the output triples: link prediction (MRR, Hits@K, MRRtypeMRR_type, ECE) Validation the assembled graph (⇒ DNRDNR, SRRSRR, LCLC, OCOC) the graph is the artifact under review, so it is observed by construction Usage the retrieved subgraph, mapped query entities, and connecting paths (⇒ QGRQGR, ERCERC, ERPERP, RRSRRS, AGRAGR, AURAUR, ARFARF, RHDRHD, RPCRPC) only the query and the answer: answer correctness (Hits@1/F1) and LLM-as-judge answer-level dimensions Table 4: Dependency/redundancy analysis. Primary metrics are independent signals worth triggering remediation on; derived metrics are definitional complements, aggregates, or expected correlates of a primary metric, and serve as diagnostic detail. No claim of statistical independence is made; relationships are definitional or hypothesized (section 4.4). Metric Role Relationship TCSTCS primary extraction confidence SCRSCR primary source coverage ECEC primary run-to-run stability CSCCSC primary cross-source agreement SCSC primary schema validity CVRCVR derived per-constraint refinement of SCSC CSRlinkCSR_link primary grounding success TRIAGE-TITRIAGE -TI, TFITFI derived weighted aggregates of the above LCLC primary graph-level consistency OCOC primary schema coverage DNRDNR primary structural inertia SRRSRR primary structural redundancy CMPCMP derived gold-anchored counterpart of OCOC QGRQGR primary query in-scope for KG ERCERC primary retrieval recall of grounded query nodes ERPERP derived precision complement of ERCERC RRSRRS primary subgraph connectivity AGRAGR primary answer grounding AURAUR derived recall complement of AGRAGR ARFARF primary relational faithfulness RPCRPC primary retrieval cost RHDRHD derived path-length distribution behind RRSRRS 4.1 KG Implementation Metrics The KG Implementation phase produces a set of extracted triples E=t^jE=\ t_j\, each t^=(s,p,o) t=(s,p,o) a candidate fact from the source documents. Evaluating them requires three complementary perspectives. Triple fidelity metrics assess correctness relative to gold triples or source evidence (and so require G or source text P). Trustworthiness metrics are intrinsic signals computable without gold annotations. KG completion metrics characterize the structural quality of the graph and contextualize why trustworthiness signals are needed alongside accuracy. Core question: is each extracted triple trustworthy? The signal to watch first is the Triple Confidence Score (TCSTCS); the other Implementation metrics refine and complement it. 4.1.1 Triple Fidelity Metrics Triple fidelity metrics measure how well extracted triples match gold knowledge or source evidence; they are benchmark-setting metrics, applicable when a reference set G or source text P is available. The encoder f is a free parameter: sentence-transformer encoders are a fast, inexpensive default well suited to short triple strings, while LLM-based embedders capture relational nuance at higher cost and latency. The metric is only as reliable as the encoder’s ability to place semantically equivalent triples near one another. Embedding similarity. Given the embedding function f, the cosine similarity between an extracted triple t t and a gold triple t is [47, 49] cos(t^,t)=f(t^)⋅f(t)‖f(t^)‖‖f(t)‖. ( t,t)= f( t)· f(t)\|f( t)\|\,\|f(t)\|. (1) Each triple is linearized into a string (e.g., s p o) before embedding, capturing semantic relatedness even when triples differ lexically. Soft matching and Soft-F1. Exact matching is too strict for LLM outputs. Using the shared component similarity sim(x,y)=cos(f(x),f(y))sim(x,y)= (f(x),f(y)) from table 1, component-wise triple similarity is [5, 51] Score(t^,t)=13(sim(st^,st)+sim(pt^,pt)+sim(ot^,ot)).Score( t,t)= 13 (sim(s_ t,s_t)+sim(p_ t,p_t)+sim(o_ t,o_t) ). (2) Set-level Soft-Precision and Soft-Recall are computed via maximum-weight bipartite matching M between E and G [32]: Soft-P=1|E|∑(t^,t)∈MScore(t^,t),Soft-R=1|G|∑(t^,t)∈MScore(t^,t),Soft -P= 1|E|\!\! _( t,t)∈ M\!\!Score( t,t), -R= 1|G|\!\! _( t,t)∈ M\!\!Score( t,t), (3) Soft-F1=2Soft-P⋅Soft-RSoft-P+Soft-R.Soft -F1= 2\,Soft -P·Soft -RSoft -P+Soft -R. (4) Entailment-based verification. To measure grounding in source evidence, triple validation is reframed as Natural Language Inference (NLI). NLI classifies a (premise, hypothesis) pair into one of three labels: entailment (the premise supports the hypothesis), contradiction (the premise refutes it), or neutral (neither). Here the premise is the source passage p(t^)p( t) from which t t was extracted (its recorded provenance from section 3.1, not the whole corpus), and the hypothesis H(t^)H( t) is a natural-language verbalization of the triple (e.g., (s,p,o)↦(s,p,o) “s p o”). Writing PNLI(ent∣p(t^),H)P_NLI(ent p( t),H) for the probability mass an NLI model assigns to the entailment label, the corpus-level entailment ratio is [7, 61, 24] EntRatio(E)=1|E|∑t^∈EPNLI(ent∣p(t^),H(t^)).EntRatio(E)= 1|E| _ t∈ EP_NLI\! (ent p( t),H( t) ). (5) Triples with low entailment and high contradiction probability are flagged as unsupported, a grounding signal complementary to Soft-F1. Triple Fidelity Index (TFITFI) [TRIAGE], benchmark-only. To summarize the three fidelity signals into a single [0,1][0,1] score, paralleling the trustworthiness aggregate TRIAGE-TITRIAGE -TI below: TFI(E)=λ1cos¯+λ2Soft-F1+λ3EntRatio,∑iλi=1,TFI(E)= _1\, + _2\,Soft -F1+ _3\,EntRatio, _i _i=1, (6) where cos¯ is the mean best-match cosine over E and the weights λi _i reflect the relative importance of semantic match, set-level agreement, and source grounding. Because two of its three components (cos¯ and Soft-F1) require the gold set G, TFITFI is a benchmark-setting aggregate and is not computable in gold-free deployment, even though its third component, EntRatio, needs only the source passages P. 4.1.2 TRIAGE Trustworthiness Metrics Unlike fidelity metrics, trustworthiness metrics are intrinsic: they require neither gold triples nor source alignment, and so are operational in any deployment. They capture extraction confidence (TCSTCS), source coverage (SCRSCR), run-to-run stability (ECEC), cross-source agreement (CSCCSC), structural validity (SCSC, CVRCVR), and grounding to canonical identifiers (CSRlinkCSR_link). Triple Confidence Score (TCSTCS) [TRIAGE]. LLM token probabilities provide an intrinsic proxy for extraction confidence [4, 30, 29]. Treating the emitted triple as its token sequence w1,…,w|t^|w_1,…,w_| t|, where wkw_k is the k-th token and w<kw_<k its predecessors, and writing PLLM(wk∣w<k)P_LLM(w_k w_<k) for the model’s next-token probability, we define TCSTCS as the geometric mean token probability, equivalently the exponentiated mean token log-probability: TCS(t^)=exp(1|t^|∑k=1|t^|logPLLM(wk∣w<k))=(∏k=1|t^|PLLM(wk∣w<k))1/|t^|∈(0,1].TCS( t)= \! ( 1| t| _k=1^| t| P_LLM(w_k w_<k) )\;=\; ( _k=1^| t|P_LLM(w_k w_<k) )^1/| t|\;∈(0,1]. (7) Under this convention higher TCSTCS is better (the model was more confident), the score is length-normalized and bounded, and a triple is flagged when TCS(t^)<τTCS( t)<τ for a threshold τ. Being already in (0,1](0,1], TCSTCS enters the aggregate index below directly, with no rescaling. TCSTCS requires access to token log-probabilities from the extracting LLM (white-box setting); it is undefined for a black-box graph. Source Coverage Ratio (SCRSCR) [TRIAGE]. Correctness does not imply completeness. SCRSCR measures how broadly the input yields at least one confident triple. With passages =p1,…,pnP=\p_1,…,p_n\ and threshold τ: SCR=|pi∈:∃t^∈E(pi)s.t.TCS(t^)≥τ|||∈[0,1].SCR= |\p_i :∃\, t∈ E(p_i)\ s.t.\ TCS( t)≥τ\ ||P|\;∈[0,1]. (8) Low SCRSCR indicates that large portions of the input fail to yield confident triples, mapping to the Knowledge Elicitation coverage notion of [2]. Extraction Consistency (ECEC) [TRIAGE]. LLM extraction is sensitive to stochastic decoding [59]. Given k independent extractions E(1),…,E(k)E^(1),…,E^(k) of the same input, ECEC is the average pairwise Jaccard similarity [26]: EC=2k(k−1)∑i<j|E(i)∩E(j)||E(i)∪E(j)|∈[0,1].EC= 2k(k-1) _i<j |E^(i)∩ E^(j)||E^(i)∪ E^(j)|\;∈[0,1]. (9) A semantic variant replaces set overlap with Soft-F1 matching to absorb paraphrastic variation across runs. Cross-Source Corroboration (CSCCSC) [TRIAGE]. Whereas ECEC measures agreement across runs of the same input, CSCCSC measures agreement across independent sources: a triple extracted independently from two or more documents is far more likely to be true. Let src(t^)src( t) be the set of distinct source documents from which a (canonicalized) triple t t is extracted. Then CSC(E)=|t^∈E:|src(t^)|≥2||E|∈[0,1].CSC(E)= |\ t∈ E:|src( t)|≥ 2\||E|\;∈[0,1]. (10) A higher CSCCSC indicates a graph whose facts are corroborated across the corpus rather than resting on single, possibly idiosyncratic, mentions. CSCCSC requires multi-document provenance and therefore applies in the white-box setting. Schema Compliance Rate (SCSC) [TRIAGE]. Schema compliance measures conformance to the ontology’s relation vocabulary and type constraints [44, 46]. With predicate set ℛR, domain/range constraints dom(p),rng(p)dom(p),rng(p), and inferred type type(x)type(x): valid(t^)=[pt^∈ℛ]⋅[type(st^)∈dom(pt^)]⋅[type(ot^)∈rng(pt^)],valid_O( t)=I[p_ t\!∈\!R]·I[type(s_ t)\!∈\!dom(p_ t)]·I[type(o_ t)\!∈\!rng(p_ t)], (11) SC(E)=1|E|∑t^∈Evalid(t^)∈[0,1].SC(E)= 1|E| _ t∈ Evalid_O( t)\;∈[0,1]. (12) SCSC requires an ontology and is undefined without one. Constraint Violation Rate (CVRCVR) [TRIAGE]. Where SCSC is a per-triple pass/fail signal, CVRCVR quantifies how frequently each constraint type is violated. The constraint set is exactly the three checks composing SCSC in eq. 11, =pred,dom,rngC=\ pred, dom, rng\ (predicate membership, domain, and range), with violc(t^)=1−Ic(t^)viol_c( t)=1-I_c( t), where Ic(t^)I_c( t) denotes the c-th indicator composing validO(t^)valid_O( t), so that CVRCVR decomposes SCSC by constraint type rather than introducing new checks: CVR(E)=1|E|∑t^∈E(1||∑c∈violc(t^))∈[0,1].CVR(E)= 1|E| _ t∈ E ( 1|C| _c viol_c( t) )\;∈[0,1]. (13) CVRCVR can be reported per constraint type (e.g., CVRdom,CVRrngCVR_ dom,CVR_ rng) to localize systematic schema drift; it too requires an ontology. Canonicalization Success Rate (CSRlinkCSR_link) [TRIAGE]. A triple may be correct at the string level yet unusable if its entities cannot be grounded to canonical identifiers [50, 27]. Let C(⋅)C(·) map surface forms to canonical identifiers, returning ⊥ when no link is found: CSRlink(E)=1|E|∑t^∈E[C(st^)≠⊥]⋅[C(ot^)≠⊥]∈[0,1].CSR_link(E)= 1|E| _ t∈ EI[C(s_ t)≠ ]·I[C(o_ t)≠ ]\;∈[0,1]. (14) The meaning of “canonical” depends on the regime of section 3.1. In the ontology-grounded regime, C maps to schema or KB identifiers and CSRlinkCSR_link measures external grounding success. In the resolution-based regime (no ontology), C maps to the internally induced canonical set produced by entity resolution, and CSRlinkCSR_link measures internal consistency rather than external grounding. This is the point at which the no-ontology consequence becomes concrete: when OCOC, SCSC, LCLC, and CMPCMP are undefined for lack of a schema, the trustworthiness load shifts onto CSRlinkCSR_link, DNRDNR, ECEC, and TCSTCS. Evidence Attribution Rate (EAREAR) [TRIAGE], optional. For auditability, EAREAR measures how often a provenance evidence span is available for a triple, enabling the verification motivated in section 2. With span(t^)span( t) the (possibly empty) evidence set: EAR(E)=1|E|∑t^∈E[span(t^)≠∅]∈[0,1].EAR(E)= 1|E| _ t∈ EI[span( t)≠ ]\;∈[0,1]. (15) Aggregated Trustworthiness Index (TRIAGE-TITRIAGE -TI) [TRIAGE], optional. To summarize complementary trustworthiness signals into one indicator (all terms already lie in [0,1][0,1] and increase with quality): TRIAGE-TI(E) -TI(E) =ω1TCS+ω2SCR+ω3EC = _1TCS+ _2SCR+ _3EC (16) +ω4CSC+ω5SC+ω6CSRlink, + _4CSC+ _5SC+ _6CSR_link, ∑iωi=1. _i _i=1. where weights ωi _i reflect application risk (e.g., higher ω5 _5 in ontology-critical domains). Because TCSTCS is now bounded in (0,1](0,1], it enters the sum directly with no rescaling, so TRIAGE-TITRIAGE -TI is comparable across datasets and runs. Each component remains independently interpretable. TRIAGE-TITRIAGE -TI is gold-free but not schema-free: its SCSC term is undefined without an ontology. When no ontology is available, SCSC is dropped and the remaining weights are renormalized to sum to one, so TRIAGE-TITRIAGE -TI stays computable and comparable on the schema-free subset. Remediation note (Implementation). Low Implementation metrics call for extraction levers, which leave graph structure and retrieval untouched: refine the extraction prompt, lower decoding temperature, add or rebalance few-shot demonstrations, or re-chunk the input. For example, low SCRSCR or CSCCSC points to under-extraction or single-source fragility (acquire or re-process documents); low ECEC points to decoding instability (reduce temperature, self-consistency voting); low SCSC/CVRCVR points to schema drift (tighten the prompt or predicate inventory). These are the extraction levers of the remediation map (section 4.5). 4.1.3 KG Completion (KGC) Metrics Knowledge Graph Completion (KGC), or link prediction, evaluates a model’s ability to predict missing entities in an existing graph: given (s,p,?)(s,p,?) or (?,p,o)(?,p,o), the relation fixed, it ranks candidate entities so the correct one appears near the top [6]. (The related task of relation prediction (s,?,o)(s,?,o) exists but is not the setup these rank-based metrics target.) As discussed above, KGC is primarily relevant in the black-box regime: when a graph arrives without source text, log-probabilities, or extraction provenance, the extraction-oriented trustworthiness metrics are undefined, and the meaningful question shifts from “was this graph extracted well?” to “is this graph complete?”, which is precisely what link prediction probes. Reporting is in the filtered setting [6]: • Mean Rank: MR=1|Q|∑qrankqMR= 1|Q| _qrank_q; • MRR [55]: MRR=1|Q|∑q1rankqMRR= 1|Q| _q 1rank_q; • Hits@K [52]: Hits@K=1|Q|∑q[rankq≤K]Hits@K= 1|Q| _qI[rank_q≤ K], with rankqrank_q the rank of the correct entity and K∈1,3,10K∈\1,3,10\. Ontology-constrained ranking (MRRtypeMRR_type) [TRIAGE]. Standard KGC ranks the correct entity against all candidates, including type-incompatible ones that the schema already rules out. When an ontology is available, TRIAGE restricts the candidate set to the type-valid entities for the relation, so that completion is judged on semantically plausible candidates rather than trivially excluded ones. Let ℰtype(p)E_type(p) be the set of entities compatible with the domain or range of relation p under the ontology, and let rankqtyperank^type_q be the rank of the correct entity within ℰtype(p)E_type(p): MRRtype=1|Q|∑q∈Q1rankqtype.MRR_type= 1|Q| _q∈ Q 1rank^type_q. (17) MRRtypeMRR_type ties link prediction to the same schema axis as SCSC and CVRCVR: it rewards a model for placing the correct entity ahead of other type-valid alternatives, not merely ahead of entities the ontology would reject anyway. It requires an ontology and is undefined without one. Calibration of completion confidence (ECEECE), adopted. A black-box graph is sometimes accompanied by per-edge confidence scores, or a completion model produces them; their usefulness depends on whether they reflect true correctness likelihood. We adopt the Expected Calibration Error [19] to assess this, binning predictions into B1,…,BMB_1,…,B_M by confidence and comparing per-bin accuracy with confidence: ECE=∑m=1M|Bm|n|acc(Bm)−conf(Bm)|.ECE= _m=1^M |B_m|n\, |acc(B_m)-conf(B_m) |. (18) ECE is a standard metric rather than a contribution of this work; what TRIAGE adds is its role in the framework: a low ECE licenses using a graph’s confidence scores as a trust signal at KG Validation, while a high ECE warns that those scores cannot be trusted and the schema-free structural metrics (DNRDNR, SRRSRR) should carry the assessment instead. This mirrors, on the black-box side, the calibration concern that affects TCSTCS on the white-box side (section 4.1.2). Limitations. Three limitations motivate caution: rank metrics depend on candidate-set size, hampering cross-dataset comparison [25, 48]; closed-world evaluation treats unobserved triples as negatives [6]; and benchmark artifacts such as inverse-relation leakage inflate scores [12, 54]. TRIAGE therefore complements KGC reporting with the intrinsic trustworthiness metrics above rather than relying on rank metrics alone. 4.2 KG Validation Metrics The KG Validation phase assesses the assembled KG before use. A key shift from section 4.1 is one of level: where Implementation metrics are computed per triple, Validation metrics are computed over the whole graph K. Logical consistency is the clearest example: here it is a global property of K under ontology axioms, detected by a reasoner over the entire graph, not a per-triple schema check. Validation metrics fall into two groups: ground-truth-free metrics, computable in any deployment, and benchmark metrics requiring a reference KG ∗K^*. Core question: does the assembled graph hold together well enough to query? The schema-free signal to watch first is the Dead Node Ratio (DNRDNR), with Logical Consistency (LCLC) and Ontology Coverage (OCOC) adding schema-based checks when an ontology is available. Ground-truth-free metrics. Logical Consistency (LCLC, μLC _LC). Absence of internal contradictions in K, detected by OWL reasoning against the ontology [2]. Let Conf()⊆Conf(K) be the set of triples that participate in at least one violated ontology axiom; defining the numerator as this triple count (rather than the number of violation instances, which can exceed |||K|) keeps the score in [0,1][0,1]: μLC=1−|Conf()|||∈[0,1], _LC=1- |Conf(K)||K|\;∈[0,1], (19) where a triple is in Conf()Conf(K) if it occurs in some triple set violating an ontology axiom (disjointness, functionality, cardinality), as found by a reasoner such as HermiT over the full graph. LCLC requires an ontology. Ontology Coverage (OCOC) [TRIAGE]. OCOC measures what fraction of the query-relevant ontology is instantiated in K: OC=|c∈rel:∃t∈instantiatingc||rel|∈[0,1].OC= |\c _rel:∃\,t \ instantiating\ c\||O_rel|\;∈[0,1]. (20) OCOC is the structural-completeness signal of this phase: not whether triples are correct (that is CORCOR) but whether the graph covers the domain as defined by the ontology. Low OCOC on a class suggests either missing source content or extraction failure for that class, both actionable. Dead Node Ratio (DNRDNR) [TRIAGE]. A dead node has degree ≤1≤ 1: isolated (degree 0) or a dangling leaf (degree 1). Such nodes cannot serve as intermediate bridges in multi-hop reasoning: a degree-1 node may still be a valid path endpoint (e.g. an answer entity), but it cannot connect two other entities along a reasoning chain. With node set V of K: DNR=|v∈V:deg(v)≤1||V|∈[0,1].DNR= |\v∈ V: (v)≤ 1\||V|\;∈[0,1]. (21) High DNRDNR signals disconnected entities: surface-form variants, over-segmented mentions, or hallucinated entities with no relational context. It complements LCLC (contradictions) and OCOC (missing concepts) by detecting structurally inert content, and is computable without an ontology. Semantic Redundancy Rate (SRRSRR) [TRIAGE]. Where DNRDNR detects too little connectivity, SRRSRR detects too much duplication: triples that are semantically equivalent or subsumed by others, inflating the graph without adding information. Using the component similarity simsim and a threshold δ, call two triples redundant if all three components match above δ; let Red()Red(K) be the set of triples that duplicate or are subsumed by another: SRR=|Red()|||∈[0,1].SRR= |Red(K)||K|\;∈[0,1]. (22) High SRRSRR indicates a bloated graph that raises retrieval cost (RPCRPC, section 4.3) without improving coverage; it is computable without an ontology. Benchmark metrics (require ∗K^*). When a reference KG ∗K^* is available (e.g., PathQuestion [71], MetaQA [68]), the following metrics from [2] provide gold-anchored evaluation and calibrate the ground-truth-free metrics above. Let crt=(∩∗)|ΠK_crt=(K ^*)|_ _K be the correct subset of K and cpt=(∩∗)|Π∗K_cpt=(K ^*)|_ _K^* the covered subset of ∗K^*. Correctness (CORCOR, μcorrect _correct) [2]. μcorrect=1−|∖crt|||∈[0,1]. _correct=1- |K _crt||K|\;∈[0,1]. (23) Completeness (CMPCMP, μcomplete _complete) [2]. μcomplete=1−|∗∖cpt||∗|∈[0,1]. _complete=1- |K^* _cpt||K^*|\;∈[0,1]. (24) Low CMPCMP is the gold-anchored counterpart of low OCOC: both signal structural incompleteness, but CMPCMP is measured against gold triples and OCOC against the ontology schema. In the TKG lifecycle [2], low CMPCMP is a direct trigger for a KG Update. Remediation note (Validation). Low Validation metrics call for graph-and-schema levers that touch the graph and its schema but not the retrieval algorithm: improve entity resolution and canonicalization (high DNRDNR), prune or merge duplicates (high SRRSRR), repair contradicting triples or axioms (low LCLC), and, importantly, refine, loosen, or extend the ontology, or acquire targeted documents, when OCOC or CMPCMP is low. Ontology refinement is the natural Validation-stage lever, available only when a schema exists; without one, the resolution-based fixes above carry the load. These are the graph-and-schema levers of the remediation map (section 4.5). 4.3 KG Usage Metrics The KG Usage phase queries the validated KG via graph-based retrieval and inference. We first fix the object these metrics operate on. For a query q over the validated KG K with vertex set V, the retrieval function induces a retrieved subgraph S⊆K_S with vertex set VS⊆V_S V. A path “in the subgraph” uses only edges in SK_S. All intrinsic metrics below are ground-truth-free and computable before or during generation; gold-requiring metrics are collected at the end. Core question: does retrieval set up a correct answer? The signal to watch first is Entity Retrieval Coverage (ERCERC), with Reasoning Readiness (RRSRRS) close behind; the answer-side metrics then check faithfulness. We sort the usage metrics into two roles and label each group below accordingly. A per-query predictor takes a graded value on a single query, so it can flag or triage that query before the answer is judged, and it can also be aggregated as the averaged score over a set of queries. A global outcome metric instead summarizes end-task success over a set of queries, as a rate or ratio, and serves as an indicator. This predictor/outcome split is orthogonal to the primary/derived distinction of table 4: the former concerns what a metric measures (a single query versus a set), the latter whether it is an independent signal or a definitional complement. 4.3.1 Query and Answer Entity Resolution Usage metrics compare entities of the query or answer against SK_S. Resolution proceeds in two stages: an extraction step turns free text into candidate entity mentions (an NER model or a noun-chunk tokenizer), and a matching step resolves each mention to a graph node by significant-token overlap against node labels. Let MqM_q be the set of surface mentions extracted from query q, and let γq:Mq→V∪⊥ _q:M_q→ V∪\ \ be the grounding function, where ⊥ denotes an unresolved mention. We define the grounded query-node set as Vqg=γq(m):m∈Mq,γq(m)≠⊥⊆V.V_q^g=\ _q(m):m∈ M_q,\ _q(m)≠ \ V. The query-side denominator remains the number of extracted surface mentions |Mq||M_q|, so that missing or unresolved mentions are penalized. Thus QGRQGR, ERCERC, and RRSRRS measure, respectively, the fraction of surface mentions grounded to the KG, the fraction whose grounded nodes are retrieved into the subgraph, and the fraction of query-mention pairs whose grounded nodes are connected there. The retrieved subgraph has node set VS⊆V_S V. All intersections such as Vqg∩VSV_q^g∩ V_S are therefore intersections between KG node identifiers, not between text mentions and graph nodes. Over-segmented multi-word mentions are then coalesced into the largest matched entity, so each real-world entity contributes a single node. All query-side metrics are measured over the same mention universe MqM_q: QGR and ERC use |Mq||M_q| directly, while RRS uses query-mention pairs through (|Mq|2) |M_q|2. Thus unresolved or unretrieved mentions are penalized consistently across scope, coverage, and connectivity. Convention for degenerate inputs. Throughout this section, a ratio with an empty denominator is reported as N/A rather than assigned a numeric value: QGRQGR, ERCERC, and RRSRRS are N/A when |Mq|=0|M_q|=0 (no entity mention could be extracted from the query); ERPERP and AURAUR are N/A when |VS|=0|V_S|=0 (empty retrieval); and AGRAGR is N/A when |Ma|=0|M_a|=0 (no entity mention could be extracted from the answer). For RHDRHD, queries whose grounded entities are disconnected in SK_S (no finite ℓ(q) (q)) are collected in a separate ℓ=∞ =∞ bucket, so the reported distribution sums to one over 0,1,2,…,∞\0,1,2,…,∞\. These degenerate cases are themselves diagnostic (an empty VqgV_q^g or VSV_S signals an out-of-scope query or a retrieval miss) and are reported as counts alongside the metric. 4.3.2 Retrieval-time Metrics These characterize the retrieved subgraph before the LLM is invoked. They are all per-query predictors. Query Grounding Rate (QGRQGR) [TRIAGE]. Fraction of extracted query mentions that exist as graph nodes at all, a KG-coverage signal distinct from retrieval quality: QGR=|Vqg||Mq|∈[0,1].QGR= |V_q^g||M_q|\;∈[0,1]. (25) Entity Retrieval Coverage (ERCERC) [TRIAGE]. Fraction of all extracted query mentions whose grounded nodes are present in the retrieved subgraph: ERC=|Vqg∩VS||Mq|∈[0,1].ERC= |V_q^g∩ V_S||M_q|\;∈[0,1]. (26) Sharing the denominator |Mq||M_q| with QGRQGR gives ERC≤QGRERC always; the gap QGR−ERCQGR-ERC is exactly the retrieval failure (grounded entities not fetched). Because MqM_q is extracted from the query surface, ERCERC never inspects the answer: it is disjoint from answer-side outcomes such as CC, so using ERCERC to predict them is not circular by construction. Entity Retrieval Precision (ERPERP) [TRIAGE]. The precision complement of ERCERC: are retrieved nodes mostly query-relevant, or bloated by neighborhood expansion? ERP=|Vqg∩VS||VS|∈[0,1].ERP= |V_q^g∩ V_S||V_S|\;∈[0,1]. (27) Because the denominator |VS||V_S| counts all retrieved nodes, typically far larger than the handful of query anchors, ERPERP is low in absolute terms by construction; it is therefore read relatively, comparing retrieval strategies or configurations on the same queries rather than against a fixed threshold. Low ERPERP alongside high ERCERC indicates over-retrieval: a large context with few anchors to the question. Reasoning Readiness Score (RRSRRS) [TRIAGE]. Whether the subgraph is connected enough to support reasoning across the query’s entities: the fraction of query-entity pairs joined by a path in SK_S, with a convention for the single-entity case where the pair count is zero: RRS=|ei,ej⊆Vqg:∃path(ei,ej)inS|(|Mq|2),|Mq|≥2,[Vqg∩VS≠∅],|Mq|=1,RRS= cases |\\e_i,e_j\ V_q^g:∃\,path(e_i,e_j)\ in\ K_S\| |M_q|2,&|M_q|≥ 2,\\[11.99998pt] I [V_q^g∩ V_S≠ ],&|M_q|=1, cases (28) By construction, RRS∈[0,1]RRS∈[0,1]. The denominator counts pairs over all extracted query mentions, so an off-graph entity is penalized. For a single-entity query, readiness reduces to whether that entity is grounded and retrieved (RRS=1RRS=1) or not (RRS=0RRS=0). Because this denominator also charges RRSRRS for entities that were never grounded or retrieved, a low RRSRRS can reflect a grounding or retrieval gap rather than a genuine connectivity failure. RRSRRS is therefore read as a pure connectivity signal only conditional on high ERCERC: the diagnostic “high ERCERC, low RRSRRS” connectivity gap of section 4.5 is exactly the regime in which the entities are present yet disconnected. A more localized variant could replace the denominator by the number of grounded, retrieved query-entity pairs, e.g., pairs in Vqg∩VSV_q^g∩ V_S, but we keep the global denominator over MqM_q because RRS is used in the diagnostic chain as an end-to-end readiness signal: it should decrease not only when retrieved entities are disconnected, but also when grounding or retrieval has already failed. Retrieval Path Cost (RPCRPC) [TRIAGE]. The intrinsic metrics above say nothing about cost, yet the central question of section 1 includes the price of traversal. RPCRPC captures it as the retrieval effort expended per query, measured as the number of nodes expanded during subgraph construction (a hardware-independent proxy) and, optionally, wall-clock path-search time: RPC(q)=|v∈V:vexpanded during retrieval forq|.RPC(q)=|\v∈ V:v\ expanded during retrieval for\ q\|. (29) RPCRPC makes the coverage/cost trade-off observable: a retrieval strategy may raise ERCERC and RRSRRS only by expanding far more of the graph, and RPCRPC exposes that price. It connects directly to SRRSRR (section 4.2), since a redundant graph inflates expansion, and to the traversal-budget failure mode of section 2. 4.3.3 Answer-time Metrics These measure faithfulness of the generated answer to the retrieved subgraph. Let MaM_a be the set of surface mentions extracted from the generated answer A, and let γa:Ma→V∪⊥ _a:M_a→ V∪\ \ be the answer grounding function. We define Vag=γa(m):m∈Ma,γa(m)≠⊥.V_a^g=\ _a(m):m∈ M_a,\ _a(m)≠ \. AGRAGR, AURAUR, and ARFARF are per-query predictors; RHDRHD is a set-level metric and so reads as a diagnostic rather than a single-query signal. Answer Grounding Rate (AGRAGR) [TRIAGE]. Fraction of answer entities present in the retrieved context, detecting hallucinated entities never retrieved: AGR=|Vag∩VS||Ma|∈[0,1].AGR= |V_a^g∩ V_S||M_a|\;∈[0,1]. (30) Answer Utilization Rate (AURAUR) [TRIAGE]. The recall complement of AGRAGR, how much of the retrieved context the answer actually used: AUR=|Vag∩VS||VS|∈[0,1].AUR= |V_a^g∩ V_S||V_S|\;∈[0,1]. (31) As with ERPERP, the denominator |VS||V_S| makes AURAUR structurally low in absolute terms, so it is interpreted relatively across queries or configurations rather than against a fixed threshold. High AGRAGR with low AURAUR suggests a correct answer drawn from a narrow slice of available evidence. Answer Reasoning Faithfulness (ARFARF) [TRIAGE]. ARFARF measures relational faithfulness: when the answer discusses two entities the KG connects, does it state their relationship correctly? This isolates relational hallucination, which entity-level AGRAGR cannot see. Rather than parsing the answer for explicit relational claims (unreliable, because extracting relations from generated text propagates the same errors that afflict any extractor and compounds them at each stage [11, 37]), ARFARF uses the retrieved triples as the reference set of relations the answer could assert. Let CqC_q be the retrieved co-mentioned triples whose endpoints are both grounded in the answer; ARFARF is the fraction whose relation also appears in the answer text A: ARF=|(s,p,o)∈Cq:p⊆A||Cq|,Cq=(s,p,o)∈S:s∈Vag∩VS∧o∈Vag∩VS,ARF= |\(s,p,o)∈ C_q:p A\||C_q|, \\ C_q=\(s,p,o) _S:s∈ V_a^g∩ V_S o∈ V_a^g∩ V_S\, (32) where p⊆Ap A tests lexically whether the relation label appears in A. ARFARF is reported as N/A when |Cq|=0|C_q|=0 (no connected pair co-mentioned). Reasoning Hop Depth (RHDRHD) [TRIAGE], diagnostic. RHDRHD characterizes the structural complexity actually exploited, as the distribution of shortest-path lengths ℓ(q) (q) connecting the grounded query entities in SK_S, with ℓ(q)=0 (q)=0 for single-entity queries whose grounded entity is retrieved, and ℓ(q)=∞ (q)=∞ when the grounded query mentions are not connected in KSK_S or cannot be evaluated because grounding/retrieval failed. RHD(k)=|q∈:ℓ(q)=k|||,k∈0,1,2,…,∞,RHD(k)= |\q : (q)=k\||Q|, k∈\0,1,2,…,∞\, (33) so that ∑kRHD(k)=1 _kRHD(k)=1 including the disconnected (∞) bucket. Mass concentrated at k≤1k≤ 1 indicates the pipeline rarely exploits multi-hop structure, suggesting the KG adds little over flat retrieval for that query class, while mass at ∞ flags queries the subgraph cannot connect at all. RHDRHD is reported as a distribution to preserve diagnostic detail. 4.3.4 Gold-requiring Usage Metrics When gold reasoning paths or gold answers are available (as in PathQuestion and MetaQA), the Usage phase admits gold-anchored metrics that give it the same two-regime treatment as the earlier phases. GPCGPC and Answer Correctness (Hits@1/F1) are global outcome metrics; RPP/RPR are per-query but, unlike the predictors above, need a gold path. Gold Path Coverage (GPCGPC) [TRIAGE]. Fraction of queries whose gold reasoning path Pq∗P^*_q is fully contained in the retrieved subgraph: GPC=|q∈:Pq∗⊆S|||∈[0,1].GPC= |\q :P^*_q _S\||Q|\;∈[0,1]. (34) GPCGPC is the gold-anchored counterpart of RRSRRS/ERCERC. Retrieval Path Precision / Recall (RPP / RPR) [TRIAGE]. Of the retrieved-subgraph edges, the fraction lying on the gold path (precision), and of the gold-path edges, the fraction retrieved (recall): RPP=|S∩Pq∗||S|,RPR=|S∩Pq∗||Pq∗|.RPP= |K_S∩ P^*_q||K_S|, = |K_S∩ P^*_q||P^*_q|. (35) Answer Correctness. The ultimate gold-anchored outcome, Hits@1 and F1 over answer sets [64], used in section 5 as the dependent variable the intrinsic metrics are designed to predict. 4.3.5 Imported Outcome Metrics To validate the intrinsic metrics and close the KG Update loop, we import four outcome metrics from recent graph-RAG benchmarks. They are not TRIAGE contributions; we adopt them as dependent variables. Unlike the intrinsic usage metrics above, three of these (CC, RFRRFR, HHRHHR) require a gold answer or correctness label, not live gold-free signals monitored at deployment; only KNIKNI, built from grounding alone, is gold-free. All four are global outcome metrics: CC and KNIKNI average a per-query label into a rate, whereas RFRRFR and HHRHHR are population-level ratios with no single-query value. KG Net Impact (KNIKNI) [33]. Counterfactual contribution of the KG: answer grounding with the retrieved triples minus that without them. A drift toward KNI≤0KNI≤ 0 across a query class signals stale or irrelevant KG content. Context Coverage (CC) [65]. Fraction of queries whose gold answer is present in the retrieved context, the answer-string analogue of ERCERC. Reasoning Failure Rate (RFRRFR) [65]. Fraction of queries where the answer is in context yet the model still answers wrongly, isolating the generation-stage residual from retrieval failure. Hard Hits Rate (HHRHHR) [70]. HHR=Hits@Hard/Hits@AnyHHR=Hits@Hard/Hits@Any, robustness under an incomplete KG; complements CMPCMP by exposing, at usage time, the queries that KG incompleteness breaks. Remediation note (Usage). Only Usage-stage breaches implicate the retrieval algorithm itself. Concretely, the signature selects the lever: a retrieval gap (high QGRQGR, low ERCERC) calls for a wider seed set or traversal budget; a connectivity gap (high ERCERC, low RRSRRS) calls for a retrieval primitive that recovers the connecting path, such as backward or path-based traversal; and high RPCRPC calls for pruning redundant expansion or a sparser primitive. A retrieval or connectivity gap that persists once extraction and graph health are adequate, or high RPCRPC, motivates architectural changes to retrieval and reasoning, for example super-relation reasoning that adds backward traversal and aggregates relational paths [58], logic-aware multi-hop traversal [38], GNN-based structural retrieval [41], RL-optimized graph indexing [56], agentic restructuring [36], or cross-document construction that adds connective structure [67]. A low QGRQGR instead marks an out-of-scope query (no KG answer to retrieve), and a generation gap (an ungrounded answer despite adequate RRSRRS) points to the generator itself, for example prompting it to cite the retrieved evidence or constraining decoding to it. These are the retrieval levers, not a prescription; the remediation map is described next. 4.4 Metric Dependencies and Redundancy The suite is deliberately broad, and several metrics are related by construction. We make these relationships explicit (table 4) so that a practitioner can distinguish primary signals, worth monitoring and triggering remediation on, from derived ones that add diagnostic detail. We do not claim statistical independence, which can only be established empirically (section 5); the relationships below are definitional or hypothesized. We group them into three kinds. Definitional complements: precision/recall pairs over the same sets, where one is the dual of the other (ERCERC and ERPERP, AGRAGR and AURAUR), and QGRQGR, which fixes the query-mention scope underlying ERCERC and RRSRRS. Aggregates: TRIAGE-TITRIAGE -TI and TFITFI are weighted functions of other metrics, and CVRCVR is a per-constraint refinement of SCSC. Expected empirical correlations (to be tested, not assumed): low TCSTCS is expected to lower SCRSCR, since SCRSCR counts confident triples; high DNRDNR is expected to lower RRSRRS, since dead nodes cannot form connecting paths; high SRRSRR is expected to raise RPCRPC, since redundant structure inflates expansion. For remediation, the primary metrics are the triggers; the derived metrics help localize why a primary metric moved. 4.5 From Metrics to Remediation The usage-stage diagnostic chain. At inference time, a correct graph-grounded answer requires a chain of conditions to hold in sequence (fig. 5), each one necessary for the next, and each observable without gold annotations. The query entities must exist in the KG (QGRQGR), be retrieved into the subgraph (ERCERC), be connected there by a path (RRSRRS), have their relation stated correctly by the answer (ARFARF), and appear in an answer grounded on the retrieved evidence (AGRAGR). Each condition presupposes the previous one: a query entity cannot be retrieved unless it is in the KG, cannot be connected unless it is retrieved, and so on. For the first two links this is exact, ERC≤QGRERC by construction; for the later links it is a logical precondition rather than a numeric bound. This ordering is what makes the chain diagnostic: reading the metrics in sequence, the first link that falls below its threshold θ localizes the failure to a single condition. We call the resulting label the failure’s signature: out-of-scope or grounding failure (low QGRQGR), retrieval gap (high QGRQGR, low ERCERC), connectivity gap (high ERCERC, low RRSRRS), relational gap (high RRSRRS, low ARFARF), and generation gap (high ARFARF, low AGRAGR). The first three links are computable before the LLM is invoked, enabling pre-generation triage; the last two require the produced answer and so apply post hoc. Each link is a necessary, observable proxy rather than a guarantee: a satisfied link is required for success but does not certify it, since a structural signal such as RRSRRS cannot distinguish a correct connecting path from a spurious one. The thresholds θ separating “low” from “adequate” are not assumed; section 5 specifies how to estimate them empirically. QGR<θQGR<θ? out-of-scope/grounding failure: entities not grounded in the KG ERC<θERC<θ? retrieval gap: in KG, not retrieved RRS<θRRS<θ? connectivity gap: retrieved, not connected ARF<θARF<θ? relational gap: connected, relation wrong AGR<θAGR<θ? generation gap: answer not grounded answer supportedyesyesyesyesyesnonononono Figure 5: The usage-stage diagnostic chain as a decision cascade. Reading the metrics in order, the first one below its threshold θ names the failure’s signature; if all pass, the answer is structurally supported. Each signature then maps to the levers that can remedy it; the chain localizes the failing condition, whose root cause may lie upstream. What the chain does and does not claim. The chain is a usage-stage detector: it operates on a query and the validated KG, and it localizes which condition failed, not why. The cause of a broken link may lie upstream, in extraction or graph construction, so the build- and validation-time metrics (section 4.1, section 4.2) are the candidate explanations, and the localization test in Section 5.2 examines whether they co-move with chain failures. From a signature to remediation. Diagnosis is the contribution; what to do about a signature is a map from the failing stage to the levers that can act on it, not a fixed order. Each stage has characteristic levers: 1. Extraction levers. Low TCSTCS, SCRSCR, ECEC, CSCCSC, or SCSC/CVRCVR points to extraction, addressable by prompt refinement, lower decoding temperature, rebalanced few-shot demonstrations, re-chunking, or targeted re-extraction of low-coverage regions, with no change to graph structure or retrieval. 2. Graph and schema levers. Low OCOC, high DNRDNR, high SRRSRR, low μLC _LC, or low μcomplete _complete points to the assembled graph or its schema, addressable by improved entity resolution and canonicalization, duplicate pruning, contradiction repair, ontology refinement or extension, and document acquisition, all of which touch the KG but not the retrieval algorithm. 3. Retrieval levers. A retrieval or connectivity gap, or high RPCRPC, that persists once extraction and graph health are adequate points to the retrieval and reasoning algorithm; example directions are given in the Usage remediation note above. We deliberately do not rank these levers by cost: re-extraction reprocesses the whole corpus, a graph fix reprocesses the assembled graph, and a retrieval change may be a configuration swap or a new method, so their relative cost is deployment-specific rather than universal. Which lever is cheapest and most effective for a class of failures is an empirical question for the validation protocol in Section 5.2 and future remediation studies. What TRIAGE adds over a lifecycle that triggers an undifferentiated update is this targeting: the diagnosis names the stage and the levers that can address it. Closing the loop. Each lever set corresponds to re-entering a KG Update cycle at a different depth: extraction levers re-run extraction, graph-and-schema levers revise the assembled graph or ontology, and retrieval levers alter the retrieval algorithm. A KG Update modifies the shared graph, so it must be justified by population behavior rather than a single query: the build- and validation-time metrics are already graph-level, whereas the usage-stage metrics are per-query and must be aggregated before they can trigger an update. For a per-query metric m and a query window W, an update fires when aggq∈Wm(q)<θagg_q∈ W\,m(q)<θ, with aggagg a mean, a low quantile, or the rate of queries failing the threshold; a single low m(q)m(q) drives only per-query triage, a fallback or a warning, not a change to the graph. A persistent aggregate breach then triggers an update targeted at the diagnosed stage, which operationalizes the TKG lifecycle dimension [2] as a control loop for automated graph-RAG deployments. 5 Recommended Evaluation Protocol This section makes one empirical point and frames the rest. Because TRIAGE’s deployed instrument is gold-free, its evaluation is a question of construct validity resting on three claims: that the suite is computable gold-free on real graphs, predictive of downstream failure, and localizing of where a failure originates. Two of these we establish here, on the usage metrics. Computability we show by construction, computing the full gold-free usage suite across five retrievers and 480480 query-by-retriever observations on a real KG with no gold annotation at scoring time (section 5.1). The proof of concept provides preliminary evidence of predictive validity: the structural metrics indicate whether a faithful answer’s evidence is present in the retrieved subgraph, when the reader may often produce an answer that seems correct from parametric memory, although retrieval has failed. In that case end-answer accuracy would certify a graph our metrics correctly flag as deficient; this silent-success residual is a concrete case for TRIAGE’s design, scoring each stage gold-free and before the answer rather than trusting end-to-end accuracy. The localizing claim and threshold calibration need larger, more varied benchmarks; rather than run them here we specify them as a falsification protocol (section 5.2) that a later instantiation can execute to test, not assume, them. The extraction metrics, which need a document-grounded corpus, are left to that protocol’s extensions. 5.1 Proof of concept We instantiate the usage stage on PathQuestion [71] over its 3-hop knowledge base (PQ-3H: 1,836 entities, 2,839 triples). Because natural PathQuestion items name a single topic entity, leaving ERCERC and RRSRRS saturated at 11, we build a distance-controlled probe set: 108 synthetic questions (12 per category) naming two or more KB entities at controlled distances. Each category stresses one link of the diagnostic chain: off-topic questions contain only non-KB entities, semi questions contain one KB entity and one fictional entity, pair-k questions contain two entities at undirected distance k = 1,…,4, disc questions contain two entities in different components, and triple/quad questions contain three or four entities on a connected path. The 108 questions are designed for diagnostic coverage rather than as a natural query distribution. The 12 off-topic questions test the out-of-scope link of the chain, but they are excluded from the predictive-validity analysis because their gold answers are undefined with respect to the KG. The predictive analysis therefore uses 96 distinct questions. Since each question is run through five retrievers, the pooled analysis contains 480 question-by-retriever observations. These 480 observations should not be read as 480 independent questions; they are five retrieval views of the same 96-question probe. Entities are sampled at the target structure, verbalized into a fixed template (“how are X and Y related?”), and re-extracted into MqM_q and grounded into VqgV_q^g by the deployment pipeline; answers come from an LLM reader (mistral-small-3.2). The structural metrics need no gold answer, and the probe is reproducible from the KB alone. Retrieval configurations used as a diagnostic testbed. We do not evaluate these retrievers as systems in their own right. Instead, we use five retrieval configurations as a representative diagnostic testbed: they induce different coverage, connectivity, and cost profiles, allowing us to test whether the proposed structural metrics track evidence availability. Each configuration is a lightweight re-implementation following an existing approach’s core principle rather than an exact reproduction. The first four share a common n-hop beam expansion and differ only in their seed entities: gold-entity (the gold topic entity, an oracle upper bound on linking), extracted-entity (its deployable counterpart, from LLM-extracted, graph-linked entities), and, following LightRAG’s dual-level retrieval [20], light-entity (local mode: top-k entities most similar to the question) and light-relation (global mode: endpoints of the top-k most similar triples). The fifth, path, follows PathRAG [9]: it forgoes beam expansion and extracts flow-pruned relational paths between anchors. Predictive validity: structural metrics track context coverage. On the PQ-3H probe set we ask whether the structural metrics are associated with the framework’s two gold-anchored outcome metrics (section 4). For each question-retriever pair, we store the retrieved subgraph and the generated answer once. We then compute C and Hits@1 offline from these stored outputs, without rerunning retrieval or generation: we pool all five retrieval configurations into one population of 480 question-by-retriever observations (off-topic excluded), and for each we score two outcomes against the probe’s gold answers: Context Coverage (C: all gold-answer entities lie in the retrieved subgraph, a reader-free outcome) and Answer Correctness (Hits@1: a gold answer appears in the generated answer). The predictors ERC and RRS are computed without gold annotations; gold answers are used only afterwards to compute C and Hits@1 as validation outcomes. We then stratify the population by entity coverage (ERCERC) and by connectivity (RRSRRS). Table 5 reports both outcomes per stratum. Example: why answer correctness can hide a retrieval failure. Consider a query “How are X and Y related?”, whose gold answer requires an intermediate entity Z. A retriever may return both query entities X and Y, giving ERC=1ERC=1, but fail to retrieve a connecting path between them, giving RRS=0RRS=0. If Z is absent from the retrieved subgraph, then CC=0C=0, because the evidence needed for a faithful answer is missing. A strong reader may nevertheless answer with Z from parametric memory, yielding Hits@1=1Hits@1=1. In such a case, answer correctness marks the run as successful, whereas the structural metrics flag that the retrieved evidence was insufficient. In this controlled probe, the structural metrics strongly separate cases where the gold evidence is present from cases where it is missing. Reader-free CC rises with ERCERC and jumps sharply once retrieval is complete, from 44.3%44.3\% when ERC<0.5ERC<0.5 (and a near-flat 45.5%45.5\% for partial coverage) to 98.4%98.4\% at ERC=1ERC=1 (a 5454-point gap), and the Youden-optimal operating point for predicting low CC sits at θERC=0.67 _ERC=0.67 (Youden J=0.67J=0.67). Connectivity tells the same story on its own axis: a disconnected retrieval (RRS=0RRS=0) covers the gold answer only 54.8%54.8\% of the time against 93.2%93.2\% when RRS>0RRS>0. This suggests, on the PQ-3H probe and across retrieval configurations, that the structural metrics can indicate whether the evidence a faithful answer needs is present. The reader masks the gap, which is why the structural metrics are scored independently. Reader Answer Correctness (Hits@1) tracks the same metrics but far more weakly: it climbs only from 77.1%77.1\% at ERC<0.5ERC<0.5 to 91.4%91.4\% at full coverage (a 1414-point gap), and stays near 80%80\% even where retrieval is structurally inadequate. The reader is answering many low-coverage questions from parametric memory rather than from the retrieved subgraph, the silent-success residual of section 2, here measured directly: a correct answer co-occurs with broken retrieval, so end-answer accuracy alone would certify a graph the structural metrics correctly flag as deficient. This is the concrete motivation for TRIAGE scoring retrieval adequacy before and independently of the answer. It also bounds what this probe can show: because the reader floor is high, a clean Hits@1 threshold needs harder questions and a weaker-prior reader, and a calibrated θ with significance testing is deferred to the larger benchmarks of the protocol below. Two construct caveats accompany this probe. The predictor and outcome are disjoint by construction: ERCERC scores grounded query mentions while CC scores gold-answer entities, so the separation is not tautological, though the same extraction-and-linking pipeline builds MqM_q, VqgV_q^g, and the retrieval, which can inflate ERCERC independently of graph quality (a coupling the gold-entity oracle retriever isolates). The threshold θERC _ERC is fit in-sample on these 480480 pooled, non-independent observations (five retrievers per question), so it is descriptive rather than a validated operating point, with calibrated thresholds and clustered confidence intervals deferred to the protocol below. Table 5: Predictive validity on the PQ-3H probe set, pooling all five retrievers (480480 question-by-retriever observations, off-topic excluded). Context Coverage (CC) is reader-free (all gold-answer entities in the retrieved subgraph); Answer Correctness (Hits@1) is the reader outcome. Both rise with entity coverage (ERCERC) and connectivity (RRSRRS), but CC separates far more sharply: the structural metrics indicate whether the evidence is present in this controlled probe, while the strong reader partly masks the gap at answer time (the parametric silent-success residual). The Youden-optimal threshold for predicting low CC is θERC=0.67 _ERC=0.67. Boldface marks the values discussed in the text. The table is descriptive: confidence intervals are not reported here because the pooled observations are not independent, with each question appearing under five retrievers; calibrated clustered uncertainty estimates are deferred to the full protocol in Section 5.2. Stratum n CC (%) Hits@1 (%) by entity coverage (ERCERC) ERC<0.5ERC<0.5 70 44.3 77.1 0.5≤ERC<10.5 <1 165 45.5 84.2 ERC=1ERC=1 245 98.4 91.4 by connectivity (RRSRRS) RRS=0RRS=0 261 54.8 82.8 RRS>0RRS>0 219 93.2 91.8 5.2 A falsification protocol for trust metrics The proof of concept above is deliberately small and structurally controlled. It is sufficient to show that the usage metrics are computable and that, in one setting, they separate evidence-present from evidence-missing cases. It is not sufficient to validate thresholds, prove localization, or demonstrate operational triage utility. We therefore treat the remaining claims as hypotheses and specify, for each one, the expected effect, the concrete test, and the outcome that would refute it, so that a later instantiation tests the claim rather than searching for support. All comparisons share one reader; answer quality uses Hits@1 and answer-set F1; confidence intervals use BCa bootstrap resampling [15, 14] and paired comparisons a paired bootstrap, and a claim holds only when its effect lies in the predicted direction with a confidence interval excluding the null. Predictive validity (full study). Expected: Hits@1 increases monotonically across ERCERC (and separately RRSRRS) quartiles, with a sharp drop below a threshold θ. Test: stratify queries by quartile on WebQSP [64] (4,737 natural questions over Freebase, 1–2 hop) and report per-quartile Hits@1 with paired-bootstrap significance for the Q4–Q1 gap; estimate θ by the Youden index for predicting answer failure; repeat per hop depth on PathQuestion [71] (≈ 2.6K synthetic multi-hop questions with gold reasoning paths) and MetaQA [68]. Falsified if the Q4–Q1 gap is null or the ordering is non-monotone; passing extends the proof of concept to natural questions and measures the parametric silent-success residual of section 2, where the reader succeeds despite low ERCERC. Localization. Expected: low upstream graph-health metrics (e.g. TCSTCS at Implementation, CMPCMP at Validation) co-move with low downstream ERCERC/RRSRRS, and the five diagnostic signatures of section 4.5 (out-of-scope, retrieval gap, connectivity gap, relational gap, generation gap) are mutually distinguishable by their metric profiles. Test: report per-query Spearman ρ between each upstream metric and ERCERC/RRSRRS, with confidence bands and stratification by query type, together with the fraction of failures in each signature; this is an alignment study, not a causal-propagation claim. Falsified if the correlations are null or the signatures are not separable, in particular if the relational and generation gaps fail to isolate failures that survive the structural links (high ERCERC/RRSRRS). Utility. Expected: routing low-ERCERC/RRSRRS queries before generation, to a fallback retrieval strategy or a low-confidence warning, improves end-task reliability over no triage. Test: compare the same reader over the same KG with and without ERCERC/RRSRRS pre-inference triage, reporting Hits@1/F1 and the fraction of silent failures averted. Falsified if triage yields no reliability gain at any operating point. Two threads are left to future work. Triple-level metrics that depend on human judgment (e.g. fidelity calibration) require double annotation against the source, labelling each triple Supported, Unsupported, or Contradicted with a third annotator adjudicating, and reporting Cohen’s κ [10] and Krippendorff’s α [31]. Three validations also remain open: the extraction metrics on a document-grounded corpus (white-box construction with source text), whether ontology grounding raises structural quality (OCOC up, DNRDNR and SRRSRR down), and the remediation map, namely that the lever indicated by a diagnosed stage resolves a substantial share of failures at a measurable, deployment-specific cost. 6 Conclusion We presented TRIAGE, a stage-aware instrumentation framework that instruments automated, document-grounded graph-RAG pipelines end to end, with construction-stage metrics that apply to any extracted KG, automated or not. Informed by the TKG engineering methodology [2] and adapted to the automated setting, TRIAGE attaches stage-specific, independently interpretable confidence metrics to three phases (KG Implementation, KG Validation, and KG Usage) without requiring a single composite score and without relying on gold annotations at deployment time. Its central question is deliberately twofold: not only whether the graph underpinning retrieval can be trusted, but at what retrieval and computational cost it can be used. Summary of contributions. Three ideas distinguish TRIAGE beyond the metric definitions themselves. First, it organizes metrics by two computability axes, the external reference each needs (none, an ontology, or a gold standard) and the visibility of the extraction (white-box or black-box), making explicit which signals survive in gold-free, schema-free, or black-box settings and showing that the core assessment degrades gracefully rather than failing when conditions are not ideal; the gold-requiring metrics sit apart as offline calibration yardsticks rather than deployment signals. Second, it separates primary signals from derived ones through a dependency analysis, yielding a minimal set worth monitoring. Third, and most distinctively, it shows that the usage-stage metrics form a diagnostic chain whose first broken link localizes a failure, and that this localization maps to the stage levers that can remedy it: extraction, graph-and-schema, or retrieval. Targeting the update to the diagnosed stage is what separates TRIAGE from a lifecycle that triggers an undifferentiated update; the relative cost of the levers is deployment-specific and left to empirical measurement (section 5). A usage-stage proof of concept gives preliminary, single-probe evidence for this diagnostic claim: a capable reader can answer from its parametric knowledge when retrieval has missed the needed evidence, so retrieval adequacy must be scored before and independently of the answer (section 5). TRIAGE as one iteration of the TKG lifecycle. TRIAGE covers the methodology dimension of TKG for one KG version. Metric breaches, such as a rising DNRDNR or low aggregate ERCERC/RRSRRS over a query class, or rising RPCRPC, provide principled triggers for a new KG Update cycle, targeted at the diagnosed stage. This closes the loop with the TKG lifecycle dimension and points toward continuous, monitored KG evolution in operational deployments. Limitations. Several limitations should be acknowledged. First, validation is still partial: a usage-stage proof of concept exercises the predictive claim, but the full suite, the remediation map, and the cross-stage links are specified rather than tested at scale; section 5 gives the protocol for the remaining validation. Second, TCSTCS relies on LLM token log-probabilities as a confidence proxy, which are known to be imperfectly calibrated for factual correctness [30, 29]; a miscalibrated TCSTCS may flag correct triples or pass incorrect ones. Third, the KG Validation expert step is specified methodologically but its cost and reliability at scale remain open. Fourth, the schema-dependent metrics (OCOC, SCSC, LCLC, CMPCMP) assume an available ontology; without one, assessment relies on the schema-free subset, as section 4 makes explicit. Fifth, ARFARF uses lexical relation matching, which may miss paraphrased or implicit relational claims. Sixth, the separation of primary from derived metrics is definitional rather than statistical: genuine non-redundancy can only be established empirically (section 5), and some primaries may prove correlated in practice. Seventh, the usage-stage links are necessary but not sufficient conditions: a structural signal such as RRSRRS cannot distinguish a correct connecting path from a spurious one, so a satisfied chain certifies retrievability, not answer correctness. Eighth, the usage metrics depend on reliable query- and answer-entity extraction, so errors in MqM_q, MaM_a, or their grounding into VqgV_q^g and VagV_a^g propagate into QGRQGR, ERCERC, RRSRRS, AGRAGR, AURAUR, and ARFARF, and the decision thresholds θ estimated on a benchmark may not transfer unchanged to deployment. Future work. Beyond empirical validation, TRIAGE opens several directions: (i) extending to multi-hop, multi-entity query decomposition; (i) automating KG Update triggers from metric thresholds over a rolling query window, with the diagnosis selecting the levers; (i) formalizing the KG Validation expert step with active learning to reduce annotation cost; (iv) replacing lexical ARFARF with an NLI-based relational-faithfulness check; and (v) surfacing the stage-localized metrics in a per-phase diagnostic interface that maps metric drifts onto the remediation map, enabling continuous, cost-aware KG health monitoring in operational deployments. Acknowledgments The authors thank Sabrina Chaouche and Emna Amdouni for their help and remarks. References [1] O. K. Akindele, B. K. Mishra, and K. Y. Wertheim (2025) A knowledge graph and a tripartite evaluation framework make retrieval-augmented generation scalable and transparent. arXiv preprint arXiv:2509.19209. Cited by: §2.2.1. [2] E. Amdouni, L. Mattioli, F. Adjed, A. Awadid, M. Gonzalez, L. Cantat, and J. Mattioli (2026) An end-to-end trustworthy knowledge graph engineering methodology. In 16th International Conference on Performance, Safety and Robustness in Complex Systems and Applications (PESARO), Cited by: item 1, item 2, §1, §2.3, §3.2, §3.3, §3, §4.1.2, §4.2, §4.2, §4.2, §4.2, §4.2, §4.5, Table 2, Table 2, Table 2, §4, §6. [3] M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni (2007) Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI), p. 2670–2676. Cited by: §2.1.1. [4] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin (2003) A neural probabilistic language model. Journal of Machine Learning Research 3, p. 1137–1155. External Links: Link Cited by: §4.1.2. [5] S. Bhardwaj, S. Aggarwal, and Mausam (2019) CaRB: a crowdsourced benchmark for open IE. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 6262–6267. External Links: Document, Link Cited by: §2.1, §4.1.1, Table 2. [6] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko (2013) Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, Note: Introduces the standard link prediction evaluation by ranking corrupted head/tail entities and reports MR and Hits@10. External Links: Link Cited by: §2.1, §2.1, §4.1.3, §4.1.3, Table 2. [7] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015) A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, p. 632–642. External Links: Document, Link Cited by: §4.1.1. [8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, p. 1877–1901. External Links: Link Cited by: §2.1.1, §2.1.1. [9] B. Chen, Z. Guo, Z. Yang, Y. Chen, J. Chen, Z. Liu, C. Shi, and C. Yang (2026) Pathrag: pruning graph-based retrieval augmented generation with relational paths. In Proceedings of the AAAI conference on artificial intelligence, Vol. 40, p. 30183–30191. Cited by: §1, §1, §2.2.1, §2.2, §2.2, §5.1. [10] J. Cohen (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1), p. 37–46. External Links: Document Cited by: §5.2. [11] L. Cui, F. Wei, and M. Zhou (2018) Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), p. 407–413. External Links: Document Cited by: §4.3.3. [12] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel (2018) Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, p. 1811–1818. Note: Uses the standard ranking protocol for link prediction and reports MRR and Hits@K (commonly Hits@1/3/10). External Links: Document, Link Cited by: §2.1, §2.1, §4.1.3. [13] D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson (2024) From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130. Cited by: §1, §1, §2.2.1, §2.2, §2.2. [14] B. Efron and R. J. Tibshirani (1994) An introduction to the bootstrap. Chapman and Hall/CRC, New York, NY, USA. External Links: Document, Link Cited by: §5.2. [15] B. Efron (1987) Better bootstrap confidence intervals. Journal of the American Statistical Association 82 (397), p. 171–185. External Links: Document, Link Cited by: §5.2. [16] D. Fan, Z. Xue, S. Liu, and Q. Tan (2026) Do we still need graphrag? benchmarking rag and graphrag for agentic search systems. arXiv preprint arXiv:2604.09666. Cited by: §2.2.1. [17] M. Färber, F. Bartscherer, C. Menne, and A. Rettinger (2018) Linked data quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO. Semantic Web 9 (1), p. 77–129. External Links: Document Cited by: §2.3. [18] R. Gelin (2024) Confiance.ai program: software engineering for a trustworthy AI. In Producing Artificial Intelligent Systems, Studies in Computational Intelligence, p. 11–29. External Links: Document Cited by: §2.3. [19] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), Vol. 70, p. 1321–1330. External Links: Link Cited by: §4.1.3, Table 2. [20] Z. Guo, L. Xia, Y. Yu, T. Ao, and C. Huang (2024) Lightrag: simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779 2 (3). Cited by: §1, §1, §2.2.1, §2.2, §2.2, §5.1. [21] H. Han, L. Ma, Y. Wang, H. Shomer, Y. Lei, Z. Qi, K. Guo, Z. Hua, B. Long, H. Liu, et al. (2025) Rag vs. graphrag: a systematic evaluation and key insights. arXiv preprint arXiv:2502.11371. Cited by: §1, §2.2.1. [22] High-Level Expert Group on Artificial Intelligence (AI HLEG) (2019) Ethics guidelines for trustworthy AI. Technical report European Commission, Brussels. External Links: Link Cited by: §2.3. [23] A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. de Melo, C. Gutierrez, S. Kirrane, J. E. L. Gayo, R. Navigli, S. Neumaier, et al. (2021) Knowledge graphs. ACM Computing Surveys 54 (4), p. 1–37. External Links: Document Cited by: §1. [24] O. Honovich, R. Aharoni, J. Herzig, H. Taitelbaum, D. Kukliansy, V. Cohen, T. Scialom, I. Szpektor, A. Hassidim, and Y. Matias (2022) TRUE: re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, p. 3905–3920. External Links: Document, Link Cited by: §2.1.1, §2.1, §2.2.2, §4.1.1, Table 2. [25] C. T. Hoyt, M. Berrendorf, M. Galkin, V. Tresp, and B. M. Gyori (2022) A unified framework for rank-based evaluation metrics for link prediction in knowledge graphs. Note: arXiv preprintSurveys and formalizes rank-based metrics for KG link prediction, motivating their use in the absence of explicit negatives. External Links: 2203.07544, Link Cited by: §2.1, §2.1, §4.1.3. [26] P. Jaccard (1901) Étude comparative de la distribution florale dans une portion des alpes et du jura. Bulletin de la Société Vaudoise des Sciences Naturelles 37, p. 547–579. External Links: Link Cited by: §4.1.2. [27] H. Ji, J. Nothman, and B. Hachey (2014) Overview of TAC-KBP2014 entity discovery and linking tasks. In Proceedings of the Text Analysis Conference (TAC), External Links: Link Cited by: §4.1.2. [28] Z. Ji, N. Lee, R. Frieske, et al. (2023) Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), p. 1–38. External Links: Document Cited by: §1, §2.1.1. [29] Z. Jiang, J. Araki, H. Ding, and G. Neubig (2021) How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9, p. 962–977. External Links: Document, Link Cited by: §4.1.2, §6. [30] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. (2022) Language models (mostly) know what they know. External Links: 2207.05221, Document, Link Cited by: §4.1.2, §6. [31] K. Krippendorff (2004) Content analysis: an introduction to its methodology. 2 edition, SAGE Publications, Thousand Oaks, CA. Cited by: §5.2. [32] H. W. Kuhn (1955) The hungarian method for the assignment problem. Naval Research Logistics Quarterly 2 (1–2), p. 83–97. External Links: Document Cited by: §4.1.1. [33] M. Kumar, B. Sarmah, and S. Pasquali (2026) FinReflectKG–hallubench: graphrag hallucination benchmark for financial question answering systems. arXiv preprint arXiv:2603.20252. Cited by: §1, §2.2.1, §4.3.5, Table 2. [34] C. Laudy, V. Alonso, C. Reverdy, and J. Dreo (2025) First high-level information fusion competition: feedback and lessons learned. In Proceedings of the 28th International Conference on Information Fusion (FUSION), Cited by: §2.3. [35] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, Vol. 33, p. 9459–9474. Cited by: §1. [36] J. Lin, S. Liu, J. Song, S. Wang, J. Shun, and Y. Zhu (2025) Efficient and transferable agentic knowledge graph RAG via reinforcement learning. arXiv preprint arXiv:2509.26383. Cited by: §4.3.5. [37] C. Ling, X. Zhao, X. Zhang, W. Cheng, Y. Liu, Y. Sun, M. Oishi, T. Osaki, K. Matsuda, J. Chen, X. Bo, T. Zhang, et al. (2023) Improving open information extraction with large language models: a study on demonstration uncertainty. arXiv preprint arXiv:2309.03433. Cited by: §4.3.3. [38] H. Liu et al. (2025) HopRAG: multi-hop reasoning for logic-aware retrieval-augmented generation. arXiv preprint arXiv:2502.12442. Cited by: §4.3.5. [39] L. Luo, Z. Zhao, G. Haffari, Y. Li, C. Gong, and S. Pan (2024) Graph-constrained reasoning: faithful reasoning on knowledge graphs with large language models. arXiv preprint arXiv:2410.13080. Cited by: §2.2.2, §3.3, §3.3. [40] J. Mattioli, L. Mattioli, and M. Gonzalez (2025) A brief overview of key quality metrics for knowledge graph solution: illustration on digital NOTAMs. Proceedings of the AAAI Symposium Series 7 (1), p. 206–213. External Links: Document Cited by: §2.3. [41] C. Mavromatis and G. Karypis (2024) Gnn-rag: graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139. Cited by: §4.3.5. [42] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald (2020) On faithfulness and factuality in abstractive summarization. In Proceedings of ACL, Cited by: §1, §2.1.1, §2.2.2. [43] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022) Rethinking the role of demonstrations: what makes in-context learning work?. In Proceedings of EMNLP, Cited by: §2.1.1. [44] M. Nickel, K. Murphy, V. Tresp, and E. Gabrilovich (2015) A review of relational machine learning for knowledge graphs. Proceedings of the IEEE 104 (1), p. 11–33. External Links: Document Cited by: §2.1, §4.1.2. [45] OECD (2019) Recommendation of the council on artificial intelligence. Note: OECD Legal Instruments, OECD/LEGAL/0449 External Links: Link Cited by: §2.3. [46] H. Paulheim (2017) Knowledge graph refinement: a survey of approaches and evaluation methods. Semantic Web 8 (3), p. 489–508. External Links: Document Cited by: §2.1, §2.3, §4.1.2. [47] N. Reimers and I. Gurevych (2019) Sentence-bert: sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 3980–3990. External Links: Document, Link Cited by: §4.1.1, Table 2. [48] D. Ruffinelli, S. Broscheit, and R. Gemulla (2020) You can teach an old dog new tricks! on training knowledge graph embeddings. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4.1.3. [49] G. Salton and M. J. McGill (1983) Introduction to modern information retrieval. McGraw-Hill, New York, NY, USA. External Links: ISBN 9780070544840 Cited by: §4.1.1. [50] W. Shen, J. Wang, and J. Han (2015) Entity linking with a knowledge base: issues, techniques, and solutions. IEEE Transactions on Knowledge and Data Engineering 27 (2), p. 443–460. External Links: Document, Link Cited by: §4.1.2. [51] G. Stanovsky and I. Dagan (2016) Creating a large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, p. 2300–2305. External Links: Document, Link Cited by: §2.1.1, §2.1, §4.1.1. [52] Z. Sun, Z. Deng, J. Nie, and J. Tang (2019) RotatE: knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations (ICLR), Note: A standard KGE paper evaluating link prediction with rank-based metrics such as MRR and Hits@K. External Links: Link Cited by: §2.1, 3rd item. [53] E. Tabassi (2023) Artificial intelligence risk management framework (AI RMF 1.0). Technical report Technical Report NIST AI 100-1, National Institute of Standards and Technology. External Links: Document Cited by: §2.3. [54] K. Toutanova and D. Chen (2015) Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), Beijing, China, p. 57–66. External Links: Document, Link Cited by: §2.1, §4.1.3. [55] T. Trouillon, J. Welbl, S. Riedel, E. Gaussier, and G. Bouchard (2016) Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 48, p. 2071–2080. Note: Reports standard link prediction metrics including MRR and Hits@K on common benchmarks. External Links: Link Cited by: §2.1, §2.1, 2nd item. [56] H. T. Tsang, J. Bai, H. Huang, Q. Xiao, T. Zheng, B. Xu, S. Liu, and Y. Song (2025) AutoGraph-R1: end-to-end reinforcement learning for knowledge graph construction. arXiv preprint arXiv:2510.15339. Cited by: §4.3.5. [57] P. Wang, B. Xu, L. Zhang, S. Wang, M. Du, C. Zhu, and Z. Mao (2026) WildGraphBench: benchmarking graphrag with wild-source corpora. arXiv preprint arXiv:2602.02053. Cited by: §1, §2.2.1. [58] S. Wang et al. (2025) Reasoning of large language models over knowledge graphs with super-relations. In International Conference on Learning Representations (ICLR), Note: arXiv:2503.22166 Cited by: §4.3.5. [59] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023) Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), External Links: 2203.11171, Document, Link Cited by: §2.1.1, §4.1.2. [60] J. Wei, X. Wang, D. Schuurmans, M. Bosma, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Cited by: §2.1.1. [61] A. Williams, N. Nangia, and S. Bowman (2018) A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, p. 1112–1122. External Links: Document, Link Cited by: §4.1.1. [62] Z. Xiang, C. Wu, Q. Zhang, S. Chen, Z. Hong, X. Huang, and J. Su (2025) When to use graphs in rag: a comprehensive analysis for graph retrieval-augmented generation. arXiv preprint arXiv:2506.05690. Cited by: §1, §2.2.1. [63] S. Yao, J. Zhao, D. Yu, et al. (2023) ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, Cited by: §2.1.1. [64] W. Yih, M. Richardson, C. Meek, M. Chang, and J. Suh (2016) The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), p. 201–206. External Links: Document Cited by: §4.3.4, Table 2, §5.2. [65] Y. Zarrinkia, V. Srinivasan, and A. Thomo (2026) The reasoning bottleneck in graph-rag: structured prompting and context compression for multi-hop qa. arXiv preprint arXiv:2603.14045. Cited by: §1, §2.2.1, §4.3.5, §4.3.5, Table 2, Table 2. [66] A. Zaveri, A. Rula, A. Maurino, R. Pietrobon, J. Lehmann, and S. Auer (2016) Quality assessment for linked data: a survey. Semantic Web 7 (1), p. 63–93. External Links: Document Cited by: §2.3. [67] H. Zhang, J. Si, G. Yan, B. Qi, P. Cai, S. Mao, D. Wang, and B. Shi (2025) RAKG: document-level retrieval augmented knowledge graph construction. arXiv preprint arXiv:2504.09823. Cited by: §4.3.5. [68] Y. Zhang, H. Dai, Z. Kozareva, A. Smola, and L. Song (2018) Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §2.2.2, §4.2, §5.2. [69] T. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh (2021) Calibrate before use: improving few-shot performance of language models. In Proceedings of ICML, Cited by: §2.1.1. [70] D. Zhou, Y. Zhu, X. Wang, H. Zhou, Y. He, J. Chen, S. Staab, and E. Kharlamov (2025) What breaks knowledge graph based rag? benchmarking and empirical insights into reasoning under incomplete knowledge. arXiv preprint arXiv:2508.08344. Cited by: §1, §2.2.1, §4.3.5, Table 2. [71] M. Zhou, M. Huang, and X. Zhu (2018) An interpretable reasoning network for multi-relation question answering. In Proceedings of the 27th international conference on computational linguistics, p. 2010–2022. Cited by: §2.2.2, §4.2, §5.1, §5.2.