Paper deep dive
A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala, Naman Kabadi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/11/2026, 5:30:19 AM
Summary
This paper introduces a grounded and decomposed framework for evaluating relation-level hallucination in abstractive text summarization. The authors propose the Relation Hallucination Index (RHI), which utilizes a dependency-aware relation extraction algorithm to capture subject-verb-object triples with high structural fidelity. The framework includes normalization techniques to ensure scale-invariant comparison across datasets and models. Extensive evaluations on XSUM, XLSUM, and CNN/DailyMail datasets demonstrate that the proposed RHI provides more stable and discriminative measurements of relational faithfulness compared to existing metrics like ROUGE, EHI, and SummaC.
Entities (15)
Relation Signals (11)
Naman Kabadi → authored → A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
confidence 99% · 4th Naman Kabadi... A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Praveen Kumar Katwe → authored → A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
confidence 99% · 1st Praveen Kumar Katwe... A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Kali Prasad Vittala → authored → A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
confidence 99% · 3rd Kali Prasad Vittala... A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Rakesh Chandra Balabantaray → authored → A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
confidence 99% · 2nd Rakesh Chandra Balabantaray... A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
BART-large-cnn → evaluatedon → XLSUM
confidence 95% · The experimental dataset consists of ... XLSUM... We evaluate four representative transformer-based abstractive summarization models, namely BART-large-CNN
BART-large-cnn → evaluatedon → XSUM
confidence 95% · The experimental dataset consists of XSUM... We evaluate four representative transformer-based abstractive summarization models, namely BART-large-CNN
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis.
Tags
Links
- Source: https://arxiv.org/abs/2608.08180v1
- Canonical: https://arxiv.org/abs/2608.08180v1
Trouble viewing inline? Open PDF directly →
Full Text
31,942 characters extracted from source content.
Expand or collapse full text
A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization 1st Praveen Kumar Katwe 2nd Rakesh Chandra Balabantaray 3rd Kali Prasad Vittala 4th Naman Kabadi Abstract Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis. I Introduction Recent advances in natural text generation have significantly improved the fluency and readability of automatically generated summaries. However, fluency does not guarantee factual reliability. Abstractive summarization systems frequently introduce relational distortions, where entities are preserved but the connections between them are inaccurately constructed. Such relation-level hallucinations can subtly alter the meaning of source content, raising serious concerns in applications where correctness is critical. Although existing evaluation metrics emphasize lexical overlap or entity matching, they often fail to capture inconsistencies in structured relationships. Assessing relational faithfulness therefore requires explicit modeling of subject–verb–object structures and systematic comparison between source documents, references, and generated summaries. In this work, we present a grounded and normalized framework for evaluating relation-level hallucination. We refine relation extraction using linguistically informed mechanisms and introduce a normalized formulation of the Relation Hallucination Index (RHI) to enable stable comparison across models and datasets. Our approach provides a structured and interpretable mechanism for analyzing relational fidelity and coherence [20] in abstractive summarization systems. I Background and Problem Formulation I-A Relation-Level Hallucination Relation-level hallucination arises when a summarization system preserves surface entities from the source document, but incorrectly constructs or alters the relationships among them. Modern transformer-based models and large language models generate summaries by predicting tokens conditioned on contextual representations rather than explicitly modeling structured relational dependencies. As a result, these systems may inadvertently introduce fabricated interactions, misattribute actions, or exaggerate causal links between entities. Unlike simple factual omissions, relational distortions modify the semantic structure of the original content and may lead to misleading interpretations. Detecting such inconsistencies, therefore, requires a structured comparison of subject–predicate–object relations across source texts, references, and generated summaries. So, relation-aware extraction and evaluation frameworks are essential to quantify these structural deviations and assess summary faithfulness beyond lexical similarity. I-B Limitations of Existing Metrics Widely adopted evaluation measures such as ROUGE primarily rely on n-gram overlap, which captures surface similarity but does not reflect relational correctness. Embedding-based metrics, including BERT-derived similarity scores, assess semantic proximity yet lack explicit modeling of structured entity interactions. Entity-centric metrics improve entity coverage evaluation but remain insensitive to incorrect predicate assignments or distorted connections between entities. Consequently, existing approaches often fail to identify relation-level inconsistencies, motivating the need for structured and normalized relational evaluation mechanisms. I Related Work I-A Evaluation Metrics for Summarization Automatic evaluation of summarization systems has historically relied on lexical overlap measures, most prominently ROUGE [4]. These metrics quantify n-gram correspondence between system outputs and reference summaries and remain widely adopted for benchmarking informativeness. However, lexical similarity does not guarantee factual correctness, and high overlap scores may still accompany structurally distorted content. As neural generation models became more expressive, limitations of purely surface-based metrics became increasingly apparent, motivating research into faithfulness-oriented evaluation strategies. I-B Approaches to Mitigating Hallucination To improve the reliability of abstractive summaries, several modeling strategies have been proposed. Entity-aware decoding mechanisms incorporate named entity signals during generation to encourage preservation of salient source entities [15]. Knowledge-grounded frameworks further attempt to constrain generation by referencing structured repositories such as knowledge graphs [16]. Other approaches employ adversarial training, where a discriminator guides the generator toward producing outputs that are not only fluent but also factually aligned with source content [7]. While these methods enhance generation quality, they focus primarily on model training rather than post-hoc structural evaluation. I-C Entity-Level Faithfulness Metrics Beyond generation strategies, evaluation metrics specifically targeting hallucination have been introduced. Entity-level measures, such as Entity F1 [17], compare named entities across source and generated summaries to quantify factual preservation. Subsequent work expanded this perspective by categorizing hallucinations into subject, object, and relation dimensions [19]. Entity-centric indices, including the Entity Hallucination Index (EHI) [5], provide finer-grained insight into entity consistency. Nevertheless, entity-level comparison does not guarantee relational correctness. A summary may preserve correct entities while misassigning roles, altering predicates, or fabricating interactions between otherwise accurate mentions. I-D Structured and Knowledge-Based Evaluation Structured evaluation methods model textual content as relational triples or knowledge graph representations. Benchmarks such as Text2KG frameworks [18] evaluate extraction quality under predefined ontologies. While effective for structured prediction tasks, such approaches typically require domain-specific schemas or world knowledge alignment, limiting their applicability to general summarization evaluation. Moreover, many existing relation-based pipelines rely on basic extraction heuristics that are sensitive to syntactic variation and duplication artifacts, potentially affecting hallucination measurements. I-E Gap and Motivation Existing research demonstrates substantial progress in entity preservation and semantic similarity evaluation; however, systematic quantification of relation-level hallucination without reliance on external ontologies remains underexplored. In particular, prior methods lack linguistically grounded extraction refinements and normalized scoring mechanisms that ensure stable cross-dataset comparison. The present work addresses these gaps by integrating dependency-aware relation extraction, refined hallucination decomposition, normalized RHI computation, and coherence-aware relational continuity analysis within a unified evaluation framework. IV Methodology The proposed framework consists of four major components: dataset preparation, summary generation, relation extraction, and relation-level hallucination evaluation. The overall pipeline constructs structured relational representations from input, reference, and generated summaries, and evaluates relational faithfulness using a normalized Relation Hallucination Index. IV-A Dataset Selection and Preprocessing IV-A1 Dataset Selection To evaluate relation-level hallucination across varying abstraction styles, we employ three widely used summarization benchmarks covering complementary news domains. The experimental dataset consists of XSUM (200 instances), XLSUM (200 instances), and the CNN/DailyMail SumEval subset (400 instances), resulting in a total of 800 evaluation samples. XSUM is selected for its highly abstractive single-sentence summaries, which frequently induce factual distortions. XLSUM introduces stylistic and linguistic diversity through cross-domain news reporting, enabling robustness assessment under varied writing structures. The CNN/DailyMail SumEval dataset is incorporated due to its established use in factual consistency evaluation (e.g., SummaC), providing longer multi-sentence summaries with stronger extractive grounding. This combination enables balanced evaluation across extreme abstraction, moderate abstraction, and factual consistency-oriented summarization settings, improving generalizability of relation hallucination analysis. IV-A2 Dataset Preparation All documents undergo normalization, removal of noisy symbols, and sentence-level segmentation prior to processing. Inputs exceeding transformer token limits are truncated while preserving contextual coherence. The cleaned texts are subsequently passed to the relation extraction pipeline for Subject–Verb–Object (SVO) tuple generation, as illustrated in Fig. 1. IV-B Summary Generation IV-B1 Model Selection We evaluate four representative transformer-based abstractive summarization models, namely BART-large-CNN, PEGASUS, T5-large, and GPT-3.5, covering diverse pretraining objectives and abstraction behaviors. The human reference summary is additionally considered as a pseudo-model (RefSum) to establish an upper-bound benchmark for relational consistency. Figure 1: Process flow diagram illustrating dataset preparation, summary generation, relation extraction, and metric evaluation pipeline. IV-B2 Model Output Generation Each model generates zero-shot summaries for the selected dataset. The generated summaries, along with input documents and reference summaries, are processed to extract structured relation triples. These triples form the basis for computing relational overlap and hallucination factors. IV-C Relation Extraction Framework We propose a dependency-aware relation extraction framework designed to enhance structural robustness and reduce spurious tuple generation. Relations are represented as normalized subject–verb–object triples extracted using syntactic dependency parsing. The overall relation extraction is formalized in Algorithm 1. The verb-centric extraction procedure, detailed in Algorithm 2, identifies predicate-driven relational structures while incorporating multiple linguistic constraints. Nominal fallback extraction for prepositional noun patterns is described in Algorithm 3, and normalization together with subject grounding mechanisms are specified in Algorithm 4. The framework integrates several linguistic refinements to improve structural stability. Lemmatization-based normalization reduces morphological variation across predicates and arguments. To mitigate underspecified subject ambiguity, named entity grounded subject resolution replaces generic subjects with entity-aligned mentions when available. Passive constructions are addressed through agent recovery to restore implicit actors, while negation-aware verb modeling preserves relational polarity. Reporting verbs are filtered to exclude meta-discourse structures that do not contribute to factual content. Furthermore, nominal fallback extraction captures noun–preposition–object configurations, and clausal propagation enables relation recovery from embedded predicate structures. Finally, systematic deduplication prevents redundant triples from inflating hallucination measurements. These refinements collectively improve structural consistency of extracted relations and reduce artificial inflation of hallucination factors. The resulting triples serve as structured inputs for computing the refined evaluation metrics, including EF1 and the normalized Relation Hallucination Index (RHI), described in subsequent sections. Figure 2: Venn Diagram representing the six factors to identify the relation hallucination Algorithm 1 Grounded Relation Extraction Algorithm 1:function ExtractRelationsWithGrounding(texttext) 2: Relations←∅Relations← 3: text←CleanText(text)text← CleanText(text) 4: Chunks←SentenceChunks(text)Chunks← SentenceChunks(text) 5: for each chunk∈Chunkschunk∈ Chunks do 6: Parse chunkchunk using dependency parser 7: for each sentencesentence in chunkchunk do 8: Relations←Relations∪Relations← Relations∪ 9: ExtractVerbRelations(sentencesentence) 10: Relations←Relations∪Relations← Relations∪ 11: ExtractNominalRelations(sentencesentence) 12: end for 13: end for 14: Remove duplicate triples 15: return RelationsRelations 16:end function Algorithm 2 Verb-Based Relation Extraction 1:function ExtractVerbRelations(sentencesentence) 2: R←∅R← 3: for each tokentoken in sentencesentence do 4: if token.pos≠token.pos≠ VERB then 5: continue 6: end if 7: v←Lemma(token)v← Lemma(token) 8: if v∈ReportingVerbsv∈ ReportingVerbs then 9: continue 10: end if 11: if token has negation then 12: vfinal←v_final← “not_” ++ v 13: else 14: vfinal←v_final← v 15: end if 16: Identify SubjectsSubjects from nsubj,nsubjpass 17: Identify ObjectsObjects from dobj,obj,attr ⊳ Passive Agent Recovery 18: for each agent dependency do 19: Recover subject 20: end for 21: for each s∈Subjects∈ Subjects do 22: if s.pos∉NOUN,PROPNs.pos∉\NOUN,PROPN\ then 23: continue 24: end if 25: snorm←s_norm← GroundSubject(s,sentences,sentence) 26: if snorm=NULLs_norm=NULL then 27: continue 28: end if 29: for each o∈Objectso∈ Objects do 30: onorm←o_norm← Normalize(o) 31: if onorm≠NULLo_norm≠ NULL then 32: Add (snorm,vfinal,onorm)(s_norm,v_final,o_norm) to R 33: end if 34: end for⊳ Clausal Propagation 35: for each child c with dep ∈xcomp,ccomp∈\xcomp,ccomp\ do 36: Extract inner verb and object 37: Add propagated relation 38: end for⊳ Verb Conjunction Handling 39: for each conjunct verb do 40: Add conjunction relation 41: end for 42: end for 43: end for 44: return R 45:end function Algorithm 3 Nominal Relation Fallback 1:function ExtractNominalRelations(sentencesentence) 2: R←∅R← 3: for each noun n in sentencesentence do 4: for each child with dep = prep do 5: for each grandchild with dep = pobj do 6: s←s← Normalize(n) 7: v←Lemma(prep)v← Lemma(prep) 8: o←o← Normalize(pobjpobj) 9: if s≠NULLs≠ NULL and o≠NULLo≠ NULL then 10: Add (s,v,o)(s,v,o) to R 11: end if 12: end for 13: end for 14: end for 15: return R 16:end function Algorithm 4 Normalization and Subject Grounding 1:function Normalize(tokentoken) 2: if tokentoken is stopword or punctuation then 3: return NULL 4: end if 5: if token.pos∉NOUN,PROPN,VERB,ADJtoken.pos∉\NOUN,PROPN,VERB,ADJ\ then 6: return NULL 7: end if 8: return lowercase lemma of tokentoken 9:end function 10:function GroundSubject(token,sentencetoken,sentence) 11: if tokentoken not in generic subject list then 12: return Normalize(tokentoken) 13: end if 14: for each named entity in sentencesentence do 15: if entity.label∈PERSON,ORG,GPE,NORPentity.label∈\PERSON,ORG,GPE,NORP\ then 16: return lowercase entity text 17: end if 18: end for 19: return Normalize(tokentoken) 20:end function IV-D Evaluation Metrics To comprehensively assess factual and relational faithfulness, we employ lexical, semantic, and relation-aware evaluation metrics. Conventional measures including ROUGE [4] and SummaC [21] are used as comparative baselines, while relation-oriented metrics such as EF1 and the proposed Relation Hallucination Index (RHI) quantify structured factual consistency. The Entity Hallucination Index (EHI) [5] is additionally considered to contrast entity-level behavior with relation-level evaluation. IV-D1 Lexical and Consistency Metrics ROUGE evaluates lexical overlap between reference (R) and generated summary (S): ROUGE-N=∑gramn∈RCountmatch(gramn)∑gramn∈RCount(gramn)ROUGE-N= _gram_n∈ RCount_match(gram_n) _gram_n∈ RCount(gram_n) (1) Relational alignment is summarized using Extractive-Faithfulness F1 (EF1) [22]: EF1=2×Precision×RecallPrecision+RecallEF1= 2× Precision× RecallPrecision+Recall (2) Precision=|I∩G||G|,Recall=|R∩G||R|Precision= |I∩ G||G|, Recall= |R∩ G||R| SummaC measures document–summary consistency using entailment aggregation [21]: SummaC(D,S)=1|S|∑si∈Smaxdj∈DEntail(dj,si)SummaC(D,S)= 1|S| _s_i∈ S _d_j∈ DEntail(d_j,s_i) (3) Coherence is computed as the average semantic similarity between adjacent sentences using sentence-level embeddings: Coherence(S)=1N−1∑i=1N−1cos(i,i+1)Coherence(S)= 1N-1 _i=1^N-1 (e_i,e_i+1 ) (4) where S=s1,s2,…,sNS=\s_1,s_2,...,s_N\ denotes the generated summary consisting of N sentences, and ie_i represents the sentence embedding of sentence sis_i. Higher values indicate stronger semantic continuity between consecutive sentences [20]. IV-D2 Entity Hallucination Index Entity-level factual reliability is measured using EHI [5]: EHI=ePH+eEFePH+eEF+eNH+eOF+eLFEHI= e^PH+e^EFe^PH+e^EF+e^NH+e^OF+e^LF (5) IV-D3 Relation Hallucination Decomposition Let I, R, and G denote relation triples extracted from input, reference, and generated summaries. EF=3|I∩R∩G||I|+|R|+|G|EF= 3|I∩ R∩ G||I|+|R|+|G| (6) PH=2|R∩G||R|+|G|PH= 2|R∩ G||R|+|G| (7) OF=2(|I∩G|−|I∩R∩G|)|I|+|G|OF= 2(|I∩ G|-|I∩ R∩ G|)|I|+|G| (8) NH=||G|−(|R∩G|+|I∩G|−|I∩R∩G|)||G|NH= ||G|-(|R∩ G|+|I∩ G|-|I∩ R∩ G|) ||G| (9) LF=|R|−(|I∩R|−|I∩R∩G|)|R|+|G|LF= |R|-(|I∩ R|-|I∩ R∩ G|)|R|+|G| (10) LH=|I|−|I∩G||I|+|G|LH= |I|-|I∩ G||I|+|G| (11) IV-D4 Relation Hallucination Index The proposed RHI aggregates positive and hallucinated relational behaviors: RHI=1+EF+PH2−OF+NH+LH+LF4RHI=1+ EF+PH2- OF+NH+LH+LF4 (12) RHInorm=RHI−RHIminRHImax−RHIminRHI_norm= RHI-RHI_minRHI_max-RHI_min (13) V Results We evaluate relational faithfulness using EF1, EHI, ROUGE, SummaC, coherence, and the proposed Relation Hallucination Index (RHI). Results are reported separately for abstractive news datasets (XSUM+XLSUM) and the CNN/DailyMail SumEval benchmark to analyze cross-domain consistency. V-A Average Performance on XSUM + XLSUM TABLE I: Average Metrics on XSUM + XLSUM Metric BART PEGASUS T5 GPT3.5 RefSum EF1 0.1230 0.1412 0.1149 0.1633 0.2885 EHI 0.6112 0.6162 0.5274 0.5279 0.9580 RHI 0.7211 0.6961 0.7012 0.6830 0.9206 SummaC 0.4312 0.3553 0.2555 0.0553 0.7838 Coherence 0.2840 0.8747 0.2812 0.3533 0.9823 ROUGE-1 0.2140 0.4387 0.2162 0.2194 1.0000 Table I shows that lexical and entailment-based metrics favor PEGASUS due to stronger surface alignment and discourse coherence. However, RHI assigns the highest score to BART, indicating improved preservation of source relations despite moderate ROUGE values. This divergence highlights RHI’s ability to detect relational correctness beyond token overlap or semantic similarity. GPT-3.5 achieves higher EF1 but lower RHI, suggesting increased relation insertion variability typical of generative models under highly abstractive settings. V-B Performance on CNN/DailyMail SumEval TABLE I: Average Metrics on CNN/DailyMail SumEval Metric BART PEGASUS T5 DistilBERT RefSum EF1 0.1900 0.1600 0.1554 0.0000 0.7495 EHI 0.5712 0.5890 0.5454 0.5478 0.9495 RHI 0.7681 0.7470 0.7476 0.7188 0.9588 ROUGE-1 0.3511 0.3419 0.3035 0.0000 1.0000 SummaC 0.0906 0.0747 0.0453 0.1570 0.8460 RHInorm 0.7912 0.7556 0.6987 0.5677 0.9851 Across the SumEval benchmark (Table I), BART consistently achieves the highest RHI and EF1, demonstrating stable relational grounding on longer documents. DistilBART records near-zero ROUGE and EF1 scores, indicating failure in effective summary generation rather than metric bias. Despite moderate SummaC values, its reduced RHI confirms substantial loss of relational structure, validating the sensitivity of RHI to generation collapse scenarios. V-C Statistical Validation Statistical analysis across both the XSUM+XLSUM and CNN/DailyMail SumEval datasets confirms the discriminative strength of the proposed RHI metric. Paired t-test results show statistically significant performance differences among summarization models (p<0.01p<0.01), indicating that RHI consistently distinguishes summaries with well-preserved relations from those containing higher levels of hallucination. In contrast to ROUGE, EF1, EHI, and SummaC, which often yield comparable scores across models, RHI provides clearer separation and stable ranking across datasets, demonstrating its effectiveness for relation-level factual evaluation. V-D Baseline vs Updated RHI Framework TABLE I: Baseline vs Updated RHI (XSUM+XLSUM) Model Baseline Updated Δ BART 0.6378 0.7211 +0.0833 PEGASUS 0.6137 0.6961 +0.0824 T5 0.6240 0.7012 +0.0772 GPT-3.5 0.6156 0.6830 +0.0674 Tables I and IV show consistent RHI improvements across all models, where refinement contributes the major performance gain (Δref _ref) and normalization provides additional stabilization (Δnorm _norm). The cumulative improvements match the overall Δ observed in Table I, confirming the effectiveness of the proposed framework design. TABLE IV: Ablation Study of RHI Components (XSUM+XLSUM) Model RHIbase RHIrefined RHInorm Δref _ref Δnorm _norm BART 0.6378 0.7012 0.7211 +0.0634 +0.0199 PEGASUS 0.6137 0.6754 0.6961 +0.0617 +0.0207 T5 0.6240 0.6889 0.7012 +0.0649 +0.0123 GPT-3.5 0.6156 0.6627 0.6830 +0.0471 +0.0203 V-E Cross-Dataset Hallucination Behaviour TABLE V: Cross-Dataset Hallucination Factors and Normalized RHI XSUM + XLSUM BART PEGASUS T5 GPT-3.5 EF 0.0249 0.0270 0.0213 0.0294 PH 0.0635 0.1478 0.0569 0.0798 OF 0.2616 0.0707 0.2089 0.2125 NH 0.0873 0.3364 0.1711 0.4846 LF 0.2418 0.2903 0.2410 0.1682 LH 0.6907 0.8277 0.7272 0.6171 RHInorm 0.6912 0.6553 0.7055 0.6155 CNN / SumEval BART PEGASUS T5 DistilBART EF 0.0347 0.0261 0.0260 0.0000 PH 0.1256 0.1154 0.0957 0.0000 OF 0.1319 0.0886 0.1376 0.0000 NH 0.1266 0.1902 0.1790 0.0000 LF 0.1258 0.1217 0.1045 0.1249 LH 0.8267 0.8701 0.8177 1.0000 RHInorm 0.7912 0.7556 0.6987 0.5677 Across both datasets, hallucination decomposition reveals consistent relational trends supporting RHI evaluation. Models with balanced Extractiveness (EF) and Positive Hallucination (PH) while maintaining reduced Over-Focus (OF) achieve higher normalized RHI scores, indicating stable relation preservation. Improvements observed on the CNN/DailyMail benchmark demonstrate that the proposed framework remains robust under longer and less abstractive summaries. Notably, generation failure in DistilBART results in near-zero relational factors, leading to reduced RHInorm despite high coherence, confirming that RHI evaluates factual grounding rather than fluency. The consistent separation of models across datasets validates that aggregated hallucination factors collectively strengthen RHI’s sensitivity and enable reliable cross-domain hallucination assessment. V-F Distributional Analysis of RHI Figure 3: Cumulative Distribution of RHI across models for XLSum/XSum Dataset Figure 4: Cumulative Distribution of RHI across models for CNN/Daily Mail Dataset Figures 3 and 4 present the cumulative distribution of RHI scores across the abstractive (XSUM+XLSUM) and long-document (CNN/DailyMail) evaluation settings. In both datasets, BART and PEGASUS exhibit smoother right-shifted distributions, indicating consistent preservation of source relations across a large proportion of summaries. T5 shows moderate dispersion, reflecting variable relational grounding under different contextual conditions. In contrast, DistilBART demonstrates an earlier saturation and compressed distribution in Fig. 4, revealing reduced relational coverage and confirming performance degradation observed in quantitative evaluation. The consistent ordering of model curves across Fig. 3 and Fig. 4 further indicates that RHI maintains stable discriminative behaviour independent of dataset abstraction level or document length. V-G Discussion The experimental evaluation demonstrates that analysing summaries through relational structure provides insights that are not reflected by overlap- or entailment-oriented metrics. Across both evaluation settings, models exhibiting comparable lexical quality show noticeable differences when assessed using relation consistency, indicating that factual reliability depends strongly on preserved entity–event interactions rather than surface similarity. The grounded extraction strategy enables stable identification of meaningful relations, reducing sensitivity to stylistic variation across datasets. Furthermore, normalized RHI scores maintain consistent model ordering under both highly abstractive and long-document summarization scenarios. The observed distributional separation confirms that relation decomposition effectively exposes hallucination patterns arising from omission, distortion, or unsupported relation generation. These findings suggest that relation-aware evaluation offers a practical mechanism for examining factual behaviour at structural granularity while remaining adaptable across model architectures and dataset characteristics. VI Conclusion and Future Work This paper introduced a grounded framework for assessing relation-level hallucination in abstractive summarization through the proposed Relation Hallucination Index (RHI). Experimental results across diverse datasets demonstrate that the proposed formulation provides stable and discriminative evaluation of factual consistency, enabling clearer differentiation between relation-preserving and hallucination-prone summarization systems. The study shows that modelling factuality at the level of entity–event interactions offers improved diagnostic capability compared to traditional evaluation measures focused primarily on lexical or semantic similarity. Future research will investigate extending the framework toward document-level reasoning by incorporating cross-sentence relation modeling and richer contextual representations. Another promising direction involves leveraging RHI-driven feedback during model optimization to encourage generation mechanisms that prioritize factual grounding alongside linguistic fluency. References [1] P. Katwe, R. C. Balabantaray, and K. P. Vittala, “Evaluating Relation Hallucination in Text Summarization: An Introduction to the Relation Hallucination Index,” in Proc. 16th Annu. Meeting Forum Inf. Retrieval Eval. (FIRE), 2024, p. 88–94, doi: 10.1145/3734947.3734958. [2] Z. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Comput. Surveys, vol. 55, no. 12, p. 1–38, 2023, doi: 10.1145/3571730. [3] S. Qi, L. Gui, Y. He, and Z. Yuan, “A Survey of Automatic Hallucination Evaluation on Natural Language Generation,” arXiv preprint arXiv:2404.12041, 2025. [4] C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” in Text Summarization Branches Out, Barcelona, Spain, 2004, p. 74–81. [5] K. Praveenkumar, R. C. Balabantaray, and K. P. Vittala, “Entity Hallucination Index in Abstractive Summarization—A Metric,” in Proc. 2023 Int. Conf. Commun., Circuits, and Systems (IC3S), 2023, p. 1–5. [6] P. Qi et al., “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” arXiv preprint arXiv:2003.07082, 2020. [7] W. Wu et al., “Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation,” arXiv preprint arXiv:2210.12367, 2022. [8] M. Lewis et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” in Proc. ACL, 2020. [9] J. Zhang et al., “PEGASUS: Pre-training with Extracted Gap-Sentences for Abstractive Summarization,” in Proc. ICML, 2020. [10] C. Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” J. Mach. Learn. Res., vol. 21, no. 140, p. 1–67, 2020. [11] T. B. Brown et al., “Language Models are Few-Shot Learners,” in Proc. NeurIPS, 2020. [12] M.-C. de Marneffe and C. D. Manning, “The Stanford Typed Dependencies Representation,” in Proc. Cross-Framework and Cross-Domain Parser Evaluation, 2008, p. 1–8. [13] D. Zeng, K. Liu, S. Lai, G. Zhou, and J. Zhao, “Relation Classification via Convolutional Deep Neural Network,” in Proc. COLING, 2014. [14] L. He, K. Lee, M. Lewis, and L. Zettlemoyer, “Deep Semantic Role Labeling: What Works and What’s Next,” in Proc. ACL, 2017. [15] H. Zhou, W. Ren, G. Liu, B. Su, and W. Lu, “Entity-Aware Abstractive Multi-Document Summarization,” in Findings of the Association for Computational Linguistics: ACL, 2021, p. 351–362, doi: 10.18653/v1/2021.findings-acl.30. [16] W. Chen, Y. Su, X. Yan, and W. Y. Wang, “KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation,” arXiv preprint arXiv:2010.02307, 2020. [17] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural Architectures for Named Entity Recognition,” in Proc. NAACL-HLT, San Diego, CA, USA, 2016, p. 260–270, doi: 10.18653/v1/N16-1030. [18] N. Mihindukulasooriya, S. Tiwari, C. F. Enguix, and K. Lata, “Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text,” arXiv preprint arXiv:2308.02357, 2023. [19] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, “On Faithfulness and Factuality in Abstractive Summarization,” in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics (ACL), 2020, p. 1906–1919. [20] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proc. EMNLP, 2019. [21] P. Laban, T. Schnabel, P. N. Bennett, and M. A. Hearst, “SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization,” arXiv preprint arXiv:2111.09525, 2021. [22] M. Olek, “About Evaluation of F1 Score for RECENT Relation Extraction System,” arXiv preprint arXiv:2305.09410, 2023.