Paper deep dive
Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation
Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/14/2026, 5:52:23 AM
Summary
This paper introduces Fact-Ablated Evaluation (FAE), a framework to assess whether Large Language Models (LLMs) rely on provided evidence or parametric knowledge for fact-checking. The authors find that current LLMs often maintain predictions even when evidence is removed. To address this, they propose REAL (Rigorous Evidence Ablation Learning), a training framework using counterfactual supervision to improve evidence grounding.
Entities (7)
Relation Signals (5)
Llama-3.1-8B-Instruct → evaluatedon → FEVER
confidence 95% · Ideal vs actual ablation trajectories for claims annotated as SUPPORT/REFUTE (Llama-3.1-8B-Instruct on FEVER)
REAL → uses → Counterfactual Supervision
confidence 95% · REAL introduces counterfactual supervision by training on paired inputs with and without supporting evidence
REAL → improves → Evidence Grounding
confidence 92% · REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence
Fact-Ablated Evaluation → measuresdependencyof → Parametric Knowledge
confidence 90% · FAE evaluates LLM-based verifier grounding behaviour... indicating persistent reliance on parametric knowledge
LLM → exhibitsfailuremodein → Fact-Ablated Evaluation
confidence 88% · current LLM-based verifiers often maintain their original predictions even after the supporting evidence has been ablated
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.
Tags
Links
- Source: https://arxiv.org/abs/2609.08943v1
- Canonical: https://arxiv.org/abs/2609.08943v1
Trouble viewing inline? Open PDF directly →
Full Text
71,804 characters extracted from source content.
Expand or collapse full text
Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation Conference: Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 07–11, 2026; Rome, ItalyProceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, ItalyDOI: 10.1145/3799682.3841076ISBN: 979-8-4007-2539-5/2026/11CCS: Computing methodologies Natural language processing Xingyu Deng Affiliation: University of Sheffield, Sheffield, UK email: xdeng37@sheffield.ac.uk , Mingzi Cao Affiliation: University of Sheffield, Sheffield, UK email: mcao20@sheffield.ac.uk , Nikolaos Aletras Affiliation: University of Sheffield, Sheffield, UK email: n.aletras@sheffield.ac.uk , Xi Wang Affiliation: University of Sheffield, Sheffield, UK email: xi.wang@sheffield.ac.uk and Mark Stevenson Affiliation: University of Sheffield, Sheffield, UK email: mark.stevenson@sheffield.ac.uk © c Abstract. Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence. Keywords: Fact-Checking, Retrieval-Augmented Generation †c-license: by 1. Introduction Automated fact checking, the process of assessing the veracity of claims, acts as a safeguard against misinformation (Guo et al., 2022b; Zeng et al., 2021). Fact-checking systems are expected not only to predict claim veracity labels, but also to retrieve and present supporting evidence from external knowledge sources such as textual documents and knowledge graphs (Guo et al., 2022a; Zeng et al., 2021). This evidence helps to explain algorithm decisions, which is particularly important in high-stakes domains, like public health or law, and supports the integration of fact-checking systems into human decision-making workflows (Vladika and Matthes, 2023). The traditional approach to fact-checking involves a two-stage approach: (1) identification of evidence from a collection ranked by retrievers (Chen et al., 2022; Wu et al., 2021; Hanselowski et al., 2019; Hu et al., 2023; Zheng et al., 2024; Deng et al., 2025a; Luken et al., 2018; Nie et al., 2019) followed by (2) veracity prediction (Vlachos and Riedel, 2014; Guo et al., 2022a; Zeng et al., 2021). More recently, Large Language Models (LLMs) have demonstrated strong fact-checking performance through application of Retrieval-Augmented Generation (RAG) architectures (Lewis et al., 2020) in which retrieved documents are used as additional context prior to veracity prediction (Schlichtkrull et al., 2024; Akhtar et al., 2025; Pan et al., 2023; Lewis et al., 2020). However, in RAG settings, LLMs can make use of both retrieved evidence and memorised knowledge when assessing claims, making it difficult to determine the source of their predictions. Since parametric knowledge can become outdated (ChenghaoZhu et al., 2025) or reflect inaccuracies introduced during training (Das et al., 2025), reliance on such knowledge may affect the reliability of fact-checking decisions (Vladika et al., 2025; Kassem et al., 2025; Deng et al., 2025b; Vladika and Matthes, 2024; Deng, 2026). Strong performance may partly reflect prior exposure to relevant facts included in pretraining data, making it difficult to determine whether predictions are primarily supported by retrieved evidence or memorised knowledge (Wei et al., 2025; Di Palma et al., 2025; Carlini et al., 2021). Consequently, superior verification accuracy alone cannot reveal whether a verifier model truly depends on the provided evidence or merely recovers memorised knowledge. Figure 1. Observed failure mode in FEVER dataset produced by LLM verifier (LLaMa3-8B-Instruct) The figure about some failure examples observed in FEVER verified by Llama3-8B To ensure evidence-based fact-checking, verification decisions are expected to be grounded in the evidence cited as their justification. This expectation demands that the decision should be sensitive to changes in the cited evidence, especially when that evidence is no longer available. As shown in Figure 1, if a claim is supported by deterministic evidence (i.e., Sentence [1]), the verifier should no longer be able to confidently justify the same prediction once that evidence is removed. In practice, however, LLM-based verifiers often maintain the same decision by “hallucinating” alternative justifications (e.g., citing Sentence [7]) or without providing evidence. Standard fact-checking metrics (Guo et al., 2022a; Zeng et al., 2021; Vladika and Matthes, 2023; Vlachos and Riedel, 2014) fail to capture this behaviour, as they assess label accuracy only once without testing the causal dependence between evidence and prediction. Therefore, evidence grounding requires counterfactual evaluation by observing whether the decision changes when its supporting evidence is ablated. To address this limitation, we introduce Fact Ablated Evaluation (FAE), a behavioural evaluation framework that measures the causal dependency between predictions and model-selected evidence through iterative counterfactual intervention. FAE evaluates LLM-based verifier grounding behaviour through three complementary metrics that quantify decision shifts across the entire ablation process. For claims initially labelled SUPPORT or REFUTE, an evidence-grounded verifier should immediately abstain (i.e., return NOT_ENOUGH_INFO) once the supporting evidence is ablated. However, we observe that current LLM-based verifiers stabilise at a non-zero decision plateau across ablations, indicating persistent reliance on parametric knowledge despite explicit control of evidence completeness. These observations suggest that current LLM-based fact-checking systems, despite achieving strong veracity prediction accuracy, do not necessarily learn to make predictions that remain dependent on the provided evidence. To promote truly evidence-grounded fact-checking, we further propose REAL (Rigorous Evidence Ablation Learning), a training framework for LLM-based verifiers. Rather than relying on off-the-shelf LLMs (Rothermel et al., 2024; Braun et al., 2025; Ullrich et al., 2024; Park et al., 2024; Jannah et al., 2025; Li et al., 2025; Wang et al., 2024; Wei et al., 2024; Geng et al., 2025; Yoon et al., 2024; Mohammadkhani et al., 2024; Zhou et al., 2025; Xie et al., 2025; Yoon et al., 2025; Ullrich and Drchal, 2025; Chowdhury et al., 2025) and standard supervised fine-tuning (SFT) on claim-evidence pairs only (Shcharbakova et al., 2025; Cheung and Lam, 2023; Kumar et al., 2025; Putta et al., 2025; Yoon et al., 2024), REAL introduces counterfactual supervision by training on paired inputs with and without supporting evidence. It enforces abstention when evidence is ablated, thus creating a causal dependency between evidence and verdict. This paper makes three main contributions: • We identify a failure mode in LLM-based fact-checking, where predictions remain stable when removing supporting evidence. • We propose Fact Ablated Evaluation (FAE), a behavioural evaluation framework that measures evidence dependence through progressive evidence ablation, and show that parametric knowledge contributes to this failure mode. • We introduce REAL, a training framework that uses counterfactual supervision and evidence enhancement to improve evidence grounding without sacrificing fact-checking performance. Figure 2. Illustration of proposed evaluating framework FAE (Top) and training framework REAL (Bottom)The figure about the proposed FAE and REAL frameworks 2. Related Work The tendency of LLMs to memorise data observed during training is now well-established (Kiyomaru et al., 2024; Wei et al., 2025; Di Palma et al., 2025; Carlini et al., 2021). Consequently, the adoption of RAG for such applications has necessitated extensive research into LLM grounding and refusal mechanisms (Gao et al., 2023). This work can be broadly categorised into three directions. First, regarding robustness to irrelevant or insufficient context, several studies investigate how LLMs manage situations where retrieved documents are non-relevant or incomplete (Yang et al., 2025; Thakur et al., 2024; Joren et al., 2025; Niu et al., 2024). They emphasise the model’s ability to “know when it doesn’t know” and abstain from answering to prevent hallucinations. Second, in terms of evaluation frameworks, new metrics and benchmarks have been proposed to diagnose retrieval and generation errors (Ru et al., 2024; Song et al., 2025), with a focus on measuring semantic consistency between answers and citations. Third, for methodological improvements, various adaptation and alignment strategies are developed to enhance grounding (Ye et al., 2024; Hsu et al., 2024; Huang et al., 2024). For instance, Song et al. (Song et al., 2025) focus on aligning LLMs to abstain from answering under insufficient evidence, whereas Huang et al. (Huang et al., 2024) enhance verifiability by training models to ground claims in specific fine-grained textual evidence rather than coarse document identifiers. While these works share our motivation to reduce reliance on internal parametric knowledge, they primarily operate in generation-centric settings, such as question-answering. In such contexts, grounding is typically defined as the semantic alignment between a free-form response and its cited sources. Consequently, while some evaluation frameworks (Tang et al., 2024; Honovich et al., 2022) utilise fact-checking related principles, they still tend to simplify the task by treating REFUTE and NOT_ENOUGH_INFO into a single NON-SUPPORT category. In contrast, our work focuses on the task of fact-checking itself, which adopts a stricter formulation than generation-centric grounding by requiring granular labels and explicit causal dependency on evidence. Our work also differs from Akhtar et al. (Akhtar et al., 2024) who removed gold evidence to validate an evaluation metric, but this static approach cannot diagnose whether a model actually uses that evidence for its decision. Instead, we introduce a dynamic ablation process that ablates the model’s own predicted evidence. This allows us to characterise the causal dependency between the evidence a model selects and its final verdict, revealing when a model relies on parametric knowledge instead. 3. Background 3.1. Problem Statement Given a set of sentences (S), a fact-checking system (FCFC) assess a claim (c) and outputs a veracity label (l∈ℒl ) together with the supporting evidence sentences (E⊆SE S): (1) FC(c,S)→(l,E).FC(c,S)→(l,E). FCFC performs verification by maximising the joint probability of the target veracity label and the evidence subset: (2) (l,E) (l,E) =argmaxl∈ℒ,E⊆SP(l,E∣c,S) = l ,\,E S \;P(l,E c,S) ℒ=,,__/L=\ SUPPORT, REFUTE, NOT\_ENOUGH\_INFO/NEI\ is a commonly used a set veracity labels (Guo et al., 2022a; Zeng et al., 2021; Vladika and Matthes, 2023). 3.2. Fact-Checking System Evaluation Existing evaluation protocols for fact-checking systems primarily assess two independent aspects: (1) whether a model predicts the correct veracity label supported by identified evidence, and (2) whether the predicted evidence set covers the gold evidence. (1) Veracity prediction is evaluated at the claim level, typically via two metrics (Guo et al., 2022a; Zeng et al., 2021; Wadden et al., 2020; Wang et al., 2023; Thorne et al., 2018; Schlichtkrull et al., 2023): (i) Label Accuracy, which measures the correctness of the predicted label l against the gold label l∗l^*. (i) Label-Evidence Joint (Strict) Accuracy, which only credits a prediction if the veracity label is correct and at least one gold evidence sentence is correctly identified. (2) Evidence selection is usually evaluated by comparing the predicted evidence set E against the annotated gold evidence E∗E^* from a global perspective (Guo et al., 2022a; Zeng et al., 2021; Wadden et al., 2020; Wang et al., 2023; Thorne et al., 2018). Three standard retrieval metrics are reported: Precision, Recall, and F_1 score. These metrics are calculated by measuring the overlap between the set of all predicted evidence sentences and the set of all gold evidence. 4. FAE: Fact Ablated Evaluation 4.1. Evidence Ablation Process To verify whether the model’s predictions are strictly grounded in the provided evidence, we introduce Fact Ablated Evaluation (FAE), which evaluates model behaviour under varying levels of evidence availability. Figure 2 (top) presents an overview of the FAE evaluation framework with an illustrative comparison between actual observations and the ideal performance of a fact-checking system. Given a claim c and an evidence set S, FCFC produces a prediction (l(0),E(0))(l^(0),E^(0)), referred to as the round 00 predication. The evidence set is then ablated by removing the predicted evidence E(0)E^(0): S1=S∖E(0).S^1=S E^(0). FCFC then predicts the same claim c again using S1S^1 to produce the round 1 prediction and then FAE updates the evidence set for round 22: (l(1),E(1)),S2=S1∖E(1).(l^(1),E^(1)),\ \ S^2=S^1 E^(1). This ablation process is repeated iteratively, dropping the predicted evidence E(r)E^(r) following each round r and re-evaluating FCFC with evidence set Sr∖E(r)S^r E^(r) in the next round. The ablation process is terminated after a fixed number of rounds R. The resulting sequence l(0),l(1),l(2),…l(r)l^(0),\,l^(1),\,l^(2),\,…\ l^(r) captures how FCFC behaviour changes as cited evidence is progressively eliminated. 4.2. Ideal vs. Actual Trajectories The figure about Ideal vs. Actual Observation in FAE evaluation framework Figure 3. Ideal vs. actual ablation trajectories for claims annotated as SUPPORT/REFUTE (Llama-3.1-8B-Instruct on FEVER).The figure about Ideal vs. Actual Observation in FAE evaluation framework After an initial SUPPORT or REFUTE prediction at round r=0r=0, an ideal FCFC should switch to NOT_ENOUGH_INFO once the supporting evidence has been ablated. However, experimental observations illustrated in Figure 2 demonstrate that current LLM-based verifiers fail to exhibit this ideal behaviour, often maintaining non-NEI predictions across subsequent ablation rounds despite the unavailability of supporting evidence. Figure 3 further contrasts these ideal and observed trajectories over a sequence of ablation rounds. This observation suggests that the decisions of current LLM-based FCFC do not rely solely on the provided evidence. Hence, we propose the following hypothesis: ℋaH_a: Parametric knowledge acquired during training in LLMs interferes with the fidelity of strictly evidence-grounded verification. To test this hypothesis, we define two fact-checking paradigms: evidence-based and knowledge-based. Evidence-based paradigm aims to verify claims using only the provided evidence, while the knowledge-based paradigm aims to omit the evidence and prompt the LLM to rely completely on its parametric knowledge. Following prior work (Tang et al., 2025), we record neuron activations on a calibration set. We rank each neuron by activation magnitude and select those that cumulatively account for 90%90\% of the total signal as highly associated neurons. This procedure is applied separately to the two paradigms to obtain paradigm-specific neuron sets, which are visualised in Figure 4. We observe that there is substantial overlap between the two paradigms (73.3%73.3\%), indicating shared fact-checking behaviour. However, the two paradigms also exhibit distinct paradigm-specific neuron subsets. This suggests that evidence-based and knowledge-based fact-checking are not identical at the neuronal level. Since the knowledge-based paradigm removes the evidence list, its specific neurons reflect the use of knowledge acquired during training. This supports ℋaH_a by showing that LLM verifiers retain a neuron-level signal associated with parametric knowledge, which may compete with the provided evidence when strictly evidence-grounded verification is required. The Venn graph about activated neuron sets in fact checking using internal knowledge and provided evidence. Figure 4. Comparison of activated neuron sets in fact checking using internal knowledge and provided evidence.The Venn graph about activated neuron sets in fact checking using internal knowledge and provided evidence. 4.3. FAE Metrics To systematically evaluate behaviour under evidence ablation, we first compute accuracy at each ablation round, normalised by the accuracy obtained before any evidence ablation. (3) gr=ara0g_r= a_ra_0 where ara_r denotes the FCFC’s accuracy at ablation round r. The resulting sequence g=g0,g1,…,gR\g\=\g_0,g_1,…,g_R\ describes how verification accuracy is retained as supporting evidence is progressively ablated. Based on g\g\, we further propose three evidence-grounded metrics. ISIS, ERER, and IOIO respectively capture the immediate response to evidence ablation, the persistence of non-NEI predictions at the final ablation round, and the cumulative deviation from ideal evidence-grounded behaviour. Immediate Sensitivity (ISIS) measures the immediate response of a verifier’s predictions to evidence ablation: (4) IS=1−g1IS=1-g_1 A high ISIS indicates an immediate collapse after the first ablation step, suggesting that the initial prediction depends directly on the cited evidence. Conversely, a low ISIS indicates that the prediction remains unchanged after initial evidence ablation, reflecting either incorrect evidence selection or limited reliance on the cited evidence. End-State Retention (ERER) measures behaviour at the final round: (5) ER=1−gRER=1-g_R A low ERER indicates persistent non-NEI predictions despite complete evidence ablation, a behaviour often associated with spurious or missing evidence, suggesting that the decision is driven by parametric knowledge rather than evidence dependence. Ideal Offset (IOIO) measures the cumulative deviation from ideal evidence-grounded behaviour across the ablation process. Unlike ISIS and ERER, which capture only the initial and final ablation rounds, IOIO reflects behaviour dynamics across all ablation rounds. (6) IO=1−1R∑r=1Rar.IO=1- 1R _r=1^Ra_r. This metric penalises FCFC that remain confident as evidence availability degrades across intermediate ablation rounds and rewards a rapid, sustained performance drop following evidence ablation. We compute IOIO using ara_r instead of grg_r, without relying on a0a_0. 5. REAL Using FAE, we observe that current LLM-based verifiers often maintain their original predictions even after the supporting evidence has been ablated, as illustrated in Figure 3. This behaviour suggests that verification decisions are not fully grounded in the retrieved evidence, but are partially sustained through parametric knowledge acquired during pre-training. Such behaviour undermines evidence-dependent verification, since the final prediction can remain stable despite changes in the supporting context. To address this limitation, we propose Rigorous Evidence Ablation Learning (REAL), a training framework designed to strengthen the dependency between verification decisions and evidence availability, shown in Figure 2. Unlike standard supervised fine-tuning (SFT), which optimises prediction correctness only under evidence-complete conditions, REAL additionally supervises the verifier’s behaviour under evidence-ablated conditions where the supporting evidence has been intentionally removed. The key intuition is that a verifier should not only learn what prediction to make when sufficient evidence is available, but also when not to maintain the same prediction once the supporting evidence becomes unavailable. 5.1. Evidence Ablation Previous work on LLM-based fact-checking typically fine-tunes models using pairs consisting only of a claim and its supporting evidence (Shcharbakova et al., 2025; Cheung and Lam, 2023; Kumar et al., 2025; Putta et al., 2025; Yoon et al., 2024). Under this paradigm, the verifier is exposed exclusively to evidence-complete training conditions, where the required evidence is always available together with the correct label. Consequently, the model is never explicitly trained to perform evidence-grounded verification when supporting evidence becomes unavailable. As a result, the verifier is not trained to change its prediction when supporting evidence is removed. During training, maintaining the original prediction after evidence ablation is never penalised as long as the prediction remains correct under the original evidence-complete input. Consequently, the verifier may learn to preserve predictions through parametric knowledge instead of adapting its behaviour according to evidence availability. To address this issue, REAL represents each training instance as a triplet (c,S,l)(c,S,l), where c is a claim, S is the sentence set, and E⊆SE S denotes the gold evidence supporting the gold label l∗l^*. For each training instance, REAL constructs two contrasting supervision conditions: • Positive Input: Full context S, with target output (l,E)(l,E). • Negative Input: Ablated context Sabl=S∖E∗S_abl=S E^*, with target output (NEI,∅)( NEI, ), as shown in the bottom-right of Figure 2. For example, as illustrated in Figure 1, if the claim “Queen is a British rock band formed in London” is supported by ’Sentence [1]’, the ablated input removes this supporting sentence while preserving the remaining document context. REAL then supervises the verifier to abstain rather than maintain the original SUPPORT prediction through unrelated or hallucinated evidence. By jointly supervising these two contrasting conditions, REAL converts evidence availability into a counterfactual supervision signal. The verifier is therefore required not only to produce correct predictions under sufficient evidence, but also to revise its behaviour once the supporting evidence becomes unavailable. Formally, REAL optimises the following objective: (7) ℒREAL=−logPθ(l∗,E∗∣c,S)−logPθ(NEI,∅∣c,Sabl)L_REAL=- P_θ(l^*,E^* c,S)- P_θ( NEI, c,S_abl) where the first term supervises standard evidence-grounded verification under complete evidence conditions, and the second term penalises prediction persistence after the supporting evidence is removed. This paired objective encourages the verifier to produce a veracity prediction only when sufficient evidence is available, and to abstain when the evidence required for that prediction is no longer present. 5.2. Evidence Enhancement The construction of evidence-ablated inputs assumes that the gold evidence set exhaustively captures all sentences in S that support or refute the claim. However, evidence annotations in fact-checking benchmarks, including FEVER (Thorne et al., 2018), are typically designed for sufficiency rather than exhaustiveness (Derczynski et al., 2020; Bekoulis et al., 2021). In practice, multiple valid evidence sentences may coexist within the same document even though only a subset is annotated. This incompleteness introduces a critical challenge for evidence-ablated supervision. Positive samples may omit valid supporting evidence, while negative samples constructed through evidence ablation may still contain unannotated evidence capable of supporting the original claim. Consequently, logically valid verifier predictions may be incorrectly penalised during training, weakening the reliability of the counterfactual supervision signal. To mitigate this issue, REAL augments the original evidence annotations through cross-model evidence verification. Specifically, we use GPT-4o-mini (Hurst et al., 2024), Gemini-2.5-Flash-Lite (Comanici et al., 2025), and Qwen-2.5-32B-Instruct (Qwen et al., 2025) to independently identify supporting evidence sentences from the sentence pool S for each claim. Each model independently identifies supporting evidence, and we retain sentences supported by the majority of models to construct the augmented evidence set. By aggregating evidence predictions across multiple LLMs, this process reduces the likelihood of leaving valid supporting evidence inside ablated contexts. This helps mitigate incorrect counterfactual supervision caused by incomplete evidence annotations during REAL training. 6. Experiments 6.1. Models Baselines. A wide range of LLMs have been adopted as verifiers for verdict prediction in fact-checking (e.g. GPT∗ (Rothermel et al., 2024; Braun et al., 2025; Ullrich et al., 2024; Park et al., 2024; Jannah et al., 2025; Li et al., 2025; Wang et al., 2024; Xie et al., 2025; Wei et al., 2024; Geng et al., 2025), Llama∗ (Yoon et al., 2024; Mohammadkhani et al., 2024; Zhou et al., 2025; Xie et al., 2025; Geng et al., 2025), Qwen∗ (Yoon et al., 2025; Ullrich and Drchal, 2025; Zhou et al., 2025; Geng et al., 2025) and Gemini∗ (Jannah et al., 2025; Chowdhury et al., 2025; Wei et al., 2024; Geng et al., 2025)), leveraging their capability to reason over claims and supporting evidence for veracity prediction. However, these studies mainly employ the models in a zero-shot or few-shot manner through prompting, without task-specific fine-tuning to ensure evidence-grounded consistency. Hence, we evaluate REAL against two categories of baselines to demonstrate its effectiveness in veracity prediction and grounding consistency: Off-the-shelf LLMs (§7.1, §7.3): We include proprietary LLMs (GPT-4o-mini (Hurst et al., 2024), Gemini-2.5-Flash-lite (Comanici et al., 2025)) and open-source models (Qwen-2.5-32B/7B-Instruct (Qwen et al., 2025), Llama-3.1-8B-Instruct(Grattafiori et al., 2024)) to evaluate fact-checking behaviour without task-specific fine-tuning (§7.1, §7.2 ). Fine-tuned LLMs (§7.3): We also compare against the standard supervised fine-tuning following a few prior studies which applied this approach to fact-checking (Shcharbakova et al., 2025; Cheung and Lam, 2023; Kumar et al., 2025; Putta et al., 2025; Yoon et al., 2024) by directly leveraging supervision data provided by claim and gold evidence pairs only (Table 2, §7.3 ). Base LLMs for REAL. Following prior work (Shcharbakova et al., 2025; Cheung and Lam, 2023; Kumar et al., 2025; Putta et al., 2025; Yoon et al., 2024), we adopt Llama-3.1-8B-Instruct (Grattafiori et al., 2024) as the primary backbone of REAL. To demonstrate framework generality, we also evaluate REAL under Qwen-2.5-7B-Instruct (Qwen et al., 2025) in an ablation study (Table 3, §7.3 ). 6.2. Hyperparameter Details Models are fine-tuned using LoRA (Hu et al., 2022) with rank r=16r=16, scaling factor α=32α=32, and dropout 0.050.05, applied to the attention projection. Training is performed using AdamW (Loshchilov and Hutter, 2019) with a learning rate of 1×10−41× 10^-4, and a global batch size of 32. The maximum input sequence length is set to 4096, which covers total length of all pairs for the used datasets. Models are fine-tuned using a supervised assistant-only objective (Ouyang et al., 2022), to generate structured outputs consisting of a veracity label and evidence sentence indices. Models are trained for two epochs, with a fixed random seed across all experiments. Code for data preparation and experiments is available in github11 1 https://github.com/xingyu-deng/FAE_REAL. 6.3. Datasets This work includes four datasets for in-domain (FEVER) and out-of-domain evaluation (SciFact, Climate-FEVER and Check-COVID): • FEVER (Thorne et al., 2018) consists of 185,445 human-written claims verified against Wikipedia articles, with sentence-level annotations. • SciFact (Wadden et al., 2020) contains 1,409 scientific claims verified against research paper abstracts, with domain expert evidence annotations. • Climate-FEVER (Diggelmann et al., 2020) is a domain-specific benchmark focused on climate-related claims, verified against Wikipedia. • Check-COVID (Wang et al., 2023) consists of COVID-19 related claims verified against scientific and medical sources, representing a challenging real-world verification setting. Claims labelled as NEI require an external retrieval process to obtain documents, which introduces unnecessary variables (e.g., retriever quality and label noise) that are outside the scope of this study, and are therefore excluded due to their limited relevance to the core research focus of this work. Training. We train LLMs under proposed REAL framework on the train split of FEVER (Thorne et al., 2018) only, as it provides sufficient scale for generalisable evidence-grounding behaviour. Evaluation. We evaluate LLM-based verifiers under two settings. For in-domain evaluation, we test on the FEVER test split. To assess generalisation ability, we conduct out-of-domain evaluation on SciFact (Wadden et al., 2020), Climate-FEVER (Diggelmann et al., 2020), and Check-COVID (Wang et al., 2023) without additional training, using all available splits. For SciFact, we exclude the test split, as it is blind for the shared task (Wadden and Lo, 2021). 6.4. Evaluation Metrics Standard fact-checking metrics. Following standard protocols, as defined in §3.2, we evaluate the initial prediction using Label Accuracy and Strict Accuracy for veracity prediction, and Precision, Recall, and F1F_1 Score for evidence selection. Fact Ablated Evaluation metrics. We assess evidence dependence using the FAE metrics (IS, ER and IO) described in §4.3 over four ablation rounds following the initial prediction (R=4R=4 in Equation 3). We set R=4R=4 based on our empirical observations that the verifier’s accuracy converges to a stable plateau by this stage. Beyond this point, continued ablation of the residual context adds minimal information regarding the model’s evidence dependence. Instead, the verifier often fails by hallucinating non-existing evidence identifiers (the third failure mode in Figure 1). The figure about the experiment results in FEVER, SciFact, Climate-FEVER and Check-COVID. Figure 5. Ablation trajectories in FEVER, SciFact, Climate-FEVER and Check-COVID. The figure about the experiment results in FEVER, SciFact, Climate-FEVER and Check-COVID. Table 1. Fact-Checking performance (left) and FAE metrics (right) in four datasets Model/Metrics Fact-Checking Metrics (Initial) FAE Metrics Evidence Selection Veracity prediction FEVER Prec Rec F1 Acc Strict ISIS ERER IOIO GPT-4o-mini 74.46 70.09 72.21 87.24 81.97 54.91 78.21 71.27 Gemini-2.5-flash-lite 75.69 66.66 70.89 80.78 78.46 75.15 98.33 90.72 Qwen-2.5-32B-Instruct 79.51 65.17 71.63 83.19 78.81 66.95 95.45 84.98 Llama-3.1-8B-Instruct 56.30 69.40 62.17 83.08 76.27 43.79 67.85 62.83 REAL (ours) 71.64 85.89 78.12 95.66 94.28 99.06 99.99 99.69 SciFact Prec Rec F1 Acc Strict ISIS ERER IOIO GPT-4o-mini 66.41 74.62 70.28 83.44 82.54 62.84 99.07 87.06 Gemini-2.5-flash-lite 64.72 63.05 63.87 77.44 74.58 76.26 97.81 92.05 Qwen-2.5-32B-Instruct 69.76 62.69 66.04 76.33 73.22 73.56 99.83 92.32 Llama-3.1-8B-Instruct 63.75 67.51 65.57 82.66 77.88 47.72 93.25 76.48 REAL (ours) 68.80 72.97 70.82 89.39 84.86 92.19 100 97.54 Climate-FEVER Prec Rec F1 Acc Strict ISIS ERER IOIO GPT-4o-mini 74.01 57.78 64.90 65.49 62.62 75.08 96.63 92.80 Gemini-2.5-flash-lite 76.98 38.15 51.02 48.62 46.20 73.70 96.83 94.12 Qwen-2.5-32B-Instruct 78.32 42.97 55.50 51.71 49.28 79.74 100 96.14 Llama-3.1-8B-Instruct 70.04 59.95 64.60 69.35 63.40 37.14 72.94 70.90 REAL (ours) 71.39 62.56 66.68 74.86 68.69 95.58 100 98.82 Check-COVID Prec Rec F1 Acc Strict ISIS ERER IOIO GPT-4o-mini 54.12 74.85 62.82 88.60 87.02 66.33 97.43 86.32 Gemini-2.5-flash-lite 54.10 62.06 57.80 79.94 73.49 70.86 96.39 89.20 Qwen-2.5-32B-Instruct 53.20 63.70 57.98 83.71 76.53 68.89 99.74 89.17 Llama-3.1-8B-Instruct 53.26 63.58 57.96 83.05 72.35 44.72 92.81 74.30 REAL (ours) 54.10 72.50 61.97 88.90 83.65 92.64 100 97.65 6.5. Research Questions To evaluate the effectiveness of REAL and analyse its underlying mechanisms, we address the following research questions: • RQ1: Veracity and Grounding. Does REAL improve veracity accuracy while enforcing strict evidence grounding? We address this by comparing REAL against baselines on the in-domain evaluation benchmark (i.e., FEVER) (§7.1). • RQ2: Domain Generalisation. Does the evidence-dependent behaviour learned by REAL transfer to out-of-domain settings? We evaluate this on the out-of-domain benchmarks (§7.2). • RQ3: Ablation Studies. How do individual components in REAL contribute to accuracy, and does REAL generalise across model architectures? We answer this via detailed ablation studies (§7.3). • RQ4: Internal Mechanism. What internal representational changes underpin evidence-dependent behaviour in LLMs fine-tuned by REAL? We analyse this by extracting the dynamics of representation space during evidence ablation (§7.4). 7. Results 7.1. Veracity and Grounding (RQ1) Table 2. Ablation study for proposed REAL training framework. Model/Metrics Fact-Checking Metrics (Initial) FAE Metrics Evidence Selection Veracity prediction FEVER Prec Rec F1 Acc Strict ISIS ERER IOIO Standard SFT 75.45 85.10 79.99 96.18 93.87 10.75 11.02 14.36 w/o Evidence Ablation (§ 5.1) 71.14 86.61 78.12 96.06 94.52 10.80 11.04 14.67 w/o Evidence Enhancement (§ 5.2) 77.90 83.29 80.50 94.28 92.24 98.82 99.99 99.61 REAL 71.64 85.89 78.12 95.66 94.28 99.06 99.99 99.69 SciFact Prec Rec F1 Acc Strict ISIS ERER IOIO Standard SFT 71.95 42.77 53.65 90.04 68.56 9.77 15.37 22.81 w/o Evidence Ablation (§ 5.1) 64.87 75.27 69.68 91.07 86.16 14.35 21.45 25.49 w/o Evidence Enhancement (§ 5.2) 70.65 42.06 52.73 87.97 67.40 92.79 100 97.46 REAL 68.80 72.97 70.82 89.39 84.86 92.19 100 97.54 Climate-FEVER Prec Rec F1 Acc Strict ISIS ERER IOIO Standard SFT 69.86 32.89 44.72 84.45 61.85 3.92 8.09 20.03 w/o Evidence Ablation (§ 5.1) 69.88 63.70 66.65 84.79 74.86 7.68 7.68 22.47 w/o Evidence Enhancement (§ 5.2) 68.69 29.49 41.26 77.40 56.56 93.30 100 97.98 REAL 71.39 62.56 66.68 74.86 68.69 95.58 100 98.82 Check-COVID Prec Rec F1 Acc Strict ISIS ERER IOIO Standard SFT 67.84 50.23 57.73 87.91 66.70 9.24 18.26 23.82 w/o Evidence Ablation (§ 5.1) 55.94 73.24 63.43 91.08 85.53 13.10 19.54 24.09 w/o Evidence Enhancement (§ 5.2) 68.83 48.42 56.85 83.45 62.83 93.82 100 98.12 REAL 58.95 72.50 65.03 88.90 83.65 92.64 100 97.65 Table 1 reports standard fact-checking metrics and grounding-oriented FAE metrics on FEVER for in-domain evaluation. We observe that REAL consistently outperforms all other baselines across both verification accuracy and evidence-grounded behaviour. On veracity prediction, compared to Llama-3.1-8B-Instruct, REAL improves label accuracy from 83.0883.08 to 95.6695.66, and strict accuracy from 76.2776.27 to 94.2894.28. Interestingly, despite being built on Llama-3.1-8B-Instruct, REAL surpasses substantially larger proprietary and open-source LLMs across both fact-checking and FAE metrics. For example, compared with GPT-4o-mini, REAL improves strict accuracy from 81.9781.97 to 94.2894.28, while increasing the ISIS score from 54.9154.91 to 99.0699.06. Compared with Gemini-2.5-Flash-Lite, REAL further improves the IOIO score from 90.7290.72 to 99.6999.69. Regarding evidence selection, REAL achieves the best recall (85.8985.89) and F1 (78.1278.12) among baselines, indicating more complete and accurate identification of supporting evidence. Importantly, the improvements in evidence dependency do not come at the expense of fact-checking performance. REAL simultaneously achieves stronger grounding behaviour and higher verification accuracy than all baselines. In contrast, several baseline models maintain relatively high verification performance even after substantial evidence ablation, indicating that strong initial accuracy alone may not fully reflect genuine evidence dependency during verification. Regarding FAE metrics, we observe a clear difference in evidence-grounded behaviour. Baselines retain relatively high accuracy after evidence ablation, whereas REAL exhibits substantially stronger sensitivity to evidence availability. REAL achieves an ISIS of 99.0699.06 and an IOIO of 99.6999.69, compared to 75.1575.15 and 90.7290.72 for the best baseline (Gemini-2.5-Flash-Lite). Consistently, Figure 5 shows that REAL’s accuracy rapidly collapses after the first ablation round, closely approaching the ideal evidence-grounded trajectory illustrated in Figure 3. In contrast, baseline models stabilise at non-zero performance plateaus, indicating persistent prediction behaviour despite the removal of selected evidence. In summary, the above results demonstrate that REAL improves both veracity accuracy and evidence-dependent behaviour for in-domain evaluation, addressing RQ1. By enforcing strict reliance on retrieved evidence, REAL achieves the best evidence-grounded scores, effectively mitigating the influence of parametric knowledge in standard LLMs for fact-checking while preserving strong prediction accuracy. Table 3. Ablation study for proposed REAL trained with model from Qwen family Model/Metrics Fact-Checking Metrics (Initial) FAE Metrics Evidence Selection Veracity prediction FEVER Prec Rec F1 Acc Strict ISIS ERER IOIO Qwen-2.5-7B-Instruct 70.82 67.40 69.07 81.94 77.18 59.95 83.49 77.11 +REAL 81.40 79.73 80.56 95.23 93.43 99.04 100 99.66 SciFact Prec Rec F1 Acc Strict ISIS ERER IOIO Qwen-2.5-7B-Instruct 65.46 64.99 65.22 76.84 73.87 62.46 97.14 86.37 +REAL 67.03 67.22 67.12 89.52 81.76 86.71 99.86 95.73 Climate-FEVER Prec Rec F1 Acc Strict ISIS ERER IOIO Qwen-2.5-7B-Instruct 70.25 53.45 60.71 63.29 58.43 55.40 82.40 82.40 +REAL 71.64 51.85 60.13 78.83 66.70 94.97 99.86 98.60 Check-COVID Prec Rec F1 Acc Strict ISIS ERER IOIO Qwen-2.5-7B-Instruct 52.86 66.40 58.86 83.94 77.30 53.13 93.15 77.96 +REAL 56.82 69.80 62.68 90.49 83.05 90.14 99.89 96.73 7.2. Domain Generalisation (RQ2) Table 1 further reports out-of-domain evaluation results on SciFact, Climate-FEVER, and Check-COVID. REAL consistently maintains strong fact-checking performance while achieving the strongest FAE scores (ISIS, ERER, and IOIO) across all evaluation domains, demonstrating robust transfer of evidence-grounded behaviour beyond the FEVER training distribution. Importantly, the improvements in evidence dependency do not come at the expense of fact-checking performance. REAL achieves the best label accuracy and strict accuracy on SciFact (89.39/84.8689.39/84.86) and Climate-FEVER (74.86/68.6974.86/68.69), while remaining competitive on Check-COVID. More broadly, REAL is the only framework that consistently maintains both strong verification accuracy and strong evidence dependency across all evaluation datasets. Among baselines, a clear trade-off emerges between evidence-grounded behaviour and fact-checking performance. Qwen-2.5-32B-Instruct and Gemini-2.5-Flash-Lite exhibit relatively stronger evidence dependency but weaker verification accuracy, whereas GPT-4o-mini and Llama-3.1-8B-Instruct achieve stronger initial prediction performance while remaining substantially less sensitive to evidence ablation. This observation suggests that stronger evidence-grounded behaviour does not naturally emerge from model scale or general reasoning capability alone, but instead requires explicit supervision over evidence availability. The transfer behaviour is further visualised in Figure 5. Unlike the immediate collapse observed on FEVER, REAL’s accuracy on out-of-domain datasets approaches zero after approximately two ablation rounds. Although domain mismatch and fragmented evidence distributions prevent perfectly ideal collapse trajectories, REAL still maintains substantially stronger evidence sensitivity than all baselines throughout the ablation process. Overall, these results demonstrate that the evidence-grounded behaviour learned through REAL generalises effectively across domains, addressing RQ2. REAL enforces a strict dependence on retrieved evidence, enabling a strong transfer of evidence-grounded behaviour, yielding a more reliable verifier in general, even in out-of-domain settings. 7.3. Ablation Studies (RQ3) This section examines the effectiveness of REAL from two perspectives: the contribution of its individual components and its generalisation capability across model families. Different Design Choices. To assess the contribution of each component in REAL, we conduct ablation studies (Table 2) by isolating the effects of Evidence Ablation (§5.1) and Evidence Enhancement (§5.2). Directional alignment to knowledge-based fact-checking on FEVER, SciFact, Climate-FEVER and Check-COVID. Figure 6. Directional alignment to knowledge-based fact-checking.Directional alignment to knowledge-based fact-checking on FEVER, SciFact, Climate-FEVER and Check-COVID. Evidence Ablation is the primary driver of evidence-grounded behaviour, revealing a clear disconnect between veracity prediction accuracy and evidence dependency. Removing this component causes FAE scores to collapse across all datasets despite relatively stable initial fact-checking performance. On FEVER, the ISIS score drops from 99.0699.06 to 10.7510.75 under Standard SFT and to 10.8010.80 without Evidence Ablation. Similar degradation is consistently observed across all out-of-domain benchmarks. This observation aligns with the motivation of REAL: standard supervised fine-tuning alone does not constrain verifier behaviour under missing-evidence conditions, allowing prediction persistence through parametric knowledge. In contrast, Evidence Enhancement primarily improves supervision quality rather than grounding behaviour itself. Removing this component while retaining Evidence Ablation preserves strong FAE scores, but substantially degrades fact-checking performance, particularly on SciFact and Climate-FEVER. For example, on Climate-FEVER, the evidence selection F1 score decreases from 66.6866.68 to 41.2641.26 without Evidence Enhancement. This degradation arises because incomplete evidence annotations leave residual supporting evidence inside ablated contexts, introducing incorrect counterfactual supervision signals during training. Taken together, the two components of REAL play distinct and complementary roles. The Evidence Ablation improves evidence-grounded verification by constraining reliance on parametric knowledge when evidence is absent. The Evidence Enhancement, in contrast, improves veracity prediction and evidence selection under evidence-grounded behaviour. Different Model Families. We further evaluate the generality of REAL by applying it to Qwen-2.5-7B-Instruct. As shown in Table 3, REAL consistently improves both veracity prediction and FAE scores across all datasets over the base model. These observed superior performances are in line with the Llama-3 family, confirming that REAL generalises across model architectures and pre-training corpora. 7.4. Analysis of Representation Shifts under Evidence Ablation (RQ4) FAE results demonstrate that standard verifier predictions often persist despite the ablation of supporting evidence. To investigate whether this persistence is driven by a shift toward parametric decision-making, we analyse the verifier’s internal dynamics in the representation space. Following exploration in §4.2, we further investigate these two fact-checking paradigms: Knowledge-based Fact-Checking. In the knowledge-based paradigm, the verifier is provided only with the system instruction and claim c, without any documentary evidence, yielding a representation vector (K)(c)h^(K)(c) that relies solely on parametric knowledge. Evidence-based Fact-Checking. During the iterative evidence ablation in FAE §4.1, we extract a representation vector (r)(c,Sr)h^(r)(c,S^r) at each ablation round r to track the evolution of the verifier’s decision state. These representations are obtained from the final-layer hidden state at the last input token, immediately before output generation (Figure 7). This fixed probe position ensures a consistent measurement of the decision state as the availability of evidence decreases. Directional alignment to knowledge-based. To quantify the shift toward parametric decision-making during evidence ablation, we measure the directional alignment between the ablation-induced shift and the vector leading to the knowledge-based state (Li et al., 2016; Carlini et al., 2021). For each ablation round r≥1r≥ 1, we compute the cosine similarity between the current representational displacement and the vector directing from the evidence-based state to the knowledge-based state: DirK(r)=cos((r)(c,Sr)−(0)(c,S0),(K)(c)−(0)(c,S0)).Dir_K^(r)= \! (h^(r)(c,S^r)-h^(0)(c,S^0),\;h^(K)(c)-h^(0)(c,S^0) ). This formulation isolates the direction of representational change induced by evidence ablation, rather than absolute similarity between states. A higher DirK(r)Dir_K^(r) indicates that ablating evidence causes the model’s internal state to drift increasingly along the "evidence-to-knowledge" axis. This progressive drift signifies that as evidence becomes unavailable, the verifier systematically reverts to its parametric knowledge-based state to make a decision. Evidence-Based Fact-checking (with context). [SYSTEM] Verify the claim strictly against the provided evidence. [USER] Claim: c Document: line12: ... line27: ... line45: ... ↑ probe position Knowledge-Based Fact-checking (zero-context). [SYSTEM] Verify the claim using general knowledge. [USER] Claim: c ↑ probe position Figure 7. Illustration of two paradigms. The probe extracts the final-layer hidden state at the last input token, immediately before output generation, under identical formatting conditions.Illustration of evidence-based and knowledge-based inputs. The probe extracts the final-layer hidden state at the last input token, immediately before output generation, under identical formatting conditions. Results. Figure 6 reports the alignment trajectories of verifiers’ hidden states relative to the knowledge-based paradigm. All trajectories and their 95% bootstrap confidence intervals are computed over 2,000 iterations to ensure statistical robustness. On FEVER, SciFact, and Check-COVID, the base verifier shows a sharp, monotonic increase in DirK(r)Dir_K^(r) as evidence is ablated. This trend confirms that the base model’s hidden states shift toward its parametric knowledge state. Notably, on FEVER and Check-COVID, the base model’s confidence intervals become entirely disjoint from those of the REAL-trained model by the second ablation round. Even on SciFact, the base model’s mean alignment increases from 0.427 to 0.456, confirming a systematic reversion to parametric knowledge. In contrast, REAL-trained verifiers maintain flatter trajectories across all benchmarks. Despite the loss of evidence, their hidden states remain distinct from the knowledge-based paradigm. This suppression of directional drift is clear on FEVER, where the model does not shift toward its parametric knowledge. Similar stability is observed in SciFact and Check-COVID, where the trained model maintains a stable mean alignment (e.g., approximately 0.430 on SciFact). These results suggest that REAL decouples the decision process from parametric knowledge, preventing the model from falling back on parametric knowledge when evidence is unavailable. On Climate-FEVER, both models show higher proximity to the knowledge-based paradigm with overlapping confidence intervals. This reflects a stronger tendency of models to rely on parametric knowledge, likely due to the specific nature of this dataset. Unlike the other three datasets which provide coherent and complete documents, Climate-FEVER consists of isolated sentences from multiple sources, making it harder to form a robust evidentiary representation. Nevertheless, REAL still maintains a lower mean trajectory than the base model, showing persistent grounding pressure even in such fragmented, knowledge-intensive scenarios. These representational probes reveal the mechanism behind the observed prediction persistence. REAL suppresses the drift toward parametric baselines, ensuring that the verifier’s hidden states remain anchored to the availability of external evidence. 7.5. Prediction Stability In some rare cases, the accuracy drop does not correspond to the model switching its prediction to NOT_ENOUGH_INFO after evidence ablation but instead reflects unstable behaviour, in which the model changes its answers arbitrarily, in contrast to the ideal behaviour (§4.2). For example, a model flipping its judgment from SUPPORT to REFUTE after an ablation round indicates guessing rather than evidence-based fact-checking. A model should ideally transition toward the neutral state (NOT_ENOUGH_INFO) when evidence is ablated, rather than switching to the opposite label. To ensure FAE is reliably evaluated, we measure the frequency of contradictory label flips, where the prediction switches directly between SUPPORT and REFUTE during any ablation round. Table 4 reports the average percentage of such flips. We observe that contradictory flips are remarkably rare, where the highest flip rate is only 2.98%2.98\%, observed with Llama-3.1-8B-Instruct on the SciFact task. This low average frequency confirms that the performance decay observed in FAE is not an artefact of label instability. Instead, it demonstrates that models are consistently shifting toward a “neutral” stance as information is ablated, reinforcing the reliability of our ablation framework. Table 4. Percentage of SUPPORT–REFUTE label flips. Mode FEVER SciFact Climate- FEVER Check- COVID GPT-4o-mini 0.50 1.00 0.91 0.79 Gemini-2.5-flash-lite 0.16 0.72 0.41 0.37 Qwen-2.5-32B-Instruct 0.15 0.45 0.36 0.59 Llama-3.1-8B-Instruct 1.00 2.98 2.51 2.22 REAL 0.05 0.42 0.22 0.57 Interestingly, we further observe that after applying REAL, the percentage of such label flips decreases (from 2.98%2.98\% to 0.42%0.42\%) and this trend is consistent across all evaluation datasets. This observation further highlights the effectiveness of REAL in inducing truly evidence-grounded behaviour in fact-checking. 8. Conclusion This paper investigates a critical limitation in LLM-based automated fact-checking that strong verification accuracy does not necessarily imply evidence-grounded reasoning. Through Fact Ablated Evaluation (FAE), we show that current LLM verifiers often preserve their predictions even after the supporting evidence has been removed, revealing substantial reliance on parametric knowledge during verification. To address this issue, we propose Rigorous Evidence Ablation Learning (REAL), a training framework that explicitly supervises verifier behaviour under both complete evidence and ablated evidence conditions through counterfactual evidence supervision and enhanced evidence annotation. Experiments across four fact-checking benchmarks demonstrate that REAL substantially improves evidence-grounded behaviour while maintaining strong fact-checking performance under both in-domain and out-of-domain evaluation. Our results further show that many existing verifiers can maintain correct predictions even after the supporting evidence has been removed, indicating that standard fact-checking accuracy alone may not fully reflect whether predictions are genuinely supported by the retrieved evidence. By explicitly evaluating and supervising verifier behaviour under evidence ablation, this work provides a practical framework for analysing and improving evidence dependency in LLM-based fact-checking systems. Limitations FAE requires iterative evidence ablation and repeated verifier inference, making evaluation computationally more expensive than standard single-pass fact-checking evaluation. Consequently, FAE is currently suitable as a diagnostic framework for analysing evidence dependency rather than as a lightweight large-scale evaluation protocol. In addition, evidence redundancy and incomplete annotations in existing fact-checking benchmarks may still allow models to preserve correct predictions after ‘gold’ evidence removal, making it difficult to perfectly separate evidence-grounded verification from parametric recall. Although REAL partially mitigates this issue through evidence enhancement, complete isolation of all supporting evidence cannot always be guaranteed. GenAI Usage Disclosure This manuscript has benefited from the use of Generative AI (ChatGPT) to improve the quality of text produced by the authors. The authors remain responsible for the content. References Akhtar et al. (2025) M. Akhtar, R. Aly, Y. Chen, Z. Deng, M. Schlichtkrull, C. Whitehouse, and A. Vlachos The 2nd automated verification of textual claims (AVeriTeC) shared task: open-weights, reproducible and efficient systems. In Proc. of FEVER, Cited by: §1. Akhtar et al. (2024) M. Akhtar, M. Schlichtkrull, and A. Vlachos Ev2r: evaluating evidence retrieval in automated fact-checking. arXiv preprint arXiv:2411.05375. Cited by: §2. Bekoulis et al. (2021) G. Bekoulis, C. Papagiannopoulou, and N. Deligiannis A review on fact extraction and verification. ACM Computing Surveys (CSUR) 55 (1), p. 1–35. Cited by: §5.2. Braun et al. (2025) T. Braun, M. Rothermel, M. Rohrbach, and A. Rohrbach DEFAME: dynamic evidence-based FAct-checking with multimodal experts. In Proc. of ICML, Cited by: §1, §6.1. Carlini et al. (2021) N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), p. 2633–2650. Cited by: §1, §2, §7.4. Chen et al. (2022) J. Chen, R. Zhang, J. Guo, Y. Fan, and X. Cheng GERE: generative evidence retrieval for fact verification. In Proc. of SIGIR, Cited by: §1. ChenghaoZhu et al. (2025) C. ChenghaoZhu, N. Chen, Y. Gao, Y. Zhang, P. Tiwari, and B. Wang Is your llm outdated? a deep look at temporal generalization. In Proc. of NAACL, p. 7433–7457. Cited by: §1. Cheung and Lam (2023) T. Cheung and K. Lam Factllama: optimizing instruction-following language models with external knowledge for automated fact-checking. In 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), p. 846–853. Cited by: §1, §5.1, §6.1, §6.1. Chowdhury et al. (2025) S. Chowdhury, S. Fang, and S. Muresan FACT5: a novel benchmark and pipeline for nuanced fact-checking of complex statements. In Proc. of FEVER, Cited by: §1, §6.1. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §5.2, §6.1. Das et al. (2025) B. C. Das, M. H. Amini, and Y. Wu Security and privacy challenges of large language models: a survey. ACM Computing Surveys 57 (6), p. 1–39. Cited by: §1. Deng et al. (2025a) X. Deng, X. Wang, and M. Stevenson + VeriRel: verification feedback to enhance document retrieval for scientific fact checking. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 4706–4711. Cited by: §1. Deng et al. (2025b) X. Deng, X. Wang, and M. Stevenson The next phase of scientific fact-checking: advanced evidence retrieval from complex structured academic papers. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR), p. 436–448. Cited by: §1. Deng (2026) X. Deng Towards evidence-aware retrieval and verification for scientific fact-checking. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 5309–5309. Cited by: §1. Derczynski et al. (2020) L. Derczynski, J. Binau, and H. Schulte Maintaining quality in FEVER annotation. In Proc. of FEVER, Cited by: §5.2. Di Palma et al. (2025) D. Di Palma, F. A. Merra, M. Sfilio, V. W. Anelli, F. Narducci, and T. Di Noia Do llms memorize recommendation datasets? a preliminary study on movielens-1m. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §1, §2. Diggelmann et al. (2020) T. Diggelmann, J. Boyd-Graber, J. Bulian, M. Ciaramita, and M. Leippold Climate-fever: a dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614. Cited by: 3rd item, §6.3. Gao et al. (2023) Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang Retrieval-augmented generation for large language models: a survey. arXiv preprint arXiv:2312.10997 2 (1). Cited by: §2. Geng et al. (2025) J. Geng, J. Tonglet, and I. Gurevych M4FC: a multimodal, multilingual, multicultural, multitask real-world fact-checking dataset. arXiv preprint arXiv:2510.23508. Cited by: §1, §6.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §6.1, §6.1. Guo et al. (2022a) Z. Guo, M. Schlichtkrull, and A. Vlachos A survey on automated fact-checking. Transactions of the Association for Computational Linguistics 10. Cited by: §1, §1, §1, §3.1, §3.2, §3.2. Guo et al. (2022b) Z. Guo, M. Schlichtkrull, and A. Vlachos A survey on automated fact-checking. Transactions of the Association for Computational Linguistics 10, p. 178–206. Cited by: §1. Hanselowski et al. (2019) A. Hanselowski, C. Stab, C. Schulz, Z. Li, and I. Gurevych A richly annotated corpus for different tasks in automated fact-checking. In Proc. of CoNLL, Cited by: §1. Honovich et al. (2022) O. Honovich, R. Aharoni, J. Herzig, H. Taitelbaum, D. Kukliansy, V. Cohen, T. Scialom, I. Szpektor, A. Hassidim, and Y. Matias TRUE: re-evaluating factual consistency evaluation. In Proc. of NAACL, Cited by: §2. Hsu et al. (2024) I. Hsu, Z. Wang, L. Le, L. Miculicich, N. Peng, C. Lee, and T. Pfister CaLM: contrasting large and small language models to verify grounded generation. In Findings of the Association for Computational Linguistics: ACL 2024, Cited by: §2. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: low-rank adaptation of large language models.. In Proc. of ICLR, Cited by: §6.2. Hu et al. (2023) X. Hu, Z. Hong, Z. Guo, L. Wen, and P. Yu Read it twice: towards faithfully interpretable fact verification by revisiting evidence. In Proc. of SIGIR, Cited by: §1. Huang et al. (2024) L. Huang, X. Feng, W. Ma, Y. Gu, W. Zhong, X. Feng, W. Yu, W. Peng, D. Tang, D. Tu, and B. Qin Learning fine-grained grounded citations for attributed large language models. In Findings of the Association for Computational Linguistics: ACL 2024, Cited by: §2. Hurst et al. (2024) A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §5.2, §6.1. Jannah et al. (2025) S. Z. Jannah, E. Aco, S. Peng, S. Wakamiya, and E. Aramaki Multilingual symptom detection on social media: enhancing health-related fact-checking with LLMs. In Proc. of FEVER, Cited by: §1, §6.1. Joren et al. (2025) H. Joren, J. Zhang, C. Ferng, D. Juan, A. Taly, and C. Rashtchian Sufficient context: a new lens on retrieval augmented generation systems. In Proc. of ICLR, Cited by: §2. Kassem et al. (2025) A. M. Kassem, O. Mahmoud, N. Mireshghallah, H. Kim, Y. Tsvetkov, Y. Choi, S. Saad, and S. Rana ALPACA AGAINST VICUNA: using LLMs to uncover memorization of LLMs. In Proc. of NAACL, Cited by: §1. Kiyomaru et al. (2024) H. Kiyomaru, I. Sugiura, D. Kawahara, and S. Kurohashi A comprehensive analysis of memorization in large language models. In Proceedings of the 17th International Natural Language Generation Conference, Cited by: §2. Kumar et al. (2025) G. Kumar, D. Mazumder, A. Garg, and J. Patro Improving the fact-checking performance of language models by relying on their entailment ability. arXiv preprint arXiv:2505.15050. Cited by: §1, §5.1, §6.1, §6.1. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc. of NeurIPS, Cited by: §1. Li et al. (2025) H. Li, J. Huang, M. Ji, Y. Yang, and R. An Use of retrieval-augmented large language model for covid-19 fact-checking: development and usability study. Journal of medical Internet research 27, p. e66098. Cited by: §1, §6.1. Li et al. (2016) J. Li, W. Monroe, and D. Jurafsky Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. Cited by: §7.4. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §6.2. Luken et al. (2018) J. Luken, N. Jiang, and M. de Marneffe QED: a fact verification system for the FEVER shared task. In Proc. of FEVER, Cited by: §1. Mohammadkhani et al. (2024) M. G. Mohammadkhani, A. G. Mohammadkhani, and H. Beigy Zero-shot learning and key points are all you need for automated fact-checking. In Proc. of FEVER, Cited by: §1, §6.1. Nie et al. (2019) Y. Nie, H. Chen, and M. Bansal Combining fact extraction and verification with neural semantic matching networks. In Proc. of AAAI, Cited by: §1. Niu et al. (2024) C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, R. Zhong, J. Song, and T. Zhang RAGTruth: a hallucination corpus for developing trustworthy retrieval-augmented language models. In Proc of ACL, Cited by: §2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §6.2. Pan et al. (2023) L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y. Wang, M. Kan, and P. Nakov Fact-checking complex claims with program-guided reasoning. In Proc. of ACL, Cited by: §1. Park et al. (2024) H. Park, D. Lee, J. Kim, C. Park, and C. Park Dunamu-ml’s submissions on AVERITEC shared task. In Proc. of FEVER, Cited by: §1, §6.1. Putta et al. (2025) A. R. Putta, J. Devasier, and C. Li ClaimCheck: automatic fact-checking of textual claims using web evidence. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, Cited by: §1, §5.1, §6.1, §6.1. Qwen et al. (2025) Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §5.2, §6.1, §6.1. Rothermel et al. (2024) M. Rothermel, T. Braun, M. Rohrbach, and A. Rohrbach InFact: a strong baseline for automated fact-checking. In Proc. of FEVER, Cited by: §1, §6.1. Ru et al. (2024) D. Ru, L. Qiu, X. Hu, T. Zhang, P. Shi, S. Chang, C. Jiayang, C. Wang, S. Sun, H. Li, et al. RAGCHECKER: a fine-grained framework for diagnosing retrieval-augmented generation. In Proc. of NeurIPS, Cited by: §2. Schlichtkrull et al. (2024) M. Schlichtkrull, Y. Chen, C. Whitehouse, Z. Deng, M. Akhtar, R. Aly, Z. Guo, C. Christodoulopoulos, O. Cocarascu, A. Mittal, J. Thorne, and A. Vlachos The automated verification of textual claims (AVeriTeC) shared task. In Proc. of FEVER, Cited by: §1. Schlichtkrull et al. (2023) M. S. Schlichtkrull, Z. Guo, and A. Vlachos AVeritec: a dataset for real-world claim verification with evidence from the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: §3.2. Shcharbakova et al. (2025) H. Shcharbakova, T. Anikina, N. Skachkova, and J. V. Genabith When scale meets diversity: evaluating language models on fine-grained multilingual claim verification. In Proc. of FEVER, Cited by: §1, §5.1, §6.1, §6.1. Song et al. (2025) M. Song, S. H. Sim, R. Bhardwaj, H. L. Chieu, N. Majumder, and S. Poria Measuring and enhancing trustworthiness of LLMs in RAG through grounded attributions and learning to refuse. In Proc. of ICLR, Cited by: §2. Tang et al. (2024) L. Tang, P. Laban, and G. Durrett MiniCheck: efficient fact-checking of LLMs on grounding documents. In Proc. of EMNLP, Cited by: §2. Tang et al. (2025) Y. Tang, K. Zhou, Y. Min, W. X. Zhao, J. Sha, Z. Sheng, and S. Wang Enhancing chain-of-thought reasoning via neuron activation differential analysis. In Proc. of EMNLP, Cited by: §4.2. Thakur et al. (2024) N. Thakur, L. Bonifacio, C. Zhang, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, B. Chen, M. Rezagholizadeh, and J. Lin “Knowing when you don’t know”: a multilingual relevance assessment dataset for robust retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, Cited by: §2. Thorne et al. (2018) J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal FEVER: a large-scale dataset for fact extraction and VERification. In Proc. of NAACL, Cited by: §3.2, §3.2, §5.2, 1st item, §6.3. Ullrich and Drchal (2025) H. Ullrich and J. Drchal AIC CTU@FEVER 8: on-premise fact checking through long context RAG. In Proc. of FEVER, Cited by: §1, §6.1. Ullrich et al. (2024) H. Ullrich, T. Mlynář, and J. Drchal AIC CTU system at AVeriTeC: re-framing automated fact-checking as a simple RAG task. In Proc. of FEVER, Cited by: §1, §6.1. Vlachos and Riedel (2014) A. Vlachos and S. Riedel Fact checking: task definition and dataset construction. In Proc. of ACL, Cited by: §1, §1. Vladika et al. (2025) J. Vladika, M. Dhaini, and F. Matthes Facts fade fast: evaluating memorization of outdated medical knowledge in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, Cited by: §1. Vladika and Matthes (2023) J. Vladika and F. Matthes Scientific fact-checking: a survey of resources and approaches. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: §1, §1, §3.1. Vladika and Matthes (2024) J. Vladika and F. Matthes Improving health question answering with reliable and time-aware evidence retrieval. In Findings of the Association for Computational Linguistics: NAACL 2024, Cited by: §1. Wadden et al. (2020) D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi Fact or fiction: verifying scientific claims. In Proc. of EMNLP, Cited by: §3.2, §3.2, 2nd item, §6.3. Wadden and Lo (2021) D. Wadden and K. Lo Overview and insights from the SCIVER shared task on scientific claim verification. In Proceedings of the Second Workshop on Scholarly Document Processing, Cited by: §6.3. Wang et al. (2023) G. Wang, K. Harwood, L. Chillrud, A. Ananthram, M. Subbiah, and K. McKeown Check-COVID: fact-checking COVID-19 news claims with scientific evidence. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: §3.2, §3.2, 4th item, §6.3. Wang et al. (2024) Y. Wang, R. Gangi Reddy, Z. M. Mujahid, A. Arora, A. Rubashevskii, J. Geng, O. Mohammed Afzal, L. Pan, N. Borenstein, A. Pillai, I. Augenstein, I. Gurevych, and P. Nakov Factcheck-bench: fine-grained evaluation benchmark for automatic fact-checkers. In Findings of the Association for Computational Linguistics: EMNLP 2024, Cited by: §1, §6.1. Wei et al. (2024) J. Wei, C. Yang, X. Song, Y. Lu, N. Z. Hu, J. Huang, D. Tran, D. Peng, R. Liu, D. Huang, C. Du, and Q. V. Le Long-form factuality in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §1, §6.1. Wei et al. (2025) J. Wei, Y. Zhang, L. Y. Zhang, M. Ding, C. Chen, K. Ong, J. Zhang, and Y. Xiang Memorization in deep learning: a survey. ACM Computing Surveys 58 (4), p. 1–35. Cited by: §1, §2. Wu et al. (2021) L. Wu, Y. Rao, X. Yang, W. Wang, and A. Nazir Evidence-aware hierarchical interactive attention networks for explainable claim verification. In Proc. of IJCAI, Cited by: §1. Xie et al. (2025) Z. Xie, R. Xing, Y. Wang, J. Geng, H. Iqbal, D. Sahnan, I. Gurevych, and P. Nakov FIRE: fact-checking with iterative retrieval and verification. In Findings of the Association for Computational Linguistics: NAACL 2025, Cited by: §1, §6.1. Yang et al. (2025) D. Yang, L. Zeng, J. Rao, and Y. Zhang Knowing you don’t know: learning when to continue search in multi-round rag through self-practicing. In Proc. of SIGIR, Cited by: §2. Ye et al. (2024) X. Ye, R. Sun, S. Arik, and T. Pfister Effective large language model adaptation for improved grounding and citation generation. In Proc. of NAACL, Cited by: §2. Yoon et al. (2024) Y. Yoon, J. Jung, S. Yoon, and K. Park HerO at AVeriTeC: the herd of open large language models for verifying real-world claims. In Proc. of FEVER, Cited by: §1, §5.1, §6.1, §6.1, §6.1. Yoon et al. (2025) Y. Yoon, J. Jung, S. Yoon, and K. Park Team HUMANE at AVeriTeC 2025: HerO 2 for efficient fact verification. In Proc. of FEVER, Cited by: §1, §6.1. Zeng et al. (2021) X. Zeng, A. S. Abumansour, and A. Zubiaga Automated fact-checking: a survey. Language and Linguistics Compass 15 (10), p. e12438. Cited by: §1, §1, §1, §3.1, §3.2, §3.2. Zheng et al. (2024) L. Zheng, C. Li, X. Zhang, Y. Shang, F. Huang, and H. Jia Evidence retrieval is almost all you need for fact verification. In Findings of the Association for Computational Linguistics: ACL 2024, Cited by: §1. Zhou et al. (2025) D. Zhou, R. Pop, Y. Zhu, and E. Kharlamov GQC: LLM-based grouped QA consolidation for open-domain fact verification at AVeriTeC. In Proc. of FEVER, Cited by: §1, §6.1.