Paper deep dive
Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems
Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/24/2026, 6:06:31 AM
Summary
This paper introduces an Evaluation Agent, a middleware for Retrieval-Augmented Generation (RAG) systems designed to detect misinformation and knowledge poisoning. The agent combines Natural Language Inference (NLI) for factual verification, a five-signal poison detector, and a Trust Index formula to assess context integrity before generation. Evaluated on TruthfulQA and FEVER benchmarks using Llama 3.3 70B, the agent achieves 91% accuracy and 100% precision, effectively blocking instruction injection attacks. However, it struggles with subtle in-place edits like entity swaps. The study highlights a 'Security-Reliability Gap' where high semantic relevance does not guarantee factual truth.
Entities (12)
Relation Signals (7)
Evaluation Agent → detects → Instruction Injection
confidence 97% · with 100% recall on instruction injection...
Evaluation Agent → protectsagainst → Knowledge Poisoning
confidence 96% · Adversaries exploit this through knowledge poisoning... We propose an Evaluation Agent... to detect misinformation and knowledge poisoning
Evaluation Agent → uses → Natural Language Inference
confidence 95% · middleware that combines Natural Language Inference (NLI) factual verification...
Evaluation Agent → evaluateson → TruthfulQA
confidence 94% · On TruthfulQA with Llama 3.3 70B, the agent reaches 91% accuracy...
Evaluation Agent → calculates → Trust Index
confidence 93% · Trust Index T = 0.4 F + 0.35 C + 0.25 (1 - P )...
Evaluation Agent → struggleswith → Entity Swap
confidence 92% · while in-place edits, such as entity swaps, remain hard to detect.
Evaluation Agent → appliedto → OWASP Top 10
confidence 90% · In a software-engineering use case, a secure-coding assistant over guidance from the Open Worldwide Application Security Project (OWASP) Top 10...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) outputs in external knowledge, but RAG systems usually trust whatever they retrieve, creating a Security-Reliability Gap: high semantic relevance does not guarantee factual truth. Adversaries exploit this through knowledge poisoning, inserting malicious documents to cause targeted misinformation. We propose an Evaluation Agent, middleware that combines Natural Language Inference (NLI) factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index T = 0.4 F + 0.35 C + 0.25 (1 - P ) with a non-linear dampener for high-contamination contexts. On TruthfulQA with Llama 3.3 70B, the agent reaches 91% accuracy and 100% precision, with 100% recall on instruction injection, while in-place edits, such as entity swaps, remain hard to detect. Across three LLMs the Trust Index stays discriminative, with a Receiver Operating Characteristic Area Under the Curve (ROC-AUC) of 0.73 to 0.81; generation style matters more than model size, and per-LLM threshold calibration restores baseline competitive accuracy, whereas a weaker FEVER result shows that cross-dataset generalization requires domain-specific calibration. In a software-engineering use case, a secure-coding assistant over guidance from the Open Worldwide Application Security Project (OWASP) Top 10 and the Common Weakness Enumeration (CWE), the agent reliably blocks instruction injection of unsafe advice (F1 92%), while contradiction and subtle semantic weakening remain hard. Throughout, the agent measures detection of poisoned context before generation, not whether the LLM adopts the injected misinformation. We release the proposed approach, attack generator, and experimental artifacts at the link: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.21095v1
- Canonical: https://arxiv.org/abs/2608.21095v1
Trouble viewing inline? Open PDF directly →
Full Text
42,766 characters extracted from source content.
Expand or collapse full text
Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems Balkrishna Giri , Md Toufique Hasan , Jussi Rasku , Muhammad Waseem , and Pekka Abrahamsson Affiliation: Faculty of Information Technology and Communication Sciences, Tampere University Tampere, Finland e-mail: balkrishna.giri, mdtoufique.hasan, jussi.rasku, muhammad.waseem, pekka.abrahamsson@tuni.fi Abstract Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) outputs in external knowledge, but RAG systems usually trust whatever they retrieve, creating a Security-Reliability Gap: high semantic relevance does not guarantee factual truth. Adversaries exploit this through knowledge poisoning, inserting malicious documents to cause targeted misinformation. We propose an Evaluation Agent, middleware that combines Natural Language Inference (NLI) factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index T=0.4F+0.35C+0.25(1−P)T=0.4\,F+0.35\,C+0.25\,(1-P) with a non-linear dampener for high-contamination contexts. On TruthfulQA with Llama 3.3 70B, the agent reaches 91% accuracy and 100% precision, with 100% recall on instruction injection, while in-place edits, such as entity swaps, remain hard to detect. Across three LLMs the Trust Index stays discriminative, with a Receiver Operating Characteristic Area Under the Curve (ROC-AUC) of 0.73 to 0.81; generation style matters more than model size, and per-LLM threshold calibration restores baseline-competitive accuracy, whereas a weaker FEVER result shows that cross-dataset generalization requires domain-specific calibration. In a software-engineering use case, a secure-coding assistant over guidance from the Open Worldwide Application Security Project (OWASP) Top 10 and the Common Weakness Enumeration (CWE), the agent reliably blocks instruction injection of unsafe advice (F1 92%), while contradiction and subtle semantic weakening remain hard. Throughout, the agent measures detection of poisoned context before generation, not whether the LLM adopts the injected misinformation. We release the proposed approach, attack generator, and experimental artifacts at the link: https://github.com/GPT-Laboratory/TrustworthyRAG. Keywords: Retrieval-augmented generation (RAG); large language models (LLMs); knowledge poisoning; trustworthy AI; misinformation detection; LLM security; natural language inference (NLI); AI safety. I Introduction Large Language Models (LLMs) built on the Transformer architecture [1] now power knowledge-intensive applications, such as question answering, search, and code assistance, yet their parametric knowledge is frozen at training time and they often hallucinate, producing fluent text that is factually wrong [2]. Retrieval-Augmented Generation (RAG) [3] addresses this by retrieving documents from an external corpus and conditioning generation on them, and it has become a default pattern for deploying LLMs over private or fast-changing knowledge [4]. Grounding generation in retrieved text creates a new dependency: the answer is only as trustworthy as the corpus behind it. Standard RAG assumes a benign knowledge base, and existing evaluation frameworks, such as RAGAS [5] and ARES [6], measure faithfulness to the retrieved context rather than the integrity of that context. In open-world deployments, however, an adversary can insert malicious documents through knowledge poisoning [7]; injecting only five malicious passages per question into a corpus of millions (≈ 0.0002%) can drive attack success to roughly 90%. A system that answers faithfully from a poisoned document therefore scores well under current metrics while emitting compromised output. We call this the Security-Reliability Gap: high semantic relevance is treated as a proxy for truth, and the few available poisoning defenses are mostly offline corpus-cleaning steps with no online trust layer inside the inference loop. This paper presents an Evaluation Agent that closes this Security-Reliability Gap. It acts as defensive middleware that screens retrieved context before generation and outputs an interpretable Trust Index that fuses Natural Language Inference (NLI) factual verification [8][9], a five-signal poison detector (Section I), and a cross-document consistency estimate. We evaluate the agent on two public benchmarks (TruthfulQA and FEVER) across three LLMs and four attack strategies, and then apply it to a software-engineering use case, a secure-coding assistant that retrieves from guidance based on the Open Worldwide Application Security Project (OWASP) Top 10 and the Common Weakness Enumeration (CWE). We study three research questions: • RQ1. How effectively does the Evaluation Agent detect misinformation and knowledge poisoning in retrieved context? • RQ2. How does NLI-based factual verification change the trustworthiness of RAG output? • RQ3. How resilient is the approach across LLMs, datasets, and attack strategies, including a secure-coding setting in the software-development lifecycle (SDLC)? The RQs operationalize this gap: RQ1 targets detection, RQ2 the NLI trust layer in the inference loop, and RQ3 robustness across generators, domains, and attacks; Section VI answers each explicitly. Code, prompts, the attack generator, and experimental artifacts are publicly available [10]. We find that the agent detects overt poisoning well (100% recall on instruction injection, 91% accuracy and 100% precision on TruthfulQA mixed attacks) but that in-place edits, such as entity swaps and subtle weakening, stay near-undetectable, which is a limit of surface-signal detection rather than of tuning. We also show that NLI-based trust scoring depends on the generating LLM (generation style matters more than model size), is stable across repeated runs, and is invariant to retrieval depth, while a weaker FEVER result bounds external validity. These findings extend prior RAG evaluation by adding an online check on context integrity, showing where such checks succeed and where world-knowledge verification is needed. I Background and Related Work RAG architectures. RAG has developed from retrieval-augmented language models [11] and retrieve-then-generate pipelines [12] into modular systems [4]. Modular RAG separates the retriever, generator, and evaluator into exchangeable components. Self-RAG [13] adds reflection tokens for retrieval and critique, but it is unclear whether self-correction can resist adversarial inputs designed to manipulate the critique process. Evaluation. Retrieval quality is often measured with Information Retrieval (IR) metrics, such as top-k retrieval accuracy [14] and Mean Reciprocal Rank (MRR). These metrics measure relevance, but not safety. A poisoned document that is highly relevant to a query can still score well. Generation quality has moved from n-gram overlap toward factuality-based measures that separate faithfulness from factual correctness [8]. TruthfulQA [15] focuses on imitative falsehoods, and the TRUE benchmark [9] supports factual-consistency evaluation with metrics, such as NLI. Recent hallucination detectors mainly verify generated text rather than the integrity of the corpus. SelfCheckGPT [16] measures self-consistency across sampled generations, while fine-grained factuality scores [17] check atomic facts against a trusted knowledge source. RAGAS [5] and ARES [6] use LLMs to judge faithfulness and relevance, but they do not evaluate whether the retrieved corpus itself is trustworthy. Attacks and defenses. Indirect prompt injection places malicious instructions inside retrievable content [18], and poisoning the web-scale datasets used for model pre-training can be done at low cost [19]. PoisonedRAG [7] studies knowledge poisoning in the inference corpus, and adversarial passages can be optimized so that they enter the retrieved set for many queries [20]. On the defense side, retrieval augmentation can reduce hallucination [21], and NLI-based verification can compare a retrieved document as the premise with the generated answer as the hypothesis to estimate factual consistency [8][9]. However, many poisoning defenses are still offline, such as filtering or cleaning the corpus before indexing. These methods cannot detect poisoned content that appears at inference time. Toxicity classifiers also miss misinformation that is written in neutral and professional language. Closest to our setting are online RAG defenses that filter or vote over the retrieved passages themselves: certifiably robust aggregation generates an answer per isolated passage and securely aggregates the results [22], and trust-aware retrieval filtering discards suspicious passages before generation [23]. These defenses act on the retriever side and often need multiple generations per query or corpus-level assumptions; our agent instead scores the retrieved set and the generated answer jointly and emits an interpretable verdict without modifying retriever or generator. Our work combines factual verification and poison detection into one interpretable online score, and shows where this type of surface-signal detection works and where it fails. Security in the software-development lifecycle. Software-engineering research increasingly studies how generative Artificial Intelligence (AI) is used in development [24] and how AI is adopted in quality assurance [25], and how digital experimentation supports sustainability in software-intensive industry [26]. LLM coding assistants are now widely used during development, but they can generate insecure code, and developers may over-trust their suggestions [27][28]. The OWASP Top 10 for LLM Applications lists data and model poisoning and misinformation as major risks for these systems [29]. When a coding assistant uses RAG over an organization’s secure-coding guidance, such as the OWASP Top 10 [30] and CWE [31], that guidance corpus becomes a poisoning target. A single corrupted rule can lead the assistant to give unsafe recommendations. This motivates evaluating the Evaluation Agent in an SDLC setting (Section V-G). I Proposed Approach Threat model. We assume a black-box knowledge-injection adversary that can insert or modify documents in the retrieval corpus but cannot access the model weights, user prompt, or the Evaluation Agent. The adversary’s goal is to make the generator emit attacker-chosen misinformation or unsafe recommendations. This reflects RAG deployments that ingest third-party, web-sourced, or community-contributed content. The Evaluation Agent is defensive middleware positioned between retrieval and the final trust decision (Figure 1). It orchestrates three modules: an NLI verifier, a poison detector, and a Trust Index calculator, and emits a numeric trust score, a categorical trust level, and human-readable warnings. Operationally, the agent uses two stages. The poison detector and consistency estimator screen retrieved documents before generation, while the NLI verifier runs afterward using each document as the premise and the generated answer as the hypothesis. The signals are fused into a verdict, while deployments may also use the pre-generation signals alone. Input: query, retrieved docs, scores, embeddingsStage 1: pre-generation screeningPoison DetectorConsistency EstimatorLLM generation (answer)Stage 2: post-generation verificationNLI Verifier (docs vs. answer)Trust Index CalculatorEvaluationResult: score, level, warnings Figure 1: Two-stage Evaluation Agent pipeline: poison detection and consistency screening before generation (Stage 1), NLI verification of the generated answer after it (Stage 2). Color coding: gray input and generation steps, blue analysis modules, orange score fusion, green output verdict. I-A Factual Verification via NLI The NLI verifier uses facebook/bart-large-mnli as a sequence classifier. For each retrieved document (premise) and the generated answer (hypothesis) it produces entailment, neutral, and contradiction probabilities. The factuality score SfactualityS_factuality aggregates entailment over documents with a meaningful entailment signal; when none exists, an inconclusive baseline of 0.50.5 is returned. This entailment-focused design is motivated by an empirical observation: bart-large-mnli assigns near-maximal contradiction (≈0.99≈ 0.99) to both genuine contradictions and merely unrelated text, so using raw contradiction as a negative signal would systematically penalize off-topic but benign documents. Strong negative signals are therefore reserved for the dedicated poison-detection pathway. I-B Multi-Signal Poison Detection The poison detector combines five independent signals per document: (1) linguistic patterns (instruction-override phrases, contradiction markers); (2) structural anomalies (excessive uppercase, suspicious repetition); (3) intra-document consistency (NLI between the first and second halves of a document); (4) cross-document consistency (pairwise contradiction checks, scoped so a single poisoned document cannot inflate the poison probability of clean neighbors); and (5) semantic-outlier analysis (embedding deviation from the batch centroid). When retrieval scores are available, document-level probabilities are combined with relevance weighting, Poverall=∑i=1kwiPi,wi=si∑jsj,P_overall= _i=1^kw_i\,P_i, w_i= s_i _js_j, (1) where sis_i is the retrieval similarity of document i. This prevents low-relevance suspicious neighbors from dominating the batch verdict. The linguistic and intra-document signals target overt injections and explicit contradictions; the cross-document and semantic-outlier signals target inconsistent or anomalous insertions; and the structural signal flags formatting artifacts. The set was derived from the attack surface of the threat model, and each signal is grounded in prior observations: linguistic override patterns follow the payload style of indirect prompt injection [18]; the intra- and cross-document checks instantiate NLI-based consistency evaluation [8][9]; and semantic-outlier analysis adapts embedding-space anomaly detection to the retrieved batch, motivated by corpus-poisoning attacks that insert semantically deviant passages [20]. In-place value substitutions that preserve surface form fall, by construction, outside this signal set, a limit we quantify in Section V-G. I-C The Trust Index The Trust Index T∈[0,1]T∈[0,1] is a weighted combination of three components: T=αSfactuality+βSconsistency+γ(1−Ppoison),T=α\,S_factuality+β\,S_consistency+γ\,(1-P_poison), (2) with default weights α=0.40α=0.40, β=0.35β=0.35, γ=0.25γ=0.25 (α+β+γ=1α+β+γ=1). With these defaults, Eq. (2) is exactly the Trust Index stated in the abstract, T=0.4F+0.35C+0.25(1−P)T=0.4\,F+0.35\,C+0.25\,(1-P). The weights reflect signal reliability: NLI entailment (factuality) is the most precise, cross-document agreement (consistency) a strong corpus-integrity proxy, and heuristic poison signals are weighted lowest to limit false-positive influence. Section V-B quantifies each component’s contribution and shows that the operating point is stable under moderate perturbations of these manually chosen weights. Non-linear dampener. A purely linear T fails under high contamination: if the LLM ignores a poisoned passage and answers from the legitimate part, factuality and consistency stay high while PpoisonP_poison is large. Because γ=0.25γ=0.25, the poison term can reduce T by at most 0.250.25, so T can exceed the threshold τ=0.5τ=0.5 despite Ppoison=0.9P_poison=0.9. We therefore apply a multiplicative dampener when Ppoison>0.70P_poison>0.70: d(P)=1.0P≤0.701.0−0.4P−0.700.30P>0.70Tfinal=T⋅d(P).d(P)= cases1.0&P≤ 0.70\\[2.0pt] 1.0-0.4\, P-0.700.30&P>0.70 cases T_final=T· d(P). (3) The dampener is continuous at the threshold (d(0.70)=1d(0.70)=1), bounded (d(1.0)=0.6d(1.0)=0.6, i.e. at most −40%-40\%), and proportional to contamination confidence. For the masking example (T=0.59T=0.59, P=0.9P=0.9), d(0.9)=0.733d(0.9)=0.733 gives Tfinal=0.43<τT_final=0.43<τ, correctly flagging the response. For clean contexts the dampener is inactive; the multiplier falls smoothly with contamination, so only high-confidence poisoning is penalized. A secondary modifier reduces T when the best retrieval similarity falls below 0.300.30 (low-confidence retrieval). The final score maps to four trust levels (HIGH/MEDIUM/LOW/VERY LOW); binary decision uses τ=0.5τ=0.5. LLM dependency. Because SfactualityS_factuality is an NLI entailment score against the generated answer, it depends on generation style, not only content. Hedged prefixes (e.g., “Based on the provided context…”) lower entailment and thus T, even for correct answers. This makes τ and the weights effectively LLM-specific hyperparameters (Section V-C). IV Experimental Design Datasets. We use two public benchmarks. TruthfulQA [15] documents are formatted as “question + best answer” (≈ 20 to 30 words); its counterintuitive correct answers make it a realistic poisoning testbed. FEVER (Fact Extraction and VERification) [32] documents are enriched to “question + verdict + claim” (≈ 30 to 40 words); bare claims of 8 to 12 words caused ∼ 70% of factuality scores to fall back to the inconclusive 0.50.5 baseline, motivating the enrichment. For a software-engineering setting we additionally construct a secure-coding knowledge base of 40 rules curated from the OWASP Top 10 [30] and CWE [31], evaluated in Section V-G. Each rule is a short natural-language recommendation in the same “question + best answer” format as the benchmark corpora, e.g., “How can Structured Query Language (SQL) injection be prevented?” paired with parameterized-query guidance referencing CWE-89. The rules were written by the authors as paraphrases of the normative guidance in the source documents, cross-checked against them, and cover the major OWASP categories; the full rule set is included in the released artifacts for independent inspection [10]. Protocol. Each run has two parts: 50 clean queries (Part A) and the same 50 queries against a corpus with 30% poisoned documents (Part B), for 100 samples per run. We evaluate four rule-based poisoning strategies: contradiction (appends a contradicting statement), instruction injection (appends an override directive), entity swap (in-place replacement of entities/numbers, with no appended artifacts), and subtle manipulation (false qualifiers/hedging). A mixed setting assigns strategies randomly across poisoned documents. Part A contributes 50 clean samples and Part B contributes 50 samples of which 30% (≈ 15) are labelled poisoned, so a 100-sample run contains about 15 poisoned and 85 clean samples. A naive always-trust baseline that labels every sample clean is therefore correct on 85 clean samples, i.e. ≈ 85% accuracy; hence Δ and F1, not accuracy, are the informative metrics. Models and metrics. Documents are embedded with the Sentence-Transformers model all-MiniLM-L6-v2 [33] (384-d) or snowflake-arctic-embed2 (1024-d), indexed in Facebook AI Similarity Search (FAISS) (cosine, top-K). Generation uses llama3.3:70b (primary) or qwen3.5:35b; NLI runs on a Central Processing Unit (CPU). We report accuracy, precision, recall, F1, and trust score separation Δ=T¯clean−T¯poisoned = T_clean- T_poisoned, a threshold-independent measure of discriminability. We add 95% confidence intervals (CIs): Wilson intervals for proportions and a percentile bootstrap (B=20 000B=20\,000) for F1. Ground truth. The clean and poisoned corpora are paired by index, so each query maps to a fixed source document. A sample is labelled poisoned when its aligned source document was modified by a poisoning strategy in Part B, and clean otherwise. This document-level label fits the paired benchmark but is stricter than a retrieval-grounded label, which would count a sample as poisoned only when a poisoned document enters the top-K set. Consequently, if the poisoned source document does not enter the retrieved context, the sample still counts as a missed detection although the agent never observed it. Reported recall is therefore a conservative lower bound of what the agent actually sees. Reproducibility. Generation uses temperature 0.70.7, max_tokens=512=512, and provider-default top-p; NLI (facebook/bart-large-mnli) runs on CPU; retrieval uses FAISS (cosine) over 512512-token chunks (overlap 5050) at K=5K=5. Sampling and poison assignment use a fixed seed (4242), while LLM decoding stays stochastic (quantified by the variance in Section V-D). Calibration fits τ on a held-out clean split (10th percentile) evaluated on disjoint clean/poisoned samples. All LLMs run on the FARMI/Ollama endpoint. V Results This section presents the experimental results. It evaluates the Evaluation Agent’s detection performance, examines the contribution of its components, tests its robustness across different configurations, and assesses its use in a secure-coding RAG setting. V-A Detection Performance and Per-Strategy Hierarchy On the primary TruthfulQA mixed run (Llama 3.3 70B + MiniLM, K=5K=5) the agent attains 91% accuracy (95% CI 83.8 to 95.2%), 100% precision (zero false positives), 40% recall (19.8 to 64.3%), F1 57.1% (25.0 to 80.0%), and Δ=0.225 =0.225, a +7% accuracy gain over the naive baseline. The system is deliberately conservative, preferring missed detections over false alarms. Table I shows a wide detection hierarchy across strategies. Instruction injection is reliably detected in our tested setting (100% recall [79.6 to 100%], F1 96.8%, Δ=0.498 =0.498): override directives trigger deterministic linguistic rules and produce strong intra-document contradiction. Contradiction is moderate (53.3% recall [30.1 to 75.2%]). Subtle manipulation is weak (20% [7.0 to 45.2%]) and entity swap is undetected (0% [0 to 20.4%], Δ=0.053 =0.053): these in-place edits leave no surface artifacts. With only 15 poisoned samples per run, recall CIs are wide; we treat the per-strategy ranking as indicative, though it is consistent across repeats. TABLE I: Per-strategy detection on TruthfulQA (100 samples each; Llama 3.3 70B + all-MiniLM-L6-v2, K=5K=5). Strategy Acc. Prec. Recall F1 Δ Instruction injection 99% 93.8% 100% 96.8% 0.498 Contradiction 92% 88.9% 53.3% 66.7% 0.311 Subtle manipulation 88% 100% 20.0% 33.3% 0.149 Entity swap 85% n/a 0% 0% 0.053 Mixed 91% 100% 40.0% 57.1% 0.225 The dampener explains the extremes: poisoned injection contexts reach mean Ppoison≈0.99P_poison≈ 0.99, consistently activating Eq. (3), whereas entity-swap contexts stay near 0.320.32, below the 0.700.70 trigger, so trust scores barely move. V-B Component Ablation (RQ2) To isolate each signal’s contribution (RQ2), we recompute the trust verdict on the primary run by re-weighting the stored per-sample factuality (F), consistency (C), and poison (P) scores at τ=0.5τ=0.5; no new generation is required. Table I shows that NLI factual verification alone is a weak poison detector (0% recall): by design it scores entailment against the generated answer, not adversarial intent, and reserves strong negative signals for the poison pathway (Section I). Its role is precision and the trust baseline: adding consistency and the poison term raises precision to 100%, and the non-linear dampener lifts recall from 0% to 40%, recovering the full operating point (91% accuracy, 57.1% F1). The poison detector alone gives the highest standalone recall (53.3%, F1 64.0%) but lower precision (80%); fusing all three trades recall for zero false positives – factuality and consistency supply precision, the poison pathway recall. Using the same offline recomputation, we assessed sensitivity to the manually chosen weights: perturbing each of α, β, and γ by up to ±0.10± 0.10 and renormalizing (125 settings) keeps accuracy at 90.0–91.0%, F1 at 54.5–57.1%, and precision at 85.7–100%, with no setting outperforming the defaults. The operating point is thus robust to moderate weight changes; threshold sensitivity is addressed by the ROC analysis and per-LLM calibration (Section V-D). TABLE I: Component ablation of the Trust Index (RQ2): detection on the primary TruthfulQA mixed run (Llama 3.3 70B + MiniLM, K=5K=5, 100 samples), recomputed by re-weighting stored per-sample signals at τ=0.5τ=0.5. Configuration Acc. Prec. Rec. F1 Poison detector only 91.0 80.0 53.3 64.0 NLI only 84.0 0.0 0.0 0.0 NLI + Consistency 84.0 40.0 13.3 20.0 Full Trust Index 87.0 100.0 13.3 23.5 Full + Dampener 91.0 100.0 40.0 57.1 V-C LLM and Embedding Sensitivity Table I reports a 2×22× 2 factorial study (K=3K=3). Two findings stand out. First, LLM choice dominates: both Llama configurations are identical (91%, 100% precision) regardless of embedding model, while both Qwen configurations underperform the naive baseline by 14 points (71%) with precision collapsing to 25 to 28%. The cause is generation style: Qwen’s hedged, verbose answers lower NLI entailment, depressing the mean clean trust score to ≈ 0.64 (vs. ≈ 0.83 for Llama) and producing 21 to 23 false positives out of 85 clean samples. Second, embedding dimensionality is near-irrelevant for a concise-output LLM. Thus τ=0.5τ=0.5 is implicitly calibrated for Llama, motivating per-LLM calibration. TABLE I: 2×22× 2 LLM × embedding grid (TruthfulQA, 100 samples, mixed, K=3K=3). LLM Embedding Acc. Prec. Rec. F1 Δ Llama 3.3 70B MiniLM 91% 100% 40.0% 57.1% 0.240 Llama 3.3 70B Snowflake 91% 100% 40.0% 57.1% 0.188 Qwen 3.5 35B MiniLM 71% 25.0% 46.7% 32.6% 0.161 Qwen 3.5 35B Snowflake 71% 28.1% 60.0% 38.3% 0.199 Naive always-trust 85% n/a n/a n/a n/a V-D Stability: Variance, Threshold-Independence, and Calibration TABLE IV: Per-LLM threshold calibration (TruthfulQA mixed, K=5K=5). τ fit on clean scores only; evaluated on held-out clean + poisoned. ROC-AUC is threshold-independent. LLM τ Acc. Prec. Rec. F1 ROC-AUC Llama 3.3 70B 0.50 84.1 96.1 40.8 57.3 0.807 0.71 83.0 71.9 57.5 63.9 Qwen 3.5 35B 0.50 65.5 38.4 53.3 44.7 0.728 0.43 74.5 51.4 40.0 45.0 Mistral 7B Instruct 0.50 79.3 66.7 40.0 50.0 0.789 0.58 74.1 50.0 53.3 51.6 Run-to-run variance. We re-ran each configuration 2 to 5 times with independent generations. Despite LLM sampling stochasticity, detection metrics are highly stable (Table V): the primary mixed configuration scores 90.6±0.5%90.6±0.5\% accuracy and 56.1±1.3%56.1±1.3\% F1 over five repeats, and injection is invariant at 99.0±0.0%99.0±0.0\%. The low variance indicates that the agent’s verdicts are driven by the retrieved evidence rather than by the surface wording of any single generation. TABLE V: Run-to-run variance: mean± over R independent repeats (TruthfulQA, Llama 3.3 70B + MiniLM, K=5K=5, 100 samples/run). Strategy R Acc. (%) Recall (%) F1 (%) Instruction injection 2 99.0± 0.0 100.0± 0.0 96.8± 0.0 Contradiction 2 92.5± 0.5 53.3± 0.0 68.1± 1.4 Subtle manip. 2 88.0± 0.0 20.0± 0.0 33.3± 0.0 Entity swap 2 85.0± 0.0 3.3± 3.3 5.9± 5.9 Mixed 5 90.6± 0.5 40.0± 0.0 56.1± 1.3 Threshold-independent performance. The fixed τ=0.5τ=0.5 operating point understates the Trust Index. Pooling the mixed-strategy runs, it attains a Receiver Operating Characteristic Area Under the Curve (ROC-AUC) of 0.810.81 (Llama), 0.790.79 (Mistral 7B), and 0.730.73 (Qwen). All three sit well above chance, so Qwen’s weak accuracy at τ=0.5τ=0.5 is a thresholding artifact, not an absence of signal. Per-LLM calibration. We also add a third LLM, Mistral 7B Instruct, which reaches 87%87\% accuracy at the default τ=0.5τ=0.5 on the primary K=5K=5 MiniLM mixed run, far above Qwen 3.5 35B (69%69\% in the same setting; Table I reports 71%71\% for Qwen at K=3K=3) despite being five times smaller. This indicates the Trust Index’s performance is governed by generation style (Qwen’s hedged phrasing depresses NLI entailment) rather than model scale. Fitting τ per LLM as in Section IV and evaluating on disjoint samples (Table IV) restores baseline-competitive accuracy: Qwen rises from 65.5%65.5\% to 74.5%74.5\% by cutting false positives. The three LLMs require different optimal thresholds (τ⋆=0.71τ =0.71, 0.580.58, 0.430.43 for Llama, Mistral, and Qwen), confirming that τ is LLM-specific. Absolute accuracies in Table IV are lower than in Table V because the calibration split is deliberately poison-enriched; the meaningful comparison is before vs. after calibration on the same split. V-E Retrieval Depth and Overhead Increasing retrieval depth from K=3K=3 to K=5K=5 leaves the mixed-strategy outcome identical (91% accuracy, 100% precision, 40% recall, F1 57.1%; Δ changes →0.2250.240\!→\!0.225, within noise). The undetected contexts are entity-swap and subtle attacks that leave no textual signal at any depth, so additional documents add no poison evidence. Detection is thus retrieval-depth-invariant, with the recall ceiling set by attack difficulty; K=3K=3 achieves the same result at roughly half the NLI cost. The evaluation adds ≈ 14.7 s per sample (≈ 17× overhead over baseline RAG), dominated by up to 20 CPU NLI passes at K=5K=5. This suits batch settings rather than interactive use; Graphics Processing Unit (GPU) inference is projected to cut evaluation below 2 s. The added cost grows linearly with K and is independent of corpus size, since only the retrieved set is analyzed; in latency-sensitive deployments the agent can run as an asynchronous or batch-mode gate (Section VI). V-F Cross-Dataset Generalization On FEVER, the full 100-sample run reaches 73% accuracy and underperforms the 85% baseline by 12 points, with separation collapsing to Δ=0.035 =0.035. Although enrichment supplies adequate NLI context, the clean/poisoned trust distributions overlap, indicating that the Trust Index does not generalize out-of-the-box and requires dataset-specific calibration. This negative result bounds the external validity of the TruthfulQA findings and motivates per-domain calibration. V-G Use Case: Secure-Coding RAG Assistant in the SDLC To ground the agent in a software-engineering setting, we apply it to a secure-coding assistant: a RAG-based coding/code-review aid that retrieves from a curated knowledge base of 40 secure-coding rules drawn from the OWASP Top 10 and CWE (e.g., parameterized queries for SQL injection, adaptive password hashing, Transport Layer Security (TLS) configuration). A developer query (“How should passwords be stored?”) retrieves guidance that the Evaluation Agent screens before the LLM emits a recommendation, so unsafe guidance can be flagged rather than acted upon. We poison 30% of the rules with security-specific versions of the five strategies: instruction injection appends a spurious “CORRECTION:” directive; contradiction appends a “common misconception” negation; entity swap alters an identifier or value; and subtle manipulation adds misleading qualifiers. TABLE VI: Secure-coding RAG assistant: poison detection by attack strategy (40 OWASP/CWE secure-coding rules; Llama 3.3 70B + MiniLM, K=5K=5). Strategy Acc. Prec. Rec. F1 Δ Instruction injection 97.5 85.7 100.0 92.3 0.542 Contradiction 85.0 n/a 0.0 0.0 0.156 Subtle manipulation 85.0 n/a 0.0 0.0 0.066 Entity swap 72.5 29.2 58.3 38.9 0.291 Mixed 86.2 100.0 8.3 15.4 0.163 Table VI reports detection by strategy (Llama 3.3 70B, K=5K=5), and the pattern follows the benchmark findings. Injected malicious directives are detected almost perfectly (F1 92.3%, Δ=0.54 =0.54): the agent blocks clear attempts to insert unsafe advice before they reach the developer. Entity swaps are now partially detected (58% recall, versus 0% on open-domain TruthfulQA), because changing a structured identifier or value creates cross-document inconsistencies that NLI signals can detect. However, contradiction and subtle manipulation still evade detection (0% recall): a weakened recommendation that presents an insecure practice as acceptable leaves no surface artifact and passes review. The agent can thus guard against overt unsafe-guidance injection in the SDLC, while subtle semantic weakening requires external world-knowledge verification (future work). VI Discussion Answering the research questions. RQ1 (detection effectiveness): overt poisoning is detected reliably in our tested setting (injection F1 96.8%, mixed accuracy 91% with 100% precision), while in-place edits remain near-undetectable; effectiveness is governed by the visibility of attack artifacts, not by tuning (Section V-A). RQ2 (effect of NLI): NLI alone is a precision instrument, not a detector: the ablation (Section V-B) shows it supplies the trust baseline and zero false positives, while the poison pathway and dampener supply recall. RQ3 (resilience): verdicts are stable across runs, retrieval depths, and embeddings, and carry over to the secure-coding setting, but are bounded by generator style and domain shift (FEVER), which calibration mitigates (Sections V-C to V-G). Together these quantify how far the Security-Reliability Gap of Section I can be closed by an online trust layer. An architectural limit, not a tuning failure. Entity-swap and subtle attacks change facts in place without textual artifacts. No amount of threshold tuning, reweighting, or deeper retrieval recovers them (Sections V-A, V-C); detecting them requires external world knowledge (e.g., a structured knowledge base). The 40% mixed-strategy recall ceiling is therefore intrinsic to surface-signal detection. False-positive spillover. A clean query that retrieves a poisoned neighbor receives elevated poison probability – security-correct (the environment is contaminated) but costly for precision; relevance-weighted aggregation mitigates but does not eliminate it. Deployment in the SDLC. The agent fits as a workflow gate: it can screen a secure-coding knowledge base before indexing and, at ≈ 17 s per query, act as an asynchronous check in code review or continuous integration rather than in the edit loop, with a human reviewing flagged retrievals. Scope and validity considerations. The poisoning strategies are rule-based and cover common attack patterns, but future work should also evaluate stronger optimization-crafted poisoned passages, including collision-style documents [7][20]. This study measures detection of poisoned context before generation, while measuring whether the LLM adopts injected misinformation, i.e., attack success rate, requires end-to-end evaluation. Relatedly, the document-level ground truth counts a sample as missed even when the poisoned document never enters the top-K context (Section IV), so reported recall is a conservative lower bound. The experiments use controlled sample sizes, with 15 poisoned samples per run, and the secure-coding study focuses on one curated domain. Because NLI scores depend on generation style, per-LLM calibration requires a small labeled clean set for each model. VII Conclusion and Future Work We presented an Evaluation Agent that adds a trust layer to RAG by combining NLI-based factual verification, multi-signal poison detection, and a Trust Index with a non-linear dampener. The agent detects clear poisoning, including 100% recall for instruction injection and 100% precision on TruthfulQA, and the secure-coding use case shows it can help block injected unsafe advice in the SDLC. The results also show important boundaries: in-place edits are harder to detect, NLI-based trust scoring depends on the generating LLM, and the FEVER result shows that dataset-specific calibration is important for generalization. Future work includes evaluating stronger poisoning attacks, calibrating the Trust Index across LLMs, adding world-knowledge sources such as, Wikidata, to address entity-swap attacks, accelerating evaluation with GPUs, and measuring misinformation adoption, or attack success rate, beyond detection. Three further directions follow from the reviewers’ suggestions. First, end-to-end attack evaluation should measure whether poisoned guidance changes generated artifacts, e.g., whether a corrupted secure-coding rule yields vulnerable code or altered developer decisions; this requires code-level security oracles, such as static analyzers, and user studies. Second, evaluation on larger real-world RAG applications with heterogeneous, continuously updated corpora, where calibration data are scarce. Third, jointly optimizing the weights, dampener parameters, and threshold per deployment, extending the perturbation analysis of Section V-B. The implementation, attack generator, and experimental artifacts are available for reproducibility [10]. Acknowledgment This work was partly supported by the AI Native Software Engineering (ANSE) project, funded by Business Finland. The authors declare no conflicts of interest. References [1] A. Vaswani et al., “Attention is all you need,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017, p. 5998–6008. [2] Z. Ji et al., “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, Art. no. 248, p. 1–38, 2023. [3] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 33, 2020, p. 9459–9474. [4] Y. Gao et al., “Retrieval-augmented generation for large language models: A survey,” arXiv:2312.10997, 2024. [5] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “RAGAs: Automated evaluation of retrieval augmented generation,” in Proc. 18th Conf. Eur. Chapter ACL (Demos), 2024, p. 150–158. [6] J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia, “ARES: An automated evaluation framework for retrieval-augmented generation systems,” in Proc. NAACL, 2024, p. 338–354. [7] W. Zou, R. Geng, B. Wang, and J. Jia, “PoisonedRAG: Knowledge corruption attacks to retrieval-augmented generation of large language models,” in Proc. 34th USENIX Security Symp., 2025, p. 3827–3844. [8] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, “On faithfulness and factuality in abstractive summarization,” in Proc. 58th Annu. Meeting ACL, 2020, p. 1906–1919. [9] O. Honovich et al., “TRUE: Re-evaluating factual consistency evaluation,” in Proc. NAACL, 2022, p. 3905–3920. [10] GPT Laboratory, “TrustworthyRAG: Implementation, attack generator, and experimental artifacts,” 2026. [Online]. Available: https://github.com/GPT-Laboratory/TrustworthyRAG [retrieved: August, 2026] [11] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, “REALM: Retrieval-augmented language model pre-training,” in Proc. 37th Int. Conf. Machine Learning (ICML), 2020, p. 3929–3938. [12] G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,” in Proc. 16th Conf. Eur. Chapter ACL (EACL), 2021, p. 874–880. [13] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to retrieve, generate, and critique through self-reflection,” in Int. Conf. Learn. Represent. (ICLR), 2024. [14] V. Karpukhin et al., “Dense passage retrieval for open-domain question answering,” in Proc. EMNLP, 2020, p. 6769–6781. [15] S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring how models mimic human falsehoods,” in Proc. 60th Annu. Meeting ACL, 2022, p. 3214–3252. [16] P. Manakul, A. Liusie, and M. J. F. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” in Proc. EMNLP, 2023, p. 9004–9017. [17] S. Min et al., “FActScore: Fine-grained atomic evaluation of factual precision in long form text generation,” in Proc. EMNLP, 2023, p. 12076–12100. [18] K. Greshake et al., “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. 16th ACM Workshop Artif. Intell. Secur., 2023, p. 79–90. [19] N. Carlini et al., “Poisoning web-scale training datasets is practical,” in Proc. IEEE Symp. Security and Privacy (S&P), 2024, p. 407–425. [20] Z. Zhong, Z. Huang, A. Wettig, and D. Chen, “Poisoning retrieval corpora by injecting adversarial passages,” in Proc. EMNLP, 2023, p. 13764–13775. [21] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” in Findings of ACL: EMNLP, 2021, p. 3784–3803. [22] C. Xiang et al., “Certifiably robust RAG against retrieval corruption,” arXiv:2405.15556, 2024. [23] Z. Cheng et al., “Secure retrieval-augmented generation against poisoning attacks,” arXiv:2510.25025, 2025. [24] T. Tuomisto and L. Harjumaa, “Ethical considerations of using generative AI in software development,” in Proc. 20th Int. Conf. Software Engineering Advances (ICSEA), 2025, p. 53–59. [25] K. Karhu and J. Kasurinen, “Barriers and enablers of AI adoption in software testing: A secondary study,” in Proc. 20th Int. Conf. Software Engineering Advances (ICSEA), 2025, p. 46–52. [26] M. Jäntti, J. Koponen, and M. Aho, “How digital experiments support sustainability in a forest machine operator company: A case study,” in Proc. 20th Int. Conf. Software Engineering Advances (ICSEA), 2025, p. 1–6. [27] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions,” in Proc. IEEE Symp. Security and Privacy (S&P), 2022, p. 754–768. [28] N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with AI assistants?” in Proc. ACM SIGSAC Conf. Computer and Communications Security (CCS), 2023, p. 2785–2799. [29] OWASP Foundation, “OWASP Top 10 for LLM Applications 2025,” OWASP GenAI Security Project, 2025. [Online]. Available: https://genai.owasp.org/llm-top-10/ [retrieved: August, 2026] [30] OWASP Foundation, “OWASP Top 10 (2021).” [Online]. Available: https://owasp.org/Top10/2021/ [retrieved: August, 2026] [31] MITRE, “Common Weakness Enumeration (CWE).” [Online]. Available: https://cwe.mitre.org/ [retrieved: August, 2026] [32] J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: A large-scale dataset for fact extraction and verification,” in Proc. NAACL, 2018, p. 809–819. [33] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. EMNLP-IJCNLP, 2019, p. 3982–3992.