Paper deep dive
DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense
Trung V. Phan, Tri Gia Nguyen, Thomas Bauschert
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Advanced Persistent Threats (APTs) are difficult to detect and interpret due to their multi-stage and stealthy nature. While recent autonomous defense systems leverage provenance graphs and learning-based models for detection and mitigation, their outputs remain largely machine-oriented and difficult for analysts to interpret. Large language models (LLMs) offer a promising interface for report generation, but often produce hallucinated or weakly grounded content. In this paper, we propose DeepFaith, an evidence-grounded framework for faithful incident reporting in multi-stage APT defense. DeepFaith transforms structured outputs from autonomous defense and explainability modules into natural-language reports that are explicitly aligned with underlying system evidence. The framework integrates a unified evidence representation, evidence-grounded prompting, faithfulness-aware generation, and post-generation verification to ensure that all generated statements are supported. Experiments in a realistic enterprise testbed demonstrate that DeepFaith improves faithfulness from 0.68 to 0.92, reduces unsupported claims from 0.32 to 0.08, and increases temporal consistency from 0.6 to 0.88, while maintaining concise reports and lower error rates than existing template-based and LLM-based solutions. These results show that evidence-grounded generation enables reliable, interpretable, and actionable reporting for security operations centers.
Tags
Links
- Source: https://arxiv.org/abs/2607.24348v1
- Canonical: https://arxiv.org/abs/2607.24348v1
Trouble viewing inline? Open PDF directly →
Full Text
49,523 characters extracted from source content.
Expand or collapse full text
DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense Trung V. Phan2, Tri Gia Nguyen4 and Thomas Bauschert2 Email: trung.phan-van@etit.tu-chemnitz.de, tri@ieee.org, thomas.bauschert@etit.tu-chemnitz.de Abstract Advanced Persistent Threats (APTs) are difficult to detect and interpret due to their multi-stage and stealthy nature. While recent autonomous defense systems leverage provenance graphs and learning-based models for detection and mitigation, their outputs remain largely machine-oriented and difficult for analysts to interpret. Large language models (LLMs) offer a promising interface for report generation, but often produce hallucinated or weakly grounded content. In this paper, we propose DeepFaith, an evidence-grounded framework for faithful incident reporting in multi-stage APT defense. DeepFaith transforms structured outputs from autonomous defense and explainability modules into natural-language reports that are explicitly aligned with underlying system evidence. The framework integrates a unified evidence representation, evidence-grounded prompting, faithfulness-aware generation, and post-generation verification to ensure that all generated statements are supported. Experiments in a realistic enterprise testbed demonstrate that DeepFaith improves faithfulness from 0.68 to 0.92, reduces unsupported claims from 0.32 to 0.08, and increases temporal consistency from 0.6 to 0.88, while maintaining concise reports and lower error rates than existing template-based and LLM-based solutions. These results show that evidence-grounded generation enables reliable, interpretable, and actionable reporting for security operations centers. Index Terms: Advanced Persistent Threat (APT), Autonomous Cyber Defense, Explainable AI (XAI), Large Language Models (LLMs), Security Operations Centers (SOCs). I Introduction Advanced Persistent Threats (APTs) [21] represent one of the most challenging classes of cyber attacks due to their stealthy, multi-stage, and long-term nature. Unlike conventional attacks, APTs unfold over extended periods, traverse multiple hosts, and progress through stages such as reconnaissance, initial compromise, lateral movement, and data exfiltration. Detecting and mitigating such attacks requires not only analyzing large volumes of heterogeneous telemetry data, but also reasoning over complex causal dependencies and temporal evolution across the system [30]. Recent research has therefore shifted toward provenance-based and learning-driven approaches that model system behavior as structured graphs and leverage deep learning for detection [20]. Building on these advances, autonomous cyber defense frameworks have been proposed to perform adaptive mitigation in real time [17]. In particular, our prior stage-aware system DeepStage [18] combines graph neural networks (GNNs) with reinforcement learning (RL) to infer attack progression and select defense actions. In addition, our prior complementary work DeepXplain [19] introduces explainability mechanisms that extract structural, temporal, and decision-level evidence from these models. While these approaches improve detection accuracy and policy effectiveness, their outputs remain largely machine-centric, consisting of latent embeddings, probability distributions, and explanation signals. As a result, human analysts in security operations centers (SOCs) must still manually interpret and synthesize this information, slowing response. At the same time, large language models (LLMs) have emerged as a promising tool for assisting SOC workflows, including alert triage, log analysis, and incident reporting [9, 10]. However, recent studies show that unconstrained LLMs often produce hallucinated or unverifiable content, especially in complex and safety-critical domains [12]. In the context of cyber defense, such errors can lead to incorrect conclusions and inappropriate mitigation actions. This limitation suggests that LLMs should not be used as standalone reasoning engines, but rather as communication layers that translate verified evidence into human-readable outputs. These observations reveal a fundamental gap: while upstream systems can detect, explain, and respond to APT attacks, there is no principled mechanism to convert their outputs into concise, faithful, and analyst-friendly reports. Existing reporting approaches are either template-based, which lack flexibility, or LLM-based, which lack grounding and reliability [28, 11]. Bridging this gap is critical for enabling effective human–AI collaboration in SOC environments. To address this challenge, we propose DeepFaith, a framework for evidence-grounded incident report generation. DeepFaith takes structured outputs from autonomous defense and explainability modules and transforms them into natural-language reports that are both informative and verifiably supported by evidence. The key idea is to treat report generation as a constrained conditional generation problem, where the language model is guided by structured evidence and explicitly evaluated for faithfulness. DeepFaith introduces three main contributions. First, we define a unified evidence representation that integrates stage estimation, provenance structure, temporal information, and action rationale into a common interface. Second, we propose a faithfulness-aware generation framework incorporating evidence-grounded prompting, faithfulness regularization, temporal consistency constraints, and confidence-aware expression. Third, we design a verification and regeneration mechanism that evaluates reports against underlying evidence and filters unsupported outputs. We evaluate DeepFaith in a realistic enterprise testbed building on our prior DeepStage [18] and DeepXplain [19] frameworks, using multi-stage APT scenarios generated via an adversarial emulation framework [6]. Results show that DeepFaith significantly improves faithfulness, reduces hallucination, and enhances temporal alignment compared to template-based and LLM-based baselines. I Related Work I-A Autonomous APT Detection and Defense Advanced Persistent Threats (APTs) are difficult to detect due to their long duration, multi-host propagation, and weak local indicators. As a result, research has moved beyond signature-based methods toward graph-based and learning-driven approaches that capture causal dependencies and temporal evolution. Surveys highlight that modern APT defense increasingly relies on integrated telemetry, attack-stage reasoning, and adaptive response [2, 30]. Provenance graphs have emerged as a powerful representation for modeling system behavior, preserving both structural and temporal relationships among entities. Systems such as FLASH demonstrate that graph representation learning over provenance data can significantly improve intrusion detection performance in enterprise environments [21]. However, practical deployment remains challenging due to scalability and attribution limitations [4]. Reinforcement learning (RL) has been widely explored for adaptive cyber defense, enabling sequential decision-making under uncertainty in dynamic environments [16, 25]. Recent work incorporates risk-aware or entity-centric state representations for effective mitigation planning [13, 23]. More recently, our prior stage-aware framework DeepStage [18] integrates provenance graphs with attack-stage estimation to guide hierarchical RL policies. While these approaches improve defense effectiveness, their outputs remain largely machine-oriented, limiting direct usability for human analysts. I-B Explainability for Reinforcement Learning and Graph-Based Security Interpretability in sequential decision-making has motivated increasing interest in explainable reinforcement learning (XRL). Existing approaches focus on feature attribution, policy summarization, and explanation generation, but are often loosely coupled with decision-making and remain domain-agnostic [15]. This limitation is critical in cyber defense, where high-impact mitigation actions require verifiable justification. Explainability for graph neural networks (GNNs) has also been extensively studied. Methods such as GNNExplainer identify important subgraphs by learning masks over nodes and edges [29], while recent surveys emphasize evaluation criteria such as correctness, robustness, and compactness [14, 7]. Although effective for static prediction tasks, these methods are less suited for dynamic, action-driven environments. To address this, our prior DeepXplain [19] integrates structural, temporal, and policy-level explanations into autonomous defense through evidence alignment and confidence-aware learning. While this improves interpretability at the model level, the outputs remain machine-centric and do not directly support analyst-facing communication. I-C LLMs for Security Operations and Incident Reporting Large language models (LLMs) are increasingly explored for SOC tasks such as alert triage, log analysis, and incident reporting. Recent surveys highlight their potential for workflow acceleration, while emphasizing risks related to hallucination, inconsistency, and unverifiable reasoning [9, 10]. In particular, Kramer et al. show that fully autonomous LLM-based incident summaries often omit key details and introduce factual errors, whereas human-in-the-loop approaches yield more reliable results [12]. This suggests that LLMs are better suited as communication layers rather than standalone reasoning systems. I-D Grounded Generation from Structured Evidence A growing body of work studies text generation from structured inputs such as databases and knowledge graphs. These approaches emphasize not only fluency, but also faithfulness, schema adherence, and execution-grounded correctness [28, 11]. Such considerations are particularly important in SOC environments, where unsupported claims can lead to operational risk. However, existing grounded-generation methods are largely domain-agnostic and do not account for the specific characteristics of cyber defense outputs, such as stage distributions, provenance subgraphs, temporal evidence, and action rationales. Conversely, prior cyber-defense research focuses on detection and explanation, but not on grounded natural-language reporting. I-E Research Gap In summary, prior work shows that (i) provenance-based and stage-aware models, including our prior DeepStage framework, provide a strong foundation for APT defense [21, 18], (i) explainability methods, including our prior DeepXplain framework, expose structural and temporal evidence but remain machine-centric [15, 14, 19], and (i) LLMs are effective for analyst-facing communication only when properly grounded [12, 9]. Therefore, DeepFaith is designed at this intersection, enabling faithful and interpretable report generation from structured defense outputs. Figure 1: Data and control flow of the proposed DeepFaith framework building upon the DeepStage [18] and the DeepXplain [19] solutions. I System Model I-A Autonomous Defense and Explanation Pipeline We build on our prior APT defense architecture DeepStage [18] together with our prior explainability pipeline DeepXplain [19]. As shown in Fig. 1, enterprise telemetry from host and network monitoring is transformed into a dynamic provenance graph Gt=(Vt,Et)G_t=(V_t,E_t) at time t, where nodes represent system entities (e.g., processes, files, sockets) and edges encode causal interactions such as execution, access, and communication. This representation preserves structural and temporal dependencies essential for modeling multi-stage APT behavior in the enterprise network environment. A graph neural network encodes GtG_t into an embedding gt=fGNN(Gt)g_t=f_GNN(G_t), summarizing system interactions in a compact latent vector. Temporal dynamics are captured by a recurrent stage estimator, pt=fLSTM(g1,…,gt),p_t=f_LSTM(g_1,…,g_t), where pt∈ℝKp_t ^K denotes the probability distribution over K attack stages. To handle partial observability, a belief state btb_t is maintained as bt=fLSTM(bt−1,ot),ot=[gt,pt,at−1],b_t=f_LSTM(b_t-1,o_t), o_t=[g_t,p_t,a_t-1], which integrates current observations and past actions. A policy then selects a mitigation action at∼πθ(a∣bt)a_t _θ(a b_t) for adaptive defense [18]. To improve interpretability, an explainability module [19] operates on the system state t=Gt,gt,pt,bt,atX_t=\G_t,g_t,p_t,b_t,a_t\ and produces an explanation signal et=(Gt∗,It,At),e_t=(G_t^*,I_t,A_t), where Gt∗G_t^* is a salient subgraph capturing critical interactions, ItI_t identifies influential time steps, and AtA_t provides the rationale for action selection. These components expose structural, temporal, and decision-level evidence for downstream use. I-B Evidence Representation The outputs of the defense and XAI modules are heterogeneous and not directly suitable for reporting. To bridge this gap, we define a unified evidence representation: ℰt=pt,bt,at,Gt∗,It,At.E_t=\p_t,b_t,a_t,G_t^*,I_t,A_t\. (1) This representation aggregates the key information required to describe system state and defense decisions: ptp_t encodes stage probabilities, btb_t captures latent context, ata_t specifies the selected action, Gt∗G_t^* provides structural evidence, ItI_t captures temporal progression, and AtA_t explains the action rationale. As such, ℰtE_t serves as a structured interface between the autonomous defense system and the DeepFaith framework. IV Problem Formulation Given structured evidence ℰtE_t, the goal is to generate a natural-language incident report yty_t that accurately reflects the system state and defense decision: yt∼P(y∣ℰt).y_t P(y _t). (2) The report yty_t consists of three components: (i) an incident summary describing the attack stage and risk level, (i) an attack timeline capturing key events, and (i) a mitigation rationale explaining the selected action. Unlike standard text generation, outputs must remain grounded in ℰtE_t, ensuring that all statements are supported by model-derived evidence. IV-A Faithfulness Constraint To enforce grounding, we define a faithfulness function measuring alignment between yty_t and ℰtE_t. Let ϕ(yt)φ(y_t) extract atomic claims from the report, and let ψ(ℰt)ψ(E_t) denote evidence-supported facts derived from: stage information from ptp_t, events from Gt∗G_t^*, timestamps from ItI_t, action rationale from AtA_t. A claim c∈ϕ(yt)c∈φ(y_t) is supported if it matches a fact in ψ(ℰt)ψ(E_t). The faithfulness score is: F(yt,ℰt)=1|ϕ(yt)|∑c∈ϕ(yt)[c∈ψ(ℰt)].F(y_t,E_t)= 1|φ(y_t)| _c∈φ(y_t)1[c∈ψ(E_t)]. (3) This metric captures the proportion of evidence-supported claims, penalizing unsupported content. IV-B Temporal Consistency The report must preserve the temporal structure of the attack. Let π=(e1,…,en)π=(e_1,…,e_n) denote the generated event sequence. Temporal consistency is defined with respect to key timestamps ItI_t: Ctime=1n∑i=1n[timestamp(ei)∈It].C_time= 1n _i=1^n1[timestamp(e_i)∈ I_t]. (4) Chronological ordering must satisfy timestamp(ei)≤timestamp(ei+1)timestamp(e_i) (e_i+1), and stage progression should not regress over time. These constraints ensure coherent and evidence-aligned timelines. IV-C Confidence-Aware Expression Let ct=maxkpt(k)c_t= _kp_t^(k) denote the stage confidence. This value regulates linguistic expression: high confidence permits assertive statements, while lower confidence encourages cautious phrasing. A mapping ut=fconf(ct)u_t=f_conf(c_t) is incorporated into prompting to align language with model uncertainty. IV-D Optimization Objective The objective combines generation quality with grounding constraints: ℒ=ℒgen+λ1(1−F(yt,ℰt))+λ2ℒcompact,L=L_gen+ _1(1-F(y_t,E_t))+ _2L_compact, (5) where λ1 _1 and λ2 _2 balance fluency, faithfulness, and conciseness. The generation loss is: ℒgen=−∑i=1|yt|logPθ(yi∣y<i,t),L_gen=- _i=1^|y_t| P_θ(y_i y_<i,P_t), (6) and the compactness term penalizes excessive length: ℒcompact=max(0,|yt|−Lmax).L_compact= (0,|y_t|-L_ ). (7) This formulation encourages reports that are accurate, interpretable, and concise. V Design of DeepFaith Framework In this section, we introduce the DeepFaith framework designed to transform structured evidence ℰtE_t into a human-readable incident report yty_t that is informative and faithful to the system state. As illustrated in Fig. 1, the framework acts as a module bridging machine-level outputs and analyst-facing reports. The transformation from ℰtE_t to yty_t is non-trivial because the input combines graph structures (Gt∗G_t^*), probability distributions (ptp_t), temporal indices (ItI_t), and action explanations (AtA_t). To address this heterogeneity, DeepFaith decomposes report generation into four phases: • Evidence serialization: which converts structured evidence into normalized textual fields; • Evidence-grounded prompt construction: which organizes serialized evidence into a constrained prompt; • Faithfulness-aware language generation: which produces an initial report while enforcing grounding and consistency constraints; • Verification and regeneration: which evaluates the generated report against the structured evidence and regenerates outputs that do not satisfy faithfulness requirements. V-A Evidence Serialization The first phase converts the structured evidence ℰt=pt,bt,at,Gt∗,It,AtE_t=\p_t,b_t,a_t,G_t^*,I_t,A_t\ into a normalized intermediate representation ℰ~t=(ℰt) E_t=S(E_t), where (⋅)S(·) is a deterministic serialization function. The purpose of serialization is to transform heterogeneous inputs into semantically meaningful textual fields that can be consumed by a language model. Formally, the serialized representation can be expressed as: ℰ~t=st,τt,γt,αt, E_t=\s_t, _t, _t, _t\, (8) where sts_t, τt _t, γt _t, and αt _t denote stage, temporal, structural, and action-level evidence fields, respectively. This step is essential because it creates a stable and interpretable interface between the autonomous defense model and the language model. It also ensures that prompting is based on normalized evidence rather than raw heterogeneous outputs. V-B Evidence-Grounded Prompt Construction The serialized evidence ℰ~t E_t is then converted into a structured prompt t=(ℰ~t)P_t=P( E_t), which serves as the conditioning context for the language model. The prompt contains three main components: • Task instruction: which specifies that the model must generate an incident report; • Evidence fields: which provide the serialized stage, temporal, structural, and action information; • Grounding constraints: which explicitly instruct the model to avoid unsupported claims and remain faithful to the provided evidence. Formally, the prompt can be represented as: t=[ℐ,st,τt,γt,αt],P_t=[I,s_t, _t, _t, _t], (9) where ℐI denotes the instruction template. This prompt design imposes a strong inductive bias on the language model: rather than generating free-form text from loosely related context, the model is explicitly guided to produce outputs grounded in the evidence fields. V-C Faithfulness-Aware Language Generation Given the structured prompt tP_t, DeepFaith generates the incident report through a conditional language model: yt∼Pθ(y∣t),y_t P_θ(y _t), (10) where θ denotes the model parameters. To align language generation with the optimization objective defined in Equation (5), DeepFaith incorporates three complementary mechanisms. V-C1 Faithfulness Regularization The faithfulness loss is defined as: ℒfaith=1−F(yt,ℰt),L_faith=1-F(y_t,E_t), (11) which penalizes claims that are not supported by the evidence. This term explicitly constrains generated text to remain consistent with the underlying model state. As a result, the report is encouraged to reflect actual system behavior rather than unsupported inferences. V-C2 Confidence-Aware Conditioning The stage confidence score ct=maxkpt,kc_t= _kp_t,k is incorporated into the prompt: t←t∪ct.P_t _t∪\c_t\. (12) This allows the language model to calibrate its phrasing according to uncertainty. For example, high-confidence situations may justify assertive wording, whereas low-confidence situations should lead to cautious descriptions. This helps align generated language with the model’s epistemic uncertainty. V-C3 Length Regularization To ensure concise and actionable reports, DeepFaith employs a compactness loss, as defined in Equation (7), to penalize outputs whose length exceeds a predefined budget LmaxL_ , where |yt||y_t| denotes the length of the generated report. This constraint is particularly important in SOC environments, where analysts typically prefer short, information-dense summaries. By limiting unnecessary verbosity, the model is encouraged to produce focused reports that highlight only the most relevant information. V-D Verification and Regeneration At inference time, the incident report is generated by maximizing the conditional likelihood under the language model: yt=argmaxyPθ(y∣t),y_t= _yP_θ(y _t), (13) where tP_t is the evidence-grounded prompt and PθP_θ denotes the parameterized language model. In practice, this maximization can be approximated using decoding strategies such as beam search or nucleus sampling [9]. To further improve reliability, DeepFaith incorporates a post-generation verification step that evaluates the faithfulness score F(yt,ℰt)F(y_t,E_t), which quantifies the extent to which the generated report is supported by the structured evidence ℰtE_t. If the faithfulness score falls below a predefined threshold τF∈[0,1] _F∈[0,1], i.e., F(yt,ℰt)<τF(y_t,E_t)< _F, the generated report is flagged for regeneration, as indicated by the red dashed feedback loop in Fig. 1. The threshold τF _F controls the strictness of the verification process and can be tuned according to application requirements. This mechanism acts as a safeguard against unsupported or hallucinated content, ensuring that only reliable and evidence-grounded reports are presented to analysts, which is critical in safety-sensitive SOC environments. V-E Human-AI Collaboration in SOC DeepFaith reports support SOC workflows such as triage, mitigation, and investigation. By providing structured summaries, timelines, and rationales, the system improves situational awareness and reduces cognitive load. Confidence-aware expressions further help analysts calibrate trust, enabling effective human–AI collaboration. VI Case Study: Realistic APT Lateral Movement VI-A Attack Scenario Overview We consider a stage space with K=7K=7 discrete states, indexed as k∈0,1,2,3,4,5,6k∈\0,1,2,3,4,5,6\, where k=0k=0 denotes the no-attack (benign) state, and k=1k=1 to 66 correspond to six APT stages [6]: reconnaissance, initial compromise, privilege escalation, lateral movement, command-and-control, and data exfiltration, respectively. In this scenario, an attacker initially compromises a user workstation in the LAN zone (Host A) through credential leakage and attempts to move laterally to a database server in the server zone (Host B). The attack leverages credential dumping and remote service execution over SMB, while benign background activities are simultaneously present. VI-B Step-by-Step Attack Execution Step 1: Initial Compromise and Credential Access. The attacker executes a credential dumping script on Host A: • /usr/bin/python3 cred_dump.py which accesses sensitive files such as: • /etc/shadow Step 2: Credential Reuse and Remote Execution. Using the extracted credentials, the attacker initiates: • psexec.py -target HostB -u admin -p ***** This results in SMB authentication and remote execution from Host A (LAN) to Host B (server zone). Step 3: Provenance Graph Construction. The telemetry is fused into a provenance graph GtG_t, where nodes represent processes (python3, psexec), files (/etc/shadow), and network sockets, and edges encode causal interactions across hosts and network zones. Step 4: Stage Estimation and Defense Action. The GNN encodes GtG_t into embedding gtg_t, and the stage estimator produces the following probability distribution over K=7K=7 stages: pt=[0.03, 0.04, 0.05, 0.06, 0.78, 0.03, 0.01]p_t=[0.03,\,0.04,\,0.05,\,0.06,\,0.78,\,0.03,\,0.01]. The confidence score is ct=maxkpt(k)=0.78c_t= _kp_t^(k)=0.78, indicating that the system is highly confident the attack is in the lateral movement stage (k=4k=4). Based on the inferred belief state and the stage estimation, the DRL-based policy πθ _θ selects a defense action ata_t, which in this case corresponds to network interface isolation (a19a_19 in the DeepStage [18]). This action aims to contain the compromised host and prevent further lateral propagation within the enterprise network. Step 5: Explanation Generation. The XAI evidence module extracts: Gt∗G_t^*: a critical subgraph capturing python3 → /etc/shadow → psexec → SMB connection, ItI_t: key timestamps (e.g., 10:32:15 and 10:33:02), AtA_t: rationale recommending network interface isolation. Step 6: Evidence-Grounded Report Generation. DeepFaith constructs prompt tP_t from ℰtE_t and generates: [DeepFaith SOC Report] ---------------------------------------- Stage: Lateral Movement (k=4) Confidence: 0.78 [Timeline] 10:32:15-Credential file access detected. 10:33:02-Remote execution to internal server. [Analysis] Credential reuse and unauthorized lateral movement detected across network zones. [Recommended Action] Isolate the affected host to prevent further spread. ---------------------------------------- Step 7: Faithfulness Verification. The report is decomposed into claims ϕ(yt)φ(y_t) and matched against evidence ψ(ℰt)ψ(E_t). Suppose: F(yt,ℰt)=0.83F(y_t,E_t)=0.83. Since F(yt,ℰt)≥τF=0.8F(y_t,E_t)≥ _F=0.8, the report is accepted. Step 8: SOC Response. The analyst isolates Host A, blocks SMB connections, resets credentials, initiates threat hunting. Note that this behavior corresponds to the Lateral Movement tactic (TA0008) and Remote Services technique (T1021) in the MITRE ATT&CK framework [6]. VII Evaluation Setup VII-A Enterprise Testbed and Data Sources Following our prior DeepStage [18], experiments are conducted in a realistic enterprise testbed segmented into four zones: LAN, DMZ, server, and management. This setup reflects typical enterprise architectures and enables evaluation of multi-stage APT behavior across domains. Host telemetry is collected from endpoint monitoring tools (e.g., Auditd), while network events are captured by Zeek sensors. These data sources are normalized and fused into provenance graphs GtG_t, where nodes represent system entities (e.g., processes, files, sockets) and edges encode causal interactions such as execution and access. To emulate realistic APT campaigns, we use CALDERA-driven adversarial playbooks [6], covering all attack stages from reconnaissance to data exfiltration. Benign background activity is executed concurrently to introduce noise and reflect real SOC environments. The same attack-stage definitions, telemetry intervals, and graph construction settings as our prior DeepStage are used for consistency. Results are averaged over 10 runs to account for stochastic variability. VII-B APT Defense and XAI Setup We follow the pipeline of our prior DeepStage [18] and DeepXplain [19] frameworks. The provenance graph GtG_t is encoded by a GNN into embedding gtg_t. A recurrent estimator produces stage distributions ptp_t, while a DRL agent generates belief states btb_t and defense actions ata_t. The prior DeepXplain module produces structured explanations (Gt∗,It,At)(G_t^*,I_t,A_t), which are integrated into the unified evidence representation ℰtE_t used by DeepFaith. VII-C Language Model and Implementation DeepFaith uses a large language model with structured prompting. We adopt LLaMA-2-13B [24] as the backbone for evidence-grounded report generation. The model is conditioned on serialized evidence to produce faithful and interpretable outputs (see Table I). The serialization module converts Gt∗G_t^* into entity-relation triples and ItI_t into ordered timestamps, forming structured textual inputs. Confidence scores ctc_t are included in prompts to regulate linguistic uncertainty. A verification module computes F(yt,ℰt)F(y_t,E_t) and triggers regeneration if it falls below a threshold, ensuring outputs remain grounded in evidence. Note that in the current implementation, Eq. (5) serves as a design objective, while its components are approximated through structured prompting, confidence-aware conditioning, compact decoding, and post-generation verification rather than end-to-end fine-tuning. Table I: DeepFaith Model and Module Configuration Module Component Setting Verification Faithfulness metric F(yt,ℰt)F(y_t,E_t) Threshold τF=0.8 _F=0.8 Module Regeneration Triggered if F<τF< _F Language Model LLaMA-2-13B [24] Decoding Beam search (B=5B=5) Model Maximum length Lmax=256L_ =256 tokens Prompt Prompt type Structured prompt tP_t Input fields st,τt,γt,αt\s_t, _t, _t, _t\ Construction Confidence input ctc_t appended to prompt Evidence Graph representation Entity-relation triples from Gt∗G_t^* Temporal encoding Ordered timestamps from ItI_t Serialization Action explanation Textual rationale from AtA_t VII-D Baselines We compare the DeepFaith framework against three representative baselines that reflect common approaches for incident report generation in SOC environments. All baselines use the same underlying LLM to ensure fair comparison, differing only in input representation and prompting strategy. VII-D1 Template-Based Reporting This baseline implements a rule-based reporting system using predefined templates. Specifically, structured inputs from ℰtE_t are directly mapped into fixed textual fields. For example, the stage is selected as k^t=argmaxkpt(k) k_t= _kp_t^(k), timestamps are taken from ItI_t, and the selected action ata_t is converted into a predefined description. Similar template-based reporting is widely used in SIEM systems [3]. VII-D2 Vanilla LLM This baseline uses the same LLM backbone (LLaMA-2-13B) but without structured evidence grounding. The model is prompted using raw or minimally processed inputs, i.e., our prior DeepXplain outputs (Gt∗,It,At)(G_t^*,I_t,A_t). No structured serialization or grounding constraints are applied. This baseline follows standard LLM usage for text generation [5]. VII-D3 Chain-of-Thought (CoT) Prompting This baseline extends the Vanilla LLM by incorporating reasoning-oriented prompts. The model first produces a reasoning chain (e.g., identifying suspicious processes, inferring attack stage), followed by a final report. This baseline follows the chain-of-thought prompting paradigm [26]. VIII Results and Analysis VIII-A Quantitative Comparison Fig. 2 summarizes the performance of DeepFaith compared to three representative baselines across key evaluation metrics. We evaluate report quality using four complementary metrics: (i) Faithfulness (F), defined as the proportion of claims in the generated report that are supported by structured evidence ℰtE_t; (i) Unsupported Claim Rate (UCR), measuring the fraction of claims not grounded in ℰtE_t; (i) Temporal Consistency (CtimeC_time), capturing alignment between generated timelines and important timestamps ItI_t; and (iv) Report Length, indicating conciseness. Together, these metrics evaluate correctness, completeness, coherence, and operational usability. DeepFaith achieves the highest faithfulness (F=0.92F=0.92), significantly outperforming Template-Based Reporting (0.720.72), Vanilla LLM (0.680.68), and CoT Prompting (0.750.75). This demonstrates that conditioning generation on structured evidence ℰtE_t, combined with faithfulness-aware constraints, effectively ensures that generated reports remain aligned with model-derived information. In contrast, Vanilla LLM produces fluent but weakly grounded outputs due to reliance on implicit knowledge, while CoT Prompting improves reasoning coherence but lacks mechanisms to enforce consistency with structured evidence. These results highlight that reasoning alone is insufficient without explicit grounding. This improvement is further reflected in the Unsupported Claim Rate (UCR), where DeepFaith achieves the lowest value (0.080.08), compared to Template (0.280.28), Vanilla LLM (0.320.32), and CoT (0.250.25). Since UCR directly measures hallucinated or unsupported content, this reduction confirms the effectiveness of combining structured prompting with post-generation verification. The verification mechanism acts as a safeguard that filters or regenerates outputs that deviate from ℰtE_t, which is critical in safety-sensitive SOC environments. DeepFaith also achieves the highest temporal consistency (Ctime=0.88C_time=0.88), indicating strong alignment between generated timelines and evidence-supported timestamps ItI_t. In contrast, Vanilla LLM (0.600.60) often produces incomplete or misordered event sequences, while CoT Prompting (0.680.68) provides only moderate improvements without explicit temporal grounding. These results show that incorporating temporal evidence directly into the generation process is essential for reconstructing coherent multi-stage attack progression. In terms of conciseness, DeepFaith produces compact reports (130 tokens), comparable to Template-Based Reporting (120 tokens) and substantially shorter than Vanilla LLM (180 tokens) and CoT Prompting (200 tokens). This demonstrates that the compactness constraint effectively balances informativeness and brevity. Unlike template-based methods, which limit expressiveness, DeepFaith maintains rich contextual information while avoiding redundancy, resulting in more information-dense outputs. Figure 2: Performance comparison across four methods. Figure 3: Error type distribution across methods (%). VIII-B Error Type Analysis To analyze limitations of generated reports, we perform error analysis across methods. For each report yty_t, we extract claims ϕ(yt)φ(y_t) and compare them with evidence ψ(ℰt)ψ(E_t) to identify errors affecting faithfulness, temporal consistency, and decision reliability. We define the following error categories: (i) Hallucinated Event: A claim c∈ϕ(yt)c∈φ(y_t) that cannot be matched to any fact in ψ(ℰt)ψ(E_t), indicating unsupported or fabricated content; (i) Temporal Error: An event whose timestamp is inconsistent with ItI_t, including misordered events or missing temporal anchors; (i) Stage Misclassification: The reported attack stage does not correspond to the maximum-probability stage in ptp_t; (iv) Incorrect Action Rationale: The generated explanation of action ata_t does not match the rationale encoded in AtA_t; and (v) Missing Critical Evidence: Key facts from ψ(ℰt)ψ(E_t) (e.g., important entities or events in Gt∗G_t^*) are not reflected in ϕ(yt)φ(y_t). We analyze 200 reports from multi-stage APT scenarios, compute error frequencies, and normalize them to obtain error rates. Fig. 3 summarizes the results. DeepFaith consistently reduces all error types compared to baselines. Hallucinated events drop from 21.4% (Vanilla LLM) and 15.8% (CoT) to 6.1%, confirming that evidence-grounded prompting and verification suppress unsupported claims. Temporal errors are also lower (11.2% vs. 25.8%), showing that incorporating temporal evidence ItI_t improves alignment. DeepFaith further achieves the lowest rate of incorrect action rationales (7.5%), indicating that structured explanations AtA_t guide decision-consistent reporting. Template-based reporting has low hallucination (5.2%) but high missing evidence (22.5%), reflecting limited expressiveness. Vanilla LLM produces more complete but less reliable outputs with high hallucination and temporal inconsistency, while CoT reduces some reasoning errors but still lacks explicit grounding, leading to unsupported or incomplete outputs. Overall, this analysis shows that DeepFaith not only improves aggregate metrics such as faithfulness and temporal consistency, but also systematically reduces specific error types. These improvements directly contribute to more reliable and actionable reports for effective SOC decision-making. Figure 4: Temporal analysis across multi-stage APT progression. VIII-C Visualization of Temporal Metrics To complement Fig. 2, Fig. 4 shows temporal trends of three metrics: Faithfulness (F), the proportion of claims supported by ℰtE_t; Cumulative Unsupported Claims (CUC), the accumulated number of unsupported claims over time; and Evidence Coverage Ratio (ECR), the fraction of evidence in ψ(ℰt)ψ(E_t) reflected in the generated report. DeepFaith maintains consistently high F, indicating stable grounding as attack complexity increases, while Vanilla LLM degrades and CoT shows moderate improvement. DeepFaith also slows the growth of CUC, demonstrating effective control of long-horizon error propagation. In addition, it achieves the highest ECR, capturing more complete structural, temporal, and action-level evidence. In contrast, Template-based reporting omits contextual details, while Vanilla LLM and CoT exhibit inconsistent evidence usage. Overall, DeepFaith provides more reliable reporting across all attack stages. Table I: Ablation Study of DeepFaith Method F↑F UCR ↓ Ctime↑C_time Length w/o Serialization 0.82 0.18 0.78 145 w/o Prompt 0.84 0.16 0.80 142 w/o Faithfulness 0.79 0.21 0.81 138 w/o Confidence 0.90 0.10 0.87 131 w/o Temporal 0.88 0.12 0.74 134 w/o Verification 0.86 0.14 0.84 136 DeepFaith 0.92 0.08 0.88 130 VIII-D Ablation Study We evaluate six ablated variants of DeepFaith by removing one component at a time while keeping the same backbone, scenarios, and decoding setup: (i) w/o Serialization, (i) w/o Prompt, (i) w/o Faithfulness, (iv) w/o Confidence, (v) w/o Temporal, and (vi) w/o Verification. The full model includes all components. Table I shows that all components contribute to performance. Removing serialization or structured prompting reduces faithfulness and increases unsupported claims, highlighting the importance of normalized evidence inputs. Removing faithfulness-aware conditioning leads to the largest drop in F and highest UCR, confirming its key role in suppressing hallucination. Temporal removal causes the largest decline in CtimeC_time, indicating the importance of temporal constraints. Disabling verification degrades all grounding-related metrics, showing that post-generation validation is critical. Confidence-aware conditioning has a smaller but consistent effect on overall quality. Overall, the full DeepFaith model achieves the best trade-off across all metrics. Table I: Illustrative Evaluation of DeepFaith with Compact Open-Source LLM Backbones Backbone F↑F UCR ↓ Ctime↑C_time Length Llama-3.2-3B 0.84 0.16 0.80 122 Qwen2.5-3B 0.88 0.12 0.84 126 Gemma-3-4B-IT 0.89 0.11 0.85 124 Falcon3-7B 0.90 0.10 0.86 127 LLaMA-2-13B 0.92 0.08 0.88 130 VIII-E Evaluation with Compact Open-Source LLM Backbones To evaluate portability, we replace the default backbone with four compact open-source models: Qwen2.5-3B-Instruct [27], Gemma-3-4B-IT [22], Llama-3.2-3B-Instruct [8], and Falcon3-7B-Instruct [1]. All models use the same DeepFaith pipeline, evidence serialization, prompt template, verification threshold, and decoding configuration with identical LmaxL_ . This setting isolates the effect of backbone capacity while keeping the grounding mechanism unchanged. Table I shows a clear scaling trend across model sizes. The LLaMA-2-13B backbone achieves the best performance (F=0.92F=0.92, UCR=0.08=0.08, Ctime=0.88C_time=0.88), while compact models exhibit gradual degradation as model size decreases. Among smaller backbones, Falcon3-7B performs closest to the 13B model (F=0.90F=0.90, UCR=0.10=0.10, Ctime=0.86C_time=0.86), followed by Gemma-3-4B-IT and Qwen2.5-3B-Instruct with moderate drops in faithfulness (0.890.89 and 0.880.88) and slightly higher UCR. Llama-3.2-3B-Instruct shows the lowest performance (F=0.84F=0.84, UCR=0.16=0.16, Ctime=0.80C_time=0.80), indicating reduced capability in maintaining grounding and temporal alignment. Nevertheless, all compact backbones maintain acceptable faithfulness and temporal consistency, suggesting that structured evidence conditioning reduces dependence on model scale. Overall, these results show that while larger models provide incremental gains, DeepFaith achieves reliable performance even with lightweight backbones. This makes the framework suitable for deployment in resource-constrained SOC environments, where local inference, privacy preservation, and reduced computational cost are often important requirements. IX Conclusion This paper presented DeepFaith, an evidence-grounded framework for generating faithful and interpretable incident reports in autonomous APT defense systems. DeepFaith integrates structured evidence, constrained generation, and verification mechanisms to align reports with model-derived information. Experimental results show that DeepFaith consistently achieves higher faithfulness, lower unsupported claim rate, and better temporal consistency than template-based and LLM-based baselines, while producing concise reports. Error analysis further confirms significant reductions in hallucination and temporal inconsistencies. Ablation results demonstrate that grounding, temporal constraints, and verification are key to performance. DeepFaith also maintains strong performance with compact open-source LLMs, indicating that its effectiveness largely stems from structured evidence integration rather than model scale. Overall, DeepFaith enables more reliable and efficient SOC reporting. Future work includes extending DeepFaith toward interactive human-in-the-loop SOC systems, enabling bidirectional communication between analysts and autonomous defense models. Acknowledgment This work has been performed in the framework of the SUSTAINET-Advance project, funded by the German BMFTR (ID:16KIS2280). References [1] E. Almazrouei et al. (2023) The falcon series of open language models. Note: Available on https://arxiv.org/abs/2311.16867 External Links: 2311.16867, Link Cited by: §VIII-E. [2] A. Alshamrani et al. (2019) A survey on advanced persistent threats: techniques, solutions, challenges, and research opportunities. IEEE Communications Surveys & Tutorials 21 (2), p. 1851–1877. Cited by: §I-A. [3] A. Behl et al. (2019) A review of siem systems. Journal of Information Security. Cited by: §VII-D1. [4] T. Bilot et al. (2025) Sometimes simpler is better: a comprehensive analysis of state-of-the-art provenance-based intrusion detection systems. In USENIX Security Symposium, Cited by: §I-A. [5] T. B. Brown et al. (2020) Language models are few-shot learners. NeurIPS. Cited by: §VII-D2. [6] T. M. Corporation (2024) MITRE Caldera: Automated Adversary Emulation Platform. Note: Open-source adversary-emulation system used for breach-and-attack simulation of APT playbooks External Links: Link Cited by: §I, §VI-A, §VI-B, §VII-A. [7] E. Dai et al. (2024) A comprehensive survey on trustworthy graph neural networks: privacy, robustness, fairness, and explainability. International Journal of Automation and Computing. Cited by: §I-B. [8] A. Grattafiori et al. (2024) The llama 3 herd of models. Note: Available on https://arxiv.org/abs/2407.21783 External Links: 2407.21783, Link Cited by: §VIII-E. [9] A. Habibzadeh, F. Feyzi, and R. E. Atani (2025) Large language models for security operations centers: a comprehensive survey. Note: Available on https://arxiv.org/abs/2509.10858 External Links: 2509.10858, Link Cited by: §I, §I-C, §I-E, §V-D. [10] N. O. Jaffal et al. (2025) Large language models in cybersecurity: a survey of applications, vulnerabilities, and defenses. AI 6 (9), p. 216. Cited by: §I, §I-C. [11] P. Jiang et al. (2025-08) Retrieval and structuring augmented generation with large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, p. 6032–6042. External Links: Link, Document Cited by: §I, §I-D. [12] D. Kramer et al. (2025) Integrating large language models into security incident response. In Symposium on Usable Privacy and Security, Cited by: §I, §I-C, §I-E. [13] A. T. Le et al. (2024) Automated apt defense using reinforcement learning and attack graph risk-based situation awareness. In Workshop on Autonomous Cybersecurity, Cited by: §I-A. [14] X. Li et al. (2025) Can graph neural networks be adequately explained? a survey. ACM Computing Surveys. Cited by: §I-B, §I-E. [15] S. Milani et al. (2024) Explainable reinforcement learning: a survey and comparative review. ACM Computing Surveys 56 (7). Cited by: §I-B, §I-E. [16] T. T. Nguyen et al. (2023) Deep reinforcement learning for cyber security. IEEE Transactions on Neural Networks and Learning Systems 34 (8), p. 3779–3795. Cited by: §I-A. [17] T. V. Phan et al. (2022) DeepAir: deep reinforcement learning for adaptive intrusion response in software-defined networks. IEEE Transactions on Network and Service Management 19 (3), p. 2207–2218. External Links: Document Cited by: §I. [18] T. V. Phan et al. (2026) DeepStage: learning autonomous defense policies against multi-stage apt campaigns. In IEEE International Conference on Cyber Security and Resilience (CSR), Note: [Accepted for Publication]. Available on https://arxiv.org/abs/2603.16969 Cited by: §I, §I, Figure 1, §I-A, §I-E, §I-A, §I-A, §VI-B, §VII-A, §VII-B. [19] T. V. Phan et al. (2026) DeepXplain: xai-guided autonomous defense against multi-stage apt campaigns. In IEEE Global Communications Conference (GLOBECOM), Note: [Under Review]. Available on https://arxiv.org/abs/2603.21296 Cited by: §I, §I, Figure 1, §I-B, §I-E, §I-A, §I-A, §VII-B. [20] T. V. Phan et al. (2026) Learning the apt kill chain: temporal reasoning over provenance data for attack stage estimation. In IEEE International Conference on Communications (ICC), Cited by: §I. [21] M. U. Rehman et al. (2024) FLASH: a comprehensive approach to intrusion detection via provenance graph representation learning. In IEEE Symposium on Security and Privacy, Cited by: §I, §I-A, §I-E. [22] G. Team et al. (2024) Gemma: open models based on gemini research and technology. Note: Available on https://arxiv.org/abs/2403.08295 External Links: 2403.08295, Link Cited by: §VIII-E. [23] S. Thompson et al. (2024) Entity-based reinforcement learning for autonomous cyber defence. In Workshop on Autonomous Cybersecurity, p. 56–67. Cited by: §I-A. [24] H. Touvron et al. (2023) Llama 2: open foundation and fine-tuned chat models. Note: Available on https://arxiv.org/abs/2307.09288 External Links: 2307.09288, Link Cited by: §VII-C, Table I. [25] S. Vyas et al. (2025) Towards the deployment of realistic autonomous cyber network defence: a systematic review. ACM Computing Surveys. Cited by: §I-A. [26] J. Wei et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. NeurIPS. Cited by: §VII-D3. [27] A. Yang et al. (2025) Qwen2.5 technical report. Note: Available on https://arxiv.org/abs/2412.15115 External Links: 2412.15115, Link Cited by: §VIII-E. [28] W. Yang et al. (2025) A comprehensive survey on integrating large language models with knowledge bases. Knowledge-Based Systems. Note: Survey on LLM and knowledge integration Cited by: §I, §I-D. [29] R. Ying et al. (2019) GNNExplainer: generating explanations for graph neural networks. In Advances in Neural Information Processing Systems, Cited by: §I-B. [30] B. Zhang et al. (2024) A survey on advanced persistent threat detection: a unified framework, challenges, and countermeasures. ACM Computing Surveys 57 (). Cited by: §I, §I-A.