Paper deep dive
Evaluating the Reliability of Digital Forensic Evidence Discovered by Large Language Model: A Case Study
Jeel Piyushkumar Khatiwala, Daniel Kwaku Ntiamoah Addai, Weifeng Xu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 8:15:24 PM
Summary
This paper proposes a structured framework for evaluating the reliability of digital forensic evidence identified by Large Language Models (LLMs). The methodology involves automating artifact extraction from forensic images, refining data via LLM-driven analysis, and validating results using a Digital Forensic Knowledge Graph (DFKG). The framework utilizes deterministic Unique Identifiers (UIDs) to ensure traceability and chain-of-custody adherence. Evaluated on a 13 GB dataset, the approach achieved over 95% accuracy in artifact extraction and demonstrated robust contextual consistency, addressing challenges in AI-assisted forensic integrity and legal admissibility.
Entities (10)
Relation Signals (5)
University of Baltimore → authored → Evaluating the Reliability of Digital Forensic Evidence Discovered by Large Language Model: A Case Study
confidence 99% · Jeel Piyushkumar Khatiwala School of Criminal Justice College of Public Affairs University of Baltimore
Unique Identifier → supports → Chain-of-Custody
confidence 93% · ensures artifact traceability and evidentiary consistency through deterministic Unique Identifiers (UIDs)
Large Language Model → usedin → Digital Forensic Knowledge Graph
confidence 92% · refines data through LLM-driven analysis, and validates results using a Digital Forensic Knowledge Graph (DFKG)
Digital Forensic Knowledge Graph → ensures → Chain-of-Custody
confidence 90% · strong support of chain-of-custody adherence
Cellebrite → provided → Evidence Extraction Accuracy
confidence 85% · ground truth data from Cellebrite’s Capture-the-Flag (CTF) competitions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The growing reliance on AI-identified digital evidence raises significant concerns about its reliability, particularly as large language models (LLMs) are increasingly integrated into forensic investigations. This paper proposes a structured framework that automates forensic artifact extraction, refines data through LLM-driven analysis, and validates results using a Digital Forensic Knowledge Graph (DFKG). Evaluated on a 13 GB forensic image dataset containing 61 applications, 2,864 databases, and 5,870 tables, the framework ensures artifact traceability and evidentiary consistency through deterministic Unique Identifiers (UIDs) and forensic cross-referencing. We propose this methodology to address challenges in ensuring the credibility and forensic integrity of AI-identified evidence, reducing classification errors, and advancing scalable, auditable methodologies. A comprehensive case study on this dataset demonstrates the framework's effectiveness, achieving over 95 percent accuracy in artifact extraction, strong support of chain-of-custody adherence, and robust contextual consistency in forensic relationships. Key results validate the framework's ability to enhance reliability, reduce errors, and establish a legally sound paradigm for AI-assisted digital forensics.
Tags
Links
- Source: https://arxiv.org/abs/2602.20202v1
- Canonical: https://arxiv.org/abs/2602.20202v1
Trouble viewing inline? Open PDF directly →
Full Text
47,275 characters extracted from source content.
Expand or collapse full text
Evaluating the Reliability of Digital Forensic Evidence Discovered by Large Language Model: A Case Study Jeel Piyushkumar Khatiwala School of Criminal Justice College of Public Affairs University of Baltimore Maryland, USA Jeel.khatiwala@ubalt.edu Daniel Kwaku Ntiamoah Addai School of Criminal Justice College of Public Affairs University of Baltimore Maryland, USA daniel.addai@ubalt.edu Weifeng Xu School of Criminal Justice College of Public Affairs University of Baltimore Maryland, USA wxu@ubalt.edu Abstract—The growing reliance on AI-identified digital evi- dence raises significant concerns about its reliability, particularly as large language models (LLMs) are increasingly integrated into forensic investigations. This paper proposes a structured framework that automates forensic artifact extraction, refines data through LLM-driven analysis, and validates results using a Digital Forensic Knowledge Graph (DFKG). Evaluated on a 13 GB forensic image dataset containing 61 applications, 2,864 databases, and 5,870 tables, the framework ensures artifact traceability and evidentiary consistency through deterministic Unique Identifiers (UIDs) and forensic cross-referencing. We propose this methodology to address challenges in ensuring the credibility and forensic integrity of AI-identified evidence, reducing classification errors, and advancing scalable, auditable methodologies. A comprehensive case study on this dataset demonstrates the framework’s effectiveness, achieving over 95% accuracy in artifact extraction, strong support of chain-of- custody adherence, and robust contextual consistency in forensic relationships. Key results validate the framework’s ability to enhance reliability, reduce errors, and establish a legally sound paradigm for AI-assisted digital forensics. Index Terms—Digital Forensic, Large Language Model, Foren- sic Knowledge Graphs, AI-Driven Forensic Analysis, Legal Ad- missibility of Evidence I. INTRODUCTION The rapid expansion of digital data and the complexity of forensic investigations have created an urgent need for scalable, automated solutions to ensure the reliability, integrity, and legal admissibility of digital evidence [1, 2]. Large lan- guage models (LLMs) show strong potential in this area, of- fering capabilities for processing unstructured data, extracting entities, and performing contextual correlation [3, 4]. However, concerns persist regarding the authenticity, accountability, and forensic soundness of AI-generated outputs [5], stemming from biases in generated artifacts, inconsistent refinements, and the absence of structured validation. A central challenge in existing forensic workflows is the lack of standardized frameworks for evidence discovery, ar- tifact validation, and traceability [5, 6]. Traditional methods often struggle with heterogeneous data such as call logs, This material is based upon work supported in part by the National Science Foundation under Grant No. 2333949 and by the Bureau of Justice Assistance under Award No. 2019-DF-BX-K001. messages, and encrypted databases, leading to incomplete ex- tractions and insufficient context reconstruction. This impacts investigative outcomes, disrupts chain of custody, and raises questions of admissibility. To address these issues, this paper presents a forensic eval- uation framework that automates artifact discovery, extraction, and validation. It integrates advanced data extraction with LLM-driven refinement to improve accuracy and contextual clarity. A Digital Forensic Knowledge Graph (DFKG) models relationships among artifacts to support structured and veri- fiable analysis [7]. The framework uses deterministic Unique Identifiers (UIDs) for artifact traceability, hypothesis testing against ground truth data, and forensic-specific metrics to assess evidentiary reliability. This work addresses the following research questions: • RQ1: How can a standardized framework be imple- mented to evaluate the reliability of forensic evidence refined by LLMs using knowledge graph visualization? • RQ2: How can the reliability of LLM-identified forensic evidence be measured in terms of accuracy, false posi- tives, and artifact coverage? • RQ3: What are the key sources of error or bias in LLM- generated forensic evidence, and how do they impact artifact classification? • RQ4: How can LLM-assisted forensic evidence be vali- dated to ensure traceability and accuracy through cross- referencing with original sources? • RQ5: What best practices can enhance the admissibility and trustworthiness of LLM-derived evidence, focusing on traceability, standardized extraction, and validation workflows? The contributions of this paper are as follows: • An automated forensic framework that unifies data extrac- tion, LLM refinement, and graph-based correlation into a scalable pipeline. • A transformation method that converts heterogeneous forensic databases into LLM-readable, audit-ready CSV format while preserving metadata for chain-of-custody compliance. arXiv:2602.20202v1 [cs.CR] 22 Feb 2026 • A set of forensic-specific evaluation metrics, including Evidence Extraction Accuracy (EEA), Forensic Artifact Precision (FAP), Forensic Artifact Recall (FAR), Knowl- edge Graph Connectivity Accuracy (KGCA), Chain-of- Custody Adherence (CCA), and Contextual Consistency Score (CCS), for assessing AI-generated forensic outputs. • A UID-based tagging system that assigns globally unique identifiers to each artifact, enabling traceability, valida- tion, and auditability. By addressing these challenges, this study advances digital forensic practice with a scalable, reliable, and legally defen- sible framework for AI-assisted investigations. The remainder of the paper is organized as follows: Section I reviews related work; Section I outlines the proposed framework; Section IV presents the case study and results; and Section V concludes with key insights and directions for future research. I. RELATED WORK The application of Artificial Intelligence (AI), particularly large language models (LLMs), in digital forensics has signifi- cantly advanced investigative workflows by automating artifact extraction, contextual analysis, and relationship visualization [3, 4]. These capabilities help address the increasing volume and complexity of digital evidence across platforms. Traditional techniques such as regular expressions (regex) have long supported forensic preprocessing by identifying patterns in unstructured data. However, these approaches often require manual oversight and lack mechanisms to contex- tualize relationships, limiting their scalability for large-scale investigations [2]. In contrast, LLMs can process semantically rich data with greater automation and contextual awareness. Recent studies highlight LLMs’ potential in automating core forensic tasks, including entity lineage tracking and relation- ship identification. Wickramasekara et al. [3] demonstrated im- proved efficiency and traceability through LLM-based work- flows, while specialized models such as ForensicLLM were introduced to increase domain-specific performance [8]. Alongside LLMs, knowledge graphs (KGs) have emerged as powerful tools for modeling relationships between foren- sic artifacts. Xu and Xu [5] proposed the Digital Forensic Knowledge Graph (DFKG) to support hypothesis-driven rea- soning, and Addai et al. [6] explored multi-device relationship discovery. These graph-based frameworks help expose latent connections that traditional tools may overlook. Graph theory continues to contribute to forensic linkage analysis at scale. Karim et al. [7] demonstrated how graph algorithms can uncover critical relational structures. Pan et al. [9] outlined integration strategies for combining LLMs with KGs to support hypothesis generation, and Wickramasekara et al. [10] proposed hybrid models to enhance investigative outcomes in complex environments. Additionally, Gen-AI platforms like ForenSift [11] have in- troduced automation in digital forensics and incident response (DFIR). Built on the LangChain framework, ForenSift accel- erates case analysis and improves artifact coverage through advanced entity correlation. Despite these developments, challenges persist in scaling automated forensic analysis and maintaining contextual accu- racy. Continued integration of LLMs with knowledge graphs, supported by structured forensic frameworks, offers a promis- ing direction for enhancing the reliability and scalability of digital investigations [2, 4]. I. PROPOSED METHODOLOGY We propose a structured framework to assess the reliability and accuracy of digital evidence extracted by large language models (LLMs). This framework ensures evidence integrity for both legal and investigative purposes, providing a holistic method for processing, analyzing, and validating forensic artifacts. As illustrated in Fig. 1, our methodology encom- passes four key steps: (1) identifying forensic artifacts from mobile device databases (the “digital evidence repository”), (2) transforming heterogeneous databases into a structured, LLM-readable CSV format, (3) constructing a Digital Forensic Knowledge Graph (DFKG) to model relationships among artifacts, and (4) applying robust forensic metrics to validate extracted evidence. By measuring forensic reliability at each stage, the framework enables end-to-end traceability and sup- ports legally admissible digital forensic workflows [2, 4]. A. Data Extraction and Preprocessing 1) Artifact Identification: Artifact extraction focuses on mobile device databases, collectively termed the “digital ev- idence repository,” including call logs, messages, applica- tion records, and location data, essential for reconstructing user behavior. File signature analysis identifies and classifies databases based on unique byte patterns, enabling detection even when filenames or extensions are obfuscated. For exam- ple, databases such as contacts2.db (contacts), sms.db (messages), and mmssms.db (multimedia) are extracted from their respective directories. Each is exported as a CSV file with metadata (database name, table name, row number) preserved to ensure integrity and traceability [5]. 2) Data Transformation: To support LLM-based process- ing, the evidence repository is converted into a flat CSV format. This transformation preserves the structural and se- mantic attributes of the original data, ensuring consistency and auditability, both essential for forensic authenticity. Each database table is exported as an individual CSV file containing metadata such as database name, table name, file path, and row number. Flattening relational data simplifies complex structures, resulting in a uniform format optimized for LLM analysis. B. Mathematical Formulations 1) Unique Identifier Generation (UID): A reliable Unique Identifier (UID) scheme ensures data integrity and verifiability, especially when identical databases exist across multiple loca- tions. We generate a UID by applying a cryptographic hash to the Device ID, File Path, and Database Name, then appending the Table Name and Row Number. Truncating the hash to eight characters provides computational efficiency while main- taining uniqueness. The appended components ensure row- level granularity. Fig. 1: Proposed evaluation workflow: identifying artifacts containing digital evidence, transforming the digital evidence repository to an LLM-readable form, constructing the Digital Forensic Knowledge Graph (DFKG), and integrating LLM- refined evidence into the final graph representation. U ID = SHA-256(DeviceID + FilePath + DatabaseName) [:8] +“” + Table + “” + LID (1) where: • DeviceID: Represents the unique identifier of the source mobile device. • FilePath: Indicates the exact location of the database file within the device. • DatabaseName: Denotes the name of the extracted database file. • Table: Identifies the table from which the forensic record is extracted. • LID: Refers to the row number within the table to maintain intra-table uniqueness. a) Example: For instance, consider a WhatsApp database (msgstore.db) extracted from a device with: (DeviceID,A1B2C3D4E5F6G7H8) (FilePath,/data/com.whatsapp/databases/) (DatabaseName,msgstore.db) (Tname,messages) (LID,42) (HashInput,A1B2C3D4E5F6G7H8 + /data/com.w hatsapp/databases/ + msgstore.db) (ComputedHash,788492af8249d22829c49c9...) (ExtractedFirst8Characters,788492af) The computed UID becomes: U ID = 788492af messages42 This process ensures that identical databases from different sources generate distinct, verifiable UIDs. 2) CSV Transformation for LLM Processing: To ensure structured, traceable, and uniquely identifiable forensic data, each database table is converted into a CSV format, preserving metadata such as database name, table name, file path, UID, and row index alongside actual column values. Formally, a table row r ′ is defined as: CSV =(database, table, path, UID, LID)∪(c, r’(c)) | c∈ C (2) where: • database - Name of the database from which the table is extracted. • table - Name of the table from which the row originates. • path - The original file path of the database on the device. • UID - Unique identifier ensuring global distinctiveness. • LID - Row index for intra-table differentiation. • (c, r ′ (c))| c∈ C - Each column-value pair in the row, where C represents the set of column names. Each row r ′ is a mapping r ′ : A→ V , where A is the set of attributes and V their corresponding values: r ′ =(database, db), (table, t), (path, p), (UID, u), (LID, l) ∪(c, r’(c))| c∈ C (3) a) Example: An entry from the UserStore table of core.db (Snapchat) might appear as: (Database,core.db);(Table,UserStore); (FilePath,...m.snapchat.android/databases/); (UID,9f97eac5_UserStore_114);(LID,114); (realval,...x19heisenbergercarro@gmail.com- - 0...) All CSV data is merged into a unified row-based file, with each record represented as a set of attribute-value pairs. To manage data volume, the unified file is partitioned and every sixth row is sampled for reliability checks. Extraneous Android system and configuration metadata are excluded to prioritize user-centric forensic content, improving computational effi- ciency while maintaining integrity for LLM-based refinement and investigation. C. LLM-Assisted Artifact Refinement Once the repository is transformed into CSV format, large language models (LLMs) are applied to enhance the clarity and contextual accuracy of forensic artifacts. The LLM cor- rects inconsistencies, removes obfuscations, and reconstructs missing values. Each artifact remains linked to its source via its UID, enabling reliable consolidation across diverse data sources. Key entities, such as Names, Phone Numbers, Emails, Timestamps, Application Names, Google Search Queries, and MAC Addresses, are automatically extracted [3, 8]. Tostandardizeandvalidatetheseforensic artifacts, we employed the following LLM prompt: 1 numbersep=5pt, 2 linenos, 3 breaklines=true, 4 frame=single, 5 fontsize= , 6 breaklines=true, 7 bgcolor=white 8 ]text 9 You are a forensic artifact refinement engine. Your task is to analyze each input row from a CSV file extracted from a mobile application database. Each row contains metadata and column-value pairs. Identify valid forensic artifacts, refine them, assign confidence scores, and output each artifact individually in the required format. 10 Each row contains: 11 Database Name (DB), Table Name (TN), File Path (FP), Row Line Number (LID), Unique Identifier (UID), One or more column-value pairs 12 Your task is to: 13 1) Use column names and metadata to determine the context of each value. 14 2) Extract only valid forensic artifacts of the following types: 15 Email 16 Phone Number 17 Human Name (real human names) 18 Username 19 App Name (convert package names to recognizable names) 20 Timestamp (convert to human-readable format) 21 Search Keyword (from queries and titles) 22 Message (user-generated text like SMS or chat) 23 MAC Address 24 Longitude 25 Latitude 26 Address (only identifiable physical locations) 27 3) For each artifact: 28 Refine the value by correcting inconsistencies, removing obfuscations (e.g., encoded characters), and converting to a human-readable forensic format. 29 Assign a confidence score between 1 (low certainty) and 10 (high certainty). 30 Only retain artifacts with a confidence score of 5 or higher. 31 4) Output Structure (F): 32 For every valid artifact identified, output a structured entry including: 33 Entity Type (Exact label from the list below) 34 Refined Value (The cleaned, normalized, human-readable artifact) 35 Confidence Score (An integer from 1 to 10) 36 Each extracted artifact must be listed separately, even if multiple artifacts are extracted from the same input row. 37 In Output Use the exact entity type labels listed below for all output entries: 38 App Name, Username, Human Name, Phone Number, Email, Search keyword, Message, MAC Address, Longitude, Latitude, Address, Timestamp. 39 5) Important rules: 40 Do not include irrelevant system-level fields or Android internal configuration metadata 41 Only use values that can be reliably interpreted based on column names and context 42 Do not infer or invent artifact types not listed above 43 If a value is partially recovered or decoded, include it only if confidence $≥$ 5 44 Each artifact must be written individually, not grouped or merged To strengthen forensic reliability and limit speculative in- ferences, each entity is assigned a confidence score between 1 (least certain) and 10 (most certain). Artifacts with scores below a defined threshold (e.g., 5) are excluded during post- processing. This filtering mitigates the inclusion of low- certainty outputs, especially in cases involving incomplete metadata or fragmented records. Retained artifacts not only preserve chain-of-custody via UID, but also satisfy an added standard of inferential confidence, supporting evidentiary in- tegrity. a) Example: Consider an entry from the UserStore table of core.db (Snapchat). After CSV conversion, it may appear as: (Database,core.db);(Table,UserStore); (FilePath,...m.snapchat.android/databases/); (UID,9f97eac5_UserStore_114);(LID,114); (realval,...x19heisenbergercarro@gmail.com- - 0...) After processing by the LLM using the above prompt, the refined artifacts are stored in separate CSV files: • Emails.csv: heisenbergcarro@gmail.com, Confi- dence: 10 • Name.csv: Heisenberg White, Confidence: 7 • AppNames.csv: Snapchat, Confidence: 9 Each refined artifact retains the original UID, maintaining chain-of-custody. If an artifact, such as an obfuscated search query or partially reconstructed name, receives a confidence score below 5, it is excluded to ensure only high-integrity evidence is used for further forensic correlation. D. Knowledge Graph Construction Following LLM-based refinement, artifacts from separate CSV files are aligned using UIDs. Entries sharing a UID are merged into consolidated records, integrating complementary attributes (e.g., names, timestamps, application references). This reduces duplication while preserving contextual integrity required for the construction of the Digital Forensic Knowl- edge Graph (DFKG). The DFKG employs a node-edge model, where nodes represent forensic entities (e.g., email address, timestamp) and edges define relationships (e.g., communication links, temporal correlations). The summary nodes, shown in Table I, trace the transformation from raw input to refined output, strengthening traceability and auditability. Isolated nodes, those without direct links, are grouped alphabetically by application name. Since all applications originate from the same device, this enhances readability and supports intra-device analysis without implying semantic linkage. It aids contextualization while preserving forensic independence. To ensure data quality, only artifacts with a score of confidence ≥5 are retained in the DFKG. This threshold, set during LLM refinement, filters out low-certainty outputs from fragmented metadata. As a result, the graph emphasizes high- confidence relationships, minimizing false associations and supporting hypothesis validation. Each node and edge is tied to a single device via deter- ministic UIDs, ensuring cross-device separation and preserv- ing evidentiary integrity in accordance with chain-of-custody principles. An example graph is shown in Fig.2. The DFKG provides a unified structure for modeling entities and relationships, supporting scalable, auditable forensic investigations. Its UID- based alignment and contextual linkage enhance clarity and uphold forensic standards[5, 8]. While structurally similar to threat intelligence formats such as STIX [12], the DFKG is independently developed. Its de- sign prioritizes reproducibility, traceability, and forensic prove- nance to support legal validation. Future work will explore exporting UID-linked entities in STIX-compatible formats for interoperability across forensic and intelligence platforms. E. Evaluation Metrics We evaluated the proposed forensic framework using ground truth data from Cellebrite’s Capture-the-Flag (CTF) competi- tions [13, 14], validating results against Cellebrite solutions and expert forensic analysis. A total of 2,000 rows were processed. The LLM-based extraction initially identified 42 potential evidence items, of which 40 were correct and 2 were false positives. After UID-based consolidation, these were merged into 26 distinct evidence records, with 24 correct and 2 incorrect records. The final Digital Forensic Knowledge Graph (DFKG) constructed from these records contained 72 connec- tions, of which 68 accurately represented true relationships. Metric 1: Evidence Extraction Accuracy (EEA) EEA = True Extractions Total Potential Extractions × 100% EEA = 40 42 × 100%≈ 95.24% Explanation: EEA quantifies the accuracy of the initial LLM- based extraction process. Metric 2: Evidence Consolidation Accuracy (ECA) ECA = Correctly Consolidated Records Total Consolidated Records × 100% ECA = 24 26 × 100%≈ 92.31% Explanation: ECA reflects the effectiveness of merging over- lapping evidence items using UID. Metric 3: Knowledge Graph Connectivity Accuracy (KGCA) KGCA = Correctly Established Connections Total Connections × 100% KGCA = 68 72 × 100%≈ 94.44% Explanation: KGCA measures the accuracy of relationships formed in the final knowledge graph. Metric 4: Forensic Artifact Precision (FAP) FAP = TP TP + FP × 100% = 40 40 + 2 × 100%≈ 95.24% Explanation: FAP quantifies the proportion of correctly iden- tified artifacts among all identified artifacts. Metric 5: Forensic Artifact Recall (FAR) Assuming no relevant evidence was missed (FN = 0): FAR = TP TP + FN × 100% = 40 40 + 0 × 100% = 100% Explanation: FAR measures the system’s ability to extract all relevant evidence items. Metric 6: Artifact Integrity Score (AIS) AIS = Correctly Extracted Artifacts Total Extracted Artifacts × 100% AIS = 40 42 × 100%≈ 95.24% Explanation: AIS evaluates fidelity to original evidence based on ground truth. TABLE I: Summary of the End-to-End Workflow for Extracting and Validating Forensic Artifacts Artifact TypeDatabase Source (Column, Row Data)Extracted ArtifactsDestination CSV file Email(realVal,...a 19heisenbergercarro@gmail.com 0...)heisenbergercarro@gmail.comEmail.csv Timestamp(proto,. . . x03 0f“1617477858090” 2 03 . . . )03 April 2021 15:24:18Timestamp.csv Phone Number(blobVal,. . . n 0c+16506808040 12 01. . . )+16506808040Phonenumber.csv Name(Data,. . ."ullname":"Marsha Mellos","pro. . . )Marsha MellosName.csv Google Search Query(title, hidden photos apps - Google Search)hidden photos appsGoogleSearch.csv MAC Address(address, 34:C7:31:F8:61:3B)34:C7:31:F8:61:3BMacaddr.csv App Name(DPath, . . . ser/0/com.instagram.android/databases/. . . )InstagramAppname.csv Note: This table summarizes the end-to-end flow for data from Database Source to Extracted Artifacts. Abbreviations: Artifact Type = Type of forensic artifact; Process = Technique used for artifact extraction. Metric 7: Chain of Custody Adherence (CCA) CCA = 100% Explanation: CCA confirms that all artifacts retain key meta- data (e.g., UID), preserving legal admissibility. Metric 8: Contextual Consistency Score (CCS) CCS = Artifacts Matching Expected Contexts Total Extracted Artifacts × 100% CCS = 40 40 × 100% = 100% Explanation: CCS measures how well extracted evidence aligns with expected forensic context. The initial LLM-based extraction achieved an EEA of 95.24%, identifying 42 artifacts with 40 correct and 2 false positives. UID-based consolidation merged these into 26 records with an ECA of 92.31%. The resulting DFKG formed 72 connections, of which 68 were accurate, yielding a KGCA of 94.44%. Although these metrics reflect the complete LLM output, a post-processing confidence threshold of 5 (on a 1 to 10 scale) was applied to exclude low-certainty artifacts from downstream analysis and graph construction. This safeguard enhances reliability by filtering speculative inferences and reinforcing evidentiary integrity. IV. CASE STUDY: MOBILE PHONE FORENSICS This section presents a case study applying the proposed forensic framework to assess the reliability and soundness of extracted digital evidence in a mobile phone investigation. The evaluation is structured around five research questions, each addressing a distinct aspect of the forensic pipeline, including artifact identification, data transformation, LLM refinement, knowledge graph construction, and overall integrity. A. Case Study Description The case centers on a suspected auto theft and resale scheme involving three individuals: Heisenberg, Beth Dutton, and Marsha Mellos. On July 21 st , law enforcement apprehended Beth at a restaurant in Vienna, VA, following an invitation from Heisenberg. During questioning, Beth asserted that both she and her sister Marsha were innocent, claiming Heisen- berg was the orchestrator behind the operation. The sisters, who operate a cattle business in Montana, stated they were unintentionally linked to his activities. The primary source of digital evidence was Heisenberg’s Android smartphone, seized to evaluate his potential involve- ment. Key indicators included Google searches related to auto theft, Bluetooth connections to vehicles, and communications involving all three individuals, each contributing context re- garding Heisenberg’s actions and associations. To test the framework, a forensic image of a Samsung Galaxy Note 10 from Cellebrite’s 2021 and 2022 Capture- the-Flag (CTF) competitions was used [13, 14]. The 13 GB image included data from 61 applications (both user-installed and system), 2,864 databases, and 5,870 tables. The framework was applied to automate artifact extraction, normalize records, refine values using OpenAI’s GPT-4, and conduct semantic analysis with emphasis on metadata integrity, accountability, and traceability. Entity relationships were modeled using the Digital Forensic Knowledge Graph (DFKG), facilitating systematic event re- construction and uncovering connections between the suspects. This case study demonstrates the framework’s capability to automate digital evidence discovery, improve artifact account- ability, and support dependable forensic investigations. Perfor- mance metrics, including accuracy, precision, recall, F1-score, and legal admissibility (Section I-E), further validate the reliability of LLM-assisted evidence processing under realistic conditions. B. RQ 1: How can a standardized framework be implemented to evaluate the reliability of forensic evidence refined by LLM using visualization of knowledge graphs? The proposed methodology introduces a structured frame- work for evaluating the reliability, accuracy, and forensic validity of digital evidence refined by large language models (LLMs). This is achieved by integrating database extrac- tion, structured transformation, artifact refinement, and Digital Forensic Knowledge Graph (DFKG) visualization. SQLite databases, commonly used to store communication logs, app usage records, and geolocation data, are extracted from forensic images [4]. These are converted into structured CSV files while preserving metadata such as database name, table name, and row index. A deterministic Unique Identi- fier (UID) scheme (see Section I) ensures traceability and integrity. Each row in the CSV is then processed by an LLM module, which extracts and refines key entities, includ- ing Name, Phone Number, Email, Timestamps, and Fig. 2: Visualization of the DFKG with Artifact Relationships Application Name. The model corrects inconsistencies and obfuscations while maintaining UID-based linkage to the original data. Refined artifacts are embedded into the DFKG, where nodes represent entities and edges capture contextual relation- ships. This graph-based structure supports evidence correla- tion, hypothesis testing, and contextual validation. Ground- truth datasets from Cellebrite’s CTF competitions [13, 14] are used for evaluation, with records categorized as true positives, false positives, or false negatives. To assess performance, the framework employs a suite of forensic-specific metrics: Evidence Extraction Accuracy (EEA), Forensic Artifact Precision (FAP), Forensic Arti- fact Recall (FAR), Knowledge Graph Connectivity Accuracy (KGCA), Chain of Custody Adherence (CCA), and Contex- tual Consistency Score (CCS) (Section I-E). These metrics validate both artifact authenticity and contextual traceability. By Combining UID traceability, LLM refinement, graph structuring, and hypothesis validation, the framework enables transparent and reproducible evaluation of forensic evidence in AI-assisted investigations. TABLE I: Explanation and Validation of Hypotheses for Edge Types in the DFKG Edge TypeExample of HypothesisTotalValidInvalid Timestamps and Application Name User logged into Google Drive on 2021-04-03 at 15:24:18 EDT. 24222 User searched for “interesting car apps” in Chrome on 2021-07-08 00:02:39 EDT. User initiated a Bluetooth connection on 2021-05-20 at 21:18:53 EDT. Email and Application Name User accessed Google Drive associated with the email heisenbergcarro@gmail.com. 532 User account associated with the email heisenbergcarro@gmail.com was linked to Snapchat. The Twitter account associated with the email CryptoWendyO@protonmail.com is linked to Heisenberg’s Twitter interactions. Application Name and Google Search Query User performed a Google search for “italianos near me” in Chrome. 13130 User performed a Google search for “hidden photos apps” in Chrome. User performed a Google search for “interesting car apps” in Chrome. MAC Address and Application Name Device with MAC address 34:C7:31:F8:61:3B was connected via Bluetooth. 330 Device with MAC address 2C:6B:7D:1D:21 was connected via Bluetooth. Device with MAC address F0:8A:76:C4:F8 was connected via Bluetooth. Timestamps and Email User interacted with CryptoWendyO@protonmail.com content on Twitter at 2021-07-19 17:48 EDT. 550 User interacted with content associated with the email thedogecoinmillionaire@gmail.com on Twitter on 2021-06-25 02:29:19 EDT. Timestamp and Google Search Query User searched for “chester springs pa zip code” at 2021-07-07 23:51:40 EDT. 13130 User searched for “hidden photos apps” at 2021-06-15 13:46:34 EDT. User searched for “interesting car apps” at 2021-07-08 00:02:39 EDT. Timestamp and MAC Address Device with MAC 34:C7:31:F8:61:3B connected via Bluetooth on 2021-04-04 at 23:15:36 EDT. 330 Device with MAC F0:8A:76:C4:F8 connected via Bluetooth on 2021-05-09 at 14:58:04 EDT. Device with MAC 2C:6B:7D:1D:21 connected via Bluetooth on 2021-05-20 at 21:18:53 EDT. Name and Timestamp User interacted with Beth Dutton on Snapchat on 2021-04-05 at 16:28:23 EDT. 220 User interacted with Marsha Mellos on Twitter on 2021-06-19 at 21:38:00 EDT. Name and Application Name User interacted with Beth Dutton on Snapchat. 220 User interacted with Marsha Mellos on Twitter. Phone Number and Application Name The phone number +16506808040 is associated with Snapchat.110 Phone Number and Email The email heisenbergcarro@gmail.com is associated with the phone number +16506808040 on Snapchat.110 Total72684 Note: This table provides insights into the validation of hypotheses for relationships between artifact types in the Digital Forensic Knowledge Graph (DFKG). Abbreviations: Total = total number of hypotheses tested; Valid = number of validated hypotheses; Invalid = number of hypotheses that failed validation. C. RQ 2: How can the reliability of LLM-identified forensic evidence be measured in terms of accuracy, false positives, and artifact coverage? The reliability of LLM-assisted forensic evidence is as- sessed using forensic-specific metrics, including accuracy, false positive rates, and artifact coverage. These metrics eval- uate the framework’s ability to extract, validate, and refine forensic artifacts while minimizing classification errors. The empirical results confirm the framework’s effectiveness in ensuring forensic integrity. Evidence Extraction Accuracy (EEA) measures the pro- portion of correctly identified forensic artifacts relative to the total extractions. Out of 42 potential evidence items, 40 were correctly identified, with 2 false positives, resulting in an EEA of 95.24%, demonstrating the framework’s reliability in initial LLM-based extraction. Forensic Artifact Precision (FAP) quantifies the correct- ness of positively identified artifacts. Since 40 out of 42 extracted artifacts were correctly classified, the precision score is 95.24%, confirming that the majority of classified artifacts were forensically valid. Forensic Artifact Recall (FAR) evaluates the framework’s ability to detect all relevant forensic artifacts. Since no relevant evidence was missed (false negatives = 0), the recall was measured at 100%, ensuring complete forensic data retrieval. Forensic Artifact F1-Score (FAF1) balances precision and recall, calculated as 97.56%, reinforcing the system’s reliability in forensic artifact classification. Specifically, the LLM-based framework achieved a precision of 95.24%, a recall of 100%, and an F1-score of 97.56%, highlighting its robust performance across key forensic metrics. Evidence Consolidation Accuracy (ECA) assesses the merging of overlapping forensic evidence based on Unique Identifiers (UIDs). Initially, 42 evidence items were identified, but after UID-based consolidation, they were merged into 26 distinct forensic records, with 24 correctly consolidated and 2 incorrectly merged. This results in an ECA of 92.31%, highlighting the effectiveness of UID-driven consolidation. Knowledge Graph Connectivity Accuracy (KGCA) mea- sures the integrity of relationships within the Digital Foren- sic Knowledge Graph (DFKG). The forensic graph initially contained 72 established connections, out of which 68 were correctly mapped, leading to a KGCA of 94.44 Contextual Consistency Score (CCS) validates the align- ment of extracted forensic artifacts with expected forensic patterns. The system achieved 100% CCS, ensuring that all re- fined forensic evidence maintained consistency with expected investigative scenarios. Chain-of-Custody Adherence (CCA) ensures the preserva- tion of metadata integrity throughout the forensic process. The system maintained 100% CCA, confirming that all forensic artifacts retained their original metadata, ensuring traceability and legal admissibility. These findings confirm that the proposed forensic frame- work achieves high artifact extraction accuracy, precise con- solidation, strong forensic graph relationship integrity, and full traceability. The results validate its ability to maintain forensic authenticity and enhance investigative reliability. D. RQ 3: What are the key sources of error or bias in LLM- generated forensic evidence and how do they impact artifact classification? The framework identifies key sources of error and bias that affect the reliability of LLM-generated forensic evidence. These include incomplete metadata, fragmented or deleted records, and contextual inference bias. Each impacts how digital evidence is interpreted, particularly when relationships are inferred by large language models (LLMs). Incomplete metadata weakens classification by omitting essential context like user IDs, session logs, or app usage traces. Artifacts may appear syntactically correct, yet lack the contextual fields needed for accurate association in the Digital Forensic Knowledge Graph (DFKG). As shown in Fig- ure 3, the timestamp 2021-06-25 02:29:19 EDT was mistakenly linked to Twitter due to missing metadata. This misclassification was only identified during graph-based vali- dation against Cellebrite’s ground truth, despite an Evidence Extraction Accuracy (EEA) of 95.24% (40 of 42 correct). Fig. 3: Invalid hypothesis for Timestamps and Application Names. Fragmentation and deleted data disrupt artifact recon- struction by stripping structural cues. For example, a corrupted email 19xxheisenbergcarro@gmail.comx1 was ini- tially misclassified. Although the LLM accurately refined it to heisenbergcarro@gmail.com, insufficient metadata prevented reliable linkage to an application. This relationship was flagged during DFKG validation and contributed to two false positives in Forensic Artifact Precision (FAP), recorded at 95.24%. Contextual bias in LLM inference occurs when re- lationships are inferred from co-occurrence rather than Fig. 4: Invalid hypothesis for Email and Application Names. substantiated connections. While malformed entries like marketing@get@upside were correctly filtered, sub- tler errors persisted. Figure 4 shows one such case where CryptoWendyO@protonmail.com was wrongly linked to Twitter, despite the absence of corroborating logs. This contributed to a Knowledge Graph Connectivity Accuracy (KGCA) of 94.44% (68 of 72 valid relationships). These false positives emerged only during graph validation. To mitigate such issues, the framework assigns confidence scores (1–10) during refinement and excludes artifacts below a threshold of 5. This removes speculative outputs before graph construction. Manual graph-based hypothesis testing serves as a secondary safeguard, revealing context-related errors that automated filters may miss. Although all metrics include pre-filtering data for transparency, this dual-layer validation improves overall forensic reliability. E. RQ 4: How can LLM-assisted forensic evidence be val- idated to ensure traceability and accuracy through cross- referencing with sources? The validation of LLM-assisted forensic evidence is achieved through deterministic Unique Identifiers (UIDs) and structured cross-referencing within the Digital Forensic Knowledge Graph (DFKG). Each artifact is assigned a UID derived from key attributes, including the database name, table name, file path, and row index, preserving forensic provenance and ensuring end-to-end auditability. This approach safeguards the chain of custody, enabling investigators to trace refined artifacts back to their original raw sources. Empirical validation confirms that every artifact in the DFKG retains its UID throughout the forensic pipeline, al- lowing discrepancies introduced during LLM processing to be detected. The framework achieved an Evidence Consolidation Accuracy (ECA) of 92.31%, correctly merging 24 out of 26 consolidated forensic records, ensuring UID-based integrity. Additionally, it maintained 100% Chain-of-Custody Adher- ence (CCA), verifying that all extracted artifacts preserved their original metadata. The Knowledge Graph Connectivity Accuracy (KGCA) of 94.44% further validated the integrity of forensic relationships. While two artifacts were incorrectly consolidated, UID- driven cross-referencing identified these inconsistencies, demonstrating the system’s ability to correct forensic errors. By embedding UID-based traceability, enforcing structured cross-referencing, and ensuring consistent metadata preserva- tion, the framework maintains evidentiary integrity, supporting a legally admissible and auditable forensic methodology. F. RQ 5: What best practices can improve the admissibility and trustworthiness of LLM-derived evidence, focusing on traceability, standardized extraction, and validation work- flows? A structured forensic workflow enhances the admissibility and trustworthiness of LLM-derived evidence. In this study, a 13 GB forensic image containing 61 applications, 2,864 databases, and 5,870 tables was converted into structured CSV files, preserving comprehensive metadata and enabling artifact traceability. Provenance is maintained using deterministic Unique Iden- tifiers (UIDs), generated by SHA-256 hashing of device ID, database name, file path, table name, and row index. This achieves 100% Chain-of-Custody Adherence (CCA), ensuring each artifact links to its original source. StandardizedCSVexportsensureconsistent formatting,enablingLLMstoreconstructfragmented artifacts.Forinstance,theobfuscatedemailaddress 19xxheisenbergcarro@gmail.comx1 was accurately refined to heisenbergcarro@gmail.com, contributing to a Forensic Artifact Precision (FAP) of 95.24 Validation confirmed an Evidence Extraction Accuracy (EEA) of 95.24% (40 out of 42 correct) and a Knowledge Graph Connectivity Accuracy (KGCA) of 94.44%, affirming accurate relationship mapping. The framework adheres to ISO/IEC 27037 guidelines on reproducibility and authenticity, supporting its legal admissibility. In summary, practices such as standardized extraction, UID-based traceability, LLM refinement, and robust cross- referencing ensure that LLM-derived forensic evidence is reliable, reproducible, and legally defensible. V. CONCLUSION AND FUTURE WORK This study presented a structured framework to evaluate the reliability and forensic soundness of digital evidence refined by large language models (LLMs). Through hypothesis validation, UID-based cross-referencing, and Digital Forensic Knowledge Graph (DFKG) construction, the framework en- sures artifact integrity, traceability, and contextual accuracy in AI-assisted investigations. Empirical results demonstrated strong forensic performance: 95.24% Evidence Extraction Accuracy (EEA), 95.24% Foren- sic Artifact Precision (FAP), and 100% Forensic Artifact Recall (FAR), indicating minimal false positives and complete artifact retrieval. The system also achieved a 97.56% Forensic Artifact F1-Score (FAF1), 94.44% Knowledge Graph Connec- tivity Accuracy (KGCA), and 100% scores for both Chain-of- Custody Adherence (CCA) and Contextual Consistency Score (CCS), supporting legal admissibility and reliability. Key challenges include handling ambiguous artifacts from incomplete logs, fragmented metadata, or static databases. These conditions may affect relationship reconstruction and require enhanced validation and broader cross-referencing. The current confidence-based filtering mechanism, though effective in minimizing false positives, may omit valid artifacts lacking sufficient context due to deletion, encryption, or cor- ruption. Future work will explore adaptive thresholding, con- textual re-validation, and deferred artifact review to mitigate premature exclusions. Ongoing development aims to scale the framework for multi-device investigations and assess its performance in real- world cybercrime scenarios. Planned integration with AI- driven forensic reasoning systems will further improve classi- fication and timeline reconstruction. To support interoperability, future versions will export UID- linked artifacts using Structured Threat Information Expres- sion (STIX) [12]. Although the current design prioritizes de- terministic identifiers for auditability, STIX-compliant output will enable standardized evidence exchange across forensic tools and collaborative environments. REFERENCES [1] W. Xu, J. Yan, and H. Chi, “A forensic evidence acquisition model for data leakage attacks,” in Proc. 17th IEEE Int. Conf. Intelligence and Security Informatics (ISI), 2019, p. 53–58. [Online]. Available: https: //ieeexplore.ieee.org/document/8823398 [2] T. Wu, F. Breitinger, and S. O’Shaughnessy, “Digital forensic tools: Recent advances and enhancing the status quo,” Forensic Sci. Int.: Digital Investigation, vol. 34, p. 300999, 2020. [Online]. Available: https: //doi.org/10.1016/j.fsidi.2020.300999 [3] A. Wickramasekara, F. Breitinger, and M. Scanlon, “Exploring the potential of large language models for improving digital forensic investigation efficiency,” 2024, arXiv:2402.19366. [Online]. Available: https: //arxiv.org/pdf/2402.19366 [4] M. Scanlon, B. Nikkel, and Z. Geradts, “Digital forensic investigation in the age of chatgpt,” Forensic Sci. Int.: Digital Investigation, vol. 44, p. 301543, 2023. [Online]. Available: https://doi.org/10.1016/j.fsidi.2023.301543 [5] W. Xu and D. Xu, “Visualizing and reasoning about presentable digital forensic evidence with knowledge graphs,” in Proc. 19th Int. Conf. Privacy, Security & Trust (PST), 2022, p. 1–10. [Online]. Available: https://ieeexplore.ieee.org/document/9851972 [6] D. Addai, S. Shaikh, E. Xu, W. Zhang, and W. Xu, “A graph-based approach for discovering evidence relation- ships across multiple devices in group crimes,” in Proc. 24th IEEE Int. Conf. Software Quality, Reliability, and Security Companion (QRS-C), 2024, p. 1312–1313. [7] A. Karim et al., “Forensic artifacts’ analysis using graph theory,” in CEUR Workshop Proc., 2024. [Online]. Available: https://ceur-ws.org/Vol-3792/paper26.pdf [8] E. Xu, W. Zhang, and W. Xu, “Transforming digital forensics with large language models: Unlocking automation, insights, and justice,” in Proc. ACM Int. Conf. Information and Knowledge Management (CIKM), 2024, p. 1450–1462. [Online]. Available: https://dl.acm.org/doi/pdf/10.1145/3627673.3679091 [9] S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu, “Unifying large language models and knowledge graphs: A roadmap,” IEEE Trans. Knowl. Data Eng., 2024. [Online]. Available: https://ieeexplore.ieee.org/ document/10387715 [10] A. Wickramasekara, F. Breitinger, and M. Scanlon, “Where is the potential for large language models in digital forensic investigations?” in Proc. Digital Forensics Research Workshop (DFRWS EU), 2024, p. 78–89. [Online]. Available: https://dfrws.org/wp-content/ uploads/2024/03/DFRWS EU2024paper4391.pdf [11] S. Talekar et al., “Forensift: Gen-ai powered integrated digital forensics and incident response platform using langchain framework,” Int. J. Multidisciplinary Research (IJFMR), 2024. [Online]. Available: https://w.ijfmr. com/research-paper.php?id=31692 [12] OASIS Committee Specification, “Stix version 2.1. part 1: Stix core concepts,” Online, March 2020, accessed: 2025-05-16. [Online]. Available: https://docs.oasis-open. org/cti/stix/v2.1/stix-v2.1-part1-stix-core.html [13] C.2021,“Overviewofthe2021 cellebritecapturetheflagevent,”2021. [Online].Available:https://cellebrite.com/en/ overview-of-the-2021-cellebrite-capture-the-flag-event/ [14] C. 2022, “Digital forensics 101: The value of ‘capture the flag’ events,” 2022. [Online]. Available: https: //cellebrite.com/en/blog/final-ctf-2022-round-up/