Paper deep dive
GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDs
Achmad Anggawirya Alimin, Artur M. Schweidtmann
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 1:35:50 AM
Summary
ChatP&ID is an agentic framework that utilizes Graph Retrieval-Augmented Generation (GraphRAG) to enable natural-language interaction with Piping and Instrumentation Diagrams (P&IDs). By transforming DEXPI-standardized P&IDs into structured knowledge graphs, the system improves query accuracy by 18% over raw images and reduces token costs by 85% compared to direct file ingestion, facilitating tasks like HAZOP analysis.
Entities (5)
Relation Signals (3)
ChatP&ID → utilizes → GraphRAG
confidence 100% · ChatP&ID, an agentic framework that enables grounded and cost-effective natural-language interaction with P&IDs using Graph Retrieval-Augmented Generation (GraphRAG)
DEXPI → istransformedinto → Knowledge Graph
confidence 95% · Smart P&IDs encoded in the DEXPI standard are transformed into structured knowledge graphs
ChatP&ID → queries → Neo4j
confidence 90% · These tools query the Neo4j graph database to extract relevant nodes, edges, or sections of the knowledge graph
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG) and knowledge graphs offer new opportunities for interacting with engineering diagrams such as Piping and Instrumentation Diagrams (P&IDs). However, directly processing raw images or smart P&ID files with LLMs is often costly, inefficient, and prone to hallucinations. This work introduces ChatP&ID, an agentic framework that enables grounded and cost-effective natural-language interaction with P&IDs using Graph Retrieval-Augmented Generation (GraphRAG), a paradigm we refer to as GraphRAG for engineering diagrams. Smart P&IDs encoded in the DEXPI standard are transformed into structured knowledge graphs, which serve as the basis for graph-based retrieval and reasoning by LLM agents. This approach enables reliable querying of engineering diagrams while significantly reducing computational cost. Benchmarking across commercial LLM APIs (OpenAI, Anthropic) demonstrates that graph-based representations improve accuracy by 18% over raw image inputs and reduce token costs by 85% compared to directly ingesting smart P&ID files. While small open-source models still struggle to interpret knowledge graph formats and structured engineering data, integrating them with VectorRAG and PathRAG improves response accuracy by up to 40%. Notably, GPT-5-mini combined with ContextRAG achieves 91% accuracy at a cost of only $0.004 per task. The resulting ChatP&ID interface enables intuitive natural-language interaction with complex engineering diagrams and lays the groundwork for AI-assisted process engineering tasks such as Hazard and Operability Studies (HAZOP) and multi-agent analysis.
Tags
Links
- Source: https://arxiv.org/abs/2603.22528v1
- Canonical: https://arxiv.org/abs/2603.22528v1
Trouble viewing inline? Open PDF directly →
Full Text
137,969 characters extracted from source content.
Expand or collapse full text
GRAPHRAG FOR ENGINEERING DIAGRAMS: CHATP&ID ENABLES LLM INTERACTION WITH P&IDS A PREPRINT Achmad Anggawirya Alimin Process Intelligence Research Group Department of Chemical Engineering Delft University of Technology Artur M. Schweidtmann ∗ Process Intelligence Research Group Department of Chemical Engineering Delft University of Technology A.Schweidtmann@tudelft.nl March 25, 2026 ABSTRACT Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG) and knowl- edge graphs offer new opportunities for interacting with engineering diagrams such as Piping and Instrumentation Diagrams (P&IDs). However, directly processing raw images or smart P&ID files with LLMs is often costly, inefficient, and prone to hallucinations. This work introduces ChatP&ID, an agentic framework that enables grounded and cost-effective natural-language interaction with P&IDs using Graph Retrieval-Augmented Generation (GraphRAG), a paradigm we refer to as GraphRAG for engineering diagrams. Smart P&IDs encoded in the DEXPI standard are transformed into structured knowledge graphs, which serve as the basis for graph-based retrieval and reasoning by LLM agents. This approach enables reliable querying of engineering diagrams while significantly reducing computational cost. Benchmarking across commercial LLM APIs (OpenAI, Anthropic) demonstrates that graph-based representations improve accuracy by 18% over raw image inputs and reduce token costs by 85% compared to directly ingesting smart P&ID files. While small open-source models still struggle to interpret knowledge graph formats and structured engineering data, integrating them with VectorRAG and PathRAG improves response accuracy by up to 40%. Notably, GPT-5-mini combined with ContextRAG achieves 91% accuracy at a cost of only $0.004 per task. The resulting ChatP&ID interface enables intuitive natural-language interaction with complex engineering diagrams and lays the groundwork for AI-assisted process engineering tasks such as Hazard and Operability Studies (HAZOP) and multi-agent analysis. Keywords P&ID, Large Language Model (LLM)·Multi-agent System (MAS)·Agentic-AI·Knowledge Graph· Graph-based Retrieval Augmented Generation (GraphRAG) 1 Introduction Piping and Instrumentation Diagrams (P&IDs) serve as an essential source of information in process engineer- ing (Toghraei, 2019). Engineers rely on accurate interpretation of P&ID data throughout the lifecycle of a plant, including design, operation, maintenance, expansion, and risk assessments. However, interacting with P&IDs remains a bottleneck due to their complex structure and the heavy reliance on manual workflow. Current practices typically involve manually tracing process lines and equipment in PDF or Computer-Aided Engineering (CAE) files, a process that is both time-consuming and error-prone. Recent digitalization efforts in process engineering are shifting P&ID development toward a more data-centric paradigm. “Smart” or “intelligent” P&IDs embed database structures within traditional diagrams, enabling easier data exchange and more automated workflows. While most industrial P&IDs still exist as PDFs or physical drawings, ongoing research is converting legacy assets into smart, machine-readable formats (Theisen et al., 2023). Standardization initiatives ∗ corresponding author arXiv:2603.22528v1 [cs.IR] 23 Mar 2026 ChatP&IDA PREPRINT such as DEXPI have also been instrumental in this shift by defining a unified data schema for P&IDs (Theißen and Wiedau, 2021). Leveraging these standards, our group developed pyDEXPI, a Python implementation of the DEXPI information model (Goldstein et al., 2025). Although these advances lay a strong foundation for the future of process data management, practical solutions that seamlessly integrate into engineers’ existing workflows remain unexplored. There is a clear opportunity to unlock more value from digital P&IDs. Lately, leveraging generative AI for P&IDs has become a central focus for advancing AI in process engineering (Schwei- dtmann, 2024). Several studies have explored the use of AI to extract information from or analyze P&IDs. A common approach feeds P&ID images directly into Large Language Models (LLMs) and asks for tasks ranging from component identification (Medhane et al., 2025) to HAZOP analysis (Lee et al., 2026; Mukharror et al., 2025). However, this approach obscures the image recognition steps, limiting transparency into how the LLM extracts information. This will make systematic improvement difficult, other than changing the LLM model. Another method passes the smart P&ID files as textual context for the LLM to read and modify (Gowaikar et al., 2024). While more transparent than images, this approach remains prone to hallucinations and is challenging for engineers to inspect, as the XML is designed for CAD (Computer-Aided Design) software, which lacks semantic readability. They can also be resource-intensive; in one study (Alimin et al., 2025), a single simple P&ID page required over 150,000 tokens. Despite this exploration, to date, the optimal representation of P&IDs for AI-based flowsheet analysis remains an open question. In our previous work, we established a foundation for representing P&IDs as knowledge graphs and enabling LLMs to interpret these engineering diagrams (Alimin et al., 2025). This knowledge graph representation offers a more intuitive and interactive way for process engineers to engage with P&ID data by passing graph context to LLMs. The latest advances in graph reasoning methods have also revealed substantial untapped opportunities to leverage graph-based representations for more powerful and accurate querying. For example, recent work has shown that LLMs can extract localized information and summarize global graph semantics (Edge et al., 2024), and several studies explore iterative graph exploration guided by LLM reasoning (Sun et al., 2023; Ma et al., 2024; He et al., 2024). Furthermore, advances in LLM-generated Cypher queries showcase the potential for natural-language-driven retrieval from Labeled Property Graphs (LPG) (Szlobodnyik, 2025; Gusarov et al., 2025; Gupta et al., 2025). Yet, none of these GraphRAG techniques have been evaluated on P&IDs, and their performance has not been benchmarked in terms of computational efficiency, cost, and response accuracy when compared to image-based interpretation or direct ingestion of smart P&IDs across different LLMs. To the best of our knowledge, this is the first application of GraphRAG to structured engineering diagrams such as Piping and Instrumentation Diagrams. To address these gaps, this work extends our previous methodology by integrating state-of-the-art graph-based query methods for P&ID. We also provide a comprehensive evaluation of multiple online and offline LLMs and model scales, benchmarking them across response accuracy, computational time, and cost. Finally, we present our work as a chat interface called ChatP&ID, an agent-driven chat interface in which LLMs autonomously select and invoke GraphRAG tools to query information from the P&ID. This enables process engineers to interact with and extract insights from P&IDs in an intuitive, efficient way, while ensuring the accuracy, transparency, and traceability of the retrieved information. More broadly, we aim to demonstrate how advances in knowledge graph technology can drive practical impact in process engineering. In the long term, the integration of P&IDs into foundation models represents a foundational capability for future GenAI applications, including automated P&ID correction, AI-assisted HAZOP studies (Schweidtmann, 2024), and Multi-Agent System (MAS) for process engineering workflow (Rupprecht et al., 2025). 2 Preliminaries This section introduces the foundational concepts and technologies that enable our proposed framework. 2.1 Representing P&ID as knowledge graphs P&IDs serve as the detailed blueprint for a chemical process facility, describing equipment, piping, control logic, and safety elements in accordance with standardized industrial conventions (Toghraei, 2019). Despite their central role in engineering, safety, and operational workflows, P&ID data is often stored in static formats such as images or PDFs, making updates and information retrieval labor-intensive and error-prone. To address this limitation, the DEXPI initiative defines a standardized semantic data model that enables consistent digital representation and interoperability across CAD tools (Theißen and Wiedau, 2021). A P&ID inherently contains two categories of information: (i) topological connectivity describing how entities interact (e.g., a pump feeding a tank, or a level indicator measuring tank level), and (i) specification data describing attributes such as equipment materials, ratings, etc. While DEXPI captures these elements, its default representation is designed to be read by the CAD program, which lacks semantic expressiveness for Machine Learning (ML) and generative AI implementation. Therefore, we implement pyDEXPI, a 2 ChatP&IDA PREPRINT Python library that enhances DEXPI’s usability by enabling the extraction and export of P&ID data to formats such as tables, JSON, and, most importantly, knowledge graphs (Goldstein et al., 2025). A graph can be represented as an index-free adjacency database consisting of nodes and edges (Latora et al., 2017). Nodes represent entities in the dataset, while edges define relationships between them. Due to their flexible structure and intuitive query language, graph databases are widely used in applications such as social networking, recommender systems, and fraud detection. Knowledge graphs also demonstrated their effectiveness in process engineering for modeling entire systems, such as P&IDs (Morbach et al., 2007; Eibeck et al., 2019). This approach represents equipment and instrumentation as nodes and their relationships as semantically meaningful edges. This not only provides a richer and more intuitive abstraction of P&ID content that closely aligns with how engineers conceptualize process interactions, but also enables machine-readability, modification, and integration with language models (Alimin et al., 2025). There are two graph data formats commonly used for knowledge graph representation: (i) the Resource Description Framework (RDF), which represents information as a triple store consisting of (subject, predicate, object) state- ments (Pan, 2009). In the context of a P&ID, this could be expressed as(Pump1, hasPressure, 50psi). RDF is a W3C standard and is typically queried using the SPARQL query language. (i) The Labeled Property Graph (LPG) model represents information using nodes, edges, and associated properties that can be assigned to both nodes and relationships (Barrasa and Webber, 2023). In a P&ID, this could be represented as a node(id: 1, tag: "Pump1", type: "CentrifugalPump", pressure: 50, pressure_unit: "psi")connected to a valve node via a la- beled edge:(Pump1)-[: CONNECTED_TO flow: 10]->(Valve2). LPG-based graphs are typically queried using languages such as Cypher or Gremlin and are commonly implemented in graph databases such as Neo4j. While each of these formats has its own advantages (Baken, 2020; Di Pierro et al., 2023), the choice of representation for a P&ID should be guided by the specific requirements of the implementation context. 2.2 Graph-based Retrieval Augmented Generation (GraphRAG) LLMs are statistical models that learn to predict the probability distribution of token sequences given input to- kens. Advances in transformer architectures via the attention mechanism (Vaswani et al., 2017) and large-scale pre-training (Radford, 2018) have enabled LLMs to demonstrate impressive capabilities in understanding queries and generating more coherent responses. However, LLMs are not inherently aware of all domain knowledge up to their training date, nor are they guaranteed to have been exposed to every relevant piece of information during training. In particular, P&ID diagrams and engineering design data are often confidential, meaning LLMs may not have had access to them during training. As a result, they may produce responses that appear fluent and coherent, but factually incorrect, a phenomenon often referred to as ‘hallucination’ (Dang et al., 2025). Additionally, the finite context window of an LLM restricts the amount of external information it can effectively include during an inference (Liu et al., 2024; Kuratov et al., 2024). These limitations pose significant risks in safety-critical environments such as chemical-processing facilities, where decision-making demands high accuracy and reliability. Retrieval-augmented generation (RAG) (Lewis et al., 2020) addresses this limitation by enabling an LLM to retrieve and incorporate relevant information from external knowledge sources during inference. By grounding generation in specified domain data, RAG improves the accuracy, specificity, and factuality of model responses. The original RAG formulation is relatively simple: text documents are segmented into chunks and encoded as vectors, and a similarity- based retrieval module selects relevant content to provide as context for the model, a configuration often referred to as vector or naïve RAG (Gao et al., 2023). While this approach has demonstrated promising applicability across diverse domains, technical challenges remain, such as optimizing chunking strategies to manage retrieval latency (Hasan et al., 2025) and improving transparency and explainability, since retrieved information resides in high-dimensional embedding spaces. GraphRAG extends traditional RAG by representing contextual knowledge as a graph of entities and their relation- ships. In GraphRAG, source documents are first transformed into a knowledge graph by identifying subjects and objects as nodes and their relationships as edges. This transformation can be achieved either through named entity recognition (NER) and relation extraction using deep learning methods (Li et al., 2020), or by prompting LLMs to extract and summarize concepts (subject, object, and predicate) directly from text (Edge et al., 2024). The resulting knowledge graph can be further semantically enriched and embedded in a vector space for indexing at a later stage. Once constructed, the knowledge graph can be queried using a variety of GraphRAG techniques, which broadly fall into three categories: (i) structured graph query languages (e.g., Cypher, GQL, Gremlin), (i) vector-based retrieval using vector similarity search, and (i) LLM-driven approaches in which the model either interprets the graph directly or generates graph queries, as well as combinations thereof. Recent GraphRAG approaches combine multiple retrieval methods. For example, Edge et al. (2024) employs LLMs to summarize clusters of closely connected nodes into local semantic representations, then subsequently aggregates them 3 ChatP&IDA PREPRINT into global semantic communities for later retrieval. Sun et al. (2023) introduces Think-on-Graph (ToG), instead of simply using an LLM to query a knowledge graph (LLM⊕ KG), he use the native graph traversal algorithm, such as n−w GRAG (breadth-first) and n−d GRAG (depth-first), and LLMs as agents to interactively traverse and reason over the graph (LLM⊗ KG). Ma et al. (2024), extend this work by implementing community detection to improve retrieval speed by guiding LLMs toward semantically coherent regions of the graph. Recent work also explores how LLMs can infer missing knowledge from incomplete graph structure, for instance, Graph Inspired Veracity Exploration (GIVE) addresses retrieval challenges in GraphRAG systems (He et al., 2024). GraphRAG is particularly interested in P&ID analysis because P&IDs are inherently graph-structured representations of process designs with symbols, making graph-based modeling both intuitive and natural. Indeed, several studies have already demonstrated the effectiveness of graph-based representations for P&ID-related chemical process design and optimization tasks (Theisen et al., 2025; Balhorn et al., 2025; Alimin et al., 2025; Schulze Balhorn et al., 2025). Recent innovations in GraphRAG techniques also closely mirror how engineers interpret, trace, and abstract information from P&IDs. Despite this strong conceptual alignment, no existing work has systematically evaluated GraphRAG approaches in the P&ID and broader process engineering domains. 2.3 Agentic Systems The deployment of LLMs typically spans a spectrum of complexity, from prompt engineering to agentic workflows. In prompt engineering, strategies such as zero-shot, one-shot, and few-shot prompting are commonly used. In zero-shot prompting, the LLM is given no examples; in one-shot prompting, it receives a single example; and in few-shot prompting, it is provided with a small set of examples to guide task completion. These approaches rely entirely on the model’s pre-trained internal weights to generate responses from instructions or examples, without access to external data (Brown et al., 2020). Techniques such as chain-of-thought prompting (Wei et al., 2022) can further improve reasoning by encouraging the model to generate intermediate reasoning steps. While effective for general-purpose tasks, these "stateless" methods often underperform in specialized engineering domains because the model lacks access to proprietary information and cannot verify its outputs, leading to frequent hallucinations. To address these limitations, recent research has shifted towards agentic workflows. Fundamentally, an agent is an intelligent system capable of perceiving its environment, planning actions, and executing tasks to achieve specific goals; this planning process requires understanding of context, multi-step reasoning, and adaptive decision-making (Russell et al., 1995; Ghallab et al., 2004). In the context of LLMs, this is often formalized as the ReAct (Reason and Act) workflow, where the model interleaves reasoning traces with executable actions (Yao et al., 2022). As illustrated in Figure 1, an agentic workflow operates through a dynamic interaction loop initiated by a user prompt. The prompt conveys the user’s intent and positions the LLM as the central reasoning engine. Rather than producing an immediate static response, the model first evaluates whether external information is required. If so, it invokes provided tools to interact with the environment, for example, querying a P&ID knowledge graph database, performing web searches, or executing calculation modules. The outputs from these tools are then incorporated into the model’s context, allowing the agent to iteratively refine its understanding and generate responses grounded in validated engineering data rather than purely probabilistic inference. In process engineering, the task can be complex and may require multiple steps to solve. Consequently, recent approaches to solving this problem have focused on implementing a multi-agent framework for chemical engineering (Rupprecht et al., 2025). Rather than attempting to solve a complex engineering problem in a single agent, agentic systems decompose it into smaller, well-defined subtasks, such as retrieving equipment specifications, accessing chemical safety data sheets (MSDS), chemical properties (Bran et al., 2023), or performing vapor–liquid equilibrium (VLE) calculations. A supervisor agent can be designed to plan these subtasks and delegate them to multiple specialized agents equipped with distinct tools. This multi-agent coordination potentially reduces the likelihood of hallucinated yet plausible answers and enables iterative refinement of intermediate results when solving complex engineering problems. Furthermore, agentic workflows improve system traceability, as errors can be localized to specific reasoning steps or tool interactions, facilitating debugging and systematic improvement. 3 Methods: ChatP&ID This section presents our proposed workflow of ChatP&ID, as illustrated in Figure 2. The framework consists of three main components: (i) the agentic framework, (i) the flowsheet knowledge graph generation, and (i) the GraphRAG tools. In Section 3.1, we first describe the design and implementation of our agentic framework, including the underlying libraries, agent arrangement, and configuration of the chat interface. Next, Section 3.2 details the construction of the flowsheet knowledge graph, the abstraction process, and the associated semantic enrichment and embedding procedures. 4 ChatP&IDA PREPRINT Input Prompt Output Response Model 2 1 To o l s 2a Database Validation, Intervention Simulator... call tool tool output (context) Quality of input: •Clarity (intent) •Context (information) •Structure (organization) Figure 1: Agentic workflow illustrating how LLMs interact with external tools. The input prompt (2) provides intent, context, and structured information to the model (1). When needed, the model invokes tools (2a) to obtain additional information or perform tasks, and the tool’s output serves as context that improves the final response. Finally, Section 3.3 introduces the proposed GraphRAG methodology, which comprises ContextRAG, VectorRAG, PathRAG, and CypherRAG. 3.1 Agentic framework At the core of ChatP&ID is an LLM implemented as a reasoning engine within an agentic framework. We set up the agentic framework using the LangGraph library (LangChain, 2024a) and configured it as illustrated in Figure 3. When a user query is received, it is first routed to the ChatP&ID agent, which determines whether a GraphRAG tool is needed and selects the appropriate information-retrieval method. To support context grounding, ChatP&ID integrates a suite of graph-retrieval tools: ContextRAG, VectorRAG, PathRAG, and CypherRAG, which are described in Section 3.3. These tools query the Neo4j graph database to extract relevant nodes, edges, or sections of the knowledge graph and return them as textual information to the ChatP&ID agent. Once a tool is selected, the ChatP&ID agent also provides the necessary execution parameters for the tool, such as query mode (topology or graph) for ContextRAG (Section 3.3.1), index (local or global) for VectorRAG (Section 3.3.2), maximum traversal (depth and breadth) for PathRAG (Section 3.3.3), or query for CypherRAG. The tool then executes the retrieval task and returns the results to the ChatP&ID agent as additional context. The agent iteratively assesses whether the retrieved knowledge is sufficient to answer the user query. If additional context is required, the LLM may invoke additional GraphRAG methods until the information is sufficient or the predefined maximum number of tool calls is reached. The collected information is then aggregated and synthesized into a final, context-aware response. To enable fluid multi-turn interactions, a memory module is integrated into the workflow, allowing conversation history to persist as contextual input across dialogue turns. This ensures continuity and enhances the consistency of reasoning across successive user interactions. Additionally, token streaming is employed to progressively display generated responses, improving user experience during longer reasoning or retrieval processes. Finally, a dedicated chat interface is developed to facilitate user interaction with ChatP&ID. The interface streams tokens in real time, displays the conversation history, and provides visibility into the agent’s execution state, including tool invocation events and the corresponding retrieval results. 3.2 Flowsheet knowledge graphs This section details the construction, abstraction, semantic enrichment, and embedding of a flowsheet knowledge graph derived from DEXPI P&ID. 3.2.1 Knowledge graph generation and abstraction The knowledge graph is generated from DEXPI P&ID data using our open-source package pyDEXPI (Goldstein et al., 2025), where each pyDEXPI instance is represented as a graph node, and the class attribute is represented as a graph relationship and property. The pyDEXPI data model defines three attribute types: compositional, reference, and data. In our representation, compositional attributes are modeled as has-type relationships that form compositional edges between nodes. Reference attributes, such as send_to, control, manipulate, and other interaction-based relationships, are translated into relational edges connecting the corresponding nodes. Data attributes are preserved as node properties 5 ChatP&IDA PREPRINT transform [2] SmartP&ID Knowledge GraphAbstractions (complete, process, conceptual) Flowsheet KnowledgeGraph database Agent condense [3] and semantic enrichment digitization [1] context-enriched answer context tool calls (graph-RAG) Piping& Instrumentation Diagram (P&ID) Figure 2: ChatP&ID Workflow. The ChatP&ID takes a flowsheet knowledge graph as its input. The knowledge graph can be generated by digitizing a PDF or image (Theisen et al., 2023) and exporting it to a DEXPI-conformant smart P&ID. The DEXPI P&ID then transformed into a knowledge graph using pyDEXPI (Goldstein et al., 2025). Using the pyDEXPI object, each flowsheet element is mapped one-to-one to a node in the knowledge graph. From this knowledge graph, multiple abstraction layers are created, from a complete graph to process-level and conceptual-level. GraphRAG methods: ContextRAG, VectorRAG, PathRAG, or CypherRAG, are implemented as tools to retrieve relevant information from a knowledge graph. The LLMs are integrated into the workflow as an agent that determines the appropriate retrieval method to invoke. Finally, a context-aware response is generated to answer user queries. while maintaining the original naming conventions defined in pyDEXPI. Furthermore, the pyDEXPI class hierarchy is encoded as node labels. For the development of a chat-based assistant, it is essential that the P&ID representation incorporates comprehensive classification information (e.g., specific valve types such as globe, butterfly, needle, or gate valves) and detailed engineering specifications (e.g., fail-safe positions, nominal diameters, and material constraints). Given the broad scope of the proposed interface, the system must be able to retrieve and present all relevant P&ID data. The LPG graph format is particularly well-suited for representing such attribute-rich engineering data, as it allows properties to be directly associated with nodes and edges. Therefore, in this work, we choose LPG as the underlying graph data model and host the graph in the Neo4j database. In our previous work (Alimin et al., 2025), we constructed a knowledge graph that provided both a one-to-one mapping from pyDEXPI and a high-level abstraction. This abstraction is necessary because the one-to-one mapping captures excessive detail that often diverges from how a process engineer conceptualizes a P&ID. For instance, in the one-to-one mapping, a piping network system is modeled as a composition of segments, which in turn are compositions of individual components (e.g., valves, pipes, and fittings). This hierarchy yields a highly complex, convoluted graph structure. To address this, we previously condensed the graph to focus directly on piping components, aligning the representation 6 ChatP&IDA PREPRINT Figure 3: ChatP&ID agentic workflow integrating graph database and graph-based retrieval method. with standard P&ID interpretation. However, this reduction was performed as a single-step transformation from the complete graph to the high-level graph, thereby risking the loss of significant topological information. Therefore, in this work, we implement a multi-step graph condensation process based on node classes (e.g., piping, equipment, and instrumentation) and refine the naming convention to provide more intuitive interpretations. Specifically, we construct three levels of graph abstraction: (1) the complete-level graph, which provides a direct one-to-one mapping of all pyDEXPI entities and relationships; (2) the process-level graph, which condenses the piping system from the complete-level graph; and (3) the conceptual-level graph, which further condenses piping systems, instrumentation systems, and equipment from the process-level graph. For each abstraction, we apply: (i) pruning of domain-specific information and connection nodes, (i) collapsing low-information nodes into their parent nodes, and (i) removal of non-process-related node properties. The resulting graph is formally denoted asG = (V, E), where each nodeV contains data attributes and the corresponding pyDEXPI object, with node labels representing pyDEXPI classes, and edges E encode compositional and reference relationships extracted from pyDEXPI. 3.2.2 Knowledge graph semantic enrichment and embedding In this section, we describe the process of enriching the knowledge graph generated in Section 3.2.1 and projecting its nodes into a vector space. As discussed in Section 2.2, many GraphRAG techniques rely on vector-based retrieval to perform similarity searches over graph entities. Consequently, generating vector embeddings for each node is a prerequisite. These embeddings are critical for VectorRAG (Section 3.3.2), PathRAG (Section 3.3.3), and other methods that utilize semantic similarity indexing to identify the nodes most relevant to a given query. To set up these embeddings, we prompt LLMs to generate textual descriptions of each node that capture its semantics. We construct two types of semantic information for each node: (i) local semantics, which describe the node’s role relative to its immediate neighbors, reflecting its function within a localized subgraph, and (i) global semantics, which describe the node’s role within the entire flowsheet, providing a high-level process perspective. We design this indexing strategy to support a coarse-to-fine retrieval process: global semantics enable the identification of specific nodes across the entire graph based on query similarity, while local semantics facilitate refinement within the identified neighborhood. This local and global approach has also been studied within the graphRAG system across different setups; for instance, Edge et al. (2024) uses LLMs to generate (local) community summaries from graph nodes and edges. This community summary is subsequently aggregated for global reasoning, whereas Shi et al. (2025) uses LLM-generated semantic representations for text nodes and embeds them into graph transformer models to capture both local and global graph structural information. In our setup, both local and global semantics are generated by prompting an LLM; in this case, we use OpenAI GPT-4o to describe the node either in the context of its immediate neighbors or within the full graph (as graphml). Recent studies indicate that frontier models from Anthropic and OpenAI, such as Claude and GPT, demonstrate high proficiency in interpreting code and structured data formats, such as XML (Nam et al., 2024; DeRose, 2024). Hence, selected GPT-4o (OpenAI frontier model) based on our previous study (Alimin et al., 2025), which demonstrated its high accuracy in describing flowsheet information regardless of graph complexity. Furthermore, it maintains a reasonable token cost, 7 ChatP&IDA PREPRINT which is critical given that this algorithm is executed for every node in the graph. We implement an algorithm that collects node information, local context, and global context, packages them into a single textual input along with the prompt, and passes it to the LLM. The full semantic enrichment procedure is detailed in Algorithm 1, while the specific prompts and examples of the resulting semantic descriptions are provided in Appendix A. Once the semantic information is generated, the textual descriptions are converted into vector representations using embedding models. In this work, we use the Voyage embedding model, specifically voyage-3.5-lite, which has demonstrated high accuracy among publicly available models (Butler et al., 2025; Ponwitayarat et al., 2025; Goel et al., 2025). The model generates 1024-dimensional vectors, which serve as node embedding in downstream tasks. These embeddings enable similarity searches based on either local or global semantics in VectorRAG (Section 3.3.2) and PathRAG (Section 3.3.3). Algorithm 1 Semantic Enrichment and Embedding 1: procedure SEMANTICENRICHMENTEMBEDDING(embeddingModel) 2:Input: embeddingModel 3:Output: V.embedding▷ Semantic embeddings for each node 4:Initialize: embeddingModel, neo4jDriver, LLM ▷ Step 1: Retrieve all relevant nodes from graph 5: nodes← GETALLNODES(graph) ▷ Step 2: Generate global and local semantics 6:for each n∈ nodes do 7:global_semantic← LLM.GENERATEGLOBALSEMANTIC(n.labels, n.properties, graph) 8:neighbors← GETNEIGHBORS(n) 9:local_semantic← LLM.GENERATELOCALSEMANTIC(n, neighbors, relationships(n)) 10:n.semantic← (global_semantic, local_semantic) 11:end for ▷ Step 3: Encode semantics into vector embeddings using provided model 12:for each n∈ nodes do 13:n.embedding ← EMBEDDINGMODEL.ENCODE(n.semantic) 14:end for 15:return nodes 16: end procedure 3.3 GraphRAG for P&ID To enable information retrieval from P&ID knowledge graph during question answering, we propose four GraphRAG tools: ContextRAG, VectorRAG, PathRAG, and CypherRAG. These tools implement graph-based retrieval techniques, including LLM reads, vector search, and Cypher queries, or combinations thereof. They are implemented as LLM tools that can be invoked directly by the LLM during execution. When called, the tool executes the query algorithm on the Neo4j database and returns the results as textual context. The output can vary depending on the query and the tool, ranging from a single node or a list of nodes to a path or a section of the knowledge graph. 3.3.1 ContextRAG ContextRAG is designed to extract a condensed, noise-free textual context from the knowledge graph. As background, LLMs have demonstrated significant capabilities for interpreting structured data, including code and native graph formats such as XML (Nam et al., 2024; Alimin et al., 2025). Frontier models, such as GPT-4 and Claude, are particularly effective at analyzing XML schemas, inferring relationships from markup, and identifying structural patterns from trees. However, these capabilities degrade significantly with input length. When processing large XML files, LLMs often encounter stability issues, such as losing track of global instructions, applying inconsistent reasoning, or failing to correctly associate related yet distant elements within the file (DeRose, 2024). In this tool, the Neo4j knowledge graph is serialized to a GraphML file, an XML-based format that explicitly encodes the graph topology, including nodes, edges, and attributes, as an XML tree. However, GraphML generated from pyDEXPI exports contains extensive metadata irrelevant to semantic reasoning, which increases noise and inflates token usage. To address this, in ContextRAG, we employ a filtering mechanism that retains only essential process structures and semantics. By stripping away extraneous attributes and focusing solely on flowsheet-relevant data, the system ensures that the retrieved context is both semantically dense and token-efficient. 8 ChatP&IDA PREPRINT ContextRAG proceeds in three stages. First, the target graph is exported from the database. Second, the raw GraphML data is processed to remove non-semantic artifacts, such as internal identifiers, URIs, and vector embeddings. Finally, the graph is reconstructed according to a agent-selected simplification mode: (i) Graph mode, which preserves key node and edge attributes for detailed contextual reasoning by retaining essential P&ID information from each component, including labels, tag number, and design specifications (e.g., design pressure, temperature ratings) following the pyDEXPI data attribute schema, alongside edge labels defining relationship types; meanwhile (i) Topology mode retains only node labels and connectivity for a lightweight structural overview. This approach yields a refined graph context that leverages the LLM’s ability to parse structured XML to mitigate the stability issues arising from excessive length and noise. 3.3.2 VectorRAG VectorRAG is designed to retrieve nodes through semantic similarity search. This tool performs a vector search over the global embeddings of the knowledge graph (detailed in Section 3.2.2) to identify nodes most relevant to the user’s query. During execution, either the original user query or an LLM-refined version is transformed into a vector embedding using the Voyage-3.5-lite model, the same embedding model employed during the knowledge graph enrichment phase. This model projects the query into a 1024-dimensional vector space and then queries the pre-indexed node embeddings stored in the vector database. A cosine similarity search is performed to rank nodes by their semantic proximity to the query. Specifically, the similarity scores i between the query embeddingv q and a node embeddingv i is computed as: s i = cos(v q ,v i ) = v q ·v i ∥v q ∥v i ∥ (1) The top-kmatched nodes, along with their enriched semantic content, are returned as contextual knowledge. The execution workflow is presented in Algorithm 2, which covers vector index initialization, similarity search over embeddings, and attribute filtering. By default, VectorRAG operates using global semantic embeddings. Local semantic embeddings are used when VectorRAG is invoked within PathRAG to identify semantically related neighboring nodes. This approach enables the integration of both global and local semantic representations for retrieval in downstream tasks. Algorithm 2 VectorRAG: Semantic node retrieval 1: procedure VECTORRAG(indexName, query, topK) 2:Input: indexName∈“global_semantic_index”, “local_semantic_index” ▷ Vector index for embeddings 3:Input: query▷ User query or question 4:Input: topK (optional)▷ Max number of nodes to retrieve 5:Output: results▷ List of relevant nodes with scores 6:Initialize: embeddingModel, neo4jDriver 7: retriever ← VectorRetriever(indexName, embeddingModel, neo4jDriver) 8: ranked← retriever.search(query, topK) 9: results←∅ 10:for each item in ranked.items do 11:content← ParseNodeContent(item.content) 12:results← results∪ 13:score: item.metadata.score, 14:elementId: item.metadata.id, 15:nodeLabels: item.metadata.nodeLabels, 16:content: content 17: 18:end for 19:return results 20: end procedure 3.3.3 PathRAG In PathRAG, we extend the VectorRAG framework by integrating path-based exploration into the retrieval process. This methodology mimics the cognitive workflow of process engineers, who typically analyze P&IDs by first locating a 9 ChatP&IDA PREPRINT OffPageConector ProcesStream Pump HE-1 GloveValve OffPageConnector CWS OffPageConnector CWR TIC Actuator Tank-1 OffPageConector ProcesStream PathRAG Manage temperatureforprocessstream? startingnode: HE-1, path: HE-1 -> TIC->Actuator->GlobeValve->OPC CWR depth:4, Figure 4: Illustration of the path exploration algorithm. To answer the query “how to control process stream temperature,” VectorRAG is first applied to the global embedding space to identify a suitable starting node. The Heat Exchanger (HE) is selected as the starting point, as it is the semantically closest node to the query. From this node, two candidate paths exist: (i) a direct connection to the Globe Valve, or (i) a path through the Temperature Indicating Controller (TIC). VectorRAG is then applied to the local embedding of neighboring nodes to resolve this choice, selecting the TIC. The exploration proceeds through the Actuator and Globe Valve, and terminates at the Off-Page Connector (OPC) for Cooling Water Return (CWR). The resulting path is returned to the LLM to support the generation of the final answer. specific component or area of interest and then tracing the connected process lines to establish context. Also, answering flowsheet queries often requires an understanding of these local topological relationships rather than finding specific information. For instance, to determine the isolation procedure for a vessel, an engineer must first identify the equipment and then trace the upstream and downstream paths to locate the corresponding isolation valves. Similarly, verifying a fluid transfer operation requires starting at a pump and traversing the discharge piping to confirm it connects to the intended tank. PathRAG is designed to operate on the ’locate-and-trace’ strategy, enabling the system to derive answers from both the semantic identity of nodes and their structural connectivity. PathRAG incrementally traverses the graph along multiple paths, accumulating contextual knowledge at each step. The approach combines global and local semantic search to ensure that both relevant starting points and intermediate nodes are considered during query resolution. The procedure begins by embedding the user query and performing a global vector search to identify semantically relevant starting nodes. Each starting node serves as the root of a path exploration process, which is limited by a maximum depth and maximum breadth parameter to control computational complexity. The procedure is presented as in Algorithm 3, where at each step along a path, we do (1) accumulates context from all previously visited nodes along the path, (2) evaluates, via the LLM, whether the accumulated context suffices to answer the query, (3) generates a next-hop query to guide local semantic search for the most relevant unvisited neighbor, and (4) updates the path with the selected node and its relationships, as visualized in Figure 4. This process continues until an answer is found, the maximum path depth is reached, or no unvisited neighbors remain. Multiple paths are explored in parallel from different starting nodes to enhance computational performance and robustness. Once all paths have been explored, the final answer is determined by the presence of relevant information along each path. If no path contains relevant information, the tool returns None. If at least one path contains relevant information, the tool returns the corresponding path(s), and the final answer is constructed by the LLM, which evaluates which path provides the most relevant information to answer the query. 3.3.4 CypherRAG CypherRAG leverages an LLM to translate natural language questions into executable Cypher queries for the Neo4j database. This approach is designed to enhance safety, as the database engine acts as a validator; any malformed or hallucinated syntax is rejected during execution rather than yielding incorrect data. Central to this process is the injection of the graph schema into the LLM’s context window. Since the knowledge graph is constructed using pyDEXPI, this schema is intrinsically tied to the pyDEXPI data model. The schema is provided to the LLM as structural context alongside the user’s query, enabling the model to map natural language intent to specific node labels and relationship types. Our implementation wraps the Neo4j Python library for Cypher-based question answering (LangChain Contributors, 2025). The overall workflow is summarized in Algorithm 4. 10 ChatP&IDA PREPRINT Algorithm 3 PathRAG: Path exploration retrieval 1: procedure PATHRAG(query, maxDepth, maxBreadth) 2:Input: query▷ User query or question 3:Input: maxDepth▷ Maximum number of hops 4:Input: maxBreadth▷ Maximum parallel starting paths 5:Output: result▷ Answer with exploration path 6:Initialize: neo4jDriver, LLM ▷ Step 1: Global search for starting nodes 7: startingNodes← VectorRAG(“global_semantic_index”, query, maxBreadth ) 8:if startingNodes =∅ then return NoAnswerFound 9:end if 10: paths←∅ ▷ Step 2: Expand paths from each starting node 11:for each nodeId in startingNodes do 12:path← [nodeId] 13:visitedNodes←nodeId 14:contexts[nodeId]← GetNodeContext(nodeId) 15:accContext← BuildContext(contexts) 16:if EvaluateContext(query, accContext).hasAnswer then 17:append (path, accContext, answer) to paths 18:continue 19:end if 20:currentNode← nodeId, depth← 1 21:while depth < maxDepth do 22:neighbors← GetNeighbors(currentNode) excluding visitedNodes 23:if neighbors =∅ then break 24:end if 25:nextNode← VectorRAG(“local_semantic_index”, LLM.NextHopQuery(query, accContext)) 26:if nextNode = None then break 27:end if 28:append nextNode to path 29:visitedNodes← visitedNodes∪nextNode 30:contexts[nextNode]← GetNodeContext(nextNode) 31:accContext← BuildContext(contexts) 32:if LLM.EvaluateContext(query, accContext).hasAnswer then 33:append (path, accContext, answer) to paths 34:break 35:end if 36:currentNode← nextNode, depth← depth + 1 37:end while 38:append (path, accContext, answer) to paths 39:end for ▷ Step 3: Select best answer 40: result← LLM.SelectBestAnswer(paths) 41:return result 42: end procedure 11 ChatP&IDA PREPRINT Algorithm 4 CypherRAG: Generates and executes Cypher for grounded retrieval 1: procedure CYPHERRAG(query) 2:Input: query▷ User query or question 3:Output: answer, cypher▷ Grounded answer and executed query 4:Initialize: neo4jDriver, LLM 5: schema← GetGraphSchema()▷ Provide graph schema as context for Cypher generation 6: cypher ← LLM.GenerateCypher(query, schema)▷ LLM translates question and schema into Cypher 7: context← ExecuteCypher(cypher)▷ Run Cypher query on Neo4j for factual grounding 8: answer ← LLM.Answer(query, context)▷ LLM generates answer constrained to retrieved context 9:returnanswer, cypher 10: end procedure 4 Illustrative case study This section presents an illustrative case study used to evaluate the proposed framework, which extends to the test configuration, question sets, and evaluation procedures. 4.1 Configuration The P&ID used in this case study is the DEXPIEX01.xml from (Theißen and Wiedau, 2021), as shown in Figure 5. The flowsheet is processed through the pyDEXPI pipeline, converted into a knowledge graph, and stored in a graph database for use within the ChatP&ID workflow. The evaluation spans the following dimensions: (1) Tasks: graph summarization, graph querying, path exploration, and knowledge inference; (2) Flowsheet representation: raw image, DEXPI Proteus file, and knowledge graphs at three abstraction levels (complete, process, and conceptual); (3) Tools: ContextRAG, VectorRAG, PathRAG, CypherRAG, read-image, and read-file; and (4) LLMs: OpenAI (GPT-4o, GPT-4o-mini, GPT-5, GPT-5-mini), Anthropic (Haiku 3.5, Sonnet 3.7, Sonnet 4, Opus 4.1), and Ollama-hosted models (Llama3.1:8B, Qwen3:4B, Qwen3:8B, Qwen3:14B, GPT-OSS:20B). Online models (OpenAI and Anthropic) are accessed via API, while offline models are executed locally using Ollama on a Mac Mini M4 with 24 GB RAM. As noted earlier, direct interpretation of flowsheet images and DEXPI files by LLMs is included as a baseline method. The corresponding multimodal context is implemented by passing the P&ID image as a base64-encoded string. Modern frontier LLMs already provide native multimodal capabilities, which enable image processing without external pre- processing. For image inputs, we ensure that equipment specifications are visible by providing high-resolution images (5,485 by 3,186 pixels, 330 DPI, 1,2MB) that include the associated specification tables, as illustrated in Figure 5. The Protheus context is implemented by passing the DEXPI Protheus file as a raw string to the LLM. A tool limiter is added to limit the number of tool calls. Each tool can be called only once per test to evaluate its performance without iteration. For vector search, the maximum breadth is limited to 2 and the maximum depth to 3, considering the small size of our case P&ID. Each benchmark configuration is executed twice to reduce stochastic variability in the reported results while keeping costs limited. 4.2 Question-set To evaluate the performance of GraphRAG and baseline methods for flowsheet information retrieval, we constructed a targeted flowsheet question–answer dataset consisting of 19 QA pairs. The questions are organized into four categories, each illustrated with a representative example. This dataset is designed to test a model’s ability to: (1) extract specific or multiple pieces of information, (2) comprehend the overall process structure, (3) explore graph topology and flow paths, and (4) combine concept in diagram information with inherent engineering information in the model. Representative samples from each category are detailed below: 1. Graph query (10 questions) Objective: Evaluate the completeness of retrieving specific or multiple components and their attributes directly from the P&ID. Sample Question: (single query, 8 questions) What is the design volumetric flow rate of pump P4712? Sample Question: (multi query, 2 questions) List all valves in the P&ID along with their specifications? 12 ChatP&IDA PREPRINT Figure 5: Sample of DEXPI P&ID used in this case study (Theißen and Wiedau, 2021). 2. Path exploration (5 questions) Objective: Test the model’s ability to trace flow paths, identify alternative routes and isolation points, and infer operational or control logic from the graph. Sample Question: If you need to isolate tank T4750 from all upstream equipment, which valves would you need to close? 3. Knowledge inference (3 questions) Objective: Test the model’s ability to derive insights from the diagram and connect it to the concept from internal information in the model. Sample Question: What would be the effect if the heating fluid temperature in Heat Exchanger H1007 increased from the design temperature to 120 °C? 4. Graph summarization (1 question) Objective: Assess the model’s ability to interpret diagram components, relationships, and topology, and generate a coherent process narrative. Sample Question: Based on the P&ID, describe the process flow from the inlet to the final outlet, identifying major equipment, intermediate streams, and key control points in sequence. For each question, we provide a reference answer, to the best of our knowledge, derived from the P&ID. This reference answer will help us compare with the LLM response. The set of our Q&A is available in Appendix B. 4.3 Evaluation To assess the framework’s performance, we evaluate both the response accuracy and the computational cost. The response accuracy is assessed using two scoring approaches (1) Semantic similarity, where we compute the cosine similarity between the model-generated response and a reference answer in embedding space, as defined in Equation 1. Both the model response and the reference answer are encoded into 1024-dimensional vectors using the Voyage-3.5-lite model. (2) LLM-as-a-Judge, where we employ LLM to assign a score to model responses relative to the reference answer based on predefined scoring rubrics. The evaluation is based on four criteria: relatedness, completeness, correctness, and coherence. The detailed rubric is shown in Table 1, which is adopted from Schulze Balhorn et al. (2024). During evaluation, the LLM receives both the model output and the reference answer and is tasked with producing a score and a justification in accordance with the rubric. Scores range from 1 to 5, where higher scores indicate responses that are more aligned with the reference answer. For example, if the reference answer is “10 bar” and the model output is “5 bar,” the response would be considered incorrect but still related, partially complete, and coherent. To enable comparison with the semantic similarity metric, the rubric scores are linearly rescaled to the 0–1 range. 13 ChatP&IDA PREPRINT Table 1: Rubric criteria for scientific answer evaluation for LLM (adapted from Schulze Balhorn et al., 2024). Criterion12345 RelatednessNot related at allMostly not relatedPartly relatedMostly relatedCompletely related CompletenessIncomplete, key details missing Incomplete, insufficient details Complete, insufficient details Complete, with most details Complete and detailed answer CorrectnessCompletely incorrect Mostly incorrectPartly correctMostly correct Completely correct CoherenceUnable to draw conclusions from relevant knowledge Barely able to draw conclusions from established knowledge Can draw conclusions with some difficulty Can independently draw conclusions based on established knowledge Can independently draw correct conclusions based on state-of-the-art knowledge We use LLMs as judges due to the large number of evaluations required. Each configuration comprises 19 questions, each tested twice, and each test result is evaluated across five metrics. This yields more than 15,000 individual scoring points. Manually annotating this dataset would be labor-intensive; assuming a conservative rate of one minute per evaluation, the process would require approximately 250 person-hours (or 31 full working days). Prior studies have demonstrated the effectiveness of LLMs as judges (Tan et al., 2024; Gu et al., 2024). Notably, GPT-4 has been shown to achieve agreement rates above 80% with human experts (Zheng et al., 2023). Consequently, we adopt GPT-5-mini, a lighter variant of the latest GPT-5 model from OpenAI, as our LLM judge for this study. However, it is important to note that LLM-as-judge may still result in hallucination when scoring; hence, we require the LLM to provide a step-by-step justification for every assigned score, and we analyze the reliability of these evaluations in Section 5.1 by comparing their performance against semantic similarity metrics. The computational cost was assessed to get the overview of (i) execution time (seconds), which shows the latency between query and last token from final response, (i) cost ($), the associated cost to answer one question, reflecting the cumulative number of input and output tokens. Both answer accuracy and computational cost were evaluated using the LangSmith library (LangChain, 2024b) using our developed evaluation module. The experiments were conducted in September 2025, and the reported costs correspond to that period. Token-based cost analysis is applicable only to online models. 5 Results This section presents an overview of ChatP&ID’s performance across different configurations and evaluation methods. We begin by comparing two evaluation approaches, LLM-as-judge and semantic similarity, which will be the basis for interpreting LLM performance in the Results and Discussion sections. Then, we provide a high-level overview of the results, followed by a detailed analysis of how each configuration affects performance. Specifically, we examine the impact of flowsheet representations and graph abstraction levels, LLM selection, and GraphRAG tools across different task types. 5.1 Contrasting LLM-based scoring with semantic similarity As described in Section 4.3, we evaluate ChatP&ID responses using two scoring approaches: (i) semantic similarity and (i) LLM-as-judge. In this section, we compare these approaches based on their scoring results when evaluating LLM-generated responses. Figure 6 (left) shows the average score of each metric across different task types. Across the board, all metrics exhibit similar distributions, with a variation of approximately 15%. Overall, the LLM-as-judge scores are fairly consistent across rubrics category. Completeness tends to score the lowest because responses must include all key information to receive a full score. In contrast, relatedness generally scores highest because it requires only that the response be relevant to the reference answer, not necessarily correct (e.g., providing an incorrect pressure value for an equipment specification may still be considered highly related). Task-specific trends reveal a notable discrepancy between semantic similarity and LLM-as-judge scores for the graph query task (bottom right of the heatmap). While semantic similarity tends to score higher across other task types, the graph query result scores the lowest. This pattern is notable because, unlike other task types (where the reference answer may consist of a longer explanatory passage), the reference answer for graph queries is typically short and precise (e.g., “10 bar” or “4.0 m”). Consequently, graph queries demand exact factual matching rather than semantic or descriptive overlap. 14 ChatP&IDA PREPRINT To further investigate, we examined the distribution of semantic similarity versus LLM-as-judge scores across tasks in Figure 6 (right). The two scoring methods do not exhibit a linear correlation; instead, semantic similarity values are compressed toward the higher end. This relationship is sensitive to both the choice of embedding model (for semantic similarity) and the rubric scoring design (for LLM-as-judge). We also find that semantic similarity aligns more closely with average LLM-as-judge scores when graph query tasks are excluded. The scatter plot further shows that graph query dots deviate disproportionately from the overall trend, exhibiting substantially lower similarity scores than LLM-as-judge scores (in the middle top of the scatter plot). Completeness CorrectnessCoherenceRelatedness Semantic Similarity Path Exploration Graph Summarization Knowledge Inference Graph Query Task Type 0.490.600.620.780.75 0.530.660.670.750.84 0.550.740.700.760.83 0.680.760.770.840.65 0.00.20.40.60.81.0 Metrics 0.30.40.50.60.70.80.91.0 Semantic Similarity Score 0.2 0.4 0.6 0.8 1.0 LLM-as-judge Score Graph Query Graph Summarization Knowledge Inference Path Exploration (y = x) Quadratic fit excl. Graph Query (x², R² = 0.4) Figure 6: Evaluation of RAG strategies across task types: (left) Heatmap of response accuracy scores, (right) Rela- tionship between average LLM-as-judge scores and semantic similarity. Small random noise (±0.05) is added to LLM scores for visibility. We further examine this behavior with examples from our evaluation benchmark, shown in Figure 7. In this case, we compare responses from three LLMs: GPT-5, Claude Opus 4.1, and Claude Haiku 3.5, for finding the set pressure of the PSV. Both GPT-5 and Claude Opus 4.1 produce correct answers, differing primarily in the formulation of their responses. GPT-5 provides a concise and direct answer, "6.0 bar," for the PSV set pressure, achieving a perfect score (1.00) across all evaluation metrics, including semantic similarity. In contrast, Claude Opus 4.1 delivers a more detailed explanation. Although the answer and all supporting details are correct, its semantic similarity score is only 0.58, while the LLM-as-judge assigns a perfect score of 1.00. This discrepancy arises because the reference answer is brief, making it semantically distant from a longer, more descriptive response. In contrast, Claude Haiku 3.5 provides a similarly detailed explanation but reports an incorrect numerical value. In this instance, the semantic similarity score is comparable to that of Claude Opus 4.1, yet the LLM-as-judge assigns a low correctness score (0.2) and explicitly explains its reasoning. The relatedness score remains relatively high (0.6), as the response addresses the same topic on the PSV set pressure but reports an incorrect value. These results demonstrate that semantic similarity reflects vector-space closeness to the reference answer rather than factual or logical correctness. Consequently, a response may be correct despite low semantic similarity, while a highly similar response may still be incorrect. Moreover, LLM-as-judge evaluations can provide interpretability by generating explanations for their scores, offering insights that go beyond what semantic similarity alone can capture. Although semantic similarity is often used as a proxy for accuracy, our findings suggest that it primarily measures topical relevance rather than reasoning accuracy or correctness. Nevertheless, it remains a useful high-level indicator for semantic indexing. Based on these observations, we adopt LLM-as-judge as our primary evaluation method and report response accuracy by averaging the full set of accuracy metrics produced by the LLM-based evaluator. 15 ChatP&IDA PREPRINT What is set pressure of safety valve SV104.01? ReasoningScoreEvaluation N/A0.58Semantic Similarity LLM-as-Judge In addition to the required set pressure (6.0 bar), the answer provides several relevant valve specifications (position number, type, nominal diameter, piping class, fluid code), making it complete and detailed. 1.00-Completeness The answer logically identifies the valve from the process data and correctly concludes its set pressure, demonstrating clear and coherent reasoning. 1.00-Coherence The reported set pressure matches the reference answer exactly (6.0 bar) and the additional specification details are presented consistently with typical valve data. 1.00-Correctness The model answer directly responds to the question by giving the set pressure value for the safety valve and identifying the specific valve (SV 104.01). 1.00-Relatedness Reference answer: 6.0bar Based on the process data, I can see that safety valve SV 104.01 has a set pressure of 6.0 bar. This safety valve is a Spring LoadedGlobe Safety Valve with the following specifications: -Position number: SV 104.01 -Type: SpringLoadedGlobeSafetyValve -Set pressure (high): 6.0 bar -Nominal diameter: DN 50 -Piping class code: 73HG12 -Fluid code: MNc –Claude Opus 4.1 6.0 bar tool_call[ContextRAG] ReasoningScoreEvaluation N/A1.00Semantic Similarity LLM-as-Judge The response provides the full expected result (6.0 bar). Given the reference is a single numeric value, the answer is complete. 1.00-Completeness The model draws the correct conclusion (6.0 bar) consistent with the reference, demonstrating coherent reasoning to the expected result. 1.00-Coherence The model answer exactly matches the reference answer (6.0 bar), so it is completely related to the question. 1.00-Correctness The model's value matches the reference exactly, so the content is completely correct.1.00-Relatedness Based on the analysis of the P&ID drawing, the safety valve SV 104.01 has a set pressure of 25/60 bar, which means it is set to open at either 25 bar or 60 bar. The specific context or application would determine which precise set pressure is applicable in this case. Is there anything else you would like to know about this safety valve or the P&ID drawing? –Claude Haiku 3.5 ReasoningScoreEvaluation N/A0.48Semantic Similarity LLM-as-Judge The response gives an ambiguous '25/60 bar' statement and lacks a clear, supported numeric answer or explanation tied to the P&ID, so key details are missing. 0.40-Completeness The model attempts to draw a conclusion about set pressure but presents an unclear and unsupported interpretation ('either 25 or 60 bar'), showing weak and inconsistent reasoning. 0.40-Coherence The numeric value contradicts the reference (6.0 bar) and the '25/60 bar' claim is either incorrect or misinterpreted, so the content is essentially incorrect. 0.20-Correctness The model answer addresses the same subject (the safety valve set pressure) but does not provide the specific value from the reference; thereforeit is only partly related. 0.60-Relatedness –GPT-5 tool_call[ContextRAG] tool_call[ReadImage] Figure 7: Comparison of semantic similarity and LLM-as-judge evaluations for a graph-query task retrieving the set pressure of a safety valve: (i) GPT-5 with ContextRAG, (i) Claude Opus 4.1 with ContextRAG, and (i) Claude Haiku 3.5 using read image. 5.2 Performance overview Figure 8 summarizes GraphRAG performance across different LLM configurations and task types. The accuracy score is calculated by averaging all LLM-as-judge scores: correctness, relatedness, coherence, and completeness. Across all tasks, ContextRAG (green) achieved the highest average score (0.84), followed by Proteus context (orange) at 0.80. Notably, ContextRAG reached this performance at four times lower cost per task ($0.13) compared to Proteus context ($0.45). The next-highest-performing methods were the vector-similarity-based approaches PathRAG (purple) and VectorRAG (red), with average accuracies of 0.72 and 0.71, respectively. These methods incurred very low token costs per task ($0.010–$0.015 per question), largely due to lower token usage enabled by vector similarity search. CypherRAG (brown) followed with an average accuracy score of 0.67. Multimodal Context achieved the lowest overall average score (0.66) but also the lowest token cost per task ($0.009). largely due to the lower cost of multimodal tokens (per tile) compared to text tokens, with a variable of image size and resolution. However, its performance was highly task-dependent: for graph queries and path exploration that require precise information retrieval (e.g., equipment specifications, flow tracing), accuracy dropped to 0.65 and 0.63, respectively. Finally, it is important to note that these average scores are influenced by the choice of LLM, as increasing flowsheet representation and tool complexity require more capable (larger) models, a factor we discuss further in the next section. 5.3 Effect of flowsheet representation We evaluate LLMs’ performance across different flowsheet representations and levels of graph abstraction, as sum- marized in Table 2. Overall, representing a P&ID as a graph at any level of abstraction consistently improves LLM performance compared to directly providing the raw flowsheet image or Proteus file. For flagship models such as GPT-5, accuracy increases by approximately 5% on average when using a conceptual-level graph as context rather than the Proteus file. In addition to accuracy gains, the conceptual graph substantially reduces computational cost by up to 85% relative to the Proteus file, resulting in an absolute reduction of approximately $0.15 per task. Smaller models show more significant improvements; for example, GPT-4o mini achieves approximately 7% higher accuracy with conceptual graph representations compared to direct Proteus file input, a trend also observed in other online LLMs. 16 ChatP&IDA PREPRINT Graph Query Path Exploration Knowledge InferenceGraph Summarization Task Type 0.0 0.2 0.4 0.6 0.8 1.0 Response Accuracy 0.92 $0.127 0.72 $0.133 0.79 $0.137 0.73 $0.145 0.79 $0.008 0.62 $0.012 0.67 $0.017 0.65 $0.017 0.72 $0.009 0.56 $0.018 0.63 $0.018 0.57 $0.022 0.71 $0.005 0.57 $0.007 0.65 $0.016 0.60 $0.012 0.65 $0.006 0.63 $0.008 0.72 $0.015 0.70 $0.020 0.83 $0.426 0.72 $0.489 0.83 $0.482 0.79 $0.500 Legend Tool Configurations: ContextRAG PathRAG VectorRAG CypherRAG MultimodalContext ProteusContext Model Vendors: OpenAI Anthropic Ollama Token Cost: Low (Q1: $0.0002) Medium (Q2: $0.0060) High (Q3: $0.0367) Figure 8: Performance overview of ChatP&ID. The scatter plot groups results by four tasks: graph query, path exploration, knowledge inference, and graph summarization. The y-axis represents the response accuracy, calculated by averaging all LLM-as-judge metrics. Data points are color-coded by tool configuration and shaped by the LLM model host (OpenAI, Anthropic, Ollama). The size of each marker indicates the token cost per QA pair, with larger markers corresponding to higher costs. Horizontal bars with annotations display the mean accuracy score and mean cost for each tool configuration. The average of the accuracy score and cost excludes the Ollama-hosted model, as not all tasks can be solved by the model. Offline Ollama hosted models that we selected (Llama3.1, GPT-OSS, and Qwen3) were excluded from this evaluation due to limitations in processing multimodal images and the extended time required to parse Proteus files. The complete set of results is provided in the Appendix C. When comparing levels of graph abstraction, flagship models such as GPT-5 leverage the complex relationships within complete graphs to achieve high accuracy on single-query tasks. However, other tasks achieve higher accuracy when using higher-level graph abstractions. Smaller models, such as GPT-4o mini, perform best at the highest level of abstraction, with accuracy decreasing as graph complexity increases across all task types. Directly providing raw flowsheet images yielded the lowest overall accuracy across the evaluation, averaging 0.76 for GPT-5 and 0.55 for GPT-4o-mini. The most significant performance degradation occurred in tasks requiring high technical precision, such as single- and multi-query graph retrieval. Conversely, vision-based methods performed best in high-level overview tasks, including graph summarization and knowledge inference. While this multimodal approach offers the lowest operational cost of all tested methods, its reliability varies significantly across task types. Consequently, while raw image processing may suffice for general descriptive tasks, it remains inadequate for complex industrial reasoning compared to the Proteus context or the GraphRAG method. 5.4 LLM model evaluation and selection We assessed the performance of various LLMs across GraphRAG methods, multimodal context, and Proteus context. Our evaluation included models from Anthropic, OpenAI, and Ollama, tested across different tool configurations. For the GraphRAG implementation, we use conceptual-level knowledge graphs as the flowsheet graph representation; as discussed in Section 5.3, this format consistently yields the highest accuracy across both large and small models, 17 ChatP&IDA PREPRINT Table 2: Response accuracy and token cost per task across different knowledge graph abstractions and flowsheet representations. Tools ContextRAG is used to query from knowledge graphs, image read for flowsheet image, and file read for Proteus File. GPT-5 and GPT-4o-mini results are for comparison. The response accuracy is the average of LLM-as-judge score: completeness, coherence, correctness, and relatedness. RepresentationGraph Query (single query) Graph Query (multi-query) Graph Sum- marization Knowledge Inference Path Exploration Average GPT-5 Graph Abstractions - Conceptual Level0.91 / $0.0111.00 / $0.0331.00 / $0.0640.98 / $0.0550.92 / $0.0270.94 / $0.027 - Process Level0.93 / $0.0350.96 / $0.0601.00 / $0.0960.98 / $0.0760.87 / $0.0630.93 / $0.055 - Complete Level0.95 / $0.0860.75 / $0.1261.00 / $0.1680.98 / $0.1260.82 / $0.1320.90 / $0.113 Flowsheet Image0.79 / $0.0020.61 / $0.0220.93 / $0.0270.87 / $0.0200.67 / $0.0080.76 / $0.001 Proteus File0.89 / $0.1520.74 / $0.1951.00 / $0.2160.98 / $0.1910.87 / $0.1840.89 / $0.175 GPT-4o-mini Graph Abstractions - Conceptual Level0.98 / $0.0010.78 / $0.0010.50 / $0.0010.63 / $0.0010.61 / $0.0010.78 / $0.001 - Process Level0.91 / $0.0040.66 / $0.0040.53 / $0.0040.61 / $0.0030.55 / $0.0040.72 / $0.004 - Complete Level0.93 / $0.0100.58 / $0.0100.50 / $0.0100.61 / $0.0070.53 / $0.0100.71 / $0.010 Flowsheet Image0.57 / $0.00010.33 / $0.00030.68 / $0.00040.62 / $0.00030.55 / $0.00010.55 / $0.0002 Proteus File0.85 / $0.0180.48 / $0.0180.58 / $0.0190.74 / $0.0120.60 / $0.0180.71/ $0.017 ensuring a fair comparison when varying model scale. Additionally, multimodal and Proteus contexts were evaluated for all models to serve as performance baselines. The results are summarized in Table 3. Overall, larger models tend to achieve higher accuracy, though at the cost of greater computational resources. Online models from Anthropic and OpenAI exhibited an average task completion time of approximately 30 seconds, peaking at 90 seconds when using the PathRAG tool, due to the path exploration algorithm’s longer execution time. While individual inference times are not detailed here (refer to the Appendix C for the full breakdown), they stand in contrast to locally hosted Ollama models. These open-source offline configurations required more processing time, with durations ranging from several minutes to upwards of 15 minutes per task. Note that these offline Ollama-hosted models were run on a Mac mini M4. For online models, ContextRAG consistently yielded the highest accuracy across tasks (average 0.84) for both Anthropic and OpenAI models. GPT-5-mini and GPT-5 demonstrated the highest accuracy, with GPT-5-mini achieving it at a fivefold lower cost, and reliably handling multimodal and Proteus contexts with an average accuracy of 0.85. For offline models, ContextRAG scores were lower for smaller models, reflecting the limited reasoning capability. Larger offline models achieved higher accuracy but incurred significantly longer execution times. Tools such as VectorRAG and PathRAG improved offline model accuracy, increasing the score from 20% to 40%. VectorRAG tasks were generally completed within one minute, while PathRAG tasks took around two minutes due to the need to traverse multiple nodes. CypherRAG tasks were computationally more complex, and only a few offline models could produce answers within the practical 15-minute time limit. Meanwhile, Proteus contexts presented additional challenges for offline models. In many cases, these offline models failed to produce results within the 15-minute cutoff. When answers were eventually produced, they were generally unsatisfactory and prone to hallucinations. The selected offline model does not support multimodal context. 5.5 Tool performance comparison We examine GraphRAG’s performance across different query tasks and compare it with baseline methods. For this analysis, we selected GPT-5 mini as ChatP&ID LLM, following the findings in Section 5.4. This model was chosen for its consistently high performance across GraphRAG tools, multimodal, and Proteus contexts. By utilizing a model that performs reliably across all input formats, we ensure an equitable comparison of the underlying tool capabilities. The comparative results are summarized in Table 4. 18 ChatP&IDA PREPRINT Table 3: LLMs performance across GraphRAG tools, multimodal context, and Proteus context. Various models from Anthropic, OpenAI, and Ollama are tested. Anthropic and OpenAI are compared by the response accuracy and token cost per task. While the Ollama model is measured by response accuracy and the time needed to complete the task. The Ollama was run in a Mac mini M4 with 24GB RAM. The flowsheet is represented as a conceptual graph for GraphRAG, a flowsheet image for Multimodal Context, and a Proteus XML file for Proteus Context. The response accuracy is the average of LLM-as-judge score: completeness, coherence, correctness, and relatedness. ModelContextRAGVectorRAGPathRAGCypherRAGMultimodal Context Proteus Context Anthropic (response accuracy / cost per task) Claude-3-5-Haiku0.82 / $0.0080.71 / $0.0030.72 / $0.0020.54 / $0.0010.59 / $0.0020.76 / $0.123 Claude-3-7-Sonnet0.88 / $0.0320.71 / $0.0150.72 / $0.0090.61 / $0.0050.62 / $0.0070.81 / $0.464 Claude-Sonnet-40.87 / $0.0330.71 / $0.0140.72 / $0.0090.69 / $0.0050.68 / $0.0090.81 / $0.465 Claude-Opus-4-10.85 / $0.162 0.69 / $0.0680.72 / $0.0450.68 / $0.0240.66 / $0.0380.78 / $2.077 OpenAI (response accuracy/ cost per task) GPT-4o-mini0.78 / $0.0010.64 / $0.0002 0.64 / $0.0002 0.54 / $0.0001 0.55 / $0.00020.71 / $0.017 GPT-4o0.83 / $0.0270.64 / $0.0030.63 / $0.0030.60 / $0.0020.58 / $0.0020.73 / $0.286 GPT-5-mini0.91 / $0.004 0.82 / $0.0020.83 / $0.0020.86 / $0.0020.83 / $0.0020.88 / $0.034 GPT-50.94 / $0.0270.79 / $0.0170.80 / $0.0150.79 / $0.0120.76 / $0.0100.89 / $0.175 Ollama (response accuracy / execution time per task) LLaMA3.1:8B0.28 / 50s0.59 / 13s0.62 / 100s**** Qwen3:4B0.37 / 275s0.63 / 54s0.68 / 141s**** Qwen3:8B0.35 / 173s0.67 / 51s0.70 / 145s0.33 / 86s*** Qwen3:14B0.43 / 284s0.63 / 84s0.68 / 139s0.52 / 273s*** GPT-OSS:20B0.30 / 380s0.72 / 38s0.70 / 81s0.45 / 73s*** * Model exceeded the 900s execution time. ** The model did not have multimodal capability. ContextRAG consistently achieves the highest overall accuracy (0.91) across all task types, with a low cost per task ($0.004) and a fast execution time (24s). VectorRAG and PathRAG have similar average accuracies (0.82-0.83), but their performance varies across tasks. VectorRAG is more accurate on graph query tasks, while PathRAG achieves better accuracy on graph summarization, knowledge inference, and path exploration. PathRAG achieves this higher score with a path-focused algorithm at the cost of longer execution time. CypherRAG performs well with an overall average score of 0.86, peaking in graph summarization (1.00) and single- graph queries (0.88). Among the baselines, the multimodal context is in the lower end, with an average accuracy score of 0.83. However, it remains a cost-effective alternative that performs well on high-level tasks such as summarization (0.95) and knowledge inference (0.91), though it struggles with precision-intensive tasks such as path exploration (0.80) and graph queries (0.79). In contrast, the Proteus Context achieved a high accuracy score of 0.88, but at a significantly higher cost ($0.034 per task) and a longer average runtime of 52 seconds. A comprehensive breakdown of these results is available in the Appendix C. 6 Discussions In this section, we analyze how variations in P&ID representation, querying tools, and LLM selection influence the observed results and discuss the implications for system design, efficiency, and accuracy. 6.1 Role of representation in performance and cost Our findings indicate that P&ID representation provides modest improvements in accuracy for frontier models but yields substantial cost savings. Frontier models, such as GPT-5, possess robust reasoning capabilities for complex patterns (Du et al., 2025; Ke et al., 2025), allowing them to handle dense information sources, such as Proteus XML or complete-level graphs. However, this depth comes at a significant computational premium. For instance, processing a 19 ChatP&IDA PREPRINT Table 4: Tool performance across tasks: response accuracy, cost per task ($), and execution time (s). The LLM examined is GPT-5-mini, the graph abstraction at the conceptual level is used to test GraphRAG, the image read tool is used to read Multimodal Context, and file read for Proteus Context. The response accuracy is the average of LLM-as-judge score: completeness, coherence, correctness, and relatedness. ToolGraph Query (single) Graph Query (multi) Graph Sum- marization Knowledge Inference Path Exploration Average response accuracy ContextRAG0.890.951.000.980.880.91 VectorRAG0.900.790.780.870.680.82 PathRAG0.870.650.880.880.790.83 CypherRAG0.880.810.930.860.840.86 Multimodal Context0.790.850.950.910.800.83 Proteus Context0.900.791.000.990.790.88 cost per task ($) ContextRAG0.00210.00580.00990.00780.00450.0044 VectorRAG0.00120.00210.00380.00480.00230.0023 PathRAG0.00090.00260.00300.00480.00210.0021 CypherRAG0.00040.00230.00280.00450.00140.0016 Multimodal Context0.00050.00220.00480.00460.00120.0018 Proteus Context0.03030.03930.04000.03630.03580.0342 execution time (s) ContextRAG7.49s34.36s62.53s48.86s24.91s24.33s VectorRAG14.04s20.52s31.93s36.16s34.05s24.42s PathRAG51.62s40.54s24.29s45.92s76.41s54.64s CypherRAG19.35s51.99s52.30s74.94s42.60s39.42s Multimodal Context18.48s42.67s100.35s103.02s44.59s45.55s Proteus Context15.32s96.70s140.49s69.96s64.52s52.05s Proteus context requires 150K input tokens (around $0.175 per task in GPT-5), whereas a complete graph uses 66K tokens ($0.113) and a process graph utilizes 32K tokens ($0.055). At the highest level of abstraction, a conceptual graph requires only 7K tokens ($0.027). Beyond the cost reduction, there is also a modest improvement (by 5%) in response accuracy when using a conceptual graph than the Proteus context. By contrast, for smaller, more economical models such as GPT-4o mini, the input representation plays a much more decisive role in accuracy. Semantically dense and curated graph contexts help these models by removing noise and highlighting critical relationships. Proteus files often contain machine-oriented artifacts, such as URIs and structural metadata, which introduce "semantic noise" that can distort meaningful information transfer (Noruzi, 2024; Hockett, 1952). Smaller models frequently struggle to filter out this noise, whereas flagship models can navigate it successfully. This suggests a diverging design requirement: while larger models benefit from abstraction primarily for cost-efficiency, smaller models require high-level abstraction simply to maintain operational accuracy. The limitations of multimodal representation underscore a trade-off between cost and precision. Although raw image inputs incur minimal token costs, they suffer from substantial performance penalties for specific graph-based queries. A common failure mode involves misreading textual data from the images; for example, in Figure 7 (i), the model failed to correctly identify PSV set pressure. The internal pre-processing and downscaling inherent in LLM vision encoders likely contributed to these errors. However, there is no publicly available information on how this internal image processing is performed; hence, it is difficult to improve it. Furthermore, it is worth noting that the P&ID image we use in this study is a relatively simple, high-resolution, single-page document (5,485 by 3,186 pixels at 330 DPI, 1.2MB). In contrast, typical industrial P&IDs are far denser, with more objects and details per page, resulting in smaller 20 ChatP&IDA PREPRINT font and symbol sizes. This may further reduce accuracy when represented as an image. Thus, while multimodal inputs remain satisfactory for general image summarization for high-level information, they are currently insufficient for tasks requiring high-fidelity data extraction. Overall, these findings underscore that careful representation design is essential for optimizing both cost and accuracy, particularly when deploying smaller models or large-scale multi-agent systems, especially in process engineering, where incorrect information can have severe consequences. Overall, these findings highlight that careful representation design can lead to meaningful improvements in both cost efficiency and task accuracy, particularly when deploying models or operating large-scale multi-agent systems. 6.2 Strengths and limitations of tooling approaches The evaluation of ContextRAG demonstrates the effectiveness of graph abstraction for efficient knowledge retrieval, providing a robust balance of accuracy, cost, and execution speed. By pre-processing the graph to be semantically dense and filtering out irrelevant metadata (semantic noise), ContextRAG minimizes the number of processed tokens. However, its primary limitation emerges when flowsheets span multiple pages; because the entire graph must be passed to the LLM for each inference, costs scale linearly with graph size. To mitigate this, we implemented a "topology mode" that passes only the lightweight structural framework without data attributes. While this reduces overhead, extracting specific parameters still requires specialized tooling, necessitating more granular methods such as VectorRAG, PathRAG, and CypherRAG. A comparison between VectorRAG and PathRAG reveals distinct operational strengths despite their shared vector- based foundation. PathRAG employs iterative traversal to improve its ability to locate specific technical nodes, but this precision comes at the cost of higher latency (averaging 54.6s per query). Conversely, VectorRAG leverages precomputed embeddings to cover a broader semantic space more rapidly and at a lower per-query cost. These methodologies represent an evolving frontier in graph-based retrieval, with significant potential for refinement in future research, alongside emerging research on graph traversal (Edge et al., 2024; Sun et al., 2023; Ma et al., 2024; He et al., 2024). CypherRAG demonstrates that schema-aware querying can achieve reasonable accuracy, though its performance is heavily dependent on the underlying model. In our evaluation, only frontier models such as GPT-5 and GPT-5-mini achieve average accuracies above 0.75. In the experiment, to test the model’s capability, we limit the query generation to a single attempt without iterative refinement. Because LLMs lack innate knowledge of the global graph’s connectivity during initial query generation, query errors are frequent. A standard mitigation strategy integrates database-aware verification and syntactic feedback loops, enabling the LLM (or a human-in-the-loop) to refine the query until it is valid. This feedback-and-iteration approach has also been highlighted in prior studies (Gusarov et al., 2025; Yang et al., 2025; Gupta et al., 2025). However, as P&ID complexity increases due to additional components and process variations, the graph size grows substantially, even if the schema remains relatively similar. This structural expansion complicates Cypher formulation; while the LLM may recognize all possible connections, it often struggles to identify the specific path required, necessitating multiple reasoning cycles to produce a functional query. Meanwhile, in this study, we used a standard Neo4j implementation; further customizing the Cypher generation for the pyDEXPI graph could significantly improve the robustness of LLM query generation. Ultimately, while this evaluation focuses on a single-page, simple P&ID, vector indices and schema-based methods (VectorRAG, PathRAG, and CypherRAG) will become increasingly important as diagrams grow in scale. These approaches possess the unique potential to offload query processing from the LLM to the database layer, which is essential for maintaining performance in large-scale industrial applications. 6.3 Considerations for LLM selection, common failure, and deployment In Section 5.4, we showed that LLM performance reflects a clear trade-off between model size, accuracy, and computational cost across task types. Here, we highlight practical insights for deploying LLMs in agentic systems for P&ID tasks, as well as common failure modes observed during deployment. LLM tasks for P&ID systems span a wide spectrum, ranging from highly precise information queries that require accuracy to higher-level summarization tasks that demand broader graph-level understanding. Larger models are generally advantageous due to their longer context windows and greater model complexity, but at the cost of com- putational power. Meanwhile, smaller online models remain highly capable but are constrained by shorter context lengths, requiring careful context management. In the flowsheet setting, this implies maintaining dense, noise-free graph representations to maximize the amount of useful context. Offline models, although limited to smaller sizes in our experiments (≤20B parameters), can also be viable when combined with effective retrieval mechanisms. In particular, structured representations such as VectorRAG and PathRAG can compensate for limited model size by improving 21 ChatP&IDA PREPRINT retrieval accuracy. In these tool-usage cases, more of the decision-making is handled by deterministic algorithms rather than the LLM, reducing the need for more complex models. Examining agent states in our evaluation bench reveals several recurring failure modes in LLM-based workflows: •The model is too small: Smaller models may struggle to interpret complex GraphML or Proteus XML structures, leading to incomplete or incorrect outputs. Similar effects are observed in multimodal models. • Too much semantic noise: Machine-oriented artifacts (e.g., URIs, raw identifiers, structural metadata) introduce semantic noise that smaller models in particular struggle to filter. •Context is too long: While modern models support long context windows and caching methods, using the entire context window does not necessarily improve accuracy and often increases token cost and inference time. •Infinite tools iteration: Tools can improve performance, but poorly constrained agents may invoke unproductive tool-call loops that exhaust context, inflate token costs, and significantly increase execution time. Prompt strategy constraints and tool-call limiters are required to prevent this failure mode. •Underspecified response formats: Without explicit output constraints, agents may produce verbose and unfocused answers, commonly known as LLMs yapping (Borisov et al., 2026). Specifying the preferred response format via a system prompt and enforcing structured outputs via schemas mitigate this behavior. • Impractical time: Offline models, such as those deployed via Ollama, frequently fail to complete complex tasks within practical time limits when run on a local desktop machine. •Inefficient prompt reuse and caching: Repeatedly passing large static artifacts (e.g., P&ID diagrams or lengthy system prompts) in every inference incurs unnecessary token cost and latency. Modern frontier LLM platforms (e.g., OpenAI GPT, Anthropic Claude, Google Gemini) support prompt caching or persistent context mechanisms - some of which are automatic, others that require explicit configuration. Leveraging these features amortizes token usage across multi-step workflows, significantly reducing long-term costs. During deployment, managing slow execution times and achieving reliable end-to-end accuracy often requires substantial compute resources. To avoid a sluggish user experience, it is critical to provide continuous feedback on task progress, including which tools are invoked, the agent’s current state, and the latest execution status. Hence, in our ChatP&ID, we use the streaming token method and provide information on the latest workflow status. To further optimize execution time and accuracy, we also suggest that multi-agent architectures may offer a promising deployment strategy. In such configurations, a more capable online model can act as a supervisory coordinator, overseeing multiple tool-augmented agents executing queries in parallel (Rupprecht et al., 2025). Their outputs can then be compared, verified, or aggregated by the supervisor agent to improve both accuracy and robustness. 7 Conclusion We introduced ChatP&ID, a framework that enables intuitive interaction with smart P&IDs using GraphRAG and knowledge graphs. In this framework, DEXPI P&IDs are encoded as a knowledge graph, and state-of-the-art GraphRAG methods are implemented to allow LLMs to query information directly from the P&ID graph database. We evaluated the framework across diverse query tasks, graph abstractions, GraphRAG methods, and LLM selections, benchmarking performance against baselines that provide raw P&ID images or DEXPI smart P&ID files directly to the LLM. This work demonstrates the potential of GraphRAG for engineering diagrams by enabling grounded natural-language interaction with structured P&IDs. Our results demonstrate that graph-based representation improves LLM accuracy while significantly reducing token costs. Among the GraphRAG methods tested (ContextRAG, VectorRAG, PathRAG, and CypherRAG), ContextRAG achieved the highest accuracy, lowest cost per task, and fastest execution time. For offline models, vector-based approaches such as VectorRAG and PathRAG further improve accuracy by shifting the workload to vector-similarity algorithms. Additionally, we found that using LLMs as judges yields more reliable scores than traditional semantic similarity metrics, which measure semantic distance but do not always capture logical correctness. During development and testing, we identified opportunities to further enhance framework accuracy, such as imple- menting a multi-agent system in which multiple tools operate concurrently, overseen by a supervisor agent that makes decisions based on worker-agent outputs (Rupprecht et al., 2025). While CypherRAG already performs well, its effectiveness could be further improved by designing a specialized Cypher query generator tailored for pyDEXPI graphs. Future work will expand these methods into broader tools, including flowsheet modification, and extend their application to help process engineers with daily tasks such as HAZOPs. 22 ChatP&IDA PREPRINT Acknowledgement We gratefully acknowledge the support from the Indonesia Endowment Fund for Education Agency (LPDP; A, 202401220100409). References Alimin, A. A., Goldstein, D. P., Balhorn, L. S., and Schweidtmann, A. M. (2025). Talking like piping and instrumentation diagrams (p&ids). arXiv preprint arXiv:2502.18928. Baken, N. (2020). Linked data for smart homes: Comparing rdf and labeled property graphs. In LDAC2020—8th Linked Data in Architecture and Construction Workshop, pages 23–36. Balhorn, L. S., Degens, K., and Schweidtmann, A. M. (2025). Graph-to-sfiles: Control structure prediction from process topologies using generative artificial intelligence. Computers & Chemical Engineering, page 109121. Barrasa, J. and Webber, J. (2023). Building knowledge graphs. " O’Reilly Media, Inc.". Borisov, V., Gröger, M., Mikhael, M., and Schreiber, R. H. (2026). Do chatbot llms talk too much? the yapbench benchmark. arXiv preprint arXiv:2601.00624. Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, P. (2023). Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Butler, U., Butler, A.-R., and Malec, A. L. (2025). The massive legal embedding benchmark (mleb). arXiv preprint arXiv:2510.19365. Dang, H. A., Tran, V., and Nguyen, L.-M. (2025). Survey and analysis of hallucinations in large language models: attribution to prompting strategies or model behavior. Frontiers in Artificial Intelligence, 8:1622292. DeRose, S. J. (2024). Can llms help with xml? In Balisage: The Markup Conference. Di Pierro, D., Ferilli, S., and Redavid, D. (2023). Lpg-based knowledge graphs: A survey, a proposal and current trends. Information, 14(3):154. Du, Z., Kang, H., Han, S., Krishna, T., and Zhu, L. (2025). Ockbench: Measuring the efficiency of llm reasoning. arXiv preprint arXiv:2511.05722. Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., and Larson, J. (2024). From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. Eibeck, A., Lim, M. Q., and Kraft, M. (2019). J-Park Simulator: An ontology-based platform for cross-domain scenarios in process industry. Computers & Chemical Engineering, 131:106586. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1). Ghallab, M., Nau, D., and Traverso, P. (2004). Automated Planning: theory and practice. Elsevier. Goel, S., Lee, R. J., and Ramchandran, K. (2025). Sage: A realistic benchmark for semantic understanding. arXiv preprint arXiv:2509.21310. Goldstein, D. P., Alimin, A. A., Schulze Balhorn, L., and Schweidtmann, A. M. (2025). pydexpi:A Python framework for piping and instrumentation diagrams using the DEXPI information model. In Proceedings of the 35th European Symposium on Computer Aided Process Engineering (ESCAPE35), Ghent, Belgium. Gowaikar, S., Iyengar, S., Segal, S., and Kalyanaraman, S. (2024). An agentic approach to automatic creation of p&id diagrams from natural language descriptions. arXiv preprint arXiv:2412.12898. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024). A survey on llm-as-a-judge. The Innovation. Gupta, M., Wei, C., Czerniawski, T., and Eiris, R. (2025). Pidqa—question answering on piping and instrumentation diagrams. Machine Learning and Knowledge Extraction, 7(2):39. Gusarov, A., Volkova, A., Khrulkov, V., Kuznetsov, A., Maslov, E., and Oseledets, I. (2025). Multi-agent graphrag: A text-to-cypher framework for labeled property graphs. arXiv preprint arXiv:2511.08274. 23 ChatP&IDA PREPRINT Hasan, M. T., Waseem, M., Kemell, K.-K., Khan, A. A., Saari, M., and Abrahamsson, P. (2025). Engineering rag systems for real-world applications: Design, development, and evaluation. In Euromicro Conference on Software Engineering and Advanced Applications, pages 143–158. Springer. He, J., Ma, M. D., Fan, J., Roth, D., Wang, W., and Ribeiro, A. (2024). Give: Structured reasoning of large language models with knowledge graph inspired veracity extrapolation. arXiv preprint arXiv:2410.08475. Hockett, C. F. (1952). An approach to the quantification of semantic noise. Philosophy of science, 19(4):257–260. Ke, Z., Jiao, F., Ming, Y., Nguyen, X.-P., Xu, A., Long, D. X., Li, M., Qin, C., Wang, P., Savarese, S., et al. (2025). A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037. Kuratov, Y., Bulatov, A., Anokhin, P., Sorokin, D., Sorokin, A., and Burtsev, M. (2024). In search of needles in a 11m haystack: Recurrent memory finds what llms miss. arXiv preprint arXiv:2402.10790. LangChain (2024a). Langgraph: Building resilient language agents as graphs. Accessed: 2026-02-18. LangChain (2024b). Langsmith: A platform for building, testing, and monitoring llm applications. Accessed: 2026-02-18. LangChain Contributors (2025). Neo4j cypher integration — langchain documentation. Accessed: 2025-02-03. Latora, V., Nicosia, V., and Russo, G. (2017). Complex networks: principles, methods and applications. Cambridge University Press. Lee, J., Park, S., Oh, S., and Ma, B. (2026). Can large language models automate the hazop process without human intervention? Safety Science, 194:107039. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474. Li, J., Sun, A., Han, J., and Li, C. (2020). A survey on deep learning for named entity recognition. IEEE transactions on knowledge and data engineering, 34(1):50–70. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173. Ma, S., Xu, C., Jiang, X., Li, M., Qu, H., and Guo, J. (2024). Think-on-graph 2.0: Deep and interpretable large language model reasoning with knowledge graph-guided retrieval. arXiv e-prints, pages arXiv–2407. Medhane, S., Khalkar, R., Dhumane, A., Mestry, R., Khan, A., and Agarwal, S. (2025). Automated counting of piping and instrumentation diagram using artificial intelligence. Journal of Integrated Science and Technology, 13(6):1147–1147. Morbach, J., Yang, A., and Marquardt, W. (2007). Ontocape—a large-scale ontology for chemical process engineering. Engineering applications of artificial intelligence, 20(2):147–161. Mukharror, D. A., Susilo, H., and Putra, Z. A. (2025). On the use of artificial intelligent with large language model as a tool for evaluating quality of hazard and operability study. In 2025 9th International Conference on Instrumentation, Control, and Automation (ICA), pages 337–342. IEEE. Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., and Myers, B. (2024). Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. Noruzi, A. (2024). Semantic noise in information representation and retrieval. informology. Pan, J. Z. (2009). Resource description framework. In Handbook on ontologies, pages 71–90. Springer. Ponwitayarat, W., Ng, R., Montalan, J. R., Aung, T., Ngui, J. G., Susanto, Y., Tjhi, W., Tasawong, P., Cambria, E., Chuangsuwanich, E., et al. (2025). Sea-bed: Southeast asia embedding benchmark. arXiv preprint arXiv:2508.12243. Radford, A. (2018). Improving language understanding with unsupervised learning. OpenAI Res. Rupprecht, S., Gao, Q., Karia, T., and Schweidtmann, A. M. (2025). Multi-agent systems for chemical engineering: A review and perspective. Russell, S., Norvig, P., and Intelligence, A. (1995). A modern approach. Artificial Intelligence. Prentice-Hall, Egnlewood Cliffs, 25(27):79–80. Schulze Balhorn, L., Seijsener, N., Dao, K., Kim, M., Goldstein, D. P., Driessen, G. H., and Schweidtmann, A. M. (2025). Rule-based autocorrection of piping and instrumentation diagrams (p&ids) on graphs. arXiv e-prints, pages arXiv–2502. 24 ChatP&IDA PREPRINT Schulze Balhorn, L., Weber, J. M., Buijsman, S., Hildebrandt, J. R., Ziefle, M., and Schweidtmann, A. M. (2024). Empirical assessment of chatgpt’s answering capabilities in natural science and engineering. Scientific Reports, 14(1):4998. Schweidtmann, A. M. (2024). Generative artificial intelligence in chemical engineering. Nature Chemical Engineering, 1(3):193–193. Shi, C., Tao, Y., Zhang, H., Wang, L., Du, S., Shen, Y., and Shen, Y. (2025). Deep semantic graph learning via llm based node enhancement. arXiv preprint arXiv:2502.07982. Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L. M., Shum, H.-Y., and Guo, J. (2023). Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697. Szlobodnyik, G. (2025). Feedback-guided prompt injection defense in retrieval-augmented text-to-cypher generation. Available at SSRN 5669662. Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. (2024). Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. Theisen, M. F., Flores, K. N., Schulze Balhorn, L., and Schweidtmann, A. M. (2023). Digitization of chemical process flow diagrams using deep convolutional neural networks. Digital Chemical Engineering, 6:100072. Theisen, M. F., Meesters, G. M., and Schweidtmann, A. M. (2025). Graph neural networks for soft sensors: Learning from process topology and operational data. Computers & Chemical Engineering, page 109532. Theißen, M. and Wiedau, M. (2021). DEXPI P&ID specification. DEXPI Initiative. Version 1.3. Toghraei, M. (2019). Piping and instrumentation diagram development. John Wiley & Sons. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. Yang, N., Reynolds, J., Prast, L., and Corvino, R. (2025). Askgraph: A dependency-aware code assistant powered by code graphs and llm-generated cypher queries. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 644–655. IEEE. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. (2022). React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623. 25 ChatP&IDA PREPRINT A Generation of Local and Global Semantic Descriptions and Embedding This appendix describes how semantic descriptions are generated for each node in the process flowsheet graph and embedded in the vector space. Two complementary semantic views are created: • Global semantic: the functional role of a node in the complete flowsheet. • Local semantic: the immediate relational context of a node with its first-level neighbors. Both descriptions are produced using an LLM that receives structured graph information extracted from Neo4j. A.1 Global Semantic Generation The global semantics describes the overall purpose of a component within the full process flowsheet. The LLM receives the node attributes together with the complete graph representation, enabling reasoning about system-wide function. Global Semantic Prompt <instruction> You are a process engineering assistant. Describe the role and function of this node in the process flowsheet (global context), based on given information. Focus on what the equipment/component does, using its name or tag (not IDs). Be clear and concise. </instruction> <node> Labels: labels Properties: properties_json This node is a component of a process flowsheet. </node> <flowsheet> full_graph_representation </flowsheet> A.2 Local Semantic Generation The local semantics describes how a node interacts with its immediate neighbors, capturing direct process connectivity such as upstream and downstream relationships. Local Semantic Prompt <instruction> You are a process engineering assistant. Describe the local context of this node in the process flowsheet, focusing on its immediate relationships. Use the node’s labels and properties, and explain how it connects to its neighbors. Focus on what the equipment/component does, using its name or tag (not IDs). Be clear and concise. </instruction> <node> Central Node: Labels: node_labels Properties: node_properties_json This node is a component of a process flowsheet. 26 ChatP&IDA PREPRINT </node> <connections> Incoming Connections: incoming_connections_json Outgoing Connections: outgoing_connections_json </connections> A.3 Semantic Representation Output For each node n, the LLM returns: • global_description n — functional role in full flowsheet • local_description n — interaction with neighbors These are combined into a unified semantic representation: semantic n = (global_description n , local_description n ) This semantic tuple is subsequently encoded into a vector embedding, as described in Algorithm 1. A.4 Example This is the example for Global and Local semantic of Tank T4750 Global semantic: <data key="global_semantic"> The component with the tag "T4750" is a Tank, designed as a vessel for fluid storage within the process flowsheet. Its main role is to receive and collect fluids, serving as an intermediate storage point before the fluid is distributed for further processing. The tank is cylindrical with a length of 4.0 meters. The tank receives fluid from upstream valves and distributes it downstream, functioning as a critical storage and transfer point in the overall material flow. It also integrates with process instrumentation that monitors temperature signals, indicating its importance in process control and safety operations. </data> Global semantic embedding: <data key="global_semantic_embedding">[0.016520526,-0.014927468 -0.020687385,-0.0301709,-0.008441978,...,-0.012176747] Local semantic: <data key="local_semantic"> The central node in this process flowsheet is equipment "T4750", classified as a tank vessel used for fluid storage and distribution. Incoming connections include multiple globe valves supplying fluids (codes MNc and MNb) and a spring-loaded safety valve that relieves overpressure into the tank. Outgoing connections include a butterfly valve distributing stored fluid downstream and a process signal generating function (T4750.03) providing instrumentation and monitoring. The tank contains two internal chambers with specified 27 ChatP&IDA PREPRINT materials and pressure–temperature design limits. This configuration shows the tank acting as a receiving, storage, and controlled distribution node within the flowsheet. </data> Local semantic embedding: <data key="local_semantic_embedding">[-3.1762512E-4,-0.006226967, -0.02870905,-0.048853412,0.0532493,0.03661649,...,-0.01421397] B Dataset of Questions and Reference Answers for Graph Tasks App. Table 1: Dataset of Questions and Reference Answers for Graph Tasks Task TypeQuestionReference Answer Graph Query (Single) What is the cylinder length of tank T4750? 4.0 m Graph Query (Single) What is the design heat trans- fer area of heat exchanger H1007? 46.8 m 2 Graph Query (Single) What is the design pressure of pump P4711? 10.0 m (pressure head) Graph Query (Single) What is the design shaft power of pump P4711? 60.0 kW Graph Query (Single) What is the design volume flow rate of pump P4712? 420.0 m 3 /h Graph Query (Single) What is the nominal diameter of valve 66KL21? DN 80 Graph Query (Single) What is the set pressure of safety valve SV 104.01? 6.0 bar Graph Query (Single) What is the upper design tem- perature of tank T4750? 100.0 ◦ C Continues on the next page . . . 28 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer Graph Query (Multi) List all valves in the P&ID along with their specifica- tion. 1. Globe Valves Valve 1: - Type: Globe Valve (Operated) - Tag/Component: C1 - Nominal Diameter: DN 80 - Piping Class: 75HB13 - Fluid Code: MNb - Segment: S1 - Actuation: HV4750.01 (Fail Close) Valve 2: - Type: Globe Valve (Operated) - Tag/Component: C1 - Nominal Diameter: DN 50 - Piping Class: 75HB13 - Fluid Code: MNc - Segment: S1 - Insulation: 80.0 m thickness, Type Q - Actuation: PV4712.02 (Fail Close) Continues on the next page . . . 29 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer Valve 3: - Type: Globe Valve (Operated) - Tag/Component: C1 - Nominal Diameter: DN 50 - Piping Class: 75HB13 - Fluid Code: QSb - Segment: S1 - Insulation: 80.0 m thickness, Type Q - Actuation: TV4750.03 (Fail Open) 2. Butterfly Valves Valve 4: - Type: Butterfly Valve (Operated) - Component Name: 66KL21 - Tag/Component: C1 - Nominal Diameter: DN 80 - Piping Class: 75HB13 - Fluid Code: MNc - Segment: S1 3. Check Valves Valve 5: - Type: Swing Check Valve - Component Name: 75SA21 - Tag/Component: C2 - Nominal Diameter: DN 80 - Piping Class: 75HB13 - Fluid Code: MNc - Segment: S1 4. Ball Valves Valve 6: - Type: Ball Valve (Operated) - Component Name: 73KH12 - Tag/Component: C4 - Nominal Diameter: DN 50 - Piping Class: 73HG12 - Fluid Code: MNc - Line Number: 47124 - Segment: S3 Valve 7: - Type: Ball Valve (Operated) - Component Name: 73KH12 - Tag/Component: C2 - Nominal Diameter: DN 25 - Piping Class: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment: S2 Continues on the next page . . . 30 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer Valve 8: - Type: Ball Valve (Operated) - Component Name: 73KH12 - Tag/Component: C5 - Nominal Diameter: DN 25 - Piping Class: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment: S5 Valve 9: - Type: Ball Valve (Operated) - Component Name: 73KH12 - Tag/Component: C7 - Nominal Diameter: DN 50 - Piping Class: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment: S6 Valve 10: - Type: Ball Valve (Operated) - Component Name: 73KH12 - Tag/Component: C10 - Nominal Diameter: DN 25 - Piping Class: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment: S9 5. Safety Valve Valve 11: - Position number: SV 104.01 - Type: SpringLoadedGlobeSafetyValve - Piping class: 73HG12 - Set pressure: 6.0 bar - Nominal diameter: DN 50 - Fluid code: MNc - Segment: S1. Graph Query (Multi) List all pipe fitting in the P&ID along with their speci- fication. 1. Pipe Reducers - Type: PipeReducer - Component Number: C3 - Nominal Diameter: DN 50 - Piping Class Code: 73HG12 - Fluid Code: MNc - Line Number: 47124 - Segment Number: S2 Continues on the next page . . . 31 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer 2. Pipe Tees Tee 1: - Type: PipeTee - Component Number: C3 - Nominal Diameter: DN 25 - Piping Class Code: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment Number: S3 Tee 2: - Type: PipeTee - Component Number: C1 - Nominal Diameter: DN 25 - Piping Class Code: 75HB13 - Fluid Code: MNc - Segment Number: S1 Tee 3: - Type: PipeTee - Component Number: C4 - Nominal Diameter: DN 25 - Piping Class Code: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment Number: S4 Tee 4: - Type: PipeTee - Component Number: C8 - Nominal Diameter: DN 50 - Piping Class Code: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment Number: S6 Tee 5: - Type: PipeTee - Component Number: C9 - Nominal Diameter: DN 25 - Piping Class Code: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment Number: S8 3. Blind Flanges Flange 1: - Type: BlindFlange - Component Number: C6 - Nominal Diameter: DN 25 - Piping Class Code: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment Number: S5 Continues on the next page . . . 32 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer Flange 2: - Type: BlindFlange - Component Number: C11 - Nominal Diameter: DN 25 - Piping Class Code: 75HB13 - Fluid Code: MNc - Line Number: 47126 - Segment Number: S9 PathExplo- ration If heat exchanger H1007 needs maintenance, can flow still reach tank T4750? De- scribe the alternative path. The operational strategy depends on the criticality of H1007. If raw material can be supplied at required temperature or no control is needed, H1007 could be bypassed. However, this requires a tie- in point along line MNb47123 which currently does not exist, or a connection via N3 of T4750. H1008 controls tank temperature, so using it as an alternative path is not recommended. PathExplo- ration If you need to isolate tank T4750 from all upstream equipment, which valves would you need to close? To fully isolate Tank T4750: - Shut the main DN80 feed globe valve (HV4750.01). - Close the DN50 globe valve on the recirculation line (PV4712.02). PathExplo- ration Trace the flow path from tank T4750 to pump P4712. Path: T4750 (Tank)→66KL21 (Butterfly valve)→75SA21 (Swing check valve)→Pipe reducer (DN80 to DN50)→ 73KH12 (Ball valve)→ P4712 (Reciprocating pump) PathExplo- ration Working backwards from tank T4750, what are the two possible inlet paths and their source equipment? Path 1: T4750←Globe Valve (HV4750.01)←H1007 (Plate Heat Exchanger)← P4711 (Centrifugal Pump) Path 2: T4750←Globe Valve (TV4712.02)←H1008 (Tubular Heat Exchanger) PathExplo- ration What is the flow path from pump P4711 to tank T4750? P4711 (Centrifugal Pump)→H1007 (Plate Heat Exchanger)→ HV4750.01 (Globe Valve)→ T4750 (Tank) Knowledge In- ference Analyze the flowsheet and give recommendations re- garding process safety Critical Safety Observations: 1.Safety valve set pressure vs vessel design pressure mismatch Current State: Safety valve SV104.01 is set at 6.0 bar and discharges to Tank T4750. However, the chambers of T4750 have upper design pressure limits of only 0.1 bar and 0.05 bar, indicating that the tank is essentially atmospheric or very low pressure. Under these conditions, a PSV set at 6 bar provides no effective protection to the tank. The tank would be subjected to severe overpressure and could fail catastrophically well before the safety valve opens. Recommendation: Each pressure-containing boundary must be protected by appropriately sized and correctly set overpressure protection devices. PSV set pressures shall not exceed the MAWP of the protected equipment. Continues on the next page . . . 33 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer 2. Missing pressure relief coverage for connected pressurized equipment Current State: Pumps P4711 and P4712, as well as heat exchangers H1007 and H1008, are specified with chamber design pressures reaching up to 60 bar. Despite this, pressure relief provisions for these high-pressure components are missing or unclear. Recommendation: Pressure relief analysis shall be updated to explicitly include operating conditions. Pressure relief devices shall be installed on all high-pressure equipment. 3. Process Fluid Current State: Multiple fluid codes are referenced (MNb, MNc, QSb) but no clear information is provided regarding their abbreviation. Recommendation: Provide detailed information on all process fluids and utility.Re-evaluate safety assessment based on confirmed fluids. 4. Isolation Current State: Blind flanges are installed on several branch connections without clear isolation points. Major equipment items do not clearly show complete isolation points on all inlets and outlets. Recommendation: Isolation and line-breaking procedures should be formally reviewed.Positive isolation points should be provided for all major equipment. 5. Fail-Safe Action Analysis Current State: Valve fail actions are defined (PV4712.02 closed, HV4750.01 closed, TV4750.03 open). For TV4750.03, it is unclear if the fluid is heating or cooling, making fail-open potentially unsafe. Critical Recommendation: Comprehensive review of fail-to-safe- state philosophy for all critical valves. Knowledge In- ference List all control valves along with its control philosophy, why and what ifs fail. Control Valves and Their Control Philosophies 1. HV4750.01 (Flow Control Valve) HV4750.01 is a manually operated globe valve used to regulate flow into Tank T4750. It is configured to fail closed. The fail-close action ensures a safe shutdown position by preventing uncontrolled flow into the tank in the event of actuator failure. If the valve fails, it closes and isolates flow to T4750, which may result in a low-level condition in the tank. 2. PV4712.02 (Pressure Control Valve) PV4712.02 is a globe valve used for pressure regulation within the downstream piping system and is controlled via a pressure control loop (PICSA 4712.02). It is also configured to fail closed. This fail-safe position prevents the process stream from returning to Tank T4750 if actuator power is lost. If the valve fails, flow through the return line is stopped, which can increase downstream pressure. Continues on the next page . . . 34 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer 3. TV4750.03 (Temperature Control Valve) TV4750.03 is a globe valve that regulates the flow of heating or cooling medium to heat exchanger H1008. It is configured to fail open. The fail-open design ensures continuous flow of the utility medium in case of actuator failure. If the valve fails, the medium flows at maximum capacity, which could lead to overcooling or overheating depending on the service. Knowledge In- ference What would be the effect if the heating fluid temperature in Heat Exchanger H1007 in- creased from the design tem- perature to 120 ◦ C? Context and Analysis Framework: H1007 is a Plate Heat Exchanger (Design: 313 kW, 46.8 m 2 ). Expected Effects: 1. Heat Transfer Performance: Increased∆T would enhance heat transfer rate, potentially exceeding the design heat flow rate of 313 kW. Process fluid outlet temperature would increase. 2.Equipment Safety: Chamber design temperature limits are 100 ◦ C. Heating to 120 ◦ C would exceed design temperature by 20 ◦ C, risking thermal stress and equipment damage. 3.Process Impact: Downstream equipment (Tank T4750) and processes would be affected. The temperature control loop (TV4750.03) would need to compensate. 4.System Response:Control valve TV4750.03 might need to throttle more to maintain desired process temperature. GraphSum- marization Based on the P&ID, make a narrative to describe the pro- cess flow from the inlet to the final outlet, identifying all major equipment, inter- mediate streams, and control points in sequence. Process Flow Description from Inlet to Outlet 1. Inlet and Feed Pressurization - The process fluid enters the system through the off-page inlet connector. From this inlet, the stream is routed to centrifugal feed pump P4711. This pump raises the fluid pressure to the level required for downstream operations. 2. First Heat Exchange - The discharge from P4711 is directed to plate heat exchanger H1007, where the fluid undergoes the initial heat transfer duty. - Downstream of H1007, flow passes through an actuated globe valve HS4750.01, which regulates the flow rate entering Tank T4750. 3. Tank - The controlled process stream enters Tank T4750. - The tank is fitted with a temperature transmitter and controller, TICSA4750.03, which continuously measures the vessel temperature and provides feedback to the temperature control loop. Continues on the next page . . . 35 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer 4. Tank Outlet Isolation and Conditioning - Fluid exits the bottom of T4750 through a butterfly valve (66KL21). A swing check valve (75SA21) downstream prevents reverse flow. - The line size is then reduced from DN80 to DN50, and an operated ball valve (73KH12) provides additional isolation upstream of the downstream pump. 5. Secondary Pumping - The stream then continues to the reciprocating pump P4712, which delivers the pressurized stream into a downstream of the system. 6. Pump Discharge Protection and Distribution - Immediately downstream of P4712, the discharge enters a tee arrangement. - One branch contains a spring-loaded safety relief valve SV104.01, set at 6.0 bar, to protect the pump discharge piping from overpressure. Any relieved fluid is returned to Tank T4750. - The main discharge continues through additional tees, for pressure measurement and downstream process. 7. Instrumentation and Test Point - A branch from the post-pump tee passes through a ball valve to a blind flange equipped with pressure transmitter PI4712.01, providing local pressure measurement. - Further downstream, another tee divides the flow between the main process line and a controlled heat exchanger loop. 8. Tubular Heat Exchanger Loop and Tank Return - One branch from this tee supplies tubular heat exchanger H1008, which is part of the tank temperature control system. - The tank temperature controller TICSA 4750.03 modulates control valve TV4750.03 to regulate the flow of heating or cooling medium through H1008. - The process outlet from H1008 may be routed back to Tank T4750 via a controlled return line fitted with globe valve PV4712.02. - This recirculation loop enables control of the tank temperature. 9. Main Outlet Line - The main run from the tee bypasses H1008 and continues toward the system outlet. - Near the outlet, a small branch fitted with a ball valve and blind flange houses pressure transmitter and controller PICSA 4712.02. This instrument provides the pressure signal used to regulate the tank return line MNc47127 via control valve PV4712.02. - The main line terminates at the off-page outlet connector. Continues on the next page . . . 36 ChatP&IDA PREPRINT . . . App. Table 1, continued from the previous page. Task TypeQuestionReference Answer Control and Instrumentation Summary Flow Routing Control: The actuated globe valve HS4750.01, located between H1007 and T4750, controls flow into the tank. The valve is operated from the central control system and is designed to fail closed. Pressure Control:PICSA 4712.02 measures pressure in the outlet line and generates the control signal for the tank return line. This controller modulates globe valve PV4712.02 in the H1008 return line, adjusting recirculation back to Tank T4750 to maintain the required line pressure. PV4712.02 is configured to fail closed. Temperature Control:TICSA 4750.03 measures the tem- perature in Tank T4750 and controls the temperature via the tank return loop. The controller actuates valve TV4750.03 to regulate the heating or cooling medium flow through H1008, maintaining the target tank temperature. TV4750.03 is designed to fail open. C Complete Results App. Table 2: Response accuracy and token cost per task across different knowledge graph abstractions and flowsheet representations. Tools: ContextRAG is used to query knowledge graphs; image read for flowsheet images; and file read for Proteus File. Comprehensive results for Anthropic and OpenAI models. The response accuracy is the average of LLM-as-judge score: completeness, coherence, correctness, and relatedness. RepresentationGraph Query (single) Graph Query (multi) Graph Summarization Knowledge InferencePath ExplorationAverage OpenAI: GPT-5 Graph Abstractions - Conceptual Level0.91 / $0.0111.00 / $0.0331.00 / $0.0640.98 / $0.0550.92 / $0.0270.94 / $0.027 - Process Level0.93 / $0.0350.96 / $0.0601.00 / $0.0960.98 / $0.0760.87 / $0.0630.93 / $0.055 - Complete Level0.95 / $0.0860.75 / $0.1261.00 / $0.1680.98 / $0.1260.82 / $0.1320.90 / $0.113 Flowsheet Image0.79 / $0.0020.61 / $0.0220.93 / $0.0270.87 / $0.0200.67 / $0.0080.76 / $0.001 Proteus File0.89 / $0.1520.74 / $0.1951.00 / $0.2160.98 / $0.1910.87 / $0.1840.89 / $0.175 OpenAI: GPT-5-mini Graph Abstractions - Conceptual Level0.89 / $0.0020.95 / $0.0061.00 / $0.0100.98 / $0.0080.88 / $0.0050.91 / $0.004 - Process Level0.99 / $0.0070.98 / $0.0120.93 / $0.0170.96 / $0.0130.79 / $0.0120.93 / $0.010 - Complete Level0.94 / $0.0170.84 / $0.0220.85 / $0.0230.99 / $0.0230.73 / $0.0230.88 / $0.021 Flowsheet Image0.79 / $0.0010.85 / $0.0020.95 / $0.0050.91 / $0.0050.80 / $0.0010.83 / $0.002 Proteus File0.90 / $0.0300.79 / $0.0391.00 / $0.0400.99 / $0.0360.79 / $0.0360.88 / $0.034 OpenAI: GPT-4o Graph Abstractions - Conceptual Level0.99 / $0.0310.81 / $0.0240.60 / $0.0240.64 / $0.0250.74 / $0.0250.83 / $0.027 - Process Level0.94 / $0.0640.81 / $0.0710.60 / $0.0700.68 / $0.0470.59 / $0.0670.77 / $0.063 - Complete Level0.90 / $0.1650.70 / $0.1710.68 / $0.1710.65 / $0.1150.57 / $0.1680.74 / $0.159 Flowsheet Image0.62 / $0.0010.39 / $0.0020.60 / $0.0060.64 / $0.0050.56 / $0.0020.58 / $0.002 Proteus File0.89 / $0.3000.60 / $0.3050.70 / $0.3050.70 / $0.2040.55 / $0.3020.73 / $0.286 OpenAI: GPT-4o-mini Graph Abstractions - Conceptual Level0.98 / $0.0010.78 / $0.0010.50 / $0.0010.63 / $0.0010.61 / $0.0010.78 / $0.001 - Process Level0.91 / $0.0040.66 / $0.0040.53 / $0.0040.61 / $0.0030.55 / $0.0040.72 / $0.004 - Complete Level0.93 / $0.0100.58 / $0.0100.50 / $0.0100.61 / $0.0070.53 / $0.0100.71 / $0.010 Flowsheet Image0.57 / $0.00010.33 / $0.00030.68 / $0.00040.62 / $0.00030.55 / $0.00010.55 / $0.0002 Continued on next page . . . 37 ChatP&IDA PREPRINT . . . App. Table 2, continued from the previous page. RepresentationGraph Query (single) Graph Query (multi) Graph Summarization Knowledge InferencePath ExplorationAverage Proteus File0.85 / $0.0180.48 / $0.0180.58 / $0.0190.74 / $0.0120.60 / $0.0180.71 / $0.017 Anthropic: Claude Opus 4.1 Graph Abstractions - Conceptual Level0.90 / $0.1350.93 / $0.1920.85 / $0.2030.78 / $0.2100.77 / $0.1570.85 / $0.162 - Process Level1.00 / $0.4650.89 / $0.5150.78 / $0.5260.83 / $0.5150.78 / $0.4930.89 / $0.489 - Complete Level0.93 / $1.1570.66 / $1.2070.80 / $1.2140.84 / $1.2020.72 / $1.1750.83 / $1.177 Flowsheet Image0.71 / $0.0180.51 / $0.0610.65 / $0.0830.69 / $0.0610.62 / $0.0380.66 / $0.038 Proteus File0.80 / $1.8670.61 / $1.7680.85 / $2.3510.85 / $2.3480.75 / $2.3200.78 / $2.077 Anthropic: Claude Sonnet 4 Graph Abstractions - Conceptual Level0.96 / $0.0270.91 / $0.0390.68 / $0.0410.81 / $0.0410.79 / $0.0320.87 / $0.033 - Process Level1.00 / $0.0930.79 / $0.1030.73 / $0.1050.82 / $0.1020.78 / $0.0990.88 / $0.098 - Complete Level0.98 / $0.2320.73 / $0.2430.73 / $0.2430.79 / $0.2420.71 / $0.2380.84 / $0.237 Flowsheet Image0.72 / $0.0040.46 / $0.0130.60 / $0.0200.73 / $0.0130.67 / $0.0090.68 / $0.009 Proteus File0.91 / $0.4610.61 / $0.4730.78 / $0.4730.83 / $0.4700.74 / $0.4640.81 / $0.465 Anthropic: Claude 3.7 Sonnet Graph Abstractions - Conceptual Level0.95 / $0.0270.89 / $0.0390.83 / $0.0400.79 / $0.0400.83 / $0.0300.88 / $0.032 - Process Level0.98 / $0.0930.79 / $0.1050.63 / $0.1050.78 / $0.1020.75 / $0.0970.85 / $0.097 - Complete Level0.97 / $0.2310.70 / $0.2410.63 / $0.2430.80 / $0.2410.69 / $0.2360.82 / $0.236 Flowsheet Image0.66 / $0.0050.44 / $0.0110.63 / $0.0150.66 / $0.0120.61 / $0.0070.62 / $0.007 Proteus File0.90 / $0.4600.58 / $0.4730.80 / $0.4730.81 / $0.4690.74 / $0.4640.81 / $0.464 Anthropic: Claude 3.5 Haiku Graph Abstractions - Conceptual Level0.93 / $0.0070.81 / $0.0100.65 / $0.0100.73 / $0.0090.72 / $0.0080.82 / $0.008 - Process Level0.93 / $0.0240.85 / $0.0280.60 / $0.0270.72 / $0.0270.66 / $0.0250.80 / $0.025 - Complete Level0.95 / $0.0610.63 / $0.0630.58 / $0.0640.66 / $0.0640.61 / $0.0620.76 / $0.062 Flowsheet Image0.58 / $0.0010.49 / $0.0020.58 / $0.0030.68 / $0.0030.60 / $0.0020.59 / $0.002 Proteus File0.85 / $0.1230.58 / $0.1250.65 / $0.1260.73 / $0.1240.72 / $0.1230.76 / $0.123 App. Table 3: Tool performance across tasks: response accuracy, cost per task ($), and execution time (s). Comprehen- sive results for Anthropic, OpenAI, and Ollama models. The response accuracy is the average of LLM-as-judge score: completeness, coherence, correctness, and relatedness. ToolGraph Query (single) Graph Query (multi) Graph Sum.Know. Inf.Path Expl.Average Anthropic: Claude 3.5 Haiku ContextRAG0.93 / $0.007 / 6.1s0.81 / $0.010 / 14.3s0.65 / $0.010 / 19.7s0.73 / $0.009 / 16.4s0.72 / $0.008 / 8.1s0.82 / $0.008 / 9.8s VectorRAG0.85 / $0.002 / 5.7s0.66 / $0.003 / 10.8s0.53 / $0.004 / 16.2s0.64 / $0.004 / 16.9s0.58 / $0.003 / 10.7s0.71 / $0.003 / 9.9s PathRAG0.84 / $0.002 / 55.6s0.64 / $0.002 / 139.1s0.58 / $0.004 / 103.3s0.67 / $0.004 / 110.1s0.62 / $0.002 / 108.1s0.72 / $0.002 / 89.3s CypherRAG0.51 / $0.000 / 2.0s0.63 / $0.001 / 4.3s0.43 / $0.001 / 3.5s0.54 / $0.002 / 8.0s0.57 / $0.001 / 3.0s0.54 / $0.001 / 3.5s Multimodal Context0.58 / $0.001 / 7.1s0.49 / $0.002 / 16.9s0.58 / $0.003 / 23.2s0.68 / $0.003 / 21.0s0.60 / $0.002 / 12.2s0.59 / $0.002 / 12.5s Proteus Context0.85 / $0.123 / 88.6s0.58 / $0.125 / 97.9s0.65 / $0.126 / 103.1s0.73 / $0.124 / 97.9s0.72 / $0.123 / 91.3s0.76 / $0.123 / 92.5s Anthropic: Claude 3.7 Sonnet ContextRAG0.95 / $0.027 / 9.2s0.89 / $0.039 / 17.1s0.83 / $0.040 / 24.9s0.79 / $0.040 / 21.3s0.83 / $0.030 / 11.8s0.88 / $0.032 / 13.4s VectorRAG0.85 / $0.007 / 21.2s0.59 / $0.015 / 36.0s0.65 / $0.020 / 58.5s0.62 / $0.019 / 36.2s0.61 / $0.023 / 58.4s0.71 / $0.015 / 36.9s PathRAG0.85 / $0.006 / 59.5s0.56 / $0.009 / 131.2s0.60 / $0.014 / 132.6s0.65 / $0.012 / 112.8s0.63 / $0.009 / 114.2s0.72 / $0.009 / 93.7s CypherRAG0.60 / $0.002 / 4.4s0.65 / $0.006 / 7.7s0.63 / $0.012 / 15.7s0.64 / $0.010 / 13.1s0.59 / $0.003 / 6.0s0.61 / $0.005 / 7.1s Multimodal Context0.66 / $0.005 / 15.0s0.44 / $0.011 / 26.5s0.63 / $0.015 / 34.4s0.66 / $0.012 / 27.5s0.61 / $0.007 / 19.4s0.62 / $0.007 / 20.4s Proteus Context0.90 / $0.460 / 107.9s0.58 / $0.473 / 111.4s0.80 / $0.473 / 110.4s0.81 / $0.469 / 105.4s0.74 / $0.464 / 101.5s0.81 / $0.464 / 106.3s Anthropic: Claude Opus 4.1 ContextRAG0.90 / $0.135 / 8.7s0.93 / $0.192 / 29.0s0.85 / $0.203 / 38.7s0.78 / $0.210 / 29.7s0.77 / $0.157 / 17.6s0.85 / $0.162 / 18.1s VectorRAG0.83 / $0.034 / 17.7s0.49 / $0.099 / 54.4s0.55 / $0.120 / 65.2s0.67 / $0.084 / 42.8s0.60 / $0.090 / 46.4s0.69 / $0.068 / 35.6s PathRAG0.87 / $0.031 / 60.0s0.61 / $0.049 / 128.4s0.53 / $0.072 / 119.6s0.68 / $0.065 / 103.5s0.59 / $0.047 / 108.9s0.72 / $0.045 / 90.1s CypherRAG0.73 / $0.013 / 10.0s0.73 / $0.029 / 17.4s0.49 / $0.036 / 35.7s0.68 / $0.047 / 26.4s0.62 / $0.023 / 16.1s0.68 / $0.024 / 16.3s Multimodal Context0.71 / $0.018 / 14.3s0.51 / $0.061 / 34.1s0.65 / $0.083 / 50.1s0.69 / $0.061 / 41.1s0.62 / $0.038 / 25.9s0.66 / $0.038 / 25.6s Proteus Context0.80 / $1.867 / 15.6s0.61 / $1.768 / 24.3s0.85 / $2.351 / 42.3s0.85 / $2.348 / 41.4s0.75 / $2.320 / 30.3s0.78 / $2.077 / 25.8s Anthropic: Claude Sonnet 4 ContextRAG0.96 / $0.027 / 7.7s0.91 / $0.039 / 16.0s0.68 / $0.041 / 29.1s0.81 / $0.041 / 22.3s0.79 / $0.032 / 12.9s0.87 / $0.033 / 13.4s Continued on next page . . . 38 ChatP&IDA PREPRINT . . . App. Table 3, continued from the previous page. ToolGraph Query (single) Graph Query (multi) Graph Sum.Know. Inf.Path Expl.Average VectorRAG0.87 / $0.007 / 8.5s0.48 / $0.020 / 23.9s0.55 / $0.023 / 32.6s0.64 / $0.018 / 26.1s0.64 / $0.019 / 29.0s0.71 / $0.014 / 19.6s PathRAG0.85 / $0.006 / 53.1s0.50 / $0.010 / 119.0s0.63 / $0.014 / 114.4s0.68 / $0.012 / 101.1s0.63 / $0.010 / 102.9s0.72 / $0.009 / 83.9s CypherRAG0.73 / $0.003 / 9.0s0.77 / $0.006 / 15.8s0.59 / $0.006 / 19.5s0.65 / $0.009 / 24.6s0.66 / $0.005 / 15.0s0.69 / $0.005 / 14.3s Multimodal Context0.72 / $0.004 / 11.0s0.46 / $0.013 / 26.5s0.60 / $0.020 / 55.7s0.73 / $0.013 / 36.7s0.67 / $0.009 / 19.9s0.68 / $0.009 / 21.4s Proteus Context0.91 / $0.461 / 11.2s0.61 / $0.473 / 20.2s0.78 / $0.473 / 31.7s0.83 / $0.470 / 24.8s0.74 / $0.464 / 16.8s0.81 / $0.465 / 16.8s OpenAI: GPT-5-mini ContextRAG0.89 / $0.002 / 7.5s0.95 / $0.006 / 34.4s1.00 / $0.010 / 62.5s0.98 / $0.008 / 48.9s0.88 / $0.005 / 24.9s0.91 / $0.004 / 24.3s VectorRAG0.90 / $0.001 / 14.0s0.79 / $0.002 / 20.5s0.78 / $0.004 / 31.9s0.87 / $0.005 / 36.2s0.68 / $0.002 / 34.0s0.82 / $0.002 / 24.4s PathRAG0.87 / $0.001 / 51.6s0.65 / $0.003 / 40.5s0.88 / $0.003 / 24.3s0.88 / $0.005 / 45.9s0.79 / $0.002 / 76.4s0.83 / $0.002 / 54.6s CypherRAG0.88 / $0.000 / 19.4s0.81 / $0.002 / 52.0s0.93 / $0.003 / 52.3s0.86 / $0.005 / 74.9s0.84 / $0.001 / 42.6s0.86 / $0.002 / 39.4s Multimodal Context0.79 / $0.001 / 18.5s0.85 / $0.002 / 42.7s0.95 / $0.005 / 100.4s0.91 / $0.005 / 103.0s0.80 / $0.001 / 44.6s0.83 / $0.002 / 45.6s Proteus Context0.90 / $0.030 / 15.3s0.79 / $0.039 / 96.7s1.00 / $0.040 / 140.5s0.99 / $0.036 / 70.0s0.79 / $0.036 / 64.5s0.88 / $0.034 / 52.1s OpenAI: GPT-4o ContextRAG0.99 / $0.031 / 2.6s0.81 / $0.024 / 13.1s0.60 / $0.024 / 17.4s0.64 / $0.025 / 11.0s0.74 / $0.025 / 5.9s0.83 / $0.027 / 6.7s VectorRAG0.80 / $0.002 / 2.7s0.43 / $0.004 / 3.9s0.63 / $0.004 / 6.0s0.54 / $0.003 / 4.8s0.52 / $0.004 / 4.4s0.64 / $0.003 / 3.8s PathRAG0.78 / $0.003 / 12.5s0.53 / $0.004 / 17.1s0.68 / $0.003 / 4.8s0.46 / $0.003 / 12.7s0.51 / $0.004 / 16.6s0.63 / $0.003 / 13.7s CypherRAG0.78 / $0.001 / 4.7s0.64 / $0.005 / 12.1s0.48 / $0.003 / 11.4s0.46 / $0.002 / 9.4s0.42 / $0.002 / 8.8s0.60 / $0.002 / 7.6s Multimodal Context0.62 / $0.001 / 7.1s0.39 / $0.002 / 12.4s0.60 / $0.006 / 26.7s0.64 / $0.005 / 21.8s0.56 / $0.002 / 11.7s0.58 / $0.002 / 12.2s Proteus Context0.89 / $0.300 / 11.5s0.60 / $0.305 / 35.2s0.70 / $0.305 / 39.2s0.70 / $0.204 / 24.6s0.55 / $0.302 / 20.2s0.73 / $0.286 / 19.8s OpenAI: GPT-4o-mini ContextRAG0.98 / $0.001 / 2.0s0.78 / $0.001 / 10.2s0.50 / $0.001 / 8.7s0.63 / $0.001 / 10.4s0.61 / $0.001 / 5.6s0.78 / $0.001 / 5.5s VectorRAG0.82 / $0.0001 / 3.2s0.33 / $0.0003 / 8.3s0.58 / $0.0001 / 2.8s0.61 / $0.0004 / 12.4s0.52 / $0.0002 / 7.9s0.64 / $0.0002 / 6.4s PathRAG0.82 / $0.0001 / 12.9s0.66 / $0.0003 / 20.2s0.50 / $0.0001 / 2.1s0.50 / $0.0003 / 21.5s0.46 / $0.0003 / 20.4s0.64 / $0.0002 / 16.4s CypherRAG0.68 / $0.0001 / 5.5s0.48 / $0.0002 / 20.3s0.50 / $0.0002 / 13.6s0.55 / $0.0002 / 15.1s0.37 / $0.0001 / 11.0s0.54 / $0.0001 / 10.4s Multimodal Context0.57 / $0.0001 / 7.0s0.33 / $0.0003 / 17.4s0.68 / $0.0004 / 20.4s0.62 / $0.0003 / 26.6s0.55 / $0.0001 / 10.7s0.55 / $0.0002 / 12.9s Proteus Context0.85 / $0.018 / 8.2s0.48 / $0.018 / 24.8s0.58 / $0.019 / 27.3s0.74 / $0.012 / 16.1s0.60 / $0.018 / 13.5s0.71 / $0.017 / 13.6s OpenAI: GPT-5 ContextRAG0.91 / $0.011 / 9.6s1.00 / $0.033 / 55.0s1.00 / $0.064 / 124.7s0.98 / $0.055 / 85.8s0.92 / $0.027 / 40.7s0.94 / $0.027 / 40.6s VectorRAG0.85 / $0.008 / 21.6s0.73 / $0.019 / 45.6s0.73 / $0.022 / 33.8s0.86 / $0.033 / 61.3s0.68 / $0.017 / 46.1s0.79 / $0.017 / 37.5s PathRAG0.86 / $0.006 / 62.2s0.63 / $0.010 / 142.1s0.83 / $0.026 / 42.8s0.87 / $0.036 / 112.4s0.74 / $0.017 / 131.2s0.80 / $0.015 / 95.7s CypherRAG0.82 / $0.006 / 48.1s0.80 / $0.010 / 69.8s0.95 / $0.021 / 192.8s0.83 / $0.029 / 199.5s0.68 / $0.010 / 107.3s0.79 / $0.012 / 97.5s Multimodal Context0.79 / $0.002 / 38.2s0.61 / $0.022 / 167.9s0.93 / $0.027 / 231.8s0.87 / $0.020 / 161.3s0.67 / $0.008 / 144.5s0.76 / $0.010 / 109.5s Proteus Context0.89 / $0.152 / 21.2s0.74 / $0.195 / 123.4s1.00 / $0.216 / 212.2s0.98 / $0.191 / 98.9s0.87 / $0.184 / 95.0s0.89 / $0.175 / 73.7s Ollama: GPT-OSS 20b ContextRAG0.25 / - / 531.9s0.25 / - / 394.6s0.20 / - / 757.2s0.38 / - / 201.9s0.37 / - / 160.7s0.30 / - / 379.5s VectorRAG0.86 / - / 18.4s0.59 / - / 36.0s0.55 / - / 50.8s0.68 / - / 47.8s0.61 / - / 62.4s0.72 / - / 38.2s PathRAG0.81 / - / 55.2s0.58 / - / 91.0s0.60 / - / 124.3s0.74 / - / 105.0s0.58 / - / 94.1s0.70 / - / 80.7s CypherRAG0.33 / - / 40.2s0.40 / - / 88.8s0.43 / - / 158.2s0.64 / - / 73.1s0.56 / - / 102.2s0.45 / - / 73.0s Ollama: Llama 3.1 8b ContextRAG0.27 / - / 44.2s0.23 / - / 55.6s0.45 / - / 52.6s0.24 / - / 51.0s0.33 / - / 54.9s0.28 / - / 49.7s VectorRAG0.77 / - / 8.5s0.34 / - / 17.6s0.45 / - / 26.8s0.50 / - / 22.0s0.51 / - / 11.7s0.59 / - / 13.4s PathRAG0.79 / - / 65.0s0.50 / - / 148.6s0.45 / - / 133.2s0.52 / - / 124.2s0.50 / - / 115.5s0.62 / - / 100.0s CypherRAG- / - / -- / - / -- / - / -- / - / -- / - / -- / - / - Ollama: Qwen 3 14b ContextRAG0.34 / - / 243.3s0.38 / - / 332.8s0.73 / - / 329.3s0.50 / - / 275.6s0.49 / - / 326.8s0.43 / - / 284.4s VectorRAG0.79 / - / 55.8s0.38 / - / 96.7s0.40 / - / 89.9s0.55 / - / 101.9s0.58 / - / 112.9s0.63 / - / 84.2s PathRAG0.81 / - / 106.8s0.48 / - / 158.3s0.50 / - / 112.7s0.63 / - / 188.7s0.60 / - / 158.4s0.68 / - / 139.1s CypherRAG- / - / -- / - / -- / - / -0.48 / - / 295.1s0.55 / - / 257.9s0.52 / - / 272.7s Ollama: Qwen 3 8b ContextRAG0.27 / - / 182.2s0.38 / - / 163.9s0.55 / - / 196.8s0.45 / - / 147.8s0.38 / - / 172.3s0.35 / - / 173.0s VectorRAG0.83 / - / 35.0s0.50 / - / 45.0s0.60 / - / 90.6s0.59 / - / 71.2s0.56 / - / 58.0s0.67 / - / 50.7s PathRAG0.85 / - / 96.5s0.56 / - / 183.5s0.58 / - / 232.5s0.63 / - / 163.4s0.58 / - / 177.2s0.70 / - / 144.6s CypherRAG- / - / -- / - / -- / - / -0.33 / - / 86.0s- / - / -0.33 / - / 86.0s Ollama: Qwen 3 4b ContextRAG0.31 / - / 216.7s0.41 / - / 534.6s0.53 / - / 319.2s0.47 / - / 242.2s0.36 / - / 276.7s0.37 / - / 275.4s VectorRAG0.83 / - / 30.6s0.40 / - / 54.3s0.50 / - / 115.9s0.54 / - / 63.5s0.49 / - / 72.0s0.63 / - / 53.6s PathRAG0.82 / - / 85.9s0.54 / - / 178.3s0.50 / - / 213.8s0.61 / - / 156.6s0.59 / - / 189.0s0.68 / - / 140.7s CypherRAG- / - / -- / - / -- / - / -- / - / -- / - / -- / - / - 39