Paper deep dive
KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs
Ghanshyam Verma, Simanta Sarkar, Devishree Pillai, Hotaka Shiokawa, Yourong Xu, Fiona Veazey, Peter Hubbert, Hui Su, Paul Buitelaar
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Answering complex conditional questions using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) remains a challenge, particularly in domain-specific contexts where general-purpose LLMs and RAG tend to underperform. We hypothesize that augmenting RAG with unstructured and structured knowledge, extracted from both documents and knowledge graphs (KGs), can improve reasoning and answer accuracy for such tasks. To test this, we propose KGCaRe, a hybrid approach that combines neural retrieval with symbolic reasoning over LLM-generated KGs. KGCaRe constructs a KG from documents using a multi-prompt extraction strategy and stores it in a graph database. Simultaneously, the documents are embedded into a vector store to enable neural retrieval. KGCaRe performs innovative iterative graph traversal guided by the LLM to extract relevant triples, prune irrelevant information, and uses additional clue entities to traverse the graph again if the initial traversal does not provide satisfactory context to generate the answer. The relevant triples extracted from the KG in path form, along with semantically retrieved text passages, are then fed into custom KGCaRe prompts to generate answers to the complex conditional questions with explanations. We evaluate KGCaRe on two complex conditional QA datasets. Our results on these datasets show that KGCaRe consistently outperforms existing baselines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA, across multiple LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o. We publicly release the software pipeline that we developed to implement the proposed KGCaRe approach.
Tags
Links
- Source: https://arxiv.org/abs/2608.09779v1
- Canonical: https://arxiv.org/abs/2608.09779v1
Trouble viewing inline? Open PDF directly â
Full Text
53,695 characters extracted from source content.
Expand or collapse full text
KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs Ghanshyam Verma 1* , Simanta Sarkar 1 , Devishree Pillai 1 , Hotaka Shiokawa 2 , Yourong Xu 2 , Fiona Veazey 3 , Peter Hubbert 3 , Hui Su 2 , Paul Buitelaar 1 1* Insight Research Ireland Centre for Data Analytics, Data Science Institute, University of Galway, IDA Business Park, Galway, H91 AEX4, Co. Galway, Ireland. 2 Fidelity Investments, Boston, Massachusetts, USA. 3 Fidelity Investments, Dublin, Co. Dublin, Ireland. *Corresponding author(s). E-mail(s): ghanshyam.verma@universityofgalway.ie; Contributing authors: simanta.sarkar@universityofgalway.ie; devishree.pillai23@gmail.com;hotaka.shiokawa@fmr.com; yourong.xu@fmr.com;fiona.veazey@fmr.com;peter.hubbert@fmr.com; Hui.Su@fmr.com;paul.buitelaar@universityofgalway.ie; Abstract Answering complex conditional questions using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) remains a challenge, particularly in domain-specific contexts where general-purpose LLMs andRAG tend to underperform. We hypothesize that augmenting RAG with unstructured and structured knowledge, extracted from both documents and knowledge graphs (KGs), can improve reasoning and answer accuracy for such tasks. To test this, we propose KGCaRe, a hybrid approach that combines neural retrieval with symbolic reasoning over LLM-generated KGs.KGCaRe constructs a KG from documents using a multi-prompt extraction strategy and stores it in a graph database. Simultaneously, the documents are embedded into a vector store 1 arXiv:2608.09779v1 [cs.CL] 10 Aug 2026 to enable neural retrieval. KGCaRe performs innovative iterative graph traver- sal guided by the LLM to extract relevant triples, prune irrelevant information, and uses additional clue entities to traverse the graph again if the initial traver- sal does not provide satisfactory context to generate the answer. The relevant triples extracted from the KG in path form, along with semantically retrieved text passages, are then fed into custom KGCaRe prompts to generate answers to the complex conditional questions with explanations. We evaluate KGCaRe on two complex conditional QA datasets. Our results on these datasets show that KGCaRe consistently outperforms existing base- lines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA, across multiple LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o. We publicly release the software pipeline that we developed to implement the proposed KGCaRe approach. Keywords:Natural Language Processing, Knowledge Graph, Large language models, Retrieval-Augmented Generation, Complex Question Answering 1 Introduction Answering domain-specific conditional questions using LLMs presents a significant challenge. Although LLMs perform well on general-purpose QA tasks, their effective- ness diminishes considerably when the input involves conditional logic or requires reasoning across domain-specific content [ 1,2]. One of the reasons behind this per- formance drop is the absence of fine-tuning on specialized corpora. Additionally, when presented with insufficient context, LLMs are prone to hallucinateâgenerating answers that appear plausible but lack grounding in the source material [3]. Condi- tional questions are particularly sensitive to missing or implicit constraints; the LLM must be aware of both the question and the conditions under which an answer is valid. These challenges point to the need for enhanced strategies that go beyond prompting or fine-tuning. A promising direction is to incorporate external knowledge sourcessuch as knowl- edge graphs (KGs), which organize information in semantically structured, explainable formats [4,5]. KGs can complement LLMs by providing factual precision and enabling symbolic reasoning [6]. However, extracting high-quality triples from raw domain- specific documents can be error-prone and incomplete, limiting the usefulness of the KG. On the other hand, semantic vector retrieval from the same documents offers a flexible way to supplement the LLMâs understanding with textual evidence. This motivates our hypothesis: a hybrid context composed of both symbolic and neural representations could improve performance on complex conditional QA tasks. To validate this, we propose a new approach called KGCaRe (KnowledgeGraph ContextawareReasoning for complex Question Answering). It combines symbolic retrieval from a LLM-generated KG with vector-based semantic search from the same document corpus. The KG is constructed using a multi-stage prompt strategy and stored in a Neo4j database [7,8], while the vector index is built using embeddings stored in Facebook AI Similarity Search (FAISS) [9] supported vector storage. During 2 inference, KGCaRe performs iterative graph traversal over theKG to identify relevant triples and combine them with semantically retrieved passages. This fused context is then used to generate answers that are both accurate and explainable. We evaluate KGCaRe using two publicly available QA datasets: ConditionalQA [10] and HotpotQA [11]. ConditionalQA contains complex, multi-hop, conditional questions derived from UK policy documents. HotpotQA contains complex multi-hop questions derived from Wikipedia articles. We also conduct an ablationstudy to eval- uate the performance of the proposed and existing approaches on specific types of questions. The key contributions of this work are: ⢠We design and implement an end-to-end pipeline for automatic KG construction from long documents with complex structure, using a multi-prompt LLM-based triple extraction approach. ⢠We introduce KGCaRe, a hybrid approach that leverages both KG and vector context for answering complex conditional questions. ⢠We empirically show that KGCaRe outperforms baseline methods suchas Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA across multiple LLMs. ⢠We publicly release the end-to-end software pipeline code and the associated prompts used to implement our approach 1 , to support reproducibility and further research. The rest of the paper is structured as follows. In Section2, we describe related work. Section3describes the ConditionalQA and HotpotQA datasets. In Section4, we explain our proposed approach. Section5describes the experimental design that we used. In Section6, we discuss and compare results in detail. Finally, we conclude in Section7. 2 Related Work Despite impressive gains in general-domain question answering, LLMscontinue to struggle with tasks requiring multi-hop, conditional reasoning overlong and complex documents. In this context, the ConditionalQA dataset introduced by Sun et al. [ 10] stands out for its focus on realistic, policy-based QA that includes multiple reasoning steps and conditional constraints. Puerto et al. proposed a code prompting strategy [12] that converts natural lan- guage queries and context from documents into code, which is thenused as input to the LLM prompts for answer generation. While this approach provides the query and context in a more machine-understandable format, it lacks explicit incorporation of structured knowledge, such as knowledge from KGs. Combining KGs with LLMs is a growing area of research. Prior works have shown that knowledge graphs can enhance factual grounding, improve faithfulness, and sup- port reasoning in domains like law and medicine [ 1]. For instance, MindMap [13] 1 https://github.com/GhanshyamVerma/KGCaRe 3 integrates biomedical triples from EMCKG into LLM prompts and shows gains on medical QA benchmarks. Think-on-Graph [14] and Think-on-Graph 2.0 [15] perform iterative beam search with the help of LLMs to extract context from the KG and use that context to gen- erate the answers. Luo et al. proposed a method called Reasoning on Graphs (RoG) that synergises LLMs with KGs so that faithful and interpretable reasoning can be performed [16]. These approaches can uncover multi-hop paths, but they oftendepend on dense graph connectivity. In contrast, our approach works even when the KG is sparse by supplementing it with semantically retrieved passages. Retrieval-Augmented Generation (RAG) frameworks have also been explored as a solution for grounding LLMs in external documents [17]. However, most RAG imple- mentations rely on simple keyword or similarity search, which may fail to retrieve critical conditions or logical dependencies [18]. HybridContextQA [19] is an existing RAG-based hybrid approach that uses context from both documents and KGs to generate answers to complex andconditional ques- tions. While HybridContextQA has shown promising results on the ConditionalQA dataset, it performs simple keyword-based search to extract context from KGs. Tree-of-Traversal [20] is a zero-shot approach that enables augmentation of LLMs with one or more KGs to perform tree search over KGs. This approach significantly improves performance on KG question answering tasks; however,it does not consider context extraction from documents for answer generation. To handle these issues, our KGCaRe approach combines symbolic graph traversal and semantic retrieval into a unified pipeline for better answer generation of complex and conditional questions. 3 Datasets We evaluate our approach on two publicly available benchmark complexQA datasets. The first dataset is ConditionalQA [ 10], which has been specifically designed to test a modelâs capability in multi-hop reasoning and conditional answer generation. The dataset is constructed from UK public policy documents and reflects realistic scenar- ios where understanding legal and procedural nuances is essential for answering the questions correctly. The ConditionalQA dataset includes four answer categories: yes/no answers, span- based answers that involve extracting text from the document, conditional answers which are only valid when certain criteria are met, and not-answerable cases where the document does not contain sufficient information to produce a valid response. One of the key challenges in ConditionalQA is that conditions relevant to the answer are often not explicitly stated in the question but are buried within various sections of the document. As a result, a model must perform multi-hop reasoning to locate and assemble these conditions into a coherent justification for its answer. The second QA dataset is HotpotQA [11], a publicly available large-scale complex QA dataset that requires multi-hop reasoning to answer the questions. This dataset mainly contains two types of multi-hop questions: yes/no type and span type. In this dataset, there is no specific category of conditional-type questions that require explicit 4 conditions to be generated along with the answer. However, the questions require implicit conditions to be checked in order to generate accurate answers, or involve comparison between two or more entities based on shared properties [11]. Please refer to AppendixAfor further details on the datasets. 4 Proposed Approach KGCaRe introduces a comprehensive and modular architecture designed to support explainable reasoning across both unstructured and structuredsources of informa- tion. The core of the KGCaRe approach lies in its hybrid retriever that seamlessly integrates symbolic reasoning over a knowledge graph with semanticsimilarity-based vector retrieval. Below, we detail each component of the proposed KGCaRe approach. 4.1 Multi-Prompt Triple Extraction and Knowledge Graph Construction To enable precise and context-rich knowledge representation, KGCaRe employs a multi-stage, prompt-driven approach for KG construction using LLMs. This process is designed to extract highly accurate and logically connected (subject, predicate, object) triples from raw document text, using a pipeline of progressive prompt stages. These prompt stages are as follows: ⢠Stage 1: Contextual Understanding and Initial Entity-Relation Extraction The first stage uses a prompt (MLT PROMPT1) that instructs the LLM to analyze the input text deeply, identify key entities, and extract allpossible rela- tions. This prompt emphasizes understanding temporal, causal, procedural, and conditional constructs within the document. It encourages the model to extract granular, context-aware triples by first summarizing the purposeand conditions within the text and then listing entity-relationship pairs. For example: (âApplicantâ, âmust submitâ, âformâ) (âformâ, âmust be submitted withinâ, â30 daysâ) ⢠Stage 2: Conditional and Alternative Scenario Expansion Next, a second prompt (MLT PROMPT2) is used to enhance the previously extracted content by focusing on conditional logic, exceptions, and alternatives within the text. This step is crucial for capturing real-world nuances such as regulatory conditions, procedural dependencies, and branchingdecision logic. The prompt instructs the LLM to explicitly recognize if-then structures, alternatives, and exceptions, and to augment the triple set accordingly. For instance: Condition: If consent is not given, permission must be obtained fromthe court. Triples: (âConsentâ, ânot givenâ, âget permission from courtâ) ⢠Stage 3: Refinement and Logical Graph Expansion The third prompt (MLT PROMPT3) refines the extracted triples by validating their completeness, enhancing specificity, and interlinking them forbetter logical consistency. It pushes the LLM to derive new triples from the previously extracted 5 set using transitive or implied logic. This step ensures that the final graph includes both explicitly stated and inferable relationships. For example: Given: (âGuardianâ, âneeds consent fromâ, âParentsâ) (âParentsâ, âcan delegate consent toâ, âCourt of Protectionâ) Inferred: (âGuardianâ, âcan delegate consent toâ, âCourt of Protectionâ) ⢠Stage 4: Output Normalization and Graph Ingestion The final output from these prompts is formatted as a set of triples that pre- serve the semantics of the source text. These triples are then standardized and ingested into a Neo4j graph database, enabling scalable storage and advanced traversal logic during the retrieval phase. This multi-prompt approach not only improves the quality and structure of extracted knowledge but also allows the KG to represent intricate dependencies and nuanced regulatory logicthat single-pass extraction pipelines may miss (see Appendix B). 4.2 Semantic Vector Indexing In parallel with KG construction, documents are embedded into high-dimensional vector representations using an LLM-powered embedder. Theseembeddings are stored in a FAISS index [ 9], allowing fast approximate nearest-neighbor search. This vector index captures the global semantic context of the documents andis used to retrieve relevant passages based on the similarity to the userâs query. 4.3 Knowledge Graph Traversal The central component of KGCaRe is its symbolic, LLM-guided KG traversal mech- anism, formally outlined in Algorithm 1. This traversal process enables KGCaRe to iteratively explore and reason over the constructed KG to extract relevant contextual information for complex question answering. Given an input questionq, the pipeline begins with a preprocessing step where topic entities (E topic ) are extracted using an LLM. These topic entities act as anchors for initiating graph traversal. KGCaRe then initializes multiple memory modules, including the triple memoryM, clue entity memoryE clue , clue triple memoryM clue , traversal path memoryM path traversed , andvisited entities. Traversal proceeds in a depth-bounded iterative manner up to a maximum depth D. At each level, KGCaRe searches for triples within the KG. For the initial depth (d= 1), the retrieval is based on partial matching to allow broader exploration, including subword matches. For subsequent depths (d >1), exact matching is used to ensure more focused reasoning. In parallel, KGCaRe also searches for triples linked to clue entities (E clue ) using partial matching, enabling the discovery of supporting evidence not captured by topic entities alone. The use of the clue entity in our KGCaRe approach is inspired by the work of [20], who proposed the Tree-of-Traversal approach. 6 Algorithm 1Knowledge Graph Traversal in KGCaRe Input:q(input question),D(maximum depth),LLM(language model) Output:triples pruned , Memory (M) 1:InitializeE topic fromq, memoryM, clue entitiesE clue , clue memoryM clue , path memory M pathtraversed , andvisitedentities 2:fordepth= 1 toDdo 3:for alle topic i âE topic do 4:ifdepth== 1then 5:Find partial matches fore topic i withvisited entities 6:else 7:Find exact matches fore topic i withvisited entities 8:end if 9:end for 10:for alle clue i âE clue do 11:Find partial matches fore clue i withvisited entities 12:end for 13:triples pruned âPruneTriples(LLM,q, extracted triples) 14:Save candidates for next round usingtriples pruned ,i,M,M pathtraversed 15:Updatevisited entities,M, andM pathtraversed 16:iftriples pruned 6=â then 17:Perform Reasoning(LLM,q,triples pruned ,M clue ) 18:ifknowledge is sufficientthen 19:returntriples pruned , Memory (M) 20:else 21:UpdateE clue andM clue with new clues 22:ifE clue 6=â orE topic 6=â then 23:continue 24:else 25:break 26:end if 27:end if 28:else 29:Perform Reasoning(LLM,q) 30:UpdateE clue andM clue with clues found 31:ifE clue 6=â orE topic 6=â then 32:continue 33:else 34:break 35:end if 36:end if 37:end for 38:returntriples pruned , Memory (M) The retrieved triples are then passed through an LLM-powered pruning function: triples pruned âPruneTriples(LLM, q,triples) 7 This step filters out irrelevant or redundant triples, retaining only those deemed con- textually useful for answering the input question. The pruned triples are saved to the memoryMand are also used to update the visited entity set and traversal path memory. Next, an LLM reasoning prompt is used to evaluate whether the accumulated context is sufficient to answerq. If so, KGCaRe generates the final answer and halts traversal. If not, the LLM is prompted to extract new clue entitiesthat may assist in further traversal. These new entities are appended toE clue , and the process continues to the next depth level. If no clue or topic entities remain, or no relevant triples are found, KGCaRe triggers a fallback mode where the LLM performs reasoning directly over the question to extract additional clues for KG traversal. If traversal reaches the maximum depthDor sufficient context is retrieved from the KG to generate the answer, it returns the KG context. KGCaRe invokes a final mechanism, where KGCaRe uses an LLM with a custom prompt to generate the answer using all accumulated information from memoryM(context from KG) and the context retrieved from the vector store. Throughout the process, KGCaRe maintains an explicit reasoning trace, including the triples retrieved, pruned, and traversed, as well as the order in which entities were visited. This traceability allows KGCaRe to generate not only an answer but also an explanation of the logical steps and conditions leading to that answer, ensuring interpretability and transparency. By dynamically combining symbolic graph traversal with LLM-guided clue gen- eration and pruning, KGCaRe enables robust, multi-hop reasoning over sparse or incomplete KGs in complex conditional question answering scenarios. 4.4 Combined Retrieval and Answer Generation Once relevant information has been gathered from both the KG traversal and the semantic vector store, KGCaRe merges these two context sources into a unified prompt for final answer generation. This fusion stage ensures that the LLM benefits from the structured precision of symbolic reasoning and the semantic richness of neural retrieval. The graph-based retriever contributes a curated set of triples and their associated traversal paths, representing the logical structure of the reasoning process. Simulta- neously, the semantic retriever returns top-ranked passages from the FAISS vector index, capturing global textual context relevant to the question. These passages may include supporting explanations, alternate phrasings, or implicit conditions that are not explicitly modeled in the KG. Both the symbolic and neural contexts are formatted into a custom structured prompt that instructs the LLM to synthesize a coherent and accurate answer. The prompt explicitly asks the model to incorporate evidence from bothsources and, where applicable, to identify and state any conditions under which the answer holds. This is particularly important for conditional questions, where part of the answer often depends on latent assumptions or constraints embedded in the retrieved context. By combining symbolic and neural retrieval, KGCaRe improves not onlyanswer accuracy but also explanation quality. This integrated reasoning strategy is essential for complex multi-hop questions that require connecting disparatefacts and resolving 8 conditional logic, especially in domains such as public policy, law, or regulations where answer faithfulness and traceability are critical. 5 Experimental Design To evaluate the effectiveness of KGCaRe, we design a series of experiments to com- pare against several baselines and existing state-of-the-art methods. The objective of these experiments is to examine how well KGCaRe performs in answering complex, conditional questions compared to existing KG-based, prompt-based, and RAG-based approaches. We conduct comparative evaluations against two baseline promptingtechniques: Text PromptandCode Prompt, as introduced by Puerto et al. [ 12]. These approaches operate solely on raw document context without incorporating anystructured knowl- edge. In parallel, we benchmark our model againstThink-on-Graph[14], which utilizes a KG to guide the LLM through a structured reasoning process. This comparison is crucial to assess the improvements brought by our hybrid design over purely symbolic KG traversal methods. We also perform comparative evaluations against Vanilla LLM, where the LLM is prompted directly using the raw question without any external context, and Vanilla RAG, where a standard RAG pipeline retrieves passages from the documents and feeds them into the LLM without incorporating any KG-based reasoning. These experiments are conducted using four LLMs: Mistral [ 21], Mixtral [22], GPT-3.5, and GPT-4o. Furthermore, we evaluate our method against an existing hybrid approach called HybridContextQA [19]. We performed the evaluation using two datasets: ConditionalQA andHotpotQA. For both datasets, the development set is used as the evaluation benchmark for all comparative experiments, as the test set is not publicly available. ConditionalQA consists of 2,338 training QA pairs and 271 development QA pairs, each associated with UK policy documents. HotpotQA consists of 90,564 QA pairs in thetraining set and 500 QA pairs in the development set. Please refer to AppendixAfor further details on the datasets. We use some QA examples from the training set for few-shot prompting to provide the LLMs with in-context examples. For all experiments, we test KGCaRe and baseline models using four different LLMs: GPT-3.5, GPT-4o, Mistral, and Mixtral. Please refer to AppendixD.2for implementation details and AppendixD.3for experimental environment details. This diverse set of LLMs allows us to evaluate the generalizability of our approach across both proprietary and open-source LLMs, and to observe how model scale and archi- tecture influence the effectiveness of hybrid retrieval strategies. Similar to [19], we use the exact match based F1 score as an evaluation metric. Please refer to AppendixD.1 for further details on the evaluation metric. Through these experiments, we aim to provide a comprehensive analysis of KGCaReâs strengths and limitations in the context of complex and conditional QA task, and to demonstrate its consistent advantages over both RAG and KG-only baselines. 9 6 Results We evaluate KGCaRe using two datasets across four different LLMs: Mistral, Mixtral, GPT-3.5, and GPT-4o. The ConditionalQA dataset contains 271 QA pairs spanning Yes/No, span-based, and conditional answer types (see Table1). The performance of the proposed KGCaRe is compared against several baselines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA. Table1shows that on both the datasets across all models, the proposedKGCaRe approach consistently achieves the highest performance considering all the questions (Avg F1 Score), validating the effectiveness of the hybrid retrieval and reasoning strategy. On ConditionalQA, for the Mistral model, baseline methods perform poorly, with Vanilla LLM and Code Prompt achieving an average F1 of 31.84 and 28.26, respec- tively, and Text Prompt scoring 27.36. The KG-only approach (Think-on-Graph) performs worse at 16.24, reflecting its limitations in better contextretrieval from KG and answer generation for the complex conditional questions in comparison to other approaches. Vanilla RAG and HybridContextQA improve performance to 40.12 and 45.17, respectively, while KGCaRe significantly boosts the average F1 to 57.89. Notably, it achieves strong gains in the conditional category (F1 = 57.17) and Yes/No category (F1 = 73.42), underscoring its ability to handle multi-hop and condition- sensitive reasoning even with small models. On HotpotQA, for the Mistral model, KGCaRe outperforms all other existing approaches for all QA categories. Mixtral, a more capable open-source model, shows higher baseline performance. On ConditionalQA, Vanilla LLM, Code Prompt and Text Prompt achieve 41.35, 40.88 Table 1Results of different prompting and context integration approaches for various LLMs on the ConditionalQA and HotpotQA datasets. LLM/ Model used Approach Dataset 1: ConditionalQADataset 2: HotpotQA Avg F1 Score [All ] F1 Score [Yes/No ] F1 Score [Span ] F1 Score [Conditional ] Avg F1 Score [All ] F1 Score [Yes/No ] F1 Score [Span ] Mistral Vanilla LLM31.8452.838.425.1124.3168.0022.01 Code Prompt28.2641.7416.3022.7609.6900.0009.69 Text Prompt27.3631.5825.6423.5116.9404.0017.62 Think-on-Graph16.2421.5812.055.5304.7604.0004.80 Vanilla RAG40.1252.8825.8621.1149.4156.0049.06 HybridContextQA45.1760.0033.5534.3951.1632.0052.17 KGCaRe (ours)57.8973.4242.0857.1753.5372.0054.00 Mixtral Vanilla LLM41.3562.2917.9639.8327.4556.0025.95 Code Prompt40.8844.8040.9934.6211.0728.0010.18 Text Prompt46.6056.9540.1539.3622.3520.0022.47 Think-on-Graph17.4023.1412.9005.5404.8700.0005.13 Vanilla RAG53.5473.5430.9149.4653.8448.0054.15 HybridContextQA53.7174.1336.4542.5953.6848.0053.98 KGCaRe (ours)59.4578.3238.3747.9358.2756.0058.39 GPT 3.5 Vanilla LLM33.0261.8824.2735.1038.8676.0036.91 Code Prompt48.2770.2023.7641.3654.1184.0052.54 Text Prompt57.1573.1345.5446.2560.2388.0058.77 Think-on-Graph16.2913.9720.6709.4814.3216.0014.24 Vanilla RAG57.7971.9142.0154.2761.8496.0060.05 HybridContextQA55.0371.2142.9743.5261.7992.0060.20 KGCaRe (ours)60.0172.0246.5954.8371.4888.0070.61 GPT 4o Vanilla LLM34.7258.5008.1514.7850.5396.0048.15 Code Prompt53.2976.9226.8936.3668.8292.0067.60 Text Prompt59.1681.8240.3248.2065.0784.0064.07 Think-on-Graph20.4021.4521.4607.8510.7900.0011.36 Vanilla RAG62.2679.4843.0145.3265.0592.0063.64 HybridContextQA63.7181.5850.7151.9973.3796.0072.18 KGCaRe (ours)67.5583.1050.0548.9080.2196.0079.38 10 and 46.60 average F1 scores, respectively, with Think-on-Graph again underperforming at 17.40. Vanilla RAG and HybridContextQA perform well with an average F1 of 53.54 and 53.71, respectively. KGCaRe outperforms all baselines with an average F1 of 59.45, showing robust performance. On HotpotQA, for the Mixtral model, KGCaRe again outperforms all other existing approaches for all QA categories. These results highlight the value of hybrid retrieval in moderately strong LLMs, especially when symbolic reasoning is paired with semantic context. GPT-3.5 delivers strong baseline results, on ConditionalQA, with VanillaLLM (Avg F1 = 33.02), Code Prompt (Avg F1 = 48.27) and Text Prompt (Avg F1 = 57.15) outperforming the Think-on-Graph method (Avg F1 = 16.29)by a wide margin. Interestingly, Text Prompt yields the highest Yes/No F1 (73.13) across all GPT-3.5 configurations. However, KGCaRe achieves the best overall performance (Avg F1 = 60.01) and the highest conditional F1 (54.83), as well as the best span-based F1 (46.59). On HotpotQA, for the GPT-3.5 model, KGCaRe again outperforms allother existing approaches for all QA categories, except Yes/No. This shows that while prompting alone performs well, complex reasoning benefits from the hybrid design. As the most powerful model in the comparison, GPT-4o achieves high scores across all methods. On ConditionalQA, Vanilla LLM, Code Prompt and Text Prompt reach average F1 scores of 34.72, 53.29 and 59.16. Think-on-Graph remains ineffective even in this setting (Avg F1 = 20.40). HybridContextQA reaches an average F1 of 63.71, and KGCaRe improves further to 67.55, achieving the highest Yes/No F1 (83.10) and competitive results in the span (50.05) and conditional (48.90) categories. On HotpotQA, for the GPT-4o model, KGCaRe again outperforms all other existing approaches with the highest F1 score in all categories, except theYes/No category, where HybridContextQA and KGCaRe achieve similar F1 scores. These overall gains confirm the utility of symbolic augmentation even in high-capacity LLMs. The results in Table1clearly show that Think-on-Graph lags significantly across all models and question types. On HotpotQA, Think-on-Graph remains ineffective and produces an F1 score of 0 with Mixtral and GPT-4o for the Yes/No category. We found that this occurs because both models generate verboseanswers without explicitly including the word âyesâ or ânoâ. Vanilla RAG, Code and Text Prompt per- form reasonably well on Yes/No and span questions, especially with stronger LLMs. KGCaRe consistently outperforms all baselines across models and question types at Avg F1 Score, and outperforms most models for Yes/No, Span, and Conditional types, demonstrating the effectiveness of integrating symbolic KG traversal with vector-based semantic retrieval. These results support our core hypothesis: combining structured and unstructured context, and reasoning over both, yields superior performance on complex, conditional QA tasks, regardless of the underlying LLMâs capacity. The largest performance margins are observed in conditional and Yes/No questions, where reasoning steps are often multi-hop, conditional, and dispersed across documents, highlighting situations where the hybrid model excels. 6.1 Ablation Study During error analysis on the ConditionalQA dataset, we observed that out of the 136 Yes/No type questions, 30 had multiple valid answers in the ground truth. To 11 Table 2Comparative analysis of KGCaRe and existing approaches on 106 Yes/No QA pairs from the ConditionalQA dataset. LLM / Model usedApproach F1 Score [Yes/No type QA ] Mistral Vanilla LLM58.49 Code Prompt40.58 Text Prompt30.89 Think-on-Graph39.94 Vanilla RAG62.26 HybridContextQA57.13 KGCaRe (ours)69.81 Mixtral Vanilla LLM64.22 Code Prompt43.86 Text Prompt57.34 Think-on-Graph45.65 Vanilla RAG74.84 HybridContextQA76.27 KGCaRe (ours)82.07 GPT 3.5 Vanilla LLM66.03 Code Prompt72.07 Text Prompt70.95 Think-on-Graph44.34 Vanilla RAG70.75 HybridContextQA68.86 KGCaRe (ours)71.75 GPT 4o Vanilla LLM71.69 Code Prompt84.90 Text Prompt85.85 Think-on-Graph43.03 Vanilla RAG83.96 HybridContextQA80.97 KGCaRe (ours)88.67 better evaluate the performance of KGCaRe and existing approaches, we focused on the subset of Yes/No questions with a single definitive answer. Table2presents a comparative analysis of KGCaRe and baseline methods on these 106 Yes/No QA pairs having a single definitive answer. For the Mistral model, the Vanilla LLM, Code Prompt, and Think-on-Graph base- lines achieve modest F1 scores of 58.49, 40.58 and 39.94, respectively (see Table2), while Text Prompt performs slightly worse at 30.89. The HybridContextQA and Vanilla RAG improve performance significantly to 57.13 and 62.26, respectively, and KGCaRe further enhances it to 69.81, demonstrating the benefit of combining symbolic reasoning with neural retrieval even in smaller open-source LLMs. With Mixtral, all approaches perform better than on Mistral, showing Mixtralâs stronger capability on the QA task. The baseline Vanilla LLM and Text Prompt achieve 64.22 and 57.34, respectively, while Think-on-Graph performs slightly lower 12 at 45.65. Vanilla RAG and HybridContextQA achieve an F1 score of 74.84 and 76.27, respectively, and KGCaRe again improves on this, achieving the highest score of 82.07. This shows that the hybrid reasoning strategy is particularly effective when leveraged with more powerful open-source models. For GPT-3.5, the Code Prompt surprisingly performs very well with an F1 score of 72.07, surpassing Vanilla LLM (66.03), Text Prompt (70.95), Think-on-Graph (44.34), Vanilla RAG (70.75), and even HybridContextQA (68.86). However, KGCaRe still manages to slightly improve over HybridContextQA, reaching an F1 score of 71.75. Although the gain is smaller compared to open-source models, this suggests that GPT- 3.5 also benefits from hybrid context to a limited but still positive extent. In the case of GPT-4o, both Code Prompt and Text Prompt alreadyperform extremely well with F1 scores of 84.90 and 85.85, respectively. Despite this strong baseline, KGCaRe still achieves the highest F1 score of 88.67, marginally outperform- ing all other approaches. This indicates that even for high-capacity LLMs, KGCaRe can offer marginal yet meaningful improvements, particularly by enhancing reasoning consistency and contextual faithfulness. Across all LLMs, we observe that KGCaRe consistently outperforms the KG-only (Think-on-Graph) and vector-only (Text Prompt, Code Prompt,and Vanilla RAG) baselines, reaffirming the value of combining symbolic and neural retrieval methods. The largest relative gains are observed in Mistral and Mixtral, wherethe base modelâs reasoning capabilities benefit most from structured traversal and context integration. In summary, Table2illustrates that KGCaRe achieves the best or competitive per- formance across most model settings on Yes/No questions with a single definitive answer. 6.2 Explanation and Visualization Our proposed approach can also provide an explanation for the generated answer by showing the paths in the KG that lead to that answer (see Figure 1). As shown in Figure1, we have a complex question as an example from the ConditionalQA dataset. We have a complex and conditional question from an applicant whose husband died at work. The applicant has mentioned some details and wants to knowabout the eligibility for the Bereavement Support Payment. Now, to extract the relevant context from the KG, our KGCaRe approach will go through the KG for a specified number of depths until it extracts the satisfactory context to answer the question or satisfy the termination condition, which is the number of depths. Then it uses the knowledge or paths traversedfrom all the depths to generate the final answer. In Depth 1, it looks into the KG to get keyword matches of the entities from the question. There are different ways to do this matching, but here it uses the direct keyword match. We can see here matches for thetwo keywords from the question: âmoneyâ and âdiedâ. We can also see the associated triples extracted from the KG. Here, initially, it extracts all the information, and thenit performs pruning. In pruning, it keeps only those triples that are most relevant for answering the question. In this case, it found the three most relevant triplesas shown in Figure 1. Then it performs reasoning using an LLM. It asks the LLM if the information found so far is enough to answer the question. In the case of Depth 1, the LLM finds 13 Fig. 1Explaining the Knowledge Graph traversal steps of the proposed algorithm using an example. that the information is not sufficient. Therefore, the approach asks the LLM to suggest other entities to search for in the KG and which might help to answer the question. Here we can see the LLM suggested entities as shown in Figure 1. It also saves these 3 triples to a variable named âPath Traversedâ so that it has a record of them, and it can use them in answer generation. Now it goes into the next depth, that is Depth 2, and looks for expanding with keywords from what we saw in the previous depth, like âclaimantâ, âapplicantâ, etc. It finds some new information related to these, which it will then prune to keep only those relevant to the question (three more triples). Now the approach again asks the LLM if, at this depth, the information is sufficient to answer the question. The LLM still decides this is not enough to answer the question and returns some more clue entities which might be useful. Now, before it goes to the next depth, it again saves the triples found so far in Depth 2 to âPath Traversedâ. It found one new triple related to âapplicantâ that it adds to the existing path. It also found two more triples related to the claimant, one states that they must have lived in the UK, and another states that the claimant must be eligible for BSP. Now it goes to the next depth that is Depth 3, and this time it didnât findany new unique triple, and since this is the last depth considered by our approach, it generates the answer to the question using the information of paths it has so far. The final answer generated is Yes, and the condition is that the applicant must be under pension age and must be living in the UK to be eligible for Bereavement support. It also provides an explanation showing which triples were used to reachthis answer, as shown in Figure1. Overall, our approach enhances trust by providing human-understandable expla- nations for the generated answer. 14 7 Conclusion In this work, we proposed KGCaRe, a hybrid retrieval-augmented question answer- ing approach that tightly integrates symbolic reasoning over a knowledge graph with neural retrieval from a vector store. Our approach is specificallydesigned to address the challenges of answering complex conditional domain-specific questionsâan area where general-purpose LLMs often fall short due to missing context or inability to interpret conditions. By constructing a KG using a multi-prompt LLM-based pipeline and pairing it with FAISS-based neural retrieval, KGCaRe leverages complementarystrengths of both structured and unstructured context. During inference, KGCaRe performs iterative graph traversal guided by an LLM, prunes irrelevant paths, and dynamically updates its memory to support multi-hop reasoning. The final answer is generated using a prompt that combines the curated triples and semantically retrieved text, allowing for both factual accuracy and traceable explainability. We evaluated KGCaRe on two complex conditional QA datasets, demonstrating consistent improvements across all answer types and LLMs compared to strong base- lines such as Vanilla LLM, Text Prompt, Code Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA. In particular, our KGCaRe approach shows significant gains for conditional answers, validating our hypothesis that hybrid retrieval can better handle multi-faceted reasoning in complex domains. Looking forward, our framework opens new directions for explainable QA systems in high-stakes settings like healthcare, law, and public policy, where both precision and transparency are critical. Declarations Funding.This publication has emanated from research conducted with the financial support of Research Ireland under Grant Number 12/RC/2289 P2 - Insight Research Ireland Centre for Data Analytics and a grant from Fidelity Investments. For the purpose of Open Access, the author has applied a C BY public copyright licence to any Author Accepted Manuscript version arising from this submission. Data Availability.The datasets used in this study are publicly available. Code availability.Code is available at the following GitHub repository: https://github.com/GhanshyamVerma/KGCaRe References [1] Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., Wu, X.: Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering (2024) [2] Kandpal, N., Deng, H., Roberts, A., Wallace, E., Raffel, C.: Large language mod- els struggle to learn long-tail knowledge. In: Proceedings of the 40th International 15 Conference on Machine Learning. ICMLâ23, vol. 641, p. 15696â15707. JMLR.org, Honolulu, Hawaii, USA (2023) [3] Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al.: Sirenâs song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023) [4] Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proceedings of the IEEE104(1), 11â33 (2015) [5] Wang, Q., Mao, Z., Wang, B., Guo, L.: Knowledge graph embedding: Asur- vey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering29(12), 2724â2743 (2017) [6] Cheng, K., Ahmed, N.K., Rossi, R.A., Willke, T., Sun, Y.: Neural-symbolicmeth- ods for knowledge graph reasoning: A survey. ACM Trans. Knowl. Discov. Data 18(9) (2024) https://doi.org/10.1145/3686806 [7] Vukotic, A., Watt, N., Abedrabbo, T., Fox, D., Partner, J.: Neo4jin Action. Manning Publications Co., Shelter Island, NY (2014) [8] Monteiro, J., S Ěa, F., Bernardino, J.: Experimental Evaluation ofGraph Databases: JanusGraph, Nebula Graph, Neo4j, and TigerGraph.Applied Sciences 13(9), 5770 (2023) [9] Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazar Ěe, P.-E., Lomeli, M., Hosseini, L., J Ěegou, H.: The FAISS Library. IEEE Transactions on Big Data, 1â17 (2025) https://doi.org/10.1109/TBDATA.2025.3618474 [10] Sun, H., Cohen, W., Salakhutdinov, R.: ConditionalQA: A complex read- ing comprehension dataset with conditional answers. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3627â3637. Association for Computational Linguis- tics, Dublin, Ireland (2022).https://doi.org/10.18653/v1/2022.acl-long.253. https://aclanthology.org/2022.acl-long.253 [11] Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov,R., Man- ning, C.D.: HotpotQA: A dataset for diverse, explainable multi-hop question answering. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018) [12] Puerto, H., Tutek, M., Aditya, S., Zhu, X., Gurevych, I.: Code prompt- ing elicits conditional reasoning abilities in Text+Code LLMs. In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, p. 11234â11258. Association for Computational Linguistics,Miami, 16 Florida, USA (2024).https://doi.org/10.18653/v1/2024.emnlp-main.629. https://aclanthology.org/2024.emnlp-main.629/ [13] Wen, Y., Wang, Z., Sun, J.: MindMap: Knowledge graph prompt- ing sparks graph of thoughts in large language models. In: Ku, L.- W., Martins, A., Srikumar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 10370â10388. Association for ComputationalLinguis- tics, Bangkok, Thailand (2024). https://doi.org/10.18653/v1/2024.acl-long.558. https://aclanthology.org/2024.acl-long.558/ [14] Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L.M., Shum, H.- Y., Guo, J.: Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In: Proceedings of the 12th International Conference on Learning Representations (ICLR 2024) (2024) [15] Ma, S., Xu, C., Jiang, X., Li, M., Qu, H., Yang, C., Mao, J., Guo, J.: Think- on-graph 2.0: Deep and faithful large language model reasoning with knowledge- guided retrieval augmented generation. In: Proceedings of the 13th International Conference on Learning Representations (ICLR 2025) (2025) [16] Luo, L., Li, Y.-F., Haffari, G., Pan, S.: Reasoning on graphs: Faithful and inter- pretable large language model reasoning. In: Proceedings of the 12th International Conference on Learning Representations (ICLR 2024) (2024) [17] Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H.: Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023) [18] Sawarkar, K., Mangal, A., Solanki, S.R.: Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers. In: 2024 IEEE 7th International Conference on Mul- timedia Information Processing and Retrieval (MIPR), p. 155â161 (2024). https://doi.org/10.1109/MIPR62202.2024.00031 [19] Verma, G., Sarkar, S., Pillai, D., Shiokawa, H., Shahbazi, H., Veazey, F., Hubbert, P., Su, H., Buitelaar, P.: HybridContextQA: A Hybrid Approach for Complex Question Answering Using Knowledge Graph Construction and Context Retrieval With LLMs. In: Proceedings of the 2nd ISWC Workshop on KnowledgeBase Construction From Pre-Trained Language Models (ISWC KBC-LM),Baltimore, USA, p. 1â14 (2024) [20] Markowitz, E., Ramakrishna, A., Dhamala, J., Mehrabi, N., Peris, C., Gupta, R., Chang, K.-W., Galstyan, A.: Tree-of-traversals: A zero-shot reasoning algorithm for augmenting black-box language models with knowledge graphs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 12302â12319 (2024) 17 [21] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.- A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7B. arXiv preprint arXiv:2310.06825 (2023) [22] Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D., Hanna, E.B., Bressand, F., Lengyel, G., Bour, G., Lam- ple, G., Lavaud, L.R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T.L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mixtral of Experts. arXiv preprint arXiv:2401.04088 (2024) Appendix A Data Appendix This section provides further details about the dataset used for experiments. A.1 ConditionalQA In the ConditionalQA dataset [10], each question-answer pair consists of three com- ponents. The first is the document itself, which is a structured policy text organized into hierarchical sections and often contains cross-referencesto other sections. These documents were scraped from official UK government websites andprocessed into structured text by serializing the Document Object Model (DOM) trees into a flat- tened sequence of HTML elements [ 10]. The second component is the question, which is typically centered around eligibility, procedural compliance, or exceptions and may require reasoning across multiple parts of the document. These questions vary in type and include yes/no responses, extractive span-based answers,and cases that may not be answerable given the context. The third component is the user/applicant scenario, which adds contextual background representing a specific userâssituation or constraint. This narrative helps simulate real-world query complexity and often includes implicit conditions that must be resolved during the process of generatinganswers. For the ConditionalQA dataset [10], we use the development set as the evaluation benchmark for all comparative experiments, as the test set is notpublicly avail- able. ConditionalQA consists of 2,338 training QA pairs and 285 development QA pairs. Out of 285 questions from the ConditionalQA development set, we removed the fourteen unanswerable questions. After removing the unanswerable questions, in the ConditionalQA development set, we were left with 271 questions, as shown in Table A1. The ConditionalQA dataset has three types of questions: Yes/No type, Span (extractive) type, and conditional type. The ConditionalQA development set has 143 Yes/No QA pairs, 102 Span QA pairs, and 63 Conditional QA pairs. A.2 HotpotQA The HotpotQA dataset [ 11] is a large-scale complex QA dataset that requires reason- ing to be performed over multiple documents to generate the answer to the multi-hop questions. This dataset is created using crowdsourcing based on Wikipedia articles, 18 showing multiple context documents to crowd workers such that they can create questions that require multi-hop reasoning. HotpotQA mainly has two types of questions: Yes/No and Span (extractive) type. The span type questions include the questions that require bridge entity identification and comparison between two entities to answer the multi-hop questions [11]. The HotpotQA train set has 90,564 QA pairs, having a combination of easy, medium, and hard questions. We evaluate the existing and proposedapproaches on a subset of the HotpotQA development set. The original HotpotQAdevelopment set had 7,405 QA pairs. All the questions in the development set are hard/complex multi- hop questions. We selected a set of 500 QA pairs (see TableA1) from the original development set using random stratified sampling. The HotpotQA development set has 25 Yes/No QA pairs and 475 Span QA pairs. Table A1Number of QA pairs in the training and development sets of the ConditionalQA and HotpotQA datasets. DatasetTrain SetDev Set ConditionalQA2,338271 HotpotQA90,564500 Datasets and the code to preprocess them are available at the GitHub repository: https://github.com/GhanshyamVerma/KGCaRe Appendix B Knowledge Graph Appendix We constructed two KGs using our proposed KGCaRe approach: one from Condition- alQA documents and another from HotpotQA documents. The KG constructed using ConditionalQA has 7,031 triples and 7,366 entities, while the KG constructed using HotpotQA has 17,776 triples and 13,286 entities, as shown in TableB2. Table B2Number of entities and triples in the generated KGs using KGCaRe. KG Creation ApproachConditionalQA DatasetHotpot QA Dataset EntitiesTriplesEntities Triples KGCaRe736670311328617776 Knowledge Graphs and the developed software pipeline to construct them are available at the GitHub repository: https://github.com/GhanshyamVerma/KGCaRe 19 Appendix C Code Appendix Code for the existing approaches and the proposed KGCaRe approach is available at the following GitHub repository: https://github.com/GhanshyamVerma/KGCaRe Appendix D Technical Appendix This section provides technical details. D.1 Evaluation Avg F1 score:The Avg F1 score represents the calculated average F1 score using the exact match policy between predicted and ground truth answer, considering all the questions of the development set [ 10,11]. F1 score:The F1 score represents the calculated F1 score using the exact match policy between the predicted and ground truth answer [10,11]. The evaluation scripts were provided with the publicly available benchmark datasets, ConditionalQA [10] and HotpotQA [11]. We used the same scripts for our evaluation. Our code repository includes the evaluation script as well. Link to the code:https://github.com/GhanshyamVerma/KGCaRe D.2 Implementation Details ⢠Code Prompt: Note:Same parameters were used as mentioned in [ 12]. ⢠Text Prompt: Note:Same parameters were used as mentioned in [ 12]. ⢠Think on Graph: Temperature:0.1 Maximum Output Length:256 Width and Depth of Exploration:3 Note:Temperature is set to 0.1 for all models. The rest of the parameters are the same as mentioned in [ 14]. ⢠HybridContextQA: Note:Same parameters were used as mentioned in [ 19]. ⢠KGCaRe: Experimental Setup:A combination of commercial APIs and local model infer- ence was used. LLM Models and Prompting: âGPT-3.5 / GPT-4o(via OpenAI API using LlamaIndex): Used as-is without modifying default decoding parameters. The default tempera- ture of 0.7 from LlamaIndex was retained. âMistral-7B and Mixtral-8x7B(via vLLM + OpenAILike interface): Default decoding parameters (e.g., temperature, top p) as defined by 20 vLLM were used without manual modification. In-Context Examples: âConditionalQA:4 examples randomly selected from the training set. âHotpotQA:4 examples randomly selected from the training set. Retrievers: âVector-based Retriever(VectorIndexRetriever from LlamaIndex): similaritytopk = 10 âKnowledge Graph Traversal Retriever:maxdepth = 3, maxentities = 20 These parameters were fixed based on preliminary trials and prior work; no hyperparameter search was conducted. Reranking: âCohere Reranker:rerank-english-v3.0 with topn = 2 â No other configuration parameters were modified. Selection Criteria:Most parameters were adopted from the default settings of the respective tools (LlamaIndex, vLLM, Cohere). Deviations, such as reduc- ing the number of in-context examples, were based on practical constraints (e.g., token limits). No grid search or extensive hyperparameter tuning was conducted. D.3 Experimental Environment All experiments were conducted on a high-performance Linux machine with the following specifications: ⢠Operating System:Ubuntu 22.04.4 LTS (Jammy) ⢠CPU:AMD EPYC 7313P 16-Core Processor â 32 threads (16 cores, 2 threads per core) â Base frequency: 1.5 GHz â Max frequency: 3.0 GHz ⢠RAM:256 GB ⢠GPU:4ĂNVIDIA A40 (48 GB each) â Experiments utilized approximately 43 GB per GPU for inference ⢠CUDA: â CUDA Version: 12.8 â CUDA Toolkit: 12.6 (Build V12.6.85) â Driver Version: 570.133.07 ⢠Python Version:3.10.16 21