Paper deep dive

GroupRAG: Cognitively Inspired Group-Aware Retrieval and Reasoning via Knowledge-Driven Problem Structuring

Xinyi Duan, Yuanrong Tang, Jiangtao Gong

Year: 2026Venue: arXiv preprintArea: cs.IRType: PreprintEmbeddings: 42

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/31/2026, 1:33:00 AM

Summary

GroupRAG is a cognitively inspired framework that improves retrieval-augmented generation by decomposing complex, unstructured problems into latent structural groups. By identifying key information points and organizing them into knowledge-driven groups, the system performs parallel local reasoning and integrates these into a coherent global chain-of-thought, outperforming traditional RAG and CoT methods on the MedQA dataset.

Entities (5)

GroupRAG · framework · 100%MedQA · dataset · 100%Chain-of-Thought · reasoning-paradigm · 98%Retrieval-Augmented Generation · technique · 98%Llama3.1-8B · model · 95%

Relation Signals (3)

GroupRAG → evaluateson → MedQA

confidence 100% · To evaluate the effectiveness of GroupRAG... we adopt MedQA.

GroupRAG → outperforms → RAG-based baselines

confidence 95% · Experiments on MedQA show that GroupRAG outperforms representative RAG- and CoT-based baselines.

GroupRAG → uses → Llama3.1-8B

confidence 95% · We choose LLaMA3.1-8B as the base model.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The performance of language models is commonly limited by insufficient knowledge and constrained reasoning. Prior approaches such as Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) address these issues by incorporating external knowledge or enforcing linear reasoning chains, but often degrade in real-world settings. Inspired by cognitive science, which characterizes human problem solving as search over structured problem spaces rather than single inference chains, we argue that inadequate awareness of problem structure is a key overlooked limitation. We propose GroupRAG, a cognitively inspired, group-aware retrieval and reasoning framework based on knowledge-driven keypoint grouping. GroupRAG identifies latent structural groups within a problem and performs retrieval and reasoning from multiple conceptual starting points, enabling fine-grained interaction between the two processes. Experiments on MedQA show that GroupRAG outperforms representative RAG- and CoT-based baselines. These results suggest that explicitly modeling problem structure, as inspired by human cognition, is a promising direction for robust retrieval-augmented reasoning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

41,462 characters extracted from source content.

Expand or collapse full text

GroupRAG: Cognitively Inspired Group-Aware Retrieval and Reasoning via Knowledge-Driven Problem Structuring Xinyi Duan 1 , Yuanrong Tang 1 and Jiangtao Gong 1∗ 1 Tsinghua University duanxy23@mails.tsinghua.edu.cn, tangxtong2022@gmail.com, gongjiangtao2@gmail.com Abstract The performance of language models is com- monly limited by insufficient knowledge and constrained reasoning.Prior approaches such as Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) address these issues by incorporating external knowledge or enforcing lin- ear reasoning chains, but often degrade in real- world settings.Inspired by cognitive science, which characterizes human problem solving as search over structured problem spaces rather than single inference chains, we argue that inadequate awareness of problem structure is a key overlooked limitation.We propose GroupRAG, a cogni- tively inspired, group-aware retrieval and reasoning framework based on knowledge-driven keypoint grouping. GroupRAG identifies latent structural groups within a problem and performs retrieval and reasoning from multiple conceptual starting points, enabling fine-grained interaction between the two processes. Experiments on MedQA show that GroupRAG outperforms representative RAG- and CoT-based baselines. These results suggest that explicitly modeling problem structure, as in- spired by human cognition, is a promising direction for robust retrieval-augmented reasoning. 1 Introduction Language models have achieved remarkable progress across a wide range of tasks, yet they continue to struggle with com- plex, knowledge-dense questions that involve long contexts, multiple information sources, and intricate reasoning require- ments. Such challenges are particularly prominent in real- world scenarios. In medical decision making, for example, relevant information is often scattered, heterogeneous, and embedded in lengthy, partially noisy descriptions. Prior studies and empirical observations suggest that fail- ures on such problems can largely be attributed to two fac- tors: insufficient access to relevant knowledge and limited ability to reason over that knowledge. To address these is- sues, two major lines of research have emerged. Retrieval- ∗ Corresponding author. Figure 1: Reasoning Paradigms Comparison. Traditional CoT fol- lows Linear/Divergent paths on unstructured sequences. GroupRAG transforms monolithic inputs into a structured problem space, em- ploying a Convergent net via keypoint grouping for real-world align- ment. Augmented Generation (RAG) incorporates external infor- mation sources to alleviate the reliance on parametric mem- ory for knowledge-dense tasks [ Lewis et al., 2020 ] . In par- allel, Chain-of-Thought (CoT) prompting and related distil- lation methods improve reasoning performance by explic- itly modeling intermediate inference steps [ Wei et al., 2022; Hsieh et al., 2023 ] . Despite their effectiveness, existing RAG- and CoT-based approaches exhibit notable limitations in complex, real-world settings. In RAG systems, retrieved chunks often fail to pre- cisely match the information actually required to answer the question, and models may struggle to align, filter, and in- tegrate retrieved content into coherent reasoning [ Izacard et al., 2023 ] . CoT-based methods, while improving reasoning fluency, remain heavily dependent on the model’s internal knowledge: when critical facts are missing or misaligned, the resulting reasoning chains may appear coherent but are grounded in incomplete or incorrect premises. Consequently, simply retrieving more information or generating longer rea- soning chains is often insufficient for reliably solving com- plex problems. Recent work has attempted to bridge this gap through ei- ther structured retrieval mechanisms or tighter integration of arXiv:2603.26807v1 [cs.IR] 26 Mar 2026 retrieval and reasoning, such as organizing external knowl- edge using graphs and interleaving retrieval steps within rea- soning traces, respectively [ Trivedi et al., 2023; Wang et al., 2024; Fan et al., 2025; Guo et al., 2024 ] . While promising, these approaches often increase system complexity or still op- erate at a coarse granularity, treating each question as a single undifferentiated unit for retrieval and reasoning. A key insight underlying this work is that the difficulty of complex real-world questions—unlike formal mathemat- ical or symbolic reasoning problems—lies not only in miss- ing knowledge or insufficient reasoning capacity, but in how the problem is internally structured and represented [ Newell and Simon, 1972; Barto et al., 2013; Eckstein and Collins, 2021 ] . Cognitive science has long shown that human prob- lem solving is highly sensitive to problem representation: complex tasks are understood and solved by organizing in- formation into structured problem spaces rather than treating them as undifferentiated sequences [ Newell and Simon, 1972; Cushen and Wiley, 2012; Ho et al., 2022 ] . In real-world ques- tion answering, inputs are rarely monolithic. For instance, when a patient describes their condition to a physician, the information is often conveyed through a lengthy, loosely or- ganized narrative that interleaves symptoms, medical history, test results, and irrelevant details. However, language mod- els are typically forced to process such inputs as a single flat sequence, leading retrieval to operate at an inappropriate granularity. This representational mismatch in turn gives rise to entangled and error-prone reasoning [ Zhang et al., 2023; Eckstein and Collins, 2021 ] . This perspective motivates a different design principle: rather than pushing models to retrieve more information or to generate increasingly long reasoning chains, we aim to enable them to reason over a structured problem space by explic- itly uncovering the latent structure of a question. From this view, effective real-world reasoning resembles human prob- lem solving: it begins by identifying meaningful substruc- tures, proceeds through parallel inference from multiple con- ceptual starting points, and gradually integrates these partial inferences into a coherent conclusion, as illustrated in Fig- ure 1. Accordingly, we focus on identifying key information points within a question and organizing them into meaningful groups that reflect underlying knowledge relations, thereby providing an explicit structural scaffold for both retrieval and reasoning. Based on this design principle, we propose GroupRAG, a cognitively inspired, group-aware retrieval-and-reasoning framework based on knowledge-driven keypoint grouping. GroupRAG treats grouping as a first-class operation that makes the internal structure of a question explicit, trans- forming unstructured inputs into a set of structured reasoning units. Retrieval and reasoning are then performed at the group level and subsequently integrated, allowing the two processes to be temporarily decoupled yet mutually reinforcing: re- trieval provides group-specific knowledge at an appropriate granularity, while reasoning over each group guides the se- lection and integration of relevant information toward a final answer. By explicitly discovering the latent grouping structure of knowledge points within a question, our approach reframes complex real-world reasoning as structured exploration over a problem space. This decomposition transforms a mono- lithic reasoning task into a set of lower-complexity, group- level subtasks, mirroring how humans organize and navigate complex problem representations in cognitive science. As a result, retrieval and reasoning are jointly constrained at an ap- propriate granularity, yielding modular and interpretable in- termediate reasoning traces that are amenable to supervision, distillation, and alignment. • We introduce a cognitively inspired, group-aware retrieval-and-reasoning framework that explicitly mod- els the internal structure of complex questions by or- ganizing key information points into knowledge-driven groups, enabling retrieval and reasoning to operate at an appropriate, structure-aware granularity. • We reformulate conventional Chain-of-Thought reason- ing from a single linear chain or divergent tree into a convergent reasoning net, in which inference is initi- ated from multiple grouped reasoning roots, augmented with group-specific retrieval, and progressively inte- grated into a coherent global conclusion. • We demonstrate that GroupRAG consistently outper- forms a wide range of RAG-based and CoT-based meth- ods on knowledge-intensive medical question answer- ing, providing empirical evidence that explicit prob- lem structuring—rather than longer reasoning chains alone—is critical for robust real-world reasoning. 2 Related Work 2.1 Chain-of-Thought Chain-of-Thought (CoT) prompting was introduced to elicit explicit multi-step reasoning in large language models by generating intermediate inference steps before a final an- swer, improving performance on reasoning tasks [ Wei et al., 2022 ] . Subsequent enhancements improve robustness and precision, including methods that sample multiple reason- ing paths, structure reasoning from simple to complex sub- problems, or generate executable representations for struc- tured reasoning [ Wang et al., 2022; Zhou et al., 2022; Chen et al., 2022 ] .Beyond linear chains, recent work adopts richer reasoning structures such as trees, forests, or more complex graphs to explore multiple solution trajecto- ries [ Yao et al., 2023; Chen et al., 2025; Bi et al., 2024; Pandey et al., 2025 ] . Despite these advances, most CoT ap- proaches—including those using tree, forest, or more com- plex graph structures—still rely on a single start point, high- lighting the continued reliance on single-start inference paths in current CoT and its extensions. 2.2 Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) improves language models by retrieving external knowledge, reducing re- liance on parametric memory for knowledge-intensive tasks [ Lewis et al., 2020 ] .Early RAG frameworks retrieve unstructured text passages to condition genera- tion [ Izacard et al., 2023 ] , while more recent work en- hances retrieval relevance and efficiency through task-aware Figure 2: Illustration of the GroupRAG reasoning process on a clinical case, featuring keypoint extraction, knowledge-driven grouping, local and global reasoning, and answer alignment. queries or structured, knowledge-graph-based representa- tions [ Lee et al., 2024; Guo et al., 2024; Fan et al., 2025 ] . To better connect external knowledge with reasoning, recent work combines RAG with Chain-of-Thought, integrating re- trieval with intermediate reasoning steps to produce retrieval- augmented chains of thought applicable to multi-step and domain-specific tasks [ Trivedi et al., 2023; Wang et al., 2024; Mavi et al., 2023; Li et al., 2024; Ma et al., 2023 ] . 2.3 Reasoning in Cognitive Science Human problem solving has long been characterized in cog- nitive science as a process of constructing and navigating structured problem spaces, rather than following a single lin- ear chain of inference. Classic work by Newell and Simon conceptualizes reasoning as search over an internal problem space defined by states, operators, and goals, with success depending on how the problem is represented and explored rather than on a fixed inference trajectory [ Newell and Simon, 1972 ] . This perspective highlights that human reasoning nat- urally involves multiple paths and intermediate states shaped by understanding the problem’s internal structure. Subsequent research in cognitive psychology shows that reasoning is highly sensitive to problem representation: inap- propriate initial representations can lead to impasses, while restructuring or dynamically adapting task representations enables new solution paths and guides exploration [ Barto et al., 2013; Cushen and Wiley, 2012; Ho et al., 2022 ] . These findings collectively indicate that real-world reasoning is not purely chain-based, but involves recognizing latent problem structure and exploring solutions from multiple conceptual starting points, gradually converging towards a solution. 3 Method 3.1 Group-Aware Retrieval and Reasoning To illustrate the workflow of GroupRAG, we take complex real-world medical problems as representative examples. Our goal is to model such problems in order to identify multiple starting points for reasoning chains. Inspired by how human students approach complex problems, we first employ a lan- guage model to extract key information points from the prob- lem. This step, termed Keypoint Extraction, is analogous to how students highlight or circle important information in a problem. The extracted keypoints are then organized to achieve a structured representation of the problem. We implement a Knowledge-Driven Grouping strategy, where a fine-tuned model leverages retrieved external knowledge to group re- lated keypoints. This process resembles how students loop up reference materials to link strongly associated information points. Unlike conventional semantic-matching-based group- ing, our approach is knowledge-driven, enabling more mean- ingful organization. After grouping, each group corresponds to a specific knowledge concept or category label. Through Keypoint Extraction and Knowledge-Driven Grouping, we achieve information structuring, transforming complex and lengthy problems into keypoint groups. Each group is then treated as an independent starting point for reasoning. We perform groupwise retrieval and reasoning, constrained by problem conditions. This approach narrows retrieval keywords, shifting from coarse-grained (problem- level) to fine-grained (group-level) retrieval, while also limit- ing the reasoning scope to reduce interference from unrelated domains. The outcome is multiple Local Reasoning conclu- sions, each corresponding to a keypoint group. These local reasoning conclusions can be categorized into three types according to their relevance and contribution to the problem: core conclusions (Core), supporting conclu- sions (Support), and noise (Noise). Building upon this, we utilize a model to identify and select the conclusions cate- gorized as Core or Support, and subsequently integrate them into a coherent global Chain-of-Thought (CoT). The CoT ob- tained by fusing multiple local reasoning conclusions consti- tutes the Global Reasoning. Global Reasoning produces a readable and high- confidence reasoning chain.To align with downstream evaluation, we perform an Answer Alignment step. Starting from Global Reasoning, the model conducts fine-grained retrieval over candidate answer options, outputting the correct choice, option analysis, and rationale.Empirical studies have shown that this step is crucial for small language models, preventing situations where the CoT is correct but the final selected option is wrong. 3.2 System Design Modular Pipeline and Stage-wise Training Building on the five-stage workflow of GroupRAG— Keypoint Extraction, Knowledge-Driven Grouping, Local Reasoning, Global Reasoning, and Answer Alignment—this section presents the design details of the system. Each stage has independent inputs and outputs, functioning as sequen- tially connected modules. To ensure high-quality outputs at each module while reduc- ing deployment and experimental costs, we implement each module using a dedicated, fine-tuned small language model. Concretely, we first run a large training dataset of medical questions through the complete pipeline, recording the inputs and outputs at each of the five modules. These records are then used as hard labels to fine-tune five base models sepa- rately, effectively creating experts specialized for each sub- task. Among these modules, the Global Reasoning module is tasked with evaluating contributions of multiple local reason- ing conclusions to answering the question. This process is inherently soft and context-dependent as multiple combina- tions of conclusions can be valid. Supervised fine-tuning with hard labels is thus limited in capturing these nuanced depen- dencies. To better align the module with this task, we adopt a reinforcement learning (RL) approach, specifically utiliz- ing a policy gradient method to fine-tune the selection policy against a custom-designed reward function. Global Reasoning Optimization The Global Reasoning module consists of two steps. First, a selection model identifies and selects the local reasoning con- clusions that serve as Core or Support conclusions. Second, a synthesis model combines the selected conclusions into a co- herent global Chain-of-Thought. We independently fine-tune the models used in each step of Global Reasoning, and further optimize the first-step model with a policy gradient method to enhance the selection precision. The key principle in Global Reasoning is to ensure that all Cores are fully included, Noises are avoided, and Supports are encouraged. To quantitatively guide the model towards this goal, we design a reward function that captures the se- lection quality of local reasoning conclusions, defined as the Weighted Inference F-score (WIF). Weighted Inference F-score (WIF). Let the sets of local reasoning conclusions be C =C i (Core),S =S j (Support),N =N k (Noise). For a model-selected subset of local reasoning P ⊂ C∪ S∪ N , we define: Core Recall: R c = |P ∩ C| |C| or 1 (if|C| = 0) Support Recall: R s = |P ∩ S| |S| or 0 (if|S| = 0) Noise Recall: R n = |P ∩ N| |N| or 0 (if|N| = 0) WIF(P ) = R α c · (1− R n ) β · (1 + γR s ) where the hyperparameters are chosen such that α≥ β ≫ γ, prioritizing coverage of Cores, penalizing selection of Noises, and mildly encouraging inclusion of Supports. In practice, we set α = 2.5, β = 2, γ = 0.5. If P = ∅, we assign WIF(P ) = 0 to discourage the model from skipping all conclusions. Policy Optimization. We treat the model as a stochastic pol- icy π θ (P | x) that outputs a subset of local reasoning conclu- sions P given a problem x. For each problem, we generate K candidate selections (rollouts) P k K k=1 and compute the corresponding rewards R k = WIF(P k ). The baseline reward for advantage estimation is the mean reward over rollouts: ̄ R = 1 K K X k=1 R k , ˆ A k = R k − ̄ R std(R i ) + ε . We parameterize the model π θ (P | x) as independent Bernoulli distributions over each local reasoning conclusion. For conclusion l i , the model outputs a probability p i = π θ (l i = 1 | x) of being selected. A rollout P k is gener- ated by sampling each l i independently according to p i . The log-probability of P k is then computed as logπ θ (P k | x) = X l i ∈P k logp i + X l i /∈P k log(1− p i ), The policy is then updated using a policy gradient loss: L =− 1 K K X k=1 ˆ A k · logπ θ (P k | x).(1) This training procedure ensures that the model learns to se- lect local reasoning conclusions that maximize WIF, produc- ing a coherent global Chain-of-Thought that prioritizes Core while avoiding Noise. Figure 3: An Abstract Overview of GroupRAG. 3.3 Retrieval-Augmented Generation in GroupRAG Retrieval-Augmented Generation (RAG) plays a crucial role in GroupRAG and is integrated into three key modules: Knowledge-Driven Grouping, Local Reasoning, and Answer Alignment. Rather than serving as a standalone retrieval com- ponent, RAG is tightly coupled with the groupwise reason- ing structure and operates at different granularities across the pipeline. In Knowledge-Driven Grouping, RAG is applied at the keypoint level. For each extracted keypoint, the model re- trieves external knowledge to ground it in a relevant knowl- edge context.When multiple keypoints retrieve overlap- ping or highly related knowledge, they are more likely to be grouped together, as they are inferred to be associated with the same underlying medical concept. This retrieval-guided grouping enables the model to organize keypoints based on shared knowledge relevance, rather than surface-level seman- tic similarity. In Local Reasoning, RAG operates at the group level. Each group of keypoints is treated as a unified query for retrieval, with the objective of identifying knowledge that jointly ex- plains multiple keypoints. This groupwise retrieval is par- ticularly important in knowledge-intensive domains such as medicine. For example, retrieving information for two symp- toms independently may lead to different candidate diseases, whereas retrieving their combination may reveal that they are associated symptoms of the same disease. By conditioning retrieval on grouped keypoints, the model is able to perform more precise and context-aware reasoning. RAG in Answer Alignment is demonstrated in Subsec- tion 3.1. Overall, RAG in GroupRAG is adapted to different stages of the reasoning process, operating at varying granu- larities from the keypoint level to the group level and the op- tion level. This group-aware integration of retrieval enables GroupRAG to progressively refine both the granularity and relevance of retrieved knowledge, forming a solid foundation for the subsequent reasoning process. 4 Experiments 4.1 Dataset and Model To evaluate the effectiveness of GroupRAG in address- ing complex real-world reasoning problems, we adopt MedQA [ Jin et al., 2021 ] , a USMLE-style medical dataset, for both training and evaluation. MedQA consists of long and information-dense clinical case descriptions that require multi-step reasoning and span a wide range of domains in ba- sic and clinical medicine. From MedQA, we randomly select 2,000 questions as the training set, ensuring balanced cover- age across different medical specialties and question types. Following the procedure described in Subsection 3.2, each training question is sequentially processed by the five mod- ules of GroupRAG, each instantiated with GPT-4o [ OpenAI, 2024 ] , with the module-wise outputs recorded as intermedi- ate supervision signals for training. We choose LLaMA3.1-8B [ Dubey et al., 2024 ] as the base model. Using the collected intermediate data, the base model is trained and specialized into five lightweight sub-models, each corresponding to a specific sub-task in the GroupRAG pipeline. For evaluation, we construct a separate test set of 400 information-dense questions, some of which include distracting or irrelevant details, aiming to assess the robust- ness and reasoning capability of GroupRAG under challeng- ing conditions. 4.2 Evaluation Metrics We design stage-wise evaluation metrics for the five modules of GroupRAG. Keypoint Extraction. Extraction performance is evaluated using precision, recall, and F1 score. For each question, the keypoints extracted by the trained lightweight model are com- pared against a gold standard set of keypoints extracted by GPT-4o. Precision is defined as the proportion of predicted keypoints that correctly match the gold standard keypoints, while recall measures the proportion of gold standard key- points that are successfully recovered. The F1 score is com- puted as the harmonic mean of precision and recall, reflecting the accuracy of keypoint extraction. Knowledge-Driven Grouping. The grouping stage aims to partition the extracted keypoints into groups associated with the same pieces of knowledge, which naturally constitutes a clustering task rather than a classification task. We adopt the BCubed evaluation, which is specifically designed for entity-level clustering evaluation [ Bagga and Baldwin, 1998 ] . BCubed Precision measures, for each keypoint, the propor- tion of other keypoints in the same predicted group that also belong to the same gold-standard group generated by GPT- 4o. BCubed Recall measures the proportion of keypoints in the gold-standard group that are correctly placed into the same predicted group. BCubed F1 is computed as the har- monic mean of BCubed Precision and BCubed Recall, and the final score is obtained by averaging over all keypoints. Local and Global Reasoning. For local reasoning based on each keypoint group, we use GPT-4o to evaluate the factual and logical correctness of each inference, and then compute the overall accuracy over all inferences. For global reasoning, we adopt the WIF function defined in Subsection 3.2 to assess the model’s ability to distinguish Core, Support, and Noise lo- cal conclusions. This ensures that the assembled global rea- soning covers all essential conclusions while filtering out dis- tracting or irrelevant inferences. Answer Alignment. To evaluate the final output of GroupRAG, we compare the selected answer options with the correct options in the dataset, and calculate the overall accuracy across the test set. This accuracy serves as the pri- mary metric for horizontal comparison with other models and methods. 4.3 Experimental Design To systematically evaluate the effectiveness of GroupRAG and analyze the contribution of its individual modules, we design three sets of experiments: leave-one-out ablation, pro- gressive ablation, and joint comparison across different mod- els and methods. Leave-One-Out Ablation. In this setting, we assess the marginal contribution of each module in GroupRAG by re- moving one module at a time while keeping all other compo- nents unchanged. Specifically, for each of the five modules, we either replace the specially trained model with the base model, or remove the RAG component within the module, if it originally contains one. Each ablation variant is evaluated using the same set of metrics as the complete GroupRAG system. The performance of each variant is then compared against the complete GroupRAG pipeline, enabling a fine- grained analysis of the individual impact of each module. Progressive Ablation. While leave-one-out ablation fo- cuses on isolated effects, progressive ablation is designed to examine the cumulative contribution of GroupRAG’s mod- ules. Starting from the complete GroupRAG system, we pro- gressively replace trained models with the base model and sequentially remove RAG modules in the order they appear in the pipeline. This process continues until the system de- generates into a baseline configuration composed entirely of Extract F1 Group F1 Local Acc.(%) Global WIF Answer Acc.(%) GroupRAG0.962 0.802 73.14 1.13 71.75 w/o Ext. Train 0.951 0.81670.251.0668.50 w/o Gro. Train0.962 0.71464.520.9363.00 w/o Loc. Train0.962 0.802 61.870.8564.25 w/o Glo. Train0.962 0.80273.14 0.6968.25 w/o Ans. Train0.962 0.80273.141.13 67.50 w/o Gro. RAG0.962 0.70662.870.9164.50 w/o Loc. RAG0.962 0.802 59.540.8663.25 w/o Ans. RAG0.962 0.80273.141.13 67.25 Table 1: Leave-One-Out Ablation Results. Each variant is compared with the full pipeline (first row) to demonstrate the marginal contri- bution of each module. “w/o” indicates removing each component independently from the full pipeline. base models and without any RAG component. Each experi- mental setting is compared with the preceding one, allowing us to observe how performance evolves as modules are incre- mentally removed. Joint Comparison Across Models and Methods. In the final set of experiments, we aim to evaluate the capability gap of small language models under different reasoning and retrieval paradigms, and to compare their performance with a reference model. To this end, we conduct a horizontal comparison across different models and methods, evaluating LLM, SLM, and trained SLM under CoT prompting, standard RAG, and GroupRAG. Specifically, for the trained SLM set- ting, models are separately fine-tuned under different supervi- sion signals corresponding to each method, such as question- answer pairs, CoT, and the intermediate data of GroupRAG. For the LLM setting, GPT-4o is included as a reference model to provide an approximate upper bound on task performance. For clarity and consistency, we focus on final answer accu- racy as the sole evaluation metric in this comparison. 4.4 Results Leave-One-Out Ablation. Results are shown in Table 1. Since the five modules are executed sequentially, ablating a specific module only affects its own evaluation metric and those of downstream modules, while leaving upstream met- rics unchanged. Compared with the complete GroupRAG system, all ablation variants exhibit varying degrees of degra- dation in final answer accuracy, indicating that each of the five trained modules and their associated RAG components plays a role in the overall performance of GroupRAG. A closer comparison across ablation variants shows that four configurations experience the largest drops in answer accu- racy (approximately 8%), corresponding to the removal of trained models and RAG components in the Knowledge- Driven Grouping and Local Reasoning modules. In contrast, ablating the Keypoint Extraction and Global Reasoning mod- ules leads to relatively smaller decreases in final accuracy (ap- proximately 3%). Finally, when comparing each ablation set- ting with the complete GroupRAG configuration, we observe consistent degradation in downstream evaluation metrics fol- Extract F1 Group F1 Local Acc.(%) Global WIF Answer Acc.(%) Acc. ∆(%) GroupRAG 0.962 0.802 73.14 1.13 71.75 −Ext. Train 0.951 0.81670.251.0668.50 -3.25 −Gro. Train0.951 0.72462.160.9462.75 -5.75 −Gro. RAG0.951 0.70261.890.8861.00 -1.75 −Loc. Train0.951 0.702 58.570.7556.75 -4.25 −Loc. RAG0.951 0.702 55.740.6956.00 -0.75 −Glo. Train0.951 0.70255.74 0.5553.75 -2.25 −Ans. Train0.951 0.70255.740.55 52.75 -1.00 −Ans. RAG0.951 0.70255.740.55 51.00 -1.75 Table 2: Progressive Ablation Results. Each variant is compared with the previous row to demonstrate the cumulative effect of the integrated modules. “−” indicates cumulative removal. Model Method -(%) +CoT Prompting(%) +Naive RAG(%) +Group- RAG(%) GPT-4o89.0089.7587.7585.25 L3.1-8B(base)48.2554.5053.5061.00 L3.1-8B(trained) 52.7561.5058.2571.75 Table 3: Joint Comparison Across Models and Methods. L3.1-8B refers to LLaMA 3.1-8B. lowing the removal of upstream modules. Progressive Ablation. Results are shown in Table 2. As modules are removed sequentially following the pipeline or- der, evaluation metrics of downstream stages exhibit a grad- ual decline, forming a stage-wise degradation pattern that aligns with the pipeline structure. As fewer modules are re- tained, degradation in upstream outputs and intermediate rea- soning quality accumulates and results in a monotonic de- crease in final answer accuracy. The final column, denoted as ∆, reports the accuracy drop of each configuration relative to the previous one. Among all transitions, removing the trained models of the Grouping and Local Reasoning modules leads to the largest accuracy declines. Joint Comparison Across Models and Methods. Re- sults are shown in Table 3.For the untrained base model LLaMA3.1-8B, CoT prompting and Naive RAG yield accuracy improvements of approximately 5-6%, whereas GroupRAG leads to a substantially larger gain of around 13%.Across all retrieval and reasoning methods, the trained LLaMA3.1-8B outperforms its untrained counter- part. Among these configurations, applying GroupRAG to the trained small language model achieves the highest accu- racy within the small model setting, reaching 71.75%. As a reference, GPT-4o achieves the best performance under both the non-augmented setting (89.00%) and CoT prompt- ing (89.75%), while incorporating Naive RAG or GroupRAG results in slight performance degradation. 4.5 Discussion Results of the leave-one-out ablation study indicate that prop- erly uncovering problem structure and conducting local rea- soning are central to accurate problem solving. In contrast, tasks that are more procedural in nature—for example, ex- tracting information—can be properly handled by untrained small models, and thus have limited impact on the final an- swer accuracy. Results of the progressive ablation study demonstrate that the performance gain of GroupRAG arises from the cumula- tive synergy between modules. The quality of upstream out- puts directly affects subsequent reasoning, and any upstream degradation is amplified through the pipeline, ultimately im- pacting the final answer. Joint comparison across models and methods indicates that GroupRAG effectively compensates for the knowledge and reasoning limitations of small language models, enabling them to answer complex questions more accurately and ro- bustly. In contrast, for large language models with strong inherent knowledge coverage and implicit reasoning capabil- ities (e.g., GPT-4o), introducing GroupRAG may slightly re- duce accuracy. This suggests that external retrieval and struc- tured reasoning may introduce redundant information or in- terfere with the efficient internal reasoning process of large language models. Overall, this experiment highlights the sig- nificant benefits of GroupRAG for small language models, while providing a reasonable explanation for the observed ef- fects on large language models. Future work could explore incorporating a multi-agent collaboration mechanism within the GroupRAG framework. This would replace the current fixed modular reasoning pipeline, allowing different agents to coordinate dynamically during reasoning and uncover potential latent structures in the problem. Another direction is to develop more sophisticated methods for modeling internal problem structures, which could better constrain retrieval and reasoning, thereby im- proving both accuracy and robustness across tasks and model scales. 5 Conclusion In this paper, we propose GroupRAG, a cognitively inspired, group-aware retrieval and reasoning framework that explic- itly models the internal structure of complex questions. By reformulating conventional Chain-of-Thought into a conver- gent reasoning net, GroupRAG enables inference to proceed from multiple grouped reasoning roots in parallel and eventu- ally converge into a coherent global conclusion. GroupRAG allows retrieval and reasoning to be temporarily decoupled yet mutually reinforcing, with retrieval providing knowledge at an appropriate granularity and reasoning guiding the se- lection and integration of relevant information. Empirical evaluations on knowledge-intensive medical problems vali- date that GroupRAG outperforms representative RAG- and CoT-based methods. Ultimately, this work highlights that ex- plicitly modeling problem structure is a promising direction for robust real-world reasoning, beyond the mere extension of reasoning chains. References [ Bagga and Baldwin, 1998 ] Amit Bagga and Breck Baldwin. Entity-based cross-document coreferencing using the vec- tor space model. In COLING 1998 Volume 1: The 17th in- ternational conference on computational linguistics, 1998. [ Barto et al., 2013 ] Andrew G Barto, George Konidaris, and Christopher Vigorito. Behavioral hierarchy: exploration and representation. In Computational and robotic models of the hierarchical organization of behavior, pages 13–46. Springer, 2013. [ Bi et al., 2024 ] Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test- time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078, 2024. [ Chen et al., 2022 ] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.Program of thoughts prompting:Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. [ Chen et al., 2025 ] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che.To- wards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025. [ Cushen and Wiley, 2012 ] Patrick J. Cushen and Jennifer Wiley. Cues to solution, restructuring patterns, and reports of insight in creative problem solving. Consciousness and Cognition, 21(3):1166–1175, 2012. [ Dubey et al., 2024 ] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407, 2024. [ Eckstein and Collins, 2021 ] Maria K Eckstein and Anne GE Collins. How the mind creates structure: Hierarchical learning of action sequences. In Cogsci... annual confer- ence of the cognitive science society. cognitive science so- ciety (us). conference, volume 43, page 618, 2021. [ Fan et al., 2025 ] Tianyu Fan, Jingyuan Wang, Xubin Ren, and Chao Huang.Minirag: Towards extremely sim- ple retrieval-augmented generation. arXiv preprint arXiv:2501.06713, 2025. [ Guo et al., 2024 ] Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang.Lightrag:Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779, 2024. [ Ho et al., 2022 ] Mark K Ho, David Abel, Carlos G Correa, Michael L Littman, Jonathan D Cohen, and Thomas L Griffiths. People construct simplified mental representa- tions to plan. Nature, 606(7912):129–136, 2022. [ Hsieh et al., 2023 ] Cheng-Yu Hsieh, Chun-Liang Li, Chih- Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Dis- tilling step-by-step! outperforming larger language mod- els with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, 2023. [ Izacard et al., 2023 ] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learn- ing Research, 24(251):1–43, 2023. [ Jin et al., 2021 ] Di Jin, Eileen Pan, Nassim Oufattole, Wei- Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- ease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. [ Lee et al., 2024 ] Yoonsang Lee, Minsoo Kim, and Seung- won Hwang.Disentangling questions from query generation for task-adaptive retrieval. arXiv preprint arXiv:2409.16570, 2024. [ Lewis et al., 2020 ] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ̈ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈ aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural infor- mation processing systems, 33:9459–9474, 2020. [ Li et al., 2024 ] Mingchen Li, Huixue Zhou, Han Yang, and Rui Zhang. Rt: a retrieving and chain-of-thought frame- work for few-shot medical named entity recognition. Jour- nal of the American Medical Informatics Association, 31(9):1929–1938, 2024. [ Ma et al., 2023 ] Xilai Ma, Jing Li, and Min Zhang. Chain of thought with explicit evidence reasoning for few-shot re- lation extraction. arXiv preprint arXiv:2311.05922, 2023. [ Mavi et al., 2023 ] VaibhavMavi,AbulhairSaparov, and Chen Zhao.Retrieval-augmented chain-of- thought in semi-structured domains. arXiv preprint arXiv:2310.14435, 2023. [ Newell and Simon, 1972 ] Allen Newell and Herbert A. Si- mon. Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ, 1972. [ OpenAI, 2024 ] OpenAI.GPT-4o system card.https:// openai.com/index/gpt-4o-system-card/, 2024. [ Pandey et al., 2025 ] Tushar Pandey, Ara Ghukasyan, Ok- tay Goktas, and Santosh Kumar Radha.Adaptive graph of thoughts: Test-time adaptive reasoning unify- ing chain, tree, and graph structures. arXiv preprint arXiv:2502.05078, 2025. [ Trivedi et al., 2023 ] Harsh Trivedi, Niranjan Balasubrama- nian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational lin- guistics (volume 1: long papers), pages 10014–10037, 2023. [ Wang et al., 2022 ] Xuezhi Wang, Jason Wei, Dale Schu- urmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. [ Wang et al., 2024 ] Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. Rat: Retrieval aug- mented thoughts elicit context-aware reasoning in long- horizon generation. arXiv preprint arXiv:2403.05313, 2024. [ Wei et al., 2022 ] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. [ Yao et al., 2023 ] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural informa- tion processing systems, 36:11809–11822, 2023. [ Zhang et al., 2023 ] Jiajie Zhang, Shulin Cao, Tingjian Zhang, Xin Lv, Juanzi Li, Lei Hou, Jiaxin Shi, and Qi Tian. Reasoning over hierarchical question decomposition tree for explainable question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14556–14570, 2023. [ Zhou et al., 2022 ] Denny Zhou, Nathanael Sch ̈ arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least- to-most prompting enables complex reasoning in large lan- guage models. arXiv preprint arXiv:2205.10625, 2022.

Instant research discovery