Paper deep dive
GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG
Paolo Pedinotti, Enrico Santus
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/24/2026, 3:02:13 AM
Summary
The paper introduces GradRAG, a framework for cross-component prompt adaptation in multi-agent Retrieval-Augmented Generation (RAG) systems. It models the RAG pipeline as a computational graph where an Evaluator critiques downstream answers and propagates structured feedback to update upstream agents (retrievers, graph constructors) via a Prompt Optimizer. Evaluated on SQuALITY and QMSum benchmarks, GradRAG outperforms one-step refinement baselines by 12-15 percentage points in LLM-judged pairwise comparisons, demonstrating effective coordination across flat chunk-based and graph-based retrieval paradigms.
Entities (10)
Relation Signals (8)
GradRAG → uses → Prompt Optimizer
confidence 95% · a Prompt Optimizer uses to iteratively update adaptive agents
GradRAG → uses → Evaluator
confidence 95% · An Evaluator critiques downstream answers and supporting evidence, producing actionable feedback
GradRAG → evaluatedon → SQuALITY
confidence 92% · We evaluate GradRAG on the SQuALITY and QMSum benchmarks
GradRAG → evaluatedon → QMSum
confidence 92% · We evaluate GradRAG on the SQuALITY and QMSum benchmarks
GradRAG → outperforms → One-Step Refinement Baselines
confidence 90% · GradRAG consistently outperforms one-step refinement baselines... achieving a 12-15 percentage point net preference margin
Evaluator → triggers → Early Stopping
confidence 90% · The Evaluator also triggers early stopping when the output is deemed satisfactory
IRCoT → proposedby → Trivedi et al.
confidence 88% · IRCoT-style query refinement (Trivedi et al., 2023)
GradRAG → implementedwith → Microsoft AutoGen
confidence 85% · All pipelines are implemented using the Microsoft AutoGen framework
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Retrieval-Augmented Generation (RAG) systems increasingly employ multiple LLM agents. Yet, most prior work optimizes components in isolation rather than coordinating improvements across the pipeline. We introduce GRADRAG, a framework for cross-component prompt adaptation that models the RAG pipeline as a computational graph and propagates structured evaluation feedback to update upstream agents. An Evaluator critiques downstream answers and supporting evidence, producing actionable feedback that a Prompt Optimizer uses to iteratively update adaptive agents, such as retrievers, graph constructors, and answerers. The Evaluator also triggers early stopping when the output is deemed satisfactory. We evaluate GRADRAG on the SQUALITY and QMSUM benchmarks under two retrieval paradigms: flat chunk-based retrieval using IRCoT-style query refinement (Trivedi et al., 2023), and graph-based retrieval that constructs and iteratively enriches an entity-relation graph from the document. Across both settings, GRADRAG consistently outperforms one-step refinement baselines that update only the final generator, achieving a 12-15 percentage point net preference margin in LLM-judged pairwise comparisons, with most gains realized within two refinement iterations.
Tags
Links
- Source: https://arxiv.org/abs/2607.21324v1
- Canonical: https://arxiv.org/abs/2607.21324v1
Trouble viewing inline? Open PDF directly →
Full Text
55,192 characters extracted from source content.
Expand or collapse full text
GradRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG Paolo Pedinotti Bloomberg pedinotti.paolo@gmail.com &Enrico Santus Bloomberg esantus@bloomberg.net Paolo Pedinotti contributed to this work during his internship at Bloomberg.Equal contribution. Abstract Retrieval-Augmented Generation (RAG) systems increasingly employ multiple LLM agents. Yet, most prior work optimizes components in isolation rather than coordinating improvements across the pipeline. We introduce GradRAG, a framework for cross-component prompt adaptation that models the RAG pipeline as a computational graph and propagates structured evaluation feedback to update upstream agents. An Evaluator critiques downstream answers and supporting evidence, producing actionable feedback that a Prompt Optimizer uses to iteratively update adaptive agents (e.g., retrievers, graph constructors, answerers). The Evaluator also triggers early stopping when the output is deemed satisfactory. We evaluate GradRAG on the SQuALITY and QMSum benchmarks under two retrieval paradigms: (i) flat (chunk-based) retrieval using IRCoT-style query refinement (Trivedi et al., 2023), and (i) graph-based retrieval that constructs and iteratively enriches an entity–relation graph from the document. Across both settings, GradRAG consistently outperforms one-step refinement baselines that update only the final generator, achieving a 12–15 percentage point net preference margin in LLM-judged pairwise comparisons, with most gains realized within two refinement iterations. GradRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG Paolo Pedinotti†thanks: Paolo Pedinotti contributed to this work during his internship at Bloomberg. Bloomberg pedinotti.paolo@gmail.com Enrico Santus†thanks: Equal contribution. Bloomberg esantus@bloomberg.net Figure 1: GradRAG: cross-component prompt adaptation in an agentic RAG pipeline. Grey (forward pass): A Retrieval Agent gathers evidence from either document chunks (Vector RAG) or a structured entity–relation graph (GraphRAG, constructed by a separate agent that is not shown in the mock up), which is assembled into a context and passed to an Answer Generation Agent to produce a candidate answer. Orange (evaluation and control): An Evaluator reviews the answer and its supporting evidence and returns feedback with a binary Stop? decision. If Yes, the answer is accepted. If No, a Prompt Optimizer updates the prompts of one or more agents (as indicated by the evaluation), and the next forward pass is executed. 1 Introduction Retrieval-Augmented Generation (RAG) has become a central paradigm for enabling large language models (LLMs) to reason over information beyond their parametric knowledge. Early RAG systems followed a simple retrieve-then-generate workflow in which a retriever selects relevant text and a generator produces an answer conditioned on it. More recently, RAG architectures have evolved into increasingly agentic systems, where multiple LLM-based components cooperate to formulate retrieval queries, organize evidence, perform multi-step reasoning, and evaluate or revise intermediate outputs. Despite these advances, most RAG pipelines still optimize components in isolation. Refinements are typically applied locally—for example, through query rewriting, evidence filtering, or post-hoc answer editing via self-reflection or self-correction. Consequently, errors introduced early in the pipeline, such as incomplete retrieval or poorly structured evidence, often propagate downstream and limit overall performance. This reveals a coordination problem in multi-agent RAG pipelines: evaluation signals produced at the final generation stage are rarely used to improve upstream components such as retrievers or graph construction modules. In this work, we study document-grounded query answering and query-focused summarization, where retrieval operates within a given document (or document set) by selecting relevant spans for each query. This setting corresponds to benchmarks such as SQuALITY and QMSum, where each query is paired with long documents. Although these documents fit within modern LLM context windows, our goal is not to compare RAG with full-context inference but to use these benchmarks as a controlled testbed for analyzing how different RAG pipelines behave and how coordination between their components can be improved. We introduce GradRAG, a framework for cross-component prompt adaptation in agentic RAG systems (Figure 1). GradRAG models the RAG pipeline as a computational graph whose nodes correspond to agents such as the retriever, graph constructor, and answer generator. Instead of refining only the final answer, the framework propagates evaluation feedback to multiple upstream agents responsible for retrieval and evidence construction. At test time, an Evaluator agent analyzes the generated answer together with its supporting evidence and produces structured feedback identifying missing information, weak reasoning links, or irrelevant context. A Prompt Optimizer converts this feedback into prompt updates for adaptive agents across refinement iterations. The process operates entirely through prompt updates and does not modify model parameters. By allowing downstream critiques to influence upstream decisions, GradRAG enables coordinated improvements beyond one-step answer rewriting. Rather than optimizing retrieval, structuring, and generation independently, the framework aligns their behavior through feedback-driven prompt refinement within a bounded test-time budget. We evaluate GradRAG on SQuALITY and QMSum using two representative retrieval paradigms. The first is a flat (chunk-based) retrieval pipeline, where the system iteratively rewrites search queries and retrieves document chunks using a hybrid dense–lexical retriever following an IRCoT-style strategy (Trivedi et al., 2023). The second is a graph-based retrieval pipeline that extracts and iteratively enriches an entity–relation graph to assemble evidence for answering the query. Within each paradigm we compare GradRAG against controlled one-step refinement baselines in which only the final answer generator is updated, isolating the effect of cross-component prompt adaptation. Across both settings, GradRAG consistently outperforms these baselines, with most gains realized within the first two refinement iterations. Section 5 shows that cross-component prompt adaptation yields a 12–15 percentage point net preference margin in LLM-guided pairwise evaluations. GradRAG therefore provides a general framework for coordinating improvements across heterogeneous components in modular RAG pipelines. This work makes the following contributions: • We introduce a framework for cross-component prompt adaptation in agentic RAG, enabling evaluator feedback to update the prompts of multiple upstream agents at test time. • We show that coordinated agent refinement via shared evaluative feedback improves alignment between retrieval, structuring, and generation. • We propose a modular graph-based abstraction that applies the same feedback-driven optimization across both flat (chunk-based) and graph-based RAG pipelines. 2 Related Work 2.1 Agentic Approaches and RAG Recent work in Retrieval-Augmented Generation (RAG) increasingly incorporates agentic components. Asai et al. (2023) introduce a reflection-based mechanism in which the model produces intermediate tokens representing retrieval decisions and evidence assessments, enabling iterative retrieval and answer refinement guided by a learned critic. This work established that RAG performance can benefit from multi-step, agentic decision-making. Subsequent approaches distribute retrieval and reasoning across specialized components. For example, Jeong et al. (2024) and Lee et al. (2025) propose routing agents that decide whether and how extensively to retrieve and which backend to query. Trivedi et al. (2023) frame retrieval as a sequential decision process that iteratively refines search queries, while Jiang et al. (2025) use Monte Carlo Tree Search guided by learned reward models. In this literature, multi-agent typically refers to pipelines composed of multiple LLM calls with distinct prompts, each specializing in a subtask such as retrieval routing, evidence assessment, or answer synthesis. Multi-agent retrieval optimization has become a common strategy. Yan et al. (2024) train a relevance judge that decides whether to continue querying or switch to generation, while Chang et al. (2025) generate partial answers from retrieved documents and use a judge to select the most informative ones. Despite their success, these approaches apply feedback locally, focusing on individual stages such as retrieval control or answer selection. Consequently, evaluation signals generated at the final answer stage rarely influence upstream components such as retrieval or evidence construction. In contrast, our work addresses this coordination problem directly: how feedback on downstream answer quality can be used to adapt multiple upstream components of a RAG pipeline in a unified manner. 2.2 LLMs as Critic Agents and Prompt Optimizers The reflection pattern can be viewed as a three-stage loop of generation, evaluation, and adaptation. Early systems (Madaan et al., 2023; Shinn et al., 2023) demonstrated inference-time refinement in which models iteratively revise outputs using self- or evaluator-generated critiques (later described as textual gradients by Yuksekgonul et al. (2024), or verbal reinforcement). These approaches treat natural-language critiques as optimization signals guiding subsequent reasoning steps. This idea was later generalized into automatic prompt optimization, where an LLM searches for prompts that improve downstream performance Zhou et al. (2023); Opsahl-Ong et al. (2024). Frameworks such as DSPy Khattab et al. (2023) and TextGrad Yuksekgonul et al. (2024) provide programming abstractions for this process: DSPy focuses on optimizing prompting structure and demonstrations, while TextGrad formalizes critique-driven prompt updates over computational graphs. Our work builds on these foundations but focuses specifically on coordinating prompt adaptation across multiple components of a RAG pipeline. GradRAG instantiates critique-driven prompt optimization within agentic RAG systems and empirically evaluates how coordinating updates across retrieval, structuring, and generation components improves retrieval-augmented reasoning over long documents. 3 Method: GradRAG GradRAG is a test-time framework for coordinating multiple agents in Retrieval-Augmented Generation (RAG) pipelines through cross-component prompt adaptation. GradRAG builds on critique-driven prompt optimization and reflection-based approaches, instantiating them in agentic RAG systems where feedback on downstream answer quality is used to adapt multiple upstream components in a unified manner. Overview. As illustrated in Figure 1, GradRAG executes a RAG pipeline in iterative refinement cycles. Each cycle consists of a forward pass that produces intermediate artifacts (e.g., retrieved text chunks or extracted graph structures) and a final answer, followed by an evaluation step. An Evaluator agent reviews the answer together with its supporting evidence and produces structured natural-language feedback. If refinement is required, a Prompt Optimizer converts this feedback into updated prompts for one or more adaptive agents. The next cycle then runs with the updated prompts. This process operates entirely through prompt updates and does not modify model parameters. The full refinement loop is summarized in Algorithm 1. Computational graph abstraction. GradRAG models the RAG workflow as a computational graph whose nodes represent agents, programs, or data artifacts, and whose edges represent computational transformations. Nodes corresponding to LLM-based agents are associated with prompts, which may be either fixed or adaptive. In our framework, adaptive agents are those whose prompts can be updated across refinement cycles based on evaluator feedback. This abstraction allows heterogeneous components — such as classical retrievers and LLM-based agents — to coexist within the same pipeline. Cross-component prompt adaptation. Unlike one-step reflection approaches that refine only the final answer generator, GradRAG allows evaluator feedback to update the prompts of multiple upstream agents. For example, feedback indicating missing causal or explanatory information may trigger changes to both the retrieval strategy (to target more relevant evidence) and the answer generation style (to emphasize causal structure). Prompt updates are applied iteratively within a bounded test-time budget, enabling coordinated improvements across components. Early stopping. Following prior reflection-based architectures (Madaan et al., 2023; Shinn et al., 2023), GradRAG employs an early-stopping mechanism. After each evaluation step, the Evaluator returns a discrete decision indicating whether the current output is satisfactory. Refinement cycles stop when the Evaluator signals satisfaction or when a maximum number of iterations is reached (Algorithm 1, lines 6–8). Algorithm 1 GradRAG: Cross-Component Prompt Adaptation 1:Query q, document(s) D, initial prompts pa\p_a\ for agents a∈a , max iterations T 2:for t=1t=1 to T do 3: Forward pass: 4: Execute RAG pipeline with current prompts pa\p_a\: 5: – generate intermediate artifacts (e.g., retrieved chunks or extracted entity–relation graphs) 6: – assemble supporting evidence (e.g., via chunk aggregation or graph community summaries) 7: – produce answer yty_t 8: Evaluation: 9: Evaluator reviews (yt,supporting evidence)(y_t,supporting evidence) and returns feedback ftf_t and decision dtd_t 10: if dt=SATISFACTORYd_t=SATISFACTORY then 11: return yty_t 12: end if 13: Prompt adaptation: 14: for each adaptive agent a∈adapta _adapt do 15: Update prompt pa←Optimize(pa,ft)p_a (p_a,f_t) 16: end for 17:end for 18:return final answer yTy_T 3.1 Vector RAG Technical Details For flat retrieval, we implement a document-grounded Vector RAG pipeline in which the input document is segmented into overlapping text chunks. A Retrieval Agent follows an IRCoT-style strategy (Trivedi et al., 2023), iteratively generating sub-queries based on the context retrieved so far. Each sub-query triggers a hybrid retrieval step that combines dense similarity search and lexical matching to select relevant chunks. Within GradRAG, the Retrieval Agent is adaptive: its prompt may be updated across refinement cycles based on evaluator feedback. For instance, critiques indicating missing explanations or temporal relationships can lead the agent to generate more targeted sub-queries in subsequent iterations. Retrieved chunks are aggregated to form the context passed to the Answer Generation Agent, as specified in the forward pass of Algorithm 1. 3.2 GraphRAG Technical Details Our graph-based RAG pipeline builds on prior graph-oriented approaches (Edge et al., 2025; Pedinotti and Santus, 2026). The document is segmented into larger text spans, from which a Graph Extraction Agent identifies entities and relations. Extracted entities are merged into a consolidated graph, and community detection is performed using the Leiden algorithm to identify groups of related entities. For each community, the system generates a natural-language summary and a set of candidate answers, each annotated with a helpfulness score indicating relevance to the query. Only answers with positive scores are retained and concatenated to form the evidence context supplied to the Answer Generation Agent. As in the vector-based setting, both the Graph Extraction Agent and the Answer Generation Agent are adaptive. Evaluator feedback highlighting missing entities, relations, or explanatory structure is used to refine the extraction prompt, guiding subsequent graph enrichment toward information that is most useful for answering the query. Over successive refinement cycles, this feedback-driven adaptation effectively induces a task-oriented ontology, shaping which types of entities and relations are prioritized during graph construction (Pedinotti and Santus, 2026; Pedinotti et al., 2026). The resulting enriched graph is then used to assemble evidence for the final answer in the next forward pass (Algorithm 1, lines 2–5). Implementation. All pipelines are implemented using the Microsoft AutoGen framework (Wu et al., 2023). The prompts are provided in Appendices A and B to support reproducibility. 4 Experimental Setup Research Questions. We study whether coordinating prompt adaptation across multiple agents improves RAG pipelines. Specifically, we ask: (1) Does cross-component prompt adaptation — where evaluator feedback updates upstream agents such as retrievers or graph extractors — improve performance compared to one-step refinement that updates only the final answer generator? (2) Are these improvements consistent across different RAG architectures, including flat (chunk-based) and graph-based retrieval pipelines? GradRAG Systems and Baselines. To answer these questions, we evaluate GradRAG under two retrieval paradigms: flat (chunk-based) retrieval and graph-based retrieval. For each paradigm, we compare a one-step refinement baseline, in which feedback updates only the answer generator, against a full GradRAG variant that propagates feedback to multiple agents. The systems are: • IRCoT (Trivedi et al., 2023) + One-Step Refinement: A vector-based RAG pipeline in which only the Answer Generation Agent is adaptive. The Retrieval Agent uses a fixed prompt, so improvements arise solely from iterative answer rewriting. • Full GradRAG Vector: The same pipeline, but evaluator feedback updates both the Answer Generation Agent and the Retrieval Agent, enabling refinement of sub-query generation and evidence selection. • GraphRAG (Edge et al., 2025) + One-Step Refinement: A graph-based RAG pipeline where only the Answer Generation Agent is adaptive. The Graph Extraction Agent uses a fixed prompt, so improvements arise only from refining the final answer. • Full GradRAG Graph: The same pipeline, but evaluator feedback updates both the Answer Generation Agent and the Graph Extraction Agent, enabling adaptive graph enrichment. This setup forms a controlled ablation: within each retrieval paradigm, the baseline and GradRAG variants share the same architecture, model, and retrieval process, differing only in whether evaluator feedback updates upstream agents. This isolates the effect of cross-component prompt adaptation. Task. We evaluate all systems on the Query-Focused Summarization (QFS) task. In QFS, the model receives a document (or document set) d and a query q, and must produce a summary that answers the query using information drawn from the document. The task lies at the intersection of abstractive summarization and question answering, requiring both global document understanding and targeted evidence selection. Datasets. We conduct experiments on two standard QFS datasets: SQuALITY (Wang et al., 2022) and QMSum (Zhong et al., 2021). These datasets represent complementary domains: narrative texts (SQuALITY) and long meeting transcripts (QMSum). Both datasets are also included in the ZeroSCROLLS (Shaham et al., 2023) and LongBench (Bai et al., 2024) benchmarks. Although these documents fit within the context windows of modern LLMs, we use these datasets as controlled benchmarks for studying the behavior of RAG pipelines and evaluating coordination between their components. Table 1 summarizes their main characteristics. Dataset Docs Questions Avg Doc Length Avg Resp Length SQuALITY 52 260 4,995 words 241 words QMSum 35 281 10,668 words 65 words Table 1: Statistics about the test sets. Experimental Protocol. GradRAG operates in a test-time refinement setting. For each test instance (query–document pair), the system executes an iterative loop consisting of a forward pass followed by evaluation and, if necessary, prompt adaptation. During refinement, an Evaluator Agent analyzes the generated answer together with its supporting evidence and produces structured feedback identifying missing information or reasoning gaps. Importantly, the Evaluator does not have access to the reference answer, preventing data leakage during optimization. If the Evaluator determines that the answer is unsatisfactory, its critique is used to update the prompts of the adaptive agents, and another refinement cycle is executed. Otherwise, the process terminates through an early-stopping decision. Each test instance is processed independently and optimized prompts are reset after each example, ensuring that no information is carried across test instances. This protocol enables a controlled comparison between one-step refinement and cross-component adaptation by isolating the effect of prompt updates within a single query. While this per-instance refinement increases computational cost, it avoids cross-example contamination. In deployment scenarios, efficiency could be improved by batching multiple instances before updating prompts, which we leave for future work. Evaluation Metrics. Traditional evaluation metrics for query-focused summarization, such as ROUGE and BERTScore, correlate poorly with human judgments because they primarily measure lexical or embedding overlap rather than higher-level properties such as coherence, factuality, and reasoning (Kryscinski et al., 2019; Liu et al., 2023; Nguyen et al., 2024). During refinement, GradRAG uses an internal Evaluator Agent that analyzes generated answers together with their supporting evidence and produces critiques guiding prompt updates. This evaluator operates without access to gold reference answers to prevent data leakage. For final system comparison, we instead adopt an LLM-as-a-Judge evaluation paradigm in which an external LLM compares system outputs using a structured rubric. Given two candidate answers for the same input, the judge selects the better response based on overall answer quality and use of supporting evidence. We report win rates, defined as the proportion of pairwise comparisons in which one system is preferred over the other (Liu et al., 2025). To improve evaluation robustness, we follow recommendations from recent work on LLM-based evaluation (Baumann et al., 2025). Specifically, we report position consistency (Shi et al., 2025), which measures how often the judge’s preference remains unchanged when the order of candidate answers is reversed. We also apply a two-sided binomial sign test to assess statistical significance. We also conduct human validation on a subset of examples. Annotators are shown the same pairwise outputs and asked to select the better response using the same rubric. Agreement between LLM and human preferences is measured using accuracy and Cohen’s κ computed on majority-vote labels. Finally, to analyze refinement dynamics (Section 5), we track several per-iteration diagnostics: average response length, lexical density (content-word ratio), ROUGE-1 between the system response and the gold reference answer, and topic entropy, defined as the Shannon entropy of topic distributions obtained by fitting LDA to the set of system responses at each iteration. Implementation Details and Hyperparameters. For flat (chunk-based) retrieval, documents are segmented into overlapping text chunks using fixed character lengths (default 400 characters with a 40-character overlap). Retrieval follows an IRCoT-style strategy (Trivedi et al., 2023), in which the Retrieval Agent iteratively generates sub-queries based on the context retrieved so far. Each sub-query triggers a retrieval step, where the Retrieval Agent selects the retrieval strategy to apply: dense similarity search via FAISS (Douze et al., 2025) or lexical matching via BM25 (Robertson et al., 1995). We perform three retrieval iterations and select the top retrieved chunks at each step, aggregating them to form the final context passed to the Answer Generation Agent. The choice of retrieval strategy is controlled by the agent’s prompt and may change across refinement iterations as a result of evaluator feedback. For graph-based retrieval, documents are segmented into larger overlapping spans (default 1000 characters with a 200-character overlap) to support more reliable entity and relation extraction. A Graph Extraction Agent identifies entities and relations from these spans, which are aggregated into a graph structure that is incrementally enriched across refinement iterations. The resulting graph is then used to assemble evidence for answer generation. Across all systems, the maximum number of refinement iterations is capped at three. Models. All generation agents (Retriever, Graph Extractor, and Answer Generator) use Gemini-2.5-Flash (Comanici et al., 2025) (June 2025 version) via the Google Generative Language API.111https://ai.google.dev/gemini-api/docs/models Repetitive subtasks—such as sub-query generation in flat retrieval, entity–relation extraction in graph retrieval, and community-level summarization and candidate-answer generation—use the more cost-efficient Gemini-2.5-Flash-Lite (July 2025 version) through the same API. For vector indexing, we use the text-embedding-3-small embedding model.222https://platform.openai.com/docs/guides/embeddings For evaluation, we use the open DeepSeek-V3.1 model (DeepSeek-AI, 2025) as an external LLM judge. The prompts used by the system and the evaluator are provided in Appendices A and B to support reproducibility. Computational Cost. We analyze the additional overhead introduced by cross-component prompt adaptation using the GraphRAG pipeline on a sample of 13 query–document pairs. A full refinement cycle requires on average 35s per query compared to 30.5s for the one-step baseline, corresponding to roughly a 10% increase. Most runtime (about 70%) is spent in the forward pipeline—graph extraction and answer generation—which is identical across systems. Token usage increases from 6,040 tokens per query in the baseline to 8,140 tokens in GradRAG, primarily due to critique processing during prompt updates. Retrieval Paradigm SQuALITY QMSum Flat (Vector RAG) Full GradRAG win rate (%) 56.5 56.0 Position consistency (%) 79.3 55.1 p-value 0.028 0.090 Graph-based RAG Full GradRAG win rate (%) 57.5 55.8 Position consistency (%) 82.5 87.9 p-value 0.005 0.020 Table 2: Pairwise evaluation results. Values report the percentage win rate of GradRAG against the corresponding one-step baseline. Metric SQuALITY QMSum Full GradRAG win rate (%) 58.5 61.5 Position consistency (%) 79.3 59.7 p-value 0.0038 0.0007 Table 3: Effect of an additional refinement iteration for flat retrieval. Method SQuALITY QMSum Flat Retrieval IRCoT + One-step refinement 1.82 1.60 Full GradRAG Vector 1.84 1.63 Graph Retrieval GraphRAG + One-step refinement 1.83 1.63 Full GradRAG Graph 1.72 1.68 Table 4: Average number of refinement iterations before early stopping. (a) Length (b) Lexical density (c) ROUGE-1 (d) Topic entropy Figure 2: Evolution of system responses across iterations. Retrieval Paradigm SQuALITY QMSum Iteration 2 Flat (Vector) GradRAG win rate (%) 54.0 54.6 Graph-based GradRAG win rate (%) 53.6 55.6 Iteration 3 Flat (Vector) GradRAG win rate (%) 56.5 56.0 Graph-based GradRAG win rate (%) 57.5 55.8 Table 5: Pairwise preference rates for GradRAG over one-step baselines when the number of refinement iterations is fixed to a given value (without early stopping). 5 Results Human Validation. Human evaluation results show substantial agreement between LLM-based judgments and human preferences. Across the annotated subset, the LLM achieves a 75.86%75.86\% agreement rate with human evaluators, a Cohen’s κ of 0.590.59, and an F1 score of 0.80.8. Human preferences exhibit moderate variability, with an average annotation entropy of 0.6420.642, indicating partial disagreement on more challenging examples, particularly those involving longer responses. Main Results. Table 2 reports pairwise comparison results between one-step baselines and full GradRAG variants. Comparisons are performed within each retrieval paradigm (flat vs. graph-based), isolating the effect of cross-component prompt adaptation from differences in retrieval architecture. Across both datasets and retrieval paradigms, full GradRAG variants consistently outperform their one-step counterparts. Win rates range between 55%55\% and 57%57\%, corresponding to a net preference margin of approximately 1212–1515 percentage points when compared with the corresponding baseline preferences (e.g., 56%56\% vs. 44%44\%). These results indicate that allowing evaluator feedback to update multiple upstream agents yields consistent improvements over refining only the final answer generator. Evaluation robustness diagnostics are reported alongside preferences (Section 4). Three of the four comparisons yield statistically significant differences at the p<0.05p<0.05 level. The remaining case—flat retrieval on QMSum—shows marginal significance, reflecting greater variability in evaluator preferences for this setting. To further investigate this case, we run an additional refinement iteration for flat retrieval on QMSum. As shown in Table 3, the additional iteration increases the performance gap and yields statistically significant results, suggesting that the earlier marginal outcome was likely due to insufficient refinement rather than instability in the evaluation procedure. Effect of Refinement Budget. Table 4 reports the average number of refinement iterations before early stopping. Across systems, refinement typically terminates after fewer than two iterations, indicating that satisfactory outputs are often reached well before the maximum iteration budget. Table 5 reports pairwise preference rates as a function of a fixed number of refinement iterations, evaluated without early stopping. For flat (chunk-based) retrieval, performance generally improves as additional refinement iterations are permitted, with win rates increasing from iteration 2 to iteration 3 on both datasets. Graph-based GradRAG shows a similar pattern, achieving preference rates above 50%50\% at both iteration limits across datasets. While gains are not strictly monotonic in all cases—most notably on QMSum, where the win rate slightly decreases from iteration 2 to iteration 3—the graph-based variant remains consistently preferred over the one-step baseline. Overall, these results suggest that both flat and graph-based pipelines benefit from additional refinement under fixed budgets, while early stopping helps balance refinement gains against potential over-specialization in structured retrieval settings. Response Evolution Across Iterations. Figure 2 illustrates how system outputs evolve across refinement iterations. Response length increases sharply after the first refinement and then stabilizes, while lexical density increases steadily across iterations. ROUGE-1 scores improve monotonically, and topic entropy decreases over time, indicating progressively more focused responses. Taken together, these trends suggest that refinement primarily improves responses by increasing informational density and topical focus rather than by simply adding more content. This pattern is consistent across retrieval paradigms and supports the interpretation that cross-component prompt adaptation produces more targeted and coherent outputs. 6 Conclusion We introduced GradRAG, a framework for coordinating multiple agents in Retrieval-Augmented Generation (RAG) pipelines through cross-component prompt adaptation. By modeling RAG systems as computational graphs and propagating evaluation feedback across agents, GradRAG enables coordinated refinement of retrieval, evidence construction, and answer generation at test time. Across both vector- and graph-based retrieval settings, GradRAG achieves a 12–15 percentage point net preference margin in pairwise comparisons. These results indicate that allowing evaluation feedback to influence upstream components—rather than refining only the final answer—consistently improves system outputs. More broadly, our findings highlight the importance of coordinating adaptation across heterogeneous components in modular RAG pipelines. 7 Limitations While GradRAG introduces a general framework for agentic optimization in RAG systems, several aspects offer opportunities for further refinement: • Computational cost. The approach is computationally intensive, as both the forward execution and refinement phases rely on LLM generation. However, this cost scales predictably with the number of adaptive agents and remains manageable for moderate configurations. Moreover, efficiency gains can be achieved through parallelization. • Evaluation vs. deployment gap. Our evaluation protocol — where each refinement cycle is triggered after a single forward execution — was designed to ensure data isolation and interpretability. While this setup is less reflective of deployment conditions, it provides a clean benchmark for studying agentic feedback dynamics. In contrast, recent approaches to automatic prompt optimization, such as DSPy (Khattab et al., 2023), derive optimized prompts from multiple examples. Future research may extend GradRAG to batch- or continual-feedback regimes to better approximate real-world usage. • Evaluator configuration. The Evaluator Agent operates using general summarization principles (e.g., coherence, fluency) without external reward signals. This design isolates the intrinsic capability of LLMs to self-evaluate while avoiding data leakage. Future implementations could integrate additional supervision signals, such as reward models or heuristic metrics, to strengthen evaluation fidelity. References A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi (2023) Self-rag: learning to retrieve, generate, and critique through self-reflection. External Links: 2310.11511, Link Cited by: §2.1. Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li (2024) LongBench: a bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3119–3137. External Links: Link, Document Cited by: §4. J. Baumann, P. Röttger, A. Urman, A. Wendsjö, F. M. Plaza-del-Arco, J. B. Gruber, and D. Hovy (2025) Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation. External Links: 2509.08825, Link Cited by: §4. C. Chang, Z. Jiang, V. Rakesh, M. Pan, C. M. Yeh, G. Wang, M. Hu, Z. Xu, Y. Zheng, M. Das, and N. Zou (2025) MAIN-RAG: multi-agent filtering retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 2607–2622. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.1. G. Comanici, E. Bieber, M. Schaekermann, et al. (2025) Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. External Links: 2507.06261, Link Cited by: §4. DeepSeek-AI (2025) DeepSeek-V3 Technical Report. External Links: 2412.19437, Link Cited by: §4. M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou (2025) The faiss library. External Links: 2401.08281, Link Cited by: §4. D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson (2025) From local to global: a graph rag approach to query-focused summarization. External Links: 2404.16130, Link Cited by: §3.2, 3rd item. S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. Park (2024) Adaptive-RAG: learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 7036–7050. External Links: Link, Document Cited by: §2.1. J. Jiang, J. Chen, J. Li, R. Ren, S. Wang, X. Zhao, Y. Song, and T. Zhang (2025) RAG-star: enhancing deliberative reasoning with retrieval augmented verification and refinement. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 7064–7074. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: §2.1. O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts (2023) DSPy: compiling declarative language model calls into self-improving pipelines. External Links: 2310.03714, Link Cited by: §2.2, 2nd item. W. Kryscinski, N. S. Keskar, B. McCann, C. Xiong, and R. Socher (2019) Neural text summarization: a critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 540–551. External Links: Link, Document Cited by: §4. M. Lee, Q. Zhu, C. Mavromatis, Z. Han, S. Adeshina, V. N. Ioannidis, H. Rangwala, and C. Faloutsos (2025) HybGRAG: hybrid retrieval-augmented generation on textual and relational knowledge bases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 879–893. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.1. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 2511–2522. External Links: Link, Document Cited by: §4. Y. Liu, H. Zhou, Z. Guo, E. Shareghi, I. Vulić, A. Korhonen, and N. Collier (2025) Aligning with human judgement: the role of pairwise preference in large language model evaluators. External Links: 2403.16950, Link Cited by: §4. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023) SELF-refine: iterative refinement with self-feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §2.2, §3. H. Nguyen, H. Chen, L. Pobbathi, and J. Ding (2024) A comparative study of quality evaluation methods for text summarization. External Links: 2407.00747, Link Cited by: §4. K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab (2024) Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 9340–9366. External Links: Link, Document Cited by: §2.2. P. Pedinotti, P. Baumann, N. Jessurun, L. Barrett, and E. Santus (2026) MetaGraph: a large-scale meta-analysis of GenAI in financial NLP (2022–2025). In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), San Diego, California, USA, p. 848–861. External Links: Link, Document Cited by: §3.2. P. Pedinotti and E. Santus (2026) StructSurvey: structured agentic retrieval for automated survey paper generation. In Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026), San Diego, California, United States, p. 162–181. External Links: Link, Document Cited by: §3.2, §3.2. S. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford (1995) Okapi at trec-3. In Overview of the Third Text REtrieval Conference (TREC-3), Overview of the Third Text REtrieval Conference (TREC–3) edition, p. 109–126. External Links: Link Cited by: §4. U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy (2023) ZeroSCROLLS: a zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 7977–7989. External Links: Link, Document Cited by: §4. L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi (2025) Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. External Links: 2406.07791, Link Cited by: §4. N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §2.2, §3. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2023) Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 10014–10037. External Links: Link, Document Cited by: §1, §2.1, §3.1, 1st item, §4. A. Wang, R. Y. Pang, A. Chen, J. Phang, and S. R. Bowman (2022) SQuALITY: building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), Abu Dhabi, United Arab Emirates, p. 1139–1156. External Links: Link, Document Cited by: §4. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang (2023) AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. External Links: 2308.08155, Link Cited by: §3.2. S. Yan, J. Gu, Y. Zhu, and Z. Ling (2024) Corrective retrieval augmented generation. External Links: 2401.15884, Link Cited by: §2.1. M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou (2024) TextGrad: automatic "differentiation" via text. External Links: 2406.07496, Link Cited by: §2.2, §2.2. M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, A. Celikyilmaz, Y. Liu, X. Qiu, and D. Radev (2021) QMSum: a new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou (Eds.), Online, p. 5905–5921. External Links: Link, Document Cited by: §4. Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba (2023) Large language models are human-level prompt engineers. External Links: 2211.01910, Link Cited by: §2.2. Appendix A Prompts for Evaluator Agents A.1 Flat Retrieval Response evaluator agent ⬇ You are an expert judge evaluating both the retrieved context and the generated answer for completeness, coherence, and overall quality. Your task: 1. Context Evaluation: - First, imagine an ideal, fully satisfying answer to the question - one that captures all key facts, causal and temporal connections, character motivations, outcomes, and thematic meaning. - Then compare this ideal answer with the retrieved context. Identify what information, links, or nuances are missing, underdeveloped, ambiguous, or irrelevant. - Note any missing elements that would prevent a reader from forming a complete understanding or that weaken the logical or emotional flow of the narrative. 2. Answer Evaluation: - Assess whether the answer uses the available context effectively. - Evaluate clarity, coherence, factual accuracy, completeness, and narrative flow. - Check if it connects events and entities logically, avoids contradictions, and reads fluently in tone and length. Decision: - SATISFACTORY: The context includes all major information and connections needed for an ideal answer, and the answer expresses them clearly and coherently. - NEEDS_REFINEMENT: Any important fact, relation, or narrative link is missing, unclear, or the answer lacks fluency, structure, or accuracy. Critique must include: 1. What an ideal answer would contain that is not fully supported by the context. 2. Which parts of the context are irrelevant or insufficient. 3. How the answer could improve in factual precision, logical or causal connections, completeness, and stylistic quality. Output format: DECISION: [SATISFACTORY or NEEDS_REFINEMENT] CRITIQUE: [detailed explanation] The question was the following: original_query The answer of the system was: generated_answer When generating the answer, the system had access to the following context: retrieved_context Answer Generation Agent critique prompt ⬇ You are evaluating the prompt used for answer generation in a RAG system. Your goal is to evaluate the prompt and propose actionable improvements based on feedback obtained from the system’s output. The feedback contains strategies to improve both the retrieved content and the style of the answer; focus only on the style. Current answer generation prompt: current_prompt Based on this information, determine if there is a problem with the answer generation prompt that needs to be fixed. Provide a detailed critique to the prompt reporting what you learned in the previous critique. The critique must be actionable: it must include concrete actions that would likely improve the output. If false, leave the critique empty. Answer generation prompt optimizer prompt ⬇ You are optimizing a prompt for answer generation in a RAG system. The prompt must be general, but it also has to be as adherent as possible to the following critique. The current prompt is: current_prompt The current critique of the answer generation process is: answer_generation_critique Based on this critique, generate a new prompt that will be used to instruct the LLM how to better generate answers from retrieved vector information. The prompt should incorporate the feedback to improve answer quality, relevance, and coherence. Provide only the optimized prompt without additional commentary. """ Retrieval agent critique prompt ⬇ You are evaluating the queries made by an iterative content retriever in a RAG system. Your goal is to provide an accurate evaluation of the queries based on the following feedback. The original question of the user was: question QUERIES MADE: queries_formatted RESPONSE EVALUATOR FEEDBACK: response_evaluator_critique The previous critique for the answer generator was: answer_generator_critique Based on the response evaluator’s feedback, provide: 1. TYPES OF QUERIES TO FOCUS ON: - What types of queries would address the issues mentioned in the feedback? - What aspects should future queries target to improve the answer? - Remember that each query must target a specific part of the text. A good query targets specific content that can be found in a single chunk. Your task is not to rate the queries, but to identify what types of new queries would best address the issues raised in the feedback. Relate your suggestions directly to the user original question and the retrieved content, specifying what missing details or narrative connections new queries should aim to uncover. Provide a clear, actionable analysis. Retrieval agent prompt optimizer ⬇ You are optimizing a prompt for a query agent in a vector RAG system. The current prompt is: current_prompt IMPORTANT CLARIFICATION: - The planner you are optimizing asks the model to generate a sub-query to answer the question, based on the context retrieved so far and the previous sub-queries. - The planner does NOT control the retrieval/embedding mechanism itself - Your instructions should focus on which aspects a sub-query must target You will receive: 1. The original question to answer 2. The sub-queries that were made 3. A critique of the retrieval plan identifying what worked and what needs to be improved. original_question previous_queries retrieval_plan_critique Based on this information, generate an optimized prompt that instructs the query agent to create a good sub-query. Provide only the optimized prompt as instructions for the query planner, without additional commentary. Generate a full optimized prompt (not partial additions) that will replace the current one used by the query agent. A.2 Graph Retrieval Response evaluator agent ⬇ You are an expert judge evaluating both the retrieved context and the generated answer for completeness, coherence, and overall quality. Your task: 1. Context Evaluation: - First, imagine an ideal, fully satisfying answer to the question - one that captures all key facts, causal and temporal connections, character motivations, outcomes, and thematic meaning. - Then compare this ideal answer with the retrieved context. Identify what information, links, or nuances are missing, underdeveloped, ambiguous, or irrelevant. - Note any missing elements that would prevent a reader from forming a complete understanding or that weaken the logical or emotional flow of the narrative. 2. Answer Evaluation: - Assess whether the answer uses the available context effectively. - Evaluate clarity, coherence, factual accuracy, completeness, and narrative flow. - Check if it connects events and entities logically, avoids contradictions, and reads fluently in tone and length. Decision: - SATISFACTORY: The context includes all major information and connections needed for an ideal answer, and the answer expresses them clearly and coherently. - NEEDS_REFINEMENT: Any important fact, relation, or narrative link is missing, unclear, or the answer lacks fluency, structure, or accuracy. Critique must include: 1. What an ideal answer would contain that is not fully supported by the context. 2. Which parts of the context are irrelevant or insufficient. 3. How the answer could improve in factual precision, logical or causal connections, completeness, and stylistic quality. Output format: DECISION: [SATISFACTORY or NEEDS_REFINEMENT] CRITIQUE: [detailed explanation] The question was the following: original_query The answer of the system was: generated_answer When generating the answer, the system had access to the following context: retrieved_context Answer generation critique prompt ⬇ You are evaluating the prompt used for answer generation in a GraphRAG system. Your goal is to evaluate the prompt and propose actionable improvements based on feedback obtained from the system’s output. The feedback contains strategies to improve both the retrieved content and the style of the answer; FOCUS ONLY ON THE STYLE. Current answer generation prompt: current_prompt Feedback from the response evaluation: response_evaluator_output Based on this information, determine if there is a problem with the answer generation prompt that needs to be fixed. First, provide your reasoning explaining why there is or isn’t a problem. Then, provide a critique focusing on the specific issue. Answer generation prompt optimizer ⬇ You are optimizing a prompt for answer generation in a GraphRAG system. The current critique of the answer generation process is: answer_generation_critique Based on this critique, generate a new prompt that will be used to instruct the LLM how to better generate answers from retrieved graph information. The prompt should incorporate the feedback to improve answer quality, relevance, and coherence. Provide only the optimized prompt without additional commentary. Graph Extraction agent critique prompt ⬇ You are evaluating the prompt given to an LLM to enrich a given graph with the information extracted from the text. The prompt tells the system how to extract entities and relationships. You will be provided with feedback explaining which information is missing in the graph. You have to think: which entity/relationship types would make this information available in the graph? Clearly specify the entities and relationship types in your answer. Include only a few entities and relationship types (not more than 6-7). Based on this, you have to identify entity and relationship types that should be included in the graph and are not specified in the current prompt. Focus only on the most crucial entity/relationship types to meet the evaluation requirement. Specify only a few entity/relationship types, the ones that are most important. For each entity/relationship, you have to include examples (each example is a phrase or a sentence). The feedback from the system is: response_evaluator_output The previous critique for the answer generator was: answer_generator_critique The current prompt is: current_prompt Please, determine which entities/relationships the current prompt is missing based on the feedback. Graph Extraction agent prompt optimizer ⬇ Your goal is to generate an instruction for an LLM that enriches a graph using information from the text. The instruction must have the following format: "Focus on:" + LIST OF ENTITY/RELATIONSHIP TYPES The instruction tells the model which entity/relationship types to expand the graph with. Include only A FEW entities and relationship types (NOT MORE THAN 6-7 IN TOTAL). You have to accompany each entity/relationship with a full explanation and some examples. Each example is a phrase or a sentence. Specify only the relationships and the entities, don’t use formatted examples since they can mislead the model output format. Your suggestions must be based on this feedback: graph_extraction_feedback Provide only the instruction without additional commentary. Appendix B LLM-as-a-Judge Evaluation Details Model • Model: deepseek-ai/DeepSeek-V3.1 with thinking mode enabled • Size: 685B parameters Prompt Template ⬇ You are evaluating two answers to a question. Your task is to determine which answer better adheres to the gold-standard reference answer. **Question:** question **Gold Standard Reference Answer:** gold_standard **Answer A (from system_a):** answer_a **Answer B (from system_b):** answer_b **Instructions:** 1. First, provide your reasoning about which answer is better and why. 2. After your reasoning, you MUST end your response with EXACTLY one of these options on the last line, with nothing else: A B Do not add any punctuation, explanation, or other text on the final line. Just the single letter or word.