Paper deep dive
Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam
Michael Majurski, Cynthia Matuszek
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/20/2026, 6:27:23 AM
Summary
This paper investigates the impact of Answer-Free Context (AFC) on query disambiguation and language model accuracy. The authors propose a method where a user's question is rewritten using retrieved background information that does not contain the direct answer. This approach significantly improves benchmark performance on datasets like Humanity's Last Exam compared to standard RAG baselines or simple context prepending, demonstrating that distinct rewriting and answering phases are crucial for reducing ambiguity.
Entities (6)
Relation Signals (5)
Query Rewriting → improves → Accuracy
confidence 98% · rewriting the question to reduce ambiguity produces benchmark improvements without changing the answer itself
Humanity's Last Exam → evaluatedon → GPT-5-mini
confidence 95% · Using gpt-oss-20b to rewrite a subset of Humanity's Last Exam using answer-free grounding context improves gpt-5-mini accuracy
GPT-OSS-20B → performs → Query Rewriting
confidence 95% · Using gpt-oss-20b to rewrite a subset of Humanity's Last Exam
Answer-Free Context → usedfor → Query Rewriting
confidence 95% · Given a user question with associated answer-free grounding context, rewriting the question to reduce ambiguity produces benchmark improvements
Query Rewriting → outperforms → Context Prepending
confidence 92% · produces benchmark improvements without changing the answer itself, even compared to prepending that context before the question.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:How carefully and unambiguously a question is phrased has a profound impact on the quality of the response, for Language Models (LMs) as well as people. While model capabilities continue to advance, the interplay between grounding context and query formulation remains under-explored. This work investigates how the quality of background grounding information in a model's context window affects accuracy. We find that combining well-grounded dynamic context construction (i.e, RAG) with query rewriting reduces question ambiguity, resulting in significant accuracy gains. Given a user question with associated answer-free grounding context, rewriting the question to reduce ambiguity produces benchmark improvements without changing the answer itself, even compared to prepending that context before the question. Using \texttt{gpt-oss-20b} to rewrite a subset of Humanity's Last Exam using answer-free grounding context improves \texttt{gpt-5-mini} accuracy from 0.14 to 0.37. We demonstrate that this accuracy improvement cannot be fully recovered just through prompting at inference time; rather, distinct rewriting and answering phases are required. Code and data are available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2603.04454v1
- Canonical: https://arxiv.org/abs/2603.04454v1
Trouble viewing inline? Open PDF directly →
Full Text
63,368 characters extracted from source content.
Expand or collapse full text
Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity’s Last Exam Michael Majurski 1,2∗ Cynthia Matuszek 2 1 National Institute of Standards and Technology 2 University of Maryland Baltimore County michael.majurski@nist.gov cmat@umbc.edu Abstract How carefully and unambiguously a question is phrased has a profound impact on the quality of the response, for Language Models (LMs) as well as people. While model capabilities continue to advance, the interplay between grounding context and query formulation re- mains under-explored. This work investigates how the quality of background grounding infor- mation in a model’s context window affects ac- curacy. We find that combining well-grounded dynamic context construction (i.e, RAG) with query rewriting reduces question ambiguity, re- sulting in significant accuracy gains. Given a user question with associated answer-free grounding context, rewriting the question to reduce ambiguity produces benchmark im- provements without changing the answer it- self, even compared to prepending that con- text before the question. Usinggpt-oss-20b to rewrite a subset of Humanity’s Last Exam using answer-free grounding context improves gpt-5-miniaccuracy from 0.14 to 0.37. We demonstrate that this accuracy improvement cannot be fully recovered just through prompt- ing at inference time; rather, distinct rewriting and answering phases are required. Code and data are available athttps://github.com/ mmajurski/lm-rewrite-uplift 1 Introduction The ongoing explosion of Language Model (LM) capability is largely a consequence of increases in training compute, model parameters, and dataset size as described by scaling laws [4]; which have demonstrated relationships between performance and parameter count, dataset size, and computa- tion. This capability growth has opened a widening chasm between what modern LMs can demonstra- bly do and how we measure their spiky intelligence. Leaderboard-style benchmarks have been a primary engine and evaluator of progress in machine learn- ing, providing an objective scalable way to measure Figure 1: When RAG systems surface relevant infor- mation, LM performance can be enhanced by rewriting the initial query using context—added information that, without providing the answer, gives relevant background knowledge and direction. and compare capabilities. However, human-curated static benchmarking is a brittle standard. LM users often implicitly assume the LM shares their men- tal model, including their background knowledge, context, and intent. This can cause them to omit critical background and related information when formulating a query, believing it to be self-evident. The model, lacking this context, operates instead on training data patterns and the explicit text of Figure 2: Rewriting questions using answer-free ground- ing context yields significant accuracy improvement (distance above line) over the original questions, as eval- uated on a subset of Humanity’s Last Exam (HLE). See Figure 4 for a complete legend. arXiv:2603.04454v1 [cs.CL] 27 Feb 2026 the prompt. When faced with an under-specified query, its response will reflect implicit (or explicit) assumptions. If these assumptions do not align with the user’s unstated background expectations, the resulting answer (while potentially factually correct under that assumed interpretation) will be perceived as incorrect or irrelevant.We focus on the specific version of this problem where implicit information can be clarified within RAG search systems. In this work, we explore a the impact on sys- tem performance of adding Answer-Free Context (AFC)—background information that is relevant to the question but which does not contain the actual answer. We demonstrate that even when RAG fails to surface a direct answer, the retrieved background information can be used to perform a contextu- ally grounded disambiguation rewrite (Figure 1) to improve an underspecified query before it is an- swered. This method can meaningfully improve accuracy on the human validated subset of Human- ity’s Last Exam (HLE) (Figure 2), producing better results than simply including theAFCin the ques- tion. An example of an original question (from Squadv2 [19]), associatedAFC, and rewritten ques- tion are shown below. Original question: What kind of lasers are crystals of zinc suflde (sic) used in? Answer-free context: Zinc chloride is often added to lumber as a fire retardant and can be used as a wood preservative. It is also used to make other chemicals. Zinc methyl (CH 3 Zn) is used in a number of organic syntheses. Zinc sulfide (ZnS) is used in luminescent pigments such as on the hands of clocks, X-ray and television screens, and luminous paints. Crystals of ZnS are used in lasers. Zinc sulfate is a chemical in dyes and pigments. Zinc pyrithione is used in antifouling paints. Rewritten question: Which portion of the electromag- netic spectrum do lasers that incorporate zinc sulfide (ZnS) crystals generally operate in? (Prompts are given in the Appendix.) This work makes the following contributions: 1.Introduces a method for leveraging Answer- Free Context (AFC) to disambiguate queries, yielding significant accuracy gains. 2.Analyzes the performance differential in RAG systems when retrieving direct answers versus purely auxiliary background information. 3. Demonstrates that this accuracy improvement necessitates a distinct rewriting phase, and cannot be replicated by simply prepending the retrieved context to a prompt. 2 Related Works While LMs store significant parametric knowledge, their responses are not grounded in reliable exter- nal information sources. Considerable background information lets these models score highly on tests of general knowledge, but they often struggle with domain-specific or proprietary data absent from public training sets. Dynamic context construction approaches like Retrieval-Augmented Generation (RAG) address this by retrieving external evidence during the generation process, which significantly improves factual accuracy [23]. The RAG pipeline introduces its own optimization challenges, span- ning document chunking, search, ranking [34], and post-retrieval processing [6]. A particularly critical component, with a long history in information retrieval, is the transfor- mation, expansion, and normalization of user queries [20,21]. In professional and enterprise workflows, system performance can depend on RAG to dynamically construct context for the LM based upon user queries [14,34]. However, cur- rent RAG evaluation focuses on retrieval rank- ing [31,22] rather than how models utilize retrieved documents—information that might be relevant without explicitly containing the answer. The initial user query is often an imperfect ex- pression of the information needed, suffering from ambiguity, missing context, or poorly aligned ter- minology with a target document corpus. To bridge this lexical gap, modern RAG systems employ query expansion and rewriting [9,36,15]. Query expansion reformulates the question to better sur- face relevant documents and improve the signal- to-noise ratio of the retrieved context [7]. LMs have proven adept at this rewriting, expansion, and contextualization task, disambiguating queries us- ing carefully designed prompts [29,7,26]. Other approaches explore post-RAG retrieval optimiza- tions for improving signal to noise in the sur- faced documents, shifting from a retrieval-generate paradigm to retrieval-note-generate in which the discovered documents are summarized into high level notes [6]. Other dynamic context and prompting strategies focus on getting the most out of the context. For ex- ample, Step-Back Prompting attempts to have LMs derive high-level concepts from first principles to guide reasoning in query answering [35]. Frame- works like AGREE have the LM cite its sources within the context window (drawn from ground- 2 Figure 3: The quality of context presented to an LM has an impact on question-answering performance. Ac- curacy is highest when RAG systems surface context containing the answer (cyan), but when the question is presented without context (orange) or the surfaced information does not contain the answer (green), per- formance (without query rewriting) suffers. The act of interpretation during question rewriting (pink) produces an accuracy improvement beyond just prepending the Question + AnswerFree Contextinformation used to rewrite before the question; despite not including the AFC when asking an LM the rewritten question. ing documents) [33]. Other approaches involve extensions to chain-of-thought, including Meta- CoT, which models how to determine what under- lying reasoning is required to arrive at a particular CoT [30]. CoT-type reasoning approaches enable the LM to employ self-directed question rewriting during the process of figuring out how best to re- spond. Our work demonstrates that prompting LMs to disambiguate questions in COT is less effective than a separate rewriting phase. 3 Methodology This work explores applying RAG query rewrite/- expansion techniques directly to queries posed to LMs, which results in an accuracy improve- ment on evaluation benchmarks. Benchmarks routinely evaluate knowledge by posing factual questions [28]. Figure 3 shows the impact of added context ongpt-oss-120b’s performance (across all datasets used in this study)(see Table 1), and demonstrates that naively addingAFCwith- out rewriting the query to be clearer and less am- biguous does not improve the results as signifi- cantly as desired. However, as shown in Figure 2, usingAFCto rewrite the question can both dis- ambiguate what is being asked and fill in rele- vant background assumptions, producing robust accuracy gains. Just prepending theAFCtext be- fore the question does not produce an equiva- lent accuracy improvement, as shown in Figure 3, whereQuestion + AnswerFree Contextunder- performsRewritten Question. Unless otherwise stated theRewrittencondition never includes the AFC. We hypothesize that the accuracy improvement is caused by the act of interpretation during rewrit- ing, via clarifying and disambiguating the ques- tion separately from attempting to answer it (Sec- tion 4.3 provides evidence for this). It is worth noting that question rewriting usingAFCdoes not re- cover the full performance improvement of adding context which directly contains the correct answer. Prior work on RAG systems has demonstrated the utility of query expansion [29] and prior work on generative benchmarking demonstrated a correla- tion between generated question length and model benchmark accuracy [16]. Applying rewriting to knowledge benchmarks produces an improvement in accuracy. A potential mechanism of action is that the rewritten prompt may shift the embedding space trajectory during generation to more closely align with a latent representation that contains the correct answer. We explore that trend and charac- terize how well the effect generalizes. 3.1 Datasets & Models To explore the impact of question rewriting on LM benchmark performance, datasets are needed that contain pairs of questions with associated context that provides the correct answer. While most RAG systems are designed to produce that type of infor- mation, validated reference data is in comparatively short supply, as benchmark creation across a va- riety of domains is historically an expensive and frequently manually intensive process. Addition- ally, in order to evaluate whether observed results stem from benchmark question memorization via training data contamination, datasets are needed that contain information that postdates model train- ing. Table 1 lists the datasets used in this study and gives the release data of the benchmark. Only Humanity’s Last Exam (HLE) [18] and generative benchmarking data was post knowledge cutoff for all models tested. All datasets (except HLE) contain human cu- rated associated context with each question. The originally published HLE dataset only contains answer rationale, which is insufficient context. However, FutureHouse released manual validations for a subset of chemistry/biology questions with grounding literature [25]. This work uses that Fu- tureHouse validated subset of HLE which contains 3 3.2 Question Rewriting Procedure Table 1: Evaluation Datasets and Publication Dates DatasetRelease Date Public Datasets Humanity’s Last Exam [18]2025 Squadv2 [19]2018 HotpotQA [32]2018 TrivaQA-web [11]2017 NaturalQuestionsShort [13]2019 PubMedQA [10]2019 BoolQ [5]2019 FermiQA [12]2021 MS-MARCO-QA [2]2016 MusiqueQA [27]2022 2WikiMultiHopQA [8]2020 Generative Benchmarks (constructed using [16]) arXiv_2502_17521v1 [3]2025 America’s AI Action Plan [1]2025 Generative Benchmarks (constructed using [24]) arXiv_2502_17521v1 [3]2025 America’s AI Action Plan [1]2025 (per question) human vetted evidence and context drawn from published sources. Evaluating the efficacy of query rewriting for LM benchmark evaluation requires two categories of models. First is the model performing question rewriting (which may not be the same as the model under evaluation via the benchmark). We separate the question rewriting and the evaluation phases to ensure that all evaluation models are given the same rewritten questions to reduce measurement noise. Second are the set of models evaluated using the rewritten questions; these span a wide range of size and capability. Table 2 outlines the models we evaluated on the rewritten question benchmarks. Table 2: Evaluation Models Knowledge Cutoff ModelKnowledge- Cutoff Public- Release gpt-5Sep 2024Aug 2025 gpt-5-miniMay 2024Aug 2025 gpt-5-nanoMay 2024Aug 2025 gpt-oss-20bJun 2024Aug 2025 gpt-oss-120bJun 2024Aug 2025 gemma-3-1b-itAug 2024Mar 2025 gemma-3-4b-itAug 2024Mar 2025 gemma-3-12b-itAug 2024Mar 2025 gemma-3-27b-itAug 2024Mar 2025 Llama-3.2-3B-InstructDec 2023Sep 2024 Llama-3.1-8B-InstructDec 2023Jul 2024 Llama-3.3-70B-InstructDec 2023Dec 2024 Llama-4-Maverick-Instruct-FP8Aug 2024Apr 2025 phi-4June 2024Dec 2024 Qwen3-1.7BApr 2025 Qwen3-4B-Instruct-2507Aug 2025 Qwen2.5-7B-InstructSep 2024 Qwen3-30B-A3B-Instruct-2507Jul 2025 Qwen3-235B-A22B-Instruct-2507Jul 2025 3.2 Question Rewriting Procedure The question rewriting and expansion was approachedasapre-processingstepbe- forebenchmarkingtheevaluationmodels against the modified questions. We evaluated three different LMs for question rewriting: gpt-oss-20b(small),gpt-oss-120b(medium), andQwen3-235B-A22B-Instruct-2507(large). For each question in each dataset the rewriting LM was provided the original question, correct answer, and grounding context during prompting (Ap- pendix B) to rewrite the question to disambiguate what was being asked. In addition to rewriting the question the rewriting model must generate the correct answer to the rewritten question, which serves as a validation check against topic drift as its later evaluated for semantic similarity to the original question. Note, the rewritten answer is never used. It only serves as a validation check. During benchmark evaluation human curated answer is always the “correct” answer. This results in three rewritten variants of every question, one for each of the rewriting models (bold models in Table 2). We filter the rewritten questions to ensure the that all final benchmark questions have the acceptable meta-properties. An LM-Judge (gpt-oss-120bprompt in Appendix E) is used to extract the following properties: reformatted question similarity to the original question, refor- matted answer similarity to the original answer, reformatted question “giveaway” score, and orig- inal question “giveaway” score. The reformatted question and answer similarity scores are used to discard any questions which are too dissimilar to the original. The giveaway scores rate how much the answer was given away (or contained within) the question. Any reformatted questions with give- away scores higher than the original are discarded. This removes rewritten questions that are easier from the LM-judge’s perspective, producing a new benchmark which is strictly more difficult, with disambiguated questions. 3.3 Evaluation methodology Evaluating the accuracy improvement provided by question rewriting and expansion requires bench- marking each model under evaluation for three con- figurations: 1.Original Question: the normal benchmarking use case, 2.Original Question with Context Prepended: the RAG retrieval use case, and 3.Rewritten Question: the improvement im- parted by rewriting. 4 Note that the “Context” could either contain the answer, or be Answer-Free. To quantify the accuracy improvement, we com- pute the following average per-model and per- dataset differences: 1.(Rewritten Question - Original Question): the direct rewrite improvement, 2. (Rewritten Question - Original Question with Context Prepended): the rewrite improvement compared to RAG retrieval This comparison between the rewritten ques- tion and the original question with the con- text prepended enables evaluation of whether the rewrite outperforms benchmarking models with a RAG database which contains only background in- formation. However, if the context contains the answer, one would expect any benchmarking with the context included to yield very high accuracy. To mitigate this we developed a version of each grounding context that is “Answer-Free”.AFCwas developed using a single LM (gpt-oss-120b) to rewrite the context (see Appendix C for the prompt). The results section covers the accuracy improve- ment using the original context (the less interest- ing use case) as well as the Answer-Free Context (AFC).AFCreplicates the use case where a RAG system surfaces background information, but not the actual answer to the user’s question. Our results demonstrate accuracy improvement in benchmark performance when question rewriting is performed using this Answer-Free Context. This improvement effect is larger than just prepending the AFC. To test one hypothesis for why this effect shows up, we leverage an embedding model (e5-mistral-7b-instruct) to understand the change in cosine distance between the original question and the context compared to the cosine dis- tance between the rewritten question and the con- text. We demonstrate that improvements in bench- mark accuracy correlate with reductions in the co- sine distance between the question and context. In other words, the rewritten question appears to place the model closer to the right ‘frame of mind’ to answer the question, as the cosine distance be- tween the question and the context is smaller after the rewrite. Assuming the answer-containing con- text points in a certain direction within the model’s embedding space, rewriting the question moves the original question towards that correct answer direction. Regardless of whether the original or rewritten question is given to the LM under test, the correct answer is always defined by the bench- mark dataset and is never modified by the rewriting process. 4 Results While the improvedgpt-5-miniperformance on HLE-subset (Figure 2) is already an informative result, in this section we investigate the impact of query disambiguation via Answer-Free Context for all models (Table 2) and datasets (Table 1) across three dimensions: (a) the validity of our Answer-Free Context construction, (b) the accu- racy improvement provided by rewriting compared to standard RAG baselines, and (c) the underlying mechanisms driving these performance gains. 4.1 Validation of Answer-Free Context To ensure that performance gains result from query disambiguation rather than information leakage, we first verify that our Answer-Free Context does not inadvertently contain the answer to the bench- mark question. Figure 4 (top left) illustrates per- formance when the correct answer is present in the context. As expected, prepending answers yields a significant improvement in accuracy. This confirms that models successfully utilize answer information when it is present. In contrast, Fig- ure 4 (top right) plots model performance when Answer-Free Contextis prepended to the original question. The absence of consistent performance improvement shows that the AFC creation process successfully removed the direct answer. Conse- quently, any accuracy improvement in subsequent experiments can be attributed to query disambigua- tion rather than rote answer extraction. 4.2 Accuracy improvement from Query Rewriting We compare the performance of the Rewritten Question (Rewrite_Q) against two baselines: the Original Question (Orig_Q) and the Original Ques- tion with AFC prepended (Orig_Q+AFC). Unless otherwise stated theRewrite_Qcondition never includes AFC, it is only the rewritten question. General improvementAs shown in Figure 2, ap- plying the rewrite strategy to the HLE-subset yields substantial gains, improvinggpt-5-miniaccuracy from 13.9% (Orig_Q) to 37.2% (Rewrite_Q). This trend generalizes across datasets. On average, ask- ing theRewrite_Qquestions provides an accuracy 5 4.2 Accuracy improvement from Query Rewriting Figure 4: Performance of various LMs with and without answer-containing context and answer-free context. The x-axis shows the original question benchmark accuracy while the y-axis shows the rewritten question benchmark accuracy; distance above the line conveys improvement. Colors represent the different models tested (table 2) and shapes represent the dataset (table 1). (top left) Answers are present in the context (baseline): Benchmark performance significantly improves with the addition of context that contains the answer. (top right) Answers not present in the context: Simply providing relevant answer free context without question rewriting does not improve performance. (bottom left) Questions rewritten using answer-free context (our approach): Improvement in accuracy caused by rewriting the question only usingAFCwhere during benchmarking only the rewritten question is presented to the LM (AFC is withheld). improvement of 13.03% compared toOrig_Qques- tions when neither have theAFCincluded with the question. The distribution of benchmark accura- cies across all models and datasets is available in Appendix A. Figure 4 (bottom left) further breaks this accu- racy improvement down by dataset, showing that most (model,dataset) combinations fall above the identity line (y = x), confirming that rewriting rarely degrades performance relative to the original query (at the benchmark level), and never degrades performance for any dataset pre-model knowledge cutoff. Additionally,somedatasetslike flashrag_fermidemonstrate significant im- provement stemming from disambiguation. Comparison to RAG BaselinesConsidering that rewritten questions typically contain more detailed information drawn from context, comparing them directly against the original is not a fair compar- ison. A stricter evaluation compares results on the rewritten question against a standard RAG approach where the model receives the original 6 4.2 Accuracy improvement from Query Rewriting question with the context prepended (Orig_Q+AFC). This isolates the value of the rewrite versus sim- ply having access to theAFCinformation. We note that the question rewriting process only uses the answer-free context, so the rewritten question and the original question with AFC are on even footing information-wise. Figure 5 displays the distribu- tion of (Rewrite_Q− Orig_Q+AFC). Note that dur- ing evaluation Rewrite_Q does not include AFC, it only presents the rewritten question to the model; whereasOrig_Q+AFCasks the original question af- ter the AFC. For the majority of questions across all models and datasets, the rewrite strategy outperforms the RAG baseline. Each violin plot (per-model) is a distribution of benchmark accuracy deltas. In this case, the model under evaluation (Orig_Q+AFC) has access to all information the rewriting model did. This demonstrates that rewriting the questions in a separate rewrite-then-answer paradigm produces an accuracy improvement that is not achieved by prepending the same information into the models’ context during evaluation. Figure 5: Per-dataset per-model difference in bench- mark accuracy between the rewritten questions and the original questions with associated answer-free context. The violin plot distribution highlights the range of ac- curacy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1346. The Reasoning Gap A notable exception is ob- served in benchmarks that focus less on fact re- call (are more reasoning intensive), specifically the HLE-subset and generative benchmarks. As detailed in Figure 6, these datasets show no con- sistent improvement when comparingRewrite_Q toOrig_Q+AFC. While rewriting improves on the isolatedOrig_Qquery, more complex reasoning tasks benefit more from having the full raw context available in the window during inference than from a condensed, rewritten query. In Figure 6, the left-most five datasets are from after the knowledge cutoff (HLE, ai_plan Figure 6: Per-dataset difference in benchmark accuracy between the rewritten questions withoutAFCand the original questions withAFC. The violin plot distribution highlights the range of accuracy deltas over all models for each dataset evaluated. Benchmark accuracy im- proved by an average of 0.1346. & arXiv_2502_17521v1). Additionally, due to a paucity of datasets released since the beginning of the year, all but HLE-subset were generatively cre- ated using the methods of either Majurski and Ma- tuszek[16]or Shashidhar et al.[24]. Those bench- marks tend to have more complex questions that are less fact-based than the extractive QA datasets which make up the majority. Unfortunately, the datasets available for this study have two conflated effects.The more reasoning-intensive benchmarks (generative and HLE-subset) are also the datasets that are post knowledge cutoff for all models (see Table 1 and Table 2. It is possible either that the lack of im- provement from rewriting stems from the questions not being subject to training data contamination, or from the fact that the questions require more rea- soning and less recall. This indicates a divergence in optimal strategies, with factual disambiguation favoring a rewrite and more complex queries favor- ing raw context inclusion. Figure 7: Per-dataset improvement in benchmark accuracy fromRewrite_Q+AFCcompared to the Orig_Q+AFCduring benchmark evaluation. Benchmark accuracy improved by an average of 0.0875. Combining both strategies limits the poten- tial downsides to rewriting shown in Figure 6 7 4.3 Alignment and Task Separation where benchmark performance might be reduced slightly from the rewrite (compared to original questions withAFC). Figure 7 demonstrates that Rewrite_Q+AFClimits the potential accuracy im- provement (average improvement of 0.0875 in- stead of 0.1346) but reduces the number of datasets which show reductions in accuracy. This may indi- cate that disambiguation and context inclusion are complementary. 4.3 Alignment and Task Separation To understand the drivers behind the accuracy im- provements, we analyze the semantic alignment between queries (questions) and their grounding contexts. Figure 8 plots the change in downstream benchmark accuracy (x-axis) against the change in query-context cosine similarity (y-axis) when changing from the original query (Orig_Q) to the rewritten query (Rewrite_Q), measured using the e5-mistral-7b-instructembedding model. We observe a strong positive relationship: Rewritten questions systematically exhibit higher cosine sim- ilarity to the context than the original questions. This effect is highly consistent, with91%of the dataset points (222 out of 243) falling into the upper-right quadrant (x > 0, y > 0). This clus- tering demonstrates that queries rewritten to have tighter semantic alignment with the context reliably produce higher benchmark accuracy. It is worth noting that every model and dataset combination drawn from before the knowledge cut- off (where training data contamination might be in effect) appear in the upper right quadrant. How- ever, for some models in the post-knowledge cutoff datasets, there is a drop in benchmark accuracy, as evidenced by those points in the upper left quad- rant. Each dataset has a distribution of accuracy improvements across the evaluated models, which is why the datasets in Figure 8 form horizontal bands. This suggests the rewriting process effec- tively aligns the query with the latent space of the relevant information from the context. Finally, we test whether this rewriting must be an explicit preprocessing step or if it can be induced via Chain-of-Thought (CoT). We implemented an “In Situ” baseline where the model is prompted to rewrite the question internally using identical prompts andAFCbefore answering. Figure 9 shows that the accuracy improvement disappears under this paradigm. This holds regardless of whether the model supports<thinking>tags. CoT rewriting was evaluated using the same prompt, lightly modi- Figure 8: Accuracy improvement on the x-axis vs. im- provement in cosine similarity between the question and context for rewritten questions on the y-axis. x- axis: delta benchmark accuracyRewrite_Q− Orig_Q; larger values indicate that the rewritten question im- proved benchmark accuracy. y-axis: increase in co- sine similarity between the question and context due to rewriting; larger values indicate better alignment be- tweenRewrite_QandAFCcompared toOrig_Qand AFC. The fact that most points are in the upper right quadrant demonstrates that improvements in embedding alignment between the question and grounding context correlate with improved benchmark accuracy. Figure 9: Per-dataset improvement in benchmark accu- racy from performing an in situ rewrite of the question using answer-free context during benchmark evaluation. This combines the rewrite-then-answer method into a single operation. The prior accuracy improvement dis- appears, highlighting the impact of task separation be- tween the rewrite and answer phases. fied to have the LM first rewrite for disambiguation before answering the rewritten question. This re- sult highlights the necessity of task separation: the ‘cognitive load’ or context window dynamics of rewriting and answering in a single pass negates the benefits of disambiguation. 4.4 Limitations Our evaluations into the impact of question rewrit- ing rely heavily on extractive QA datasets which 8 have pairs of question and context. The excep- tion is HLE-subset, where the questions are post facto grounded using internet search by domain ex- perts [25]. Thus, many of the fact-based extractive QA dataset questions are easily answerable when the LM is presented the answer-containing context. This limitation—which can be summarized as the AFCavailable with the datasets originally contained the answer—is an inevitable consequence of the availability of public datasets containing questions paired with grounding context. We use LM-as-a-judge throughout, following [17], in which LM-judge approaches were human validated. Nonetheless, future work in which one or more judge components are replaced with (possibly crowdsourced) human evaluation would strengthen these findings further. 5 Conclusion This work demonstrates a novel utilization of the information-dynamic context retrieval systems sur- face for a query. RAG utility extends beyond a binary success or failure of surfacing the direct answer; we introduce and validate query disam- biguation leveraging answer-free context informa- tion. This demonstrates that surfaced background information can enhance LM performance even when it does not contain the answer. By usingAFC to rewrite and disambiguate user queries prior to inference, we observed substantial accuracy im- provement across multiple benchmarks, most no- tably doubling the performance ofgpt-5-minion a subset of Humanity’s Last Exam (HLE). Our analysis produces three insights for the de- sign of future dynamic context grounded systems: (1) The mechanism of rewrite improvement is mea- surable and predictable, as accuracy gains correlate with increased semantic alignment (cosine similar- ity) between the rewritten query and the grounding context. (2) We identify two behaviors in response to query rewriting: Primarily factual queries bene- fit maximally from rewriting alone; more complex questions (such as HLE) achieve peak performance when the model is provided with both the disam- biguated query and the raw context. (3) We estab- lish the necessity of task separation. The inability to replicate rewriting-driven performance improve- ment during an in situ Chain-of-Thought exper- iment implies that query refinement and answer generation compete for cognitive resources or con- text attention. This holds true for models both with and without formalized “reasoning” capability. This work suggests treating dynamic context construction methodologies as not purely as evi- dence collectors, but as a collaborative partner in query formulation. Aligning the user’s query to the document corpus and clarifying the requested infor- mation allows LMs to more precisely and correctly provide the information a user actually wants. 5.1 Future work This study only used single-shot query rewriting. Accuracy improvements are likely possible by ex- tending this to a best-of-N approach that attempts to disambiguate the user’s questions in multiple direc- tions based on surfaced evidence, before asking the user which direction actually aligns with their in- tent. Another improvement would be a single multi- turn process which surfaces evidence, rewrites the query, and then performs another search of the cor- pus with the clarified query. Additional work is re- quired to characterize the additional inference time costs to this approach (how many additional tokens are being spent and RAG database calls made to perform query disambiguation) or how small an LM can be used for this process while still main- taining result fidelity. This methodology provides another test time parameter that can be modified depending on the requirements of the situation. References [1] America’s AI Action Plan. [2]Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jian- feng Gao, Xiaodong Liu, Rangan Majumder, Andrew Mc- Namara, Bhaskar Mitra, Tri Nguyen, and 1 others. 2016. Ms marco: A human generated machine reading compre- hension dataset. arXiv preprint arXiv:1611.09268. [3] Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhong- wei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and 1 others. 2025. Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation. arXiv preprint arXiv:2502.17521. [4]Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. 2025. Scaling laws for predicting downstream performance in LLMs. Transac- tions on Machine Learning Research. [5]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. [6]Yuqin Dai, Guoqing Wang, Yuan Wang, Kairan Dou, Kaichen Zhou, Zhanwei Zhang, Shuo Yang, Fei Tang, Jun Yin, Pengyu Zeng, and 1 others. 2025. Evinote-rag: En- hancing rag models via answer-supportive evidence notes. arXiv preprint arXiv:2509.00877. 9 References [7]Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jin- liu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented genera- tion for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1). [8]Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. [9]Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models.arXiv preprint arXiv:2305.03653. [10]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Co- hen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 2567–2577, Hong Kong, China. Associa- tion for Computational Linguistics. [11] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettle- moyer. 2017. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e- prints, arXiv:1705.03551. [12] Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. 2021. How much coffee was consumed during emnlp 2019? fermi prob- lems: A new reasoning challenge for ai. arXiv preprint arXiv:2110.14207. [13]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466. [14] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küt- tler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 oth- ers. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in neural information pro- cessing systems, 33:9459–9474. [15] Zhicong Li, Jiahao Wang, Zhishu Jiang, Hangyu Mao, Zhongxia Chen, Jiazhen Du, Yuanxing Zhang, Fuzheng Zhang, Di Zhang, and Yong Liu. 2024.Dmqr-rag: Diverse multi-query rewriting for rag. arXiv preprint arXiv:2411.13154. [16]Michael Majurski and Cynthia Matuszek. 2025. Ground- ing generative evaluations of language models in unsuper- vised document corpora. arXiv preprint arXiv:2505.08905. [17]Michael Majurski and Cynthia Matuszek. 2026. Ground- ing generative evaluations of language models in unsuper- vised document corpora. Transactions on Machine Learn- ing Research. [18] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mo- hamed Shaaban, John Ling, Sean Shi, and 1 others. 2025. Humanity’s last exam. arXiv preprint arXiv:2501.14249. [19]Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don‘t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. As- sociation for Computational Linguistics. [20]Pushpendre Rastogi, Arpit Gupta, and Lambert Mathias. 2019. Contextual query rewriting (cqr): Natural language as interface for dialog state tracking. [21]Andreia Rodrıguez Rivas, Eva Lorenzo Iglesias, and L Borrajo. 2014. Study of query expansion techniques and their application in the biomedical information retrieval. The Scientific World Journal, 2014(1):132158. [22]Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. Ragchecker: A fine- grained framework for diagnosing retrieval-augmented gen- eration. In Advances in Neural Information Processing Sys- tems, volume 37, pages 21999–22027. Curran Associates, Inc. [23]Chaitanya Sharma. 2025.Retrieval-augmented gen- eration: A comprehensive survey of architectures, en- hancements, and robustness frontiers.arXiv preprint arXiv:2506.00054. [24]Sumuk Shashidhar, Clémentine Fourrier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, and Dilek Hakkani-Tür. 2025. Yourbench: Easy custom evaluation sets for everyone. arXiv preprint arXiv:2504.01833. [25] Michael Skarlinski, Jon Laurent, Albert Bou, and Andrew White. 2025. Humanity’s last exam (hle) bio/chem gold. [26]Mengzhou Sun, Sendong Zhao, Jianyu Chen, and Bin Qin. 2025.Picos-rag: Pico-supported query rewrit- ing for retrieval-augmented generation in evidence-based medicine. arXiv preprint arXiv:2510.23998. [27] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop ques- tions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539– 554. [28] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu- pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Informa- tion Processing Systems, 37:95266–95290. [29]Raymond Wilson, Chase Carter, and Cole Graham. 2025. Contextualizing search queries in-context learn- ing for conversational rewriting with llms. arXiv preprint arXiv:2502.15009. [30] Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Al- balak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, and 1 others. 2025. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-thought. arXiv preprint arXiv:2501.04682. 10 [31]Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, and 1 others. 2024. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470–10490. [32] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). [33]Xi Ye, Ruoxi Sun, Sercan Ö Arik, and Tomas Pfister. 2023. Effective large language model adaptation for im- proved grounding and citation generation. arXiv preprint arXiv:2311.09533. [34]Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval- augmented generation in llms. In Advances in Neural Information Processing Systems, volume 37, pages 121156– 121184. Curran Associates, Inc. [35] Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H Chi, Quoc V Le, and Denny Zhou. 2023. Take a step back: Evoking reasoning via abstraction in large language models.arXiv preprint arXiv:2310.06117. [36]Yingxue Zhou, Jie Hao, Mukund Rungta, Yang Liu, Eu- nah Cho, Xing Fan, Yanbin Lu, Vishal Vasudevan, Kellen Gillespie, and Zeynab Raeesy. 2023. Unified contextual query rewriting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 608–615. Appendices A Impact of Rewrite on Benchmarks Figure 10 presents the distribution of accuracy im- provement measured as (Rewrite_Q−Orig_Q) per- evaluation model across all datasets. This demon- strates question rewriting broadly improves bench- mark accuracy. Figure 10: Per-dataset per-model difference in bench- mark accuracy between the rewritten question and the original question. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1303. Figure 11 presents the distribution of accuracy improvement measured as (Rewrite_Q− Orig_Q) per-dataset instead of per-model like Figure 10. Figure 11: Per-dataset per-model difference in bench- mark accuracy between the rewritten question and the original question. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1303. Figure 12 presents the distribution of accu- racy improvement measured as (Rewrite_Q− Orig_Q+AFC) per-evaluation model across all datasets. Figure 12: Per-model difference in benchmark accuracy between the rewritten questions and the original ques- tions with associated answer-free context. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1346. Figure 13 presents the same information as Fig- ure 12, but with each violin distribution per-dataset instead of per-model. 11 Figure 13: Per-dataset per-model difference in bench- mark accuracy between the rewritten questions and the original questions with associated answer-free context. The violin plot distribution highlights the range of ac- curacy deltas over all models for each dataset evaluated. Benchmark accuracy improved by an average of 0.1346. Figure 14 demonstrates the benchmark accu- racy distribution per-dataset ofRewrite_Q+AFC - Orig_Q+AFC. This combination limits the potential accuracy improvement but reduces the number of datasets which show no average improvement in accuracy. This may indicate that disambiguation and context inclusion are complementary. Figure 14: Per-dataset improvement in benchmark accu- racy fromRewrite_QwithAFCcompared to theOrig_Q withAFCduring benchmark evaluation. Benchmark ac- curacy improved by an average of 0.0875. B Question Rewriting Prompt 1 QUESTION_REFORMAT_PROMPT = """ 2 ## Your Role 3 4 You are an expert educational content creator specializing in editing and improving evaluation questions to determine the competency of domain experts based on the provided textual information. 5 6 ## Input Structure 7 8 Your input consists of: 9 10 <question > 11 [A question to be answered.] 12 </question > 13 14 <answer > 15 [The correct answer to the question.] 16 </answer > 17 18 <context > 19 [The text segment containing information relevant to the question.] 20 </context > 21 22 ## Primary Objective 23 24 Your goal is to reformat, rephrase, and rewrite the question according to the provided instructions. The rewritten question should be semantically equivalent to the original question, rewritten for clarity while preserving the same correct answer. This should only be accomplished by filling in background information and explicitly stating assumptions. You are creating a test/quiz question, so DO NOT include the answer information in the question, as that would be a giveaway which skews the results. NEVER include the answer or information which would give away the answer in the rewritten question. 25 26 ## Analysis Phase 27 28 Conduct careful analysis within`< document_analysis >` tags, following these steps: 29 30 1. ** Thoughtful Content Examination ** 31 - Carefully analyze the given context, question, and answer; identifying central ideas, nuanced themes, and significant relationships within it. 32 33 2. ** Concept Exploration ** 34 - Consider implicit assumptions, subtle details, underlying theories, and potential applications of the provided information. 35 36 3. ** Intentional Question Planning ** 37 - Plan how the question can invite deeper understanding, meaningful reflection, or critical engagement, ensuring the question is purposeful. 38 39 4. ** Detailed Assumption Expansion ** 40 - Consider what knowledge the question is asking about, and what information and assumptions have been made when formatting the question. Your goal is to provide all the background information and explicitly state assumptions to enhance the clarity of the question. 41 42 5. ** Giving Away the Answer ** 43 - Plan how to avoid giving away the answer in the rewritten question. 44 - NEVER include the answer or information which would give away the answer in the rewritten question. 45 46 ### Documentation in Analysis: 47 48 - Clearly document the rationale in the`< document_analysis >` tags, explaining your reasons for exclusion or inclusion decisions. 49 - Clearly document what elements of the question need to be disambiguated. What steps need to be taken and what information needs to be include most clearly and concisely disambiguate the question. 50 - Clearly document what information needs to be avoided in the rewritten question to prevent giving away the answer. For example if the question asks about what year a person was born, the question should not include birthday in the biographical details. 51 52 53 ## Question Rewriting Guidelines 54 12 55 ### Encouraged Question Characteristics: 56 57 - ** Thoughtful Engagement **: Prioritize creating questions that inspire deeper thought and nuanced consideration. 58 - **Deep Understanding and Insight **: Ensure that the question and answers require a deep understanding of the content by a professional domain expert. 59 - **Self -contained Clarity **: Questions and answers should contain sufficient context, clearly understandable independently of external references. 60 - ** Brevity **: The rewritten question should be as short as is reasonable while still being clear, understandable, self - contained, and unambiguous. 61 62 ### Permitted Question Types: 63 64 - Analytical 65 - Application -based 66 - Clarification 67 - Counterfactual 68 - Understanding 69 - Conceptual 70 - Factual 71 - Open -ended 72 - False -premise 73 - Edge -case 74 - Inference 75 - Implication 76 - Prediction 77 78 (You do not need to use every question type, only those naturally fitting the content and instructions .) 79 80 ## Output Structure 81 82 Present your final output strictly adhering the`<output_format >` tags. 83 <output_format > 84 Question: [ Question Text ] 85 Explanation: [Brief explanation of why the answer is correct] 86 Correct Answer: [Short answer] 87 </output_format > 88 89 ## Output 90 91 Begin by thoughtfully analyzing the provided context within`<document_analysis >` tags. Then present the resulting formatted question answer pair clearly within`<output_format >` tags. 92 93 ## Important Notes 94 95 - NEVER modify the core element the question is asking about. The knowledge being evaluated shall not change. 96 - Question disambiguation and modification must be grounded in the`<context >`. 97 - Maintain clear, direct, and accurate citations/explanations drawn verbatim from the provided context. 98 - Each "thought_process" should reflect careful consideration and reasoning behind your response. 99 - When rewriting questions, NEVER include phrases like'as per the text,′ according to the document,' or any similar explicit references. Questions should inherently integrate content naturally and stand independently without explicit references to the source material. Make sure that the question is answerable by a domain expert ** without the context paragraph **. 100 - Include all relevant context information in the question. Make the question as long and detailed as required so that the test taker can fully understand what is being asked. 101 - NEVER include the answer in the rewritten question. 102 - Ensure rigorous adherence to output formatting and generate a single`< output_format >` tag block. 103 - Verify that the correct answer is in fact correct and the best version of that answer. 104 - Verify that the question and answer are semantically equivalent to the original question and answer. 105 106 107 108 <question >question </question > 109 <answer >answer </answer > 110 <context >context </context > 111 """ C Answer-Free Context Creation 1 ANSWER_FREE_CONTEXT_PROMPT = """ 2 ## Your Role 3 4 You are an expert educational content creator specializing in editing and improving evaluation questions to determine the competency of domain experts based on the provided textual information. 5 6 ## Input Structure 7 8 Your input consists of: 9 10 <question > 11 [A question to be answered.] 12 </question > 13 14 <answer > 15 [The correct answer to the question.] 16 </answer > 17 18 <context > 19 [The text segment containing information relevant to the question.] 20 </context > 21 22 ## Primary Objective 23 24 Your goal is to reformat, rephrase, and rewrite the context information according to the provided instructions. The rewritten context should be minimally modified, and semantically equivalent to the original context. The rewrite should only remove the information which gives away the answer to the question. You are creating background material for a test/quiz question, so you need to COMPLETLELY remove the information which gives away the answer to the question from the context. NEVER include the answer or information which would give away the answer in the rewritten context. 25 26 ## Analysis Phase 27 28 Conduct careful analysis within`< document_analysis >` tags, following these steps: 29 30 1. ** Thoughtful Content Examination ** 31 - Carefully analyze the given context, question, and answer; identifying central ideas, nuanced themes, and significant relationships within it. 32 33 2. ** Concept Exploration ** 34 - Consider implicit assumptions, subtle details, underlying theories, and potential applications of the provided information. 35 36 3. ** Intentional Context Planning ** 37 - Plan how the context information can support disambiguation of the question, 13 while not giving away the answer; ensuring the question is purposeful. 38 39 4. ** Detailed Assumption Expansion ** 40 - Consider what knowledge the question is asking about, and what information and assumptions have been made when formatting the question. Your goal is to edit the context to remove the information which would give the questions answer away to the test taker. 41 42 5. ** Giving Away the Answer ** 43 - Plan how to avoid giving away the answer in the rewritten context. 44 - Figure out what minimal set of information needs to be removed to avoid giving away the answer. 45 - NEVER include the answer or information which would give away the answer in the rewritten context. 46 47 ### Documentation in Analysis: 48 49 - Clearly document the rationale in the`< document_analysis >` tags, explaining your reasons for exclusion or inclusion decisions. 50 - Clearly document what elements of the context need to be modified. What steps need to be taken and what information needs to be include most clearly and concisely (with minimal modification) remove the answer information from the context. 51 - Clearly document what information needs to be avoided in the rewritten context to prevent giving away the answer. For example if the question asks about what year a person was born, the context should not include birthday in the biographical details. 52 53 54 ## Context Rewriting Guidelines 55 56 ## Output Structure 57 58 Present your final output strictly adhering the`<output_format >` tags. 59 <output_format > 60 [ Rewritten Context ] 61 </output_format > 62 63 ## Output 64 65 Begin by thoughtfully analyzing the provided question, answer and context within`< document_analysis >` tags. Then present the resulting edited context within`< output_format >` tags. 66 67 ## Important Notes 68 69 - NEVER modify what the question is asking about. NEVER modify the answer. The knowledge being evaluated SHALL NOT change. 70 - Each "thought_process" should reflect careful consideration and reasoning behind your response. 71 - NEVER include the answer in the rewritten context. 72 - ONLY minimally modify the context as required to remove the answer information. The modified context should be as similar to the original as possible, with the answer information removed. 73 - ONLY remove answer information, do not add new information, and do not remove extraneous information. 74 - Ensure rigorous adherence to output formatting and generate a single`< output_format >` tag block. 75 76 77 78 <question >question </question > 79 <answer >answer </answer > 80 <context >context </context > 81 """ D Answer Explanation Validation Prompt 1 EXPLANATION_VALIDATION_PROMPT = """ 2 ## Your Role 3 4 You are an expert evaluator of educational content. Your goal is to produce meaningful, insightful knowledge about domain expert evaluations designed to determine competence and knowledge. 5 6 ## Input Structure 7 8 Your input consists of: 9 10 <question > 11 [A question to be answered.] 12 </question > 13 14 <answer > 15 [The student's answer to the question.] 16 </answer > 17 18 <explanation > 19 [An explanation for why the answer is correct.] 20 </explanation > 21 22 <context > 23 [The text segment containing information relevant to the question.] 24 </context > 25 26 ## Primary Objective 27 28 You will be evaluating and judging the whether the student's answer and their explanation of why their answer is correct makes sense and is logically valid. 29 30 Your goal is to judge whether the information presented in`<answer >` is in fact the correct answer to the`< question >` given the information in the `<context >` and whether the`< explanation >` for why the answer is correct is valid. The information in`< context >` and`<question >` can be assumed true, only the context of`< answer >` needs to be validated for correctness. 31 32 ### Metrics 33 34 1. ** Answer Correctness:** Rate from 1 to 10 how correct the provided student answer is given the information in the`< question >` and`<context >`. A rating of 1 indicates the answer is incorrect. A rating of 10 indicates the answer is correct and complete. 35 36 2. ** Explanation Validity:** Rate from 1 to 10 how valid the students`<explanation >` of their answer is. The`<explanation >` should explain their thinking and the information used to determine the correct answer given the context and question. Low ratings indicate the explanation is not valid, correct, or that there is some flaw in the thinking or logic of the student. High ratings indicate the explanation is valid, correct, and explains why the answer is what it is. 37 38 ## Analysis Phase 39 40 Conduct careful analysis within`< 14 document_analysis >` tags, following these steps: 41 42 1. ** Thoughtful Content Examination ** 43 - Carefully analyze the given context, identifying central ideas, nuanced themes, and significant relationships within it. 44 45 2. ** Concept Exploration ** 46 - Consider implicit assumptions, subtle details, underlying theories, and potential applications of the provided information. 47 48 ## Output Structure 49 50 Present your final output strictly adhering the`<output_format >` tags. 51 <output_format > 52 Answer Correctness: [ Correctness Rating. Respond with a number in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ] 53 Explanation Validity: [ Validity Rating. Respond with a number in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ] 54 </output_format > 55 56 ## Output 57 58 Begin by thoughtfully analyzing the provided context within`<document_analysis >` tags. Then present the resulting formatted question answer pair clearly within`<output_format >` tags. 59 60 ## Important Notes 61 62 - Each "thought_process" should reflect careful consideration and reasoning behind your ratings. 63 - Ensure rigorous adherence to output formatting. 64 65 66 <question >question </question > 67 <answer >answer </answer > 68 <explanation >explanation </explanation > 69 <context >context </context > 70 """ E Question Property Validation Prompt 1 PROPERTIES_PROMPT = """ 2 ## Your Role 3 4 You are an expert evaluator of educational content. Your goal is to produce meaningful, insightful knowledge about domain expert evaluations designed to determine competence and knowledge. 5 6 ## Input Structure 7 8 Your input consists of: 9 10 <question > 11 [A question to be answered.] 12 </question > 13 14 <answer > 15 [The correct answer to the question.] 16 </answer > 17 18 <context > 19 [The text segment containing information relevant to the question.] 20 </context > 21 22 ## Primary Objective 23 24 You will be evaluating and judging the quality of test and evaluation questions across a variety of metrics. Your goal is to judge and evaluate the quality of various test and evaluation questions across a variety of metrics. The`< question >` and`<answer >` pair is grounded and drawn from the`<context >`. 25 26 ### Metrics 27 28 1. ** Clarity:** Rate from 1 to 10 the clarity and comprehensibility (how understandable it is) of the provided`< question >`. A rating of 1 is unclear and cannot be understood or cannot be understood without the`<context >`. A rating of 10 is used for questions that are self contained, understandable, and coherent (even if the topic is complex and difficult). Questions that are missing information required to understand what is being asked rate a 1. "As of the 2015 NFL season, how many Super Bowl titles had the Denver Broncos won?" is a 10. "What event in 1861 contributed to the temporary strength of republicanism in Britain during Queen Victoria's reign ?" is a 10. "In which year was the country not a member of FIFA, as indicated in the table ?" is a 1 . "As of the census of 2000, how many families were residing in the city?" is a 1. 29 30 2. ** Difficulty:** Rate form 1 to 10 the difficulty of the`<question >`. A rating of 10 is reserved for questions which require a deep understanding of the question and what is being asked by a professional domain expert. 31 32 3. ** Groundedness:** Rate form 1 to 10 how grounded the provided`<question >` is in the`<context >`. A rating of 10 requires the question and answer information can found within the`< context >`. A rating of 1 indicates the question and answer information is not present in the`<context >`. This metric is only concerned with information found in the`<context >`, not outside information. The more outside information (not contained in the`< context >`) that is required to answer the question, the lower the rating. 33 34 4. ** Answer Give Away:** Rate from 1 to 10 how much the provided`<answer >` is given away by information in the`< question >`. A rating of 1 indicates that the information requried to answer the question is not present in the question itself. A rating of 10 indicates that the information required to answer the question is present in the question. 35 36 ## Analysis Phase 37 38 Conduct careful analysis within`< document_analysis >` tags, following these steps: 39 40 1. ** Thoughtful Content Examination ** 41 - Carefully analyze the given context, identifying central ideas, nuanced themes, and significant relationships within it. 42 43 2. ** Concept Exploration ** 44 - Consider implicit assumptions, subtle details, underlying theories, and potential applications of the provided information. 45 46 ## Output Structure 47 48 Present your final output strictly adhering the`<output_format >` tags. 49 <output_format > 50 Clarity: [ Clarity Rating (one of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ) ] 15 51 Difficulty: [ Difficulty Rating (one of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ) ] 52 Groundedness: [ Groundedness Rating (one of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ) ] 53 Answer Giveaway: [ Answer Giveaway Rating ( one of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ) ] 54 </output_format > 55 56 ## Output 57 58 Begin by thoughtfully analyzing the provided context within`<document_analysis >` tags. Then present the resulting formatted question answer pair clearly within`<output_format >` tags. 59 60 ## Important Notes 61 62 - Each "thought_process" should reflect careful consideration and reasoning behind your ratings. 63 - Ensure rigorous adherence to output formatting. 64 65 66 <question >question </question > 67 <answer >answer </answer > 68 <context >context </context > 69 """ 16