Paper deep dive
Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving
Xin Cheng, Junming Yang, Ning Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require multi-step decisions to reach a solution. However, existing puzzle benchmarks typically focus on narrow environments with rigid mechanics and tightly constrained solution targets. To bridge this gap, we introduce the Active Probing Benchmark (APB), which generalizes puzzle-solving by casting concept identification as an interactive reasoning task that integrates specialized knowledge assessment into the puzzle setting. Specifically, APB evaluates whether models can acquire information via multi-turn active probing and inference to pinpoint 960 target concepts of specialized difficulty across eight diverse academic domains. Our evaluation reveals a counter-intuitive phenomenon: most open-source reasoning models significantly underperform their standard instruction counterparts. This finding suggests that in interactive settings, these models struggle to leverage procedural reasoning to reliably access and apply specialized domain knowledge. Beyond these empirical insights, APB provides a scalable pipeline for generating benchmarks that probe knowledge reasoning under interaction across diverse settings.
Tags
Links
- Source: https://arxiv.org/abs/2610.16133v1
- Canonical: https://arxiv.org/abs/2610.16133v1
Trouble viewing inline? Open PDF directly ā
Full Text
77,319 characters extracted from source content.
Expand or collapse full text
Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Xin Cheng 1 Junming Yang 1 Ning Xu 1 Abstract Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose ver- ifiable goals under partial information and require multi-step decisions to reach a solution. However, existing puzzle benchmarks typically focus on narrow environments with rigid mechanics and tightly constrained solution targets. To bridge this gap, we introduce the Active Probing Bench- mark (APB), which generalizes puzzle-solving by casting concept identification as an interactive rea- soning task that integrates specialized knowledge assessment into the puzzle setting. Specifically, APB evaluates whether models can acquire infor- mation via multi-turn active probing and inference to pinpoint 960 target concepts of specialized dif- ficulty across eight diverse academic domains. Our evaluation reveals a counter-intuitive phe- nomenon: most open-source reasoning models significantly underperform their standard instruc- tion counterparts. This finding suggests that in interactive settings, these models struggle to lever- age procedural reasoning to reliably access and apply specialized domain knowledge. Beyond these empirical insights, APB provides a scalable pipeline for generating benchmarks that probe knowledge reasoning under interaction across di- verse settings. 1. Introduction The rapid progress of Large Language Models (LLMs) has heightened the demand for evaluation paradigms that re- liably reflect underlying model capabilities (Singh et al., 2024). Although existing benchmarks have enabled system- atic comparison, their validity is increasingly constrained by practical challenges in benchmark design and upkeep. Prior work has noted that overlap between evaluation data and 1 School of Computer Science and Engineering, Southeast University, Nanjing, China. Correspondence to: Ning Xu<xn- ing@seu.edu.cn>. Preprint. October 11, 2026. large-scale pre-training corpora may confound performance estimates in some settings (Li et al., 2023; Jacovi et al., 2023; Roberts et al., 2023), motivating interest in evaluation protocols that emphasize reasoning over memorization (Xu et al., 2024; Zhou et al., 2025). Dynamic benchmarks have emerged as one response (Kiela et al., 2021; Shirali et al., 2022), including continuously updated datasets such as Lat- estEval (Li et al., 2024) and LiveBench (White et al., 2024). However, such approaches depend on sustained data cura- tion and may be difficult to scale. Other paradigms, such as crowdsourced pairwise comparison in Chatbot Arena (Chi- ang et al., 2024), introduce variability driven by subjective or stylistic factors. Meanwhile, agent-based benchmarks (Liu et al., 2023; Zhou et al., 2023; Chan et al., 2024) primarily target domains with explicit verification signals, leaving the evaluation of general knowledge reasoning relatively underexplored. Puzzle tasks have become a widely used evaluation paradigm for LLMs. With clearly defined objectives and verifiable endpoints, they enable assessment not only of outcomes but also of trajectory-level signals of exploratory reasoning (Giadikiaroglou et al., 2024). Prior work has of- ten focused on specific classes of puzzles (Chollet, 2019; Lin et al., 2021; Zhang et al., 2025), including riddles, Su- doku, and related logic or combinatorial games. Recent benchmarks have advanced this approach. For instance, SPLAT (Chen et al., 2024) frames tasks as clue-seeking within pre-scripted narratives, and MTP (Badola et al., 2025) assesses models through carefully designed interactive en- vironments. However, such environments typically define narrow action spaces that encourage models to depend on fixed strategies rather than engaging in deep knowledge rea- soning. Moreover, the highly constrained goals of these scenarios restrict the evaluation method from being gener- alized to broader contexts. Our work extends the use of puzzle-solving by integrating multi-turn active probing with domain knowledge, aiming instead to bridge this gap and evaluate whether models can demonstrate a general under- standing of any established concept. In this work, we introduce the Active Probing Benchmark (APB), which expands the scope of puzzle-solving eval- uation (Figure 1). Specifically, APB assesses knowledge 1 arXiv:2610.16133v1 [cs.CL] 12 Oct 2026 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Data Construction Generalized Puzzle Source (Wikipedia / Docs) Domain Puzzle Set Puzzle Target (P ķ”ķķķķķ” ) Reference Document (ķ¼ ķķķ ) The Agents QueryingAgent (Model Under Test) Knowledge Oracle Probing P ķ”ķķķķķ” with Query Q ķ” (Question ķ ķ” , Options ķ ķ” ) in each turn from ķ» ķ”ā1 Feedback ķ“ ķ” of ķ ķ” from Verification Function ķ(ķ ķ” ,ķ¼ ķķķ ) Multi-turn Active Probing Process Turn 1 Turn 2 Turn t ... ... Outcome Success: P ķ”ķķķķķ” āQ ķ” Failure: t> ķķ“ķ_ķ ķķķķ·ķ ķÆ ķ ā ķ Question ķ 1 : ...? Options ķ 1 :A)B)C)D) A 1 : (A B C D ) Question ķ ķ” : ...? Options ķ ķ” :A)B)C)D) A ķ” : (A B C D ) Question ķ 2 : ...? Options ķ 2 :A)B)C)D) A 2 : (A B C D ) 0 102030 AverageSuccessRate(%) 6.6 7.2 13.0 12.5 12.6 13.4 16.4 19.6 o3 Deepseek V3 Gemini 2.5 Pro No-Thinking Qwen3-235B-A22B-Instruct Gemini 2.5 Pro Qwen3-235B-A22B-Thinking Deepseek R1 GPT-4o GPT-OSS-120B 25.3 Figure 1. Overview of the Active Probing Benchmark (APB) and representative performance results. (Left) Data construction: each puzzle specifies a target conceptP target drawn from a generalized knowledge source (e.g., Wikipedia or domain documents) along with a reference documentI ref . (Middle) Multi turn active probing: the model under test iteratively asks multiple-choice queriesQ t , receives feedbackA t from a knowledge oracle, and updates its hypotheses to reduce the candidate space until the target is identified or a turn budget is exhausted. (Right) Performance comparison: average success rates show that several open source āThinkingā models underperform their āNo-Thinkingā counterparts. Full results are reported in Table 2. reasoning under interaction: the ability to identify a target concept through active multi-turn inquiry. This setup par- allels practical decision making settings such as medical diagnosis, where a physician narrows possibilities through sequential questioning. Similarly, APB requires models to iteratively ask informative questions (Figure 2), use the re- turned signals to refine hypotheses, and efficiently reduce the candidate space until the target concept is identified. Since Wikipedia is a major source in the pre-training cor- pora of foundation models, APB constructs 960 puzzles centered on concepts of specialized difficulty spanning eight academic domains, including STEM, the humanities, and health sciences. By emphasizing information seeking inter- action over fixed game mechanics, APB aims to evaluate how effectively LLMs can access and apply specialized knowledge in a controlled setting with explicit objectives. Evaluation of a wide range of leading LLMs on APB re- veals the benchmarkās significant challenge where even the top-performing modelGPT-5achieved a success rate of only 33.4% (Table 2).The results uncover a counter-intuitive phenomenon: most open-source reasoning- enhanced models significantly underperform their no- thinking counterparts, a trend exemplified by models in- cluding includingGPT-OSS-120B,DeepSeek R1, and Qwen3-235B-A22B-Thinking(Table 2). Detailed analysis of response patterns also highlights distinct fail- ure modes in these models. For instance, Geminiās rela- tive weakness stems from a tendency to frequently offer a conclusive guess which points to potential inherent flaws. Additionally, open source reasoning models tend to commit to specific candidate targets early and then probe in a confir- matory manner, leading to inefficient probing when the true target lies outside the initial hypothesis set. These analysis suggests that in interactive settings, the models struggle to leverage procedural reasoning to reliably access and apply specialized domain knowledge. Beyond these empirical findings, the construction of the APB introduces a scalable method to rapidly generate diag- nostic benchmarks in specific specialized domain, enabling a robust evaluation of knowledge reasoning. We further showcase its application by building an IC Back-End Puzzle Benchmark, featuring 480 puzzles from four EDA com- mand tools in the IC back-end design (Appendix A). The core contribution can be summarized as follows: ā¢ We generalize the paradigm of puzzle-solving assess- ment by expanding the action and target spaces, shifting the focus from game mechanics to general knowledge reasoning across broader domains. ā¢We introduce the Active Probing Benchmark (APB) and a scalable pipeline for generating evaluations of spe- cialized difficulty, instantiated by 960 puzzles derived from expert-level concepts. ā¢ We identify a critical proficiency gap in leading open- source reasoning models, observing that under the in- teractive evaluation paradigm of APB, these models struggle to leverage procedural reasoning to reliably access and apply specialized domain knowledge. 2 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Puzzle: Reticular Dysgenesis (is a rare, inherited autosomal recessive disease...) You are a human disease expert. I will have a preset human disease name/term/concept in mind... Please begin with your first question. I'l help you identify the disease. Let me start with a broad categorization question. Q1: Which body system is primarily affected by this disease? A) Cardiovascular system ā B) Respiratory system ā C) Nervous system ā D) Endocrine/Metabolic system ā So we're dealing with an endocrine/metabolic disorder. Let me narrow this down further. Q2: Which one best describes the primary metabolic abnormality in this disease? A) Abnormal glucose/sugar metabolism ā B) Abnormal thyroid hormone levels ā C) Abnormal lipid/fat metabolism ā D) Abnormal mineral metabolism ā Ā· Ā· Ā· I need to reconsider my approach... Let me think about other metabolic causes of SCID. Q8: Is this disease: A) Adenosine kinase deficiency ā B) Nucleoside phosphorylase deficiency... ā C) A disorder of folate metabolism... ā D) Reticular dysgenesis (a form of...) ā Round 0 Round 1 Round 2 Round 3~7 Round 8 Figure 2. Example episode from the Human Disease domain in APB, where the agent successfully identifies the target concept āReticular dysgenesisā. Across turns, the agent acquires diagnostic evidence through queries and feedback, progressively narrowing the hypothesis set until the target is determined. The resulting interaction trace provides process-level evidence of how the model accesses and applies specialized knowledge. 2. Related Work 2.1. Benchmark Evaluation The evaluation of large language models has been largely driven by a suite of comprehensive benchmarks designed to assess their knowledge and reasoning abilities across a multitude of domains (Cobbe et al., 2021; Liu et al., 2024b; Rein et al., 2024; Clark et al., 2018; Hendrycks et al., 2021). Prominent examples such as MMLU (Hendrycks et al., 2020), BIG-Bench (Srivastava et al., 2023) and Hel- laSwag (Zellers et al., 2019) have become standard tools for measuring model performance (Contributors, 2023; Duan et al., 2024). A common characteristic of these benchmarks is their reliance on static, single-turn formats, typically in- volving multiple-choice questions or direct answer gener- ation (Wang et al., 2024b; Suzgun et al., 2022; Yue et al., 2024; Lu et al., 2022). However, static benchmarks are vul- nerable to data leakage, since portions of their test data may inadvertently be included in the massive datasets used for model pre-training (Balloccu et al., 2024; Xu et al., 2024; Ni et al., 2025; Li et al., 2023). One way to mitigate cheat- ing is through dynamically updated benchmarks, such as LatestEval (Li et al., 2024) and LiveBench (White et al., 2024), which continuously incorporate the latest real-world data. While this approach alleviates the issue to some ex- tent, it still fails to enable fine-grained analysis and precise evaluation of the internal knowledge a model has acquired. To address this, we propose a dynamic, multi-turn frame- work designed to preclude rote memorization and compel the model to externalize its deductive reasoning. 2.2. Puzzles and Games as Evaluation Paradigms Puzzles, broadly defined as structured problems that require reasoning to reach a solution, have long served as effective probes of intelligence (Guo et al., 2024; Estermann et al., 2024; Shah et al., 2024). They constrain the search space while still demanding creativity, logical inference, and the integration of disparate knowledge. Accordingly, puzzles and games have been widely adopted as formal evaluation paradigms, from classic board games like Chess and Go (Sil- ver et al., 2017) to text-based adventure games (Wang et al., 2024a; Gallotta et al., 2024; Zhang & Long, 2025) and in- teractive formats such as 20 Questions (Vinyals et al., 2019; Liang et al., 2023; Hu et al., 2018). A common feature of these settings is sustained, multi-turn interaction within a structured environment, which makes them well-suited for assessing strategic planning, logical deduction, and efficient use of information (Chen et al., 2024; Li et al., 2025; Chen et al., 2025). Motivated by this tradition, we frame evalua- tion as a multi-turn process that requires a model to external- ize its reasoning trajectory rather than merely produce final answers. The Active Probing framework repurposes the puzzle format to evaluate how a model retrieves, integrates, 3 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving and reasons over its internalized knowledge to identify a target concept it is expected to know, thereby prioritizing the knowledge structure and logical coherence of the reasoning trajectory over simple binary success metrics. 3. Method ACTIVE PROBING BENCHMARK is founded on a fundamen- tal generalization of the puzzle definition, which enables the rapid construction of custom reasoning benchmarks. Section 3.1 details this generalized puzzle setting, along with our evaluation settings and metrics. Section 3.2 then describes the construction details in the APB. 3.1. Multi-turn Active Probing Process Our evaluation method is a multi-turn interactive protocol. Its extensibility comes from a general definition of a puz- zle as target concept identification through active evidence acquisition. Generalized Puzzle Definition.Each instance specifies a hidden target conceptP target (e.g., a specialized Wikipedia title or a technical command) and an associated reference documentI ref that is accessible only to the oracle. The model must identifyP target by asking questions and integrat- ing turn-level feedback rather than by reading I ref directly. Problem Formulation. LetVdenote the concept space and initialize the candidate set asS 0 = V. The model interacts for at mostT = MAXROUNDSturns. At each turnt, the model produces a multiple-choice queryQ t = (q t ,O t )and optionsQ t = (q t ,O t )including questionq t and options O t , and receives feedbackA t . The interaction history is H t =(Q 1 ,A 1 ),..., (Q t ,A t ). Interaction Mechanism. Algorithm 1 summarizes the inter- action loop. At each turnt, the Querying Agent conditions on the accumulated historyH tā1 and outputs a multiple- choice queryQ t = (q t ,O t ). IfP target ā Q t , the episode ter- minates withSUCCESS; otherwise, the Knowledge Oracle consults the referenceI ref and returns turn-level validation feedbackA t =V(Q t ,I ref )without exposing the reference text. The pair(Q t ,A t )is appended to formH t for the next turn, and the process continues until success or the budget T is exhausted. 3.2. The Active Probing Benchmark To assess the specialized knowledge that models internalize from core training corpora like Wikipedia, we constructed the Active Probing Benchmark, featuring 960 puzzles derived from Wikipedia across eight distinct domains. Data Construction.The benchmark targets were curated via a rigorous, multi stage filtering pipeline (Figure 3) that Algorithm 1 APB multi-turn active probing process Input: Round budget T , puzzle target P target , reference I ref 1: H 0 āā 2: for t = 1 to T do 3: Q t = (q t ,O t )ā Ļ Īø (H tā1 ) ā· Querying Agent: input H tā1 , output Q t 4:if P target ā Q t then 5:return SUCCESS 6:end if 7: A t āV(Q t ,I ref ) ā· Knowledge Oracle: option validation feedback 8: H t ā H tā1 āŖ(Q t ,A t ) 9: end for 10: return FAILURE distills highly specialized concepts from a dump of 6.4M English Wikipedia titles. The pipeline acts as a coarse to fine funnel: it first removes broad classes of common entities and then progressively concentrates on concepts that require advanced domain expertise. The selection process consists of the following four key stages: ā¢ Stage 1 (Broad filter): We rapidly removed common entities, pop culture references, and general knowledge terms, yielding an initial pool of candidates. ā¢Stage 2 (Expertise filter): We applied prompt based crite- ria that retain only titles judged to require doctorate level specialized knowledge claimed in prompts, discarding undergraduate or introductory concepts. ā¢Stage 3 (Domain stratification): The remaining candi- dates were categorized into eight academic domains to ensure balanced coverage (e.g., biology, physics, humani- ties, and health sciences). ā¢Stage 4 (Difficulty selection): Within each domain, we ran a dynamic tournament style selection: from batches of roughly 150 specialized candidates, we retained only the 5ā10 most challenging items according to model based difficulty scoring. This step concentrates the final bench- mark on the extreme tail of the difficulty distribution. Across stages, the pipeline retains approximately 0.14% of the original title set, producing a pool of roughly 9k advanced candidates and randomly selecting 120 targets per domain (960 puzzles total). The filtering code and prompts are released for reproducibility. 1 For each puzzle, the cleaned text from the corresponding Wikipedia article serves as the reference document I ref . Difficulty validation. To provide quantitative evidence that the selected targets correspond to specialized, long tail knowledge, we analyze Wikipediaās link graph struc- ture. For entities at different pipeline stages, we compute their in-link count (backlink count), which serves as a proxy for graph centrality (and correlates with common- 1 https://anonymous.4open.science/r/ wikipedia_filter-7870 4 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving R Specialized Knowledge Screening Doctoral- Level Filtering Discipline Categorization Final Selection (120 per Domain) Reference Documents Wikipedia Dump Figure 3. The multi-stage, coarse-to-fine data filtering pipeline used to construct the Active Probing Benchmark from the English Wikipedia dump. Table 1. Topological Analysis of Concept Difficulty. In-link counts drop drastically, confirming the selection of āedgeā knowledge. GroupNMeanMed.Type G1: Rejected (Stage 1)1k379.2500 + Common G2: Rejected (Stage 4)1k86.719.0Specialized G3: Final (APB)1k59.316.0Deep/Edge ness). As shown in Table 1, entities rejected in Stage 1 exhibit saturated in-link counts (median> 500, capped by the Wikipedia API), consistent with central and widely refer- enced concepts. In contrast, the final APB targets have much smaller in-link counts (median 16), indicating that they are substantially more peripheral in the knowledge graph and thus more specialized than the rejected groups. 3.3. The Active Probing Benchmark Benchmark Composition. The Active Probing Bench- mark is composed of 960 puzzles, which are systematically categorized into eight distinct domains: Biology, Human Disease, Chemistry, Physics, Mathematics, Computer Sci- ence, Economics, and Philosophy. These domains were chosen to ensure comprehensive coverage of major aca- demic fields, including the humanities and social sciences, the natural and engineering sciences, and health. Each domain comprises 120 puzzles distributed across 7 to 8 primary sub-disciplines (Figure 4) to ensure diverse topical coverage. For example, the Physics domain sources items from eight distinct subfields, with key contributions from Quantum Field Theory and Mathematical Physics. We also provide additional granularity within each subfield. For instance, the āSpecialized Topicsā sub-discipline covers Plasma Physics, Atomic, Molecular, and Optical (AMO) Physics, and Particle and Nuclear Physics. This multi-topic setup is a fundamental aspect of our benchmark across all domains, supporting a comprehensive evaluation of expert knowledge. Full details are described in the Appendix D. 4. Experiments and Analyses 4.1. Setup This section outlines the key parameter configurations and the set of models evaluated in our experiments. Biology Molecular and Genomic Bio Ecology and Evolution Biotech System & Omics Cell Bio Biochem Immunology Human Disease Systemic & Genetic Dis. Neurogenetic Oncology Neurology Immuno Endo Organ-System Chemistry Organic Chem Biochem and Chemical Bio Physical Chem Organometall ic Analytical Chem Applied Chem Physics Quantum Field Theo Math Phys Quantum Mech General Relativity Specialized Condensed Matter String Theo Mathematics Geometry & Topology Number Theory Analysis Crypto & Logic Algebra & Group Theo Phys Computer Science Theoretical CS AI/ML PL & Formal Scientific Computing Cryptography Algorithm Theo Quantum Computing Economics Pluralist Econ Microeconomi cs Econometrics Macroeconomics Math Econ Game Theory Finance Philosophy Religion and Interdiscipl inary Phil Logic and Math Phil Epistemology Metaphysics & Ontology Continental Phil Eastern Phil Figure 4. Hierarchical distribution of puzzle targets across domains and sub-disciplines. Eight distinct color families represent the primary domains, while individual segments denote specific sub- topics. The area of each segment is proportional to the number of questions in that category. Exact distributions are provided in Appendix D. Maximum Rounds. We setMAXROUNDSto 15. This de- cision is justified by empirical data (Table 3), which shows that a successful session concludes in 9.78 turns on average (9.15 for the top-performingGPT-5). A limit of 15 thus pro- vides sufficient opportunity for models to converge without being cost-prohibitive. Simultaneously, this cap manages the escalating token costs associated with including the full conversation history in each turn. While increasing the limit to 20 would have nearly doubled the evaluation cost. Temperature. We standardized the temperature at 0.2 to ensure reproducibility. Our ablation study (Appendix B.2) confirms the robustness of this choice, demonstrating that alternative settings (e.g., higher temperatures for reasoning models) yield negligible gains and do not alter the perfor- mance hierarchy.GPT-5used its default āmediumā effort, 5 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Table 2. Success Rate (%) for various models across eight domains. The best performance in each domain is highlighted in bold. The bottom row, āDomain Avg.ā, summarizes the average performance for each domain. ModelBioDisChemPhysMathCSEconPhilAvg. GPT-534.237.533.334.234.225.039.230.033.4 Claude Opus 427.540.829.226.718.319.231.727.527.6 Claude Sonnet 429.235.023.323.317.518.342.521.726.4 o328.333.321.729.220.010.035.025.0 25.3 GPT-4o26.724.220.017.511.710.825.020.819.6 Grok-414.230.816.713.310.013.320.821.717.6 DeepSeek V321.722.513.315.011.711.718.316.7 16.4 GPT-OSS-120B19.221.712.510.85.88.320.88.313.4 Gemini 2.5 Pro No-Thinking9.228.312.56.75.87.516.717.513.0 Qwen3-235B-A22B-Instruct5.827.55.011.75.810.015.020.0 12.6 Gemini 2.5 Pro7.526.711.713.31.78.317.513.312.5 DeepSeek R15.821.75.01.75.03.310.05.07.2 Qwen3-235B-A22B-Thinking7.511.75.83.31.75.09.28.3 6.6 Domain Avg.18.227.816.215.911.511.623.218.117.8 verified as optimal in our analysis.ā Knowledge Oracle. We designatedDeepSeek V3as the Knowledge Oracle responsible for providing correctness feedback. This model was chosen for its strong instruction- following capabilities and consistent outputs at a low tem- perature. To ensure our main findings are not sensitive to this specific choice, we conducted a detailed ablation study (Appendix B.1). This includes both a subset analysis and a comprehensive cross-validation usingClaude Sonnet 4on the full 960-puzzle benchmark, confirming that the relative performance hierarchy of models remains stable (Ļ = 0.96) despite absolute scoring shifts. Evaluated Models. Our evaluation on the APB covers a comprehensive suite of 13 leading closed-source and open- source LLMs to assess their capabilities. We list all the evaluated models below: ā¢Closed-Sourcemodels:GPT-5,o3and GPT-4o(Hurst et al., 2024),Claude Opus 4 andClaude Sonnet 4,Gemini 2.5 Proand Gemini 2.5 Pro No-Thinking(Comanici et al., 2025), Grok-4. ā¢ Open-Sourcemodels:Deepseek V3(Liu et al., 2024a) andDeepseek R1(Guo et al., 2025),GPT-OSS-120B(Agarwaletal., 2025),Qwen3-235B-A22B-Instructand Qwen-235B-A22B-Thinking(Yang et al., 2025). 4.2. Main Results The main evaluation results are presented in Table 2, cov- ering detailed success rate assessments across 8 domains: Biology (Bio), Human Disease (Dis), Chemistry (Chem), Physics (Phys), Mathematics (Math), Computer Science (CS), Economics (Econ), and Philosophy (Phil). Analysis of Main Results. The evaluation results demon- strate pronounced disparities across 8 domains (Table 2). In the strongest-performing domains, Human Disease (27.8%) and Economics (23.2%) achieved the highest average suc- cess rates. The high accuracy in Human Disease can be attributed to its comprehensive and well-structured classifi- cation system, while the strong performance in Economics may stem from the relatively lower depth and complexity of its specialized knowledge. In the most challenging do- mains, Mathematics (11.5%) and Computer Science (11.6%) recorded the lowest success rates, where evenGPT-5, the strongest model, attained only 34.2% and 25.0% respec- tively. Intermediate performance was observed in Biology, Chemistry, Physics, and Philosophy. From a model-centric perspective, the results reveal a clear performance hierarchy. WhileGPT-5leads the pack, its predecessorGPT-4o(19.6%) demonstrates remarkable longevity, effectively separating the top-performing mod- els and the rest. When compared to other closed-source offerings, the relatively modest results from theGemini series point to specific weaknesses for this reasoning task. Furthermore, the struggles ofDeepSeek R1(7.2%) and Qwen3-235B-A22B-Thinking(6.6%) underscore the current limitations and vulnerabilities within the open- source reasoning model landscape. 4.3. Detailed Analyses This section provides a detailed analysis of model perfor- mance based on Table 3, focusing on the behavioral patterns observed during the knowledge reasoning process. 6 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving ModelPLIGIOISSRTR GPT-53000.326.163.619.1513.02 Claude Opus 44940.486.414.1110.2313.60 Claude Sonnet 45650.445.133.149.8113.54 o33570.526.323.599.6613.67 GPT-4o4730.816.574.3410.6814.06 Grok-419301.154.873.089.7214.04 DeepSeek V35512.295.312.689.9614.07 GPT-OSS-120B4091.216.133.649.5314.23 Gemini 2.5 Pro No-Thinking 9082.125.513.8210.4814.37 Qwen3-235B-A22B-Instruct13961.437.154.759.2014.21 Gemini 2.5 Pro9062.205.283.7010.3114.37 DeepSeek R15745.133.632.2210.2014.66 Qwen3-235B-A22B-Thinking 14952.984.152.918.2114.48 Avg. 7971.625.593.519.7814.02 Table 3. Detailed behavioral analysis metrics. PL: Probing Length, IG: Incorrect Guess rounds, IO: Incorrect Option rounds, IS: Incor- rect Streak rounds, SR: Success Rounds, TR: Total Rounds. Blue bars visualize the normalized magnitude of each metric within its column to highlight behavioral patterns. Probing Length. The Probing Length (PL) measures the average character length of the agentās output per turn. This output typically includes not only the multiple-choice ques- tion but also the modelās explicit reasoning based on prior rounds. Our analysis reveals that increased verbosity does not correlate with better performance on this task. Indeed, efficiency in knowledge probing rounds appears to be a hallmark of the top models likeGPT-5(PL = 300). This trend generally holds for the other top-performing models with relatively concise probing lengths. Furthermore, we observe that PL is often strongly correlated with a modelās developer family, suggesting that different series of models possess inherent and consistent styles for expressing their reasoning process in this interactive setting. For instance, bothGeminimodels exhibit a similarly high verbosity (PL ā907), as do theDeepSeekmodels (PLā560) and the OpenAImodels (299 < PL < 500). This suggests that different series of models possess inherent and consistent styles for expressing their probing process in this interactive setting. Incorrect Guess Rounds. The Incorrect Guess Rounds (IG) metric quantifies the average number of times an agent makes a conclusive but incorrect guess of the target concept per session. We interpret a high IG as an indicator of model overconfidence, where the agent prematurely commits to a wrong answer despite having insufficient information. Our results show a strong negative correlation between IG and Success Rate, suggesting that early incorrect commitments are associated with persistent mislocalization in the knowl- edge space, which limits the usefulness of subsequent prob- ing. For instance, the top-performing modelGPT-5has the lowest IG (0.32), while the struggling DeepSeek R1 has the highest (5.13). From a strategic standpoint, such actions represent a low-utility approach by invalidating only a single hypothesis. In a context like Human Disease, a propensity for making frequent, incorrect assertions is a harmful failure mode that could lead to detrimental outcomes, underscoring the importance of penalizing such overconfident behavior in model evaluation. Incorrect Probing and Error Correction. We analyze two metrics related to probing errors: Incorrect Option rounds (IO) and Incorrect Streak rounds (IS). IO measures the aver- age number of rounds where all four proposed options are incorrect, suggesting inaccurate concept grounding during knowledge localization. IS measures the longest consecu- tive sequence of such rounds, reflecting the modelās ability to recover from a flawed reasoning path. However, these metrics are not simple indicators of poor performance. An āall-incorrectā round can be highly informative for a capa- ble agent, as it allows for significant pruning of the search space and a major strategy correction. This can paradoxi- cally help a strong model quickly locate an obscure concept. Thus, these metrics offer insights into a modelās strategic behavior. The case ofDeepSeek R1is particularly illus- trative: despite having the lowest IO (3.63) and IS (2.22), it achieves one of the lowest success rates. This suggests that its probing strategy is ineffective at leveraging infor- mation and asking questions to find theP target , a strategic failure that helps explain its high rate of premature guesses, as reflected in its top-ranking IG. Overall Efficiency. We evaluate efficiency using two com- plementary metrics: Average Success Rounds (SR) and Average Total Rounds (TR). SR measures the average num- ber of rounds in successful sessions only, reflecting speed on solvable problems. TR provides a more holistic score by in- cluding failed sessions, which are counted asMAXROUNDS (15), thus blending success rate with efficiency. Relying solely on SR can be misleading for models with low success rates, as it only reflects performance on easier tasks they could solve. TR, therefore, offers a more comprehensive view. For example,GPT-5ās top performance gives it the best TR (13.02) and leading SR (9.15), confirming its po- sition as the most effective model overall. This combined metric also reveals nuances in reasoning efficiency. For instance, despiteClaude Opus 4having a higher Suc- 7 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving cess Rate thanClaude Sonnet 4,Sonnetās lower TR (13.54 vs. 13.60) and SR (9.81 vs 10.23) suggest it has a relative advantage in the efficiency of its reasoning process. 4.4. Error Analysis Open-Source Reasoning Models.To understand the relatively weak performance of open source reasoning models, we analyzed the publicly available chain-of- thought traces forGPT-OSS-120B,DeepSeek R1, and Qwen3-235B-A22B-Thinking. We observe a recur- ring pattern that is best characterized as a failure of knowl- edge localization during active probing. Rather than using early turns to acquire broadly discriminative evidence, these models often anchor on several concrete candidate targets forP target at the beginning of an episode (Figure 5). Subse- quent multiple-choice queries are then constructed to seek confirmation for these candidates, instead of eliciting infor- mation that would re-partition the broader candidate space. This premature anchoring has two downstream effects. First, when the trueP target is not among the initial hypotheses, the modelās questions tend to target attributes that are irrelevant to the correct region of the domain knowledge space, yield- ing feedback that is weakly informative for correction. As a result, the interaction trace exhibits repetitive cycles of querying around the wrong neighborhood, with many low- utility terms and little reduction of the remaining uncertainty. Second, the same anchoring increases the likelihood of con- clusive but unsupported commitments, producing more fre- quent incorrect final guesses. This is consistent with the comparatively high IG values observed for these models and with their lower success rates overall. These traces suggest that the performance gap is driven less by a lack of procedural reasoning steps and more by difficulty in using multi-turn feedback to maintain and up- date a well-grounded hypothesis set. In other words, the models often fail to treat early turns as evidence acquisition for concept identification, and instead collapse prematurely onto specific candidates, which limits their ability to recover when the initial localization is wrong. Gemini Series Models. TheGemini 2.5 Promodels exhibit a distinct behavioral pattern that helps explain their relatively weak performance. A primary characteristic is their high Probing Length (PLā907), far exceeding other top closed-source models. We observe that this verbosity stems from a tendency to repeatedly summarize and elab- orate on prior turns before issuing the next query. From a knowledge probing perspective, this behavior shifts effort from acquiring new discriminative evidence to reprocessing self-generated context, which can dilute the influence of objective turn-level feedback. As a result, the model may persist in a mislocalized hypothesis set and continue probing within an irrelevant region of the knowledge space, culmi- Deepseek R1 Reasoning Content for Reticular Dysgenesis (Round 5) Extracted Hypotheses Generated Question Okay, the user said my guess of Sickle Cell Anemia was wrong. Let me think again... Cystic Fibrosis? No, thatās autosomal... Thalassemia ? Also... Or maybe something like Gaucherās ? Wait... Or maybe Fanconi anemia ? Bone marrow... Beta-thalassemia? Affects blood... H. spherocytosis, PK deficiency... Let me ask about clinical features... Cystic Fibrosis Thalassemia Gaucher's Fanconi Anemia Beta-thalassemia Hemophilia H. Spherocytosis PK Def. G6PD Def. Sickle Cell Which clinical feature is most characteristic of this disease? A) Recurrent infections and pancreatic insufficiency B) Severe anemia with target cells on blood smear C) Episodic pain crises and vascular occlusion D) Easy bruising and prolonged bleeding Figure 5.DeepSeek R1reasoning trace for the target āReticular dysgenesisā (Round 5). Top: The internal reasoning content shows the model brainstorming specific disease candidates following an incorrect guess. Middle: The extracted hypotheses visualize the diverse set of conditions considered during this step. Bottom: The generated follow-up question focuses on differentiating clinical features. nating in more frequent unsupported final commitments, reflected by their notably high IG (2.1ā2.2) relative to other leading closed-source models. 5. Conclusion We introduced APB, an interactive benchmark that extends puzzle evaluation from narrow game-like settings to knowl- edge reasoning over specialized concepts. By casting con- cept identification as multi-turn information seeking with verifiable endpoints, APB provides process-level evidence of how models acquire and use domain knowledge. Exper- iments show that APB is challenging even for strong sys- tems (GPT-5reaches 33.4% success), and that several open source reasoning models lag behind their no-thinking coun- terparts, consistent with difficulties in grounding multi-turn inference in specialized knowledge. Beyond these results, APB offers a practical construction pipeline for producing challenging diagnostic evaluations in additional specialized domains, helping broaden the empirical study of knowledge reasoning under interaction. 8 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Impact Statement The primary objective of this work is to evaluate and better understand the limitations of interactive knowledge reason- ing in large language models, contributing to more trustwor- thy and informative model assessment. APB is intended for research use and reports objective success criteria together with qualitative failure analyses to reduce over-interpretation of single metrics. This study was conducted independently and adheres to standard ethical research principles, includ- ing considerations of bias, fairness, privacy, security, and legal compliance. References Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. Badola, K., Simon, J., Hosseini, A., Carthy, S. M. M., Munkhdalai, T., Goyal, A., Ko Ė cisk ` y, T., Upadhyay, S., Fatemi, B., and Kazemi, M. Multi-turn puzzles: Evaluat- ing interactive reasoning and strategic dialogue in llms. arXiv preprint arXiv:2508.10142, 2025. Balloccu, S., Schmidtov Ģ a, P., Lango, M., and Du Ė sek, O. Leak, cheat, repeat: Data contamination and evalua- tion malpractices in closed-source llms. arXiv preprint arXiv:2402.03927, 2024. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patward- han, T., et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024. Chen, G., Xu, W., Zhang, H., Chan, H. P., Liu, C., Bing, L., Zhao, D., Luu, A. T., and Rong, Y. Finereason: Evaluat- ing and improving llmsā deliberate reasoning through re- flective puzzle solving. arXiv preprint arXiv:2502.20238, 2025. Chen, Q., Zhang, B., Wang, G., and Wu, Q. Weak-eval- strong: Evaluating and eliciting lateral thinking of llms with situation puzzles. Advances in Neural Information Processing Systems, 37:79642ā79665, 2024. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., et al. Chatbot arena: An open platform for evaluat- ing llms by human preference. In Forty-first International Conference on Machine Learning, 2024. Chollet, F. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Contributors, O. Opencompass: A universal evaluation platform for foundation models, 2023. Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, p. 11198ā 11201, 2024. Estermann, B., Lanzend Ģ orfer, L., Niedermayr, Y., and Wat- tenhofer, R. Puzzles: A benchmark for neural algorithmic reasoning. Advances in Neural Information Processing Systems, 37:127059ā127098, 2024. Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., To- gelius, J., and Yannakakis, G. N. Large language models and games: A survey and roadmap. IEEE Transactions on Games, 2024. Giadikiaroglou, P., Lymperaiou, M., Filandrianos, G., and Stamou, G.Puzzle solving using reasoning of large language models: A survey.arXiv preprint arXiv:2402.11291, 2024. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Guo, K., Nan, B., Zhou, Y., Guo, T., Guo, Z., Surve, M., Liang, Z., Chawla, N., Wiest, O., and Zhang, X. Can llms solve molecule puzzles? a multimodal benchmark for molecular structure elucidation. Advances in Neu- ral Information Processing Systems, 37:134721ā134746, 2024. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 9 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021. Hu, H., Wu, X., Luo, B., Tao, C., Xu, C., Wu, W., and Chen, Z. Playing 20 question game with policy-based reinforcement learning. arXiv preprint arXiv:1808.07645, 2018. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Jacovi, A., Caciularu, A., Goldman, O., and Goldberg, Y. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation bench- marks. arXiv preprint arXiv:2305.10160, 2023. Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021. Li, K., Chen, X., Song, T., Zhou, C., Liu, Z., Zhang, Z., Guo, J., and Shan, Q. Solving situation puzzles with large language model and external reformulation. arXiv preprint arXiv:2503.18394, 2025. Li, Y., Guerin, F., and Lin, C. An open source data contami- nation report for large language models. arXiv preprint arXiv:2310.17589, 2023. Li, Y., Guerin, F., and Lin, C. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, p. 18600ā18607, 2024. Liang, T., He, Z., Huang, J.-t., Wang, W., Jiao, W., Wang, R., Yang, Y., Tu, Z., Shi, S., and Wang, X. Leverag- ing word guessing games to assess the intelligence of large language models. arXiv preprint arXiv:2310.20499, 2023. Lin, B. Y., Wu, Z., Yang, Y., Lee, D.-H., and Ren, X. Rid- dlesense: Reasoning about riddle questions featuring lin- guistic creativity and commonsense knowledge. arXiv preprint arXiv:2101.00376, 2021. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., Zhang, W., Zhang, S., Lin, D., and Chen, K. Math- bench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. arXiv preprint arXiv:2405.12209, 2024b. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evalu- ating llms as agents. arXiv preprint arXiv:2308.03688, 2023. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507ā2521, 2022. Ni, S., Kong, X., Li, C., Hu, X., Xu, R., Zhu, J., and Yang, M. Training on the benchmark is not all you need. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, p. 24948ā24956, 2025. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. Roberts, M., Thakur, H., Herlihy, C., White, C., and Dooley, S. To the cutoff... and beyond? a longitudinal perspective on llm data contamination. In The Twelfth International Conference on Learning Representations, 2023. Shah, K., Dikkala, N., Wang, X., and Panigrahy, R. Causal language modeling can elicit search and reasoning capa- bilities on logic puzzles. Advances in Neural Information Processing Systems, 37:56674ā56702, 2024. Shirali, A., Abebe, R., and Hardt, M. A theory of dynamic benchmarks. arXiv preprint arXiv:2210.03165, 2022. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354ā359, 2017. Singh, A. K., Kocyigit, M. Y., Poulton, A., Esiobu, D., Lomeli, M., Szilvasy, G., and Hupkes, D. Evaluation data contamination in llms: how do we measure it and (when) does it matter? arXiv preprint arXiv:2411.03923, 2024. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga- Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023. Suzgun, M., Scales, N., Sch Ģ arli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., 10 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Zhou, D., , and Wei, J. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jader- berg, M., Czarnecki, W. M., Dudzik, A., Huang, A., Georgiev, P., Powell, R., et al. Alphastar: Mastering the real-time strategy game starcraft i. DeepMind blog, 2:20, 2019. Wang, R., Todd, G., Xiao, Z., Yuan, X., C Ė ot Ģ e, M.-A., Clark, P., and Jansen, P. Can language models serve as text-based world simulators?arXiv preprint arXiv:2406.06485, 2024a. Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language un- derstanding benchmark. Advances in Neural Information Processing Systems, 37:95266ā95290, 2024b. White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., et al. Livebench: A challenging, contamination-limited llm benchmark. arXiv preprint arXiv:2406.19314, 2024. Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. Benchmark- ing benchmark leakage in large language models. arXiv preprint arXiv:2404.18824, 2024. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9556ā9567, 2024. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. Zhang, J. and Long, Y. Learning to play like humans: A framework for llm adaptation in interactive fiction games. arXiv preprint arXiv:2505.12439, 2025. Zhang, Z., Chen, Z., Zhang, Z., Sun, Y., Tian, Y., Jia, Z., Li, C., Liu, X., Min, X., and Zhai, G. Puzzlebench: A fully dynamic evaluation framework for large multimodal mod- els on puzzle solving. arXiv preprint arXiv:2504.10885, 2025. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., and Lo, D. Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks. arXiv preprint arXiv:2502.06215, 2025. 11 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Appendix A. Rapid Custom Benchmark Construction Our framework provides a scalable and generalizable method for rapidly constructing diagnostic benchmarks in any specialized domain. The process is straightforward and consists of two main stages: ā¢ Target Identification and Curation: First, a domain of interest and a corresponding source corpus of knowl- edge are identified (e.g., technical manuals, API doc- umentation, academic articles). From this corpus, a list of key concepts, terms, commands, or entities is systematically extracted to serve as the puzzle targets (P target ) ā¢Reference Document Generation: For each puzzle tar- get, a āgoldenā reference document (I ref ) is established (e.g., the specific documentation page for a command). This document serves as the ground truth for the envi- ronmentās feedback during the multi-turn evaluation. A.1. Case Study: The IC Back-End Puzzle Benchmark To demonstrate the efficacy of this method, we applied it to the niche, industrial domain of IC back-end design. We used the official documentation for four widely-used EDA command setsāSynopsys ICC, ICC2, and Cadence Innovus (both common and legacy)āas our source corpus. Table 4. Success rates (%) on the IC Back-End Puzzle Benchmark. Tools are grouped by vendor: Synopsys (ICC, ICC2) and Ca- dence Innovus (Common, Legacy). āNTā denotes āNo-Thinkingā. Qwen3 models refer to the Qwen3-235B-A22B variants. Best results in each category are bolded. SynopsysCadence Innovus ModelICCICC2CommonLegacyAvg. Claude Opus 427.514.216.722.520.2 GPT-525.815.014.220.018.8 grok-415.010.818.324.217.1 o320.015.012.512.515.0 Claude Sonnet 414.210.89.218.313.1 Gemini 2.5 Pro10.88.35.016.7 10.2 Gemini 2.5 Pro NT 10.88.36.712.59.6 GPT-4o6.77.57.57.57.3 DeepSeek V310.84.26.75.86.9 GPT-OSS-120B8.37.54.24.2 6.0 Qwen3 Instruct4.25.83.34.24.4 Qwen3 Thinking6.72.53.35.04.4 DeepSeek R14.20.82.52.52.5 Domain Avg.12.78.58.512.010.4 Following our process, we curated 480 puzzles by sam- pling 120 command names from each of the four sets. This benchmark exemplifies how our framework can be used to quickly create robust, expert-level evaluations that test practical knowledge in any specialized domain. We evaluated 13 leading LLMs on this benchmark, with the results presented in Table 4. The findings underscore the benchmarkās significant challenge, with an average success rate of only 10.4% across all models. A clear performance hierarchy emerged, with Claude Opus 4 (20.2%), GPT-5 (18.8%), and Grok-4 (17.1%) forming a distinct top tier. No- tably, reasoning-enhanced models like DeepSeek R1 (2.5%) and Qwen3 235B Thinking (4.4%) performed very poorly, reinforcing our main paperās finding of a gap between pro- cedural reasoning and applied domain knowledge. The domain-specific results also reveal non-uniform knowledge; for instance, Grok-4 excels on Innovus commands but is weaker on ICC, suggesting that model knowledge of these specialized tools is often fragmented. This case study vali- dates our methodās ability to create challenging benchmarks that effectively differentiate model capabilities in highly specialized, real-world domains. A.2. Broader Applications and Future Directions The methodology detailed in this section is designed for broad applicability beyond the technical domains demon- strated. We encourage the research community to leverage our scalable process to construct new benchmarks across a wide array of fields. For example, puzzle targets could be drawn from the humanities (e.g., famous figures, artworks, film titles) or the natural sciences (e.g., celestial bodies, materials, biological species). The creation of such diverse benchmarks would facilitate a more holistic evaluation of an LLMās all-around knowl- edge reasoning abilities. Moreover, the resulting multi-turn interaction data would provide a rich resource for deeper, cross-domain pattern recognition, enabling a more com- prehensive exploration of model reasoning strategies and behaviors. B. Ablation Study To validate the robustness of our evaluation framework, we conducted a comprehensive ablation study identifying potential sources of variance in our experimental pipeline. Our objective is to confirm that the observed model perfor- mance tiers are intrinsic to the agentsā capabilities rather than artifacts of the evaluation configuration. Specifically, we address two critical dimensions: (1) the reliability of the Knowledge Oracle (Judge Consistency), and (2) the sensi- tivity of performance to hyperparameters and inference- time compute (Temperature and Reasoning Effort). 12 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving 0 20 40 Score (Claude Sonnet 4) Ļ=0.79 Bio Ļ=1.00 Dis Ļ=0.89 Chem Ļ=0.86 Phys 02040 Score (DeepSeek V3) 0 20 40 Score (Claude Sonnet 4) Ļ=0.88 Math 02040 Score (DeepSeek V3) Ļ=0.93 CS 02040 Score (DeepSeek V3) Ļ=0.87 Econ 02040 Score (DeepSeek V3) Ļ=0.94 Phil GPT-5 Claude Opus 4 Grok-4 GPT-4o DeepSeek V3 GPT-OSS-120B DeepSeek R1 Figure 6. Ranking consistency analysis (DeepSeek V3 vs. Claude Sonnet 4) across eight domains. Each point represents a modelās performance in a specific domain. The distinct clustering of model colors and the high Spearmanās rank correlation (Ļ) within each domain confirm that relative model rankings remain robust despite the absolute score shift caused by the stricter judge. B.1. Cross-Validation with Claude Sonnet 4 We employed Claude Sonnet 4 to cross-validate the main results against DeepSeek V3. This evaluation covered the full core benchmark (960 puzzles) across seven representa- tive models, covering the spectrum from top-tier proprietary models to open-source and reasoning-specific models. Rank Stability Across Different JudgesFigure 6 visual- izes the domain-level performance correlation between the two judges. While absolute success rates tend to be higher under Claude Sonnet 4 (as indicated by the data points generally lying above they = xdiagonal), the relative performance ordering remains remarkably stable: ā¢Visual Consistency: As shown in the facet plots, mod- els form distinct clusters. Top-performing models (e.g., GPT-5, Claude Opus 4) consistently appear in the upper- right quadrant, while lower-tier models remain in the lower-left across all domains. ā¢High Correlation: The quantitative evidence supports this visual trend. We observe a Spearman rank correla- tion ofĻ = 0.96for the overall model rankings, indicat- ing near-identical leaderboard structures. This consis- tency remains robust at the granular level, with an aver- age correlation ofĻ = 0.89across individual domains (exemplified byĻ = 1.00in Disease andĻ = 0.94in Philosophy). ā¢Robust Failure Modes: āReasoning-enhancedā mod- els (DeepSeek R1, GPT-OSS-120B) remain at the bot- tom regardless of the judge. Notably, even with the more capable Claude Sonnet 4 as the judge, DeepSeek R1 achieves only 8.6% accuracy compared to 7.2% un- der V3. This confirms that the failure of these models stems from their internal strategies (i.e., the concrete-to- abstract fallacy), not judge-induced noise. Explanation for Spikes and Judge DifferencesOur case analysis identified two primary factors driving the observed differences between judges: context robustness and seman- tic validation style. ā¢Grok-4ās Outlier Performance (Context Robustness): Grok-4 exhibits an anomalously high probing length (avg. 1930 characters/turn). DeepSeek V3 struggles to maintain precise instruction-following when parsing such verbose contexts. In contrast, Claude Sonnet 4 effectively filters through Grok-4ās lengthy outputs to extract the core logic, explaining why Grok-4 shows the most dramatic improvement (+11.5%) under the stronger judge. ā¢ Judging Characteristics (Association vs.Preci- sion): DeepSeek V3 tends toward associative valida- tionāoccasionally accepting highly related āattributesā as the target āentityā itself. In contrast, Claude Sonnet 4 enforces stricter semantic precision. In conclusion, replacing DeepSeek V3 with Claude Sonnet 4 acts primarily as a linear scaling factor. It uniformly āliftsā the scores of capable models but does not fundamentally 13 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving alter the leaderboard. The high rank correlation confirms that DeepSeek V3 remains a valid, cost-effective proxy for differentiating model tiers. B.2.Robustness to Sampling Parameters and Reasoning Effort To ensure that our main findings are not artifacts of sub- optimal hyperparameters, we conducted a targeted abla- tion study on a representative subset of the benchmark (N = 120). We isolate the effects of sampling tempera- ture on open-source reasoning models and the impact of test-time compute scaling on proprietary models. Temperature Sensitivity in Open-Source ModelsA po- tential hypothesis for the underperformance of open-source reasoning models (e.g., DeepSeek R1, Qwen3) is that the default low-temperature settings (T = 0.2) might con- strain their exploratory capabilities. To test this, we evalu- ated DeepSeek R1 at its recommended high temperature (T = 1.0) and Qwen3 Thinking at a moderate setting (T = 0.6). As detailed in Table 5, the results indicate that temperature is not the bottleneck. ā¢DeepSeek R1: While increasing the temperature to T = 1.0yielded a marginal improvement (6.7%ā 8.3%), the model remains firmly in the bottom tier. ā¢ Qwen3: Increasing the temperature resulted in a slight performance fluctuation (11.7%ā10.8%), offering no statistical benefit. These findings reinforce our conclusion that the fail- ure modes of these modelsāspecifically the concrete- to-abstract fallacyāare structural rather than parameter- dependent. Table 5. Ablation results on sampling temperature and reasoning effort (N = 120). The āTierā column reflects the relative ranking within the full benchmark context. ModelConfigurationSuccess RateTier Effect of Temperature (Open-Source Reasoning) DeepSeek R1 Temp = 0.2 (Default)6.7%Bottom Temp = 1.0 (Recommended)8.3%Bottom Qwen3 Thinking Temp = 0.2 (Default)11.7%Low Temp = 0.610.8%Low Effect of Reasoning Effort (GPT-5) GPT-5 Low Effort33.3%Top Medium Effort (Default)38.3%Top High Effort34.6%Top Scaling Laws of Test-Time ComputeWe further investi- gated the relationship between reasoning effort and perfor- mance using GPT-5. By varying the inference budget across Low, Medium (Default), and High settings, we observed a distinct inverted-U trajectory: performance peaks at Medium effort (38.3%) and declines at High effort (34.6%). This observation leads to two critical insights: 1. Diminishing Returns: In the constrained paradigm of strategic questioning, excessive test-time compute does not linearly translate to better outcomes. āOver- thinkingā may lead to hallucinated constraints or overly convoluted search paths. 2. Strategy Quality over Quantity: Crucially, even at the āLowā effort setting, GPT-5 (33.3%) significantly out- performs the best open-source reasoning models. This suggests that the quality of the underlying reasoning strategyāhow the model prunes the search spaceāis a more determinant factor than the raw quantity of infer- ence steps or token generation. C. Token Usage This section details the token consumption patterns, aver- aged per puzzle, for both the Querying Agent (QA) and the Knowledge Oracle (KO). It is important to note that a single puzzle evaluation consists of multiple rounds of interaction. The token cost of the Knowledge Oracle (KO) is character- ized by highly efficient outputs but demanding inputs. The KOās completion tokens are minimal and stable across all tests (averaging 460 tokens), as its responses are restricted to a fixed format providing only correctness judgments. This ensures a fair and consistent evaluation. However, its prompt token requirements are substantial, averaging 18,347 tokens per puzzle, because the KO must process the full reference document (I ref ) in each round to provide accurate feedback. Similarly, the QA prompt costs are significant, averaging 18,826 tokens per puzzle, as the full interaction history is in- cluded in the context for each new turn. The vast majority of performance variance and economic cost, however, comes from the SAās completion tokens, which reflect model ver- bosity (Table 6). Notably, some of the poorest-performing open-source reasoning models were paradoxically the most computationally expensive. GPT-OSS-120B was the most costly, consuming over 72,000 total QA tokens on average, followed by Qwen3 235B Thinking (approx. 57,600 tokens) and DeepSeek R1 (approx. 43,300 tokens). Collectively, these three open-source models averaged 57,784 QA to- kens per puzzleāalmost exactly double the 28,705-token average of the remaining models. This contrasts sharply with top-performing models like GPT-5, which achieved the highest success rate while using less than half the tokens of GPT-OSS-120B. This suggests that excessive verbosity is not only an unsuccessful strategy but also an economically inefficient one for this task. 14 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Table 6. A summary of token usage for representative models. āComp.ā stands for Completion, āTok.ā for Tokens. KO output is low due to fixed formatting. ModelQAQAQA TotalKO Comp. Prompt Comp.Tok.Tok. (Best Performance) GPT-59,82124,30134,122430 (Most Verbosity) GPT-OSS-120B14,034 58,47172,505473 Qwen3 235B Think. 33,414 24,17057,584473 Deepseek R115,242 28,02143,263458 (Most Concise) GPT-4o13,4411,51314,954466 Average18,826 16,58335,416460 D. Benchmark Composition Table 7. Biology Domain Composition. Sub-DisciplineCount (%) Molecular and Genomic Biology29 (24.2%) Genes, Proteins, and Nucleic Acids16 (13.3%) Gene Expression and Regulation7 (5.8%) DNA Structure and Dynamics5 (4.2%) Genetics and Inheritance1 (0.8%) Ecology and Evolution24 (20.0%) Medical Biology and Pathology10 (8.3%) Evolutionary Biology and Genetics5 (4.2%) Biophysics and Theoretical Biology5 (4.2%) Ecology and Environmental Science4 (3.3%) Biotechnology and Bioengineering20 (16.7%) Molecular and Genetic Techniques9 (7.5%) Other Techniques and Models5 (4.2%) Imaging and Microscopy3 (2.5%) Drug Discovery and Delivery3 (2.5%) Systems and Omics Biology12 (10.0%) Systems-Level Concepts and Models8 (6.7%) ā-omicsā Fields4 (3.3%) Cellular and Subcellular Biology10 (8.3%) Cellular Processes3 (2.5%) Cell Death3 (2.5%) Organelles and Subcellular Structures2 (1.7%) Cytoskeleton and Molecular Motors2 (1.7%) Biochemistry and Metabolism10 (8.3%) Proteins and Enzymes8 (6.7%) Metabolic Pathways2 (1.7%) Immunology9 (7.5%) Core Concepts and Processes5 (4.2%) Signaling and Receptors4 (3.3%) Neuroscience6 (5.0%) Molecular and Cellular Neuroscience2 (1.7%) Systems and Cognitive Neuroscience2 (1.7%) Neuropharmacology1 (0.8%) Clinical Neuroscience1 (0.8%) 15 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Table 8. Human Disease Domain Composition. Sub-DisciplineCount (%) Systemic & Structural Genetic Disorders35 (29.2%) Multi-system Genetic Syndromes14 (11.7%) Skeletal and Overgrowth Syndromes13 (10.8%) Syndromes with Major Dermatologic Fea- tures 4 (3.3%) Genomic and Inheritance Disorders4 (3.3%) Neurogenetic Disorders18 (15.0%) Neurodevelopmental Syndromes13 (10.8%) Hereditary Neuromuscular and Neurodegen- erative Diseases 5 (4.2%) Oncology15 (12.5%) Hematologic Malignancies6 (5.0%) Solid Tumors6 (5.0%) Hereditary Cancer Syndromes2 (1.7%) Transmissible Cancers1 (0.8%) Neurology14 (11.7%) Neurodegenerative and Movement Disorders4 (3.3%) Epilepsy and Seizure Disorders3 (2.5%) Other Neurological Conditions3 (2.5%) Neuroinflammatory and Demyelinating Dis- eases 2 (1.7%) Congenital and Structural Brain Disorders2 (1.7%) Immuno & Rheum14 (11.7%) Autoinflammatory and Autoimmune Syn- dromes 7 (5.8%) Systemic Vasculitides4 (3.3%) Primary Immunodeficiencies2 (1.7%) Systemic and Catastrophic Conditions1 (0.8%) Endo & Metab13 (10.8%) Inborn Errors of Metabolism11 (9.2%) Disorders of Mineral and Porphyrin Metabolism 2 (1.7%) Organ-System11 (9.2%) Hematologic Disorders4 (3.3%) Gastroenterology and Hepatology3 (2.5%) Dermatologic Disorders1 (0.8%) Pulmonology1 (0.8%) Infectious and Transmissible Diseases1 (0.8%) Other Conditions1 (0.8%) Table 9. Chemistry Domain Composition. Sub-DisciplineCount (%) Organic Chemistry27 (22.5%) Named Reactions and Rearrangements11 (9.2%) Synthesis, Reagents, and Specific Com- pounds 8 (6.7%) Reaction Mechanisms and Concepts5 (4.2%) Stereochemistry and Molecular Structure3 (2.5%) Biochemistry and Chemical Biology27 (22.5%) Enzymes and Cofactors11 (9.2%) Biological Processes and Pathways8 (6.7%) Bio-analytical and Chemical Biology Tools6 (5.0%) Bio-inorganic Chemistry2 (1.7%) Physical Chemistry19 (15.8%) Quantum Chemistry and Spectroscopy6 (5.0%) Electrochemistry4 (3.3%) Thermodynamics and Calorimetry3 (2.5%) Kinetics and Dynamics3 (2.5%) Condensed Matter and Surface Chemistry3 (2.5%) Inorganometallics and Organometallics17 (14.2%) Organometallic Compounds8 (6.7%) Specific Inorganic Compounds6 (5.0%) Bonding and Molecular Structure3 (2.5%) Analytical Chemistry11 (9.2%) Spectroscopy and Spectrometry5 (4.2%) Instrumentation and Databases3 (2.5%) Microscopy and Surface Analysis2 (1.7%) Chromatography and Separation1 (0.8%) Applied Chemistry8 (6.7%) Photochemistry and Photophysics2 (1.7%) Magnetochemistry2 (1.7%) Industrial and Applied Chemistry2 (1.7%) Radiochemistry and Exotic Matter1 (0.8%) Theoretical Physics in Chemistry1 (0.8%) Materials Chemistry6 (5.0%) Key Concepts and Materials6 (5.0%) Theoretical Chemistry5 (4.2%) Concepts and Models3 (2.5%) Methods and Techniques2 (1.7%) 16 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Table 10. Physics Domain Composition. Sub-DisciplineCount (%) Quantum Field Theory22 (18.3%) Foundational and Axiomatic QFT9 (7.5%) Gauge Theories and the Standard Model6 (5.0%) Non-Perturbative and Topological QFT3 (2.5%) Supersymmetry and Beyond2 (1.7%) QFT in Curved Spacetime2 (1.7%) Mathematical Physics19 (15.8%) Differential Geometry and Topology6 (5.0%) Group Theory and Algebra5 (4.2%) Functional Analysis and Operator Theory5 (4.2%) Integrable Systems and Exact Solutions2 (1.7%) Twistor Theory1 (0.8%) Quantum Mechanics16 (13.3%) Foundational Concepts and Formalism6 (5.0%) Quantum Systems and Dynamics4 (3.3%) Quantum Computation and Information4 (3.3%) Quantum States and Phenomena2 (1.7%) General Relativity16 (13.3%) Alternative and Extended Theories of Grav- ity 7 (5.8%) Formalisms and Solutions5 (4.2%) Astrophysical Relativity3 (2.5%) Spacetime and Black Holes1 (0.8%) Specialized Topics in Physics16 (13.3%) Atomic, Molecular, and Optical (AMO) Physics 7 (5.8%) Particle and Nuclear Physics5 (4.2%) Plasma Physics4 (3.3%) Condensed Matter14 (11.7%) Quantum Materials and Phases6 (5.0%) Electronic and Spintronic Effects4 (3.3%) Many-Body and Superconducting Systems2 (1.7%) Statistical and Soft Matter2 (1.7%) String Theory12 (10.0%) Core Concepts and Dualities6 (5.0%) Advanced and Related Theories5 (4.2%) Cosmological Applications1 (0.8%) Classical Physics and Electrodynamics5 (4.2%) Core Concepts and Phenomena5 (4.2%) Table 11. Mathematics Domain Composition. Sub-DisciplineCount (%) Geometry & Topology37 (30.8%) Algebraic Topology & Homotopy Theory12 (10.0%) General & Geometric Topology12 (10.0%) Differential & Geometric Analysis11 (9.2%) K-Theory2 (1.7%) Number Theory19 (15.8%) Arithmetic & Algebraic Geometry12 (10.0%) Algebraic Number Theory & Class Field Theory 5 (4.2%) Analytic Number Theory2 (1.7%) Analysis18 (15.0%) Operator Theory & Operator Algebras9 (7.5%) Real, Complex, & Harmonic Analysis8 (6.7%) Control Theory1 (0.8%) Cryptography & Logic18 (15.0%) Cryptography7 (5.8%) Theoretical Computer Science & Algorithms6 (5.0%) Logic & Set Theory5 (4.2%) Algebra & Group14 (11.7%) Homological & Abstract Algebra8 (6.7%) Representation Theory4 (3.3%) Group Theory2 (1.7%) Theo Phys8 (6.7%) Quantum Mechanics & Quantum Field The- ory 3 (2.5%) General Relativity & Spacetime Physics2 (1.7%) Plasma & Fluid Dynamics1 (0.8%) Physical Chemistry1 (0.8%) Dynamical Systems1 (0.8%) Statistics6 (5.0%) Probability & Statistics4 (3.3%) Stochastic Analysis2 (1.7%) 17 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Table 12. Computer Science Domain Composition. Sub-DisciplineCount (%) Theoretical Computer Science43 (35.8%) Mathematical Foundations25 (20.8%) Logic in Computer Science12 (10.0%) Computability and Complexity Theory6 (5.0%) Artificial Intelligence and Machine Learning17 (14.2%) Bayesian and Probabilistic Methods5 (4.2%) Learning Theory and Models4 (3.3%) Specialized Domains and Applications4 (3.3%) Logic, Reasoning, and Knowledge Represen- tation 3 (2.5%) Robotics and Embodied AI1 (0.8%) Programming Languages and Formal Meth- ods 13 (10.8%) Formal Verification4 (3.3%) Automata and Formal Languages4 (3.3%) Type Theory and Program Analysis3 (2.5%) Formal Semantics and Rewriting Systems2 (1.7%) Scientific Computing13 (10.8%) Computational Physics and Engineering6 (5.0%) Numerical Methods5 (4.2%) Optimization and Control2 (1.7%) Cryptography12 (10.0%) Cryptosystems and Primitives4 (3.3%) Cryptanalysis4 (3.3%) Secure Protocols2 (1.7%) Foundational Concepts2 (1.7%) Algorithmic Theory11 (9.2%) Graph and String Algorithms3 (2.5%) Algorithmic Information Theory3 (2.5%) Data Structures and Compression2 (1.7%) Combinatorial Optimization2 (1.7%) Algorithmic Game Theory1 (0.8%) Quantum Computing11 (9.2%) Quantum Algorithms and Information4 (3.3%) Quantum Foundations and Simulation4 (3.3%) Quantum Complexity and Models2 (1.7%) Interdisciplinary Applications1 (0.8%) Table 13. Economics Domain Composition. Sub-DisciplineCount (%) Pluralist Econ34 (28.3%) Behavioral and Experimental Economics8 (6.7%) Public Economics and Social Choice Theory8 (6.7%) Marxian Economics7 (5.8%) Other Theories and Applied Fields6 (5.0%) Economic History and History of Thought5 (4.2%) Microeconomics19 (15.8%) Consumer and Firm Theory13 (10.8%) General Equilibrium and Market Design6 (5.0%) Econometrics15 (12.5%) Models and Methods11 (9.2%) Statistical Concepts4 (3.3%) Macroeconomics14 (11.7%) Economic Models and Growth8 (6.7%) Monetary Policy and Critiques6 (5.0%) Math/Comp Econ14 (11.7%) Models and Methods14 (11.7%) Game Theory9 (7.5%) Concepts and Models9 (7.5%) Finance9 (7.5%) Asset Pricing and Risk Management9 (7.5%) International Economics6 (5.0%) Trade and Finance6 (5.0%) 18 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving Table 14. Philosophy Domain Composition. Sub-DisciplineCount (%) Philosophy of Religion and Interdisciplinary31 (25.8%) History of Philosophy and Specific Texts10 (8.3%) Philosophy of Religion and Theology9 (7.5%) Other Interdisciplinary Concepts8 (6.7%) Philosophy of Language4 (3.3%) Logic and Philosophy of Mathematics20 (16.7%) Modal and Non-Classical Logics7 (5.8%) Logic in Computation and Language7 (5.8%) Set Theory and Mathematical Foundations6 (5.0%) Epistemology & Phil Sci18 (15.0%) Epistemology10 (8.3%) Philosophy of Science4 (3.3%) Philosophy of Physics4 (3.3%) Metaphysics & Ontology16 (13.3%) Core Concepts9 (7.5%) Theories of Reality4 (3.3%) Metaphysics of Mind3 (2.5%) Continental Philosophy13 (10.8%) Phenomenology and Existentialism5 (4.2%) Post-Structuralism and Deconstruction4 (3.3%) Lacanian Psychoanalysis4 (3.3%) Eastern Philosophy12 (10.0%) Indian Philosophy (Hindu Schools)6 (5.0%) Indian Philosophy (Buddhist Schools)5 (4.2%) Indian Religious/Philosophical Concepts1 (0.8%) Social Philosophy6 (5.0%) Legal and Social Philosophy5 (4.2%) Political Philosophy1 (0.8%) Mind4 (3.3%) Core Concepts3 (2.5%) Psychoanalytic Philosophy1 (0.8%) 19 Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving E. Active Probing Prompt Querying Agent Prompt You are adomainexpert. I will have a presetdomainname/term/concept in mind, and you need to continuously ask questions and reason until you can identify the name/term/concept. You can ask questions to narrow down the possibilities. Your questions must be in the form of a single, four-option multiple-choice question at a time. You can only ask one multiple-choice question at a time. Each question is crucial and should be based on your reasoning, professional knowledge, and the previous rounds of questions and answers. Please try to identify the term/concept in as few rounds as possible. I have now preset a name/term/concept. Please begin with your first question. Figure 7. The system prompt is for the querying agent. In each round, it poses strategic queries to quickly identify theP target by interacting with an environment. Thedomainplaceholder is replaced with the specific domain name (e.g., Human Disease, Physics) for each evaluation. Knowledge Oracle Prompt You are adomainexpert. You will have a presetdomainname/term/concept in mind, and I need to identify it by asking questions and using reasoning. I can ask questions to narrow down the scope, and my method of questioning will be a single, four-option multiple-choice question at a time. The specific details regarding my question are all within the<qaresponse>part. You will respond to my questions based on some āgolden informationā (reference material). The name/term/concept you have preset is: puzzle name Reference material: <referencematerial>puzzlereference </referencematerial> Inyourreply,youmustincludea<pureparse>...</pureparse>blockanda <pureanswer>...</pureanswer> block. For the <pureparse>...</pureparse> block: You will only judge the correctness of the options. For example, when option A, D are correct, and option B, C are incorrect: <pureparse>A is correct, B is incorrect, C is incorrect, D is correct</pureparse> When all options are incorrect, reply with<pureparse>A is incorrect, B is incorrect, C is incorrect, D is incorrect</pureparse> If the<qaresponse>part provides only an incorrect final guess without including a multiple- choicequestion,replywith<pureparse>Your guess is incorrect, please ask a new question</pure parse> and <pureanswer>None</pureanswer>. The<pureparse>block is forbidden from containing the name of the preset name/term/concept or any other content from the <referencematerial> part. For the <pureanswer>...</pureanswer> block: If the message you receive in the<qaresponse>part (or one of the options in the<qaresponse>part) explicitly states the name of your preset name/term/concept (or its alias), then include<pureanswer>Your preset name</pureanswer>in your reply (Here, āYour preset nameā is replaced with the actual name that appears). If it does not, then include <pureanswer>None</pureanswer>. All of your replies must include both of these blocks. Figure 8. The system prompt for the Knowledge Oracle. This agent acts as an oracle, providing structured and non-leaking feedback to the strategic agent based on the P target and I ref . 20