Paper deep dive
From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models
Si'an Xie, Jiaxun Liu, Biao Yang, Wei Yuan, Fan Yang, Tingting Gao, Ming Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/13/2026, 5:34:37 AM
Summary
This paper introduces MPAR-Bench, a bilingual benchmark designed to evaluate 'reasoning breadth' in Large Language Models (LLMs), defined as the ability to integrate multiple semantically diverse clues into a single coherent answer. Unlike traditional benchmarks focusing on reasoning depth (linear chains), MPAR-Bench uses a multi-agent pipeline to generate 1,000 items inspired by the game 'Just One'. The study evaluates various LLMs (GPT-5, Gemini, Qwen, etc.) under standard and perturbed conditions (clue masking, order shuffling, distractor injection, multi-step inference). Results indicate that while thinking modes improve accuracy, they do not consistently enhance robustness against perturbations, suggesting that current LLMs lack robust multi-point associative reasoning capabilities.
Entities (13)
Relation Signals (10)
MPAR-Bench â evaluates â Reasoning Breadth
confidence 95% ¡ We introduce MPAR-Bench, a bilingual English-Chinese benchmark that isolates reasoning breadth through multi-point associative reasoning.
MPAR-Bench â containsperturbation â Clue Masking
confidence 90% ¡ four perturbations: clue masking, order shuffling, distractor injection, and multi-step clues.
MPAR-Bench â containsperturbation â Order Shuffling
confidence 90% ¡ four perturbations: clue masking, order shuffling, distractor injection, and multi-step clues.
MPAR-Bench â containsperturbation â Distractor Injection
confidence 90% ¡ four perturbations: clue masking, order shuffling, distractor injection, and multi-step clues.
MPAR-Bench â containsperturbation â Multi-step Inferring
confidence 90% ¡ four perturbations: clue masking, order shuffling, distractor injection, and multi-step clues.
MPAR-Bench â inspiredby â Just One
confidence 90% ¡ Inspired by the cooperative game Just One, each item asks a model to recover a hidden target from several independently generated, semantically diverse clues.
Reasoning Breadth â isorthogonalto â Reasoning Depth
confidence 90% ¡ Reasoning breadth is thus an orthogonal axis, and one on which depth-oriented benchmarks provide little discrimination.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains. This progress primarily reflects reasoning depth. A complementary and comparatively unexamined capability is reasoning breadth: exploring multiple semantic directions in parallel and integrating the resulting clues into one coherent answer. We introduce MPAR-Bench, a bilingual English-Chinese benchmark that isolates reasoning breadth through multi-point associative reasoning. Inspired by the cooperative game Just One, each item asks a model to recover a hidden target from several independently generated, semantically diverse clues. We construct 1,000 items using a multi-agent clue-generation pipeline, embedding-based diversity filtering, and human verification. Only the answer space is drawn from public word lists, whereas every clue set is generated from scratch. Beyond exact-match accuracy, we evaluate models using accuracy, ANLS, embedding similarity, reasoning-trace verification, and four perturbations: clue masking, order shuffling, distractor injection, and multi-step clues. Across evaluated models, perturbations reduce accuracy by 9-18 percentage points in English and 5-12 percentage points in Chinese. Thinking mode improves standard-setting accuracy, especially in English, but does not consistently reduce sensitivity to perturbations. Case-level analysis also shows that extended reasoning can overturn an initially correct hypothesis. These results indicate that greater reasoning depth does not automatically confer robust reasoning breadth, and that reasoning breadth remains largely uncovered by current benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2608.10444v2
- Canonical: https://arxiv.org/abs/2608.10444v2
Trouble viewing inline? Open PDF directly â
Full Text
60,915 characters extracted from source content.
Expand or collapse full text
From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models Preprint, compiled August 13, 2026 Siâan Xie 1,â , Jiaxun Liu 2,â , Biao Yang 3 , Wei Yuan 3,â , Fan Yang 3 , Tingting Gao 3 , Ming Wu 1,â 1 Beijing University of Posts and Telecommunications 2 Peking University 3 Kuaishou Technology â Equal contribution. â Corresponding authors. Abstract Large language models (LLMs) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains. This progress primarily reflects reasoning depth. A complementary and comparatively unexamined capability is reasoning breadth: exploring multiple semantic directions in parallel and integrating the resulting clues into one coherent answer. We introduce MPAR-Bench, a bilingual (EnglishâChinese) benchmark that isolates reasoning breadth through multi-point associative reasoning. Inspired by the cooperative game Just One, each item asks a model to recover a hidden target from several independently generated, semantically diverse clues. We construct 1,000 items using a multi-agent clue-generation pipeline, embedding-based diversity filtering, and human verification. Crucially, only the answer space is drawn from public word lists, whereas every clue set is generated from scratch. Beyond exact-match accuracy, we evaluate models along a coarse-to-fine protocol (accuracy, ANLS, embedding similarity, and reasoning-trace verification) and a perturbation suite that probes four distinct robustness axes: clue masking, order shuffling, distractor injection, and multi-step clues. Across evaluated models, perturbations reduce accuracy by 9â18 points in English and 5â12 points in Chinese. Enabling thinking mode improves standard-setting accuracyâespecially in Englishâbut does not consistently reduce sensitivity to perturbations, and case-level trace analysis shows that extended reasoning can overturn an initially correct hypothesis. These results indicate that greater reasoning depth does not automatically confer robust reasoning breadth, and that breadth is a measurement axis that current benchmarks largely leave uncovered. Introduction Large language models (LLMs) have rapidly evolved from early neural language models to highly capable Transformer- based systems [1] such as GPT, Gemini, Qwen, and others [2,3,4,5,6,7,8,9]. Through techniques such as reinforcement learning [10], supervised finetuning [11], Chain-of-Thought (CoT) [12] and Retrieval-Augmented Generation [13], mod- ern LLMs have achieved remarkable success and demonstrated near-human performance in solving practical problems, which depends on their step-by-step linear reasoning. Humans possess another important capability: multi-point associative reasoning, which enables the structured integration of a wide range of con- cepts. This ability allows humans not only to explore problems in depth, but also to bridge existing concepts and synthesize them into novel ideas. These ability differences are shown in Fig. 1. An important question remains unresolved: do current LLMs possess the non-linear, cross-domain associative reason- ing abilities? Existing LLM benchmarks [14,15,16,17,18] primarily empha- size reasoning âdepth,â evaluating step-by-step logical deduction and procedural reasoning. In contrast, the evaluation of rea- soning âbreadthââthe ability to aggregate dispersed semantic signals and perform abstract conceptual convergenceâremains largely unexplored. This capability matters whenever the rele- vant evidence is distributed across different semantic perspec- tives rather than arranged as a single derivation. Multi-document synthesis, cross-domain analogy, hypothesis generation, and rea- Figure 1: Linear Reasoning and Multi-Point Associative Rea- soning Flow Chart. soning under incomplete or distracting evidence all require a model to hold several partial relations in view and reconcile them into a final prediction [19]. A model may therefore reason deeply along one path while still failing to combine information available across several paths. To fill this gap, we introduce MPAR-Bench, a cognitively in- spired benchmark designed to systematically evaluate multi- arXiv:2608.10444v2 [cs.CL] 12 Aug 2026 Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models2 Figure 2: A brief introduction of MPAR-Bench. point associative reasoning in LLMs. To improve evaluation reliability and robustness, we further construct high-quality bilin- gual test sets with 1,000 questions through a carefully designed multi-agent generation and verification pipeline [20, 21, 22]. Beyond exact-match metrics, we propose a fine-grained evalu- ation framework that analyzes model behavior from reasoning perspectives. This framework enables a more comprehensive investigation of how LLMs âreason broadâ, offering deeper in- sight into the current capabilities and limitations of associative reasoning in modern language models. A general overview of MPAR-Bench is shown in Fig. 2. In general, our contributions are summarized as follows: â˘A benchmark targeting reasoning breadth. MPAR- Bench operationalizes many-to-one integration from multiple, semantically diverse clues. Unlike RAT-style tests (three fixed compound-word cues) and existing game benchmarks (clue giving, or grouping a fixed word set), MPAR-Bench isolates the guesser-side in- tegration of an open number of free-form clues, and pairs it with a controlled perturbation suite. ⢠An innovative multi-agent clue-synthesis pipeline. We propose a multi-agent collaborative clue-synthesis framework with embedding-based filtering and human verification, enabling the construction of semantically diverse, high-difficulty evaluation instances while sub- stantially reducing memorization risk. â˘A coarse-to-fine evaluation protocol. Beyond exact match, we combine ANLS, embedding similarity, and reasoning-trace verification, and analyze robustness per perturbation type rather than as a single aggregate, exposing a blank space in reasoning breadth. Related Work LLM in Reasoning Depth The dominant trajectory of LLM reasoning research extends in- ferential depth. Chain-of-thought prompting [12], tree- and plan- structured search [23,24], self-verification [25], and process su- pervision [26] all lengthen or stabilize a single reasoning trajec- tory, and reinforcement-learned thinking modes push test-time computation further [27]. A parallel line of work documents the failure mode of overthinking, in which additional reasoning steps degrade rather than improve answers [28,29]. Depth-oriented benchmarksâmathematical [14,16], knowledge-intensive [15], and broad-coverage suites [17,18]âare increasingly saturated for frontier models. These tasks answer how far a model can push one chain; they do not answer whether a model can in- tegrate evidence across chains. Reasoning breadth is thus an orthogonal axis, and one on which depth-oriented benchmarks provide little discrimination. Associative Reasoning Associative reasoning has long been studied in cognitive psy- chology through convergent-thinking instruments, most notably the Remote Associates Test (RAT) [30]. RAT has recently been repurposed to probe LLMs: Schon et al.[31]model associative reasoning processes, Kumar et al.[32]study humanâAI conver- gent and divergent thinking, and generative models have been reported to match or exceed humans on such tests [33,34]â though such results are hard to interpret, since the test items are publicly available and may have been seen during pre-training [35, 36]. Related open-ended formulations argue for process-based rather than multiple-choice evaluation [37] and for generating explicit Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models3 Figure 3: Introduction of Board Game Just One. associative paths [38]. We inherit the construct of convergent association from this tradition but deliberately depart from its instrument: RAT items are public, largely three-cue, and dom- inated by fixed phrasal collocations that next token prediction learns readily. MPAR-Bench instead poses an open, variable number of free-form semantic clues that must be integrated, with every clue set synthesized de novo to substantially re- duce overlap with public clueâtarget pairings and lower the risk of memorization. This is what MPAR-Bench adds over prior RAT-on-LLM evaluations: a breadth-oriented benchmark with reduced contamination risk, rather than a re-run of a public convergent-thinking test. Boardgame-Based Benchmarks Cooperative and word-association games offer constrained rules with large state spaces, which helps mitigate contamination and yields human-aligned semantic tasks. Codenames has been used to evaluate one-to-many clue giving and ad-hoc concept forming [39,40]; the NYT Connections game requires partitioning a fixed set of words into latent groups [41]; and the Word Synchro- nization Challenge measures two agents converging on a shared word without communication [42]. MPAR-Bench differs along three axes: (i) clue cardinalityâmodels must jointly integrate an open, variable number of clues rather than fixed cues; (i) association typeâclues are free-form semantic descriptions spanning lexical, cultural, phonetic, and world-knowledge rela- tions, not compound-word completions or fixed candidate pools; and (i) item availabilityâall clue sets are synthesized from scratch, leaving no public clueâtarget pairing to memorize. Methodology Task Definition Given a clue setC=c 1 , c 2 ,..., c n , the task is to recover a targetysuch that each cluec i contributes an independently informative semantic relation toy. Reasoning breadth, in this setting, is the ability to integrate multiple semantically distinct and non-redundant clues into a single coherent answer. Figure 4: Word Cloud of MPAR-Bench. We ensure that each item genuinely requires breadth through two construction-side safeguards. First, clues are generated from diverse semantic angles to maximize the range of associations a model must reconcile. Second, a judge agent and an embedding- based filter remove synonyms, paraphrases, and near-duplicates, so that each retained clue carries non-overlapping information. At evaluation time, we measure not only whether a modelâs prediction matches the target, but also whether that prediction re- mains stable under perturbationâclue masking, order shuffling, distractor injection, and multi-step inferenceâwhich probes whether the integration is robust or merely superficial. Just One The design borrows the constraint structure of the coopera- tive game Just One, in which players give single-word hints to help a guesser infer a hidden target, while direct synonyms, translations, homophones, and duplicate clues are forbidden. These constraints are what make the game a clean instrument for breadth. Rather than rewarding the most obvious lexical association, they force clue writers to approach the target from distinct, indirect angles; the guesser must then integrate frag- mented, non-overlapping signals rather than pattern-match a single cue. This yields a constrained multi-point associative reasoning task emphasizing semantic abstraction, conceptual bridging, and integration. An illustrative round is shown in Fig. 3. Dataset Construction Answer space. Target words are drawn from public word listsâ RAT-derived vocabulary (collected on the internet) and Just One word cards. Multi-agent clue generation. Given a target and the clues al- ready accepted, LLM-based agents iteratively propose new clues that remain semantically relevant to the target while minimizing redundancy with existing clues. Each agent is assigned a dis- tinct association angle to encourage coverage across semantic directions. A judge agent then removes clues that are the answer itself, direct synonyms/translations/homophones/morphological variants, exact or near-duplicates of accepted clues, or genuinely low quality [43,44]. This division of labor mirrors the indepen- dent clue-provider and arbiter structure of Just One. Notably, all agentsâ prompt are provided in Appendix. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models4 Category Count Rate 95% Wilson CI Unique232 92.8% [88.6%, 95.4%] Ambiguity187.2% [4.6%, 11.1%] Table 1: Answer uniqueness with 95% Wilson CI Embedding-based diversity filtering.We use Qwen3- Embedding-8B [45] to score clueâanswer and clueâclue similar- ity, discarding clues that are either trivially close to the answer or too weakly related to be informative, and clue pairs that are near-duplicates [46]. An experimental embedding similarity threshold ranges from 0.3 to 0.8, which serves as a primary filtering step in benchmark construction. Answer uniqueness and graded acceptability. A central con- cern is that a clue set may admit more than one reasonable target. We address this in two ways. (i) Construction: the judge stage filters out clue sets that jointly under-determine the target, and problematic items are reconstructed. (i) Human verification: on a randomly sampled subset of 250 items, two masterâs students majoring in NLP independently assess whether each item admits a unique, unambiguous target. Table 1 reports the results with 95% Wilson CI, indicating that 92.8% of the items were judged as having a unique answer. Bilingual design. MPAR-Bench includes English and Chinese subsets built with the same pipeline (500 validated items each), combining synthesized and native-speaker-authored samples. The English subset emphasizes lexical and abstract associations; the Chinese subset additionally incorporates idioms, character- level and pictographic properties, and contemporary cultural memes. Word clouds for each subset are shown in Fig. 4. Benchmark rationale. MPAR-Bench is intentionally con- structed to emphasize long-range, multi-point association rather than lexical overlap or frequent collocations. By synthesizing clues from complementary perspectives while enforcing low redundancy, each item requires integrating sparse and seman- tically distant evidence into a single target, making successful prediction less dependent on next token co-occurrence patterns and more on associative reasoning. Difficulty Settings and Perturbation We distinguish two complementary facets of reasoning breadth. The Standard setting measures baseline breadth: given com- plete, well-formed clues under ideal conditions, can the model integrate multiple semantic signals into a correct answer? To further evaluate whether this integration capability remains reli- able under more realistic conditions, we introduce an Enhanced setting that systematically perturbs the standard test protocol to simulate information-restricted or noisy environments. These two settings provide a complete picture: the Standard setting establishes what a model can achieve under favorable conditions, while the Enhanced setting reveals whether that capability is resilient enough to matter in practice [47]. Specifically, we implement the following enhanced transforma- tions and perturbations: ⢠Clue Masking: Randomly masking clues to evaluate model reasoning ability under information deficiency. Clue Masking Standard â TobleroneSolitaire Kidney Pascal Python Pyramid â Clue Masking â SolitairePascal Python Computer â Order Shuffling Standard â TurnGold Spring NodeCanada Leaf â Order Shuffling â TurnNodeGoldCanada Spring Maple â Multi-step Inferring Standard â PoisonVectorCRISPRPandora Payload Virus â Multi-step Inferring â PoisonArrowScissorsPandora Payload Box â Distractors Standard â GallandRobinThree AgrabahRing Aladdin â Distractors â Agrabah GallandHoodThree MusketeersRobin Ring Thieves â Figure 5: Examples of Enhanced settings in MPAR-Bench. â˘Order Shuffling: Shuffling the clues order to find whether the modelâs reasoning process is sensitive to order. ⢠Distractors: Injecting semantically misleading or ir- relevant cue words to test model resistance to noisy contexts and spurious correlations. â˘Multi-step Inferring: Increasing the associative se- mantic distance between clues and the mystery word, forcing models to generate intermediate latent con- nections rather than relying on direct surface co- occurrence. Each enhanced setting evenly distributes words from the stan- dard task. We refer to this setting as the Enhanced MPAR-Bench in the remainder of the paper. As shown in Fig. 5, each question has a corresponding enhanced variant, posing a greater challenge to LLMs across multiple dimensions of robustness. Evaluation Associative reasoning cannot be fully captured by a single exact- match metric, since semantically reasonable predictions may differ lexically from the ground truth, and some correct predic- tions may come from flawed reasoning processes. To obtain a more comprehensive understanding of LLM behavior, we evalu- ate models at three progressively finer granularities: accuracy, word-based similarity, and the validity of reasoning trace, i.e., the explanation of how the answer and clues are connecting with each other. Accuracy We first evaluate model performance using exact-match accu- racy. A prediction is considered correct if and only if it exactly matches the ground-truth answer. We report accuracy across different subsets, including multilingual Standard and Enhanced Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models5 Model English Chinese Acc ANLS Emb TraceAcc ANLS Emb Trace GPT-5.277.6 0.792 0.848 0.93364.4 0.740 0.823 0.954 Gemini-3.1pro 86.8 0.884 0.915 0.94172.2 0.802 0.869 0.957 Sonnet-4.579.0 0.803 0.856 0.86567.4 0.767 0.839 0.927 Qwen3-max73.2 0.743 0.800 0.878 65.0 0.733 0.813 0.928 Kimi-k271.6 0.729 0.795 0.87157.4 0.675 0.770 0.888 Deepseek-v3.2 69.8 0.716 0.784 0.84861.2 0.686 0.793 0.893 Seed-2-pro71.4 0.724 0.802 0.87564.6 0.739 0.820 0.937 Table 2: Standard MPAR-Bench Results on Thinking Models Model EnglishChinese Acc ANLS Emb TraceAcc ANLS Emb Trace GPT-5.266.6 0.681 0.764 0.90258.8 0.669 0.772 0.932 Gemini-3.1pro 76.9 0.778 0.842 0.906 64.0 0.722 0.815 0.926 Sonnet-4.563.4 0.646 0.739 0.83661.0 0.689 0.789 0.873 Qwen3-max56.8 0.591 0.679 0.84154.8 0.621 0.732 0.911 Kimi-k257.6 0.596 0.687 0.82745.8 0.547 0.683 0.850 Deepseek-v3.2 52.0 0.531 0.639 0.79452.4 0.596 0.721 0.842 Seed-2-pro56.6 0.579 0.682 0.81758.2 0.666 0.768 0.887 Table 3: Enhanced MPAR-Bench Results on Thinking Models settings, to measure modelsâ associative retrieval capability un- der varying semantic and linguistic conditions. Word-based Evaluation To further evaluate semantic proximity between predictions and target concepts, we adopt both Average Normalized Levenshtein Similarity (ANLS) and word embedding similarity to judge at the semantic level. ANLS [48] is computed using normalized Levenshtein edit dis- tance: ANLS( Ëy, y) = 1â d lev (Ëy, y) max(| Ëy|,|y|) ,(1) whereËyandydenote the model prediction and ground truth respectively,d lev (¡,¡) denotes the Levenshtein edit distance, and |¡| denotes string length. Moreover, we compute word embedding similarity using fastText [49] as a word embedding model: Sim(Ëy emb , y emb ) = Ëy ⤠emb y emb ⼠Ëy emb ⼠2 âĽy emb ⼠2 .(2) whereËy emb andy emb denote the word embedding of the model prediction and ground truth in fastText respectively, andâĽÂˇâĽ 2 denotes their Euclidean (â 2 ) norms. Reasoning Trace Evaluation Beyond final-answer accuracy, we assess the validity of inter- mediate processes via reasoning trace evaluation, decomposed into two dimensions: logical verification and factual verification [50,25,26]. Logical verification examines whether reasoning trajectories follow coherent inferential steps from clues to pre- dictions, while factual verification checks whether intermediate claims are factually grounded. This dual assessment distin- guishes valid associative reasoning from spurious correlations Model English Chinese Acc ANLS Emb TraceAcc ANLS Emb Trace GPT-5.259.6 0.614 0.705 0.81161.8 0.718 0.804 0.943 Gemini-3flash 70.0 0.717 0.787 0.83167.0 0.765 0.836 0.920 Sonnet-4.570.4 0.716 0.791 0.83868.8 0.776 0.843 0.921 Qwen3-max55.4 0.572 0.669 0.792 64.4 0.748 0.820 0.930 DeepSeek-v3.2 51.4 0.536 0.644 0.73060.6 0.702 0.788 0.906 Seed-2-pro59.8 0.619 0.711 0.74364.2 0.726 0.817 0.910 Table 4: Standard MPAR-Bench Results on Non-thinking Mod- els Model EnglishChinese Acc ANLS Emb TraceAcc ANLS Emb Trace GPT-5.244.2 0.460 0.586 0.75454.4 0.632 0.749 0.906 Gemini-3flash 56.0 0.580 0.675 0.76361.4 0.695 0.791 0.870 Sonnet-4.555.8 0.572 0.679 0.77462.0 0.707 0.797 0.865 Qwen3-max44.8 0.464 0.586 0.75156.0 0.662 0.758 0.874 Deepseek-v3.2 42.6 0.444 0.566 0.685 54.0 0.618 0.738 0.851 Seed-2-pro43.4 0.452 0.586 0.693 56.0 0.646 0.761 0.866 Table 5: Enhanced MPAR-Bench Results on Non-thinking Mod- els #1 EN strawberry fuzzytakegutMario peachtennis #2 EN PETDesertNurserySymphonyPacific oceanbottle #3 EN BoardEchoFourVolumeCat speakerroom #4 ZH čč˘ head duang boing éžçäź äşş Dragon's heir çť fine éŁć Rejoice 头ĺ hair éťĺ black hair #5 ZH ç˝ćźčŻşĺ¤Ť Romanov ĺ头隰 two-headed eagle 輿䟯ĺŠäş Siberia 彟ĺžĺ¤§ĺ¸ Peter the Great ĺŹĺŽŤ Winter Palace äżç˝ćŻ Russia ć˛ç Tsar #6 ZH ĺšść side by side 厢ĺ č§č˝ living-room corner ç§čŻ whisper 软座 soft seat ĺ ąäşŤ shared ć˛ĺ sofa ĺäşşć˛ĺ loveseat Figure 6: Challenging Cases in MPAR-Bench. Examples where most models fail to identify the correct answer. and hallucinated paths. We manually review a randomly sam- pled subset of 300 reasoning trace predictions, confirming high consistency (98.7% on factual verification, 94.7% on logical verification) between human and LLM judgement. Evaluation prompts are provided in Appendix. Experiments and Results Implementation Details We benchmark a diverse set of representative LLM familiesâ including GPT, Gemini, Sonnet, Qwen, Kimi, DeepSeek, and Seed series [2,3,4,5,6,7,8,9]; the concrete information is shown in Appendix. Evaluations are conducted across both thinking and non-thinking modes, under standard and enhanced settings, on the bilingual subsets of MPAR-Bench. We also report fine-grained metrics including ANLS, word em- bedding similarity, and reasoning trace failure analysis to dissect the reasoning behaviors of LLMs from lexical, semantic, and reasoning process perspectives. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models6 Main Results We analyze the results from three key perspectives: cross-model comparison, the effect of thinking mode, and robustness under perturbation. Challenging cases in MPAR-Bench are shown in Fig. 6, and more detailed experimental results are provided in Appendix. Model Comparisons.Table 2 and Table 4 report the standard MPAR-Bench results under thinking and non-thinking modes. Under the thinking mode, Gemini-3.1pro leads on both En- glish (86.8%) and Chinese (72.2%), followed by GPT-5.2 and Sonnet-4.5 in English. Under the non-thinking mode, Sonnet-4.5 achieves the highest accuracy in both languages. Standard vs. Enhanced. Comparing standard (Tables 2, 4) with enhanced settings (Tables 3, 5) reveals consistent degra- dation under perturbation. For instance, in thinking mode, Deepseek-v3.2 exhibits the largest decline in English, while Kimi-k2 drops the most in Chinese. In non-thinking mode, Seed- 2-pro shows the largest decline in English, whereas Qwen3-max drops the sharpest in Chinese. These contrasts indicate that robustness varies substantially across models: some suffer pro- nounced degradation under perturbation while others remain comparatively stable. Notably, this robustness is strongly model- and language-dependent. Detailed information and analyses are shown in Appendix. Thinking vs. Non-Thinking. Comparing thinking-mode re- sults (Tables 2, 3) against their non-thinking counterparts (Ta- bles 4, 5), we find that thinking mode [26,27] consistently im- proves most indicators, but the magnitude of the gain is markedly larger on English than on Chinese: averaged across models, thinking lifts English accuracy by a substantially wider margin and produces clear, stable gains for every model, whereas its effect on Chinese is much smaller and model-dependent (Sonnet- 4.5 even shows a slight regression), and the improvement under perturbation is non-monotonic. Moreover, thinking mode is not always reliable: models occasionally overthink and override cor- rect intermediate answers, as discussed in the next section and Appendix. This suggests a crucial distinction: while thinking mode improves reasoning depth, it may not necessarily enhance reasoning breadth. Discussion The benchmark results above establish that current LLMs ex- hibit measurable but imperfect reasoning breadth, and that this capability degrades under perturbation. A natural follow-up question is: under what conditions does reasoning breadth im- prove, and what mechanisms cause it to fail? We investigate this question through four complementary lenses. First, we examine overthinking, a failure mode in which extended reasoning ac- tively harms breadth by overriding correct intermediate answers. Second, we characterize information gain curve, asking how breadth scales as more clues become available. Third, we ex- plore scaling laws to determine whether larger models inherently develop broader reasoning. Fourth, we test whether semantic feedback can steer models toward correct answers across multi- ple refinement rounds. We additionally try to create a structured Model English Chinese Wrong Ex. Ans. Mention Token Len. Wrong Ex. Ans. Mention Token Len. Sonnet-4.51050.4570.3711630.7060.571 Qwen3-max1340.5900.5151740.8740.690 Kimi-k21420.5850.415 2130.8640.366 Deepseek-v3.2 1510.6230.5101940.8300.634 Seed-2-pro1430.4970.4061770.7510.599 Table 6: Overthinking Results on models. 2345 Context Size (Number of Words) 35 40 45 50 55 60 65 70 Accuracy (%) Accuracy vs. Size Chinese English 2â33â44â5 Context Size Transition 2 4 6 8 10 12 14 16 Absolute Gain (%) Accuracy Gain Chinese English Figure 7: Information Gain Curve of Seed-2-pro. As number of words increases, accuracy rises but at a decreasing rate. reasoning skill as an intervention strategy; results are reported in Appendix. Overthinking A notable failure pattern we observe is overthinking [28,29]: models initially arrive at the correct answer but subsequently override it during extended reasoning, often drifting toward a semantically related but incorrect concept. This behavior is particularly pronounced in Qwen3-max and Kimi-k2. For instance, given the answer word Philosophy, the model outputs Plato, over-focusing on a representative entity implied by the clues rather than the academic discipline itself. Table 6 reports a detailed overthinking analysis. âWrong Ex.â is the total count of incorrect predictions. âAns. Mentionâ (ratio) is the proportion of those incorrect cases in which the correct answer appeared in the reasoning trace but was subsequently overridden. âToken Len.â (ratio) is the proportion of incorrect cases whose reasoning length exceeded the modelâs average reasoning length on correct samples. Information Gain Curve The information gain curve characterizes how a modelâs reason- ing accuracy scales with the number of provided clue words; we utilize it to evaluate Seed-2-proâs capacity to leverage incremen- tal semantic evidence for multi-point associative reasoning. As shown in Fig. 7, accuracy consistently improves as the number of clue words increases, suggesting that the model is able to ac- cumulate and integrate incremental semantic information across multiple clues. This provides preliminary evidence that LLMs possess multi-point associative reasoning capabilityâthe ability to jointly combine several semantically distinct clues into a co- herent answer. However, the marginal gain progressively slows down, indicating that while models benefit from richer semantic context, they saturate beyond a certain evidence threshold. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models7 0.6B1.7B4B8B14B32B Model Parameters 0.0 0.1 0.2 0.3 0.4 0.5 Performance Score Accuracy 0.6B1.7B4B8B14B32B Model Parameters 0.0 0.1 0.2 0.3 0.4 0.5 0.6 ANLS 0.6B1.7B4B8B14B32B Model Parameters 0.2 0.3 0.4 0.5 0.6 0.7 FastText Chinese (Standard)Chinese (Enhanced)English (Standard)English (Enhanced) Figure 8: Scaling Law of Qwen3 family in Chinese MPAR- Bench. As model scales up, all metrics improve, except for Qwen3-32B, which has a more severe overthinking issue. Scaling Law We investigate how model scale affects performance on our benchmark. In particular, Fig. 8 reports the scaling behavior of the locally deployed Qwen3 family in thinking mode (0.6B to 32B parameters), evaluated on accuracy, ANLS, and fastText embedding similarity across both English and Chinese datasets under Standard and Enhanced settings. Except for Qwen3-32B, accuracy consistently improves with increasing model size [51, 52,53]. Case-level inspection finds that Qwen3-32B suffers from overthinking, causing it to reject correct answers during an extended reasoning process. Feedback The proposed feedback method iteratively delivers semantic similarity metrics (ANLS and average word embedding sim- ilarity), guiding the model to refine its responses toward the answer across multiple rounds. Fig. 9 visualizes the correspond- ing trajectories (rounds 3â6) where the mechanism successfully corrected Qwen3-maxâs outputs. The results show that LLMs are not fully sensitive to word embedding similarity as a guid- ance signal; instead, they rely on other internal strategies. While feedback provides opportunities for exploration and sometimes enables recovery of the answer, the revision trajectory remains weakly aligned with semantic indicators, suggesting that models do not naturally exploit surface-level semantic proximity for iterative refinement. Conclusion We introduced MPAR-Bench, a bilingual benchmark that evalu- ates multi-point associative reasoningâreasoning breadthâin LLMs through 1,000 boardgame-rule-based questions and a coarse-to-fine evaluation protocol spanning accuracy, ANLS, embedding similarity, and reasoning-trace verification, comple- mented by a four-axis perturbation suite. Our experiments yield three findings. First, reasoning breadth re- mains far from solved: the best models reach 86.8%/72.2% accu- racy in English/Chinese, with perturbations causing 9â18/5â12 point drops. Second, greater reasoning depth does not automati- cally confer breadth: thinking mode improves standard-setting accuracy but does not consistently reduce perturbation sensi- tivity, and case-level analysis reveals that extended reasoning can override correct answers through overthinking. Third, im- proving breadth appears challenging: scaling model size, adding reasoning strategies, and iterative feedback each bring only par- R 0R 1R 2 Feedback Round 0.0 0.2 0.4 0.6 0.8 1.0 Cosine Similarity Solved in 3 Rounds (N=21) Average Trend R 0R 1R 2R 3 Feedback Round 0.0 0.2 0.4 0.6 0.8 1.0 Solved in 4 Rounds (N=15) Average Trend R 0R 1R 2R 3R 4 Feedback Round 0.2 0.4 0.6 0.8 1.0 Cosine Similarity Solved in 5 Rounds (N=5) Average Trend R 0R 1R 2R 3R 4R 5 Feedback Round 0.0 0.2 0.4 0.6 0.8 1.0 Solved in 6 Rounds (N=10) Average Trend Figure 9: Chinese MPAR-Bench feedback results for Qwen3- max in thinking mode, rounds 3â6. Blue lines show individual cases; green lines show average trends. tial gains, suggesting that reasoning breadth may be a capability that current training paradigms do not naturally optimize for. We release MPAR-Bench and its pipeline to encourage the com- munity to move beyond depth-oriented evaluation and toward a more complete picture of reasoningâone that values breadth as much as depth. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models8 References [1]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ĺukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [2] OpenAI.Gpt-5.2.https://openai.com/index/ introducing-gpt-5-2/, 2025. [3]Google.Gemini3flash:frontierintelli- gence built for speed.https://blog.google/ products-and-platforms/products/gemini/ gemini-3-flash/, 2025. [4] Google.Gemini 3.1 pro:A smarter model for your most complex tasks.https://blog.google/ innovation-and-ai/models-and-research/ gemini-models/gemini-3-1-pro/, 2026. [5]Anthropic. Introducing claude sonnet 4.5.https://w. anthropic.com/news/claude-sonnet-4-5, 2025. [6]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [7]Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chao- fan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. [8]Yifan Bai, Yiping Bao, Y Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. [9] Bytedance. Seed 2.0.https://seed.bytedance.com/ en/blog/seed-2-0-official-launch, 2026. [10]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Ja- cob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow in- structions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 27730â27744, 2022. [11]Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero- shot learners. In International Conference on Learning Representations (ICLR), 2022. [12]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits rea- soning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824â24837, 2022. [13]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich KĂźt- tler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9459â9474, 2020. [14]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021. [15]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. In Inter- national Conference on Learning Representations (ICLR), 2021. [16]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [17]Aarohi Srivastava et al. Beyond the imitation game: Quan- tifying and extrapolating the capabilities of language mod- els. Transactions on Machine Learning Research (TMLR), 2023. [18]Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic evaluation of language models. Transactions on Machine Learning Research (TMLR), 2023. [19]Johannes Treutlein, Dami Choi, Jan Betley, Sam Marks, Cem Anil, Roger Grosse, and Owain Evans. Connecting the dots: Llms can infer and verbalize latent structure from disparate training data. Advances in Neural Information Processing Systems, 37:140667â140730, 2024. [20]Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Com- municative agents for âmindâ exploration of large language model society. In Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 36, 2023. [21]Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Ha- jishirzi. Self-instruct: Aligning language models with self- generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 13484â13508, 2023. [22]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenen- baum, and Igor Mordatch. Improving factuality and rea- soning in language models through multiagent debate. In Proceedings of the 41st International Conference on Ma- chine Learning (ICML), 2024. [23]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 36, 2023. [24]Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reason- ing by large language models. In Proceedings of the 61st Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models9 Annual Meeting of the Association for Computational Lin- guistics (ACL), pages 2609â2634, 2023. [25]Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. In Findings of the Association for Computational Linguis- tics: EMNLP, pages 2550â2575, 2023. [26]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Ed- wards, Bowen Baker, Teddy Lee, Jan Leike, John Schul- man, Ilya Sutskever, and Karl Cobbe. Letâs verify step by step. In International Conference on Learning Representa- tions (ICLR), 2024. [27]DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning ca- pability in LLMs via reinforcement learning. Nature, 2025. [28]Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of o1-like LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. [29] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop over- thinking: A survey on efficient reasoning for large lan- guage models. arXiv preprint arXiv:2503.16419, 2025. [30] Sarnoff Mednick. The associative basis of the creative process. Psychological review, 69(3):220, 1962. [31]Claudia Schon, Ulrich Furbach, and Marco Ragni. Mod- eling associative reasoning processes. arXiv preprint arXiv:2201.00716, 2022. [32] Harsh Kumar, Jonathan Vincentius, Ewan Jordan, and Ash- ton Anderson. Human creativity in the age of llms: Ran- domized experiments on divergent and convergent think- ing. In Proceedings of the 2025 CHI conference on human factors in computing systems, pages 1â18, 2025. [33] Astrid Carolus, Martin J Koch, and Shuyan Feng. Time- on-task and instructions help humans to keep up with ai: replication and extension of a comparison of creative per- formances. Scientific reports, 15(1):20173, 2025. [34]Vikram Arora, Alex Thabane, Sameer Parpia, Goran Calic, and Mohit Bhandari. Generative artificial intelligence models outperform students on divergent and convergent thinking assessments. Scientific Reports, 15(1):36987, 2025. [35]Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. Unveiling the spectrum of data contamination in language model: A survey from detection to remediation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16078â16092, 2024. [36] Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhong- wei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, et al. Benchmarking large language models un- der data contamination: A survey from static to dynamic evaluation. In Proceedings of the 2025 Conference on Em- pirical Methods in Natural Language Processing, pages 10091â10109, 2025. [37]Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, et al. Mm-opera: Benchmarking open- ended association reasoning for large vision-language mod- els. arXiv preprint arXiv:2510.26937, 2025. [38]Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li, and Greg Durrett. Create: Testing llms for associative creativity. arXiv preprint arXiv:2603.09970, 2026. [39]Matthew Stephenson, Matthew Sidji, and BenoĂŽt Ronval. Codenames as a benchmark for large language models. IEEE Transactions on Games, 2025. [40] Sherzod Hakimov, Lara Pfennigschmidt, and David Schlangen. Ad-hoc concept forming in the game code- names as a means for evaluating large language models. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 728â740, 2025. [41]Tim Merino, Sam Earle, Ryan Sudhakaran, Shyam Sud- hakaran, and Julian Togelius. Making new connections: Llms as puzzle generators for the new york timesâ connec- tions word game. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertain- ment, volume 20, pages 87â96, 2024. [42]Tanguy Cazalets and Joni Dambre. Word synchronization challenge: A benchmark for word association responses for large language models. In International Conference on Human-Computer Interaction, pages 3â19. Springer, 2025. [43]Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven syn- thetic data generation, curation, and evaluation: A survey. In Findings of the Association for Computational Linguis- tics: ACL 2024, pages 11065â11082, 2024. [44]Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel FernĂĄndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238â255, 2025. [45] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Day- iheng Liu, Junyang Lin, et al. Qwen3 embedding: Ad- vancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. [46] Amro Abbas, Kushal Tirumala, DĂĄniel Simig, Surya Gan- guli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023. URL https://arxiv. org/abs/2303.09540, 2021. [47]Guangxiang Zhao, Saier Hu, Xiaoqi Jian, Jinzhu Wu, Yuhan Wu, Lin Sun, and Xiangzheng Zhang. Stress test- ing generalization: How minor modifications undermine large language model performance. arXiv e-prints, pages arXivâ2502, 2025. [48]Ali Furkan Biten, Rubèn Tito, Andres Mafla, Lluis Gomez, Marçal RusiĂąol, Ernest Valveny, C. V. Jawahar, and Di- mosthenis Karatzas. Scene text visual question answering. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models10 In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4291â4301, 2019. [49]Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computa- tional Linguistics, 5:135â146, 2017. ISSN 2307-387X. [50]Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Mea- suring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. [51]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [52]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large lan- guage models. In Advances in Neural Information Process- ing Systems (NeurIPS), volume 35, pages 30016â30030, 2022. [53]Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. Limitations While this benchmark provides an initial step toward evaluat- ing the multi-point associative reasoning capabilities of LLMs, there remains some room to further broaden its coverage and ecological validity. Future work may extend the benchmark with more diverse and interactive settings to better capture associative reasoning behaviors that arise in real-world environments. In addition, we hope this benchmark can serve as a foundation for systematically studying non-linear associative and creative capa- bilities in LLMs, as well as for developing evaluation protocols and modeling principles that more closely align with practical applications of multi-point associative reasoning. Ethical Considerations MPAR-Bench is built upon lexical materials sourced from cog- nitive psychology (the Remote Associates Test) and cooperative gaming (Just One). Throughout dataset construction, we adopted a series of safeguards to preempt potential ethical risks. The benchmark consists solely of word-level clues and semantic as- sociations, carrying no personally identifiable information and requiring no collection of human subject data. All clues were either drawn from publicly available game corpora or synthe- sized through an LLM-based multi-agent pipeline, after which embedding-based filtering and manual verification were applied to exclude content that could introduce demographic, cultural, or other unintended biases, in accordance with established eth- ical standards. The bilingual design of our benchmark further ensures equitable treatment of English and Chinese linguistic contexts, with dedicated attention to cultural appropriateness within each subset. Beyond its primary evaluation purpose, MPAR-Bench also serves as a diagnostic instrument for prob- ing model reasoning behaviors and uncovering failure modes, thereby supporting the broader goal of identifying and mitigating ethical vulnerabilities in deployed AI systems. Multi-agent Generation Prompts Prompts are used in English and Chinese in generating, evaluat- ing, and judging models separately. Questioner Prompt You are playing the board game âJust Oneâ. Secret answer: word Requirements: 1. Give exactly 1 English clue word. 2. The clue must be a common English word or a widely recognized proper noun. 3. The clue must not be the answer itself, a translation, a synonym, a homophone, a made-up word, an obvious morpho- logical variant, or contain the answer as a substring. 4. The clue should be indirect and moderately difficult, but useful when combined with other clues. 5. You MUST approach the answer from this specific associa- tion angle: angle Judger Prompt You are the judge for the board game âJust Oneâ. Secret answer: word Already approved clues (LOCKED - do NOT remove or re- evaluate these): locked_clues New candidate clues to evaluate: clue_list. Your tasks (apply only to the NEW candidates above): 1. Remove any new clue that IS the answer itself, a direct syn- onym, a translation, a homophone, an obvious morphological variant, or contains the full answer as a substring. 2. Remove EXACT duplicates among the new candidates. 3. Remove any new clue that refers to the SAME specific con- cept, entity, or phrase as an already-approved clue or another new candidate. Clues that merely belong to the same broad category (e.g., two different fictional characters, two different countries) are NOT duplicates - keep both. 4. Remove genuinely LOW-QUALITY new clues: completely obscure, made-up, grammatically wrong, or with no logical connection to the answer. Do NOT remove a clue just because it requires one step of reasoning. 5. Do NOT remove a clue simply because it seems âtoo directâ unless using it would immediately give away the answer with zero reasoning required. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models11 Player Prompt You are the one who guess the mystery word for âJust Oneâ. Based on the clue words given, guess the mystery word and explain the logical connection between each related clue word and the mystery word. The clue words are âclue_word1â, âclue_word2â, âclue_word3â Requirements: 1. The mystery word is only one word. 2. Provide the connection between each clue word and the mystery word. Evaluation Prompt You are a rigorous AI logic auditor. Your task is to evaluate each reasoning step produced by an Agent in a word-guessing game. Break the reasoning into independent steps (atoms) and judge each atom on two dimensions: âFactual Accuracyâ and âLogi- cal Soundnessâ. For each step: 1. Fact_Check: Are the objective claims in this step (numbers, ingredients, mechanisms, historical origins, etc.) factually correct? (Pass / Fail) 2. Logic_Check: Is the reasoning chain from the clue to the predicted answer natural and sound? Are there signs of over-generalization, edge-case pandering, or multi-layer rein- terpretation? (Pass / Fail) Judging Principles (you MUST follow these): - Scrutinize specific claims (numbers, ingredients, mechanisms, definitions) â any factual error â Fact Fail - If a step cherry-picks a marginal meaning of the clue to fit the answer, ignoring a more obvious association â Logic Fail - If the clue clearly points to a more specific/precise word, but the step generalizes it to a broad concept â Logic Fail - If the step requires more than two layers of inference or reinterpretation to connect the clue to the answer â Logic Fail - Only when the reasoning feels natural, direct, and requires no mental gymnastics for an ordinary person should it be judged Logic Pass Asset Distribution and Compliance To ensure reproducibility, both the English and Chinese subsets of MPAR-Bench, along with our evaluation scripts, will be publicly released under the MIT License upon publication. Our evaluation items are generated via frontier LLM APIs (e.g., GPT, Gemini, Qwen), and we have verified that our pipeline adheres to the respective terms of service of these model providers, restricting the usage of our dataset strictly to non-commercial academic benchmarking. Experiment Setup For all evaluated models we keep the official default sampling parameters specified in each providerâs release. For reason- ing models, we additionally set reasoning_effort=high where the API supports it. We do not perform any per-model hyper- parameter tuning. Detailed model configurations for all experi- ments are provided in Table 7. And detailed software versions are provided in Table 8. Detailed Results Token Usage Tables 9 and 10 report the average token consumption of mod- els operating in thinking mode, encompassing prompt tokens, Chain-of-Thought reasoning tokens, and output tokens. Token consumption increases consistently from the Standard to the Enhanced setting for nearly all models, reflecting the greater reasoning demand imposed by the four perturbation conditions. The magnitude of this increase, however, varies markedly across models, pointing to fundamentally different strategies for allocating reasoning computation. The most pronounced outlier is Qwen3-max, which exhibits dis- proportionately high token consumption in English across both settings, averaging roughly 10,000 tokens in Standard MPAR- Bench and 11,700 in Enhanced MPAR-Bench. Qualitative in- spection of its generated reasoning trace reveals a consistent ten- dency toward repetitive self-verification: the model frequently revisits intermediate conclusions, generates multiple redundant candidate answers, and enters circular deliberation loops before converging on a final prediction. This overthinking behavior substantially inflates token usage. Enhanced MPAR-Bench Result Analysis Tables 11 and 12 present the detailed accuracy of model across four perturbation variants in the Enhanced MPAR-Bench setting: Clue Masking, Order Shuffling, Distractor Injection, and Multi- step Inferring. The four perturbation types exhibit different impacts on model performance. Order Shuffling consistently yields the highest accuracy across all models, some of which are even higher than Standard setting. In contrast, Clue Masking causes the most severe degradation in English, with an average drop of 20.0% from Order Shuffling. Distractor Injection also substantially reduces performance, particularly for Qwen3-max and Seed-2- pro, both dropping by over 28% from their respective Standard accuracies. This indicates that spurious semantic correlations introduced by irrelevant clue words can effectively derail the modelâs reasoning trajectory. Multi-step Inferring occupies an intermediate difficulty level, suggesting that extending the as- sociative chain length moderately taxes the modelâs long-range semantic mapping capacity but does not fundamentally break the reasoning process. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models12 ModelsModel SizeAccessVersionProvider GPT-5.2undisclosedapigpt-5.2-2025-12-11OpenAI Gemini-3.1proundisclosedapiGemini-3.1pro-preview Google Gemini-3flashundisclosedapiGemini-3flash-preview Sonnet-4.5undisclosedapiclaude-sonnet-4-5-20250929Anthropic Qwen3-maxundisclosedapiqwen3-max-2026-01-23 Qwen Qwen3-0.6B0.6Bweights- Qwen3-1.7B1.7Bweights- Qwen3-4B4Bweights- Qwen3-8B8Bweights- Qwen3-14B14Bweights- Qwen3-32B32Bweights- Kimi-k21Tapikimi-k2-thinking-251104Moonshot AI Deepseek-v3.2671BapiDeepseek-v3.2DeepSeek Seed-2-proundisclosedapidoubao-seed-2-0-pro-260215ByteDance Table 7: Summary of Evaluated Models ComponentVersion Python3.11.14 PyTorch2.6.0 transformers4.57.1 accelerate1.10.1 vllm0.16.0rc2 openai1.96.1 tiktoken0.9.0 fastText0.9.3 gensim4.3.0 Table 8: Software versions used in our experiments Model Avg Token (English) Avg Token (Chinese) GPT-5.21355.11263.6 Gemini-3.1pro3551.11883.3 Sonnet-4.51468.91729.2 Qwen3-max9983.3777.9 Kimi-k24659.53290.7 Deepseek-v3.23091.02194.4 Seed-2-pro1865.21563.8 Table 9: Token Usage in Thinking Mode LLMs (Standard) Reasoning Trace Error Analysis Tables 13 through 16 present the reasoning trace evaluation re- sults, decomposing reasoning failures into âfact failâ and âlogic failâ. Across all models, logical error rates substantially exceed factual error rates. In the English thinking standard setting, Deepseek- v3.2 exhibits a logical error rate of 43.60%, Kimi-k2 40.56%, and Qwen3-max 38.21%. Even the best-performing model, Gemini-3.1pro, shows a logical error rate of 20.61%. By con- trast, factual error rates in the same setting are considerably Model Avg Token (English) Avg Token (Chinese) GPT-5.21741.21338.9 Gemini-3.1pro4940.72409.8 Sonnet-4.51577.41838.7 Qwen3-max11743.44782.4 Kimi-k24790.14120.7 Deepseek-v3.23255.02307.8 Seed-2-pro2140.71858.6 Table 10: Token Usage in Thinking Mode LLMs (Enhanced) Model English Acc(%)Chinese Acc(%) Mask Shuf. Dis. Multi step Mask Shuf. Dis. Multi step GPT-5.256.8 80.8 60.8 68.048.8 68.0 67.2 51.2 Gemini-3.1pro 67.2 86.3 76.7 77.655.2 73.6 66.4 60.8 Sonnet-4.553.6 73.6 56.8 69.6 49.6 70.4 67.2 56.8 Qwen3-max56.0 68.8 43.2 59.248.0 63.2 58.4 49.6 Kimi-k249.6 69.6 49.6 61.6 35.2 57.6 48.8 41.6 Deepseek-v3.2 40.8 64.8 44.8 57.644.8 60.0 57.6 47.2 Seed-2-pro47.2 67.2 43.2 68.8 51.2 68.0 60.8 52.8 Table 11: Detailed Results of Enhanced Benchmark on Thinking Models lower: Deepseek-v3.2 at 13.60%, Kimi-k2 at 15.77%, and Gemini-3.1pro at 11.52%. This indicates that the primary fail- ure in multi-point associative reasoning is not incorrect factual knowledge but rather invalid inferential jumps in constructing the reasoning chain from clues to the answer. Moreover, logical error rates in Chinese reasoning are consistently lower than in English. Furthermore, in the non-thinking mode, factual error rates in- crease substantially, which suggests that without an explicit reasoning process, models are more susceptible to factual hallu- cination. Preprint â From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models13 Model English Acc(%) Chinese Acc(%) Mask Shuf. Dis. Multi step Mask Shuf. Dis. Multi step GPT-5.232.0 54.5 39.2 51.242.4 67.2 64.8 43.2 Gemini-3flash 43.2 68.8 51.2 60.849.6 76.0 51.2 68.8 Sonnet-4.545.6 72.8 51.2 53.6 53.6 71.2 71.2 52.0 Qwen3-max35.2 55.2 44.0 44.844.0 65.6 67.2 47.2 Deepseek-v3.2 32.0 56.8 39.2 42.447.2 61.6 62.4 44.8 Seed-2-pro31.2 56.0 33.6 52.847.2 64.8 58.4 53.6 Table 12: Detailed Results of Enhanced Benchmark on Non- thinking Models Model EnglishChinese Fact(%)Logic(%)Fact(%)Logic(%) â Ăâ Ăâ Ăâ Ă GPT-5.21.75 6.616.39 24.640.87 2.255.59 11.69 Gemini-3.1pro 2.58 11.52 6.08 20.612.16 2.455.71 7.77 Sonnet-4.54.30 17.14 8.71 31.623.44 5.288.55 14.72 Qwen3-max4.37 18.968.03 38.212.22 4.949.22 15.30 Kimi-k23.91 15.779.66 40.564.32 7.6111.01 24.13 Deepseek-v3.2 6.02 13.6012.89 43.603.59 7.7310.78 24.85 Seed-2-pro5.27 20.14 7.90 34.692.48 5.087.06 13.33 Table 13: Detailed Fact Fail and Logic Fail of Reasoning Trace on Standard MPAR-Bench on Thinking Models Structured Reasoning Skill Experiment We designed a structured three-step reasoning skill to test whether explicit multi-step prompting can enhance reasoning breadth beyond what thinking mode alone achieves. The skill instructs the model to: (1) examine all clues comprehensively and identify their underlying associations; (2) prioritize specific concepts over abstract hypernyms; and (3) perform reverse veri- fication by reasoning backward from the predicted answer to the provided clues. We evaluate this skill on Seed-2-pro (thinking mode, standard setting). On the English subset, accuracy rises marginally from 71.4% to 72.4% (+1.0p), ANLS from 0.724 to 0.734, and embed- ding similarity from 0.802 to 0.807. On the Chinese subset, gains are somewhat larger: accuracy from 64.6% to 67.8% (+3.2p), ANLS from 0.739 to 0.765, and embedding similarity from 0.820 to 0.838. The modest magnitude of improvementâ particularly on Englishâsuggests that thinking-mode models may already perform these associative integration steps implic- itly during extended reasoning, and that prompt-level interven- tions alone offer limited leverage on the core challenge of multi- source evidence integration. Model EnglishChinese Fact(%)Logic(%)Fact(%)Logic(%) â Ăâ Ăâ Ăâ Ă GPT-5.25.30 24.069.93 47.031.75 3.876.34 12.88 Gemini-3flash 4.74 35.337.89 47.734.24 6.798.78 15.52 Sonnet-4.55.40 9.94 27.30 45.413.84 7.189.24 14.10 Qwen3-max4.55 31.308.52 45.652.61 4.498.32 15.28 Deepseek-v3.2 8.95 29.7115.10 55.972.18 8.838.84 21.83 Seed-2-pro10.03 37.5115.65 52.143.43 8.278.47 20.34 Table 14: Detailed Fact Fail and Logic Fail of Reasoning Trace on Standard MPAR-Bench on Non-thinking Models Model EnglishChinese Fact(%)Logic(%)Fact(%)Logic(%) â Ăâ Ăâ Ăâ Ă GPT-5.22.52 6.2210.38 26.852.27 2.309.94 13.31 Gemini-3.1pro 4.07 10.719.85 23.513.11 3.5910.97 12.56 Sonnet-4.56.79 16.4912.93 39.216.11 8.3515.73 22.32 Qwen3-max4.74 15.7811.86 35.903.15 4.3011.83 17.13 Kimi-k25.96 14.03 14.02 40.255.61 10.1518.18 25.07 Deepseek-v3.2 7.09 14.37 17.11 45.325.97 9.8017.11 31.04 Seed-2-pro7.07 21.60 12.21 37.825.83 7.0014.87 18.23 Table 15: Detailed Fact Fail and Logic Fail of Reasoning Trace on Enhanced MPAR-Bench on Thinking Models Model EnglishChinese Fact(%)Logic(%)Fact(%)Logic(%) â Ăâ Ăâ Ăâ Ă GPT-5.27.16 24.1913.97 47.393.38 5.7310.11 19.29 Gemini-3flash 7.22 33.0013.62 48.066.38 11.2213.60 24.33 Sonnet-4.58.52 26.7014.79 46.045.97 10.4415.73 25.45 Qwen3-max6.25 25.56 13.21 48.913.32 9.2914.47 25.21 Deepseek-v3.2 11.79 29.6320.09 56.304.58 11.5014.49 30.75 Seed-2-pro12.97 33.9518.03 50.885.36 11.3614.08 24.81 Table 16: Detailed Fact Fail and Logic Fail of Reasoning Trace on Enhanced MPAR-Bench on Non-thinking Models