Paper deep dive

MASTraceBench: Diagnosing Collaboration Gains through Proposal Trajectories in LLM-Based Multi-Agent Systems

Yapeng Li, Songze Li, Shuang Yu, Jing Yu, Zhixin Liu, Liqiang Wen, Tonghua Su

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 89

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 9/29/2026, 11:22:49 AM

Summary

The paper introduces MASTraceBench, a benchmark designed to diagnose collaboration gains in LLM-based Multi-Agent Systems (MAS) by tracking proposal trajectories rather than just final outcomes. It evaluates six tasks and reveals that final MAS answers rarely surpass the strongest initial proposal, often lifting weaker ones instead. To address this, the authors propose CLEARS, a method using claim-level evaluation to preserve strong proposals, achieving higher Collaboration Gain on five of six tasks.

Entities (11)

MASTraceBench · benchmark · 98%CLEARS · method · 97%LLM-based Multi-Agent Systems · technology · 95%Collaboration Gain · metric · 92%Qwen3-32b · model · 90%MAPF · task · 88%EvidenceChain · task · 85%Gomoku · task · 85%Negotiation · task · 85%OptionGen · task · 85%TacticDuel · task · 85%

Relation Signals (10)

MASTraceBench → evaluates → LLM-based Multi-Agent Systems

confidence 95% · Using MASTraceBench, we systematically compare representative MAS methods...

CLEARS → improves → Collaboration Gain

confidence 93% · CLEARS more often preserves or improves upon the strongest initial proposal and achieves the highest Collaboration Gain on five of the six tasks.

MASTraceBench → includestasks → Gomoku

confidence 90% · MASTraceBench includes... board-game move selection... Gomoku

MASTraceBench → includestasks → EvidenceChain

confidence 90% · MASTraceBench includes... evidence-chain reasoning... EvidenceChain

MASTraceBench → includestasks → MAPF

confidence 90% · MASTraceBench includes multi-agent path finding... These tasks span cooperative and competitive...

MASTraceBench → includestasks → Negotiation

confidence 90% · MASTraceBench includes... private-value negotiation... Negotiation

MASTraceBench → includestasks → TacticDuel

confidence 90% · MASTraceBench includes... turn-based tactical decision-making... TacticDuel

MASTraceBench → includestasks → OptionGen

confidence 90% · MASTraceBench includes... option generation... OptionGen

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:LLM-based multi-agent systems (MAS) have shown promise in complex problem solving. As MAS methods diversify, systematic evaluation becomes increasingly challenging. However, existing benchmarks largely focus on final outcomes, leaving unclear how collaboration gains arise, are preserved, or are lost. To address this limitation, we introduce MASTraceBench, a benchmark for diagnosing collaboration gains through proposal trajectories in MAS. Across six cooperative and competitive tasks, MASTraceBench tracks and grades proposal trajectories and provides a multi-layer metric suite covering Task Score, Collaboration Gain, proposal-trajectory indicators, and Token Cost. Using MASTraceBench, we systematically compare representative MAS methods not only by final performance, but also by how agent proposals evolve and are aggregated into the final answer. This analysis reveals a recurring pattern: final MAS answers rarely surpass the strongest initial proposal; interaction often lifts initially weaker proposals toward it, while strong initial proposals are seldom further improved and may regress. To reduce this risk, we propose CLEARS, which replaces whole-proposal exchange with claim-level evaluation across agents to guide reliable synthesis. CLEARS more often preserves or improves upon the strongest initial proposal and achieves the highest Collaboration Gain on five of the six tasks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

85,716 characters extracted from source content.

Expand or collapse full text

MASTraceBench: Diagnosing Collaboration Gains through Proposal Trajectories in LLM-Based Multi-Agent Systems Yapeng Li 1 , Songze Li 1 , Shuang Yu 1 , Jing Yu 1 , Zhixin Liu 1 , Liqiang Wen 2 , Tonghua Su 1∗ 1 Harbin Institute of Technology, Harbin, China 2 Peking University, Beijing, China liyapeng, lisongze, yujing, Zhixin_Liu@stu.hit.edu.cn, yushuang, thsu@hit.edu.cn, wenlq@pku.edu.cn Abstract LLM-based multi-agent systems (MAS) have shown promise in complex problem solving. As MAS methods diversify, sys- tematic evaluation becomes increasingly challenging. How- ever, existing benchmarks largely focus on final outcomes, leaving unclear how collaboration gains arise, are preserved, or are lost. To address this limitation, we introduce MAS- TraceBench, a benchmark for diagnosing collaboration gains through proposal trajectories in MAS. Across six cooperative and competitive tasks, MASTraceBench tracks and grades proposal trajectories and provides a multi-layer metric suite covering Task Score, Collaboration Gain, proposal-trajectory indicators, and Token Cost. Using MASTraceBench, we sys- tematically compare representative MAS methods not only by final performance, but also by how agent proposals evolve and are aggregated into the final answer. This analysis re- veals a recurring pattern: final MAS answers rarely surpass the strongest initial proposal; interaction often lifts initially weaker proposals toward it, while strong initial proposals are seldom further improved and may regress. To reduce this risk, we propose CLEARS, which replaces whole-proposal ex- change with claim-level evaluation across agents to guide re- liable synthesis. CLEARS more often preserves or improves upon the strongest initial proposal and achieves the highest Collaboration Gain on five of the six tasks. Introduction Agents based on large language models (LLMs) have demon- strated strong capabilities in understanding task instruc- tions, reasoning and planning, and taking actions toward task goals (Wang et al. 2024a; Xi et al. 2025). Building on these capabilities, LLM-based multi-agent systems (MAS) coordi- nate multiple agents with different roles, expertise, or view- points to exchange information and jointly solve complex tasks (Wu et al. 2024; Hong et al. 2024; Du et al. 2024). As MAS methods become diverse in their agent configurations, communication protocols, and decision mechanisms, sys- tematically evaluating them under comparable settings has become essential for clarifying the practical value of MAS. Reliable assessment of MAS requires evaluation ap- proaches that account for its collaborative nature. Despite growing efforts in MAS evaluation, existing benchmarks offer limited diagnostic granularity beyond final outcomes. ∗ Corresponding author Evaluation Scope Metric Dimensions Performance Overall Accuracy Task Score Collaboration Gain Proposal Trajectory Token Cost Not provided Token Cost QueryMAS Answer Correct Narrow Scope Incorrect Query Answer v v initial proposals final proposals ... Scores: 70/80 Scores: 85/70 Expanded Scope MAS Proposal Trajectory Traditional Bench MASTraceBench Agent Refinement Aggregation Loss Gain Against Best Not provided MASTraceBench Traditional Bench Score: 85 Figure 1: Comparison of evaluation scope and metric dimen- sions between traditional benchmarks and MASTraceBench. Most existing studies on LLM-based MAS (Chen, Saha, and Bansal 2024; Fan et al. 2026) reuse general-purpose bench- marks for tasks such as mathematical reasoning (Cobbe et al. 2021) and code generation (Chen et al. 2021), retaining con- ventional protocols that judge only whether the final answer is correct, as illustrated in Fig. 1. This outcome-only evalua- arXiv:2609.34496v1 [cs.MA] 28 Sep 2026 tion obscures fine-grained improvements, such as transitions from incorrect to partially correct solutions, and makes it difficult to determine whether collaboration gains arise from the initial proposal pool, subsequent interaction, or final ag- gregation. As fixed and widely reused test sets, these bench- marks also increasingly face performance saturation (Akhtar et al. 2026) and potential training-data exposure (Sainz et al. 2023), potentially biasing comparisons among MAS meth- ods. Recent MAS benchmarks broaden task coverage to include coordination, competition, and sequential decision- making in multi-agent settings, with some further extending the evaluation metrics beyond final outcomes to assess in- termediate processes (Sun et al. 2025; Zhu et al. 2025; Lin et al. 2025). However, they are primarily designed to compare model backbones in multi-agent settings rather than to com- pare and diagnose MAS methods. Their intermediate metrics do not directly track the trajectories of candidate solutions during interaction and aggregation, providing limited insight into how collaboration gains are produced, preserved, or lost. To address this gap, we introduce MASTraceBench, a benchmark for diagnosing MAS collaboration gains through proposal-trajectory analysis, as illustrated in Fig. 1. We de- fine a proposal as an evaluable candidate solution gener- ated within the MAS for the current query or state. Rather than evaluating only the final MAS answer, MASTraceBench records the initial and final proposals of Proposer Agents together with the final MAS answer, thereby capturing pro- posal trajectories during interaction and aggregation. It as- signs graded scores to these outputs, making fine-grained changes in proposal quality directly measurable. Accord- ingly, MASTraceBench reports a broader metric suite be- yond Task Score, covering Collaboration Gain, proposal- trajectory indicators, and Token Cost. It covers six coopera- tive and competitive tasks in which proposals are observable and scoreable, enabling comparison across MAS methods. Using MASTraceBench, we evaluate representative MAS methods and uncover a recurring pattern hidden by final per- formance: final MAS answers rarely surpass the strongest initial proposal; interaction more often lifts initially weaker proposals toward it, while initially strong proposals are sel- dom further improved and may regress. Rather than meet- ing the common expectation that multi-agent interaction cre- ates stronger proposals, the evaluated methods often gain by preserving and propagating existing strong proposals. This result highlights a key limitation in how current interac- tion mechanisms use proposal diversity: whole-proposal ex- change transmits useful and unreliable components together and may not reliably preserve strong proposals, as reflected in the observed regressions. To reduce this risk, we introduce CLEARS, a method that replaces whole-proposal exchange with claim-level evaluation across agents to guide reliable synthesis. CLEARS more often preserves or improves upon the strongest initial proposal and achieves the highest Col- laboration Gain on five of the six benchmarks. Our contributions are summarized as follows: • We introduce MASTraceBench, which moves beyond outcome-only MAS evaluation to enable fine-grained diagnosis of collaboration gains. To the best of our knowledge, it is the first benchmark to support proposal- trajectory diagnosis at the method level. • Using MASTraceBench, we systematically evaluate rep- resentative MAS methods across six tasks, revealing a recurring pattern underlying collaboration gains: strong initial proposals are rarely improved and may regress, whereas weaker ones are often lifted toward them. • We introduce CLEARS, which enables selective synthesis by preserving useful proposal components and avoiding unreliable ones. CLEARS more reliably preserves or im- proves upon the strongest initial proposal and achieves the highest Collaboration Gain on five of the six benchmarks. Related Work LLM-based Multi-Agent Systems LLM-based MAS have rapidly evolved, enabled by recent advances in large language models. Early work on self- consistency (Wang et al. 2023) shows that aggregating mul- tiple reasoning paths can improve performance. Reflection- based approaches (Madaan et al. 2023; Shinn et al. 2023) fur- ther refine model outputs through iterative feedback. Later work explores explicit interaction among multiple agents, including debate (Du et al. 2024; Liang et al. 2024), peer review (Xu et al. 2023), and verification (Lifshitz, McIlraith, and Du 2025). Another important line of work focuses on automatic or adaptive MAS construction, including Agent- Verse (Chen et al. 2024b), which dynamically adjusts multi- agent group composition, and MAS-GPT (Ye et al. 2025), which generates query-adaptive MASs from user queries. MASTraceBench evaluates these representative methods and provides diagnostics of their collaboration gains. MAS Benchmarks Benchmarks used in MAS studies can be broadly divided into two categories: general-purpose benchmarks reused for MAS evaluation and benchmarks specifically designed for multi-agent settings. General-purpose benchmarks such as MATH (Hendrycks et al. 2021), HumanEval+ (Liu et al. 2023), MMLU-Pro (Wang et al. 2024c), and GPQA (Rein et al. 2024) cover mathematical reasoning, code generation and multidomain reasoning, but typically assess only final- answer correctness. MAS-specific benchmarks broaden this scope: Collab-Overcooked (Sun et al. 2025) and BattleAgent- Bench (Wang et al. 2024b) evaluate sequential collaboration and cooperative and competitive task execution in structured interactive environments, respectively. LLMArena (Chen et al. 2024a) spans diverse dynamic games to assess strategic planning, communication, opponent modeling, and team col- laboration. MultiAgentBench (Zhu et al. 2025) further mea- sures task progress and coordination quality through mile- stone, communication, and planning scores, while GAME- BoT (Lin et al. 2025) decomposes game reasoning into pre- defined subproblems with rule-based intermediate verifica- tion. These benchmarks primarily compare model backbones in particular multi-agent settings, whereas MASTraceBench focuses on method-level diagnosis by tracking proposal tra- jectories during interaction and aggregation. Sources and Reliability of MAS Gains Recent studies examine how MAS gains arise and under what conditions they remain reliable. Debate-or-Vote (Choi, Zhu, and Li 2026) decomposes debate gains into majority voting and inter-agent discussion. A controlled study (Wu, Li, and Li 2025) examines the effects of initial reasoning strength, group diversity, and majority pressure, while MAST (Pan et al. 2025) identifies recurrent MAS failure modes from ex- ecution traces. A scaling-oriented study (Kim et al. 2025) further shows that MAS gains depend on model capabil- ity, task properties, and MAS architecture. MASTraceBench advances this research by quantitatively tracking proposal trajectories to reveal how collaboration gains are produced, preserved, or lost. MASTraceBench We introduce MASTraceBench with (1) a unified evaluation framework for consistently capturing proposal trajectories across MAS methods and task scenarios; (2) a multi-layer metric suite for diagnosing collaboration gains beyond final outcomes; and (3) a diverse benchmark suite spanning six cooperative and competitive tasks with graded scoring along proposal trajectories. Unified Evaluation Framework To consistently capture and evaluate proposal trajectories across MAS methods and task scenarios, we organize the evaluation framework around agent modeling and environ- ment modeling. Agent Modeling. We distinguish agents by whether their outputs can be directly evaluated as proposals. Within an MAS, agents that produce candidate proposals for a given query or state are treated as Proposer Agents. MAS- TraceBench records and scores their initial and final pro- posals together with the final MAS answer; these outputs constitute the MAS’s proposal trajectory for that query or state. Agents that provide feedback, verification, critique, or judgment without producing such proposals are treated as Function Agents, whose outputs serve as auxiliary interac- tion signals rather than independently evaluated proposals. Environment Modeling. We model different benchmark tasks under a unified query–MAS–answer paradigm. For static tasks, the MAS receives a query and produces a fi- nal MAS answer, while MASTraceBench records the corre- sponding proposal trajectory. For dynamic tasks, the MAS repeatedly observes the current state, outputs an action, and receives the next state according to (q,s t )→ MAS→ a t , s t+1 =T (s t ,a t ), (1) where q is the task query, s t is the environment state at step t, a t is the MAS answer at that step, and T is the tran- sition function. MASTraceBench records a corresponding proposal trajectory for each action decision, and the com- plete sequence of actions determines the final task outcome. Multi-Layer Metrics To address the limited diagnostic granularity of outcome- only evaluation, we define a multi-layer metric suite for di- agnosing collaboration gains, with proposal-level diagnostics grounded in the recorded proposal trajectories. Layer-1: How much gain does the MAS achieve? We use Task Score (TS) to measure the task-level performance of a MAS: TS = E z∈Z [R MAS (z)],(2) whereZ is the set of task-level instances, i.e., samples for static tasks and episodes for dynamic tasks. R MAS (z) is the final MAS answer score for static tasks and the episode score determined by the MAS action sequence for dynamic tasks. To quantify the overall performance improvement achieved by a MAS over the Single-Agent baseline, we define Collaboration Gain (CG) as: CG = TS− TS Single ,(3) where TS Single denotes the Task Score achieved by the Single-Agent baseline under the same benchmark and eval- uation protocol. Layer-2: How does the MAS produce, preserve, or lose gains? A final MAS answer score alone can hide very different proposal trajectories. Consider a MAPF instance where scores are arrival rates and, in all cases, the final MAS answer scores 66.7%. If all initial proposals score 33.3%, this answer indicates newly created value beyond the initial proposal pool. If the initial proposals score 100.0%, 33.3%, and 33.3%, the same answer instead fails to preserve the strongest initial proposal, even though some individual pro- posals may have been refined from 33.3% to 66.7% during collaboration. If a final Proposer proposal still scores 100.0% but the aggregated MAS answer scores 66.7%, the loss oc- curs during aggregation. These cases motivate GAB, AR, and AL, which respectively compare the final MAS answer with the best initial proposal, track the initial-to-final evolution of individual Proposer proposals, and measure aggregation loss. We formalize these proposal trajectory diagnostics over a set of evaluation unitsD, where eachx∈D corresponds to a sample in a static task or a state-conditioned action decision in a dynamic task. For each x, let S MAS (x) denote the score of the final MAS answer. LetP m denote the set of Proposer Agents in MAS method m, and let S init i (x) and S final i (x) denote the scores of agent i’s initial and final proposals, respectively. For subsequent diagnostics, we define b x = max i∈P m S init i (x) as the best initial proposal score and let S ⋆ x denote the maximum attainable score for x. Gain Against Best (GAB). For each x ∈ D, we define GAB label as: GAB(x) =      Improve, S MAS (x) > b x , Regress, S MAS (x) < b x , OptKeep, S MAS (x) = b x = S ⋆ x , SubKeep, S MAS (x) = b x < S ⋆ x . The overall GAB is reported as the percentage distribution of these labels overD. Competition Gomoku MAPF Current Board Selects (4, 4) to block the white extension while simultaneously forming a three-in-a-row for black. Selects (4, 5) to block black’s three-in-a-row and prevent an immediate threat. Query: Plan collision-free paths for two robots. EvidenceChain Negotiation Query: Given a question and a list of numbered facts, please select and order the evidence fact IDs as a logical chain supporting the answer. evidencechain: [0, 3, 4] Fact 0 → Fact 3 → Fact 4 → Answer Objects Pool: My allocation proposal: MAS-1 gets: 2 shirts, 0 diamonds, 4 food MAS-2 gets: 1 shirts, 2 diamonds, 0 food Reject. My allocation proposal: ... private valuesprivate values Time discount . . . . . . . . . . . . . . . . . . TacticDuel OptionGen Charge.Build energy early for a future Burst. Heavy. Deal 4 damage while Counter is unlikely on turn 1. Query: Given a passage and a question, please generate one correct answer and three distractors. . . . . . . . . . . . . A: By train.B:By bus. C:On foot.D:By bike Correct option: A Two players start with 20 HP / 0 Energy. Each round, both simultaneously pick one of 7 actions: Quick, Heavy, Guard, Counter, Feint, Charge, Burst. Cooperation HP HP Energy Energy: 0 . . . . . . Robot A: [up, wait, ...] Robot B: [left, left, ...] MAS MAS MAS MAS-2 MAS-1 MAS-1 MAS-2 MAS-1 MAS-2 Figure 2: Benchmark design in MASTraceBench. Red dashed boxes indicate the proposal-level outputs evaluated by S(·). BenchmarkSettingTask Score (TS)Graded Proposal Scoring Function S(·) MAPF157 samplesMean arrival rateFraction of robots reaching their goals OptionGen235 samplesMean option-set scoreLLM-judge score of the correct option and distractor set EvidenceChain100 samplesMean chain scoreSet F1 and ordering agreement with ground truth GomokuDynamicWin rateSelf-formation score plus defense-interception score NegotiationDynamicMean episode payoff ratioPayoff ratio of proposed allocation TacticDuelDynamicWin rateRule-based tactical value of the proposed action Table 1: Tasks and evaluation specifications in MASTraceBench. The first three tasks are static tasks that require one-shot solutions for independent samples, while tasks marked as Dynamic require MASs to repeatedly observe the current state and output actions until an episode ends. Additional implementation details are provided in the supplementary material. Agent Refinement (AR). We define AR to track pro- posal evolution from S init i (x) to S final i (x). To examine how proposal refinement varies with initial quality, we partition agents on each instance x into Strong Agents and Weak Agents according to their initial proposal scores: Strong Agents are those inB x = i ∈ P m | S init i (x) = b x , while Weak Agents are those inW x =P m x . For agents inB x , their initial score is b x . We define their refinement label AR strong (x,i) using the same four conditions as GAB(x), simply by replacing S MAS (x) with S final i (x). For agents inW x , we measure their refinement relative to both their own initial score and the strong baseline b x . Their refinement label is defined as: AR weak (x,i) =          Surpass, S final i (x) > b x , Reach, S final i (x) = b x , Partial, S init i (x) < S final i (x) < b x , Keep, S final i (x) = S init i (x), Degrade, S final i (x) < S init i (x). For each AR group, we report label percentages within that group. Aggregation Loss (AL). Even after proposal refinement, gains may still be lost when the final MAS answer is aggre- gated. We therefore define AL as the gap between the final MAS answer and the best final proposal: AL = E x∈D max i∈P m S final i (x)− S MAS (x) ,(4) Layer-3: How costly are these gains? Since MAS gains may rely on additional agent calls, interaction rounds, or verification steps, we use Total Token Cost (TC) to measure the average token consumption required by a MAS: TC = E x∈D [C MAS (x)],(5) where C MAS (x) denotes the total token cost incurred by all agent calls in the MAS from receiving the input context to producing the corresponding MAS answer. Benchmark Design General-purpose benchmark tasks often require an MAS to select an option or output a final numerical answer, so the resulting outputs are ultimately judged only as correct or incorrect. An improved but still incorrect proposal therefore remains indistinguishable from its predecessor, while quality differences among correct proposals are also hidden. To cap- ture the proposal-quality differences required for proposal- trajectory analysis, we construct six evaluation tasks. As shown in Fig. 2, MASTraceBench includes multi-agent path finding, option generation, evidence-chain reasoning, board-game move selection, private-value negotiation, and turn-based tactical decision-making. These tasks span coop- erative and competitive, static and dynamic settings: a single 2 Cross-Agent Claim Evaluation 3 Dual-Signal Synthesis 1 Claim-Level Decomposition Agent 1 Proposal1 Claim 1-1 Agent 2 Proposal2 Claim 2-1 Claim 2-2 ... Agent 1 Agent 2 Agent 3 Agent 2 C 2-1 C 2-2 Agent 1 Agent 3 — C 1-1 C 1-2 C 3-1 C 3-2 C 1-1 C 1-2 C 2-1 C 2-2 C 3-1 C 3-2 — — Endorse Challenge Synthesizer Task Prompt Final Answer Top Challenged Top Endorsed Claim 1-1 Claim 2-2 Claim 3-1 Incorporate Avoid / Revise Claim 1-2 ... ... Figure 3: Overview of CLEARS. MAS solves cooperative tasks, whereas multiple MASs in- teract head-to-head in competitive tasks. Each task assigns graded scores to intermediate proposals and final MAS an- swers. For example, in MAPF, guiding two rather than one of three robots to their goals raises a proposal’s score from 33.3% to 66.7%. Such scoring makes fine-grained changes in proposal quality observable and supports proposal-trajectory analysis across diverse forms of multi-agent interaction. Table 1 summarizes each task’s setting, Task Score, and proposal-level scoring function S(·). CLEARS: Claim-Level Evaluation Across Agents for Reliable Synthesis To reduce the risk caused by propagating useful and unreli- able proposal components together, we introduce CLEARS, a diagnosis-guided MAS method that shifts interaction and synthesis from whole proposals to claim-level signals. For an input x, let Y x = y i | i ∈ P m denote the proposals generated by the Proposer Agents. For analysis, we view each proposal as consisting of semantically meaningful solution components, such as factual assertions, reasoning steps, or local decisions: φ(y i ) = (U i ,E i ),(6) where U i and E i denote latent components that contribute positively and negatively to the overall quality of proposal y i , respectively. In whole-proposal interaction, Agent i revises its proposal by conditioning on the other proposals in their entirety: ̃y i ∼ p θ (·| x,y i ,y j j̸=i ).(7) Providing each y j intact leavesU j andE j undifferentiated during revision, while subsequent majority voting or LLM- based selection may overlook U j when the corresponding proposal is not selected. As shown in Fig. 3, CLEARS translates this latent dis- tinction into a three-stage procedure. Each Proposer Agent first decomposes its proposal into claims, each explicitly representing a solution component. Cross-agent evaluation aggregates different agent perspectives to identify the top- endorsed claims C + x and top-challenged claims C − x , which respectively approximate the useful components U i i∈P m and unreliable componentsE i i∈P m across proposals. The final MAS answer is then generated as ˆy CLEARS ∼ p θ ·| x,C + x ,C − x .(8) During synthesis, C + x provides content to preserve or com- bine, whereas C − x identifies content to avoid, enabling CLEARS to construct a new final MAS answer from in- formation distributed across proposals. Experiments Setup Evaluated Methods. We evaluate representative MAS methods from four categories (Table 2): selection-based, debate-based, revision-based, and adaptive MAS construc- tion. The first three capture common collaboration primi- tives, while the last automatically configures agent roles and workflows. All methods use Qwen3-32B (Yang et al. 2025) as the shared backbone. For methods with configurable multi- Proposer settings, where multiple Proposer Agents generate candidate proposals, we use three Proposers; Int. Debate and MAPR use one interaction round. Evaluation Settings. For static tasks, we report the main results as averages over three runs per method. For competi- tive tasks, each method plays a fixed Single-Agent baseline, avoiding opponent-set-dependent tournaments. We conduct 50 Gomoku and TacticDuel games and evaluate 100 Nego- tiation configurations, each defined by an item pool and two private-value profiles. To control order effects, we alternate the first mover in Gomoku and run each Negotiation config- uration twice, reversing which side makes the first offer. Results and Analysis We present the main results on MASTraceBench and then provide a detailed analysis of representative MAS methods. Observation 1: Multi-proposer interaction brings larger gains in MAS. As shown in Table 2, MAS methods ex- hibit clear differences in gains over the Single-Agent base- line. Interactive protocols with multiple Proposer Agents, such as Int. Debate and MAPR, generally achieve stronger TS and CG than passive aggregation, external verification, or single-proposer refinement methods. For example, on Gomoku, Int. Debate reaches 80.00, outperforming Consis- tency (50.00), MAV (46.00), and Reflection (58.00), while MAPR also achieves a strong score of 72.00. Adaptive frame- works such as AgentVerse and MAS-GPT perform reason- ably on several benchmarks, but they do not consistently match the strongest interactive protocols, suggesting that workflow adaptation alone does not guarantee the most suit- able collaboration mechanism. Notably, Adv. Debate’s poor performance may stem from noisy counter-proposals induced by its forced pro–con roles. Observation 2: Interaction among multi-proposers pro- duces limited new gains in MAS. Fig. 4 reports GAB for this class of methods, for which cross-proposer debate or revision would intuitively be expected to create gains be- yond the strongest initial proposal. However, the Improve Method MAPFOptionGen EvidenceChain Gomoku Negotiation TacticDuel TS CG TS CG TSCGTS CG TS CG TS CG Single-Agent69.66 – 80.56 – 74.02–50.00 – 41.00 – 50.00 – Consistency (Wang et al. 2023)72.91 3.25 81.83 1.27 75.19 1.17 50.00 0.00 42.33 1.33 55.00 5.00 MAV (Lifshitz, McIlraith, and Du 2025) 77.13 7.47 82.19 1.63 74.40 0.38 46.00 -4.00 43.21 2.21 54.00 4.00 Int. Debate (Du et al. 2024)77.317.6583.70 3.14 77.943.9280.00 30.00 44.08 3.08 64.0014.00 Adv. Debate (Liang et al. 2024)67.50 -2.16 81.32 0.76 65.83 -8.19 48.00 -2.00 43.28 2.28 21.00 -29.00 Reflection (Renze and Guven 2024)75.24 5.58 81.22 0.66 75.75 1.73 58.00 8.00 42.51 1.51 36.00 -14.00 MAPR (Xu et al. 2023)76.33 6.67 84.033.4777.81 3.79 72.00 22.00 44.753.7535.00 -15.00 AgentVerse (Chen et al. 2024b)76.04 6.38 81.62 1.06 76.46 2.44 64.00 14.00 41.78 0.78 52.00 2.00 MAS-GPT (Ye et al. 2025)71.92 2.26 81.56 1.00 77.33 3.31 68.00 18.00 42.19 1.19 60.00 10.00 CLEARS (ours)78.72 9.06 84.55 3.99 80.44 6.42 76.0026.0045.24 4.24 66.00 16.00 CLEARS w/o CCE74.52 4.86 82.65 2.09 79.58 5.56 72.00 22.00 42.74 1.74 53.00 3.00 Table 2: Performance of MAS methods across benchmarks. TS and CG denote Task Score (0–100 scale) and Collaboration Gain over the Single-Agent baseline; Int. Debate and Adv. Debate refer to interactive and adversarial debate. MAPFOptionGenGomokuEvidenceChainNegotiationTacticDuel 1.1% 65.5% 19.3% 14.1% 1.7% 65.1% 14.4% 18.7% 16.8% 1.3% 27.1% 54.8% 18.2% 1.1% 24.0% 56.7% 4.4% 14.4% 41.5% 39.8% 11.3% 11.0% 33.0% 44.7% 0.3% 62.1% 26.1% 11.5% 2.0% 56.7% 23.9% 17.3% 4.0% 0.0% 71.5% 24.5% 6.6% 0.0% 68.0% 25.4% 2.5% 37.7% 48.3% 11.6% 8.4% 25.0% 43.9% 22.7% Improve OptKeepSubKeep Regress 2.4% 66.1% 16.7% 14.8% 17.3% 1.3% 27.7% 53.7% 7.3% 15.3% 42.0% 35.3% 1.7% 55.2% 31.5% 11.6% 11.2% 0.0% 73.6% 15.2% 4.0% 38.2% 47.6% 10.2% Figure 4: GAB analysis. Each bar reports the percentage distribution across GAB labels. ratio remains limited on most benchmarks, showing that new gains are rare. Most cases instead fall into OptKeep or Sub- Keep, suggesting that observed gains mainly come from car- rying forward an already available strong proposal rather than forming a stronger answer. The non-negligible Regress ratios, especially on OptionGen and EvidenceChain, further show that even this preservation is not guaranteed. Observation 3: Weak Agents are lifted while Strong Agents are not reliably preserved. Fig. 5 further exam- ines Observation 2 at the agent level. Weak Agents often fall into Reach or Partial, moving toward the strongest initial proposal but rarely beyond it. For Strong Agents, Improve re- mains limited and Regress is non-negligible, indicating that initially strong proposals are hard to further improve and are not always preserved during interaction. This points to a limitation of whole-proposal interaction: agents exchange full proposals or full-proposal critiques, and whole-proposal revision may carry errors along with useful information. For example, noisy signals from weaker proposals may be ab- sorbed during revision and interfere with otherwise strong proposals. Such strong-proposal degradation is an important source of the GAB Regress observed in Observation 2. Observation 4: Aggregation loss depends on proposal convergence, not only the aggregation rule. Fig. 6 shows that the final MAS answer can fall below the strongest fi- nal proposal among Proposer Agents. Selection-based meth- ods suffer large AL when strong proposals remain isolated. For example, on MAPF, Consistency has only 3.5% final- proposal agreement and an AL of 11.63, while Int. Debate raises this agreement to 58.2% and reduces AL to 2.22. Thus, effective aggregation requires proposal convergence before final decision, not merely a stronger aggregation rule. Agg. Debate further supports this view: replacing Int. De- bate’s majority-vote aggregation with LLM-based aggrega- AR-StrongAR-Weak 0%20%40%60%80%100%0%20%40%60%80%100% Gomoku Int.Debate 66%27%61%33% MAPR 60%26%53%36% 0%20%40%60%80%100%0%20%40%60%80%100% EvidenceChain Int.Debate 20%56%20%31%18%38% MAPR 34%39%40%21%16% Surpass Reach Partial Keep Degrade Improve OptKeep SubKeepRegress Figure 5: AR analysis on EvidenceChain and Gomoku. For each sample, Proposer Agents are grouped as AR-Strong if their initial proposal achieves the best initial score. MAPFOptionGenEvidenceChainGomokuNegotiationTacticDuel Consistency 11.636.229.8711.926.323.16 Int.Debate 2.223.274.261.261.400.72 Agg.Debate 1.913.683.552.111.120.91 MAPR 5.834.585.866.675.743.99 MAV 6.155.989.878.805.772.31 Figure 6: AL analysis. Lower AL indicates stronger aggre- gation capability of the MAS. tion yields comparable rather than uniformly better AL. Observation 5: Token cost reveals the efficiency of MAS gains. Extra tokens support sampling, critique, verifica- tion, and revision, but are valuable only when they yield collaboration gains. Taken together, Tables 2 and 3 show that, compared with Consistency, Int. Debate introduces de- bate among agents and converts the additional token cost into stronger CG. In contrast, the substantially costlier MAV and MAPR do not consistently yield further gains. CLEARS incurs relatively higher TC, much of which is attributable to cross-agent claim evaluation. The subsequent component ab- lation isolates this effect through CLEARS w/o CCE: adding CCE increases TC but consistently improves CG, indicat- ing that the added token budget is effectively converted into Collaboration Gain. CLEARS Results. Table 2 shows that CLEARS achieves the highest CG on five of the six benchmarks. On Ev- idenceChain, CLEARS reaches a CG of 6.42, exceeding the strongest baseline, Int. Debate (3.92), by 2.50 points. The GAB results show a clearer advantage on Negotiation: CLEARS increases the Improve ratio to 11.2%, compared with 4.0% for Int. Debate and 6.6% for MAPR, while reduc- ing Regress to 15.2% from 24.5% and 25.4%, respectively (Fig. 4). These results indicate that CLEARS improves col- laboration gains while mitigating proposal regression. Nev- ertheless, the Improve ratio remains limited, and surpassing MethodMAPF Gomoku Single-Agent1.00×1.00× Reflection2.40×2.38× Consistency3.04×3.04× Int. Debate5.14×4.64× AgentVerse5.75×6.35× Adv. Debate5.79×5.92× MAS-GPT6.16×6.23× MAV8.62×9.50× MAPR10.50× 10.63× CLEARS (ours)8.52×8.38× CLEARS w/o CCE 5.71×5.53× Table 3: Token cost multipliers over the Single-Agent base- line on MAPF and Gomoku. Setting Int. DebateMAPR GI↑ GR↓ WI↑ SR↓ GI↑ GR↓ WI↑ SR↓ Main (3P+1R) 4.4 39.8 54.9 20.3 11.3 44.7 71.1 39.4 5P+1R3.0 49.0 56.2 20.1 9.0 50.0 63.1 39.9 7P+1R4.0 53.0 57.4 30.5 3.0 50.0 61.1 37.5 3P+2R11.0 48.0 63.4 39.7 16.0 38.0 72.8 33.8 3P+3R7.0 41.0 70.6 32.9 6.0 42.0 67.9 34.8 5P+2R4.0 45.0 66.2 31.1 9.0 45.0 61.3 33.6 Scale-Het.7.0 40.0 55.9 23.8 7.0 46.0 68.1 48.2 Family-Het. 15.0 61.0 59.8 50.5 11.0 72.0 53.8 63.1 Table 4: Configuration ablation on EvidenceChain. P and R denote the numbers of Proposers and interaction rounds, respectively; GI/GR denote GAB Improve/Regress; WI/SR denote Surpass+Reach+Partial and Regress for initially weak and strong agents, respectively. Under 3P+1R, Scale-Het. and Family-Het. denote scale- and family-heterogeneous teams composed of Qwen3-32B,14B,8B and Qwen3-8B, Llama-3.1-8B-Instruct, Mistral-8B-Instruct, respectively. the strongest initial proposal remains an open challenge. Ablation Study CLEARS component ablation. We compare CLEARS with CLEARS w/o CCE, which bypasses cross-agent claim evaluation and directly synthesizes all decomposed claims without filtering. Its consistently lower CG across all six benchmarks (Table 2) confirms the contribution of CCE. Interaction scale and model heterogeneity. We vary the number of Proposers, interaction rounds and model com- position for Int. Debate and MAPR on EvidenceChain. As shown in Table 4, GAB Improve remains limited across all configurations, whereas GAB Regress and strong-proposal regression remain substantial despite frequent improvements to initially weak proposals. These results suggest that Obser- vations 2–3 are robust across configurations. Conclusion In this work, we introduce MASTraceBench for method- level comparison and proposal-trajectory diagnosis of LLM- based MAS. Its analyses reveal recurring limitations in ex- isting methods and motivate CLEARS, whose effectiveness is validated across six tasks. These results show that MAS- TraceBench can serve as a testbed for studying collaboration mechanisms and developing more reliable MAS. References Akhtar, M.; Reuel, A.; Soni, P.; Ahuja, S.; Ammanamanchi, P. S.; Rawal, R.; Zouhar, V.; Yadav, S.; Whitehouse, C.; Ki, D.; et al. 2026. When ai benchmarks plateau: A sys- tematic study of benchmark saturation. arXiv preprint arXiv:2602.16763. Chen, J.; Hu, X.; Liu, S.; Huang, S.; Tu, W.-W.; He, Z.; and Wen, L. 2024a. Llmarena: Assessing capabilities of large language models in dynamic multi-agent environments. In Proceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), 13055–13077. Chen, J.; Saha, S.; and Bansal, M. 2024. Reconcile: Round- table conference improves reasoning via consensus among diverse llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7066–7085. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. Chen, W.; Su, Y.; Zuo, J.; Yang, C.; Yuan, C.; Chan, C.-M.; Yu, H.; Lu, Y.; Hung, Y.-H.; Qian, C.; et al. 2024b. Agent- Verse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. In ICLR. Choi, H. K.; Zhu, J.; and Li, S. 2026. Debate or vote: Which yields better decisions in multi-agent large language mod- els? Advances in Neural Information Processing Systems, 38: 101732–101764. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; et al. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2024. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, 11733– 11763. Fan, Y.; Zhang, J.; Cai, K.; Yang, J.; Tang, C.; Wang, J.; and Wang, K. 2026. Cost-effective communication: An auction- based method for language agent interaction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 29412–29420. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Wang, J.; Zhang, C.; Yau, S.; Lin, Z.; Zhou, L.; et al. 2024. MetaGPT: Meta programming for a multi-agent collabora- tive framework. In International Conference on Learning Representations, volume 2024, 23247–23275. Kim, Y.; Gu, K.; Park, C.; Park, C.; Schmidgall, S.; Heydari, A. A.; Yan, Y.; Zhang, Z.; Zhuang, Y.; Malhotra, M.; et al. 2025. Towards a science of scaling agent systems. arXiv preprint arXiv:2512.08296. Lai, G.; Xie, Q.; Liu, H.; Yang, Y.; and Hovy, E. 2017. Race: Large-scale reading comprehension dataset from examina- tions. In Proceedings of the 2017 conference on empirical methods in natural language processing, 785–794. Liang, T.; He, Z.; Jiao, W.; Wang, X.; Wang, Y.; Wang, R.; Yang, Y.; Shi, S.; and Tu, Z. 2024. Encouraging divergent thinking in large language models through multi-agent de- bate. In Proceedings of the 2024 conference on empirical methods in natural language processing, 17889–17904. Lifshitz, S.; McIlraith, S. A.; and Du, Y. 2025. Multi-agent verification: Scaling test-time compute with multiple veri- fiers. arXiv preprint arXiv:2502.20379. Lin, W.; Roberts, J.; Yang, Y.; Albanie, S.; Lu, Z.; and Han, K. 2025. GAMEBoT: Transparent assessment of LLM rea- soning in games. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7656–7682. Liu, J.; Xia, C. S.; Wang, Y.; and Zhang, L. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Eval- uation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Process- ing Systems. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Sys- tems, 36: 46534–46594. Pan, M. Z.; Cemri, M.; Agrawal, L. A.; Yang, S.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Ramchandran, K.; Klein, D.; et al. 2025. Why do multiagent systems fail? In ICLR 2025 Workshop on Building Trust in Language Models and Applications. Rein, D.; Hou, B. L.; Stickland, A. C.; Petty, J.; Pang, R. Y.; et al. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling. Renze, M.; and Guven, E. 2024. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682. Sainz, O.; Campos, J.; García-Ferrero, I.; Etxaniz, J.; de La- calle, O. L.; and Agirre, E. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, 10776–10787. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language agents with verbal re- inforcement learning. Advances in Neural Information Pro- cessing Systems, 36: 8634–8652. Sun, H.; Zhang, S.; Niu, L.; Ren, L.; Xu, H.; Fu, H.; Zhao, F.; Yuan, C.; and Wang, X. 2025. Collab-overcooked: Bench- marking and evaluating large language models as collabo- rative agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 4922– 4951. Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y.; et al. 2024a. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186345. Wang, W.; Zhang, D.; Feng, T.; Wang, B.; and Tang, J. 2024b. Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi- agent systems. arXiv preprint arXiv:2408.15971. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. Self- Consistency Improves Chain of Thought Reasoning in Lan- guage Models. In The Eleventh International Conference on Learning Representations. Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; et al. 2024c. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. arXiv preprint arXiv:2406.01574. Wu, H.; Li, Z.; and Li, L. 2025. Can LLM Agents Really De- bate? A Controlled Study of Multi-Agent Debate in Logical Reasoning. arXiv preprint arXiv:2511.07784. Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Li, B.; Zhu, E.; Jiang, L.; Zhang, X.; Zhang, S.; Liu, J.; Awadallah, A. H.; White, R. W.; Burger, D.; and Wang, C. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations. In First Conference on Language Modeling. Xi, Z.; Chen, W.; Guo, X.; He, W.; Ding, Y.; Hong, B.; Zhang, M.; Wang, J.; Jin, S.; Zhou, E.; et al. 2025. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2): 121101. Xu, Z.; Shi, S.; Hu, B.; Yu, J.; Li, D.; Zhang, M.; and Wu, Y. 2023. Towards reasoning in large language mod- els via multi-agent peer review collaboration. arXiv preprint arXiv:2311.08152. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Ye, R.; Tang, S.; Ge, R.; Du, Y.; Yin, Z.; Chen, S.; and Shao, J. 2025. MAS-GPT: Training LLMs to Build LLM- based Multi-Agent Systems. In Forty-second International Conference on Machine Learning. Zhu, K.; Du, H.; Hong, Z.; Yang, X.; Guo, S.; Wang, D. Z.; Wang, Z.; Qian, C.; Tang, R.; Ji, H.; et al. 2025. Multiagent- bench: Evaluating the collaboration and competition of llm agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8580–8622. Appendix Benchmark Details This section provides the detailed task settings and scoring functions of the six benchmarks in MASTraceBench. For static tasks, the task score (TS) is obtained by averaging the scores of the final MAS answers over all samples. For dynamic tasks, TS is computed over complete interaction episodes, whereas the proposal scoring function S(·) evalu- ates each action proposed at an intermediate state. MAPF Scenario. Multi-Agent Path Finding (MAPF) is a coopera- tive planning task in which multiple robots move simultane- ously on a shared grid map. Each instance specifies the map boundary, obstacle locations, and the start and goal positions of all robots. Given this information, the MAS produces a joint plan containing one action sequence for each robot. At every time step, a robot may move up, down, left, or right, or remain at its current position. The joint plan is executed for at most 30 steps and is checked for invalid movements and inter-robot collisions. We use 157 independently constructed MAPF instances with varying map layouts and robot con- figurations. A representative sample is illustrated in Fig. 7, showing the grid layout, obstacles, and robot–goal assign- ments. Scoring Definitions. For each MAPF instance x, the pro- posal score is defined as the fraction of robots that reach their designated goals when the proposed joint plan y is executed: S MAPF (y) = # robots reaching their goals # robots in x . (9) This graded score assigns partial credit when only a subset of robots successfully reaches its goals. TS is the mean arrival rate of the final MAS plans over all instances. For reporting, both proposal scores and TS are multiplied by 100. OptionGen Scenario. OptionGen is constructed from RACE (Lai et al. 2017), which originally formulates reading comprehension as a multiple-choice task: given an article, a question, and four candidate options, a model selects the correct answer. We reformulate it into an open-ended option-generation task, where the MAS is provided only with the article and question and must generate a complete set of four options, including one correct answer and three distractors. The correct op- tion should accurately answer the question according to the article, while the distractors should be relevant and super- ficially plausible but remain clearly incorrect under careful reading. They should also reflect distinct misunderstanding patterns rather than trivial paraphrases of one another. We select 235 instances from RACE for evaluation; the original reference options and derived scoring criteria are withheld from the MAS and used only for scoring. Figure 8 presents a representative sample, including the source article, reference question and options, and the instance-specific criteria used for evaluating the generated correct option and distractor set. Instance name: swap_5x5_2r_i1 map_size: 5 scenario_type: bottleneck Robot 0: start (3, 1) → goal (3, 3) Robot 1: start (3, 3) → goal (3, 1) c Map View · Robot–Goal Pairing MAPF Sample · Bottleneck_5×5_2r_i1 0 1 2 3 4 0 1 2 3 4 Row Col Figure 7: A representative MAPF sample with two robots in a 5× 5 bottleneck map. The visualization shows the obstacles, initial robot positions, and corresponding goals. Scoring Definitions. OptionGen adopts criterion- grounded LLM evaluation rather than asking the judge to assign unconstrained holistic scores. For each instance, we first use the original four RACE options and the ground-truth label to construct two instance-specific scoring rubrics. The correct-option rubric specifies the key information that a valid answer should contain and common errors it should avoid, while the distractor rubric specifies why the reference distractors appear plausible, why they remain incorrect, and what properties suitable distractors should satisfy. These rubrics are generated before evaluating MAS outputs and are not provided to the MAS. Following the extracted rubrics, the judge independently assigns a correct-option score s cor (y) ∈ [0, 100] and a distractor-set score s dis (y) ∈ [0, 100]. The former evaluates the correctness, completeness, and consistency of the gen- erated answer with the article, whereas the latter evaluates the plausibility, relevance, incorrectness, and diversity of the three distractors. Both evaluations follow six anchored score intervals, each with explicit descriptions of the expected out- put quality, thereby encouraging consistent application of the scoring criteria. The judge is instantiated with Qwen3-32B, with the temperature set to 0 to ensure deterministic and reproducible scores. The proposal score is S OptionGen (y) = s cor (y) + s dis (y) 2 .(10) TS is the mean score of the option sets produced as the final MAS answers over all evaluation instances. Agreement with Human Evaluation. To validate the LLM judge used in OptionGen, we sample N h = 50 bench- mark instances, stratified by their mean LLM-judge scores ComparisonPearson r Spearman ρ LLM Judge vs. Human Mean0.9190.818 Human–Human Average0.7840.661 Table 5: Agreement between the LLM judge and human eval- uation on 550 OptionGen outputs from 50 sampled bench- mark instances. Human–Human Average denotes the mean pairwise correlation among the three annotators. across methods to cover a broad score range. For each in- stance, we collect the outputs generated by the 11 methods included in this validation, resulting in 550 option sets. Three graduate student annotators independently evaluate each op- tion set using the same two-component rubric and 0–100 score anchors as the LLM judge. The annotators are blinded to both the generating method and the LLM-judge score, and the outputs are presented in randomized order. Following the automatic evaluation protocol, each anno- tator separately scores the quality of the correct option and the distractor set. The two component scores are averaged to obtain the annotator’s score for each output, after which the scores of the three annotators are averaged. We mea- sure human–LLM consistency using Pearson’s correlation and Spearman’s rank correlation between the LLM-judge scores and the mean human scores. As shown in Table 5, the LLM judge achieves a Pearson correlation of 0.919 and a Spearman correlation of 0.818 with the mean human ratings, indicating strong consistency with human assessment. EvidenceChain Scenario. EvidenceChain is a cooperative evidence- reasoning task built around detective stories. Each instance contains a narrative, a question about the underlying case, and a numbered list of candidate facts. Among these facts, only a subset constitutes the evidence needed to resolve the question, while the remaining facts are irrelevant or poten- tially distracting. The MAS is required to identify the IDs of the supporting facts and arrange them in the order in which they should be used for reasoning. The resulting ID sequence should form a coherent evidence chain whose facts, when fol- lowed in order, logically support the answer to the question. Therefore, a valid output must not only select the relevant evidence, but also organize it into a reasoning sequence that connects the given facts to the final conclusion. We con- struct 100 such instances for evaluation. Figure 9 shows a representative sample containing the story, question–answer pair, indexed candidate facts, and the ordered gold reasoning chain. Scoring Definitions. Let P (y) be the ordered list of fact identifiers predicted in proposal y, and let G be the ground- truth evidence chain. We first evaluate whether the proposal contains the correct facts using set-level precision and recall: Prec(P,G) = |P ∩ G| |P| ,Rec(P,G) = |P ∩ G| |G| , (11) from which the set-level F1 score is computed as F set (P,G) = 2 Prec(P,G) Rec(P,G) Prec(P,G) + Rec(P,G) .(12) We additionally compute an ordering score O(P,G) ∈ [0, 1] using a normalized Kendall-style pairwise concordance measure over the facts shared by the predicted and ground- truth chains. Let H = P ∩ G denote the shared facts. The ordering score is the proportion of fact pairs in H whose relative order in P agrees with that in G: O(P,G) = P u,v⊆H I[ord P (u,v) = ord G (u,v)] |H| 2 . (13) The proposal score combines set correctness and ordering consistency: S EvidenceChain (y) = F set (P (y),G) + O(P (y),G) 2 . (14) TS is the mean score of the evidence chains produced as the final MAS answers over all evaluation instances. For reporting, both proposal scores and TS are multiplied by 100. Gomoku Scenario. Gomoku is instantiated as a competitive, se- quential board-game task on a 15 × 15 board. Two MAS methods act as opposing players and alternately place stones on unoccupied intersections, with the player controlling black moving first. The first player to form five consecutive stones horizontally, vertically, or diagonally wins the game. A game terminates when either player forms five stones or when the board is full. To control for first-move bias, each evaluated pairing is played for 50 games, with each MAS method con- trolling black in 25 games and white in the remaining 25. Scoring Definitions. For each completed game z, let p m denote the player controlled by MAS methodm. The episode score of method m is defined as R m Gomoku (z) = 1, if player p m wins game z, 0, otherwise. (15) TS is the mean episode score over all games, which is equiv- alent to the win rate. For reporting, TS is multiplied by 100. Because the binary game outcome does not distinguish the quality of individual moves, we additionally assign each proposed move a local rule-based score. Let a t denote a proposed move by the current player in board state s t . In- valid moves, including out-of-bound positions and occupied intersections, receive a score of zero. The local score consists of two components. The self- formation score F (a t ;s t ) is computed by hypothetically placing the current player’s stone at a t and scoring the re- sulting pattern. The defense-interception score D(a t ;s t ) is computed by hypothetically placing an opponent stone at a t and scoring the pattern the opponent would form there. It therefore measures the threat prevented when the current player occupiesa t instead. The scoring rules are summarized in Table 6. Pattern Open Endpoints Self Defense Five2 / 15 / 5 2 / 5 Four2 / 14 / 2 4 / 3 Three2 / 13 / 2 3 / 2 Other–00 Table 6: Rule-based scoring of proposed Gomoku moves. Paired values correspond to patterns with two and one open endpoints, respectively. Patterns with no open endpoint or at most two consecutive stones receive zero. Benchmark ρ p ρ rp r Gomoku0.884 0.0003 0.945 < 0.0001 TacticDuel 0.882 0.0003 0.951 < 0.0001 Table 7: Correlation between the mean proposal score and TS across the 11 evaluated MAS methods. ρ and r denote Spearman’s rank correlation and Pearson’s correlation, re- spectively. For five-stone threats, the defense score is lower when both endpoints are open because occupying one endpoint does not eliminate the opponent’s remaining winning move. Occupy- ing the only open endpoint, by contrast, directly removes the immediate threat and receives the highest defense score. When a move creates or blocks multiple scorable patterns, their contributions are accumulated. The proposal score is defined as S Gomoku (a t ;s t ) = F (a t ;s t ) + D(a t ;s t ). (16) For reporting, the resulting proposal scores are multiplied by 20 to normalize them to a 0–100 scale. Consistency between proposal score and TS. TS mea- sures game-level performance, whereas the proposal score evaluates individual moves. Because our proposal-trajectory metrics use these move-level scores to identify improvement, preservation, and regression, we need to verify that the scor- ing function captures task-relevant action quality rather than merely reflecting local pattern changes. We therefore com- pute each method’s mean proposal score over all replayed moves and correlate it with its TS. As shown in Table 7, both correlation measures show a strong positive association, in- dicating that methods with higher move-level scores also tend to achieve stronger game-level performance. This pro- vides empirical support for using the graded proposal score in proposal-trajectory diagnosis. Negotiation Scenario. Negotiation is a competitive multi-round resource-allocation task. Two MAS methods negotiate how to divide a shared collection of items. Each player observes the available item quantities and its own private valuation of each item, but does not observe the opponent’s valuations. During each round, one player proposes a complete alloca- tion and the other player decides whether to accept or reject it. If the proposal is accepted, the negotiation terminates with the proposed allocation. Otherwise, the roles are exchanged and the negotiation continues. An episode lasts for at most 10 rounds and uses a discount factor ofδ = 0.95, making de- layed agreements less valuable. If no agreement is reached, both players receive zero payoff. Each scenario is evaluated twice with the player roles and first proposer exchanged to reduce first-mover and role bias. Scoring Definitions. Let u i (a) denote the payoff received by player i under allocation a. We define u max i as the ideal- ized maximum payoff of player i, obtained by assigning all negotiable items to that player and evaluating them according to its private item values. For each proposal a t recorded in the proposal trajectory, we assess whether it would be accepted by the opponent un- der the corresponding negotiation context. Specifically, we retain the original negotiation history, replace the outward MAS proposal with a t , and query the opponent with tem- perature 0 to obtain a deterministic acceptance decision. Let A −i (a t ) ∈ 0, 1 denote this decision. The proposal score from player i’s perspective is S i Neg (a t ) = A −i (a t )δ t−1 u i (a t ) u max i .(17) Thus, an accepted proposal receives its discounted normal- ized payoff, whereas a rejected proposal receives zero. If an agreement on allocation a ⋆ is reached in round t, the episode-level task score of player i is R i Neg (z) = δ t−1 u i (a ⋆ ) u max i .(18) The episode score is zero when no agreement is reached. TS is the mean episode-level task score over all scenarios and both role assignments. For reporting, both proposal scores and episode-level task scores are multiplied by 100. TacticDuel Scenario. TacticDuel is a deterministic and symmetric simultaneous-action combat task. Each player starts with 20 hit points (HP) and zero energy, with no upper bound on accumulated energy. At each round, the two competing MAS methods independently select one action from Quick, Heavy, Guard, Counter, Feint, Charge, and Burst. Nei- ther method observes the opponent’s current selection be- fore making its decision. The two selected actions are then revealed and resolved simultaneously according to determin- istic transition rules. Table 8 summarizes the implemented action effects. Burst is legal only when the acting player has at least two energy, while all other actions have no energy requirement. A player cannot repeat the action selected in the immediately preced- ing round. An episode terminates when either player’s HP reaches zero or below, or when the configured maximum number of rounds R max is reached. If the maximum round limit is reached without either player being defeated, the player with higher remaining HP wins; equal remaining HP results in a draw. For each evaluated method pair, we conduct 50 duels. ActionEnergy cost Base effectConditional interaction Quick0Deals 2 damage.None. Heavy0Deals 4 damage.Deals 0 damage if the opponent selects Counter. Guard0Deals no damage.Reduces incoming non-Feint damage by 4, lower-bounded by 0. Counter0Deals 1 damage.Deals 4 damage if the opponent selects Heavy. Feint0Deals 1 damage.Deals 4 damage against Guard and bypasses its damage reduction. Charge0Gains 2 energy.Deals no damage in the current round. Burst2Deals 6 damage.Its damage can be reduced by Guard. Table 8: Action effects in TacticDuel. The two players’ selected actions are resolved simultaneously in each round. Scoring Definitions. For each completed duel z, let p m denote the player controlled by MAS methodm. The episode score of method m is defined as R m TacticDuel (z) =    1, if player p m wins duel z, 0.5, if duel z ends in a draw, 0, if player p m loses duel z. (19) TS is the mean episode score over all duels. For reporting, TS is multiplied by 100. To support proposal-trajectory analysis, each proposed ac- tion is additionally evaluated against the opponent action that was actually executed in the same round. Let a t denote the evaluated action, a − t the opponent’s realized action, and s t the acting player’s state before the round. We first calculate the immediate net combat payoff ∆(a t ,a − t ) = d(a t ,a − t )− d(a − t ,a t ),(20) where d(a,b) is the damage dealt by action a against action b after applying the interaction rules in Table 8. Thus, the score accounts for both the damage inflicted on the opponent and the damage received by the acting player. The implementation also assigns a fixed energy- management bonus to Charge: B(a t ) = 1, a t = Charge, 0, otherwise. (21) The proposal score for action a t is computed as S(a t ,a − t ) = ∆(a t ,a − t ) + B(a t ).(22) The resulting value ranges from−5 to 6 and is shifted and lin- early scaled to [0, 5]. For reporting, the scores are multiplied by 20 to obtain a 0–100 scale. Consistency between proposal score and TS. Following the same analysis, we correlate each method’s mean proposal score over all replayed actions with its TS in TacticDuel. As shown in Table 7, both correlation measures show a strong positive association. Methods producing actions with higher tactical scores therefore also tend to achieve higher duel win rates, supporting the task relevance of the graded TacticDuel proposal score. Benchmark Task Prompts Figures 11–16 present the task-prompt templates used for the six benchmarks. Each prompt specifies the task input, rules, and decision require- ments, with fields enclosed in braces instantiated using the current sample or environment state. Static benchmarks pro- vide the complete instance in a single prompt, whereas dy- namic benchmarks update state-dependent information at each decision step. The same benchmark prompt is pro- vided to all evaluated methods, while their role-definition and workflow prompts remain method-specific and benchmark- agnostic. Additional Implementation Details Model and Inference Settings All models are deployed using vLLM 0.9.2 and Transformers 4.53.1 on a server equipped with eight NVIDIA GeForce RTX 4090 D GPUs, and are accessed through an OpenAI-compatible API. The sampling temperature is set to 1.0 to encourage diverse out- puts across agents, and the maximum generation length is set to max_tokens= 16,384. For Qwen3-32B, thinking mode is enabled on MAPF and Gomoku due to their higher rea- soning difficulty and disabled on the other four benchmarks. Main-Experiment Method Configurations All methods use the common model and inference settings described above, and their method-specific configurations remain fixed across benchmarks. For each method, the role-definition and workflow prompts are benchmark-agnostic: the same coor- dination protocol is applied to all benchmarks, while task- specific information is provided only through the benchmark input and output specification. This design decouples MAS coordination strategies from benchmark-specific logic and enables controlled comparison across tasks. Single-agent and selection-based methods. Single-Agent directly generates one response without collaboration. Self- Consistency independently samples three proposals without inter-proposal interaction and selects the final response by majority voting. MAV also generates three independent pro- posals, but selects among them by external validation rather than output-frequency voting. Five functional validators in- dependently assess each proposal in terms of correctness, completeness, clarity, reasoning, and relevance, and the pro- posal receiving the most approvals is selected. Debate-based methods. Int. Debate uses three agents and one interaction round. Each agent first generates an indepen- dent proposal, observes the current proposals of the other two agents, and then either retains or revises its own answer. The final proposals are aggregated by majority voting. Agg. Debate uses the same three-agent, one-round interaction pro- cess, but replaces voting with an independent LLM aggrega- tor. The aggregator is instructed to select the strongest final proposal and reproduce it without modification. Adv. De- bate assigns two agents the affirmative and negative roles. The negative agent is initially required to challenge the af- firmative proposal and provide a distinct alternative. After two rebuttal rounds, an independent judge evaluates the final positions and produces the system response. Revision-based methods. Reflection uses a reasoner and a reflector. The reasoner first generates an initial response, af- ter which the reflector either approves it or provides revision feedback. We allow at most two reflection–revision cycles and terminate early upon approval. MAPR uses three agents and one critique–revision round. Each agent reviews the so- lutions of the other two agents, receives both peer reviews, and updates its own solution before majority voting. Adaptive MAS construction. AgentVerse uses a vertical collaboration structure consisting of one solver and two crit- ics. Expert roles are dynamically assigned rather than fixed across rounds. In each iteration, the critics independently re- view the current solution, and the solver integrates feedback that identifies errors or potential improvements. An evaluator then scores the revised solution, and the procedure terminates when the score reaches 8 out of 10 or after three refinement rounds. MAS-GPT uses a separately fine-tuned MAS-GPT-32B model to generate a task-specific multi-agent program, which is subsequently executed with Qwen3-32B as the task-solving model. Dynamic MAS generation is enabled in the main experiments. If program generation or execution fails, the fallback procedure generates three candidate solutions and uses an LLM aggregator to produce the final response. CLEARS and its ablation. CLEARS uses three proposer agents, with each proposal decomposed into 2–4 verifiable claims. For synthesis, CLEARS selects the claim with the highest number of endorsements as the positive signal and the claim with the highest number of challenges as the neg- ative signal. If multiple claims receive the same highest count, all tied claims are retained. CLEARS w/o CCE uses the same proposer and claim-decomposition settings, but re- moves cross-agent claim evaluation and directly provides all extracted claims to the synthesizer. Additional Experimental Results Output Validity and Final-Proposal Agreement To as- sess the potential influence of output-format and extraction errors on proposal-trajectory analysis, we report Output Va- lidity (OV). Let Y m denote all evaluated outputs produced by method m, including the initial and final proposals of Proposer Agents and the final MAS answers. For each output y ∈ Y m , let v(y) = 1 if the required proposal or answer can be correctly extracted and scored by the corresponding scoring function S(·), and v(y) = 0 otherwise. We define OV m = 1 |Y m | X y∈Y m v(y).(23) OV measures the proportion of recorded outputs that are successfully converted into valid, scoreable proposals or MAS answers, rather than their semantic quality. To im- prove extraction reliability, we specify structured JSON out- put requirements and implement robust benchmark-specific extraction mechanisms. As shown in Table 9, OV ranges from 93.4% to 100.0% across all method–benchmark set- tings. These consistently high values indicate that malformed outputs and extraction failures have little influence on the re- ported proposal-trajectory diagnostics. We further report Final-Proposal Agreement (FPA) to quantify proposal convergence before aggregation. Let y final i (x) denote the final proposal produced by Proposer Agent i∈ P m for evaluation unit x. FPA is defined as FPA m = E x∈D I y final i (x) = y final j (x), ∀i,j ∈ P m . (24) We report FPA only for methods with multiple comparable final proposals; otherwise, it is marked as not applicable. The results further support Observation 4 in the main pa- per. Consistency exhibits only 3.5% final-proposal agree- ment on MAPF, whereas Int. Debate increases it to 58.2%. Similar differences appear across the other benchmarks. Agg. Debate achieves agreement levels broadly comparable to Int. Debate, while MAPR has lower FPA than Int. Debate on all six benchmarks. These results reinforce that aggregation loss depends strongly on whether interaction brings Proposer Agents toward a shared final proposal, rather than on the ag- gregation rule alone. Full Agent Refinement Results Figure 10 presents the AR distributions on the four benchmarks not shown in the main paper, complementing the results on EvidenceChain and Gomoku. The overall patterns are consistent with Obser- vation 3. For initially weak proposals, interaction frequently moves them toward the strongest initial proposal, mainly through Reach and Partial, whereas Surpass remains rel- atively limited. For initially strong proposals, OptKeep and SubKeep account for most outcomes, while Improve is gen- erally infrequent and Regress remains non-negligible. Compared with Int. Debate, MAPR tends to revise propos- als more aggressively. This can increase upward refinement among initially weak proposals, but it also produces higher regression ratios for initially strong proposals, particularly on TacticDuel. Overall, the additional results reinforce that ex- isting interaction methods are more effective at lifting weaker proposals than at reliably improving and preserving already strong ones. Full Token Cost Results Table 10 complements the token- cost results reported in the main paper. The additional benchmarks exhibit a similar overall pattern: interactive MAS methods require substantially more tokens than Single- Agent, while higher token usage does not necessarily produce stronger collaboration gains. MAPR consistently incurs the largest cost among the evaluated methods. CLEARS also requires a relatively large token budget, much of which is introduced by cross-agent claim evaluation. Removing CCE reduces its token cost by approximately half across the four benchmarks, while the main results show that CCE consis- tently improves CG. These results further support the main- text analysis of how additional computation is converted into collaboration gains. Method MAPFOptionGenEvidenceChainGomokuNegotiationTacticDuel OV FPA OVFPAOVFPAOV FPA OVFPAOVFPA Single-Agent98.1–100.0–100.0–97.0–99.5–100.0– Consistency97.7 3.5 100.0 23.0 100.09.097.0 35.5 99.96.0 100.0 67.2 MAV99.6 6.0 100.0 19.1 99.77.799.5 32.3 100.0 11.5 100.0 66.2 Int. Debate99.6 58.2 99.4 77.2 100.050.798.3 88.8 99.7 75.0 100.0 89.7 Agg. Debate100.0 71.3 98.2 79.2 100.048.098.6 86.4 99.9 81.5 100.0 90.2 Adv. Debate98.9 87.2 99.3 62.4 100.020.793.4 79.6 99.66.2 100.0 20.8 Reflection98.9–100.0–100.0–96.2–99.7–100.0– MAPR98.9 25.1 99.9 51.4 100.029.797.5 57.3 99.7 19.3 100.0 58.2 AgentVerse94.5–100.0–100.0–95.2–99.2–100.0– MAS-GPT100.0–97.4–99.5–97.9–99.6–100.0– CLEARS99.2 4.7 99.9 21.0 100.09.099.1 32.4 99.46.6 100.0 100.0 CLEARS w/o CCE 99.8 5.1 99.3 22.3 100.07.799.7 34.8 99.99.5 100.0 65.1 Table 9: Output Validity (OV, %) and Final-Proposal Agreement (FPA, %) across methods and benchmarks. A dash indicates that FPA is not applicable or is trivially determined by the method configuration. MethodOptionGenEvidenceChainNegotiationTacticDuel Single-Agent1.00×1.00×1.00×1.00× Consistency3.00×2.99×2.94×2.96× MAV20.46×20.64×19.69×21.60× Int. Debate8.85×8.09×7.98×7.37× Agg. Debate10.92×9.82×9.62×8.98× Adv. Debate15.84×13.60×16.12×15.34× Reflection3.09×7.12×8.10×8.47× MAPR35.73×30.54×29.54×36.21× AgentVerse9.10×10.31×9.32×13.87× MAS-GPT6.84×6.79×7.84×6.69× CLEARS24.18×21.09×18.97×22.29× CLEARS w/o CCE11.21×10.12×9.26×10.37× Table 10: Additional token cost multipliers over the Single-Agent baseline on the four benchmarks not included in the main-text token cost results. Article Whales live in the ocean, but they are not fish. They are warm-blooded mammals and must have air to breathe. The babies are born alive and drink milk from the mother's body. It took millions of years for the whale to develop as it is today. Every spring groups of whales swim hundreds of kilometers to warm places to bear their young. Each group goes to the same place every year. The mothers take very good care of their babies. The fully grown whales are very gentle and playful with each other, and with humans. They "talk" to each other with a high noise that sounds beautiful. This "talking" can be heard for more than 300 kilometers in open waters. Yet humans kill over a hundred whales a day. They kill them to make whale oils and a lot of other things. Killing them is a modern business with modern machines and even harpoons that explode inside the whales. A whale dies very slowly and in great pain. Some ships kill every whale they can find, even the mothers and babies. Then there are no young whales to grow up or to bear more young. Question and Options (ground truth) Question: Every spring each group of whales A. return to the same place as they did before (correct option) B. swim only a few kilometers C. go to different places D. go to cold places to bear their young Distractor Criteria Why they seem correct: •Distractors may seem correct because they involve plausible whale behaviors that a student might infer (e.g., swimming distances, choosing warm or cold places). Why they are wrong: •Distractor 1 is incorrect because the passage says whales swim 'hundreds of kilometers' not 'only a few kilometers'. •Distractor 2 is incorrect because the passage clearly states that each group goes to the same place, not different ones. what_to_avoid: ...... Correct-Option Criteria Why correct: •The passage states: 'Each group goes to the same place every year.' This directly supports the correct answer that each group of whales returns to the same place. •The word 'return' implies a repetition of a previous action, which aligns with the passage's mention that the groups go to the same place annually. What to avoid: •Avoid using vague or general statements that do not directly reference the passage's specific details. •Avoid confusing the purpose of the journey (e.g., mixing up why they go there vs. where they go). OptionGenSample · IDX 84 【 FILL HERE 】 . Figure 8: A representative OptionGen sample. In addition to the source article and reference question and options, each sample provides instance-specific criteria describing the de- sired correct option and the plausibility and incorrectness of the distractors. Story 'I couldn't wait to be announced,' said George Collins, Florida's foremost prohibition agent, and a great wit, as he stepped into the Professor's office.'How long are you going to be in New York?' asked Fordney, as the two friends shook hands.'I'm due back in Miami now,' replied Collins, 'but I wanted to congratulate you on your success in the Hicks murder case. I wish interesting things like that would happen in my game. However, I did have an amusing experience last December.'Sneaky Joe, a stool-pigeon, tipped me off to a huge still he said was working about forty miles from headquarters. A narrow road through dense woods led to the spot. Arriving there, I found an old dilapidated shanty screened by trees. As I entered the woods, I smelled alcohol. Sneaky Joe was right, after all, I thought, as I drove up to the shanty.'I got out and peered cautiously around, but the place seemed deserted. After opening the door and entering the house, however, I knew liquor was not being made there. I searched the woods, but found nothing. As I was driving back along the road at a good rate, I discovered the alcohol I smelled was coming from my own radiator! Imagine my chagrin!'The Professor laughed heartily and said to his friend, 'Stay over for my birthday party tonight, the ladies would love to hear that yarn of yours.' Question and Answer Question: What did the Professor find preposterous in Collins's story? Answer: Not even a prohibition agent would use alcohol in an automobile radiator in or about Miami! Gold Reasoning Chain reasoning_chain: [6,3,1,0] Candidate Facts [0] The area around Miami is warm, so an automobile radiator would not need alcohol as antifreeze; a prohibition agent especially would not use alcohol in his car. [1] Collins claimed that the smell of alcohol actually came from his own automobile radiator. [2] Sneaky Joe told Collins that a huge still was operating about forty miles from headquarters. [3] Collins said the experience happened last December, about forty miles from headquarters, meaning near Miami. [4] Sneaky Joe's tip and the wooded road make the story sound like a genuine still investigation, but they do not explain the radiator claim. [5] Collins entered Fordney's office without waiting to be announced, showing that he and Fordney were familiar with each other. [6] Collins was Florida's foremost prohibition agent, and he said he was due back at headquarters in Miami. [7] Collins searched the shanty and the woods but found no liquor-making operation. EvidenceChain Sample · IDX 2 Figure 9: A representative EvidenceChain sample. The task requires identifying and ordering the relevant facts from the indexed candidate set, while the gold reasoning chain pro- vides the reference ordering used for evaluation. AR-StrongAR-Weak 0%20%40%60%80%100%0%20%40%60%80%100% MAPF Int.Debate MAPR 0%20%40%60%80%100%0%20%40%60%80%100% OptionGen Int.Debate MAPR Surpass Reach Partial Keep Degrade Improve OptKeep SubKeepRegress 73%19%54%37% 72% 12% 14% 50%36% 18%37%42%18%29%22% 21% 16% 34%43%17%30%17%23% 15% 0%20%40%60%80%100%0%20%40%60%80%100% Int.Debate MAPR 0%20%40%60%80%100%0%20%40%60%80%100% TacticDuel Int.Debate MAPR 89%41%37% 81% 14% 36%23% 41%49% 21% 64%28% 26%43%21%53%34% Negotiation Figure 10: Agent Refinement distributions of Int. Debate and MAPR on MAPF, OptionGen, Negotiation, and TacticDuel. ## Task: Multi-Robot Path Coordination Generate a COMPLETE action plan for ALL timesteps at once. ## Optimization Goals (in priority order): 1. **Collision-free**: ALL robots must reach their goals WITHOUT any collision 2. **Efficient**: Minimize the total number of timesteps needed Note: Goal 1 has HIGHER priority than Goal 2. Never sacrifice safety for speed. ## Collision Rules (collisions checked AFTER each timestep): ### 1. Robot Collision (SAME position at SAME timestep) ### 2. Crossing Collision (robots PASS THROUGH each other in SAME timestep) ### 3. Reached-Robot Collision(Once a robot reaches its goal, other robots CANNOT move into that position to avoid robot collision) ### 4. Boundary/Obstacle Collision: Moving outside map bounds (e.g., robot at row 0 cannot move "up"); Moving into a position with an obstacle ## Goal Reaching Rule: Once a robot reaches its goal position, it STOPS and will NOT execute any further actions: ## Strategy Tips: Individual shortest path ≠ Global optimal solution -When paths conflict, some robots may need to take a detour or wait: -If two robots would collide on their direct paths, one must yield -Yielding options: take a detour (go around) OR wait for the other to pass -A longer individual path can lead to a shorter overall plan Task Prompt-MAPF Figure 11: Task-prompt template for MAPF, including the op- timization priorities, collision rules, and coordination guid- ance. ## Task: Generate Multiple-Choice Options Read the following article and generate four options for the given question. ### Article: article ### Question: question ### Your Task: Generate exactly 4 options for this question: - 1 correct answer (the best answer based on the article) - 3 distractors (plausible but incorrect answers) Requirements for distractors: - Must be related to the article content - Must seem reasonable but be definitively incorrect - Should test different aspects of understanding Task Prompt-OptionGen Figure 12: Task-prompt template for OptionGen, specifying the generation of one correct option and three plausible but incorrect distractors. ## Task: Identify the Supporting Evidence Chain Read the story, the question, and the numbered candidate facts below. Among these candidate facts, some can support answering the question, while others may be irrelevant. You need to judge for yourself which facts are truly supportive. ### Story: story ### Question: question ### Candidate Facts: facts ### Your Task: 1. Identify which fact IDs constitute the supporting evidence chain that can answer the question. 2. Arrange the supporting fact IDs in logical reasoning order (the order in which they should be combined to reach the answer). 3. The evidence chain must be **complete** (include all facts necessary for the reasoning), **minimal** (exclude any fact that is not essential), and **reasonable** (the logical order must coherently lead to the answer). Task Prompt-EvidenceChain Figure 13: Task-prompt template for EvidenceChain, requir- ing a complete, minimal, and logically ordered supporting evidence chain. ## Gomoku (Five in a Row) You are playing as **player_name**. ### Board Legend: - `your_stone` = YOUR stones (You control these) - `opponent_stone` = OPPONENT's stones (Do NOT place here) - `·` = Empty positions (You can place here) ### Rules: - Board: board_sizexboard_size grid - Black (○) moves first, White (●) moves second - First to get 5 stones in a row wins (horizontal, vertical, or diagonal) ### Winning Directions (EQUALLY IMPORTANT): -You can win by connecting 5 stones in ANY of these 4 directions: -1. **Horizontal** (→): Same row, consecutive columns (e.g., (7,3)→(7,4)→(7,5)→(7,6)→(7,7)) -2. **Vertical** (↓): Same column, consecutive rows (e.g., (3,7)→(4,7)→(5,7)→(6,7)→(7,7)) -3. **Diagonal **: Both row and column increase together (e.g., (3,3)→(4,4)→(5,5)→(6,6)→(7,7)) -4. **Diagonal **: Row increases, column decreases (e.g., (3,11)→(4,10)→(5,9)→(6,8)→(7,7)) ### Current Board: -board ### Your Turn: -You are **player_name**. Choose an EMPTY position (`·`) to place your `your_stone` stone. -**CRITICAL**: You can ONLY place on empty positions marked with `·`. -Do NOT choose positions with `your_stone` (already your stone) -Do NOT choose positions with `opponent_stone` (opponent's stone) ### Think strategically: -1. Can you win immediately? (Make 5 of YOUR `your_stone` stones in a row in ANY direction) -2. Do you need to block opponent's winning threat? (Check ALL 4 directions for their line of 4 `opponent_stone`) -3. Can you create a strong formation? (Build "open three" - 3 stones with both ends empty, or "open four" - 4 stones with both ends empty, which are hard to block) Task Prompt-Gomoku Figure 14: Task-prompt template for Gomoku, including the current board, legal-placement constraints, and tactical pri- orities. ## Negotiation Game -Two players COMPETE to divide n item types: item_list. -Each player has PRIVATE valuations — opponent does NOT know your values. -Your SOLE objective: maximize YOUR payoff. Every point the opponent gets is a point you didn't get. ### Available Items -pool_str ### Your Private Valuations -val_str ### Payoff Calculation -Your payoff = sum of (items you get) × (your value for each item) -Your max possible payoff = sum of (pool quantity × your value) for all items — what you'd get if you took everything. -discount_info ### Rules -Maximum max_rounds rounds -Each round: one proposes allocation, other accepts/rejects -If reject: responder becomes proposer next round -No agreement by round max_rounds → both get 0 -Valid proposal: the items P1 gets + items P2 gets = total pool -Example: if pool has A×3, B×2, then valid: P1 gets A×2,B×1 and P2 gets A×1,B×1 ### Strategic Framework -Before every decision, you MUST calculate: -1. Your payoff for the proposal on the table -2. Your max possible payoff (all items × your values) and what share of it you're getting -3. Your BATNA (Best Alternative To Negotiated Agreement): if you reject, you propose next round — what could you realistically get? ###Key principles: -Exploit information asymmetry: opponent doesn't know your values. Items you value highly may seem worthless to them — claim those -Anchor high: open with an ambitious offer. You can always concede, but you can never take back what you give away -Don't accept just because an offer is "reasonable" — only accept when it's genuinely hard to improve upon -Rejecting is not failure: if your proposal is rejected, you become proposer next round and control the agenda -Infer opponent's preferences from their proposals and rejections — this tells you what they value Task Prompt-Negotiation Figure 15: Task-prompt template for Negotiation, including private valuations, payoff calculation, interaction rules, and strategic guidance. ## TacticDuel You are **player_name** in a deterministic, symmetric, simultaneous-action tactical duel. ### Game Rules - Each player starts with HP=20, Energy=0 (no cap). - Each round, both players simultaneously choose ONE action. - Game ends when a player's HP <= 0 or after state['max_rounds'] rounds. - Winner: higher HP; tie if equal. ### Action Space | Action | Cost | Effect | |---------|------|--------| | Quick | 0 | Deal 2 damage. | | Heavy | 0 | Deal 4 damage; deals 0 if opponent uses Counter. | | Guard | 0 | Reduce non-Feint damage taken by 4 (min 0). | | Counter | 0 | Deal 4 damage if opponent uses Heavy; otherwise deal 1. | | Feint | 0 | Deal 1 damage; deal 4 if opponent uses Guard (ignores Guard reduction). | | Charge | 0 | Gain 2 Energy (no cap); deal 0 damage this round. | | Burst | 2 Energy | Consume 2 Energy, deal 6 damage (can be reduced by Guard). | ### Legality Rule - Burst requires at least 2 Energy; all other actions are always legal. ### Current State (Round state['round']/state['max_rounds']) - Your HP: state['my_hp'] | Your Energy: state['my_energy'] | Your Last Action: state['my_last_action'] or 'None (first round)' - Opponent HP: state['opp_hp'] | Opponent Energy: state['opp_energy'] | Opponent Last Action: state['opp_last_action'] or 'None (first round)' ### Your Legal Actions This Round legal_str ### Round History history_str if history_str else "No rounds played yet." ### Strategy Tips - Counter punishes Heavy but is weak otherwise (only 1 damage). - Feint punishes Guard but is weak otherwise (only 1 damage). - Heavy is strong (4 damage) but risky if opponent might Counter. - Guard blocks most attacks completely (reduces 4 damage) but is vulnerable to Feint. - Charge sacrifices immediate damage for future Burst potential. - Burst (6 damage) is the strongest attack but costs 2 Energy. - You cannot repeat your last action — plan your sequences carefully. Task Prompt-TacticDuel Figure 16: Task-prompt template for TacticDuel, including the current combat state, action effects, legal actions, and round history.

Instant research discovery