Paper deep dive
ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures
Muhammad Ashar Ishfaq, Glaucia Melo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/16/2026, 7:32:11 AM
Summary
The paper introduces ToMAS, a pilot benchmark and pipeline for converting diagnosed inter-agent misalignment failures (FC2) from the MAST-Data dataset into functional partner-state reasoning items. The authors apply four convertibility criteria to filter execution traces, producing 39 'Clean' items from 242 eligible non-AG2 traces. They conduct a small-scale Group Relative Policy Optimization (GRPO) experiment using Qwen2.5-1.5B to test if these items can serve as binary rewards. The experiment fails to show a training effect because the LoRA updates were numerically negligible, resulting in identical decoding to the untrained checkpoint. The paper highlights limitations such as provenance gaps and lexical-overlap scoring issues, proposing ToMAS as a preliminary rubric for failure-grounded Theory-of-Mind training.
Entities (10)
Relation Signals (7)
FC2 â isa â inter-agent misalignment
confidence 96% · FC2 refers to inter-agent misalignment: cases where agents fail to coordinate because they do not correctly track, interpret, or respond to other agentsâ roles, knowledge, or intentions.
ToMAS â uses â MAST-Data
confidence 95% · ToMAS applies four explicit convertibility criteria to diagnosed execution traces from MAST-Data.
ToMAS â converts â FC2 traces
confidence 94% · ToMAS converts each retained trace into an item asking what a peer knew and what the acting agent should therefore have done.
GRPO â isusedwith â Qwen2.5-1.5B
confidence 93% · We then used the converted items as binary rewards in a small-scale GRPO feasibility experiment with Qwen2.5-1.5B.
LoRA â produces â negligible updates
confidence 92% · the LoRA update remained numerically negligible (max abs Delta W about 7e-6)
ToMAS â uses â ROUGE-L
confidence 91% · For ToMAS-RL training we use a single transparent reward: binary ROUGE-L F1 match on Clean training items
AG2 â isexcludedfrom â ToMAS conversion
confidence 88% · A full conversion pass over 242 eligible non-AG2 training traces produced 39 CLEAN items.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-based multi-agent systems can fail even when communication succeeds because agents do not correctly track their peers' roles, knowledge, or intentions. We investigate whether such inter-agent misalignment cases, labelled FC2 in MAST-Data, can be converted into functional partner-state reasoning items. ToMAS applies four explicit convertibility criteria to diagnosed execution traces. A full conversion pass over 242 eligible non-AG2 training traces produced 39 CLEAN items. In an 18-trace reliability pilot, two annotators achieved 94.4% raw agreement and Cohen's kappa = 0.92. We then used the converted items as binary rewards in a small-scale GRPO feasibility experiment with Qwen2.5-1.5B. On a 28-item held-out Magentic GAIA diagnostic, every evaluated condition exceeded the ROUGE-L threshold on the same 2 of 28 items. Post-hoc adapter checks show why: under the learning rate used, the LoRA update remained numerically negligible (max abs Delta W about 7e-6), so all conditions decode identically to the untrained checkpoint. The experiment therefore does not show a training effect and cannot establish one; it reports an executable pipeline together with two limitations that any conclusive study must address: a provenance gap between the training and evaluation items, and lexical-overlap scoring. ToMAS provides a preliminary rubric and pipeline for converting diagnosed coordination failures into trainable partner-state reasoning items and identifies the requirements for a conclusive matched-domain evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2609.16986v1
- Canonical: https://arxiv.org/abs/2609.16986v1
Trouble viewing inline? Open PDF directly â
Full Text
41,809 characters extracted from source content.
Expand or collapse full text
ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures Muhammad Ashar Ishfaq1 and Glaucia Melo2 Affiliation: 1The Islamia University of Bahawalpur, Pakistan s24barin1m01132@iub.edu.pk Affiliation: 2Toronto Metropolitan University, Toronto, Canada glaucia@torontomu.ca Abstract LLM-based multi-agent systems can fail even when communication succeeds because agents do not correctly track their peersâ roles, knowledge, or intentions. We investigate whether such inter-agent misalignment cases, labelled FC2 in MAST-Data, can be converted into functional partner-state reasoning items. ToMAS applies four explicit convertibility criteria to diagnosed execution traces. A full conversion pass over 242 eligible non-AG2 training traces produced 39 Clean items. In an 18-trace reliability pilot, two annotators achieved 94.4% raw agreement and Cohenâs Îș=0.92Îș=0.92. We then used the converted items as binary rewards in a small-scale GRPO feasibility experiment with Qwen2.5-1.5B. On a 28-item held-out Magentic GAIA diagnostic, every evaluated condition exceeded the ROUGE-L threshold on the same 2 of 28 items. Post-hoc adapter checks show why: under the learning rate used, the LoRA update remained numerically negligible (maxâĄ|ÎâW|â7Ă10â6 | W|â 7Ă10^-6), so all conditions decode identically to the untrained checkpoint. The experiment therefore does not show a training effect and cannot establish one; it reports an executable pipeline together with two limitations that any conclusive study must address: a provenance gap between the training and evaluation items, and lexical-overlap scoring. ToMAS provides a preliminary rubric and pipeline for converting diagnosed coordination failures into trainable partner-state reasoning items and identifies the requirements for a conclusive matched-domain evaluation. Index Terms: multi-agent systems, large language models, Theory of Mind, reinforcement learning, failure taxonomy I Introduction LLM-based multi-agent systems distribute complex work across specialized agentsâfor software engineering [1, 2], mathematical reasoning [3], and general-purpose automation [4]. The premise is that role specialization enables decomposition, cross-checking, and stronger collective performance than a single agent. Empirically, coordination frequently breaks down when agents hold different knowledge and responsibilities. Analysis of 1,642 annotated execution traces across seven frameworks identifies inter-agent misalignment as a persistent failure mode that survives structural fixes to other coordination surfaces [5]. Coordination failures in multi-agent LLM systems have drawn two largely separate research responses. Failure taxonomies characterize when coordination breaks downâagents misalign on roles, knowledge, or intentionsâfrom annotated execution traces or controlled fault injection [5, 6, 7]. ToM benchmarks and RL methods train social reasoning for coordination in abstract games, embodied households, and cooperative MARL environments [8, 9, 10, 11, 12, 13]. Diagnostic work typically stops at recommending system changes; ToM and RL training typically use authored tasks rather than FC2 failures from observed execution traces [5, 9, 12]. None uses MAST-Data FC2 cases as the joint source for benchmark items and RL reward signals [5]. Diagnosed coordination failures seldom become the explicit objects against which agents are trained and evaluated. Selected diagnosed FC2 failures can serve as supervision rather than only as explanation. The subset of MAST FC2 failures that can be represented as functional partner-state reasoning problems can supply benchmark items and RL rewards from the same source. FC2 refers to inter-agent misalignment: cases where agents fail to coordinate because they do not correctly track, interpret, or respond to other agentsâ roles, knowledge, or intentions. FC2 is broader than Theory of Mind; other FC2 labels, such as tool-loop or protocol failures without a peer belief or knowledge target, cannot be represented as partner-state items. This paper investigates two research questions: 1. RQ1: Which MAST FC2 inter-agent misalignment traces can be converted into functional partner-state reasoning items under explicit inclusion criteria, and how does convertibility vary across the available frameworks and model outputs? 2. RQ2: What does a small-scale GRPO feasibility experiment reveal about the use of converted FC2 items as binary reward signals and their generalization to a separately sourced evaluation domain? We introduce ToMAS, a pilot benchmark-construction and training pipeline that filters FC2-labeled traces from MAST-Data through four explicit convertibility criteria. ToMAS converts each retained trace into an item asking what a peer knew and what the acting agent should therefore have done. The paper makes three contributions: âą a four-criterion rubric for determining whether an FC2 trace contains an isolable functional partner-state reasoning problem; âą an empirical conversion analysis of 242 eligible non-AG2 training traces, producing 39 confirmed Clean items; and âą a small-scale GRPO diagnostic showing that the resulting reward pipeline can be executed end to end, together with a transparent post-hoc account of why this run cannot support a training claim: the optimization left the policy effectively unchanged, so the held-out comparison is uninformative about transfer. The purpose of this pilot is therefore to establish the feasibility and boundaries of failure-grounded benchmark construction. We do not claim that ToMAS solves multi-agent coordination or that the present experiment demonstrates a reduction in FC2 failures. Establishing such an effect requires a larger benchmark, a matched-domain held-out evaluation, and correctness measures that go beyond lexical overlap. I Related Work I-A Multi-agent complexity Multi-agent LLM systems are built on the premise that dividing work across specialized roles lets a system decompose tasks and cross-check its own outputs [1, 2, 3, 4]. Under matched cost and compute, the simpler baseline generally wins [14, 15, 16]. Empirically, this promise has not materialized at scale. A simple agentless pipeline achieves among the highest open-source performance on SWE-Bench Lite at lower cost [14]. Best-of-N sampling frequently matches or outperforms multi-agent designs under equalized budget [15]. Adding agents can degrade outcomes once coordination overhead compounds [16]. These results do not merely recommend fewer agents. They point at coordinationânot model capabilityâas the suspect: agents still fail when knowledge and responsibility are distributed [5]. Understanding why coordination breaks, not merely that it does, is the precondition for any training intervention. I-B Diagnosis: coordination as the bottleneck Systematic diagnosis has since located that bottleneck. A 14-mode taxonomy derived from 1,642 annotated execution traces across seven frameworks shows that most failures trace back to specification and coordination defects rather than base-model weaknesses [5]. Within this taxonomy, FC2 denotes inter-agent misalignment: cases where agents fail to coordinate because they do not correctly track, interpret, or respond to other agentsâ roles, knowledge, or intentions. Structural interventions do work on other surfaces. Closed-loop architectures neutralize over 40% of faults that cause catastrophic collapse in linear pipelines [6]. Encoded role specifications and SOPs reduce design-level failures [1, 2]. Yet inter-agent misalignment survives them. Even when agents successfully exchange messages, they often fail to synthesize distributed information. The breakdown is localized to reasoning integration rather than message exchange [7]. Diagnosis therefore names the residue. Diagnosis has not, so far, been trained against it. I-C Partner-state reasoning, and the literal/functional split One candidate explanation for that residue is a capability gap rather than a design gap. Models score above 80% on comprehension of a coordination environment while falling below 40% on questions requiring them to model a partnerâs knowledge [8, 17]. Strong literal belief prediction does not imply functional use of that belief when choosing an action [10, 18]. We do not equate all FC2 with ToM. We hypothesize that some FC2 failures reflect deficits in functional partner-state reasoning. Other FC2 cases may be protocol, tool-use, or design issues without a peer belief or knowledge target. Frontier models can report partner beliefs and still score 0% on hard functional ToM splits in embodied coordination [9]. Related work on zero-cost collaboration shows that capable models can still coordinate poorly when helping is free [18]. This is a cooperation deficit distinct from FC2 misalignment under distributed knowledge. This evidence comes from abstract games, isolated probes, or embodied households. The evidence does not come from the convertible subset of MAS execution failures documented in diagnostic research [5]. The subset qualifier is earned here: we do not treat every FC2 label as a ToM failure. I-D Training social reasoning: a toolkit with an authored reward Reinforcement learning has become the standard route to capabilities that supervised fine-tuning does not reach [19]. Applied to social reasoning, it improves cooperation through step-wise credit assignment [11], embodied functional-ToM objectives [9], language-level credit assignment [12], and explicit partner modeling before action selection [13]. These methods provide useful training tools. Their rewards are authored from constructed environments rather than observed MAS execution failures. ToMAS treats selected labelled failures as supervision. ToMAS converts cases with an identifiable peer state into benchmark items and RL rewards, under explicit inclusion criteria [5]. Structural repair and generic social-reasoning training remain necessary for other failure modes. I Methodology We organize the methodology around benchmark construction (RQ1) and the exploratory training diagnostic (RQ2). Source traces and convertibility criteria determine which FC2 failures can produce partner-state items. Item construction and reward specification define the ToMAS training signal. The training conditions and held-out evaluation assess whether the current experimental design can support a comparative training claim. Figure 1 presents an overview of our methodology. Fig. 1: ToMAS workflow. Benchmark construction filters FC2+ training traces into 39 Clean partner-state items (RQ1). The items are then used as binary ROUGE-L rewards in a small-scale GRPO experiment. A separately sourced held-out pool is used to diagnose the limitations of the present training and evaluation setup (RQ2). I-A Source traces We use the public Hugging Face release of MAST-Data [5] (mcemri/MAST-Data, accessed 27 August 2026), which contains 1,642 annotated traces across seven MAS frameworks (AG2, ChatDev, MetaGPT, AppWorld, HyperAgent, Magentic, OpenManus) and released model families including GPT-4o, Claude, Qwen, and CodeLlama. Only FC2-positive traces (failure modes 2.1â2.6) are candidates for conversion. The pilot and AG2-scoping decisions use a locked Step 1 train/hold-out inventory of train FC2-positive n=423n=423 (Table I) and hold-out FC2-positive n=209n=209 (Table VI) under an AG2-by-benchmark split: train units are AG2/GSM, AG2/Olympiad, ChatDev, MetaGPT, AppWorld, and HyperAgent; hold-out units are Magentic, OpenManus, and AG2/MMLU. Hold-out status is inferred from that framework-based split; MAD_full_dataset.json contains no explicit train/test field. The conversion pilot samples only from the train pool. Table I is the scoping inventory for the pilot and for AG2 exclusion; it is not the MetaGPT sampling frame for the later conversion pass. The locked conversion pass then codes non-AG2 traces under the same AG2-by-benchmark train definition, but expands MetaGPT coding across the four released model families available in the working extract (GPT-4o, Claude, Qwen, CodeLlama). Table I is therefore the source of truth for conversion counts: MetaGPT contributes 217 coded traces there, not the Step 1 MetaGPT cell of 100. The two counts differ because Step 1 was scoped for the pilot sample rather than for full conversion: it drew MetaGPT traces from a single model family, whereas the conversion pass covers all four MetaGPT model families present in the working extract (the locally cached subset of MAD_full_dataset.json used throughout, released with the deposit). The MetaGPT FC2+ train pool in that extract totals 217 traces, all of which were coded. AG2 GSM/Olympiad remains documentation-only Structural Mismatch under both tables. I-B Convertibility criteria An FC2-labeled trace enters the ToMAS conversion pool only if all four of the following hold: (a) a peer agent has an identifiable role, knowledge state, belief, or intention; (b) another agent acts without correctly accounting for that state; (c) the trace contains enough evidence to identify a better-coordinated action; and (d) a reasonably clear evaluation question and answer can be constructed. Meeting the MAST FC2 label alone is not sufficient. Traces that fail any criterion are coded Ambiguous, Not Convertible, or Structural Mismatch and never generate items or rewards. Structural Mismatch applies when inclusion (a) failsâfor example, AG2 traces pairing an assistant with a mathproxyagent that only executes code. Ambiguous applies when a peer exists but multiple incompatible readings are equally supported, or a non-ToM co-cause (e.g., a framework gui_design flag) blocks isolation of the partner-state question. To assess the reliability of the four-category scheme, the first author and supervisor independently categorized all 18 sampled traces using the rubric defined above. Raw agreement was 94.4% (17/18 items), yielding Cohenâs Îș=0.92Îș=0.92, indicating almost perfect agreement [20]. The single disagreement occurred on trace #13 (MetaGPT), where the first author assigned AMBIGUOUS and the supervisor assigned NOT CONVERTIBLE; the supervisorâs coding was adopted as the final label, yielding CLEAN: 4, AMBIGUOUS: 4, NOT CONVERTIBLE: 2, STRUCTURAL MISMATCH: 8. Of the 423 train FC2+ traces in the locked Step 1 inventory (Table I), the 258 AG2 GSM/Olympiad cases (127 GSM + 131 Olympiad) are scoped out of the convertible partner-state pool and retained only for documentation as Structural Mismatch: their FC2 labels reflect solverâtool-proxy protocol issues rather than an isolable peer belief or knowledge target (confirmed on all 8/8 AG2 traces in the reliability pilot). The Step 1 non-AG2 remainder is 165 traces (ChatDev 53, MetaGPT 100, AppWorld 6, HyperAgent 6); that MetaGPT cell of 100 is the pilot-era inventory count, not the conversion count. The locked conversion log records 242 coded non-AG2 traces (Table I), of which MetaGPT accounts for 217 traces across four model families. ChatDev coding was stopped after 13 GPT-4o traces because every coded ChatDev case was Ambiguous under the locked rubric (often gui_design co-cause), so the remaining 40 ChatDev inventory traces were not converted into items. AppWorld and HyperAgent were fully coded (6/6 each). The coded pool yields 39 confirmed Clean items. Category counts are taken from the locked conversion log (conversion_log.json). TABLE I: Locked Step 1 train FC2+ inventory used for the conversion pilot and AG2 scoping (n=423n=423). This table does not supersede Table I for MetaGPT conversion counts. Unit Train FC2+ Role in this paper AG2/GSM 127 Excluded before partner-state coding (structural mismatch) AG2/Olympiad 131 Excluded before partner-state coding (structural mismatch) ChatDev 53 Pilot inventory; coding later stopped after 13 GPT-4o traces MetaGPT 100 Pilot inventory only; conversion uses Table I (217) AppWorld 6 Pilot inventory; all 6 later coded HyperAgent 6 Pilot inventory; all 6 later coded AG2 subtotal 258 Documentation only Non-AG2 subtotal 165 Pilot-era conversion pool Total 423 Pilot / AG2-scoping lock TABLE I: Conversion-pass accounting by framework and model (authoritative for coded counts). AMB = Ambiguous; NC = Not Convertible; SM = Structural Mismatch. Coded is the row sum. MetaGPT 217=53+47+31+86217=53+47+31+86 supersedes the Step 1 MetaGPT cell of 100. Framework Model Coded CLEAN AMB NC SM MetaGPT GPT-4o 53 28 2 23 0 MetaGPT Claude 47 4 0 43 0 MetaGPT Qwen 31 4 2 25 0 MetaGPT CodeLlama 86 0 2 48 36 AppWorld GPT-4o 6 2 0 4 0 HyperAgent Claude 6 1 2 3 0 ChatDev GPT-4o 13 0 13 0 0 Total 242 39 21 146 36 I-C Item construction and reward Each Clean trace becomes one functional partner-state item. The model receives a truncated coordination context (task text, roles, prior messages up to the miscoordinated action, visible tool outputs) but not the MAST failure label or later revealing turns. The model must state the peerâs relevant informational state and/or the better-coordinated next action; scoring is against fixed gold under a documented rubric sheet. For ToMAS-RL training we use a single transparent reward: binary ROUGE-L F1 match on Clean training items (+1+1 if ROUGE-L F1 exceeds 0.2; 00 otherwise). Scores are computed with rouge-scoreâs RougeScorer(["rougeL"]) default tokenizer on the extracted [ANSWER] span versus the fixed gold string; the reported value is F1 (not precision or recall alone). The training threshold 0.20.2 is intentionally loose and was selected because short partner-state answers rarely reach high lexical overlap, and a stricter cutoff collapsed the binary reward toward all zeros on the 39-item pool in preliminary checks, removing the GRPO learning signal (0.2 selected for a usable training signal; held-out uses 0.40.4). Held-out evaluation uses the stricter >0.4>0.4 cutoff to reduce spurious lexical matches on longer Magentic/GAIA answers where partial n-gram overlap is common even when the partner-state claim is wrong. Gold is assigned manually at construction; matching at train time is automatic. Only Clean items enter the reward pool. Circularity is controlled by source separation: rewards use train-split Clean items, while evaluation uses separate held-out Clean items. Training prompts never include MAST failure tags or post-failure turns. The present diagnostic scores generated answers against fixed held-out gold strings and is not an independent estimate of the MAST FC2 failure rate. I-D Illustrative conversion examples Table I shows two Clean items and one Ambiguous case from the approved conversion batches, clarifying when a partner-state gap is isolable under criteria (a)â(d). TABLE I: Illustrative Clean and Ambiguous conversion examples. Label Example Clean A AppWorld #15. Trace: AppWorld/Test-C/GPT-4o/18. Category: Clean. Context: Supervisor agent issues show_account_passwords; Spotify agent has previously requested an access token. Question: What credential does the Spotify agent require, and what should the Supervisor provide instead? Gold: The Spotify agent requires an OAuth access token, not an account password; the Supervisor should call the appropriate token-retrieval action. Rationale: Peer (Spotify) knowledge state is isolable; Supervisor action ignores the prior token request (criteria (a)â(d)). Supporting quote: âaccess_tokenâ in Spotify agent request. Clean B MetaGPT #12. Trace: MetaGPT/ProgramDev/Claude/91. Category: Clean. Context: Coder emits cron string with 0 in hour field using H:M format; Tester checks scheduling behavior. Question: What cron format does the Coder treat as valid for daily execution? Gold: The Coder treats 0 H:M as valid but the hour field requires an integer 0â23, not H:M; Tester should check field format before testing schedule behavior. Rationale: Coderâs format belief is recoverable from the trace; Tester should condition on that state before schedule checks (criteria (a)â(d)). Supporting quote: cron string in Coder output. Ambiguous ChatDev #11. Trace: ChatDev/ProgramDev/GPT-4o/82. Category: Ambiguous. Context: Customer requests CLI interface; Programmer implements tkinter GUI; gui_design parameter also set. Question / gold: Not issuedâco-cause blocks a single partner-state item. Rationale: FC2 peer-state gap exists (Programmer ignores CLI requirement) but gui_design system parameter independently influences output, preventing clean isolation of the inter-agent belief failure. I-E Training conditions and evaluation protocol We evaluate three conditions: a policy trained with the ToMAS reward (ToMAS-RL), a policy trained with adapted LLM-Coordination items (Coord-RL), and the untrained base checkpoint (Base). Both trained conditions use Qwen/Qwen2.5-1.5B-Instruct (Hugging Face Hub default revision at download time; revision not pinned; adapter configs record revision=null), the same optimization budget, and two random seeds (42 and 43). Each training step generates four completions. Training generation_kwargs set only max_new_tokens; temperature and top-p were left at Hugging Face GenerationConfig defaults (temperature=1.0temperature=1.0, topâ_âp=1.0top\_p=1.0). Held-out evaluation uses greedy decoding (do_sample=False), so temperature and top-p are unused at test time. We use Group Relative Policy Optimization (GRPO) with the binary reward defined in Section I-C. We selected the 1.5B Instruct checkpoint because it supports LoRA training and GRPO group rollouts on a Colab T4 without gated access. We use GRPO rather than dynamic sampling because the confirmed Clean pool contains only 39 items; removing all-correct or all-incorrect groups could further reduce an already limited training signal. Table IV summarizes the shared GRPO training configuration. Each step samples one item from the 39-item training pool, generates 4 completions (batch == 1 prompt Ă 4 completions), and applies binary ROUGE-L reward. Over 200 steps, each item is presented approximately 200/39â5.1200/39â 5.1 times on average (one âpassâ == one presentation of that item). TABLE IV: Shared GRPO training configuration. Parameter Value Base model Qwen/Qwen2.5-1.5B-Instruct Hub revision Default at download (not pinned; revision=null) Libraries trl 1.12.0; transformers 5.15.1; peft 0.20.0; torch 2.11.0+cu128 LoRA rank 8, alpha 16, dropout 0; targets q_proj, v_proj RL / optimizer GRPO (trl.GRPOTrainer); AdamW (trl default) LR schedule TrainingArguments defaults (linear; warmup_ratio=0); LR 1Ă10â61Ă10^-6 Steps / batch 200 steps; per step: 1 prompt Ă 4 completions (GRPO group size 4); grad. accum. 1 Lengths / stop Prompt 512; completion 128; stop [/ANSWER] Train sampling Temperature 1.01.0, top-p 1.01.0 (HF defaults; no override) Held-out decoding Greedy (do_sample=False) Reward ROUGE-L F1; train >0.2>0.2 (0.2 selected); held-out >0.4>0.4 Sampling schedule 1 item/step Ă 200 steps; â 5.1 presentations/item (200/39200/39) Seeds / hardware 42, 43; Colab T4 ⌠3.5GB; runtime not archived Code / logs train_condition.py; checkpoint-200; logs/*_rewards.jsonl Each trained condition runs for 200 optimization steps. The ToMAS-RL condition trains on the 39 converted items. The Coord-RL condition uses an equal number of adapted LLM-Coordination items under the same binary reward format and training budget. Adaptation procedure: from the public LLM-Coordination CoordQA CSVs (OvercookedFixedHeadings.csv, HanabiFixedHeadings.csv, CollabGamesFixedHeadings.csv), we retained Theory-of-Mind questions only (excluding Environment Comprehension and Joint Planning), yielding 66 eligible rows; we then sampled 39 items with seed 42 (Overcooked 15, Hanabi 14, Collab Capture 6, Collab Escape 4). Each row was mapped to the ToMAS JSONL schema (id, source, prompt, question, gold, supporting_quote), with scenario/state text as prompt, the ToM question as question, and the published ToM answer as gold, then scored with the same binary ROUGE-L reward as ToMAS-RL. The held-out diagnostic contains 28 Magentic GAIA items. One observation is one generated response to one held-out item. TABLE V: Conditions retained in the feasibility experiment. Condition Signal ToMAS-RL Binary partner-state reward on 39 converted Clean training items Coord-RL 39 LLM-Coordination ToM items (seed 42 subsample of 66); mapped to ToMAS JSONL; matched reward format and budget Base Untrained Qwen2.5-1.5B-Instruct checkpoint We considered EnactToM as a functional Theory-of-Mind baseline. However, its hard items require PDDL-predicate JSON responses, whereas the retained conditions use natural-language answers and ROUGE-L scoring. Because adapting these formats would introduce an additional methodological difference, EnactToM was excluded before the comparison. Consequently, the present experiment does not include a direct functional Theory-of-Mind baseline. The diagnostic reports binary ROUGE-L outcomes rather than an independent estimate of the MAST FC2 failure rate. Given the small held-out pool and the domain mismatch described below, we treat the results descriptively and do not perform inferential comparisons. IV Results We report findings in relation to RQ1 (convertibility) and RQ2 (small-scale GRPO training diagnostic / cross-domain evaluation). IV-A Dataset inventory The Step 1 lock reports 423 train FC2+ (Table I) and 209 hold-out FC2+ traces (Table VI: Magentic 103, OpenManus 6, AG2/MMLU 100) under the AG2-by-benchmark split. That inventory supports the pilot sample and AG2 scoping; conversion outcomes follow Table I. Neither inventory is an RL outcome. TABLE VI: Locked Step 1 hold-out FC2+ inventory under the same AG2-by-benchmark split (n=209n=209). The held-out diagnostic uses only the Magentic slice. Unit Hold-out FC2+ Role in this paper Magentic 103 Source pool for the 28-item diagnostic OpenManus 6 Not used in the present diagnostic AG2/MMLU 100 Not used in the present diagnostic Total 209 Step 1 hold-out lock IV-B Conversion pilot From the train FC2-positive traces we drew a stratified random sample of n=18n=18 (seed=42seed=42). Each trace was coded Clean, Ambiguous, Not Convertible, or Structural Mismatch under the criteria in Section I. TABLE VII: Conversion-pilot outcomes on n=18n=18 train FC2-positive traces. Code AG2 Non-AG2 Total (n=8n=8) (n=10n=10) (n=18n=18) CLEAN 0 4 4 AMBIGUOUS 0 4 4 NOT CONVERTIBLE 0 2 2 STRUCTURAL MISMATCH 8 0 8 Table VII reports the measured codes. All 8/8 sampled AG2 GSM/Olympiad traces were coded Structural Mismatchâlabeled FC2 in those traces reflects solverâtool-proxy protocol issues rather than peer-belief reasoning in this small sample. We treat that pilot AG2 pattern as confirmed for scoping: the AG2 GSM/Olympiad train FC2+ traces (n=258n=258 in the locked inventory, including the 8 AG2 pilot samples) are retained for documentation as Structural Mismatch, not as convertible partner-state items, and are excluded from the partner-state conversion pool. Among 10 non-AG2 samples, 4 were CLEAN (40%), 4 Ambiguous (including a ChatDev gui_design co-cause), and 2 Not Convertible. A conversion pass recorded 242 coded non-AG2 traces in the locked log (Table I; MetaGPT 217 across four model families) and yielded 39 confirmed CLEAN items with further yield limited under the locked rubric. CLEAN rates differed sharply by model within MetaGPT: GPT-4o â 53% (28/53), Qwen â 13% (4/31), Claude â 9% (4/47), and CodeLlama 0% (0/86); ChatDev was 0% CLEAN across 13 coded traces (all Ambiguous). These descriptive rates (Table I) show that recoverable partner-state items were substantially more common in the available GPT-4o MetaGPT traces than in the other coded sources. However, they should not be interpreted as causal evidence that one model has stronger partner-state reasoning than another because the trace contents, tasks, and failure distributions were not experimentally controlled. Qualitatively, many non-convertible CodeLlama and ChatDev traces were dominated by prompt echo, repetitive questionâanswer loops, or framework-level configuration conflicts rather than an isolable peer-state reasoning failure. Two confirmed Clean drafts illustrate the item format (AppWorld token confusion; MetaGPT crontab field misuse). A ChatDev CLI-vs-GUI case is retained as an Ambiguous illustration because framework-injected GUI configuration co-causes the drift. The pilot establishes that FC2 label â ToMAS-convertible under our criteria. IV-C Training and held-out evaluation TABLE VIII: Training reward at selected optimizer steps. Each cell is the trl-logged rewards/binary_reward/mean written by the reward callback at that step (logs every 10 steps; trl 1.12.0). Values equal exact small-denominator ratios recovered from experiments/logs/*_rewards.jsonl (e.g., 0.417=5/120.417=5/12, 0.833=5/60.833=5/6, 0.125=1/80.125=1/8), so they are not means over a fixed 40-sample (4Ă10) window. Seeds 42 and 43 use different random initializations. Condition Seed Step 50 Step 100 Step 150 Step 200 ToMAS-RL 42 0.417 0.750 0.833 0.125 ToMAS-RL 43 0.417 0.500 0.167 0.875 Coord-RL 42 0.667 0.250 0.583 0.250 Coord-RL 43 0.750 0.625 0.167 0.500 All trained conditions produced nonzero logged rewards throughout training (Table VIII), confirming that the reward function fired and the GRPO loop ran to completion. RQ2 adapter checks Identical greedy held-out answers across ToMAS-RL, Coord-RL, and Base (Tables IXâX) prompted three checks. (a) Evaluation attaches adapters via PeftModel.from_pretrained on each checkpoint-200 directory (experiments/evaluate.py); logs confirm âAttached adapterâ for all four trained runs. (b) On the 39 training items under the same greedy protocol, Coord-RL matched Base on 39/3939/39 items (both seeds). ToMAS-RL matched on 37/3937/39 (seed 42) and 38/3938/39 (seed 43); the few mismatches are minor surface paraphrases on the same items (item_34, and item_35 for seed 42 only), not systematic answer changes. (c) Adapter norms: LoRA A dominates (L2â12.22L2\!â\!12.22, typical of nonzero init), while LoRA B remains near zero (L2â0.005L2\!â\!0.005â0.0060.006, maxâĄ|w|â4Ă10â5 |w|\!â\!4Ă10^-5); with α/r=2α/r=2, effective maxâĄ|ÎâW|â7Ă10â6 | W|\!â\!7Ă10^-6. Plainly: adapters were loaded and training ran, but under LR 1Ă10â61Ă10^-6 the policy barely movedâheld-out outputs are identical to Base, and training-set outputs are identical or near-identicalâso RQ2 does not establish a coordination-training effect. Held-out items were drawn from the Magentic slice of the locked hold-out FC2+ pool (Table VI: Magentic 103, OpenManus 6, AG2/MMLU 100; total 209). OpenManus and AG2/MMLU were not used for this diagnostic. The first author applied the same locked convertibility rubric used in training conversion and retained traces that yielded isolable partner-state items with recoverable gold, producing the released set of 28 Clean Magentic/GAIA items (IVH_01âIVH_28). Gold answers for all 28 held-out items were created by the first author using the same locked convertibility rubric applied to the training conversion; no second annotator re-checked held-out gold for this pilot. Per-item conversion rationales for held-out items follow the locked rubric and are included in the Zenodo release alongside truncated context, question, and gold. No held-out items overlap with training items by source trace ID (verified by ID intersection check). We did not run a blinded human or semantic-equivalence assessment in this pilot; the reported held-out metric is therefore lexical ROUGE-L F1 only, which can credit phrasing overlap without verifying partner-state reasoning. TABLE IX: Held-out items with binary ROUGE-L >0.4>0.4 by condition and seed (n=28n=28 items each). Positive item IDs and generated answers are identical across ToMAS-RL, Coord-RL, and Base under greedy decoding (do_sample=False). Answers are the extracted [ANSWER] spans from the evaluation runs summarized here; the same two items fire under every condition. Condition Seed n>0.4n>0.4 Positive item IDs ToMAS-RL 42 2 IVH_04, IVH_12 ToMAS-RL 43 2 IVH_04, IVH_12 Coord-RL 42 2 IVH_04, IVH_12 Coord-RL 43 2 IVH_04, IVH_12 Base â 2 IVH_04, IVH_12 TABLE X: Generated answers for the two threshold-passing held-out items (identical under every condition/seed in Table IX). Item ROUGE-L Generated answer IVH_04 0.404 WebSurfer knows 2072 Akaikai Loop sold for $1,190,000 (14 Dec 2022) while 2017 Komo Mai Drive sold for $1,080,000 (17 Aug 2023); the acting agent should report which 2022 sale was higher. IVH_12 0.476 Peers establish honey density 1.38â1.45 kg/L (20âC) and mayonnaise 0.91 kg/L; the acting agent should compute cups of honey to remove so honey weighs less than mayonnaise. Because every condition decodes identically to Base (Tables IX and X), the held-out numbers describe base-model behaviour on the diagnostic and carry no information about training. We report them for completeness and because they expose a second limitation independent of the optimization failure: a provenance gap between the two item pools. Training items are drawn predominantly from MetaGPT code-coordination failures (36/39 items, with 2 AppWorld and 1 HyperAgent), whereas held-out items are general-purpose GAIA tasks. This asymmetry follows from the AG2-by-benchmark split fixed at Step 1 and was therefore a property of the design, not a result of the experiment. The two threshold-passing items also illustrate the scoring limitation: both clear 0.40.4 on phrasing overlap with gold rather than on a verified partner-state claim. Since every condition and seed produced the same two item IDs and the same extracted answers under greedy decoding, we report the compact hit table rather than a redundant 28Ă28Ăcondition matrix; non-hitting items scored â€0.4†0.4 throughout. A conclusive study requires a matched-domain held-out pool, an effective learning rate, and semantic or human scoring. V Discussion The conversion pilot and full conversion pass show that a nonempty convertible subset of FC2 can be represented as functional partner-state problems, and that the same items are well-formed as binary rewards. They also tied the claim: convertibility must be filtered, not assumed from the FC2 label alone. The AG2 GSM/Olympiad pattern is treated as Structural Mismatch for ToMAS construction; ChatDev and CodeLlama MetaGPT are excluded after yielding 0% CLEAN under the locked rubric. Model-level convertibility (highest for GPT-4o MetaGPT; near-zero for CodeLlama and ChatDev) is reported as a finding about which execution traces support partner-state items, rather than merely a limit on pool size. The present 28-item diagnostic does not establish a training effect (RQ2): adapters load at evaluation, LoRA ÎâW W remains near zero under LR 1Ă10â61Ă10^-6, held-out greedy outputs match Base, and training-set greedy outputs match on 3737â39/3939/39 items (only minor paraphrases elsewhere). Convertibility findings address RQ1. Two further limitations would have blocked a conclusive result even had the policy updated: the provenance gap between MetaGPT-derived training items and GAIA-derived held-out items, and scoring by lexical overlap alone. Cross-framework generalization therefore remains open. We note also that a positive result under a corrected setup would show that ToMAS items are trainable, not that failure grounding per se is what produced the gain; isolating that would require a matched control trained on authored items of equivalent difficulty. Convertibility analysis under RQ1 is limited by cell size in the 39 confirmed CLEAN items from the conversion pass. VI Conclusion Our study highlights how systematic diagnosis of agent errors can serve as a direct source for improving social reasoning in large language models. ToMAS demonstrates that a subset of MAST FC2 failures can support a failure-grounded Theory of Mind benchmark under a structured conversion rubric with near-perfect inter-annotator agreement (Îș=0.92Îș=0.92). Convertibility varies markedly with the structure of the source modelâs output. The training diagnostic did not produce a measurable policy update and therefore yields no evidence about coordination training; together with the item-provenance gap and lexical scoring, it defines what a conclusive follow-up study must control. Data Availability ToMAS evaluation items, annotation materials, scoring scripts, train_condition.py, GRPO reward logs, and Coord-RL training items are available at https://doi.org/10.5281/zenodo.22685705 [21] under Creative Commons Attribution 4.0 International (C-BY-4.0). Items derived from MAST-Data must also attribute mcemri/MAST-Data [5]. References [1] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber (2024) MetaGPT: meta programming for a multi-agent collaborative framework. In ICLR, Cited by: §I, §I-A, §I-B. [2] C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun (2024) ChatDev: communicative agents for software development. In ACL, Cited by: §I, §I-A, §I-B. [3] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang (2024) AutoGen: enabling next-gen LLM applications via multi-agent conversations. In COLM, Cited by: §I, §I-A. [4] A. Fourney, G. Bansal, H. Mozannar, C. Tan, E. Salinas, E. Zhu, F. Niedtner, G. Proebsting, G. Bassman, J. Gerrits, J. Alber, P. Chang, R. Loynd, R. West, V. Dibia, A. Awadallah, E. Kamar, R. Hosn, and S. Amershi (2024) Magentic-One: a generalist multi-agent system for solving complex tasks. Note: arXiv preprintarXiv:2411.04468 Cited by: §I, §I-A. [5] M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica (2025) Why do multi-agent LLM systems fail?. In NeurIPS, Cited by: §I, §I, §I-A, §I-B, §I-C, §I-D, §I-A, Data Availability. [6] J. Jia, Z. Deng, Z. Chen, Y. Wang, and Z. Zheng (2026) MAS-FIRE: fault injection and reliability evaluation for LLM-based multi-agent systems. Note: arXiv preprintarXiv:2602.19843 Cited by: §I, §I-B. [7] Y. Zhang, F. Liu, Y. Shan, X. Huang, X. Yang, Y. Zhu, X. Cheng, C. Liu, K. Zeng, T. J. Zhang, and W. Jiang (2026) Silo-Bench: a scalable environment for evaluating distributed coordination in multi-agent LLM systems. Note: arXiv preprintarXiv:2603.01045 External Links: 2603.01045 Cited by: §I, §I-B. [8] S. Agashe, Y. Fan, A. Reyna, and X. E. Wang (2025) LLM-Coordination: evaluating and analyzing multi-agent coordination abilities in large language models. In NAACL Findings, Cited by: §I, §I-C. [9] G. Juneja, D. Lu, S. Agashe, P. Diwane, E. Gunn, J. Srinivasa, G. Liu, W. Y. Wang, Y. Du, and X. E. Wang (2026) EnactToM: an evolving benchmark for functional theory of mind in embodied agents. Note: arXiv preprintarXiv:2605.09826 Cited by: §I, §I-C, §I-D. [10] M. Riemer, Z. Ashktorab, D. Bouneffouf, P. Das, M. Liu, J. D. Weisz, and M. Campbell (2025) Theory of mind benchmarks are broken for large language models. Note: OpenReviewhttps://openreview.net/forum?id=BCP8U2BcU Cited by: §I, §I-C. [11] Y. Cai, Z. Gu, J. Zhang, and P. Chen (2026) MARO: learning stronger reasoning from social interaction. Note: arXiv preprintarXiv:2601.12323 Cited by: §I, §I-D. [12] H. Yao, L. Da, X. Liu, C. Fleming, T. Chen, and H. Wei (2026) LangMARL: natural language multi-agent reinforcement learning. Note: arXiv preprintarXiv:2604.00722 Cited by: §I, §I-D. [13] X. Li, T. Zhang, C. Liu, S. Xu, and B. Xu (2025) Theory of mind inspired large reasoning language model improved multi-agent reinforcement learning algorithm for robust and adaptive partner modelling. Machine Intelligence Research. Cited by: §I, §I-D. [14] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang (2025) Agentless: demystifying LLM-based software engineering agents. In FSE, Cited by: §I-A. [15] S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan (2024) AI agents that matter. Note: arXiv preprintarXiv:2407.01502 Cited by: §I-A. [16] K. Venkatesh and J. Cui (2026) Do agent societies develop intellectual elites? the hidden power laws of collective cognition in LLM multi-agent systems. Note: arXiv preprintarXiv:2604.02674 Cited by: §I-A. [17] E. La Malfa, G. La Malfa, S. Marro, J. M. Zhang, E. Black, M. Luck, P. Torr, and M. J. Wooldridge (2025) LLMs miss the multi-agent mark. Note: arXiv preprintarXiv:2505.21298 Cited by: §I-C. [18] A. Yadav, S. Black, and O. Sourbut (2026) More capable, less cooperative? when LLMs fail at zero-cost collaboration. Note: arXiv preprintarXiv:2604.07821 Cited by: §I-C. [19] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, J. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang (2025) DAPO: an open-source LLM reinforcement learning system at scale. In NeurIPS, Cited by: §I-D. [20] J. R. Landis and G. G. Koch (1977) The measurement of observer agreement for categorical data. Biometrics 33 (1), p. 159â174. Cited by: §I-B. [21] M. A. Ishfaq and G. Melo (2026) ToMAS: theory-of-mind evaluation items for multi-agent coordination failures (v1.2). Note: Zenododoi:10.5281/zenodo.22685705 Cited by: Data Availability.