Paper deep dive
MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, Zibin Zheng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/20/2026, 3:33:52 PM
Summary
The paper introduces MAS-FIRE, a systematic framework for fault injection and reliability evaluation of LLM-based Multi-Agent Systems (MAS). It defines a taxonomy of 15 fault types (intra-agent and inter-agent) and three non-invasive injection mechanisms (prompt modification, response rewriting, message routing manipulation). The study evaluates three MAS architectures, revealing that architectural topology (e.g., iterative closed-loop designs) significantly impacts robustness, neutralizing over 40% of faults that cause failure in linear workflows, while stronger foundation models do not uniformly improve resilience.
Entities (14)
Relation Signals (8)
Jin Jia → affiliatedwith → Sun Yat-sen University
confidence 99% · Jin Jia ∗ Sun Yat-sen University
Zhuangbin Chen → affiliatedwith → Sun Yat-sen University
confidence 99% · Zhuangbin Chen † Sun Yat-sen University
Zhiling Deng → affiliatedwith → Sun Yat-sen University
confidence 99% · Zhiling Deng ∗ Sun Yat-sen University
MAS-FIRE → evaluates → Multi-Agent Systems
confidence 95% · MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
MAS-FIRE → uses → Prompt Modification
confidence 95% · We define a taxonomy of 15 fault types ... and inject them via three non-invasive mechanisms: prompt modification...
MAS-FIRE → uses → Message Routing Manipulation
confidence 95% · We define a taxonomy of 15 fault types ... and inject them via three non-invasive mechanisms: ... message routing manipulation.
MAS-FIRE → uses → Response Rewriting
confidence 95% · We define a taxonomy of 15 fault types ... and inject them via three non-invasive mechanisms: ... response rewriting...
Iterative Closed-Loop Design → mitigates → Linear Workflow
confidence 90% · iterative, closed-loop designs neutralizing over 40% of faults that cause catastrophic collapse in linear workflows.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As LLM-based Multi-Agent Systems (MAS) are increasingly deployed for complex tasks, ensuring their reliability has become a pressing challenge. Since MAS coordinate through unstructured natural language rather than rigid protocols, they are prone to semantic failures (e.g., hallucinations, misinterpreted instructions, and reasoning drift) that propagate silently without raising runtime exceptions. Prevailing evaluation approaches, which measure only end-to-end task success, offer limited insight into how these failures arise or how effectively agents recover from them. To bridge this gap, we propose MAS-FIRE, a systematic framework for fault injection and reliability evaluation of MAS. We define a taxonomy of 15 fault types covering intra-agent cognitive errors and inter-agent coordination failures, and inject them via three non-invasive mechanisms: prompt modification, response rewriting, and message routing manipulation. Applying MAS-FIRE to three representative MAS architectures, we uncover a rich set of fault-tolerant behaviors that we organize into four tiers: mechanism, rule, prompt, and reasoning. This tiered view enables fine-grained diagnosis of where and why systems succeed or fail. Our findings reveal that stronger foundation models do not uniformly improve robustness. We further show that architectural topology plays an equally decisive role, with iterative, closed-loop designs neutralizing over 40% of faults that cause catastrophic collapse in linear workflows. MAS-FIRE provides the process-level observability and actionable guidance needed to systematically improve multi-agent systems.
Tags
Links
- Source: https://arxiv.org/abs/2602.19843v1
- Canonical: https://arxiv.org/abs/2602.19843v1
Trouble viewing inline? Open PDF directly →
Full Text
98,620 characters extracted from source content.
Expand or collapse full text
MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems Jin Jia ∗ Sun Yat-sen University Zhuhai, Guangdong, China jiaj9@mail2.sysu.edu.cn Zhiling Deng ∗ Sun Yat-sen University Zhuhai, Guangdong, China dengzhling3@mail2.sysu.edu.cn Zhuangbin Chen † Sun Yat-sen University Zhuhai, Guangdong, China chenzhb36@mail.sysu.edu.cn Yingqi Wang Sun Yat-sen University Zhuhai, Guangdong, China wangyq376@mail2.sysu.edu.cn Zibin Zheng Sun Yat-sen University Zhuhai, Guangdong, China zhzibin@mail.sysu.edu.cn Abstract As LLM-based Multi-Agent Systems (MAS) are increasingly de- ployed for complex tasks, ensuring their reliability has become a pressing challenge. Since MAS coordinate through unstructured natural language rather than rigid protocols, they are prone to se- mantic failures (e.g., hallucinations, misinterpreted instructions, and reasoning drift) that propagate silently without raising runtime exceptions. Prevailing evaluation approaches, which measure only end-to-end task success, offer limited insight into how these fail- ures arise or how effectively agents recover from them. To bridge this gap, we propose MAS-FIRE, a systematic framework for fault injection and reliability evaluation of MAS. We define a taxonomy of 15 fault types covering intra-agent cognitive errors and inter- agent coordination failures, and inject them via three non-invasive mechanisms: prompt modification, response rewriting, and message routing manipulation. Applying MAS-FIRE to three representative MAS architectures, we uncover a rich set of fault-tolerant behav- iors that we organize into four tiers: mechanism, rule, prompt, and reasoning. This tiered view enables fine-grained diagnosis of where and why systems succeed or fail. Our findings reveal that stronger foundation models do not uniformly improve robustness. We fur- ther show that architectural topology plays an equally decisive role, with iterative, closed-loop designs neutralizing over 40% of faults that cause catastrophic collapse in linear workflows. MAS-FIRE provides the process-level observability and actionable guidance needed to systematically improve multi-agent systems. CCS Concepts • Software and its engineering→Software testing and debug- ging; Software verification and validation. ∗ Both authors contributed equally to this research. † Zhuangbin Chen is the corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2026/02 https://doi.org/X.X Keywords Fault Injection, Robustness Evaluation, Multi-agent Systems ACM Reference Format: Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi- Agent Systems. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’X). ACM, New York, NY, USA, 16 pages. https://doi.org/X.X 1 Introduction The rapid advancement of LLMs has catalyzed a paradigm shift in intelligent software, moving from monolithic chatbots to orches- trated Multi-Agent Systems (MAS). By assigning specialized roles (e.g., planning, coding, and reviewing) to distinct agent instances, MAS can tackle complex, long-horizon tasks through collabora- tion [1,8,10,36,47]. These systems have demonstrated impressive capabilities in domains ranging from automated software engi- neering to scientific discovery. However, as MAS transition from experimental prototypes to mission-critical components in pro- duction environments, their reliability and fault tolerance become paramount concerns. Traditional fault-tolerance mechanisms of distributed systems typically address well-defined failures such as component crashes or network timeouts [16,28]. However, these paradigms are in- sufficient for MAS due to a fundamental architectural difference. Unlike traditional distributed systems that rely on rigid protocols (e.g., gRPC, REST), MAS utilize natural language as their primary interface for coordination. While this flexibility enables dynamic collaboration, it introduces a unique class of reliability challenges where the system state is defined not by deterministic variables, but by the semantic context of unstructured dialogue. Consequently, failures rarely manifest as explicit crashes; instead, they appear as “soft” semantic deviations (e.g. hallucinations [13, 51, 62], ambigu- ous interpretations [64], or reasoning drift [11,44,49]) that silently propagate through the system without runtime exceptions. Current evaluation methodologies for MAS are ill-equipped to diagnose these semantic vulnerabilities. They primarily rely on outcome-oriented metrics, such as binary task success rates or sub- goal completion percentages [24,26]. While effective for bench- marking capabilities, these metrics treat the system as a black box, obscuring the process of failure and recovery. They fail to answer critical questions: Did the system succeed because it is robust, or arXiv:2602.19843v1 [cs.SE] 23 Feb 2026 Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng Intra-agent Fault Inter-agent Fault Inexecution Plan (IP): Agent generates plans that cannot be executed due to non-existent tools or fabricated subtasks Critical Information Loss (CIL): Essential information is lost, leading to incomplete or incorrect reasoning and potentially affecting subsequent decisions Planning Fault Memory Loss (ML): Agent forgets previously observed information, which affects subsequent reasoning and actions Context Length Violation (CLV): Input exceeds the context window, causing prompt truncation and unreasonable outputs Memory Fault Hallucination (H): Agent produces unsupported assumptions, incorrect summaries, or fabricated facts Reasoning Fault Parameter Filling Error (PFE): Agent provides incorrect parameter values in a valid tool call, causing unexpected behavior Action Fault Role Ambiguity (RA): Ambiguous, overlapping, or underspecified roles in system prompts lead to unexpected execution flows at runtime Blind Trust (BT): Agent accepts information from others without verification, allowing errors to propagate and affect subsequent reasoning Configuration Fault Message Cycle (MC): Agents repeatedly exchange messages without making any real progress Communication Fault Instruciton Logic Conflict (ILC): Instructions contain conflicting requirements, forcing agents to rationalize, ignore, or only partially fulfill them Instruction Ambiguity (IA): Vague instructions lead agents to guess or ignore missing details, possibly causing task failure Instruction Fault Message Storm (MS): Agents send a large volume of messages in a short time, overwhelming the system Message Broadcast Amplification (MBA): Agents broadcast messages to unrelated agents, triggering unnecessary or conflicting responses Tool Format Error (TFE): Agent violates the tool call format, using an incorrect name or invalid syntax Tool Selection Error (TSE): Agent chooses an inappropriate tool for the task Figure 1: Fault Injection Model for MAS due to a lucky retry? Did it fail because of a logic error, or because a rigid architecture prevented an agent from asking for clarifica- tion? Without fine-grained observability into how agents respond to anomalies, whether they self-correct, or stall, improving system robustness remains a trial-and-error process. To address this gap, we introduce MAS-FIRE, a systematic frame- work for Fault Injection and Robustness Evaluation of Multi-Agent Systems. MAS-FIRE moves beyond simple success metrics to pro- vide a granular analysis of agent resilience. We establish a grounded taxonomy of 15 distinct fault types, categorized into intra-agent faults (affecting internal cognitive processes) and inter-agent faults (disrupting coordination). To simulate realistic production failures, we design non-invasive injection mechanisms that introduce per- turbations through prompt modification, response rewriting, and message routing, preserving the system’s internal architecture. Using this framework, we evaluate three representative MAS architectures across the 15 fault types in our taxonomy. Our analysis identifies a comprehensive set of fault-tolerant behaviors, revealing the specific processes through which agents detect, mitigate, and recover from semantic and structural anomalies. We categorize these observed behaviors into four hierarchical tiers: mechanism, rule, prompt, and reasoning. These tiers provides a structured lens for diagnosing MAS resilience, allowing developers to decouple the contributions of system architecture from those of model reasoning. Furthermore, we find that while advanced models excel at semantic reasoning, they are paradoxically more vulnerable to prompt-level corruption due to strict instruction compliance. Structural design serves as the an equally important safeguard, with iterative, closed- loop topologies neutralizing over 40% of the faults that dismantle linear, waterfall-style workflows. The major contributions of this work are summarized as follows: •We propose MAS-FIRE, a fault injection and robustness evalu- ation framework for MAS, which includes three non-invasive injection mechanisms and a comprehensive suite of robustness metrics that quantify both system-level stability and process- level fault-tolerant effectiveness. •We establish a fault taxonomy categorizing 15 MAS-specific fault types across intra-agent faults and inter-agent faults, and a behavioral taxonomy characterizing fault-tolerant responses along four system dimensions, enabling a fine-grained diagno- sis of how systems fail or recover. • Through extensive evaluation on three representative MAS architectures, we systematically identify and quantify the fault- tolerant behaviors exhibited by MAS under different fault types. Our analysis reveals multi-dimensional nature of MAS robust- ness that provide actionable insights essential for designing and deploying robust MAS. 2 Background 2.1 LLM-Based Multi-Agent Systems as Intelligent Software The emergence MAS powered by LLMs marks a transition toward a new type of intelligent software. Unlike traditional monolithic MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY applications or standard microservices [6], LLM-based MAS operate by orchestrating autonomous agents that function as specialized computing units [7,10,18,19,32,50]. Each agent is typically en- dowed with distinct capabilities (e.g., planning, memory retention, and tool execution), allowing the collective system to decompose and solve complex, long-horizon problems that exceed the capacity of a single model instance. In this paradigm, the LLM functions as the cognitive core, responsible for interpreting instructions, reason- ing through sub-tasks, and generating executable actions based on environmental feedback. A defining characteristic of this intelligent software is its re- liance on natural language as the primary interface for coordination. Whereas traditional distributed systems communicate via rigid, pre- defined protocols (e.g., gRPC, REST), agents in an MAS collaborate through unstructured semantic dialogue. This reliance on natural language introduces a unique layer of complexity: the system state is not defined by deterministic variables but by the semantic context of the conversation history [29,63]. Thus, the architectural topol- ogy, whether organized as a sequential pipeline [32], a hierarchical hierarchy [10], or a cooperative network [18], plays a critical role in defining how information flows and how effectively the system can maintain logical consistency across diverse agent interactions. 2.2 Fault Injection and the Reliability of MAS Fault Injection (FI) is a well-established technique in reliability en- gineering used to assess a system’s resilience by deliberately intro- ducing perturbations [22,25]. Historically, FI methodologies have been applied across various abstraction layers, from hardware-level signal interference to software-level logic mutation and interface data corruption [3,27]. The primary objective is to accelerate the occurrence of rare failure modes, thereby allowing developers to validate error-handling mechanisms and ensure that local faults do not cascade into catastrophic system failures. However, applying existing FI methodologies to LLM-based in- telligent software presents significant challenges due to their prob- abilistic and semantic nature. Traditional software faults typically manifest as explicit crashes, exceptions, or timeouts that are eas- ily detectable by runtime monitors. In contrast, failures in MAS often appear as “silent” semantic deviations such as hallucinated facts [13,62], misinterpreted instructions [41,64], or reasoning drift [2,5], where the system continues to operate without trig- gering technical exceptions. Furthermore, because agent behavior is non-deterministic, a minor perturbation in a prompt or a slight distortion in a message can lead to vastly different outcomes de- pending on the interaction context. Thus, evaluating the robustness of MAS requires shifting the focus of injection from purely syntactic or structural code mutations to semantic perturbations that directly challenge the agents’ cognitive and collaborative capabilities. 3 An MAS Fault Injection and Robustness Evaluation Framework In this section, we present MAS-FIRE, a systematic framework for evaluating MAS robustness through controlled fault injection. We begin by establishing a grounded MAS fault taxonomy, which synthesize potential failure modes into distinct intra-agent and inter-agent categories based on architectural boundaries. Based on these faults, we detail our fault jnjection mechanisms, employing non-invasive techniques (prompt modification, interception and response rewriting, and message routing manipulation) to simulate realistic and representative anomalies during execution. Finally, we define a set of quantitative robustness metrics designed to measure both the overall system resilience and the specific efficacy of fault- tolerance mechanisms. 3.1 MAS Fault Taxonomy To establish a rigorous foundation for MAS robustness evalua- tion, we derive a comprehensive fault taxonomy based on system- atic literature review of empirical evaluations [12,37,52], bench- marks [4,24,26,30,35,39,39,42,61,64], and specialized failure studies [11,13,23,29,34,38,49,51,54,56,59,62,63]. We categorize faults according to where they arise in MAS execution: Intra-agent Faults emerge within an individual agent’s internal processing (plan- ning, memory, reasoning, action), while Inter-agent Faults affect coordination and information flow across multiple agents (configu- ration, instruction, communication). This distinction is critical, as internal reasoning faults require fundamentally different injection strategies than coordination failures. As illustrated in Fig. 1, the taxonomy contains seven fault categories, serving as the unified basis for MAS-FIRE. 3.1.1 Intra-agent Faults. Intra-agent faults encompass failures in- herent to an individual agent’s cognitive pipeline, directly degrad- ing its ability to plan, retain context, reason, or execute commands. Following standard agent capability decompositions [48,53], we classify these faults into four subclasses. Planning Faults stem from deficiencies in task decomposition and execution scheduling, manifesting when agents generate inexecutable plans (by invoking non-existent tools or hallucinating subtasks) or omit essential task constraints [20,44,49,54,58]. Memory Faults arise from incor- rect information retention or management, including aggressive context compression leading to critical information loss or context overflow exceeding effective processing windows [17,23,29,63]. Reasoning Faults reflect errors in the agent’s inference engine, commonly manifesting as hallucinations where agents generate incorrect summaries, unsupported assumptions, or fabricated facts that propagate to subsequent stages [13,51,62]. Action Faults occur during external tool interactions, involving inappropriate tool selection, invocation format violations, or invalid parameter supply [21, 24, 33, 35, 39]. 3.1.2Inter-agent Faults. Inter-agent faults capture failures emerg- ing from misaligned assumptions, faulty dependencies, or abnormal interactions between agents. These faults propagate across inter- agent communications, amplifying errors and causing global task failures even when individual agents function correctly in isola- tion. We divide these into three subclasses. Configuration Faults refer to failures caused by flawed agent role definitions and de- pendency assumptions established prior to execution, manifesting when agent roles specified through natural-language prompts are ambiguous, overlapping, or underspecified, or when agents blindly trust information from other agents without validation [34,45,61]. Instruction Faults denote failures introduced by defects in user- provided task instructions shared across agents, including logical Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng User def discount(sum): if sum > 100: off = sum * 0.1 if off > 5: off = 5 return sum - off Implement a discount function: apply a 10% discount to orders over $100, and the discount amount must not exceed $5. User Implement a discount function: apply a 10% discount to orders over $100. PM Agent There’s a conflict: 10% off orders over $100 is already at least $10, which is higher than the $5 limit. Which rule should we change? Got it. Share this with the dev team. Proactive requirement clarification INJECTOR Based on LLM PM Agent Dev Agent Fail to identify conflicting instructions (a) Prompt Modification Coordinate repair with other agents Buy a silk scarf. > order(item="silk scarf") > order(item="wool socks") Retry original request. > order(item="silk scarf") Wool socks are out of stock. The user might accept a wool scarf instead. > order(item="wool scarf") Error: wool socks not found. Respond to the error but ultimately fail Success: wool scarf ordered. I've successfully ordered a scarf. Original user intent is "silk scarf", not "wool socks". Re-verify input. User Tool Tool INJECTOR Based on LLM Agent Agent Agent Agent Agent Verifier (b) Intercept. and Resp. Rewriting [ID 45] Page 4 browsing complete. [ID 46] Browse the next page. [ID 46] Browse the next page. [ID 47] Now browsing Page 7. (Recieved 3 messages from Agent B) [ID 46] Browse the next page. [ID 46] Browse the next page. [ID 47] Now browsing Page 5. (Recieved 1 message from Agent B) Fail to filter and cause abnormal behavior Correctly filter messages Filter messages by ID Replay message: [ID 46] x3 INJECTOR Programmatic Agent A Agent A Agent A Agent B Agent B Message Pool > click_next_page() > click_next_page() > click_next_page() Agent A > click_next_page() (c) Message Routing Manipulation Figure 2: Examples of Fault Injection Mechanisms and Multi-Agent Recovery Behaviors. (a) Instruction Logic Conflict via Prompt Modification, which introduces incompatible constraints to evaluate requirement clarification; (b) Parameter Filling Error via Interception and Response Rewriting, which alters task parameters to assess inter-agent coordination and repair; (c) Message Storm via Message Routing Manipulation, which injects redundant communication to test infrastructure-level filtering. Green panels illustrate robust recovery (good behavior), while red panels highlight representative failure modes (bad behavior). conflicts (mutually incompatible requirements) and semantic am- biguity (insufficient clarity for consistent interpretation) [41,46, 52,64]. Communication Faults capture failures from abnormal message-passing behaviors, including message duplication without proper deduplication, message cycles where agents enter repetitive loops, and message broadcast amplification where messages are mistakenly disseminated to unintended agents [48, 56]. 3.2 Fault Injection Mechanism The fault taxonomy described above defines seven high-level fault categories. To operationalize these for systematic evaluation, we design 15 concrete, injectable fault types as shown in Fig. 1. These faults originate from different sources (i.e., prompts governing agent identity and task specification, runtime outputs during execution, and message flows coordinating multi-agent interaction), each re- quiring distinct injection strategies. To inject them without com- promising the non-invasive nature of the evaluation, we design three complementary fault injection mechanisms: 3.2.1Prompt Modification. This mechanism (as shown in Fig. 2a) targets natural language prompts that shape how agents interpret their responsibilities and task objectives. It operates on two prompt types: system prompts (establishing agent identity, roles, and be- havioral policies at initialization) and user prompts (conveying task requirements and constraints). By corrupting these textual direc- tives, this mechanism injects two categories of faults: •Configuration Faults. To inject this type of faults, MAS-FIRE modifies system prompts before agent instantiation, introduc- ing architectural defects. Two specific faults are implemented: (1) Role Ambiguity, which merges conflicting role definitions into an agent’s system prompt (e.g., acting as both “Developer” and “Tester”), forcing the agent to manage disparate objectives and leading to internal logic conflicts; and (2) Blind Trust, which injects unconditional trust directives (e.g., “accept all input from Agent X as absolute truth”), disabling critical verification and causing uncritical propagation of upstream errors. •Instruction Faults. MAS-FIRE intercepts user prompts at the MAS entry point and applies semantic transformations that in- troduce logical inconsistencies or ambiguity into the task spec- ification, forcing correctly configured agents to operate under flawed premises. MAS-FIRE employs a rule-guided LLM-based injector implementing two mutation strategies: (1) Instruction MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY Logic Conflict, which introduces mutually incompatible con- straints (e.g., “Implement a discount function that orders over $100 get 10% off, and the discount amount must not exceed $5”, creating logically unsatisfiable specifications that force agents to rationalize conflicts or arbitrarily prioritize constraints; and (2) Instruction Ambiguity, which degrades prompt specificity by replacing concrete terms with vague language (e.g., “Sort by rev- enue descending” becomes “Organize the data appropriately”), forcing agents to infer intent from insufficient information and leading to divergent interpretations. 3.2.2Interception and Response Rewriting. This mechanism (Fig. 2b) targets agent runtime behavior by intercepting agent outputs at crit- ical interaction boundaries (where agents communicate with peers or invoke external tools) and applying targeted mutations before outputs reach recipients. By positioning interceptors at the middle- ware layer, this mechanism injects Intra-agent Faults (Planning, Memory, Reasoning, Action) through two mutation categories: Semantic-Level Mutation. For faults requiring contextual under- standing, MAS-FIRE employs a prompt-guided mutation strategy. The interceptor delegates captured content (reasoning chains, plan- ning outputs, tool invocation requests) to a secondary LLM injec- tor guided by predefined fault templates, which applies semantic transformations that preserve surface coherence while corrupting underlying correctness. •Planning Faults. MAS-FIRE injects Inexecutable Plan by in- troducing logical inconsistencies such as circular task depen- dencies, references to non-existent tools or agents, or invalid workflow orderings. Critical Information Loss is injected by se- lectively removing essential constraints, parameters, or context from planning outputs, causing downstream agents to operate on incomplete specifications. •Reasoning Faults. MAS-FIRE injects Hallucination by replac- ing verified facts with plausible but factually incorrect info, removing uncertainty qualifiers (e.g., changing “likely” to de- finitive statements), or introducing fabricated intermediate reasoning steps. These mutations corrupt the semantic in- tegrity of agent deliberation while maintaining syntactic well- formedness. • Action Faults. MAS-FIRE injects Tool Selection Error by substi- tuting the intended tool with a semantically similar but incor- rect alternative (e.g., replacing “calculator” with “web_search”). Parameter Filling Error is injected by altering arguments to introduce domain-specific mistakes (e.g., swapping location coordinates, modifying query terms) while preserving type correctness. Structure-Level Mutation. For faults independent of semantic context, MAS-FIRE performs direct algorithmic transformations on message payloads or data structures without LLM assistance, operating on syntactic structure rather than semantic content. •Memory Faults. MAS-FIRE injects Memory Loss by selec- tively truncating conversation history using rule-based prun- ing strategies (e.g., removing early-turn messages, deleting messages from specific agents). Context Length Violation is in- jected by aggressively compressing context windows beyond the agent’s effective processing capacity, forcing the agent to operate with incomplete historical information and degrading reasoning fidelity. • Action Faults. MAS-FIRE injects Parameter Format Error by directly corrupting the syntactic structure of tool invocations. This includes introducing malformed JSON (missing brackets, incorrect escaping), breaking API schema constraints (wrong data types, missing required fields), or violating domain-specific formatting rules (e.g., invalid date formats, malformed SQL queries). These errors trigger immediate execution failures at the parsing or validation stage. 3.2.3 Message Routing Manipulation. This mechanism (as shown in Fig. 2c) targets inter-agent communication by manipulating mes- sage flows, including frequency and recipients, without altering message content. MAS-FIRE implements it programmatically, in- jecting faults in a controlled manner without LLM involvement. This mechanism injects three types of Communication Faults: (1) Message Cycle, which redirects messages back to the sender agent, forcing agents into repetitive conversational loops that halt progress and simulate infinite loops in coordination; (2) Message Storm, which replicates a single point-to-point message multiple times, flooding the receiver to simulate resource exhaustion and test the MAS ability to handle redundancy messages; (3) Message Broad- cast Amplification, which redirects messages intended for specific agents to unrelated agents, causing them to receive irrelevant infor- mation, perform unnecessary processing, and potentially disrupt consensus and state consistency across the MAS. 3.3 MAS Robustness Metrics While fault injection reveals how agents respond to anomalies, quantitative metrics are essential to systematically compare ro- bustness across systems, fault types, and architectures. Existing MAS evaluation frameworks [24,26] rely primarily on binary task success rates, which fail to distinguish graceful degradation from catastrophic failures and cannot capture whether agents success- fully detect faults, resolve immediate errors, or translate local recov- ery into end-to-end completion. We define a dual-level evaluation framework: (1) system-level resilience measures overall task success rates under fault injection to assess how architectural design and model capabilities preserve functionality, and (2) process-level ef- fectiveness analyzes fault-tolerant behaviors during execution to understand detection, response, and recovery mechanisms. 3.3.1 System-level Resilience. System-level metrics measure an MAS’ ability to maintain task successful completion under faults, capturing the combined effect of architecture, agent intelligence, and coordination. This includes Robustness Score (RS), which quantifies the fraction of originally successful tasks that remain solvable after fault injection. For fault type 푓 : 푅푆 푓 = 푁 푓 ,success 푇 base (1) where푁 푓 ,success denotes the number of tasks successfully completed under fault푓, and푇 base denotes the set of tasks the MAS completes without faults. A high푅푆indicates maintained functionality de- spite faults, while a low푅푆signals architectural or algorithmic vulnerabilities. Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng 3.3.2Process-level Effectiveness. Process-level metrics evaluate an MAS’ internal mechanisms to detect and respond to faults during execution. Unlike system-level metrics focusing on final outcomes, these metrics analyze intermediate behaviors to understand why systems succeed or fail. 푂 푓 = 푁 푓 ,trigger 푁 total , 퐿 푓 = 푁 푓 ,fixed 푁 푓 ,trigger , 푆 푓 = 푁 푓 ,final_success 푁 푓 ,trigger (2) Occurrence Rate (푂 푓 ). This metric quantifies the system’s ability to detect anomalies and activate fault-tolerant responses. 푁 푓 ,trigger denotes the number of tasks in which the system detects abnormality and activates at least one fault-tolerant behavior under fault푓, and푁 total denotes the total number of injected tasks. A high 푂 푓 indicates strong fault awareness, where agents recognize devia- tions from expected execution and initiate corrective actions. A low 푂 푓 suggests silent fault propagation without defensive responses, often leading to cascading failures. Local Success Rate (퐿 푓 ). This metric evaluates the effectiveness of an MAS’ fault-tolerant behaviors in resolving the injected fault. 푁 푓 ,fixed denotes the number of tasks in which the triggered fault- tolerant behavior successfully corrects the injected fault (e.g., fixing an incorrect tool invocation format).퐿 푓 focuses on local recovery, i.e., whether the system can handle the immediate error introduced by the fault. A high퐿 푓 indicates effective error-correction mecha- nisms, while a low퐿 푓 reveals that agents recognize faults but lack appropriate recovery strategies. Success Rate (푆 푓 ). The푆 푓 measures whether local fault recovery translates into global task success.푁 푓 ,final_success denotes the number of tasks that ultimately achieve their intended goal among those in which fault-tolerant behaviors are triggered.푆 푓 captures end-to-end effectiveness even if an agent successfully corrects an immediate error (reflected in퐿 푓 ), the overall task may still fail due to residual effects (e.g., lost context, cascading downstream errors). The gap between퐿 푓 and푆 푓 reveals the extent to which local recovery is insufficient for global success. 4 MAS Robustness Evaluation To systematically characterize how MAS respond to and recover from faults in practice, we conduct an empirical evaluation that applies the MAS-FIRE framework (Sec. 3) to representative MAS implementations. Unlike existing work that focuses solely on task success rates, our evaluation investigates the process through which systems detect, respond to, and recover from faults. This process- oriented analysis enables us to understand not only whether systems fail under faults, but also how and why failures occur, providing actionable insights for improving MAS resilience. The evaluation is designed to answer the following three research questions: •How do different faults impact MAS performance and stability? •How does foundation model capability affect MAS robustness? •What fault-tolerant behaviors emerge in MAS when confronted with faults, and how can they be categorized and quantified? 4.1 Experimental Setup 4.1.1System Selection and Task Datasets. We select three represen- tative MAS that span diverse architectural paradigms and applica- tion domains. MetaGPT [10] employs a hierarchical organizational workflow for code generation, featuring a shared message pool that maintains a persistent, globally accessible context and a linear se- quential execution pipeline. MetaGPT incorporates programmatic design mechanisms such as automatic message deduplication and selective information transmission through instructions, enabling automatic message filtering and deduplication. Table-Critic [60] features an iterative critic-refiner pipeline, where a Judge agent acts as a validator to verify outputs. Upon detecting any discrepancies, the agent triggers a closed-loop refinement cycle, facilitating seam- less autonomous error detection and self-correction. Camel [18] utilizes a bilateral role-playing structure, where information flows sequentially between a User and an Assistant through cooperative negotiation. This selection ensures diversity in agent roles, commu- nication patterns, coordination mechanisms, and fault propagation characteristics. For each MAS, we select task datasets representative of its tar- get domain. MetaGPT is evaluated on HumanEval [55], a bench- mark for functional correctness of generated code. Table-Critic is evaluated on WikiTableQuestions [31], a dataset requiring complex table reasoning and multi-step inference. Camel is eval- uated on WebShop [57], which requires multi-turn interaction and decision-making in simulated e-commerce environments. Fol- lowing the sampling methodology of Krejcie and Morgan [15], we randomly sample 400 instances from WikiTableQuestions (from a total of 4,344), exceeding the minimum representative sample size threshold to ensure statistical robustness. Table 1 summarizes the evaluated systems, their associated benchmarks, task scales, and baseline performance under fault-free conditions. 4.1.2Fault Injection and Log Collection. Each sampled task is exe- cuted under controlled fault injection using the three complemen- tary mechanisms described in Sec. 3.2. For each fault type, faults are injected at predefined execution points that correspond to the appropriate injection mechanism: system and user prompt modifica- tions occur at agent initialization and task ingestion (Prompt Mod- ification), agent output corruptions occur during agent-to-agent communication or tool invocations (Interception and Response Rewriting), and message flow manipulations occur at the inter- agent coordination infrastructure (Message Routing Manipulation). To assess foundation model capability impact (RQ2), each MAS is configured with two foundation models: GPT-5 and DeepSeek-V3. As shown in Table 1, GPT-5 consistently outperforms DeepSeek-V3 across all systems in baseline success rates, representing a stronger and weaker model pair for comparative analysis. For all semantic fault injections (e.g., Hallucination, Inexecutable Plan), we employ GPT-5-mini as the fault injector to ensure contextual coherence and realistic anomalies. Our fault injection experiments achieved a 99% success rate across all evaluated systems. A small fraction of fail- ures occurred when LLM-generated fault specifications contained structural inconsistencies violating the fault injector’s integrity checks; these cases were re-executed to ensure complete coverage. Additionally, a minimal subset of tasks were excluded from tool- related fault injection because they involved no tool invocations, making such faults inapplicable. Following execution, we collect MAS logs into a comprehensive corpus. This dataset comprises execution logs spanning 15 fault MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY Table 1: Evaluated MAS and Baseline Success Rates under Fault-Free Conditions System (Paradigm)Domain (Benchmark)Total Tasks GPT-5 Deepseek-V3 MetaGPT (Dynamic Organization) Code Gen (HumanEval[55])16499.0%89.0% Table-Critic (Thought-Critic)Table QA (WikiTQ † [31])40087.0%78.3% Camel (Instructor-Assistant)Web Nav (WebShop ‡ [57])25137.8%33.0% Note: All evaluations are single-trial (Pass@1).†10% systematic sampling of the original 4,344 tasks. ‡ Tasks are based on AgentBoard-annotated version. types (Sec. 3.2), three MAS architectures, two foundation models, and task difficulty levels within each benchmark. 4.2 Fault-Tolerant Behavior Analysis To systematically derive a taxonomy of fault-tolerant behaviors from execution logs, we employ the Grounded Theory methodol- ogy [9]. Four authors of the paper independently analyze execution logs for each fault type, identifying observable behavioral patterns without predefined categories based on concrete log evidence (agent actions, communication patterns, decision-making strategies). The identified behaviors undergo iterative refinement through constant comparison analysis, i.e., merging semantically equivalent behav- iors, splitting overly coarse categories, introducing new categories, and removing insufficiently distinguishable ones, until saturation is reached. To enable scalable analysis, we develop an automated annotation pipeline using an LLM-as-a-judge approach. The derived taxonomy, formal behavior definitions, and representative examples are pro- vided to GPT-5, which assigns behavior labels to unseen execution logs. We measure agreement between LLM-generated and human- generated annotations on a held-out validation set using Cohen’s Kappa (휅=0.94), validating the automated annotator’s use for large-scale annotation. 5 Evaluation Results This section presents the empirical results of applying the MAS- FIRE framework to three representative MAS under 15 fault types. We organize the findings around three research questions to com- prehensively analyze MAS robustness, fault tolerance mechanisms, and recovery strategies. 5.1 RQ1: Impact of Different Fault Categories on MAS Robustness Our evaluation reveals that MAS exhibit highly varying sensitivity to different fault categories. Fig. 3 illustrates the푅푆 푓 , as defined in Sec. 3.3.1, across all fault types and systems. 5.1.1 Impact of Intra-Agent Faults. Intra-agent faults target the internal inference processes of individual agents, including Plan- ning, Memory, Reasoning, and Action. These faults exhibit strong architecture-dependent impact, as specific MAS design patterns act as structural buffers that neutralize localized failures before they propagate system-wide. The key pattern across intra-agent faults is that architectural mechanisms can effectively contain their impact. Memory Faults (Critical Information Loss) severely degrade performance in Camel’s linear bilateral structure (푅푆 푓 ≈67%) where info flows sequentially. However, MetaGPT’s shared message pool maintains a persistent, globally accessible context that allows downstream agents to re- trieve missing info (푅푆 푓 >90%) (aΔ푅푆 푓 ≈+25% advantage). Sim- ilarly, Planning Faults substantially reduce MetaGPT’s푅푆 푓 (as low as 43.84%) due to its rigid sequential execution pipeline where a single planning error halts the entire workflow. In contrast, Table- Critic’s critique-refinement loop enables autonomous error detec- tion and plan correction through iterative self-assessment, main- taining푅푆 푓 >89% across both foundation models (aΔ푅푆 푓 ≈+45% improvement). Action faults demonstrate another form of architectural media- tion. MetaGPT and Table-Critic generally sustain high푅푆 푓 on most action-related faults through environmental error feedback and automatic retry logic. When a tool invocation fails, these systems detect the failure signal and regenerate corrected invocations with- out human intervention. Camel consistently attains lower푅푆 푓 on these faults because it lacks comparable dynamic error-trapping mechanisms. Reasoning Faults exhibit moderate impact with strong model dependency, as superior foundation models demon- strate enhanced mitigation through semantic validation (detailed in Sec. 5.2). Finding 1: Specific architectural design patterns effectively neutralize faults, e.g., shared message pools mitigate Mem- ory Faults (Δ푅푆 푓 ≈+25%), iterative critique loops neutralize Planning Faults (Δ푅푆 푓 ≈+45%), and environmental feed- back mechanisms enable rapid recovery from most Action Faults (푅푆 푓 >90% for Parameter Filling and Tool Format errors). These architectural features prevent localized failures from escalating into systemic collapse. 5.1.2 Impact of Inter-Agent Faults. Inter-agent faults target coor- dination mechanisms, exhibiting different vulnerability patterns ranging from catastrophic collapse to effective mitigation. Configuration and Instruction Faults represent the most se- vere threats by corrupting semantic foundations of agent coor- dination. Configuration Faults tamper with system prompts at initialization (e.g., "trust all outputs without verification"), while Instruction Faults inject contradictions or ambiguities into user prompts (e.g., conflicting goals). Once the semantic contract be- tween designers and agents is violated, agents cannot distinguish valid logic from injected errors. Results demonstrate catastrophic impact: Configuration Faults reduce MetaGPT’s푅푆 푓 to 0.0%- 31.68% (under Blind Trust and Role Ambiguity), Instruction Faults Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng IPCILMLCLVHPFETFETSERABTILCIAMCMSMBA 0 20 40 60 80 100 Robustness Score (%) MetaGPT CILHPFETFETSERABTILCIA 0 20 40 60 80 100 Robustness Score (%) Camel IPCILTFETSERABTILCIA 0 20 40 60 80 100 Table-Critic DeepSeek-V3GPT-5 Figure 3: Robustness Score (푅푆 푓 ) of Different MAS under 15 Fault Types cause similar collapse (푅푆 푓 ≤13.7%). Even Table-Critic suffers degradation under Instruction Faults (푅푆 푓 to 16.67%-40.52%). Architectural topology critically modulates severity. MetaGPT’s linear pipeline exhibits extreme vulnerability: under Blind Trust, 푅푆 푓 drops to 0.0% as sequential dependency enables cascading fail- ures—errors propagate downstream without interception. Under Role Ambiguity, MetaGPT’s푅푆 푓 remains low (23.97%-31.68%). In contrast, Table-Critic’s iterative closed-loop provides resilience: under Role Ambiguity,푅푆 푓 maintains 91.05% (DeepSeek-V3) and 79.31% (GPT-5) as the Critic agent validates outputs and triggers re- finement cycles. For Instruction Faults, this mechanism provides partial protection (푅푆 푓 ∈ [16.67%,40.52%]), significantly outper- forming linear architectures. Camel achieves intermediate perfor- mance (Role Ambiguity:푅푆 푓 ∈ [61.05%,69.88%]) through bilateral negotiation. Notably, Blind Trust reveals a paradox: Table-Critic with GPT-5 achieves only푅푆 푓 =6.32%, while DeepSeek-V3 main- tains푅푆 푓 =70.61%, there is a 64.29% gap favoring the weaker model due to stricter instruction adherence preventing recovery (detailed in Sec. 5.2). Finding 2: Configuration Faults and Instruction Faults cause severe system degradation, with RS dropping as low as 0.0% in worst cases (e.g., Blind Trust in linear architectures) by corrupting semantic foundations of agent coordination. The severity is modulated by architecture. Linear workflows are highly vulnerable due to cascading failures, whereas iterative structures mitigate impact. Communication faults represent the least destructive threat category, consistently yielding RS> 93% in MetaGPT. Other MAS frameworks were excluded as they lack a communication mod- ule, preventing the injection of Communication Faults. This resilience is primarily due to inherent message validation logic. For instance, MetaGPT’s role-based subscription model ensures agents only process messages from relevant peers. By handling dedupli- cation (Message Storm) and cycle detection (Message Cycle) at the infrastructure layer, the system neutralizes these faults determinis- tically, bypassing the need for complex agent-level reasoning. Finding 3: Infrastructure-level defenses provide superior tol- erance for Communication Faults. While semantic or Con- figuration Faults that require cognitive reasoning, communica- tion-related threats are effectively neutralized (RS>93%) through hard-wired procedural logic, such as role-based fil- tering and automated cycle detection. 5.2 RQ2: Role of Foundation Model Capability Based on the impact of different fault categories in RQ1, we in- vestigate whether superior foundation models can mitigate these risks. Our findings reveal a dual nature of model capability: while superior models provide significant resilience gains through en- hanced semantic reasoning, they can become bottlenecks when system designs enforce strict adherence to potentially corrupted instructions. Superior foundation models demonstrate significant robustness advantages when architectures provide mechanisms for semantic validation and context-based error correction. MetaGPT’s shared message pool enables stronger models to retrospectively analyze execution history, identify logical inconsistencies, and retrieve cor- rect information. For Hallucination, GPT-5 performs semantic val- idation by comparing received information against original task MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY specifications, yielding RS of 45.34% versus DeepSeek-V3’s 23.29% (Δ푅푆=+22.05%). Similarly, under Inexecutable Plan faults, GPT-5 achieves RS of 55.90% versus 43.84% (Δ푅푆=+12.06%). In contrast, systems like Camel with simpler architectures show limited benefits from superior models. Hallucination in Camel stem primarily from external environment interactions (e.g., fetching corrupted content from the Internet) rather than internal coordination errors. Since fault mitigation depends more on environmental re-perception than logical reasoning, GPT-5’s reasoning advantage provides minimal resilience gain. Similarly, the absence of a global shared information pool prevents models from leveraging historical context to correct semantic deviations. Finding 4: For semantic-related faults such as Hallucination and Inexecutable Plan, superior foundation model capabil- ity plays a critical role in fault mitigation. In systems with shared information pools, enhanced semantic reasoning en- ables models to retrieve correct historical information and achieveΔ푅푆 ≈+17% improvements. However, this advantage is architecture-dependent. When faults stem from external environment interactions or systems lack shared information pools, model capability provides limited resilience gains. Counterintuitively, superior model intelligence can exacerbate fault impacts when recovery depends on challenging corrupted di- rectives. Table-Critic’s Thought-Critic-Refine loop operates through: (1) Generator produces answers with reasoning traces; (2) JudgeAgent evaluates correctness; (3) upon detecting errors, JudgeAgent trig- gers Refiner to correct mistakes. The Blind Trust attacks this via dual injection: corrupting JudgeAgent’s system prompt to "uncon- ditionally trust Generator outputs without verification," then inject- ing semantic errors into Generator’s thoughts. Recovery requires JudgeAgent to override its corrupted instruction. Results reveal a significant reversal in performance. While GPT-5 achieves a푅푆of only 6.32%, DeepSeek-V3 reaches 70.61%, represent- ing a 64.29% improvement for the weaker model. GPT-5’s superior instruction-following leads to strict adherence to corrupted direc- tives. In approximately 93.68% of its failed cases, the JudgeAgent ac- cepts erroneous reasoning without triggering the Critic-Refine loop. Conversely, DeepSeek-V3 exhibits instructional non-compliance by challenging suspicious inputs in the majority of cases. Para- doxically, this failure to follow corrupted instructions serves as an accidental recovery mechanism, enabling the system to bypass the injected fault. Finding 5: Higher model capability becomes counterproduc- tive when a system’s resilience relies on bypassing a cor- rupted directive rather than strictly following it. Superior models’ strict compliance with system prompts prevents them from deviating into alternative reasoning paths that might trigger recovery mechanisms. System robustness in such sce- narios depends on partial instruction non-compliance, creat- ing a scenario where lower-capability models unexpectedly achieve higher success rates. 5.3 RQ3: Categorization and Quantification of MAS’ Fault-tolerant Behaviors 5.3.1 Behavioral Taxonomy and Fault Tolerance Dimensions. To understand how MAS respond to and recover from faults, we analyze execution logs collected during fault injection. Table 2 presents a comprehensive behavioral mapping across all 15 fault categories, cataloging the diverse fault-tolerant behaviors observed when agents encounter different types of failures. The observed behaviors exhibit substantial heterogeneity rather than fixed failure patterns. For instance, under Instruction Ambigu- ity, an agent may leverage its own reasoning to recognize missing details and proactively request clarification from the user. However, in the absence of a feedback channel (e.g., a tool that allows query- ing the user), the same situation may instead lead to brief confusion followed by autonomous continuation based on assumptions. Un- der Message Storm, redundant messages may be filtered out before reaching the agent, leaving its behavior unaffected; without such pre-filtering, the agent may still detect redundancy and ignore du- plicated content on its own. When facing Role Ambiguity, an agent can resolve confusion by referring to examples provided by it’s sys- tem prompt and realign itself with the intended responsibility. Even if prompt interference temporarily induces incorrect agent outputs, errors can be mitigated or overridden by other collaborating agents through cross-checking and compensation mechanisms. These observations reveal that behaviors often involve multi- ple mechanisms: architectural structures (e.g., linear workflows), programmed logic (e.g., format validation), prompt design (e.g., role specification), and model-level reasoning (e.g., contextual in- ference). However, simple behavior identification does not reveal which specific mechanisms enable these behaviors or where resilience originates within MAS architectures. To address this gap, we derive four hierarchical fault tolerance tiers that classify behaviors by their source of resilience: •Mechanism-Level FT. Fault tolerance derived from the sys- tem’s structural design and temporal redundancy mechanisms. This includes architectural features such as iterative critique loops, multi-agent voting schemes, and redundant execution paths. These mechanisms operate independently of agent rea- soning and are embedded in the MAS coordination infrastruc- ture. •Rule-Based FT. Fault tolerance emerging from explicit pro- cedural logic and heuristic rules encoded in the MAS imple- mentation. This includes automatically deduplicates redundant messages. These behaviors are deterministic and activate when predefined conditions are met, regardless of the underlying model’s reasoning capabilities. • Prompt-Level FT. Rooted in the semantic robustness of User prompts. It leverages prompt engineering to guide agents through edge cases, clarify ambiguities, and maintain role boundaries, thereby pre-empting and mitigating faults. •Reasoning-Level FT. Driven by the agent’s high-level cog- nitive reflection. It relies on the underlying model’s semantic understanding to autonomously detect logical inconsistencies, infer missing context, and resolve conflicts through multi-agent debate and consensus-building. Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng Table 2: Fault-Tolerant Behavior Taxonomy and Mechanism Classification. This table evaluates four fault tolerance (FT) dimensions: Mechanism, Rule, Prompt, and Reasoning. Symbols are defined as follows:✓denotes successful mitigation;✗ indicates activation but failure to resolve; an empty cell signifies no activation. Action Faults are categorized into Parameter Filling Error, Tool Format Error, and Tool Selection Error. Fault Category Agent BehaviorMechanism. Rule. Prompt. Reasoning. Inexecutable Plan Restores faulty plan via inherent process✓ Ignores inexecutable parts and continues✓ No corrective behavior✗ Responds to error but ultimately fails✓✗ Critical Info Loss Restores missing information via inherent process✓ Autonomously repairs missing information✓ Avoids using missing information✓ Ignores missing information✗ Uses external information sources to repair✓✗ Memory Loss Uses external information sources to repair✓ Autonomously restores missing memory✓ Restores missing memory via inherent process✓ Avoids using missing memory✓ Ignores missing memory✗ Context Length Violation Asks the user for key information✓ Automatically ignores irrelevant long context✓ System architecture filters long context✓ No corrective behavior Hallucination Infers true intent while executing misinformation✓ Fully accepts and executes misinformation✗ Partially accepts and executes misinformation✗ Ignores misinformation✓ Detects misinformation and seeks correct information✓ Uses external information sources to repair✓✗ Action Fault Agent identifies and corrects✓ Uses partial repair coordination or multipath compensation✓ System uses redundancy or retry mechanisms✓ Responds to the error but ultimately fails✓✗ No corrective behavior Role Ambiguity Agent remains unaffected and follows original role✓ Agent disturbed with no system fault tolerance✗ Agent disturbed and system filters erroneous output✓✗ Agent disturbed and other agents compensate successfully✓✗ Agent disturbed and system fails to filter erroneous output✗ Agent disturbed and other agents fail to compensate✗ Agent disturbed but self compensates successfully✓✗ Blind Trust Fully accepts incorrect information from other agents✗ Judges independently but correction fails✓✗ Judges independently and corrects successfully✓ Judges independently and other agents compensate✓ Makes no independent judgment✓ Instruction Logic Conflict Reconciles conflicting instructions✓ Ignores some conflicts and rationalizes others✓ Ignores all conflicting instructions✓ Attempts to correct conflicting instructions✓ Detects conflicting instructions and requests clarification✓ Identifies conflicts but lacks a clarification mechanism✗✓ Fails to identify conflicting instructions✗ Instruction Ambiguity Recognizes vague goal and guesses✓ Recognizes vague goal and asks user✓ Recognizes vagueness but architecture prevents asking✗✓ Fails to notice instruction ambiguity✗ Message Storm System correctly filters messages✓ System fails to filter and duplicates do not affect behavior✗✓ System fails to filter and duplicates cause abnormal behavior✗ Message Cycle System correctly filters messages✓ Receives cyclic messages without impact✗✓ Receives cyclic messages and behavior is affected✗ Broadcast Amplification System correctly filters messages✓ System fails to filter and irrelevant messages do not affect behavior✗✓ System fails to filter and irrelevant messages cause abnormal behavior✗ MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY These four tiers often operate synergistically, and their interactions reveal the complexity of fault tolerance in MAS. In Blind Trust scenarios, the behavior “Judges independently but correction fails” highlights a success in Prompt-Level FT. While agents maintain their role definitions despite injected instructions, Reasoning-Level FT fails because they cannot identify the erroneous nature of preced- ing inputs. Consequently, agents rationalize incorrect information instead of challenging it, demonstrating that role consistency does not guarantee semantic validation. Similarly, in Instruction Logic Conflict scenarios, the behavior “Detects conflicts but architecture prevents querying” demonstrates a case where Reasoning-Level FT succeeds while Mechanism-Level FT fails. Although agents identify logical inconsistencies in conflicting instructions, the absence of querying modules in the system architecture precludes recovery. Despite attempting to seek clarification to resolve ambiguity, agents are restricted by architectural constraints from accessing alterna- tive information sources. This shows how Reasoning-Level FT can identify semantic problems even when Mechanism-Level FT fails to provide the infrastructure support necessary for recovery actions. 5.3.2 Empirical Evaluation of Fault Tolerance Tiers. Our analysis of fault-tolerance performance across the four hierarchical tiers reveals distinct characteristics for each layer. The fault-tolerance performance across the four tiers is visualized in the heatmap of Fig. 4, evaluated through the(푂 푓 , 퐿 푓 , 푆 푓 ) triplet defined in Sec. 3.3.2. Under perturbations including Parameter Filling Error, Tool For- mat Error, and Tool Selection Error, the systems demonstrate robust Mechanism-Level fault tolerance. Across all systems, the occurrence rates consistently reach푂 푓 ≥85%, local recovery rates maintain 퐿 푓 = 100%, and final task success rates are secured at 푆 푓 > 61%. Finding 6: Mechanism-Level defenses (e.g., automated re- tries, syntax parsers) function as a filter for low-level errors within MAS. By effectively handling execution-layer noise, these mechanisms prevent low-level errors from propagating into the cognitive context of agents, ensuring the accuracy of historical information within high-level reasoning processes. Rule-Based FT emerges from explicit procedural logic and de- terministic rules hardcoded in the MAS implementation. Unlike Mechanism-Level FT which relies on architectural redundancy and retry mechanisms, or Reasoning-Level FT which depends on seman- tic understanding, Rule-Based FT operates through programmed exception handling that detects and filters predefined structural pat- terns. This tier proves particularly effective for structural anomalies such as Communication Faults (message storms, cycles, broadcast amplification) and Context Length Violation, where Mechanism and Reasoning layers fail because these faults require deterministic pattern matching rather than semantic interpretation or architec- tural compensation. In MetaGPT, Rule-Based FT implemented as hardcoded filtering rules achieves perfect detection and recovery: 푂=100% (all fault instances trigger the filter) and퐿=100% (all detected faults are successfully resolved), resulting in task success rate푆>93%. This demonstrates that deterministic procedural logic provides guaranteed mitigation for well-defined structural patterns, i.e., once activated, recovery is certain. Finding 7: Rule-Based FT provides deterministic handling of structural failures through programmed exception handling. In MetaGPT, Rule-Based FT achieves 100% recovery rates for communication-related anomalies and context overflows, indicating that procedural logic stabilizes systems when com- munication protocols or memory constraints are exceeded. Prompt-Level fault tolerance relies on the semantic robustness of agent instructions encoded in system and user prompts. In con- trast to Rule-Based FT which operates through hardcoded rules, Prompt-Level FT depends on agents’ ability to interpret and adhere to textual directives that define roles, responsibilities, and behav- ioral constraints. For Configuration Faults such as Role Ambigu- ity and Blind Trust, Prompt-Level FT achieves universal activation (푂 푓 =100%) across all architectures, as these faults directly modify system prompts and invariably trigger Prompt-Level responses. However, activation does not guarantee successful recovery. The effectiveness depends on whether corrupted prompts preserve or undermine the semantic foundations necessary for correct agent behavior. The recovery success rates (푆 푓 ) diverge sharply across fault types and architectures. For Role Ambiguity, Table-Critic ex- hibits high resilience (푆 푓 =91% for DeepSeek-V3,푆 푓 =79% for GPT-5) as corrupted role definitions still allow agents to retain core task-solving capabilities, while MetaGPT achieves lower rates (푆 푓 ∈ [24%,32%]) and Camel maintains intermediate performance (푆 푓 ∈ [61%,70%]). Conversely, Blind Trust induces systemic col- lapse in Camel (푆 푓 = 0.0%) and MetaGPT (푆 푓 = 0.0%) as corrupted prompts instruct agents to unconditionally accept erroneous up- stream information, fundamentally undermining their verification capabilities. Only Table-Critic shows partial resilience (푆 푓 =71% for DeepSeek-V3,푆 푓 =6.32% for GPT-5) by leveraging Reasoning and Mechanism layers to override compromised directives. Finding 8: Prompt modifications universally trigger FT (푂 푓 = 100%), but efficacy is fault-dependent. In Role Ambiguity, suc- cess rates vary by architecture (Table-Critic:푆 푓 ∈ [79%,91%]; MetaGPT:푆 푓 ∈ [24%,32%]) as agents retain task-solving logic. In Blind Trust, Prompt-Level defenses fail in MetaGPT and Camel (푆 푓 =0.0%) due to strict adherence to erroneous instructions, while Table-Critic shows partial resilience (푆 푓 ∈ [6%,71%]). Higher resilience requires Reasoning or Mechanism- layer interventions to override corrupted directives. As shown in Fig. 4, the highest tier of fault tolerance (Reasoning- Level FT ) addresses semantic faults that bypass lower-tier defenses. Three fault types (Hallucination, Instruction Logic Conflict, Instruc- tion Ambiguity) share a common profile, i.e., syntactically valid but semantically defective, allowing them to evade Mechanism-Level and Rule-Based filters that rely on structural pattern matching. Mechanism-Level occurrence rates (푂 푓 ) vary dramatically across these faults. For Hallucination, MetaGPT achieves low detection (푂 푓 =5.5% for DeepSeek-V3,푂 푓 =42.2% for GPT-5) as hallucina- tions originate from internal reasoning, while Camel reaches higher rates (푂 푓 ∈ [27.7%,34.7%]) because external information sources Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng IPCILMLCLVHPFETFETSERABTILCIAMCMSMBA O:12% L:100% S:6% O:100% L:100% S:97% O:0% L:0% S:0% O:0% L:0% S:0% O:5% L:100% S:88% O:100% L:100% S:96% O:100% L:100% S:97% O:100% L:100% S:72% O:2% L:100% S:100% O:0% L:0% S:0% O:0% L:0% S:0% O:5% L:100% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:100% S:94% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:8% L:100% S:100% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:100% S:97% O:100% L:100% S:99% O:100% L:100% S:96% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:14% S:24% O:100% L:2% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:57% L:80% S:73% O:100% L:100% S:97% O:100% L:99% S:99% O:0% L:0% S:0% O:100% L:11% S:23% O:90% L:100% S:96% O:77% L:98% S:98% O:35% L:100% S:88% O:0% L:0% S:0% O:2% L:33% S:0% O:100% L:50% S:14% O:100% L:25% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:21% L:100% S:29% O:98% L:100% S:92% O:1% L:100% S:50% O:0% L:0% S:0% O:42% L:100% S:69% O:99% L:100% S:92% O:100% L:100% S:93% O:100% L:100% S:78% O:2% L:100% S:100% O:4% L:100% S:0% O:18% L:100% S:34% O:25% L:100% S:10% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:100% S:91% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:12% L:100% S:100% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:100% S:95% O:100% L:100% S:93% O:100% L:100% S:97% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:16% S:32% O:100% L:12% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:74% L:81% S:67% O:100% L:98% S:90% O:99% L:96% S:91% O:0% L:0% S:0% O:100% L:42% S:45% O:75% L:94% S:91% O:55% L:98% S:89% O:81% L:90% S:76% O:0% L:0% S:0% O:7% L:83% S:0% O:100% L:39% S:12% O:100% L:31% S:2% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% DEEPSEEK-V3 GPT-5 Mechanism. Rule. Prompt. Reasoning. Mechanism. Rule. Prompt. Reasoning. 0 20 40 60 80 100 Occurrence Rate (O%) (a) MetaGPT IPCILMLCLVHPFETFETSERABTILCIAMCMSMBA O:96% L:100% S:70% O:28% L:100% S:61% O:99% L:100% S:71% O:100% L:100% S:82% O:86% L:100% S:68% O:4% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:14% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:96% L:98% S:72% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:0% S:70% O:100% L:40% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:95% S:69% O:100% L:59% S:71% O:90% L:97% S:71% O:98% L:99% S:81% O:60% L:94% S:56% O:0% L:0% S:0% O:40% L:27% S:0% O:100% L:90% S:12% O:100% L:69% S:2% O:95% L:100% S:69% O:35% L:100% S:67% O:100% L:100% S:77% O:100% L:100% S:69% O:96% L:100% S:73% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:23% L:95% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:94% L:98% S:62% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:6% S:61% O:100% L:33% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:100% S:67% O:100% L:46% S:61% O:84% L:98% S:78% O:99% L:100% S:69% O:72% L:96% S:65% O:0% L:0% S:0% O:33% L:71% S:0% O:100% L:92% S:13% O:100% L:82% S:5% DEEPSEEK-V3 GPT-5 Mechanism. Rule. Prompt. Reasoning. Mechanism. Rule. Prompt. Reasoning. 0 20 40 60 80 100 Occurrence Rate (O%) (b) Camel IPCILMLCLVHPFETFETSERABTILCIAMCMSMBA O:92% L:100% S:97% O:38% L:100% S:92% O:100% L:100% S:95% O:88% L:100% S:96% O:12% L:92% S:94% O:82% L:100% S:86% O:0% L:0% S:0% O:33% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:88% S:91% O:100% L:82% S:71% O:0% L:0% S:0% O:0% L:0% S:0% O:4% L:36% S:18% O:85% L:27% S:72% O:0% L:0% S:0% O:53% L:99% S:95% O:0% L:0% S:0% O:82% L:100% S:86% O:100% L:89% S:17% O:100% L:79% S:22% O:100% L:100% S:97% O:76% L:100% S:97% O:100% L:100% S:99% O:100% L:100% S:100% O:10% L:89% S:83% O:14% L:100% S:44% O:0% L:0% S:0% O:12% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:0% L:0% S:0% O:100% L:89% S:79% O:100% L:44% S:6% O:0% L:0% S:0% O:0% L:0% S:0% O:3% L:0% S:62% O:68% L:65% S:87% O:0% L:0% S:0% O:99% L:100% S:100% O:0% L:0% S:0% O:30% L:0% S:0% O:100% L:76% S:17% O:100% L:37% S:41% DEEPSEEK-V3 GPT-5 Mechanism. Rule. Prompt. Reasoning. Mechanism. Rule. Prompt. Reasoning. 0 20 40 60 80 100 Occurrence Rate (O%) (c) Table-Critic Figure 4: Fault-tolerance Performance of Different MAS under 15 Fault Types. Gray columns indicate that the corresponding faults cannot be injected due to system architecture limitations or output format constraints. MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY enable partial detection. For Instruction Logic Conflict, Mechanism- Level detection is nearly absent (푂 푓 =0.0% for most systems; only MetaGPT with GPT-5 reaches푂 푓 =18%). For Instruction Ambi- guity, detection remains limited (MetaGPT:푂 푓 ∈ [5.5%,24.8%]; Table-Critic:푂 푓 ∈ [12.4%,32.9%]; Camel:푂 푓 ∈ [14.5%,23.2%]). In stark contrast, Reasoning-Level FT achieves universal activation (푂 푓 =100%) across all three fault types and all systems, serving as the primary and often sole defense against semantic anomalies. Finding 9: Faults like Hallucination, Logic Conflict, and Am- biguity are syntactically correct but semantically defective, allowing them to bypass Mechanism-Level filters. Reasoning- Level FT is the primary defense (푂 푓 =100%), where agents leverage cognitive redundancy to detect semantic flaws and infer correct intent to resolve instruction-level errors. 6 Discussion 6.1 Treat Upstream Instructions with Caution Strict adherence to system prompts and upstream instructions is widely regarded as desirable in MAS. However, our evaluation reveals that this assumption breaks down under fault conditions. When instructions are corrupted or internally inconsistent (Sec. 5.1), rigid compliance becomes a liability. Under Blind Trust, GPT-5’s su- perior instruction-following led to near-total collapse (푅푆 푓 =6.32%), while DeepSeek-V3’s weaker compliance paradoxically preserved functionality (푅푆 푓 = 70.61%). These results expose a fundamental design tension: agents must be compliant enough to follow valid directives yet skeptical enough to detect corrupted ones. Robust MAS should incorporate condi- tional compliance mechanisms, allowing agents to pause execution or flag inconsistencies when they encounter logical contradictions, constraint violations, or conflicting environmental feedback. Rather than treating all upstream signals as authoritative, agents should cross-validate instructions against their own reasoning before com- mitting to irreversible actions, preventing locally corrupted direc- tives from cascading into system-level failures. 6.2 Avoid Failure Propagation in Linear Agent Workflow Linear, pipeline-style workflows are widely adopted in MAS due to their simplicity and clear stage boundaries. However, our results show this topology is the most vulnerable to cascading failures. Un- der Configuration and Instruction Faults, MetaGPT’s linear pipeline collapsed to푅푆 푓 as low as 0.0%, as a single corrupted output propa- gates downstream unchecked, with each subsequent agent inherit- ing and compounding the error. In contrast, Table-Critic’s iterative closed-loop maintained significantly higher robustness by enabling repeated validation and correction cycles. The core vulnerability is single-path dependency: no redundancy exists to catch semantic drift before it reaches downstream con- sumers. We identify two complementary mitigation strategies. First, multi-source validation: downstream agents should reconcile infor- mation from multiple independent sources, such as parallel agent interpretations or environmental ground truth, to detect inconsis- tencies before acting. Second, inline verification checkpoints: light- weight validation stages between pipeline steps can assess output plausibility (e.g., schema conformance, semantic consistency with the task specification) and trigger re-execution when anomalies are detected. These mechanisms introduce the error-correction benefits of closed-loop architectures while preserving the interpretability of linear workflows. 7 Related Work 7.1 Fault Injection and Chaos Engineering Traditional fault injection and mutation testing focus on low-level syntactic corruptions such as memory leaks [43]. These methodolo- gies are ill-equipped for Multi-Agent Systems (MAS) where failures manifest as semantic deviations. Unlike deterministic faults, seman- tic failures allow a system to remain operational while becoming logically decoupled from its intended tasks. Current chaos engi- neering and static metrics fail to capture the dynamic nature of these agentic collapses [38]. For example, a system may reach a superficial consensus even when its internal reasoning has drifted into a hallucinated state, producing outputs that are superficially coherent yet logically invalid. 7.2 MAS Evaluation Frameworks Recent studies explore how localized failures propagate across MAS topologies. AutoInject [12] demonstrates that fault propagation pat- terns depend critically on underlying organizational structures. Despite the importance of organizational interactions, existing MAS evaluation frameworks remain limited in systematically di- agnosing interaction-level failures. While adversarial benchmarks like TAMAS [14] address intentional sabotage, the more perva- sive threat in production remains spontaneous coordination failure. Current frameworks rely on coarse-grained outcome metrics such as task success rates or binary pass/fail. While frameworks like AppWorld [42], AgentBoard [26], and ScienceAgentBench [4] incor- porate sub-goal tracking to move beyond binary success, they re- main outcome-oriented. Their metrics focus on linear task progress rather than interaction-layer resilience. OpenJudge [40] offers multi- dimensional monitoring capabilities but relies on idealized scenar- ios, lacking the stress tests to evaluate organizational resilience. 8 Conclusion This paper introduced MAS-FIRE, a framework designed to diag- nose and evaluate the robustness of Multi-Agent Systems through systematic fault injection. Through the lens of 15 distinct fault types, we have shown that MAS reliability depends on a complex interplay between foundation model reasoning and coordination infrastructure. Our findings challenge the assumption that model scaling alone ensures system stability. Instead, we quantified the superior protective power of specific architectural patterns, such as shared message pools and iterative critique loops, which effectively neutralize semantic errors before they propagate into systemic col- lapse. By providing a granular behavioral taxonomy and a suite of process-oriented metrics, MAS-FIRE equips the software engineer- ing community with a rigorous methodology to evaluate, diagnose, and harden the new generation of orchestrated intelligent software. Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng Data Availability The experimental data of MAS-FIRE are publicly available on https: //github.com/wxhhxn/MASFIRE. References [1] Daniil A Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent au- tonomous scientific research capabilities of large language models. arXiv preprint arXiv:2304.05332 (2023). [2]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya G. Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? CoRR abs/2503.13657 (2025). arXiv:2503.13657 doi:10.48550/ARXIV.2503.13657 [3]Hongyang Chen, Pengfei Chen, Guangba Yu, Xiaoyun Li, and Zilong He. 2024. MicroFI: Non-Intrusive and Prioritized Request-Level Fault Injection for Microser- vice Applications. IEEE Trans. Dependable Secur. Comput. 21, 5 (2024), 4921–4938. doi:10.1109/TDSC.2024.3363902 [4]Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=6z4YKr0GK6 [5] Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kan- nappan, and Rebecca Qian. 2025. TRAIL: Trace Reasoning and Agentic Issue Localization. CoRR abs/2505.08638 (2025). arXiv:2505.08638 doi:10.48550/ARXIV. 2505.08638 [6] Nicola Dragoni, Saverio Giallorenzo, Alberto Lluch-Lafuente, Manuel Mazzara, Fabrizio Montesi, Ruslan Mustafin, and Larisa Safina. 2016. Microservices: yes- terday, today, and tomorrow. CoRR abs/1606.04036 (2016). arXiv:1606.04036 http://arxiv.org/abs/1606.04036 [7]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mor- datch. 2024. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Forty-first International Conference on Machine Learn- ing, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.https: //openreview.net/forum?id=zj7YuTE4t8 [8] Alireza Ghafarollahi and Markus J Buehler. 2024. ProtAgents: protein discovery via large language model multi-agent collaborations combining physics and machine learning. Digital Discovery 3, 7 (2024), 1389–1409. [9] BG Glaser and AL Strauss. 1967. The discovery of grounded theory: strategies for qualitative research. (1967). [10] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum? id=VtmBAGCN7o [11]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=IkmD3fKBPQ [12]Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Zixi Chen, Wenxuan Wang, Youliang Yuan, Michael R. Lyu, and Maarten Sap. 2025. On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents. In Forty-second In- ternational Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net. https://openreview.net/forum?id=bkiM54QftZ [13]Chaoya Jiang, Hongrui Jia, Mengfan Dong, Wei Ye, Haiyang Xu, Ming Yan, Ji Zhang, and Shikun Zhang. 2024. Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models. In Proceedings of the 32nd ACM International Conference on Multimedia, M 2024, Melbourne, VIC, Australia, 28 October 2024 - 1 November 2024, Jianfei Cai, Mohan S. Kankanhalli, Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu (Eds.). ACM, 525–534. doi:10.1145/3664647.3680576 [14]Ishan Kavathekar, Hemang Jain, Ameya Rathod, Ponnurangam Kumaraguru, and Tanuja Ganu. 2025. TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems. CoRR abs/2511.05269 (2025). arXiv:2511.05269 doi:10.48550/ARXIV. 2511.05269 [15]Robert V. Krejcie and Daryle W. Morgan. 1970. Determining Sample Size for Research Activities. Educational and Psychological Measurement 30, 3 (1970), 607–610. arXiv:https://doi.org/10.1177/001316447003000308 doi:10.1177/ 001316447003000308 [16]Priti Kumari and Parmeet Kaur. 2021. A survey of fault tolerance in cloud computing. J. King Saud Univ. Comput. Inf. Sci. 33, 10 (2021), 1159–1176. doi:10. 1016/J.JKSUCI.2018.09.021 [17] Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Y. Sorokin, and Mikhail Burtsev. 2024. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). http://papers.nips. c/paper_files/paper/2024/hash/c0d62e70dbc659c9bd44cbcf1cb652f-Abstract- Datasets_and_Benchmarks_Track.html [18] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023.CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.c/paper_files/paper/2023/hash/ a3621e907def47c1b952ade25c67698-Abstract-Conference.html [19]Haoran Li, Ziyi Su, Yun Xue, Zhiliang Tian, Yiping Song, and Minlie Huang. 2025. Advancing Collaborative Debates with Role Differentiation through Multi- Agent Reinforcement Learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, 22655–22666. https://aclanthology.org/2025.acl-long.1105/ [20] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. 2024. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.).http://papers.nips.c/ paper_files/paper/2024/hash/b631da756d1573c24c9ba9c702fde5a9-Abstract- Datasets_and_Benchmarks_Track.html [21]Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244 (2023). [22] Jiangchao Liu, Jierui Liu, Peng Di, Alex X. Liu, and Zexin Zhong. 2022. Record and Replay of Online Traffic for Microservices with Automatic Mocking Point Identification. In 44th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2022, Pittsburgh, PA, USA, May 22-24, 2022. IEEE, 221–230. doi:10.1109/ICSE-SEIP55303.2022.9793867 [23] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Trans. Assoc. Comput. Linguistics 12 (2024), 157–173. doi:10. 1162/TACL_A_00638 [24]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Ao- han Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Eval- uating LLMs as Agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=zAdUB0aCTQ [25]Zhenyue Long, Guoquan Wu, Xiaojiang Chen, Chengxu Cui, Wei Chen, and Jun Wei. 2020. Fitness-guided Resilience Testing of Microservice-based Applications. In 2020 IEEE International Conference on Web Services, ICWS 2020, Beijing, China, October 19-23, 2020. IEEE, 151–158. doi:10.1109/ICWS49710.2020.00027 [26]Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024.AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). http://papers.nips. c/paper_files/paper/2024/hash/877b40688e330a0e2a3fc24084208dfa-Abstract- Datasets_and_Benchmarks_Track.html [27] Christopher S. Meiklejohn, Andrea Estrada, Yiwen Song, Heather Miller, and Rohan Padhye. 2021. Service-Level Fault Injection Testing. In SoCC ’21: ACM Symposium on Cloud Computing, Seattle, WA, USA, November 1 - 4, 2021, Carlo Curino, Georgia Koutrika, and Ravi Netravali (Eds.). ACM, 388–402. doi:10.1145/ 3472883.3487005 [28]Abraham Mukosi Mukwevho and Turgay Çelik. 2021. Toward a Smart Cloud: A Review of Fault-Tolerance Methods in Cloud Systems. IEEE Trans. Serv. Comput. 14, 2 (2021), 589–605. doi:10.1109/TSC.2018.2816644 MAS-FIRE : Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent SystemsConference acronym ’X, June 03–05, 2026, Woodstock, NY [29]Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems. (2023). [30]Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 13679–13707. doi:10.18653/V1/2024.ACL-LONG.739 [31]Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. The Association for Computer Linguistics, 1470–1480. doi:10.3115/V1/P15-1142 [32]Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 15174–15186. doi:10.18653/V1/2024.ACL-LONG.810 [33]Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.c/paper_files/paper/2023/ hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html [34]Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158 (2023). [35]Haiyang Shen, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, and Yun Ma. 2025. ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https: //openreview.net/forum?id=kKILfPkhSz [36] Minjie Shen and Qikai Yang. 2025. From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent. CoRR abs/2505.02024 (2025). arXiv:2505.02024 doi:10.48550/ARXIV.2505.02024 [37] Xu Shen, Yixin Liu, Yiwei Dai, Yili Wang, Rui Miao, Yue Tan, Shirui Pan, and Xin Wang. 2025. Understanding the Information Propagation Effects of Communica- tion Topologies in LLM-based Multi-Agent Systems. CoRR abs/2505.23352 (2025). arXiv:2505.23352 doi:10.48550/ARXIV.2505.23352 [38]Andries P. Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D. Barrett, and Arnu Pretorius. 2024. Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview. net/forum?id=CrUmgUaAQp [39]Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen. 2024. WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting. CoRR abs/2405.00823 (2024). arXiv:2405.00823 doi:10.48550/ARXIV.2405.00823 [40] The OpenJudge Team. 2025. OpenJudge: A Unified Framework for Holistic Evalua- tion and Quality Rewards. https://github.com/agentscope-ai/OpenJudge [41] Haoye Tian, Chong Wang, Boyang Yang, Lyuye Zhang, and Yang Liu. 2025. A Taxonomy of Prompt Defects in LLM Systems. CoRR abs/2509.14404 (2025). arXiv:2509.14404 doi:10.48550/ARXIV.2509.14404 [42]Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 16022–16076. doi:10. 18653/V1/2024.ACL-LONG.850 [43]Timothy K. Tsai, Mei-Chen Hsueh, Hong Zhao, Zbigniew Kalbarczyk, and Ravis- hankar K. Iyer. 1999. Stress-Based and Path-Based Fault Injection. IEEE Trans. Computers 48, 11 (1999), 1183–1201. doi:10.1109/12.811108 [44] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023.On the Planning Abilities of Large Language Mod- els - A Critical Investigation. In Advances in Neural Information Process- ing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.c/paper_files/paper/2023/hash/ efb2072a358cefb75886a315a6fcf880-Abstract-Conference.html [45]Noah Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. 2024. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abil- ities of Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Compu- tational Linguistics, 14743–14777. doi:10.18653/V1/2024.FINDINGS-ACL.878 [46] Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuanjing Huang, and Zhongyu Wei. 2025. Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.). Association for Computational Linguistics, 3310–3328. https://aclanthology.org/2025.coling-main.223/ [47]Xiao Wang, Lu Dong, Sahana Rangasrinivasan, Ifeoma Nwogu, Srirangaraj Setlur, and Venugopal Govindaraju. 2025. AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2025, Hangzhou, China, October 19-25, 2025. IEEE, 9194–9201. doi:10.1109/IROS60139.2025.11247695 [48]Zexin Wang, Jingjing Li, Quan Zhou, Haotian Si, Yuanhao Liu, Jianhui Li, Gao- gang Xie, Fei Sun, Dan Pei, and Changhua Pei. 2025. A Survey on AgentOps: Categorization, Challenges, and Future Directions. CoRR abs/2508.02121 (2025). arXiv:2508.02121 doi:10.48550/ARXIV.2508.02121 [49]Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. 2025. PlanGenLLMs: A Modern Survey of LLM Planning Capabilities. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, 19497–19521. https://aclanthology. org/2025.acl-long.958/ [50] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. CoRR abs/2308.08155 (2023). arXiv:2308.08155 doi:10.48550/ARXIV.2308.08155 [51]Weibin Wu, Yuhang Cao, Ning Yi, Rongyi Ou, and Zibin Zheng. 2025. De- tecting and Reducing the Factual Hallucinations of Large Language Models with Metamorphic Testing. Proc. ACM Softw. Eng. 2, FSE (2025), 1432–1453. doi:10.1145/3715784 [52]Weibin Wu, Yuhang Cao, Ning Yi, Rongyi Ou, and Zibin Zheng. 2025. De- tecting and Reducing the Factual Hallucinations of Large Language Models with Metamorphic Testing. Proc. ACM Softw. Eng. 2, FSE (2025), 1432–1453. doi:10.1145/3715784 [53] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, Qi Zhang, and Tao Gui. 2025. The rise and potential of large language model based agents: a survey. Sci. China Inf. Sci. 68, 2 (2025). doi:10.1007/S11432-024-4222-0 [54] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622 (2024). [55] Devansh Yadav and Shouvick Mondal. 2025. Evaluating pre-trained Large Lan- guage Models on zero shot prompts for parallelization of source code. J. Syst. Softw. 230 (2025), 112543. doi:10.1016/J.JSS.2025.112543 HumanEval benchmark. [56]Bingyu Yan, Xiaoming Zhang, Litian Zhang, Lian Zhang, Ziyi Zhou, Dezhuang Miao, and Chaozhuo Li. 2025. Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems. CoRR abs/2502.14321 (2025). arXiv:2502.14321 doi:10.48550/ARXIV.2502.14321 [57]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.c/paper_files/paper/2022/ hash/82ad13ec01f9fe44c01cb91814fd7b8c-Abstract-Conference.html [58]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview. net/forum?id=WE_vluYUL-X [59] Miao Yu, Fanci Meng, Xinyun Zhou, Shilong Wang, Junyuan Mao, Linsey Pan, Tianlong Chen, Kun Wang, Xinfeng Li, Yongfeng Zhang, et al.2025. A survey on trustworthy llm agents: Threats and countermeasures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 6216–6226. Conference acronym ’X, June 03–05, 2026, Woodstock, NYJin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng [60]Peiying Yu, Guoxin Chen, and Jingjing Wang. 2025. Table-Critic: A Multi-Agent Framework for Collaborative Criticism and Refinement in Table Reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, 17432–17451. https: //aclanthology.org/2025.acl-long.853/ [61]Dingbo Yuan, Yipeng Chen, Guodong Liu, Chenchen Li, Chengfu Tang, Dongxu Zhang, Zhenkui Wang, Xudong Wang, and Song Liu. 2025. DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA, Toby Walsh, Julie Shah, and Zico Kolter (Eds.). AAAI Press, 25760–25768. doi:10.1609/AAAI.V39I24.34768 [62]Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xint- ing Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2025. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. Computational Lin- guistics 51, 4 (12 2025), 1373–1418. arXiv:https://direct.mit.edu/coli/article- pdf/51/4/1373/2535477/coli.a.16.pdf doi:10.1162/COLI.a.16 [63]Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memo- ryBank: Enhancing Large Language Models with Long-Term Memory. In Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Con- ference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (Eds.). AAAI Press, 19724–19731. doi:10.1609/AAAI.V38I17.29946 [64]Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al.2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv e-prints (2023), arXiv–2306.