Paper deep dive
Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?
Franka Bause, Jonas Niederle, Martin Pawelczyk, Rebekka Burkholz
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 7/8/2026, 8:32:32 PM
Summary
The paper models multi-agent LLM deliberation using the Friedkin-Johnsen opinion dynamics framework, demonstrating that input-dependent parameters transform multi-agent systems into adaptive mixture-of-experts systems. It shows that routing influence toward locally competent agents—proxied by self-assessed confidence and initial belief alignment—enables multi-agent systems to outperform single agents and static ensembles.
Entities (10)
Relation Signals (10)
Friedkin-Johnsen model → exhibits → Input-Dependence
confidence 95% · We show that the FJ parameters are input-dependent
Multi-Agent Systems → modeledby → Friedkin-Johnsen model
confidence 95% · We study this mechanism through the lens of Friedkin-Johnsen (FJ) opinion dynamics...
Routing → drivenby → Self-Assessed Confidence
confidence 90% · relative confidence is the primary factor influencing the routing
Friedkin-Johnsen model → hasparameter → Stubbornness
confidence 90% · γ_i ∈ [0, 1] denotes the attachment to innate beliefs also called stubbornness
Friedkin-Johnsen model → hasparameter → Peer Influence
confidence 90% · Peer Influence Pull
Multi-Agent Systems → outperforms → Single Agents
confidence 90% · multi-agent systems can outperform single agents and static ensembles when routing reflects agent competence.
Self-Assessed Confidence → proxiesfor → Competence
confidence 90% · Since competence is latent in practice, we analyze how influence is established through observable proxies: agents' self-assessed confidence...
Mixture of Experts → requires → Routing
confidence 90% · routing reflects agent competence.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The effectiveness of multi-agent LLM deliberation depends not only on the agents' individual predictions, but also on how they communicate and collaborate. We study this mechanism through the lens of Friedkin-Johnsen (FJ) opinion dynamics, a tractable model for analyzing stubbornness, influence, and opinion change in multi-agent systems that captures empirically observed deliberation patterns. We show that the FJ parameters are input-dependent, turning multi-agent deliberation into a mixture of experts. This perspective implies that multi-agent systems can outperform single agents and static ensembles when routing reflects agent competence. Since competence is latent in practice, we analyze how influence is established through observable proxies: agents' self-assessed confidence, their perceived confidence, and initial alignment with other agents' views.
Tags
Links
- Source: https://arxiv.org/abs/2605.25929v2
- Canonical: https://arxiv.org/abs/2605.25929v2
Trouble viewing inline? Open PDF directly →
Full Text
100,994 characters extracted from source content.
Expand or collapse full text
Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer? Franka Bause * 1 Jonas Niederle * 1 Martin Pawelczyk 2 Rebekka Burkholz 1 Abstract The effectiveness of multi-agent LLM deliber- ation depends not only on the agents’ individ- ual predictions, but also on how they communi- cate and collaborate. We study this mechanism through the lens of Friedkin-Johnsen (FJ) opin- ion dynamics, a tractable model for analyzing stubbornness, influence, and opinion change in multi-agent systems that captures empirically ob- served deliberation patterns. We show that the FJ parameters are input-dependent, turning multi- agent deliberation into a mixture of experts. This perspective implies that multi-agent systems can outperform single agents and static ensembles when routing reflects agent competence. Since competence is latent in practice, we analyze how influence is established through observable prox- ies: agents’ self-assessed confidence, their per- ceived confidence, and initial alignment with other agents’ views. 1. Introduction Large language models (LLMs) have led to significant ad- vancements in natural language processing across various tasks. Recently, multi-agent systems (MASs) composed of interacting LLMs have attracted attention for their poten- tial to improve performance, particularly in tasks involving strategic reasoning, negotiation, and generative design (Du et al., 2024; Wu et al., 2024; Qian et al., 2024; Chen et al., 2024; Hong et al., 2024). In MASs, multiple agents commu- nicate iteratively to deliberate and refine predictions, with the promise that diverse agents can contribute complemen- tary expertise and improve decision-making. * Equal contribution 1 CISPA Helmholtz Center for Informa- tion Security, Saarbr ̈ ucken, Germany 2 Faculty of Computer Sci- ence, University of Vienna, Vienna, Austria. Correspondence to: Franka Bause<franka.bause@cispa.de>, Jonas Niederle <jonas.niederle@cispa.com>. Accepted to the 2nd Workshop on Compositional Learning at ICML 2026, Seoul, South Korea. This arXiv version corre- sponds to the workshop paper. Workshop website:https: //compositional-learning.github.io/ However, the empirical benefits of MASs over single-agent models or static ensembles are mixed (Smit et al., 2024; Zhang et al., 2025; Tran & Kiela, 2026). A key factor in the success of MASs is understanding how influence is distributed during deliberation. Not all agents are equally persuasive, and the central question is: What makes some agents more influential than others in a multi-agent deliberation? We analyze this problem through the Friedkin-Johnsen (FJ) model of opinion dynamics (Friedkin & Johnsen, 1990), which has been successfully applied to model belief prop- agation in social networks and, more recently, in LLM de- liberations (Abedini et al., 2026). While FJ dynamics have been used to assess security risks in MASs, we focus on the observation that the FJ model’s parameters are input- dependent, which leads to an important insight: multi-agent deliberation can be interpreted as a mixture of experts (MoE) system (Jacobs et al., 1991). In this framework, each agent contributes to the final decision based on its initial belief and its influence weight, which varies depending on the input. Accordingly, the MAS implicitly implements an adaptive routing mechanism. Its performance hinges on the degree to which influence is directed toward the most competent agents for a given input. Since true competence is latent, we focus on observable proxies for competence, such as self- assessed confidence, peer influence, and initial alignment with the other agents. Our findings demonstrate that relative confidence is the primary factor influencing the routing, with the initial alignment and social behavior also playing relevant roles. Contributions. 1) We use the Friedkin-Johnsen model to provide a tractable description of belief propagation, stub- bornness, and peer influence in LLM-based MASs. 2) We show that input-dependent FJ parameters induce a mixture- of-experts interpretation of multi-agent deliberation. 3) We theoretically and empirically analyze how influence emerges from observable proxies for latent competence, identifying confidence and opinion alignment as central drivers of the implicit routing mechanism. Together, these results suggest that successful MAS design requires not only diverse agents, but also reliable mechanisms for routing influence toward agents that are locally competent. 1 arXiv:2605.25929v2 [cs.MA] 17 Jun 2026 Multi-Agent Systems are Mixtures of Experts 1.1. Related work LLM-based MASs and deliberation. Recent work ex- plores the use of LLMs interacting through structured or unstructured communication to improve reasoning and ro- bustness (Wu et al., 2024; Li et al., 2023). Techniques such as self-consistency (Wang et al., 2023) and chain-of-thought prompting (Wei et al., 2022) demonstrate that aggregating diverse opinions and reasoning can outperform single-pass inference. However, existing methods largely lack a princi- pled framework on how opinions evolve through communi- cation and how influence shapes the outcome. Opinion dynamics and social influence models. The Friedkin-Johnsen (FJ) model (DeGroot, 1974; Friedkin & Johnsen, 1990; 2011) has been applied extensively in the so- cial sciences to understand how group consensus is formed, how stubbornness and belief retention interact with peer influence, and how changes in network structure can affect collective decision-making (Parsegov et al., 2017; Tian & Wang, 2018; Abedini et al., 2026). Recently, it has been used to model opinions co-evolving with platform learning systems (Wu et al., 2026) and creating simulation environ- ments of human opinion formation (He et al., 2026; Wang et al., 2025). Recent work has demonstrated that FJ dy- namics can effectively capture the belief formation process in MASs to study the systemic risk induced by stubborn agents (Abedini et al., 2026). Surprisingly, compared to more complex cascades processes (Burkholz et al., 2018b;a; Burkholz & Schweitzer, 2018; Burkholz & Quackenbush, 2021; Gotovos et al., 2021) of social influence (Burkholz et al., 2016; Flache et al., 2017), simple linear dynamics suffice to capture LLM deliberation. Interpretability of MASs. Research on the interpretability of MASs has explored how agents’ behavior can be ex- plained, particularly in the context of collaborative problem- solving and collective decision-making (Wooldridge, 2018; Rosenfeld & Richardson, 2019; Lee et al., 2025). A chal- lenge in multi-agent settings is understanding the emergent behaviors that arise from complex inter-agent interactions. Our work adds to this growing body of research by provid- ing a framework for explaining influence in MASs through the lens of Friedkin-Johnsen belief propagation (Abedini et al., 2026) and a mixture-of-experts interpretation. Mixture of experts. The concept of mixture-of-experts (MoE) models has been widely studied in the machine learn- ing literature, where multiple models (or experts) are as- signed responsibility for different regions of the input space. In these models, a gating network determines which ex- pert(s) to trust based on the input, effectively routing the problem to the most relevant expert. MoE models have been used to improve performance on complex tasks by allowing specialized models to handle different subproblems (Jacobs et al., 1991; Jordan & Jacobs, 1994; Chen et al., 2022; Sun et al., 2026). In contrast to the standard MoE setting, we focus on collaborative MAS, where routing is not explic- itly learned in a supervised setting but implicitly realized through deliberation. 2. Theoretical framework: Agentic deliberation as mixtures of experts Our work builds on the observation that LLM deliberation dynamics can be described by the Friedkin-Johnsen model (see also Table 1), which enables us to cast multi-agent LLM systems as mixture of experts. Accordingly, influence emerges as the result of a routing mechanism, which we set out to understand based on the confidence of agents and communication behavior. Friedkin-Johnsen (FJ) model. Let each agenti∈ Vin an agentic networkG = (V,E)withn = |V|agents hold a beliefb i (t)∈ ∆ d , where∆ d ⊂ [0, 1] d is thed-dimensional simplex representing a probability distribution over potential outcomes, e.g., answers to a multiple-choice question. Each agentiis characterized by an innate beliefb i (0) = s i ∈ ∆ d , which may be interpreted as the agent’s individual prediction before deliberation. The belief update at timet + 1in the Friedkin-Johnsen model (Friedkin & Johnsen, 1990) for belief dynamics is defined as (Abedini et al., 2026): b i (t + 1) = γ i s i |z Prior Belief Pull + (1− γ i )α i b i (t) | z Belief Retention + (1− γ i )(1− α i ) X j∈Ni w ij b j (t) | z Peer Influence Pull , (1) whereγ i ∈ [0, 1]denotes the attachment to innate be- liefs also called stubbornness,α i ∈ [0, 1]represents the weight given to the previous state, andW = [w ij ]is a row-stochastic influence matrix where P j w ij = 1 (and w i = 0, as self-loops are covered byα i ). The term (1 − γ i )(1 − α i )represents the agent’s susceptibility to external influence. We can also formulate this in matrix no- tation. LetB(t)∈ R n×d denote the matrix whosei-th row isb i (t) ⊤ , and letS ∈ R n×d denote the matrix whosei-th row iss ⊤ i . Define diagonal matricesΓ = diag(γ 1 ,...,γ n ), A = diag(α 1 ,...,α n ), andH = (I− Γ) A + (I−A)W . Then the deliberation dynamics can be written compactly as B(t + 1) = ΓS + HB(t). As LLM deliberation dynamics are captured by the FJ model, the multi-agent system (MAS) computes a convex combination of the initial beliefs of its agents. Proposition 2.1 (Graph-induced convex combination (Fried- kin & Johnsen, 1990; Proskurnikova & Tempo, 2017)). As- sume thatρ(H) < 1, whereρ(·)denotes the spectral radius. 2 Multi-Agent Systems are Mixtures of Experts 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (a) Parameter variability. W: weighted in-degrees (avg. over senders). InitialFinal round 0.0 0.2 0.4 0.6 Disagreement (b) Belief disagreement. Figure 1. FJ induces MoE. (a) FJ parameter variability across MMLU-Pro questions for GPT-5.4 Mini. (b) Despite diverse initial opinions (left), consensus is often reached (right). Then the FJ dynamics converge to the unique equilibrium B ⋆ = (I − H) −1 ΓS.M := (I − H) −1 Γ = [m ij ]is nonnegative and row-stochastic. Consequently, each equi- librium belief is a convex combination of the innate be- liefs:b ⋆ i = P n j=1 m ij s j . If the agentic network belief b ⋆ out = P n i=1 η i b ⋆ i = P n j=1 π j s j withπ ⊤ = η ⊤ Mis formed by a linear aggregation of all agents’ beliefs (typ- icallyη i = 1/N), the agentic network is equivalent to an ensemble with weightsπ j that depend on the FJ parameters (Γ,A,W ). A system that follows Friedkin-Johnsen dynamics with static parameters(Γ,A,W )would therefore implement an ensem- ble. This can be already beneficial compared to a single- agent model, as ensembles can combine the opinion of diverse agents, achieve a variance reduction and are poten- tially more robust to distribution shift (Dietterich, 2000; Reddy et al., 2026a;b). A direct implication of this insight is that agentic networks tend to benefit from high agent diversity. Yet, our empirical findings suggest that MASs can be even more powerful, as they do not rely on static FJ parameters, but adapt them to the input questionsx, resulting in FJ pa- rameters(Γ(x),A(x),W (x))that induce input-dependent routing of agent influences π i (x). Hypothesis 2.2. A multi-agent LLM system (MAS) can be cast as mixture of experts (MoEs)b ⋆ (x) = P n j=1 π j (x)s j (x) , where the routerπ j (x)depends on the input x. Fig. 1a provides empirical evidence for this hypothesis. It implies that the two central quantities of interest that deter- mine the performance of the MAS are a) the diversity of agents and b) the routing. While a) is a critical design choice, it is often given and only mildly influenced by prompting in our LLM deliberation experiments. b) Routing is performed automatically by the LLM agents. The main purpose of the introduced FJ framework is to facilitate the discovery of its main underlying mechanisms. Our analysis reveals that routing primarily relies on initial beliefss i (x), even though these are not explicitly communicated by the LLM agents. However, they serve as proxy for the agents’ states that also informs their behavior. They define their confidence, com- petence, and to which degree their opinions align initially. Based on this hypothesis, i.e. an MAS acts as an MoE that performs routing based on initial agent beliefs (π = π(S)), for which we provide empirical evidence, our following theoretical discussion aims to derive conditions when MASs outperform single agents and even ensembles. 2.1. When do MASs outperform single agents and ensembles? MASs have shown promising performance gains over single agents. This is especially true when the task has compo- sitional structure, agents contribute diverse information or reasoning strategies, and communication allows for error correction (Du et al., 2024; Wu et al., 2024; Qian et al., 2024; Hong et al., 2024). However, recent evaluations find only modest or inconsistent gains over strong single-agent baselines (Smit et al., 2024; Zhang et al., 2025; Cemri et al., 2025; Tran & Kiela, 2026), calling for a deeper understand- ing of successful MAS design principles. Our link to MoEs allows us to draw on theoretical insights. MoEs are a special case of ensembles. For ensembles, the routing is constant across inputs. While it is well known that agent diversity is critical to their function (Dietterich, 2000), theoretical studies of MoEs have focused on router properties (Jacobs et al., 1991; Jordan & Jacobs, 1994; Jiang & Tanner, 1999) or how clustered structure in the input data can be exploited (Chen et al., 2022; Sun et al., 2026) in the context of supervised learning problems, where the router has access to predictive features of labels. In our context of LLM deliberation, no router has been trained explicitly and the MAS does not have access to label information. Intuitively, an ideal router would assign high influence to competent agents. Yet, agent competence is 3 Multi-Agent Systems are Mixtures of Experts input question dependent, latent, and therefore unavailable information. Agent beliefs, however, can be interrogated. We therefore ask the question: When can belief-derived signals identify which agent is locally competent and drive MAS performance? To answer this question, our analysis decomposes perfor- mance into three terms: global ensemble diversity, local competence diversity, and routing regret. This lets us com- pare single agents, static ensembles, and MoE-style delibera- tion in terms of agent diversity, complementary information, and confidence-based routing that rests on sufficiently well- calibrated agents, whose confidence reflects competence. In particular, we show that adaptive deliberation improves over fixed ensembling only when the gain from local spe- cialization exceeds the diversity lost by no longer averaging all agents and the regret incurred by imperfect confidence- based routing. Belief-dependent routing, diversity, and performance. To formalize this intuition, we assume that each agent j ∈ [n]outputs an initial beliefs j (X) ∈ ∆ d about the right answer to questionX, which is a random variable over the set of possible questions, and writeS(X) = (s 1 (X),...,s n (X)) for the collection of all beliefs. A MAS outputsf (X) = P n j=1 π j (S(X))s j (X) and its performance is measured with the Brier lossℓ(y,p) = ∥p− e y ∥ 2 2 , wheree y denotes the one-hot vector for belief in the correct answery. For each agent, define the belief- conditional riskr j (S) = E ∥s j (X)− e Y ∥ 2 2 | S(X) = S . This is the true local competence of agentjgiven the observable beliefsS, which is generally unknown to the system. For any weight vectora ∈ ∆ n , define the mix- ture ̄s a = P n j=1 a j s j and the local initial belief diversity, which measures how diverse the agents’ beliefs are under weights a: D a (S) = n X j=1 a j ∥s j − ̄s a ∥ 2 2 = 1 2 n X i,j=1 a i a j ∥s i − s j ∥ 2 2 . (2) Lemma 2.3 (Local ambiguity decomposition). For any belief-dependent weights a(S)∈ ∆ n , E n X j=1 a j (S)s j (X)− e Y 2 2 | S = n X j=1 a j (S)r j (S)− D a(S) (S). This composition has the following interpretation: The first term rewards putting weight on locally competent agents. The second term rewards averaging diverse beliefs. Comparing MAS, single agents, and ensembles. This composition enables us to compare different routing choices a(S) ∈ ∆ n over the set of all questions by taking an average also overXandSadditionally toY, yielding L = P n j=1 E[a j (S)r j (S)]− E[D a(S) (S)]. In the follow- ing, we use the notationa(S)for a general router,π(S)for a MoE, andΨ(S) = Ψfor an ensemble, where the routing is independent of the input. Theorem 2.4 (MAS vs. single agent). A multi-agent system with the mixture of experts routingπ(S)outperforms the best single agent j ∗ with lowest risk min j∈[n] E[r j (S)] if E[r j ∗ (S)− min j r j (S)] | z specialization gain + E D π(S) (S) | z local diversity > E[δ π (S)] |z routing regret , (3) whereδ π (S) = P n j=1 π j (S)r j (S)−min j∈[n] r j (S)defines the routing regret. The theorem is verified experimentally by Fig. 20. The proof follows directly from taking the average over the respective losses and demanding the loss of the MoE to be lower. The first term is the specialization advantage over the best global agent. It is large when no single agent is best everywhere. The second term is the local diversity benefit retained by the MoE. The right-hand side is the cost of imperfect routing. Therefore, MoE beats the best single agent if specialization gain plus local diversity exceeds routing regret. This implies that a multi-agent system does not improve sim- ply by combining many agents. Its agents have to be locally competent, complementary, and the routing mechanism can identify them from observable belief signals. However, note that this analysis does not exclude the exis- tence of another more powerful and potentially more com- plex single expert model, which could e.g. be obtained by distillation of the MAS. It could also implement a similar mechanism as routing by choosing good answers from a sample like self-consistency (SC) (Zhang et al., 2025). At the same time, such single models could potentially also be valuable experts in a MAS if they are combined with other complementary experts. Note that ensembles are special cases of MoEs and there- fore follow a similar logic. However, they cannot achieve a local specialization gain through routing and the above condition simplifies toE[D a (S)] > P n j=1 a j E[r j (S)]− min j∈[n] E[r j (S)], highlighting the relevance of expert di- versity. The difference in this analysis points also out how MoEs can potentially outperform ensembles, i.e. by exploit- ing local diversity. 4 Multi-Agent Systems are Mixtures of Experts 0.000.250.500.751.00 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Belief in correct answer r per MAS: 0.65 ± 0.61 Correct Incorrect Figure 2. Association of competence and confidence for ChatGPT- 5.4 Mini on MMLU-Pro. Theorem 2.5 (MAS vs. ensemble). A multi-agent system with the mixture of experts routingπ(S)outperforms an ensemble with constant routing weights ψ if E n X j=1 ψ j − π j (S) r j (S) |z local routing gain > E D ψ (S)− D π(S) (S) | z diversity loss . (4) The left-hand side is the gain (resulting from lower aver- age risk) from assigning more weight to locally competent agents. The right-hand side is the diversity that could be lost by moving away from the fixed ensemble. Thus, MoE wins if local competence routing gain exceeds lost ensemble diversity. Hard routing towards the most competent agent. As a special case, we can also consider hard routing, where a sin- gle agentj ′ (S)receives all weight, leading to the condition E h P n j=1 ψ j r j (S)− r j ′ (S) (S) i > E[D ψ (S)], because the local diversityD π(S) (S) = 0. Thus, hard routing must com- pensate for the diversity benefit with a sufficiently better local expert. Our analysis of real LLM deliberation reveals a preference of MAS to select a single or only a few experts, suggesting a strong tendency towards local specialization rather than exploiting diversity through ensembling. Inter- estingly, even though the agents do not appear to be very diverse with respect to their average performance, their in- trinsic randomness leads to local diversity in the their initial beliefs, which is exploited by the routing. In the following, we discuss scenarios how such routing is enabled by proxies for local expert competence in the initial beliefs. 2.2. Confidence-based routing as a competence proxy The local risksr j (S)are not observed during deliberation. The router must therefore use proxies derived from the be- liefs. A natural self-assessment of competence is their confi- denceC j (S)∈ [0, 1]in a specific answer, which we propose to measure based on the entropyH(s j )of their initial belief: C j (S) = 1− 1 logd H(s j ) = 1 + 1 logd d X c=1 s j,c logs j,c . (5) A confidence-based router may take the formπ j (S) = exp(βC j (S))/ P n ℓ=1 exp(βC ℓ (S)), whereβ ≥ 0controls how strongly the router favors confident agents. Fig. 3 presents an example, where the MAS simply picks the opin- ion of the initially most confident agent. As we show, not only absolute confidence of agents matters but also confi- dence relative to the other agents. Thus, all predictions are taken into account. Such routers are beneficial only when confidence is suffi- ciently calibrated with competence (see also Fig. 2). One way to formalize calibration is to assume that there ex- ists a decreasing functionφsuch thatr j (S) ≈ φ(C j (S)). Equivalently, agents with higher confidence should have lower conditional risk. Letj C (S) ∈ arg max j C j (S)be the most confident agent, and letj ⋆ (S) ∈ arg min j r j (S) be the most competent agent. For hard confidence rout- ing, the routing regret isδ C (S) = r j C (S) (S)− r j ⋆ (S) (S). The confidence-routed MoE beats a fixed ensemble with weightsψifE[G ψ (S)] > E[δ C (S)] + E[D ψ (S)],where G ψ (S) = G w (S) = P n j=1 ψ j r j (S)− min j∈[n] r j (S) mea- sures how much a fixed ensemble wastes probability mass on agents that are not locally optimal. This condition cap- tures both the strength and the limitation of confidence rout- ing. If confidence reliably identifies competence, thenδ C (S) is small and the MoE can exploit local specialization. If agents are overconfident when wrong, thenδ C (S)can be large and the MoE may underperform a fixed ensemble. To get an intuitive understanding of this, let us discuss two examples. Case: Mutually exclusive and optimally calibrated agents. Ideal agents of a MAS have a specialized local expertise that is complementary to other agents and signal reliably when they are competent. Optimal agentsjwith limited performance budgetB j would specialize and thus spend their budget on disjoint sets of questions. Let the input space decompose into disjoint regions,X = S n j=1 X j ,X i ∩X j = ∅fori ̸= jwithPr(X ∈ X j ) = ρ j , 5 Multi-Agent Systems are Mixtures of Experts Answers 0 .25 .5 .75 1 concise Answers 0 .25 .5 .75 1 emotional Answers 0 .25 .5 .75 1 elaborate Answers 0 .25 .5 .75 1 balanced Answers 0 .25 .5 .75 1 step-by-step Init Final 0.00 0.25 0.50 0.75 1.00 W value Indicator (intensity=conf.) CorrectIncorrect Figure 3. Example with communication styles on GPT-5.4 Mini: Initial beliefs of agents (left). FJ weight matrix,γ, and color coded belief in correct answer (right). The most confident agent convinces the majority to change their answer. ρ j > 0, P n j=1 ρ j = 1. On regionX j , agentjis competent and confident, while all other agents are uninformative. Let p = 1− εwithε ∈ (0, 1− 1/d), and letu = 1/ddenote the probability assigned to the true class by the uniform distribution. For every inputx ∈ X j , assume that agent jassigns probabilitypto the true class, while every other agent assigns probabilityuto the true class:s j,Y (x) = p, s i,Y (x) = ufor alli̸= j. Thus, each agent is competent on exactly one region and uninformative elsewhere. Assuming a symmetric belief that assigns probabilityqto the correct label and distributes the remaining mass uniformly over the incorrect labels, achieves a Brier loss ofB d (q) = (1− q) 2 + (d− 1) 1−q d−1 2 = d d−1 (1− q) 2 . A MoE router that selects the competent agent on every region thus achieves Brier lossL MoE =B d (p). In comparison, a fixed ensemble with weightsψ ∈ ∆ n assigns on regionX j probability q j (ψ) = u+(p−u)ψ j to the correct label. Its expected Brier loss is therefore L ens (ψ) = P n j=1 ρ j B d (u + (p− u)ψ j ). Proposition 2.6 (MoE advantage under mutually exclusive competence). Assumen≥ 2,ρ j > 0for allj, andp > u. Then L MoE < inf ψ∈∆ N L ens (ψ). For balanced regionsρ j = 1/nfor allj, symmetry and strict convexity imply that the optimal fixed ensem- ble is the uniform ensembleψ ⋆ j = 1/n. Writingq n = u + p−u n , the loss gap then becomesL ⋆ ens − L MoE = d d−1 (1− q n ) 2 − (1− p) 2 > 0 . For fixedpandd, this gap increases with the number of agents. The best sin- gle agentj glob ∈ arg max j∈[n] ρ j is the agent whose competence region has largest probability.Its loss is L ⋆ single = ρ j glob B d (p) + 1− ρ j glob B d (u) . Consequently, L ⋆ single − L MoE = 1 − ρ j glob [B d (u)−B d (p)]. This means, the MoE therefore strictly improves over the best single agent whenever at least two competence regions have positive probability. Effect of routing errors. The preceding result assumes that confidence perfectly identifies the competent agent. Sup- pose instead that the router selects the correct agent with probability1− δand selects an uninformative agent with probabilityδ. Then the routed system assigns true-class probabilitypwith probability1− δanduwith probability δ, Its expected Brier loss isL router (δ) = (1− δ)B d (p) + δB d (u) . In the balanced setting, this router improves over the optimal fixed ensemble if and only ifδ < B d (q n )−B d (p) B d (u)−B d (p) . Thus, confidence-based routing is beneficial only when the confidence signal is sufficiently reliable. This formalizes the central limitation of task-dependent FJ deliberation: adap- tive influence can realize MoE-like gains, but miscalibrated confidence can route trust to the wrong agent and erase the advantage. As experts are mutually exclusive in this ideal case, a MAS can generally route towards the single most competent ex- pert, while the other agents focus their performance budget on other questions. Note that the maximum belief predic- tions of ensembles are still correct (albeit with lower cer- tainty). As a consequence, expert optimization could make complex MAS interaction unnecessary. The real advantage of MoE routing is realized with imperfect agents, which are more common in practice. Case: Routing with imperfect agents. More interesting cases that highlight the advantages of MoE routing assume imperfect agents. Let us consider a scenario that is compa- rable with the observed case in Fig. 3, where the majority of agents makes a wrong prediction, confusing an ensem- ble, but the most confident agents convinces the rest during deliberation. As before, we observe disjoint regions of com- petence but with less well calibrated agents that correlate their belief in wrong labels, which makes the problem harder to solve. Assumed≥ 3. On regionX k , lety k be the correct label and letz k ̸= y k be a common incorrect label. The competent agentkpredictss k,y k = p,s k,c = b := 1−p d−1 forc ̸= y k . Every non-competent agent predictss j,y k = u,s j,z k = c, s j,ℓ = r := 1−u−c d−2 forj ̸= kandℓ /∈ y k ,z k . Assume 0 < u < 1 d < p < 1 ,0 < c < 1− u,c > maxu,r. The last condition formalizes that the non-competent agents concentrate their probability on the same wrong answer. We further assume that the competent agent is the most confident:C k (S(x)) > C j (S(x))for everyj ̸= kand 6 Multi-Agent Systems are Mixtures of Experts x ∈ X k . Hard confidence routing selects the competent agent and achieves L MoE =B d (p) as before. For a fixed ensembleψ ∈ ∆ n , its prediction on regionX k has probabilitiesq y,k = u + (p− u)ψ k ,q z,k = c + (b− c)ψ k andq o,k = r + (b− r)ψ k on each of the remaining d− 2labels. For balanced regions, its expected Brier loss is L ens (ψ) = 1 n P n k=1 h (1− q y,k ) 2 + q 2 z,k + (d− 2)q 2 o,k i . Proposition 2.7 (Confidence routing beats fixed ensem- bling). Assumen ≥ 2,d ≥ 3, and the conditions above. Then the optimal fixed ensemble is the uniform ensemble withψ ⋆ k = 1 n and confidence routing strictly improves over it:L MoE < L ens (ψ ⋆ ). More precisely, witht = n−1 n , the loss gap is L ens (ψ ⋆ )− L MoE = 2t(p− u)(1− p + b) + t 2 (p− u) 2 + (c− b) 2 + (d− 2)(r− b) 2 > 0. Under the uniform ensemble, the correct and common in- correct labels receive probabilities ̄q y = p+(n−1)u n , ̄q z = b+(n−1)c n . Becausec > r, the common incorrect label is the most likely incorrect label. The uniform ensemble there- fore predicts the wrong answer wheneverb + (n− 1)c > p + (n− 1)u.By contrast, hard confidence routing predicts correctly on every region because p > 1/d implies p > b. Strengths and limitations. This theoretical perspective clarifies both the strengths and the limitations of FJ-based multi-agent deliberation. Its strength is that deliberation can implement a task-adaptive mixture of agents: when the FJ parameters track competence, the system can route influence toward agents that are locally reliable. This enables gains from specialization that a fixed ensemble cannot realize. Its limitation is that routing depends on imperfect competence signals. If confidence is miscalibrated, if behavioral con- fidence is uninformative, or if opinion alignment merely reflects correlated errors, then the induced influence weights may amplify the wrong agents. In particular, high agreement among agents can be beneficial when it reflects independent corroboration, but harmful when it reflects shared bias. Sim- ilarly, high confidence can be beneficial when calibrated, but harmful when associated with overconfident mistakes. 3. Experiments The main goal of our experiments is to demonstrate the utility of the Friedkin-Johnsen modeling approach and mix- ture of expert interpretation for studying the emergence of influence in MASs. Our experimental evaluation considers the MMLU- Pro (Wang et al., 2024), BBQ (Parrish et al., 2022) and CommonsenseQA (CSQA) (Talmor et al., 2019) dataset. We adopt the setup and subset of100questions of CSQA from (Abedini et al., 2026) and sample300questions from MMLU-Pro and BBQ each (balanced for the categories). We conduct our experiments with three representative lan- guage models, GPT-5.4 Mini, Qwen2.5-14B-Instruct, and Qwen2.5-72B-Instruct-GPTQ-Int8, enabling comparison across different model scales. We run our experiments over 3 different seeds. We use the same base system prompt for each agent and ex- tend it to create diversity by prompting agents to assume dif- ferent roles (Kong et al., 2024), like doctor, mathematician, or careless student, or answering in a specific communica- tion style, e.g. concise, balanced, or emotional. Prompting agents to act as experts in supercategories (Ko et al., 2026) of the MMLU-Pro dataset and giving them access to spe- cific example questions can also introduce agent diversity. The scenario neutral refers to no diversifying prompts being added in the system prompt. For the user prompts, com- munication between agents, and generating prompts for the next deliberation round, we use the setup of (Abedini et al., 2026). The prompts can be found in Appendix E. We use5 agents, communicating in a complete graph structure over 5 rounds. Competence. We measure agent competence by the mag- nitude of their belief in the correct answer. This measure is unknown to the system but we can evaluate it based on known labels. To understand on what basis a MAS approxi- mates agent competence, we define several other variables. Confidence. The confidence of an agentjis defined in Eq. (5). The relative-confidence of agentjis defined as R j (S) = C j (S) C (n−1) (S) whereC (n−1) (S)is the second most confident agent in the system. Influence. The influence of agentj, normalized by the maximum agent influence, is defined asI j (S) = π j max i π i , where π j is defined in Prop. 2.1. Peer Influence. LetB = A + (I − A)Wand define ̃ B asBwith zero diagonal entries, i.e., ̃ B ij = B ij 1[i ̸= j]. Here,AandWrefer to FJ matrices, as defined in Section 2. The peer influence of agentjisP j = π peer j max k π peer k , where π peer j = P n i=1 ̃ B ij and π peer j is the j-th column sum of ̃ B. Disagreement. The disagreement of a systemSis the mean distance of each agent’s initial belief from the average system opinion: Di(S) = 1 n P n j=1 s j − 1 n P n i=1 s i 2 . Alignment. The alignment of an agent with the MAS is given by the cosine similarity of its initial belief to the mean opinion: Al j (S) = s j · 1 N P N i=1 s i ∥s j ∥ 2 ·∥ 1 N P N i=1 s i ∥ 2 7 Multi-Agent Systems are Mixtures of Experts humanities others social science stem student teacher mathematician doctor rookie Agent 0.25 ±0.05 0.29 ±0.15 0.29 ±0.01 0.30 ±0.06 0.55 ±0.12 0.53 ±0.18 0.56 ±0.07 0.55 ±0.08 0.52 ±0.07 0.50 ±0.23 0.52 ±0.07 0.56 ±0.06 0.49 ±0.06 0.51 ±0.07 0.59 ±0.14 0.57 ±0.12 0.31 ±0.14 0.32 ±0.06 0.37 ±0.07 0.33 ±0.06 0.30 0.35 0.40 0.45 0.50 0.55 Figure 4. Influence (mean±95% confidence interval) for different roles, differences in influence suggest perceived confidence plays a role in FJ dynamics. Alignment score. The alignment score is a binary variable, describing whether an agents initial answer matches the groups initial answer: As j (S) = ( 1, if arg max(s j ) = arg max( 1 N P N i=1 s i ) 0, else Alignment count. The alignment count of an agent gives the number of other agents, that share the same initial an- swer:Ac j (S) = P N i=1,i̸=j δ arg max(s j ),arg max(s i ) , whereδ is the Kronecker delta. We empirically study the FJ dynamics in agentic systems and investigate how the routing weights are dependent on the different variables. If not indicated otherwise, the plots show results from the MMLU-pro dataset with role prompts with GPT-5.4 Mini. Plots for the other configurations can be found in Appendix C and in the supplementary material. Agentic system often outperforms ensemble. We compare the performance of the agentic system to the baseline of tak- ing the answer with maximum belief (averaging over the initial agent beliefs), and using a task independent ensemble by fitting constant FJ parameters over all samples, optimiz- ing for predictive performance. Table 3 in the appendix shows that agentic systems can perform better than both baseline and FJ ensemble, underlining the theory that the routing is input dependent, suggesting a MoE model. MASs implement MoEs. Table 1 shows that the FJ model fits the dynamics of the agentic collaboration well for all datasets, models and prompts styles. We can therefore lever- age these fitted parameters to investigate the variability in agent weight between tasks. High variability of the FJ pa- rameters over samples, as seen in Fig. 1a, suggests that the parameters are input dependent. Consequently, the fi- nal weighting of the agents’ initial beliefs,πin Proposi- confidence student teacher mathematician doctor Rel. Confidence Alignment Alignm. Score Alignm. Count 2 0 2 4 6 Coefficient Acc (train) = 0.747 ± 0.010 Acc (CV) = 0.786 ± 0.020 Figure 5. Logistic regression coefficients classifying the most influential agent. Confidence and competence are the strongest positive predictors. Role coefficients suggest perceived confidence plays a role in FJ dynamics. tion 2.1, are also input dependent, empirically confirming Hypothesis 2.2. Tendency towards consensus in MASs. Despite the diver- sity in initial opinions (see Fig. 1), agents consistently reach consensus in the final round. This is unexpected under FJ dynamics and only happens under specific conditions, e.g. when little or no stubbornness is present. Furthermore, we observe that influence is concentrated on a small subset of agent, as can be seen in Fig. 19 and in the individual sam- ples in Fig. 22 (see appendix). Influence is strongly related to stubbornness (γ), as can be seen in Fig. 17b: Within an agentic system, stubborn agents become the most influential. The fact that a small subset of agents dominate the final consensus and that influence is input dependent, shows that agentic systems make strong routing decisions, and moti- vates the need for understanding this routing behavior. Competence is associated with influence. Fig. 18 in the ap- pendix shows that there is a tendency for competent agents to get more influence, which is an indicator of effective routing. Particularly, confidence relative to other agents is a stronger predictor than absolute confidence, highlighting the social aspect of influence emergence. Interestingly, the role of an agent can change its influence, suggesting that perceived confidence also plays a role in the FJ dynamics (see Figs. 4 and. 5). Confidence leads to influence. To understand the routing behavior, we investigate the relation between metrics of the initial beliefs and competence with influence of agents by training random forests to regress influence to classify (Fig. 16, appendix) if an agent becomes the most influential one. The models achieve a testR 2 of0.7and accuracy of 8 Multi-Agent Systems are Mixtures of Experts Table 1. Model fit measured in KL divergence and MSE aggregated across datasets, prompt types, seeds and models. MetricMean± 95% CI KL Divergence0.0470± 0.0034 MSE0.00198± 0.00026 0.9for regression and classification, respectively, which suggests that influence can largely be explained by the con- sidered variables. We also use logistic regression to gain insights into the relative impact of different variables. Fig. 5 shows that confidence both in absolute terms and relative to the second most confident agent is predictive of influence. Furthermore, competence, prompt style and alignment have an effect. Other datasets show similar behavior (see supple- mentary material). The positive relation of confidence to influence as well as peer-influence can also be observed in Figs. 9-11 in the appendix. Confident agents are more stubborn. Naturally, alignment with the majority opinion plays in a role in the rate to which an agent sticks to its opinion. Agents that align with the majority stick to their opinion more frequently. However, both for aligned and misaligned agents, confident agents tend to stick more to their initial belief, as can be seen in Fig. 12 in the appendix. As noted before, there is a strong correlation between influ- ence and stubbornness (see Fig. 17b), suggesting that the routing behavior emerging in MASs is shaped also by an agents’ resistance to changing their opinion. While we see that confident agents tend to be more stubborn, this confi- dence is not necessarily grounded in competence, making it crucial to understand when to trust such social signals. 4. Discussion The primary objective of our work is to understand accord- ing to which criteria influence emerges through multi-agent deliberation by casting MASs as mixture of experts (MoE) with adaptive routing. Our analysis of the router revealed not only confidence but also relative confidence as most dominant factor to explain which agent dominates the opin- ion formation process. This suggests that the overall dis- tribution of confidence among agents is highly predictive of the deliberation outcome. To gain deeper insights into influence emergence, a natural next step in future would be to build more accurate router models considering all, not only the most confident agent. Graph Neural Networks (GNNs) (Wu et al., 2020; Corso et al., 2024), in particu- lar, GNNs with attention mechanism (Brody et al., 2022; Mustafa et al., 2023; Mustafa & Burkholz, 2024b;a) could be good router candidates, as they are permutation invariant with respect to specific agent identities. They also have the potential to generalize across different communication graph topologies and even provide an opportunity to design more sophisticated and expressive multi-agent routing in future (Bause et al., 2024; 2025; Kummer et al., 2025; Roth et al., 2024). However, recent insights (Rubio-Madrigal & Burkholz, 2026a;b) imply that more complex communica- tion patterns might not even be required to obtain high per- formance, especially, if agents are sufficiently complex and if the communication graph can be optimized (Jamadandi et al., 2024; Rubio-Madrigal et al., 2025). How much can be gained from optimal routing, compared to the routing that emerges from deliberation, remains an open question, that could lead to build more accurate, efficient, and trustworthy agentic systems. 5. Conclusion We have provided a framework to study the mechanisms of collaboration in multi-agent LLM systems (MASs) and gained new insights into the factors governing the emer- gence of influence, including the confidence of agents, their communication behavior and opinion alignment. Building on the observation that the Friedkin-Johnsen model cap- tures MAS deliberation dynamics, we have cast MASs as mixtures of experts (MoE) with adaptive routing. This has allowed us to derive conditions when MASs can outperform simpler ensembles and single agent models. Accordingly, their strengths arise from adaptive routing and local spe- cialization of diverse and well calibrated agents, while its limitations arise from miscalibrated agent confidence, mis- leading consensus, and routing errors. We see multiple important directions for future work: To understand fac- tors contributing to the emergence of influence, we could learn more advanced routing mechanisms that take relevant features into account. Graph neural networks would be a natural approach to identify general routing mechanisms that apply to different agent communication topologies. Ex- tending the framework to other tasks, such as open-ended generation and tool use, could transfer our insights to differ- ent agentic use cases. This could also aid mitigating overly confident agents and improving communication to enhance overall performance. Acknowledgments.We are grateful for funding from the European Research Council (ERC) under the Horizon Eu- rope Framework Programme (HORIZON) for proposal num- ber 101116395 SPARSE-ML. References Abedini, S., Mavali, S., Sch ̈ onherr, L., Pawelczyk, M., and Burkholz, R. Don’t trust stubborn neighbors: A security framework for agentic networks. CoRR, abs/2603.15809, 2026. doi: 10.48550/ARXIV.2603.15809. 9 Multi-Agent Systems are Mixtures of Experts Bause, F., Moustafa, S., Langguth, J., Gansterer, W. N., and Kriege, N. M. On the two sides of redundancy in graph neural networks. In Machine Learning and Knowledge Discovery in Databases. Research Track, p. 371–388, Cham, 2024. Springer Nature Switzerland. ISBN 978-3- 031-70365-2. Bause, F., Jogl, F., Indri, P., Drucks, T., Penz, D., Kriege, N. M., G ̈ artner, T., Welke, P., and Thiessen, M. Maxi- mally expressive GNNs for outerplanar graphs. Transac- tions on Machine Learning Research, 2025. ISSN 2835- 8856. URLhttps://openreview.net/forum? id=XxbQAsxrRC. Brody, S., Alon, U., and Yahav, E. How attentive are graph attention networks?In International Confer- ence on Learning Representations, 2022. URLhttps: //openreview.net/forum?id=F72ximsx7C1. Burkholz, R. and Quackenbush, J. Cascade size distribu- tions: Why they matter and how to compute them ef- ficiently. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, p. 6840–6849, 2021. Burkholz, R. and Schweitzer, F. Framework for cascade size calculations on random networks. Physical Review E, 97(4):042312, 2018. Burkholz, R., Garas, A., and Schweitzer, F. How damage diversification can reduce systemic risk. Physical Review E, 93(4):042313, 2016. Burkholz, R., Herrmann, H. J., and Schweitzer, F. Explicit size distributions of failure cascades redefine systemic risk on finite networks. Scientific reports, 8(1):6878, 2018a. Burkholz, R., Herrmann, H. J., and Schweitzer, F. Explicit size distributions of failure cascades redefine systemic risk on finite networks. Scientific reports, 8(1):6878, 2018b. Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J. E., and Stoica, I. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025. Chen, S., Liu, Y., Han, W., Zhang, W., and Liu, T. A survey on llm-based multi-agent system: Recent ad- vances and new frontiers in application. arXiv preprint arXiv:2412.17481, 2024. Chen, Z., Deng, Y., Wu, Y., Gu, Q., and Li, Y. Towards understanding mixture-of-experts layer in deep learning. In Advances in Neural Information Processing Systems, volume 35, 2022. Corso, G., Stark, H., Jegelka, S., Jaakkola, T., and Barzilay, R. Graph neural networks. Nature Reviews Methods Primers, 4(1):17, 2024. DeGroot, M. H.Reaching a consensus.Journal of the American Statistical Association, 69(345):118–121, 1974. ISSN 01621459, 1537274X. URLhttp://w. jstor.org/stable/2285509. Dietterich, T. G. Ensemble methods in machine learning. In Multiple Classifier Systems, p. 1–15. Springer, 2000. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 11409–11431. PMLR, 2024. Flache, A., M ̈ as, M., Feliciani, T., Chattoe-Brown, E., Def- fuant, G., Huet, S., and Lorenz, J. Models of social in- fluence: Towards the next frontiers. Journal of Artificial Societies and Social Simulation, 20(4), 2017. Friedkin, N. E. and Johnsen, E. C. Social influence and opinions. Journal of Mathematical Sociology, 15(3–4): 193–206, 1990. Friedkin, N. E. and Johnsen, E. C. Social Influence Network Theory: A Sociological Examination of Small Group Dy- namics. Cambridge University Press, 2011. Gotovos, A., Burkholz, R., Quackenbush, J., and Jegelka, S. Scaling up continuous-time markov chains helps re- solve underspecification. Advances in Neural Information Processing Systems, 34:14580–14592, 2021. He, Y., Zhang, D., Kovalchuk, S., Li, P., and Sedakov, A.Opinion dynamics and mutual influence with llm agents through dialog simulation. arXiv preprint arXiv:2602.12583, 2026. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., and Schmidhuber, J. Metagpt: Meta programming for a multi-agent collaborative frame- work. In International Conference on Learning Repre- sentations, 2024. Hu, Z., Rostami, M., and Thomason, J. Expert personas improve llm alignment but damage accuracy: Bootstrap- ping intent-based persona routing with prism, 2026. URL https://arxiv.org/abs/2603.18507. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991. 10 Multi-Agent Systems are Mixtures of Experts Jamadandi, A., Rubio-Madrigal, C., and Burkholz, R. Spectral graph pruning against over-squashing and over- smoothing. In Thirty-eighth Conference on Neural In- formation Processing Systems, 2024. URLhttps: //openreview.net/forum?id=EMkrwJY2de. Jiang, W. and Tanner, M. A. Hierarchical mixtures-of- experts for generalized linear models. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, 1999. Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2): 181–214, 1994. Ko, C., Shin, J., Song, H., Lee, H., Hwang, E. J., and Park, J. C. Social dynamics as critical vulnerabilities that undermine objective decision-making in LLM collectives, 2026. Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Sun, R., Zhou, X., Wang, E., and Dong, X. Better zero-shot reasoning with role-play prompting. In Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, p. 4099–4113. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.NAACL-LONG.228. Kummer, L., Moustafa, S., Ehrlich, A., Bause, F., Suess, N., Gansterer, W. N., and Kriege, N. M. Weisfeiler and leman go gambling: Why expressive lottery tickets win. In Forty-second International Conference on Machine Learning, 2025. URLhttps://openreview.net/ forum?id=7EtP9u7JNw. Lee, J. H., Lauscher, A., and Albrecht, S. V. Towards ethical multi-agent systems of large language models: A mechanistic interpretability perspective. arXiv preprint arXiv:2512.04691, 2025. Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. CAMEL: Communicative agents for ”mind” exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/ forum?id=3IyL2XWDkG. Mustafa, N. and Burkholz, R. Dynamic rescaling for train- ing GNNs. In Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URLhttps: //openreview.net/forum?id=IfZwSRpqHl. Mustafa, N. and Burkholz, R. GATE: How to keep out intrusive neighbors. In Forty-first International Con- ference on Machine Learning, 2024b. URLhttps: //openreview.net/forum?id=Sjv5RcqfuH. Mustafa, N., Bojchevski, A., and Burkholz, R. Are GATs out of balance? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps: //openreview.net/forum?id=qY7UqLoora. Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. BBQ: A hand-built bias benchmark for question answering. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Find- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, p. 2086– 2105. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.FINDINGS-ACL.165. Parsegov, S. E., Proskurnikov, A. V., Tempo, R., and Fried- kin, N. E. Novel multidimensional models of opinion dynamics in social networks. IEEE Transactions on Au- tomatic Control, 62(5):2270–2285, May 2017. ISSN 1558-2523. doi: 10.1109/tac.2016.2613905. URLhttp: //dx.doi.org/10.1109/TAC.2016.2613905. Proskurnikova, A. V. and Tempo, R. A tutorial on modeling and analysis of dynamic social networks. part i. Annual Reviews in Control, 43:65–79, 2017. Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. Chatdev: Commu- nicative agents for software development. In Proceed- ings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), p. 15174–15186, 2024. Reddy, A. G., Rubio-Madrigal, C., Burkholz, R., and Muan- det, K. When shift happens - confounding is to blame. In The Fourteenth International Conference on Learning Representations, 2026a. Reddy, A. G., Verma, R., Rubio-Madrigal, C., Muandet, K., and Burkholz, R. Boosting for predictive sufficiency. In The Fourteenth International Conference on Learning Representations, 2026b. Rosenfeld, A. and Richardson, A. Explainability in human– agent systems. Autonomous agents and multi-agent sys- tems, 33(6):673–705, 2019. Roth, A., Bause, F., Kriege, N. M., and Liebig, T. Pre- venting representational rank collapse in MPNNs by splitting the computational graph. In The Third Learn- ing on Graphs Conference, 2024. URLhttps:// openreview.net/forum?id=DOh3hW1OZu. Rubio-Madrigal, C. and Burkholz, R. Fixed aggregation features can rival GNNs. In Forty-third International Conference on Machine Learning, 2026a. URLhttps: //openreview.net/forum?id=gSZhNPp103. 11 Multi-Agent Systems are Mixtures of Experts Rubio-Madrigal, C. and Burkholz, R. Linear GCNs need better bias, not more expressive power. In Workshop on Graph Foundation Models: A New Era for Graph Ma- chine Learning (GFM) @ ICML, 2026b. URLhttps: //openreview.net/forum?id=cUJMrMzYVG. Rubio-Madrigal, C., Jamadandi, A., and Burkholz, R. GNNs getting comfy: Community and feature similarity guided rewiring. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps:// openreview.net/forum?id=g6v09VxgFw. Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., and Pretorius, A. Should we be going MAD? a look at multi- agent debate strategies for LLMs. In Proceedings of the 41st International Conference on Machine Learning, vol- ume 235 of Proceedings of Machine Learning Research. PMLR, 2024. Sun, D., Nittala, R., and Burkholz, R. Robustness of mix- tures of experts to feature noise. In Forty-third Inter- national Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2601.14792. Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), p. 4149–4158. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1421. Tian, Y. and Wang, L. Opinion dynamics in social net- works with stubborn agents: An issue-based perspective. Automatica, 96:213–223, 2018. Tran, D. and Kiela, D. Single-agent llms outperform multi- agent systems on multi-hop reasoning under equal think- ing token budgets. arXiv preprint arXiv:2604.02460, 2026. Wang, C., Liu, Z., Yang, D., and Chen, X. Decoding echo chambers: LLM-powered simulations revealing polariza- tion in social networks. In Rambow, O., Wanner, L., Apid- ianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schock- aert, S. (eds.), Proceedings of the 31st International Con- ference on Computational Linguistics, p. 3913–3923, Abu Dhabi, UAE, January 2025. Association for Compu- tational Linguistics. URLhttps://aclanthology. org/2025.coling-main.264/. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, 2023. URLhttps: //openreview.net/forum?id=1PL1NIMMrw. Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps: //openreview.net/forum?id=y10DM6R2r3. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., and Oh, A. (eds.), Advances in Neu- ral Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. Wooldridge, M. An introduction to multiagent systems. Wiley, 2018. Wu, J., Abebe, R., and Mendler-D ̈ unner, C. Opinion dynamics in learning systems, 2026. URLhttps: //arxiv.org/abs/2603.12137. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. Autogen: En- abling next-gen LLM applications via multi-agent con- versations. In First Conference on Language Modeling, 2024. URLhttps://openreview.net/forum? id=BAakY1hNKS. Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu, P. S. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32 (1):4–24, 2020. Zhang, H. et al. Stop overvaluing multi-agent debate – we must rethink evaluation and embrace model heterogeneity. arXiv preprint arXiv:2502.08788, 2025. 12 Multi-Agent Systems are Mixtures of Experts A. Theory: Proofs of theorems and discussion Equilibrium beliefs are convex ensembles.The Friedkin-Johnsen dynamics induce a graph-dependent ensemble over the initial agent beliefs. Proposition A.1 (Graph-induced convex ensemble). Assume thatρ(H) < 1, whereρ(·)denotes the spectral radius. Then the FJ dynamics converge to the unique equilibriumB ⋆ = (I − H) −1 ΓS.M := (I − H) −1 Γis nonnegative and row-stochastic. Consequently, each equilibrium belief is a convex combination of the innate beliefs: b ⋆ i = P n j=1 M ij s j . If the agentic network beliefb ⋆ out = P n i=1 η i b ⋆ i = P n j=1 π j s j withπ ⊤ = η ⊤ Mis formed by a linear aggregation (typically η i = 1/N) of all agents’ beliefs, the agentic network is equivalent to a fixed ensemble with weightsπ j that depend on the FJ parameters. Sketch. Unrolling the FJ dynamics gives B(t) = t−1 X τ =0 H τ ΓS + H t B(0). If ρ(H) < 1, then H t → 0 and the Neumann series converges: ∞ X τ =0 H τ = (I − H) −1 . Nonnegativity follows becauseHandΓare nonnegative. Row-stochasticity follows from conservation of mass on the simplex: if all innate beliefs are equal to a constant simplex vectors, then all beliefs remain equal tosunder the dynamics, implying M1 = 1. Thus, deliberation over a fixed graph does not produce an arbitrary new predictor. It produces a graph-induced convex ensemble of the agents’ initial beliefs. For a designated readout agent r, the final prediction is b ⋆ r = n X j=1 m rj s j .(6) More generally, if the system output is a linear readoutη ∈ ∆ n over the final agent beliefs (typically an average with η i = 1/N ), then b ⋆ out = n X i=1 η i b ⋆ i = n X j=1 π j s j , π ⊤ = η ⊤ M. Hence, a fixed agentic deliberation graph is equivalent to a fixed convex ensemble over agent priors, with weights determined by the network topology and the agents’ stubbornness, memory, and susceptibility parameters. Static deliberation versus mixture-of-experts deliberation. The preceding result shows that a fixed Friedkin–Johnsen network implements a static ensemble: f FJ (x) = n X j=1 π j s j (x),(7) where s j (x) is the prediction of agent j on input x, and π j is independent of x. If, however, the graph, influence weights, or deliberation parameters depend on the input, then the equilibrium weights also become input-dependent. Writing W = W (x),Γ = Γ(x), A = A(x), we obtain M (x) = I − H(x) −1 Γ(x), 13 Multi-Agent Systems are Mixtures of Experts and therefore f FJ−MoE (x) = n X j=1 π j (x)s j (x).(8) This is a mixture-of-experts architecture: the agents are experts, while the graph-dependent deliberation mechanism acts as a router or gating function. The static ensemble in(7)can only assign global importance to agents. By contrast, the mixture form in (8) can assign local, input-dependent responsibility to agents. LemmaA.2(Localambiguitydecomposition).Foranybelief-dependentweightsa(S) ∈∆ n , E P n j=1 a j (S)s j (X)− e Y 2 2 | S = P n j=1 a j (S)r j (S)− D a(S) (S). Proof. For any fixed S and any label y, X j a j ∥s j − e y ∥ 2 2 = X j a j s j − e y 2 2 + X j a j s j − X k a k s k 2 2 . Taking conditional expectation over Y | S gives the claim. Proposition A.3 (MoE advantage under mutually exclusive competence). Assumen≥ 2,ρ j > 0for allj, andp > u. Then L MoE < inf ψ∈∆ N L ens (ψ). Proof. For a symmetric prediction that assigns probabilityqto the correct class and distributes the remaining mass uniformly over the d− 1 incorrect classes, the Brier loss is B d (q) = (q− 1) 2 + (d− 1) 1− q d− 1 2 = d d− 1 (1− q) 2 . (9) Because q < 1, this function is strictly decreasing in q. On regionX j , a fixed ensemble with weights ψ assigns probability q j (ψ) = u + (p− u)ψ j to the correct class. Sinceψ j ≤ 1andp > u,q j (ψ)≤ p,with equality if and only ifψ j = 1. Therefore,B d (q j (ψ))≥B d (p), with equality if and only if ψ j = 1. For the fixed ensemble to match the MoE on every positive-probability region, it would needψ j = 1for everyj. This is impossible when n≥ 2 and P j ψ j = 1. Consequently, at least one region with ρ j > 0 satisfies q j (ψ) < p, and hence L ens (ψ) = n X j=1 ρ j B d (q j (ψ)) > n X j=1 ρ j B d (p) =B d (p) = L MoE . This holds for every ψ ∈ ∆ n . Optimal ensemble for balanced mutually exclusive regions. When ρ j = 1/n, define h(t) =B d (u + (p− u)t). SinceB d is strictly convex and p̸= u, h is strictly convex. Jensen’s inequality gives L ens (ψ) = 1 n n X j=1 h(ψ j )≥ h 1 n n X j=1 ψ j = h 1 n . Equality holds if and only if ψ 1 =· = ψ n = 1 n . 14 Multi-Agent Systems are Mixtures of Experts Thus, the uniform ensemble is the unique optimum and L ⋆ ens =B d u + p− u n . Subtracting L MoE =B d (p) gives the stated gap. Comparison with the best single agent. Agent j obtains lossB d (p) on its competence region andB d (u) elsewhere. Thus, L single,j = ρ j B d (p) + (1− ρ j )B d (u). BecauseB d (p) <B d (u), this expression is minimized by an agent with maximum ρ j . Therefore, Lsingle ⋆ − LMoE = ρj glob B d (p) + 1− ρj glob B d (u)−B d (p) = 1− ρj glob [B d (u)−B d (p)]. This is strictly positive whenever no single region has probability one. Routing-error threshold.A router that selects the competent agent with probability1− δand an uninformative agent with probability δ has expected loss L router (δ) = (1− δ)B d (p) + δB d (u). In the balanced case, the optimal fixed ensemble has loss L ⋆ ens =B d (q n ), q n = u + p− u n . The router is better if and only if B d (p) + δ [B d (u)−B d (p)] <B d (q n ). SinceB d (u) >B d (p), division by the positive denominator gives δ < B d (q n )−B d (p) B d (u)−B d (p) . Substituting B d (q) = d d− 1 (1− q) 2 and cancelling the common factor gives δ < (1− q n ) 2 − (1− p) 2 (1− u) 2 − (1− p) 2 . Proposition A.4. Assumen≥ 2,d≥ 3, and the conditions above. Then the optimal fixed ensemble is the uniform ensemble withψ ⋆ k = 1 n and confidence routing strictly improves over it:L MoE < L ens (ψ ⋆ ). More precisely, witht = n−1 n , the loss gap is L ens (ψ ⋆ )− L MoE = 2t(p− u)(1− p + b) + t 2 (p− u) 2 + (c− b) 2 + (d− 2)(r− b) 2 > 0. Proof.On regionX k , letv k denote the competent agent’s prediction andw k the common prediction of every non-competent agent. For any fixed ensemble ψ, its prediction on this region is q k (ψ k ) = ψ k v k + (1− ψ k )w k . Define h(t) =∥tv k + (1− t)w k − e y k ∥ 2 2 . 15 Multi-Agent Systems are Mixtures of Experts The functionhis strictly convex because it is a squared norm of an affine function andv k ̸= w k . By symmetry, the same function applies to every region. Hence, L ens (ψ) = 1 n n X k=1 h(ψ k ). Jensen’s inequality gives L ens (ψ)≥ h 1 n n X k=1 ψ k ! = h 1 n , with equality if and only if ψ k = 1/n for every k. Thus, the uniform ensemble is the unique optimal fixed ensemble. Let t = n− 1 n . The uniform ensemble prediction on regionX k is ̄q k = 1 n v k + n− 1 n w k = v k + t(w k − v k ). Its excess Brier loss over the competent agent is ∥ ̄q k − ey k ∥ 2 2 −∥v k − e y k ∥ 2 2 = 2t⟨v k − e y k ,w k − v k ⟩ + t 2 ∥w k − v k ∥ 2 2 . The inner product is ⟨v k − e y k ,w k − v k ⟩ = (p− 1)(u− p) + b [(1− u)− (1− p)] = (1− p)(p− u) + b(p− u) = (p− u)(1− p + b). Furthermore, ∥w k − v k ∥ 2 2 = (p− u) 2 + (c− b) 2 + (d− 2)(r− b) 2 . Therefore, L ens (ψ ⋆ )− L MoE = 2t(p− u)(1− p + b) + t 2 (p− u) 2 + (c− b) 2 + (d− 2)(r− b) 2 . Since t > 0 and p > u, the first term is strictly positive, while the second is nonnegative. Hence, L ens (ψ ⋆ ) > L MoE . Finally, under the uniform ensemble, ̄q y = p + (n− 1)u n , ̄q z = b + (n− 1)c n . Becausec > r, the common wrong label receives more probability than every other wrong label. Thus, the ensemble predicts the wrong label whenever b + (n− 1)c > p + (n− 1)u. The confidence-routed MoE predicts correctly because p > 1 d ⇐⇒ p > 1− p d− 1 = b. 16 Multi-Agent Systems are Mixtures of Experts B. Experimental evaluation We evaluate how well the FJ model fits in detail and present the results in Table 2. Over all datasets, we see a good fit. We compare the performance of the agentic system to the baseline of taking the answer with maximum belief (averaging over the initial agent beliefs), and using a task independent ensemble by fitting constant FJ parameters over all samples, optimizing for predictive performance. Table 3 that agentic systems can perform better than both baseline and FJ ensemble, underlining the theory that the routing is input dependent, suggesting a MoE model. In some cases, the agentic system slightly underperforms the baseline when prompted with specific persona or communication style prompts. Similar behavior was described in (Hu et al., 2026), which shows personas can potentially damage the discriminative performance. Regardless, utilizing these prompts introduces diversity in the agentic system, enabling a better study of its dynamics. In rare cases(< 1%), the FJ solver fails to converge within a reasonable time, so we skip the affected sample. In4%of the successful samples, the fitted FJ parameters result in the condition number of(I− H)≥ 10 4 , consequenltyπ j ’s are noisy and no longer sum exactly to one, so we exclude these cases from the influence regressions and plots. C. Additional Plots We present additional plots for our experiments. More results can be found in the supplementary material. We see the same variability of the FJ parameters and tendency towards consensus in the other datasets and when using Qwen2.5-14B-Instruct and Qwen2.5-72B-Instruct-GPTQ-Int8 as the underlying models (Figs. 6, 7 and 8). Figs. 9, 10, and 11 show the (peer-)influence of agents in relation to their confidence. We see that confident agents tend to be more influential. Fig. 12 shows the probability of an agent changing their answer to the majority answer when the agent is confident/not confident. We see that agents are more likely to change their answer if they are not confident. Interestingly, between the models, the Qwen2.5-14B-Instruct agents are less confident, but also less likely to change their answer to the majority answer when initially disagreeing with it. Figs. 13, 14, and 15 show the impact of different diversifying prompts on the peer influence of the agents, grouped by question (super-)category on dataset MMLU-Pro. The influence of an agent seems to be driven also by perceived confidence, and roles seem to have a relatively big effect that is consistent over different models. Fig. 16 shows coefficient for random forests regression influence and classifying the most influential agent in an MAS. We observe a good fit for regression and a near perfect fit on the classification task. This shows, the defined variables are predictive of FJ dynamics. We investigate the relation between confidence and competence, as well as between influence and stubbornness in Fig. 17. We can see that an agents influence is highly correlated with its stubbornness. The relation between actual competence and influence is less prominent (see Fig. 18). Figs. 20 and 21 show the number of cases where Eq. 3 of Theorem 2.4 is satisfied for individual samples, compared to when it does not, and whether the MAS outperforms the best single agent. Theorem 2.4 states that, if Eq.3 is satisfied, the MAS should outperform the best single agent, so large values in the upper left and lower right entries of the confusion matrix point to Theorem 2.4 holding. We can see, that in most cases, the equation is satisfied, and routing wins for GPT-5.4 Mini. For Qwen2.5-72B-Instruct-GPTQ-Int8 the MAS loses more often, but in most of these cases Eq. 3 is not satisfied. Figs. 22 and 23 show weight heatmaps for different samples of the MMLU-Pro dataset (with communication style prompts). We can see that the weight matrices are task dependent. 17 Multi-Agent Systems are Mixtures of Experts 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (a) MMLU-Pro InitialFinal round 0.0 0.2 0.4 0.6 Disagreement (b) Belief disagreement. 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (c) BBQ InitialFinal round 0.0 0.2 0.4 0.6 Disagreement (d) Belief disagreement. 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (e) CSQA InitialFinal round 0.0 0.1 0.2 0.3 0.4 0.5 Disagreement (f) Belief disagreement. Figure 6. Parameter variability over all samples no special prompts for GPT-5.4 Mini, receiving weights (W1-5) are averaged over senders (left). Tendency towards consensus for the corresponding dataset (right). 18 Multi-Agent Systems are Mixtures of Experts 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (a) MMLU-Pro InitialFinal round 0.0 0.2 0.4 0.6 Disagreement (b) Belief disagreement. 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (c) BBQ InitialFinal round 0.0 0.2 0.4 0.6 Disagreement (d) Belief disagreement. 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (e) CSQA InitialFinal round 0.0 0.1 0.2 0.3 0.4 Disagreement (f) Belief disagreement. Figure 7. Parameter variability over all samples no special prompts for Qwen2.5-14B-Instruct, receiving weights (W1-5) are averaged over senders (left). Tendency towards consensus for the corresponding dataset (right). 19 Multi-Agent Systems are Mixtures of Experts 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (a) MMLU-Pro InitialFinal round 0.0 0.2 0.4 0.6 Disagreement (b) Belief disagreement. 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (c) BBQ InitialFinal round 0.0 0.1 0.2 0.3 0.4 0.5 Disagreement (d) Belief disagreement. 12345 12345 W1W2W3W4W5 0.0 0.2 0.4 0.6 0.8 1.0 Parameter value (e) CSQA InitialFinal round 0.0 0.1 0.2 0.3 Disagreement (f) Belief disagreement. Figure 8. Parameter variability over all samples no special prompts for Qwen2.5-72B-Instruct-GPTQ-Int8, receiving weights (W1-5) are averaged over senders (left). Tendency towards consensus for the corresponding dataset (right). 20 Multi-Agent Systems are Mixtures of Experts 0.02.55.07.510.0 Influence (norm. by 2nd largest) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Confidence (norm. by 2nd largest) r per MAS: 0.61 ± 0.43 Correct Incorrect 02468 Relative peer influence 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.77 ± 0.39 Correct Incorrect (a) MMLU-Pro: Confidence vs. (Peer-)Influence 0.02.55.07.510.0 Influence (norm. by 2nd largest) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Confidence (norm. by 2nd largest) r per MAS: 0.59 ± 0.39 Correct Incorrect 051015 Relative peer influence 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Confidence (norm. by 2nd largest) r per MAS: 0.70 ± 0.37 Correct Incorrect (b) BBQ: Confidence vs. (Peer-)Influence 02468 Influence (norm. by 2nd largest) 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.63 ± 0.38 Correct Incorrect 051015 Relative peer influence 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.87 ± 0.19 Correct Incorrect (c) CSQA: Confidence vs. (Peer-)Influence Figure 9. Confidence vs. Influence (left) and confidence vs. peer influence (right) with no special prompts for GPT-5.4 Mini. 21 Multi-Agent Systems are Mixtures of Experts 0.02.55.07.510.0 Influence (norm. by 2nd largest) 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.36 ± 0.45 Correct Incorrect 051015 Relative peer influence 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Confidence (norm. by 2nd largest) r per MAS: 0.44 ± 0.49 Correct Incorrect (a) MMLU-Pro: Confidence vs. (Peer-)Influence 02468 Influence (norm. by 2nd largest) 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.51 ± 0.40 Correct Incorrect 0102030 Relative peer influence 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.60 ± 0.45 Correct Incorrect (b) BBQ: Confidence vs. (Peer-)Influence 0.02.55.07.510.0 Influence (norm. by 2nd largest) 0.0 0.5 1.0 1.5 2.0 2.5 Confidence (norm. by 2nd largest) r per MAS: 0.42 ± 0.43 Correct Incorrect 051015 Relative peer influence 0.0 0.5 1.0 1.5 2.0 2.5 Confidence (norm. by 2nd largest) r per MAS: 0.58 ± 0.41 Correct Incorrect (c) CSQA: Confidence vs. (Peer-)Influence Figure 10. Confidence vs. Influence (left) and confidence vs. peer influence (right) with no special prompts for Qwen2.5-14B-Instruct. 22 Multi-Agent Systems are Mixtures of Experts 0.02.55.07.510.0 Influence (norm. by 2nd largest) 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.41 ± 0.46 Correct Incorrect 0102030 Relative peer influence 0.0 0.5 1.0 1.5 2.0 Confidence (norm. by 2nd largest) r per MAS: 0.43 ± 0.57 Correct Incorrect (a) MMLU-Pro: Confidence vs. (Peer-)Influence 02468 Influence (norm. by 2nd largest) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Confidence (norm. by 2nd largest) r per MAS: 0.46 ± 0.50 Correct Incorrect 0102030 Relative peer influence 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Confidence (norm. by 2nd largest) r per MAS: 0.54 ± 0.56 Correct Incorrect (b) BBQ: Confidence vs. (Peer-)Influence 02468 Influence (norm. by 2nd largest) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Confidence (norm. by 2nd largest) r per MAS: 0.47 ± 0.50 Correct Incorrect 010203040 Relative peer influence 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Confidence (norm. by 2nd largest) r per MAS: 0.61 ± 0.48 Correct Incorrect (c) CSQA: Confidence vs. (Peer-)Influence Figure 11. Confidence vs. Influence (left) and confidence vs. peer influence (right) with no special prompts for Qwen2.5-72B-Instruct- GPTQ-Int8. 23 Multi-Agent Systems are Mixtures of Experts Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.20.40.60.81.0 Opinion change rate n=649 n=698 n=15 n=123 Overall: 0.11 ± 0.01 Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.20.40.60.81.0 Opinion change rate n=106 n=1089 n=12 n=278 Overall: 0.23 ± 0.02 Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.51.0 Opinion change rate n=23 n=1299 n=1 n=166 Overall: 0.13 ± 0.01 (a) MMLU-Pro Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.20.40.60.81.0 Opinion change rate n=1233 n=211 n=20 n=23 Overall: 0.03 ± 0.01 Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.20.40.60.81.0 Opinion change rate n=678 n=729 n=16 n=57 Overall: 0.05 ± 0.01 Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.000.250.500.751.00 Opinion change rate n=830 n=606 n=5 n=44 Overall: 0.03 ± 0.01 (b) BBQ Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.51.0 Opinion change rate n=134 n=350 n=0 n=14 Overall: 0.03 ± 0.01 Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.20.40.60.81.0 Opinion change rate n=9 n=459 n=0 n=28 Overall: 0.06 ± 0.04 Confident + Agrees Not confident + Agrees Confident + Disagrees Not confident + Disagrees 0.00.20.40.60.81.0 Opinion change rate n=1 n=481 n=0 n=16 Overall: 0.03 ± 0.00 (c) CSQA Figure 12. Probability of changing answer to the majority answer for cases where the agent is confident, p ( change | confident > = 0 . 75 , agree with mean = True ) , ̄ n indicates average sample size over seeds rounded to nearest integer. With no special prompts for GPT-5.4 Mini (left), Qwen2.5-14B-Instruct (middle), and Qwen2.5-72B-Instruct-GPTQ-Int8 (right). 24 Multi-Agent Systems are Mixtures of Experts humanities others social science stem student teacher mathematician doctor rookie Agent 0.25 ±0.05 0.29 ±0.15 0.29 ±0.01 0.30 ±0.06 0.55 ±0.12 0.53 ±0.18 0.56 ±0.07 0.55 ±0.08 0.52 ±0.07 0.50 ±0.23 0.52 ±0.07 0.56 ±0.06 0.49 ±0.06 0.51 ±0.07 0.59 ±0.14 0.57 ±0.12 0.31 ±0.14 0.32 ±0.06 0.37 ±0.07 0.33 ±0.06 0.30 0.35 0.40 0.45 0.50 0.55 (a) Peer influence for different roles. humanities others social science stem stem social humanities other no prompt Agent 0.48 ±0.15 0.52 ±0.13 0.56 ±0.18 0.57 ±0.01 0.40 ±0.15 0.44 ±0.08 0.40 ±0.20 0.42 ±0.06 0.38 ±0.23 0.42 ±0.07 0.37 ±0.16 0.51 ±0.04 0.37 ±0.05 0.40 ±0.01 0.48 ±0.22 0.41 ±0.11 0.42 ±0.09 0.43 ±0.12 0.42 ±0.18 0.44 ±0.05 0.375 0.400 0.425 0.450 0.475 0.500 0.525 0.550 (b) Peer influence for different experts. humanities others social science stem concise emotional elaborate balanced step-by-step Agent 0.42 ±0.09 0.48 ±0.13 0.49 ±0.10 0.37 ±0.06 0.45 ±0.18 0.38 ±0.05 0.47 ±0.11 0.45 ±0.02 0.44 ±0.12 0.51 ±0.07 0.52 ±0.14 0.56 ±0.07 0.34 ±0.10 0.41 ±0.12 0.32 ±0.13 0.42 ±0.03 0.45 ±0.18 0.48 ±0.03 0.41 ±0.22 0.50 ±0.01 0.35 0.40 0.45 0.50 0.55 (c) Peer influence for different communication styles. humanities others social science stem Agent_0 Agent_1 Agent_2 Agent_3 Agent_4 Agent 0.44 ±0.05 0.40 ±0.11 0.51 ±0.15 0.48 ±0.05 0.41 ±0.05 0.46 ±0.09 0.48 ±0.32 0.47 ±0.07 0.45 ±0.07 0.48 ±0.15 0.39 ±0.14 0.47 ±0.11 0.37 ±0.04 0.42 ±0.03 0.44 ±0.29 0.50 ±0.06 0.46 ±0.15 0.49 ±0.10 0.43 ±0.11 0.44 ±0.05 0.38 0.40 0.42 0.44 0.46 0.48 0.50 (d) Peer influence without special prompts. Figure 13. Results for GPT-5.4 Mini on the MMLU-Pro dataset. Roles seem to have an the biggest influence on perceived confidence as seen by the weights agents receive from the others. 25 Multi-Agent Systems are Mixtures of Experts humanities others social science stem student teacher mathematician doctor rookie Agent 0.37 ±0.03 0.26 ±0.10 0.27 ±0.11 0.22 ±0.04 0.58 ±0.08 0.69 ±0.08 0.67 ±0.10 0.68 ±0.03 0.55 ±0.06 0.64 ±0.11 0.63 ±0.03 0.60 ±0.16 0.62 ±0.08 0.58 ±0.04 0.60 ±0.25 0.58 ±0.11 0.24 ±0.08 0.18 ±0.13 0.21 ±0.20 0.15 ±0.03 0.2 0.3 0.4 0.5 0.6 (a) Peer influence for different roles. humanities others social science stem stem social humanities other no prompt Agent 0.55 ±0.11 0.64 ±0.09 0.64 ±0.31 0.62 ±0.07 0.44 ±0.07 0.58 ±0.09 0.54 ±0.12 0.57 ±0.12 0.44 ±0.20 0.58 ±0.18 0.45 ±0.02 0.52 ±0.03 0.48 ±0.10 0.53 ±0.12 0.49 ±0.05 0.55 ±0.04 0.47 ±0.15 0.48 ±0.17 0.45 ±0.12 0.48 ±0.05 0.45 0.50 0.55 0.60 (b) Peer influence for different experts. humanities others social science stem concise emotional elaborate balanced step-by-step Agent 0.53 ±0.07 0.56 ±0.09 0.54 ±0.02 0.51 ±0.02 0.54 ±0.13 0.50 ±0.12 0.49 ±0.10 0.49 ±0.10 0.59 ±0.14 0.62 ±0.04 0.57 ±0.04 0.62 ±0.14 0.38 ±0.06 0.48 ±0.12 0.45 ±0.19 0.52 ±0.07 0.50 ±0.05 0.61 ±0.07 0.52 ±0.12 0.57 ±0.03 0.40 0.45 0.50 0.55 0.60 (c) Peer influence for different communication styles. humanities others social science stem Agent_0 Agent_1 Agent_2 Agent_3 Agent_4 Agent 0.46 ±0.04 0.52 ±0.10 0.54 ±0.15 0.54 ±0.00 0.55 ±0.09 0.57 ±0.13 0.57 ±0.23 0.55 ±0.01 0.52 ±0.12 0.56 ±0.16 0.50 ±0.08 0.52 ±0.11 0.48 ±0.03 0.57 ±0.04 0.47 ±0.13 0.53 ±0.04 0.45 ±0.10 0.51 ±0.17 0.47 ±0.07 0.52 ±0.06 0.46 0.48 0.50 0.52 0.54 0.56 (d) Peer influence without special prompts. Figure 14. Results for Qwen2.5-14B-Instruct on the MMLU-Pro dataset. The same tendencies as for GPT-5.4 Mini can be seen. 26 Multi-Agent Systems are Mixtures of Experts humanities others social science stem student teacher mathematician doctor rookie Agent 0.45 ±0.17 0.39 ±0.07 0.45 ±0.16 0.34 ±0.03 0.64 ±0.07 0.73 ±0.06 0.74 ±0.06 0.74 ±0.07 0.62 ±0.05 0.71 ±0.18 0.75 ±0.13 0.71 ±0.05 0.66 ±0.05 0.73 ±0.02 0.72 ±0.14 0.71 ±0.03 0.32 ±0.08 0.24 ±0.09 0.35 ±0.23 0.25 ±0.10 0.3 0.4 0.5 0.6 0.7 (a) Peer influence for different roles. humanities others social science stem stem social humanities other no prompt Agent 0.59 ±0.17 0.71 ±0.06 0.65 ±0.19 0.66 ±0.05 0.50 ±0.03 0.62 ±0.09 0.63 ±0.03 0.63 ±0.07 0.49 ±0.19 0.61 ±0.10 0.56 ±0.08 0.63 ±0.07 0.57 ±0.04 0.59 ±0.06 0.58 ±0.20 0.64 ±0.04 0.61 ±0.20 0.67 ±0.06 0.58 ±0.11 0.61 ±0.07 0.50 0.55 0.60 0.65 0.70 (b) Peer influence for different experts. humanities others social science stem concise emotional elaborate balanced step-by-step Agent 0.59 ±0.18 0.70 ±0.01 0.67 ±0.12 0.62 ±0.12 0.55 ±0.08 0.59 ±0.03 0.66 ±0.11 0.60 ±0.04 0.62 ±0.13 0.69 ±0.09 0.63 ±0.04 0.70 ±0.09 0.45 ±0.03 0.51 ±0.09 0.52 ±0.07 0.53 ±0.08 0.63 ±0.12 0.67 ±0.06 0.58 ±0.20 0.68 ±0.10 0.45 0.50 0.55 0.60 0.65 0.70 (c) Peer influence for different communication styles. humanities others social science stem Agent_0 Agent_1 Agent_2 Agent_3 Agent_4 Agent 0.63 ±0.07 0.59 ±0.10 0.64 ±0.09 0.64 ±0.04 0.56 ±0.08 0.65 ±0.08 0.61 ±0.04 0.63 ±0.04 0.60 ±0.12 0.67 ±0.14 0.67 ±0.18 0.65 ±0.05 0.56 ±0.08 0.63 ±0.08 0.61 ±0.12 0.68 ±0.01 0.62 ±0.09 0.65 ±0.11 0.57 ±0.16 0.66 ±0.06 0.58 0.60 0.62 0.64 0.66 (d) Peer influence without special prompts. Figure 15. Results for Qwen2.5-72B-Instruct-GPTQ-Int8 on the MMLU-Pro dataset. The same tendencies as for GPT-5.4 Mini can be seen. confidence concise emotional elaborate balanced step-by-step Rel. Confidence Alignment Alignm. Score Alignm. Count 0.0 0.2 0.4 0.6 0.8 Gini Importance R 2 (train) = 0.814 ± 0.111 R 2 (CV) = 0.720 ± 0.028 confidence concise emotional elaborate balanced step-by-step Rel. Confidence Alignment Alignm. Score Alignm. Count 0.0 0.2 0.4 0.6 Gini Importance Acc (train) = 0.985 ± 0.050 Acc (CV) = 0.919 ± 0.013 Figure 16. Results for the MMLU-Pro dataset with GPT-5.4 Mini and communication style prompts. Coefficients for a random forest regression predicting influence (a) and a random forest classifying the most influential agent in a MAS (b), with reported CVR 2 andAcc respectively. The high CV score, especially on the classification task (b) suggests the defined variables are highly predictive of the FJ dynamics. 27 Multi-Agent Systems are Mixtures of Experts 0.000.250.500.751.00 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Belief in correct answer r per MAS: 0.63 ± 0.61 Correct Incorrect (a) Confidence. 0.00.20.40.60.81.0 Gamma (norm. by max) 0.0 0.2 0.4 0.6 0.8 1.0 Influence r per MAS: 0.96 ± 0.10 Correct Incorrect (b) Influence vs. γ. Figure 17. Results for the MMLU-Pro dataset with GPT-5.4 Mini and communication style prompts. When relating confidence and relative confidence to competence (a), we see a clear trend: confidence agents are more competent. We report Spearman’s ranked correlation per MAS. (b) An agents influence is related to its stubbornness γ. 0.000.250.500.751.00 Influence 0.0 0.2 0.4 0.6 0.8 1.0 Belief in correct answer r per MAS: 0.59 ± 0.48 Correct Incorrect 01020 Relative peer influence 0.0 0.2 0.4 0.6 0.8 1.0 Belief in correct answer r per MAS: 0.69 ± 0.53 Correct Incorrect Figure 18. Results for the MMLU-Pro dataset with GPT-5.4 Mini and communication style prompts. More competent agents gain more influence (a) and relative influence (b), we again report Spearman’s ranked correlation per MAS. sample 113sample 139sample 181sample 193sample 215sample 265sample 287sample 289 Init Final 0 1 W 0 1 gamma Indicator (intensity = conf.) Correct Incorrect Figure 19. Exemplary questions of MMLU-Pro with communication style prompts with GPT-5.4 Mini, where the MAS outperforms the optimal ensemble. Weight heatmaps with indicators whether an agent was right (blue) or wrong (orange) initially (top) and in the final round (bottom), showing the γ (stubbornness) of each agent on the side (yellow being highest). 28 Multi-Agent Systems are Mixtures of Experts Routing winsRouting loses Eq. satisfied Eq. not satisfied 17618 6237 (a) Roles Routing winsRouting loses Eq. satisfied Eq. not satisfied 18713 5141 (b) Communication styles Routing winsRouting loses Eq. satisfied Eq. not satisfied 18912 5140 (c) Experts Routing winsRouting loses Eq. satisfied Eq. not satisfied 18710 5042 (d) Neutral Figure 20. Confusion matrices showing empirical result of whether per sample version of Eq. 3 holds (Theorem 2.4) for different prompt styles of GPT on MMLU-Pro, averaged over seeds. On average, MoE routing outperforms a single agent if Eq. 3 holds. 29 Multi-Agent Systems are Mixtures of Experts Routing winsRouting loses Eq. satisfied Eq. not satisfied 12118 47107 (a) Roles Routing winsRouting loses Eq. satisfied Eq. not satisfied 14813 3297 (b) Communication styles Routing winsRouting loses Eq. satisfied Eq. not satisfied 14015 3895 (c) Experts Routing winsRouting loses Eq. satisfied Eq. not satisfied 14415 3789 (d) Neutral Figure 21. Confusion matrices showing empirical result of whether per sample version of Eq. 3 holds (Theorem 2.4) for different prompt styles of Qwen2.5-72B-Instruct-GPTQ-Int8 on MMLU-Pro, averaged over seeds. On average, MoE routing outperforms a single agent if Eq. 3 holds. 30 Multi-Agent Systems are Mixtures of Experts Table 2. Model fit (KL divergence and MSE) across datasets, prompt types, and models. DatasetPromptModelKLMSE MMLU-Pro neutral GPT-5.4 Mini0.0829± 0.01470.00247± 0.00097 Qwen2.5-14B-Instruct0.0503± 0.00130.00201± 0.00004 Qwen2.5-72B-Instruct0.0282± 0.00400.000972± 0.000176 experts GPT-5.4 Mini0.0714± 0.01040.00206± 0.00040 Qwen2.5-14B-Instruct0.0467± 0.00440.00182± 0.00032 Qwen2.5-72B-Instruct0.0268± 0.00680.000935± 0.000226 comm. styles GPT-5.4 Mini0.0740± 0.00960.00219± 0.00052 Qwen2.5-14B-Instruct0.0505± 0.01480.00200± 0.00067 Qwen2.5-72B-Instruct0.0263± 0.00770.000859± 0.000326 roles GPT-5.4 Mini0.0786± 0.02120.00227± 0.00090 Qwen2.5-14B-Instruct0.0575± 0.00610.00225± 0.00032 Qwen2.5-72B-Instruct0.0320± 0.01170.00114± 0.00049 BBQ neutral GPT-5.4 Mini0.0420± 0.00930.000373± 0.000100 Qwen2.5-14B-Instruct0.0403± 0.00430.00117± 0.00006 Qwen2.5-72B-Instruct0.0238± 0.00050.000876± 0.000135 experts GPT-5.4 Mini0.0423± 0.00750.000324± 0.000091 Qwen2.5-14B-Instruct0.0541± 0.00760.00151± 0.00053 Qwen2.5-72B-Instruct0.0239± 0.01520.000963± 0.000827 comm. styles GPT-5.4 Mini0.0459± 0.00520.000436± 0.000063 Qwen2.5-14B-Instruct0.0354± 0.00390.00105± 0.00011 Qwen2.5-72B-Instruct0.0221± 0.00160.000837± 0.000125 roles GPT-5.4 Mini0.0479± 0.00390.000422± 0.000135 Qwen2.5-14B-Instruct0.0508± 0.00170.00169± 0.00031 Qwen2.5-72B-Instruct0.0214± 0.00570.000817± 0.000305 CSQA neutral GPT-5.4 Mini0.0513± 0.00400.00247± 0.00019 Qwen2.5-14B-Instruct0.0754± 0.01460.00570± 0.00197 Qwen2.5-72B-Instruct0.0364± 0.00340.00264± 0.00021 experts GPT-5.4 Mini0.0517± 0.00150.00256± 0.00027 Qwen2.5-14B-Instruct0.0599± 0.01060.00414± 0.00110 Qwen2.5-72B-Instruct0.0331± 0.00190.00238± 0.00004 comm. styles GPT-5.4 Mini0.0516± 0.00420.00244± 0.00020 Qwen2.5-14B-Instruct0.0647± 0.00860.00443± 0.00047 Qwen2.5-72B-Instruct0.0316± 0.00260.00222± 0.00026 roles GPT-5.4 Mini0.0534± 0.00370.00272± 0.00032 Qwen2.5-14B-Instruct0.0770± 0.01210.00592± 0.00133 Qwen2.5-72B-Instruct0.0306± 0.00490.00223± 0.00063 31 Multi-Agent Systems are Mixtures of Experts Table 3. Accuracy of baseline, FJ ensemble, and agent system, across datasets, prompts, and models. DatasetPromptModelBaselineFJ Ens.MAS MMLU-Pro neutral GPT-5.4 Mini0.804± 0.0130.820± 0.013 0.826± 0.027 Qwen2.5-14B-Instruct0.569± 0.0270.577± 0.034 0.604± 0.023 Qwen2.5-72B-Instruct0.695± 0.0360.694± 0.075 0.712± 0.016 experts GPT-5.4 Mini0.816± 0.0050.819± 0.009 0.835± 0.008 Qwen2.5-14B-Instruct0.637± 0.0190.633± 0.034 0.640± 0.016 Qwen2.5-72B-Instruct0.728± 0.0430.729± 0.007 0.742± 0.037 comm. styles GPT-5.4 Mini0.797± 0.0120.809± 0.029 0.824± 0.029 Qwen2.5-14B-Instruct0.591± 0.023 0.614± 0.0290.606± 0.016 Qwen2.5-72B-Instruct0.727± 0.0190.736± 0.017 0.743± 0.015 roles GPT-5.4 Mini0.805± 0.0230.806± 0.029 0.824± 0.004 Qwen2.5-14B-Instruct0.557± 0.0140.576± 0.018 0.596± 0.009 Qwen2.5-72B-Instruct0.700± 0.0400.706± 0.045 0.720± 0.060 BBQ neutral GPT-5.4 Mini0.916± 0.014 0.916± 0.0280.916± 0.024 Qwen2.5-14B-Instruct0.866± 0.0280.867± 0.015 0.876± 0.021 Qwen2.5-72B-Instruct0.882± 0.002 0.888± 0.0160.875± 0.015 experts GPT-5.4 Mini0.932± 0.0170.932± 0.012 0.934± 0.012 Qwen2.5-14B-Instruct0.880± 0.017 0.886± 0.0090.870± 0.012 Qwen2.5-72B-Instruct0.836± 0.015 0.858± 0.0310.840± 0.006 comm. styles GPT-5.4 Mini0.906± 0.0250.903± 0.015 0.916± 0.014 Qwen2.5-14B-Instruct0.865± 0.0050.882± 0.020 0.887± 0.018 Qwen2.5-72B-Instruct0.857± 0.020 0.875± 0.0340.870± 0.025 roles GPT-5.4 Mini0.904± 0.018 0.918± 0.0180.915± 0.018 Qwen2.5-14B-Instruct0.874± 0.0340.866± 0.026 0.898± 0.028 Qwen2.5-72B-Instruct0.860± 0.009 0.887± 0.0130.882± 0.018 CSQA neutral GPT-5.4 Mini 0.726± 0.0130.722± 0.0100.722± 0.017 Qwen2.5-14B-Instruct0.775± 0.011 0.782± 0.0130.782± 0.018 Qwen2.5-72B-Instruct 0.803± 0.0120.799± 0.0280.793± 0.011 experts GPT-5.4 Mini0.723± 0.014 0.733± 0.0140.730± 0.025 Qwen2.5-14B-Instruct0.787± 0.014 0.803± 0.0140.797± 0.029 Qwen2.5-72B-Instruct0.806± 0.017 0.816± 0.0130.803± 0.016 comm. styles GPT-5.4 Mini0.730± 0.025 0.733± 0.0520.730± 0.025 Qwen2.5-14B-Instruct0.777± 0.014 0.787± 0.0380.763± 0.057 Qwen2.5-72B-Instruct0.799± 0.0470.806± 0.013 0.806± 0.036 roles GPT-5.4 Mini0.727± 0.029 0.737± 0.0140.720± 0.000 Qwen2.5-14B-Instruct0.777± 0.0140.793± 0.052 0.803± 0.014 Qwen2.5-72B-Instruct 0.809± 0.0270.806± 0.0300.799± 0.003 32 Multi-Agent Systems are Mixtures of Experts D. Computing resources required for reproducing Experiments for Qwen2.5-14b-Instruct and Qwen2.5-72B-Instruct-GPTQ-Int8 require A100 GPU’s. Experiments can be run in parallel on multiple GPU’s or on a single GPU, depending on GPU ram available per GPU. GPT-5.4-mini, requires access to OpenAI credits but no additional specialized hardware for the researcher. The time to run each experiment depends on the available hardware and level of parallelization. Given our hardware setup the time to complete one combination of model, dataset and prompt style experiment running on a single A100 GPU is: approximately 2.5 hours for GPT-5.4-mini and approximately 3.5 hours for Qwen-14b. Running Qwen-72b on 4 GPU’s takes approximately 19 hours for a single experiment combination. E. Additional information on prompts used All agents share a base system prompt (Abedini et al., 2026), that instructs them to solve the given task, exchange reasoning and their answer, as well as a distribution over their belief in the answer options. For the different modes (communication styles, roles, and experts) the corresponding prompt for the specific communication style etc. is appended to the base system prompt. Base System Prompt ###Instruction### Your task is to work collaboratively with other agents to solve the user’s question. Always keep the user’s question in mind. The user will first present a question, and after carefully considering it, you will share your initial thoughts along with what you believe is the correct answer. Then, other agents will contribute their own thoughts and answers. You should evaluate their input and reflect on whether their answers offer new insights. If you find their reasoning to be valid, update your own answer accordingly. If you believe your original answer is correct, keep it unchanged. Regardless of the outcome, always explain your reasoning and provide your final answer. STEM Expert Prompt ###Expertise### You are a multidisciplinary STEM expert with strong competence in mathematics, physics, chemistry, engineering, biology, and computer science. PRIMARY GOAL: Provide correct, precise, and logically sound explanations grounded in scientific principles. REASONING RULES: - Always prioritize correctness over simplicity - Break complex problems into structured steps when solving - Explicitly state assumptions before solving problems - Use formal reasoning (equations, mechanisms, algorithms) when appropriate - Verify consistency of results where possible STYLE: - Technical, precise, and structured - Minimal ambiguity - Prefer formal notation when relevant ###Examples### Three example questions including solutions from MMLU-Pro categories fitting STEM expert. 33 Multi-Agent Systems are Mixtures of Experts sample 0sample 1sample 2sample 3sample 4sample 5sample 6sample 7sample 8sample 9 sample 10sample 11sample 12sample 13sample 14sample 15sample 16sample 17sample 18sample 19 sample 20sample 21sample 22sample 23sample 24sample 25sample 26sample 27sample 28sample 29 sample 30sample 31sample 32sample 33sample 34sample 35sample 36sample 37sample 38sample 39 sample 40sample 41sample 42sample 43sample 44sample 45sample 46sample 47sample 48sample 49 sample 50sample 51sample 52sample 53sample 54sample 55sample 56sample 57sample 58sample 59 sample 60sample 61sample 62sample 63sample 64sample 65sample 66sample 67sample 68sample 69 sample 70sample 71sample 72sample 73sample 74sample 75sample 76sample 77sample 78sample 79 sample 80sample 81sample 82sample 83sample 84sample 85sample 86sample 87sample 88sample 89 sample 90sample 91sample 92sample 93sample 94sample 95sample 96sample 97sample 98sample 99 sample 100sample 101sample 102sample 103sample 104sample 105sample 106sample 107sample 108sample 109 sample 110sample 111sample 112sample 113sample 114sample 115sample 116sample 117sample 118sample 119 sample 120sample 121sample 122sample 123sample 124sample 125sample 126sample 127sample 128sample 129 sample 130sample 131sample 132sample 133sample 134sample 135sample 136sample 137sample 138sample 139 sample 140sample 141sample 142sample 143sample 144sample 145sample 146sample 147 Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final 0.0 0.2 0.4 0.6 0.8 1.0 W 0.0 0.2 0.4 0.6 0.8 1.0 gamma Indicator (intensity = conf.) Correct Incorrect Figure 22. Weight heatmaps for samples from the MMLU-Pro dataset with GPT-5.4 Mini and communication style prompts, with indicators whether an agent was right (blue) or wrong (orange) initially (top) and in the final round (bottom), showing theγ(stubbornness) of each agent on the side (yellow being highest). 34 Multi-Agent Systems are Mixtures of Experts sample 148sample 149sample 150sample 151sample 152sample 153sample 154sample 155sample 156sample 157 sample 158sample 159sample 160sample 161sample 162sample 163sample 164sample 165sample 166sample 167 sample 168sample 169sample 170sample 171sample 172sample 173sample 174sample 175sample 176sample 177 sample 178sample 179sample 180sample 181sample 182sample 183sample 184sample 185sample 186sample 187 sample 188sample 189sample 190sample 191sample 192sample 193sample 194sample 195sample 196sample 197 sample 198sample 199sample 200sample 201sample 202sample 203sample 204sample 205sample 206sample 207 sample 208sample 209sample 210sample 211sample 212sample 213sample 214sample 215sample 216sample 217 sample 218sample 219sample 220sample 221sample 222sample 223sample 224sample 225sample 226sample 227 sample 228sample 229sample 230sample 231sample 232sample 233sample 234sample 235sample 236sample 237 sample 238sample 239sample 240sample 241sample 242sample 243sample 244sample 245sample 246sample 247 sample 248sample 249sample 250sample 251sample 252sample 253sample 254sample 255sample 256sample 257 sample 258sample 259sample 260sample 261sample 262sample 263sample 264sample 265sample 266sample 267 sample 268sample 269sample 270sample 271sample 272sample 273sample 274sample 275sample 276sample 277 sample 278sample 279sample 280sample 281sample 282sample 283sample 284sample 285sample 286sample 287 sample 288sample 289sample 290sample 291sample 292sample 293sample 294sample 295sample 296 Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final Init Final 0.0 0.2 0.4 0.6 0.8 1.0 W 0.0 0.2 0.4 0.6 0.8 1.0 gamma Indicator (intensity = conf.) Correct Incorrect Figure 23. Weight heatmaps for samples from the MMLU-Pro dataset with GPT-5.4 Mini and communication style prompts, with indicators whether an agent was right (blue) or wrong (orange) initially (top) and in the final round (bottom), showing theγ(stubbornness) of each agent on the side (yellow being highest). 35 Multi-Agent Systems are Mixtures of Experts Social Science Expert Prompt ###Expertise### You are a social science expert specializing in economics and psychology. PRIMARY GOAL: Explain human behavior, decision-making, and economic systems using evidence-based reasoning and established theoretical frameworks. REASONING RULES: - Distinguish clearly between empirical findings, theories, and assumptions - When relevant, reference causal mechanisms (not just correlations) - Consider multiple competing explanations for observed behavior - Acknowledge uncertainty where evidence is mixed STYLE: - Analytical but accessible - Balanced and interpretive - Avoid overconfidence in conclusions ###Examples### Three example questions including solutions from MMLU-Pro categories fitting social science expert. Humanities Expert Prompt ###Expertise### You are a humanities expert specializing in law, philosophy, and history. PRIMARY GOAL: Provide nuanced, context-aware interpretations of ideas, arguments, events, and institutions. REASONING RULES: - Emphasize interpretation, context, and perspective - When discussing arguments, present multiple viewpoints fairly - In historical analysis, distinguish facts from interpretation - In philosophy, clearly separate premises, assumptions, and conclusions STYLE: - Reflective, precise, and interpretive - Balanced and intellectually rigorous - Avoid oversimplification of complex ideas ###Examples### Three example questions including solutions from MMLU-Pro categories fitting humanities expert. 36 Multi-Agent Systems are Mixtures of Experts Other Expert Prompt ###Expertise### You are an applied knowledge expert specializing in health, business, and general real-world problem solving. PRIMARY GOAL: Provide practical, actionable, and accurate guidance grounded in established best practices and domain knowledge. REASONING RULES: - Prioritize real-world applicability and usefulness - When relevant, include trade-offs, risks, or constraints - Use structured reasoning for decisions or recommendations - Distinguish between general advice and context-dependent advice STYLE: - Clear, practical, and solution-oriented - Moderately structured - Avoid unnecessary abstraction ###Examples### Three example questions including solutions from MMLU-Pro categories health, business and other. Role Prompts ”teacher”: ###Role### You are an excellent teacher and always teach your students problems correctly. ”mathematician”: ###Role### You are an excellent mathematician who can always explain math problems in an easy-to-understand manner. ”doctor”: ###Role### You are a respectable doctor. You have profound medical knowledge and have saved many lives. ”careless student”: ###Role### Please act as a careless student. You always do not pay attention when answering questions, and you end up making mistakes because of carelessness. ”rookie”: ###Role### Please act as a rookie. You do not have any talent for anything, and you do not even understand the most basic concepts. So you always make mistakes when answering questions. 37 Multi-Agent Systems are Mixtures of Experts Communication Style Prompts ”concise”: ###Communication Style### You are an agent that produces concise answers. RULES: - Use the minimum number of words needed to be correct - Remove all unnecessary explanation or context - Prefer direct statements over elaboration - 1-3 sentences maximum - No bullet points unless absolutely necessary - No background explanation unless explicitly requested ”elaborate”: ###Communication Style### You are an agent that provides detailed and comprehensive explanations. RULES: - Fully explain the topic with necessary depth - Include reasoning, context, and relevant background - Expand on key ideas rather than summarizing them - Multi-paragraph response - Structured explanation where helpful - Include examples or clarifications when useful ”step bystep”: ###Communication Style### You are an agent that explains reasoning step by step. RULES: - Break down reasoning into sequential steps - Make logical progression explicit - Do not skip intermediate steps - Numbered steps preferred - Clear progression from premise to conclusion - Final answer clearly separated at the end ”balanced”: ###Communication Style### You are an agent that provides balanced and objective analysis. RULES: - Present multiple perspectives when relevant - Avoid strong bias or persuasive framing - Highlight trade-offs and uncertainties - Structured comparison when needed - Neutral tone - Concluding summary with balanced judgment ”emotional”: ###Communication Style### You are an agent that communicates in an emotionally engaging way. RULES: - Use vivid and expressive language - Emphasize human impact and lived experience - Make the explanation feel relatable and engaging - Narrative or paragraph form - Emotionally rich language allowed - Avoid overly technical structure 38