Paper deep dive
Discovering Efficient and Explainable Communication Topologies for LLM-based Multi-Agent Systems via Causal Inference
Junzhi Li, Peng He, Qirui Ji, Wei Wang, Lixiang Liu, Chuxiong Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/14/2026, 5:32:40 AM
Summary
The paper introduces E2-Explainer, a model-agnostic framework for generating interpretable explanations of communication topologies in LLM-based multi-agent systems (MAS). It formulates topology explanation as a causal attribution problem using a Granger-style objective to identify compact, task-preserving subgraphs. By measuring the impact of masking communication edges on task outcomes and response stability, E2-Explainer identifies critical subgraphs that can prune redundant edges, reducing communication costs while maintaining performance. An amortized explainer is trained to predict these subgraphs efficiently at deployment.
Entities (7)
Relation Signals (6)
E2-Explainer → trains → Amortized Explainer
confidence 94% · The resulting budgeted subgraphs are then distilled into an amortized explainer
E2-Explainer → uses → Granger-style Objective
confidence 92% · We obtain this evidence with a Granger-style objective that measures how masking each communication channel changes the task outcome
E2-Explainer → reduces → Communication costs
confidence 91% · These subgraphs can also be executed directly to prune redundant communication edges, substantially reducing communication costs
E2-Explainer → identifies → Communication Topology
confidence 90% · E2-Explainer identifies critical communication subgraphs that preserve successful collaboration.
Amortized Explainer → enables → Efficient Post-hoc Explanation
confidence 89% · enabling efficient post-hoc explanation without repeated edge-level evaluations at deployment.
Communication Topology → contains → Redundant Edges
confidence 87% · we observe that the communication topologies generated by existing methods still contain substantial redundancy.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The performance of large language model (LLM)-based multi-agent systems (MAS) largely depends on effective communication topologies. Existing topology generation methods, however, typically learn communication topologies through black-box optimization driven solely by task-level rewards. While effective, such optimization provides little insight into why particular communication edges are selected, making it difficult to identify the critical communication subgraphs responsible for successful collaboration. To address this limitation, we propose E2-Explainer, a model-agnostic framework for providing interpretable explanations of communication topologies produced by arbitrary topology generators. Specifically, we formulate topology explanation as a causal attribution problem that identifies compact communication subgraphs supported by edge-level evidence of task preservation. We obtain this evidence with a Granger-style objective that measures how masking each communication channel changes the task outcome and the stability of the final response. The resulting budgeted subgraphs are then distilled into an amortized explainer, enabling efficient post-hoc explanation without repeated edge-level evaluations at deployment. Extensive experiments on multiple reasoning and coding benchmarks demonstrate that E2-Explainer identifies critical communication subgraphs that preserve successful collaboration. These subgraphs can also be executed directly to prune redundant communication edges, substantially reducing communication costs while maintaining competitive task performance.
Tags
Links
- Source: https://arxiv.org/abs/2608.12921v1
- Canonical: https://arxiv.org/abs/2608.12921v1
Trouble viewing inline? Open PDF directly →
Full Text
82,839 characters extracted from source content.
Expand or collapse full text
Discovering Efficient and Explainable Communication Topologies for LLM-based Multi-Agent Systems via Causal Inference Junzhi Li Peng He Qirui Ji Wei Wang Lixiang Liu Chuxiong Sun Abstract The performance of large language model (LLM)-based multi-agent systems (MAS) largely depends on effective communication topologies. Existing topology generation methods, however, typically learn communication topologies through black-box optimization driven solely by task-level rewards. While effective, such optimization provides little insight into why particular communication edges are selected, making it difficult to identify the critical communication subgraphs responsible for successful collaboration. To address this limitation, we propose E2-Explainer, a model-agnostic framework for providing interpretable explanations of communication topologies produced by arbitrary topology generators. Specifically, we formulate topology explanation as a causal attribution problem that identifies compact communication subgraphs supported by edge-level evidence of task preservation. We obtain this evidence with a Granger-style objective that measures how masking each communication channel changes the task outcome and the stability of the final response. The resulting budgeted subgraphs are then distilled into an amortized explainer, enabling efficient post-hoc explanation without repeated edge-level evaluations at deployment. Extensive experiments on multiple reasoning and coding benchmarks demonstrate that E2-Explainer identifies critical communication subgraphs that preserve successful collaboration. These subgraphs can also be executed directly to prune redundant communication edges, substantially reducing communication costs while maintaining competitive task performance. Introduction Large language model (LLM)-based agents have demonstrated remarkable capabilities across a wide range of complex tasks, including question answering (39), code generation (44), and autonomous driving (15). However, organizing multiple LLM-based agents into a cohesive team capable of exhibiting human-like collective intelligence poses a range of new challenges (7; 38; 19; 30; 13). Among these challenges, a fundamental one lies in designing the communication topology—that is, determining how agents should be connected and how they should transmit information during collaboration—as it directly shapes information flow, coordination effectiveness, and ultimately the collective intelligence of the system. Early studies typically relied on manually specified and predefined topologies (7; 38; 19; 30; 13). Although simple and effective, these topologies are largely task-agnostic, limiting their adaptability to diverse tasks and evolving collaboration demands. To improve task adaptivity and communication efficiency, recent works have increasingly explored learnable communication topology designers, which broadly fall into two paradigms. Pruning-based approaches begin with an existing collaboration workflow and remove unnecessary agents or communication links through learned gating mechanisms, structured dropout, or importance estimation (42; 35; 18). Generation-based approaches, by contrast, formulate topology design as a task-conditioned graph generation problem and synthesize collaboration topologies using autoregressive node-and-edge decoding, mixture-of-experts graph generators, or guided graph diffusion (20; 21; 14; 45). Figure 1: Random edge masking on G-Designer-generated communication graphs for MMLU and HumanEval. The edge masking ratio ranges from 0.05 to 0.80 in increments of 0.05, with ten random seeds evaluated at each ratio. Each point denotes the accuracy obtained with one random mask, while the dashed line marks the performance of the original unmasked G-Designer graph. Most random masks degrade performance, whereas only a few preserve or improve accuracy, indicating that task-preserving compact subgraphs exist but are difficult to identify through random pruning. Despite their impressive performance, we observe that the communication topologies generated by existing methods still contain substantial redundancy. As illustrated in Figure 1, removing several communication links in the topologies generated by G-Designer (43) has little impact on the final task performance and, in some cases, even improves it. This observation suggests that the learned topologies may contain not only redundant edges but also communication patterns that are merely correlated with, rather than causally relevant to, successful collaboration. Such redundancy is difficult to discover under the black-box optimization paradigm adopted by existing methods, where topologies are optimized solely using topology-level task rewards without explicitly attributing the final outcome to individual communication decisions or substructures. Consequently, existing topology designers provide little insight into which communication substructures are truly responsible for successful collaboration. This motivates the need for a post-hoc explanation framework that can identify critical causal communication substructures, thereby revealing why a learned topology succeeds and enabling redundant communication edges to be pruned for improved efficiency. In this work, we explicitly investigate the causal contribution of individual communication edges to the task outcomes achieved by the overall communication topology. Our formulation is inspired by the notion of Granger causality, which considers xi→Yx_i→ Y causal if the information provided by xix_i improves the prediction of Y (9; 37). Analogously, in a communication topology, if the information transmitted through an edge improves the cooperative performance of the MAS, that edge can be regarded as causally important to the resulting collaboration. Building upon this intuition, we extend the Granger-style criterion from individual communication edges to local communication substructures, aiming to identify compact causal subgraphs that preserve task-relevant interactions while eliminating redundant communication. Specifically, we propose E2-Explainer, a model-agnostic post-hoc framework for identifying compact causal subgraphs from communication topologies generated by arbitrary topology designers. Given a communication topology, E2-Explainer estimates the Granger-style causal contribution of each edge by systematically masking it and measuring the resulting change in a task evaluation signal. This signal is primarily defined by the final task outcome and is supplemented with result-level semantic entropy changes to provide denser feedback under sparse rewards. The estimated edge contributions are then integrated with structural constraints to identify compact causal subgraphs that preserve task-relevant communication while eliminating redundant interactions. Since this attribution process requires repeated executions of the underlying LLM-based MAS, we further train an amortized explainer using the discovered causal subgraphs as supervision. At deployment, the explainer directly predicts a compact causal subgraph from a newly generated topology, avoiding repeated edge-masking evaluations. Extensive experiments across six reasoning and coding benchmarks demonstrate that E2-Explainer consistently improves the performance–communication cost trade-off of multiple topology designers while exhibiting strong transferability across different designers and agent scales. Our contributions are summarized as follows: • We formulate post-hoc explanation of optimized LLM-MAS communication graphs as the identification of compact subgraphs that preserve the task behavior of the original topology, and introduce a Granger-style criterion for evaluating the contribution of communication edges. • We propose E2-Explainer, which distills edge-level causal evidence obtained from offline masking evaluations into budget-specific subgraph supervision and trains an amortized explainer to directly generate compact, collaboration-preserving communication subgraphs without requiring edge-contribution evaluations at test time. • We conduct extensive experiments across multiple benchmarks, topology optimizers, and agent scales, showing that E2-Explainer reduces communication costs while maintaining competitive task performance and exhibits cross-generator and cross-scale transferability. Related Work LLM-based multi-agent systems. LLM-based multi-agent systems organize multiple language-model agents into collaborative workflows, where agents exchange intermediate solutions, critique one another, and aggregate final decisions (7; 38; 19; 30; 13). They build on decomposed reasoning, self-consistency, and multi-perspective deliberation to improve LLM reliability (36; 34; 16), and have been summarized in recent surveys on agent workflows and applications (11; 22; 5). Our work focuses on their communication structure rather than new roles or prompting strategies. Communication topology optimization. A growing line of work studies how to design efficient communication topologies for LLM-MAS. Recent methods can be roughly grouped into generative topology methods, which construct task-adaptive collaboration graphs, such as G-Designer and ARG-Designer (43; 20), and pruning or selection methods, which remove unnecessary links, agents, or graph components from an existing workflow, such as AgentPrune, AgentDropout, Cut the Crap, and adaptive graph pruning methods (42; 35; 18; 2). These methods show that topology strongly affects the performance–cost trade-off. E2-Explainer is complementary: it treats generated topologies as candidate graphs and learns to refine them into smaller task-preserving communication subgraphs. Causality in machine learning. Causality provides a principled framework for reasoning about interventions, counterfactuals, and invariant mechanisms (28; 29; 32). In machine learning, causal ideas have been used to analyze model behavior under controlled perturbations and to identify factors that are responsible for predictions (3; 33). Granger causality characterizes causal informativeness by whether access to one variable improves the prediction of another (9; 10; 1). In graph learning, this intuition has been connected to explanation methods that identify compact graph structures responsible for preserving or changing model behavior, such as GNNExplainer, PGExplainer, counterfactual explainers, and GEM (41; 26; 25; 23). Inspired by these ideas, we adapt Granger-style causal reasoning from differentiable graph models to frozen LLM-MAS communication graphs. Unlike prior graph explanation methods that primarily produce explanations for inspection, E2-Explainer distills offline edge-masking evidence into an amortized explainer that directly outputs budgeted task-preserving communication subgraphs at deployment. Problem Formulation LLM-MAS Communication Graphs We consider an LLM-based multi-agent system with agents V=a1,…,anV=\a_1,…,a_n\. Given query x, a topology generator ϕG_φ produces a directed communication graph G=ϕ(x)=(V,E),G=G_φ(x)=(V,E), where e=(ai,aj)∈Ee=(a_i,a_j)∈ E delivers the output of agent aia_i to agent aja_j. Executing G yields a final response y(x,G)y(x,G) and a dataset-specific task score Q(x,G)Q(x,G), such as exact-match accuracy, execution correctness, or an LLM-judged score. Post-hoc Communication Explanation Given a generated graph G, we seek a compact and executable subgraph H=(VH,EH),H⊆G,H=(V_H,E_H), H G, that preserves its task behavior. The retained edges explain the communication paths supporting the original collaboration, while executing H may reduce communication overhead. We denote the realized online cost by C(x,H)C(x,H), measured using actual execution tokens. Because removing an edge changes downstream prompts and messages, C(x,H)C(x,H) need not vary monotonically with the number of retained edges or nodes. Objective and Deployment Constraint Let b=(ρE,ρV)b=( _E, _V) denote a subgraph budget, where ρE _E and ρV _V specify the desired edge-pruning and node-pruning ratios, and let ℋb(G)H_b(G) be the corresponding family of compact subgraphs. A valid explanation under budget b should satisfy Hb⋆∈ℋb(G),Q(x,Hb⋆)≈Q(x,G).H_b _b(G), Q(x,H_b )≈ Q(x,G). (1) The budget determines compactness, while the task-behavior condition defines faithfulness. We report C(x,Hb⋆)C(x,H_b ) as measured efficiency rather than assuming that structural sparsity guarantees a fixed token reduction. Because validating candidate subgraphs requires repeated LLM-MAS executions and may require task feedback, we learn an amortized explainer FθF_θ: H^b=Fθ(x,G,b),H^b⊆G. H_b=F_θ(x,G,b), H_b G. (2) Figure 2: Overview of E2-Explainer. Edge masking estimates preservation utilities from task-score and semantic-entropy changes, which yield budgeted subgraphs for supervision. The amortized explainer then predicts a task-preserving subgraph for an unseen topology in one forward pass. E2-Explainer As illustrated in Figure 2, E2-Explainer is a model-agnostic post-hoc framework for identifying compact causal subgraphs from communication topologies generated by arbitrary topology designers. Given a communication topology, E2-Explainer estimates the Granger-style causal contribution of each edge by systematically masking it and measuring the resulting change in a task evaluation signal. This signal is primarily derived from the final task outcome and further incorporates topology-level semantic entropy changes to provide denser feedback when the task outcome is sparse or coarse-grained. The estimated causal contributions are then combined with budget constraints and structural requirements to construct executable compact subgraphs that preserve task-relevant communication patterns. Since this attribution process requires repeated edge-masking evaluations of the underlying LLM-based MAS, we use the discovered subgraphs as supervision to train an amortized explainer, which can directly predict a budget-specific subgraph for a newly generated topology at test time. Edge-Level Granger-Style Causal Attribution To estimate the Granger-style causal contribution of each communication edge under the original topology, we perform single-edge removal interventions while keeping all other factors fixed. For a query x, graph G, and edge e=(u,v)e=(u,v), we preserve the same agents, prompts, decoding configurations, and all other communication channels, while intervening on e by blocking the message transmitted from source agent u to target agent v. Following the Granger criterion, we quantify the contribution of e by measuring the change in the evaluation signal before and after this intervention. Task-level causal contribution. For each active edge e∈Ee∈ E of the generated graph G=(V,E)G=(V,E), we construct the intervened graph G−e=G∖eG^-e=G \e\ and re-execute the frozen LLM-MAS. Let Q(x,G)Q(x,G) and Q(x,G−e)Q(x,G^-e) denote the task scores obtained before and after intervening on e, respectively. We define the task-level causal contribution of e as Δetask=Q(x,G)−Q(x,G−e). ^task_e=Q(x,G)-Q(x,G^-e). (3) A positive value indicates that retaining edge e improves the task outcome relative to removing it, whereas a negative value suggests that the information transmitted through e may be detrimental to the task outcome. Since our explanation focuses on communication that supports the original collaboration, we retain only the nonnegative preservation contribution: uetask=max(0,Δetask).u_e^task= (0, _e^task ). (4) A larger value provides stronger evidence that edge e contributes to preserving the task behavior of the candidate graph, while zero indicates no positive preservation evidence under the observed task metric. Auxiliary semantic signal. The final task outcome can be too coarse to distinguish edge masks that receive the same task score. We therefore use result-level semantic entropy as an auxiliary signal to characterize the stability of the final response (17; 8). For the response set Y obtained from M stochastic executions, we group semantically equivalent responses into classes ()C(Y) and compute H¯()=−∑c∈()p(c)logp(c), H(Y)=- _c (Y)p(c) p(c), (5) where p(c)p(c) is the empirical frequency of class c among the M executions. We use M=5M=5 independently sampled executions for both the original graph and each edge-masked graph. The auxiliary semantic effect of edge e is defined as uesem=max(0,H¯(G−e)−H¯(G)).u^sem_e= (0,\, H(Y^G^-e)- H(Y^G) ). (6) A positive value indicates that masking e increases the uncertainty of the final response, providing additional evidence that the edge contributes to preserving the behavior of the original graph. Unified edge effect. We combine the task-level causal effect with the auxiliary semantic stability signal. The resulting edge utility is ue=clip[0,1](α⋅u~etask+β⋅u~esem),u_e=clip_[0,1] (α· u^task_e+β· u^sem_e ), (7) where ⋅~ · denotes normalization within the current graph or calibration batch, and α,β≥0α,β≥ 0 control the two signals. The utilities induce a preservation ordering over the edges of G: πG=(e(1),e(2),…,e(|E|)),ue(1)≥ue(2)≥⋯≥ue(|E|). _G=(e_(1),e_(2),…,e_(|E|)), u_e_(1)≥ u_e_(2)≥·s≥ u_e_(|E|). (8) This ordering connects edge-level Granger-style attribution to budgeted subgraph construction. Causal Subgraph Extraction The edge-level causal contributions obtained above provide a principled basis for extracting compact communication subgraphs. Given a target budget b, E2-Explainer selects edges according to their preservation utilities and constructs a subgraph that retains the most task-relevant information flows while satisfying the structural constraints inherited from the original workflow. This budget-aware extraction process achieves a desirable trade-off between communication sparsity and task preservation without requiring exhaustive subgraph enumeration. Specifically, the budget determines the number of retained edges, while the causal ordering determines which communication paths are preserved. Budget-Aware Subgraph Selection. The budget b=(ρE,ρV)b=( _E, _V) specifies the desired retention ratios of edges and optional nodes. For each graph, these ratios are converted into retention counts kE(b)k_E(b) and kV(b)k_V(b) according to the graph size. We first select the most important edges according to the preservation ordering obtained from edge attribution: Eb⋆=TopK(πG,kE(b)).E _b=TopK ( _G,k_E(b) ). (9) The node budget is subsequently enforced jointly with the structural projection described below to obtain an executable communication subgraph. Executability-Constrained Projection. To transform the selected high-utility edges into an executable communication topology, we apply an executability-constrained projection: Hb⋆=ℛvalid(G,Eb⋆,kV(b)).H _b=R_valid(G,E _b,k_V(b)). (10) The projection ℛvalidR_valid only selects edges from the original edge set E and never introduces new links. Therefore, when the original topology G is a directed acyclic graph, the extracted subgraph naturally preserves acyclicity. Specifically, the projection retains mandatory input and final-aggregation agents, removes communication links incident to excluded optional agents, and discards optional agents that no longer participate in valid directed communication paths. When the node budget is active, optional agents are prioritized according to the aggregated preservation utilities of their incident retained edges. The resulting Hb⋆H_b therefore satisfies both the specified budget constraints and the execution requirements of the underlying LLM-MAS. Budget-Specific Explanation Family. Applying Eq. (10) to the budget set ℬB yields an explanation family ℱ⋆(x,G)=Hb⋆:b∈ℬ.F (x,G)=\H _b:b \. (11) Each member provides an explanation of the same candidate topology under a different compactness requirement. Collectively, these subgraphs characterize different trade-offs between communication efficiency and task preservation. As the retention budget increases, the extracted subgraphs progressively incorporate lower-ranked communication edges to recover additional information flows. Amortized Subgraph Explainer The causal attribution and structural projection procedures described above provide a reliable mechanism for identifying compact, task-preserving subgraphs from arbitrary communication topologies. However, directly applying this procedure to a newly generated topology at test time is computationally expensive, as it requires repeated edge-masking interventions and multiple executions of the underlying LLM-MAS to estimate edge contributions. To overcome this limitation, we amortize the causal extraction process by collecting extracted subgraphs from generated topologies as graph-to-subgraph supervision. Specifically, these causally grounded subgraphs serve as explanation targets that teach an explainer to predict which communication paths should be preserved under different compactness budgets. After training, the amortized explainer can directly generate budget-specific causal subgraphs for unseen topologies without additional causal evaluations. Graph-to-subgraph supervision. For each query xix_i and candidate graph GiG_i, the extraction procedure produces one target subgraph Hi,b⋆H _i,b for every budget b∈ℬb . These targets form the training set expl=(xi,Gi,b,Hi,b⋆)∣b∈ℬ.D_expl= \(x_i,G_i,b,H _i,b) b \. (12) Thus, the explanation family provides supervision rather than an input to the explainer. For a fixed graph, the targets share the same causal preservation ordering but correspond to different pruning budgets. Learning from the complete family therefore teaches the explainer both which communication elements are important and how their retention changes with the requested compactness, allowing one model to serve multiple budgets. Explainer mapping. We train a parameterized explainer FθF_θ to approximate the graph-to-subgraph mapping induced by the causal extraction procedure: H^i,b=Fθ(xi,Gi,b),H^i,b⊆Gi. H_i,b=F_θ(x_i,G_i,b), H_i,b G_i. (13) Given a query, a candidate graph, and a pruning budget, FθF_θ internally predicts edge and node retention probabilities and decodes them into the corresponding budget-specific subgraph. The explainer is independent of the topology generation mechanism, with architectural details deferred to the Appendix. Training objective. Let Ai,b,e⋆A _i,b,e indicate whether edge e belongs to Hi,b⋆H _i,b, and let Ri,b,v⋆R _i,b,v indicate whether node v is retained. The explainer predicts the corresponding probabilities A^i,b,e A_i,b,e and R^i,b,v R_i,b,v, which are collected into the edge- and node-retention vectors ^i,b A_i,b and ^i,b R_i,b, respectively. We optimize ℒedge _edge =∑(xi,Gi,b,Hi,b⋆)∈expl1|Ei|∑e∈EiBCE(Ai,b,e⋆,A^i,b,e), = _(x_i,G_i,b,H _i,b) _expl 1|E_i| _e∈ E_iBCE (A _i,b,e, A_i,b,e ), (14) ℒnode _node =∑(xi,Gi,b,Hi,b⋆)∈expl1|Vi|∑v∈ViBCE(Ri,b,v⋆,R^i,b,v), = _(x_i,G_i,b,H _i,b) _expl 1|V_i| _v∈ V_iBCE (R _i,b,v, R_i,b,v ), with the final objective ℒ=ℒedge+λnodeℒnode+λwd‖θ‖22.L=L_edge+ _nodeL_node+ _wd\|θ\|_2^2. (15) Method Metric MMLU GSM8K MultiArith SVAMP AQuA HumanEval Avg. Vanilla Acc. 72.78 88.15 96.67 93.40 82.68 81.37 85.84 CoT Acc. 76.81 91.81 97.67 93.60 83.07 87.58 88.42 SC(CoT) Acc. 77.34 92.48 97.67 93.83 84.13 87.08 88.76 ARG-Designer Acc. 76.68 89.18 97.81 93.57 83.95 86.78 88.00 ARG-Designer + E2-Explainer Acc. 77.99↑ 1.31 86.94↓ 2.24 97.17↓ 0.64 94.67↑ 1.10 84.58↑ 0.63 89.26↑ 2.48 88.44↑ 0.44 Token Δ -12.1% -15.5% -23.3% -21.1% -29.8% -14.3% -20.1% OFA-MAS Acc. 79.91 91.01 98.21 93.97 84.11 89.26 89.41 OFA-MAS + E2-Explainer Acc. 81.87↑ 1.96 90.40↓ 0.61 97.41↓ 0.80 92.65↓ 1.32 86.67↑ 2.56 90.62↑ 1.36 89.94↑ 0.53 Token Δ -15.1% -27.2% -25.5% -23.8% -17.8% -26.8% -25.6% AgentPrune Acc. 78.43 92.78 98.33 94.90 85.04 87.58 89.51 AgentPrune + E2-Explainer Acc. 79.08↑ 0.65 94.09↑ 1.31 98.33± 0.00 94.20↓ 0.70 86.61↑ 1.57 88.20↑ 0.62 90.09↑ 0.58 Token Δ -14.7% -26.7% -27.3% -22.9% -38.1% -10.6% -24.9% G-Designer Acc. 78.65 93.05 98.13 95.00 84.52 88.40 89.63 G-Designer + E2-Explainer Acc. 79.74↑ 1.09 93.77↑ 0.72 98.17↑ 0.04 94.60↓ 0.40 87.30↑ 2.78 90.69↑ 2.29 90.71↑ 1.09 Token Δ -31.5% -44.0% -28.0% -20.1% -37.5% -19.3% -25.1% Table 1: Main results with Qwen3-8B as the backbone LLM. Boldface indicates the best accuracy for each dataset. For E2-Explainer variants, Acc. arrows indicate changes relative to the corresponding candidate generator, while the Avg. column reports average accuracy and weighted token reduction across all datasets. Subgraph prediction. The predicted probabilities are first converted into budget-compatible candidate sets: E^i,b E_i,b =TopK(^i,b,kE(b)), =TopK ( A_i,b,k_E(b) ), (16) V^i,b V_i,b =TopK(^i,b,kV(b)). =TopK ( R_i,b,k_V(b) ). Accordingly, the final subgraph is obtained as H^i,b=Fθ(xi,Gi,b)=ℛvalid(Gi,E^i,b,V^i,b), H_i,b=F_θ(x_i,G_i,b)=R_valid (G_i, E_i,b, V_i,b ), (17) where mandatory nodes are retained by ℛvalidR_valid. At deployment, this process requires neither task labels nor repeated edge interventions. Experiment Experimental Setup Datasets and backbone. We evaluate on six benchmarks: AQuA, GSM8K, MultiArith, and SVAMP for mathematical reasoning (24; 6; 31; 27), MMLU for knowledge-intensive reasoning (12), and HumanEval for code generation (4). All methods use Qwen3-8B as the backbone LLM (40), with Vanilla, CoT (36), and SC(CoT) (34) as single-agent baselines. Candidate generators and calibration. We train a single explainer using only G-Designer-generated candidate graphs. For each dataset, 40 calibration inputs are sampled from the training split without overlap with the test split, and edge-masking probes construct graph-to-subgraph supervision. The frozen explainer is then applied without retraining to graphs produced by ARG-Designer, OFA-MAS, AgentPrune, and G-Designer. Subgraph setting and metrics. The main comparison uses E25+N20E25+N20 as the default subgraph-size setting, while other settings are examined in the appendix. We report task accuracy and actual online token usage; changes for E2-Explainer are computed relative to the corresponding candidate generator. Results and Analysis Main comparison. Table 1 evaluates E2-Explainer as a post-hoc module for four representative topology optimizers. Across all four optimizers, E2-Explainer reduces weighted token usage by 20.1%–25.6%, while preserving or improving their average accuracy. Specifically, it improves the average accuracy of ARG-Designer, OFA-MAS, AgentPrune, and G-Designer by 0.44, 0.53, 0.58, and 1.09 points, respectively. G-Designer+E2-Explainer achieves the highest average accuracy of 90.71 while consuming 25.1% fewer tokens than the original G-Designer. These results directly support the primary objective of E2-Explainer: reducing communication overhead without sacrificing the collaborative effectiveness of an already optimized topology. The fine-grained results show that the reduction in communication cost is more consistent than the change in task accuracy. Across the 24 optimizer–dataset combinations, E2-Explainer reduces token usage in every case, with dataset-level reductions ranging from 10.6% to 44.0%. Meanwhile, accuracy improves in 16 combinations and remains unchanged in one. Improvements are particularly consistent on MMLU, AQuA, and HumanEval, where all four topology optimizers benefit from post-processing. Results on GSM8K, MultiArith, and SVAMP are more mixed, indicating that the amount of removable communication depends on both the task and the initial candidate topology. Nevertheless, the universal token reduction and predominantly preserved or improved accuracy show that compact task-preserving subgraphs can often be recovered from optimized communication graphs. The additional accuracy gains further suggest that some candidate topologies still retain redundant or distracting communication whose removal can benefit answer quality. A key observation is that the improvement transfers beyond the generator used for calibration. The explainer is trained only on G-Designer graphs but applied without retraining to ARG-Designer, OFA-MAS, and AgentPrune. These unseen generators have different construction biases, covering autoregressive graph generation, mixture-of-experts graph generation, and pruning-based topology optimization. Nevertheless, E2-Explainer improves their average accuracy by 0.44–0.58 points and reduces weighted online cost by 20.1%–25.6%. Since these generators are unseen during calibration, the transfer is unlikely to come from memorizing G-Designer-specific adjacency patterns. Instead, the explainer appears to capture reusable cues about which communication paths are necessary under a given task and graph context. Setting MMLU HumanEval Acc. Tok. ↓ Acc. Tok. ↓ Original (6-agent) 79.43 – 89.26 – E2-Explainer (5→ 6-agent) 81.70 10.16% 90.08 19.38% Table 2: Generalization across agent scales on G-Designer-generated communication graphs. The explainer is trained on 5-agent graphs and evaluated on 6-agent graphs. Token reductions are measured against the original 6-agent topology. Generalization across agent scales. Table 2 further tests cross-scale generalization by training the explainer on 5-agent G-Designer graphs and directly evaluating it on 6-agent graphs. Without retraining, E2-Explainer improves MMLU accuracy from 79.43 to 81.70 and HumanEval accuracy from 89.26 to 90.08, while also reducing token usage by 10.16% and 19.38%, respectively. These results suggest that the learned explainer transfers beyond the graph size observed during training and can still identify compact task-preserving subgraphs under moderate changes in agent scale. Method MMLU HumanEval Acc. Token Δ Acc. Token Δ w/o Causal 75.82 -11.16% 86.88 -5.05% w/o Sem. Entropy 77.78 -23.42% 87.50 -14.32% E2-Explainer 79.74 -31.50% 90.69 -19.25% Table 3: Component ablation of E2-Explainer on G-Designer-generated communication graphs for MMLU and HumanEval. Component ablation. Table 3 evaluates the roles of the causal and auxiliary semantic-entropy signals. Removing the causal signal leads to a larger degradation, reducing accuracy from 79.74 to 75.82 on MMLU and from 90.69 to 86.88 on HumanEval. The causal-only variant, namely w/o Sem. Entropy, performs better than the semantic-entropy-only variant, suggesting that task-level causal effects provide the primary preservation-aware signal for edge valuation. However, causal-only still falls short of the full model. This indicates that the result-level semantic entropy proxy complements causal supervision by densifying the refinement signal when final-task feedback is sparse. Source MMLU HumanEval Acc. Token Δ Acc. Token Δ Target: G-Designer Original 78.65 – 88.40 – G-Designer 79.74 -31.50% 90.69 -19.25% AgentPrune 79.08 -14.43% 88.75 -15.40% Mixed 80.18 -24.55% 89.30 -20.61% Target: AgentPrune Original 78.43 – 87.58 – G-Designer 79.08 -14.70% 88.20 -10.60% AgentPrune 81.70 -24.56% 90.07 -19.18% Mixed 79.08 -26.69% 88.82 -13.43% Table 4: Effect of calibration sources under E25+N20E25+N20. Original denotes unrefined target graphs, and Mixed combines G-Designer and AgentPrune calibration graphs. Boldface marks the best refined accuracy. Effect of Calibration Sources. Table 4 compares explainers trained on G-Designer graphs, AgentPrune graphs, or their mixture and evaluated on both target optimizers. All six calibration–target combinations improve accuracy while reducing token usage, providing evidence of bidirectional cross-optimizer transfer. Source matching remains beneficial: AgentPrune-only calibration achieves the highest accuracy on both datasets for AgentPrune targets, while G-Designer-only performs best on HumanEval for G-Designer targets. Mixed-source calibration obtains the best G-Designer result on MMLU and improves both target optimizers, but does not uniformly outperform single-source calibration. These results suggest that E2-Explainer learns transferable communication patterns while still benefiting from structural characteristics specific to the target optimizer. Conclusion We present E2-Explainer, a post-hoc framework for identifying compact and collaboration-preserving communication subgraphs from optimized LLM-MAS topologies. E2-Explainer distills offline edge-level evaluations into an amortized explainer, enabling efficient test-time generation of compact communication subgraphs that preserve collaborative task performance without requiring ground-truth answers or repeated edge-level evaluations. Experiments across multiple topology optimizers show that it reduces communication costs while maintaining task performance, and transfers across graph sources and agent scales. References Bressler and Seth (2011) S. L. Bressler and A. K. Seth Wiener–granger causality: a well established methodology. NeuroImage 58 (2), p. 323–329. Cited by: Causality in machine learning.. Cang et al. (2026) Y. Cang, X. Zhang, E. Zhao, Z. Ji, Y. Liu, Y. He, Z. Ning, Y. Chen, W. Que, and L. Shi Graph-grpo: stabilizing multi-agent topology learning via group relative policy optimization. In Findings of the Association for Computational Linguistics: ACL 2026, p. 20222–20231. Cited by: Communication topology optimization.. Chattopadhyay et al. (2019) A. Chattopadhyay, P. Manupriya, A. Sarkar, and V. N. Balasubramanian Neural network attributions: a causal perspective. In International Conference on Machine Learning, Cited by: Causality in machine learning.. Chen et al. (2021) M. Chen et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: Datasets and backbone.. Chen et al. (2024) W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Qian, C. Chan, Y. Qin, Y. Lu, R. Xie, Z. Liu, and M. Sun A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. Cited by: LLM-based multi-agent systems.. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Datasets and backbone.. Du et al. (2024) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235. Cited by: Introduction, Introduction, LLM-based multi-agent systems.. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630, p. 625–630. Cited by: Auxiliary semantic signal.. Granger (1969) C. W. J. Granger Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37 (3), p. 424–438. Cited by: Introduction, Causality in machine learning.. Granger (1980) C. W. J. Granger Testing for causality: a personal viewpoint. Journal of Economic Dynamics and Control 2, p. 329–352. Cited by: Causality in machine learning.. Guo et al. (2024) T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang Large language model based multi-agents: a survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, p. 8048–8057. External Links: Document Cited by: LLM-based multi-agent systems.. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, Cited by: Datasets and backbone.. Hong et al. (2024) S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber MetaGPT: meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, Cited by: Introduction, Introduction, LLM-based multi-agent systems.. Jiang et al. (2026) E. H. Jiang, L. Li, F. Wan, X. Liang, S. Yin, Y. Wu, X. Li, Y. Sun, W. Wang, K. Chang, and Y. N. Wu Dynamic generation of multi llm agents communication topologies with graph diffusion models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 38042–38060. External Links: Document Cited by: Introduction. Jin et al. (2023) Y. Jin, X. Shen, H. Peng, X. Liu, J. Qin, J. Li, J. Xie, P. Gao, G. Zhou, and J. Gong Surrealdriver: designing generative driver agent simulation framework in urban contexts based on large language model. arXiv preprint arXiv:2309.13193 5 (7), p. 8. Cited by: Introduction. Kojima et al. (2022) T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, Vol. 35, p. 22199–22213. Cited by: LLM-based multi-agent systems.. Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations, Cited by: Auxiliary semantic signal.. Li et al. (2025) B. Li, Z. Zhao, D. Lee, and G. Wang Adaptive graph pruning for multi-agent communication. arXiv preprint arXiv:2506.02951. Cited by: Introduction, Communication topology optimization.. Li et al. (2023) G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem CAMEL: communicative agents for “mind” exploration of large scale language model society. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: Introduction, Introduction, LLM-based multi-agent systems.. Li et al. (2026a) S. Li, Y. Liu, Q. Wen, C. Zhang, and S. Pan Assemble your crew: automatic multi-agent communication topology design via autoregressive graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 23142–23150. External Links: Document Cited by: Introduction, Communication topology optimization.. Li et al. (2026b) S. Li, Y. Liu, Y. Zheng, M. Li, Q. V. H. Nguyen, and S. Pan OFA-mas: one-for-all multi-agent system topology design based on mixture-of-experts graph generative models. In Proceedings of the ACM Web Conference 2026, p. 1333–1344. Cited by: Introduction. Li et al. (2024) X. Li, S. Wang, S. Zeng, Y. Wu, and Y. Yang A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 1 (1), p. 9. Cited by: LLM-based multi-agent systems.. Lin et al. (2021) W. Lin, H. Lan, and B. Li Generative causal explanations for graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, p. 6666–6679. Cited by: Causality in machine learning.. Ling et al. (2017) W. Ling, D. Yogatama, C. Dyer, and P. Blunsom Program induction by rationale generation: learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 158–167. External Links: Document Cited by: Datasets and backbone.. Lucic et al. (2022) A. Lucic, M. A. Ter Hoeve, G. Tolomei, M. de Rijke, and F. Silvestri CF-gnnexplainer: counterfactual explanations for graph neural networks. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 151, p. 4499–4511. Cited by: Causality in machine learning.. Luo et al. (2020) D. Luo, W. Cheng, D. Xu, W. Yu, B. Zong, H. Chen, and X. Zhang Parameterized explainer for graph neural network. In Advances in Neural Information Processing Systems, Vol. 33, p. 19620–19631. Cited by: Causality in machine learning.. Patel et al. (2021) A. Patel, S. Bhattamishra, and N. Goyal Are NLP models really able to solve simple math word problems?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 2080–2094. External Links: Document Cited by: Datasets and backbone.. Pearl (2009) J. Pearl Causality: models, reasoning, and inference. 2 edition, Cambridge University Press. Cited by: Causality in machine learning.. Peters et al. (2017) J. Peters, D. Janzing, and B. Schölkopf Elements of causal inference: foundations and learning algorithms. MIT Press. Cited by: Causality in machine learning.. Qian et al. (2024) C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: Introduction, Introduction, LLM-based multi-agent systems.. Roy and Roth (2015) S. Roy and D. Roth Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, p. 1743–1752. External Links: Document Cited by: Datasets and backbone.. Schölkopf et al. (2021) B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio Toward causal representation learning. Proceedings of the IEEE 109 (5), p. 612–634. Cited by: Causality in machine learning.. Schwab and Karlen (2019) P. Schwab and W. Karlen CXPlain: causal explanations for model interpretation under uncertainty. Advances in Neural Information Processing Systems 32. Cited by: Causality in machine learning.. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, Cited by: LLM-based multi-agent systems., Datasets and backbone.. Wang et al. (2025) Z. Wang, Y. Wang, X. Liu, L. Ding, M. Zhang, J. Liu, and M. Zhang AgentDropout: dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, p. 24013–24035. Cited by: Introduction, Communication topology optimization.. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 24824–24837. Cited by: LLM-based multi-agent systems., Datasets and backbone.. Wiener (1956) N. Wiener The theory of prediction. Modern Mathematics for Engineers, p. 165–190. Cited by: Introduction. Wu et al. (2024) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang AutoGen: enabling next-gen llm applications via multi-agent conversation. In First Conference on Language Modeling, Cited by: Introduction, Introduction, LLM-based multi-agent systems.. Xu et al. (2024) Y. Xu, S. He, J. Chen, Z. Wang, Y. Song, H. Tong, G. Liu, J. Zhao, and K. Liu Generate-on-graph: treat LLM as both agent and KG for incomplete knowledge graph question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), p. 18410–18430. External Links: Link, Document Cited by: Introduction. Yang et al. (2025) A. Yang, A. Li, B. Yang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Datasets and backbone.. Ying et al. (2019) Z. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec GNNExplainer: generating explanations for graph neural networks. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: Causality in machine learning.. Zhang et al. (2025a) G. Zhang, Y. Yue, Z. Li, S. Yun, G. Wan, K. Wang, D. Cheng, J. X. Yu, and T. Chen Cut the crap: an economical communication pipeline for llm-based multi-agent systems. In International Conference on Learning Representations, Cited by: Introduction, Communication topology optimization.. Zhang et al. (2025b) G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng G-designer: architecting multi-agent communication topologies via graph neural networks. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267. Cited by: Introduction, Communication topology optimization.. Zhang et al. (2024) K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin Codeagent: enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13643–13658. Cited by: Introduction. Zhang et al. (2026) Z. Zhang, W. Zhou, J. Li, H. Fei, J. Wen, and W. Ji RADAR: redundancy-aware diffusion for multi-agent communication structure generation. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306. Cited by: Introduction. Appendix A Implementation Details Explainer Input and Output The amortized explainer learns the budget-conditioned mapping Fθ:(x,G,b)↦H^b,H^b⊆G,F_θ:(x,G,b) H_b, H_b G, (18) where x is the task query, G=(V,E)G=(V,E) is a candidate communication graph, and b=(ρE,ρV)b=( _E, _V) specifies the edge- and node-pruning ratios. All inputs are available before executing the predicted subgraph; the explainer does not require ground-truth answers, repeated edge interventions, or internal parameters of the topology generator. For each edge e=(u,v)∈Ee=(u,v)∈ E and node v∈Vv∈ V, the network predicts an edge-retention probability A^b,e∈[0,1] A_b,e∈[0,1] and a node-retention probability R^b,v∈[0,1] R_b,v∈[0,1]. These probabilities are intermediate outputs rather than the final explanation. They are converted into budget-compatible candidate sets and passed through the validity projection to obtain the executable subgraph H^b H_b. The explainer therefore learns a budget-conditioned graph-to-subgraph mapping. The same candidate graph can yield different explanations under different pruning budgets, while the supervision is defined by the edge and node membership of the corresponding compact target subgraph Hb⋆H_b . Budget-Conditioned Explainer Architecture Input representations. Let xq_x denote a fixed embedding of query x, and let vr_v denote a fixed embedding of the role or system prompt assigned to agent v. Each node is also associated with a structural feature vector vt_v, containing graph-local information available from the candidate topology, such as normalized in-degree and out-degree and topological position. A shared node encoder produces v=ϕnode([x‖v‖v]),G=1|V|∑v∈Vv,h_v= _node ([q_x\|r_v\|t_v] ), _G= 1|V| _v∈ Vh_v, (19) where ∥\| denotes concatenation and Gg_G is a permutation-invariant graph-level summary. The pruning ratios are encoded as b=ϕb([ρE,ρV]).z_b= _b([ _E, _V]). (20) Edge and node prediction heads. For an active edge e=(u,v)e=(u,v), let es_e collect edge-local structural features, including the edge type and normalized structural statistics of its endpoints. The shared edge head predicts A^b,e=σ(ϕedge([u‖v‖G‖b‖e])). A_b,e=σ\! ( _edge([h_u\|h_v\|g_G\|z_b\|s_e]) ). (21) The node head predicts R^b,v=σ(ϕretain([v‖G‖b])). R_b,v=σ\! ( _retain([h_v\|g_G\|z_b]) ). (22) All node and edge instances share the same encoders and prediction heads. Consequently, the number of trainable parameters does not depend on the numbers of agents or communication links, which allows the explainer to process graphs of different sizes. The modules ϕnode _node, ϕb _b, ϕedge _edge, and ϕretain _retain are implemented as lightweight multilayer perceptrons with parameters shared across graph elements. Fixed task and role representations are computed once and reused during explainer training and inference. Graph-to-Subgraph Supervision and Decoding For each calibration pair (xi,Gi)(x_i,G_i), causal subgraph construction produces one target Hi,b⋆H_i,b for every budget b∈ℬb . The resulting training set is expl=(xi,Gi,b,Hi,b⋆)∣b∈ℬ.D_expl=\(x_i,G_i,b,H_i,b ) b \. (23) Each target subgraph defines binary edge- and node-retention labels Ai,b,e⋆=[e∈E(Hi,b⋆)],Ri,b,v⋆=[v∈V(Hi,b⋆)].A_i,b,e =I[e∈ E(H_i,b )], R_i,b,v =I[v∈ V(H_i,b )]. (24) The explainer is optimized with the edge and node binary-cross-entropy losses defined in the main paper. Because one candidate graph contributes one target for each budget, the model learns both which communication elements should be retained and how the selected subgraph changes with the requested pruning ratios. At inference time, the predicted probabilities are converted into candidate sets according to the graph-specific retention counts: E^b=TopK(^b,kE(b)),V^b=TopK(^b,kV(b)). E_b=TopK( A_b,k_E(b)), V_b=TopK( R_b,k_V(b)). (25) The final explanation is H^b=ℛvalid(G,E^b,V^b), H_b=R_valid(G, E_b, V_b), (26) where ℛvalidR_valid enforces the requested node budget together with a minimum-node constraint, removes every edge incident to an excluded node, and retains only feasible directed links between selected agents. The projection ranks feasible elements using the predicted retention scores and selects only nodes and edges already present in G, so it never introduces a new communication link. Causal Supervision Construction Details Repeated execution and task score. For every active edge e, we independently execute both the original graph G and the intervened graph G−eG^-e for M=5M=5 runs. The task score is the mean outcome over these executions: Q(x,G)=1M∑m=1MQ(m)(x,G).Q(x,G)= 1M _m=1^MQ^(m)(x,G). (27) The original graph is re-executed for each evaluated edge rather than sharing one response set across all interventions. All executions use a decoding temperature of 0.20.2. The original and intervened runs are sampled independently and do not use paired random seeds. The same response sets are used to compute both the mean task outcome and the auxiliary semantic signal. Result-level semantic signal. Semantic entropy is computed from the final system responses, rather than from the local outputs of the two agents incident to an edge. This choice aligns the auxiliary signal with the system-level quantity whose causal contribution is being estimated. The M final responses are grouped into semantic equivalence classes using the same equivalence procedure for the original and intervened graphs, after which the empirical class entropy is computed as defined in the main paper. Score normalization. The mean task outcome and its non-negative intervention difference already lie in [0,1][0,1]. We therefore retain the task contribution on its original scale and normalize the semantic contribution by the maximum empirical entropy of M samples: u~etask=uetask,u~esem=min(1,uesemlogM). u^task_e=u^task_e, u^sem_e= (1, u^sem_e M ). (28) The combined preservation utility is then computed with (α,β)=(0.8,0.2)(α,β)=(0.8,0.2). Pruning counts and validity projection. The budget b=(ρE,ρV)b=( _E, _V) specifies removal ratios. For a candidate graph G=(V,E)G=(V,E), the numbers of removed elements are rE(b)=⌈ρE|E|⌉,rV(b)=⌈ρV|V|⌉,r_E(b)= _E|E| , r_V(b)= _V|V| , (29) with retention counts kE(b)=|E|−rE(b)k_E(b)=|E|-r_E(b) and kV(b)=|V|−rV(b)k_V(b)=|V|-r_V(b). Thus, E25 denotes removing 25%25\% of the active edges and N20 denotes removing 20%20\% of the active nodes. After budgeted selection, the validity projection applies the selected node mask, removes incident edges, and retains the highest-ranked feasible links subject to the edge budget and the minimum-node constraint. Scope of the causal attribution. The intervention procedure estimates a first-order conditional contribution for each edge in the context of the remaining full topology. Following the Granger principle, it measures whether excluding one information channel reduces the mean outcome or the stability of the final response while all other channels remain available. We do not enumerate multi-edge coalitions, whose cost grows combinatorially with graph size. Instead, these conditional edge contributions provide a tractable signal for constructing graph-level supervision, while the end-to-end evaluation of the predicted subgraph measures the aggregate outcome after multiple communication elements are removed. Implementation Settings Table 5 summarizes the main settings used for causal supervision construction and explainer training. We train one dataset-specific explainer for each benchmark. The six explainers use the same architecture, optimization configuration, and budget set, but each is trained from the 40 calibration queries of its corresponding dataset. The same frozen Qwen3-8B backbone is used for all methods. Item Setting Semantic samples M 5 Task/semantic weights (α,β)(α,β) (0.8,0.2)(0.8,0.2) Calibration queries per dataset 40 Training budgets ℬB E25+N0, E50+N0, E75+N0, E25+N20, E50+N20, E25+N40 Shared hidden dimension 128 Dropout 0.15 Optimizer AdamW Learning rate 1×10−31× 10^-3 Weight decay 1×10−41× 10^-4 Batch size / epochs 64 / 200 Backbone LLM Qwen3-8B Decoding temperature 0.2 Reported evaluation runs 3 Hardware 4× NVIDIA GeForce RTX 4090 GPUs Inference framework PyTorch and vLLM Token accounting Online prompt + completion tokens Table 5: Main implementation settings. Dataset-specific explainers share the same architecture and optimization configuration. Dataset and Evaluation Details We evaluate E2-Explainer on six reasoning and code-generation benchmarks using the data scale and evaluation protocol adopted by G-Designer and AgentPrune. For datasets with a dedicated training split, 40 calibration queries are selected from that split and are not used as evaluation queries. For HumanEval, we follow the AgentPrune data split: 40 problems are used for calibration and the remaining 124 problems are used for evaluation. G-Designer, AgentPrune, OFA-MAS, ARG-Designer, and all variants equipped with E2-Explainer are evaluated on exactly the same query set for each benchmark. Dataset Task Calib. Eval. Metric MMLU Knowledge MCQ 40 153 Accuracy GSM8K Math reasoning 40 1,319 Exact match MultiArith Math reasoning 40 600 Exact match SVAMP Math reasoning 40 1,000 Exact match AQuA Algebra MCQ 40 254 Accuracy HumanEval Code generation 40 124 Pass@1 Table 6: Dataset statistics and evaluation protocols. HumanEval uses 40 calibration problems and the remaining 124 problems for evaluation. All methods use identical evaluation queries. MMLU. MMLU evaluates knowledge-intensive reasoning across subjects from STEM, the humanities, and the social sciences. Each query contains a question and four candidate answers. We evaluate on the 153-query validation subset used by G-Designer and AgentPrune and report multiple-choice accuracy. GSM8K, MultiArith, and SVAMP. These benchmarks evaluate multi-step arithmetic reasoning over natural-language word problems. Their evaluation sets contain 1,319, 600, and 1,000 queries, respectively. Performance is measured by exact match after extracting and normalizing the final numerical answer. AQuA. AQuA contains algebraic word problems with five candidate answers. We evaluate on 254 queries and report multiple-choice accuracy based on the final selected option. HumanEval. HumanEval evaluates function-level Python code generation from a function signature and natural-language specification. Following AgentPrune, 40 problems are used for calibration and the remaining 124 problems are used for evaluation. Pass@1 is computed by executing the generated implementation against the associated unit tests. Answer extraction and scoring. We follow the answer-processing conventions of the released G-Designer evaluation implementation, which are also used by AgentPrune for the overlapping benchmarks. For MMLU, the evaluator locates the final answer statement and extracts the option label from A–D; AQuA is processed analogously with labels A–E. For GSM8K, MultiArith, and SVAMP, the parser first checks for an explicit final-answer statement or a expression and otherwise uses the last numerical expression in the response. Commas, surrounding spaces, trailing punctuation, and superficial LaTeX formatting are removed before exact comparison with the normalized reference value. For HumanEval, Markdown code fences are removed and the generated Python function is executed against the provided unit tests; a query is counted as correct only when all tests pass within the execution timeout. The same extraction and scoring pipeline is used for every topology generator and its corresponding E2-Explainer variant. Additional Evaluation Protocols Controlled candidate-graph comparison. For each query, a topology generator and its E2-Explainer variant start from exactly the same candidate graph and use the same agent roles, prompts, decoding temperature, and evaluation procedure. The reported accuracy and token results are averaged over three independent runs. Token accounting. The online communication cost includes the prompt and completion tokens of all retained agent calls and the final aggregation call during execution of the candidate or predicted subgraph. It excludes the offline edge interventions, semantic-entropy sampling, topology-optimizer training, explainer training, and the lightweight explainer forward pass. Causal ablations. Both ablations reconstruct the target graph-to-subgraph supervision and retrain the explainer. “w/o Causal” uses only the semantic signal by setting (α,β)=(0,1)(α,β)=(0,1), whereas “w/o Sem. Entropy” uses only the task-level causal contribution by setting (α,β)=(1,0)(α,β)=(1,0). Calibration-source analysis. The G-Designer-only and AgentPrune-only variants each use 40 candidate graph instances generated from the corresponding 40 calibration queries. In the mixed setting, the same 40 queries are processed by both topology generators, yielding 80 candidate graph instances in total. Agent-scale generalization. For MMLU and HumanEval, the scale-generalization experiment trains the explainer on 40 five-agent calibration graphs and directly evaluates it on six-agent graphs generated separately under a six-role G-Designer configuration. The six-agent roles are reconstructed following the role-design logic of G-Designer. The E25+N20 budget and evaluation queries are identical to those used in the main comparison. Meaning of model-agnostic. In this work, model-agnostic specifically denotes independence from the candidate topology generator. E2-Explainer does not access generator parameters or generator-specific optimization states; it operates on the candidate graph together with the task and agent-role information. Table provides a more complete view of the accuracy–efficiency trade-off controlled by the edge- and node-pruning budgets. Edge-only pruning largely preserves overall task performance, with the three settings changing average accuracy by only +0.42+0.42, +0.15+0.15, and +0.51+0.51 percentage points. However, even E75+N0 reduces weighted token usage by only 5.27%. This shows that structural sparsity alone does not translate into proportional execution savings when all agents remain active: each retained agent still performs its own reasoning, and the lengths of the remaining messages can change after the topology is modified. At the same time, the gains on AQuA and HumanEval indicate that removing low-contribution channels can suppress redundant or misleading information rather than merely reducing cost. Introducing node pruning produces substantially larger token reductions, but the benefit depends strongly on the budget. E25+N20 provides the most favorable overall trade-off, improving average accuracy from 89.63 to 90.71 while reducing weighted token usage by 25.10%. It preserves or improves performance on five of the six benchmarks, including gains of 1.09, 2.78, and 2.29 percentage points on MMLU, AQuA, and HumanEval, respectively. Increasing the node-pruning ratio to 40% further reduces token usage by 42.63%, but decreases average accuracy to 88.71, largely because HumanEval drops by 6.52 percentage points. This result suggests that aggressive node removal is particularly risky for code generation, where different roles may provide complementary implementation and verification signals. E50+N20 also removes more edges than E25+N20 but achieves both lower accuracy and a smaller token reduction. The realized cost is therefore jointly determined by node selection, edge selection, structural projection, and the lengths of the remaining messages, rather than by the nominal sparsity ratio alone. We perform the complete budget sweep only on G-Designer candidate graphs because its purpose is to characterize budget sensitivity and select one common operating point, rather than tune a different ratio for every topology generator. After selecting E25+N20, we keep the budget fixed when transferring E2-Explainer to AgentPrune, OFA-MAS, and ARG-Designer. This isolates cross-generator generalization from generator-specific budget tuning and evaluates all topology optimizers under the same compression requirement. Selecting a separate budget for each generator on the evaluation set would introduce method-specific tuning advantages and make the comparison less controlled. Qualitative Case Studies and Execution Traces We examine four E25N20 examples covering all possible transitions between correct and incorrect predictions. Table 8 summarizes their outcomes and online token usage. In every example, the original candidate graph contains six active spatial masks, whereas E25N20 retains two active spatial masks after removing one agent. The executable acyclic trace contains three same-round communication edges before refinement and one afterward. Active masks describe the selected candidate topology, while the trace records the directed messages that are actually executed after acyclic projection. Outcome transition Dataset / index Original E25N20 Online tokens Token change Correct → Correct MultiArith / 419 49 49 6,383 → 4,567 −28.5%-28.5\% Wrong → Wrong MMLU / 62 A A 6,314 → 5,550 −12.1%-12.1\% Correct → Wrong MMLU / 151 B C 11,811 → 10,216 −13.5%-13.5\% Wrong → Correct SVAMP / 415 47 3 6,059 → 4,965 −18.1%-18.1\% Table 8: Overview of the four qualitative cases. Online tokens include the prompt and completion tokens of the agent calls and final aggregation call for the corresponding query. Per-query totals are obtained from the sequential usage records within each evaluation scope. The original and E25N20 outputs are independent executions with decoding temperature 0.20.2. Accordingly, the cases illustrate observed changes in information propagation and aggregation under different topologies, rather than paired single-sample proofs that one removed element alone causes the change in correctness. Case 1: Correct → Correct with lower token usage. The MultiArith query asks how much longer a painter needs to finish 12 rooms when each room requires 7 hours and 5 rooms have already been painted. The reference answer is (12−5)×7=49(12-5)× 7=49. The original trace is 4tCg→3LK5 4tCg\!→\! 3LK5, 4tCg→3Ssf 4tCg\!→\! 3Ssf, and 3LK5→3Ssf 3LK5\!→\! 3Ssf. All four agents return 49. E25N20 removes 4tCg (Mathematical Analyst), leaving only 3LK5→3Ssf 3LK5\!→\! 3Ssf; the three retained agents and the final node still return 49. Node Observed input in the realized trace Logged output summary Original G-Designer execution 6bRH (Math Solver) Query only Computes 7 remaining rooms and returns 7×7=497× 7=49. 4tCg (Mathematical Analyst) Query only Computes total time 12×7=8412× 7=84, elapsed time 5×7=355× 7=35, and returns 84−35=4984-35=49. 3LK5 (Programming Expert) Query and the output of 4tCg Generates a program that subtracts the painted rooms and returns 49. 3Ssf (Inspector) Query and the outputs of 4tCg and 3LK5 Verifies the remaining-room calculation and returns 49. FinalRefer Final outputs of all four agents Observes four consistent solutions and returns 49. E25N20 execution 6bRH (Math Solver) Query only Returns 7×7=497× 7=49. 3LK5 (Programming Expert) Query only Generates an equivalent program and returns 49. 3Ssf (Inspector) Query and the output of 3LK5 Checks the calculation and returns 49. FinalRefer Final outputs of the three retained agents Receives three consistent answers and returns 49. Table 9: Execution trace for the correct-to-correct MultiArith case. The removed Mathematical Analyst is correct, but its derivation duplicates information already available from the other roles and is forwarded to two downstream agents. Removing this node shortens both the number of calls and the prompts received downstream. Total online usage falls from 6,383 to 4,567 tokens, a reduction of 28.5%28.5\%, while the answer remains correct. Case 2: Wrong → Wrong under a shared domain misconception. The MMLU query asks which isolated pancreatic-enzyme loss would have the most extensive effect on nutrient absorption in cystic fibrosis. The reference option is C, trypsinogen. Both executions instead select A, lipase. In the original trace, 4MEK→4BNb 4MEK\!→\! 4BNb, 4MEK→4Cwg 4MEK\!→\! 4Cwg, and 4BNb→4Cwg 4BNb\!→\! 4Cwg are executed. E25N20 removes 4MEK (Knowledgeable Expert) and retains only 4BNb→4Cwg 4BNb\!→\! 4Cwg. Node Observed input in the realized trace Logged output summary Original G-Designer execution 4MEK (Knowledgeable Expert) Query only Produces search terms covering cystic fibrosis, pancreatic function, lipase, trypsinogen, and nutrient absorption, without selecting an option. 6wZy (Critic) Query only Notes that trypsinogen activates proteolytic enzymes, but its response ends without a final option. 4BNb (Mathematician) Query and the output of 4MEK Prioritizes fat and fat-soluble-vitamin malabsorption and selects A, lipase. 3gkY (Psychologist) Query only Uses the same fat-malabsorption argument and selects A. 4Cwg (Historian) Query and the outputs of 4MEK and 4BNb Adopts 4BNb’s comparison and selects A. FinalRefer Final outputs of all five agents Returns A, which disagrees with the reference option C. E25N20 execution 6wZy (Critic) Query only Again discusses the alternatives but does not produce a complete final option. 4BNb (Mathematician) Query only Again argues that lipase has the broadest nutritional consequences and selects A. 3gkY (Psychologist) Query only Selects A using the same fat-digestion rationale. 4Cwg (Historian) Query and the output of 4BNb Repeats the lipase argument and selects A. FinalRefer Final outputs of the four retained agents Returns A again. Table 10: Execution trace for the wrong-to-wrong MMLU case. This failure is not primarily caused by one removable noisy message. The active reasoning roles share the same substantive misconception and rank the direct consequences of lipase loss above the broader downstream role of trypsinogen. Pruning reduces token usage from 6,314 to 5,550, but the retained subgraph continues to support the same incorrect answer. This case exposes a limit of topology refinement: removing redundant communication cannot repair an error already shared by the retained agents. Case 3: Correct → Wrong after removing a useful knowledge source. The MMLU query asks which worker uses a “paddy wagon,” with reference option B, police officer. The original execution returns B. Its realized trace is 4MEK→4BNb 4MEK\!→\! 4BNb, 4MEK→4Cwg 4MEK\!→\! 4Cwg, and 4BNb→4Cwg 4BNb\!→\! 4Cwg. E25N20 removes 4MEK (Knowledgeable Expert), retains only 4BNb→4Cwg 4BNb\!→\! 4Cwg, and returns C, rice farmer. Node Observed input in the realized trace Logged output summary Original G-Designer execution 4MEK (Knowledgeable Expert) Query only Supplies lexical cues linking “paddy wagon” with police officer, while also listing the competing options. 6wZy (Critic) Query only Treats the wording as regionally ambiguous but identifies B as the most likely answer. 4BNb (Mathematician) Query and the output of 4MEK Compares the police-van and agricultural readings and selects the common police-van meaning, B. 3gkY (Psychologist) Query only Independently interprets the term as a police transport vehicle and selects B. 4Cwg (Historian) Query and the outputs of 4MEK and 4BNb Follows the common slang interpretation and selects B. FinalRefer Final outputs of all five agents Returns B, matching the reference answer. E25N20 execution 6wZy (Critic) Query only Still identifies B as the best answer, although with reservations about ambiguity. 4BNb (Mathematician) Query only Reinterprets the term literally as a wagon for transporting unmilled rice and selects C. 3gkY (Psychologist) Query only Independently retains the police-van interpretation and selects B. 4Cwg (Historian) Query and the output of 4BNb Adopts the agricultural interpretation from 4BNb and selects C. FinalRefer Final outputs of the four retained agents Chooses C despite two independent agents selecting B. Table 11: Execution trace for the correct-to-wrong MMLU case. The removed node provides a useful lexical cue in the original run. After its removal, the only retained communication path propagates an incorrect literal interpretation from 4BNb to 4Cwg. The execution uses 13.5%13.5\% fewer tokens, but the connected pair reinforces option C and the final node rejects two independent correct responses. This example shows that a lower-cost topology can lose complementary knowledge and preserve a misleading information path. Case 4: Wrong → Correct after topology refinement. Figure 3 visualizes the SVAMP example in which Dave initially has 59 files and 15 apps, and retains 30 files and 12 apps after deletion. The requested quantity is the number of deleted apps, so the reference answer is 15−12=315-12=3. The original G-Designer execution returns 47, whereas E25N20 returns 3. Figure 3: Wrong-to-correct case on SVAMP index 415. The original graph contains four active agents and six active spatial masks, and its realized acyclic trace contains three same-round communication edges. E25N20 removes the Programming Expert, retains three agents and two active spatial masks, and realizes one same-round communication edge. The original realized trace is 6A7e→8fKH 6A7e\!→\! 8fKH, 6A7e→7ivC 6A7e\!→\! 7ivC, and 8fKH→7ivC 8fKH\!→\! 7ivC. E25N20 removes 8fKH (Programming Expert), after which only 6A7e→7ivC 6A7e\!→\! 7ivC is executed. Node Observed input in the realized trace Logged output summary Original G-Designer execution 46bj (Math Solver) Query only Correctly uses the app counts and returns 15−12=315-12=3. 6A7e (Mathematical Analyst) Query only Misbinds 59 files as the original number of apps and returns 59−12=4759-12=47. 8fKH (Programming Expert) Query and the output of 6A7e Sets original_apps=59 in its generated program and returns 47. 7ivC (Inspector) Query and the outputs of 6A7e and 8fKH Accepts the same variable assignment and returns 47. FinalRefer Final outputs of all four agents Selects 47 from three mutually consistent but incorrect responses and rejects the isolated answer 3. E25N20 execution 46bj (Math Solver) Query only Returns 15−12=315-12=3. 6A7e (Mathematical Analyst) Query only Correctly binds 59 and 30 to files and 15 and 12 to apps, then returns 3. 7ivC (Inspector) Query and the output of 6A7e Verifies the app-based subtraction and returns 3. FinalRefer Final outputs of the three retained agents Receives three consistent answers of 3 and returns 3. Table 12: Execution trace for the wrong-to-correct SVAMP case in Figure 3. In the original run, the incorrect variable binding is repeated along the connected path and outweighs the only correct response at aggregation. In the independent E25N20 run, all retained agents distinguish files from apps and agree on 3. Token usage also falls from 6,059 to 4,965. The case illustrates how a compact topology can coincide with both lower cost and a more reliable aggregation outcome, while the independent sampling caveat prevents attributing the correction solely to removal of 8fKH. Generalization to Hand-Crafted Communication Topologies We further examine whether the frozen explainer trained only on G-Designer-generated graphs can transfer to manually specified communication structures. We consider five representative topology families, including complete, random, layered, chain, and star graphs. E2-Explainer is directly applied to these graphs without additional causal supervision or retraining. Table shows that E2-Explainer transfers effectively beyond the generated graph distribution used for training. It improves or matches accuracy in 26 of the 30 dataset–topology combinations while reducing token usage across every topology and benchmark. The strongest gains appear on the star topology, including improvements of 1.92 points on AQuA and 2.88 points on HumanEval. The layered topology also benefits consistently, reaching 89.11 on HumanEval while reducing token usage by 19.9%. Even for the chain topology, whose sparse structure leaves less removable redundancy, the explainer largely preserves task performance and still reduces token usage on all six benchmarks. These results are notable because the explainer is calibrated exclusively on G-Designer graphs but is directly applied to fixed topologies with substantially different structural patterns. Its improvements therefore cannot be attributed solely to memorizing G-Designer-specific adjacency structures. Instead, the explainer captures transferable task- and structure-dependent cues for retaining useful communication elements. Additional Redundancy Analysis on OFA-MAS To examine whether the redundancy observed on G-Designer is specific to a particular topology designer, we repeat the same random edge-masking analysis on OFA-MAS, as shown in Figure 4. Similar to the observations on G-Designer, most random edge masks degrade performance on both MMLU and HumanEval, while only a small fraction of them preserve or slightly improve the original accuracy. This pattern suggests that OFA-MAS-generated topologies also contain redundant communication links and that task-preserving compact subgraphs do exist. Figure 4: Random edge masking on OFA-MAS-generated communication graphs for MMLU and HumanEval. The edge masking ratio ranges from 0.05 to 0.80 in increments of 0.05. Each point denotes the accuracy obtained under one retained random mask, while the dashed line marks the performance of the original unmasked OFA-MAS graph. For visualization, one extreme run is omitted at each masking ratio. The resulting distribution shows that although a few random masks preserve or even improve the original performance, most random masks degrade accuracy, indicating that task-preserving compact subgraphs exist but are difficult to identify through unguided pruning. At the same time, the rarity of these successful random masks shows that such subgraphs are difficult to identify through unguided pruning alone. In other words, redundancy does not imply that arbitrary pruning is safe. This further motivates the need for a task-conditioned explainer that selects compact communication subgraphs according to causal contribution rather than random deletion. Appendix C Algorithmic Details Causal Supervision Construction and Explainer Prediction Algorithms 1 and 2 summarize the offline construction of graph-to-subgraph supervision and the training and test-time prediction procedures, respectively. Algorithm 1 Causally Grounded Supervision Construction 1: cal,ϕ,ℬ,M,α,βD_cal,G_φ,B,M,α,β 2: Graph-to-subgraph dataset explD_expl 3: expl←∅D_expl← 4: for each x∈calx _cal do 5: G=(V,E)←ϕ(x)G=(V,E) _φ(x) 6: for each active edge e∈Ee∈ E do 7: (QGe,G,e)←Evaluate(x,G,M)(Q_G^e,Y^G,e)← Evaluate(x,G,M) 8: G−e←Intervene(G,e)G^-e← Intervene(G,e) 9: (Qe,G−e)←Evaluate(x,G−e,M)(Q_e,Y^G^-e)← Evaluate(x,G^-e,M) 10: uetask←max(0,QGe−Qe)u^task_e← (0,Q_G^e-Q_e) 11: uesem←max(0,H¯(G−e)−H¯(G,e))u^sem_e← \! (0, H(Y^G^-e)- H(Y^G,e) ) 12: u~etask←uetask u^task_e← u^task_e 13: u~esem←min(1,uesem/logM) u^sem_e← (1,u^sem_e/ M) 14: ue←clip[0,1](αu~etask+βu~esem)u_e _[0,1](α u^task_e+β u^sem_e) 15: end for 16: πG←SortDescending(E,uee∈E) _G← SortDescending(E,\u_e\_e∈ E) 17: for each budget b∈ℬb do 18: (kE,kV)←RetentionCounts(G,b)(k_E,k_V)← RetentionCounts(G,b) 19: Eb⋆←TopK(πG,kE)E_b ← TopK( _G,k_E) 20: Hb⋆←ℛvalid(G,Eb⋆,kV)H_b _valid(G,E_b ,k_V) 21: expl←expl∪(x,G,b,Hb⋆)D_expl _expl∪\(x,G,b,H_b )\ 22: end for 23: end for 24: return explD_expl Algorithm 2 Explainer Training and Test-Time Prediction 1: expl,λnode,λwdD_expl, _node, _wd 2: Trained explainer FθF_θ 3: Initialize parameters θ 4: Training 5: for each minibatch ⊆explS _expl do 6: for each (xi,Gi,b,Hi,b⋆)∈(x_i,G_i,b,H_i,b ) do 7: (i,b⋆,i,b⋆)←Indicators(Gi,Hi,b⋆)(A_i,b ,R_i,b )← Indicators(G_i,H_i,b ) 8: (^i,b,^i,b)←Fθ(xi,Gi,b)( A_i,b, R_i,b)← F_θ(x_i,G_i,b) 9: end for 10: Compute ℒedgeL_edge and ℒnodeL_node on S 11: ℒ←ℒedge+λnodeℒnode+λwd∥θ∥22L _edge+ _nodeL_node+ _wd θ _2^2 12: Update θ by minimizing ℒL 13: end for 14: Test-time prediction 15: Receive query x, graph G, and budget b 16: (^b,^b)←Fθ(x,G,b)( A_b, R_b)← F_θ(x,G,b) 17: (kE,kV)←RetentionCounts(G,b)(k_E,k_V)← RetentionCounts(G,b) 18: E^b←TopK(^b,kE) E_b← TopK( A_b,k_E) 19: V^b←TopK(^b,kV) V_b← TopK( R_b,k_V) 20: H^b←ℛvalid(G,E^b,V^b) H_b _valid(G, E_b, V_b) 21: return H^b H_b