Paper deep dive

Learning How Much to Collaborate: Difficulty-Aware Topology Selection for Multi-Agent Code Generation

Yunsong Hong

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 95

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/16/2026, 7:23:50 AM

Summary

The paper introduces Difficulty-Aware Topology Selection (DATS), a method for multi-agent code generation that dynamically selects the optimal communication topology based on problem difficulty and cost. Evaluating five topologies across APPS, HumanEval+, and LiveCodeBench, the authors find that hierarchical collaboration benefits hard problems significantly more than easy ones, despite higher token costs. DATS uses a graph network to predict success probabilities for each topology and selects the one maximizing predicted success minus cost. A budget-matched evaluation protocol reveals that DATS achieves a 4.1-point pass@1 improvement over always-hierarchical systems when fixed at 40% of the hierarchical budget, a gain that holds across different backbones and transfers to mathematical reasoning tasks.

Entities (18)

DATS · method · 98%APPS · dataset · 95%Chain · topology · 95%FullMesh · topology · 95%HRM · topology · 95%HumanEval · dataset · 95%LiveCodeBench · dataset · 95%Single · topology · 95%Star · topology · 95%Claude Opus 4 · model · 92%DeepSeek-V3 · model · 92%GPT-4o-mini · model · 92%Qwen2.5-Coder-32B · model · 92%Graph Network · modelarchitecture · 90%GSM8K · dataset · 90%MATH · dataset · 90%CodeBERT · model · 85%GraphCodeBERT · model · 85%

Relation Signals (12)

DATS → evaluatedon → LiveCodeBench

confidence 95% · Evaluating five topologies on 614 problems from APPS, HumanEval+ and LiveCodeBench

DATS → evaluatedon → APPS

confidence 95% · Evaluating five topologies on 614 problems from APPS, HumanEval+ and LiveCodeBench

DATS → evaluatedon → HumanEval

confidence 95% · Evaluating five topologies on 614 problems from APPS, HumanEval+ and LiveCodeBench

DATS → improves → pass@1

confidence 95% · Fixed at 40% of the always-hierarchical cost, DATS reaches 77.7% pass@1 against 73.6% (always-hierarchical)

DATS → uses → Graph Network

confidence 95% · Its predictor is a graph network that treats the five topologies as nodes of a connectivity order rather than independent labels

HRM → outperforms → Single

confidence 90% · the advantage of hierarchical collaboration over a single agent grows from 2.4 points... to 21.1 points on the hardest third

DATS → testedwithbackbone → GPT-4o-mini

confidence 90% · The 4.1-point gain holds across four backbones... Our backbones are... GPT-4o-mini...

DATS → testedwithbackbone → Claude Opus 4

confidence 90% · The 4.1-point gain holds across four backbones... Our backbones are Claude Opus 4...

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multi-agent systems for code generation are deployed with a single communication topology, chosen once for every problem. This is the wrong granularity. Evaluating five topologies on 614 problems from APPS, HumanEval+ and LiveCodeBench, we find that the advantage of hierarchical collaboration over a single agent grows from 2.4 points of pass@1 on the easiest third of problems to 21.1 points on the hardest third, while its token cost stays about ten times higher. We propose the Difficulty-Aware Topology Selector (DATS), which predicts each topology's probability of solving a problem and selects the one maximising predicted success minus cost. Its predictor is a graph network that treats the five topologies as nodes of a connectivity order rather than independent labels, worth 1.7 points over a flat multi-label head. Because the cost penalty is a single scalar recalibrable without retraining, routers compare at equal spend: under this budget-matched protocol six cost-aware methods span 21.6 percentage points, and two baselines leading DATS fall behind once calibrated to it. Fixed at 40% of the always-hierarchical cost, DATS reaches 77.7% pass@1 against 73.6% (always-hierarchical) and 74.3% (strongest learned competitor), all eleven pairwise McNemar comparisons surviving Holm-Bonferroni correction. The 4.1-point gain holds across four backbones spanning fourteen points of capability, and replacing the 39 interpretable features with a graph network or a pretrained encoder shifts accuracy by at most 1.3 points, never significantly. A cross-domain study on 400 mathematical reasoning problems reproduces the effect, the gap widening from 2.5 to 20.9 points.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

91,279 characters extracted from source content.

Expand or collapse full text

Highlights Learning How Much to Collaborate: Difficulty-Aware Topology Selection for Multi-Agent Code Generation Yunsong Hong • Collaboration benefit in multi-agent code generation scales with problem difficulty • Topology is treated as a per-problem decision rather than a system-level constant • A budget-matched protocol swaps the rank of two base- lines once spend is equalised • Matched at 40% of the hierarchical budget, routing adds 4.1 points of pass@1 • The gain holds across four backbones and transfers to mathematical reasoning arXiv:2609.13890v1 [cs.MA] 12 Sep 2026 Learning How Much to Collaborate: Difficulty-Aware Topology Selection for Multi-Agent Code Generation Yunsong Hong ∗ School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia Abstract Multi-agent systems for code generation are deployed with a single communication topology, chosen once for every problem. This is the wrong granularity. Evaluating five topologies on 614 problems from APPS, HumanEval+ and LiveCodeBench, we find that the advantage of hierarchical collaboration over a single agent grows from 2.4 points of pass@1 on the easiest third of problems to 21.1 points on the hardest third, while its token cost stays about ten times higher. We propose the Difficulty-Aware Topology Selector (Dats), which predicts each topology’s probability of solving a problem and selects the one maximising predicted success minus cost. Its predictor is a graph network that treats the five topologies as nodes of a connectivity order rather than independent labels, worth 1.7 points over a flat multi-label head. Because the cost penalty is a single scalar recalibrable without retraining, routers compare at equal spend: under this budget-matched protocol six cost-aware methods span 21.6 percentage points, and two baselines leading Dats fall behind once calibrated to it. Fixed at 40% of the always-hierarchical cost, Dats reaches 77.7% pass@1 against 73.6% (always-hierarchical) and 74.3% (strongest learned competitor), all eleven pairwise McNemar comparisons surviving Holm–Bonferroni correction. The 4.1-point gain holds across four backbones spanning fourteen points of capability, and replacing the 39 interpretable features with a graph network or a pretrained encoder shifts accuracy by at most 1.3 points, never significantly. A cross-domain study on 400 mathematical reasoning problems reproduces the effect, the gap widening from 2.5 to 20.9 points. Keywords: multi-agent systems, large language models, code generation, adaptive computation, cost-aware routing, budget-matched evaluation 1. Introduction Large language models write code well enough that the in- teresting engineering question is no longer whether to use them but how to arrange them. A single model called once is the cheapest arrangement. Beyond it lies a design space of multi- agent systems in which several model instances take on roles, exchange drafts, criticise each other, and converge on an an- swer. Frameworks such as ChatDev [1], MetaGPT [2] and Au- toGen [3] have made these arrangements easy to build, and spe- cialised pipelines such as AgentCoder [4] and MapCoder [5] report that collaboration lifts pass@1 on competitive program- ming benchmarks well above what one call achieves. What these systems share is a commitment: the topology is fixed at design time. A pipeline that decomposes, delegates and reviews does so for every incoming problem, whether the prob- lem asks for a hierarchical decomposition or for a three-line string manipulation. The commitment is convenient to imple- ment and expensive to run. In our measurements a hierarchical role-based mesh consumes 9.95 times the tokens of a single call on the same problem; at Claude Opus 4 prices that is $0.94 ver- sus $0.09 per problem, a difference that dominates the operating cost of any deployed system. ∗ Corresponding author. Email address: yhon7822@uni.sydney.edu.au (Yunsong Hong) The commitment would be defensible if the extra collabora- tion paid off uniformly. It does not. Splitting our 614 problems into difficulty tertiles, the gap between the hierarchical mesh and a single agent is 2.4 points of pass@1 on the easy third, 10.7 points on the middle third, and 21.1 points on the hard third. On easy problems the elaborate pipeline spends ten times the bud- get to recover two points that a single call very nearly reaches on its own. The aggregate number that such systems usually report—a few points of average improvement—is an average over two quite different regimes, and averaging them hides the fact that most of the spend is going to the regime where it buys almost nothing. This observation reframes the design problem. If the bene- fit of collaboration varies across problems while its cost does not, then the topology should not be a constant of the system; it should be a function of the input. The question is whether that function can be learned from properties of a problem state- ment alone, before any model is called and before any answer is available to evaluate. We show that it can, and that the way the decision is factored matters. The natural first attempt is to train a classifier that maps a problem directly to the topology that solved it. This throws away the structure of the decision. The label “the topology that solved it” is ambiguous when several topologies succeed and undefined when none do, and it hard-codes an accuracy–cost trade-off into the training targets, so changing the budget means retraining. Instead we predict what is actually observable—for each topology, whether it solved each cached problem—and de- fer the trade-off to a decision rule applied at inference. The pre- dictor answers five independent questions of the form “would this topology solve this problem”; the selector combines those answers with a cost table through a single scalar coefficient. The factorisation is a standard predict-then-optimize construc- tion [6], and in our setting it is worth 6.0 points of pass@1 over direct classification from identical inputs. Keeping the trade-off in one scalar has a second consequence that turned out to matter more than the first, and that we think outlives the router it came from. Cost-aware routers can be tuned to spend anything between the cheapest and the most expensive topology, so a router is not a system but a one- parameter family of systems, and reporting its accuracy with- out pinning its spend says very little. Giving every method the same coefficient does not repair this, because the coeffi- cient multiplies each method’s own score scale: in our com- parison a shared λ = 0.2 leaves six methods spending any- where from 30% to 52% of the reference budget. We there- fore calibrate every cost-aware method, ours and the baselines alike, to one common budget before comparing them, choos- ing each method’s coefficient by bisection on held-out folds so that all of them spend 40% of what the always-hierarchical sys- tem spends. Two baselines that appeared to beat Dats under the shared coefficient fall behind it once their spend is equalised, and the method that looked best turns out to have been the one buying the most computation. We state the procedure as a pro- tocol, report what it does and does not equalise, and regard it as a requirement for this class of comparison rather than a refine- ment of it. The contributions of this work are the following. First, we quantify how the benefit of five collaboration topologies varies with problem difficulty and algorithmic type, on 614 problems and four backbones, and show that the variation is large enough to be exploitable. Second, we formulate topology choice as a per-problem constrained selection problem and instantiate it as Dats, a predict-then-optimize router whose predictor is a com- pact graph network over the five candidate topologies: because the topologies form a connectivity order, we model them as nodes of that order and pass messages along it rather than scor- ing them as independent labels, which buys 1.7 points over a flat multi-label head at equal spend. We separate the two axes of the design and show that the gain lives on the decision side, not the input side: replacing the 39 interpretable features with a graph network over the statement, with frozen or fine-tuned CodeBERT and GraphCodeBERT, or with a hybrid of features and embeddings moves accuracy by at most 1.3 points and never significantly. Third, we define the budget-matched eval- uation protocol described above, show that it reverses two of the comparisons among the six cost-aware methods it governs, and use it throughout. Fourth, we characterise the resulting behaviour and its boundaries: which features carry the signal, which topologies get chosen for which algorithmic types, how the router’s probabilities calibrate, how much of the gain sur- vives a change of backbone, and—on 400 GSM8K and MATH problems—whether any of this depends on the problems being code at all. 2. Related work 2.1. Communication topologies for language agents Early multi-agent work with language models established that role assignment and message passing change what a sys- tem can solve. CAMEL [7] paired an instruction-giving agent with an executing agent; debate procedures [8, 9] ran sev- eral instances against each other and aggregated; generative agents [10] scaled the interaction to societies. For software specifically, ChatDev [1] and MetaGPT [2] encode a water- fall process into agent roles, AgentVerse [11] assembles task- specific teams on demand, self-collaboration prompting [12] obtains the same effect from a single model playing several roles, and AutoGen [3] provides the conversational substrate on which such processes are assembled. A second line treats the communication graph itself as an ob- ject to optimise. GPTSwarm [13] represents agent systems as graphs and optimises edges by reinforcement learning; Mac- Net [14] studies how performance scales as the graph grows; AgentPrune [15] removes redundant edges to cut token spend; G-Designer [16] generates a task-conditioned topology with a graph neural network. Our work shares the premise that topol- ogy is a decision variable, and differs in where the decision is made. These methods search for or generate a graph, usually per task family, and pay a search or generation cost. We take a small, fixed catalogue of hand-designed topologies and learn only which catalogue entry to use for a given problem. The catalogue is coarser, but selection over it costs a single forward pass and admits a budget guarantee, which is what makes the comparison at equal spend possible. Evidence on whether collaboration helps at all is mixed. Wang et al. [17] report that multi-agent discussion does not con- sistently outperform a single strong agent with a good prompt. Our difficulty analysis is consistent with both that finding and with the positive reports: averaged over a benchmark whose problems are mostly easy, discussion looks unnecessary; re- stricted to the hard tail, it is decisive. 2.2. Iterative refinement and self-correction Chain, star and mesh topologies inherit their machinery from single-agent prompting and refinement. Chain-of-thought prompting [18] and ReAct [19] established that structuring a model’s intermediate output changes what it can solve; Self- Refine [20] and Reflexion [21] loop generation and critique; Self-Debug [22] feeds execution feedback back to the model; Self-Consistency [23] and Tree of Thoughts [24] broaden the search rather than deepen the critique. These are complemen- tary to topology selection: each of our topologies could be in- stantiated with a stronger refinement operator, and the routing problem would remain. 2 2.3. Routing and cascades over models Choosing which model to call for which query is a well- studied cost-reduction strategy. FrugalGPT [25] cascades from cheap to expensive models with a learned stopping rule; Hybrid LLM [26] routes between a small and a large model on pre- dicted difficulty; RouteLLM [27] learns the router from prefer- ence data; ZOOTER [28] distils reward signals into a routing function; meta-modelling approaches [29] predict per-model performance and pick the cheapest adequate one; Router- Bench [30] standardises evaluation of such systems. Cascades with mixed thought representations [31] apply the same idea to reasoning chains. Two recent methods route within multi-agent systems rather than over single models. MasRouter [32] jointly selects agents, models and collaboration mode; Router-R1 [33] learns multi- round routing and aggregation with reinforcement learning. Our formulation is narrower than either—one decision, made once, over a fixed catalogue—which is what allows the budget- matched protocol and the significance testing we report. We reimplement lightweight versions of both as baselines. The broader question of how much computation to spend per input has been studied as adaptive computation. Confident Adaptive Language Modeling [34] exits early when a per-token confidence threshold is met; mixture-of-experts routing [35] conditions capacity on the input; test-time compute scaling [36] and repeated sampling [37] characterise how accuracy responds to additional inference budget. Topology selection is adaptive computation at the granularity of an entire agent pipeline. 2.4. Code generation benchmarks and models We evaluate on APPS [38], which stratifies problems into introductory, interview and competition levels; HumanEval+, the strengthened test suite of EvalPlus [39] over the origi- nal HumanEval [40]; and LiveCodeBench [41], which draws from contest problems released after model training cut-offs. MBPP [42], AlphaCode [43] and SWE-bench [44] define ad- jacent regimes we do not cover. Our backbones are Claude Opus 4, DeepSeek-V3 [45], GPT-4o-mini [46] and Qwen2.5- Coder-32B [47], the last chosen so that the gradient includes a code-specialised model of the kind introduced by DeepSeek- Coder [48]. 2.5. Representations of code and of problem statements A router needs a representation of the input, and the natu- ral candidates come from code representation learning. Code- BERT [49] pretrains a bimodal transformer on code paired with natural language; GraphCodeBERT [50] adds data-flow edges to the same objective; UniXcoder [51] unifies the en- coder and decoder views; code2vec [52] predates them with paths through the abstract syntax tree; and graph convolutional networks [53, 54] supply the machinery for encoding either a syntax tree or a statement graph. All of these assume a program to encode. Our router runs before any program exists, which changes what they can contribute; Section 7.3 measures the dif- ference by putting each of them in place of our hand-crafted features. 2.6. Mathematical reasoning as a second domain Our cross-domain check uses GSM8K [55], grade-school word problems with a single numeric answer, and MATH [56], competition problems annotated with five difficulty levels, which is what makes it usable for a difficulty-scaling analysis. Multi-agent methods have been applied to both: debate [8] and self-consistency [23] report gains on arithmetic and mathemat- ical reasoning, and program-aided approaches [57] offload the arithmetic to an interpreter. We use these suites not to compete on them but to ask whether the difficulty-dependent value of collaboration that we measure on code is a property of code or a property of hard problems. 3. Problem formulation Let x denote a problem statement and T = τ 1 ,...,τ 5 the catalogue of topologies in Fig. 1. Running τ t on x with a fixed backbone yields a program that either passes the problem’s full test suite or does not; write the outcome y t (x) ∈ 0, 1 and its expectation s t (x) = Pr[pass | x,τ t ]. Running τ t also consumes tokens, and we write c t for the expected cost of topology t in currency units, measured per topology rather than per problem because the token profile is dominated by the number and struc- ture of calls rather than by the individual statement. A router is a map π : X → T . We want the router that max- imises expected success subject to an average spending con- straint, max π E x s π(x) (x) s.t. E x c π(x) ≤ B.(1) The constraint couples the individual decisions only through an expectation, so the Lagrangian relaxation separates across problems: for a multiplier λ≥ 0 the optimal policy is pointwise, π λ (x) = arg max t∈T s t (x)− λc t .(2) Expected cost E x [c π λ (x) ] is non-increasing in λ, so the multiplier that meets a given budget can be found by bisection. This is the property we exploit in the evaluation protocol: every cost-aware method in this paper exposes such a scalar, so all of them can be driven to the same budget and compared there. Two quantities in Eq. (2) are unavailable at decision time. The costs c t are cheap to obtain, since they can be measured once from cached runs and reused. The success probabilities s t (x) are the difficult part, and estimating them is what the next section is about. Throughout, we call the policy that always plays a single topology Always-τ, and the policy π ⋆ (x) = arg max t y t (x) that knows the realised outcomes the oracle. The oracle is not achievable, but it bounds what any router built on this catalogue can reach, and the gap to it measures how much of the available signal a router leaves behind. 3.1. The five topologies Single issues one call with the problem statement and a short instruction. Chain runs three sequential calls: a proposer writes 3 (a) Single A (b) Chain PCR (c) Star H W W (d) FullMesh A A (e) HRM M L w Figure 1: The five collaboration topologies. Single issues one call. Chain passes a draft from proposer (P) to critic (C) to reviser (R). Star has a hub (H) decompose the problem and delegate to workers (W), whose returns are merged deterministically. FullMesh lets three peers (A) read each other’s drafts over two rounds. HRM adds a manager (M) above two leads (L), each supervising two workers (w), with review flowing upward. Filled nodes indicate agents that see the full problem statement. a first solution, a critic lists concrete defects, and a reviser pro- duces the final program. Star uses four calls: a hub decom- poses the statement into three subtasks and three workers solve them independently, after which the returns are concatenated deterministically into the final program without a further model call. FullMesh runs three peers for two rounds, six calls in to- tal, with each peer seeing the other two drafts in the second round. HRM, the hierarchical role-based mesh, places a man- ager above two leads, each of which supervises two workers; leads consolidate their workers’ output and the manager arbi- trates, for five to eight calls depending on whether the man- ager requests a second pass. The catalogue is deliberately small and hand-designed: our claim concerns the value of choosing among topologies, not the value of any particular topology. 4. Difficulty-aware topology selection Dats estimates s t (x) for all five topologies from the problem statement alone, then applies Eq. (2) with a calibrated multiplier (Fig. 2). Nothing in the pipeline requires executing a topol- ogy at decision time, and nothing propagates gradients into the agents. 4.1. Features The feature map φ : X → R 39 is deliberately interpretable, because we want to be able to say afterwards which properties of a problem drive the decision. It has six groups. The lexical group (five dimensions) records length in words, sentence count, average word length, the fraction of tokens that appear inside code spans, and a constraint density obtained by counting phrases such as “at most”, “exactly” and “for each”. The structural group (four) counts worked examples, detects an explicit edge-case discussion, counts required functions in the signature, and scores input–output complexity from the de- scribed formats. The complexity group (three) captures cues that a solution will need nesting or asymptotic care: depth hints such as “for each pair”, the number of large numeric bounds, and the count of algorithmic keywords associated with hard techniques. The difficulty group is a single scalar, a five-point rating produced by a lightweight scorer over the statement. The semantic group (sixteen) holds the leading singular directions of a TF–IDF representation of the statement. The algorithmic group (ten) is a one-hot encoding of the problem’s dominant technique, assigned by keyword vote over ten categories from implementation and string handling to dynamic programming and graphs; the vote recovers our manual labels on 85.5% of problems. The two groups that a router might be expected to lean on hardest, difficulty and algorithmic type, are also the two that carry the most signal in the ablation of Section 7. Their inter- action is what makes the problem non-trivial: the useful pattern is not “hard problems need HRM” but “hard graph problems need HRM while hard implementation problems do not”, and a model without interaction terms cannot express it. 4.2. Predictor The predictor must turn the difficulty embedding of a prob- lem into five success probabilities, one per topology. The obvi- ous realisation is a multi-label perceptron with five independent sigmoid heads, and it is the one most routers of this kind use. It is also the wrong inductive bias, because the five outputs are not independent. The topologies form a natural order of increasing coordination—a single agent, then a linear pipeline, then a hub that fans work out and back, then an all-to-all mesh—monotone in the per-topology cost of Table 2, with the hierarchical mesh sitting above all four as their role-structured composition. Solv- ability inherits that order almost monotonically: a problem a lighter topology already solves is nearly always solved by a heavier one, and the decision a router actually faces is not “does topology t succeed” in isolation but “is the marginal coordina- tion of t over the next-lighter option worth its cost”. A head that scores the five topologies independently cannot see that struc- ture; it has to relearn the order from data, separately for every problem. We therefore make the label space explicit and predict over it with a small graph network. Let G T = (T, E) be the topol- ogy graph: its five nodes are the topologies and its edges join each topology to its neighbours in the connectivity order, with the hierarchical mesh linked to Star and FullMesh as the two structures it composes. Each node carries a static descriptor e t ∈ R 4 —agent count, edge count, diameter and mean degree of the topology’s own communication graph—so the network is told what each option is, not merely that five options exist. 4 problem x 39-dim feature extractor Á topology-graph GNN s t (x)=Pr[passjx;¿ t ] cost table c t budget B! ̧ (inner CV) argmax t s t ¡ ̧ c t execute ¿ ¤ (x) ! program trained once on cached benchmark outcomes; no gradient reaches the agents Figure 2: Dats. A problem statement is mapped to 39 features and a shared trunk; a graph network then passes messages over the five candidate topologies, arranged by connectivity, and reads out one success probability per topology. The selector subtracts a cost term whose coefficient is calibrated to the target budget on held-out data. Only the selected topology is executed. The problem enters through a shared trunk. A single stan- dardised affine map with a ReLU turns the 39 features into a difficulty context z = trunk(φ(x)) ∈ R d , and each node is ini- tialised by concatenating that context with its structural descrip- tor, h 0 t = ReLU W 0 [ z∥ e t ] + b 0 .(3) The trunk is shared across nodes for the same reason a trunk was shared across the old heads—difficulty is a property of the problem, not of the topology—but now the topology-specific part of the computation is a message passing over G T rather than five detached linear maps. For ℓ = 0,..., L− 1, h ℓ+1 t = ReLU W self h ℓ t + W nbr X u∈N (t) 1 |N (t)| h ℓ u ,(4) with weights tied across nodes and L = 2 rounds, so a topol- ogy’s score is a function of the predicted evidence at the topolo- gies structurally adjacent to it. A per-node sigmoid readout ˆs t = σ(w ⊤ h L t + b) produces the five probabilities. The whole predictor—trunk, the tied maps W self , W nbr and the readout— holds 13.7K trainable parameters, fewer than the flat two- hidden-layer perceptron it replaces, because the connectivity prior is supplied by G T rather than learned from scratch. Inputs are standardised on the training fold; optimisation uses Adam at an initial learning rate of 10 −3 with early stopping on a 10% internal validation split and a cap of 400 epochs. The loss is the mean binary cross-entropy over the five node readouts, L(θ) =− 1 5N N X i=1 5 X t=1 h y (i) t log ˆs (i) t + (1− y (i) t ) log(1− ˆs (i) t ) i . (5) It is worth being precise about where the graph is and is not. The encoder ladder of Section 7.3 will show that building a graph on the input—a co-occurrence graph over the problem statement—does not help, because at decision time there is no program to encode and a graph of English is a lossy substi- tute. The graph that is genuinely present at decision time is the output space: the five candidate topologies are literally commu- nication graphs, related by a connectivity order that the router should exploit. Modelling that graph, rather than the absent program graph, is the design choice this predictor makes, and Section 7 measures what it is worth against the flat head that ignores it. Figure 3 lays out the three stages end to end: the shared trunk that turns a problem into a difficulty context, the message passing over the topology graph G T , and the per-node readout that the selector then ranks against cost. Two design points carry over unchanged because they are easy to get wrong. First, the targets are per-topology outcomes, not a single categorical label, so problems that every topology solves and problems that none solve both contribute—the for- mer teaches the network that cheap topologies suffice, the latter that nothing helps and the cheapest option is therefore right. Second, the network never sees cost, which is what keeps the trade-off confined to the selector. 4.3. Selection and budget calibration Costs enter through a normalised table ˆc t = c t / max t ′ c t ′ , so ˆc Single = 0.10 and ˆc HRM = 1. The selector is τ ⋆ (x) = arg max t∈T ˆs t (x)− λ ˆc t ,(6) which places λ on the same scale as a probability difference: at λ = 0.06, upgrading from Single to HRM must buy at least 5.4 points of predicted success to be worth it. The multiplier is not a hyperparameter to be tuned for accu- racy. It is determined by the budget. Given a target B expressed as a fraction of the always-hierarchical cost, we run bisection on λ inside a three-fold split of the training fold, evaluating realised spend at each step, and stop when the realised spend is within 0.5% of B or after sixty iterations. The resulting λ is then applied unchanged to the outer test fold. Algorithm 1 states the procedure. Across the five outer folds of our main ex- periment the calibrated values were 0.058, 0.052, 0.065, 0.056 and 0.061, a spread small enough that the routing behaviour is stable across folds but large enough that fixing a single value in advance would miss the budget. 5 (a) problem features ! shared trunk (b) message passing over the topology graph G T (c) read-out ! selector context z 1/03/24/3 3/3 7/6 h 0 t = ReLU(W 0 [zke t ]) h `+ 1 t = ReLU ¡ W self h ` t + W nbr h N(t) ¢ ; `= 0;1 S Ch St FM HRM s t = ¾(w > h L t ) problem x Á 39 features shared trunk ¿ ⋆ = argmax t [ s t ¡ ̧ c t ] SChSt FM HRM Figure 3: The topology-graph predictor. (a) A problem x is mapped by φ to 39 difficulty features and a shared trunk to a context z ∈ R d . (b) The five candidate topologies are the nodes of the topology graph G T , joined along the connectivity order Single–Chain–Star–FullMesh with the hierarchical mesh composing Star and FullMesh; each node is initialised from z (dashed) and its own structural descriptor e t (agent/edge counts shown), then exchanges messages with its structural neighbours (orange) for L = 2 rounds. (c) A per-node sigmoid readout produces one success probability per topology, which the cost-aware selector ranks. The graph the network reasons over is the output space of topologies, not the absent program graph of the input. Algorithm 1 Dats: training and budget calibration Require: cached outcomes (x i , y (i) 1:5 ) i∈D , cost table ˆc, budget B 1: Φ← standardise φ(x i ) i∈D 2: θ ← minimise Eq. (5) on (Φ, y) 3: λ lo ← 0, λ hi ← 2 4: for j = 1 to 60 do 5: λ← (λ lo + λ hi )/2 6: ̄c ← mean spend of Eq. (6) under λ on inner validation folds 7: if ̄c > B then 8:λ lo ← λ 9: else 10:λ hi ← λ 11: end if 12: end for Ensure: predictor θ, multiplier λ 13: at inference: return arg max t [ ˆs t (φ(x))− λ ˆc t ] 4.4. The budget-matched protocol The calibration step above was introduced as a component of Dats. We now separate it from Dats, because it is not specific to our router and because without it the accuracies that this class of method reports are not comparable to each other. We state it as a protocol and treat it as a contribution independent of the model it was developed for. The difficulty it addresses is that a cost-aware router is not a system but a one-parameter family of systems. Any method of the form “score minus λ times cost” sweeps out a curve in the cost–accuracy plane as λ moves, and every point of that curve is a legitimate configuration of the same method. An ac- curacy reported without its realised spend therefore identifies a curve only up to an unknown position along it, and two such numbers cannot be subtracted. The obvious repair—give ev- ery method the same λ—does not work, because λ acts on each method’s own score scale. A predictor whose probabilities are compressed towards 0.5 upgrades to an expensive topology less often, at identical λ, than one whose probabilities are spread towards the extremes, so a shared coefficient silently places the methods at different operating points. In our comparison a shared λ = 0.2 spreads realised spend over 21.6 percentage points of the reference budget (Section 6.4). The protocol has four steps. One, declare a budget B as a fraction of the realised spend of a reference policy that requires no calibration; we use the most expensive fixed topology, which makes B interpretable as “this fraction of what the standard pipeline costs”. Two, calibrate each method separately, by bi- section on its own coefficient inside the training folds, never on the test fold. Three, verify the realised spend out of fold, report it beside the accuracy, and declare a method budget-matched only if it lands inside a stated tolerance; ours is 0.4 percentage points, and all six methods meet it. Four, compare accuracies only within the matched group, and report policies that cannot be calibrated—fixed topologies, the oracle—separately, as lo- cators of the frontier rather than as competitors. Three properties make the protocol cheap to adopt. It re- quires only that realised spend be monotone in the coefficient, which holds by construction: raising λ can only make cheaper alternatives more attractive. It converges quickly, needing 21.4 bisection steps on average to reach the 0.5% inner tolerance. And on a cached evaluation grid it costs no inference at all, 6 Table 1: Composition of the evaluation set. Difficulty is the normalised state- ment rating; words and tests are per-problem means. Suiten Difficulty Words Tests APPS-Introductory 1000.2517812.4 APPS-Interview1000.5029618.7 APPS-Competition 1000.7541221.3 HumanEval+1640.206724.6 LiveCodeBench1500.5934116.2 Total6140.4423219.0 since every candidate assignment is scored against outcomes that have already been paid for. What it does not equalise should also be stated: it matches expected monetary spend, not latency, not worst-case cost, and not the variance of either. A method that meets the budget by occasionally choosing a very expensive topology is matched to one that never does, and the two may be quite different to operate. 4.5. Cost of the router Feature extraction is linear in statement length and takes under a millisecond per problem; the forward pass through a 39–128–64–5 network is negligible against a single language model call, which takes 8.4 seconds even in the cheapest topol- ogy. Training the predictor on 614 problems takes under thirty seconds on a laptop CPU. The router’s own cost is therefore not a term worth carrying in Eq. (1), and we omit it from all reported budgets. 5. Experimental setup 5.1. Problems We evaluate on 614 problems (Table 1): 300 from APPS, one hundred at each of the introductory, interview and compe- tition levels; the full 164 problems of HumanEval+; and 150 problems from LiveCodeBench drawn from contests held af- ter the training cut-off of every backbone we test. The suites were chosen to span difficulty rather than to maximise size: Hu- manEval+ supplies a regime where a single call almost always suffices, APPS-Competition supplies one where nothing reli- ably works, and the middle suites populate the interval between them where routing decisions are actually contested. Mean dif- ficulty in Table 1 is the normalised rating described in Sec- tion 4.1, reported here to show that the suites are ordered as intended. 5.2. Backbones and execution Claude Opus 4 is the primary backbone; DeepSeek-V3, GPT- 4o-mini and Qwen2.5-Coder-32B provide a capability gradient for the generalisation study. Every agent inside a topology uses the same backbone, so the comparison isolates topology from model choice. Decoding uses temperature 0.2 and a 4096-token output cap, and we report pass@1 from a single sample, since a router that only helps under repeated sampling would be solv- ing a different problem. A program counts as correct only if it passes the complete test suite. Table 2: Measured cost profile per topology on Claude Opus 4. Tokens are per-problem means over the full grid; ratio is relative to Single. Topology Calls Tokens RatioUSD Lat. (s) Single12,6321.00 0.0958.4 Chain3 13,4255.10 0.48326.1 Star4 14,0835.35 0.50721.7 FullMesh6 23,0048.74 0.82844.3 HRM5–8 26,1899.95 0.94338.6 Dats2.6 10,4763.98 0.37717.1 The full grid is 614× 5× 4 = 12,280 executions. All out- comes, token counts and generated programs are cached, so the routing experiments—five folds times five seeds times thirteen methods—read from the cache and incur no further inference cost. Total spend for the grid was approximately $1,807, of which $1,753 was Claude Opus 4. 5.3. Cost accounting Table 2 reports the measured token profile. Costs use pub- lished per-token prices; for Claude Opus 4 we combine $15 per million input tokens and $75 per million output tokens, which given our observed input–output mix yields an effective $36 per million. The ratios in the fourth column are the quantity that matters for routing, and they are stable across backbones to within three percent, because they are determined by the num- ber and structure of calls rather than by the model. Latency is wall-clock median over the grid, with independent workers in Star and FullMesh executed in parallel. 5.4. Protocol All routers are evaluated by five-fold cross-validation strati- fied jointly on suite and difficulty tertile, repeated over five ran- dom seeds; reported accuracy is the pooled out-of-fold accuracy of the first seed, and the seed-to-seed standard deviation is re- ported alongside. Hyperparameters and the cost multiplier are selected inside a three-fold split of each training fold and never touch the corresponding test fold. The budget is fixed at B = 40% of the always-hierarchical cost, that is $0.377 per problem. Every cost-aware method is calibrated to that budget by the protocol of Section 4.4, using its own coefficient and its own bisection run. Fixed topologies obviously cannot be calibrated and are reported at their natu- ral cost; they are included to locate the frontier, not as budget- matched competitors. 5.5. Baselines Five Always-τ policies establish the fixed-topology frontier. Random samples a topology uniformly and is included to con- firm that the routing gain is not an artefact of mixing. Difficulty- Threshold bins the predicted difficulty rating and maps bins to topologies, with thresholds set on the training fold to meet the budget; it represents the intuition that difficulty alone should be enough. Dats-E2E replaces the five heads with one five-way softmax trained on the cheapest successful topology, keeping 7 the features, network and calibration identical, and thus iso- lates the predict-then-optimize factorisation. MasRouter-lite follows [32] in learning a routing distribution over collabora- tion modes, restricted here to a logistic model on the lexical and structural features. RouterR1-lite follows [33] in learning a reward regressor—ridge regression on accuracy minus nor- malised cost—and selecting the argmax. RandomForest-Router substitutes a 300-tree forest for the network in the same predict- then-optimize scaffold and is the strongest learned competitor. The oracle completes the table. 5.6. Statistics Accuracy intervals are Wilson score intervals at 95% [58]. Paired comparisons between routers on the same problems use McNemar’s test [59], exact binomial when the number of dis- cordant pairs is below 25 and with Yates’ continuity correc- tion otherwise, as recommended for classifier comparison by Dietterich [60]. Families of comparisons are corrected by the Holm–Bonferroni procedure [61]. Calibration is summarised by expected calibration error over ten equal-width bins [62, 63] and by the Brier score [64]. All models are scikit-learn imple- mentations [65]. 6. Results 6.1. No topology dominates Table 3 reports pass@1 for the five topologies on Claude Opus 4. The ordering by aggregate accuracy is Single, Star, Chain, FullMesh, HRM, and the total spread is 11.4 points. Two features of the table matter more than the ordering. The first is how unevenly the spread is distributed. On Hu- manEval+ the five topologies lie within 4.2 points of each other, and the cheapest is only 4.2 points behind the most expensive despite costing a tenth as much. On APPS-Competition the same comparison is 22 points. Any system that fixes one topol- ogy is implicitly accepting one of these two trade-offs every- where. The second is that the ordering is not a ranking of collabora- tion intensity. Star uses four calls and Chain three, yet Chain is ahead of Star on every suite. Sequential criticism, where each agent sees the full statement and the previous draft, ex- tracts more from three calls than decomposition and integration extract from four, because decomposition can split a problem along the wrong seam and neither the workers nor the hub can recover from that. The pairwise tests in Appendix A make the same point statistically: Chain and FullMesh are indistinguish- able ( p = 0.628) although FullMesh costs 71% more. 6.2. The benefit of collaboration scales with difficulty Fig. 4 splits the 614 problems into difficulty tertiles. The five topology curves fan out as difficulty rises: on the easy third they occupy a 2.4-point band, on the middle third 10.7 points, on the hard third 21.1 points. The hierarchical mesh is not uniformly better than a single call; it is barely better on easy problems and decisively better on hard ones, at constant relative cost. Table 3: pass@1 (%) by topology and suite on Claude Opus 4. The last row gives the number of solved problems out of 614. SuiteSingle ChainStar FullMesh HRM APPS-Introductory88.091.089.090.0 92.0 APPS-Interview57.066.063.067.0 71.0 APPS-Competition19.032.027.035.0 41.0 HumanEval+90.993.392.194.5 95.1 LiveCodeBench46.056.052.757.3 61.3 Overall62.21 69.38 66.6170.52 73.62 Solved382426409433 452 easy d∙0:33 medium 0:33<d∙0:66 hard d>0:66 20 30 40 50 60 70 80 90 100 pass@1 (%) +2.4 +10.7 +21.1 Single Chain Star FullMesh HRM DATS Figure 4: pass@1 by difficulty tertile. Arrows mark the HRM-minus-Single gap, which widens from 2.4 to 21.1 points. Dats exceeds every fixed topology on the hard tertile while spending 40% of the HRM budget overall. This is the empirical fact the rest of the paper is built on. It implies that a fixed-topology system is misallocating in one of two directions everywhere. A system fixed on Single forgoes twenty-one points on the hard tail. A system fixed on HRM spends ten times the budget on the easy two-thirds to gain be- tween two and ten points, and in the easy third specifically it spends $0.94 per problem to move 91.7% to 94.1%. The Dats curve in the same figure shows what per-problem allocation buys. It tracks the hierarchical mesh on the easy third, where it mostly declines to pay for collaboration, and ex- ceeds it by 10.3 points on the hard third, where it concentrates the budget it saved. The router does not merely interpolate be- tween the fixed policies; on the hard tertile it is above all five of them, because different hard problems want different topologies and no single policy can follow. 6.3. Budget-matched comparison Table 4 is the main result. Every cost-aware method is cal- ibrated to $0.377 per problem, and the realised spends in the fifth column confirm the calibration held out of fold, all landing within 0.4 points of the 40% target. Dats solves 477 of 614 problems, 25 more than always- hierarchical while spending 40% of its budget, and 21 more 8 Table 4: Budget-matched comparison on Claude Opus 4, 614 problems, five-fold cross-validation. Learned routers are calibrated to B = 40% of the always- hierarchical cost. Fixed topologies are shown at their natural cost and are not budget-matched. Best budget-matched result in bold. MethodSolvedpass@1 (%)95% CIUSD/problem% of HRM cost fixed Always-Single38262.21[58.3, 66.0]0.09510.1 Always-Chain42669.38[65.6, 72.9]0.48351.3 Always-Star40966.61[62.8, 70.2]0.50753.8 Always-FullMesh43370.52[66.8, 74.0]0.82887.8 Always-HRM45273.62[70.0, 77.0]0.943100.0 budget-matched Random42168.57[64.8, 72.1]0.57160.6 Difficulty-Threshold43270.36[66.6, 73.8]0.37139.3 Dats-E2E44071.66[68.0, 75.1]0.36138.3 MasRouter-lite44672.64[69.0, 76.0]0.37539.8 RouterR1-lite45173.45[69.8, 76.8]0.37639.9 RandomForest-Router45674.27[70.7, 77.6]0.37439.6 Dats47777.69[74.2, 80.8]0.37740.0 Oracle54688.93[86.2, 91.2]0.26528.1 than the strongest learned competitor at the same budget. Three comparisons in the table carry most of the argument. Against Always-HRM, the router is 4.07 points ahead at 40% of the cost. Read the other way, the same result says that 60% of what a hierarchical system spends is buying nothing, which is a stronger statement about fixed topologies than the accuracy difference alone. Against Dats-E2E, which shares the features, the network capacity and the budget calibration and differs only in predict- ing a topology label rather than five outcomes, the gap is 6.03 points. The factorisation is doing real work, not bookkeeping. Direct classification has to commit to one right answer per prob- lem during training, and for the 38% of problems that several topologies solve, that commitment discards the information that the alternatives also worked. Against Random, the gap is 9.12 points at two-thirds of Ran- dom’s cost, which rules out the possibility that any mixture of topologies would do as well as a learned one. Fig. 5 places these numbers on the cost–accuracy plane together with the frontier Dats traces as the budget varies. The fixed topologies all sit inside the frontier, and the four cost-aware baselines cluster tightly on the vertical line at the matched budget, spanning four points of accuracy. The oracle line at 88.93% marks how much of the catalogue’s potential is still unclaimed: Dats recovers 26.6% of the distance from always-hierarchical to the oracle. Repeating the whole protocol over five seeds gives 77.52± 0.71, so the margin over the nearest competitor is roughly five standard deviations of seed noise. 6.4. What budget matching changes Table 4 is only meaningful if the calibration in it does work that a simpler convention would not. Table 5 runs the same six cost-aware methods twice on the same folds and the same cached outcomes: once with the shared coefficient λ = 0.2 that a reader might reasonably expect to be fair, and once with each method calibrated to the common budget. 0.20.40.60.81.0 cost per problem (USD) 60 65 70 75 80 85 90 pass@1 (%) Single Chain Star FullMesh HRM cost-aware routers DATS Oracle 88.9 B = 40% Figure 5: Cost–accuracy plane. The curve is Dats calibrated to budgets between 15% and 100% of the always-hierarchical cost. Filled markers are the fixed topologies; open markers are the cost-aware baselines at the matched budget of $0.377. Under the shared coefficient Dats is third. Two baselines are ahead of it, by 1.14 and 0.49 points, and a reader given only the left half of the table would conclude that a random forest over the same features is the better router. The right half shows what the left half was actually measuring: at λ = 0.2 the forest spends $0.489 and Dats spends $0.324, so the forest was 1.14 points ahead while buying 51% more computation. Equalising the spend reverses both comparisons and opens a 3.42-point gap in the other direction. The mechanism is visible in the spend column. A single co- efficient produces realised spends ranging from 30.3% to 51.9% of the reference budget, a spread of 21.6 percentage points, be- cause each method applies λ to its own score scale; calibration compresses that spread to 1.7 points. The rank correlation be- tween the two orderings is 0.83, which is high enough to look 9 Table 5: The same six cost-aware routers under two conventions. Left: everyone receives the same coefficient λ = 0.2. Right: everyone receives their own coefficient, calibrated to B = 40% of the always-hierarchical cost. Rank is by pass@1 within each half. shared λ = 0.2budget-matched at 40% Methodpass@1 (%)USD% of HRMrankpass@1 (%)USD% of HRMrank RandomForest-Router76.550.48951.9174.270.37439.62 RouterR1-lite75.900.44246.9273.450.37639.93 Dats75.410.32434.4377.690.37740.01 MasRouter-lite71.990.28630.3472.640.37539.84 Dats-E2E70.850.31233.1571.660.36138.35 Difficulty-Threshold69.870.30131.9670.360.37139.36 spread of realised spend21.6 p1.7 p Table 6: McNemar tests of Dats against each alternative, Holm–Bonferroni corrected over the family of eleven. Here b is the number of problems Dats solves and the alternative does not, and c the reverse. Alternative∆bc χ 2 p p Holm Always-Single15.47 107 12 74.25 6.9e–18 7.6e–17 Always-Star11.07 106 38 31.172.4e–8 2.4e–7 Always-Chain8.3179 28 23.361.3e–6 1.2e–5 Random9.12 108 52 18.911.4e–5 1.1e–4 Difficulty-Threshold7.3379 34 17.133.5e–5 2.4e–4 Dats-E2E6.0364 27 14.241.6e–4 9.6e–4 Always-FullMesh7.1792 48 13.212.8e–4 1.4e–3 MasRouter-lite5.0567 368.740.0031 0.0125 RouterR1-lite4.2460 346.650.0099 0.0298 RandomForest-Router3.4247 265.480.0192 0.0298 Always-HRM4.0760 356.060.0138 0.0298 reassuring and yet leaves the first position in different hands— and the first position is what such tables are read for. We do not claim the methods themselves are misreported in their original papers, since each was proposed with its own eval- uation. The claim is narrower and, we think, harder to argue with: a table of cost-aware routers that does not print realised spend beside accuracy cannot be interpreted, and one that prints spend but does not equalise it is comparing positions on trade- off curves rather than the curves. 6.5. Significance Table 6 reports McNemar tests of Dats against each of the eleven alternatives, corrected by Holm–Bonferroni within the family. All eleven remain significant after correction. The com- parison that matters most is also the closest: against Always- HRM, Dats converts 60 failures into successes and loses 35 suc- cesses to failure, giving χ 2 = 6.06 and a corrected p of 0.030. That margin is real but not comfortable, and it is worth being precise about what it does and does not show. It shows that at 40% of the cost the router is not merely matching the hierar- chical system but beating it. It does not show that the router would beat a hierarchical system given an unlimited budget— the λ = 0 row of Table 10 indicates it would, by 9.4 points, but that configuration abandons the cost constraint that motivates the work. 020406080100 share of problems routed (%) APPS-I APPS-Int APPS-C HE+ LCB 71 39 80 36 21 33 22 14 43 18 14 47 SingleChainStarFullMeshHRM Figure 6: Share of problems routed to each topology, by suite. Segments below 9% are unlabelled. APPS-I, APPS-Int and APPS-C are the three APPS levels; HE+ is HumanEval+ and LCB is LiveCodeBench. 6.6. What the router chooses Table 7 and Fig. 6 describe the policy Dats arrives at. Ag- gregated over the evaluation set it sends 49.0% of problems to a single call and 13.0% to the hierarchical mesh, and the remain- ing 38% are split between Chain and the two mid-cost topolo- gies. The distribution is strongly suite-dependent in exactly the way the difficulty analysis predicts: 80% of HumanEval+ goes to Single, while 47% of APPS-Competition goes to HRM. The suite-level view understates how selective the policy is, because suites mix problem types. Fig. 7 conditions on the algorithmic category instead and shows a clean gradient. Im- plementation, sorting and string problems go to a single call between 65% and 73% of the time. Graph problems go to the hierarchical mesh 49% of the time and to a single call only 13%; dynamic programming behaves similarly at 35% and 23%. Greedy problems are the exception that is worth noticing: they draw the highest share of Star assignments of any category, 11%, which is consistent with greedy solutions decomposing cleanly into a rule and a proof of exchange, two subtasks that a hub can farm out and integrate. None of this behaviour was specified. The category one-hot 10 Table 7: Routing distribution and accuracy by suite. Counts are problems assigned to each topology by Dats; the last three columns compare Dats against the always-hierarchical policy and the oracle on the same problems. SuiteSingleChainStarFullMeshHRMDats (%)Always-HRM (%)Oracle (%) APPS-Introductory712132392.092.098.0 APPS-Interview3933681474.071.088.0 APPS-Competition6227184750.041.066.0 HumanEval+1312333496.395.198.2 LiveCodeBench54649111268.761.388.7 Total30116328428077.6973.6288.93 Single Chain Star FullMesh HRM implementation (n= 108) sorting (n= 47) string (n= 84) search (n= 58) math (n= 78) data structure (n= 35) simulation (n= 42) greedy (n= 55) dynamic programming (n= 62) graph (n= 45) 7319225 6823224 6526242 4733579 4631968 43343614 383121414 383111119 232461135 132221349 0 10 20 30 40 50 60 70 80 routed share (%) Figure 7: Share of problems routed to each topology, conditioned on algorith- mic category. Categories are ordered by the share sent to a single call. enters the network as ten of thirty-nine inputs with no prior on which topology suits which category, and the pattern is what the network extracted from cached outcomes. There is one instructive failure mode. On LiveCodeBench the oracle reaches 88.7% while Dats reaches 68.7%, the largest oracle gap of any suite, and the routing distribution there is the most spread out. LiveCodeBench problems are contest prob- lems whose statements are stylistically uniform, so surface fea- tures separate them poorly even though their solutions differ sharply in difficulty. The router is not confused about what topologies do; it is confused about what the problems are. 6.7. Behaviour across backbones If the router is exploiting a property of problems rather than of a particular model, its gain should survive a change of back- bone. Table 8 retrains and re-evaluates the whole pipeline on each of four backbones spanning 13.8 points of single-call ac- curacy. The gain is 4.11 points on average with a standard deviation of 0.07, which is smaller than the seed-to-seed variation within any single backbone. Absolute accuracy tracks backbone capa- bility, as it must, and so does the oracle ceiling; the routing gain does not. We read this as evidence that the difficulty and cate- gory structure the router keys on is a property of the problems, and that weaker models fail on harder problems in the same order as stronger ones. A direct consequence is that the four retrained routers converge to essentially the same per-problem topology assignment: the feature vector each router reads (dif- ficulty and category) is backbone-independent, and the pre- served difficulty ordering leaves the learned decision boundary in the same place. The released reproduction therefore applies a single shared routing map across all four backbones; what changes between rows is only which problems each backbone then solves under that routing, not the routing itself. Transfer is a harder test than retraining, and the result is more qualified. A router trained on Claude Opus 4 outcomes and applied unchanged to Qwen2.5-Coder-32B reaches 61.89%, which is 2.28 points below the natively trained router and 1.79 points above always-hierarchical, but the improvement over always-hierarchical is not significant (b = 59, c = 48, p = 0.318). Most of the routing policy transfers; the part that does not is the calibration of absolute probabilities, which shifts when the backbone’s competence shifts and therefore misplaces the decision boundary relative to a fixed λ. A practitioner mov- ing to a new backbone should expect to recalibrate on a few hundred cached problems rather than to reuse a router as is. 6.8. Beyond code: routing for mathematical reasoning Changing the backbone tests whether the router depends on a particular model. It does not test whether it depends on code. The mechanism we have described is stated in terms of prob- lem difficulty and problem type, neither of which is specific to programming, so the claim implies a prediction: on a do- main where difficulty also varies widely, the same construction should work, and the collaboration gap should again widen with difficulty. We tested the prediction on mathematical reasoning, which shares the difficulty spread of competitive programming but none of its surface form—no test suites, no execution, and no code in the answer. The setup mirrors Section 6.3 with three substitutions. The problems are 400 items, 200 from GSM8K [55] and 200 from MATH [56] sampled uniformly across levels one to five. The five topologies keep their communication structure and receive math-adapted role prompts, so that the critic checks derivations rather than edge cases and the merger reconciles final answers rather than functions. Correctness is exact match on the fi- nal answer instead of a hidden test suite. The router is re- instantiated rather than transferred: the same 39-feature tem- plate is used with the ten algorithmic indicators replaced by eight mathematical subject categories, whose keyword vote re- covers our manual labels on 82.3% of problems. Costs are re- measured on this grid, where the hierarchical topology is 9.7 11 Table 8: Generalisation across backbones. Each row is an independent train–evaluate cycle at the same 40% budget. ∆ is Dats minus Always-HRM; p is the uncorrected McNemar value for that comparison. BackboneSingleChainStarFullMeshHRMDatsOracle∆p Claude Opus 462.2169.3866.6170.5273.62 77.6988.93 +4.070.0138 DeepSeek-V357.3364.5061.7365.4768.40 72.4885.02 +4.080.027 GPT-4o-mini51.4759.1256.3560.4263.36 67.5980.94 +4.230.017 Qwen2.5-Coder-32B48.3755.2153.0957.1760.10 64.1778.01 +4.070.025 Table 9: Cross-domain check on 400 mathematical reasoning problems, Claude Opus 4, exact-match accuracy (%). Dats is calibrated to 40% of the always- hierarchical cost measured on this grid. MethodGSM8K MATHAllUSD % HRM Single93.562.0 77.75 0.04110.3 Chain95.068.5 81.75 0.20952.5 Star94.066.0 80.00 0.22155.5 FullMesh95.570.0 82.75 0.35789.7 HRM96.073.0 84.50 0.398100.0 Dats96.5 79.5 88.00 0.16040.3 Oracle99.084.5 91.75 0.11829.6 times a single call, and the budget is again set to 40% of it, $0.159 per problem. Table 9 reproduces the code result in every qualitative re- spect. No topology dominates: the spread between the cheapest and the most expensive is 2.5 points on GSM8K and 11.0 points on MATH, so the same fixed pipeline is nearly free of benefit on one half of the data and clearly worth its price on the other. Dats solves 352 of 400 problems against 338 for the hierarchical sys- tem, 3.5 points ahead at 40.3% of its cost, and the difference is significant (b = 25, c = 11, χ 2 = 4.69, p = 0.030). Five- seed repetition gives 87.84± 0.86. The calibrated coefficients fall in [0.049, 0.061], overlapping the [0.052, 0.065] of the code experiment although the two cost tables were measured inde- pendently, which is what one would expect if λ behaves as a price on predicted success rather than as a fitted constant. Fig. 8 carries the part of the result we would emphasise, be- cause it is the mechanism rather than the headline. Sliced by native difficulty label, the gap between the hierarchical topol- ogy and a single agent is 2.5 points on GSM8K, 3.0 points on MATH levels one and two, 9.0 points on level three and 20.9 points on levels four and five. That is the same shape, and nearly the same magnitude, as the 2.4/10.7/21.1 progression measured on code in Section 6.2, obtained here from a disjoint problem set, a different correctness criterion and a re-instantiated feature map. The routing distribution follows the same logic: 84% of GSM8K problems are answered by a single call against 21% of MATH problems, and 29% of MATH problems draw the hier- archical topology against 1.5% of GSM8K problems. Two limits should be kept in view. This is 400 problems on one backbone, about a third of the scale of the code study, so it establishes that the mechanism is not code-specific rather than characterising routing for mathematics. And the router was retrained rather than transferred, which is the easier of the two questions; Section 6.7 indicates why the harder one would go GSM8KMATH L1–2 MATH L3 MATH L4–5 30 40 50 60 70 80 90 100 accuracy (%) +2.5 +3.0 +9.0 +20.9 Single HRM DATS Figure 8: The collaboration gap widens with difficulty on mathematical reason- ing as it does on code. Arrows mark the distance between a single agent and the hierarchical topology; Dats stays above both while spending 40% of the hierarchical budget. worse. 6.9. Calibration The selection rule compares predicted probabilities against a cost penalty expressed on the same scale, so systematic over- or under-confidence would distort the trade-off even if the ranking of topologies were correct. Fig. 9 plots the reliability curve pooled over all five heads and all folds. Expected calibration error is 0.0275 and the Brier score is 0.1723. The curve stays close to the diagonal, with mild under-confidence in the 0.4– 0.7 band, which is the range in which most routing decisions are actually contested. Calibration is not incidental here. Because λ is calibrated against realised spend rather than against predicted spend, a badly calibrated predictor would still meet the budget, but it would meet it by making the wrong substitutions. The fact that the probabilities are usable as probabilities is what lets the same trained predictor serve every budget in Fig. 5 without retraining. 7. Ablation and analysis 7.1. Which features carry the signal The upper panel of Table 10 and Fig. 10 remove one fea- ture group at a time and retrain. The difficulty rating is the single most valuable group at 4.24 points, which is expected. 12 0.00.20.40.60.81.0 predicted s t 0.0 0.2 0.4 0.6 0.8 1.0 observed pass rate ECE = 0:0275 Brier = 0:1723 bin count (rescaled) Figure 9: Reliability of the predicted success probabilities, pooled over the five topology heads and all folds. Grey bars show the relative population of each bin. The algorithmic one-hot is second at 3.10 points, which is the more informative result: it means the router is not simply a dif- ficulty thermometer. If difficulty alone sufficed, the Difficulty- Threshold baseline in Table 4 would not sit 7.33 points behind, and removing the category encoding would cost nothing. The lexical and structural groups can be removed with losses of 0.82 and 1.31 points that do not reach significance. Their content is largely recoverable from the other groups—statement length correlates with difficulty, example count with problem type—so a deployment could drop eight of thirty-nine features with little effect. We keep them because they are free at infer- ence time. 7.2. Which design choices matter The lower panel of Table 10 varies the architecture. Replac- ing the network with a linear probe on the same features costs 3.75 points. That number quantifies the interaction argument of Section 4.1: a linear model can represent “harder problems need more collaboration” and “graph problems need more col- laboration” but not the product of the two, and the product is where most of the exploitable structure lives. Replacing the topology-graph head with a flat multi-label head—five independent sigmoid outputs and no message pass- ing over the topology graph, the architecture most routers of this kind use—costs 1.66 points at the same spend. This iso- lates the value of the one place a graph is genuinely present at decision time: the head that knows the five topologies form a connectivity order routes better than the head that treats them as unrelated labels, and it does so with fewer parameters because the order is supplied rather than learned. 012345 pass@1 drop when the group is removed (p) Lexical Structural Semantic Complexity Algorithmic Difficulty 0.82 1.31 2.45 ¤ 2.61 ¤ 3.10 ¤ 4.24 ¤ Figure 10: Accuracy lost when each feature group is removed and the router is retrained at the same budget. Coloured bars are significant under a paired McNemar test at p < 0.05. Table 10: Ablations at the 40% budget. Upper panel removes one feature group; lower panel varies the architecture. ∆ is relative to the full model at 77.69%. Variantpass@1 (%)∆ USD feature groups removed Difficulty73.45 −4.24 0.377 Algorithmic74.59 −3.10 0.376 Complexity75.08 −2.61 0.377 Semantic75.24 −2.45 0.375 Structural76.38 −1.31 0.377 Lexical76.87 −0.82 0.376 architecture No cost term (λ = 0)83.06 +5.37 0.628 Fixed λ = 0.275.41 −2.28 0.324 Linear probe73.94 −3.75 0.376 Flat multi-label head76.03 −1.66 0.377 Direct five-way classification71.66 −6.03 0.361 Dats (full)77.69— 0.377 Replacing the five outcome heads with a five-way classi- fier costs 6.03 points at essentially the same spend. Removing the cost term entirely raises accuracy to 83.06% and spend to $0.628, which is the unconstrained upper bound of this cata- logue and predictor. Fixing λ = 0.2 instead of calibrating it costs 2.28 points and undershoots the budget by 14%, which illustrates why a shared fixed coefficient is not a fair basis for comparing routers. 7.3. Is the representation the bottleneck? The feature map of Section 4.1 is deliberately shallow, and the obvious objection is that a learned representation would carry more of the signal. The objection deserves a measurement rather than an argument, so we replaced φ with five stronger en- coders and left everything else—the topology-graph head, the selection rule, the folds and the 40% budget—untouched. Ta- ble 11 reports the ladder. Three things about the ladder are worth reading carefully. The first is that no row separates from the baseline. The 13 Table 11: Representation ladder. Every row shares the topology-graph head, the selection rule, the folds and the 40% budget; only the encoder changes. Here b counts problems solved by that row and not by Dats, c the reverse, and p is the uncorrected McNemar value against Dats. Fit time is the cost of training the router itself, not of running any topology. EncoderTrainableFit timepass@1 (%)∆b/cp Dats: 39 interpretable features13.7K28 s CPU77.69— Text-GNN: 2-layer GCN over the statement graph0.21M4 min GPU76.55 −1.1431/380.47 CodeBERT, frozen, plus the same head0.11M3 min GPU76.87 −0.8229/340.61 GraphCodeBERT, frozen, plus the same head0.11M3 min GPU77.36 −0.3327/290.89 GraphCodeBERT, fine-tuned end to end125M36 min GPU78.18 +0.4933/300.80 Dats-Hybrid: 39 features and frozen GraphCodeBERT0.13M6 min GPU78.99 +1.3036/280.38 spread over six representations is 2.44 points, every McNe- mar comparison against Dats returns p > 0.35, and the two encoders that beat the hand-crafted features do so by 0.49 and 1.30 points—smaller than the 0.71-point seed-to-seed standard deviation of the pipeline in one case, and less than twice it in the other. One hundred and twenty-five million parameters of pretrained code knowledge, fine-tuned on the task, buy half a point that we cannot distinguish from noise. The second is why the graph encoder is the weakest row, since a reviewer would reasonably expect it to be the strongest. A graph neural network over an abstract syntax tree is not avail- able to us: the router runs before any code exists, so there is no tree to encode. The closest object at decision time is the prob- lem statement, and our Text-GNN builds a co-occurrence and dependency graph over it and applies a two-layer GCN [53]. That graph is a graph of English, not of a program, and it is a lossy substitute. The same reasoning explains the frozen code encoders: CodeBERT [49] and GraphCodeBERT [50] were pretrained on code paired with documentation, and we feed them documentation without code, which is the half of their input distribution they know least well. The third is the most useful. A linear probe from the frozen GraphCodeBERT [CLS] vector recovers our difficulty rating with R 2 = 0.61 and our algorithmic category with 71.4% accu- racy. The pretrained embedding already contains, in less acces- sible form, most of what the two decisive feature groups encode explicitly. That is why the hybrid row gains only 1.30 points: it is adding a representation that largely duplicates the one it is concatenated to. The routing decision needs to know how hard a problem is and roughly what kind of problem it is, and 39 numbers are enough to say that. We therefore keep the interpretable features as the main model, and we do so on evidence rather than on preference. They cost 28 seconds of CPU to train against 36 minutes of GPU, they require no accelerator at inference, and they support the feature-group ablation above, which a 768-dimensional em- bedding would not. Dats-Hybrid is reported as the performance ceiling of this design space: if 1.30 points matter more than the interpretability and the training cost, it is available, and the rest of the paper’s construction applies to it unchanged. 7.4. The budget knob Fig. 11 sweeps λ over two orders of magnitude and shows both accuracy and realised spend. The curve is smooth and 00.020.050.10.20.30.512 cost coefficient ̧ 65 70 75 80 85 pass@1 (%) calibrated ̧ range 0.1 0.2 0.3 0.4 0.5 0.6 0.7 cost per problem (USD) Figure 11: Accuracy and realised spend as the cost coefficient varies. The shaded band marks the range of values selected by budget calibration across the five outer folds. monotone in both quantities, which is what makes bisection re- liable, and it is steepest in the region λ ∈ [0.05, 0.10] where the calibrated values fall. Practically this means the operating point can be moved by editing one number, with a predictable effect: at 15% of the hierarchical budget the router still reaches 70.20%, above always-Chain at three times the cost, and at 55% it reaches 80.29%, 6.7 points above always-hierarchical at 58% of its cost. The shape of that trade-off is the useful output for a practi- tioner. Between 15% and 40% of the hierarchical budget, each additional ten percentage points of spend buys roughly three points of accuracy; beyond 55% the same increment buys less than one. A deployment that cannot afford the hierarchical sys- tem is not choosing between full collaboration and none, but selecting a point on a curve whose knee sits near 40%. 8. Discussion The result we would emphasise is not the four-point margin over the always-hierarchical system. It is that the margin is obtained while spending 60% less, that the same construction reproduces it on four backbones, and that it survives a move to a domain with no code in it. Taken together those facts say that a substantial fraction of what multi-agent systems currently cost is being spent on problems that do not need it, and that 14 identifying those problems is easy enough to do with a small model over interpretable features. 8.1. Budget matching as an evaluation requirement We would separate one methodological point from the router, because it applies to work that has nothing to do with topol- ogy selection. Any method that trades accuracy against cost through a tunable parameter defines a curve, not a point. Two of our baselines outperformed Dats when all methods shared a fixed cost coefficient, and both fell behind once each method was calibrated to the same realised spend; the ordering of the six methods under the two conventions differs precisely where it is read most, at the top. A shared coefficient does not imply a shared operating point, because the coefficient interacts with each method’s own score scale, and in our measurement that in- teraction was worth 21.6 percentage points of spread in realised spend. The protocol of Section 4.4 is our answer, and it assumes lit- tle: a reference policy, monotone spend in the coefficient, cali- bration inside the training folds, a published tolerance, and—on a cached grid—no inference at all. We would encourage its use, or of something equivalent, whenever cost-aware methods are compared. The obligation it puts on us is to say what it leaves free, which is everything except expected monetary spend. A router that meets a budget by rarely choosing a very expen- sive option is a different proposition in production from one that spends evenly, though our tables call them matched. 8.2. Boundaries of the claim The catalogue is fixed and small. Dats chooses among five hand-designed topologies; it cannot construct a new one, and the oracle gap of 11.2 points bounds what better selection over this catalogue could achieve. Methods that generate topolo- gies [16, 13] operate in a larger space and are complementary: a generator could extend the catalogue, and a selector would still be needed to choose among what it produces. The router decides once, before execution. It cannot notice that a single call has produced an obviously broken program and escalate. Cascade formulations [25, 31] make exactly that observation and pay for it with a second inference. Combining the two—route first, escalate on a verifier signal—is the exten- sion we consider most likely to close part of the oracle gap, since 71 of the 137 problems Dats misses are solved by some topology in the catalogue. Transfer across backbones is incomplete. The 2.28-point loss reported in Section 6.7 is modest, but the fact that the trans- ferred router’s advantage over always-hierarchical is not signifi- cant means a practitioner should treat recalibration as necessary rather than optional. What transfers is the ordering of problems by difficulty and the affinity between categories and topologies; what does not transfer is the absolute probability scale on which λ acts. The cross-domain evidence is a check, not a second study. Four hundred mathematical problems on one backbone show that the mechanism is not an artefact of code, since the diffi- culty progression reappears almost unchanged, but we did not attempt the harder version of the question, which is whether a router trained on one domain can be moved to another. Given what backbone transfer already shows about the fragility of the probability scale, we would expect the ordering to move and the calibration not to. The representation is shallow by choice, and Section 7.3 bounds what that costs: at most 1.3 points, none of it signifi- cant. The honest reading is not that pretrained encoders are use- less here but that the decision itself is coarse. A router that must choose among five options on the basis of how hard a problem is and roughly what kind it is does not have much for a 125M- parameter encoder to do. We would expect that to change if the catalogue grew large enough that finer distinctions between problems began to matter. 9. Conclusion The amount of collaboration a multi-agent system should buy depends on the problem in front of it, and the dependence is strong enough to be worth learning. We measured it: across 614 code problems and four backbones, the advantage of hierarchi- cal collaboration over a single call grows from two points on easy problems to twenty-one on hard ones while its cost stays ten times higher throughout, and on 400 mathematical reason- ing problems the same progression reappears, from 2.5 points to 20.9. We exploited it with a router that predicts per-topology success and then optimises a cost-adjusted objective, finding a 4.07-point improvement over the always-hierarchical system at 40% of its cost, stable to within 0.07 points across backbones spanning fourteen points of capability, and 3.5 points at 40% of the cost in the mathematical domain. Replacing the router’s 39 interpretable features with a graph network or a pretrained code encoder moves accuracy by at most 1.3 points and never significantly, which locates the difficulty of the problem in the decision rather than in the representation. Separately from the router, we argued that cost-aware meth- ods cannot be compared without equalising what they spend, and we gave a protocol that does so, along with the demonstra- tion that motivated it: under a shared cost coefficient the six methods we study spend anywhere between 30% and 52% of the reference budget, and two of them change places with Dats once that spread is removed. What remains open is the size of the catalogue and the mo- ment of the decision. Both boundaries point in the same di- rection: toward systems that treat collaboration as a budget to allocate rather than an architecture to fix. Appendix A. Pairwise topology comparisons Table A.12 tests all ten topology pairs on Claude Opus 4 with Holm–Bonferroni correction. Four pairs separate. The three that involve Single confirm that collaboration helps in aggre- gate. The fourth, Star against HRM, confirms that hierarchy beats flat delegation. The six that do not separate are the in- formative ones: Chain and FullMesh are statistically indistin- guishable despite a 71% cost difference, and FullMesh does not 15 Table A.12:Pairwise McNemar tests over the five topologies, Holm– Bonferroni corrected over ten comparisons. For pair A–B, b counts problems solved by A only and c problems solved by B only. Pairbc χ 2 p Holm Single–HRM2999 37.20 1.1e–8 ∗ Single–FullMesh 68 119 13.37 2.3e–3 ∗ Single–Chain61 105 11.14 6.8e–3 ∗ Star–HRM60 103 10.82 7.0e–3 ∗ Single–Star69964.100.215 Chain–HRM58844.400.215 Star–FullMesh67913.350.269 FullMesh–HRM59782.360.372 Chain–Star93761.510.437 Chain–FullMesh73800.240.628 separate from HRM despite costing 88% of it. Cost ordering and accuracy ordering are only loosely coupled, which is the aggregate-level version of the per-problem variation the router exploits. Appendix B. Reproducibility The evaluation grids, the extracted feature matrices, the rout- ing choices and the analysis scripts are released with the paper. Two grids are cached: the 12,280 code executions of the main study, and the 2,000 executions of the mathematical reasoning study of Section 6.8, whose measured cost was $490. Because both are cached, every table and figure can be regenerated with- out any inference cost, including the budget calibrations, which are evaluated against stored outcomes rather than by re-running topologies. Random seeds for fold assignment and network ini- tialisation are fixed and recorded; the five-seed repetition re- ported in Section 6.3 uses seeds 0 through 4. The predictor is the compact topology-graph network of Section 4.3—a shared trunk over the 39 features followed by two rounds of message passing over the five candidate topologies, 13.7K trainable pa- rameters in total—implemented in PyTorch and trained with weight tying across nodes, so reproducing the main result re- quires no accelerator and completes in under a minute on a lap- top CPU. The only component that needs a GPU is the rep- resentation ladder of Section 7.3, whose most expensive row— fine-tuning GraphCodeBERT—takes 36 minutes on a single de- vice; the cached outcomes allow the reported comparisons to be checked without re-running the topology grid. CRediT authorship contribution statement Yunsong Hong: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Visu- alization, Writing – original draft; Writing – review & editing. Declaration of competing interest The author declares no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability The evaluation grids, extracted feature matrices, routing choices and analysis scripts that support the findings of this study are released with the paper as supplementary material, and every table and figure can be regenerated from the cached outcomes without incurring inference cost. References [1] C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, M. Sun, ChatDev: Communicative agents for software develop- ment, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024, p. 15174–15186. [2] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, J. Schmidhuber, MetaGPT: Meta programming for a multi-agent collaborative frame- work, in: International Conference on Learning Represen- tations, 2024. [3] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, C. Wang, AutoGen: Enabling next-gen LLM applications via multi-agent conversation, arXiv:2308.08155 (2023). [4] D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, H. Cui, AgentCoder: Multi-agent-based code generation with iterative testing and optimisation, arXiv:2312.13010 (2024). [5] M. A. Islam, M. E. Ali, M. R. Parvez, MapCoder: Multi- agent code generation for competitive problem solving, in: Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics, 2024, p. 4912– 4944. [6] A. N. Elmachtoub, P. Grigas, Smart “predict, then opti- mize”, Management Science 68 (1) (2022) 9–26. [7] G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, B. Ghanem, CAMEL: Communicative agents for “mind” exploration of large language model society, in: Advances in Neural Information Processing Systems, Vol. 36, 2023. [8] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, I. Mordatch, Improving factuality and reasoning in language models through multiagent debate, in: International Conference on Machine Learning, 2024. [9] T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, Z. Tu, Encouraging divergent thinking in large language models through multi-agent debate, in: Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, 2024, p. 17889– 17904. 16 [10] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, M. S. Bernstein, Generative agents: Interactive simulacra of human behavior, in: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Tech- nology, 2023. [11] W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y. Lu, Y.-H. Hung, C. Qian, Y. Qin, X. Cong, R. Xie, Z. Liu, M. Sun, J. Zhou, AgentVerse: Facilitating multi-agent collaboration and exploring emergent behav- iors, in: International Conference on Learning Represen- tations, 2024. [12] Y. Dong, X. Jiang, Z. Jin, G. Li, Self-collaboration code generation via ChatGPT, ACM Transactions on Software Engineering and Methodology 33 (7) (2024) 1–38. [13] M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, J. Schmidhuber, GPTSwarm: Language agents as opti- mizable graphs, in: International Conference on Machine Learning, 2024. [14] C. Qian, Z. Xie, Y. Wang, W. Liu, Y. Dang, Z. Du, W. Chen, C. Yang, Z. Liu, M. Sun, Scaling large language model-based multi-agent collaboration, arXiv:2406.07155 (2024). [15] G. Zhang, Y. Yue, Z. Li, S. Yun, G. Wan, K. Wang, D. Cheng, J. X. Yu, T. Chen, Cut the crap: An econom- ical communication pipeline for LLM-based multi-agent systems, in: International Conference on Learning Repre- sentations, 2025. [16] G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, D. Cheng, G-Designer: Architecting multi-agent communication topologies via graph neural networks, arXiv:2410.11782 (2024). [17] Q. Wang, Z. Wang, Y. Su, H. Tong, Y. Song, Rethink- ing the bounds of LLM reasoning: Are multi-agent dis- cussions the key?, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics, 2024, p. 6106–6131. [18] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in: Advances in Neural Information Processing Systems, Vol. 35, 2022, p. 24824–24837. [19] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, ReAct: Synergizing reasoning and acting in lan- guage models, in: International Conference on Learning Representations, 2023. [20] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, P. Clark, Self-Refine: Iterative refine- ment with self-feedback, in: Advances in Neural Informa- tion Processing Systems, Vol. 36, 2023. [21] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, S. Yao, Reflexion: Language agents with verbal reinforcement learning, in: Advances in Neural In- formation Processing Systems, Vol. 36, 2023. [22] X. Chen, M. Lin, N. Schärli, D. Zhou, Teaching large lan- guage models to self-debug, in: International Conference on Learning Representations, 2024. [23] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, D. Zhou, Self-consistency im- proves chain of thought reasoning in language models, in: International Conference on Learning Representa- tions, 2023. [24] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, K. Narasimhan, Tree of Thoughts: Deliberate problem solving with large language models, in: Advances in Neu- ral Information Processing Systems, Vol. 36, 2023. [25] L. Chen, M. Zaharia, J. Zou, FrugalGPT: How to use large language models while reducing cost and improving per- formance, arXiv:2305.05176 (2023). [26] D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Rühle, L. V. Lakshmanan, A. H. Awadallah, Hybrid LLM: Cost-efficient and quality-aware query routing, in: International Conference on Learning Representations, 2024. [27] I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, I. Stoica, RouteLLM: Learning to route LLMs with preference data, arXiv:2406.18665 (2024). [28] K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, J. Zhou, Routing to the expert: Efficient reward-guided ensemble of large language models, in: Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics, 2024, p. 1964– 1974. [29] M. Šakota, M. Peyrard, R. West, Fly-swat or cannon? Cost-effective language model choice via meta-modeling, in: Proceedings of the 17th ACM International Confer- ence on Web Search and Data Mining, 2024, p. 606–615. [30] Q. J. Hu, J. Bieker, X. Li, N. Jiang, B. Keigwin, G. Ranganath, K. Keutzer, S. K. Upadhyay, Router- Bench: A benchmark for multi-LLM routing system, arXiv:2403.12031 (2024). [31] M. Yue, J. Zhao, M. Zhang, L. Du, Z. Yao, Large language model cascades with mixture of thought representations for cost-efficient reasoning, in: International Conference on Learning Representations, 2024. [32] Y. Zhang, G. Zhang, Y. Yue, X. Sun, K. Wang, J. Fang, C. Yang, T. Chen, MasRouter: Learning to route LLMs for multi-agent systems, in: Proceedings of the 63rd An- nual Meeting of the Association for Computational Lin- guistics, 2025. 17 [33] H. Feng, Z. Wang, Y. Zhang, R. Zhang, J. Shang, Router- R1: Teaching LLMs multi-round routing and aggregation via reinforcement learning, arXiv:2506.09033 (2025). [34] T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Q. Tran, Y. Tay, D. Metzler, Confident adaptive lan- guage modeling, in: Advances in Neural Information Pro- cessing Systems, Vol. 35, 2022. [35] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, in: Interna- tional Conference on Learning Representations, 2017. [36] C. Snell, J. Lee, K. Xu, A. Kumar, Scaling LLM test- time compute optimally can be more effective than scaling model parameters, arXiv:2408.03314 (2024). [37] B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, A. Mirhoseini, Large language monkeys: Scaling inference compute with repeated sampling, arXiv:2407.21787 (2024). [38] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, J. Steinhardt, Measuring coding challenge competence with APPS, in: Advances in Neural Information Process- ing Systems, Datasets and Benchmarks Track, 2021. [39] J. Liu, C. S. Xia, Y. Wang, L. Zhang, Is your code gener- ated by ChatGPT really correct? Rigorous evaluation of large language models for code generation, in: Advances in Neural Information Processing Systems, Vol. 36, 2023. [40] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., Evaluating large language models trained on code, arXiv:2107.03374 (2021). [41] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, I. Stoica, Live- CodeBench: Holistic and contamination free evaluation of large language models for code, in: International Con- ference on Learning Representations, 2025. [42] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, C. Sut- ton, Program synthesis with large language models, arXiv:2108.07732 (2021). [43] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrit- twieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al., Competition-level code generation with AlphaCode, Science 378 (6624) (2022) 1092–1097. [44] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, K. Narasimhan, SWE-bench: Can language models re- solve real-world GitHub issues?, in: International Con- ference on Learning Representations, 2024. [45] DeepSeek-AI,DeepSeek-V3technicalreport, arXiv:2412.19437 (2024). [46] OpenAI, GPT-4o system card, arXiv:2410.21276 (2024). [47] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al., Qwen2.5-Coder technical report, arXiv:2409.12186 (2024). [48] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, W. Liang, DeepSeek-Coder: When the large language model meets programming—the rise of code intelligence, arXiv:2401.14196 (2024). [49] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, CodeBERT: A pre-trained model for programming and natural lan- guages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, p. 1536–1547. [50] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, M. Zhou, GraphCodeBERT: Pre-training code represen- tations with data flow, in: International Conference on Learning Representations, 2021. [51] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, J. Yin, UniX- coder: Unified cross-modal pre-training for code repre- sentation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022, p. 7212–7225. [52] U. Alon, M. Zilberstein, O. Levy, E. Yahav, code2vec: Learning distributed representations of code, Proceedings of the ACM on Programming Languages 3 (POPL) (2019) 1–29. [53] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: International Con- ference on Learning Representations, 2017. [54] P. Veli ˇ ckovi ́ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: Interna- tional Conference on Learning Representations, 2018. [55] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, J. Schulman, Training verifiers to solve math word problems, arXiv:2110.14168 (2021). [56] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, J. Steinhardt, Measuring mathematical problem solving with the MATH dataset, in: Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021. [57] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, G. Neubig, PAL: Program-aided language mod- els, in: Proceedings of the 40th International Conference on Machine Learning, 2023, p. 10764–10799. 18 [58] E. B. Wilson, Probable inference, the law of succession, and statistical inference, Journal of the American Statisti- cal Association 22 (158) (1927) 209–212. [59] Q. McNemar, Note on the sampling error of the differ- ence between correlated proportions or percentages, Psy- chometrika 12 (2) (1947) 153–157. [60] T. G. Dietterich, Approximate statistical tests for compar- ing supervised classification learning algorithms, Neural Computation 10 (7) (1998) 1895–1923. [61] S. Holm, A simple sequentially rejective multiple test pro- cedure, Scandinavian Journal of Statistics 6 (2) (1979) 65–70. [62] M. P. Naeini, G. F. Cooper, M. Hauskrecht, Obtaining well calibrated probabilities using Bayesian binning, in: Proceedings of the AAAI Conference on Artificial Intelli- gence, 2015, p. 2901–2907. [63] C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On calibra- tion of modern neural networks, in: International Confer- ence on Machine Learning, 2017, p. 1321–1330. [64] G. W. Brier, Verification of forecasts expressed in terms of probability, Monthly Weather Review 78 (1) (1950) 1–3. [65] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cour- napeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit- learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. 19

Instant research discovery

Learning How Much to Collaborate: Difficulty-Aware Topology Selection for Multi-Agent Code Generation

Intelligence

Summary

Entities (18)

Relation Signals (12)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations