Paper deep dive
AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering
Kaituo Zhang, Zhen Xiong, Zhimeng Jiang, Mingyu Zhong, Zhouyuan Yuan, Zhecheng Li, Bowen Lin, Chia-Yuan Chang, Mingzhi Hu, Huazheng Wang, Ying Lin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.
Tags
Links
- Source: https://arxiv.org/abs/2609.32662v1
- Canonical: https://arxiv.org/abs/2609.32662v1
Trouble viewing inline? Open PDF directly â
Full Text
85,679 characters extracted from source content.
Expand or collapse full text
AsynCodeBench ASYNCODEBENCH: BENCHMARKING COLLABORATION OF ASYNCHRONOUS MULTI-AGENT SYSTEMS IN SOFTWARE ENGINEERING Kaituo Zhang University of Houston Zhen Xiong New York University Zhimeng Jiang Texas A&M University Mingyu Zhong University of Houston Zhouyuan Yuan UCSD Zhecheng Li UCSD Bowen Lin University of Houston Chia-Yuan Chang Texas A&M University Mingzhi Hu Worcester Polytechnic Institute Huazheng Wang Oregon State University Ying Lin University of Houston ABSTRACT Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent sys- tems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual cod- ing capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineer- ing that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: A synchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and D ependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency- level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window. These findings po- sition dependency dynamics as a useful lens for understanding and interpreting collaboration behavior in asynchronous multi-agent software engineering. § github.com/KaituoZhang/AsynCodeBench AsynCodeBench.org 1INTRODUCTION Large language model agents have rapidly expanded from code generation to repository-level soft- ware engineering, where they can modify files, execute tests, and iteratively repair implementa- tions (Jimenez et al., 2024; Yang et al., 2024a). Multi-agent software development has conse- quently emerged as an active direction, with systems such as ChatDev, MetaGPT, RTADev, CAID, 1 arXiv:2609.32662v1 [cs.SE] 26 Sep 2026 AsynCodeBench and CodeTeam distributing planning, implementation, testing, and coordination across specialized agents (Qian et al., 2024; Hong et al., 2024; Liu et al., 2025; Wang et al., 2026; Geng & Neubig, 2026). Recent work has also begun to study interaction and coordination more explicitly, includ- ing recovery from out-of-sync states and corrective interaction during coding (Guo et al., 2025; Wu et al., 2026). Yet multi-agent software engineering introduces a capability beyond solving the task itself: agents must coordinate their independently produced work. Strong performance in single- agent coding therefore does not necessarily imply effective collaboration in a multi-agent setting. Evaluation, however, remains largely focused on final task outcomes. Repository-level benchmarks such as SWE-bench assess the final implementation through executable tests (Jimenez et al., 2024), while multi-agent coding systems often report gains using the same end-task metrics over single- agent baselines (Geng & Neubig, 2026; Wang et al., 2026). These comparisons show whether multi-agent execution performs better, but do not directly measure collaboration: final scores con- flate individual coding ability with cross-agent coordination and may hide unresolved dependencies across independently implemented components. We therefore use cross-component software dependencies as a lens for collaboration and introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engi- neering. Unlike existing evaluations that characterize collaboration through final task outcomes or isolated interaction behaviors, AsynCodeBench makes cross-agent dependency resolution observ- able throughout execution. We introduce Asynchronous Dependency Pass Rate (ADPR) to capture dependency resolution and Dependency Resolution Step (DRS) to capture resolution timing, en- abling collaboration to be evaluated both by its outcome and by how it develops over time. Our study provides a dependency-centric view of collaboration in multi-agent software engineering, separating collaboration capability from conventional coding performance and making its evolution observable throughout execution. Using AsynCodeBench, we find that multi-agent execution is not uniformly stronger than single-agent execution, and that within the evaluated Qwen family, stronger single-agent coding performance does not improve all aspects of collaboration. Dependency tra- jectories further reveal a recurring hopping phenomenon, where substantial dependency resolution is often established through concentrated phases whose timing varies across models and execution protocols. Finally, dependency-aware evaluation can lead to different qualityâcost assessments from conventional test-based metrics. These findings highlight collaboration as a distinct capability that cannot be fully characterized by final coding outcomes alone. 2RELATED WORK 2.1REPOSITORY-LEVEL CODING AGENTS AND BENCHMARKS Repository-level coding benchmarks have emerged to bridge the gap between closed-form code generation and real software engineering, where solving a task requires reasoning over an existing codebase rather than completing an isolated function (Ding et al., 2023; Liu et al., 2024; Jimenez et al., 2024). This setting preserves challenges that are largely abstracted away by function-level evaluation, including cross-file reasoning, software dependencies, execution feedback, and consis- tency across interacting changes (Gautam et al., 2025; Li et al., 2025; Deng et al., 2025; Le et al., 2026; Ding et al., 2026). To retain these challenges, benchmark construction has increasingly relied on real repositories and development artifacts together with executable validation, preserving much of the surrounding software context encountered in practice (Jimenez et al., 2024; Li et al., 2025; Pan et al., 2025; Miserendino et al., 2025; Badertdinov et al., 2025; Vergopoulos et al., 2025). This repository-grounded paradigm has since expanded across software ecosystems and toward increas- ingly broad and long-horizon development tasks, ranging from feature and library implementation to software evolution and repository-scale transformation (Yang et al., 2024b; Zan et al., 2025; Rashid et al., 2025; Zhao et al., 2024; Yang et al., 2025; Badertdinov et al., 2026; Le et al., 2026; Hong et al., 2026). Yet as the tasks themselves have become more realistic and extended over time, their evaluation still largely reduces execution to final task completion, leaving intermediate progress and failure dynamics comparatively under-characterized. 2 AsynCodeBench 2.2MULTI-AGENT SOFTWARE ENGINEERING Multi-agent software engineering has rapidly expanded from role-based workflows toward repository-scale collaboration, with increasingly sophisticated mechanisms for specialization, dele- gation, integration, and concurrent execution (Qian et al., 2024; Hong et al., 2024; Liu et al., 2025; Wang et al., 2026; Oliveira et al., 2026; Costa, 2026; Kumar et al., 2026). Despite this shift, evalua- tion still largely inherits a single-agent perspective, judging multi-agent systems by end-task perfor- mance or gains over single-agent baselines rather than measuring collaboration itself (Khatua et al., 2026; Donato et al., 2026; Destefanis & Aste, 2026). Recent work has begun to expose this gap through failure analysis, out-of-sync recovery, role-separated coordination, distributed communica- tion, and interactive coding (Cemri et al., 2025; Guo et al., 2025; Kim et al., 2026; Zhang et al., 2026b; Wu et al., 2026). AsynCodeBench goes further by tracing the dependencies that must be re- solved across independently implemented components, revealing failed handoffs and coordination dynamics that can remain hidden behind final task performance. 3BENCHMARK CONSTRUCTION 3.1PROBLEM DEFINITION Multi-agent software engineering often decomposes a coding task across multiple agents, each re- sponsible for a different implementation unit. We refer to collaboration as asynchronous when these agents proceed independently without necessarily waiting for dependent agents to complete or propagate their updates. When dependencies exist across their subproblems, this execution parser.py api.py schema.py tests.py Execution results(outcome-only) <parser.py> Passed <api.py> Failed <tests.py> Failed <schema.py> Passed Outcome-only evaluation: Only know the task failed Update parser for feature extraction Adapt api / tests to parser commit abc1234 parser.py updated api.py / tests.py still depend on old parser í í í í í í Asynchronous collaboration failed on dependent tasks Multi-file and Multi-agent coding task Dependent and asynchronous process working... Still using the old dependency Updated Failed AgentA AgentB Dependency matters Figure 1: A small illustration of the existing gap. pattern can create coordination failures:a downstream agent may continue working with incomplete or outdated upstream behavior, causing the independently produced implemen- tations to become inconsistent when they are integrated. We refer to failures arising from this interaction between asynchronous execu- tion and software dependencies as the asyn- chronous coordination problem studied in AsynCodeBench.A key difficulty is that these failures are often hidden by traditional outcome-only evaluation, since dependencies across agentsâ subproblems are typically im- plicit and not explicitly tracked. As illustrated in Figure 1, the evaluator may identify failed components without revealing the stale depen- dency between agents that produced them. This makes it difficult to determine whether the failure reflects limitations in individual coding capability or in multi-agent collaboration. The key question is therefore whether dependencies between agentsâ implementations are correctly maintained during collaboration, rather than only whether the final repository succeeds. In Asyn- CodeBench, a dependency exists when the correctness of code assigned to one agent depends on code implemented by another, such as a required function, interface, data structure, or shared state. We make these dependencies explicit through task-specific dependency graphs, evaluate them with executable Dependency Checkers, and track them across integration checkpoints to determine whether and when they are resolved. 3.2DEPENDENCY GRAPH AND EXECUTABLE CHECKERS We operationalize asynchronous coordination through four benchmark components, B = (R,G,C,E), where B denotes AsynCodeBench, R the repository and target coding task, G the task-specific dependency graph,C the executable Dependency Checkers defined over the dependencies inG, and E the benchmark evaluation. 3 AsynCodeBench Each task is represented by a directed dependency graph G = (V,A), whereV is the set of implementation units andA â V ĂV is the set of directed dependency arcs. Each d â A connects an upstream producer to a downstream consumer whose correctness depends on behavior provided by the producer. The graph is defined before model evaluation. Appendix A.1 details the dependency construction procedure and supporting repository evidence. For each dependency dâA, we pre-register three groups of executable Boolean checkers, C d = C U d ,C D d ,C I d , C =C d dâA . Each constituent checker c returns a binary pass/fail outcome on a repository workspace W , with c(W) = 1 indicating that the tested condition is satisfied. The upstream groupC U d checks whether the producer provides the required behavior, the downstream groupC D d checks whether the consumer correctly uses or preserves it, and the integrated group C I d checks whether the producerâconsumer dependency holds after integration. For X âU,D,I, the group-level outcome is C X d (W) = 1 âcâC X d , c(W) = 1 . The upstream and downstream outcomes provide diagnostic evidence about the two sides of a depen- dency, whereas C I d determines whether the dependency itself is satisfied in the integrated workspace. Let W t denote the retained integrated workspace at checkpoint t, t = 1,...,T . We define I d,t = C I d (W t ), I d = (I d,1 ,...,I d,T ), where I d,t â 0, 1 is the satisfaction state of dependency d at checkpoint t. Because the same dependency is evaluated over an ordered sequence of integrated repository states, I d records its res- olution trajectory over execution. It therefore distinguishes dependencies that never resolve, resolve only after several integration steps, or resolve and later regressâinformation that is lost when only the final workspace is evaluated. Appendix C details checkpoint construction and indexing. 3.3EVALUATION METRICS Test Pass Rate. Unlike a binary task success measure, Test Pass Rate quantifies the fraction of evaluation points obtained from the repository-level test suite: TestPass = P N , where P is the number of passed test points and N is the total number of test points for the current task. It provides proportional credit for partially correct implementations, but does not directly assess whether the benchmark-defined cross-component dependencies are satisfied. Asynchronous Dependency Pass Rate. Using the pre-registered dependency graph and exe- cutable checker states defined above, ADPR measures the fraction of dependencies satisfied in the final integrated workspace: ADPR = 1 |A| X dâA I d,T . ADPR therefore provides a dependency-level measure of how successfully the cross-component dependencies identified in AsynCodeBench are resolved after integration. Dependency Resolution Step. Based on the ordered integrated-checker outcomes defined above, DRS measures the first checkpoint at which a dependency passes its integrated checker: DRS d = mint : I d,t = 1, with DRS d = T + 1 when dependency d never passes during execution. Since the number of retained checkpoints varies across runs, we normalize the resolution step as ] DRS d = DRS d T + 1 , 4 AsynCodeBench where smaller values indicate earlier resolution and 1 denotes non-resolution. By using the ordered dependency trajectory rather than only its final state, DRS distinguishes dependencies that resolve early from those that become valid only after later integration steps, providing a temporal view of dependency resolution that final ADPR alone cannot capture. DRS is defined over logical integration checkpoints rather than wall-clock time. Since SINGLE exposes only the final workspace, it does not provide a dependency trajectory and is therefore excluded from DRS. 3.4TASK CURATION AND HUMAN VALIDATION We curate public repository tasks whose implementations naturally span behaviorally coupled files or modules, preserving existing software boundaries rather than introducing artificial code parti- tions. AsynCodeBench contains 19 tasks from 16 repositories, covering framework and library engineering, systems and storage, compiler/IR engineering, network protocols, and security. Each task is packaged with a fixed repository revision, an answer-free task instruction, scoped writable paths, and a task-specific pytest evaluator. All execution protocols use the same task package, OpenHands (Wang et al., 2025) harness, and task-specific Docker environment. To ensure that the dependency structure reflects genuine repository constraints, all dependency graphs undergo independent human validation. Three reviewers verify the validity and direction of each dependency together with its supporting executable evidence, and only validated depen- dencies are retained. The complete task inventory, packaging details, and dependency-validation protocol are reported in Appendix A.2, with human-review details in Appendix A.3. 4EVALUATION PROTOCOLS We organize the five execution protocols as a progression from independent full-task solving to increasingly structured collaboration. SINGLE assigns the entire task to one agent. SERIAL intro- duces specialist decomposition while preserving producer-to-consumer handoffs, whereas ASYNC- PRIVATE keeps the same decomposition but allows specialists to proceed concurrently in private workspaces. The two manager variants then add an explicit coordination layer, first without write access and then with bounded repair. The contrast between SERIAL and ASYNC-PRIVATE isolates the effect of asynchronous execution under the same specialist decomposition. In SERIAL, each downstream specialist begins only after the corresponding upstream artifact has been integrated; in ASYNC-PRIVATE, specialists work from the same initial repository state and cannot observe one anotherâs in-flight changes. This creates the setting in which unresolved cross-agent dependencies can emerge. The manager variants test whether these failures can be mitigated through coordination. ASYNC- MANAGER-RO allows a read-only manager to inspect artifacts, provide feedback, and coordinate integration without modifying production code. ASYNC-MANAGER further permits bounded repair after specialist integration. This distinction separates coordination through information exchange from direct code intervention. Full protocol details are provided in appendix B. 5EXPERIMENTS 5.1EXPERIMENT SETTINGS We evaluate ten open-weight models from the Qwen, DeepSeek, Gemma, Muse, GLM, and Nemotron families, covering dense and mixture-of-experts architectures from 9B dense parame- ters to 284B total parameters. We use shortened model names. Complete checkpoint identifiers, parameter counts, architectures, and weight precisions are provided in Appendix D. Using AsynCodeBench, we systematically investigate multi-agent software engineering across four research questions, progressing from collaboration effectiveness and model capability to dependency-resolution dynamics, and resource costs. RQ1 How does multi-agent collaboration compare with single-agent execution, and what aspects of collaboration are overlooked by conventional task-level metrics? 5 AsynCodeBench RQ2 How do model scale and generation relate to collaborative performance and dependency resolution? RQ3 How do execution protocols shape dependency-resolution outcomes and closure dynamics over successive integration checkpoints? RQ4 How do token costs vary across execution protocols, and how does the choice of quality metric affect the observed qualityâcost trade-offs? 5.2COLLABORATION GAINS AND HIDDEN DEPENDENCY FAILURES We first ask whether multi-agent execution consistently improves over standalone coding across models. Figure 2(a) compares each modelâs task-macro Test Pass Rate under Single with its best result among the four multi-agent protocols. This Best Multi score is intentionally optimistic: it asks whether any collaborative protocol can improve upon the same model acting alone, without yet attributing the gain to a particular protocol. The answer varies markedly across models. Six of the ten models improve under their best multi- agent protocol, with gains of 7.8â22.7%, while four remain worse than SINGLE even after selecting their best collaborative protocol. The contrast is particularly clear for models with high standalone performance: Qwen3.8-27B and DeepSeek-V4 both reach a 93.6% Test Pass Rate under SINGLE, yet their best multi-agent results decrease to 86.7% and 89.9%, respectively. Thus, multi-agent decomposition can provide substantial gains, but these gains are neither universal nor guaranteed to preserve a modelâs standalone performance. Task-level progress, however, does not alone reveal whether collaboration succeeds across compo- nent boundaries. Figure 2(b) examines this distinction under ASYNC-MANAGER-RO using all 190 modelâtask outcomes. Test Pass Rate and ADPR are positively but imperfectly associated (Pearson r = 0.617; Spearman Ï = 0.613), and their disagreement is strongly asymmetric. Test Pass Rate exceeds ADPR in 136 of 190 outcomes (71.6%), with a mean Test Pass Rate of 48.0% compared with only 18.8% ADPR. More strikingly, 32 outcomes achieve at least 80% Test Pass Rate while re- solving fewer than half of their declared dependencies, including 26 with zero ADPR; we observe no corresponding low-test, high-ADPR cases. Results for the other four protocols are in Appendix E. Overall, multi-agent execution is not uniformly beneficial, and task-level success alone provides an incomplete picture of collaboration. While Test Pass Rate measures broad implementation progress, ADPR directly captures whether dependencies across specialists are successfully resolved, exposing coordination failures that may remain hidden behind high aggregate test performance. 5.3MODEL CAPABILITY: QWEN SCALE AND GENERATION The cross-model results above suggest that improvements in standalone coding performance do not necessarily translate into comparable gains under multi-agent execution. To examine this pat- tern under a more controlled setting, we turn to the Qwen family. We compare Qwen3.5-9B with Qwen3.5-27B to study scaling within the same generation, and Qwen3.5-27B, Qwen3.6-27B, and Qwen3.8-27B to study model evolution at a fixed nominal scale. Figure 3 reports Test Pass Rate, ADPR, and normalized DRS. Scaling Qwen3.5 from 9B to 27B consistently improves overall test progress, but its effect on de- pendency resolution is less uniform. Test Pass Rate increases under every protocol, including from 24.4% to 42.1% for SINGLE and from 36.9% to 59.2% for ASYNC-MANAGER. ADPR follows a different pattern: SINGLE changes only from 14.2% to 12.3%, while ASYNC-MANAGER increases from 2.8% to 31.6%. Thus, additional model capacity clearly improves implementation progress, but the extent to which this improvement carries over to cross-component dependency resolution depends strongly on the execution protocol. The contrast becomes more pronounced across generations. At 27B, SINGLE improves substantially from Qwen3.5 to Qwen3.6 and Qwen3.8, with Test Pass Rate increasing from 42.1% to 57.4% to 93.6% and ADPR from 12.3% to 36.8% to 77.2%. Multi-agent execution also improves, but not at the same rate. Under ASYNC-MANAGER, ADPR rises from 31.6% for Qwen3.5 to 63.2% for Qwen3.6, but changes only marginally to 64.9% for Qwen3.8. In other words, the large improve- 6 AsynCodeBench Qwen3.5-9B Gemma-4 Nemotron-3.5 Qwen3.5-27B GLM-4.7 Muse-Glimmer Qwen3-Coder Qwen3.6Qwen3.8 DeepSeek-V4 20% 30% 40% 50% 60% 70% 80% 90% 100% Test Pass Rate (a) Single vs. Best Multi-Agent Performance Single Best Multi Multi gain Multi loss +12.5% +22.7% +10.9% +17.0% -7.2% +7.8% -5.2% +19.3% -7.0% -3.8% 0%20%40%60%80%100% Test Pass Rate 0% 20% 40% 60% 80% 100% ADPR (b) Test Pass Rate & ADPR DeepSeek-V4 Qwen3.8-27B Qwen3.6-27B Qwen3.5-27B Qwen3.5-9B Gemma-4-26B Qwen3-Coder-Next Muse-Glimmer-30B GLM-4.7-Flash Nemotron-3.5 Final success Low Test Pass, Low ADPR High Test Pass, Low ADPR Low Test Pass, High ADPRHigh Test Pass, High ADPR Figure 2: Collaboration gains and hidden dependency failures. (a) For each model, SINGLE is com- pared with the highest task-macro Test Pass Rate achieved by any of the four multi-agent protocols; arrows show the signed gain. (b) Test Pass Rate versus ADPR for 190 ASYNC-MANAGER-RO modelâtask outcomes. The dashed line indicates equality between the two metrics, and the 50% boundaries provide a visual partition. Qwen3.5 9B Qwen3.5 27B Qwen3.6 27B Qwen3.8 27B 0% 20% 40% 60% 80% 100% Rate SCALEGENERATION (a) Test Pass Rate â Qwen3.5 9B Qwen3.5 27B Qwen3.6 27B Qwen3.8 27B 0% 20% 40% 60% 80% 100% (b) ADPR â Qwen3.5 9B Qwen3.5 27B Qwen3.6 27B Qwen3.8 27B 0.5 0.6 0.7 0.8 0.9 1.0 Normalized step (c) DRS â SingleSerialAsync-PrivateAsync-Manager-ROAsync-ManagerScale · Qwen3.5Generation · 27B Figure 3: Qwen scale and generation analysis under five matched execution protocols. The left re- gion compares Qwen3.5-9B and Qwen3.5-27B, while the right region compares Qwen3.5, Qwen3.6, and Qwen3.8 at a nominal scale of 27B. Panels report Test Pass Rate, ADPR, and normalized DRS; lower DRS indicates earlier dependency resolution. ment observed in standalone dependency resolution from Qwen3.6 to Qwen3.8 is not matched by a comparable gain under collaborative execution. The trajectory analysis reveals a further distinction that final ADPR alone cannot capture. Under ASYNC-MANAGER, normalized DRS decreases from 0.787 for Qwen3.5 to 0.641 for Qwen3.6, indicating earlier dependency resolution, but rises again to 0.719 for Qwen3.8 despite their similar final ADPR. Two models can therefore reach comparable final dependency coverage while arriving there through different coordination trajectories. 5.4PROTOCOL EFFECTS AND DEPENDENCY CLOSURE DYNAMICS The Qwen analysis shows that stronger standalone capability does not uniformly translate into stronger collaboration, and that similar final dependency coverage can arise from different reso- 7 AsynCodeBench lution trajectories. We therefore next examine the other side of the interaction: how the execution protocol shapes both the amount and timing of dependency resolution. Figure 4(a) examines how execution protocols affect dependency resolution by comparing each collaborative setting against SINGLE under the same model. The results reveal substantial model- dependent variation: a protocol that improves dependency resolution for one model may degrade it for another, and even the same model can exhibit opposite outcomes across protocols. For example, Qwen3.6-27B ranges from a 14.9% decrease under ASYNC-PRIVATE to a 26.3% improvement under ASYNC-MANAGER, yielding a 41.2% spread. Moreover, no collaborative protocol consistently improves dependency resolution across all models. These results indicate that collaboration gains depend not only on the underlying model but also on how multi-agent execution is organized. Figure 4(b) reveals a further pattern that is invisible from final ADPR alone. Using the first- resolution checkpoints captured by DRS, we plot the cumulative fraction of dependencies that have achieved closure over normalized logical progress. The horizontal axis represents each checkpointâs relative position in its execution trajectory, while the vertical axis measures the cumulative fraction of dependencies ever resolved by that point. Across all four collaborative protocols, the curves exhibit a pronounced increase in the middle-to- late portion of the observed trajectories. Across the aggregated trajectories, a substantial fraction of eventual dependency closures concentrates within 0.50 < x †0.75. We refer to this region as a representative hopping window, while noting that the onset and duration of the hopping process vary across individual models and protocols. This interval accounts for 34.9%â67.8% of eventual closure mass across protocols. This observation suggests that multi-agent collaboration cannot be fully characterized by its initial decomposition or final outcome. When dependencies become resolved is itself an important dimension of collaboration capability. Appendix F.1 further examines the prevalence of this pattern and the effect of checkpoint normalization. The aggregate trajectories in Figure 4(b) suggest that the hopping process is closely tied to the for- mation of substantial dependency resolution. Figure 5 makes this relationship more explicit at the model level. For Gemma-4-26B, protocols whose trajectories remain nearly flat establish little de- pendency closure, whereas ASYNC-MANAGER develops a pronounced hopping phase and reaches a substantially higher final ADPR. Qwen3-Coder-Next shows that the location of this phase is not fixed: its ASYNC-MANAGER trajectory undergoes a sharp closure burst much earlier in execution, around x â 0.4, before settling into its final plateau. DeepSeek-V4 exhibits another form of the same phenomenon, with all four protocols undergoing pronounced bursts of dependency closure at later stages of their trajectories. Across these cases, the common factor is not a particular temporal interval, but the emergence of a concentrated phase in which a substantial share of dependencies becomes resolved. This pattern also holds across the full set of 40 modelâprotocol configurations: 28 exhibit closure growth within the representative interval 0.50 < x †0.75, including all 16 configurations reaching at least 20% final ADPR, while configurations with little final dependency resolution often remain nearly flat throughout execution. The onset and duration of the hopping process nevertheless vary substan- tially across models and protocols, as illustrated by the earlier burst of Qwen3-Coder-Next. These results suggest that substantial dependency resolution is often established through a concentrated hopping process rather than steady accumulation, while the location of that process is configuration- dependent. Complete trajectories for all models are provided in Appendix F.2. 5.5QUALITYâCOST TRADE-OFFS Beyond collaboration effectiveness, we examine the token costs incurred by different execution protocols. Multi-agent execution introduces additional computational overhead, yet greater token consumption does not necessarily translate into better dependency resolution. To examine this trade- off, we construct within-model Pareto frontiers, identifying protocols that offer quality gains not attainable at a lower token cost. We use mean reported tokens per task as the cost measure and independently evaluate quality using Test Pass Rate and ADPR. Figure 6 illustrates this metric-dependent trade-off using Nemotron-3.5.Compared with Single, Async-Private consumes more tokens and improves Test Pass Rate from 28.99% to 32.94%, but achieves the same ADPR of 1.05%.It therefore lies on 8 AsynCodeBench Serial Async-Private Async-Manager-RO Async-Manager Qwen3.5-9B Gemma-4-26B Nemotron-3.5 Qwen3.5-27B GLM-4.7-Flash Muse-Glimmer-30B Qwen3-Coder-Next Qwen3.6-27B Qwen3.8-27B DeepSeek-V4 -8.2-14.2-13.2-11.4 +0.0+0.0+4.9+19.1 -1.1+0.0+1.8+6.7 +2.6-5.1+7.9+19.3 +7.7-1.1+1.8+5.3 -4.6-14.0-15.8+7.7 -23.7-27.2-25.4-14.2 +12.3-14.9-14.0+26.3 -19.3-21.9-19.3-12.3 -12.3-33.3-14.7-7.0 (a) ADPR change from Single 0.00.20.40.60.81.0 Normalized logical step 0 10 20 30 40 Cumulative dependencies ever resolved (%) (b) Dependency closure dynamics Serial Async-Private Async-Manager-RO Async-Manager Final ADPR Hopping window â30 â20 â10 0 10 20 30 Î ADPR Figure 4: Protocol effects and dependency-closure dynamics. (a) ADPR change relative to Single across models and multi-agent protocols. (b) Cumulative dependency closure over normalized logi- cal steps k/T , showing a recurrent middle-to-late hopping window (0.50 < k/T †0.75). 0.00.20.40.60.81.0 Normalized logical step 0 20 40 60 80 100 Cumulative dependencies ever resolved (%) (a) Gemma-4-26B Serial Async-Private Async-Manager-RO Async-Manager Final ADPR 0.00.20.40.60.81.0 Normalized logical step (b) Qwen3-Coder-Next Early closure burst 0.00.20.40.60.81.0 Normalized logical step (c) DeepSeek-V4 Hopping window Figure 5: Representative dependency-closure trajectories across execution protocols. Each panel shows the cumulative fraction of dependencies ever resolved over normalized logical steps for one model. The shaded region marks the hopping window. 681012141618 Mean reported tokens per task (M) 0 2 4 6 8 ADPR (%) ADPR frontier (left axis) Test Pass frontier (right axis) Single Serial Async-Private Async-Manager-RO Async-Manager Single Manager-RO 28 30 32 34 36 38 40 Test Pass Rate (%) Serial Private Manager Figure 6:Qualityâcost Pareto frontiers for Nemotron-3.5 under ADPR and Test Pass Rate. Both use identical protocols and token costs, yet identify different non-dominated configurations. Marker shapes indicate execution protocols, and lines connect their respective Pareto frontiers. the Test Pass Rate frontier but is dominated by Single under ADPR. Conversely, Async- Manager-RO improves ADPR to 2.81%, plac- ing it on the ADPR frontier. It is dominated by Async-Private under Test Pass Rate, as it incurs higher token costs while achieving lower test performance. These differences reveal that the qualityâcost assessment of collaboration depends on the metric used to measure quality.Across all ten models, five exhibit different within-model Pareto frontiers under the two metrics, as de- tailed in Appendix G. Extending our earlier observations of metric mismatch, these results show that task-level performance alone may overlook dependency-resolution gains when evaluating the resource efficiency of multi- agent collaboration. ADPR provides a com- plementary, dependency-centric perspective on these trade-offs. 9 AsynCodeBench 6CONCLUSION Asynchronous multi-agent software engineering introduces a challenge beyond task completion: independently produced components must be coordinated correctly. AsynCodeBench studies this challenge through repository-grounded dependencies, making collaboration and its evolution di- rectly measurable during execution. Our results reveal a persistent gap between standalone coding capability and collaborative performance, while also showing that conventional task-level outcomes can hide unresolved coordination failures. By exposing these hidden dependencies and their dy- namics over execution, AsynCodeBench provides a complementary view of multi-agent software engineering beyond final task success. We hope this benchmark supports the development of coding agents that are effective both individually and collaboratively. AI USE STATEMENT We used OpenAI ChatGPT to assist with language polishing and improving the clarity of the manuscript, literature retrieval and discovery, brainstorming and refinement of research ideas, pro- viding feedback on research methodology and experimental design, and code development and de- bugging. All methodological choices, benchmark construction, experimental configurations, result verification, and scientific conclusions were independently reviewed, determined, and validated by the authors. The authors take full responsibility for the content of this paper. REFERENCES Anthropic. Building effective agents. Anthropic Engineering, December 2024. URL https: //w.anthropic.com/engineering/building-effective-agents. Published December 19, 2024; accessed September 3, 2026. Anthropic.Claudeopus4.6systemcard.Technicalreport,An- thropic,February2026a.URL https://w-cdn.anthropic.com/ c788cbc0a3da9135112f97cdf6dcd06f2c16cee2.pdf. Section 2.21.3.2 describes a multi-agent evaluation in which the orchestrator has no direct tools and delegates to subagents. Anthropic.How and when to use subagents in claude code.Claude Blog, April 2026b. URL https://claude.com/blog/subagents-in-claude-code. Published April 7, 2026; accessed September 3, 2026. Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineer- ing agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id= nMpJoVmRy1. Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, and Alexander Golubev. SWE- rebench v2: Language-agnostic SWE task collection at scale.In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id= UCAda9kS57. Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Ti- wari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? arXiv preprint arXiv:2503.13657, 2025. doi: 10.48550/arXiv.2503.13657. URL https://arxiv.org/ abs/2503.13657. Igor Costa. Agentspawn: Adaptive multi-agent collaboration through dynamic spawning for long- horizon code generation, 2026. URL https://arxiv.org/abs/2602.07072. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, 10 AsynCodeBench Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?, 2025. URL https://arxiv.org/abs/2509.16941. Giuseppe Destefanis and Tomaso Aste. When agents coordinate: Measuring coordination in multi- agent ai coding, 2026. URL https://arxiv.org/abs/2608.16801. Jingzhe Ding, Shengda Long, Changxin PU, Ge Zhang, zhou huan, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, FEI HU, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Chen Qizhi, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Minghao Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Haoran Wang, Zihan Wang, Weihao Xie, Yishuo Yuan, Jiayu Zhang, Yunfei Zhao, He Zhu, LIYA ZHU, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, and Wenhao Huang. NL2repo- bench: Towards long-horizon repository generation evaluation of coding agents. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/ forum?id=wqQam1muOQ. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Kr- ishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodee- val: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=wgDcbBMSfh. Benedetta Donato, Noah Hagar-Dent, Aaron Worsnop, Leonardo Mariani, and Valerio Terragni. An evaluation of role-based multiagent code generation on repository-scale problems. IEEE Software, 43(5):56â64, 2026. ISSN 1937-4194. doi: 10.1109/ms.2026.3701516. URL http://dx.doi. org/10.1109/MS.2026.3701516. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-one: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468, 2024.doi: 10.48550/arXiv.2411.04468.URL https: //arxiv.org/abs/2411.04468. Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam. Refactorbench: Evaluating stateful reasoning in language agents through code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=NiNIthntx7. Jiayi Geng and Graham Neubig. Effective strategies for asynchronous software engineering agents, 2026. URL https://arxiv.org/abs/2603.21489. Xuehang Guo, Xingyao Wang, Yangyi Chen, Sha Li, Chi Han, Manling Li, and Heng Ji. SyncMind: Measuring agent out-of-sync recovery in collaborative software engineering. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 20993â21066. PMLR, 13â19 Jul 2025. URL https://proceedings.mlr.press/v267/guo25l.html. Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, and Qinhuai Na. Swe refactor bench: Can coding agents complete a long- horizon, whole-repository stack migration?, 2026. URL https://arxiv.org/abs/2608. 23564. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, zili wang, Steven Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J Ì urgen Schmidhuber.Metagpt: Meta programming for a multi-agent collabora- tive framework.In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, p. 23247â23275, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/file/ 6507b115562b0a305f1958c87355a-Paper-Conference.pdf. 11 AsynCodeBench Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=VTF8yNQM66. Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann Kaspar Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J Ryan, Jiaxin Pei, and Diyi Yang. Cooperbench: Why coding agents cannot be your teammates yet. In Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=qo1hehBPVf. Yubin Kim, Chanwoo Park, Taehan Kim, Eugene Park, Samuel Schmidgall, Salman Rahman, Chunjong Park, Cynthia Breazeal, Xin Liu, Hamid Palangi, Hae Won Park, and Daniel Mc- Duff. Teambench: Evaluating agent coordination under enforced role separation, 2026. URL https://arxiv.org/abs/2605.07073. Rajesh Kumar, Waqar Ali, Junaid Ahmed, Najma Imtiaz Ali, and Shaban Usman. Agentforge: Execution-grounded multi-agent llm framework for autonomous software engineering, 2026. URL https://arxiv.org/abs/2604.13120. Tue Le, Minh V. T. Thai, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios, 2026. URL https: //arxiv.org/abs/2512.18470. Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation, 2025.URL https://arxiv.org/abs/2503. 06680. Jie Liu, Guohua Wang, Ronghui Yang, Jiajie Zeng, Mengchen Zhao, and Yi Cai. RTADev: Inten- tion aligned multi-agent framework for software development. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Com- putational Linguistics: ACL 2025, p. 1548â1581, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.80. URL https://aclanthology.org/2025.findings-acl.80/. Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2024. URL https://arxiv.org/abs/2306.03091. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=xZXhFg43EI. Mariama Celi Serafim De Oliveira, Motunrayo Osatohanmen Ibiyo, Marco Gianrusso, Claudio Di Sipio, Davide Di Ruscio, and Phuong T. Nguyen. Developing llm-based multi-agent systems in software engineering: A mixed-method experience report, 2026. URL https://arxiv.org/ abs/2608.11965. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-gym. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id= Cq1BNvHx74. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chat- Dev: Communicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), p. 15174â15186, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.810. URL https://aclanthology.org/2024.acl-long.810/. 12 AsynCodeBench Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop De- oras, Giovanni Zappella, and Laurent Callot. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents, 2025. URL https://arxiv.org/abs/2504. 08703. Dongsheng Shi, Yue Li, Xin Yi, and Linlin Wang. Emr: Self-evolving medical multi-agent system via experience mining and reuse. arXiv preprint arXiv:2609.15161, 2026. Konstantinos Vergopoulos, Mark Niklas M Ì uller, and Martin Vechev. Automated benchmark gen- eration for repository-level coding tasks, 2025. URL https://arxiv.org/abs/2503. 07701. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for ai software developers as generalist agents, 2025. URL https://arxiv.org/abs/2407.16741. Yifei Wang, Ruiyin Li, Peng Liang, Qiong Feng, Zengyang Li, Mojtaba Shahin, and Arif Ali Khan. CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation. arXiv preprint arXiv:2606.22082, 2026. Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, and Shengzhi Li. Swe-together: Evaluating coding agents in interactive user sessions. arXiv preprint arXiv:2606.29957, 2026. URL https://arxiv. org/pdf/2606.29957. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://arxiv.org/abs/2405.15793. John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024b. URL https://arxiv.org/abs/2410.03859. John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum? id=63iVrXc8cC. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, RUI LONG, Ming Ding, and liang xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview. net/forum?id=MhBZzkz4h9. Kaituo Zhang, Zhen Xiong, Mingyu Zhong, Zhimeng Jiang, Zhouyuan Yuan, Zhecheng Li, and Ying Lin. Are tools all we need? unveiling the tool-use tax in llm agents, 2026a. URL https: //arxiv.org/abs/2605.00136. Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang, Xin Yang, Yueqi Zhu, Xuxin Cheng, Cao Liu, Ke Zeng, Terry Jingchen Zhang, and Wenyuan Jiang. SILO-BENCH: A scalable environment for evaluating distributed coordination in multi-agent LLM systems. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 29379â29398, San Diego, California, United States, July 2026b. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1354. URL https://aclanthology. org/2026.acl-long.1354/. 13 AsynCodeBench Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gall Ì e, and Alexan- der M Rush. Commit0: Library generation from scratch, 2024. URL https://arxiv.org/ abs/2412.01769. 14 AsynCodeBench ATASK DETAILS A.1TASK-LEVEL DEPENDENCY GRAPHS For each benchmark task, we first identify the repository-grounded implementation units that must be modified or completed to satisfy the task specification. We then construct a directed graph G = (V,A), whereV contains the implementation units andA contains directed producerâconsumer dependen- cies among them. An arc d = (u,v) â A is included only when the correctness of consumer v depends on behavior, state, an interface, or another artifact provided by producer u. The direction therefore follows the flow of required behavior from producer to consumer. Dependency arcs are grounded in repository evidence rather than inferred from file proximity or task decomposition alone. For each candidate relation, we inspect the relevant source code, referenced symbols or interfaces, task decomposition, and available public tests to verify that the consumerâs correctness genuinely depends on the producer. Relations are excluded when the two implementa- tion units merely co-occur in the same task but can be implemented correctly without satisfying a cross-unit dependency. The resulting graphs are finalized before model evaluation and remain fixed across all models and execution protocols. Each dependency is manually reviewed for both its direction and supporting evidence, and disagreements during construction are resolved through additional inspection of the repository and associated tests. Freezing the graph before evaluation prevents model behavior or observed outcomes from influencing which dependencies are included. We present the dependency graph structure for each task. As shown in Figure 7, AsynCodeBench contains nine types of dependency relations. Each dependency specifies well-defined upstream and downstream implementation units, providing the basis for dependency-level evaluation with ADPR. Task TVM 20073, Parsel, Filesystem_spec TVM 20107, Requests Portalocker, Tinydb, Wcwidth, TVM Cachetools Cookiecutter Flask, pexpect Imapclient Marshmallow Python-rsa, Simpy Dependency Implemented Unit Figure 7: Task-level dependency graphs in AsynCodeBench. Across the benchmark, we identify nine types of dependency relations, each defined by explicit upstream and downstream implementa- tion units. These dependencies form the basis for dependency-level evaluation with ADPR. A.2TASK INVENTORY AND PROVENANCE AsynCodeBench contains 19 tasks from 16 public repositories, covering 64 repository-grounded implementation units and 52 directed dependency edges.Fifteen tasks are derived from 15 AsynCodeBench Table 1: Complete AsynCodeBench task inventory and provenance. Agents denotes the number of specialists used in a multi-agent condition; Files denotes the number of distinct implementation files writable by specialists; Edges denotes the number of registered dependency edges; Tests denotes the number of test files included in the final evaluator; and Ov. denotes the number of checksum-pinned public overlays. Domain abbreviations are F/L (framework and libraries), S/S (systems and storage), C/IR (compiler and IR), N/P (network protocols), and SEC (security). TaskSource recordDomainLang.Agents Files Edges Tests Ov. cachetoolsCommit0F/LPy225120 deprecatedCommit0F/LPy22380 portalockerCommit0S/SPy22351 tinydbCommit0S/SPy27381 wcwidthCommit0S/SPy22240 requestsCommit0N/PPy313353 simpyCommit0S/SPy493100 parselCommit0F/LPy35261 filesystem specCommit0S/SPy35244 marshmallowCommit0F/LPy393124 imapclientCommit0N/PPy39395 pexpectCommit0S/SPy49341 flaskCommit0F/LPy4103711 python-rsaCommit0SECPy49380 cookiecutterCommit0F/LPy4153220 apache-tvm-20018 TVM PR #20018 C/IRPy+C++3282141 apache-tvm-20073 TVM PR #20073 C/IRPy+C++37221 apache-tvm-20107 TVM PR #20107 C/IRPy+C++322251 apache-tvm-20153 TVM PR #20153 C/IRPy36232 AsynCodeBench 32% 32% 21% 11% 5% Framework & Library Engineering Systems & Storage Compiler / IR Engineering Network Protocols Security Figure 8: Domain distribution of tasks in AsynCodeBench. Commit0 (Zhao et al., 2024) and four from merged Apache TVM pull requests; 16 tasks are Python-only, while three involve both Python and C++. Evaluating ev- ery task under the five matched execution protocols pro- duces 95 taskâprotocol scenarios. Across all task packages, the evaluators reference 148 task-local public test files and collect 4,884 pytest items in completed-state sanity runs. The 52 dependency edges are instrumented with upstream, downstream, and integrated checker groups, yielding 156 checker groups and 508 selector references in total: 125 upstream, 162 downstream, and 221 integrated. After within-task de- duplication, these correspond to 274 task-qualified unique test selectors. Task-level evaluators measure overall im- plementation correctness, whereas Dependency Checkers provide edge-level observations over the execution trajec- tory; individual selectors may therefore be shared across multiple edges or checker groups. A.3HUMAN REVIEW DETAILS Three reviewers independently examined each benchmark task to verify that it contains meaningful inter-component dependencies and admits a natural decomposition across agents. Each reviewer assessed three fields: include, indicating whether the task satisfies the inclusion criteria of Asyn- CodeBench; parallelizability label, categorized as parallelizable, partially paralleliz- able, or effectively serial; and rationale, providing the justification for the decision. For each dependency edge, reviewers also verified its validity and direction, together with the supporting repository evidence from source code, symbols, tests, or executable Dependency Checkers; dis- agreements were resolved through discussion. Reviewer judgments were additionally compared 16 AsynCodeBench Table 2: Human-review sheet used for task validation. Each task is reviewed independently by three reviewers. Task Review Sheet Task <task-name> ReviewerReviewer 1 Include⥠Yes ⥠No Parallelizability ⥠Parallelizable ⥠Partially parallelizable ⥠Effectively serial RationaleReviewer justification for whether the task contains meaningful dependencies and can be naturally decomposed across agents. Table 3: Execution protocols in AsynCodeBench. âWritable managerâ denotes scope-constrained patches produced in a private manager worktree; the manager never directly modifies the integrated workspace. ProtocolDecomp.ConcurrentManagerWritable Manager SINGLEâ SERIALââ ASYNC-PRIVATEââ ASYNC-MANAGER-RO ââ ASYNC-MANAGERâ against the benchmark dependency graph, and any discrepancy was explicitly recorded. Table 2 provides an example review sheet illustrating the human-review process used for benchmark valida- tion. The results are in our anonymized repository. BPROTOCOL DETAILS This section provides the full specification of the five execution protocols summarized in Table 3. Across all protocols, we hold fixed the initial repository state, task instruction, model configura- tion, final evaluator, and Dependency Checkers. The four multi-agent protocols additionally share the same manifest-defined specialist roles and writable paths, and all specialist artifacts are scope- validated before integration. The protocols differ only in how specialist work is scheduled, what intermediate state is visible, whether a manager coordinates execution, and whether the manager may submit scoped production-code patches. SINGLE.SINGLE assigns the complete task to one agent with access to the full repository and task instruction. The agent edits the task workspace directly and produces the final implementation without specialist decomposition, handoffs, or manager intervention. This protocol serves as the full-task ownership baseline, avoiding cross-agent coordination while also forgoing specialization and concurrent execution. Because only the final integrated workspace is observed, SINGLE is included in Test Pass Rate and ADPR comparisons but excluded from trajectory-based metrics such as DRS. SERIAL.SERIAL uses the manifest-defined specialist decomposition but executes specialists se- quentially in dependency order. A downstream specialist begins only after the relevant upstream ar- tifact has been scope-validated and integrated, and therefore observes the latest integrated repository state when starting its work. This protocol preserves explicit specialist handoffs while minimizing stale-context effects, at the cost of concurrency. ASYNC-PRIVATE.ASYNC-PRIVATE uses the same specialist roles and writable scopes as SE- RIAL, but runs specialists concurrently in isolated private worktrees created from the same initial 17 AsynCodeBench repository state. Specialists cannot observe one anotherâs in-flight changes or receive adaptive feed- back during their initial execution. Completed artifacts are scope-validated and integrated in a deter- ministic dependency order. The contrast with SERIAL therefore isolates the effect of asynchronous execution under the same task decomposition: dependent implementations are produced indepen- dently and must compose only after integration. ASYNC-MANAGER-RO.ASYNC-MANAGER-RO retains concurrent private specialist execution and introduces a read-only manager. The manager may inspect the task specification, dependency graph, repository state, specialist artifacts, merge outcomes, and checker diagnostics; it may also plan work, provide feedback, request scoped follow-up attempts, and coordinate integration. It cannot modify production code, so every production patch must originate from a scope-constrained specialist. This separation follows the orchestratorâworker pattern used in prior multi-agent systems (Fourney et al., 2024; Anthropic, 2024), where a coordinating agent delegates work to specialized agents; related planner-mediated coordination has also been explored in domain-specific multi-agent sys- tems (Shi et al., 2026). Role-specific tool permissions provide a further mechanism for separating coordination from execution (Anthropic, 2026b). A related design appears in Anthropicâs reported DeepSearchQA configuration, where the orchestrator acts through delegated subagents rather than direct tools (Anthropic, 2026a). We use read-only authority as an experimental control: it allows explicit coordination without turning the manager into an additional full-context coding agent, while keeping production changes attributable to specialists. ASYNC-MANAGER.ASYNC-MANAGER preserves the same coordination workflow but addition- ally permits the manager to submit scoped production-code interventions after specialist integration. The manager operates in its own private worktree, and its writable region is restricted to the union of manifest-defined specialist production paths. Tests, Dependency Checkers, evaluators, task man- ifests, Git metadata, and out-of-scope files remain protected. The manager never edits the integrated workspace or specialist worktrees directly. Instead, each proposed patch is recorded, scope-validated, checked against its base revision, committed in the manager worktree, and integrated only if all policy checks pass. Each accepted state-changing intervention creates a new integrated-workspace checkpoint; rejected or no-change interventions do not alter the trajectory. This protocol tests whether coordination benefits from bounded direct repair in addition to planning, feedback, and delegation. These are controlled system-level contrasts, not an assumed capability ordering. More generally, agentic protocols can themselves introduce execution overhead or alter model behavior beyond the underlying model capability (Zhang et al., 2026a), motivating our explicit comparison of execution protocols rather than treating the surrounding agent framework as neutral. CINTEGRATED-WORKSPACE CHECKPOINT CONSTRUCTION Figure 9 illustrates how dependency checkpoints are constructed during execution. The harness records checker outcomes in both private specialist workspaces and the integrated workspace. Pri- vate artifact checkpoints are used only for local diagnostics and are not included in the dependency trajectory. Instead, we retain the integrated-workspace observations recorded after specialist inte- gration attempts, accepted state-changing manager interventions when applicable, and final eval- uation. These retained observations are ordered by execution order and indexed consecutively as t = 1,...,T . Because the number of integration events and accepted manager interventions varies across runs, T is run-specific. Consequently, the checkpoint index represents logical progress through the ob- served integrated-workspace trajectory rather than a model turn, tool call, unit of wall-clock time, or fixed amount of computation. SINGLE exposes only the final integrated workspace and is therefore evaluated with ADPR but excluded from temporal DRS analysis. 18 AsynCodeBench Table 4: Models evaluated in AsynCodeBench. Active denotes the approximate number of parame- ters activated per token. For dense models, all language-model parameters are active. Paper nameEvaluated checkpoint Archi- tecture TotalActive Qwen3.5-9BQwen/ Qwen3.5-9B Dense9B9B Qwen3.5-27BQwen/ Qwen3.5-27B Dense27B27B Qwen3.6-27BQwen/ Qwen3.6-27B Dense27B27B Qwen3.8-27BQwen/ Qwen3.8-27B Dense27B27B Qwen3-Coder- Next Qwen/ Qwen3-Coder- Next-FP8 MoE80B3B DeepSeek- V4-Flash deepseek/ deepseek-v4- flash-0731 MoE284B13B Gemma-4-26Bgoogle/ gemma-4-26B- A4B-it MoE25.2B3.8B Muse-Glimmer- 30B meta-models/ Muse-Glimmer- 30B DenseâŒ29.6BâŒ29.6B GLM-4.7-Flashzai-org/ GLM-4.7-Flash MoE30B3B Nemotron- 3.5-Lightning nvidia/ NVIDIA-Nemotron- 3.5-Lightning- 30B-A3B-BF16 Hybrid MoE 30B3B DEVALUATED MODEL CHECKPOINTS Table 4 reports the exact model checkpoints used in our experiments and their corresponding names in the main paper. Because the model set includes both dense and mixture-of-experts (MoE) ar- chitectures, we report total and per-token activated parameter counts separately. For dense models, the activated parameter count equals the language-model parameter count. For MoE models, the activated count represents the approximate subset of parameters used for each token and can be substantially smaller than the total model size. Weight formats such as FP8 and BF16 describe numerical precision and do not change the parameter count. ETEST PROGRESS AND DEPENDENCY RESOLUTION ACROSS PROTOCOLS Section 5.2 uses ASYNC-MANAGER-RO to illustrate the mismatch between task-level test progress and dependency resolution. Figure 10 extends the same analysis to the remaining four protocols. Each panel contains 190 modelâtask outcomes from the same ten models and 19 tasks, with identical axes and visual encodings to enable direct comparison. The same asymmetric mismatch appears across protocols. As shown in Table 5, high-test, low- ADPR outcomes account for 22.1â27.4% of observations under every protocol. By contrast, the low- test, high-ADPR region is empty for four protocols and contains only three outcomes under ASYNC- MANAGER. Thus, substantial test progress frequently occurs without corresponding dependency closure, whereas the reverse pattern is rare. 19 AsynCodeBench Table 5: Distribution of modelâtask outcomes across the four regions in Figure 10 and Figure 2(b). Each protocol contains 190 outcomes. Entries report count (percentage). The 50% threshold is used only as a descriptive visual partition. Blue cells denote the best value according to the indicated direction, while orange cells highlight values discussed in the accompanying analysis. Protocol Low Test / Low ADPRâ Low Test / High ADPR High Test / Low ADPRâ High Test / High ADPRâ SINGLE97 (51.1%)0 (0.0%)42 (22.1%)51 (26.8%) SERIAL104 (54.7%)0 (0.0%)44 (23.2%)42 (22.1%) ASYNC-PRIVATE116 (61.1%)0 (0.0%)49 (25.8%)25 (13.2%) ASYNC-MANAGER-RO105 (55.3%)0 (0.0%)52 (27.4%)33 (17.4%) ASYNC-MANAGER83 (43.7%)3 (1.6%)50 (26.3%)54 (28.4%) Table 6: Correlation and mismatch diagnostics by protocol. Percentages are computed over 190 modelâtask outcomes per protocol. The signed gap is Test Pass Rate minus ADPR; the absolute gap ignores direction. Arrows indicate the preferred direction where applicable.Blue cells denote the best value according to the indicated direction, whileorange cells highlight values discussed in the accompanying analysis. MetricSingleSerialAsync-P.Manager- RO Manager Pearson r0.7160.6670.5990.6170.628 Spearman Ï0.7370.6510.5590.6130.657 Mean Test Pass Rateâ51.0%47.9%41.9%48.0%57.4% Mean ADPRâ27.4%22.7%14.2%18.8%31.3% Mean signed gapâ23.7%25.2%27.8%29.2%26.1% Test Pass Rate > ADPRâ62.6%70.5%77.9%71.6%67.4% Test Passâ„ 80%, ADPR < 50%â12.6%12.6%15.8%16.8%16.3% Test Passâ„ 80%, ADPR = 0â10.0%12.1%12.1%13.7%14.2% Final task successâ21.6%13.2%8.4%13.2%21.1% Table 6 provides complementary continuous diagnostics. Test Pass Rate and ADPR remain posi- tively associated under all five protocols, with Pearson correlations of 0.599â0.716 and Spearman correlations of 0.559â0.737. However, Test Pass Rate exceeds ADPR in 62.6â77.9% of outcomes, with mean signed gaps of 23.7â29.2%. Moreover, 12.6â16.8% of outcomes achieve at least 80% Test Pass Rate while resolving fewer than half of their declared dependencies. The discrepancy observed in Section 5.2 is therefore not specific to ASYNC-MANAGER-RO. FDIAGNOSING DEPENDENCY CLOSURE DYNAMICS F.1AGGREGATE CLOSURE CONCENTRATION AND NORMALIZATION Figure 4(b) shows that first dependency closures are not distributed uniformly over the aggregated execution trajectories. A particularly strong concentration appears within 0.50 < x†0.75, where x denotes the normalized logical step. We treat this interval as a representative region of the aggregate hopping phenomenon rather than as a fixed temporal definition, since the onset and duration of concentrated closure may vary across individual models and protocols. Table 7 quantifies this aggregate concentration. The first three columns report the cumulative frac- tion of dependencies that have achieved closure by the halfway point, the three-quarter point, and at least once during execution, respectively. The final column reports the fraction of eventual clo- sure mass accumulated within the representative interval. Under SERIAL, for example, cumulative closure increases from 4.9% to 20.3% between x = 0.50 and x = 0.75, accounting for 67.8% of its eventual closure mass. Across the four protocols, 34.9%â67.8% of eventual closure mass is accumulated within this interval, confirming that a substantial portion of dependency resolution is concentrated within a relatively short segment of the execution trajectory. 20 AsynCodeBench Table 7: Cumulative dependency closure and hopping-window concentration across execution pro- tocols. Cumulative closure (%) ProtocolBy 50%By 75%Ever closedClosure in (0.50, 0.75] (%) SERIAL4.920.322.767.8 ASYNC-PRIVATE4.612.614.256.5 ASYNC-MANAGER-RO3.011.918.947.3 ASYNC-MANAGER13.824.731.334.9 Table 8: Effect of checkpoint normalization on the hopping-window pattern. ProtocolPeak events from k = 2,T = 3First closure at raw k = 2 SERIAL68.9%58.6% ASYNC-PRIVATE73.9%60.2% ASYNC-MANAGER-RO66.7%38.4% ASYNC-MANAGER50.0%48.6% A natural concern is whether this concentration is induced by the normalization used to align trajec- tories of different lengths. Let k denote the first integrated checkpoint at which a dependency closes and T the total number of integrated checkpoints in the corresponding run, giving the normalized position x = k/T . Although this mapping allows trajectories with different numbers of checkpoints to be compared on a common scale, discrete checkpoint structures can accumulate at shared ratios such as 2/3. Table 8 shows that normalization indeed sharpens the aggregate peak: 50.0%â73.9% of first closures around this region correspond to k = 2,T = 3. However, the underlying concentration remains visible in raw checkpoint space. Across all four protocols, k = 2 is still the most common first- closure checkpoint, accounting for 38.4%â60.2% of eventual closures. Normalization therefore accentuates the observed hopping pattern but does not create the underlying burst of dependency closure. The aggregate concentration, however, does not imply that every modelâprotocol configuration fol- lows the same temporal pattern. Figure 11 exposes substantial heterogeneity beneath the protocol- level averages: while some configurations place a large share of their eventual first closures within 0.50 < x†0.75, others contribute little or no closure in this region. The interval therefore captures where closure is most strongly concentrated in aggregate, but not a fixed boundary that governs every execution. This distinction motivates a closer examination of the complete model-level trajec- tories in Section F.2, where we show that the hopping process can emerge earlier or later depending on the model and execution protocol. F.2MODEL-LEVEL VARIATION IN HOPPING DYNAMICS The variation in Figure 11 raises a natural question: if the aggregate hopping concentration is not shared uniformly across configurations, how does dependency closure actually unfold at the model level? Figure 12 shows the complete trajectories for the remaining seven models, complementing the three representative cases in Figure 5. The trajectories reveal that the main variation lies not in whether closure must occur within a par- ticular normalized interval, but in when and how strongly a concentrated closure phase emerges. Low-closure configurations, such as many protocols for Qwen3.5-9B, Nemotron-3.5, and GLM- 4.7-Flash, remain largely flat throughout execution. By contrast, configurations with substantial final closure exhibit pronounced bursts, but their timing differs considerably. For Qwen3.5-27B and Qwen3.6-27B, ASYNC-MANAGER begins accumulating dependencies well before the repre- sentative aggregate interval, whereas Qwen3.8-27B concentrates much of its closure later in the trajectory. Similar differences also appear across protocols within the same model. These trajec- tories therefore support a broader interpretation of hopping as a configuration-dependent phase of 21 AsynCodeBench Table 9: Comparison of within-model Pareto frontiers under ADPR and Test Pass Rate. The two quality metrics are evaluated using identical protocols and token costs. Only metric-exclusive con- figurations are listed. ModelSame membership?ADPR-only protocolsTest-Pass-only protocols DeepSeek-V4Yesâ Qwen3.8-27BYesâ Qwen3.6-27BYesâ Qwen3.5-27BYesâ Qwen3-Coder-NextYesâ Qwen3.5-9BNoâAsync-Manager Gemma-4-26BNoâSerial Muse-Glimmer-30BNoâSerial GLM-4.7-FlashNoSerialâ Nemotron-3.5NoAsync-Manager-ROAsync-Private concentrated dependency closure, whose onset and magnitude vary across models and execution protocols. GADDITIONAL QUALITYâCOST ANALYSIS To complement the Nemotron-3.5 case study in the main text, we extend the qualityâcost analysis to all ten evaluated models. For each model, we independently construct Pareto frontiers using ADPR and Test Pass Rate as alternative quality measures, while keeping the candidate protocols and their token costs fixed. A protocol is dominated if another protocol achieves equal or higher quality at equal or lower token cost, with at least one strict improvement. Table 9 summarizes the resulting differences in frontier membership. Five models exhibit different frontiers, yielding two ADPR-only and four Test-Pass-only configurations. Figure 13 presents the other four models with differing frontiers. For Qwen3.5-9B, Single achieves the highest ADPR while consuming the fewest tokens, leaving all collaborative protocols dominated under ADPR. However, Async-Manager substantially improves Test Pass Rate and therefore enters the corresponding frontier. A similar metric-dependent pattern appears in Gemma-4-26B and Muse- Glimmer-30B: Serial improves Test Pass Rate over Single but achieves equal or lower ADPR at a higher token cost, making it non-dominated only under Test Pass Rate. GLM-4.7-Flash exhibits the opposite pattern. Serial improves ADPR over Single and enters the ADPR frontier, whereas Single achieves higher Test Pass Rate at a lower token cost, dominating all collaborative protocols under the conventional metric. These cases demonstrate that improvements in test performance do not necessarily translate into dependency-resolution gains, and vice versa. Consequently, the same execution protocols can yield different non-dominated configurations depending on the quality metric used. Together with the Nemotron-3.5 case study, these results extend the metric mismatch observed in the main experiments to cost-aware evaluation, illustrating the complementary role of dependency-centric metrics. The complete numerical results underlying these comparisons are reported in Tables 10 and 11. HCASE STUDY: WHEN HIGH TEST PASS RATE MASKS UNRESOLVED DEPENDENCIES Figure 14 illustrates why task-level and dependency-level evaluation can lead to different views of the same execution. Under DeepSeek-V4 with ASYNC-MANAGER on TVM-20073, the final workspace passes 24 of 28 evaluator items, yielding a Test Pass Rate of 85.7%. From the final test outcome alone, the implementation appears largely complete. The dependency trajectory reveals a different structure behind these results. The task contains two dependencies: source-coordinate mapping and IRBuilder active-span state must both be incorpo- 22 AsynCodeBench rated into a shared span-propagation implementation. Source mapping is available from the first retained checkpoint, and both upstream requirements are satisfied by checkpoint 3. However, the downstream propagation logic remains unsatisfied after two subsequent attempts, leaving both de- pendencies unresolved at final evaluation and yielding an ADPR of 0/2 = 0%. The discrepancy arises because the two metrics count different units. The four failed evaluator items all exercise the same missing span-propagation behavior, whereas ADPR identifies the two unre- solved dependencies underlying those failures. Thus, several test failures that appear separately in the final evaluator trace back to a shared coordination bottleneck: the required upstream capabilities are available, but they are never successfully composed in the downstream implementation. This case highlights the complementary information provided by dependency-aware evaluation. Test Pass Rate summarizes repository-level correctness over evaluator items, while the dependency tra- jectory identifies whether required cross-agent dependencies are established and when they become available. The latter therefore exposes coordination structure that is not visible from the final test outcome alone. IDETAILED RESULTS OF ALL MODELS Tables 10 and 11 report the complete experimental results for all ten models under five execution protocols, covering Test Pass Rate, ADPR, normalized DRS, and mean token consumption per task. The results are aggregated over the same 19 benchmark tasks for each modelâprotocol configuration, providing the numerical basis for the model capability, protocol, and qualityâcost analyses in the main text. Test Pass Rate and ADPR are reported as percentages, with higher values indicating better performance, whereas lower DRS indicates earlier dependency resolution. DRS is not reported for Single, which has no intermediate integration checkpoints. Token consumption is expressed in millions per task; the marked Qwen3.5-9B Async-Manager result represents a lower bound due to one incomplete token record. We also report the Final Task Success Rate in Table 12. Unlike the Test Pass Rate, the Final Task Success Rate adopts a stricter criterion: a task is considered successful only if it achieves both a com- plete task pass and full dependency resolution (ADPR = 1). For each modelâprotocol combination, we report the average success rate across the 19 tasks. 23 AsynCodeBench Table 10: Complete experimental results for the Qwen model family. Test Pass Rate and ADPR are reported as percentages. Tokens denote mean reported tokens per task in millions. ProtocolTest PassâADPRâDRSâTokens (M)â Qwen3.5-9B Single24.4414.21â5.18 Serial15.835.960.9628.01 Async-Private15.130.001.0008.32 Async-Manager-RO28.441.050.99215.53 Async-Manager36.932.810.98815.48 â Qwen3.5-27B Single42.1312.28â4.13 Serial47.9114.910.9206.18 Async-Private31.157.190.9516.67 Async-Manager-RO49.3320.180.91312.41 Async-Manager59.1531.580.78714.00 Qwen3.6-27B Single57.4536.84â4.96 Serial64.0149.120.7548.35 Async-Private55.7921.930.8788.18 Async-Manager-RO63.3222.810.88013.88 Async-Manager76.7063.160.64114.49 Qwen3.8-27B Single93.6277.19â3.45 Serial79.5357.890.7178.02 Async-Private81.7655.260.7277.09 Async-Manager-RO82.3357.890.73312.78 Async-Manager86.6764.910.71913.73 Qwen3-Coder-Next Single51.1830.70â4.69 Serial39.777.020.9679.49 Async-Private33.553.510.9828.66 Async-Manager-RO28.835.260.97717.89 Async-Manager45.9716.490.88616.38 24 AsynCodeBench Specialist completes a round Artifact produced Private workspace: run U / D / I Artifact checkpoint: diagnostic only Artifact integration attempt Integrated workspace: run U / D / I Integration checkpoint: enters trajectory Accepted state-changing manager intervention? Continue Run U / D / I again Manager checkpoint: enters trajectory NoYes Next artifact / round Final evaluation Integrated workspace: run U / D / I Final checkpoint: enters trajectory Figure 9: Event-triggered checkpoint construction in AsynCodeBench. The artifact checkpoint eval- uates the three checker groups in a private specialist workspace and is used only for local diagnostics. Integrated checker outcomes are recorded at the gold-bordered trajectory checkpoints after integra- tion attempts, accepted state-changing manager interventions when applicable, and final evaluation. Only these ordered integrated-workspace outcomes form the dependency trajectory used by ADPR and DRS. 25 AsynCodeBench 0%20%40%60%80%100% Test Pass Rate 0% 20% 40% 60% 80% 100% ADPR (a) Single Low Test Pass, Low ADPR High Test Pass, Low ADPR Low Test Pass, High ADPRHigh Test Pass, High ADPR 0%20%40%60%80%100% Test Pass Rate 0% 20% 40% 60% 80% 100% ADPR (b) Serial Low Test Pass, Low ADPR High Test Pass, Low ADPR Low Test Pass, High ADPRHigh Test Pass, High ADPR 0%20%40%60%80%100% Test Pass Rate 0% 20% 40% 60% 80% 100% ADPR (c) Async-Private Low Test Pass, Low ADPR High Test Pass, Low ADPR Low Test Pass, High ADPRHigh Test Pass, High ADPR 0%20%40%60%80%100% Test Pass Rate 0% 20% 40% 60% 80% 100% ADPR (d) Async-Manager Low Test Pass, Low ADPR High Test Pass, Low ADPR Low Test Pass, High ADPRHigh Test Pass, High ADPR DeepSeek-V4 Qwen3.8-27B Qwen3.6-27B Qwen3.5-27B Qwen3.5-9B Gemma-4-26B Qwen3-Coder-Next Muse-Glimmer-30B GLM-4.7-Flash Nemotron-3.5 Final success Figure 10: Test Pass Rate versus ADPR under the four protocols not shown in Figure 2(b). Each point represents a modelâtask outcome and colors denote models. The dashed line indicates equality between Test Pass Rate and ADPR, while the 50% boundaries provide a common visual partition. All panels use identical axes and encodings. 26 AsynCodeBench Serial Async-Private Async-Manager-RO Async-Manager DeepSeek-V4 GLM-4.7-Flash Gemma-4-26B Muse-Glimmer-30B Nemotron-3.5 Qwen3-Coder-Next Qwen3.5-27B Qwen3.5-9B Qwen3.6-27B Qwen3.8-27B 60%58%36%43% 67%N/A50%0% 0%0%88%38% 70%33%50%22% N/A0%0%25% 100%100%0%0% 66%0%36%6% 38%N/A0%100% 50%35%54%24% 71%54%41%64% 0 20 40 60 80 100 Eventual closures (%) Figure 11: Modelâprotocol variation in hopping-window concentration. Each cell shows the share of eventual first closures occurring within the hopping window (0.50, 0.75]. N/A denotes no observed dependency closure. 0.00.20.40.60.81.0 0 20 40 60 80 100 (a) Qwen3.5-9B 0.00.20.40.60.81.0 (b) Nemotron-3.5 0.00.20.40.60.81.0 (c) Qwen3.5-27B 0.00.20.40.60.81.0 (d) GLM-4.7-Flash 0.00.20.40.60.81.0 Normalized logical step 0 20 40 60 80 100 (e) Muse-Glimmer-30B 0.00.20.40.60.81.0 Normalized logical step (f) Qwen3.6-27B 0.00.20.40.60.81.0 Normalized logical step (g) Qwen3.8-27B Cumulative dependencies ever resolved (%) SerialAsync-PrivateAsync-Manager-ROAsync-ManagerFinal ADPR Figure 12: Dependency-closure trajectories for the seven models not shown in the main-text case studies. Curves report cumulative dependencies ever resolved over normalized logical execution, and crosses denote final ADPR. The shaded region marks the representative aggregate concentration 0.50 < x†0.75. Individual trajectories show that the onset and magnitude of concentrated closure vary substantially across models and protocols. 27 AsynCodeBench 691215 0 4 8 12 ADPR (%) (a) Qwen3.5-9B 061218 0 5 10 15 20 (b) Gemma-4-26B 81216 0 8 16 24 (c) Muse-Glimmer-30B 36912 0 2 5 8 10 (d) GLM-4.7-Flash 18 24 30 36 24 30 36 42 48 40 45 50 55 30 35 40 45 Test Pass Rate (%) Mean reported tokens per task (M) ADPR frontier (left axes)Test Pass frontier (right axes) SingleSerialAsync-PrivateAsync-Manager-ROAsync-Manager Figure 13: Metric-dependent qualityâcost trade-offs for four additional models. Each panel shows the within-model Pareto frontiers under ADPR (blue, left axis) and Test Pass Rate (orange, right axis). Marker shapes distinguish execution protocols, and lines connect non-dominated configura- tions under each metric. Dominated configurations remain visible. Axis ranges are independently scaled across models. Source mapping ready at k= 1 IRBuilder state ready at k= 3 Shared span propagation not satisfied (two attempts) Integrated dependencies: 0 / 2 resolved 1 Mapping ready 2 IRBuilder merged 3 Both upstream requirements ready 4 Consumer attempt 1 5 Consumer attempt 2 6 0 / 2 dependencies resolved Retained integrated-workspace checkpoints (a) Upstream requirements are satisfied, but dependencies remain unresolved 0%25%50%75%100% Test Pass Rate = 85.7% ADPR = 0% 24 passed 4 failed 0 resolved / 2 unresolved 4 failed evaluator items same missing span-propagation behavior 2 dependencies unresolved (b) The same execution looks different at task and dependency levels Figure 14: Case study of DeepSeek-V4 under ASYNC-MANAGER on TVM-20073. Although both upstream requirements become available during execution, the shared downstream span-propagation logic remains unsatisfied after two attempts, leaving both dependencies unresolved. The final workspace nevertheless passes 24 of 28 evaluator items (85.7% Test Pass Rate), while ADPR re- mains 0%, illustrating how task-level performance can obscure unresolved cross-agent dependen- cies and the coordination failures behind them. 28 AsynCodeBench Table 11: Complete experimental results for the remaining model families. Test Pass Rate and ADPR are reported as percentages. Tokens denote mean reported tokens per task in millions. ProtocolTest PassâADPRâDRSâTokens (M)â DeepSeek-V4 Single93.6478.95â2.54 Serial87.8466.670.6656.89 Async-Private75.7545.610.7696.92 Async-Manager-RO80.3664.210.74811.61 Async-Manager89.8571.930.69013.65 Gemma-4-26B Single26.511.05â0.93 Serial32.651.050.9922.49 Async-Private26.371.050.9923.14 Async-Manager-RO36.405.960.9723.64 Async-Manager49.1820.180.90520.20 Muse-Glimmer-30B Single47.6720.35â4.83 Serial49.4815.790.9218.32 Async-Private38.816.320.9617.99 Async-Manager-RO41.674.560.97614.06 Async-Manager55.4528.070.83216.08 GLM-4.7-Flash Single44.771.05â3.50 Serial33.858.770.9517.06 Async-Private28.200.001.0007.10 Async-Manager-RO37.582.810.98312.34 Async-Manager34.686.320.95713.79 Nemotron-3.5 Single28.991.05â5.72 Serial28.140.001.0009.82 Async-Private32.941.050.9929.82 Async-Manager-RO31.502.810.98014.04 Async-Manager39.897.720.95917.79 29 AsynCodeBench Table 12: Number of tasks with successful outcomes across models and execution protocols. Each entry reports the number of successful tasks out of 19, with the corresponding percentage in paren- theses. SingleSerialAsync-PrivateAsync- Manager-RO Async- Manager Qwen3.5-9B 2/19 (10.5%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%) Gemma-4-26B 0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)1/19 (5.3%) Nemotron-3.5 0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%) Qwen3.5-27B 0/19 (0.0%)1/19 (5.3%)0/19 (0.0%)3/19 (15.8%)2/19 (10.5%) GLM-4.7-Flash 0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%)0/19 (0.0%) Muse-Glimmer-30B 1/19 (5.3%)2/19 (10.5%)0/19 (0.0%)0/19 (0.0%)3/19 (15.8%) Qwen3-Coder-Next 3/19 (15.8%)0/19 (0.0%)0/19 (0.0%)1/19 (5.3%)2/19 (10.5%) Qwen3.6-27B 6/19 (31.6%)5/19 (26.3%)2/19 (10.5%)3/19 (15.8%)7/19 (36.8%) Qwen3.8-27B 14/19 (73.7%)8/19 (42.1%)7/19 (36.8%)7/19 (36.8%)12/19 (63.2%) DeepSeek-V4 15/19 (78.9%)9/19 (47.4%)7/19 (36.8%)11/19 (57.9%)13/19 (68.4%) 30