Paper deep dive
OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime
Chun-Wah Hsu, Kai Gong, Yu Wu, Xianhe Chen, Mengyang Liu, Jie Li, Hanyu Li, Zhixuan Liu, Naisheng Tang, Jiaying Chi, Ziheng Fan, Xuning He, Xiaokang Yang, Xue Jiang, Yihong Dong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 10/1/2026, 7:49:52 AM
Summary
The paper introduces OpenCollab, a multi-agent coding framework designed to address the 'illusion of collaboration' where agents fail to follow declared organizational structures. OpenCollab provides a unified infrastructure with programmable collaboration, controllable shared runtime, and fine-grained event streams to track execution. It introduces the 'Adherence' metric to quantify the alignment between declared and realized organizations. Experiments show that changing configuration dimensions significantly impacts adherence (47.2% to 97.2%). OpenCollab's two-coder workflow achieves state-of-the-art performance on SWE-bench Pro, Terminal-Bench 2.1, and DeepSWE, outperforming harnesses like Mini-SWE-agent, Codex CLI, and Claude Code, while its single-agent configuration uses the fewest tokens.
Entities (12)
Relation Signals (12)
OpenCollab → developedby → Chun-Wah Hsu
confidence 99% · OpenCollab: A Multi-Agent Coding Framework ... Chun-Wah Hsu 1,*
OpenCollab → developedby → Kai Gong
confidence 99% · OpenCollab: A Multi-Agent Coding Framework ... Kai Gong 1,*
Kai Gong → affiliatedwith → Shanghai Jiao Tong University
confidence 98% · Kai Gong 1 ... 1 Shanghai Jiao Tong University
Chun-Wah Hsu → affiliatedwith → Shanghai Jiao Tong University
confidence 98% · Chun-Wah Hsu 1 ... 1 Shanghai Jiao Tong University
OpenCollab → usesmetric → Adherence
confidence 98% · we define Adherence to quantify whether the declared organization is actually realized.
OpenCollab → achievesstateofthearton → SWE-bench Pro
confidence 95% · OpenCollab establishes new SOTA performance on SWE-bench Pro (64.25%)
OpenCollab → achievesstateofthearton → Terminal-Bench 2.1
confidence 95% · Terminal-Bench 2.1 (83.15%)
OpenCollab → achievesstateofthearton → DeepSWE
confidence 95% · and DeepSWE (69.91%)
OpenCollab → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent coding systems are designed to tackle complex software engineering tasks through collaboration. However, existing evaluations typically assume configured organizations are followed faithfully, whereas reality differs. This behavioral gap, combined with differences in underlying system components, prevents clear attribution of observed gains. To this end, we introduce OpenCollab, a multi-agent coding framework that provides a unified infrastructure for programmable collaboration and controllable runtime. Specifically, OpenCollab unifies organization design, enforces experimental control on a shared runtime, and tracks execution through fine-grained event streams. On this basis, we define Adherence to quantify whether the declared organization is actually realized. Our experiments reveal that agents collaborate very differently across configurations: changing any single dimension shifts Adherence, from 47.2% to as high as 97.2%. Furthermore, extensive agentic coding benchmarks show that a two-coder workflow built on OpenCollab establishes new SOTA performance compared to the mainstream harnesses such as Mini-SWE-agent, Codex CLI, and Claude Code, showing that a well-designed organization can outperform strong existing harnesses, while OpenCollab's single-agent configuration uses the fewest tokens across all evaluated suites. OpenCollab establishes a unified multi-agent infrastructure for easy programmable collaboration and controlled causal evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2609.38345v1
- Canonical: https://arxiv.org/abs/2609.38345v1
Trouble viewing inline? Open PDF directly →
Full Text
120,186 characters extracted from source content.
Expand or collapse full text
September 29, 2026 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Chun-Wah Hsu 1,* Kai Gong 1,* Yu Wu 1,* Xianhe Chen 2 Mengyang Liu 1 Jie Li 3 Hanyu Li 1 Zhixuan Liu 1 Naisheng Tang 4 Jiaying Chi 1 Ziheng Fan 5 Xuning He 1 Xiaokang Yang 1 Xue Jiang 6,7 Yihong Dong 1,B 1 Shanghai Jiao Tong University 2 University of Cambridge 3 Nanyang Technological University 4 The University of Hong Kong 5 Imperial College London 6 Peking University 7 Tencent ABSTRACT Multi-agent coding systems are designed to tackle complex software engineering tasks through collaboration. However, existing evaluations typically assume configured organi- zations are followed faithfully, whereas reality differs. This behavioral gap, combined with differences in underlying system components, prevents clear attribution of observed gains. To this end, we introduce OpenCollab, a multi-agent coding framework that provides a unified infrastructure for programmable collaboration and controllable runtime. Specifically, OpenCollab unifies organization design, enforces experimental control on a shared runtime, and tracks execution through fine-grained event streams. On this basis, we define Adherence to quantify whether the declared organization is actually realized. Our experiments reveal that agents collaborate very differently across configurations: changing any single dimension shifts Adherence, from 47.2% to as high as 97.2%. Furthermore, extensive agentic coding benchmarks show that a two-coder workflow built on OpenCollab establishes new SOTA performance compared to the mainstream harnesses such as Mini-SWE-agent, Codex CLI, and Claude Code, showing that a well-designed organization can outperform strong existing harnesses, while OpenCollab’s single-agent configuration uses the fewest tokens across all evaluated suites. OpenCollab establishes a unified multi-agent infrastructure for easy programmable collaboration and controlled causal evaluation. Project page / Code Correspondence: dongyh@sjtu.edu.cn 1 Introduction Autonomous coding agents have achieved remarkable progress, demonstrating promising capabilities to resolve complex issues in real-world software repositories (Jimenez et al., 2024; Wang et al., 2025). As real-world programming tasks grow in complexity, managing vast contexts and multi-stage execution within a single model becomes increasingly challenging. To push the boundaries of performance, the paradigm is shifting toward multi-agent systems (Li et al., 2023; Hong et al., 2024; Qian et al., 2024; Wu et al., 2024; Dong et al., 2024). By assigning specialized organization, such as analysts, developers, and testers, these systems tackle intricate tasks through division of labor and interactive coordination. This design assumes that structured organization improves performance, yet configured collaboration may not actually take place. Despite their potential, current multi-agent frameworks lack the infrastructure to evaluate actual * Equal contribution, work done as research interns at RISE-X Lab, School of Computer Science, Shanghai Jiao Tong University. B: Corresponding author. 1 arXiv:2609.38345v1 [cs.SE] 29 Sep 2026 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime (a) Organization(b) Benefit / cost(c) Execution(d) Validity Conversation AutoGen / CAMEL Role workflow self-collaboration Graph control statetool LangGraph Shared tasks model / tools specified budget policy SingleTeam vs. ? T BenefitCost Worth the extra cost? Configured Analyst CoderTester Observed events time Weak Collaboration PolicyTrace Collaboration audit Handoffs Participation Role boundary Evidence, not labels Figure 1. Perspectives on multi-agent collaboration: (a) organization, (b) benefit and cost, (c) configured versus observed execution, and (d) collaboration auditing. effectiveness. First, existing systems lack a unified mechanism to explicitly declare an organizational topology. Second, multi-agent evaluations are rarely controlled, as harness differences and run-to-run variance can obscure what produced a performance gap (Zhang et al., 2026; Kaliyev and Maryanskyy, 2026). Third, intermediate collaborative processes suffer from poor observability, since benchmarks re- port only final outcomes (Kirgis et al., 2026) and logs are written for human debugging, not attribution. Finally, this opacity obscures a fundamental mismatch between declared and realized organizations, where agents frequently work in isolation without actual delegation (Figure 1). Consequently, standard benchmarks cannot determine whether an organization actually helps, because they wouldn’t verify whether collaboration took place. In this paper, we introduce OpenCollab, a multi-agent infrastructure for programmable collaboration and controllable runtime. To enable explicit declaration, OpenCollab provides unified abstractions to statically specify role boundaries and communication topologies. To controlled comparisons, its runtime executes all configurations on a shared substrate, holding non-organizational factors constant. To structured logs, it captures step-by-step actions and resource usage through fine-grained event streams. Building on it, OpenCollab introduces Adherence to quantify the realized organization and, by recording it per run, checks when the complier average causal effect (CACE) (Bloom, 1984; Angrist et al., 1996) can be attributed to the organization. To systematically evaluate multi-agent systems, we leverage OpenCollab’s event streams to dissect their execution across five dimensions (model, tool set, budget, context, and topology). We find that agents collaborate very differently across configurations: changing any single dimension shifts Adherence, from 47.2% to as high as 97.2%; at 47.2%, lead agents monopolize budgets or bypass teammates. Furthermore, our audit of ten mainstream agent frameworks reveals that none natively guarantees controlled experimental comparison or verifies behavioral delivery. Against Claude Code, Codex CLI, and Mini-SWE-agent, OpenCollab establishes new SOTA performance on SWE-bench Pro (64.25%), Terminal-Bench 2.1 (83.15%), and DeepSWE (69.91%). Finally, OpenCollab demonstrates that effective multi-agent collaboration provides substantial gains over working alone. Its collaborative design OpenCollab (Duo) outperforms its single-agent counterpart OpenCollab (Base) by +3.4 points on Terminal-Bench 2.1 and +14.2 points on DeepSWE. We summarize our main contributions as follows: •We demonstrate that agents collaborate very differently across configurations: changing a single dimension moves the collaboration rate from 47.2% to as high as 97.2%. • We formalize Adherence metric to quantify the alignment between a declared organization and its actual runtime execution, grounded in causal identification theory. Furthermore, our rigorous audit of ten mainstream agent frameworks reveals that none can compute Adherence for every organization it runs or ensure controlled comparisons. • We present OpenCollab, an infrastructure for programmable collaboration and controllable runtime, supporting teams and workflows. It isolates experimental variables and accurately reconstructs 2 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 1. Audit of seven controlled evaluation conditions.✓: by the artifact’s own settings and records; –: only through the researcher’s code, or in part;×: not met. Held: can the factor be set explicitly?Verified: can the run be checked? ArtifactModelToolsBudgetContextTopologyRealizedCompared Multi Agent frameworks AutoGen (Wu et al., 2024)✓–✓× AG2 (ag2ai, 2024)– ×–× LangGraph (LangChain, 2023)– ×–× Coding agents SWE-agent (Yang et al., 2024)✓–✓×✓× OpenHands (Wang et al., 2025)✓–× Claude Code (Anthropic, 2026)✓– ×–× Codex CLI (OpenAI, 2025)✓– × ×–× DeepSeek Harness (DeepSeek, 2026)✓ ×–× Evaluation frameworks Inspect AI (AI Security Institute, UK, 2024)✓–✓– HAL (Kapoor et al., 2026)– × × ×–× OpenCollab (Ours)✓ execution trajectories via fine-grained event streams. •On extensive mainstream coding benchmarks, our sample multi-agent OpenCollab (Duo) sets new SOTA performance over other mainstream harnesses, and surpasses the single-agent OpenCollab (Base) by +3.4 points on Terminal-Bench and +14.2 on DeepSWE. 2 Motivation Deploying multi-agent teams on repository-level tasks typically begins by configuring specialized roles, such as an Analyst, Coder, and Tester (Dong et al., 2024). While intended to divide labor, trajectory analysis reveals that delegation rarely occurs when models decide autonomously. Mostly, the lead agent bypasses configured teammates to resolve issues in isolation, while in other cases, it exhausts its token allowance before issuing any handoff. Consequently, the rate at which the declared organization is realized (formalized as Adherence) frequently falls below half as the system collapses into a single-agent loop. We term this divergence between configured and executed organizations the illusion of collaboration. Attempting to resolve this illusion via prompt engineering or log filtering introduces deeper dilemmas. Incrementally mandating delegation in prompts causes Adherence to swing wildly, demonstrating that collaboration is highly sensitive to phrasing rather than organizational structure alone. Conversely, filtering logs to collaborating runs introduces severe post-treatment selection bias. Agents solve easy tasks alone and seek help only when stuck, so this subset skews toward harder tasks. That penalizes the multi-agent arm and invalidates causal claims. The persistence of this illusion and the failure of simple workarounds stem from underlying architec- tural limitations in current agent frameworks. First, decentralized resource management allows agents to exhaust budgets locally within isolated execution loops. Because token limits are not governed by a shared pre-call gate across the entire team, lead agents frequently consume the run allowance before reaching a state where delegation could occur. Second, current platforms record execution through conversational prose logs designed for human debugging rather than programmatic analysis. These text transcripts lack role-level attribution, making it difficult to verify which agent executed a specific tool or consumed a given portion of the budget. This opacity prevents automated tools from confirming whether declared topology edges were walked, concealing unexecuted organizations beneath overall benchmark scores. Addressing these limitations requires three core system capabilities. To prevent unobserved budget exhaustion and eliminate confounders, the framework must operate on a shared runtime enforcing pre-call admission across all arms. To overcome opaque prose logs, execution must be serialized into a structured event stream binding every action and token cost to a distinct role. Finally, to resolve 3 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime SpecificationEvaluation taskmodel role prompts toolsets topology tokens / context tool policies Control Single one session Teammodel-directed AnalystCoderTester Workflowscripted AnalyzeCodeTest shared session interface Shared session runtime Analyst prompt / tools context / budget Coder prompt / tools context / budget Tester prompt / tools context / budget Workspaces isolated worktrees CoderTester patch / commit graded entry Session loop PrecheckModelResponseToolsAutosave Done next step Assigned (Z) roles / policies Event stream run / step / seat actions / usage Realized (A) Audit: Z + trace A participation / delegation info flow / role bounds Result (Y) Cost (C) Metrics: Z, A, Y, C CACE / Adherence overhead / balance Figure 2. OpenCollab overview. Single, Team, and Workflow controllers share a session runtime. Configuration and execution traces support Adherence and resource costs auditing. selection bias, the system must compute Adherence from these streams and use CACE to support adherence-aware causal attribution of organizational effects. Together, these capabilities define an infrastructure that unifies programmable collaboration with controlled evaluation, providing the foundation for OpenCollab. 3 Methodology We next describe OpenCollab in detail. In particular, we discuss 1) the architecture of OpenCollab (§3.1), 2) how to program custom collaborative organizations (§3.2), 3) how to ensure fair and controlled comparisons (§3.3), 4) how to trace executions using structured event streams (§3.4), and 5) how to measure collaboration effectiveness with built-in metrics (§3.5). Figure 2 provides an overview of OpenCollab. 3.1 Architecture Overview OpenCollab introduces a decoupled, two-layer architecture, departing from traditional frameworks where each agent encapsulates its own isolated execution loop. As illustrated in Figure 13 (Ap- pendix C.1), this design separates the high-level orchestration of agent interactions from the low-level mechanics of model execution. At the foundational layer, OpenCollab operates on a shared session runtime. The fundamental unit of execution is a session, representing a single role in an organization. Every session runs through an identical ten-state finite state machine loop detailed in Appendix C.1. This substrate handles context management, model invocations, and tool execution, ensuring unit consistency across runs. Built atop the shared runtime, the orchestration layer operates through independent controllers. These controllers govern session topology, lifecycle, and message routing, abstracting control flow away from individual agent implementations. Because all controllers converge on this shared substrate, OpenCollab isolates structural design from low-level runtime mechanics. This decoupling supports diverse organizational paradigms without altering the execution engine, removes infrastructure noise, and facilitates both programmable collaboration and controlled evaluation. 3.2 Programmable Collaboration To structure multi-agent execution, the orchestration layer supports two primary collaboration paradigms alongside single-agent baselines, summarized as experimental arms in Table 11 (Ap- 4 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime pendix C.1): a model-directed team mode and a programmatic workflow mode. The team controller enables a model-directed collaboration paradigm. Users provide a declarative configuration specifying role prompts, accessible toolsets for each role, and a directed communication topology indicating which roles can address one another, following the declaration schema in Ap- pendix C.3. Once configured, the language model operates within these boundaries, deciding whether, when, and how to delegate subtasks to available teammates. This mode mimics workplace dynamics, allowing multiple agent sessions to cooperate concurrently and pass messages along declared topology edges. The workflow controller enables programmatic and deterministic agent orchestration. OpenCollab exposes an interface that allows researchers to explicitly govern the collaboration graph. Through simple Python scripts, users can define call orders, fan-out structures, and conditional state transitions across multiple sessions. The script directly controls message routing and termination, ensuring structured execution rather than relying on volatile model decisions. By separating orchestration from execution, these controllers drastically reduce implementation overhead: reproducing the core multi-agent pattern of edict (∼24k lines of code) requires merely 239 lines in OpenCollab. While this enables effortless organizational prototyping, evaluating true collaborative effectiveness still requires strictly controlled conditions. 3.3 Controlled Evaluation and Comparison OpenCollab embeds control primitives into the shared runtime to enforce identical conditions across all experimental arms. Specifically, Table 1 defines seven conditions for controlled comparison, com- prising five factors held constant at runtime and two structural verification requirements. Table 7 (Appendix B.1.1) details how the five controlled factors guarantee that non-organizational variables remain aligned across arms, including the base model, per-role tool availability, token and cost budgets, context compaction policies, and the communication topology. The two verification requirements ensure auditability by recording each run’s declaration next to its realized trajectory and evaluating one against the other. As our audit of ten artifacts in Appendix C.4 reveals, none of them satisfies all seven conditions, whereas OpenCollab meets all seven. OpenCollab enforces these conditions through dedicated runtime gates rather than post-hoc filtering. Tool calls and inter-agent messages are intercepted at invocation time, where any request outside the declared role permissions or communication topology is refused and recorded, following the declaration rules in Appendix C.3. For resource control, Figure 13 shows how the runtime enforces an identical per-role token ceiling via pre-call admission within the session state machine. Before each model call, the runtime validates an estimated token reservation against the remaining allowance, halting the session with an explicit termination record if exceeded. Appendix C.5 provides the exact termination codes, which prevent unmetered overruns and cleanly separate runs that exhausted their budget from those that finished voluntarily. These explicit termination records and invocation-time gates form the precise foundation of OpenCollab’s execution traceability. 3.4 Execution Traceability and Observability To capture these records, OpenCollab replaces prose logs with an ordered, append-only event stream. Throughout an execution, every operational step emits a structured record containing a monotonic step index, a timestamp, a run identifier, a role identifier, an event type, and a typed payload. As summarized in Table 13 (Appendix C.5), the runtime defines seven distinct event types to capture the full lifecycle of each role, covering model calls, tool executions, refused actions, session terminations, spawn attempts, worktree modifications, and context compaction triggers. Because every event is explicitly stamped with its role identifier, the event stream allows automated scripts to reconstruct the realized execution topology and attribute resource costs without parsing natural language text. To ensure physical delivery attribution on the filesystem, non-entry roles in a team operate in isolated Git worktrees over a shared object store (Appendix C.1). Only the entry role workspace is graded upon run completion. Appendix C.5 details this event schema. 5 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime 3.5 Evaluation of Cooperation Effectiveness Leveraging the structured event stream, OpenCollab extracts runtime variables to compute automated metrics, addressing the evaluation biases outlined in Section 2. Each runris represented by the assigned configurationZ r ∈Single,Workflow,Team, the realized-organization indicatorA r ∈0, 1, the task resultY r ∈ 0, 1, and the token costC s (r)for each roles ∈ S, whereC(r) = ∑ s∈S C s (r) is the total run cost. For Team runs, A r = adh(r), the full structural-adherence indicator defined below. Structural Adherence. Adherence establishes internal validity by verifying that an assigned organiza- tional topology was delivered at runtime. Table 15 (Appendix C.6) defines six verification axes: partici- pation, delegation, role boundary, budget sharing, information flow, and context policy. For each appli- cable axisx, the runtime produces an objective audit verdictv x (r)∈adherent,deviant,unverifiable. A run is judged adherent if and only if every applicable axis passes:adh(r) = ∏ x∈X (Z r ) 1[v x (r) = adherent], whereX (Z r )denotes the set of applicable verification axes for assignmentZ r . The Adher- ence rate observed across all runsR a within an arm, the sample analogue ofα adh = Pr(A(z) =1) (Appendix A.1), is: b α adh = 1 |R a | ∑ r∈R a adh(r).(1) A deviant or unverifiable verdict on any applicable axis givesadh(r) =0, so delegation alone does not make a run adherent: a run in which only one of two required teammates acts hasadh(r) =0. BecauseA r = adh(r)on Team runs, b α adh is the empirical rate at which the assigned Team organization is realized. Configuration-level complier effect. To resolve the dilution of observed task success gains, OpenCollab divides the Pass@1 difference from Single (the intention-to-treat effect, ITT) by the Adherence rate, giving the complier average causal effect (CACE): \ CACE := d ITT / b α adh =∆ \ Pass@1 / b α adh .(2) Under mean exclusion,δ 0 =0 (Eq.(8)), the population ratioITT/α adh identifies the configuration-level complier effect: the effect of assigning the target Team configuration rather than Single among runs in which the assigned Team organization is fully realized, not an isolated effect of coordination itself. This causal interpretation relies on two conditions. One-sided compliance holds because the Single arm cannot realize the target Team organization (Appendix A.1). The mean exclusion restriction requires that, among tasks whose Team execution hasA r =0, assigning the Team configuration rather than Single leaves the mean outcome unchanged. Without it, ITT/α adh differs from the complier effect by(1− α adh )/α adh times thatA r =0 contrast (Theorem A.1, Proposition A.2). While standard evaluations can only assume this restriction, recordingA r per run on paired tasks allows theA r =0 Team-versus-Single contrast to be estimated, which provides evidence about mean exclusion rather than proof of it (Appendix A.3). Cost metrics. The same records support two cost metrics, the organizational overhead of runs that do not realize the assigned organization and the workload balance of runs that do; Appendix C.5 defines both, and our experiments do not report them. 4 Experiments Our evaluations focus on two primary objectives: measuring system-level effectiveness and diagnosing internal collaborative mechanics. We evaluate OpenCollab on SWE-bench Pro (Deng et al., 2026), Terminal-Bench 2.1 (Merrill et al., 2026; Terminal-Bench, 2026), and DeepSWE (Huang et al., 2026). For cross-harness evaluations, OpenCollab is benchmarked against Claude Code (Anthropic, 2026), Codex CLI (OpenAI, 2025), and Mini-SWE-agent (Yang et al., 2024). We compare three models: DeepSeek- V4.1-Flash, Qwen3.8-Flash, and GPT-5.6-Luna. Extensive experiments and detailed setup can be found in Appendices B.1–B.5. 4.1 System-Level Performance and Organizational Value We first evaluate system-level performance to establish OpenCollab’s competitiveness. Table 2 runs five harnesses with GPT-5.6-Luna on SWE-bench Pro, Terminal-Bench 2.1, and DeepSWE. OpenCollab 6 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 2. Cross-harness evaluation on SWE-bench Pro, Terminal-Bench 2.1, and DeepSWE. BenchmarkHarnessPass@1 (%)↑Avg. tokens (M)↓Avg. cost ($)↓Cache hit (%)↑ SWE-bench Pro Mini-SWE-Agent61.664.160.9036.65 Codex CLI63.737.310.7390.27 Claude Code58.039.382.5412.50 OpenCollab (Base)63.213.890.4188.03 OpenCollab (Duo)64.256.500.7384.95 Terminal-Bench 2.1 Mini-SWE-Agent76.402.970.4567.36 Codex CLI80.902.860.2693.88 Claude Code77.5321.952.7379.32 OpenCollab (Base)79.781.540.1978.38 OpenCollab (Duo)83.154.600.6076.47 DeepSWE Mini-SWE-Agent61.9519.363.2758.81 Codex CLI47.7915.011.3096.47 Claude Code56.6477.907.6691.16 OpenCollab (Base)55.7511.951.1990.53 OpenCollab (Duo)69.9126.112.6789.36 (Duo) attains the highest Pass@1 on all three (64.25%, 83.15%, and 69.91%). OpenCollab (Base) uses the fewest tokens and the lowest cost on all three benchmarks, and OpenCollab (Duo) costs less than Claude Code on each (Appendix B.2). OpenCollab (Duo) is a Workflow in which code issues every handoff, so both of its Coders run on every task. Paired by task against OpenCollab (Base), it solves 21 DeepSWE tasks that the single agent fails and fails 5 that the single agent solves (69.91% vs. 55.75%); on Terminal-Bench 2.1 the split is 6 to 3 (83.15% vs. 79.78%), too few to separate the two. The workflow spends more tokens, since it runs two complete solving processes and selects between their candidates (Appendix B.1.3). Table 3. Single-dimension ablations relative to the reference team. Unverified runs are those terminated before collaboration; brackets report 95% Clopper–Pearson confidence intervals. Dimension VariantAdherence (%)↑95% CIUnverified (%) Pass@1 (%)↑ Tokens (M)↓ Model Qwen3.8-Flash47.2[30.4, 64.5]13.975.03.05 DeepSeek-V4.1-Flash66.7[49.0, 81.4]22.272.23.65 GPT-5.6-Luna97.2[85.5, 99.9]2.861.15.79 Tools All tools47.2[30.4, 64.5]13.975.03.05 No edit tools86.1[70.5, 95.3]0.066.73.24 Read-only94.4[81.3, 99.3]2.852.84.24 Budget 0.5M / role47.2[30.4, 64.5]2.844.40.88 2M / role75.0[57.8, 87.9]8.369.44.17 4M / role77.8[60.8, 89.9]2.875.05.20 Context Open card47.2[30.4, 64.5]13.975.03.05 Optional card63.9[46.2, 79.2]16.761.13.58 Mandatory card91.7[77.5, 98.2]8.372.24.04 Topology Star47.2[30.4, 64.5]13.975.03.05 Ring + shortcut66.7[49.0, 81.4]13.966.73.92 Ring91.7[77.5, 98.2]8.363.94.21 7 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime 4.2 Deconstructing Collaboration: Dimensions of Adherence Standard multi-agent evaluations assume that configured roles operate exactly as declared. We test this by deploying a model-directed team and auditing its execution through OpenCollab’s event streams across five configuration dimensions: model, tool set, budget, context, and topology. To isolate the impact of each dimension, we define a baseline reference team using Qwen3.8-Flash, open prompt cards, unrestricted tools, and a budget of 2M tokens per role. For ring topology variants, we instead deploy an analyst-coder-tester structure under the Mandatory card setting (Appendix B.1.1). Evaluating single-dimension perturbations against this reference team, Table 3 shows that unconstrained defaults yield only 47.2% Adherence. However, enforcing constraints like restricted tool boundaries prevents isolated execution, surging Adherence above 90% and demonstrating that actual collaboration is systematically controllable. Table 4. Deployment contrasts and adherence-adjusted estimates across configurations on SWE-bench Pro. ITT is the Pass@1 difference from Single. Adherence is the share of runs withA r =1. CACE is ITT divided by Adherence. Brackets show sensitivity to the labeling rule for unverifiable verdicts and are not confidence intervals. ConfigurationAdherence (%)↑Pass@1 (%)↑ITT (%)↑CACE (%)↑ Single–69.4– Open card (reference)47.2 [47.2, 61.1]75.0+5.6+11.8 [+9.1, +11.8] Mandatory card91.7 [91.7, 100.0]72.2+2.8+3.0 [+2.8, +3.0] Budget stated, 2M75.0 [75.0, 83.3]69.40.00.0 [0.0, 0.0] Read-only94.4 [94.4, 97.2]52.8−16.7 −17.6 [−17.6,−17.1] 4.3 Causal Validity and Trace Diagnostics To compute deployment contrasts, each configuration runs the tasks once and is paired task by task with the Single agent (Table 4). Adherence sets how far CACE (calculated as ITT divided by Adherence, Eq.(2)) rescales the observed ITT. By default, a run with an unverifiable verdict is assignedA r =0 (Appendix A.1); brackets in our results show the sensitivity of counting these as adherent instead. Under the unconstrained Open card, 47.2% Adherence rescales an ITT of+5.6 to+11.8; this gap follows from low Adherence alone and does not by itself indicate a violation of mean exclusion. Reading+11.8 as the configuration-level complier effect (Appendix A) further requires theA r =0 Team-versus-Single contrast to have mean zero (Theorem A.1). The ratio departs from the complier effect by(1− α adh )/α adh times this contrast (Proposition A.2), a factor that grows as Adherence falls and is small above 90%. Adherence levels alone cannot verify mean exclusion; only theA r =0 contrast bears on it. Because OpenCollab records A r for every run on tasks also run by Single, this contrast is computed rather than assumed (Appendix A.6). Under the Mandatory card, the paired estimate over adherent runs coincides with CACE in this sample (+3.0); under the Open card, the+5.6 ITT arises on the 19 tasks withA r =0 (+15.8), whereas the 17 adherent tasks show−5.9, so+11.8 should not be read as the complier effect. OpenCollab’s event streams show whether the assigned organization is realized at runtime (Figure 3). In the reference team (Figure 3b), the adopter briefs both coders but monopolizes execution until exhausting its token budget, never submitting teammate work. Restricting the adopter to Read-only tools (Figure 3a) removes its ability to edit files, compelling delegation to Coders and the successful adoption of Coder A’s patch. Further trace analyses are in Appendix B.4. 5 Related Work Autonomous coding agents increasingly adopt multi-agent architectures, drawing on classical orga- nization theory (Fox, 1981; Galbraith, 1974; Malone and Crowston, 1994; Decker and Lesser, 1993, 1995; HORLING and LESSER, 2004). Modern frameworks operationalize these concepts through multi-agent interaction (Li et al., 2023; Hong et al., 2024; Qian et al., 2024; Chen et al., 2024b; Wu et al., 2024), stateful graphs (LangChain, 2023), or shared-state management (Liu et al., 2026a). These 8 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime 02040 60 adopts A's commit Coder A's code, passes Adopter Coder A Coder B Graded code (a) Read-only adopter: resolved 02040 60 briefs both, keeps working the adopter never ends its turn; coders run once its budget is out Adopter's own code, fails min (b) Reference team: unresolved readedittest / buildmessagecommit / adoptout of budgetpassesfails Figure 3. Two execution trajectories from Table 3 on a shared time axis. Lanes denote agents; blocks represent model calls colored by tool usage. frameworks lower the engineering barrier to deploying complex collaborative topologies. However, existing platforms entangle organizational design with low-level execution, forcing agents to manage clients, contexts, and tools independently. OpenCollab breaks this coupling by isolating orchestration from the runtime, providing a unified substrate to evaluate diverse structures under identical system mechanics. Yet, whether multi-agent collaboration outperforms single-agent baselines remains contested. Early work highlighted the benefits of multi-agent debate (Du et al., 2024), voting ensembles (Li et al., 2024), and repeated sampling (Brown et al., 2024). Conversely, recent studies report mixed or negative returns from adding agents or calls (Smit et al., 2024; Wang et al., 2024; Chen et al., 2024a; Zhao et al., 2026; Bertalaniˇc and Fortuna, 2026; Cemri et al., 2025), even under equal token budgets (Tran and Kiela, 2026; Kim et al., 2026a). Interacting agent teams often incur substantial coordination overhead (Khatua et al., 2026), while reported gains often fall below the replication noise floor (Kaliyev and Maryanskyy, 2026). These conflicting findings are hard to reconcile because comparisons rarely hold budgets, toolsets, or harnesses constant (Zhang et al., 2026). By enforcing pre-call budget gates and uniform execution primitives, OpenCollab ensures observed performance gaps reflect organizational design rather than infrastructural confounders. Beyond controlling runtime factors, evaluating collaboration requires confirming that the assigned organization actually materialized during execution. Analyzing interaction networks, Destefanis and Aste (2026) show that designated coordinator roles frequently fail to form communication hubs in practice. Related studies examine plan compliance (Liu et al., 2026b), reversible execution traces (Yu et al., 2026), and role separation (Kim et al., 2026b), while Kirgis et al. (2026) emphasize that agent evaluations require log analysis. Yet existing frameworks treat execution as an unverified black box, reporting outcomes without verifying actual communication. OpenCollab bridges this gap via structured event streams and formal Adherence, providing the infrastructure to audit delivery and support causal attribution of organizational gains under explicit identification assumptions. 6 Conclusion Evaluating multi-agent systems is challenging because existing frameworks entangle organizational design with execution mechanics. To address this, we present OpenCollab, a unified multi-agent coding platform that cleanly decouples design from execution. This decoupling allows researchers to easily construct diverse multi-agent systems in a unified manner on a shared substrate with identical runtime semantics. Furthermore, OpenCollab tracks runtime Adherence to verify that the configured collaboration actually occurs. Our findings also reveal that the agents collaborate very differently across configurations, so evaluations that do not verify collaboration cannot attribute their gains to it. Moreover, OpenCollab provides both the unified programmability to effortlessly build agent organizations and the controlled runtime required to attribute their effects causally under the explicit identification. 9 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Acknowledgments We sincerely thank Fanjian Su, Lirui Guan, Peize Li, and Weiming Wang for their contributions to the early stages of this work. References ag2ai. AG2 (formerly AutoGen): The open-source AgentOS. GitHub repository, 2024. URL https://github.com/ ag2ai/ag2. AI Security Institute, UK. Inspect AI: Framework for large language model evaluations, 2024. URL https: //zenodo.org/doi/10.5281/zenodo.18434278. Joshua D. Angrist, Guido W. Imbens, and Donald B. Rubin. Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434):444–455, 1996. doi: 10.1080/01621459.1996. 10476902. Anthropic. Overview - Claude Code docs, 2026. URL https://code.claude.com/docs/en/overview. Blaž Bertalaniˇc and Carolina Fortuna. The ringelmann effect in multi-agent LLM systems: A scaling law for effective team size. arXiv preprint arXiv:2606.02646, 2026. URL https://arxiv.org/abs/2606.02646. Howard S. Bloom. Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8(2):225–246, 1984. doi: 10.1177/0193841x8400800205. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. URL https://arxiv.org/abs/2407.21787. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei A Zaharia, Joseph Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems, volume 38, Main Conference. Curran Associates, Inc., 2025. doi: 10.52202/085713-4082. URL https://proceedings.neurips.c/paper_files/ paper/2025/file/b1041e52d3be19f0a9bc491657488e4a-Paper-Datasets_and_Benchmarks_Track.pdf. Lingjiao Chen, Jared Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou.Are more LLM calls all you need? towards the scaling properties of compound AI systems.In Ad- vances in Neural Information Processing Systems, volume 37, pages 45767–45790. Curran Associates, Inc., 2024a.doi: 10.52202/079017-1455.URL https://proceedings.neurips.c/paper_files/paper/2024/file/ 51173cf34c5faac9796a47dc2fdd3a71-Paper-Conference.pdf. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors. In International Conference on Learning Representations, volume 2024, pages 20094–20136, 2024b. URL https://proceedings.iclr.c/paper_files/paper/ 2024/file/578e65cdee35d00c708d4c64bce32971-Paper-Conference.pdf. Keith S. Decker and Victor R. Lesser. Quantitative modeling of complex environments. Intelligent Systems in Accounting, Finance and Management, 2(4):215–234, 1993. doi: 10.1002/j.1099-1174.1993.tb00044.x. Keith S. Decker and Victor R. Lesser. Designing a family of coordination algorithms. In Proceedings of the First International Conference on Multiagent Systems (ICMAS-95), pages 73–80, San Francisco, CA, USA, 1995. AAAI Press. URL https://cdn.aaai.org/ICMAS/1995/ICMAS95-010.pdf. DeepSeek. DeepSeek Harness: Everything is a plugin. GitHub repository, 2026. URL https://github.com/ deepseek-ai/deepseek-harness. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-bench pro: Can AI agents solve long-horizon software engineering tasks? In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. URL https://icml.c/virtual/2026/poster/61047. Giuseppe Destefanis and Tomaso Aste. When agents coordinate: Measuring coordination in multi-agent AI coding. arXiv preprint arXiv:2608.16801, 2026. URL https://arxiv.org/abs/2608.16801. Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via ChatGPT. ACM Transactions on Software Engineering and Methodology, 33(7):1–38, 2024. doi: 10.1145/3672459. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on 10 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11733–11763. PMLR, 2024. URL https://proceedings.mlr.press/v235/du24e.html. Mark S. Fox. An organizational view of distributed systems. IEEE Transactions on Systems, Man, and Cybernetics, 11 (1):70–80, 1981. doi: 10.1109/tsmc.1981.4308580. Jay R. Galbraith. Organization design: An information processing view. Interfaces, 4(3):28–36, 1974. doi: 10.1287/inte.4.3.28. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, volume 2024, pages 23247–23275, 2024. URL https://proceedings.iclr.c/paper_files/paper/ 2024/file/6507b115562b0a305f1958c87355a-Paper-Conference.pdf. Bryan HORLING and Victor LESSER. A survey of multi-agent organizational paradigms. The Knowledge Engineering Review, 19(4):281–316, 2004. doi: 10.1017/s0269888905000317. Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026. URL https://arxiv.org/abs/2607.07946. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024. URL https://proceedings.iclr.c/paper_files/paper/ 2024/file/edac78c3e300629acfe6cbe9ca88fb84-Paper-Conference.pdf. Alibek T Kaliyev and Artem Maryanskyy. How much coordination gain is real? a paired noise-floor protocol for multi-agent LLM benchmarks. arXiv preprint arXiv:2606.20695, 2026. URL https://arxiv.org/abs/2606.20695. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, and Arvind Narayanan. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. In International Conference on Learning Representations, volume 2026, pages 98778–98849, 2026. URL https://proceedings.iclr.c/paper_files/ paper/2026/file/a0928f924a344aaebbb7f6cd8d56e34c-Paper-Conference.pdf. Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K. Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J. Ryan, Jiaxin Pei, and Diyi Yang. CooperBench: Why coding agents cannot be your teammates yet. arXiv preprint arXiv:2601.13295, 2026. URL https://arxiv.org/abs/2601.13295. Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, Paul Pu Liang, Hae Won Park, Yuzhe Yang, Xuhai Xu, Yilun Du, Shwetak Patel, Tim Althoff, Daniel McDuff, and Xin Liu. Capable language models can outgrow the benefits of collaboration. Nature Machine Intelligence, 8(7):1157–1172, 2026a. doi: 10.1038/s42256-026-01268-y. Yubin Kim, Chanwoo Park, Taehan Kim, Eugene Park, Samuel Schmidgall, Salman Rahman, Chunjong Park, Cynthia Breazeal, Xin Liu, Hamid Palangi, Hae Won Park, and Daniel McDuff. TeamBench: Evaluating agent coordination under enforced role separation. arXiv preprint arXiv:2605.07073, 2026b. URL https://arxiv.org/ abs/2605.07073. Peter Kirgis, Sayash Kapoor, Stephan Rabanser, Nitya Nadgir, Cozmin Ududec, Magda Dubois, J Allaire, Conrad Stosz, Marius Hobbhahn, Jacob Steinhardt, and Arvind Narayanan. Log analysis is necessary for credible evaluation of AI agents. arXiv preprint arXiv:2605.08545, 2026. URL https://arxiv.org/abs/2605.08545. LangChain. LangGraph: Build resilient agents. GitHub repository, 2023. URL https://github.com/langchain-ai/ langgraph. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.CAMEL: Communicative agents for "mind" exploration of large language model society.In Advances in Neural Information Processing Systems, volume 36, pages 51991–52008. Curran Associates, Inc., 2023.doi: 10.52202/075280-2264.URL https://proceedings.neurips.c/paper_files/paper/2023/file/ a3621e907def47c1b952ade25c67698-Paper-Conference.pdf. Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=bgzUSZ8aeg. Mengyang Liu, Taozhi Chen, Zhenhua Xu, Xue Jiang, and Yihong Dong. Multi-agent collaboration with state management. arXiv preprint arXiv:2605.20563, 2026a. URL https://arxiv.org/abs/2605.20563. Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand. From plan to action: How well do agents follow the plan? arXiv preprint arXiv:2604.12147, 2026b. URL https://arxiv.org/abs/2604.12147. 11 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Thomas W. Malone and Kevin Crowston. The interdisciplinary study of coordination. ACM Computing Surveys, 26(1):87–119, 1994. doi: 10.1145/174666.174668. Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong (Ryan) Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Rytting, Ryan Marten, Yixin Wang, Jenia Jitsev, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, volume 2026, pages 40903–40986, 2026. OpenAI. Codex CLI: Lightweight coding agent that runs in your terminal. GitHub repository, 2025. URL https://github.com/openai/codex. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174–15186, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.810. URL https://aclanthology.org/2024.acl-long.810/. Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going MAD? A look at multi-agent debate strategies for LLMs. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 45883–45905. PMLR, 2024. URL https://proceedings.mlr.press/v235/smit24a.html. Terminal-Bench. Terminal-Bench 2.1, 2026. URL https://w.tbench.ai/news/terminal-bench-2-1. Dat Tran and Douwe Kiela. Single-agent LLMs outperform multi-agent systems on multi-hop reasoning under equal thinking token budgets. arXiv preprint arXiv:2604.02460, 2026. URL https://arxiv.org/abs/2604.02460. Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6106–6131, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.331. URL https://aclanthology.org/2024.acl-long.331/. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025. URL https://proceedings.iclr.c/ paper_files/paper/2025/file/a4b6ad6b48850c0c331d1259fc66a69c-Paper-Conference.pdf. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling (COLM 2024), 2024. URL https://colmweb.org/2024/AcceptedPapers.html. John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.SWE-agent: Agent-computer interfaces enable automated software engineering.In Ad- vances in Neural Information Processing Systems, volume 37, pages 50528–50652. Curran Associates, Inc., 2024.doi: 10.52202/079017-1601.URL https://proceedings.neurips.c/paper_files/paper/2024/file/ 5a7c947568c1b1328c5230172e1e7c-Paper-Conference.pdf. Simon Yu, Derek Chong, Ananjan Nandi, Dilara Soylu, Jiuding Sun, Christopher D Manning, and Weiyan Shi. Shepherd: Enabling programmable meta-agents via reversible agentic execution traces. arXiv preprint arXiv:2605.10913, 2026. URL https://arxiv.org/abs/2605.10913. Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K. Reddy. Stop comparing LLM agents without disclosing the harness. arXiv preprint arXiv:2605.23950, 2026. URL https://arxiv.org/abs/2605. 23950. Yuxuan Zhao, Sijia Chen, and Ningxin Su. When does multi-agent collaboration help? an entropy perspective. arXiv preprint arXiv:2602.04234, 2026. URL https://arxiv.org/abs/2602.04234. 12 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime A Identification theory for the CACE Terminology: Section 3.5 and Table 4 call the ratioITT /α adh CACE for brevity. The derivations below keep the two apart: the ratio is the Wald ratio (its sample value the Wald estimate), and CACE denotes the complier effectE[∆ | A =1]of Appendix A.1, which the main text calls the configuration-level complier effect. Section 3.5 reports the Wald ratioITT /α adh , which is read as a complier average causal effect (CACE) only under the mean exclusion restriction below. Hereα adh is the probability that a target Team run satisfies the full operational structural-adherence rule. Every applicable axis must be explicitly judged adherent. The Single arm cannot realize that target organization, giving one-sided compliance. The ratio identifies the effect of the assigned Team configuration relative to Single among fully adherent Team runs when the non-adherent Team-versus-Single contrast has mean zero. Compatible paired records can estimate that contrast; otherwise the exact discrepancy and a sensitivity bound state what the ratio may miss. A.1 Setup and notation Using the notation of Section 3.5, writeZ r ∈ Single,Workflow,Teamfor the assigned arm,A r ∈ 0, 1for the realized-organization indicator, and retain outcomeY r ∈ 0, 1, role costC s (r), and arm-specific structural adherenceadh(r). Fix a target Team configurationzand write 0 for the Single baseline. For a task and its pre-specified execution randomness, letY(z),Y(0)∈0, 1be the configuration-level potential outcomes under the same scoring rule. Their joint law uses a specified coupling of task and run randomness; their marginal means require only comparable arm distributions. We use the same task weights in both arms and assume consistency of observed runs with their assigned potential outcomes, isolated resets, stable model and runtime versions, and complete scoring under a fixed rule. In particular, failures and timeouts are not dropped according to their outcomes. For a task and its pre-specified execution randomness, letA(z)∈0, 1denote the result of applying the operational structural-adherence rule of Section 3.5 to the execution that would be generated under target Team configurationz. Withv x (z)denoting the operational axis verdict generated under that configuration, define A(z) := ∏ x∈X (z) 1v x (z) = adherent.(3) For an observed Team runrassigned configurationz, consistency gives the observed labelA r = A(z) = adh(r) for that run’s task and execution randomness. An adherent verdict contributes one; a deviant or unverifiable verdict contributes zero. ThusA r =1 exactly when every applicable axis is explicitly judged adherent, andA r =0 otherwise. Every retained Team run has this label, including runs with unverifiable axis verdicts. WriteA := A(z)below and define the population rateα adh := P A(z) =1. The finite-sample rate b α adh reported in Section 3.5 is the mean of the labelsA r over the evaluated Team runs, the sample analogue of this population rate; likewise d ITTis the difference of observed arm means, and b τ Wald = d ITT/ b α adh is the sample Wald estimate. Delegation is one component of the structural-adherence criterion. As illustrated in Appendix C.6, a run can delegate to one teammate while another declared participant never acts; such a run hasA r =0 despite delegation. The cost variables do not enter this outcome identification argument. DefineA(0) =0 only as a statement that a Single run cannot realize the target Team organization. It does not score Single’s adherence to its own applicable axes as zero; Single can be adherent to its own topology. The pair(A(0),A(z))is therefore either(0, 0)or(0, 1). This is the one-sided-compliance, and hence monotonicity, property relevant to the Wald argument (Bloom, 1984; Angrist et al., 1996). With∆ := Y(z)− Y(0), define ITT := E[∆],(4) CACE := E[∆| A = 1] (α adh > 0),(5) δ 0 := E[∆| A = 0] (α adh < 1).(6) We refer to this configuration-level complier effect as CACE: the effect of assigning the target Team configuration rather than Single among runs that satisfy the full operational structural-adherence 13 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime rule. It is not, without further intervention assumptions, an isolated effect of coordination itself. The contrastδ 0 concerns the entire non-adherent stratumA =0; non-delegating runs generally form only a subset of this stratum. A.2 CACE decomposition and Wald identification Theorem A.1 (Decomposition and identification of the CACE). Under the common consistency and stability conditions above, if 0 < α adh < 1, then ITT = α adh CACE +(1− α adh ) δ 0 .(7) The additional mean exclusion restriction is δ 0 = E[Y(z)− Y(0)| A = 0] = 0(8) When it holds, the ratio identifies CACE: CACE = ITT α adh .(9) Ifα adh =1, the same equality holds without a restriction on the emptyA =0 stratum. Ifα adh =0, the CACE is undefined. Proof. The law of total expectation, applied to∆, gives E[∆] = P(A = 1)E[∆| A = 1] + P(A = 0)E[∆| A = 0] = α adh CACE +(1− α adh ) δ 0 . Substituting Eq.(8)and dividing byα adh >0 proves Eq.(9). Forα adh =1,A =1 almost surely, so ITT = CACE directly. The decomposition follows simply by conditioning onA(z)and does not useA(0) =0. The latter condition supports the one-sided-compliance and “complier” interpretation; it does not itself imply the Wald ratio. Mean exclusion is the additional conditional mean restriction on all target-Team runs withA =0, including those with unverifiable axis verdicts; it does not require equal outcomes on every such run. Thus Eq.(9)is the one-sided Wald identification result under mean exclusion and the common conditions, not an unconditional algebraic identity. The arm means identifyITTunder the common conditions, and the operational Team adherence labels identifyα adh ; the additional restriction supplies the ratio’s CACE interpretation. A.3 Paired diagnostic for the exclusion restriction The matched-task design can provide a stronger observation regime than the two arm means. If the recorded tuple(I,A,Y(z),Y(0))is a compatible paired observation from the specified joint law, where I is the task, then CACE = E[∆| A = 1],δ 0 = E[∆| A = 0]when P(A = 0) > 0.(10) Hence theA =0 paired contrast directly estimates the contrast relevant to mean exclusion, and the A =1 paired contrast estimates CACE without using exclusion. A finite-sample estimate of theA =0 contrast provides evidence about the restriction rather than establishing exact equality. This diagnostic uses the full structural-adherence label and includesA =0 runs in which some delegation occurs. Merely sharing a task identifier does not fix the cross-arm dependence of stochastic outcomes: the seed or another explicitly specified coupling protocol (including the product coupling that makes the two arms’ execution randomness independent conditional on the task) must make the baseline outcome in each tuple compatible with the same task-and-run coupling used in the definitions above. An unpaired comparison of adherent Team outcomes with the unconditional Single mean need not estimate Eq. (5). 14 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime A.4 Sensitivity to exclusion violations Proposition A.2 (Exact discrepancy and sensitivity bounds). For 0< α adh <1, without mean exclusion, ITT α adh − CACE = 1− α adh α adh δ 0 .(11) If|δ 0 |≤ D max for a nonnegative scalarD max , distinct from the indicatorA(z), the absolute discrepancy is at most (1− α adh )D max /α adh . For ε > 0, this bound is at most ε if and only if α adh ≥ D max D max + ε .(12) If instead δ 0 ∈ [ℓ 0 , u 0 ], a valid sensitivity interval is CACE∈ ITT−(1− α adh )u 0 α adh , ITT−(1− α adh )ℓ 0 α adh ∩ [−1, 1].(13) Proof. Rearrange Eq.(7)to obtain Eq.(11). Taking absolute values gives the bound; solving(1− α adh )D max /α adh ≤ εgives Eq.(12). Finally,CACE = ITT−(1− α adh )δ 0 /α adh decreases withδ 0 , giving the interval. Binary outcomes constrain the CACE to [−1, 1]. The discrepancy in Eq.(11)is a population identification discrepancy, not the finite-sample bias of a random ratio. The interval is valid but need not be sharp after all observed outcome constraints are imposed. A non-delegating-only diagnostic does not boundδ 0 unless it also covers the remaining A = 0 runs. A.5 Reporting requirements Table 5 separates the deployment effect from the two ways of learning about configuration-level CACE. All rows require the common scoring, consistency, reset, stability, and task-distribution conditions stated above. Table 5: Observation regimes and requirements for the Section 3.5 effect. TargetObservable inputAdditional condition or diagnostic ITT = E[Y(z)− Y(0)]Task-weighted arm outcomesStable, complete execution identifies the deployment contrast. CACE = ITT /α adh Arm outcomes and target-Team labels A r ; α adh = P(A = 1) α adh > 0, target-Team one-sided compliance A(0) = 0, and mean exclusion δ 0 = 0. Paired CACE and diagnostic Compatible (I, A, Y(z), Y(0)) tuples Directly estimate E[∆| A = 1] and, when A = 0 occurs, the exclusion-relevant contrast δ 0 = E[∆| A = 0]. A report should retain the task, assigned arm and configuration, outcome, per-axis adherence verdicts (including unverifiable), and the seed or coupling protocol used for any pairing. It should state whether the CACE was obtained from the Wald ratio under mean exclusion or directly from compatible pairs. These are statements about the assigned configuration’s effect among Team runs that satisfy the full operational structural-adherence rule, not a separate coordination-only estimand. A.6 Empirical check on the adherence experiment Table 6 computes the two paired contrasts of Appendix A.3 for the four configurations of Table 4. Each configuration is paired with the Single agent on the same 36 SWE-bench Pro tasks, and A r is the full structural-adherence label of that table, with failed and unverifiable runs atA r =0. For this analysis we take the cross-arm coupling of Appendix A.1 to be the product coupling conditional on the task: conditional on the taskI, the Team and Single execution randomness are independent draws. The 15 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 6. Wald ratio and paired contrasts against the Single agent on SWE-bench Pro. Paired,A r =1 is the mean Team-minus-Single difference over the tasks whose Team run is adherent, the direct estimate of Eq.(10)under the product coupling above; Paired,A r =0 is the same difference over the remaining tasks, the estimate ofδ 0 . By construction d ITT = b α adh · (Paired, A r =1) + (1− b α adh )· (Paired, A r =0). Differences in percentage points. ConfigurationAdherence (%)Tasks A r =1 / 0ITTWaldPaired, A r =1Paired, A r =0 Open card47.217 / 19+5.6 +11.8 −5.9+15.8 Budget stated, 2M75.027 / 90.00.0 −3.7+11.1 Mandatory card91.733 / 3+2.8 +3.0+3.00.0 Read-only94.434 / 2 −16.7 −17.6 −14.7−50.0 task-matched tuples in Table 6 are interpreted under this specified coupling; they do not rely on a shared execution seed. Under it, a Team run’sA r is independent of the Single run’s outcome on the same task. The gap between the Wald ratio and the paired estimate over adherent runs narrows as Adherence rises: 17.6 points at 47.2%, 3.7 at 75.0%, and at most 2.9 above 90%, as the factor(1− α adh )/α adh of Proposition A.2 predicts. Under the Mandatory card the three non-adherent tasks show no difference and the two estimates coincide. Under the Open card the+5.6 ITT arises on the 19 non-adherent tasks, while the 17 adherent tasks show−5.9; the Wald ratio of+11.8 therefore should not be read as the configuration-level CACE of this configuration. These estimates are descriptive. Re-running the Single agent unchanged flips the outcome of 5 of the 36 tasks (Appendix B.1), and among the 17 adherent Open-card tasks the two arms disagree on one task only, so the sign of−5.9 is not established. The table supports the direction of the Wald–paired gap across Adherence levels, not a collaboration effect of any single configuration. B Supplementary Experiments This part expands Section 4: the setup of its three experiments (B.1), every result table (B.2), how runs end and what they cost (B.3), traced runs (B.4), and the limitations (B.5). Appendix A gives the identification theory behind the Wald estimate of Section 3.5, and Appendix C describes the system. B.1 Experimental Setup Section 4 reports three experiments on one runtime. The adherence experiment changes one setting of a team at a time and records whether the declared organization happened (Table 3). The cross-harness comparison runs OpenCollab and three external harnesses on three benchmarks (Table 2). The Duo– single comparison runs a team whose handoffs are fixed by code against a single agent on the same tasks. This subsection gives the task sets, the organizations, the shared settings and the metrics, and then the design of each experiment. Benchmarks and task sets. The cross-harness comparison uses 193 tasks of SWE-bench Pro (Deng et al., 2026), 89 tasks of Terminal-Bench 2.1 (Merrill et al., 2026) and 113 tasks of DeepSWE (Huang et al., 2026). The adherence experiment uses 36 SWE-bench Pro tasks drawn at random, and every configuration runs on the same 36 tasks. The Duo–single comparison uses the task sets of the cross- harness comparison and is paired task by task on Terminal-Bench 2.1 and DeepSWE. A SWE-bench Pro run is resolved when every fail-to-pass test passes and no pass-to-pass test breaks, as judged by the benchmark’s official evaluator. A run whose tests produce no result counts as unresolved. DeepSWE is graded the same way on each task’s fail-to-pass and pass-to-pass tests. A Terminal-Bench 2.1 run is graded by the task’s own verifier, which checks the files, program output or running service in the candidate’s container; it passes when the verifier returns reward 1. When grading itself failed for an infrastructure reason (a download, a dependency, the verifier not starting), the same candidate was graded again, with no new model call. Nine Claude Code runs that did not end were stopped by hand and scored as failed: eight on DeepSWE andextract-elfon Terminal-Bench 2.1; their records keep the verifier’s missing reward. 16 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Organizations. The single agent, OpenCollab (Base) in Table 2, is one agent with six tools (bash, file_read, file_write, apply_patch, grep, git_diff) and no teammate. A team runs under one of two controllers (Appendix C.1). In a Workflow code issues every handoff. In a Team the model decides whether to brief a teammate, whom, and how much to do itself. Duo, OpenCollab (Duo) of Table 2, is a Workflow: two Coders each produce a candidate fix and code selects one (Appendix B.1.3). The adherence experiment runs two Team rosters, named by their topology. In Star an Adopter receives the task, may brief two Coders, and chooses and submits the final patch. Each Coder works in its own copy of the repository and reports back with a commit. Whether the Coders run is therefore the Adopter’s decision, which is what adherence records. Ring has an analyst that receives the task and submits, a coder and a tester. Every agent starts from the single agent’s configuration and adds its role card. Figure 4 draws Duo and the three Team topologies. Shared settings. In the adherence experiment every agent runs with reasoning effort at its highest setting and a streamed response, a limit of 200 steps, and the runtime’s reminder to edit after repeated reads switched off. Each agent has its own token allowance of 2M unless a Budget row changes it, and the runtime refuses a call that would exceed it (Appendix C.1). A run is stopped after 5,400 s of wall clock with Qwen3.8-Flash and after 7,200 s with DeepSeek-V4.1-Flash and GPT-5.6-Luna. In this experiment the container in which the agents run has no network access, so an agent cannot fetch a fix from outside the repository; only the runtime, outside the container, reaches the model endpoint. Which runs are kept.A run that ends for a reason the model caused stays in every denominator: it reached its allowance or the wall clock, produced no patch, or failed the tests. Its patch is graded as it stands. A run lost to an infrastructure fault is launched again on the same task at the same code. A run that still has no result counts as unverified for Adherence and as not resolved for Pass@1, so every configuration is read over its 36 runs. Metrics. A run is adherent when it is adherent on every one of the six axes of Table 15 that applies to its arm. In these experiments the runtime holds four of them (role boundary, budget sharing, information flow and context policy), so only participation and delegation vary (Appendix C.6 gives the reasons). A Team run is therefore adherent when every teammate, every agent other than the entry agent, did work: each spent tokens and produced at least one model output. The teammates are the two Coders in Star and the coder and the tester in Ring, so the two rosters count different agents. A run is unverified when the entry agent had already begun to delegate but the run was cut off before every teammate had worked, by the entry agent’s token allowance, by the wall clock, or by a teammate that stopped without replying, or when it failed after being launched again. Such a run might have collaborated had it not been cut off, so the log cannot settle it. A run in which the entry agent never delegated is not unverified. Unverified runs count as not adherent (A r = 0, Appendix A.1); counting every unverified run as adherent instead shows how much Adherence depends on this rule. Pass@1 is the share of the 36 runs that are resolved, and Tokens is the median per run over all agents. Intervals are Clopper–Pearson 95%. Two configurations are compared task by task on all 36 tasks, with an exact two-sided sign test and no correction for multiple comparisons. Re-running the single agent unchanged flips the outcome of 5 of the 36 tasks, so a difference of a few tasks between two rows is within run-to-run variation. B.1.1 The adherence experiment Table 7 lists every configuration of this experiment with the code used below. Every row of Table 3 is one configuration run once on each of the 36 tasks, twelve configurations in all. Every row except the two rings changes one setting of a single reference team: Star with Qwen3.8-Flash, the Open card, all tools, and a 2M-token allowance per agent that the card does not mention. The single agent runs the same 36 tasks twice, the second time unchanged, to measure run-to-run variation. Model. The same team with DeepSeek-V4.1-Flash or GPT-5.6-Luna in every role. GPT-5.6-Luna is served through a different endpoint from the other two models, so its row changes the endpoint as well as the model. 17 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 7. The configurations of the adherence experiment. Each row but S ′ is a row of Table 3, or the single agent it is compared with in Table 4; S ′ reruns S unchanged. Every agent has a 2M-token allowance except in B1 and B3. Each configuration runs the 36 tasks once. S and S ′ are the single agent; Star has an Adopter and two Coders, and both rings use the Ring roster (an analyst, a coder and a tester). Code Table 2 row CardModelEntry tools Topology S–Qwen3.8-Flashall– S ′ –Qwen3.8-Flashall– RreferenceOpenQwen3.8-FlashallStar M1ModelOpenDeepSeek-V4.1-Flash allStar M2ModelOpenGPT-5.6-LunaallStar T1ToolsOpenQwen3.8-Flashno editStar T2ToolsOpenQwen3.8-Flashread-onlyStar B1BudgetOpen + 0.5M stated Qwen3.8-FlashallStar B2BudgetOpen + 2M statedQwen3.8-FlashallStar B3BudgetOpen + 4M statedQwen3.8-FlashallStar C1ContextOptionalQwen3.8-FlashallStar C2ContextMandatoryQwen3.8-FlashallStar G1TopologyMandatoryQwen3.8-FlashallRing + shortcut G2TopologyMandatoryQwen3.8-FlashallRing Tools. All tools is the reference. No edit tools removes the Adopter’s two file-editing tools (file write and apply patch); it keeps the shell. Read-only leaves the Adopter only its reading tools andadopt, which checks out a Coder’s commit into the Adopter’s own tree. The Coders’ tools are unchanged. Budget.Each row adds two sentences to the Adopter ’s card and sets every agent’s allowance to the stated value: “Your budget for this run isXtokens, and each Coder’s is the same. On these tasks, a single agent working alone used 1,320,000 tokens on average (median 1,470,000).” The 2M row therefore differs from the reference only in those two sentences. Context: how the three cards are written.Every Adopter card has the same body: the role, the tools, the two Coders and what each is asked for, and how a run ends. The three cards differ only in the closing section, which Table 8 gives in full. The Open card leaves every choice to the Adopter and says that neither keeping the work nor handing it over is expected. The Mandatory card opens by saying that the division of the work is not the Adopter’s to decide, and then gives one procedure: implementation belongs to the Coders, who are asked one at a time, the second after the first has answered, and the Adopter chooses which candidate to deliver. The Optional card gives the same procedure without that opening sentence, and adds one sentence that lets the Adopter depart from it and implement the fix itself. The three cards thus form a ladder: no instruction, an instruction the Adopter may set aside, and an instruction it may not. Because the second Coder is briefed after the first has answered, the two candidates are not independent draws. In the Optional row a Coder that stopped without replying did not notify the Adopter, so an Adopter waiting for that reply could end the run; its Adherence and Pass@1 are therefore not strictly comparable with the other rows. Topology.The three rows differ in who can message whom (Figure 4). Star is the reference team: the Adopter and each Coder can message each other, and the two Coders cannot message each other. The two rings use the Ring roster, each agent with the reference’s tools. The analyst’s card opens with the same two sentences as the Mandatory card and then asks the analyst to leave the implementation to the coder. In the Ring, the analyst can message only the coder, the coder only the tester and the tester only the analyst. Ring + shortcut adds one edge, from the coder to the analyst, so the coder may report directly and skip the tester. B.1.2 The cross-harness comparison This comparison asks whether OpenCollab is a competitive harness, so that what the adherence experiment finds is not a property of a weak one. Five harnesses run on the three benchmarks, all with GPT-5.6-Luna: mini-swe-agent, Codex CLI, Claude Code, OpenCollab (Base) and OpenCollab (Duo). 18 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 8. The closing section of the Adopter’s card, the only text that differs between the Open, Optional and Mandatory cards; the rest of the card is identical. Text the Optional and Mandatory cards share is in gray; what distinguishes each card is in black. CardClosing section OpenWhat is yours to judge. Nothing above tells you what order to do things in, whom to talk to, or how much of the work to do yourself. Keeping all of it and handing parts of it over are both open to you, and neither one is what you are expected to do. Those are your calls, and you make them the way you would judge any piece of work: what the request actually needs, what is worth another agent’s attention, what it costs to describe a piece of it well enough to hand over, and what it costs to carry all of it in one budget and one conversation. OptionalWhat this run asks of you. Implementation is the Coders’. There are two of them and neither can see what the other is doing, so ask them one at a time: send the first a message withmessage_agentdescribing the problem and let it work out its own fix, and when its answer is back, write the second its own brief and let it work out its own. What goes into that second brief is yours – what you learned from the first is yours to use or to leave out. Which of the two this run delivers is yours to choose. You may depart from this and do the implementation yourself; that is your call, and there are runs where it is the better one. Useteam_statusfirst if you want to see who is live. Everything else about how you work – what you read, how you diagnose, what you put in each message – is still yours. MandatoryWhat this run asks of you. This run is not asking you to decide how to divide the work. It is asking you to divide it a particular way, so do that. Implementation is the Coders’. There are two of them and neither can see what the other is doing, so ask them one at a time: send the first a message withmessage_agentdescribing the problem and let it work out its own fix, and when its answer is back, write the second its own brief and let it work out its own. What goes into that second brief is yours – what you learned from the first is yours to use or to leave out. Which of the two this run delivers is yours to choose. Useteam_statusfirst if you want to see who is live. Everything else about how you work – what you read, how you diagnose, what you put in each message – is still yours. code Coder ACoder B Adjudicator Duo, Workflow Adopter Coder ACoder B Star analyst coder tester Ring + shortcut analyst coder tester Ring Figure 4. The four organizations. Left: Duo, a Workflow, OpenCollab (Duo) of Table 2; every arrow is a handoff that code makes on every run, and the dashed one runs only when the selection rule does not decide. The other three are the topologies of Table 3, run as Teams: an arrow means the agent at its tail can message the agent at its head, and whether it does is the model’s choice. Network access and answer leakage.On Terminal-Bench 2.1 the runner gives a task internet access when the task’s ownallow_internetflag allows it, and no network otherwise. The trajectories of OpenCollab (Base), OpenCollab (Duo), Claude Code and mini-swe-agent contain successful fetches of external content, so these harnesses did reach the internet on some tasks; for Codex CLI we found no fetch of reference material. Ten generation attempts, on nine harness–task pairs, received material that gives away the answer; all ten are on Terminal-Bench 2.1. We found them by searching the trajectories for paths into the benchmark’s repository,solutionandtestsdirectories, reference implementations and reference values, and then re-reading each hit: the tool call, the full text it returned, what the model did next, and whether the resulting candidate was selected. An attempt counts only if the model received the material; a 404, a failed fetch, ordinary project documentation or a search result showing only titles does not count. Most affected tasks were run again from the original task statement and base image in a fresh session, with the old candidate, the old session and any grading feedback kept out, and with a filter on the source and content of what a tool returns; ordinary dependency downloads stayed allowed, so this is not a rerun with the network off. Table 2 reports the results after this handling: every pass obtained with leaked material was withdrawn or replaced. B.1.3 The Duo–single comparison The adherence experiment shows that a Team’s handoffs may not happen, so a difference in Pass@1 between a Team and a single agent cannot be read as the effect of the organization. OpenCollab (Duo) is a Workflow: code issues every handoff, so both of its Coders run on every task. 19 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 9. OpenCollab (Duo) against OpenCollab (Base), paired by task.pis the exact two-sided sign test on the two middle columns. BenchmarkBoth passOnly DuoOnly BaseBoth failp Terminal-Bench 2.1 (89)6863120.51 DeepSWE (113)58215290.0025 Table 10. Adherence and Pass@1 of every configuration, with paired comparisons (only this configuration succeeded : only the other did, exact sign testp). Adherence for a Star run counts both Coders; for a Ring run, the coder and the tester. AdherencePass@1 Code% [95% CI]Unverified %vs reference%vs referencevs S S–69.4– S ′ –66.72:3 (1.00)– R47.2 [30.4, 64.5]13.9–75.0–3:1 (0.63) M166.7 [49.0, 81.4]22.213:6 (0.17)72.22:3 (1.00)– M297.2 [85.5, 99.9]2.819:1 (<0.001) 61.12:7 (0.18)– T186.1 [70.5, 95.3]0.015:1 (<0.001) 66.71:4 (0.38)1:2 (1.00) T294.4 [81.3, 99.3]2.818:1 (<0.001) 52.80:8 (0.008)2:8 (0.11) B147.2 [30.4, 64.5]2.89:9 (1.00)44.40:11 (0.001)0:9 (0.004) B275.0 [57.8, 87.9]8.314:4 (0.03)69.41:3 (0.63)3:3 (1.00) B377.8 [60.8, 89.9]2.813:2 (0.007)75.01:1 (1.00)3:1 (0.63) C163.9 [46.2, 79.2]16.7–61.1– C291.7 [77.5, 98.2]8.319:3 (<0.001) 72.21:2 (1.00)1:0 (1.00) G166.7 [49.0, 81.4]13.9–66.7–2:3 (1.00) G291.7 [77.5, 98.2]8.310:1 (0.01)63.93:4 (1.00)4:6 (0.75) The workflow.OpenCollab (Duo) has two solving agents and a judge that is called only when needed. Coder A looks for the narrowest root cause and writes a minimal complete fix. Coder B works from the same original task and aims at completeness across components, APIs, lifecycles and boundary conditions; it may receive the public test commands that Coder A ran. Each works in its own candidate environment, with the single agent’s configuration and six tools. The code then selects between the two candidates by a fixed rule; when the rule does not decide, it calls a read-only Adjudicator, which compares the candidates’ evidence with the public requirements of the task (on Terminal-Bench 2.1 it also has a read-only tool that pages through that evidence). The code, not an agent, applies the selected candidate. Paired results. Table 9 pairs OpenCollab (Duo) with OpenCollab (Base) task by task, with an exact two-sided sign test on the tasks where only one of them passed. The difference is significant on DeepSWE and not on Terminal-Bench 2.1. B.2 Full Results Table 10 gives, for every configuration of Table 7, the rates behind Table 3 and the task-by-task comparisons behind its reading. A comparisona:bcounts the tasks on which only this configuration succeeded (a) and those on which only the other one did (b), over the 36 tasks, with the exact two-sided sign test in parentheses. The reference is R for every Star row and G1 for G2. C1 is not compared, for the reason given under Context in Appendix B.1.1. The single-agent column is left empty for M1 and M2, whose models have no single-agent run. B.3 How Runs End and What They Cost This section covers the Qwen3.8-Flash configurations of Table 7, that is, every configuration except M1 and M2. A run ends in one of three ways (Figure 5). It completes when the entry agent (the Adopter, or the analyst in a Ring) submits. It is stopped for its token budget when the entry agent has used up its own allowance before submitting. It is stopped for wall clock when it reaches the 5,400 s limit. Budget stops dominate only where the allowance is smallest: in B1 92% of runs stop for budget and 8% 20 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime complete. With the largest allowance, B3, a third of the runs reach the wall clock instead. T2 completes most often, in 89% of its runs. Figures 6 and 7 show what a run costs in tokens and in time, and how a team run’s cost splits across its three agents. A single-agent run uses 1.3M to 1.4M tokens on average against an allowance of 2M. A team run uses 2.9M to 4.2M against 6M, except in the two Budget rows whose allowance differs: B1 uses 0.9M of 1.5M and B3 4.7M of 12M. The entry agent takes 33% to 50% of a team’s tokens, except in T2, where it can only read and adopt a commit and takes 17%. Because exactly one agent runs at any moment (Appendix B.5), a run’s wall clock divides into the time each agent spends in its own model calls and tool calls, plus the harness’s time between turns, which stays below 2%. The time split follows the token split: the entry agent’s share ranges from 27% in T2 to 59% in B1. A single-agent run takes 28 to 30 minutes on average and a team run 44 to 59 minutes, except in B1 (20 minutes). Figure 8 shows when each agent spends its tokens. Each run is cut into ten equal slices of time, from its start to its last model call, and the tokens in each slice are divided among the agents. The entry agent spends nearly all tokens in the first slices, the other two agents take over in the middle, and in every configuration the entry agent’s share rises again in the last slice. In B1 the entry agent spends all the tokens of the first three slices; in T2 it spends almost nothing in the middle of the run. 025%50%75%100% share of the 36 runs S S′ R T1 T2 B1 B2 B3 C1 C2 G1 G2 64%33% 53%47% 58%11%31% 67%17%17% 89% 8%92% 50%44% 58%33%8% 72%8%14% 72%22% 67%33% 69%8%22% completed (submitted)stopped: wall clockstopped: token budget failed Figure 5. How runs end, as a share of each configuration’s 36 runs (Appendix B.1). Completed: the entry agent submitted. Token budget: the entry agent used up its allowance before submitting. Wall clock: the run reached 5,400 s. Failed: no result after the run was launched again. Configuration codes as in Table 7. B.4 Three Runs, Traced Figures 9–11 draw three runs of the adopter–two-coder team from their event streams. Each lane is one agent; an outlined bar is one turn, and a block inside it is one model call, coloured by what the call did. Only one agent runs at a time, so a message to another agent is drawn as a vertical arrow at the moment it is sent, and the dotted line after it is the time the woken agent waits before its turn starts. Labelled points mark moments checked against the trajectory, and the bottom row of each panel is the code that would be graded if the run stopped at that moment; the legend sits above Figure 9, and all three share one page. Each is a single run chosen to show one way the declared organization is or is not realized, so none of them estimates how often that happens, and their time axes differ. Delegation that works (Figure 9). In the Read-only variant the adopter can read files and adopt a commit, but cannot edit. It sends the same task to both coders. Coder A writes the fix, and every 21 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime 024681012 tokens per run (M): dots = runs, bar = mean, red = allowance S S′ R T1 T2 B1 B2 B3 C1 C2 G1 G2 1.3 / 2 1.4 / 2 2.9 / 6 3.3 / 6 3.7 / 6 0.9 / 1.5 3.9 / 6 4.7 / 12 3.4 / 6 3.8 / 6 3.9 / 6 4.2 / 6 (a)Tokens per run (mean / allowance) 050%100% share of mean tokens per run 462925 372637 173943 502426 412733 432533 363430 333236 373330 333235 (b)By agent Adopter (Ring: analyst; S, S′: the one agent)Coder A (Ring: coder)Coder B (Ring: tester) Figure 6. Tokens per run. (a) Each dot is a run; the bar is the mean and the red tick is the run’s allowance (each agent’s allowance times the number of agents), with “mean / allowance” in millions at the right. (b) The mean split across agents, in percent. In a Ring the three agents are the analyst, the coder and the tester. 0306090 minutes per run: dots = runs, bar = mean, red = wall-clock limit S S′ R T1 T2 B1 B2 B3 C1 C2 G1 G2 30 / 90 28 / 90 44 / 90 53 / 90 50 / 90 20 / 90 55 / 90 59 / 90 50 / 90 51 / 90 47 / 90 52 / 90 (a)Wall clock per run (mean / limit, min) 050%100% share of mean wall clock per run 482922 402336 273437 591920 452629 452628 393327 363033 482229 442233 (b)By agent Adopter (Ring: analyst; S, S′: the one agent)Coder A (Ring: coder)Coder B (Ring: tester) harness, between turns Figure 7. Wall clock per run. (a) Each dot is a run; the bar is the mean and the red tick is the 5,400 s limit, with “mean / limit” in minutes at the right. (b) The mean split across agents, where an agent’s time is the duration of its own model calls and tool calls; grey is the harness’s time between turns, a thin sliver at the right of each bar. 22 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime 0 100% RT1T2B1B2 startend 0 100% B3 startend C1 startend C2 startend G1 startend G2 Adopter (Ring: analyst; S, S′: the one agent)Coder A (Ring: coder)Coder B (Ring: tester) Figure 8. Share of tokens by agent in each tenth of a run, from its start to its last model call, averaged over the configuration’s runs. One panel per team configuration of Table 7 with Qwen3.8-Flash. line of the graded patch comes from that edit; Coder B independently writes an equivalent one. The adopter adopts Coder A’s commit and asks Coder B to review it, and Coder B agrees. Briefed but not waited for (Figure 10).In the reference team the adopter briefs both coders at minute 12.7. The tool that sends the brief replies that the teammate runs on its next turn and that ending the turn is how the adopter waits. The adopter instead implements the change itself; at minute 27.2 it writes that it will wait for the coders’ replies, then makes five more model calls. Its budget runs out at minute 33 before it submits, and only then do the coders run. Coder A reports its commit to the adopter, which has already stopped. The five target tests the graded code fails reference a constant name that the task statement does not give, so this failure does not follow from the missing delegation. Cross-review that picks the failing commit (Figure 11).Again in the Read-only variant, the adopter adopts Coder A’s commit, then Coder B’s to compare them. Coder A reports that Coder B’s commit passes all fifteen of its probes and recommends keeping it, and the adopter ships it. Coder B’s code reads a file in a way that bypasses the permission error that a target test injects, and fails that test. The figures show forms the organization takes; they are not a comparison with the single agent. B.5 Limitations Each limitation below narrows how far a number in this paper reaches. Three models, not a model sweep. The three model snapshots of Table 3 show that Adherence moves when the model changes, but three snapshots cannot estimate how it varies across models in general. One agent at a time.All agents in a team share one lock, so exactly one agent runs at any moment; a message to a busy agent is delivered when its turn ends. Organizations whose value depends on agents working truly in parallel are therefore outside what these experiments measure. One workflow, not the space of workflows.OpenCollab (Duo) is a single decomposition of the task. A different script could deliver a different organization just as faithfully and perform differently. 23 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime readedittest / buildmessagecommit / adopt / submitturn woken, waiting for its turnmessage that wakes itout of budgetgraded code passesfails 0 5 10 15 20 Coder A's code Adopter Coder A Coder B Graded code min sends both coders the same task writes every graded line writes an equivalent fix adopts A's commit reviews it: “Agree” Figure 9. Read-only variant (the adopter can only read files and adopt a commit), resolved: two coders fix the task and the adopter adopts Coder A’s commit. A SWE-bench Pro task from the teleport repository. 010203040 50 60 Adopter's own code 5 of 32 target tests fail Adopter Coder A Coder B Graded code min briefs both coderswrites the fix itself“Let me wait”, keeps working out of budget, never submits reports its commit; the adopter has stopped Figure 10. Reference team, unresolved: the adopter briefs both coders but keeps working until its budget ends; the coders run only after its turn ends. A SWE-bench Pro task from the flipt repository. 0 5 10 15 20 25 30 35 40 4550 A's codeB's code B's code fails TestDMI/realistic Adopter Coder A Coder B Graded code min sends both coders the same taskadopts A's commitadopts B's, to compare tie-break: “keep B's” ships B's Figure 11. Read-only variant, unresolved: cross-review ships the commit that fails. A SWE-bench Pro task from the teleport repository. 24 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime C System Details This part describes the code behind Sections 3.1–3.5: the runtime and its two controllers (C.1), how the code is organized (C.2), how a collaboration is declared (C.3), an audit of ten agent artifacts, with OpenCollab scored by the same rule, against the conditions of Section 3.3 (C.4), the run record (C.5), and how adherence is computed from it (C.6). C.1 Runtime and Control Regimes Sections 3.1 and 3.2 describe one runtime that carries three control regimes: a Team, a script-ordered Workflow, and a single agent. This appendix gives the limits every session runs under, its context compaction, the two controllers above it, the workspaces, and the token budget. The session loop. Every session, whichever controller opened it, runs the same loop, a state machine with ten states.IDLEprecedes the first step.PRECHECKadmits or refuses the next model call,CALLING_LLMmakes it, andHANDLING_RESPONSEsends a reply with tool calls to EXECUTING_TOOLS;AUTOSAVINGrecords the step and returns toPRECHECK. A tool whose result is deferred parks the session inAWAITING_EVENTSuntil the controller resumes it.DONE,STOPPEDand ERRORare the end states (Table 14). A transition that is not in the table of legal transitions raises an error instead of being taken (Figure 12). STOPPED controlled stop; reason recorded DONE final answer, or submit called ERROR unhandled fault solid edges: PHASE_TRANSITIONS IDLEPRECHECKCALLING_LLMHANDLING_RESPONSE EXECUTING_TOOLSAUTOSAVING AWAITING_EVENTS tool calls empty reply, retried once next step all tools immediate deferred tool all pending rows filled step called submit final answer limit or interruptover budget or overflowoutput truncated checked transitionescape from any state: fail() to ERROR, cancel() to STOPPED terminal state; each returns to IDLE only when a new turn starts Figure 12. The session state machine that every session runs. Solid edges are the checked transition table; a transition not in the table raises an error instead of being taken. Two escapes bypass the table from any state: an unhandled fault ends the session inERROR, and a cancellation from outside the loop ends it inSTOPPED. STOPPEDis a controlled stop at a limit or a refusal, with the reason recorded (Table 14), not a fault. Each end state returns to IDLE only when a new turn starts. Limits.Every limit is checked before a call is made.PRECHECKstops the session once it has reached its step limit, its limit on repeated tool calls, or its token allowance. InCALLING_LLMthe call is then priced with a conservative estimate of its input, and refused if it would not fit in what remains of the allowance. One check also runs after the call, because a call can still cost more than its estimate; a role that ends up past its allowance in this way stops at once. A role stopped at any of these points ends in STOPPEDwith the reason recorded (Table 14). Because an organization decides only which sessions exist and when they start, never what happens inside one, these limits bind identically whichever controller opened the session. Context compaction.Before each model call the session compacts a copy of its history in four stages applied in order. The first caps every tool result at 16,000 characters and runs on every call. The other 25 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Team controller application/scheduler*.py the model decides whether to hand off: message_agent, along a declared edge an undeclared edge: refused and recorded a busy recipient: the message waits for its turn to end Workflow controller application/workflow.py a Python script fixes the order of work each step of the script opens one session and the script passes its output on the run's token pool spent: the next call is refused Session runtime application/session_run.py one session per seat; a single-agent run is one session, no controller Limits PRECHECK: per-seat step, token limits Model call CALLING_LLM: refused if it cannot fit Compaction on a copy, before each call opens a session per declared role, each with its own allowanceopens a session per script step Workspaces adapters/worktree_pool.py entry role: the graded tree each other Team role: its own git worktree Workflow roles: one shared directory Tracer adapters/trace.py one ordered event stream per run each event: timestamp, step, run id, type, payload also the controllers' message and budget records tools act in events TW Figure 13. Two controllers over one session runtime. The Team controller opens a session for every declared role, each with its own token allowance; the model decides whether to hand work on withmessage_agent, and a message along an undeclared edge is refused and recorded. The Workflow controller is a Python script that fixes the order of work and opens one session per step; a single-agent run is one session with no controller above it. All sessions run the same code, which checks every limit before a model call, acts in the role’s workspace, and writes to one ordered event stream per run (Appendix C.5). Table 11. The three arms. The last column says how a run is known to have realized its arm. Only in the Team is it a measurement of the model’s choice. In the Workflow the script, not the model, issues every handoff it reaches. ArmDefinitionDiffers from Single inRealized Singleone agent works the task to completion—trivially Workflowshared role cards; a script decides the order of work, what each role receives, and when to stop organization, enforcedby code Teamshared role cards; the model decides whether to delegate, what to send, and when to stop organization, offeredmeasured, b α adh three act only when the estimated history exceeds a trigger set 33,000 tokens below the model’s context window. They clear the content of old tool results while keeping the five most recent verbatim, then drop whole old tool exchanges, oldest first, and last replace the remaining old span with a summary written by a separate model call. Each stage that fires writes a record to the event log. If the provider still rejects the prompt as too long, a forced pass ignores the trigger and the call is retried once; a second rejection stops the session. The stored history is never altered by compaction. Two controllers. Above the loop, a controller decides which sessions exist, when each one starts, and how their outputs travel (Figure 13). The Team controller opens a session for every declared role and leaves it to the model whether to send work along the declared edges. The Workflow controller is a Python script that opens sessions in a fixed order and passes their outputs on itself. A single-agent run is one session on the same runtime. Table 11 lists the three arms built from these regimes. Workspaces. The entry role is the role that receives the task. Its workspace is what the benchmark grades when the run ends, and we call that workspace the graded tree. The Team controller builds every declared teammate before the first model call, in its own git worktree over the same object store, and gives it no task: a teammate starts working only when a message reaches it. The token budget. Each role has its own fixed token allowance, set before the run. Building a teammate takes nothing from the team’s tokens, so a teammate that is never used spends nothing. 26 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Grading runs after the run has ended and is not charged to any role’s allowance. C.2 How the Code Is Organized The system is two code bases. The runtime, the Python package opencollab, runs the sessions and the two controllers of Appendix C.1. The evaluator, the packageopencollab_eval, defines the arms, drives the batches and prepares each run’s patch for grading. Every path and count below is taken from the code of the adherence experiment. Four layers in the runtime.Of the runtime’s 167 Python files, 158 fall into four layers (Table 12), and apart from five import statements in the command-line interface, described below, no module imports from a layer above its own. The domain layer holds the session state machine and the other plain data types, and it uses nothing outside the Python standard library. The application layer holds the session loop with its limits, the two controllers and context compaction. The adapters layer holds the code that reaches outside the process: model providers, tools, workspaces and git worktrees, and the event log. The bootstrap layer reads a team file, resolves its tool names to implementations, and assembles the other three layers. Above the layers sits a public surface of nine files: the client, whoseagent,team and workflow calls start the three regimes, and the modules for tools, environments, team files and workflows. Figure 14 draws the four layers as the rings of a clean architecture; the layer contract in the code carries that name. Bootstrap wires the layers together from a team file Adapters models, tools, workspaces, event log Application session loop, controllers, compaction Domain state machine and plain data, stdlib only source dependencies point inward ports: declared by the application, implemented by the adapters Bootstrap Adapters Application Domain state machine Figure 14. The runtime’s four layers as a clean architecture. Every import points inward, from bootstrap to adapters to application to domain; the application declares ports, interfaces that the adapters implement, so it never imports a model provider, a tool or a workspace directly. Figure 15 gives the files in each layer. Figure 15 draws the same layers with the modules this appendix names. The layering is checked. Four import contracts, run in continuous integration, state the rules: the order of the layers; that the domain imports none of the runtime’s third-party dependencies; that the client and the modules for tools, environments and workflows never import the adapters or the domain directly; and that no two modules of the application, adapters or domain layer import each other in a cycle, except one known cycle between two modules of the Workflow controller. All four hold. We confirmed this by running the contracts on the code of the experiments, and confirmed that the check can fail by adding two forbidden imports to the domain, which broke exactly the two contracts they violate. Counting import statements directly agrees: every import in the four layers 27 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 12. The runtime’s layers, top to bottom. Apart from five statements in the command-line interface (see text), no module imports from a layer above its own. Lines counts every line of the layer’s Python files. LayerFilesLinesWhat it holds Public surface9894The client (sdk/client.py); the modules for tools, environments, team files and workflows Bootstrap226,690Loading a team file (team_config.py, Appendix C.3); resolving tool names (tool_registry.py); assembling a session, a team or a workflow run Adapters7216,171Model providers and the per-model capability table (llm/); the tools, among them the patch tool, the shell and message_agent (tools/); local, container and worktree workspaces; the sandbox policy (safety.py); the per-run event stream (trace.py, Appendix C.5) Application5115,202The session loop and its limits (session_run.py); the Team controller (application/scheduler.py and its scheduler_ * and _scheduler_ * modules); the Workflow controller (workflow.pyand itsworkflow_ * modules); context compaction (shaping/) Domain131,820The ten-state session machine and its legal transitions (session.py, Appendix C.1); the team topology (team.py); the table of sessions that the Team controller keeps, as plain data (domain/scheduler.py); agents, events and token estimates points into its own layer or a lower one, except five statements in three files of the command-line interface, which belongs to the adapters layer, that reach up into the bootstrap layer. The contract file lists these as known exceptions. The experiments never start the runtime through the command-line interface. The evaluator.The evaluator ’s modules import the runtime only through its public surface. An arm is therefore defined by what it passes through that surface, namely role prompts, tool bundles, and a team file or a script, while the session loop and the code that enforces its limits are the runtime’s own in every arm. The team files are kept in the runtime’s repository, one per configuration. The batch driver (generation/gen_prediction_batch.py) runs every arm over the same ordered task list. Theexperiment/package holds the batch specification that fixes a configuration before it is launched, and the per-configuration report that computes adherence and its intervals from the run records. C.3 Declaring a Collaboration A team is declared in one file (Figure 2b). The file names the entry role, gives each role a system prompt, the agent profile it starts from, the names of the tools it may call and its token allowance, optionally overrides the role’s model, names the context policy every session runs, and lists the directed edges along which one role may message another. Tool names are resolved to implementations from a registry when the team is assembled, so one file yields the same tool set in every run that uses it. Roles are separated by what they can call, not only by what they are asked to do. A role whose tool set contains no file-editing tool cannot edit through the file tools, whatever its prompt says. A boundary set in the tool set is enforced at call time, so a call outside it is refused rather than left to the model’s choice. The runtime enforces the declaration at the moment of each call. A tool call outside the calling role’s tool set, and a message along an edge the file does not declare, are refused, and the refusal is recorded as an event (Appendix C.5). The edges decide who can reach whom. In the Star team of Figure 4 the entry role and each Coder can message each other, and the two Coders cannot message each other; in the Ring the analyst can 28 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Evaluator opencollab_eval workflows/ the Workflow arm's script generation/ batch driver experiment/ specs, reports, arm registry opencollab Runtime Public surface 9 files 894 lines sdk/client.py agent, team, workflow calls tools.py environments.py public modules teams.py workflows.py public modules Bootstrap 22 files 6,690 lines team_config.py loads a team file scheduler_factory.py assembles a team workflow_runtime.py runs a workflow Adapters 72 files 16,171 lines llm/ model providers tools/ the agents' tools worktree_pool.py git worktrees trace.py event stream cli/ command line Application 51 files 15,202 lines session_run.py session loop, limits scheduler*.py workflow*.py the two controllers shaping/ context compaction Domain 13 files 1,820 lines session.py ten-state machine team.py team topology scheduler.py session table (data) imports only the public surface import directionupward import (cli/ only), a listed exception cli/: 5 imports into this layer Figure 15. The two code bases. The runtime’s layers run top to bottom, each with its file and line counts (Table 12), and a module imports only from its own layer or the ones below; the one exception is five imports from the command-line interface into the bootstrap layer (dashed), which the experiments do not use. The evaluator reaches the runtime only through its public surface. message only the coder, the coder only the tester and the tester only the analyst. Every team needs a path back to the entry role: without one, a teammate could not return its result, and a delegation that did happen but could not be returned would look like a model that never delegated. The roster is fixed before the first model call and cannot change during a run; a request to add an agent is refused and recorded. C.4 Audit of Ten Agent Artifacts Section 3.3 names seven conditions for a controlled comparison of agent organizations. Five ask whether a factor can be set explicitly before a run: the model, the tools, the token budget, the context policy and the topology. Two ask whether a run can be checked afterwards against what it was declared to be (Realized, Compared). Table 1 marks each condition on ten public agent artifacts, with OpenCollab scored by the same rule in the last row. This appendix gives the rule and the reason behind every mark other than✓. How the marks were placed. Nine artifacts were read in code at a pinned version; Claude Code is closed-source, and its row rests on vendor documentation. A✓records that a researcher sets the factor in a setting the artifact itself reads, such as a configuration file, a declarative specification or a command-line option, and that the artifact applies it to every agent. A – records that the artifact ships a mechanism but takes it only in the researcher ’s own code, or meets the condition only in part. A× records that no mechanism meets the condition even in part; a setting that only moves the threshold of a built-in behavior or switches it off does not count as one. Each row is read on the mechanism that leaves the organization to the model, such as teams, handoffs, subagents or agents called as tools, as OpenCollab’s row is read on its Team controller; a script the researcher fixes before the run is outside the table. Marks follow each artifact’s default settings. An artifact that runs a single agent is marked on that agent, and× under Topology and Compared. A condition is marked✓ when: •Model: every agent’s model can be set, and a declared model is not replaced by another without notice; 29 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime • Tools: each role can be given its own tool set; •Budget: each model call is admitted against the calling agent’s remaining allowance before it is sent, and refused if it does not fit; • Context: a setting chooses the policy that decides what each model call sees; •Topology: the roles and the edges between them are fixed before the run rather than started by the model while it runs; • Realized: each run’s record holds the whole declared organization, including roles that never act, with every model call and tool call attributed to its run and agent; •Compared: Realized holds, every message between agents names its sender and receiver, and every attempt to leave the declaration is refused or flagged in the record. The pinned versions and the file-and-line evidence behind every mark are released with the paper. Why the other rows fall short.On Model and Tools, AG2 and LangGraph take both only as constructor arguments in the researcher’s code, HAL passes a model name on to the agent’s own code, and a Codex CLI role file can only switch tools off from a fixed set; HAL leaves the tools to the agent. On Budget, no external artifact prices a call against the remaining allowance before sending it. AutoGen, SWE-agent, OpenHands, Claude Code and Inspect AI stop an agent once its limit has been reached or after a call has returned; AG2 only raises alerts, LangGraph limits steps, DeepSeek Harness caps the output of a single call, Codex CLI’s token budget sits behind a feature marked as not ready for external use, and HAL bounds a task by wall-clock time. On Context, AutoGen and SWE-agent choose the policy in configuration; OpenHands, DeepSeek Harness, AG2, LangGraph and Inspect AI let it be chosen only in part or only in code; Claude Code and Codex CLI expose only a threshold and an off switch, and HAL does not run the agent’s loop. On Topology, AutoGen and Inspect AI hold the agents and whom each may call in a configuration file; AG2 and LangGraph wire agents in code; in Claude Code, Codex CLI, DeepSeek Harness and OpenHands the roles are declared but the model decides at run time which agents to start, and the edges can be held only by a hook the researcher writes; HAL leaves the organization to the agent’s own code. On Realized, SWE-agent stores its full configuration with every run; seven rows record only part of the declaration, such as the settings of the agents that were started but not of those that were not, or roles without their models; AutoGen’s logs carry no run identifier, and LangGraph’s traces hold what each call sent but not the graph. On Compared, only Inspect AI refuses or flags departures in its record, and only for agents joined as tools; SWE-agent’s record is complete, but it runs a single agent. Our own row. OpenCollab’s row is read on its Team controller. Its team file sets each role’s model, tools, token allowance and context policy together with the edges; every call is admitted against the calling role’s remaining allowance before it is sent (Appendix C.1); each run’s record opens with the declared organization, every role with its model and tools and every edge, so a declared teammate that never acts still appears; and a message along an undeclared edge or a call outside a role’s tool set is refused and recorded (Appendix C.5). C.5 The Run Record Section 3.4 replaces prose logs with an ordered event stream. This appendix gives the files each configuration writes, the event types, and the recorded termination reasons. Three files per configuration. The batch manifest holds the assigned condition: arm, task list, role cards with their sha256, per-role allowance, the runtime and evaluator versions, and the driver’s environment. The event log holds the realized side, one ordered stream per run. The metrics file holds one row per run, with the image the run was generated in. The run id joins the three files, and every event carries it. Rates are computed from individual runs, never from pre-averaged configurations, and adherence is computed from the manifest and the event log after the run (Appendix C.6). Events.Every event carries a timestamp, a step number that increases within the run, the run id, the event type, and a payload whose fields depend on the type and, for every type the analyses below 30 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 13. The seven event types. Model calls and agent finishes settle whether a teammate worked; tool executions ofmessage_agentshow who briefed whom; a session termination, read with the token counts of the model calls, identifies the runs whose decision the allowance cut short. Event typeWhat it adds to the common fieldsWhat it supports Model callToken counts and latency of the stepPer-role spend; the organizational overhead of Appendix C.5 Tool executionTool name, arguments and resultWho briefed whom; which edges were addressed. Refused messages and refused tool calls are written as separate events Agent finishWhether the role produced contentWhether a teammate worked, read together with its model calls Session terminationThe role’s termination reason (Table 14) Which runs were cut off by the allowance after delegation began Refused spawnThe reason for the refusalThat the roster cannot change within a run (Section 3.2); that a single-agent run cannot coordinate Worktree changeCommit shaWhich role’s commits reached which worktree (Section 3.4) Context shapingWhich compaction stage fired this turn Whether a call’s input was compacted (Section 3.3) Table 14. The end states of a session (Figure 12) and the reasons recorded with them. Each reason is a string whose fixed prefix is shown; some continue with a count, such as the tokens used. The wall-clock limit is not among them: the evaluator stops a run that reaches it from outside the runtime and records that in the run’s metrics row. StateRecorded reasonWhen it is recorded DONE completed; submittedThe role gave a final answer, or called the submit tool. STOPPED budget exceededAt PRECHECK, the role had already spent its allowance. budget exhausted before model call The next call, priced before it was sent, would not fit in what remained. budget exceeded after model callThe last call cost more than its estimate and took the role past its allowance. step limit reachedThe session reached its step limit. loop block limit reachedThe role kept repeating a tool call without progress. context overflowThe prompt exceeded the model’s context window even after compaction. output truncatedThe provider cut the response off at its generation limit. required tool was not called after correction The role was required to call a tool, was reminded once, and still did not. interrupted by user; cancelledThe session was stopped from outside. ERRORthe exception’s type and messageAn unhandled fault, including a role whose container or workspace was withdrawn during the run. read, name the role. Table 13 lists the seven types the analyses read and what each one supports; the controllers also write message and budget records. Because the payload names the role, what each role spent, how many steps it took and which tools it called are read directly off the stream, with no prose log to parse. Cost metrics. Organizational overhead. Multi-agent configurations incur prompt costs even when the assigned organization is not fully realized. Because Table 13 logs token usage by role, OpenCollab measures this prefix tax by comparing, on the same tasks, the cost of Team runs withA r =0 against the cost of the single-agent baseline. WithC(z)andC(0)the potential total costs under target Team 31 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime configuration z and Single, defined like Y(z) and Y(0) in Appendix A.1, Overhead 0 := E[C(z)− C(0)| A(z) = 0], \ Overhead 0 = 1 n 0 ∑ i: A i (z)=0 C i (z)− C i (0) , (14) where the sum runs over then 0 tasks whose Team run hasA r =0 and pairs each with the Single run of the same task. This metric is the additional cost of carrying the Team configuration among runs that do not fully realize it. Comparing these Team runs with the unconditional Single mean would instead mix that cost with differences in task composition, since non-adherent runs need not fall on typical tasks. Workload balance. For fully adherent Team runs (A r =1), OpenCollab measures how computational effort is divided across the active roles. Lettingp s (r) = C s (r)/C(r)denote the token proportion consumed by roles, the framework defines workload balance using normalized Shannon entropy across the active rolesS : Balance(r) =− ∑ s∈S p s (r) log p s (r) log|S| .(15) The aggregate balance metric is computed as the expectation ofBalance(r)across all adherent runs whereA r =1. A value near one reflects an egalitarian division of labor, whereas a value near zero indicates that a single role dominated execution despite the presence of teammates. C.6 Adherence, Axis by Axis Section 3.5 defines a run as adherent when every axis that applies to its arm is adherent. This appendix states the six axes (Table 15), how a verdict is reached, and which axes can vary in these experiments. How a verdict is reached. Each run has an assigned side, written in the configuration’s manifest before the run and restated as events in the run’s event log, and a realized side, the rest of that log. Each axis compares the two and returns adherent, deviant or unverifiable. b α adh is the share of a configuration’s runs that are adherent on every axis that applies. An axis that does not apply to an arm, such as delegation for the single agent, is left out of that arm’s conjunction rather than scored as a pass. Which axes can vary here. An adherent verdict says that a run did not depart from its declaration on that axis, which is not the same as the axis having had anything to measure. Four of the six axes are held by the runtime in these experiments. The runtime offers each agent only the tools of its assigned set and refuses any other call, so no agent can act outside it, which is the one deviation the role-boundary axis records. An allowance refuses the call that would exceed it and records the refusal, so no role can pass its allowance unrecorded, the budget-sharing deviation. A message along an edge the topology does not declare is refused. Every role runs the same context policy. In the Team the conjunction is therefore decided by participation and delegation. A teammate starts working only when a message reaches it (Appendix C.1), so a run in which every declared teammate worked has also delegated, and the conjunction comes down to one test: a Team run is adherent when every declared teammate, every role other than the entry role, spent tokens and produced at least one model output. This is computed for every run from the run record: the roles from the declaration restated in the event log, and each role’s tokens and outputs from its saved session. The four held axes still have to be declared: a comparison that gives each model a different tool set, allowance or context window is exactly the case they exist for. Unverifiable counts against adherence. When the log cannot establish what happened on an axis, the verdict is unverifiable and the run is not counted as adherent. This happens, for instance, when a run was cut off after the entry agent had begun to delegate but before every teammate had worked (Appendix B.1 lists the cases). Such a run hasA r =0 (Appendix A.1). Missing evidence can therefore lower a reported rate but never raise it; counting every unverifiable run as adherent instead shows how much the rate depends on this rule. 32 OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime Table 15. The six adherence axes. The assigned value is written before the run, and the realized value is read from the event log. AxisRealized value, read fromDeviation Participationthe teammates that spent tokens and produced at least one model output a declared teammate did no work Delegationa teammate that spent tokens and produced at least one model output no such teammate, where the assigned regime offers or requires delegation Role boundarythe tool calls attributed to each agenta call outside the agent’s assigned tool set is executed Budget sharingeach role’s token draw against its own allowance a role reached its allowance and no refusal was recorded Information flow message_agent calls in the event log, in every arm a message travels along an edge the topology does not declare Context policythe compaction level applied on each turna level other than the assigned one fires 33