Paper deep dive
Can AI Scientists Coordinate at Runtime?
Zijian Liu, Yangzhixin Luo, Junyu Lu, Yi Li, Yu Chen, David Xu, William F. Shen, Xinchi Qiu, Xisen Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/3/2026, 1:53:32 AM
Summary
The paper introduces Runtime Agent Coordination (RAC), a framework for multi-agent AI scientists that selects agents, assigns scoped work contracts, and provides artifact-grounded verification at runtime. Evaluated on Agent Laboratory, EvoScientist, and ARK using ResearchClawBench, the study finds that runtime selection (R2) yields the highest mean scores, while adding contracts and verification (R3) reduces performance relative to R2 but varies against native execution (N0).
Entities (8)
Relation Signals (10)
Agent Laboratory → evaluatedon → ResearchClawBench
confidence 99% · evaluation across Agent Laboratory... on ResearchClawBench
ARK → evaluatedon → ResearchClawBench
confidence 99% · evaluation across... ARK on ResearchClawBench
EvoScientist → evaluatedon → ResearchClawBench
confidence 99% · evaluation across... EvoScientist... on ResearchClawBench
Runtime Agent Coordination → includesmechanism → Runtime Selection
confidence 95% · RAC... selects agents from existing AI-scientist hosts during execution
Runtime Agent Coordination → includesmechanism → Artifact-Grounded Verification
confidence 92% · provides artifact-grounded verification
Runtime Agent Coordination → includesmechanism → Scoped Work Contracts
confidence 92% · assigns scoped work contracts
Runtime Selection → yieldshighestscore → EvoScientist
confidence 90% · Runtime selection yields the highest observed mean score for each host
Runtime Selection → yieldshighestscore → ARK
confidence 90% · Runtime selection yields the highest observed mean score for each host
Runtime Selection → yieldshighestscore → Agent Laboratory
confidence 90% · Runtime selection yields the highest observed mean score for each host
Artifact-Grounded Verification → informs → Subsequent Agents
confidence 88% · Verification informs subsequent agents without blocking transitions or discarding artifacts
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent AI scientists have shown improving performance across a diverse range of tasks. Yet a common approach is design-time agentic orchestration, which typically relies on fixed workflows. In contrast, human scientists coordinate and adjust their division of labor at runtime. We therefore ask: can AI scientists also coordinate at runtime? To this end, we introduce Runtime Agent Coordination (RAC), which selects agents from existing AI-scientist hosts during execution, assigns scoped work contracts, and provides artifact-grounded verification. Verification informs subsequent agents without blocking transitions or discarding artifacts. We conduct a single-seed exploratory evaluation across Agent Laboratory, EvoScientist, and ARK on ResearchClawBench, preserving host models, tools, and permissions under host-calibrated budgets. Four cumulative conditions separate native execution, runtime communication, runtime selection, and the combined addition of contracts and verification. Runtime selection yields the highest observed mean score for each host; adding contracts and verification reduces these means, with host-dependent outcomes relative to native execution. These results motivate runtime coordination while exposing the limits of additional coordination mechanisms under constrained budgets. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.00980v1
- Canonical: https://arxiv.org/abs/2610.00980v1
Trouble viewing inline? Open PDF directly →
Full Text
116,606 characters extracted from source content.
Expand or collapse full text
Systemind Research CAN AI SCIENTISTS COORDINATE AT RUNTIME? Zijian Liu 1 , Yangzhixin Luo 2 , Junyu Lu 3 , Yi Li 1 , Yu Chen 1 , David Xu 1 , William F. Shen 4,* , Xinchi Qiu 4,* , Xisen Wang 1,† 1 University of Oxford (ox.ac.uk) 2 King Abdullah University of Science and Technology (kaust.edu.sa) 3 University of Sydney (usyd.edu.au) 4 University of Cambridge (cam.ac.uk) ABSTRACT Multi-agent AI scientists have shown improving performance across a diverse range of tasks. Yet a common approach is design-time agentic orchestration, which typically relies on fixed workflows. In contrast, human scientists coor- dinate and adjust their division of labor at runtime. We therefore ask: can AI scientists also coordinate at runtime? To this end, we introduce Runtime Agent Coordination (RAC), which selects agents from existing AI-scientist hosts dur- ing execution, assigns scoped work contracts, and provides artifact-grounded verification. Verification informs subsequent agents without blocking transi- tions or discarding artifacts. We conduct a single-seed exploratory evaluation across Agent Laboratory, EvoScientist, and ARK on ResearchClawBench, pre- serving host models, tools, and permissions under host-calibrated budgets. Four cumulative conditions separate native execution, runtime communication, run- time selection, and the combined addition of contracts and verification. Run- time selection yields the highest observed mean score for each host; adding con- tracts and verification reduces these means, with host-dependent outcomes rel- ative to native execution. These results motivate runtime coordination while exposing the limits of additional coordination mechanisms under constrained budgets. Code is available at https://github.com/systemind-team/ Runtime-AI-Scientist. 1INTRODUCTION AI scientists automate increasingly large parts of the research lifecycle, including idea generation, literature synthesis, experiment design, implementation, analysis, and paper writing (Lu et al., 2024; Yamada et al., 2025; Schmidgall et al., 2025; Goel et al., 2025; Gottweis et al., 2025; Yang et al., 2026). Multi-agent systems distribute these activities across specialized roles that exchange prompts, critiques, code, and research artifacts. This communication determines what agents share. Coordi- nation is a separate decision: which agent should act next, what it should produce, whether its output is valid, and how the run should continue. Many prominent AI-scientist systems organize agent interactions through predefined stages, loops, and fallback rules (Lu et al., 2024; Yamada et al., 2025; Tang et al., 2025; Schmidgall et al., 2025; Xin et al., 2026). Search, repair, and debate add flexibility within these workflows, but do not necessarily let the system select the most relevant agent across stages (Yamada et al., 2025; Liu et al., 2026). Other systems, such as EvoScientist (Lyu et al., 2026), incorporate persistent memory and self-evolution. These differences motivate us to investigate whether a shared runtime coordination approach can improve research performance across hosts with varying degrees of native flexibility. Selecting the next agent based on the current research state matters because long-horizon scientific failures are state dependent. A missing result calls for an experimenter, a broken bibliography calls for a deterministic check or writer, or a contradictory figure may require code rather than another prose revision. Recent evaluations locate substantial AI-scientist failures in experimental design, grounding, and scientific reasoning rather than only in software crashes (Xu et al., 2026; Fei et al., * William F. Shen and Xinchi Qiu serve only in an advisory role. † Corresponding author. Contact: research@systemind.io 1 arXiv:2610.00980v1 [cs.MA] 1 Oct 2026 Systemind Research 2026). A fixed workflow cannot remove these failures merely by adding more agents. It must either execute all contingencies, which increases cost, or omit contingencies that later turn out to be necessary. We therefore ask whether runtime coordination is a useful alternative to fixed workflows for multi- agent AI scientists (Wang, 2026). To address this question, we propose Runtime Agent Coordination (RAC), illustrated in Figure 1, which uses current artifacts, open problems, execution history, and remaining budget to select an authorized existing agent. Scoped contracts define what each selected agent should accomplish, while artifact-grounded verification provides feedback on its output for subsequent work. Every verdict retains the workspace and host transition. We apply the same framework across different hosts under matched budgets, keeping their underlying agents and tools fixed to assess the benefits of communication and coordination. Our core contributions are threefold. First, we formally study runtime coordination in multi-agent AI scientists and formulate next-agent selection as a state-dependent decision, distinguishing it from communication under native workflow control. Second, we instantiate the RAC framework in multi- agent AI scientists such as Agent Laboratory, EvoScientist, and ARK without replacing their native agents, tools, or permissions. Third, we empirically test whether runtime coordination improves research performance under matched budgets, reporting research scores, token usage, and estimated inference costs across a range of tasks. (a) Overall coordination framework Comparing native workflow control with runtime coordination (RAC) under the same setup. Research task + initial artifacts same input Native workflow control predefined transitions and fallback rules Runtime coordination (RAC) At each step, the current agent selects the next agent Research understand problem Experiment run experiments Write draft paper Review check quality predefined fallback Available host agents (same set) ResearchExperimentWriteReview Inter-agent communication · R1–R3 Exchange research context, artifacts, and messages Decisions stay with the current agent. No separate orchestrator. R2–R3 Current state Current agent 1. analyze current state 2. assess candidate agents 3. select the next agent Execute selected agent Write Update state with new results iterate until task completion Controlled comparison (identical across both workflows) Agents same set of host agents Models same model family/versions Tools same tool set and access Permissions same permissions and sandbox Budget matched compute budget Only the coordination mechanism changes. (b) Controlled factors Kept identical across all conditions. Same agents identical set of host agents Same models same model family/versions Same tools same tool set and access Same permissions same permissions and sandbox Matched budget same compute budget held fixed across conditions (c) RAC decision loop At each step, the current agent decides who acts next. new evidence state signals available capabilities Current state research state, artifacts, goals Artifact / result feedback new artifacts, updated state Candidate agents available capabilities, specialties, costs Selected agent next agent to execute (d) Condition ladder Progressively adding coordination capabilities. N0 native lifecycle R1 + communication R2 + runtime selection R3 + contracts cumulative evaluation conditions Example: a formatting error calls for Write rather than Experiment. formatting error Fixed-workflow example Follows the host-native successor policy with predefined fallback transitions. + verification (a) Overall coordination framework Comparing native workflow control with runtime coordination (RAC) under the same setup. Research task + initial artifacts same input Native workflow control predefined transitions and fallback rules Runtime coordination (RAC) At each step, the current agent selects the next agent Research understand problem Experiment run experiments Write draft paper Review check quality predefined fallback Available host agents (same set) ResearchExperimentWriteReview Inter-agent communication · R1–R3 Exchange research context, artifacts, and messages Decisions stay with the current agent. No separate orchestrator. R2–R3 Current state Current agent 1. analyze current state 2. assess candidate agents 3. select the next agent Execute selected agent Write Update state with new results iterate until task completion Controlled comparison (identical across both workflows) Agents same set of host agents Models same model family/versions Tools same tool set and access Permissions same permissions and sandbox Budget matched compute budget Only the coordination mechanism changes. (b) Controlled factors Kept identical across all conditions. Same agents identical set of host agents Same models same model family/versions Same tools same tool set and access Same permissions same permissions and sandbox Matched budget same compute budget held fixed across conditions (c) RAC decision loop At each step, the current agent decides who acts next. new evidence state signals available capabilities Current state research state, artifacts, goals Artifact / result feedback new artifacts, updated state Candidate agents available capabilities, specialties, costs Selected agent next agent to execute (d) Condition ladder Progressively adding coordination capabilities. N0 native lifecycle R1 + communication R2 + runtime selection R3 + contracts cumulative evaluation conditions Example: a formatting error calls for Write rather than Experiment. formatting error Fixed-workflow example Follows the host-native successor policy with predefined fallback transitions. + verification Figure 1: Same agents, different coordination. (a) Native transitions versus state-dependent se- lection by the current agent, with inter-agent communication for exchanging context, artifacts, and messages; this schematic does not imply that all hosts lack runtime delegation. (b) Agents, models, tools, permissions, and run-level budget are held fixed. (c) At each step, the current agent selects the next available agent based on current state, without a separate orchestrator; returned artifacts update the state for the next decision. (d) The cumulative conditions add communication (R1), runtime selection (R2), and contracts plus verification (R3) to the native lifecycle (N0). 2RELATED WORK 2.1AI SCIENTISTS AI scientists aim to automate research activities ranging from idea generation and experimentation to analysis and scientific writing (Lu et al., 2024; Schmidgall et al., 2025; Tang et al., 2025). The 2 Systemind Research AI Scientist organizes these activities into an end-to-end workflow, while AI Scientist-v2 introduces agentic tree search within experimentation (Lu et al., 2024; Yamada et al., 2025). Agent Laboratory assigns specialized agents to research phases, AI-Researcher combines literature exploration with experimental development, and ARK provides a human-steered research harness (Schmidgall et al., 2025; Tang et al., 2025; Xin et al., 2026). These systems demonstrate different ways to organize specialized capabilities across the research lifecycle. Other systems emphasize iterative feedback and adaptation. Co-scientist combines hypothesis gen- eration, debate, and ranking (Gottweis et al., 2025); AutoResearchClaw integrates multi-agent de- bate, execution repair, and verifiable reporting (Liu et al., 2026); and ARIS uses cross-model re- view and evidence checks to assess research claims (Yang et al., 2026). EvoScientist accumulates reusable research experience through persistent memory and self-evolution (Lyu et al., 2026). Mov- ing beyond predefined pipelines, freephdlabor supports workflows determined by real-time agent reasoning, while AutoScientists organizes decentralized teams around evolving experimental evi- dence (Li et al., 2025; Gao et al., 2026). Our work examines whether a shared runtime coordination approach can improve existing hosts with different native control mechanisms. Scientific output also requires evaluation beyond successful execution. ResearchClawBench eval- uates end-to-end research on real-paper-derived tasks, while diagnostic studies examine failures in experimental design, grounding, and scientific reasoning (Xu et al., 2026; Fei et al., 2026). Auto- mated review introduces further uncertainty because assessments can vary across evaluator mod- els (Ravideshik & Kejriwal, 2026). We therefore distinguish feedback used during research from external benchmark evaluation of the final outputs. 2.2MULTI-AGENT COORDINATION Multi-agent systems combine specialized roles through communication and structured collaboration. AutoGen supports programmable agent conversations, MetaGPT combines role-specific workflows with shared structured messages, and multi-agent debate iteratively exchanges candidate answers and critiques (Wu et al., 2023; Hong et al., 2023; Du et al., 2023). These mechanisms shape the information available to agents, while coordination determines how their contributions are organized and which agent acts next. Several approaches explicitly optimize or adapt this organization. GPTSwarm represents agents as computational graphs and optimizes prompts and inter-agent connectivity (Zhuge et al., 2024). Dy- LAN selects agent teams and supports dynamic interactions during task solving (Liu et al., 2023). In scientific applications, freephdlabor and AutoScientists further demonstrate reasoning-driven work- flows and self-organizing collaboration (Li et al., 2025; Gao et al., 2026). Building on these di- rections, RAC studies runtime selection across existing AI-scientist hosts while preserving their underlying agents, tools, and permissions. Its cumulative conditions distinguish communication under native control from runtime selection and the combined addition of contracts and verification. Feedback and resource use are integral to this comparison. MetaGPT incorporates executable feed- back, while analyses of multi-agent failures identify inter-agent misalignment and verification or termination problems (Hong et al., 2023; Cemri et al., 2025). In RAC, scoped contracts define the selected agent’s task, and artifact-grounded verification informs subsequent work without veto- ing transitions. Because additional coordination can increase inference costs, we follow cost-aware agent evaluation principles (Kapoor et al., 2024; 2025) and compare research scores, token usage, and estimated costs under matched run-level budgets. 3RUNTIME AGENT COORDINATION FOR AI SCIENTISTS 3.1PROBLEM FORMULATION At decision step t, an AI scientist has a research state s t comprising its current objective, saved artifacts, unresolved problems, execution history, and remaining budget. LetA t denote the agents currently permitted by the host. Communication adds information to this state, while runtime coor- dination uses it to select the next agent and specify its task. At a nonterminal handoff, the next agent 3 Systemind Research is selected as a t+1 = ( π host (s t ),native control (N0, R1), π RAC (s t ,A t ), runtime coordination (R2, R3), a t+1 ∈A t .(1) Here, π host is the host’s native successor policy, and π RAC is the shared runtime selection policy. Native policies may already include conditional branches, fallback rules, or adaptive delegation. Our comparison examines the effect of replacing this native selection policy with a shared state- dependent policy while retaining the host’s permitted agents. Each condition generates its own research trajectory, so s t need not be identical across conditions. 3.2ENGINEERING IMPLEMENTATION Figure 2 summarizes the execution cycle. We use the same RAC package and condition config- uration across Agent Laboratory, EvoScientist, and ARK. Each host bridge serializes native state, invokes an existing capability, and returns artifacts and usage records through a shared interface. Bridges contain no host-specific runtime selection rules, verification criteria, or benchmark hints. This design preserves each host’s agents, tools, and permissions without requiring identical host internals. Inter-agent communication work.request → work.result → + work.verification (in R3) Shared context, artifacts, and messages across agents and tools. Communication Checkpoint Host-native Runtime selection R2 · Shared policy Work contract R3 · Task specification Execute Host-native Verifier R3 · Artifact-grounded • Research state & artifacts • Open issues & execution history • Remaining budget • Select authorized capability • Read current state • Return next action • Objective • Readable / writable artifacts • Required output (format, criteria, etc.) Execute selected host capability (with existing tools and permissions) Return artifacts Contract (task spec) Returned artifacts Verify persisted artifacts against the contract ✓ supported × refuted ? inconclusive Retain state → Fresh checkpoint for next cycle (research state, artifacts, history, budget) Verification feedback → next selected agent Advisory, non-blocking (informs next cycle without blocking transitions) RAC levels Progressive capabilities Each level adds new capabilities to the previous. N0 native lifecycle R1 communication R2 + selection R3 + contracts + verifier N0/R1: native workflow / communication only; R2/R3: runtime coordination. i Stopping conditions Independent of verification. Budget exhaustion Hard hop limits Terminal provider errors ! Genuine host completion Agents, tools, and permissions unchanged. 1 2 3 Figure 2: The RAC execution cycle. At each handoff, agents exchange requests and results. In R2– R3, the current agent selects the next authorized host capability from a fresh checkpoint, without a separate orchestrator. R3 additionally specifies a work contract and verifies the returned artifacts, passing feedback to the next selected agent. N0 executes the host-native lifecycle without additional communication mechanisms or a RAC phase loop. R1 introduces inter-agent communication while preserving the native successor at every handoff. R2 retains this communication mechanism and enables the current agent to select the next authorized capability using artifacts, execution history, unresolved problems, and remaining 4 Systemind Research budget. Agents exchange structured requests and results through a common communication channel. Thus, R1–R2 changes next-agent selection while retaining the same communication mechanism. Communication state is isolated between runs, and cross-run memory is reset across all conditions. R3 adds scoped work contracts and artifact-grounded verification. Each contract specifies the ob- jective, readable and writable artifacts, and required output. The verifier checks persisted artifacts against the contract and records a supported, refuted, or inconclusive verdict. All ver- dicts preserve the workspace and host transition; feedback enters the next selected agent’s prompt rather than directly triggering retries, rerouting, rollback, or stopping. Runs terminate on budget exhaustion, hard hop limits, terminal provider errors, or native host completion. Checkpoints and event records document coordination decisions, while external benchmark evaluation assesses final outputs separately from in-loop verification. 4EXPERIMENTAL SETUP 4.1BASELINES We evaluated RAC with Agent Laboratory (Schmidgall et al., 2025), EvoScientist (Lyu et al., 2026), and ARK (Xin et al., 2026). These hosts span predefined control loops, fixed research phases, and native adaptation, allowing us to examine runtime coordination across different native control mechanisms. Each host’s native lifecycle serves as its N0 baseline. 4.2BENCHMARK AND CONDITIONS ResearchClawBench provides real-paper-derived tasks across scientific domains. Each task supplies a research question, related literature, raw data, and an executable environment while withholding the target paper (Xu et al., 2026). Our evaluation covers mathematics, neuroscience, information science, energy, life science, and physics. We evaluate Agent Laboratory and EvoScientist on ten tasks each and ARK on five tasks. All runs use seed 0. Table 1 defines the cumulative conditions used to examine communication, runtime selection, and contracts with verification. R3 versus N0 compares the full framework with native execution; R2–R3 treats contracts and verification as a joint addition rather than isolating their individual effects. Table 1: Four cumulative conditions. R1 preserves native successors, R2 adds shared runtime selec- tion, and R3 jointly adds scoped contracts and verification. IDConfigurationNewly enabled mechanism N0Native lifecycleNative execution without additional communication mechanisms or a RAC phase loop. R1CommunicationInter-agent communication with the native successor at each handoff. R2+ Runtime selectionSelect a host capability from a fresh checkpoint using the shared policy. R3+ Contracts and verification Scope the task and check artifacts; pass supported, refuted, or inconclusive feedback to the next selected agent. Retain state and transitions for all verdicts. 4.3CONTROLS AND ANALYSIS Within each host, we hold base models, native role prompts, tools, permissions, artifact interfaces, and task inputs fixed across conditions. Hop ceilings are calibrated to the corresponding host’s N0 hop count rather than set uniformly across hosts. The budget-matching design concerns config- ured resource ceilings, not equal realized consumption; routing and verification are charged to the run’s budget. Budget accounting is described in Appendix B.2. Cross-run memory and evolution stores are reset before each run to isolate within-run coordination. Model choices and inference-cost accounting are detailed in Appendix B. The primary outcome is the official ResearchClawBench weighted final score. We report input and output token usage and estimated inference costs separately. Each task–host–condition contributes one final score from the consolidated results record, including numeric zeros. A zero is an evaluator 5 Systemind Research outcome, not an encoding of a missing report or a failed evaluation. Within-task differences from N0 are averaged over the same evaluated tasks within each host. Table 4 provides task-level scores; all summaries use unrounded values before display rounding. 5RESULTS 5.1FULL RUNTIME COORDINATION ACROSS HOST SYSTEMS Table 2 compares full RAC (R3) with native execution. R3 increased mean scores for Agent Lab- oratory and ARK, while EvoScientist remained nearly unchanged. Recorded input-token usage and estimated costs decreased for Agent Laboratory but increased for the other two hosts. Table 2: Means over recorded results, with equal task weighting of final evaluation scores. Mean total input tokens are shown in millions (M). Costs are standardized estimates in USD from recorded input and output tokens using the cost-accounting convention in Appendix B. HostConditionMean score Mean total input tokens (M)Est. cost (USD) ARKNative (N0)16.6648.8569.31 ARKFull RAC (R3)17.9868.7696.69 Agent LaboratoryNative (N0)5.474.267.09 Agent LaboratoryFull RAC (R3)9.882.564.48 EvoScientistNative (N0)15.9911.4915.44 EvoScientistFull RAC (R3)15.9638.5851.50 5.2CUMULATIVE CONDITION COMPARISON R2 had the highest observed mean for all three hosts (Table 3). R1 also exceeded N0 for Agent Laboratory and ARK but fell below it for EvoScientist. Adding contracts and verification in R3 reduced the mean relative to R2 in every host. Table 3: Mean scores across all available tasks for each host and condition. Tasks are equally weighted using their final evaluation scores; recorded zeros are retained. Shared configurationARKAgent Lab.EvoScientist N0: Native lifecycle16.665.4715.99 R1: Runtime communication17.409.6312.07 R2: + Runtime selection18.4212.0818.53 R3: + Contracts and verification17.989.8815.96 Table 4 shows that improvements over N0 outnumber declines under R2 and R3, with the broadest gains under R2; R1 produces a more evenly mixed pattern. R3 improved Math-000 and Math-001 for both Agent Laboratory and EvoScientist, but reduced EvoScientist’s scores on Physics-002 and Math-002 and Math-003. Gains persisted across all three conditions for ARK on Physics-002 and Agent Laboratory on Life-001. Figure 3 contrasts mean improvements with their task-level spread. R2 increased scores over N0 by 6.61 points for Agent Laboratory, 2.55 for EvoScientist, and 1.76 for ARK. Both gains and de- clines occurred within every host and condition, showing that positive means did not imply uniform benefits. Detailed statistics of token usage and estimated inference costs are reported in Appendix C.1. 5.3TRANSFERABILITY TEST We additionally evaluated RAC on an exploratory subset of five DiscoveryBench (Majumder et al., 2025) tasks, without a predefined sampling rule. Agent Laboratory covers all five tasks; EvoScientist 6 Systemind Research Table 4: Task-level final scores at seed 0. The upper block reports the five tasks evaluated with all three hosts. The lower block reports five additional tasks evaluated with Agent Laboratory and EvoScientist. Each cell gives the score; R1–R3 additionally show the change from the same host’s N0 in brackets. Dark green and red denote gains and declines exceeding 3 points, respectively; smaller changes remain uncolored. Colors describe magnitude, not statistical significance. HostCondition Neuroscience 000 Energy 001 Life 001 Neuroscience 002 Physics 002 ARK N010.0029.309.806.6027.60 R17.60 [-2.40]27.60 [-1.70]9.40 [-0.40]5.70 [-0.90]36.70 [+9.10] R28.40 [-1.60]31.90 [+2.60]14.10 [+4.30]5.70 [-0.90]32.00 [+4.40] R313.80 [+3.80]27.90 [-1.40]10.45 [+0.65]3.15 [-3.45]34.60 [+7.00] Agent Lab. N09.009.201.303.300.00 R18.00 [-1.00]10.50 [+1.30]4.65 [+3.35]5.40 [+2.10]6.60 [+6.60] R23.40 [-5.60]12.20 [+3.00]6.90 [+5.60]5.70 [+2.40]24.55 [+24.55] R310.40 [+1.40]5.90 [-3.30]2.90 [+1.60]5.05 [+1.75]4.30 [+4.30] EvoScientist N08.4017.7016.207.8039.20 R16.60 [-1.80]19.90 [+2.20]13.50 [-2.70]4.60 [-3.20]10.40 [-28.80] R26.00 [-2.40]25.80 [+8.10]13.75 [-2.45]8.30 [+0.50]43.10 [+3.90] R313.60 [+5.20]28.50 [+10.80]14.10 [-2.10]7.05 [-0.75]16.45 [-22.75] HostCondition Math 000 Math 001 Math 002 Math 003 Information 003 Agent Lab. N017.1511.100.000.003.60 R118.90 [+1.75]23.10 [+12.00]3.60 [+3.60]10.05 [+10.05]5.45 [+1.85] R222.50 [+5.35]27.50 [+16.40]8.40 [+8.40]6.60 [+6.60]3.00 [-0.60] R318.30 [+1.15]29.90 [+18.80]4.80 [+4.80]9.00 [+9.00]8.20 [+4.60] EvoScientist N016.0516.8010.6016.3010.80 R19.60 [-6.45]32.60 [+15.80]5.30 [-5.30]9.45 [-6.85]8.70 [-2.10] R221.60 [+5.55]36.20 [+19.40]11.00 [+0.40]11.25 [-5.05]8.30 [-2.50] R322.35 [+6.30]37.70 [+20.90]3.20 [-7.40]9.70 [-6.60]6.90 [-3.90] R1R2R3 Condition −4 −2 0 2 4 6 8 Mean score difference from N0 +4.16 +6.61 +4.41 -3.92 +2.55 -0.03 +0.74 +1.76 +1.32 (a) R1R2R3 Condition −30 −20 −10 0 10 20 Task-level score difference from N0 (b) Each point: one task | Solid line: host mean Agent LaboratoryEvoScientistARK Figure 3: Within-task score improvements. (a) Mean score differences from N0 for each host under R1–R3. (b) Individual task-level differences from the same host’s N0; solid black segments indicate host-specific means. Points are horizontally offset for visibility. All conditions use the same task set within each host. The panels use different vertical scales and show descriptive results, not uncertainty across independent runs. 7 Systemind Research covers only NLS SES, giving six task–host pairs rather than a complete two-host matrix. Figure 4 shows mean improvements over N0 of 19.05, 18.07, and 25.92 points under R1–R3, respectively. R3 achieved the largest aggregate gain, extending the evaluation beyond ResearchClawBench without establishing representative cross-benchmark performance. R2’s slightly lower mean than R1 was driven by the incarceration and archaeology tasks, despite improvements on the other three tasks. In incarceration, R2 analyzed simulated data and reported synthetic year indices rather than the requested survey years. In archaeology, additional quantities in R2’s answer reduced the evaluator’s variable-matching score, while relation and context matching remained unchanged. Both runs completed the same six-stage sequence as R1 without exhausting their budgets. These cases indicate task drift and scoring sensitivity rather than a demonstrated disadvantage of runtime selection. N0R1R2R3 Condition 0 10 20 30 Mean score di f erence from N0 0 +19.05 +18.07 +25.92 (a) R1R2R3 Condition Pair 1 Pair 2 Pair 3 Pair 4 Pair 5 Pair 6 +13.33+22.22+100.00 +31.94+31.94-12.50 +20.000.00+66.67 +2.62+15.62-9.38 +50.00+28.57+28.57 0.00+19.050.00 (b) −100 −50 0 50 100 Figure 4: Transfer to DiscoveryBench. (a) Mean score improvements over N0, computed within each host and task, averaged across available hosts within each task, and then equally across the five tasks. (b) Improvements for the six task–host pairs, labeled Pairs 1–6; blue indicates gains and red indicates decreases. Table 7 lists the tasks, hosts, and absolute scores; the first two rows share the same task. Scores are rescaled to 0–100; the improvement relative to N0 is zero by construction for N0. Points in (a) show descriptive means, and connecting lines are visual guides. 5.4TRACE-LEVEL ANALYSIS: THREE COORDINATION EFFECTS We inspected 1,181 hop excerpts and 180 external scoring receipts from the 60 cells covering five shared tasks, three hosts, and N0–R3. Selected trajectories illustrate coordination mechanisms and failure modes, not their prevalence or the causes of aggregate score differences. The excerpts are incomplete, scientific claims were not independently recomputed, and hop indices are zero-based. Evidence–action closure effect. In Physics-002 with EvoScientist R2, the native reviewer re- ported at hop 6 that the report contained only 39 bytes despite existing code, analyses, and figures. At hop 7, the selected debug capability reported producing a 337-line report of approximately 22 KB. Ten recorded scientific-artifact changes and the final inventory support the conclusion that a report was produced. This handoff connected a specific defect to corrective work, although subsequent debug calls made no recorded scientific-artifact changes before the hop limit. The feedback came from the host reviewer, not R3’s verifier; delivery repair does not establish scientific correctness. Planning stagnation after diagnosis. In Physics-002 with EvoScientist R3, the reviewer iden- tified a standard-error discrepancy: 0.011 in the report versus 0.015 in the data. At hop 8, the planner attributed it to duplicated observations and proposed code changes followed by regeneration of results. However, hops 8–13 remained planner calls with no recorded scientific-artifact changes, ending at the 14-hop limit. The trajectory illustrates a failure to move from diagnosis to imple- mentation. Claims of repeated verification appeared in agent messages but were not independently confirmed from verifier events. 8 Systemind Research Self-consistent drift effect. In Neuroscience-000 with EvoScientist R3, the native reviewer re- ported that numerical and code–report consistency checks passed and assigned a score of 8/10. External evaluation nevertheless identified missing cross-laboratory, sex, and environment SHAP comparisons, yielding a final score of 13.6. The two scores use different criteria and scales and are not directly comparable. Similarly, Life-001 with Agent Laboratory R2 analyzed ten synthetic patients rather than the supplied seven-patient dataset. These cases distinguish internal consistency from coverage of the research objective, without identifying whether divergence originated in task interpretation, data limitations, or coordination. Appendix D provides detailed trajectories and evidence boundaries, including additional Math cases outside the shared-task subset. 6DISCUSSION Runtime selection achieved the highest observed means on ResearchClawBench, whereas contracts and verification produced the largest aggregate gain on the exploratory DiscoveryBench subset. This contrast suggests that the benefits of additional coordination depend on the host and research setting, rather than increasing uniformly with each mechanism. Under matched budgets, communication and verification add context and inference costs that may displace useful research work. Runtime selection can offset this overhead by directing work to- ward relevant agents, but R2’s lower recorded costs in ARK and Agent Laboratory did not extend to EvoScientist. Additional budget might accommodate R3’s overhead, yet the repeated planning trajectory in Section 5.4 shows that diagnosis need not lead to implementation. Distracting context or inaccurate verifier feedback could also weaken outcomes. Budget sweeps with per-hop usage and artifact changes would help distinguish resource constraints from ineffective delegation. Host structure may further shape these trade-offs. Agent Laboratory’s fixed phases may benefit from information carried between stages, whereas EvoScientist already incorporates native adaptation. Differences in native prompts and token consumption may also affect how effectively hosts use additional context. These remain hypotheses: aggregate scores and usage do not establish why individual runs improve or decline. Our single-seed evaluation does not measure variability across independent runs, and bounded benchmark tasks do not establish performance in open-ended discovery. Costs are standardized token-based estimates rather than provider invoices. R3 jointly adds contracts and verification, so their individual effects remain unresolved. 7CONCLUSION We presented RAC as an alternative coordination approach for multi-agent AI scientists. It selects existing agents from current state and, in R3, adds scoped contracts and artifact-grounded verifica- tion feedback. Across all available ResearchClawBench scores, R2 had the highest task-weighted mean for each host, whereas R3’s performance relative to N0 varied across hosts. These rankings are descriptive and do not establish robustness across independent runs. Standardized token-based costs describe resource use without establishing a general performance or cost advantage. ETHICS AND REPRODUCIBILITY STATEMENT Adaptive scientific workflows may accelerate research but can also amplify unsupported claims or optimize for fallible reviewer signals. RAC preserves host permissions, scopes tasks through contracts, and records verification evidence, but verification is not a safety gate and does not pre- vent unsupported outputs from propagating; accountable human review remains necessary, par- ticularly in safety-critical, dual-use, or regulated domains. Sections 3 and 4 describe the host adaptation boundary and experimental conditions, Table 4 reports task-level scores, Appendix B specifies the execution and judge models and token-based cost calculation, and Appendix B.4 de- fines task weighting and the paired comparisons in Figure 3. The implementation is available at https://github.com/systemind-team/Runtime-AI-Scientist. 9 Systemind Research REFERENCES Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, et al. Why do multi-agent LLM systems fail? arXiv preprint arXiv:2503.13657, 2025. URL https://arxiv.org/abs/2503.13657. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. URL https://arxiv.org/abs/2305.14325. Yanlin Fei, Nazhou Liu, Xinmiao Yu, Shaolong Chen, Lei Li, Rahul Thapa, et al. How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks. arXiv preprint arXiv:2608.14905, 2026. URL https://arxiv.org/abs/2608. 14905. Shanghua Gao, Ada Fang, and Marinka Zitnik. AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation. arXiv preprint arXiv:2605.28655, 2026. URL https://arxiv.org/abs/2605.28655. Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F Shen, Ilias Leon- tiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, et al. Training AI co-scientists using rubric rewards. arXiv preprint arXiv:2512.23707, 2025. URL https://arxiv.org/abs/ 2512.23707. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Petar Sirkovic, Artiom Myaskovsky, et al. Accelerating Scientific Discovery with Co-Scientist. arXiv preprint arXiv:2502.18864, 2025. URL https://arxiv.org/abs/2502.18864. Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent col- laborative framework. arXiv preprint arXiv:2308.00352, 2023. URL https://arxiv.org/ abs/2308.00352. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter. arXiv preprint arXiv:2407.01502, 2024. URL https://arxiv.org/ abs/2407.01502. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S. Siegel, et al. HAL: Holistic Agent Leaderboard. https://hal.cs.princeton.edu, 2025. Ed Li, Junyu Ren, Xintian Pan, Cat Yan, Chuanhao Li, Dirk Bergemann, and Zhuoran Yang. Build Your Personalized Research Group: A Multiagent Framework for Continual and Interactive Sci- ence Automation. arXiv preprint arXiv:2510.15624, 2025. URL https://arxiv.org/ abs/2510.15624. Jiaqi Liu, Shi Qiu, Mairui Li, Bingzhou Li, Haonian Ji, Siwei Han, Xinyu Ye, Peng Xia, Zihan Dong, Meng Chen, Congyu Zhang, et al. AutoResearchClaw: Self-reinforcing autonomous research with human-AI collaboration. arXiv preprint arXiv:2605.20025, 2026. URL https://arxiv. org/abs/2605.20025. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic LLM-powered agent network for task-oriented agent collaboration. arXiv preprint arXiv:2310.02170, 2023. URL https://arxiv.org/abs/2310.02170. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha.The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.arXiv preprint arXiv:2408.06292, 2024. URL https://arxiv.org/abs/2408.06292. Yougang Lyu, Xi Zhang, Xinhao Yi, Yuyue Zhao, Shuyu Guo, Wenxiang Hu, Jan Piotrowski, Jakub Kaliski, Jacopo Urbani, Zaiqiao Meng, Lun Zhou, and Xiaohui Yan. EvoScientist: Towards multi- agent evolving AI scientists for end-to-end scientific discovery. arXiv preprint arXiv:2603.08127, 2026. URL https://arxiv.org/abs/2603.08127. 10 Systemind Research Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark.DiscoveryBench:Towards data-driven discovery with large lan- guage models.In International Conference on Learning Representations,2025. URL https://proceedings.iclr.c/paper_files/paper/2025/hash/ 0d70af566e69f1dfb687791ecf955e28-Abstract-Conference.html. Vaibhava Lakshmi Ravideshik and Mayank Kejriwal. Can AI Evaluate AI Scientists? A Benchmark- ing Study of Autonomous Research Generation Systems Using Automated Multi-Model Review. arXiv preprint arXiv:2607.28631, 2026. URL https://arxiv.org/abs/2607.28631. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, et al. Agent Laboratory: Using LLM Agents as Research Assistants. arXiv preprint arXiv:2501.04227, 2025. URL https://arxiv.org/abs/2501.04227. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous scien- tific innovation. arXiv preprint arXiv:2505.18705, 2025. URL https://arxiv.org/abs/ 2505.18705. Xisen Wang. Position: Stop hardcoding multi-agent workflows that general agents will outgrow. In ICML 2026 Workshop on Agents in the Wild: Safety, Security, and Beyond, 2026. URL https: //openreview.net/forum?id=Icz1RJ2t0j. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. URL https://arxiv.org/abs/2308.08155. Jihao Xin, Yangzhixin Luo, Muhammad Bilal, and Marco Canini. ARK: A human-steered research harness for AI-assisted scientific discovery. In ICML 2026 AI for Science Workshop, 2026. URL https://openreview.net/forum?id=fSAiWmpk0O. Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, et al. Research- ClawBench: A Benchmark for End-to-End Autonomous Scientific Research. arXiv preprint arXiv:2606.07591, 2026. URL https://arxiv.org/abs/2606.07591. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. arXiv preprint arXiv:2504.08066, 2025. URL https://arxiv.org/abs/ 2504.08066. Ruofeng Yang, Yongcan Li, and Shuai Li. ARIS: Autonomous Research via Adversarial Multi- Agent Collaboration. arXiv preprint arXiv:2605.03042, 2026. URL https://arxiv.org/ abs/2605.03042. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Proceedings of the 41st International Conference on Machine Learning, volume 235, p. 62743–62767. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhuge24a.html. 11 Systemind Research APPENDIX CONTENTS A Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A.1 Coordination records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A.2 Host bridges and termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B Experimental Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.1 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.2 Budget calibration and accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.3 Cost accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 B.4 Evaluation protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 C Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 C.1 Token usage and estimated cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 C.2 DiscoveryBench task-level scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 C.3 ResearchClawBench criterion-level scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 C.4 ResearchClawBench criterion summaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D Detailed Case Studies of RAC Coordination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D.1 ARK on Life-001: direct corrective handoffs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D.2 ARK on Physics-002: repair followed by cross-capability checking . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 D.3 EvoScientist on Math-001: explicit delivery verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 D.4 Agent Laboratory on Math-003: greater task relevance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 D.5 EvoScientist on Math-003: local contracts, unfinished objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 E Condensed All-Hop Trajectories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 E.1Physics-002 / ARK / R3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 E.2Life-001 / ARK / R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 F Generative AI Use Disclosure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 12 Systemind Research AIMPLEMENTATION DETAILS This appendix specifies the records that RAC writes during a run and the boundary between RAC and each host. All conditions share the same RAC package and condition configuration; only the host bridge differs across hosts. A.1COORDINATION RECORDS Table 5 lists the records that document coordination decisions. Requests and results are typed work.request and work.result events exchanged through the common communication channel. R3 adds contracts and work.verification records. Every verdict is retained and passed to the next selected agent; no verdict discards the workspace or alters the host transition. Table 5: Coordination records written by RAC. Checkpoints and event records document coordina- tion decisions; external benchmark evaluation assesses final outputs separately. RecordContentsUse CheckpointRun and hop identifiers, objective, native state, artifacts, unresolved problems, remaining budget, and available capabilities. State from which R2–R3 select the next authorized capability. work.requestStructured request to the selected agent.Hands off work through the communication channel. work.resultReturned result with artifacts and usage records.Returns work and charges usage to the run. Contract (R3)Objective, readable and writable artifacts, and required output. Scopes the selected agent’s task. work.verification (R3) Verdict (supported, refuted, or inconclusive), checks, reasons, and feedback status. Feedback for the next selected agent’s prompt. ProvenanceCommunication backend and non-secret room identifier; never invite tokens. Attributes each trace to its channel. A.2HOST BRIDGES AND TERMINATION Each host bridge serializes native state, invokes an existing capability, and returns artifacts and usage records through a shared interface. Bridges contain no host-specific runtime selection rules, verifica- tion criteria, or benchmark hints, so each host keeps its agents, tools, and permissions. Verification feedback enters the next selected agent’s prompt rather than directly triggering retries, rerouting, rollback, or stopping. Runs terminate on budget exhaustion, hard hop limits, terminal provider errors, or native host completion. Communication state is isolated between runs, and cross-run memory and evolution stores are reset before each run. BEXPERIMENTAL DETAILS This appendix gives the models, budget calibration, cost accounting, and evaluation protocol used for all reported comparisons. B.1MODELS We use DeepSeek V4 Pro as the execution model for the AI-scientist hosts and GPT-5.5 as the external ResearchClawBench judge. External judging is separate from the in-loop verifier and does not control agent selection or host transitions. B.2BUDGET CALIBRATION AND ACCOUNTING Hop ceilings are tailored to each host using its N0 hop count as the calibration reference. They are not a common cross-host constant: a native phase may itself contain many model or tool calls. 13 Systemind Research Configured limits and realized usage must therefore be distinguished. Token and call accounting includes coordination and verification, while external judging is excluded. Limits checked after an agent returns can be exceeded by that completed call; the excess is retained in usage totals, and subsequent work is stopped. The archived configurations contain multiple ceiling settings, and the consolidated score table alone does not identify every final configuration. We therefore do not infer a single numerical budget vector for an entire host or equate the standardized USD estimates with an enforced provider-spend limit. Budget matching describes the intended comparison design, not an independently verified equality of every archived resource ceiling. The results should be read as exploratory comparisons rather than exact compute-equated effects. B.3COST ACCOUNTING We estimate inference costs from recorded input and output token usage using a common DeepSeek V4 Pro tariff. We apply the official peak rates of US$1.32 per million non-cached input tokens and US$3.96 per million output tokens, as listed in the DeepSeek API pricing documentation on 23 September 2026. For each episode, the estimated cost is (1.32T in + 3.96T out )/10 6 USD. Here, T in and T out are its recorded input and output token counts. We apply no cache-hit or off-peak discounts. Table entries average episode estimates before rounding. These are standardized inference-cost estimates, not Azure invoices, and exclude external GPT-5.5 judging charges. B.4EVALUATION PROTOCOL The primary comparison is R3 versus N0 under matched budgets. Condition means equally weight final task–host–condition scores. For each host and each of R1–R3, we subtract the N0 score on the same task and average these differences over the same evaluated task set within that host. Figure 3 reports these descriptive paired means. The bars do not measure variability across independent runs. Adjacent conditions are compared on the same tasks, but R2–R3 does not separate contracts from verification. CADDITIONAL RESULTS This appendix reports resource usage, task-level DiscoveryBench scores, and criterion-level Re- searchClawBench scores that support the aggregates in the main text. C.1TOKEN USAGE AND ESTIMATED COST Table 6 summarizes mean input and output tokens and standardized inference costs for the evaluated tasks. Agent Laboratory has its lowest input-token and cost means in R2, but its lowest output-token mean in R1. EvoScientist has its lowest token and cost means in N0. ARK has its lowest input- token, output-token, and cost means in R2. These are within-host resource summaries; the hosts use different task sets. C.2DISCOVERYBENCH TASK-LEVEL SCORES Table 7 maps the row labels in Figure 4(b) to their task–host pairs and reports absolute final scores on the 0–100 scale. Pair 1 and Pair 2 share the same benchmark task but use different hosts; they are averaged before computing the task-level mean in panel (a). DiscoveryBench evaluates generated hypotheses against target hypotheses using variable, relation, and context matching. We use its recorded final_score, originally on a 0–1 scale, multiplied by 100. Means and paired differences are computed before rounding. Evaluation uses GPT-5.5 through Azure Chat Completions, separately from the research agents. This exploratory task subset was not randomly sampled and does not support a claim of representative benchmark coverage. 14 Systemind Research Table 6: Mean recorded token usage and estimated inference cost. Input tokens are in millions (M); output tokens are in thousands (K). n usage /n score gives the number of task records contributing to resource and score means. Costs use the uniform tariff in Appendix B.3 and exclude external judging. Hosts cover different tasks, and resource means for Agent Laboratory R3 and EvoScientist R1 have smaller denominators than their score means. HostCondition n usage /n score Input (M)Output (K)Est. cost (USD) Agent LaboratoryN010/104.26369.427.09 Agent LaboratoryR110/102.90269.914.89 Agent LaboratoryR210/102.36308.184.33 Agent LaboratoryR39/102.56279.244.48 EvoScientistN010/1011.4970.3415.44 EvoScientistR18/1018.31126.3724.67 EvoScientistR210/1037.74146.9750.40 EvoScientistR310/1038.58145.3051.50 ARKN05/548.851,219.6969.31 ARKR15/563.061,425.7688.88 ARKR25/545.471,113.5864.42 ARKR35/568.761,496.8596.69 Table 7: DiscoveryBench scores and row-label mapping for Figure 4(b). IDTaskHostN0R1R2R3 Pair 1nls_ses / metadata_0Agent Laboratory0.0013.3322.22100.00 Pair 2nls_ses / metadata_0EvoScientist12.5044.4444.440.00 Pair 3nls_incarceration / metadata_2Agent Laboratory0.0020.000.0066.67 Pair 4meta_regression / metadata_0Agent Laboratory9.3812.0025.000.00 Pair 5archaeology / metadata_13Agent Laboratory0.0050.0028.5728.57 Pair 6requirements_engineering_ for_ML_enabled_systems / metadata_0 Agent Laboratory0.000.0019.050.00 C.3RESEARCHCLAWBENCH CRITERION-LEVEL SCORES The table reports unweighted criterion scores and their weighted final scores, consistent with the main analysis. Criterion descriptions are concise summaries of the supplied rubric excerpts, not the complete benchmark rubric. Table 8: Criterion-level scores and final scores, arranged in two columns. AL denotes Agent Laboratory; Evo denotes EvoScientist. Weights follow criterion order in each task header. A dash denotes an unused criterion. Neuroscience-000 Weights: 20%, 20%, 20%, 20%, 20%. HostCond.C1C2C3C4C5Final ALN012500289.00 ALR125300128.00 ALR25000123.40 ALR3240002810.40 EvoN024000188.40 EvoR12800056.60 EvoR218000126.00 EvoR32822001813.60 ARKN0243501810.00 ARKR18828127.60 ARKR222800128.40 ARKR32212522813.80 Energy-001 Weights: 10%, 30%, 20%, 20%, 20%. HostCond.C1C2C3C4C5Final ALN020812259.20 ALR1158322810.50 ALR22130212512.20 ALR33000285.90 EvoN02425832817.70 EvoR1351212183419.90 EvoR282837182825.80 EvoR3852218183128.50 ARKN0453818183129.30 ARKR135353282827.60 ARKR2254234222831.90 ARKR3254218222427.90 Continued on the next page 15 Systemind Research Table 8 (continued) Life-001 Weights: 35%, 20%, 15%, 30%. HostCond.C1C2C3C4C5Final ALN02002–1.30 ALR130126–4.65 ALR2123121–6.90 ALR322120–2.90 EvoN01824340–16.20 EvoR11812320–13.50 EvoR2818414–13.75 EvoR31818280–14.10 ARKN088284–9.80 ARKR1812280–9.40 ARKR21812322–14.10 ARKR3128310–10.45 Neuroscience-002 Weights: 15%, 25%, 25%, 20%, 15%. HostCond.C1C2C3C4C5Final ALN02120003.30 ALR101201205.40 ALR201201225.70 ALR30120835.05 EvoN001201887.80 EvoR18120204.60 EvoR201401888.30 EvoR301201837.05 ARKN001201286.60 ARKR121201205.70 ARKR201201225.70 ARKR31120003.15 Math-000 Weights: 15%, 15%, 70%. HostCond.C1C2C3C4C5Final ALN045328–17.15 ALR1323812–18.90 ALR2443615–22.50 ALR3343212–18.30 EvoN0242712–16.05 EvoR13512–9.60 EvoR2243618–21.60 EvoR3283718–22.35 Math-001 Weights: 40%, 30%, 30%. HostCond.C1C2C3C4C5Final ALN00829–11.10 ALR104631–23.10 ALR283942–27.50 ALR3233831–29.90 EvoN062226–16.80 EvoR1353428–32.60 EvoR2324533–36.20 EvoR3354336–37.70 Math-002 Weights: 40%, 30%, 30%. HostCond.C1C2C3C4C5Final ALN0000–0.00 ALR1353–3.60 ALR212012–8.40 ALR31200–4.80 EvoN07818–10.60 EvoR1565–5.30 EvoR214315–11.00 EvoR3522–3.20 Math-003 Weights: 40%, 35%, 25%. HostCond.C1C2C3C4C5Final ALN0000–0.00 ALR112150–10.05 ALR26120–6.60 ALR312120–9.00 EvoN03193–16.30 EvoR12130–9.45 EvoR215150–11.25 EvoR31853–9.70 Physics-002 Weights: 20%, 20%, 15%, 20%, 25%. HostCond.C1C2C3C4C5Final ALN0000000.00 ALR151221226.60 ALR218323283424.55 ALR32321504.30 EvoN0324528384839.20 EvoR168528510.40 EvoR2323624327843.10 EvoR31225882516.45 ARKN0283524123627.60 ARKR1384234184836.70 ARKR2313222284232.00 ARKR3324228184834.60 Information-003 Weights: 15%, 20%, 30%, 20%, 15%. HostCond.C1C2C3C4C5Final ALN0880403.60 ALR11580805.45 ALR2806003.00 ALR312612808.20 EvoN0822128010.80 EvoR118612608.70 EvoR212223608.30 EvoR314312306.90 16 Systemind Research C.4RESEARCHCLAWBENCH CRITERION SUMMARIES The descriptions below summarize what each criterion assesses. They describe benchmark targets, not findings established by the evaluated agents. Neuroscience-000. C1: attack-classifier precision–recall performance across conditions. C2: cross-laboratory SHAP comparisons. C3: male–female SHAP comparisons. C4: RI–CSDS en- vironment SHAP comparisons. C5: permutation-based feature importance. Energy-001. C1: construction of the Scottish wind-curtailment power-system model. C2: con- strained versus unconstrained optimal power flow. C3: dispatched and curtailed wind over time. C4: loading of the Scotland–England boundary link. C5: the open-source PyPSA-GB representa- tion of the British power system. Life-001. C1: immune-response distributions for optimized vaccines. C2: coverage across re- sponse thresholds. C3: optimization runtime scaling. C4: recall of experimentally validated neoanti- gens against ranking-based comparators. Neuroscience-002. C1: FlyTracing dataset construction. C2: fusion of morphological and image features. C3: EmbedNet weighting ablations. C4: robustness to imaging artifacts. C5: visualization of neuron-segment embeddings. Math-000. C1: pseudo-depth estimation from image geometry. C2: depth cascade matching using depth-partitioned association. C3: SparseTrack tracking performance on MOT17, MOT20, and DanceTrack. Math-001. C1: accelerated convergence on a high-dimensional Lasso problem. C2: convergence trajectories and stability-related evidence. C3: robustness on an ill-conditioned regression problem. Math-002. C1: multi-agent path-finding performance in high-density settings. C2: remaining collisions and solution cost. C3: zero-shot generalization across map structures. Math-003. C1: geometry-proof performance on IMO-AG-30. C2: synthetic training-data scale. C3: traceback-based identification of an unused premise and the resulting theorem generalization. Physics-002. C1: fidelity scaling with qubit count at fixed depth. C2: fidelity scaling with depth for 40 qubits. C3: depth-dependent fidelity estimates for 56 qubits. C4: gate-counting predictions and uncertainty. C5: counts-weighted linear XEB reproduction on the 40-qubit verification subset. Information-003. C1: evidence connecting feature entanglement with detection failures. C2: the DIDS-MFL disentanglement and graph-diffusion framework. C3: few-shot intrusion-detection per- formance across benchmarks. C4: component ablations and visualizations. C5: comparison with large language models. DDETAILED CASE STUDIES OF RAC COORDINATION We examine five trajectories to illustrate how RAC changes the work performed between agent boundaries. The cases cover direct reassignment of corrective work, artifact-grounded verification without a change of agent order, improved task relevance under communication-enabled execution, and a failure to translate local completion into global progress. We emphasize the concrete interven- tion visible in each trace: which defect was identified, which capability acted next, what changed on disk, and what the external judge subsequently assessed. Evidence and interpretation. These are selected qualitative cases, not additional controlled ab- lations. The ARK records are exported hop summaries paired with artifacts and external scoring receipts. The Math cases are identified by individual episode records. Numerical outcomes refer to the final scores reported in Table 4; older receipts are not used to decompose those scores. Their 17 Systemind Research differences describe the N0 and RAC executions, rather than isolated effects of a coordination com- ponent. All hop indices are zero-based. Quoted passages below reproduce short excerpts, with Markdown styling removed where needed; the accompanying descriptions are our reconstruction from the records, not additional agent dialogue. We distinguish the host reviewer, which critiques research outputs, the RAC verifier, which checks work-contract evidence, and the external judge, which applies the benchmark criteria. We have not independently rerun the scientific computations. D.1ARK ON LIFE-001: DIRECT CORRECTIVE HANDOFFS Outcome and coordination change. The useful change in this R2 trajectory was a shorter path from a concrete review finding to a revised artifact. Whereas the recorded native successor after review was planner, RAC repeatedly selected writer directly. This capability did more than edit prose: it reported modifying analysis code, regenerating comparisons, and checking claims against the underlying data. Thus, the observed benefit was not the introduction of review—ARK already had a native review loop—but a change in who acted immediately on its findings. Trace: repairing the comparison being reported. 1. Hop 3, reviewer: identify a method–result mismatch. The reviewer reported that the headline coverage curve used the greedy baseline rather than the proposed Min- Sum objective. It also identified missing threshold sensitivity and unsupported gen- eralization from a single simulated tumor. The recorded successor was planner. “E1 coverage curve uses greedy, not MinSum” 2. Hop 4, writer: act directly on the implicated artifacts. The actual next capability was writer. It reported adding compute_greedy_minsum_curve() to the experiment code and regenerating a figure with both curves, rather than merely re- naming the existing result. The export recorded 15 scientific-artifact changes at this hop. “regenerated figure with both curves; rewrote E1 text to position greedy as baseline and MinSum as proposed method” 3. Hop 5, reviewer: check the repair. The reviewer explicitly recognized the added MinSum curve, threshold analysis, corrected coverage numbers, and narrower claims. It then requested that replicate variability be made visible and that clinical-feasibility language be qualified. 4. Hop 6, writer: complete the next local correction. RAC again selected the writer directly. The report acquired the replicate-level range of mean response probabili- ties, and claims of clinical feasibility were replaced with claims about computational tractability at the tested scale. The final report corroborates the central repair: it contains separate greedy and MinSum cov- erage curves, with MinSum coverage of 97.9% at budget four and 99.2% at budget ten. It also reports coverage at multiple response thresholds rather than relying entirely on a single cutoff. These persisted changes connect the selected capability to the artifact that the reviewer had challenged. Trace: replacing an unsupported explanation with a data check. A later handoff shows that the same pattern could improve interpretation as well as presen- tation. At hop 11, the reviewer challenged the explanation for low-response cells. At hop 12, the writer inspected per-peptide scores and reported that five of the eight cells had near- zero scores for every candidate, while the other three had their highest scores among already selected elements. Its summary stated: 18 Systemind Research “the claim was not just unsupported” . . . “contradicted by the data” The revision replaced speculation about excluded vaccine elements with a description grounded in those observations. This is a local evidence–action closure: feedback specified a claim, the next capability checked the relevant data, and the explanation changed. Relation to the reported outcome. Table 4 reports final scores of 9.80 for N0 and 14.10 for R2, a difference of 4.30 points. The trace supports the local repairs described above, not a quantitative attribution of the final score gain to individual handoffs. The case nevertheless provides a concrete positive example of RAC changing the native successor and closing local defects. Its boundary is equally specific: efficacy experiments still used a single simulated tumor with 100 cells per replicate, rather than the required seven-patient, 10,000-cell analysis. The run also continued through repeated revisions to the 46-hop limit. Direct corrective handoffs improved a partially aligned study without completing the benchmark’s broader scientific requirements. Delivered-paper evidence: the correction survives into the manuscript. The final paper explic- itly separates the proposed objective from the baseline and qualifies its comparative result: “Under the proposed MinSum objective, coverage rises from 53.2% [50.1%, 56.3%] at k=1 to 94.0% [92.3%, 95.4%] at k=3, reaches 97.9% [96.8%, 98.7%] at k=4, and plateaus at 99.2% [98.4%, 99.7%] by k=7.” “The MinSum point-estimate curve is higher than greedy at every budget k ≥ 3 (though CI overlap at k=3 and the non-significant E5 comparison at p=0.0625 caution against claims of strict superiority).” This is more informative than a generic claim that a figure was fixed: the delivered text identifies which curve represents MinSum and distinguishes point-estimate differences from statistical supe- riority. The paper also bounds the experiment: “Vaccine efficacy experiments (E1–E3, E5) use a single simulated tumor model with M = 12 mutations and N = 100 cells per replicate. Runtime scaling (E4) spans 7 independently parameterized instances to provide cross-instance evidence for computational cost.” The distinction prevents seven runtime instances from being read as seven independent efficacy stud- ies. These passages support the positive interpretation of the trace: corrective handoffs improved the fidelity and qualification of the delivered comparison, even though they did not supply the missing patient-scale evaluation. Residual inconsistency: repaired delivery does not certify the objective. The same paper de- fines a minimization objective that decomposes as L min (S) = 1 N N X i=1 log(1− P i (S)) = X j∈S w j , w j = 1 N N X i=1 log(1− p ij ), but its accompanying instruction reads: “The optimal k-element solution therefore reduces to ranking elements by w j (descending) and selecting the top-k.” For the stated minimization of a sum of fixed weights, the selected weights should instead be the smallest ones. This establishes a contradiction in the paper’s description, not that the implementation necessarily used the wrong order. The paper also acknowledges that this is “structurally the same ranking paradigm the Introduction critiques”. Consequently, the corrected comparison is evidence of improved reporting, not evidence of a newly established combinatorial optimization paradigm. RAC’s observed success here is local: it closes particular reviewer-identified defects while leaving a mathematical description that warrants further checking. 19 Systemind Research D.2ARK ON PHYSICS-002: REPAIR FOLLOWED BY CROSS-CAPABILITY CHECKING Outcome and coordination change. The selected R3 snapshot scored 34.6, compared with 27.6 for N0. RAC departed from the recorded native successor at several boundaries and assigned con- crete defects across writer, coder, and experimenter capabilities. The resulting trajectory is useful because it includes both a correction to existing evidence and a second check that caught an error introduced while extending a figure. Trace: reconciling reported values with computed results. 1. Hop 7, reviewer: localize the discrepancy. The reviewer identified XEB table val- ues that disagreed with the experiment manifest, a missing N56 result table, undis- closed differences in shot counts, and incorrect significance statements. “d=10: paper reports 0.557 vs manifest 0.4502” The recorded successor was planner. 2. Hop 8, writer: revise before replanning. RAC selected writer instead. The writer added the N56 table, disclosed 100 shots per instance at depth 24 versus 20 at other depths, and corrected a caption that described a nonexistent second axis. 3. Hop 9, coder: inspect the partially repaired report. Rather than proceeding di- rectly to the writer’s proposed reviewer successor, execution moved to coder. It verified the seven XEB depth values and found a residual discrepancy between the discussion and conclusion: “The Discussion section already correctly stated ‘23σ’ — the Conclusion was the only place with the stale ‘26σ’.” 4. Hop 10, experimenter: check against numerical artifacts. The experimenter checked the XEB values against the instance records and reported a maximum dis- crepancy of 0.00028, within the stated rounding tolerance. It also corrected Transport confidence intervals and added the corresponding N56 table. The final report contains the corrected depth-10 XEB value of 0.450. The significance state- ments in these excerpts remain agent calculations, not independent validation by our audit; the directly checked fact is that the handoff led to reconciliation of the persisted report and its cited outputs. Trace: detecting a bug in a newly added figure. At hop 42, the coder handled a figure-specific issue and expanded the summary figure from two panels to three, adding the N56 MB depth scan. At hop 43, the experimenter checked that addition and identified a denominator error in the pooled confidence intervals: “The panel (c) code hardcoded n_total = n * 20” The depth-24 data used 100 shots per instance, not 20. The experimenter reported changing the interval from [0.1014, 0.1428] to [0.1119, 0.1302]. The final report both references panel (c) and records the rounded corrected interval [0.112, 0.130], together with 100 shots per instance and 5,000 total trials. This sequence links a scoped figure task to implementation, a distinct checking step, and a corrected delivered result. RAC’s contribution and its scope. The archived provenance for this ARK R3 case identifies an expanded-budget continuation. We therefore use its trace as qualitative evidence of local repairs, not as a matched-budget estimate of their benefit. Unlike a fixed traversal of role names, the observed routing followed the type of outstanding work: document reconciliation, code and figure repair, and numerical checking. Typed issue identifiers in the summaries made the requested changes concrete. 20 Systemind Research The cross-capability check was productive because it found a specific new bug, not merely because a second agent declared the work complete. The export does not include the underlying RAC verifier ledger for these ARK hops, so this example establishes a corrective handoff pattern rather than independently verified contract compliance at every boundary. N0’s report had introduced an additional “unbiased XEB” rescaling, whereas R3 retained values on the target scale. This between-run method difference must not be mistaken for a logged repair of N0’s code by R3. The within-R3 evidence above instead shows how concrete inconsistencies were repaired and checked before delivery; it does not decompose the 7.0-point final-score difference. Gate-counting uncertainty bands and the required mirror-circuit inference remained incomplete. Delivered-paper evidence: a correct patch and an obsolete claim coexist. The final manuscript provides unusually direct evidence of both the value and the limits of local repair. Its comparison paragraph retains the following sentence: “The largest within-CI spread at any single depth is at d=12 (F XEB = 0.457 vs. F MB = 0.476, CIs [0.376,0.538] and [0.445,0.507] respectively).” Yet the same paragraph subsequently states: “The largest between-method differences occur at d=10 (F XEB = 0.450 vs. F MB = 0.518, ∆ = 0.068) and d=16 (F XEB = 0.368 vs. F MB = 0.305, ∆ = 0.063).” The table supports the latter values: the depth-12 difference is only 0.019, compared with 0.068 and 0.063 at depths 10 and 16. The corresponding intervals also overlap at those depths, so the “within- CI” qualification does not resolve the obsolete superlative. Both sentences occur in the original main.tex, not only in the Markdown export. A corrected statement was delivered without fully reconciling the surrounding text. The figure repair is more cleanly reflected in the final artifact. The paper reports the depth-24 pooled interval as [0.112, 0.130], and its table lists 100 shots per instance and 5,000 total trials. These are the quantities implicated in the hop-43 denominator correction. Thus, the paper corroborates a successful numerical patch while exposing a separate synchronization failure. The mechanism-level lesson is not that cross-capability checking failed altogether, but that checking a named value or figure can succeed without checking every dependent statement in the manuscript. The higher score and the remaining contradiction can therefore coexist. D.3EVOSCIENTIST ON MATH-001: EXPLICIT DELIVERY VERIFICATION WITH UNCHANGED AGENT ORDER Outcome and coordination change. The selected R3 episode scored 37.7, compared with 16.8 for N0. Its executed sequence remained planner, research, code, debug, data analysis, and writing. This case therefore illustrates a different RAC contribution: a work contract distinguished an agent’s nonempty response from an actual implementation, even though the next agent retained the native position in the workflow. Trace: a nonempty response fails the delivery contract. 1. Hop 2, code contract. The decision required changed code/result artifacts and nonempty output. These requirements were recorded separately, rather than treating a successful return as evidence of implementation. 2. Hop 2, code invocation. The agent announced that it was about to delegate, but did not deliver the required implementation: “Good. Both /code/ and /outputs/ are empty — we’re starting from scratch.” 3. Hop 2, RAC verifier. The actual evaluation event recorded nonempty_output: passed, artifact_changed: failed, and verdict refuted: 21 Systemind Research “verification advisory (refuted): one or more contract evidence checks failed” 4. Hop 3, debug invocation. The next capability delivered the missing implementation rather than only commenting on it. Its output began: “Implementation complete.” . . . “Code & Results Delivered” The artifact record includes solver modules, a Lasso experiment, convergence plots, and method-comparison results. The subsequent verifier event recorded supported and listed the changed code and output files. Why this is a useful positive contract example. RAC added an explicit, machine-recorded dis- tinction between intention to perform work and evidence that the work was delivered. That distinc- tion exposed a gap that a nonempty-output check alone would have missed. The following native debug capability then filled the gap, after which analysis and writing produced a substantive re- port. Thus, a visibly unchanged agent order did not imply unchanged coordination semantics: the boundary now carried an auditable delivery verdict. The scope of the inference matters. The code invocation had already proposed debug as its succes- sor; the trace does not show a verifier-induced reroute, forced retry, or rollback. Nor does this single trajectory isolate whether debug would have repaired the missing implementation without the ad- visory. The supported observation is the sequence of explicit detection followed by artifact-backed recovery, which is consistent with a beneficial role for contracts and verification without changing the agent roster or order. Relation to the result. The N0 receipt describes a VOS method that underperformed its baseline; the R3 submission provided a more complete implementation, comparator analysis, convergence evidence, and a 460-line report. These differences explain why the external judge had more relevant evidence to assess, while the trace identifies a concrete delivery gap successfully closed along the way. The result was still partial: the run exceeded its input-token budget after writing, before a native review invocation, and did not fully establish the requested convergence target. The 20.9- point difference is therefore an outcome associated with this recovered trajectory, not a measured causal effect of the verifier alone. Delivered-paper evidence: a working comparator, not a new winning algorithm. The gener- ated paper explicitly characterizes the implemented method as matching an established baseline: “VOS matches FISTA: At r = 3, the VOS discretization is algorithmically equivalent to FISTA with gradient restart. The identical convergence trajectories (92 iterations, rates within 0.01 of each other) confirm that the VOS framework correctly reproduces Nesterov acceleration.” Its comparison table gives both FISTA and VOS 92 iterations, with fitted rates of 6.60 and 6.61, respectively. This grounds the positive case more precisely: the recovered implementation supplies a functioning accelerated baseline and relevant comparison evidence. It does not demonstrate a distinct algorithm outperforming FISTA. That narrower interpretation is consistent with the delivery contract: the missing implementation was supplied, but the verifier did not establish algorithmic novelty. Residual numerical and theoretical overreach. Two passages show why artifact-backed delivery must be distinguished from validation of the scientific conclusions. First, the paper states: “All three methods converge to essentially the same point (objective within 3× 10 −9 of each other).” The comparison table instead lists final suboptimalities of 1.40× 10 −5 for FISTA, 1.40× 10 −6 for ADMM, and 1.37 × 10 −5 for VOS. If these use a common reference optimum, the FISTA– ADMM objective difference is approximately 1.26× 10 −5 , inconsistent with the quoted tolerance. If different references were used, the paper would need to explain how the cross-method claim was 22 Systemind Research obtained. We flag an unreconciled reporting discrepancy rather than claim to have recomputed the objectives. Second, the paper’s Lyapunov diagnostic uses changing reference values: “where f ⋆ and x ⋆ are dynamically updated to the best observed values.” “Non-monotonic steps: 13 out of 92 (14.1%)” The discussion acknowledges that “the dynamic update of f ⋆ and x ⋆ introduces reference-point shifts”, but the conclusion nevertheless describes the experiment as “exhibiting the predicted Lya- punov monotonicity (85.9% monotonic steps, p = 2.87 decay exponent)”. A moving-reference, partly non-monotone diagnostic is not a direct verification of a fixed-reference monotonicity the- orem. Nor do these discrete observations by themselves refute a continuous-time theorem. The appropriate claim is evidence of empirical decay under the implemented diagnostic. This separates RAC’s observable contribution—detecting and closing a delivery gap—from stronger mathematical claims that its artifact checks did not validate. D.4AGENT LABORATORY ON MATH-003: GREATER TASK RELEVANCE UNDER COMMUNICATION-ENABLED EXECUTION Outcome and coordination condition. The selected R1 episode scored 10.05, compared with 0 for N0. N0 studied an unrelated synthetic-transformation setup. R1 instead developed a geometry- proof experiment with self-play and comparison baselines. Because R1 preserved the native successor and did not enable work-contract verification, this case examines improvement under communication-enabled execution rather than dynamic selection or verifier-guided repair. Trace: carrying a task-relevant plan into experiments. 1. Runtime decisions. The recorded decision reason was: “native fixed successor with SharedNet communication” The workflow retained literature review, plan formulation, data preparation, exper- iments, interpretation, and report writing. Evaluation events explicitly stated that verification was disabled. 2. Hop 1, plan formulation. The planning dialogue framed proof steps as actions in a geometry-solving system, combining sequence generation, search, and self-play. Its stated goal was: “autonomously proving olympiad-level geometry theorems without human demonstra- tions” 3. Hop 3, experiments. Execution attempted the task-relevant experimental plan, en- countering several 600-second code-execution timeouts and native repair attempts before producing results. These repairs belong to the host’s internal execution ma- chinery, not to an R3 verifier. 4. Hop 4, interpretation. The next stage discussed the actual failures of that experiment rather than substituting a different research question: “0/10 on the random walk baseline, 0/10 on the supervised imitation learning baseline, and 0/10 on our SPNS-EPS self-play approach” 5. Hop 5, writing. A report of the geometry experiment was delivered. Input-token exhaustion prevented the subsequent refinement stage. The communication-related interpretation. The positive change was continuity of a relevant research objective across planning, implementation, and interpretation. Under the communication- enabled condition, the host produced a connected, albeit unsuccessful, geometry experiment instead 23 Systemind Research of an off-task study. This motivates communication as a way of improving what successive agents coordinate around without changing their order. However, the recorded host dialogue is not itself proof of SharedNet message consumption. The available evidence does not link a particular Shared- Net message to a downstream decision, and R1 differs from N0 in its integration context as well as communication availability. We therefore treat this as a communication-consistent example, not a component-level causal demonstration. The score increase should also be read as improved relevance, not improved theorem-proving per- formance: every reported method solved 0 of the ten evaluated problems, and the study did not reproduce the full target benchmark. Planning consumed approximately 70.5% of the episode’s recorded input tokens. The trajectory shows the value of preserving a meaningful task across hand- offs, while also showing that relevance alone does not ensure successful execution or efficient use of the budget. Delivered-paper evidence: relevant and candid negative results. The R1 manuscript reports the failure directly rather than manufacturing successful proof results: “Our experiments reveal a stark negative result: all three approaches achieve exactly 0/10 (0.0%) proof success on the IMO test set. The SPNS-EPS self-play agent achieves an immediate +1.000 reward at episode 0 and maintains this reward with identically zero policy loss and zero value loss across all 120 training episodes.” It positions its contribution accordingly: “The primary contribution of this work is not a novel method that achieves state-of-the-art performance, but rather an empirical characterization of the failure modes” The latter is an excerpt from a longer sentence. Together, these passages reinforce the positive but limited interpretation: R1 delivered an on-topic experiment and openly reported unsuccessful outcomes. The relevance gain does not depend on accepting the manuscript’s explanation of those outcomes. Code evidence: the failure explanation misses implementation-level causes. The paper inter- prets a zero supervised loss as successful memorization: “A loss of exactly zero indicates that the network assigns probability 1.0 to the target action index for every training example, achieving perfect memorization of the training set.” However, the delivered training loop constructs its legal-action mask from the target action alone. The following contiguous excerpt is from run_experiments.py: mask, _ = build_legal_mask([target_action]) target_idx = action_to_idx(target_action) mask[0, target_idx] = 1.0 _, logits = sup_net([gdata], goal_t, mask) loss = F.cross_entropy(logits, torch.tensor([target_idx])) The forward pass masks other logits, and the loss is cross-entropy against that same target. With only the target class left available, probability one and a near-zero loss follow from the mask; they are not evidence that the network learned to distinguish competing actions. This static implementation evidence supports a more specific explanation than memorization. Theimmediateself-playrewardhasasimilarlyconcretecause.In random_provable_problem(), the generator repeatedly applies a theorem and updates state = ns. It then sets the goal to the conclusion of the last applied theorem and returns that already-expanded state, rather than the original premises. The decisive terminal statements are: state.goal = goal return state 24 Systemind Research The theorem application has already inserted its conclusion into the state’s relations; the goal check tests membership in those relations. When there is no recorded application, the alternative branch chooses an existing relation as the goal. Thus, the returned training instance already contains its goal. This explains why immediate success can occur without a learned proof policy. The paper instead discusses forward chaining as producing goals reachable “in zero or one steps”, framing the problem primarily as an overly easy curriculum. These are static code findings, not results of a rerun or proof that fixing the two issues would solve the benchmark. They identify a missed distinction between a genuinely informative negative experiment and an implementation artifact. R1 improved task continuity and reporting relevance, but did not provide the semantic checking needed to catch these defects before they became an explanatory narrative. Since verification was disabled in R1, this is not a failed R3 verifier verdict. D.5EVOSCIENTIST ON MATH-003: LOCAL CONTRACTS CAN LEAVE THE GLOBAL OBJECTIVE UNFINISHED Outcome and failure boundary. The selected R3 episode scored 9.7, below the N0 score of 16.3. This is not a case in which RAC broke an otherwise successful prover: N0 also reported 0/30 solved problems. The useful contrast is that R3 developed and validated a narrower intermediate artifact, then proceeded to analyze and write about it while the planned system remained incomplete. Trace: stage completion substitutes for system completion. 1. Hop 0, planner. The plan specified six implementation stages, including a neural policy/value component and Monte Carlo tree search for geometry proving. 2. Hop 2, code. The implementation stopped at the parser and deduction-engine foun- dation: “Stage 0 Complete: Parser & Deduction Engine” It reported 89 passing tests and successful parsing of all 30 problems. The contract’s executable evidence checks required changed code/result artifacts and nonempty out- put; the RAC verifier returned supported. 3. Hop 3, debug. Debugging fixed five reported bugs and increased the test count to 102. These were genuine local improvements to the implemented foundation. 4. Hop 4, data analysis. Instead of returning to the unimplemented neural and search components, execution moved to analysis of the existing artifact: relation counts, predicate distributions, and related diagnostics, with 11 generated figures. Its contract also received supported. 5. Hops 5–6, writing and host review. The writer produced an 861-line report centered on Stage 0. The host reviewer gave 8/10 after checking such properties as the passing tests and the reported 0/30 result. All seven RAC verification events in the episode were supported. What the negative case says about RAC. The verification results were compatible with their local evidence rules: files changed, outputs were nonempty, and the parser was tested. The weakness was the connection between those local checks and the remaining scientific obligations. The contract objective named autonomous geometry solving, but its executable criteria did not establish that the full planned solver had been implemented. Runtime selection continued along the native sequence, allowing analysis and writing to consolidate the partial system rather than return to the missing stages. This illustrates a failure mode in which locally supported work accumulates without closing the global objective. RAC made the intermediate work explicit and checkable, but did not prevent premature progression from a useful foundation to a final report. The episode ultimately exceeded its input-token limit; nevertheless, the narrowing to Stage 0 was already visible before termination. Budget exhaustion alone therefore does not explain the observed sequence. The case motivates connecting contracts and subsequent selection to unresolved end-to-end requirements, rather than 25 Systemind Research interpreting supported as scientific completion. It does not establish that contracts necessarily caused the narrowing or generally reduce performance. Delivered-paper evidence: the missing system becomes future work. The final manuscript makes the gap between local completion and the research objective explicit. It reports: “The most important empirical result: 0/30 problems are solvable by forward chaining alone.” Elsewhere, it describes its foundation in stronger terms: “Our implementation—parser, definition expander, rule compiler, and forward chainer—is a faithful re- production that handles all 30 IMO-AG-30 problems correctly. The 102-test suite provides confidence in the symbolic foundation.” Handling the input representation and passing local tests are compatible with solving none of the benchmark problems, but “handles ... correctly” should not be read as end-to-end proof success. Likewise, the observed 0/30 is a result for this implementation and configuration; it does not estab- lish a general impossibility result for forward chaining. The paper’s appeal to the upstream system’s performance does not validate the completeness of its own reproduction. Most revealingly, the manuscript’s “Next Steps (Stage 1–5)” section places the planned core com- ponents after the delivered study: “Synthetic Data Generation”, “GNN Architecture and Training”, “MCTS Integration”, “Self-Play Expert Iteration”, and “Evaluation”. The proposed MCTS success criterion is to solve at least three of the 30 problems within 10,000 simulations. This is a future target, not an achieved result. The paper therefore corroborates the trace more strongly than a low score alone: the planned learning-and-search system was explicitly deferred while the foundation was packaged as the final contribution. RAC’s local contracts did capture real work—parser repairs, tests, and diagnostic artifacts—but their supported verdicts did not keep the missing stages active as prerequisites for final delivery. The negative case is consequently about the granularity and coverage of coordination obligations, not the uselessness of the intermediate work. A polished account of an incomplete system can satisfy local delivery checks while losing external credit for the actual research objective. Cross-case synthesis. The two ARK trajectories illustrate altered corrective handoffs that pro- duced observable repairs. EvoScientist on Math-001 shows how delivery verification can distin- guish a missing implementation from a nonempty response without changing the next agent. Agent Laboratory on Math-003 shows greater task relevance under communication-enabled fixed-order execution, with weaker evidence for the communication mechanism itself. The final case bounds these benefits: artifact-level success must remain connected to the unfinished research objective. Together, the cases locate RAC’s potential value at the interfaces between agents—what evidence is handed over, what obligation is assigned, and what work follows—rather than in the number of agents invoked or the presence of a positive local verdict. 26 Systemind Research ECONDENSED ALL-HOP TRAJECTORIES We summarize all 92 exported hops from Physics-002 / ARK / R3 and Life-001 / ARK / R2, re- taining one row per hop and the chronological order. The tables condense reported work, corrective handoffs, and repeated checking; selected complete exported summaries follow. Phase headings are editorial groupings, not additional workflow stages. Reading the tables. Work descriptions paraphrase agent-reported actions; they do not indepen- dently validate scientific results. In the final column, recorded next is the successor specified by the host’s native workflow, whereas observed next is the agent selected by RAC and actually invoked next. Blue entries highlight departures from the native successor; the record alone does not es- tablish the rationale for each selection. Res., Exp., Wri., Rev., Plan., and Code denote researcher, experimenter, writer, reviewer, planner, and coder. A dash means no subsequent hop is exported. The host reviewer is distinct from the RAC verifier and the external benchmark judge. These are qualitative records, not isolated estimates of causal benefit or matched-budget efficiency. The boxes following each table preserve selected complete exported summaries. Internal ratings are not external benchmark scores. Markdown syntax is retained as text, with typography and emoji markers adapted for print. E.1PHYSICS-002 / ARK / R3 The trajectory alternated textual revisions with coder and experimenter checks, including a numeri- cal defect introduced in a later figure. The final benchmark score was 34.60; the archive stopped at the hard hop limit. Hops 3, 12, and 44 have no summary. Table 9: Physics-002: all 46 exported hops, summarized chronologically. Work and checks are agent-reported. Hop AgentReported work and outcomeRecorded→ observed next Initial research and first refinement (hops 0–6) 0ResearcherEstablished the RCS fidelity-estimation project context.Exp. → Exp. 1ExperimenterExecuted the initial five experiments and summarized results.Wri. → Wri. 2WriterWrote and compiled the initial manuscript.Rev. → Rev. 3ReviewerNo summary available; reviewer invocation retained.Rev. → Plan. 4PlannerPrepared an action plan from the available review state.Exp. → Exp. 5ExperimenterChecked the pipeline and extended experimental evidence.Wri. → Wri. 6WriterRevised the manuscript against the action plan.Rev. → Rev. Cross-agent numerical repairs (hops 7–15) 7ReviewerIdentified inconsistent XEB values and missing numerical evidence. Plan. → Wri. 8WriterRepaired tables, shot-count disclosure and figure caption.Rev. → Code 9CoderCross-checked XEB values and corrected stale significance wording from 26 to 23 standard errors. Wri. → Exp. 10ExperimenterChecked numerical artifacts, corrected Transport confidence intervals, and added the N56 Transport table. Wri. → Wri. 11WriterAdded discussion of XEB/MB divergence.Rev. → Rev. 12ReviewerNo summary available; reviewer invocation retained.Rev. → Plan. 13PlannerPrepared the next correction and verification plan.Exp. → Exp. 14ExperimenterRe-ran the pipeline and cross-checked six tables.Wri. → Wri. 15WriterHedged residual confirmation claims and checked presentation.Rev. → Rev. Continued on the next page 27 Systemind Research Table 9 (continued) Hop AgentReported work and outcomeRecorded→ observed next Disclosure and interpretation revisions (hops 16–28) 16ReviewerReviewed repaired evidence and remaining disclosure issues.Plan. → Wri. 17WriterAdded disclosure, significance method and result-file paths.Rev. → Plan. 18PlannerPlanned follow-up work after writer corrections.Exp. → Code 19CoderChecked prior presentation fixes and clarified the interpretation of the significance range. Wri. → Exp. 20ExperimenterChecked the N56 MB table against experiment outputs.Wri. → Wri. 21WriterChecked prior fixes and updated the manuscript.Rev. → Rev. 22ReviewerReviewed the paper and remaining minor issues.Plan. → Wri. 23WriterClarified Transport wording, novelty and significance values.Rev. → Plan. 24PlannerPrepared the next plan despite only line-level issues.Exp. → Exp. 25ExperimenterQuantified the Transport hit-rate distribution and applied fixes.Wri. → Wri. 26WriterChecked experimenter edits and resolved remaining plan items.Rev. → Rev. 27ReviewerReviewed resolved issues and remaining interpretation claims.Plan. → Wri. 28WriterSeparated monotonic from exponential decay and qualified language. Rev. → Plan. Interval methods and statistical claims (hops 29–38) 29PlannerRecorded an action plan and protocol-coverage audit.Exp. → Code 30CoderChanged plotted intervals to pooled Clopper–Pearson intervals and updated captions and evidence records. Wri. → Exp. 31ExperimenterRe-ran experiments and verified the implemented corrections.Wri. → Wri. 32WriterChecked all five issues as resolved on disk.Rev. → Rev. 33ReviewerFound a false within-one-standard-error agreement claim: the reported N = 40 discrepancy was 1.83σ. Plan. → Wri. 34WriterReplaced that claim with overlapping 95% confidence intervals and explicitly reported the 1.83σ discrepancy. Rev. → Plan. 35PlannerPlanned follow-up work with the main issue already corrected.Exp. → Code 36CoderChecked earlier figure, execution and framing repairs.Wri. → Exp. 37ExperimenterRegenerated page images and added a significance-metric caution. Wri. → Wri. 38WriterChecked the iteration-9 corrections in the manuscript.Rev. → Rev. Final checks, new figure, and residual repair (hops 39–45) 39ReviewerReviewed the manuscript and identified remaining consistency issues. Plan. → Wri. 40WriterHedged the abstract Transport claim and revised presentation.Rev. → Plan. 41PlannerReported zero outstanding issues; the next observed agent was still a coder. Exp. → Code 42CoderAdded an N56 depth-scan panel and qualified the Transport claim. Wri. → Exp. 43ExperimenterFound a hardcoded shot-count denominator in the new panel; reported replacing it with actual totals. Wri. → Wri. 44WriterNo summary available; the recorded error was qualification_request_limit. Wri. → Wri. 45WriterReported prior fixes verified and corrected full-width figure layout; no later hop is exported. Rev. → – 28 Systemind Research Hops 9–10 document consecutive coder and experimenter checks after the writer’s revision. Hop 43 reports a defect in a newly added figure. The exported summaries for hops 10 and 43 may be truncated upstream at 2,000 characters; their available text is reproduced without reconstructing the missing tail. Hop 9 · coder Task / reported work: Check the writer repairs and correct stale significance wording. Editorial heading inferred from the exported summary, not an assignment quote. Handoff: Recorded next: writer; observed next at hop 10: experimenter. The two fields differ. Artifact changes: Exported scientific_changes = 3. No file-level diff is provided for this hop. Recorded status: done = false; error = null. Complete available summary ## Summary I reviewed the state of ‘report/main.tex‘ against the reviewer’s four acceptance conditions. The previous writer agent had already addressed all major issues (M1-M5), but I discovered and fixed one residual error. ### Change Made ** ‘report/main.tex‘ -- Conclusion (§6), line 234 ** : Corrected the d=24 Nσ claim from ** 26σ → 23σ ** . The actual computed value from ‘outputs/n56_mb_transport_depth_scan_instances.jsonl‘ is 23.0σ (mean=0.1208, SE=0.005244). The Discussion section already correctly stated "23σ" -- the Conclusion was the only place with the stale "26σ". ### Verification Results | Acceptance Condition | Status | Evidence | |--|--|--| | (1) XEB table values reconciled | [check] | All 7 depths match JSONL data (d=8:0.632, d=10:0.450, d=12:0.457, d=14:0.397, d=16:0.368, d=18:0.266, d=20:0.302) | | (2) Nσ claims corrected | [check] | d=12: 26σ, d=24: 23σ (Discussion + Conclusion consistent) | | (3) Figure 1 caption fixed | [check] | "plotted on the same [0,1] axis for compactness" | | (4) N56 MB table added | [check] | Appendix table with 8 depths, CIs, shot counts | LaTeX compiles clean (10 pages, 0 errors). Hop 10 · experimenter Task / reported work: Cross-check numerical artifacts and correct confidence intervals. Editorial head- ing inferred from the exported summary, not an assignment quote. Handoff: Recorded next: writer; observed next at hop 11: writer. The two fields agree. Artifact changes: Exported scientific_changes = 6. No file-level diff is provided for this hop. Recorded status: done = false; error = null. Complete available summary ## Completed: Data Integrity Corrections for RCS Paper (Iteration 3 Review) ### Summary of Changes All reviewer-identified issues resolved. Paper now reconciles with experiment data at every claim. ### Changes Made to ‘report/main.tex‘ | Issue | Fix | |--|--| | ** M1 ** : XEB table values | All 7 depths verified within ±0.0005 of JSONL data -- already correct | 29 Systemind Research | ** M2 ** : Nσ claims | d=12: 26σ, d=24: 23σ (both Discussion and Conclusion -- already correct from previous iteration) | | ** M3 ** : Figure caption | "same $[0,1]$ axis for compactness" -- already correct | | ** M4 ** : N56 MB table | Already present in Appendix (Table B) | | ** M5 ** : Shot-count inconsistency | Disclosed in §3.1 and appendix table caption (d=24 uses 100 shots vs 20 elsewhere) | | ** m1 ** : Monotonicity claim | Text now acknowledges d=18→20 uptick (0.266→0.302), notes overlapping CIs indicate statistical fluctuation | | ** m2 ** : Transport CI values | Table 2 corrected: N=16 CI [0.946,1.000]→[0.967,1.000]; N=24 [0.812,0.944]→[0.867,0.913]; N=32 [0.800,0.936]→[0.814,0.946]; N=40 [0.664,0.840]→[0.663,0.857]; N=48 [0.664,0.840]→[0.658,0.862]; N=56 [0.559,0.752]→[0.520,0.800] | | ** m4 ** : N56 Transport table | Added Appendix Table D with 6 depths, Clopper-Pearson CIs | | ** m6 ** : N40→N56 gap | Exact per-depth gaps now listed, d=16 anomaly (gap=0.057) noted, SPAM vs gate-error interpretation added | ### Verifiable Evidence - ‘report/main.tex‘ -- corrected LaTeX source (line 129: monotonicity fix; lines 185-190: Transport CI fixes; lines 296-317: N56 Transport table; line 219: gap discussion) - ‘report/main.pdf‘ -- compiles clean (11 pages, 0 errors) - ‘auto_research/state/findings.json‘ -- updated with review verification finding (f0_review_verification) and coverage entry ### Correctness Audit Results - XEB depth scan: max delta 0.00028 (within rounding) - Nσ: d=12 = 25.9σ ≈ 26σ [check], d=24 = 23.0σ [check] - Transport CIs: recomputed with t(df=9), truncated to [0,1] - N40-N56 gaps: 0.111, 0.109, 0.089, 0.087, 0.057, 0 Hop 43 · experimenter Task / reported work: Check the new panel and repair its shot-count denominator. Editorial heading inferred from the exported summary, not an assignment quote. Handoff: Recorded next: writer; observed next at hop 44: writer. The two fields agree. Artifact changes: Exported scientific_changes = 6. No file-level diff is provided for this hop. Recorded status: done = false; error = null. Complete available summary ## Experimenter Verification -- Iteration 10 Review Fixes ### Status: Both reviewer issues verified [check], plus one bug found and fixed [bug] ### 1. ‘review:execution:982d33ab5e‘ (Hedge transport claim in Introduction) -- ** VERIFIED ON DISK ** The abstract (line 33) and Introduction (line 51) both use hedged language. The abstract correctly frames the DeCross analysis as " * relied primarily on XEB * " and describes this paper’s contribution as extending with three complementary methods -- no claim that alternatives are "unavailable." The Introduction (line 51) reads: " * evidence that ion transport does not introduce a dominant transport-specific error source beyond standard decoherence scaling, though a small (5-10%) contribution cannot be excluded at the current statistical resolution. * " ### 2. ‘review:figure:b72031b754‘ (Add N56 MB panel to summary figure) -- ** VERIFIED ON DISK ** Panel (c) exists in ‘outputs/summary_figure.pdf‘ showing N=56 MB depth scan (d=8-24) with Transport 1QRB overlay (d=4-96). Figure caption describes panel (c) with correct CI methodology. Body text (§4.2) references panel (c). 30 Systemind Research ### 3. Data-Integrity Bug Found and Fixed [red marker] ** Root cause ** : The panel (c) code hardcoded ‘n_total = n * 20‘ for computing pooled Clopper-Pearson CIs, assuming all MB instances have 20 shots. However, ** d=24 has 100 shots per instance ** (50 instances × 100 = 5000 trials). This inflated the d=24 CI by ~30%. | | Buggy | Correct (fixed) | Appendix B Table | |--|--|--|--| | d=24 MB CI low | 0.1014 | 0.1119 | 0.112 | | d=24 MB CI high | 0.1428 | 0.1302 | 0.130 | ** Fix ** : Rewrote the panel (c) data loader to accumulate actual ‘n_ ideal‘ (successes) and ‘n_total‘ (trials) from each JSONL row, then compute CP CIs from the actual totals. Same fix applied to Transport for robustness. Code changes in ‘code/run_experiments.py‘ lines 801-846. ### Score adjustment: 8.0 → 6.9/10 The hardcoded-shot-count bug is a data-integrity issue -- d=24 in panel (c) is the single most important poi E.2LIFE-001 / ARK / R2 The early trajectory repeatedly passed corrective work from the reviewer directly to the writer. Later hops returned to planning and repeated checks, including checks after a positive internal review. The final benchmark score was 14.10; the archive stopped at the hard hop limit. Table 10: Life-001: all 46 exported hops, summarized chronologically. Work and checks are agent-reported. Hop AgentReported work and outcomeRecorded→ observed next Initial research and report (hops 0–2) 0ResearcherEstablished the vaccine-design project context.Exp. → Exp. 1ExperimenterRan five experiments; reported repairing pivot indexing and verifying 17 output files. Wri. → Wri. 2WriterWrote and compiled the initial manuscript.Rev. → Rev. Direct reviewer–writer corrections (hops 3–20) 3ReviewerReviewed the method–result mismatch and missing evidence.Plan. → Wri. 4WriterAdded the MinSum curve alongside the greedy baseline; revised efficacy scope and numerical claims. Rev. → Rev. 5ReviewerReviewed the first corrections and remaining scope claims.Plan. → Wri. 6WriterQualified feasibility and added replicate variability.Rev. → Rev. 7ReviewerChecked earlier corrections and identified further text issues.Plan. → Wri. 8WriterSoftened runtime scaling claims and revised text.Rev. → Rev. 9ReviewerReviewed quantitative claims against outputs.Plan. → Wri. 10WriterCorrected the count of instances and scoped the budget claim.Rev. → Rev. 11ReviewerReviewed the revised manuscript and underlying cell data.Plan. → Wri. 12WriterChecked low-response cells against source data and replaced a contradicted explanation. Rev. → Rev. 13ReviewerReviewed the remaining runtime claim.Plan. → Wri. 14WriterCorrected runtime from under one second to the measured range. Rev. → Rev. 15ReviewerIdentified MinSum separability and other text issues.Plan. → Wri. 16WriterAcknowledged separable ranking and revised the method description. Rev. → Rev. Continued on the next page 31 Systemind Research Table 10 (continued) Hop AgentReported work and outcomeRecorded→ observed next 17ReviewerReviewed separability disclosure and threshold interpretation.Plan. → Wri. 18WriterCorrected the threshold range from 0.5–0.9 to 0.5–0.7.Rev. → Rev. 19ReviewerReviewed the threshold fix and remaining terminology.Plan. → Wri. 20WriterResolved three text-level issues from review iteration 9.Rev. → Rev. Renewed planning and checks (hops 21–28) 21ReviewerReported no remaining issues and a borderline-accept host-review verdict; execution nevertheless continued. Plan. → Plan. 22PlannerPrepared an action plan and audited protocol coverage.Exp. → Exp. 23ExperimenterAudited findings against experiment outputs.Wri. → Wri. 24WriterReported an empty remediation list and no file modifications after rechecking the paper. Rev. → Rev. 25ReviewerReviewed the manuscript again and identified further qualifications. Plan. → Plan. 26PlannerPrepared the next action plan.Exp. → Exp. 27ExperimenterApplied writing-only edits as the experimenter; reported running no new experiments. Wri. → Wri. 28WriterChecked corrections already made by the experimenter; reported no further writer edits were needed. Rev. → Rev. Further textual revisions and final review (hops 29–45) 29ReviewerReviewed the updated claims and comparisons.Plan. → Wri. 30WriterQualified dominance language and applied three fixes.Rev. → Rev. 31ReviewerReviewed title, claims and remaining issues.Plan. → Wri. 32WriterResolved six issues including the coverage-guarantee title.Rev. → Rev. 33ReviewerReviewed the next manuscript revision.Plan. → Wri. 34WriterApplied six review-requested corrections.Rev. → Rev. 35ReviewerReviewed consistency after the corrections.Plan. → Wri. 36WriterResolved instance labels and related textual issues.Rev. → Rev. 37ReviewerChecked prior fixes and flagged unsupported parameterization.Plan. → Wri. 38WriterRevised the unsupported instance-parameterization claim.Rev. → Rev. 39ReviewerReviewed the revised manuscript and presentation.Plan. → Wri. 40WriterResolved page-length and four further issues.Rev. → Rev. 41ReviewerReviewed the revised runtime framing.Plan. → Wri. 42WriterReplaced linear-runtime claims with sub-quadratic wording.Rev. → Rev. 43ReviewerReviewed remaining baseline and statistical descriptions.Plan. → Wri. 44WriterAdded a justification for omitting top-k ranking, clarified statistical tests, and reduced abstract density. Rev. → Rev. 45ReviewerConfirmed prior fixes but flagged an unresolved abstract confidence-interval claim. Plan. → – Hop 12 reports replacing a speculative explanation with a data check. Hop 24 reports no remaining remediation items and no modifications, but the all-hop table shows that execution continued. Hop 45 records a remaining confidence-interval issue at the end of the exported trajectory. 32 Systemind Research Hop 12 · writer Task / reported work: Replace the low-response-cell speculation with a data check. Editorial heading inferred from the exported summary, not an assignment quote. Handoff: Recorded next: reviewer; observed next at hop 13: reviewer. The two fields agree. Artifact changes: Exported scientific_changes = 4. No file-level diff is provided for this hop. Recorded status: done = false; error = null. Complete available summary The compilation confirms: - Body ends at ‘[9]‘ (page 9), then ‘.bbl‘ loads - 12 pages total (9 body + 3 appendix/references), clean, no errors The fix is complete. Here’s a summary of what was done: -- ## Summary ** Task m1 -- Fix E2 speculation at L147-148 ** : The original text claimed the 8 low-response cells "likely represent a subclone whose presented peptides derive exclusively from a mutation not included in the vaccine." I cross-tabulated every low-response cell against its per-peptide scores in the raw data (‘vaccine-elements.scores.100-cells.10x.rep- * .csv‘) and found the claim was not just unsupported -- it was ** contradicted by the data ** : - ** 5 of 8 cells ** have uniformly near-zero scores across all 12 candidates (p ≈ 2×10 -6 for every peptide), meaning no vaccine composition could cover them. - ** The remaining 3 cells ** have their highest individual scores from ** included ** elements (mut26, mut44) -- not excluded ones -- but aggregate response still falls below 0.5. ** Fix applied ** : Replaced the speculative claim with a data-grounded factual description at L147: > "Of these 8 cells, 5 exhibit uniformly near-zero per-peptide response probabilities across all 12 candidates and would remain uncovered under any vaccine composition; the remaining 3 have moderate individual scores from included elements but insufficient aggregate response to clear the threshold." All values are verified against the source CSVs. PDF compiles successfully (12 pages, body ends on page 9). Hop 24 · writer Task / reported work: Check a plan with no listed remediation issues. Editorial heading inferred from the exported summary, not an assignment quote. Handoff: Recorded next: reviewer; observed next at hop 25: reviewer. The two fields agree. Artifact changes: Exported scientific_changes = 0. No file-level diff is provided for this hop. Recorded status: done = false; error = null. Complete available summary ## Summary The action plan (‘auto_research/state/action_plan.yaml‘) explicitly states ** ‘issues: []‘ ** -- no issues to remediate. The Iteration 10 reviewer gave a ** Borderline Accept (7.5/10) ** verdict with: "No further revisions needed. The manuscript is submission-ready." I verified the paper is complete and correct: ### Compilation - ** 12 pages ** , body ends on ** page 8 ** (well within 9-page limit) - Compiles cleanly with ‘pdflatex‘ + ‘bibtex‘ -- no errors - Only cosmetic underfull box warnings (typical for LaTeX) ### Quantitative Claim Verification 33 Systemind Research Every number in the paper cross-checked against result files: | Claim | Source | Verified | |--|--|--| | Greedy coverage: 55.7% [52.6%, 58.8%] at k=1 | ‘coverage_by_ budget_aggregated.csv‘ | [check] | | Greedy plateau: 96.4% [95.1%, 97.5%] at k=6+ | ‘coverage_by_ budget_aggregated.csv‘ | [check] | | MinSum k=4: 97.9%, k=10: 99.2% [98.4%, 99.7%] | ‘coverage_by_ budget_minsum.csv‘ | [check] | | Mean P_i = 0.943, median = 0.963, IQR = 0.047 | ‘cell_response_ summary.json‘ | [check] | | 99.2% > 0.5, 88.7% > 0.9 | ‘cell_response_summary.json‘ | [check] | | 5 near-zero cells across reps 0, 1, 5, 8 | ‘cell_response_ distribution.csv‘ | [check] | | All 45 IoU = 1.0 | ‘vaccine_stability_iou.csv‘ | [check] | | β range 0.99-1.53, mean 1.23, all R 2 ≥ 0.94 | ‘runtime_fit_ results.csv‘ | [check] | | MinSum 99.2% vs Greedy 96.4% (5 wins, 5 ties) | ‘objective_ comparison.csv‘ | [check] | | Random 91.9%, p=0.002 Wilcoxon | ‘objective_comparison.csv‘ | [check] | ### Figure Placement All 8 figures are present in ‘report/images/‘ and correctly referenced: - Body: ‘fig_coverage_curve.pdf‘, ‘fig_response_distribution.pdf‘, ‘fig_iou_heatmap.pdf‘, ‘fig_runtime_scaling.pdf‘, ‘fig_objective_ comparison.pdf‘ - Appendix: ‘fig_coverage_curve_per_rep.pdf‘, ‘fig_response_ distribution_per_rep.pdf‘, ‘fig_runtime_exponents.pdf‘ ### No Modifications Made The action plan has zero issues. The reviewer and planner both say the paper is submission-ready. No files were modified. Hop 45 · reviewer Task / reported work: Final recorded review: identify a remaining abstract CI ambiguity. Editorial heading inferred from the exported summary, not an assignment quote. Handoff: Recorded next: planner; no subsequent hop in this selected export. Do not infer execution of that successor. Artifact changes: Exported scientific_changes = 0. No file-level diff is provided for this hop. Recorded status: done = false; error = null. Complete available summary ## Review Complete -- Iteration 20 ** Rating: Borderline Accept (7.3/10) ** ### Summary I thoroughly reviewed all 12 PDF pages and the full LaTeX source. The paper has reached a high level of polish. All four issues from Iteration 19 are ** resolved ** : - ** M1 ** : Top-k baseline justification now lives in the E5 body (line 210) with explicit appendix cross-reference [check] - ** m1 ** : "near-linear" survivor at line 183 → now reads "sub-quadratic" [check] - ** m2 ** : Implementation/Checklist now correctly describes mixed Wilcoxon/sign-test methodology (lines 121, 263, 267) [check] - ** m3 ** : Abstract numerical density reduced from ~18 to ~10 claims [check] ** One new minor issue identified (m1) ** : The abstract’s claim "95% CI excludes β=1.0" is ambiguous. Per-instance CIs for instances 1 and 2 include 1.0, and the paper never computes a CI for the mean β. 34 Systemind Research The body text correctly reports "95% CIs exclude β=2.0 for six of seven instances" -- the abstract should match this claim or compute the mean CI. ** Anonymity ** : Clean -- Anonymous Author(s)/Anonymous Institution, required ARK disclosure present, no identity leaks. ** Figures ** : All clean, readable, properly scaled. Coverage curve, response distribution, IoU heatmap, runtime scaling, and objective comparison all present and well-formatted. ** Protocol alignment ** : All 5 experiments delivered. Single-model limitation disclosed honestly. Findings.json surprises (perfect IoU, sub-quadratic runtime, single-population data) properly surfaced in the paper. ** No placeholder citations found ** -- all references appear to be real publications. The review is saved to ‘auto_research/state/latest_review.md‘. FGENERATIVE AI USE DISCLOSURE Generative AI systems are intrinsic experimental components of the evaluated AI-scientist work- flows. They generate plans, code, experimental analyses, reviews, and manuscript drafts inside each evaluated system. Human authors are responsible for the research question, system design, evaluation protocol, verification of reported values, literature checking, interpretation, and final manuscript. All AI-assisted content was checked against persisted artifacts and source records be- fore submission. 35